2024-09-10T18:24:03.024 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T18:24:03.040 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T18:24:03.111 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236 branch: wip-guits-quincy-2024-09-09-1810-quincy description: orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7900236' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-guits-quincy-2024-09-09-1810-quincy ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) mon_bind_msgr2: false sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 workunit: branch: quincy-orch-centos-9-teuth sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4625 sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: quincy-orch-centos-9-teuth suite_path: /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa suite_relpath: qa suite_repo: https://github.com/adk3798/ceph.git suite_sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:53:37 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T18:24:03.111 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa; will attempt to use it 2024-09-10T18:24:03.112 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa/tasks 2024-09-10T18:24:03.112 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T18:24:03.113 INFO:teuthology.task.internal:Checking packages... 2024-09-10T18:24:03.134 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5be57395ea235535910512ae73ba8e7ff25fe588' 2024-09-10T18:24:03.134 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T18:24:03.134 INFO:teuthology.packaging:ref: None 2024-09-10T18:24:03.134 INFO:teuthology.packaging:tag: None 2024-09-10T18:24:03.134 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:24:03.134 INFO:teuthology.packaging:sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:24:03.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:24:03.276 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1606-g5be57395-1focal 2024-09-10T18:24:03.278 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T18:24:03.287 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T18:24:03.287 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T18:24:03.309 INFO:teuthology.task.internal:Saving configuration 2024-09-10T18:24:03.321 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T18:24:03.333 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T18:24:03.350 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:17:49.905592', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-10T18:24:03.366 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:17:49.906901', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-10T18:24:03.366 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T18:24:03.378 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-10T18:24:03.379 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-10T18:24:03.379 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T18:24:03.430 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fec4566fac0>, signals=[15]) 2024-09-10T18:24:03.430 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T18:24:03.439 INFO:teuthology.task.internal:Opening connections... 2024-09-10T18:24:03.439 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-09-10T18:24:03.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:24:03.536 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T18:24:03.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:24:03.624 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T18:24:03.633 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-09-10T18:24:03.640 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-09-10T18:24:03.640 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="Ubuntu" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:ID=ubuntu 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE=debian 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="20.04" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-10T18:24:03.691 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_CODENAME=focal 2024-09-10T18:24:03.692 INFO:teuthology.orchestra.run.smithi083.stdout:UBUNTU_CODENAME=focal 2024-09-10T18:24:03.692 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-09-10T18:24:03.714 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-10T18:24:03.719 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-10T18:24:03.720 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="Ubuntu" 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:ID=ubuntu 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE=debian 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-10T18:24:03.766 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="20.04" 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_CODENAME=focal 2024-09-10T18:24:03.767 INFO:teuthology.orchestra.run.smithi165.stdout:UBUNTU_CODENAME=focal 2024-09-10T18:24:03.768 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-10T18:24:03.789 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T18:24:03.801 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T18:24:03.813 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T18:24:03.813 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:24:03.815 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:24:03.821 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T18:24:03.831 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T18:24:03.831 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:24:03.834 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:24:03.884 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T18:24:03.932 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T18:24:03.947 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T18:24:03.947 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T18:24:03.947 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:24:03.948 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:24:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.953 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-10T18:24:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:24:03.954 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-10T18:24:03.960 INFO:teuthology.orchestra.run.smithi083.stdout:5.4.0-124-generic 2024-09-10T18:24:03.961 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.4.0-124-generic 2024-09-10T18:24:03.961 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2024-09-10T18:24:03.962 INFO:teuthology.orchestra.run.smithi165.stdout:5.4.0-124-generic 2024-09-10T18:24:03.962 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.4.0-124-generic 2024-09-10T18:24:03.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-10T18:24:04.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-10T18:24:04.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-10T18:24:04.262 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:24:04.263 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-10T18:24:04.263 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-10T18:24:04.338 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-10T18:24:04.392 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:24:04.392 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-10T18:24:04.431 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-10T18:24:04.653 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,166 kB] 2024-09-10T18:24:04.777 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [807 kB] 2024-09-10T18:24:04.786 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-10T18:24:04.790 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [469 kB] 2024-09-10T18:24:04.790 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,550 kB] 2024-09-10T18:24:04.796 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-10T18:24:04.796 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,082 kB] 2024-09-10T18:24:04.818 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-10T18:24:04.819 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [431 kB] 2024-09-10T18:24:04.822 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-10T18:24:04.822 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [680 kB] 2024-09-10T18:24:04.827 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,009 kB] 2024-09-10T18:24:04.833 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-10T18:24:04.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-10T18:24:04.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-10T18:24:04.835 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-10T18:24:04.841 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-10T18:24:04.847 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-10T18:24:04.869 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [807 kB] 2024-09-10T18:24:04.896 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,027 kB] 2024-09-10T18:24:04.904 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [548 kB] 2024-09-10T18:24:04.908 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-10T18:24:04.908 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-10T18:24:04.908 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,215 kB] 2024-09-10T18:24:04.930 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [450 kB] 2024-09-10T18:24:04.934 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-10T18:24:04.934 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-10T18:24:04.939 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-10T18:24:04.947 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-10T18:24:04.950 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-10T18:24:04.950 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-10T18:24:04.950 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-10T18:24:04.950 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-10T18:24:04.956 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-10T18:24:04.956 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-10T18:24:04.956 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-10T18:24:04.973 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-10T18:24:04.973 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-10T18:24:04.974 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-10T18:24:04.974 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-10T18:24:05.007 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,027 kB] 2024-09-10T18:24:05.230 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,166 kB] 2024-09-10T18:24:05.335 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,550 kB] 2024-09-10T18:24:05.422 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [469 kB] 2024-09-10T18:24:05.436 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-10T18:24:05.436 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-10T18:24:05.438 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,082 kB] 2024-09-10T18:24:05.515 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [548 kB] 2024-09-10T18:24:05.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [431 kB] 2024-09-10T18:24:05.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-10T18:24:05.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-10T18:24:05.529 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,215 kB] 2024-09-10T18:24:05.542 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-10T18:24:05.542 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,009 kB] 2024-09-10T18:24:05.569 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [680 kB] 2024-09-10T18:24:05.591 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-10T18:24:05.594 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-10T18:24:05.594 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-10T18:24:05.594 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-10T18:24:05.599 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-10T18:24:05.623 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [450 kB] 2024-09-10T18:24:05.636 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-10T18:24:05.636 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-10T18:24:05.665 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-10T18:24:05.667 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-10T18:24:05.698 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-10T18:24:05.701 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-10T18:24:05.701 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-10T18:24:05.702 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-10T18:24:05.702 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-10T18:24:05.760 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-10T18:24:05.761 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-10T18:24:05.762 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-10T18:24:05.763 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-10T18:24:05.842 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-10T18:24:05.842 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-10T18:24:05.843 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-10T18:24:09.240 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 21.7 MB in 3s (7,230 kB/s) 2024-09-10T18:24:09.793 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 21.7 MB in 4s (6,105 kB/s) 2024-09-10T18:24:10.820 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:24:10.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T18:24:10.922 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:24:11.144 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-10T18:24:11.145 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-10T18:24:11.338 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-10T18:24:11.338 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-10T18:24:11.338 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-10T18:24:11.339 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-10T18:24:11.339 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-10T18:24:11.340 INFO:teuthology.orchestra.run.smithi165.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-10T18:24:11.340 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-10T18:24:11.340 INFO:teuthology.orchestra.run.smithi165.stdout: thermald 2024-09-10T18:24:11.393 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-10T18:24:11.393 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-10T18:24:11.393 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-10T18:24:11.393 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-10T18:24:11.395 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2024-09-10T18:24:11.395 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-10T18:24:11.482 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:24:11.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T18:24:11.577 INFO:teuthology.orchestra.run.smithi165.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-10T18:24:11.577 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 77.2 MB of archives. 2024-09-10T18:24:11.577 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-10T18:24:11.578 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-10T18:24:11.586 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:24:11.803 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-10T18:24:11.804 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-10T18:24:12.004 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2024-09-10T18:24:12.004 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-10T18:24:12.004 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-10T18:24:12.004 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-10T18:24:12.005 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2024-09-10T18:24:12.005 INFO:teuthology.orchestra.run.smithi083.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-10T18:24:12.006 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2024-09-10T18:24:12.006 INFO:teuthology.orchestra.run.smithi083.stdout: thermald 2024-09-10T18:24:12.052 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2024-09-10T18:24:12.052 INFO:teuthology.orchestra.run.smithi083.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-10T18:24:12.053 INFO:teuthology.orchestra.run.smithi083.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-10T18:24:12.053 INFO:teuthology.orchestra.run.smithi083.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-10T18:24:12.054 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2024-09-10T18:24:12.055 INFO:teuthology.orchestra.run.smithi083.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-10T18:24:12.107 INFO:teuthology.orchestra.run.smithi083.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-10T18:24:12.107 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 77.2 MB of archives. 2024-09-10T18:24:12.107 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-10T18:24:12.107 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-10T18:24:12.442 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-10T18:24:12.533 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-10T18:24:12.741 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-10T18:24:13.020 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-10T18:24:13.020 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-10T18:24:13.020 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-10T18:24:13.064 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-10T18:24:13.113 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-10T18:24:13.124 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-10T18:24:13.882 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 77.2 MB in 1s (72.7 MB/s) 2024-09-10T18:24:14.064 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-10T18:24:14.325 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-10T18:24:14.325 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-10T18:24:14.325 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-10T18:24:14.607 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-10T18:24:14.620 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-10T18:24:15.346 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 77.2 MB in 3s (24.0 MB/s) 2024-09-10T18:24:15.531 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-10T18:24:15.817 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-10T18:24:15.822 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:15.945 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:17.261 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-10T18:24:17.265 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:17.413 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:18.054 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-10T18:24:18.067 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:18.145 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:19.290 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-10T18:24:19.305 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:19.348 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:19.355 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-10T18:24:19.371 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:19.472 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:20.525 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-10T18:24:20.538 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:20.566 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:24.355 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:24.479 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:24.840 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:24.974 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:25.319 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-10T18:24:25.332 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-10T18:24:25.385 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-10T18:24:25.431 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:25.558 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:25.949 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:26.069 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:26.398 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-10T18:24:26.413 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-10T18:24:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-10T18:24:30.937 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-10T18:24:30.959 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:31.012 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:32.369 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-10T18:24:32.390 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-10T18:24:32.435 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:33.748 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:33.903 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:34.267 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:35.234 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-10T18:24:35.387 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-10T18:24:35.509 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:35.744 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:36.658 INFO:teuthology.orchestra.run.smithi083.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:24:36.658 INFO:teuthology.orchestra.run.smithi083.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-10T18:24:36.828 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:36.942 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:37.996 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-10T18:24:38.164 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-10T18:24:38.290 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:38.416 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-10T18:24:38.493 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:24:38.493 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-10T18:24:38.551 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-10T18:24:38.652 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:38.702 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-10T18:24:38.702 INFO:teuthology.orchestra.run.smithi083.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:24:38.985 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:40.162 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-10T18:24:40.288 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-10T18:24:40.439 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:40.564 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-10T18:24:40.716 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-10T18:24:40.842 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-10T18:24:40.909 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-10T18:24:40.910 INFO:teuthology.orchestra.run.smithi165.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:25:01.318 INFO:teuthology.orchestra.run.smithi083.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-10T18:25:01.319 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub' 2024-09-10T18:25:01.339 INFO:teuthology.orchestra.run.smithi083.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T18:25:01.368 INFO:teuthology.orchestra.run.smithi083.stdout:Generating grub configuration file ... 2024-09-10T18:25:01.707 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:25:01.730 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:25:02.313 INFO:teuthology.orchestra.run.smithi083.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-10T18:25:02.325 INFO:teuthology.orchestra.run.smithi083.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-10T18:25:02.715 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-10T18:25:02.715 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub' 2024-09-10T18:25:02.730 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T18:25:02.755 INFO:teuthology.orchestra.run.smithi165.stdout:Generating grub configuration file ... 2024-09-10T18:25:03.143 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:25:03.165 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:25:03.641 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-10T18:25:03.652 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-10T18:25:04.998 INFO:teuthology.orchestra.run.smithi083.stdout:done 2024-09-10T18:25:05.466 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 20 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2024-09-10T18:25:05.509 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.4.0.195.193 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2024-09-10T18:25:05.510 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2024-09-10T18:25:05.510 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-10T18:25:05.510 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:25:05.510 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:25:05.511 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T18:25:05.511 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:25:05.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get clean 2024-09-10T18:25:05.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-10T18:25:05.794 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-10T18:25:05.794 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:25:05.801 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-10T18:25:05.818 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-10T18:25:06.072 INFO:teuthology.orchestra.run.smithi165.stdout:done 2024-09-10T18:25:06.494 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 20 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-10T18:25:06.556 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.4.0.195.193 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-10T18:25:06.557 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-10T18:25:06.557 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-10T18:25:06.557 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:25:06.557 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:25:06.557 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T18:25:06.557 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:25:06.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2024-09-10T18:25:06.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-10T18:25:06.766 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:25:06.773 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-10T18:25:06.790 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-10T18:25:06.885 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-10T18:25:07.634 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:25:07.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T18:25:07.729 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:25:07.943 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-10T18:25:07.944 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-10T18:25:08.182 INFO:teuthology.orchestra.run.smithi083.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-10T18:25:08.182 INFO:teuthology.orchestra.run.smithi083.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-10T18:25:08.184 DEBUG:teuthology.orchestra.run.smithi083:> dpkg -s linux-image-generic 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Package: linux-image-generic 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Status: install ok installed 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Priority: optional 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Section: kernel 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Installed-Size: 20 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: amd64 2024-09-10T18:25:08.204 INFO:teuthology.orchestra.run.smithi083.stdout:Source: linux-meta 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout:Version: 5.4.0.195.193 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout:Recommends: thermald 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout:Description: Generic Linux kernel image 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout: This package will always depend on the latest generic kernel image 2024-09-10T18:25:08.205 INFO:teuthology.orchestra.run.smithi083.stdout: available. 2024-09-10T18:25:08.205 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-10T18:25:08.251 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.VFKE7tpCCo 2024-09-10T18:25:08.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VFKE7tpCCo 2024-09-10T18:25:08.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.VFKE7tpCCo 2024-09-10T18:25:08.309 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.VFKE7tpCCo is 9KB 2024-09-10T18:25:08.322 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.VFKE7tpCCo 2024-09-10T18:25:08.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-10T18:25:08.383 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:25:08.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-10T18:25:08.384 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-10T18:25:08.414 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-10T18:25:08.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo update-grub 2024-09-10T18:25:08.623 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:25:08.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T18:25:08.717 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-10T18:25:08.944 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-10T18:25:09.189 INFO:teuthology.orchestra.run.smithi165.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-10T18:25:09.189 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-10T18:25:09.191 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 20 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T18:25:09.213 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.4.0.195.193 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2024-09-10T18:25:09.214 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2024-09-10T18:25:09.215 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-10T18:25:09.221 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.NSywRjYmXS 2024-09-10T18:25:09.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NSywRjYmXS 2024-09-10T18:25:09.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.NSywRjYmXS 2024-09-10T18:25:09.321 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.NSywRjYmXS is 9KB 2024-09-10T18:25:09.334 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.NSywRjYmXS 2024-09-10T18:25:09.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-10T18:25:09.396 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:25:09.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-10T18:25:09.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-10T18:25:09.430 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-10T18:25:09.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo update-grub 2024-09-10T18:25:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub' 2024-09-10T18:25:09.940 INFO:teuthology.orchestra.run.smithi083.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T18:25:09.950 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-09-10T18:25:10.192 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:25:10.204 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:25:10.556 INFO:teuthology.orchestra.run.smithi083.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-10T18:25:10.568 INFO:teuthology.orchestra.run.smithi083.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-10T18:25:10.885 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub' 2024-09-10T18:25:10.892 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T18:25:10.903 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-10T18:25:11.169 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-10T18:25:11.181 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-10T18:25:11.609 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-10T18:25:11.621 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-10T18:25:12.174 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-09-10T18:25:12.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-09-10T18:25:13.027 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-10T18:25:13.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-10T18:25:42.205 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T18:25:42.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-10T18:25:42.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:25:43.034 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T18:25:43.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:25:43.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:26:17.537 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-10T18:26:26.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:26:26.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:26:42.225 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T18:26:51.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-10T18:26:51.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:26:51.632 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-10T18:26:52.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-10T18:26:52.516 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-10T18:26:52.516 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-10T18:26:52.522 INFO:teuthology.orchestra.run.smithi083.stdout:5.4.0-195-generic 2024-09-10T18:26:52.522 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-10T18:26:52.522 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:26:52.522 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-10T18:26:53.523 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T18:26:53.523 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:26:53.672 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-09-10T18:26:53.702 DEBUG:teuthology.parallel:result is None 2024-09-10T18:27:26.549 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T18:27:38.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:27:38.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:27:38.804 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-10T18:27:39.392 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:27:39.392 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-10T18:27:39.392 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-10T18:27:39.399 INFO:teuthology.orchestra.run.smithi165.stdout:5.4.0-195-generic 2024-09-10T18:27:39.399 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-10T18:27:39.399 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:27:39.399 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-10T18:27:40.400 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T18:27:40.400 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:27:40.487 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-10T18:27:40.517 DEBUG:teuthology.parallel:result is None 2024-09-10T18:27:40.517 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T18:27:40.525 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T18:27:40.526 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:27:40.528 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:27:40.535 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T18:27:40.561 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T18:27:40.570 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T18:27:40.570 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:27:40.573 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:27:40.598 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T18:27:40.604 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T18:27:40.604 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:27:40.629 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:27:40.652 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:27:40.662 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:27:40.663 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:27:40.674 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:27:40.676 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T18:27:40.682 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T18:27:40.683 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:27:40.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:27:40.699 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T18:27:40.708 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T18:27:40.708 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:27:40.710 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:27:40.717 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:27:40.762 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:27:40.769 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:27:40.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:27:40.792 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:27:40.800 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:27:40.807 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:27:40.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:27:40.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-09-10T18:27:40.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-10T18:27:40.876 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T18:27:40.883 INFO:teuthology.task.internal:Starting timer... 2024-09-10T18:27:40.884 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T18:27:40.897 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T18:27:40.905 DEBUG:teuthology.task.selinux:Excluding smithi083: OS 'ubuntu' does not support SELinux 2024-09-10T18:27:40.905 DEBUG:teuthology.task.selinux:Excluding smithi165: OS 'ubuntu' does not support SELinux 2024-09-10T18:27:40.905 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T18:27:40.905 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-10T18:27:40.906 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T18:27:40.906 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T18:27:40.914 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T18:27:40.925 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T18:27:40.926 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T18:33:16.269 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-09-10T18:33:16.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-10T18:33:16.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:33:16.361 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-10T18:33:16.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-10T18:33:16.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:33:16.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:33:16.575 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-10T18:33:16.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T18:33:16.703 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T18:33:16.713 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T18:33:16.713 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:33:16.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:33:16.716 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:33:16.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:33:16.748 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-10T18:33:16.748 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Command line: ntpd -gq 2024-09-10T18:33:16.749 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: proto: precision = 0.066 usec (-24) 2024-09-10T18:33:16.749 INFO:teuthology.orchestra.run.smithi083.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T18:33:16.750 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T18:33:16.750 INFO:teuthology.orchestra.run.smithi083.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-10T18:33:16.752 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: restrict ::: KOD does nothing without LIMITED. 2024-09-10T18:33:16.752 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen and drop on 0 v6wildcard [::]:123 2024-09-10T18:33:16.752 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-10T18:33:16.752 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen normally on 2 lo 127.0.0.1:123 2024-09-10T18:33:16.753 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen normally on 3 enp3s0f1 172.21.15.83:123 2024-09-10T18:33:16.753 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen normally on 4 lo [::1]:123 2024-09-10T18:33:16.753 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce91%5]:123 2024-09-10T18:33:16.753 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:16 ntpd[13702]: Listening on routing socket on fd #22 for interface updates 2024-09-10T18:33:16.754 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-10T18:33:16.754 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Command line: ntpd -gq 2024-09-10T18:33:16.755 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: proto: precision = 0.138 usec (-23) 2024-09-10T18:33:16.755 INFO:teuthology.orchestra.run.smithi165.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T18:33:16.755 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T18:33:16.756 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: restrict ::: KOD does nothing without LIMITED. 2024-09-10T18:33:16.756 INFO:teuthology.orchestra.run.smithi165.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen and drop on 0 v6wildcard [::]:123 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen normally on 2 lo 127.0.0.1:123 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen normally on 3 enp3s0f1 172.21.15.165:123 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen normally on 4 lo [::1]:123 2024-09-10T18:33:16.757 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f63%5]:123 2024-09-10T18:33:16.758 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:16 ntpd[13716]: Listening on routing socket on fd #22 for interface updates 2024-09-10T18:33:17.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:17.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:17 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:17.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:17.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:17.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:17.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:17.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:17 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:19.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:19.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:19 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:19.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:19.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:19.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:19.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:19.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:19 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:20.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:20.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:20.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:20.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:20.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:20.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:20 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:21.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:21 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:21.755 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:21.755 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:21 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:22.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:22.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:22.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:22.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:22.751 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:22.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:22 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:23.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.749 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:10 Sep 18:33:23 ntpd[13702]: ntpd: time slew -0.002083 s 2024-09-10T18:33:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:ntpd: time slew -0.002083s 2024-09-10T18:33:23.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-10T18:33:23.751 INFO:teuthology.orchestra.run.smithi083.stderr:10 Sep 18:33:23 ntpd[13702]: can't open /var/log/ntpstats/loopstats.20240910: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T18:33:23.752 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T18:33:23.753 INFO:teuthology.orchestra.run.smithi165.stdout:10 Sep 18:33:23 ntpd[13716]: ntpd: time slew +0.002963 s 2024-09-10T18:33:23.753 INFO:teuthology.orchestra.run.smithi165.stdout:ntpd: time slew +0.002963s 2024-09-10T18:33:23.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-10T18:33:23.753 INFO:teuthology.orchestra.run.smithi165.stderr:10 Sep 18:33:23 ntpd[13716]: can't open /var/log/ntpstats/loopstats.20240910: Permission denied 2024-09-10T18:33:23.821 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T18:33:23.821 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-10T18:33:23.822 INFO:teuthology.orchestra.run.smithi165.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.822 INFO:teuthology.orchestra.run.smithi165.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.822 INFO:teuthology.orchestra.run.smithi165.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.822 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.823 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T18:33:23.823 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2024-09-10T18:33:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.824 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-10T18:33:23.824 INFO:teuthology.run_tasks:Running task install... 2024-09-10T18:33:23.835 DEBUG:teuthology.task.install:project ceph 2024-09-10T18:33:23.835 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}} 2024-09-10T18:33:23.836 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'} 2024-09-10T18:33:23.836 INFO:teuthology.task.install:Using flavor: default 2024-09-10T18:33:23.842 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T18:33:23.842 INFO:teuthology.task.install:extra packages: [] 2024-09-10T18:33:23.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-key list | grep Ceph 2024-09-10T18:33:23.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-key list | grep Ceph 2024-09-10T18:33:23.936 INFO:teuthology.orchestra.run.smithi083.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-10T18:33:23.962 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-10T18:33:23.996 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-10T18:33:23.996 INFO:teuthology.orchestra.run.smithi083.stdout:uid [ unknown] Ceph.com (release key) 2024-09-10T18:33:23.997 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-10T18:33:23.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:33:24.041 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-10T18:33:24.041 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph.com (release key) 2024-09-10T18:33:24.042 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-10T18:33:24.042 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:33:24.139 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default/ 2024-09-10T18:33:24.139 INFO:teuthology.task.install.deb:Package version is 17.2.7-1606-g5be57395-1focal 2024-09-10T18:33:24.182 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default/ 2024-09-10T18:33:24.182 INFO:teuthology.task.install.deb:Package version is 17.2.7-1606-g5be57395-1focal 2024-09-10T18:33:24.253 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:33:24.253 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-10T18:33:24.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo apt-get update 2024-09-10T18:33:24.307 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:33:24.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-10T18:33:24.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2024-09-10T18:33:24.408 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:33:24.409 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-10T18:33:24.416 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-10T18:33:24.433 INFO:teuthology.orchestra.run.smithi083.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-10T18:33:24.464 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal InRelease 2024-09-10T18:33:24.485 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-10T18:33:24.510 INFO:teuthology.orchestra.run.smithi083.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal Release.gpg 2024-09-10T18:33:24.532 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main i386 Packages [3,376 B] 2024-09-10T18:33:24.563 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal InRelease 2024-09-10T18:33:24.565 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-10T18:33:24.585 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-10T18:33:24.608 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal Release.gpg 2024-09-10T18:33:24.636 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-10T18:33:24.637 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-10T18:33:24.637 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-10T18:33:24.669 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main i386 Packages [3,376 B] 2024-09-10T18:33:24.706 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-10T18:33:24.787 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-10T18:33:25.066 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 28.4 kB in 1s (45.2 kB/s) 2024-09-10T18:33:25.289 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 28.4 kB in 1s (37.3 kB/s) 2024-09-10T18:33:26.210 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:33:26.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1606-g5be57395-1focal cephadm=17.2.7-1606-g5be57395-1focal ceph-mds=17.2.7-1606-g5be57395-1focal ceph-mgr=17.2.7-1606-g5be57395-1focal ceph-common=17.2.7-1606-g5be57395-1focal ceph-fuse=17.2.7-1606-g5be57395-1focal ceph-test=17.2.7-1606-g5be57395-1focal ceph-volume=17.2.7-1606-g5be57395-1focal radosgw=17.2.7-1606-g5be57395-1focal python3-rados=17.2.7-1606-g5be57395-1focal python3-rgw=17.2.7-1606-g5be57395-1focal python3-cephfs=17.2.7-1606-g5be57395-1focal python3-rbd=17.2.7-1606-g5be57395-1focal libcephfs2=17.2.7-1606-g5be57395-1focal libcephfs-dev=17.2.7-1606-g5be57395-1focal librados2=17.2.7-1606-g5be57395-1focal librbd1=17.2.7-1606-g5be57395-1focal rbd-fuse=17.2.7-1606-g5be57395-1focal 2024-09-10T18:33:26.306 INFO:teuthology.orchestra.run.smithi083.stdout:Reading package lists... 2024-09-10T18:33:26.448 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:33:26.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1606-g5be57395-1focal cephadm=17.2.7-1606-g5be57395-1focal ceph-mds=17.2.7-1606-g5be57395-1focal ceph-mgr=17.2.7-1606-g5be57395-1focal ceph-common=17.2.7-1606-g5be57395-1focal ceph-fuse=17.2.7-1606-g5be57395-1focal ceph-test=17.2.7-1606-g5be57395-1focal ceph-volume=17.2.7-1606-g5be57395-1focal radosgw=17.2.7-1606-g5be57395-1focal python3-rados=17.2.7-1606-g5be57395-1focal python3-rgw=17.2.7-1606-g5be57395-1focal python3-cephfs=17.2.7-1606-g5be57395-1focal python3-rbd=17.2.7-1606-g5be57395-1focal libcephfs2=17.2.7-1606-g5be57395-1focal libcephfs-dev=17.2.7-1606-g5be57395-1focal librados2=17.2.7-1606-g5be57395-1focal librbd1=17.2.7-1606-g5be57395-1focal rbd-fuse=17.2.7-1606-g5be57395-1focal 2024-09-10T18:33:26.519 INFO:teuthology.orchestra.run.smithi083.stdout:Building dependency tree... 2024-09-10T18:33:26.520 INFO:teuthology.orchestra.run.smithi083.stdout:Reading state information... 2024-09-10T18:33:26.553 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2024-09-10T18:33:26.709 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T18:33:26.710 INFO:teuthology.orchestra.run.smithi083.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-10T18:33:26.710 INFO:teuthology.orchestra.run.smithi083.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T18:33:26.710 INFO:teuthology.orchestra.run.smithi083.stdout:The following additional packages will be installed: 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-10T18:33:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-10T18:33:26.712 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: socat xmlstarlet 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout:Suggested packages: 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-10T18:33:26.713 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout:Recommended packages: 2024-09-10T18:33:26.714 INFO:teuthology.orchestra.run.smithi083.stdout: btrfs-tools 2024-09-10T18:33:26.770 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2024-09-10T18:33:26.771 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2024-09-10T18:33:26.831 INFO:teuthology.orchestra.run.smithi083.stdout:The following NEW packages will be installed: 2024-09-10T18:33:26.831 INFO:teuthology.orchestra.run.smithi083.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-10T18:33:26.831 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-10T18:33:26.832 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-10T18:33:26.832 INFO:teuthology.orchestra.run.smithi083.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-10T18:33:26.832 INFO:teuthology.orchestra.run.smithi083.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-10T18:33:26.832 INFO:teuthology.orchestra.run.smithi083.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-10T18:33:26.832 INFO:teuthology.orchestra.run.smithi083.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-10T18:33:26.833 INFO:teuthology.orchestra.run.smithi083.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout:The following packages will be upgraded: 2024-09-10T18:33:26.834 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 librbd1 2024-09-10T18:33:26.897 INFO:teuthology.orchestra.run.smithi083.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-10T18:33:26.897 INFO:teuthology.orchestra.run.smithi083.stdout:Need to get 201 MB of archives. 2024-09-10T18:33:26.897 INFO:teuthology.orchestra.run.smithi083.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-10T18:33:26.897 INFO:teuthology.orchestra.run.smithi083.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-10T18:33:26.970 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T18:33:26.971 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-10T18:33:26.971 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T18:33:26.971 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2024-09-10T18:33:26.971 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-10T18:33:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-10T18:33:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-10T18:33:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-10T18:33:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-10T18:33:26.972 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-10T18:33:26.973 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: socat xmlstarlet 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-10T18:33:26.974 INFO:teuthology.orchestra.run.smithi165.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2024-09-10T18:33:26.975 INFO:teuthology.orchestra.run.smithi165.stdout: btrfs-tools 2024-09-10T18:33:26.981 INFO:teuthology.orchestra.run.smithi083.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-10T18:33:26.981 INFO:teuthology.orchestra.run.smithi083.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1606-g5be57395-1focal [3,507 kB] 2024-09-10T18:33:26.983 INFO:teuthology.orchestra.run.smithi083.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-10T18:33:26.992 INFO:teuthology.orchestra.run.smithi083.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-10T18:33:26.998 INFO:teuthology.orchestra.run.smithi083.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-10T18:33:27.024 INFO:teuthology.orchestra.run.smithi083.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-10T18:33:27.025 INFO:teuthology.orchestra.run.smithi083.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-10T18:33:27.030 INFO:teuthology.orchestra.run.smithi083.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-10T18:33:27.063 INFO:teuthology.orchestra.run.smithi083.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-10T18:33:27.065 INFO:teuthology.orchestra.run.smithi083.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-10T18:33:27.071 INFO:teuthology.orchestra.run.smithi083.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-10T18:33:27.073 INFO:teuthology.orchestra.run.smithi083.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-10T18:33:27.073 INFO:teuthology.orchestra.run.smithi083.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-10T18:33:27.075 INFO:teuthology.orchestra.run.smithi083.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-10T18:33:27.075 INFO:teuthology.orchestra.run.smithi083.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-10T18:33:27.076 INFO:teuthology.orchestra.run.smithi083.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-10T18:33:27.077 INFO:teuthology.orchestra.run.smithi083.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-10T18:33:27.081 INFO:teuthology.orchestra.run.smithi083.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-10T18:33:27.095 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2024-09-10T18:33:27.095 INFO:teuthology.orchestra.run.smithi165.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-10T18:33:27.095 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-10T18:33:27.095 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-10T18:33:27.097 INFO:teuthology.orchestra.run.smithi083.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-10T18:33:27.097 INFO:teuthology.orchestra.run.smithi083.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-10T18:33:27.097 INFO:teuthology.orchestra.run.smithi083.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-10T18:33:27.098 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-10T18:33:27.099 INFO:teuthology.orchestra.run.smithi165.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-10T18:33:27.100 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2024-09-10T18:33:27.100 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 librbd1 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-10T18:33:27.101 INFO:teuthology.orchestra.run.smithi083.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-10T18:33:27.116 INFO:teuthology.orchestra.run.smithi083.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-10T18:33:27.117 INFO:teuthology.orchestra.run.smithi083.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-10T18:33:27.117 INFO:teuthology.orchestra.run.smithi083.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-10T18:33:27.133 INFO:teuthology.orchestra.run.smithi083.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-10T18:33:27.133 INFO:teuthology.orchestra.run.smithi083.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-10T18:33:27.134 INFO:teuthology.orchestra.run.smithi083.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-10T18:33:27.135 INFO:teuthology.orchestra.run.smithi083.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-10T18:33:27.136 INFO:teuthology.orchestra.run.smithi083.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-10T18:33:27.139 INFO:teuthology.orchestra.run.smithi083.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-10T18:33:27.140 INFO:teuthology.orchestra.run.smithi083.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-10T18:33:27.150 INFO:teuthology.orchestra.run.smithi083.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-10T18:33:27.151 INFO:teuthology.orchestra.run.smithi083.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-10T18:33:27.152 INFO:teuthology.orchestra.run.smithi083.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-10T18:33:27.155 INFO:teuthology.orchestra.run.smithi083.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1606-g5be57395-1focal [3,560 kB] 2024-09-10T18:33:27.167 INFO:teuthology.orchestra.run.smithi083.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-10T18:33:27.167 INFO:teuthology.orchestra.run.smithi083.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-10T18:33:27.168 INFO:teuthology.orchestra.run.smithi083.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-10T18:33:27.205 INFO:teuthology.orchestra.run.smithi083.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1606-g5be57395-1focal [761 kB] 2024-09-10T18:33:27.220 INFO:teuthology.orchestra.run.smithi083.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1606-g5be57395-1focal [414 kB] 2024-09-10T18:33:27.224 INFO:teuthology.orchestra.run.smithi083.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1606-g5be57395-1focal [22.6 kB] 2024-09-10T18:33:27.225 INFO:teuthology.orchestra.run.smithi083.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1606-g5be57395-1focal [200 kB] 2024-09-10T18:33:27.228 INFO:teuthology.orchestra.run.smithi083.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1606-g5be57395-1focal [49.1 kB] 2024-09-10T18:33:27.228 INFO:teuthology.orchestra.run.smithi083.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1606-g5be57395-1focal [389 kB] 2024-09-10T18:33:27.235 INFO:teuthology.orchestra.run.smithi083.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1606-g5be57395-1focal [5,637 kB] 2024-09-10T18:33:27.249 INFO:teuthology.orchestra.run.smithi083.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-10T18:33:27.265 INFO:teuthology.orchestra.run.smithi165.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-10T18:33:27.265 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 201 MB of archives. 2024-09-10T18:33:27.265 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-10T18:33:27.265 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1606-g5be57395-1focal [3,507 kB] 2024-09-10T18:33:27.265 INFO:teuthology.orchestra.run.smithi083.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-10T18:33:27.266 INFO:teuthology.orchestra.run.smithi083.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-10T18:33:27.278 INFO:teuthology.orchestra.run.smithi083.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-10T18:33:27.278 INFO:teuthology.orchestra.run.smithi083.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-10T18:33:27.292 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-10T18:33:27.346 INFO:teuthology.orchestra.run.smithi083.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1606-g5be57395-1focal [116 kB] 2024-09-10T18:33:27.346 INFO:teuthology.orchestra.run.smithi083.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-10T18:33:27.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1606-g5be57395-1focal [479 kB] 2024-09-10T18:33:27.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-10T18:33:27.348 INFO:teuthology.orchestra.run.smithi083.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-10T18:33:27.349 INFO:teuthology.orchestra.run.smithi083.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-10T18:33:27.350 INFO:teuthology.orchestra.run.smithi083.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-10T18:33:27.352 INFO:teuthology.orchestra.run.smithi083.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-10T18:33:27.353 INFO:teuthology.orchestra.run.smithi083.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1606-g5be57395-1focal [27.0 MB] 2024-09-10T18:33:27.353 INFO:teuthology.orchestra.run.smithi083.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-10T18:33:27.363 INFO:teuthology.orchestra.run.smithi083.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-10T18:33:27.366 INFO:teuthology.orchestra.run.smithi083.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-10T18:33:27.367 INFO:teuthology.orchestra.run.smithi083.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-10T18:33:27.381 INFO:teuthology.orchestra.run.smithi083.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-10T18:33:27.382 INFO:teuthology.orchestra.run.smithi083.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-10T18:33:27.388 INFO:teuthology.orchestra.run.smithi083.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-10T18:33:27.392 INFO:teuthology.orchestra.run.smithi083.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-10T18:33:27.408 INFO:teuthology.orchestra.run.smithi083.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-09-10T18:33:27.410 INFO:teuthology.orchestra.run.smithi083.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-10T18:33:27.453 INFO:teuthology.orchestra.run.smithi083.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-10T18:33:27.454 INFO:teuthology.orchestra.run.smithi083.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-10T18:33:27.454 INFO:teuthology.orchestra.run.smithi083.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-10T18:33:27.455 INFO:teuthology.orchestra.run.smithi083.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-10T18:33:27.455 INFO:teuthology.orchestra.run.smithi083.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-10T18:33:27.455 INFO:teuthology.orchestra.run.smithi083.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-10T18:33:27.467 INFO:teuthology.orchestra.run.smithi083.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-10T18:33:27.467 INFO:teuthology.orchestra.run.smithi083.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-10T18:33:27.468 INFO:teuthology.orchestra.run.smithi083.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-10T18:33:27.470 INFO:teuthology.orchestra.run.smithi083.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-10T18:33:27.488 INFO:teuthology.orchestra.run.smithi083.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-10T18:33:27.488 INFO:teuthology.orchestra.run.smithi083.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-10T18:33:27.489 INFO:teuthology.orchestra.run.smithi083.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-10T18:33:27.491 INFO:teuthology.orchestra.run.smithi083.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-10T18:33:27.492 INFO:teuthology.orchestra.run.smithi083.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-10T18:33:27.694 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-10T18:33:27.701 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-10T18:33:27.744 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-10T18:33:27.757 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-10T18:33:27.878 INFO:teuthology.orchestra.run.smithi083.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1606-g5be57395-1focal [5,178 kB] 2024-09-10T18:33:27.883 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-10T18:33:27.889 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-10T18:33:27.918 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-10T18:33:27.922 INFO:teuthology.orchestra.run.smithi083.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1606-g5be57395-1focal [223 kB] 2024-09-10T18:33:27.923 INFO:teuthology.orchestra.run.smithi083.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1606-g5be57395-1focal [157 kB] 2024-09-10T18:33:27.925 INFO:teuthology.orchestra.run.smithi083.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1606-g5be57395-1focal [1,593 kB] 2024-09-10T18:33:27.952 INFO:teuthology.orchestra.run.smithi083.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1606-g5be57395-1focal [4,686 kB] 2024-09-10T18:33:28.031 INFO:teuthology.orchestra.run.smithi083.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1606-g5be57395-1focal [23.1 MB] 2024-09-10T18:33:28.059 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-10T18:33:28.067 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-10T18:33:28.089 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-10T18:33:28.096 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-10T18:33:28.097 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-10T18:33:28.104 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-10T18:33:28.105 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-10T18:33:28.105 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-10T18:33:28.107 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-10T18:33:28.140 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-10T18:33:28.193 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-10T18:33:28.194 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-10T18:33:28.195 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-10T18:33:28.196 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-10T18:33:28.198 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-10T18:33:28.198 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-10T18:33:28.199 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-10T18:33:28.201 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-10T18:33:28.222 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-10T18:33:28.284 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-10T18:33:28.286 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-10T18:33:28.287 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-10T18:33:28.296 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-10T18:33:28.297 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-10T18:33:28.297 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-10T18:33:28.298 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-10T18:33:28.304 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-10T18:33:28.306 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-10T18:33:28.384 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-10T18:33:28.385 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-10T18:33:28.388 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-10T18:33:28.390 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-10T18:33:28.392 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-10T18:33:28.393 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-10T18:33:28.394 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-10T18:33:28.539 INFO:teuthology.orchestra.run.smithi083.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1606-g5be57395-1focal [3,936 B] 2024-09-10T18:33:28.540 INFO:teuthology.orchestra.run.smithi165.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1606-g5be57395-1focal [3,560 kB] 2024-09-10T18:33:28.576 INFO:teuthology.orchestra.run.smithi083.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1606-g5be57395-1focal [871 kB] 2024-09-10T18:33:28.590 INFO:teuthology.orchestra.run.smithi083.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1606-g5be57395-1focal [2,415 kB] 2024-09-10T18:33:28.607 INFO:teuthology.orchestra.run.smithi083.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1606-g5be57395-1focal [81.4 kB] 2024-09-10T18:33:28.608 INFO:teuthology.orchestra.run.smithi083.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1606-g5be57395-1focal [120 kB] 2024-09-10T18:33:28.609 INFO:teuthology.orchestra.run.smithi083.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1606-g5be57395-1focal [6,653 kB] 2024-09-10T18:33:28.707 INFO:teuthology.orchestra.run.smithi083.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1606-g5be57395-1focal [7,889 kB] 2024-09-10T18:33:28.718 INFO:teuthology.orchestra.run.smithi165.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-10T18:33:28.744 INFO:teuthology.orchestra.run.smithi165.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-10T18:33:28.746 INFO:teuthology.orchestra.run.smithi165.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-10T18:33:28.814 INFO:teuthology.orchestra.run.smithi165.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-10T18:33:28.814 INFO:teuthology.orchestra.run.smithi165.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-10T18:33:28.814 INFO:teuthology.orchestra.run.smithi165.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-10T18:33:28.815 INFO:teuthology.orchestra.run.smithi165.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-10T18:33:28.816 INFO:teuthology.orchestra.run.smithi165.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-10T18:33:28.816 INFO:teuthology.orchestra.run.smithi165.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-10T18:33:28.817 INFO:teuthology.orchestra.run.smithi165.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-10T18:33:28.850 INFO:teuthology.orchestra.run.smithi083.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1606-g5be57395-1focal [4,052 B] 2024-09-10T18:33:28.850 INFO:teuthology.orchestra.run.smithi083.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1606-g5be57395-1focal [53.4 MB] 2024-09-10T18:33:28.882 INFO:teuthology.orchestra.run.smithi165.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-10T18:33:28.883 INFO:teuthology.orchestra.run.smithi165.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-10T18:33:28.963 INFO:teuthology.orchestra.run.smithi165.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-10T18:33:28.968 INFO:teuthology.orchestra.run.smithi165.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-10T18:33:28.970 INFO:teuthology.orchestra.run.smithi165.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-10T18:33:28.973 INFO:teuthology.orchestra.run.smithi165.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-10T18:33:28.974 INFO:teuthology.orchestra.run.smithi165.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-10T18:33:28.979 INFO:teuthology.orchestra.run.smithi165.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-10T18:33:28.981 INFO:teuthology.orchestra.run.smithi165.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-10T18:33:29.056 INFO:teuthology.orchestra.run.smithi165.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2024-09-10T18:33:29.060 INFO:teuthology.orchestra.run.smithi165.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-10T18:33:29.064 INFO:teuthology.orchestra.run.smithi165.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1606-g5be57395-1focal [761 kB] 2024-09-10T18:33:29.140 INFO:teuthology.orchestra.run.smithi165.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1606-g5be57395-1focal [414 kB] 2024-09-10T18:33:29.194 INFO:teuthology.orchestra.run.smithi165.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1606-g5be57395-1focal [22.6 kB] 2024-09-10T18:33:29.195 INFO:teuthology.orchestra.run.smithi165.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1606-g5be57395-1focal [200 kB] 2024-09-10T18:33:29.220 INFO:teuthology.orchestra.run.smithi165.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-10T18:33:29.222 INFO:teuthology.orchestra.run.smithi165.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-10T18:33:29.222 INFO:teuthology.orchestra.run.smithi165.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-10T18:33:29.223 INFO:teuthology.orchestra.run.smithi165.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-10T18:33:29.224 INFO:teuthology.orchestra.run.smithi165.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-10T18:33:29.226 INFO:teuthology.orchestra.run.smithi165.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-10T18:33:29.238 INFO:teuthology.orchestra.run.smithi165.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-10T18:33:29.238 INFO:teuthology.orchestra.run.smithi165.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-10T18:33:29.238 INFO:teuthology.orchestra.run.smithi165.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-10T18:33:29.258 INFO:teuthology.orchestra.run.smithi165.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1606-g5be57395-1focal [49.1 kB] 2024-09-10T18:33:29.260 INFO:teuthology.orchestra.run.smithi165.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1606-g5be57395-1focal [389 kB] 2024-09-10T18:33:29.299 INFO:teuthology.orchestra.run.smithi165.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1606-g5be57395-1focal [5,637 kB] 2024-09-10T18:33:29.303 INFO:teuthology.orchestra.run.smithi165.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-10T18:33:29.384 INFO:teuthology.orchestra.run.smithi165.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-10T18:33:29.385 INFO:teuthology.orchestra.run.smithi165.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-10T18:33:29.386 INFO:teuthology.orchestra.run.smithi165.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-10T18:33:29.389 INFO:teuthology.orchestra.run.smithi165.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-10T18:33:29.390 INFO:teuthology.orchestra.run.smithi165.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-10T18:33:29.861 INFO:teuthology.orchestra.run.smithi165.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1606-g5be57395-1focal [116 kB] 2024-09-10T18:33:29.864 INFO:teuthology.orchestra.run.smithi165.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1606-g5be57395-1focal [479 kB] 2024-09-10T18:33:29.899 INFO:teuthology.orchestra.run.smithi165.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1606-g5be57395-1focal [27.0 MB] 2024-09-10T18:33:30.142 INFO:teuthology.orchestra.run.smithi083.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1606-g5be57395-1focal [231 kB] 2024-09-10T18:33:30.147 INFO:teuthology.orchestra.run.smithi083.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1606-g5be57395-1focal [21.9 kB] 2024-09-10T18:33:30.147 INFO:teuthology.orchestra.run.smithi083.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1606-g5be57395-1focal [17.2 MB] 2024-09-10T18:33:30.595 INFO:teuthology.orchestra.run.smithi083.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1606-g5be57395-1focal [80.7 kB] 2024-09-10T18:33:30.704 INFO:teuthology.orchestra.run.smithi165.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1606-g5be57395-1focal [5,178 kB] 2024-09-10T18:33:30.779 INFO:teuthology.orchestra.run.smithi165.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1606-g5be57395-1focal [223 kB] 2024-09-10T18:33:30.781 INFO:teuthology.orchestra.run.smithi165.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1606-g5be57395-1focal [157 kB] 2024-09-10T18:33:30.783 INFO:teuthology.orchestra.run.smithi165.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1606-g5be57395-1focal [1,593 kB] 2024-09-10T18:33:30.799 INFO:teuthology.orchestra.run.smithi165.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1606-g5be57395-1focal [4,686 kB] 2024-09-10T18:33:30.854 INFO:teuthology.orchestra.run.smithi165.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1606-g5be57395-1focal [23.1 MB] 2024-09-10T18:33:31.243 INFO:teuthology.orchestra.run.smithi083.stdout:Fetched 201 MB in 4s (53.6 MB/s) 2024-09-10T18:33:31.318 INFO:teuthology.orchestra.run.smithi165.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1606-g5be57395-1focal [3,936 B] 2024-09-10T18:33:31.318 INFO:teuthology.orchestra.run.smithi165.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1606-g5be57395-1focal [871 kB] 2024-09-10T18:33:31.330 INFO:teuthology.orchestra.run.smithi165.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1606-g5be57395-1focal [2,415 kB] 2024-09-10T18:33:31.347 INFO:teuthology.orchestra.run.smithi165.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1606-g5be57395-1focal [81.4 kB] 2024-09-10T18:33:31.348 INFO:teuthology.orchestra.run.smithi165.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1606-g5be57395-1focal [120 kB] 2024-09-10T18:33:31.379 INFO:teuthology.orchestra.run.smithi165.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1606-g5be57395-1focal [6,653 kB] 2024-09-10T18:33:31.452 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-10T18:33:31.458 INFO:teuthology.orchestra.run.smithi165.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1606-g5be57395-1focal [7,889 kB] 2024-09-10T18:33:31.507 INFO:teuthology.orchestra.run.smithi083.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-10T18:33:31.511 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-10T18:33:31.576 INFO:teuthology.orchestra.run.smithi165.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1606-g5be57395-1focal [4,052 B] 2024-09-10T18:33:31.576 INFO:teuthology.orchestra.run.smithi165.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1606-g5be57395-1focal [53.4 MB] 2024-09-10T18:33:31.600 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-10T18:33:31.945 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../001-librbd1_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:32.046 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librbd1 (17.2.7-1606-g5be57395-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-10T18:33:32.711 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../002-librados2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:32.720 INFO:teuthology.orchestra.run.smithi165.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1606-g5be57395-1focal [231 kB] 2024-09-10T18:33:32.725 INFO:teuthology.orchestra.run.smithi165.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1606-g5be57395-1focal [21.9 kB] 2024-09-10T18:33:32.726 INFO:teuthology.orchestra.run.smithi165.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1606-g5be57395-1focal [17.2 MB] 2024-09-10T18:33:32.875 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librados2 (17.2.7-1606-g5be57395-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-10T18:33:33.027 INFO:teuthology.orchestra.run.smithi165.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1606-g5be57395-1focal [80.7 kB] 2024-09-10T18:33:33.496 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs2. 2024-09-10T18:33:33.515 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:33.574 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:33.681 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 201 MB in 6s (34.0 MB/s) 2024-09-10T18:33:33.870 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-10T18:33:33.915 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rados. 2024-09-10T18:33:33.926 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-10T18:33:33.931 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-10T18:33:33.932 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:33.973 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rados (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:34.044 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-10T18:33:34.318 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-10T18:33:34.336 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:34.376 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-argparse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:34.413 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../001-librbd1_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:34.565 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librbd1 (17.2.7-1606-g5be57395-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-10T18:33:34.578 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cephfs. 2024-09-10T18:33:34.597 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:34.628 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cephfs (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:34.947 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-10T18:33:34.964 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:35.005 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:35.271 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../002-librados2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:35.351 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-prettytable. 2024-09-10T18:33:35.366 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-10T18:33:35.369 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librados2 (17.2.7-1606-g5be57395-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-10T18:33:35.409 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-10T18:33:35.662 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rbd. 2024-09-10T18:33:35.677 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:35.711 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rbd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:36.014 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-10T18:33:36.032 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-10T18:33:36.064 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-10T18:33:36.108 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs2. 2024-09-10T18:33:36.123 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:36.165 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:36.409 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-10T18:33:36.425 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-10T18:33:36.467 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-10T18:33:36.534 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rados. 2024-09-10T18:33:36.552 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:36.592 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rados (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:36.845 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-10T18:33:36.861 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-10T18:33:36.903 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-10T18:33:36.929 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-10T18:33:36.948 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:37.020 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-argparse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:37.281 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-10T18:33:37.297 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-10T18:33:37.314 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cephfs. 2024-09-10T18:33:37.337 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:37.339 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-10T18:33:37.381 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cephfs (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:37.642 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-10T18:33:37.660 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-10T18:33:37.693 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-10T18:33:37.767 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-10T18:33:37.783 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:37.825 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:38.158 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-prettytable. 2024-09-10T18:33:38.170 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-10T18:33:38.179 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-10T18:33:38.197 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:38.229 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-10T18:33:38.248 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:38.482 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rbd. 2024-09-10T18:33:38.499 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:38.540 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rbd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:38.799 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-10T18:33:38.815 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:38.857 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:38.893 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-10T18:33:38.911 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-10T18:33:38.976 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-10T18:33:39.219 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-10T18:33:39.234 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:39.277 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:39.346 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-10T18:33:39.361 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-10T18:33:39.404 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-10T18:33:39.638 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-10T18:33:39.654 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-10T18:33:39.688 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-10T18:33:39.815 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-10T18:33:39.832 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-10T18:33:39.874 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-10T18:33:40.008 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package librgw2. 2024-09-10T18:33:40.027 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../019-librgw2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:40.066 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking librgw2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:40.268 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-10T18:33:40.285 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-10T18:33:40.318 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-10T18:33:40.671 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-10T18:33:40.688 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-10T18:33:40.771 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-10T18:33:40.852 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rgw. 2024-09-10T18:33:40.869 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:40.927 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:41.255 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liboath0. 2024-09-10T18:33:41.270 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-10T18:33:41.300 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-10T18:33:41.313 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-10T18:33:41.316 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:41.368 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:41.616 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libradosstriper1. 2024-09-10T18:33:41.635 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:41.707 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libradosstriper1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:41.845 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-10T18:33:41.863 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:41.895 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:42.169 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-common. 2024-09-10T18:33:42.185 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:42.227 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:42.364 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-10T18:33:42.383 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-10T18:33:42.423 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:33:42.836 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-10T18:33:42.855 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-10T18:33:42.886 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-10T18:33:43.247 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librgw2. 2024-09-10T18:33:43.265 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../019-librgw2_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:43.339 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librgw2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:44.083 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rgw. 2024-09-10T18:33:44.102 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:44.158 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:44.728 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liboath0. 2024-09-10T18:33:44.744 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-10T18:33:44.749 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-base. 2024-09-10T18:33:44.765 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:44.786 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-10T18:33:44.876 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-base (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:45.156 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libradosstriper1. 2024-09-10T18:33:45.173 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:45.214 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libradosstriper1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:45.659 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-common. 2024-09-10T18:33:45.675 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:45.709 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:45.813 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cryptography. 2024-09-10T18:33:45.829 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-10T18:33:45.871 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-10T18:33:46.233 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-openssl. 2024-09-10T18:33:46.249 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-10T18:33:46.282 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-10T18:33:46.602 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-packaging. 2024-09-10T18:33:46.622 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-10T18:33:46.660 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-10T18:33:46.938 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-10T18:33:46.956 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:46.996 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:47.401 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-10T18:33:47.420 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-10T18:33:47.459 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-10T18:33:47.787 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-10T18:33:47.804 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-10T18:33:47.846 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-10T18:33:48.182 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-natsort. 2024-09-10T18:33:48.200 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-10T18:33:48.241 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-10T18:33:48.373 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-base. 2024-09-10T18:33:48.390 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:48.442 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-base (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:48.585 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jwt. 2024-09-10T18:33:48.601 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-10T18:33:48.644 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-10T18:33:48.955 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-logutils. 2024-09-10T18:33:48.973 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-10T18:33:49.013 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-10T18:33:49.350 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-10T18:33:49.366 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-10T18:33:49.386 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cryptography. 2024-09-10T18:33:49.404 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-10T18:33:49.408 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-10T18:33:49.444 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-10T18:33:49.711 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-mako. 2024-09-10T18:33:49.729 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-10T18:33:49.769 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-10T18:33:50.131 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-10T18:33:50.131 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-openssl. 2024-09-10T18:33:50.146 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-10T18:33:50.149 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-10T18:33:50.231 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-10T18:33:50.311 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-10T18:33:50.559 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-packaging. 2024-09-10T18:33:50.577 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-10T18:33:50.618 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-10T18:33:50.758 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-10T18:33:50.777 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-10T18:33:50.808 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-10T18:33:50.934 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-10T18:33:50.952 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:33:50.995 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:51.095 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webob. 2024-09-10T18:33:51.110 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-10T18:33:51.153 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-10T18:33:51.384 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-10T18:33:51.401 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-10T18:33:51.467 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-10T18:33:51.506 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-10T18:33:51.521 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-10T18:33:51.564 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-10T18:33:51.803 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-10T18:33:51.820 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-10T18:33:51.861 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-10T18:33:51.866 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-bs4. 2024-09-10T18:33:51.882 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-10T18:33:51.924 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-10T18:33:52.265 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-natsort. 2024-09-10T18:33:52.281 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-10T18:33:52.311 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-waitress. 2024-09-10T18:33:52.323 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-10T18:33:52.329 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-10T18:33:52.373 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-10T18:33:52.677 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jwt. 2024-09-10T18:33:52.693 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-10T18:33:52.698 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-tempita. 2024-09-10T18:33:52.716 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-10T18:33:52.726 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-10T18:33:52.776 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-10T18:33:53.005 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-logutils. 2024-09-10T18:33:53.021 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-10T18:33:53.063 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-10T18:33:53.090 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-paste. 2024-09-10T18:33:53.108 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-10T18:33:53.148 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-10T18:33:53.441 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-10T18:33:53.457 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-10T18:33:53.469 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-10T18:33:53.487 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-10T18:33:53.499 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-10T18:33:53.544 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-10T18:33:53.785 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-mako. 2024-09-10T18:33:53.804 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-10T18:33:53.835 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-10T18:33:53.846 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-10T18:33:53.862 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-10T18:33:53.905 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-10T18:33:54.197 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-10T18:33:54.208 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webtest. 2024-09-10T18:33:54.213 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-10T18:33:54.225 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-10T18:33:54.255 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-10T18:33:54.282 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-10T18:33:54.591 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-10T18:33:54.594 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pecan. 2024-09-10T18:33:54.609 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-10T18:33:54.610 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-10T18:33:54.652 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-10T18:33:54.683 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-10T18:33:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-10T18:33:55.040 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-10T18:33:55.044 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webob. 2024-09-10T18:33:55.061 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-10T18:33:55.082 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-10T18:33:55.102 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-10T18:33:55.456 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-10T18:33:55.473 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-10T18:33:55.486 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-10T18:33:55.502 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:55.505 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-10T18:33:55.561 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:55.800 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bs4. 2024-09-10T18:33:55.819 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-10T18:33:55.849 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-10T18:33:55.930 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr. 2024-09-10T18:33:55.946 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:55.990 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:56.236 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-waitress. 2024-09-10T18:33:56.253 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-10T18:33:56.297 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-10T18:33:56.498 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mon. 2024-09-10T18:33:56.516 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:56.556 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mon (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:56.681 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempita. 2024-09-10T18:33:56.697 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-10T18:33:56.739 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-10T18:33:57.075 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-paste. 2024-09-10T18:33:57.093 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-10T18:33:57.159 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-10T18:33:57.435 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-osd. 2024-09-10T18:33:57.452 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:57.492 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-osd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:57.504 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-10T18:33:57.521 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-10T18:33:57.562 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-10T18:33:57.923 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-10T18:33:57.940 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-10T18:33:57.981 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-10T18:33:58.302 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webtest. 2024-09-10T18:33:58.320 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-10T18:33:58.373 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-10T18:33:58.713 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pecan. 2024-09-10T18:33:58.729 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-10T18:33:58.771 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-10T18:33:59.197 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-10T18:33:59.213 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-10T18:33:59.255 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-10T18:33:59.625 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-10T18:33:59.641 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:59.645 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph. 2024-09-10T18:33:59.662 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../053-ceph_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:33:59.699 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:59.703 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:33:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-fuse. 2024-09-10T18:34:00.008 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:00.047 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:00.069 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr. 2024-09-10T18:34:00.088 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:00.311 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:00.475 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mds. 2024-09-10T18:34:00.492 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:00.533 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mds (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:01.273 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mon. 2024-09-10T18:34:01.292 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:01.356 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mon (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:01.887 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package cephadm. 2024-09-10T18:34:01.907 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../056-cephadm_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:01.954 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:02.193 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-osd. 2024-09-10T18:34:02.209 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:02.251 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-osd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:02.257 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-jinja2. 2024-09-10T18:34:02.272 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-10T18:34:02.315 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-10T18:34:02.651 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-10T18:34:02.666 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:02.709 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:03.029 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-10T18:34:03.045 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-10T18:34:03.096 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-10T18:34:03.382 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjs-underscore. 2024-09-10T18:34:03.397 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-10T18:34:03.440 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-10T18:34:03.717 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-10T18:34:03.733 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-10T18:34:03.775 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-10T18:34:04.103 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-routes. 2024-09-10T18:34:04.123 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-10T18:34:04.161 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-10T18:34:04.427 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph. 2024-09-10T18:34:04.445 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../053-ceph_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:04.488 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:04.489 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-10T18:34:04.505 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:04.547 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:04.841 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-fuse. 2024-09-10T18:34:04.857 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:04.924 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:05.353 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mds. 2024-09-10T18:34:05.372 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:05.444 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mds (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:05.878 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-decorator. 2024-09-10T18:34:05.896 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-10T18:34:05.936 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-10T18:34:06.064 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package cephadm. 2024-09-10T18:34:06.080 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../056-cephadm_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:06.122 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:06.256 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-10T18:34:06.275 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-10T18:34:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-10T18:34:06.483 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jinja2. 2024-09-10T18:34:06.503 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-10T18:34:06.541 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-10T18:34:06.633 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-scipy. 2024-09-10T18:34:06.649 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-10T18:34:06.691 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-10T18:34:06.894 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-10T18:34:06.910 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:06.978 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:07.348 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-10T18:34:07.367 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-10T18:34:07.406 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-10T18:34:07.658 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjs-underscore. 2024-09-10T18:34:07.676 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-10T18:34:07.725 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-10T18:34:07.993 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-10T18:34:08.003 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-10T18:34:08.011 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-10T18:34:08.020 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-10T18:34:08.051 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:08.061 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-10T18:34:08.455 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-routes. 2024-09-10T18:34:08.473 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-10T18:34:08.513 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-10T18:34:08.630 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-joblib. 2024-09-10T18:34:08.646 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-10T18:34:08.688 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-10T18:34:08.867 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-10T18:34:08.885 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:08.950 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:09.044 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-sklearn. 2024-09-10T18:34:09.058 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-10T18:34:09.125 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:09.775 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-10T18:34:09.794 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:09.858 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:10.341 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-decorator. 2024-09-10T18:34:10.360 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-10T18:34:10.398 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-10T18:34:10.760 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-10T18:34:10.776 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-10T18:34:10.818 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-10T18:34:11.011 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-cachetools. 2024-09-10T18:34:11.031 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-10T18:34:11.069 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-10T18:34:11.137 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-scipy. 2024-09-10T18:34:11.156 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-10T18:34:11.195 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-10T18:34:11.405 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-10T18:34:11.421 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-10T18:34:11.463 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-10T18:34:11.792 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-10T18:34:11.809 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-10T18:34:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-10T18:34:12.186 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-rsa. 2024-09-10T18:34:12.204 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-10T18:34:12.244 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-10T18:34:12.563 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-10T18:34:12.564 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-google-auth. 2024-09-10T18:34:12.579 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-10T18:34:12.581 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-10T18:34:12.621 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:12.622 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-10T18:34:12.959 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-blinker. 2024-09-10T18:34:12.978 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-10T18:34:13.008 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-10T18:34:13.225 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-joblib. 2024-09-10T18:34:13.244 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-10T18:34:13.283 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-10T18:34:13.311 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-10T18:34:13.329 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-10T18:34:13.369 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-10T18:34:13.690 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-10T18:34:13.698 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn. 2024-09-10T18:34:13.708 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-10T18:34:13.717 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-10T18:34:13.747 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-10T18:34:13.761 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:14.051 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-websocket. 2024-09-10T18:34:14.069 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-10T18:34:14.109 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-10T18:34:14.446 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-10T18:34:14.461 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-10T18:34:14.504 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-10T18:34:14.512 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-10T18:34:14.529 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:14.561 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:15.129 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-10T18:34:15.145 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:15.187 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:15.507 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-10T18:34:15.523 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-10T18:34:15.565 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-10T18:34:15.773 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cachetools. 2024-09-10T18:34:15.790 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-10T18:34:15.831 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-10T18:34:15.985 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-10T18:34:16.004 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-10T18:34:16.043 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:16.209 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-10T18:34:16.226 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-10T18:34:16.267 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-10T18:34:16.325 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package jq. 2024-09-10T18:34:16.341 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-10T18:34:16.382 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:16.654 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-10T18:34:16.660 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package socat. 2024-09-10T18:34:16.670 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-10T18:34:16.677 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-10T18:34:16.719 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-10T18:34:16.737 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-10T18:34:17.031 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package xmlstarlet. 2024-09-10T18:34:17.049 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-10T18:34:17.082 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rsa. 2024-09-10T18:34:17.089 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-10T18:34:17.100 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-10T18:34:17.140 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-10T18:34:17.477 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-test. 2024-09-10T18:34:17.496 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:17.511 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-google-auth. 2024-09-10T18:34:17.527 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-10T18:34:17.535 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-test (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:17.569 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-10T18:34:17.964 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-blinker. 2024-09-10T18:34:17.981 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-10T18:34:18.022 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-10T18:34:18.350 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-10T18:34:18.366 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-10T18:34:18.433 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-10T18:34:18.796 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-10T18:34:18.813 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-10T18:34:18.854 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-10T18:34:19.224 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-websocket. 2024-09-10T18:34:19.241 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-10T18:34:19.282 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-10T18:34:19.611 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-10T18:34:19.629 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-10T18:34:19.693 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-10T18:34:20.381 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-10T18:34:20.400 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:20.473 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:20.834 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-10T18:34:20.850 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-10T18:34:20.892 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-10T18:34:21.311 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-10T18:34:21.328 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-10T18:34:21.478 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:21.889 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package jq. 2024-09-10T18:34:21.908 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-10T18:34:21.939 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:22.225 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package socat. 2024-09-10T18:34:22.242 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-10T18:34:22.300 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-10T18:34:22.457 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package ceph-volume. 2024-09-10T18:34:22.476 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:22.515 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking ceph-volume (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:22.654 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package xmlstarlet. 2024-09-10T18:34:22.671 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-10T18:34:22.712 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-10T18:34:22.841 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-10T18:34:22.857 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-10T18:34:22.898 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:23.120 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-test. 2024-09-10T18:34:23.137 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:23.177 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-test (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:23.428 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-10T18:34:23.444 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:23.486 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libcephfs-dev (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:23.847 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package nvme-cli. 2024-09-10T18:34:23.865 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-09-10T18:34:23.965 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-09-10T18:34:24.296 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-10T18:34:24.313 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-10T18:34:24.354 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:25.952 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-10T18:34:25.973 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-10T18:34:26.002 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-10T18:34:26.297 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-10T18:34:26.313 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-10T18:34:26.354 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-10T18:34:26.624 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-attr. 2024-09-10T18:34:26.644 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-10T18:34:26.674 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-10T18:34:27.019 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-webencodings. 2024-09-10T18:34:27.039 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-10T18:34:27.077 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-10T18:34:27.372 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-html5lib. 2024-09-10T18:34:27.389 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-10T18:34:27.429 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-10T18:34:27.731 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-volume. 2024-09-10T18:34:27.750 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1606-g5be57395-1focal_all.deb ... 2024-09-10T18:34:27.789 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-volume (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:27.808 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-10T18:34:27.825 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-10T18:34:27.865 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-10T18:34:28.137 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-10T18:34:28.154 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-10T18:34:28.220 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:28.361 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pastescript. 2024-09-10T18:34:28.378 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-10T18:34:28.419 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-10T18:34:28.739 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pluggy. 2024-09-10T18:34:28.757 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-10T18:34:28.774 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-10T18:34:28.792 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:28.797 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-10T18:34:28.832 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs-dev (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:29.117 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-psutil. 2024-09-10T18:34:29.136 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-10T18:34:29.175 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-10T18:34:29.285 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package nvme-cli. 2024-09-10T18:34:29.304 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2024-09-10T18:34:29.360 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2024-09-10T18:34:29.503 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-py. 2024-09-10T18:34:29.520 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-10T18:34:29.561 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-10T18:34:29.740 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-10T18:34:29.755 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-10T18:34:29.797 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:29.914 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-10T18:34:29.933 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-10T18:34:29.972 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-10T18:34:30.275 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-10T18:34:30.295 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-10T18:34:30.333 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-10T18:34:30.644 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-pytest. 2024-09-10T18:34:30.664 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-10T18:34:30.702 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-10T18:34:31.039 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package python3-simplejson. 2024-09-10T18:34:31.057 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-10T18:34:31.097 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-10T18:34:31.400 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-10T18:34:31.417 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-10T18:34:31.458 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-10T18:34:31.595 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-10T18:34:31.616 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-10T18:34:31.686 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-10T18:34:32.056 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-10T18:34:32.074 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-10T18:34:32.134 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-10T18:34:32.238 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package radosgw. 2024-09-10T18:34:32.254 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../108-radosgw_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:32.296 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking radosgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:32.475 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-attr. 2024-09-10T18:34:32.496 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-10T18:34:32.533 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-10T18:34:32.869 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webencodings. 2024-09-10T18:34:32.886 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-10T18:34:32.944 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-10T18:34:33.280 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-html5lib. 2024-09-10T18:34:33.298 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-10T18:34:33.338 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-10T18:34:33.749 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-10T18:34:33.767 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-10T18:34:33.807 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-10T18:34:34.066 INFO:teuthology.orchestra.run.smithi083.stdout:Selecting previously unselected package rbd-fuse. 2024-09-10T18:34:34.082 INFO:teuthology.orchestra.run.smithi083.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:34.124 INFO:teuthology.orchestra.run.smithi083.stdout:Unpacking rbd-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:34.294 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastescript. 2024-09-10T18:34:34.313 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-10T18:34:34.352 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-10T18:34:34.444 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-10T18:34:34.738 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pluggy. 2024-09-10T18:34:34.740 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-10T18:34:34.757 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-10T18:34:34.805 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-10T18:34:34.866 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-10T18:34:34.992 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-09-10T18:34:35.183 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-psutil. 2024-09-10T18:34:35.200 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-10T18:34:35.218 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:35.241 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-10T18:34:35.622 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-py. 2024-09-10T18:34:35.645 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-10T18:34:35.693 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-10T18:34:35.799 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user cephadm....done 2024-09-10T18:34:35.966 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:36.046 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-10T18:34:36.063 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-10T18:34:36.092 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-10T18:34:36.128 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-10T18:34:36.432 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-10T18:34:36.446 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-10T18:34:36.449 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-10T18:34:36.490 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-10T18:34:36.773 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:36.885 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pytest. 2024-09-10T18:34:36.898 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-10T18:34:36.904 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-10T18:34:36.942 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-10T18:34:37.025 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-10T18:34:37.324 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplejson. 2024-09-10T18:34:37.344 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-10T18:34:37.359 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-10T18:34:37.415 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-10T18:34:37.710 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-10T18:34:37.726 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-10T18:34:37.767 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-10T18:34:37.778 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-10T18:34:38.063 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-10T18:34:38.347 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-10T18:34:38.563 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package radosgw. 2024-09-10T18:34:38.579 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../108-radosgw_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:38.640 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-10T18:34:38.646 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking radosgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:39.009 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-argparse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:39.318 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:39.478 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-10T18:34:39.762 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-10T18:34:40.047 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-10T18:34:40.172 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-10T18:34:40.441 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package rbd-fuse. 2024-09-10T18:34:40.461 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1606-g5be57395-1focal_amd64.deb ... 2024-09-10T18:34:40.499 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking rbd-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:40.590 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-10T18:34:40.824 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-10T18:34:40.883 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-10T18:34:41.188 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-10T18:34:41.192 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-10T18:34:41.315 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-10T18:34:41.466 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2024-09-10T18:34:41.544 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-10T18:34:41.693 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:41.845 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-10T18:34:42.172 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-10T18:34:42.346 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user cephadm....done 2024-09-10T18:34:42.539 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-10T18:34:42.540 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:42.691 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-10T18:34:42.848 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-10T18:34:42.974 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-10T18:34:43.044 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-10T18:34:43.100 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-10T18:34:43.345 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-10T18:34:43.418 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-10T18:34:43.496 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-10T18:34:43.630 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-10T18:34:43.820 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-10T18:34:43.998 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-10T18:34:44.227 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-10T18:34:44.450 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-10T18:34:44.589 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-10T18:34:44.740 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:44.751 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-10T18:34:45.036 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-10T18:34:45.058 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:34:45.168 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-10T18:34:45.328 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-10T18:34:45.511 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-10T18:34:45.637 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-10T18:34:45.696 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-argparse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:45.763 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-10T18:34:46.005 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:46.056 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-10T18:34:46.131 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-10T18:34:46.391 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:34:46.466 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-10T18:34:46.517 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-10T18:34:46.618 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-10T18:34:46.758 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-10T18:34:46.906 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-10T18:34:46.928 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-10T18:34:47.221 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-10T18:34:47.343 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-10T18:34:47.522 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-10T18:34:47.678 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-10T18:34:47.681 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-10T18:34:47.970 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-10T18:34:48.007 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-10T18:34:48.330 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-10T18:34:48.333 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-10T18:34:48.634 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-10T18:34:48.664 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-10T18:34:48.974 INFO:teuthology.orchestra.run.smithi083.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-10T18:34:49.007 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-10T18:34:49.221 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-10T18:34:49.347 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-10T18:34:49.374 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-10T18:34:49.681 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-10T18:34:49.692 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-10T18:34:49.818 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-10T18:34:49.969 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-10T18:34:50.074 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-10T18:34:50.200 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-10T18:34:50.245 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-10T18:34:50.318 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-10T18:34:50.647 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-10T18:34:50.652 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-10T18:34:50.954 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librados2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:51.015 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-10T18:34:51.079 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-10T18:34:51.409 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-10T18:34:51.534 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:51.885 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:34:52.019 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-10T18:34:52.552 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:52.708 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:52.749 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-10T18:34:52.837 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-10T18:34:52.905 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-10T18:34:53.031 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-10T18:34:53.340 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-10T18:34:53.480 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-10T18:34:53.683 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:34:53.781 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-10T18:34:53.809 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-10T18:34:53.968 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-10T18:34:54.099 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:34:54.250 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-10T18:34:54.295 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-10T18:34:54.601 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:54.623 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-10T18:34:54.736 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-10T18:34:54.924 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-10T18:34:55.050 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-10T18:34:55.062 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libradosstriper1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:55.189 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-10T18:34:55.384 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-10T18:34:55.523 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-10T18:34:55.677 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-10T18:34:55.969 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-10T18:34:56.337 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-10T18:34:56.570 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librbd1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:56.621 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-10T18:34:56.696 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-10T18:34:56.756 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-10T18:34:56.822 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-10T18:34:56.948 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:34:57.074 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-10T18:34:57.115 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-10T18:34:57.409 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-10T18:34:57.483 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-10T18:34:57.534 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-10T18:34:57.642 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-10T18:34:57.768 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-10T18:34:57.836 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-modules-core (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:57.987 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:58.119 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-10T18:34:58.216 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T18:34:58.216 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T18:34:58.420 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librados2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:58.579 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-10T18:34:58.655 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up libcephfs-dev (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:58.866 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-10T18:34:59.167 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-10T18:34:59.503 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-10T18:34:59.863 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rados (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:34:59.989 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up librgw2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:00.060 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:00.132 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rbd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:00.186 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:35:00.258 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up rbd-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:00.345 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-10T18:35:00.384 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-10T18:35:00.685 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-10T18:35:00.704 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-10T18:35:00.995 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-rgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:01.039 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-10T18:35:01.121 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-cephfs (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:01.248 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-10T18:35:01.324 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-10T18:35:01.449 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-10T18:35:01.851 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:01.997 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-10T18:35:02.176 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:02.320 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libradosstriper1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:02.445 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-10T18:35:02.745 INFO:teuthology.orchestra.run.smithi083.stdout:Adding group ceph....done 2024-09-10T18:35:02.805 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-10T18:35:03.155 INFO:teuthology.orchestra.run.smithi083.stdout:Adding system user ceph....done 2024-09-10T18:35:03.332 INFO:teuthology.orchestra.run.smithi083.stdout:Setting system user ceph properties....done 2024-09-10T18:35:03.357 INFO:teuthology.orchestra.run.smithi083.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-10T18:35:03.505 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-10T18:35:03.820 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-10T18:35:03.893 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librbd1 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:04.044 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-10T18:35:04.170 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-10T18:35:04.278 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-test (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:04.329 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-10T18:35:04.404 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up radosgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:04.489 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-10T18:35:04.710 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:35:04.710 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:35:04.790 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-10T18:35:04.916 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-10T18:35:05.258 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-modules-core (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:05.364 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-base (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:05.384 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:05.627 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T18:35:05.627 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T18:35:05.734 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-10T18:35:06.102 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs-dev (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:06.191 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mds (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:06.330 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-10T18:35:06.401 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T18:35:06.401 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T18:35:06.623 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-10T18:35:06.859 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:06.999 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-10T18:35:07.085 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T18:35:07.085 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T18:35:07.360 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rados (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:07.519 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librgw2 (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:07.535 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-osd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:07.645 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rbd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:07.804 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up rbd-fuse (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:07.957 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-10T18:35:08.019 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T18:35:08.019 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T18:35:08.274 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-10T18:35:08.437 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:08.575 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:08.726 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cephfs (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:08.781 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:08.852 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-10T18:35:08.982 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mon (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:09.197 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T18:35:09.197 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T18:35:09.658 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:09.776 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:09.796 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-common (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:09.902 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-mgr-dashboard (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:10.045 INFO:teuthology.orchestra.run.smithi083.stdout:Setting up ceph-volume (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-10T18:35:10.340 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-10T18:35:10.407 INFO:teuthology.orchestra.run.smithi165.stdout:Adding group ceph....done 2024-09-10T18:35:10.641 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-10T18:35:10.817 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user ceph....done 2024-09-10T18:35:11.018 INFO:teuthology.orchestra.run.smithi165.stdout:Setting system user ceph properties....done 2024-09-10T18:35:11.045 INFO:teuthology.orchestra.run.smithi165.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-10T18:35:11.190 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-10T18:35:11.496 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-10T18:35:11.972 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-test (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:12.098 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up radosgw (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:12.402 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:35:12.402 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:35:13.101 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-base (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:13.460 INFO:teuthology.orchestra.run.smithi083.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-10T18:35:13.525 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-10T18:35:13.985 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mds (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:14.055 INFO:teuthology.orchestra.run.smithi083.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-10T18:35:14.060 DEBUG:teuthology.parallel:result is None 2024-09-10T18:35:14.203 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T18:35:14.204 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T18:35:14.678 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:14.904 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T18:35:14.925 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T18:35:15.374 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-osd (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:15.897 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T18:35:15.897 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T18:35:16.389 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:16.514 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:16.699 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mon (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:16.915 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T18:35:16.915 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T18:35:17.383 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-cephadm (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:17.526 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:17.685 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-dashboard (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:17.861 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-volume (17.2.7-1606-g5be57395-1focal) ... 2024-09-10T18:35:17.988 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-10T18:35:18.173 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-10T18:35:18.583 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-10T18:35:21.444 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-10T18:35:22.099 INFO:teuthology.orchestra.run.smithi165.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-10T18:35:22.103 DEBUG:teuthology.parallel:result is None 2024-09-10T18:35:22.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:35:22.243 DEBUG:teuthology.orchestra.run.smithi083:> dpkg-query -W -f '${Version}' ceph 2024-09-10T18:35:22.267 INFO:teuthology.orchestra.run.smithi083.stdout:17.2.7-1606-g5be57395-1focal 2024-09-10T18:35:22.267 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606-g5be57395-1focal 2024-09-10T18:35:22.267 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606-g5be57395-1focal is installed. 2024-09-10T18:35:22.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:35:22.404 DEBUG:teuthology.orchestra.run.smithi165:> dpkg-query -W -f '${Version}' ceph 2024-09-10T18:35:22.428 INFO:teuthology.orchestra.run.smithi165.stdout:17.2.7-1606-g5be57395-1focal 2024-09-10T18:35:22.428 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606-g5be57395-1focal 2024-09-10T18:35:22.428 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606-g5be57395-1focal is installed. 2024-09-10T18:35:22.430 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T18:35:22.430 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:35:22.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:35:22.446 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:35:22.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:35:22.486 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T18:35:22.486 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:35:22.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:35:22.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:35:22.556 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:35:22.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:35:22.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:35:22.586 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T18:35:22.587 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:35:22.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:35:22.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:35:22.661 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:35:22.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:35:22.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:35:22.733 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T18:35:22.734 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:35:22.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:35:22.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:35:22.765 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:35:22.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:35:22.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:35:22.837 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-10T18:35:22.918 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'cephadm_mode': 'root'} 2024-09-10T18:35:22.918 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:35:22.919 INFO:tasks.cephadm:Cluster fsid is 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:35:22.919 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-10T18:35:22.919 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.83:6789]', 'mon.c': '[v1:172.21.15.83:6790]', 'mon.b': '[v1:172.21.15.165:6789]'} 2024-09-10T18:35:22.919 INFO:tasks.cephadm:First mon is mon.a on smithi083 2024-09-10T18:35:22.919 INFO:tasks.cephadm:First mgr is y 2024-09-10T18:35:22.919 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-10T18:35:22.919 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-09-10T18:35:22.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-09-10T18:35:22.953 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref 5be57395ea235535910512ae73ba8e7ff25fe588)... 2024-09-10T18:35:22.953 DEBUG:teuthology.orchestra.run.smithi083:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T18:35:22.983 INFO:teuthology.orchestra.run.smithi083.stderr:Cloning into 'testrepo'... 2024-09-10T18:36:22.286 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-rw-r-- 1 ubuntu ubuntu 378585 Sep 10 18:36 /home/ubuntu/cephtest/cephadm 2024-09-10T18:36:22.287 DEBUG:teuthology.orchestra.run.smithi165:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T18:36:22.295 INFO:teuthology.orchestra.run.smithi165.stderr:Cloning into 'testrepo'... 2024-09-10T18:37:23.128 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-rw-r-- 1 ubuntu ubuntu 378585 Sep 10 18:37 /home/ubuntu/cephtest/cephadm 2024-09-10T18:37:23.129 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T18:37:23.139 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T18:37:23.191 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 on all hosts... 2024-09-10T18:37:23.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:37:23.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:37:23.477 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:37:23.482 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:38:15.854 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:38:23.829 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:38:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:38:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:38:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-09-10T18:38:23.829 INFO:teuthology.orchestra.run.smithi083.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:38:23.830 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:38:23.830 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:38:23.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-09-10T18:38:23.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-09-10T18:38:23.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-09-10T18:38:23.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-09-10T18:38:23.946 INFO:tasks.cephadm:Writing seed config... 2024-09-10T18:38:23.947 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-10T18:38:23.947 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-10T18:38:23.947 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-10T18:38:23.948 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-10T18:38:23.949 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-10T18:38:23.950 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-10T18:38:23.950 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-10T18:38:23.950 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:38:23.951 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-10T18:38:23.957 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 707a4ada-6fa3-11ef-bcea-c7b262605968 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-10T18:38:23.957 DEBUG:teuthology.orchestra.run.smithi083:mon.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service 2024-09-10T18:38:23.960 DEBUG:teuthology.orchestra.run.smithi083:mgr.y> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y.service 2024-09-10T18:38:24.002 INFO:tasks.cephadm:Bootstrapping... 2024-09-10T18:38:24.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 -v bootstrap --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.83:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:38:24.167 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:38:24.168 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588', '-v', 'bootstrap', '--fsid', '707a4ada-6fa3-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.83:6789]', '--skip-admin-label'] 2024-09-10T18:38:24.168 INFO:teuthology.orchestra.run.smithi083.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-10T18:38:24.169 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying podman|docker is present... 2024-09-10T18:38:24.169 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying lvm2 is present... 2024-09-10T18:38:24.169 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying time synchronization is in place... 2024-09-10T18:38:24.174 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:38:24.174 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:38:24.182 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:38:24.183 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.189 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-10T18:38:24.189 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-10T18:38:24.196 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-10T18:38:24.196 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.201 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-10T18:38:24.202 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout masked 2024-09-10T18:38:24.208 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-10T18:38:24.208 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.216 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-10T18:38:24.216 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-10T18:38:24.223 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-10T18:38:24.223 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.232 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout enabled 2024-09-10T18:38:24.238 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout active 2024-09-10T18:38:24.238 INFO:teuthology.orchestra.run.smithi083.stdout:Unit ntp.service is enabled and running 2024-09-10T18:38:24.238 INFO:teuthology.orchestra.run.smithi083.stdout:Repeating the final host check... 2024-09-10T18:38:24.238 INFO:teuthology.orchestra.run.smithi083.stdout:docker (/usr/bin/docker) is present 2024-09-10T18:38:24.239 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl is present 2024-09-10T18:38:24.239 INFO:teuthology.orchestra.run.smithi083.stdout:lvcreate is present 2024-09-10T18:38:24.246 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:38:24.246 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:38:24.253 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:38:24.253 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.260 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-09-10T18:38:24.260 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-09-10T18:38:24.266 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-09-10T18:38:24.267 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.274 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-09-10T18:38:24.274 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout masked 2024-09-10T18:38:24.281 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-09-10T18:38:24.281 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.288 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-09-10T18:38:24.288 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-09-10T18:38:24.295 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-09-10T18:38:24.295 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout inactive 2024-09-10T18:38:24.301 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout enabled 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stdout active 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Unit ntp.service is enabled and running 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Host looks OK 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Cluster fsid: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Acquiring lock 140098793761376 on /run/cephadm/707a4ada-6fa3-11ef-bcea-c7b262605968.lock 2024-09-10T18:38:24.309 INFO:teuthology.orchestra.run.smithi083.stdout:Lock 140098793761376 acquired on /run/cephadm/707a4ada-6fa3-11ef-bcea-c7b262605968.lock 2024-09-10T18:38:24.310 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying IP 172.21.15.83 port 6789 ... 2024-09-10T18:38:24.310 INFO:teuthology.orchestra.run.smithi083.stdout:Base mon IP(s) is [172.21.15.83:6789], mon addrv is [v1:172.21.15.83:6789] 2024-09-10T18:38:24.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 2024-09-10T18:38:24.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.83 metric 100 2024-09-10T18:38:24.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-09-10T18:38:24.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.83 2024-09-10T18:38:24.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout 172.21.15.254 dev enp3s0f1 proto dhcp scope link src 172.21.15.83 metric 100 2024-09-10T18:38:24.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-10T18:38:24.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-09-10T18:38:24.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 expires 1794sec pref medium 2024-09-10T18:38:24.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-10T18:38:24.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-10T18:38:24.324 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:Mon IP `172.21.15.83` is in CIDR network `172.21.0.0/20` 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:Mon IP `172.21.15.83` is in CIDR network `172.21.15.254/32` 2024-09-10T18:38:24.325 INFO:teuthology.orchestra.run.smithi083.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.15.254/32'] 2024-09-10T18:38:24.326 INFO:teuthology.orchestra.run.smithi083.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-10T18:38:24.327 INFO:teuthology.orchestra.run.smithi083.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:38:24.357 INFO:journalctl@ceph.mgr.y.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:38:24.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:38:24.620 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/docker: stdout 5be57395ea235535910512ae73ba8e7ff25fe588: Pulling from ceph-ci/ceph 2024-09-10T18:38:24.620 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/docker: stdout Digest: sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:38:24.620 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:38:24.620 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:38:26.344 INFO:teuthology.orchestra.run.smithi083.stdout:ceph: stdout ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:38:26.344 INFO:teuthology.orchestra.run.smithi083.stdout:Ceph version: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:38:26.344 INFO:teuthology.orchestra.run.smithi083.stdout:Extracting ceph user uid/gid from container image... 2024-09-10T18:38:27.891 INFO:teuthology.orchestra.run.smithi083.stdout:stat: stdout 167 167 2024-09-10T18:38:27.892 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial keys... 2024-09-10T18:38:29.506 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQAkkuBmIWb2NxAAmk/GsY1AJDkew0YN7pheiQ== 2024-09-10T18:38:31.087 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQAmkuBmlCNCHhAAn/d1bgui4H4yMHIBgN9M5A== 2024-09-10T18:38:32.718 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-authtool: stdout AQAokuBmOGgaBxAAqJ9PJaARnUMno1FNKMP4eQ== 2024-09-10T18:38:32.719 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial monmap... 2024-09-10T18:38:34.308 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:38:34.308 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-10T18:38:34.308 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool for a [v1:172.21.15.83:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = octopus 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: set fsid to 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:38:34.309 INFO:teuthology.orchestra.run.smithi083.stdout:Creating mon... 2024-09-10T18:38:35.941 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:38:35.941 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 1 imported monmap: 2024-09-10T18:38:35.941 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-09-10T18:38:33.730216+0000 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr created 2024-09-10T18:38:33.730216+0000 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr min_mon_release 15 (octopus) 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 0: v1:172.21.15.83:6789/0 mon.a 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 0 /usr/bin/ceph-mon: set fsid to 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: RocksDB version: 6.15.5 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:38:35.942 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Compile date Sep 9 2024 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: DB SUMMARY 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: DB Session ID: YK3MHNEVEY6C7JKB9WD3 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.error_if_exists: 0 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.create_if_missing: 1 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.paranoid_checks: 1 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:38:35.943 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.env: 0x562c8a62e440 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.fs: Posix File System 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.info_log: 0x562c8b5a8660 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.statistics: (nil) 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.use_fsync: 0 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_log_file_size: 0 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_fallocate: 1 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:38:35.944 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.use_direct_reads: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.db_log_dir: 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.write_buffer_manager: 0x562c8b5f1ec0 2024-09-10T18:38:35.945 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.unordered_write: 0 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:38:35.946 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.row_cache: None 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.wal_filter: None 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.preserve_deletes: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.two_write_queues: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.atomic_flush: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.log_readahead_size: 0 2024-09-10T18:38:35.947 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_background_jobs: 2 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_background_compactions: -1 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_subcompactions: 1 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:38:35.948 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_open_files: -1 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Options.max_background_flushes: -1 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Compression algorithms supported: 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kZSTD supported: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kXpressCompression supported: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kBZip2Compression supported: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:38:35.949 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kLZ4Compression supported: 1 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kZlibCompression supported: 1 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: kSnappyCompression supported: 1 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.368+0000 7fdf1be8b900 4 rocksdb: [db/db_impl/db_impl_open.cc:281] Creating manifest 1 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.merge_operator: 2024-09-10T18:38:35.950 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_filter: None 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562c8b548ee0) 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-09-10T18:38:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr hash_index_allow_collision: 1 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr checksum: 1 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x562c8b5811f0 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-09-10T18:38:35.952 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr format_version: 4 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-09-10T18:38:35.953 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression: NoCompression 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.num_levels: 7 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:38:35.954 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:38:35.955 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:38:35.956 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:38:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.inplace_update_support: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.bloom_locality: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.max_successive_merges: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:38:35.958 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.ttl: 2592000 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.enable_blob_files: false 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.min_blob_size: 0 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.372+0000 7fdf1be8b900 4 rocksdb: [db/version_set.cc:4082] Creating manifest 3 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.376+0000 7fdf1be8b900 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562c8b5b2700 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.376+0000 7fdf1be8b900 4 rocksdb: DB pointer 0x562c8b584000 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.376+0000 7fdf13619640 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.376+0000 7fdf13619640 4 rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:38:35.960 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:38:35.961 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:38:35.962 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:38:35.963 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:35.964 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.380+0000 7fdf1be8b900 4 rocksdb: [db/db_impl/db_impl.cc:446] Shutdown: canceling all background work 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.380+0000 7fdf1be8b900 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph-mon: stderr debug 2024-09-10T18:38:35.380+0000 7fdf1be8b900 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-09-10T18:38:35.965 INFO:teuthology.orchestra.run.smithi083.stdout:create mon.a on 2024-09-10T18:38:36.142 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-09-10T18:38:36.345 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-09-10T18:38:36.543 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target -> /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target. 2024-09-10T18:38:36.544 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target -> /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target. 2024-09-10T18:38:36.770 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a 2024-09-10T18:38:36.770 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to reset failed state of unit ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-10T18:38:36.958 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target.wants/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service -> /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@.service. 2024-09-10T18:38:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld does not appear to be present 2024-09-10T18:38:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:38:36.971 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mon to start... 2024-09-10T18:38:36.971 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mon... 2024-09-10T18:38:37.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:36 smithi083 systemd[1]: Started Ceph mon.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:38:38.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:38 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.475854+0000 mon.a (mon.0) 0 : cluster [INF] mkfs 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:38.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:38 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.469417+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:38:39.257 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout cluster: 2024-09-10T18:38:39.257 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:39.257 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-10T18:38:39.257 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:39.257 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout services: 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.218827s) 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout data: 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout pgs: 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.556+0000 7f8f9f9d3640 1 Processor -- start 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.556+0000 7f8f9f9d3640 1 -- start start 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.556+0000 7f8f9f9d3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f98104fa0 con 0x7f8f98102180 2024-09-10T18:38:39.258 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.556+0000 7f8f9d76f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8f98102180 0x7f8f98102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54528/0 (socket says 172.21.15.83:54528) 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.556+0000 7f8f9d76f640 1 -- 172.21.15.83:0/3236772420 learned_addr learned my addr 172.21.15.83:0/3236772420 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233349442 0 0) 0x7f8f98104fa0 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f840034e0 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161591334 0 0) 0x7f8f840034e0 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98106140 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8f94001590 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f94001820 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8ffff640 1 -- 172.21.15.83:0/3236772420 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8f94001b70 con 0x7f8f98102180 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/3236772420 >> v1:172.21.15.83:6789/0 conn(0x7f8f98102180 legacy=0x7f8f98102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/3236772420 shutdown_connections 2024-09-10T18:38:39.259 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/3236772420 >> 172.21.15.83:0/3236772420 conn(0x7f8f980fc240 msgr2=0x7f8f980fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/3236772420 shutdown_connections 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/3236772420 wait complete. 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 librados: starting msgr at 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 librados: starting objecter 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 Processor -- start 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- start start 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 librados: setting wanted keys 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 librados: calling monclient init 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9f9d3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f9810f3d0 con 0x7f8f98102180 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9d76f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8f98102180 0x7f8f9810ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54532/0 (socket says 172.21.15.83:54532) 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f9d76f640 1 -- 172.21.15.83:0/421792108 learned_addr learned my addr 172.21.15.83:0/421792108 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:39.260 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 10 client.?.objecter ms_handle_connect 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669038356 0 0) 0x7f8f9810f3d0 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f84004500 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 229927500 0 0) 0x7f8f84004500 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9810f3d0 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.560+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8f94001550 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2288356777 0 0) 0x7f8f9810f3d0 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9810f560 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9810f830 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f9f9d3640 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f98113440 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f94004bd0 con 0x7f8f98102180 2024-09-10T18:38:39.261 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f9f9d3640 1 librados: init done 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8f94004ee0 con 0x7f8f98102180 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f8f94005340 con 0x7f8f98102180 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f8f940063b0 con 0x7f8f98102180 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 10 client.4098.objecter ms_dispatch 0x7f8f98000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f8dffb640 20 client.4098.objecter dump_active .. 0 homeless 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.564+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9805ed70 con 0x7f8f98102180 2024-09-10T18:38:39.262 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.568+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f8f940065f0 con 0x7f8f98102180 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.688+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f8f98108a50 con 0x7f8f98102180 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.688+0000 7f8f8dffb640 1 -- 172.21.15.83:0/421792108 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 3389395351) 0x7f8f940057d0 con 0x7f8f98102180 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 10 librados: watch_flush enter 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 10 librados: watch_flush exit 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 20 client.4098.objecter shutdown clearing up homeless session... 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 10 client.4098.objecter successfully canceled tick 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 >> v1:172.21.15.83:6789/0 conn(0x7f8f98102180 legacy=0x7f8f9810ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:39.263 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 shutdown_connections 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 >> 172.21.15.83:0/421792108 conn(0x7f8f980fc240 msgr2=0x7f8f98104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 shutdown_connections 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 -- 172.21.15.83:0/421792108 wait complete. 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:38.692+0000 7f8f9f9d3640 1 librados: shutdown 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:mon is available 2024-09-10T18:38:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:Assimilating anything we can from ceph.conf... 2024-09-10T18:38:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.474670+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:38:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.474894+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.83:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:38:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.477674+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-09-10T18:38:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.481214+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-10T18:38:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: cluster 2024-09-10T18:38:38.482003+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-09-10T18:38:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:39 smithi083 bash[18124]: audit 2024-09-10T18:38:38.693439+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.83:0/421792108' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-10T18:38:40.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:40 smithi083 bash[18124]: audit 2024-09-10T18:38:40.462087+0000 mon.a (mon.0) 8 : audit [INF] from='client.? v1:172.21.15.83:0/3437093257' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:38:40.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:40 smithi083 bash[18124]: audit 2024-09-10T18:38:40.468507+0000 mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.83:0/3437093257' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout fsid = 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.83:6789] 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:41.051 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 Processor -- start 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- start start 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecd41071a0 con 0x7fecd4104380 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fecd4104380 0x7fecd4104790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54536/0 (socket says 172.21.15.83:54536) 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd37fe640 1 -- 172.21.15.83:0/2693628645 learned_addr learned my addr 172.21.15.83:0/2693628645 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689921030 0 0) 0x7fecd41071a0 con 0x7fecd4104380 2024-09-10T18:38:41.052 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc00034e0 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266516406 0 0) 0x7fecc00034e0 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4108340 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fecc4001590 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc40018f0 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecd2ffd640 1 -- 172.21.15.83:0/2693628645 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fecc4001c40 con 0x7fecd4104380 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- 172.21.15.83:0/2693628645 >> v1:172.21.15.83:6789/0 conn(0x7fecd4104380 legacy=0x7fecd4104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- 172.21.15.83:0/2693628645 shutdown_connections 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- 172.21.15.83:0/2693628645 >> 172.21.15.83:0/2693628645 conn(0x7fecd4076320 msgr2=0x7fecd4076740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- 172.21.15.83:0/2693628645 shutdown_connections 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.328+0000 7fecda047640 1 -- 172.21.15.83:0/2693628645 wait complete. 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 librados: starting msgr at 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 librados: starting objecter 2024-09-10T18:38:41.053 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 Processor -- start 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 -- start start 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 librados: setting wanted keys 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 librados: calling monclient init 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecd41115b0 con 0x7fecd4104380 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fecd4104380 0x7fecd4110ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54538/0 (socket says 172.21.15.83:54538) 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd37fe640 1 -- 172.21.15.83:0/3437093257 learned_addr learned my addr 172.21.15.83:0/3437093257 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 10 client.?.objecter ms_handle_connect 0x7fecd4104380 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447162190 0 0) 0x7fecd41115b0 con 0x7fecd4104380 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc00044b0 con 0x7fecd4104380 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3229209948 0 0) 0x7fecc00044b0 con 0x7fecd4104380 2024-09-10T18:38:41.054 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecd41115b0 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fecc4001550 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2441049884 0 0) 0x7fecd41115b0 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4111740 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd4111a10 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd4115620 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecda047640 1 librados: init done 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecc4004c70 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fecc4004f80 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fecc40053a0 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7fecc4006370 con 0x7fecd4104380 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 10 client.4100.objecter ms_dispatch 0x7fecd4000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:41.055 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.332+0000 7fecd0ff9640 20 client.4100.objecter dump_active .. 0 homeless 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.336+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4005040 con 0x7fecd4104380 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.340+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7fecc4005830 con 0x7fecd4104380 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.456+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fecb4005870 con 0x7fecd4104380 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 10 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fecc4005a50 con 0x7fecd4104380 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecd0ff9640 1 -- 172.21.15.83:0/3437093257 <== mon.0 v1:172.21.15.83:6789/0 11 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+353 (unknown 1213389831 0 2940280748) 0x7fecc40076c0 con 0x7fecd4104380 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecda047640 10 librados: watch_flush enter 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecda047640 10 librados: watch_flush exit 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecda047640 20 client.4100.objecter shutdown clearing up homeless session... 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.464+0000 7fecda047640 10 client.4100.objecter successfully canceled tick 2024-09-10T18:38:41.056 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 >> v1:172.21.15.83:6789/0 conn(0x7fecd4104380 legacy=0x7fecd4110ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 shutdown_connections 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 >> 172.21.15.83:0/3437093257 conn(0x7fecd4076320 msgr2=0x7fecd4101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 shutdown_connections 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 -- 172.21.15.83:0/3437093257 wait complete. 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:40.468+0000 7fecda047640 1 librados: shutdown 2024-09-10T18:38:41.057 INFO:teuthology.orchestra.run.smithi083.stdout:Generating new minimal ceph.conf... 2024-09-10T18:38:42.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:42 smithi083 bash[18124]: audit 2024-09-10T18:38:42.270937+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.83:0/3178457154' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab2dc4c640 1 Processor -- start 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab2dc4c640 1 -- start start 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab2dc4c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab280ff3f0 con 0x7fab280fe8d0 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fab280fe8d0 0x7fab280fece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54546/0 (socket says 172.21.15.83:54546) 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab277fe640 1 -- 172.21.15.83:0/3844257796 learned_addr learned my addr 172.21.15.83:0/3844257796 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608907931 0 0) 0x7fab280ff3f0 con 0x7fab280fe8d0 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab140034e0 con 0x7fab280fe8d0 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400415814 0 0) 0x7fab140034e0 con 0x7fab280fe8d0 2024-09-10T18:38:42.879 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab28106280 con 0x7fab280fe8d0 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.132+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fab20001590 con 0x7fab280fe8d0 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fab20001bf0 con 0x7fab280fe8d0 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab267fc640 1 -- 172.21.15.83:0/3844257796 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fab20005600 con 0x7fab280fe8d0 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3844257796 >> v1:172.21.15.83:6789/0 conn(0x7fab280fe8d0 legacy=0x7fab280fece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3844257796 shutdown_connections 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3844257796 >> 172.21.15.83:0/3844257796 conn(0x7fab280fa430 msgr2=0x7fab280fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3844257796 shutdown_connections 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3844257796 wait complete. 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 librados: starting msgr at 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 librados: starting objecter 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 Processor -- start 2024-09-10T18:38:42.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- start start 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 librados: setting wanted keys 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 librados: calling monclient init 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab2dc4c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab2810fc50 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fab280fe8d0 0x7fab2810f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54554/0 (socket says 172.21.15.83:54554) 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.136+0000 7fab277fe640 1 -- 172.21.15.83:0/3178457154 learned_addr learned my addr 172.21.15.83:0/3178457154 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 10 client.?.objecter ms_handle_connect 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771672384 0 0) 0x7fab2810fc50 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab140056b0 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2392921618 0 0) 0x7fab140056b0 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab2810fc50 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fab20000ed0 con 0x7fab280fe8d0 2024-09-10T18:38:42.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2095933372 0 0) 0x7fab2810fc50 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2810fde0 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab281100b0 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab2dc4c640 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab28113cc0 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab2dc4c640 1 librados: init done 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fab20004c10 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fab20006070 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fab200075b0 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7fab20006620 con 0x7fab280fe8d0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 10 client.4102.objecter ms_dispatch 0x7fab28000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:42.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab1ffff640 20 client.4102.objecter dump_active .. 0 homeless 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.140+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab08005040 con 0x7fab280fe8d0 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.148+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7fab20005940 con 0x7fab280fe8d0 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fab080052f0 con 0x7fab280fe8d0 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab1ffff640 1 -- 172.21.15.83:0/3178457154 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+147 (unknown 1452402520 0 2211039233) 0x7fab20007a40 con 0x7fab280fe8d0 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 10 librados: watch_flush enter 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 10 librados: watch_flush exit 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 20 client.4102.objecter shutdown clearing up homeless session... 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 10 client.4102.objecter successfully canceled tick 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 >> v1:172.21.15.83:6789/0 conn(0x7fab280fe8d0 legacy=0x7fab2810f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 shutdown_connections 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 >> 172.21.15.83:0/3178457154 conn(0x7fab280fa430 msgr2=0x7fab28102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 shutdown_connections 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 -- 172.21.15.83:0/3178457154 wait complete. 2024-09-10T18:38:42.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:42.268+0000 7fab2dc4c640 1 librados: shutdown 2024-09-10T18:38:42.884 INFO:teuthology.orchestra.run.smithi083.stdout:Restarting the monitor... 2024-09-10T18:38:43.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:42 smithi083 systemd[1]: Stopping Ceph mon.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T18:38:43.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 bash[18124]: debug 2024-09-10T18:38:43.024+0000 7f0ad303d640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-10T18:38:43.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 bash[18124]: debug 2024-09-10T18:38:43.024+0000 7f0ad303d640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-10T18:38:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 bash[18529]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-mon-a 2024-09-10T18:38:43.797 INFO:teuthology.orchestra.run.smithi083.stdout:Setting public_network to 172.21.15.254/32,172.21.0.0/20 in mon config section 2024-09-10T18:38:43.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 bash[18591]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-mon-a 2024-09-10T18:38:43.952 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service: Succeeded. 2024-09-10T18:38:43.952 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 systemd[1]: Stopped Ceph mon.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:38:43.952 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:43 smithi083 systemd[1]: Started Ceph mon.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.280+0000 7facac898900 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.280+0000 7facac898900 0 ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.280+0000 7facac898900 0 pidfile_write: ignore empty --pid-file 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 0 load: jerasure load: lrc 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: RocksDB version: 6.15.5 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Compile date Sep 9 2024 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: DB SUMMARY 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: DB Session ID: OJAZUUJM2BIY5MPNLL4X 2024-09-10T18:38:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: CURRENT file: CURRENT 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 89518 ; 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.error_if_exists: 0 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.create_if_missing: 0 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.paranoid_checks: 1 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.env: 0x55d600326440 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.fs: Posix File System 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.info_log: 0x55d6011bee80 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:38:45.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.statistics: (nil) 2024-09-10T18:38:45.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.use_fsync: 0 2024-09-10T18:38:45.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_log_file_size: 0 2024-09-10T18:38:45.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:38:45.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_fallocate: 1 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.use_direct_reads: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.db_log_dir: 2024-09-10T18:38:45.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.write_buffer_manager: 0x55d60127c780 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:38:45.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.unordered_write: 0 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.row_cache: None 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.wal_filter: None 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:38:45.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.preserve_deletes: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.two_write_queues: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.atomic_flush: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.log_readahead_size: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:38:45.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_background_jobs: 2 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_background_compactions: -1 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_subcompactions: 1 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_open_files: -1 2024-09-10T18:38:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_background_flushes: -1 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Compression algorithms supported: 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kZSTD supported: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kXpressCompression supported: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kBZip2Compression supported: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kLZ4Compression supported: 1 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kZlibCompression supported: 1 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:38:45.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: kSnappyCompression supported: 1 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.merge_operator: 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_filter: None 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d601167a38) 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cache_index_and_filter_blocks: 1 2024-09-10T18:38:45.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: pin_top_level_index_and_filter: 1 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: index_type: 0 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: data_block_index_type: 0 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: index_shortening: 1 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: hash_index_allow_collision: 1 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: checksum: 1 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: no_block_cache: 0 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_cache: 0x55d60119b350 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_cache_name: BinnedLRUCache 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_cache_options: 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: capacity : 536870912 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: num_shard_bits : 4 2024-09-10T18:38:45.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: strict_capacity_limit : 0 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: high_pri_pool_ratio: 0.000 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_cache_compressed: (nil) 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: persistent_cache: (nil) 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_size: 4096 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_size_deviation: 10 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_restart_interval: 16 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: index_block_restart_interval: 1 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: metadata_block_size: 4096 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: partition_filters: 0 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: use_delta_encoding: 1 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: whole_key_filtering: 1 2024-09-10T18:38:45.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: verify_compression: 0 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: read_amp_bytes_per_bit: 0 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: format_version: 4 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: enable_index_compression: 1 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: block_align: 0 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression: NoCompression 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.num_levels: 7 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:38:45.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:38:45.554 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:38:45.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:38:45.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:38:45.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.inplace_update_support: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.bloom_locality: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.max_successive_merges: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:38:45.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.ttl: 2592000 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.enable_blob_files: false 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.min_blob_size: 0 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.284+0000 7facac898900 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993525291664, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-10T18:38:45.559 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993525293127, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 85385, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 83475, "index_size": 244, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 11875, "raw_average_key_size": 48, "raw_value_size": 76955, "raw_average_value_size": 316, "num_data_blocks": 11, "num_entries": 243, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725993525, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "76d03179-a010-42c5-b4cf-e43547e7bd6d", "db_session_id": "OJAZUUJM2BIY5MPNLL4X"}} 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.288+0000 7facac898900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993525293990, "job": 1, "event": "recovery_finished"} 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7facac898900 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7facac898900 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55d6011d0700 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7facac898900 4 rocksdb: DB pointer 0x55d60119e000 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7faca2664640 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7faca2664640 4 rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** DB Stats ** 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:38:45.560 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** Compaction Stats [default] ** 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: L0 2/0 85.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Sum 2/0 85.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** Compaction Stats [default] ** 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:45.561 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Cumulative compaction: 0.00 GB write, 14.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Interval compaction: 0.00 GB write, 14.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** Compaction Stats [default] ** 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:45.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: L0 2/0 85.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Sum 2/0 85.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** Compaction Stats [default] ** 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 84.0 0.00 0.00 1 0.001 0 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:38:45.563 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Cumulative compaction: 0.00 GB write, 14.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7facac898900 0 starting mon.a rank 0 at public addrs v1:172.21.15.83:6789/0 at bind addrs v1:172.21.15.83:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.292+0000 7facac898900 1 mon.a@-1(???) e1 preinit fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).mds e1 new map 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).mds e1 print_map 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: e1 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: legacy client fscid: -1 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: 2024-09-10T18:38:45.564 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: No filesystems configured 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: debug 2024-09-10T18:38:45.296+0000 7facac898900 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cluster 2024-09-10T18:38:45.303788+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cluster 2024-09-10T18:38:45.303887+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.83:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cluster 2024-09-10T18:38:45.304526+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cluster 2024-09-10T18:38:45.304574+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-09-10T18:38:45.565 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:45 smithi083 bash[18629]: cluster 2024-09-10T18:38:45.305169+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-09-10T18:38:46.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.396+0000 7effcdc13640 1 Processor -- start 2024-09-10T18:38:46.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.396+0000 7effcdc13640 1 -- start start 2024-09-10T18:38:46.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.396+0000 7effcdc13640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7effc81015c0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.396+0000 7effc77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7effc8100aa0 0x7effc8100eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60182/0 (socket says 172.21.15.83:60182) 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.396+0000 7effc77fe640 1 -- 172.21.15.83:0/4095683641 learned_addr learned my addr 172.21.15.83:0/4095683641 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506145424 0 0) 0x7effc81015c0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb00034e0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2458556876 0 0) 0x7effb00034e0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8105fb0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7effb8001590 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7effb8001bf0 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effc67fc640 1 -- 172.21.15.83:0/4095683641 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7effb8005600 con 0x7effc8100aa0 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- 172.21.15.83:0/4095683641 >> v1:172.21.15.83:6789/0 conn(0x7effc8100aa0 legacy=0x7effc8100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- 172.21.15.83:0/4095683641 shutdown_connections 2024-09-10T18:38:46.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- 172.21.15.83:0/4095683641 >> 172.21.15.83:0/4095683641 conn(0x7effc80fc280 msgr2=0x7effc80fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- 172.21.15.83:0/4095683641 shutdown_connections 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- 172.21.15.83:0/4095683641 wait complete. 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 librados: starting msgr at 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 librados: starting objecter 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 Processor -- start 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 -- start start 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 librados: setting wanted keys 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.400+0000 7effcdc13640 1 librados: calling monclient init 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effcdc13640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7effc810fc00 con 0x7effc8100aa0 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effc77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7effc8100aa0 0x7effc810f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60196/0 (socket says 172.21.15.83:60196) 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effc77fe640 1 -- 172.21.15.83:0/2429841122 learned_addr learned my addr 172.21.15.83:0/2429841122 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 10 client.?.objecter ms_handle_connect 0x7effc8100aa0 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:46.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487763588 0 0) 0x7effc810fc00 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb00056b0 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550722684 0 0) 0x7effb00056b0 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effc810fc00 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7effb8000ed0 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4169931533 0 0) 0x7effc810fc00 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc810fd90 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effcdc13640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc8110060 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effcdc13640 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effcdc13640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effc8113c70 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effcdc13640 1 librados: init done 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7effb80017b0 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7effb80061a0 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7effb8006660 con 0x7effc8100aa0 2024-09-10T18:38:46.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7effb8007630 con 0x7effc8100aa0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 10 client.14098.objecter ms_dispatch 0x7effc8000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effaffff640 20 client.14098.objecter dump_active .. 0 homeless 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.404+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8005010 con 0x7effc8100aa0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.408+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7effb8001920 con 0x7effc8100aa0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.520+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7effa80052c0 con 0x7effc8100aa0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effaffff640 1 -- 172.21.15.83:0/2429841122 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 141+0+0 (unknown 223993786 0 0) 0x7effb8005a30 con 0x7effc8100aa0 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 10 librados: watch_flush enter 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 10 librados: watch_flush exit 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 20 client.14098.objecter shutdown clearing up homeless session... 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 10 client.14098.objecter successfully canceled tick 2024-09-10T18:38:46.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 >> v1:172.21.15.83:6789/0 conn(0x7effc8100aa0 legacy=0x7effc810f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:46.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 shutdown_connections 2024-09-10T18:38:46.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 >> 172.21.15.83:0/2429841122 conn(0x7effc80fc280 msgr2=0x7effc81063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:46.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 shutdown_connections 2024-09-10T18:38:46.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 -- 172.21.15.83:0/2429841122 wait complete. 2024-09-10T18:38:46.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:45.528+0000 7effadffb640 1 librados: shutdown 2024-09-10T18:38:46.230 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-10T18:38:46.230 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:38:46.230 INFO:teuthology.orchestra.run.smithi083.stdout:Creating mgr... 2024-09-10T18:38:46.230 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-10T18:38:46.230 INFO:teuthology.orchestra.run.smithi083.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-10T18:38:46.439 INFO:teuthology.orchestra.run.smithi083.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y 2024-09-10T18:38:46.439 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Failed to reset failed state of unit ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-10T18:38:46.639 INFO:teuthology.orchestra.run.smithi083.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968.target.wants/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y.service -> /etc/systemd/system/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@.service. 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld does not appear to be present 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld does not appear to be present 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr to start... 2024-09-10T18:38:46.649 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr... 2024-09-10T18:38:46.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:46 smithi083 bash[18629]: audit 2024-09-10T18:38:45.531742+0000 mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.83:0/2429841122' entity='client.admin' 2024-09-10T18:38:48.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:48 smithi083 bash[18629]: audit 2024-09-10T18:38:48.490507+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.83:0/3872831362' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:38:48.679 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:48 smithi083 bash[18896]: debug 2024-09-10T18:38:48.348+0000 7f83528f9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "707a4ada-6fa3-11ef-bcea-c7b262605968", 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:38:49.057 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:38:49.058 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:38:49.059 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:38:49.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:38:49.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:38:38.476379+0000", 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:38:49.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.356+0000 7f65c1913640 1 Processor -- start 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.356+0000 7f65c1913640 1 -- start start 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.356+0000 7f65c1913640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65bc100460 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.356+0000 7f65baffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f65bc0ff940 0x7f65bc0ffd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60230/0 (socket says 172.21.15.83:60230) 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.356+0000 7f65baffd640 1 -- 172.21.15.83:0/3287959447 learned_addr learned my addr 172.21.15.83:0/3287959447 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619142975 0 0) 0x7f65bc100460 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a40034e0 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808414059 0 0) 0x7f65a40034e0 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc104e50 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f65ac001590 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f65ac001bf0 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65b9ffb640 1 -- 172.21.15.83:0/3287959447 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f65ac005600 con 0x7f65bc0ff940 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- 172.21.15.83:0/3287959447 >> v1:172.21.15.83:6789/0 conn(0x7f65bc0ff940 legacy=0x7f65bc0ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- 172.21.15.83:0/3287959447 shutdown_connections 2024-09-10T18:38:49.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- 172.21.15.83:0/3287959447 >> 172.21.15.83:0/3287959447 conn(0x7f65bc0fb140 msgr2=0x7f65bc0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- 172.21.15.83:0/3287959447 shutdown_connections 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- 172.21.15.83:0/3287959447 wait complete. 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 librados: starting msgr at 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 librados: starting objecter 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 Processor -- start 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- start start 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 librados: setting wanted keys 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 librados: calling monclient init 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65c1913640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65bc111e10 con 0x7f65bc0ff940 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65baffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f65bc0ff940 0x7f65bc111700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60232/0 (socket says 172.21.15.83:60232) 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65baffd640 1 -- 172.21.15.83:0/3872831362 learned_addr learned my addr 172.21.15.83:0/3872831362 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65a37fe640 10 client.?.objecter ms_handle_connect 0x7f65bc0ff940 2024-09-10T18:38:49.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65a37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555461195 0 0) 0x7f65bc111e10 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.360+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a40056b0 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3055878801 0 0) 0x7f65a40056b0 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65bc111e10 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f65ac000ed0 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3599560863 0 0) 0x7f65bc111e10 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc111fa0 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc112270 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65c1913640 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc115e80 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65c1913640 1 librados: init done 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f65ac004b10 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f65ac0060c0 con 0x7f65bc0ff940 2024-09-10T18:38:49.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f65ac0064a0 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f65ac006a90 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 10 client.14102.objecter ms_dispatch 0x7f65bc000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65a37fe640 20 client.14102.objecter dump_active .. 0 homeless 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.364+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a8005040 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.368+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f65ac006c50 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.484+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f65a80052f0 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65a37fe640 1 -- 172.21.15.83:0/3872831362 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (unknown 4201413639 0 1037453160) 0x7f65ac005940 con 0x7f65bc0ff940 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 10 librados: watch_flush enter 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 10 librados: watch_flush exit 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 20 client.14102.objecter shutdown clearing up homeless session... 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 10 client.14102.objecter successfully canceled tick 2024-09-10T18:38:49.074 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 >> v1:172.21.15.83:6789/0 conn(0x7f65bc0ff940 legacy=0x7f65bc111700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 shutdown_connections 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 >> 172.21.15.83:0/3872831362 conn(0x7f65bc0fb140 msgr2=0x7f65bc0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 shutdown_connections 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 -- 172.21.15.83:0/3872831362 wait complete. 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:48.488+0000 7f65c1913640 1 librados: shutdown 2024-09-10T18:38:49.075 INFO:teuthology.orchestra.run.smithi083.stdout:mgr not available, waiting (1/15)... 2024-09-10T18:38:49.389 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.108+0000 7f83528f9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:38:49.389 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.172+0000 7f83528f9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:38:49.389 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.236+0000 7f83528f9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:38:49.693 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.384+0000 7f83528f9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:38:49.693 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.628+0000 7f83528f9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:38:50.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.688+0000 7f83528f9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:38:50.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:49 smithi083 bash[18896]: debug 2024-09-10T18:38:49.820+0000 7f83528f9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:38:50.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:50 smithi083 bash[18896]: debug 2024-09-10T18:38:50.444+0000 7f83528f9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:38:50.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:50 smithi083 bash[18896]: debug 2024-09-10T18:38:50.516+0000 7f83528f9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:38:50.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:50 smithi083 bash[18896]: debug 2024-09-10T18:38:50.588+0000 7f83528f9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:38:51.154 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:50 smithi083 bash[18896]: debug 2024-09-10T18:38:50.864+0000 7f83528f9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:38:51.154 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:51 smithi083 bash[18896]: debug 2024-09-10T18:38:50.996+0000 7f83528f9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:38:51.525 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:51 smithi083 bash[18896]: debug 2024-09-10T18:38:51.308+0000 7f83528f9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:38:51.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:51 smithi083 bash[18896]: debug 2024-09-10T18:38:51.520+0000 7f83528f9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:38:52.237 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:51 smithi083 bash[18896]: debug 2024-09-10T18:38:51.936+0000 7f83528f9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:38:52.237 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:51.996+0000 7f83528f9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:38:52.237 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.064+0000 7f83528f9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:38:52.237 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.168+0000 7f83528f9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:38:52.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18629]: audit 2024-09-10T18:38:52.394565+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.83:0/2664376532' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:38:52.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.232+0000 7f83528f9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:38:52.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.316+0000 7f83528f9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:38:52.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.468+0000 7f83528f9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:38:53.006 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "707a4ada-6fa3-11ef-bcea-c7b262605968", 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:38:53.007 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:38:53.008 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:38:53.018 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:38:53.019 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:38:38.476379+0000", 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.248+0000 7f866aac7640 1 Processor -- start 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.248+0000 7f866aac7640 1 -- start start 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.248+0000 7f866aac7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8664100460 con 0x7f86640ff940 2024-09-10T18:38:53.021 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f8668863640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f86640ff940 0x7f86640ffd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60248/0 (socket says 172.21.15.83:60248) 2024-09-10T18:38:53.023 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f8668863640 1 -- 172.21.15.83:0/46463909 learned_addr learned my addr 172.21.15.83:0/46463909 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713024311 0 0) 0x7f8664100460 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86540034e0 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183370609 0 0) 0x7f86540034e0 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8664104e50 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8658001590 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f8658001cc0 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f86637fe640 1 -- 172.21.15.83:0/46463909 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f86580056d0 con 0x7f86640ff940 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- 172.21.15.83:0/46463909 >> v1:172.21.15.83:6789/0 conn(0x7f86640ff940 legacy=0x7f86640ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- 172.21.15.83:0/46463909 shutdown_connections 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- 172.21.15.83:0/46463909 >> 172.21.15.83:0/46463909 conn(0x7f86640fb140 msgr2=0x7f86640fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:53.024 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- 172.21.15.83:0/46463909 shutdown_connections 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- 172.21.15.83:0/46463909 wait complete. 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 librados: starting msgr at 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 librados: starting objecter 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 Processor -- start 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- start start 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 librados: setting wanted keys 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 librados: calling monclient init 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.252+0000 7f866aac7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f866410fbc0 con 0x7f86640ff940 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f8668863640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f86640ff940 0x7f866410f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60260/0 (socket says 172.21.15.83:60260) 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f8668863640 1 -- 172.21.15.83:0/2664376532 learned_addr learned my addr 172.21.15.83:0/2664376532 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 10 client.?.objecter ms_handle_connect 0x7f86640ff940 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:53.025 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277915811 0 0) 0x7f866410fbc0 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654005660 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200536632 0 0) 0x7f8654005660 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866410fbc0 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8658001060 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1486643136 0 0) 0x7f866410fbc0 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866410fd50 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8664110020 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f866aac7640 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8664113c30 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f866aac7640 1 librados: init done 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f86580047a0 con 0x7f86640ff940 2024-09-10T18:38:53.026 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8658006070 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f8658006530 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f8658006b20 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 10 client.14104.objecter ms_dispatch 0x7f8664000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f86617fa640 20 client.14104.objecter dump_active .. 0 homeless 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.256+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8648005040 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.260+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f8658006d60 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f86480052f0 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f86617fa640 1 -- 172.21.15.83:0/2664376532 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (unknown 4201413639 0 783874904) 0x7f8658005a00 con 0x7f86640ff940 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 10 librados: watch_flush enter 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 10 librados: watch_flush exit 2024-09-10T18:38:53.027 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 20 client.14104.objecter shutdown clearing up homeless session... 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 10 client.14104.objecter successfully canceled tick 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 >> v1:172.21.15.83:6789/0 conn(0x7f86640ff940 legacy=0x7f866410f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 shutdown_connections 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 >> 172.21.15.83:0/2664376532 conn(0x7f86640fb140 msgr2=0x7f86640fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 shutdown_connections 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 -- 172.21.15.83:0/2664376532 wait complete. 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:52.392+0000 7f866aac7640 1 librados: shutdown 2024-09-10T18:38:53.028 INFO:teuthology.orchestra.run.smithi083.stdout:mgr not available, waiting (2/15)... 2024-09-10T18:38:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:52 smithi083 bash[18896]: debug 2024-09-10T18:38:52.544+0000 7f83528f9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:38:53.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18896]: debug 2024-09-10T18:38:53.448+0000 7f83528f9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:38:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: cluster 2024-09-10T18:38:53.472161+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-09-10T18:38:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: cluster 2024-09-10T18:38:53.478650+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00676848s) 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.481298+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.481662+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.481958+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.482254+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.482577+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.482879+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.483157+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.483535+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: cluster 2024-09-10T18:38:53.493808+0000 mon.a (mon.0) 19 : cluster [INF] Manager daemon y is now available 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.513734+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:38:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:53 smithi083 bash[18629]: audit 2024-09-10T18:38:53.518165+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:38:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:54 smithi083 bash[18629]: audit 2024-09-10T18:38:53.524695+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' 2024-09-10T18:38:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:54 smithi083 bash[18629]: audit 2024-09-10T18:38:53.534928+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' 2024-09-10T18:38:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:54 smithi083 bash[18629]: audit 2024-09-10T18:38:53.540995+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 v1:172.21.15.83:0/1290025894' entity='mgr.y' 2024-09-10T18:38:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:54 smithi083 bash[18629]: cluster 2024-09-10T18:38:54.485419+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e3: y(active, since 1.01353s) 2024-09-10T18:38:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:56 smithi083 bash[18629]: cluster 2024-09-10T18:38:55.536473+0000 mon.a (mon.0) 26 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsid": "707a4ada-6fa3-11ef-bcea-c7b262605968", 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:38:57.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:57.225 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:38:57.226 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.227 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:38:38.476379+0000", 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:38:57.228 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:38:57.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:38:57.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:38:57.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f0636967640 1 Processor -- start 2024-09-10T18:38:57.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f0636967640 1 -- start start 2024-09-10T18:38:57.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f0636967640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f063007b380 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0630107cf0 0x7f063010a0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43410/0 (socket says 172.21.15.83:43410) 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062ffff640 1 -- 172.21.15.83:0/713102786 learned_addr learned my addr 172.21.15.83:0/713102786 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121657900 0 0) 0x7f063007b380 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c0034e0 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424937200 0 0) 0x7f061c0034e0 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.184+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063007b510 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f0620001590 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f0620001bf0 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062f7fe640 1 -- 172.21.15.83:0/713102786 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f0620005600 con 0x7f0630107cf0 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- 172.21.15.83:0/713102786 >> v1:172.21.15.83:6789/0 conn(0x7f0630107cf0 legacy=0x7f063010a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- 172.21.15.83:0/713102786 shutdown_connections 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- 172.21.15.83:0/713102786 >> 172.21.15.83:0/713102786 conn(0x7f0630076320 msgr2=0x7f0630078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:57.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- 172.21.15.83:0/713102786 shutdown_connections 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- 172.21.15.83:0/713102786 wait complete. 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 librados: starting msgr at 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 librados: starting objecter 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 Processor -- start 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- start start 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 librados: setting wanted keys 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 librados: calling monclient init 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f0636967640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f06301161f0 con 0x7f0630107cf0 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0630107cf0 0x7f0630115ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43418/0 (socket says 172.21.15.83:43418) 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062ffff640 1 -- 172.21.15.83:0/1565323910 learned_addr learned my addr 172.21.15.83:0/1565323910 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:57.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062d7fa640 10 client.?.objecter ms_handle_connect 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034971100 0 0) 0x7f06301161f0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.188+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c0056b0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2148344364 0 0) 0x7f061c0056b0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06301161f0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f0620000ed0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4175323967 0 0) 0x7f06301161f0 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0630116380 con 0x7f0630107cf0 2024-09-10T18:38:57.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0630116650 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f0620004b50 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f06200060a0 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f0636967640 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f063011a260 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f0636967640 1 librados: init done 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 45446+0+0 (unknown 1173535182 0 0) 0x7f0620006200 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f0620048520 con 0x7f0630107cf0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 10 client.14112.objecter ms_dispatch 0x7f0630000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 20 client.14112.objecter dump_active .. 0 homeless 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f062d7fa640 10 client.14112.objecter ms_handle_connect 0x7f061c03c130 2024-09-10T18:38:57.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.192+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f063005ed70 con 0x7f0630107cf0 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.200+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f06200079a0 con 0x7f0630107cf0 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.643+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f063010f880 con 0x7f0630107cf0 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.643+0000 7f062d7fa640 1 -- 172.21.15.83:0/1565323910 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (unknown 4201413639 0 1606952275) 0x7f06200163b0 con 0x7f0630107cf0 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 10 librados: watch_flush enter 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 10 librados: watch_flush exit 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 20 client.14112.objecter shutdown clearing up homeless session... 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 10 client.14112.objecter successfully canceled tick 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 >> v1:172.21.15.83:6800/3275901971 conn(0x7f061c03c130 legacy=0x7f061c03e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 >> v1:172.21.15.83:6789/0 conn(0x7f0630107cf0 legacy=0x7f0630115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 shutdown_connections 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 >> 172.21.15.83:0/1565323910 conn(0x7f0630076320 msgr2=0x7f063010c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 shutdown_connections 2024-09-10T18:38:57.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 -- 172.21.15.83:0/1565323910 wait complete. 2024-09-10T18:38:57.235 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:56.647+0000 7f0636967640 1 librados: shutdown 2024-09-10T18:38:57.235 INFO:teuthology.orchestra.run.smithi083.stdout:mgr is available 2024-09-10T18:38:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:57 smithi083 bash[18629]: audit 2024-09-10T18:38:56.649334+0000 mon.a (mon.0) 27 : audit [DBG] from='client.? v1:172.21.15.83:0/1565323910' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:38:59.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:38:58 smithi083 bash[18629]: audit 2024-09-10T18:38:58.749305+0000 mon.a (mon.0) 28 : audit [INF] from='client.? v1:172.21.15.83:0/3204507104' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:38:59.344 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:59.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:38:59.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout fsid = 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:38:59.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:38:59.347 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 2024-09-10T18:38:59.347 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:38:59.347 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:38:59.347 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:38:59.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.403+0000 7f3f6cad7640 1 Processor -- start 2024-09-10T18:38:59.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.403+0000 7f3f6cad7640 1 -- start start 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.403+0000 7f3f6cad7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f68075950 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.403+0000 7f3f6659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3f68076e10 0x7f3f68075240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43422/0 (socket says 172.21.15.83:43422) 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.403+0000 7f3f6659c640 1 -- 172.21.15.83:0/3688963013 learned_addr learned my addr 172.21.15.83:0/3688963013 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989728585 0 0) 0x7f3f68075950 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f540034e0 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1146599 0 0) 0x7f3f540034e0 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68075ae0 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3f60001910 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f3f60002040 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6559a640 1 -- 172.21.15.83:0/3688963013 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3f60005a50 con 0x7f3f68076e10 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3688963013 >> v1:172.21.15.83:6789/0 conn(0x7f3f68076e10 legacy=0x7f3f68075240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3688963013 shutdown_connections 2024-09-10T18:38:59.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3688963013 >> 172.21.15.83:0/3688963013 conn(0x7f3f680fc240 msgr2=0x7f3f680fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3688963013 shutdown_connections 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3688963013 wait complete. 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 librados: starting msgr at 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 librados: starting objecter 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 Processor -- start 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 -- start start 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 librados: setting wanted keys 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.407+0000 7f3f6cad7640 1 librados: calling monclient init 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3f68111dd0 con 0x7f3f68076e10 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3f68076e10 0x7f3f681116c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43434/0 (socket says 172.21.15.83:43434) 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6659c640 1 -- 172.21.15.83:0/3204507104 learned_addr learned my addr 172.21.15.83:0/3204507104 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 10 client.?.objecter ms_handle_connect 0x7f3f68076e10 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:38:59.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096220207 0 0) 0x7f3f68111dd0 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f54005660 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3372356759 0 0) 0x7f3f54005660 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f68111dd0 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3f600013e0 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3378019049 0 0) 0x7f3f68111dd0 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68111f60 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f68112230 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f68115e40 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 1 librados: init done 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f3f600059f0 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3f60006320 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 45446+0+0 (unknown 1173535182 0 0) 0x7f3f60011610 con 0x7f3f68076e10 2024-09-10T18:38:59.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f48005040 con 0x7f3f68076e10 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f3f60047a90 con 0x7f3f68076e10 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 10 client.14114.objecter ms_dispatch 0x7f3f68000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.411+0000 7f3f5effd640 20 client.14114.objecter dump_active .. 0 homeless 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.415+0000 7f3f5effd640 10 client.14114.objecter ms_handle_connect 0x7f3f5403c0c0 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.423+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f3f60012f30 con 0x7f3f68076e10 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.743+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3f48005870 con 0x7f3f68076e10 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.747+0000 7f3f5effd640 1 -- 172.21.15.83:0/3204507104 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+318 (unknown 1187553405 0 1854333688) 0x7f3f60016740 con 0x7f3f68076e10 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 10 librados: watch_flush enter 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 10 librados: watch_flush exit 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 20 client.14114.objecter shutdown clearing up homeless session... 2024-09-10T18:38:59.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 10 client.14114.objecter successfully canceled tick 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 >> v1:172.21.15.83:6800/3275901971 conn(0x7f3f5403c0c0 legacy=0x7f3f5403e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 >> v1:172.21.15.83:6789/0 conn(0x7f3f68076e10 legacy=0x7f3f681116c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 shutdown_connections 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 >> 172.21.15.83:0/3204507104 conn(0x7f3f680fc240 msgr2=0x7f3f681051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 shutdown_connections 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 -- 172.21.15.83:0/3204507104 wait complete. 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:38:58.751+0000 7f3f6cad7640 1 librados: shutdown 2024-09-10T18:38:59.357 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling cephadm module... 2024-09-10T18:39:01.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18629]: audit 2024-09-10T18:39:00.943431+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.83:0/3900114253' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-10T18:39:01.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18896]: ignoring --setuser ceph since I am not root 2024-09-10T18:39:01.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18896]: ignoring --setgroup ceph since I am not root 2024-09-10T18:39:01.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18896]: debug 2024-09-10T18:39:01.147+0000 7f9e7e62eec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:39:01.598 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3bfb3b7640 1 Processor -- start 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3bfb3b7640 1 -- start start 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3bfb3b7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bf4104fa0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3bf9153640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3bf4102180 0x7f3bf4102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43436/0 (socket says 172.21.15.83:43436) 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3bf9153640 1 -- 172.21.15.83:0/935287285 learned_addr learned my addr 172.21.15.83:0/935287285 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439911070 0 0) 0x7f3bf4104fa0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be00034e0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2121894953 0 0) 0x7f3be00034e0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4106140 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3be8001590 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f3be8001cc0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.515+0000 7f3be7fff640 1 -- 172.21.15.83:0/935287285 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3be80056c0 con 0x7f3bf4102180 2024-09-10T18:39:01.599 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/935287285 >> v1:172.21.15.83:6789/0 conn(0x7f3bf4102180 legacy=0x7f3bf4102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/935287285 shutdown_connections 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/935287285 >> 172.21.15.83:0/935287285 conn(0x7f3bf40fc240 msgr2=0x7f3bf40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/935287285 shutdown_connections 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/935287285 wait complete. 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 librados: starting msgr at 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 librados: starting objecter 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 Processor -- start 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- start start 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 librados: setting wanted keys 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 librados: calling monclient init 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bf4111db0 con 0x7f3bf4102180 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bf9153640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3bf4102180 0x7f3bf41116a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43448/0 (socket says 172.21.15.83:43448) 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bf9153640 1 -- 172.21.15.83:0/3900114253 learned_addr learned my addr 172.21.15.83:0/3900114253 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:01.600 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 10 client.?.objecter ms_handle_connect 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001625770 0 0) 0x7f3bf4111db0 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0005630 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3235279743 0 0) 0x7f3be0005630 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bf4111db0 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3be80047a0 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4136881052 0 0) 0x7f3bf4111db0 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf4111f40 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf4112210 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf4115e20 con 0x7f3bf4102180 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3bfb3b7640 1 librados: init done 2024-09-10T18:39:01.601 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.519+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f3be80011c0 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f3be8006010 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 4) v1 ==== 45446+0+0 (unknown 1173535182 0 0) 0x7f3be80062c0 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f3be8047800 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 10 client.14116.objecter ms_dispatch 0x7f3bf4000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 20 client.14116.objecter dump_active .. 0 homeless 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd8005010 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.523+0000 7f3be5ffb640 10 client.14116.objecter ms_handle_connect 0x7f3be003bdc0 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.531+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f3be8016b90 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.939+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f3bd80052c0 con 0x7f3bf4102180 2024-09-10T18:39:01.602 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.995+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 5) v1 ==== 45457+0+0 (unknown 1231082550 0 0) 0x7f3be80076d0 con 0x7f3bf4102180 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.995+0000 7f3be5ffb640 1 -- 172.21.15.83:0/3900114253 <== mon.0 v1:172.21.15.83:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7f3be8016dc0 con 0x7f3bf4102180 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.999+0000 7f3bfb3b7640 10 librados: watch_flush enter 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.999+0000 7f3bfb3b7640 10 librados: watch_flush exit 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.999+0000 7f3bfb3b7640 20 client.14116.objecter shutdown clearing up homeless session... 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:00.999+0000 7f3bfb3b7640 10 client.14116.objecter successfully canceled tick 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 >> v1:172.21.15.83:6800/3275901971 conn(0x7f3be003bdc0 legacy=0x7f3be003e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 >> v1:172.21.15.83:6789/0 conn(0x7f3bf4102180 legacy=0x7f3bf41116a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 shutdown_connections 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 >> 172.21.15.83:0/3900114253 conn(0x7f3bf40fc240 msgr2=0x7f3bf40fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 shutdown_connections 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 -- 172.21.15.83:0/3900114253 wait complete. 2024-09-10T18:39:01.603 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:01.003+0000 7f3bfb3b7640 1 librados: shutdown 2024-09-10T18:39:02.003 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18896]: debug 2024-09-10T18:39:01.875+0000 7f9e7e62eec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:39:02.003 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:01 smithi083 bash[18896]: debug 2024-09-10T18:39:01.943+0000 7f9e7e62eec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:39:02.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18629]: audit 2024-09-10T18:39:01.001296+0000 mon.a (mon.0) 30 : audit [INF] from='client.? v1:172.21.15.83:0/3900114253' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-10T18:39:02.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18629]: cluster 2024-09-10T18:39:01.001394+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e5: y(active, since 7s) 2024-09-10T18:39:02.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18896]: debug 2024-09-10T18:39:02.007+0000 7f9e7e62eec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:39:02.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18896]: debug 2024-09-10T18:39:02.159+0000 7f9e7e62eec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:39:02.706 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18896]: debug 2024-09-10T18:39:02.407+0000 7f9e7e62eec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:39:02.706 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18896]: debug 2024-09-10T18:39:02.471+0000 7f9e7e62eec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:39:02.707 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:02 smithi083 bash[18896]: debug 2024-09-10T18:39:02.607+0000 7f9e7e62eec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:39:03.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18629]: audit 2024-09-10T18:39:03.245733+0000 mon.a (mon.0) 32 : audit [DBG] from='client.? v1:172.21.15.83:0/1378228792' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:39:03.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18896]: debug 2024-09-10T18:39:03.251+0000 7f9e7e62eec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:39:03.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18896]: debug 2024-09-10T18:39:03.327+0000 7f9e7e62eec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:39:03.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18896]: debug 2024-09-10T18:39:03.407+0000 7f9e7e62eec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.819+0000 7f8d528cc640 1 Processor -- start 2024-09-10T18:39:03.840 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.819+0000 7f8d528cc640 1 -- start start 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.819+0000 7f8d528cc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d4c077140 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8d4c076d30 0x7f8d4c075260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43486/0 (socket says 172.21.15.83:43486) 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4bfff640 1 -- 172.21.15.83:0/294635036 learned_addr learned my addr 172.21.15.83:0/294635036 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766439941 0 0) 0x7f8d4c077140 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d340034e0 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914524285 0 0) 0x7f8d340034e0 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c075970 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8d3c001590 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f8d3c001cc0 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d4affd640 1 -- 172.21.15.83:0/294635036 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8d3c0056d0 con 0x7f8d4c076d30 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- 172.21.15.83:0/294635036 >> v1:172.21.15.83:6789/0 conn(0x7f8d4c076d30 legacy=0x7f8d4c075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- 172.21.15.83:0/294635036 shutdown_connections 2024-09-10T18:39:03.841 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- 172.21.15.83:0/294635036 >> 172.21.15.83:0/294635036 conn(0x7f8d4c0fc1a0 msgr2=0x7f8d4c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- 172.21.15.83:0/294635036 shutdown_connections 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- 172.21.15.83:0/294635036 wait complete. 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 librados: starting msgr at 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 librados: starting objecter 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 Processor -- start 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- start start 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 librados: setting wanted keys 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 librados: calling monclient init 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.823+0000 7f8d528cc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d4c111d60 con 0x7f8d4c076d30 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d4bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8d4c076d30 0x7f8d4c111650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43500/0 (socket says 172.21.15.83:43500) 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d4bfff640 1 -- 172.21.15.83:0/1378228792 learned_addr learned my addr 172.21.15.83:0/1378228792 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 10 client.?.objecter ms_handle_connect 0x7f8d4c076d30 2024-09-10T18:39:03.842 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539273296 0 0) 0x7f8d4c111d60 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34005660 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3420263471 0 0) 0x7f8d34005660 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d4c111d60 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8d3c001060 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2718501440 0 0) 0x7f8d4c111d60 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c111ef0 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c1121c0 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d528cc640 10 client.14120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c115dd0 con 0x7f8d4c076d30 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d528cc640 1 librados: init done 2024-09-10T18:39:03.843 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f8d3c0047a0 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8d3c006080 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 5) v1 ==== 45457+0+0 (unknown 1231082550 0 0) 0x7f8d3c011440 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f8d3c048740 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 10 client.14120.objecter ms_dispatch 0x7f8d4c000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 3 client.14120.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 3 client.14120.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d48ff9640 20 client.14120.objecter dump_active .. 0 homeless 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d4b7fe640 1 -- 172.21.15.83:0/1378228792 >> v1:172.21.15.83:6800/3275901971 conn(0x7f8d3403c000 legacy=0x7f8d3403e4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/3275901971 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.827+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c04b720 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:02.835+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8d3c012e00 con 0x7f8d4c076d30 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.031+0000 7f8d4b7fe640 1 -- 172.21.15.83:0/1378228792 >> v1:172.21.15.83:6800/3275901971 conn(0x7f8d3403c000 legacy=0x7f8d3403e4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/3275901971 2024-09-10T18:39:03.844 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.239+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f8d4c116160 con 0x7f8d4c076d30 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.239+0000 7f8d48ff9640 1 -- 172.21.15.83:0/1378228792 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f8d3c0165d0 con 0x7f8d4c076d30 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 10 librados: watch_flush enter 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 10 librados: watch_flush exit 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 20 client.14120.objecter shutdown clearing up homeless session... 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 10 client.14120.objecter successfully canceled tick 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 >> v1:172.21.15.83:6800/3275901971 conn(0x7f8d3403c000 legacy=0x7f8d3403e4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 >> v1:172.21.15.83:6789/0 conn(0x7f8d4c076d30 legacy=0x7f8d4c111650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 shutdown_connections 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 >> 172.21.15.83:0/1378228792 conn(0x7f8d4c0fc1a0 msgr2=0x7f8d4c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 shutdown_connections 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 -- 172.21.15.83:0/1378228792 wait complete. 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:03.243+0000 7f8d528cc640 1 librados: shutdown 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for the mgr to restart... 2024-09-10T18:39:03.845 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr epoch 5... 2024-09-10T18:39:03.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18896]: debug 2024-09-10T18:39:03.687+0000 7f9e7e62eec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:39:03.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:03 smithi083 bash[18896]: debug 2024-09-10T18:39:03.823+0000 7f9e7e62eec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:39:04.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:04 smithi083 bash[18896]: debug 2024-09-10T18:39:04.139+0000 7f9e7e62eec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:39:04.778 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:04 smithi083 bash[18896]: debug 2024-09-10T18:39:04.355+0000 7f9e7e62eec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:39:05.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:04 smithi083 bash[18896]: debug 2024-09-10T18:39:04.771+0000 7f9e7e62eec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:39:05.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:04 smithi083 bash[18896]: debug 2024-09-10T18:39:04.835+0000 7f9e7e62eec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:39:05.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:04 smithi083 bash[18896]: debug 2024-09-10T18:39:04.903+0000 7f9e7e62eec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:39:05.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:05 smithi083 bash[18896]: debug 2024-09-10T18:39:05.011+0000 7f9e7e62eec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:39:05.382 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:05 smithi083 bash[18896]: debug 2024-09-10T18:39:05.079+0000 7f9e7e62eec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:39:05.382 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:05 smithi083 bash[18896]: debug 2024-09-10T18:39:05.155+0000 7f9e7e62eec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:39:05.382 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:05 smithi083 bash[18896]: debug 2024-09-10T18:39:05.303+0000 7f9e7e62eec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:39:05.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:05 smithi083 bash[18896]: debug 2024-09-10T18:39:05.375+0000 7f9e7e62eec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:39:06.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18896]: debug 2024-09-10T18:39:06.311+0000 7f9e7e62eec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:39:06.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: cluster 2024-09-10T18:39:06.336226+0000 mon.a (mon.0) 33 : cluster [INF] Active manager daemon y restarted 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: cluster 2024-09-10T18:39:06.336678+0000 mon.a (mon.0) 34 : cluster [INF] Activating manager daemon y 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: cluster 2024-09-10T18:39:06.342318+0000 mon.a (mon.0) 35 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: cluster 2024-09-10T18:39:06.342567+0000 mon.a (mon.0) 36 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00605245s) 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.345430+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.347602+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.348662+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.349045+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.349351+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: cluster 2024-09-10T18:39:06.360226+0000 mon.a (mon.0) 42 : cluster [INF] Manager daemon y is now available 2024-09-10T18:39:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:06 smithi083 bash[18629]: audit 2024-09-10T18:39:06.383634+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: cephadm 2024-09-10T18:39:06.376637+0000 mgr.y (mgr.14118) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-09-10T18:39:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: audit 2024-09-10T18:39:06.393426+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: audit 2024-09-10T18:39:06.403012+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: audit 2024-09-10T18:39:06.406849+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: audit 2024-09-10T18:39:06.423401+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:39:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: audit 2024-09-10T18:39:06.431807+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:39:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:07 smithi083 bash[18629]: cluster 2024-09-10T18:39:07.348378+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.01186s) 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.103+0000 7f24eb59e640 1 Processor -- start 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.103+0000 7f24eb59e640 1 -- start start 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f24ec1035f0 con 0x7f24ec102ad0 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24ea59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24ec102ad0 0x7f24ec102ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39524/0 (socket says 172.21.15.83:39524) 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24ea59c640 1 -- 172.21.15.83:0/2863494344 learned_addr learned my addr 172.21.15.83:0/2863494344 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:07.982 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553240667 0 0) 0x7f24ec1035f0 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d80034e0 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2576423411 0 0) 0x7f24d80034e0 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec107fe0 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f24dc001590 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f24dc001bf0 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24e9d9b640 1 -- 172.21.15.83:0/2863494344 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f24dc005600 con 0x7f24ec102ad0 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- 172.21.15.83:0/2863494344 >> v1:172.21.15.83:6789/0 conn(0x7f24ec102ad0 legacy=0x7f24ec102ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- 172.21.15.83:0/2863494344 shutdown_connections 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- 172.21.15.83:0/2863494344 >> 172.21.15.83:0/2863494344 conn(0x7f24ec0760b0 msgr2=0x7f24ec0764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- 172.21.15.83:0/2863494344 shutdown_connections 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 -- 172.21.15.83:0/2863494344 wait complete. 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 librados: starting msgr at 2024-09-10T18:39:07.983 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.107+0000 7f24eb59e640 1 librados: starting objecter 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 Processor -- start 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 -- start start 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 librados: setting wanted keys 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 librados: calling monclient init 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f24ec111bf0 con 0x7f24ec102ad0 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24ea59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24ec102ad0 0x7f24ec1114e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39536/0 (socket says 172.21.15.83:39536) 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24ea59c640 1 -- 172.21.15.83:0/3963658006 learned_addr learned my addr 172.21.15.83:0/3963658006 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 10 client.?.objecter ms_handle_connect 0x7f24ec102ad0 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825599835 0 0) 0x7f24ec111bf0 con 0x7f24ec102ad0 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d80056b0 con 0x7f24ec102ad0 2024-09-10T18:39:07.984 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657869951 0 0) 0x7f24d80056b0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24ec111bf0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f24dc000ed0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4051655080 0 0) 0x7f24ec111bf0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec111d80 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec112050 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec115c60 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24eb59e640 1 librados: init done 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f24dc001670 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f24dc0060c0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 5) v1 ==== 45457+0+0 (unknown 1231082550 0 0) 0x7f24dc0113c0 con 0x7f24ec102ad0 2024-09-10T18:39:07.985 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6800/3275901971 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24dc000ed0 con 0x7f24d803c060 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 1901766985 0 0) 0x7f24dc047bb0 con 0x7f24ec102ad0 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 10 client.14122.objecter ms_dispatch 0x7f24ec000b90 osd_map(1..1 src has 1..1) v4 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e2ffd640 20 client.14122.objecter dump_active .. 0 homeless 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.111+0000 7f24e3fff640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6800/3275901971 conn(0x7f24d803c060 legacy=0x7f24d803e510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/3275901971 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.315+0000 7f24e3fff640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6800/3275901971 conn(0x7f24d803c060 legacy=0x7f24d803e510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/3275901971 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:05.719+0000 7f24e3fff640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6800/3275901971 conn(0x7f24d803c060 legacy=0x7f24d803e510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/3275901971 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:06.335+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mgrmap(e 6) v1 ==== 45259+0+0 (unknown 935769217 0 0) 0x7f24dc046ca0 con 0x7f24ec102ad0 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:06.335+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6800/3275901971 conn(0x7f24d803c060 legacy=0x7f24d803e510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.339+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 7) v1 ==== 45351+0+0 (unknown 136603975 0 0) 0x7f24dc046f50 con 0x7f24ec102ad0 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.339+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6800/2452184936 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24dc000ed0 con 0x7f24d8040980 2024-09-10T18:39:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.339+0000 7f24e2ffd640 10 client.14122.objecter ms_handle_connect 0x7f24d8040980 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.343+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (unknown 0 0 4193150809) 0x7f24dc000ed0 con 0x7f24d8040980 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 --> v1:172.21.15.83:6800/2452184936 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f24cc002560 con 0x7f24d8040980 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24e2ffd640 1 -- 172.21.15.83:0/3963658006 <== mgr.14118 v1:172.21.15.83:6800/2452184936 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 96372106) 0x7f24cc002560 con 0x7f24d8040980 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 10 librados: watch_flush enter 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 10 librados: watch_flush exit 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 20 client.14122.objecter shutdown clearing up homeless session... 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 10 client.14122.objecter successfully canceled tick 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6800/2452184936 conn(0x7f24d8040980 legacy=0x7f24d8042d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 >> v1:172.21.15.83:6789/0 conn(0x7f24ec102ad0 legacy=0x7f24ec1114e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 shutdown_connections 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.355+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 >> 172.21.15.83:0/3963658006 conn(0x7f24ec0760b0 msgr2=0x7f24ec108420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.359+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 shutdown_connections 2024-09-10T18:39:07.987 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.359+0000 7f24eb59e640 1 -- 172.21.15.83:0/3963658006 wait complete. 2024-09-10T18:39:07.988 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:07.359+0000 7f24eb59e640 1 librados: shutdown 2024-09-10T18:39:07.988 INFO:teuthology.orchestra.run.smithi083.stdout:mgr epoch 5 is available 2024-09-10T18:39:07.988 INFO:teuthology.orchestra.run.smithi083.stdout:Setting orchestrator backend to cephadm... 2024-09-10T18:39:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18629]: audit 2024-09-10T18:39:07.346283+0000 mgr.y (mgr.14118) 2 : audit [DBG] from='client.14122 v1:172.21.15.83:0/3963658006' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T18:39:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18629]: audit 2024-09-10T18:39:07.361354+0000 mgr.y (mgr.14118) 3 : audit [DBG] from='client.14122 v1:172.21.15.83:0/3963658006' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T18:39:09.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: debug 2024-09-10T18:39:08.903+0000 7f9e30a02640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:39:08] ENGINE Error in HTTPServer.serve 2024-09-10T18:39:09.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: Traceback (most recent call last): 2024-09-10T18:39:09.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: self._connections.run(self.expiration_interval) 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: self._run(expiration_interval) 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: s = self.context.wrap_socket( 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: return self.sslsocket_class._create( 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: self.do_handshake() 2024-09-10T18:39:09.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:39:09.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: self._sslobj.do_handshake() 2024-09-10T18:39:09.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:08 smithi083 bash[18896]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: cluster 2024-09-10T18:39:08.409074+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: cephadm 2024-09-10T18:39:08.779414+0000 mgr.y (mgr.14118) 4 : cephadm [INF] [10/Sep/2024:18:39:08] ENGINE Bus STARTING 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: cephadm 2024-09-10T18:39:08.903575+0000 mgr.y (mgr.14118) 5 : cephadm [INF] [10/Sep/2024:18:39:08] ENGINE Serving on https://172.21.15.83:7150 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: cephadm 2024-09-10T18:39:08.903740+0000 mgr.y (mgr.14118) 6 : cephadm [INF] [10/Sep/2024:18:39:08] ENGINE Bus STARTED 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: cephadm 2024-09-10T18:39:08.907221+0000 mgr.y (mgr.14118) 7 : cephadm [ERR] [10/Sep/2024:18:39:08] ENGINE Error in HTTPServer.serve 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: Traceback (most recent call last): 2024-09-10T18:39:09.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: self._connections.run(self.expiration_interval) 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: self._run(expiration_interval) 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: s = self.context.wrap_socket( 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: return self.sslsocket_class._create( 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:39:09.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: self.do_handshake() 2024-09-10T18:39:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:39:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: self._sslobj.do_handshake() 2024-09-10T18:39:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:39:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: audit 2024-09-10T18:39:08.909066+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:09 smithi083 bash[18629]: audit 2024-09-10T18:39:08.910524+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 Processor -- start 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- start start 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa73c079c70 con 0x7fa73c078df0 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa73c078df0 0x7fa73c079200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39602/0 (socket says 172.21.15.83:39602) 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73a59c640 1 -- 172.21.15.83:0/3620556153 learned_addr learned my addr 172.21.15.83:0/3620556153 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202533269 0 0) 0x7fa73c079c70 con 0x7fa73c078df0 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7240034e0 con 0x7fa73c078df0 2024-09-10T18:39:10.184 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3818817321 0 0) 0x7fa7240034e0 con 0x7fa73c078df0 2024-09-10T18:39:10.185 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c07ae10 con 0x7fa73c078df0 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fa72c001590 con 0x7fa73c078df0 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fa72c001bf0 con 0x7fa73c078df0 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa73959a640 1 -- 172.21.15.83:0/3620556153 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fa72c005600 con 0x7fa73c078df0 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- 172.21.15.83:0/3620556153 >> v1:172.21.15.83:6789/0 conn(0x7fa73c078df0 legacy=0x7fa73c079200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- 172.21.15.83:0/3620556153 shutdown_connections 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- 172.21.15.83:0/3620556153 >> 172.21.15.83:0/3620556153 conn(0x7fa73c076320 msgr2=0x7fa73c076740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:10.189 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- 172.21.15.83:0/3620556153 shutdown_connections 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.155+0000 7fa740a95640 1 -- 172.21.15.83:0/3620556153 wait complete. 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 librados: starting msgr at 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 librados: starting objecter 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 Processor -- start 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 -- start start 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 librados: setting wanted keys 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 librados: calling monclient init 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa73c116250 con 0x7fa73c078df0 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa73a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa73c078df0 0x7fa73c115b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39616/0 (socket says 172.21.15.83:39616) 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa73a59c640 1 -- 172.21.15.83:0/2102693775 learned_addr learned my addr 172.21.15.83:0/2102693775 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 10 client.?.objecter ms_handle_connect 0x7fa73c078df0 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:10.190 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233375697 0 0) 0x7fa73c116250 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7240056b0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3737115009 0 0) 0x7fa7240056b0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa73c116250 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fa72c000ed0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2777084906 0 0) 0x7fa73c116250 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73c1163e0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73c1166b0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73c11a270 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa740a95640 1 librados: init done 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fa72c004bf0 con 0x7fa73c078df0 2024-09-10T18:39:10.191 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.159+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fa72c0060f0 con 0x7fa73c078df0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7fa72c007380 con 0x7fa73c078df0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71c005040 con 0x7fa73c078df0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7fa72c0486b0 con 0x7fa73c078df0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 10 client.14130.objecter ms_dispatch 0x7fa73c000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 20 client.14130.objecter dump_active .. 0 homeless 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.163+0000 7fa72affd640 10 client.14130.objecter ms_handle_connect 0x7fa72403c110 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.171+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa72c007ba0 con 0x7fa73c078df0 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.519+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa71c000bc0 con 0x7fa72403c110 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.539+0000 7fa72affd640 1 -- 172.21.15.83:0/2102693775 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7fa71c000bc0 con 0x7fa72403c110 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 10 librados: watch_flush enter 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 10 librados: watch_flush exit 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 20 client.14130.objecter shutdown clearing up homeless session... 2024-09-10T18:39:10.192 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 10 client.14130.objecter successfully canceled tick 2024-09-10T18:39:10.193 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 >> v1:172.21.15.83:6800/2452184936 conn(0x7fa72403c110 legacy=0x7fa72403e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:10.193 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 >> v1:172.21.15.83:6789/0 conn(0x7fa73c078df0 legacy=0x7fa73c115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:10.194 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 shutdown_connections 2024-09-10T18:39:10.194 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 >> 172.21.15.83:0/2102693775 conn(0x7fa73c076320 msgr2=0x7fa73c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:10.194 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 shutdown_connections 2024-09-10T18:39:10.194 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 -- 172.21.15.83:0/2102693775 wait complete. 2024-09-10T18:39:10.194 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:09.543+0000 7fa740a95640 1 librados: shutdown 2024-09-10T18:39:10.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:10 smithi083 bash[18629]: audit 2024-09-10T18:39:09.527237+0000 mgr.y (mgr.14118) 8 : audit [DBG] from='client.14130 v1:172.21.15.83:0/2102693775' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:10.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:10 smithi083 bash[18629]: audit 2024-09-10T18:39:09.538294+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:10.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:10 smithi083 bash[18629]: audit 2024-09-10T18:39:09.558098+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:12.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:12 smithi083 bash[18629]: audit 2024-09-10T18:39:11.724615+0000 mgr.y (mgr.14118) 9 : audit [DBG] from='client.14132 v1:172.21.15.83:0/4258925668' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:12.307 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-10T18:39:12.307 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc34f056640 1 Processor -- start 2024-09-10T18:39:12.307 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc34f056640 1 -- start start 2024-09-10T18:39:12.314 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc34f056640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3480759b0 con 0x7fc348076e70 2024-09-10T18:39:12.314 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc34cdf2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc348076e70 0x7fc3480752a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39632/0 (socket says 172.21.15.83:39632) 2024-09-10T18:39:12.314 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc34cdf2640 1 -- 172.21.15.83:0/2077961072 learned_addr learned my addr 172.21.15.83:0/2077961072 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:12.314 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493436991 0 0) 0x7fc3480759b0 con 0x7fc348076e70 2024-09-10T18:39:12.314 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3300034e0 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2442786040 0 0) 0x7fc3300034e0 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348075b40 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fc33c001590 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fc33c001bf0 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.375+0000 7fc3477fe640 1 -- 172.21.15.83:0/2077961072 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fc33c005600 con 0x7fc348076e70 2024-09-10T18:39:12.315 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- 172.21.15.83:0/2077961072 >> v1:172.21.15.83:6789/0 conn(0x7fc348076e70 legacy=0x7fc3480752a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- 172.21.15.83:0/2077961072 shutdown_connections 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- 172.21.15.83:0/2077961072 >> 172.21.15.83:0/2077961072 conn(0x7fc3480fc2e0 msgr2=0x7fc3480fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- 172.21.15.83:0/2077961072 shutdown_connections 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- 172.21.15.83:0/2077961072 wait complete. 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 librados: starting msgr at 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 librados: starting objecter 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 Processor -- start 2024-09-10T18:39:12.316 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- start start 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 librados: setting wanted keys 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 librados: calling monclient init 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34f056640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc348111e40 con 0x7fc348076e70 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34cdf2640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc348076e70 0x7fc348111730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39634/0 (socket says 172.21.15.83:39634) 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc34cdf2640 1 -- 172.21.15.83:0/4258925668 learned_addr learned my addr 172.21.15.83:0/4258925668 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:12.317 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc3457fa640 10 client.?.objecter ms_handle_connect 0x7fc348076e70 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc3457fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194670455 0 0) 0x7fc348111e40 con 0x7fc348076e70 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3300056b0 con 0x7fc348076e70 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.379+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509169018 0 0) 0x7fc3300056b0 con 0x7fc348076e70 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc348111e40 con 0x7fc348076e70 2024-09-10T18:39:12.318 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fc33c000ed0 con 0x7fc348076e70 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1123288640 0 0) 0x7fc348111e40 con 0x7fc348076e70 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc348111fd0 con 0x7fc348076e70 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3481122a0 con 0x7fc348076e70 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc34f056640 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc348115eb0 con 0x7fc348076e70 2024-09-10T18:39:12.319 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc34f056640 1 librados: init done 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fc33c004bf0 con 0x7fc348076e70 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fc33c005fb0 con 0x7fc348076e70 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc334005040 con 0x7fc348076e70 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7fc33c0112d0 con 0x7fc348076e70 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7fc33c047940 con 0x7fc348076e70 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 10 client.14132.objecter ms_dispatch 0x7fc348000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:12.320 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 20 client.14132.objecter dump_active .. 0 homeless 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.383+0000 7fc3457fa640 10 client.14132.objecter ms_handle_connect 0x7fc33003c0c0 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.391+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc33c012ab0 con 0x7fc348076e70 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.719+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc334000bc0 con 0x7fc33003c0c0 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.719+0000 7fc3457fa640 1 -- 172.21.15.83:0/4258925668 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7fc334000bc0 con 0x7fc33003c0c0 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 10 librados: watch_flush enter 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 10 librados: watch_flush exit 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 20 client.14132.objecter shutdown clearing up homeless session... 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 10 client.14132.objecter successfully canceled tick 2024-09-10T18:39:12.321 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 >> v1:172.21.15.83:6800/2452184936 conn(0x7fc33003c0c0 legacy=0x7fc33003e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 >> v1:172.21.15.83:6789/0 conn(0x7fc348076e70 legacy=0x7fc348111730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 shutdown_connections 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 >> 172.21.15.83:0/4258925668 conn(0x7fc3480fc2e0 msgr2=0x7fc348105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 shutdown_connections 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 -- 172.21.15.83:0/4258925668 wait complete. 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:11.723+0000 7fc34f056640 1 librados: shutdown 2024-09-10T18:39:12.322 INFO:teuthology.orchestra.run.smithi083.stdout:Generating ssh key... 2024-09-10T18:39:14.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: Generating public/private rsa key pair. 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: Your identification has been saved in /tmp/tmpkjbkepso/key 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: Your public key has been saved in /tmp/tmpkjbkepso/key.pub 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: The key fingerprint is: 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: SHA256:mCkHcqbG+894zJ3cwbCXhGYuUyPchpIZjalsYf5Xed8 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: The key's randomart image is: 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: +---[RSA 3072]----+ 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | + | 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | o + . | 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | +.o+= o o | 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | .==+.++% o | 2024-09-10T18:39:14.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | .+...=OSB o . | 2024-09-10T18:39:14.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | . ..o+ o = . E | 2024-09-10T18:39:14.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | . + = + . | 2024-09-10T18:39:14.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | . o+ + . | 2024-09-10T18:39:14.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: | ooo | 2024-09-10T18:39:14.292 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:13 smithi083 bash[18896]: +----[SHA256]-----+ 2024-09-10T18:39:14.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa6690f640 1 Processor -- start 2024-09-10T18:39:14.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa6690f640 1 -- start start 2024-09-10T18:39:14.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa6690f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa60107110 con 0x7ffa601042f0 2024-09-10T18:39:14.557 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa5ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffa601042f0 0x7ffa60104700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32976/0 (socket says 172.21.15.83:32976) 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa5ffff640 1 -- 172.21.15.83:0/4244654232 learned_addr learned my addr 172.21.15.83:0/4244654232 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384136832 0 0) 0x7ffa60107110 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.411+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c0034e0 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491242391 0 0) 0x7ffa4c0034e0 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa601082b0 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7ffa50001590 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7ffa50001cc0 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5effd640 1 -- 172.21.15.83:0/4244654232 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7ffa500056c0 con 0x7ffa601042f0 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- 172.21.15.83:0/4244654232 >> v1:172.21.15.83:6789/0 conn(0x7ffa601042f0 legacy=0x7ffa60104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- 172.21.15.83:0/4244654232 shutdown_connections 2024-09-10T18:39:14.558 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- 172.21.15.83:0/4244654232 >> 172.21.15.83:0/4244654232 conn(0x7ffa60076350 msgr2=0x7ffa60076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- 172.21.15.83:0/4244654232 shutdown_connections 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- 172.21.15.83:0/4244654232 wait complete. 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 librados: starting msgr at 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 librados: starting objecter 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 Processor -- start 2024-09-10T18:39:14.564 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- start start 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 librados: setting wanted keys 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 librados: calling monclient init 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa6690f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa601121b0 con 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffa60111690 0x7ffa60111aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32978/0 (socket says 172.21.15.83:32978) 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5ffff640 1 -- 172.21.15.83:0/2717023754 learned_addr learned my addr 172.21.15.83:0/2717023754 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5cff9640 10 client.?.objecter ms_handle_connect 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392780438 0 0) 0x7ffa601121b0 con 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.415+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa4c005630 con 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408510019 0 0) 0x7ffa4c005630 con 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa601121b0 con 0x7ffa60111690 2024-09-10T18:39:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7ffa500047a0 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 616671768 0 0) 0x7ffa601121b0 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60116be0 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa60115bd0 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa6690f640 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa601160a0 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa6690f640 1 librados: init done 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7ffa50001160 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7ffa50005f70 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7ffa50011270 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa40005040 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7ffa50047880 con 0x7ffa60111690 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 10 client.14134.objecter ms_dispatch 0x7ffa60000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:14.566 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 20 client.14134.objecter dump_active .. 0 homeless 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.419+0000 7ffa5cff9640 10 client.14134.objecter ms_handle_connect 0x7ffa4c03c120 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.423+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ffa50012c30 con 0x7ffa60111690 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.751+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa40000bc0 con 0x7ffa4c03c120 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.967+0000 7ffa5cff9640 1 -- 172.21.15.83:0/2717023754 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7ffa40000bc0 con 0x7ffa4c03c120 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 10 librados: watch_flush enter 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 10 librados: watch_flush exit 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 20 client.14134.objecter shutdown clearing up homeless session... 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 10 client.14134.objecter successfully canceled tick 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 >> v1:172.21.15.83:6800/2452184936 conn(0x7ffa4c03c120 legacy=0x7ffa4c03e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 >> v1:172.21.15.83:6789/0 conn(0x7ffa60111690 legacy=0x7ffa60111aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:14.567 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 shutdown_connections 2024-09-10T18:39:14.568 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 >> 172.21.15.83:0/2717023754 conn(0x7ffa60076350 msgr2=0x7ffa60100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:14.568 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 shutdown_connections 2024-09-10T18:39:14.568 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 -- 172.21.15.83:0/2717023754 wait complete. 2024-09-10T18:39:14.568 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:13.971+0000 7ffa6690f640 1 librados: shutdown 2024-09-10T18:39:15.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:14 smithi083 bash[18629]: audit 2024-09-10T18:39:13.755333+0000 mgr.y (mgr.14118) 10 : audit [DBG] from='client.14134 v1:172.21.15.83:0/2717023754' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:15.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:14 smithi083 bash[18629]: cephadm 2024-09-10T18:39:13.755761+0000 mgr.y (mgr.14118) 11 : cephadm [INF] Generating ssh key... 2024-09-10T18:39:15.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:14 smithi083 bash[18629]: audit 2024-09-10T18:39:13.963258+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:15.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:14 smithi083 bash[18629]: audit 2024-09-10T18:39:13.968731+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfjy+Sz0+yVv8FRjRmCspdqDVuL5Xuy2woPtsqQemUIDlItq20uPM02HNIxTRblrPtHTJXgmlgnSwEMYAL0Rq16I+yZZElIB+vcChQEmGXkuHdjLJ0GT8nYxsy1t94tQauT5JB4axkHsHfA1r6gM4C/OZDz1fcV+EU6sbDsBHGY43/0nzNMgTSyQ26hjr3pBhvp59anKNdbu67iOc2/L9Pe6bIeK6l+k2/eWsATm77whFsVZPZCou+g07DbZSe+b+e2F6qQtQTf9Iu7dLuIrMABhOXwqMDxTvPHfkV4SLf7tcLAZ5JDr/xbJbS0JiGT1tvJ8VymG/6heiJ95s24LL/nXXd/eEuKSaKyAmQdgsqgdRt1xNtuRFHgKVmGcjRCWFmRokuEtHi7vDe21oWABuvov/aJJzcnX4xzkxzcIlgPTPhvb9MGZwaiNms/i6X0EOBRW8SUzZlbnBZ1aQz09/fy+UXdITP7z+Ksj6bdgTzcOOduL7InTBCdzIulgjjaCM= ceph-707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.739+0000 7f4f97515640 1 Processor -- start 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.739+0000 7f4f97515640 1 -- start start 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.739+0000 7f4f97515640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f90104f80 con 0x7f4f90102160 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.739+0000 7f4f952b1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4f90102160 0x7f4f90102570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32984/0 (socket says 172.21.15.83:32984) 2024-09-10T18:39:16.681 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.739+0000 7f4f952b1640 1 -- 172.21.15.83:0/2851650435 learned_addr learned my addr 172.21.15.83:0/2851650435 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847617780 0 0) 0x7f4f90104f80 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f780034e0 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3269019619 0 0) 0x7f4f780034e0 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90106120 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4f80001590 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f4f80001cc0 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f87fff640 1 -- 172.21.15.83:0/2851650435 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4f800056c0 con 0x7f4f90102160 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 -- 172.21.15.83:0/2851650435 >> v1:172.21.15.83:6789/0 conn(0x7f4f90102160 legacy=0x7f4f90102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:16.687 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 -- 172.21.15.83:0/2851650435 shutdown_connections 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 -- 172.21.15.83:0/2851650435 >> 172.21.15.83:0/2851650435 conn(0x7f4f900fc240 msgr2=0x7f4f900fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 -- 172.21.15.83:0/2851650435 shutdown_connections 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 -- 172.21.15.83:0/2851650435 wait complete. 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 librados: starting msgr at 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 librados: starting objecter 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.743+0000 7f4f97515640 1 Processor -- start 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 -- start start 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 librados: setting wanted keys 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 librados: calling monclient init 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f90076c50 con 0x7f4f90102160 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f952b1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4f90102160 0x7f4f90075220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:32992/0 (socket says 172.21.15.83:32992) 2024-09-10T18:39:16.688 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f952b1640 1 -- 172.21.15.83:0/2608271391 learned_addr learned my addr 172.21.15.83:0/2608271391 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 10 client.?.objecter ms_handle_connect 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101138104 0 0) 0x7f4f90076c50 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78005630 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680394767 0 0) 0x7f4f78005630 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f90076c50 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4f800047a0 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409964978 0 0) 0x7f4f90076c50 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f90076de0 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f90077090 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f90075b50 con 0x7f4f90102160 2024-09-10T18:39:16.689 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 librados: init done 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f4f800057f0 con 0x7f4f90102160 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9005ed70 con 0x7f4f90102160 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.747+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4f80006060 con 0x7f4f90102160 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7f4f800114a0 con 0x7f4f90102160 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f4f80048860 con 0x7f4f90102160 2024-09-10T18:39:16.690 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 10 client.14136.objecter ms_dispatch 0x7f4f90000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 20 client.14136.objecter dump_active .. 0 homeless 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.751+0000 7f4f85ffb640 10 client.14136.objecter ms_handle_connect 0x7f4f7803c0a0 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:15.759+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4f80012ef0 con 0x7f4f90102160 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.087+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f90075e40 con 0x7f4f7803c0a0 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.087+0000 7f4f85ffb640 1 -- 172.21.15.83:0/2608271391 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 374764277) 0x7f4f90075e40 con 0x7f4f7803c0a0 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 10 librados: watch_flush enter 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 10 librados: watch_flush exit 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 20 client.14136.objecter shutdown clearing up homeless session... 2024-09-10T18:39:16.691 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 10 client.14136.objecter successfully canceled tick 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 >> v1:172.21.15.83:6800/2452184936 conn(0x7f4f7803c0a0 legacy=0x7f4f7803e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 >> v1:172.21.15.83:6789/0 conn(0x7f4f90102160 legacy=0x7f4f90075220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 shutdown_connections 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 >> 172.21.15.83:0/2608271391 conn(0x7f4f900fc240 msgr2=0x7f4f901063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 shutdown_connections 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 -- 172.21.15.83:0/2608271391 wait complete. 2024-09-10T18:39:16.692 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:16.091+0000 7f4f97515640 1 librados: shutdown 2024-09-10T18:39:16.693 INFO:teuthology.orchestra.run.smithi083.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-10T18:39:16.693 INFO:teuthology.orchestra.run.smithi083.stdout:Adding key to root@localhost authorized_keys... 2024-09-10T18:39:16.693 INFO:teuthology.orchestra.run.smithi083.stdout:Adding host smithi083... 2024-09-10T18:39:17.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:17 smithi083 bash[18629]: audit 2024-09-10T18:39:16.091362+0000 mgr.y (mgr.14118) 12 : audit [DBG] from='client.14136 v1:172.21.15.83:0/2608271391' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:19.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:19 smithi083 bash[18629]: audit 2024-09-10T18:39:18.227772+0000 mgr.y (mgr.14118) 13 : audit [DBG] from='client.14138 v1:172.21.15.83:0/211132576' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:20.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:20 smithi083 bash[18629]: cephadm 2024-09-10T18:39:19.480576+0000 mgr.y (mgr.14118) 14 : cephadm [INF] Deploying cephadm binary to smithi083 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Added host 'smithi083' with addr '172.21.15.83' 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe09961640 1 Processor -- start 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe09961640 1 -- start start 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe09961640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe04104fa0 con 0x7fbe04102180 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe02ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbe04102180 0x7fbe04102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33002/0 (socket says 172.21.15.83:33002) 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe02ffd640 1 -- 172.21.15.83:0/3069811202 learned_addr learned my addr 172.21.15.83:0/3069811202 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:22.911 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612799169 0 0) 0x7fbe04104fa0 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf00034e0 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119234489 0 0) 0x7fbdf00034e0 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04106140 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbdf4001590 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fbdf4001bf0 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.871+0000 7fbe027fc640 1 -- 172.21.15.83:0/3069811202 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbdf40055f0 con 0x7fbe04102180 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/3069811202 >> v1:172.21.15.83:6789/0 conn(0x7fbe04102180 legacy=0x7fbe04102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/3069811202 shutdown_connections 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/3069811202 >> 172.21.15.83:0/3069811202 conn(0x7fbe040fc240 msgr2=0x7fbe040fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/3069811202 shutdown_connections 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/3069811202 wait complete. 2024-09-10T18:39:22.912 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 librados: starting msgr at 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 librados: starting objecter 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 Processor -- start 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- start start 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 librados: setting wanted keys 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 librados: calling monclient init 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe04111e10 con 0x7fbe04102180 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe02ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbe04102180 0x7fbe04111700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33006/0 (socket says 172.21.15.83:33006) 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe02ffd640 1 -- 172.21.15.83:0/211132576 learned_addr learned my addr 172.21.15.83:0/211132576 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 10 client.?.objecter ms_handle_connect 0x7fbe04102180 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431876692 0 0) 0x7fbe04111e10 con 0x7fbe04102180 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf0005680 con 0x7fbe04102180 2024-09-10T18:39:22.913 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501630277 0 0) 0x7fbdf0005680 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe04111e10 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbdf4001770 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4105491479 0 0) 0x7fbe04111e10 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04111fa0 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe04112270 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.875+0000 7fbe09961640 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe04115e80 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbe09961640 1 librados: init done 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7fbdf40056e0 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbdf40060d0 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7fbdf40113d0 con 0x7fbe04102180 2024-09-10T18:39:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde4005040 con 0x7fbe04102180 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7fbdf4048750 con 0x7fbe04102180 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 10 client.14138.objecter ms_dispatch 0x7fbe04000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 20 client.14138.objecter dump_active .. 0 homeless 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.879+0000 7fbdfbfff640 10 client.14138.objecter ms_handle_connect 0x7fbdf003bfb0 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:17.887+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fbdf4012b70 con 0x7fbe04102180 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:18.223+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}) v1 -- 0x7fbde4000bc0 con 0x7fbdf003bfb0 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.051+0000 7fbdfbfff640 1 -- 172.21.15.83:0/211132576 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (unknown 0 0 3934659554) 0x7fbde4000bc0 con 0x7fbdf003bfb0 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 10 librados: watch_flush enter 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 10 librados: watch_flush exit 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 20 client.14138.objecter shutdown clearing up homeless session... 2024-09-10T18:39:22.915 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 10 client.14138.objecter successfully canceled tick 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 >> v1:172.21.15.83:6800/2452184936 conn(0x7fbdf003bfb0 legacy=0x7fbdf003e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 >> v1:172.21.15.83:6789/0 conn(0x7fbe04102180 legacy=0x7fbe04111700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 shutdown_connections 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.055+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 >> 172.21.15.83:0/211132576 conn(0x7fbe040fc240 msgr2=0x7fbe040fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.059+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 shutdown_connections 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.059+0000 7fbe09961640 1 -- 172.21.15.83:0/211132576 wait complete. 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:22.059+0000 7fbe09961640 1 librados: shutdown 2024-09-10T18:39:22.916 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying unmanaged mon service... 2024-09-10T18:39:23.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:23 smithi083 bash[18629]: audit 2024-09-10T18:39:22.055388+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:23.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:23 smithi083 bash[18629]: cephadm 2024-09-10T18:39:22.055895+0000 mgr.y (mgr.14118) 15 : cephadm [INF] Added host smithi083 2024-09-10T18:39:23.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:23 smithi083 bash[18629]: audit 2024-09-10T18:39:22.056734+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:25.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:25 smithi083 bash[18629]: audit 2024-09-10T18:39:25.045671+0000 mgr.y (mgr.14118) 16 : audit [DBG] from='client.14140 v1:172.21.15.83:0/2593501106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:25.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:25 smithi083 bash[18629]: audit 2024-09-10T18:39:25.053897+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:25.933 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e437b640 1 Processor -- start 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e437b640 1 -- start start 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e437b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42dc100460 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e2117640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f42dc0ff940 0x7f42dc0ffd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45244/0 (socket says 172.21.15.83:45244) 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e2117640 1 -- 172.21.15.83:0/3064800682 learned_addr learned my addr 172.21.15.83:0/3064800682 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489607315 0 0) 0x7f42dc100460 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c80034e0 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195751762 0 0) 0x7f42c80034e0 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc104e50 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.659+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f42d8001590 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f42d8001bf0 con 0x7f42dc0ff940 2024-09-10T18:39:25.934 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e1115640 1 -- 172.21.15.83:0/3064800682 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f42d8005600 con 0x7f42dc0ff940 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/3064800682 >> v1:172.21.15.83:6789/0 conn(0x7f42dc0ff940 legacy=0x7f42dc0ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/3064800682 shutdown_connections 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/3064800682 >> 172.21.15.83:0/3064800682 conn(0x7f42dc0fb140 msgr2=0x7f42dc0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/3064800682 shutdown_connections 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/3064800682 wait complete. 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 librados: starting msgr at 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 librados: starting objecter 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 Processor -- start 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- start start 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 librados: setting wanted keys 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 librados: calling monclient init 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42dc074b30 con 0x7f42dc0ff940 2024-09-10T18:39:25.935 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e2117640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f42dc0ff940 0x7f42dc112210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45250/0 (socket says 172.21.15.83:45250) 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e2117640 1 -- 172.21.15.83:0/2593501106 learned_addr learned my addr 172.21.15.83:0/2593501106 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 10 client.?.objecter ms_handle_connect 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372771822 0 0) 0x7f42dc074b30 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c80056b0 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666391986 0 0) 0x7f42c80056b0 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42dc074b30 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f42d8000ed0 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2906043682 0 0) 0x7f42dc074b30 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc074cc0 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc074f70 con 0x7f42dc0ff940 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:25.936 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc113bb0 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.663+0000 7f42e437b640 1 librados: init done 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 976+0+0 (unknown 4014920612 0 0) 0x7f42d8004b10 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f42d8005f50 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7f42d8011250 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f42d8048620 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 10 client.14140.objecter ms_dispatch 0x7f42dc000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0005040 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 20 client.14140.objecter dump_active .. 0 homeless 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.667+0000 7f42d2ffd640 10 client.14140.objecter ms_handle_connect 0x7f42c803c160 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:24.675+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f42d8012a40 con 0x7f42dc0ff940 2024-09-10T18:39:25.937 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.039+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f42c0000bc0 con 0x7f42c803c160 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42d2ffd640 1 -- 172.21.15.83:0/2593501106 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f42c0000bc0 con 0x7f42c803c160 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 10 librados: watch_flush enter 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 10 librados: watch_flush exit 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 20 client.14140.objecter shutdown clearing up homeless session... 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 10 client.14140.objecter successfully canceled tick 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 >> v1:172.21.15.83:6800/2452184936 conn(0x7f42c803c160 legacy=0x7f42c803e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 >> v1:172.21.15.83:6789/0 conn(0x7f42dc0ff940 legacy=0x7f42dc112210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 shutdown_connections 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 >> 172.21.15.83:0/2593501106 conn(0x7f42dc0fb140 msgr2=0x7f42dc105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 shutdown_connections 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.051+0000 7f42e437b640 1 -- 172.21.15.83:0/2593501106 wait complete. 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:25.055+0000 7f42e437b640 1 librados: shutdown 2024-09-10T18:39:25.938 INFO:teuthology.orchestra.run.smithi083.stdout:Deploying unmanaged mgr service... 2024-09-10T18:39:26.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:26 smithi083 bash[18629]: cephadm 2024-09-10T18:39:25.048984+0000 mgr.y (mgr.14118) 17 : cephadm [INF] Saving service mon spec with placement count:5 2024-09-10T18:39:26.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:26 smithi083 bash[18629]: audit 2024-09-10T18:39:25.555204+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:26.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:26 smithi083 bash[18629]: audit 2024-09-10T18:39:25.869238+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:27.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:27 smithi083 bash[18629]: cluster 2024-09-10T18:39:26.350325+0000 mgr.y (mgr.14118) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:39:27.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:27 smithi083 bash[18629]: audit 2024-09-10T18:39:27.488900+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f11393f1640 1 Processor -- start 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f11393f1640 1 -- start start 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f11393f1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1134103eb0 con 0x7f1134101090 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1132ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1134101090 0x7f11341014a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45252/0 (socket says 172.21.15.83:45252) 2024-09-10T18:39:28.120 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1132ffd640 1 -- 172.21.15.83:0/912840762 learned_addr learned my addr 172.21.15.83:0/912840762 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708662747 0 0) 0x7f1134103eb0 con 0x7f1134101090 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111c0034e0 con 0x7f1134101090 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3688735467 0 0) 0x7f111c0034e0 con 0x7f1134101090 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1134105050 con 0x7f1134101090 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.115+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f1124001590 con 0x7f1134101090 2024-09-10T18:39:28.121 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1124001ce0 con 0x7f1134101090 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f1131ffb640 1 -- 172.21.15.83:0/912840762 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f1124005700 con 0x7f1134101090 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- 172.21.15.83:0/912840762 >> v1:172.21.15.83:6789/0 conn(0x7f1134101090 legacy=0x7f11341014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- 172.21.15.83:0/912840762 shutdown_connections 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- 172.21.15.83:0/912840762 >> 172.21.15.83:0/912840762 conn(0x7f11340fb190 msgr2=0x7f11340fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- 172.21.15.83:0/912840762 shutdown_connections 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- 172.21.15.83:0/912840762 wait complete. 2024-09-10T18:39:28.126 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 librados: starting msgr at 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 librados: starting objecter 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 Processor -- start 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- start start 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 librados: setting wanted keys 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 librados: calling monclient init 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f11393f1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f113410fb80 con 0x7f1134101090 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f1132ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1134101090 0x7f113410f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45264/0 (socket says 172.21.15.83:45264) 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.119+0000 7f1132ffd640 1 -- 172.21.15.83:0/2755912466 learned_addr learned my addr 172.21.15.83:0/2755912466 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 10 client.?.objecter ms_handle_connect 0x7f1134101090 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978305648 0 0) 0x7f113410fb80 con 0x7f1134101090 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f111c005630 con 0x7f1134101090 2024-09-10T18:39:28.127 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994870175 0 0) 0x7f111c005630 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113410fb80 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f1124001060 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2107154894 0 0) 0x7f113410fb80 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113410fd10 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113410ffe0 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f11393f1640 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1134113bf0 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f11393f1640 1 librados: init done 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1124004990 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f1124006120 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7f11240062a0 con 0x7f1134101090 2024-09-10T18:39:28.128 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f1124048750 con 0x7f1134101090 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 10 client.14142.objecter ms_dispatch 0x7f1134000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f111b7fe640 20 client.14142.objecter dump_active .. 0 homeless 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.123+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1120005040 con 0x7f1134101090 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.127+0000 7f111b7fe640 10 client.14142.objecter ms_handle_connect 0x7f111c03c1a0 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.131+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1124006b00 con 0x7f1134101090 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.475+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 --> v1:172.21.15.83:6800/2452184936 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1120000bc0 con 0x7f111c03c1a0 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.483+0000 7f111b7fe640 1 -- 172.21.15.83:0/2755912466 <== mgr.14118 v1:172.21.15.83:6800/2452184936 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f1120000bc0 con 0x7f111c03c1a0 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 10 librados: watch_flush enter 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 10 librados: watch_flush exit 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 20 client.14142.objecter shutdown clearing up homeless session... 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 10 client.14142.objecter successfully canceled tick 2024-09-10T18:39:28.129 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 >> v1:172.21.15.83:6800/2452184936 conn(0x7f111c03c1a0 legacy=0x7f111c03e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 >> v1:172.21.15.83:6789/0 conn(0x7f1134101090 legacy=0x7f113410f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 shutdown_connections 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 >> 172.21.15.83:0/2755912466 conn(0x7f11340fb190 msgr2=0x7f1134073ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 shutdown_connections 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 -- 172.21.15.83:0/2755912466 wait complete. 2024-09-10T18:39:28.130 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:27.487+0000 7f11393f1640 1 librados: shutdown 2024-09-10T18:39:29.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:28 smithi083 bash[18629]: audit 2024-09-10T18:39:27.481813+0000 mgr.y (mgr.14118) 19 : audit [DBG] from='client.14142 v1:172.21.15.83:0/2755912466' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:29.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:28 smithi083 bash[18629]: cephadm 2024-09-10T18:39:27.483906+0000 mgr.y (mgr.14118) 20 : cephadm [INF] Saving service mgr spec with placement count:2 2024-09-10T18:39:29.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:29 smithi083 bash[18629]: cluster 2024-09-10T18:39:28.350824+0000 mgr.y (mgr.14118) 21 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:39:30.391 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.395+0000 7f8732b1f640 1 Processor -- start 2024-09-10T18:39:30.391 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.395+0000 7f8732b1f640 1 -- start start 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f872c102cc0 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f87308bb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f872c1021a0 0x7f872c1025b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45268/0 (socket says 172.21.15.83:45268) 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f87308bb640 1 -- 172.21.15.83:0/2592445812 learned_addr learned my addr 172.21.15.83:0/2592445812 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326194954 0 0) 0x7f872c102cc0 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87180034e0 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420623834 0 0) 0x7f87180034e0 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c1060d0 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f871c001590 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f871c001c10 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f872b7fe640 1 -- 172.21.15.83:0/2592445812 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f871c005630 con 0x7f872c1021a0 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- 172.21.15.83:0/2592445812 >> v1:172.21.15.83:6789/0 conn(0x7f872c1021a0 legacy=0x7f872c1025b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:30.392 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- 172.21.15.83:0/2592445812 shutdown_connections 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- 172.21.15.83:0/2592445812 >> 172.21.15.83:0/2592445812 conn(0x7f872c0fa960 msgr2=0x7f872c0fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- 172.21.15.83:0/2592445812 shutdown_connections 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.399+0000 7f8732b1f640 1 -- 172.21.15.83:0/2592445812 wait complete. 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 librados: starting msgr at 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 librados: starting objecter 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 Processor -- start 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 -- start start 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 librados: setting wanted keys 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 librados: calling monclient init 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f872c10fc00 con 0x7f872c1021a0 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87308bb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f872c1021a0 0x7f872c10f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45284/0 (socket says 172.21.15.83:45284) 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87308bb640 1 -- 172.21.15.83:0/3828122768 learned_addr learned my addr 172.21.15.83:0/3828122768 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 10 client.?.objecter ms_handle_connect 0x7f872c1021a0 2024-09-10T18:39:30.393 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162175214 0 0) 0x7f872c10fc00 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87180056b0 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251741784 0 0) 0x7f87180056b0 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f872c10fc00 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f871c0047a0 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3740514517 0 0) 0x7f872c10fc00 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c10fd90 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872c110060 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c113c70 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f8732b1f640 1 librados: init done 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f871c004d30 con 0x7f872c1021a0 2024-09-10T18:39:30.394 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.403+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f871c006040 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7f871c006270 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f871c005960 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 10 client.14144.objecter ms_dispatch 0x7f872c000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870c005040 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 20 client.14144.objecter dump_active .. 0 homeless 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.407+0000 7f87297fa640 10 client.14144.objecter ms_handle_connect 0x7f871803c040 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.411+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f871c007010 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.739+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f870c0052f0 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.743+0000 7f87297fa640 1 -- 172.21.15.83:0/3828122768 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7f871c0165a0 con 0x7f872c1021a0 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 10 librados: watch_flush enter 2024-09-10T18:39:30.395 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 10 librados: watch_flush exit 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 20 client.14144.objecter shutdown clearing up homeless session... 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 10 client.14144.objecter successfully canceled tick 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 >> v1:172.21.15.83:6800/2452184936 conn(0x7f871803c040 legacy=0x7f871803e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 >> v1:172.21.15.83:6789/0 conn(0x7f872c1021a0 legacy=0x7f872c10f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 shutdown_connections 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 >> 172.21.15.83:0/3828122768 conn(0x7f872c0fa960 msgr2=0x7f872c106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 shutdown_connections 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 -- 172.21.15.83:0/3828122768 wait complete. 2024-09-10T18:39:30.396 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:29.747+0000 7f8732b1f640 1 librados: shutdown 2024-09-10T18:39:31.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:30 smithi083 bash[18629]: audit 2024-09-10T18:39:29.749744+0000 mon.a (mon.0) 63 : audit [INF] from='client.? v1:172.21.15.83:0/3828122768' entity='client.admin' 2024-09-10T18:39:32.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:31 smithi083 bash[18629]: cluster 2024-09-10T18:39:30.351287+0000 mgr.y (mgr.14118) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:39:32.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:31 smithi083 bash[18629]: audit 2024-09-10T18:39:31.259525+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:32.644 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1ca6cb640 1 Processor -- start 2024-09-10T18:39:32.644 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1ca6cb640 1 -- start start 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1ca6cb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1c4104fa0 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb1c4102180 0x7fb1c4102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45286/0 (socket says 172.21.15.83:45286) 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c3fff640 1 -- 172.21.15.83:0/3595415013 learned_addr learned my addr 172.21.15.83:0/3595415013 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804187811 0 0) 0x7fb1c4104fa0 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ac0034e0 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3703239572 0 0) 0x7fb1ac0034e0 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4106140 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb1b8001590 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb1b8001ce0 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.603+0000 7fb1c2ffd640 1 -- 172.21.15.83:0/3595415013 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb1b8005700 con 0x7fb1c4102180 2024-09-10T18:39:32.645 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/3595415013 >> v1:172.21.15.83:6789/0 conn(0x7fb1c4102180 legacy=0x7fb1c4102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:32.650 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/3595415013 shutdown_connections 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/3595415013 >> 172.21.15.83:0/3595415013 conn(0x7fb1c40fc2e0 msgr2=0x7fb1c40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/3595415013 shutdown_connections 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/3595415013 wait complete. 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 librados: starting msgr at 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 librados: starting objecter 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 Processor -- start 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- start start 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 librados: setting wanted keys 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 librados: calling monclient init 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1ca6cb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1c4111dd0 con 0x7fb1c4102180 2024-09-10T18:39:32.651 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb1c4102180 0x7fb1c41116c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45294/0 (socket says 172.21.15.83:45294) 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c3fff640 1 -- 172.21.15.83:0/2792380066 learned_addr learned my addr 172.21.15.83:0/2792380066 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 10 client.?.objecter ms_handle_connect 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690608643 0 0) 0x7fb1c4111dd0 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ac005630 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883101102 0 0) 0x7fb1ac005630 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1c4111dd0 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.607+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb1b8001060 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3981158481 0 0) 0x7fb1c4111dd0 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4111f60 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c4112230 con 0x7fb1c4102180 2024-09-10T18:39:32.652 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1ca6cb640 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c4115e40 con 0x7fb1c4102180 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1ca6cb640 1 librados: init done 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb1b8004990 con 0x7fb1c4102180 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb1b8005ff0 con 0x7fb1c4102180 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7fb1b80112f0 con 0x7fb1c4102180 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7fb1b80487e0 con 0x7fb1c4102180 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 10 client.14146.objecter ms_dispatch 0x7fb1c4000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:32.653 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 20 client.14146.objecter dump_active .. 0 homeless 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b0005010 con 0x7fb1c4102180 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.611+0000 7fb1c0ff9640 10 client.14146.objecter ms_handle_connect 0x7fb1ac03c160 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.619+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fb1b8012b70 con 0x7fb1c4102180 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.951+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb1b0005840 con 0x7fb1c4102180 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.955+0000 7fb1c0ff9640 1 -- 172.21.15.83:0/2792380066 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7fb1b80165c0 con 0x7fb1c4102180 2024-09-10T18:39:32.654 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 10 librados: watch_flush enter 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 10 librados: watch_flush exit 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 20 client.14146.objecter shutdown clearing up homeless session... 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 10 client.14146.objecter successfully canceled tick 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 >> v1:172.21.15.83:6800/2452184936 conn(0x7fb1ac03c160 legacy=0x7fb1ac03e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 >> v1:172.21.15.83:6789/0 conn(0x7fb1c4102180 legacy=0x7fb1c41116c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.959+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 shutdown_connections 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.963+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 >> 172.21.15.83:0/2792380066 conn(0x7fb1c40fc2e0 msgr2=0x7fb1c40fc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.963+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 shutdown_connections 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.963+0000 7fb1ca6cb640 1 -- 172.21.15.83:0/2792380066 wait complete. 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:31.963+0000 7fb1ca6cb640 1 librados: shutdown 2024-09-10T18:39:32.655 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling the dashboard module... 2024-09-10T18:39:33.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:32 smithi083 bash[18629]: audit 2024-09-10T18:39:31.900841+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14118 v1:172.21.15.83:0/1599587621' entity='mgr.y' 2024-09-10T18:39:33.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:32 smithi083 bash[18629]: audit 2024-09-10T18:39:31.961142+0000 mon.a (mon.0) 66 : audit [INF] from='client.? v1:172.21.15.83:0/2792380066' entity='client.admin' 2024-09-10T18:39:34.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:33 smithi083 bash[18629]: cluster 2024-09-10T18:39:32.351674+0000 mgr.y (mgr.14118) 23 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:39:35.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:34 smithi083 bash[18629]: audit 2024-09-10T18:39:34.354948+0000 mon.a (mon.0) 67 : audit [INF] from='client.? v1:172.21.15.83:0/3377904614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-10T18:39:35.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18896]: ignoring --setuser ceph since I am not root 2024-09-10T18:39:35.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18896]: ignoring --setgroup ceph since I am not root 2024-09-10T18:39:35.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18896]: debug 2024-09-10T18:39:35.154+0000 7f7134a2bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:39:35.671 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb547ff0640 1 Processor -- start 2024-09-10T18:39:35.671 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb547ff0640 1 -- start start 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb547ff0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb540104fa0 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb545d8c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb540102180 0x7fb540102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41928/0 (socket says 172.21.15.83:41928) 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb545d8c640 1 -- 172.21.15.83:0/2240691463 learned_addr learned my addr 172.21.15.83:0/2240691463 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756803207 0 0) 0x7fb540104fa0 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c0034e0 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227271753 0 0) 0x7fb52c0034e0 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540106140 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb53c001910 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb53c001f90 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb544d8a640 1 -- 172.21.15.83:0/2240691463 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb53c005a10 con 0x7fb540102180 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.918+0000 7fb547ff0640 1 -- 172.21.15.83:0/2240691463 >> v1:172.21.15.83:6789/0 conn(0x7fb540102180 legacy=0x7fb540102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:35.672 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/2240691463 shutdown_connections 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/2240691463 >> 172.21.15.83:0/2240691463 conn(0x7fb5400fc2e0 msgr2=0x7fb5400fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/2240691463 shutdown_connections 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/2240691463 wait complete. 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 librados: starting msgr at 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 librados: starting objecter 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 Processor -- start 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- start start 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 librados: setting wanted keys 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 librados: calling monclient init 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb54010fbd0 con 0x7fb540102180 2024-09-10T18:39:35.673 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb545d8c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb540102180 0x7fb54010f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41940/0 (socket says 172.21.15.83:41940) 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb545d8c640 1 -- 172.21.15.83:0/3377904614 learned_addr learned my addr 172.21.15.83:0/3377904614 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 10 client.?.objecter ms_handle_connect 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956191184 0 0) 0x7fb54010fbd0 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c005680 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641990370 0 0) 0x7fb52c005680 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb54010fbd0 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb53c0057d0 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1328216892 0 0) 0x7fb54010fbd0 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54010fd60 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540110030 con 0x7fb540102180 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:35.674 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb53c0050e0 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb540113c40 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb547ff0640 1 librados: init done 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.922+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fb53c0065d0 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb524005040 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 8) v1 ==== 45457+0+0 (unknown 724222658 0 0) 0x7fb53c0068a0 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7fb53c047e50 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 10 client.14148.objecter ms_dispatch 0x7fb540000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 20 client.14148.objecter dump_active .. 0 homeless 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.926+0000 7fb5327fc640 10 client.14148.objecter ms_handle_connect 0x7fb52c03bf80 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:33.934+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fb53c0070c0 con 0x7fb540102180 2024-09-10T18:39:35.675 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.350+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb5240052f0 con 0x7fb540102180 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.970+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 9) v1 ==== 45470+0+0 (unknown 2459378537 0 0) 0x7fb53c046e90 con 0x7fb540102180 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.970+0000 7fb5327fc640 1 -- 172.21.15.83:0/3377904614 <== mon.0 v1:172.21.15.83:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7fb53c047210 con 0x7fb540102180 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 10 librados: watch_flush enter 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 10 librados: watch_flush exit 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 20 client.14148.objecter shutdown clearing up homeless session... 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 10 client.14148.objecter successfully canceled tick 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 >> v1:172.21.15.83:6800/2452184936 conn(0x7fb52c03bf80 legacy=0x7fb52c03e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 >> v1:172.21.15.83:6789/0 conn(0x7fb540102180 legacy=0x7fb54010f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 shutdown_connections 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 >> 172.21.15.83:0/3377904614 conn(0x7fb5400fc2e0 msgr2=0x7fb540106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 shutdown_connections 2024-09-10T18:39:35.676 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 -- 172.21.15.83:0/3377904614 wait complete. 2024-09-10T18:39:35.677 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:34.978+0000 7fb547ff0640 1 librados: shutdown 2024-09-10T18:39:36.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18629]: audit 2024-09-10T18:39:34.977901+0000 mon.a (mon.0) 68 : audit [INF] from='client.? v1:172.21.15.83:0/3377904614' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-10T18:39:36.058 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18629]: cluster 2024-09-10T18:39:34.977989+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e9: y(active, since 28s) 2024-09-10T18:39:36.058 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18896]: debug 2024-09-10T18:39:35.914+0000 7f7134a2bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:39:36.058 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:35 smithi083 bash[18896]: debug 2024-09-10T18:39:35.982+0000 7f7134a2bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:39:36.453 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:36 smithi083 bash[18896]: debug 2024-09-10T18:39:36.046+0000 7f7134a2bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:39:36.453 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:36 smithi083 bash[18896]: debug 2024-09-10T18:39:36.202+0000 7f7134a2bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:39:36.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:36 smithi083 bash[18896]: debug 2024-09-10T18:39:36.446+0000 7f7134a2bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:39:36.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:36 smithi083 bash[18896]: debug 2024-09-10T18:39:36.510+0000 7f7134a2bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:39:36.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:36 smithi083 bash[18896]: debug 2024-09-10T18:39:36.650+0000 7f7134a2bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:39:37.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18629]: audit 2024-09-10T18:39:37.196959+0000 mon.a (mon.0) 70 : audit [DBG] from='client.? v1:172.21.15.83:0/2047577584' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:39:37.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18896]: debug 2024-09-10T18:39:37.286+0000 7f7134a2bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:39:37.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18896]: debug 2024-09-10T18:39:37.358+0000 7f7134a2bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:39:37.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18896]: debug 2024-09-10T18:39:37.434+0000 7f7134a2bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:39:37.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:39:37.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-10T18:39:37.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:39:37.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.758+0000 7f9e1a5f4640 1 Processor -- start 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.758+0000 7f9e1a5f4640 1 -- start start 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.758+0000 7f9e1a5f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e14104fa0 con 0x7f9e14102180 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e13fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9e14102180 0x7f9e14102590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41966/0 (socket says 172.21.15.83:41966) 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e13fff640 1 -- 172.21.15.83:0/3991473240 learned_addr learned my addr 172.21.15.83:0/3991473240 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022535334 0 0) 0x7f9e14104fa0 con 0x7f9e14102180 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc0034e0 con 0x7f9e14102180 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032596589 0 0) 0x7f9dfc0034e0 con 0x7f9e14102180 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14106140 con 0x7f9e14102180 2024-09-10T18:39:37.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f9e04001590 con 0x7f9e14102180 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9e04001ce0 con 0x7f9e14102180 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e12ffd640 1 -- 172.21.15.83:0/3991473240 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f9e04005700 con 0x7f9e14102180 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/3991473240 >> v1:172.21.15.83:6789/0 conn(0x7f9e14102180 legacy=0x7f9e14102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/3991473240 shutdown_connections 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/3991473240 >> 172.21.15.83:0/3991473240 conn(0x7f9e140fc2e0 msgr2=0x7f9e140fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/3991473240 shutdown_connections 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/3991473240 wait complete. 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 librados: starting msgr at 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 librados: starting objecter 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 Processor -- start 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- start start 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 librados: setting wanted keys 2024-09-10T18:39:37.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 librados: calling monclient init 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e1a5f4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e1410fbc0 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e13fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9e14102180 0x7f9e1410f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41974/0 (socket says 172.21.15.83:41974) 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e13fff640 1 -- 172.21.15.83:0/2047577584 learned_addr learned my addr 172.21.15.83:0/2047577584 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e10ff9640 10 client.?.objecter ms_handle_connect 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e10ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991660495 0 0) 0x7f9e1410fbc0 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.762+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc005630 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756983161 0 0) 0x7f9dfc005630 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e1410fbc0 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f9e04001060 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3583876057 0 0) 0x7f9e1410fbc0 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1410fd50 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e14110020 con 0x7f9e14102180 2024-09-10T18:39:37.887 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e1a5f4640 10 client.14152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e14113c30 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9e040057f0 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e1a5f4640 1 librados: init done 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f9e04006220 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 9) v1 ==== 45470+0+0 (unknown 2459378537 0 0) 0x7f9e04011530 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f9e04048870 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 10 client.14152.objecter ms_dispatch 0x7f9e14000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 3 client.14152.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 3 client.14152.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e10ff9640 20 client.14152.objecter dump_active .. 0 homeless 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e137fe640 1 -- 172.21.15.83:0/2047577584 >> v1:172.21.15.83:6800/2452184936 conn(0x7f9dfc03c160 legacy=0x7f9dfc03e610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/2452184936 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.766+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00005040 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.778+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9e04012be0 con 0x7f9e14102180 2024-09-10T18:39:37.888 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:36.970+0000 7f9e137fe640 1 -- 172.21.15.83:0/2047577584 >> v1:172.21.15.83:6800/2452184936 conn(0x7f9dfc03c160 legacy=0x7f9dfc03e610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/2452184936 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.190+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9e00005b00 con 0x7f9e14102180 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.190+0000 7f9e10ff9640 1 -- 172.21.15.83:0/2047577584 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f9e04016650 con 0x7f9e14102180 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 10 librados: watch_flush enter 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 10 librados: watch_flush exit 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 20 client.14152.objecter shutdown clearing up homeless session... 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 10 client.14152.objecter successfully canceled tick 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 >> v1:172.21.15.83:6800/2452184936 conn(0x7f9dfc03c160 legacy=0x7f9dfc03e610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 >> v1:172.21.15.83:6789/0 conn(0x7f9e14102180 legacy=0x7f9e1410f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:37.889 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 shutdown_connections 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 >> 172.21.15.83:0/2047577584 conn(0x7f9e140fc2e0 msgr2=0x7f9e14106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 shutdown_connections 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 -- 172.21.15.83:0/2047577584 wait complete. 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:37.194+0000 7f9e1a5f4640 1 librados: shutdown 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for the mgr to restart... 2024-09-10T18:39:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:Waiting for mgr epoch 9... 2024-09-10T18:39:37.972 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18896]: debug 2024-09-10T18:39:37.714+0000 7f7134a2bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:39:37.972 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:37 smithi083 bash[18896]: debug 2024-09-10T18:39:37.850+0000 7f7134a2bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:39:38.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:38 smithi083 bash[18896]: debug 2024-09-10T18:39:38.162+0000 7f7134a2bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:39:38.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:38 smithi083 bash[18896]: debug 2024-09-10T18:39:38.374+0000 7f7134a2bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:39:39.080 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:38 smithi083 bash[18896]: debug 2024-09-10T18:39:38.790+0000 7f7134a2bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:39:39.080 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:38 smithi083 bash[18896]: debug 2024-09-10T18:39:38.850+0000 7f7134a2bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:39:39.080 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:38 smithi083 bash[18896]: debug 2024-09-10T18:39:38.918+0000 7f7134a2bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:39:39.081 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:39 smithi083 bash[18896]: debug 2024-09-10T18:39:39.010+0000 7f7134a2bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:39:39.374 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:39 smithi083 bash[18896]: debug 2024-09-10T18:39:39.074+0000 7f7134a2bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:39:39.374 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:39 smithi083 bash[18896]: debug 2024-09-10T18:39:39.146+0000 7f7134a2bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:39:39.374 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:39 smithi083 bash[18896]: debug 2024-09-10T18:39:39.294+0000 7f7134a2bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:39:39.665 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:39 smithi083 bash[18896]: debug 2024-09-10T18:39:39.366+0000 7f7134a2bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:39:40.543 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18896]: debug 2024-09-10T18:39:40.334+0000 7f7134a2bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:39:40.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: cluster 2024-09-10T18:39:40.343197+0000 mon.a (mon.0) 71 : cluster [INF] Active manager daemon y restarted 2024-09-10T18:39:40.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: cluster 2024-09-10T18:39:40.343743+0000 mon.a (mon.0) 72 : cluster [INF] Activating manager daemon y 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: cluster 2024-09-10T18:39:40.347481+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: cluster 2024-09-10T18:39:40.347619+0000 mon.a (mon.0) 74 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00406424s) 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.351015+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.353073+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.354018+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.354178+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.354364+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: cluster 2024-09-10T18:39:40.361663+0000 mon.a (mon.0) 80 : cluster [INF] Manager daemon y is now available 2024-09-10T18:39:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:40 smithi083 bash[18629]: audit 2024-09-10T18:39:40.387845+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:41.678 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18629]: audit 2024-09-10T18:39:40.420088+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:39:41.678 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18629]: audit 2024-09-10T18:39:40.427224+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:39:41.678 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18629]: cluster 2024-09-10T18:39:41.355703+0000 mon.a (mon.0) 84 : cluster [DBG] mgrmap e11: y(active, since 1.01213s) 2024-09-10T18:39:42.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: debug 2024-09-10T18:39:41.670+0000 7f70e557c640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:39:41] ENGINE Error in HTTPServer.serve 2024-09-10T18:39:42.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: Traceback (most recent call last): 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: self._connections.run(self.expiration_interval) 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: self._run(expiration_interval) 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: s = self.context.wrap_socket( 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: return self.sslsocket_class._create( 2024-09-10T18:39:42.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:39:42.042 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: self.do_handshake() 2024-09-10T18:39:42.042 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:39:42.042 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: self._sslobj.do_handshake() 2024-09-10T18:39:42.042 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:39:41 smithi083 bash[18896]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:39:42.067 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:39:42.067 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-10T18:39:42.067 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45d1946640 1 Processor -- start 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45d1946640 1 -- start start 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45d1946640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45cc1071a0 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45caffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f45cc104380 0x7f45cc104790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41988/0 (socket says 172.21.15.83:41988) 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45caffd640 1 -- 172.21.15.83:0/4002271504 learned_addr learned my addr 172.21.15.83:0/4002271504 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283804427 0 0) 0x7f45cc1071a0 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b80034e0 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1619137327 0 0) 0x7f45b80034e0 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc108340 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f45bc001910 con 0x7f45cc104380 2024-09-10T18:39:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f45bc001f90 con 0x7f45cc104380 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.862+0000 7f45c9ffb640 1 -- 172.21.15.83:0/4002271504 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f45bc005a10 con 0x7f45cc104380 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/4002271504 >> v1:172.21.15.83:6789/0 conn(0x7f45cc104380 legacy=0x7f45cc104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/4002271504 shutdown_connections 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/4002271504 >> 172.21.15.83:0/4002271504 conn(0x7f45cc076320 msgr2=0x7f45cc076740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/4002271504 shutdown_connections 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/4002271504 wait complete. 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 librados: starting msgr at 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 librados: starting objecter 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 Processor -- start 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- start start 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 librados: setting wanted keys 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 librados: calling monclient init 2024-09-10T18:39:42.069 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f45cc111df0 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45caffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f45cc104380 0x7f45cc1116e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41998/0 (socket says 172.21.15.83:41998) 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45caffd640 1 -- 172.21.15.83:0/2201394614 learned_addr learned my addr 172.21.15.83:0/2201394614 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 10 client.?.objecter ms_handle_connect 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375443258 0 0) 0x7f45cc111df0 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b8005680 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288149233 0 0) 0x7f45b8005680 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45cc111df0 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f45bc0057d0 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1072544694 0 0) 0x7f45cc111df0 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc111f80 con 0x7f45cc104380 2024-09-10T18:39:42.070 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc112250 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc115e60 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45d1946640 1 librados: init done 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f45bc001e20 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.866+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f45bc006420 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 9) v1 ==== 45470+0+0 (unknown 2459378537 0 0) 0x7f45bc011750 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (unknown 2381955277 0 0) 0x7f45bc048bf0 con 0x7f45cc104380 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 10 client.14154.objecter ms_dispatch 0x7f45cc000b90 osd_map(2..2 src has 1..2) v4 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45ca7fc640 1 -- 172.21.15.83:0/2201394614 >> v1:172.21.15.83:6800/2452184936 conn(0x7f45b803c050 legacy=0x7f45b803e500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/2452184936 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45c37fe640 20 client.14154.objecter dump_active .. 0 homeless 2024-09-10T18:39:42.071 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:39.870+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6800/2452184936 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45ac000d40 con 0x7f45b803c050 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:40.074+0000 7f45ca7fc640 1 -- 172.21.15.83:0/2201394614 >> v1:172.21.15.83:6800/2452184936 conn(0x7f45b803c050 legacy=0x7f45b803e500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.83:6800/2452184936 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:40.342+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mgrmap(e 10) v1 ==== 45272+0+0 (unknown 154714084 0 0) 0x7f45bc005d40 con 0x7f45cc104380 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:40.342+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 >> v1:172.21.15.83:6800/2452184936 conn(0x7f45b803c050 legacy=0x7f45b803e500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.346+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 11) v1 ==== 45364+0+0 (unknown 1020138881 0 0) 0x7f45bc034c20 con 0x7f45cc104380 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.346+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6800/2630800463 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45ac000d40 con 0x7f45b8040a90 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.346+0000 7f45c37fe640 10 client.14154.objecter ms_handle_connect 0x7f45b8040a90 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.346+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (unknown 0 0 4193150809) 0x7f45ac000d40 con 0x7f45b8040a90 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 --> v1:172.21.15.83:6800/2630800463 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f45ac002730 con 0x7f45b8040a90 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45c37fe640 1 -- 172.21.15.83:0/2201394614 <== mgr.14150 v1:172.21.15.83:6800/2630800463 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7f45ac002730 con 0x7f45b8040a90 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 10 librados: watch_flush enter 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 10 librados: watch_flush exit 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 20 client.14154.objecter shutdown clearing up homeless session... 2024-09-10T18:39:42.072 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 10 client.14154.objecter successfully canceled tick 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 >> v1:172.21.15.83:6800/2630800463 conn(0x7f45b8040a90 legacy=0x7f45b8042e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 >> v1:172.21.15.83:6789/0 conn(0x7f45cc104380 legacy=0x7f45cc1116e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 shutdown_connections 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 >> 172.21.15.83:0/2201394614 conn(0x7f45cc076320 msgr2=0x7f45cc0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 shutdown_connections 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 -- 172.21.15.83:0/2201394614 wait complete. 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:41.362+0000 7f45d1946640 1 librados: shutdown 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:mgr epoch 9 is available 2024-09-10T18:39:42.073 INFO:teuthology.orchestra.run.smithi083.stdout:Generating a dashboard self-signed certificate... 2024-09-10T18:39:43.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: cephadm 2024-09-10T18:39:41.547938+0000 mgr.y (mgr.14150) 3 : cephadm [INF] [10/Sep/2024:18:39:41] ENGINE Bus STARTING 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: cephadm 2024-09-10T18:39:41.674460+0000 mgr.y (mgr.14150) 4 : cephadm [INF] [10/Sep/2024:18:39:41] ENGINE Serving on https://172.21.15.83:7150 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: cephadm 2024-09-10T18:39:41.674675+0000 mgr.y (mgr.14150) 5 : cephadm [INF] [10/Sep/2024:18:39:41] ENGINE Bus STARTED 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: cephadm 2024-09-10T18:39:41.678112+0000 mgr.y (mgr.14150) 6 : cephadm [ERR] [10/Sep/2024:18:39:41] ENGINE Error in HTTPServer.serve 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: Traceback (most recent call last): 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: self._connections.run(self.expiration_interval) 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: self._run(expiration_interval) 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:39:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: s = self.context.wrap_socket( 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: return self.sslsocket_class._create( 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: self.do_handshake() 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: self._sslobj.do_handshake() 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:39:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:42 smithi083 bash[18629]: audit 2024-09-10T18:39:41.681602+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:43.940 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:43 smithi083 bash[18629]: cluster 2024-09-10T18:39:42.692867+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-09-10T18:39:44.344 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-10T18:39:44.344 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f1322b640 1 Processor -- start 2024-09-10T18:39:44.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f1322b640 1 -- start start 2024-09-10T18:39:44.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f1322b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2f0c1071a0 con 0x7f2f0c104380 2024-09-10T18:39:44.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f10fc7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2f0c104380 0x7f2f0c104790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57782/0 (socket says 172.21.15.83:57782) 2024-09-10T18:39:44.345 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f10fc7640 1 -- 172.21.15.83:0/4169369893 learned_addr learned my addr 172.21.15.83:0/4169369893 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:44.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325457578 0 0) 0x7f2f0c1071a0 con 0x7f2f0c104380 2024-09-10T18:39:44.346 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef80034e0 con 0x7f2f0c104380 2024-09-10T18:39:44.351 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 515545838 0 0) 0x7f2ef80034e0 con 0x7f2f0c104380 2024-09-10T18:39:44.351 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c108340 con 0x7f2f0c104380 2024-09-10T18:39:44.351 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f2ef4001590 con 0x7f2f0c104380 2024-09-10T18:39:44.351 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2ef4001c10 con 0x7f2f0c104380 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.290+0000 7f2f0b7fe640 1 -- 172.21.15.83:0/4169369893 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f2ef4005690 con 0x7f2f0c104380 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/4169369893 >> v1:172.21.15.83:6789/0 conn(0x7f2f0c104380 legacy=0x7f2f0c104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/4169369893 shutdown_connections 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/4169369893 >> 172.21.15.83:0/4169369893 conn(0x7f2f0c076320 msgr2=0x7f2f0c076740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/4169369893 shutdown_connections 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/4169369893 wait complete. 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 librados: starting msgr at 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 librados: starting objecter 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 Processor -- start 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- start start 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 librados: setting wanted keys 2024-09-10T18:39:44.352 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 librados: calling monclient init 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2f0c111e00 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f10fc7640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2f0c104380 0x7f2f0c1116f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57788/0 (socket says 172.21.15.83:57788) 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f10fc7640 1 -- 172.21.15.83:0/608322743 learned_addr learned my addr 172.21.15.83:0/608322743 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 10 client.?.objecter ms_handle_connect 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625480138 0 0) 0x7f2f0c111e00 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8005680 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4205420592 0 0) 0x7f2ef8005680 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f0c111e00 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f2ef4005450 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3461716200 0 0) 0x7f2f0c111e00 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c111f90 con 0x7f2f0c104380 2024-09-10T18:39:44.353 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0c112260 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0c115e70 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f1322b640 1 librados: init done 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2ef4001aa0 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.294+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f2ef4006000 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 12) v1 ==== 45470+0+0 (unknown 3990653655 0 0) 0x7f2ef4011310 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f2ef4047a30 con 0x7f2f0c104380 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 10 client.14162.objecter ms_dispatch 0x7f2f0c000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 20 client.14162.objecter dump_active .. 0 homeless 2024-09-10T18:39:44.354 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec005040 con 0x7f2f0c104380 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.298+0000 7f2f097fa640 10 client.14162.objecter ms_handle_connect 0x7f2ef803c110 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.306+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2ef4012cd0 con 0x7f2f0c104380 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.638+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f2eec000bc0 con 0x7f2ef803c110 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.702+0000 7f2f097fa640 1 -- 172.21.15.83:0/608322743 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7f2eec000bc0 con 0x7f2ef803c110 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 10 librados: watch_flush enter 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 10 librados: watch_flush exit 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 20 client.14162.objecter shutdown clearing up homeless session... 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 10 client.14162.objecter successfully canceled tick 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 >> v1:172.21.15.83:6800/2630800463 conn(0x7f2ef803c110 legacy=0x7f2ef803e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 >> v1:172.21.15.83:6789/0 conn(0x7f2f0c104380 legacy=0x7f2f0c1116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 shutdown_connections 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 >> 172.21.15.83:0/608322743 conn(0x7f2f0c076320 msgr2=0x7f2f0c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:44.355 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 shutdown_connections 2024-09-10T18:39:44.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 -- 172.21.15.83:0/608322743 wait complete. 2024-09-10T18:39:44.356 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:43.706+0000 7f2f1322b640 1 librados: shutdown 2024-09-10T18:39:44.356 INFO:teuthology.orchestra.run.smithi083.stdout:Creating initial admin user... 2024-09-10T18:39:44.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:44 smithi083 bash[18629]: audit 2024-09-10T18:39:43.645023+0000 mgr.y (mgr.14150) 7 : audit [DBG] from='client.14162 v1:172.21.15.83:0/608322743' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:44.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:44 smithi083 bash[18629]: audit 2024-09-10T18:39:43.700381+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:44.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:44 smithi083 bash[18629]: audit 2024-09-10T18:39:43.707479+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:46.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:46 smithi083 bash[18629]: audit 2024-09-10T18:39:45.972289+0000 mgr.y (mgr.14150) 8 : audit [DBG] from='client.14164 v1:172.21.15.83:0/263932923' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:39:46.880 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$UnI8SFy4VFE95E9hZNDstuR99tTHHbsI7z7xxl2pTFuVOJyZMOpHK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725993586, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f83780c8640 1 Processor -- start 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f83780c8640 1 -- start start 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f83780c8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8370100460 con 0x7f83700ff940 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8375e64640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f83700ff940 0x7f83700ffd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57804/0 (socket says 172.21.15.83:57804) 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8375e64640 1 -- 172.21.15.83:0/504932827 learned_addr learned my addr 172.21.15.83:0/504932827 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:46.881 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883540787 0 0) 0x7f8370100460 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c0034e0 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 968367106 0 0) 0x7f835c0034e0 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370104e50 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.594+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f836c001590 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f836c001c10 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f8374e62640 1 -- 172.21.15.83:0/504932827 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f836c005630 con 0x7f83700ff940 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/504932827 >> v1:172.21.15.83:6789/0 conn(0x7f83700ff940 legacy=0x7f83700ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/504932827 shutdown_connections 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/504932827 >> 172.21.15.83:0/504932827 conn(0x7f83700fb140 msgr2=0x7f83700fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/504932827 shutdown_connections 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/504932827 wait complete. 2024-09-10T18:39:46.882 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 librados: starting msgr at 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 librados: starting objecter 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 Processor -- start 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- start start 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 librados: setting wanted keys 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 librados: calling monclient init 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8370110060 con 0x7f83700ff940 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f8375e64640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f83700ff940 0x7f8370114670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57810/0 (socket says 172.21.15.83:57810) 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f8375e64640 1 -- 172.21.15.83:0/263932923 learned_addr learned my addr 172.21.15.83:0/263932923 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 10 client.?.objecter ms_handle_connect 0x7f83700ff940 2024-09-10T18:39:46.883 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904436861 0 0) 0x7f8370110060 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c0056b0 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3869764272 0 0) 0x7f835c0056b0 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8370110060 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f836c0047a0 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240400864 0 0) 0x7f8370110060 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8370074ae0 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8370074d90 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8370115f30 con 0x7f83700ff940 2024-09-10T18:39:46.884 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.598+0000 7f83780c8640 1 librados: init done 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f836c004db0 con 0x7f83700ff940 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f836c0060c0 con 0x7f83700ff940 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 12) v1 ==== 45470+0+0 (unknown 3990653655 0 0) 0x7f836c006370 con 0x7f83700ff940 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f836c047c60 con 0x7f83700ff940 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 10 client.14164.objecter ms_dispatch 0x7f8370000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 20 client.14164.objecter dump_active .. 0 homeless 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83667fc640 10 client.14164.objecter ms_handle_connect 0x7f835c03c090 2024-09-10T18:39:46.885 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.602+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8354005040 con 0x7f83700ff940 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.610+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f836c007d80 con 0x7f83700ff940 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:45.966+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8354002420 con 0x7f835c03c090 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.198+0000 7f83667fc640 1 -- 172.21.15.83:0/263932923 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 1014719772) 0x7f8354002420 con 0x7f835c03c090 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 10 librados: watch_flush enter 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 10 librados: watch_flush exit 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 20 client.14164.objecter shutdown clearing up homeless session... 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 10 client.14164.objecter successfully canceled tick 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 >> v1:172.21.15.83:6800/2630800463 conn(0x7f835c03c090 legacy=0x7f835c03e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 >> v1:172.21.15.83:6789/0 conn(0x7f83700ff940 legacy=0x7f8370114670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 shutdown_connections 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 >> 172.21.15.83:0/263932923 conn(0x7f83700fb140 msgr2=0x7f8370105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 shutdown_connections 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 -- 172.21.15.83:0/263932923 wait complete. 2024-09-10T18:39:46.886 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:46.202+0000 7f83780c8640 1 librados: shutdown 2024-09-10T18:39:46.887 INFO:teuthology.orchestra.run.smithi083.stdout:Fetching dashboard port number... 2024-09-10T18:39:47.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:47 smithi083 bash[18629]: audit 2024-09-10T18:39:46.117161+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:47.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:47 smithi083 bash[18629]: audit 2024-09-10T18:39:46.202042+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:47.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:47 smithi083 bash[18629]: audit 2024-09-10T18:39:46.692860+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:48.473 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:48 smithi083 bash[18629]: cluster 2024-09-10T18:39:47.206975+0000 mon.a (mon.0) 92 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stdout 8443 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.098+0000 7f7c7c166640 1 Processor -- start 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.098+0000 7f7c7c166640 1 -- start start 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.098+0000 7f7c7c166640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c74079c70 con 0x7f7c74078df0 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.098+0000 7f7c79f02640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7c74078df0 0x7f7c74079200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57822/0 (socket says 172.21.15.83:57822) 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c79f02640 1 -- 172.21.15.83:0/2217778072 learned_addr learned my addr 172.21.15.83:0/2217778072 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:49.090 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480231031 0 0) 0x7f7c74079c70 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c600034e0 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1842110889 0 0) 0x7f7c600034e0 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7407ae10 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f7c64001590 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7c64001c10 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c78f00640 1 -- 172.21.15.83:0/2217778072 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f7c64005630 con 0x7f7c74078df0 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- 172.21.15.83:0/2217778072 >> v1:172.21.15.83:6789/0 conn(0x7f7c74078df0 legacy=0x7f7c74079200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- 172.21.15.83:0/2217778072 shutdown_connections 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- 172.21.15.83:0/2217778072 >> 172.21.15.83:0/2217778072 conn(0x7f7c74076320 msgr2=0x7f7c74076740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- 172.21.15.83:0/2217778072 shutdown_connections 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- 172.21.15.83:0/2217778072 wait complete. 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 librados: starting msgr at 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 librados: starting objecter 2024-09-10T18:39:49.091 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 Processor -- start 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- start start 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 librados: setting wanted keys 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 librados: calling monclient init 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c7c166640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c74116230 con 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c79f02640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7c74078df0 0x7f7c74115b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57830/0 (socket says 172.21.15.83:57830) 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c79f02640 1 -- 172.21.15.83:0/251272062 learned_addr learned my addr 172.21.15.83:0/251272062 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c6a7fc640 10 client.?.objecter ms_handle_connect 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.102+0000 7f7c6a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601535501 0 0) 0x7f7c74116230 con 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c600056b0 con 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578989323 0 0) 0x7f7c600056b0 con 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c74116230 con 0x7f7c74078df0 2024-09-10T18:39:49.092 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f7c640047a0 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1286659320 0 0) 0x7f7c74116230 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c741163c0 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c74116690 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c7c166640 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7411a2a0 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c7c166640 1 librados: init done 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7c64004d30 con 0x7f7c74078df0 2024-09-10T18:39:49.093 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f7c64006040 con 0x7f7c74078df0 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f7c64011400 con 0x7f7c74078df0 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c58005040 con 0x7f7c74078df0 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f7c640489c0 con 0x7f7c74078df0 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 10 client.14166.objecter ms_dispatch 0x7f7c74000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:39:49.094 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 20 client.14166.objecter dump_active .. 0 homeless 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.106+0000 7f7c6a7fc640 10 client.14166.objecter ms_handle_connect 0x7f7c6003c1a0 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.114+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7c64012ca0 con 0x7f7c74078df0 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.438+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7c58005870 con 0x7f7c74078df0 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.438+0000 7f7c6a7fc640 1 -- 172.21.15.83:0/251272062 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7f7c640166f0 con 0x7f7c74078df0 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 10 librados: watch_flush enter 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 10 librados: watch_flush exit 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 20 client.14166.objecter shutdown clearing up homeless session... 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 10 client.14166.objecter successfully canceled tick 2024-09-10T18:39:49.095 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 >> v1:172.21.15.83:6800/2630800463 conn(0x7f7c6003c1a0 legacy=0x7f7c6003e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 >> v1:172.21.15.83:6789/0 conn(0x7f7c74078df0 legacy=0x7f7c74115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 shutdown_connections 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 >> 172.21.15.83:0/251272062 conn(0x7f7c74076320 msgr2=0x7f7c74101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 shutdown_connections 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 -- 172.21.15.83:0/251272062 wait complete. 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:48.442+0000 7f7c7c166640 1 librados: shutdown 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:firewalld does not appear to be present 2024-09-10T18:39:49.096 INFO:teuthology.orchestra.run.smithi083.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-10T18:39:49.097 INFO:teuthology.orchestra.run.smithi083.stdout:Ceph Dashboard is now available at: 2024-09-10T18:39:49.097 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:49.097 INFO:teuthology.orchestra.run.smithi083.stdout: URL: https://smithi083.front.sepia.ceph.com:8443/ 2024-09-10T18:39:49.097 INFO:teuthology.orchestra.run.smithi083.stdout: User: admin 2024-09-10T18:39:49.098 INFO:teuthology.orchestra.run.smithi083.stdout: Password: j90fwtm3zk 2024-09-10T18:39:49.098 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:49.098 INFO:teuthology.orchestra.run.smithi083.stdout:Saving cluster configuration to /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config directory 2024-09-10T18:39:49.098 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling autotune for osd_memory_target 2024-09-10T18:39:49.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:49 smithi083 bash[18629]: audit 2024-09-10T18:39:48.445537+0000 mon.a (mon.0) 93 : audit [DBG] from='client.? v1:172.21.15.83:0/251272062' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-10T18:39:51.287 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a42e21640 1 Processor -- start 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a42e21640 1 -- start start 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a42e21640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a3c103830 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a40bbd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8a3c102d10 0x7f8a3c103120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57834/0 (socket says 172.21.15.83:57834) 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a40bbd640 1 -- 172.21.15.83:0/113821358 learned_addr learned my addr 172.21.15.83:0/113821358 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035410574 0 0) 0x7f8a3c103830 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.318+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a280034e0 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574643635 0 0) 0x7f8a280034e0 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c108220 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8a34001910 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8a34001f90 con 0x7f8a3c102d10 2024-09-10T18:39:51.288 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a3b7fe640 1 -- 172.21.15.83:0/113821358 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8a340059b0 con 0x7f8a3c102d10 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- 172.21.15.83:0/113821358 >> v1:172.21.15.83:6789/0 conn(0x7f8a3c102d10 legacy=0x7f8a3c103120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- 172.21.15.83:0/113821358 shutdown_connections 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- 172.21.15.83:0/113821358 >> 172.21.15.83:0/113821358 conn(0x7f8a3c076370 msgr2=0x7f8a3c076770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- 172.21.15.83:0/113821358 shutdown_connections 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- 172.21.15.83:0/113821358 wait complete. 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 librados: starting msgr at 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 librados: starting objecter 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 Processor -- start 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- start start 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 librados: setting wanted keys 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 librados: calling monclient init 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a42e21640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a3c111e00 con 0x7f8a3c102d10 2024-09-10T18:39:51.289 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a40bbd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8a3c102d10 0x7f8a3c1116f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57840/0 (socket says 172.21.15.83:57840) 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a40bbd640 1 -- 172.21.15.83:0/228147835 learned_addr learned my addr 172.21.15.83:0/228147835 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a397fa640 10 client.?.objecter ms_handle_connect 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.322+0000 7f8a397fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781949597 0 0) 0x7f8a3c111e00 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a280056b0 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194620489 0 0) 0x7f8a280056b0 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a3c111e00 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8a34004b20 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 108189960 0 0) 0x7f8a3c111e00 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c111f90 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c112260 con 0x7f8a3c102d10 2024-09-10T18:39:51.290 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a42e21640 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:51.291 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c115df0 con 0x7f8a3c102d10 2024-09-10T18:39:51.291 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a42e21640 1 librados: init done 2024-09-10T18:39:51.291 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8a340050b0 con 0x7f8a3c102d10 2024-09-10T18:39:51.291 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8a34006390 con 0x7f8a3c102d10 2024-09-10T18:39:51.291 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f8a340116d0 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f8a34048020 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 10 client.14168.objecter ms_dispatch 0x7f8a3c000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c005040 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 20 client.14168.objecter dump_active .. 0 homeless 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.326+0000 7f8a397fa640 10 client.14168.objecter ms_handle_connect 0x7f8a2803c200 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.334+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8a34012f70 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.662+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f8a1c005870 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.662+0000 7f8a397fa640 1 -- 172.21.15.83:0/228147835 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7f8a34016530 con 0x7f8a3c102d10 2024-09-10T18:39:51.292 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 10 librados: watch_flush enter 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 10 librados: watch_flush exit 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 20 client.14168.objecter shutdown clearing up homeless session... 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 10 client.14168.objecter successfully canceled tick 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 >> v1:172.21.15.83:6800/2630800463 conn(0x7f8a2803c200 legacy=0x7f8a2803e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 >> v1:172.21.15.83:6789/0 conn(0x7f8a3c102d10 legacy=0x7f8a3c1116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 shutdown_connections 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 >> 172.21.15.83:0/228147835 conn(0x7f8a3c076370 msgr2=0x7f8a3c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 shutdown_connections 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 -- 172.21.15.83:0/228147835 wait complete. 2024-09-10T18:39:51.293 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:50.666+0000 7f8a42e21640 1 librados: shutdown 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57d2b50640 1 Processor -- start 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57d2b50640 1 -- start start 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57d2b50640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f57cc100460 con 0x7f57cc0ff940 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57d08ec640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f57cc0ff940 0x7f57cc0ffd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57850/0 (socket says 172.21.15.83:57850) 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57d08ec640 1 -- 172.21.15.83:0/2947907650 learned_addr learned my addr 172.21.15.83:0/2947907650 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492023435 0 0) 0x7f57cc100460 con 0x7f57cc0ff940 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b40034e0 con 0x7f57cc0ff940 2024-09-10T18:39:54.224 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065435359 0 0) 0x7f57b40034e0 con 0x7f57cc0ff940 2024-09-10T18:39:54.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc104e50 con 0x7f57cc0ff940 2024-09-10T18:39:54.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f57c4001910 con 0x7f57cc0ff940 2024-09-10T18:39:54.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 <== mon.0 v1:172.21.15.83:6789/0 4 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f57c4002060 con 0x7f57cc0ff940 2024-09-10T18:39:54.229 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.914+0000 7f57cb7fe640 1 -- 172.21.15.83:0/2947907650 <== mon.0 v1:172.21.15.83:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f57c4005a80 con 0x7f57cc0ff940 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- 172.21.15.83:0/2947907650 >> v1:172.21.15.83:6789/0 conn(0x7f57cc0ff940 legacy=0x7f57cc0ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- 172.21.15.83:0/2947907650 shutdown_connections 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- 172.21.15.83:0/2947907650 >> 172.21.15.83:0/2947907650 conn(0x7f57cc0fb140 msgr2=0x7f57cc0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- 172.21.15.83:0/2947907650 shutdown_connections 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- 172.21.15.83:0/2947907650 wait complete. 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 librados: starting msgr at 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 librados: starting objecter 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 Processor -- start 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- start start 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 librados: setting wanted keys 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 librados: calling monclient init 2024-09-10T18:39:54.230 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d2b50640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f57cc074b90 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d08ec640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f57cc0ff940 0x7f57cc112190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57858/0 (socket says 172.21.15.83:57858) 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57d08ec640 1 -- 172.21.15.83:0/837982457 learned_addr learned my addr 172.21.15.83:0/837982457 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 10 client.?.objecter ms_handle_connect 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205980530 0 0) 0x7f57cc074b90 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4005660 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1134063724 0 0) 0x7f57b4005660 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57cc074b90 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f57c40013e0 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 865608036 0 0) 0x7f57cc074b90 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.918+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc074d20 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57cc1138b0 con 0x7f57cc0ff940 2024-09-10T18:39:54.231 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57d2b50640 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57cc113be0 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57d2b50640 1 librados: init done 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f57c4005a70 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f57c4006530 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f57c4011890 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b8005040 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f57c4048020 con 0x7f57cc0ff940 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 10 client.14170.objecter ms_dispatch 0x7f57cc000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 20 client.14170.objecter dump_active .. 0 homeless 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.922+0000 7f57c97fa640 10 client.14170.objecter ms_handle_connect 0x7f57b403c170 2024-09-10T18:39:54.232 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:52.926+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f57c4013090 con 0x7f57cc0ff940 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.438+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f57b8005870 con 0x7f57cc0ff940 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57c97fa640 1 -- 172.21.15.83:0/837982457 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (unknown 1169358022 0 0) 0x7f57c4016a80 con 0x7f57cc0ff940 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 10 librados: watch_flush enter 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 10 librados: watch_flush exit 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 20 client.14170.objecter shutdown clearing up homeless session... 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 10 client.14170.objecter successfully canceled tick 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 >> v1:172.21.15.83:6800/2630800463 conn(0x7f57b403c170 legacy=0x7f57b403e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:54.233 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 >> v1:172.21.15.83:6789/0 conn(0x7f57cc0ff940 legacy=0x7f57cc112190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:39:54.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 shutdown_connections 2024-09-10T18:39:54.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 >> 172.21.15.83:0/837982457 conn(0x7f57cc0fb140 msgr2=0x7f57cc105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:39:54.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.442+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 shutdown_connections 2024-09-10T18:39:54.234 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.446+0000 7f57d2b50640 1 -- 172.21.15.83:0/837982457 wait complete. 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout:/usr/bin/ceph: stderr 2024-09-10T18:39:53.446+0000 7f57d2b50640 1 librados: shutdown 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout:Or, if you are only running a single cluster on this host: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: ceph telemetry on 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.235 INFO:teuthology.orchestra.run.smithi083.stdout:For more information see: 2024-09-10T18:39:54.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.236 INFO:teuthology.orchestra.run.smithi083.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-10T18:39:54.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:39:54.236 INFO:teuthology.orchestra.run.smithi083.stdout:Bootstrap complete. 2024-09-10T18:39:54.269 INFO:tasks.cephadm:Fetching config... 2024-09-10T18:39:54.269 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:39:54.269 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-10T18:39:54.274 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-10T18:39:54.274 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:39:54.274 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-10T18:39:54.281 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-10T18:39:54.281 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:39:54.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-10T18:39:54.294 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-10T18:39:54.294 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:39:54.294 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-10T18:39:54.348 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-10T18:39:54.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfjy+Sz0+yVv8FRjRmCspdqDVuL5Xuy2woPtsqQemUIDlItq20uPM02HNIxTRblrPtHTJXgmlgnSwEMYAL0Rq16I+yZZElIB+vcChQEmGXkuHdjLJ0GT8nYxsy1t94tQauT5JB4axkHsHfA1r6gM4C/OZDz1fcV+EU6sbDsBHGY43/0nzNMgTSyQ26hjr3pBhvp59anKNdbu67iOc2/L9Pe6bIeK6l+k2/eWsATm77whFsVZPZCou+g07DbZSe+b+e2F6qQtQTf9Iu7dLuIrMABhOXwqMDxTvPHfkV4SLf7tcLAZ5JDr/xbJbS0JiGT1tvJ8VymG/6heiJ95s24LL/nXXd/eEuKSaKyAmQdgsqgdRt1xNtuRFHgKVmGcjRCWFmRokuEtHi7vDe21oWABuvov/aJJzcnX4xzkxzcIlgPTPhvb9MGZwaiNms/i6X0EOBRW8SUzZlbnBZ1aQz09/fy+UXdITP7z+Ksj6bdgTzcOOduL7InTBCdzIulgjjaCM= ceph-707a4ada-6fa3-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:39:54.374 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfjy+Sz0+yVv8FRjRmCspdqDVuL5Xuy2woPtsqQemUIDlItq20uPM02HNIxTRblrPtHTJXgmlgnSwEMYAL0Rq16I+yZZElIB+vcChQEmGXkuHdjLJ0GT8nYxsy1t94tQauT5JB4axkHsHfA1r6gM4C/OZDz1fcV+EU6sbDsBHGY43/0nzNMgTSyQ26hjr3pBhvp59anKNdbu67iOc2/L9Pe6bIeK6l+k2/eWsATm77whFsVZPZCou+g07DbZSe+b+e2F6qQtQTf9Iu7dLuIrMABhOXwqMDxTvPHfkV4SLf7tcLAZ5JDr/xbJbS0JiGT1tvJ8VymG/6heiJ95s24LL/nXXd/eEuKSaKyAmQdgsqgdRt1xNtuRFHgKVmGcjRCWFmRokuEtHi7vDe21oWABuvov/aJJzcnX4xzkxzcIlgPTPhvb9MGZwaiNms/i6X0EOBRW8SUzZlbnBZ1aQz09/fy+UXdITP7z+Ksj6bdgTzcOOduL7InTBCdzIulgjjaCM= ceph-707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:39:54.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfjy+Sz0+yVv8FRjRmCspdqDVuL5Xuy2woPtsqQemUIDlItq20uPM02HNIxTRblrPtHTJXgmlgnSwEMYAL0Rq16I+yZZElIB+vcChQEmGXkuHdjLJ0GT8nYxsy1t94tQauT5JB4axkHsHfA1r6gM4C/OZDz1fcV+EU6sbDsBHGY43/0nzNMgTSyQ26hjr3pBhvp59anKNdbu67iOc2/L9Pe6bIeK6l+k2/eWsATm77whFsVZPZCou+g07DbZSe+b+e2F6qQtQTf9Iu7dLuIrMABhOXwqMDxTvPHfkV4SLf7tcLAZ5JDr/xbJbS0JiGT1tvJ8VymG/6heiJ95s24LL/nXXd/eEuKSaKyAmQdgsqgdRt1xNtuRFHgKVmGcjRCWFmRokuEtHi7vDe21oWABuvov/aJJzcnX4xzkxzcIlgPTPhvb9MGZwaiNms/i6X0EOBRW8SUzZlbnBZ1aQz09/fy+UXdITP7z+Ksj6bdgTzcOOduL7InTBCdzIulgjjaCM= ceph-707a4ada-6fa3-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:39:54.410 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfjy+Sz0+yVv8FRjRmCspdqDVuL5Xuy2woPtsqQemUIDlItq20uPM02HNIxTRblrPtHTJXgmlgnSwEMYAL0Rq16I+yZZElIB+vcChQEmGXkuHdjLJ0GT8nYxsy1t94tQauT5JB4axkHsHfA1r6gM4C/OZDz1fcV+EU6sbDsBHGY43/0nzNMgTSyQ26hjr3pBhvp59anKNdbu67iOc2/L9Pe6bIeK6l+k2/eWsATm77whFsVZPZCou+g07DbZSe+b+e2F6qQtQTf9Iu7dLuIrMABhOXwqMDxTvPHfkV4SLf7tcLAZ5JDr/xbJbS0JiGT1tvJ8VymG/6heiJ95s24LL/nXXd/eEuKSaKyAmQdgsqgdRt1xNtuRFHgKVmGcjRCWFmRokuEtHi7vDe21oWABuvov/aJJzcnX4xzkxzcIlgPTPhvb9MGZwaiNms/i6X0EOBRW8SUzZlbnBZ1aQz09/fy+UXdITP7z+Ksj6bdgTzcOOduL7InTBCdzIulgjjaCM= ceph-707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:39:54.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-10T18:39:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:54 smithi083 bash[18629]: audit 2024-09-10T18:39:53.447152+0000 mon.a (mon.0) 94 : audit [INF] from='client.? v1:172.21.15.83:0/837982457' entity='client.admin' 2024-09-10T18:39:59.274 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.806807+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.812036+0000 mon.a (mon.0) 96 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.813072+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.819760+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.843759+0000 mon.a (mon.0) 99 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:39:59.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:39:58 smithi083 bash[18629]: audit 2024-09-10T18:39:57.852690+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:01.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:01 smithi083 bash[18629]: cluster 2024-09-10T18:40:00.355555+0000 mgr.y (mgr.14150) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:02.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2976563214 >> v1:172.21.15.83:6789/0 conn(0x7f94600fe300 legacy=0x7f94600fe710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:02.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2976563214 shutdown_connections 2024-09-10T18:40:02.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2976563214 >> 172.21.15.83:0/2976563214 conn(0x7f94600f9d20 msgr2=0x7f94600fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:02.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2976563214 shutdown_connections 2024-09-10T18:40:02.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2976563214 wait complete. 2024-09-10T18:40:02.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 librados: starting msgr at 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 librados: starting objecter 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 Processor -- start 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- start start 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 librados: setting wanted keys 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 librados: calling monclient init 2024-09-10T18:40:02.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946010e1f0 con 0x7f94600fe300 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f945f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f94600fe300 0x7f946010dae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51350/0 (socket says 172.21.15.83:51350) 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f945f7fe640 1 -- 172.21.15.83:0/2847418047 learned_addr learned my addr 172.21.15.83:0/2847418047 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 10 client.?.objecter ms_handle_connect 0x7f94600fe300 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373075828 0 0) 0x7f946010e1f0 con 0x7f94600fe300 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94480056b0 con 0x7f94600fe300 2024-09-10T18:40:02.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2702353962 0 0) 0x7f94480056b0 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946010e1f0 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f9454001490 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668274763 0 0) 0x7f946010e1f0 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946010e380 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946010e650 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.866+0000 7f946610c640 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9460112260 con 0x7f94600fe300 2024-09-10T18:40:02.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f946610c640 1 librados: init done 2024-09-10T18:40:02.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9454002360 con 0x7f94600fe300 2024-09-10T18:40:02.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f94540037c0 con 0x7f94600fe300 2024-09-10T18:40:02.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f94540039e0 con 0x7f94600fe300 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f9454047c60 con 0x7f94600fe300 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 10 client.14172.objecter ms_dispatch 0x7f9460000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 20 client.14172.objecter dump_active .. 0 homeless 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c005040 con 0x7f94600fe300 2024-09-10T18:40:02.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.870+0000 7f9447fff640 10 client.14172.objecter ms_handle_connect 0x7f944803c0f0 2024-09-10T18:40:02.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:02.874+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9454016ca0 con 0x7f94600fe300 2024-09-10T18:40:03.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.050+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f944c005870 con 0x7f94600fe300 2024-09-10T18:40:03.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.058+0000 7f9447fff640 1 -- 172.21.15.83:0/2847418047 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7f94540166a0 con 0x7f94600fe300 2024-09-10T18:40:03.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 10 librados: watch_flush enter 2024-09-10T18:40:03.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 10 librados: watch_flush exit 2024-09-10T18:40:03.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 20 client.14172.objecter shutdown clearing up homeless session... 2024-09-10T18:40:03.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 10 client.14172.objecter successfully canceled tick 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 >> v1:172.21.15.83:6800/2630800463 conn(0x7f944803c0f0 legacy=0x7f944803e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 >> v1:172.21.15.83:6789/0 conn(0x7f94600fe300 legacy=0x7f946010dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 shutdown_connections 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 >> 172.21.15.83:0/2847418047 conn(0x7f94600f9d20 msgr2=0x7f94600fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 shutdown_connections 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 -- 172.21.15.83:0/2847418047 wait complete. 2024-09-10T18:40:03.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:03.062+0000 7f946610c640 1 librados: shutdown 2024-09-10T18:40:03.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:03 smithi083 bash[18629]: cluster 2024-09-10T18:40:02.355986+0000 mgr.y (mgr.14150) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:03 smithi083 bash[18629]: audit 2024-09-10T18:40:03.063158+0000 mon.a (mon.0) 101 : audit [INF] from='client.? v1:172.21.15.83:0/2847418047' entity='client.admin' 2024-09-10T18:40:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:03 smithi083 bash[18629]: audit 2024-09-10T18:40:03.078823+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:03 smithi083 bash[18629]: audit 2024-09-10T18:40:03.085084+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:03.728 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-10T18:40:03.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-10T18:40:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:05 smithi083 bash[18629]: cluster 2024-09-10T18:40:04.356535+0000 mgr.y (mgr.14150) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:07.444 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:07 smithi083 bash[18629]: cluster 2024-09-10T18:40:06.357013+0000 mgr.y (mgr.14150) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:09 smithi083 bash[18629]: cluster 2024-09-10T18:40:08.357512+0000 mgr.y (mgr.14150) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:10.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.442+0000 7f104b59e640 1 -- 172.21.15.83:0/1454154622 >> v1:172.21.15.83:6789/0 conn(0x7f104c0fe130 legacy=0x7f104c0fe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.442+0000 7f104b59e640 1 -- 172.21.15.83:0/1454154622 shutdown_connections 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.442+0000 7f104b59e640 1 -- 172.21.15.83:0/1454154622 >> 172.21.15.83:0/1454154622 conn(0x7f104c0f9d20 msgr2=0x7f104c0fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.442+0000 7f104b59e640 1 -- 172.21.15.83:0/1454154622 shutdown_connections 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.442+0000 7f104b59e640 1 -- 172.21.15.83:0/1454154622 wait complete. 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 librados: starting msgr at 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 librados: starting objecter 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 Processor -- start 2024-09-10T18:40:10.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 -- start start 2024-09-10T18:40:10.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 librados: setting wanted keys 2024-09-10T18:40:10.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 librados: calling monclient init 2024-09-10T18:40:10.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f104c10e030 con 0x7f104c0fe130 2024-09-10T18:40:10.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f104c0fe130 0x7f104c10d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45132/0 (socket says 172.21.15.83:45132) 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104a59c640 1 -- 172.21.15.83:0/7959724 learned_addr learned my addr 172.21.15.83:0/7959724 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 10 client.?.objecter ms_handle_connect 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537879619 0 0) 0x7f104c10e030 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1034005660 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654829716 0 0) 0x7f1034005660 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f104c10e030 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f1040002050 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 341365553 0 0) 0x7f104c10e030 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c10e1c0 con 0x7f104c0fe130 2024-09-10T18:40:10.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c10e490 con 0x7f104c0fe130 2024-09-10T18:40:10.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:10.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c1120a0 con 0x7f104c0fe130 2024-09-10T18:40:10.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f104b59e640 1 librados: init done 2024-09-10T18:40:10.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1040002430 con 0x7f104c0fe130 2024-09-10T18:40:10.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.446+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f10400036c0 con 0x7f104c0fe130 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c005040 con 0x7f104c0fe130 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f10400038c0 con 0x7f104c0fe130 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f1040047aa0 con 0x7f104c0fe130 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 10 client.14174.objecter ms_dispatch 0x7f104c000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:40:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 20 client.14174.objecter dump_active .. 0 homeless 2024-09-10T18:40:10.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.450+0000 7f103affd640 10 client.14174.objecter ms_handle_connect 0x7f103403c150 2024-09-10T18:40:10.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.462+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1040016bd0 con 0x7f104c0fe130 2024-09-10T18:40:10.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.642+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f102c000bc0 con 0x7f103403c150 2024-09-10T18:40:10.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.650+0000 7f103affd640 1 -- 172.21.15.83:0/7959724 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f102c000bc0 con 0x7f103403c150 2024-09-10T18:40:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 10 librados: watch_flush enter 2024-09-10T18:40:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 10 librados: watch_flush exit 2024-09-10T18:40:10.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 20 client.14174.objecter shutdown clearing up homeless session... 2024-09-10T18:40:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 10 client.14174.objecter successfully canceled tick 2024-09-10T18:40:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 >> v1:172.21.15.83:6800/2630800463 conn(0x7f103403c150 legacy=0x7f103403e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 >> v1:172.21.15.83:6789/0 conn(0x7f104c0fe130 legacy=0x7f104c10d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 shutdown_connections 2024-09-10T18:40:10.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 >> 172.21.15.83:0/7959724 conn(0x7f104c0f9d20 msgr2=0x7f104c0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 shutdown_connections 2024-09-10T18:40:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 -- 172.21.15.83:0/7959724 wait complete. 2024-09-10T18:40:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:10.654+0000 7f104b59e640 1 librados: shutdown 2024-09-10T18:40:11.357 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-09-10T18:40:11.358 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:40:11.358 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-09-10T18:40:11.365 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:40:11.365 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:40:11.413 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-09-10T18:40:11.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch host add smithi165 2024-09-10T18:40:12.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: cluster 2024-09-10T18:40:10.357937+0000 mgr.y (mgr.14150) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:10.643775+0000 mgr.y (mgr.14150) 15 : audit [DBG] from='client.14174 v1:172.21.15.83:0/7959724' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:10.651445+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:10.652524+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:10.655687+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:10.657385+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: cephadm 2024-09-10T18:40:10.659662+0000 mgr.y (mgr.14150) 16 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: cephadm 2024-09-10T18:40:10.765581+0000 mgr.y (mgr.14150) 17 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: cephadm 2024-09-10T18:40:10.877011+0000 mgr.y (mgr.14150) 18 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:11.167539+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:11.172043+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:11 smithi083 bash[18629]: audit 2024-09-10T18:40:11.179398+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:13.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:12 smithi083 bash[18629]: cephadm 2024-09-10T18:40:11.042158+0000 mgr.y (mgr.14150) 19 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:40:14.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:13 smithi083 bash[18629]: cluster 2024-09-10T18:40:12.358406+0000 mgr.y (mgr.14150) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:15 smithi083 bash[18629]: cluster 2024-09-10T18:40:14.358861+0000 mgr.y (mgr.14150) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:16.138 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:17.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:17 smithi083 bash[18629]: cluster 2024-09-10T18:40:16.359329+0000 mgr.y (mgr.14150) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:19.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.109+0000 7f6c56b22640 1 -- 172.21.15.83:0/1212369626 >> v1:172.21.15.83:6789/0 conn(0x7f6c50104330 legacy=0x7f6c50104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:19.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.109+0000 7f6c56b22640 1 -- 172.21.15.83:0/1212369626 shutdown_connections 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.109+0000 7f6c56b22640 1 -- 172.21.15.83:0/1212369626 >> 172.21.15.83:0/1212369626 conn(0x7f6c500fa200 msgr2=0x7f6c500fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- 172.21.15.83:0/1212369626 shutdown_connections 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- 172.21.15.83:0/1212369626 wait complete. 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 librados: starting msgr at 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 librados: starting objecter 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 Processor -- start 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- start start 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 librados: setting wanted keys 2024-09-10T18:40:19.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 librados: calling monclient init 2024-09-10T18:40:19.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6c501140e0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c548be640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6c50104330 0x7f6c501139d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42486/0 (socket says 172.21.15.83:42486) 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c548be640 1 -- 172.21.15.83:0/3449687605 learned_addr learned my addr 172.21.15.83:0/3449687605 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 10 client.?.objecter ms_handle_connect 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984201753 0 0) 0x7f6c501140e0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c380056b0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3729618769 0 0) 0x7f6c380056b0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c501140e0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f6c400014d0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2455159705 0 0) 0x7f6c501140e0 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c50114270 con 0x7f6c50104330 2024-09-10T18:40:19.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c50114540 con 0x7f6c50104330 2024-09-10T18:40:19.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:19.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c501169f0 con 0x7f6c50104330 2024-09-10T18:40:19.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c56b22640 1 librados: init done 2024-09-10T18:40:19.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6c40002360 con 0x7f6c50104330 2024-09-10T18:40:19.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f6c40003500 con 0x7f6c50104330 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.113+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f6c40003680 con 0x7f6c50104330 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c005010 con 0x7f6c50104330 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f6c40047a00 con 0x7f6c50104330 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 10 client.14176.objecter ms_dispatch 0x7f6c50000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 20 client.14176.objecter dump_active .. 0 homeless 2024-09-10T18:40:19.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.117+0000 7f6c4d7fa640 10 client.14176.objecter ms_handle_connect 0x7f6c3803c210 2024-09-10T18:40:19.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.125+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f6c400169a0 con 0x7f6c50104330 2024-09-10T18:40:19.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:19.301+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c3c000b90 con 0x7f6c3803c210 2024-09-10T18:40:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:19 smithi083 bash[18629]: cluster 2024-09-10T18:40:18.359758+0000 mgr.y (mgr.14150) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:20 smithi083 bash[18629]: audit 2024-09-10T18:40:19.306682+0000 mgr.y (mgr.14150) 24 : audit [DBG] from='client.14176 v1:172.21.15.83:0/3449687605' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:21.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:21 smithi083 bash[18629]: cluster 2024-09-10T18:40:20.360121+0000 mgr.y (mgr.14150) 25 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:21.992 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:21 smithi083 bash[18629]: cephadm 2024-09-10T18:40:20.555395+0000 mgr.y (mgr.14150) 26 : cephadm [INF] Deploying cephadm binary to smithi165 2024-09-10T18:40:23.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.189+0000 7f6c4d7fa640 1 -- 172.21.15.83:0/3449687605 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 563005145) 0x7f6c3c000b90 con 0x7f6c3803c210 2024-09-10T18:40:23.192 INFO:teuthology.orchestra.run.smithi083.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-09-10T18:40:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 10 librados: watch_flush enter 2024-09-10T18:40:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 10 librados: watch_flush exit 2024-09-10T18:40:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 20 client.14176.objecter shutdown clearing up homeless session... 2024-09-10T18:40:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 10 client.14176.objecter successfully canceled tick 2024-09-10T18:40:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 >> v1:172.21.15.83:6800/2630800463 conn(0x7f6c3803c210 legacy=0x7f6c3803e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 >> v1:172.21.15.83:6789/0 conn(0x7f6c50104330 legacy=0x7f6c501139d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 shutdown_connections 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 >> 172.21.15.83:0/3449687605 conn(0x7f6c500fa200 msgr2=0x7f6c500fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 shutdown_connections 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 -- 172.21.15.83:0/3449687605 wait complete. 2024-09-10T18:40:23.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:23.193+0000 7f6c56b22640 1 librados: shutdown 2024-09-10T18:40:23.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-10T18:40:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:23 smithi083 bash[18629]: cluster 2024-09-10T18:40:22.360509+0000 mgr.y (mgr.14150) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:23 smithi083 bash[18629]: audit 2024-09-10T18:40:23.190438+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:23 smithi083 bash[18629]: audit 2024-09-10T18:40:23.191926+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:23 smithi083 bash[18629]: audit 2024-09-10T18:40:23.487948+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:25.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:24 smithi083 bash[18629]: cephadm 2024-09-10T18:40:23.191158+0000 mgr.y (mgr.14150) 28 : cephadm [INF] Added host smithi165 2024-09-10T18:40:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:25 smithi083 bash[18629]: cluster 2024-09-10T18:40:24.361014+0000 mgr.y (mgr.14150) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:25 smithi083 bash[18629]: audit 2024-09-10T18:40:24.781975+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:25 smithi083 bash[18629]: audit 2024-09-10T18:40:25.461435+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:27 smithi083 bash[18629]: cluster 2024-09-10T18:40:26.361447+0000 mgr.y (mgr.14150) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:28.569 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:29.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:29 smithi083 bash[18629]: cluster 2024-09-10T18:40:28.361957+0000 mgr.y (mgr.14150) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- 172.21.15.83:0/703289115 >> v1:172.21.15.83:6789/0 conn(0x7f87880716e0 legacy=0x7f878806fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- 172.21.15.83:0/703289115 shutdown_connections 2024-09-10T18:40:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- 172.21.15.83:0/703289115 >> 172.21.15.83:0/703289115 conn(0x7f87880f7b90 msgr2=0x7f87880f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- 172.21.15.83:0/703289115 shutdown_connections 2024-09-10T18:40:31.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- 172.21.15.83:0/703289115 wait complete. 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 librados: starting msgr at 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 librados: starting objecter 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 Processor -- start 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- start start 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 librados: setting wanted keys 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 librados: calling monclient init 2024-09-10T18:40:31.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878f4a1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87881184e0 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878e49f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f87880716e0 0x7f8788117dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41572/0 (socket says 172.21.15.83:41572) 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f878e49f640 1 -- 172.21.15.83:0/3297218224 learned_addr learned my addr 172.21.15.83:0/3297218224 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f877effd640 10 client.?.objecter ms_handle_connect 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.637+0000 7f877effd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512907139 0 0) 0x7f87881184e0 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8774005660 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 416783584 0 0) 0x7f8774005660 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87881184e0 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8780002050 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 416879632 0 0) 0x7f87881184e0 con 0x7f87880716e0 2024-09-10T18:40:31.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8788118670 con 0x7f87880716e0 2024-09-10T18:40:31.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8788118940 con 0x7f87880716e0 2024-09-10T18:40:31.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f878f4a1640 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:31.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f878811ada0 con 0x7f87880716e0 2024-09-10T18:40:31.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f878f4a1640 1 librados: init done 2024-09-10T18:40:31.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8780002430 con 0x7f87880716e0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f8780003630 con 0x7f87880716e0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f8780003870 con 0x7f87880716e0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876c005010 con 0x7f87880716e0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7f8780047900 con 0x7f87880716e0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 10 client.14178.objecter ms_dispatch 0x7f8788000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:40:31.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 20 client.14178.objecter dump_active .. 0 homeless 2024-09-10T18:40:31.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.641+0000 7f877effd640 10 client.14178.objecter ms_handle_connect 0x7f877403c0b0 2024-09-10T18:40:31.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.645+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8780016b30 con 0x7f87880716e0 2024-09-10T18:40:31.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.829+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f876c000b90 con 0x7f877403c0b0 2024-09-10T18:40:31.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.829+0000 7f877effd640 1 -- 172.21.15.83:0/3297218224 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (unknown 0 0 1005664802) 0x7f876c000b90 con 0x7f877403c0b0 2024-09-10T18:40:31.834 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:40:31.835 INFO:teuthology.orchestra.run.smithi083.stdout:[{"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-09-10T18:40:31.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 10 librados: watch_flush enter 2024-09-10T18:40:31.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 10 librados: watch_flush exit 2024-09-10T18:40:31.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 20 client.14178.objecter shutdown clearing up homeless session... 2024-09-10T18:40:31.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 10 client.14178.objecter successfully canceled tick 2024-09-10T18:40:31.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 >> v1:172.21.15.83:6800/2630800463 conn(0x7f877403c0b0 legacy=0x7f877403e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 >> v1:172.21.15.83:6789/0 conn(0x7f87880716e0 legacy=0x7f8788117dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 shutdown_connections 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 >> 172.21.15.83:0/3297218224 conn(0x7f87880f7b90 msgr2=0x7f87880f9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 shutdown_connections 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 -- 172.21.15.83:0/3297218224 wait complete. 2024-09-10T18:40:31.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:31.833+0000 7f878f4a1640 1 librados: shutdown 2024-09-10T18:40:32.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:31 smithi083 bash[18629]: cluster 2024-09-10T18:40:30.362489+0000 mgr.y (mgr.14150) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:32.544 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-09-10T18:40:32.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd crush tunables jewel 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.833135+0000 mgr.y (mgr.14150) 33 : audit [DBG] from='client.14178 v1:172.21.15.83:0/3297218224' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.964161+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.971960+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.980329+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.987528+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.988774+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.990407+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:31.991721+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: cephadm 2024-09-10T18:40:31.993604+0000 mgr.y (mgr.14150) 34 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:32.540735+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:32.548658+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:32 smithi083 bash[18629]: audit 2024-09-10T18:40:32.556136+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:34.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:33 smithi083 bash[18629]: cephadm 2024-09-10T18:40:32.123076+0000 mgr.y (mgr.14150) 35 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:34.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:33 smithi083 bash[18629]: cephadm 2024-09-10T18:40:32.255820+0000 mgr.y (mgr.14150) 36 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:40:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:33 smithi083 bash[18629]: cluster 2024-09-10T18:40:32.362840+0000 mgr.y (mgr.14150) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:33 smithi083 bash[18629]: cephadm 2024-09-10T18:40:32.396876+0000 mgr.y (mgr.14150) 38 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:40:36.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:35 smithi083 bash[18629]: cluster 2024-09-10T18:40:34.363255+0000 mgr.y (mgr.14150) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:37.253 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:40:38.269 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:37 smithi083 bash[18629]: cluster 2024-09-10T18:40:36.363700+0000 mgr.y (mgr.14150) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- 172.21.15.83:0/703751623 >> v1:172.21.15.83:6789/0 conn(0x7fbd60102120 legacy=0x7fbd601024f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- 172.21.15.83:0/703751623 shutdown_connections 2024-09-10T18:40:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- 172.21.15.83:0/703751623 >> 172.21.15.83:0/703751623 conn(0x7fbd600f7cd0 msgr2=0x7fbd600fa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- 172.21.15.83:0/703751623 shutdown_connections 2024-09-10T18:40:40.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- 172.21.15.83:0/703751623 wait complete. 2024-09-10T18:40:40.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 librados: starting msgr at 2024-09-10T18:40:40.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 librados: starting objecter 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 Processor -- start 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- start start 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 librados: setting wanted keys 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 librados: calling monclient init 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd654ca640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd60070120 con 0x7fbd60102120 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd5effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbd60102120 0x7fbd6006fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39568/0 (socket says 172.21.15.83:39568) 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd5effd640 1 -- 172.21.15.83:0/2713813847 learned_addr learned my addr 172.21.15.83:0/2713813847 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 10 client.?.objecter ms_handle_connect 0x7fbd60102120 2024-09-10T18:40:40.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955967817 0 0) 0x7fbd60070120 con 0x7fbd60102120 2024-09-10T18:40:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd44005660 con 0x7fbd60102120 2024-09-10T18:40:40.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026916527 0 0) 0x7fbd44005660 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd60070120 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.237+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbd50001550 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434681597 0 0) 0x7fbd60070120 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd600702b0 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60070560 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd654ca640 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd60116b00 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd654ca640 1 librados: init done 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fbd50002790 con 0x7fbd60102120 2024-09-10T18:40:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7fbd50003630 con 0x7fbd60102120 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7fbd50002330 con 0x7fbd60102120 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (unknown 1020018565 0 0) 0x7fbd500479c0 con 0x7fbd60102120 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 10 client.14180.objecter ms_dispatch 0x7fbd60000b90 osd_map(3..3 src has 1..3) v4 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 20 client.14180.objecter dump_active .. 0 homeless 2024-09-10T18:40:40.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd48005010 con 0x7fbd60102120 2024-09-10T18:40:40.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.241+0000 7fbd577fe640 10 client.14180.objecter ms_handle_connect 0x7fbd4403c150 2024-09-10T18:40:40.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.249+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fbd50015f40 con 0x7fbd60102120 2024-09-10T18:40:40.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:39 smithi083 bash[18629]: cluster 2024-09-10T18:40:38.364118+0000 mgr.y (mgr.14150) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:40.437+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7fbd480052c0 con 0x7fbd60102120 2024-09-10T18:40:41.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.329+0000 7fbd577fe640 1 -- 172.21.15.83:0/2713813847 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (unknown 2099115640 0 0) 0x7fbd50015960 con 0x7fbd60102120 2024-09-10T18:40:41.334 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to jewel 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 10 librados: watch_flush enter 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 10 librados: watch_flush exit 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 20 client.14180.objecter shutdown clearing up homeless session... 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 10 client.14180.objecter successfully canceled tick 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 >> v1:172.21.15.83:6800/2630800463 conn(0x7fbd4403c150 legacy=0x7fbd4403e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 >> v1:172.21.15.83:6789/0 conn(0x7fbd60102120 legacy=0x7fbd6006fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 shutdown_connections 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 >> 172.21.15.83:0/2713813847 conn(0x7fbd600f7cd0 msgr2=0x7fbd600fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 shutdown_connections 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 -- 172.21.15.83:0/2713813847 wait complete. 2024-09-10T18:40:41.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:40:41.333+0000 7fbd654ca640 1 librados: shutdown 2024-09-10T18:40:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:41 smithi083 bash[18629]: cluster 2024-09-10T18:40:40.364452+0000 mgr.y (mgr.14150) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:41 smithi083 bash[18629]: audit 2024-09-10T18:40:40.438811+0000 mon.a (mon.0) 126 : audit [INF] from='client.? v1:172.21.15.83:0/2713813847' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-09-10T18:40:41.972 INFO:tasks.cephadm:Adding mon.a on smithi083 2024-09-10T18:40:41.973 INFO:tasks.cephadm:Adding mon.c on smithi083 2024-09-10T18:40:41.973 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-09-10T18:40:41.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b' 2024-09-10T18:40:42.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:42 smithi083 bash[18629]: audit 2024-09-10T18:40:41.333584+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.83:0/2713813847' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-09-10T18:40:42.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:42 smithi083 bash[18629]: cluster 2024-09-10T18:40:41.333701+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:40:43.206 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:43 smithi083 bash[18629]: cluster 2024-09-10T18:40:42.364841+0000 mgr.y (mgr.14150) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:45.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:45 smithi083 bash[18629]: cluster 2024-09-10T18:40:44.365229+0000 mgr.y (mgr.14150) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:46.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- 172.21.15.165:0/2308248392 >> v1:172.21.15.83:6789/0 conn(0x7f0a601020f0 legacy=0x7f0a601024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:46.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- 172.21.15.165:0/2308248392 shutdown_connections 2024-09-10T18:40:46.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- 172.21.15.165:0/2308248392 >> 172.21.15.165:0/2308248392 conn(0x7f0a600f7ca0 msgr2=0x7f0a600fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:46.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- 172.21.15.165:0/2308248392 shutdown_connections 2024-09-10T18:40:46.205 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- 172.21.15.165:0/2308248392 wait complete. 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 librados: starting msgr at 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 librados: starting objecter 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 Processor -- start 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- start start 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 librados: setting wanted keys 2024-09-10T18:40:46.206 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 librados: calling monclient init 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a67a5f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a60111eb0 con 0x7f0a601020f0 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a657fb640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0a601020f0 0x7f0a601117a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.165:43792/0 (socket says 172.21.15.165:43792) 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a657fb640 1 -- 172.21.15.165:0/3455832752 learned_addr learned my addr 172.21.15.165:0/3455832752 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 10 client.?.objecter ms_handle_connect 0x7f0a601020f0 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356713816 0 0) 0x7f0a60111eb0 con 0x7f0a601020f0 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a4c005660 con 0x7f0a601020f0 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 376628872 0 0) 0x7f0a4c005660 con 0x7f0a601020f0 2024-09-10T18:40:46.207 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a60111eb0 con 0x7f0a601020f0 2024-09-10T18:40:46.208 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.198+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f0a580019c0 con 0x7f0a601020f0 2024-09-10T18:40:46.208 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4261804379 0 0) 0x7f0a60111eb0 con 0x7f0a601020f0 2024-09-10T18:40:46.208 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60112040 con 0x7f0a601020f0 2024-09-10T18:40:46.208 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a60112310 con 0x7f0a601020f0 2024-09-10T18:40:46.208 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a67a5f640 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:46.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a601147c0 con 0x7f0a601020f0 2024-09-10T18:40:46.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a67a5f640 1 librados: init done 2024-09-10T18:40:46.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0a580026d0 con 0x7f0a601020f0 2024-09-10T18:40:46.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f0a580035b0 con 0x7f0a601020f0 2024-09-10T18:40:46.209 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a40005010 con 0x7f0a601020f0 2024-09-10T18:40:46.210 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f0a58003730 con 0x7f0a601020f0 2024-09-10T18:40:46.210 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7f0a580478a0 con 0x7f0a601020f0 2024-09-10T18:40:46.210 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 10 client.14182.objecter ms_dispatch 0x7f0a60000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:40:46.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 3 client.14182.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:40:46.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 3 client.14182.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:40:46.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.202+0000 7f0a5dffb640 20 client.14182.objecter dump_active .. 0 homeless 2024-09-10T18:40:46.211 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.206+0000 7f0a5dffb640 10 client.14182.objecter ms_handle_connect 0x7f0a4c03c220 2024-09-10T18:40:46.221 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.214+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0a58003ad0 con 0x7f0a601020f0 2024-09-10T18:40:46.411 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.402+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a40000c60 con 0x7f0a4c03c220 2024-09-10T18:40:46.427 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a5dffb640 1 -- 172.21.15.165:0/3455832752 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f0a40000c60 con 0x7f0a4c03c220 2024-09-10T18:40:46.427 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-09-10T18:40:46.429 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 10 librados: watch_flush enter 2024-09-10T18:40:46.429 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 10 librados: watch_flush exit 2024-09-10T18:40:46.429 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 20 client.14182.objecter shutdown clearing up homeless session... 2024-09-10T18:40:46.429 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 10 client.14182.objecter successfully canceled tick 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 >> v1:172.21.15.83:6800/2630800463 conn(0x7f0a4c03c220 legacy=0x7f0a4c03e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 >> v1:172.21.15.83:6789/0 conn(0x7f0a601020f0 legacy=0x7f0a601117a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 shutdown_connections 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 >> 172.21.15.165:0/3455832752 conn(0x7f0a600f7ca0 msgr2=0x7f0a600fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 shutdown_connections 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 -- 172.21.15.165:0/3455832752 wait complete. 2024-09-10T18:40:46.430 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:46.422+0000 7f0a67a5f640 1 librados: shutdown 2024-09-10T18:40:47.231 DEBUG:teuthology.orchestra.run.smithi083:mon.c> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.c.service 2024-09-10T18:40:47.233 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.b.service 2024-09-10T18:40:47.235 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:40:47.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:40:47.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: cluster 2024-09-10T18:40:46.365677+0000 mgr.y (mgr.14150) 45 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.413493+0000 mgr.y (mgr.14150) 46 : audit [DBG] from='client.14182 v1:172.21.15.165:0/3455832752' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: cephadm 2024-09-10T18:40:46.418858+0000 mgr.y (mgr.14150) 47 : cephadm [INF] Saving service mon spec with placement smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b;count:3 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.426494+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.427838+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.431139+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.432722+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.440017+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.443240+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: audit 2024-09-10T18:40:46.444596+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:47 smithi083 bash[18629]: cephadm 2024-09-10T18:40:46.446275+0000 mgr.y (mgr.14150) 48 : cephadm [INF] Deploying daemon mon.b on smithi165 2024-09-10T18:40:48.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:40:48.515 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:49 smithi083 bash[18629]: cluster 2024-09-10T18:40:48.366108+0000 mgr.y (mgr.14150) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:51 smithi083 bash[18629]: cluster 2024-09-10T18:40:50.366535+0000 mgr.y (mgr.14150) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 -- 172.21.15.165:0/1601249517 >> v1:172.21.15.83:6789/0 conn(0x7f4cb406d960 legacy=0x7f4cb406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 -- 172.21.15.165:0/1601249517 shutdown_connections 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 -- 172.21.15.165:0/1601249517 >> 172.21.15.165:0/1601249517 conn(0x7f4cb4067660 msgr2=0x7f4cb4067a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 -- 172.21.15.165:0/1601249517 shutdown_connections 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 -- 172.21.15.165:0/1601249517 wait complete. 2024-09-10T18:40:52.791 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 librados: starting msgr at 2024-09-10T18:40:52.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 librados: starting objecter 2024-09-10T18:40:52.792 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.782+0000 7f4cbc091640 1 Processor -- start 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 -- start start 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 librados: setting wanted keys 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 librados: calling monclient init 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4cb4120e50 con 0x7f4cb406d960 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cb9e2d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4cb406d960 0x7f4cb4120740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.165:43812/0 (socket says 172.21.15.165:43812) 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cb9e2d640 1 -- 172.21.15.165:0/3366169319 learned_addr learned my addr 172.21.15.165:0/3366169319 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 10 client.?.objecter ms_handle_connect 0x7f4cb406d960 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732611787 0 0) 0x7f4cb4120e50 con 0x7f4cb406d960 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca0005660 con 0x7f4cb406d960 2024-09-10T18:40:52.793 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237329814 0 0) 0x7f4ca0005660 con 0x7f4cb406d960 2024-09-10T18:40:52.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cb4120e50 con 0x7f4cb406d960 2024-09-10T18:40:52.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4ca4002050 con 0x7f4cb406d960 2024-09-10T18:40:52.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1693885159 0 0) 0x7f4cb4120e50 con 0x7f4cb406d960 2024-09-10T18:40:52.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4120fe0 con 0x7f4cb406d960 2024-09-10T18:40:52.794 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb41212b0 con 0x7f4cb406d960 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb41236e0 con 0x7f4cb406d960 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4cbc091640 1 librados: init done 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4ca4002430 con 0x7f4cb406d960 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 572405882 0 0) 0x7f4ca4003580 con 0x7f4cb406d960 2024-09-10T18:40:52.795 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.786+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f4ca4001880 con 0x7f4cb406d960 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c98005010 con 0x7f4cb406d960 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7f4ca4047880 con 0x7f4cb406d960 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 10 client.14184.objecter ms_dispatch 0x7f4cb4000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:40:52.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 20 client.14184.objecter dump_active .. 0 homeless 2024-09-10T18:40:52.797 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.790+0000 7f4caa7fc640 10 client.14184.objecter ms_handle_connect 0x7f4ca003c1d0 2024-09-10T18:40:52.805 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:52.798+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4ca40028b0 con 0x7f4cb406d960 2024-09-10T18:40:53.071 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.062+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c980052c0 con 0x7f4cb406d960 2024-09-10T18:40:53.072 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4caa7fc640 1 -- 172.21.15.165:0/3366169319 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+680 (unknown 2237029548 0 2925643384) 0x7f4ca40158c0 con 0x7f4cb406d960 2024-09-10T18:40:53.072 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-09-10T18:40:53.072 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T18:40:53.072 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","modified":"2024-09-10T18:38:33.730216Z","created":"2024-09-10T18:38:33.730216Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T18:40:53.074 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 10 librados: watch_flush enter 2024-09-10T18:40:53.074 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 10 librados: watch_flush exit 2024-09-10T18:40:53.074 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 20 client.14184.objecter shutdown clearing up homeless session... 2024-09-10T18:40:53.074 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 10 client.14184.objecter successfully canceled tick 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 >> v1:172.21.15.83:6800/2630800463 conn(0x7f4ca003c1d0 legacy=0x7f4ca003e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 >> v1:172.21.15.83:6789/0 conn(0x7f4cb406d960 legacy=0x7f4cb4120740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 shutdown_connections 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 >> 172.21.15.165:0/3366169319 conn(0x7f4cb4067660 msgr2=0x7f4cb406b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 shutdown_connections 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 -- 172.21.15.165:0/3366169319 wait complete. 2024-09-10T18:40:53.075 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:40:53.066+0000 7f4cbc091640 1 librados: shutdown 2024-09-10T18:40:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: cluster 2024-09-10T18:40:52.366944+0000 mgr.y (mgr.14150) 51 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.072014+0000 mon.a (mon.0) 136 : audit [DBG] from='client.? v1:172.21.15.165:0/3366169319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:40:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.253671+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.259378+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.264360+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.265255+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:40:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:53 smithi083 bash[18629]: audit 2024-09-10T18:40:53.266577+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:54.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:54 smithi083 bash[18629]: cephadm 2024-09-10T18:40:53.268130+0000 mgr.y (mgr.14150) 52 : cephadm [INF] Deploying daemon mon.c on smithi083 2024-09-10T18:40:54.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:54 smithi165 bash[22591]: debug 2024-09-10T18:40:54.626+0000 7f6f05893640 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T18:40:55.117 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:40:55.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:40:58.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:57 smithi083 systemd[1]: Started Ceph mon.c for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:40:58.844 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:40:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.828+0000 7f143e069900 0 set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:40:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.828+0000 7f143e069900 0 ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.828+0000 7f143e069900 0 pidfile_write: ignore empty --pid-file 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.832+0000 7f143e069900 0 load: jerasure load: lrc 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: RocksDB version: 6.15.5 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Compile date Sep 9 2024 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: DB SUMMARY 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: DB Session ID: ZPF08XTMPIF4LA2G82CV 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: CURRENT file: CURRENT 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-10T18:40:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 476 ; 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.error_if_exists: 0 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.create_if_missing: 0 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.paranoid_checks: 1 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.env: 0x561100de4440 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.fs: Posix File System 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.info_log: 0x56110205ce60 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.statistics: (nil) 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.use_fsync: 0 2024-09-10T18:40:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_log_file_size: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_fallocate: 1 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.use_direct_reads: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:40:59.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.db_log_dir: 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.write_buffer_manager: 0x56110211a7e0 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:40:59.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:40:59.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:40:59.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:40:59.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:40:59.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:40:59.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:40:59.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:40:59.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.unordered_write: 0 2024-09-10T18:40:59.048 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.row_cache: None 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.wal_filter: None 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.preserve_deletes: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.two_write_queues: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.atomic_flush: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:40:59.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.log_readahead_size: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_background_jobs: 2 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_background_compactions: -1 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_subcompactions: 1 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:40:59.050 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_open_files: -1 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:40:59.051 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_background_flushes: -1 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Compression algorithms supported: 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kZSTD supported: 0 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kXpressCompression supported: 0 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kBZip2Compression supported: 0 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kLZ4Compression supported: 1 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kZlibCompression supported: 1 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: kSnappyCompression supported: 1 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:40:59.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.merge_operator: 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_filter: None 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561102005a38) 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cache_index_and_filter_blocks: 1 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:40:59.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: pin_top_level_index_and_filter: 1 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: index_type: 0 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: data_block_index_type: 0 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: index_shortening: 1 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: hash_index_allow_collision: 1 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: checksum: 1 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: no_block_cache: 0 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_cache: 0x561102039350 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_cache_name: BinnedLRUCache 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_cache_options: 2024-09-10T18:40:59.054 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: capacity : 536870912 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: num_shard_bits : 4 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: strict_capacity_limit : 0 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: high_pri_pool_ratio: 0.000 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_cache_compressed: (nil) 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: persistent_cache: (nil) 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_size: 4096 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_size_deviation: 10 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_restart_interval: 16 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: index_block_restart_interval: 1 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: metadata_block_size: 4096 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: partition_filters: 0 2024-09-10T18:40:59.055 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: use_delta_encoding: 1 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: whole_key_filtering: 1 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: verify_compression: 0 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: read_amp_bytes_per_bit: 0 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: format_version: 4 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: enable_index_compression: 1 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: block_align: 0 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression: NoCompression 2024-09-10T18:40:59.056 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.num_levels: 7 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:40:59.057 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:40:59.058 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:40:59.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:40:59.060 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.inplace_update_support: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.bloom_locality: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.max_successive_merges: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.ttl: 2592000 2024-09-10T18:40:59.061 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.enable_blob_files: false 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.min_blob_size: 0 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658840543, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 3 rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:40:59.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658840967, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1536, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725993658, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "d9f7f889-4572-41bb-85e4-0fe44ba1133c", "db_session_id": "ZPF08XTMPIF4LA2G82CV"}} 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658841828, "job": 1, "event": "recovery_finished"} 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.836+0000 7f143e069900 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f143e069900 4 rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x56110206e700 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f143e069900 4 rocksdb: DB pointer 0x56110203c000 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f1433e35640 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f1433e35640 4 rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** DB Stats ** 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:40:59.063 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** Compaction Stats [default] ** 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: L0 1/0 1.50 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Sum 1/0 1.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** Compaction Stats [default] ** 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.064 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Cumulative compaction: 0.00 GB write, 0.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Interval compaction: 0.00 GB write, 0.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** Compaction Stats [default] ** 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: L0 1/0 1.50 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.065 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Sum 1/0 1.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** Compaction Stats [default] ** 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Cumulative compaction: 0.00 GB write, 0.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:40:59.066 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f143e069900 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.840+0000 7f143e069900 0 using public_addrv v1:172.21.15.83:6790/0 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.844+0000 7f143e069900 0 starting mon.c rank -1 at public addrs v1:172.21.15.83:6790/0 at bind addrs v1:172.21.15.83:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.848+0000 7f143e069900 1 mon.c@-1(???) e0 preinit fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f1432632640 4 rocksdb: [db/db_impl/db_impl_write.cc:1736] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f143b644640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.869282) [db/db_impl/db_impl_compaction_flush.cc:2394] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f143b644640 4 rocksdb: [db/flush_job.cc:335] [default] [JOB 3] Flushing memtable with next log file: 13 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f143b644640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658869354, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1173, "num_deletes": 20, "total_data_size": 2242162, "memory_usage": 2263424, "flush_reason": "Manual Compaction"} 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f143b644640 4 rocksdb: [db/flush_job.cc:364] [default] [JOB 3] Level-0 flush table #14: started 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f1436e3b640 0 mon.c@-1(synchronizing).mds e1 new map 2024-09-10T18:40:59.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f1436e3b640 0 mon.c@-1(synchronizing).mds e1 print_map 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: e1 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: legacy client fscid: -1 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: No filesystems configured 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.864+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T18:40:59.068 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.868+0000 7f1436e3b640 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:00.355555+0000 mgr.y (mgr.14150) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:02.355986+0000 mgr.y (mgr.14150) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:03.063158+0000 mon.a (mon.0) 101 : audit [INF] from='client.? v1:172.21.15.83:0/2847418047' entity='client.admin' 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:03.078823+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:03.085084+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:04.356535+0000 mgr.y (mgr.14150) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:06.357013+0000 mgr.y (mgr.14150) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:08.357512+0000 mgr.y (mgr.14150) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:10.357937+0000 mgr.y (mgr.14150) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:10.643775+0000 mgr.y (mgr.14150) 15 : audit [DBG] from='client.14174 v1:172.21.15.83:0/7959724' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:59.069 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:10.651445+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:10.652524+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:10.655687+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:10.657385+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:10.659662+0000 mgr.y (mgr.14150) 16 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:10.765581+0000 mgr.y (mgr.14150) 17 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:10.877011+0000 mgr.y (mgr.14150) 18 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:11.167539+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:11.172043+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:11.179398+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:11.042158+0000 mgr.y (mgr.14150) 19 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:12.358406+0000 mgr.y (mgr.14150) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:14.358861+0000 mgr.y (mgr.14150) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.070 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:16.359329+0000 mgr.y (mgr.14150) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:18.359758+0000 mgr.y (mgr.14150) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:19.306682+0000 mgr.y (mgr.14150) 24 : audit [DBG] from='client.14176 v1:172.21.15.83:0/3449687605' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:20.360121+0000 mgr.y (mgr.14150) 25 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:20.555395+0000 mgr.y (mgr.14150) 26 : cephadm [INF] Deploying cephadm binary to smithi165 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:22.360509+0000 mgr.y (mgr.14150) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:23.190438+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:23.191926+0000 mon.a (mon.0) 112 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:23.487948+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:23.191158+0000 mgr.y (mgr.14150) 28 : cephadm [INF] Added host smithi165 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:24.361014+0000 mgr.y (mgr.14150) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:24.781975+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:25.461435+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.071 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:26.361447+0000 mgr.y (mgr.14150) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:28.361957+0000 mgr.y (mgr.14150) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:30.362489+0000 mgr.y (mgr.14150) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.833135+0000 mgr.y (mgr.14150) 33 : audit [DBG] from='client.14178 v1:172.21.15.83:0/3297218224' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.964161+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.971960+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.980329+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.987528+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.988774+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.990407+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:31.991721+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:31.993604+0000 mgr.y (mgr.14150) 34 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:40:59.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:32.540735+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:32.548658+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:32.556136+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:32.123076+0000 mgr.y (mgr.14150) 35 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:32.255820+0000 mgr.y (mgr.14150) 36 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:32.362840+0000 mgr.y (mgr.14150) 37 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:32.396876+0000 mgr.y (mgr.14150) 38 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:34.363255+0000 mgr.y (mgr.14150) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:36.363700+0000 mgr.y (mgr.14150) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:38.364118+0000 mgr.y (mgr.14150) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:40.364452+0000 mgr.y (mgr.14150) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:40.438811+0000 mon.a (mon.0) 126 : audit [INF] from='client.? v1:172.21.15.83:0/2713813847' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:41.333584+0000 mon.a (mon.0) 127 : audit [INF] from='client.? v1:172.21.15.83:0/2713813847' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-09-10T18:40:59.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:41.333701+0000 mon.a (mon.0) 128 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:42.364841+0000 mgr.y (mgr.14150) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:44.365229+0000 mgr.y (mgr.14150) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:46.365677+0000 mgr.y (mgr.14150) 45 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.413493+0000 mgr.y (mgr.14150) 46 : audit [DBG] from='client.14182 v1:172.21.15.165:0/3455832752' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:46.418858+0000 mgr.y (mgr.14150) 47 : cephadm [INF] Saving service mon spec with placement smithi083:[v1:172.21.15.83:6789]=a;smithi083:[v1:172.21.15.83:6790]=c;smithi165:[v1:172.21.15.165:6789]=b;count:3 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.426494+0000 mon.a (mon.0) 129 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.427838+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.431139+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.432722+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.440017+0000 mon.a (mon.0) 133 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.443240+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:46.444596+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:59.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:46.446275+0000 mgr.y (mgr.14150) 48 : cephadm [INF] Deploying daemon mon.b on smithi165 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:48.366108+0000 mgr.y (mgr.14150) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:50.366535+0000 mgr.y (mgr.14150) 50 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cluster 2024-09-10T18:40:52.366944+0000 mgr.y (mgr.14150) 51 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.072014+0000 mon.a (mon.0) 136 : audit [DBG] from='client.? v1:172.21.15.165:0/3366169319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.253671+0000 mon.a (mon.0) 137 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.259378+0000 mon.a (mon.0) 138 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.264360+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.265255+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: audit 2024-09-10T18:40:53.266577+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: cephadm 2024-09-10T18:40:53.268130+0000 mgr.y (mgr.14150) 52 : cephadm [INF] Deploying daemon mon.c on smithi083 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f1436e3b640 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658876864, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1981006, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 1975204, "index_size": 2572, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2245, "raw_key_size": 25539, "raw_average_key_size": 29, "raw_value_size": 1957490, "raw_average_value_size": 2286, "num_data_blocks": 121, "num_entries": 856, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725993658, "oldest_key_time": 1725993658, "file_creation_time": 1725993658, "db_id": "d9f7f889-4572-41bb-85e4-0fe44ba1133c", "db_session_id": "ZPF08XTMPIF4LA2G82CV"}} 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: [db/flush_job.cc:424] [default] [JOB 3] Level-0 flush table #14: 1981006 bytes OK 2024-09-10T18:40:59.075 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.876927) [db/memtable_list.cc:449] [default] Level-0 commit table #14 started 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.877098) [db/memtable_list.cc:628] [default] Level-0 commit table #14: memtable #1 done 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.877118) EVENT_LOG_v1 {"time_micros": 1725993658877113, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.877135) [db/db_impl/db_impl_compaction_flush.cc:233] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: [db/db_impl/db_impl_files.cc:415] [JOB 3] Try to delete WAL files size 2233979, prev total WAL file size 2234202, number of live WAL files 2. 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143b644640 4 rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f1432632640 4 rocksdb: [db/db_impl/db_impl_compaction_flush.cc:1615] [default] Manual compaction starting 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143be45640 4 rocksdb: (Original Log Time 2024/09/10-18:40:58.877633) [db/db_impl/db_impl_compaction_flush.cc:2712] [default] Manual compaction from level-0 to level-6 from '6D6772737461740030' seq:72057594037927935, type:20 .. '6D677273746174003231' seq:0, type:0; will stop at (end) 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143be45640 4 rocksdb: [db/compaction/compaction_job.cc:1881] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143be45640 4 rocksdb: [db/compaction/compaction_job.cc:1887] [default] Compaction start summary: Base version 4 Base level 0, inputs: [14(1934KB) 8(1536B)] 2024-09-10T18:40:59.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:40:58 smithi083 bash[32381]: debug 2024-09-10T18:40:58.872+0000 7f143be45640 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1725993658877733, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 1982542} 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:54.367383+0000 mgr.y (mgr.14150) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:54.643951+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:54.644175+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:54.645822+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:55.639464+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:56.367793+0000 mgr.y (mgr.14150) 54 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:56.639733+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:56.639927+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:57.639096+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:58.368310+0000 mgr.y (mgr.14150) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:58.639858+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:58.883502+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:59.640074+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.665767+0000 mon.a (mon.0) 152 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.673458+0000 mon.a (mon.0) 153 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.674612+0000 mon.a (mon.0) 154 : cluster [DBG] fsmap 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.674646+0000 mon.a (mon.0) 155 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.674826+0000 mon.a (mon.0) 156 : cluster [DBG] mgrmap e13: y(active, since 79s) 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: cluster 2024-09-10T18:40:59.674971+0000 mon.a (mon.0) 157 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:59.680971+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:59.689156+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:59.696425+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:40:59 smithi083 bash[18629]: audit 2024-09-10T18:40:59.704067+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:54.367383+0000 mgr.y (mgr.14150) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:54.643951+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:54.644175+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:54.645822+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:55.639464+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:56.367793+0000 mgr.y (mgr.14150) 54 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:56.639733+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:56.639927+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:57.639096+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:58.368310+0000 mgr.y (mgr.14150) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:58.639858+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:58.883502+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:59.640074+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.665767+0000 mon.a (mon.0) 152 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.673458+0000 mon.a (mon.0) 153 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.674612+0000 mon.a (mon.0) 154 : cluster [DBG] fsmap 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.674646+0000 mon.a (mon.0) 155 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.674826+0000 mon.a (mon.0) 156 : cluster [DBG] mgrmap e13: y(active, since 79s) 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: cluster 2024-09-10T18:40:59.674971+0000 mon.a (mon.0) 157 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:00.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:59.680971+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:59.689156+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:59.696425+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:00.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:40:59 smithi165 bash[22591]: audit 2024-09-10T18:40:59.704067+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:01.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:00 smithi083 bash[18629]: audit 2024-09-10T18:40:59.748098+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:01.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:00 smithi083 bash[18629]: audit 2024-09-10T18:40:59.882559+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:01.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:00 smithi083 bash[18629]: audit 2024-09-10T18:41:00.419283+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:00 smithi083 bash[18629]: audit 2024-09-10T18:41:00.640001+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:01.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:00 smithi165 bash[22591]: audit 2024-09-10T18:40:59.748098+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:00 smithi165 bash[22591]: audit 2024-09-10T18:40:59.882559+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:00 smithi165 bash[22591]: audit 2024-09-10T18:41:00.419283+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:00 smithi165 bash[22591]: audit 2024-09-10T18:41:00.640001+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:05.933 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- 172.21.15.165:0/4215249055 >> v1:172.21.15.83:6789/0 conn(0x7f2990003520 legacy=0x7f29900059a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:05.933 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- 172.21.15.165:0/4215249055 shutdown_connections 2024-09-10T18:41:05.933 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- 172.21.15.165:0/4215249055 >> 172.21.15.165:0/4215249055 conn(0x7f29ac071150 msgr2=0x7f29ac071550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:05.933 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- 172.21.15.165:0/4215249055 shutdown_connections 2024-09-10T18:41:05.933 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- 172.21.15.165:0/4215249055 wait complete. 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 librados: starting msgr at 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 librados: starting objecter 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 Processor -- start 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- start start 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 librados: setting wanted keys 2024-09-10T18:41:05.934 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 librados: calling monclient init 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29ac11df20 con 0x7f29ac11a260 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29ac11f0c0 con 0x7f29ac114410 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29b1acb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f29ac1202c0 con 0x7f2990003520 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29abfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f29ac11a260 0x7f29ac11c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.165:49280/0 (socket says 172.21.15.165:49280) 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29aaffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f29ac114410 0x7f29ac118b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:42662/0 (socket says 172.21.15.165:42662) 2024-09-10T18:41:05.935 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29abfff640 1 -- 172.21.15.165:0/528064484 learned_addr learned my addr 172.21.15.165:0/528064484 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:41:05.936 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.926+0000 7f29ab7fe640 1 --1- 172.21.15.165:0/528064484 >> v1:172.21.15.83:6790/0 conn(0x7f2990003520 0x7f29ac113d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.165:41178/0 (socket says 172.21.15.165:41178) 2024-09-10T18:41:05.936 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter ms_handle_connect 0x7f29ac11a260 2024-09-10T18:41:05.936 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter ms_handle_connect 0x7f29ac114410 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter ms_handle_connect 0x7f2990003520 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422304424 0 0) 0x7f29ac11df20 con 0x7f29ac11a260 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f298c003b80 con 0x7f29ac11a260 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417234830 0 0) 0x7f298c003b80 con 0x7f29ac11a260 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29ac11df20 con 0x7f29ac11a260 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f299c001b80 con 0x7f29ac11a260 2024-09-10T18:41:05.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2227961587 0 0) 0x7f29ac11df20 con 0x7f29ac11a260 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 >> v1:172.21.15.83:6790/0 conn(0x7f2990003520 legacy=0x7f29ac113d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 >> v1:172.21.15.165:6789/0 conn(0x7f29ac114410 legacy=0x7f29ac118b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac1214c0 con 0x7f29ac11a260 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ac1204b0 con 0x7f29ac11a260 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f299c002150 con 0x7f29ac11a260 2024-09-10T18:41:05.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f299c005df0 con 0x7f29ac11a260 2024-09-10T18:41:05.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29b1acb640 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:41:05.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ac120940 con 0x7f29ac11a260 2024-09-10T18:41:05.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29b1acb640 1 librados: init done 2024-09-10T18:41:05.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.930+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f299c011260 con 0x7f29ac11a260 2024-09-10T18:41:05.940 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ac0fb780 con 0x7f29ac11a260 2024-09-10T18:41:05.940 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 10 client.14190.objecter ms_handle_connect 0x7f298c03c260 2024-09-10T18:41:05.941 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7f299c0167a0 con 0x7f29ac11a260 2024-09-10T18:41:05.941 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 10 client.14190.objecter ms_dispatch 0x7f29ac000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:41:05.941 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:41:05.941 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:41:05.941 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.934+0000 7f29a3fff640 20 client.14190.objecter dump_active .. 0 homeless 2024-09-10T18:41:05.951 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:05.942+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f299c016a00 con 0x7f29ac11a260 2024-09-10T18:41:06.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.186+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29ac05ed70 con 0x7f29ac11a260 2024-09-10T18:41:06.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.186+0000 7f29a3fff640 1 -- 172.21.15.165:0/528064484 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1113 (unknown 409899127 0 4169750816) 0x7f299c016400 con 0x7f29ac11a260 2024-09-10T18:41:06.194 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-09-10T18:41:06.194 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T18:41:06.194 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","modified":"2024-09-10T18:41:00.885365Z","created":"2024-09-10T18:38:33.730216Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6790","nonce":0}]},"addr":"172.21.15.83:6790/0","public_addr":"172.21.15.83:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-10T18:41:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 10 librados: watch_flush enter 2024-09-10T18:41:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 10 librados: watch_flush exit 2024-09-10T18:41:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 20 client.14190.objecter shutdown clearing up homeless session... 2024-09-10T18:41:06.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 10 client.14190.objecter successfully canceled tick 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 >> v1:172.21.15.83:6800/2630800463 conn(0x7f298c03c260 legacy=0x7f298c03e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 >> v1:172.21.15.83:6789/0 conn(0x7f29ac11a260 legacy=0x7f29ac11c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 shutdown_connections 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 >> 172.21.15.165:0/528064484 conn(0x7f29ac071150 msgr2=0x7f29ac0f9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 shutdown_connections 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 -- 172.21.15.165:0/528064484 wait complete. 2024-09-10T18:41:06.197 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:06.190+0000 7f29b1acb640 1 librados: shutdown 2024-09-10T18:41:06.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:00.368789+0000 mgr.y (mgr.14150) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:00.891516+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:06.232 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:00.891922+0000 mon.a (mon.0) 168 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:00.892112+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:00.894415+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:00.894898+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:01.883040+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:02.369207+0000 mgr.y (mgr.14150) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:02.883281+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:02.887008+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:03.883293+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:04.369624+0000 mgr.y (mgr.14150) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:04.883549+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:05.883572+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.917957+0000 mon.a (mon.0) 176 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.925138+0000 mon.a (mon.0) 177 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0,c=v1:172.21.15.83:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.926442+0000 mon.a (mon.0) 178 : cluster [DBG] fsmap 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.926533+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.926792+0000 mon.a (mon.0) 180 : cluster [DBG] mgrmap e13: y(active, since 85s) 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: cluster 2024-09-10T18:41:05.927012+0000 mon.a (mon.0) 181 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:05.933667+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:05.940686+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:05.953871+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:05 smithi165 bash[22591]: audit 2024-09-10T18:41:05.958376+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:00.368789+0000 mgr.y (mgr.14150) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:00.891516+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:00.891922+0000 mon.a (mon.0) 168 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:00.892112+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:00.894415+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:00.894898+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:01.883040+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:02.369207+0000 mgr.y (mgr.14150) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:02.883281+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:02.887008+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:03.883293+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:04.369624+0000 mgr.y (mgr.14150) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:04.883549+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:05.883572+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.917957+0000 mon.a (mon.0) 176 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:41:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.925138+0000 mon.a (mon.0) 177 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0,c=v1:172.21.15.83:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.926442+0000 mon.a (mon.0) 178 : cluster [DBG] fsmap 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.926533+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.926792+0000 mon.a (mon.0) 180 : cluster [DBG] mgrmap e13: y(active, since 85s) 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: cluster 2024-09-10T18:41:05.927012+0000 mon.a (mon.0) 181 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:05.933667+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:05.940686+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:54.367383+0000 mgr.y (mgr.14150) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:54.643951+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:54.644175+0000 mon.a (mon.0) 144 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:54.645822+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:55.639464+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:56.367793+0000 mgr.y (mgr.14150) 54 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:56.639733+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:56.639927+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:57.639096+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:58.368310+0000 mgr.y (mgr.14150) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:58.639858+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:58.883502+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.640074+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.665767+0000 mon.a (mon.0) 152 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.673458+0000 mon.a (mon.0) 153 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.674612+0000 mon.a (mon.0) 154 : cluster [DBG] fsmap 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.674646+0000 mon.a (mon.0) 155 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.674826+0000 mon.a (mon.0) 156 : cluster [DBG] mgrmap e13: y(active, since 79s) 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:40:59.674971+0000 mon.a (mon.0) 157 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.680971+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.689156+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.696425+0000 mon.a (mon.0) 160 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:05.953871+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[18629]: audit 2024-09-10T18:41:05.958376+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.704067+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.748098+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:40:59.882559+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:00.419283+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:00.640001+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:00.368789+0000 mgr.y (mgr.14150) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:00.891516+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:00.891922+0000 mon.a (mon.0) 168 : cluster [INF] mon.a calling monitor election 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:00.892112+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:00.894415+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:00.894898+0000 mon.a (mon.0) 170 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:01.883040+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:02.369207+0000 mgr.y (mgr.14150) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:02.883281+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:02.887008+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:03.883293+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:04.369624+0000 mgr.y (mgr.14150) 58 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:04.883549+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:05.883572+0000 mon.a (mon.0) 175 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.917957+0000 mon.a (mon.0) 176 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.925138+0000 mon.a (mon.0) 177 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.83:6789/0,b=v1:172.21.15.165:6789/0,c=v1:172.21.15.83:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.926442+0000 mon.a (mon.0) 178 : cluster [DBG] fsmap 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.926533+0000 mon.a (mon.0) 179 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.926792+0000 mon.a (mon.0) 180 : cluster [DBG] mgrmap e13: y(active, since 85s) 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: cluster 2024-09-10T18:41:05.927012+0000 mon.a (mon.0) 181 : cluster [INF] overall HEALTH_OK 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:05.933667+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:05.940686+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:05.953871+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.298 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:05 smithi083 bash[32381]: audit 2024-09-10T18:41:05.958376+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:06.977 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-10T18:41:06.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.193896+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? v1:172.21.15.165:0/528064484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.327442+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.329116+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.596754+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.602093+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.609282+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.616780+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.624408+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.667763+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.669055+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.670168+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18629]: audit 2024-09-10T18:41:06.883604+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.193896+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? v1:172.21.15.165:0/528064484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.327442+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.329116+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.596754+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.602093+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.609282+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.616780+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.624408+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.667763+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.669055+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.670168+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[32381]: audit 2024-09-10T18:41:06.883604+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.193896+0000 mon.a (mon.0) 186 : audit [DBG] from='client.? v1:172.21.15.165:0/528064484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.327442+0000 mon.a (mon.0) 187 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.329116+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.596754+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.602093+0000 mon.a (mon.0) 190 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.609282+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.616780+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.624408+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:07.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.667763+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:07.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.669055+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:07.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.670168+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:07.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:07 smithi165 bash[22591]: audit 2024-09-10T18:41:06.883604+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:41:08.176 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:41:07 smithi083 bash[18896]: debug 2024-09-10T18:41:07.880+0000 7f7118f63640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.331123+0000 mgr.y (mgr.14150) 59 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.331465+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cluster 2024-09-10T18:41:06.370017+0000 mgr.y (mgr.14150) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.469083+0000 mgr.y (mgr.14150) 62 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.482471+0000 mgr.y (mgr.14150) 63 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.667238+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[18629]: cephadm 2024-09-10T18:41:06.671372+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.a on smithi083 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.331123+0000 mgr.y (mgr.14150) 59 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.331465+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:41:08.177 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cluster 2024-09-10T18:41:06.370017+0000 mgr.y (mgr.14150) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:08.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.469083+0000 mgr.y (mgr.14150) 62 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.482471+0000 mgr.y (mgr.14150) 63 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.667238+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:41:08.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:08 smithi083 bash[32381]: cephadm 2024-09-10T18:41:06.671372+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.a on smithi083 2024-09-10T18:41:08.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.331123+0000 mgr.y (mgr.14150) 59 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.331465+0000 mgr.y (mgr.14150) 60 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cluster 2024-09-10T18:41:06.370017+0000 mgr.y (mgr.14150) 61 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.469083+0000 mgr.y (mgr.14150) 62 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.482471+0000 mgr.y (mgr.14150) 63 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.667238+0000 mgr.y (mgr.14150) 64 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:41:08.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:08 smithi165 bash[22591]: cephadm 2024-09-10T18:41:06.671372+0000 mgr.y (mgr.14150) 65 : cephadm [INF] Reconfiguring daemon mon.a on smithi083 2024-09-10T18:41:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: cluster 2024-09-10T18:41:08.370488+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: audit 2024-09-10T18:41:08.822748+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: audit 2024-09-10T18:41:08.829732+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: cephadm 2024-09-10T18:41:08.830815+0000 mgr.y (mgr.14150) 67 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: audit 2024-09-10T18:41:08.831202+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: audit 2024-09-10T18:41:08.832199+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: audit 2024-09-10T18:41:08.833245+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:09 smithi165 bash[22591]: cephadm 2024-09-10T18:41:08.834502+0000 mgr.y (mgr.14150) 68 : cephadm [INF] Reconfiguring daemon mon.c on smithi083 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: cluster 2024-09-10T18:41:08.370488+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: audit 2024-09-10T18:41:08.822748+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: audit 2024-09-10T18:41:08.829732+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: cephadm 2024-09-10T18:41:08.830815+0000 mgr.y (mgr.14150) 67 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: audit 2024-09-10T18:41:08.831202+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: audit 2024-09-10T18:41:08.832199+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: audit 2024-09-10T18:41:08.833245+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[32381]: cephadm 2024-09-10T18:41:08.834502+0000 mgr.y (mgr.14150) 68 : cephadm [INF] Reconfiguring daemon mon.c on smithi083 2024-09-10T18:41:10.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: cluster 2024-09-10T18:41:08.370488+0000 mgr.y (mgr.14150) 66 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: audit 2024-09-10T18:41:08.822748+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: audit 2024-09-10T18:41:08.829732+0000 mon.a (mon.0) 199 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: cephadm 2024-09-10T18:41:08.830815+0000 mgr.y (mgr.14150) 67 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: audit 2024-09-10T18:41:08.831202+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: audit 2024-09-10T18:41:08.832199+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: audit 2024-09-10T18:41:08.833245+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:10.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:09 smithi083 bash[18629]: cephadm 2024-09-10T18:41:08.834502+0000 mgr.y (mgr.14150) 68 : cephadm [INF] Reconfiguring daemon mon.c on smithi083 2024-09-10T18:41:11.706 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:12.080 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: cluster 2024-09-10T18:41:10.370891+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:12.080 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: audit 2024-09-10T18:41:10.822564+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.080 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: audit 2024-09-10T18:41:10.831249+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: cephadm 2024-09-10T18:41:10.832452+0000 mgr.y (mgr.14150) 70 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-10T18:41:12.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: audit 2024-09-10T18:41:10.832964+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:12.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: audit 2024-09-10T18:41:10.834178+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:12.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: audit 2024-09-10T18:41:10.835320+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:12.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:11 smithi165 bash[22591]: cephadm 2024-09-10T18:41:10.836711+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Reconfiguring daemon mon.b on smithi165 2024-09-10T18:41:12.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: cluster 2024-09-10T18:41:10.370891+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:12.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: audit 2024-09-10T18:41:10.822564+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: audit 2024-09-10T18:41:10.831249+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: cephadm 2024-09-10T18:41:10.832452+0000 mgr.y (mgr.14150) 70 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: audit 2024-09-10T18:41:10.832964+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: audit 2024-09-10T18:41:10.834178+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: audit 2024-09-10T18:41:10.835320+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[18629]: cephadm 2024-09-10T18:41:10.836711+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Reconfiguring daemon mon.b on smithi165 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: cluster 2024-09-10T18:41:10.370891+0000 mgr.y (mgr.14150) 69 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: audit 2024-09-10T18:41:10.822564+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: audit 2024-09-10T18:41:10.831249+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:12.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: cephadm 2024-09-10T18:41:10.832452+0000 mgr.y (mgr.14150) 70 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-09-10T18:41:12.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: audit 2024-09-10T18:41:10.832964+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:41:12.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: audit 2024-09-10T18:41:10.834178+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:41:12.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: audit 2024-09-10T18:41:10.835320+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:12.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:11 smithi083 bash[32381]: cephadm 2024-09-10T18:41:10.836711+0000 mgr.y (mgr.14150) 71 : cephadm [INF] Reconfiguring daemon mon.b on smithi165 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: cluster 2024-09-10T18:41:12.371324+0000 mgr.y (mgr.14150) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.876003+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.884822+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.886831+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.889458+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.890873+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[18629]: audit 2024-09-10T18:41:12.900657+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: cluster 2024-09-10T18:41:12.371324+0000 mgr.y (mgr.14150) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.876003+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.884822+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.886831+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.889458+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.890873+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:14.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:13 smithi083 bash[32381]: audit 2024-09-10T18:41:12.900657+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: cluster 2024-09-10T18:41:12.371324+0000 mgr.y (mgr.14150) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.876003+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.884822+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.886831+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.889458+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.890873+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:13 smithi165 bash[22591]: audit 2024-09-10T18:41:12.900657+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:14.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.804+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/1976192832 >> v1:172.21.15.83:6790/0 conn(0x7f0cdc104280 legacy=0x7f0cdc106660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:14.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/1976192832 shutdown_connections 2024-09-10T18:41:14.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/1976192832 >> 172.21.15.83:0/1976192832 conn(0x7f0cdc0f70d0 msgr2=0x7f0cdc0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:14.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/1976192832 shutdown_connections 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/1976192832 wait complete. 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 librados: starting msgr at 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 librados: starting objecter 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 Processor -- start 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- start start 2024-09-10T18:41:14.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 librados: setting wanted keys 2024-09-10T18:41:14.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 librados: calling monclient init 2024-09-10T18:41:14.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0cdc11afd0 con 0x7f0cdc104280 2024-09-10T18:41:14.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0cdc11c1d0 con 0x7f0cdc101510 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0cdc11d3d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cdbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0cdc104280 0x7f0cdc104f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38270/0 (socket says 172.21.15.83:38270) 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cdbfff640 1 -- 172.21.15.83:0/19849713 learned_addr learned my addr 172.21.15.83:0/19849713 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter ms_handle_connect 0x7f0cdc107de0 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter ms_handle_connect 0x7f0cdc104280 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:14.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter ms_handle_connect 0x7f0cdc101510 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282602053 0 0) 0x7f0cdc11afd0 con 0x7f0cdc104280 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc8003120 con 0x7f0cdc104280 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438503221 0 0) 0x7f0cdc11d3d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc11afd0 con 0x7f0cdc107de0 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563591365 0 0) 0x7f0cdc11c1d0 con 0x7f0cdc101510 2024-09-10T18:41:14.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc11d3d0 con 0x7f0cdc101510 2024-09-10T18:41:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184772521 0 0) 0x7f0cdc11afd0 con 0x7f0cdc107de0 2024-09-10T18:41:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cdc11c1d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0cd40017c0 con 0x7f0cdc107de0 2024-09-10T18:41:14.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841429657 0 0) 0x7f0cc8003120 con 0x7f0cdc104280 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cdc11afd0 con 0x7f0cdc104280 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0ccc001890 con 0x7f0cdc104280 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635526727 0 0) 0x7f0cdc11d3d0 con 0x7f0cdc101510 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cc8003120 con 0x7f0cdc101510 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0cc40017b0 con 0x7f0cdc101510 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3406367632 0 0) 0x7f0cdc11c1d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 >> v1:172.21.15.165:6789/0 conn(0x7f0cdc101510 legacy=0x7f0cdc1047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 >> v1:172.21.15.83:6789/0 conn(0x7f0cdc104280 legacy=0x7f0cdc104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc11e5d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc11b1c0 con 0x7f0cdc107de0 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 10 client.24104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:41:14.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc11b610 con 0x7f0cdc107de0 2024-09-10T18:41:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.808+0000 7f0ce2bb6640 1 librados: init done 2024-09-10T18:41:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0cd4002d00 con 0x7f0cdc107de0 2024-09-10T18:41:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0cd4002f50 con 0x7f0cdc107de0 2024-09-10T18:41:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7f0cd400e290 con 0x7f0cdc107de0 2024-09-10T18:41:14.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc005010 con 0x7f0cdc107de0 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7f0cd40024d0 con 0x7f0cdc107de0 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 10 client.24104.objecter ms_dispatch 0x7f0cdc000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 3 client.24104.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 3 client.24104.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 20 client.24104.objecter dump_active .. 0 homeless 2024-09-10T18:41:14.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.812+0000 7f0cd97fa640 10 client.24104.objecter ms_handle_connect 0x7f0cc803e5b0 2024-09-10T18:41:14.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.820+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0cd400fda0 con 0x7f0cdc107de0 2024-09-10T18:41:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.992+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0cbc0052c0 con 0x7f0cdc107de0 2024-09-10T18:41:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.992+0000 7f0cd97fa640 1 -- 172.21.15.83:0/19849713 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+191 (unknown 2189217731 0 3213663321) 0x7f0cd4013640 con 0x7f0cdc107de0 2024-09-10T18:41:14.998 INFO:teuthology.orchestra.run.smithi083.stdout:# minimal ceph.conf for 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:41:14.998 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-09-10T18:41:14.998 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:41:14.998 INFO:teuthology.orchestra.run.smithi083.stdout: mon_host = 172.21.15.83:6789/0 172.21.15.165:6789/0 v1:172.21.15.83:6790/0 2024-09-10T18:41:15.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 10 librados: watch_flush enter 2024-09-10T18:41:15.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 10 librados: watch_flush exit 2024-09-10T18:41:15.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 20 client.24104.objecter shutdown clearing up homeless session... 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 10 client.24104.objecter successfully canceled tick 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 >> v1:172.21.15.83:6800/2630800463 conn(0x7f0cc803e5b0 legacy=0x7f0cc8040a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 >> v1:172.21.15.83:6790/0 conn(0x7f0cdc107de0 legacy=0x7f0cdc1196d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 shutdown_connections 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 >> 172.21.15.83:0/19849713 conn(0x7f0cdc0f70d0 msgr2=0x7f0cdc10aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:15.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 shutdown_connections 2024-09-10T18:41:15.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:14.996+0000 7f0ce2bb6640 1 -- 172.21.15.83:0/19849713 wait complete. 2024-09-10T18:41:15.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:15.000+0000 7f0ce2bb6640 1 librados: shutdown 2024-09-10T18:41:15.669 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-10T18:41:15.669 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:41:15.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:41:15.686 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:41:15.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:41:15.743 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:41:15.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:41:15.759 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:41:15.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:41:15.771 INFO:tasks.cephadm:Adding mgr.y on smithi083 2024-09-10T18:41:15.772 INFO:tasks.cephadm:Adding mgr.x on smithi165 2024-09-10T18:41:15.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi083=y;smithi165=x' 2024-09-10T18:41:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:15 smithi083 bash[18629]: cluster 2024-09-10T18:41:14.371732+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:15 smithi083 bash[18629]: audit 2024-09-10T18:41:14.997647+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.83:0/19849713' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:15 smithi083 bash[32381]: cluster 2024-09-10T18:41:14.371732+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:15 smithi083 bash[32381]: audit 2024-09-10T18:41:14.997647+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.83:0/19849713' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:16.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:15 smithi165 bash[22591]: cluster 2024-09-10T18:41:14.371732+0000 mgr.y (mgr.14150) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:16.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:15 smithi165 bash[22591]: audit 2024-09-10T18:41:14.997647+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.83:0/19849713' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:18.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:17 smithi083 bash[18629]: cluster 2024-09-10T18:41:16.372180+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:17 smithi083 bash[32381]: cluster 2024-09-10T18:41:16.372180+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:18.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:17 smithi165 bash[22591]: cluster 2024-09-10T18:41:16.372180+0000 mgr.y (mgr.14150) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:19.485 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:41:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:19 smithi083 bash[18629]: cluster 2024-09-10T18:41:18.372576+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:19 smithi083 bash[32381]: cluster 2024-09-10T18:41:18.372576+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:20.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:19 smithi165 bash[22591]: cluster 2024-09-10T18:41:18.372576+0000 mgr.y (mgr.14150) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:22.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:21 smithi083 bash[18629]: cluster 2024-09-10T18:41:20.372988+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:22.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:21 smithi083 bash[32381]: cluster 2024-09-10T18:41:20.372988+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:22.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:21 smithi165 bash[22591]: cluster 2024-09-10T18:41:20.372988+0000 mgr.y (mgr.14150) 76 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:22.606 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- 172.21.15.165:0/83778080 >> v1:172.21.15.165:6789/0 conn(0x7efe640fc590 legacy=0x7efe640fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:22.607 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- 172.21.15.165:0/83778080 shutdown_connections 2024-09-10T18:41:22.607 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- 172.21.15.165:0/83778080 >> 172.21.15.165:0/83778080 conn(0x7efe640f7d70 msgr2=0x7efe640fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:22.607 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- 172.21.15.165:0/83778080 shutdown_connections 2024-09-10T18:41:22.607 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- 172.21.15.165:0/83778080 wait complete. 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 librados: starting msgr at 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 librados: starting objecter 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 Processor -- start 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 -- start start 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 librados: setting wanted keys 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.601+0000 7efe6b6c9640 1 librados: calling monclient init 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe641196d0 con 0x7efe641046b0 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe6411a8d0 con 0x7efe640fc590 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe6411bad0 con 0x7efe64100a90 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe69465640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7efe640fc590 0x7efe64071320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:46180/0 (socket says 172.21.15.165:46180) 2024-09-10T18:41:22.609 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe69465640 1 -- 172.21.15.165:0/662309915 learned_addr learned my addr 172.21.15.165:0/662309915 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter ms_handle_connect 0x7efe640fc590 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter ms_handle_connect 0x7efe64100a90 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter ms_handle_connect 0x7efe641046b0 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757068825 0 0) 0x7efe6411a8d0 con 0x7efe640fc590 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe500054b0 con 0x7efe640fc590 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333433222 0 0) 0x7efe6411bad0 con 0x7efe64100a90 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6411a8d0 con 0x7efe64100a90 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090832921 0 0) 0x7efe641196d0 con 0x7efe641046b0 2024-09-10T18:41:22.610 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6411bad0 con 0x7efe641046b0 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185234200 0 0) 0x7efe500054b0 con 0x7efe640fc590 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe641196d0 con 0x7efe640fc590 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe540021f0 con 0x7efe640fc590 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676393007 0 0) 0x7efe6411a8d0 con 0x7efe64100a90 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe500054b0 con 0x7efe64100a90 2024-09-10T18:41:22.611 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe60001840 con 0x7efe64100a90 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512630469 0 0) 0x7efe641196d0 con 0x7efe640fc590 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 >> v1:172.21.15.83:6790/0 conn(0x7efe64100a90 legacy=0x7efe64114510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 >> v1:172.21.15.83:6789/0 conn(0x7efe641046b0 legacy=0x7efe64117dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6411ccd0 con 0x7efe640fc590 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6411bcc0 con 0x7efe640fc590 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7efe54002390 con 0x7efe640fc590 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 10 client.24109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe540038e0 con 0x7efe640fc590 2024-09-10T18:41:22.612 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6411c110 con 0x7efe640fc590 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 librados: init done 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6404bf60 con 0x7efe640fc590 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.605+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 13) v1 ==== 45513+0+0 (unknown 2097086416 0 0) 0x7efe54003b20 con 0x7efe640fc590 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7efe54047d10 con 0x7efe640fc590 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 10 client.24109.objecter ms_dispatch 0x7efe64000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 3 client.24109.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 3 client.24109.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 20 client.24109.objecter dump_active .. 0 homeless 2024-09-10T18:41:22.613 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.609+0000 7efe59ffb640 10 client.24109.objecter ms_handle_connect 0x7efe5003c080 2024-09-10T18:41:22.616 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.613+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7efe54016d00 con 0x7efe640fc590 2024-09-10T18:41:22.796 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.793+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi165=x", "target": ["mon-mgr", ""]}) v1 -- 0x7efe64105b40 con 0x7efe5003c080 2024-09-10T18:41:22.809 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe59ffb640 1 -- 172.21.15.165:0/662309915 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7efe64105b40 con 0x7efe5003c080 2024-09-10T18:41:22.809 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 10 librados: watch_flush enter 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 10 librados: watch_flush exit 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 20 client.24109.objecter shutdown clearing up homeless session... 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 10 client.24109.objecter successfully canceled tick 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 >> v1:172.21.15.83:6800/2630800463 conn(0x7efe5003c080 legacy=0x7efe5003e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:22.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 >> v1:172.21.15.165:6789/0 conn(0x7efe640fc590 legacy=0x7efe64071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 shutdown_connections 2024-09-10T18:41:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.805+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 >> 172.21.15.165:0/662309915 conn(0x7efe640f7d70 msgr2=0x7efe64107ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.809+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 shutdown_connections 2024-09-10T18:41:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.809+0000 7efe6b6c9640 1 -- 172.21.15.165:0/662309915 wait complete. 2024-09-10T18:41:22.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:41:22.809+0000 7efe6b6c9640 1 librados: shutdown 2024-09-10T18:41:23.655 DEBUG:teuthology.orchestra.run.smithi165:mgr.x> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.x.service 2024-09-10T18:41:23.657 INFO:tasks.cephadm:Deploying OSDs... 2024-09-10T18:41:23.657 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:41:23.657 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:41:23.665 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:41:23.665 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 370 Links: 1 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-10 18:31:50.507293897 +0000 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-10 18:31:50.287282900 +0000 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-10 18:31:50.287282900 +0000 2024-09-10T18:41:23.674 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-10T18:41:23.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:41:23.733 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-10T18:41:23.733 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-10T18:41:23.733 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000429352 s, 1.2 MB/s 2024-09-10T18:41:23.734 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:41:23.743 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-09-10T18:41:23.749 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:41:23.749 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-10 18:31:50.735305296 +0000 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-10 18:31:50.731305097 +0000 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-10 18:31:50.731305097 +0000 2024-09-10T18:41:23.750 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-10T18:41:23.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:41:23.763 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-10T18:41:23.764 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-10T18:41:23.764 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000426513 s, 1.2 MB/s 2024-09-10T18:41:23.764 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:41:23.815 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 383 Links: 1 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-10 18:31:51.391337707 +0000 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-10 18:31:51.191327908 +0000 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-10 18:31:51.191327908 +0000 2024-09-10T18:41:23.821 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-10T18:41:23.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:41:23.830 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-10T18:41:23.831 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-10T18:41:23.831 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00032237 s, 1.6 MB/s 2024-09-10T18:41:23.832 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:41:23.882 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 6h/6d Inode: 388 Links: 1 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-10 18:31:51.651350445 +0000 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-10 18:31:51.647350251 +0000 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-10 18:31:51.647350251 +0000 2024-09-10T18:41:23.890 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: - 2024-09-10T18:41:23.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:41:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-10T18:41:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-10T18:41:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000398714 s, 1.3 MB/s 2024-09-10T18:41:23.905 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:41:23.912 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:41:23.913 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:41:23.920 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:41:23.920 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-10T18:41:23.970 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:41:23.970 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:23.970 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-10T18:41:23.970 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:23.971 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 18:31:50.467113320 +0000 2024-09-10T18:41:23.971 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 18:31:50.259118110 +0000 2024-09-10T18:41:23.971 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 18:31:50.259118110 +0000 2024-09-10T18:41:23.971 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-10T18:41:23.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:41:23.985 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T18:41:23.986 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T18:41:23.986 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000430613 s, 1.2 MB/s 2024-09-10T18:41:23.987 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:41:23.994 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-10T18:41:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: cluster 2024-09-10T18:41:22.373446+0000 mgr.y (mgr.14150) 77 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.798391+0000 mgr.y (mgr.14150) 78 : audit [DBG] from='client.24109 v1:172.21.15.165:0/662309915' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: cephadm 2024-09-10T18:41:22.801331+0000 mgr.y (mgr.14150) 79 : cephadm [INF] Saving service mgr spec with placement smithi083=y;smithi165=x;count:2 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.808232+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.809845+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.813096+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.814826+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.823501+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.825583+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.830691+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.831953+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: audit 2024-09-10T18:41:22.833298+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[18629]: cephadm 2024-09-10T18:41:22.834654+0000 mgr.y (mgr.14150) 80 : cephadm [INF] Deploying daemon mgr.x on smithi165 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: cluster 2024-09-10T18:41:22.373446+0000 mgr.y (mgr.14150) 77 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.798391+0000 mgr.y (mgr.14150) 78 : audit [DBG] from='client.24109 v1:172.21.15.165:0/662309915' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: cephadm 2024-09-10T18:41:22.801331+0000 mgr.y (mgr.14150) 79 : cephadm [INF] Saving service mgr spec with placement smithi083=y;smithi165=x;count:2 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.808232+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.809845+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.813096+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.814826+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.823501+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.825583+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.830691+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:41:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.831953+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: audit 2024-09-10T18:41:22.833298+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:23 smithi083 bash[32381]: cephadm 2024-09-10T18:41:22.834654+0000 mgr.y (mgr.14150) 80 : cephadm [INF] Deploying daemon mgr.x on smithi165 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 18:31:50.863104200 +0000 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 18:31:50.659108898 +0000 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 18:31:50.659108898 +0000 2024-09-10T18:41:24.044 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-10T18:41:24.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:41:24.056 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T18:41:24.056 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T18:41:24.056 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00046098 s, 1.1 MB/s 2024-09-10T18:41:24.057 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:41:24.108 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-10T18:41:24.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: cluster 2024-09-10T18:41:22.373446+0000 mgr.y (mgr.14150) 77 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.798391+0000 mgr.y (mgr.14150) 78 : audit [DBG] from='client.24109 v1:172.21.15.165:0/662309915' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi083=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: cephadm 2024-09-10T18:41:22.801331+0000 mgr.y (mgr.14150) 79 : cephadm [INF] Saving service mgr spec with placement smithi083=y;smithi165=x;count:2 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.808232+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.809845+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.813096+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.814826+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.823501+0000 mon.a (mon.0) 218 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.825583+0000 mon.a (mon.0) 219 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.830691+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.831953+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: audit 2024-09-10T18:41:22.833298+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:23 smithi165 bash[22591]: cephadm 2024-09-10T18:41:22.834654+0000 mgr.y (mgr.14150) 80 : cephadm [INF] Deploying daemon mgr.x on smithi165 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 373 Links: 1 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 18:31:51.287094435 +0000 2024-09-10T18:41:24.114 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 18:31:51.083099132 +0000 2024-09-10T18:41:24.115 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 18:31:51.083099132 +0000 2024-09-10T18:41:24.115 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-10T18:41:24.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:41:24.126 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T18:41:24.126 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T18:41:24.126 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000412128 s, 1.2 MB/s 2024-09-10T18:41:24.127 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:41:24.180 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-10T18:41:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:41:24.187 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:41:24.187 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-10T18:41:24.188 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:41:24.188 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 18:31:51.699084946 +0000 2024-09-10T18:41:24.188 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 18:31:51.483089922 +0000 2024-09-10T18:41:24.188 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 18:31:51.483089922 +0000 2024-09-10T18:41:24.188 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2024-09-10T18:41:24.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:41:24.201 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T18:41:24.201 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T18:41:24.201 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000289526 s, 1.8 MB/s 2024-09-10T18:41:24.202 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:41:24.210 INFO:tasks.cephadm:Deploying osd.0 on smithi083 with /dev/vg_nvme/lv_4... 2024-09-10T18:41:24.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:41:24.489 INFO:journalctl@ceph.mgr.x.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:41:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:25 smithi165 bash[22591]: cluster 2024-09-10T18:41:24.373779+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:25 smithi165 bash[22591]: audit 2024-09-10T18:41:25.750800+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:25 smithi165 bash[22591]: audit 2024-09-10T18:41:25.757502+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:25 smithi165 bash[22591]: audit 2024-09-10T18:41:25.763808+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:25 smithi165 bash[22591]: audit 2024-09-10T18:41:25.769398+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[18629]: cluster 2024-09-10T18:41:24.373779+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:26.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[18629]: audit 2024-09-10T18:41:25.750800+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[18629]: audit 2024-09-10T18:41:25.757502+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[18629]: audit 2024-09-10T18:41:25.763808+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[18629]: audit 2024-09-10T18:41:25.769398+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[32381]: cluster 2024-09-10T18:41:24.373779+0000 mgr.y (mgr.14150) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[32381]: audit 2024-09-10T18:41:25.750800+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[32381]: audit 2024-09-10T18:41:25.757502+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[32381]: audit 2024-09-10T18:41:25.763808+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:25 smithi083 bash[32381]: audit 2024-09-10T18:41:25.769398+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:27.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:26 smithi165 bash[22591]: audit 2024-09-10T18:41:25.816649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:27.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:26 smithi083 bash[18629]: audit 2024-09-10T18:41:25.816649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:27.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:26 smithi083 bash[32381]: audit 2024-09-10T18:41:25.816649+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:27.361 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:27 smithi165 bash[23586]: debug 2024-09-10T18:41:27.009+0000 7f146f9abec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:41:28.031 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:27 smithi165 bash[22591]: cluster 2024-09-10T18:41:26.374221+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:28.031 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:27 smithi165 bash[23586]: debug 2024-09-10T18:41:27.749+0000 7f146f9abec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:41:28.032 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:27 smithi165 bash[23586]: debug 2024-09-10T18:41:27.813+0000 7f146f9abec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:41:28.032 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:27 smithi165 bash[23586]: debug 2024-09-10T18:41:27.877+0000 7f146f9abec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:41:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:27 smithi083 bash[18629]: cluster 2024-09-10T18:41:26.374221+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:28.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:27 smithi083 bash[32381]: cluster 2024-09-10T18:41:26.374221+0000 mgr.y (mgr.14150) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:28.336 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:28 smithi165 bash[23586]: debug 2024-09-10T18:41:28.025+0000 7f146f9abec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:41:28.337 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:28 smithi165 bash[23586]: debug 2024-09-10T18:41:28.269+0000 7f146f9abec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:41:28.337 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:28 smithi165 bash[23586]: debug 2024-09-10T18:41:28.333+0000 7f146f9abec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:41:28.611 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:28 smithi165 bash[23586]: debug 2024-09-10T18:41:28.461+0000 7f146f9abec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:41:28.942 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:29.361 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.093+0000 7f146f9abec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:41:29.362 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.161+0000 7f146f9abec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:41:29.362 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.237+0000 7f146f9abec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:41:29.828 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.513+0000 7f146f9abec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:41:29.828 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.645+0000 7f146f9abec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:41:30.080 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:29 smithi083 bash[18629]: cluster 2024-09-10T18:41:28.374672+0000 mgr.y (mgr.14150) 83 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:30.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:29 smithi083 bash[32381]: cluster 2024-09-10T18:41:28.374672+0000 mgr.y (mgr.14150) 83 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:30.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[22591]: cluster 2024-09-10T18:41:28.374672+0000 mgr.y (mgr.14150) 83 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:30.111 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:29 smithi165 bash[23586]: debug 2024-09-10T18:41:29.953+0000 7f146f9abec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:41:30.583 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.165+0000 7f146f9abec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:41:30.861 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.577+0000 7f146f9abec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:41:30.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.649+0000 7f146f9abec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:41:30.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.717+0000 7f146f9abec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:41:30.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.817+0000 7f146f9abec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:41:31.196 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.885+0000 7f146f9abec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:41:31.196 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:30 smithi165 bash[23586]: debug 2024-09-10T18:41:30.965+0000 7f146f9abec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:41:31.196 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[23586]: debug 2024-09-10T18:41:31.113+0000 7f146f9abec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:41:31.611 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[23586]: debug 2024-09-10T18:41:31.189+0000 7f146f9abec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: cluster 2024-09-10T18:41:30.375075+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:30.427466+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:30.840094+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:30.847332+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.217476+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.219082+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.230335+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.273554+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.275108+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:31 smithi165 bash[22591]: audit 2024-09-10T18:41:31.276281+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: cluster 2024-09-10T18:41:30.375075+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:30.427466+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:30.840094+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:30.847332+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.217476+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.219082+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.230335+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.273554+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.275108+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[18629]: audit 2024-09-10T18:41:31.276281+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: cluster 2024-09-10T18:41:30.375075+0000 mgr.y (mgr.14150) 84 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:30.427466+0000 mon.a (mon.0) 228 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:30.840094+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:30.847332+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.217476+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.219082+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.230335+0000 mon.a (mon.0) 233 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.273554+0000 mon.a (mon.0) 234 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.275108+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:41:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:31 smithi083 bash[32381]: audit 2024-09-10T18:41:31.276281+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:32.440 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[23586]: debug 2024-09-10T18:41:32.109+0000 7f146f9abec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:41:32.486 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:41:32.682 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:41:32.704 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-09-10T18:41:32.704 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-09-10T18:41:32.704 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0181744 s, 577 MB/s 2024-09-10T18:41:32.704 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-09-10T18:41:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: cephadm 2024-09-10T18:41:31.272975+0000 mgr.y (mgr.14150) 85 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:41:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: cephadm 2024-09-10T18:41:31.277513+0000 mgr.y (mgr.14150) 86 : cephadm [INF] Reconfiguring daemon mgr.y on smithi083 2024-09-10T18:41:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: audit 2024-09-10T18:41:32.136350+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:41:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: cluster 2024-09-10T18:41:32.136554+0000 mon.a (mon.0) 237 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:41:32.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: audit 2024-09-10T18:41:32.136964+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:41:32.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: audit 2024-09-10T18:41:32.138492+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:41:32.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[18629]: audit 2024-09-10T18:41:32.139018+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:41:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: cephadm 2024-09-10T18:41:31.272975+0000 mgr.y (mgr.14150) 85 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:41:32.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: cephadm 2024-09-10T18:41:31.277513+0000 mgr.y (mgr.14150) 86 : cephadm [INF] Reconfiguring daemon mgr.y on smithi083 2024-09-10T18:41:32.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: audit 2024-09-10T18:41:32.136350+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:41:32.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: cluster 2024-09-10T18:41:32.136554+0000 mon.a (mon.0) 237 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:41:32.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: audit 2024-09-10T18:41:32.136964+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:41:32.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: audit 2024-09-10T18:41:32.138492+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:41:32.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:32 smithi083 bash[32381]: audit 2024-09-10T18:41:32.139018+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: cephadm 2024-09-10T18:41:31.272975+0000 mgr.y (mgr.14150) 85 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: cephadm 2024-09-10T18:41:31.277513+0000 mgr.y (mgr.14150) 86 : cephadm [INF] Reconfiguring daemon mgr.y on smithi083 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: audit 2024-09-10T18:41:32.136350+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: cluster 2024-09-10T18:41:32.136554+0000 mon.a (mon.0) 237 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: audit 2024-09-10T18:41:32.136964+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: audit 2024-09-10T18:41:32.138492+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:41:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:32 smithi165 bash[22591]: audit 2024-09-10T18:41:32.139018+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3112370175' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:41:33.696 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[18629]: cluster 2024-09-10T18:41:32.375525+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:33.696 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[18629]: cluster 2024-09-10T18:41:32.450767+0000 mon.a (mon.0) 238 : cluster [DBG] mgrmap e14: y(active, since 112s), standbys: x 2024-09-10T18:41:33.697 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[18629]: audit 2024-09-10T18:41:32.451187+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:41:33.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[32381]: cluster 2024-09-10T18:41:32.375525+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:33.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[32381]: cluster 2024-09-10T18:41:32.450767+0000 mon.a (mon.0) 238 : cluster [DBG] mgrmap e14: y(active, since 112s), standbys: x 2024-09-10T18:41:33.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:33 smithi083 bash[32381]: audit 2024-09-10T18:41:32.451187+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:41:33.698 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-09-10T18:41:33.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:33 smithi165 bash[22591]: cluster 2024-09-10T18:41:32.375525+0000 mgr.y (mgr.14150) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:33 smithi165 bash[22591]: cluster 2024-09-10T18:41:32.450767+0000 mon.a (mon.0) 238 : cluster [DBG] mgrmap e14: y(active, since 112s), standbys: x 2024-09-10T18:41:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:33 smithi165 bash[22591]: audit 2024-09-10T18:41:32.451187+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.171095+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.178495+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.180711+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.183340+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.184595+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: audit 2024-09-10T18:41:34.191547+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[18629]: cluster 2024-09-10T18:41:34.376012+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.171095+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.178495+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.180711+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.183340+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.184595+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: audit 2024-09-10T18:41:34.191547+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:35 smithi083 bash[32381]: cluster 2024-09-10T18:41:34.376012+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.171095+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.178495+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.180711+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.183340+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.184595+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: audit 2024-09-10T18:41:34.191547+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:41:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:35 smithi165 bash[22591]: cluster 2024-09-10T18:41:34.376012+0000 mgr.y (mgr.14150) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:37 smithi083 bash[18629]: cluster 2024-09-10T18:41:36.376503+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:37 smithi083 bash[32381]: cluster 2024-09-10T18:41:36.376503+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:37 smithi165 bash[22591]: cluster 2024-09-10T18:41:36.376503+0000 mgr.y (mgr.14150) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:38.434 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:41:39.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:39 smithi083 bash[18629]: cluster 2024-09-10T18:41:38.376949+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:39.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:39 smithi083 bash[32381]: cluster 2024-09-10T18:41:38.376949+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:39.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:39 smithi165 bash[22591]: cluster 2024-09-10T18:41:38.376949+0000 mgr.y (mgr.14150) 90 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:41.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- 172.21.15.83:0/2277328921 >> v1:172.21.15.83:6789/0 conn(0x7f862c100930 legacy=0x7f862c102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:41.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- 172.21.15.83:0/2277328921 shutdown_connections 2024-09-10T18:41:41.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- 172.21.15.83:0/2277328921 >> 172.21.15.83:0/2277328921 conn(0x7f862c0f6c60 msgr2=0x7f862c0f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- 172.21.15.83:0/2277328921 shutdown_connections 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- 172.21.15.83:0/2277328921 wait complete. 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 librados: starting msgr at 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 librados: starting objecter 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 Processor -- start 2024-09-10T18:41:41.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 -- start start 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 librados: setting wanted keys 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.499+0000 7f86330b0640 1 librados: calling monclient init 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f862c10d350 con 0x7f862c066830 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f862c118b50 con 0x7f862c100930 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f8630e4c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f862c066830 0x7f862c10c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52330/0 (socket says 172.21.15.83:52330) 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f8630e4c640 1 -- 172.21.15.83:0/3784279661 learned_addr learned my addr 172.21.15.83:0/3784279661 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:41:41.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f862c119cf0 con 0x7f862c1046b0 2024-09-10T18:41:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter ms_handle_connect 0x7f862c1046b0 2024-09-10T18:41:41.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter ms_handle_connect 0x7f862c066830 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028751808 0 0) 0x7f862c119cf0 con 0x7f862c1046b0 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86140034e0 con 0x7f862c1046b0 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384487175 0 0) 0x7f862c10d350 con 0x7f862c066830 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c119cf0 con 0x7f862c066830 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter ms_handle_connect 0x7f862c100930 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413479021 0 0) 0x7f862c118b50 con 0x7f862c100930 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c10d350 con 0x7f862c100930 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042260294 0 0) 0x7f86140034e0 con 0x7f862c1046b0 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f862c118b50 con 0x7f862c1046b0 2024-09-10T18:41:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4233277271 0 0) 0x7f862c119cf0 con 0x7f862c066830 2024-09-10T18:41:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86140034e0 con 0x7f862c066830 2024-09-10T18:41:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8624001c40 con 0x7f862c1046b0 2024-09-10T18:41:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8620002970 con 0x7f862c066830 2024-09-10T18:41:41.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409816166 0 0) 0x7f862c118b50 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 >> v1:172.21.15.165:6789/0 conn(0x7f862c100930 legacy=0x7f862c112cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 >> v1:172.21.15.83:6789/0 conn(0x7f862c066830 legacy=0x7f862c10c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c11aed0 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c117ba0 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 10 client.24122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c118070 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86330b0640 1 librados: init done 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8624002310 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.503+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8624003470 con 0x7f862c1046b0 2024-09-10T18:41:41.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 14) v1 ==== 90689+0+0 (unknown 3087547402 0 0) 0x7f8624019820 con 0x7f862c1046b0 2024-09-10T18:41:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8618005040 con 0x7f862c1046b0 2024-09-10T18:41:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (unknown 2809340345 0 0) 0x7f86240862a0 con 0x7f862c1046b0 2024-09-10T18:41:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 10 client.24122.objecter ms_dispatch 0x7f862c000b90 osd_map(4..4 src has 1..4) v4 2024-09-10T18:41:41.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 3 client.24122.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T18:41:41.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 3 client.24122.objecter handle_osd_map decoding full epoch 4 2024-09-10T18:41:41.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 20 client.24122.objecter dump_active .. 0 homeless 2024-09-10T18:41:41.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.507+0000 7f86297fa640 10 client.24122.objecter ms_handle_connect 0x7f86140756f0 2024-09-10T18:41:41.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.515+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8624054450 con 0x7f862c1046b0 2024-09-10T18:41:41.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:41.699+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8618000bc0 con 0x7f86140756f0 2024-09-10T18:41:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:41 smithi083 bash[18629]: cluster 2024-09-10T18:41:40.377379+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:41.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:41 smithi083 bash[32381]: cluster 2024-09-10T18:41:40.377379+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:41 smithi165 bash[22591]: cluster 2024-09-10T18:41:40.377379+0000 mgr.y (mgr.14150) 91 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:42.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[18629]: audit 2024-09-10T18:41:41.705131+0000 mgr.y (mgr.14150) 92 : audit [DBG] from='client.24122 v1:172.21.15.83:0/3784279661' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[18629]: audit 2024-09-10T18:41:41.711730+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[18629]: audit 2024-09-10T18:41:41.716855+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[18629]: audit 2024-09-10T18:41:41.718003+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[32381]: audit 2024-09-10T18:41:41.705131+0000 mgr.y (mgr.14150) 92 : audit [DBG] from='client.24122 v1:172.21.15.83:0/3784279661' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[32381]: audit 2024-09-10T18:41:41.711730+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[32381]: audit 2024-09-10T18:41:41.716855+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:41:42.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:42 smithi083 bash[32381]: audit 2024-09-10T18:41:41.718003+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:42 smithi165 bash[22591]: audit 2024-09-10T18:41:41.705131+0000 mgr.y (mgr.14150) 92 : audit [DBG] from='client.24122 v1:172.21.15.83:0/3784279661' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:41:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:42 smithi165 bash[22591]: audit 2024-09-10T18:41:41.711730+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:41:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:42 smithi165 bash[22591]: audit 2024-09-10T18:41:41.716855+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:41:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:42 smithi165 bash[22591]: audit 2024-09-10T18:41:41.718003+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:41:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:43 smithi083 bash[18629]: cluster 2024-09-10T18:41:42.377787+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:43.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:43 smithi083 bash[32381]: cluster 2024-09-10T18:41:42.377787+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:43.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:43 smithi165 bash[22591]: cluster 2024-09-10T18:41:42.377787+0000 mgr.y (mgr.14150) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:45.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:45 smithi083 bash[18629]: cluster 2024-09-10T18:41:44.378242+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:45.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:45 smithi083 bash[32381]: cluster 2024-09-10T18:41:44.378242+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:45.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:45 smithi165 bash[22591]: cluster 2024-09-10T18:41:44.378242+0000 mgr.y (mgr.14150) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:46.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:46.507+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:41:47.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:47 smithi083 bash[18629]: cluster 2024-09-10T18:41:46.378620+0000 mgr.y (mgr.14150) 95 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:47.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:47 smithi083 bash[32381]: cluster 2024-09-10T18:41:46.378620+0000 mgr.y (mgr.14150) 95 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:47.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:47 smithi165 bash[22591]: cluster 2024-09-10T18:41:46.378620+0000 mgr.y (mgr.14150) 95 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:49 smithi083 bash[18629]: cluster 2024-09-10T18:41:48.379061+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:49 smithi083 bash[32381]: cluster 2024-09-10T18:41:48.379061+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:49.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:49 smithi165 bash[22591]: cluster 2024-09-10T18:41:48.379061+0000 mgr.y (mgr.14150) 96 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:50.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[32381]: audit 2024-09-10T18:41:50.206924+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.83:0/2253438368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[32381]: audit 2024-09-10T18:41:50.207409+0000 mon.a (mon.0) 249 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[32381]: audit 2024-09-10T18:41:50.214181+0000 mon.a (mon.0) 250 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]': finished 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[32381]: cluster 2024-09-10T18:41:50.214356+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[32381]: audit 2024-09-10T18:41:50.214544+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[18629]: audit 2024-09-10T18:41:50.206924+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.83:0/2253438368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[18629]: audit 2024-09-10T18:41:50.207409+0000 mon.a (mon.0) 249 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[18629]: audit 2024-09-10T18:41:50.214181+0000 mon.a (mon.0) 250 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]': finished 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[18629]: cluster 2024-09-10T18:41:50.214356+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:41:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:50 smithi083 bash[18629]: audit 2024-09-10T18:41:50.214544+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:41:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:50 smithi165 bash[22591]: audit 2024-09-10T18:41:50.206924+0000 mon.c (mon.2) 3 : audit [INF] from='client.? v1:172.21.15.83:0/2253438368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:50 smithi165 bash[22591]: audit 2024-09-10T18:41:50.207409+0000 mon.a (mon.0) 249 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]: dispatch 2024-09-10T18:41:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:50 smithi165 bash[22591]: audit 2024-09-10T18:41:50.214181+0000 mon.a (mon.0) 250 : audit [INF] from='client.24128 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "35ab8328-74f1-4af8-aa84-797914199445"}]': finished 2024-09-10T18:41:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:50 smithi165 bash[22591]: cluster 2024-09-10T18:41:50.214356+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:41:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:50 smithi165 bash[22591]: audit 2024-09-10T18:41:50.214544+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:41:51.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:51.511+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:41:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:51 smithi083 bash[18629]: cluster 2024-09-10T18:41:50.379522+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:51 smithi083 bash[18629]: audit 2024-09-10T18:41:50.810362+0000 mon.a (mon.0) 253 : audit [DBG] from='client.? v1:172.21.15.83:0/260484696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:41:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:51 smithi083 bash[32381]: cluster 2024-09-10T18:41:50.379522+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:51 smithi083 bash[32381]: audit 2024-09-10T18:41:50.810362+0000 mon.a (mon.0) 253 : audit [DBG] from='client.? v1:172.21.15.83:0/260484696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:41:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:51 smithi165 bash[22591]: cluster 2024-09-10T18:41:50.379522+0000 mgr.y (mgr.14150) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:51 smithi165 bash[22591]: audit 2024-09-10T18:41:50.810362+0000 mon.a (mon.0) 253 : audit [DBG] from='client.? v1:172.21.15.83:0/260484696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:41:53.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:53 smithi083 bash[32381]: cluster 2024-09-10T18:41:52.380009+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:53 smithi083 bash[18629]: cluster 2024-09-10T18:41:52.380009+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:53.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:53 smithi165 bash[22591]: cluster 2024-09-10T18:41:52.380009+0000 mgr.y (mgr.14150) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:55 smithi083 bash[18629]: cluster 2024-09-10T18:41:54.380434+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:55 smithi083 bash[32381]: cluster 2024-09-10T18:41:54.380434+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:55.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:55 smithi165 bash[22591]: cluster 2024-09-10T18:41:54.380434+0000 mgr.y (mgr.14150) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:56.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:41:56.515+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:41:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:57 smithi083 bash[18629]: cluster 2024-09-10T18:41:56.380936+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:57 smithi083 bash[32381]: cluster 2024-09-10T18:41:56.380936+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:57.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:57 smithi165 bash[22591]: cluster 2024-09-10T18:41:56.380936+0000 mgr.y (mgr.14150) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:59.744 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:41:59 smithi083 bash[18629]: cluster 2024-09-10T18:41:58.381356+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:59.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:41:59 smithi083 bash[32381]: cluster 2024-09-10T18:41:58.381356+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:41:59.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:41:59 smithi165 bash[22591]: cluster 2024-09-10T18:41:58.381356+0000 mgr.y (mgr.14150) 101 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:01.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:01.519+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:42:01.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:01 smithi083 bash[18629]: cluster 2024-09-10T18:42:00.381812+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:01.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:01 smithi083 bash[32381]: cluster 2024-09-10T18:42:00.381812+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:01.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:01 smithi165 bash[22591]: cluster 2024-09-10T18:42:00.381812+0000 mgr.y (mgr.14150) 102 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[18629]: cluster 2024-09-10T18:42:02.382349+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[18629]: audit 2024-09-10T18:42:02.566818+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:42:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[18629]: audit 2024-09-10T18:42:02.568111+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[18629]: cephadm 2024-09-10T18:42:02.569065+0000 mgr.y (mgr.14150) 104 : cephadm [INF] Deploying daemon osd.0 on smithi083 2024-09-10T18:42:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[32381]: cluster 2024-09-10T18:42:02.382349+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[32381]: audit 2024-09-10T18:42:02.566818+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:42:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[32381]: audit 2024-09-10T18:42:02.568111+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:03 smithi083 bash[32381]: cephadm 2024-09-10T18:42:02.569065+0000 mgr.y (mgr.14150) 104 : cephadm [INF] Deploying daemon osd.0 on smithi083 2024-09-10T18:42:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:03 smithi165 bash[22591]: cluster 2024-09-10T18:42:02.382349+0000 mgr.y (mgr.14150) 103 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:03 smithi165 bash[22591]: audit 2024-09-10T18:42:02.566818+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:42:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:03 smithi165 bash[22591]: audit 2024-09-10T18:42:02.568111+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:03 smithi165 bash[22591]: cephadm 2024-09-10T18:42:02.569065+0000 mgr.y (mgr.14150) 104 : cephadm [INF] Deploying daemon osd.0 on smithi083 2024-09-10T18:42:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:05 smithi083 bash[18629]: cluster 2024-09-10T18:42:04.382801+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:05 smithi083 bash[32381]: cluster 2024-09-10T18:42:04.382801+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:05 smithi165 bash[22591]: cluster 2024-09-10T18:42:04.382801+0000 mgr.y (mgr.14150) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:06.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:06.523+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:42:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[18629]: cluster 2024-09-10T18:42:06.383233+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[18629]: audit 2024-09-10T18:42:06.971036+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[18629]: audit 2024-09-10T18:42:06.975572+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[18629]: audit 2024-09-10T18:42:06.983558+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[32381]: cluster 2024-09-10T18:42:06.383233+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[32381]: audit 2024-09-10T18:42:06.971036+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[32381]: audit 2024-09-10T18:42:06.975572+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:07 smithi083 bash[32381]: audit 2024-09-10T18:42:06.983558+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:07 smithi165 bash[22591]: cluster 2024-09-10T18:42:06.383233+0000 mgr.y (mgr.14150) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:07 smithi165 bash[22591]: audit 2024-09-10T18:42:06.971036+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:07 smithi165 bash[22591]: audit 2024-09-10T18:42:06.975572+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:07 smithi165 bash[22591]: audit 2024-09-10T18:42:06.983558+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:09 smithi083 bash[18629]: cluster 2024-09-10T18:42:08.383708+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:09 smithi083 bash[32381]: cluster 2024-09-10T18:42:08.383708+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:09 smithi165 bash[22591]: cluster 2024-09-10T18:42:08.383708+0000 mgr.y (mgr.14150) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:11.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:11.526+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:42:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:11 smithi165 bash[22591]: cluster 2024-09-10T18:42:10.384047+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:11.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:11 smithi083 bash[18629]: cluster 2024-09-10T18:42:10.384047+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:11 smithi083 bash[32381]: cluster 2024-09-10T18:42:10.384047+0000 mgr.y (mgr.14150) 108 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:13 smithi083 bash[18629]: cluster 2024-09-10T18:42:12.384522+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:13.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:13 smithi083 bash[32381]: cluster 2024-09-10T18:42:12.384522+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:13 smithi165 bash[22591]: cluster 2024-09-10T18:42:12.384522+0000 mgr.y (mgr.14150) 109 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:14.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.461815+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.468723+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.851169+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.852486+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.861246+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[32381]: audit 2024-09-10T18:42:13.957030+0000 mon.a (mon.0) 264 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.461815+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.468723+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.851169+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:14.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.852486+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:14.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.861246+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:14 smithi083 bash[18629]: audit 2024-09-10T18:42:13.957030+0000 mon.a (mon.0) 264 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.461815+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.468723+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.851169+0000 mon.a (mon.0) 261 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.852486+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.861246+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:14 smithi165 bash[22591]: audit 2024-09-10T18:42:13.957030+0000 mon.a (mon.0) 264 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:42:16.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[18629]: cluster 2024-09-10T18:42:14.385022+0000 mgr.y (mgr.14150) 110 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:16.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[18629]: audit 2024-09-10T18:42:14.871767+0000 mon.a (mon.0) 265 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:42:16.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[18629]: cluster 2024-09-10T18:42:14.871864+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:42:16.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[18629]: audit 2024-09-10T18:42:14.872036+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:16.234 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[18629]: audit 2024-09-10T18:42:14.872522+0000 mon.a (mon.0) 268 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:42:16.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[32381]: cluster 2024-09-10T18:42:14.385022+0000 mgr.y (mgr.14150) 110 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:16.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[32381]: audit 2024-09-10T18:42:14.871767+0000 mon.a (mon.0) 265 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:42:16.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[32381]: cluster 2024-09-10T18:42:14.871864+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:42:16.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[32381]: audit 2024-09-10T18:42:14.872036+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:16.235 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:15 smithi083 bash[32381]: audit 2024-09-10T18:42:14.872522+0000 mon.a (mon.0) 268 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:42:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:15 smithi165 bash[22591]: cluster 2024-09-10T18:42:14.385022+0000 mgr.y (mgr.14150) 110 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:15 smithi165 bash[22591]: audit 2024-09-10T18:42:14.871767+0000 mon.a (mon.0) 265 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:42:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:15 smithi165 bash[22591]: cluster 2024-09-10T18:42:14.871864+0000 mon.a (mon.0) 266 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:42:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:15 smithi165 bash[22591]: audit 2024-09-10T18:42:14.872036+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:15 smithi165 bash[22591]: audit 2024-09-10T18:42:14.872522+0000 mon.a (mon.0) 268 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:42:16.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.530+0000 7f8629ffb640 10 client.24122.objecter tick 2024-09-10T18:42:16.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86297fa640 1 -- 172.21.15.83:0/3784279661 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 16467172) 0x7f8618000bc0 con 0x7f86140756f0 2024-09-10T18:42:16.911 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 0 on host 'smithi083' 2024-09-10T18:42:16.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 10 librados: watch_flush enter 2024-09-10T18:42:16.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 10 librados: watch_flush exit 2024-09-10T18:42:16.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 20 client.24122.objecter shutdown clearing up homeless session... 2024-09-10T18:42:16.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 10 client.24122.objecter successfully canceled tick 2024-09-10T18:42:16.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 >> v1:172.21.15.83:6800/2630800463 conn(0x7f86140756f0 legacy=0x7f8614077ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:42:16.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.906+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 >> v1:172.21.15.83:6790/0 conn(0x7f862c1046b0 legacy=0x7f862c116420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:42:16.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.910+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 shutdown_connections 2024-09-10T18:42:16.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.910+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 >> 172.21.15.83:0/3784279661 conn(0x7f862c0f6c60 msgr2=0x7f862c106b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:42:16.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.910+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 shutdown_connections 2024-09-10T18:42:16.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.910+0000 7f86330b0640 1 -- 172.21.15.83:0/3784279661 wait complete. 2024-09-10T18:42:16.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:16.910+0000 7f86330b0640 1 librados: shutdown 2024-09-10T18:42:17.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[18629]: audit 2024-09-10T18:42:15.874402+0000 mon.a (mon.0) 269 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[18629]: cluster 2024-09-10T18:42:15.874536+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[18629]: audit 2024-09-10T18:42:15.875136+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[18629]: audit 2024-09-10T18:42:15.885160+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[32381]: audit 2024-09-10T18:42:15.874402+0000 mon.a (mon.0) 269 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[32381]: cluster 2024-09-10T18:42:15.874536+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[32381]: audit 2024-09-10T18:42:15.875136+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:16 smithi083 bash[32381]: audit 2024-09-10T18:42:15.885160+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:16 smithi165 bash[22591]: audit 2024-09-10T18:42:15.874402+0000 mon.a (mon.0) 269 : audit [INF] from='osd.0 v1:172.21.15.83:6801/2535527483' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:42:17.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:16 smithi165 bash[22591]: cluster 2024-09-10T18:42:15.874536+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:42:17.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:16 smithi165 bash[22591]: audit 2024-09-10T18:42:15.875136+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:16 smithi165 bash[22591]: audit 2024-09-10T18:42:15.885160+0000 mon.a (mon.0) 272 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:17.529 DEBUG:teuthology.orchestra.run.smithi083:osd.0> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T18:42:17.532 INFO:tasks.cephadm:Deploying osd.1 on smithi083 with /dev/vg_nvme/lv_3... 2024-09-10T18:42:17.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T18:42:17.811 INFO:journalctl@ceph.osd.0.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: cluster 2024-09-10T18:42:14.943725+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: cluster 2024-09-10T18:42:14.944161+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: cluster 2024-09-10T18:42:16.385492+0000 mgr.y (mgr.14150) 111 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: cluster 2024-09-10T18:42:16.885951+0000 mon.a (mon.0) 273 : cluster [INF] osd.0 v1:172.21.15.83:6801/2535527483 boot 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: cluster 2024-09-10T18:42:16.886011+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: audit 2024-09-10T18:42:16.886632+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: audit 2024-09-10T18:42:16.887724+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: audit 2024-09-10T18:42:16.900215+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[18629]: audit 2024-09-10T18:42:16.907061+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: cluster 2024-09-10T18:42:14.943725+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: cluster 2024-09-10T18:42:14.944161+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: cluster 2024-09-10T18:42:16.385492+0000 mgr.y (mgr.14150) 111 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: cluster 2024-09-10T18:42:16.885951+0000 mon.a (mon.0) 273 : cluster [INF] osd.0 v1:172.21.15.83:6801/2535527483 boot 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: cluster 2024-09-10T18:42:16.886011+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: audit 2024-09-10T18:42:16.886632+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: audit 2024-09-10T18:42:16.887724+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: audit 2024-09-10T18:42:16.900215+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:17 smithi083 bash[32381]: audit 2024-09-10T18:42:16.907061+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: cluster 2024-09-10T18:42:14.943725+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: cluster 2024-09-10T18:42:14.944161+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: cluster 2024-09-10T18:42:16.385492+0000 mgr.y (mgr.14150) 111 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: cluster 2024-09-10T18:42:16.885951+0000 mon.a (mon.0) 273 : cluster [INF] osd.0 v1:172.21.15.83:6801/2535527483 boot 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: cluster 2024-09-10T18:42:16.886011+0000 mon.a (mon.0) 274 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: audit 2024-09-10T18:42:16.886632+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: audit 2024-09-10T18:42:16.887724+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:42:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: audit 2024-09-10T18:42:16.900215+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:18.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:17 smithi165 bash[22591]: audit 2024-09-10T18:42:16.907061+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:19.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:18 smithi083 bash[18629]: cluster 2024-09-10T18:42:17.910734+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:42:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:18 smithi083 bash[32381]: cluster 2024-09-10T18:42:17.910734+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:42:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:18 smithi165 bash[22591]: cluster 2024-09-10T18:42:17.910734+0000 mon.a (mon.0) 279 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:42:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:19 smithi083 bash[18629]: cluster 2024-09-10T18:42:18.385963+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:19 smithi083 bash[32381]: cluster 2024-09-10T18:42:18.385963+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:20.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:19 smithi165 bash[22591]: cluster 2024-09-10T18:42:18.385963+0000 mgr.y (mgr.14150) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:21.667 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:22.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:21 smithi083 bash[18629]: cluster 2024-09-10T18:42:20.386453+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:22.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:21 smithi083 bash[32381]: cluster 2024-09-10T18:42:20.386453+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:22.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:21 smithi165 bash[22591]: cluster 2024-09-10T18:42:20.386453+0000 mgr.y (mgr.14150) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:24.059 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:23 smithi083 bash[18629]: cluster 2024-09-10T18:42:22.386922+0000 mgr.y (mgr.14150) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:24.059 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:23 smithi083 bash[32381]: cluster 2024-09-10T18:42:22.386922+0000 mgr.y (mgr.14150) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:24.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:23 smithi165 bash[22591]: cluster 2024-09-10T18:42:22.386922+0000 mgr.y (mgr.14150) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:26.194 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:25 smithi083 bash[32381]: cluster 2024-09-10T18:42:24.387421+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:26.194 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:25 smithi083 bash[18629]: cluster 2024-09-10T18:42:24.387421+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:26.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:25 smithi165 bash[22591]: cluster 2024-09-10T18:42:24.387421+0000 mgr.y (mgr.14150) 115 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:26.511 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T18:42:26.712 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T18:42:26.733 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-09-10T18:42:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-09-10T18:42:26.733 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0170804 s, 614 MB/s 2024-09-10T18:42:26.733 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-09-10T18:42:27.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-09-10T18:42:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:27 smithi083 bash[18629]: cluster 2024-09-10T18:42:26.387820+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:27 smithi083 bash[32381]: cluster 2024-09-10T18:42:26.387820+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:27 smithi165 bash[22591]: cluster 2024-09-10T18:42:26.387820+0000 mgr.y (mgr.14150) 116 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: cephadm 2024-09-10T18:42:28.208812+0000 mgr.y (mgr.14150) 117 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.217564+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.224340+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.225746+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: cephadm 2024-09-10T18:42:28.226668+0000 mgr.y (mgr.14150) 118 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 16247M 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.231917+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.234302+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.235689+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: audit 2024-09-10T18:42:28.243837+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[18629]: cluster 2024-09-10T18:42:28.388311+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: cephadm 2024-09-10T18:42:28.208812+0000 mgr.y (mgr.14150) 117 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.217564+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.224340+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.225746+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: cephadm 2024-09-10T18:42:28.226668+0000 mgr.y (mgr.14150) 118 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 16247M 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.231917+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.234302+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.235689+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: audit 2024-09-10T18:42:28.243837+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:29 smithi083 bash[32381]: cluster 2024-09-10T18:42:28.388311+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: cephadm 2024-09-10T18:42:28.208812+0000 mgr.y (mgr.14150) 117 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.217564+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.224340+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.225746+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: cephadm 2024-09-10T18:42:28.226668+0000 mgr.y (mgr.14150) 118 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 16247M 2024-09-10T18:42:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.231917+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.234302+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:29.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.235689+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:42:29.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: audit 2024-09-10T18:42:28.243837+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:42:29.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:29 smithi165 bash[22591]: cluster 2024-09-10T18:42:28.388311+0000 mgr.y (mgr.14150) 119 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:31 smithi165 bash[22591]: cluster 2024-09-10T18:42:30.388797+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:31 smithi083 bash[18629]: cluster 2024-09-10T18:42:30.388797+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:31.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:31 smithi083 bash[32381]: cluster 2024-09-10T18:42:30.388797+0000 mgr.y (mgr.14150) 120 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:32.116 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:42:33.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:33 smithi083 bash[32381]: cluster 2024-09-10T18:42:32.389301+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:33 smithi083 bash[18629]: cluster 2024-09-10T18:42:32.389301+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:33 smithi165 bash[22591]: cluster 2024-09-10T18:42:32.389301+0000 mgr.y (mgr.14150) 121 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:35.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- 172.21.15.83:0/3579560703 >> v1:172.21.15.83:6789/0 conn(0x7f68481008d0 legacy=0x7f6848102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:42:35.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- 172.21.15.83:0/3579560703 shutdown_connections 2024-09-10T18:42:35.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- 172.21.15.83:0/3579560703 >> 172.21.15.83:0/3579560703 conn(0x7f68480f6c60 msgr2=0x7f68480f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:42:35.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- 172.21.15.83:0/3579560703 shutdown_connections 2024-09-10T18:42:35.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- 172.21.15.83:0/3579560703 wait complete. 2024-09-10T18:42:35.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 librados: starting msgr at 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 librados: starting objecter 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 Processor -- start 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- start start 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 librados: setting wanted keys 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 librados: calling monclient init 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f684810d300 con 0x7f68481008d0 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6848118b00 con 0x7f68480668b0 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f68503ba640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6848119ca0 con 0x7f68481046b0 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f684e957640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f68481046b0 0x7f68481163d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51870/0 (socket says 172.21.15.83:51870) 2024-09-10T18:42:35.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f684e957640 1 -- 172.21.15.83:0/466668571 learned_addr learned my addr 172.21.15.83:0/466668571 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f684d955640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f68481008d0 0x7f6848112ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56020/0 (socket says 172.21.15.83:56020) 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter ms_handle_connect 0x7f68481046b0 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter ms_handle_connect 0x7f68481008d0 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter ms_handle_connect 0x7f68480668b0 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:42:35.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613152860 0 0) 0x7f6848119ca0 con 0x7f68481046b0 2024-09-10T18:42:35.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68300034e0 con 0x7f68481046b0 2024-09-10T18:42:35.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696397251 0 0) 0x7f684810d300 con 0x7f68481008d0 2024-09-10T18:42:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.050+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6848119ca0 con 0x7f68481008d0 2024-09-10T18:42:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332076638 0 0) 0x7f6848118b00 con 0x7f68480668b0 2024-09-10T18:42:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684810d300 con 0x7f68480668b0 2024-09-10T18:42:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817815686 0 0) 0x7f6848119ca0 con 0x7f68481008d0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6848118b00 con 0x7f68481008d0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f683c0015d0 con 0x7f68481008d0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657765211 0 0) 0x7f68300034e0 con 0x7f68481046b0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6848119ca0 con 0x7f68481046b0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6838002950 con 0x7f68481046b0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468043143 0 0) 0x7f684810d300 con 0x7f68480668b0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68300034e0 con 0x7f68480668b0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1281072745 0 0) 0x7f6848118b00 con 0x7f68481008d0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 >> v1:172.21.15.83:6790/0 conn(0x7f68481046b0 legacy=0x7f68481163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 >> v1:172.21.15.165:6789/0 conn(0x7f68480668b0 legacy=0x7f684810c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684811ae80 con 0x7f68481008d0 2024-09-10T18:42:35.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6848117b50 con 0x7f68481008d0 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f68503ba640 10 client.14241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6848118020 con 0x7f68481008d0 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f68503ba640 1 librados: init done 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f683c0011f0 con 0x7f68481008d0 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f683c003720 con 0x7f68481008d0 2024-09-10T18:42:35.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6834005040 con 0x7f68481008d0 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.054+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 14) v1 ==== 90689+0+0 (unknown 3087547402 0 0) 0x7f683c001880 con 0x7f68481008d0 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1537+0+0 (unknown 3419078859 0 0) 0x7f683c088d30 con 0x7f68481008d0 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 10 client.14241.objecter ms_dispatch 0x7f6848000b90 osd_map(9..9 src has 1..9) v4 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 3 client.14241.objecter handle_osd_map got epochs [9,9] > 0 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 3 client.14241.objecter handle_osd_map decoding full epoch 9 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 20 client.14241.objecter dump_active .. 0 homeless 2024-09-10T18:42:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.058+0000 7f6846ffd640 10 client.14241.objecter ms_handle_connect 0x7f6830071540 2024-09-10T18:42:35.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.062+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f683c056d10 con 0x7f68481008d0 2024-09-10T18:42:35.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:35.242+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6834000bc0 con 0x7f6830071540 2024-09-10T18:42:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[18629]: cluster 2024-09-10T18:42:34.389814+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[18629]: audit 2024-09-10T18:42:35.256673+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[18629]: audit 2024-09-10T18:42:35.261957+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[18629]: audit 2024-09-10T18:42:35.263203+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[32381]: cluster 2024-09-10T18:42:34.389814+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[32381]: audit 2024-09-10T18:42:35.256673+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[32381]: audit 2024-09-10T18:42:35.261957+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:42:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:35 smithi083 bash[32381]: audit 2024-09-10T18:42:35.263203+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:35 smithi165 bash[22591]: cluster 2024-09-10T18:42:34.389814+0000 mgr.y (mgr.14150) 122 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:35 smithi165 bash[22591]: audit 2024-09-10T18:42:35.256673+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:42:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:35 smithi165 bash[22591]: audit 2024-09-10T18:42:35.261957+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:42:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:35 smithi165 bash[22591]: audit 2024-09-10T18:42:35.263203+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:36 smithi083 bash[18629]: audit 2024-09-10T18:42:35.250668+0000 mgr.y (mgr.14150) 123 : audit [DBG] from='client.14241 v1:172.21.15.83:0/466668571' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:42:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:36 smithi083 bash[32381]: audit 2024-09-10T18:42:35.250668+0000 mgr.y (mgr.14150) 123 : audit [DBG] from='client.14241 v1:172.21.15.83:0/466668571' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:42:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:36 smithi165 bash[22591]: audit 2024-09-10T18:42:35.250668+0000 mgr.y (mgr.14150) 123 : audit [DBG] from='client.14241 v1:172.21.15.83:0/466668571' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:42:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:37 smithi083 bash[18629]: cluster 2024-09-10T18:42:36.390297+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:37 smithi083 bash[32381]: cluster 2024-09-10T18:42:36.390297+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:37 smithi165 bash[22591]: cluster 2024-09-10T18:42:36.390297+0000 mgr.y (mgr.14150) 124 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:39 smithi083 bash[18629]: cluster 2024-09-10T18:42:38.390781+0000 mgr.y (mgr.14150) 125 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:39 smithi083 bash[32381]: cluster 2024-09-10T18:42:38.390781+0000 mgr.y (mgr.14150) 125 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:39 smithi165 bash[22591]: cluster 2024-09-10T18:42:38.390781+0000 mgr.y (mgr.14150) 125 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:40.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:40.058+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:42:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:41 smithi083 bash[18629]: cluster 2024-09-10T18:42:40.391204+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:41 smithi083 bash[32381]: cluster 2024-09-10T18:42:40.391204+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:41 smithi165 bash[22591]: cluster 2024-09-10T18:42:40.391204+0000 mgr.y (mgr.14150) 126 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:43 smithi083 bash[18629]: cluster 2024-09-10T18:42:42.391717+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:43.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:43 smithi083 bash[32381]: cluster 2024-09-10T18:42:42.391717+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:43 smithi165 bash[22591]: cluster 2024-09-10T18:42:42.391717+0000 mgr.y (mgr.14150) 127 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: audit 2024-09-10T18:42:43.653402+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.83:0/3932552664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: audit 2024-09-10T18:42:43.653852+0000 mon.a (mon.0) 290 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: audit 2024-09-10T18:42:43.661310+0000 mon.a (mon.0) 291 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]': finished 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: cluster 2024-09-10T18:42:43.661448+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: audit 2024-09-10T18:42:43.661600+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[18629]: audit 2024-09-10T18:42:44.346182+0000 mon.a (mon.0) 294 : audit [DBG] from='client.? v1:172.21.15.83:0/3173433093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:42:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: audit 2024-09-10T18:42:43.653402+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.83:0/3932552664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: audit 2024-09-10T18:42:43.653852+0000 mon.a (mon.0) 290 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: audit 2024-09-10T18:42:43.661310+0000 mon.a (mon.0) 291 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]': finished 2024-09-10T18:42:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: cluster 2024-09-10T18:42:43.661448+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:42:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: audit 2024-09-10T18:42:43.661600+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:42:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:44 smithi083 bash[32381]: audit 2024-09-10T18:42:44.346182+0000 mon.a (mon.0) 294 : audit [DBG] from='client.? v1:172.21.15.83:0/3173433093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:42:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: audit 2024-09-10T18:42:43.653402+0000 mon.c (mon.2) 4 : audit [INF] from='client.? v1:172.21.15.83:0/3932552664' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: audit 2024-09-10T18:42:43.653852+0000 mon.a (mon.0) 290 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]: dispatch 2024-09-10T18:42:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: audit 2024-09-10T18:42:43.661310+0000 mon.a (mon.0) 291 : audit [INF] from='client.24155 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad"}]': finished 2024-09-10T18:42:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: cluster 2024-09-10T18:42:43.661448+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:42:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: audit 2024-09-10T18:42:43.661600+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:42:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:44 smithi165 bash[22591]: audit 2024-09-10T18:42:44.346182+0000 mon.a (mon.0) 294 : audit [DBG] from='client.? v1:172.21.15.83:0/3173433093' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:42:45.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:45.062+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:42:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:45 smithi083 bash[18629]: cluster 2024-09-10T18:42:44.392135+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:45 smithi083 bash[32381]: cluster 2024-09-10T18:42:44.392135+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:45 smithi165 bash[22591]: cluster 2024-09-10T18:42:44.392135+0000 mgr.y (mgr.14150) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:47 smithi083 bash[18629]: cluster 2024-09-10T18:42:46.392632+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:47 smithi083 bash[32381]: cluster 2024-09-10T18:42:46.392632+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:47 smithi165 bash[22591]: cluster 2024-09-10T18:42:46.392632+0000 mgr.y (mgr.14150) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:49 smithi083 bash[18629]: cluster 2024-09-10T18:42:48.393147+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:49 smithi083 bash[32381]: cluster 2024-09-10T18:42:48.393147+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:50.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:50.065+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:42:50.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:49 smithi165 bash[22591]: cluster 2024-09-10T18:42:48.393147+0000 mgr.y (mgr.14150) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:51 smithi165 bash[22591]: cluster 2024-09-10T18:42:50.393635+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:51 smithi083 bash[18629]: cluster 2024-09-10T18:42:50.393635+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:51.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:51 smithi083 bash[32381]: cluster 2024-09-10T18:42:50.393635+0000 mgr.y (mgr.14150) 131 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:53.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:53 smithi083 bash[18629]: cluster 2024-09-10T18:42:52.394108+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:53.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:53 smithi083 bash[32381]: cluster 2024-09-10T18:42:52.394108+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:53 smithi165 bash[22591]: cluster 2024-09-10T18:42:52.394108+0000 mgr.y (mgr.14150) 132 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:55.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:42:55.069+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:42:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:55 smithi083 bash[18629]: cluster 2024-09-10T18:42:54.394642+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:55 smithi083 bash[32381]: cluster 2024-09-10T18:42:54.394642+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:55 smithi165 bash[22591]: cluster 2024-09-10T18:42:54.394642+0000 mgr.y (mgr.14150) 133 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:56 smithi083 bash[18629]: audit 2024-09-10T18:42:56.261294+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:42:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:56 smithi083 bash[18629]: audit 2024-09-10T18:42:56.262757+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:56 smithi083 bash[32381]: audit 2024-09-10T18:42:56.261294+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:42:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:56 smithi083 bash[32381]: audit 2024-09-10T18:42:56.262757+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:56 smithi165 bash[22591]: audit 2024-09-10T18:42:56.261294+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:42:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:56 smithi165 bash[22591]: audit 2024-09-10T18:42:56.262757+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:42:57.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:57 smithi083 bash[18629]: cephadm 2024-09-10T18:42:56.264088+0000 mgr.y (mgr.14150) 134 : cephadm [INF] Deploying daemon osd.1 on smithi083 2024-09-10T18:42:57.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:57 smithi083 bash[18629]: cluster 2024-09-10T18:42:56.395036+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:57 smithi083 bash[32381]: cephadm 2024-09-10T18:42:56.264088+0000 mgr.y (mgr.14150) 134 : cephadm [INF] Deploying daemon osd.1 on smithi083 2024-09-10T18:42:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:57 smithi083 bash[32381]: cluster 2024-09-10T18:42:56.395036+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:57 smithi165 bash[22591]: cephadm 2024-09-10T18:42:56.264088+0000 mgr.y (mgr.14150) 134 : cephadm [INF] Deploying daemon osd.1 on smithi083 2024-09-10T18:42:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:57 smithi165 bash[22591]: cluster 2024-09-10T18:42:56.395036+0000 mgr.y (mgr.14150) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:59.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:42:59 smithi083 bash[18629]: cluster 2024-09-10T18:42:58.395460+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:59.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:42:59 smithi083 bash[32381]: cluster 2024-09-10T18:42:58.395460+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:42:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:42:59 smithi165 bash[22591]: cluster 2024-09-10T18:42:58.395460+0000 mgr.y (mgr.14150) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:00.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:00.073+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:43:01.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[18629]: cluster 2024-09-10T18:43:00.395788+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[18629]: audit 2024-09-10T18:43:00.771083+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[18629]: audit 2024-09-10T18:43:00.777212+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[18629]: audit 2024-09-10T18:43:00.782614+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[32381]: cluster 2024-09-10T18:43:00.395788+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[32381]: audit 2024-09-10T18:43:00.771083+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[32381]: audit 2024-09-10T18:43:00.777212+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:01 smithi083 bash[32381]: audit 2024-09-10T18:43:00.782614+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:01 smithi165 bash[22591]: cluster 2024-09-10T18:43:00.395788+0000 mgr.y (mgr.14150) 137 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:01 smithi165 bash[22591]: audit 2024-09-10T18:43:00.771083+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:01 smithi165 bash[22591]: audit 2024-09-10T18:43:00.777212+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:01 smithi165 bash[22591]: audit 2024-09-10T18:43:00.782614+0000 mon.a (mon.0) 299 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:03.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:03 smithi083 bash[18629]: cluster 2024-09-10T18:43:02.396362+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:03.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:03 smithi083 bash[32381]: cluster 2024-09-10T18:43:02.396362+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:03 smithi165 bash[22591]: cluster 2024-09-10T18:43:02.396362+0000 mgr.y (mgr.14150) 138 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:05.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:05.077+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:43:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:05 smithi083 bash[18629]: cluster 2024-09-10T18:43:04.396933+0000 mgr.y (mgr.14150) 139 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:05 smithi083 bash[32381]: cluster 2024-09-10T18:43:04.396933+0000 mgr.y (mgr.14150) 139 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:05 smithi165 bash[22591]: cluster 2024-09-10T18:43:04.396933+0000 mgr.y (mgr.14150) 139 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: cluster 2024-09-10T18:43:06.397487+0000 mgr.y (mgr.14150) 140 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: audit 2024-09-10T18:43:07.345252+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: audit 2024-09-10T18:43:07.350995+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: audit 2024-09-10T18:43:07.352282+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: audit 2024-09-10T18:43:07.353806+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[18629]: audit 2024-09-10T18:43:07.361649+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: cluster 2024-09-10T18:43:06.397487+0000 mgr.y (mgr.14150) 140 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: audit 2024-09-10T18:43:07.345252+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: audit 2024-09-10T18:43:07.350995+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: audit 2024-09-10T18:43:07.352282+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: audit 2024-09-10T18:43:07.353806+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:07 smithi083 bash[32381]: audit 2024-09-10T18:43:07.361649+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: cluster 2024-09-10T18:43:06.397487+0000 mgr.y (mgr.14150) 140 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: audit 2024-09-10T18:43:07.345252+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: audit 2024-09-10T18:43:07.350995+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: audit 2024-09-10T18:43:07.352282+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: audit 2024-09-10T18:43:07.353806+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:07 smithi165 bash[22591]: audit 2024-09-10T18:43:07.361649+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:08 smithi083 bash[18629]: audit 2024-09-10T18:43:07.603238+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:43:08.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:08 smithi083 bash[32381]: audit 2024-09-10T18:43:07.603238+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:43:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:08 smithi165 bash[22591]: audit 2024-09-10T18:43:07.603238+0000 mon.a (mon.0) 305 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:43:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[18629]: cluster 2024-09-10T18:43:08.397993+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[18629]: audit 2024-09-10T18:43:08.498561+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[18629]: cluster 2024-09-10T18:43:08.498621+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[18629]: audit 2024-09-10T18:43:08.498826+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[18629]: audit 2024-09-10T18:43:08.499591+0000 mon.a (mon.0) 309 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[32381]: cluster 2024-09-10T18:43:08.397993+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[32381]: audit 2024-09-10T18:43:08.498561+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[32381]: cluster 2024-09-10T18:43:08.498621+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[32381]: audit 2024-09-10T18:43:08.498826+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:09 smithi083 bash[32381]: audit 2024-09-10T18:43:08.499591+0000 mon.a (mon.0) 309 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:43:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:09 smithi165 bash[22591]: cluster 2024-09-10T18:43:08.397993+0000 mgr.y (mgr.14150) 141 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:09 smithi165 bash[22591]: audit 2024-09-10T18:43:08.498561+0000 mon.a (mon.0) 306 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:43:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:09 smithi165 bash[22591]: cluster 2024-09-10T18:43:08.498621+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:43:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:09 smithi165 bash[22591]: audit 2024-09-10T18:43:08.498826+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:09 smithi165 bash[22591]: audit 2024-09-10T18:43:08.499591+0000 mon.a (mon.0) 309 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:43:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:10.081+0000 7f68477fe640 10 client.14241.objecter tick 2024-09-10T18:43:10.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[18629]: audit 2024-09-10T18:43:09.500459+0000 mon.a (mon.0) 310 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:43:10.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[18629]: cluster 2024-09-10T18:43:09.500566+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[18629]: audit 2024-09-10T18:43:09.501221+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[18629]: audit 2024-09-10T18:43:09.509485+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[32381]: audit 2024-09-10T18:43:09.500459+0000 mon.a (mon.0) 310 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[32381]: cluster 2024-09-10T18:43:09.500566+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[32381]: audit 2024-09-10T18:43:09.501221+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:10.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:10 smithi083 bash[32381]: audit 2024-09-10T18:43:09.509485+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:10 smithi165 bash[22591]: audit 2024-09-10T18:43:09.500459+0000 mon.a (mon.0) 310 : audit [INF] from='osd.1 v1:172.21.15.83:6805/2309624637' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:43:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:10 smithi165 bash[22591]: cluster 2024-09-10T18:43:09.500566+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:43:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:10 smithi165 bash[22591]: audit 2024-09-10T18:43:09.501221+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:10 smithi165 bash[22591]: audit 2024-09-10T18:43:09.509485+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:11.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.097+0000 7f6846ffd640 1 -- 172.21.15.83:0/466668571 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1559145040) 0x7f6834000bc0 con 0x7f6830071540 2024-09-10T18:43:11.102 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 1 on host 'smithi083' 2024-09-10T18:43:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 10 librados: watch_flush enter 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 10 librados: watch_flush exit 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 20 client.14241.objecter shutdown clearing up homeless session... 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 10 client.14241.objecter successfully canceled tick 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 >> v1:172.21.15.83:6800/2630800463 conn(0x7f6830071540 legacy=0x7f68300739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 >> v1:172.21.15.83:6789/0 conn(0x7f68481008d0 legacy=0x7f6848112ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.101+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 shutdown_connections 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.105+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 >> 172.21.15.83:0/466668571 conn(0x7f68480f6c60 msgr2=0x7f6848104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:43:11.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.105+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 shutdown_connections 2024-09-10T18:43:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.105+0000 7f68503ba640 1 -- 172.21.15.83:0/466668571 wait complete. 2024-09-10T18:43:11.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:11.105+0000 7f68503ba640 1 librados: shutdown 2024-09-10T18:43:11.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: cluster 2024-09-10T18:43:08.624331+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: cluster 2024-09-10T18:43:08.624441+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: cluster 2024-09-10T18:43:10.398581+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: cluster 2024-09-10T18:43:10.511906+0000 mon.a (mon.0) 314 : cluster [INF] osd.1 v1:172.21.15.83:6805/2309624637 boot 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: cluster 2024-09-10T18:43:10.511995+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: audit 2024-09-10T18:43:10.512558+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: audit 2024-09-10T18:43:11.084332+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: audit 2024-09-10T18:43:11.091070+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[18629]: audit 2024-09-10T18:43:11.098876+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.788 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: cluster 2024-09-10T18:43:08.624331+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: cluster 2024-09-10T18:43:08.624441+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: cluster 2024-09-10T18:43:10.398581+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: cluster 2024-09-10T18:43:10.511906+0000 mon.a (mon.0) 314 : cluster [INF] osd.1 v1:172.21.15.83:6805/2309624637 boot 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: cluster 2024-09-10T18:43:10.511995+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: audit 2024-09-10T18:43:10.512558+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: audit 2024-09-10T18:43:11.084332+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: audit 2024-09-10T18:43:11.091070+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:11 smithi083 bash[32381]: audit 2024-09-10T18:43:11.098876+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.790 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T18:43:11.793 INFO:tasks.cephadm:Deploying osd.2 on smithi083 with /dev/vg_nvme/lv_2... 2024-09-10T18:43:11.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T18:43:11.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: cluster 2024-09-10T18:43:08.624331+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: cluster 2024-09-10T18:43:08.624441+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: cluster 2024-09-10T18:43:10.398581+0000 mgr.y (mgr.14150) 142 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: cluster 2024-09-10T18:43:10.511906+0000 mon.a (mon.0) 314 : cluster [INF] osd.1 v1:172.21.15.83:6805/2309624637 boot 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: cluster 2024-09-10T18:43:10.511995+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: audit 2024-09-10T18:43:10.512558+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: audit 2024-09-10T18:43:11.084332+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: audit 2024-09-10T18:43:11.091070+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:11 smithi165 bash[22591]: audit 2024-09-10T18:43:11.098876+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:11.870 INFO:journalctl@ceph.osd.1.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:43:12.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:12 smithi083 bash[18629]: cluster 2024-09-10T18:43:11.529483+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:43:12.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:12 smithi083 bash[32381]: cluster 2024-09-10T18:43:11.529483+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:43:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:12 smithi165 bash[22591]: cluster 2024-09-10T18:43:11.529483+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:43:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:13 smithi083 bash[18629]: cluster 2024-09-10T18:43:12.399121+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:13 smithi083 bash[32381]: cluster 2024-09-10T18:43:12.399121+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:13 smithi165 bash[22591]: cluster 2024-09-10T18:43:12.399121+0000 mgr.y (mgr.14150) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:15.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:15 smithi083 bash[18629]: cluster 2024-09-10T18:43:14.399610+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:15 smithi083 bash[32381]: cluster 2024-09-10T18:43:14.399610+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:15 smithi165 bash[22591]: cluster 2024-09-10T18:43:14.399610+0000 mgr.y (mgr.14150) 144 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:15.863 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:17.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:17 smithi083 bash[18629]: cluster 2024-09-10T18:43:16.400097+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:17 smithi165 bash[22591]: cluster 2024-09-10T18:43:16.400097+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:17.942 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:17 smithi083 bash[32381]: cluster 2024-09-10T18:43:16.400097+0000 mgr.y (mgr.14150) 145 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:19 smithi165 bash[22591]: cluster 2024-09-10T18:43:18.400589+0000 mgr.y (mgr.14150) 146 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:19 smithi083 bash[18629]: cluster 2024-09-10T18:43:18.400589+0000 mgr.y (mgr.14150) 146 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:19 smithi083 bash[32381]: cluster 2024-09-10T18:43:18.400589+0000 mgr.y (mgr.14150) 146 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T18:43:20.757 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T18:43:20.780 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-09-10T18:43:20.780 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-09-10T18:43:20.780 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0190602 s, 550 MB/s 2024-09-10T18:43:20.780 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-09-10T18:43:21.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-09-10T18:43:21.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:21 smithi083 bash[18629]: cluster 2024-09-10T18:43:20.400936+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:21 smithi083 bash[32381]: cluster 2024-09-10T18:43:20.400936+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:21.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:21 smithi165 bash[22591]: cluster 2024-09-10T18:43:20.400936+0000 mgr.y (mgr.14150) 147 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: cluster 2024-09-10T18:43:22.401369+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: cephadm 2024-09-10T18:43:22.487161+0000 mgr.y (mgr.14150) 149 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.495071+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.502454+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.504102+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.505424+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: cephadm 2024-09-10T18:43:22.506402+0000 mgr.y (mgr.14150) 150 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 8123M 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.512546+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.514923+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.516167+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[18629]: audit 2024-09-10T18:43:22.523057+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: cluster 2024-09-10T18:43:22.401369+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: cephadm 2024-09-10T18:43:22.487161+0000 mgr.y (mgr.14150) 149 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.495071+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.502454+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.504102+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.505424+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: cephadm 2024-09-10T18:43:22.506402+0000 mgr.y (mgr.14150) 150 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 8123M 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.512546+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.514923+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.516167+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:23 smithi083 bash[32381]: audit 2024-09-10T18:43:22.523057+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: cluster 2024-09-10T18:43:22.401369+0000 mgr.y (mgr.14150) 148 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: cephadm 2024-09-10T18:43:22.487161+0000 mgr.y (mgr.14150) 149 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.495071+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.502454+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.504102+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.505424+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: cephadm 2024-09-10T18:43:22.506402+0000 mgr.y (mgr.14150) 150 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 8123M 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.512546+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.514923+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.516167+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:43:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:23 smithi165 bash[22591]: audit 2024-09-10T18:43:22.523057+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:25.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:25 smithi083 bash[18629]: cluster 2024-09-10T18:43:24.401918+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:25 smithi083 bash[32381]: cluster 2024-09-10T18:43:24.401918+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:25 smithi165 bash[22591]: cluster 2024-09-10T18:43:24.401918+0000 mgr.y (mgr.14150) 151 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:43:27.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:27 smithi083 bash[18629]: cluster 2024-09-10T18:43:26.402525+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:27.658 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:27 smithi083 bash[32381]: cluster 2024-09-10T18:43:26.402525+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:27 smithi165 bash[22591]: cluster 2024-09-10T18:43:26.402525+0000 mgr.y (mgr.14150) 152 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:28.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[32381]: cluster 2024-09-10T18:43:27.231450+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:28.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[32381]: cluster 2024-09-10T18:43:27.231569+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:28.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[32381]: cluster 2024-09-10T18:43:27.509683+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-10T18:43:28.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[18629]: cluster 2024-09-10T18:43:27.231450+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:28.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[18629]: cluster 2024-09-10T18:43:27.231569+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:28.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:28 smithi083 bash[18629]: cluster 2024-09-10T18:43:27.509683+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-10T18:43:28.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:28 smithi165 bash[22591]: cluster 2024-09-10T18:43:27.231450+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:43:28.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:28 smithi165 bash[22591]: cluster 2024-09-10T18:43:27.231569+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:43:28.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:28 smithi165 bash[22591]: cluster 2024-09-10T18:43:27.509683+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e15: 2 total, 2 up, 2 in 2024-09-10T18:43:29.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- 172.21.15.83:0/1258284755 >> v1:172.21.15.83:6789/0 conn(0x7fcc200fdbf0 legacy=0x7fcc200fe000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:43:29.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- 172.21.15.83:0/1258284755 shutdown_connections 2024-09-10T18:43:29.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- 172.21.15.83:0/1258284755 >> 172.21.15.83:0/1258284755 conn(0x7fcc200f63f0 msgr2=0x7fcc200f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:43:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- 172.21.15.83:0/1258284755 shutdown_connections 2024-09-10T18:43:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- 172.21.15.83:0/1258284755 wait complete. 2024-09-10T18:43:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 librados: starting msgr at 2024-09-10T18:43:29.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 librados: starting objecter 2024-09-10T18:43:29.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 Processor -- start 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- start start 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 librados: setting wanted keys 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 librados: calling monclient init 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc20119a10 con 0x7fcc20100bf0 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc2011ac10 con 0x7fcc201046f0 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc2545e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc2011be10 con 0x7fcc200fdbf0 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc1effd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fcc200fdbf0 0x7fcc200fd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38036/0 (socket says 172.21.15.83:38036) 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc1effd640 1 -- 172.21.15.83:0/389213199 learned_addr learned my addr 172.21.15.83:0/389213199 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc1e7fc640 1 --1- 172.21.15.83:0/389213199 >> v1:172.21.15.83:6789/0 conn(0x7fcc20100bf0 0x7fcc201148b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60752/0 (socket says 172.21.15.83:60752) 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter ms_handle_connect 0x7fcc200fdbf0 2024-09-10T18:43:29.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter ms_handle_connect 0x7fcc20100bf0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter ms_handle_connect 0x7fcc201046f0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629515073 0 0) 0x7fcc2011be10 con 0x7fcc200fdbf0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c0034e0 con 0x7fcc200fdbf0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260957842 0 0) 0x7fcc20119a10 con 0x7fcc20100bf0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2011be10 con 0x7fcc20100bf0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668223647 0 0) 0x7fcc2011ac10 con 0x7fcc201046f0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc20119a10 con 0x7fcc201046f0 2024-09-10T18:43:29.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1173109832 0 0) 0x7fcc0c0034e0 con 0x7fcc200fdbf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc2011ac10 con 0x7fcc200fdbf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 555817506 0 0) 0x7fcc2011be10 con 0x7fcc20100bf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0c0034e0 con 0x7fcc20100bf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301493348 0 0) 0x7fcc20119a10 con 0x7fcc201046f0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc2011be10 con 0x7fcc201046f0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.484+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc18002c00 con 0x7fcc200fdbf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc08001c40 con 0x7fcc20100bf0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc100017c0 con 0x7fcc201046f0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1719257691 0 0) 0x7fcc2011be10 con 0x7fcc201046f0 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 >> v1:172.21.15.83:6790/0 conn(0x7fcc200fdbf0 legacy=0x7fcc200fd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 >> v1:172.21.15.83:6789/0 conn(0x7fcc20100bf0 legacy=0x7fcc201148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:43:29.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2011d010 con 0x7fcc201046f0 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc20119c00 con 0x7fcc201046f0 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc2545e640 10 client.24175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2011a0d0 con 0x7fcc201046f0 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc2545e640 1 librados: init done 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fcc10002130 con 0x7fcc201046f0 2024-09-10T18:43:29.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc10003070 con 0x7fcc201046f0 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc00005040 con 0x7fcc201046f0 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 14) v1 ==== 90689+0+0 (unknown 3087547402 0 0) 0x7fcc10001cd0 con 0x7fcc201046f0 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(15..15 src has 1..15) v4 ==== 1829+0+0 (unknown 388525272 0 0) 0x7fcc100860d0 con 0x7fcc201046f0 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 10 client.24175.objecter ms_dispatch 0x7fcc20000b90 osd_map(15..15 src has 1..15) v4 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 3 client.24175.objecter handle_osd_map got epochs [15,15] > 0 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 3 client.24175.objecter handle_osd_map decoding full epoch 15 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.488+0000 7fcc177fe640 20 client.24175.objecter dump_active .. 0 homeless 2024-09-10T18:43:29.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.492+0000 7fcc177fe640 10 client.24175.objecter ms_handle_connect 0x7fcc0c071630 2024-09-10T18:43:29.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.500+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fcc10053f90 con 0x7fcc201046f0 2024-09-10T18:43:29.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:29 smithi083 bash[32381]: cluster 2024-09-10T18:43:28.403071+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:29.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:29 smithi083 bash[18629]: cluster 2024-09-10T18:43:28.403071+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:29.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:29.684+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc00000bc0 con 0x7fcc0c071630 2024-09-10T18:43:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:29 smithi165 bash[22591]: cluster 2024-09-10T18:43:28.403071+0000 mgr.y (mgr.14150) 153 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:30.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[18629]: audit 2024-09-10T18:43:29.690529+0000 mgr.y (mgr.14150) 154 : audit [DBG] from='client.24175 v1:172.21.15.83:0/389213199' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:43:30.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[18629]: audit 2024-09-10T18:43:29.693971+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[18629]: audit 2024-09-10T18:43:29.699419+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[18629]: audit 2024-09-10T18:43:29.700666+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[32381]: audit 2024-09-10T18:43:29.690529+0000 mgr.y (mgr.14150) 154 : audit [DBG] from='client.24175 v1:172.21.15.83:0/389213199' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[32381]: audit 2024-09-10T18:43:29.693971+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[32381]: audit 2024-09-10T18:43:29.699419+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:43:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:30 smithi083 bash[32381]: audit 2024-09-10T18:43:29.700666+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:30 smithi165 bash[22591]: audit 2024-09-10T18:43:29.690529+0000 mgr.y (mgr.14150) 154 : audit [DBG] from='client.24175 v1:172.21.15.83:0/389213199' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:43:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:30 smithi165 bash[22591]: audit 2024-09-10T18:43:29.693971+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:43:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:30 smithi165 bash[22591]: audit 2024-09-10T18:43:29.699419+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:43:30.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:30 smithi165 bash[22591]: audit 2024-09-10T18:43:29.700666+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:31.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:31 smithi083 bash[18629]: cluster 2024-09-10T18:43:30.403564+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:31 smithi083 bash[32381]: cluster 2024-09-10T18:43:30.403564+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:31 smithi165 bash[22591]: cluster 2024-09-10T18:43:30.403564+0000 mgr.y (mgr.14150) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:33 smithi083 bash[18629]: cluster 2024-09-10T18:43:32.404056+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:33.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:33 smithi083 bash[32381]: cluster 2024-09-10T18:43:32.404056+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:33 smithi165 bash[22591]: cluster 2024-09-10T18:43:32.404056+0000 mgr.y (mgr.14150) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:34.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:34.492+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:43:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:35 smithi083 bash[18629]: cluster 2024-09-10T18:43:34.404563+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:35 smithi083 bash[32381]: cluster 2024-09-10T18:43:34.404563+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:35 smithi165 bash[22591]: cluster 2024-09-10T18:43:34.404563+0000 mgr.y (mgr.14150) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:37 smithi083 bash[18629]: cluster 2024-09-10T18:43:36.404994+0000 mgr.y (mgr.14150) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:37.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:37 smithi083 bash[32381]: cluster 2024-09-10T18:43:36.404994+0000 mgr.y (mgr.14150) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:37 smithi165 bash[22591]: cluster 2024-09-10T18:43:36.404994+0000 mgr.y (mgr.14150) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[18629]: audit 2024-09-10T18:43:38.141915+0000 mon.c (mon.2) 5 : audit [INF] from='client.? v1:172.21.15.83:0/733694898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[18629]: audit 2024-09-10T18:43:38.142544+0000 mon.a (mon.0) 333 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[18629]: audit 2024-09-10T18:43:38.147640+0000 mon.a (mon.0) 334 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]': finished 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[18629]: cluster 2024-09-10T18:43:38.147685+0000 mon.a (mon.0) 335 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[18629]: audit 2024-09-10T18:43:38.147899+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[32381]: audit 2024-09-10T18:43:38.141915+0000 mon.c (mon.2) 5 : audit [INF] from='client.? v1:172.21.15.83:0/733694898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[32381]: audit 2024-09-10T18:43:38.142544+0000 mon.a (mon.0) 333 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[32381]: audit 2024-09-10T18:43:38.147640+0000 mon.a (mon.0) 334 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]': finished 2024-09-10T18:43:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[32381]: cluster 2024-09-10T18:43:38.147685+0000 mon.a (mon.0) 335 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:43:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:38 smithi083 bash[32381]: audit 2024-09-10T18:43:38.147899+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:43:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:38 smithi165 bash[22591]: audit 2024-09-10T18:43:38.141915+0000 mon.c (mon.2) 5 : audit [INF] from='client.? v1:172.21.15.83:0/733694898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:38 smithi165 bash[22591]: audit 2024-09-10T18:43:38.142544+0000 mon.a (mon.0) 333 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]: dispatch 2024-09-10T18:43:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:38 smithi165 bash[22591]: audit 2024-09-10T18:43:38.147640+0000 mon.a (mon.0) 334 : audit [INF] from='client.24182 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6193b508-03d9-458a-b49f-6b2e24283282"}]': finished 2024-09-10T18:43:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:38 smithi165 bash[22591]: cluster 2024-09-10T18:43:38.147685+0000 mon.a (mon.0) 335 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:43:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:38 smithi165 bash[22591]: audit 2024-09-10T18:43:38.147899+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:43:39.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:39.496+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:43:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:39 smithi165 bash[22591]: cluster 2024-09-10T18:43:38.405462+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:39 smithi165 bash[22591]: audit 2024-09-10T18:43:38.805888+0000 mon.a (mon.0) 337 : audit [DBG] from='client.? v1:172.21.15.83:0/2975570816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:43:40.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:39 smithi083 bash[18629]: cluster 2024-09-10T18:43:38.405462+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:40.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:39 smithi083 bash[18629]: audit 2024-09-10T18:43:38.805888+0000 mon.a (mon.0) 337 : audit [DBG] from='client.? v1:172.21.15.83:0/2975570816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:43:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:39 smithi083 bash[32381]: cluster 2024-09-10T18:43:38.405462+0000 mgr.y (mgr.14150) 159 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:39 smithi083 bash[32381]: audit 2024-09-10T18:43:38.805888+0000 mon.a (mon.0) 337 : audit [DBG] from='client.? v1:172.21.15.83:0/2975570816' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:43:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:41 smithi083 bash[18629]: cluster 2024-09-10T18:43:40.406004+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:41.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:41 smithi083 bash[32381]: cluster 2024-09-10T18:43:40.406004+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:41 smithi165 bash[22591]: cluster 2024-09-10T18:43:40.406004+0000 mgr.y (mgr.14150) 160 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:43 smithi165 bash[22591]: cluster 2024-09-10T18:43:42.406501+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:44.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:43 smithi083 bash[18629]: cluster 2024-09-10T18:43:42.406501+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:43 smithi083 bash[32381]: cluster 2024-09-10T18:43:42.406501+0000 mgr.y (mgr.14150) 161 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:44.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:44.500+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:43:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:45 smithi165 bash[22591]: cluster 2024-09-10T18:43:44.407007+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:45 smithi083 bash[18629]: cluster 2024-09-10T18:43:44.407007+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:46.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:45 smithi083 bash[32381]: cluster 2024-09-10T18:43:44.407007+0000 mgr.y (mgr.14150) 162 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:47 smithi165 bash[22591]: cluster 2024-09-10T18:43:46.407488+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:47.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:47 smithi083 bash[18629]: cluster 2024-09-10T18:43:46.407488+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:47.910 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:47 smithi083 bash[32381]: cluster 2024-09-10T18:43:46.407488+0000 mgr.y (mgr.14150) 163 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:49.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:49.504+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:43:49.836 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:49 smithi083 bash[18629]: cluster 2024-09-10T18:43:48.407997+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:49.837 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:49 smithi083 bash[32381]: cluster 2024-09-10T18:43:48.407997+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:49 smithi165 bash[22591]: cluster 2024-09-10T18:43:48.407997+0000 mgr.y (mgr.14150) 164 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[18629]: cluster 2024-09-10T18:43:50.408451+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[18629]: audit 2024-09-10T18:43:50.897164+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[18629]: audit 2024-09-10T18:43:50.898640+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[18629]: cephadm 2024-09-10T18:43:50.899659+0000 mgr.y (mgr.14150) 166 : cephadm [INF] Deploying daemon osd.2 on smithi083 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[32381]: cluster 2024-09-10T18:43:50.408451+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[32381]: audit 2024-09-10T18:43:50.897164+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[32381]: audit 2024-09-10T18:43:50.898640+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:51 smithi083 bash[32381]: cephadm 2024-09-10T18:43:50.899659+0000 mgr.y (mgr.14150) 166 : cephadm [INF] Deploying daemon osd.2 on smithi083 2024-09-10T18:43:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:51 smithi165 bash[22591]: cluster 2024-09-10T18:43:50.408451+0000 mgr.y (mgr.14150) 165 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:51 smithi165 bash[22591]: audit 2024-09-10T18:43:50.897164+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:43:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:51 smithi165 bash[22591]: audit 2024-09-10T18:43:50.898640+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:43:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:51 smithi165 bash[22591]: cephadm 2024-09-10T18:43:50.899659+0000 mgr.y (mgr.14150) 166 : cephadm [INF] Deploying daemon osd.2 on smithi083 2024-09-10T18:43:53.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:53 smithi083 bash[18629]: cluster 2024-09-10T18:43:52.408922+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:53 smithi083 bash[32381]: cluster 2024-09-10T18:43:52.408922+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:53 smithi165 bash[22591]: cluster 2024-09-10T18:43:52.408922+0000 mgr.y (mgr.14150) 167 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:54.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:54.508+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:43:55.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:55 smithi083 bash[32381]: cluster 2024-09-10T18:43:54.409345+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:55.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:55 smithi083 bash[18629]: cluster 2024-09-10T18:43:54.409345+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:55 smithi165 bash[22591]: cluster 2024-09-10T18:43:54.409345+0000 mgr.y (mgr.14150) 168 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[18629]: audit 2024-09-10T18:43:55.665267+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[18629]: audit 2024-09-10T18:43:55.669797+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[18629]: audit 2024-09-10T18:43:55.677340+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[32381]: audit 2024-09-10T18:43:55.665267+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[32381]: audit 2024-09-10T18:43:55.669797+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:56 smithi083 bash[32381]: audit 2024-09-10T18:43:55.677340+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:56 smithi165 bash[22591]: audit 2024-09-10T18:43:55.665267+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:43:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:56 smithi165 bash[22591]: audit 2024-09-10T18:43:55.669797+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:56 smithi165 bash[22591]: audit 2024-09-10T18:43:55.677340+0000 mon.a (mon.0) 342 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:43:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:57 smithi083 bash[18629]: cluster 2024-09-10T18:43:56.409811+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:57 smithi083 bash[32381]: cluster 2024-09-10T18:43:56.409811+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:57 smithi165 bash[22591]: cluster 2024-09-10T18:43:56.409811+0000 mgr.y (mgr.14150) 169 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:43:59.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:43:59.512+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:44:00.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:43:59 smithi083 bash[18629]: cluster 2024-09-10T18:43:58.410283+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:43:59 smithi083 bash[32381]: cluster 2024-09-10T18:43:58.410283+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:43:59 smithi165 bash[22591]: cluster 2024-09-10T18:43:58.410283+0000 mgr.y (mgr.14150) 170 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:01 smithi083 bash[18629]: cluster 2024-09-10T18:44:00.410741+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:01 smithi083 bash[32381]: cluster 2024-09-10T18:44:00.410741+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:01 smithi165 bash[22591]: cluster 2024-09-10T18:44:00.410741+0000 mgr.y (mgr.14150) 171 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:03.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.011168+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.021247+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.387461+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.388698+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.394472+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[18629]: audit 2024-09-10T18:44:02.408538+0000 mon.a (mon.0) 348 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.011168+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.021247+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.387461+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:03.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.388698+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:03.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.394472+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:03 smithi083 bash[32381]: audit 2024-09-10T18:44:02.408538+0000 mon.a (mon.0) 348 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:44:03.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.011168+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.021247+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.387461+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:03.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.388698+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:03.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.394472+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:03.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:03 smithi165 bash[22591]: audit 2024-09-10T18:44:02.408538+0000 mon.a (mon.0) 348 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[18629]: cluster 2024-09-10T18:44:02.411035+0000 mgr.y (mgr.14150) 172 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[18629]: audit 2024-09-10T18:44:03.405575+0000 mon.a (mon.0) 349 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[18629]: cluster 2024-09-10T18:44:03.405724+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[18629]: audit 2024-09-10T18:44:03.405972+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[18629]: audit 2024-09-10T18:44:03.406555+0000 mon.a (mon.0) 352 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[32381]: cluster 2024-09-10T18:44:02.411035+0000 mgr.y (mgr.14150) 172 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:04.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[32381]: audit 2024-09-10T18:44:03.405575+0000 mon.a (mon.0) 349 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:44:04.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[32381]: cluster 2024-09-10T18:44:03.405724+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:44:04.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[32381]: audit 2024-09-10T18:44:03.405972+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:04.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:04 smithi083 bash[32381]: audit 2024-09-10T18:44:03.406555+0000 mon.a (mon.0) 352 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:04.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:04 smithi165 bash[22591]: cluster 2024-09-10T18:44:02.411035+0000 mgr.y (mgr.14150) 172 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:04 smithi165 bash[22591]: audit 2024-09-10T18:44:03.405575+0000 mon.a (mon.0) 349 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:44:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:04 smithi165 bash[22591]: cluster 2024-09-10T18:44:03.405724+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:44:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:04 smithi165 bash[22591]: audit 2024-09-10T18:44:03.405972+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:04 smithi165 bash[22591]: audit 2024-09-10T18:44:03.406555+0000 mon.a (mon.0) 352 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:04.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:04.516+0000 7fcc17fff640 10 client.24175.objecter tick 2024-09-10T18:44:05.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[18629]: audit 2024-09-10T18:44:04.411248+0000 mon.a (mon.0) 353 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[18629]: cluster 2024-09-10T18:44:04.411368+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[18629]: cluster 2024-09-10T18:44:04.411490+0000 mgr.y (mgr.14150) 173 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[18629]: audit 2024-09-10T18:44:04.411568+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[18629]: audit 2024-09-10T18:44:04.419171+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[32381]: audit 2024-09-10T18:44:04.411248+0000 mon.a (mon.0) 353 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[32381]: cluster 2024-09-10T18:44:04.411368+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[32381]: cluster 2024-09-10T18:44:04.411490+0000 mgr.y (mgr.14150) 173 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[32381]: audit 2024-09-10T18:44:04.411568+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:05 smithi083 bash[32381]: audit 2024-09-10T18:44:04.419171+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:05.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.840+0000 7fcc177fe640 1 -- 172.21.15.83:0/389213199 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3100678540) 0x7fcc00000bc0 con 0x7fcc0c071630 2024-09-10T18:44:05.843 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 2 on host 'smithi083' 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.840+0000 7fcc2545e640 10 librados: watch_flush enter 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.840+0000 7fcc2545e640 10 librados: watch_flush exit 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.840+0000 7fcc2545e640 20 client.24175.objecter shutdown clearing up homeless session... 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.840+0000 7fcc2545e640 10 client.24175.objecter successfully canceled tick 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 >> v1:172.21.15.83:6800/2630800463 conn(0x7fcc0c071630 legacy=0x7fcc0c073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 >> v1:172.21.15.165:6789/0 conn(0x7fcc201046f0 legacy=0x7fcc20118110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:44:05.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 shutdown_connections 2024-09-10T18:44:05.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 >> 172.21.15.83:0/389213199 conn(0x7fcc200f63f0 msgr2=0x7fcc20106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:44:05.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 shutdown_connections 2024-09-10T18:44:05.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 -- 172.21.15.83:0/389213199 wait complete. 2024-09-10T18:44:05.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:05.844+0000 7fcc2545e640 1 librados: shutdown 2024-09-10T18:44:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:05 smithi165 bash[22591]: audit 2024-09-10T18:44:04.411248+0000 mon.a (mon.0) 353 : audit [INF] from='osd.2 v1:172.21.15.83:6809/386094220' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:05 smithi165 bash[22591]: cluster 2024-09-10T18:44:04.411368+0000 mon.a (mon.0) 354 : cluster [DBG] osdmap e18: 3 total, 2 up, 3 in 2024-09-10T18:44:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:05 smithi165 bash[22591]: cluster 2024-09-10T18:44:04.411490+0000 mgr.y (mgr.14150) 173 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:44:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:05 smithi165 bash[22591]: audit 2024-09-10T18:44:04.411568+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:05 smithi165 bash[22591]: audit 2024-09-10T18:44:04.419171+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:06.487 DEBUG:teuthology.orchestra.run.smithi083:osd.2> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.2.service 2024-09-10T18:44:06.490 INFO:tasks.cephadm:Deploying osd.3 on smithi083 with /dev/vg_nvme/lv_1... 2024-09-10T18:44:06.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T18:44:06.614 INFO:journalctl@ceph.osd.2.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: cluster 2024-09-10T18:44:03.372959+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: cluster 2024-09-10T18:44:03.373079+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: cluster 2024-09-10T18:44:05.411660+0000 mon.a (mon.0) 357 : cluster [INF] osd.2 v1:172.21.15.83:6809/386094220 boot 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: cluster 2024-09-10T18:44:05.411710+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: audit 2024-09-10T18:44:05.413095+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: audit 2024-09-10T18:44:05.823547+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: audit 2024-09-10T18:44:05.830581+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[18629]: audit 2024-09-10T18:44:05.837678+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:06.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: cluster 2024-09-10T18:44:03.372959+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: cluster 2024-09-10T18:44:03.373079+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: cluster 2024-09-10T18:44:05.411660+0000 mon.a (mon.0) 357 : cluster [INF] osd.2 v1:172.21.15.83:6809/386094220 boot 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: cluster 2024-09-10T18:44:05.411710+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: audit 2024-09-10T18:44:05.413095+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: audit 2024-09-10T18:44:05.823547+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: audit 2024-09-10T18:44:05.830581+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:06 smithi083 bash[32381]: audit 2024-09-10T18:44:05.837678+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: cluster 2024-09-10T18:44:03.372959+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: cluster 2024-09-10T18:44:03.373079+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: cluster 2024-09-10T18:44:05.411660+0000 mon.a (mon.0) 357 : cluster [INF] osd.2 v1:172.21.15.83:6809/386094220 boot 2024-09-10T18:44:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: cluster 2024-09-10T18:44:05.411710+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:44:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: audit 2024-09-10T18:44:05.413095+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:44:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: audit 2024-09-10T18:44:05.823547+0000 mon.a (mon.0) 360 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: audit 2024-09-10T18:44:05.830581+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:06 smithi165 bash[22591]: audit 2024-09-10T18:44:05.837678+0000 mon.a (mon.0) 362 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[18629]: cluster 2024-09-10T18:44:06.412035+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[18629]: cluster 2024-09-10T18:44:06.436344+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:44:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[18629]: audit 2024-09-10T18:44:06.513740+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:44:07.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[32381]: cluster 2024-09-10T18:44:06.412035+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:07.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[32381]: cluster 2024-09-10T18:44:06.436344+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:44:07.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:07 smithi083 bash[32381]: audit 2024-09-10T18:44:06.513740+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:44:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:07 smithi165 bash[22591]: cluster 2024-09-10T18:44:06.412035+0000 mgr.y (mgr.14150) 174 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:07 smithi165 bash[22591]: cluster 2024-09-10T18:44:06.436344+0000 mon.a (mon.0) 363 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:44:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:07 smithi165 bash[22591]: audit 2024-09-10T18:44:06.513740+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:44:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[18629]: audit 2024-09-10T18:44:07.442043+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[18629]: cluster 2024-09-10T18:44:07.442286+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[18629]: audit 2024-09-10T18:44:07.444593+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[18629]: audit 2024-09-10T18:44:08.444303+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[18629]: cluster 2024-09-10T18:44:08.444465+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[32381]: audit 2024-09-10T18:44:07.442043+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[32381]: cluster 2024-09-10T18:44:07.442286+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[32381]: audit 2024-09-10T18:44:07.444593+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[32381]: audit 2024-09-10T18:44:08.444303+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:44:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:08 smithi083 bash[32381]: cluster 2024-09-10T18:44:08.444465+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:44:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:08 smithi165 bash[22591]: audit 2024-09-10T18:44:07.442043+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:44:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:08 smithi165 bash[22591]: cluster 2024-09-10T18:44:07.442286+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:44:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:08 smithi165 bash[22591]: audit 2024-09-10T18:44:07.444593+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:44:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:08 smithi165 bash[22591]: audit 2024-09-10T18:44:08.444303+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:44:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:08 smithi165 bash[22591]: cluster 2024-09-10T18:44:08.444465+0000 mon.a (mon.0) 369 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-09-10T18:44:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:09 smithi083 bash[18629]: cluster 2024-09-10T18:44:08.412684+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v145: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:09 smithi083 bash[18629]: cluster 2024-09-10T18:44:08.447633+0000 mon.a (mon.0) 370 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:44:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:09 smithi083 bash[32381]: cluster 2024-09-10T18:44:08.412684+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v145: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:09 smithi083 bash[32381]: cluster 2024-09-10T18:44:08.447633+0000 mon.a (mon.0) 370 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:44:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:09 smithi165 bash[22591]: cluster 2024-09-10T18:44:08.412684+0000 mgr.y (mgr.14150) 175 : cluster [DBG] pgmap v145: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:09 smithi165 bash[22591]: cluster 2024-09-10T18:44:08.447633+0000 mon.a (mon.0) 370 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:44:10.618 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:10.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:10 smithi083 bash[18629]: cluster 2024-09-10T18:44:09.472821+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-09-10T18:44:10.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:10 smithi083 bash[18629]: audit 2024-09-10T18:44:10.336645+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:10.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:10 smithi083 bash[32381]: cluster 2024-09-10T18:44:09.472821+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-09-10T18:44:10.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:10 smithi083 bash[32381]: audit 2024-09-10T18:44:10.336645+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:10 smithi165 bash[22591]: cluster 2024-09-10T18:44:09.472821+0000 mon.a (mon.0) 371 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-09-10T18:44:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:10 smithi165 bash[22591]: audit 2024-09-10T18:44:10.336645+0000 mon.a (mon.0) 372 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:11.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[18629]: cluster 2024-09-10T18:44:10.413172+0000 mgr.y (mgr.14150) 176 : cluster [DBG] pgmap v148: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[18629]: cluster 2024-09-10T18:44:10.483785+0000 mon.a (mon.0) 373 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[18629]: cluster 2024-09-10T18:44:11.370460+0000 mon.a (mon.0) 374 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[18629]: cluster 2024-09-10T18:44:11.370523+0000 mon.a (mon.0) 375 : cluster [INF] Cluster is now healthy 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[32381]: cluster 2024-09-10T18:44:10.413172+0000 mgr.y (mgr.14150) 176 : cluster [DBG] pgmap v148: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[32381]: cluster 2024-09-10T18:44:10.483785+0000 mon.a (mon.0) 373 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[32381]: cluster 2024-09-10T18:44:11.370460+0000 mon.a (mon.0) 374 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:44:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:11 smithi083 bash[32381]: cluster 2024-09-10T18:44:11.370523+0000 mon.a (mon.0) 375 : cluster [INF] Cluster is now healthy 2024-09-10T18:44:11.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:11 smithi165 bash[22591]: cluster 2024-09-10T18:44:10.413172+0000 mgr.y (mgr.14150) 176 : cluster [DBG] pgmap v148: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:11 smithi165 bash[22591]: cluster 2024-09-10T18:44:10.483785+0000 mon.a (mon.0) 373 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-09-10T18:44:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:11 smithi165 bash[22591]: cluster 2024-09-10T18:44:11.370460+0000 mon.a (mon.0) 374 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:44:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:11 smithi165 bash[22591]: cluster 2024-09-10T18:44:11.370523+0000 mon.a (mon.0) 375 : cluster [INF] Cluster is now healthy 2024-09-10T18:44:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[18629]: audit 2024-09-10T18:44:12.075204+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[18629]: audit 2024-09-10T18:44:12.075460+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[18629]: audit 2024-09-10T18:44:12.075697+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[18629]: audit 2024-09-10T18:44:12.076049+0000 mon.a (mon.0) 379 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[32381]: audit 2024-09-10T18:44:12.075204+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[32381]: audit 2024-09-10T18:44:12.075460+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[32381]: audit 2024-09-10T18:44:12.075697+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:12 smithi083 bash[32381]: audit 2024-09-10T18:44:12.076049+0000 mon.a (mon.0) 379 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:12 smithi165 bash[22591]: audit 2024-09-10T18:44:12.075204+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:12 smithi165 bash[22591]: audit 2024-09-10T18:44:12.075460+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:12 smithi165 bash[22591]: audit 2024-09-10T18:44:12.075697+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:12 smithi165 bash[22591]: audit 2024-09-10T18:44:12.076049+0000 mon.a (mon.0) 379 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:15.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: cluster 2024-09-10T18:44:12.413733+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:12.962598+0000 mon.a (mon.0) 380 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:12.969632+0000 mon.b (mon.1) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:12.971958+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:12.972193+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:12.972381+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:13.849237+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:13.885780+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:13.889253+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:13.889473+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[18629]: audit 2024-09-10T18:44:13.889711+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: cluster 2024-09-10T18:44:12.413733+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:12.962598+0000 mon.a (mon.0) 380 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:12.969632+0000 mon.b (mon.1) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:12.971958+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:12.972193+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:12.972381+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:13.849237+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:13.885780+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:13.889253+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:13.889473+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:14 smithi083 bash[32381]: audit 2024-09-10T18:44:13.889711+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: cluster 2024-09-10T18:44:12.413733+0000 mgr.y (mgr.14150) 177 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:12.962598+0000 mon.a (mon.0) 380 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:12.969632+0000 mon.b (mon.1) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:12.971958+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:12.972193+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:12.972381+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:13.849237+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:13.885780+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:13.889253+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:44:15.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:13.889473+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:44:15.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:14 smithi165 bash[22591]: audit 2024-09-10T18:44:13.889711+0000 mon.a (mon.0) 387 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:44:15.191 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T18:44:15.386 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T18:44:15.407 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-09-10T18:44:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-09-10T18:44:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0173759 s, 603 MB/s 2024-09-10T18:44:15.407 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-09-10T18:44:15.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[32381]: audit 2024-09-10T18:44:13.889527+0000 mon.c (mon.2) 6 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:15.788 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[32381]: cluster 2024-09-10T18:44:14.414187+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:15.788 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[32381]: audit 2024-09-10T18:44:14.782026+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[18629]: audit 2024-09-10T18:44:13.889527+0000 mon.c (mon.2) 6 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[18629]: cluster 2024-09-10T18:44:14.414187+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:15 smithi083 bash[18629]: audit 2024-09-10T18:44:14.782026+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:15 smithi165 bash[22591]: audit 2024-09-10T18:44:13.889527+0000 mon.c (mon.2) 6 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:44:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:15 smithi165 bash[22591]: cluster 2024-09-10T18:44:14.414187+0000 mgr.y (mgr.14150) 178 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:15 smithi165 bash[22591]: audit 2024-09-10T18:44:14.782026+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:44:16.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: cluster 2024-09-10T18:44:16.414597+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.156205+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.163233+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.164725+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.165795+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.166766+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.173519+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.175736+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.176954+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[18629]: audit 2024-09-10T18:44:17.184230+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: cluster 2024-09-10T18:44:16.414597+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.156205+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.163233+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.164725+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.165795+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.166766+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.173519+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.175736+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.176954+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:18.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:17 smithi083 bash[32381]: audit 2024-09-10T18:44:17.184230+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: cluster 2024-09-10T18:44:16.414597+0000 mgr.y (mgr.14150) 179 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.156205+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.163233+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.164725+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.165795+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.166766+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:44:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.173519+0000 mon.a (mon.0) 393 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:18.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.175736+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:18.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.176954+0000 mon.a (mon.0) 395 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:18.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:17 smithi165 bash[22591]: audit 2024-09-10T18:44:17.184230+0000 mon.a (mon.0) 396 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:19.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:18 smithi165 bash[22591]: cephadm 2024-09-10T18:44:17.147935+0000 mgr.y (mgr.14150) 180 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:44:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:18 smithi165 bash[22591]: cephadm 2024-09-10T18:44:17.167507+0000 mgr.y (mgr.14150) 181 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 5415M 2024-09-10T18:44:19.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:18 smithi083 bash[18629]: cephadm 2024-09-10T18:44:17.147935+0000 mgr.y (mgr.14150) 180 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:44:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:18 smithi083 bash[18629]: cephadm 2024-09-10T18:44:17.167507+0000 mgr.y (mgr.14150) 181 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 5415M 2024-09-10T18:44:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:18 smithi083 bash[32381]: cephadm 2024-09-10T18:44:17.147935+0000 mgr.y (mgr.14150) 180 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:44:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:18 smithi083 bash[32381]: cephadm 2024-09-10T18:44:17.167507+0000 mgr.y (mgr.14150) 181 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 5415M 2024-09-10T18:44:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:19 smithi165 bash[22591]: cluster 2024-09-10T18:44:18.415119+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:44:20.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:19 smithi083 bash[32381]: cluster 2024-09-10T18:44:18.415119+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:44:20.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:19 smithi083 bash[18629]: cluster 2024-09-10T18:44:18.415119+0000 mgr.y (mgr.14150) 182 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-10T18:44:20.876 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:44:22.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:21 smithi083 bash[18629]: cluster 2024-09-10T18:44:20.415694+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:44:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:21 smithi165 bash[22591]: cluster 2024-09-10T18:44:20.415694+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:44:22.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:21 smithi083 bash[32381]: cluster 2024-09-10T18:44:20.415694+0000 mgr.y (mgr.14150) 183 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-10T18:44:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- 172.21.15.83:0/2702680580 >> v1:172.21.15.83:6790/0 conn(0x7fd70c100bf0 legacy=0x7fd70c102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- 172.21.15.83:0/2702680580 shutdown_connections 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- 172.21.15.83:0/2702680580 >> 172.21.15.83:0/2702680580 conn(0x7fd70c0f63f0 msgr2=0x7fd70c0f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- 172.21.15.83:0/2702680580 shutdown_connections 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- 172.21.15.83:0/2702680580 wait complete. 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 librados: starting msgr at 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 librados: starting objecter 2024-09-10T18:44:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 Processor -- start 2024-09-10T18:44:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- start start 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 librados: setting wanted keys 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 librados: calling monclient init 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd70c1199d0 con 0x7fd70c1046f0 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd70c11abd0 con 0x7fd70c0fdbf0 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd713b4d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd70c11bdd0 con 0x7fd70c100bf0 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7110e8640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd70c100bf0 0x7fd70c103440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59100/0 (socket says 172.21.15.83:59100) 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7110e8640 1 -- 172.21.15.83:0/2503962951 learned_addr learned my addr 172.21.15.83:0/2503962951 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter ms_handle_connect 0x7fd70c1046f0 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter ms_handle_connect 0x7fd70c100bf0 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:44:24.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96163086 0 0) 0x7fd70c1199d0 con 0x7fd70c1046f0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f80034e0 con 0x7fd70c1046f0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter ms_handle_connect 0x7fd70c0fdbf0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317294370 0 0) 0x7fd70c11bdd0 con 0x7fd70c100bf0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c1199d0 con 0x7fd70c100bf0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393778908 0 0) 0x7fd70c11abd0 con 0x7fd70c0fdbf0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70c11bdd0 con 0x7fd70c0fdbf0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.007+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3716235526 0 0) 0x7fd6f80034e0 con 0x7fd70c1046f0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd70c11abd0 con 0x7fd70c1046f0 2024-09-10T18:44:24.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd7040018c0 con 0x7fd70c1046f0 2024-09-10T18:44:24.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023940909 0 0) 0x7fd70c1199d0 con 0x7fd70c100bf0 2024-09-10T18:44:24.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6f80034e0 con 0x7fd70c100bf0 2024-09-10T18:44:24.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd6fc001430 con 0x7fd70c100bf0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3466510252 0 0) 0x7fd70c11abd0 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 >> v1:172.21.15.83:6790/0 conn(0x7fd70c100bf0 legacy=0x7fd70c103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 >> v1:172.21.15.165:6789/0 conn(0x7fd70c0fdbf0 legacy=0x7fd70c102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70c11cfd0 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c11bfc0 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd713b4d640 10 client.14304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70c11c490 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd713b4d640 1 librados: init done 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd704001f40 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd704003110 con 0x7fd70c1046f0 2024-09-10T18:44:24.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f0005040 con 0x7fd70c1046f0 2024-09-10T18:44:24.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.011+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fd704003270 con 0x7fd70c1046f0 2024-09-10T18:44:24.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 2507+0+0 (unknown 4015459947 0 0) 0x7fd704086470 con 0x7fd70c1046f0 2024-09-10T18:44:24.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 10 client.14304.objecter ms_dispatch 0x7fd70c000b90 osd_map(23..23 src has 1..23) v4 2024-09-10T18:44:24.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 3 client.14304.objecter handle_osd_map got epochs [23,23] > 0 2024-09-10T18:44:24.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 3 client.14304.objecter handle_osd_map decoding full epoch 23 2024-09-10T18:44:24.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 20 client.14304.objecter dump_active .. 0 homeless 2024-09-10T18:44:24.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.015+0000 7fd7027fc640 10 client.14304.objecter ms_handle_connect 0x7fd6f8071450 2024-09-10T18:44:24.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.019+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd704053b80 con 0x7fd70c1046f0 2024-09-10T18:44:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:23 smithi165 bash[22591]: cluster 2024-09-10T18:44:22.416253+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:44:24.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:24.199+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6f0000bc0 con 0x7fd6f8071450 2024-09-10T18:44:24.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:23 smithi083 bash[18629]: cluster 2024-09-10T18:44:22.416253+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:44:24.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:23 smithi083 bash[32381]: cluster 2024-09-10T18:44:22.416253+0000 mgr.y (mgr.14150) 184 : cluster [DBG] pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-10T18:44:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:24 smithi165 bash[22591]: audit 2024-09-10T18:44:24.210344+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:44:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:24 smithi165 bash[22591]: audit 2024-09-10T18:44:24.213152+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:44:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:24 smithi165 bash[22591]: audit 2024-09-10T18:44:24.214168+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:25.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[18629]: audit 2024-09-10T18:44:24.210344+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:44:25.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[18629]: audit 2024-09-10T18:44:24.213152+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:44:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[18629]: audit 2024-09-10T18:44:24.214168+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:25.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[32381]: audit 2024-09-10T18:44:24.210344+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:44:25.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[32381]: audit 2024-09-10T18:44:24.213152+0000 mon.a (mon.0) 398 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:44:25.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:24 smithi083 bash[32381]: audit 2024-09-10T18:44:24.214168+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:25 smithi165 bash[22591]: audit 2024-09-10T18:44:24.207792+0000 mgr.y (mgr.14150) 185 : audit [DBG] from='client.14304 v1:172.21.15.83:0/2503962951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:44:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:25 smithi165 bash[22591]: cluster 2024-09-10T18:44:24.416773+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:26.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:25 smithi083 bash[32381]: audit 2024-09-10T18:44:24.207792+0000 mgr.y (mgr.14150) 185 : audit [DBG] from='client.14304 v1:172.21.15.83:0/2503962951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:44:26.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:25 smithi083 bash[32381]: cluster 2024-09-10T18:44:24.416773+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:26.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:25 smithi083 bash[18629]: audit 2024-09-10T18:44:24.207792+0000 mgr.y (mgr.14150) 185 : audit [DBG] from='client.14304 v1:172.21.15.83:0/2503962951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:44:26.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:25 smithi083 bash[18629]: cluster 2024-09-10T18:44:24.416773+0000 mgr.y (mgr.14150) 186 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:27 smithi165 bash[22591]: cluster 2024-09-10T18:44:26.417290+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:27 smithi083 bash[18629]: cluster 2024-09-10T18:44:26.417290+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:28.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:27 smithi083 bash[32381]: cluster 2024-09-10T18:44:26.417290+0000 mgr.y (mgr.14150) 187 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:29.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:29.015+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:29 smithi165 bash[22591]: cluster 2024-09-10T18:44:28.417771+0000 mgr.y (mgr.14150) 188 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:30.200 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:29 smithi083 bash[18629]: cluster 2024-09-10T18:44:28.417771+0000 mgr.y (mgr.14150) 188 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:30.201 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:29 smithi083 bash[32381]: cluster 2024-09-10T18:44:28.417771+0000 mgr.y (mgr.14150) 188 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-10T18:44:31.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[18629]: cluster 2024-09-10T18:44:30.284110+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[18629]: cluster 2024-09-10T18:44:30.284315+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[18629]: cluster 2024-09-10T18:44:30.418348+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[18629]: cluster 2024-09-10T18:44:30.836328+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e24: 3 total, 3 up, 3 in 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[32381]: cluster 2024-09-10T18:44:30.284110+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[32381]: cluster 2024-09-10T18:44:30.284315+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[32381]: cluster 2024-09-10T18:44:30.418348+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:31 smithi083 bash[32381]: cluster 2024-09-10T18:44:30.836328+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e24: 3 total, 3 up, 3 in 2024-09-10T18:44:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:31 smithi165 bash[22591]: cluster 2024-09-10T18:44:30.284110+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:31 smithi165 bash[22591]: cluster 2024-09-10T18:44:30.284315+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:31 smithi165 bash[22591]: cluster 2024-09-10T18:44:30.418348+0000 mgr.y (mgr.14150) 189 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:31 smithi165 bash[22591]: cluster 2024-09-10T18:44:30.836328+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e24: 3 total, 3 up, 3 in 2024-09-10T18:44:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: cluster 2024-09-10T18:44:32.418837+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: audit 2024-09-10T18:44:32.765926+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.83:0/3769215183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: audit 2024-09-10T18:44:32.766627+0000 mon.a (mon.0) 401 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: audit 2024-09-10T18:44:32.776801+0000 mon.a (mon.0) 402 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]': finished 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: cluster 2024-09-10T18:44:32.776981+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: audit 2024-09-10T18:44:32.777207+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[18629]: audit 2024-09-10T18:44:33.433635+0000 mon.a (mon.0) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/2640453948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: cluster 2024-09-10T18:44:32.418837+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: audit 2024-09-10T18:44:32.765926+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.83:0/3769215183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: audit 2024-09-10T18:44:32.766627+0000 mon.a (mon.0) 401 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: audit 2024-09-10T18:44:32.776801+0000 mon.a (mon.0) 402 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]': finished 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: cluster 2024-09-10T18:44:32.776981+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:44:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: audit 2024-09-10T18:44:32.777207+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:33 smithi083 bash[32381]: audit 2024-09-10T18:44:33.433635+0000 mon.a (mon.0) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/2640453948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: cluster 2024-09-10T18:44:32.418837+0000 mgr.y (mgr.14150) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: audit 2024-09-10T18:44:32.765926+0000 mon.c (mon.2) 8 : audit [INF] from='client.? v1:172.21.15.83:0/3769215183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: audit 2024-09-10T18:44:32.766627+0000 mon.a (mon.0) 401 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]: dispatch 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: audit 2024-09-10T18:44:32.776801+0000 mon.a (mon.0) 402 : audit [INF] from='client.24218 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c"}]': finished 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: cluster 2024-09-10T18:44:32.776981+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: audit 2024-09-10T18:44:32.777207+0000 mon.a (mon.0) 404 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:33 smithi165 bash[22591]: audit 2024-09-10T18:44:33.433635+0000 mon.a (mon.0) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/2640453948' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:44:34.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:34.019+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:35 smithi083 bash[18629]: cluster 2024-09-10T18:44:34.419394+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:35 smithi083 bash[32381]: cluster 2024-09-10T18:44:34.419394+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:35 smithi165 bash[22591]: cluster 2024-09-10T18:44:34.419394+0000 mgr.y (mgr.14150) 191 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:37 smithi083 bash[18629]: cluster 2024-09-10T18:44:36.419922+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:37 smithi083 bash[32381]: cluster 2024-09-10T18:44:36.419922+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:37 smithi165 bash[22591]: cluster 2024-09-10T18:44:36.419922+0000 mgr.y (mgr.14150) 192 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:39.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:39.023+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:39 smithi083 bash[18629]: cluster 2024-09-10T18:44:38.420455+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:39 smithi083 bash[32381]: cluster 2024-09-10T18:44:38.420455+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:39 smithi165 bash[22591]: cluster 2024-09-10T18:44:38.420455+0000 mgr.y (mgr.14150) 193 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:41 smithi083 bash[18629]: cluster 2024-09-10T18:44:40.421012+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:41 smithi083 bash[32381]: cluster 2024-09-10T18:44:40.421012+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:41 smithi165 bash[22591]: cluster 2024-09-10T18:44:40.421012+0000 mgr.y (mgr.14150) 194 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:43 smithi083 bash[18629]: cluster 2024-09-10T18:44:42.421562+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:43.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:43 smithi083 bash[32381]: cluster 2024-09-10T18:44:42.421562+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:43 smithi165 bash[22591]: cluster 2024-09-10T18:44:42.421562+0000 mgr.y (mgr.14150) 195 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:44.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:44.027+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:45.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:45 smithi083 bash[32381]: cluster 2024-09-10T18:44:44.422096+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:45.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:45 smithi083 bash[18629]: cluster 2024-09-10T18:44:44.422096+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:45 smithi165 bash[22591]: cluster 2024-09-10T18:44:44.422096+0000 mgr.y (mgr.14150) 196 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:46.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[18629]: audit 2024-09-10T18:44:45.529138+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:44:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[18629]: audit 2024-09-10T18:44:45.530308+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[18629]: cephadm 2024-09-10T18:44:45.531207+0000 mgr.y (mgr.14150) 197 : cephadm [INF] Deploying daemon osd.3 on smithi083 2024-09-10T18:44:46.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[32381]: audit 2024-09-10T18:44:45.529138+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:44:46.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[32381]: audit 2024-09-10T18:44:45.530308+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:46.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:46 smithi083 bash[32381]: cephadm 2024-09-10T18:44:45.531207+0000 mgr.y (mgr.14150) 197 : cephadm [INF] Deploying daemon osd.3 on smithi083 2024-09-10T18:44:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:46 smithi165 bash[22591]: audit 2024-09-10T18:44:45.529138+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:44:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:46 smithi165 bash[22591]: audit 2024-09-10T18:44:45.530308+0000 mon.a (mon.0) 407 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:46 smithi165 bash[22591]: cephadm 2024-09-10T18:44:45.531207+0000 mgr.y (mgr.14150) 197 : cephadm [INF] Deploying daemon osd.3 on smithi083 2024-09-10T18:44:47.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:47 smithi083 bash[18629]: cluster 2024-09-10T18:44:46.422576+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:47.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:47 smithi083 bash[32381]: cluster 2024-09-10T18:44:46.422576+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:47 smithi165 bash[22591]: cluster 2024-09-10T18:44:46.422576+0000 mgr.y (mgr.14150) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:49.031+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:49 smithi083 bash[18629]: cluster 2024-09-10T18:44:48.423153+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:49.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:49 smithi083 bash[32381]: cluster 2024-09-10T18:44:48.423153+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:49 smithi165 bash[22591]: cluster 2024-09-10T18:44:48.423153+0000 mgr.y (mgr.14150) 199 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:50.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[32381]: audit 2024-09-10T18:44:50.187344+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:50.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[32381]: audit 2024-09-10T18:44:50.191656+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[32381]: audit 2024-09-10T18:44:50.197988+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[18629]: audit 2024-09-10T18:44:50.187344+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[18629]: audit 2024-09-10T18:44:50.191656+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:50 smithi083 bash[18629]: audit 2024-09-10T18:44:50.197988+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:50 smithi165 bash[22591]: audit 2024-09-10T18:44:50.187344+0000 mon.a (mon.0) 408 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:44:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:50 smithi165 bash[22591]: audit 2024-09-10T18:44:50.191656+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:50 smithi165 bash[22591]: audit 2024-09-10T18:44:50.197988+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:51.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:51 smithi083 bash[18629]: cluster 2024-09-10T18:44:50.423472+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:51.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:51 smithi083 bash[32381]: cluster 2024-09-10T18:44:50.423472+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:51 smithi165 bash[22591]: cluster 2024-09-10T18:44:50.423472+0000 mgr.y (mgr.14150) 200 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[18629]: cluster 2024-09-10T18:44:52.008523+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[18629]: cluster 2024-09-10T18:44:52.008642+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[18629]: cluster 2024-09-10T18:44:52.424078+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[18629]: cluster 2024-09-10T18:44:52.535932+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[18629]: audit 2024-09-10T18:44:52.536206+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[32381]: cluster 2024-09-10T18:44:52.008523+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[32381]: cluster 2024-09-10T18:44:52.008642+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[32381]: cluster 2024-09-10T18:44:52.424078+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[32381]: cluster 2024-09-10T18:44:52.535932+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:44:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:53 smithi083 bash[32381]: audit 2024-09-10T18:44:52.536206+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:53 smithi165 bash[22591]: cluster 2024-09-10T18:44:52.008523+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:44:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:53 smithi165 bash[22591]: cluster 2024-09-10T18:44:52.008642+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:44:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:53 smithi165 bash[22591]: cluster 2024-09-10T18:44:52.424078+0000 mgr.y (mgr.14150) 201 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:53 smithi165 bash[22591]: cluster 2024-09-10T18:44:52.535932+0000 mon.a (mon.0) 411 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-09-10T18:44:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:53 smithi165 bash[22591]: audit 2024-09-10T18:44:52.536206+0000 mon.a (mon.0) 412 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:54.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:54.034+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:55 smithi083 bash[18629]: cluster 2024-09-10T18:44:54.424633+0000 mgr.y (mgr.14150) 202 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:55 smithi083 bash[32381]: cluster 2024-09-10T18:44:54.424633+0000 mgr.y (mgr.14150) 202 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:55 smithi165 bash[22591]: cluster 2024-09-10T18:44:54.424633+0000 mgr.y (mgr.14150) 202 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: cluster 2024-09-10T18:44:56.425116+0000 mgr.y (mgr.14150) 203 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:56.554120+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:56.562694+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:56.564176+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:56.565368+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:56.572863+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:57.262634+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:57 smithi165 bash[22591]: audit 2024-09-10T18:44:57.263082+0000 mon.a (mon.0) 418 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: cluster 2024-09-10T18:44:56.425116+0000 mgr.y (mgr.14150) 203 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:56.554120+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:56.562694+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:56.564176+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:56.565368+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:56.572863+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:57.262634+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[18629]: audit 2024-09-10T18:44:57.263082+0000 mon.a (mon.0) 418 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: cluster 2024-09-10T18:44:56.425116+0000 mgr.y (mgr.14150) 203 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:56.554120+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:56.562694+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:56.564176+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:56.565368+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:56.572863+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:57.262634+0000 mon.c (mon.2) 9 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:57 smithi083 bash[32381]: audit 2024-09-10T18:44:57.263082+0000 mon.a (mon.0) 418 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:44:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:58 smithi165 bash[22591]: audit 2024-09-10T18:44:57.579711+0000 mon.a (mon.0) 419 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:44:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:58 smithi165 bash[22591]: cluster 2024-09-10T18:44:57.579791+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-10T18:44:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:58 smithi165 bash[22591]: audit 2024-09-10T18:44:57.580015+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:58 smithi165 bash[22591]: audit 2024-09-10T18:44:57.580725+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:58 smithi165 bash[22591]: audit 2024-09-10T18:44:57.581262+0000 mon.a (mon.0) 422 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:59.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[18629]: audit 2024-09-10T18:44:57.579711+0000 mon.a (mon.0) 419 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:44:59.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[18629]: cluster 2024-09-10T18:44:57.579791+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[18629]: audit 2024-09-10T18:44:57.580015+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[18629]: audit 2024-09-10T18:44:57.580725+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[18629]: audit 2024-09-10T18:44:57.581262+0000 mon.a (mon.0) 422 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[32381]: audit 2024-09-10T18:44:57.579711+0000 mon.a (mon.0) 419 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[32381]: cluster 2024-09-10T18:44:57.579791+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[32381]: audit 2024-09-10T18:44:57.580015+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[32381]: audit 2024-09-10T18:44:57.580725+0000 mon.c (mon.2) 10 : audit [INF] from='osd.3 v1:172.21.15.83:6813/1279688021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:58 smithi083 bash[32381]: audit 2024-09-10T18:44:57.581262+0000 mon.a (mon.0) 422 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T18:44:59.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:44:59.038+0000 7fd702ffd640 10 client.14304.objecter tick 2024-09-10T18:44:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:59 smithi165 bash[22591]: cluster 2024-09-10T18:44:58.425669+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:59 smithi165 bash[22591]: audit 2024-09-10T18:44:58.584150+0000 mon.a (mon.0) 423 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:59 smithi165 bash[22591]: cluster 2024-09-10T18:44:58.584342+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e28: 4 total, 3 up, 4 in 2024-09-10T18:44:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:59 smithi165 bash[22591]: audit 2024-09-10T18:44:58.585132+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:44:59 smithi165 bash[22591]: audit 2024-09-10T18:44:58.591731+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.870 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[18629]: cluster 2024-09-10T18:44:58.425669+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:59.870 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[18629]: audit 2024-09-10T18:44:58.584150+0000 mon.a (mon.0) 423 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:59.870 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[18629]: cluster 2024-09-10T18:44:58.584342+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e28: 4 total, 3 up, 4 in 2024-09-10T18:44:59.870 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[18629]: audit 2024-09-10T18:44:58.585132+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.870 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[18629]: audit 2024-09-10T18:44:58.591731+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[32381]: cluster 2024-09-10T18:44:58.425669+0000 mgr.y (mgr.14150) 204 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:44:59.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[32381]: audit 2024-09-10T18:44:58.584150+0000 mon.a (mon.0) 423 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-09-10T18:44:59.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[32381]: cluster 2024-09-10T18:44:58.584342+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e28: 4 total, 3 up, 4 in 2024-09-10T18:44:59.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[32381]: audit 2024-09-10T18:44:58.585132+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:44:59.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:44:59 smithi083 bash[32381]: audit 2024-09-10T18:44:58.591731+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:45:00.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.470+0000 7fd7027fc640 1 -- 172.21.15.83:0/2503962951 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3838146360) 0x7fd6f0000bc0 con 0x7fd6f8071450 2024-09-10T18:45:00.478 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 3 on host 'smithi083' 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.474+0000 7fd713b4d640 10 librados: watch_flush enter 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.474+0000 7fd713b4d640 10 librados: watch_flush exit 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.474+0000 7fd713b4d640 20 client.14304.objecter shutdown clearing up homeless session... 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.474+0000 7fd713b4d640 10 client.14304.objecter successfully canceled tick 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 >> v1:172.21.15.83:6800/2630800463 conn(0x7fd6f8071450 legacy=0x7fd6f8073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 >> v1:172.21.15.83:6789/0 conn(0x7fd70c1046f0 legacy=0x7fd70c118120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 shutdown_connections 2024-09-10T18:45:00.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 >> 172.21.15.83:0/2503962951 conn(0x7fd70c0f63f0 msgr2=0x7fd70c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:45:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 shutdown_connections 2024-09-10T18:45:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 -- 172.21.15.83:0/2503962951 wait complete. 2024-09-10T18:45:00.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:45:00.478+0000 7fd713b4d640 1 librados: shutdown 2024-09-10T18:45:00.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: cluster 2024-09-10T18:44:58.256984+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: cluster 2024-09-10T18:44:58.257073+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: cluster 2024-09-10T18:44:59.598545+0000 mon.a (mon.0) 427 : cluster [INF] osd.3 v1:172.21.15.83:6813/1279688021 boot 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: cluster 2024-09-10T18:44:59.598592+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: audit 2024-09-10T18:44:59.599027+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: audit 2024-09-10T18:45:00.459336+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: audit 2024-09-10T18:45:00.465822+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[18629]: audit 2024-09-10T18:45:00.474505+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: cluster 2024-09-10T18:44:58.256984+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: cluster 2024-09-10T18:44:58.257073+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: cluster 2024-09-10T18:44:59.598545+0000 mon.a (mon.0) 427 : cluster [INF] osd.3 v1:172.21.15.83:6813/1279688021 boot 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: cluster 2024-09-10T18:44:59.598592+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: audit 2024-09-10T18:44:59.599027+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: audit 2024-09-10T18:45:00.459336+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: audit 2024-09-10T18:45:00.465822+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:00 smithi083 bash[32381]: audit 2024-09-10T18:45:00.474505+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: cluster 2024-09-10T18:44:58.256984+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: cluster 2024-09-10T18:44:58.257073+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: cluster 2024-09-10T18:44:59.598545+0000 mon.a (mon.0) 427 : cluster [INF] osd.3 v1:172.21.15.83:6813/1279688021 boot 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: cluster 2024-09-10T18:44:59.598592+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: audit 2024-09-10T18:44:59.599027+0000 mon.a (mon.0) 429 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: audit 2024-09-10T18:45:00.459336+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: audit 2024-09-10T18:45:00.465822+0000 mon.a (mon.0) 431 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:00 smithi165 bash[22591]: audit 2024-09-10T18:45:00.474505+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:01.176 DEBUG:teuthology.orchestra.run.smithi083:osd.3> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.3.service 2024-09-10T18:45:01.178 INFO:tasks.cephadm:Deploying osd.4 on smithi165 with /dev/vg_nvme/lv_4... 2024-09-10T18:45:01.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:45:01.361 INFO:journalctl@ceph.osd.3.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:45:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:01 smithi083 bash[18629]: cluster 2024-09-10T18:45:00.426163+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:45:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:01 smithi083 bash[18629]: cluster 2024-09-10T18:45:00.618518+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-09-10T18:45:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:01 smithi083 bash[32381]: cluster 2024-09-10T18:45:00.426163+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:45:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:01 smithi083 bash[32381]: cluster 2024-09-10T18:45:00.618518+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-09-10T18:45:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:01 smithi165 bash[22591]: cluster 2024-09-10T18:45:00.426163+0000 mgr.y (mgr.14150) 205 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:45:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:01 smithi165 bash[22591]: cluster 2024-09-10T18:45:00.618518+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-09-10T18:45:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:03 smithi083 bash[18629]: cluster 2024-09-10T18:45:02.426719+0000 mgr.y (mgr.14150) 206 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:03 smithi083 bash[32381]: cluster 2024-09-10T18:45:02.426719+0000 mgr.y (mgr.14150) 206 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:04.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:03 smithi165 bash[22591]: cluster 2024-09-10T18:45:02.426719+0000 mgr.y (mgr.14150) 206 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:05 smithi083 bash[18629]: cluster 2024-09-10T18:45:04.427224+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:05 smithi083 bash[32381]: cluster 2024-09-10T18:45:04.427224+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:05.899 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:45:06.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:05 smithi165 bash[22591]: cluster 2024-09-10T18:45:04.427224+0000 mgr.y (mgr.14150) 207 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:07.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:07 smithi083 bash[18629]: cluster 2024-09-10T18:45:06.427757+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:07.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:07 smithi083 bash[32381]: cluster 2024-09-10T18:45:06.427757+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:07.919 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:07 smithi165 bash[22591]: cluster 2024-09-10T18:45:06.427757+0000 mgr.y (mgr.14150) 208 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:09.233 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:45:09.393 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-09-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-09-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146436 s, 716 MB/s 2024-09-10T18:45:09.411 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-09-10T18:45:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:09 smithi165 bash[22591]: cluster 2024-09-10T18:45:08.428565+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:09.994 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:09 smithi083 bash[18629]: cluster 2024-09-10T18:45:08.428565+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:09.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:09 smithi083 bash[32381]: cluster 2024-09-10T18:45:08.428565+0000 mgr.y (mgr.14150) 209 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:09.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-09-10T18:45:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: cluster 2024-09-10T18:45:10.429335+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: cephadm 2024-09-10T18:45:10.673755+0000 mgr.y (mgr.14150) 211 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:45:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.680586+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.689981+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.692230+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.693697+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.695042+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.696473+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: cephadm 2024-09-10T18:45:10.697561+0000 mgr.y (mgr.14150) 212 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 4061M 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.706292+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.707941+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.709013+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[18629]: audit 2024-09-10T18:45:10.716023+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: cluster 2024-09-10T18:45:10.429335+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: cephadm 2024-09-10T18:45:10.673755+0000 mgr.y (mgr.14150) 211 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.680586+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.689981+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.692230+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.693697+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.695042+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.696473+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: cephadm 2024-09-10T18:45:10.697561+0000 mgr.y (mgr.14150) 212 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 4061M 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.706292+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.707941+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.709013+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:12.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:11 smithi083 bash[32381]: audit 2024-09-10T18:45:10.716023+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: cluster 2024-09-10T18:45:10.429335+0000 mgr.y (mgr.14150) 210 : cluster [DBG] pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: cephadm 2024-09-10T18:45:10.673755+0000 mgr.y (mgr.14150) 211 : cephadm [INF] Detected new or changed devices on smithi083 2024-09-10T18:45:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.680586+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.689981+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.692230+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.693697+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.695042+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.696473+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: cephadm 2024-09-10T18:45:10.697561+0000 mgr.y (mgr.14150) 212 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 4061M 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.706292+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.707941+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.709013+0000 mon.a (mon.0) 442 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:11 smithi165 bash[22591]: audit 2024-09-10T18:45:10.716023+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[32381]: cluster 2024-09-10T18:45:12.429937+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[32381]: cluster 2024-09-10T18:45:12.758370+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:45:14.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[32381]: cluster 2024-09-10T18:45:12.767797+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:45:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[18629]: cluster 2024-09-10T18:45:12.429937+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[18629]: cluster 2024-09-10T18:45:12.758370+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:45:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:13 smithi083 bash[18629]: cluster 2024-09-10T18:45:12.767797+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:45:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:13 smithi165 bash[22591]: cluster 2024-09-10T18:45:12.429937+0000 mgr.y (mgr.14150) 213 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:13 smithi165 bash[22591]: cluster 2024-09-10T18:45:12.758370+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:45:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:13 smithi165 bash[22591]: cluster 2024-09-10T18:45:12.767797+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:45:14.716 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:45:15.942 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:15 smithi165 bash[22591]: cluster 2024-09-10T18:45:14.430525+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:15 smithi083 bash[18629]: cluster 2024-09-10T18:45:14.430525+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:16.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:15 smithi083 bash[32381]: cluster 2024-09-10T18:45:14.430525+0000 mgr.y (mgr.14150) 214 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:17.650 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- 172.21.15.165:0/1327857744 >> v1:172.21.15.165:6789/0 conn(0x7fc7dc1069a0 legacy=0x7fc7dc108dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- 172.21.15.165:0/1327857744 shutdown_connections 2024-09-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- 172.21.15.165:0/1327857744 >> 172.21.15.165:0/1327857744 conn(0x7fc7dc0f63f0 msgr2=0x7fc7dc0f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- 172.21.15.165:0/1327857744 shutdown_connections 2024-09-10T18:45:17.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- 172.21.15.165:0/1327857744 wait complete. 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 librados: starting msgr at 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 librados: starting objecter 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 Processor -- start 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- start start 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 librados: setting wanted keys 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 librados: calling monclient init 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7dc10f380 con 0x7fc7dc0fcbe0 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7dc11ab80 con 0x7fc7dc102ea0 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e3865640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7dc11bd20 con 0x7fc7dc1069a0 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e0e00640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc7dc102ea0 0x7fc7dc114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:40534/0 (socket says 172.21.15.165:40534) 2024-09-10T18:45:17.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7e0e00640 1 -- 172.21.15.165:0/1516688970 learned_addr learned my addr 172.21.15.165:0/1516688970 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:45:17.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter ms_handle_connect 0x7fc7dc102ea0 2024-09-10T18:45:17.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter ms_handle_connect 0x7fc7dc0fcbe0 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter ms_handle_connect 0x7fc7dc1069a0 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528806597 0 0) 0x7fc7dc11ab80 con 0x7fc7dc102ea0 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c40054b0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590009762 0 0) 0x7fc7dc10f380 con 0x7fc7dc0fcbe0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc11ab80 con 0x7fc7dc0fcbe0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261043434 0 0) 0x7fc7dc11bd20 con 0x7fc7dc1069a0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc10f380 con 0x7fc7dc1069a0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381205778 0 0) 0x7fc7c40054b0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc11bd20 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.644+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7d0001990 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 74930061 0 0) 0x7fc7dc11bd20 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 >> v1:172.21.15.83:6790/0 conn(0x7fc7dc1069a0 legacy=0x7fc7dc118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 >> v1:172.21.15.83:6789/0 conn(0x7fc7dc0fcbe0 legacy=0x7fc7dc10e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc11cf00 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc119bd0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7e3865640 10 client.24238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc11a0a0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7e3865640 1 librados: init done 2024-09-10T18:45:17.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc7d0001d20 con 0x7fc7dc102ea0 2024-09-10T18:45:17.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7d0005b60 con 0x7fc7dc102ea0 2024-09-10T18:45:17.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8005040 con 0x7fc7dc102ea0 2024-09-10T18:45:17.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.648+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fc7d00021b0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(30..30 src has 1..30) v4 ==== 2799+0+0 (unknown 737695302 0 0) 0x7fc7d0088fd0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 10 client.24238.objecter ms_dispatch 0x7fc7dc000b90 osd_map(30..30 src has 1..30) v4 2024-09-10T18:45:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 3 client.24238.objecter handle_osd_map got epochs [30,30] > 0 2024-09-10T18:45:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 3 client.24238.objecter handle_osd_map decoding full epoch 30 2024-09-10T18:45:17.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 20 client.24238.objecter dump_active .. 0 homeless 2024-09-10T18:45:17.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.652+0000 7fc7d9ffb640 10 client.24238.objecter ms_handle_connect 0x7fc7c4071290 2024-09-10T18:45:17.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.656+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc7d00565c0 con 0x7fc7dc102ea0 2024-09-10T18:45:17.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:17.836+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7c8000bc0 con 0x7fc7c4071290 2024-09-10T18:45:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:17 smithi165 bash[22591]: cluster 2024-09-10T18:45:16.431084+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:17 smithi083 bash[18629]: cluster 2024-09-10T18:45:16.431084+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:17 smithi083 bash[32381]: cluster 2024-09-10T18:45:16.431084+0000 mgr.y (mgr.14150) 215 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[18629]: audit 2024-09-10T18:45:17.845421+0000 mgr.y (mgr.14150) 216 : audit [DBG] from='client.24238 v1:172.21.15.165:0/1516688970' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:45:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[18629]: audit 2024-09-10T18:45:17.852030+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:45:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[18629]: audit 2024-09-10T18:45:17.858166+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:45:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[18629]: audit 2024-09-10T18:45:17.859704+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[32381]: audit 2024-09-10T18:45:17.845421+0000 mgr.y (mgr.14150) 216 : audit [DBG] from='client.24238 v1:172.21.15.165:0/1516688970' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:45:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[32381]: audit 2024-09-10T18:45:17.852030+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:45:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[32381]: audit 2024-09-10T18:45:17.858166+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:45:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:18 smithi083 bash[32381]: audit 2024-09-10T18:45:17.859704+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:18 smithi165 bash[22591]: audit 2024-09-10T18:45:17.845421+0000 mgr.y (mgr.14150) 216 : audit [DBG] from='client.24238 v1:172.21.15.165:0/1516688970' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:45:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:18 smithi165 bash[22591]: audit 2024-09-10T18:45:17.852030+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:45:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:18 smithi165 bash[22591]: audit 2024-09-10T18:45:17.858166+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:45:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:18 smithi165 bash[22591]: audit 2024-09-10T18:45:17.859704+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:20.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[32381]: cluster 2024-09-10T18:45:18.431762+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:20.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[32381]: cluster 2024-09-10T18:45:19.322761+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[32381]: cluster 2024-09-10T18:45:19.322892+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[18629]: cluster 2024-09-10T18:45:18.431762+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[18629]: cluster 2024-09-10T18:45:19.322761+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:19 smithi083 bash[18629]: cluster 2024-09-10T18:45:19.322892+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:19 smithi165 bash[22591]: cluster 2024-09-10T18:45:18.431762+0000 mgr.y (mgr.14150) 217 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:19 smithi165 bash[22591]: cluster 2024-09-10T18:45:19.322761+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:19 smithi165 bash[22591]: cluster 2024-09-10T18:45:19.322892+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:20 smithi083 bash[18629]: cluster 2024-09-10T18:45:19.712612+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-09-10T18:45:21.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:20 smithi083 bash[32381]: cluster 2024-09-10T18:45:19.712612+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-09-10T18:45:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:20 smithi165 bash[22591]: cluster 2024-09-10T18:45:19.712612+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-09-10T18:45:22.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:21 smithi083 bash[18629]: cluster 2024-09-10T18:45:20.432414+0000 mgr.y (mgr.14150) 218 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:22.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:21 smithi083 bash[32381]: cluster 2024-09-10T18:45:20.432414+0000 mgr.y (mgr.14150) 218 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:21 smithi165 bash[22591]: cluster 2024-09-10T18:45:20.432414+0000 mgr.y (mgr.14150) 218 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:22.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:22.652+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:23 smithi083 bash[18629]: cluster 2024-09-10T18:45:22.433061+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:24.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:23 smithi083 bash[32381]: cluster 2024-09-10T18:45:22.433061+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:24.097 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:23 smithi165 bash[22591]: cluster 2024-09-10T18:45:22.433061+0000 mgr.y (mgr.14150) 219 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:25.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:25 smithi165 bash[22591]: cluster 2024-09-10T18:45:24.433694+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:25 smithi083 bash[18629]: cluster 2024-09-10T18:45:24.433694+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:26.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:25 smithi083 bash[32381]: cluster 2024-09-10T18:45:24.433694+0000 mgr.y (mgr.14150) 220 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[18629]: audit 2024-09-10T18:45:26.255979+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.165:0/1366807745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[18629]: audit 2024-09-10T18:45:26.260236+0000 mon.a (mon.0) 448 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[18629]: audit 2024-09-10T18:45:26.269050+0000 mon.a (mon.0) 449 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]': finished 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[18629]: cluster 2024-09-10T18:45:26.269166+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[18629]: audit 2024-09-10T18:45:26.271906+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[32381]: audit 2024-09-10T18:45:26.255979+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.165:0/1366807745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[32381]: audit 2024-09-10T18:45:26.260236+0000 mon.a (mon.0) 448 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[32381]: audit 2024-09-10T18:45:26.269050+0000 mon.a (mon.0) 449 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]': finished 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[32381]: cluster 2024-09-10T18:45:26.269166+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:45:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:26 smithi083 bash[32381]: audit 2024-09-10T18:45:26.271906+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:26 smithi165 bash[22591]: audit 2024-09-10T18:45:26.255979+0000 mon.b (mon.1) 9 : audit [INF] from='client.? v1:172.21.15.165:0/1366807745' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:26 smithi165 bash[22591]: audit 2024-09-10T18:45:26.260236+0000 mon.a (mon.0) 448 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]: dispatch 2024-09-10T18:45:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:26 smithi165 bash[22591]: audit 2024-09-10T18:45:26.269050+0000 mon.a (mon.0) 449 : audit [INF] from='client.24244 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b"}]': finished 2024-09-10T18:45:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:26 smithi165 bash[22591]: cluster 2024-09-10T18:45:26.269166+0000 mon.a (mon.0) 450 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-09-10T18:45:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:26 smithi165 bash[22591]: audit 2024-09-10T18:45:26.271906+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:27.661 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:27.656+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:27 smithi083 bash[18629]: cluster 2024-09-10T18:45:26.434347+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:27 smithi083 bash[18629]: audit 2024-09-10T18:45:26.909460+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.165:0/3883755905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:45:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:27 smithi083 bash[32381]: cluster 2024-09-10T18:45:26.434347+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:27 smithi083 bash[32381]: audit 2024-09-10T18:45:26.909460+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.165:0/3883755905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:45:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:27 smithi165 bash[22591]: cluster 2024-09-10T18:45:26.434347+0000 mgr.y (mgr.14150) 221 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:27 smithi165 bash[22591]: audit 2024-09-10T18:45:26.909460+0000 mon.b (mon.1) 10 : audit [DBG] from='client.? v1:172.21.15.165:0/3883755905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:45:29.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:29 smithi165 bash[22591]: cluster 2024-09-10T18:45:28.435015+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:30.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:29 smithi083 bash[18629]: cluster 2024-09-10T18:45:28.435015+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:29 smithi083 bash[32381]: cluster 2024-09-10T18:45:28.435015+0000 mgr.y (mgr.14150) 222 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:31.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:31 smithi083 bash[32381]: cluster 2024-09-10T18:45:30.435621+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:31 smithi083 bash[18629]: cluster 2024-09-10T18:45:30.435621+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:31 smithi165 bash[22591]: cluster 2024-09-10T18:45:30.435621+0000 mgr.y (mgr.14150) 223 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:32.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:32.660+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:33 smithi083 bash[18629]: cluster 2024-09-10T18:45:32.436269+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:33.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:33 smithi083 bash[32381]: cluster 2024-09-10T18:45:32.436269+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:33 smithi165 bash[22591]: cluster 2024-09-10T18:45:32.436269+0000 mgr.y (mgr.14150) 224 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:35.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:35 smithi083 bash[32381]: cluster 2024-09-10T18:45:34.436901+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:35 smithi083 bash[18629]: cluster 2024-09-10T18:45:34.436901+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:35 smithi165 bash[22591]: cluster 2024-09-10T18:45:34.436901+0000 mgr.y (mgr.14150) 225 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:37.669 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:37.664+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:37.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:37 smithi083 bash[32381]: cluster 2024-09-10T18:45:36.437557+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:37 smithi083 bash[18629]: cluster 2024-09-10T18:45:36.437557+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:37.800 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:37 smithi165 bash[22591]: cluster 2024-09-10T18:45:36.437557+0000 mgr.y (mgr.14150) 226 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:39.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[18629]: cluster 2024-09-10T18:45:38.438228+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[18629]: audit 2024-09-10T18:45:38.736516+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[18629]: audit 2024-09-10T18:45:38.738122+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[18629]: cephadm 2024-09-10T18:45:38.739605+0000 mgr.y (mgr.14150) 228 : cephadm [INF] Deploying daemon osd.4 on smithi165 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[32381]: cluster 2024-09-10T18:45:38.438228+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[32381]: audit 2024-09-10T18:45:38.736516+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[32381]: audit 2024-09-10T18:45:38.738122+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:39 smithi083 bash[32381]: cephadm 2024-09-10T18:45:38.739605+0000 mgr.y (mgr.14150) 228 : cephadm [INF] Deploying daemon osd.4 on smithi165 2024-09-10T18:45:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:39 smithi165 bash[22591]: cluster 2024-09-10T18:45:38.438228+0000 mgr.y (mgr.14150) 227 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:39 smithi165 bash[22591]: audit 2024-09-10T18:45:38.736516+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T18:45:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:39 smithi165 bash[22591]: audit 2024-09-10T18:45:38.738122+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:39 smithi165 bash[22591]: cephadm 2024-09-10T18:45:38.739605+0000 mgr.y (mgr.14150) 228 : cephadm [INF] Deploying daemon osd.4 on smithi165 2024-09-10T18:45:41.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:41 smithi083 bash[18629]: cluster 2024-09-10T18:45:40.438779+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:41 smithi083 bash[32381]: cluster 2024-09-10T18:45:40.438779+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:41 smithi165 bash[22591]: cluster 2024-09-10T18:45:40.438779+0000 mgr.y (mgr.14150) 229 : cluster [DBG] pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:42.673 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:42.668+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:43.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:43 smithi165 bash[22591]: cluster 2024-09-10T18:45:42.439317+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:43.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:43 smithi165 bash[22591]: audit 2024-09-10T18:45:43.320831+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:43.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:43 smithi165 bash[22591]: audit 2024-09-10T18:45:43.328944+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:43.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:43 smithi165 bash[22591]: audit 2024-09-10T18:45:43.336851+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[18629]: cluster 2024-09-10T18:45:42.439317+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[18629]: audit 2024-09-10T18:45:43.320831+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[18629]: audit 2024-09-10T18:45:43.328944+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:43.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[18629]: audit 2024-09-10T18:45:43.336851+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:43.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[32381]: cluster 2024-09-10T18:45:42.439317+0000 mgr.y (mgr.14150) 230 : cluster [DBG] pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:43.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[32381]: audit 2024-09-10T18:45:43.320831+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:43.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[32381]: audit 2024-09-10T18:45:43.328944+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:43.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:43 smithi083 bash[32381]: audit 2024-09-10T18:45:43.336851+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:45.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:45 smithi083 bash[18629]: cluster 2024-09-10T18:45:44.439823+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:45.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:45 smithi083 bash[32381]: cluster 2024-09-10T18:45:44.439823+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:45 smithi165 bash[22591]: cluster 2024-09-10T18:45:44.439823+0000 mgr.y (mgr.14150) 231 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:47.677 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:47.671+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:47.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:47 smithi083 bash[18629]: cluster 2024-09-10T18:45:46.440332+0000 mgr.y (mgr.14150) 232 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:47.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:47 smithi083 bash[32381]: cluster 2024-09-10T18:45:46.440332+0000 mgr.y (mgr.14150) 232 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:47 smithi165 bash[22591]: cluster 2024-09-10T18:45:46.440332+0000 mgr.y (mgr.14150) 232 : cluster [DBG] pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[18629]: cluster 2024-09-10T18:45:48.440871+0000 mgr.y (mgr.14150) 233 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[18629]: audit 2024-09-10T18:45:49.430069+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[18629]: audit 2024-09-10T18:45:49.439573+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[32381]: cluster 2024-09-10T18:45:48.440871+0000 mgr.y (mgr.14150) 233 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[32381]: audit 2024-09-10T18:45:49.430069+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:49 smithi083 bash[32381]: audit 2024-09-10T18:45:49.439573+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:49 smithi165 bash[22591]: cluster 2024-09-10T18:45:48.440871+0000 mgr.y (mgr.14150) 233 : cluster [DBG] pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:49 smithi165 bash[22591]: audit 2024-09-10T18:45:49.430069+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:49 smithi165 bash[22591]: audit 2024-09-10T18:45:49.439573+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:50.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[18629]: audit 2024-09-10T18:45:49.809730+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:50.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[18629]: audit 2024-09-10T18:45:49.811051+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[18629]: audit 2024-09-10T18:45:49.821346+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[18629]: audit 2024-09-10T18:45:50.044608+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[18629]: audit 2024-09-10T18:45:50.048768+0000 mon.a (mon.0) 462 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[32381]: audit 2024-09-10T18:45:49.809730+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[32381]: audit 2024-09-10T18:45:49.811051+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[32381]: audit 2024-09-10T18:45:49.821346+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[32381]: audit 2024-09-10T18:45:50.044608+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:50 smithi083 bash[32381]: audit 2024-09-10T18:45:50.048768+0000 mon.a (mon.0) 462 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:50 smithi165 bash[22591]: audit 2024-09-10T18:45:49.809730+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:45:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:50 smithi165 bash[22591]: audit 2024-09-10T18:45:49.811051+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:45:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:50 smithi165 bash[22591]: audit 2024-09-10T18:45:49.821346+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:50 smithi165 bash[22591]: audit 2024-09-10T18:45:50.044608+0000 mon.b (mon.1) 11 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:50 smithi165 bash[22591]: audit 2024-09-10T18:45:50.048768+0000 mon.a (mon.0) 462 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T18:45:52.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: cluster 2024-09-10T18:45:50.441389+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:52.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: audit 2024-09-10T18:45:50.824627+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: audit 2024-09-10T18:45:50.826689+0000 mon.a (mon.0) 463 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:45:52.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: cluster 2024-09-10T18:45:50.826868+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-10T18:45:52.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: audit 2024-09-10T18:45:50.827090+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:52.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:51 smithi165 bash[22591]: audit 2024-09-10T18:45:50.828663+0000 mon.a (mon.0) 466 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: cluster 2024-09-10T18:45:50.441389+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:52.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: audit 2024-09-10T18:45:50.824627+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: audit 2024-09-10T18:45:50.826689+0000 mon.a (mon.0) 463 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: cluster 2024-09-10T18:45:50.826868+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: audit 2024-09-10T18:45:50.827090+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[18629]: audit 2024-09-10T18:45:50.828663+0000 mon.a (mon.0) 466 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: cluster 2024-09-10T18:45:50.441389+0000 mgr.y (mgr.14150) 234 : cluster [DBG] pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: audit 2024-09-10T18:45:50.824627+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 v1:172.21.15.165:6800/3225515265' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: audit 2024-09-10T18:45:50.826689+0000 mon.a (mon.0) 463 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: cluster 2024-09-10T18:45:50.826868+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: audit 2024-09-10T18:45:50.827090+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:51 smithi083 bash[32381]: audit 2024-09-10T18:45:50.828663+0000 mon.a (mon.0) 466 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:45:52.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:52.675+0000 7fc7da7fc640 10 client.24238.objecter tick 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: cluster 2024-09-10T18:45:51.755661+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: cluster 2024-09-10T18:45:51.755756+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: audit 2024-09-10T18:45:51.833514+0000 mon.a (mon.0) 467 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: cluster 2024-09-10T18:45:51.833754+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: audit 2024-09-10T18:45:51.834359+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:52 smithi165 bash[22591]: audit 2024-09-10T18:45:51.842465+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.166 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.159+0000 7fc7d9ffb640 1 -- 172.21.15.165:0/1516688970 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2934005994) 0x7fc7c8000bc0 con 0x7fc7c4071290 2024-09-10T18:45:53.167 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 4 on host 'smithi165' 2024-09-10T18:45:53.171 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 10 librados: watch_flush enter 2024-09-10T18:45:53.171 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 10 librados: watch_flush exit 2024-09-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 20 client.24238.objecter shutdown clearing up homeless session... 2024-09-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 10 client.24238.objecter successfully canceled tick 2024-09-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 >> v1:172.21.15.83:6800/2630800463 conn(0x7fc7c4071290 legacy=0x7fc7c4073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 >> v1:172.21.15.165:6789/0 conn(0x7fc7dc102ea0 legacy=0x7fc7dc114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:45:53.172 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 shutdown_connections 2024-09-10T18:45:53.173 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 >> 172.21.15.165:0/1516688970 conn(0x7fc7dc0f63f0 msgr2=0x7fc7dc106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:45:53.173 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 shutdown_connections 2024-09-10T18:45:53.173 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 -- 172.21.15.165:0/1516688970 wait complete. 2024-09-10T18:45:53.174 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:45:53.163+0000 7fc7e3865640 1 librados: shutdown 2024-09-10T18:45:53.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: cluster 2024-09-10T18:45:51.755661+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:53.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: cluster 2024-09-10T18:45:51.755756+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: audit 2024-09-10T18:45:51.833514+0000 mon.a (mon.0) 467 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: cluster 2024-09-10T18:45:51.833754+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: audit 2024-09-10T18:45:51.834359+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[18629]: audit 2024-09-10T18:45:51.842465+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: cluster 2024-09-10T18:45:51.755661+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: cluster 2024-09-10T18:45:51.755756+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: audit 2024-09-10T18:45:51.833514+0000 mon.a (mon.0) 467 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: cluster 2024-09-10T18:45:51.833754+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: audit 2024-09-10T18:45:51.834359+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:52 smithi083 bash[32381]: audit 2024-09-10T18:45:51.842465+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:53.843 DEBUG:teuthology.orchestra.run.smithi165:osd.4> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.4.service 2024-09-10T18:45:53.846 INFO:tasks.cephadm:Deploying osd.5 on smithi165 with /dev/vg_nvme/lv_3... 2024-09-10T18:45:53.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T18:45:54.112 INFO:journalctl@ceph.osd.4.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: cluster 2024-09-10T18:45:51.052862+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: cluster 2024-09-10T18:45:51.052967+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: cluster 2024-09-10T18:45:52.441920+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: audit 2024-09-10T18:45:52.845333+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: cluster 2024-09-10T18:45:52.851229+0000 mon.a (mon.0) 472 : cluster [INF] osd.4 v1:172.21.15.165:6800/3225515265 boot 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: cluster 2024-09-10T18:45:52.851281+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:45:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: audit 2024-09-10T18:45:52.851484+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: audit 2024-09-10T18:45:53.147321+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:54.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: audit 2024-09-10T18:45:53.155162+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:54.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:53 smithi165 bash[22591]: audit 2024-09-10T18:45:53.162707+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:54.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: cluster 2024-09-10T18:45:51.052862+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: cluster 2024-09-10T18:45:51.052967+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: cluster 2024-09-10T18:45:52.441920+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: audit 2024-09-10T18:45:52.845333+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: cluster 2024-09-10T18:45:52.851229+0000 mon.a (mon.0) 472 : cluster [INF] osd.4 v1:172.21.15.165:6800/3225515265 boot 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: cluster 2024-09-10T18:45:52.851281+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: audit 2024-09-10T18:45:52.851484+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: audit 2024-09-10T18:45:53.147321+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: audit 2024-09-10T18:45:53.155162+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[18629]: audit 2024-09-10T18:45:53.162707+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:54.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: cluster 2024-09-10T18:45:51.052862+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: cluster 2024-09-10T18:45:51.052967+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: cluster 2024-09-10T18:45:52.441920+0000 mgr.y (mgr.14150) 235 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: audit 2024-09-10T18:45:52.845333+0000 mon.a (mon.0) 471 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: cluster 2024-09-10T18:45:52.851229+0000 mon.a (mon.0) 472 : cluster [INF] osd.4 v1:172.21.15.165:6800/3225515265 boot 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: cluster 2024-09-10T18:45:52.851281+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: audit 2024-09-10T18:45:52.851484+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: audit 2024-09-10T18:45:53.147321+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: audit 2024-09-10T18:45:53.155162+0000 mon.a (mon.0) 476 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:53 smithi083 bash[32381]: audit 2024-09-10T18:45:53.162707+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:45:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:54 smithi165 bash[22591]: cluster 2024-09-10T18:45:53.130684+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:54 smithi165 bash[22591]: cluster 2024-09-10T18:45:53.130769+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:54 smithi165 bash[22591]: cluster 2024-09-10T18:45:53.855569+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-10T18:45:55.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[18629]: cluster 2024-09-10T18:45:53.130684+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[18629]: cluster 2024-09-10T18:45:53.130769+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[18629]: cluster 2024-09-10T18:45:53.855569+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-10T18:45:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[32381]: cluster 2024-09-10T18:45:53.130684+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:45:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[32381]: cluster 2024-09-10T18:45:53.130769+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:45:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:54 smithi083 bash[32381]: cluster 2024-09-10T18:45:53.855569+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-09-10T18:45:56.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:55 smithi083 bash[18629]: cluster 2024-09-10T18:45:54.442485+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:45:56.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:55 smithi083 bash[18629]: cluster 2024-09-10T18:45:54.858706+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-10T18:45:56.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:55 smithi083 bash[32381]: cluster 2024-09-10T18:45:54.442485+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:45:56.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:55 smithi083 bash[32381]: cluster 2024-09-10T18:45:54.858706+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-10T18:45:56.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:55 smithi165 bash[22591]: cluster 2024-09-10T18:45:54.442485+0000 mgr.y (mgr.14150) 236 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:45:56.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:55 smithi165 bash[22591]: cluster 2024-09-10T18:45:54.858706+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-09-10T18:45:57.922 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:45:58.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:57 smithi083 bash[18629]: cluster 2024-09-10T18:45:56.443038+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:45:58.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:57 smithi083 bash[32381]: cluster 2024-09-10T18:45:56.443038+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:45:58.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:57 smithi165 bash[22591]: cluster 2024-09-10T18:45:56.443038+0000 mgr.y (mgr.14150) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:45:59 smithi165 bash[22591]: cluster 2024-09-10T18:45:58.443632+0000 mgr.y (mgr.14150) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:46:00.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:45:59 smithi083 bash[18629]: cluster 2024-09-10T18:45:58.443632+0000 mgr.y (mgr.14150) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:46:00.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:45:59 smithi083 bash[32381]: cluster 2024-09-10T18:45:58.443632+0000 mgr.y (mgr.14150) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T18:46:02.177 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:01 smithi165 bash[22591]: cluster 2024-09-10T18:46:00.444172+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T18:46:02.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:01 smithi083 bash[18629]: cluster 2024-09-10T18:46:00.444172+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T18:46:02.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:01 smithi083 bash[32381]: cluster 2024-09-10T18:46:00.444172+0000 mgr.y (mgr.14150) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T18:46:02.628 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T18:46:02.830 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T18:46:02.851 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-09-10T18:46:02.851 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-09-10T18:46:02.851 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0181893 s, 576 MB/s 2024-09-10T18:46:02.851 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-09-10T18:46:03.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-09-10T18:46:04.148 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:03 smithi165 bash[22591]: cluster 2024-09-10T18:46:02.444806+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-10T18:46:04.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:03 smithi083 bash[18629]: cluster 2024-09-10T18:46:02.444806+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-10T18:46:04.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:03 smithi083 bash[32381]: cluster 2024-09-10T18:46:02.444806+0000 mgr.y (mgr.14150) 240 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-10T18:46:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: cluster 2024-09-10T18:46:04.445378+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: cephadm 2024-09-10T18:46:04.555652+0000 mgr.y (mgr.14150) 242 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.563530+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.571864+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.573373+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: cephadm 2024-09-10T18:46:04.574395+0000 mgr.y (mgr.14150) 243 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 17271M 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.582660+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.584264+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.585536+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:05 smithi165 bash[22591]: audit 2024-09-10T18:46:04.593459+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: cluster 2024-09-10T18:46:04.445378+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: cephadm 2024-09-10T18:46:04.555652+0000 mgr.y (mgr.14150) 242 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.563530+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.571864+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.573373+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: cephadm 2024-09-10T18:46:04.574395+0000 mgr.y (mgr.14150) 243 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 17271M 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.582660+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.584264+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.585536+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[18629]: audit 2024-09-10T18:46:04.593459+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: cluster 2024-09-10T18:46:04.445378+0000 mgr.y (mgr.14150) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: cephadm 2024-09-10T18:46:04.555652+0000 mgr.y (mgr.14150) 242 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.563530+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.571864+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.573373+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: cephadm 2024-09-10T18:46:04.574395+0000 mgr.y (mgr.14150) 243 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 17271M 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.582660+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.584264+0000 mon.a (mon.0) 484 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.585536+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:05 smithi083 bash[32381]: audit 2024-09-10T18:46:04.593459+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:07 smithi165 bash[22591]: cluster 2024-09-10T18:46:06.445952+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-10T18:46:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:07 smithi083 bash[18629]: cluster 2024-09-10T18:46:06.445952+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-10T18:46:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:07 smithi083 bash[32381]: cluster 2024-09-10T18:46:06.445952+0000 mgr.y (mgr.14150) 244 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-10T18:46:08.286 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:46:09.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:09 smithi165 bash[22591]: cluster 2024-09-10T18:46:08.446533+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-10T18:46:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:09 smithi083 bash[18629]: cluster 2024-09-10T18:46:08.446533+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-10T18:46:10.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:09 smithi083 bash[32381]: cluster 2024-09-10T18:46:08.446533+0000 mgr.y (mgr.14150) 245 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-10T18:46:11.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/3091238587 >> v1:172.21.15.165:6789/0 conn(0x7fe0200ff9a0 legacy=0x7fe020101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:46:11.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/3091238587 shutdown_connections 2024-09-10T18:46:11.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/3091238587 >> 172.21.15.165:0/3091238587 conn(0x7fe0200f6c20 msgr2=0x7fe0200f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:46:11.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/3091238587 shutdown_connections 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/3091238587 wait complete. 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 librados: starting msgr at 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 librados: starting objecter 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 Processor -- start 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- start start 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 librados: setting wanted keys 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 librados: calling monclient init 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe02010d320 con 0x7fe0201035c0 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe020118ad0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe020119c70 con 0x7fe0200fb4a0 2024-09-10T18:46:11.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe024e04640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe0200ff9a0 0x7fe020112c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:45540/0 (socket says 172.21.15.165:45540) 2024-09-10T18:46:11.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe024e04640 1 -- 172.21.15.165:0/2673699369 learned_addr learned my addr 172.21.15.165:0/2673699369 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter ms_handle_connect 0x7fe0200ff9a0 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter ms_handle_connect 0x7fe0201035c0 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter ms_handle_connect 0x7fe0200fb4a0 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434055972 0 0) 0x7fe020118ad0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0080034e0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760134684 0 0) 0x7fe020119c70 con 0x7fe0200fb4a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020118ad0 con 0x7fe0200fb4a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527062491 0 0) 0x7fe02010d320 con 0x7fe0201035c0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020119c70 con 0x7fe0201035c0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364700145 0 0) 0x7fe0080034e0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe02010d320 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe0140015d0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3365828244 0 0) 0x7fe02010d320 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 >> v1:172.21.15.83:6790/0 conn(0x7fe0200fb4a0 legacy=0x7fe02010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 >> v1:172.21.15.83:6789/0 conn(0x7fe0201035c0 legacy=0x7fe0201163a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02011ae50 con 0x7fe0200ff9a0 2024-09-10T18:46:11.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe020117b20 con 0x7fe0200ff9a0 2024-09-10T18:46:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 10 client.24265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:46:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe020117ff0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe027869640 1 librados: init done 2024-09-10T18:46:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe014002400 con 0x7fe0200ff9a0 2024-09-10T18:46:11.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.247+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe014003930 con 0x7fe0200ff9a0 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c005040 con 0x7fe0200ff9a0 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fe0140031a0 con 0x7fe0200ff9a0 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(37..37 src has 1..37) v4 ==== 3201+0+0 (unknown 1722148719 0 0) 0x7fe014089590 con 0x7fe0200ff9a0 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 10 client.24265.objecter ms_dispatch 0x7fe020000b90 osd_map(37..37 src has 1..37) v4 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 3 client.24265.objecter handle_osd_map got epochs [37,37] > 0 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 3 client.24265.objecter handle_osd_map decoding full epoch 37 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 20 client.24265.objecter dump_active .. 0 homeless 2024-09-10T18:46:11.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.251+0000 7fe01dffb640 10 client.24265.objecter ms_handle_connect 0x7fe008071210 2024-09-10T18:46:11.267 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.259+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe014056e70 con 0x7fe0200ff9a0 2024-09-10T18:46:11.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:11.443+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe00c000bc0 con 0x7fe008071210 2024-09-10T18:46:11.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:11 smithi083 bash[18629]: cluster 2024-09-10T18:46:10.447060+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:11.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:11 smithi083 bash[32381]: cluster 2024-09-10T18:46:10.447060+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:11 smithi165 bash[22591]: cluster 2024-09-10T18:46:10.447060+0000 mgr.y (mgr.14150) 246 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:12.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[18629]: audit 2024-09-10T18:46:11.453778+0000 mgr.y (mgr.14150) 247 : audit [DBG] from='client.24265 v1:172.21.15.165:0/2673699369' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:46:12.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[18629]: audit 2024-09-10T18:46:11.459659+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[18629]: audit 2024-09-10T18:46:11.465472+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[18629]: audit 2024-09-10T18:46:11.466931+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[32381]: audit 2024-09-10T18:46:11.453778+0000 mgr.y (mgr.14150) 247 : audit [DBG] from='client.24265 v1:172.21.15.165:0/2673699369' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[32381]: audit 2024-09-10T18:46:11.459659+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[32381]: audit 2024-09-10T18:46:11.465472+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:46:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:12 smithi083 bash[32381]: audit 2024-09-10T18:46:11.466931+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:12 smithi165 bash[22591]: audit 2024-09-10T18:46:11.453778+0000 mgr.y (mgr.14150) 247 : audit [DBG] from='client.24265 v1:172.21.15.165:0/2673699369' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:46:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:12 smithi165 bash[22591]: audit 2024-09-10T18:46:11.459659+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:46:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:12 smithi165 bash[22591]: audit 2024-09-10T18:46:11.465472+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:46:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:12 smithi165 bash[22591]: audit 2024-09-10T18:46:11.466931+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:13 smithi083 bash[18629]: cluster 2024-09-10T18:46:12.447620+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:13.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:13 smithi083 bash[32381]: cluster 2024-09-10T18:46:12.447620+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:13 smithi165 bash[22591]: cluster 2024-09-10T18:46:12.447620+0000 mgr.y (mgr.14150) 248 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:15.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:15 smithi083 bash[18629]: cluster 2024-09-10T18:46:14.448166+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:15 smithi083 bash[32381]: cluster 2024-09-10T18:46:14.448166+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:15 smithi165 bash[22591]: cluster 2024-09-10T18:46:14.448166+0000 mgr.y (mgr.14150) 249 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:16.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:16.251+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:17 smithi165 bash[22591]: cluster 2024-09-10T18:46:16.448702+0000 mgr.y (mgr.14150) 250 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:17.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:17 smithi083 bash[18629]: cluster 2024-09-10T18:46:16.448702+0000 mgr.y (mgr.14150) 250 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:17.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:17 smithi083 bash[32381]: cluster 2024-09-10T18:46:16.448702+0000 mgr.y (mgr.14150) 250 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:19.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:19 smithi083 bash[18629]: cluster 2024-09-10T18:46:18.449244+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:19.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:19 smithi083 bash[32381]: cluster 2024-09-10T18:46:18.449244+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:19 smithi165 bash[22591]: cluster 2024-09-10T18:46:18.449244+0000 mgr.y (mgr.14150) 251 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:20.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[18629]: audit 2024-09-10T18:46:19.880219+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.165:0/769569269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[18629]: audit 2024-09-10T18:46:19.884437+0000 mon.a (mon.0) 490 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[18629]: audit 2024-09-10T18:46:19.893411+0000 mon.a (mon.0) 491 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]': finished 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[18629]: cluster 2024-09-10T18:46:19.893607+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[18629]: audit 2024-09-10T18:46:19.893864+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[32381]: audit 2024-09-10T18:46:19.880219+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.165:0/769569269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[32381]: audit 2024-09-10T18:46:19.884437+0000 mon.a (mon.0) 490 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[32381]: audit 2024-09-10T18:46:19.893411+0000 mon.a (mon.0) 491 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]': finished 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[32381]: cluster 2024-09-10T18:46:19.893607+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:46:20.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:20 smithi083 bash[32381]: audit 2024-09-10T18:46:19.893864+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:20 smithi165 bash[22591]: audit 2024-09-10T18:46:19.880219+0000 mon.b (mon.1) 13 : audit [INF] from='client.? v1:172.21.15.165:0/769569269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:20 smithi165 bash[22591]: audit 2024-09-10T18:46:19.884437+0000 mon.a (mon.0) 490 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]: dispatch 2024-09-10T18:46:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:20 smithi165 bash[22591]: audit 2024-09-10T18:46:19.893411+0000 mon.a (mon.0) 491 : audit [INF] from='client.24271 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e"}]': finished 2024-09-10T18:46:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:20 smithi165 bash[22591]: cluster 2024-09-10T18:46:19.893607+0000 mon.a (mon.0) 492 : cluster [DBG] osdmap e38: 6 total, 5 up, 6 in 2024-09-10T18:46:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:20 smithi165 bash[22591]: audit 2024-09-10T18:46:19.893864+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:21.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:21.255+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:21.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:21 smithi083 bash[18629]: cluster 2024-09-10T18:46:20.449788+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:21.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:21 smithi083 bash[18629]: audit 2024-09-10T18:46:20.557188+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.165:0/3327722216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:46:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:21 smithi083 bash[32381]: cluster 2024-09-10T18:46:20.449788+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:21 smithi083 bash[32381]: audit 2024-09-10T18:46:20.557188+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.165:0/3327722216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:46:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:21 smithi165 bash[22591]: cluster 2024-09-10T18:46:20.449788+0000 mgr.y (mgr.14150) 252 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:21 smithi165 bash[22591]: audit 2024-09-10T18:46:20.557188+0000 mon.b (mon.1) 14 : audit [DBG] from='client.? v1:172.21.15.165:0/3327722216' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:46:23.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:23 smithi083 bash[18629]: cluster 2024-09-10T18:46:22.450354+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:23.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:23 smithi083 bash[32381]: cluster 2024-09-10T18:46:22.450354+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:23 smithi165 bash[22591]: cluster 2024-09-10T18:46:22.450354+0000 mgr.y (mgr.14150) 253 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:25.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:25 smithi083 bash[18629]: cluster 2024-09-10T18:46:24.450969+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:25.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:25 smithi083 bash[32381]: cluster 2024-09-10T18:46:24.450969+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:25.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:25 smithi165 bash[22591]: cluster 2024-09-10T18:46:24.450969+0000 mgr.y (mgr.14150) 254 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:26.265 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:26.259+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:27.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[18629]: cluster 2024-09-10T18:46:26.451488+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:27.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[18629]: cluster 2024-09-10T18:46:26.630908+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:27.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[18629]: cluster 2024-09-10T18:46:26.631029+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:27.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[32381]: cluster 2024-09-10T18:46:26.451488+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:27.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[32381]: cluster 2024-09-10T18:46:26.630908+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:27.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:27 smithi083 bash[32381]: cluster 2024-09-10T18:46:26.631029+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:27 smithi165 bash[22591]: cluster 2024-09-10T18:46:26.451488+0000 mgr.y (mgr.14150) 255 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:27 smithi165 bash[22591]: cluster 2024-09-10T18:46:26.630908+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:27 smithi165 bash[22591]: cluster 2024-09-10T18:46:26.631029+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:28.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:28 smithi083 bash[18629]: cluster 2024-09-10T18:46:27.532289+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e39: 6 total, 5 up, 6 in 2024-09-10T18:46:28.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:28 smithi083 bash[18629]: audit 2024-09-10T18:46:27.533026+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:28.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:28 smithi083 bash[32381]: cluster 2024-09-10T18:46:27.532289+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e39: 6 total, 5 up, 6 in 2024-09-10T18:46:28.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:28 smithi083 bash[32381]: audit 2024-09-10T18:46:27.533026+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:28 smithi165 bash[22591]: cluster 2024-09-10T18:46:27.532289+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e39: 6 total, 5 up, 6 in 2024-09-10T18:46:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:28 smithi165 bash[22591]: audit 2024-09-10T18:46:27.533026+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:29.841 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:29 smithi165 bash[22591]: cluster 2024-09-10T18:46:28.452047+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:30.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:29 smithi083 bash[18629]: cluster 2024-09-10T18:46:28.452047+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:29 smithi083 bash[32381]: cluster 2024-09-10T18:46:28.452047+0000 mgr.y (mgr.14150) 256 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:31.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:31.263+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:31.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:31 smithi083 bash[18629]: cluster 2024-09-10T18:46:30.452599+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:31.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:31 smithi083 bash[32381]: cluster 2024-09-10T18:46:30.452599+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:31.838 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:31 smithi165 bash[22591]: cluster 2024-09-10T18:46:30.452599+0000 mgr.y (mgr.14150) 257 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:32.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:32 smithi083 bash[18629]: audit 2024-09-10T18:46:32.473975+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:46:32.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:32 smithi083 bash[18629]: audit 2024-09-10T18:46:32.475481+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:32 smithi083 bash[32381]: audit 2024-09-10T18:46:32.473975+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:46:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:32 smithi083 bash[32381]: audit 2024-09-10T18:46:32.475481+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:32.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:32 smithi165 bash[22591]: audit 2024-09-10T18:46:32.473975+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T18:46:32.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:32 smithi165 bash[22591]: audit 2024-09-10T18:46:32.475481+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:33 smithi083 bash[18629]: cluster 2024-09-10T18:46:32.453154+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:33 smithi083 bash[18629]: cephadm 2024-09-10T18:46:32.476662+0000 mgr.y (mgr.14150) 259 : cephadm [INF] Deploying daemon osd.5 on smithi165 2024-09-10T18:46:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:33 smithi083 bash[32381]: cluster 2024-09-10T18:46:32.453154+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:33 smithi083 bash[32381]: cephadm 2024-09-10T18:46:32.476662+0000 mgr.y (mgr.14150) 259 : cephadm [INF] Deploying daemon osd.5 on smithi165 2024-09-10T18:46:33.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:33 smithi165 bash[22591]: cluster 2024-09-10T18:46:32.453154+0000 mgr.y (mgr.14150) 258 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:33.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:33 smithi165 bash[22591]: cephadm 2024-09-10T18:46:32.476662+0000 mgr.y (mgr.14150) 259 : cephadm [INF] Deploying daemon osd.5 on smithi165 2024-09-10T18:46:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:35 smithi083 bash[18629]: cluster 2024-09-10T18:46:34.453732+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:35 smithi083 bash[32381]: cluster 2024-09-10T18:46:34.453732+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:35 smithi165 bash[22591]: cluster 2024-09-10T18:46:34.453732+0000 mgr.y (mgr.14150) 260 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:36.273 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:36.266+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:37.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: cluster 2024-09-10T18:46:36.454290+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: cluster 2024-09-10T18:46:36.476099+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: cluster 2024-09-10T18:46:36.476241+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: cluster 2024-09-10T18:46:36.551689+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: audit 2024-09-10T18:46:36.551948+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: audit 2024-09-10T18:46:37.041689+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: audit 2024-09-10T18:46:37.049916+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:37 smithi165 bash[22591]: audit 2024-09-10T18:46:37.057913+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:38.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: cluster 2024-09-10T18:46:36.454290+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: cluster 2024-09-10T18:46:36.476099+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: cluster 2024-09-10T18:46:36.476241+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: cluster 2024-09-10T18:46:36.551689+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: audit 2024-09-10T18:46:36.551948+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: audit 2024-09-10T18:46:37.041689+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: audit 2024-09-10T18:46:37.049916+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[18629]: audit 2024-09-10T18:46:37.057913+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: cluster 2024-09-10T18:46:36.454290+0000 mgr.y (mgr.14150) 261 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: cluster 2024-09-10T18:46:36.476099+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: cluster 2024-09-10T18:46:36.476241+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: cluster 2024-09-10T18:46:36.551689+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e40: 6 total, 5 up, 6 in 2024-09-10T18:46:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: audit 2024-09-10T18:46:36.551948+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: audit 2024-09-10T18:46:37.041689+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: audit 2024-09-10T18:46:37.049916+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:37 smithi083 bash[32381]: audit 2024-09-10T18:46:37.057913+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:39 smithi165 bash[22591]: cluster 2024-09-10T18:46:38.454772+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:40.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:39 smithi083 bash[18629]: cluster 2024-09-10T18:46:38.454772+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:40.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:39 smithi083 bash[32381]: cluster 2024-09-10T18:46:38.454772+0000 mgr.y (mgr.14150) 262 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:40.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:40 smithi165 bash[22591]: cluster 2024-09-10T18:46:39.566385+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:46:40.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:40 smithi165 bash[22591]: cluster 2024-09-10T18:46:39.570239+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:46:41.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:40 smithi083 bash[18629]: cluster 2024-09-10T18:46:39.566385+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:46:41.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:40 smithi083 bash[18629]: cluster 2024-09-10T18:46:39.570239+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:46:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:40 smithi083 bash[32381]: cluster 2024-09-10T18:46:39.566385+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:46:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:40 smithi083 bash[32381]: cluster 2024-09-10T18:46:39.570239+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:46:41.277 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:41.270+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:41 smithi165 bash[22591]: cluster 2024-09-10T18:46:40.455284+0000 mgr.y (mgr.14150) 263 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:42.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:41 smithi083 bash[18629]: cluster 2024-09-10T18:46:40.455284+0000 mgr.y (mgr.14150) 263 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:41 smithi083 bash[32381]: cluster 2024-09-10T18:46:40.455284+0000 mgr.y (mgr.14150) 263 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:43.842 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: cluster 2024-09-10T18:46:42.455880+0000 mgr.y (mgr.14150) 264 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:43.842 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: audit 2024-09-10T18:46:43.456260+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:43.842 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: audit 2024-09-10T18:46:43.465288+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:43.842 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: audit 2024-09-10T18:46:43.467155+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:43.843 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: audit 2024-09-10T18:46:43.468812+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:43.843 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:43 smithi165 bash[22591]: audit 2024-09-10T18:46:43.478599+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: cluster 2024-09-10T18:46:42.455880+0000 mgr.y (mgr.14150) 264 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:44.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: audit 2024-09-10T18:46:43.456260+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: audit 2024-09-10T18:46:43.465288+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: audit 2024-09-10T18:46:43.467155+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: audit 2024-09-10T18:46:43.468812+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[18629]: audit 2024-09-10T18:46:43.478599+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: cluster 2024-09-10T18:46:42.455880+0000 mgr.y (mgr.14150) 264 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: audit 2024-09-10T18:46:43.456260+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: audit 2024-09-10T18:46:43.465288+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: audit 2024-09-10T18:46:43.467155+0000 mon.a (mon.0) 505 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: audit 2024-09-10T18:46:43.468812+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:43 smithi083 bash[32381]: audit 2024-09-10T18:46:43.478599+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:44 smithi165 bash[22591]: audit 2024-09-10T18:46:43.899307+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:44 smithi165 bash[22591]: audit 2024-09-10T18:46:43.903511+0000 mon.a (mon.0) 508 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:45.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:44 smithi083 bash[18629]: audit 2024-09-10T18:46:43.899307+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:45.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:44 smithi083 bash[18629]: audit 2024-09-10T18:46:43.903511+0000 mon.a (mon.0) 508 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:45.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:44 smithi083 bash[32381]: audit 2024-09-10T18:46:43.899307+0000 mon.b (mon.1) 15 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:45.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:44 smithi083 bash[32381]: audit 2024-09-10T18:46:43.903511+0000 mon.a (mon.0) 508 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T18:46:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: cluster 2024-09-10T18:46:44.456418+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: audit 2024-09-10T18:46:44.571425+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: audit 2024-09-10T18:46:44.573597+0000 mon.a (mon.0) 509 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:46:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: cluster 2024-09-10T18:46:44.573742+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-10T18:46:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: audit 2024-09-10T18:46:44.573985+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:45 smithi165 bash[22591]: audit 2024-09-10T18:46:44.575465+0000 mon.a (mon.0) 512 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: cluster 2024-09-10T18:46:44.456418+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: audit 2024-09-10T18:46:44.571425+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: audit 2024-09-10T18:46:44.573597+0000 mon.a (mon.0) 509 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: cluster 2024-09-10T18:46:44.573742+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: audit 2024-09-10T18:46:44.573985+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[18629]: audit 2024-09-10T18:46:44.575465+0000 mon.a (mon.0) 512 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: cluster 2024-09-10T18:46:44.456418+0000 mgr.y (mgr.14150) 265 : cluster [DBG] pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: audit 2024-09-10T18:46:44.571425+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 v1:172.21.15.165:6804/571173777' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: audit 2024-09-10T18:46:44.573597+0000 mon.a (mon.0) 509 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: cluster 2024-09-10T18:46:44.573742+0000 mon.a (mon.0) 510 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: audit 2024-09-10T18:46:44.573985+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:45 smithi083 bash[32381]: audit 2024-09-10T18:46:44.575465+0000 mon.a (mon.0) 512 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:46:46.281 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.274+0000 7fe01e7fc640 10 client.24265.objecter tick 2024-09-10T18:46:46.747 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.738+0000 7fe01dffb640 1 -- 172.21.15.165:0/2673699369 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4076113502) 0x7fe00c000bc0 con 0x7fe008071210 2024-09-10T18:46:46.747 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 5 on host 'smithi165' 2024-09-10T18:46:46.751 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 10 librados: watch_flush enter 2024-09-10T18:46:46.751 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 10 librados: watch_flush exit 2024-09-10T18:46:46.751 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 20 client.24265.objecter shutdown clearing up homeless session... 2024-09-10T18:46:46.751 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 10 client.24265.objecter successfully canceled tick 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 >> v1:172.21.15.83:6800/2630800463 conn(0x7fe008071210 legacy=0x7fe0080736c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 >> v1:172.21.15.165:6789/0 conn(0x7fe0200ff9a0 legacy=0x7fe020112c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 shutdown_connections 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 >> 172.21.15.165:0/2673699369 conn(0x7fe0200f6c20 msgr2=0x7fe020106a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 shutdown_connections 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 -- 172.21.15.165:0/2673699369 wait complete. 2024-09-10T18:46:46.752 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:46:46.742+0000 7fe027869640 1 librados: shutdown 2024-09-10T18:46:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:46 smithi165 bash[22591]: audit 2024-09-10T18:46:45.577593+0000 mon.a (mon.0) 513 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:46:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:46 smithi165 bash[22591]: cluster 2024-09-10T18:46:45.577776+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-10T18:46:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:46 smithi165 bash[22591]: audit 2024-09-10T18:46:45.578511+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:46 smithi165 bash[22591]: audit 2024-09-10T18:46:45.585684+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[18629]: audit 2024-09-10T18:46:45.577593+0000 mon.a (mon.0) 513 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:46:47.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[18629]: cluster 2024-09-10T18:46:45.577776+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-10T18:46:47.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[18629]: audit 2024-09-10T18:46:45.578511+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[18629]: audit 2024-09-10T18:46:45.585684+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[32381]: audit 2024-09-10T18:46:45.577593+0000 mon.a (mon.0) 513 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:46:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[32381]: cluster 2024-09-10T18:46:45.577776+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-09-10T18:46:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[32381]: audit 2024-09-10T18:46:45.578511+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:46 smithi083 bash[32381]: audit 2024-09-10T18:46:45.585684+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.400 DEBUG:teuthology.orchestra.run.smithi165:osd.5> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.5.service 2024-09-10T18:46:47.403 INFO:tasks.cephadm:Deploying osd.6 on smithi165 with /dev/vg_nvme/lv_2... 2024-09-10T18:46:47.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T18:46:47.618 INFO:journalctl@ceph.osd.5.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:46:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: cluster 2024-09-10T18:46:44.882179+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: cluster 2024-09-10T18:46:44.882324+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: cluster 2024-09-10T18:46:46.456950+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: audit 2024-09-10T18:46:46.589655+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: cluster 2024-09-10T18:46:46.594774+0000 mon.a (mon.0) 518 : cluster [INF] osd.5 v1:172.21.15.165:6804/571173777 boot 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: cluster 2024-09-10T18:46:46.594834+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: audit 2024-09-10T18:46:46.595095+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: audit 2024-09-10T18:46:46.726335+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: audit 2024-09-10T18:46:46.734120+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:47.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:47 smithi165 bash[22591]: audit 2024-09-10T18:46:46.742365+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: cluster 2024-09-10T18:46:44.882179+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: cluster 2024-09-10T18:46:44.882324+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: cluster 2024-09-10T18:46:46.456950+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: audit 2024-09-10T18:46:46.589655+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: cluster 2024-09-10T18:46:46.594774+0000 mon.a (mon.0) 518 : cluster [INF] osd.5 v1:172.21.15.165:6804/571173777 boot 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: cluster 2024-09-10T18:46:46.594834+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: audit 2024-09-10T18:46:46.595095+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: audit 2024-09-10T18:46:46.726335+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: audit 2024-09-10T18:46:46.734120+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[18629]: audit 2024-09-10T18:46:46.742365+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: cluster 2024-09-10T18:46:44.882179+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: cluster 2024-09-10T18:46:44.882324+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: cluster 2024-09-10T18:46:46.456950+0000 mgr.y (mgr.14150) 266 : cluster [DBG] pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: audit 2024-09-10T18:46:46.589655+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: cluster 2024-09-10T18:46:46.594774+0000 mon.a (mon.0) 518 : cluster [INF] osd.5 v1:172.21.15.165:6804/571173777 boot 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: cluster 2024-09-10T18:46:46.594834+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e43: 6 total, 6 up, 6 in 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: audit 2024-09-10T18:46:46.595095+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: audit 2024-09-10T18:46:46.726335+0000 mon.a (mon.0) 521 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: audit 2024-09-10T18:46:46.734120+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:47 smithi083 bash[32381]: audit 2024-09-10T18:46:46.742365+0000 mon.a (mon.0) 523 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:48 smithi165 bash[22591]: cluster 2024-09-10T18:46:47.600722+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-10T18:46:49.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:48 smithi083 bash[18629]: cluster 2024-09-10T18:46:47.600722+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-10T18:46:49.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:48 smithi083 bash[32381]: cluster 2024-09-10T18:46:47.600722+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e44: 6 total, 6 up, 6 in 2024-09-10T18:46:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:49 smithi165 bash[22591]: cluster 2024-09-10T18:46:48.457492+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:49 smithi165 bash[22591]: cluster 2024-09-10T18:46:48.610361+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-10T18:46:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:49 smithi083 bash[18629]: cluster 2024-09-10T18:46:48.457492+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:49 smithi083 bash[18629]: cluster 2024-09-10T18:46:48.610361+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-10T18:46:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:49 smithi083 bash[32381]: cluster 2024-09-10T18:46:48.457492+0000 mgr.y (mgr.14150) 267 : cluster [DBG] pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:49 smithi083 bash[32381]: cluster 2024-09-10T18:46:48.610361+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e45: 6 total, 6 up, 6 in 2024-09-10T18:46:51.511 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:46:51.673 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:51 smithi165 bash[22591]: cluster 2024-09-10T18:46:50.458028+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:51 smithi083 bash[18629]: cluster 2024-09-10T18:46:50.458028+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:51.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:51 smithi083 bash[32381]: cluster 2024-09-10T18:46:50.458028+0000 mgr.y (mgr.14150) 268 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:53 smithi083 bash[18629]: cluster 2024-09-10T18:46:52.458630+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:53.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:53 smithi083 bash[32381]: cluster 2024-09-10T18:46:52.458630+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:53 smithi165 bash[22591]: cluster 2024-09-10T18:46:52.458630+0000 mgr.y (mgr.14150) 269 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:54 smithi165 bash[22591]: cluster 2024-09-10T18:46:53.788718+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:54 smithi165 bash[22591]: cluster 2024-09-10T18:46:53.788815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:54 smithi083 bash[18629]: cluster 2024-09-10T18:46:53.788718+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:54.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:54 smithi083 bash[18629]: cluster 2024-09-10T18:46:53.788815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:54 smithi083 bash[32381]: cluster 2024-09-10T18:46:53.788718+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:46:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:54 smithi083 bash[32381]: cluster 2024-09-10T18:46:53.788815+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:46:55.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:55 smithi083 bash[18629]: cluster 2024-09-10T18:46:54.459235+0000 mgr.y (mgr.14150) 270 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:55.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:55 smithi083 bash[18629]: cluster 2024-09-10T18:46:54.533956+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-10T18:46:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:55 smithi083 bash[32381]: cluster 2024-09-10T18:46:54.459235+0000 mgr.y (mgr.14150) 270 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:55 smithi083 bash[32381]: cluster 2024-09-10T18:46:54.533956+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-10T18:46:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:55 smithi165 bash[22591]: cluster 2024-09-10T18:46:54.459235+0000 mgr.y (mgr.14150) 270 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:55 smithi165 bash[22591]: cluster 2024-09-10T18:46:54.533956+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e46: 6 total, 6 up, 6 in 2024-09-10T18:46:55.950 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T18:46:56.125 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T18:46:56.147 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-09-10T18:46:56.148 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-09-10T18:46:56.148 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0187537 s, 559 MB/s 2024-09-10T18:46:56.148 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-09-10T18:46:56.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-09-10T18:46:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:57 smithi083 bash[18629]: cluster 2024-09-10T18:46:56.459776+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:57 smithi083 bash[32381]: cluster 2024-09-10T18:46:56.459776+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:57 smithi165 bash[22591]: cluster 2024-09-10T18:46:56.459776+0000 mgr.y (mgr.14150) 271 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:46:59.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: cephadm 2024-09-10T18:46:57.656638+0000 mgr.y (mgr.14150) 272 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.665536+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.680315+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.681453+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.682372+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: cephadm 2024-09-10T18:46:57.683386+0000 mgr.y (mgr.14150) 273 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 8635M 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.691389+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.692605+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.693567+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[18629]: audit 2024-09-10T18:46:57.700172+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: cephadm 2024-09-10T18:46:57.656638+0000 mgr.y (mgr.14150) 272 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.665536+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.680315+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.681453+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.682372+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: cephadm 2024-09-10T18:46:57.683386+0000 mgr.y (mgr.14150) 273 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 8635M 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.691389+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.692605+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.693567+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:58 smithi083 bash[32381]: audit 2024-09-10T18:46:57.700172+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: cephadm 2024-09-10T18:46:57.656638+0000 mgr.y (mgr.14150) 272 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:46:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.665536+0000 mon.a (mon.0) 527 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.680315+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.681453+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.682372+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: cephadm 2024-09-10T18:46:57.683386+0000 mgr.y (mgr.14150) 273 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 8635M 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.691389+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.692605+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.693567+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:46:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:58 smithi165 bash[22591]: audit 2024-09-10T18:46:57.700172+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:46:59 smithi083 bash[18629]: cluster 2024-09-10T18:46:58.460342+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:46:59 smithi083 bash[32381]: cluster 2024-09-10T18:46:58.460342+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:46:59 smithi165 bash[22591]: cluster 2024-09-10T18:46:58.460342+0000 mgr.y (mgr.14150) 274 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:01.549 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:47:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:01 smithi083 bash[18629]: cluster 2024-09-10T18:47:00.460892+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:01 smithi083 bash[32381]: cluster 2024-09-10T18:47:00.460892+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:01 smithi165 bash[22591]: cluster 2024-09-10T18:47:00.460892+0000 mgr.y (mgr.14150) 275 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:03 smithi165 bash[22591]: cluster 2024-09-10T18:47:02.461446+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:03 smithi083 bash[18629]: cluster 2024-09-10T18:47:02.461446+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:03 smithi083 bash[32381]: cluster 2024-09-10T18:47:02.461446+0000 mgr.y (mgr.14150) 276 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:04.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.646+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3717632881 >> v1:172.21.15.165:6789/0 conn(0x7fc9e40ff9a0 legacy=0x7fc9e4101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:04.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.646+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3717632881 shutdown_connections 2024-09-10T18:47:04.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.646+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3717632881 >> 172.21.15.165:0/3717632881 conn(0x7fc9e40f6c20 msgr2=0x7fc9e40f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:47:04.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.646+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3717632881 shutdown_connections 2024-09-10T18:47:04.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3717632881 wait complete. 2024-09-10T18:47:04.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 librados: starting msgr at 2024-09-10T18:47:04.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 librados: starting objecter 2024-09-10T18:47:04.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 Processor -- start 2024-09-10T18:47:04.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 -- start start 2024-09-10T18:47:04.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 librados: setting wanted keys 2024-09-10T18:47:04.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 librados: calling monclient init 2024-09-10T18:47:04.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc9e410c300 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc9e4117b00 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.660 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e8afc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc9e4118ca0 con 0x7fc9e41035c0 2024-09-10T18:47:04.660 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e259c640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc9e40fb4a0 0x7fc9e410b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:35864/0 (socket says 172.21.15.165:35864) 2024-09-10T18:47:04.660 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9e259c640 1 -- 172.21.15.165:0/3401697117 learned_addr learned my addr 172.21.15.165:0/3401697117 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:47:04.661 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter ms_handle_connect 0x7fc9e40fb4a0 2024-09-10T18:47:04.661 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter ms_handle_connect 0x7fc9e41035c0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter ms_handle_connect 0x7fc9e40ff9a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577269156 0 0) 0x7fc9e4117b00 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d00054b0 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16558146 0 0) 0x7fc9e4118ca0 con 0x7fc9e41035c0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e4117b00 con 0x7fc9e41035c0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559335043 0 0) 0x7fc9e410c300 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e4118ca0 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695229873 0 0) 0x7fc9d00054b0 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.662 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9e410c300 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc9d4001940 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3299121718 0 0) 0x7fc9e4117b00 con 0x7fc9e41035c0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9d00054b0 con 0x7fc9e41035c0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173998502 0 0) 0x7fc9e4118ca0 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9e4117b00 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc9dc001a30 con 0x7fc9e41035c0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.650+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc9cc000f50 con 0x7fc9e40ff9a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3223484223 0 0) 0x7fc9e410c300 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 >> v1:172.21.15.83:6790/0 conn(0x7fc9e41035c0 legacy=0x7fc9e41153d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 >> v1:172.21.15.83:6789/0 conn(0x7fc9e40ff9a0 legacy=0x7fc9e4111ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e4119e80 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e4116b50 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.663 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9e8afc640 10 client.24292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:47:04.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e4117020 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9e8afc640 1 librados: init done 2024-09-10T18:47:04.664 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc9d4001cd0 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc9d4005c40 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c4005040 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fc9d4002270 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(46..46 src has 1..46) v4 ==== 3493+0+0 (unknown 2395709936 0 0) 0x7fc9d4089350 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 10 client.24292.objecter ms_dispatch 0x7fc9e4000b90 osd_map(46..46 src has 1..46) v4 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 3 client.24292.objecter handle_osd_map got epochs [46,46] > 0 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 3 client.24292.objecter handle_osd_map decoding full epoch 46 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.654+0000 7fc9daffd640 20 client.24292.objecter dump_active .. 0 homeless 2024-09-10T18:47:04.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.658+0000 7fc9daffd640 10 client.24292.objecter ms_handle_connect 0x7fc9d0071750 2024-09-10T18:47:04.670 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.662+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc9d4056af0 con 0x7fc9e40fb4a0 2024-09-10T18:47:04.853 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:04.842+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9c4000bc0 con 0x7fc9d0071750 2024-09-10T18:47:06.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[18629]: cluster 2024-09-10T18:47:04.462013+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:06.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[18629]: audit 2024-09-10T18:47:04.854886+0000 mgr.y (mgr.14150) 278 : audit [DBG] from='client.24292 v1:172.21.15.165:0/3401697117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[18629]: audit 2024-09-10T18:47:04.861148+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[18629]: audit 2024-09-10T18:47:04.867080+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[18629]: audit 2024-09-10T18:47:04.868614+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[32381]: cluster 2024-09-10T18:47:04.462013+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[32381]: audit 2024-09-10T18:47:04.854886+0000 mgr.y (mgr.14150) 278 : audit [DBG] from='client.24292 v1:172.21.15.165:0/3401697117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[32381]: audit 2024-09-10T18:47:04.861148+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[32381]: audit 2024-09-10T18:47:04.867080+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:47:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:05 smithi083 bash[32381]: audit 2024-09-10T18:47:04.868614+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:05 smithi165 bash[22591]: cluster 2024-09-10T18:47:04.462013+0000 mgr.y (mgr.14150) 277 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:05 smithi165 bash[22591]: audit 2024-09-10T18:47:04.854886+0000 mgr.y (mgr.14150) 278 : audit [DBG] from='client.24292 v1:172.21.15.165:0/3401697117' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:47:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:05 smithi165 bash[22591]: audit 2024-09-10T18:47:04.861148+0000 mon.a (mon.0) 535 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:47:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:05 smithi165 bash[22591]: audit 2024-09-10T18:47:04.867080+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:47:06.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:05 smithi165 bash[22591]: audit 2024-09-10T18:47:04.868614+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:07 smithi083 bash[18629]: cluster 2024-09-10T18:47:06.462585+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:07 smithi083 bash[32381]: cluster 2024-09-10T18:47:06.462585+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:07 smithi165 bash[22591]: cluster 2024-09-10T18:47:06.462585+0000 mgr.y (mgr.14150) 279 : cluster [DBG] pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:09.665 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:09.658+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:09 smithi083 bash[18629]: cluster 2024-09-10T18:47:08.463189+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:09 smithi083 bash[32381]: cluster 2024-09-10T18:47:08.463189+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:09 smithi165 bash[22591]: cluster 2024-09-10T18:47:08.463189+0000 mgr.y (mgr.14150) 280 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:11.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:11 smithi083 bash[18629]: cluster 2024-09-10T18:47:10.463659+0000 mgr.y (mgr.14150) 281 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:11 smithi083 bash[32381]: cluster 2024-09-10T18:47:10.463659+0000 mgr.y (mgr.14150) 281 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:11 smithi165 bash[22591]: cluster 2024-09-10T18:47:10.463659+0000 mgr.y (mgr.14150) 281 : cluster [DBG] pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: cluster 2024-09-10T18:47:12.464221+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: audit 2024-09-10T18:47:13.306792+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.165:0/4134471485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: audit 2024-09-10T18:47:13.311186+0000 mon.a (mon.0) 538 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: audit 2024-09-10T18:47:13.320003+0000 mon.a (mon.0) 539 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]': finished 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: cluster 2024-09-10T18:47:13.320142+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[18629]: audit 2024-09-10T18:47:13.320384+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: cluster 2024-09-10T18:47:12.464221+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: audit 2024-09-10T18:47:13.306792+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.165:0/4134471485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: audit 2024-09-10T18:47:13.311186+0000 mon.a (mon.0) 538 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: audit 2024-09-10T18:47:13.320003+0000 mon.a (mon.0) 539 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]': finished 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: cluster 2024-09-10T18:47:13.320142+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-09-10T18:47:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:13 smithi083 bash[32381]: audit 2024-09-10T18:47:13.320384+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: cluster 2024-09-10T18:47:12.464221+0000 mgr.y (mgr.14150) 282 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: audit 2024-09-10T18:47:13.306792+0000 mon.b (mon.1) 17 : audit [INF] from='client.? v1:172.21.15.165:0/4134471485' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: audit 2024-09-10T18:47:13.311186+0000 mon.a (mon.0) 538 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]: dispatch 2024-09-10T18:47:13.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: audit 2024-09-10T18:47:13.320003+0000 mon.a (mon.0) 539 : audit [INF] from='client.24298 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2"}]': finished 2024-09-10T18:47:13.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: cluster 2024-09-10T18:47:13.320142+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e47: 7 total, 6 up, 7 in 2024-09-10T18:47:13.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:13 smithi165 bash[22591]: audit 2024-09-10T18:47:13.320384+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:14.666 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:14.662+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:14.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:14 smithi083 bash[32381]: audit 2024-09-10T18:47:13.944326+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.165:0/1937523883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:47:14.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:14 smithi083 bash[18629]: audit 2024-09-10T18:47:13.944326+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.165:0/1937523883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:47:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:14 smithi165 bash[22591]: audit 2024-09-10T18:47:13.944326+0000 mon.b (mon.1) 18 : audit [DBG] from='client.? v1:172.21.15.165:0/1937523883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:47:15.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:15 smithi083 bash[18629]: cluster 2024-09-10T18:47:14.464794+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:15 smithi083 bash[32381]: cluster 2024-09-10T18:47:14.464794+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:15 smithi165 bash[22591]: cluster 2024-09-10T18:47:14.464794+0000 mgr.y (mgr.14150) 283 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:17.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:17 smithi083 bash[18629]: cluster 2024-09-10T18:47:16.465358+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:17.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:17 smithi083 bash[32381]: cluster 2024-09-10T18:47:16.465358+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:17 smithi165 bash[22591]: cluster 2024-09-10T18:47:16.465358+0000 mgr.y (mgr.14150) 284 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:18 smithi165 bash[22591]: cluster 2024-09-10T18:47:17.728819+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:18 smithi165 bash[22591]: cluster 2024-09-10T18:47:17.728917+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:18 smithi083 bash[18629]: cluster 2024-09-10T18:47:17.728819+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:18 smithi083 bash[18629]: cluster 2024-09-10T18:47:17.728917+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:18 smithi083 bash[32381]: cluster 2024-09-10T18:47:17.728819+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:18 smithi083 bash[32381]: cluster 2024-09-10T18:47:17.728917+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:19.669 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:19.666+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:19 smithi165 bash[22591]: cluster 2024-09-10T18:47:18.465924+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:19 smithi165 bash[22591]: cluster 2024-09-10T18:47:18.550518+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e48: 7 total, 6 up, 7 in 2024-09-10T18:47:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:19 smithi165 bash[22591]: audit 2024-09-10T18:47:18.550786+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[18629]: cluster 2024-09-10T18:47:18.465924+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[18629]: cluster 2024-09-10T18:47:18.550518+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e48: 7 total, 6 up, 7 in 2024-09-10T18:47:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[18629]: audit 2024-09-10T18:47:18.550786+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[32381]: cluster 2024-09-10T18:47:18.465924+0000 mgr.y (mgr.14150) 285 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[32381]: cluster 2024-09-10T18:47:18.550518+0000 mon.a (mon.0) 542 : cluster [DBG] osdmap e48: 7 total, 6 up, 7 in 2024-09-10T18:47:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:19 smithi083 bash[32381]: audit 2024-09-10T18:47:18.550786+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:21 smithi165 bash[22591]: cluster 2024-09-10T18:47:20.466538+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:22.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:21 smithi083 bash[18629]: cluster 2024-09-10T18:47:20.466538+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:22.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:21 smithi083 bash[32381]: cluster 2024-09-10T18:47:20.466538+0000 mgr.y (mgr.14150) 286 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:23.823 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:23 smithi165 bash[22591]: cluster 2024-09-10T18:47:22.467114+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:23.823 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:23 smithi165 bash[22591]: cluster 2024-09-10T18:47:22.596685+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:23.823 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:23 smithi165 bash[22591]: cluster 2024-09-10T18:47:22.596778+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[18629]: cluster 2024-09-10T18:47:22.467114+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[18629]: cluster 2024-09-10T18:47:22.596685+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[18629]: cluster 2024-09-10T18:47:22.596778+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[32381]: cluster 2024-09-10T18:47:22.467114+0000 mgr.y (mgr.14150) 287 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[32381]: cluster 2024-09-10T18:47:22.596685+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:23 smithi083 bash[32381]: cluster 2024-09-10T18:47:22.596778+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:24.674 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:24.669+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:24.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:24 smithi165 bash[22591]: cluster 2024-09-10T18:47:23.566987+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-10T18:47:24.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:24 smithi165 bash[22591]: audit 2024-09-10T18:47:23.567262+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:25.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:24 smithi083 bash[32381]: cluster 2024-09-10T18:47:23.566987+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-10T18:47:25.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:24 smithi083 bash[32381]: audit 2024-09-10T18:47:23.567262+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:24 smithi083 bash[18629]: cluster 2024-09-10T18:47:23.566987+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e49: 7 total, 6 up, 7 in 2024-09-10T18:47:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:24 smithi083 bash[18629]: audit 2024-09-10T18:47:23.567262+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:25.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:25 smithi165 bash[22591]: cluster 2024-09-10T18:47:24.467654+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:25 smithi083 bash[18629]: cluster 2024-09-10T18:47:24.467654+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:25 smithi083 bash[32381]: cluster 2024-09-10T18:47:24.467654+0000 mgr.y (mgr.14150) 288 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:26.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:26 smithi165 bash[22591]: audit 2024-09-10T18:47:25.778882+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:47:26.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:26 smithi165 bash[22591]: audit 2024-09-10T18:47:25.780457+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:26.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:26 smithi165 bash[22591]: cephadm 2024-09-10T18:47:25.781825+0000 mgr.y (mgr.14150) 289 : cephadm [INF] Deploying daemon osd.6 on smithi165 2024-09-10T18:47:27.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[18629]: audit 2024-09-10T18:47:25.778882+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:47:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[18629]: audit 2024-09-10T18:47:25.780457+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[18629]: cephadm 2024-09-10T18:47:25.781825+0000 mgr.y (mgr.14150) 289 : cephadm [INF] Deploying daemon osd.6 on smithi165 2024-09-10T18:47:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[32381]: audit 2024-09-10T18:47:25.778882+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T18:47:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[32381]: audit 2024-09-10T18:47:25.780457+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:26 smithi083 bash[32381]: cephadm 2024-09-10T18:47:25.781825+0000 mgr.y (mgr.14150) 289 : cephadm [INF] Deploying daemon osd.6 on smithi165 2024-09-10T18:47:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:27 smithi165 bash[22591]: cluster 2024-09-10T18:47:26.468219+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:27 smithi083 bash[18629]: cluster 2024-09-10T18:47:26.468219+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:27 smithi083 bash[32381]: cluster 2024-09-10T18:47:26.468219+0000 mgr.y (mgr.14150) 290 : cluster [DBG] pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:29.678 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:29.673+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:29.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:29 smithi165 bash[22591]: cluster 2024-09-10T18:47:28.468856+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:30.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:29 smithi083 bash[18629]: cluster 2024-09-10T18:47:28.468856+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:30.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:29 smithi083 bash[32381]: cluster 2024-09-10T18:47:28.468856+0000 mgr.y (mgr.14150) 291 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:30 smithi165 bash[22591]: audit 2024-09-10T18:47:30.238525+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:30.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:30 smithi165 bash[22591]: audit 2024-09-10T18:47:30.246876+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:30.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:30 smithi165 bash[22591]: audit 2024-09-10T18:47:30.256402+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:31.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[18629]: audit 2024-09-10T18:47:30.238525+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[18629]: audit 2024-09-10T18:47:30.246876+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[18629]: audit 2024-09-10T18:47:30.256402+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[32381]: audit 2024-09-10T18:47:30.238525+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[32381]: audit 2024-09-10T18:47:30.246876+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:30 smithi083 bash[32381]: audit 2024-09-10T18:47:30.256402+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:31 smithi165 bash[22591]: cluster 2024-09-10T18:47:30.469449+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:32.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:31 smithi083 bash[32381]: cluster 2024-09-10T18:47:30.469449+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:32.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:31 smithi083 bash[18629]: cluster 2024-09-10T18:47:30.469449+0000 mgr.y (mgr.14150) 292 : cluster [DBG] pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:33 smithi165 bash[22591]: cluster 2024-09-10T18:47:32.470038+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:34.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:33 smithi083 bash[32381]: cluster 2024-09-10T18:47:32.470038+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:34.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:33 smithi083 bash[18629]: cluster 2024-09-10T18:47:32.470038+0000 mgr.y (mgr.14150) 293 : cluster [DBG] pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:34.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:34.677+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:35 smithi165 bash[22591]: cluster 2024-09-10T18:47:34.470654+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:36.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:35 smithi083 bash[18629]: cluster 2024-09-10T18:47:34.470654+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:36.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:35 smithi083 bash[32381]: cluster 2024-09-10T18:47:34.470654+0000 mgr.y (mgr.14150) 294 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:38.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: cluster 2024-09-10T18:47:36.471243+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:36.706185+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:36.714895+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:36.966025+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:36.970267+0000 mon.a (mon.0) 553 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:37.096874+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:37.098527+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[18629]: audit 2024-09-10T18:47:37.108060+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: cluster 2024-09-10T18:47:36.471243+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:36.706185+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:36.714895+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:36.966025+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:36.970267+0000 mon.a (mon.0) 553 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:37.096874+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:37.098527+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:37 smithi083 bash[32381]: audit 2024-09-10T18:47:37.108060+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: cluster 2024-09-10T18:47:36.471243+0000 mgr.y (mgr.14150) 295 : cluster [DBG] pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:38.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:36.706185+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:36.714895+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:36.966025+0000 mon.b (mon.1) 19 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:36.970267+0000 mon.a (mon.0) 553 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:37.096874+0000 mon.a (mon.0) 554 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:37.098527+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:38.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:37 smithi165 bash[22591]: audit 2024-09-10T18:47:37.108060+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:39.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[18629]: audit 2024-09-10T18:47:37.718370+0000 mon.a (mon.0) 557 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:47:39.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[18629]: cluster 2024-09-10T18:47:37.718482+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[18629]: audit 2024-09-10T18:47:37.718643+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[18629]: audit 2024-09-10T18:47:37.719769+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[18629]: audit 2024-09-10T18:47:37.723983+0000 mon.a (mon.0) 560 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[32381]: audit 2024-09-10T18:47:37.718370+0000 mon.a (mon.0) 557 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[32381]: cluster 2024-09-10T18:47:37.718482+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[32381]: audit 2024-09-10T18:47:37.718643+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[32381]: audit 2024-09-10T18:47:37.719769+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:38 smithi083 bash[32381]: audit 2024-09-10T18:47:37.723983+0000 mon.a (mon.0) 560 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:38 smithi165 bash[22591]: audit 2024-09-10T18:47:37.718370+0000 mon.a (mon.0) 557 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T18:47:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:38 smithi165 bash[22591]: cluster 2024-09-10T18:47:37.718482+0000 mon.a (mon.0) 558 : cluster [DBG] osdmap e50: 7 total, 6 up, 7 in 2024-09-10T18:47:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:38 smithi165 bash[22591]: audit 2024-09-10T18:47:37.718643+0000 mon.a (mon.0) 559 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:38 smithi165 bash[22591]: audit 2024-09-10T18:47:37.719769+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 v1:172.21.15.165:6808/1317922936' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:38 smithi165 bash[22591]: audit 2024-09-10T18:47:37.723983+0000 mon.a (mon.0) 560 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:47:39.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.681+0000 7fc9db7fe640 10 client.24292.objecter tick 2024-09-10T18:47:39.932 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.925+0000 7fc9daffd640 1 -- 172.21.15.165:0/3401697117 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 382385538) 0x7fc9c4000bc0 con 0x7fc9d0071750 2024-09-10T18:47:39.933 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 6 on host 'smithi165' 2024-09-10T18:47:39.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 10 librados: watch_flush enter 2024-09-10T18:47:39.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 10 librados: watch_flush exit 2024-09-10T18:47:39.937 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 20 client.24292.objecter shutdown clearing up homeless session... 2024-09-10T18:47:39.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 10 client.24292.objecter successfully canceled tick 2024-09-10T18:47:39.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 >> v1:172.21.15.83:6800/2630800463 conn(0x7fc9d0071750 legacy=0x7fc9d0073c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:39.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 >> v1:172.21.15.165:6789/0 conn(0x7fc9e40fb4a0 legacy=0x7fc9e410b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:39.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 shutdown_connections 2024-09-10T18:47:39.938 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 >> 172.21.15.165:0/3401697117 conn(0x7fc9e40f6c20 msgr2=0x7fc9e4105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:47:39.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 shutdown_connections 2024-09-10T18:47:39.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 -- 172.21.15.165:0/3401697117 wait complete. 2024-09-10T18:47:39.939 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:39.933+0000 7fc9e8afc640 1 librados: shutdown 2024-09-10T18:47:40.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[18629]: cluster 2024-09-10T18:47:38.471774+0000 mgr.y (mgr.14150) 296 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[18629]: audit 2024-09-10T18:47:38.723641+0000 mon.a (mon.0) 561 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[18629]: cluster 2024-09-10T18:47:38.723883+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[18629]: audit 2024-09-10T18:47:38.724609+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[18629]: audit 2024-09-10T18:47:38.732270+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[32381]: cluster 2024-09-10T18:47:38.471774+0000 mgr.y (mgr.14150) 296 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[32381]: audit 2024-09-10T18:47:38.723641+0000 mon.a (mon.0) 561 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[32381]: cluster 2024-09-10T18:47:38.723883+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[32381]: audit 2024-09-10T18:47:38.724609+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:39 smithi083 bash[32381]: audit 2024-09-10T18:47:38.732270+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:39 smithi165 bash[22591]: cluster 2024-09-10T18:47:38.471774+0000 mgr.y (mgr.14150) 296 : cluster [DBG] pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:39 smithi165 bash[22591]: audit 2024-09-10T18:47:38.723641+0000 mon.a (mon.0) 561 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:47:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:39 smithi165 bash[22591]: cluster 2024-09-10T18:47:38.723883+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e51: 7 total, 6 up, 7 in 2024-09-10T18:47:40.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:39 smithi165 bash[22591]: audit 2024-09-10T18:47:38.724609+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:39 smithi165 bash[22591]: audit 2024-09-10T18:47:38.732270+0000 mon.a (mon.0) 564 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:40.653 DEBUG:teuthology.orchestra.run.smithi165:osd.6> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.6.service 2024-09-10T18:47:40.656 INFO:tasks.cephadm:Deploying osd.7 on smithi165 with /dev/vg_nvme/lv_1... 2024-09-10T18:47:40.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T18:47:40.897 INFO:journalctl@ceph.osd.6.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:47:41.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: cluster 2024-09-10T18:47:37.999874+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: cluster 2024-09-10T18:47:37.999981+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: cluster 2024-09-10T18:47:39.734076+0000 mon.a (mon.0) 565 : cluster [INF] osd.6 v1:172.21.15.165:6808/1317922936 boot 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: cluster 2024-09-10T18:47:39.734295+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e52: 7 total, 7 up, 7 in 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: audit 2024-09-10T18:47:39.734934+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: audit 2024-09-10T18:47:39.916009+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: audit 2024-09-10T18:47:39.923434+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: audit 2024-09-10T18:47:39.929310+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[18629]: cluster 2024-09-10T18:47:40.434038+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e53: 7 total, 7 up, 7 in 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: cluster 2024-09-10T18:47:37.999874+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: cluster 2024-09-10T18:47:37.999981+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: cluster 2024-09-10T18:47:39.734076+0000 mon.a (mon.0) 565 : cluster [INF] osd.6 v1:172.21.15.165:6808/1317922936 boot 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: cluster 2024-09-10T18:47:39.734295+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e52: 7 total, 7 up, 7 in 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: audit 2024-09-10T18:47:39.734934+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: audit 2024-09-10T18:47:39.916009+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: audit 2024-09-10T18:47:39.923434+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: audit 2024-09-10T18:47:39.929310+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:40 smithi083 bash[32381]: cluster 2024-09-10T18:47:40.434038+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e53: 7 total, 7 up, 7 in 2024-09-10T18:47:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: cluster 2024-09-10T18:47:37.999874+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: cluster 2024-09-10T18:47:37.999981+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: cluster 2024-09-10T18:47:39.734076+0000 mon.a (mon.0) 565 : cluster [INF] osd.6 v1:172.21.15.165:6808/1317922936 boot 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: cluster 2024-09-10T18:47:39.734295+0000 mon.a (mon.0) 566 : cluster [DBG] osdmap e52: 7 total, 7 up, 7 in 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: audit 2024-09-10T18:47:39.734934+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: audit 2024-09-10T18:47:39.916009+0000 mon.a (mon.0) 568 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: audit 2024-09-10T18:47:39.923434+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: audit 2024-09-10T18:47:39.929310+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:41.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:40 smithi165 bash[22591]: cluster 2024-09-10T18:47:40.434038+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e53: 7 total, 7 up, 7 in 2024-09-10T18:47:42.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:41 smithi083 bash[18629]: cluster 2024-09-10T18:47:40.472305+0000 mgr.y (mgr.14150) 297 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:41 smithi083 bash[18629]: cluster 2024-09-10T18:47:41.443293+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-10T18:47:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:41 smithi083 bash[32381]: cluster 2024-09-10T18:47:40.472305+0000 mgr.y (mgr.14150) 297 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:41 smithi083 bash[32381]: cluster 2024-09-10T18:47:41.443293+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-10T18:47:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:41 smithi165 bash[22591]: cluster 2024-09-10T18:47:40.472305+0000 mgr.y (mgr.14150) 297 : cluster [DBG] pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T18:47:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:41 smithi165 bash[22591]: cluster 2024-09-10T18:47:41.443293+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e54: 7 total, 7 up, 7 in 2024-09-10T18:47:44.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:43 smithi083 bash[18629]: cluster 2024-09-10T18:47:42.472883+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v285: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:43 smithi083 bash[32381]: cluster 2024-09-10T18:47:42.472883+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v285: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:43 smithi165 bash[22591]: cluster 2024-09-10T18:47:42.472883+0000 mgr.y (mgr.14150) 298 : cluster [DBG] pgmap v285: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:44.697 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:47:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:45 smithi083 bash[18629]: cluster 2024-09-10T18:47:44.473445+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v286: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:45 smithi083 bash[32381]: cluster 2024-09-10T18:47:44.473445+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v286: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:45 smithi165 bash[22591]: cluster 2024-09-10T18:47:44.473445+0000 mgr.y (mgr.14150) 299 : cluster [DBG] pgmap v286: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[18629]: cluster 2024-09-10T18:47:46.474040+0000 mgr.y (mgr.14150) 300 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[18629]: cluster 2024-09-10T18:47:46.950896+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:47:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[18629]: cluster 2024-09-10T18:47:46.955556+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:47:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[32381]: cluster 2024-09-10T18:47:46.474040+0000 mgr.y (mgr.14150) 300 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[32381]: cluster 2024-09-10T18:47:46.950896+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:47:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:47 smithi083 bash[32381]: cluster 2024-09-10T18:47:46.955556+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:47:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:47 smithi165 bash[22591]: cluster 2024-09-10T18:47:46.474040+0000 mgr.y (mgr.14150) 300 : cluster [DBG] pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:47 smithi165 bash[22591]: cluster 2024-09-10T18:47:46.950896+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:47:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:47 smithi165 bash[22591]: cluster 2024-09-10T18:47:46.955556+0000 osd.6 (osd.6) 4 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:47:49.708 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T18:47:49.935 INFO:teuthology.orchestra.run.smithi165.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T18:47:49.960 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10+0 records in 2024-09-10T18:47:49.960 INFO:teuthology.orchestra.run.smithi165.stderr:10+0 records out 2024-09-10T18:47:49.960 INFO:teuthology.orchestra.run.smithi165.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.02161 s, 485 MB/s 2024-09-10T18:47:49.960 INFO:teuthology.orchestra.run.smithi165.stderr:--> Zapping successful for: 2024-09-10T18:47:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:49 smithi083 bash[18629]: cluster 2024-09-10T18:47:48.474606+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:50.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:49 smithi083 bash[32381]: cluster 2024-09-10T18:47:48.474606+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:50.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:49 smithi165 bash[22591]: cluster 2024-09-10T18:47:48.474606+0000 mgr.y (mgr.14150) 301 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:50.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-09-10T18:47:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[18629]: cluster 2024-09-10T18:47:50.475153+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[18629]: cluster 2024-09-10T18:47:50.593544+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[18629]: cluster 2024-09-10T18:47:50.593640+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[32381]: cluster 2024-09-10T18:47:50.475153+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[32381]: cluster 2024-09-10T18:47:50.593544+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:51 smithi083 bash[32381]: cluster 2024-09-10T18:47:50.593640+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:51 smithi165 bash[22591]: cluster 2024-09-10T18:47:50.475153+0000 mgr.y (mgr.14150) 302 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:51 smithi165 bash[22591]: cluster 2024-09-10T18:47:50.593544+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:51 smithi165 bash[22591]: cluster 2024-09-10T18:47:50.593640+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: cluster 2024-09-10T18:47:51.443213+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: cephadm 2024-09-10T18:47:51.462441+0000 mgr.y (mgr.14150) 303 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.472732+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.481122+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.482862+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.484224+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.485513+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: cephadm 2024-09-10T18:47:51.486433+0000 mgr.y (mgr.14150) 304 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 5757M 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.494347+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.496406+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.497631+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[18629]: audit 2024-09-10T18:47:51.505631+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: cluster 2024-09-10T18:47:51.443213+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: cephadm 2024-09-10T18:47:51.462441+0000 mgr.y (mgr.14150) 303 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.472732+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.481122+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.482862+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.484224+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.485513+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: cephadm 2024-09-10T18:47:51.486433+0000 mgr.y (mgr.14150) 304 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 5757M 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.494347+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.496406+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.497631+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:52.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:52 smithi083 bash[32381]: audit 2024-09-10T18:47:51.505631+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: cluster 2024-09-10T18:47:51.443213+0000 mon.a (mon.0) 573 : cluster [DBG] osdmap e55: 7 total, 7 up, 7 in 2024-09-10T18:47:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: cephadm 2024-09-10T18:47:51.462441+0000 mgr.y (mgr.14150) 303 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:47:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.472732+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.481122+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.482862+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.484224+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.485513+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: cephadm 2024-09-10T18:47:51.486433+0000 mgr.y (mgr.14150) 304 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 5757M 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.494347+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.496406+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.497631+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:47:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:52 smithi165 bash[22591]: audit 2024-09-10T18:47:51.505631+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:47:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:53 smithi083 bash[18629]: cluster 2024-09-10T18:47:52.475625+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:53 smithi083 bash[32381]: cluster 2024-09-10T18:47:52.475625+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:53 smithi165 bash[22591]: cluster 2024-09-10T18:47:52.475625+0000 mgr.y (mgr.14150) 305 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:55.385 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:47:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:55 smithi083 bash[18629]: cluster 2024-09-10T18:47:54.476141+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:55 smithi083 bash[32381]: cluster 2024-09-10T18:47:54.476141+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:55 smithi165 bash[22591]: cluster 2024-09-10T18:47:54.476141+0000 mgr.y (mgr.14150) 306 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:56 smithi165 bash[22591]: cluster 2024-09-10T18:47:55.676976+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:56 smithi165 bash[22591]: cluster 2024-09-10T18:47:55.677068+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:57.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:56 smithi083 bash[18629]: cluster 2024-09-10T18:47:55.676976+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:57.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:56 smithi083 bash[18629]: cluster 2024-09-10T18:47:55.677068+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:56 smithi083 bash[32381]: cluster 2024-09-10T18:47:55.676976+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:47:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:56 smithi083 bash[32381]: cluster 2024-09-10T18:47:55.677068+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:47:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:57 smithi165 bash[22591]: cluster 2024-09-10T18:47:56.476702+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:57 smithi165 bash[22591]: cluster 2024-09-10T18:47:56.546735+0000 mon.a (mon.0) 583 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-10T18:47:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:57 smithi083 bash[18629]: cluster 2024-09-10T18:47:56.476702+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:57 smithi083 bash[18629]: cluster 2024-09-10T18:47:56.546735+0000 mon.a (mon.0) 583 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-10T18:47:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:57 smithi083 bash[32381]: cluster 2024-09-10T18:47:56.476702+0000 mgr.y (mgr.14150) 307 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:57 smithi083 bash[32381]: cluster 2024-09-10T18:47:56.546735+0000 mon.a (mon.0) 583 : cluster [DBG] osdmap e56: 7 total, 7 up, 7 in 2024-09-10T18:47:58.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 -- 172.21.15.165:0/2811188382 >> v1:172.21.15.165:6789/0 conn(0x7f76d41070d0 legacy=0x7f76d41094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:58.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 -- 172.21.15.165:0/2811188382 shutdown_connections 2024-09-10T18:47:58.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 -- 172.21.15.165:0/2811188382 >> 172.21.15.165:0/2811188382 conn(0x7f76d4071ef0 msgr2=0x7f76d4074320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:47:58.681 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 -- 172.21.15.165:0/2811188382 shutdown_connections 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 -- 172.21.15.165:0/2811188382 wait complete. 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 librados: starting msgr at 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.673+0000 7f76d914d640 1 librados: starting objecter 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 Processor -- start 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- start start 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 librados: setting wanted keys 2024-09-10T18:47:58.682 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 librados: calling monclient init 2024-09-10T18:47:58.683 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76d4113900 con 0x7f76d41070d0 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76d411f100 con 0x7f76d410acf0 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76d41202a0 con 0x7f76d4102bd0 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d359e640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f76d410acf0 0x7f76d411c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:46186/0 (socket says 172.21.15.165:46186) 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d359e640 1 -- 172.21.15.165:0/1188781746 learned_addr learned my addr 172.21.15.165:0/1188781746 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d2d9d640 1 --1- 172.21.15.165:0/1188781746 >> v1:172.21.15.83:6790/0 conn(0x7f76d4102bd0 0x7f76d4112d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.165:51192/0 (socket says 172.21.15.165:51192) 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76d410acf0 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76d4102bd0 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:58.684 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76d41070d0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845611337 0 0) 0x7f76d411f100 con 0x7f76d410acf0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c00034e0 con 0x7f76d410acf0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879503435 0 0) 0x7f76d41202a0 con 0x7f76d4102bd0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d411f100 con 0x7f76d4102bd0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749094047 0 0) 0x7f76d4113900 con 0x7f76d41070d0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d41202a0 con 0x7f76d41070d0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1017805752 0 0) 0x7f76c00034e0 con 0x7f76d410acf0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76d4113900 con 0x7f76d410acf0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f76cc002cb0 con 0x7f76d410acf0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418798813 0 0) 0x7f76d411f100 con 0x7f76d4102bd0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76c00034e0 con 0x7f76d4102bd0 2024-09-10T18:47:58.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f76c40018c0 con 0x7f76d4102bd0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471897896 0 0) 0x7f76d41202a0 con 0x7f76d41070d0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76d411f100 con 0x7f76d41070d0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805164572 0 0) 0x7f76d4113900 con 0x7f76d410acf0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 >> v1:172.21.15.83:6790/0 conn(0x7f76d4102bd0 legacy=0x7f76d4112d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 >> v1:172.21.15.83:6789/0 conn(0x7f76d41070d0 legacy=0x7f76d41192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4121480 con 0x7f76d410acf0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d411f2f0 con 0x7f76d410acf0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 10 client.24319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d411f740 con 0x7f76d410acf0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 librados: init done 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f76cc003060 con 0x7f76d410acf0 2024-09-10T18:47:58.686 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f76cc005e70 con 0x7f76d410acf0 2024-09-10T18:47:58.687 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.677+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b4005040 con 0x7f76d410acf0 2024-09-10T18:47:58.687 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7f76cc0019a0 con 0x7f76d410acf0 2024-09-10T18:47:58.687 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(56..56 src has 1..56) v4 ==== 3785+0+0 (unknown 4208427012 0 0) 0x7f76cc0897e0 con 0x7f76d410acf0 2024-09-10T18:47:58.687 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 10 client.24319.objecter ms_dispatch 0x7f76d4000b90 osd_map(56..56 src has 1..56) v4 2024-09-10T18:47:58.687 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 3 client.24319.objecter handle_osd_map got epochs [56,56] > 0 2024-09-10T18:47:58.688 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 3 client.24319.objecter handle_osd_map decoding full epoch 56 2024-09-10T18:47:58.688 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 20 client.24319.objecter dump_active .. 0 homeless 2024-09-10T18:47:58.688 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.681+0000 7f76cb7fe640 10 client.24319.objecter ms_handle_connect 0x7f76c0071600 2024-09-10T18:47:58.694 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.689+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f76cc056e10 con 0x7f76d410acf0 2024-09-10T18:47:58.873 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:47:58.865+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f76b4000bc0 con 0x7f76c0071600 2024-09-10T18:47:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:59 smithi165 bash[22591]: cluster 2024-09-10T18:47:58.477269+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:47:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:59 smithi165 bash[22591]: audit 2024-09-10T18:47:58.875410+0000 mgr.y (mgr.14150) 309 : audit [DBG] from='client.24319 v1:172.21.15.165:0/1188781746' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:47:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:59 smithi165 bash[22591]: audit 2024-09-10T18:47:58.881553+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:47:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:59 smithi165 bash[22591]: audit 2024-09-10T18:47:58.887497+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:47:59.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:47:59 smithi165 bash[22591]: audit 2024-09-10T18:47:58.889303+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:00.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[32381]: cluster 2024-09-10T18:47:58.477269+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:00.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[32381]: audit 2024-09-10T18:47:58.875410+0000 mgr.y (mgr.14150) 309 : audit [DBG] from='client.24319 v1:172.21.15.165:0/1188781746' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[32381]: audit 2024-09-10T18:47:58.881553+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[32381]: audit 2024-09-10T18:47:58.887497+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[32381]: audit 2024-09-10T18:47:58.889303+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[18629]: cluster 2024-09-10T18:47:58.477269+0000 mgr.y (mgr.14150) 308 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[18629]: audit 2024-09-10T18:47:58.875410+0000 mgr.y (mgr.14150) 309 : audit [DBG] from='client.24319 v1:172.21.15.165:0/1188781746' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[18629]: audit 2024-09-10T18:47:58.881553+0000 mon.a (mon.0) 584 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[18629]: audit 2024-09-10T18:47:58.887497+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:48:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:47:59 smithi083 bash[18629]: audit 2024-09-10T18:47:58.889303+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:01 smithi165 bash[22591]: cluster 2024-09-10T18:48:00.477839+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:01 smithi083 bash[18629]: cluster 2024-09-10T18:48:00.477839+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:01 smithi083 bash[32381]: cluster 2024-09-10T18:48:00.477839+0000 mgr.y (mgr.14150) 310 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:03.685 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:03.681+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:03.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:03 smithi165 bash[22591]: cluster 2024-09-10T18:48:02.478454+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:03 smithi083 bash[18629]: cluster 2024-09-10T18:48:02.478454+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:04.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:03 smithi083 bash[32381]: cluster 2024-09-10T18:48:02.478454+0000 mgr.y (mgr.14150) 311 : cluster [DBG] pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:05 smithi165 bash[22591]: cluster 2024-09-10T18:48:04.479103+0000 mgr.y (mgr.14150) 312 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:06.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:05 smithi083 bash[32381]: cluster 2024-09-10T18:48:04.479103+0000 mgr.y (mgr.14150) 312 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:05 smithi083 bash[18629]: cluster 2024-09-10T18:48:04.479103+0000 mgr.y (mgr.14150) 312 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: cluster 2024-09-10T18:48:06.479757+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: audit 2024-09-10T18:48:07.539381+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.165:0/1599290299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: audit 2024-09-10T18:48:07.543538+0000 mon.a (mon.0) 587 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: audit 2024-09-10T18:48:07.554621+0000 mon.a (mon.0) 588 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]': finished 2024-09-10T18:48:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: cluster 2024-09-10T18:48:07.554787+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e57: 8 total, 7 up, 8 in 2024-09-10T18:48:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:07 smithi165 bash[22591]: audit 2024-09-10T18:48:07.555002+0000 mon.a (mon.0) 590 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: cluster 2024-09-10T18:48:06.479757+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: audit 2024-09-10T18:48:07.539381+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.165:0/1599290299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: audit 2024-09-10T18:48:07.543538+0000 mon.a (mon.0) 587 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: audit 2024-09-10T18:48:07.554621+0000 mon.a (mon.0) 588 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]': finished 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: cluster 2024-09-10T18:48:07.554787+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e57: 8 total, 7 up, 8 in 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[18629]: audit 2024-09-10T18:48:07.555002+0000 mon.a (mon.0) 590 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: cluster 2024-09-10T18:48:06.479757+0000 mgr.y (mgr.14150) 313 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: audit 2024-09-10T18:48:07.539381+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.165:0/1599290299' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: audit 2024-09-10T18:48:07.543538+0000 mon.a (mon.0) 587 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]: dispatch 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: audit 2024-09-10T18:48:07.554621+0000 mon.a (mon.0) 588 : audit [INF] from='client.24325 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f"}]': finished 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: cluster 2024-09-10T18:48:07.554787+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e57: 8 total, 7 up, 8 in 2024-09-10T18:48:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:07 smithi083 bash[32381]: audit 2024-09-10T18:48:07.555002+0000 mon.a (mon.0) 590 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:08.690 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:08.685+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:08 smithi165 bash[22591]: audit 2024-09-10T18:48:08.246400+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.165:0/3560452374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:48:09.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:08 smithi083 bash[18629]: audit 2024-09-10T18:48:08.246400+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.165:0/3560452374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:48:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:08 smithi083 bash[32381]: audit 2024-09-10T18:48:08.246400+0000 mon.b (mon.1) 22 : audit [DBG] from='client.? v1:172.21.15.165:0/3560452374' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:48:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:09 smithi165 bash[22591]: cluster 2024-09-10T18:48:08.480375+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:09 smithi083 bash[18629]: cluster 2024-09-10T18:48:08.480375+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:09 smithi083 bash[32381]: cluster 2024-09-10T18:48:08.480375+0000 mgr.y (mgr.14150) 314 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:11.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:11 smithi083 bash[18629]: cluster 2024-09-10T18:48:10.480938+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:11.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:11 smithi083 bash[32381]: cluster 2024-09-10T18:48:10.480938+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:11 smithi165 bash[22591]: cluster 2024-09-10T18:48:10.480938+0000 mgr.y (mgr.14150) 315 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:13.693 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:13.689+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[18629]: cluster 2024-09-10T18:48:12.481525+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[18629]: cluster 2024-09-10T18:48:12.890759+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[18629]: cluster 2024-09-10T18:48:12.890880+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[32381]: cluster 2024-09-10T18:48:12.481525+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[32381]: cluster 2024-09-10T18:48:12.890759+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:13 smithi083 bash[32381]: cluster 2024-09-10T18:48:12.890880+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:13 smithi165 bash[22591]: cluster 2024-09-10T18:48:12.481525+0000 mgr.y (mgr.14150) 316 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:13 smithi165 bash[22591]: cluster 2024-09-10T18:48:12.890759+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:13 smithi165 bash[22591]: cluster 2024-09-10T18:48:12.890880+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:14 smithi165 bash[22591]: cluster 2024-09-10T18:48:13.549011+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e58: 8 total, 7 up, 8 in 2024-09-10T18:48:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:14 smithi165 bash[22591]: audit 2024-09-10T18:48:13.549268+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:15.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:14 smithi083 bash[18629]: cluster 2024-09-10T18:48:13.549011+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e58: 8 total, 7 up, 8 in 2024-09-10T18:48:15.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:14 smithi083 bash[18629]: audit 2024-09-10T18:48:13.549268+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:14 smithi083 bash[32381]: cluster 2024-09-10T18:48:13.549011+0000 mon.a (mon.0) 591 : cluster [DBG] osdmap e58: 8 total, 7 up, 8 in 2024-09-10T18:48:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:14 smithi083 bash[32381]: audit 2024-09-10T18:48:13.549268+0000 mon.a (mon.0) 592 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:15 smithi165 bash[22591]: cluster 2024-09-10T18:48:14.482086+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:16.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:15 smithi083 bash[18629]: cluster 2024-09-10T18:48:14.482086+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:15 smithi083 bash[32381]: cluster 2024-09-10T18:48:14.482086+0000 mgr.y (mgr.14150) 317 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:16 smithi165 bash[22591]: cluster 2024-09-10T18:48:15.664645+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:16 smithi165 bash[22591]: cluster 2024-09-10T18:48:15.664741+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:17.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:16 smithi083 bash[18629]: cluster 2024-09-10T18:48:15.664645+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:17.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:16 smithi083 bash[18629]: cluster 2024-09-10T18:48:15.664741+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:16 smithi083 bash[32381]: cluster 2024-09-10T18:48:15.664645+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:16 smithi083 bash[32381]: cluster 2024-09-10T18:48:15.664741+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:17 smithi165 bash[22591]: cluster 2024-09-10T18:48:16.482683+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:17 smithi165 bash[22591]: cluster 2024-09-10T18:48:16.570368+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-10T18:48:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:17 smithi165 bash[22591]: audit 2024-09-10T18:48:16.571567+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:18.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[32381]: cluster 2024-09-10T18:48:16.482683+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:18.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[32381]: cluster 2024-09-10T18:48:16.570368+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-10T18:48:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[32381]: audit 2024-09-10T18:48:16.571567+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[18629]: cluster 2024-09-10T18:48:16.482683+0000 mgr.y (mgr.14150) 318 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[18629]: cluster 2024-09-10T18:48:16.570368+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e59: 8 total, 7 up, 8 in 2024-09-10T18:48:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:17 smithi083 bash[18629]: audit 2024-09-10T18:48:16.571567+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:18.697 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:18.693+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:19 smithi165 bash[22591]: cluster 2024-09-10T18:48:18.483253+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:20.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:19 smithi083 bash[32381]: cluster 2024-09-10T18:48:18.483253+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:19 smithi083 bash[18629]: cluster 2024-09-10T18:48:18.483253+0000 mgr.y (mgr.14150) 319 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:21 smithi165 bash[22591]: cluster 2024-09-10T18:48:20.483850+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:21 smithi165 bash[22591]: audit 2024-09-10T18:48:20.601471+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:48:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:21 smithi165 bash[22591]: audit 2024-09-10T18:48:20.602985+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:21 smithi165 bash[22591]: cephadm 2024-09-10T18:48:20.604268+0000 mgr.y (mgr.14150) 321 : cephadm [INF] Deploying daemon osd.7 on smithi165 2024-09-10T18:48:22.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[18629]: cluster 2024-09-10T18:48:20.483850+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[18629]: audit 2024-09-10T18:48:20.601471+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[18629]: audit 2024-09-10T18:48:20.602985+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[18629]: cephadm 2024-09-10T18:48:20.604268+0000 mgr.y (mgr.14150) 321 : cephadm [INF] Deploying daemon osd.7 on smithi165 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[32381]: cluster 2024-09-10T18:48:20.483850+0000 mgr.y (mgr.14150) 320 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[32381]: audit 2024-09-10T18:48:20.601471+0000 mon.a (mon.0) 595 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[32381]: audit 2024-09-10T18:48:20.602985+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:21 smithi083 bash[32381]: cephadm 2024-09-10T18:48:20.604268+0000 mgr.y (mgr.14150) 321 : cephadm [INF] Deploying daemon osd.7 on smithi165 2024-09-10T18:48:23.701 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:23.697+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:23 smithi165 bash[22591]: cluster 2024-09-10T18:48:22.484463+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:23 smithi083 bash[18629]: cluster 2024-09-10T18:48:22.484463+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:23 smithi083 bash[32381]: cluster 2024-09-10T18:48:22.484463+0000 mgr.y (mgr.14150) 322 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:25.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:25 smithi165 bash[22591]: cluster 2024-09-10T18:48:24.485014+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:25.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:25 smithi165 bash[22591]: audit 2024-09-10T18:48:25.225759+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:25 smithi165 bash[22591]: audit 2024-09-10T18:48:25.232132+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:25 smithi165 bash[22591]: audit 2024-09-10T18:48:25.239994+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[18629]: cluster 2024-09-10T18:48:24.485014+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[18629]: audit 2024-09-10T18:48:25.225759+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[18629]: audit 2024-09-10T18:48:25.232132+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[18629]: audit 2024-09-10T18:48:25.239994+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[32381]: cluster 2024-09-10T18:48:24.485014+0000 mgr.y (mgr.14150) 323 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[32381]: audit 2024-09-10T18:48:25.225759+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[32381]: audit 2024-09-10T18:48:25.232132+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:25 smithi083 bash[32381]: audit 2024-09-10T18:48:25.239994+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:27 smithi165 bash[22591]: cluster 2024-09-10T18:48:26.485560+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:27 smithi083 bash[18629]: cluster 2024-09-10T18:48:26.485560+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:27 smithi083 bash[32381]: cluster 2024-09-10T18:48:26.485560+0000 mgr.y (mgr.14150) 324 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:28.705 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:28.701+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:29.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:29 smithi165 bash[22591]: cluster 2024-09-10T18:48:28.486105+0000 mgr.y (mgr.14150) 325 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:30.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:29 smithi083 bash[18629]: cluster 2024-09-10T18:48:28.486105+0000 mgr.y (mgr.14150) 325 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:29 smithi083 bash[32381]: cluster 2024-09-10T18:48:28.486105+0000 mgr.y (mgr.14150) 325 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:31.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:31 smithi083 bash[18629]: cluster 2024-09-10T18:48:30.486691+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:31 smithi083 bash[32381]: cluster 2024-09-10T18:48:30.486691+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:31 smithi165 bash[22591]: cluster 2024-09-10T18:48:30.486691+0000 mgr.y (mgr.14150) 326 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:32.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:31.485640+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:31.493313+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:31.494887+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:31.496007+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:31.502696+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:32.017151+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[18629]: audit 2024-09-10T18:48:32.021307+0000 mon.a (mon.0) 605 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:31.485640+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:31.493313+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:31.494887+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:31.496007+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:31.502696+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:32.017151+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:32.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:32 smithi083 bash[32381]: audit 2024-09-10T18:48:32.021307+0000 mon.a (mon.0) 605 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:31.485640+0000 mon.a (mon.0) 600 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:31.493313+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:31.494887+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:32.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:31.496007+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:32.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:31.502696+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:32.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:32.017151+0000 mon.b (mon.1) 23 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:32.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:32 smithi165 bash[22591]: audit 2024-09-10T18:48:32.021307+0000 mon.a (mon.0) 605 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T18:48:33.709 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:33.705+0000 7f76cbfff640 10 client.24319.objecter tick 2024-09-10T18:48:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: cluster 2024-09-10T18:48:32.487289+0000 mgr.y (mgr.14150) 327 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:33.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: audit 2024-09-10T18:48:32.505145+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: audit 2024-09-10T18:48:32.505430+0000 mon.a (mon.0) 606 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: cluster 2024-09-10T18:48:32.505514+0000 mon.a (mon.0) 607 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: audit 2024-09-10T18:48:32.505667+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[18629]: audit 2024-09-10T18:48:32.509089+0000 mon.a (mon.0) 609 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: cluster 2024-09-10T18:48:32.487289+0000 mgr.y (mgr.14150) 327 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: audit 2024-09-10T18:48:32.505145+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: audit 2024-09-10T18:48:32.505430+0000 mon.a (mon.0) 606 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:48:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: cluster 2024-09-10T18:48:32.505514+0000 mon.a (mon.0) 607 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-10T18:48:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: audit 2024-09-10T18:48:32.505667+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:33 smithi083 bash[32381]: audit 2024-09-10T18:48:32.509089+0000 mon.a (mon.0) 609 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: cluster 2024-09-10T18:48:32.487289+0000 mgr.y (mgr.14150) 327 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: audit 2024-09-10T18:48:32.505145+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 v1:172.21.15.165:6812/1610151650' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: audit 2024-09-10T18:48:32.505430+0000 mon.a (mon.0) 606 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: cluster 2024-09-10T18:48:32.505514+0000 mon.a (mon.0) 607 : cluster [DBG] osdmap e60: 8 total, 7 up, 8 in 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: audit 2024-09-10T18:48:32.505667+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:33.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:33 smithi165 bash[22591]: audit 2024-09-10T18:48:32.509089+0000 mon.a (mon.0) 609 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-10T18:48:34.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:34 smithi165 bash[22591]: audit 2024-09-10T18:48:33.510077+0000 mon.a (mon.0) 610 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:48:34.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:34 smithi165 bash[22591]: cluster 2024-09-10T18:48:33.510408+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-10T18:48:34.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:34 smithi165 bash[22591]: audit 2024-09-10T18:48:33.511078+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:34.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:34 smithi165 bash[22591]: audit 2024-09-10T18:48:33.519311+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:34.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[18629]: audit 2024-09-10T18:48:33.510077+0000 mon.a (mon.0) 610 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[18629]: cluster 2024-09-10T18:48:33.510408+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[18629]: audit 2024-09-10T18:48:33.511078+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[18629]: audit 2024-09-10T18:48:33.519311+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[32381]: audit 2024-09-10T18:48:33.510077+0000 mon.a (mon.0) 610 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[32381]: cluster 2024-09-10T18:48:33.510408+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e61: 8 total, 7 up, 8 in 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[32381]: audit 2024-09-10T18:48:33.511078+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:34.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:34 smithi083 bash[32381]: audit 2024-09-10T18:48:33.519311+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.438 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.432+0000 7f76cb7fe640 1 -- 172.21.15.165:0/1188781746 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1256149814) 0x7f76b4000bc0 con 0x7f76c0071600 2024-09-10T18:48:35.438 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 7 on host 'smithi165' 2024-09-10T18:48:35.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 10 librados: watch_flush enter 2024-09-10T18:48:35.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 10 librados: watch_flush exit 2024-09-10T18:48:35.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 20 client.24319.objecter shutdown clearing up homeless session... 2024-09-10T18:48:35.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 10 client.24319.objecter successfully canceled tick 2024-09-10T18:48:35.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 >> v1:172.21.15.83:6800/2630800463 conn(0x7f76c0071600 legacy=0x7f76c0073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:35.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 >> v1:172.21.15.165:6789/0 conn(0x7f76d410acf0 legacy=0x7f76d411c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:35.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 shutdown_connections 2024-09-10T18:48:35.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 >> 172.21.15.165:0/1188781746 conn(0x7f76d4071ef0 msgr2=0x7f76d410d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:48:35.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 shutdown_connections 2024-09-10T18:48:35.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 -- 172.21.15.165:0/1188781746 wait complete. 2024-09-10T18:48:35.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:48:35.436+0000 7f76d914d640 1 librados: shutdown 2024-09-10T18:48:35.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:33.006158+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:33.006319+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:34.487915+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: audit 2024-09-10T18:48:34.519909+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:34.528246+0000 mon.a (mon.0) 615 : cluster [INF] osd.7 v1:172.21.15.165:6812/1610151650 boot 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:34.528309+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: audit 2024-09-10T18:48:34.528529+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: audit 2024-09-10T18:48:35.416029+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: audit 2024-09-10T18:48:35.423372+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: audit 2024-09-10T18:48:35.434038+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[18629]: cluster 2024-09-10T18:48:35.445315+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:48:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:33.006158+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:33.006319+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:34.487915+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: audit 2024-09-10T18:48:34.519909+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:34.528246+0000 mon.a (mon.0) 615 : cluster [INF] osd.7 v1:172.21.15.165:6812/1610151650 boot 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:34.528309+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: audit 2024-09-10T18:48:34.528529+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: audit 2024-09-10T18:48:35.416029+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: audit 2024-09-10T18:48:35.423372+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: audit 2024-09-10T18:48:35.434038+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:35 smithi083 bash[32381]: cluster 2024-09-10T18:48:35.445315+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:48:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:33.006158+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:33.006319+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:34.487915+0000 mgr.y (mgr.14150) 328 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: audit 2024-09-10T18:48:34.519909+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:34.528246+0000 mon.a (mon.0) 615 : cluster [INF] osd.7 v1:172.21.15.165:6812/1610151650 boot 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:34.528309+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: audit 2024-09-10T18:48:34.528529+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: audit 2024-09-10T18:48:35.416029+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: audit 2024-09-10T18:48:35.423372+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: audit 2024-09-10T18:48:35.434038+0000 mon.a (mon.0) 620 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:35 smithi165 bash[22591]: cluster 2024-09-10T18:48:35.445315+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-09-10T18:48:36.163 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.7.service 2024-09-10T18:48:36.165 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-10T18:48:36.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T18:48:36.219 INFO:journalctl@ceph.osd.7.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:48:37.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:37 smithi083 bash[32381]: cluster 2024-09-10T18:48:36.449834+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:48:37.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:37 smithi083 bash[32381]: cluster 2024-09-10T18:48:36.488511+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:37 smithi083 bash[18629]: cluster 2024-09-10T18:48:36.449834+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:48:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:37 smithi083 bash[18629]: cluster 2024-09-10T18:48:36.488511+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:37.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:37 smithi165 bash[22591]: cluster 2024-09-10T18:48:36.449834+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-09-10T18:48:37.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:37 smithi165 bash[22591]: cluster 2024-09-10T18:48:36.488511+0000 mgr.y (mgr.14150) 329 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:39 smithi165 bash[22591]: cluster 2024-09-10T18:48:38.489103+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:40.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:39 smithi083 bash[18629]: cluster 2024-09-10T18:48:38.489103+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:39 smithi083 bash[32381]: cluster 2024-09-10T18:48:38.489103+0000 mgr.y (mgr.14150) 330 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:40.924 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:48:41.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:41 smithi165 bash[22591]: cluster 2024-09-10T18:48:40.489649+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:41.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:41 smithi083 bash[32381]: cluster 2024-09-10T18:48:40.489649+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:41 smithi083 bash[18629]: cluster 2024-09-10T18:48:40.489649+0000 mgr.y (mgr.14150) 331 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:43 smithi165 bash[22591]: cluster 2024-09-10T18:48:42.490320+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:44.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:43 smithi083 bash[18629]: cluster 2024-09-10T18:48:42.490320+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:43 smithi083 bash[32381]: cluster 2024-09-10T18:48:42.490320+0000 mgr.y (mgr.14150) 332 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- 172.21.15.83:0/1173801569 >> v1:172.21.15.83:6789/0 conn(0x7f7cf40fb480 legacy=0x7f7cf40fb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- 172.21.15.83:0/1173801569 shutdown_connections 2024-09-10T18:48:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- 172.21.15.83:0/1173801569 >> 172.21.15.83:0/1173801569 conn(0x7f7cf40f6c60 msgr2=0x7f7cf40f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:48:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- 172.21.15.83:0/1173801569 shutdown_connections 2024-09-10T18:48:44.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- 172.21.15.83:0/1173801569 wait complete. 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 librados: starting msgr at 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 librados: starting objecter 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 Processor -- start 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- start start 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 librados: setting wanted keys 2024-09-10T18:48:44.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 librados: calling monclient init 2024-09-10T18:48:44.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7cf41176f0 con 0x7f7cf41035a0 2024-09-10T18:48:44.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7cf41188f0 con 0x7f7cf40fb480 2024-09-10T18:48:44.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf8825640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7cf4119af0 con 0x7f7cf40ff980 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf1d9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7cf40ff980 0x7f7cf4112440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38558/0 (socket says 172.21.15.83:38558) 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7cf41035a0 0x7f7cf4115df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49742/0 (socket says 172.21.15.83:49742) 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf1d9b640 1 -- 172.21.15.83:0/2059982102 learned_addr learned my addr 172.21.15.83:0/2059982102 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7cf259c640 1 --1- 172.21.15.83:0/2059982102 >> v1:172.21.15.165:6789/0 conn(0x7f7cf40fb480 0x7f7cf40665f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:47644/0 (socket says 172.21.15.83:47644) 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cf40ff980 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:44.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cf40fb480 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f7cf41035a0 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388749664 0 0) 0x7f7cf4119af0 con 0x7f7cf40ff980 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cdc0034e0 con 0x7f7cf40ff980 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944387534 0 0) 0x7f7cf41176f0 con 0x7f7cf41035a0 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf4119af0 con 0x7f7cf41035a0 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236031873 0 0) 0x7f7cf41188f0 con 0x7f7cf40fb480 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf41176f0 con 0x7f7cf40fb480 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 461553839 0 0) 0x7f7cdc0034e0 con 0x7f7cf40ff980 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf41188f0 con 0x7f7cf40ff980 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.261+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7ce8001980 con 0x7f7cf40ff980 2024-09-10T18:48:44.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316048454 0 0) 0x7f7cf4119af0 con 0x7f7cf41035a0 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cdc0034e0 con 0x7f7cf41035a0 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475906698 0 0) 0x7f7cf41176f0 con 0x7f7cf40fb480 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf4119af0 con 0x7f7cf40fb480 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7cec001bb0 con 0x7f7cf41035a0 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7ce4001510 con 0x7f7cf40fb480 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1839463 0 0) 0x7f7cf41188f0 con 0x7f7cf40ff980 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 >> v1:172.21.15.165:6789/0 conn(0x7f7cf40fb480 legacy=0x7f7cf40665f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 >> v1:172.21.15.83:6789/0 conn(0x7f7cf41035a0 legacy=0x7f7cf4115df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf411acf0 con 0x7f7cf40ff980 2024-09-10T18:48:44.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf41178e0 con 0x7f7cf40ff980 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7cf8825640 10 client.24347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf4117d30 con 0x7f7cf40ff980 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7cf8825640 1 librados: init done 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7ce8002d00 con 0x7f7cf40ff980 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7ce8005ac0 con 0x7f7cf40ff980 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd4005040 con 0x7f7cf40ff980 2024-09-10T18:48:44.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.265+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7f7ce8001ea0 con 0x7f7cf40ff980 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(65..65 src has 14..65) v4 ==== 4077+0+0 (unknown 2387566313 0 0) 0x7f7ce8089510 con 0x7f7cf40ff980 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 10 client.24347.objecter ms_dispatch 0x7f7cf4000b90 osd_map(65..65 src has 14..65) v4 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 3 client.24347.objecter handle_osd_map got epochs [65,65] > 0 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 3 client.24347.objecter handle_osd_map decoding full epoch 65 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 20 client.24347.objecter dump_active .. 0 homeless 2024-09-10T18:48:44.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.269+0000 7f7ce2ffd640 10 client.24347.objecter ms_handle_connect 0x7f7cdc071650 2024-09-10T18:48:44.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.273+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7ce80565a0 con 0x7f7cf40ff980 2024-09-10T18:48:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.453+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7cd40052f0 con 0x7f7cf40ff980 2024-09-10T18:48:44.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.453+0000 7f7ce2ffd640 1 -- 172.21.15.83:0/2059982102 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v65) v1 ==== 74+0+130 (unknown 127696662 0 2682017352) 0x7f7ce80563c0 con 0x7f7cf40ff980 2024-09-10T18:48:44.457 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 10 librados: watch_flush enter 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 10 librados: watch_flush exit 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 20 client.24347.objecter shutdown clearing up homeless session... 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 10 client.24347.objecter successfully canceled tick 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 >> v1:172.21.15.83:6800/2630800463 conn(0x7f7cdc071650 legacy=0x7f7cdc073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:44.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 >> v1:172.21.15.83:6790/0 conn(0x7f7cf40ff980 legacy=0x7f7cf4112440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 shutdown_connections 2024-09-10T18:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 >> 172.21.15.83:0/2059982102 conn(0x7f7cf40f6c60 msgr2=0x7f7cf40f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 shutdown_connections 2024-09-10T18:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 -- 172.21.15.83:0/2059982102 wait complete. 2024-09-10T18:48:44.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:44.457+0000 7f7cf8825640 1 librados: shutdown 2024-09-10T18:48:44.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: cluster 2024-09-10T18:48:43.201340+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: cluster 2024-09-10T18:48:43.201481+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: cluster 2024-09-10T18:48:43.425080+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: cluster 2024-09-10T18:48:43.425154+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: cluster 2024-09-10T18:48:43.555815+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[18629]: audit 2024-09-10T18:48:44.456524+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.83:0/2059982102' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: cluster 2024-09-10T18:48:43.201340+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: cluster 2024-09-10T18:48:43.201481+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: cluster 2024-09-10T18:48:43.425080+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: cluster 2024-09-10T18:48:43.425154+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: cluster 2024-09-10T18:48:43.555815+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:48:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:44 smithi083 bash[32381]: audit 2024-09-10T18:48:44.456524+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.83:0/2059982102' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:48:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: cluster 2024-09-10T18:48:43.201340+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: cluster 2024-09-10T18:48:43.201481+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: cluster 2024-09-10T18:48:43.425080+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: cluster 2024-09-10T18:48:43.425154+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: cluster 2024-09-10T18:48:43.555815+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-09-10T18:48:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:44 smithi165 bash[22591]: audit 2024-09-10T18:48:44.456524+0000 mon.c (mon.2) 11 : audit [DBG] from='client.? v1:172.21.15.83:0/2059982102' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T18:48:45.216 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":65,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994114,"num_in_osds":8,"osd_in_since":1725994087,"num_remapped_pgs":0} 2024-09-10T18:48:45.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:48:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: cluster 2024-09-10T18:48:44.490717+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.388908+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.396868+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.398055+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.399027+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.400005+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.400749+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.408738+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.410035+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.410983+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:45 smithi165 bash[22591]: audit 2024-09-10T18:48:45.418970+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: cluster 2024-09-10T18:48:44.490717+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.388908+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.396868+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.398055+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.399027+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.400005+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.400749+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.408738+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.410035+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.410983+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[18629]: audit 2024-09-10T18:48:45.418970+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: cluster 2024-09-10T18:48:44.490717+0000 mgr.y (mgr.14150) 333 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.388908+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.396868+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.398055+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.399027+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.400005+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.400749+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.408738+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.410035+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:48:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.410983+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:48:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:45 smithi083 bash[32381]: audit 2024-09-10T18:48:45.418970+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:48:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:46 smithi165 bash[22591]: cephadm 2024-09-10T18:48:45.380802+0000 mgr.y (mgr.14150) 334 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:48:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:46 smithi165 bash[22591]: cephadm 2024-09-10T18:48:45.401322+0000 mgr.y (mgr.14150) 335 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 4317M 2024-09-10T18:48:47.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:46 smithi083 bash[32381]: cephadm 2024-09-10T18:48:45.380802+0000 mgr.y (mgr.14150) 334 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:48:47.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:46 smithi083 bash[32381]: cephadm 2024-09-10T18:48:45.401322+0000 mgr.y (mgr.14150) 335 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 4317M 2024-09-10T18:48:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:46 smithi083 bash[18629]: cephadm 2024-09-10T18:48:45.380802+0000 mgr.y (mgr.14150) 334 : cephadm [INF] Detected new or changed devices on smithi165 2024-09-10T18:48:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:46 smithi083 bash[18629]: cephadm 2024-09-10T18:48:45.401322+0000 mgr.y (mgr.14150) 335 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 4317M 2024-09-10T18:48:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:47 smithi165 bash[22591]: cluster 2024-09-10T18:48:46.491263+0000 mgr.y (mgr.14150) 336 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:47 smithi083 bash[18629]: cluster 2024-09-10T18:48:46.491263+0000 mgr.y (mgr.14150) 336 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:47 smithi083 bash[32381]: cluster 2024-09-10T18:48:46.491263+0000 mgr.y (mgr.14150) 336 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:49 smithi165 bash[22591]: cluster 2024-09-10T18:48:48.491815+0000 mgr.y (mgr.14150) 337 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:49.977 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:48:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:49 smithi083 bash[18629]: cluster 2024-09-10T18:48:48.491815+0000 mgr.y (mgr.14150) 337 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:49 smithi083 bash[32381]: cluster 2024-09-10T18:48:48.491815+0000 mgr.y (mgr.14150) 337 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:50 smithi165 bash[22591]: cluster 2024-09-10T18:48:49.034313+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:50 smithi165 bash[22591]: cluster 2024-09-10T18:48:49.034442+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:50 smithi165 bash[22591]: cluster 2024-09-10T18:48:49.593067+0000 mon.a (mon.0) 634 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:48:51.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[18629]: cluster 2024-09-10T18:48:49.034313+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[18629]: cluster 2024-09-10T18:48:49.034442+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[18629]: cluster 2024-09-10T18:48:49.593067+0000 mon.a (mon.0) 634 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:48:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[32381]: cluster 2024-09-10T18:48:49.034313+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:48:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[32381]: cluster 2024-09-10T18:48:49.034442+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:48:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:50 smithi083 bash[32381]: cluster 2024-09-10T18:48:49.593067+0000 mon.a (mon.0) 634 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-09-10T18:48:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:51 smithi083 bash[18629]: cluster 2024-09-10T18:48:50.492410+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:51 smithi083 bash[32381]: cluster 2024-09-10T18:48:50.492410+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:51 smithi165 bash[22591]: cluster 2024-09-10T18:48:50.492410+0000 mgr.y (mgr.14150) 338 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:53.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- 172.21.15.83:0/1796294772 >> v1:172.21.15.83:6789/0 conn(0x7f5ab8104810 legacy=0x7f5ab8104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:53.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- 172.21.15.83:0/1796294772 shutdown_connections 2024-09-10T18:48:53.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- 172.21.15.83:0/1796294772 >> 172.21.15.83:0/1796294772 conn(0x7f5ab80fa3d0 msgr2=0x7f5ab80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:48:53.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- 172.21.15.83:0/1796294772 shutdown_connections 2024-09-10T18:48:53.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- 172.21.15.83:0/1796294772 wait complete. 2024-09-10T18:48:53.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 librados: starting msgr at 2024-09-10T18:48:53.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 librados: starting objecter 2024-09-10T18:48:53.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 Processor -- start 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- start start 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 librados: setting wanted keys 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 librados: calling monclient init 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ab811e1b0 con 0x7f5ab810b130 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ab811f3b0 con 0x7f5ab8104810 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5abd936640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ab81205b0 con 0x7f5ab81075a0 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5ab67fc640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5ab81075a0 0x7f5ab8100ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35874/0 (socket says 172.21.15.83:35874) 2024-09-10T18:48:53.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5ab67fc640 1 -- 172.21.15.83:0/282275052 learned_addr learned my addr 172.21.15.83:0/282275052 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5ab810b130 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5ab81075a0 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter ms_handle_connect 0x7f5ab8104810 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236595683 0 0) 0x7f5ab811e1b0 con 0x7f5ab810b130 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa00034e0 con 0x7f5ab810b130 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176674696 0 0) 0x7f5ab81205b0 con 0x7f5ab81075a0 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab811e1b0 con 0x7f5ab81075a0 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635739842 0 0) 0x7f5ab811f3b0 con 0x7f5ab8104810 2024-09-10T18:48:53.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab81205b0 con 0x7f5ab8104810 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2244964094 0 0) 0x7f5aa00034e0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab811f3b0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.236+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ab0001c40 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4262747490 0 0) 0x7f5ab811f3b0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 >> v1:172.21.15.83:6790/0 conn(0x7f5ab81075a0 legacy=0x7f5ab8100ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 >> v1:172.21.15.165:6789/0 conn(0x7f5ab8104810 legacy=0x7f5ab81007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab81217b0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab81207a0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5abd936640 10 client.14445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab8120c70 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5abd936640 1 librados: init done 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5ab00022c0 con 0x7f5ab810b130 2024-09-10T18:48:53.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ab00034d0 con 0x7f5ab810b130 2024-09-10T18:48:53.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa4005010 con 0x7f5ab810b130 2024-09-10T18:48:53.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.240+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7f5ab0001e60 con 0x7f5ab810b130 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(66..66 src has 14..66) v4 ==== 4077+0+0 (unknown 3142520624 0 0) 0x7f5ab0086f40 con 0x7f5ab810b130 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 10 client.14445.objecter ms_dispatch 0x7f5ab8000b90 osd_map(66..66 src has 14..66) v4 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 3 client.14445.objecter handle_osd_map got epochs [66,66] > 0 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 3 client.14445.objecter handle_osd_map decoding full epoch 66 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 20 client.14445.objecter dump_active .. 0 homeless 2024-09-10T18:48:53.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.244+0000 7f5a9f7fe640 10 client.14445.objecter ms_handle_connect 0x7f5aa00712f0 2024-09-10T18:48:53.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.248+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5ab0054450 con 0x7f5ab810b130 2024-09-10T18:48:53.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.424+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aa40052c0 con 0x7f5ab810b130 2024-09-10T18:48:53.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.424+0000 7f5a9f7fe640 1 -- 172.21.15.83:0/282275052 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+11159 (unknown 1914645027 0 501731695) 0x7f5ab0053df0 con 0x7f5ab810b130 2024-09-10T18:48:53.429 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:48:53.429 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":66,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","created":"2024-09-10T18:38:38.475667+0000","modified":"2024-09-10T18:48:49.585483+0000","last_up_change":"2024-09-10T18:48:34.513026+0000","last_in_change":"2024-09-10T18:48:07.544682+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:44:06.952078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"35ab8328-74f1-4af8-aa84-797914199445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6801","nonce":2535527483}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6802","nonce":2535527483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6804","nonce":2535527483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6803","nonce":2535527483}]},"public_addr":"172.21.15.83:6801/2535527483","cluster_addr":"172.21.15.83:6802/2535527483","heartbeat_back_addr":"172.21.15.83:6804/2535527483","heartbeat_front_addr":"172.21.15.83:6803/2535527483","state":["exists","up"]},{"osd":1,"uuid":"8e1d1072-767b-49fb-b2f2-fce60263e2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6805","nonce":2309624637}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6806","nonce":2309624637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6808","nonce":2309624637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6807","nonce":2309624637}]},"public_addr":"172.21.15.83:6805/2309624637","cluster_addr":"172.21.15.83:6806/2309624637","heartbeat_back_addr":"172.21.15.83:6808/2309624637","heartbeat_front_addr":"172.21.15.83:6807/2309624637","state":["exists","up"]},{"osd":2,"uuid":"6193b508-03d9-458a-b49f-6b2e24283282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6809","nonce":386094220}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6810","nonce":386094220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6812","nonce":386094220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6811","nonce":386094220}]},"public_addr":"172.21.15.83:6809/386094220","cluster_addr":"172.21.15.83:6810/386094220","heartbeat_back_addr":"172.21.15.83:6812/386094220","heartbeat_front_addr":"172.21.15.83:6811/386094220","state":["exists","up"]},{"osd":3,"uuid":"a28f94ff-2fff-458e-8b69-c5c65d720d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6813","nonce":1279688021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6814","nonce":1279688021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6816","nonce":1279688021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6815","nonce":1279688021}]},"public_addr":"172.21.15.83:6813/1279688021","cluster_addr":"172.21.15.83:6814/1279688021","heartbeat_back_addr":"172.21.15.83:6816/1279688021","heartbeat_front_addr":"172.21.15.83:6815/1279688021","state":["exists","up"]},{"osd":4,"uuid":"a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":3225515265}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":3225515265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":3225515265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":3225515265}]},"public_addr":"172.21.15.165:6800/3225515265","cluster_addr":"172.21.15.165:6801/3225515265","heartbeat_back_addr":"172.21.15.165:6803/3225515265","heartbeat_front_addr":"172.21.15.165:6802/3225515265","state":["exists","up"]},{"osd":5,"uuid":"10543223-db5a-493b-88f3-8705f2b31a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":571173777}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":571173777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":571173777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":571173777}]},"public_addr":"172.21.15.165:6804/571173777","cluster_addr":"172.21.15.165:6805/571173777","heartbeat_back_addr":"172.21.15.165:6807/571173777","heartbeat_front_addr":"172.21.15.165:6806/571173777","state":["exists","up"]},{"osd":6,"uuid":"fcea0ca4-4521-4161-b407-9b1fab1a2ae2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":1317922936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":1317922936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":1317922936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":1317922936}]},"public_addr":"172.21.15.165:6808/1317922936","cluster_addr":"172.21.15.165:6809/1317922936","heartbeat_back_addr":"172.21.15.165:6811/1317922936","heartbeat_front_addr":"172.21.15.165:6810/1317922936","state":["exists","up"]},{"osd":7,"uuid":"8531d286-5ebb-4c96-a026-63d8a17cb21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1610151650}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1610151650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1610151650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1610151650}]},"public_addr":"172.21.15.165:6812/1610151650","cluster_addr":"172.21.15.165:6813/1610151650","heartbeat_back_addr":"172.21.15.165:6815/1610151650","heartbeat_front_addr":"172.21.15.165:6814/1610151650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:15.664745+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:49.034451+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:47:55.677072+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:47:50.593646+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:43.425157+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:12.890884+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:43.201485+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:48:33.006328+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/1156549307":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/2659452540":"2024-09-11T18:39:40.343252+0000","172.21.15.83:6800/2452184936":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/4112054239":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3076437878":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3555551632":"2024-09-11T18:39:40.343252+0000","172.21.15.83:6800/3275901971":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/867303943":"2024-09-11T18:39:06.336264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:48:53.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 10 librados: watch_flush enter 2024-09-10T18:48:53.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 10 librados: watch_flush exit 2024-09-10T18:48:53.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 20 client.14445.objecter shutdown clearing up homeless session... 2024-09-10T18:48:53.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 10 client.14445.objecter successfully canceled tick 2024-09-10T18:48:53.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 >> v1:172.21.15.83:6800/2630800463 conn(0x7f5aa00712f0 legacy=0x7f5aa00737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 >> v1:172.21.15.83:6789/0 conn(0x7f5ab810b130 legacy=0x7f5ab811c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 shutdown_connections 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 >> 172.21.15.83:0/282275052 conn(0x7f5ab80fa3d0 msgr2=0x7f5ab810b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 shutdown_connections 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 -- 172.21.15.83:0/282275052 wait complete. 2024-09-10T18:48:53.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:48:53.428+0000 7f5abd936640 1 librados: shutdown 2024-09-10T18:48:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:53 smithi083 bash[18629]: cluster 2024-09-10T18:48:52.493002+0000 mgr.y (mgr.14150) 339 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:53 smithi083 bash[18629]: audit 2024-09-10T18:48:53.428120+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? v1:172.21.15.83:0/282275052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:48:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:53 smithi083 bash[32381]: cluster 2024-09-10T18:48:52.493002+0000 mgr.y (mgr.14150) 339 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:53 smithi083 bash[32381]: audit 2024-09-10T18:48:53.428120+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? v1:172.21.15.83:0/282275052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:48:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:53 smithi165 bash[22591]: cluster 2024-09-10T18:48:52.493002+0000 mgr.y (mgr.14150) 339 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:53 smithi165 bash[22591]: audit 2024-09-10T18:48:53.428120+0000 mon.a (mon.0) 635 : audit [DBG] from='client.? v1:172.21.15.83:0/282275052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:48:54.152 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:44:06.952078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-10T18:48:54.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-10T18:48:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:55 smithi165 bash[22591]: cluster 2024-09-10T18:48:54.493574+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:56.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:55 smithi083 bash[18629]: cluster 2024-09-10T18:48:54.493574+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:56.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:55 smithi083 bash[32381]: cluster 2024-09-10T18:48:54.493574+0000 mgr.y (mgr.14150) 340 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:57 smithi083 bash[18629]: cluster 2024-09-10T18:48:56.494167+0000 mgr.y (mgr.14150) 341 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:57 smithi083 bash[32381]: cluster 2024-09-10T18:48:56.494167+0000 mgr.y (mgr.14150) 341 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:57 smithi165 bash[22591]: cluster 2024-09-10T18:48:56.494167+0000 mgr.y (mgr.14150) 341 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:48:58.960 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:49:00.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:48:59 smithi083 bash[18629]: cluster 2024-09-10T18:48:58.494771+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:48:59 smithi083 bash[32381]: cluster 2024-09-10T18:48:58.494771+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:48:59 smithi165 bash[22591]: cluster 2024-09-10T18:48:58.494771+0000 mgr.y (mgr.14150) 342 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:01 smithi083 bash[18629]: cluster 2024-09-10T18:49:00.495300+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:02.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:01 smithi083 bash[32381]: cluster 2024-09-10T18:49:00.495300+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:01 smithi165 bash[22591]: cluster 2024-09-10T18:49:00.495300+0000 mgr.y (mgr.14150) 343 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:02.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- 172.21.15.83:0/1603905419 >> v1:172.21.15.83:6790/0 conn(0x7fac14101500 legacy=0x7fac141018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:02.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- 172.21.15.83:0/1603905419 shutdown_connections 2024-09-10T18:49:02.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- 172.21.15.83:0/1603905419 >> 172.21.15.83:0/1603905419 conn(0x7fac140f7060 msgr2=0x7fac140f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:02.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- 172.21.15.83:0/1603905419 shutdown_connections 2024-09-10T18:49:02.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- 172.21.15.83:0/1603905419 wait complete. 2024-09-10T18:49:02.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 librados: starting msgr at 2024-09-10T18:49:02.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 librados: starting objecter 2024-09-10T18:49:02.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 Processor -- start 2024-09-10T18:49:02.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- start start 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 librados: setting wanted keys 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 librados: calling monclient init 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac141079a0 con 0x7fac14104270 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac14107b50 con 0x7fac14107dd0 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac19df8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac14120670 con 0x7fac14101500 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac12ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fac14104270 0x7fac14119800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:42966/0 (socket says 172.21.15.83:42966) 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac12ffd640 1 -- 172.21.15.83:0/2921056297 learned_addr learned my addr 172.21.15.83:0/2921056297 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter ms_handle_connect 0x7fac14104270 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter ms_handle_connect 0x7fac14101500 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:02.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748473169 0 0) 0x7fac141079a0 con 0x7fac14104270 2024-09-10T18:49:02.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf8003120 con 0x7fac14104270 2024-09-10T18:49:02.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663266997 0 0) 0x7fac14120670 con 0x7fac14101500 2024-09-10T18:49:02.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac141079a0 con 0x7fac14101500 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter ms_handle_connect 0x7fac14107dd0 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.328+0000 7fac0bfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4205303963 0 0) 0x7fac141079a0 con 0x7fac14101500 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac14120670 con 0x7fac14101500 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725767288 0 0) 0x7fabf8003120 con 0x7fac14104270 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac141079a0 con 0x7fac14104270 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac00001960 con 0x7fac14104270 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489091169 0 0) 0x7fac14107b50 con 0x7fac14107dd0 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf8003120 con 0x7fac14107dd0 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac04001510 con 0x7fac14101500 2024-09-10T18:49:02.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 437581996 0 0) 0x7fac14120670 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 >> v1:172.21.15.165:6789/0 conn(0x7fac14107dd0 legacy=0x7fac1411cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 >> v1:172.21.15.83:6789/0 conn(0x7fac14104270 legacy=0x7fac14119800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14121810 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14120860 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fac04002300 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac04003690 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac19df8640 10 client.24359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac14120cb0 con 0x7fac14101500 2024-09-10T18:49:02.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac19df8640 1 librados: init done 2024-09-10T18:49:02.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.332+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac140fcc00 con 0x7fac14101500 2024-09-10T18:49:02.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fac04002af0 con 0x7fac14101500 2024-09-10T18:49:02.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(66..66 src has 14..66) v4 ==== 4077+0+0 (unknown 3142520624 0 0) 0x7fac04086f80 con 0x7fac14101500 2024-09-10T18:49:02.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 10 client.24359.objecter ms_dispatch 0x7fac14000b90 osd_map(66..66 src has 14..66) v4 2024-09-10T18:49:02.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 3 client.24359.objecter handle_osd_map got epochs [66,66] > 0 2024-09-10T18:49:02.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 3 client.24359.objecter handle_osd_map decoding full epoch 66 2024-09-10T18:49:02.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 20 client.24359.objecter dump_active .. 0 homeless 2024-09-10T18:49:02.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.336+0000 7fac0bfff640 10 client.24359.objecter ms_handle_connect 0x7fabf8071610 2024-09-10T18:49:02.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.340+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fac040544f0 con 0x7fac14101500 2024-09-10T18:49:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.516+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fac14109840 con 0x7fac14101500 2024-09-10T18:49:02.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.516+0000 7fac0bfff640 1 -- 172.21.15.83:0/2921056297 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (unknown 3721626206 0 2170607528) 0x7fac04053e90 con 0x7fac14101500 2024-09-10T18:49:02.520 INFO:teuthology.orchestra.run.smithi083.stdout:pg_num: 1 2024-09-10T18:49:02.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 10 librados: watch_flush enter 2024-09-10T18:49:02.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 10 librados: watch_flush exit 2024-09-10T18:49:02.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 20 client.24359.objecter shutdown clearing up homeless session... 2024-09-10T18:49:02.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 10 client.24359.objecter successfully canceled tick 2024-09-10T18:49:02.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 >> v1:172.21.15.83:6800/2630800463 conn(0x7fabf8071610 legacy=0x7fabf8073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 >> v1:172.21.15.83:6790/0 conn(0x7fac14101500 legacy=0x7fac141070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 shutdown_connections 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 >> 172.21.15.83:0/2921056297 conn(0x7fac140f7060 msgr2=0x7fac140fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 shutdown_connections 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 -- 172.21.15.83:0/2921056297 wait complete. 2024-09-10T18:49:02.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:49:02.520+0000 7fac19df8640 1 librados: shutdown 2024-09-10T18:49:03.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:02 smithi083 bash[18629]: audit 2024-09-10T18:49:02.520071+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.83:0/2921056297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:49:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:02 smithi083 bash[32381]: audit 2024-09-10T18:49:02.520071+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.83:0/2921056297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:49:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:02 smithi165 bash[22591]: audit 2024-09-10T18:49:02.520071+0000 mon.c (mon.2) 12 : audit [DBG] from='client.? v1:172.21.15.83:0/2921056297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T18:49:03.227 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi083 2024-09-10T18:49:03.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi083=foo.a' 2024-09-10T18:49:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:03 smithi083 bash[18629]: cluster 2024-09-10T18:49:02.495844+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:04.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:03 smithi083 bash[32381]: cluster 2024-09-10T18:49:02.495844+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:04.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:03 smithi165 bash[22591]: cluster 2024-09-10T18:49:02.495844+0000 mgr.y (mgr.14150) 344 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:06.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:05 smithi083 bash[18629]: cluster 2024-09-10T18:49:04.496395+0000 mgr.y (mgr.14150) 345 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:05 smithi083 bash[32381]: cluster 2024-09-10T18:49:04.496395+0000 mgr.y (mgr.14150) 345 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:05 smithi165 bash[22591]: cluster 2024-09-10T18:49:04.496395+0000 mgr.y (mgr.14150) 345 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:07.978 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[18629]: cluster 2024-09-10T18:49:06.496953+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[18629]: cluster 2024-09-10T18:49:06.996890+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:49:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[18629]: cluster 2024-09-10T18:49:07.003709+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:49:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[32381]: cluster 2024-09-10T18:49:06.496953+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[32381]: cluster 2024-09-10T18:49:06.996890+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:49:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:07 smithi083 bash[32381]: cluster 2024-09-10T18:49:07.003709+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:49:08.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:07 smithi165 bash[22591]: cluster 2024-09-10T18:49:06.496953+0000 mgr.y (mgr.14150) 346 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:08.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:07 smithi165 bash[22591]: cluster 2024-09-10T18:49:06.996890+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T18:49:08.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:07 smithi165 bash[22591]: cluster 2024-09-10T18:49:07.003709+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T18:49:09.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:08 smithi083 bash[18629]: cluster 2024-09-10T18:49:07.997955+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:09.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:08 smithi083 bash[18629]: cluster 2024-09-10T18:49:07.998042+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:08 smithi083 bash[32381]: cluster 2024-09-10T18:49:07.997955+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:08 smithi083 bash[32381]: cluster 2024-09-10T18:49:07.998042+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:09.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:08 smithi165 bash[22591]: cluster 2024-09-10T18:49:07.997955+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:09.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:08 smithi165 bash[22591]: cluster 2024-09-10T18:49:07.998042+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:09.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:09 smithi165 bash[22591]: cluster 2024-09-10T18:49:08.497500+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:09.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:09 smithi165 bash[22591]: cluster 2024-09-10T18:49:08.652589+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:49:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:09 smithi083 bash[18629]: cluster 2024-09-10T18:49:08.497500+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:09 smithi083 bash[18629]: cluster 2024-09-10T18:49:08.652589+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:49:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:09 smithi083 bash[32381]: cluster 2024-09-10T18:49:08.497500+0000 mgr.y (mgr.14150) 347 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:09 smithi083 bash[32381]: cluster 2024-09-10T18:49:08.652589+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-09-10T18:49:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 -- 172.21.15.165:0/3453274658 >> v1:172.21.15.165:6789/0 conn(0x7fc4d80668b0 legacy=0x7fc4d80ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 -- 172.21.15.165:0/3453274658 shutdown_connections 2024-09-10T18:49:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 -- 172.21.15.165:0/3453274658 >> 172.21.15.165:0/3453274658 conn(0x7fc4d80f6c40 msgr2=0x7fc4d80f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 -- 172.21.15.165:0/3453274658 shutdown_connections 2024-09-10T18:49:11.188 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 -- 172.21.15.165:0/3453274658 wait complete. 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 librados: starting msgr at 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.180+0000 7fc4e0314640 1 librados: starting objecter 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 Processor -- start 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- start start 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 librados: setting wanted keys 2024-09-10T18:49:11.189 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 librados: calling monclient init 2024-09-10T18:49:11.190 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4d810d2f0 con 0x7fc4d8104690 2024-09-10T18:49:11.190 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4d8118a60 con 0x7fc4d81008b0 2024-09-10T18:49:11.190 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4d8119c40 con 0x7fc4d80668b0 2024-09-10T18:49:11.190 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4dd8af640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc4d81008b0 0x7fc4d8112c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:60158/0 (socket says 172.21.15.165:60158) 2024-09-10T18:49:11.190 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4dd8af640 1 -- 172.21.15.165:0/402574384 learned_addr learned my addr 172.21.15.165:0/402574384 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter ms_handle_connect 0x7fc4d81008b0 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter ms_handle_connect 0x7fc4d8104690 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter ms_handle_connect 0x7fc4d80668b0 2024-09-10T18:49:11.191 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:11.192 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825608402 0 0) 0x7fc4d8118a60 con 0x7fc4d81008b0 2024-09-10T18:49:11.192 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c40034e0 con 0x7fc4d81008b0 2024-09-10T18:49:11.192 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718361455 0 0) 0x7fc4d810d2f0 con 0x7fc4d8104690 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d8118a60 con 0x7fc4d8104690 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215266844 0 0) 0x7fc4d8119c40 con 0x7fc4d80668b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d810d2f0 con 0x7fc4d80668b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812166244 0 0) 0x7fc4c40034e0 con 0x7fc4d81008b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4d8119c40 con 0x7fc4d81008b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4d4001980 con 0x7fc4d81008b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484391723 0 0) 0x7fc4d8118a60 con 0x7fc4d8104690 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c40034e0 con 0x7fc4d8104690 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4d0001c00 con 0x7fc4d8104690 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 280256208 0 0) 0x7fc4d810d2f0 con 0x7fc4d80668b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4d8118a60 con 0x7fc4d80668b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4c8001ee0 con 0x7fc4d80668b0 2024-09-10T18:49:11.193 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3860879727 0 0) 0x7fc4d8119c40 con 0x7fc4d81008b0 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 >> v1:172.21.15.83:6790/0 conn(0x7fc4d80668b0 legacy=0x7fc4d810c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 >> v1:172.21.15.83:6789/0 conn(0x7fc4d8104690 legacy=0x7fc4d8116330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d811ae40 con 0x7fc4d81008b0 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d8117ab0 con 0x7fc4d81008b0 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 10 client.24364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d8117f80 con 0x7fc4d81008b0 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4e0314640 1 librados: init done 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc4d4002000 con 0x7fc4d81008b0 2024-09-10T18:49:11.194 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.184+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4d4005a10 con 0x7fc4d81008b0 2024-09-10T18:49:11.195 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc005040 con 0x7fc4d81008b0 2024-09-10T18:49:11.195 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fc4d401be00 con 0x7fc4d81008b0 2024-09-10T18:49:11.195 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(67..67 src has 14..67) v4 ==== 4077+0+0 (unknown 1424691752 0 0) 0x7fc4d4089580 con 0x7fc4d81008b0 2024-09-10T18:49:11.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 10 client.24364.objecter ms_dispatch 0x7fc4d8000b90 osd_map(67..67 src has 14..67) v4 2024-09-10T18:49:11.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 3 client.24364.objecter handle_osd_map got epochs [67,67] > 0 2024-09-10T18:49:11.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 3 client.24364.objecter handle_osd_map decoding full epoch 67 2024-09-10T18:49:11.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 20 client.24364.objecter dump_active .. 0 homeless 2024-09-10T18:49:11.196 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.188+0000 7fc4ceffd640 10 client.24364.objecter ms_handle_connect 0x7fc4c40716d0 2024-09-10T18:49:11.203 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.196+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc4d4056610 con 0x7fc4d81008b0 2024-09-10T18:49:11.384 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.376+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi083=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4bc000bc0 con 0x7fc4c40716d0 2024-09-10T18:49:11.400 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.392+0000 7fc4ceffd640 1 -- 172.21.15.165:0/402574384 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7fc4bc000bc0 con 0x7fc4c40716d0 2024-09-10T18:49:11.401 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.foo.a update... 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 10 librados: watch_flush enter 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 10 librados: watch_flush exit 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 20 client.24364.objecter shutdown clearing up homeless session... 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 10 client.24364.objecter successfully canceled tick 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 >> v1:172.21.15.83:6800/2630800463 conn(0x7fc4c40716d0 legacy=0x7fc4c4073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:11.403 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 >> v1:172.21.15.165:6789/0 conn(0x7fc4d81008b0 legacy=0x7fc4d8112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:11.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 shutdown_connections 2024-09-10T18:49:11.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 >> 172.21.15.165:0/402574384 conn(0x7fc4d80f6c40 msgr2=0x7fc4d8106b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:11.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 shutdown_connections 2024-09-10T18:49:11.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 -- 172.21.15.165:0/402574384 wait complete. 2024-09-10T18:49:11.404 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:11.396+0000 7fc4e0314640 1 librados: shutdown 2024-09-10T18:49:11.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[32381]: cluster 2024-09-10T18:49:10.498064+0000 mgr.y (mgr.14150) 348 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:11.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[32381]: audit 2024-09-10T18:49:11.399574+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[32381]: audit 2024-09-10T18:49:11.401011+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[18629]: cluster 2024-09-10T18:49:10.498064+0000 mgr.y (mgr.14150) 348 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[18629]: audit 2024-09-10T18:49:11.399574+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:11 smithi083 bash[18629]: audit 2024-09-10T18:49:11.401011+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:11 smithi165 bash[22591]: cluster 2024-09-10T18:49:10.498064+0000 mgr.y (mgr.14150) 348 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:11 smithi165 bash[22591]: audit 2024-09-10T18:49:11.399574+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:11 smithi165 bash[22591]: audit 2024-09-10T18:49:11.401011+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:12.072 DEBUG:teuthology.orchestra.run.smithi083:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-10T18:49:12.074 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi165 2024-09-10T18:49:12.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-10T18:49:12.588 INFO:journalctl@ceph.rgw.foo.a.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:49:12.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.386550+0000 mgr.y (mgr.14150) 349 : audit [DBG] from='client.24364 v1:172.21.15.165:0/402574384' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi083=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: cephadm 2024-09-10T18:49:11.390027+0000 mgr.y (mgr.14150) 350 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.779589+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.781167+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.792382+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.795896+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.804570+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.814723+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: audit 2024-09-10T18:49:11.815947+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[18629]: cephadm 2024-09-10T18:49:11.817331+0000 mgr.y (mgr.14150) 351 : cephadm [INF] Deploying daemon rgw.foo.a on smithi083 2024-09-10T18:49:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.386550+0000 mgr.y (mgr.14150) 349 : audit [DBG] from='client.24364 v1:172.21.15.165:0/402574384' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi083=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: cephadm 2024-09-10T18:49:11.390027+0000 mgr.y (mgr.14150) 350 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.779589+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.781167+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.792382+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.795896+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.804570+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.814723+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: audit 2024-09-10T18:49:11.815947+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:12 smithi083 bash[32381]: cephadm 2024-09-10T18:49:11.817331+0000 mgr.y (mgr.14150) 351 : cephadm [INF] Deploying daemon rgw.foo.a on smithi083 2024-09-10T18:49:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.386550+0000 mgr.y (mgr.14150) 349 : audit [DBG] from='client.24364 v1:172.21.15.165:0/402574384' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi083=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: cephadm 2024-09-10T18:49:11.390027+0000 mgr.y (mgr.14150) 350 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.779589+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.781167+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.792382+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.795896+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.804570+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.814723+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: audit 2024-09-10T18:49:11.815947+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:12 smithi165 bash[22591]: cephadm 2024-09-10T18:49:11.817331+0000 mgr.y (mgr.14150) 351 : cephadm [INF] Deploying daemon rgw.foo.a on smithi083 2024-09-10T18:49:13.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[18629]: cluster 2024-09-10T18:49:12.498650+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:13.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[18629]: cluster 2024-09-10T18:49:12.843741+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[18629]: cluster 2024-09-10T18:49:12.843837+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[32381]: cluster 2024-09-10T18:49:12.498650+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[32381]: cluster 2024-09-10T18:49:12.843741+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:13 smithi083 bash[32381]: cluster 2024-09-10T18:49:12.843837+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:13 smithi165 bash[22591]: cluster 2024-09-10T18:49:12.498650+0000 mgr.y (mgr.14150) 352 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:13 smithi165 bash[22591]: cluster 2024-09-10T18:49:12.843741+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:13 smithi165 bash[22591]: cluster 2024-09-10T18:49:12.843837+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:14.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:14 smithi083 bash[18629]: cluster 2024-09-10T18:49:13.479854+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:49:14.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:14 smithi083 bash[32381]: cluster 2024-09-10T18:49:13.479854+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:49:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:14 smithi165 bash[22591]: cluster 2024-09-10T18:49:13.479854+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-09-10T18:49:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: cluster 2024-09-10T18:49:14.499193+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.575270+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.581092+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.587446+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: cephadm 2024-09-10T18:49:14.588332+0000 mgr.y (mgr.14150) 354 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.592932+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.600088+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:14.639566+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:15 smithi165 bash[22591]: audit 2024-09-10T18:49:15.452028+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: cluster 2024-09-10T18:49:14.499193+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.575270+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.581092+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.587446+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: cephadm 2024-09-10T18:49:14.588332+0000 mgr.y (mgr.14150) 354 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.592932+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.600088+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:14.639566+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[18629]: audit 2024-09-10T18:49:15.452028+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: cluster 2024-09-10T18:49:14.499193+0000 mgr.y (mgr.14150) 353 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.575270+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.581092+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.587446+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: cephadm 2024-09-10T18:49:14.588332+0000 mgr.y (mgr.14150) 354 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi083=foo.a;count:1 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.592932+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.600088+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:14.639566+0000 mon.a (mon.0) 652 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:15 smithi083 bash[32381]: audit 2024-09-10T18:49:15.452028+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:16.832 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:17 smithi165 bash[22591]: cluster 2024-09-10T18:49:16.499755+0000 mgr.y (mgr.14150) 355 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:17 smithi165 bash[22591]: cluster 2024-09-10T18:49:16.591145+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:49:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:17 smithi165 bash[22591]: audit 2024-09-10T18:49:16.597835+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:17 smithi165 bash[22591]: audit 2024-09-10T18:49:16.598351+0000 mon.a (mon.0) 655 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:18.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[18629]: cluster 2024-09-10T18:49:16.499755+0000 mgr.y (mgr.14150) 355 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[18629]: cluster 2024-09-10T18:49:16.591145+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[18629]: audit 2024-09-10T18:49:16.597835+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[18629]: audit 2024-09-10T18:49:16.598351+0000 mon.a (mon.0) 655 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[32381]: cluster 2024-09-10T18:49:16.499755+0000 mgr.y (mgr.14150) 355 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[32381]: cluster 2024-09-10T18:49:16.591145+0000 mon.a (mon.0) 654 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[32381]: audit 2024-09-10T18:49:16.597835+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:17 smithi083 bash[32381]: audit 2024-09-10T18:49:16.598351+0000 mon.a (mon.0) 655 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T18:49:18.870 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:18 smithi165 bash[22591]: audit 2024-09-10T18:49:17.598151+0000 mon.a (mon.0) 656 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:49:18.870 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:18 smithi165 bash[22591]: cluster 2024-09-10T18:49:17.598282+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:49:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:18 smithi083 bash[18629]: audit 2024-09-10T18:49:17.598151+0000 mon.a (mon.0) 656 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:49:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:18 smithi083 bash[18629]: cluster 2024-09-10T18:49:17.598282+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:49:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:18 smithi083 bash[32381]: audit 2024-09-10T18:49:17.598151+0000 mon.a (mon.0) 656 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T18:49:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:18 smithi083 bash[32381]: cluster 2024-09-10T18:49:17.598282+0000 mon.a (mon.0) 657 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-09-10T18:49:19.979 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- 172.21.15.165:0/2769091330 >> v1:172.21.15.165:6789/0 conn(0x7f06941046d0 legacy=0x7f0694106b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:19.979 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- 172.21.15.165:0/2769091330 shutdown_connections 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- 172.21.15.165:0/2769091330 >> 172.21.15.165:0/2769091330 conn(0x7f06940f7d70 msgr2=0x7f06940fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- 172.21.15.165:0/2769091330 shutdown_connections 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- 172.21.15.165:0/2769091330 wait complete. 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 librados: starting msgr at 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 librados: starting objecter 2024-09-10T18:49:19.980 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 Processor -- start 2024-09-10T18:49:19.981 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- start start 2024-09-10T18:49:19.981 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 librados: setting wanted keys 2024-09-10T18:49:19.981 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 librados: calling monclient init 2024-09-10T18:49:19.981 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f069410ff00 con 0x7f0694100ab0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f069411f060 con 0x7f06941046d0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069c1cc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0694120280 con 0x7f06940fc5b0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069a769640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f06941046d0 0x7f069411c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:38456/0 (socket says 172.21.15.165:38456) 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f069a769640 1 -- 172.21.15.165:0/316215529 learned_addr learned my addr 172.21.15.165:0/316215529 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter ms_handle_connect 0x7f0694100ab0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter ms_handle_connect 0x7f06941046d0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter ms_handle_connect 0x7f06940fc5b0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683615104 0 0) 0x7f069411f060 con 0x7f06941046d0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680003720 con 0x7f06941046d0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036365310 0 0) 0x7f0694120280 con 0x7f06940fc5b0 2024-09-10T18:49:19.982 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.972+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069411f060 con 0x7f06940fc5b0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894981599 0 0) 0x7f0680003720 con 0x7f06941046d0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0694120280 con 0x7f06941046d0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f068c001950 con 0x7f06941046d0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058676004 0 0) 0x7f069411f060 con 0x7f06940fc5b0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0680003720 con 0x7f06940fc5b0 2024-09-10T18:49:19.983 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0684001a10 con 0x7f06940fc5b0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1834272287 0 0) 0x7f0694120280 con 0x7f06941046d0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 >> v1:172.21.15.83:6790/0 conn(0x7f06940fc5b0 legacy=0x7f0694102db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 >> v1:172.21.15.83:6789/0 conn(0x7f0694100ab0 legacy=0x7f06941034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694121480 con 0x7f06941046d0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069411f250 con 0x7f06941046d0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f069c1cc640 10 client.24379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f069411f720 con 0x7f06941046d0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f069c1cc640 1 librados: init done 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f068c002780 con 0x7f06941046d0 2024-09-10T18:49:19.984 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f068c003cd0 con 0x7f06941046d0 2024-09-10T18:49:19.985 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.976+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678005040 con 0x7f06941046d0 2024-09-10T18:49:19.985 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7f068c003e50 con 0x7f06941046d0 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(72..72 src has 14..72) v4 ==== 4801+0+0 (unknown 475823313 0 0) 0x7f068c089fd0 con 0x7f06941046d0 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 10 client.24379.objecter ms_dispatch 0x7f0694000b90 osd_map(72..72 src has 14..72) v4 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 3 client.24379.objecter handle_osd_map got epochs [72,72] > 0 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 3 client.24379.objecter handle_osd_map decoding full epoch 72 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 20 client.24379.objecter dump_active .. 0 homeless 2024-09-10T18:49:19.986 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.980+0000 7f068a7fc640 10 client.24379.objecter ms_handle_connect 0x7f0680071600 2024-09-10T18:49:19.997 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:19.992+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f068c057210 con 0x7f06941046d0 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[18629]: cluster 2024-09-10T18:49:18.500415+0000 mgr.y (mgr.14150) 356 : cluster [DBG] pgmap v349: 33 pgs: 5 creating+activating, 25 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[18629]: cluster 2024-09-10T18:49:18.610503+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[18629]: audit 2024-09-10T18:49:18.611923+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[18629]: audit 2024-09-10T18:49:18.621955+0000 mon.a (mon.0) 659 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[32381]: cluster 2024-09-10T18:49:18.500415+0000 mgr.y (mgr.14150) 356 : cluster [DBG] pgmap v349: 33 pgs: 5 creating+activating, 25 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[32381]: cluster 2024-09-10T18:49:18.610503+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[32381]: audit 2024-09-10T18:49:18.611923+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:19 smithi083 bash[32381]: audit 2024-09-10T18:49:18.621955+0000 mon.a (mon.0) 659 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:19 smithi165 bash[22591]: cluster 2024-09-10T18:49:18.500415+0000 mgr.y (mgr.14150) 356 : cluster [DBG] pgmap v349: 33 pgs: 5 creating+activating, 25 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:19 smithi165 bash[22591]: cluster 2024-09-10T18:49:18.610503+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-09-10T18:49:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:19 smithi165 bash[22591]: audit 2024-09-10T18:49:18.611923+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:19 smithi165 bash[22591]: audit 2024-09-10T18:49:18.621955+0000 mon.a (mon.0) 659 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T18:49:20.177 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:20.168+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f0678005870 con 0x7f06941046d0 2024-09-10T18:49:21.034 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'datapool' created 2024-09-10T18:49:21.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.024+0000 7f068a7fc640 1 -- 172.21.15.165:0/316215529 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v73) v1 ==== 160+0+0 (unknown 1386380281 0 0) 0x7f068c056bb0 con 0x7f06941046d0 2024-09-10T18:49:21.036 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 10 librados: watch_flush enter 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 10 librados: watch_flush exit 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 20 client.24379.objecter shutdown clearing up homeless session... 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 10 client.24379.objecter successfully canceled tick 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 >> v1:172.21.15.83:6800/2630800463 conn(0x7f0680071600 legacy=0x7f0680073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 >> v1:172.21.15.165:6789/0 conn(0x7f06941046d0 legacy=0x7f069411c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 shutdown_connections 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.028+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 >> 172.21.15.165:0/316215529 conn(0x7f06940f7d70 msgr2=0x7f06940fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.032+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 shutdown_connections 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.032+0000 7f069c1cc640 1 -- 172.21.15.165:0/316215529 wait complete. 2024-09-10T18:49:21.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:21.032+0000 7f069c1cc640 1 librados: shutdown 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:19.615747+0000 mon.a (mon.0) 660 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: cluster 2024-09-10T18:49:19.616600+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.078650+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.086560+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.087990+0000 mon.a (mon.0) 664 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.089147+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: cephadm 2024-09-10T18:49:20.097086+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.174235+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.165:0/316215529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[18629]: audit 2024-09-10T18:49:20.178299+0000 mon.a (mon.0) 666 : audit [INF] from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:19.615747+0000 mon.a (mon.0) 660 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: cluster 2024-09-10T18:49:19.616600+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.078650+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.086560+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.087990+0000 mon.a (mon.0) 664 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.089147+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: cephadm 2024-09-10T18:49:20.097086+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.174235+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.165:0/316215529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:21 smithi083 bash[32381]: audit 2024-09-10T18:49:20.178299+0000 mon.a (mon.0) 666 : audit [INF] from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:19.615747+0000 mon.a (mon.0) 660 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: cluster 2024-09-10T18:49:19.616600+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.078650+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.086560+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.087990+0000 mon.a (mon.0) 664 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.089147+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:21.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: cephadm 2024-09-10T18:49:20.097086+0000 mgr.y (mgr.14150) 357 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:21.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.174235+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.165:0/316215529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:21 smithi165 bash[22591]: audit 2024-09-10T18:49:20.178299+0000 mon.a (mon.0) 666 : audit [INF] from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T18:49:21.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-10T18:49:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: cluster 2024-09-10T18:49:20.501233+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v352: 65 pgs: 4 creating+peering, 5 creating+activating, 48 unknown, 8 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T18:49:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: audit 2024-09-10T18:49:21.026855+0000 mon.a (mon.0) 667 : audit [INF] from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:49:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: cluster 2024-09-10T18:49:21.026949+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:49:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: audit 2024-09-10T18:49:21.029304+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.83:0/2284258763' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: audit 2024-09-10T18:49:21.029789+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: audit 2024-09-10T18:49:21.036769+0000 mon.a (mon.0) 669 : audit [INF] from='client.24404 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:22 smithi165 bash[22591]: audit 2024-09-10T18:49:21.037093+0000 mon.a (mon.0) 670 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: cluster 2024-09-10T18:49:20.501233+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v352: 65 pgs: 4 creating+peering, 5 creating+activating, 48 unknown, 8 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: audit 2024-09-10T18:49:21.026855+0000 mon.a (mon.0) 667 : audit [INF] from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: cluster 2024-09-10T18:49:21.026949+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: audit 2024-09-10T18:49:21.029304+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.83:0/2284258763' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: audit 2024-09-10T18:49:21.029789+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: audit 2024-09-10T18:49:21.036769+0000 mon.a (mon.0) 669 : audit [INF] from='client.24404 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[18629]: audit 2024-09-10T18:49:21.037093+0000 mon.a (mon.0) 670 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: cluster 2024-09-10T18:49:20.501233+0000 mgr.y (mgr.14150) 358 : cluster [DBG] pgmap v352: 65 pgs: 4 creating+peering, 5 creating+activating, 48 unknown, 8 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: audit 2024-09-10T18:49:21.026855+0000 mon.a (mon.0) 667 : audit [INF] from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: cluster 2024-09-10T18:49:21.026949+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: audit 2024-09-10T18:49:21.029304+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.83:0/2284258763' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: audit 2024-09-10T18:49:21.029789+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.83:0/2025706139' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: audit 2024-09-10T18:49:21.036769+0000 mon.a (mon.0) 669 : audit [INF] from='client.24404 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:22 smithi083 bash[32381]: audit 2024-09-10T18:49:21.037093+0000 mon.a (mon.0) 670 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T18:49:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: audit 2024-09-10T18:49:22.031890+0000 mon.a (mon.0) 671 : audit [INF] from='client.24404 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: audit 2024-09-10T18:49:22.032030+0000 mon.a (mon.0) 672 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: cluster 2024-09-10T18:49:22.032244+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:49:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: cluster 2024-09-10T18:49:23.031232+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:49:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: audit 2024-09-10T18:49:23.033574+0000 mon.a (mon.0) 675 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: audit 2024-09-10T18:49:23.034052+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:23 smithi165 bash[22591]: audit 2024-09-10T18:49:23.037833+0000 mon.a (mon.0) 676 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: audit 2024-09-10T18:49:22.031890+0000 mon.a (mon.0) 671 : audit [INF] from='client.24404 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: audit 2024-09-10T18:49:22.032030+0000 mon.a (mon.0) 672 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: cluster 2024-09-10T18:49:22.032244+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: cluster 2024-09-10T18:49:23.031232+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: audit 2024-09-10T18:49:23.033574+0000 mon.a (mon.0) 675 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: audit 2024-09-10T18:49:23.034052+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[18629]: audit 2024-09-10T18:49:23.037833+0000 mon.a (mon.0) 676 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: audit 2024-09-10T18:49:22.031890+0000 mon.a (mon.0) 671 : audit [INF] from='client.24404 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: audit 2024-09-10T18:49:22.032030+0000 mon.a (mon.0) 672 : audit [INF] from='client.24374 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: cluster 2024-09-10T18:49:22.032244+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: cluster 2024-09-10T18:49:23.031232+0000 mon.a (mon.0) 674 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: audit 2024-09-10T18:49:23.033574+0000 mon.a (mon.0) 675 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: audit 2024-09-10T18:49:23.034052+0000 mon.b (mon.1) 26 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:23.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:23 smithi083 bash[32381]: audit 2024-09-10T18:49:23.037833+0000 mon.a (mon.0) 676 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T18:49:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: cluster 2024-09-10T18:49:22.501980+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v355: 100 pgs: 9 creating+peering, 5 creating+activating, 41 unknown, 45 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: cluster 2024-09-10T18:49:23.042402+0000 mon.a (mon.0) 677 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: audit 2024-09-10T18:49:24.037170+0000 mon.a (mon.0) 678 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: audit 2024-09-10T18:49:24.037376+0000 mon.a (mon.0) 679 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: cluster 2024-09-10T18:49:24.037553+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: audit 2024-09-10T18:49:24.039745+0000 mon.a (mon.0) 681 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: audit 2024-09-10T18:49:24.040106+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:24 smithi165 bash[22591]: audit 2024-09-10T18:49:24.044950+0000 mon.a (mon.0) 682 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: cluster 2024-09-10T18:49:22.501980+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v355: 100 pgs: 9 creating+peering, 5 creating+activating, 41 unknown, 45 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: cluster 2024-09-10T18:49:23.042402+0000 mon.a (mon.0) 677 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: audit 2024-09-10T18:49:24.037170+0000 mon.a (mon.0) 678 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: audit 2024-09-10T18:49:24.037376+0000 mon.a (mon.0) 679 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: cluster 2024-09-10T18:49:24.037553+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:49:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: audit 2024-09-10T18:49:24.039745+0000 mon.a (mon.0) 681 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: audit 2024-09-10T18:49:24.040106+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[18629]: audit 2024-09-10T18:49:24.044950+0000 mon.a (mon.0) 682 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: cluster 2024-09-10T18:49:22.501980+0000 mgr.y (mgr.14150) 359 : cluster [DBG] pgmap v355: 100 pgs: 9 creating+peering, 5 creating+activating, 41 unknown, 45 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: cluster 2024-09-10T18:49:23.042402+0000 mon.a (mon.0) 677 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: audit 2024-09-10T18:49:24.037170+0000 mon.a (mon.0) 678 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: audit 2024-09-10T18:49:24.037376+0000 mon.a (mon.0) 679 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: cluster 2024-09-10T18:49:24.037553+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: audit 2024-09-10T18:49:24.039745+0000 mon.a (mon.0) 681 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: audit 2024-09-10T18:49:24.040106+0000 mon.b (mon.1) 27 : audit [INF] from='client.? v1:172.21.15.83:0/461710557' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:24.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:24 smithi083 bash[32381]: audit 2024-09-10T18:49:24.044950+0000 mon.a (mon.0) 682 : audit [INF] from='client.24409 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: cluster 2024-09-10T18:49:24.502799+0000 mgr.y (mgr.14150) 360 : cluster [DBG] pgmap v358: 132 pgs: 10 creating+peering, 40 unknown, 82 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.036542+0000 mon.a (mon.0) 683 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.036673+0000 mon.a (mon.0) 684 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: cluster 2024-09-10T18:49:25.036894+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.261283+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.272224+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.281474+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.345829+0000 mon.a (mon.0) 689 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.348443+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:26.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.349337+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[32381]: audit 2024-09-10T18:49:25.662052+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: cluster 2024-09-10T18:49:24.502799+0000 mgr.y (mgr.14150) 360 : cluster [DBG] pgmap v358: 132 pgs: 10 creating+peering, 40 unknown, 82 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.036542+0000 mon.a (mon.0) 683 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.036673+0000 mon.a (mon.0) 684 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: cluster 2024-09-10T18:49:25.036894+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.261283+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.272224+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.281474+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.345829+0000 mon.a (mon.0) 689 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.348443+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:26.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.349337+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:26.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:26 smithi083 bash[18629]: audit 2024-09-10T18:49:25.662052+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: cluster 2024-09-10T18:49:24.502799+0000 mgr.y (mgr.14150) 360 : cluster [DBG] pgmap v358: 132 pgs: 10 creating+peering, 40 unknown, 82 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T18:49:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.036542+0000 mon.a (mon.0) 683 : audit [INF] from='client.? v1:172.21.15.83:0/703687015' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.036673+0000 mon.a (mon.0) 684 : audit [INF] from='client.24409 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T18:49:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: cluster 2024-09-10T18:49:25.036894+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-09-10T18:49:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.261283+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.272224+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.281474+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.345829+0000 mon.a (mon.0) 689 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.348443+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.349337+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:26 smithi165 bash[22591]: audit 2024-09-10T18:49:25.662052+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:26.439 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:27 smithi165 bash[22591]: cephadm 2024-09-10T18:49:25.353625+0000 mgr.y (mgr.14150) 361 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:27.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:27 smithi083 bash[18629]: cephadm 2024-09-10T18:49:25.353625+0000 mgr.y (mgr.14150) 361 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:27 smithi083 bash[32381]: cephadm 2024-09-10T18:49:25.353625+0000 mgr.y (mgr.14150) 361 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-09-10T18:49:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:28 smithi165 bash[22591]: cluster 2024-09-10T18:49:26.504190+0000 mgr.y (mgr.14150) 362 : cluster [DBG] pgmap v360: 132 pgs: 10 creating+peering, 12 unknown, 110 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 92 op/s 2024-09-10T18:49:28.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:28 smithi083 bash[18629]: cluster 2024-09-10T18:49:26.504190+0000 mgr.y (mgr.14150) 362 : cluster [DBG] pgmap v360: 132 pgs: 10 creating+peering, 12 unknown, 110 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 92 op/s 2024-09-10T18:49:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:28 smithi083 bash[32381]: cluster 2024-09-10T18:49:26.504190+0000 mgr.y (mgr.14150) 362 : cluster [DBG] pgmap v360: 132 pgs: 10 creating+peering, 12 unknown, 110 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 92 op/s 2024-09-10T18:49:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:30 smithi165 bash[22591]: cluster 2024-09-10T18:49:28.505300+0000 mgr.y (mgr.14150) 363 : cluster [DBG] pgmap v361: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-09-10T18:49:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:30 smithi165 bash[22591]: audit 2024-09-10T18:49:29.822406+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.165:0/3568408522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:30 smithi165 bash[22591]: audit 2024-09-10T18:49:29.826353+0000 mon.a (mon.0) 693 : audit [INF] from='client.24466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:30.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[18629]: cluster 2024-09-10T18:49:28.505300+0000 mgr.y (mgr.14150) 363 : cluster [DBG] pgmap v361: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-09-10T18:49:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[18629]: audit 2024-09-10T18:49:29.822406+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.165:0/3568408522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[18629]: audit 2024-09-10T18:49:29.826353+0000 mon.a (mon.0) 693 : audit [INF] from='client.24466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[32381]: cluster 2024-09-10T18:49:28.505300+0000 mgr.y (mgr.14150) 363 : cluster [DBG] pgmap v361: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 5.0 KiB/s wr, 151 op/s 2024-09-10T18:49:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[32381]: audit 2024-09-10T18:49:29.822406+0000 mon.b (mon.1) 28 : audit [INF] from='client.? v1:172.21.15.165:0/3568408522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:30 smithi083 bash[32381]: audit 2024-09-10T18:49:29.826353+0000 mon.a (mon.0) 693 : audit [INF] from='client.24466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T18:49:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:31 smithi165 bash[22591]: audit 2024-09-10T18:49:30.065710+0000 mon.a (mon.0) 694 : audit [INF] from='client.24466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:49:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:31 smithi165 bash[22591]: cluster 2024-09-10T18:49:30.065832+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:49:31.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:31 smithi083 bash[18629]: audit 2024-09-10T18:49:30.065710+0000 mon.a (mon.0) 694 : audit [INF] from='client.24466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:49:31.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:31 smithi083 bash[18629]: cluster 2024-09-10T18:49:30.065832+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:49:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:31 smithi083 bash[32381]: audit 2024-09-10T18:49:30.065710+0000 mon.a (mon.0) 694 : audit [INF] from='client.24466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T18:49:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:31 smithi083 bash[32381]: cluster 2024-09-10T18:49:30.065832+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-09-10T18:49:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:32 smithi165 bash[22591]: cluster 2024-09-10T18:49:30.506052+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 163 op/s 2024-09-10T18:49:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:32 smithi165 bash[22591]: cluster 2024-09-10T18:49:31.061113+0000 mon.a (mon.0) 696 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:49:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:32 smithi165 bash[22591]: cluster 2024-09-10T18:49:31.061166+0000 mon.a (mon.0) 697 : cluster [INF] Cluster is now healthy 2024-09-10T18:49:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:32 smithi165 bash[22591]: cluster 2024-09-10T18:49:31.075336+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:49:32.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[18629]: cluster 2024-09-10T18:49:30.506052+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 163 op/s 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[18629]: cluster 2024-09-10T18:49:31.061113+0000 mon.a (mon.0) 696 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[18629]: cluster 2024-09-10T18:49:31.061166+0000 mon.a (mon.0) 697 : cluster [INF] Cluster is now healthy 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[18629]: cluster 2024-09-10T18:49:31.075336+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[32381]: cluster 2024-09-10T18:49:30.506052+0000 mgr.y (mgr.14150) 364 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 5.4 KiB/s wr, 163 op/s 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[32381]: cluster 2024-09-10T18:49:31.061113+0000 mon.a (mon.0) 696 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[32381]: cluster 2024-09-10T18:49:31.061166+0000 mon.a (mon.0) 697 : cluster [INF] Cluster is now healthy 2024-09-10T18:49:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:32 smithi083 bash[32381]: cluster 2024-09-10T18:49:31.075336+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-09-10T18:49:32.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.165 --placement '1;smithi165=iscsi.a' 2024-09-10T18:49:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:33 smithi165 bash[22591]: cluster 2024-09-10T18:49:32.078792+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:49:33.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:33 smithi083 bash[18629]: cluster 2024-09-10T18:49:32.078792+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:49:33.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:33 smithi083 bash[32381]: cluster 2024-09-10T18:49:32.078792+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-09-10T18:49:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:34 smithi165 bash[22591]: cluster 2024-09-10T18:49:32.506757+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.3 KiB/s wr, 123 op/s 2024-09-10T18:49:34.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:34 smithi083 bash[18629]: cluster 2024-09-10T18:49:32.506757+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.3 KiB/s wr, 123 op/s 2024-09-10T18:49:34.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:34 smithi083 bash[32381]: cluster 2024-09-10T18:49:32.506757+0000 mgr.y (mgr.14150) 365 : cluster [DBG] pgmap v366: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.3 KiB/s wr, 123 op/s 2024-09-10T18:49:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:35 smithi165 bash[22591]: cluster 2024-09-10T18:49:34.719870+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:35 smithi165 bash[22591]: cluster 2024-09-10T18:49:34.719987+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:35.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:35 smithi083 bash[18629]: cluster 2024-09-10T18:49:34.719870+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:35.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:35 smithi083 bash[18629]: cluster 2024-09-10T18:49:34.719987+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:35 smithi083 bash[32381]: cluster 2024-09-10T18:49:34.719870+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:35 smithi083 bash[32381]: cluster 2024-09-10T18:49:34.719987+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:36 smithi165 bash[22591]: cluster 2024-09-10T18:49:34.507802+0000 mgr.y (mgr.14150) 366 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1023 B/s wr, 41 op/s 2024-09-10T18:49:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:36 smithi165 bash[22591]: cluster 2024-09-10T18:49:35.102184+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:49:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:36 smithi165 bash[22591]: cluster 2024-09-10T18:49:35.630860+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:49:36.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[18629]: cluster 2024-09-10T18:49:34.507802+0000 mgr.y (mgr.14150) 366 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1023 B/s wr, 41 op/s 2024-09-10T18:49:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[18629]: cluster 2024-09-10T18:49:35.102184+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:49:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[18629]: cluster 2024-09-10T18:49:35.630860+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:49:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[32381]: cluster 2024-09-10T18:49:34.507802+0000 mgr.y (mgr.14150) 366 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 1023 B/s wr, 41 op/s 2024-09-10T18:49:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[32381]: cluster 2024-09-10T18:49:35.102184+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-09-10T18:49:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:36 smithi083 bash[32381]: cluster 2024-09-10T18:49:35.630860+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-09-10T18:49:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:37 smithi165 bash[22591]: cluster 2024-09-10T18:49:36.686946+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:37 smithi165 bash[22591]: cluster 2024-09-10T18:49:36.687043+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:37.532 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:37.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:37 smithi083 bash[18629]: cluster 2024-09-10T18:49:36.686946+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:37.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:37 smithi083 bash[18629]: cluster 2024-09-10T18:49:36.687043+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:37.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:37 smithi083 bash[32381]: cluster 2024-09-10T18:49:36.686946+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:37.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:37 smithi083 bash[32381]: cluster 2024-09-10T18:49:36.687043+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:38 smithi165 bash[22591]: cluster 2024-09-10T18:49:36.508680+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 376 B/s wr, 0 op/s 2024-09-10T18:49:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:38 smithi165 bash[22591]: cluster 2024-09-10T18:49:37.113191+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:49:38.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:38 smithi083 bash[18629]: cluster 2024-09-10T18:49:36.508680+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 376 B/s wr, 0 op/s 2024-09-10T18:49:38.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:38 smithi083 bash[18629]: cluster 2024-09-10T18:49:37.113191+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:49:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:38 smithi083 bash[32381]: cluster 2024-09-10T18:49:36.508680+0000 mgr.y (mgr.14150) 367 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 376 B/s wr, 0 op/s 2024-09-10T18:49:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:38 smithi083 bash[32381]: cluster 2024-09-10T18:49:37.113191+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-09-10T18:49:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:40 smithi083 bash[18629]: cluster 2024-09-10T18:49:38.509734+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:40 smithi083 bash[32381]: cluster 2024-09-10T18:49:38.509734+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:40 smithi165 bash[22591]: cluster 2024-09-10T18:49:38.509734+0000 mgr.y (mgr.14150) 368 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:40.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 -- 172.21.15.165:0/1063152504 >> v1:172.21.15.165:6789/0 conn(0x7fe138100af0 legacy=0x7fe138102ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:40.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 -- 172.21.15.165:0/1063152504 shutdown_connections 2024-09-10T18:49:40.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 -- 172.21.15.165:0/1063152504 >> 172.21.15.165:0/1063152504 conn(0x7fe1380f7db0 msgr2=0x7fe1380fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:40.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 -- 172.21.15.165:0/1063152504 shutdown_connections 2024-09-10T18:49:40.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 -- 172.21.15.165:0/1063152504 wait complete. 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 librados: starting msgr at 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.803+0000 7fe13da27640 1 librados: starting objecter 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 Processor -- start 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- start start 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 librados: setting wanted keys 2024-09-10T18:49:40.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 librados: calling monclient init 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe13810d320 con 0x7fe138104710 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe138118b20 con 0x7fe138100af0 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe138119cc0 con 0x7fe1380fc5f0 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe1367fc640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe138100af0 0x7fe138112cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:38200/0 (socket says 172.21.15.165:38200) 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe1367fc640 1 -- 172.21.15.165:0/1704553161 learned_addr learned my addr 172.21.15.165:0/1704553161 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter ms_handle_connect 0x7fe138100af0 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter ms_handle_connect 0x7fe1380fc5f0 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481640576 0 0) 0x7fe138118b20 con 0x7fe138100af0 2024-09-10T18:49:40.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1240034e0 con 0x7fe138100af0 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter ms_handle_connect 0x7fe138104710 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572498495 0 0) 0x7fe138119cc0 con 0x7fe1380fc5f0 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe138118b20 con 0x7fe1380fc5f0 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898824707 0 0) 0x7fe13810d320 con 0x7fe138104710 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe138119cc0 con 0x7fe138104710 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 183422691 0 0) 0x7fe1240034e0 con 0x7fe138100af0 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe13810d320 con 0x7fe138100af0 2024-09-10T18:49:40.816 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe120001430 con 0x7fe138100af0 2024-09-10T18:49:40.817 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588413829 0 0) 0x7fe13810d320 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 >> v1:172.21.15.83:6790/0 conn(0x7fe1380fc5f0 legacy=0x7fe13810c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 >> v1:172.21.15.83:6789/0 conn(0x7fe138104710 legacy=0x7fe1381163f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13811aea0 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe138119e50 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 10 client.24472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13811a320 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe13da27640 1 librados: init done 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe120002400 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.807+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe120003920 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118005040 con 0x7fe138100af0 2024-09-10T18:49:40.818 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 15) v1 ==== 90745+0+0 (unknown 1414673027 0 0) 0x7fe120002570 con 0x7fe138100af0 2024-09-10T18:49:40.819 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(84..84 src has 14..84) v4 ==== 6054+0+0 (unknown 2150527058 0 0) 0x7fe12008a230 con 0x7fe138100af0 2024-09-10T18:49:40.819 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 10 client.24472.objecter ms_dispatch 0x7fe138000b90 osd_map(84..84 src has 14..84) v4 2024-09-10T18:49:40.819 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 3 client.24472.objecter handle_osd_map got epochs [84,84] > 0 2024-09-10T18:49:40.820 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 3 client.24472.objecter handle_osd_map decoding full epoch 84 2024-09-10T18:49:40.820 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.811+0000 7fe12f7fe640 20 client.24472.objecter dump_active .. 0 homeless 2024-09-10T18:49:40.821 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.815+0000 7fe12f7fe640 10 client.24472.objecter ms_handle_connect 0x7fe1240718a0 2024-09-10T18:49:40.826 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:40.819+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe120056f90 con 0x7fe138100af0 2024-09-10T18:49:41.011 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.003+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fe118000c90 con 0x7fe1240718a0 2024-09-10T18:49:41.027 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.019+0000 7fe12f7fe640 1 -- 172.21.15.165:0/1704553161 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (unknown 0 0 803663096) 0x7fe118000c90 con 0x7fe1240718a0 2024-09-10T18:49:41.027 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled iscsi.datapool update... 2024-09-10T18:49:41.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 10 librados: watch_flush enter 2024-09-10T18:49:41.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 10 librados: watch_flush exit 2024-09-10T18:49:41.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 20 client.24472.objecter shutdown clearing up homeless session... 2024-09-10T18:49:41.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 10 client.24472.objecter successfully canceled tick 2024-09-10T18:49:41.029 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 >> v1:172.21.15.83:6800/2630800463 conn(0x7fe1240718a0 legacy=0x7fe124073d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 >> v1:172.21.15.165:6789/0 conn(0x7fe138100af0 legacy=0x7fe138112cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 shutdown_connections 2024-09-10T18:49:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 >> 172.21.15.165:0/1704553161 conn(0x7fe1380f7db0 msgr2=0x7fe138106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 shutdown_connections 2024-09-10T18:49:41.030 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 -- 172.21.15.165:0/1704553161 wait complete. 2024-09-10T18:49:41.031 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:41.023+0000 7fe13da27640 1 librados: shutdown 2024-09-10T18:49:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.390852+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-09-10T18:49:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.391409+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.391772+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.392135+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.632763+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.632897+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.633012+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:40.633104+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: cluster 2024-09-10T18:49:40.633169+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:41.026137+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:41 smithi165 bash[22591]: audit 2024-09-10T18:49:41.027698+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.390852+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.391409+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.391772+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.392135+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.632763+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.632897+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.633012+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:40.633104+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: cluster 2024-09-10T18:49:40.633169+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:49:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:41.026137+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[32381]: audit 2024-09-10T18:49:41.027698+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.390852+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.391409+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.391772+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.392135+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.632763+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.632897+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.633012+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:40.633104+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T18:49:41.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: cluster 2024-09-10T18:49:40.633169+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-09-10T18:49:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:41.026137+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:41 smithi083 bash[18629]: audit 2024-09-10T18:49:41.027698+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:41.652 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-10T18:49:41.652 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:49:41.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T18:49:41.660 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:49:41.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T18:49:41.671 DEBUG:teuthology.orchestra.run.smithi165:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-10T18:49:41.673 INFO:tasks.cephadm:Adding prometheus.a on smithi165 2024-09-10T18:49:41.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi165=a' 2024-09-10T18:49:41.790 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: cluster 2024-09-10T18:49:40.510669+0000 mgr.y (mgr.14150) 369 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.012966+0000 mgr.y (mgr.14150) 370 : audit [DBG] from='client.24472 v1:172.21.15.165:0/1704553161' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: cephadm 2024-09-10T18:49:41.018642+0000 mgr.y (mgr.14150) 371 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.320335+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.327750+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: cluster 2024-09-10T18:49:41.630472+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.703068+0000 mon.a (mon.0) 717 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.704252+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.712407+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.715831+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:49:42.406 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.722361+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:49:42.407 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:42 smithi165 bash[22591]: audit 2024-09-10T18:49:41.734491+0000 mon.a (mon.0) 722 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: cluster 2024-09-10T18:49:40.510669+0000 mgr.y (mgr.14150) 369 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.012966+0000 mgr.y (mgr.14150) 370 : audit [DBG] from='client.24472 v1:172.21.15.165:0/1704553161' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: cephadm 2024-09-10T18:49:41.018642+0000 mgr.y (mgr.14150) 371 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.320335+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.327750+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: cluster 2024-09-10T18:49:41.630472+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.703068+0000 mon.a (mon.0) 717 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.704252+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.712407+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.715831+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.722361+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[18629]: audit 2024-09-10T18:49:41.734491+0000 mon.a (mon.0) 722 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: cluster 2024-09-10T18:49:40.510669+0000 mgr.y (mgr.14150) 369 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.012966+0000 mgr.y (mgr.14150) 370 : audit [DBG] from='client.24472 v1:172.21.15.165:0/1704553161' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: cephadm 2024-09-10T18:49:41.018642+0000 mgr.y (mgr.14150) 371 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.320335+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.327750+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: cluster 2024-09-10T18:49:41.630472+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-09-10T18:49:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.703068+0000 mon.a (mon.0) 717 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.704252+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.712407+0000 mon.a (mon.0) 719 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.715831+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T18:49:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.722361+0000 mon.a (mon.0) 721 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T18:49:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:42 smithi083 bash[32381]: audit 2024-09-10T18:49:41.734491+0000 mon.a (mon.0) 722 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:43.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:43 smithi165 bash[22591]: cephadm 2024-09-10T18:49:41.735769+0000 mgr.y (mgr.14150) 372 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi165 2024-09-10T18:49:43.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:43 smithi083 bash[18629]: cephadm 2024-09-10T18:49:41.735769+0000 mgr.y (mgr.14150) 372 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi165 2024-09-10T18:49:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:43 smithi083 bash[32381]: cephadm 2024-09-10T18:49:41.735769+0000 mgr.y (mgr.14150) 372 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi165 2024-09-10T18:49:44.287 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:44 smithi165 bash[22591]: cluster 2024-09-10T18:49:42.511408+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v376: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:44.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:44 smithi083 bash[18629]: cluster 2024-09-10T18:49:42.511408+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v376: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:44 smithi083 bash[32381]: cluster 2024-09-10T18:49:42.511408+0000 mgr.y (mgr.14150) 373 : cluster [DBG] pgmap v376: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:49:44.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:44 smithi165 systemd[1]: Started Ceph iscsi.iscsi.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:49:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: cluster 2024-09-10T18:49:44.512499+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v377: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 B/s, 0 objects/s recovering 2024-09-10T18:49:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:44.552196+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:44.561172+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:44.569734+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: cephadm 2024-09-10T18:49:44.570710+0000 mgr.y (mgr.14150) 375 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:44.589323+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: cluster 2024-09-10T18:49:44.606600+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: cluster 2024-09-10T18:49:44.606699+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:44.631053+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:45 smithi165 bash[22591]: audit 2024-09-10T18:49:45.464483+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: cluster 2024-09-10T18:49:44.512499+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v377: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 B/s, 0 objects/s recovering 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:44.552196+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:44.561172+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:44.569734+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: cephadm 2024-09-10T18:49:44.570710+0000 mgr.y (mgr.14150) 375 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:44.589323+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: cluster 2024-09-10T18:49:44.606600+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:46.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: cluster 2024-09-10T18:49:44.606699+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:44.631053+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[32381]: audit 2024-09-10T18:49:45.464483+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: cluster 2024-09-10T18:49:44.512499+0000 mgr.y (mgr.14150) 374 : cluster [DBG] pgmap v377: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 B/s, 0 objects/s recovering 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:44.552196+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:44.561172+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:44.569734+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: cephadm 2024-09-10T18:49:44.570710+0000 mgr.y (mgr.14150) 375 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:44.589323+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: cluster 2024-09-10T18:49:44.606600+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: cluster 2024-09-10T18:49:44.606699+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:44.631053+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:49:46.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:45 smithi083 bash[18629]: audit 2024-09-10T18:49:45.464483+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:46.425 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:46 smithi165 bash[22591]: cluster 2024-09-10T18:49:45.047714+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:46 smithi165 bash[22591]: cluster 2024-09-10T18:49:45.047872+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:46 smithi165 bash[22591]: cluster 2024-09-10T18:49:45.604812+0000 mon.a (mon.0) 729 : cluster [DBG] mgrmap e16: y(active, since 10m), standbys: x 2024-09-10T18:49:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:46 smithi165 bash[22591]: cluster 2024-09-10T18:49:45.612127+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:49:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:46 smithi165 bash[22591]: cluster 2024-09-10T18:49:45.631875+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:49:47.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[18629]: cluster 2024-09-10T18:49:45.047714+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[18629]: cluster 2024-09-10T18:49:45.047872+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[18629]: cluster 2024-09-10T18:49:45.604812+0000 mon.a (mon.0) 729 : cluster [DBG] mgrmap e16: y(active, since 10m), standbys: x 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[18629]: cluster 2024-09-10T18:49:45.612127+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[18629]: cluster 2024-09-10T18:49:45.631875+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[32381]: cluster 2024-09-10T18:49:45.047714+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[32381]: cluster 2024-09-10T18:49:45.047872+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[32381]: cluster 2024-09-10T18:49:45.604812+0000 mon.a (mon.0) 729 : cluster [DBG] mgrmap e16: y(active, since 10m), standbys: x 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[32381]: cluster 2024-09-10T18:49:45.612127+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-09-10T18:49:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:46 smithi083 bash[32381]: cluster 2024-09-10T18:49:45.631875+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-09-10T18:49:47.265 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54512]: log file path now is '/var/log/tcmu-runner.log' 2024-09-10T18:49:47.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug Started the configuration object watcher 2024-09-10T18:49:47.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug Checking for config object changes every 1s 2024-09-10T18:49:47.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug Processing osd blocklist entries for this node 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug Reading the configuration object to update local LIO configuration 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug Configuration does not have an entry for this host(smithi165.front.sepia.ceph.com) - nothing to define to LIO 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: * Environment: production 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: Use a production WSGI server instead. 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: * Debug mode: off 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug * Running on all addresses. 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: * Running on all addresses. 2024-09-10T18:49:47.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T18:49:47.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: debug * Running on http://[fe80::ec4:7aff:fe88:7f63]:5000/ (Press CTRL+C to quit) 2024-09-10T18:49:47.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[54535]: * Running on http://[fe80::ec4:7aff:fe88:7f63]:5000/ (Press CTRL+C to quit) 2024-09-10T18:49:47.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[22591]: cluster 2024-09-10T18:49:46.513292+0000 mgr.y (mgr.14150) 376 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 B/s, 0 objects/s recovering 2024-09-10T18:49:47.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:47 smithi165 bash[22591]: audit 2024-09-10T18:49:47.628032+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.165:0/2372369551' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:49:48.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:47 smithi083 bash[18629]: cluster 2024-09-10T18:49:46.513292+0000 mgr.y (mgr.14150) 376 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 B/s, 0 objects/s recovering 2024-09-10T18:49:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:47 smithi083 bash[18629]: audit 2024-09-10T18:49:47.628032+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.165:0/2372369551' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:49:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:47 smithi083 bash[32381]: cluster 2024-09-10T18:49:46.513292+0000 mgr.y (mgr.14150) 376 : cluster [DBG] pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 B/s, 0 objects/s recovering 2024-09-10T18:49:48.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:47 smithi083 bash[32381]: audit 2024-09-10T18:49:47.628032+0000 mon.b (mon.1) 29 : audit [DBG] from='client.? v1:172.21.15.165:0/2372369551' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T18:49:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- 172.21.15.165:0/1876584747 >> v1:172.21.15.165:6789/0 conn(0x7f16740fdbf0 legacy=0x7f16740fe000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- 172.21.15.165:0/1876584747 shutdown_connections 2024-09-10T18:49:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- 172.21.15.165:0/1876584747 >> 172.21.15.165:0/1876584747 conn(0x7f16740f63d0 msgr2=0x7f16740f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- 172.21.15.165:0/1876584747 shutdown_connections 2024-09-10T18:49:49.808 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- 172.21.15.165:0/1876584747 wait complete. 2024-09-10T18:49:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 librados: starting msgr at 2024-09-10T18:49:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 librados: starting objecter 2024-09-10T18:49:49.809 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 Processor -- start 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 -- start start 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 librados: setting wanted keys 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.799+0000 7f1678c5e640 1 librados: calling monclient init 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f167410d390 con 0x7f16740fdbf0 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1674118b40 con 0x7f16741046f0 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1674119ce0 con 0x7f1674100bf0 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1672d9d640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f16741046f0 0x7f1674116410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:59228/0 (socket says 172.21.15.165:59228) 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1672d9d640 1 -- 172.21.15.165:0/1824476301 learned_addr learned my addr 172.21.15.165:0/1824476301 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter ms_handle_connect 0x7f16741046f0 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter ms_handle_connect 0x7f1674100bf0 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:49.810 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter ms_handle_connect 0x7f16740fdbf0 2024-09-10T18:49:49.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:49:49.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674816273 0 0) 0x7f1674118b40 con 0x7f16741046f0 2024-09-10T18:49:49.811 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165c0054b0 con 0x7f16741046f0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435881856 0 0) 0x7f1674119ce0 con 0x7f1674100bf0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1674118b40 con 0x7f1674100bf0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909087190 0 0) 0x7f167410d390 con 0x7f16740fdbf0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1674119ce0 con 0x7f16740fdbf0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326485738 0 0) 0x7f165c0054b0 con 0x7f16741046f0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f167410d390 con 0x7f16741046f0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f166c001900 con 0x7f16741046f0 2024-09-10T18:49:49.812 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2405234965 0 0) 0x7f167410d390 con 0x7f16741046f0 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 >> v1:172.21.15.83:6790/0 conn(0x7f1674100bf0 legacy=0x7f1674112ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 >> v1:172.21.15.83:6789/0 conn(0x7f16740fdbf0 legacy=0x7f167410c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167411aec0 con 0x7f16741046f0 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1674119ed0 con 0x7f16741046f0 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 10 client.24508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f167411a3a0 con 0x7f16741046f0 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 librados: init done 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f166c001c90 con 0x7f16741046f0 2024-09-10T18:49:49.813 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f166c005b60 con 0x7f16741046f0 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.803+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1654005040 con 0x7f16741046f0 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 16) v1 ==== 90796+0+0 (unknown 425156702 0 0) 0x7f166c0029a0 con 0x7f16741046f0 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(87..87 src has 14..87) v4 ==== 6026+0+0 (unknown 2801437762 0 0) 0x7f166c089e20 con 0x7f16741046f0 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 10 client.24508.objecter ms_dispatch 0x7f1674000b90 osd_map(87..87 src has 14..87) v4 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 3 client.24508.objecter handle_osd_map got epochs [87,87] > 0 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 3 client.24508.objecter handle_osd_map decoding full epoch 87 2024-09-10T18:49:49.814 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 20 client.24508.objecter dump_active .. 0 homeless 2024-09-10T18:49:49.815 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.807+0000 7f1662ffd640 10 client.24508.objecter ms_handle_connect 0x7f165c0714f0 2024-09-10T18:49:49.820 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:49.811+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f166c056af0 con 0x7f16741046f0 2024-09-10T18:49:50.021 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.015+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1654000bc0 con 0x7f165c0714f0 2024-09-10T18:49:50.033 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1662ffd640 1 -- 172.21.15.165:0/1824476301 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7f1654000bc0 con 0x7f165c0714f0 2024-09-10T18:49:50.034 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled prometheus update... 2024-09-10T18:49:50.036 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 10 librados: watch_flush enter 2024-09-10T18:49:50.036 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 10 librados: watch_flush exit 2024-09-10T18:49:50.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 20 client.24508.objecter shutdown clearing up homeless session... 2024-09-10T18:49:50.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 10 client.24508.objecter successfully canceled tick 2024-09-10T18:49:50.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 >> v1:172.21.15.83:6800/2630800463 conn(0x7f165c0714f0 legacy=0x7f165c0739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:50.037 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.027+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 >> v1:172.21.15.165:6789/0 conn(0x7f16741046f0 legacy=0x7f1674116410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:49:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.031+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 shutdown_connections 2024-09-10T18:49:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.031+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 >> 172.21.15.165:0/1824476301 conn(0x7f16740f63d0 msgr2=0x7f1674106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:49:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.031+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 shutdown_connections 2024-09-10T18:49:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.031+0000 7f1678c5e640 1 -- 172.21.15.165:0/1824476301 wait complete. 2024-09-10T18:49:50.038 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:49:50.031+0000 7f1678c5e640 1 librados: shutdown 2024-09-10T18:49:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:49 smithi083 bash[18629]: cluster 2024-09-10T18:49:48.514347+0000 mgr.y (mgr.14150) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 B/s, 0 objects/s recovering 2024-09-10T18:49:50.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:49 smithi083 bash[32381]: cluster 2024-09-10T18:49:48.514347+0000 mgr.y (mgr.14150) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 B/s, 0 objects/s recovering 2024-09-10T18:49:50.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:49 smithi165 bash[22591]: cluster 2024-09-10T18:49:48.514347+0000 mgr.y (mgr.14150) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 B/s, 0 objects/s recovering 2024-09-10T18:49:50.808 DEBUG:teuthology.orchestra.run.smithi165:prometheus.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-10T18:49:50.811 INFO:tasks.cephadm:Adding node-exporter.a on smithi083 2024-09-10T18:49:50.811 INFO:tasks.cephadm:Adding node-exporter.b on smithi165 2024-09-10T18:49:50.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi083=a;smithi165=b' 2024-09-10T18:49:50.985 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:49:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.023297+0000 mgr.y (mgr.14150) 378 : audit [DBG] from='client.24508 v1:172.21.15.165:0/1824476301' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: cephadm 2024-09-10T18:49:50.025200+0000 mgr.y (mgr.14150) 379 : cephadm [INF] Saving service prometheus spec with placement smithi165=a;count:1 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.032629+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.099152+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.107848+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.109190+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.110500+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: audit 2024-09-10T18:49:50.118300+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: cephadm 2024-09-10T18:49:50.137876+0000 mgr.y (mgr.14150) 380 : cephadm [INF] Deploying daemon prometheus.a on smithi165 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: cluster 2024-09-10T18:49:50.378563+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:51.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:51 smithi165 bash[22591]: cluster 2024-09-10T18:49:50.378664+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.023297+0000 mgr.y (mgr.14150) 378 : audit [DBG] from='client.24508 v1:172.21.15.165:0/1824476301' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: cephadm 2024-09-10T18:49:50.025200+0000 mgr.y (mgr.14150) 379 : cephadm [INF] Saving service prometheus spec with placement smithi165=a;count:1 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.032629+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.099152+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.107848+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.109190+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.110500+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: audit 2024-09-10T18:49:50.118300+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: cephadm 2024-09-10T18:49:50.137876+0000 mgr.y (mgr.14150) 380 : cephadm [INF] Deploying daemon prometheus.a on smithi165 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: cluster 2024-09-10T18:49:50.378563+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[18629]: cluster 2024-09-10T18:49:50.378664+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.023297+0000 mgr.y (mgr.14150) 378 : audit [DBG] from='client.24508 v1:172.21.15.165:0/1824476301' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: cephadm 2024-09-10T18:49:50.025200+0000 mgr.y (mgr.14150) 379 : cephadm [INF] Saving service prometheus spec with placement smithi165=a;count:1 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.032629+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.099152+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.107848+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.109190+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.110500+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: audit 2024-09-10T18:49:50.118300+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: cephadm 2024-09-10T18:49:50.137876+0000 mgr.y (mgr.14150) 380 : cephadm [INF] Deploying daemon prometheus.a on smithi165 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: cluster 2024-09-10T18:49:50.378563+0000 osd.6 (osd.6) 7 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:51 smithi083 bash[32381]: cluster 2024-09-10T18:49:50.378664+0000 osd.6 (osd.6) 8 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:52.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:52 smithi083 bash[18629]: cluster 2024-09-10T18:49:50.515165+0000 mgr.y (mgr.14150) 381 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 B/s, 0 objects/s recovering 2024-09-10T18:49:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:52 smithi083 bash[18629]: cluster 2024-09-10T18:49:51.133542+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:49:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:52 smithi083 bash[32381]: cluster 2024-09-10T18:49:50.515165+0000 mgr.y (mgr.14150) 381 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 B/s, 0 objects/s recovering 2024-09-10T18:49:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:52 smithi083 bash[32381]: cluster 2024-09-10T18:49:51.133542+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:49:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:52 smithi165 bash[22591]: cluster 2024-09-10T18:49:50.515165+0000 mgr.y (mgr.14150) 381 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 B/s, 0 objects/s recovering 2024-09-10T18:49:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:52 smithi165 bash[22591]: cluster 2024-09-10T18:49:51.133542+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-09-10T18:49:54.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:54 smithi083 bash[18629]: cluster 2024-09-10T18:49:52.516032+0000 mgr.y (mgr.14150) 382 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 148 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:54 smithi083 bash[32381]: cluster 2024-09-10T18:49:52.516032+0000 mgr.y (mgr.14150) 382 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 148 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:54 smithi165 bash[22591]: cluster 2024-09-10T18:49:52.516032+0000 mgr.y (mgr.14150) 382 : cluster [DBG] pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 148 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:55.619 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:49:56.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:56 smithi083 bash[18629]: cluster 2024-09-10T18:49:54.517094+0000 mgr.y (mgr.14150) 383 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 460 B/s rd, 115 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:56 smithi083 bash[18629]: cluster 2024-09-10T18:49:55.646102+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:49:56.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:56 smithi083 bash[32381]: cluster 2024-09-10T18:49:54.517094+0000 mgr.y (mgr.14150) 383 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 460 B/s rd, 115 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:56.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:56 smithi083 bash[32381]: cluster 2024-09-10T18:49:55.646102+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:49:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:56 smithi165 bash[22591]: cluster 2024-09-10T18:49:54.517094+0000 mgr.y (mgr.14150) 383 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 460 B/s rd, 115 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:49:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:56 smithi165 bash[22591]: cluster 2024-09-10T18:49:55.646102+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-09-10T18:49:57.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:49:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:49:58.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:58 smithi083 bash[18629]: cluster 2024-09-10T18:49:56.517890+0000 mgr.y (mgr.14150) 384 : cluster [DBG] pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:49:58.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:58 smithi083 bash[32381]: cluster 2024-09-10T18:49:56.517890+0000 mgr.y (mgr.14150) 384 : cluster [DBG] pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:49:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:58 smithi165 bash[22591]: cluster 2024-09-10T18:49:56.517890+0000 mgr.y (mgr.14150) 384 : cluster [DBG] pgmap v387: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:49:59.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[32381]: audit 2024-09-10T18:49:57.354037+0000 mgr.y (mgr.14150) 385 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:49:59.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[32381]: cluster 2024-09-10T18:49:58.459304+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[32381]: cluster 2024-09-10T18:49:58.459420+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[18629]: audit 2024-09-10T18:49:57.354037+0000 mgr.y (mgr.14150) 385 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:49:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[18629]: cluster 2024-09-10T18:49:58.459304+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:49:59 smithi083 bash[18629]: cluster 2024-09-10T18:49:58.459420+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:49:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:59 smithi165 bash[22591]: audit 2024-09-10T18:49:57.354037+0000 mgr.y (mgr.14150) 385 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:49:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:59 smithi165 bash[22591]: cluster 2024-09-10T18:49:58.459304+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:49:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:49:59 smithi165 bash[22591]: cluster 2024-09-10T18:49:58.459420+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:00.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[18629]: cluster 2024-09-10T18:49:58.518973+0000 mgr.y (mgr.14150) 386 : cluster [DBG] pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:50:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[18629]: cluster 2024-09-10T18:49:59.161342+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:50:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[18629]: cluster 2024-09-10T18:50:00.000170+0000 mon.a (mon.0) 741 : cluster [INF] overall HEALTH_OK 2024-09-10T18:50:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[32381]: cluster 2024-09-10T18:49:58.518973+0000 mgr.y (mgr.14150) 386 : cluster [DBG] pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:50:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[32381]: cluster 2024-09-10T18:49:59.161342+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:50:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:00 smithi083 bash[32381]: cluster 2024-09-10T18:50:00.000170+0000 mon.a (mon.0) 741 : cluster [INF] overall HEALTH_OK 2024-09-10T18:50:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:00 smithi165 bash[22591]: cluster 2024-09-10T18:49:58.518973+0000 mgr.y (mgr.14150) 386 : cluster [DBG] pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 127 B/s wr, 2 op/s 2024-09-10T18:50:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:00 smithi165 bash[22591]: cluster 2024-09-10T18:49:59.161342+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-09-10T18:50:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:00 smithi165 bash[22591]: cluster 2024-09-10T18:50:00.000170+0000 mon.a (mon.0) 741 : cluster [INF] overall HEALTH_OK 2024-09-10T18:50:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:01 smithi083 bash[18629]: cluster 2024-09-10T18:50:00.519779+0000 mgr.y (mgr.14150) 387 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:02.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:01 smithi083 bash[18629]: cluster 2024-09-10T18:50:00.639006+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:50:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:01 smithi083 bash[32381]: cluster 2024-09-10T18:50:00.519779+0000 mgr.y (mgr.14150) 387 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:01 smithi083 bash[32381]: cluster 2024-09-10T18:50:00.639006+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:50:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:01 smithi165 bash[22591]: cluster 2024-09-10T18:50:00.519779+0000 mgr.y (mgr.14150) 387 : cluster [DBG] pgmap v390: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:01 smithi165 bash[22591]: cluster 2024-09-10T18:50:00.639006+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-09-10T18:50:03.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/414694802 >> v1:172.21.15.165:6789/0 conn(0x7f095c0fb4a0 legacy=0x7f095c0fb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:03.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/414694802 shutdown_connections 2024-09-10T18:50:03.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/414694802 >> 172.21.15.165:0/414694802 conn(0x7f095c0f6c20 msgr2=0x7f095c0f9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:03.350 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/414694802 shutdown_connections 2024-09-10T18:50:03.351 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/414694802 wait complete. 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 librados: starting msgr at 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 librados: starting objecter 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 Processor -- start 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- start start 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 librados: setting wanted keys 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 librados: calling monclient init 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c10bfc0 con 0x7f095c1035c0 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c1177c0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.353 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c118960 con 0x7f095c0ff9a0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0960b94640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f095c0fb4a0 0x7f095c10b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:54708/0 (socket says 172.21.15.165:54708) 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0960b94640 1 -- 172.21.15.165:0/3602119523 learned_addr learned my addr 172.21.15.165:0/3602119523 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter ms_handle_connect 0x7f095c0fb4a0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564722569 0 0) 0x7f095c1177c0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003720 con 0x7f095c0fb4a0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter ms_handle_connect 0x7f095c1035c0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter ms_handle_connect 0x7f095c0ff9a0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126023550 0 0) 0x7f095c10bfc0 con 0x7f095c1035c0 2024-09-10T18:50:03.354 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c1177c0 con 0x7f095c1035c0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141704435 0 0) 0x7f093c003720 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095c10bfc0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f094c001430 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216290731 0 0) 0x7f095c118960 con 0x7f095c0ff9a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003720 con 0x7f095c0ff9a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1881709642 0 0) 0x7f095c10bfc0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 >> v1:172.21.15.83:6790/0 conn(0x7f095c0ff9a0 legacy=0x7f095c111960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 >> v1:172.21.15.83:6789/0 conn(0x7f095c1035c0 legacy=0x7f095c115090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c119b40 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c116810 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 10 client.24514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c116ce0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.355 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.343+0000 7f0962df8640 1 librados: init done 2024-09-10T18:50:03.356 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f094c002470 con 0x7f095c0fb4a0 2024-09-10T18:50:03.356 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f094c003790 con 0x7f095c0fb4a0 2024-09-10T18:50:03.356 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0940005040 con 0x7f095c0fb4a0 2024-09-10T18:50:03.356 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 16) v1 ==== 90796+0+0 (unknown 425156702 0 0) 0x7f094c0038f0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(91..91 src has 14..91) v4 ==== 6026+0+0 (unknown 1859357816 0 0) 0x7f094c08a2f0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 10 client.24514.objecter ms_dispatch 0x7f095c000b90 osd_map(91..91 src has 14..91) v4 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 3 client.24514.objecter handle_osd_map got epochs [91,91] > 0 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 3 client.24514.objecter handle_osd_map decoding full epoch 91 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 20 client.24514.objecter dump_active .. 0 homeless 2024-09-10T18:50:03.357 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.347+0000 7f09597fa640 10 client.24514.objecter ms_handle_connect 0x7f093c0713d0 2024-09-10T18:50:03.363 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.355+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f094c056fc0 con 0x7f095c0fb4a0 2024-09-10T18:50:03.548 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.539+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 --> v1:172.21.15.83:6800/2630800463 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi165=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f0940000bc0 con 0x7f093c0713d0 2024-09-10T18:50:03.561 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.551+0000 7f09597fa640 1 -- 172.21.15.165:0/3602119523 <== mgr.14150 v1:172.21.15.83:6800/2630800463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7f0940000bc0 con 0x7f093c0713d0 2024-09-10T18:50:03.561 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled node-exporter update... 2024-09-10T18:50:03.565 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 10 librados: watch_flush enter 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 10 librados: watch_flush exit 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 20 client.24514.objecter shutdown clearing up homeless session... 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 10 client.24514.objecter successfully canceled tick 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 >> v1:172.21.15.83:6800/2630800463 conn(0x7f093c0713d0 legacy=0x7f093c073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 >> v1:172.21.15.165:6789/0 conn(0x7f095c0fb4a0 legacy=0x7f095c10b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:03.566 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 shutdown_connections 2024-09-10T18:50:03.567 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 >> 172.21.15.165:0/3602119523 conn(0x7f095c0f6c20 msgr2=0x7f095c103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:03.567 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 shutdown_connections 2024-09-10T18:50:03.567 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 -- 172.21.15.165:0/3602119523 wait complete. 2024-09-10T18:50:03.567 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:03.559+0000 7f0962df8640 1 librados: shutdown 2024-09-10T18:50:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:03 smithi165 bash[22591]: cluster 2024-09-10T18:50:02.520747+0000 mgr.y (mgr.14150) 388 : cluster [DBG] pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:50:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:03 smithi165 bash[22591]: audit 2024-09-10T18:50:03.560173+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:03 smithi083 bash[18629]: cluster 2024-09-10T18:50:02.520747+0000 mgr.y (mgr.14150) 388 : cluster [DBG] pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:50:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:03 smithi083 bash[18629]: audit 2024-09-10T18:50:03.560173+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:03 smithi083 bash[32381]: cluster 2024-09-10T18:50:02.520747+0000 mgr.y (mgr.14150) 388 : cluster [DBG] pgmap v392: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-09-10T18:50:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:03 smithi083 bash[32381]: audit 2024-09-10T18:50:03.560173+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:04.715 DEBUG:teuthology.orchestra.run.smithi083:node-exporter.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-10T18:50:04.717 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.b> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-10T18:50:04.719 INFO:tasks.cephadm:Adding alertmanager.a on smithi083 2024-09-10T18:50:04.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi083=a' 2024-09-10T18:50:04.905 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:50:04.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:04 smithi165 bash[22591]: audit 2024-09-10T18:50:03.549929+0000 mgr.y (mgr.14150) 389 : audit [DBG] from='client.24514 v1:172.21.15.165:0/3602119523' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:04.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:04 smithi165 bash[22591]: cephadm 2024-09-10T18:50:03.553019+0000 mgr.y (mgr.14150) 390 : cephadm [INF] Saving service node-exporter spec with placement smithi083=a;smithi165=b;count:2 2024-09-10T18:50:05.010 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:50:05.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:04 smithi083 bash[18629]: audit 2024-09-10T18:50:03.549929+0000 mgr.y (mgr.14150) 389 : audit [DBG] from='client.24514 v1:172.21.15.165:0/3602119523' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:05.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:04 smithi083 bash[18629]: cephadm 2024-09-10T18:50:03.553019+0000 mgr.y (mgr.14150) 390 : cephadm [INF] Saving service node-exporter spec with placement smithi083=a;smithi165=b;count:2 2024-09-10T18:50:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:04 smithi083 bash[32381]: audit 2024-09-10T18:50:03.549929+0000 mgr.y (mgr.14150) 389 : audit [DBG] from='client.24514 v1:172.21.15.165:0/3602119523' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi083=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:04 smithi083 bash[32381]: cephadm 2024-09-10T18:50:03.553019+0000 mgr.y (mgr.14150) 390 : cephadm [INF] Saving service node-exporter spec with placement smithi083=a;smithi165=b;count:2 2024-09-10T18:50:06.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:05 smithi083 bash[18629]: cluster 2024-09-10T18:50:04.521828+0000 mgr.y (mgr.14150) 391 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:05 smithi083 bash[32381]: cluster 2024-09-10T18:50:04.521828+0000 mgr.y (mgr.14150) 391 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:05 smithi165 bash[22591]: cluster 2024-09-10T18:50:04.521828+0000 mgr.y (mgr.14150) 391 : cluster [DBG] pgmap v393: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:50:07.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:07 smithi083 bash[18629]: cluster 2024-09-10T18:50:06.522926+0000 mgr.y (mgr.14150) 392 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:07 smithi083 bash[32381]: cluster 2024-09-10T18:50:06.522926+0000 mgr.y (mgr.14150) 392 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:07 smithi165 bash[22591]: cluster 2024-09-10T18:50:06.522926+0000 mgr.y (mgr.14150) 392 : cluster [DBG] pgmap v394: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:09.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[23586]: ignoring --setuser ceph since I am not root 2024-09-10T18:50:09.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[23586]: ignoring --setgroup ceph since I am not root 2024-09-10T18:50:09.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[23586]: debug 2024-09-10T18:50:09.311+0000 7f7978523ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:50:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[22591]: audit 2024-09-10T18:50:07.363774+0000 mgr.y (mgr.14150) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[22591]: audit 2024-09-10T18:50:08.110637+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[22591]: audit 2024-09-10T18:50:08.120186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[22591]: audit 2024-09-10T18:50:08.126075+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[22591]: audit 2024-09-10T18:50:08.129857+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:50:09.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18896]: ignoring --setuser ceph since I am not root 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18896]: ignoring --setgroup ceph since I am not root 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18896]: debug 2024-09-10T18:50:09.294+0000 7f52269c5ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[32381]: audit 2024-09-10T18:50:07.363774+0000 mgr.y (mgr.14150) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[32381]: audit 2024-09-10T18:50:08.110637+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[32381]: audit 2024-09-10T18:50:08.120186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[32381]: audit 2024-09-10T18:50:08.126075+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[32381]: audit 2024-09-10T18:50:08.129857+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:50:09.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18629]: audit 2024-09-10T18:50:07.363774+0000 mgr.y (mgr.14150) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:09.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18629]: audit 2024-09-10T18:50:08.110637+0000 mon.a (mon.0) 744 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18629]: audit 2024-09-10T18:50:08.120186+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18629]: audit 2024-09-10T18:50:08.126075+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' 2024-09-10T18:50:09.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:09 smithi083 bash[18629]: audit 2024-09-10T18:50:08.129857+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T18:50:09.863 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:50:09 smithi165 bash[55673]: ts=2024-09-10T18:50:09.398Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T18:50:10.290 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.030+0000 7f52269c5ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.098+0000 7f52269c5ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.166+0000 7f52269c5ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18629]: audit 2024-09-10T18:50:09.141754+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18629]: cluster 2024-09-10T18:50:09.141823+0000 mon.a (mon.0) 749 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[32381]: audit 2024-09-10T18:50:09.141754+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:50:10.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[32381]: cluster 2024-09-10T18:50:09.141823+0000 mon.a (mon.0) 749 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-10T18:50:10.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[22591]: audit 2024-09-10T18:50:09.141754+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14150 v1:172.21.15.83:0/3871068215' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T18:50:10.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[22591]: cluster 2024-09-10T18:50:09.141823+0000 mon.a (mon.0) 749 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-09-10T18:50:10.356 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.059+0000 7f7978523ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:50:10.356 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.127+0000 7f7978523ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:50:10.356 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.195+0000 7f7978523ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:50:10.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.347+0000 7f7978523ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:50:10.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.595+0000 7f7978523ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:50:10.630 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.318+0000 7f52269c5ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:50:10.631 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.562+0000 7f52269c5ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:50:11.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.626+0000 7f52269c5ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:50:11.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:10 smithi083 bash[18896]: debug 2024-09-10T18:50:10.762+0000 7f52269c5ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:50:11.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.659+0000 7f7978523ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:50:11.113 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:10 smithi165 bash[23586]: debug 2024-09-10T18:50:10.791+0000 7f7978523ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:50:11.568 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:50:11.693 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:11 smithi165 bash[23586]: debug 2024-09-10T18:50:11.411+0000 7f7978523ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:50:11.693 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:11 smithi165 bash[23586]: debug 2024-09-10T18:50:11.479+0000 7f7978523ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:50:11.693 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:11 smithi165 bash[23586]: debug 2024-09-10T18:50:11.555+0000 7f7978523ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:50:11.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:11 smithi083 bash[18896]: debug 2024-09-10T18:50:11.386+0000 7f52269c5ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:50:11.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:11 smithi083 bash[18896]: debug 2024-09-10T18:50:11.458+0000 7f52269c5ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:50:11.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:11 smithi083 bash[18896]: debug 2024-09-10T18:50:11.530+0000 7f52269c5ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:50:11.977 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:11 smithi165 bash[23586]: debug 2024-09-10T18:50:11.835+0000 7f7978523ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:50:12.261 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:11 smithi083 bash[18896]: debug 2024-09-10T18:50:11.810+0000 7f52269c5ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:50:12.262 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:11 smithi083 bash[18896]: debug 2024-09-10T18:50:11.942+0000 7f52269c5ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:50:12.289 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:11 smithi165 bash[23586]: debug 2024-09-10T18:50:11.967+0000 7f7978523ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:50:12.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:12 smithi083 bash[18896]: debug 2024-09-10T18:50:12.254+0000 7f52269c5ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:50:12.540 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:12 smithi083 bash[18896]: debug 2024-09-10T18:50:12.470+0000 7f52269c5ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:50:12.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:12 smithi165 bash[23586]: debug 2024-09-10T18:50:12.279+0000 7f7978523ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:50:12.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:12 smithi165 bash[23586]: debug 2024-09-10T18:50:12.495+0000 7f7978523ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:50:13.063 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:12 smithi165 bash[23586]: debug 2024-09-10T18:50:12.923+0000 7f7978523ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:50:13.063 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:12 smithi165 bash[23586]: debug 2024-09-10T18:50:12.987+0000 7f7978523ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:50:13.172 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:12 smithi083 bash[18896]: debug 2024-09-10T18:50:12.882+0000 7f52269c5ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:50:13.172 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:12 smithi083 bash[18896]: debug 2024-09-10T18:50:12.946+0000 7f52269c5ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:50:13.172 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.010+0000 7f52269c5ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:50:13.172 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.102+0000 7f52269c5ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:50:13.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.055+0000 7f7978523ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:50:13.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.147+0000 7f7978523ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:50:13.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.215+0000 7f7978523ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:50:13.363 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.287+0000 7f7978523ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:50:13.466 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.166+0000 7f52269c5ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:50:13.466 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.238+0000 7f52269c5ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:50:13.466 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.386+0000 7f52269c5ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:50:13.786 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.431+0000 7f7978523ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:50:13.786 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:13 smithi165 bash[23586]: debug 2024-09-10T18:50:13.507+0000 7f7978523ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:50:13.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:13 smithi083 bash[18896]: debug 2024-09-10T18:50:13.458+0000 7f52269c5ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.421412+0000 mon.a (mon.0) 750 : cluster [INF] Active manager daemon y restarted 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.422114+0000 mon.a (mon.0) 751 : cluster [INF] Activating manager daemon y 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.434615+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.435541+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0136447s), standbys: x 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.442743+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.442904+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:50:14.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.443028+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.444443+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.444612+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.444893+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.445139+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.445362+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.445656+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.445971+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.446181+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.446407+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.446626+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.446941+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:50:14.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.447135+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.421412+0000 mon.a (mon.0) 750 : cluster [INF] Active manager daemon y restarted 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.422114+0000 mon.a (mon.0) 751 : cluster [INF] Activating manager daemon y 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.434615+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.435541+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0136447s), standbys: x 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.442743+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.442904+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:50:14.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.443028+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.444443+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.444612+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.444893+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.445139+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.445362+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.445656+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.445971+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.446181+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.446407+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.446626+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:50:14.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.446941+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.447135+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.447662+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.458566+0000 mon.a (mon.0) 754 : cluster [INF] Manager daemon y is now available 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.468618+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.468897+0000 mon.a (mon.0) 755 : cluster [DBG] Standby manager daemon x restarted 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: cluster 2024-09-10T18:50:14.469073+0000 mon.a (mon.0) 756 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:50:14.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18629]: audit 2024-09-10T18:50:14.469225+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.447662+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.458566+0000 mon.a (mon.0) 754 : cluster [INF] Manager daemon y is now available 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.468618+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.468897+0000 mon.a (mon.0) 755 : cluster [DBG] Standby manager daemon x restarted 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: cluster 2024-09-10T18:50:14.469073+0000 mon.a (mon.0) 756 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[32381]: audit 2024-09-10T18:50:14.469225+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: debug 2024-09-10T18:50:14.394+0000 7f52269c5ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: [10/Sep/2024:18:50:14] ENGINE Bus STARTING 2024-09-10T18:50:14.714 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: CherryPy Checker: 2024-09-10T18:50:14.715 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: The Application mounted at '' has an empty config. 2024-09-10T18:50:14.715 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: [10/Sep/2024:18:50:14] ENGINE Serving on http://:::9283 2024-09-10T18:50:14.715 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:14 smithi083 bash[18896]: [10/Sep/2024:18:50:14] ENGINE Bus STARTED 2024-09-10T18:50:14.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: debug 2024-09-10T18:50:14.459+0000 7f7978523ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:50:14.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: [10/Sep/2024:18:50:14] ENGINE Bus STARTING 2024-09-10T18:50:14.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: CherryPy Checker: 2024-09-10T18:50:14.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: The Application mounted at '' has an empty config. 2024-09-10T18:50:14.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: [10/Sep/2024:18:50:14] ENGINE Serving on http://:::9283 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[23586]: [10/Sep/2024:18:50:14] ENGINE Bus STARTED 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.421412+0000 mon.a (mon.0) 750 : cluster [INF] Active manager daemon y restarted 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.422114+0000 mon.a (mon.0) 751 : cluster [INF] Activating manager daemon y 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.434615+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.435541+0000 mon.a (mon.0) 753 : cluster [DBG] mgrmap e18: y(active, starting, since 0.0136447s), standbys: x 2024-09-10T18:50:14.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.442743+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.442904+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.443028+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.444443+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.444612+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.444893+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.445139+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.445362+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.445656+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.445971+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:50:14.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.446181+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.446407+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.446626+0000 mon.c (mon.2) 29 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.446941+0000 mon.c (mon.2) 30 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.447135+0000 mon.c (mon.2) 31 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.447662+0000 mon.c (mon.2) 32 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.458566+0000 mon.a (mon.0) 754 : cluster [INF] Manager daemon y is now available 2024-09-10T18:50:14.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.468618+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:50:14.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.468897+0000 mon.a (mon.0) 755 : cluster [DBG] Standby manager daemon x restarted 2024-09-10T18:50:14.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: cluster 2024-09-10T18:50:14.469073+0000 mon.a (mon.0) 756 : cluster [DBG] Standby manager daemon x started 2024-09-10T18:50:14.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:14 smithi165 bash[22591]: audit 2024-09-10T18:50:14.469225+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- 172.21.15.165:0/978131141 >> v1:172.21.15.83:6790/0 conn(0x7ff7b80fa9d0 legacy=0x7ff7b80fae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- 172.21.15.165:0/978131141 shutdown_connections 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- 172.21.15.165:0/978131141 >> 172.21.15.165:0/978131141 conn(0x7ff7b80f63f0 msgr2=0x7ff7b80f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- 172.21.15.165:0/978131141 shutdown_connections 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- 172.21.15.165:0/978131141 wait complete. 2024-09-10T18:50:14.911 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 librados: starting msgr at 2024-09-10T18:50:14.912 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 librados: starting objecter 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 Processor -- start 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- start start 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 librados: setting wanted keys 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 librados: calling monclient init 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b81199c0 con 0x7ff7b8100b70 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b811abc0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7bd472640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7b811bdc0 con 0x7ff7b81046d0 2024-09-10T18:50:14.913 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7b6ffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff7b80fa9d0 0x7ff7b80fd290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:35998/0 (socket says 172.21.15.165:35998) 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff7b6ffd640 1 -- 172.21.15.165:0/3457922504 learned_addr learned my addr 172.21.15.165:0/3457922504 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7b8100b70 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7b80fa9d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7b81046d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846384282 0 0) 0x7ff7b81199c0 con 0x7ff7b8100b70 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7a00034e0 con 0x7ff7b8100b70 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345879595 0 0) 0x7ff7b811abc0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b81199c0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800887854 0 0) 0x7ff7b811bdc0 con 0x7ff7b81046d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b811abc0 con 0x7ff7b81046d0 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.903+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050639199 0 0) 0x7ff7a00034e0 con 0x7ff7b8100b70 2024-09-10T18:50:14.914 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7b811bdc0 con 0x7ff7b8100b70 2024-09-10T18:50:14.915 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff7ac001840 con 0x7ff7b8100b70 2024-09-10T18:50:14.915 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758985059 0 0) 0x7ff7b811abc0 con 0x7ff7b81046d0 2024-09-10T18:50:14.915 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7a00034e0 con 0x7ff7b81046d0 2024-09-10T18:50:14.915 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2873390955 0 0) 0x7ff7b81199c0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.915 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7b811abc0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff7b0001b40 con 0x7ff7b81046d0 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff7a80015d0 con 0x7ff7b80fa9d0 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744690391 0 0) 0x7ff7b811bdc0 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 >> v1:172.21.15.83:6790/0 conn(0x7ff7b81046d0 legacy=0x7ff7b81180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 >> v1:172.21.15.165:6789/0 conn(0x7ff7b80fa9d0 legacy=0x7ff7b80fd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b811cfc0 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b811adb0 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff7bd472640 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b811b280 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff7bd472640 1 librados: init done 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff7ac0021c0 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff7ac003140 con 0x7ff7b8100b70 2024-09-10T18:50:14.916 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a4005040 con 0x7ff7b8100b70 2024-09-10T18:50:14.917 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.907+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 18) v1 ==== 90462+0+0 (unknown 4270087168 0 0) 0x7ff7ac001d60 con 0x7ff7b8100b70 2024-09-10T18:50:14.917 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.911+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(92..92 src has 14..92) v4 ==== 6284+0+0 (unknown 1435856268 0 0) 0x7ff7ac086eb0 con 0x7ff7b8100b70 2024-09-10T18:50:14.917 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.911+0000 7ff79f7fe640 10 client.14625.objecter ms_dispatch 0x7ff7b8000b90 osd_map(92..92 src has 14..92) v4 2024-09-10T18:50:14.917 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.911+0000 7ff79f7fe640 3 client.14625.objecter handle_osd_map got epochs [92,92] > 0 2024-09-10T18:50:14.918 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.911+0000 7ff79f7fe640 3 client.14625.objecter handle_osd_map decoding full epoch 92 2024-09-10T18:50:14.918 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.911+0000 7ff79f7fe640 20 client.14625.objecter dump_active .. 0 homeless 2024-09-10T18:50:14.926 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:14.919+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff7ac0507b0 con 0x7ff7b8100b70 2024-09-10T18:50:15.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.435+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mgrmap(e 19) v1 ==== 90554+0+0 (unknown 2337580828 0 0) 0x7ff7ac04fad0 con 0x7ff7b8100b70 2024-09-10T18:50:15.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.435+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi083=a", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7a007baf0 con 0x7ff7a0077ff0 2024-09-10T18:50:15.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.435+0000 7ff79f7fe640 10 client.14625.objecter ms_handle_connect 0x7ff7a0077ff0 2024-09-10T18:50:15.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff79f7fe640 1 -- 172.21.15.165:0/3457922504 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7ff7a007baf0 con 0x7ff7a0077ff0 2024-09-10T18:50:15.454 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled alertmanager update... 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 10 librados: watch_flush enter 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 10 librados: watch_flush exit 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 20 client.14625.objecter shutdown clearing up homeless session... 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 10 client.14625.objecter successfully canceled tick 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff7a0077ff0 legacy=0x7ff7a007a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.447+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 >> v1:172.21.15.83:6789/0 conn(0x7ff7b8100b70 legacy=0x7ff7b8114800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.451+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 shutdown_connections 2024-09-10T18:50:15.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.451+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 >> 172.21.15.165:0/3457922504 conn(0x7ff7b80f63f0 msgr2=0x7ff7b80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:15.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.451+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 shutdown_connections 2024-09-10T18:50:15.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.451+0000 7ff7bd472640 1 -- 172.21.15.165:0/3457922504 wait complete. 2024-09-10T18:50:15.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:15.451+0000 7ff7bd472640 1 librados: shutdown 2024-09-10T18:50:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.472533+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:50:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.481133+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:50:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.489628+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.531984+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.539427+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.547921+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.548227+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.579337+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:14.579848+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: cluster 2024-09-10T18:50:15.443545+0000 mon.a (mon.0) 760 : cluster [DBG] mgrmap e19: y(active, since 1.02164s), standbys: x 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[32381]: audit 2024-09-10T18:50:15.453959+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.472533+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:50:15.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.481133+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.489628+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.531984+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.539427+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.547921+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.548227+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.579337+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:14.579848+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: cluster 2024-09-10T18:50:15.443545+0000 mon.a (mon.0) 760 : cluster [DBG] mgrmap e19: y(active, since 1.02164s), standbys: x 2024-09-10T18:50:15.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:15 smithi083 bash[18629]: audit 2024-09-10T18:50:15.453959+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.472533+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:50:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.481133+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.? v1:172.21.15.165:0/3833922463' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:50:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.489628+0000 mon.c (mon.2) 33 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.531984+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.539427+0000 mon.c (mon.2) 34 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.547921+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.548227+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.579337+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:14.579848+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: cluster 2024-09-10T18:50:15.443545+0000 mon.a (mon.0) 760 : cluster [DBG] mgrmap e19: y(active, since 1.02164s), standbys: x 2024-09-10T18:50:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:15 smithi165 bash[22591]: audit 2024-09-10T18:50:15.453959+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:16.252 DEBUG:teuthology.orchestra.run.smithi083:alertmanager.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-10T18:50:16.255 INFO:tasks.cephadm:Adding grafana.a on smithi165 2024-09-10T18:50:16.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi165=a' 2024-09-10T18:50:16.299 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: debug 2024-09-10T18:50:16.038+0000 7f51c973b640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:50:16] ENGINE Error in HTTPServer.serve 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: Traceback (most recent call last): 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: self._connections.run(self.expiration_interval) 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: self._run(expiration_interval) 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:50:16.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: s = self.context.wrap_socket( 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: return self.sslsocket_class._create( 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: self.do_handshake() 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: self._sslobj.do_handshake() 2024-09-10T18:50:16.542 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18896]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cephadm 2024-09-10T18:50:15.450245+0000 mgr.y (mgr.24526) 2 : cephadm [INF] Saving service alertmanager spec with placement smithi083=a;count:1 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cluster 2024-09-10T18:50:15.450394+0000 mgr.y (mgr.24526) 3 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cluster 2024-09-10T18:50:15.644749+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cephadm 2024-09-10T18:50:15.914701+0000 mgr.y (mgr.24526) 4 : cephadm [INF] [10/Sep/2024:18:50:15] ENGINE Bus STARTING 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cephadm 2024-09-10T18:50:16.041967+0000 mgr.y (mgr.24526) 5 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Serving on https://172.21.15.83:7150 2024-09-10T18:50:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cephadm 2024-09-10T18:50:16.042140+0000 mgr.y (mgr.24526) 6 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Bus STARTED 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: cephadm 2024-09-10T18:50:16.045332+0000 mgr.y (mgr.24526) 7 : cephadm [ERR] [10/Sep/2024:18:50:16] ENGINE Error in HTTPServer.serve 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: Traceback (most recent call last): 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: self._connections.run(self.expiration_interval) 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:50:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: self._run(expiration_interval) 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:50:17.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: s = self.context.wrap_socket( 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: return self.sslsocket_class._create( 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: self.do_handshake() 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: self._sslobj.do_handshake() 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[18629]: audit 2024-09-10T18:50:16.049811+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cephadm 2024-09-10T18:50:15.450245+0000 mgr.y (mgr.24526) 2 : cephadm [INF] Saving service alertmanager spec with placement smithi083=a;count:1 2024-09-10T18:50:17.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cluster 2024-09-10T18:50:15.450394+0000 mgr.y (mgr.24526) 3 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cluster 2024-09-10T18:50:15.644749+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cephadm 2024-09-10T18:50:15.914701+0000 mgr.y (mgr.24526) 4 : cephadm [INF] [10/Sep/2024:18:50:15] ENGINE Bus STARTING 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cephadm 2024-09-10T18:50:16.041967+0000 mgr.y (mgr.24526) 5 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Serving on https://172.21.15.83:7150 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cephadm 2024-09-10T18:50:16.042140+0000 mgr.y (mgr.24526) 6 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Bus STARTED 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: cephadm 2024-09-10T18:50:16.045332+0000 mgr.y (mgr.24526) 7 : cephadm [ERR] [10/Sep/2024:18:50:16] ENGINE Error in HTTPServer.serve 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: Traceback (most recent call last): 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: self._connections.run(self.expiration_interval) 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: self._run(expiration_interval) 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:50:17.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: s = self.context.wrap_socket( 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: return self.sslsocket_class._create( 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: self.do_handshake() 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: self._sslobj.do_handshake() 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:50:17.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:16 smithi083 bash[32381]: audit 2024-09-10T18:50:16.049811+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cephadm 2024-09-10T18:50:15.450245+0000 mgr.y (mgr.24526) 2 : cephadm [INF] Saving service alertmanager spec with placement smithi083=a;count:1 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cluster 2024-09-10T18:50:15.450394+0000 mgr.y (mgr.24526) 3 : cluster [DBG] pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cluster 2024-09-10T18:50:15.644749+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cephadm 2024-09-10T18:50:15.914701+0000 mgr.y (mgr.24526) 4 : cephadm [INF] [10/Sep/2024:18:50:15] ENGINE Bus STARTING 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cephadm 2024-09-10T18:50:16.041967+0000 mgr.y (mgr.24526) 5 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Serving on https://172.21.15.83:7150 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cephadm 2024-09-10T18:50:16.042140+0000 mgr.y (mgr.24526) 6 : cephadm [INF] [10/Sep/2024:18:50:16] ENGINE Bus STARTED 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: cephadm 2024-09-10T18:50:16.045332+0000 mgr.y (mgr.24526) 7 : cephadm [ERR] [10/Sep/2024:18:50:16] ENGINE Error in HTTPServer.serve 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: Traceback (most recent call last): 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: self._connections.run(self.expiration_interval) 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: self._run(expiration_interval) 2024-09-10T18:50:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: s = self.context.wrap_socket( 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: return self.sslsocket_class._create( 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: self.do_handshake() 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: self._sslobj.do_handshake() 2024-09-10T18:50:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:50:17.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:16 smithi165 bash[22591]: audit 2024-09-10T18:50:16.049811+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:17.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:18.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:17 smithi083 bash[18629]: cluster 2024-09-10T18:50:16.446040+0000 mgr.y (mgr.24526) 8 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:17 smithi083 bash[18629]: cluster 2024-09-10T18:50:17.452842+0000 mon.a (mon.0) 764 : cluster [DBG] mgrmap e20: y(active, since 3s), standbys: x 2024-09-10T18:50:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:17 smithi083 bash[32381]: cluster 2024-09-10T18:50:16.446040+0000 mgr.y (mgr.24526) 8 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:17 smithi083 bash[32381]: cluster 2024-09-10T18:50:17.452842+0000 mon.a (mon.0) 764 : cluster [DBG] mgrmap e20: y(active, since 3s), standbys: x 2024-09-10T18:50:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:17 smithi165 bash[22591]: cluster 2024-09-10T18:50:16.446040+0000 mgr.y (mgr.24526) 8 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:17 smithi165 bash[22591]: cluster 2024-09-10T18:50:17.452842+0000 mon.a (mon.0) 764 : cluster [DBG] mgrmap e20: y(active, since 3s), standbys: x 2024-09-10T18:50:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[18629]: cluster 2024-09-10T18:50:17.342541+0000 osd.4 (osd.4) 9 : cluster [DBG] 2.f scrub starts 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[18629]: cluster 2024-09-10T18:50:17.347587+0000 osd.4 (osd.4) 10 : cluster [DBG] 2.f scrub ok 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[18629]: audit 2024-09-10T18:50:17.374963+0000 mgr.y (mgr.24526) 9 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[18629]: cluster 2024-09-10T18:50:17.657369+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[18629]: cluster 2024-09-10T18:50:17.660199+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[32381]: cluster 2024-09-10T18:50:17.342541+0000 osd.4 (osd.4) 9 : cluster [DBG] 2.f scrub starts 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[32381]: cluster 2024-09-10T18:50:17.347587+0000 osd.4 (osd.4) 10 : cluster [DBG] 2.f scrub ok 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[32381]: audit 2024-09-10T18:50:17.374963+0000 mgr.y (mgr.24526) 9 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[32381]: cluster 2024-09-10T18:50:17.657369+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:50:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:18 smithi083 bash[32381]: cluster 2024-09-10T18:50:17.660199+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:50:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:18 smithi165 bash[22591]: cluster 2024-09-10T18:50:17.342541+0000 osd.4 (osd.4) 9 : cluster [DBG] 2.f scrub starts 2024-09-10T18:50:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:18 smithi165 bash[22591]: cluster 2024-09-10T18:50:17.347587+0000 osd.4 (osd.4) 10 : cluster [DBG] 2.f scrub ok 2024-09-10T18:50:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:18 smithi165 bash[22591]: audit 2024-09-10T18:50:17.374963+0000 mgr.y (mgr.24526) 9 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:18 smithi165 bash[22591]: cluster 2024-09-10T18:50:17.657369+0000 osd.1 (osd.1) 17 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:50:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:18 smithi165 bash[22591]: cluster 2024-09-10T18:50:17.660199+0000 osd.1 (osd.1) 18 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:50:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[18629]: cluster 2024-09-10T18:50:18.446836+0000 mgr.y (mgr.24526) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[18629]: cluster 2024-09-10T18:50:18.480026+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[18629]: cluster 2024-09-10T18:50:18.483807+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[18629]: cluster 2024-09-10T18:50:18.663582+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[32381]: cluster 2024-09-10T18:50:18.446836+0000 mgr.y (mgr.24526) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[32381]: cluster 2024-09-10T18:50:18.480026+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[32381]: cluster 2024-09-10T18:50:18.483807+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:50:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:19 smithi083 bash[32381]: cluster 2024-09-10T18:50:18.663582+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:50:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:19 smithi165 bash[22591]: cluster 2024-09-10T18:50:18.446836+0000 mgr.y (mgr.24526) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:19 smithi165 bash[22591]: cluster 2024-09-10T18:50:18.480026+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:50:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:19 smithi165 bash[22591]: cluster 2024-09-10T18:50:18.483807+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:50:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:19 smithi165 bash[22591]: cluster 2024-09-10T18:50:18.663582+0000 mon.a (mon.0) 765 : cluster [DBG] mgrmap e21: y(active, since 4s), standbys: x 2024-09-10T18:50:20.782 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:50:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: cluster 2024-09-10T18:50:19.467376+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: cluster 2024-09-10T18:50:19.472687+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: audit 2024-09-10T18:50:20.352826+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: audit 2024-09-10T18:50:20.362835+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: audit 2024-09-10T18:50:20.625537+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[18629]: audit 2024-09-10T18:50:20.633962+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: cluster 2024-09-10T18:50:19.467376+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: cluster 2024-09-10T18:50:19.472687+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: audit 2024-09-10T18:50:20.352826+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: audit 2024-09-10T18:50:20.362835+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: audit 2024-09-10T18:50:20.625537+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:20 smithi083 bash[32381]: audit 2024-09-10T18:50:20.633962+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: cluster 2024-09-10T18:50:19.467376+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: cluster 2024-09-10T18:50:19.472687+0000 osd.5 (osd.5) 10 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: audit 2024-09-10T18:50:20.352826+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: audit 2024-09-10T18:50:20.362835+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: audit 2024-09-10T18:50:20.625537+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:20 smithi165 bash[22591]: audit 2024-09-10T18:50:20.633962+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:21.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:50:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:50:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: cluster 2024-09-10T18:50:20.364013+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:50:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: cluster 2024-09-10T18:50:20.372699+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:50:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: cluster 2024-09-10T18:50:20.448018+0000 mgr.y (mgr.24526) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.022621+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.031999+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.034438+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.034960+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.036735+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.037208+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.039430+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.039860+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.041262+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.041710+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.042694+0000 mgr.y (mgr.24526) 12 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3805M 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.049355+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.250713+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.264532+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.267286+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.267828+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.269345+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: cluster 2024-09-10T18:50:20.364013+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: cluster 2024-09-10T18:50:20.372699+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: cluster 2024-09-10T18:50:20.448018+0000 mgr.y (mgr.24526) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.022621+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.031999+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.034438+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.034960+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.036735+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.037208+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.039430+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.039860+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.041262+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.041710+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.042694+0000 mgr.y (mgr.24526) 12 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3805M 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.049355+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.250713+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.264532+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.267286+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.267828+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.269345+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.269779+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.271219+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.271640+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.273066+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.273523+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.286023+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.288162+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.289504+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.822179+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.295 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.829818+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.842917+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.849962+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[18629]: audit 2024-09-10T18:50:21.857603+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.269779+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.271219+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.271640+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.273066+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.273523+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.286023+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.288162+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.289504+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.822179+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.829818+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.842917+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.849962+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[32381]: audit 2024-09-10T18:50:21.857603+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: cluster 2024-09-10T18:50:20.364013+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: cluster 2024-09-10T18:50:20.372699+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: cluster 2024-09-10T18:50:20.448018+0000 mgr.y (mgr.24526) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.022621+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.031999+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.034438+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.034960+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.036735+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.037208+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.039430+0000 mon.c (mon.2) 39 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.039860+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.041262+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.041710+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.042694+0000 mgr.y (mgr.24526) 12 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3805M 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.049355+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.250713+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.264532+0000 mon.a (mon.0) 778 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.267286+0000 mon.c (mon.2) 41 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.267828+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.269345+0000 mon.c (mon.2) 42 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.269779+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.271219+0000 mon.c (mon.2) 43 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.271640+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.273066+0000 mon.c (mon.2) 44 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.273523+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.286023+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.288162+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.289504+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.822179+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.829818+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.842917+0000 mon.a (mon.0) 786 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.849962+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:22 smithi165 bash[22591]: audit 2024-09-10T18:50:21.857603+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:22.640 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:22 smithi083 systemd[1]: Started Ceph node-exporter.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:50:23.032 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:22 smithi083 bash[65618]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-09-10T18:50:23.290 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: v1.5.0: Pulling from prometheus/node-exporter 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.274702+0000 mgr.y (mgr.24526) 13 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3805M 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.291520+0000 mgr.y (mgr.24526) 14 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.291997+0000 mgr.y (mgr.24526) 15 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.433287+0000 mgr.y (mgr.24526) 16 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.437986+0000 mgr.y (mgr.24526) 17 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.560479+0000 mgr.y (mgr.24526) 18 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.566308+0000 mgr.y (mgr.24526) 19 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.686562+0000 mgr.y (mgr.24526) 20 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.691228+0000 mgr.y (mgr.24526) 21 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: cephadm 2024-09-10T18:50:21.864001+0000 mgr.y (mgr.24526) 22 : cephadm [INF] Deploying daemon node-exporter.a on smithi083 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: audit 2024-09-10T18:50:22.619913+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: audit 2024-09-10T18:50:22.628832+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[18629]: audit 2024-09-10T18:50:22.635670+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.274702+0000 mgr.y (mgr.24526) 13 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3805M 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.291520+0000 mgr.y (mgr.24526) 14 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.291997+0000 mgr.y (mgr.24526) 15 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.433287+0000 mgr.y (mgr.24526) 16 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.437986+0000 mgr.y (mgr.24526) 17 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.560479+0000 mgr.y (mgr.24526) 18 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.566308+0000 mgr.y (mgr.24526) 19 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.686562+0000 mgr.y (mgr.24526) 20 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.691228+0000 mgr.y (mgr.24526) 21 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: cephadm 2024-09-10T18:50:21.864001+0000 mgr.y (mgr.24526) 22 : cephadm [INF] Deploying daemon node-exporter.a on smithi083 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: audit 2024-09-10T18:50:22.619913+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: audit 2024-09-10T18:50:22.628832+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[32381]: audit 2024-09-10T18:50:22.635670+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.274702+0000 mgr.y (mgr.24526) 13 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3805M 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.291520+0000 mgr.y (mgr.24526) 14 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.conf 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.291997+0000 mgr.y (mgr.24526) 15 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.conf 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.433287+0000 mgr.y (mgr.24526) 16 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.437986+0000 mgr.y (mgr.24526) 17 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.560479+0000 mgr.y (mgr.24526) 18 : cephadm [INF] Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.566308+0000 mgr.y (mgr.24526) 19 : cephadm [INF] Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.686562+0000 mgr.y (mgr.24526) 20 : cephadm [INF] Updating smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.691228+0000 mgr.y (mgr.24526) 21 : cephadm [INF] Updating smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: cephadm 2024-09-10T18:50:21.864001+0000 mgr.y (mgr.24526) 22 : cephadm [INF] Deploying daemon node-exporter.a on smithi083 2024-09-10T18:50:23.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: audit 2024-09-10T18:50:22.619913+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: audit 2024-09-10T18:50:22.628832+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[22591]: audit 2024-09-10T18:50:22.635670+0000 mon.a (mon.0) 791 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:23.613 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:23 smithi165 systemd[1]: Started Ceph node-exporter.b for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:50:23.613 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:23 smithi165 bash[56588]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 22b70bddd3ac: Pulling fs layer 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 5c12815fee55: Pulling fs layer 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: c0e87333d380: Pulling fs layer 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 5c12815fee55: Download complete 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 22b70bddd3ac: Verifying Checksum 2024-09-10T18:50:23.717 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 22b70bddd3ac: Download complete 2024-09-10T18:50:24.033 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: c0e87333d380: Verifying Checksum 2024-09-10T18:50:24.033 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: c0e87333d380: Download complete 2024-09-10T18:50:24.033 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:23 smithi083 bash[65618]: 22b70bddd3ac: Pull complete 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.137294+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.f scrub starts 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.140598+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.f scrub ok 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.338396+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.343881+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.448781+0000 mgr.y (mgr.24526) 23 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cephadm 2024-09-10T18:50:22.638090+0000 mgr.y (mgr.24526) 24 : cephadm [INF] Deploying daemon node-exporter.b on smithi165 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.674812+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:50:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: cluster 2024-09-10T18:50:22.679704+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: audit 2024-09-10T18:50:23.406747+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: audit 2024-09-10T18:50:23.419357+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: audit 2024-09-10T18:50:23.427803+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: audit 2024-09-10T18:50:23.433660+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[18629]: audit 2024-09-10T18:50:23.445436+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.137294+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.f scrub starts 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.140598+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.f scrub ok 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.338396+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.343881+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:50:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.448781+0000 mgr.y (mgr.24526) 23 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cephadm 2024-09-10T18:50:22.638090+0000 mgr.y (mgr.24526) 24 : cephadm [INF] Deploying daemon node-exporter.b on smithi165 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.674812+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: cluster 2024-09-10T18:50:22.679704+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: audit 2024-09-10T18:50:23.406747+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: audit 2024-09-10T18:50:23.419357+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: audit 2024-09-10T18:50:23.427803+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: audit 2024-09-10T18:50:23.433660+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[32381]: audit 2024-09-10T18:50:23.445436+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.363 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: v1.5.0: Pulling from prometheus/node-exporter 2024-09-10T18:50:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.137294+0000 osd.7 (osd.7) 7 : cluster [DBG] 3.f scrub starts 2024-09-10T18:50:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.140598+0000 osd.7 (osd.7) 8 : cluster [DBG] 3.f scrub ok 2024-09-10T18:50:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.338396+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:50:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.343881+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.448781+0000 mgr.y (mgr.24526) 23 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cephadm 2024-09-10T18:50:22.638090+0000 mgr.y (mgr.24526) 24 : cephadm [INF] Deploying daemon node-exporter.b on smithi165 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.674812+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: cluster 2024-09-10T18:50:22.679704+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: audit 2024-09-10T18:50:23.406747+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: audit 2024-09-10T18:50:23.419357+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: audit 2024-09-10T18:50:23.427803+0000 mon.a (mon.0) 794 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: audit 2024-09-10T18:50:23.433660+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[22591]: audit 2024-09-10T18:50:23.445436+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- 172.21.15.165:0/3817092238 >> v1:172.21.15.165:6789/0 conn(0x7f9d60108d30 legacy=0x7f9d6010b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- 172.21.15.165:0/3817092238 shutdown_connections 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- 172.21.15.165:0/3817092238 >> 172.21.15.165:0/3817092238 conn(0x7f9d600f7fe0 msgr2=0x7f9d600fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- 172.21.15.165:0/3817092238 shutdown_connections 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- 172.21.15.165:0/3817092238 wait complete. 2024-09-10T18:50:24.623 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 librados: starting msgr at 2024-09-10T18:50:24.624 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 librados: starting objecter 2024-09-10T18:50:24.624 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 Processor -- start 2024-09-10T18:50:24.624 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- start start 2024-09-10T18:50:24.624 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 librados: setting wanted keys 2024-09-10T18:50:24.624 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 librados: calling monclient init 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d60122670 con 0x7f9d601051d0 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d60123810 con 0x7f9d60108d30 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d65990640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d601249b0 con 0x7f9d601023f0 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d6518f640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f9d60108d30 0x7f9d60120f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:36018/0 (socket says 172.21.15.165:36018) 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d6518f640 1 -- 172.21.15.165:0/2737798106 learned_addr learned my addr 172.21.15.165:0/2737798106 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d60108d30 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:24.626 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d601023f0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter ms_handle_connect 0x7f9d601051d0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193848838 0 0) 0x7f9d60123810 con 0x7f9d60108d30 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d500039c0 con 0x7f9d60108d30 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262836008 0 0) 0x7f9d601249b0 con 0x7f9d601023f0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d60123810 con 0x7f9d601023f0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902915015 0 0) 0x7f9d60122670 con 0x7f9d601051d0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d601249b0 con 0x7f9d601051d0 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 487295025 0 0) 0x7f9d500039c0 con 0x7f9d60108d30 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d60122670 con 0x7f9d60108d30 2024-09-10T18:50:24.627 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9d58003020 con 0x7f9d60108d30 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.615+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300115509 0 0) 0x7f9d60123810 con 0x7f9d601023f0 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d500039c0 con 0x7f9d601023f0 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804184103 0 0) 0x7f9d601249b0 con 0x7f9d601051d0 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d60123810 con 0x7f9d601051d0 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1818393795 0 0) 0x7f9d60122670 con 0x7f9d60108d30 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 >> v1:172.21.15.83:6790/0 conn(0x7f9d601023f0 legacy=0x7f9d60106e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 >> v1:172.21.15.83:6789/0 conn(0x7f9d601051d0 legacy=0x7f9d60107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60125b50 con 0x7f9d60108d30 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d60124ba0 con 0x7f9d60108d30 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d65990640 10 client.24562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d60124ff0 con 0x7f9d60108d30 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d65990640 1 librados: init done 2024-09-10T18:50:24.628 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9d58001510 con 0x7f9d60108d30 2024-09-10T18:50:24.629 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9d580038e0 con 0x7f9d60108d30 2024-09-10T18:50:24.629 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d44005010 con 0x7f9d60108d30 2024-09-10T18:50:24.629 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.619+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9d58003ba0 con 0x7f9d60108d30 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(93..93 src has 14..93) v4 ==== 6284+0+0 (unknown 3658611096 0 0) 0x7f9d5808a4e0 con 0x7f9d60108d30 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 10 client.24562.objecter ms_dispatch 0x7f9d60000b90 osd_map(93..93 src has 14..93) v4 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 3 client.24562.objecter handle_osd_map got epochs [93,93] > 0 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 3 client.24562.objecter handle_osd_map decoding full epoch 93 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 20 client.24562.objecter dump_active .. 0 homeless 2024-09-10T18:50:24.630 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.623+0000 7f9d5d7fa640 10 client.24562.objecter ms_handle_connect 0x7f9d500718e0 2024-09-10T18:50:24.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.627+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9d58057150 con 0x7f9d60108d30 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 22b70bddd3ac: Pulling fs layer 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 5c12815fee55: Pulling fs layer 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: c0e87333d380: Pulling fs layer 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 22b70bddd3ac: Verifying Checksum 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 22b70bddd3ac: Download complete 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 5c12815fee55: Verifying Checksum 2024-09-10T18:50:24.787 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 5c12815fee55: Download complete 2024-09-10T18:50:24.790 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[65618]: 5c12815fee55: Pull complete 2024-09-10T18:50:24.838 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.831+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d44000b90 con 0x7f9d500718e0 2024-09-10T18:50:24.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.843+0000 7f9d5d7fa640 1 -- 172.21.15.165:0/2737798106 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7f9d44000b90 con 0x7f9d500718e0 2024-09-10T18:50:24.852 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled grafana update... 2024-09-10T18:50:24.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 10 librados: watch_flush enter 2024-09-10T18:50:24.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 10 librados: watch_flush exit 2024-09-10T18:50:24.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 20 client.24562.objecter shutdown clearing up homeless session... 2024-09-10T18:50:24.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 10 client.24562.objecter successfully canceled tick 2024-09-10T18:50:24.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9d500718e0 legacy=0x7f9d50073d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 >> v1:172.21.15.165:6789/0 conn(0x7f9d60108d30 legacy=0x7f9d60120f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 shutdown_connections 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 >> 172.21.15.165:0/2737798106 conn(0x7f9d600f7fe0 msgr2=0x7f9d6010c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 shutdown_connections 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 -- 172.21.15.165:0/2737798106 wait complete. 2024-09-10T18:50:24.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:24.847+0000 7f9d65990640 1 librados: shutdown 2024-09-10T18:50:25.042 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: c0e87333d380: Verifying Checksum 2024-09-10T18:50:25.043 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: c0e87333d380: Download complete 2024-09-10T18:50:25.043 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:24 smithi165 bash[56588]: 22b70bddd3ac: Pull complete 2024-09-10T18:50:25.223 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:24 smithi083 bash[65618]: c0e87333d380: Pull complete 2024-09-10T18:50:25.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[18629]: cephadm 2024-09-10T18:50:23.459693+0000 mgr.y (mgr.24526) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi083 2024-09-10T18:50:25.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.306959+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:25.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.307089+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:25.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[18629]: audit 2024-09-10T18:50:24.532228+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[18629]: audit 2024-09-10T18:50:24.850833+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[32381]: cephadm 2024-09-10T18:50:23.459693+0000 mgr.y (mgr.24526) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi083 2024-09-10T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.306959+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.307089+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[32381]: audit 2024-09-10T18:50:24.532228+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[32381]: audit 2024-09-10T18:50:24.850833+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:25 smithi165 bash[22591]: cephadm 2024-09-10T18:50:23.459693+0000 mgr.y (mgr.24526) 25 : cephadm [INF] Deploying daemon alertmanager.a on smithi083 2024-09-10T18:50:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:25 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.306959+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:25 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.307089+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:25 smithi165 bash[22591]: audit 2024-09-10T18:50:24.532228+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:25 smithi165 bash[22591]: audit 2024-09-10T18:50:24.850833+0000 mon.a (mon.0) 798 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:25.540 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[65618]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-09-10T18:50:25.540 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:25 smithi083 bash[65618]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T18:50:26.097 DEBUG:teuthology.orchestra.run.smithi165:grafana.a> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@grafana.a.service 2024-09-10T18:50:26.100 INFO:tasks.cephadm:Setting up client nodes... 2024-09-10T18:50:26.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T18:50:26.363 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[56588]: 5c12815fee55: Pull complete 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.449416+0000 mgr.y (mgr.24526) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.641860+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.641957+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.814712+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:24.817357+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:50:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: audit 2024-09-10T18:50:24.840488+0000 mgr.y (mgr.24526) 27 : audit [DBG] from='client.24562 v1:172.21.15.165:0/2737798106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cephadm 2024-09-10T18:50:24.844074+0000 mgr.y (mgr.24526) 28 : cephadm [INF] Saving service grafana spec with placement smithi165=a;count:1 2024-09-10T18:50:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.046836+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:50:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.109385+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:50:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.112880+0000 osd.7 (osd.7) 10 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:50:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.646553+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-10T18:50:26.431 INFO:journalctl@ceph.grafana.a.smithi165.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T18:50:26.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.449416+0000 mgr.y (mgr.24526) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.641860+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.641957+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.814712+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:24.817357+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: audit 2024-09-10T18:50:24.840488+0000 mgr.y (mgr.24526) 27 : audit [DBG] from='client.24562 v1:172.21.15.165:0/2737798106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cephadm 2024-09-10T18:50:24.844074+0000 mgr.y (mgr.24526) 28 : cephadm [INF] Saving service grafana spec with placement smithi165=a;count:1 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.046836+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.109385+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.112880+0000 osd.7 (osd.7) 10 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.646553+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-10T18:50:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.449416+0000 mgr.y (mgr.24526) 26 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.641860+0000 osd.2 (osd.2) 11 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.641957+0000 osd.2 (osd.2) 12 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.814712+0000 osd.3 (osd.3) 11 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:24.817357+0000 osd.3 (osd.3) 12 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: audit 2024-09-10T18:50:24.840488+0000 mgr.y (mgr.24526) 27 : audit [DBG] from='client.24562 v1:172.21.15.165:0/2737798106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cephadm 2024-09-10T18:50:24.844074+0000 mgr.y (mgr.24526) 28 : cephadm [INF] Saving service grafana spec with placement smithi165=a;count:1 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.046836+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.109385+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.112880+0000 osd.7 (osd.7) 10 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:50:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:26 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.646553+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-09-10T18:50:26.985 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[56588]: c0e87333d380: Pull complete 2024-09-10T18:50:26.985 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[56588]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-09-10T18:50:26.985 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:26 smithi165 bash[56588]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T18:50:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:27 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.889109+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:50:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:27 smithi165 bash[22591]: cluster 2024-09-10T18:50:25.895772+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:50:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:27 smithi165 bash[22591]: cluster 2024-09-10T18:50:26.679993+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:50:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:27 smithi165 bash[22591]: cluster 2024-09-10T18:50:26.684074+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:50:27.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.889109+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[18629]: cluster 2024-09-10T18:50:25.895772+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[18629]: cluster 2024-09-10T18:50:26.679993+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[18629]: cluster 2024-09-10T18:50:26.684074+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.889109+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[32381]: cluster 2024-09-10T18:50:25.895772+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[32381]: cluster 2024-09-10T18:50:26.679993+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:50:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:27 smithi083 bash[32381]: cluster 2024-09-10T18:50:26.684074+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:50:27.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[22591]: cluster 2024-09-10T18:50:26.310239+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:50:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[22591]: cluster 2024-09-10T18:50:26.315887+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:50:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[22591]: cluster 2024-09-10T18:50:26.450227+0000 mgr.y (mgr.24526) 29 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[22591]: cluster 2024-09-10T18:50:27.178687+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:50:28.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[22591]: cluster 2024-09-10T18:50:27.184119+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[18629]: cluster 2024-09-10T18:50:26.310239+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[18629]: cluster 2024-09-10T18:50:26.315887+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[18629]: cluster 2024-09-10T18:50:26.450227+0000 mgr.y (mgr.24526) 29 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[18629]: cluster 2024-09-10T18:50:27.178687+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[18629]: cluster 2024-09-10T18:50:27.184119+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[32381]: cluster 2024-09-10T18:50:26.310239+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:50:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[32381]: cluster 2024-09-10T18:50:26.315887+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:50:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[32381]: cluster 2024-09-10T18:50:26.450227+0000 mgr.y (mgr.24526) 29 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[32381]: cluster 2024-09-10T18:50:27.178687+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:50:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[32381]: cluster 2024-09-10T18:50:27.184119+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.194Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.194Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.195Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.195Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T18:50:28.542 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T18:50:28.543 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.196Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T18:50:28.544 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T18:50:28.545 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T18:50:28.546 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 18:50:28 smithi083 bash[65618]: ts=2024-09-10T18:50:28.197Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T18:50:28.863 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.526Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.526Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.528Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.529Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.529Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.529Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.530Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.530Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.530Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T18:50:28.864 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T18:50:28.865 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T18:50:28.866 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.531Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T18:50:28.867 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T18:50:28.868 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T18:50:28.868 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T18:50:28.868 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.532Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T18:50:28.868 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.533Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T18:50:28.868 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 18:50:28 smithi165 bash[56588]: ts=2024-09-10T18:50:28.533Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:27.312528+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:27.317807+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: audit 2024-09-10T18:50:27.386259+0000 mgr.y (mgr.24526) 30 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.160377+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.163892+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.755568+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.759196+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:27.312528+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:27.317807+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T18:50:29.330 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: audit 2024-09-10T18:50:27.386259+0000 mgr.y (mgr.24526) 30 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:29.331 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.160377+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:50:29.331 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.163892+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:50:29.331 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.755568+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:50:29.331 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:29 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.759196+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:27.312528+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:27.317807+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: audit 2024-09-10T18:50:27.386259+0000 mgr.y (mgr.24526) 30 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.160377+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.163892+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.755568+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:50:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:29 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.759196+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:50:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.451021+0000 mgr.y (mgr.24526) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.684669+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.a scrub starts 2024-09-10T18:50:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: cluster 2024-09-10T18:50:28.688770+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.a scrub ok 2024-09-10T18:50:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: cluster 2024-09-10T18:50:29.190837+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:50:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: cluster 2024-09-10T18:50:29.195490+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:50:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:30 smithi165 bash[22591]: audit 2024-09-10T18:50:29.539176+0000 mon.c (mon.2) 47 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:30.406 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.451021+0000 mgr.y (mgr.24526) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.684669+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.a scrub starts 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: cluster 2024-09-10T18:50:28.688770+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.a scrub ok 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: cluster 2024-09-10T18:50:29.190837+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: cluster 2024-09-10T18:50:29.195490+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[32381]: audit 2024-09-10T18:50:29.539176+0000 mon.c (mon.2) 47 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.451021+0000 mgr.y (mgr.24526) 31 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.684669+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.a scrub starts 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: cluster 2024-09-10T18:50:28.688770+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.a scrub ok 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: cluster 2024-09-10T18:50:29.190837+0000 osd.7 (osd.7) 15 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:50:30.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: cluster 2024-09-10T18:50:29.195490+0000 osd.7 (osd.7) 16 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:50:30.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:30 smithi083 bash[18629]: audit 2024-09-10T18:50:29.539176+0000 mon.c (mon.2) 47 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:31.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[18629]: cluster 2024-09-10T18:50:30.202034+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.12 deep-scrub starts 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[18629]: cluster 2024-09-10T18:50:30.205568+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.12 deep-scrub ok 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[18629]: cluster 2024-09-10T18:50:30.648921+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[18629]: cluster 2024-09-10T18:50:30.652285+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[32381]: cluster 2024-09-10T18:50:30.202034+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.12 deep-scrub starts 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[32381]: cluster 2024-09-10T18:50:30.205568+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.12 deep-scrub ok 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[32381]: cluster 2024-09-10T18:50:30.648921+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:50:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:31 smithi083 bash[32381]: cluster 2024-09-10T18:50:30.652285+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:50:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:31 smithi165 bash[22591]: cluster 2024-09-10T18:50:30.202034+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.12 deep-scrub starts 2024-09-10T18:50:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:31 smithi165 bash[22591]: cluster 2024-09-10T18:50:30.205568+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.12 deep-scrub ok 2024-09-10T18:50:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:31 smithi165 bash[22591]: cluster 2024-09-10T18:50:30.648921+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:50:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:31 smithi165 bash[22591]: cluster 2024-09-10T18:50:30.652285+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:50:31.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:50:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:50:32.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[18629]: cluster 2024-09-10T18:50:30.452401+0000 mgr.y (mgr.24526) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:32.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[18629]: cluster 2024-09-10T18:50:31.796750+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:50:32.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[18629]: cluster 2024-09-10T18:50:31.799912+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:50:32.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[32381]: cluster 2024-09-10T18:50:30.452401+0000 mgr.y (mgr.24526) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:32.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[32381]: cluster 2024-09-10T18:50:31.796750+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:50:32.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:32 smithi083 bash[32381]: cluster 2024-09-10T18:50:31.799912+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:50:32.253 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:50:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:32 smithi165 bash[22591]: cluster 2024-09-10T18:50:30.452401+0000 mgr.y (mgr.24526) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:32 smithi165 bash[22591]: cluster 2024-09-10T18:50:31.796750+0000 osd.0 (osd.0) 21 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:50:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:32 smithi165 bash[22591]: cluster 2024-09-10T18:50:31.799912+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:50:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:33 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.469230+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.a scrub starts 2024-09-10T18:50:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:33 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.472535+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.a scrub ok 2024-09-10T18:50:33.377 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:33 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.469230+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.a scrub starts 2024-09-10T18:50:33.377 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:33 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.472535+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.a scrub ok 2024-09-10T18:50:33.377 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:33 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.469230+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.a scrub starts 2024-09-10T18:50:33.377 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:33 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.472535+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.a scrub ok 2024-09-10T18:50:34.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.385459+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.389659+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.453271+0000 mgr.y (mgr.24526) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.858187+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[18629]: cluster 2024-09-10T18:50:32.862301+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.385459+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.389659+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.453271+0000 mgr.y (mgr.24526) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:34.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.858187+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:50:34.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:34 smithi083 bash[32381]: cluster 2024-09-10T18:50:32.862301+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:50:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:34 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.385459+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:50:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:34 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.389659+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:50:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:34 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.453271+0000 mgr.y (mgr.24526) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:34 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.858187+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:50:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:34 smithi165 bash[22591]: cluster 2024-09-10T18:50:32.862301+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:50:35.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: cluster 2024-09-10T18:50:33.411712+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: cluster 2024-09-10T18:50:33.415588+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: audit 2024-09-10T18:50:34.859498+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: audit 2024-09-10T18:50:34.866924+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: audit 2024-09-10T18:50:34.872946+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: audit 2024-09-10T18:50:34.878802+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[18629]: audit 2024-09-10T18:50:34.921542+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: cluster 2024-09-10T18:50:33.411712+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:50:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: cluster 2024-09-10T18:50:33.415588+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:50:35.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: audit 2024-09-10T18:50:34.859498+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: audit 2024-09-10T18:50:34.866924+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: audit 2024-09-10T18:50:34.872946+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: audit 2024-09-10T18:50:34.878802+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:35 smithi083 bash[32381]: audit 2024-09-10T18:50:34.921542+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: cluster 2024-09-10T18:50:33.411712+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:50:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: cluster 2024-09-10T18:50:33.415588+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:50:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: audit 2024-09-10T18:50:34.859498+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: audit 2024-09-10T18:50:34.866924+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: audit 2024-09-10T18:50:34.872946+0000 mon.a (mon.0) 803 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: audit 2024-09-10T18:50:34.878802+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:35 smithi165 bash[22591]: audit 2024-09-10T18:50:34.921542+0000 mon.c (mon.2) 48 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:50:36.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: cluster 2024-09-10T18:50:34.453846+0000 mgr.y (mgr.24526) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: audit 2024-09-10T18:50:35.223203+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: audit 2024-09-10T18:50:35.231607+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: cluster 2024-09-10T18:50:35.297599+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: cluster 2024-09-10T18:50:35.300949+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: cluster 2024-09-10T18:50:35.712308+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[18629]: cluster 2024-09-10T18:50:35.715597+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: cluster 2024-09-10T18:50:34.453846+0000 mgr.y (mgr.24526) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: audit 2024-09-10T18:50:35.223203+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: audit 2024-09-10T18:50:35.231607+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: cluster 2024-09-10T18:50:35.297599+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:50:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: cluster 2024-09-10T18:50:35.300949+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:50:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: cluster 2024-09-10T18:50:35.712308+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:50:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:36 smithi083 bash[32381]: cluster 2024-09-10T18:50:35.715597+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:50:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: cluster 2024-09-10T18:50:34.453846+0000 mgr.y (mgr.24526) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:50:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: audit 2024-09-10T18:50:35.223203+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: audit 2024-09-10T18:50:35.231607+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: cluster 2024-09-10T18:50:35.297599+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:50:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: cluster 2024-09-10T18:50:35.300949+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:50:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: cluster 2024-09-10T18:50:35.712308+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:50:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:36 smithi165 bash[22591]: cluster 2024-09-10T18:50:35.715597+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.446522+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.452022+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.468391+0000 osd.5 (osd.5) 13 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.476877+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.662435+0000 osd.1 (osd.1) 21 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:50:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.667594+0000 osd.1 (osd.1) 22 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:50:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/3668345961 >> v1:172.21.15.83:6789/0 conn(0x7fa21c108f90 legacy=0x7fa21c10b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/3668345961 shutdown_connections 2024-09-10T18:50:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/3668345961 >> 172.21.15.83:0/3668345961 conn(0x7fa21c0f8230 msgr2=0x7fa21c0fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/3668345961 shutdown_connections 2024-09-10T18:50:37.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/3668345961 wait complete. 2024-09-10T18:50:37.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 librados: starting msgr at 2024-09-10T18:50:37.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 librados: starting objecter 2024-09-10T18:50:37.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 Processor -- start 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- start start 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 librados: setting wanted keys 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 librados: calling monclient init 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c0ff400 con 0x7fa21c102670 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c11d4b0 con 0x7fa21c105400 2024-09-10T18:50:37.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c11e650 con 0x7fa21c108f90 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa21b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa21c102670 0x7fa21c0fe7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55744/0 (socket says 172.21.15.83:55744) 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa21b7fe640 1 -- 172.21.15.83:0/1063039934 learned_addr learned my addr 172.21.15.83:0/1063039934 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa21bfff640 1 --1- 172.21.15.83:0/1063039934 >> v1:172.21.15.83:6790/0 conn(0x7fa21c108f90 0x7fa21c11ad80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60684/0 (socket says 172.21.15.83:60684) 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter ms_handle_connect 0x7fa21c102670 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter ms_handle_connect 0x7fa21c108f90 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451002469 0 0) 0x7fa21c0ff400 con 0x7fa21c102670 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2040034e0 con 0x7fa21c102670 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844384482 0 0) 0x7fa21c11e650 con 0x7fa21c108f90 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c0ff400 con 0x7fa21c108f90 2024-09-10T18:50:37.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter ms_handle_connect 0x7fa21c105400 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840037680 0 0) 0x7fa2040034e0 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa21c11e650 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa20c001980 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032881525 0 0) 0x7fa21c11d4b0 con 0x7fa21c105400 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2040034e0 con 0x7fa21c105400 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767888323 0 0) 0x7fa21c11e650 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 >> v1:172.21.15.83:6790/0 conn(0x7fa21c108f90 legacy=0x7fa21c11ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 >> v1:172.21.15.165:6789/0 conn(0x7fa21c105400 legacy=0x7fa21c117650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c11f850 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c11e840 con 0x7fa21c102670 2024-09-10T18:50:37.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:50:37.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c11ec90 con 0x7fa21c102670 2024-09-10T18:50:37.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa221f1e640 1 librados: init done 2024-09-10T18:50:37.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa20c002cc0 con 0x7fa21c102670 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.470+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa20c005b30 con 0x7fa21c102670 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa208005040 con 0x7fa21c102670 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa20c01c020 con 0x7fa21c102670 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(95..95 src has 14..95) v4 ==== 6284+0+0 (unknown 3237259073 0 0) 0x7fa20c08a0b0 con 0x7fa21c102670 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 10 client.14643.objecter ms_dispatch 0x7fa21c000b90 osd_map(95..95 src has 14..95) v4 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 3 client.14643.objecter handle_osd_map got epochs [95,95] > 0 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 3 client.14643.objecter handle_osd_map decoding full epoch 95 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 20 client.14643.objecter dump_active .. 0 homeless 2024-09-10T18:50:37.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.474+0000 7fa203fff640 10 client.14643.objecter ms_handle_connect 0x7fa204071610 2024-09-10T18:50:37.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.478+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa20c0023c0 con 0x7fa21c102670 2024-09-10T18:50:37.540 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[66125]: ts=2024-09-10T18:50:37.272Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.446522+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.452022+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.468391+0000 osd.5 (osd.5) 13 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.476877+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.662435+0000 osd.1 (osd.1) 21 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.667594+0000 osd.1 (osd.1) 22 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.446522+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.452022+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:50:37.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.468391+0000 osd.5 (osd.5) 13 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T18:50:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.476877+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T18:50:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.662435+0000 osd.1 (osd.1) 21 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:50:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:37 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.667594+0000 osd.1 (osd.1) 22 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:50:37.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.722+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa208005870 con 0x7fa21c102670 2024-09-10T18:50:37.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa203fff640 1 -- 172.21.15.83:0/1063039934 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 464748285) 0x7fa20c055cd0 con 0x7fa21c102670 2024-09-10T18:50:37.735 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-09-10T18:50:37.735 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQD9lOBm5zp/KxAA5mbWDjlxW6/cBNem111WCg== 2024-09-10T18:50:37.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 10 librados: watch_flush enter 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 10 librados: watch_flush exit 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 20 client.14643.objecter shutdown clearing up homeless session... 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 10 client.14643.objecter successfully canceled tick 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa204071610 legacy=0x7fa204073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.730+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 >> v1:172.21.15.83:6789/0 conn(0x7fa21c102670 legacy=0x7fa21c0fe7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.734+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 shutdown_connections 2024-09-10T18:50:37.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.734+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 >> 172.21.15.83:0/1063039934 conn(0x7fa21c0f8230 msgr2=0x7fa21c0f8600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:37.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.734+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 shutdown_connections 2024-09-10T18:50:37.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.734+0000 7fa221f1e640 1 -- 172.21.15.83:0/1063039934 wait complete. 2024-09-10T18:50:37.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:50:37.734+0000 7fa221f1e640 1 librados: shutdown 2024-09-10T18:50:37.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[18629]: cluster 2024-09-10T18:50:36.455355+0000 mgr.y (mgr.24526) 35 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[18629]: cluster 2024-09-10T18:50:37.463959+0000 osd.4 (osd.4) 25 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[18629]: cluster 2024-09-10T18:50:37.468561+0000 osd.4 (osd.4) 26 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[18629]: audit 2024-09-10T18:50:37.729490+0000 mon.a (mon.0) 807 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[18629]: audit 2024-09-10T18:50:37.734217+0000 mon.a (mon.0) 808 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[32381]: cluster 2024-09-10T18:50:36.455355+0000 mgr.y (mgr.24526) 35 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[32381]: cluster 2024-09-10T18:50:37.463959+0000 osd.4 (osd.4) 25 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[32381]: cluster 2024-09-10T18:50:37.468561+0000 osd.4 (osd.4) 26 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:50:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[32381]: audit 2024-09-10T18:50:37.729490+0000 mon.a (mon.0) 807 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:38.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:38 smithi083 bash[32381]: audit 2024-09-10T18:50:37.734217+0000 mon.a (mon.0) 808 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:38.506 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:50:38.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-10T18:50:38.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T18:50:38.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T18:50:38.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:38 smithi165 bash[22591]: cluster 2024-09-10T18:50:36.455355+0000 mgr.y (mgr.24526) 35 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:38.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:38 smithi165 bash[22591]: cluster 2024-09-10T18:50:37.463959+0000 osd.4 (osd.4) 25 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:50:38.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:38 smithi165 bash[22591]: cluster 2024-09-10T18:50:37.468561+0000 osd.4 (osd.4) 26 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:50:38.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:38 smithi165 bash[22591]: audit 2024-09-10T18:50:37.729490+0000 mon.a (mon.0) 807 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:38.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:38 smithi165 bash[22591]: audit 2024-09-10T18:50:37.734217+0000 mon.a (mon.0) 808 : audit [INF] from='client.? v1:172.21.15.83:0/1063039934' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:39.540 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[66125]: ts=2024-09-10T18:50:39.219Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000231342s 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[18629]: audit 2024-09-10T18:50:37.397574+0000 mgr.y (mgr.24526) 36 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[18629]: cluster 2024-09-10T18:50:37.836104+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[18629]: cluster 2024-09-10T18:50:37.841619+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[18629]: cluster 2024-09-10T18:50:38.774847+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.a scrub starts 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[18629]: cluster 2024-09-10T18:50:38.778193+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.a scrub ok 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[32381]: audit 2024-09-10T18:50:37.397574+0000 mgr.y (mgr.24526) 36 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[32381]: cluster 2024-09-10T18:50:37.836104+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:50:39.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[32381]: cluster 2024-09-10T18:50:37.841619+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:50:39.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[32381]: cluster 2024-09-10T18:50:38.774847+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.a scrub starts 2024-09-10T18:50:39.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:39 smithi083 bash[32381]: cluster 2024-09-10T18:50:38.778193+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.a scrub ok 2024-09-10T18:50:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:39 smithi165 bash[22591]: audit 2024-09-10T18:50:37.397574+0000 mgr.y (mgr.24526) 36 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:39 smithi165 bash[22591]: cluster 2024-09-10T18:50:37.836104+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:50:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:39 smithi165 bash[22591]: cluster 2024-09-10T18:50:37.841619+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:50:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:39 smithi165 bash[22591]: cluster 2024-09-10T18:50:38.774847+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.a scrub starts 2024-09-10T18:50:39.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:39 smithi165 bash[22591]: cluster 2024-09-10T18:50:38.778193+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.a scrub ok 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:38.456063+0000 mgr.y (mgr.24526) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:38.698387+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:38.702083+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.340941+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.344443+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.485126+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.488590+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: audit 2024-09-10T18:50:39.545752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.726390+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:50:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.730263+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:38.456063+0000 mgr.y (mgr.24526) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:38.698387+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:38.702083+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.340941+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.344443+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.485126+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.488590+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: audit 2024-09-10T18:50:39.545752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.726390+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:50:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:40 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.730263+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:50:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:38.456063+0000 mgr.y (mgr.24526) 37 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:38.698387+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:50:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:38.702083+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:50:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.340941+0000 osd.7 (osd.7) 21 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:50:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.344443+0000 osd.7 (osd.7) 22 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:50:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.485126+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:50:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.488590+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:50:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: audit 2024-09-10T18:50:39.545752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.726390+0000 osd.6 (osd.6) 15 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:50:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:40 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.730263+0000 osd.6 (osd.6) 16 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE Bus STOPPING 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE Bus STOPPED 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE Bus STARTING 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE Serving on http://:::9283 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18896]: [10/Sep/2024:18:50:41] ENGINE Bus STARTED 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.836298+0000 osd.3 (osd.3) 17 : cluster [DBG] 3.b deep-scrub starts 2024-09-10T18:50:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.845385+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.b deep-scrub ok 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.871759+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.c scrub starts 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:39.874665+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.c scrub ok 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.335849+0000 osd.7 (osd.7) 23 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.340429+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.481152+0000 osd.5 (osd.5) 15 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.485069+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:40.886606+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:40.896186+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:41.043948+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:41.054878+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:41.057762+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:41.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:41.059171+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[32381]: audit 2024-09-10T18:50:41.068107+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.836298+0000 osd.3 (osd.3) 17 : cluster [DBG] 3.b deep-scrub starts 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.845385+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.b deep-scrub ok 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.871759+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.c scrub starts 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:39.874665+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.c scrub ok 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.335849+0000 osd.7 (osd.7) 23 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.340429+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.481152+0000 osd.5 (osd.5) 15 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.485069+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:40.886606+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:40.896186+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:41.043948+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:41.054878+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:41.057762+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:41.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:41.059171+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:41.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:41 smithi083 bash[18629]: audit 2024-09-10T18:50:41.068107+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:50:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:50:41.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.836298+0000 osd.3 (osd.3) 17 : cluster [DBG] 3.b deep-scrub starts 2024-09-10T18:50:41.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.845385+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.b deep-scrub ok 2024-09-10T18:50:41.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.871759+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.c scrub starts 2024-09-10T18:50:41.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:39.874665+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.c scrub ok 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.335849+0000 osd.7 (osd.7) 23 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.340429+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.481152+0000 osd.5 (osd.5) 15 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.485069+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:40.886606+0000 mon.a (mon.0) 810 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:40.896186+0000 mon.a (mon.0) 811 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:41.043948+0000 mon.a (mon.0) 812 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:41.054878+0000 mon.a (mon.0) 813 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:41.057762+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:41.059171+0000 mon.c (mon.2) 50 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:50:41.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:41 smithi165 bash[22591]: audit 2024-09-10T18:50:41.068107+0000 mon.a (mon.0) 814 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.457039+0000 mgr.y (mgr.24526) 38 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.847157+0000 osd.3 (osd.3) 19 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.851155+0000 osd.3 (osd.3) 20 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.881164+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e scrub starts 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:40.884611+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e scrub ok 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: audit 2024-09-10T18:50:41.176821+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: audit 2024-09-10T18:50:41.186127+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: audit 2024-09-10T18:50:41.189502+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: audit 2024-09-10T18:50:41.190218+0000 mgr.y (mgr.24526) 39 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: audit 2024-09-10T18:50:41.200294+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.761873+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.766177+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.766823+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.770002+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.457039+0000 mgr.y (mgr.24526) 38 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.847157+0000 osd.3 (osd.3) 19 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.851155+0000 osd.3 (osd.3) 20 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.881164+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e scrub starts 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:40.884611+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e scrub ok 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: audit 2024-09-10T18:50:41.176821+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: audit 2024-09-10T18:50:41.186127+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: audit 2024-09-10T18:50:41.189502+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: audit 2024-09-10T18:50:41.190218+0000 mgr.y (mgr.24526) 39 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: audit 2024-09-10T18:50:41.200294+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.761873+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.766177+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.766823+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:50:42.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:42 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.770002+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:50:42.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.457039+0000 mgr.y (mgr.24526) 38 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:42.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.847157+0000 osd.3 (osd.3) 19 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.851155+0000 osd.3 (osd.3) 20 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.881164+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e scrub starts 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:40.884611+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e scrub ok 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: audit 2024-09-10T18:50:41.176821+0000 mon.a (mon.0) 815 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: audit 2024-09-10T18:50:41.186127+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: audit 2024-09-10T18:50:41.189502+0000 mon.c (mon.2) 51 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: audit 2024-09-10T18:50:41.190218+0000 mgr.y (mgr.24526) 39 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: audit 2024-09-10T18:50:41.200294+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.761873+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:50:42.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.766177+0000 osd.0 (osd.0) 24 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:50:42.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.766823+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:50:42.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:42 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.770002+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:50:43.074 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:50:43.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cephadm 2024-09-10T18:50:41.221419+0000 mgr.y (mgr.24526) 40 : cephadm [INF] Deploying daemon grafana.a on smithi165 2024-09-10T18:50:43.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.847910+0000 osd.2 (osd.2) 19 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:50:43.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:41.851274+0000 osd.2 (osd.2) 20 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:50:43.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.282889+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.286630+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.447000+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.c scrub starts 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.450632+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.c scrub ok 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.517324+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d scrub starts 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.521191+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d scrub ok 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.759185+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:50:43.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:43 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.762613+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cephadm 2024-09-10T18:50:41.221419+0000 mgr.y (mgr.24526) 40 : cephadm [INF] Deploying daemon grafana.a on smithi165 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.847910+0000 osd.2 (osd.2) 19 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:41.851274+0000 osd.2 (osd.2) 20 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.282889+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.286630+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.447000+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.c scrub starts 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.450632+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.c scrub ok 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.517324+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d scrub starts 2024-09-10T18:50:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.521191+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d scrub ok 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.759185+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.762613+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cephadm 2024-09-10T18:50:41.221419+0000 mgr.y (mgr.24526) 40 : cephadm [INF] Deploying daemon grafana.a on smithi165 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.847910+0000 osd.2 (osd.2) 19 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:41.851274+0000 osd.2 (osd.2) 20 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.282889+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.286630+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.447000+0000 osd.4 (osd.4) 29 : cluster [DBG] 4.c scrub starts 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.450632+0000 osd.4 (osd.4) 30 : cluster [DBG] 4.c scrub ok 2024-09-10T18:50:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.517324+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.d scrub starts 2024-09-10T18:50:43.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.521191+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d scrub ok 2024-09-10T18:50:43.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.759185+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:50:43.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:43 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.762613+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.457732+0000 mgr.y (mgr.24526) 41 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.802218+0000 osd.1 (osd.1) 25 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:42.807038+0000 osd.1 (osd.1) 26 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:43.566010+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:43.569851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:43.783136+0000 osd.0 (osd.0) 27 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[18629]: cluster 2024-09-10T18:50:43.788049+0000 osd.0 (osd.0) 28 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.457732+0000 mgr.y (mgr.24526) 41 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.802218+0000 osd.1 (osd.1) 25 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:50:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:42.807038+0000 osd.1 (osd.1) 26 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:50:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:43.566010+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:50:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:43.569851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:50:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:43.783136+0000 osd.0 (osd.0) 27 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:50:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:44 smithi083 bash[32381]: cluster 2024-09-10T18:50:43.788049+0000 osd.0 (osd.0) 28 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.457732+0000 mgr.y (mgr.24526) 41 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.802218+0000 osd.1 (osd.1) 25 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:42.807038+0000 osd.1 (osd.1) 26 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:43.566010+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:43.569851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:50:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:43.783136+0000 osd.0 (osd.0) 27 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:50:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:44 smithi165 bash[22591]: cluster 2024-09-10T18:50:43.788049+0000 osd.0 (osd.0) 28 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:50:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:45 smithi165 bash[22591]: cluster 2024-09-10T18:50:44.433290+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-09-10T18:50:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:45 smithi165 bash[22591]: cluster 2024-09-10T18:50:44.436231+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-09-10T18:50:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:45 smithi165 bash[22591]: audit 2024-09-10T18:50:44.539685+0000 mon.c (mon.2) 52 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:45 smithi165 bash[22591]: cluster 2024-09-10T18:50:44.851105+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:50:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:45 smithi165 bash[22591]: cluster 2024-09-10T18:50:44.854537+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[18629]: cluster 2024-09-10T18:50:44.433290+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[18629]: cluster 2024-09-10T18:50:44.436231+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[18629]: audit 2024-09-10T18:50:44.539685+0000 mon.c (mon.2) 52 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[18629]: cluster 2024-09-10T18:50:44.851105+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[18629]: cluster 2024-09-10T18:50:44.854537+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[32381]: cluster 2024-09-10T18:50:44.433290+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[32381]: cluster 2024-09-10T18:50:44.436231+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-09-10T18:50:45.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[32381]: audit 2024-09-10T18:50:44.539685+0000 mon.c (mon.2) 52 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:50:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[32381]: cluster 2024-09-10T18:50:44.851105+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:50:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:45 smithi083 bash[32381]: cluster 2024-09-10T18:50:44.854537+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:44.458444+0000 mgr.y (mgr.24526) 42 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.373611+0000 osd.7 (osd.7) 27 : cluster [DBG] 3.e scrub starts 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.377648+0000 osd.7 (osd.7) 28 : cluster [DBG] 3.e scrub ok 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.421098+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.424733+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.789357+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.793016+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:50:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:44.458444+0000 mgr.y (mgr.24526) 42 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.373611+0000 osd.7 (osd.7) 27 : cluster [DBG] 3.e scrub starts 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.377648+0000 osd.7 (osd.7) 28 : cluster [DBG] 3.e scrub ok 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.421098+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.424733+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.789357+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:50:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:46 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.793016+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:44.458444+0000 mgr.y (mgr.24526) 42 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.373611+0000 osd.7 (osd.7) 27 : cluster [DBG] 3.e scrub starts 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.377648+0000 osd.7 (osd.7) 28 : cluster [DBG] 3.e scrub ok 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.421098+0000 osd.4 (osd.4) 33 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.424733+0000 osd.4 (osd.4) 34 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:50:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.789357+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:50:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:46 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.793016+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:50:47.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.725241+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:50:47.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[22591]: cluster 2024-09-10T18:50:45.728800+0000 osd.2 (osd.2) 22 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:50:47.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[22591]: cluster 2024-09-10T18:50:46.459897+0000 mgr.y (mgr.24526) 43 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:47.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[22591]: cluster 2024-09-10T18:50:46.565904+0000 osd.5 (osd.5) 21 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:50:47.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[22591]: cluster 2024-09-10T18:50:46.569899+0000 osd.5 (osd.5) 22 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:50:47.540 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[66125]: ts=2024-09-10T18:50:47.222Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003102947s 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.725241+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[32381]: cluster 2024-09-10T18:50:45.728800+0000 osd.2 (osd.2) 22 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[32381]: cluster 2024-09-10T18:50:46.459897+0000 mgr.y (mgr.24526) 43 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[32381]: cluster 2024-09-10T18:50:46.565904+0000 osd.5 (osd.5) 21 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[32381]: cluster 2024-09-10T18:50:46.569899+0000 osd.5 (osd.5) 22 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:50:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.725241+0000 osd.2 (osd.2) 21 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:50:47.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[18629]: cluster 2024-09-10T18:50:45.728800+0000 osd.2 (osd.2) 22 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:50:47.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[18629]: cluster 2024-09-10T18:50:46.459897+0000 mgr.y (mgr.24526) 43 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:47.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[18629]: cluster 2024-09-10T18:50:46.565904+0000 osd.5 (osd.5) 21 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:50:47.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:47 smithi083 bash[18629]: cluster 2024-09-10T18:50:46.569899+0000 osd.5 (osd.5) 22 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:50:47.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:46.722704+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:46.725966+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.391395+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.394953+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: audit 2024-09-10T18:50:47.405836+0000 mgr.y (mgr.24526) 44 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.470836+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.473787+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.853787+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:50:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.857339+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:46.722704+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:46.725966+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.391395+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.394953+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: audit 2024-09-10T18:50:47.405836+0000 mgr.y (mgr.24526) 44 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.470836+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.473787+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.853787+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:50:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:48 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.857339+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:50:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:46.722704+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:50:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:46.725966+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:50:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.391395+0000 osd.7 (osd.7) 29 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:50:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.394953+0000 osd.7 (osd.7) 30 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:50:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: audit 2024-09-10T18:50:47.405836+0000 mgr.y (mgr.24526) 44 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.470836+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T18:50:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.473787+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T18:50:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.853787+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:50:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:48 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.857339+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.723548+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.727389+0000 osd.3 (osd.3) 22 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.770319+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[18629]: cluster 2024-09-10T18:50:47.773664+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[18629]: cluster 2024-09-10T18:50:48.460616+0000 mgr.y (mgr.24526) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.723548+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.727389+0000 osd.3 (osd.3) 22 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.770319+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-09-10T18:50:49.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[32381]: cluster 2024-09-10T18:50:47.773664+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-09-10T18:50:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:49 smithi083 bash[32381]: cluster 2024-09-10T18:50:48.460616+0000 mgr.y (mgr.24526) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:49 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.723548+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:50:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:49 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.727389+0000 osd.3 (osd.3) 22 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:50:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:49 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.770319+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-09-10T18:50:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:49 smithi165 bash[22591]: cluster 2024-09-10T18:50:47.773664+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-09-10T18:50:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:49 smithi165 bash[22591]: cluster 2024-09-10T18:50:48.460616+0000 mgr.y (mgr.24526) 45 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- 172.21.15.165:0/498703967 >> v1:172.21.15.165:6789/0 conn(0x7ff6dc1042e0 legacy=0x7ff6dc1066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- 172.21.15.165:0/498703967 shutdown_connections 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- 172.21.15.165:0/498703967 >> 172.21.15.165:0/498703967 conn(0x7ff6dc075320 msgr2=0x7ff6dc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- 172.21.15.165:0/498703967 shutdown_connections 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- 172.21.15.165:0/498703967 wait complete. 2024-09-10T18:50:49.964 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 librados: starting msgr at 2024-09-10T18:50:49.965 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 librados: starting objecter 2024-09-10T18:50:49.965 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 Processor -- start 2024-09-10T18:50:49.966 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- start start 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 librados: setting wanted keys 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 librados: calling monclient init 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6dc0ff3e0 con 0x7ff6dc101520 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6dc11d490 con 0x7ff6dc107e70 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e303b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6dc11e630 con 0x7ff6dc1042e0 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e15d8640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff6dc107e70 0x7ff6dc11ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:51508/0 (socket says 172.21.15.165:51508) 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.954+0000 7ff6e15d8640 1 -- 172.21.15.165:0/330209062 learned_addr learned my addr 172.21.15.165:0/330209062 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter ms_handle_connect 0x7ff6dc107e70 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter ms_handle_connect 0x7ff6dc101520 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:49.967 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter ms_handle_connect 0x7ff6dc1042e0 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783184158 0 0) 0x7ff6dc11d490 con 0x7ff6dc107e70 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc0034e0 con 0x7ff6dc107e70 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985823062 0 0) 0x7ff6dc0ff3e0 con 0x7ff6dc101520 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc11d490 con 0x7ff6dc101520 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875707084 0 0) 0x7ff6dc11e630 con 0x7ff6dc1042e0 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc0ff3e0 con 0x7ff6dc1042e0 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325987161 0 0) 0x7ff6cc0034e0 con 0x7ff6dc107e70 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6dc11e630 con 0x7ff6dc107e70 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff6d4001c80 con 0x7ff6dc107e70 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545281777 0 0) 0x7ff6dc11d490 con 0x7ff6dc101520 2024-09-10T18:50:49.968 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6cc0034e0 con 0x7ff6dc101520 2024-09-10T18:50:49.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1920208861 0 0) 0x7ff6dc11e630 con 0x7ff6dc107e70 2024-09-10T18:50:49.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 >> v1:172.21.15.83:6790/0 conn(0x7ff6dc1042e0 legacy=0x7ff6dc117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:49.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 >> v1:172.21.15.83:6789/0 conn(0x7ff6dc101520 legacy=0x7ff6dc0fe870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:49.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc11f810 con 0x7ff6dc107e70 2024-09-10T18:50:49.969 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc11d680 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6e303b640 10 client.24574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc11db50 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6e303b640 1 librados: init done 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff6d40027a0 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc0fcca0 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff6d4005e00 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.958+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff6d401c300 con 0x7ff6dc107e70 2024-09-10T18:50:49.970 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(95..95 src has 14..95) v4 ==== 6284+0+0 (unknown 3237259073 0 0) 0x7ff6d408b000 con 0x7ff6dc107e70 2024-09-10T18:50:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 10 client.24574.objecter ms_dispatch 0x7ff6dc000b90 osd_map(95..95 src has 14..95) v4 2024-09-10T18:50:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 3 client.24574.objecter handle_osd_map got epochs [95,95] > 0 2024-09-10T18:50:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 3 client.24574.objecter handle_osd_map decoding full epoch 95 2024-09-10T18:50:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 20 client.24574.objecter dump_active .. 0 homeless 2024-09-10T18:50:49.971 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.962+0000 7ff6d97fa640 10 client.24574.objecter ms_handle_connect 0x7ff6cc071700 2024-09-10T18:50:49.975 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:49.966+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff6d4057cd0 con 0x7ff6dc107e70 2024-09-10T18:50:50.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.250+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff6dc04bf60 con 0x7ff6dc107e70 2024-09-10T18:50:50.266 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6d97fa640 1 -- 172.21.15.165:0/330209062 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v19) v1 ==== 170+0+59 (unknown 1383567223 0 3886950002) 0x7ff6d4057670 con 0x7ff6dc107e70 2024-09-10T18:50:50.267 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-09-10T18:50:50.267 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQAKleBm5x5yDxAAsdN76ES955LvYFFNQkH2tg== 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 10 librados: watch_flush enter 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 10 librados: watch_flush exit 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 20 client.24574.objecter shutdown clearing up homeless session... 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 10 client.24574.objecter successfully canceled tick 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff6cc071700 legacy=0x7ff6cc073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:50.269 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.258+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 >> v1:172.21.15.165:6789/0 conn(0x7ff6dc107e70 legacy=0x7ff6dc11ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:50:50.270 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.262+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 shutdown_connections 2024-09-10T18:50:50.270 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.262+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 >> 172.21.15.165:0/330209062 conn(0x7ff6dc075320 msgr2=0x7ff6dc064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:50:50.270 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.262+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 shutdown_connections 2024-09-10T18:50:50.270 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.262+0000 7ff6e303b640 1 -- 172.21.15.165:0/330209062 wait complete. 2024-09-10T18:50:50.270 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T18:50:50.262+0000 7ff6e303b640 1 librados: shutdown 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:48.734867+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:48.738647+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.404976+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.d scrub starts 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.409327+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.d scrub ok 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.917545+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:50:50.494 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:50 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.921183+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:50:50.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:48.734867+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:50:50.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:48.738647+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.404976+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.d scrub starts 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.409327+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.d scrub ok 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.917545+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.921183+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:48.734867+0000 osd.3 (osd.3) 23 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:48.738647+0000 osd.3 (osd.3) 24 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.404976+0000 osd.7 (osd.7) 31 : cluster [DBG] 3.d scrub starts 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.409327+0000 osd.7 (osd.7) 32 : cluster [DBG] 3.d scrub ok 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.917545+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:50:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:50 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.921183+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.737158+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.741151+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.751071+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:49.756057+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: audit 2024-09-10T18:50:50.254954+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.165:0/330209062' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: audit 2024-09-10T18:50:50.258821+0000 mon.a (mon.0) 818 : audit [INF] from='client.24574 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: audit 2024-09-10T18:50:50.266232+0000 mon.a (mon.0) 819 : audit [INF] from='client.24574 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.461983+0000 mgr.y (mgr.24526) 46 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.790377+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.790462+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.737158+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.741151+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.751071+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:49.756057+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: audit 2024-09-10T18:50:50.254954+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.165:0/330209062' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: audit 2024-09-10T18:50:50.258821+0000 mon.a (mon.0) 818 : audit [INF] from='client.24574 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: audit 2024-09-10T18:50:50.266232+0000 mon.a (mon.0) 819 : audit [INF] from='client.24574 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.461983+0000 mgr.y (mgr.24526) 46 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:51.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.790377+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:51.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:51 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.790462+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.737158+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.741151+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.751071+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:49.756057+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: audit 2024-09-10T18:50:50.254954+0000 mon.b (mon.1) 34 : audit [INF] from='client.? v1:172.21.15.165:0/330209062' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: audit 2024-09-10T18:50:50.258821+0000 mon.a (mon.0) 818 : audit [INF] from='client.24574 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T18:50:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: audit 2024-09-10T18:50:50.266232+0000 mon.a (mon.0) 819 : audit [INF] from='client.24574 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T18:50:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.461983+0000 mgr.y (mgr.24526) 46 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.790377+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.790462+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:51.614 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:50:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:50:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.737639+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.d scrub starts 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.740790+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.d scrub ok 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.753239+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:50.756313+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.275416+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.425585+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.430068+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:50:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.922053+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.926520+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.737639+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.d scrub starts 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.740790+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.d scrub ok 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.753239+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:50.756313+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.275416+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.425585+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.430068+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.922053+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-10T18:50:52.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:52 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.926520+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.737639+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.d scrub starts 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.740790+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.d scrub ok 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.753239+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:50.756313+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.275416+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-09-10T18:50:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.425585+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:50:52.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.430068+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:50:52.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.922053+0000 osd.6 (osd.6) 25 : cluster [DBG] 3.a scrub starts 2024-09-10T18:50:52.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:52 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.926520+0000 osd.6 (osd.6) 26 : cluster [DBG] 3.a scrub ok 2024-09-10T18:50:53.510 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T18:50:53.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-10T18:50:53.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T18:50:53.529 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T18:50:53.530 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-10T18:50:53.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.631555+0000 osd.2 (osd.2) 23 : cluster [DBG] 6.f scrub starts 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.634790+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.f scrub ok 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.766138+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:51.769808+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.462707+0000 mgr.y (mgr.24526) 47 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.470270+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.473882+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.516787+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.520029+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.955156+0000 osd.6 (osd.6) 27 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:50:53.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.958784+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.631555+0000 osd.2 (osd.2) 23 : cluster [DBG] 6.f scrub starts 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.634790+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.f scrub ok 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.766138+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:51.769808+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.462707+0000 mgr.y (mgr.24526) 47 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.470270+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.473882+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.516787+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.520029+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.955156+0000 osd.6 (osd.6) 27 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:50:53.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:53 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.958784+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.631555+0000 osd.2 (osd.2) 23 : cluster [DBG] 6.f scrub starts 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.634790+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.f scrub ok 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.766138+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:51.769808+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.462707+0000 mgr.y (mgr.24526) 47 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.470270+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.b scrub starts 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.473882+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.b scrub ok 2024-09-10T18:50:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.516787+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:50:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.520029+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:50:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.955156+0000 osd.6 (osd.6) 27 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:50:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:53 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.958784+0000 osd.6 (osd.6) 28 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.631742+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:52.635172+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:53.429806+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:53.433705+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:53.513792+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[32381]: cluster 2024-09-10T18:50:53.516740+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.631742+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:52.635172+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:50:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:53.429806+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:50:54.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:53.433705+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:50:54.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:53.513792+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:50:54.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:54 smithi083 bash[18629]: cluster 2024-09-10T18:50:53.516740+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.631742+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:52.635172+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:53.429806+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:53.433705+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:53.513792+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:50:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:54 smithi165 bash[22591]: cluster 2024-09-10T18:50:53.516740+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:50:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:55 smithi165 bash[22591]: cluster 2024-09-10T18:50:54.463403+0000 mgr.y (mgr.24526) 48 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:55 smithi165 bash[22591]: cluster 2024-09-10T18:50:54.555795+0000 osd.4 (osd.4) 43 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:50:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:55 smithi165 bash[22591]: cluster 2024-09-10T18:50:54.559198+0000 osd.4 (osd.4) 44 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:50:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[18629]: cluster 2024-09-10T18:50:54.463403+0000 mgr.y (mgr.24526) 48 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[18629]: cluster 2024-09-10T18:50:54.555795+0000 osd.4 (osd.4) 43 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:50:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[18629]: cluster 2024-09-10T18:50:54.559198+0000 osd.4 (osd.4) 44 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:50:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[32381]: cluster 2024-09-10T18:50:54.463403+0000 mgr.y (mgr.24526) 48 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:50:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[32381]: cluster 2024-09-10T18:50:54.555795+0000 osd.4 (osd.4) 43 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:50:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:55 smithi083 bash[32381]: cluster 2024-09-10T18:50:54.559198+0000 osd.4 (osd.4) 44 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:54.734182+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:54.738762+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:55.394576+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:55.397534+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:55.400760+0000 osd.5 (osd.5) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:55.400835+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[18629]: cluster 2024-09-10T18:50:55.659480+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:54.734182+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:54.738762+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:50:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:55.394576+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:50:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:55.397534+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:50:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:55.400760+0000 osd.5 (osd.5) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:55.400835+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:56 smithi083 bash[32381]: cluster 2024-09-10T18:50:55.659480+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-10T18:50:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:54.734182+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:50:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:54.738762+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:50:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:55.394576+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:50:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:55.397534+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:50:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:55.400760+0000 osd.5 (osd.5) 23 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:55.400835+0000 osd.5 (osd.5) 24 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:56 smithi165 bash[22591]: cluster 2024-09-10T18:50:55.659480+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-09-10T18:50:57.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:50:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:50:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:57 smithi165 bash[22591]: cluster 2024-09-10T18:50:56.464133+0000 mgr.y (mgr.24526) 49 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:57 smithi165 bash[22591]: cluster 2024-09-10T18:50:56.769213+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T18:50:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:57 smithi165 bash[22591]: cluster 2024-09-10T18:50:56.774004+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T18:50:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[18629]: cluster 2024-09-10T18:50:56.464133+0000 mgr.y (mgr.24526) 49 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[18629]: cluster 2024-09-10T18:50:56.769213+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T18:50:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[18629]: cluster 2024-09-10T18:50:56.774004+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T18:50:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[32381]: cluster 2024-09-10T18:50:56.464133+0000 mgr.y (mgr.24526) 49 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:50:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[32381]: cluster 2024-09-10T18:50:56.769213+0000 osd.3 (osd.3) 31 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T18:50:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:57 smithi083 bash[32381]: cluster 2024-09-10T18:50:56.774004+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T18:50:58.302 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: audit 2024-09-10T18:50:57.417083+0000 mgr.y (mgr.24526) 50 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.418012+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.418115+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.449898+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.455393+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.475897+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.482079+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.682961+0000 mon.a (mon.0) 822 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.819202+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:50:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[18629]: cluster 2024-09-10T18:50:57.822926+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: audit 2024-09-10T18:50:57.417083+0000 mgr.y (mgr.24526) 50 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.418012+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.418115+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.449898+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.455393+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.475897+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.482079+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.682961+0000 mon.a (mon.0) 822 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.819202+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:50:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:58 smithi083 bash[32381]: cluster 2024-09-10T18:50:57.822926+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:50:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: audit 2024-09-10T18:50:57.417083+0000 mgr.y (mgr.24526) 50 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:50:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.418012+0000 osd.0 (osd.0) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:50:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.418115+0000 osd.0 (osd.0) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:50:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.449898+0000 osd.5 (osd.5) 25 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:50:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.455393+0000 osd.5 (osd.5) 26 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:50:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.475897+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:50:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.482079+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:50:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.682961+0000 mon.a (mon.0) 822 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-09-10T18:50:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.819202+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:50:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:58 smithi165 bash[22591]: cluster 2024-09-10T18:50:57.822926+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[18629]: cluster 2024-09-10T18:50:58.464837+0000 mgr.y (mgr.24526) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[18629]: cluster 2024-09-10T18:50:58.644283+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[18629]: cluster 2024-09-10T18:50:58.647538+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[18629]: audit 2024-09-10T18:50:59.540150+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[32381]: cluster 2024-09-10T18:50:58.464837+0000 mgr.y (mgr.24526) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T18:51:00.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[32381]: cluster 2024-09-10T18:50:58.644283+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-09-10T18:51:00.030 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[32381]: cluster 2024-09-10T18:50:58.647538+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-09-10T18:51:00.030 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:50:59 smithi083 bash[32381]: audit 2024-09-10T18:50:59.540150+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:59 smithi165 bash[22591]: cluster 2024-09-10T18:50:58.464837+0000 mgr.y (mgr.24526) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T18:51:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:59 smithi165 bash[22591]: cluster 2024-09-10T18:50:58.644283+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.b scrub starts 2024-09-10T18:51:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:59 smithi165 bash[22591]: cluster 2024-09-10T18:50:58.647538+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.b scrub ok 2024-09-10T18:51:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:50:59 smithi165 bash[22591]: audit 2024-09-10T18:50:59.540150+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:00.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:00 smithi165 bash[22591]: cluster 2024-09-10T18:50:59.606484+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:51:00.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:00 smithi165 bash[22591]: cluster 2024-09-10T18:50:59.610412+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:51:00.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:00 smithi165 bash[22591]: cluster 2024-09-10T18:50:59.745814+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:51:00.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:00 smithi165 bash[22591]: cluster 2024-09-10T18:50:59.749176+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:51:00.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:00 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.665130+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-10T18:51:01.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[18629]: cluster 2024-09-10T18:50:59.606484+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[18629]: cluster 2024-09-10T18:50:59.610412+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[18629]: cluster 2024-09-10T18:50:59.745814+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[18629]: cluster 2024-09-10T18:50:59.749176+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.665130+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[32381]: cluster 2024-09-10T18:50:59.606484+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[32381]: cluster 2024-09-10T18:50:59.610412+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[32381]: cluster 2024-09-10T18:50:59.745814+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[32381]: cluster 2024-09-10T18:50:59.749176+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:51:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:00 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.665130+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-09-10T18:51:01.701 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:51:01.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.793+0000 7f4482d6b640 1 -- 172.21.15.83:0/1375565001 >> v1:172.21.15.83:6790/0 conn(0x7f447c104810 legacy=0x7f447c104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:01.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.793+0000 7f4482d6b640 1 -- 172.21.15.83:0/1375565001 shutdown_connections 2024-09-10T18:51:01.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.793+0000 7f4482d6b640 1 -- 172.21.15.83:0/1375565001 >> 172.21.15.83:0/1375565001 conn(0x7f447c0fa370 msgr2=0x7f447c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:01.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.793+0000 7f4482d6b640 1 -- 172.21.15.83:0/1375565001 shutdown_connections 2024-09-10T18:51:01.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- 172.21.15.83:0/1375565001 wait complete. 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 librados: starting msgr at 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 librados: starting objecter 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 Processor -- start 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- start start 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 librados: setting wanted keys 2024-09-10T18:51:01.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 librados: calling monclient init 2024-09-10T18:51:01.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f447c1040f0 con 0x7f447c1075a0 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f447c11f650 con 0x7f447c10b130 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f447c120850 con 0x7f447c104810 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f447bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f447c1075a0 0x7f447c1197f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34440/0 (socket says 172.21.15.83:34440) 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4480b07640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f447c104810 0x7f447c103540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59558/0 (socket says 172.21.15.83:59558) 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f447bfff640 1 -- 172.21.15.83:0/2201623605 learned_addr learned my addr 172.21.15.83:0/2201623605 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter ms_handle_connect 0x7f447c1075a0 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter ms_handle_connect 0x7f447c104810 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832375349 0 0) 0x7f447c1040f0 con 0x7f447c1075a0 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44640034e0 con 0x7f447c1075a0 2024-09-10T18:51:01.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717409992 0 0) 0x7f447c120850 con 0x7f447c104810 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c1040f0 con 0x7f447c104810 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter ms_handle_connect 0x7f447c10b130 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1538266723 0 0) 0x7f44640034e0 con 0x7f447c1075a0 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f447c120850 con 0x7f447c1075a0 2024-09-10T18:51:01.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4470002950 con 0x7f447c1075a0 2024-09-10T18:51:01.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805057033 0 0) 0x7f447c11f650 con 0x7f447c10b130 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44640034e0 con 0x7f447c10b130 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030227209 0 0) 0x7f447c1040f0 con 0x7f447c104810 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f447c11f650 con 0x7f447c104810 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f446c0017b0 con 0x7f447c104810 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1873564139 0 0) 0x7f447c120850 con 0x7f447c1075a0 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 >> v1:172.21.15.83:6790/0 conn(0x7f447c104810 legacy=0x7f447c103540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 >> v1:172.21.15.165:6789/0 conn(0x7f447c10b130 legacy=0x7f447c11cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c121a50 con 0x7f447c1075a0 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c120a40 con 0x7f447c1075a0 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 10 client.14655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c120e90 con 0x7f447c1075a0 2024-09-10T18:51:01.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.797+0000 7f4482d6b640 1 librados: init done 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4470001750 con 0x7f447c1075a0 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4470005a80 con 0x7f447c1075a0 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468005040 con 0x7f447c1075a0 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f44700012f0 con 0x7f447c1075a0 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(99..99 src has 14..99) v4 ==== 6284+0+0 (unknown 3795323966 0 0) 0x7f4470089e30 con 0x7f447c1075a0 2024-09-10T18:51:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 10 client.14655.objecter ms_dispatch 0x7f447c000b90 osd_map(99..99 src has 14..99) v4 2024-09-10T18:51:01.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 3 client.14655.objecter handle_osd_map got epochs [99,99] > 0 2024-09-10T18:51:01.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 3 client.14655.objecter handle_osd_map decoding full epoch 99 2024-09-10T18:51:01.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 20 client.14655.objecter dump_active .. 0 homeless 2024-09-10T18:51:01.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.801+0000 7f44797fa640 10 client.14655.objecter ms_handle_connect 0x7f4464071740 2024-09-10T18:51:01.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:01.809+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4470056b00 con 0x7f447c1075a0 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.356251+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.361214+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.464178+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.466261+0000 mgr.y (mgr.24526) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.467183+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:51:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.723895+0000 osd.1 (osd.1) 39 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:51:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:01 smithi165 bash[22591]: cluster 2024-09-10T18:51:00.728144+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:51:02.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.029+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f44680052f0 con 0x7f447c1075a0 2024-09-10T18:51:02.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.037+0000 7f44797fa640 1 -- 172.21.15.83:0/2201623605 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) v1 ==== 74+0+176622 (unknown 3290845145 0 2453218339) 0x7f44700564a0 con 0x7f447c1075a0 2024-09-10T18:51:02.040 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.356251+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.361214+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.464178+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.466261+0000 mgr.y (mgr.24526) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.467183+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.723895+0000 osd.1 (osd.1) 39 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:51:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[32381]: cluster 2024-09-10T18:51:00.728144+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.356251+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.361214+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.464178+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.466261+0000 mgr.y (mgr.24526) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.467183+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.723895+0000 osd.1 (osd.1) 39 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:51:02.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:01 smithi083 bash[18629]: cluster 2024-09-10T18:51:00.728144+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:51:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 10 librados: watch_flush enter 2024-09-10T18:51:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 10 librados: watch_flush exit 2024-09-10T18:51:02.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 20 client.14655.objecter shutdown clearing up homeless session... 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 10 client.14655.objecter successfully canceled tick 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 >> v1:172.21.15.83:6800/3071793735 conn(0x7f4464071740 legacy=0x7f4464073bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 >> v1:172.21.15.83:6789/0 conn(0x7f447c1075a0 legacy=0x7f447c1197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 shutdown_connections 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 >> 172.21.15.83:0/2201623605 conn(0x7f447c0fa370 msgr2=0x7f447c10b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 shutdown_connections 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 -- 172.21.15.83:0/2201623605 wait complete. 2024-09-10T18:51:02.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:02.041+0000 7f4482d6b640 1 librados: shutdown 2024-09-10T18:51:02.757 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":21,"flags":0,"active_gid":24526,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6800","nonce":3071793735}]},"active_addr":"172.21.15.83:6800/3071793735","active_change":"2024-09-10T18:50:14.421885+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24523,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":92,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":1146618556}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":2419028730}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":902517130}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":295516604}]}]} 2024-09-10T18:51:02.761 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-10T18:51:02.762 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T18:51:02.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:51:02.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:02 smithi165 bash[22591]: cluster 2024-09-10T18:51:01.565127+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:51:02.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:02 smithi165 bash[22591]: cluster 2024-09-10T18:51:01.568564+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:51:02.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:02 smithi165 bash[22591]: audit 2024-09-10T18:51:02.031494+0000 mon.a (mon.0) 824 : audit [DBG] from='client.? v1:172.21.15.83:0/2201623605' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:51:03.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[18629]: cluster 2024-09-10T18:51:01.565127+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[18629]: cluster 2024-09-10T18:51:01.568564+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:51:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[18629]: audit 2024-09-10T18:51:02.031494+0000 mon.a (mon.0) 824 : audit [DBG] from='client.? v1:172.21.15.83:0/2201623605' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[32381]: cluster 2024-09-10T18:51:01.565127+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[32381]: cluster 2024-09-10T18:51:01.568564+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:51:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:02 smithi083 bash[32381]: audit 2024-09-10T18:51:02.031494+0000 mon.a (mon.0) 824 : audit [DBG] from='client.? v1:172.21.15.83:0/2201623605' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T18:51:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:03 smithi165 bash[22591]: cluster 2024-09-10T18:51:02.467016+0000 mgr.y (mgr.24526) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-09-10T18:51:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:03 smithi165 bash[22591]: cluster 2024-09-10T18:51:02.564547+0000 osd.4 (osd.4) 47 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:51:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:03 smithi165 bash[22591]: cluster 2024-09-10T18:51:02.568886+0000 osd.4 (osd.4) 48 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:51:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:03 smithi165 bash[22591]: cluster 2024-09-10T18:51:02.775043+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:51:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:03 smithi165 bash[22591]: cluster 2024-09-10T18:51:02.778236+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:51:03.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:03 smithi165 systemd[1]: Started Ceph grafana.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:51:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[18629]: cluster 2024-09-10T18:51:02.467016+0000 mgr.y (mgr.24526) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[18629]: cluster 2024-09-10T18:51:02.564547+0000 osd.4 (osd.4) 47 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[18629]: cluster 2024-09-10T18:51:02.568886+0000 osd.4 (osd.4) 48 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[18629]: cluster 2024-09-10T18:51:02.775043+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[18629]: cluster 2024-09-10T18:51:02.778236+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[32381]: cluster 2024-09-10T18:51:02.467016+0000 mgr.y (mgr.24526) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 751 B/s rd, 0 op/s 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[32381]: cluster 2024-09-10T18:51:02.564547+0000 osd.4 (osd.4) 47 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[32381]: cluster 2024-09-10T18:51:02.568886+0000 osd.4 (osd.4) 48 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[32381]: cluster 2024-09-10T18:51:02.775043+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:51:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:03 smithi083 bash[32381]: cluster 2024-09-10T18:51:02.778236+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:51:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.036839+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:51:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.040485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:51:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.526848+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.c scrub starts 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.530320+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.c scrub ok 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.554359+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:03.556569+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:03.856175+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:03.866565+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:03.873904+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:03.882534+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:03.927212+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: audit 2024-09-10T18:51:04.572888+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:04.590315+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:51:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:04 smithi165 bash[22591]: cluster 2024-09-10T18:51:04.593950+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.036839+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.040485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.526848+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.c scrub starts 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.530320+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.c scrub ok 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.554359+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:03.556569+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:03.856175+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:03.866565+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:03.873904+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:03.882534+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:03.927212+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:05.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: audit 2024-09-10T18:51:04.572888+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:04.590315+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[18629]: cluster 2024-09-10T18:51:04.593950+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.036839+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.040485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.526848+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.c scrub starts 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.530320+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.c scrub ok 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.554359+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:03.556569+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:03.856175+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:03.866565+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:03.873904+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:03.882534+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:03.927212+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: audit 2024-09-10T18:51:04.572888+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:04.590315+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:51:05.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:04 smithi083 bash[32381]: cluster 2024-09-10T18:51:04.593950+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376282527Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376493275Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376527018Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376547322Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376564758Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376576314Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.37658781Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376598953Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376611419Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376621973Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-10T18:51:05.638 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.37663487Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376646598Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.37667148Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=settings t=2024-09-10T18:51:05.376687136Z level=info msg="App mode production" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=sqlstore t=2024-09-10T18:51:05.376770523Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=sqlstore t=2024-09-10T18:51:05.376797549Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.377844574Z level=info msg="Starting DB migrations" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.377950915Z level=info msg="Executing migration" id="create migration_log table" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.43276879Z level=info msg="Executing migration" id="create user table" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.466280876Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.538272459Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.591990348Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-10T18:51:05.639 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.633817518Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-10T18:51:05.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[22591]: cluster 2024-09-10T18:51:04.467750+0000 mgr.y (mgr.24526) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:05.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[22591]: cluster 2024-09-10T18:51:05.601258+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:51:05.888 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.670587081Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-10T18:51:05.888 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.712560389Z level=info msg="Executing migration" id="create user table v2" 2024-09-10T18:51:05.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.737828187Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-10T18:51:05.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.821717221Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-10T18:51:05.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.872110678Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[22591]: cluster 2024-09-10T18:51:05.604795+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.914033942Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.950703387Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.967735741Z level=info msg="Executing migration" id="Update user table charset" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:05 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:05.992978867Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.084983828Z level=info msg="Executing migration" id="Add missing user data" 2024-09-10T18:51:06.139 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.110181524Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-10T18:51:06.140 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.135452321Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-10T18:51:06.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[18629]: cluster 2024-09-10T18:51:04.467750+0000 mgr.y (mgr.24526) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:06.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[18629]: cluster 2024-09-10T18:51:05.601258+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:51:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[18629]: cluster 2024-09-10T18:51:05.604795+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:51:06.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[32381]: cluster 2024-09-10T18:51:04.467750+0000 mgr.y (mgr.24526) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:06.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[32381]: cluster 2024-09-10T18:51:05.601258+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:51:06.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:05 smithi083 bash[32381]: cluster 2024-09-10T18:51:05.604795+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:51:06.402 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.177522964Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-10T18:51:06.402 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.206138164Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-10T18:51:06.402 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.231452458Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-10T18:51:06.403 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.315120336Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-10T18:51:06.403 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.35708061Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.398974127Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.440958101Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.477825979Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.561370208Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.603278913Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-10T18:51:06.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.628536178Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-10T18:51:06.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[22591]: cluster 2024-09-10T18:51:05.765488+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:51:06.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[22591]: cluster 2024-09-10T18:51:05.769632+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:51:06.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.637742+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:51:06.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.641190+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.670567524Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.695893938Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.724994387Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.808647335Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.850646188Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-10T18:51:06.934 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.893706089Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.929882683Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:06 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:06.98063338Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.05574611Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.081057367Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.12300734Z level=info msg="Executing migration" id="create star table" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.151979684Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-10T18:51:07.206 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.177291712Z level=info msg="Executing migration" id="create org table v1" 2024-09-10T18:51:07.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[18629]: cluster 2024-09-10T18:51:05.765488+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:51:07.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[18629]: cluster 2024-09-10T18:51:05.769632+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.637742+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.641190+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[32381]: cluster 2024-09-10T18:51:05.765488+0000 osd.3 (osd.3) 37 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[32381]: cluster 2024-09-10T18:51:05.769632+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.637742+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:51:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:06 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.641190+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:51:07.482 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:07.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.202400245Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-10T18:51:07.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.227682042Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-10T18:51:07.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.302911942Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-10T18:51:07.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.33209077Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-10T18:51:07.482 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.349068474Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-10T18:51:07.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.390951017Z level=info msg="Executing migration" id="Update org table charset" 2024-09-10T18:51:07.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.407719489Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-10T18:51:07.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.449756104Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-10T18:51:07.483 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.478738398Z level=info msg="Executing migration" id="create dashboard table" 2024-09-10T18:51:07.543 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:07.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.554091251Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-10T18:51:07.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.612622458Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-10T18:51:07.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.654762194Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-10T18:51:07.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.683894472Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-10T18:51:07.738 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.709128704Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-10T18:51:07.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.376399+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:51:07.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.380584+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:51:07.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.469216+0000 mgr.y (mgr.24526) 55 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:07.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.580562+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:51:07.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.584249+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:51:08.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.376399+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:51:08.068 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.380584+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:51:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.469216+0000 mgr.y (mgr.24526) 55 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.580562+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:51:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.584249+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:51:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.771257+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:51:08.069 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[22591]: cluster 2024-09-10T18:51:06.774480+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:51:08.069 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.734145878Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-10T18:51:08.069 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.817944637Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-10T18:51:08.069 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.843086189Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-10T18:51:08.069 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.868403706Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-10T18:51:08.069 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.914044003Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-10T18:51:08.070 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.956305338Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-10T18:51:08.070 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:07 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:07.981454808Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-10T18:51:08.070 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.064952916Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-10T18:51:08.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.771257+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:51:08.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[18629]: cluster 2024-09-10T18:51:06.774480+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.376399+0000 osd.7 (osd.7) 43 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.380584+0000 osd.7 (osd.7) 44 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.469216+0000 mgr.y (mgr.24526) 55 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.580562+0000 osd.5 (osd.5) 31 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.584249+0000 osd.5 (osd.5) 32 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.771257+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:51:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:07 smithi083 bash[32381]: cluster 2024-09-10T18:51:06.774480+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:51:08.336 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.094090837Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.119309508Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.136246876Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.161553336Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.186760737Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.212640245Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.241053128Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-10T18:51:08.337 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.333027243Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.349866752Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.391927759Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.412401824Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.454420727Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.529682149Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-10T18:51:08.609 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.580067602Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-10T18:51:08.859 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.605381615Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.63445929Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.676374596Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.693287281Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.718532278Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.802088069Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-10T18:51:08.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.856147182Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-10T18:51:09.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[18629]: audit 2024-09-10T18:51:07.428487+0000 mgr.y (mgr.24526) 56 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[18629]: cluster 2024-09-10T18:51:07.538841+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:51:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[18629]: cluster 2024-09-10T18:51:07.542234+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:51:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[32381]: audit 2024-09-10T18:51:07.428487+0000 mgr.y (mgr.24526) 56 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[32381]: cluster 2024-09-10T18:51:07.538841+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:51:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:08 smithi083 bash[32381]: cluster 2024-09-10T18:51:07.542234+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:51:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[22591]: audit 2024-09-10T18:51:07.428487+0000 mgr.y (mgr.24526) 56 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[22591]: cluster 2024-09-10T18:51:07.538841+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:51:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[22591]: cluster 2024-09-10T18:51:07.542234+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:51:09.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.881398912Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-10T18:51:09.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.923660078Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-10T18:51:09.114 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:08 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:08.965577315Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-10T18:51:09.114 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.060368792Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-10T18:51:09.114 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.085639009Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-10T18:51:09.424 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.127625737Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-10T18:51:09.424 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.15286867Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-10T18:51:09.424 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.178082005Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-10T18:51:09.424 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.220100402Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-10T18:51:09.425 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.303881691Z level=info msg="Executing migration" id="create data_source table" 2024-09-10T18:51:09.425 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.329084941Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-10T18:51:09.425 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.376866519Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.419052456Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.444395614Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.486340966Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.57191854Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.613918878Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-10T18:51:09.704 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.658137957Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.700287217Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.732250297Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.816119301Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.858122708Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.900073664Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-10T18:51:09.990 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.942555247Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: cluster 2024-09-10T18:51:08.423592+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: cluster 2024-09-10T18:51:08.423696+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: cluster 2024-09-10T18:51:08.469985+0000 mgr.y (mgr.24526) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: cluster 2024-09-10T18:51:08.918726+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: audit 2024-09-10T18:51:09.603175+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:09.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[22591]: audit 2024-09-10T18:51:09.613831+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:10.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: cluster 2024-09-10T18:51:08.423592+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: cluster 2024-09-10T18:51:08.423696+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: cluster 2024-09-10T18:51:08.469985+0000 mgr.y (mgr.24526) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: cluster 2024-09-10T18:51:08.918726+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: audit 2024-09-10T18:51:09.603175+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[18629]: audit 2024-09-10T18:51:09.613831+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: cluster 2024-09-10T18:51:08.423592+0000 osd.7 (osd.7) 45 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: cluster 2024-09-10T18:51:08.423696+0000 osd.7 (osd.7) 46 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: cluster 2024-09-10T18:51:08.469985+0000 mgr.y (mgr.24526) 57 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: cluster 2024-09-10T18:51:08.918726+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-09-10T18:51:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: audit 2024-09-10T18:51:09.603175+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:09 smithi083 bash[32381]: audit 2024-09-10T18:51:09.613831+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:10.315 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:09 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:09.986609753Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.072421728Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.114471389Z level=info msg="Executing migration" id="Add uid column" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.139908253Z level=info msg="Executing migration" id="Update uid value" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.167052354Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.192361305Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-10T18:51:10.316 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.217816182Z level=info msg="Executing migration" id="create api_key table" 2024-09-10T18:51:10.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.311969168Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-10T18:51:10.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.353977675Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-10T18:51:10.576 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.398268362Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-10T18:51:10.577 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.426654924Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-10T18:51:10.577 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.470414347Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-10T18:51:10.577 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.545850592Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-10T18:51:10.577 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.572872113Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.598363636Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.623602941Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.65072914Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.692934351Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.718392874Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-10T18:51:10.860 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.814137731Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-10T18:51:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.877+0000 7ff2d03b4640 1 -- 172.21.15.83:0/2513227442 >> v1:172.21.15.83:6789/0 conn(0x7ff2c81075a0 legacy=0x7ff2c8109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.877+0000 7ff2d03b4640 1 -- 172.21.15.83:0/2513227442 shutdown_connections 2024-09-10T18:51:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.877+0000 7ff2d03b4640 1 -- 172.21.15.83:0/2513227442 >> 172.21.15.83:0/2513227442 conn(0x7ff2c80fa370 msgr2=0x7ff2c80fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.877+0000 7ff2d03b4640 1 -- 172.21.15.83:0/2513227442 shutdown_connections 2024-09-10T18:51:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.877+0000 7ff2d03b4640 1 -- 172.21.15.83:0/2513227442 wait complete. 2024-09-10T18:51:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 librados: starting msgr at 2024-09-10T18:51:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 librados: starting objecter 2024-09-10T18:51:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 Processor -- start 2024-09-10T18:51:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- start start 2024-09-10T18:51:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 librados: setting wanted keys 2024-09-10T18:51:10.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 librados: calling monclient init 2024-09-10T18:51:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c81040f0 con 0x7ff2c810b130 2024-09-10T18:51:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c811f650 con 0x7ff2c81075a0 2024-09-10T18:51:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c8120850 con 0x7ff2c8104810 2024-09-10T18:51:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2ce150640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff2c8104810 0x7ff2c8103540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60324/0 (socket says 172.21.15.83:60324) 2024-09-10T18:51:10.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2ce150640 1 -- 172.21.15.83:0/3758419884 learned_addr learned my addr 172.21.15.83:0/3758419884 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter ms_handle_connect 0x7ff2c8104810 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter ms_handle_connect 0x7ff2c810b130 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter ms_handle_connect 0x7ff2c81075a0 2024-09-10T18:51:10.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406618785 0 0) 0x7ff2c8120850 con 0x7ff2c8104810 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b4003720 con 0x7ff2c8104810 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685596744 0 0) 0x7ff2c81040f0 con 0x7ff2c810b130 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c8120850 con 0x7ff2c810b130 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588194883 0 0) 0x7ff2c811f650 con 0x7ff2c81075a0 2024-09-10T18:51:10.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c81040f0 con 0x7ff2c81075a0 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336353329 0 0) 0x7ff2b4003720 con 0x7ff2c8104810 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c811f650 con 0x7ff2c8104810 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026810373 0 0) 0x7ff2c8120850 con 0x7ff2c810b130 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b4003720 con 0x7ff2c810b130 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff2c4001940 con 0x7ff2c8104810 2024-09-10T18:51:10.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff2c0001c40 con 0x7ff2c810b130 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2876015473 0 0) 0x7ff2c81040f0 con 0x7ff2c81075a0 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c8120850 con 0x7ff2c81075a0 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff2bc001430 con 0x7ff2c81075a0 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787209369 0 0) 0x7ff2b4003720 con 0x7ff2c810b130 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 >> v1:172.21.15.83:6790/0 conn(0x7ff2c8104810 legacy=0x7ff2c8103540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:10.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 >> v1:172.21.15.165:6789/0 conn(0x7ff2c81075a0 legacy=0x7ff2c81197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8121a50 con 0x7ff2c810b130 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c811e6a0 con 0x7ff2c810b130 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 10 client.14661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c811eaf0 con 0x7ff2c810b130 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.881+0000 7ff2d03b4640 1 librados: init done 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.885+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff2c00024d0 con 0x7ff2c810b130 2024-09-10T18:51:10.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.885+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff2c0003340 con 0x7ff2c810b130 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.885+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac005010 con 0x7ff2c810b130 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff2c0019760 con 0x7ff2c810b130 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(101..101 src has 49..101) v4 ==== 6284+0+0 (unknown 4267342977 0 0) 0x7ff2c00876c0 con 0x7ff2c810b130 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 10 client.14661.objecter ms_dispatch 0x7ff2c8000b90 osd_map(101..101 src has 49..101) v4 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 3 client.14661.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 3 client.14661.objecter handle_osd_map decoding full epoch 101 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 20 client.14661.objecter dump_active .. 0 homeless 2024-09-10T18:51:10.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.889+0000 7ff2baffd640 10 client.14661.objecter ms_handle_connect 0x7ff2b4071770 2024-09-10T18:51:10.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:10.897+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff2c0054330 con 0x7ff2c810b130 2024-09-10T18:51:11.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[18629]: cluster 2024-09-10T18:51:09.268716+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:11.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[18629]: cluster 2024-09-10T18:51:09.268832+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:11.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[18629]: cluster 2024-09-10T18:51:09.928453+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-10T18:51:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[32381]: cluster 2024-09-10T18:51:09.268716+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[32381]: cluster 2024-09-10T18:51:09.268832+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:10 smithi083 bash[32381]: cluster 2024-09-10T18:51:09.928453+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-10T18:51:11.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.073+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2ac0052c0 con 0x7ff2c810b130 2024-09-10T18:51:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2baffd640 1 -- 172.21.15.83:0/3758419884 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+19422 (unknown 901664539 0 179534950) 0x7ff2c0053cd0 con 0x7ff2c810b130 2024-09-10T18:51:11.080 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:51:11.080 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":101,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","created":"2024-09-10T18:38:38.475667+0000","modified":"2024-09-10T18:51:09.909233+0000","last_up_change":"2024-09-10T18:48:34.513026+0000","last_in_change":"2024-09-10T18:48:07.544682+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:44:06.952078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T18:49:16.239173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:49:18.096790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:49:20.070555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"datapool","create_time":"2024-09-10T18:49:20.615000+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":80,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:49:22.582739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"35ab8328-74f1-4af8-aa84-797914199445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6801","nonce":2535527483}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6802","nonce":2535527483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6804","nonce":2535527483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6803","nonce":2535527483}]},"public_addr":"172.21.15.83:6801/2535527483","cluster_addr":"172.21.15.83:6802/2535527483","heartbeat_back_addr":"172.21.15.83:6804/2535527483","heartbeat_front_addr":"172.21.15.83:6803/2535527483","state":["exists","up"]},{"osd":1,"uuid":"8e1d1072-767b-49fb-b2f2-fce60263e2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6805","nonce":2309624637}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6806","nonce":2309624637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6808","nonce":2309624637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6807","nonce":2309624637}]},"public_addr":"172.21.15.83:6805/2309624637","cluster_addr":"172.21.15.83:6806/2309624637","heartbeat_back_addr":"172.21.15.83:6808/2309624637","heartbeat_front_addr":"172.21.15.83:6807/2309624637","state":["exists","up"]},{"osd":2,"uuid":"6193b508-03d9-458a-b49f-6b2e24283282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6809","nonce":386094220}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6810","nonce":386094220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6812","nonce":386094220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6811","nonce":386094220}]},"public_addr":"172.21.15.83:6809/386094220","cluster_addr":"172.21.15.83:6810/386094220","heartbeat_back_addr":"172.21.15.83:6812/386094220","heartbeat_front_addr":"172.21.15.83:6811/386094220","state":["exists","up"]},{"osd":3,"uuid":"a28f94ff-2fff-458e-8b69-c5c65d720d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6813","nonce":1279688021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6814","nonce":1279688021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6816","nonce":1279688021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6815","nonce":1279688021}]},"public_addr":"172.21.15.83:6813/1279688021","cluster_addr":"172.21.15.83:6814/1279688021","heartbeat_back_addr":"172.21.15.83:6816/1279688021","heartbeat_front_addr":"172.21.15.83:6815/1279688021","state":["exists","up"]},{"osd":4,"uuid":"a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":3225515265}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":3225515265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":3225515265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":3225515265}]},"public_addr":"172.21.15.165:6800/3225515265","cluster_addr":"172.21.15.165:6801/3225515265","heartbeat_back_addr":"172.21.15.165:6803/3225515265","heartbeat_front_addr":"172.21.15.165:6802/3225515265","state":["exists","up"]},{"osd":5,"uuid":"10543223-db5a-493b-88f3-8705f2b31a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":571173777}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":571173777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":571173777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":571173777}]},"public_addr":"172.21.15.165:6804/571173777","cluster_addr":"172.21.15.165:6805/571173777","heartbeat_back_addr":"172.21.15.165:6807/571173777","heartbeat_front_addr":"172.21.15.165:6806/571173777","state":["exists","up"]},{"osd":6,"uuid":"fcea0ca4-4521-4161-b407-9b1fab1a2ae2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":1317922936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":1317922936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":1317922936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":1317922936}]},"public_addr":"172.21.15.165:6808/1317922936","cluster_addr":"172.21.15.165:6809/1317922936","heartbeat_back_addr":"172.21.15.165:6811/1317922936","heartbeat_front_addr":"172.21.15.165:6810/1317922936","state":["exists","up"]},{"osd":7,"uuid":"8531d286-5ebb-4c96-a026-63d8a17cb21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1610151650}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1610151650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1610151650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1610151650}]},"public_addr":"172.21.15.165:6812/1610151650","cluster_addr":"172.21.15.165:6813/1610151650","heartbeat_back_addr":"172.21.15.165:6815/1610151650","heartbeat_front_addr":"172.21.15.165:6814/1610151650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:57.418119+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:49:58.459425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:24.641962+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:24.307097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:50.790465+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:55.400840+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:51:09.268836+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:51:08.423724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/3645093631":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/3275901971":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3555551632":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/3076437878":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/4112054239":"2024-09-11T18:39:06.336264+0000","172.21.15.83:6800/2452184936":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/2659452540":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/867303943":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/1156549307":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/1687990153":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1246256198":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/3225538750":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/2630800463":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1423545952":"2024-09-11T18:50:14.421453+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:51:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 10 librados: watch_flush enter 2024-09-10T18:51:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 10 librados: watch_flush exit 2024-09-10T18:51:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 20 client.14661.objecter shutdown clearing up homeless session... 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 10 client.14661.objecter successfully canceled tick 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff2b4071770 legacy=0x7ff2b4073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 >> v1:172.21.15.83:6789/0 conn(0x7ff2c810b130 legacy=0x7ff2c811cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 shutdown_connections 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 >> 172.21.15.83:0/3758419884 conn(0x7ff2c80fa370 msgr2=0x7ff2c810b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 shutdown_connections 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.077+0000 7ff2d03b4640 1 -- 172.21.15.83:0/3758419884 wait complete. 2024-09-10T18:51:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:11.081+0000 7ff2d03b4640 1 librados: shutdown 2024-09-10T18:51:11.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.856333995Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-10T18:51:11.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.898110868Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-10T18:51:11.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.940242915Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-10T18:51:11.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:10.978091767Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-10T18:51:11.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.078451575Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-10T18:51:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[22591]: cluster 2024-09-10T18:51:09.268716+0000 osd.6 (osd.6) 31 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:11.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[22591]: cluster 2024-09-10T18:51:09.268832+0000 osd.6 (osd.6) 32 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:11.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:10 smithi165 bash[22591]: cluster 2024-09-10T18:51:09.928453+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-09-10T18:51:11.405 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:51:11.405 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.120577999Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-10T18:51:11.405 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.162422623Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-10T18:51:11.405 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.204538909Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-10T18:51:11.405 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.233587983Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-10T18:51:11.406 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.308908361Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-10T18:51:11.406 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.334270243Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-10T18:51:11.406 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.359426707Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-10T18:51:11.406 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.401526509Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-10T18:51:11.681 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.426998828Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.464406748Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.498103285Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.573488725Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.598845453Z level=info msg="Executing migration" id="create quota table v1" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.640643565Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-10T18:51:11.682 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.678077102Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-10T18:51:11.932 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.70339447Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-10T18:51:11.932 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.745497921Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-10T18:51:11.933 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.8291223Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-10T18:51:11.933 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.871024669Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-10T18:51:11.933 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.896291314Z level=info msg="Executing migration" id="create session table" 2024-09-10T18:51:11.933 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.921653419Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-10T18:51:11.933 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: cluster 2024-09-10T18:51:10.447093+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:51:11.951 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T18:51:11.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T18:51:12.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: cluster 2024-09-10T18:51:10.447093+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:51:12.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: cluster 2024-09-10T18:51:10.452368+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:51:12.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: cluster 2024-09-10T18:51:10.471264+0000 mgr.y (mgr.24526) 58 : cluster [DBG] pgmap v40: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.077655+0000 mon.a (mon.0) 834 : audit [DBG] from='client.? v1:172.21.15.83:0/3758419884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.134658+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.144444+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.149503+0000 mon.c (mon.2) 55 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.150836+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: audit 2024-09-10T18:51:11.161201+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: cluster 2024-09-10T18:51:11.649541+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:51:12.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[22591]: cluster 2024-09-10T18:51:11.652488+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:51:12.207 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:11 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:11.963617156Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-10T18:51:12.207 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.059218622Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-10T18:51:12.207 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.101333284Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-10T18:51:12.208 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.126590542Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-10T18:51:12.208 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.15174483Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-10T18:51:12.208 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.177269925Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-10T18:51:12.208 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.20256368Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-10T18:51:12.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: cluster 2024-09-10T18:51:10.452368+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: cluster 2024-09-10T18:51:10.471264+0000 mgr.y (mgr.24526) 58 : cluster [DBG] pgmap v40: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.077655+0000 mon.a (mon.0) 834 : audit [DBG] from='client.? v1:172.21.15.83:0/3758419884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.134658+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.144444+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.149503+0000 mon.c (mon.2) 55 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.150836+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: audit 2024-09-10T18:51:11.161201+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: cluster 2024-09-10T18:51:11.649541+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:51:12.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[18629]: cluster 2024-09-10T18:51:11.652488+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: cluster 2024-09-10T18:51:10.447093+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: cluster 2024-09-10T18:51:10.452368+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: cluster 2024-09-10T18:51:10.471264+0000 mgr.y (mgr.24526) 58 : cluster [DBG] pgmap v40: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.077655+0000 mon.a (mon.0) 834 : audit [DBG] from='client.? v1:172.21.15.83:0/3758419884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.134658+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.144444+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.149503+0000 mon.c (mon.2) 55 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:12.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.150836+0000 mon.c (mon.2) 56 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:12.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: audit 2024-09-10T18:51:11.161201+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:12.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: cluster 2024-09-10T18:51:11.649541+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:51:12.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:11 smithi083 bash[32381]: cluster 2024-09-10T18:51:11.652488+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:51:12.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.231559138Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-10T18:51:12.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.315365521Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-10T18:51:12.484 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.365597572Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-10T18:51:12.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.390984593Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-10T18:51:12.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.416230709Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-10T18:51:12.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.441513761Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-10T18:51:12.485 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.480299529Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-10T18:51:12.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.563915331Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-10T18:51:12.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.619637797Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-10T18:51:12.736 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.645095066Z level=info msg="Executing migration" id="create alert table v1" 2024-09-10T18:51:12.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.682236113Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-10T18:51:12.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.707443966Z level=info msg="Executing migration" id="add index alert state" 2024-09-10T18:51:12.737 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.732683118Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-10T18:51:13.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[22591]: cephadm 2024-09-10T18:51:11.206267+0000 mgr.y (mgr.24526) 59 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:51:13.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[22591]: cephadm 2024-09-10T18:51:11.213629+0000 mgr.y (mgr.24526) 60 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi083 2024-09-10T18:51:13.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[22591]: cluster 2024-09-10T18:51:11.481335+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:51:13.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[22591]: cluster 2024-09-10T18:51:11.486135+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:51:13.001 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.81637287Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-10T18:51:13.001 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.858301937Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-10T18:51:13.001 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.900508866Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-10T18:51:13.001 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.946537579Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-10T18:51:13.001 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.971691399Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-10T18:51:13.002 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:12 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:12.996997265Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.089158325Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.114359541Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.14804021Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.185279801Z level=info msg="Executing migration" id="Add column is_default" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.210454428Z level=info msg="Executing migration" id="Add column frequency" 2024-09-10T18:51:13.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.252406707Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-10T18:51:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[18629]: cephadm 2024-09-10T18:51:11.206267+0000 mgr.y (mgr.24526) 59 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[18629]: cephadm 2024-09-10T18:51:11.213629+0000 mgr.y (mgr.24526) 60 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi083 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[18629]: cluster 2024-09-10T18:51:11.481335+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[18629]: cluster 2024-09-10T18:51:11.486135+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[32381]: cephadm 2024-09-10T18:51:11.206267+0000 mgr.y (mgr.24526) 59 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[32381]: cephadm 2024-09-10T18:51:11.213629+0000 mgr.y (mgr.24526) 60 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi083 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[32381]: cluster 2024-09-10T18:51:11.481335+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:51:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:12 smithi083 bash[32381]: cluster 2024-09-10T18:51:11.486135+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:51:13.593 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.34480145Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-10T18:51:13.593 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.370076252Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-10T18:51:13.593 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.412300946Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-10T18:51:13.594 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.453543653Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-10T18:51:13.594 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.495609114Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-10T18:51:13.594 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.589454347Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-10T18:51:13.854 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.614794336Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-10T18:51:13.854 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.639943906Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-10T18:51:13.854 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.665192908Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-10T18:51:13.854 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.70172268Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-10T18:51:13.854 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.726991598Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-10T18:51:13.855 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.754887645Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-10T18:51:13.855 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.850524496Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-10T18:51:14.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[22591]: cluster 2024-09-10T18:51:12.464483+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:51:14.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[22591]: cluster 2024-09-10T18:51:12.468129+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:51:14.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[22591]: cluster 2024-09-10T18:51:12.472010+0000 mgr.y (mgr.24526) 61 : cluster [DBG] pgmap v41: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:14.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[22591]: cluster 2024-09-10T18:51:12.605455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:51:14.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[22591]: cluster 2024-09-10T18:51:12.608761+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.875793388Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.901078396Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.926306292Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.968292422Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:13 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:13.993677154Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.018937976Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-10T18:51:14.111 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.10663077Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-10T18:51:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[18629]: cluster 2024-09-10T18:51:12.464483+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:51:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[18629]: cluster 2024-09-10T18:51:12.468129+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:51:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[18629]: cluster 2024-09-10T18:51:12.472010+0000 mgr.y (mgr.24526) 61 : cluster [DBG] pgmap v41: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[18629]: cluster 2024-09-10T18:51:12.605455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:51:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[18629]: cluster 2024-09-10T18:51:12.608761+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:51:14.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[32381]: cluster 2024-09-10T18:51:12.464483+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:51:14.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[32381]: cluster 2024-09-10T18:51:12.468129+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:51:14.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[32381]: cluster 2024-09-10T18:51:12.472010+0000 mgr.y (mgr.24526) 61 : cluster [DBG] pgmap v41: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:14.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[32381]: cluster 2024-09-10T18:51:12.605455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:51:14.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:13 smithi083 bash[32381]: cluster 2024-09-10T18:51:12.608761+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:51:14.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.14864153Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-10T18:51:14.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.173893145Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-10T18:51:14.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.215906023Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-10T18:51:14.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.241154418Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-10T18:51:14.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.266556304Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-10T18:51:14.633 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.36087168Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.386310988Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.428256677Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.453559944Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.478790756Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.50414448Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.576484613Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-10T18:51:14.634 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.630089957Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-10T18:51:14.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.663737069Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-10T18:51:14.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.709309215Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-10T18:51:14.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.734653142Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-10T18:51:14.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.759920467Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-10T18:51:14.889 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.843516406Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-10T18:51:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[18629]: cluster 2024-09-10T18:51:13.561770+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:51:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[18629]: cluster 2024-09-10T18:51:13.565662+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:51:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[18629]: audit 2024-09-10T18:51:14.540495+0000 mon.c (mon.2) 57 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:15.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[32381]: cluster 2024-09-10T18:51:13.561770+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:51:15.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[32381]: cluster 2024-09-10T18:51:13.565662+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:51:15.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:14 smithi083 bash[32381]: audit 2024-09-10T18:51:14.540495+0000 mon.c (mon.2) 57 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:15.158 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[22591]: cluster 2024-09-10T18:51:13.561770+0000 osd.5 (osd.5) 37 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:51:15.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[22591]: cluster 2024-09-10T18:51:13.565662+0000 osd.5 (osd.5) 38 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:51:15.159 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[22591]: audit 2024-09-10T18:51:14.540495+0000 mon.c (mon.2) 57 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:15.159 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.885465458Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-10T18:51:15.159 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.910776586Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-10T18:51:15.159 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.93232622Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-10T18:51:15.159 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:14 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.974482762Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-10T18:51:15.160 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:14.999786916Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-10T18:51:15.160 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.024976525Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-10T18:51:15.160 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.104521078Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-10T18:51:15.160 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.155205007Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-10T18:51:15.433 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.20548296Z level=info msg="Executing migration" id="Move region to single row" 2024-09-10T18:51:15.433 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.230867949Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-10T18:51:15.433 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.27018488Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-10T18:51:15.433 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.337222548Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-10T18:51:15.434 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.362377939Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-10T18:51:15.434 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.404415216Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-10T18:51:15.434 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.429689963Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-10T18:51:15.540 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:15 smithi083 systemd[1]: Stopping Ceph alertmanager.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T18:51:15.540 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[66125]: ts=2024-09-10T18:51:15.395Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T18:51:15.694 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.472017818Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-10T18:51:15.694 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.514033525Z level=info msg="Executing migration" id="create test_data table" 2024-09-10T18:51:15.694 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.597773931Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-10T18:51:15.694 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.622952897Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-10T18:51:15.694 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.648327813Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-10T18:51:15.955 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.690439131Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-10T18:51:15.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.732199635Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-10T18:51:15.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.807472919Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-10T18:51:15.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.858076377Z level=info msg="Executing migration" id="create team table" 2024-09-10T18:51:15.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.883422944Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-10T18:51:15.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.925515086Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-10T18:51:16.229 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[22591]: cluster 2024-09-10T18:51:14.472710+0000 mgr.y (mgr.24526) 62 : cluster [DBG] pgmap v42: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:16.229 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[22591]: cluster 2024-09-10T18:51:14.701072+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:51:16.229 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[22591]: cluster 2024-09-10T18:51:14.705018+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.967505303Z level=info msg="Executing migration" id="create team member table" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:15 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:15.990428389Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.082451441Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.107829534Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.133066961Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.174963982Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.200361911Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-10T18:51:16.230 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.225712263Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-10T18:51:16.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[32381]: cluster 2024-09-10T18:51:14.472710+0000 mgr.y (mgr.24526) 62 : cluster [DBG] pgmap v42: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:16.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[32381]: cluster 2024-09-10T18:51:14.701072+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:51:16.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[32381]: cluster 2024-09-10T18:51:14.705018+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:51:16.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[18629]: cluster 2024-09-10T18:51:14.472710+0000 mgr.y (mgr.24526) 62 : cluster [DBG] pgmap v42: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:16.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[18629]: cluster 2024-09-10T18:51:14.701072+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:51:16.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[18629]: cluster 2024-09-10T18:51:14.705018+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:51:16.242 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:15 smithi083 bash[67598]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T18:51:16.242 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[67662]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T18:51:16.242 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:16 smithi083 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@alertmanager.a.service: Succeeded. 2024-09-10T18:51:16.242 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:16 smithi083 systemd[1]: Stopped Ceph alertmanager.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:51:16.242 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:16 smithi083 systemd[1]: Started Ceph alertmanager.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:51:16.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.25100464Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-10T18:51:16.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.33468745Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-10T18:51:16.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.376620594Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-10T18:51:16.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.419438403Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-10T18:51:16.723 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:16.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.486601978Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-10T18:51:16.770 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.71662239Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-10T18:51:17.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.766903126Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-10T18:51:17.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.900537945Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-10T18:51:17.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:16.98519852Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cluster 2024-09-10T18:51:15.481342+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cluster 2024-09-10T18:51:15.484871+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: audit 2024-09-10T18:51:16.140509+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: audit 2024-09-10T18:51:16.149915+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cephadm 2024-09-10T18:51:16.152362+0000 mgr.y (mgr.24526) 63 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cephadm 2024-09-10T18:51:16.164612+0000 mgr.y (mgr.24526) 64 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi165 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cluster 2024-09-10T18:51:16.547803+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:51:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:16 smithi165 bash[22591]: cluster 2024-09-10T18:51:16.552391+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cluster 2024-09-10T18:51:15.481342+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cluster 2024-09-10T18:51:15.484871+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: audit 2024-09-10T18:51:16.140509+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: audit 2024-09-10T18:51:16.149915+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cephadm 2024-09-10T18:51:16.152362+0000 mgr.y (mgr.24526) 63 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:51:17.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cephadm 2024-09-10T18:51:16.164612+0000 mgr.y (mgr.24526) 64 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi165 2024-09-10T18:51:17.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cluster 2024-09-10T18:51:16.547803+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:51:17.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[18629]: cluster 2024-09-10T18:51:16.552391+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:51:17.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cluster 2024-09-10T18:51:15.481342+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:51:17.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cluster 2024-09-10T18:51:15.484871+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: audit 2024-09-10T18:51:16.140509+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: audit 2024-09-10T18:51:16.149915+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cephadm 2024-09-10T18:51:16.152362+0000 mgr.y (mgr.24526) 63 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cephadm 2024-09-10T18:51:16.164612+0000 mgr.y (mgr.24526) 64 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi165 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cluster 2024-09-10T18:51:16.547803+0000 osd.5 (osd.5) 39 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:51:17.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:16 smithi083 bash[32381]: cluster 2024-09-10T18:51:16.552391+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:51:17.592 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:17.593 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:17 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:17.252511359Z level=info msg="Executing migration" id="create tag table" 2024-09-10T18:51:17.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:17 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:17.588983271Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-10T18:51:17.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:17 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:17.661312751Z level=info msg="Executing migration" id="create login attempt table" 2024-09-10T18:51:17.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:17 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:17.741171942Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:17 smithi083 bash[67687]: ts=2024-09-10T18:51:17.958Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:17 smithi083 bash[67687]: ts=2024-09-10T18:51:17.958Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:17 smithi083 bash[67687]: ts=2024-09-10T18:51:17.960Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.83 port=9094 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:17 smithi083 bash[67687]: ts=2024-09-10T18:51:17.963Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[67687]: ts=2024-09-10T18:51:18.003Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[67687]: ts=2024-09-10T18:51:18.005Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[67687]: ts=2024-09-10T18:51:18.009Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T18:51:18.090 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[67687]: ts=2024-09-10T18:51:18.009Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T18:51:18.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.10290533Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-10T18:51:18.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.260142044Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-10T18:51:18.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.335196897Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-10T18:51:18.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[22591]: cluster 2024-09-10T18:51:16.474336+0000 mgr.y (mgr.24526) 65 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:18.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[22591]: cluster 2024-09-10T18:51:17.505203+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:51:18.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[22591]: cluster 2024-09-10T18:51:17.508946+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:51:18.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[18629]: cluster 2024-09-10T18:51:16.474336+0000 mgr.y (mgr.24526) 65 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:18.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[18629]: cluster 2024-09-10T18:51:17.505203+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:51:18.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[18629]: cluster 2024-09-10T18:51:17.508946+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:51:18.429 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[32381]: cluster 2024-09-10T18:51:16.474336+0000 mgr.y (mgr.24526) 65 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:18.429 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[32381]: cluster 2024-09-10T18:51:17.505203+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:51:18.429 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:18 smithi083 bash[32381]: cluster 2024-09-10T18:51:17.508946+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:51:18.775 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.418487564Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-10T18:51:18.775 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.544809479Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-10T18:51:18.775 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.586882679Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-10T18:51:18.775 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.652872853Z level=info msg="Executing migration" id="create user auth table" 2024-09-10T18:51:19.025 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.771592296Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-10T18:51:19.025 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.822334562Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-10T18:51:19.026 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.847363655Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-10T18:51:19.026 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.889851614Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-10T18:51:19.026 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:18 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:18.964940749Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-10T18:51:19.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[22591]: audit 2024-09-10T18:51:17.440059+0000 mgr.y (mgr.24526) 66 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:19.282 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.041099369Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-10T18:51:19.282 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.114981209Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-10T18:51:19.282 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.230675658Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-10T18:51:19.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:19 smithi083 bash[18629]: audit 2024-09-10T18:51:17.440059+0000 mgr.y (mgr.24526) 66 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:19 smithi083 bash[32381]: audit 2024-09-10T18:51:17.440059+0000 mgr.y (mgr.24526) 66 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:19.613 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.378492864Z level=info msg="Executing migration" id="create server_lock table" 2024-09-10T18:51:20.044 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.719903917Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-10T18:51:20.044 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.893070366Z level=info msg="Executing migration" id="create user auth token table" 2024-09-10T18:51:20.044 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:19 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:19.960365628Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-10T18:51:20.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- 172.21.15.83:0/3230164212 >> v1:172.21.15.83:6789/0 conn(0x7f95e810b130 legacy=0x7f95e810d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- 172.21.15.83:0/3230164212 shutdown_connections 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- 172.21.15.83:0/3230164212 >> 172.21.15.83:0/3230164212 conn(0x7f95e80fa370 msgr2=0x7f95e80fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- 172.21.15.83:0/3230164212 shutdown_connections 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- 172.21.15.83:0/3230164212 wait complete. 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 librados: starting msgr at 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 librados: starting objecter 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 Processor -- start 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- start start 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 librados: setting wanted keys 2024-09-10T18:51:20.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 librados: calling monclient init 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95e81040a0 con 0x7f95e810b130 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95e811f630 con 0x7f95e81075a0 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95eccee640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95e81207d0 con 0x7f95e8104810 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95e659c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f95e8104810 0x7f95e81034f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60566/0 (socket says 172.21.15.83:60566) 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95e659c640 1 -- 172.21.15.83:0/4046524166 learned_addr learned my addr 172.21.15.83:0/4046524166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter ms_handle_connect 0x7f95e8104810 2024-09-10T18:51:20.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter ms_handle_connect 0x7f95e81075a0 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820353328 0 0) 0x7f95e81207d0 con 0x7f95e8104810 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d40034e0 con 0x7f95e8104810 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter ms_handle_connect 0x7f95e810b130 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.237+0000 7f95deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799448642 0 0) 0x7f95e811f630 con 0x7f95e81075a0 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e81207d0 con 0x7f95e81075a0 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850357884 0 0) 0x7f95e81040a0 con 0x7f95e810b130 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e811f630 con 0x7f95e810b130 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4017725432 0 0) 0x7f95d40034e0 con 0x7f95e8104810 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e81040a0 con 0x7f95e8104810 2024-09-10T18:51:20.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f95e0001980 con 0x7f95e8104810 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510987781 0 0) 0x7f95e811f630 con 0x7f95e810b130 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95d40034e0 con 0x7f95e810b130 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f95d8002310 con 0x7f95e810b130 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014704600 0 0) 0x7f95e81207d0 con 0x7f95e81075a0 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e811f630 con 0x7f95e81075a0 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f95d0001840 con 0x7f95e81075a0 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1015223800 0 0) 0x7f95e81040a0 con 0x7f95e8104810 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 >> v1:172.21.15.165:6789/0 conn(0x7f95e81075a0 legacy=0x7f95e81197d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 >> v1:172.21.15.83:6789/0 conn(0x7f95e810b130 legacy=0x7f95e811cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e81219b0 con 0x7f95e8104810 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e811f820 con 0x7f95e8104810 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95eccee640 10 client.24587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:20.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e811fc70 con 0x7f95e8104810 2024-09-10T18:51:20.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95eccee640 1 librados: init done 2024-09-10T18:51:20.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f95e0002d00 con 0x7f95e8104810 2024-09-10T18:51:20.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f95e0005b30 con 0x7f95e8104810 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c8005010 con 0x7f95e8104810 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f95e0001f60 con 0x7f95e8104810 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(101..101 src has 49..101) v4 ==== 6284+0+0 (unknown 4267342977 0 0) 0x7f95e0089f90 con 0x7f95e8104810 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 10 client.24587.objecter ms_dispatch 0x7f95e8000b90 osd_map(101..101 src has 49..101) v4 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 3 client.24587.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.241+0000 7f95deffd640 3 client.24587.objecter handle_osd_map decoding full epoch 101 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.245+0000 7f95deffd640 20 client.24587.objecter dump_active .. 0 homeless 2024-09-10T18:51:20.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.245+0000 7f95deffd640 10 client.24587.objecter ms_handle_connect 0x7f95d40717b0 2024-09-10T18:51:20.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.249+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f95e00567e0 con 0x7f95e8104810 2024-09-10T18:51:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:20 smithi083 bash[18629]: cluster 2024-09-10T18:51:18.475000+0000 mgr.y (mgr.24526) 67 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:51:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:20 smithi083 bash[32381]: cluster 2024-09-10T18:51:18.475000+0000 mgr.y (mgr.24526) 67 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:51:20.291 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:19 smithi083 bash[67687]: ts=2024-09-10T18:51:19.963Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000626571s 2024-09-10T18:51:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[22591]: cluster 2024-09-10T18:51:18.475000+0000 mgr.y (mgr.24526) 67 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:51:20.365 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.040472818Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-10T18:51:20.365 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.125149778Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-10T18:51:20.365 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.178839907Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-10T18:51:20.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.417+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95c80052c0 con 0x7f95e8104810 2024-09-10T18:51:20.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.417+0000 7f95deffd640 1 -- 172.21.15.83:0/4046524166 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+19422 (unknown 901664539 0 179534950) 0x7f95e0056600 con 0x7f95e8104810 2024-09-10T18:51:20.422 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:51:20.422 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":101,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","created":"2024-09-10T18:38:38.475667+0000","modified":"2024-09-10T18:51:09.909233+0000","last_up_change":"2024-09-10T18:48:34.513026+0000","last_in_change":"2024-09-10T18:48:07.544682+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:44:06.952078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T18:49:16.239173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:49:18.096790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:49:20.070555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"datapool","create_time":"2024-09-10T18:49:20.615000+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":80,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:49:22.582739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"35ab8328-74f1-4af8-aa84-797914199445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6801","nonce":2535527483}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6802","nonce":2535527483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6804","nonce":2535527483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6803","nonce":2535527483}]},"public_addr":"172.21.15.83:6801/2535527483","cluster_addr":"172.21.15.83:6802/2535527483","heartbeat_back_addr":"172.21.15.83:6804/2535527483","heartbeat_front_addr":"172.21.15.83:6803/2535527483","state":["exists","up"]},{"osd":1,"uuid":"8e1d1072-767b-49fb-b2f2-fce60263e2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6805","nonce":2309624637}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6806","nonce":2309624637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6808","nonce":2309624637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6807","nonce":2309624637}]},"public_addr":"172.21.15.83:6805/2309624637","cluster_addr":"172.21.15.83:6806/2309624637","heartbeat_back_addr":"172.21.15.83:6808/2309624637","heartbeat_front_addr":"172.21.15.83:6807/2309624637","state":["exists","up"]},{"osd":2,"uuid":"6193b508-03d9-458a-b49f-6b2e24283282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6809","nonce":386094220}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6810","nonce":386094220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6812","nonce":386094220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6811","nonce":386094220}]},"public_addr":"172.21.15.83:6809/386094220","cluster_addr":"172.21.15.83:6810/386094220","heartbeat_back_addr":"172.21.15.83:6812/386094220","heartbeat_front_addr":"172.21.15.83:6811/386094220","state":["exists","up"]},{"osd":3,"uuid":"a28f94ff-2fff-458e-8b69-c5c65d720d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6813","nonce":1279688021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6814","nonce":1279688021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6816","nonce":1279688021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6815","nonce":1279688021}]},"public_addr":"172.21.15.83:6813/1279688021","cluster_addr":"172.21.15.83:6814/1279688021","heartbeat_back_addr":"172.21.15.83:6816/1279688021","heartbeat_front_addr":"172.21.15.83:6815/1279688021","state":["exists","up"]},{"osd":4,"uuid":"a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":3225515265}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":3225515265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":3225515265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":3225515265}]},"public_addr":"172.21.15.165:6800/3225515265","cluster_addr":"172.21.15.165:6801/3225515265","heartbeat_back_addr":"172.21.15.165:6803/3225515265","heartbeat_front_addr":"172.21.15.165:6802/3225515265","state":["exists","up"]},{"osd":5,"uuid":"10543223-db5a-493b-88f3-8705f2b31a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":571173777}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":571173777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":571173777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":571173777}]},"public_addr":"172.21.15.165:6804/571173777","cluster_addr":"172.21.15.165:6805/571173777","heartbeat_back_addr":"172.21.15.165:6807/571173777","heartbeat_front_addr":"172.21.15.165:6806/571173777","state":["exists","up"]},{"osd":6,"uuid":"fcea0ca4-4521-4161-b407-9b1fab1a2ae2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":1317922936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":1317922936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":1317922936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":1317922936}]},"public_addr":"172.21.15.165:6808/1317922936","cluster_addr":"172.21.15.165:6809/1317922936","heartbeat_back_addr":"172.21.15.165:6811/1317922936","heartbeat_front_addr":"172.21.15.165:6810/1317922936","state":["exists","up"]},{"osd":7,"uuid":"8531d286-5ebb-4c96-a026-63d8a17cb21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1610151650}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1610151650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1610151650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1610151650}]},"public_addr":"172.21.15.165:6812/1610151650","cluster_addr":"172.21.15.165:6813/1610151650","heartbeat_back_addr":"172.21.15.165:6815/1610151650","heartbeat_front_addr":"172.21.15.165:6814/1610151650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:57.418119+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:49:58.459425+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:24.641962+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:24.307097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:50.790465+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:50:55.400840+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:51:09.268836+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:51:08.423724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/3645093631":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/3275901971":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3555551632":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/3076437878":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/4112054239":"2024-09-11T18:39:06.336264+0000","172.21.15.83:6800/2452184936":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/2659452540":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/867303943":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/1156549307":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/1687990153":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1246256198":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/3225538750":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/2630800463":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1423545952":"2024-09-11T18:50:14.421453+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 10 librados: watch_flush enter 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 10 librados: watch_flush exit 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 20 client.24587.objecter shutdown clearing up homeless session... 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 10 client.24587.objecter successfully canceled tick 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 >> v1:172.21.15.83:6800/3071793735 conn(0x7f95d40717b0 legacy=0x7f95d4073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 >> v1:172.21.15.83:6790/0 conn(0x7f95e8104810 legacy=0x7f95e81034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 shutdown_connections 2024-09-10T18:51:20.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.421+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 >> 172.21.15.83:0/4046524166 conn(0x7f95e80fa370 msgr2=0x7f95e80fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:20.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.425+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 shutdown_connections 2024-09-10T18:51:20.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.425+0000 7f95eccee640 1 -- 172.21.15.83:0/4046524166 wait complete. 2024-09-10T18:51:20.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:20.425+0000 7f95eccee640 1 librados: shutdown 2024-09-10T18:51:20.593 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.362356184Z level=info msg="Executing migration" id="create cache_data table" 2024-09-10T18:51:20.773 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.589976006Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-10T18:51:21.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.76968726Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-10T18:51:21.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.853411587Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-10T18:51:21.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:20 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:20.942207925Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-10T18:51:21.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T18:51:21.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T18:51:21.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T18:51:21.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T18:51:21.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T18:51:21.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T18:51:21.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T18:51:21.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T18:51:21.467 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.320838553Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-10T18:51:21.468 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[22591]: cluster 2024-09-10T18:51:20.399677+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T18:51:21.468 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[22591]: cluster 2024-09-10T18:51:20.402912+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T18:51:21.468 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[22591]: audit 2024-09-10T18:51:20.420938+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.83:0/4046524166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:21.468 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:51:21.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[18629]: cluster 2024-09-10T18:51:20.399677+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T18:51:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[18629]: cluster 2024-09-10T18:51:20.402912+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T18:51:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[18629]: audit 2024-09-10T18:51:20.420938+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.83:0/4046524166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[32381]: cluster 2024-09-10T18:51:20.399677+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T18:51:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[32381]: cluster 2024-09-10T18:51:20.402912+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T18:51:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:21 smithi083 bash[32381]: audit 2024-09-10T18:51:20.420938+0000 mon.c (mon.2) 58 : audit [DBG] from='client.? v1:172.21.15.83:0/4046524166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T18:51:21.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.463835317Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-10T18:51:21.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.592983508Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-10T18:51:21.763 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.673519367Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-10T18:51:22.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.759450082Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-10T18:51:22.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.824475708Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-10T18:51:22.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.900236664Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-10T18:51:22.056 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:21 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:21.966422121Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-10T18:51:22.057 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.008809238Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-10T18:51:22.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.105838334Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-10T18:51:22.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.179054589Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-10T18:51:22.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.227210128Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-10T18:51:22.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.353245943Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-10T18:51:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[22591]: cluster 2024-09-10T18:51:20.476169+0000 mgr.y (mgr.24526) 68 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[22591]: cluster 2024-09-10T18:51:21.379777+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:51:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[22591]: cluster 2024-09-10T18:51:21.383316+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:51:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[22591]: cluster 2024-09-10T18:51:21.662522+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:51:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[22591]: cluster 2024-09-10T18:51:21.666836+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:51:22.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[18629]: cluster 2024-09-10T18:51:20.476169+0000 mgr.y (mgr.24526) 68 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[18629]: cluster 2024-09-10T18:51:21.379777+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[18629]: cluster 2024-09-10T18:51:21.383316+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[18629]: cluster 2024-09-10T18:51:21.662522+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[18629]: cluster 2024-09-10T18:51:21.666836+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[32381]: cluster 2024-09-10T18:51:20.476169+0000 mgr.y (mgr.24526) 68 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[32381]: cluster 2024-09-10T18:51:21.379777+0000 osd.0 (osd.0) 51 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:51:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[32381]: cluster 2024-09-10T18:51:21.383316+0000 osd.0 (osd.0) 52 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:51:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[32381]: cluster 2024-09-10T18:51:21.662522+0000 osd.5 (osd.5) 41 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:51:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:22 smithi083 bash[32381]: cluster 2024-09-10T18:51:21.666836+0000 osd.5 (osd.5) 42 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:51:22.831 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.434066219Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-10T18:51:22.831 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.551928701Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-10T18:51:23.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.827342464Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-10T18:51:23.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:22 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:22.993019851Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-10T18:51:23.448 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 systemd[1]: Stopping Ceph prometheus.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T18:51:23.448 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.156798228Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-10T18:51:23.449 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.232507896Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-10T18:51:23.449 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.316947705Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-10T18:51:23.449 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.400715614Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-10T18:51:23.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.444570641Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-10T18:51:23.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.51221712Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-10T18:51:23.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.562826576Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-10T18:51:23.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.692503609Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-10T18:51:23.814 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T18:51:23.814 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T18:51:23.814 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.491Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.492Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.492Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.492Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T18:51:23.815 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[55673]: ts=2024-09-10T18:51:23.492Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T18:51:24.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:23 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:23.809823111Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-10T18:51:24.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.002117174Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-10T18:51:24.441 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[58241]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T18:51:24.441 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.282195311Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-10T18:51:24.441 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[22591]: cluster 2024-09-10T18:51:22.476839+0000 mgr.y (mgr.24526) 69 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:24.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[22591]: cluster 2024-09-10T18:51:22.699946+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.c scrub starts 2024-09-10T18:51:24.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[22591]: cluster 2024-09-10T18:51:22.704046+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.c scrub ok 2024-09-10T18:51:24.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18629]: cluster 2024-09-10T18:51:22.476839+0000 mgr.y (mgr.24526) 69 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18629]: cluster 2024-09-10T18:51:22.699946+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.c scrub starts 2024-09-10T18:51:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18629]: cluster 2024-09-10T18:51:22.704046+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.c scrub ok 2024-09-10T18:51:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[32381]: cluster 2024-09-10T18:51:22.476839+0000 mgr.y (mgr.24526) 69 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[32381]: cluster 2024-09-10T18:51:22.699946+0000 osd.3 (osd.3) 43 : cluster [DBG] 6.c scrub starts 2024-09-10T18:51:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[32381]: cluster 2024-09-10T18:51:22.704046+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.c scrub ok 2024-09-10T18:51:24.778 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[58305]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T18:51:24.778 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:24 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@prometheus.a.service: Succeeded. 2024-09-10T18:51:24.778 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:24 smithi165 systemd[1]: Stopped Ceph prometheus.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:51:24.778 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:24 smithi165 systemd[1]: Started Ceph prometheus.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T18:51:24.778 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.43752397Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-10T18:51:24.778 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.563224849Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-10T18:51:24.779 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.605829001Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-10T18:51:24.779 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.656124297Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-10T18:51:25.032 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18896]: [10/Sep/2024:18:51:24] ENGINE Bus STOPPING 2024-09-10T18:51:25.033 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18896]: [10/Sep/2024:18:51:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:51:25.033 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18896]: [10/Sep/2024:18:51:24] ENGINE Bus STOPPED 2024-09-10T18:51:25.033 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:24 smithi083 bash[18896]: [10/Sep/2024:18:51:24] ENGINE Bus STARTING 2024-09-10T18:51:25.045 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Serving on http://:::9283 2024-09-10T18:51:25.045 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STARTED 2024-09-10T18:51:25.045 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STOPPING 2024-09-10T18:51:25.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.774264851Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-10T18:51:25.113 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:24 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:24.950534897Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STOPPED 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STARTING 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Serving on http://:::9283 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STARTED 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STOPPING 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: cluster 2024-09-10T18:51:23.679804+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:51:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: cluster 2024-09-10T18:51:23.683158+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.638626+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.649586+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.658044+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.660611+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.667595+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.681265+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.689850+0000 mon.c (mon.2) 62 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.698948+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.712934+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.717723+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.727108+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18629]: audit 2024-09-10T18:51:24.734172+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: cluster 2024-09-10T18:51:23.679804+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: cluster 2024-09-10T18:51:23.683158+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.638626+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.649586+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.658044+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.660611+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.667595+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.681265+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.689850+0000 mon.c (mon.2) 62 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.698948+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.712934+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:25.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.717723+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:25.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.727108+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[32381]: audit 2024-09-10T18:51:24.734172+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:25.519 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:25.268014641Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-10T18:51:25.520 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:25.372393545Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: cluster 2024-09-10T18:51:23.679804+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: cluster 2024-09-10T18:51:23.683158+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.638626+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.649586+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.658044+0000 mon.c (mon.2) 59 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:25.520 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.660611+0000 mon.c (mon.2) 60 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.667595+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.681265+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.689850+0000 mon.c (mon.2) 62 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.698948+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.712934+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.717723+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.727108+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:25.521 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[22591]: audit 2024-09-10T18:51:24.734172+0000 mon.c (mon.2) 65 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:25.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:25.516082159Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-10T18:51:25.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:25.669752095Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-10T18:51:26.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:51:26.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STOPPED 2024-09-10T18:51:26.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STARTING 2024-09-10T18:51:26.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Serving on http://:::9283 2024-09-10T18:51:26.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:25 smithi083 bash[18896]: [10/Sep/2024:18:51:25] ENGINE Bus STARTED 2024-09-10T18:51:26.066 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.067 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.067 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.070 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.070 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.074 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.075 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.075 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:26.256 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:25 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:25.982522397Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-10T18:51:26.257 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.099279725Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-10T18:51:26.257 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: cluster 2024-09-10T18:51:24.477517+0000 mgr.y (mgr.24526) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:26.257 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.658880+0000 mgr.y (mgr.24526) 71 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:26.257 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.661181+0000 mgr.y (mgr.24526) 72 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:26.258 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.682092+0000 mgr.y (mgr.24526) 73 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:26.258 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.690653+0000 mgr.y (mgr.24526) 74 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:26.258 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.713751+0000 mgr.y (mgr.24526) 75 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:26.258 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[22591]: audit 2024-09-10T18:51:24.718551+0000 mgr.y (mgr.24526) 76 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:26.524 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.2527315Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-10T18:51:26.525 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.370271434Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-10T18:51:26.525 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.45029192Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: cluster 2024-09-10T18:51:24.477517+0000 mgr.y (mgr.24526) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.658880+0000 mgr.y (mgr.24526) 71 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.661181+0000 mgr.y (mgr.24526) 72 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.682092+0000 mgr.y (mgr.24526) 73 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.690653+0000 mgr.y (mgr.24526) 74 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.713751+0000 mgr.y (mgr.24526) 75 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[18629]: audit 2024-09-10T18:51:24.718551+0000 mgr.y (mgr.24526) 76 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: cluster 2024-09-10T18:51:24.477517+0000 mgr.y (mgr.24526) 70 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.658880+0000 mgr.y (mgr.24526) 71 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T18:51:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.661181+0000 mgr.y (mgr.24526) 72 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi083.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T18:51:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.682092+0000 mgr.y (mgr.24526) 73 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T18:51:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.690653+0000 mgr.y (mgr.24526) 74 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T18:51:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.713751+0000 mgr.y (mgr.24526) 75 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T18:51:26.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:26 smithi083 bash[32381]: audit 2024-09-10T18:51:24.718551+0000 mgr.y (mgr.24526) 76 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T18:51:26.778 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.520995086Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-10T18:51:26.778 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.621546746Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-10T18:51:26.778 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.709640033Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.774257126Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.864653998Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.890078427Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:26.940768547Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.000870171Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.087282247Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-10T18:51:27.091 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.934Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.934Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.934Z caller=main.go:570 level=info host_details="(Linux 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 smithi165 (none))" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.935Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.935Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.939Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.940Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.941Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.941Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.944Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.944Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.264µs 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.944Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-10T18:51:27.092 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.945Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.947Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.947Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.956µs wal_replay_duration=3.129997ms wbl_replay_duration=142ns total_replay_duration=3.196013ms 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.949Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.949Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.949Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.978Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.721644ms db_storage=1.25µs remote_storage=1.58µs web_handler=568ns query_engine=1.056µs scrape=531.849µs scrape_sd=51.622µs notify=33.867µs notify_sd=9.914µs rules=27.5706ms tracing=10.584µs 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.978Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-10T18:51:27.093 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 18:51:26 smithi165 bash[58330]: ts=2024-09-10T18:51:26.978Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T18:51:27.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[22591]: cluster 2024-09-10T18:51:25.624030+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:51:27.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[22591]: cluster 2024-09-10T18:51:25.626682+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:51:27.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[22591]: cluster 2024-09-10T18:51:26.358498+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:27.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[22591]: cluster 2024-09-10T18:51:26.358612+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:27.343 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.112525978Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-10T18:51:27.343 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.13779322Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-10T18:51:27.343 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.179831445Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-10T18:51:27.344 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.222030214Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-10T18:51:27.344 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.247315082Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-10T18:51:27.344 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.339294678Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[18629]: cluster 2024-09-10T18:51:25.624030+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[18629]: cluster 2024-09-10T18:51:25.626682+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[18629]: cluster 2024-09-10T18:51:26.358498+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[18629]: cluster 2024-09-10T18:51:26.358612+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[32381]: cluster 2024-09-10T18:51:25.624030+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[32381]: cluster 2024-09-10T18:51:25.626682+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:51:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[32381]: cluster 2024-09-10T18:51:26.358498+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:27.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[32381]: cluster 2024-09-10T18:51:26.358612+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:27.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:27.613 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.381431124Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-10T18:51:27.613 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.423307681Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-10T18:51:27.613 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.448500698Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-10T18:51:27.614 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.49072459Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-10T18:51:27.614 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.574409368Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-10T18:51:27.880 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.616378443Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-10T18:51:27.880 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.658457505Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-10T18:51:27.880 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.700511209Z level=info msg="Executing migration" id="create alert_image table" 2024-09-10T18:51:27.880 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.725928264Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-10T18:51:27.880 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.817771432Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-10T18:51:28.141 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.876460191Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-10T18:51:28.141 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.901826722Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-10T18:51:28.141 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.927004527Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T18:51:28.141 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.927378044Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T18:51:28.141 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.969088821Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:27 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:27.99435525Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.086696371Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.087179591Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.087620017Z level=info msg="alerts found to migrate" alerts=0 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.112067953Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-10T18:51:28.142 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.137417667Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-10T18:51:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[18629]: cluster 2024-09-10T18:51:26.478983+0000 mgr.y (mgr.24526) 77 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[18629]: cluster 2024-09-10T18:51:27.210462+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-10T18:51:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[18629]: audit 2024-09-10T18:51:27.451591+0000 mgr.y (mgr.24526) 78 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[32381]: cluster 2024-09-10T18:51:26.478983+0000 mgr.y (mgr.24526) 77 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[32381]: cluster 2024-09-10T18:51:27.210462+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-10T18:51:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:28 smithi083 bash[32381]: audit 2024-09-10T18:51:27.451591+0000 mgr.y (mgr.24526) 78 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:28.291 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 18:51:27 smithi083 bash[67687]: ts=2024-09-10T18:51:27.966Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002983228s 2024-09-10T18:51:28.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[22591]: cluster 2024-09-10T18:51:26.478983+0000 mgr.y (mgr.24526) 77 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:28.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[22591]: cluster 2024-09-10T18:51:27.210462+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-09-10T18:51:28.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[22591]: audit 2024-09-10T18:51:27.451591+0000 mgr.y (mgr.24526) 78 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:28.410 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.162565781Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-10T18:51:28.410 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.187853506Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-10T18:51:28.411 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.229984629Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-10T18:51:28.411 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.255210108Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-10T18:51:28.411 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.338907585Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-10T18:51:28.411 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.380946142Z level=info msg="Executing migration" id="create data_keys table" 2024-09-10T18:51:28.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.406338549Z level=info msg="Executing migration" id="create secrets table" 2024-09-10T18:51:28.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.452111754Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-10T18:51:28.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.510743476Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-10T18:51:28.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.602929987Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-10T18:51:28.674 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.628194899Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.670461593Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.729153617Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.754512518Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.838119759Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.863411312Z level=info msg="Executing migration" id="create permission table" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.913799012Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-10T18:51:28.968 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.964206222Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:28 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:28.989459204Z level=info msg="Executing migration" id="create role table" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.014792388Z level=info msg="Executing migration" id="add column display_name" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.098488637Z level=info msg="Executing migration" id="add column group_name" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.140622919Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.165835851Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.191142179Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-10T18:51:29.237 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[22591]: cluster 2024-09-10T18:51:28.479711+0000 mgr.y (mgr.24526) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:29.237 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[22591]: cluster 2024-09-10T18:51:28.790117+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:51:29.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[22591]: cluster 2024-09-10T18:51:28.793774+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.233267557Z level=info msg="Executing migration" id="create team role table" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.258578332Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.342296862Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.36756511Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.392830785Z level=info msg="Executing migration" id="create user role table" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.434903811Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.460234072Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-10T18:51:29.506 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.502296262Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-10T18:51:29.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[32381]: cluster 2024-09-10T18:51:28.479711+0000 mgr.y (mgr.24526) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[32381]: cluster 2024-09-10T18:51:28.790117+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:51:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[32381]: cluster 2024-09-10T18:51:28.793774+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:51:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[18629]: cluster 2024-09-10T18:51:28.479711+0000 mgr.y (mgr.24526) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[18629]: cluster 2024-09-10T18:51:28.790117+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:51:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:29 smithi083 bash[18629]: cluster 2024-09-10T18:51:28.793774+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.577579905Z level=info msg="Executing migration" id="create builtin role table" 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.602806357Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.628159502Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.65343268Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.695686393Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-10T18:51:29.829 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.741969253Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-10T18:51:29.830 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.825502687Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-10T18:51:30.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.876002734Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-10T18:51:30.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.926331371Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-10T18:51:30.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.968404689Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-10T18:51:30.081 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:29 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:29.993652995Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-10T18:51:30.333 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.077546111Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-10T18:51:30.333 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.119593128Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-10T18:51:30.333 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.144830554Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-10T18:51:30.334 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.170129568Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-10T18:51:30.334 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.195417112Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-10T18:51:30.334 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.237760427Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:28.590972+0000 osd.3 (osd.3) 49 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:28.596662+0000 osd.3 (osd.3) 50 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:29.400523+0000 osd.7 (osd.7) 47 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:29.405888+0000 osd.7 (osd.7) 48 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: audit 2024-09-10T18:51:29.541233+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:29.822293+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:51:30.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[22591]: cluster 2024-09-10T18:51:29.827598+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:51:30.582 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:28.590972+0000 osd.3 (osd.3) 49 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:28.596662+0000 osd.3 (osd.3) 50 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:29.400523+0000 osd.7 (osd.7) 47 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:29.405888+0000 osd.7 (osd.7) 48 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: audit 2024-09-10T18:51:29.541233+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:29.822293+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[18629]: cluster 2024-09-10T18:51:29.827598+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:28.590972+0000 osd.3 (osd.3) 49 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:28.596662+0000 osd.3 (osd.3) 50 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:29.400523+0000 osd.7 (osd.7) 47 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:51:30.583 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:29.405888+0000 osd.7 (osd.7) 48 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:51:30.584 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: audit 2024-09-10T18:51:29.541233+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:30.584 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:29.822293+0000 osd.4 (osd.4) 53 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:51:30.584 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:30 smithi083 bash[32381]: cluster 2024-09-10T18:51:29.827598+0000 osd.4 (osd.4) 54 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.329689858Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.371809286Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.397152933Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.430825436Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.472787583Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.498055148Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-10T18:51:30.585 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.581709912Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-10T18:51:30.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.607108341Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-10T18:51:30.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.649136008Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-10T18:51:30.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.691262032Z level=info msg="Executing migration" id="add correlation config column" 2024-09-10T18:51:30.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.733386667Z level=info msg="Executing migration" id="create entity_events table" 2024-09-10T18:51:30.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.810126419Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-10T18:51:31.162 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.884053537Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:51:31.162 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.884341292Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:51:31.162 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.926242975Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.926655329Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.968434037Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:30 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:30.998854824Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.091028675Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.133229387Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T18:51:31.163 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.15858601Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-10T18:51:31.414 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.20047169Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T18:51:31.414 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.242415302Z level=info msg="Executing migration" id="Drop public config table" 2024-09-10T18:51:31.414 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.33480156Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-10T18:51:31.414 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.360046838Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-10T18:51:31.414 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.385229227Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T18:51:31.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.41059032Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-10T18:51:31.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.435971646Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-10T18:51:31.573 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.486319109Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-10T18:51:31.666 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.570072009Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-10T18:51:31.667 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.595496073Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-10T18:51:31.667 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.620608186Z level=info msg="Executing migration" id="add share column" 2024-09-10T18:51:31.918 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[22591]: cluster 2024-09-10T18:51:30.481124+0000 mgr.y (mgr.24526) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:31.918 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.662977838Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-10T18:51:31.918 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.713365777Z level=info msg="Executing migration" id="create file table" 2024-09-10T18:51:31.918 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.739140697Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-10T18:51:31.919 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.822557857Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-10T18:51:31.919 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.864575886Z level=info msg="Executing migration" id="create file_meta table" 2024-09-10T18:51:31.919 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.889991811Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-10T18:51:32.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:31 smithi083 bash[18629]: cluster 2024-09-10T18:51:30.481124+0000 mgr.y (mgr.24526) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:31 smithi083 bash[32381]: cluster 2024-09-10T18:51:30.481124+0000 mgr.y (mgr.24526) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.914984868Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.940411497Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:31 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:31.982401814Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.066160311Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.113143255Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-10T18:51:32.184 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.138550383Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-10T18:51:32.436 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.180558205Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-10T18:51:32.436 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.222587963Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-10T18:51:32.436 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.247669025Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-10T18:51:32.437 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.339937621Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-10T18:51:32.437 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.365147212Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-10T18:51:32.437 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.407313967Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-10T18:51:32.437 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.432627978Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-10T18:51:32.696 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.457847378Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-10T18:51:32.697 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.516612455Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-10T18:51:32.697 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.583721192Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-10T18:51:32.697 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.625650896Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-10T18:51:32.697 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.667758636Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-10T18:51:32.697 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.692872422Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-10T18:51:32.955 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.718279587Z level=info msg="Executing migration" id="create folder table" 2024-09-10T18:51:32.955 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.743481095Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-10T18:51:32.955 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.827136291Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-10T18:51:32.955 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.869263967Z level=info msg="Executing migration" id="Update folder title length" 2024-09-10T18:51:32.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.894612023Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-10T18:51:32.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=migrator t=2024-09-10T18:51:32.919826465Z level=info msg="migrations completed" performed=485 skipped=0 duration=27.541906611s 2024-09-10T18:51:32.956 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[57601]: logger=sqlstore t=2024-09-10T18:51:32.92122654Z level=info msg="Created default organization" 2024-09-10T18:51:32.988 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:32] "GET /metrics HTTP/1.1" 200 220056 "" "Prometheus/2.43.0" 2024-09-10T18:51:33.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:32 smithi083 bash[18629]: audit 2024-09-10T18:51:31.953690+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:32 smithi083 bash[18629]: audit 2024-09-10T18:51:31.963781+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:32 smithi083 bash[32381]: audit 2024-09-10T18:51:31.953690+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:32 smithi083 bash[32381]: audit 2024-09-10T18:51:31.963781+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[22591]: audit 2024-09-10T18:51:31.953690+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:32 smithi165 bash[22591]: audit 2024-09-10T18:51:31.963781+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:33.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=plugin.loader t=2024-09-10T18:51:33.022388168Z level=info msg="Plugin registered" pluginID=input 2024-09-10T18:51:33.363 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=plugin.loader t=2024-09-10T18:51:33.051207197Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-10T18:51:33.364 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=plugin.loader t=2024-09-10T18:51:33.051235459Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-10T18:51:33.364 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=secrets t=2024-09-10T18:51:33.051283188Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-10T18:51:33.364 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=query_data t=2024-09-10T18:51:33.053170279Z level=info msg="Query Service initialization" 2024-09-10T18:51:33.364 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[57601]: logger=live.push_http t=2024-09-10T18:51:33.057574872Z level=info msg="Live Push Gateway initialization" 2024-09-10T18:51:34.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[22591]: cluster 2024-09-10T18:51:32.481749+0000 mgr.y (mgr.24526) 81 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:34.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[22591]: cluster 2024-09-10T18:51:32.826366+0000 osd.4 (osd.4) 55 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:51:34.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:33 smithi165 bash[22591]: cluster 2024-09-10T18:51:32.830639+0000 osd.4 (osd.4) 56 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:51:34.215 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=infra.usagestats.collector t=2024-09-10T18:51:34.035580013Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-10T18:51:34.215 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=provisioning.datasources t=2024-09-10T18:51:34.036369328Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-10T18:51:34.215 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=provisioning.datasources t=2024-09-10T18:51:34.068948102Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-10T18:51:34.215 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=provisioning.datasources t=2024-09-10T18:51:34.139940342Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-10T18:51:34.215 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=provisioning.alerting t=2024-09-10T18:51:34.205532311Z level=info msg="starting to provision alerting" 2024-09-10T18:51:34.216 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=provisioning.alerting t=2024-09-10T18:51:34.205599583Z level=info msg="finished to provision alerting" 2024-09-10T18:51:34.216 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=ngalert.state.manager t=2024-09-10T18:51:34.205903578Z level=info msg="Warming state cache for startup" 2024-09-10T18:51:34.216 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=grafanaStorageLogger t=2024-09-10T18:51:34.20624318Z level=info msg="storage starting" 2024-09-10T18:51:34.216 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=http.server t=2024-09-10T18:51:34.211357415Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-10T18:51:34.251 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[18629]: cluster 2024-09-10T18:51:32.481749+0000 mgr.y (mgr.24526) 81 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:34.252 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[18629]: cluster 2024-09-10T18:51:32.826366+0000 osd.4 (osd.4) 55 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:51:34.252 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[18629]: cluster 2024-09-10T18:51:32.830639+0000 osd.4 (osd.4) 56 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:51:34.252 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[32381]: cluster 2024-09-10T18:51:32.481749+0000 mgr.y (mgr.24526) 81 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:34.252 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[32381]: cluster 2024-09-10T18:51:32.826366+0000 osd.4 (osd.4) 55 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:51:34.252 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:33 smithi083 bash[32381]: cluster 2024-09-10T18:51:32.830639+0000 osd.4 (osd.4) 56 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:51:34.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T18:51:34.217248638Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:51:34.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=ngalert.state.manager t=2024-09-10T18:51:34.272099963Z level=info msg="State cache has been initialized" states=0 duration=66.195363ms 2024-09-10T18:51:34.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=ticker t=2024-09-10T18:51:34.272240892Z level=info msg=starting first_tick=2024-09-10T18:51:40Z 2024-09-10T18:51:34.504 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=ngalert.multiorg.alertmanager t=2024-09-10T18:51:34.272241025Z level=info msg="starting MultiOrg Alertmanager" 2024-09-10T18:51:34.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T18:51:34.500359061Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T18:51:34.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T18:51:34.511722776Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-10T18:51:34.863 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T18:51:34.523246262Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-10T18:51:35.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:34 smithi083 bash[18629]: cluster 2024-09-10T18:51:33.448170+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:51:35.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:34 smithi083 bash[18629]: cluster 2024-09-10T18:51:33.452490+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:51:35.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:34 smithi083 bash[18629]: cluster 2024-09-10T18:51:33.611819+0000 osd.3 (osd.3) 51 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:51:35.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:34 smithi083 bash[18629]: cluster 2024-09-10T18:51:33.617390+0000 osd.3 (osd.3) 52 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:51:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[22591]: cluster 2024-09-10T18:51:33.448170+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:51:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[22591]: cluster 2024-09-10T18:51:33.452490+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:51:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[22591]: cluster 2024-09-10T18:51:33.611819+0000 osd.3 (osd.3) 51 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:51:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:34 smithi165 bash[22591]: cluster 2024-09-10T18:51:33.617390+0000 osd.3 (osd.3) 52 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:51:35.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:33.448170+0000 osd.7 (osd.7) 49 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:51:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:33.452490+0000 osd.7 (osd.7) 50 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:51:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:33.611819+0000 osd.3 (osd.3) 51 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:51:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:33.617390+0000 osd.3 (osd.3) 52 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:51:35.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.940+0000 7fe63a2a1640 1 -- 172.21.15.83:0/905665325 >> v1:172.21.15.83:6790/0 conn(0x7fe634107e00 legacy=0x7fe63410a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- 172.21.15.83:0/905665325 shutdown_connections 2024-09-10T18:51:35.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- 172.21.15.83:0/905665325 >> 172.21.15.83:0/905665325 conn(0x7fe6340f7070 msgr2=0x7fe6340f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:35.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- 172.21.15.83:0/905665325 shutdown_connections 2024-09-10T18:51:35.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- 172.21.15.83:0/905665325 wait complete. 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 librados: starting msgr at 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 librados: starting objecter 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 Processor -- start 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- start start 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 librados: setting wanted keys 2024-09-10T18:51:35.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 librados: calling monclient init 2024-09-10T18:51:35.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe63411c290 con 0x7fe634107e00 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe63411d430 con 0x7fe6341042a0 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63a2a1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe63411e5d0 con 0x7fe634101530 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63883e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe634107e00 0x7fe63411ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37702/0 (socket says 172.21.15.83:37702) 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe63883e640 1 -- 172.21.15.83:0/2403077101 learned_addr learned my addr 172.21.15.83:0/2403077101 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter ms_handle_connect 0x7fe634107e00 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter ms_handle_connect 0x7fe6341042a0 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter ms_handle_connect 0x7fe634101530 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066540925 0 0) 0x7fe63411c290 con 0x7fe634107e00 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61c0034e0 con 0x7fe634107e00 2024-09-10T18:51:35.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215846139 0 0) 0x7fe63411d430 con 0x7fe6341042a0 2024-09-10T18:51:35.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63411c290 con 0x7fe6341042a0 2024-09-10T18:51:35.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811891628 0 0) 0x7fe63411e5d0 con 0x7fe634101530 2024-09-10T18:51:35.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63411d430 con 0x7fe634101530 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660604888 0 0) 0x7fe61c0034e0 con 0x7fe634107e00 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe63411e5d0 con 0x7fe634107e00 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe62c001950 con 0x7fe634107e00 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103274781 0 0) 0x7fe63411c290 con 0x7fe6341042a0 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe61c0034e0 con 0x7fe6341042a0 2024-09-10T18:51:35.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2856528388 0 0) 0x7fe63411d430 con 0x7fe634101530 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe63411c290 con 0x7fe634101530 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe6280017f0 con 0x7fe6341042a0 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.944+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3426945344 0 0) 0x7fe63411e5d0 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 >> v1:172.21.15.83:6790/0 conn(0x7fe634101530 legacy=0x7fe634107120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 >> v1:172.21.15.165:6789/0 conn(0x7fe6341042a0 legacy=0x7fe634117440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63411f770 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63411d620 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 10 client.14673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63411da70 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 1 librados: init done 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe62c003220 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe62c0038a0 con 0x7fe634107e00 2024-09-10T18:51:35.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 10 client.14673.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:35.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 15 client.14673.objecter _session_command_op_assign -1 1 2024-09-10T18:51:35.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 20 client.14673.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:35.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 10 client.14673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:35.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6200028e0 con 0x7fe634107e00 2024-09-10T18:51:35.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe62c003aa0 con 0x7fe634107e00 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(102..102 src has 49..102) v4 ==== 6284+0+0 (unknown 1799075404 0 0) 0x7fe62c08a370 con 0x7fe634107e00 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 10 client.14673.objecter ms_dispatch 0x7fe634000b90 osd_map(102..102 src has 49..102) v4 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 3 client.14673.objecter handle_osd_map got epochs [102,102] > 0 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.948+0000 7fe630ff9640 3 client.14673.objecter handle_osd_map decoding full epoch 102 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter checking command 1 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter _get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 15 client.14673.objecter _session_command_op_remove -1 1 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter _get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 15 client.14673.objecter _session_command_op_assign 4 1 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 4 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter _send_command 1 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.165:6800/3225515265 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe61c07ff60 con 0x7fe61c07c260 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter dump_active .. 0 homeless 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (unknown 690086368 0 0) 0x7fe62c08a7a0 con 0x7fe634107e00 2024-09-10T18:51:35.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter ms_handle_connect 0x7fe61c0718a0 2024-09-10T18:51:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter ms_handle_connect 0x7fe61c07c260 2024-09-10T18:51:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe61c07ff60 con 0x7fe61c07c260 2024-09-10T18:51:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter ms_dispatch 0x7fe634000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:35.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 10 client.14673.objecter _finish_command 1 = system:0 2024-09-10T18:51:35.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 4 2024-09-10T18:51:35.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.952+0000 7fe630ff9640 15 client.14673.objecter _session_command_op_remove 4 1 2024-09-10T18:51:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 client.14673.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 15 client.14673.objecter _session_command_op_assign -1 2 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter _get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter _get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 15 client.14673.objecter _session_command_op_remove -1 2 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter get_session s=0x7fe61c07bbb0 osd=4 3 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 15 client.14673.objecter _session_command_op_assign 4 2 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 4 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 client.14673.objecter _send_command 2 2024-09-10T18:51:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 --> v1:172.21.15.165:6800/3225515265 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6200011c0 con 0x7fe61c07c260 2024-09-10T18:51:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe630ff9640 1 -- 172.21.15.83:0/2403077101 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1648478145) 0x7fe6200011c0 con 0x7fe61c07c260 2024-09-10T18:51:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe630ff9640 10 client.14673.objecter ms_dispatch 0x7fe634000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe630ff9640 10 client.14673.objecter _finish_command 2 = system:0 2024-09-10T18:51:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe630ff9640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 4 2024-09-10T18:51:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe630ff9640 15 client.14673.objecter _session_command_op_remove 4 2 2024-09-10T18:51:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 librados: watch_flush enter 2024-09-10T18:51:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 librados: watch_flush exit 2024-09-10T18:51:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 client.14673.objecter close_session for osd.4 2024-09-10T18:51:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 >> v1:172.21.15.165:6800/3225515265 conn(0x7fe61c07c260 legacy=0x7fe61c07e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter put_session s=0x7fe61c07bbb0 osd=4 1 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 20 client.14673.objecter shutdown clearing up homeless session... 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 10 client.14673.objecter successfully canceled tick 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe61c0718a0 legacy=0x7fe61c073d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 >> v1:172.21.15.83:6789/0 conn(0x7fe634107e00 legacy=0x7fe63411ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63883e640 1 -- 172.21.15.83:0/2403077101 reap_dead start 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 shutdown_connections 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 >> 172.21.15.83:0/2403077101 conn(0x7fe6340f7070 msgr2=0x7fe6340f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 shutdown_connections 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 -- 172.21.15.83:0/2403077101 wait complete. 2024-09-10T18:51:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:35.972+0000 7fe63a2a1640 1 librados: shutdown 2024-09-10T18:51:36.308 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[18629]: cluster 2024-09-10T18:51:34.482491+0000 mgr.y (mgr.24526) 82 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:36.308 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[18629]: cluster 2024-09-10T18:51:35.489087+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:51:36.308 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[18629]: cluster 2024-09-10T18:51:35.492453+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:51:36.308 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:34.482491+0000 mgr.y (mgr.24526) 82 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:36.309 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:35.489087+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:51:36.309 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:35 smithi083 bash[32381]: cluster 2024-09-10T18:51:35.492453+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:51:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:35 smithi165 bash[22591]: cluster 2024-09-10T18:51:34.482491+0000 mgr.y (mgr.24526) 82 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:35 smithi165 bash[22591]: cluster 2024-09-10T18:51:35.489087+0000 osd.1 (osd.1) 51 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:51:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:35 smithi165 bash[22591]: cluster 2024-09-10T18:51:35.492453+0000 osd.1 (osd.1) 52 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:51:37.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/3968858423 >> v1:172.21.15.83:6789/0 conn(0x7fc4b80fd5b0 legacy=0x7fc4b80fd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/3968858423 shutdown_connections 2024-09-10T18:51:38.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/3968858423 >> 172.21.15.83:0/3968858423 conn(0x7fc4b80f7060 msgr2=0x7fc4b80f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/3968858423 shutdown_connections 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/3968858423 wait complete. 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 librados: starting msgr at 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 librados: starting objecter 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 Processor -- start 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- start start 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 librados: setting wanted keys 2024-09-10T18:51:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 librados: calling monclient init 2024-09-10T18:51:38.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4b811c3f0 con 0x7fc4b8118640 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4b811d5f0 con 0x7fc4b80fd5b0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc4b811e7f0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4b6ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc4b8108ee0 0x7fc4b8104820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60088/0 (socket says 172.21.15.83:60088) 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4b6ffd640 1 -- 172.21.15.83:0/2985265570 learned_addr learned my addr 172.21.15.83:0/2985265570 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter ms_handle_connect 0x7fc4b8108ee0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546084720 0 0) 0x7fc4b811e7f0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a0003720 con 0x7fc4b8108ee0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter ms_handle_connect 0x7fc4b80fd5b0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter ms_handle_connect 0x7fc4b8118640 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835775214 0 0) 0x7fc4a0003720 con 0x7fc4b8108ee0 2024-09-10T18:51:38.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4b811e7f0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4ac001940 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548222009 0 0) 0x7fc4b811c3f0 con 0x7fc4b8118640 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a0003720 con 0x7fc4b8118640 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440661922 0 0) 0x7fc4b811d5f0 con 0x7fc4b80fd5b0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b811c3f0 con 0x7fc4b80fd5b0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 598287448 0 0) 0x7fc4b811e7f0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 >> v1:172.21.15.165:6789/0 conn(0x7fc4b80fd5b0 legacy=0x7fc4b8104110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 >> v1:172.21.15.83:6789/0 conn(0x7fc4b8118640 legacy=0x7fc4b811aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b811f9f0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b811d7e0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 10 client.24599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b811dc30 con 0x7fc4b8108ee0 2024-09-10T18:51:38.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc4bda63640 1 librados: init done 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc4ac002d00 con 0x7fc4b8108ee0 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.180+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc4ac005bd0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc4bda63640 10 client.24599.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc4bda63640 15 client.24599.objecter _session_command_op_assign -1 1 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc4bda63640 20 client.24599.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc4bda63640 10 client.24599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:38.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc4a40028b0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc4ac001fa0 con 0x7fc4b8108ee0 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(103..103 src has 49..103) v4 ==== 6284+0+0 (unknown 3837731416 0 0) 0x7fc4ac08a050 con 0x7fc4b8108ee0 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc49ffff640 10 client.24599.objecter ms_dispatch 0x7fc4b8000b90 osd_map(103..103 src has 49..103) v4 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc49ffff640 3 client.24599.objecter handle_osd_map got epochs [103,103] > 0 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.184+0000 7fc49ffff640 3 client.24599.objecter handle_osd_map decoding full epoch 103 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter checking command 1 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter _get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 15 client.24599.objecter _session_command_op_remove -1 1 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter _get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 15 client.24599.objecter _session_command_op_assign 6 1 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 4 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter _send_command 1 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.165:6808/1317922936 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4a007fd10 con 0x7fc4a007c010 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter dump_active .. 0 homeless 2024-09-10T18:51:38.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (unknown 3677219870 0 0) 0x7fc4ac08a420 con 0x7fc4b8108ee0 2024-09-10T18:51:38.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter ms_handle_connect 0x7fc4a00715d0 2024-09-10T18:51:38.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter ms_handle_connect 0x7fc4a007c010 2024-09-10T18:51:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc4a007fd10 con 0x7fc4a007c010 2024-09-10T18:51:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter ms_dispatch 0x7fc4b8000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 10 client.24599.objecter _finish_command 1 = system:0 2024-09-10T18:51:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 4 2024-09-10T18:51:38.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.188+0000 7fc49ffff640 15 client.24599.objecter _session_command_op_remove 6 1 2024-09-10T18:51:38.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 10 client.24599.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:38.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 15 client.24599.objecter _session_command_op_assign -1 2 2024-09-10T18:51:38.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 20 client.24599.objecter _get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 20 client.24599.objecter _get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 15 client.24599.objecter _session_command_op_remove -1 2 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 20 client.24599.objecter get_session s=0x7fc4a007b960 osd=6 3 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 15 client.24599.objecter _session_command_op_assign 6 2 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 4 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 10 client.24599.objecter _send_command 2 2024-09-10T18:51:38.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.204+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 --> v1:172.21.15.165:6808/1317922936 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc4a4001190 con 0x7fc4a007c010 2024-09-10T18:51:38.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc49ffff640 1 -- 172.21.15.83:0/2985265570 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1427340890) 0x7fc4a4001190 con 0x7fc4a007c010 2024-09-10T18:51:38.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc49ffff640 10 client.24599.objecter ms_dispatch 0x7fc4b8000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:38.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc49ffff640 10 client.24599.objecter _finish_command 2 = system:0 2024-09-10T18:51:38.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc49ffff640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 4 2024-09-10T18:51:38.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc49ffff640 15 client.24599.objecter _session_command_op_remove 6 2 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 10 librados: watch_flush enter 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 10 librados: watch_flush exit 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 10 client.24599.objecter close_session for osd.6 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 >> v1:172.21.15.165:6808/1317922936 conn(0x7fc4a007c010 legacy=0x7fc4a007e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 20 client.24599.objecter put_session s=0x7fc4a007b960 osd=6 1 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 20 client.24599.objecter shutdown clearing up homeless session... 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 10 client.24599.objecter successfully canceled tick 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc4a00715d0 legacy=0x7fc4a0073a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 >> v1:172.21.15.83:6790/0 conn(0x7fc4b8108ee0 legacy=0x7fc4b8104820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4b7fff640 1 -- 172.21.15.83:0/2985265570 reap_dead start 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 shutdown_connections 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 >> 172.21.15.83:0/2985265570 conn(0x7fc4b80f7060 msgr2=0x7fc4b810b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 shutdown_connections 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 -- 172.21.15.83:0/2985265570 wait complete. 2024-09-10T18:51:38.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:38.208+0000 7fc4bda63640 1 librados: shutdown 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[18629]: cluster 2024-09-10T18:51:36.483923+0000 mgr.y (mgr.24526) 83 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[18629]: cluster 2024-09-10T18:51:36.533500+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[18629]: cluster 2024-09-10T18:51:36.536984+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.467849+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.467947+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[32381]: cluster 2024-09-10T18:51:36.483923+0000 mgr.y (mgr.24526) 83 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[32381]: cluster 2024-09-10T18:51:36.533500+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[32381]: cluster 2024-09-10T18:51:36.536984+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:51:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.467849+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:38.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:37 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.467947+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[22591]: cluster 2024-09-10T18:51:36.483923+0000 mgr.y (mgr.24526) 83 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[22591]: cluster 2024-09-10T18:51:36.533500+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:51:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[22591]: cluster 2024-09-10T18:51:36.536984+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:51:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.467849+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:37 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.467947+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:39.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.342005+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.345364+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: audit 2024-09-10T18:51:37.461290+0000 mgr.y (mgr.24526) 84 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.689197+0000 osd.3 (osd.3) 53 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.694127+0000 osd.3 (osd.3) 54 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[18629]: cluster 2024-09-10T18:51:37.995936+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.342005+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.345364+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: audit 2024-09-10T18:51:37.461290+0000 mgr.y (mgr.24526) 84 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.689197+0000 osd.3 (osd.3) 53 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:51:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.694127+0000 osd.3 (osd.3) 54 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:51:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:38 smithi083 bash[32381]: cluster 2024-09-10T18:51:37.995936+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-10T18:51:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.342005+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:51:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.345364+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:51:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: audit 2024-09-10T18:51:37.461290+0000 mgr.y (mgr.24526) 84 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.689197+0000 osd.3 (osd.3) 53 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:51:39.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.694127+0000 osd.3 (osd.3) 54 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:51:39.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:38 smithi165 bash[22591]: cluster 2024-09-10T18:51:37.995936+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-09-10T18:51:40.001 INFO:teuthology.orchestra.run.smithi083.stdout:150323855431 2024-09-10T18:51:40.002 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:38.484703+0000 mgr.y (mgr.24526) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:38.647314+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:38.650765+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:38.758266+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:38.761849+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:51:40.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.619495+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.622989+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:38.484703+0000 mgr.y (mgr.24526) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:38.647314+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:38.650765+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:38.758266+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:38.761849+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.619495+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:51:40.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:40 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.622989+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:38.484703+0000 mgr.y (mgr.24526) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:38.647314+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:38.650765+0000 osd.3 (osd.3) 56 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:38.758266+0000 osd.4 (osd.4) 57 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:38.761849+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:51:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.619495+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:51:40.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:40 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.622989+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:51:41.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.635989+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.639693+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.967363+0000 osd.3 (osd.3) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[18629]: cluster 2024-09-10T18:51:39.967448+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.016468+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.635989+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.639693+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.967363+0000 osd.3 (osd.3) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[32381]: cluster 2024-09-10T18:51:39.967448+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:41 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.016468+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-10T18:51:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.635989+0000 osd.3 (osd.3) 57 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:51:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.639693+0000 osd.3 (osd.3) 58 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:51:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.967363+0000 osd.3 (osd.3) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:51:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[22591]: cluster 2024-09-10T18:51:39.967448+0000 osd.3 (osd.3) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:51:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.016468+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-09-10T18:51:41.481 INFO:teuthology.orchestra.run.smithi083.stdout:223338299441 2024-09-10T18:51:41.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T18:51:41.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:51:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:51:42.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- 172.21.15.83:0/42051971 >> v1:172.21.15.83:6789/0 conn(0x7f8ba00fe1d0 legacy=0x7f8ba0107680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- 172.21.15.83:0/42051971 shutdown_connections 2024-09-10T18:51:42.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- 172.21.15.83:0/42051971 >> 172.21.15.83:0/42051971 conn(0x7f8ba0075290 msgr2=0x7f8ba00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2995155634 >> v1:172.21.15.83:6789/0 conn(0x7ff5bc101530 legacy=0x7ff5bc101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2995155634 shutdown_connections 2024-09-10T18:51:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2995155634 >> 172.21.15.83:0/2995155634 conn(0x7ff5bc0f70b0 msgr2=0x7ff5bc0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2995155634 shutdown_connections 2024-09-10T18:51:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- 172.21.15.83:0/42051971 shutdown_connections 2024-09-10T18:51:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2995155634 wait complete. 2024-09-10T18:51:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- 172.21.15.83:0/42051971 wait complete. 2024-09-10T18:51:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 librados: starting msgr at 2024-09-10T18:51:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 librados: starting objecter 2024-09-10T18:51:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 Processor -- start 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 librados: starting msgr at 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 librados: starting objecter 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 Processor -- start 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 -- start start 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 librados: setting wanted keys 2024-09-10T18:51:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7f8ba83d6640 1 librados: calling monclient init 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8ba83d6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ba0122840 con 0x7f8ba0108f30 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8ba83d6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ba0123a40 con 0x7f8ba00fe1d0 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8ba83d6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ba0124c40 con 0x7f8ba00fd600 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8ba6973640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8ba0108f30 0x7f8ba0120f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37774/0 (socket says 172.21.15.83:37774) 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8ba6973640 1 -- 172.21.15.83:0/2879315736 learned_addr learned my addr 172.21.15.83:0/2879315736 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba0108f30 2024-09-10T18:51:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba00fd600 2024-09-10T18:51:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ba00fe1d0 2024-09-10T18:51:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- start start 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 librados: setting wanted keys 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 librados: calling monclient init 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5bc11bf80 con 0x7ff5bc101530 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5bc11d140 con 0x7ff5bc1042a0 2024-09-10T18:51:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.224+0000 7ff5c40ef640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5bc11e340 con 0x7ff5bc107e00 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c268c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff5bc107e00 0x7ff5bc11a680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60118/0 (socket says 172.21.15.83:60118) 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c268c640 1 -- 172.21.15.83:0/2832975651 learned_addr learned my addr 172.21.15.83:0/2832975651 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter ms_handle_connect 0x7ff5bc107e00 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter ms_handle_connect 0x7ff5bc1042a0 2024-09-10T18:51:42.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888661594 0 0) 0x7f8ba0122840 con 0x7f8ba0108f30 2024-09-10T18:51:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c0054b0 con 0x7f8ba0108f30 2024-09-10T18:51:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4163076732 0 0) 0x7f8b8c0054b0 con 0x7f8ba0108f30 2024-09-10T18:51:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ba0122840 con 0x7f8ba0108f30 2024-09-10T18:51:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8b98001c80 con 0x7f8ba0108f30 2024-09-10T18:51:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293618271 0 0) 0x7f8ba0123a40 con 0x7f8ba00fe1d0 2024-09-10T18:51:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008391494 0 0) 0x7ff5bc11d140 con 0x7ff5bc1042a0 2024-09-10T18:51:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a8003720 con 0x7ff5bc1042a0 2024-09-10T18:51:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter ms_handle_connect 0x7ff5bc101530 2024-09-10T18:51:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808489484 0 0) 0x7ff5bc11e340 con 0x7ff5bc107e00 2024-09-10T18:51:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5bc11d140 con 0x7ff5bc107e00 2024-09-10T18:51:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218248070 0 0) 0x7ff5bc11bf80 con 0x7ff5bc101530 2024-09-10T18:51:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b8c0054b0 con 0x7f8ba00fe1d0 2024-09-10T18:51:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5bc11e340 con 0x7ff5bc101530 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448512296 0 0) 0x7ff5bc11d140 con 0x7ff5bc107e00 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5bc11bf80 con 0x7ff5bc107e00 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff5b4001c40 con 0x7ff5bc107e00 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52814522 0 0) 0x7ff5a8003720 con 0x7ff5bc1042a0 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5bc11d140 con 0x7ff5bc1042a0 2024-09-10T18:51:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff5ac001840 con 0x7ff5bc1042a0 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655775338 0 0) 0x7ff5bc11bf80 con 0x7ff5bc107e00 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 >> v1:172.21.15.165:6789/0 conn(0x7ff5bc1042a0 legacy=0x7ff5bc105990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 >> v1:172.21.15.83:6789/0 conn(0x7ff5bc101530 legacy=0x7ff5bc105280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc11f540 con 0x7ff5bc107e00 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5bc11d330 con 0x7ff5bc107e00 2024-09-10T18:51:42.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c40ef640 10 client.24605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:42.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5bc11d800 con 0x7ff5bc107e00 2024-09-10T18:51:42.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5c40ef640 1 librados: init done 2024-09-10T18:51:42.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff5b4002340 con 0x7ff5bc107e00 2024-09-10T18:51:42.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff5b4005d90 con 0x7ff5bc107e00 2024-09-10T18:51:42.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455882299 0 0) 0x7f8ba0124c40 con 0x7f8ba00fd600 2024-09-10T18:51:42.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba0123a40 con 0x7f8ba00fd600 2024-09-10T18:51:42.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3987404404 0 0) 0x7f8ba0122840 con 0x7f8ba0108f30 2024-09-10T18:51:42.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 >> v1:172.21.15.83:6790/0 conn(0x7f8ba00fd600 legacy=0x7f8ba01065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 >> v1:172.21.15.165:6789/0 conn(0x7f8ba00fe1d0 legacy=0x7f8ba0106cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.228+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba0125e40 con 0x7f8ba0108f30 2024-09-10T18:51:42.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba0124e30 con 0x7f8ba0108f30 2024-09-10T18:51:42.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:42.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba0125280 con 0x7f8ba0108f30 2024-09-10T18:51:42.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 1 librados: init done 2024-09-10T18:51:42.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5c40ef640 10 client.24605.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:42.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5c40ef640 15 client.24605.objecter _session_command_op_assign -1 1 2024-09-10T18:51:42.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5c40ef640 20 client.24605.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:42.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5c40ef640 10 client.24605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:42.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5a00028b0 con 0x7ff5bc107e00 2024-09-10T18:51:42.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8b98002050 con 0x7f8ba0108f30 2024-09-10T18:51:42.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8b98005f10 con 0x7f8ba0108f30 2024-09-10T18:51:42.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 10 client.14688.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:42.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 15 client.14688.objecter _session_command_op_assign -1 1 2024-09-10T18:51:42.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 20 client.14688.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:42.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:42.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b840028b0 con 0x7f8ba0108f30 2024-09-10T18:51:42.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff5b4001ee0 con 0x7ff5bc107e00 2024-09-10T18:51:42.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7ff5b408a250 con 0x7ff5bc107e00 2024-09-10T18:51:42.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 10 client.24605.objecter ms_dispatch 0x7ff5bc000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:42.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 3 client.24605.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:42.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 3 client.24605.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 10 client.24605.objecter checking command 1 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter _get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 15 client.24605.objecter _session_command_op_remove -1 1 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter _get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 15 client.24605.objecter _session_command_op_assign 7 1 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 4 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 10 client.24605.objecter _send_command 1 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.165:6812/1610151650 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5a807fae0 con 0x7ff5a807bde0 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 20 client.24605.objecter dump_active .. 0 homeless 2024-09-10T18:51:42.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (unknown 2268335905 0 0) 0x7ff5b408a610 con 0x7ff5bc107e00 2024-09-10T18:51:42.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f8b98002530 con 0x7f8ba0108f30 2024-09-10T18:51:42.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 10 client.24605.objecter ms_handle_connect 0x7ff5a80714a0 2024-09-10T18:51:42.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.232+0000 7ff5b27fc640 10 client.24605.objecter ms_handle_connect 0x7ff5a807bde0 2024-09-10T18:51:42.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7f8b9808a110 con 0x7f8ba0108f30 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter ms_dispatch 0x7f8ba0000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 3 client.14688.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 3 client.14688.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter checking command 1 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter _get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 15 client.14688.objecter _session_command_op_remove -1 1 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter _get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 15 client.14688.objecter _session_command_op_assign 2 1 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 4 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter _send_command 1 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6809/386094220 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b8c07f9a0 con 0x7f8b8c07bca0 2024-09-10T18:51:42.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter dump_active .. 0 homeless 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (unknown 2268335905 0 0) 0x7f8b9808a510 con 0x7f8ba0108f30 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter ms_handle_connect 0x7f8b8c07bca0 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter ms_handle_connect 0x7f8b8c071260 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8b8c07f9a0 con 0x7f8b8c07bca0 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter ms_dispatch 0x7f8ba0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 10 client.14688.objecter _finish_command 1 = system:0 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 4 2024-09-10T18:51:42.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7f8b96ffd640 15 client.14688.objecter _session_command_op_remove 2 1 2024-09-10T18:51:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff5a807fae0 con 0x7ff5a807bde0 2024-09-10T18:51:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7ff5b27fc640 10 client.24605.objecter ms_dispatch 0x7ff5bc000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7ff5b27fc640 10 client.24605.objecter _finish_command 1 = system:0 2024-09-10T18:51:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7ff5b27fc640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 4 2024-09-10T18:51:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.236+0000 7ff5b27fc640 15 client.24605.objecter _session_command_op_remove 7 1 2024-09-10T18:51:42.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 10 client.14688.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:42.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 15 client.14688.objecter _session_command_op_assign -1 2 2024-09-10T18:51:42.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 20 client.14688.objecter _get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 20 client.14688.objecter _get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 15 client.14688.objecter _session_command_op_remove -1 2 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 20 client.14688.objecter get_session s=0x7f8b8c07b5f0 osd=2 3 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 15 client.14688.objecter _session_command_op_assign 2 2 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 4 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 10 client.14688.objecter _send_command 2 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 --> v1:172.21.15.83:6809/386094220 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b84001190 con 0x7f8b8c07bca0 2024-09-10T18:51:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8b96ffd640 1 -- 172.21.15.83:0/2879315736 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 372940400) 0x7f8b84001190 con 0x7f8b8c07bca0 2024-09-10T18:51:42.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8b96ffd640 10 client.14688.objecter ms_dispatch 0x7f8ba0000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:42.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8b96ffd640 10 client.14688.objecter _finish_command 2 = system:0 2024-09-10T18:51:42.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8b96ffd640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 4 2024-09-10T18:51:42.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.248+0000 7f8b96ffd640 15 client.14688.objecter _session_command_op_remove 2 2 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 10 librados: watch_flush enter 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 10 librados: watch_flush exit 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 10 client.14688.objecter close_session for osd.2 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 >> v1:172.21.15.83:6809/386094220 conn(0x7f8b8c07bca0 legacy=0x7f8b8c07e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 20 client.14688.objecter put_session s=0x7f8b8c07b5f0 osd=2 1 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 20 client.14688.objecter shutdown clearing up homeless session... 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 10 client.14688.objecter successfully canceled tick 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8b8c071260 legacy=0x7f8b8c073710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 >> v1:172.21.15.83:6789/0 conn(0x7f8ba0108f30 legacy=0x7f8ba0120f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba6973640 1 -- 172.21.15.83:0/2879315736 reap_dead start 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 shutdown_connections 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 >> 172.21.15.83:0/2879315736 conn(0x7f8ba0075290 msgr2=0x7f8ba00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 shutdown_connections 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7f8ba83d6640 1 -- 172.21.15.83:0/2879315736 wait complete. 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 10 client.24605.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 15 client.24605.objecter _session_command_op_assign -1 2 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 20 client.24605.objecter _get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 20 client.24605.objecter _get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 15 client.24605.objecter _session_command_op_remove -1 2 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 20 client.24605.objecter get_session s=0x7ff5a807b730 osd=7 3 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 15 client.24605.objecter _session_command_op_assign 7 2 2024-09-10T18:51:42.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 4 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 10 client.24605.objecter _send_command 2 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 --> v1:172.21.15.165:6812/1610151650 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5a0001190 con 0x7ff5a807bde0 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5b27fc640 1 -- 172.21.15.83:0/2832975651 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3892225106) 0x7ff5a0001190 con 0x7ff5a807bde0 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5b27fc640 10 client.24605.objecter ms_dispatch 0x7ff5bc000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5b27fc640 10 client.24605.objecter _finish_command 2 = system:0 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5b27fc640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 4 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.252+0000 7ff5b27fc640 15 client.24605.objecter _session_command_op_remove 7 2 2024-09-10T18:51:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7f8ba83d6640 1 librados: shutdown 2024-09-10T18:51:42.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 10 librados: watch_flush enter 2024-09-10T18:51:42.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 10 librados: watch_flush exit 2024-09-10T18:51:42.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 10 client.24605.objecter close_session for osd.7 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff5a807bde0 legacy=0x7ff5a807e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 20 client.24605.objecter put_session s=0x7ff5a807b730 osd=7 1 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 20 client.24605.objecter shutdown clearing up homeless session... 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 10 client.24605.objecter successfully canceled tick 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff5a80714a0 legacy=0x7ff5a8073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 >> v1:172.21.15.83:6790/0 conn(0x7ff5bc107e00 legacy=0x7ff5bc11a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c268c640 1 -- 172.21.15.83:0/2832975651 reap_dead start 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 shutdown_connections 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 >> 172.21.15.83:0/2832975651 conn(0x7ff5bc0f70b0 msgr2=0x7ff5bc0fa380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 shutdown_connections 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 -- 172.21.15.83:0/2832975651 wait complete. 2024-09-10T18:51:42.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:42.256+0000 7ff5c40ef640 1 librados: shutdown 2024-09-10T18:51:42.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.486116+0000 mgr.y (mgr.24526) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.688820+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.e scrub starts 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.694621+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.e scrub ok 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.772102+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:40.776141+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.489009+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:51:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.492698+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:51:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.486116+0000 mgr.y (mgr.24526) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.688820+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.e scrub starts 2024-09-10T18:51:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.694621+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.e scrub ok 2024-09-10T18:51:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.772102+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:51:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:40.776141+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:51:42.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.489009+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:51:42.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.492698+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:51:42.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.486116+0000 mgr.y (mgr.24526) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:42.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.688820+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.e scrub starts 2024-09-10T18:51:42.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.694621+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.e scrub ok 2024-09-10T18:51:42.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.772102+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:51:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:40.776141+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:51:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.489009+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:51:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:42 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.492698+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:51:43.014 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:42] "GET /metrics HTTP/1.1" 200 220056 "" "Prometheus/2.43.0" 2024-09-10T18:51:43.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.623881+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.627136+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.702438+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1e deep-scrub starts 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[18629]: cluster 2024-09-10T18:51:41.705937+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1e deep-scrub ok 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.623881+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.627136+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.702438+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1e deep-scrub starts 2024-09-10T18:51:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:43 smithi083 bash[32381]: cluster 2024-09-10T18:51:41.705937+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1e deep-scrub ok 2024-09-10T18:51:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:43 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.623881+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:51:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:43 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.627136+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:51:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:43 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.702438+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.1e deep-scrub starts 2024-09-10T18:51:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:43 smithi165 bash[22591]: cluster 2024-09-10T18:51:41.705937+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.1e deep-scrub ok 2024-09-10T18:51:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 -- 172.21.15.83:0/1484182140 >> v1:172.21.15.83:6790/0 conn(0x7f1884102670 legacy=0x7f1884102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 -- 172.21.15.83:0/1484182140 shutdown_connections 2024-09-10T18:51:43.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 -- 172.21.15.83:0/1484182140 >> 172.21.15.83:0/1484182140 conn(0x7f1884074a20 msgr2=0x7f18840f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 -- 172.21.15.83:0/1484182140 shutdown_connections 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 -- 172.21.15.83:0/1484182140 wait complete. 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 librados: starting msgr at 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 librados: starting objecter 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.596+0000 7f188c0fd640 1 Processor -- start 2024-09-10T18:51:43.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- start start 2024-09-10T18:51:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 librados: setting wanted keys 2024-09-10T18:51:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 librados: calling monclient init 2024-09-10T18:51:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1884101b90 con 0x7f1884108f60 2024-09-10T18:51:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1884101d20 con 0x7f1884102670 2024-09-10T18:51:43.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1884101ef0 con 0x7f18841053a0 2024-09-10T18:51:43.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f1889698640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f18841053a0 0x7f1884119960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40368/0 (socket says 172.21.15.83:40368) 2024-09-10T18:51:43.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f1889698640 1 -- 172.21.15.83:0/3772557715 learned_addr learned my addr 172.21.15.83:0/3772557715 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:43.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter ms_handle_connect 0x7f18841053a0 2024-09-10T18:51:43.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter ms_handle_connect 0x7f1884108f60 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671812460 0 0) 0x7f1884101ef0 con 0x7f18841053a0 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f186c003120 con 0x7f18841053a0 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter ms_handle_connect 0x7f1884102670 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109554405 0 0) 0x7f1884101b90 con 0x7f1884108f60 2024-09-10T18:51:43.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1884101ef0 con 0x7f1884108f60 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261969240 0 0) 0x7f1884101d20 con 0x7f1884102670 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1884101b90 con 0x7f1884102670 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232701834 0 0) 0x7f186c003120 con 0x7f18841053a0 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1884101d20 con 0x7f18841053a0 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1878001960 con 0x7f18841053a0 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700086331 0 0) 0x7f1884101ef0 con 0x7f1884108f60 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f186c003120 con 0x7f1884108f60 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f18740017b0 con 0x7f1884108f60 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040732016 0 0) 0x7f1884101d20 con 0x7f18841053a0 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 >> v1:172.21.15.165:6789/0 conn(0x7f1884102670 legacy=0x7f18841013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 >> v1:172.21.15.83:6789/0 conn(0x7f1884108f60 legacy=0x7f188411d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18841217e0 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f188411e810 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 10 client.24614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f188411ec60 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f188c0fd640 1 librados: init done 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1878002420 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.600+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1878005af0 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f187801bff0 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7f187808a0a0 con 0x7f18841053a0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 10 client.24614.objecter ms_dispatch 0x7f1884000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 3 client.24614.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 3 client.24614.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:43.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 20 client.24614.objecter dump_active .. 0 homeless 2024-09-10T18:51:43.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.604+0000 7f18827fc640 10 client.24614.objecter ms_handle_connect 0x7f186c071700 2024-09-10T18:51:43.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 10 client.24614.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_assign -1 1 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 20 client.24614.objecter _get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 20 client.24614.objecter _get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_remove -1 1 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 20 client.24614.objecter get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_assign 0 1 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 4 2024-09-10T18:51:43.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 10 client.24614.objecter _send_command 1 2024-09-10T18:51:43.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1870006cc0 con 0x7f1870002f60 2024-09-10T18:51:43.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 10 client.24614.objecter ms_handle_connect 0x7f1870002f60 2024-09-10T18:51:43.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1870006cc0 con 0x7f1870002f60 2024-09-10T18:51:43.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 10 client.24614.objecter ms_dispatch 0x7f1884000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:43.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 10 client.24614.objecter _finish_command 1 = system:0 2024-09-10T18:51:43.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 4 2024-09-10T18:51:43.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.644+0000 7f18827fc640 15 client.24614.objecter _session_command_op_remove 0 1 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 client.24614.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_assign -1 2 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter _get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter _get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_remove -1 2 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter get_session s=0x7f1870002870 osd=0 3 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 15 client.24614.objecter _session_command_op_assign 0 2 2024-09-10T18:51:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 4 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 client.24614.objecter _send_command 2 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18700011c0 con 0x7f1870002f60 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f18827fc640 1 -- 172.21.15.83:0/3772557715 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 747004464) 0x7f18700011c0 con 0x7f1870002f60 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f18827fc640 10 client.24614.objecter ms_dispatch 0x7f1884000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f18827fc640 10 client.24614.objecter _finish_command 2 = system:0 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f18827fc640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 4 2024-09-10T18:51:43.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f18827fc640 15 client.24614.objecter _session_command_op_remove 0 2 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 librados: watch_flush enter 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 librados: watch_flush exit 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 client.24614.objecter close_session for osd.0 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 >> v1:172.21.15.83:6801/2535527483 conn(0x7f1870002f60 legacy=0x7f1870005410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter put_session s=0x7f1870002870 osd=0 1 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 20 client.24614.objecter shutdown clearing up homeless session... 2024-09-10T18:51:43.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 10 client.24614.objecter successfully canceled tick 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 >> v1:172.21.15.83:6800/3071793735 conn(0x7f186c071700 legacy=0x7f186c073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 >> v1:172.21.15.83:6790/0 conn(0x7f18841053a0 legacy=0x7f1884119960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.664+0000 7f188a69a640 1 -- 172.21.15.83:0/3772557715 reap_dead start 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.668+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 shutdown_connections 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.668+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 >> 172.21.15.83:0/3772557715 conn(0x7f1884074a20 msgr2=0x7f18840f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.668+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 shutdown_connections 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.668+0000 7f188c0fd640 1 -- 172.21.15.83:0/3772557715 wait complete. 2024-09-10T18:51:43.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:43.668+0000 7f188c0fd640 1 librados: shutdown 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: cluster 2024-09-10T18:51:42.487264+0000 mgr.y (mgr.24526) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: cluster 2024-09-10T18:51:42.586352+0000 osd.3 (osd.3) 63 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: cluster 2024-09-10T18:51:42.590056+0000 osd.3 (osd.3) 64 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: cluster 2024-09-10T18:51:43.646017+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: cluster 2024-09-10T18:51:43.649941+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: audit 2024-09-10T18:51:43.728493+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[18629]: audit 2024-09-10T18:51:43.736058+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: cluster 2024-09-10T18:51:42.487264+0000 mgr.y (mgr.24526) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: cluster 2024-09-10T18:51:42.586352+0000 osd.3 (osd.3) 63 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:51:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: cluster 2024-09-10T18:51:42.590056+0000 osd.3 (osd.3) 64 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:51:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: cluster 2024-09-10T18:51:43.646017+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:51:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: cluster 2024-09-10T18:51:43.649941+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:51:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: audit 2024-09-10T18:51:43.728493+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:44 smithi083 bash[32381]: audit 2024-09-10T18:51:43.736058+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: cluster 2024-09-10T18:51:42.487264+0000 mgr.y (mgr.24526) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:44.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: cluster 2024-09-10T18:51:42.586352+0000 osd.3 (osd.3) 63 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:51:44.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: cluster 2024-09-10T18:51:42.590056+0000 osd.3 (osd.3) 64 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:51:44.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: cluster 2024-09-10T18:51:43.646017+0000 osd.5 (osd.5) 45 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:51:44.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: cluster 2024-09-10T18:51:43.649941+0000 osd.5 (osd.5) 46 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:51:44.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: audit 2024-09-10T18:51:43.728493+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:44 smithi165 bash[22591]: audit 2024-09-10T18:51:43.736058+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:44.901 INFO:teuthology.orchestra.run.smithi083.stdout:81604378716 2024-09-10T18:51:44.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: cluster 2024-09-10T18:51:43.696771+0000 osd.4 (osd.4) 63 : cluster [DBG] 2.f scrub starts 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: cluster 2024-09-10T18:51:43.700280+0000 osd.4 (osd.4) 64 : cluster [DBG] 2.f scrub ok 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.118385+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.119703+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.129064+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.181424+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.183537+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.184320+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.192960+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[18629]: audit 2024-09-10T18:51:44.541357+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: cluster 2024-09-10T18:51:43.696771+0000 osd.4 (osd.4) 63 : cluster [DBG] 2.f scrub starts 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: cluster 2024-09-10T18:51:43.700280+0000 osd.4 (osd.4) 64 : cluster [DBG] 2.f scrub ok 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.118385+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.119703+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.129064+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.181424+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.183537+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.184320+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.192960+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:45 smithi083 bash[32381]: audit 2024-09-10T18:51:44.541357+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:45.339 INFO:teuthology.orchestra.run.smithi083.stdout:266287972392 2024-09-10T18:51:45.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: cluster 2024-09-10T18:51:43.696771+0000 osd.4 (osd.4) 63 : cluster [DBG] 2.f scrub starts 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: cluster 2024-09-10T18:51:43.700280+0000 osd.4 (osd.4) 64 : cluster [DBG] 2.f scrub ok 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.118385+0000 mon.c (mon.2) 67 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.119703+0000 mon.c (mon.2) 68 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.129064+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.181424+0000 mon.c (mon.2) 69 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:51:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.183537+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:51:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.184320+0000 mon.c (mon.2) 71 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:51:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.192960+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:51:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:45 smithi165 bash[22591]: audit 2024-09-10T18:51:44.541357+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:51:45.976 INFO:teuthology.orchestra.run.smithi083.stdout:34359738483 2024-09-10T18:51:45.976 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T18:51:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.120+0000 7f3a91426640 1 -- 172.21.15.83:0/3803218302 >> v1:172.21.15.83:6790/0 conn(0x7f3a8c109130 legacy=0x7f3a8c10b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.120+0000 7f3a91426640 1 -- 172.21.15.83:0/3803218302 shutdown_connections 2024-09-10T18:51:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.120+0000 7f3a91426640 1 -- 172.21.15.83:0/3803218302 >> 172.21.15.83:0/3803218302 conn(0x7f3a8c075290 msgr2=0x7f3a8c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.120+0000 7f3a91426640 1 -- 172.21.15.83:0/3803218302 shutdown_connections 2024-09-10T18:51:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.120+0000 7f3a91426640 1 -- 172.21.15.83:0/3803218302 wait complete. 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 librados: starting msgr at 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 librados: starting objecter 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 Processor -- start 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- start start 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 librados: setting wanted keys 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 librados: calling monclient init 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3a8c11e0c0 con 0x7f3a8c109130 2024-09-10T18:51:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3a8c11f260 con 0x7f3a8c0fd650 2024-09-10T18:51:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3a8c120440 con 0x7f3a8c105630 2024-09-10T18:51:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a8b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3a8c109130 0x7f3a8c11c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60274/0 (socket says 172.21.15.83:60274) 2024-09-10T18:51:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a8b7fe640 1 -- 172.21.15.83:0/3088802816 learned_addr learned my addr 172.21.15.83:0/3088802816 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:46.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter ms_handle_connect 0x7f3a8c109130 2024-09-10T18:51:46.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter ms_handle_connect 0x7f3a8c105630 2024-09-10T18:51:46.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052451133 0 0) 0x7f3a8c120440 con 0x7f3a8c105630 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a74003200 con 0x7f3a8c105630 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter ms_handle_connect 0x7f3a8c0fd650 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400549668 0 0) 0x7f3a8c11e0c0 con 0x7f3a8c109130 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8c120440 con 0x7f3a8c109130 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035303721 0 0) 0x7f3a8c11f260 con 0x7f3a8c0fd650 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8c11e0c0 con 0x7f3a8c0fd650 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2777083953 0 0) 0x7f3a74003200 con 0x7f3a8c105630 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a8c11f260 con 0x7f3a8c105630 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3a80001840 con 0x7f3a8c105630 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1938241551 0 0) 0x7f3a8c120440 con 0x7f3a8c109130 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a74003200 con 0x7f3a8c109130 2024-09-10T18:51:46.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3a7c001440 con 0x7f3a8c109130 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 515454743 0 0) 0x7f3a8c11f260 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 >> v1:172.21.15.165:6789/0 conn(0x7f3a8c0fd650 legacy=0x7f3a8c104640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 >> v1:172.21.15.83:6789/0 conn(0x7f3a8c109130 legacy=0x7f3a8c11c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c121620 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c120630 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 10 client.24620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c120a80 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 librados: init done 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f3a80002270 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3a800030d0 con 0x7f3a8c105630 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 10 client.24620.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 15 client.24620.objecter _session_command_op_assign -1 1 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 20 client.24620.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:46.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 10 client.24620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.124+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a780028b0 con 0x7f3a8c105630 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f3a80001e10 con 0x7f3a8c105630 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7f3a800874a0 con 0x7f3a8c105630 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter ms_dispatch 0x7f3a8c000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 3 client.24620.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 3 client.24620.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter checking command 1 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter _get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 15 client.24620.objecter _session_command_op_remove -1 1 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter _get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 15 client.24620.objecter _session_command_op_assign 1 1 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 4 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter _send_command 1 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6805/2309624637 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a740802a0 con 0x7f3a7407c5a0 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter dump_active .. 0 homeless 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (unknown 2268335905 0 0) 0x7f3a80087830 con 0x7f3a8c105630 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter ms_handle_connect 0x7f3a7407c5a0 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter ms_handle_connect 0x7f3a74071b90 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f3a740802a0 con 0x7f3a7407c5a0 2024-09-10T18:51:46.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter ms_dispatch 0x7f3a8c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:46.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 10 client.24620.objecter _finish_command 1 = system:0 2024-09-10T18:51:46.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 4 2024-09-10T18:51:46.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.128+0000 7f3a737fe640 15 client.24620.objecter _session_command_op_remove 1 1 2024-09-10T18:51:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 client.24620.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 15 client.24620.objecter _session_command_op_assign -1 2 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter _get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter _get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 15 client.24620.objecter _session_command_op_remove -1 2 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter get_session s=0x7f3a7407bef0 osd=1 3 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 15 client.24620.objecter _session_command_op_assign 1 2 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 4 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 client.24620.objecter _send_command 2 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 --> v1:172.21.15.83:6805/2309624637 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3a78001190 con 0x7f3a7407c5a0 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a737fe640 1 -- 172.21.15.83:0/3088802816 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2357320960) 0x7f3a78001190 con 0x7f3a7407c5a0 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a737fe640 10 client.24620.objecter ms_dispatch 0x7f3a8c000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a737fe640 10 client.24620.objecter _finish_command 2 = system:0 2024-09-10T18:51:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a737fe640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 4 2024-09-10T18:51:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a737fe640 15 client.24620.objecter _session_command_op_remove 1 2 2024-09-10T18:51:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 librados: watch_flush enter 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 librados: watch_flush exit 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 client.24620.objecter close_session for osd.1 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 >> v1:172.21.15.83:6805/2309624637 conn(0x7f3a7407c5a0 legacy=0x7f3a7407ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter put_session s=0x7f3a7407bef0 osd=1 1 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 20 client.24620.objecter shutdown clearing up homeless session... 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 10 client.24620.objecter successfully canceled tick 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 >> v1:172.21.15.83:6800/3071793735 conn(0x7f3a74071b90 legacy=0x7f3a74074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 >> v1:172.21.15.83:6790/0 conn(0x7f3a8c105630 legacy=0x7f3a8c104d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a8b7fe640 1 -- 172.21.15.83:0/3088802816 reap_dead start 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 shutdown_connections 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 >> 172.21.15.83:0/3088802816 conn(0x7f3a8c075290 msgr2=0x7f3a8c107a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 shutdown_connections 2024-09-10T18:51:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 -- 172.21.15.83:0/3088802816 wait complete. 2024-09-10T18:51:46.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.148+0000 7f3a91426640 1 librados: shutdown 2024-09-10T18:51:46.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.272+0000 7fe0fbdec640 1 -- 172.21.15.83:0/654447378 >> v1:172.21.15.83:6790/0 conn(0x7fe0f40fd640 legacy=0x7fe0f40fda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.272+0000 7fe0fbdec640 1 -- 172.21.15.83:0/654447378 shutdown_connections 2024-09-10T18:51:46.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.272+0000 7fe0fbdec640 1 -- 172.21.15.83:0/654447378 >> 172.21.15.83:0/654447378 conn(0x7fe0f4075290 msgr2=0x7fe0f40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.272+0000 7fe0fbdec640 1 -- 172.21.15.83:0/654447378 shutdown_connections 2024-09-10T18:51:46.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.272+0000 7fe0fbdec640 1 -- 172.21.15.83:0/654447378 wait complete. 2024-09-10T18:51:46.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 librados: starting msgr at 2024-09-10T18:51:46.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 librados: starting objecter 2024-09-10T18:51:46.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 Processor -- start 2024-09-10T18:51:46.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- start start 2024-09-10T18:51:46.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 librados: setting wanted keys 2024-09-10T18:51:46.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 librados: calling monclient init 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0f411c190 con 0x7fe0f40fd640 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0f411d390 con 0x7fe0f4108f40 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0f411e590 con 0x7fe0f40fe210 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0f9387640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fe0f40fe210 0x7fe0f4104a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40394/0 (socket says 172.21.15.83:40394) 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0f9387640 1 -- 172.21.15.83:0/2750980838 learned_addr learned my addr 172.21.15.83:0/2750980838 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0f9b88640 1 --1- 172.21.15.83:0/2750980838 >> v1:172.21.15.83:6789/0 conn(0x7fe0f40fd640 0x7fe0f4104320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60286/0 (socket says 172.21.15.83:60286) 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe0f40fd640 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe0f40fe210 2024-09-10T18:51:46.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter ms_handle_connect 0x7fe0f4108f40 2024-09-10T18:51:46.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144289304 0 0) 0x7fe0f411e590 con 0x7fe0f40fe210 2024-09-10T18:51:46.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e00038e0 con 0x7fe0f40fe210 2024-09-10T18:51:46.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537441572 0 0) 0x7fe0f411d390 con 0x7fe0f4108f40 2024-09-10T18:51:46.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f411e590 con 0x7fe0f4108f40 2024-09-10T18:51:46.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406761570 0 0) 0x7fe0f411c190 con 0x7fe0f40fd640 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f411d390 con 0x7fe0f40fd640 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660130494 0 0) 0x7fe0e00038e0 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0f411c190 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe0e40019f0 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3716555731 0 0) 0x7fe0f411e590 con 0x7fe0f4108f40 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0e00038e0 con 0x7fe0f4108f40 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2537478896 0 0) 0x7fe0f411c190 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 >> v1:172.21.15.165:6789/0 conn(0x7fe0f4108f40 legacy=0x7fe0f411a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 >> v1:172.21.15.83:6789/0 conn(0x7fe0f40fd640 legacy=0x7fe0f4104320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f411f790 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f411c380 con 0x7fe0f40fe210 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 10 client.24626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f411c7d0 con 0x7fe0f40fe210 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0fbdec640 1 librados: init done 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe0e4002370 con 0x7fe0f40fe210 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.276+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe0e4005970 con 0x7fe0f40fe210 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0fbdec640 10 client.24626.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0fbdec640 15 client.24626.objecter _session_command_op_assign -1 1 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0fbdec640 20 client.24626.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0fbdec640 10 client.24626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe0d80028b0 con 0x7fe0f40fe210 2024-09-10T18:51:46.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.280+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe0e40024e0 con 0x7fe0f40fe210 2024-09-10T18:51:46.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7fe0e4089d10 con 0x7fe0f40fe210 2024-09-10T18:51:46.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter ms_dispatch 0x7fe0f4000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:46.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 3 client.24626.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:46.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 3 client.24626.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/1132357957 >> v1:172.21.15.165:6789/0 conn(0x7fdd6c1042a0 legacy=0x7fdd6c106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter checking command 1 2024-09-10T18:51:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/1132357957 shutdown_connections 2024-09-10T18:51:46.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/1132357957 >> 172.21.15.83:0/1132357957 conn(0x7fdd6c075290 msgr2=0x7fdd6c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter _get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 15 client.24626.objecter _session_command_op_remove -1 1 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter _get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 15 client.24626.objecter _session_command_op_assign 5 1 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 4 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter _send_command 1 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.165:6804/571173777 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0e007fdf0 con 0x7fe0e007c0f0 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter dump_active .. 0 homeless 2024-09-10T18:51:46.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (unknown 2268335905 0 0) 0x7fe0e408a110 con 0x7fe0f40fe210 2024-09-10T18:51:46.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/1132357957 shutdown_connections 2024-09-10T18:51:46.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/1132357957 wait complete. 2024-09-10T18:51:46.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter ms_handle_connect 0x7fe0e0071630 2024-09-10T18:51:46.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 librados: starting msgr at 2024-09-10T18:51:46.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 librados: starting objecter 2024-09-10T18:51:46.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter ms_handle_connect 0x7fe0e007c0f0 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 Processor -- start 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- start start 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 librados: setting wanted keys 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 librados: calling monclient init 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd6c119d90 con 0x7fdd6c107e30 2024-09-10T18:51:46.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd6c11af50 con 0x7fdd6c1014e0 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd6c11c150 con 0x7fdd6c1042a0 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd72a2b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fdd6c107e30 0x7fdd6c118490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60314/0 (socket says 172.21.15.83:60314) 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd72a2b640 1 -- 172.21.15.83:0/918873762 learned_addr learned my addr 172.21.15.83:0/918873762 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd6c1042a0 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd6c107e30 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506457919 0 0) 0x7fdd6c11c150 con 0x7fdd6c1042a0 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd58003120 con 0x7fdd6c1042a0 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042764837 0 0) 0x7fdd6c119d90 con 0x7fdd6c107e30 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c11c150 con 0x7fdd6c107e30 2024-09-10T18:51:46.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter ms_handle_connect 0x7fdd6c1014e0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774367311 0 0) 0x7fdd6c11af50 con 0x7fdd6c1014e0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6c119d90 con 0x7fdd6c1014e0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012768963 0 0) 0x7fdd6c11c150 con 0x7fdd6c107e30 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c11af50 con 0x7fdd6c107e30 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296347817 0 0) 0x7fdd58003120 con 0x7fdd6c1042a0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c11c150 con 0x7fdd6c1042a0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdd64001c60 con 0x7fdd6c107e30 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdd68002df0 con 0x7fdd6c1042a0 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 338918209 0 0) 0x7fdd6c11af50 con 0x7fdd6c107e30 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 >> v1:172.21.15.83:6790/0 conn(0x7fdd6c1042a0 legacy=0x7fdd6c0fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 >> v1:172.21.15.165:6789/0 conn(0x7fdd6c1014e0 legacy=0x7fdd6c0fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c11d350 con 0x7fdd6c107e30 2024-09-10T18:51:46.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe0e007fdf0 con 0x7fe0e007c0f0 2024-09-10T18:51:46.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter ms_dispatch 0x7fe0f4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:46.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c119f80 con 0x7fdd6c107e30 2024-09-10T18:51:46.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 10 client.24626.objecter _finish_command 1 = system:0 2024-09-10T18:51:46.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 4 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fe0ea7fc640 15 client.24626.objecter _session_command_op_remove 5 1 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c11a450 con 0x7fdd6c107e30 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd7448e640 1 librados: init done 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.284+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fdd640022e0 con 0x7fdd6c107e30 2024-09-10T18:51:46.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdd64005df0 con 0x7fdd6c107e30 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[18629]: cluster 2024-09-10T18:51:44.488139+0000 mgr.y (mgr.24526) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[18629]: cluster 2024-09-10T18:51:45.217571+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[18629]: cluster 2024-09-10T18:51:45.221324+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[32381]: cluster 2024-09-10T18:51:44.488139+0000 mgr.y (mgr.24526) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[32381]: cluster 2024-09-10T18:51:45.217571+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:51:46.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:46 smithi083 bash[32381]: cluster 2024-09-10T18:51:45.221324+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd7448e640 10 client.14715.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd7448e640 15 client.14715.objecter _session_command_op_assign -1 1 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd7448e640 20 client.14715.objecter _get_session osd=-1 returning homeless 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd7448e640 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd6c0fcc60 con 0x7fdd6c107e30 2024-09-10T18:51:46.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fdd64001e40 con 0x7fdd6c107e30 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7fdd6408a320 con 0x7fdd6c107e30 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter ms_dispatch 0x7fdd6c000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 3 client.14715.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 3 client.14715.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter checking command 1 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter _get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 15 client.14715.objecter _session_command_op_remove -1 1 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter _get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 15 client.14715.objecter _session_command_op_assign 3 1 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 4 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter _send_command 1 2024-09-10T18:51:46.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6813/1279688021 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd58082180 con 0x7fdd5807e480 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter dump_active .. 0 homeless 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (unknown 2268335905 0 0) 0x7fdd6408a750 con 0x7fdd6c107e30 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter ms_handle_connect 0x7fdd58073ac0 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter ms_handle_connect 0x7fdd5807e480 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdd58082180 con 0x7fdd5807e480 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter ms_dispatch 0x7fdd6c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 10 client.14715.objecter _finish_command 1 = system:0 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 4 2024-09-10T18:51:46.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.288+0000 7fdd62ffd640 15 client.14715.objecter _session_command_op_remove 3 1 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 client.24626.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 15 client.24626.objecter _session_command_op_assign -1 2 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter _get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter _get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 15 client.24626.objecter _session_command_op_remove -1 2 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter get_session s=0x7fe0e007ba40 osd=5 3 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 15 client.24626.objecter _session_command_op_assign 5 2 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 4 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 client.24626.objecter _send_command 2 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 --> v1:172.21.15.165:6804/571173777 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe0d8001190 con 0x7fe0e007c0f0 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0ea7fc640 1 -- 172.21.15.83:0/2750980838 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3278825725) 0x7fe0d8001190 con 0x7fe0e007c0f0 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0ea7fc640 10 client.24626.objecter ms_dispatch 0x7fe0f4000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:46.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0ea7fc640 10 client.24626.objecter _finish_command 2 = system:0 2024-09-10T18:51:46.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0ea7fc640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 4 2024-09-10T18:51:46.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0ea7fc640 15 client.24626.objecter _session_command_op_remove 5 2 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 librados: watch_flush enter 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 librados: watch_flush exit 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 client.24626.objecter close_session for osd.5 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 >> v1:172.21.15.165:6804/571173777 conn(0x7fe0e007c0f0 legacy=0x7fe0e007e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter put_session s=0x7fe0e007ba40 osd=5 1 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 20 client.24626.objecter shutdown clearing up homeless session... 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 10 client.24626.objecter successfully canceled tick 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe0e0071630 legacy=0x7fe0e0073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 >> v1:172.21.15.83:6790/0 conn(0x7fe0f40fe210 legacy=0x7fe0f4104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fa389640 1 -- 172.21.15.83:0/2750980838 reap_dead start 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 shutdown_connections 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 >> 172.21.15.83:0/2750980838 conn(0x7fe0f4075290 msgr2=0x7fe0f40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 shutdown_connections 2024-09-10T18:51:46.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 -- 172.21.15.83:0/2750980838 wait complete. 2024-09-10T18:51:46.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.300+0000 7fe0fbdec640 1 librados: shutdown 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 10 client.14715.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 15 client.14715.objecter _session_command_op_assign -1 2 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 20 client.14715.objecter _get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 20 client.14715.objecter _get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 15 client.14715.objecter _session_command_op_remove -1 2 2024-09-10T18:51:46.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 20 client.14715.objecter get_session s=0x7fdd5807ddd0 osd=3 3 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 15 client.14715.objecter _session_command_op_assign 3 2 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 4 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 10 client.14715.objecter _send_command 2 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 --> v1:172.21.15.83:6813/1279688021 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdd6c04bf60 con 0x7fdd5807e480 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd62ffd640 1 -- 172.21.15.83:0/918873762 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 243145291) 0x7fdd6c04bf60 con 0x7fdd5807e480 2024-09-10T18:51:46.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd62ffd640 10 client.14715.objecter ms_dispatch 0x7fdd6c000b90 command_reply(tid 2: 0 ) v1 2024-09-10T18:51:46.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd62ffd640 10 client.14715.objecter _finish_command 2 = system:0 2024-09-10T18:51:46.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd62ffd640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 4 2024-09-10T18:51:46.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.304+0000 7fdd62ffd640 15 client.14715.objecter _session_command_op_remove 3 2 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 10 librados: watch_flush enter 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 10 librados: watch_flush exit 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 10 client.14715.objecter close_session for osd.3 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 >> v1:172.21.15.83:6813/1279688021 conn(0x7fdd5807e480 legacy=0x7fdd58080910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 20 client.14715.objecter put_session s=0x7fdd5807ddd0 osd=3 1 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 20 client.14715.objecter shutdown clearing up homeless session... 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 10 client.14715.objecter successfully canceled tick 2024-09-10T18:51:46.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 >> v1:172.21.15.83:6800/3071793735 conn(0x7fdd58073ac0 legacy=0x7fdd58075f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 >> v1:172.21.15.83:6789/0 conn(0x7fdd6c107e30 legacy=0x7fdd6c118490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd72a2b640 1 -- 172.21.15.83:0/918873762 reap_dead start 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 shutdown_connections 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 >> 172.21.15.83:0/918873762 conn(0x7fdd6c075290 msgr2=0x7fdd6c10b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 shutdown_connections 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 -- 172.21.15.83:0/918873762 wait complete. 2024-09-10T18:51:46.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:46.308+0000 7fdd7448e640 1 librados: shutdown 2024-09-10T18:51:46.447 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:46 smithi165 bash[22591]: cluster 2024-09-10T18:51:44.488139+0000 mgr.y (mgr.24526) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:51:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:46 smithi165 bash[22591]: cluster 2024-09-10T18:51:45.217571+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:51:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:46 smithi165 bash[22591]: cluster 2024-09-10T18:51:45.221324+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:51:47.473 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:47 smithi165 bash[22591]: cluster 2024-09-10T18:51:46.597075+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-09-10T18:51:47.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:47 smithi165 bash[22591]: cluster 2024-09-10T18:51:46.600450+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-09-10T18:51:47.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:47 smithi165 bash[22591]: cluster 2024-09-10T18:51:46.630497+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 deep-scrub starts 2024-09-10T18:51:47.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:47 smithi165 bash[22591]: cluster 2024-09-10T18:51:46.633821+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 deep-scrub ok 2024-09-10T18:51:47.569 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[18629]: cluster 2024-09-10T18:51:46.597075+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[18629]: cluster 2024-09-10T18:51:46.600450+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[18629]: cluster 2024-09-10T18:51:46.630497+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 deep-scrub starts 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[18629]: cluster 2024-09-10T18:51:46.633821+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 deep-scrub ok 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[32381]: cluster 2024-09-10T18:51:46.597075+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.c scrub starts 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[32381]: cluster 2024-09-10T18:51:46.600450+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.c scrub ok 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[32381]: cluster 2024-09-10T18:51:46.630497+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 deep-scrub starts 2024-09-10T18:51:47.570 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:47 smithi083 bash[32381]: cluster 2024-09-10T18:51:46.633821+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 deep-scrub ok 2024-09-10T18:51:47.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:48.035 INFO:teuthology.orchestra.run.smithi083.stdout:55834574952 2024-09-10T18:51:48.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T18:51:48.261 INFO:teuthology.orchestra.run.smithi083.stdout:184683593789 2024-09-10T18:51:48.261 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T18:51:48.262 INFO:teuthology.orchestra.run.smithi083.stdout:124554051668 2024-09-10T18:51:48.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:46.488897+0000 mgr.y (mgr.24526) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:51:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.154900+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.158395+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.291454+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.f deep-scrub starts 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.304143+0000 osd.7 (osd.7) 56 : cluster [DBG] 3.f deep-scrub ok 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.566164+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[18629]: cluster 2024-09-10T18:51:47.570359+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:46.488897+0000 mgr.y (mgr.24526) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.154900+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.158395+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.291454+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.f deep-scrub starts 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.304143+0000 osd.7 (osd.7) 56 : cluster [DBG] 3.f deep-scrub ok 2024-09-10T18:51:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.566164+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:51:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:48 smithi083 bash[32381]: cluster 2024-09-10T18:51:47.570359+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:51:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:46.488897+0000 mgr.y (mgr.24526) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:51:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.154900+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:51:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.158395+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:51:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.291454+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.f deep-scrub starts 2024-09-10T18:51:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.304143+0000 osd.7 (osd.7) 56 : cluster [DBG] 3.f deep-scrub ok 2024-09-10T18:51:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.566164+0000 osd.4 (osd.4) 67 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:51:48.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:48 smithi165 bash[22591]: cluster 2024-09-10T18:51:47.570359+0000 osd.4 (osd.4) 68 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:51:49.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:49 smithi083 bash[18629]: audit 2024-09-10T18:51:47.472596+0000 mgr.y (mgr.24526) 90 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:49 smithi083 bash[18629]: cluster 2024-09-10T18:51:48.489474+0000 mgr.y (mgr.24526) 91 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:49 smithi083 bash[32381]: audit 2024-09-10T18:51:47.472596+0000 mgr.y (mgr.24526) 90 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:49.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:49 smithi083 bash[32381]: cluster 2024-09-10T18:51:48.489474+0000 mgr.y (mgr.24526) 91 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:49 smithi165 bash[22591]: audit 2024-09-10T18:51:47.472596+0000 mgr.y (mgr.24526) 90 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:49.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:49 smithi165 bash[22591]: cluster 2024-09-10T18:51:48.489474+0000 mgr.y (mgr.24526) 91 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:51:50.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:50 smithi083 bash[18629]: cluster 2024-09-10T18:51:49.220419+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:51:50.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:50 smithi083 bash[18629]: cluster 2024-09-10T18:51:49.223891+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:51:50.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:50 smithi083 bash[32381]: cluster 2024-09-10T18:51:49.220419+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:51:50.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:50 smithi083 bash[32381]: cluster 2024-09-10T18:51:49.223891+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:51:50.556 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:50.556 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:50.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:50 smithi165 bash[22591]: cluster 2024-09-10T18:51:49.220419+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:51:50.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:50 smithi165 bash[22591]: cluster 2024-09-10T18:51:49.223891+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.490884+0000 mgr.y (mgr.24526) 92 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.537553+0000 osd.1 (osd.1) 55 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.544175+0000 osd.1 (osd.1) 56 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.622882+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.626501+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.671026+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[18629]: cluster 2024-09-10T18:51:50.674805+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.490884+0000 mgr.y (mgr.24526) 92 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.537553+0000 osd.1 (osd.1) 55 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:51:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.544175+0000 osd.1 (osd.1) 56 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:51:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.622882+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:51:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.626501+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:51:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.671026+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T18:51:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:51 smithi083 bash[32381]: cluster 2024-09-10T18:51:50.674805+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T18:51:51.571 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:51.572 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:51.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:51:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.490884+0000 mgr.y (mgr.24526) 92 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T18:51:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.537553+0000 osd.1 (osd.1) 55 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:51:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.544175+0000 osd.1 (osd.1) 56 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:51:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.622882+0000 osd.0 (osd.0) 55 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:51:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.626501+0000 osd.0 (osd.0) 56 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:51:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.671026+0000 osd.5 (osd.5) 49 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T18:51:51.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:51 smithi165 bash[22591]: cluster 2024-09-10T18:51:50.674805+0000 osd.5 (osd.5) 50 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T18:51:52.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.364+0000 7f25f8c2d640 1 -- 172.21.15.83:0/3111385011 >> v1:172.21.15.83:6790/0 conn(0x7f25f40fb370 legacy=0x7f25f40fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:52.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- 172.21.15.83:0/3111385011 shutdown_connections 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- 172.21.15.83:0/3111385011 >> 172.21.15.83:0/3111385011 conn(0x7f25f4075290 msgr2=0x7f25f40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- 172.21.15.83:0/3111385011 shutdown_connections 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- 172.21.15.83:0/3111385011 wait complete. 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 librados: starting msgr at 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 librados: starting objecter 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 Processor -- start 2024-09-10T18:51:52.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- start start 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 librados: setting wanted keys 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 librados: calling monclient init 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25f411e090 con 0x7f25f40fbe50 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25f411f250 con 0x7f25f410b1a0 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f8c2d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25f4120450 con 0x7f25f40fb370 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f25f40fb370 0x7f25f4107240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40420/0 (socket says 172.21.15.83:40420) 2024-09-10T18:51:52.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25f259c640 1 -- 172.21.15.83:0/902404306 learned_addr learned my addr 172.21.15.83:0/902404306 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter ms_handle_connect 0x7f25f40fb370 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter ms_handle_connect 0x7f25f40fbe50 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter ms_handle_connect 0x7f25f410b1a0 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023137059 0 0) 0x7f25f411e090 con 0x7f25f40fbe50 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25dc0034e0 con 0x7f25f40fbe50 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866086725 0 0) 0x7f25f4120450 con 0x7f25f40fb370 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f411e090 con 0x7f25f40fb370 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431954746 0 0) 0x7f25f411f250 con 0x7f25f410b1a0 2024-09-10T18:51:52.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f4120450 con 0x7f25f410b1a0 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58742443 0 0) 0x7f25f411e090 con 0x7f25f40fb370 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25f411f250 con 0x7f25f40fb370 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f25e40015d0 con 0x7f25f40fb370 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185267924 0 0) 0x7f25dc0034e0 con 0x7f25f40fbe50 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25f411e090 con 0x7f25f40fbe50 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f25e8001840 con 0x7f25f40fbe50 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 774976756 0 0) 0x7f25f4120450 con 0x7f25f410b1a0 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25dc0034e0 con 0x7f25f410b1a0 2024-09-10T18:51:52.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.368+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f25ec001bb0 con 0x7f25f410b1a0 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 747057080 0 0) 0x7f25f411e090 con 0x7f25f40fbe50 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 >> v1:172.21.15.83:6790/0 conn(0x7f25f40fb370 legacy=0x7f25f4107240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 >> v1:172.21.15.165:6789/0 conn(0x7f25f410b1a0 legacy=0x7f25f411c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f4121650 con 0x7f25f40fbe50 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f411e280 con 0x7f25f40fbe50 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25f8c2d640 10 client.14721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f411e750 con 0x7f25f40fbe50 2024-09-10T18:51:52.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25f8c2d640 1 librados: init done 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f25e8001ec0 con 0x7f25f40fbe50 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f25e80030d0 con 0x7f25f40fbe50 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d4005010 con 0x7f25f40fbe50 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f25e8001a60 con 0x7f25f40fbe50 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7f25e8087430 con 0x7f25f40fbe50 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 10 client.14721.objecter ms_dispatch 0x7f25f4000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:51:52.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 3 client.14721.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:51:52.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 3 client.14721.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:51:52.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.372+0000 7f25e2ffd640 20 client.14721.objecter dump_active .. 0 homeless 2024-09-10T18:51:52.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.376+0000 7f25e2ffd640 10 client.14721.objecter ms_handle_connect 0x7f25dc071760 2024-09-10T18:51:52.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.380+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f25e8053c80 con 0x7f25f40fbe50 2024-09-10T18:51:52.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:52 smithi083 bash[18629]: cluster 2024-09-10T18:51:51.614406+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:51:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:52 smithi083 bash[18629]: cluster 2024-09-10T18:51:51.619493+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:51:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:52 smithi083 bash[32381]: cluster 2024-09-10T18:51:51.614406+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:51:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:52 smithi083 bash[32381]: cluster 2024-09-10T18:51:51.619493+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:51:52.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.564+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f25d40052c0 con 0x7f25f40fbe50 2024-09-10T18:51:52.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.564+0000 7f25e2ffd640 1 -- 172.21.15.83:0/902404306 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 122566035) 0x7f25e8053aa0 con 0x7f25f40fbe50 2024-09-10T18:51:52.568 INFO:teuthology.orchestra.run.smithi083.stdout:150323855434 2024-09-10T18:51:52.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.564+0000 7f25f8c2d640 10 librados: watch_flush enter 2024-09-10T18:51:52.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.564+0000 7f25f8c2d640 10 librados: watch_flush exit 2024-09-10T18:51:52.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.564+0000 7f25f8c2d640 20 client.14721.objecter shutdown clearing up homeless session... 2024-09-10T18:51:52.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 10 client.14721.objecter successfully canceled tick 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 >> v1:172.21.15.83:6800/3071793735 conn(0x7f25dc071760 legacy=0x7f25dc073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 >> v1:172.21.15.83:6789/0 conn(0x7f25f40fbe50 legacy=0x7f25f4107950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 shutdown_connections 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 >> 172.21.15.83:0/902404306 conn(0x7f25f4075290 msgr2=0x7f25f40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 shutdown_connections 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 -- 172.21.15.83:0/902404306 wait complete. 2024-09-10T18:51:52.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:51:52.568+0000 7f25f8c2d640 1 librados: shutdown 2024-09-10T18:51:52.588 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:52.590 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:52.590 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:51:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:52 smithi165 bash[22591]: cluster 2024-09-10T18:51:51.614406+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:51:52.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:52 smithi165 bash[22591]: cluster 2024-09-10T18:51:51.619493+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:51:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:51:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:51:52] "GET /metrics HTTP/1.1" 200 220105 "" "Prometheus/2.43.0" 2024-09-10T18:51:53.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[18629]: cluster 2024-09-10T18:51:52.318253+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-09-10T18:51:53.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[18629]: cluster 2024-09-10T18:51:52.321655+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-09-10T18:51:53.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[18629]: cluster 2024-09-10T18:51:52.491521+0000 mgr.y (mgr.24526) 93 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:53.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[18629]: audit 2024-09-10T18:51:52.567586+0000 mon.a (mon.0) 854 : audit [DBG] from='client.? v1:172.21.15.83:0/902404306' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:51:53.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[32381]: cluster 2024-09-10T18:51:52.318253+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-09-10T18:51:53.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[32381]: cluster 2024-09-10T18:51:52.321655+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-09-10T18:51:53.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[32381]: cluster 2024-09-10T18:51:52.491521+0000 mgr.y (mgr.24526) 93 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:53.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:53 smithi083 bash[32381]: audit 2024-09-10T18:51:52.567586+0000 mon.a (mon.0) 854 : audit [DBG] from='client.? v1:172.21.15.83:0/902404306' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:51:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:53 smithi165 bash[22591]: cluster 2024-09-10T18:51:52.318253+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-09-10T18:51:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:53 smithi165 bash[22591]: cluster 2024-09-10T18:51:52.321655+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-09-10T18:51:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:53 smithi165 bash[22591]: cluster 2024-09-10T18:51:52.491521+0000 mgr.y (mgr.24526) 93 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:53 smithi165 bash[22591]: audit 2024-09-10T18:51:52.567586+0000 mon.a (mon.0) 854 : audit [DBG] from='client.? v1:172.21.15.83:0/902404306' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T18:51:54.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:54 smithi083 bash[18629]: cluster 2024-09-10T18:51:53.425403+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.e scrub starts 2024-09-10T18:51:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:54 smithi083 bash[18629]: cluster 2024-09-10T18:51:53.429200+0000 osd.4 (osd.4) 70 : cluster [DBG] 6.e scrub ok 2024-09-10T18:51:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:54 smithi083 bash[32381]: cluster 2024-09-10T18:51:53.425403+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.e scrub starts 2024-09-10T18:51:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:54 smithi083 bash[32381]: cluster 2024-09-10T18:51:53.429200+0000 osd.4 (osd.4) 70 : cluster [DBG] 6.e scrub ok 2024-09-10T18:51:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:54 smithi165 bash[22591]: cluster 2024-09-10T18:51:53.425403+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.e scrub starts 2024-09-10T18:51:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:54 smithi165 bash[22591]: cluster 2024-09-10T18:51:53.429200+0000 osd.4 (osd.4) 70 : cluster [DBG] 6.e scrub ok 2024-09-10T18:51:55.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855431 got 150323855434 for osd.4 2024-09-10T18:51:55.182 DEBUG:teuthology.parallel:result is None 2024-09-10T18:51:55.417 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[18629]: cluster 2024-09-10T18:51:53.684084+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.e scrub starts 2024-09-10T18:51:55.417 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[18629]: cluster 2024-09-10T18:51:53.688286+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.e scrub ok 2024-09-10T18:51:55.417 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[18629]: cluster 2024-09-10T18:51:54.492173+0000 mgr.y (mgr.24526) 94 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:55.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[32381]: cluster 2024-09-10T18:51:53.684084+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.e scrub starts 2024-09-10T18:51:55.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[32381]: cluster 2024-09-10T18:51:53.688286+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.e scrub ok 2024-09-10T18:51:55.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:55 smithi083 bash[32381]: cluster 2024-09-10T18:51:54.492173+0000 mgr.y (mgr.24526) 94 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:55 smithi165 bash[22591]: cluster 2024-09-10T18:51:53.684084+0000 osd.2 (osd.2) 31 : cluster [DBG] 2.e scrub starts 2024-09-10T18:51:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:55 smithi165 bash[22591]: cluster 2024-09-10T18:51:53.688286+0000 osd.2 (osd.2) 32 : cluster [DBG] 2.e scrub ok 2024-09-10T18:51:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:55 smithi165 bash[22591]: cluster 2024-09-10T18:51:54.492173+0000 mgr.y (mgr.24526) 94 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.295403+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.299381+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.505058+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.f scrub starts 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.508546+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.f scrub ok 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.515948+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[18629]: cluster 2024-09-10T18:51:55.522506+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.295403+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.299381+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:51:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.505058+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.f scrub starts 2024-09-10T18:51:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.508546+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.f scrub ok 2024-09-10T18:51:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.515948+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:51:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:56 smithi083 bash[32381]: cluster 2024-09-10T18:51:55.522506+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:51:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.295403+0000 osd.6 (osd.6) 39 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:51:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.299381+0000 osd.6 (osd.6) 40 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:51:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.505058+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.f scrub starts 2024-09-10T18:51:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.508546+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.f scrub ok 2024-09-10T18:51:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.515948+0000 osd.3 (osd.3) 65 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:51:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:56 smithi165 bash[22591]: cluster 2024-09-10T18:51:55.522506+0000 osd.3 (osd.3) 66 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:51:57.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:51:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.151382+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:51:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.154866+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.493686+0000 mgr.y (mgr.24526) 95 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.571655+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.575144+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.649052+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.652665+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.748100+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-09-10T18:51:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:57 smithi165 bash[22591]: cluster 2024-09-10T18:51:56.751382+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.151382+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.154866+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.493686+0000 mgr.y (mgr.24526) 95 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.571655+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.575144+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.649052+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.652665+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.748100+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-09-10T18:51:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[18629]: cluster 2024-09-10T18:51:56.751382+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.151382+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.154866+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.493686+0000 mgr.y (mgr.24526) 95 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.571655+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.575144+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.649052+0000 osd.2 (osd.2) 33 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.652665+0000 osd.2 (osd.2) 34 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.748100+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-09-10T18:51:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:57 smithi083 bash[32381]: cluster 2024-09-10T18:51:56.751382+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:57.416065+0000 osd.4 (osd.4) 71 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:57.420479+0000 osd.4 (osd.4) 72 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: audit 2024-09-10T18:51:57.483979+0000 mgr.y (mgr.24526) 96 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:57.530154+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:57.533272+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:58.417071+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[18629]: cluster 2024-09-10T18:51:58.430861+0000 osd.1 (osd.1) 60 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:57.416065+0000 osd.4 (osd.4) 71 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:51:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:57.420479+0000 osd.4 (osd.4) 72 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:51:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: audit 2024-09-10T18:51:57.483979+0000 mgr.y (mgr.24526) 96 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:57.530154+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:51:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:57.533272+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:51:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:58.417071+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T18:51:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:58 smithi083 bash[32381]: cluster 2024-09-10T18:51:58.430861+0000 osd.1 (osd.1) 60 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:57.416065+0000 osd.4 (osd.4) 71 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:57.420479+0000 osd.4 (osd.4) 72 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: audit 2024-09-10T18:51:57.483979+0000 mgr.y (mgr.24526) 96 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:57.530154+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:57.533272+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:51:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:58.417071+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T18:51:59.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:58 smithi165 bash[22591]: cluster 2024-09-10T18:51:58.430861+0000 osd.1 (osd.1) 60 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T18:52:00.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[18629]: cluster 2024-09-10T18:51:58.494493+0000 mgr.y (mgr.24526) 97 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[18629]: cluster 2024-09-10T18:51:58.823028+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[18629]: cluster 2024-09-10T18:51:58.828393+0000 osd.5 (osd.5) 54 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[18629]: audit 2024-09-10T18:51:59.541607+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[32381]: cluster 2024-09-10T18:51:58.494493+0000 mgr.y (mgr.24526) 97 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[32381]: cluster 2024-09-10T18:51:58.823028+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[32381]: cluster 2024-09-10T18:51:58.828393+0000 osd.5 (osd.5) 54 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:52:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:51:59 smithi083 bash[32381]: audit 2024-09-10T18:51:59.541607+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:59 smithi165 bash[22591]: cluster 2024-09-10T18:51:58.494493+0000 mgr.y (mgr.24526) 97 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T18:52:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:59 smithi165 bash[22591]: cluster 2024-09-10T18:51:58.823028+0000 osd.5 (osd.5) 53 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:52:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:59 smithi165 bash[22591]: cluster 2024-09-10T18:51:58.828393+0000 osd.5 (osd.5) 54 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:52:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:51:59 smithi165 bash[22591]: audit 2024-09-10T18:51:59.541607+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:01.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:00 smithi083 bash[18629]: cluster 2024-09-10T18:51:59.557733+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:52:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:00 smithi083 bash[18629]: cluster 2024-09-10T18:51:59.561293+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:52:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:00 smithi083 bash[32381]: cluster 2024-09-10T18:51:59.557733+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:52:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:00 smithi083 bash[32381]: cluster 2024-09-10T18:51:59.561293+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:52:01.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:00 smithi165 bash[22591]: cluster 2024-09-10T18:51:59.557733+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:52:01.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:00 smithi165 bash[22591]: cluster 2024-09-10T18:51:59.561293+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:52:01.715 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:01.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[18629]: cluster 2024-09-10T18:52:00.387033+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:52:01.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[18629]: cluster 2024-09-10T18:52:00.390379+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:52:01.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[18629]: cluster 2024-09-10T18:52:00.495984+0000 mgr.y (mgr.24526) 98 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:01.854 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[18629]: cluster 2024-09-10T18:52:00.588310+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[18629]: cluster 2024-09-10T18:52:00.591627+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[32381]: cluster 2024-09-10T18:52:00.387033+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[32381]: cluster 2024-09-10T18:52:00.390379+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[32381]: cluster 2024-09-10T18:52:00.495984+0000 mgr.y (mgr.24526) 98 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[32381]: cluster 2024-09-10T18:52:00.588310+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:52:01.855 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:01 smithi083 bash[32381]: cluster 2024-09-10T18:52:00.591627+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:52:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[22591]: cluster 2024-09-10T18:52:00.387033+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:52:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[22591]: cluster 2024-09-10T18:52:00.390379+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:52:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[22591]: cluster 2024-09-10T18:52:00.495984+0000 mgr.y (mgr.24526) 98 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[22591]: cluster 2024-09-10T18:52:00.588310+0000 osd.3 (osd.3) 69 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:52:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:01 smithi165 bash[22591]: cluster 2024-09-10T18:52:00.591627+0000 osd.3 (osd.3) 70 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:52:03.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:02] "GET /metrics HTTP/1.1" 200 220116 "" "Prometheus/2.43.0" 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[18629]: cluster 2024-09-10T18:52:01.411859+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[18629]: cluster 2024-09-10T18:52:01.415693+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[18629]: cluster 2024-09-10T18:52:01.478973+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[18629]: cluster 2024-09-10T18:52:01.484710+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[32381]: cluster 2024-09-10T18:52:01.411859+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[32381]: cluster 2024-09-10T18:52:01.415693+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:52:03.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[32381]: cluster 2024-09-10T18:52:01.478973+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:52:03.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:02 smithi083 bash[32381]: cluster 2024-09-10T18:52:01.484710+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:52:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:02 smithi165 bash[22591]: cluster 2024-09-10T18:52:01.411859+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:52:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:02 smithi165 bash[22591]: cluster 2024-09-10T18:52:01.415693+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:52:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:02 smithi165 bash[22591]: cluster 2024-09-10T18:52:01.478973+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:52:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:02 smithi165 bash[22591]: cluster 2024-09-10T18:52:01.484710+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:52:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[18629]: cluster 2024-09-10T18:52:02.350336+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.a scrub starts 2024-09-10T18:52:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[18629]: cluster 2024-09-10T18:52:02.355964+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.a scrub ok 2024-09-10T18:52:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[18629]: cluster 2024-09-10T18:52:02.496692+0000 mgr.y (mgr.24526) 99 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[32381]: cluster 2024-09-10T18:52:02.350336+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.a scrub starts 2024-09-10T18:52:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[32381]: cluster 2024-09-10T18:52:02.355964+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.a scrub ok 2024-09-10T18:52:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:03 smithi083 bash[32381]: cluster 2024-09-10T18:52:02.496692+0000 mgr.y (mgr.24526) 99 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:04.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- 172.21.15.83:0/2247651747 >> v1:172.21.15.83:6790/0 conn(0x7faee0102560 legacy=0x7faee0102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- 172.21.15.83:0/2247651747 shutdown_connections 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- 172.21.15.83:0/2247651747 >> 172.21.15.83:0/2247651747 conn(0x7faee0074a50 msgr2=0x7faee00f8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- 172.21.15.83:0/2247651747 shutdown_connections 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- 172.21.15.83:0/2247651747 wait complete. 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 librados: starting msgr at 2024-09-10T18:52:04.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 librados: starting objecter 2024-09-10T18:52:04.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 Processor -- start 2024-09-10T18:52:04.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- start start 2024-09-10T18:52:04.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 librados: setting wanted keys 2024-09-10T18:52:04.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 librados: calling monclient init 2024-09-10T18:52:04.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faee0101b30 con 0x7faee0105340 2024-09-10T18:52:04.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faee0101ce0 con 0x7faee0108e70 2024-09-10T18:52:04.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faee56ef640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faee0101ed0 con 0x7faee0102560 2024-09-10T18:52:04.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faedeffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7faee0102560 0x7faee0101330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48342/0 (socket says 172.21.15.83:48342) 2024-09-10T18:52:04.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faedeffd640 1 -- 172.21.15.83:0/2448312769 learned_addr learned my addr 172.21.15.83:0/2448312769 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:04.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faede7fc640 1 --1- 172.21.15.83:0/2448312769 >> v1:172.21.15.83:6789/0 conn(0x7faee0105340 0x7faee01198a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43208/0 (socket says 172.21.15.83:43208) 2024-09-10T18:52:04.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faec77fe640 10 client.?.objecter ms_handle_connect 0x7faee0102560 2024-09-10T18:52:04.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faec77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:04.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faec77fe640 10 client.?.objecter ms_handle_connect 0x7faee0105340 2024-09-10T18:52:04.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.088+0000 7faec77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:04.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 10 client.?.objecter ms_handle_connect 0x7faee0108e70 2024-09-10T18:52:04.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:04.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752922244 0 0) 0x7faee0101b30 con 0x7faee0105340 2024-09-10T18:52:04.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec80034e0 con 0x7faee0105340 2024-09-10T18:52:04.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059911621 0 0) 0x7faee0101ed0 con 0x7faee0102560 2024-09-10T18:52:04.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee0101b30 con 0x7faee0102560 2024-09-10T18:52:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760855149 0 0) 0x7faee0101ce0 con 0x7faee0108e70 2024-09-10T18:52:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee0101ed0 con 0x7faee0108e70 2024-09-10T18:52:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625498673 0 0) 0x7faee0101b30 con 0x7faee0102560 2024-09-10T18:52:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee0101ce0 con 0x7faee0102560 2024-09-10T18:52:04.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faed00015d0 con 0x7faee0102560 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654145659 0 0) 0x7faec80034e0 con 0x7faee0105340 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faee0101b30 con 0x7faee0105340 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faed4001840 con 0x7faee0105340 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107446097 0 0) 0x7faee0101ce0 con 0x7faee0102560 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 >> v1:172.21.15.165:6789/0 conn(0x7faee0108e70 legacy=0x7faee011cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 >> v1:172.21.15.83:6789/0 conn(0x7faee0105340 legacy=0x7faee01198a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:04.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0121720 con 0x7faee0102560 2024-09-10T18:52:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee011e750 con 0x7faee0102560 2024-09-10T18:52:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faee56ef640 10 client.24644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faee011ec20 con 0x7faee0102560 2024-09-10T18:52:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faee56ef640 1 librados: init done 2024-09-10T18:52:04.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7faed00011f0 con 0x7faee0102560 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faed0003660 con 0x7faee0102560 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.092+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc005010 con 0x7faee0102560 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7faed0001880 con 0x7faee0102560 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7faed008a190 con 0x7faee0102560 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 10 client.24644.objecter ms_dispatch 0x7faee0000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:52:04.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 3 client.24644.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:52:04.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 3 client.24644.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:52:04.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 20 client.24644.objecter dump_active .. 0 homeless 2024-09-10T18:52:04.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.096+0000 7faec77fe640 10 client.24644.objecter ms_handle_connect 0x7faec8071660 2024-09-10T18:52:04.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.104+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7faed0056e60 con 0x7faee0102560 2024-09-10T18:52:04.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:03 smithi165 bash[22591]: cluster 2024-09-10T18:52:02.350336+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.a scrub starts 2024-09-10T18:52:04.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:03 smithi165 bash[22591]: cluster 2024-09-10T18:52:02.355964+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.a scrub ok 2024-09-10T18:52:04.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:03 smithi165 bash[22591]: cluster 2024-09-10T18:52:02.496692+0000 mgr.y (mgr.24526) 99 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.288+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faecc0052c0 con 0x7faee0102560 2024-09-10T18:52:04.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.288+0000 7faec77fe640 1 -- 172.21.15.83:0/2448312769 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 2576209898) 0x7faed0056800 con 0x7faee0102560 2024-09-10T18:52:04.293 INFO:teuthology.orchestra.run.smithi083.stdout:81604378720 2024-09-10T18:52:04.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 10 librados: watch_flush enter 2024-09-10T18:52:04.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 10 librados: watch_flush exit 2024-09-10T18:52:04.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 20 client.24644.objecter shutdown clearing up homeless session... 2024-09-10T18:52:04.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 10 client.24644.objecter successfully canceled tick 2024-09-10T18:52:04.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 >> v1:172.21.15.83:6800/3071793735 conn(0x7faec8071660 legacy=0x7faec8073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:04.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 >> v1:172.21.15.83:6790/0 conn(0x7faee0102560 legacy=0x7faee0101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:04.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 shutdown_connections 2024-09-10T18:52:04.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 >> 172.21.15.83:0/2448312769 conn(0x7faee0074a50 msgr2=0x7faee010bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:04.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 shutdown_connections 2024-09-10T18:52:04.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 -- 172.21.15.83:0/2448312769 wait complete. 2024-09-10T18:52:04.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:04.292+0000 7faee56ef640 1 librados: shutdown 2024-09-10T18:52:05.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:03.180288+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:52:05.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:03.183637+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:03.326296+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.5 deep-scrub starts 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:03.329623+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.5 deep-scrub ok 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: audit 2024-09-10T18:52:04.291017+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.83:0/2448312769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:04.354350+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[18629]: cluster 2024-09-10T18:52:04.358366+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:03.180288+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:03.183637+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:03.326296+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.5 deep-scrub starts 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:03.329623+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.5 deep-scrub ok 2024-09-10T18:52:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: audit 2024-09-10T18:52:04.291017+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.83:0/2448312769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:52:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:04.354350+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:52:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:04 smithi083 bash[32381]: cluster 2024-09-10T18:52:04.358366+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:52:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:03.180288+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:52:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:03.183637+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:52:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:03.326296+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.5 deep-scrub starts 2024-09-10T18:52:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:03.329623+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.5 deep-scrub ok 2024-09-10T18:52:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: audit 2024-09-10T18:52:04.291017+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? v1:172.21.15.83:0/2448312769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T18:52:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:04.354350+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:52:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:04 smithi165 bash[22591]: cluster 2024-09-10T18:52:04.358366+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:52:05.932 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[18629]: cluster 2024-09-10T18:52:04.497396+0000 mgr.y (mgr.24526) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:05.932 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[18629]: cluster 2024-09-10T18:52:05.357090+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:52:05.932 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[18629]: cluster 2024-09-10T18:52:05.361166+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:52:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:05 smithi165 bash[22591]: cluster 2024-09-10T18:52:04.497396+0000 mgr.y (mgr.24526) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:05 smithi165 bash[22591]: cluster 2024-09-10T18:52:05.357090+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:52:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:05 smithi165 bash[22591]: cluster 2024-09-10T18:52:05.361166+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:52:06.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[32381]: cluster 2024-09-10T18:52:04.497396+0000 mgr.y (mgr.24526) 100 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:06.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[32381]: cluster 2024-09-10T18:52:05.357090+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:52:06.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:05 smithi083 bash[32381]: cluster 2024-09-10T18:52:05.361166+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:52:06.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- 172.21.15.83:0/1281049335 >> v1:172.21.15.83:6789/0 conn(0x7f71341042a0 legacy=0x7f7134106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- 172.21.15.83:0/1281049335 shutdown_connections 2024-09-10T18:52:06.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- 172.21.15.83:0/1281049335 >> 172.21.15.83:0/1281049335 conn(0x7f7134075290 msgr2=0x7f71340f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:06.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- 172.21.15.83:0/1281049335 shutdown_connections 2024-09-10T18:52:06.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- 172.21.15.83:0/1281049335 wait complete. 2024-09-10T18:52:06.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 librados: starting msgr at 2024-09-10T18:52:06.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 librados: starting objecter 2024-09-10T18:52:06.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 Processor -- start 2024-09-10T18:52:06.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 -- start start 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 librados: setting wanted keys 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.460+0000 7f7138d08640 1 librados: calling monclient init 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713411afa0 con 0x7f71341014e0 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713411c160 con 0x7f7134107e30 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713411d360 con 0x7f71341042a0 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f713259c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71341014e0 0x7f71340fd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43216/0 (socket says 172.21.15.83:43216) 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f713259c640 1 -- 172.21.15.83:0/1284935656 learned_addr learned my addr 172.21.15.83:0/1284935656 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter ms_handle_connect 0x7f71341014e0 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter ms_handle_connect 0x7f71341042a0 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889459187 0 0) 0x7f713411afa0 con 0x7f71341014e0 2024-09-10T18:52:06.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120003120 con 0x7f71341014e0 2024-09-10T18:52:06.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter ms_handle_connect 0x7f7134107e30 2024-09-10T18:52:06.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233624283 0 0) 0x7f713411d360 con 0x7f71341042a0 2024-09-10T18:52:06.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713411afa0 con 0x7f71341042a0 2024-09-10T18:52:06.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260877808 0 0) 0x7f713411c160 con 0x7f7134107e30 2024-09-10T18:52:06.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713411d360 con 0x7f7134107e30 2024-09-10T18:52:06.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274170733 0 0) 0x7f7120003120 con 0x7f71341014e0 2024-09-10T18:52:06.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f713411c160 con 0x7f71341014e0 2024-09-10T18:52:06.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7124001940 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3758800846 0 0) 0x7f713411afa0 con 0x7f71341042a0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7120003120 con 0x7f71341042a0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f711c0020a0 con 0x7f71341042a0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3418254995 0 0) 0x7f713411c160 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 >> v1:172.21.15.83:6790/0 conn(0x7f71341042a0 legacy=0x7f71340fdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 >> v1:172.21.15.165:6789/0 conn(0x7f7134107e30 legacy=0x7f71341196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713411e560 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713411d550 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 10 client.14733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f713411d9a0 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 librados: init done 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7124002420 con 0x7f71341014e0 2024-09-10T18:52:06.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7124005a80 con 0x7f71341014e0 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7124001fc0 con 0x7f71341014e0 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.464+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7114005010 con 0x7f71341014e0 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7f712408a050 con 0x7f71341014e0 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 10 client.14733.objecter ms_dispatch 0x7f7134000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 3 client.14733.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 3 client.14733.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 20 client.14733.objecter dump_active .. 0 homeless 2024-09-10T18:52:06.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.468+0000 7f712affd640 10 client.14733.objecter ms_handle_connect 0x7f7120071700 2024-09-10T18:52:06.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.472+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7124056cc0 con 0x7f71341014e0 2024-09-10T18:52:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- 172.21.15.83:0/2017015990 >> v1:172.21.15.83:6789/0 conn(0x7fee5010f8a0 legacy=0x7fee50111c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- 172.21.15.83:0/2017015990 shutdown_connections 2024-09-10T18:52:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- 172.21.15.83:0/2017015990 >> 172.21.15.83:0/2017015990 conn(0x7fee500f7080 msgr2=0x7fee500f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- 172.21.15.83:0/2017015990 shutdown_connections 2024-09-10T18:52:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- 172.21.15.83:0/2017015990 wait complete. 2024-09-10T18:52:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 librados: starting msgr at 2024-09-10T18:52:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 librados: starting objecter 2024-09-10T18:52:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 Processor -- start 2024-09-10T18:52:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- start start 2024-09-10T18:52:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 librados: setting wanted keys 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 librados: calling monclient init 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee50122a00 con 0x7fee5010f8a0 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee50123c00 con 0x7fee500fb3c0 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee55bf6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fee50124e00 con 0x7fee5011ec70 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee4effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fee5010f8a0 0x7fee5010c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43234/0 (socket says 172.21.15.83:43234) 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee4effd640 1 -- 172.21.15.83:0/1313976783 learned_addr learned my addr 172.21.15.83:0/1313976783 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter ms_handle_connect 0x7fee5010f8a0 2024-09-10T18:52:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter ms_handle_connect 0x7fee500fb3c0 2024-09-10T18:52:06.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter ms_handle_connect 0x7fee5011ec70 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964766847 0 0) 0x7fee50122a00 con 0x7fee5010f8a0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.556+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee380038b0 con 0x7fee5010f8a0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853215852 0 0) 0x7fee50123c00 con 0x7fee500fb3c0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50122a00 con 0x7fee500fb3c0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390700710 0 0) 0x7fee50124e00 con 0x7fee5011ec70 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee50123c00 con 0x7fee5011ec70 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1332174741 0 0) 0x7fee50122a00 con 0x7fee500fb3c0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee50124e00 con 0x7fee500fb3c0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fee400028f0 con 0x7fee500fb3c0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4001905116 0 0) 0x7fee380038b0 con 0x7fee5010f8a0 2024-09-10T18:52:06.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee50122a00 con 0x7fee5010f8a0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724783344 0 0) 0x7fee50123c00 con 0x7fee5011ec70 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee380038b0 con 0x7fee5011ec70 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fee44001a20 con 0x7fee5010f8a0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fee48002510 con 0x7fee5011ec70 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 868791654 0 0) 0x7fee50124e00 con 0x7fee500fb3c0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 >> v1:172.21.15.83:6790/0 conn(0x7fee5011ec70 legacy=0x7fee50121100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 >> v1:172.21.15.83:6789/0 conn(0x7fee5010f8a0 legacy=0x7fee5010c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50126000 con 0x7fee500fb3c0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee50123df0 con 0x7fee500fb3c0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee55bf6640 10 client.24664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50124240 con 0x7fee500fb3c0 2024-09-10T18:52:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee55bf6640 1 librados: init done 2024-09-10T18:52:06.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fee40002c80 con 0x7fee500fb3c0 2024-09-10T18:52:06.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fee40003230 con 0x7fee500fb3c0 2024-09-10T18:52:06.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.560+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c005010 con 0x7fee500fb3c0 2024-09-10T18:52:06.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fee40001c70 con 0x7fee500fb3c0 2024-09-10T18:52:06.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(104..104 src has 49..104) v4 ==== 6284+0+0 (unknown 796995139 0 0) 0x7fee400875a0 con 0x7fee500fb3c0 2024-09-10T18:52:06.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 10 client.24664.objecter ms_dispatch 0x7fee50000b90 osd_map(104..104 src has 49..104) v4 2024-09-10T18:52:06.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 3 client.24664.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T18:52:06.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 3 client.24664.objecter handle_osd_map decoding full epoch 104 2024-09-10T18:52:06.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 20 client.24664.objecter dump_active .. 0 homeless 2024-09-10T18:52:06.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.564+0000 7fee37fff640 10 client.24664.objecter ms_handle_connect 0x7fee38071770 2024-09-10T18:52:06.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.572+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fee40054210 con 0x7fee500fb3c0 2024-09-10T18:52:06.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378716 got 81604378720 for osd.2 2024-09-10T18:52:06.625 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:06.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.652+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f71140052c0 con 0x7f71341014e0 2024-09-10T18:52:06.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.652+0000 7f712affd640 1 -- 172.21.15.83:0/1284935656 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 363300407) 0x7f7124056660 con 0x7f71341014e0 2024-09-10T18:52:06.657 INFO:teuthology.orchestra.run.smithi083.stdout:34359738487 2024-09-10T18:52:06.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 10 librados: watch_flush enter 2024-09-10T18:52:06.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 10 librados: watch_flush exit 2024-09-10T18:52:06.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 20 client.14733.objecter shutdown clearing up homeless session... 2024-09-10T18:52:06.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 10 client.14733.objecter successfully canceled tick 2024-09-10T18:52:06.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7120071700 legacy=0x7f7120073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 >> v1:172.21.15.83:6789/0 conn(0x7f71341014e0 legacy=0x7f71340fd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 shutdown_connections 2024-09-10T18:52:06.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 >> 172.21.15.83:0/1284935656 conn(0x7f7134075290 msgr2=0x7f71340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:06.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 shutdown_connections 2024-09-10T18:52:06.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 -- 172.21.15.83:0/1284935656 wait complete. 2024-09-10T18:52:06.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.656+0000 7f7138d08640 1 librados: shutdown 2024-09-10T18:52:06.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.752+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fee3c0052c0 con 0x7fee500fb3c0 2024-09-10T18:52:06.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.752+0000 7fee37fff640 1 -- 172.21.15.83:0/1313976783 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 952080543) 0x7fee40053bb0 con 0x7fee500fb3c0 2024-09-10T18:52:06.758 INFO:teuthology.orchestra.run.smithi083.stdout:223338299447 2024-09-10T18:52:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 10 librados: watch_flush enter 2024-09-10T18:52:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 10 librados: watch_flush exit 2024-09-10T18:52:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 20 client.24664.objecter shutdown clearing up homeless session... 2024-09-10T18:52:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 10 client.24664.objecter successfully canceled tick 2024-09-10T18:52:06.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 >> v1:172.21.15.83:6800/3071793735 conn(0x7fee38071770 legacy=0x7fee38073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 >> v1:172.21.15.165:6789/0 conn(0x7fee500fb3c0 legacy=0x7fee5010bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 shutdown_connections 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 >> 172.21.15.83:0/1313976783 conn(0x7fee500f7080 msgr2=0x7fee500f7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 shutdown_connections 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 -- 172.21.15.83:0/1313976783 wait complete. 2024-09-10T18:52:06.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:06.756+0000 7fee55bf6640 1 librados: shutdown 2024-09-10T18:52:07.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[18629]: cluster 2024-09-10T18:52:05.173326+0000 osd.7 (osd.7) 61 : cluster [DBG] 3.e scrub starts 2024-09-10T18:52:07.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[18629]: cluster 2024-09-10T18:52:05.179171+0000 osd.7 (osd.7) 62 : cluster [DBG] 3.e scrub ok 2024-09-10T18:52:07.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[18629]: audit 2024-09-10T18:52:06.657410+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? v1:172.21.15.83:0/1284935656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:52:07.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[32381]: cluster 2024-09-10T18:52:05.173326+0000 osd.7 (osd.7) 61 : cluster [DBG] 3.e scrub starts 2024-09-10T18:52:07.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[32381]: cluster 2024-09-10T18:52:05.179171+0000 osd.7 (osd.7) 62 : cluster [DBG] 3.e scrub ok 2024-09-10T18:52:07.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:06 smithi083 bash[32381]: audit 2024-09-10T18:52:06.657410+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? v1:172.21.15.83:0/1284935656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:52:07.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:06 smithi165 bash[22591]: cluster 2024-09-10T18:52:05.173326+0000 osd.7 (osd.7) 61 : cluster [DBG] 3.e scrub starts 2024-09-10T18:52:07.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:06 smithi165 bash[22591]: cluster 2024-09-10T18:52:05.179171+0000 osd.7 (osd.7) 62 : cluster [DBG] 3.e scrub ok 2024-09-10T18:52:07.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:06 smithi165 bash[22591]: audit 2024-09-10T18:52:06.657410+0000 mon.a (mon.0) 855 : audit [DBG] from='client.? v1:172.21.15.83:0/1284935656' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T18:52:07.749 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- 172.21.15.83:0/1531287853 >> v1:172.21.15.83:6789/0 conn(0x7fc63c0fd610 legacy=0x7fc63c0fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- 172.21.15.83:0/1531287853 shutdown_connections 2024-09-10T18:52:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- 172.21.15.83:0/1531287853 >> 172.21.15.83:0/1531287853 conn(0x7fc63c075290 msgr2=0x7fc63c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- 172.21.15.83:0/1531287853 shutdown_connections 2024-09-10T18:52:07.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- 172.21.15.83:0/1531287853 wait complete. 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 librados: starting msgr at 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 librados: starting objecter 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 Processor -- start 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- start start 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 librados: setting wanted keys 2024-09-10T18:52:07.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 librados: calling monclient init 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c11e630 con 0x7fc63c0fd610 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c11f830 con 0x7fc63c1140c0 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc6437fb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc63c120a30 con 0x7fc63c108f40 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc640d96640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc63c108f40 0x7fc63c119620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48392/0 (socket says 172.21.15.83:48392) 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc640d96640 1 -- 172.21.15.83:0/2732865605 learned_addr learned my addr 172.21.15.83:0/2732865605 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc641597640 1 --1- 172.21.15.83:0/2732865605 >> v1:172.21.15.83:6789/0 conn(0x7fc63c0fd610 0x7fc63c1137c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43252/0 (socket says 172.21.15.83:43252) 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter ms_handle_connect 0x7fc63c0fd610 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter ms_handle_connect 0x7fc63c108f40 2024-09-10T18:52:07.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:07.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter ms_handle_connect 0x7fc63c1140c0 2024-09-10T18:52:07.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:07.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634325645 0 0) 0x7fc63c11e630 con 0x7fc63c0fd610 2024-09-10T18:52:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc628003120 con 0x7fc63c0fd610 2024-09-10T18:52:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600786506 0 0) 0x7fc63c11f830 con 0x7fc63c1140c0 2024-09-10T18:52:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c11e630 con 0x7fc63c1140c0 2024-09-10T18:52:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934764615 0 0) 0x7fc63c120a30 con 0x7fc63c108f40 2024-09-10T18:52:07.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c11f830 con 0x7fc63c108f40 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352985436 0 0) 0x7fc628003120 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc63c120a30 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.887+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc638002d20 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123670863 0 0) 0x7fc63c11f830 con 0x7fc63c108f40 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc628003120 con 0x7fc63c108f40 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc62c001980 con 0x7fc63c108f40 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 125424279 0 0) 0x7fc63c120a30 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 >> v1:172.21.15.83:6790/0 conn(0x7fc63c108f40 legacy=0x7fc63c119620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 >> v1:172.21.15.165:6789/0 conn(0x7fc63c1140c0 legacy=0x7fc63c11cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c121c30 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c120c20 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc6437fb640 10 client.14745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c121070 con 0x7fc63c0fd610 2024-09-10T18:52:07.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc6437fb640 1 librados: init done 2024-09-10T18:52:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc638002270 con 0x7fc63c0fd610 2024-09-10T18:52:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6380035e0 con 0x7fc63c0fd610 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc620005010 con 0x7fc63c0fd610 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc6380017d0 con 0x7fc63c0fd610 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(105..105 src has 49..105) v4 ==== 6284+0+0 (unknown 2684729943 0 0) 0x7fc638089ff0 con 0x7fc63c0fd610 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 10 client.14745.objecter ms_dispatch 0x7fc63c000b90 osd_map(105..105 src has 49..105) v4 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 3 client.14745.objecter handle_osd_map got epochs [105,105] > 0 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 3 client.14745.objecter handle_osd_map decoding full epoch 105 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 20 client.14745.objecter dump_active .. 0 homeless 2024-09-10T18:52:07.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.891+0000 7fc631ffb640 10 client.14745.objecter ms_handle_connect 0x7fc6280716e0 2024-09-10T18:52:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:07.899+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc638056c60 con 0x7fc63c0fd610 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:06.146073+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:06.149688+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:06.498403+0000 mgr.y (mgr.24526) 101 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:06.605978+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:06.609329+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: audit 2024-09-10T18:52:06.754705+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.83:0/1313976783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.451187+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.c scrub starts 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.455042+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.c scrub ok 2024-09-10T18:52:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:06.146073+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:06.149688+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:06.498403+0000 mgr.y (mgr.24526) 101 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:06.605978+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:06.609329+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: audit 2024-09-10T18:52:06.754705+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.83:0/1313976783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.451187+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.c scrub starts 2024-09-10T18:52:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:07 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.455042+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.c scrub ok 2024-09-10T18:52:08.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.075+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc6200052c0 con 0x7fc63c0fd610 2024-09-10T18:52:08.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.075+0000 7fc631ffb640 1 -- 172.21.15.83:0/2732865605 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 635750457) 0x7fc638056600 con 0x7fc63c0fd610 2024-09-10T18:52:08.081 INFO:teuthology.orchestra.run.smithi083.stdout:55834574956 2024-09-10T18:52:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 10 librados: watch_flush enter 2024-09-10T18:52:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 10 librados: watch_flush exit 2024-09-10T18:52:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 20 client.14745.objecter shutdown clearing up homeless session... 2024-09-10T18:52:08.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 10 client.14745.objecter successfully canceled tick 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc6280716e0 legacy=0x7fc628073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 >> v1:172.21.15.83:6789/0 conn(0x7fc63c0fd610 legacy=0x7fc63c1137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 shutdown_connections 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 >> 172.21.15.83:0/2732865605 conn(0x7fc63c075290 msgr2=0x7fc63c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 shutdown_connections 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 -- 172.21.15.83:0/2732865605 wait complete. 2024-09-10T18:52:08.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.079+0000 7fc6437fb640 1 librados: shutdown 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:06.146073+0000 osd.7 (osd.7) 63 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:06.149688+0000 osd.7 (osd.7) 64 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:06.498403+0000 mgr.y (mgr.24526) 101 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:06.605978+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:06.609329+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: audit 2024-09-10T18:52:06.754705+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? v1:172.21.15.83:0/1313976783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.451187+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.c scrub starts 2024-09-10T18:52:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:07 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.455042+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.c scrub ok 2024-09-10T18:52:08.495 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738483 got 34359738487 for osd.0 2024-09-10T18:52:08.496 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.603+0000 7f5243c15640 1 -- 172.21.15.83:0/2123915928 >> v1:172.21.15.83:6789/0 conn(0x7f523c1075a0 legacy=0x7f523c109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.603+0000 7f5243c15640 1 -- 172.21.15.83:0/2123915928 shutdown_connections 2024-09-10T18:52:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.603+0000 7f5243c15640 1 -- 172.21.15.83:0/2123915928 >> 172.21.15.83:0/2123915928 conn(0x7f523c0fa370 msgr2=0x7f523c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.603+0000 7f5243c15640 1 -- 172.21.15.83:0/2123915928 shutdown_connections 2024-09-10T18:52:08.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.603+0000 7f5243c15640 1 -- 172.21.15.83:0/2123915928 wait complete. 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 librados: starting msgr at 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 librados: starting objecter 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 Processor -- start 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- start start 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 librados: setting wanted keys 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 librados: calling monclient init 2024-09-10T18:52:08.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f523c11e270 con 0x7f523c10b130 2024-09-10T18:52:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f523c11f470 con 0x7f523c104810 2024-09-10T18:52:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f523c120670 con 0x7f523c1075a0 2024-09-10T18:52:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f52411b0640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f523c1075a0 0x7f523c101080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48418/0 (socket says 172.21.15.83:48418) 2024-09-10T18:52:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f52411b0640 1 -- 172.21.15.83:0/3774252255 learned_addr learned my addr 172.21.15.83:0/3774252255 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:08.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f52421b2640 1 --1- 172.21.15.83:0/3774252255 >> v1:172.21.15.83:6789/0 conn(0x7f523c10b130 0x7f523c11c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43272/0 (socket says 172.21.15.83:43272) 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter ms_handle_connect 0x7f523c1075a0 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter ms_handle_connect 0x7f523c104810 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209242104 0 0) 0x7f523c120670 con 0x7f523c1075a0 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52240034e0 con 0x7f523c1075a0 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter ms_handle_connect 0x7f523c10b130 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316974156 0 0) 0x7f523c11f470 con 0x7f523c104810 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c120670 con 0x7f523c104810 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543276182 0 0) 0x7f523c11e270 con 0x7f523c10b130 2024-09-10T18:52:08.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523c11f470 con 0x7f523c10b130 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3870645356 0 0) 0x7f52240034e0 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f523c11e270 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5230001510 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3316751798 0 0) 0x7f523c11e270 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 >> v1:172.21.15.165:6789/0 conn(0x7f523c104810 legacy=0x7f523c100970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 >> v1:172.21.15.83:6789/0 conn(0x7f523c10b130 legacy=0x7f523c11c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c121870 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c11f660 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 10 client.24668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c11fab0 con 0x7f523c1075a0 2024-09-10T18:52:08.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f5243c15640 1 librados: init done 2024-09-10T18:52:08.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5230002300 con 0x7f523c1075a0 2024-09-10T18:52:08.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.607+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f52300037b0 con 0x7f523c1075a0 2024-09-10T18:52:08.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5228005010 con 0x7f523c1075a0 2024-09-10T18:52:08.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5230003910 con 0x7f523c1075a0 2024-09-10T18:52:08.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(105..105 src has 49..105) v4 ==== 6284+0+0 (unknown 2684729943 0 0) 0x7f523008a040 con 0x7f523c1075a0 2024-09-10T18:52:08.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 10 client.24668.objecter ms_dispatch 0x7f523c000b90 osd_map(105..105 src has 49..105) v4 2024-09-10T18:52:08.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 3 client.24668.objecter handle_osd_map got epochs [105,105] > 0 2024-09-10T18:52:08.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 3 client.24668.objecter handle_osd_map decoding full epoch 105 2024-09-10T18:52:08.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 20 client.24668.objecter dump_active .. 0 homeless 2024-09-10T18:52:08.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.611+0000 7f523a7fc640 10 client.24668.objecter ms_handle_connect 0x7f52240711e0 2024-09-10T18:52:08.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.619+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5230056cb0 con 0x7f523c1075a0 2024-09-10T18:52:08.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 -- 172.21.15.83:0/471141480 >> v1:172.21.15.83:6789/0 conn(0x7f47381042a0 legacy=0x7f4738106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 -- 172.21.15.83:0/471141480 shutdown_connections 2024-09-10T18:52:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 -- 172.21.15.83:0/471141480 >> 172.21.15.83:0/471141480 conn(0x7f4738075290 msgr2=0x7f47380f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 -- 172.21.15.83:0/471141480 shutdown_connections 2024-09-10T18:52:08.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 -- 172.21.15.83:0/471141480 wait complete. 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 librados: starting msgr at 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.767+0000 7f473fb77640 1 librados: starting objecter 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 Processor -- start 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- start start 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 librados: setting wanted keys 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 librados: calling monclient init 2024-09-10T18:52:08.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4738122810 con 0x7f47381014e0 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47381239d0 con 0x7f47381042a0 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4738124bd0 con 0x7f4738107e30 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473d913640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f47381014e0 0x7f47380fd1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43296/0 (socket says 172.21.15.83:43296) 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473d913640 1 -- 172.21.15.83:0/2729751194 learned_addr learned my addr 172.21.15.83:0/2729751194 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter ms_handle_connect 0x7f4738107e30 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter ms_handle_connect 0x7f47381014e0 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702205588 0 0) 0x7f4738124bd0 con 0x7f4738107e30 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4724003720 con 0x7f4738107e30 2024-09-10T18:52:08.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter ms_handle_connect 0x7f47381042a0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194796509 0 0) 0x7f4738122810 con 0x7f47381014e0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4738124bd0 con 0x7f47381014e0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842198123 0 0) 0x7f47381239d0 con 0x7f47381042a0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4738122810 con 0x7f47381042a0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016310942 0 0) 0x7f4724003720 con 0x7f4738107e30 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47381239d0 con 0x7f4738107e30 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4730001b80 con 0x7f4738107e30 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536170383 0 0) 0x7f4738124bd0 con 0x7f47381014e0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4724003720 con 0x7f47381014e0 2024-09-10T18:52:08.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4734001840 con 0x7f47381014e0 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611602098 0 0) 0x7f4738122810 con 0x7f47381042a0 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4738124bd0 con 0x7f47381042a0 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4090499154 0 0) 0x7f47381239d0 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 >> v1:172.21.15.165:6789/0 conn(0x7f47381042a0 legacy=0x7f47380fd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 >> v1:172.21.15.83:6789/0 conn(0x7f47381014e0 legacy=0x7f47380fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4738125dd0 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4738124dc0 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4730002200 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4730003450 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 10 client.24674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4738125210 con 0x7f4738107e30 2024-09-10T18:52:08.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f473fb77640 1 librados: init done 2024-09-10T18:52:08.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.771+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f47300199a0 con 0x7f4738107e30 2024-09-10T18:52:08.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47380fcc60 con 0x7f4738107e30 2024-09-10T18:52:08.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(105..105 src has 49..105) v4 ==== 6284+0+0 (unknown 2684729943 0 0) 0x7f4730087990 con 0x7f4738107e30 2024-09-10T18:52:08.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 10 client.24674.objecter ms_dispatch 0x7f4738000b90 osd_map(105..105 src has 49..105) v4 2024-09-10T18:52:08.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 3 client.24674.objecter handle_osd_map got epochs [105,105] > 0 2024-09-10T18:52:08.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 3 client.24674.objecter handle_osd_map decoding full epoch 105 2024-09-10T18:52:08.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 20 client.24674.objecter dump_active .. 0 homeless 2024-09-10T18:52:08.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.775+0000 7f472e7fc640 10 client.24674.objecter ms_handle_connect 0x7f47240718d0 2024-09-10T18:52:08.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.779+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4730054660 con 0x7f4738107e30 2024-09-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.807+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f52280052c0 con 0x7f523c1075a0 2024-09-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.807+0000 7f523a7fc640 1 -- 172.21.15.83:0/3774252255 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 2251523347) 0x7f5230056650 con 0x7f523c1075a0 2024-09-10T18:52:08.813 INFO:teuthology.orchestra.run.smithi083.stdout:266287972397 2024-09-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 10 librados: watch_flush enter 2024-09-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 10 librados: watch_flush exit 2024-09-10T18:52:08.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 20 client.24668.objecter shutdown clearing up homeless session... 2024-09-10T18:52:08.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 10 client.24668.objecter successfully canceled tick 2024-09-10T18:52:08.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 >> v1:172.21.15.83:6800/3071793735 conn(0x7f52240711e0 legacy=0x7f5224073690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.811+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 >> v1:172.21.15.83:6790/0 conn(0x7f523c1075a0 legacy=0x7f523c101080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.815+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 shutdown_connections 2024-09-10T18:52:08.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.815+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 >> 172.21.15.83:0/3774252255 conn(0x7f523c0fa370 msgr2=0x7f523c10b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.815+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 shutdown_connections 2024-09-10T18:52:08.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.815+0000 7f5243c15640 1 -- 172.21.15.83:0/3774252255 wait complete. 2024-09-10T18:52:08.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.815+0000 7f5243c15640 1 librados: shutdown 2024-09-10T18:52:08.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299441 got 223338299447 for osd.6 2024-09-10T18:52:08.831 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:08.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/2160852234 >> v1:172.21.15.83:6790/0 conn(0x7faa74107e30 legacy=0x7faa7410a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/2160852234 shutdown_connections 2024-09-10T18:52:08.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/2160852234 >> 172.21.15.83:0/2160852234 conn(0x7faa74075290 msgr2=0x7faa740f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/2160852234 shutdown_connections 2024-09-10T18:52:08.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/2160852234 wait complete. 2024-09-10T18:52:08.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 librados: starting msgr at 2024-09-10T18:52:08.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 librados: starting objecter 2024-09-10T18:52:08.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 Processor -- start 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- start start 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 librados: setting wanted keys 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 librados: calling monclient init 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa7411e410 con 0x7faa741014e0 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa7411f5d0 con 0x7faa74107e30 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa741207d0 con 0x7faa741042a0 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa79033640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7faa741042a0 0x7faa740657b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48460/0 (socket says 172.21.15.83:48460) 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa79033640 1 -- 172.21.15.83:0/1951961327 learned_addr learned my addr 172.21.15.83:0/1951961327 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter ms_handle_connect 0x7faa741014e0 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter ms_handle_connect 0x7faa741042a0 2024-09-10T18:52:08.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272637224 0 0) 0x7faa7411e410 con 0x7faa741014e0 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003120 con 0x7faa741014e0 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter ms_handle_connect 0x7faa74107e30 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087974960 0 0) 0x7faa741207d0 con 0x7faa741042a0 2024-09-10T18:52:08.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7411e410 con 0x7faa741042a0 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793938178 0 0) 0x7faa7411f5d0 con 0x7faa74107e30 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa741207d0 con 0x7faa74107e30 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473151390 0 0) 0x7faa60003120 con 0x7faa741014e0 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa7411f5d0 con 0x7faa741014e0 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1805705074 0 0) 0x7faa7411e410 con 0x7faa741042a0 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa60003120 con 0x7faa741042a0 2024-09-10T18:52:08.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faa64001a10 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faa700017c0 con 0x7faa741042a0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3167100925 0 0) 0x7faa7411f5d0 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 >> v1:172.21.15.83:6790/0 conn(0x7faa741042a0 legacy=0x7faa740657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 >> v1:172.21.15.165:6789/0 conn(0x7faa74107e30 legacy=0x7faa7411cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa741219d0 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa741209c0 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 10 client.14763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74120e90 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.927+0000 7faa7ba98640 1 librados: init done 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7faa64002390 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faa64003190 con 0x7faa741014e0 2024-09-10T18:52:08.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa54005010 con 0x7faa741014e0 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7faa640195b0 con 0x7faa741014e0 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(105..105 src has 49..105) v4 ==== 6284+0+0 (unknown 2684729943 0 0) 0x7faa64087650 con 0x7faa741014e0 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 10 client.14763.objecter ms_dispatch 0x7faa74000b90 osd_map(105..105 src has 49..105) v4 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 3 client.14763.objecter handle_osd_map got epochs [105,105] > 0 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 3 client.14763.objecter handle_osd_map decoding full epoch 105 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 20 client.14763.objecter dump_active .. 0 homeless 2024-09-10T18:52:08.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.931+0000 7faa6a7fc640 10 client.14763.objecter ms_handle_connect 0x7faa60071800 2024-09-10T18:52:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.939+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7faa64054300 con 0x7faa741014e0 2024-09-10T18:52:08.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.979+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4738107660 con 0x7f4738107e30 2024-09-10T18:52:08.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.979+0000 7f472e7fc640 1 -- 172.21.15.83:0/2729751194 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 2886631821) 0x7f4730054000 con 0x7f4738107e30 2024-09-10T18:52:08.983 INFO:teuthology.orchestra.run.smithi083.stdout:124554051672 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 10 librados: watch_flush enter 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 10 librados: watch_flush exit 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 20 client.24674.objecter shutdown clearing up homeless session... 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 10 client.24674.objecter successfully canceled tick 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 >> v1:172.21.15.83:6800/3071793735 conn(0x7f47240718d0 legacy=0x7f4724073d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 >> v1:172.21.15.83:6790/0 conn(0x7f4738107e30 legacy=0x7f4738121060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 shutdown_connections 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 >> 172.21.15.83:0/2729751194 conn(0x7f4738075290 msgr2=0x7f47380f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:08.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 shutdown_connections 2024-09-10T18:52:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 -- 172.21.15.83:0/2729751194 wait complete. 2024-09-10T18:52:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:08.983+0000 7f473fb77640 1 librados: shutdown 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.194003+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.197690+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.494919+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: audit 2024-09-10T18:52:07.495376+0000 mgr.y (mgr.24526) 102 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.501532+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-09-10T18:52:09.012 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.708922+0000 osd.4 (osd.4) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.709029+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: cluster 2024-09-10T18:52:07.761606+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[18629]: audit 2024-09-10T18:52:08.081110+0000 mon.a (mon.0) 857 : audit [DBG] from='client.? v1:172.21.15.83:0/2732865605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.194003+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.197690+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.494919+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: audit 2024-09-10T18:52:07.495376+0000 mgr.y (mgr.24526) 102 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.501532+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.708922+0000 osd.4 (osd.4) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.709029+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: cluster 2024-09-10T18:52:07.761606+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-10T18:52:09.013 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:08 smithi083 bash[32381]: audit 2024-09-10T18:52:08.081110+0000 mon.a (mon.0) 857 : audit [DBG] from='client.? v1:172.21.15.83:0/2732865605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:52:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.194003+0000 osd.7 (osd.7) 65 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:52:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.197690+0000 osd.7 (osd.7) 66 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:52:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.494919+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: audit 2024-09-10T18:52:07.495376+0000 mgr.y (mgr.24526) 102 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.501532+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.708922+0000 osd.4 (osd.4) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.709029+0000 osd.4 (osd.4) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: cluster 2024-09-10T18:52:07.761606+0000 mon.a (mon.0) 856 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-09-10T18:52:09.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:08 smithi165 bash[22591]: audit 2024-09-10T18:52:08.081110+0000 mon.a (mon.0) 857 : audit [DBG] from='client.? v1:172.21.15.83:0/2732865605' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T18:52:09.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.115+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faa540052c0 con 0x7faa741014e0 2024-09-10T18:52:09.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.115+0000 7faa6a7fc640 1 -- 172.21.15.83:0/1951961327 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 947323998) 0x7faa64053ca0 con 0x7faa741014e0 2024-09-10T18:52:09.122 INFO:teuthology.orchestra.run.smithi083.stdout:184683593793 2024-09-10T18:52:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 10 librados: watch_flush enter 2024-09-10T18:52:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 10 librados: watch_flush exit 2024-09-10T18:52:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 20 client.14763.objecter shutdown clearing up homeless session... 2024-09-10T18:52:09.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 10 client.14763.objecter successfully canceled tick 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 >> v1:172.21.15.83:6800/3071793735 conn(0x7faa60071800 legacy=0x7faa60073cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 >> v1:172.21.15.83:6789/0 conn(0x7faa741014e0 legacy=0x7faa74065080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 shutdown_connections 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 >> 172.21.15.83:0/1951961327 conn(0x7faa74075290 msgr2=0x7faa7410b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 shutdown_connections 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 -- 172.21.15.83:0/1951961327 wait complete. 2024-09-10T18:52:09.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:09.119+0000 7faa5ffff640 1 librados: shutdown 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: cluster 2024-09-10T18:52:08.241073+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: cluster 2024-09-10T18:52:08.245600+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: cluster 2024-09-10T18:52:08.498954+0000 mgr.y (mgr.24526) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: audit 2024-09-10T18:52:08.813229+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.83:0/3774252255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: audit 2024-09-10T18:52:08.982814+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.83:0/2729751194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: audit 2024-09-10T18:52:09.120070+0000 mon.a (mon.0) 858 : audit [DBG] from='client.? v1:172.21.15.83:0/1951961327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.319970+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.323689+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:52:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: cluster 2024-09-10T18:52:08.241073+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: cluster 2024-09-10T18:52:08.245600+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: cluster 2024-09-10T18:52:08.498954+0000 mgr.y (mgr.24526) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: audit 2024-09-10T18:52:08.813229+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.83:0/3774252255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: audit 2024-09-10T18:52:08.982814+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.83:0/2729751194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: audit 2024-09-10T18:52:09.120070+0000 mon.a (mon.0) 858 : audit [DBG] from='client.? v1:172.21.15.83:0/1951961327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.319970+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:52:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:09 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.323689+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:52:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: cluster 2024-09-10T18:52:08.241073+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:52:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: cluster 2024-09-10T18:52:08.245600+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:52:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: cluster 2024-09-10T18:52:08.498954+0000 mgr.y (mgr.24526) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: audit 2024-09-10T18:52:08.813229+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? v1:172.21.15.83:0/3774252255' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T18:52:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: audit 2024-09-10T18:52:08.982814+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? v1:172.21.15.83:0/2729751194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T18:52:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: audit 2024-09-10T18:52:09.120070+0000 mon.a (mon.0) 858 : audit [DBG] from='client.? v1:172.21.15.83:0/1951961327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T18:52:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.319970+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:52:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:09 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.323689+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:52:10.384 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574952 got 55834574956 for osd.1 2024-09-10T18:52:10.385 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:10.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972392 got 266287972397 for osd.7 2024-09-10T18:52:10.385 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:11.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.504238+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.509366+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.655679+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.660541+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.834515+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[18629]: cluster 2024-09-10T18:52:09.838004+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.504238+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.509366+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.655679+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.660541+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:52:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.834515+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:52:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:10 smithi083 bash[32381]: cluster 2024-09-10T18:52:09.838004+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.504238+0000 osd.0 (osd.0) 65 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.509366+0000 osd.0 (osd.0) 66 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.655679+0000 osd.3 (osd.3) 73 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.660541+0000 osd.3 (osd.3) 74 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.834515+0000 osd.5 (osd.5) 55 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:52:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:10 smithi165 bash[22591]: cluster 2024-09-10T18:52:09.838004+0000 osd.5 (osd.5) 56 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:52:11.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051668 got 124554051672 for osd.3 2024-09-10T18:52:11.170 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:11.184 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593789 got 184683593793 for osd.5 2024-09-10T18:52:11.184 DEBUG:teuthology.parallel:result is None 2024-09-10T18:52:11.184 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T18:52:11.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:52:11.781 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[18629]: cluster 2024-09-10T18:52:10.500339+0000 mgr.y (mgr.24526) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.303467+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.309138+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.448382+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.451782+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[32381]: cluster 2024-09-10T18:52:10.500339+0000 mgr.y (mgr.24526) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.303467+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:52:12.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.309138+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:52:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.448382+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:52:12.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:11 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.451782+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:52:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[22591]: cluster 2024-09-10T18:52:10.500339+0000 mgr.y (mgr.24526) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.303467+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:52:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.309138+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:52:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.448382+0000 osd.2 (osd.2) 39 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:52:12.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:11 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.451782+0000 osd.2 (osd.2) 40 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:52:13.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:12] "GET /metrics HTTP/1.1" 200 220116 "" "Prometheus/2.43.0" 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.217315+0000 osd.4 (osd.4) 77 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.222581+0000 osd.4 (osd.4) 78 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.492332+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.496633+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.852832+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[18629]: cluster 2024-09-10T18:52:11.856322+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.217315+0000 osd.4 (osd.4) 77 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.222581+0000 osd.4 (osd.4) 78 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:52:13.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.492332+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:52:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.496633+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:52:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.852832+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:52:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:12 smithi083 bash[32381]: cluster 2024-09-10T18:52:11.856322+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.217315+0000 osd.4 (osd.4) 77 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.222581+0000 osd.4 (osd.4) 78 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.492332+0000 osd.6 (osd.6) 47 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.496633+0000 osd.6 (osd.6) 48 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.852832+0000 osd.5 (osd.5) 57 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:52:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:12 smithi165 bash[22591]: cluster 2024-09-10T18:52:11.856322+0000 osd.5 (osd.5) 58 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:52:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:13 smithi165 bash[22591]: cluster 2024-09-10T18:52:12.501015+0000 mgr.y (mgr.24526) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:14.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:13 smithi083 bash[18629]: cluster 2024-09-10T18:52:12.501015+0000 mgr.y (mgr.24526) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:14.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:13 smithi083 bash[32381]: cluster 2024-09-10T18:52:12.501015+0000 mgr.y (mgr.24526) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:14 smithi165 bash[22591]: audit 2024-09-10T18:52:14.549948+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:15.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:14 smithi083 bash[18629]: audit 2024-09-10T18:52:14.549948+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:15.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:14 smithi083 bash[32381]: audit 2024-09-10T18:52:14.549948+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:15.952 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[18629]: cluster 2024-09-10T18:52:14.501666+0000 mgr.y (mgr.24526) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[18629]: cluster 2024-09-10T18:52:14.534468+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[18629]: cluster 2024-09-10T18:52:14.538825+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[32381]: cluster 2024-09-10T18:52:14.501666+0000 mgr.y (mgr.24526) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[32381]: cluster 2024-09-10T18:52:14.534468+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:52:16.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:15 smithi083 bash[32381]: cluster 2024-09-10T18:52:14.538825+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:52:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:15 smithi165 bash[22591]: cluster 2024-09-10T18:52:14.501666+0000 mgr.y (mgr.24526) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:15 smithi165 bash[22591]: cluster 2024-09-10T18:52:14.534468+0000 osd.0 (osd.0) 67 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:52:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:15 smithi165 bash[22591]: cluster 2024-09-10T18:52:14.538825+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.204287+0000 osd.4 (osd.4) 79 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.209593+0000 osd.4 (osd.4) 80 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.546842+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 deep-scrub starts 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.552780+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 deep-scrub ok 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.855416+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[32381]: cluster 2024-09-10T18:52:15.859207+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.204287+0000 osd.4 (osd.4) 79 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.209593+0000 osd.4 (osd.4) 80 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:52:17.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.546842+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 deep-scrub starts 2024-09-10T18:52:17.093 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.552780+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 deep-scrub ok 2024-09-10T18:52:17.093 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.855416+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:52:17.093 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:16 smithi083 bash[18629]: cluster 2024-09-10T18:52:15.859207+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.204287+0000 osd.4 (osd.4) 79 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.209593+0000 osd.4 (osd.4) 80 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.546842+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 deep-scrub starts 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.552780+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 deep-scrub ok 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.855416+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:52:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:16 smithi165 bash[22591]: cluster 2024-09-10T18:52:15.859207+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:52:17.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:17 smithi165 bash[22591]: cluster 2024-09-10T18:52:16.233959+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:52:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:17 smithi165 bash[22591]: cluster 2024-09-10T18:52:16.237404+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:52:18.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:17 smithi165 bash[22591]: cluster 2024-09-10T18:52:16.503134+0000 mgr.y (mgr.24526) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:18.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[18629]: cluster 2024-09-10T18:52:16.233959+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:52:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[18629]: cluster 2024-09-10T18:52:16.237404+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:52:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[18629]: cluster 2024-09-10T18:52:16.503134+0000 mgr.y (mgr.24526) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[32381]: cluster 2024-09-10T18:52:16.233959+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:52:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[32381]: cluster 2024-09-10T18:52:16.237404+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:52:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:17 smithi083 bash[32381]: cluster 2024-09-10T18:52:16.503134+0000 mgr.y (mgr.24526) 107 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.247786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:52:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.252482+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.459622+0000 osd.0 (osd.0) 69 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.463930+0000 osd.0 (osd.0) 70 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: audit 2024-09-10T18:52:17.506587+0000 mgr.y (mgr.24526) 108 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.703173+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.b scrub starts 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:17.707803+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.b scrub ok 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:18.311792+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:52:19.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:18 smithi165 bash[22591]: cluster 2024-09-10T18:52:18.315452+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:52:19.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/1901319529 >> v1:172.21.15.83:6789/0 conn(0x7f9c2c0fd3b0 legacy=0x7f9c2c0faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:19.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/1901319529 shutdown_connections 2024-09-10T18:52:19.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/1901319529 >> 172.21.15.83:0/1901319529 conn(0x7f9c2c074a80 msgr2=0x7f9c2c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:19.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/1901319529 shutdown_connections 2024-09-10T18:52:19.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/1901319529 wait complete. 2024-09-10T18:52:19.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 librados: starting msgr at 2024-09-10T18:52:19.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 librados: starting objecter 2024-09-10T18:52:19.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 Processor -- start 2024-09-10T18:52:19.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- start start 2024-09-10T18:52:19.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 librados: setting wanted keys 2024-09-10T18:52:19.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 librados: calling monclient init 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9c2c11e250 con 0x7f9c2c10b230 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9c2c11f450 con 0x7f9c2c0fd3b0 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9c2c120650 con 0x7f9c2c0fd870 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c31b9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9c2c0fd870 0x7f9c2c107da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44024/0 (socket says 172.21.15.83:44024) 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c31b9d640 1 -- 172.21.15.83:0/4234137660 learned_addr learned my addr 172.21.15.83:0/4234137660 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter ms_handle_connect 0x7f9c2c0fd870 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter ms_handle_connect 0x7f9c2c10b230 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334460213 0 0) 0x7f9c2c120650 con 0x7f9c2c0fd870 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c18003720 con 0x7f9c2c0fd870 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter ms_handle_connect 0x7f9c2c0fd3b0 2024-09-10T18:52:19.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231140081 0 0) 0x7f9c2c11e250 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2c120650 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184437909 0 0) 0x7f9c2c11f450 con 0x7f9c2c0fd3b0 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c2c11e250 con 0x7f9c2c0fd3b0 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2084196837 0 0) 0x7f9c2c120650 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c2c11f450 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113106791 0 0) 0x7f9c18003720 con 0x7f9c2c0fd870 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c2c120650 con 0x7f9c2c0fd870 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9c24001aa0 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9c1c0019b0 con 0x7f9c2c0fd870 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3028619624 0 0) 0x7f9c2c11f450 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 >> v1:172.21.15.83:6790/0 conn(0x7f9c2c0fd870 legacy=0x7f9c2c107da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 >> v1:172.21.15.165:6789/0 conn(0x7f9c2c0fd3b0 legacy=0x7f9c2c107690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c121850 con 0x7f9c2c10b230 2024-09-10T18:52:19.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c11e440 con 0x7f9c2c10b230 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 10 client.14769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c11e890 con 0x7f9c2c10b230 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.167+0000 7f9c34602640 1 librados: init done 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9c24002460 con 0x7f9c2c10b230 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9c24003410 con 0x7f9c2c10b230 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c10005010 con 0x7f9c2c10b230 2024-09-10T18:52:19.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9c24002000 con 0x7f9c2c10b230 2024-09-10T18:52:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(105..105 src has 49..105) v4 ==== 6284+0+0 (unknown 2684729943 0 0) 0x7f9c240877d0 con 0x7f9c2c10b230 2024-09-10T18:52:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 10 client.14769.objecter ms_dispatch 0x7f9c2c000b90 osd_map(105..105 src has 49..105) v4 2024-09-10T18:52:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 3 client.14769.objecter handle_osd_map got epochs [105,105] > 0 2024-09-10T18:52:19.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 3 client.14769.objecter handle_osd_map decoding full epoch 105 2024-09-10T18:52:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 20 client.14769.objecter dump_active .. 0 homeless 2024-09-10T18:52:19.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.171+0000 7f9c22ffd640 10 client.14769.objecter ms_handle_connect 0x7f9c18071690 2024-09-10T18:52:19.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.179+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9c240544a0 con 0x7f9c2c10b230 2024-09-10T18:52:19.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.247786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.252482+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.459622+0000 osd.0 (osd.0) 69 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.463930+0000 osd.0 (osd.0) 70 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: audit 2024-09-10T18:52:17.506587+0000 mgr.y (mgr.24526) 108 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.703173+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.b scrub starts 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:17.707803+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.b scrub ok 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:18.311792+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[18629]: cluster 2024-09-10T18:52:18.315452+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.247786+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.252482+0000 osd.4 (osd.4) 82 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T18:52:19.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.459622+0000 osd.0 (osd.0) 69 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.463930+0000 osd.0 (osd.0) 70 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: audit 2024-09-10T18:52:17.506587+0000 mgr.y (mgr.24526) 108 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.703173+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.b scrub starts 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:17.707803+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.b scrub ok 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:18.311792+0000 osd.1 (osd.1) 69 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:52:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:18 smithi083 bash[32381]: cluster 2024-09-10T18:52:18.315452+0000 osd.1 (osd.1) 70 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:52:19.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.351+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c10000b90 con 0x7f9c18071690 2024-09-10T18:52:19.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.363+0000 7f9c22ffd640 1 -- 172.21.15.83:0/4234137660 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384356 (unknown 2965378022 0 2515083177) 0x7f9c10000b90 con 0x7f9c18071690 2024-09-10T18:52:19.374 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:52:19.377 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 10 librados: watch_flush enter 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 10 librados: watch_flush exit 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 20 client.14769.objecter shutdown clearing up homeless session... 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 10 client.14769.objecter successfully canceled tick 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9c18071690 legacy=0x7f9c18073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 >> v1:172.21.15.83:6789/0 conn(0x7f9c2c10b230 legacy=0x7f9c2c11c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 shutdown_connections 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 >> 172.21.15.83:0/4234137660 conn(0x7f9c2c074a80 msgr2=0x7f9c2c10ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 shutdown_connections 2024-09-10T18:52:19.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 -- 172.21.15.83:0/4234137660 wait complete. 2024-09-10T18:52:19.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:19.371+0000 7f9c34602640 1 librados: shutdown 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[18629]: cluster 2024-09-10T18:52:18.503810+0000 mgr.y (mgr.24526) 109 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[18629]: cluster 2024-09-10T18:52:18.561909+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[18629]: cluster 2024-09-10T18:52:18.566040+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[18629]: cluster 2024-09-10T18:52:19.318497+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[18629]: cluster 2024-09-10T18:52:19.321903+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[32381]: cluster 2024-09-10T18:52:18.503810+0000 mgr.y (mgr.24526) 109 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-09-10T18:52:20.079 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[32381]: cluster 2024-09-10T18:52:18.561909+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:52:20.080 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[32381]: cluster 2024-09-10T18:52:18.566040+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:52:20.080 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[32381]: cluster 2024-09-10T18:52:19.318497+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:52:20.080 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:19 smithi083 bash[32381]: cluster 2024-09-10T18:52:19.321903+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:52:20.081 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-10T18:52:18.503437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1027,"num_read_kb":883,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385948,"kb_used_data":7128,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575268,"statfs":{"total":767960285184,"available":765517074432,"internally_reserved":0,"allocated":7299072,"data_stored":4354771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.745333"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018569+0000","last_change":"2024-09-10T18:50:56.774372+0000","last_active":"2024-09-10T18:51:40.018569+0000","last_peered":"2024-09-10T18:51:40.018569+0000","last_clean":"2024-09-10T18:51:40.018569+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:51:40.018569+0000","last_undegraded":"2024-09-10T18:51:40.018569+0000","last_fullsized":"2024-09-10T18:51:40.018569+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:50:56.774029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:10.130141+0000","scrub_duration":0.004548547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.653552+0000","last_change":"2024-09-10T18:50:45.793175+0000","last_active":"2024-09-10T18:51:40.653552+0000","last_peered":"2024-09-10T18:51:40.653552+0000","last_clean":"2024-09-10T18:51:40.653552+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:51:40.653552+0000","last_undegraded":"2024-09-10T18:51:40.653552+0000","last_fullsized":"2024-09-10T18:51:40.653552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:45.793029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:45.793029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:14.463730+0000","scrub_duration":0.0035619169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":67,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.019426+0000","last_change":"2024-09-10T18:51:37.694332+0000","last_active":"2024-09-10T18:51:40.019426+0000","last_peered":"2024-09-10T18:51:40.019426+0000","last_clean":"2024-09-10T18:51:40.019426+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:51:40.019426+0000","last_undegraded":"2024-09-10T18:51:40.019426+0000","last_fullsized":"2024-09-10T18:51:40.019426+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:37.694143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.694143+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.236407+0000","scrub_duration":0.0048100759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017855+0000","last_change":"2024-09-10T18:50:48.738844+0000","last_active":"2024-09-10T18:51:40.017855+0000","last_peered":"2024-09-10T18:51:40.017855+0000","last_clean":"2024-09-10T18:51:40.017855+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:51:40.017855+0000","last_undegraded":"2024-09-10T18:51:40.017855+0000","last_fullsized":"2024-09-10T18:51:40.017855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:48.738683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:48.738683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036868299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:41.627328+0000","last_change":"2024-09-10T18:51:41.627258+0000","last_active":"2024-09-10T18:51:41.627328+0000","last_peered":"2024-09-10T18:51:41.627328+0000","last_clean":"2024-09-10T18:51:41.627328+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:51:41.627328+0000","last_undegraded":"2024-09-10T18:51:41.627328+0000","last_fullsized":"2024-09-10T18:51:41.627328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.627147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.627147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:48.610764+0000","scrub_duration":0.0032008610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":71,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.653118+0000","last_change":"2024-09-10T18:51:11.486336+0000","last_active":"2024-09-10T18:51:40.653118+0000","last_peered":"2024-09-10T18:51:40.653118+0000","last_clean":"2024-09-10T18:51:40.653118+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:51:40.653118+0000","last_undegraded":"2024-09-10T18:51:40.653118+0000","last_fullsized":"2024-09-10T18:51:40.653118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:11.486150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:11.486150+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:15.162538+0000","scrub_duration":0.0046833980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.691215+0000","last_change":"2024-09-10T18:50:53.516934+0000","last_active":"2024-09-10T18:51:40.691215+0000","last_peered":"2024-09-10T18:51:40.691215+0000","last_clean":"2024-09-10T18:51:40.691215+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:51:40.691215+0000","last_undegraded":"2024-09-10T18:51:40.691215+0000","last_fullsized":"2024-09-10T18:51:40.691215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:53.516754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:53.516754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:13.655847+0000","scrub_duration":0.0028258020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.690678+0000","last_change":"2024-09-10T18:50:52.520182+0000","last_active":"2024-09-10T18:51:40.690678+0000","last_peered":"2024-09-10T18:51:40.690678+0000","last_clean":"2024-09-10T18:51:40.690678+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:51:40.690678+0000","last_undegraded":"2024-09-10T18:51:40.690678+0000","last_fullsized":"2024-09-10T18:51:40.690678+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:52.520037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:52.520037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:19.642683+0000","scrub_duration":0.0031335130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.322263+0000","last_change":"2024-09-10T18:50:35.301141+0000","last_active":"2024-09-10T18:51:40.322263+0000","last_peered":"2024-09-10T18:51:40.322263+0000","last_clean":"2024-09-10T18:51:40.322263+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:51:40.322263+0000","last_undegraded":"2024-09-10T18:51:40.322263+0000","last_fullsized":"2024-09-10T18:51:40.322263+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:35.300963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:35.300963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003249409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"77'15","reported_seq":75,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620575+0000","last_change":"2024-09-10T18:50:57.455605+0000","last_active":"2024-09-10T18:51:40.620575+0000","last_peered":"2024-09-10T18:51:40.620575+0000","last_clean":"2024-09-10T18:51:40.620575+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:51:40.620575+0000","last_undegraded":"2024-09-10T18:51:40.620575+0000","last_fullsized":"2024-09-10T18:51:40.620575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:50:57.455409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:57.455409+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.020653+0000","scrub_duration":0.0053829450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.691643+0000","last_change":"2024-09-10T18:51:03.556626+0000","last_active":"2024-09-10T18:51:40.691643+0000","last_peered":"2024-09-10T18:51:40.691643+0000","last_clean":"2024-09-10T18:51:40.691643+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:51:40.691643+0000","last_undegraded":"2024-09-10T18:51:40.691643+0000","last_fullsized":"2024-09-10T18:51:40.691643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:03.556572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.556572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:23.172666+0000","scrub_duration":0.0021821369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.619048+0000","last_change":"2024-09-10T18:51:06.584417+0000","last_active":"2024-09-10T18:51:40.619048+0000","last_peered":"2024-09-10T18:51:40.619048+0000","last_clean":"2024-09-10T18:51:40.619048+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:51:40.619048+0000","last_undegraded":"2024-09-10T18:51:40.619048+0000","last_fullsized":"2024-09-10T18:51:40.619048+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:06.584260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.584260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.730218+0000","scrub_duration":0.0035935630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.321788+0000","last_change":"2024-09-10T18:50:55.397671+0000","last_active":"2024-09-10T18:51:40.321788+0000","last_peered":"2024-09-10T18:51:40.321788+0000","last_clean":"2024-09-10T18:51:40.321788+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:51:40.321788+0000","last_undegraded":"2024-09-10T18:51:40.321788+0000","last_fullsized":"2024-09-10T18:51:40.321788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:55.397540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:55.397540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:13.469774+0000","scrub_duration":0.0029184089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":88,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:43.650149+0000","last_change":"2024-09-10T18:51:43.650049+0000","last_active":"2024-09-10T18:51:43.650149+0000","last_peered":"2024-09-10T18:51:43.650149+0000","last_clean":"2024-09-10T18:51:43.650149+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:51:43.650149+0000","last_undegraded":"2024-09-10T18:51:43.650149+0000","last_fullsized":"2024-09-10T18:51:43.650149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:51:43.649947+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.649947+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.873051+0000","scrub_duration":0.003856703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:01.416181+0000","last_change":"2024-09-10T18:52:01.415959+0000","last_active":"2024-09-10T18:52:01.416181+0000","last_peered":"2024-09-10T18:52:01.416181+0000","last_clean":"2024-09-10T18:52:01.416181+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:52:01.416181+0000","last_undegraded":"2024-09-10T18:52:01.416181+0000","last_fullsized":"2024-09-10T18:52:01.416181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.415718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.415718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:25.902687+0000","scrub_duration":0.0036293950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.651768+0000","last_change":"2024-09-10T18:51:36.537160+0000","last_active":"2024-09-10T18:51:40.651768+0000","last_peered":"2024-09-10T18:51:40.651768+0000","last_clean":"2024-09-10T18:51:40.651768+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:51:40.651768+0000","last_undegraded":"2024-09-10T18:51:40.651768+0000","last_fullsized":"2024-09-10T18:51:40.651768+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:36.536996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:36.536996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.415469+0000","scrub_duration":0.0033809280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":91,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:02.356278+0000","last_change":"2024-09-10T18:52:02.356154+0000","last_active":"2024-09-10T18:52:02.356278+0000","last_peered":"2024-09-10T18:52:02.356278+0000","last_clean":"2024-09-10T18:52:02.356278+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:52:02.356278+0000","last_undegraded":"2024-09-10T18:52:02.356278+0000","last_fullsized":"2024-09-10T18:52:02.356278+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:02.355980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:02.355980+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:21.108556+0000","scrub_duration":0.0055219309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.321708+0000","last_change":"2024-09-10T18:50:52.474030+0000","last_active":"2024-09-10T18:51:40.321708+0000","last_peered":"2024-09-10T18:51:40.321708+0000","last_clean":"2024-09-10T18:51:40.321708+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:51:40.321708+0000","last_undegraded":"2024-09-10T18:51:40.321708+0000","last_fullsized":"2024-09-10T18:51:40.321708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:52.473895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:52.473895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:22.239397+0000","scrub_duration":0.003517807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.758520+0000","last_change":"2024-09-10T18:50:57.482247+0000","last_active":"2024-09-10T18:52:07.758520+0000","last_peered":"2024-09-10T18:52:07.758520+0000","last_clean":"2024-09-10T18:52:07.758520+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:52:07.758520+0000","last_undegraded":"2024-09-10T18:52:07.758520+0000","last_fullsized":"2024-09-10T18:52:07.758520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:50:57.482095+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:50:57.482095+0000","last_clean_scrub_stamp":"2024-09-10T18:50:57.482095+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060783180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.498224+0000","last_change":"2024-09-10T18:50:51.634970+0000","last_active":"2024-09-10T18:51:40.498224+0000","last_peered":"2024-09-10T18:51:40.498224+0000","last_clean":"2024-09-10T18:51:40.498224+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:51:40.498224+0000","last_undegraded":"2024-09-10T18:51:40.498224+0000","last_fullsized":"2024-09-10T18:51:40.498224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:51.634808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:51.634808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:21.182322+0000","scrub_duration":0.0031486029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":72,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018939+0000","last_change":"2024-09-10T18:50:39.845505+0000","last_active":"2024-09-10T18:51:40.018939+0000","last_peered":"2024-09-10T18:51:40.018939+0000","last_clean":"2024-09-10T18:51:40.018939+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:51:40.018939+0000","last_undegraded":"2024-09-10T18:51:40.018939+0000","last_fullsized":"2024-09-10T18:51:40.018939+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:50:39.845392+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090190909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:41.493042+0000","last_change":"2024-09-10T18:51:41.492867+0000","last_active":"2024-09-10T18:51:41.493042+0000","last_peered":"2024-09-10T18:51:41.493042+0000","last_clean":"2024-09-10T18:51:41.493042+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:51:41.493042+0000","last_undegraded":"2024-09-10T18:51:41.493042+0000","last_fullsized":"2024-09-10T18:51:41.493042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.492707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:50.150125+0000","scrub_duration":0.0035920370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:46.600767+0000","last_change":"2024-09-10T18:51:46.600626+0000","last_active":"2024-09-10T18:51:46.600767+0000","last_peered":"2024-09-10T18:51:46.600767+0000","last_clean":"2024-09-10T18:51:46.600767+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:51:46.600767+0000","last_undegraded":"2024-09-10T18:51:46.600767+0000","last_fullsized":"2024-09-10T18:51:46.600767+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.600460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.600460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.436438+0000","scrub_duration":0.0032588019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:53.429492+0000","last_change":"2024-09-10T18:51:53.429372+0000","last_active":"2024-09-10T18:51:53.429492+0000","last_peered":"2024-09-10T18:51:53.429492+0000","last_clean":"2024-09-10T18:51:53.429492+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:51:53.429492+0000","last_undegraded":"2024-09-10T18:51:53.429492+0000","last_fullsized":"2024-09-10T18:51:53.429492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:53.429216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.429216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.844395+0000","scrub_duration":0.003713091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":87,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:55.522731+0000","last_change":"2024-09-10T18:51:55.522605+0000","last_active":"2024-09-10T18:51:55.522731+0000","last_peered":"2024-09-10T18:51:55.522731+0000","last_clean":"2024-09-10T18:51:55.522731+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:51:55.522731+0000","last_undegraded":"2024-09-10T18:51:55.522731+0000","last_fullsized":"2024-09-10T18:51:55.522731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:55.522513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.522513+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.705011+0000","scrub_duration":0.0065065660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.016286+0000","last_change":"2024-09-10T18:50:54.738927+0000","last_active":"2024-09-10T18:51:40.016286+0000","last_peered":"2024-09-10T18:51:40.016286+0000","last_clean":"2024-09-10T18:51:40.016286+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:51:40.016286+0000","last_undegraded":"2024-09-10T18:51:40.016286+0000","last_fullsized":"2024-09-10T18:51:40.016286+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:54.738771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:54.738771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:11.659596+0000","scrub_duration":0.004489435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:55.508861+0000","last_change":"2024-09-10T18:51:55.508740+0000","last_active":"2024-09-10T18:51:55.508861+0000","last_peered":"2024-09-10T18:51:55.508861+0000","last_clean":"2024-09-10T18:51:55.508861+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:51:55.508861+0000","last_undegraded":"2024-09-10T18:51:55.508861+0000","last_fullsized":"2024-09-10T18:51:55.508861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:55.508582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.508582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.812679+0000","scrub_duration":0.0034042920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:50.675176+0000","last_change":"2024-09-10T18:51:50.674985+0000","last_active":"2024-09-10T18:51:50.675176+0000","last_peered":"2024-09-10T18:51:50.675176+0000","last_clean":"2024-09-10T18:51:50.675176+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:51:50.675176+0000","last_undegraded":"2024-09-10T18:51:50.675176+0000","last_fullsized":"2024-09-10T18:51:50.675176+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.674821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:16.792421+0000","scrub_duration":0.003696628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":90,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.692765+0000","last_change":"2024-09-10T18:51:29.827804+0000","last_active":"2024-09-10T18:51:40.692765+0000","last_peered":"2024-09-10T18:51:40.692765+0000","last_clean":"2024-09-10T18:51:40.692765+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:51:40.692765+0000","last_undegraded":"2024-09-10T18:51:40.692765+0000","last_fullsized":"2024-09-10T18:51:40.692765+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:51:29.827615+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.827615+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.030826+0000","scrub_duration":0.0051889199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.320884+0000","last_change":"2024-09-10T18:50:47.395085+0000","last_active":"2024-09-10T18:51:40.320884+0000","last_peered":"2024-09-10T18:51:40.320884+0000","last_clean":"2024-09-10T18:51:40.320884+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:51:40.320884+0000","last_undegraded":"2024-09-10T18:51:40.320884+0000","last_fullsized":"2024-09-10T18:51:40.320884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:47.394957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:47.394957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:15.354944+0000","scrub_duration":0.0035266519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.758541+0000","last_change":"2024-09-10T18:51:40.694756+0000","last_active":"2024-09-10T18:52:07.758541+0000","last_peered":"2024-09-10T18:52:07.758541+0000","last_clean":"2024-09-10T18:52:07.758541+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:52:07.758541+0000","last_undegraded":"2024-09-10T18:52:07.758541+0000","last_fullsized":"2024-09-10T18:52:07.758541+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:51:40.694630+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.694630+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.465301+0000","scrub_duration":0.0057213519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":49,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018834+0000","last_change":"2024-09-10T18:51:22.704230+0000","last_active":"2024-09-10T18:51:40.018834+0000","last_peered":"2024-09-10T18:51:40.018834+0000","last_clean":"2024-09-10T18:51:40.018834+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:51:40.018834+0000","last_undegraded":"2024-09-10T18:51:40.018834+0000","last_fullsized":"2024-09-10T18:51:40.018834+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:51:22.704057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:22.704057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.304502+0000","scrub_duration":0.0039967839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":70,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.653340+0000","last_change":"2024-09-10T18:50:59.610521+0000","last_active":"2024-09-10T18:51:40.653340+0000","last_peered":"2024-09-10T18:51:40.653340+0000","last_clean":"2024-09-10T18:51:40.653340+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:51:40.653340+0000","last_undegraded":"2024-09-10T18:51:40.653340+0000","last_fullsized":"2024-09-10T18:51:40.653340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:50:59.610418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:59.610418+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:23.830590+0000","scrub_duration":0.0038889670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:47.158668+0000","last_change":"2024-09-10T18:51:47.158565+0000","last_active":"2024-09-10T18:51:47.158668+0000","last_peered":"2024-09-10T18:51:47.158668+0000","last_clean":"2024-09-10T18:51:47.158668+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:51:47.158668+0000","last_undegraded":"2024-09-10T18:51:47.158668+0000","last_fullsized":"2024-09-10T18:51:47.158668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:47.158405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.158405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.185469+0000","scrub_duration":0.0034278289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.692963+0000","last_change":"2024-09-10T18:51:28.793948+0000","last_active":"2024-09-10T18:51:40.692963+0000","last_peered":"2024-09-10T18:51:40.692963+0000","last_clean":"2024-09-10T18:51:40.692963+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:51:40.692963+0000","last_undegraded":"2024-09-10T18:51:40.692963+0000","last_fullsized":"2024-09-10T18:51:40.692963+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:28.793787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.793787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.515669+0000","scrub_duration":0.0035723209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":221,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:13.674471+0000","last_change":"2024-09-10T18:52:00.591712+0000","last_active":"2024-09-10T18:52:13.674471+0000","last_peered":"2024-09-10T18:52:13.674471+0000","last_clean":"2024-09-10T18:52:13.674471+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:52:13.674471+0000","last_undegraded":"2024-09-10T18:52:13.674471+0000","last_fullsized":"2024-09-10T18:52:13.674471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:52:00.591632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.591632+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:05.247607+0000","scrub_duration":0.0032914160000000001,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:56.155148+0000","last_change":"2024-09-10T18:51:56.155039+0000","last_active":"2024-09-10T18:51:56.155148+0000","last_peered":"2024-09-10T18:51:56.155148+0000","last_clean":"2024-09-10T18:51:56.155148+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:51:56.155148+0000","last_undegraded":"2024-09-10T18:51:56.155148+0000","last_fullsized":"2024-09-10T18:51:56.155148+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.154883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.154883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.178313+0000","scrub_duration":0.0033842830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:09.660821+0000","last_change":"2024-09-10T18:52:09.660716+0000","last_active":"2024-09-10T18:52:09.660821+0000","last_peered":"2024-09-10T18:52:09.660821+0000","last_clean":"2024-09-10T18:52:09.660821+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:52:09.660821+0000","last_undegraded":"2024-09-10T18:52:09.660821+0000","last_fullsized":"2024-09-10T18:52:09.660821+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:52:09.660551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.660551+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.065594+0000","scrub_duration":0.0047491369999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:05.361453+0000","last_change":"2024-09-10T18:52:05.361338+0000","last_active":"2024-09-10T18:52:05.361453+0000","last_peered":"2024-09-10T18:52:05.361453+0000","last_clean":"2024-09-10T18:52:05.361453+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:52:05.361453+0000","last_undegraded":"2024-09-10T18:52:05.361453+0000","last_fullsized":"2024-09-10T18:52:05.361453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:05.361175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.361175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.175562+0000","scrub_duration":0.0039873549999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.759736+0000","last_change":"2024-09-10T18:51:05.769818+0000","last_active":"2024-09-10T18:52:07.759736+0000","last_peered":"2024-09-10T18:52:07.759736+0000","last_clean":"2024-09-10T18:52:07.759736+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:52:07.759736+0000","last_undegraded":"2024-09-10T18:52:07.759736+0000","last_fullsized":"2024-09-10T18:52:07.759736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:05.769650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:05.769650+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.040600+0000","scrub_duration":0.0040089660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"78'1","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.691659+0000","last_change":"2024-09-10T18:51:32.830829+0000","last_active":"2024-09-10T18:51:40.691659+0000","last_peered":"2024-09-10T18:51:40.691659+0000","last_clean":"2024-09-10T18:51:40.691659+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:51:40.691659+0000","last_undegraded":"2024-09-10T18:51:40.691659+0000","last_fullsized":"2024-09-10T18:51:40.691659+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-09-10T18:51:32.830654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:32.830654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.674413+0000","scrub_duration":0.0041675089999999998,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.690673+0000","last_change":"2024-09-10T18:51:38.762020+0000","last_active":"2024-09-10T18:51:40.690673+0000","last_peered":"2024-09-10T18:51:40.690673+0000","last_clean":"2024-09-10T18:51:40.690673+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:51:40.690673+0000","last_undegraded":"2024-09-10T18:51:40.690673+0000","last_fullsized":"2024-09-10T18:51:40.690673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.761859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.761859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:49.670551+0000","scrub_duration":0.003532154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"77'30","reported_seq":145,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:11.309395+0000","last_change":"2024-09-10T18:52:11.309315+0000","last_active":"2024-09-10T18:52:11.309395+0000","last_peered":"2024-09-10T18:52:11.309395+0000","last_clean":"2024-09-10T18:52:11.309395+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:52:11.309395+0000","last_undegraded":"2024-09-10T18:52:11.309395+0000","last_fullsized":"2024-09-10T18:52:11.309395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-10T18:52:11.309152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.309152+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.549403+0000","scrub_duration":0.0055436330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:03.183955+0000","last_change":"2024-09-10T18:52:03.183815+0000","last_active":"2024-09-10T18:52:03.183955+0000","last_peered":"2024-09-10T18:52:03.183955+0000","last_clean":"2024-09-10T18:52:03.183955+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:52:03.183955+0000","last_undegraded":"2024-09-10T18:52:03.183955+0000","last_fullsized":"2024-09-10T18:52:03.183955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.183646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.183646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.233272+0000","scrub_duration":0.0032583899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.761502+0000","last_change":"2024-09-10T18:52:07.501745+0000","last_active":"2024-09-10T18:52:07.761502+0000","last_peered":"2024-09-10T18:52:07.761502+0000","last_clean":"2024-09-10T18:52:07.761502+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:52:07.761502+0000","last_undegraded":"2024-09-10T18:52:07.761502+0000","last_fullsized":"2024-09-10T18:52:07.761502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.501543+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.400864+0000","scrub_duration":0.0064925740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"80'2","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.014076+0000","last_change":"2024-09-10T18:50:47.857492+0000","last_active":"2024-09-10T18:51:40.014076+0000","last_peered":"2024-09-10T18:51:40.014076+0000","last_clean":"2024-09-10T18:51:40.014076+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:51:40.014076+0000","last_undegraded":"2024-09-10T18:51:40.014076+0000","last_fullsized":"2024-09-10T18:51:40.014076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T18:50:47.857351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:47.857351+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:10.727526+0000","scrub_duration":0.0034194500000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.016026+0000","last_change":"2024-09-10T18:51:11.652632+0000","last_active":"2024-09-10T18:51:40.016026+0000","last_peered":"2024-09-10T18:51:40.016026+0000","last_clean":"2024-09-10T18:51:40.016026+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:51:40.016026+0000","last_undegraded":"2024-09-10T18:51:40.016026+0000","last_fullsized":"2024-09-10T18:51:40.016026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:11.652501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:11.652501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:39.556031+0000","scrub_duration":0.0028739529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":82,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:58.828690+0000","last_change":"2024-09-10T18:51:58.828562+0000","last_active":"2024-09-10T18:51:58.828690+0000","last_peered":"2024-09-10T18:51:58.828690+0000","last_clean":"2024-09-10T18:51:58.828690+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:51:58.828690+0000","last_undegraded":"2024-09-10T18:51:58.828690+0000","last_fullsized":"2024-09-10T18:51:58.828690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:58.828403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.828403+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.385475+0000","scrub_duration":0.0052576080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018485+0000","last_change":"2024-09-10T18:50:59.749352+0000","last_active":"2024-09-10T18:51:40.018485+0000","last_peered":"2024-09-10T18:51:40.018485+0000","last_clean":"2024-09-10T18:51:40.018485+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:51:40.018485+0000","last_undegraded":"2024-09-10T18:51:40.018485+0000","last_fullsized":"2024-09-10T18:51:40.018485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:59.749193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:59.749193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:10.479902+0000","scrub_duration":0.003245782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.016481+0000","last_change":"2024-09-10T18:51:05.604974+0000","last_active":"2024-09-10T18:51:40.016481+0000","last_peered":"2024-09-10T18:51:40.016481+0000","last_clean":"2024-09-10T18:51:40.016481+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:51:40.016481+0000","last_undegraded":"2024-09-10T18:51:40.016481+0000","last_fullsized":"2024-09-10T18:51:40.016481+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:05.604804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:05.604804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:31.081415+0000","scrub_duration":0.0034418669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.651484+0000","last_change":"2024-09-10T18:51:15.485056+0000","last_active":"2024-09-10T18:51:40.651484+0000","last_peered":"2024-09-10T18:51:40.651484+0000","last_clean":"2024-09-10T18:51:40.651484+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:51:40.651484+0000","last_undegraded":"2024-09-10T18:51:40.651484+0000","last_fullsized":"2024-09-10T18:51:40.651484+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:15.484881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:15.484881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.398529+0000","scrub_duration":0.0034503160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:15.209856+0000","last_change":"2024-09-10T18:52:15.209771+0000","last_active":"2024-09-10T18:52:15.209856+0000","last_peered":"2024-09-10T18:52:15.209856+0000","last_clean":"2024-09-10T18:52:15.209856+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:52:15.209856+0000","last_undegraded":"2024-09-10T18:52:15.209856+0000","last_fullsized":"2024-09-10T18:52:15.209856+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:15.209609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.209609+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.019264+0000","scrub_duration":0.005193746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":62,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:09.838305+0000","last_change":"2024-09-10T18:52:09.838167+0000","last_active":"2024-09-10T18:52:09.838305+0000","last_peered":"2024-09-10T18:52:09.838305+0000","last_clean":"2024-09-10T18:52:09.838305+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:52:09.838305+0000","last_undegraded":"2024-09-10T18:52:09.838305+0000","last_fullsized":"2024-09-10T18:52:09.838305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.838015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.838015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.447360+0000","scrub_duration":0.003415202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":58,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:09.323964+0000","last_change":"2024-09-10T18:52:09.323881+0000","last_active":"2024-09-10T18:52:09.323964+0000","last_peered":"2024-09-10T18:52:09.323964+0000","last_clean":"2024-09-10T18:52:09.323964+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:52:09.323964+0000","last_undegraded":"2024-09-10T18:52:09.323964+0000","last_fullsized":"2024-09-10T18:52:09.323964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.323704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.323704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.895803+0000","scrub_duration":0.0036260730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.019191+0000","last_change":"2024-09-10T18:51:39.639863+0000","last_active":"2024-09-10T18:51:40.019191+0000","last_peered":"2024-09-10T18:51:40.019191+0000","last_clean":"2024-09-10T18:51:40.019191+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:51:40.019191+0000","last_undegraded":"2024-09-10T18:51:40.019191+0000","last_fullsized":"2024-09-10T18:51:40.019191+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.639709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.639709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:44.720874+0000","scrub_duration":0.0036283190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":100,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:58.431079+0000","last_change":"2024-09-10T18:51:58.430973+0000","last_active":"2024-09-10T18:51:58.431079+0000","last_peered":"2024-09-10T18:51:58.431079+0000","last_clean":"2024-09-10T18:51:58.431079+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:51:58.431079+0000","last_undegraded":"2024-09-10T18:51:58.431079+0000","last_fullsized":"2024-09-10T18:51:58.431079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.430868+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.088894+0000","scrub_duration":0.013718902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.776612+0000","last_change":"2024-09-10T18:51:40.776447+0000","last_active":"2024-09-10T18:51:40.776612+0000","last_peered":"2024-09-10T18:51:40.776612+0000","last_clean":"2024-09-10T18:51:40.776612+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:51:40.776612+0000","last_undegraded":"2024-09-10T18:51:40.776612+0000","last_fullsized":"2024-09-10T18:51:40.776612+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:40.776169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.776169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.421742+0000","scrub_duration":0.0038302309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018002+0000","last_change":"2024-09-10T18:51:06.641398+0000","last_active":"2024-09-10T18:51:40.018002+0000","last_peered":"2024-09-10T18:51:40.018002+0000","last_clean":"2024-09-10T18:51:40.018002+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:51:40.018002+0000","last_undegraded":"2024-09-10T18:51:40.018002+0000","last_fullsized":"2024-09-10T18:51:40.018002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:06.641227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.641227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:11.031881+0000","scrub_duration":0.003318765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:45.221596+0000","last_change":"2024-09-10T18:51:45.221499+0000","last_active":"2024-09-10T18:51:45.221596+0000","last_peered":"2024-09-10T18:51:45.221596+0000","last_clean":"2024-09-10T18:51:45.221596+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:51:45.221596+0000","last_undegraded":"2024-09-10T18:51:45.221596+0000","last_fullsized":"2024-09-10T18:51:45.221596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:45.221333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:45.221333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.678342+0000","scrub_duration":0.0036649180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:51.619834+0000","last_change":"2024-09-10T18:51:51.619698+0000","last_active":"2024-09-10T18:51:51.619834+0000","last_peered":"2024-09-10T18:51:51.619834+0000","last_clean":"2024-09-10T18:51:51.619834+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:51:51.619834+0000","last_undegraded":"2024-09-10T18:51:51.619834+0000","last_fullsized":"2024-09-10T18:51:51.619834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:51.619511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:51.619511+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:03.627716+0000","scrub_duration":0.0049634149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:06.149958+0000","last_change":"2024-09-10T18:52:06.149853+0000","last_active":"2024-09-10T18:52:06.149958+0000","last_peered":"2024-09-10T18:52:06.149958+0000","last_clean":"2024-09-10T18:52:06.149958+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:52:06.149958+0000","last_undegraded":"2024-09-10T18:52:06.149958+0000","last_fullsized":"2024-09-10T18:52:06.149958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.149699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.149699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.887082+0000","scrub_duration":0.0035198529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:50.626759+0000","last_change":"2024-09-10T18:51:50.626660+0000","last_active":"2024-09-10T18:51:50.626759+0000","last_peered":"2024-09-10T18:51:50.626759+0000","last_clean":"2024-09-10T18:51:50.626759+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:51:50.626759+0000","last_undegraded":"2024-09-10T18:51:50.626759+0000","last_fullsized":"2024-09-10T18:51:50.626759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.626514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.626514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.083006+0000","scrub_duration":0.0035366019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:04.358951+0000","last_change":"2024-09-10T18:52:04.358732+0000","last_active":"2024-09-10T18:52:04.358951+0000","last_peered":"2024-09-10T18:52:04.358951+0000","last_clean":"2024-09-10T18:52:04.358951+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:52:04.358951+0000","last_undegraded":"2024-09-10T18:52:04.358951+0000","last_fullsized":"2024-09-10T18:52:04.358951+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:04.358415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:04.358415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:06.405553+0000","scrub_duration":0.003743377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"77'10","reported_seq":67,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018345+0000","last_change":"2024-09-10T18:51:33.617562+0000","last_active":"2024-09-10T18:51:40.018345+0000","last_peered":"2024-09-10T18:51:40.018345+0000","last_clean":"2024-09-10T18:51:40.018345+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:51:40.018345+0000","last_undegraded":"2024-09-10T18:51:40.018345+0000","last_fullsized":"2024-09-10T18:51:40.018345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:33.617399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.617399+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.865090+0000","scrub_duration":0.0054712390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620374+0000","last_change":"2024-09-10T18:51:39.623163+0000","last_active":"2024-09-10T18:51:40.620374+0000","last_peered":"2024-09-10T18:51:40.620374+0000","last_clean":"2024-09-10T18:51:40.620374+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:51:40.620374+0000","last_undegraded":"2024-09-10T18:51:40.620374+0000","last_fullsized":"2024-09-10T18:51:40.620374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.623005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.623005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.432652+0000","scrub_duration":0.0034195559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":110,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.321197+0000","last_change":"2024-09-10T18:51:33.452673+0000","last_active":"2024-09-10T18:51:40.321197+0000","last_peered":"2024-09-10T18:51:40.321197+0000","last_clean":"2024-09-10T18:51:40.321197+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:51:40.321197+0000","last_undegraded":"2024-09-10T18:51:40.321197+0000","last_fullsized":"2024-09-10T18:51:40.321197+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:51:33.452504+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.452504+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.436046+0000","scrub_duration":0.0041999869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:03.330026+0000","last_change":"2024-09-10T18:52:03.329811+0000","last_active":"2024-09-10T18:52:03.330026+0000","last_peered":"2024-09-10T18:52:03.330026+0000","last_clean":"2024-09-10T18:52:03.330026+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:52:03.330026+0000","last_undegraded":"2024-09-10T18:52:03.330026+0000","last_fullsized":"2024-09-10T18:52:03.330026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.329632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.640377+0000","scrub_duration":0.0032347410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.619320+0000","last_change":"2024-09-10T18:51:12.608925+0000","last_active":"2024-09-10T18:51:40.619320+0000","last_peered":"2024-09-10T18:51:40.619320+0000","last_clean":"2024-09-10T18:51:40.619320+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:51:40.619320+0000","last_undegraded":"2024-09-10T18:51:40.619320+0000","last_fullsized":"2024-09-10T18:51:40.619320+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:12.608777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:12.608777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:17.966100+0000","scrub_duration":0.0032288310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":78,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.322158+0000","last_change":"2024-09-10T18:50:49.409493+0000","last_active":"2024-09-10T18:51:40.322158+0000","last_peered":"2024-09-10T18:51:40.322158+0000","last_clean":"2024-09-10T18:51:40.322158+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:51:40.322158+0000","last_undegraded":"2024-09-10T18:51:40.322158+0000","last_fullsized":"2024-09-10T18:51:40.322158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:50:49.409334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:49.409334+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:17.444318+0000","scrub_duration":0.0042892800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.455355+0000","last_change":"2024-09-10T18:52:07.455227+0000","last_active":"2024-09-10T18:52:07.455355+0000","last_peered":"2024-09-10T18:52:07.455355+0000","last_clean":"2024-09-10T18:52:07.455355+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:52:07.455355+0000","last_undegraded":"2024-09-10T18:52:07.455355+0000","last_fullsized":"2024-09-10T18:52:07.455355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.455062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.455062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.587408+0000","scrub_duration":0.0037580970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:52.321917+0000","last_change":"2024-09-10T18:51:52.321821+0000","last_active":"2024-09-10T18:51:52.321917+0000","last_peered":"2024-09-10T18:51:52.321917+0000","last_clean":"2024-09-10T18:51:52.321917+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:51:52.321917+0000","last_undegraded":"2024-09-10T18:51:52.321917+0000","last_fullsized":"2024-09-10T18:51:52.321917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:52.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:52.321671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:02.059416+0000","scrub_duration":0.003316977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.320028+0000","last_change":"2024-09-10T18:50:51.430175+0000","last_active":"2024-09-10T18:51:40.320028+0000","last_peered":"2024-09-10T18:51:40.320028+0000","last_clean":"2024-09-10T18:51:40.320028+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:51:40.320028+0000","last_undegraded":"2024-09-10T18:51:40.320028+0000","last_fullsized":"2024-09-10T18:51:40.320028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:51.430075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:51.430075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:18.726618+0000","scrub_duration":0.0043979570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"77'10","reported_seq":65,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620859+0000","last_change":"2024-09-10T18:51:03.530423+0000","last_active":"2024-09-10T18:51:40.620859+0000","last_peered":"2024-09-10T18:51:40.620859+0000","last_clean":"2024-09-10T18:51:40.620859+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:51:40.620859+0000","last_undegraded":"2024-09-10T18:51:40.620859+0000","last_fullsized":"2024-09-10T18:51:40.620859+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:03.530326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.530326+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.247623+0000","scrub_duration":0.0034148749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017333+0000","last_change":"2024-09-10T18:50:50.740960+0000","last_active":"2024-09-10T18:51:40.017333+0000","last_peered":"2024-09-10T18:51:40.017333+0000","last_clean":"2024-09-10T18:51:40.017333+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:51:40.017333+0000","last_undegraded":"2024-09-10T18:51:40.017333+0000","last_fullsized":"2024-09-10T18:51:40.017333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:50.740799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:50.740799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:15.120674+0000","scrub_duration":0.0030619250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.652976+0000","last_change":"2024-09-10T18:50:58.647711+0000","last_active":"2024-09-10T18:51:40.652976+0000","last_peered":"2024-09-10T18:51:40.652976+0000","last_clean":"2024-09-10T18:51:40.652976+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:51:40.652976+0000","last_undegraded":"2024-09-10T18:51:40.652976+0000","last_fullsized":"2024-09-10T18:51:40.652976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:58.647547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:58.647547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:23.263398+0000","scrub_duration":0.0031702420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.651804+0000","last_change":"2024-09-10T18:51:17.509138+0000","last_active":"2024-09-10T18:51:40.651804+0000","last_peered":"2024-09-10T18:51:40.651804+0000","last_clean":"2024-09-10T18:51:40.651804+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:51:40.651804+0000","last_undegraded":"2024-09-10T18:51:40.651804+0000","last_fullsized":"2024-09-10T18:51:40.651804+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:17.508961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:17.508961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:21.703492+0000","scrub_duration":0.0036482350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":92,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:47.304349+0000","last_change":"2024-09-10T18:51:47.304256+0000","last_active":"2024-09-10T18:51:47.304349+0000","last_peered":"2024-09-10T18:51:47.304349+0000","last_clean":"2024-09-10T18:51:47.304349+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:51:47.304349+0000","last_undegraded":"2024-09-10T18:51:47.304349+0000","last_fullsized":"2024-09-10T18:51:47.304349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.304153+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.010387+0000","scrub_duration":0.012604194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":74,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:53.688730+0000","last_change":"2024-09-10T18:51:53.688495+0000","last_active":"2024-09-10T18:51:53.688730+0000","last_peered":"2024-09-10T18:51:53.688730+0000","last_clean":"2024-09-10T18:51:53.688730+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:51:53.688730+0000","last_undegraded":"2024-09-10T18:51:53.688730+0000","last_fullsized":"2024-09-10T18:51:53.688730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:51:53.688302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.688302+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.064936+0000","scrub_duration":0.0040928329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:46.634087+0000","last_change":"2024-09-10T18:51:46.633977+0000","last_active":"2024-09-10T18:51:46.634087+0000","last_peered":"2024-09-10T18:51:46.634087+0000","last_clean":"2024-09-10T18:51:46.634087+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:51:46.634087+0000","last_undegraded":"2024-09-10T18:51:46.634087+0000","last_fullsized":"2024-09-10T18:51:46.634087+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.633831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.754683+0000","scrub_duration":0.003224747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:56.751648+0000","last_change":"2024-09-10T18:51:56.751545+0000","last_active":"2024-09-10T18:51:56.751648+0000","last_peered":"2024-09-10T18:51:56.751648+0000","last_clean":"2024-09-10T18:51:56.751648+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:51:56.751648+0000","last_undegraded":"2024-09-10T18:51:56.751648+0000","last_fullsized":"2024-09-10T18:51:56.751648+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.751398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.751398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.229956+0000","scrub_duration":0.0032004590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:05.179688+0000","last_change":"2024-09-10T18:52:05.179430+0000","last_active":"2024-09-10T18:52:05.179688+0000","last_peered":"2024-09-10T18:52:05.179688+0000","last_clean":"2024-09-10T18:52:05.179688+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:52:05.179688+0000","last_undegraded":"2024-09-10T18:52:05.179688+0000","last_fullsized":"2024-09-10T18:52:05.179688+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:05.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.179207+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.065476+0000","scrub_duration":0.0057240640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":103,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:43.700583+0000","last_change":"2024-09-10T18:51:43.700454+0000","last_active":"2024-09-10T18:51:43.700583+0000","last_peered":"2024-09-10T18:51:43.700583+0000","last_clean":"2024-09-10T18:51:43.700583+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:51:43.700583+0000","last_undegraded":"2024-09-10T18:51:43.700583+0000","last_fullsized":"2024-09-10T18:51:43.700583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:51:43.700290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.700290+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.352626+0000","scrub_duration":0.0033939529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.757250+0000","last_change":"2024-09-10T18:51:47.570557+0000","last_active":"2024-09-10T18:52:07.757250+0000","last_peered":"2024-09-10T18:52:07.757250+0000","last_clean":"2024-09-10T18:52:07.757250+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:52:07.757250+0000","last_undegraded":"2024-09-10T18:52:07.757250+0000","last_fullsized":"2024-09-10T18:52:07.757250+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:47.570369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.570369+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.658418+0000","scrub_duration":0.0040413760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018696+0000","last_change":"2024-09-10T18:51:23.683348+0000","last_active":"2024-09-10T18:51:40.018696+0000","last_peered":"2024-09-10T18:51:40.018696+0000","last_clean":"2024-09-10T18:51:40.018696+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:51:40.018696+0000","last_undegraded":"2024-09-10T18:51:40.018696+0000","last_fullsized":"2024-09-10T18:51:40.018696+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:23.683176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:23.683176+0000","last_clean_scrub_stamp":"2024-09-10T18:51:23.683176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.223369+0000","scrub_duration":0.0032660779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.320841+0000","last_change":"2024-09-10T18:51:29.406090+0000","last_active":"2024-09-10T18:51:40.320841+0000","last_peered":"2024-09-10T18:51:40.320841+0000","last_clean":"2024-09-10T18:51:40.320841+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:51:40.320841+0000","last_undegraded":"2024-09-10T18:51:40.320841+0000","last_fullsized":"2024-09-10T18:51:40.320841+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:29.405903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.405903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:35.496037+0000","scrub_duration":0.0052412719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:59.561526+0000","last_change":"2024-09-10T18:51:59.561431+0000","last_active":"2024-09-10T18:51:59.561526+0000","last_peered":"2024-09-10T18:51:59.561526+0000","last_clean":"2024-09-10T18:51:59.561526+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:51:59.561526+0000","last_undegraded":"2024-09-10T18:51:59.561526+0000","last_fullsized":"2024-09-10T18:51:59.561526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_clean_scrub_stamp":"2024-09-10T18:51:59.561306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.793374+0000","scrub_duration":0.003474277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:01.485381+0000","last_change":"2024-09-10T18:52:01.485093+0000","last_active":"2024-09-10T18:52:01.485381+0000","last_peered":"2024-09-10T18:52:01.485381+0000","last_clean":"2024-09-10T18:52:01.485381+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:52:01.485381+0000","last_undegraded":"2024-09-10T18:52:01.485381+0000","last_fullsized":"2024-09-10T18:52:01.485381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.484734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.484734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.349048+0000","scrub_duration":0.0054940589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":56,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:11.452023+0000","last_change":"2024-09-10T18:52:11.451951+0000","last_active":"2024-09-10T18:52:11.452023+0000","last_peered":"2024-09-10T18:52:11.452023+0000","last_clean":"2024-09-10T18:52:11.452023+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:52:11.452023+0000","last_undegraded":"2024-09-10T18:52:11.452023+0000","last_fullsized":"2024-09-10T18:52:11.452023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.451797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.451797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:26.258751+0000","scrub_duration":0.0033137470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"77'4","reported_seq":66,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:11.496933+0000","last_change":"2024-09-10T18:52:11.496842+0000","last_active":"2024-09-10T18:52:11.496933+0000","last_peered":"2024-09-10T18:52:11.496933+0000","last_clean":"2024-09-10T18:52:11.496933+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:52:11.496933+0000","last_undegraded":"2024-09-10T18:52:11.496933+0000","last_fullsized":"2024-09-10T18:52:11.496933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-10T18:52:11.496670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.496670+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:24.908807+0000","scrub_duration":0.0041783929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:00.390748+0000","last_change":"2024-09-10T18:52:00.390577+0000","last_active":"2024-09-10T18:52:00.390748+0000","last_peered":"2024-09-10T18:52:00.390748+0000","last_clean":"2024-09-10T18:52:00.390748+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:52:00.390748+0000","last_undegraded":"2024-09-10T18:52:00.390748+0000","last_fullsized":"2024-09-10T18:52:00.390748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:00.390415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.390415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.045571+0000","scrub_duration":0.00325318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018979+0000","last_change":"2024-09-10T18:51:14.705208+0000","last_active":"2024-09-10T18:51:40.018979+0000","last_peered":"2024-09-10T18:51:40.018979+0000","last_clean":"2024-09-10T18:51:40.018979+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:51:40.018979+0000","last_undegraded":"2024-09-10T18:51:40.018979+0000","last_fullsized":"2024-09-10T18:51:40.018979+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:14.705032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:14.705032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.732429+0000","scrub_duration":0.0038292510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.198150+0000","last_change":"2024-09-10T18:52:07.197975+0000","last_active":"2024-09-10T18:52:07.198150+0000","last_peered":"2024-09-10T18:52:07.198150+0000","last_clean":"2024-09-10T18:52:07.198150+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:52:07.198150+0000","last_undegraded":"2024-09-10T18:52:07.198150+0000","last_fullsized":"2024-09-10T18:52:07.198150+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.197730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.197730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.782447+0000","scrub_duration":0.003470014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"77'11","reported_seq":69,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.321486+0000","last_change":"2024-09-10T18:51:00.361402+0000","last_active":"2024-09-10T18:51:40.321486+0000","last_peered":"2024-09-10T18:51:40.321486+0000","last_clean":"2024-09-10T18:51:40.321486+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:51:40.321486+0000","last_undegraded":"2024-09-10T18:51:40.321486+0000","last_fullsized":"2024-09-10T18:51:40.321486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:00.361230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:00.361230+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.900016+0000","scrub_duration":0.0048395340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620339+0000","last_change":"2024-09-10T18:51:13.565917+0000","last_active":"2024-09-10T18:51:40.620339+0000","last_peered":"2024-09-10T18:51:40.620339+0000","last_clean":"2024-09-10T18:51:40.620339+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:51:40.620339+0000","last_undegraded":"2024-09-10T18:51:40.620339+0000","last_fullsized":"2024-09-10T18:51:40.620339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:13.565686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:13.565686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.569343+0000","scrub_duration":0.0037071460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.763150+0000","last_change":"2024-09-10T18:50:49.741360+0000","last_active":"2024-09-10T18:52:07.763150+0000","last_peered":"2024-09-10T18:52:07.763150+0000","last_clean":"2024-09-10T18:52:07.763150+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:52:07.763150+0000","last_undegraded":"2024-09-10T18:52:07.763150+0000","last_fullsized":"2024-09-10T18:52:07.763150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:50:49.741167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:49.741167+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038900150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.651523+0000","last_change":"2024-09-10T18:51:20.403072+0000","last_active":"2024-09-10T18:51:40.651523+0000","last_peered":"2024-09-10T18:51:40.651523+0000","last_clean":"2024-09-10T18:51:40.651523+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:51:40.651523+0000","last_undegraded":"2024-09-10T18:51:40.651523+0000","last_fullsized":"2024-09-10T18:51:40.651523+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:51:20.402922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:24.916001+0000","scrub_duration":0.0031476540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":68,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.652366+0000","last_change":"2024-09-10T18:51:10.452572+0000","last_active":"2024-09-10T18:51:40.652366+0000","last_peered":"2024-09-10T18:51:40.652366+0000","last_clean":"2024-09-10T18:51:40.652366+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:51:40.652366+0000","last_undegraded":"2024-09-10T18:51:40.652366+0000","last_fullsized":"2024-09-10T18:51:40.652366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:10.452379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:10.452379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.851342+0000","scrub_duration":0.0051699889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:56.575415+0000","last_change":"2024-09-10T18:51:56.575310+0000","last_active":"2024-09-10T18:51:56.575415+0000","last_peered":"2024-09-10T18:51:56.575415+0000","last_clean":"2024-09-10T18:51:56.575415+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:51:56.575415+0000","last_undegraded":"2024-09-10T18:51:56.575415+0000","last_fullsized":"2024-09-10T18:51:56.575415+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.575153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.564777+0000","scrub_duration":0.0033985280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":93,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.937247+0000","last_change":"2024-09-10T18:51:21.667022+0000","last_active":"2024-09-10T18:52:07.937247+0000","last_peered":"2024-09-10T18:52:07.937247+0000","last_clean":"2024-09-10T18:52:07.937247+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:52:07.937247+0000","last_undegraded":"2024-09-10T18:52:07.937247+0000","last_fullsized":"2024-09-10T18:52:07.937247+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:21.666847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:21.666847+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:48.835363+0000","scrub_duration":0.0041978249999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.692545+0000","last_change":"2024-09-10T18:50:45.424958+0000","last_active":"2024-09-10T18:51:40.692545+0000","last_peered":"2024-09-10T18:51:40.692545+0000","last_clean":"2024-09-10T18:51:40.692545+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:51:40.692545+0000","last_undegraded":"2024-09-10T18:51:40.692545+0000","last_fullsized":"2024-09-10T18:51:40.692545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:45.424754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:45.424754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:09.941679+0000","scrub_duration":0.0034751560000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:08.245872+0000","last_change":"2024-09-10T18:52:08.245784+0000","last_active":"2024-09-10T18:52:08.245872+0000","last_peered":"2024-09-10T18:52:08.245872+0000","last_clean":"2024-09-10T18:52:08.245872+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:52:08.245872+0000","last_undegraded":"2024-09-10T18:52:08.245872+0000","last_fullsized":"2024-09-10T18:52:08.245872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:08.245617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:08.245617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.667622+0000","scrub_duration":0.0044282050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.013900+0000","last_change":"2024-09-10T18:51:03.040657+0000","last_active":"2024-09-10T18:51:40.013900+0000","last_peered":"2024-09-10T18:51:40.013900+0000","last_clean":"2024-09-10T18:51:40.013900+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:51:40.013900+0000","last_undegraded":"2024-09-10T18:51:40.013900+0000","last_fullsized":"2024-09-10T18:51:40.013900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:03.040495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.040495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:22.626127+0000","scrub_duration":0.0035544589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":49,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:11.856640+0000","last_change":"2024-09-10T18:52:11.856507+0000","last_active":"2024-09-10T18:52:11.856640+0000","last_peered":"2024-09-10T18:52:11.856640+0000","last_clean":"2024-09-10T18:52:11.856640+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:52:11.856640+0000","last_undegraded":"2024-09-10T18:52:11.856640+0000","last_fullsized":"2024-09-10T18:52:11.856640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.856337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.856337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.241036+0000","scrub_duration":0.0034053619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":58,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:14.539078+0000","last_change":"2024-09-10T18:52:14.538993+0000","last_active":"2024-09-10T18:52:14.539078+0000","last_peered":"2024-09-10T18:52:14.539078+0000","last_clean":"2024-09-10T18:52:14.539078+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:52:14.539078+0000","last_undegraded":"2024-09-10T18:52:14.539078+0000","last_fullsized":"2024-09-10T18:52:14.539078+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:14.538838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:14.538838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.876083+0000","scrub_duration":0.0042549700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"77'10","reported_seq":77,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:11.222864+0000","last_change":"2024-09-10T18:52:11.222775+0000","last_active":"2024-09-10T18:52:11.222864+0000","last_peered":"2024-09-10T18:52:11.222864+0000","last_clean":"2024-09-10T18:52:11.222864+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:52:11.222864+0000","last_undegraded":"2024-09-10T18:52:11.222864+0000","last_fullsized":"2024-09-10T18:52:11.222864+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:11.222598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.222598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.658086+0000","scrub_duration":0.0051546400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017268+0000","last_change":"2024-09-10T18:51:35.492628+0000","last_active":"2024-09-10T18:51:40.017268+0000","last_peered":"2024-09-10T18:51:40.017268+0000","last_clean":"2024-09-10T18:51:40.017268+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:51:40.017268+0000","last_undegraded":"2024-09-10T18:51:40.017268+0000","last_fullsized":"2024-09-10T18:51:40.017268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:35.492463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:35.492463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:35.871414+0000","scrub_duration":0.0032631600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.691680+0000","last_change":"2024-09-10T18:50:54.559365+0000","last_active":"2024-09-10T18:51:40.691680+0000","last_peered":"2024-09-10T18:51:40.691680+0000","last_clean":"2024-09-10T18:51:40.691680+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:51:40.691680+0000","last_undegraded":"2024-09-10T18:51:40.691680+0000","last_fullsized":"2024-09-10T18:51:40.691680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:54.559212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:54.559212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:08.677292+0000","scrub_duration":0.003302967,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:57.533522+0000","last_change":"2024-09-10T18:51:57.533423+0000","last_active":"2024-09-10T18:51:57.533522+0000","last_peered":"2024-09-10T18:51:57.533522+0000","last_clean":"2024-09-10T18:51:57.533522+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:51:57.533522+0000","last_undegraded":"2024-09-10T18:51:57.533522+0000","last_fullsized":"2024-09-10T18:51:57.533522+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:57.533281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.533281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.018032+0000","scrub_duration":0.0030396329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'6","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.653272+0000","last_change":"2024-09-10T18:51:21.383448+0000","last_active":"2024-09-10T18:51:40.653272+0000","last_peered":"2024-09-10T18:51:40.653272+0000","last_clean":"2024-09-10T18:51:40.653272+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:51:40.653272+0000","last_undegraded":"2024-09-10T18:51:40.653272+0000","last_fullsized":"2024-09-10T18:51:40.653272+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-10T18:51:21.383324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:21.383324+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.795348+0000","scrub_duration":0.003470833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620980+0000","last_change":"2024-09-10T18:51:07.542429+0000","last_active":"2024-09-10T18:51:40.620980+0000","last_peered":"2024-09-10T18:51:40.620980+0000","last_clean":"2024-09-10T18:51:40.620980+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:51:40.620980+0000","last_undegraded":"2024-09-10T18:51:40.620980+0000","last_fullsized":"2024-09-10T18:51:40.620980+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:07.542270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:07.542270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.429613+0000","scrub_duration":0.0032886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017415+0000","last_change":"2024-09-10T18:51:38.650934+0000","last_active":"2024-09-10T18:51:40.017415+0000","last_peered":"2024-09-10T18:51:40.017415+0000","last_clean":"2024-09-10T18:51:40.017415+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:51:40.017415+0000","last_undegraded":"2024-09-10T18:51:40.017415+0000","last_fullsized":"2024-09-10T18:51:40.017415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.650775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.247838+0000","scrub_duration":0.0033420289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":56,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.320650+0000","last_change":"2024-09-10T18:51:37.345540+0000","last_active":"2024-09-10T18:51:40.320650+0000","last_peered":"2024-09-10T18:51:40.320650+0000","last_clean":"2024-09-10T18:51:40.320650+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:51:40.320650+0000","last_undegraded":"2024-09-10T18:51:40.320650+0000","last_fullsized":"2024-09-10T18:51:40.320650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:37.345379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.345379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.645490+0000","scrub_duration":0.0032740450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.620816+0000","last_change":"2024-09-10T18:51:16.552571+0000","last_active":"2024-09-10T18:51:40.620816+0000","last_peered":"2024-09-10T18:51:40.620816+0000","last_clean":"2024-09-10T18:51:40.620816+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:51:40.620816+0000","last_undegraded":"2024-09-10T18:51:40.620816+0000","last_fullsized":"2024-09-10T18:51:40.620816+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:16.552401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:16.552401+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:22.084996+0000","scrub_duration":0.0044786210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:49.224195+0000","last_change":"2024-09-10T18:51:49.224075+0000","last_active":"2024-09-10T18:51:49.224195+0000","last_peered":"2024-09-10T18:51:49.224195+0000","last_clean":"2024-09-10T18:51:49.224195+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:51:49.224195+0000","last_undegraded":"2024-09-10T18:51:49.224195+0000","last_fullsized":"2024-09-10T18:51:49.224195+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:49.223905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:49.223905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.495696+0000","scrub_duration":0.003346421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017535+0000","last_change":"2024-09-10T18:51:06.774645+0000","last_active":"2024-09-10T18:51:40.017535+0000","last_peered":"2024-09-10T18:51:40.017535+0000","last_clean":"2024-09-10T18:51:40.017535+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:51:40.017535+0000","last_undegraded":"2024-09-10T18:51:40.017535+0000","last_fullsized":"2024-09-10T18:51:40.017535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:06.774496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.774496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.898966+0000","scrub_duration":0.0031373199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":48,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018850+0000","last_change":"2024-09-10T18:50:57.823090+0000","last_active":"2024-09-10T18:51:40.018850+0000","last_peered":"2024-09-10T18:51:40.018850+0000","last_clean":"2024-09-10T18:51:40.018850+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:51:40.018850+0000","last_undegraded":"2024-09-10T18:51:40.018850+0000","last_fullsized":"2024-09-10T18:51:40.018850+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:57.822942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:57.822942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:17.859503+0000","scrub_duration":0.0036355409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":49,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.321436+0000","last_change":"2024-09-10T18:51:06.380763+0000","last_active":"2024-09-10T18:51:40.321436+0000","last_peered":"2024-09-10T18:51:40.321436+0000","last_clean":"2024-09-10T18:51:40.321436+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:51:40.321436+0000","last_undegraded":"2024-09-10T18:51:40.321436+0000","last_fullsized":"2024-09-10T18:51:40.321436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:51:06.380593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.380593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.734018+0000","scrub_duration":0.0040786110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":85,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:50.544504+0000","last_change":"2024-09-10T18:51:50.544356+0000","last_active":"2024-09-10T18:51:50.544504+0000","last_peered":"2024-09-10T18:51:50.544504+0000","last_clean":"2024-09-10T18:51:50.544504+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:51:50.544504+0000","last_undegraded":"2024-09-10T18:51:50.544504+0000","last_fullsized":"2024-09-10T18:51:50.544504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:50.544186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.544186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.788233+0000","scrub_duration":0.0064898330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":52,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.619837+0000","last_change":"2024-09-10T18:51:00.467363+0000","last_active":"2024-09-10T18:51:40.619837+0000","last_peered":"2024-09-10T18:51:40.619837+0000","last_clean":"2024-09-10T18:51:40.619837+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:51:40.619837+0000","last_undegraded":"2024-09-10T18:51:40.619837+0000","last_fullsized":"2024-09-10T18:51:40.619837+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:00.467197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:00.467197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002892379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.653041+0000","last_change":"2024-09-10T18:51:12.468300+0000","last_active":"2024-09-10T18:51:40.653041+0000","last_peered":"2024-09-10T18:51:40.653041+0000","last_clean":"2024-09-10T18:51:40.653041+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:51:40.653041+0000","last_undegraded":"2024-09-10T18:51:40.653041+0000","last_fullsized":"2024-09-10T18:51:40.653041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:12.468139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:12.468139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:20.515171+0000","scrub_duration":0.0035583490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":46,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017086+0000","last_change":"2024-09-10T18:51:04.594129+0000","last_active":"2024-09-10T18:51:40.017086+0000","last_peered":"2024-09-10T18:51:40.017086+0000","last_clean":"2024-09-10T18:51:40.017086+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:51:40.017086+0000","last_undegraded":"2024-09-10T18:51:40.017086+0000","last_fullsized":"2024-09-10T18:51:40.017086+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:04.593968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:04.593968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.665764+0000","scrub_duration":0.003534648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":68,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017705+0000","last_change":"2024-09-10T18:51:28.596878+0000","last_active":"2024-09-10T18:51:40.017705+0000","last_peered":"2024-09-10T18:51:40.017705+0000","last_clean":"2024-09-10T18:51:40.017705+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:51:40.017705+0000","last_undegraded":"2024-09-10T18:51:40.017705+0000","last_fullsized":"2024-09-10T18:51:40.017705+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:28.596672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.596672+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:50.156773+0000","scrub_duration":0.0055701980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":54,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.017999+0000","last_change":"2024-09-10T18:51:02.778350+0000","last_active":"2024-09-10T18:51:40.017999+0000","last_peered":"2024-09-10T18:51:40.017999+0000","last_clean":"2024-09-10T18:51:40.017999+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:51:40.017999+0000","last_undegraded":"2024-09-10T18:51:40.017999+0000","last_fullsized":"2024-09-10T18:51:40.017999+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:02.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:02.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:09.366038+0000","scrub_duration":0.003123444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":120,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:07.940963+0000","last_change":"2024-09-10T18:51:55.299569+0000","last_active":"2024-09-10T18:52:07.940963+0000","last_peered":"2024-09-10T18:52:07.940963+0000","last_clean":"2024-09-10T18:52:07.940963+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:52:07.940963+0000","last_undegraded":"2024-09-10T18:52:07.940963+0000","last_fullsized":"2024-09-10T18:52:07.940963+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:51:55.299396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.299396+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.997434+0000","scrub_duration":0.0038567179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:41.706254+0000","last_change":"2024-09-10T18:51:41.706112+0000","last_active":"2024-09-10T18:51:41.706254+0000","last_peered":"2024-09-10T18:51:41.706254+0000","last_clean":"2024-09-10T18:51:41.706254+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:51:41.706254+0000","last_undegraded":"2024-09-10T18:51:41.706254+0000","last_fullsized":"2024-09-10T18:51:41.706254+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.705954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.045292+0000","scrub_duration":0.003398917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.014321+0000","last_change":"2024-09-10T18:50:49.921311+0000","last_active":"2024-09-10T18:51:40.014321+0000","last_peered":"2024-09-10T18:51:40.014321+0000","last_clean":"2024-09-10T18:51:40.014321+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:51:40.014321+0000","last_undegraded":"2024-09-10T18:51:40.014321+0000","last_fullsized":"2024-09-10T18:51:40.014321+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:50:49.921196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:49.921196+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:11.292657+0000","scrub_duration":0.0035193099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":72,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-10T18:52:09.509736+0000","last_change":"2024-09-10T18:52:09.509564+0000","last_active":"2024-09-10T18:52:09.509736+0000","last_peered":"2024-09-10T18:52:09.509736+0000","last_clean":"2024-09-10T18:52:09.509736+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:52:09.509736+0000","last_undegraded":"2024-09-10T18:52:09.509736+0000","last_fullsized":"2024-09-10T18:52:09.509736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:52:09.509384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.509384+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.310603+0000","scrub_duration":0.0050191510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:56.652942+0000","last_change":"2024-09-10T18:51:56.652829+0000","last_active":"2024-09-10T18:51:56.652942+0000","last_peered":"2024-09-10T18:51:56.652942+0000","last_clean":"2024-09-10T18:51:56.652942+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:51:56.652942+0000","last_undegraded":"2024-09-10T18:51:56.652942+0000","last_fullsized":"2024-09-10T18:51:56.652942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.652676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.654243+0000","scrub_duration":0.003518115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":58,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:52:06.609726+0000","last_change":"2024-09-10T18:52:06.609575+0000","last_active":"2024-09-10T18:52:06.609726+0000","last_peered":"2024-09-10T18:52:06.609726+0000","last_clean":"2024-09-10T18:52:06.609726+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:52:06.609726+0000","last_undegraded":"2024-09-10T18:52:06.609726+0000","last_fullsized":"2024-09-10T18:52:06.609726+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.609346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.609346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.679962+0000","scrub_duration":0.0031752439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":64,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:42.590358+0000","last_change":"2024-09-10T18:51:42.590260+0000","last_active":"2024-09-10T18:51:42.590358+0000","last_peered":"2024-09-10T18:51:42.590358+0000","last_clean":"2024-09-10T18:51:42.590358+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:51:42.590358+0000","last_undegraded":"2024-09-10T18:51:42.590358+0000","last_fullsized":"2024-09-10T18:51:42.590358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:42.590071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:42.590071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.690989+0000","scrub_duration":0.003624726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":78,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:57.420868+0000","last_change":"2024-09-10T18:51:57.420671+0000","last_active":"2024-09-10T18:51:57.420868+0000","last_peered":"2024-09-10T18:51:57.420868+0000","last_clean":"2024-09-10T18:51:57.420868+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:51:57.420868+0000","last_undegraded":"2024-09-10T18:51:57.420868+0000","last_fullsized":"2024-09-10T18:51:57.420868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:57.420489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.420489+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.697937+0000","scrub_duration":0.0042932839999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":50,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-10T18:51:40.018850+0000","last_change":"2024-09-10T18:51:25.626798+0000","last_active":"2024-09-10T18:51:40.018850+0000","last_peered":"2024-09-10T18:51:40.018850+0000","last_clean":"2024-09-10T18:51:40.018850+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:51:40.018850+0000","last_undegraded":"2024-09-10T18:51:40.018850+0000","last_fullsized":"2024-09-10T18:51:40.018850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:25.626688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:25.626688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.833450+0000","scrub_duration":0.0026021970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":153,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":422,"ondisk_log_size":422,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972399,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298612,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446540,"statfs":{"total":95995035648,"available":95689256960,"internally_reserved":0,"allocated":1290240,"data_stored":914202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91500000000000004,"15min":0.93500000000000005},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3129999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.044},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91900000000000004,"15min":0.93100000000000005},"min":{"1min":0.61399999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2669999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":1.119}]},{"osd":1,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.89700000000000002,"15min":0.90500000000000003},"min":{"1min":0.64800000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0309999999999999,"15min":1.0529999999999999},"min":{"1min":0.68500000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.2110000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.0640000000000001},"min":{"1min":0.64300000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.97499999999999998,"15min":0.97699999999999998},"min":{"1min":0.72399999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3340000000000001,"5min":1.353,"15min":1.353},"last":1.236}]},{"osd":3,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.998,"15min":1.0209999999999999},"min":{"1min":0.66200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.266,"5min":1.335,"15min":1.335},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.95999999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.0800000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.004,"15min":1.0269999999999999},"min":{"1min":0.58099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0209999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.0109999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99199999999999999,"15min":0.996},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1659999999999999,"15min":1.196},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.746,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.2429999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1850000000000001,"15min":1.1930000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0070000000000001,"5min":2.536,"15min":2.536},"last":1.1859999999999999}]}]},{"osd":6,"up_from":52,"seq":223338299449,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1269760,"data_stored":912441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.165,"5min":1.272,"15min":1.272},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3300000000000001,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.1479999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.91400000000000003,"15min":0.88400000000000001},"min":{"1min":0.626,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2350000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.91600000000000004,"15min":0.91300000000000003},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.2529999999999999,"5min":1.361,"15min":1.361},"last":0.66400000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.93500000000000005,"15min":0.91600000000000004},"min":{"1min":0.80700000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2809999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.79700000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":0.96699999999999997,"15min":0.94599999999999995},"min":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4370000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.95899999999999996,"15min":0.96899999999999997},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2430000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.95999999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.88,"15min":0.85599999999999998},"min":{"1min":0.57599999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.03},{"interface":"front","average":{"1min":1.075,"5min":1.006,"15min":1.006},"min":{"1min":0.68100000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.4379999999999999,"5min":1.744,"15min":1.744},"last":1.1799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98199999999999998,"15min":0.96399999999999997},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.21},{"interface":"front","average":{"1min":1.073,"5min":1.0780000000000001,"15min":1.0840000000000001},"min":{"1min":0.64800000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5009999999999999,"5min":1.665,"15min":1.665},"last":1.127}]},{"osd":7,"last update":"Tue Sep 10 18:51:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0529999999999999,"15min":1.0389999999999999},"min":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.516,"5min":1.573,"15min":1.573},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.141,"15min":1.137},"min":{"1min":0.68500000000000005,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.4279999999999999,"5min":1.488,"15min":1.488},"last":1.264}]}]},{"osd":5,"up_from":43,"seq":184683593795,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":679936,"data_stored":322152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.88800000000000001,"15min":0.83999999999999997},"min":{"1min":0.66200000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.258,"5min":1.609,"15min":1.609},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1990000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.97899999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.86899999999999999,"15min":0.85499999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2949999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.91700000000000004,"15min":0.88100000000000001},"min":{"1min":0.70699999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2230000000000001,"5min":1.635,"15min":1.635},"last":0.88400000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91500000000000004,"15min":0.88500000000000001},"min":{"1min":0.628,"5min":0.496,"15min":0.496},"max":{"1min":1.274,"5min":1.496,"15min":1.496},"last":1.107},{"interface":"front","average":{"1min":0.999,"5min":0.93899999999999995,"15min":0.89400000000000002},"min":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3180000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.94499999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.64900000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.242,"5min":1.409,"15min":1.409},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.69199999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2090000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.80200000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83099999999999996,"15min":0.77000000000000002},"min":{"1min":0.65100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1499999999999999,"5min":1.77,"15min":1.77},"last":1.083},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92400000000000004,"15min":0.85299999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3580000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.61899999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0229999999999999,"15min":1.0089999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2070000000000001,"5min":1.462,"15min":1.462},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.0580000000000001,"15min":1.032},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.361,"5min":1.653,"15min":1.653},"last":1.016}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.046,"15min":1.0680000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.294,"5min":1.837,"15min":1.837},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.1379999999999999,"15min":1.1759999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.296,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":0.82199999999999995}]}]},{"osd":4,"up_from":35,"seq":150323855439,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":716800,"data_stored":323717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.877,"15min":0.85099999999999998},"min":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.171,"5min":1.204,"15min":1.204},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91300000000000003,"15min":0.82599999999999996},"min":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56000000000000005},"max":{"1min":1.1870000000000001,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.92000000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.90400000000000003,"15min":0.88800000000000001},"min":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2869999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92600000000000005,"15min":0.84499999999999997},"min":{"1min":0.624,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.3480000000000001,"5min":1.456,"15min":1.456},"last":0.96799999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.93799999999999994,"15min":0.872},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3160000000000001,"5min":1.425,"15min":1.425},"last":1.05},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.94699999999999995,"15min":0.872},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.59699999999999998},"max":{"1min":1.3180000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.0249999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95699999999999996,"15min":0.88900000000000001},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.56399999999999995},"max":{"1min":1.327,"5min":1.714,"15min":1.714},"last":0.747},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97099999999999997,"15min":0.91700000000000004},"min":{"1min":0.79800000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.339,"5min":1.391,"15min":1.391},"last":1.0740000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.90300000000000002,"15min":0.83499999999999996},"min":{"1min":0.63200000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3939999999999999,"5min":1.484,"15min":1.484},"last":1.145},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97299999999999998,"15min":0.88100000000000001},"min":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.405,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.83899999999999997}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.96999999999999997,"15min":0.92400000000000004},"min":{"1min":0.77200000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0880000000000001,"15min":1.054},"min":{"1min":0.90200000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.5009999999999999,"5min":1.536,"15min":1.536},"last":1.2090000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.73699999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1579999999999999,"15min":1.1719999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.476,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.004}]}]},{"osd":3,"up_from":29,"seq":124554051674,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":679936,"data_stored":323314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.52500000000000002,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.01,"15min":1.2529999999999999},"last":0.79700000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.014,"5min":1.032,"15min":1.2869999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82599999999999996,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.48899999999999999,"15min":0.36699999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":1.274},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.78100000000000003},"min":{"1min":0.50600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.879},{"interface":"front","average":{"1min":0.75,"5min":0.77700000000000002,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1739999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70999999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:52:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.079,"15min":0.99099999999999999},"min":{"1min":0.83299999999999996,"5min":0.70099999999999996,"15min":0.56999999999999995},"max":{"1min":1.373,"5min":1.778,"15min":1.778},"last":1.431},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1579999999999999,"15min":1.069},"min":{"1min":0.86699999999999999,"5min":0.75700000000000001,"15min":0.59399999999999997},"max":{"1min":1.4239999999999999,"5min":1.952,"15min":1.952},"last":1.52}]},{"osd":5,"last update":"Tue Sep 10 18:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.113,"15min":1.0960000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5469999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.385},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.97899999999999998},"min":{"1min":0.72999999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4630000000000001,"5min":1.532,"15min":1.532},"last":0.93300000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.173,"15min":1.163},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.641,"5min":1.643,"15min":1.643},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.079,"15min":1.0669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.3620000000000001,"5min":1.476,"15min":1.476},"last":0.96499999999999997}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.113,"15min":1.1140000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.018},{"interface":"front","average":{"1min":1.232,"5min":1.242,"15min":1.2430000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.5940000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.472}]}]},{"osd":2,"up_from":19,"seq":81604378723,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447124,"statfs":{"total":95995035648,"available":95689854976,"internally_reserved":0,"allocated":692224,"data_stored":323511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69699999999999995,"15min":0.624},"min":{"1min":0.50800000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84799999999999998,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.78400000000000003,"15min":0.70899999999999996},"min":{"1min":0.49099999999999999,"5min":0.41399999999999998,"15min":0.30599999999999999},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.847}]},{"osd":1,"last update":"Tue Sep 10 18:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70699999999999996,"15min":0.66500000000000004},"min":{"1min":0.46999999999999997,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80900000000000005,"15min":0.72299999999999998},"min":{"1min":0.47599999999999998,"5min":0.45900000000000002,"15min":0.379},"max":{"1min":1.0980000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.698}]},{"osd":3,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88200000000000001,"15min":0.81299999999999994},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4510000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.278},{"interface":"front","average":{"1min":0.874,"5min":0.84499999999999997,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.982}]},{"osd":4,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.022,"15min":0.95599999999999996},"min":{"1min":0.745,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.425,"5min":1.542,"15min":1.542},"last":1.659},{"interface":"front","average":{"1min":1.107,"5min":1.115,"15min":1.0680000000000001},"min":{"1min":0.73299999999999998,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.627,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.448}]},{"osd":5,"last update":"Tue Sep 10 18:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0940000000000001,"15min":1.111},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.71,"5min":1.766,"15min":1.766},"last":1.802},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.004},"min":{"1min":0.72999999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5700000000000001,"5min":1.643,"15min":1.643},"last":1.4039999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:45 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.165,"15min":1.1819999999999999},"min":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.0780000000000001,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.542,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.752}]},{"osd":7,"last update":"Tue Sep 10 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2010000000000001,"15min":1.2070000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6459999999999999,"5min":1.671,"15min":1.671},"last":1.7829999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1200000000000001,"15min":1.1160000000000001},"min":{"1min":0.81699999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.633,"5min":1.708,"15min":1.708},"last":1.9339999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":688128,"data_stored":322741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.67900000000000005,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1160000000000001,"5min":1.121,"15min":1.121},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.76000000000000001,"15min":0.64300000000000002},"min":{"1min":0.496,"5min":0.46800000000000003,"15min":0.35299999999999998},"max":{"1min":1.0600000000000001,"5min":2.181,"15min":2.181},"last":0.67900000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:51:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.71599999999999997,"15min":0.64400000000000002},"min":{"1min":0.41699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.81799999999999995,"15min":0.72199999999999998},"min":{"1min":0.47299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.129,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.877,"15min":0.81899999999999995},"min":{"1min":0.629,"5min":0.435,"15min":0.435},"max":{"1min":1.2210000000000001,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77100000000000002,"15min":0.73399999999999999},"min":{"1min":0.59899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.78400000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.036,"15min":1},"min":{"1min":0.71899999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6830000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97699999999999998,"15min":0.88900000000000001},"min":{"1min":0.78400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.05}]},{"osd":5,"last update":"Tue Sep 10 18:51:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.995,"15min":1.012},"min":{"1min":0.77500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.115,"15min":1.131},"min":{"1min":0.86099999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.571,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:50 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1100000000000001,"15min":1.0760000000000001},"min":{"1min":0.82299999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.635,"5min":1.976,"15min":1.976},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.075,"15min":1.077},"min":{"1min":0.70599999999999996,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.98699999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.202,"15min":1.22},"min":{"1min":0.78300000000000003,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.569,"5min":2.125,"15min":2.125},"last":1.028},{"interface":"front","average":{"1min":1.159,"5min":1.143,"15min":1.147},"min":{"1min":0.751,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.1970000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298604,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446548,"statfs":{"total":95995035648,"available":95689265152,"internally_reserved":0,"allocated":1282048,"data_stored":912693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.56699999999999995},"min":{"1min":0.45000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.92300000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77000000000000002,"15min":0.63600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72199999999999998,"15min":0.67100000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0629999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.76700000000000002,"15min":0.67700000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.76700000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:51:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80500000000000005,"15min":0.72899999999999998},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2310000000000001,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82499999999999996,"15min":0.77100000000000002},"min":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.48699999999999999},"max":{"1min":1.4279999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.622}]},{"osd":4,"last update":"Tue Sep 10 18:52:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.1279999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0600000000000001,"15min":1.004},"min":{"1min":0.73599999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5189999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.079,"5min":1.091,"15min":1.081},"min":{"1min":0.68999999999999995,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6000000000000001,"5min":1.712,"15min":1.712},"last":0.95899999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1180000000000001,"15min":1.1000000000000001},"min":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.48,"5min":1.893,"15min":1.893},"last":1.214},{"interface":"front","average":{"1min":1.101,"5min":1.135,"15min":1.105},"min":{"1min":0.82599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.4219999999999999,"5min":2.363,"15min":2.363},"last":1.091}]},{"osd":7,"last update":"Tue Sep 10 18:51:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1539999999999999,"15min":1.1639999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.4810000000000001,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.2649999999999999,"15min":1.3080000000000001},"min":{"1min":0.71099999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4359999999999999,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.26}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:52:20.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:52:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:19 smithi165 bash[22591]: cluster 2024-09-10T18:52:18.503810+0000 mgr.y (mgr.24526) 109 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-09-10T18:52:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:19 smithi165 bash[22591]: cluster 2024-09-10T18:52:18.561909+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:52:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:19 smithi165 bash[22591]: cluster 2024-09-10T18:52:18.566040+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:52:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:19 smithi165 bash[22591]: cluster 2024-09-10T18:52:19.318497+0000 osd.1 (osd.1) 71 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:52:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:19 smithi165 bash[22591]: cluster 2024-09-10T18:52:19.321903+0000 osd.1 (osd.1) 72 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:52:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:20 smithi165 bash[22591]: audit 2024-09-10T18:52:19.356023+0000 mgr.y (mgr.24526) 110 : audit [DBG] from='client.14769 v1:172.21.15.83:0/4234137660' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:21.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:20 smithi083 bash[18629]: audit 2024-09-10T18:52:19.356023+0000 mgr.y (mgr.24526) 110 : audit [DBG] from='client.14769 v1:172.21.15.83:0/4234137660' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:21.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:20 smithi083 bash[32381]: audit 2024-09-10T18:52:19.356023+0000 mgr.y (mgr.24526) 110 : audit [DBG] from='client.14769 v1:172.21.15.83:0/4234137660' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:21.841 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:20.148937+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T18:52:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:20.152359+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T18:52:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:20.505246+0000 mgr.y (mgr.24526) 111 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:20.730736+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:52:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:20.733913+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:52:22.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.397297+0000 osd.2 (osd.2) 41 : cluster [DBG] 6.f scrub starts 2024-09-10T18:52:22.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:21 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.400842+0000 osd.2 (osd.2) 42 : cluster [DBG] 6.f scrub ok 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:20.148937+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:20.152359+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:20.505246+0000 mgr.y (mgr.24526) 111 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:20.730736+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:20.733913+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.397297+0000 osd.2 (osd.2) 41 : cluster [DBG] 6.f scrub starts 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.400842+0000 osd.2 (osd.2) 42 : cluster [DBG] 6.f scrub ok 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:20.148937+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:20.152359+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:20.505246+0000 mgr.y (mgr.24526) 111 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:22.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:20.730736+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:52:22.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:20.733913+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:52:22.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.397297+0000 osd.2 (osd.2) 41 : cluster [DBG] 6.f scrub starts 2024-09-10T18:52:22.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:21 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.400842+0000 osd.2 (osd.2) 42 : cluster [DBG] 6.f scrub ok 2024-09-10T18:52:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:22] "GET /metrics HTTP/1.1" 200 220095 "" "Prometheus/2.43.0" 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.433687+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.433787+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.729259+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.732795+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[32381]: cluster 2024-09-10T18:52:21.860913+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.433687+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.433787+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.729259+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:52:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.732795+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:52:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:22 smithi083 bash[18629]: cluster 2024-09-10T18:52:21.860913+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-10T18:52:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:22 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.433687+0000 osd.0 (osd.0) 71 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:22 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.433787+0000 osd.0 (osd.0) 72 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:22 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.729259+0000 osd.5 (osd.5) 61 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:52:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:22 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.732795+0000 osd.5 (osd.5) 62 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:52:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:22 smithi165 bash[22591]: cluster 2024-09-10T18:52:21.860913+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.181836+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.188819+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.322624+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.326306+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.505943+0000 mgr.y (mgr.24526) 112 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.688869+0000 osd.5 (osd.5) 63 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[18629]: cluster 2024-09-10T18:52:22.698366+0000 osd.5 (osd.5) 64 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:52:24.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.181836+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.188819+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.322624+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.326306+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.505943+0000 mgr.y (mgr.24526) 112 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.688869+0000 osd.5 (osd.5) 63 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:52:24.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:23 smithi083 bash[32381]: cluster 2024-09-10T18:52:22.698366+0000 osd.5 (osd.5) 64 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:52:24.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.181836+0000 osd.4 (osd.4) 85 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.188819+0000 osd.4 (osd.4) 86 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.322624+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.326306+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.505943+0000 mgr.y (mgr.24526) 112 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.688869+0000 osd.5 (osd.5) 63 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:52:24.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:23 smithi165 bash[22591]: cluster 2024-09-10T18:52:22.698366+0000 osd.5 (osd.5) 64 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:52:24.861 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.324368+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.328678+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.495001+0000 osd.7 (osd.7) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.495150+0000 osd.7 (osd.7) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.595438+0000 osd.0 (osd.0) 73 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.600123+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.710039+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.710149+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[18629]: cluster 2024-09-10T18:52:23.869797+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.324368+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:52:25.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.328678+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.495001+0000 osd.7 (osd.7) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.495150+0000 osd.7 (osd.7) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.595438+0000 osd.0 (osd.0) 73 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.600123+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.710039+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.710149+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:24 smithi083 bash[32381]: cluster 2024-09-10T18:52:23.869797+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-10T18:52:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.324368+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:52:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.328678+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:52:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.495001+0000 osd.7 (osd.7) 75 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.495150+0000 osd.7 (osd.7) 76 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.595438+0000 osd.0 (osd.0) 73 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.600123+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.710039+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.710149+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:24 smithi165 bash[22591]: cluster 2024-09-10T18:52:23.869797+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-09-10T18:52:26.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:24.506636+0000 mgr.y (mgr.24526) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:52:26.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:24.611097+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:52:26.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:24.614949+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:52:26.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:24.722770+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:24.726852+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.321688+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.325781+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:24.506636+0000 mgr.y (mgr.24526) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:24.611097+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:24.614949+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:24.722770+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:24.726852+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.321688+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:52:26.209 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:25 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.325781+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:52:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:24.506636+0000 mgr.y (mgr.24526) 113 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:52:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:24.611097+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:52:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:24.614949+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:52:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:24.722770+0000 osd.3 (osd.3) 79 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:52:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:24.726852+0000 osd.3 (osd.3) 80 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:52:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.321688+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:52:26.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:25 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.325781+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:52:27.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.175241+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.178785+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.458501+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.458632+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.580051+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.583475+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.753509+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.756871+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[18629]: cluster 2024-09-10T18:52:25.897185+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-10T18:52:27.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.175241+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.178785+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.458501+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.458632+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.580051+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.583475+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.753509+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.756871+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:52:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:26 smithi083 bash[32381]: cluster 2024-09-10T18:52:25.897185+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-10T18:52:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.175241+0000 osd.4 (osd.4) 87 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:52:27.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.178785+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.458501+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.458632+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.580051+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.583475+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.753509+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.756871+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:52:27.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:26 smithi165 bash[22591]: cluster 2024-09-10T18:52:25.897185+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-09-10T18:52:27.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:28.197 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[18629]: cluster 2024-09-10T18:52:26.139945+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:52:28.197 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[18629]: cluster 2024-09-10T18:52:26.143287+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:52:28.197 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[18629]: cluster 2024-09-10T18:52:26.507335+0000 mgr.y (mgr.24526) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:28.197 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[18629]: cluster 2024-09-10T18:52:26.612791+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:52:28.197 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[18629]: cluster 2024-09-10T18:52:26.617011+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:52:28.198 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[32381]: cluster 2024-09-10T18:52:26.139945+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:52:28.198 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[32381]: cluster 2024-09-10T18:52:26.143287+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:52:28.198 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[32381]: cluster 2024-09-10T18:52:26.507335+0000 mgr.y (mgr.24526) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:28.198 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[32381]: cluster 2024-09-10T18:52:26.612791+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:52:28.198 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:27 smithi083 bash[32381]: cluster 2024-09-10T18:52:26.617011+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:52:28.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- 172.21.15.83:0/3386505764 >> v1:172.21.15.83:6790/0 conn(0x7f7618108f10 legacy=0x7f761810b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- 172.21.15.83:0/3386505764 shutdown_connections 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- 172.21.15.83:0/3386505764 >> 172.21.15.83:0/3386505764 conn(0x7f76180f70b0 msgr2=0x7f76180f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- 172.21.15.83:0/3386505764 shutdown_connections 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- 172.21.15.83:0/3386505764 wait complete. 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 librados: starting msgr at 2024-09-10T18:52:28.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 librados: starting objecter 2024-09-10T18:52:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 Processor -- start 2024-09-10T18:52:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- start start 2024-09-10T18:52:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 librados: setting wanted keys 2024-09-10T18:52:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 librados: calling monclient init 2024-09-10T18:52:28.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f761811c3e0 con 0x7f7618108f10 2024-09-10T18:52:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f761811d5e0 con 0x7f76180fd5e0 2024-09-10T18:52:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761d183640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f761811e7e0 con 0x7f7618118670 2024-09-10T18:52:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7618108f10 0x7f7618104860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43390/0 (socket says 172.21.15.83:43390) 2024-09-10T18:52:28.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f761659c640 1 -- 172.21.15.83:0/2982694210 learned_addr learned my addr 172.21.15.83:0/2982694210 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f760f7fe640 10 client.?.objecter ms_handle_connect 0x7f7618108f10 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f760f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f760f7fe640 10 client.?.objecter ms_handle_connect 0x7f7618118670 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.299+0000 7f760f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 10 client.?.objecter ms_handle_connect 0x7f76180fd5e0 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:28.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080781434 0 0) 0x7f761811c3e0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76040038e0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833373714 0 0) 0x7f761811e7e0 con 0x7f7618118670 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761811c3e0 con 0x7f7618118670 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939312567 0 0) 0x7f761811d5e0 con 0x7f76180fd5e0 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761811e7e0 con 0x7f76180fd5e0 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964368981 0 0) 0x7f76040038e0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f761811d5e0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7600002cf0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3269449816 0 0) 0x7f761811c3e0 con 0x7f7618118670 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76040038e0 con 0x7f7618118670 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 246696181 0 0) 0x7f761811d5e0 con 0x7f7618108f10 2024-09-10T18:52:28.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 >> v1:172.21.15.83:6790/0 conn(0x7f7618118670 legacy=0x7f761811aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 >> v1:172.21.15.165:6789/0 conn(0x7f76180fd5e0 legacy=0x7f7618104150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761811f9e0 con 0x7f7618108f10 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761811d7d0 con 0x7f7618108f10 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f761d183640 10 client.14775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761811dc20 con 0x7f7618108f10 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f761d183640 1 librados: init done 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7600001bd0 con 0x7f7618108f10 2024-09-10T18:52:28.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7600005e60 con 0x7f7618108f10 2024-09-10T18:52:28.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f8005010 con 0x7f7618108f10 2024-09-10T18:52:28.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.303+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7600001730 con 0x7f7618108f10 2024-09-10T18:52:28.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(108..108 src has 49..108) v4 ==== 6284+0+0 (unknown 441160689 0 0) 0x7f760008a210 con 0x7f7618108f10 2024-09-10T18:52:28.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 10 client.14775.objecter ms_dispatch 0x7f7618000b90 osd_map(108..108 src has 49..108) v4 2024-09-10T18:52:28.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 3 client.14775.objecter handle_osd_map got epochs [108,108] > 0 2024-09-10T18:52:28.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 3 client.14775.objecter handle_osd_map decoding full epoch 108 2024-09-10T18:52:28.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 20 client.14775.objecter dump_active .. 0 homeless 2024-09-10T18:52:28.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.307+0000 7f760f7fe640 10 client.14775.objecter ms_handle_connect 0x7f7604071720 2024-09-10T18:52:28.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.311+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7600056e80 con 0x7f7618108f10 2024-09-10T18:52:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[22591]: cluster 2024-09-10T18:52:26.139945+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:52:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[22591]: cluster 2024-09-10T18:52:26.143287+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:52:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[22591]: cluster 2024-09-10T18:52:26.507335+0000 mgr.y (mgr.24526) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[22591]: cluster 2024-09-10T18:52:26.612791+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:52:28.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:27 smithi165 bash[22591]: cluster 2024-09-10T18:52:26.617011+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:52:28.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.483+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75f8000b90 con 0x7f7604071720 2024-09-10T18:52:28.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.495+0000 7f760f7fe640 1 -- 172.21.15.83:0/2982694210 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384093 (unknown 2965378022 0 737862508) 0x7f75f8000b90 con 0x7f7604071720 2024-09-10T18:52:28.506 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:52:28.508 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:52:28.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 10 librados: watch_flush enter 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 10 librados: watch_flush exit 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 20 client.14775.objecter shutdown clearing up homeless session... 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 10 client.14775.objecter successfully canceled tick 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7604071720 legacy=0x7f7604073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 >> v1:172.21.15.83:6789/0 conn(0x7f7618108f10 legacy=0x7f7618104860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 shutdown_connections 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 >> 172.21.15.83:0/2982694210 conn(0x7f76180f70b0 msgr2=0x7f76180f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 shutdown_connections 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 -- 172.21.15.83:0/2982694210 wait complete. 2024-09-10T18:52:28.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:28.503+0000 7f761d183640 1 librados: shutdown 2024-09-10T18:52:29.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[18629]: cluster 2024-09-10T18:52:27.338008+0000 osd.7 (osd.7) 77 : cluster [DBG] 3.d scrub starts 2024-09-10T18:52:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[18629]: cluster 2024-09-10T18:52:27.343250+0000 osd.7 (osd.7) 78 : cluster [DBG] 3.d scrub ok 2024-09-10T18:52:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[18629]: audit 2024-09-10T18:52:27.514093+0000 mgr.y (mgr.24526) 115 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[32381]: cluster 2024-09-10T18:52:27.338008+0000 osd.7 (osd.7) 77 : cluster [DBG] 3.d scrub starts 2024-09-10T18:52:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[32381]: cluster 2024-09-10T18:52:27.343250+0000 osd.7 (osd.7) 78 : cluster [DBG] 3.d scrub ok 2024-09-10T18:52:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:28 smithi083 bash[32381]: audit 2024-09-10T18:52:27.514093+0000 mgr.y (mgr.24526) 115 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:29.247 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-10T18:52:26.506960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1037,"num_read_kb":893,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":610,"ondisk_log_size":610,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386176,"kb_used_data":7320,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575040,"statfs":{"total":767960285184,"available":765516840960,"internally_reserved":0,"allocated":7495680,"data_stored":4459587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002788"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862041+0000","last_change":"2024-09-10T18:50:56.774372+0000","last_active":"2024-09-10T18:52:21.862041+0000","last_peered":"2024-09-10T18:52:21.862041+0000","last_clean":"2024-09-10T18:52:21.862041+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:52:21.862041+0000","last_undegraded":"2024-09-10T18:52:21.862041+0000","last_fullsized":"2024-09-10T18:52:21.862041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:50:56.774029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004548547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862741+0000","last_change":"2024-09-10T18:52:17.464193+0000","last_active":"2024-09-10T18:52:21.862741+0000","last_peered":"2024-09-10T18:52:21.862741+0000","last_clean":"2024-09-10T18:52:21.862741+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:52:21.862741+0000","last_undegraded":"2024-09-10T18:52:21.862741+0000","last_fullsized":"2024-09-10T18:52:21.862741+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:17.463956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.463956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:35.714604+0000","scrub_duration":0.004114698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":69,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861451+0000","last_change":"2024-09-10T18:51:37.694332+0000","last_active":"2024-09-10T18:52:21.861451+0000","last_peered":"2024-09-10T18:52:21.861451+0000","last_clean":"2024-09-10T18:52:21.861451+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:52:21.861451+0000","last_undegraded":"2024-09-10T18:52:21.861451+0000","last_fullsized":"2024-09-10T18:52:21.861451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:37.694143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.694143+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.236407+0000","scrub_duration":0.0048100759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860271+0000","last_change":"2024-09-10T18:52:20.734242+0000","last_active":"2024-09-10T18:52:21.860271+0000","last_peered":"2024-09-10T18:52:21.860271+0000","last_clean":"2024-09-10T18:52:21.860271+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:52:21.860271+0000","last_undegraded":"2024-09-10T18:52:21.860271+0000","last_fullsized":"2024-09-10T18:52:21.860271+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.733951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.733951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.495882+0000","scrub_duration":0.0029697920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862007+0000","last_change":"2024-09-10T18:51:41.627258+0000","last_active":"2024-09-10T18:52:21.862007+0000","last_peered":"2024-09-10T18:52:21.862007+0000","last_clean":"2024-09-10T18:52:21.862007+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:52:21.862007+0000","last_undegraded":"2024-09-10T18:52:21.862007+0000","last_fullsized":"2024-09-10T18:52:21.862007+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.627147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.627147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:48.610764+0000","scrub_duration":0.0032008610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":83,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:23.600483+0000","last_change":"2024-09-10T18:52:23.600334+0000","last_active":"2024-09-10T18:52:23.600483+0000","last_peered":"2024-09-10T18:52:23.600483+0000","last_clean":"2024-09-10T18:52:23.600483+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:52:23.600483+0000","last_undegraded":"2024-09-10T18:52:23.600483+0000","last_fullsized":"2024-09-10T18:52:23.600483+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:23.600141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.600141+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.041119+0000","scrub_duration":0.0045540590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":72,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.179049+0000","last_change":"2024-09-10T18:52:25.178969+0000","last_active":"2024-09-10T18:52:25.179049+0000","last_peered":"2024-09-10T18:52:25.179049+0000","last_clean":"2024-09-10T18:52:25.179049+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:52:25.179049+0000","last_undegraded":"2024-09-10T18:52:25.179049+0000","last_fullsized":"2024-09-10T18:52:25.179049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.178800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.178800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.952086+0000","scrub_duration":0.0034539140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938614+0000","last_change":"2024-09-10T18:50:52.520182+0000","last_active":"2024-09-10T18:52:21.938614+0000","last_peered":"2024-09-10T18:52:21.938614+0000","last_clean":"2024-09-10T18:52:21.938614+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:52:21.938614+0000","last_undegraded":"2024-09-10T18:52:21.938614+0000","last_fullsized":"2024-09-10T18:52:21.938614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:52.520037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:52.520037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031335130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.328300+0000","last_change":"2024-09-10T18:52:16.237573+0000","last_active":"2024-09-10T18:52:22.328300+0000","last_peered":"2024-09-10T18:52:22.328300+0000","last_clean":"2024-09-10T18:52:22.328300+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:52:22.328300+0000","last_undegraded":"2024-09-10T18:52:22.328300+0000","last_fullsized":"2024-09-10T18:52:22.328300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:16.237414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:16.237414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.186738+0000","scrub_duration":0.0033375729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"77'15","reported_seq":77,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171781+0000","last_change":"2024-09-10T18:50:57.455605+0000","last_active":"2024-09-10T18:52:22.171781+0000","last_peered":"2024-09-10T18:52:22.171781+0000","last_clean":"2024-09-10T18:52:22.171781+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:52:22.171781+0000","last_undegraded":"2024-09-10T18:52:22.171781+0000","last_fullsized":"2024-09-10T18:52:22.171781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:50:57.455409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:57.455409+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.020653+0000","scrub_duration":0.0053829450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938929+0000","last_change":"2024-09-10T18:51:03.556626+0000","last_active":"2024-09-10T18:52:21.938929+0000","last_peered":"2024-09-10T18:52:21.938929+0000","last_clean":"2024-09-10T18:52:21.938929+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:52:21.938929+0000","last_undegraded":"2024-09-10T18:52:21.938929+0000","last_fullsized":"2024-09-10T18:52:21.938929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:03.556572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.556572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021821369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":48,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.170946+0000","last_change":"2024-09-10T18:51:06.584417+0000","last_active":"2024-09-10T18:52:22.170946+0000","last_peered":"2024-09-10T18:52:22.170946+0000","last_clean":"2024-09-10T18:52:22.170946+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:52:22.170946+0000","last_undegraded":"2024-09-10T18:52:22.170946+0000","last_fullsized":"2024-09-10T18:52:22.170946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:06.584260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.584260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.730218+0000","scrub_duration":0.0035935630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327478+0000","last_change":"2024-09-10T18:52:22.326533+0000","last_active":"2024-09-10T18:52:22.327478+0000","last_peered":"2024-09-10T18:52:22.327478+0000","last_clean":"2024-09-10T18:52:22.327478+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:52:22.327478+0000","last_undegraded":"2024-09-10T18:52:22.327478+0000","last_fullsized":"2024-09-10T18:52:22.327478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.326315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:34.718300+0000","scrub_duration":0.003554892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":90,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172220+0000","last_change":"2024-09-10T18:51:43.650049+0000","last_active":"2024-09-10T18:52:22.172220+0000","last_peered":"2024-09-10T18:52:22.172220+0000","last_clean":"2024-09-10T18:52:22.172220+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:52:22.172220+0000","last_undegraded":"2024-09-10T18:52:22.172220+0000","last_fullsized":"2024-09-10T18:52:22.172220+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:51:43.649947+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.649947+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.873051+0000","scrub_duration":0.003856703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938971+0000","last_change":"2024-09-10T18:52:01.415959+0000","last_active":"2024-09-10T18:52:21.938971+0000","last_peered":"2024-09-10T18:52:21.938971+0000","last_clean":"2024-09-10T18:52:21.938971+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:52:21.938971+0000","last_undegraded":"2024-09-10T18:52:21.938971+0000","last_fullsized":"2024-09-10T18:52:21.938971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.415718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.415718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:25.902687+0000","scrub_duration":0.0036293950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859158+0000","last_change":"2024-09-10T18:51:36.537160+0000","last_active":"2024-09-10T18:52:21.859158+0000","last_peered":"2024-09-10T18:52:21.859158+0000","last_clean":"2024-09-10T18:52:21.859158+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:52:21.859158+0000","last_undegraded":"2024-09-10T18:52:21.859158+0000","last_fullsized":"2024-09-10T18:52:21.859158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:36.536996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:36.536996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.415469+0000","scrub_duration":0.0033809280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":93,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852437+0000","last_change":"2024-09-10T18:52:02.356154+0000","last_active":"2024-09-10T18:52:21.852437+0000","last_peered":"2024-09-10T18:52:21.852437+0000","last_clean":"2024-09-10T18:52:21.852437+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:52:21.852437+0000","last_undegraded":"2024-09-10T18:52:21.852437+0000","last_fullsized":"2024-09-10T18:52:21.852437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:02.355980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:02.355980+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:21.108556+0000","scrub_duration":0.0055219309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":54,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.328241+0000","last_change":"2024-09-10T18:50:52.474030+0000","last_active":"2024-09-10T18:52:22.328241+0000","last_peered":"2024-09-10T18:52:22.328241+0000","last_clean":"2024-09-10T18:52:22.328241+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:52:22.328241+0000","last_undegraded":"2024-09-10T18:52:22.328241+0000","last_fullsized":"2024-09-10T18:52:22.328241+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:52.473895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:52.473895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003517807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":110,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939816+0000","last_change":"2024-09-10T18:52:17.252670+0000","last_active":"2024-09-10T18:52:21.939816+0000","last_peered":"2024-09-10T18:52:21.939816+0000","last_clean":"2024-09-10T18:52:21.939816+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:52:21.939816+0000","last_undegraded":"2024-09-10T18:52:21.939816+0000","last_fullsized":"2024-09-10T18:52:21.939816+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.252498+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:23.792093+0000","scrub_duration":0.0045932949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188889+0000","last_change":"2024-09-10T18:52:21.401011+0000","last_active":"2024-09-10T18:52:22.188889+0000","last_peered":"2024-09-10T18:52:22.188889+0000","last_clean":"2024-09-10T18:52:22.188889+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:52:22.188889+0000","last_undegraded":"2024-09-10T18:52:22.188889+0000","last_fullsized":"2024-09-10T18:52:22.188889+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.400853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.400853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:30.701974+0000","scrub_duration":0.0034523370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862703+0000","last_change":"2024-09-10T18:52:17.707983+0000","last_active":"2024-09-10T18:52:21.862703+0000","last_peered":"2024-09-10T18:52:21.862703+0000","last_clean":"2024-09-10T18:52:21.862703+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:52:21.862703+0000","last_undegraded":"2024-09-10T18:52:21.862703+0000","last_fullsized":"2024-09-10T18:52:21.862703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:52:17.707813+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.707813+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.139488+0000","scrub_duration":0.0045170389999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696086+0000","last_change":"2024-09-10T18:51:41.492867+0000","last_active":"2024-09-10T18:52:22.696086+0000","last_peered":"2024-09-10T18:52:22.696086+0000","last_clean":"2024-09-10T18:52:22.696086+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:52:22.696086+0000","last_undegraded":"2024-09-10T18:52:22.696086+0000","last_fullsized":"2024-09-10T18:52:22.696086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.492707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:50.150125+0000","scrub_duration":0.0035920370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939026+0000","last_change":"2024-09-10T18:51:46.600626+0000","last_active":"2024-09-10T18:52:21.939026+0000","last_peered":"2024-09-10T18:52:21.939026+0000","last_clean":"2024-09-10T18:52:21.939026+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:52:21.939026+0000","last_undegraded":"2024-09-10T18:52:21.939026+0000","last_fullsized":"2024-09-10T18:52:21.939026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.600460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.600460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.436438+0000","scrub_duration":0.0032588019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939443+0000","last_change":"2024-09-10T18:51:53.429372+0000","last_active":"2024-09-10T18:52:21.939443+0000","last_peered":"2024-09-10T18:52:21.939443+0000","last_clean":"2024-09-10T18:52:21.939443+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:52:21.939443+0000","last_undegraded":"2024-09-10T18:52:21.939443+0000","last_fullsized":"2024-09-10T18:52:21.939443+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:53.429216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.429216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.844395+0000","scrub_duration":0.003713091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":89,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860035+0000","last_change":"2024-09-10T18:51:55.522605+0000","last_active":"2024-09-10T18:52:21.860035+0000","last_peered":"2024-09-10T18:52:21.860035+0000","last_clean":"2024-09-10T18:52:21.860035+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:52:21.860035+0000","last_undegraded":"2024-09-10T18:52:21.860035+0000","last_fullsized":"2024-09-10T18:52:21.860035+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:55.522513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.522513+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.705011+0000","scrub_duration":0.0065065660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":64,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.326169+0000","last_change":"2024-09-10T18:52:25.326041+0000","last_active":"2024-09-10T18:52:25.326169+0000","last_peered":"2024-09-10T18:52:25.326169+0000","last_clean":"2024-09-10T18:52:25.326169+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:52:25.326169+0000","last_undegraded":"2024-09-10T18:52:25.326169+0000","last_fullsized":"2024-09-10T18:52:25.326169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.325803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.325803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.547370+0000","scrub_duration":0.0039072000000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696458+0000","last_change":"2024-09-10T18:51:55.508740+0000","last_active":"2024-09-10T18:52:22.696458+0000","last_peered":"2024-09-10T18:52:22.696458+0000","last_clean":"2024-09-10T18:52:22.696458+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:52:22.696458+0000","last_undegraded":"2024-09-10T18:52:22.696458+0000","last_fullsized":"2024-09-10T18:52:22.696458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:55.508582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.508582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.812679+0000","scrub_duration":0.0034042920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171793+0000","last_change":"2024-09-10T18:51:50.674985+0000","last_active":"2024-09-10T18:52:22.171793+0000","last_peered":"2024-09-10T18:52:22.171793+0000","last_clean":"2024-09-10T18:52:22.171793+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:52:22.171793+0000","last_undegraded":"2024-09-10T18:52:22.171793+0000","last_fullsized":"2024-09-10T18:52:22.171793+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.674821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:16.792421+0000","scrub_duration":0.003696628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939085+0000","last_change":"2024-09-10T18:51:29.827804+0000","last_active":"2024-09-10T18:52:21.939085+0000","last_peered":"2024-09-10T18:52:21.939085+0000","last_clean":"2024-09-10T18:52:21.939085+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:52:21.939085+0000","last_undegraded":"2024-09-10T18:52:21.939085+0000","last_fullsized":"2024-09-10T18:52:21.939085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:51:29.827615+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.827615+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.030826+0000","scrub_duration":0.0051889199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:23.329145+0000","last_change":"2024-09-10T18:52:23.328971+0000","last_active":"2024-09-10T18:52:23.329145+0000","last_peered":"2024-09-10T18:52:23.329145+0000","last_clean":"2024-09-10T18:52:23.329145+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:52:23.329145+0000","last_undegraded":"2024-09-10T18:52:23.329145+0000","last_fullsized":"2024-09-10T18:52:23.329145+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:23.328703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.328703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.447554+0000","scrub_duration":0.0041061600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":110,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939528+0000","last_change":"2024-09-10T18:51:40.694756+0000","last_active":"2024-09-10T18:52:21.939528+0000","last_peered":"2024-09-10T18:52:21.939528+0000","last_clean":"2024-09-10T18:52:21.939528+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:52:21.939528+0000","last_undegraded":"2024-09-10T18:52:21.939528+0000","last_fullsized":"2024-09-10T18:52:21.939528+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:51:40.694630+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.694630+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.465301+0000","scrub_duration":0.0057213519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861287+0000","last_change":"2024-09-10T18:51:22.704230+0000","last_active":"2024-09-10T18:52:21.861287+0000","last_peered":"2024-09-10T18:52:21.861287+0000","last_clean":"2024-09-10T18:52:21.861287+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:52:21.861287+0000","last_undegraded":"2024-09-10T18:52:21.861287+0000","last_fullsized":"2024-09-10T18:52:21.861287+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:51:22.704057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:22.704057+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.304502+0000","scrub_duration":0.0039967839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":72,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862315+0000","last_change":"2024-09-10T18:50:59.610521+0000","last_active":"2024-09-10T18:52:21.862315+0000","last_peered":"2024-09-10T18:52:21.862315+0000","last_clean":"2024-09-10T18:52:21.862315+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:52:21.862315+0000","last_undegraded":"2024-09-10T18:52:21.862315+0000","last_fullsized":"2024-09-10T18:52:21.862315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:50:59.610418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:59.610418+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038889670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.853761+0000","last_change":"2024-09-10T18:51:47.158565+0000","last_active":"2024-09-10T18:52:21.853761+0000","last_peered":"2024-09-10T18:52:21.853761+0000","last_clean":"2024-09-10T18:52:21.853761+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:52:21.853761+0000","last_undegraded":"2024-09-10T18:52:21.853761+0000","last_fullsized":"2024-09-10T18:52:21.853761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:47.158405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.158405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.185469+0000","scrub_duration":0.0034278289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938384+0000","last_change":"2024-09-10T18:51:28.793948+0000","last_active":"2024-09-10T18:52:21.938384+0000","last_peered":"2024-09-10T18:52:21.938384+0000","last_clean":"2024-09-10T18:52:21.938384+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:52:21.938384+0000","last_undegraded":"2024-09-10T18:52:21.938384+0000","last_fullsized":"2024-09-10T18:52:21.938384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:28.793787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.793787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.515669+0000","scrub_duration":0.0035723209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-10T18:52:23.867304+0000","last_change":"2024-09-10T18:52:00.591712+0000","last_active":"2024-09-10T18:52:23.867304+0000","last_peered":"2024-09-10T18:52:23.867304+0000","last_clean":"2024-09-10T18:52:23.867304+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:52:23.867304+0000","last_undegraded":"2024-09-10T18:52:23.867304+0000","last_fullsized":"2024-09-10T18:52:23.867304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:52:00.591632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.591632+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:05.247607+0000","scrub_duration":0.0032914160000000001,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":158,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.326631+0000","last_change":"2024-09-10T18:51:56.155039+0000","last_active":"2024-09-10T18:52:22.326631+0000","last_peered":"2024-09-10T18:52:22.326631+0000","last_clean":"2024-09-10T18:52:22.326631+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:52:22.326631+0000","last_undegraded":"2024-09-10T18:52:22.326631+0000","last_fullsized":"2024-09-10T18:52:22.326631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.154883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.154883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.178313+0000","scrub_duration":0.0033842830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":91,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861505+0000","last_change":"2024-09-10T18:52:09.660716+0000","last_active":"2024-09-10T18:52:21.861505+0000","last_peered":"2024-09-10T18:52:21.861505+0000","last_clean":"2024-09-10T18:52:21.861505+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:52:21.861505+0000","last_undegraded":"2024-09-10T18:52:21.861505+0000","last_fullsized":"2024-09-10T18:52:21.861505+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:52:09.660551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.660551+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.065594+0000","scrub_duration":0.0047491369999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696760+0000","last_change":"2024-09-10T18:52:05.361338+0000","last_active":"2024-09-10T18:52:22.696760+0000","last_peered":"2024-09-10T18:52:22.696760+0000","last_clean":"2024-09-10T18:52:22.696760+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:52:22.696760+0000","last_undegraded":"2024-09-10T18:52:22.696760+0000","last_fullsized":"2024-09-10T18:52:22.696760+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:05.361175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.361175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.175562+0000","scrub_duration":0.0039873549999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":99,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859697+0000","last_change":"2024-09-10T18:51:05.769818+0000","last_active":"2024-09-10T18:52:21.859697+0000","last_peered":"2024-09-10T18:52:21.859697+0000","last_clean":"2024-09-10T18:52:21.859697+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:52:21.859697+0000","last_undegraded":"2024-09-10T18:52:21.859697+0000","last_fullsized":"2024-09-10T18:52:21.859697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:05.769650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:05.769650+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.040600+0000","scrub_duration":0.0040089660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"78'1","reported_seq":68,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938337+0000","last_change":"2024-09-10T18:51:32.830829+0000","last_active":"2024-09-10T18:52:21.938337+0000","last_peered":"2024-09-10T18:52:21.938337+0000","last_clean":"2024-09-10T18:52:21.938337+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:52:21.938337+0000","last_undegraded":"2024-09-10T18:52:21.938337+0000","last_fullsized":"2024-09-10T18:52:21.938337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-09-10T18:51:32.830654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:32.830654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.674413+0000","scrub_duration":0.0041675089999999998,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939168+0000","last_change":"2024-09-10T18:51:38.762020+0000","last_active":"2024-09-10T18:52:21.939168+0000","last_peered":"2024-09-10T18:52:21.939168+0000","last_clean":"2024-09-10T18:52:21.939168+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:52:21.939168+0000","last_undegraded":"2024-09-10T18:52:21.939168+0000","last_fullsized":"2024-09-10T18:52:21.939168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.761859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.761859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:49.670551+0000","scrub_duration":0.003532154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"77'30","reported_seq":149,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696442+0000","last_change":"2024-09-10T18:52:11.309315+0000","last_active":"2024-09-10T18:52:22.696442+0000","last_peered":"2024-09-10T18:52:22.696442+0000","last_clean":"2024-09-10T18:52:22.696442+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:52:22.696442+0000","last_undegraded":"2024-09-10T18:52:22.696442+0000","last_fullsized":"2024-09-10T18:52:22.696442+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-10T18:52:11.309152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.309152+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.549403+0000","scrub_duration":0.0055436330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.328481+0000","last_change":"2024-09-10T18:52:03.183815+0000","last_active":"2024-09-10T18:52:22.328481+0000","last_peered":"2024-09-10T18:52:22.328481+0000","last_clean":"2024-09-10T18:52:22.328481+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:52:22.328481+0000","last_undegraded":"2024-09-10T18:52:22.328481+0000","last_fullsized":"2024-09-10T18:52:22.328481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.183646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.183646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.233272+0000","scrub_duration":0.0032583899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":110,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860692+0000","last_change":"2024-09-10T18:52:07.501745+0000","last_active":"2024-09-10T18:52:21.860692+0000","last_peered":"2024-09-10T18:52:21.860692+0000","last_clean":"2024-09-10T18:52:21.860692+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:52:21.860692+0000","last_undegraded":"2024-09-10T18:52:21.860692+0000","last_fullsized":"2024-09-10T18:52:21.860692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.501543+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.400864+0000","scrub_duration":0.0064925740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":68,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852303+0000","last_change":"2024-09-10T18:52:15.552956+0000","last_active":"2024-09-10T18:52:21.852303+0000","last_peered":"2024-09-10T18:52:21.852303+0000","last_clean":"2024-09-10T18:52:21.852303+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:52:21.852303+0000","last_undegraded":"2024-09-10T18:52:21.852303+0000","last_fullsized":"2024-09-10T18:52:21.852303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.552791+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:33.020117+0000","scrub_duration":0.0058227280000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":48,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.695072+0000","last_change":"2024-09-10T18:51:11.652632+0000","last_active":"2024-09-10T18:52:22.695072+0000","last_peered":"2024-09-10T18:52:22.695072+0000","last_clean":"2024-09-10T18:52:22.695072+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:52:22.695072+0000","last_undegraded":"2024-09-10T18:52:22.695072+0000","last_fullsized":"2024-09-10T18:52:22.695072+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:11.652501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:11.652501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:39.556031+0000","scrub_duration":0.0028739529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":84,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172351+0000","last_change":"2024-09-10T18:51:58.828562+0000","last_active":"2024-09-10T18:52:22.172351+0000","last_peered":"2024-09-10T18:52:22.172351+0000","last_clean":"2024-09-10T18:52:22.172351+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:52:22.172351+0000","last_undegraded":"2024-09-10T18:52:22.172351+0000","last_fullsized":"2024-09-10T18:52:22.172351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:58.828403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.828403+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.385475+0000","scrub_duration":0.0052576080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696456+0000","last_change":"2024-09-10T18:52:18.315632+0000","last_active":"2024-09-10T18:52:22.696456+0000","last_peered":"2024-09-10T18:52:22.696456+0000","last_clean":"2024-09-10T18:52:22.696456+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:52:22.696456+0000","last_undegraded":"2024-09-10T18:52:22.696456+0000","last_fullsized":"2024-09-10T18:52:22.696456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:18.315469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.315469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:29.767066+0000","scrub_duration":0.0035610590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.695533+0000","last_change":"2024-09-10T18:51:05.604974+0000","last_active":"2024-09-10T18:52:22.695533+0000","last_peered":"2024-09-10T18:52:22.695533+0000","last_clean":"2024-09-10T18:52:22.695533+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:52:22.695533+0000","last_undegraded":"2024-09-10T18:52:22.695533+0000","last_fullsized":"2024-09-10T18:52:22.695533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:05.604804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:05.604804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:31.081415+0000","scrub_duration":0.0034418669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859533+0000","last_change":"2024-09-10T18:51:15.485056+0000","last_active":"2024-09-10T18:52:21.859533+0000","last_peered":"2024-09-10T18:52:21.859533+0000","last_clean":"2024-09-10T18:52:21.859533+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:52:21.859533+0000","last_undegraded":"2024-09-10T18:52:21.859533+0000","last_fullsized":"2024-09-10T18:52:21.859533+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:15.484881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:15.484881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.398529+0000","scrub_duration":0.0034503160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":100,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.940053+0000","last_change":"2024-09-10T18:52:15.209771+0000","last_active":"2024-09-10T18:52:21.940053+0000","last_peered":"2024-09-10T18:52:21.940053+0000","last_clean":"2024-09-10T18:52:21.940053+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:52:21.940053+0000","last_undegraded":"2024-09-10T18:52:21.940053+0000","last_fullsized":"2024-09-10T18:52:21.940053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:15.209609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.209609+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.019264+0000","scrub_duration":0.005193746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171574+0000","last_change":"2024-09-10T18:52:09.838167+0000","last_active":"2024-09-10T18:52:22.171574+0000","last_peered":"2024-09-10T18:52:22.171574+0000","last_clean":"2024-09-10T18:52:22.171574+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:52:22.171574+0000","last_undegraded":"2024-09-10T18:52:22.171574+0000","last_fullsized":"2024-09-10T18:52:22.171574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.838015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.838015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.447360+0000","scrub_duration":0.003415202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696002+0000","last_change":"2024-09-10T18:52:09.323881+0000","last_active":"2024-09-10T18:52:22.696002+0000","last_peered":"2024-09-10T18:52:22.696002+0000","last_clean":"2024-09-10T18:52:22.696002+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:52:22.696002+0000","last_undegraded":"2024-09-10T18:52:22.696002+0000","last_fullsized":"2024-09-10T18:52:22.696002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.323704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.323704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.895803+0000","scrub_duration":0.0036260730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860812+0000","last_change":"2024-09-10T18:51:39.639863+0000","last_active":"2024-09-10T18:52:21.860812+0000","last_peered":"2024-09-10T18:52:21.860812+0000","last_clean":"2024-09-10T18:52:21.860812+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:52:21.860812+0000","last_undegraded":"2024-09-10T18:52:21.860812+0000","last_fullsized":"2024-09-10T18:52:21.860812+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.639709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.639709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:44.720874+0000","scrub_duration":0.0036283190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":102,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696240+0000","last_change":"2024-09-10T18:51:58.430973+0000","last_active":"2024-09-10T18:52:22.696240+0000","last_peered":"2024-09-10T18:52:22.696240+0000","last_clean":"2024-09-10T18:52:22.696240+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:52:22.696240+0000","last_undegraded":"2024-09-10T18:52:22.696240+0000","last_fullsized":"2024-09-10T18:52:22.696240+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.430868+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.088894+0000","scrub_duration":0.013718902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188137+0000","last_change":"2024-09-10T18:51:40.776447+0000","last_active":"2024-09-10T18:52:22.188137+0000","last_peered":"2024-09-10T18:52:22.188137+0000","last_clean":"2024-09-10T18:52:22.188137+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:52:22.188137+0000","last_undegraded":"2024-09-10T18:52:22.188137+0000","last_fullsized":"2024-09-10T18:52:22.188137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:40.776169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.776169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.421742+0000","scrub_duration":0.0038302309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696548+0000","last_change":"2024-09-10T18:52:19.322071+0000","last_active":"2024-09-10T18:52:22.696548+0000","last_peered":"2024-09-10T18:52:22.696548+0000","last_clean":"2024-09-10T18:52:22.696548+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:52:22.696548+0000","last_undegraded":"2024-09-10T18:52:22.696548+0000","last_fullsized":"2024-09-10T18:52:22.696548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:19.321916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:19.321916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.895150+0000","scrub_duration":0.0033042869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327331+0000","last_change":"2024-09-10T18:51:45.221499+0000","last_active":"2024-09-10T18:52:22.327331+0000","last_peered":"2024-09-10T18:52:22.327331+0000","last_clean":"2024-09-10T18:52:22.327331+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:52:22.327331+0000","last_undegraded":"2024-09-10T18:52:22.327331+0000","last_fullsized":"2024-09-10T18:52:22.327331+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:45.221333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:45.221333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.678342+0000","scrub_duration":0.0036649180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":85,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.863084+0000","last_change":"2024-09-10T18:51:51.619698+0000","last_active":"2024-09-10T18:52:21.863084+0000","last_peered":"2024-09-10T18:52:21.863084+0000","last_clean":"2024-09-10T18:52:21.863084+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:52:21.863084+0000","last_undegraded":"2024-09-10T18:52:21.863084+0000","last_fullsized":"2024-09-10T18:52:21.863084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:51.619511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:51.619511+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:03.627716+0000","scrub_duration":0.0049634149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.325926+0000","last_change":"2024-09-10T18:52:06.149853+0000","last_active":"2024-09-10T18:52:22.325926+0000","last_peered":"2024-09-10T18:52:22.325926+0000","last_clean":"2024-09-10T18:52:22.325926+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:52:22.325926+0000","last_undegraded":"2024-09-10T18:52:22.325926+0000","last_fullsized":"2024-09-10T18:52:22.325926+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.149699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.149699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.887082+0000","scrub_duration":0.0035198529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862490+0000","last_change":"2024-09-10T18:51:50.626660+0000","last_active":"2024-09-10T18:52:21.862490+0000","last_peered":"2024-09-10T18:52:21.862490+0000","last_clean":"2024-09-10T18:52:21.862490+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:52:21.862490+0000","last_undegraded":"2024-09-10T18:52:21.862490+0000","last_fullsized":"2024-09-10T18:52:21.862490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.626514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.626514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.083006+0000","scrub_duration":0.0035366019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.695582+0000","last_change":"2024-09-10T18:52:04.358732+0000","last_active":"2024-09-10T18:52:22.695582+0000","last_peered":"2024-09-10T18:52:22.695582+0000","last_clean":"2024-09-10T18:52:22.695582+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:52:22.695582+0000","last_undegraded":"2024-09-10T18:52:22.695582+0000","last_fullsized":"2024-09-10T18:52:22.695582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:04.358415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:04.358415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:06.405553+0000","scrub_duration":0.003743377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"77'10","reported_seq":69,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861074+0000","last_change":"2024-09-10T18:51:33.617562+0000","last_active":"2024-09-10T18:52:21.861074+0000","last_peered":"2024-09-10T18:52:21.861074+0000","last_clean":"2024-09-10T18:52:21.861074+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:52:21.861074+0000","last_undegraded":"2024-09-10T18:52:21.861074+0000","last_fullsized":"2024-09-10T18:52:21.861074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:33.617399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.617399+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.865090+0000","scrub_duration":0.0054712390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171685+0000","last_change":"2024-09-10T18:51:39.623163+0000","last_active":"2024-09-10T18:52:22.171685+0000","last_peered":"2024-09-10T18:52:22.171685+0000","last_clean":"2024-09-10T18:52:22.171685+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:52:22.171685+0000","last_undegraded":"2024-09-10T18:52:22.171685+0000","last_fullsized":"2024-09-10T18:52:22.171685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.623005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.623005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.432652+0000","scrub_duration":0.0034195559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":112,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.326171+0000","last_change":"2024-09-10T18:51:33.452673+0000","last_active":"2024-09-10T18:52:22.326171+0000","last_peered":"2024-09-10T18:52:22.326171+0000","last_clean":"2024-09-10T18:52:22.326171+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:52:22.326171+0000","last_undegraded":"2024-09-10T18:52:22.326171+0000","last_fullsized":"2024-09-10T18:52:22.326171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:51:33.452504+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.452504+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.436046+0000","scrub_duration":0.0041999869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.853346+0000","last_change":"2024-09-10T18:52:03.329811+0000","last_active":"2024-09-10T18:52:21.853346+0000","last_peered":"2024-09-10T18:52:21.853346+0000","last_clean":"2024-09-10T18:52:21.853346+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:52:21.853346+0000","last_undegraded":"2024-09-10T18:52:21.853346+0000","last_fullsized":"2024-09-10T18:52:21.853346+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.329632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.640377+0000","scrub_duration":0.0032347410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172308+0000","last_change":"2024-09-10T18:52:21.732959+0000","last_active":"2024-09-10T18:52:22.172308+0000","last_peered":"2024-09-10T18:52:22.172308+0000","last_clean":"2024-09-10T18:52:22.172308+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:52:22.172308+0000","last_undegraded":"2024-09-10T18:52:22.172308+0000","last_fullsized":"2024-09-10T18:52:22.172308+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.732812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.732812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.740200+0000","scrub_duration":0.0034512259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.328089+0000","last_change":"2024-09-10T18:50:49.409493+0000","last_active":"2024-09-10T18:52:22.328089+0000","last_peered":"2024-09-10T18:52:22.328089+0000","last_clean":"2024-09-10T18:52:22.328089+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:52:22.328089+0000","last_undegraded":"2024-09-10T18:52:22.328089+0000","last_fullsized":"2024-09-10T18:52:22.328089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:50:49.409334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:50:49.409334+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042892800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188779+0000","last_change":"2024-09-10T18:52:07.455227+0000","last_active":"2024-09-10T18:52:22.188779+0000","last_peered":"2024-09-10T18:52:22.188779+0000","last_clean":"2024-09-10T18:52:22.188779+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:52:22.188779+0000","last_undegraded":"2024-09-10T18:52:22.188779+0000","last_fullsized":"2024-09-10T18:52:22.188779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.455062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.455062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.587408+0000","scrub_duration":0.0037580970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852564+0000","last_change":"2024-09-10T18:51:52.321821+0000","last_active":"2024-09-10T18:52:21.852564+0000","last_peered":"2024-09-10T18:52:21.852564+0000","last_clean":"2024-09-10T18:52:21.852564+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:52:21.852564+0000","last_undegraded":"2024-09-10T18:52:21.852564+0000","last_fullsized":"2024-09-10T18:52:21.852564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:52.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:52.321671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:02.059416+0000","scrub_duration":0.003316977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":48,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.326060+0000","last_change":"2024-09-10T18:50:51.430175+0000","last_active":"2024-09-10T18:52:22.326060+0000","last_peered":"2024-09-10T18:52:22.326060+0000","last_clean":"2024-09-10T18:52:22.326060+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:52:22.326060+0000","last_undegraded":"2024-09-10T18:52:22.326060+0000","last_fullsized":"2024-09-10T18:52:22.326060+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:51.430075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:51.430075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043979570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"77'10","reported_seq":67,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172042+0000","last_change":"2024-09-10T18:51:03.530423+0000","last_active":"2024-09-10T18:52:22.172042+0000","last_peered":"2024-09-10T18:52:22.172042+0000","last_clean":"2024-09-10T18:52:22.172042+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:52:22.172042+0000","last_undegraded":"2024-09-10T18:52:22.172042+0000","last_fullsized":"2024-09-10T18:52:22.172042+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:03.530326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.530326+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.247623+0000","scrub_duration":0.0034148749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":54,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696677+0000","last_change":"2024-09-10T18:50:50.740960+0000","last_active":"2024-09-10T18:52:22.696677+0000","last_peered":"2024-09-10T18:52:22.696677+0000","last_clean":"2024-09-10T18:52:22.696677+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:52:22.696677+0000","last_undegraded":"2024-09-10T18:52:22.696677+0000","last_fullsized":"2024-09-10T18:52:22.696677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:50.740799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:50:50.740799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030619250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860452+0000","last_change":"2024-09-10T18:50:58.647711+0000","last_active":"2024-09-10T18:52:21.860452+0000","last_peered":"2024-09-10T18:52:21.860452+0000","last_clean":"2024-09-10T18:52:21.860452+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:52:21.860452+0000","last_undegraded":"2024-09-10T18:52:21.860452+0000","last_fullsized":"2024-09-10T18:52:21.860452+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:58.647547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:50:58.647547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031702420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859370+0000","last_change":"2024-09-10T18:51:17.509138+0000","last_active":"2024-09-10T18:52:21.859370+0000","last_peered":"2024-09-10T18:52:21.859370+0000","last_clean":"2024-09-10T18:52:21.859370+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:52:21.859370+0000","last_undegraded":"2024-09-10T18:52:21.859370+0000","last_fullsized":"2024-09-10T18:52:21.859370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:17.508961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:17.508961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036482350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":94,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327701+0000","last_change":"2024-09-10T18:51:47.304256+0000","last_active":"2024-09-10T18:52:22.327701+0000","last_peered":"2024-09-10T18:52:22.327701+0000","last_clean":"2024-09-10T18:52:22.327701+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:52:22.327701+0000","last_undegraded":"2024-09-10T18:52:22.327701+0000","last_fullsized":"2024-09-10T18:52:22.327701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.304153+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.010387+0000","scrub_duration":0.012604194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":76,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188775+0000","last_change":"2024-09-10T18:51:53.688495+0000","last_active":"2024-09-10T18:52:22.188775+0000","last_peered":"2024-09-10T18:52:22.188775+0000","last_clean":"2024-09-10T18:52:22.188775+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:52:22.188775+0000","last_undegraded":"2024-09-10T18:52:22.188775+0000","last_fullsized":"2024-09-10T18:52:22.188775+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:51:53.688302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.688302+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.064936+0000","scrub_duration":0.0040928329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.170655+0000","last_change":"2024-09-10T18:51:46.633977+0000","last_active":"2024-09-10T18:52:22.170655+0000","last_peered":"2024-09-10T18:52:22.170655+0000","last_clean":"2024-09-10T18:52:22.170655+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:52:22.170655+0000","last_undegraded":"2024-09-10T18:52:22.170655+0000","last_fullsized":"2024-09-10T18:52:22.170655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.633831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.754683+0000","scrub_duration":0.003224747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171129+0000","last_change":"2024-09-10T18:51:56.751545+0000","last_active":"2024-09-10T18:52:22.171129+0000","last_peered":"2024-09-10T18:52:22.171129+0000","last_clean":"2024-09-10T18:52:22.171129+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:52:22.171129+0000","last_undegraded":"2024-09-10T18:52:22.171129+0000","last_fullsized":"2024-09-10T18:52:22.171129+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.751398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.751398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.229956+0000","scrub_duration":0.0032004590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327954+0000","last_change":"2024-09-10T18:52:05.179430+0000","last_active":"2024-09-10T18:52:22.327954+0000","last_peered":"2024-09-10T18:52:22.327954+0000","last_clean":"2024-09-10T18:52:22.327954+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:52:22.327954+0000","last_undegraded":"2024-09-10T18:52:22.327954+0000","last_fullsized":"2024-09-10T18:52:22.327954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:05.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.179207+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.065476+0000","scrub_duration":0.0057240640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":105,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.940322+0000","last_change":"2024-09-10T18:51:43.700454+0000","last_active":"2024-09-10T18:52:21.940322+0000","last_peered":"2024-09-10T18:52:21.940322+0000","last_clean":"2024-09-10T18:52:21.940322+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:52:21.940322+0000","last_undegraded":"2024-09-10T18:52:21.940322+0000","last_fullsized":"2024-09-10T18:52:21.940322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:51:43.700290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.700290+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.352626+0000","scrub_duration":0.0033939529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":109,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.938760+0000","last_change":"2024-09-10T18:51:47.570557+0000","last_active":"2024-09-10T18:52:21.938760+0000","last_peered":"2024-09-10T18:52:21.938760+0000","last_clean":"2024-09-10T18:52:21.938760+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:52:21.938760+0000","last_undegraded":"2024-09-10T18:52:21.938760+0000","last_fullsized":"2024-09-10T18:52:21.938760+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:47.570369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.570369+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.658418+0000","scrub_duration":0.0040413760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862240+0000","last_change":"2024-09-10T18:51:23.683348+0000","last_active":"2024-09-10T18:52:21.862240+0000","last_peered":"2024-09-10T18:52:21.862240+0000","last_clean":"2024-09-10T18:52:21.862240+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:52:21.862240+0000","last_undegraded":"2024-09-10T18:52:21.862240+0000","last_fullsized":"2024-09-10T18:52:21.862240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:23.683176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:23.683176+0000","last_clean_scrub_stamp":"2024-09-10T18:51:23.683176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:26.223369+0000","scrub_duration":0.0032660779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":81,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.326090+0000","last_change":"2024-09-10T18:51:29.406090+0000","last_active":"2024-09-10T18:52:22.326090+0000","last_peered":"2024-09-10T18:52:22.326090+0000","last_clean":"2024-09-10T18:52:22.326090+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:52:22.326090+0000","last_undegraded":"2024-09-10T18:52:22.326090+0000","last_fullsized":"2024-09-10T18:52:22.326090+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:29.405903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.405903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:35.496037+0000","scrub_duration":0.0052412719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188196+0000","last_change":"2024-09-10T18:51:59.561431+0000","last_active":"2024-09-10T18:52:22.188196+0000","last_peered":"2024-09-10T18:52:22.188196+0000","last_clean":"2024-09-10T18:52:22.188196+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:52:22.188196+0000","last_undegraded":"2024-09-10T18:52:22.188196+0000","last_fullsized":"2024-09-10T18:52:22.188196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_clean_scrub_stamp":"2024-09-10T18:51:59.561306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.793374+0000","scrub_duration":0.003474277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862112+0000","last_change":"2024-09-10T18:52:01.485093+0000","last_active":"2024-09-10T18:52:21.862112+0000","last_peered":"2024-09-10T18:52:21.862112+0000","last_clean":"2024-09-10T18:52:21.862112+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:52:21.862112+0000","last_undegraded":"2024-09-10T18:52:21.862112+0000","last_fullsized":"2024-09-10T18:52:21.862112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.484734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.484734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.349048+0000","scrub_duration":0.0054940589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188215+0000","last_change":"2024-09-10T18:52:11.451951+0000","last_active":"2024-09-10T18:52:22.188215+0000","last_peered":"2024-09-10T18:52:22.188215+0000","last_clean":"2024-09-10T18:52:22.188215+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:52:22.188215+0000","last_undegraded":"2024-09-10T18:52:22.188215+0000","last_fullsized":"2024-09-10T18:52:22.188215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.451797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.451797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:26.258751+0000","scrub_duration":0.0033137470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"77'4","reported_seq":68,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852067+0000","last_change":"2024-09-10T18:52:11.496842+0000","last_active":"2024-09-10T18:52:21.852067+0000","last_peered":"2024-09-10T18:52:21.852067+0000","last_clean":"2024-09-10T18:52:21.852067+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:52:21.852067+0000","last_undegraded":"2024-09-10T18:52:21.852067+0000","last_fullsized":"2024-09-10T18:52:21.852067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-10T18:52:11.496670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.496670+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:24.908807+0000","scrub_duration":0.0041783929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852561+0000","last_change":"2024-09-10T18:52:00.390577+0000","last_active":"2024-09-10T18:52:21.852561+0000","last_peered":"2024-09-10T18:52:21.852561+0000","last_clean":"2024-09-10T18:52:21.852561+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:52:21.852561+0000","last_undegraded":"2024-09-10T18:52:21.852561+0000","last_fullsized":"2024-09-10T18:52:21.852561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:00.390415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.390415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.045571+0000","scrub_duration":0.00325318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861310+0000","last_change":"2024-09-10T18:51:14.705208+0000","last_active":"2024-09-10T18:52:21.861310+0000","last_peered":"2024-09-10T18:52:21.861310+0000","last_clean":"2024-09-10T18:52:21.861310+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:52:21.861310+0000","last_undegraded":"2024-09-10T18:52:21.861310+0000","last_fullsized":"2024-09-10T18:52:21.861310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:14.705032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:14.705032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:30.732429+0000","scrub_duration":0.0038292510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.326912+0000","last_change":"2024-09-10T18:52:07.197975+0000","last_active":"2024-09-10T18:52:22.326912+0000","last_peered":"2024-09-10T18:52:22.326912+0000","last_clean":"2024-09-10T18:52:22.326912+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:52:22.326912+0000","last_undegraded":"2024-09-10T18:52:22.326912+0000","last_fullsized":"2024-09-10T18:52:22.326912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.197730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.197730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.782447+0000","scrub_duration":0.003470014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"77'11","reported_seq":71,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327034+0000","last_change":"2024-09-10T18:51:00.361402+0000","last_active":"2024-09-10T18:52:22.327034+0000","last_peered":"2024-09-10T18:52:22.327034+0000","last_clean":"2024-09-10T18:52:22.327034+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:52:22.327034+0000","last_undegraded":"2024-09-10T18:52:22.327034+0000","last_fullsized":"2024-09-10T18:52:22.327034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:00.361230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:00.361230+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.900016+0000","scrub_duration":0.0048395340000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":54,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.171379+0000","last_change":"2024-09-10T18:51:13.565917+0000","last_active":"2024-09-10T18:52:22.171379+0000","last_peered":"2024-09-10T18:52:22.171379+0000","last_clean":"2024-09-10T18:52:22.171379+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:52:22.171379+0000","last_undegraded":"2024-09-10T18:52:22.171379+0000","last_fullsized":"2024-09-10T18:52:22.171379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:13.565686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:13.565686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.569343+0000","scrub_duration":0.0037071460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":112,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-10T18:52:24.727095+0000","last_change":"2024-09-10T18:52:24.727019+0000","last_active":"2024-09-10T18:52:24.727095+0000","last_peered":"2024-09-10T18:52:24.727095+0000","last_clean":"2024-09-10T18:52:24.727095+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:52:24.727095+0000","last_undegraded":"2024-09-10T18:52:24.727095+0000","last_fullsized":"2024-09-10T18:52:24.727095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:24.726861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.726861+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.604127+0000","scrub_duration":0.0039722289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861811+0000","last_change":"2024-09-10T18:51:20.403072+0000","last_active":"2024-09-10T18:52:21.861811+0000","last_peered":"2024-09-10T18:52:21.861811+0000","last_clean":"2024-09-10T18:52:21.861811+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:52:21.861811+0000","last_undegraded":"2024-09-10T18:52:21.861811+0000","last_fullsized":"2024-09-10T18:52:21.861811+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:51:20.402922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:24.916001+0000","scrub_duration":0.0031476540000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860286+0000","last_change":"2024-09-10T18:51:10.452572+0000","last_active":"2024-09-10T18:52:21.860286+0000","last_peered":"2024-09-10T18:52:21.860286+0000","last_clean":"2024-09-10T18:52:21.860286+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:52:21.860286+0000","last_undegraded":"2024-09-10T18:52:21.860286+0000","last_fullsized":"2024-09-10T18:52:21.860286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:10.452379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:10.452379+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.851342+0000","scrub_duration":0.0051699889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.863060+0000","last_change":"2024-09-10T18:51:56.575310+0000","last_active":"2024-09-10T18:52:21.863060+0000","last_peered":"2024-09-10T18:52:21.863060+0000","last_clean":"2024-09-10T18:52:21.863060+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:52:21.863060+0000","last_undegraded":"2024-09-10T18:52:21.863060+0000","last_fullsized":"2024-09-10T18:52:21.863060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.575153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.564777+0000","scrub_duration":0.0033985280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":97,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172101+0000","last_change":"2024-09-10T18:51:21.667022+0000","last_active":"2024-09-10T18:52:22.172101+0000","last_peered":"2024-09-10T18:52:22.172101+0000","last_clean":"2024-09-10T18:52:22.172101+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:52:22.172101+0000","last_undegraded":"2024-09-10T18:52:22.172101+0000","last_fullsized":"2024-09-10T18:52:22.172101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:21.666847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:21.666847+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:48.835363+0000","scrub_duration":0.0041978249999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.189077+0000","last_change":"2024-09-10T18:52:22.188985+0000","last_active":"2024-09-10T18:52:22.189077+0000","last_peered":"2024-09-10T18:52:22.189077+0000","last_clean":"2024-09-10T18:52:22.189077+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:52:22.189077+0000","last_undegraded":"2024-09-10T18:52:22.189077+0000","last_fullsized":"2024-09-10T18:52:22.189077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.188834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.188834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.440531+0000","scrub_duration":0.0068943040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327719+0000","last_change":"2024-09-10T18:52:08.245784+0000","last_active":"2024-09-10T18:52:22.327719+0000","last_peered":"2024-09-10T18:52:22.327719+0000","last_clean":"2024-09-10T18:52:22.327719+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:52:22.327719+0000","last_undegraded":"2024-09-10T18:52:22.327719+0000","last_fullsized":"2024-09-10T18:52:22.327719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:08.245617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:08.245617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.667622+0000","scrub_duration":0.0044282050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":54,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.853306+0000","last_change":"2024-09-10T18:51:03.040657+0000","last_active":"2024-09-10T18:52:21.853306+0000","last_peered":"2024-09-10T18:52:21.853306+0000","last_clean":"2024-09-10T18:52:21.853306+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:52:21.853306+0000","last_undegraded":"2024-09-10T18:52:21.853306+0000","last_fullsized":"2024-09-10T18:52:21.853306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:03.040495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:03.040495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035544589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.170995+0000","last_change":"2024-09-10T18:52:11.856507+0000","last_active":"2024-09-10T18:52:22.170995+0000","last_peered":"2024-09-10T18:52:22.170995+0000","last_clean":"2024-09-10T18:52:22.170995+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:52:22.170995+0000","last_undegraded":"2024-09-10T18:52:22.170995+0000","last_fullsized":"2024-09-10T18:52:22.170995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.856337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.856337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.241036+0000","scrub_duration":0.0034053619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859188+0000","last_change":"2024-09-10T18:52:14.538993+0000","last_active":"2024-09-10T18:52:21.859188+0000","last_peered":"2024-09-10T18:52:21.859188+0000","last_clean":"2024-09-10T18:52:21.859188+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:52:21.859188+0000","last_undegraded":"2024-09-10T18:52:21.859188+0000","last_fullsized":"2024-09-10T18:52:21.859188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:14.538838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:14.538838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.876083+0000","scrub_duration":0.0042549700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"77'10","reported_seq":79,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939502+0000","last_change":"2024-09-10T18:52:11.222775+0000","last_active":"2024-09-10T18:52:21.939502+0000","last_peered":"2024-09-10T18:52:21.939502+0000","last_clean":"2024-09-10T18:52:21.939502+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:52:21.939502+0000","last_undegraded":"2024-09-10T18:52:21.939502+0000","last_fullsized":"2024-09-10T18:52:21.939502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:11.222598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.222598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.658086+0000","scrub_duration":0.0051546400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696467+0000","last_change":"2024-09-10T18:51:35.492628+0000","last_active":"2024-09-10T18:52:22.696467+0000","last_peered":"2024-09-10T18:52:22.696467+0000","last_clean":"2024-09-10T18:52:22.696467+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:52:22.696467+0000","last_undegraded":"2024-09-10T18:52:22.696467+0000","last_fullsized":"2024-09-10T18:52:22.696467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:35.492463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:35.492463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:35.871414+0000","scrub_duration":0.0032631600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":62,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939632+0000","last_change":"2024-09-10T18:52:20.152534+0000","last_active":"2024-09-10T18:52:21.939632+0000","last_peered":"2024-09-10T18:52:21.939632+0000","last_clean":"2024-09-10T18:52:21.939632+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:52:21.939632+0000","last_undegraded":"2024-09-10T18:52:21.939632+0000","last_fullsized":"2024-09-10T18:52:21.939632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.152369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:47.436481+0000","scrub_duration":0.0033349780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859545+0000","last_change":"2024-09-10T18:51:57.533423+0000","last_active":"2024-09-10T18:52:21.859545+0000","last_peered":"2024-09-10T18:52:21.859545+0000","last_clean":"2024-09-10T18:52:21.859545+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:52:21.859545+0000","last_undegraded":"2024-09-10T18:52:21.859545+0000","last_fullsized":"2024-09-10T18:52:21.859545+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:57.533281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.533281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.018032+0000","scrub_duration":0.0030396329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"77'6","reported_seq":63,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862570+0000","last_change":"2024-09-10T18:51:21.383448+0000","last_active":"2024-09-10T18:52:21.862570+0000","last_peered":"2024-09-10T18:52:21.862570+0000","last_clean":"2024-09-10T18:52:21.862570+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:52:21.862570+0000","last_undegraded":"2024-09-10T18:52:21.862570+0000","last_fullsized":"2024-09-10T18:52:21.862570+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'6","last_scrub_stamp":"2024-09-10T18:51:21.383324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:21.383324+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.795348+0000","scrub_duration":0.003470833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":54,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.172639+0000","last_change":"2024-09-10T18:51:07.542429+0000","last_active":"2024-09-10T18:52:22.172639+0000","last_peered":"2024-09-10T18:52:22.172639+0000","last_clean":"2024-09-10T18:52:22.172639+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:52:22.172639+0000","last_undegraded":"2024-09-10T18:52:22.172639+0000","last_fullsized":"2024-09-10T18:52:22.172639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:07.542270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:07.542270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:29.429613+0000","scrub_duration":0.0032886,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859590+0000","last_change":"2024-09-10T18:51:38.650934+0000","last_active":"2024-09-10T18:52:21.859590+0000","last_peered":"2024-09-10T18:52:21.859590+0000","last_clean":"2024-09-10T18:52:21.859590+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:52:21.859590+0000","last_undegraded":"2024-09-10T18:52:21.859590+0000","last_fullsized":"2024-09-10T18:52:21.859590+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.650775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.247838+0000","scrub_duration":0.0033420289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":58,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.325878+0000","last_change":"2024-09-10T18:51:37.345540+0000","last_active":"2024-09-10T18:52:22.325878+0000","last_peered":"2024-09-10T18:52:22.325878+0000","last_clean":"2024-09-10T18:52:22.325878+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:52:22.325878+0000","last_undegraded":"2024-09-10T18:52:22.325878+0000","last_fullsized":"2024-09-10T18:52:22.325878+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:37.345379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.345379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.645490+0000","scrub_duration":0.0032740450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":78,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.698515+0000","last_change":"2024-09-10T18:52:22.698451+0000","last_active":"2024-09-10T18:52:22.698515+0000","last_peered":"2024-09-10T18:52:22.698515+0000","last_clean":"2024-09-10T18:52:22.698515+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:52:22.698515+0000","last_undegraded":"2024-09-10T18:52:22.698515+0000","last_fullsized":"2024-09-10T18:52:22.698515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:22.698371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.698371+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:46.745548+0000","scrub_duration":0.0094695119999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.853744+0000","last_change":"2024-09-10T18:51:49.224075+0000","last_active":"2024-09-10T18:52:21.853744+0000","last_peered":"2024-09-10T18:52:21.853744+0000","last_clean":"2024-09-10T18:52:21.853744+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:52:21.853744+0000","last_undegraded":"2024-09-10T18:52:21.853744+0000","last_fullsized":"2024-09-10T18:52:21.853744+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:49.223905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:49.223905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.495696+0000","scrub_duration":0.003346421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859829+0000","last_change":"2024-09-10T18:51:06.774645+0000","last_active":"2024-09-10T18:52:21.859829+0000","last_peered":"2024-09-10T18:52:21.859829+0000","last_clean":"2024-09-10T18:52:21.859829+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:52:21.859829+0000","last_undegraded":"2024-09-10T18:52:21.859829+0000","last_fullsized":"2024-09-10T18:52:21.859829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:06.774496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.774496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.898966+0000","scrub_duration":0.0031373199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":50,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862530+0000","last_change":"2024-09-10T18:50:57.823090+0000","last_active":"2024-09-10T18:52:21.862530+0000","last_peered":"2024-09-10T18:52:21.862530+0000","last_clean":"2024-09-10T18:52:21.862530+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:52:21.862530+0000","last_undegraded":"2024-09-10T18:52:21.862530+0000","last_fullsized":"2024-09-10T18:52:21.862530+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:50:57.822942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:50:57.822942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036355409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"77'1","reported_seq":51,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.327067+0000","last_change":"2024-09-10T18:51:06.380763+0000","last_active":"2024-09-10T18:52:22.327067+0000","last_peered":"2024-09-10T18:52:22.327067+0000","last_clean":"2024-09-10T18:52:22.327067+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:52:22.327067+0000","last_undegraded":"2024-09-10T18:52:22.327067+0000","last_fullsized":"2024-09-10T18:52:22.327067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:51:06.380593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:06.380593+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.734018+0000","scrub_duration":0.0040786110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":87,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.695315+0000","last_change":"2024-09-10T18:51:50.544356+0000","last_active":"2024-09-10T18:52:22.695315+0000","last_peered":"2024-09-10T18:52:22.695315+0000","last_clean":"2024-09-10T18:52:22.695315+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:52:22.695315+0000","last_undegraded":"2024-09-10T18:52:22.695315+0000","last_fullsized":"2024-09-10T18:52:22.695315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:50.544186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.544186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.788233+0000","scrub_duration":0.0064898330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":64,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.170893+0000","last_change":"2024-09-10T18:52:15.859394+0000","last_active":"2024-09-10T18:52:22.170893+0000","last_peered":"2024-09-10T18:52:22.170893+0000","last_clean":"2024-09-10T18:52:22.170893+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:52:22.170893+0000","last_undegraded":"2024-09-10T18:52:22.170893+0000","last_fullsized":"2024-09-10T18:52:22.170893+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:15.859239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.859239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.490895+0000","scrub_duration":0.0037198740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":62,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-10T18:52:24.615205+0000","last_change":"2024-09-10T18:52:24.615123+0000","last_active":"2024-09-10T18:52:24.615205+0000","last_peered":"2024-09-10T18:52:24.615205+0000","last_clean":"2024-09-10T18:52:24.615205+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:52:24.615205+0000","last_undegraded":"2024-09-10T18:52:24.615205+0000","last_fullsized":"2024-09-10T18:52:24.615205+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:24.614967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.614967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:40.011991+0000","scrub_duration":0.0037677969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":48,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.696103+0000","last_change":"2024-09-10T18:51:04.594129+0000","last_active":"2024-09-10T18:52:22.696103+0000","last_peered":"2024-09-10T18:52:22.696103+0000","last_clean":"2024-09-10T18:52:22.696103+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:52:22.696103+0000","last_undegraded":"2024-09-10T18:52:22.696103+0000","last_fullsized":"2024-09-10T18:52:22.696103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:04.593968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:04.593968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:27.665764+0000","scrub_duration":0.003534648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.859906+0000","last_change":"2024-09-10T18:51:28.596878+0000","last_active":"2024-09-10T18:52:21.859906+0000","last_peered":"2024-09-10T18:52:21.859906+0000","last_clean":"2024-09-10T18:52:21.859906+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:52:21.859906+0000","last_undegraded":"2024-09-10T18:52:21.859906+0000","last_fullsized":"2024-09-10T18:52:21.859906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:28.596672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.596672+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:50.156773+0000","scrub_duration":0.0055701980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":56,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860417+0000","last_change":"2024-09-10T18:51:02.778350+0000","last_active":"2024-09-10T18:52:21.860417+0000","last_peered":"2024-09-10T18:52:21.860417+0000","last_clean":"2024-09-10T18:52:21.860417+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:52:21.860417+0000","last_undegraded":"2024-09-10T18:52:21.860417+0000","last_fullsized":"2024-09-10T18:52:21.860417+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:02.778242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:02.778242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003123444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":124,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.853637+0000","last_change":"2024-09-10T18:51:55.299569+0000","last_active":"2024-09-10T18:52:21.853637+0000","last_peered":"2024-09-10T18:52:21.853637+0000","last_clean":"2024-09-10T18:52:21.853637+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:52:21.853637+0000","last_undegraded":"2024-09-10T18:52:21.853637+0000","last_fullsized":"2024-09-10T18:52:21.853637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:51:55.299396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.299396+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.997434+0000","scrub_duration":0.0038567179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939303+0000","last_change":"2024-09-10T18:51:41.706112+0000","last_active":"2024-09-10T18:52:21.939303+0000","last_peered":"2024-09-10T18:52:21.939303+0000","last_clean":"2024-09-10T18:52:21.939303+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:52:21.939303+0000","last_undegraded":"2024-09-10T18:52:21.939303+0000","last_fullsized":"2024-09-10T18:52:21.939303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.705954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.045292+0000","scrub_duration":0.003398917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":73,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.852788+0000","last_change":"2024-09-10T18:52:18.566245+0000","last_active":"2024-09-10T18:52:21.852788+0000","last_peered":"2024-09-10T18:52:21.852788+0000","last_clean":"2024-09-10T18:52:21.852788+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:52:21.852788+0000","last_undegraded":"2024-09-10T18:52:21.852788+0000","last_fullsized":"2024-09-10T18:52:21.852788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:52:18.566056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.566056+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.529822+0000","scrub_duration":0.0040203510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":74,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860991+0000","last_change":"2024-09-10T18:52:09.509564+0000","last_active":"2024-09-10T18:52:21.860991+0000","last_peered":"2024-09-10T18:52:21.860991+0000","last_clean":"2024-09-10T18:52:21.860991+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:52:21.860991+0000","last_undegraded":"2024-09-10T18:52:21.860991+0000","last_fullsized":"2024-09-10T18:52:21.860991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:52:09.509384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.509384+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.310603+0000","scrub_duration":0.0050191510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:22.188575+0000","last_change":"2024-09-10T18:51:56.652829+0000","last_active":"2024-09-10T18:52:22.188575+0000","last_peered":"2024-09-10T18:52:22.188575+0000","last_clean":"2024-09-10T18:52:22.188575+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:52:22.188575+0000","last_undegraded":"2024-09-10T18:52:22.188575+0000","last_fullsized":"2024-09-10T18:52:22.188575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.652676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.654243+0000","scrub_duration":0.003518115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":60,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.861096+0000","last_change":"2024-09-10T18:52:06.609575+0000","last_active":"2024-09-10T18:52:21.861096+0000","last_peered":"2024-09-10T18:52:21.861096+0000","last_clean":"2024-09-10T18:52:21.861096+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:52:21.861096+0000","last_undegraded":"2024-09-10T18:52:21.861096+0000","last_fullsized":"2024-09-10T18:52:21.861096+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.609346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.609346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.679962+0000","scrub_duration":0.0031752439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":66,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.862975+0000","last_change":"2024-09-10T18:51:42.590260+0000","last_active":"2024-09-10T18:52:21.862975+0000","last_peered":"2024-09-10T18:52:21.862975+0000","last_clean":"2024-09-10T18:52:21.862975+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:52:21.862975+0000","last_undegraded":"2024-09-10T18:52:21.862975+0000","last_fullsized":"2024-09-10T18:52:21.862975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:42.590071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:42.590071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.690989+0000","scrub_duration":0.003624726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":80,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.939413+0000","last_change":"2024-09-10T18:51:57.420671+0000","last_active":"2024-09-10T18:52:21.939413+0000","last_peered":"2024-09-10T18:52:21.939413+0000","last_clean":"2024-09-10T18:52:21.939413+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:52:21.939413+0000","last_undegraded":"2024-09-10T18:52:21.939413+0000","last_fullsized":"2024-09-10T18:52:21.939413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:57.420489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.420489+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.697937+0000","scrub_duration":0.0042932839999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":52,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-10T18:52:21.860813+0000","last_change":"2024-09-10T18:51:25.626798+0000","last_active":"2024-09-10T18:52:21.860813+0000","last_peered":"2024-09-10T18:52:21.860813+0000","last_clean":"2024-09-10T18:52:21.860813+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:52:21.860813+0000","last_undegraded":"2024-09-10T18:52:21.860813+0000","last_fullsized":"2024-09-10T18:52:21.860813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:25.626688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:25.626688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:25.833450+0000","scrub_duration":0.0026021970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":163,"num_read_kb":158,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":422,"ondisk_log_size":422,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972401,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298640,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446512,"statfs":{"total":95995035648,"available":95689228288,"internally_reserved":0,"allocated":1314816,"data_stored":927304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91500000000000004,"15min":0.93500000000000005},"min":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3129999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91900000000000004,"15min":0.93100000000000005},"min":{"1min":0.61399999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2669999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.69399999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.89700000000000002,"15min":0.90500000000000003},"min":{"1min":0.64800000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0309999999999999,"15min":1.0529999999999999},"min":{"1min":0.68500000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0369999999999999,"15min":1.0640000000000001},"min":{"1min":0.64300000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.97499999999999998,"15min":0.97699999999999998},"min":{"1min":0.72399999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3340000000000001,"5min":1.353,"15min":1.353},"last":1.028}]},{"osd":3,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.998,"15min":1.0209999999999999},"min":{"1min":0.66200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.266,"5min":1.335,"15min":1.335},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.96399999999999997,"15min":0.95999999999999996},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.73599999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.004,"15min":1.0269999999999999},"min":{"1min":0.58099999999999996,"5min":0.443,"15min":0.443},"max":{"1min":2.1989999999999998,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.083,"5min":1.0269999999999999,"15min":1.0209999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99199999999999999,"15min":0.996},"min":{"1min":0.60499999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1659999999999999,"15min":1.196},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.109}]},{"osd":6,"last update":"Tue Sep 10 18:51:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.2350000000000001,"15min":1.256},"min":{"1min":0.746,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.2429999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1850000000000001,"15min":1.1930000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0070000000000001,"5min":2.536,"15min":2.536},"last":1.0860000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299451,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298620,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446532,"statfs":{"total":95995035648,"available":95689248768,"internally_reserved":0,"allocated":1294336,"data_stored":925543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.165,"5min":1.272,"15min":1.272},"last":1.3600000000000001},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3300000000000001,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.5109999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.91400000000000003,"15min":0.88400000000000001},"min":{"1min":0.626,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2350000000000001,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.446},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.91600000000000004,"15min":0.91300000000000003},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.2529999999999999,"5min":1.361,"15min":1.361},"last":1.1120000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.93500000000000005,"15min":0.91600000000000004},"min":{"1min":0.80700000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.2809999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.337},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.92200000000000004,"15min":0.92800000000000005},"min":{"1min":0.64600000000000002,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":1.4710000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":0.96699999999999997,"15min":0.94599999999999995},"min":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.4370000000000001,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.95899999999999996,"15min":0.96899999999999997},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.2430000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.2749999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.88,"15min":0.85599999999999998},"min":{"1min":0.57599999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.246},{"interface":"front","average":{"1min":1.075,"5min":1.006,"15min":1.006},"min":{"1min":0.68100000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.4379999999999999,"5min":1.744,"15min":1.744},"last":0.72899999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":0.98199999999999998,"15min":0.96399999999999997},"min":{"1min":0.53500000000000003,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.0780000000000001,"15min":1.0840000000000001},"min":{"1min":0.64800000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.5009999999999999,"5min":1.665,"15min":1.665},"last":0.83199999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:51:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0529999999999999,"15min":1.0389999999999999},"min":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.516,"5min":1.573,"15min":1.573},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.141,"15min":1.137},"min":{"1min":0.68500000000000005,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.4279999999999999,"5min":1.488,"15min":1.488},"last":1.405}]}]},{"osd":5,"up_from":43,"seq":184683593797,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298044,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447108,"statfs":{"total":95995035648,"available":95689838592,"internally_reserved":0,"allocated":704512,"data_stored":335254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.88800000000000001,"15min":0.83999999999999997},"min":{"1min":0.66200000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.258,"5min":1.609,"15min":1.609},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1990000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.234}]},{"osd":1,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.86899999999999999,"15min":0.85499999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2949999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.91700000000000004,"15min":0.88100000000000001},"min":{"1min":0.70699999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2230000000000001,"5min":1.635,"15min":1.635},"last":1.2709999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91500000000000004,"15min":0.88500000000000001},"min":{"1min":0.628,"5min":0.496,"15min":0.496},"max":{"1min":1.274,"5min":1.496,"15min":1.496},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.999,"5min":0.93899999999999995,"15min":0.89400000000000002},"min":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3180000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.755}]},{"osd":3,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.64900000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.242,"5min":1.409,"15min":1.409},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.69199999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2090000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.085}]},{"osd":4,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83099999999999996,"15min":0.77000000000000002},"min":{"1min":0.65100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1499999999999999,"5min":1.77,"15min":1.77},"last":1.1819999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92400000000000004,"15min":0.85299999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3580000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.99199999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:44 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0229999999999999,"15min":1.0089999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.2070000000000001,"5min":1.462,"15min":1.462},"last":1.294},{"interface":"front","average":{"1min":1.04,"5min":1.0580000000000001,"15min":1.032},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.361,"5min":1.653,"15min":1.653},"last":1.155}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.046,"15min":1.0680000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.294,"5min":1.837,"15min":1.837},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.1379999999999999,"15min":1.1759999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.296,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.377}]}]},{"osd":4,"up_from":35,"seq":150323855441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298084,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447068,"statfs":{"total":95995035648,"available":95689797632,"internally_reserved":0,"allocated":741376,"data_stored":336819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.877,"15min":0.85099999999999998},"min":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.171,"5min":1.204,"15min":1.204},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91300000000000003,"15min":0.82599999999999996},"min":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56000000000000005},"max":{"1min":1.1870000000000001,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.343}]},{"osd":1,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.90400000000000003,"15min":0.88800000000000001},"min":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2869999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92600000000000005,"15min":0.84499999999999997},"min":{"1min":0.624,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.3480000000000001,"5min":1.456,"15min":1.456},"last":1.2070000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.93799999999999994,"15min":0.872},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3160000000000001,"5min":1.425,"15min":1.425},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.94699999999999995,"15min":0.872},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.59699999999999998},"max":{"1min":1.3180000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95699999999999996,"15min":0.88900000000000001},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.56399999999999995},"max":{"1min":1.327,"5min":1.714,"15min":1.714},"last":1.079},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97099999999999997,"15min":0.91700000000000004},"min":{"1min":0.79800000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.339,"5min":1.391,"15min":1.391},"last":1.2829999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.90300000000000002,"15min":0.83499999999999996},"min":{"1min":0.63200000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3939999999999999,"5min":1.484,"15min":1.484},"last":0.872},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97299999999999998,"15min":0.88100000000000001},"min":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.405,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.2410000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.96999999999999997,"15min":0.92400000000000004},"min":{"1min":0.77200000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0880000000000001,"15min":1.054},"min":{"1min":0.90200000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.5009999999999999,"5min":1.536,"15min":1.536},"last":1.371}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.73699999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.395},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1579999999999999,"15min":1.1719999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.476,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.4430000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051676,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298044,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447108,"statfs":{"total":95995035648,"available":95689838592,"internally_reserved":0,"allocated":704512,"data_stored":336416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.52500000000000002,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.01,"15min":1.2529999999999999},"last":0.95199999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.014,"5min":1.032,"15min":1.2869999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82599999999999996,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.48899999999999999,"15min":0.36699999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":1.274},"last":0.72799999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.78100000000000003},"min":{"1min":0.50600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.77700000000000002,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1739999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.81200000000000006}]},{"osd":4,"last update":"Tue Sep 10 18:52:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.079,"15min":0.99099999999999999},"min":{"1min":0.83299999999999996,"5min":0.70099999999999996,"15min":0.56999999999999995},"max":{"1min":1.373,"5min":1.778,"15min":1.778},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1579999999999999,"15min":1.069},"min":{"1min":0.86699999999999999,"5min":0.75700000000000001,"15min":0.59399999999999997},"max":{"1min":1.4239999999999999,"5min":1.952,"15min":1.952},"last":0.98199999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.113,"15min":1.0960000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5469999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.1020000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.97899999999999998},"min":{"1min":0.72999999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4630000000000001,"5min":1.532,"15min":1.532},"last":1.595}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.173,"15min":1.163},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.641,"5min":1.643,"15min":1.643},"last":1.161},{"interface":"front","average":{"1min":1.052,"5min":1.079,"15min":1.0669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.3620000000000001,"5min":1.476,"15min":1.476},"last":1.3180000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.113,"15min":1.1140000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":1.23},{"interface":"front","average":{"1min":1.232,"5min":1.242,"15min":1.2430000000000001},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.5940000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.387}]}]},{"osd":2,"up_from":19,"seq":81604378725,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298056,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447096,"statfs":{"total":95995035648,"available":95689826304,"internally_reserved":0,"allocated":716800,"data_stored":336613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.69699999999999995,"15min":0.624},"min":{"1min":0.50800000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84799999999999998,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.78400000000000003,"15min":0.70899999999999996},"min":{"1min":0.49099999999999999,"5min":0.41399999999999998,"15min":0.30599999999999999},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":1.014}]},{"osd":1,"last update":"Tue Sep 10 18:51:21 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.70699999999999996,"15min":0.66500000000000004},"min":{"1min":0.46999999999999997,"5min":0.313,"15min":0.313},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80900000000000005,"15min":0.72299999999999998},"min":{"1min":0.47599999999999998,"5min":0.45900000000000002,"15min":0.379},"max":{"1min":1.0980000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.054}]},{"osd":3,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88200000000000001,"15min":0.81299999999999994},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4510000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.84499999999999997,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.72199999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.022,"15min":0.95599999999999996},"min":{"1min":0.745,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.425,"5min":1.542,"15min":1.542},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.107,"5min":1.115,"15min":1.0680000000000001},"min":{"1min":0.73299999999999998,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.627,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.1479999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0940000000000001,"15min":1.111},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.71,"5min":1.766,"15min":1.766},"last":1.294},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.004},"min":{"1min":0.72999999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5700000000000001,"5min":1.643,"15min":1.643},"last":1.119}]},{"osd":6,"last update":"Tue Sep 10 18:51:45 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.165,"15min":1.1819999999999999},"min":{"1min":0.76900000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":1.381},{"interface":"front","average":{"1min":1.079,"5min":1.0780000000000001,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.542,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.1870000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:51:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2010000000000001,"15min":1.2070000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6459999999999999,"5min":1.671,"15min":1.671},"last":1.337},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1200000000000001,"15min":1.1160000000000001},"min":{"1min":0.81699999999999995,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.633,"5min":1.708,"15min":1.708},"last":1.24}]}]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":712704,"data_stored":335843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.59199999999999997},"min":{"1min":0.49099999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.77500000000000002,"15min":0.65900000000000003},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.35299999999999998},"max":{"1min":1.857,"5min":2.181,"15min":2.181},"last":0.61399999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73699999999999999,"15min":0.65800000000000003},"min":{"1min":0.54200000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.83299999999999996,"15min":0.73599999999999999},"min":{"1min":0.60299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.73499999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.877,"15min":0.81899999999999995},"min":{"1min":0.629,"5min":0.435,"15min":0.435},"max":{"1min":1.2210000000000001,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77100000000000002,"15min":0.73399999999999999},"min":{"1min":0.59899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.65200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.036,"15min":1},"min":{"1min":0.71899999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6830000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97699999999999998,"15min":0.88900000000000001},"min":{"1min":0.78400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.208}]},{"osd":5,"last update":"Tue Sep 10 18:51:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.995,"15min":1.012},"min":{"1min":0.77500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.115,"15min":1.131},"min":{"1min":0.86099999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.571,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.357}]},{"osd":6,"last update":"Tue Sep 10 18:51:50 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1100000000000001,"15min":1.0760000000000001},"min":{"1min":0.82299999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.635,"5min":1.976,"15min":1.976},"last":1.46},{"interface":"front","average":{"1min":1.103,"5min":1.075,"15min":1.077},"min":{"1min":0.70599999999999996,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":1.2669999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:51:41 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.202,"15min":1.22},"min":{"1min":0.78300000000000003,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.569,"5min":2.125,"15min":2.125},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.159,"5min":1.143,"15min":1.147},"min":{"1min":0.751,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.71,"5min":1.71,"15min":1.71},"last":1.425}]}]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298636,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446516,"statfs":{"total":95995035648,"available":95689232384,"internally_reserved":0,"allocated":1306624,"data_stored":925795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:51:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.64200000000000002,"15min":0.56699999999999995},"min":{"1min":0.45000000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.92300000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.628},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77000000000000002,"15min":0.63600000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.67800000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72699999999999998,"15min":0.67400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.873},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.69099999999999995},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0249999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":1.863},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.77800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.48699999999999999},"max":{"1min":1.633,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.91200000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:52:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":2.4079999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.9750000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:51:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0600000000000001,"15min":1.004},"min":{"1min":0.73599999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5189999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.091,"15min":1.081},"min":{"1min":0.68999999999999995,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6000000000000001,"5min":1.712,"15min":1.712},"last":1.7549999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1180000000000001,"15min":1.1000000000000001},"min":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.48,"5min":1.893,"15min":1.893},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.101,"5min":1.135,"15min":1.105},"min":{"1min":0.82599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.4219999999999999,"5min":2.363,"15min":2.363},"last":3.544}]},{"osd":7,"last update":"Tue Sep 10 18:51:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.1539999999999999,"15min":1.1639999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.4810000000000001,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":2.5390000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.2649999999999999,"15min":1.3080000000000001},"min":{"1min":0.71099999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4359999999999999,"5min":1.8380000000000001,"15min":1.8380000000000001},"last":1.9039999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:52:29.253 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T18:52:29.254 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T18:52:29.254 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-10T18:52:29.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-10T18:52:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:28 smithi165 bash[22591]: cluster 2024-09-10T18:52:27.338008+0000 osd.7 (osd.7) 77 : cluster [DBG] 3.d scrub starts 2024-09-10T18:52:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:28 smithi165 bash[22591]: cluster 2024-09-10T18:52:27.343250+0000 osd.7 (osd.7) 78 : cluster [DBG] 3.d scrub ok 2024-09-10T18:52:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:28 smithi165 bash[22591]: audit 2024-09-10T18:52:27.514093+0000 mgr.y (mgr.24526) 115 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.112399+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1b deep-scrub starts 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.116069+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1b deep-scrub ok 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: audit 2024-09-10T18:52:28.488980+0000 mgr.y (mgr.24526) 116 : audit [DBG] from='client.14775 v1:172.21.15.83:0/2982694210' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.507772+0000 mgr.y (mgr.24526) 117 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.662361+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.673165+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.730060+0000 osd.3 (osd.3) 81 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:28.733314+0000 osd.3 (osd.3) 82 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:52:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.288419+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.d scrub starts 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.291927+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.d scrub ok 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[32381]: audit 2024-09-10T18:52:29.550304+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.112399+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1b deep-scrub starts 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.116069+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1b deep-scrub ok 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: audit 2024-09-10T18:52:28.488980+0000 mgr.y (mgr.24526) 116 : audit [DBG] from='client.14775 v1:172.21.15.83:0/2982694210' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.507772+0000 mgr.y (mgr.24526) 117 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.662361+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.673165+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.730060+0000 osd.3 (osd.3) 81 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:52:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:28.733314+0000 osd.3 (osd.3) 82 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:52:30.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.288419+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.d scrub starts 2024-09-10T18:52:30.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.291927+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.d scrub ok 2024-09-10T18:52:30.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:29 smithi083 bash[18629]: audit 2024-09-10T18:52:29.550304+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.112399+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.1b deep-scrub starts 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.116069+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.1b deep-scrub ok 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: audit 2024-09-10T18:52:28.488980+0000 mgr.y (mgr.24526) 116 : audit [DBG] from='client.14775 v1:172.21.15.83:0/2982694210' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.507772+0000 mgr.y (mgr.24526) 117 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.662361+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.673165+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.730060+0000 osd.3 (osd.3) 81 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:28.733314+0000 osd.3 (osd.3) 82 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.288419+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.d scrub starts 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.291927+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.d scrub ok 2024-09-10T18:52:30.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:29 smithi165 bash[22591]: audit 2024-09-10T18:52:29.550304+0000 mon.c (mon.2) 78 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:31.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.588167+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.b scrub starts 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.591518+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub ok 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.665698+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub starts 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[32381]: cluster 2024-09-10T18:52:29.670545+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.c scrub ok 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.588167+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.b scrub starts 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.591518+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub ok 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.665698+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub starts 2024-09-10T18:52:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:30 smithi083 bash[18629]: cluster 2024-09-10T18:52:29.670545+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.c scrub ok 2024-09-10T18:52:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:30 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.588167+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.b scrub starts 2024-09-10T18:52:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:30 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.591518+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.b scrub ok 2024-09-10T18:52:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:30 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.665698+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub starts 2024-09-10T18:52:31.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:30 smithi165 bash[22591]: cluster 2024-09-10T18:52:29.670545+0000 osd.5 (osd.5) 72 : cluster [DBG] 3.c scrub ok 2024-09-10T18:52:31.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:32.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[18629]: cluster 2024-09-10T18:52:30.509135+0000 mgr.y (mgr.24526) 118 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.346734+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:52:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.350526+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:52:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[32381]: cluster 2024-09-10T18:52:30.509135+0000 mgr.y (mgr.24526) 118 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.346734+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:52:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:31 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.350526+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:52:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:31 smithi165 bash[22591]: cluster 2024-09-10T18:52:30.509135+0000 mgr.y (mgr.24526) 118 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:31 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.346734+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:52:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:31 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.350526+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:52:32.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:32] "GET /metrics HTTP/1.1" 200 220150 "" "Prometheus/2.43.0" 2024-09-10T18:52:32.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.614258+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:52:32.941 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.619821+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:52:33.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.773051+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:52:33.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[18629]: cluster 2024-09-10T18:52:31.777327+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:52:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.614258+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:52:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.619821+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:52:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.773051+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:52:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:32 smithi083 bash[32381]: cluster 2024-09-10T18:52:31.777327+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:52:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:32 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.614258+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:52:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:32 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.619821+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:52:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:32 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.773051+0000 osd.3 (osd.3) 83 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:52:33.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:32 smithi165 bash[22591]: cluster 2024-09-10T18:52:31.777327+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:52:34.027 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:52:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[18629]: cluster 2024-09-10T18:52:32.509697+0000 mgr.y (mgr.24526) 119 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[18629]: cluster 2024-09-10T18:52:32.643519+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:52:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[18629]: cluster 2024-09-10T18:52:32.646972+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:52:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[18629]: cluster 2024-09-10T18:52:32.745109+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:52:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[18629]: cluster 2024-09-10T18:52:32.749543+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:52:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[32381]: cluster 2024-09-10T18:52:32.509697+0000 mgr.y (mgr.24526) 119 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[32381]: cluster 2024-09-10T18:52:32.643519+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:52:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[32381]: cluster 2024-09-10T18:52:32.646972+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:52:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[32381]: cluster 2024-09-10T18:52:32.745109+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:52:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:33 smithi083 bash[32381]: cluster 2024-09-10T18:52:32.749543+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:52:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:33 smithi165 bash[22591]: cluster 2024-09-10T18:52:32.509697+0000 mgr.y (mgr.24526) 119 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:33 smithi165 bash[22591]: cluster 2024-09-10T18:52:32.643519+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:52:34.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:33 smithi165 bash[22591]: cluster 2024-09-10T18:52:32.646972+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:52:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:33 smithi165 bash[22591]: cluster 2024-09-10T18:52:32.745109+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:52:34.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:33 smithi165 bash[22591]: cluster 2024-09-10T18:52:32.749543+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:52:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:33.560658+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:33.564274+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:33.672472+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:33.676295+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:34.324546+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[32381]: cluster 2024-09-10T18:52:34.328275+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:33.560658+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:33.564274+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:33.672472+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:33.676295+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:34.324546+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:52:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:34 smithi083 bash[18629]: cluster 2024-09-10T18:52:34.328275+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:33.560658+0000 osd.0 (osd.0) 83 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:33.564274+0000 osd.0 (osd.0) 84 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:33.672472+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:33.676295+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:34.324546+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:52:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:34 smithi165 bash[22591]: cluster 2024-09-10T18:52:34.328275+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:52:36.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:35 smithi083 bash[18629]: cluster 2024-09-10T18:52:34.510361+0000 mgr.y (mgr.24526) 120 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:35 smithi083 bash[32381]: cluster 2024-09-10T18:52:34.510361+0000 mgr.y (mgr.24526) 120 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:35 smithi165 bash[22591]: cluster 2024-09-10T18:52:34.510361+0000 mgr.y (mgr.24526) 120 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.195+0000 7fe891bcd640 1 -- 172.21.15.83:0/951603428 >> v1:172.21.15.83:6789/0 conn(0x7fe88c10b9c0 legacy=0x7fe88c10dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.195+0000 7fe891bcd640 1 -- 172.21.15.83:0/951603428 shutdown_connections 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.195+0000 7fe891bcd640 1 -- 172.21.15.83:0/951603428 >> 172.21.15.83:0/951603428 conn(0x7fe88c0723f0 msgr2=0x7fe88c074800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- 172.21.15.83:0/951603428 shutdown_connections 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- 172.21.15.83:0/951603428 wait complete. 2024-09-10T18:52:37.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 librados: starting msgr at 2024-09-10T18:52:37.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 librados: starting objecter 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 Processor -- start 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- start start 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 librados: setting wanted keys 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 librados: calling monclient init 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe88c122540 con 0x7fe88c10f520 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe88c123740 con 0x7fe88c10b9c0 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe88c124940 con 0x7fe88c108be0 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe88bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe88c10f520 0x7fe88c120c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54122/0 (socket says 172.21.15.83:54122) 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe88bfff640 1 -- 172.21.15.83:0/33050217 learned_addr learned my addr 172.21.15.83:0/33050217 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter ms_handle_connect 0x7fe88c10f520 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter ms_handle_connect 0x7fe88c108be0 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter ms_handle_connect 0x7fe88c10b9c0 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111201927 0 0) 0x7fe88c122540 con 0x7fe88c10f520 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8780054b0 con 0x7fe88c10f520 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898867388 0 0) 0x7fe88c123740 con 0x7fe88c10b9c0 2024-09-10T18:52:37.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88c122540 con 0x7fe88c10b9c0 2024-09-10T18:52:37.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885988495 0 0) 0x7fe8780054b0 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe88c123740 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe87c001950 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952953521 0 0) 0x7fe88c124940 con 0x7fe88c108be0 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8780054b0 con 0x7fe88c108be0 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2033948954 0 0) 0x7fe88c122540 con 0x7fe88c10b9c0 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe88c124940 con 0x7fe88c10b9c0 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe8740018c0 con 0x7fe88c10b9c0 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 412342002 0 0) 0x7fe88c123740 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 >> v1:172.21.15.83:6790/0 conn(0x7fe88c108be0 legacy=0x7fe88c104c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 >> v1:172.21.15.165:6789/0 conn(0x7fe88c10b9c0 legacy=0x7fe88c105360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c125b40 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c124b30 con 0x7fe88c10f520 2024-09-10T18:52:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 10 client.14781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c125000 con 0x7fe88c10f520 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe891bcd640 1 librados: init done 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe87c001ce0 con 0x7fe88c10f520 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.199+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe87c005c40 con 0x7fe88c10f520 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86c005010 con 0x7fe88c10f520 2024-09-10T18:52:37.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe87c002270 con 0x7fe88c10f520 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(108..108 src has 49..108) v4 ==== 6284+0+0 (unknown 441160689 0 0) 0x7fe87c089f10 con 0x7fe88c10f520 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 10 client.14781.objecter ms_dispatch 0x7fe88c000b90 osd_map(108..108 src has 49..108) v4 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 3 client.14781.objecter handle_osd_map got epochs [108,108] > 0 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 3 client.14781.objecter handle_osd_map decoding full epoch 108 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 20 client.14781.objecter dump_active .. 0 homeless 2024-09-10T18:52:37.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.203+0000 7fe883fff640 10 client.14781.objecter ms_handle_connect 0x7fe8780716b0 2024-09-10T18:52:37.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.211+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe87c056b80 con 0x7fe88c10f520 2024-09-10T18:52:37.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:36 smithi083 bash[18629]: cluster 2024-09-10T18:52:35.713416+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:52:37.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:36 smithi083 bash[18629]: cluster 2024-09-10T18:52:35.715768+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:52:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:36 smithi083 bash[32381]: cluster 2024-09-10T18:52:35.713416+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:52:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:36 smithi083 bash[32381]: cluster 2024-09-10T18:52:35.715768+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:52:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:36 smithi165 bash[22591]: cluster 2024-09-10T18:52:35.713416+0000 osd.3 (osd.3) 87 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:52:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:36 smithi165 bash[22591]: cluster 2024-09-10T18:52:35.715768+0000 osd.3 (osd.3) 88 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:52:37.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.423+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe86c005840 con 0x7fe88c10f520 2024-09-10T18:52:37.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.423+0000 7fe883fff640 1 -- 172.21.15.83:0/33050217 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fe87c056520 con 0x7fe88c10f520 2024-09-10T18:52:37.428 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:52:37.428 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T18:52:37.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 10 librados: watch_flush enter 2024-09-10T18:52:37.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 10 librados: watch_flush exit 2024-09-10T18:52:37.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 20 client.14781.objecter shutdown clearing up homeless session... 2024-09-10T18:52:37.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 10 client.14781.objecter successfully canceled tick 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe8780716b0 legacy=0x7fe878073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 >> v1:172.21.15.83:6789/0 conn(0x7fe88c10f520 legacy=0x7fe88c120c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 shutdown_connections 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 >> 172.21.15.83:0/33050217 conn(0x7fe88c0723f0 msgr2=0x7fe88c1119b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 shutdown_connections 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 -- 172.21.15.83:0/33050217 wait complete. 2024-09-10T18:52:37.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:37.427+0000 7fe891bcd640 1 librados: shutdown 2024-09-10T18:52:37.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:38.180 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-10T18:52:38.180 INFO:tasks.cephadm:Setup complete, yielding 2024-09-10T18:52:38.181 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-10T18:52:38.192 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T18:52:38.193 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-quincy-2024-09-09-1810-quincy'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}}, 'workunit': {'branch': 'quincy-orch-centos-9-teuth', 'sha1': 'fad5356a94e6044ce1062d748daa5a0f2ed53a7d'}} 2024-09-10T18:52:38.193 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T18:52:38.193 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-10T18:52:38.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[32381]: cluster 2024-09-10T18:52:36.511807+0000 mgr.y (mgr.24526) 121 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[32381]: cluster 2024-09-10T18:52:36.572347+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[32381]: cluster 2024-09-10T18:52:36.576854+0000 osd.0 (osd.0) 86 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[32381]: audit 2024-09-10T18:52:37.428146+0000 mon.a (mon.0) 862 : audit [DBG] from='client.? v1:172.21.15.83:0/33050217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[18629]: cluster 2024-09-10T18:52:36.511807+0000 mgr.y (mgr.24526) 121 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[18629]: cluster 2024-09-10T18:52:36.572347+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[18629]: cluster 2024-09-10T18:52:36.576854+0000 osd.0 (osd.0) 86 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:52:38.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:37 smithi083 bash[18629]: audit 2024-09-10T18:52:37.428146+0000 mon.a (mon.0) 862 : audit [DBG] from='client.? v1:172.21.15.83:0/33050217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:52:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:37 smithi165 bash[22591]: cluster 2024-09-10T18:52:36.511807+0000 mgr.y (mgr.24526) 121 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-09-10T18:52:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:37 smithi165 bash[22591]: cluster 2024-09-10T18:52:36.572347+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:52:38.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:37 smithi165 bash[22591]: cluster 2024-09-10T18:52:36.576854+0000 osd.0 (osd.0) 86 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:52:38.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:37 smithi165 bash[22591]: audit 2024-09-10T18:52:37.428146+0000 mon.a (mon.0) 862 : audit [DBG] from='client.? v1:172.21.15.83:0/33050217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T18:52:39.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[18629]: audit 2024-09-10T18:52:37.525556+0000 mgr.y (mgr.24526) 122 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[18629]: cluster 2024-09-10T18:52:37.621660+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[18629]: cluster 2024-09-10T18:52:37.625934+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[18629]: cluster 2024-09-10T18:52:38.370412+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[18629]: cluster 2024-09-10T18:52:38.372964+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[32381]: audit 2024-09-10T18:52:37.525556+0000 mgr.y (mgr.24526) 122 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[32381]: cluster 2024-09-10T18:52:37.621660+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[32381]: cluster 2024-09-10T18:52:37.625934+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:52:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[32381]: cluster 2024-09-10T18:52:38.370412+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:52:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:38 smithi083 bash[32381]: cluster 2024-09-10T18:52:38.372964+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:52:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:38 smithi165 bash[22591]: audit 2024-09-10T18:52:37.525556+0000 mgr.y (mgr.24526) 122 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:38 smithi165 bash[22591]: cluster 2024-09-10T18:52:37.621660+0000 osd.0 (osd.0) 87 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:52:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:38 smithi165 bash[22591]: cluster 2024-09-10T18:52:37.625934+0000 osd.0 (osd.0) 88 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:52:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:38 smithi165 bash[22591]: cluster 2024-09-10T18:52:38.370412+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:52:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:38 smithi165 bash[22591]: cluster 2024-09-10T18:52:38.372964+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:52:40.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[18629]: cluster 2024-09-10T18:52:38.512486+0000 mgr.y (mgr.24526) 123 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:40.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[18629]: cluster 2024-09-10T18:52:38.755513+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:52:40.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[18629]: cluster 2024-09-10T18:52:38.759366+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:52:40.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[32381]: cluster 2024-09-10T18:52:38.512486+0000 mgr.y (mgr.24526) 123 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:40.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[32381]: cluster 2024-09-10T18:52:38.755513+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:52:40.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:39 smithi083 bash[32381]: cluster 2024-09-10T18:52:38.759366+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:52:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:39 smithi165 bash[22591]: cluster 2024-09-10T18:52:38.512486+0000 mgr.y (mgr.24526) 123 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:39 smithi165 bash[22591]: cluster 2024-09-10T18:52:38.755513+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:52:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:39 smithi165 bash[22591]: cluster 2024-09-10T18:52:38.759366+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:52:41.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:40 smithi083 bash[18629]: cluster 2024-09-10T18:52:39.615846+0000 osd.0 (osd.0) 89 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T18:52:41.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:40 smithi083 bash[18629]: cluster 2024-09-10T18:52:39.623707+0000 osd.0 (osd.0) 90 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T18:52:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:40 smithi083 bash[32381]: cluster 2024-09-10T18:52:39.615846+0000 osd.0 (osd.0) 89 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T18:52:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:40 smithi083 bash[32381]: cluster 2024-09-10T18:52:39.623707+0000 osd.0 (osd.0) 90 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T18:52:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:40 smithi165 bash[22591]: cluster 2024-09-10T18:52:39.615846+0000 osd.0 (osd.0) 89 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T18:52:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:40 smithi165 bash[22591]: cluster 2024-09-10T18:52:39.623707+0000 osd.0 (osd.0) 90 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T18:52:41.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:42.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:40.438487+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:40.442023+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:40.513879+0000 mgr.y (mgr.24526) 124 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:40.751991+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:40.755111+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:41.392973+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[32381]: cluster 2024-09-10T18:52:41.396456+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:40.438487+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:40.442023+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:40.513879+0000 mgr.y (mgr.24526) 124 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:40.751991+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:52:42.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:40.755111+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:52:42.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:41.392973+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:52:42.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:41 smithi083 bash[18629]: cluster 2024-09-10T18:52:41.396456+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:40.438487+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:40.442023+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:40.513879+0000 mgr.y (mgr.24526) 124 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:40.751991+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b deep-scrub starts 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:40.755111+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b deep-scrub ok 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:41.392973+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:52:42.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:41 smithi165 bash[22591]: cluster 2024-09-10T18:52:41.396456+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:52:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:52:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:42] "GET /metrics HTTP/1.1" 200 220150 "" "Prometheus/2.43.0" 2024-09-10T18:52:44.259 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:43 smithi083 bash[18629]: cluster 2024-09-10T18:52:42.514601+0000 mgr.y (mgr.24526) 125 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:44.259 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:43 smithi083 bash[32381]: cluster 2024-09-10T18:52:42.514601+0000 mgr.y (mgr.24526) 125 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:44.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:43 smithi165 bash[22591]: cluster 2024-09-10T18:52:42.514601+0000 mgr.y (mgr.24526) 125 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: cluster 2024-09-10T18:52:43.409318+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.b scrub starts 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: cluster 2024-09-10T18:52:43.412899+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.b scrub ok 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: cluster 2024-09-10T18:52:43.772772+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: cluster 2024-09-10T18:52:43.776894+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: audit 2024-09-10T18:52:44.241471+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: audit 2024-09-10T18:52:44.550086+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: audit 2024-09-10T18:52:44.718654+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: audit 2024-09-10T18:52:44.720533+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[18629]: audit 2024-09-10T18:52:44.730814+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: cluster 2024-09-10T18:52:43.409318+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.b scrub starts 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: cluster 2024-09-10T18:52:43.412899+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.b scrub ok 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: cluster 2024-09-10T18:52:43.772772+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: cluster 2024-09-10T18:52:43.776894+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: audit 2024-09-10T18:52:44.241471+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: audit 2024-09-10T18:52:44.550086+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: audit 2024-09-10T18:52:44.718654+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: audit 2024-09-10T18:52:44.720533+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:45 smithi083 bash[32381]: audit 2024-09-10T18:52:44.730814+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:44 smithi165 bash[22591]: cluster 2024-09-10T18:52:43.409318+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.b scrub starts 2024-09-10T18:52:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: cluster 2024-09-10T18:52:43.412899+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.b scrub ok 2024-09-10T18:52:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: cluster 2024-09-10T18:52:43.772772+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.c scrub starts 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: cluster 2024-09-10T18:52:43.776894+0000 osd.3 (osd.3) 94 : cluster [DBG] 6.c scrub ok 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: audit 2024-09-10T18:52:44.241471+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: audit 2024-09-10T18:52:44.550086+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: audit 2024-09-10T18:52:44.718654+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: audit 2024-09-10T18:52:44.720533+0000 mon.c (mon.2) 82 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:45.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:45 smithi165 bash[22591]: audit 2024-09-10T18:52:44.730814+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:46.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- 172.21.15.83:0/209549975 >> v1:172.21.15.83:6790/0 conn(0x7f44700fb300 legacy=0x7f44700fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:46.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- 172.21.15.83:0/209549975 shutdown_connections 2024-09-10T18:52:46.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- 172.21.15.83:0/209549975 >> 172.21.15.83:0/209549975 conn(0x7f44700f70b0 msgr2=0x7f44700f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:46.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- 172.21.15.83:0/209549975 shutdown_connections 2024-09-10T18:52:46.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- 172.21.15.83:0/209549975 wait complete. 2024-09-10T18:52:46.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 librados: starting msgr at 2024-09-10T18:52:46.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 librados: starting objecter 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 Processor -- start 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- start start 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 librados: setting wanted keys 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 librados: calling monclient init 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f447011e0c0 con 0x7f447010b130 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f447011f2c0 con 0x7f44700fb300 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4476177640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f44701204c0 con 0x7f44700fbde0 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f446ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f447010b130 0x7f447011c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52808/0 (socket says 172.21.15.83:52808) 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f446ffff640 1 -- 172.21.15.83:0/2333453381 learned_addr learned my addr 172.21.15.83:0/2333453381 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f446effd640 1 --1- 172.21.15.83:0/2333453381 >> v1:172.21.15.83:6790/0 conn(0x7f44700fbde0 0x7f4470107910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49520/0 (socket says 172.21.15.83:49520) 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter ms_handle_connect 0x7f447010b130 2024-09-10T18:52:46.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter ms_handle_connect 0x7f44700fbde0 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272280563 0 0) 0x7f447011e0c0 con 0x7f447010b130 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c0034e0 con 0x7f447010b130 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter ms_handle_connect 0x7f44700fb300 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876766611 0 0) 0x7f44701204c0 con 0x7f44700fbde0 2024-09-10T18:52:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.139+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447011e0c0 con 0x7f44700fbde0 2024-09-10T18:52:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455143418 0 0) 0x7f445c0034e0 con 0x7f447010b130 2024-09-10T18:52:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44701204c0 con 0x7f447010b130 2024-09-10T18:52:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4460001990 con 0x7f447010b130 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341541556 0 0) 0x7f447011f2c0 con 0x7f44700fb300 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c0034e0 con 0x7f44700fb300 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4278190101 0 0) 0x7f447011e0c0 con 0x7f44700fbde0 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f447011f2c0 con 0x7f44700fbde0 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f44580019d0 con 0x7f44700fbde0 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1731238786 0 0) 0x7f44701204c0 con 0x7f447010b130 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 >> v1:172.21.15.83:6790/0 conn(0x7f44700fbde0 legacy=0x7f4470107910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 >> v1:172.21.15.165:6789/0 conn(0x7f44700fb300 legacy=0x7f4470107200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44701216c0 con 0x7f447010b130 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44701206b0 con 0x7f447010b130 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4476177640 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470120b00 con 0x7f447010b130 2024-09-10T18:52:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4476177640 1 librados: init done 2024-09-10T18:52:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4460001c50 con 0x7f447010b130 2024-09-10T18:52:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4460005a10 con 0x7f447010b130 2024-09-10T18:52:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.143+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4450005010 con 0x7f447010b130 2024-09-10T18:52:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f44600021b0 con 0x7f447010b130 2024-09-10T18:52:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(108..108 src has 49..108) v4 ==== 6284+0+0 (unknown 441160689 0 0) 0x7f4460089e50 con 0x7f447010b130 2024-09-10T18:52:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 10 client.14787.objecter ms_dispatch 0x7f4470000b90 osd_map(108..108 src has 49..108) v4 2024-09-10T18:52:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 3 client.14787.objecter handle_osd_map got epochs [108,108] > 0 2024-09-10T18:52:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 3 client.14787.objecter handle_osd_map decoding full epoch 108 2024-09-10T18:52:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 20 client.14787.objecter dump_active .. 0 homeless 2024-09-10T18:52:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.147+0000 7f4467fff640 10 client.14787.objecter ms_handle_connect 0x7f445c071520 2024-09-10T18:52:46.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.151+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4460056b20 con 0x7f447010b130 2024-09-10T18:52:46.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18629]: cluster 2024-09-10T18:52:44.371542+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.1c deep-scrub starts 2024-09-10T18:52:46.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18629]: cluster 2024-09-10T18:52:44.380089+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.1c deep-scrub ok 2024-09-10T18:52:46.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18629]: cluster 2024-09-10T18:52:44.515273+0000 mgr.y (mgr.24526) 126 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[32381]: cluster 2024-09-10T18:52:44.371542+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.1c deep-scrub starts 2024-09-10T18:52:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[32381]: cluster 2024-09-10T18:52:44.380089+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.1c deep-scrub ok 2024-09-10T18:52:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[32381]: cluster 2024-09-10T18:52:44.515273+0000 mgr.y (mgr.24526) 126 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:46.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.327+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 --> v1:172.21.15.83:6789/0 -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f4450005840 con 0x7f447010b130 2024-09-10T18:52:46.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.335+0000 7f4467fff640 1 -- 172.21.15.83:0/2333453381 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) v1 ==== 136+0+0 (unknown 398629962 0 0) 0x7f44600564c0 con 0x7f447010b130 2024-09-10T18:52:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.339+0000 7f4476177640 10 librados: watch_flush enter 2024-09-10T18:52:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.339+0000 7f4476177640 10 librados: watch_flush exit 2024-09-10T18:52:46.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.339+0000 7f4476177640 20 client.14787.objecter shutdown clearing up homeless session... 2024-09-10T18:52:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.339+0000 7f4476177640 10 client.14787.objecter successfully canceled tick 2024-09-10T18:52:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 >> v1:172.21.15.83:6800/3071793735 conn(0x7f445c071520 legacy=0x7f445c0739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 >> v1:172.21.15.83:6789/0 conn(0x7f447010b130 legacy=0x7f447011c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:46.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 shutdown_connections 2024-09-10T18:52:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 >> 172.21.15.83:0/2333453381 conn(0x7f44700f70b0 msgr2=0x7f447010d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 shutdown_connections 2024-09-10T18:52:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 -- 172.21.15.83:0/2333453381 wait complete. 2024-09-10T18:52:46.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:46.343+0000 7f4476177640 1 librados: shutdown 2024-09-10T18:52:46.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:46 smithi165 bash[22591]: cluster 2024-09-10T18:52:44.371542+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.1c deep-scrub starts 2024-09-10T18:52:46.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:46 smithi165 bash[22591]: cluster 2024-09-10T18:52:44.380089+0000 osd.7 (osd.7) 84 : cluster [DBG] 6.1c deep-scrub ok 2024-09-10T18:52:46.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:46 smithi165 bash[22591]: cluster 2024-09-10T18:52:44.515273+0000 mgr.y (mgr.24526) 126 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:52:46.695 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE Bus STOPPING 2024-09-10T18:52:46.798 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T18:52:46.798 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE Bus STOPPED 2024-09-10T18:52:46.798 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE Bus STARTING 2024-09-10T18:52:47.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE Serving on http://:::9283 2024-09-10T18:52:47.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:46 smithi083 bash[18896]: [10/Sep/2024:18:52:46] ENGINE Bus STARTED 2024-09-10T18:52:47.080 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-10T18:52:47.080 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T18:52:47.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:52:47.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:47.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: cluster 2024-09-10T18:52:45.816906+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: cluster 2024-09-10T18:52:45.821617+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: audit 2024-09-10T18:52:46.341837+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.83:0/2333453381' entity='client.admin' 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: audit 2024-09-10T18:52:46.359351+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: audit 2024-09-10T18:52:46.361703+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: audit 2024-09-10T18:52:46.362719+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: audit 2024-09-10T18:52:46.373510+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: cluster 2024-09-10T18:52:46.516782+0000 mgr.y (mgr.24526) 127 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: cluster 2024-09-10T18:52:46.824827+0000 osd.3 (osd.3) 97 : cluster [DBG] 4.17 deep-scrub starts 2024-09-10T18:52:47.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:47 smithi165 bash[22591]: cluster 2024-09-10T18:52:46.829490+0000 osd.3 (osd.3) 98 : cluster [DBG] 4.17 deep-scrub ok 2024-09-10T18:52:47.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: cluster 2024-09-10T18:52:45.816906+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: cluster 2024-09-10T18:52:45.821617+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: audit 2024-09-10T18:52:46.341837+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.83:0/2333453381' entity='client.admin' 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: audit 2024-09-10T18:52:46.359351+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: audit 2024-09-10T18:52:46.361703+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: audit 2024-09-10T18:52:46.362719+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: audit 2024-09-10T18:52:46.373510+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: cluster 2024-09-10T18:52:46.516782+0000 mgr.y (mgr.24526) 127 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: cluster 2024-09-10T18:52:46.824827+0000 osd.3 (osd.3) 97 : cluster [DBG] 4.17 deep-scrub starts 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[18629]: cluster 2024-09-10T18:52:46.829490+0000 osd.3 (osd.3) 98 : cluster [DBG] 4.17 deep-scrub ok 2024-09-10T18:52:47.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: cluster 2024-09-10T18:52:45.816906+0000 osd.3 (osd.3) 95 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: cluster 2024-09-10T18:52:45.821617+0000 osd.3 (osd.3) 96 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: audit 2024-09-10T18:52:46.341837+0000 mon.a (mon.0) 864 : audit [INF] from='client.? v1:172.21.15.83:0/2333453381' entity='client.admin' 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: audit 2024-09-10T18:52:46.359351+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: audit 2024-09-10T18:52:46.361703+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: audit 2024-09-10T18:52:46.362719+0000 mon.c (mon.2) 85 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: audit 2024-09-10T18:52:46.373510+0000 mon.a (mon.0) 865 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: cluster 2024-09-10T18:52:46.516782+0000 mgr.y (mgr.24526) 127 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: cluster 2024-09-10T18:52:46.824827+0000 osd.3 (osd.3) 97 : cluster [DBG] 4.17 deep-scrub starts 2024-09-10T18:52:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:47 smithi083 bash[32381]: cluster 2024-09-10T18:52:46.829490+0000 osd.3 (osd.3) 98 : cluster [DBG] 4.17 deep-scrub ok 2024-09-10T18:52:48.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:48 smithi083 bash[18629]: audit 2024-09-10T18:52:47.532402+0000 mgr.y (mgr.24526) 128 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:48 smithi083 bash[32381]: audit 2024-09-10T18:52:47.532402+0000 mgr.y (mgr.24526) 128 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:48 smithi165 bash[22591]: audit 2024-09-10T18:52:47.532402+0000 mgr.y (mgr.24526) 128 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:47.898347+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:47.898460+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:48.318112+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:48.328620+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:48.387736+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[18629]: cluster 2024-09-10T18:52:48.517510+0000 mgr.y (mgr.24526) 129 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:47.898347+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:47.898460+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:48.318112+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T18:52:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:48.328620+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T18:52:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:48.387736+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-10T18:52:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:49 smithi083 bash[32381]: cluster 2024-09-10T18:52:48.517510+0000 mgr.y (mgr.24526) 129 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:47.898347+0000 osd.1 (osd.1) 85 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:47.898460+0000 osd.1 (osd.1) 86 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:48.318112+0000 osd.7 (osd.7) 85 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T18:52:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:48.328620+0000 osd.7 (osd.7) 86 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T18:52:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:48.387736+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-09-10T18:52:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:49 smithi165 bash[22591]: cluster 2024-09-10T18:52:48.517510+0000 mgr.y (mgr.24526) 129 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:50.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:50 smithi083 bash[18629]: cluster 2024-09-10T18:52:49.564783+0000 osd.5 (osd.5) 77 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:52:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:50 smithi083 bash[18629]: cluster 2024-09-10T18:52:49.569803+0000 osd.5 (osd.5) 78 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:52:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:50 smithi083 bash[32381]: cluster 2024-09-10T18:52:49.564783+0000 osd.5 (osd.5) 77 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:52:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:50 smithi083 bash[32381]: cluster 2024-09-10T18:52:49.569803+0000 osd.5 (osd.5) 78 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:52:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:50 smithi165 bash[22591]: cluster 2024-09-10T18:52:49.564783+0000 osd.5 (osd.5) 77 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:52:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:50 smithi165 bash[22591]: cluster 2024-09-10T18:52:49.569803+0000 osd.5 (osd.5) 78 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:52:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[18629]: cluster 2024-09-10T18:52:50.383632+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a scrub starts 2024-09-10T18:52:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[18629]: cluster 2024-09-10T18:52:50.387151+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a scrub ok 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[18629]: cluster 2024-09-10T18:52:50.518922+0000 mgr.y (mgr.24526) 130 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[18629]: cluster 2024-09-10T18:52:50.793402+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[18629]: cluster 2024-09-10T18:52:50.797102+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[32381]: cluster 2024-09-10T18:52:50.383632+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a scrub starts 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[32381]: cluster 2024-09-10T18:52:50.387151+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a scrub ok 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[32381]: cluster 2024-09-10T18:52:50.518922+0000 mgr.y (mgr.24526) 130 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[32381]: cluster 2024-09-10T18:52:50.793402+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:52:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:51 smithi083 bash[32381]: cluster 2024-09-10T18:52:50.797102+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:52:51.862 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:52:51.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:52:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[22591]: cluster 2024-09-10T18:52:50.383632+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a scrub starts 2024-09-10T18:52:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[22591]: cluster 2024-09-10T18:52:50.387151+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a scrub ok 2024-09-10T18:52:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[22591]: cluster 2024-09-10T18:52:50.518922+0000 mgr.y (mgr.24526) 130 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[22591]: cluster 2024-09-10T18:52:50.793402+0000 osd.3 (osd.3) 99 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:52:51.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:51 smithi165 bash[22591]: cluster 2024-09-10T18:52:50.797102+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:52:52.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:52 smithi083 bash[18629]: cluster 2024-09-10T18:52:51.764119+0000 osd.3 (osd.3) 101 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:52:52.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:52 smithi083 bash[18629]: cluster 2024-09-10T18:52:51.767530+0000 osd.3 (osd.3) 102 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:52:52.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:52 smithi083 bash[32381]: cluster 2024-09-10T18:52:51.764119+0000 osd.3 (osd.3) 101 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:52:52.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:52 smithi083 bash[32381]: cluster 2024-09-10T18:52:51.767530+0000 osd.3 (osd.3) 102 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:52:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:52 smithi165 bash[22591]: cluster 2024-09-10T18:52:51.764119+0000 osd.3 (osd.3) 101 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:52:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:52 smithi165 bash[22591]: cluster 2024-09-10T18:52:51.767530+0000 osd.3 (osd.3) 102 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:52:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:52:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:52:52] "GET /metrics HTTP/1.1" 200 220134 "" "Prometheus/2.43.0" 2024-09-10T18:52:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[18629]: cluster 2024-09-10T18:52:52.519589+0000 mgr.y (mgr.24526) 131 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[18629]: cluster 2024-09-10T18:52:52.607559+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[18629]: cluster 2024-09-10T18:52:52.611099+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[18629]: cluster 2024-09-10T18:52:52.785686+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[18629]: cluster 2024-09-10T18:52:52.790423+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[32381]: cluster 2024-09-10T18:52:52.519589+0000 mgr.y (mgr.24526) 131 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[32381]: cluster 2024-09-10T18:52:52.607559+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[32381]: cluster 2024-09-10T18:52:52.611099+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[32381]: cluster 2024-09-10T18:52:52.785686+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:52:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:53 smithi083 bash[32381]: cluster 2024-09-10T18:52:52.790423+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:52:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:53 smithi165 bash[22591]: cluster 2024-09-10T18:52:52.519589+0000 mgr.y (mgr.24526) 131 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:53 smithi165 bash[22591]: cluster 2024-09-10T18:52:52.607559+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:52:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:53 smithi165 bash[22591]: cluster 2024-09-10T18:52:52.611099+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:52:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:53 smithi165 bash[22591]: cluster 2024-09-10T18:52:52.785686+0000 osd.3 (osd.3) 103 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:52:53.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:53 smithi165 bash[22591]: cluster 2024-09-10T18:52:52.790423+0000 osd.3 (osd.3) 104 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:52:54.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:54 smithi083 bash[32381]: cluster 2024-09-10T18:52:53.275151+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:52:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:54 smithi083 bash[32381]: cluster 2024-09-10T18:52:53.278821+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:52:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:54 smithi083 bash[18629]: cluster 2024-09-10T18:52:53.275151+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:52:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:54 smithi083 bash[18629]: cluster 2024-09-10T18:52:53.278821+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:52:54.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:54 smithi165 bash[22591]: cluster 2024-09-10T18:52:53.275151+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:52:54.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:54 smithi165 bash[22591]: cluster 2024-09-10T18:52:53.278821+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:52:55.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 -- 172.21.15.83:0/3819438373 >> v1:172.21.15.83:6789/0 conn(0x7fb7fc070590 legacy=0x7fb7fc1098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:55.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 -- 172.21.15.83:0/3819438373 shutdown_connections 2024-09-10T18:52:55.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 -- 172.21.15.83:0/3819438373 >> 172.21.15.83:0/3819438373 conn(0x7fb7fc0f8150 msgr2=0x7fb7fc0fa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:55.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 -- 172.21.15.83:0/3819438373 shutdown_connections 2024-09-10T18:52:55.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 -- 172.21.15.83:0/3819438373 wait complete. 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 librados: starting msgr at 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 librados: starting objecter 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.282+0000 7fb8008e4640 1 Processor -- start 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- start start 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 librados: setting wanted keys 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 librados: calling monclient init 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7fc11e550 con 0x7fb7fc070590 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7fc11f6f0 con 0x7fb7fc10b110 2024-09-10T18:52:55.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7fc120890 con 0x7fb7fc06fab0 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f9d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb7fc070590 0x7fb7fc119700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55722/0 (socket says 172.21.15.83:55722) 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7fa59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb7fc06fab0 0x7fb7fc10a2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:46882/0 (socket says 172.21.15.83:46882) 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f9d9b640 1 -- 172.21.15.83:0/885444477 learned_addr learned my addr 172.21.15.83:0/885444477 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7fc06fab0 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7fc070590 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737723031 0 0) 0x7fb7fc11e550 con 0x7fb7fc070590 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8003720 con 0x7fb7fc070590 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664363256 0 0) 0x7fb7fc120890 con 0x7fb7fc06fab0 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc11e550 con 0x7fb7fc06fab0 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fb7fc10b110 2024-09-10T18:52:55.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433233779 0 0) 0x7fb7fc11f6f0 con 0x7fb7fc10b110 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc120890 con 0x7fb7fc10b110 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3783734707 0 0) 0x7fb7e8003720 con 0x7fb7fc070590 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc11f6f0 con 0x7fb7fc070590 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb7f4002f30 con 0x7fb7fc070590 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983373653 0 0) 0x7fb7fc11e550 con 0x7fb7fc06fab0 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e8003720 con 0x7fb7fc06fab0 2024-09-10T18:52:55.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb7ec0018c0 con 0x7fb7fc06fab0 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254318980 0 0) 0x7fb7fc120890 con 0x7fb7fc10b110 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc11e550 con 0x7fb7fc10b110 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb7e40018f0 con 0x7fb7fc10b110 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2183776860 0 0) 0x7fb7fc11f6f0 con 0x7fb7fc070590 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 >> v1:172.21.15.83:6790/0 conn(0x7fb7fc06fab0 legacy=0x7fb7fc10a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 >> v1:172.21.15.165:6789/0 conn(0x7fb7fc10b110 legacy=0x7fb7fc11ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc121a30 con 0x7fb7fc070590 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc120a80 con 0x7fb7fc070590 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb7f4001e70 con 0x7fb7fc070590 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb7f4003a90 con 0x7fb7fc070590 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 10 client.14793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:52:55.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc120ed0 con 0x7fb7fc070590 2024-09-10T18:52:55.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.286+0000 7fb8008e4640 1 librados: init done 2024-09-10T18:52:55.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc0711f0 con 0x7fb7fc070590 2024-09-10T18:52:55.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fb7f4003bf0 con 0x7fb7fc070590 2024-09-10T18:52:55.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(109..109 src has 59..109) v4 ==== 6284+0+0 (unknown 2035185908 0 0) 0x7fb7f408a310 con 0x7fb7fc070590 2024-09-10T18:52:55.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 10 client.14793.objecter ms_dispatch 0x7fb7fc000b90 osd_map(109..109 src has 59..109) v4 2024-09-10T18:52:55.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 3 client.14793.objecter handle_osd_map got epochs [109,109] > 0 2024-09-10T18:52:55.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 3 client.14793.objecter handle_osd_map decoding full epoch 109 2024-09-10T18:52:55.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 20 client.14793.objecter dump_active .. 0 homeless 2024-09-10T18:52:55.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.290+0000 7fb7f2ffd640 10 client.14793.objecter ms_handle_connect 0x7fb7e8071860 2024-09-10T18:52:55.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.294+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fb7f4056fe0 con 0x7fb7fc070590 2024-09-10T18:52:55.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.474+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7fc1089c0 con 0x7fb7e8071860 2024-09-10T18:52:55.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.490+0000 7fb7f2ffd640 1 -- 172.21.15.83:0/885444477 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384677 (unknown 2965378022 0 1681814988) 0x7fb7fc1089c0 con 0x7fb7e8071860 2024-09-10T18:52:55.502 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:52:55.504 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:52:55.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 10 librados: watch_flush enter 2024-09-10T18:52:55.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 10 librados: watch_flush exit 2024-09-10T18:52:55.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 20 client.14793.objecter shutdown clearing up homeless session... 2024-09-10T18:52:55.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 10 client.14793.objecter successfully canceled tick 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 >> v1:172.21.15.83:6800/3071793735 conn(0x7fb7e8071860 legacy=0x7fb7e8073d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 >> v1:172.21.15.83:6789/0 conn(0x7fb7fc070590 legacy=0x7fb7fc119700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 shutdown_connections 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 >> 172.21.15.83:0/885444477 conn(0x7fb7fc0f8150 msgr2=0x7fb7fc0fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 shutdown_connections 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 -- 172.21.15.83:0/885444477 wait complete. 2024-09-10T18:52:55.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:52:55.498+0000 7fb8008e4640 1 librados: shutdown 2024-09-10T18:52:55.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[18629]: cluster 2024-09-10T18:52:54.520259+0000 mgr.y (mgr.24526) 132 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[18629]: cluster 2024-09-10T18:52:54.732646+0000 osd.3 (osd.3) 105 : cluster [DBG] 3.2 deep-scrub starts 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[18629]: cluster 2024-09-10T18:52:54.743248+0000 osd.3 (osd.3) 106 : cluster [DBG] 3.2 deep-scrub ok 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[18629]: cluster 2024-09-10T18:52:54.908917+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[18629]: cluster 2024-09-10T18:52:54.909023+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[32381]: cluster 2024-09-10T18:52:54.520259+0000 mgr.y (mgr.24526) 132 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[32381]: cluster 2024-09-10T18:52:54.732646+0000 osd.3 (osd.3) 105 : cluster [DBG] 3.2 deep-scrub starts 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[32381]: cluster 2024-09-10T18:52:54.743248+0000 osd.3 (osd.3) 106 : cluster [DBG] 3.2 deep-scrub ok 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[32381]: cluster 2024-09-10T18:52:54.908917+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:55.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:55 smithi083 bash[32381]: cluster 2024-09-10T18:52:54.909023+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:55 smithi165 bash[22591]: cluster 2024-09-10T18:52:54.520259+0000 mgr.y (mgr.24526) 132 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:52:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:55 smithi165 bash[22591]: cluster 2024-09-10T18:52:54.732646+0000 osd.3 (osd.3) 105 : cluster [DBG] 3.2 deep-scrub starts 2024-09-10T18:52:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:55 smithi165 bash[22591]: cluster 2024-09-10T18:52:54.743248+0000 osd.3 (osd.3) 106 : cluster [DBG] 3.2 deep-scrub ok 2024-09-10T18:52:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:55 smithi165 bash[22591]: cluster 2024-09-10T18:52:54.908917+0000 osd.3 (osd.3) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:55 smithi165 bash[22591]: cluster 2024-09-10T18:52:54.909023+0000 osd.3 (osd.3) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:56.292 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-10T18:52:55.437393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1062,"num_read_kb":918,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":626,"ondisk_log_size":626,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2390488,"kb_used_data":7488,"kb_used_omap":13,"kb_used_meta":2382898,"kb_avail":747570728,"statfs":{"total":767960285184,"available":765512425472,"internally_reserved":0,"allocated":7667712,"data_stored":4550951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2440088256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.921831"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:31.777606+0000","last_change":"2024-09-10T18:52:31.777492+0000","last_active":"2024-09-10T18:52:31.777606+0000","last_peered":"2024-09-10T18:52:31.777606+0000","last_clean":"2024-09-10T18:52:31.777606+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:52:31.777606+0000","last_undegraded":"2024-09-10T18:52:31.777606+0000","last_fullsized":"2024-09-10T18:52:31.777606+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:31.777344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.777344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.153926+0000","scrub_duration":0.0041891280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.898773+0000","last_change":"2024-09-10T18:52:17.464193+0000","last_active":"2024-09-10T18:52:25.898773+0000","last_peered":"2024-09-10T18:52:25.898773+0000","last_clean":"2024-09-10T18:52:25.898773+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:52:25.898773+0000","last_undegraded":"2024-09-10T18:52:25.898773+0000","last_fullsized":"2024-09-10T18:52:25.898773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:17.463956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.463956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:35.714604+0000","scrub_duration":0.004114698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":71,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900510+0000","last_change":"2024-09-10T18:51:37.694332+0000","last_active":"2024-09-10T18:52:25.900510+0000","last_peered":"2024-09-10T18:52:25.900510+0000","last_clean":"2024-09-10T18:52:25.900510+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:52:25.900510+0000","last_undegraded":"2024-09-10T18:52:25.900510+0000","last_fullsized":"2024-09-10T18:52:25.900510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:37.694143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.694143+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.236407+0000","scrub_duration":0.0048100759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.898660+0000","last_change":"2024-09-10T18:52:20.734242+0000","last_active":"2024-09-10T18:52:25.898660+0000","last_peered":"2024-09-10T18:52:25.898660+0000","last_clean":"2024-09-10T18:52:25.898660+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:52:25.898660+0000","last_undegraded":"2024-09-10T18:52:25.898660+0000","last_fullsized":"2024-09-10T18:52:25.898660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.733951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.733951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.495882+0000","scrub_duration":0.0029697920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900686+0000","last_change":"2024-09-10T18:51:41.627258+0000","last_active":"2024-09-10T18:52:25.900686+0000","last_peered":"2024-09-10T18:52:25.900686+0000","last_clean":"2024-09-10T18:52:25.900686+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:52:25.900686+0000","last_undegraded":"2024-09-10T18:52:25.900686+0000","last_fullsized":"2024-09-10T18:52:25.900686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.627147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.627147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:48.610764+0000","scrub_duration":0.0032008610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.898559+0000","last_change":"2024-09-10T18:52:23.600334+0000","last_active":"2024-09-10T18:52:25.898559+0000","last_peered":"2024-09-10T18:52:25.898559+0000","last_clean":"2024-09-10T18:52:25.898559+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:52:25.898559+0000","last_undegraded":"2024-09-10T18:52:25.898559+0000","last_fullsized":"2024-09-10T18:52:25.898559+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:23.600141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.600141+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.041119+0000","scrub_duration":0.0045540590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":74,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894323+0000","last_change":"2024-09-10T18:52:25.178969+0000","last_active":"2024-09-10T18:52:25.894323+0000","last_peered":"2024-09-10T18:52:25.894323+0000","last_clean":"2024-09-10T18:52:25.894323+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:52:25.894323+0000","last_undegraded":"2024-09-10T18:52:25.894323+0000","last_fullsized":"2024-09-10T18:52:25.894323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.178800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.178800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.952086+0000","scrub_duration":0.0034539140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.143606+0000","last_change":"2024-09-10T18:52:26.143465+0000","last_active":"2024-09-10T18:52:26.143606+0000","last_peered":"2024-09-10T18:52:26.143606+0000","last_clean":"2024-09-10T18:52:26.143606+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:52:26.143606+0000","last_undegraded":"2024-09-10T18:52:26.143606+0000","last_fullsized":"2024-09-10T18:52:26.143606+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:26.143297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:26.143297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:44.663812+0000","scrub_duration":0.0032662059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043896+0000","last_change":"2024-09-10T18:52:16.237573+0000","last_active":"2024-09-10T18:52:26.043896+0000","last_peered":"2024-09-10T18:52:26.043896+0000","last_clean":"2024-09-10T18:52:26.043896+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:52:26.043896+0000","last_undegraded":"2024-09-10T18:52:26.043896+0000","last_fullsized":"2024-09-10T18:52:26.043896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:16.237414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:16.237414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.186738+0000","scrub_duration":0.0033375729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:28.673381+0000","last_change":"2024-09-10T18:52:28.673257+0000","last_active":"2024-09-10T18:52:28.673381+0000","last_peered":"2024-09-10T18:52:28.673381+0000","last_clean":"2024-09-10T18:52:28.673381+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:52:28.673381+0000","last_undegraded":"2024-09-10T18:52:28.673381+0000","last_fullsized":"2024-09-10T18:52:28.673381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.673170+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:53.934483+0000","scrub_duration":0.01076011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:28.116341+0000","last_change":"2024-09-10T18:52:28.116242+0000","last_active":"2024-09-10T18:52:28.116341+0000","last_peered":"2024-09-10T18:52:28.116341+0000","last_clean":"2024-09-10T18:52:28.116341+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:52:28.116341+0000","last_undegraded":"2024-09-10T18:52:28.116341+0000","last_fullsized":"2024-09-10T18:52:28.116341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.116084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.380860+0000","scrub_duration":0.003567785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:33.676581+0000","last_change":"2024-09-10T18:52:33.676485+0000","last_active":"2024-09-10T18:52:33.676581+0000","last_peered":"2024-09-10T18:52:33.676581+0000","last_clean":"2024-09-10T18:52:33.676581+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:52:33.676581+0000","last_undegraded":"2024-09-10T18:52:33.676581+0000","last_fullsized":"2024-09-10T18:52:33.676581+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:33.676311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:33.676311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:53.929050+0000","scrub_duration":0.0037210590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046719+0000","last_change":"2024-09-10T18:52:22.326533+0000","last_active":"2024-09-10T18:52:26.046719+0000","last_peered":"2024-09-10T18:52:26.046719+0000","last_clean":"2024-09-10T18:52:26.046719+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:52:26.046719+0000","last_undegraded":"2024-09-10T18:52:26.046719+0000","last_fullsized":"2024-09-10T18:52:26.046719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.326315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:34.718300+0000","scrub_duration":0.003554892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":92,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.041539+0000","last_change":"2024-09-10T18:51:43.650049+0000","last_active":"2024-09-10T18:52:26.041539+0000","last_peered":"2024-09-10T18:52:26.041539+0000","last_clean":"2024-09-10T18:52:26.041539+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:52:26.041539+0000","last_undegraded":"2024-09-10T18:52:26.041539+0000","last_fullsized":"2024-09-10T18:52:26.041539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:51:43.649947+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.649947+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.873051+0000","scrub_duration":0.003856703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894909+0000","last_change":"2024-09-10T18:52:01.415959+0000","last_active":"2024-09-10T18:52:25.894909+0000","last_peered":"2024-09-10T18:52:25.894909+0000","last_clean":"2024-09-10T18:52:25.894909+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:52:25.894909+0000","last_undegraded":"2024-09-10T18:52:25.894909+0000","last_fullsized":"2024-09-10T18:52:25.894909+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.415718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.415718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:25.902687+0000","scrub_duration":0.0036293950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.896574+0000","last_change":"2024-09-10T18:51:36.537160+0000","last_active":"2024-09-10T18:52:25.896574+0000","last_peered":"2024-09-10T18:52:25.896574+0000","last_clean":"2024-09-10T18:52:25.896574+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:52:25.896574+0000","last_undegraded":"2024-09-10T18:52:25.896574+0000","last_fullsized":"2024-09-10T18:52:25.896574+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:36.536996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:36.536996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.415469+0000","scrub_duration":0.0033809280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":95,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040587+0000","last_change":"2024-09-10T18:52:02.356154+0000","last_active":"2024-09-10T18:52:26.040587+0000","last_peered":"2024-09-10T18:52:26.040587+0000","last_clean":"2024-09-10T18:52:26.040587+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:52:26.040587+0000","last_undegraded":"2024-09-10T18:52:26.040587+0000","last_fullsized":"2024-09-10T18:52:26.040587+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:02.355980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:02.355980+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:21.108556+0000","scrub_duration":0.0055219309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:43.413356+0000","last_change":"2024-09-10T18:52:43.413178+0000","last_active":"2024-09-10T18:52:43.413356+0000","last_peered":"2024-09-10T18:52:43.413356+0000","last_clean":"2024-09-10T18:52:43.413356+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:52:43.413356+0000","last_undegraded":"2024-09-10T18:52:43.413356+0000","last_fullsized":"2024-09-10T18:52:43.413356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:43.412924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:43.412924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:09.023724+0000","scrub_duration":0.0033475380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.960314+0000","last_change":"2024-09-10T18:52:17.252670+0000","last_active":"2024-09-10T18:52:45.960314+0000","last_peered":"2024-09-10T18:52:45.960314+0000","last_clean":"2024-09-10T18:52:45.960314+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:52:45.960314+0000","last_undegraded":"2024-09-10T18:52:45.960314+0000","last_fullsized":"2024-09-10T18:52:45.960314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.252498+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:23.792093+0000","scrub_duration":0.0045932949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043650+0000","last_change":"2024-09-10T18:52:21.401011+0000","last_active":"2024-09-10T18:52:26.043650+0000","last_peered":"2024-09-10T18:52:26.043650+0000","last_clean":"2024-09-10T18:52:26.043650+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:52:26.043650+0000","last_undegraded":"2024-09-10T18:52:26.043650+0000","last_fullsized":"2024-09-10T18:52:26.043650+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.400853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.400853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:30.701974+0000","scrub_duration":0.0034523370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":86,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900140+0000","last_change":"2024-09-10T18:52:17.707983+0000","last_active":"2024-09-10T18:52:25.900140+0000","last_peered":"2024-09-10T18:52:25.900140+0000","last_clean":"2024-09-10T18:52:25.900140+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:52:25.900140+0000","last_undegraded":"2024-09-10T18:52:25.900140+0000","last_fullsized":"2024-09-10T18:52:25.900140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:52:17.707813+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.707813+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.139488+0000","scrub_duration":0.0045170389999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-10T18:52:50.387424+0000","last_change":"2024-09-10T18:52:50.387339+0000","last_active":"2024-09-10T18:52:50.387424+0000","last_peered":"2024-09-10T18:52:50.387424+0000","last_clean":"2024-09-10T18:52:50.387424+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:52:50.387424+0000","last_undegraded":"2024-09-10T18:52:50.387424+0000","last_fullsized":"2024-09-10T18:52:50.387424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:50.387160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:52:50.387160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.160775+0000","scrub_duration":0.0034323050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895470+0000","last_change":"2024-09-10T18:51:46.600626+0000","last_active":"2024-09-10T18:52:25.895470+0000","last_peered":"2024-09-10T18:52:25.895470+0000","last_clean":"2024-09-10T18:52:25.895470+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:52:25.895470+0000","last_undegraded":"2024-09-10T18:52:25.895470+0000","last_fullsized":"2024-09-10T18:52:25.895470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.600460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.600460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.436438+0000","scrub_duration":0.0032588019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895874+0000","last_change":"2024-09-10T18:51:53.429372+0000","last_active":"2024-09-10T18:52:25.895874+0000","last_peered":"2024-09-10T18:52:25.895874+0000","last_clean":"2024-09-10T18:52:25.895874+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:52:25.895874+0000","last_undegraded":"2024-09-10T18:52:25.895874+0000","last_fullsized":"2024-09-10T18:52:25.895874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:53.429216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.429216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.844395+0000","scrub_duration":0.003713091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":91,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900489+0000","last_change":"2024-09-10T18:51:55.522605+0000","last_active":"2024-09-10T18:52:25.900489+0000","last_peered":"2024-09-10T18:52:25.900489+0000","last_clean":"2024-09-10T18:52:25.900489+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:52:25.900489+0000","last_undegraded":"2024-09-10T18:52:25.900489+0000","last_fullsized":"2024-09-10T18:52:25.900489+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:55.522513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.522513+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.705011+0000","scrub_duration":0.0065065660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042785+0000","last_change":"2024-09-10T18:52:25.326041+0000","last_active":"2024-09-10T18:52:26.042785+0000","last_peered":"2024-09-10T18:52:26.042785+0000","last_clean":"2024-09-10T18:52:26.042785+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:52:26.042785+0000","last_undegraded":"2024-09-10T18:52:26.042785+0000","last_fullsized":"2024-09-10T18:52:26.042785+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.325803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.325803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.547370+0000","scrub_duration":0.0039072000000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043557+0000","last_change":"2024-09-10T18:51:55.508740+0000","last_active":"2024-09-10T18:52:26.043557+0000","last_peered":"2024-09-10T18:52:26.043557+0000","last_clean":"2024-09-10T18:52:26.043557+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:52:26.043557+0000","last_undegraded":"2024-09-10T18:52:26.043557+0000","last_fullsized":"2024-09-10T18:52:26.043557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:55.508582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.508582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.812679+0000","scrub_duration":0.0034042920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040859+0000","last_change":"2024-09-10T18:51:50.674985+0000","last_active":"2024-09-10T18:52:26.040859+0000","last_peered":"2024-09-10T18:52:26.040859+0000","last_clean":"2024-09-10T18:52:26.040859+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:52:26.040859+0000","last_undegraded":"2024-09-10T18:52:26.040859+0000","last_fullsized":"2024-09-10T18:52:26.040859+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.674821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:16.792421+0000","scrub_duration":0.003696628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":94,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895103+0000","last_change":"2024-09-10T18:51:29.827804+0000","last_active":"2024-09-10T18:52:25.895103+0000","last_peered":"2024-09-10T18:52:25.895103+0000","last_clean":"2024-09-10T18:52:25.895103+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:52:25.895103+0000","last_undegraded":"2024-09-10T18:52:25.895103+0000","last_fullsized":"2024-09-10T18:52:25.895103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:51:29.827615+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.827615+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.030826+0000","scrub_duration":0.0051889199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046015+0000","last_change":"2024-09-10T18:52:23.328971+0000","last_active":"2024-09-10T18:52:26.046015+0000","last_peered":"2024-09-10T18:52:26.046015+0000","last_clean":"2024-09-10T18:52:26.046015+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:52:26.046015+0000","last_undegraded":"2024-09-10T18:52:26.046015+0000","last_fullsized":"2024-09-10T18:52:26.046015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:23.328703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.328703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.447554+0000","scrub_duration":0.0041061600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.960265+0000","last_change":"2024-09-10T18:51:40.694756+0000","last_active":"2024-09-10T18:52:45.960265+0000","last_peered":"2024-09-10T18:52:45.960265+0000","last_clean":"2024-09-10T18:52:45.960265+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:52:45.960265+0000","last_undegraded":"2024-09-10T18:52:45.960265+0000","last_fullsized":"2024-09-10T18:52:45.960265+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:51:40.694630+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.694630+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.465301+0000","scrub_duration":0.0057213519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":63,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:43.777206+0000","last_change":"2024-09-10T18:52:43.777074+0000","last_active":"2024-09-10T18:52:43.777206+0000","last_peered":"2024-09-10T18:52:43.777206+0000","last_clean":"2024-09-10T18:52:43.777206+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:52:43.777206+0000","last_undegraded":"2024-09-10T18:52:43.777206+0000","last_fullsized":"2024-09-10T18:52:43.777206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:52:43.776904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:43.776904+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.160729+0000","scrub_duration":0.0039940949999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:31.620273+0000","last_change":"2024-09-10T18:52:31.620025+0000","last_active":"2024-09-10T18:52:31.620273+0000","last_peered":"2024-09-10T18:52:31.620273+0000","last_clean":"2024-09-10T18:52:31.620273+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:52:31.620273+0000","last_undegraded":"2024-09-10T18:52:31.620273+0000","last_fullsized":"2024-09-10T18:52:31.620273+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:52:31.619837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.619837+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:51.678739+0000","scrub_duration":0.0054562689999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044480+0000","last_change":"2024-09-10T18:51:47.158565+0000","last_active":"2024-09-10T18:52:26.044480+0000","last_peered":"2024-09-10T18:52:26.044480+0000","last_clean":"2024-09-10T18:52:26.044480+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:52:26.044480+0000","last_undegraded":"2024-09-10T18:52:26.044480+0000","last_fullsized":"2024-09-10T18:52:26.044480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:47.158405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.158405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.185469+0000","scrub_duration":0.0034278289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894516+0000","last_change":"2024-09-10T18:51:28.793948+0000","last_active":"2024-09-10T18:52:25.894516+0000","last_peered":"2024-09-10T18:52:25.894516+0000","last_clean":"2024-09-10T18:52:25.894516+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:52:25.894516+0000","last_undegraded":"2024-09-10T18:52:25.894516+0000","last_fullsized":"2024-09-10T18:52:25.894516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:28.793787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.793787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.515669+0000","scrub_duration":0.0035723209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":260,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-10T18:52:48.759716+0000","last_change":"2024-09-10T18:52:00.591712+0000","last_active":"2024-09-10T18:52:48.759716+0000","last_peered":"2024-09-10T18:52:48.759716+0000","last_clean":"2024-09-10T18:52:48.759716+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:52:48.759716+0000","last_undegraded":"2024-09-10T18:52:48.759716+0000","last_fullsized":"2024-09-10T18:52:48.759716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:52:00.591632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.591632+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:05.247607+0000","scrub_duration":0.0032914160000000001,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":188,"num_read_kb":183,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044471+0000","last_change":"2024-09-10T18:51:56.155039+0000","last_active":"2024-09-10T18:52:26.044471+0000","last_peered":"2024-09-10T18:52:26.044471+0000","last_clean":"2024-09-10T18:52:26.044471+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:52:26.044471+0000","last_undegraded":"2024-09-10T18:52:26.044471+0000","last_fullsized":"2024-09-10T18:52:26.044471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.154883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.154883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.178313+0000","scrub_duration":0.0033842830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900839+0000","last_change":"2024-09-10T18:52:09.660716+0000","last_active":"2024-09-10T18:52:25.900839+0000","last_peered":"2024-09-10T18:52:25.900839+0000","last_clean":"2024-09-10T18:52:25.900839+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:52:25.900839+0000","last_undegraded":"2024-09-10T18:52:25.900839+0000","last_fullsized":"2024-09-10T18:52:25.900839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:52:09.660551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.660551+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.065594+0000","scrub_duration":0.0047491369999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.045077+0000","last_change":"2024-09-10T18:52:05.361338+0000","last_active":"2024-09-10T18:52:26.045077+0000","last_peered":"2024-09-10T18:52:26.045077+0000","last_clean":"2024-09-10T18:52:26.045077+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:52:26.045077+0000","last_undegraded":"2024-09-10T18:52:26.045077+0000","last_fullsized":"2024-09-10T18:52:26.045077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:05.361175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.361175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.175562+0000","scrub_duration":0.0039873549999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.963291+0000","last_change":"2024-09-10T18:52:45.821797+0000","last_active":"2024-09-10T18:52:45.963291+0000","last_peered":"2024-09-10T18:52:45.963291+0000","last_clean":"2024-09-10T18:52:45.963291+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:52:45.963291+0000","last_undegraded":"2024-09-10T18:52:45.963291+0000","last_fullsized":"2024-09-10T18:52:45.963291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:45.821628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:45.821628+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:12.908001+0000","scrub_duration":0.0045851850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"78'1","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894416+0000","last_change":"2024-09-10T18:51:32.830829+0000","last_active":"2024-09-10T18:52:25.894416+0000","last_peered":"2024-09-10T18:52:25.894416+0000","last_clean":"2024-09-10T18:52:25.894416+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:52:25.894416+0000","last_undegraded":"2024-09-10T18:52:25.894416+0000","last_fullsized":"2024-09-10T18:52:25.894416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-09-10T18:51:32.830654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:32.830654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.674413+0000","scrub_duration":0.0041675089999999998,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894826+0000","last_change":"2024-09-10T18:51:38.762020+0000","last_active":"2024-09-10T18:52:25.894826+0000","last_peered":"2024-09-10T18:52:25.894826+0000","last_clean":"2024-09-10T18:52:25.894826+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:52:25.894826+0000","last_undegraded":"2024-09-10T18:52:25.894826+0000","last_fullsized":"2024-09-10T18:52:25.894826+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.761859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.761859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:49.670551+0000","scrub_duration":0.003532154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"77'30","reported_seq":156,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.963584+0000","last_change":"2024-09-10T18:52:11.309315+0000","last_active":"2024-09-10T18:52:45.963584+0000","last_peered":"2024-09-10T18:52:45.963584+0000","last_clean":"2024-09-10T18:52:45.963584+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:52:45.963584+0000","last_undegraded":"2024-09-10T18:52:45.963584+0000","last_fullsized":"2024-09-10T18:52:45.963584+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-10T18:52:11.309152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.309152+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.549403+0000","scrub_duration":0.0055436330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044092+0000","last_change":"2024-09-10T18:52:03.183815+0000","last_active":"2024-09-10T18:52:26.044092+0000","last_peered":"2024-09-10T18:52:26.044092+0000","last_clean":"2024-09-10T18:52:26.044092+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:52:26.044092+0000","last_undegraded":"2024-09-10T18:52:26.044092+0000","last_fullsized":"2024-09-10T18:52:26.044092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.183646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.183646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.233272+0000","scrub_duration":0.0032583899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.963349+0000","last_change":"2024-09-10T18:52:07.501745+0000","last_active":"2024-09-10T18:52:45.963349+0000","last_peered":"2024-09-10T18:52:45.963349+0000","last_clean":"2024-09-10T18:52:45.963349+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:52:45.963349+0000","last_undegraded":"2024-09-10T18:52:45.963349+0000","last_fullsized":"2024-09-10T18:52:45.963349+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.501543+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.400864+0000","scrub_duration":0.0064925740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040200+0000","last_change":"2024-09-10T18:52:15.552956+0000","last_active":"2024-09-10T18:52:26.040200+0000","last_peered":"2024-09-10T18:52:26.040200+0000","last_clean":"2024-09-10T18:52:26.040200+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:52:26.040200+0000","last_undegraded":"2024-09-10T18:52:26.040200+0000","last_fullsized":"2024-09-10T18:52:26.040200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.552791+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:33.020117+0000","scrub_duration":0.0058227280000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:41.396875+0000","last_change":"2024-09-10T18:52:41.396637+0000","last_active":"2024-09-10T18:52:41.396875+0000","last_peered":"2024-09-10T18:52:41.396875+0000","last_clean":"2024-09-10T18:52:41.396875+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:52:41.396875+0000","last_undegraded":"2024-09-10T18:52:41.396875+0000","last_fullsized":"2024-09-10T18:52:41.396875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:41.396466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:41.396466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:09.624971+0000","scrub_duration":0.0033778850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":86,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.041649+0000","last_change":"2024-09-10T18:51:58.828562+0000","last_active":"2024-09-10T18:52:26.041649+0000","last_peered":"2024-09-10T18:52:26.041649+0000","last_clean":"2024-09-10T18:52:26.041649+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:52:26.041649+0000","last_undegraded":"2024-09-10T18:52:26.041649+0000","last_fullsized":"2024-09-10T18:52:26.041649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:58.828403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.828403+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.385475+0000","scrub_duration":0.0052576080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044094+0000","last_change":"2024-09-10T18:52:18.315632+0000","last_active":"2024-09-10T18:52:26.044094+0000","last_peered":"2024-09-10T18:52:26.044094+0000","last_clean":"2024-09-10T18:52:26.044094+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:52:26.044094+0000","last_undegraded":"2024-09-10T18:52:26.044094+0000","last_fullsized":"2024-09-10T18:52:26.044094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:18.315469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.315469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:29.767066+0000","scrub_duration":0.0035610590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:34.328561+0000","last_change":"2024-09-10T18:52:34.328450+0000","last_active":"2024-09-10T18:52:34.328561+0000","last_peered":"2024-09-10T18:52:34.328561+0000","last_clean":"2024-09-10T18:52:34.328561+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:52:34.328561+0000","last_undegraded":"2024-09-10T18:52:34.328561+0000","last_fullsized":"2024-09-10T18:52:34.328561+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:34.328284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:34.328284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.788608+0000","scrub_duration":0.0036424299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:37.626228+0000","last_change":"2024-09-10T18:52:37.626097+0000","last_active":"2024-09-10T18:52:37.626228+0000","last_peered":"2024-09-10T18:52:37.626228+0000","last_clean":"2024-09-10T18:52:37.626228+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:52:37.626228+0000","last_undegraded":"2024-09-10T18:52:37.626228+0000","last_fullsized":"2024-09-10T18:52:37.626228+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:37.625944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:37.625944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.595482+0000","scrub_duration":0.004216867,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":102,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895488+0000","last_change":"2024-09-10T18:52:15.209771+0000","last_active":"2024-09-10T18:52:25.895488+0000","last_peered":"2024-09-10T18:52:25.895488+0000","last_clean":"2024-09-10T18:52:25.895488+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:52:25.895488+0000","last_undegraded":"2024-09-10T18:52:25.895488+0000","last_fullsized":"2024-09-10T18:52:25.895488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:15.209609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.209609+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.019264+0000","scrub_duration":0.005193746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040855+0000","last_change":"2024-09-10T18:52:09.838167+0000","last_active":"2024-09-10T18:52:26.040855+0000","last_peered":"2024-09-10T18:52:26.040855+0000","last_clean":"2024-09-10T18:52:26.040855+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:52:26.040855+0000","last_undegraded":"2024-09-10T18:52:26.040855+0000","last_fullsized":"2024-09-10T18:52:26.040855+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.838015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.838015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.447360+0000","scrub_duration":0.003415202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043791+0000","last_change":"2024-09-10T18:52:09.323881+0000","last_active":"2024-09-10T18:52:26.043791+0000","last_peered":"2024-09-10T18:52:26.043791+0000","last_clean":"2024-09-10T18:52:26.043791+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:52:26.043791+0000","last_undegraded":"2024-09-10T18:52:26.043791+0000","last_fullsized":"2024-09-10T18:52:26.043791+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.323704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.323704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.895803+0000","scrub_duration":0.0036260730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":52,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900220+0000","last_change":"2024-09-10T18:51:39.639863+0000","last_active":"2024-09-10T18:52:25.900220+0000","last_peered":"2024-09-10T18:52:25.900220+0000","last_clean":"2024-09-10T18:52:25.900220+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:52:25.900220+0000","last_undegraded":"2024-09-10T18:52:25.900220+0000","last_fullsized":"2024-09-10T18:52:25.900220+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.639709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.639709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036283190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":104,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042323+0000","last_change":"2024-09-10T18:51:58.430973+0000","last_active":"2024-09-10T18:52:26.042323+0000","last_peered":"2024-09-10T18:52:26.042323+0000","last_clean":"2024-09-10T18:52:26.042323+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:52:26.042323+0000","last_undegraded":"2024-09-10T18:52:26.042323+0000","last_fullsized":"2024-09-10T18:52:26.042323+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.430868+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.088894+0000","scrub_duration":0.013718902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043152+0000","last_change":"2024-09-10T18:51:40.776447+0000","last_active":"2024-09-10T18:52:26.043152+0000","last_peered":"2024-09-10T18:52:26.043152+0000","last_clean":"2024-09-10T18:52:26.043152+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:52:26.043152+0000","last_undegraded":"2024-09-10T18:52:26.043152+0000","last_fullsized":"2024-09-10T18:52:26.043152+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:40.776169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.776169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.421742+0000","scrub_duration":0.0038302309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043707+0000","last_change":"2024-09-10T18:52:19.322071+0000","last_active":"2024-09-10T18:52:26.043707+0000","last_peered":"2024-09-10T18:52:26.043707+0000","last_clean":"2024-09-10T18:52:26.043707+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:52:26.043707+0000","last_undegraded":"2024-09-10T18:52:26.043707+0000","last_fullsized":"2024-09-10T18:52:26.043707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:19.321916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:19.321916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.895150+0000","scrub_duration":0.0033042869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042933+0000","last_change":"2024-09-10T18:51:45.221499+0000","last_active":"2024-09-10T18:52:26.042933+0000","last_peered":"2024-09-10T18:52:26.042933+0000","last_clean":"2024-09-10T18:52:26.042933+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:52:26.042933+0000","last_undegraded":"2024-09-10T18:52:26.042933+0000","last_fullsized":"2024-09-10T18:52:26.042933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:45.221333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:45.221333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.678342+0000","scrub_duration":0.0036649180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":87,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.897571+0000","last_change":"2024-09-10T18:51:51.619698+0000","last_active":"2024-09-10T18:52:25.897571+0000","last_peered":"2024-09-10T18:52:25.897571+0000","last_clean":"2024-09-10T18:52:25.897571+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:52:25.897571+0000","last_undegraded":"2024-09-10T18:52:25.897571+0000","last_fullsized":"2024-09-10T18:52:25.897571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:51.619511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:51.619511+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:03.627716+0000","scrub_duration":0.0049634149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046374+0000","last_change":"2024-09-10T18:52:06.149853+0000","last_active":"2024-09-10T18:52:26.046374+0000","last_peered":"2024-09-10T18:52:26.046374+0000","last_clean":"2024-09-10T18:52:26.046374+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:52:26.046374+0000","last_undegraded":"2024-09-10T18:52:26.046374+0000","last_fullsized":"2024-09-10T18:52:26.046374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.149699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.149699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.887082+0000","scrub_duration":0.0035198529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.899207+0000","last_change":"2024-09-10T18:51:50.626660+0000","last_active":"2024-09-10T18:52:25.899207+0000","last_peered":"2024-09-10T18:52:25.899207+0000","last_clean":"2024-09-10T18:52:25.899207+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:52:25.899207+0000","last_undegraded":"2024-09-10T18:52:25.899207+0000","last_fullsized":"2024-09-10T18:52:25.899207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.626514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.626514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.083006+0000","scrub_duration":0.0035366019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043276+0000","last_change":"2024-09-10T18:52:04.358732+0000","last_active":"2024-09-10T18:52:26.043276+0000","last_peered":"2024-09-10T18:52:26.043276+0000","last_clean":"2024-09-10T18:52:26.043276+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:52:26.043276+0000","last_undegraded":"2024-09-10T18:52:26.043276+0000","last_fullsized":"2024-09-10T18:52:26.043276+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:04.358415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:04.358415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:06.405553+0000","scrub_duration":0.003743377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"77'10","reported_seq":71,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.899378+0000","last_change":"2024-09-10T18:51:33.617562+0000","last_active":"2024-09-10T18:52:25.899378+0000","last_peered":"2024-09-10T18:52:25.899378+0000","last_clean":"2024-09-10T18:52:25.899378+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:52:25.899378+0000","last_undegraded":"2024-09-10T18:52:25.899378+0000","last_fullsized":"2024-09-10T18:52:25.899378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:33.617399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.617399+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.865090+0000","scrub_duration":0.0054712390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.039730+0000","last_change":"2024-09-10T18:51:39.623163+0000","last_active":"2024-09-10T18:52:26.039730+0000","last_peered":"2024-09-10T18:52:26.039730+0000","last_clean":"2024-09-10T18:52:26.039730+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:52:26.039730+0000","last_undegraded":"2024-09-10T18:52:26.039730+0000","last_fullsized":"2024-09-10T18:52:26.039730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.623005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.623005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.432652+0000","scrub_duration":0.0034195559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":114,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046669+0000","last_change":"2024-09-10T18:51:33.452673+0000","last_active":"2024-09-10T18:52:26.046669+0000","last_peered":"2024-09-10T18:52:26.046669+0000","last_clean":"2024-09-10T18:52:26.046669+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:52:26.046669+0000","last_undegraded":"2024-09-10T18:52:26.046669+0000","last_fullsized":"2024-09-10T18:52:26.046669+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:51:33.452504+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.452504+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.436046+0000","scrub_duration":0.0041999869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043044+0000","last_change":"2024-09-10T18:52:03.329811+0000","last_active":"2024-09-10T18:52:26.043044+0000","last_peered":"2024-09-10T18:52:26.043044+0000","last_clean":"2024-09-10T18:52:26.043044+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:52:26.043044+0000","last_undegraded":"2024-09-10T18:52:26.043044+0000","last_fullsized":"2024-09-10T18:52:26.043044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.329632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.640377+0000","scrub_duration":0.0032347410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042024+0000","last_change":"2024-09-10T18:52:21.732959+0000","last_active":"2024-09-10T18:52:26.042024+0000","last_peered":"2024-09-10T18:52:26.042024+0000","last_clean":"2024-09-10T18:52:26.042024+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:52:26.042024+0000","last_undegraded":"2024-09-10T18:52:26.042024+0000","last_fullsized":"2024-09-10T18:52:26.042024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.732812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.732812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.740200+0000","scrub_duration":0.0034512259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":92,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:27.343613+0000","last_change":"2024-09-10T18:52:27.343427+0000","last_active":"2024-09-10T18:52:27.343613+0000","last_peered":"2024-09-10T18:52:27.343613+0000","last_clean":"2024-09-10T18:52:27.343613+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:52:27.343613+0000","last_undegraded":"2024-09-10T18:52:27.343613+0000","last_fullsized":"2024-09-10T18:52:27.343613+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:52:27.343260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:27.343260+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.393322+0000","scrub_duration":0.005128049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043829+0000","last_change":"2024-09-10T18:52:07.455227+0000","last_active":"2024-09-10T18:52:26.043829+0000","last_peered":"2024-09-10T18:52:26.043829+0000","last_clean":"2024-09-10T18:52:26.043829+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:52:26.043829+0000","last_undegraded":"2024-09-10T18:52:26.043829+0000","last_fullsized":"2024-09-10T18:52:26.043829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.455062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.455062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.587408+0000","scrub_duration":0.0037580970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040804+0000","last_change":"2024-09-10T18:51:52.321821+0000","last_active":"2024-09-10T18:52:26.040804+0000","last_peered":"2024-09-10T18:52:26.040804+0000","last_clean":"2024-09-10T18:52:26.040804+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:52:26.040804+0000","last_undegraded":"2024-09-10T18:52:26.040804+0000","last_fullsized":"2024-09-10T18:52:26.040804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:52.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:52.321671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:02.059416+0000","scrub_duration":0.003316977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:40.442420+0000","last_change":"2024-09-10T18:52:40.442208+0000","last_active":"2024-09-10T18:52:40.442420+0000","last_peered":"2024-09-10T18:52:40.442420+0000","last_clean":"2024-09-10T18:52:40.442420+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:52:40.442420+0000","last_undegraded":"2024-09-10T18:52:40.442420+0000","last_fullsized":"2024-09-10T18:52:40.442420+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:40.442034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:40.442034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.415342+0000","scrub_duration":0.0034431399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"77'10","reported_seq":79,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:29.670961+0000","last_change":"2024-09-10T18:52:29.670736+0000","last_active":"2024-09-10T18:52:29.670961+0000","last_peered":"2024-09-10T18:52:29.670961+0000","last_clean":"2024-09-10T18:52:29.670961+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:52:29.670961+0000","last_undegraded":"2024-09-10T18:52:29.670961+0000","last_fullsized":"2024-09-10T18:52:29.670961+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:29.670555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.670555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.035420+0000","scrub_duration":0.0047380190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:29.292230+0000","last_change":"2024-09-10T18:52:29.292088+0000","last_active":"2024-09-10T18:52:29.292230+0000","last_peered":"2024-09-10T18:52:29.292230+0000","last_clean":"2024-09-10T18:52:29.292230+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:52:29.292230+0000","last_undegraded":"2024-09-10T18:52:29.292230+0000","last_fullsized":"2024-09-10T18:52:29.292230+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:29.291936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.291936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:55.077335+0000","scrub_duration":0.003409966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:29.591810+0000","last_change":"2024-09-10T18:52:29.591674+0000","last_active":"2024-09-10T18:52:29.591810+0000","last_peered":"2024-09-10T18:52:29.591810+0000","last_clean":"2024-09-10T18:52:29.591810+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:52:29.591810+0000","last_undegraded":"2024-09-10T18:52:29.591810+0000","last_fullsized":"2024-09-10T18:52:29.591810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:29.591530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.591530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.371798+0000","scrub_duration":0.003270528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.897497+0000","last_change":"2024-09-10T18:52:25.583663+0000","last_active":"2024-09-10T18:52:25.897497+0000","last_peered":"2024-09-10T18:52:25.897497+0000","last_clean":"2024-09-10T18:52:25.897497+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:52:25.897497+0000","last_undegraded":"2024-09-10T18:52:25.897497+0000","last_fullsized":"2024-09-10T18:52:25.897497+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.583492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.226811+0000","scrub_duration":0.0033363329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":96,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042029+0000","last_change":"2024-09-10T18:51:47.304256+0000","last_active":"2024-09-10T18:52:26.042029+0000","last_peered":"2024-09-10T18:52:26.042029+0000","last_clean":"2024-09-10T18:52:26.042029+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:52:26.042029+0000","last_undegraded":"2024-09-10T18:52:26.042029+0000","last_fullsized":"2024-09-10T18:52:26.042029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.304153+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.010387+0000","scrub_duration":0.012604194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":78,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044325+0000","last_change":"2024-09-10T18:51:53.688495+0000","last_active":"2024-09-10T18:52:26.044325+0000","last_peered":"2024-09-10T18:52:26.044325+0000","last_clean":"2024-09-10T18:52:26.044325+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:52:26.044325+0000","last_undegraded":"2024-09-10T18:52:26.044325+0000","last_fullsized":"2024-09-10T18:52:26.044325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:51:53.688302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.688302+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.064936+0000","scrub_duration":0.0040928329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.039141+0000","last_change":"2024-09-10T18:51:46.633977+0000","last_active":"2024-09-10T18:52:26.039141+0000","last_peered":"2024-09-10T18:52:26.039141+0000","last_clean":"2024-09-10T18:52:26.039141+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:52:26.039141+0000","last_undegraded":"2024-09-10T18:52:26.039141+0000","last_fullsized":"2024-09-10T18:52:26.039141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.633831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:56.754683+0000","scrub_duration":0.003224747,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.039974+0000","last_change":"2024-09-10T18:51:56.751545+0000","last_active":"2024-09-10T18:52:26.039974+0000","last_peered":"2024-09-10T18:52:26.039974+0000","last_clean":"2024-09-10T18:52:26.039974+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:52:26.039974+0000","last_undegraded":"2024-09-10T18:52:26.039974+0000","last_fullsized":"2024-09-10T18:52:26.039974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.751398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.751398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.229956+0000","scrub_duration":0.0032004590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":83,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046960+0000","last_change":"2024-09-10T18:52:05.179430+0000","last_active":"2024-09-10T18:52:26.046960+0000","last_peered":"2024-09-10T18:52:26.046960+0000","last_clean":"2024-09-10T18:52:26.046960+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:52:26.046960+0000","last_undegraded":"2024-09-10T18:52:26.046960+0000","last_fullsized":"2024-09-10T18:52:26.046960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:05.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.179207+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.065476+0000","scrub_duration":0.0057240640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":107,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.896640+0000","last_change":"2024-09-10T18:51:43.700454+0000","last_active":"2024-09-10T18:52:25.896640+0000","last_peered":"2024-09-10T18:52:25.896640+0000","last_clean":"2024-09-10T18:52:25.896640+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:52:25.896640+0000","last_undegraded":"2024-09-10T18:52:25.896640+0000","last_fullsized":"2024-09-10T18:52:25.896640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:51:43.700290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:43.700290+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:51.352626+0000","scrub_duration":0.0033939529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.960415+0000","last_change":"2024-09-10T18:51:47.570557+0000","last_active":"2024-09-10T18:52:45.960415+0000","last_peered":"2024-09-10T18:52:45.960415+0000","last_clean":"2024-09-10T18:52:45.960415+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:52:45.960415+0000","last_undegraded":"2024-09-10T18:52:45.960415+0000","last_fullsized":"2024-09-10T18:52:45.960415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:47.570369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.570369+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.658418+0000","scrub_duration":0.0040413760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:40.755330+0000","last_change":"2024-09-10T18:52:40.755231+0000","last_active":"2024-09-10T18:52:40.755330+0000","last_peered":"2024-09-10T18:52:40.755330+0000","last_clean":"2024-09-10T18:52:40.755330+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:52:40.755330+0000","last_undegraded":"2024-09-10T18:52:40.755330+0000","last_fullsized":"2024-09-10T18:52:40.755330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:52:40.755118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:54.859229+0000","scrub_duration":0.0030712019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":83,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.046822+0000","last_change":"2024-09-10T18:51:29.406090+0000","last_active":"2024-09-10T18:52:26.046822+0000","last_peered":"2024-09-10T18:52:26.046822+0000","last_clean":"2024-09-10T18:52:26.046822+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:52:26.046822+0000","last_undegraded":"2024-09-10T18:52:26.046822+0000","last_fullsized":"2024-09-10T18:52:26.046822+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:29.405903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.405903+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052412719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042639+0000","last_change":"2024-09-10T18:51:59.561431+0000","last_active":"2024-09-10T18:52:26.042639+0000","last_peered":"2024-09-10T18:52:26.042639+0000","last_clean":"2024-09-10T18:52:26.042639+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:52:26.042639+0000","last_undegraded":"2024-09-10T18:52:26.042639+0000","last_fullsized":"2024-09-10T18:52:26.042639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_clean_scrub_stamp":"2024-09-10T18:51:59.561306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.793374+0000","scrub_duration":0.003474277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.899033+0000","last_change":"2024-09-10T18:52:01.485093+0000","last_active":"2024-09-10T18:52:25.899033+0000","last_peered":"2024-09-10T18:52:25.899033+0000","last_clean":"2024-09-10T18:52:25.899033+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:52:25.899033+0000","last_undegraded":"2024-09-10T18:52:25.899033+0000","last_fullsized":"2024-09-10T18:52:25.899033+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.484734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.484734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.349048+0000","scrub_duration":0.0054940589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043402+0000","last_change":"2024-09-10T18:52:11.451951+0000","last_active":"2024-09-10T18:52:26.043402+0000","last_peered":"2024-09-10T18:52:26.043402+0000","last_clean":"2024-09-10T18:52:26.043402+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:52:26.043402+0000","last_undegraded":"2024-09-10T18:52:26.043402+0000","last_fullsized":"2024-09-10T18:52:26.043402+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.451797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.451797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:26.258751+0000","scrub_duration":0.0033137470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"77'4","reported_seq":70,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040654+0000","last_change":"2024-09-10T18:52:11.496842+0000","last_active":"2024-09-10T18:52:26.040654+0000","last_peered":"2024-09-10T18:52:26.040654+0000","last_clean":"2024-09-10T18:52:26.040654+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:52:26.040654+0000","last_undegraded":"2024-09-10T18:52:26.040654+0000","last_fullsized":"2024-09-10T18:52:26.040654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-10T18:52:11.496670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.496670+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:24.908807+0000","scrub_duration":0.0041783929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.041943+0000","last_change":"2024-09-10T18:52:00.390577+0000","last_active":"2024-09-10T18:52:26.041943+0000","last_peered":"2024-09-10T18:52:26.041943+0000","last_clean":"2024-09-10T18:52:26.041943+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:52:26.041943+0000","last_undegraded":"2024-09-10T18:52:26.041943+0000","last_fullsized":"2024-09-10T18:52:26.041943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:00.390415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.390415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.045571+0000","scrub_duration":0.00325318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:46.829850+0000","last_change":"2024-09-10T18:52:46.829741+0000","last_active":"2024-09-10T18:52:46.829850+0000","last_peered":"2024-09-10T18:52:46.829850+0000","last_clean":"2024-09-10T18:52:46.829850+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:52:46.829850+0000","last_undegraded":"2024-09-10T18:52:46.829850+0000","last_fullsized":"2024-09-10T18:52:46.829850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:52:46.829507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:55.683255+0000","scrub_duration":0.0045636449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042699+0000","last_change":"2024-09-10T18:52:07.197975+0000","last_active":"2024-09-10T18:52:26.042699+0000","last_peered":"2024-09-10T18:52:26.042699+0000","last_clean":"2024-09-10T18:52:26.042699+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:52:26.042699+0000","last_undegraded":"2024-09-10T18:52:26.042699+0000","last_fullsized":"2024-09-10T18:52:26.042699+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.197730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.197730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.782447+0000","scrub_duration":0.003470014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":88,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:48.328934+0000","last_change":"2024-09-10T18:52:48.328798+0000","last_active":"2024-09-10T18:52:48.328934+0000","last_peered":"2024-09-10T18:52:48.328934+0000","last_clean":"2024-09-10T18:52:48.328934+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:52:48.328934+0000","last_undegraded":"2024-09-10T18:52:48.328934+0000","last_fullsized":"2024-09-10T18:52:48.328934+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:52:48.328639+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.175263+0000","scrub_duration":0.010391842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.617312+0000","last_change":"2024-09-10T18:52:26.617200+0000","last_active":"2024-09-10T18:52:26.617312+0000","last_peered":"2024-09-10T18:52:26.617312+0000","last_clean":"2024-09-10T18:52:26.617312+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:52:26.617312+0000","last_undegraded":"2024-09-10T18:52:26.617312+0000","last_fullsized":"2024-09-10T18:52:26.617312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:26.617026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:26.617026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:29.957670+0000","scrub_duration":0.0041383920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":119,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.964094+0000","last_change":"2024-09-10T18:52:24.727019+0000","last_active":"2024-09-10T18:52:45.964094+0000","last_peered":"2024-09-10T18:52:45.964094+0000","last_clean":"2024-09-10T18:52:45.964094+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:52:45.964094+0000","last_undegraded":"2024-09-10T18:52:45.964094+0000","last_fullsized":"2024-09-10T18:52:45.964094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:24.726861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.726861+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.604127+0000","scrub_duration":0.0039722289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:33.564561+0000","last_change":"2024-09-10T18:52:33.564446+0000","last_active":"2024-09-10T18:52:33.564561+0000","last_peered":"2024-09-10T18:52:33.564561+0000","last_clean":"2024-09-10T18:52:33.564561+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:52:33.564561+0000","last_undegraded":"2024-09-10T18:52:33.564561+0000","last_fullsized":"2024-09-10T18:52:33.564561+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:33.564283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:52:33.564283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:35.408845+0000","scrub_duration":0.0035323889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":82,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:36.577153+0000","last_change":"2024-09-10T18:52:36.577038+0000","last_active":"2024-09-10T18:52:36.577153+0000","last_peered":"2024-09-10T18:52:36.577153+0000","last_clean":"2024-09-10T18:52:36.577153+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:52:36.577153+0000","last_undegraded":"2024-09-10T18:52:36.577153+0000","last_fullsized":"2024-09-10T18:52:36.577153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:36.576871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:36.576871+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:37.551929+0000","scrub_duration":0.0043998120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.898119+0000","last_change":"2024-09-10T18:51:56.575310+0000","last_active":"2024-09-10T18:52:25.898119+0000","last_peered":"2024-09-10T18:52:25.898119+0000","last_clean":"2024-09-10T18:52:25.898119+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:52:25.898119+0000","last_undegraded":"2024-09-10T18:52:25.898119+0000","last_fullsized":"2024-09-10T18:52:25.898119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.575153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.564777+0000","scrub_duration":0.0033985280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-10T18:52:49.570093+0000","last_change":"2024-09-10T18:52:49.569965+0000","last_active":"2024-09-10T18:52:49.570093+0000","last_peered":"2024-09-10T18:52:49.570093+0000","last_clean":"2024-09-10T18:52:49.570093+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:52:49.570093+0000","last_undegraded":"2024-09-10T18:52:49.570093+0000","last_fullsized":"2024-09-10T18:52:49.570093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:49.569813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:49.569813+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:03.970871+0000","scrub_duration":0.0048404019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.896330+0000","last_change":"2024-09-10T18:52:22.188985+0000","last_active":"2024-09-10T18:52:25.896330+0000","last_peered":"2024-09-10T18:52:25.896330+0000","last_clean":"2024-09-10T18:52:25.896330+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:52:25.896330+0000","last_undegraded":"2024-09-10T18:52:25.896330+0000","last_fullsized":"2024-09-10T18:52:25.896330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.188834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.188834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.440531+0000","scrub_duration":0.0068943040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":80,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043371+0000","last_change":"2024-09-10T18:52:08.245784+0000","last_active":"2024-09-10T18:52:26.043371+0000","last_peered":"2024-09-10T18:52:26.043371+0000","last_clean":"2024-09-10T18:52:26.043371+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:52:26.043371+0000","last_undegraded":"2024-09-10T18:52:26.043371+0000","last_fullsized":"2024-09-10T18:52:26.043371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:08.245617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:08.245617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.667622+0000","scrub_duration":0.0044282050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043064+0000","last_change":"2024-09-10T18:52:25.757066+0000","last_active":"2024-09-10T18:52:26.043064+0000","last_peered":"2024-09-10T18:52:26.043064+0000","last_clean":"2024-09-10T18:52:26.043064+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:52:26.043064+0000","last_undegraded":"2024-09-10T18:52:26.043064+0000","last_fullsized":"2024-09-10T18:52:26.043064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.756907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.756907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.523280+0000","scrub_duration":0.0032708659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":53,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.039582+0000","last_change":"2024-09-10T18:52:11.856507+0000","last_active":"2024-09-10T18:52:26.039582+0000","last_peered":"2024-09-10T18:52:26.039582+0000","last_clean":"2024-09-10T18:52:26.039582+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:52:26.039582+0000","last_undegraded":"2024-09-10T18:52:26.039582+0000","last_fullsized":"2024-09-10T18:52:26.039582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.856337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.856337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.241036+0000","scrub_duration":0.0034053619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.896111+0000","last_change":"2024-09-10T18:52:14.538993+0000","last_active":"2024-09-10T18:52:25.896111+0000","last_peered":"2024-09-10T18:52:25.896111+0000","last_clean":"2024-09-10T18:52:25.896111+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:52:25.896111+0000","last_undegraded":"2024-09-10T18:52:25.896111+0000","last_fullsized":"2024-09-10T18:52:25.896111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:14.538838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:14.538838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.876083+0000","scrub_duration":0.0042549700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"77'10","reported_seq":81,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895822+0000","last_change":"2024-09-10T18:52:11.222775+0000","last_active":"2024-09-10T18:52:25.895822+0000","last_peered":"2024-09-10T18:52:25.895822+0000","last_clean":"2024-09-10T18:52:25.895822+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:52:25.895822+0000","last_undegraded":"2024-09-10T18:52:25.895822+0000","last_fullsized":"2024-09-10T18:52:25.895822+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:11.222598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.222598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.658086+0000","scrub_duration":0.0051546400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:38.373087+0000","last_change":"2024-09-10T18:52:38.373031+0000","last_active":"2024-09-10T18:52:38.373087+0000","last_peered":"2024-09-10T18:52:38.373087+0000","last_clean":"2024-09-10T18:52:38.373087+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:52:38.373087+0000","last_undegraded":"2024-09-10T18:52:38.373087+0000","last_fullsized":"2024-09-10T18:52:38.373087+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:38.372968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:38.372968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:56.815767+0000","scrub_duration":0.0025253519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.894976+0000","last_change":"2024-09-10T18:52:20.152534+0000","last_active":"2024-09-10T18:52:25.894976+0000","last_peered":"2024-09-10T18:52:25.894976+0000","last_clean":"2024-09-10T18:52:25.894976+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:52:25.894976+0000","last_undegraded":"2024-09-10T18:52:25.894976+0000","last_fullsized":"2024-09-10T18:52:25.894976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.152369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:47.436481+0000","scrub_duration":0.0033349780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.897588+0000","last_change":"2024-09-10T18:51:57.533423+0000","last_active":"2024-09-10T18:52:25.897588+0000","last_peered":"2024-09-10T18:52:25.897588+0000","last_clean":"2024-09-10T18:52:25.897588+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:52:25.897588+0000","last_undegraded":"2024-09-10T18:52:25.897588+0000","last_fullsized":"2024-09-10T18:52:25.897588+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:57.533281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.533281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.018032+0000","scrub_duration":0.0030396329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":78,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:39.624203+0000","last_change":"2024-09-10T18:52:39.623937+0000","last_active":"2024-09-10T18:52:39.624203+0000","last_peered":"2024-09-10T18:52:39.624203+0000","last_clean":"2024-09-10T18:52:39.624203+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:52:39.624203+0000","last_undegraded":"2024-09-10T18:52:39.624203+0000","last_fullsized":"2024-09-10T18:52:39.624203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:52:39.623723+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.931489+0000","scrub_duration":0.0077034729999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:32.647253+0000","last_change":"2024-09-10T18:52:32.647147+0000","last_active":"2024-09-10T18:52:32.647253+0000","last_peered":"2024-09-10T18:52:32.647253+0000","last_clean":"2024-09-10T18:52:32.647253+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:52:32.647253+0000","last_undegraded":"2024-09-10T18:52:32.647253+0000","last_fullsized":"2024-09-10T18:52:32.647253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:32.646982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:32.646982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:44.711757+0000","scrub_duration":0.0033406109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":54,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.899808+0000","last_change":"2024-09-10T18:51:38.650934+0000","last_active":"2024-09-10T18:52:25.899808+0000","last_peered":"2024-09-10T18:52:25.899808+0000","last_clean":"2024-09-10T18:52:25.899808+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:52:25.899808+0000","last_undegraded":"2024-09-10T18:52:25.899808+0000","last_fullsized":"2024-09-10T18:52:25.899808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.650775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.247838+0000","scrub_duration":0.0033420289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.041066+0000","last_change":"2024-09-10T18:51:37.345540+0000","last_active":"2024-09-10T18:52:26.041066+0000","last_peered":"2024-09-10T18:52:26.041066+0000","last_clean":"2024-09-10T18:52:26.041066+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:52:26.041066+0000","last_undegraded":"2024-09-10T18:52:26.041066+0000","last_fullsized":"2024-09-10T18:52:26.041066+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:37.345379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.345379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:55.645490+0000","scrub_duration":0.0032740450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":80,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.040250+0000","last_change":"2024-09-10T18:52:22.698451+0000","last_active":"2024-09-10T18:52:26.040250+0000","last_peered":"2024-09-10T18:52:26.040250+0000","last_clean":"2024-09-10T18:52:26.040250+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:52:26.040250+0000","last_undegraded":"2024-09-10T18:52:26.040250+0000","last_fullsized":"2024-09-10T18:52:26.040250+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:22.698371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.698371+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:46.745548+0000","scrub_duration":0.0094695119999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.044369+0000","last_change":"2024-09-10T18:51:49.224075+0000","last_active":"2024-09-10T18:52:26.044369+0000","last_peered":"2024-09-10T18:52:26.044369+0000","last_clean":"2024-09-10T18:52:26.044369+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:52:26.044369+0000","last_undegraded":"2024-09-10T18:52:26.044369+0000","last_fullsized":"2024-09-10T18:52:26.044369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:49.223905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:49.223905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.495696+0000","scrub_duration":0.003346421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:38.759696+0000","last_change":"2024-09-10T18:52:38.759578+0000","last_active":"2024-09-10T18:52:38.759696+0000","last_peered":"2024-09-10T18:52:38.759696+0000","last_clean":"2024-09-10T18:52:38.759696+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:52:38.759696+0000","last_undegraded":"2024-09-10T18:52:38.759696+0000","last_fullsized":"2024-09-10T18:52:38.759696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:38.759381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:38.759381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.210016+0000","scrub_duration":0.0037681210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:32.749974+0000","last_change":"2024-09-10T18:52:32.749810+0000","last_active":"2024-09-10T18:52:32.749974+0000","last_peered":"2024-09-10T18:52:32.749974+0000","last_clean":"2024-09-10T18:52:32.749974+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:52:32.749974+0000","last_undegraded":"2024-09-10T18:52:32.749974+0000","last_fullsized":"2024-09-10T18:52:32.749974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:32.749566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:32.749566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:46.056687+0000","scrub_duration":0.0042025500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:44.380603+0000","last_change":"2024-09-10T18:52:44.380396+0000","last_active":"2024-09-10T18:52:44.380603+0000","last_peered":"2024-09-10T18:52:44.380603+0000","last_clean":"2024-09-10T18:52:44.380603+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:52:44.380603+0000","last_undegraded":"2024-09-10T18:52:44.380603+0000","last_fullsized":"2024-09-10T18:52:44.380603+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:52:44.380117+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:00.476762+0000","scrub_duration":0.0083087079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.042654+0000","last_change":"2024-09-10T18:51:50.544356+0000","last_active":"2024-09-10T18:52:26.042654+0000","last_peered":"2024-09-10T18:52:26.042654+0000","last_clean":"2024-09-10T18:52:26.042654+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:52:26.042654+0000","last_undegraded":"2024-09-10T18:52:26.042654+0000","last_fullsized":"2024-09-10T18:52:26.042654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:50.544186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.544186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.788233+0000","scrub_duration":0.0064898330000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":66,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.039394+0000","last_change":"2024-09-10T18:52:15.859394+0000","last_active":"2024-09-10T18:52:26.039394+0000","last_peered":"2024-09-10T18:52:26.039394+0000","last_clean":"2024-09-10T18:52:26.039394+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:52:26.039394+0000","last_undegraded":"2024-09-10T18:52:26.039394+0000","last_fullsized":"2024-09-10T18:52:26.039394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:15.859239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.859239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.490895+0000","scrub_duration":0.0037198740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.898961+0000","last_change":"2024-09-10T18:52:24.615123+0000","last_active":"2024-09-10T18:52:25.898961+0000","last_peered":"2024-09-10T18:52:25.898961+0000","last_clean":"2024-09-10T18:52:25.898961+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:52:25.898961+0000","last_undegraded":"2024-09-10T18:52:25.898961+0000","last_fullsized":"2024-09-10T18:52:25.898961+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:24.614967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.614967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:40.011991+0000","scrub_duration":0.0037677969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":60,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:31.350793+0000","last_change":"2024-09-10T18:52:31.350686+0000","last_active":"2024-09-10T18:52:31.350793+0000","last_peered":"2024-09-10T18:52:31.350793+0000","last_clean":"2024-09-10T18:52:31.350793+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:52:31.350793+0000","last_undegraded":"2024-09-10T18:52:31.350793+0000","last_fullsized":"2024-09-10T18:52:31.350793+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:31.350539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.350539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:37.972404+0000","scrub_duration":0.003698621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":72,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900305+0000","last_change":"2024-09-10T18:51:28.596878+0000","last_active":"2024-09-10T18:52:25.900305+0000","last_peered":"2024-09-10T18:52:25.900305+0000","last_clean":"2024-09-10T18:52:25.900305+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:52:25.900305+0000","last_undegraded":"2024-09-10T18:52:25.900305+0000","last_fullsized":"2024-09-10T18:52:25.900305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:28.596672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.596672+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:50.156773+0000","scrub_duration":0.0055701980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:28.733614+0000","last_change":"2024-09-10T18:52:28.733481+0000","last_active":"2024-09-10T18:52:28.733614+0000","last_peered":"2024-09-10T18:52:28.733614+0000","last_clean":"2024-09-10T18:52:28.733614+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:52:28.733614+0000","last_undegraded":"2024-09-10T18:52:28.733614+0000","last_fullsized":"2024-09-10T18:52:28.733614+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:28.733333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.733333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:51.670048+0000","scrub_duration":0.0031761739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":131,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:45.960346+0000","last_change":"2024-09-10T18:51:55.299569+0000","last_active":"2024-09-10T18:52:45.960346+0000","last_peered":"2024-09-10T18:52:45.960346+0000","last_clean":"2024-09-10T18:52:45.960346+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:52:45.960346+0000","last_undegraded":"2024-09-10T18:52:45.960346+0000","last_fullsized":"2024-09-10T18:52:45.960346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:51:55.299396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.299396+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.997434+0000","scrub_duration":0.0038567179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895871+0000","last_change":"2024-09-10T18:51:41.706112+0000","last_active":"2024-09-10T18:52:25.895871+0000","last_peered":"2024-09-10T18:52:25.895871+0000","last_clean":"2024-09-10T18:52:25.895871+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:52:25.895871+0000","last_undegraded":"2024-09-10T18:52:25.895871+0000","last_fullsized":"2024-09-10T18:52:25.895871+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:51:41.705954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:52.045292+0000","scrub_duration":0.003398917,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":75,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.041349+0000","last_change":"2024-09-10T18:52:18.566245+0000","last_active":"2024-09-10T18:52:26.041349+0000","last_peered":"2024-09-10T18:52:26.041349+0000","last_clean":"2024-09-10T18:52:26.041349+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:52:26.041349+0000","last_undegraded":"2024-09-10T18:52:26.041349+0000","last_fullsized":"2024-09-10T18:52:26.041349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:52:18.566056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.566056+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.529822+0000","scrub_duration":0.0040203510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":76,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.897983+0000","last_change":"2024-09-10T18:52:09.509564+0000","last_active":"2024-09-10T18:52:25.897983+0000","last_peered":"2024-09-10T18:52:25.897983+0000","last_clean":"2024-09-10T18:52:25.897983+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:52:25.897983+0000","last_undegraded":"2024-09-10T18:52:25.897983+0000","last_fullsized":"2024-09-10T18:52:25.897983+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:52:09.509384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.509384+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.310603+0000","scrub_duration":0.0050191510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:26.043624+0000","last_change":"2024-09-10T18:51:56.652829+0000","last_active":"2024-09-10T18:52:26.043624+0000","last_peered":"2024-09-10T18:52:26.043624+0000","last_clean":"2024-09-10T18:52:26.043624+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:52:26.043624+0000","last_undegraded":"2024-09-10T18:52:26.043624+0000","last_fullsized":"2024-09-10T18:52:26.043624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.652676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.654243+0000","scrub_duration":0.003518115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":62,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900562+0000","last_change":"2024-09-10T18:52:06.609575+0000","last_active":"2024-09-10T18:52:25.900562+0000","last_peered":"2024-09-10T18:52:25.900562+0000","last_clean":"2024-09-10T18:52:25.900562+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:52:25.900562+0000","last_undegraded":"2024-09-10T18:52:25.900562+0000","last_fullsized":"2024-09-10T18:52:25.900562+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.609346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.609346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.679962+0000","scrub_duration":0.0031752439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":68,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.900142+0000","last_change":"2024-09-10T18:51:42.590260+0000","last_active":"2024-09-10T18:52:25.900142+0000","last_peered":"2024-09-10T18:52:25.900142+0000","last_clean":"2024-09-10T18:52:25.900142+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:52:25.900142+0000","last_undegraded":"2024-09-10T18:52:25.900142+0000","last_fullsized":"2024-09-10T18:52:25.900142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:42.590071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:42.590071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.690989+0000","scrub_duration":0.003624726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":82,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:25.895190+0000","last_change":"2024-09-10T18:51:57.420671+0000","last_active":"2024-09-10T18:52:25.895190+0000","last_peered":"2024-09-10T18:52:25.895190+0000","last_clean":"2024-09-10T18:52:25.895190+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:52:25.895190+0000","last_undegraded":"2024-09-10T18:52:25.895190+0000","last_fullsized":"2024-09-10T18:52:25.895190+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:57.420489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.420489+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.697937+0000","scrub_duration":0.0042932839999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":64,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-10T18:52:35.715917+0000","last_change":"2024-09-10T18:52:35.715864+0000","last_active":"2024-09-10T18:52:35.715917+0000","last_peered":"2024-09-10T18:52:35.715917+0000","last_clean":"2024-09-10T18:52:35.715917+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:52:35.715917+0000","last_undegraded":"2024-09-10T18:52:35.715917+0000","last_fullsized":"2024-09-10T18:52:35.715917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:35.715788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:35.715788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:56.123687+0000","scrub_duration":0.0023162970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":188,"num_read_kb":183,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972406,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446488,"statfs":{"total":95995035648,"available":95689203712,"internally_reserved":0,"allocated":1339392,"data_stored":940356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91600000000000004,"15min":0.93500000000000005},"min":{"1min":0.621,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3380000000000001,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.022,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.69399999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.264,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":1.1499999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.91400000000000003,"15min":0.91000000000000003},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.373,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.0109999999999999,"15min":1.046},"min":{"1min":0.65400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.409,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.72299999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.0089999999999999,"15min":1.0549999999999999},"min":{"1min":0.77200000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3779999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.073},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.69799999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.236,"5min":1.353,"15min":1.353},"last":1.0429999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98499999999999999,"15min":1.0169999999999999},"min":{"1min":0.70699999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3169999999999999,"5min":1.335,"15min":1.335},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.97199999999999998,"15min":0.96299999999999997},"min":{"1min":0.73599999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3280000000000001,"5min":1.905,"15min":1.905},"last":0.96499999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.97899999999999998,"15min":1.0189999999999999},"min":{"1min":0.65800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.331,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.0269999999999999,"15min":1.0209999999999999},"min":{"1min":0.83299999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2749999999999999,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.84499999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98899999999999999,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2370000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1479999999999999,"15min":1.1899999999999999},"min":{"1min":0.77200000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.448,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.2410000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2490000000000001},"min":{"1min":0.875,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4950000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1919999999999999},"min":{"1min":0.749,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3520000000000001,"5min":2.536,"15min":2.536},"last":1.266}]}]},{"osd":6,"up_from":52,"seq":223338299456,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298644,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446508,"statfs":{"total":95995035648,"available":95689224192,"internally_reserved":0,"allocated":1318912,"data_stored":938595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.55700000000000005,"5min":0.503,"15min":0.503},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.028,"5min":0.94099999999999995,"15min":0.93500000000000005},"min":{"1min":0.64800000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.1419999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.92700000000000005,"15min":0.88800000000000001},"min":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.226,"5min":1.361,"15min":1.361},"last":1.1080000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.94399999999999995,"15min":0.91900000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.177},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.91800000000000004,"15min":0.92700000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.91200000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96899999999999997,"15min":0.94699999999999995},"min":{"1min":0.73899999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.2869999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.008,"5min":0.96599999999999997,"15min":0.97099999999999997},"min":{"1min":0.71399999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.294,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.85799999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.85999999999999999},"min":{"1min":0.61399999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0029999999999999,"15min":1.0049999999999999},"min":{"1min":0.63800000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.353,"5min":1.744,"15min":1.744},"last":0.66600000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.997,"15min":0.96899999999999997},"min":{"1min":0.71599999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.0780000000000001,"15min":1.0840000000000001},"min":{"1min":0.76400000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4259999999999999,"5min":1.665,"15min":1.665},"last":0.93400000000000005}]},{"osd":7,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.054,"15min":1.0389999999999999},"min":{"1min":0.77200000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2889999999999999,"5min":1.573,"15min":1.573},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.149,"15min":1.139},"min":{"1min":0.98399999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.405,"5min":1.488,"15min":1.488},"last":1.0649999999999999}]}]},{"osd":5,"up_from":43,"seq":184683593802,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298052,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447100,"statfs":{"total":95995035648,"available":95689830400,"internally_reserved":0,"allocated":716800,"data_stored":341780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.88800000000000001,"15min":0.83999999999999997},"min":{"1min":0.66200000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.258,"5min":1.609,"15min":1.609},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1990000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.1279999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.86899999999999999,"15min":0.85499999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2949999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.91700000000000004,"15min":0.88100000000000001},"min":{"1min":0.70699999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2230000000000001,"5min":1.635,"15min":1.635},"last":1.2769999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91500000000000004,"15min":0.88500000000000001},"min":{"1min":0.628,"5min":0.496,"15min":0.496},"max":{"1min":1.274,"5min":1.496,"15min":1.496},"last":1.2529999999999999},{"interface":"front","average":{"1min":0.999,"5min":0.93899999999999995,"15min":0.89400000000000002},"min":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3180000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.752}]},{"osd":3,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.64900000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.242,"5min":1.409,"15min":1.409},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.69199999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2090000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.002}]},{"osd":4,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83099999999999996,"15min":0.77000000000000002},"min":{"1min":0.65100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1499999999999999,"5min":1.77,"15min":1.77},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92400000000000004,"15min":0.85299999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3580000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.036}]},{"osd":6,"last update":"Tue Sep 10 18:52:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0329999999999999,"15min":1.012},"min":{"1min":0.80200000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.294,"5min":1.462,"15min":1.462},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.081,"5min":1.0700000000000001,"15min":1.036},"min":{"1min":0.67100000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.4359999999999999,"5min":1.653,"15min":1.653},"last":1.0860000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0329999999999999,"15min":1.0629999999999999},"min":{"1min":0.75900000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.4850000000000001,"5min":1.837,"15min":1.837},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.121,"15min":1.171},"min":{"1min":0.82199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.464,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.204}]}]},{"osd":4,"up_from":35,"seq":150323855446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298104,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447048,"statfs":{"total":95995035648,"available":95689777152,"internally_reserved":0,"allocated":765952,"data_stored":349871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.877,"15min":0.85099999999999998},"min":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.171,"5min":1.204,"15min":1.204},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91300000000000003,"15min":0.82599999999999996},"min":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56000000000000005},"max":{"1min":1.1870000000000001,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0780000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.90400000000000003,"15min":0.88800000000000001},"min":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2869999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92600000000000005,"15min":0.84499999999999997},"min":{"1min":0.624,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.3480000000000001,"5min":1.456,"15min":1.456},"last":1.1579999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.93799999999999994,"15min":0.872},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3160000000000001,"5min":1.425,"15min":1.425},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.94699999999999995,"15min":0.872},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.59699999999999998},"max":{"1min":1.3180000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.1339999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95699999999999996,"15min":0.88900000000000001},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.56399999999999995},"max":{"1min":1.327,"5min":1.714,"15min":1.714},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97099999999999997,"15min":0.91700000000000004},"min":{"1min":0.79800000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.339,"5min":1.391,"15min":1.391},"last":0.88}]},{"osd":5,"last update":"Tue Sep 10 18:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.90300000000000002,"15min":0.83499999999999996},"min":{"1min":0.63200000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3939999999999999,"5min":1.484,"15min":1.484},"last":0.75},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97299999999999998,"15min":0.88100000000000001},"min":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.405,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.99399999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":0.96999999999999997,"15min":0.92400000000000004},"min":{"1min":0.77200000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0880000000000001,"15min":1.054},"min":{"1min":0.90200000000000002,"5min":0.754,"15min":0.754},"max":{"1min":1.5009999999999999,"5min":1.536,"15min":1.536},"last":1.244}]},{"osd":7,"last update":"Tue Sep 10 18:52:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.167,"15min":1.177},"min":{"1min":0.746,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6120000000000001,"5min":2.008,"15min":2.008},"last":1.2},{"interface":"front","average":{"1min":1.175,"5min":1.157,"15min":1.1719999999999999},"min":{"1min":0.873,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4430000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1100000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051681,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298068,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447084,"statfs":{"total":95995035648,"available":95689814016,"internally_reserved":0,"allocated":729088,"data_stored":349468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.52500000000000002,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.01,"15min":1.2529999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.014,"5min":1.032,"15min":1.2869999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82599999999999996,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.48899999999999999,"15min":0.36699999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":1.274},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.78100000000000003},"min":{"1min":0.50600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.77700000000000002,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1739999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:52:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.079,"15min":0.99099999999999999},"min":{"1min":0.83299999999999996,"5min":0.70099999999999996,"15min":0.56999999999999995},"max":{"1min":1.373,"5min":1.778,"15min":1.778},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1579999999999999,"15min":1.069},"min":{"1min":0.86699999999999999,"5min":0.75700000000000001,"15min":0.59399999999999997},"max":{"1min":1.4239999999999999,"5min":1.952,"15min":1.952},"last":0.93500000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.113,"15min":1.0960000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5469999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.895},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.97899999999999998},"min":{"1min":0.72999999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4630000000000001,"5min":1.532,"15min":1.532},"last":1.9339999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.173,"15min":1.163},"min":{"1min":0.82099999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.641,"5min":1.643,"15min":1.643},"last":1.9690000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.079,"15min":1.0669999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.3620000000000001,"5min":1.476,"15min":1.476},"last":0.91200000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.101,"15min":1.1100000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":1.02},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2290000000000001,"15min":1.2390000000000001},"min":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":2.0230000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298080,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447072,"statfs":{"total":95995035648,"available":95689801728,"internally_reserved":0,"allocated":741376,"data_stored":349665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.69799999999999995,"15min":0.63},"min":{"1min":0.40100000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0049999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80200000000000005,"15min":0.71699999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.30599999999999999},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.69999999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70999999999999996,"15min":0.66900000000000004},"min":{"1min":0.54800000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.83699999999999997,"15min":0.74199999999999999},"min":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.379},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.85599999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88200000000000001,"15min":0.81299999999999994},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4510000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.84499999999999997,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.93400000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.022,"15min":0.95599999999999996},"min":{"1min":0.745,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.425,"5min":1.542,"15min":1.542},"last":1.242},{"interface":"front","average":{"1min":1.107,"5min":1.115,"15min":1.0680000000000001},"min":{"1min":0.73299999999999998,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.627,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.5369999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:51:56 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0940000000000001,"15min":1.111},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.71,"5min":1.766,"15min":1.766},"last":1.619},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.069,"15min":1.004},"min":{"1min":0.72999999999999998,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.5700000000000001,"5min":1.643,"15min":1.643},"last":1.2889999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.163,"15min":1.181},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.7},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.091,"15min":1.0680000000000001},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.3520000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:52:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.194,"15min":1.204},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.139,"15min":1.1220000000000001},"min":{"1min":0.90400000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.5089999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302236,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442916,"statfs":{"total":95995035648,"available":95685545984,"internally_reserved":0,"allocated":737280,"data_stored":348895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.59199999999999997},"min":{"1min":0.49099999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.77500000000000002,"15min":0.65900000000000003},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.35299999999999998},"max":{"1min":1.857,"5min":2.181,"15min":2.181},"last":1.099}]},{"osd":2,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73699999999999999,"15min":0.65800000000000003},"min":{"1min":0.54200000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.873,"5min":0.83299999999999996,"15min":0.73599999999999999},"min":{"1min":0.60299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.58699999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.877,"15min":0.81899999999999995},"min":{"1min":0.629,"5min":0.435,"15min":0.435},"max":{"1min":1.2210000000000001,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77100000000000002,"15min":0.73399999999999999},"min":{"1min":0.59899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.86899999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.036,"15min":1},"min":{"1min":0.71899999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6830000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97699999999999998,"15min":0.88900000000000001},"min":{"1min":0.78400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.97799999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:52:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0029999999999999,"15min":1.0149999999999999},"min":{"1min":0.73899999999999999,"5min":0.38,"15min":0.38},"max":{"1min":1.3560000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.034},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1319999999999999,"15min":1.137},"min":{"1min":0.86899999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.696,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:52:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.145,"15min":1.087},"min":{"1min":1.0129999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.6080000000000001,"5min":1.976,"15min":1.976},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.77500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7370000000000001,"5min":1.754,"15min":1.754},"last":1.129}]},{"osd":7,"last update":"Tue Sep 10 18:52:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2030000000000001,"15min":1.2210000000000001},"min":{"1min":0.89400000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.762,"5min":2.125,"15min":2.125},"last":1.056},{"interface":"front","average":{"1min":1.155,"5min":1.1439999999999999,"15min":1.1479999999999999},"min":{"1min":0.878,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.675,"5min":1.71,"15min":1.71},"last":1.224}]}]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298640,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446512,"statfs":{"total":95995035648,"available":95689228288,"internally_reserved":0,"allocated":1318912,"data_stored":932321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65800000000000003,"15min":0.57699999999999996},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.76500000000000001,"15min":0.65000000000000002},"min":{"1min":0.496,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2050000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.55400000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72699999999999998,"15min":0.67400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.69099999999999995},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.78800000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0249999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.77800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.48699999999999999},"max":{"1min":1.633,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.82099999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:52:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":0.90700000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:51:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0600000000000001,"15min":1.004},"min":{"1min":0.73599999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5189999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.173},{"interface":"front","average":{"1min":1.079,"5min":1.091,"15min":1.081},"min":{"1min":0.68999999999999995,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6000000000000001,"5min":1.712,"15min":1.712},"last":1.0009999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:51:48 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.1180000000000001,"15min":1.1000000000000001},"min":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.48,"5min":1.893,"15min":1.893},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.101,"5min":1.135,"15min":1.105},"min":{"1min":0.82599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.4219999999999999,"5min":2.363,"15min":2.363},"last":1.093}]},{"osd":7,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.996,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.252,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":1.0580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:52:56.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:52:56.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[18629]: cluster 2024-09-10T18:52:55.253490+0000 osd.4 (osd.4) 95 : cluster [DBG] 2.f scrub starts 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[18629]: cluster 2024-09-10T18:52:55.257900+0000 osd.4 (osd.4) 96 : cluster [DBG] 2.f scrub ok 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[18629]: cluster 2024-09-10T18:52:55.440767+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[18629]: audit 2024-09-10T18:52:55.479592+0000 mgr.y (mgr.24526) 133 : audit [DBG] from='client.14793 v1:172.21.15.83:0/885444477' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[32381]: cluster 2024-09-10T18:52:55.253490+0000 osd.4 (osd.4) 95 : cluster [DBG] 2.f scrub starts 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[32381]: cluster 2024-09-10T18:52:55.257900+0000 osd.4 (osd.4) 96 : cluster [DBG] 2.f scrub ok 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[32381]: cluster 2024-09-10T18:52:55.440767+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-10T18:52:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:56 smithi083 bash[32381]: audit 2024-09-10T18:52:55.479592+0000 mgr.y (mgr.24526) 133 : audit [DBG] from='client.14793 v1:172.21.15.83:0/885444477' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:56 smithi165 bash[22591]: cluster 2024-09-10T18:52:55.253490+0000 osd.4 (osd.4) 95 : cluster [DBG] 2.f scrub starts 2024-09-10T18:52:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:56 smithi165 bash[22591]: cluster 2024-09-10T18:52:55.257900+0000 osd.4 (osd.4) 96 : cluster [DBG] 2.f scrub ok 2024-09-10T18:52:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:56 smithi165 bash[22591]: cluster 2024-09-10T18:52:55.440767+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-09-10T18:52:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:56 smithi165 bash[22591]: audit 2024-09-10T18:52:55.479592+0000 mgr.y (mgr.24526) 133 : audit [DBG] from='client.14793 v1:172.21.15.83:0/885444477' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:52:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.363856+0000 osd.1 (osd.1) 89 : cluster [DBG] 4.f scrub starts 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.368639+0000 osd.1 (osd.1) 90 : cluster [DBG] 4.f scrub ok 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.400794+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.405848+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.521600+0000 mgr.y (mgr.24526) 134 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.363856+0000 osd.1 (osd.1) 89 : cluster [DBG] 4.f scrub starts 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.368639+0000 osd.1 (osd.1) 90 : cluster [DBG] 4.f scrub ok 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.400794+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.405848+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:52:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:57 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.521600+0000 mgr.y (mgr.24526) 134 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:57.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:52:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.363856+0000 osd.1 (osd.1) 89 : cluster [DBG] 4.f scrub starts 2024-09-10T18:52:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.368639+0000 osd.1 (osd.1) 90 : cluster [DBG] 4.f scrub ok 2024-09-10T18:52:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.400794+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:52:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.405848+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:52:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:57 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.521600+0000 mgr.y (mgr.24526) 134 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.793369+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:56.793506+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:57.394954+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:57.398876+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:57.450842+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:57.475407+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:52:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: cluster 2024-09-10T18:52:57.479132+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:52:58.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[18629]: audit 2024-09-10T18:52:57.542555+0000 mgr.y (mgr.24526) 135 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.793369+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:56.793506+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:57.394954+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:57.398876+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:57.450842+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:57.475407+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: cluster 2024-09-10T18:52:57.479132+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:52:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:58 smithi083 bash[32381]: audit 2024-09-10T18:52:57.542555+0000 mgr.y (mgr.24526) 135 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.793369+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:52:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:56.793506+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:52:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:57.394954+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:52:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:57.398876+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:52:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:57.450842+0000 mon.a (mon.0) 868 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-09-10T18:52:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:57.475407+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:52:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: cluster 2024-09-10T18:52:57.479132+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:52:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:58 smithi165 bash[22591]: audit 2024-09-10T18:52:57.542555+0000 mgr.y (mgr.24526) 135 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:52:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[18629]: cluster 2024-09-10T18:52:58.438281+0000 osd.5 (osd.5) 81 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:52:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[18629]: cluster 2024-09-10T18:52:58.444424+0000 osd.5 (osd.5) 82 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:52:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[18629]: cluster 2024-09-10T18:52:58.522414+0000 mgr.y (mgr.24526) 136 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:52:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[32381]: cluster 2024-09-10T18:52:58.438281+0000 osd.5 (osd.5) 81 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:52:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[32381]: cluster 2024-09-10T18:52:58.444424+0000 osd.5 (osd.5) 82 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:52:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:52:59 smithi083 bash[32381]: cluster 2024-09-10T18:52:58.522414+0000 mgr.y (mgr.24526) 136 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:52:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:59 smithi165 bash[22591]: cluster 2024-09-10T18:52:58.438281+0000 osd.5 (osd.5) 81 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:52:59.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:59 smithi165 bash[22591]: cluster 2024-09-10T18:52:58.444424+0000 osd.5 (osd.5) 82 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:52:59.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:52:59 smithi165 bash[22591]: cluster 2024-09-10T18:52:58.522414+0000 mgr.y (mgr.24526) 136 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:53:00.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[32381]: cluster 2024-09-10T18:52:59.237895+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:53:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[32381]: cluster 2024-09-10T18:52:59.241536+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:53:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[32381]: audit 2024-09-10T18:52:59.550730+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[18629]: cluster 2024-09-10T18:52:59.237895+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:53:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[18629]: cluster 2024-09-10T18:52:59.241536+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:53:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:00 smithi083 bash[18629]: audit 2024-09-10T18:52:59.550730+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:00 smithi165 bash[22591]: cluster 2024-09-10T18:52:59.237895+0000 osd.4 (osd.4) 97 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:53:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:00 smithi165 bash[22591]: cluster 2024-09-10T18:52:59.241536+0000 osd.4 (osd.4) 98 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:53:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:00 smithi165 bash[22591]: audit 2024-09-10T18:52:59.550730+0000 mon.c (mon.2) 86 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:01.071 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:01.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.393291+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.396766+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.400477+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub starts 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.404034+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.a scrub ok 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.424066+0000 osd.1 (osd.1) 91 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.429221+0000 osd.1 (osd.1) 92 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[18629]: cluster 2024-09-10T18:53:00.523848+0000 mgr.y (mgr.24526) 137 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.393291+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.396766+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:53:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.400477+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub starts 2024-09-10T18:53:01.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.404034+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.a scrub ok 2024-09-10T18:53:01.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.424066+0000 osd.1 (osd.1) 91 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:53:01.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.429221+0000 osd.1 (osd.1) 92 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:53:01.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:01 smithi083 bash[32381]: cluster 2024-09-10T18:53:00.523848+0000 mgr.y (mgr.24526) 137 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:01.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.393291+0000 osd.2 (osd.2) 45 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.396766+0000 osd.2 (osd.2) 46 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.400477+0000 osd.5 (osd.5) 83 : cluster [DBG] 6.a scrub starts 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.404034+0000 osd.5 (osd.5) 84 : cluster [DBG] 6.a scrub ok 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.424066+0000 osd.1 (osd.1) 91 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.429221+0000 osd.1 (osd.1) 92 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:53:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:01 smithi165 bash[22591]: cluster 2024-09-10T18:53:00.523848+0000 mgr.y (mgr.24526) 137 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:03.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:02] "GET /metrics HTTP/1.1" 200 220135 "" "Prometheus/2.43.0" 2024-09-10T18:53:03.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[32381]: cluster 2024-09-10T18:53:02.524576+0000 mgr.y (mgr.24526) 138 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:03.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[32381]: cluster 2024-09-10T18:53:03.411419+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.e scrub starts 2024-09-10T18:53:03.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[32381]: cluster 2024-09-10T18:53:03.415888+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.e scrub ok 2024-09-10T18:53:03.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:03 smithi165 bash[22591]: cluster 2024-09-10T18:53:02.524576+0000 mgr.y (mgr.24526) 138 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:03.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:03 smithi165 bash[22591]: cluster 2024-09-10T18:53:03.411419+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.e scrub starts 2024-09-10T18:53:03.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:03 smithi165 bash[22591]: cluster 2024-09-10T18:53:03.415888+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.e scrub ok 2024-09-10T18:53:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[18629]: cluster 2024-09-10T18:53:02.524576+0000 mgr.y (mgr.24526) 138 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[18629]: cluster 2024-09-10T18:53:03.411419+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.e scrub starts 2024-09-10T18:53:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:03 smithi083 bash[18629]: cluster 2024-09-10T18:53:03.415888+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.e scrub ok 2024-09-10T18:53:04.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/4174345338 >> v1:172.21.15.83:6790/0 conn(0x7f74501025d0 legacy=0x7f74501029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:04.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/4174345338 shutdown_connections 2024-09-10T18:53:04.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/4174345338 >> 172.21.15.83:0/4174345338 conn(0x7f74500f81f0 msgr2=0x7f74500fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:04.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/4174345338 shutdown_connections 2024-09-10T18:53:04.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/4174345338 wait complete. 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 librados: starting msgr at 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 librados: starting objecter 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 Processor -- start 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- start start 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 librados: setting wanted keys 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 librados: calling monclient init 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f745011c020 con 0x7f74501025d0 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f745011d220 con 0x7f7450108f20 2024-09-10T18:53:04.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f745011e420 con 0x7f7450105390 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f744effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f74501025d0 0x7f74500fe5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37304/0 (socket says 172.21.15.83:37304) 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f744effd640 1 -- 172.21.15.83:0/2255724547 learned_addr learned my addr 172.21.15.83:0/2255724547 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter ms_handle_connect 0x7f74501025d0 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter ms_handle_connect 0x7f7450105390 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter ms_handle_connect 0x7f7450108f20 2024-09-10T18:53:04.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256099917 0 0) 0x7f745011c020 con 0x7f74501025d0 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c003120 con 0x7f74501025d0 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106678893 0 0) 0x7f745011e420 con 0x7f7450105390 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745011c020 con 0x7f7450105390 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231778711 0 0) 0x7f745011d220 con 0x7f7450108f20 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745011e420 con 0x7f7450108f20 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148716929 0 0) 0x7f745011c020 con 0x7f7450105390 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f745011d220 con 0x7f7450105390 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7448001d70 con 0x7f7450105390 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279308721 0 0) 0x7f743c003120 con 0x7f74501025d0 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f745011c020 con 0x7f74501025d0 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7440002c50 con 0x7f74501025d0 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221502758 0 0) 0x7f745011e420 con 0x7f7450108f20 2024-09-10T18:53:04.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f743c003120 con 0x7f7450108f20 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7438001800 con 0x7f7450108f20 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661890668 0 0) 0x7f745011c020 con 0x7f74501025d0 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 >> v1:172.21.15.83:6790/0 conn(0x7f7450105390 legacy=0x7f74500fecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 >> v1:172.21.15.165:6789/0 conn(0x7f7450108f20 legacy=0x7f745011a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745011f620 con 0x7f74501025d0 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745011e610 con 0x7f74501025d0 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 10 client.14799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f745011ea60 con 0x7f74501025d0 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.498+0000 7f7455808640 1 librados: init done 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7440002370 con 0x7f74501025d0 2024-09-10T18:53:04.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f74400035c0 con 0x7f74501025d0 2024-09-10T18:53:04.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7430005010 con 0x7f74501025d0 2024-09-10T18:53:04.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7440003720 con 0x7f74501025d0 2024-09-10T18:53:04.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(111..111 src has 59..111) v4 ==== 6284+0+0 (unknown 1672160813 0 0) 0x7f744008a1c0 con 0x7f74501025d0 2024-09-10T18:53:04.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 10 client.14799.objecter ms_dispatch 0x7f7450000b90 osd_map(111..111 src has 59..111) v4 2024-09-10T18:53:04.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 3 client.14799.objecter handle_osd_map got epochs [111,111] > 0 2024-09-10T18:53:04.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 3 client.14799.objecter handle_osd_map decoding full epoch 111 2024-09-10T18:53:04.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 20 client.14799.objecter dump_active .. 0 homeless 2024-09-10T18:53:04.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.502+0000 7f74477fe640 10 client.14799.objecter ms_handle_connect 0x7f743c071770 2024-09-10T18:53:04.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.510+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7440056e30 con 0x7f74501025d0 2024-09-10T18:53:04.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.682+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7430000b90 con 0x7f743c071770 2024-09-10T18:53:04.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.702+0000 7f74477fe640 1 -- 172.21.15.83:0/2255724547 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384618 (unknown 2965378022 0 3903226532) 0x7f7430000b90 con 0x7f743c071770 2024-09-10T18:53:04.711 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:53:04.714 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 10 librados: watch_flush enter 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 10 librados: watch_flush exit 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 20 client.14799.objecter shutdown clearing up homeless session... 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 10 client.14799.objecter successfully canceled tick 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 >> v1:172.21.15.83:6800/3071793735 conn(0x7f743c071770 legacy=0x7f743c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 >> v1:172.21.15.83:6789/0 conn(0x7f74501025d0 legacy=0x7f74500fe5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 shutdown_connections 2024-09-10T18:53:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 >> 172.21.15.83:0/2255724547 conn(0x7f74500f81f0 msgr2=0x7f7450109380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 shutdown_connections 2024-09-10T18:53:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 -- 172.21.15.83:0/2255724547 wait complete. 2024-09-10T18:53:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:04.710+0000 7f7455808640 1 librados: shutdown 2024-09-10T18:53:05.436 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-10T18:53:04.524930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1072,"num_read_kb":928,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":631,"ondisk_log_size":631,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2390164,"kb_used_data":7072,"kb_used_omap":13,"kb_used_meta":2382898,"kb_avail":747571052,"statfs":{"total":767960285184,"available":765512757248,"internally_reserved":0,"allocated":7241728,"data_stored":4333329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2440088256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.087536"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442496+0000","last_change":"2024-09-10T18:52:31.777492+0000","last_active":"2024-09-10T18:52:55.442496+0000","last_peered":"2024-09-10T18:52:55.442496+0000","last_clean":"2024-09-10T18:52:55.442496+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:52:55.442496+0000","last_undegraded":"2024-09-10T18:52:55.442496+0000","last_fullsized":"2024-09-10T18:52:55.442496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:31.777344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.777344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.153926+0000","scrub_duration":0.0041891280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467643+0000","last_change":"2024-09-10T18:52:17.464193+0000","last_active":"2024-09-10T18:52:55.467643+0000","last_peered":"2024-09-10T18:52:55.467643+0000","last_clean":"2024-09-10T18:52:55.467643+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:52:55.467643+0000","last_undegraded":"2024-09-10T18:52:55.467643+0000","last_fullsized":"2024-09-10T18:52:55.467643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:17.463956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.463956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:35.714604+0000","scrub_duration":0.004114698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443218+0000","last_change":"2024-09-10T18:51:37.694332+0000","last_active":"2024-09-10T18:52:55.443218+0000","last_peered":"2024-09-10T18:52:55.443218+0000","last_clean":"2024-09-10T18:52:55.443218+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:52:55.443218+0000","last_undegraded":"2024-09-10T18:52:55.443218+0000","last_fullsized":"2024-09-10T18:52:55.443218+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:51:37.694143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:37.694143+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.236407+0000","scrub_duration":0.0048100759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442095+0000","last_change":"2024-09-10T18:52:20.734242+0000","last_active":"2024-09-10T18:52:55.442095+0000","last_peered":"2024-09-10T18:52:55.442095+0000","last_clean":"2024-09-10T18:52:55.442095+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:52:55.442095+0000","last_undegraded":"2024-09-10T18:52:55.442095+0000","last_fullsized":"2024-09-10T18:52:55.442095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.733951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.733951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.495882+0000","scrub_duration":0.0029697920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443402+0000","last_change":"2024-09-10T18:52:51.767710+0000","last_active":"2024-09-10T18:52:55.443402+0000","last_peered":"2024-09-10T18:52:55.443402+0000","last_clean":"2024-09-10T18:52:55.443402+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:52:55.443402+0000","last_undegraded":"2024-09-10T18:52:55.443402+0000","last_fullsized":"2024-09-10T18:52:55.443402+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:51.767543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:51.767543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:55.338203+0000","scrub_duration":0.0033286639999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467352+0000","last_change":"2024-09-10T18:52:23.600334+0000","last_active":"2024-09-10T18:52:55.467352+0000","last_peered":"2024-09-10T18:52:55.467352+0000","last_clean":"2024-09-10T18:52:55.467352+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:52:55.467352+0000","last_undegraded":"2024-09-10T18:52:55.467352+0000","last_fullsized":"2024-09-10T18:52:55.467352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:23.600141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.600141+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.041119+0000","scrub_duration":0.0045540590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.433704+0000","last_change":"2024-09-10T18:52:25.178969+0000","last_active":"2024-09-10T18:52:55.433704+0000","last_peered":"2024-09-10T18:52:55.433704+0000","last_clean":"2024-09-10T18:52:55.433704+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:52:55.433704+0000","last_undegraded":"2024-09-10T18:52:55.433704+0000","last_fullsized":"2024-09-10T18:52:55.433704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.178800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.178800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.952086+0000","scrub_duration":0.0034539140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434153+0000","last_change":"2024-09-10T18:52:26.143465+0000","last_active":"2024-09-10T18:52:55.434153+0000","last_peered":"2024-09-10T18:52:55.434153+0000","last_clean":"2024-09-10T18:52:55.434153+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:52:55.434153+0000","last_undegraded":"2024-09-10T18:52:55.434153+0000","last_fullsized":"2024-09-10T18:52:55.434153+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:26.143297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:26.143297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:44.663812+0000","scrub_duration":0.0032662059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466045+0000","last_change":"2024-09-10T18:52:16.237573+0000","last_active":"2024-09-10T18:52:55.466045+0000","last_peered":"2024-09-10T18:52:55.466045+0000","last_clean":"2024-09-10T18:52:55.466045+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:52:55.466045+0000","last_undegraded":"2024-09-10T18:52:55.466045+0000","last_fullsized":"2024-09-10T18:52:55.466045+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:16.237414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:16.237414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.186738+0000","scrub_duration":0.0033375729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465419+0000","last_change":"2024-09-10T18:52:28.673257+0000","last_active":"2024-09-10T18:52:55.465419+0000","last_peered":"2024-09-10T18:52:55.465419+0000","last_clean":"2024-09-10T18:52:55.465419+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:52:55.465419+0000","last_undegraded":"2024-09-10T18:52:55.465419+0000","last_fullsized":"2024-09-10T18:52:55.465419+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.673170+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:53.934483+0000","scrub_duration":0.01076011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434437+0000","last_change":"2024-09-10T18:52:28.116242+0000","last_active":"2024-09-10T18:52:55.434437+0000","last_peered":"2024-09-10T18:52:55.434437+0000","last_clean":"2024-09-10T18:52:55.434437+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:52:55.434437+0000","last_undegraded":"2024-09-10T18:52:55.434437+0000","last_fullsized":"2024-09-10T18:52:55.434437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.116084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.380860+0000","scrub_duration":0.003567785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.463269+0000","last_change":"2024-09-10T18:52:33.676485+0000","last_active":"2024-09-10T18:52:55.463269+0000","last_peered":"2024-09-10T18:52:55.463269+0000","last_clean":"2024-09-10T18:52:55.463269+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:52:55.463269+0000","last_undegraded":"2024-09-10T18:52:55.463269+0000","last_fullsized":"2024-09-10T18:52:55.463269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:33.676311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:33.676311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:53.929050+0000","scrub_duration":0.0037210590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465801+0000","last_change":"2024-09-10T18:52:22.326533+0000","last_active":"2024-09-10T18:52:55.465801+0000","last_peered":"2024-09-10T18:52:55.465801+0000","last_clean":"2024-09-10T18:52:55.465801+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:52:55.465801+0000","last_undegraded":"2024-09-10T18:52:55.465801+0000","last_fullsized":"2024-09-10T18:52:55.465801+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.326315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:34.718300+0000","scrub_duration":0.003554892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":104,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:52:58.444731+0000","last_change":"2024-09-10T18:52:58.444627+0000","last_active":"2024-09-10T18:52:58.444731+0000","last_peered":"2024-09-10T18:52:58.444731+0000","last_clean":"2024-09-10T18:52:58.444731+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:52:58.444731+0000","last_undegraded":"2024-09-10T18:52:58.444731+0000","last_fullsized":"2024-09-10T18:52:58.444731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:52:58.444435+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:52:58.444435+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.896723+0000","scrub_duration":0.0059810169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434459+0000","last_change":"2024-09-10T18:52:01.415959+0000","last_active":"2024-09-10T18:52:55.434459+0000","last_peered":"2024-09-10T18:52:55.434459+0000","last_clean":"2024-09-10T18:52:55.434459+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:52:55.434459+0000","last_undegraded":"2024-09-10T18:52:55.434459+0000","last_fullsized":"2024-09-10T18:52:55.434459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.415718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.415718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:25.902687+0000","scrub_duration":0.0036293950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465056+0000","last_change":"2024-09-10T18:52:52.611274+0000","last_active":"2024-09-10T18:52:55.465056+0000","last_peered":"2024-09-10T18:52:55.465056+0000","last_clean":"2024-09-10T18:52:55.465056+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:52:55.465056+0000","last_undegraded":"2024-09-10T18:52:55.465056+0000","last_fullsized":"2024-09-10T18:52:55.465056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:52.611115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:52.611115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:17.703658+0000","scrub_duration":0.0034278720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464469+0000","last_change":"2024-09-10T18:52:02.356154+0000","last_active":"2024-09-10T18:52:55.464469+0000","last_peered":"2024-09-10T18:52:55.464469+0000","last_clean":"2024-09-10T18:52:55.464469+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:52:55.464469+0000","last_undegraded":"2024-09-10T18:52:55.464469+0000","last_fullsized":"2024-09-10T18:52:55.464469+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:02.355980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:02.355980+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:21.108556+0000","scrub_duration":0.0055219309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466754+0000","last_change":"2024-09-10T18:52:43.413178+0000","last_active":"2024-09-10T18:52:55.466754+0000","last_peered":"2024-09-10T18:52:55.466754+0000","last_clean":"2024-09-10T18:52:55.466754+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:52:55.466754+0000","last_undegraded":"2024-09-10T18:52:55.466754+0000","last_fullsized":"2024-09-10T18:52:55.466754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:43.412924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:43.412924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:09.023724+0000","scrub_duration":0.0033475380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.968345+0000","last_change":"2024-09-10T18:52:17.252670+0000","last_active":"2024-09-10T18:52:55.968345+0000","last_peered":"2024-09-10T18:52:55.968345+0000","last_clean":"2024-09-10T18:52:55.968345+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:52:55.968345+0000","last_undegraded":"2024-09-10T18:52:55.968345+0000","last_fullsized":"2024-09-10T18:52:55.968345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.252498+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:23.792093+0000","scrub_duration":0.0045932949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.440786+0000","last_change":"2024-09-10T18:52:21.401011+0000","last_active":"2024-09-10T18:52:55.440786+0000","last_peered":"2024-09-10T18:52:55.440786+0000","last_clean":"2024-09-10T18:52:55.440786+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:52:55.440786+0000","last_undegraded":"2024-09-10T18:52:55.440786+0000","last_fullsized":"2024-09-10T18:52:55.440786+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.400853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.400853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:30.701974+0000","scrub_duration":0.0034523370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442793+0000","last_change":"2024-09-10T18:52:17.707983+0000","last_active":"2024-09-10T18:52:55.442793+0000","last_peered":"2024-09-10T18:52:55.442793+0000","last_clean":"2024-09-10T18:52:55.442793+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:52:55.442793+0000","last_undegraded":"2024-09-10T18:52:55.442793+0000","last_fullsized":"2024-09-10T18:52:55.442793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:52:17.707813+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:52:17.707813+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.139488+0000","scrub_duration":0.0045170389999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466513+0000","last_change":"2024-09-10T18:52:50.387339+0000","last_active":"2024-09-10T18:52:55.466513+0000","last_peered":"2024-09-10T18:52:55.466513+0000","last_clean":"2024-09-10T18:52:55.466513+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:52:55.466513+0000","last_undegraded":"2024-09-10T18:52:55.466513+0000","last_fullsized":"2024-09-10T18:52:55.466513+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:50.387160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:52:50.387160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.160775+0000","scrub_duration":0.0034323050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435319+0000","last_change":"2024-09-10T18:51:46.600626+0000","last_active":"2024-09-10T18:52:55.435319+0000","last_peered":"2024-09-10T18:52:55.435319+0000","last_clean":"2024-09-10T18:52:55.435319+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:52:55.435319+0000","last_undegraded":"2024-09-10T18:52:55.435319+0000","last_fullsized":"2024-09-10T18:52:55.435319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:46.600460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:46.600460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032588019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435165+0000","last_change":"2024-09-10T18:51:53.429372+0000","last_active":"2024-09-10T18:52:55.435165+0000","last_peered":"2024-09-10T18:52:55.435165+0000","last_clean":"2024-09-10T18:52:55.435165+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:52:55.435165+0000","last_undegraded":"2024-09-10T18:52:55.435165+0000","last_fullsized":"2024-09-10T18:52:55.435165+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:53.429216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.429216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.844395+0000","scrub_duration":0.003713091,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441567+0000","last_change":"2024-09-10T18:51:55.522605+0000","last_active":"2024-09-10T18:52:55.441567+0000","last_peered":"2024-09-10T18:52:55.441567+0000","last_clean":"2024-09-10T18:52:55.441567+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:52:55.441567+0000","last_undegraded":"2024-09-10T18:52:55.441567+0000","last_fullsized":"2024-09-10T18:52:55.441567+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:51:55.522513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.522513+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.705011+0000","scrub_duration":0.0065065660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466040+0000","last_change":"2024-09-10T18:52:25.326041+0000","last_active":"2024-09-10T18:52:55.466040+0000","last_peered":"2024-09-10T18:52:55.466040+0000","last_clean":"2024-09-10T18:52:55.466040+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:52:55.466040+0000","last_undegraded":"2024-09-10T18:52:55.466040+0000","last_fullsized":"2024-09-10T18:52:55.466040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.325803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.325803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.547370+0000","scrub_duration":0.0039072000000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:56.368848+0000","last_change":"2024-09-10T18:52:56.368771+0000","last_active":"2024-09-10T18:52:56.368848+0000","last_peered":"2024-09-10T18:52:56.368848+0000","last_clean":"2024-09-10T18:52:56.368848+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:52:56.368848+0000","last_undegraded":"2024-09-10T18:52:56.368848+0000","last_fullsized":"2024-09-10T18:52:56.368848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:56.368650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:56.368650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:17.370146+0000","scrub_duration":0.004734795,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466309+0000","last_change":"2024-09-10T18:51:50.674985+0000","last_active":"2024-09-10T18:52:55.466309+0000","last_peered":"2024-09-10T18:52:55.466309+0000","last_clean":"2024-09-10T18:52:55.466309+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:52:55.466309+0000","last_undegraded":"2024-09-10T18:52:55.466309+0000","last_fullsized":"2024-09-10T18:52:55.466309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.674821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:16.792421+0000","scrub_duration":0.003696628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":96,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434574+0000","last_change":"2024-09-10T18:51:29.827804+0000","last_active":"2024-09-10T18:52:55.434574+0000","last_peered":"2024-09-10T18:52:55.434574+0000","last_clean":"2024-09-10T18:52:55.434574+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:52:55.434574+0000","last_undegraded":"2024-09-10T18:52:55.434574+0000","last_fullsized":"2024-09-10T18:52:55.434574+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:51:29.827615+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:51:29.827615+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051889199999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.469880+0000","last_change":"2024-09-10T18:52:23.328971+0000","last_active":"2024-09-10T18:52:55.469880+0000","last_peered":"2024-09-10T18:52:55.469880+0000","last_clean":"2024-09-10T18:52:55.469880+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:52:55.469880+0000","last_undegraded":"2024-09-10T18:52:55.469880+0000","last_fullsized":"2024-09-10T18:52:55.469880+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:23.328703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:23.328703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.447554+0000","scrub_duration":0.0041061600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.968312+0000","last_change":"2024-09-10T18:51:40.694756+0000","last_active":"2024-09-10T18:52:55.968312+0000","last_peered":"2024-09-10T18:52:55.968312+0000","last_clean":"2024-09-10T18:52:55.968312+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:52:55.968312+0000","last_undegraded":"2024-09-10T18:52:55.968312+0000","last_fullsized":"2024-09-10T18:52:55.968312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:51:40.694630+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.694630+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:58.465301+0000","scrub_duration":0.0057213519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442614+0000","last_change":"2024-09-10T18:52:43.777074+0000","last_active":"2024-09-10T18:52:55.442614+0000","last_peered":"2024-09-10T18:52:55.442614+0000","last_clean":"2024-09-10T18:52:55.442614+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:52:55.442614+0000","last_undegraded":"2024-09-10T18:52:55.442614+0000","last_fullsized":"2024-09-10T18:52:55.442614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:52:43.776904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:43.776904+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.160729+0000","scrub_duration":0.0039940949999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466994+0000","last_change":"2024-09-10T18:52:31.620025+0000","last_active":"2024-09-10T18:52:55.466994+0000","last_peered":"2024-09-10T18:52:55.466994+0000","last_clean":"2024-09-10T18:52:55.466994+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:52:55.466994+0000","last_undegraded":"2024-09-10T18:52:55.466994+0000","last_fullsized":"2024-09-10T18:52:55.466994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:52:31.619837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.619837+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:51.678739+0000","scrub_duration":0.0054562689999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465452+0000","last_change":"2024-09-10T18:51:47.158565+0000","last_active":"2024-09-10T18:52:55.465452+0000","last_peered":"2024-09-10T18:52:55.465452+0000","last_clean":"2024-09-10T18:52:55.465452+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:52:55.465452+0000","last_undegraded":"2024-09-10T18:52:55.465452+0000","last_fullsized":"2024-09-10T18:52:55.465452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:47.158405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.158405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.185469+0000","scrub_duration":0.0034278289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.433976+0000","last_change":"2024-09-10T18:51:28.793948+0000","last_active":"2024-09-10T18:52:55.433976+0000","last_peered":"2024-09-10T18:52:55.433976+0000","last_clean":"2024-09-10T18:52:55.433976+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:52:55.433976+0000","last_undegraded":"2024-09-10T18:52:55.433976+0000","last_fullsized":"2024-09-10T18:52:55.433976+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:28.793787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:28.793787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035723209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":272,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:52:58.783475+0000","last_change":"2024-09-10T18:52:00.591712+0000","last_active":"2024-09-10T18:52:58.783475+0000","last_peered":"2024-09-10T18:52:58.783475+0000","last_clean":"2024-09-10T18:52:58.783475+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:52:58.783475+0000","last_undegraded":"2024-09-10T18:52:58.783475+0000","last_fullsized":"2024-09-10T18:52:58.783475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:52:00.591632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.591632+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:05.247607+0000","scrub_duration":0.0032914160000000001,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465572+0000","last_change":"2024-09-10T18:51:56.155039+0000","last_active":"2024-09-10T18:52:55.465572+0000","last_peered":"2024-09-10T18:52:55.465572+0000","last_clean":"2024-09-10T18:52:55.465572+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:52:55.465572+0000","last_undegraded":"2024-09-10T18:52:55.465572+0000","last_fullsized":"2024-09-10T18:52:55.465572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.154883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.154883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.178313+0000","scrub_duration":0.0033842830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443759+0000","last_change":"2024-09-10T18:52:09.660716+0000","last_active":"2024-09-10T18:52:55.443759+0000","last_peered":"2024-09-10T18:52:55.443759+0000","last_clean":"2024-09-10T18:52:55.443759+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:52:55.443759+0000","last_undegraded":"2024-09-10T18:52:55.443759+0000","last_fullsized":"2024-09-10T18:52:55.443759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:52:09.660551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.660551+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.065594+0000","scrub_duration":0.0047491369999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468083+0000","last_change":"2024-09-10T18:52:05.361338+0000","last_active":"2024-09-10T18:52:55.468083+0000","last_peered":"2024-09-10T18:52:55.468083+0000","last_clean":"2024-09-10T18:52:55.468083+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:52:55.468083+0000","last_undegraded":"2024-09-10T18:52:55.468083+0000","last_fullsized":"2024-09-10T18:52:55.468083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:05.361175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.361175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.175562+0000","scrub_duration":0.0039873549999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.971923+0000","last_change":"2024-09-10T18:52:45.821797+0000","last_active":"2024-09-10T18:52:55.971923+0000","last_peered":"2024-09-10T18:52:55.971923+0000","last_clean":"2024-09-10T18:52:55.971923+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:52:55.971923+0000","last_undegraded":"2024-09-10T18:52:55.971923+0000","last_fullsized":"2024-09-10T18:52:55.971923+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:45.821628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:45.821628+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:12.908001+0000","scrub_duration":0.0045851850000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"78'1","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.433858+0000","last_change":"2024-09-10T18:51:32.830829+0000","last_active":"2024-09-10T18:52:55.433858+0000","last_peered":"2024-09-10T18:52:55.433858+0000","last_clean":"2024-09-10T18:52:55.433858+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:52:55.433858+0000","last_undegraded":"2024-09-10T18:52:55.433858+0000","last_fullsized":"2024-09-10T18:52:55.433858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-09-10T18:51:32.830654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:32.830654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:59.674413+0000","scrub_duration":0.0041675089999999998,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434450+0000","last_change":"2024-09-10T18:52:53.279005+0000","last_active":"2024-09-10T18:52:55.434450+0000","last_peered":"2024-09-10T18:52:55.434450+0000","last_clean":"2024-09-10T18:52:55.434450+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:52:55.434450+0000","last_undegraded":"2024-09-10T18:52:55.434450+0000","last_fullsized":"2024-09-10T18:52:55.434450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:53.278836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:53.278836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:03.169218+0000","scrub_duration":0.00358502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"77'30","reported_seq":160,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.971630+0000","last_change":"2024-09-10T18:52:11.309315+0000","last_active":"2024-09-10T18:52:55.971630+0000","last_peered":"2024-09-10T18:52:55.971630+0000","last_clean":"2024-09-10T18:52:55.971630+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:52:55.971630+0000","last_undegraded":"2024-09-10T18:52:55.971630+0000","last_fullsized":"2024-09-10T18:52:55.971630+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'30","last_scrub_stamp":"2024-09-10T18:52:11.309152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.309152+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:28.549403+0000","scrub_duration":0.0055436330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466173+0000","last_change":"2024-09-10T18:52:03.183815+0000","last_active":"2024-09-10T18:52:55.466173+0000","last_peered":"2024-09-10T18:52:55.466173+0000","last_clean":"2024-09-10T18:52:55.466173+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:52:55.466173+0000","last_undegraded":"2024-09-10T18:52:55.466173+0000","last_fullsized":"2024-09-10T18:52:55.466173+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.183646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.183646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.233272+0000","scrub_duration":0.0032583899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":121,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.971365+0000","last_change":"2024-09-10T18:52:07.501745+0000","last_active":"2024-09-10T18:52:55.971365+0000","last_peered":"2024-09-10T18:52:55.971365+0000","last_clean":"2024-09-10T18:52:55.971365+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:52:55.971365+0000","last_undegraded":"2024-09-10T18:52:55.971365+0000","last_fullsized":"2024-09-10T18:52:55.971365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.501543+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.400864+0000","scrub_duration":0.0064925740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464091+0000","last_change":"2024-09-10T18:52:15.552956+0000","last_active":"2024-09-10T18:52:55.464091+0000","last_peered":"2024-09-10T18:52:55.464091+0000","last_clean":"2024-09-10T18:52:55.464091+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:52:55.464091+0000","last_undegraded":"2024-09-10T18:52:55.464091+0000","last_fullsized":"2024-09-10T18:52:55.464091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.552791+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:33.020117+0000","scrub_duration":0.0058227280000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465732+0000","last_change":"2024-09-10T18:52:41.396637+0000","last_active":"2024-09-10T18:52:55.465732+0000","last_peered":"2024-09-10T18:52:55.465732+0000","last_clean":"2024-09-10T18:52:55.465732+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:52:55.465732+0000","last_undegraded":"2024-09-10T18:52:55.465732+0000","last_fullsized":"2024-09-10T18:52:55.465732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:41.396466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:41.396466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:09.624971+0000","scrub_duration":0.0033778850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467152+0000","last_change":"2024-09-10T18:51:58.828562+0000","last_active":"2024-09-10T18:52:55.467152+0000","last_peered":"2024-09-10T18:52:55.467152+0000","last_clean":"2024-09-10T18:52:55.467152+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:52:55.467152+0000","last_undegraded":"2024-09-10T18:52:55.467152+0000","last_fullsized":"2024-09-10T18:52:55.467152+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:58.828403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.828403+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.385475+0000","scrub_duration":0.0052576080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468653+0000","last_change":"2024-09-10T18:52:18.315632+0000","last_active":"2024-09-10T18:52:55.468653+0000","last_peered":"2024-09-10T18:52:55.468653+0000","last_clean":"2024-09-10T18:52:55.468653+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:52:55.468653+0000","last_undegraded":"2024-09-10T18:52:55.468653+0000","last_fullsized":"2024-09-10T18:52:55.468653+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:18.315469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.315469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:29.767066+0000","scrub_duration":0.0035610590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466276+0000","last_change":"2024-09-10T18:52:34.328450+0000","last_active":"2024-09-10T18:52:55.466276+0000","last_peered":"2024-09-10T18:52:55.466276+0000","last_clean":"2024-09-10T18:52:55.466276+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:52:55.466276+0000","last_undegraded":"2024-09-10T18:52:55.466276+0000","last_fullsized":"2024-09-10T18:52:55.466276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:34.328284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:34.328284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.788608+0000","scrub_duration":0.0036424299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465207+0000","last_change":"2024-09-10T18:52:37.626097+0000","last_active":"2024-09-10T18:52:55.465207+0000","last_peered":"2024-09-10T18:52:55.465207+0000","last_clean":"2024-09-10T18:52:55.465207+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:52:55.465207+0000","last_undegraded":"2024-09-10T18:52:55.465207+0000","last_fullsized":"2024-09-10T18:52:55.465207+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:37.625944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:37.625944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.595482+0000","scrub_duration":0.004216867,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435256+0000","last_change":"2024-09-10T18:52:15.209771+0000","last_active":"2024-09-10T18:52:55.435256+0000","last_peered":"2024-09-10T18:52:55.435256+0000","last_clean":"2024-09-10T18:52:55.435256+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:52:55.435256+0000","last_undegraded":"2024-09-10T18:52:55.435256+0000","last_fullsized":"2024-09-10T18:52:55.435256+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:52:15.209609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.209609+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:45.019264+0000","scrub_duration":0.005193746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.469237+0000","last_change":"2024-09-10T18:52:09.838167+0000","last_active":"2024-09-10T18:52:55.469237+0000","last_peered":"2024-09-10T18:52:55.469237+0000","last_clean":"2024-09-10T18:52:55.469237+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:52:55.469237+0000","last_undegraded":"2024-09-10T18:52:55.469237+0000","last_fullsized":"2024-09-10T18:52:55.469237+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.838015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.838015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.447360+0000","scrub_duration":0.003415202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468408+0000","last_change":"2024-09-10T18:52:09.323881+0000","last_active":"2024-09-10T18:52:55.468408+0000","last_peered":"2024-09-10T18:52:55.468408+0000","last_clean":"2024-09-10T18:52:55.468408+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:52:55.468408+0000","last_undegraded":"2024-09-10T18:52:55.468408+0000","last_fullsized":"2024-09-10T18:52:55.468408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:09.323704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.323704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.895803+0000","scrub_duration":0.0036260730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442882+0000","last_change":"2024-09-10T18:52:50.797273+0000","last_active":"2024-09-10T18:52:55.442882+0000","last_peered":"2024-09-10T18:52:55.442882+0000","last_clean":"2024-09-10T18:52:55.442882+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:52:55.442882+0000","last_undegraded":"2024-09-10T18:52:55.442882+0000","last_fullsized":"2024-09-10T18:52:55.442882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:50.797111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:50.797111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.050531+0000","scrub_duration":0.0036202809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.469014+0000","last_change":"2024-09-10T18:51:58.430973+0000","last_active":"2024-09-10T18:52:55.469014+0000","last_peered":"2024-09-10T18:52:55.469014+0000","last_clean":"2024-09-10T18:52:55.469014+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:52:55.469014+0000","last_undegraded":"2024-09-10T18:52:55.469014+0000","last_fullsized":"2024-09-10T18:52:55.469014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:51:58.430868+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.088894+0000","scrub_duration":0.013718902999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.439788+0000","last_change":"2024-09-10T18:51:40.776447+0000","last_active":"2024-09-10T18:52:55.439788+0000","last_peered":"2024-09-10T18:52:55.439788+0000","last_clean":"2024-09-10T18:52:55.439788+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:52:55.439788+0000","last_undegraded":"2024-09-10T18:52:55.439788+0000","last_fullsized":"2024-09-10T18:52:55.439788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:40.776169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:40.776169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.421742+0000","scrub_duration":0.0038302309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468865+0000","last_change":"2024-09-10T18:52:19.322071+0000","last_active":"2024-09-10T18:52:55.468865+0000","last_peered":"2024-09-10T18:52:55.468865+0000","last_clean":"2024-09-10T18:52:55.468865+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:52:55.468865+0000","last_undegraded":"2024-09-10T18:52:55.468865+0000","last_fullsized":"2024-09-10T18:52:55.468865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:19.321916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:19.321916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.895150+0000","scrub_duration":0.0033042869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465463+0000","last_change":"2024-09-10T18:51:45.221499+0000","last_active":"2024-09-10T18:52:55.465463+0000","last_peered":"2024-09-10T18:52:55.465463+0000","last_clean":"2024-09-10T18:52:55.465463+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:52:55.465463+0000","last_undegraded":"2024-09-10T18:52:55.465463+0000","last_fullsized":"2024-09-10T18:52:55.465463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:45.221333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:45.221333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:52:57.678342+0000","scrub_duration":0.0036649180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465394+0000","last_change":"2024-09-10T18:51:51.619698+0000","last_active":"2024-09-10T18:52:55.465394+0000","last_peered":"2024-09-10T18:52:55.465394+0000","last_clean":"2024-09-10T18:52:55.465394+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:52:55.465394+0000","last_undegraded":"2024-09-10T18:52:55.465394+0000","last_fullsized":"2024-09-10T18:52:55.465394+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:51:51.619511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:51.619511+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:03.627716+0000","scrub_duration":0.0049634149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.470075+0000","last_change":"2024-09-10T18:52:06.149853+0000","last_active":"2024-09-10T18:52:55.470075+0000","last_peered":"2024-09-10T18:52:55.470075+0000","last_clean":"2024-09-10T18:52:55.470075+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:52:55.470075+0000","last_undegraded":"2024-09-10T18:52:55.470075+0000","last_fullsized":"2024-09-10T18:52:55.470075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.149699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.149699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.887082+0000","scrub_duration":0.0035198529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467096+0000","last_change":"2024-09-10T18:51:50.626660+0000","last_active":"2024-09-10T18:52:55.467096+0000","last_peered":"2024-09-10T18:52:55.467096+0000","last_clean":"2024-09-10T18:52:55.467096+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:52:55.467096+0000","last_undegraded":"2024-09-10T18:52:55.467096+0000","last_fullsized":"2024-09-10T18:52:55.467096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:50.626514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:50.626514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.083006+0000","scrub_duration":0.0035366019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468209+0000","last_change":"2024-09-10T18:52:04.358732+0000","last_active":"2024-09-10T18:52:55.468209+0000","last_peered":"2024-09-10T18:52:55.468209+0000","last_clean":"2024-09-10T18:52:55.468209+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:52:55.468209+0000","last_undegraded":"2024-09-10T18:52:55.468209+0000","last_fullsized":"2024-09-10T18:52:55.468209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:04.358415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:04.358415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:06.405553+0000","scrub_duration":0.003743377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442910+0000","last_change":"2024-09-10T18:52:54.743442+0000","last_active":"2024-09-10T18:52:55.442910+0000","last_peered":"2024-09-10T18:52:55.442910+0000","last_clean":"2024-09-10T18:52:55.442910+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:52:55.442910+0000","last_undegraded":"2024-09-10T18:52:55.442910+0000","last_fullsized":"2024-09-10T18:52:55.442910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:52:54.743264+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:14.088219+0000","scrub_duration":0.010419701999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464720+0000","last_change":"2024-09-10T18:51:39.623163+0000","last_active":"2024-09-10T18:52:55.464720+0000","last_peered":"2024-09-10T18:52:55.464720+0000","last_clean":"2024-09-10T18:52:55.464720+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:52:55.464720+0000","last_undegraded":"2024-09-10T18:52:55.464720+0000","last_fullsized":"2024-09-10T18:52:55.464720+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:39.623005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:39.623005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:07.432652+0000","scrub_duration":0.0034195559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":116,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.470626+0000","last_change":"2024-09-10T18:51:33.452673+0000","last_active":"2024-09-10T18:52:55.470626+0000","last_peered":"2024-09-10T18:52:55.470626+0000","last_clean":"2024-09-10T18:52:55.470626+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:52:55.470626+0000","last_undegraded":"2024-09-10T18:52:55.470626+0000","last_fullsized":"2024-09-10T18:52:55.470626+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:51:33.452504+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:51:33.452504+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041999869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465836+0000","last_change":"2024-09-10T18:52:03.329811+0000","last_active":"2024-09-10T18:52:55.465836+0000","last_peered":"2024-09-10T18:52:55.465836+0000","last_clean":"2024-09-10T18:52:55.465836+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:52:55.465836+0000","last_undegraded":"2024-09-10T18:52:55.465836+0000","last_fullsized":"2024-09-10T18:52:55.465836+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:52:03.329632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.640377+0000","scrub_duration":0.0032347410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467718+0000","last_change":"2024-09-10T18:52:21.732959+0000","last_active":"2024-09-10T18:52:55.467718+0000","last_peered":"2024-09-10T18:52:55.467718+0000","last_clean":"2024-09-10T18:52:55.467718+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:52:55.467718+0000","last_undegraded":"2024-09-10T18:52:55.467718+0000","last_fullsized":"2024-09-10T18:52:55.467718+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:21.732812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:21.732812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:38.740200+0000","scrub_duration":0.0034512259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465969+0000","last_change":"2024-09-10T18:52:27.343427+0000","last_active":"2024-09-10T18:52:55.465969+0000","last_peered":"2024-09-10T18:52:55.465969+0000","last_clean":"2024-09-10T18:52:55.465969+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:52:55.465969+0000","last_undegraded":"2024-09-10T18:52:55.465969+0000","last_fullsized":"2024-09-10T18:52:55.465969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:52:27.343260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:27.343260+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.393322+0000","scrub_duration":0.005128049,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.440737+0000","last_change":"2024-09-10T18:52:07.455227+0000","last_active":"2024-09-10T18:52:55.440737+0000","last_peered":"2024-09-10T18:52:55.440737+0000","last_clean":"2024-09-10T18:52:55.440737+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:52:55.440737+0000","last_undegraded":"2024-09-10T18:52:55.440737+0000","last_fullsized":"2024-09-10T18:52:55.440737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.455062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.455062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.587408+0000","scrub_duration":0.0037580970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464701+0000","last_change":"2024-09-10T18:51:52.321821+0000","last_active":"2024-09-10T18:52:55.464701+0000","last_peered":"2024-09-10T18:52:55.464701+0000","last_clean":"2024-09-10T18:52:55.464701+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:52:55.464701+0000","last_undegraded":"2024-09-10T18:52:55.464701+0000","last_fullsized":"2024-09-10T18:52:55.464701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:52.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:52.321671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:02.059416+0000","scrub_duration":0.003316977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.470283+0000","last_change":"2024-09-10T18:52:40.442208+0000","last_active":"2024-09-10T18:52:55.470283+0000","last_peered":"2024-09-10T18:52:55.470283+0000","last_clean":"2024-09-10T18:52:55.470283+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:52:55.470283+0000","last_undegraded":"2024-09-10T18:52:55.470283+0000","last_fullsized":"2024-09-10T18:52:55.470283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:40.442034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:40.442034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.415342+0000","scrub_duration":0.0034431399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"77'10","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465722+0000","last_change":"2024-09-10T18:52:29.670736+0000","last_active":"2024-09-10T18:52:55.465722+0000","last_peered":"2024-09-10T18:52:55.465722+0000","last_clean":"2024-09-10T18:52:55.465722+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:52:55.465722+0000","last_undegraded":"2024-09-10T18:52:55.465722+0000","last_fullsized":"2024-09-10T18:52:55.465722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:29.670555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.670555+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:43.035420+0000","scrub_duration":0.0047380190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467538+0000","last_change":"2024-09-10T18:52:29.292088+0000","last_active":"2024-09-10T18:52:55.467538+0000","last_peered":"2024-09-10T18:52:55.467538+0000","last_clean":"2024-09-10T18:52:55.467538+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:52:55.467538+0000","last_undegraded":"2024-09-10T18:52:55.467538+0000","last_fullsized":"2024-09-10T18:52:55.467538+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:29.291936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.291936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:55.077335+0000","scrub_duration":0.003409966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465808+0000","last_change":"2024-09-10T18:52:29.591674+0000","last_active":"2024-09-10T18:52:55.465808+0000","last_peered":"2024-09-10T18:52:55.465808+0000","last_clean":"2024-09-10T18:52:55.465808+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:52:55.465808+0000","last_undegraded":"2024-09-10T18:52:55.465808+0000","last_fullsized":"2024-09-10T18:52:55.465808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:29.591530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:29.591530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.371798+0000","scrub_duration":0.003270528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465238+0000","last_change":"2024-09-10T18:52:25.583663+0000","last_active":"2024-09-10T18:52:55.465238+0000","last_peered":"2024-09-10T18:52:55.465238+0000","last_clean":"2024-09-10T18:52:55.465238+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:52:55.465238+0000","last_undegraded":"2024-09-10T18:52:55.465238+0000","last_fullsized":"2024-09-10T18:52:55.465238+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.583492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:52.226811+0000","scrub_duration":0.0033363329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.468059+0000","last_change":"2024-09-10T18:51:47.304256+0000","last_active":"2024-09-10T18:52:55.468059+0000","last_peered":"2024-09-10T18:52:55.468059+0000","last_clean":"2024-09-10T18:52:55.468059+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:52:55.468059+0000","last_undegraded":"2024-09-10T18:52:55.468059+0000","last_fullsized":"2024-09-10T18:52:55.468059+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.304153+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.010387+0000","scrub_duration":0.012604194000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441174+0000","last_change":"2024-09-10T18:51:53.688495+0000","last_active":"2024-09-10T18:52:55.441174+0000","last_peered":"2024-09-10T18:52:55.441174+0000","last_clean":"2024-09-10T18:52:55.441174+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:52:55.441174+0000","last_undegraded":"2024-09-10T18:52:55.441174+0000","last_fullsized":"2024-09-10T18:52:55.441174+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:51:53.688302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:53.688302+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:00.064936+0000","scrub_duration":0.0040928329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:52:57.479398+0000","last_change":"2024-09-10T18:52:57.479287+0000","last_active":"2024-09-10T18:52:57.479398+0000","last_peered":"2024-09-10T18:52:57.479398+0000","last_clean":"2024-09-10T18:52:57.479398+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:52:57.479398+0000","last_undegraded":"2024-09-10T18:52:57.479398+0000","last_fullsized":"2024-09-10T18:52:57.479398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:57.479141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:52:57.479141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:21.821379+0000","scrub_duration":0.0036510710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":72,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:53:00.404294+0000","last_change":"2024-09-10T18:53:00.404213+0000","last_active":"2024-09-10T18:53:00.404294+0000","last_peered":"2024-09-10T18:53:00.404294+0000","last_clean":"2024-09-10T18:53:00.404294+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:53:00.404294+0000","last_undegraded":"2024-09-10T18:53:00.404294+0000","last_fullsized":"2024-09-10T18:53:00.404294+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:00.404042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:53:00.404042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:07.431124+0000","scrub_duration":0.003469691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467579+0000","last_change":"2024-09-10T18:52:05.179430+0000","last_active":"2024-09-10T18:52:55.467579+0000","last_peered":"2024-09-10T18:52:55.467579+0000","last_clean":"2024-09-10T18:52:55.467579+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:52:55.467579+0000","last_undegraded":"2024-09-10T18:52:55.467579+0000","last_fullsized":"2024-09-10T18:52:55.467579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:05.179207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:05.179207+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.065476+0000","scrub_duration":0.0057240640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":119,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435986+0000","last_change":"2024-09-10T18:52:55.258116+0000","last_active":"2024-09-10T18:52:55.435986+0000","last_peered":"2024-09-10T18:52:55.435986+0000","last_clean":"2024-09-10T18:52:55.435986+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:52:55.435986+0000","last_undegraded":"2024-09-10T18:52:55.435986+0000","last_fullsized":"2024-09-10T18:52:55.435986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:52:55.257933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:55.257933+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:18.698683+0000","scrub_duration":0.0043042050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":120,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.968537+0000","last_change":"2024-09-10T18:51:47.570557+0000","last_active":"2024-09-10T18:52:55.968537+0000","last_peered":"2024-09-10T18:52:55.968537+0000","last_clean":"2024-09-10T18:52:55.968537+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:52:55.968537+0000","last_undegraded":"2024-09-10T18:52:55.968537+0000","last_fullsized":"2024-09-10T18:52:55.968537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:51:47.570369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:47.570369+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:12.658418+0000","scrub_duration":0.0040413760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442589+0000","last_change":"2024-09-10T18:52:40.755231+0000","last_active":"2024-09-10T18:52:55.442589+0000","last_peered":"2024-09-10T18:52:55.442589+0000","last_clean":"2024-09-10T18:52:55.442589+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:52:55.442589+0000","last_undegraded":"2024-09-10T18:52:55.442589+0000","last_fullsized":"2024-09-10T18:52:55.442589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:52:40.755118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:54.859229+0000","scrub_duration":0.0030712019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:56.406159+0000","last_change":"2024-09-10T18:52:56.406032+0000","last_active":"2024-09-10T18:52:56.406159+0000","last_peered":"2024-09-10T18:52:56.406159+0000","last_clean":"2024-09-10T18:52:56.406159+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:52:56.406159+0000","last_undegraded":"2024-09-10T18:52:56.406159+0000","last_fullsized":"2024-09-10T18:52:56.406159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:56.405864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:56.405864+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:19.866543+0000","scrub_duration":0.0049478969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.439757+0000","last_change":"2024-09-10T18:51:59.561431+0000","last_active":"2024-09-10T18:52:55.439757+0000","last_peered":"2024-09-10T18:52:55.439757+0000","last_clean":"2024-09-10T18:52:55.439757+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:52:55.439757+0000","last_undegraded":"2024-09-10T18:52:55.439757+0000","last_fullsized":"2024-09-10T18:52:55.439757+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:59.561306+0000","last_clean_scrub_stamp":"2024-09-10T18:51:59.561306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.793374+0000","scrub_duration":0.003474277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466880+0000","last_change":"2024-09-10T18:52:01.485093+0000","last_active":"2024-09-10T18:52:55.466880+0000","last_peered":"2024-09-10T18:52:55.466880+0000","last_clean":"2024-09-10T18:52:55.466880+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:52:55.466880+0000","last_undegraded":"2024-09-10T18:52:55.466880+0000","last_fullsized":"2024-09-10T18:52:55.466880+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:01.484734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:01.484734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:22.349048+0000","scrub_duration":0.0054940589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.439683+0000","last_change":"2024-09-10T18:52:11.451951+0000","last_active":"2024-09-10T18:52:55.439683+0000","last_peered":"2024-09-10T18:52:55.439683+0000","last_clean":"2024-09-10T18:52:55.439683+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:52:55.439683+0000","last_undegraded":"2024-09-10T18:52:55.439683+0000","last_fullsized":"2024-09-10T18:52:55.439683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.451797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.451797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:26.258751+0000","scrub_duration":0.0033137470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"77'4","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.463849+0000","last_change":"2024-09-10T18:52:11.496842+0000","last_active":"2024-09-10T18:52:55.463849+0000","last_peered":"2024-09-10T18:52:55.463849+0000","last_clean":"2024-09-10T18:52:55.463849+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:52:55.463849+0000","last_undegraded":"2024-09-10T18:52:55.463849+0000","last_fullsized":"2024-09-10T18:52:55.463849+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'4","last_scrub_stamp":"2024-09-10T18:52:11.496670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.496670+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:24.908807+0000","scrub_duration":0.0041783929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464464+0000","last_change":"2024-09-10T18:52:00.390577+0000","last_active":"2024-09-10T18:52:55.464464+0000","last_peered":"2024-09-10T18:52:55.464464+0000","last_clean":"2024-09-10T18:52:55.464464+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:52:55.464464+0000","last_undegraded":"2024-09-10T18:52:55.464464+0000","last_fullsized":"2024-09-10T18:52:55.464464+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:00.390415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:00.390415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.045571+0000","scrub_duration":0.00325318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443639+0000","last_change":"2024-09-10T18:52:46.829741+0000","last_active":"2024-09-10T18:52:55.443639+0000","last_peered":"2024-09-10T18:52:55.443639+0000","last_clean":"2024-09-10T18:52:55.443639+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:52:55.443639+0000","last_undegraded":"2024-09-10T18:52:55.443639+0000","last_fullsized":"2024-09-10T18:52:55.443639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:52:46.829507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:55.683255+0000","scrub_duration":0.0045636449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465322+0000","last_change":"2024-09-10T18:52:07.197975+0000","last_active":"2024-09-10T18:52:55.465322+0000","last_peered":"2024-09-10T18:52:55.465322+0000","last_clean":"2024-09-10T18:52:55.465322+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:52:55.465322+0000","last_undegraded":"2024-09-10T18:52:55.465322+0000","last_fullsized":"2024-09-10T18:52:55.465322+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:07.197730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:07.197730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.782447+0000","scrub_duration":0.003470014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466233+0000","last_change":"2024-09-10T18:52:48.328798+0000","last_active":"2024-09-10T18:52:55.466233+0000","last_peered":"2024-09-10T18:52:55.466233+0000","last_clean":"2024-09-10T18:52:55.466233+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:52:55.466233+0000","last_undegraded":"2024-09-10T18:52:55.466233+0000","last_fullsized":"2024-09-10T18:52:55.466233+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:52:48.328639+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:58.175263+0000","scrub_duration":0.010391842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.469075+0000","last_change":"2024-09-10T18:52:26.617200+0000","last_active":"2024-09-10T18:52:55.469075+0000","last_peered":"2024-09-10T18:52:55.469075+0000","last_clean":"2024-09-10T18:52:55.469075+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:52:55.469075+0000","last_undegraded":"2024-09-10T18:52:55.469075+0000","last_fullsized":"2024-09-10T18:52:55.469075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:26.617026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:26.617026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:29.957670+0000","scrub_duration":0.0041383920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":123,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.971511+0000","last_change":"2024-09-10T18:52:24.727019+0000","last_active":"2024-09-10T18:52:55.971511+0000","last_peered":"2024-09-10T18:52:55.971511+0000","last_clean":"2024-09-10T18:52:55.971511+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:52:55.971511+0000","last_undegraded":"2024-09-10T18:52:55.971511+0000","last_fullsized":"2024-09-10T18:52:55.971511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:24.726861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.726861+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.604127+0000","scrub_duration":0.0039722289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466693+0000","last_change":"2024-09-10T18:52:33.564446+0000","last_active":"2024-09-10T18:52:55.466693+0000","last_peered":"2024-09-10T18:52:55.466693+0000","last_clean":"2024-09-10T18:52:55.466693+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:52:55.466693+0000","last_undegraded":"2024-09-10T18:52:55.466693+0000","last_fullsized":"2024-09-10T18:52:55.466693+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:33.564283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:52:33.564283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:35.408845+0000","scrub_duration":0.0035323889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465727+0000","last_change":"2024-09-10T18:52:36.577038+0000","last_active":"2024-09-10T18:52:55.465727+0000","last_peered":"2024-09-10T18:52:55.465727+0000","last_clean":"2024-09-10T18:52:55.465727+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:52:55.465727+0000","last_undegraded":"2024-09-10T18:52:55.465727+0000","last_fullsized":"2024-09-10T18:52:55.465727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:36.576871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:36.576871+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:37.551929+0000","scrub_duration":0.0043998120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466221+0000","last_change":"2024-09-10T18:51:56.575310+0000","last_active":"2024-09-10T18:52:55.466221+0000","last_peered":"2024-09-10T18:52:55.466221+0000","last_clean":"2024-09-10T18:52:55.466221+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:52:55.466221+0000","last_undegraded":"2024-09-10T18:52:55.466221+0000","last_fullsized":"2024-09-10T18:52:55.466221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:51:56.575153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:10.564777+0000","scrub_duration":0.0033985280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.968462+0000","last_change":"2024-09-10T18:52:49.569965+0000","last_active":"2024-09-10T18:52:55.968462+0000","last_peered":"2024-09-10T18:52:55.968462+0000","last_clean":"2024-09-10T18:52:55.968462+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:52:55.968462+0000","last_undegraded":"2024-09-10T18:52:55.968462+0000","last_fullsized":"2024-09-10T18:52:55.968462+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:52:49.569813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:49.569813+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:03.970871+0000","scrub_duration":0.0048404019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435515+0000","last_change":"2024-09-10T18:52:22.188985+0000","last_active":"2024-09-10T18:52:55.435515+0000","last_peered":"2024-09-10T18:52:55.435515+0000","last_clean":"2024-09-10T18:52:55.435515+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:52:55.435515+0000","last_undegraded":"2024-09-10T18:52:55.435515+0000","last_fullsized":"2024-09-10T18:52:55.435515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:22.188834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.188834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.440531+0000","scrub_duration":0.0068943040000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465815+0000","last_change":"2024-09-10T18:52:08.245784+0000","last_active":"2024-09-10T18:52:55.465815+0000","last_peered":"2024-09-10T18:52:55.465815+0000","last_clean":"2024-09-10T18:52:55.465815+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:52:55.465815+0000","last_undegraded":"2024-09-10T18:52:55.465815+0000","last_fullsized":"2024-09-10T18:52:55.465815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:08.245617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:08.245617+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:11.667622+0000","scrub_duration":0.0044282050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464645+0000","last_change":"2024-09-10T18:52:25.757066+0000","last_active":"2024-09-10T18:52:55.464645+0000","last_peered":"2024-09-10T18:52:55.464645+0000","last_clean":"2024-09-10T18:52:55.464645+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:52:55.464645+0000","last_undegraded":"2024-09-10T18:52:55.464645+0000","last_fullsized":"2024-09-10T18:52:55.464645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:25.756907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:25.756907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.523280+0000","scrub_duration":0.0032708659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.463671+0000","last_change":"2024-09-10T18:52:11.856507+0000","last_active":"2024-09-10T18:52:55.463671+0000","last_peered":"2024-09-10T18:52:55.463671+0000","last_clean":"2024-09-10T18:52:55.463671+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:52:55.463671+0000","last_undegraded":"2024-09-10T18:52:55.463671+0000","last_fullsized":"2024-09-10T18:52:55.463671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:11.856337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.856337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.241036+0000","scrub_duration":0.0034053619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.464992+0000","last_change":"2024-09-10T18:52:14.538993+0000","last_active":"2024-09-10T18:52:55.464992+0000","last_peered":"2024-09-10T18:52:55.464992+0000","last_clean":"2024-09-10T18:52:55.464992+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:52:55.464992+0000","last_undegraded":"2024-09-10T18:52:55.464992+0000","last_fullsized":"2024-09-10T18:52:55.464992+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:14.538838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:14.538838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.876083+0000","scrub_duration":0.0042549700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"77'10","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.435003+0000","last_change":"2024-09-10T18:52:11.222775+0000","last_active":"2024-09-10T18:52:55.435003+0000","last_peered":"2024-09-10T18:52:55.435003+0000","last_clean":"2024-09-10T18:52:55.435003+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:52:55.435003+0000","last_undegraded":"2024-09-10T18:52:55.435003+0000","last_fullsized":"2024-09-10T18:52:55.435003+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:52:11.222598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:11.222598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:27.658086+0000","scrub_duration":0.0051546400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467295+0000","last_change":"2024-09-10T18:52:38.373031+0000","last_active":"2024-09-10T18:52:55.467295+0000","last_peered":"2024-09-10T18:52:55.467295+0000","last_clean":"2024-09-10T18:52:55.467295+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:52:55.467295+0000","last_undegraded":"2024-09-10T18:52:55.467295+0000","last_fullsized":"2024-09-10T18:52:55.467295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:38.372968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:38.372968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:56.815767+0000","scrub_duration":0.0025253519999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434785+0000","last_change":"2024-09-10T18:52:20.152534+0000","last_active":"2024-09-10T18:52:55.434785+0000","last_peered":"2024-09-10T18:52:55.434785+0000","last_clean":"2024-09-10T18:52:55.434785+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:52:55.434785+0000","last_undegraded":"2024-09-10T18:52:55.434785+0000","last_fullsized":"2024-09-10T18:52:55.434785+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:52:20.152369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:47.436481+0000","scrub_duration":0.0033349780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441720+0000","last_change":"2024-09-10T18:51:57.533423+0000","last_active":"2024-09-10T18:52:55.441720+0000","last_peered":"2024-09-10T18:52:55.441720+0000","last_clean":"2024-09-10T18:52:55.441720+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:52:55.441720+0000","last_undegraded":"2024-09-10T18:52:55.441720+0000","last_fullsized":"2024-09-10T18:52:55.441720+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:57.533281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.533281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:13.018032+0000","scrub_duration":0.0030396329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467431+0000","last_change":"2024-09-10T18:52:39.623937+0000","last_active":"2024-09-10T18:52:55.467431+0000","last_peered":"2024-09-10T18:52:55.467431+0000","last_clean":"2024-09-10T18:52:55.467431+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:52:55.467431+0000","last_undegraded":"2024-09-10T18:52:55.467431+0000","last_fullsized":"2024-09-10T18:52:55.467431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:52:39.623723+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.931489+0000","scrub_duration":0.0077034729999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466598+0000","last_change":"2024-09-10T18:52:32.647147+0000","last_active":"2024-09-10T18:52:55.466598+0000","last_peered":"2024-09-10T18:52:55.466598+0000","last_clean":"2024-09-10T18:52:55.466598+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:52:55.466598+0000","last_undegraded":"2024-09-10T18:52:55.466598+0000","last_fullsized":"2024-09-10T18:52:55.466598+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:32.646982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:32.646982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:44.711757+0000","scrub_duration":0.0033406109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441303+0000","last_change":"2024-09-10T18:51:38.650934+0000","last_active":"2024-09-10T18:52:55.441303+0000","last_peered":"2024-09-10T18:52:55.441303+0000","last_clean":"2024-09-10T18:52:55.441303+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:52:55.441303+0000","last_undegraded":"2024-09-10T18:52:55.441303+0000","last_fullsized":"2024-09-10T18:52:55.441303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:51:38.650775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.247838+0000","scrub_duration":0.0033420289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:57.399177+0000","last_change":"2024-09-10T18:52:57.399054+0000","last_active":"2024-09-10T18:52:57.399177+0000","last_peered":"2024-09-10T18:52:57.399177+0000","last_clean":"2024-09-10T18:52:57.399177+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:52:57.399177+0000","last_undegraded":"2024-09-10T18:52:57.399177+0000","last_fullsized":"2024-09-10T18:52:57.399177+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:57.398886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:52:57.398886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:10.162921+0000","scrub_duration":0.0038228759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466402+0000","last_change":"2024-09-10T18:52:22.698451+0000","last_active":"2024-09-10T18:52:55.466402+0000","last_peered":"2024-09-10T18:52:55.466402+0000","last_clean":"2024-09-10T18:52:55.466402+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:52:55.466402+0000","last_undegraded":"2024-09-10T18:52:55.466402+0000","last_fullsized":"2024-09-10T18:52:55.466402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:22.698371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:22.698371+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:46.745548+0000","scrub_duration":0.0094695119999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465256+0000","last_change":"2024-09-10T18:51:49.224075+0000","last_active":"2024-09-10T18:52:55.465256+0000","last_peered":"2024-09-10T18:52:55.465256+0000","last_clean":"2024-09-10T18:52:55.465256+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:52:55.465256+0000","last_undegraded":"2024-09-10T18:52:55.465256+0000","last_fullsized":"2024-09-10T18:52:55.465256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:49.223905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:49.223905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:14.495696+0000","scrub_duration":0.003346421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441913+0000","last_change":"2024-09-10T18:52:38.759578+0000","last_active":"2024-09-10T18:52:55.441913+0000","last_peered":"2024-09-10T18:52:55.441913+0000","last_clean":"2024-09-10T18:52:55.441913+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:52:55.441913+0000","last_undegraded":"2024-09-10T18:52:55.441913+0000","last_fullsized":"2024-09-10T18:52:55.441913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:38.759381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:38.759381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:59.210016+0000","scrub_duration":0.0037681210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442649+0000","last_change":"2024-09-10T18:52:32.749810+0000","last_active":"2024-09-10T18:52:55.442649+0000","last_peered":"2024-09-10T18:52:55.442649+0000","last_clean":"2024-09-10T18:52:55.442649+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:52:55.442649+0000","last_undegraded":"2024-09-10T18:52:55.442649+0000","last_fullsized":"2024-09-10T18:52:55.442649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:32.749566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:32.749566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:46.056687+0000","scrub_duration":0.0042025500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465309+0000","last_change":"2024-09-10T18:52:44.380396+0000","last_active":"2024-09-10T18:52:55.465309+0000","last_peered":"2024-09-10T18:52:55.465309+0000","last_clean":"2024-09-10T18:52:55.465309+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:52:55.465309+0000","last_undegraded":"2024-09-10T18:52:55.465309+0000","last_fullsized":"2024-09-10T18:52:55.465309+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:52:44.380117+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:00.476762+0000","scrub_duration":0.0083087079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":101,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:53:00.429498+0000","last_change":"2024-09-10T18:53:00.429413+0000","last_active":"2024-09-10T18:53:00.429498+0000","last_peered":"2024-09-10T18:53:00.429498+0000","last_clean":"2024-09-10T18:53:00.429498+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:53:00.429498+0000","last_undegraded":"2024-09-10T18:53:00.429498+0000","last_fullsized":"2024-09-10T18:53:00.429498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:53:00.429232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:53:00.429232+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:22.199557+0000","scrub_duration":0.0050452159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.463303+0000","last_change":"2024-09-10T18:52:15.859394+0000","last_active":"2024-09-10T18:52:55.463303+0000","last_peered":"2024-09-10T18:52:55.463303+0000","last_clean":"2024-09-10T18:52:55.463303+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:52:55.463303+0000","last_undegraded":"2024-09-10T18:52:55.463303+0000","last_fullsized":"2024-09-10T18:52:55.463303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:15.859239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:15.859239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:17.490895+0000","scrub_duration":0.0037198740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466771+0000","last_change":"2024-09-10T18:52:24.615123+0000","last_active":"2024-09-10T18:52:55.466771+0000","last_peered":"2024-09-10T18:52:55.466771+0000","last_clean":"2024-09-10T18:52:55.466771+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:52:55.466771+0000","last_undegraded":"2024-09-10T18:52:55.466771+0000","last_fullsized":"2024-09-10T18:52:55.466771+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:24.614967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:24.614967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:40.011991+0000","scrub_duration":0.0037677969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.467073+0000","last_change":"2024-09-10T18:52:31.350686+0000","last_active":"2024-09-10T18:52:55.467073+0000","last_peered":"2024-09-10T18:52:55.467073+0000","last_clean":"2024-09-10T18:52:55.467073+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:52:55.467073+0000","last_undegraded":"2024-09-10T18:52:55.467073+0000","last_fullsized":"2024-09-10T18:52:55.467073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:31.350539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:31.350539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:37.972404+0000","scrub_duration":0.003698621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.441474+0000","last_change":"2024-09-10T18:52:52.790598+0000","last_active":"2024-09-10T18:52:55.441474+0000","last_peered":"2024-09-10T18:52:55.441474+0000","last_clean":"2024-09-10T18:52:55.441474+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:52:55.441474+0000","last_undegraded":"2024-09-10T18:52:55.441474+0000","last_fullsized":"2024-09-10T18:52:55.441474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:52:52.790437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:52.790437+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:54.157707+0000","scrub_duration":0.0046273140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442183+0000","last_change":"2024-09-10T18:52:28.733481+0000","last_active":"2024-09-10T18:52:55.442183+0000","last_peered":"2024-09-10T18:52:55.442183+0000","last_clean":"2024-09-10T18:52:55.442183+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:52:55.442183+0000","last_undegraded":"2024-09-10T18:52:55.442183+0000","last_fullsized":"2024-09-10T18:52:55.442183+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:28.733333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:28.733333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:51.670048+0000","scrub_duration":0.0031761739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":135,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.968625+0000","last_change":"2024-09-10T18:51:55.299569+0000","last_active":"2024-09-10T18:52:55.968625+0000","last_peered":"2024-09-10T18:52:55.968625+0000","last_clean":"2024-09-10T18:52:55.968625+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:52:55.968625+0000","last_undegraded":"2024-09-10T18:52:55.968625+0000","last_fullsized":"2024-09-10T18:52:55.968625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:51:55.299396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:51:55.299396+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:01.997434+0000","scrub_duration":0.0038567179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:52:59.241810+0000","last_change":"2024-09-10T18:52:59.241722+0000","last_active":"2024-09-10T18:52:59.241810+0000","last_peered":"2024-09-10T18:52:59.241810+0000","last_clean":"2024-09-10T18:52:59.241810+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:52:59.241810+0000","last_undegraded":"2024-09-10T18:52:59.241810+0000","last_fullsized":"2024-09-10T18:52:59.241810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:59.241552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:52:59.241552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:02.724788+0000","scrub_duration":0.0035493170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.465188+0000","last_change":"2024-09-10T18:52:18.566245+0000","last_active":"2024-09-10T18:52:55.465188+0000","last_peered":"2024-09-10T18:52:55.465188+0000","last_clean":"2024-09-10T18:52:55.465188+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:52:55.465188+0000","last_undegraded":"2024-09-10T18:52:55.465188+0000","last_fullsized":"2024-09-10T18:52:55.465188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:52:18.566056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:52:18.566056+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:48.529822+0000","scrub_duration":0.0040203510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.466009+0000","last_change":"2024-09-10T18:52:09.509564+0000","last_active":"2024-09-10T18:52:55.466009+0000","last_peered":"2024-09-10T18:52:55.466009+0000","last_clean":"2024-09-10T18:52:55.466009+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:52:55.466009+0000","last_undegraded":"2024-09-10T18:52:55.466009+0000","last_fullsized":"2024-09-10T18:52:55.466009+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:52:09.509384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:52:09.509384+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:32.310603+0000","scrub_duration":0.0050191510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":74,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-10T18:53:00.397045+0000","last_change":"2024-09-10T18:53:00.396942+0000","last_active":"2024-09-10T18:53:00.397045+0000","last_peered":"2024-09-10T18:53:00.397045+0000","last_clean":"2024-09-10T18:53:00.397045+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:53:00.397045+0000","last_undegraded":"2024-09-10T18:53:00.397045+0000","last_fullsized":"2024-09-10T18:53:00.397045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:00.396776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:53:00.396776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:54:19.382307+0000","scrub_duration":0.003387386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443548+0000","last_change":"2024-09-10T18:52:06.609575+0000","last_active":"2024-09-10T18:52:55.443548+0000","last_peered":"2024-09-10T18:52:55.443548+0000","last_clean":"2024-09-10T18:52:55.443548+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:52:55.443548+0000","last_undegraded":"2024-09-10T18:52:55.443548+0000","last_fullsized":"2024-09-10T18:52:55.443548+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:06.609346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:52:06.609346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:31.679962+0000","scrub_duration":0.0031752439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.442816+0000","last_change":"2024-09-10T18:51:42.590260+0000","last_active":"2024-09-10T18:52:55.442816+0000","last_peered":"2024-09-10T18:52:55.442816+0000","last_clean":"2024-09-10T18:52:55.442816+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:52:55.442816+0000","last_undegraded":"2024-09-10T18:52:55.442816+0000","last_fullsized":"2024-09-10T18:52:55.442816+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:51:42.590071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:51:42.590071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:04.690989+0000","scrub_duration":0.003624726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.434738+0000","last_change":"2024-09-10T18:51:57.420671+0000","last_active":"2024-09-10T18:52:55.434738+0000","last_peered":"2024-09-10T18:52:55.434738+0000","last_clean":"2024-09-10T18:52:55.434738+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:52:55.434738+0000","last_undegraded":"2024-09-10T18:52:55.434738+0000","last_fullsized":"2024-09-10T18:52:55.434738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:51:57.420489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:51:57.420489+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:19.697937+0000","scrub_duration":0.0042932839999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-10T18:52:55.443058+0000","last_change":"2024-09-10T18:52:35.715864+0000","last_active":"2024-09-10T18:52:55.443058+0000","last_peered":"2024-09-10T18:52:55.443058+0000","last_clean":"2024-09-10T18:52:55.443058+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:52:55.443058+0000","last_undegraded":"2024-09-10T18:52:55.443058+0000","last_fullsized":"2024-09-10T18:52:55.443058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:52:35.715788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:52:35.715788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:53:56.123687+0000","scrub_duration":0.0023162970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":442,"ondisk_log_size":442,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972408,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298608,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446544,"statfs":{"total":95995035648,"available":95689261056,"internally_reserved":0,"allocated":1282048,"data_stored":911443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91600000000000004,"15min":0.93500000000000005},"min":{"1min":0.621,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3380000000000001,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.022,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.69399999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.264,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":1.044}]},{"osd":1,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.91400000000000003,"15min":0.91000000000000003},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.373,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.0109999999999999,"15min":1.046},"min":{"1min":0.65400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.409,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.85299999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":1.0089999999999999,"15min":1.0549999999999999},"min":{"1min":0.77200000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3779999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.6579999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.69799999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.236,"5min":1.353,"15min":1.353},"last":0.78500000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98499999999999999,"15min":1.0169999999999999},"min":{"1min":0.70699999999999996,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.3169999999999999,"5min":1.335,"15min":1.335},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.997,"5min":0.97199999999999998,"15min":0.96299999999999997},"min":{"1min":0.73599999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3280000000000001,"5min":1.905,"15min":1.905},"last":1.008}]},{"osd":4,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.97899999999999998,"15min":1.0189999999999999},"min":{"1min":0.65800000000000003,"5min":0.443,"15min":0.443},"max":{"1min":1.331,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.018,"5min":1.0269999999999999,"15min":1.0209999999999999},"min":{"1min":0.83299999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2749999999999999,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.1020000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98899999999999999,"15min":0.995},"min":{"1min":0.59999999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2370000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.1479999999999999,"15min":1.1899999999999999},"min":{"1min":0.77200000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.448,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.23}]},{"osd":6,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2130000000000001,"15min":1.2490000000000001},"min":{"1min":0.875,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4950000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.169},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1919999999999999},"min":{"1min":0.749,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3520000000000001,"5min":2.536,"15min":2.536},"last":1.1990000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299458,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1265664,"data_stored":909997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.88900000000000001,"15min":0.89100000000000001},"min":{"1min":0.55700000000000005,"5min":0.503,"15min":0.503},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.028,"5min":0.94099999999999995,"15min":0.93500000000000005},"min":{"1min":0.64800000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.22}]},{"osd":1,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.92700000000000005,"15min":0.88800000000000001},"min":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.446,"5min":1.446,"15min":1.446},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.92300000000000004,"15min":0.91500000000000004},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.226,"5min":1.361,"15min":1.361},"last":0.98799999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.94399999999999995,"15min":0.91900000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.337,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.196},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.91800000000000004,"15min":0.92700000000000005},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":1.2829999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96899999999999997,"15min":0.94699999999999995},"min":{"1min":0.73899999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.2869999999999999,"5min":1.4890000000000001,"15min":1.4890000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.008,"5min":0.96599999999999997,"15min":0.97099999999999997},"min":{"1min":0.71399999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.294,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.119}]},{"osd":4,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.85999999999999999},"min":{"1min":0.61399999999999999,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.337,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.622},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0029999999999999,"15min":1.0049999999999999},"min":{"1min":0.63800000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.353,"5min":1.744,"15min":1.744},"last":0.76900000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.997,"15min":0.96899999999999997},"min":{"1min":0.71599999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.0780000000000001,"15min":1.0840000000000001},"min":{"1min":0.76400000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.4259999999999999,"5min":1.665,"15min":1.665},"last":1.028}]},{"osd":7,"last update":"Tue Sep 10 18:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.054,"15min":1.0389999999999999},"min":{"1min":0.77200000000000002,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2889999999999999,"5min":1.573,"15min":1.573},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.149,"15min":1.139},"min":{"1min":0.98399999999999999,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.405,"5min":1.488,"15min":1.488},"last":1.0669999999999999}]}]},{"osd":5,"up_from":43,"seq":184683593804,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298016,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447136,"statfs":{"total":95995035648,"available":95689867264,"internally_reserved":0,"allocated":675840,"data_stored":319708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.88800000000000001,"15min":0.83999999999999997},"min":{"1min":0.66200000000000003,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.258,"5min":1.609,"15min":1.609},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.85199999999999998,"15min":0.85799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1990000000000001,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.86899999999999999,"15min":0.85499999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2949999999999999,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.91700000000000004,"15min":0.88100000000000001},"min":{"1min":0.70699999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2230000000000001,"5min":1.635,"15min":1.635},"last":0.95499999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91500000000000004,"15min":0.88500000000000001},"min":{"1min":0.628,"5min":0.496,"15min":0.496},"max":{"1min":1.274,"5min":1.496,"15min":1.496},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.999,"5min":0.93899999999999995,"15min":0.89400000000000002},"min":{"1min":0.65000000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3180000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":1.1950000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.92800000000000005,"15min":0.91200000000000003},"min":{"1min":0.64900000000000002,"5min":0.62,"15min":0.62},"max":{"1min":1.242,"5min":1.409,"15min":1.409},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.98399999999999999,"15min":0.93500000000000005},"min":{"1min":0.69199999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2090000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.155}]},{"osd":4,"last update":"Tue Sep 10 18:52:03 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.83099999999999996,"15min":0.77000000000000002},"min":{"1min":0.65100000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.1499999999999999,"5min":1.77,"15min":1.77},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92400000000000004,"15min":0.85299999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3580000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.123}]},{"osd":6,"last update":"Tue Sep 10 18:52:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0329999999999999,"15min":1.012},"min":{"1min":0.80200000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.294,"5min":1.462,"15min":1.462},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.0700000000000001,"15min":1.036},"min":{"1min":0.67100000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.4359999999999999,"5min":1.653,"15min":1.653},"last":1.236}]},{"osd":7,"last update":"Tue Sep 10 18:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.0329999999999999,"15min":1.0629999999999999},"min":{"1min":0.75900000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.4850000000000001,"5min":1.837,"15min":1.837},"last":1.04},{"interface":"front","average":{"1min":1.109,"5min":1.121,"15min":1.171},"min":{"1min":0.82199999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.464,"5min":1.7110000000000001,"15min":1.7110000000000001},"last":1.214}]}]},{"osd":4,"up_from":35,"seq":150323855448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447104,"statfs":{"total":95995035648,"available":95689834496,"internally_reserved":0,"allocated":708608,"data_stored":320958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.877,"15min":0.85099999999999998},"min":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.171,"5min":1.204,"15min":1.204},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91300000000000003,"15min":0.82599999999999996},"min":{"1min":0.60499999999999998,"5min":0.59899999999999998,"15min":0.56000000000000005},"max":{"1min":1.1870000000000001,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.218}]},{"osd":1,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.90400000000000003,"15min":0.88800000000000001},"min":{"1min":0.65300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.2869999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92600000000000005,"15min":0.84499999999999997},"min":{"1min":0.624,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.3480000000000001,"5min":1.456,"15min":1.456},"last":1.0629999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.93799999999999994,"15min":0.872},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.3160000000000001,"5min":1.425,"15min":1.425},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.94699999999999995,"15min":0.872},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.59699999999999998},"max":{"1min":1.3180000000000001,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":1.173}]},{"osd":3,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95699999999999996,"15min":0.88900000000000001},"min":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.56399999999999995},"max":{"1min":1.327,"5min":1.714,"15min":1.714},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.97099999999999997,"15min":0.91700000000000004},"min":{"1min":0.79800000000000004,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.339,"5min":1.391,"15min":1.391},"last":1.024}]},{"osd":5,"last update":"Tue Sep 10 18:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.90300000000000002,"15min":0.83499999999999996},"min":{"1min":0.63200000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3939999999999999,"5min":1.484,"15min":1.484},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97299999999999998,"15min":0.88100000000000001},"min":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.405,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":1.117}]},{"osd":6,"last update":"Tue Sep 10 18:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98999999999999999,"15min":0.93000000000000005},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.27,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.198,"5min":1.1200000000000001,"15min":1.0640000000000001},"min":{"1min":0.83099999999999996,"5min":0.754,"15min":0.754},"max":{"1min":1.5029999999999999,"5min":1.536,"15min":1.536},"last":0.97299999999999998}]},{"osd":7,"last update":"Tue Sep 10 18:52:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.167,"15min":1.177},"min":{"1min":0.746,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.6120000000000001,"5min":2.008,"15min":2.008},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.157,"15min":1.1719999999999999},"min":{"1min":0.873,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.4430000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0920000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051683,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":671744,"data_stored":320555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.64200000000000002,"15min":0.62},"min":{"1min":0.52500000000000002,"5min":0.40999999999999998,"15min":0.39900000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.71499999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.38800000000000001},"max":{"1min":0.90900000000000003,"5min":1.01,"15min":1.2529999999999999},"last":0.98099999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.432,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.014,"5min":1.032,"15min":1.2869999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.82599999999999996,"15min":0.73899999999999999},"min":{"1min":0.60499999999999998,"5min":0.48899999999999999,"15min":0.36699999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":1.274},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:52:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.84399999999999997,"15min":0.78100000000000003},"min":{"1min":0.50600000000000001,"5min":0.504,"15min":0.504},"max":{"1min":1.101,"5min":1.4399999999999999,"15min":1.4399999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.77700000000000002,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1739999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:52:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.079,"15min":0.99099999999999999},"min":{"1min":0.83299999999999996,"5min":0.70099999999999996,"15min":0.56999999999999995},"max":{"1min":1.373,"5min":1.778,"15min":1.778},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1579999999999999,"15min":1.069},"min":{"1min":0.86699999999999999,"5min":0.75700000000000001,"15min":0.59399999999999997},"max":{"1min":1.4239999999999999,"5min":1.952,"15min":1.952},"last":1.353}]},{"osd":5,"last update":"Tue Sep 10 18:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.113,"15min":1.0960000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.5469999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.97899999999999998},"min":{"1min":0.72999999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.4630000000000001,"5min":1.532,"15min":1.532},"last":1.0369999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.1639999999999999,"15min":1.1599999999999999},"min":{"1min":0.75800000000000001,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.069,"15min":1.0640000000000001},"min":{"1min":0.68600000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.3480000000000001,"5min":1.476,"15min":1.476},"last":1.2010000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.101,"15min":1.1100000000000001},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.2290000000000001,"15min":1.2390000000000001},"min":{"1min":0.81499999999999995,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":2.0019999999999998,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":1.409}]}]},{"osd":2,"up_from":19,"seq":81604378732,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":684032,"data_stored":320752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.69799999999999995,"15min":0.63},"min":{"1min":0.40100000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0049999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.80200000000000005,"15min":0.71699999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.30599999999999999},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.85299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:52:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70999999999999996,"15min":0.66900000000000004},"min":{"1min":0.54800000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.83699999999999997,"15min":0.74199999999999999},"min":{"1min":0.54200000000000004,"5min":0.45900000000000002,"15min":0.379},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.90100000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88200000000000001,"15min":0.81299999999999994},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4510000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.84499999999999997,"15min":0.80200000000000005},"min":{"1min":0.48799999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.69499999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:52:11 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.022,"15min":0.95599999999999996},"min":{"1min":0.745,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.425,"5min":1.542,"15min":1.542},"last":1.085},{"interface":"front","average":{"1min":1.107,"5min":1.115,"15min":1.0680000000000001},"min":{"1min":0.73299999999999998,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.627,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.111}]},{"osd":5,"last update":"Tue Sep 10 18:52:56 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0980000000000001,"15min":1.1120000000000001},"min":{"1min":0.67900000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.089,"15min":1.0109999999999999},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.63800000000000001},"max":{"1min":1.4039999999999999,"5min":1.643,"15min":1.643},"last":0.99099999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.163,"15min":1.181},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.024},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.091,"15min":1.0680000000000001},"min":{"1min":0.745,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":1.0469999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:52:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.194,"15min":1.204},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.208,"5min":1.139,"15min":1.1220000000000001},"min":{"1min":0.90400000000000003,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.175}]}]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302180,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442972,"statfs":{"total":95995035648,"available":95685603328,"internally_reserved":0,"allocated":679936,"data_stored":319982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69999999999999996,"15min":0.59199999999999997},"min":{"1min":0.49099999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.77500000000000002,"15min":0.65900000000000003},"min":{"1min":0.54400000000000004,"5min":0.46800000000000003,"15min":0.35299999999999998},"max":{"1min":1.857,"5min":2.181,"15min":2.181},"last":1.1659999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:52:21 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73699999999999999,"15min":0.65800000000000003},"min":{"1min":0.54200000000000004,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.83299999999999996,"15min":0.73599999999999999},"min":{"1min":0.60299999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.0449999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:52:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.877,"15min":0.81899999999999995},"min":{"1min":0.629,"5min":0.435,"15min":0.435},"max":{"1min":1.2210000000000001,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.77100000000000002,"15min":0.73399999999999999},"min":{"1min":0.59899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.68600000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:52:07 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.036,"15min":1},"min":{"1min":0.71899999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6830000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.26},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.97699999999999998,"15min":0.88900000000000001},"min":{"1min":0.78400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.075}]},{"osd":5,"last update":"Tue Sep 10 18:52:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.0029999999999999,"15min":1.0149999999999999},"min":{"1min":0.73899999999999999,"5min":0.38,"15min":0.38},"max":{"1min":1.3560000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1319999999999999,"15min":1.137},"min":{"1min":0.86899999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.696,"5min":1.9019999999999999,"15min":1.9019999999999999},"last":1.327}]},{"osd":6,"last update":"Tue Sep 10 18:52:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.145,"15min":1.087},"min":{"1min":1.0129999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.6080000000000001,"5min":1.976,"15min":1.976},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.77500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.7370000000000001,"5min":1.754,"15min":1.754},"last":1.2130000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:52:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2030000000000001,"15min":1.2210000000000001},"min":{"1min":0.89400000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.762,"5min":2.125,"15min":2.125},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.1439999999999999,"15min":1.1479999999999999},"min":{"1min":0.878,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.675,"5min":1.71,"15min":1.71},"last":1.296}]}]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298684,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446468,"statfs":{"total":95995035648,"available":95689183232,"internally_reserved":0,"allocated":1273856,"data_stored":909934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:52:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.65800000000000003,"15min":0.57699999999999996},"min":{"1min":0.42999999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.94799999999999995,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.76500000000000001,"15min":0.65000000000000002},"min":{"1min":0.496,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2050000000000001,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.78400000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72699999999999998,"15min":0.67400000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97499999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.79200000000000004,"15min":0.69099999999999995},"min":{"1min":0.48099999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.361,"5min":1.361,"15min":1.361},"last":0.98199999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:52:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.80600000000000005,"15min":0.73399999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0249999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.77800000000000002},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.48699999999999999},"max":{"1min":1.633,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.86799999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:52:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.98599999999999999,"15min":0.96499999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.071,"15min":1.0329999999999999},"min":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.6479999999999999,"5min":1.6479999999999999,"15min":1.6479999999999999},"last":1.349}]},{"osd":5,"last update":"Tue Sep 10 18:52:54 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.075,"15min":1.0089999999999999},"min":{"1min":0.70899999999999996,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5660000000000001,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.117,"15min":1.0900000000000001},"min":{"1min":0.78700000000000003,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7549999999999999,"5min":1.7549999999999999,"15min":1.7549999999999999},"last":2.1059999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:52:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.143,"15min":1.1080000000000001},"min":{"1min":0.85799999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.861,"5min":1.893,"15min":1.893},"last":1.9570000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.169,"15min":1.117},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":3.544,"5min":3.544,"15min":3.544},"last":1.198}]},{"osd":7,"last update":"Tue Sep 10 18:52:45 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.996,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.252,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":2.0339999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:53:05.442 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T18:53:05.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump 2024-09-10T18:53:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:05 smithi083 bash[18629]: cluster 2024-09-10T18:53:04.525271+0000 mgr.y (mgr.24526) 139 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:53:05.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:05 smithi083 bash[18629]: audit 2024-09-10T18:53:04.689510+0000 mgr.y (mgr.24526) 140 : audit [DBG] from='client.14799 v1:172.21.15.83:0/2255724547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:53:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:05 smithi083 bash[32381]: cluster 2024-09-10T18:53:04.525271+0000 mgr.y (mgr.24526) 139 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:53:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:05 smithi083 bash[32381]: audit 2024-09-10T18:53:04.689510+0000 mgr.y (mgr.24526) 140 : audit [DBG] from='client.14799 v1:172.21.15.83:0/2255724547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:53:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:05 smithi165 bash[22591]: cluster 2024-09-10T18:53:04.525271+0000 mgr.y (mgr.24526) 139 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T18:53:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:05 smithi165 bash[22591]: audit 2024-09-10T18:53:04.689510+0000 mgr.y (mgr.24526) 140 : audit [DBG] from='client.14799 v1:172.21.15.83:0/2255724547' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:53:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:06 smithi165 bash[22591]: cluster 2024-09-10T18:53:05.738608+0000 osd.0 (osd.0) 93 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:53:06.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:06 smithi165 bash[22591]: cluster 2024-09-10T18:53:05.743687+0000 osd.0 (osd.0) 94 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:53:07.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:06 smithi083 bash[18629]: cluster 2024-09-10T18:53:05.738608+0000 osd.0 (osd.0) 93 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:53:07.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:06 smithi083 bash[18629]: cluster 2024-09-10T18:53:05.743687+0000 osd.0 (osd.0) 94 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:53:07.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:06 smithi083 bash[32381]: cluster 2024-09-10T18:53:05.738608+0000 osd.0 (osd.0) 93 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:53:07.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:06 smithi083 bash[32381]: cluster 2024-09-10T18:53:05.743687+0000 osd.0 (osd.0) 94 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:53:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:07 smithi165 bash[22591]: cluster 2024-09-10T18:53:06.526728+0000 mgr.y (mgr.24526) 141 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:07.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:07 smithi083 bash[18629]: cluster 2024-09-10T18:53:06.526728+0000 mgr.y (mgr.24526) 141 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:07 smithi083 bash[32381]: cluster 2024-09-10T18:53:06.526728+0000 mgr.y (mgr.24526) 141 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:08 smithi165 bash[22591]: cluster 2024-09-10T18:53:07.218432+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.c scrub starts 2024-09-10T18:53:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:08 smithi165 bash[22591]: cluster 2024-09-10T18:53:07.222076+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.c scrub ok 2024-09-10T18:53:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:08 smithi165 bash[22591]: audit 2024-09-10T18:53:07.554025+0000 mgr.y (mgr.24526) 142 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:08 smithi165 bash[22591]: cluster 2024-09-10T18:53:07.725774+0000 osd.3 (osd.3) 109 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:53:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:08 smithi165 bash[22591]: cluster 2024-09-10T18:53:07.729113+0000 osd.3 (osd.3) 110 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:53:09.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[18629]: cluster 2024-09-10T18:53:07.218432+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.c scrub starts 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[18629]: cluster 2024-09-10T18:53:07.222076+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.c scrub ok 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[18629]: audit 2024-09-10T18:53:07.554025+0000 mgr.y (mgr.24526) 142 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[18629]: cluster 2024-09-10T18:53:07.725774+0000 osd.3 (osd.3) 109 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[18629]: cluster 2024-09-10T18:53:07.729113+0000 osd.3 (osd.3) 110 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[32381]: cluster 2024-09-10T18:53:07.218432+0000 osd.4 (osd.4) 99 : cluster [DBG] 4.c scrub starts 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[32381]: cluster 2024-09-10T18:53:07.222076+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.c scrub ok 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[32381]: audit 2024-09-10T18:53:07.554025+0000 mgr.y (mgr.24526) 142 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[32381]: cluster 2024-09-10T18:53:07.725774+0000 osd.3 (osd.3) 109 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:53:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:08 smithi083 bash[32381]: cluster 2024-09-10T18:53:07.729113+0000 osd.3 (osd.3) 110 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:53:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:09 smithi165 bash[22591]: cluster 2024-09-10T18:53:08.172394+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:53:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:09 smithi165 bash[22591]: cluster 2024-09-10T18:53:08.175654+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:53:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:09 smithi165 bash[22591]: cluster 2024-09-10T18:53:08.527427+0000 mgr.y (mgr.24526) 143 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-10T18:53:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:09 smithi165 bash[22591]: cluster 2024-09-10T18:53:08.749263+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:53:09.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:09 smithi165 bash[22591]: cluster 2024-09-10T18:53:08.753021+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:53:10.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[32381]: cluster 2024-09-10T18:53:08.172394+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[32381]: cluster 2024-09-10T18:53:08.175654+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[32381]: cluster 2024-09-10T18:53:08.527427+0000 mgr.y (mgr.24526) 143 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[32381]: cluster 2024-09-10T18:53:08.749263+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[32381]: cluster 2024-09-10T18:53:08.753021+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[18629]: cluster 2024-09-10T18:53:08.172394+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[18629]: cluster 2024-09-10T18:53:08.175654+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[18629]: cluster 2024-09-10T18:53:08.527427+0000 mgr.y (mgr.24526) 143 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[18629]: cluster 2024-09-10T18:53:08.749263+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:53:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:09 smithi083 bash[18629]: cluster 2024-09-10T18:53:08.753021+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:53:10.224 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:11.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:10 smithi083 bash[32381]: cluster 2024-09-10T18:53:09.741387+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T18:53:11.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:10 smithi083 bash[32381]: cluster 2024-09-10T18:53:09.745010+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T18:53:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:10 smithi083 bash[18629]: cluster 2024-09-10T18:53:09.741387+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T18:53:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:10 smithi083 bash[18629]: cluster 2024-09-10T18:53:09.745010+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T18:53:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:10 smithi165 bash[22591]: cluster 2024-09-10T18:53:09.741387+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T18:53:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:10 smithi165 bash[22591]: cluster 2024-09-10T18:53:09.745010+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T18:53:11.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.171809+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:53:11.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.176753+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:53:11.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.213978+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:11.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.214098+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:11.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.403326+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.406967+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.528847+0000 mgr.y (mgr.24526) 144 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.623853+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.716777+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.719572+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.749933+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[18629]: cluster 2024-09-10T18:53:10.756478+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.171809+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.176753+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.213978+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.214098+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:11.762 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.403326+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.406967+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.528847+0000 mgr.y (mgr.24526) 144 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.623853+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.716777+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.719572+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.749933+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:53:11.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:11 smithi083 bash[32381]: cluster 2024-09-10T18:53:10.756478+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:53:11.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.171809+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.176753+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.213978+0000 osd.4 (osd.4) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.214098+0000 osd.4 (osd.4) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.403326+0000 osd.5 (osd.5) 85 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.406967+0000 osd.5 (osd.5) 86 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.528847+0000 mgr.y (mgr.24526) 144 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.623853+0000 mon.a (mon.0) 869 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.716777+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.719572+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.749933+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:53:11.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:11 smithi165 bash[22591]: cluster 2024-09-10T18:53:10.756478+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:11.162250+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.e scrub starts 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:11.166774+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.e scrub ok 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:11.763260+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:11.768243+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:12.277797+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18629]: cluster 2024-09-10T18:53:12.280595+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T18:53:12.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:11.162250+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.e scrub starts 2024-09-10T18:53:12.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:11.166774+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.e scrub ok 2024-09-10T18:53:12.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:11.763260+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:53:12.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:11.768243+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:53:12.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:12.277797+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T18:53:12.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[32381]: cluster 2024-09-10T18:53:12.280595+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T18:53:12.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:12] "GET /metrics HTTP/1.1" 200 220135 "" "Prometheus/2.43.0" 2024-09-10T18:53:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:11.162250+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.e scrub starts 2024-09-10T18:53:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:11.166774+0000 osd.4 (osd.4) 108 : cluster [DBG] 4.e scrub ok 2024-09-10T18:53:13.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:11.763260+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:53:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:11.768243+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:53:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:12.277797+0000 osd.2 (osd.2) 49 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T18:53:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:12 smithi165 bash[22591]: cluster 2024-09-10T18:53:12.280595+0000 osd.2 (osd.2) 50 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T18:53:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- 172.21.15.83:0/2759466095 >> v1:172.21.15.83:6789/0 conn(0x7f80dc108be0 legacy=0x7f80dc108fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- 172.21.15.83:0/2759466095 shutdown_connections 2024-09-10T18:53:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- 172.21.15.83:0/2759466095 >> 172.21.15.83:0/2759466095 conn(0x7f80dc0723f0 msgr2=0x7f80dc074800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- 172.21.15.83:0/2759466095 shutdown_connections 2024-09-10T18:53:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- 172.21.15.83:0/2759466095 wait complete. 2024-09-10T18:53:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 librados: starting msgr at 2024-09-10T18:53:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 librados: starting objecter 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 Processor -- start 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 -- start start 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 librados: setting wanted keys 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.598+0000 7f80e2115640 1 librados: calling monclient init 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80dc1226a0 con 0x7f80dc10f520 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80dc123860 con 0x7f80dc108be0 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80dc124a60 con 0x7f80dc10b9c0 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80dbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f80dc10f520 0x7f80dc120da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55552/0 (socket says 172.21.15.83:55552) 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80daffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f80dc10b9c0 0x7f80dc105380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36002/0 (socket says 172.21.15.83:36002) 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80dbfff640 1 -- 172.21.15.83:0/617475701 learned_addr learned my addr 172.21.15.83:0/617475701 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter ms_handle_connect 0x7f80dc10f520 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter ms_handle_connect 0x7f80dc10b9c0 2024-09-10T18:53:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter ms_handle_connect 0x7f80dc108be0 2024-09-10T18:53:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134038084 0 0) 0x7f80dc124a60 con 0x7f80dc10b9c0 2024-09-10T18:53:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c40034e0 con 0x7f80dc10b9c0 2024-09-10T18:53:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122274242 0 0) 0x7f80dc1226a0 con 0x7f80dc10f520 2024-09-10T18:53:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80dc124a60 con 0x7f80dc10f520 2024-09-10T18:53:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831970686 0 0) 0x7f80dc123860 con 0x7f80dc108be0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80dc1226a0 con 0x7f80dc108be0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492634766 0 0) 0x7f80c40034e0 con 0x7f80dc10b9c0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80dc123860 con 0x7f80dc10b9c0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f80cc001de0 con 0x7f80dc10b9c0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1814683026 0 0) 0x7f80dc124a60 con 0x7f80dc10f520 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80c40034e0 con 0x7f80dc10f520 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260929641 0 0) 0x7f80dc1226a0 con 0x7f80dc108be0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80dc124a60 con 0x7f80dc108be0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f80d4001950 con 0x7f80dc10f520 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f80d0002a20 con 0x7f80dc108be0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1056937657 0 0) 0x7f80dc123860 con 0x7f80dc10b9c0 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 >> v1:172.21.15.165:6789/0 conn(0x7f80dc108be0 legacy=0x7f80dc104c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 >> v1:172.21.15.83:6789/0 conn(0x7f80dc10f520 legacy=0x7f80dc120da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc125c60 con 0x7f80dc10b9c0 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc124c50 con 0x7f80dc10b9c0 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 10 client.24722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc1250a0 con 0x7f80dc10b9c0 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80e2115640 1 librados: init done 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f80cc002710 con 0x7f80dc10b9c0 2024-09-10T18:53:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.602+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f80cc005d70 con 0x7f80dc10b9c0 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8005010 con 0x7f80dc10b9c0 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f80cc002880 con 0x7f80dc10b9c0 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(112..112 src has 59..112) v4 ==== 6284+0+0 (unknown 2143280382 0 0) 0x7f80cc08a190 con 0x7f80dc10b9c0 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 10 client.24722.objecter ms_dispatch 0x7f80dc000b90 osd_map(112..112 src has 59..112) v4 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 3 client.24722.objecter handle_osd_map got epochs [112,112] > 0 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 3 client.24722.objecter handle_osd_map decoding full epoch 112 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 20 client.24722.objecter dump_active .. 0 homeless 2024-09-10T18:53:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.606+0000 7f80c3fff640 10 client.24722.objecter ms_handle_connect 0x7f80c40717f0 2024-09-10T18:53:13.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.614+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f80cc056e60 con 0x7f80dc10b9c0 2024-09-10T18:53:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[18629]: cluster 2024-09-10T18:53:12.147782+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 deep-scrub starts 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[18629]: cluster 2024-09-10T18:53:12.153545+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1 deep-scrub ok 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[18629]: cluster 2024-09-10T18:53:12.529512+0000 mgr.y (mgr.24526) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[18629]: cluster 2024-09-10T18:53:13.288340+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[18629]: cluster 2024-09-10T18:53:13.292639+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[32381]: cluster 2024-09-10T18:53:12.147782+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 deep-scrub starts 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[32381]: cluster 2024-09-10T18:53:12.153545+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1 deep-scrub ok 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[32381]: cluster 2024-09-10T18:53:12.529512+0000 mgr.y (mgr.24526) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[32381]: cluster 2024-09-10T18:53:13.288340+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:53:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:13 smithi083 bash[32381]: cluster 2024-09-10T18:53:13.292639+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:53:13.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.790+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f80c8005ad0 con 0x7f80dc10b9c0 2024-09-10T18:53:13.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.794+0000 7f80c3fff640 1 -- 172.21.15.83:0/617475701 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v112) v1 ==== 56+0+4312 (unknown 258877355 0 2043989201) 0x7f80cc056800 con 0x7f80dc10b9c0 2024-09-10T18:53:13.798 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 112 2024-09-10T18:53:13.798 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-09-10T18:38:38.475667+0000 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:modified 2024-09-10T18:53:10.607842+0000 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:crush_version 18 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:full_ratio 0.95 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:backfillfull_ratio 0.9 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:nearfull_ratio 0.85 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:require_min_compat_client luminous 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:min_compat_client luminous 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:require_osd_release quincy 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:stretch_mode_enabled false 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-10T18:53:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 application rgw 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:pool 5 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 80 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 77 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:max_osd 8 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0 up in weight 1 up_from 8 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6801/2535527483 v1:172.21.15.83:6802/2535527483 exists,up 35ab8328-74f1-4af8-aa84-797914199445 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6805/2309624637 v1:172.21.15.83:6806/2309624637 exists,up 8e1d1072-767b-49fb-b2f2-fce60263e2ad 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2 up in weight 1 up_from 19 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6809/386094220 v1:172.21.15.83:6810/386094220 exists,up 6193b508-03d9-458a-b49f-6b2e24283282 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3 up in weight 1 up_from 29 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6813/1279688021 v1:172.21.15.83:6814/1279688021 exists,up a28f94ff-2fff-458e-8b69-c5c65d720d6c 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4 up in weight 1 up_from 35 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/3225515265 v1:172.21.15.165:6801/3225515265 exists,up a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5 up in weight 1 up_from 43 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/571173777 v1:172.21.15.165:6805/571173777 exists,up 10543223-db5a-493b-88f3-8705f2b31a8e 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/1317922936 v1:172.21.15.165:6809/1317922936 exists,up fcea0ca4-4521-4161-b407-9b1fab1a2ae2 2024-09-10T18:53:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7 up in weight 1 up_from 62 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/1610151650 v1:172.21.15.165:6813/1610151650 exists,up 8531d286-5ebb-4c96-a026-63d8a17cb21f 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:pg_upmap_items 2.f [7,2] 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:pg_upmap_items 4.12 [1,5] 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:pg_upmap_items 4.18 [1,2] 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:pg_upmap_items 4.1f [1,2] 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/3645093631 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:6800/3275901971 expires 2024-09-11T18:39:06.336264+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/3555551632 expires 2024-09-11T18:39:40.343252+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/3076437878 expires 2024-09-11T18:39:06.336264+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/4112054239 expires 2024-09-11T18:39:06.336264+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:6800/2452184936 expires 2024-09-11T18:39:40.343252+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/2659452540 expires 2024-09-11T18:39:40.343252+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/867303943 expires 2024-09-11T18:39:06.336264+0000 2024-09-10T18:53:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/1156549307 expires 2024-09-11T18:39:40.343252+0000 2024-09-10T18:53:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/1687990153 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/1246256198 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/3225538750 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:6800/2630800463 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:blocklist 172.21.15.83:0/1423545952 expires 2024-09-11T18:50:14.421453+0000 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.794+0000 7f80e2115640 10 librados: watch_flush enter 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 10 librados: watch_flush exit 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 20 client.24722.objecter shutdown clearing up homeless session... 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 10 client.24722.objecter successfully canceled tick 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 >> v1:172.21.15.83:6800/3071793735 conn(0x7f80c40717f0 legacy=0x7f80c4073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 >> v1:172.21.15.83:6790/0 conn(0x7f80dc10b9c0 legacy=0x7f80dc105380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 shutdown_connections 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 >> 172.21.15.83:0/617475701 conn(0x7f80dc0723f0 msgr2=0x7f80dc111990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:13.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 shutdown_connections 2024-09-10T18:53:13.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 -- 172.21.15.83:0/617475701 wait complete. 2024-09-10T18:53:13.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:13.798+0000 7f80e2115640 1 librados: shutdown 2024-09-10T18:53:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:13 smithi165 bash[22591]: cluster 2024-09-10T18:53:12.147782+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.1 deep-scrub starts 2024-09-10T18:53:14.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:13 smithi165 bash[22591]: cluster 2024-09-10T18:53:12.153545+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.1 deep-scrub ok 2024-09-10T18:53:14.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:13 smithi165 bash[22591]: cluster 2024-09-10T18:53:12.529512+0000 mgr.y (mgr.24526) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:14.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:13 smithi165 bash[22591]: cluster 2024-09-10T18:53:13.288340+0000 osd.2 (osd.2) 51 : cluster [DBG] 2.10 deep-scrub starts 2024-09-10T18:53:14.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:13 smithi165 bash[22591]: cluster 2024-09-10T18:53:13.292639+0000 osd.2 (osd.2) 52 : cluster [DBG] 2.10 deep-scrub ok 2024-09-10T18:53:14.582 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6801/2535527483 v1:172.21.15.83:6802/2535527483 exists,up 35ab8328-74f1-4af8-aa84-797914199445', 'osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6805/2309624637 v1:172.21.15.83:6806/2309624637 exists,up 8e1d1072-767b-49fb-b2f2-fce60263e2ad', 'osd.2 up in weight 1 up_from 19 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6809/386094220 v1:172.21.15.83:6810/386094220 exists,up 6193b508-03d9-458a-b49f-6b2e24283282', 'osd.3 up in weight 1 up_from 29 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.83:6813/1279688021 v1:172.21.15.83:6814/1279688021 exists,up a28f94ff-2fff-458e-8b69-c5c65d720d6c', 'osd.4 up in weight 1 up_from 35 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/3225515265 v1:172.21.15.165:6801/3225515265 exists,up a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b', 'osd.5 up in weight 1 up_from 43 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/571173777 v1:172.21.15.165:6805/571173777 exists,up 10543223-db5a-493b-88f3-8705f2b31a8e', 'osd.6 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/1317922936 v1:172.21.15.165:6809/1317922936 exists,up fcea0ca4-4521-4161-b407-9b1fab1a2ae2', 'osd.7 up in weight 1 up_from 62 up_thru 75 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/1610151650 v1:172.21.15.165:6813/1610151650 exists,up 8531d286-5ebb-4c96-a026-63d8a17cb21f'] 2024-09-10T18:53:14.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon mon.a config show 2024-09-10T18:53:15.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[32381]: cluster 2024-09-10T18:53:13.764352+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[32381]: cluster 2024-09-10T18:53:13.767597+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[32381]: audit 2024-09-10T18:53:13.798262+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? v1:172.21.15.83:0/617475701' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[32381]: audit 2024-09-10T18:53:14.553088+0000 mon.c (mon.2) 88 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[18629]: cluster 2024-09-10T18:53:13.764352+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[18629]: cluster 2024-09-10T18:53:13.767597+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[18629]: audit 2024-09-10T18:53:13.798262+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? v1:172.21.15.83:0/617475701' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:53:15.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:14 smithi083 bash[18629]: audit 2024-09-10T18:53:14.553088+0000 mon.c (mon.2) 88 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:14 smithi165 bash[22591]: cluster 2024-09-10T18:53:13.764352+0000 osd.0 (osd.0) 97 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:53:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:14 smithi165 bash[22591]: cluster 2024-09-10T18:53:13.767597+0000 osd.0 (osd.0) 98 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:53:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:14 smithi165 bash[22591]: audit 2024-09-10T18:53:13.798262+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? v1:172.21.15.83:0/617475701' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:53:15.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:14 smithi165 bash[22591]: audit 2024-09-10T18:53:14.553088+0000 mon.c (mon.2) 88 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[18629]: cluster 2024-09-10T18:53:14.149158+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[18629]: cluster 2024-09-10T18:53:14.154063+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[18629]: cluster 2024-09-10T18:53:14.424946+0000 osd.5 (osd.5) 87 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[18629]: cluster 2024-09-10T18:53:14.432125+0000 osd.5 (osd.5) 88 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[18629]: cluster 2024-09-10T18:53:14.530165+0000 mgr.y (mgr.24526) 146 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[32381]: cluster 2024-09-10T18:53:14.149158+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[32381]: cluster 2024-09-10T18:53:14.154063+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[32381]: cluster 2024-09-10T18:53:14.424946+0000 osd.5 (osd.5) 87 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[32381]: cluster 2024-09-10T18:53:14.432125+0000 osd.5 (osd.5) 88 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:53:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:15 smithi083 bash[32381]: cluster 2024-09-10T18:53:14.530165+0000 mgr.y (mgr.24526) 146 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:15 smithi165 bash[22591]: cluster 2024-09-10T18:53:14.149158+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:53:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:15 smithi165 bash[22591]: cluster 2024-09-10T18:53:14.154063+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:53:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:15 smithi165 bash[22591]: cluster 2024-09-10T18:53:14.424946+0000 osd.5 (osd.5) 87 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:53:16.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:15 smithi165 bash[22591]: cluster 2024-09-10T18:53:14.432125+0000 osd.5 (osd.5) 88 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:53:16.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:15 smithi165 bash[22591]: cluster 2024-09-10T18:53:14.530165+0000 mgr.y (mgr.24526) 146 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:15.728121+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:15.731876+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:15.756652+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:15.760692+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:16.447914+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[18629]: cluster 2024-09-10T18:53:16.451514+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:15.728121+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:53:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:15.731876+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:53:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:15.756652+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:53:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:15.760692+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:53:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:16.447914+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T18:53:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:16 smithi083 bash[32381]: cluster 2024-09-10T18:53:16.451514+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T18:53:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:15.728121+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:53:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:15.731876+0000 osd.6 (osd.6) 60 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:53:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:15.756652+0000 osd.0 (osd.0) 99 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:53:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:15.760692+0000 osd.0 (osd.0) 100 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:53:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:16.447914+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T18:53:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:16 smithi165 bash[22591]: cluster 2024-09-10T18:53:16.451514+0000 osd.1 (osd.1) 94 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T18:53:17.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:17 smithi165 bash[22591]: cluster 2024-09-10T18:53:16.531639+0000 mgr.y (mgr.24526) 147 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:18.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:17 smithi083 bash[18629]: cluster 2024-09-10T18:53:16.531639+0000 mgr.y (mgr.24526) 147 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:17 smithi083 bash[32381]: cluster 2024-09-10T18:53:16.531639+0000 mgr.y (mgr.24526) 147 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[18629]: cluster 2024-09-10T18:53:17.499151+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[18629]: cluster 2024-09-10T18:53:17.503122+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[18629]: audit 2024-09-10T18:53:17.565443+0000 mgr.y (mgr.24526) 148 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[18629]: cluster 2024-09-10T18:53:17.872832+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[18629]: cluster 2024-09-10T18:53:17.877696+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[32381]: cluster 2024-09-10T18:53:17.499151+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[32381]: cluster 2024-09-10T18:53:17.503122+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[32381]: audit 2024-09-10T18:53:17.565443+0000 mgr.y (mgr.24526) 148 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[32381]: cluster 2024-09-10T18:53:17.872832+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:53:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:18 smithi083 bash[32381]: cluster 2024-09-10T18:53:17.877696+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:53:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:18 smithi165 bash[22591]: cluster 2024-09-10T18:53:17.499151+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.d scrub starts 2024-09-10T18:53:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:18 smithi165 bash[22591]: cluster 2024-09-10T18:53:17.503122+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.d scrub ok 2024-09-10T18:53:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:18 smithi165 bash[22591]: audit 2024-09-10T18:53:17.565443+0000 mgr.y (mgr.24526) 148 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:18 smithi165 bash[22591]: cluster 2024-09-10T18:53:17.872832+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:53:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:18 smithi165 bash[22591]: cluster 2024-09-10T18:53:17.877696+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:53:19.351 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.137731+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.142058+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.456108+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.459858+0000 osd.5 (osd.5) 92 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.532400+0000 mgr.y (mgr.24526) 149 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.702391+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub starts 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.705766+0000 osd.6 (osd.6) 62 : cluster [DBG] 4.a scrub ok 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.880289+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:53:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[18629]: cluster 2024-09-10T18:53:18.883748+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.137731+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.142058+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.456108+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.459858+0000 osd.5 (osd.5) 92 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.532400+0000 mgr.y (mgr.24526) 149 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.702391+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub starts 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.705766+0000 osd.6 (osd.6) 62 : cluster [DBG] 4.a scrub ok 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.880289+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:53:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:19 smithi083 bash[32381]: cluster 2024-09-10T18:53:18.883748+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.137731+0000 osd.7 (osd.7) 91 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.142058+0000 osd.7 (osd.7) 92 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.456108+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.459858+0000 osd.5 (osd.5) 92 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.532400+0000 mgr.y (mgr.24526) 149 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.702391+0000 osd.6 (osd.6) 61 : cluster [DBG] 4.a scrub starts 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.705766+0000 osd.6 (osd.6) 62 : cluster [DBG] 4.a scrub ok 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.880289+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:53:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:19 smithi165 bash[22591]: cluster 2024-09-10T18:53:18.883748+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:53:20.926 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.161687+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.166088+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.691479+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.695035+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.752788+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 deep-scrub starts 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.756010+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 deep-scrub ok 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.837569+0000 osd.3 (osd.3) 121 : cluster [DBG] 3.b scrub starts 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:19.842161+0000 osd.3 (osd.3) 122 : cluster [DBG] 3.b scrub ok 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:20.452378+0000 osd.1 (osd.1) 95 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[18629]: cluster 2024-09-10T18:53:20.455690+0000 osd.1 (osd.1) 96 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:53:20.927 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.161687+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.166088+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.691479+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.695035+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.752788+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 deep-scrub starts 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.756010+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 deep-scrub ok 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.837569+0000 osd.3 (osd.3) 121 : cluster [DBG] 3.b scrub starts 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:19.842161+0000 osd.3 (osd.3) 122 : cluster [DBG] 3.b scrub ok 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:20.452378+0000 osd.1 (osd.1) 95 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:53:20.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:20 smithi083 bash[32381]: cluster 2024-09-10T18:53:20.455690+0000 osd.1 (osd.1) 96 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:53:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.161687+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.166088+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.691479+0000 osd.0 (osd.0) 101 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.695035+0000 osd.0 (osd.0) 102 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.752788+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.7 deep-scrub starts 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.756010+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.7 deep-scrub ok 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.837569+0000 osd.3 (osd.3) 121 : cluster [DBG] 3.b scrub starts 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:19.842161+0000 osd.3 (osd.3) 122 : cluster [DBG] 3.b scrub ok 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:20.452378+0000 osd.1 (osd.1) 95 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:53:21.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:20 smithi165 bash[22591]: cluster 2024-09-10T18:53:20.455690+0000 osd.1 (osd.1) 96 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:53:21.743 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:21.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:21 smithi165 bash[22591]: cluster 2024-09-10T18:53:20.533811+0000 mgr.y (mgr.24526) 150 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:21.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:21 smithi165 bash[22591]: cluster 2024-09-10T18:53:21.417622+0000 osd.1 (osd.1) 97 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:53:21.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:21 smithi165 bash[22591]: cluster 2024-09-10T18:53:21.421386+0000 osd.1 (osd.1) 98 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:53:22.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[32381]: cluster 2024-09-10T18:53:20.533811+0000 mgr.y (mgr.24526) 150 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[32381]: cluster 2024-09-10T18:53:21.417622+0000 osd.1 (osd.1) 97 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:53:22.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[32381]: cluster 2024-09-10T18:53:21.421386+0000 osd.1 (osd.1) 98 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:53:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[18629]: cluster 2024-09-10T18:53:20.533811+0000 mgr.y (mgr.24526) 150 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[18629]: cluster 2024-09-10T18:53:21.417622+0000 osd.1 (osd.1) 97 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:53:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:21 smithi083 bash[18629]: cluster 2024-09-10T18:53:21.421386+0000 osd.1 (osd.1) 98 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "name": "mon.a", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster": "ceph", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "admin_socket_mode": "", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_client_required": "cephx, none", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_cluster_required": "cephx", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_debug": "true", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-10T18:53:22.818 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_service_required": "cephx", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "auth_supported": "", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_aio": "true", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_aio_poll_ms": "250", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_aio_reap_max": "16", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_async_discard": "false", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_block_size": "4096", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_debug_aio": "false", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_debug_inflight_ios": "false", 2024-09-10T18:53:22.819 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_enable_discard": "false", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_flock_retry": "3", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_inject_crash": "0", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_ioring": "false", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_ioring_hipri": "false", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bdev_type": "", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_alloc_size": "1048576", 2024-09-10T18:53:22.820 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_allocator": "hybrid", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_buffered_io": "true", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_check_for_zeros": "false", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_compact_log_sync": "false", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_max_log_runway": "4194304", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_max_prefetch": "1048576", 2024-09-10T18:53:22.821 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_min_flush_size": "524288", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_min_log_runway": "1048576", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_replay_recovery": "false", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_sync_write": "false", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_allocation_from_file": "true", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_allocator": "hybrid", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_async_db_compaction": "true", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-10T18:53:22.822 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_create": "true", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_db_create": "false", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_db_path": "", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_db_size": "0", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_path": "", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_preallocate_file": "false", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_size": "107374182400", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_wal_create": "false", 2024-09-10T18:53:22.823 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_wal_path": "", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_block_wal_size": "100663296", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bluefs": "true", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_autotune": "true", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-10T18:53:22.824 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_size": "0", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_type": "2q", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_clone_cow": "true", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-10T18:53:22.825 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_mode": "none", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_csum_type": "crc32c", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_freelist": "false", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-10T18:53:22.826 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_small_allocations": "0", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_default_buffered_read": "true", 2024-09-10T18:53:22.827 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_default_buffered_write": "false", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fail_eio": "false", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-10T18:53:22.828 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_mount": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_umount": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-10T18:53:22.829 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_ignore_data_csum": "false", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_log_op_age": "5.000000", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_alloc_size": "0", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_blob_size": "0", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_max_deferred_txc": "32", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_min_alloc_size": "0", 2024-09-10T18:53:22.830 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_nid_prealloc": "1024", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_retry_disk_reads": "3", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_rocksdb_cf": "true", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-10T18:53:22.831 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_spdk_mem": "512", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_tracing": "false", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-10T18:53:22.832 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_zero_block_detection": "false", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-10T18:53:22.833 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_cluster_require_signatures": "false", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_cluster_require_version": "2", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_require_signatures": "false", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_require_version": "2", 2024-09-10T18:53:22.834 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_service_require_signatures": "false", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_service_require_version": "2", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "cephx_sign_messages": "true", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "chdir": "", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_acl_type": "", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_asio_thread_count": "2", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_cache_mid": "0.750000", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_cache_size": "16384", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_caps_release_delay": "5", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_check_pool_perm": "true", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_debug_force_sync_read": "false", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_debug_getattr_caps": "false", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_debug_inject_features": "", 2024-09-10T18:53:22.835 INFO:teuthology.orchestra.run.smithi083.stdout: "client_debug_inject_tick_delay": "0", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_die_on_failed_remount": "false", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_dirsize_rbytes": "true", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_force_lazyio": "false", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_fs": "", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-10T18:53:22.836 INFO:teuthology.orchestra.run.smithi083.stdout: "client_inject_release_failure": "false", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_max_inline_size": "4096", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_mds_namespace": "", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_metadata": "", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_mount_gid": "-1", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_mount_timeout": "300", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_mount_uid": "-1", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_mountpoint": "/", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_notify_timeout": "10", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc": "true", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc_max_dirty": "104857600", 2024-09-10T18:53:22.837 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc_max_objects": "1000", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc_size": "209715200", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_oc_target_dirty": "8388608", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_permissions": "true", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_quota": "true", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_quota_df": "true", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_readahead_max_bytes": "0", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_readahead_max_periods": "4", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_readahead_min": "131072", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_reconnect_stale": "false", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_shutdown_timeout": "30", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_snapdir": ".snap", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_tick_interval": "1", 2024-09-10T18:53:22.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_trace": "", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "client_try_dentry_invalidate": "false", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "client_use_faked_inos": "false", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "client_use_random_mds": "false", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_graylog": "false", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_graylog_port": "12201", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_monitors": "default=true", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_syslog": "false", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "clog_to_syslog_level": "info", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "-", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_network": "", 2024-09-10T18:53:22.839 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_network_interface": "", 2024-09-10T18:53:22.840 INFO:teuthology.orchestra.run.smithi083.stdout: "compressor_zlib_isal": "false", 2024-09-10T18:53:22.840 INFO:teuthology.orchestra.run.smithi083.stdout: "compressor_zlib_level": "5", 2024-09-10T18:53:22.840 INFO:teuthology.orchestra.run.smithi083.stdout: "compressor_zlib_winsize": "-15", 2024-09-10T18:53:22.840 INFO:teuthology.orchestra.run.smithi083.stdout: "compressor_zstd_level": "1", 2024-09-10T18:53:22.840 INFO:teuthology.orchestra.run.smithi083.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crimson_alien_op_num_threads": "6", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location": "", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location_hook": "", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_location_hook_timeout": "10", 2024-09-10T18:53:22.841 INFO:teuthology.orchestra.run.smithi083.stdout: "daemonize": "false", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_alienstore": "0/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_allow_any_pool_priority": "false", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_asok": "1/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_asok_assert_abort": "false", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_asserts_on_shutdown": "true", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_auth": "1/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_bdev": "1/3", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_bluefs": "1/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_bluestore": "1/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_buffer": "0/1", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_ceph_exporter": "1/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_cephfs_mirror": "0/5", 2024-09-10T18:53:22.842 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_cephsqlite": "0/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_civetweb": "1/10", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_client": "0/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_compressor": "1/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_context": "0/1", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_crush": "1/1", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_crypto": "1/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_deliberately_leak_memory": "false", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_disable_randomized_ping": "false", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_dpdk": "1/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_eventtrace": "1/5", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_filer": "0/1", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_filestore": "1/3", 2024-09-10T18:53:22.843 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_finisher": "1/1", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_fuse": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_heartbeat_testing_span": "0", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_heartbeatmap": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_javaclient": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_journal": "1/3", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_journaler": "0/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_kstore": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_leveldb": "4/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_lockdep": "0/1", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mclock": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds": "1/5", 2024-09-10T18:53:22.844 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds_balancer": "1/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds_locker": "1/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds_log": "1/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds_log_expire": "1/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mds_migrator": "1/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_memdb": "4/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mgr": "2/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mgrc": "20/20", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_mon": "20/20", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_monc": "0/10", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_ms": "1/1", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_none": "0/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_objclass": "0/5", 2024-09-10T18:53:22.845 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_objectcacher": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_objecter": "0/1", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_optracker": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_osd": "1/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_paxos": "20/20", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_perfcounter": "1/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_prioritycache": "1/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rados": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rbd": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rbd_mirror": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rbd_pwl": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rbd_replay": "0/5", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_refs": "0/0", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_reserver": "1/1", 2024-09-10T18:53:22.846 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rgw": "1/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rgw_datacache": "1/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rgw_sync": "1/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_rocksdb": "4/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_cache": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_cleaner": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_device": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_journal": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_lba": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_odata": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_omap": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_onode": "0/5", 2024-09-10T18:53:22.847 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_seastore_tm": "0/5", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_striper": "0/1", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_test": "0/5", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_throttle": "1/1", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_timer": "0/1", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "debug_tp": "0/5", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "device_failure_prediction_mode": "none", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "err_to_graylog": "false", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "err_to_journald": "false", 2024-09-10T18:53:22.848 INFO:teuthology.orchestra.run.smithi083.stdout: "err_to_stderr": "true", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "err_to_syslog": "false", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event_tracing": "false", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_addr": "0.0.0.0", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_http_port": "9926", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_prio_limit": "5", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_sort_metrics": "true", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "exporter_stats_period": "5", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "fake_statfs_for_testing": "0", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "fatal_signal_handlers": "true", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "filer_max_purge_ops": "10", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "filer_max_truncate_ops": "128", 2024-09-10T18:53:22.849 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_apply_finisher_threads": "1", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_blackhole": "false", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_btrfs_clone_range": "true", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_btrfs_snap": "true", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_caller_concurrency": "10", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_collect_device_partition_information": "true", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_commit_timeout": "600.000000", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_debug_inject_read_err": "false", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_debug_omap_check": "false", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_debug_verify_split": "false", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_dump_file": "", 2024-09-10T18:53:22.850 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fadvise": "true", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fail_eio": "true", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fd_cache_shards": "16", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fd_cache_size": "128", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fiemap": "false", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fiemap_threshold": "4096", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_inject_stall": "0", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_journal_parallel": "false", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_journal_trailing": "false", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_journal_writeahead": "false", 2024-09-10T18:53:22.851 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_kill_at": "0", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattrs": "0", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_xattr_value_size": "0", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-10T18:53:22.852 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_merge_threshold": "-10", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_odsync_write": "false", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_omap_backend": "rocksdb", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_omap_backend_path": "", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_op_thread_timeout": "60", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_op_threads": "2", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_punch_hole": "false", 2024-09-10T18:53:22.853 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_queue_max_ops": "50", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_seek_data_hole": "false", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_sloppy_crc": "false", 2024-09-10T18:53:22.854 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_splice": "false", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_split_multiple": "2", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_split_rand_factor": "20", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_update_to": "1000", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_enable": "true", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-10T18:53:22.855 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_xfs_extsize": "false", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "filestore_zfs_snap": "false", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fio_dir": "/tmp/fio", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "707a4ada-6fa3-11ef-bcea-c7b262605968", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_allow_other": "true", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_atomic_o_trunc": "true", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_big_writes": "true", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_debug": "false", 2024-09-10T18:53:22.856 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_default_permissions": "false", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_disable_pagecache": "false", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_max_write": "0", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_multithreaded": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_require_active_mds": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_set_user_groups": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_splice_move": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_splice_read": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_splice_write": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "fuse_use_invalidate_cb": "true", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "gss_target_name": "ceph", 2024-09-10T18:53:22.857 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_file": "", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_inject_failure": "0", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_interval": "5", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "host": "smithi083", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-10T18:53:22.858 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "inject_early_sigterm": "false", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "jaeger_tracing_enable": "false", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_aio": "true", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_align_min_size": "65536", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_block_align": "true", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_block_size": "4096", 2024-09-10T18:53:22.859 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_dio": "true", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_discard": "false", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_force_aio": "false", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_ignore_corruption": "false", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_max_write_bytes": "10485760", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_max_write_entries": "100", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_replay_from": "0", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_write_header_frequency": "0", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journal_zero_on_create": "false", 2024-09-10T18:53:22.860 INFO:teuthology.orchestra.run.smithi083.stdout: "journaler_prefetch_periods": "10", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "journaler_prezero_periods": "5", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "journaler_write_head_interval": "15", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "key": "", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "keyfile": "", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_backend": "rocksdb", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_default_stripe_size": "65536", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_fsck_on_mount": "false", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_max_bytes": "67108864", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_max_ops": "512", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_nid_prealloc": "1024", 2024-09-10T18:53:22.861 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_onode_map_size": "1024", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_sync_submit_transaction": "false", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "kstore_sync_transaction": "false", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_block_size": "65536", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_bloom_size": "0", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_cache_size": "536870912", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_compact_on_mount": "false", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_compression": "false", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_log": "", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_max_open_files": "0", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_paranoid": "false", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-10T18:53:22.862 INFO:teuthology.orchestra.run.smithi083.stdout: "librados_thread_count": "2", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "lockdep": "false", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "lockdep_force_backtrace": "false", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_coarse_timestamps": "true", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_flush_on_exit": "false", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_graylog_host": "127.0.0.1", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_graylog_port": "12201", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_max_new": "1000", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_max_recent": "10000", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_stderr_prefix": "debug ", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_stop_at_utilization": "0.970000", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_to_file": "true", 2024-09-10T18:53:22.863 INFO:teuthology.orchestra.run.smithi083.stdout: "log_to_graylog": "false", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "log_to_journald": "false", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "log_to_stderr": "false", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "log_to_syslog": "false", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "max_rotating_auth_attempts": "10", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_action_on_write_error": "1", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_alternate_name_max": "8192", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_asio_thread_count": "2", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_export_pin": "true", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_fragment_dirs": "true", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_fragment_interval": "5", 2024-09-10T18:53:22.864 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_interval": "10", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_max": "-1", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_max_until": "-1", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_merge_size": "50", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_midchunk": "0.300000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_min_start": "0.200000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_minchunk": "0.001000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_mode": "0", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_need_max": "1.200000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_need_min": "0.800000", 2024-09-10T18:53:22.865 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_split_bits": "3", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_split_size": "10000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_target_decay": "10.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_beacon_grace": "15.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_beacon_interval": "4.000000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_mid": "0.700000", 2024-09-10T18:53:22.866 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_release_free_interval": "10", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_reservation": "0.050000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_trim_interval": "1", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache_trim_threshold": "262144", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_client_prealloc_inos": "1000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_connect_bootstrapping": "false", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_damage_table_max_entries": "10000", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_debug_auth_pins": "false", 2024-09-10T18:53:22.867 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_debug_frag": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_debug_scatterstat": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_debug_subtrees": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_decay_halflife": "5.000000", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_default_dir_hash": "2", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_defer_session_stale": "true", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_deny_all_reconnect": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dir_keys_per_op": "16384", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dir_max_commit_size": "10", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dir_max_entries": "0", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dump_cache_on_map": "false", 2024-09-10T18:53:22.868 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_early_reply": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_enable_op_tracker": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_enforce_unique_name": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_export_ephemeral_random": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-10T18:53:22.869 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_health_summarize_threshold": "10", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_health_dummy": "false", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_migrator_session_race": "false", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_join_fs": "", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_journal_format": "1", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_create_at": "0", 2024-09-10T18:53:22.870 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_export_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_import_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_journal_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_journal_expire_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_journal_replay_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_link_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_mdstable_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_openc_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_rename_at": "0", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_events_per_segment": "1024", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_max_events": "-1", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_max_segments": "128", 2024-09-10T18:53:22.871 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_pause": "false", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_segment_size": "0", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log_warn_factor": "2.000000", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_caps_per_client": "1048576", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_completed_flushes": "100000", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_completed_requests": "100000", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_export_size": "20971520", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_file_recover": "32", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_purge_files": "64", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_purge_ops": "8192", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-10T18:53:22.872 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_snaps_per_dir": "100", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_metrics_update_interval": "2", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_min_caps_per_client": "100", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_min_caps_working_set": "10000", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_numa_node": "-1", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_complaint_time": "30.000000", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_history_duration": "600", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_history_size": "20", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_history_slow_op_size": "20", 2024-09-10T18:53:22.873 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_op_log_threshold": "5", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_ping_grace": "15", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_ping_interval": "5", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_max_caps": "30000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_recall_warning_threshold": "262144", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_replay_interval": "1.000000", 2024-09-10T18:53:22.874 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_root_ino_gid": "0", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_root_ino_uid": "0", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-10T18:53:22.875 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_shutdown_check": "0", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_skip_ino": "0", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_snap_max_uid": "4294967294", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_snap_min_uid": "0", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_snap_rstat": "false", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_standby_replay_damaged": "false", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_symlink_recovery": "true", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_thrash_exports": "0", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_thrash_fragments": "0", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_tick_interval": "5.000000", 2024-09-10T18:53:22.876 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_valgrind_exit": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_verify_backtrace": "1", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_verify_scatter": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_wipe_sessions": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool_debug": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "memstore_device_bytes": "1073741824", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "memstore_page_set": "false", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "memstore_page_size": "65536", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_client_bytes": "134217728", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_client_messages": "512", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-10T18:53:22.877 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_max_pg_num_change": "128", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_mds_bytes": "134217728", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_mds_messages": "128", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_mon_bytes": "134217728", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_mon_messages": "128", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_osd_bytes": "536870912", 2024-09-10T18:53:22.878 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_osd_messages": "8192", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_pool": "true", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_standby_modules": "true", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_stats_period": "5", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_stats_threshold": "5", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_tick_period": "2", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_allow_pool_delete": "true", 2024-09-10T18:53:22.879 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_allow_pool_size_one": "false", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_auth_validate_all_caps": "true", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_bytes": "104857600", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_directed_command_retry": "2", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_hunt_parallel": "3", 2024-09-10T18:53:22.880 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_log_interval": "1.000000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_ping_interval": "10.000000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_level": "debug", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_file": "false", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_journald": "false", 2024-09-10T18:53:22.881 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_stderr": "true", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_compact_on_bootstrap": "false", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_compact_on_start": "false", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_compact_on_trim": "true", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_cpu_threads": "4", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_crush_min_required_version": "hammer", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_daemon_bytes": "419430400", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-10T18:53:22.882 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_data_avail_crit": "5", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_data_avail_warn": "5", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_data_size_warn": "16106127360", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_dump_json": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_dump_transactions": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_extra_checks": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_no_require_pacific": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_no_require_quincy": "false", 2024-09-10T18:53:22.883 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_down_mkfs_grace": "60", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_election_default_strategy": "1", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_election_timeout": "5.000000", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_elector_ping_divisor": "2", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_enable_op_tracker": "true", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_fake_pool_delete": "false", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_force_quorum_join": "false", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_globalid_prealloc": "10000", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_detail_to_clog": "true", 2024-09-10T18:53:22.884 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_log_update_period": "5", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_max_detail": "50", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_to_clog": "true", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_to_clog_interval": "600", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_host": "172.21.15.83:6789/0", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_host_override": "", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_initial_members": "", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-10T18:53:22.885 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_lease": "5.000000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_log_full_interval": "50", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_log_max": "10000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_log_max_summary": "50", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_log_epochs": "500", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_osd": "10000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_pg_per_osd": "10000", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_pool_pg_num": "65536", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-10T18:53:22.886 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mds_force_trim_to": "0", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mds_skip_sanity": "false", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_memory_autotune": "true", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_memory_target": "2147483648", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_beacon_grace": "30", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_digest_period": "5", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_inactive_grace": "60", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_min_osdmap_epochs": "50", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_complaint_time": "30", 2024-09-10T18:53:22.887 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_history_duration": "600", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_history_size": "20", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_history_slow_op_size": "20", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_op_log_threshold": "5", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_auto_mark_in": "false", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_cache_size": "500", 2024-09-10T18:53:22.888 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_down_out_interval": "600", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_force_trim_to": "0", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-10T18:53:22.889 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_min_down_reporters": "2", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_report_timeout": "900", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-10T18:53:22.890 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_stuck_threshold": "60", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-10T18:53:22.891 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_probe_timeout": "2.000000", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_reweight_max_change": "0.050000", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_reweight_max_osds": "4", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_scrub_interval": "86400", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_scrub_max_keys": "100", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_scrub_timeout": "300", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_session_timeout": "300", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_smart_report_timeout": "5", 2024-09-10T18:53:22.892 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_stat_smooth_intervals": "6", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_stretch_pool_min_size": "2", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_stretch_pool_size": "4", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_debug": "false", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_provider_kill_at": "0", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_requester_kill_at": "0", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_sync_timeout": "60.000000", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_target_pg_per_osd": "100", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_tick_interval": "5", 2024-09-10T18:53:22.893 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_timecheck_interval": "300.000000", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_older_version_delay": "604800", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_misplaced": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_older_version": "true", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-10T18:53:22.894 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "monmap": "", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_op_threads": "3", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_cm": "false", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_device_name": "", 2024-09-10T18:53:22.895 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_dscp": "96", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_local_gid": "", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_port_num": "1", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_sl": "3", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_support_srq": "true", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_rdma_type": "ib", 2024-09-10T18:53:22.896 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_async_reap_threshold": "5", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_before_connect": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_ipv4": "true", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_ipv6": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_msgr1": "true", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_msgr2": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_port_max": "7568", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_port_min": "6800", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_retry_count": "3", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_bind_retry_delay": "5", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_blackhole_client": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_blackhole_mds": "false", 2024-09-10T18:53:22.897 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_blackhole_mgr": "false", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_blackhole_mon": "false", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_blackhole_osd": "false", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_client_mode": "crc secure", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_cluster_mode": "crc secure", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_cluster_type": "", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_compress_secure": "false", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_connection_idle_timeout": "900", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_connection_ready_timeout": "10", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_crc_data": "true", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_crc_header": "true", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_die_on_bad_msg": "false", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_die_on_bug": "true", 2024-09-10T18:53:22.898 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_die_on_old_message": "true", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_die_on_skipped_message": "false", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_coremask": "0xF", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_enable_tso": "true", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_hugepages": "", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_lro": "true", 2024-09-10T18:53:22.899 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_memory_channel": "4", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_pmd": "", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_port_id": "0", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_dump_on_send": "false", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_initial_backoff": "0.200000", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_inject_delay_max": "1.000000", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_inject_delay_type": "", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_inject_socket_failures": "0", 2024-09-10T18:53:22.900 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_learn_addr_from_peer": "true", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_max_accept_failures": "4", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_max_backoff": "15.000000", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_mon_client_mode": "secure crc", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_mon_service_mode": "secure crc", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_osd_compress_min_size": "1024", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_osd_compress_mode": "none", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_pq_min_cost": "65536", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_public_type": "", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_service_mode": "crc secure", 2024-09-10T18:53:22.901 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_tcp_listen_backlog": "512", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_tcp_nodelay": "true", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_tcp_rcvbuf": "0", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "ms_type": "async+posix", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "no_config_file": "false", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_completion_locks_per_session": "32", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_inflight_ops": "1024", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_tick_interval": "5.000000", 2024-09-10T18:53:22.902 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter_timeout": "10.000000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "objectstore_blackhole": "false", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "openssl_engine_opts": "", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_delay_time": "5.000000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_hist_halflife": "1000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_max_low_ops": "2", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_max_ops": "4", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_agent_slop": "0.020000", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-10T18:53:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_asio_thread_count": "2", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_async_recovery_min_cost": "100", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backfill_scan_max": "512", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backfill_scan_min": "64", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backoff_on_degraded": "false", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backoff_on_peering": "false", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_backoff_on_unfound": "true", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_beacon_report_interval": "300", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bench_duration": "30", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bench_max_block_size": "67108864", 2024-09-10T18:53:22.904 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_blkin_trace_all": "false", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_check_for_log_corruption": "false", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_class_update_on_start": "true", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_client_message_cap": "256", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_client_message_size_cap": "524288000", 2024-09-10T18:53:22.905 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_client_op_priority": "63", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_client_watch_timeout": "30", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_command_max_records": "256", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_compact_on_start": "false", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_crush_update_on_start": "true", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_crush_update_weight_set": "true", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-10T18:53:22.906 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_feed_pullee": "-1", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_misdirected_ops": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_no_acting_change": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_no_purge_strays": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_op_order": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_shutdown": "false", 2024-09-10T18:53:22.907 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_verify_snaps": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_keys": "1024", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_stride": "524288", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-10T18:53:22.908 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_default_notify_timeout": "30", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_delete_sleep": "0.000000", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_discard_disconnected_ops": "true", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_enable_op_tracker": "true", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_fast_info": "true", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_fast_shutdown": "true", 2024-09-10T18:53:22.909 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_function_tracing": "false", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_grace": "20", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_interval": "6", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_min_peers": "10", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_min_size": "2000", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_stale": "600", 2024-09-10T18:53:22.910 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_hit_set_max_size": "100000", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_hit_set_min_size": "1000", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_journal_size": "5120", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_kill_backfill_at": "0", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_map_cache_size": "50", 2024-09-10T18:53:22.911 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_map_dedup": "true", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_map_message_max": "40", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_map_share_max_epochs": "40", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_attr_name_len": "100", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_attr_size": "0", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_backfills": "1", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_markdown_count": "5", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_markdown_period": "600", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_object_name_len": "2048", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_object_namespace_len": "256", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_object_size": "134217728", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-10T18:53:22.912 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_pg_blocked_by": "16", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_pg_log_entries": "10000", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_pgls": "1024", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_push_cost": "8388608", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_push_objects": "10", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_scrubs": "3", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_trimming_pgs": "2", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_write_op_reply_len": "64", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_max_write_size": "90", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-10T18:53:22.913 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_profile": "balanced", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-10T18:53:22.914 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_base": "805306368", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_cache_min": "134217728", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_target": "4294967296", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_target_autotune": "false", 2024-09-10T18:53:22.915 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_min_pg_log_entries": "250", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_min_recovery_priority": "0", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mon_report_interval": "5", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_num_cache_shards": "32", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_num_op_tracker_shard": "32", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_numa_auto_affinity": "true", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_numa_node": "-1", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_numa_prefer_iface": "true", 2024-09-10T18:53:22.916 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_objecter_finishers": "1", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_objectstore": "bluestore", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_objectstore_fuse": "false", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_objectstore_tracing": "false", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_complaint_time": "30.000000", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_history_duration": "600", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_history_size": "20", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_history_slow_op_size": "20", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_log_threshold": "5", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_shards": "0", 2024-09-10T18:53:22.917 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_shards_hdd": "5", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_shards_ssd": "8", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_pq_min_cost": "65536", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_queue_cut_off": "high", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_op_thread_timeout": "15", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_open_classes_on_start": "true", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_os_flags": "0", 2024-09-10T18:53:22.918 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_peering_op_priority": "255", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_delete_cost": "1048576", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_delete_priority": "5", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_log_trim_max": "10000", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_log_trim_min": "100", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-10T18:53:22.919 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_flags": "0", 2024-09-10T18:53:22.920 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_min_size": "0", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_pg_num": "32", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_pgp_num": "0", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_size": "3", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_default_type": "replicated", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_push_per_object_cost": "1000", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recover_clone_overlap": "true", 2024-09-10T18:53:22.921 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_cost": "20971520", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_active": "0", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_max_single_start": "1", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_op_priority": "3", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_priority": "5", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_sleep": "0.000000", 2024-09-10T18:53:22.922 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_repair_during_recovery": "false", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_requested_scrub_priority": "120", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_auto_repair": "false", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_begin_hour": "0", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_begin_week_day": "0", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_chunk_max": "25", 2024-09-10T18:53:22.923 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_chunk_min": "5", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_cost": "52428800", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_during_recovery": "false", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_end_hour": "0", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_end_week_day": "0", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_event_cost": "4096", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_invalid_stats": "true", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_max_preemptions": "5", 2024-09-10T18:53:22.924 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_priority": "5", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_scrub_sleep": "0.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_skip_data_digest": "false", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_smart_report_timeout": "5", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_cost": "1048576", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_priority": "5", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-10T18:53:22.925 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_target_transaction_size": "30", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_tracing": "false", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_use_stale_snap": "false", 2024-09-10T18:53:22.926 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "osdc_blkin_trace_all": "false", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_kill_at": "0", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_max_join_drift": "10", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_min": "500", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_min_wait": "0.050000", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_propose_interval": "1.000000", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_service_trim_max": "500", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_service_trim_min": "10", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_stash_full_interval": "25", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_trim_max": "500", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "paxos_trim_min": "250", 2024-09-10T18:53:22.927 INFO:teuthology.orchestra.run.smithi083.stdout: "perf": "true", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "pid_file": "", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "-", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrv": "", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "public_bind_addr": "-", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "public_network": "", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "public_network_interface": "", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "qat_compressor_enabled": "false", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "rados_mon_op_timeout": "0", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "rados_osd_op_timeout": "0", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "rados_tracing": "false", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_atime_update_interval": "60", 2024-09-10T18:53:22.928 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_balance_parent_reads": "false", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_balance_snap_reads": "false", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_blkin_trace_all": "false", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache": "true", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_policy": "writearound", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_size": "33554432", 2024-09-10T18:53:22.929 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_clone_copy_on_read": "false", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_compression_hint": "none", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_concurrent_management_ops": "10", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_clone_format": "auto", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_data_pool": "", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_features": "61", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_format": "2", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_map_options": "", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_order": "22", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_pool": "rbd", 2024-09-10T18:53:22.930 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_stripe_count": "0", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_default_stripe_unit": "0", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_enable_alloc_hint": "true", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_io_scheduler": "simple", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-10T18:53:22.931 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_order": "24", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_pool": "", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_journal_splay_width": "4", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_localize_parent_reads": "false", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_localize_snap_reads": "false", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-10T18:53:22.932 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-10T18:53:22.933 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-10T18:53:22.934 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_mtime_update_interval": "60", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_non_blocking_aio": "true", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_op_thread_timeout": "60", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_op_threads": "1", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_parent_cache_enabled": "false", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_plugins": "", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_bps_burst": "0", 2024-09-10T18:53:22.935 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_bps_limit": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_exclude_ops": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_iops_burst": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_iops_limit": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-10T18:53:22.936 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_read_from_replica_policy": "default", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-10T18:53:22.937 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_skip_partial_discard": "true", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_tracing": "false", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_validate_names": "true", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd_validate_pool": "true", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "restapi_base_url": "", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "restapi_log_level": "", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_acl_grants_max_num": "100", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_admin_entry": "admin", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_backend_store": "rados", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_barbican_url": "", 2024-09-10T18:53:22.938 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_beast_enable_async": "true", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_cache_enabled": "true", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_cache_expiry_interval": "900", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_cache_lru_size": "10000", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_content_length_compat": "false", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_copy_obj_progress": "true", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_cors_rules_max_num": "100", 2024-09-10T18:53:22.939 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_cross_domain_policy": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_addr": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_password": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_kmip_username": "", 2024-09-10T18:53:22.940 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_require_ssl": "true", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-10T18:53:22.941 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_addr": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_auth": "token", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_namespace": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_prefix": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_token_file": "", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-10T18:53:22.942 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_curl_buffersize": "524288", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_curl_low_speed_time": "300", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-10T18:53:22.943 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data_log_changes_size": "1000", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data_log_num_shards": "128", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data_log_window": "30", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-10T18:53:22.944 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-10T18:53:22.945 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dns_name": "", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dns_s3website_name": "", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_dynamic_resharding": "true", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_gc_threads": "true", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_lc_threads": "true", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_ops_log": "false", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_quota_threads": "true", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_static_website": "false", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enable_usage_log": "false", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_enforce_swift_acls": "true", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_exit_timeout_secs": "120", 2024-09-10T18:53:22.946 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_expose_bucket": "false", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_extended_http_attrs": "", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_frontends": "beast port=7480", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_deferred": "50", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_objs": "32", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_gc_processor_period": "3600", 2024-09-10T18:53:22.947 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_init_timeout": "300", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_kafka_connection_idle": "300", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_domain": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_password": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_password_path": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_project": "", 2024-09-10T18:53:22.948 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_tenant": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_token": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_token_path": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_admin_user": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_api_version": "2", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_barbican_domain": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_barbican_password": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_barbican_project": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_barbican_user": "", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-10T18:53:22.949 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_url": "", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_debug_interval": "-1", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_lock_max_time": "90", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_max_objs": "32", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_max_rules": "1000", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_max_worker": "3", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lc_thread_delay": "0", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_dnattr": "uid", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-10T18:53:22.950 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_searchfilter": "", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ldap_uri": "", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_log_http_headers": "", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_log_object_name_utc": "false", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_attr_name_len": "0", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_attr_size": "0", 2024-09-10T18:53:22.951 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_chunk_size": "4194304", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_listing_results": "1000", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_notify_retries": "10", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_put_param_size": "1048576", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_put_size": "5368709120", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_max_slo_entries": "1000", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_md_log_max_shards": "64", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-10T18:53:22.952 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_mp_lock_max_time": "600", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_max_gc": "300", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-10T18:53:22.953 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_num_async_rados_threads": "32", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_num_control_oids": "8", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_numa_node": "-1", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_objexp_chunk_size": "100", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_objexp_gc_interval": "600", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_op_thread_timeout": "600", 2024-09-10T18:53:22.954 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_op_tracing": "false", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_opa_token": "", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_opa_url": "", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_opa_verify_ssl": "true", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ops_log_file_path": "", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ops_log_rados": "true", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_ops_log_socket_path": "", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_period_push_interval": "2.000000", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-10T18:53:22.955 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_print_continue": "true", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_rados_tracing": "false", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_realm": "", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_realm_id": "", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_region": "", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-10T18:53:22.956 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_request_uri": "", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_reshard_batch_size": "64", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_reshard_max_aio": "128", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_reshard_num_logs": "16", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_reshard_thread_interval": "600", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_resolve_cname": "false", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_run_sync_thread": "true", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-10T18:53:22.957 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_scheduler_type": "throttler", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_script_uri": "", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_service_provider_name": "", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_client_id": "", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_client_secret": "", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_entry": "sts", 2024-09-10T18:53:22.958 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_key": "sts", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_min_session_duration": "900", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sts_token_introspection_url": "", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_account_in_url": "false", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_auth_entry": "auth", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_auth_url": "", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_custom_header": "", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_need_stats": "true", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_tenant_name": "", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_token_expiration": "86400", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_url": "", 2024-09-10T18:53:22.959 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_url_prefix": "swift", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_lease_period": "120", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-10T18:53:22.960 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_thread_pool_size": "512", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_comment": "", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_createby": "", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_encoding": "", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_flag": "false", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_origin": "", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_torrent_tracker": "", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_trust_forwarded_https": "false", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_usage_max_shards": "32", 2024-09-10T18:53:22.961 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_usage_max_user_shards": "1", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_use_opa_authz": "false", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_max_buckets": "1000", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_user_unique_email": "true", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_verify_ssl": "true", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zone": "", 2024-09-10T18:53:22.962 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zone_id": "", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zonegroup": "", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zonegroup_id": "", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_block_size": "4096", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_size": "536870912", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-10T18:53:22.963 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_index_type": "binary_search", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_partition_filters": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_perf": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-10T18:53:22.964 INFO:teuthology.orchestra.run.smithi083.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "run_dir": "/var/run/ceph", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_block_create": "true", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_cache_lru_size": "67108864", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_default_max_object_size": "16777216", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_device_size": "107374182400", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_journal_batch_capacity": "16", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "seastore_segment_size": "67108864", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "setgroup": "ceph", 2024-09-10T18:53:22.965 INFO:teuthology.orchestra.run.smithi083.stdout: "setuser": "ceph", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "setuser_match_path": "", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "thp": "false", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "threadpool_default_timeout": "60", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout: "throttler_perf_counter": "true" 2024-09-10T18:53:22.966 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:53:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:22 smithi083 bash[18629]: cluster 2024-09-10T18:53:21.561853+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:53:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:22 smithi083 bash[18629]: cluster 2024-09-10T18:53:21.565088+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:53:23.042 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:22] "GET /metrics HTTP/1.1" 200 220117 "" "Prometheus/2.43.0" 2024-09-10T18:53:23.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:22 smithi083 bash[32381]: cluster 2024-09-10T18:53:21.561853+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:53:23.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:22 smithi083 bash[32381]: cluster 2024-09-10T18:53:21.565088+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:53:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:22 smithi165 bash[22591]: cluster 2024-09-10T18:53:21.561853+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:53:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:22 smithi165 bash[22591]: cluster 2024-09-10T18:53:21.565088+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:53:23.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-10T18:53:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[18629]: cluster 2024-09-10T18:53:22.534562+0000 mgr.y (mgr.24526) 151 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-10T18:53:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[18629]: cluster 2024-09-10T18:53:22.798832+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:53:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[18629]: cluster 2024-09-10T18:53:22.802375+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:53:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[32381]: cluster 2024-09-10T18:53:22.534562+0000 mgr.y (mgr.24526) 151 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-10T18:53:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[32381]: cluster 2024-09-10T18:53:22.798832+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:53:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:23 smithi083 bash[32381]: cluster 2024-09-10T18:53:22.802375+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:53:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:23 smithi165 bash[22591]: cluster 2024-09-10T18:53:22.534562+0000 mgr.y (mgr.24526) 151 : cluster [DBG] pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-10T18:53:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:23 smithi165 bash[22591]: cluster 2024-09-10T18:53:22.798832+0000 osd.3 (osd.3) 123 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:53:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:23 smithi165 bash[22591]: cluster 2024-09-10T18:53:22.802375+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:53:25.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[18629]: cluster 2024-09-10T18:53:23.117821+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[18629]: cluster 2024-09-10T18:53:23.121346+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.188604+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.c scrub starts 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.192326+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.c scrub ok 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[32381]: cluster 2024-09-10T18:53:23.117821+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[32381]: cluster 2024-09-10T18:53:23.121346+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.188604+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.c scrub starts 2024-09-10T18:53:25.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:24 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.192326+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.c scrub ok 2024-09-10T18:53:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:24 smithi165 bash[22591]: cluster 2024-09-10T18:53:23.117821+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:53:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:24 smithi165 bash[22591]: cluster 2024-09-10T18:53:23.121346+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:53:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:24 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.188604+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.c scrub starts 2024-09-10T18:53:25.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:24 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.192326+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.c scrub ok 2024-09-10T18:53:26.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.535281+0000 mgr.y (mgr.24526) 152 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.676611+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.679890+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.754834+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:24.758470+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.303718+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.308270+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.535281+0000 mgr.y (mgr.24526) 152 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.676611+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:53:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.679890+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:53:26.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.754834+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:53:26.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:24.758470+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:53:26.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.303718+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:53:26.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:25 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.308270+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:53:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.535281+0000 mgr.y (mgr.24526) 152 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.676611+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:53:26.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.679890+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:53:26.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.754834+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:53:26.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:24.758470+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:53:26.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.303718+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:53:26.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:25 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.308270+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:53:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.202258+0000 osd.4 (osd.4) 113 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:53:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.207182+0000 osd.4 (osd.4) 114 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:53:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.767585+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:53:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[18629]: cluster 2024-09-10T18:53:25.771177+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:53:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.202258+0000 osd.4 (osd.4) 113 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:53:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.207182+0000 osd.4 (osd.4) 114 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:53:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.767585+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:53:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:26 smithi083 bash[32381]: cluster 2024-09-10T18:53:25.771177+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:53:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:26 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.202258+0000 osd.4 (osd.4) 113 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:53:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:26 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.207182+0000 osd.4 (osd.4) 114 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:53:27.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:26 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.767585+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:53:27.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:26 smithi165 bash[22591]: cluster 2024-09-10T18:53:25.771177+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:53:27.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[22591]: cluster 2024-09-10T18:53:26.249317+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.e scrub starts 2024-09-10T18:53:27.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[22591]: cluster 2024-09-10T18:53:26.252997+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.e scrub ok 2024-09-10T18:53:27.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[22591]: cluster 2024-09-10T18:53:26.536705+0000 mgr.y (mgr.24526) 153 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:27.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[22591]: cluster 2024-09-10T18:53:27.231759+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:53:27.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:27 smithi165 bash[22591]: cluster 2024-09-10T18:53:27.236084+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[18629]: cluster 2024-09-10T18:53:26.249317+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.e scrub starts 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[18629]: cluster 2024-09-10T18:53:26.252997+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.e scrub ok 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[18629]: cluster 2024-09-10T18:53:26.536705+0000 mgr.y (mgr.24526) 153 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[18629]: cluster 2024-09-10T18:53:27.231759+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[18629]: cluster 2024-09-10T18:53:27.236084+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[32381]: cluster 2024-09-10T18:53:26.249317+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.e scrub starts 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[32381]: cluster 2024-09-10T18:53:26.252997+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.e scrub ok 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[32381]: cluster 2024-09-10T18:53:26.536705+0000 mgr.y (mgr.24526) 153 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[32381]: cluster 2024-09-10T18:53:27.231759+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:53:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:27 smithi083 bash[32381]: cluster 2024-09-10T18:53:27.236084+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:53:28.332 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:29.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[18629]: audit 2024-09-10T18:53:27.576805+0000 mgr.y (mgr.24526) 154 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[18629]: cluster 2024-09-10T18:53:27.760381+0000 osd.6 (osd.6) 69 : cluster [DBG] 3.a scrub starts 2024-09-10T18:53:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[18629]: cluster 2024-09-10T18:53:27.765901+0000 osd.6 (osd.6) 70 : cluster [DBG] 3.a scrub ok 2024-09-10T18:53:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[32381]: audit 2024-09-10T18:53:27.576805+0000 mgr.y (mgr.24526) 154 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[32381]: cluster 2024-09-10T18:53:27.760381+0000 osd.6 (osd.6) 69 : cluster [DBG] 3.a scrub starts 2024-09-10T18:53:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:28 smithi083 bash[32381]: cluster 2024-09-10T18:53:27.765901+0000 osd.6 (osd.6) 70 : cluster [DBG] 3.a scrub ok 2024-09-10T18:53:29.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:28 smithi165 bash[22591]: audit 2024-09-10T18:53:27.576805+0000 mgr.y (mgr.24526) 154 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:29.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:28 smithi165 bash[22591]: cluster 2024-09-10T18:53:27.760381+0000 osd.6 (osd.6) 69 : cluster [DBG] 3.a scrub starts 2024-09-10T18:53:29.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:28 smithi165 bash[22591]: cluster 2024-09-10T18:53:27.765901+0000 osd.6 (osd.6) 70 : cluster [DBG] 3.a scrub ok 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: cluster 2024-09-10T18:53:28.172827+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.d scrub starts 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: cluster 2024-09-10T18:53:28.177240+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.d scrub ok 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: cluster 2024-09-10T18:53:28.537391+0000 mgr.y (mgr.24526) 155 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: cluster 2024-09-10T18:53:29.054275+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: cluster 2024-09-10T18:53:29.059005+0000 osd.7 (osd.7) 98 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[32381]: audit 2024-09-10T18:53:29.553340+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: cluster 2024-09-10T18:53:28.172827+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.d scrub starts 2024-09-10T18:53:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: cluster 2024-09-10T18:53:28.177240+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.d scrub ok 2024-09-10T18:53:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: cluster 2024-09-10T18:53:28.537391+0000 mgr.y (mgr.24526) 155 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: cluster 2024-09-10T18:53:29.054275+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:53:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: cluster 2024-09-10T18:53:29.059005+0000 osd.7 (osd.7) 98 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:53:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:29 smithi083 bash[18629]: audit 2024-09-10T18:53:29.553340+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: cluster 2024-09-10T18:53:28.172827+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.d scrub starts 2024-09-10T18:53:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: cluster 2024-09-10T18:53:28.177240+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.d scrub ok 2024-09-10T18:53:30.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: cluster 2024-09-10T18:53:28.537391+0000 mgr.y (mgr.24526) 155 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:30.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: cluster 2024-09-10T18:53:29.054275+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:53:30.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: cluster 2024-09-10T18:53:29.059005+0000 osd.7 (osd.7) 98 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:53:30.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:29 smithi165 bash[22591]: audit 2024-09-10T18:53:29.553340+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:30 smithi165 bash[22591]: cluster 2024-09-10T18:53:29.847711+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:53:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:30 smithi165 bash[22591]: cluster 2024-09-10T18:53:29.852703+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:53:31.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:30 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.040702+0000 osd.7 (osd.7) 99 : cluster [DBG] 3.f scrub starts 2024-09-10T18:53:31.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:30 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.046184+0000 osd.7 (osd.7) 100 : cluster [DBG] 3.f scrub ok 2024-09-10T18:53:31.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[18629]: cluster 2024-09-10T18:53:29.847711+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:53:31.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[18629]: cluster 2024-09-10T18:53:29.852703+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.040702+0000 osd.7 (osd.7) 99 : cluster [DBG] 3.f scrub starts 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.046184+0000 osd.7 (osd.7) 100 : cluster [DBG] 3.f scrub ok 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[32381]: cluster 2024-09-10T18:53:29.847711+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[32381]: cluster 2024-09-10T18:53:29.852703+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.040702+0000 osd.7 (osd.7) 99 : cluster [DBG] 3.f scrub starts 2024-09-10T18:53:31.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:30 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.046184+0000 osd.7 (osd.7) 100 : cluster [DBG] 3.f scrub ok 2024-09-10T18:53:31.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.697+0000 7f4557d56640 1 -- 172.21.15.83:0/4080257821 >> v1:172.21.15.83:6789/0 conn(0x7f4550100c70 legacy=0x7f4550101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.697+0000 7f4557d56640 1 -- 172.21.15.83:0/4080257821 shutdown_connections 2024-09-10T18:53:31.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.697+0000 7f4557d56640 1 -- 172.21.15.83:0/4080257821 >> 172.21.15.83:0/4080257821 conn(0x7f4550074a20 msgr2=0x7f45500f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:31.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.697+0000 7f4557d56640 1 -- 172.21.15.83:0/4080257821 shutdown_connections 2024-09-10T18:53:31.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.697+0000 7f4557d56640 1 -- 172.21.15.83:0/4080257821 wait complete. 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 librados: starting msgr at 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 librados: starting objecter 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 Processor -- start 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- start start 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 librados: setting wanted keys 2024-09-10T18:53:31.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 librados: calling monclient init 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4550100470 con 0x7f4550103a50 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4550100660 con 0x7f4550100c70 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f455011c340 con 0x7f45501075b0 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45552f1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4550103a50 0x7f45500fcf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37702/0 (socket says 172.21.15.83:37702) 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45552f1640 1 -- 172.21.15.83:0/2499350288 learned_addr learned my addr 172.21.15.83:0/2499350288 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter ms_handle_connect 0x7f45501075b0 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter ms_handle_connect 0x7f4550103a50 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:31.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter ms_handle_connect 0x7f4550100c70 2024-09-10T18:53:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642894144 0 0) 0x7f455011c340 con 0x7f45501075b0 2024-09-10T18:53:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c0034e0 con 0x7f45501075b0 2024-09-10T18:53:31.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140427871 0 0) 0x7f4550100470 con 0x7f4550103a50 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455011c340 con 0x7f4550103a50 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722494133 0 0) 0x7f4550100660 con 0x7f4550100c70 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4550100470 con 0x7f4550100c70 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3341716870 0 0) 0x7f453c0034e0 con 0x7f45501075b0 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4550100660 con 0x7f45501075b0 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4548001b80 con 0x7f45501075b0 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1792892960 0 0) 0x7f4550100660 con 0x7f45501075b0 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.165:6789/0 conn(0x7f4550100c70 legacy=0x7f45500fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6789/0 conn(0x7f4550103a50 legacy=0x7f45500fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455011d520 con 0x7f45501075b0 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455011c530 con 0x7f45501075b0 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 10 client.24728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f455011c980 con 0x7f45501075b0 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f4557d56640 1 librados: init done 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4548002200 con 0x7f45501075b0 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.701+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4548003450 con 0x7f45501075b0 2024-09-10T18:53:31.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4534005060 con 0x7f45501075b0 2024-09-10T18:53:31.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4548001da0 con 0x7f45501075b0 2024-09-10T18:53:31.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(113..113 src has 59..113) v4 ==== 6284+0+0 (unknown 4030417130 0 0) 0x7f45480878b0 con 0x7f45501075b0 2024-09-10T18:53:31.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 10 client.24728.objecter ms_dispatch 0x7f4550000b90 osd_map(113..113 src has 59..113) v4 2024-09-10T18:53:31.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 3 client.24728.objecter handle_osd_map got epochs [113,113] > 0 2024-09-10T18:53:31.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 3 client.24728.objecter handle_osd_map decoding full epoch 113 2024-09-10T18:53:31.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.705+0000 7f45467fc640 20 client.24728.objecter dump_active .. 0 homeless 2024-09-10T18:53:31.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1113 (unknown 409899127 0 4169750816) 0x7f4548087d30 con 0x7f45501075b0 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f453c071470 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f4534004d70 con 0x7f4534002760 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f4534002760 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716454232 0 0) 0x7f4534004d70 con 0x7f4534002760 2024-09-10T18:53:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4550100660 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2960899152 0 0) 0x7f4550100660 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534004d70 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540001f00 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1278017465 0 0) 0x7f4534004d70 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f453400a550 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4540005840 con 0x7f4534002760 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6789/0 conn(0x7f4534002760 legacy=0x7f45340055d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.709+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6789/0 conn(0x7f4534002760 legacy=0x7f45340055d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f453400adb0 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399290633 0 0) 0x7f453400adb0 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004d70 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3921238781 0 0) 0x7f4534004d70 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453400adb0 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540002730 con 0x7f453400a8e0 2024-09-10T18:53:31.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3795930922 0 0) 0x7f453400adb0 con 0x7f453400a8e0 2024-09-10T18:53:31.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.717+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f453400c030 con 0x7f453400a8e0 2024-09-10T18:53:31.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.0 v1:172.21.15.83:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f4540002bb0 con 0x7f453400a8e0 2024-09-10T18:53:31.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6789/0 conn(0x7f453400a8e0 legacy=0x7f4534008d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6789/0 conn(0x7f453400a8e0 legacy=0x7f4534008d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.723 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a: {} 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f45340021a0 con 0x7f4534001d40 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f4534001d40 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700455494 0 0) 0x7f45340021a0 con 0x7f4534001d40 2024-09-10T18:53:31.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453400adb0 con 0x7f4534001d40 2024-09-10T18:53:31.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1191854318 0 0) 0x7f453400adb0 con 0x7f4534001d40 2024-09-10T18:53:31.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45340021a0 con 0x7f4534001d40 2024-09-10T18:53:31.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540005c70 con 0x7f4534001d40 2024-09-10T18:53:31.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 763435477 0 0) 0x7f45340021a0 con 0x7f4534001d40 2024-09-10T18:53:31.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.721+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4534012a80 con 0x7f4534001d40 2024-09-10T18:53:31.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.725+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4540007fc0 con 0x7f4534001d40 2024-09-10T18:53:31.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.725+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.165:6789/0 conn(0x7f4534001d40 legacy=0x7f453400b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.725+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.165:6789/0 conn(0x7f4534001d40 legacy=0x7f453400b700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.725+0000 7f45562f3640 1 -- 172.21.15.83:0/2499350288 reap_dead start 2024-09-10T18:53:31.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f4534013160 con 0x7f4534012d40 2024-09-10T18:53:31.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f4534012d40 2024-09-10T18:53:31.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758479236 0 0) 0x7f4534013160 con 0x7f4534012d40 2024-09-10T18:53:31.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45340021a0 con 0x7f4534012d40 2024-09-10T18:53:31.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3654572128 0 0) 0x7f45340021a0 con 0x7f4534012d40 2024-09-10T18:53:31.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534013160 con 0x7f4534012d40 2024-09-10T18:53:31.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540006390 con 0x7f4534012d40 2024-09-10T18:53:31.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1583981339 0 0) 0x7f4534013160 con 0x7f4534012d40 2024-09-10T18:53:31.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.733+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.165:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f4534014300 con 0x7f4534012d40 2024-09-10T18:53:31.740 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b: {} 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.1 v1:172.21.15.165:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f4540006670 con 0x7f4534012d40 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.165:6789/0 conn(0x7f4534012d40 legacy=0x7f4534011230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.165:6789/0 conn(0x7f4534012d40 legacy=0x7f4534011230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f4534013fe0 con 0x7f45340132f0 2024-09-10T18:53:31.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588995343 0 0) 0x7f4534013fe0 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534013160 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1953746882 0 0) 0x7f4534013160 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534013fe0 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.737+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540006c40 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.741+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3525466575 0 0) 0x7f4534013fe0 con 0x7f45340132f0 2024-09-10T18:53:31.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.741+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f453401ad50 con 0x7f45340132f0 2024-09-10T18:53:31.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.741+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4540008250 con 0x7f45340132f0 2024-09-10T18:53:31.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.741+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6790/0 conn(0x7f45340132f0 legacy=0x7f4534013730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.741+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6790/0 conn(0x7f45340132f0 legacy=0x7f4534013730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7f453401b2b0 con 0x7f453401aec0 2024-09-10T18:53:31.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 10 client.24728.objecter ms_handle_connect 0x7f453401aec0 2024-09-10T18:53:31.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 10 client.24728.objecter resend_mon_ops 2024-09-10T18:53:31.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203854770 0 0) 0x7f453401b2b0 con 0x7f453401aec0 2024-09-10T18:53:31.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534013160 con 0x7f453401aec0 2024-09-10T18:53:31.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1499352141 0 0) 0x7f4534013160 con 0x7f453401aec0 2024-09-10T18:53:31.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453401b2b0 con 0x7f453401aec0 2024-09-10T18:53:31.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4540007180 con 0x7f453401aec0 2024-09-10T18:53:31.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 880181935 0 0) 0x7f453401b2b0 con 0x7f453401aec0 2024-09-10T18:53:31.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.753+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 --> v1:172.21.15.83:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f453401c4a0 con 0x7f453401aec0 2024-09-10T18:53:31.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 <== mon.2 v1:172.21.15.83:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7f4540006fe0 con 0x7f453401aec0 2024-09-10T18:53:31.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6790/0 conn(0x7f453401aec0 legacy=0x7f4534019500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f45467fc640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6790/0 conn(0x7f453401aec0 legacy=0x7f4534019500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T18:53:31.759 INFO:teuthology.orchestra.run.smithi083.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T18:53:31.759 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c: {} 2024-09-10T18:53:31.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 10 librados: watch_flush enter 2024-09-10T18:53:31.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 10 librados: watch_flush exit 2024-09-10T18:53:31.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 20 client.24728.objecter shutdown clearing up homeless session... 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 10 client.24728.objecter successfully canceled tick 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6800/3071793735 conn(0x7f453c071470 legacy=0x7f453c073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 >> v1:172.21.15.83:6790/0 conn(0x7f45501075b0 legacy=0x7f4550118c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f45562f3640 1 -- 172.21.15.83:0/2499350288 reap_dead start 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 shutdown_connections 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 >> 172.21.15.83:0/2499350288 conn(0x7f4550074a20 msgr2=0x7f455010a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 shutdown_connections 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 -- 172.21.15.83:0/2499350288 wait complete. 2024-09-10T18:53:31.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:31.757+0000 7f4557d56640 1 librados: shutdown 2024-09-10T18:53:31.801 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:32.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.329715+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.329829+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.538786+0000 mgr.y (mgr.24526) 156 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:30.806081+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.020677+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.024363+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.243082+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.f scrub starts 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.246823+0000 osd.2 (osd.2) 58 : cluster [DBG] 6.f scrub ok 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.290849+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.293476+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:53:32.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:31 smithi165 bash[22591]: audit 2024-09-10T18:53:31.708828+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.83:0/2499350288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.329715+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.329829+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.538786+0000 mgr.y (mgr.24526) 156 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:30.806081+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.020677+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.024363+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.243082+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.f scrub starts 2024-09-10T18:53:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.246823+0000 osd.2 (osd.2) 58 : cluster [DBG] 6.f scrub ok 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.290849+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.293476+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[18629]: audit 2024-09-10T18:53:31.708828+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.83:0/2499350288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.329715+0000 osd.0 (osd.0) 107 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.329829+0000 osd.0 (osd.0) 108 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.538786+0000 mgr.y (mgr.24526) 156 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:30.806081+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-09-10T18:53:32.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.020677+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.024363+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.243082+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.f scrub starts 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.246823+0000 osd.2 (osd.2) 58 : cluster [DBG] 6.f scrub ok 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.290849+0000 osd.1 (osd.1) 101 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.293476+0000 osd.1 (osd.1) 102 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:53:32.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:31 smithi083 bash[32381]: audit 2024-09-10T18:53:31.708828+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.83:0/2499350288' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:53:32.481 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T18:53:32.495 INFO:tasks.workunit:Pulling workunits from ref fad5356a94e6044ce1062d748daa5a0f2ed53a7d 2024-09-10T18:53:32.495 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T18:53:32.495 DEBUG:teuthology.orchestra.run.smithi083:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T18:53:32.496 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-10T18:53:32.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T18:53:32.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T18:53:32.497 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-10T18:53:32.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:32.497 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-10T18:53:32.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:53:32.498 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-10T18:53:32.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:53:32.498 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-10T18:53:32.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:53:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:32.507 INFO:teuthology.orchestra.run.smithi083.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T18:53:32.508 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T18:53:32.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:32.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:32.554 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T18:53:32.555 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T18:53:32.560 INFO:tasks.workunit:timeout=3h 2024-09-10T18:53:32.560 INFO:tasks.workunit:cleanup=True 2024-09-10T18:53:32.561 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/adk3798/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout fad5356a94e6044ce1062d748daa5a0f2ed53a7d 2024-09-10T18:53:32.567 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T18:53:32.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:32.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:32.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:32.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:32.785 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:32] "GET /metrics HTTP/1.1" 200 220131 "" "Prometheus/2.43.0" 2024-09-10T18:53:32.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:32.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:32.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:32.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:32.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:33.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.165496+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.169308+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.455780+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[18629]: cluster 2024-09-10T18:53:31.459313+0000 osd.5 (osd.5) 96 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.165496+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.169308+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.455780+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:53:33.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:32 smithi083 bash[32381]: cluster 2024-09-10T18:53:31.459313+0000 osd.5 (osd.5) 96 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:53:33.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:33.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:32 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.165496+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:53:33.064 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:32 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.169308+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:53:33.064 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:32 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.455780+0000 osd.5 (osd.5) 95 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:53:33.064 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:32 smithi165 bash[22591]: cluster 2024-09-10T18:53:31.459313+0000 osd.5 (osd.5) 96 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:53:33.084 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:33.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:33.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:33.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:33.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:33.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:33.501 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:33.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:33.636 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:33.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:33.773 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:33.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:32.539298+0000 mgr.y (mgr.24526) 157 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:32.750327+0000 osd.6 (osd.6) 71 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:32.756721+0000 osd.6 (osd.6) 72 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:32.826386+0000 osd.0 (osd.0) 109 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:32.830053+0000 osd.0 (osd.0) 110 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:33.341792+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 deep-scrub starts 2024-09-10T18:53:33.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[32381]: cluster 2024-09-10T18:53:33.355772+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 deep-scrub ok 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:32.539298+0000 mgr.y (mgr.24526) 157 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:32.750327+0000 osd.6 (osd.6) 71 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:32.756721+0000 osd.6 (osd.6) 72 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:32.826386+0000 osd.0 (osd.0) 109 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:32.830053+0000 osd.0 (osd.0) 110 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:33.341792+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 deep-scrub starts 2024-09-10T18:53:33.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:33 smithi083 bash[18629]: cluster 2024-09-10T18:53:33.355772+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 deep-scrub ok 2024-09-10T18:53:33.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:33.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:34.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:32.539298+0000 mgr.y (mgr.24526) 157 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:34.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:32.750327+0000 osd.6 (osd.6) 71 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T18:53:34.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:32.756721+0000 osd.6 (osd.6) 72 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T18:53:34.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:32.826386+0000 osd.0 (osd.0) 109 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:53:34.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:32.830053+0000 osd.0 (osd.0) 110 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:53:34.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:33.341792+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 deep-scrub starts 2024-09-10T18:53:34.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:33 smithi165 bash[22591]: cluster 2024-09-10T18:53:33.355772+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 deep-scrub ok 2024-09-10T18:53:34.049 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:34.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:34.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:34.326 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:34.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:34.466 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:34.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:34.604 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:34.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.746 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:34.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:34.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:34.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:34.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:35.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:34 smithi165 bash[22591]: cluster 2024-09-10T18:53:33.246943+0000 osd.4 (osd.4) 121 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:53:35.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:34 smithi165 bash[22591]: cluster 2024-09-10T18:53:33.250816+0000 osd.4 (osd.4) 122 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:53:35.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:34 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.323255+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:53:35.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:34 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.326574+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:53:35.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:35.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[18629]: cluster 2024-09-10T18:53:33.246943+0000 osd.4 (osd.4) 121 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:53:35.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[18629]: cluster 2024-09-10T18:53:33.250816+0000 osd.4 (osd.4) 122 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:53:35.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.323255+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:53:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.326574+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:53:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[32381]: cluster 2024-09-10T18:53:33.246943+0000 osd.4 (osd.4) 121 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:53:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[32381]: cluster 2024-09-10T18:53:33.250816+0000 osd.4 (osd.4) 122 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:53:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.323255+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:53:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:34 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.326574+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:53:35.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:35.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:35.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:35.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:35.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:35.425 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:35.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:35.560 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:35.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:35.692 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:35.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:35.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.539957+0000 mgr.y (mgr.24526) 158 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:35.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.548855+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:53:35.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.552190+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:53:35.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.733243+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:53:35.828 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[18629]: cluster 2024-09-10T18:53:34.736693+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:53:35.854 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:35.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:35.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:35.986 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:35.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:36.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:36.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:35 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.539957+0000 mgr.y (mgr.24526) 158 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:36.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:35 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.548855+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:53:36.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:35 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.552190+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:53:36.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:35 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.733243+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:53:36.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:35 smithi165 bash[22591]: cluster 2024-09-10T18:53:34.736693+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:53:36.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:36.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.539957+0000 mgr.y (mgr.24526) 158 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.548855+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:53:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.552190+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:53:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.733243+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:53:36.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:35 smithi083 bash[32381]: cluster 2024-09-10T18:53:34.736693+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:53:36.259 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:36.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:36.395 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:36.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.533 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:36.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:36.672 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:36.776 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.802 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:36.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:36.936 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:36.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:37.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:37.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[18629]: cluster 2024-09-10T18:53:35.736345+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:53:37.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[18629]: cluster 2024-09-10T18:53:35.739352+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:53:37.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[18629]: cluster 2024-09-10T18:53:35.833541+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:53:37.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[18629]: cluster 2024-09-10T18:53:35.836200+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:53:37.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[32381]: cluster 2024-09-10T18:53:35.736345+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:53:37.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[32381]: cluster 2024-09-10T18:53:35.739352+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:53:37.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[32381]: cluster 2024-09-10T18:53:35.833541+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:53:37.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:36 smithi083 bash[32381]: cluster 2024-09-10T18:53:35.836200+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:53:37.072 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:36 smithi165 bash[22591]: cluster 2024-09-10T18:53:35.736345+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:53:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:36 smithi165 bash[22591]: cluster 2024-09-10T18:53:35.739352+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:53:37.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:36 smithi165 bash[22591]: cluster 2024-09-10T18:53:35.833541+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:53:37.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:36 smithi165 bash[22591]: cluster 2024-09-10T18:53:35.836200+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:53:37.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:37.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:37.273 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:37.274 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:37.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:37.345 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:37.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:37.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:37.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:37.616 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:37.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:37.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:37.747 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:37.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:37.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.092193+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:53:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.097188+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:53:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.541106+0000 mgr.y (mgr.24526) 159 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.585244+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:37.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.585365+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.839673+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-10T18:53:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.861669+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:53:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:37 smithi165 bash[22591]: cluster 2024-09-10T18:53:36.866547+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:53:37.875 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:37.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:37.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.013 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:38.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.092193+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:53:38.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.097188+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:53:38.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.541106+0000 mgr.y (mgr.24526) 159 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.585244+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.585365+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.839673+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.861669+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[18629]: cluster 2024-09-10T18:53:36.866547+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.092193+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.097188+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.541106+0000 mgr.y (mgr.24526) 159 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.585244+0000 osd.7 (osd.7) 105 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.585365+0000 osd.7 (osd.7) 106 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:38.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.839673+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-09-10T18:53:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.861669+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:53:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:37 smithi083 bash[32381]: cluster 2024-09-10T18:53:36.866547+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:53:38.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:38.146 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:38.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:38.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.413 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:38.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:38.552 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:38.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:38.685 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:38.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:38.815 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:38.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.954 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:38.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:39.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:37.374764+0000 osd.6 (osd.6) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:37.374884+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: audit 2024-09-10T18:53:37.588123+0000 mgr.y (mgr.24526) 160 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:37.663149+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:37.663261+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:37.848366+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.114732+0000 osd.4 (osd.4) 123 : cluster [DBG] 3.14 deep-scrub starts 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.301315+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.304784+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:37.374764+0000 osd.6 (osd.6) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:37.374884+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: audit 2024-09-10T18:53:37.588123+0000 mgr.y (mgr.24526) 160 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:37.663149+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:37.663261+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:37.848366+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-10T18:53:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.114732+0000 osd.4 (osd.4) 123 : cluster [DBG] 3.14 deep-scrub starts 2024-09-10T18:53:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.301315+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T18:53:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:38 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.304784+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T18:53:39.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:37.374764+0000 osd.6 (osd.6) 77 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:37.374884+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: audit 2024-09-10T18:53:37.588123+0000 mgr.y (mgr.24526) 160 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:37.663149+0000 osd.5 (osd.5) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:37.663261+0000 osd.5 (osd.5) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:37.848366+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.114732+0000 osd.4 (osd.4) 123 : cluster [DBG] 3.14 deep-scrub starts 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.301315+0000 osd.1 (osd.1) 107 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T18:53:39.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:38 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.304784+0000 osd.1 (osd.1) 108 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T18:53:39.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:39.195 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:39.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:39.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:39.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:39.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:39.489 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:39.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:39.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:39.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:39.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:39.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:39.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:39.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:39.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:39.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:39.994 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.119675+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:53:39.994 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.123834+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.127104+0000 osd.4 (osd.4) 124 : cluster [DBG] 3.14 deep-scrub ok 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[32381]: cluster 2024-09-10T18:53:38.541716+0000 mgr.y (mgr.24526) 161 : cluster [DBG] pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.119675+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.123834+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.127104+0000 osd.4 (osd.4) 124 : cluster [DBG] 3.14 deep-scrub ok 2024-09-10T18:53:39.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:39 smithi083 bash[18629]: cluster 2024-09-10T18:53:38.541716+0000 mgr.y (mgr.24526) 161 : cluster [DBG] pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-10T18:53:40.010 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:40.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:40.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:39 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.119675+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:53:40.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:39 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.123834+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:53:40.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:39 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.127104+0000 osd.4 (osd.4) 124 : cluster [DBG] 3.14 deep-scrub ok 2024-09-10T18:53:40.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:39 smithi165 bash[22591]: cluster 2024-09-10T18:53:38.541716+0000 mgr.y (mgr.24526) 161 : cluster [DBG] pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-09-10T18:53:40.140 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:40.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:40.276 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:40.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:40.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:40.543 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:40.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:40.679 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:40.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:40.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:40.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:40.947 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:40.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:41.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:41.062 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:40 smithi083 bash[18629]: cluster 2024-09-10T18:53:39.555386+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:53:41.062 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:40 smithi083 bash[18629]: cluster 2024-09-10T18:53:39.558685+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:53:41.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:40 smithi083 bash[32381]: cluster 2024-09-10T18:53:39.555386+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:53:41.062 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:40 smithi083 bash[32381]: cluster 2024-09-10T18:53:39.558685+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:53:41.078 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:41.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:41.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:40 smithi165 bash[22591]: cluster 2024-09-10T18:53:39.555386+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:53:41.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:40 smithi165 bash[22591]: cluster 2024-09-10T18:53:39.558685+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:53:41.215 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:41.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:41.355 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:41.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:41.486 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:41.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:41.601 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:41.621 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:41.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:41.751 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:41.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:41.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:41.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:41.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:42.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[32381]: cluster 2024-09-10T18:53:40.122436+0000 osd.7 (osd.7) 109 : cluster [DBG] 3.e scrub starts 2024-09-10T18:53:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[32381]: cluster 2024-09-10T18:53:40.127468+0000 osd.7 (osd.7) 110 : cluster [DBG] 3.e scrub ok 2024-09-10T18:53:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[32381]: cluster 2024-09-10T18:53:40.542788+0000 mgr.y (mgr.24526) 162 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[18629]: cluster 2024-09-10T18:53:40.122436+0000 osd.7 (osd.7) 109 : cluster [DBG] 3.e scrub starts 2024-09-10T18:53:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[18629]: cluster 2024-09-10T18:53:40.127468+0000 osd.7 (osd.7) 110 : cluster [DBG] 3.e scrub ok 2024-09-10T18:53:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:41 smithi083 bash[18629]: cluster 2024-09-10T18:53:40.542788+0000 mgr.y (mgr.24526) 162 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:42.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:42.138 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:41 smithi165 bash[22591]: cluster 2024-09-10T18:53:40.122436+0000 osd.7 (osd.7) 109 : cluster [DBG] 3.e scrub starts 2024-09-10T18:53:42.138 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:41 smithi165 bash[22591]: cluster 2024-09-10T18:53:40.127468+0000 osd.7 (osd.7) 110 : cluster [DBG] 3.e scrub ok 2024-09-10T18:53:42.138 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:41 smithi165 bash[22591]: cluster 2024-09-10T18:53:40.542788+0000 mgr.y (mgr.24526) 162 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:42.156 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:42.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:42.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:42.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:42.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:42.565 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:42.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:42.704 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:42.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:42.830 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:42.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:42.941 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:42] "GET /metrics HTTP/1.1" 200 220131 "" "Prometheus/2.43.0" 2024-09-10T18:53:42.955 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:42.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:43.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:43.093 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:43.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:43.234 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:43.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:43.363 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:43.466 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:43.496 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:43.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:43.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:43.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:43.763 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:43.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:43.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[32381]: cluster 2024-09-10T18:53:42.543494+0000 mgr.y (mgr.24526) 163 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:43.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[32381]: cluster 2024-09-10T18:53:42.774223+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:53:43.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[32381]: cluster 2024-09-10T18:53:42.782863+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:53:43.903 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:43.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:44.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:44.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:43 smithi165 bash[22591]: cluster 2024-09-10T18:53:42.543494+0000 mgr.y (mgr.24526) 163 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:43 smithi165 bash[22591]: cluster 2024-09-10T18:53:42.774223+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:53:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:43 smithi165 bash[22591]: cluster 2024-09-10T18:53:42.782863+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:53:44.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:44.155 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[18629]: cluster 2024-09-10T18:53:42.543494+0000 mgr.y (mgr.24526) 163 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:44.155 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[18629]: cluster 2024-09-10T18:53:42.774223+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.1f scrub starts 2024-09-10T18:53:44.156 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:43 smithi083 bash[18629]: cluster 2024-09-10T18:53:42.782863+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.1f scrub ok 2024-09-10T18:53:44.178 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:44.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:44.308 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:44.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:44.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:44.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.586 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:44.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:44.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:44.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.860 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:44.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.601909+0000 osd.5 (osd.5) 103 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T18:53:44.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.611745+0000 osd.5 (osd.5) 104 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T18:53:44.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.756730+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:53:44.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.759395+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:53:44.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.845687+0000 osd.0 (osd.0) 115 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:53:44.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: cluster 2024-09-10T18:53:43.850597+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:53:44.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:44 smithi165 bash[22591]: audit 2024-09-10T18:53:44.552968+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:44.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:44.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:45.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:45.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:45.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:45.268 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:45.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.601909+0000 osd.5 (osd.5) 103 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T18:53:45.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.611745+0000 osd.5 (osd.5) 104 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T18:53:45.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.756730+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.759395+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.845687+0000 osd.0 (osd.0) 115 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: cluster 2024-09-10T18:53:43.850597+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[18629]: audit 2024-09-10T18:53:44.552968+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.601909+0000 osd.5 (osd.5) 103 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.611745+0000 osd.5 (osd.5) 104 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.756730+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.759395+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.845687+0000 osd.0 (osd.0) 115 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: cluster 2024-09-10T18:53:43.850597+0000 osd.0 (osd.0) 116 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:53:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:44 smithi083 bash[32381]: audit 2024-09-10T18:53:44.552968+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:53:45.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:45.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:45.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:45.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:45.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:45.667 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:45.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:45.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:45.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:45 smithi083 bash[32381]: cluster 2024-09-10T18:53:44.543997+0000 mgr.y (mgr.24526) 164 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:53:45.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:45.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:45.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:46.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:46.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:45 smithi083 bash[18629]: cluster 2024-09-10T18:53:44.543997+0000 mgr.y (mgr.24526) 164 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:53:46.070 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:46.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:46.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:46.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:45 smithi165 bash[22591]: cluster 2024-09-10T18:53:44.543997+0000 mgr.y (mgr.24526) 164 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:53:46.332 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:46.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:46.455 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:46.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:46.582 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:46.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:46.709 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:46.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:46.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:46.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:46.955 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:46 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.244778+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 deep-scrub starts 2024-09-10T18:53:46.956 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:46 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.247293+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 deep-scrub ok 2024-09-10T18:53:46.956 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:46 smithi165 bash[22591]: audit 2024-09-10T18:53:46.419526+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:53:46.956 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:46 smithi165 bash[22591]: audit 2024-09-10T18:53:46.874083+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:53:46.956 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:46 smithi165 bash[22591]: audit 2024-09-10T18:53:46.875421+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:53:46.975 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:46.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:47.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:47.091 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.244778+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 deep-scrub starts 2024-09-10T18:53:47.091 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.247293+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 deep-scrub ok 2024-09-10T18:53:47.091 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[18629]: audit 2024-09-10T18:53:46.419526+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[18629]: audit 2024-09-10T18:53:46.874083+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[18629]: audit 2024-09-10T18:53:46.875421+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.244778+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.2 deep-scrub starts 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.247293+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.2 deep-scrub ok 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[32381]: audit 2024-09-10T18:53:46.419526+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[32381]: audit 2024-09-10T18:53:46.874083+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:53:47.092 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:46 smithi083 bash[32381]: audit 2024-09-10T18:53:46.875421+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:53:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:47.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:47.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:47.244 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:47.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:47.377 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:47.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.512 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:47.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:47.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:47.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:47.781 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:47.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:47.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:48.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.192482+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.196113+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.545292+0000 mgr.y (mgr.24526) 165 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.597225+0000 osd.5 (osd.5) 105 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: cluster 2024-09-10T18:53:46.600667+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T18:53:48.033 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:47 smithi165 bash[22591]: audit 2024-09-10T18:53:46.885747+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:53:48.045 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:48.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:48.182 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:48.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.192482+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:53:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.196113+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:53:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.545292+0000 mgr.y (mgr.24526) 165 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:53:48.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.597225+0000 osd.5 (osd.5) 105 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: cluster 2024-09-10T18:53:46.600667+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[18629]: audit 2024-09-10T18:53:46.885747+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.192482+0000 osd.4 (osd.4) 125 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.196113+0000 osd.4 (osd.4) 126 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.545292+0000 mgr.y (mgr.24526) 165 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.597225+0000 osd.5 (osd.5) 105 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: cluster 2024-09-10T18:53:46.600667+0000 osd.5 (osd.5) 106 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T18:53:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:47 smithi083 bash[32381]: audit 2024-09-10T18:53:46.885747+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:53:48.319 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:48.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:48.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:48.456 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:48.559 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:48.584 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:48.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:48.721 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:48.824 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:48.854 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:48.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:48.971 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:48 smithi165 bash[22591]: cluster 2024-09-10T18:53:47.218911+0000 osd.4 (osd.4) 127 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:53:48.971 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:48 smithi165 bash[22591]: cluster 2024-09-10T18:53:47.224292+0000 osd.4 (osd.4) 128 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:53:48.972 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:48 smithi165 bash[22591]: audit 2024-09-10T18:53:47.592461+0000 mgr.y (mgr.24526) 166 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:48.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:48.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:49.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.128 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:49.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.265 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:49.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[18629]: cluster 2024-09-10T18:53:47.218911+0000 osd.4 (osd.4) 127 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:53:49.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[18629]: cluster 2024-09-10T18:53:47.224292+0000 osd.4 (osd.4) 128 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:53:49.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[18629]: audit 2024-09-10T18:53:47.592461+0000 mgr.y (mgr.24526) 166 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:49.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[32381]: cluster 2024-09-10T18:53:47.218911+0000 osd.4 (osd.4) 127 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:53:49.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[32381]: cluster 2024-09-10T18:53:47.224292+0000 osd.4 (osd.4) 128 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:53:49.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:48 smithi083 bash[32381]: audit 2024-09-10T18:53:47.592461+0000 mgr.y (mgr.24526) 166 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:49.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:49.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:49.541 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:49.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:49.674 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:49.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:49.807 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:49.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:49.947 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:49.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:50.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[18629]: cluster 2024-09-10T18:53:48.545771+0000 mgr.y (mgr.24526) 167 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[18629]: cluster 2024-09-10T18:53:48.798447+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[18629]: cluster 2024-09-10T18:53:48.802650+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[18629]: cluster 2024-09-10T18:53:48.845796+0000 osd.0 (osd.0) 117 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[18629]: cluster 2024-09-10T18:53:48.848973+0000 osd.0 (osd.0) 118 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[32381]: cluster 2024-09-10T18:53:48.545771+0000 mgr.y (mgr.24526) 167 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[32381]: cluster 2024-09-10T18:53:48.798447+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[32381]: cluster 2024-09-10T18:53:48.802650+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[32381]: cluster 2024-09-10T18:53:48.845796+0000 osd.0 (osd.0) 117 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:53:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:49 smithi083 bash[32381]: cluster 2024-09-10T18:53:48.848973+0000 osd.0 (osd.0) 118 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:53:50.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:50.081 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:50.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:50.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:49 smithi165 bash[22591]: cluster 2024-09-10T18:53:48.545771+0000 mgr.y (mgr.24526) 167 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 956 B/s rd, 0 op/s 2024-09-10T18:53:50.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:49 smithi165 bash[22591]: cluster 2024-09-10T18:53:48.798447+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:53:50.197 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:49 smithi165 bash[22591]: cluster 2024-09-10T18:53:48.802650+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:53:50.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:49 smithi165 bash[22591]: cluster 2024-09-10T18:53:48.845796+0000 osd.0 (osd.0) 117 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:53:50.198 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:49 smithi165 bash[22591]: cluster 2024-09-10T18:53:48.848973+0000 osd.0 (osd.0) 118 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:53:50.216 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:50.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:50.356 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:50.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:50.496 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:50.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:50.635 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:50.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:50.769 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:50.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:50.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:50.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:51.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:51.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:50 smithi165 bash[22591]: cluster 2024-09-10T18:53:49.105040+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:53:51.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:50 smithi165 bash[22591]: cluster 2024-09-10T18:53:49.108825+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:53:51.043 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:51.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:50 smithi083 bash[18629]: cluster 2024-09-10T18:53:49.105040+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:53:51.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:50 smithi083 bash[18629]: cluster 2024-09-10T18:53:49.108825+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:53:51.082 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:50 smithi083 bash[32381]: cluster 2024-09-10T18:53:49.105040+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:53:51.082 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:50 smithi083 bash[32381]: cluster 2024-09-10T18:53:49.108825+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:53:51.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:51.196 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:51.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:51.333 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:51.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:51.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:51.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:51.601 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:51.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:53:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:53:51.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.637+0000 7f4a7b51d640 1 -- 172.21.15.83:0/1690311938 >> v1:172.21.15.83:6790/0 conn(0x7f4a741014e0 legacy=0x7f4a741018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.637+0000 7f4a7b51d640 1 -- 172.21.15.83:0/1690311938 shutdown_connections 2024-09-10T18:53:51.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.637+0000 7f4a7b51d640 1 -- 172.21.15.83:0/1690311938 >> 172.21.15.83:0/1690311938 conn(0x7f4a74075290 msgr2=0x7f4a740f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:51.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.637+0000 7f4a7b51d640 1 -- 172.21.15.83:0/1690311938 shutdown_connections 2024-09-10T18:53:51.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.637+0000 7f4a7b51d640 1 -- 172.21.15.83:0/1690311938 wait complete. 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 librados: starting msgr at 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 librados: starting objecter 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 Processor -- start 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 -- start start 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 librados: setting wanted keys 2024-09-10T18:53:51.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 librados: calling monclient init 2024-09-10T18:53:51.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a7406ea40 con 0x7f4a741014e0 2024-09-10T18:53:51.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a7406ebf0 con 0x7f4a741042a0 2024-09-10T18:53:51.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a7b51d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4a7406ede0 con 0x7f4a74107e30 2024-09-10T18:53:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a79aba640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f4a74107e30 0x7f4a7406e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:53594/0 (socket says 172.21.15.83:53594) 2024-09-10T18:53:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a78ab8640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4a741042a0 0x7f4a74065610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:46274/0 (socket says 172.21.15.83:46274) 2024-09-10T18:53:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a79aba640 1 -- 172.21.15.83:0/2075645610 learned_addr learned my addr 172.21.15.83:0/2075645610 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:51.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter ms_handle_connect 0x7f4a741042a0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter ms_handle_connect 0x7f4a74107e30 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter ms_handle_connect 0x7f4a741014e0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253190367 0 0) 0x7f4a7406ebf0 con 0x7f4a741042a0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a600038b0 con 0x7f4a741042a0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073245235 0 0) 0x7f4a7406ede0 con 0x7f4a74107e30 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7406ebf0 con 0x7f4a74107e30 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946556579 0 0) 0x7f4a7406ea40 con 0x7f4a741014e0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7406ede0 con 0x7f4a741014e0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335453466 0 0) 0x7f4a7406ede0 con 0x7f4a741014e0 2024-09-10T18:53:51.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a7406ea40 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4a6c0015d0 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470432221 0 0) 0x7f4a600038b0 con 0x7f4a741042a0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.641+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a7406ede0 con 0x7f4a741042a0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4a64002990 con 0x7f4a741042a0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1010255898 0 0) 0x7f4a7406ea40 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6790/0 conn(0x7f4a74107e30 legacy=0x7f4a7406e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.165:6789/0 conn(0x7f4a741042a0 legacy=0x7f4a74065610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7406efd0 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7406f280 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a7b51d640 10 client.14823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7406f6d0 con 0x7f4a741014e0 2024-09-10T18:53:51.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a7b51d640 1 librados: init done 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4a6c001b30 con 0x7f4a741014e0 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4a6c0034b0 con 0x7f4a741014e0 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4a6c019a40 con 0x7f4a741014e0 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(115..115 src has 59..115) v4 ==== 6284+0+0 (unknown 3940423219 0 0) 0x7f4a6c0878e0 con 0x7f4a741014e0 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 osd_map(115..115 src has 59..115) v4 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 3 client.14823.objecter handle_osd_map got epochs [115,115] > 0 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 3 client.14823.objecter handle_osd_map decoding full epoch 115 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 20 client.14823.objecter dump_active .. 0 homeless 2024-09-10T18:53:51.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.645+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a600716a0 2024-09-10T18:53:51.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.653+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4a58004ff0 con 0x7f4a741014e0 2024-09-10T18:53:51.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v115) v1 ==== 54+0+16 (unknown 553586118 0 2420000192) 0x7f4a6c04fe60 con 0x7f4a741014e0 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 1 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 1 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 0 1 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 4 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 10 client.14823.objecter _send_command 1 2024-09-10T18:53:51.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58005290 con 0x7f4a58001b90 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.657+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58001b90 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.661+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58005290 con 0x7f4a58001b90 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.661+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.661+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 1 = system:0 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.661+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 4 2024-09-10T18:53:51.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.661+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 0 1 2024-09-10T18:53:51.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 2 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 2 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58005810 osd=0 3 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 0 2 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 4 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 10 client.14823.objecter _send_command 2 2024-09-10T18:53:51.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.673+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58002200 con 0x7f4a58001b90 2024-09-10T18:53:51.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58002200 con 0x7f4a58001b90 2024-09-10T18:53:51.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 4 2024-09-10T18:53:51.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 0 2 2024-09-10T18:53:51.679 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:53:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:53:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 3 2024-09-10T18:53:51.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 1 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 4 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 10 client.14823.objecter _send_command 3 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58003b40 con 0x7f4a580035e0 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a580035e0 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58003b40 con 0x7f4a580035e0 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 3 = system:0 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 4 2024-09-10T18:53:51.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.677+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 1 3 2024-09-10T18:53:51.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:51.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 4 2024-09-10T18:53:51.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 4 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a580023b0 osd=1 3 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 1 4 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 4 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 10 client.14823.objecter _send_command 4 2024-09-10T18:53:51.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.701+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a580030d0 con 0x7f4a580035e0 2024-09-10T18:53:51.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a580030d0 con 0x7f4a580035e0 2024-09-10T18:53:51.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 4 2024-09-10T18:53:51.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 1 4 2024-09-10T18:53:51.708 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 5 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 5 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 2 5 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 4 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 10 client.14823.objecter _send_command 5 2024-09-10T18:53:51.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58004d70 con 0x7f4a5800d350 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a5800d350 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58004d70 con 0x7f4a5800d350 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 5 = system:0 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 4 2024-09-10T18:53:51.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.705+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 2 5 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 6 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 6 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58004610 osd=2 3 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 2 6 2024-09-10T18:53:51.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 4 2024-09-10T18:53:51.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 10 client.14823.objecter _send_command 6 2024-09-10T18:53:51.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.725+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58004100 con 0x7f4a5800d350 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58004100 con 0x7f4a5800d350 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 4 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 2 6 2024-09-10T18:53:51.732 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:53:51.733 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:53:51.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 7 2024-09-10T18:53:51.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 7 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 3 7 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 4 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 10 client.14823.objecter _send_command 7 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a5800d170 con 0x7f4a58011fd0 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58011fd0 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a5800d170 con 0x7f4a58011fd0 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 7 = system:0 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 4 2024-09-10T18:53:51.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.729+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 3 7 2024-09-10T18:53:51.738 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:51.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 8 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 8 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a5800c990 osd=3 3 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 3 8 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 4 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 10 client.14823.objecter _send_command 8 2024-09-10T18:53:51.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.753+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a5800d170 con 0x7f4a58011fd0 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a5800d170 con 0x7f4a58011fd0 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 4 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 3 8 2024-09-10T18:53:51.760 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 9 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 9 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 4 9 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 4 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 10 client.14823.objecter _send_command 9 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58011bc0 con 0x7f4a58011610 2024-09-10T18:53:51.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.757+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58011610 2024-09-10T18:53:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.761+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58011bc0 con 0x7f4a58011610 2024-09-10T18:53:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.761+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:53:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.761+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 9 = system:0 2024-09-10T18:53:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.761+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 4 2024-09-10T18:53:51.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.761+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 4 9 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 10 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 10 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a5800d170 osd=4 3 2024-09-10T18:53:51.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 4 10 2024-09-10T18:53:51.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 4 2024-09-10T18:53:51.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 10 client.14823.objecter _send_command 10 2024-09-10T18:53:51.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.781+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58011bc0 con 0x7f4a58011610 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58011bc0 con 0x7f4a58011610 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 4 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 4 10 2024-09-10T18:53:51.788 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 11 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 11 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 5 11 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 4 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 10 client.14823.objecter _send_command 11 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58017cc0 con 0x7f4a58017870 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.785+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58017870 2024-09-10T18:53:51.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.789+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58017cc0 con 0x7f4a58017870 2024-09-10T18:53:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.789+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:53:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.789+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 11 = system:0 2024-09-10T18:53:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.789+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 4 2024-09-10T18:53:51.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.789+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 5 11 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 12 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 12 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58011bc0 osd=5 3 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 5 12 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 4 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 10 client.14823.objecter _send_command 12 2024-09-10T18:53:51.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.813+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58017cc0 con 0x7f4a58017870 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58017cc0 con 0x7f4a58017870 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 4 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 5 12 2024-09-10T18:53:51.819 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 13 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 13 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 6 13 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 4 2024-09-10T18:53:51.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 10 client.14823.objecter _send_command 13 2024-09-10T18:53:51.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58022890 con 0x7f4a58017eb0 2024-09-10T18:53:51.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58017eb0 2024-09-10T18:53:51.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58022890 con 0x7f4a58017eb0 2024-09-10T18:53:51.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:53:51.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 13 = system:0 2024-09-10T18:53:51.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 4 2024-09-10T18:53:51.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.817+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 6 13 2024-09-10T18:53:51.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 14 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 14 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58017cc0 osd=6 3 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 6 14 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 4 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 10 client.14823.objecter _send_command 14 2024-09-10T18:53:51.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.845+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58022890 con 0x7f4a58017eb0 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58022890 con 0x7f4a58017eb0 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 4 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 6 14 2024-09-10T18:53:51.850 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 15 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 15 2024-09-10T18:53:51.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 7 15 2024-09-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 4 2024-09-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 10 client.14823.objecter _send_command 15 2024-09-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a58023ae0 con 0x7f4a58024680 2024-09-10T18:53:51.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 10 client.14823.objecter ms_handle_connect 0x7f4a58024680 2024-09-10T18:53:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4a58023ae0 con 0x7f4a58024680 2024-09-10T18:53:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:53:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 15 = system:0 2024-09-10T18:53:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 4 2024-09-10T18:53:51.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.849+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 7 15 2024-09-10T18:53:51.868 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:51.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:51.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 10 client.14823.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign -1 16 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 20 client.14823.objecter _get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_remove -1 16 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 20 client.14823.objecter get_session s=0x7f4a58022890 osd=7 3 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 15 client.14823.objecter _session_command_op_assign 7 16 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 4 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 10 client.14823.objecter _send_command 16 2024-09-10T18:53:51.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.873+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4a58023ae0 con 0x7f4a58024680 2024-09-10T18:53:51.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.877+0000 7f4a69ffb640 1 -- 172.21.15.83:0/2075645610 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4a58023ae0 con 0x7f4a58024680 2024-09-10T18:53:51.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.877+0000 7f4a69ffb640 10 client.14823.objecter ms_dispatch 0x7f4a74000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:53:51.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.877+0000 7f4a69ffb640 10 client.14823.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:53:51.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.877+0000 7f4a69ffb640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 4 2024-09-10T18:53:51.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.877+0000 7f4a69ffb640 15 client.14823.objecter _session_command_op_remove 7 16 2024-09-10T18:53:51.882 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:53:51.883 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:53:51.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 librados: watch_flush enter 2024-09-10T18:53:51.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 librados: watch_flush exit 2024-09-10T18:53:51.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.0 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6801/2535527483 conn(0x7f4a58001b90 legacy=0x7f4a58002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58005810 osd=0 1 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.1 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6805/2309624637 conn(0x7f4a580035e0 legacy=0x7f4a5800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a580023b0 osd=1 1 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.2 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6809/386094220 conn(0x7f4a5800d350 legacy=0x7f4a5800f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58004610 osd=2 1 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.3 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6813/1279688021 conn(0x7f4a58011fd0 legacy=0x7f4a580143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800c990 osd=3 1 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.4 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a79aba640 1 -- 172.21.15.83:0/2075645610 reap_dead start 2024-09-10T18:53:51.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.165:6800/3225515265 conn(0x7f4a58011610 legacy=0x7f4a58016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a79aba640 1 -- 172.21.15.83:0/2075645610 reap_dead start 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a5800d170 osd=4 1 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.5 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.165:6804/571173777 conn(0x7f4a58017870 legacy=0x7f4a5801d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58011bc0 osd=5 1 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.6 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.165:6808/1317922936 conn(0x7f4a58017eb0 legacy=0x7f4a58021000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58017cc0 osd=6 1 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter close_session for osd.7 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.165:6812/1610151650 conn(0x7f4a58024680 legacy=0x7f4a58024ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter put_session s=0x7f4a58022890 osd=7 1 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 20 client.14823.objecter shutdown clearing up homeless session... 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 10 client.14823.objecter successfully canceled tick 2024-09-10T18:53:51.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6800/3071793735 conn(0x7f4a600716a0 legacy=0x7f4a60073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a79aba640 1 -- 172.21.15.83:0/2075645610 reap_dead start 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> v1:172.21.15.83:6789/0 conn(0x7f4a741014e0 legacy=0x7f4a74064f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 shutdown_connections 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 >> 172.21.15.83:0/2075645610 conn(0x7f4a74075290 msgr2=0x7f4a740f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 shutdown_connections 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 -- 172.21.15.83:0/2075645610 wait complete. 2024-09-10T18:53:51.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:51.881+0000 7f4a7b51d640 1 librados: shutdown 2024-09-10T18:53:51.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:51.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:51 smithi165 bash[22591]: cluster 2024-09-10T18:53:50.546564+0000 mgr.y (mgr.24526) 168 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:51.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:51 smithi165 bash[22591]: cluster 2024-09-10T18:53:50.693444+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:53:51.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:51 smithi165 bash[22591]: cluster 2024-09-10T18:53:50.696238+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:53:51.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:51 smithi165 bash[22591]: audit 2024-09-10T18:53:51.658796+0000 mon.a (mon.0) 874 : audit [DBG] from='client.? v1:172.21.15.83:0/2075645610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:53:52.007 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:52.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:52.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[18629]: cluster 2024-09-10T18:53:50.546564+0000 mgr.y (mgr.24526) 168 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:52.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[18629]: cluster 2024-09-10T18:53:50.693444+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:53:52.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[18629]: cluster 2024-09-10T18:53:50.696238+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:53:52.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[18629]: audit 2024-09-10T18:53:51.658796+0000 mon.a (mon.0) 874 : audit [DBG] from='client.? v1:172.21.15.83:0/2075645610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:53:52.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[32381]: cluster 2024-09-10T18:53:50.546564+0000 mgr.y (mgr.24526) 168 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:52.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[32381]: cluster 2024-09-10T18:53:50.693444+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:53:52.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[32381]: cluster 2024-09-10T18:53:50.696238+0000 osd.3 (osd.3) 130 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:53:52.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:51 smithi083 bash[32381]: audit 2024-09-10T18:53:51.658796+0000 mon.a (mon.0) 874 : audit [DBG] from='client.? v1:172.21.15.83:0/2075645610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:53:52.133 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:52.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:52.273 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:53:52.273 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:53:52.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.389 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:52.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:52.517 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:52.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:52.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:52.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:52.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.780 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:52.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:52.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:52.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:52.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:52 smithi083 bash[18629]: cluster 2024-09-10T18:53:51.811711+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.0 deep-scrub starts 2024-09-10T18:53:52.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:52 smithi083 bash[18629]: cluster 2024-09-10T18:53:51.813894+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 deep-scrub ok 2024-09-10T18:53:52.958 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:53:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:53:52] "GET /metrics HTTP/1.1" 200 220156 "" "Prometheus/2.43.0" 2024-09-10T18:53:53.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:53.049 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:53.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:53.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:52 smithi165 bash[22591]: cluster 2024-09-10T18:53:51.811711+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.0 deep-scrub starts 2024-09-10T18:53:53.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:52 smithi165 bash[22591]: cluster 2024-09-10T18:53:51.813894+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 deep-scrub ok 2024-09-10T18:53:53.185 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:53.252 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:52 smithi083 bash[32381]: cluster 2024-09-10T18:53:51.811711+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.0 deep-scrub starts 2024-09-10T18:53:53.253 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:52 smithi083 bash[32381]: cluster 2024-09-10T18:53:51.813894+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.0 deep-scrub ok 2024-09-10T18:53:53.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.312 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:53.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- 172.21.15.83:0/784532581 >> v1:172.21.15.83:6789/0 conn(0x7f08d81014e0 legacy=0x7f08d81018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:53.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- 172.21.15.83:0/784532581 shutdown_connections 2024-09-10T18:53:53.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- 172.21.15.83:0/784532581 >> 172.21.15.83:0/784532581 conn(0x7f08d8075290 msgr2=0x7f08d80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:53.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- 172.21.15.83:0/784532581 shutdown_connections 2024-09-10T18:53:53.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- 172.21.15.83:0/784532581 wait complete. 2024-09-10T18:53:53.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 librados: starting msgr at 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 librados: starting objecter 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 Processor -- start 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- start start 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 librados: setting wanted keys 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 librados: calling monclient init 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f08d8122780 con 0x7f08d81014e0 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f08d8123920 con 0x7f08d81042a0 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08dd36e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f08d8124ac0 con 0x7f08d8107e30 2024-09-10T18:53:53.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08d6ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f08d81014e0 0x7f08d8064ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43552/0 (socket says 172.21.15.83:43552) 2024-09-10T18:53:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08d77fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f08d8107e30 0x7f08d8121060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47922/0 (socket says 172.21.15.83:47922) 2024-09-10T18:53:53.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08d77fe640 1 -- 172.21.15.83:0/2030836995 learned_addr learned my addr 172.21.15.83:0/2030836995 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d8107e30 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d81014e0 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d81042a0 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469616532 0 0) 0x7f08d8122780 con 0x7f08d81014e0 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c0003200 con 0x7f08d81014e0 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478334140 0 0) 0x7f08d8124ac0 con 0x7f08d8107e30 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d8122780 con 0x7f08d8107e30 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850376298 0 0) 0x7f08d8123920 con 0x7f08d81042a0 2024-09-10T18:53:53.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d8124ac0 con 0x7f08d81042a0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1426634166 0 0) 0x7f08c0003200 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d8123920 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f08c8002050 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712288729 0 0) 0x7f08d8122780 con 0x7f08d8107e30 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08c0003200 con 0x7f08d8107e30 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.389+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f08d0001bc0 con 0x7f08d8107e30 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2993288784 0 0) 0x7f08d8123920 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 >> v1:172.21.15.83:6790/0 conn(0x7f08d8107e30 legacy=0x7f08d8121060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 >> v1:172.21.15.165:6789/0 conn(0x7f08d81042a0 legacy=0x7f08d80655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8125c60 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d8122970 con 0x7f08d81014e0 2024-09-10T18:53:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08dd36e640 10 client.14829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d8122dc0 con 0x7f08d81014e0 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08dd36e640 1 librados: init done 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f08c80022b0 con 0x7f08d81014e0 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f08c80036c0 con 0x7f08d81014e0 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4005010 con 0x7f08d81014e0 2024-09-10T18:53:53.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f08c80017f0 con 0x7f08d81014e0 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(115..115 src has 59..115) v4 ==== 6284+0+0 (unknown 3940423219 0 0) 0x7f08c8089ff0 con 0x7f08d81014e0 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 10 client.14829.objecter ms_dispatch 0x7f08d8000b90 osd_map(115..115 src has 59..115) v4 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 3 client.14829.objecter handle_osd_map got epochs [115,115] > 0 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 3 client.14829.objecter handle_osd_map decoding full epoch 115 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 20 client.14829.objecter dump_active .. 0 homeless 2024-09-10T18:53:53.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.393+0000 7f08bf7fe640 10 client.14829.objecter ms_handle_connect 0x7f08c0071a70 2024-09-10T18:53:53.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.401+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f08c8056c80 con 0x7f08d81014e0 2024-09-10T18:53:53.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:53.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:53.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:53.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.613+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f08c4005840 con 0x7f08d81014e0 2024-09-10T18:53:53.683 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.709 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:53.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.840 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:53.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.921+0000 7f08bf7fe640 1 -- 172.21.15.83:0/2030836995 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v116) v1 ==== 87+0+0 (unknown 4254056820 0 0) 0x7f08c8056620 con 0x7f08d81014e0 2024-09-10T18:53:53.930 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T18:53:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.929+0000 7f08dd36e640 10 librados: watch_flush enter 2024-09-10T18:53:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.929+0000 7f08dd36e640 10 librados: watch_flush exit 2024-09-10T18:53:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.929+0000 7f08dd36e640 20 client.14829.objecter shutdown clearing up homeless session... 2024-09-10T18:53:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.929+0000 7f08dd36e640 10 client.14829.objecter successfully canceled tick 2024-09-10T18:53:53.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 >> v1:172.21.15.83:6800/3071793735 conn(0x7f08c0071a70 legacy=0x7f08c0073f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:53.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 >> v1:172.21.15.83:6789/0 conn(0x7f08d81014e0 legacy=0x7f08d8064ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:53.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 shutdown_connections 2024-09-10T18:53:53.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 >> 172.21.15.83:0/2030836995 conn(0x7f08d8075290 msgr2=0x7f08d80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:53.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 shutdown_connections 2024-09-10T18:53:53.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 -- 172.21.15.83:0/2030836995 wait complete. 2024-09-10T18:53:53.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:53.933+0000 7f08dd36e640 1 librados: shutdown 2024-09-10T18:53:53.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:53.969 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:53.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:54.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:54.080 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.075100+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.078158+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.546987+0000 mgr.y (mgr.24526) 169 : cluster [DBG] pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.597560+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.601877+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.677373+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.c scrub starts 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: cluster 2024-09-10T18:53:52.680770+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.c scrub ok 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[32381]: audit 2024-09-10T18:53:53.615725+0000 mon.a (mon.0) 875 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.075100+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.078158+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.546987+0000 mgr.y (mgr.24526) 169 : cluster [DBG] pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:54.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.597560+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:53:54.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.601877+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:53:54.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.677373+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.c scrub starts 2024-09-10T18:53:54.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: cluster 2024-09-10T18:53:52.680770+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.c scrub ok 2024-09-10T18:53:54.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:53 smithi083 bash[18629]: audit 2024-09-10T18:53:53.615725+0000 mon.a (mon.0) 875 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:53:54.094 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:54.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.075100+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.078158+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.546987+0000 mgr.y (mgr.24526) 169 : cluster [DBG] pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.597560+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.601877+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.677373+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.c scrub starts 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: cluster 2024-09-10T18:53:52.680770+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.c scrub ok 2024-09-10T18:53:54.211 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:53 smithi165 bash[22591]: audit 2024-09-10T18:53:53.615725+0000 mon.a (mon.0) 875 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:53:54.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- 172.21.15.83:0/618234323 >> v1:172.21.15.83:6790/0 conn(0x7f5f141042a0 legacy=0x7f5f14106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- 172.21.15.83:0/618234323 shutdown_connections 2024-09-10T18:53:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- 172.21.15.83:0/618234323 >> 172.21.15.83:0/618234323 conn(0x7f5f140f7070 msgr2=0x7f5f140f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- 172.21.15.83:0/618234323 shutdown_connections 2024-09-10T18:53:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- 172.21.15.83:0/618234323 wait complete. 2024-09-10T18:53:54.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 librados: starting msgr at 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 librados: starting objecter 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 Processor -- start 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 -- start start 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 librados: setting wanted keys 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.249+0000 7f5f1b77b640 1 librados: calling monclient init 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f1411e130 con 0x7f5f14107e00 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f1411f2f0 con 0x7f5f141042a0 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f141204f0 con 0x7f5f14101530 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f19517640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5f14101530 0x7f5f14065b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47932/0 (socket says 172.21.15.83:47932) 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f19517640 1 -- 172.21.15.83:0/95383989 learned_addr learned my addr 172.21.15.83:0/95383989 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter ms_handle_connect 0x7f5f14107e00 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:54.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter ms_handle_connect 0x7f5f14101530 2024-09-10T18:53:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456628788 0 0) 0x7f5f1411e130 con 0x7f5f14107e00 2024-09-10T18:53:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f00003200 con 0x7f5f14107e00 2024-09-10T18:53:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter ms_handle_connect 0x7f5f141042a0 2024-09-10T18:53:54.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873986791 0 0) 0x7f5f141204f0 con 0x7f5f14101530 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f1411e130 con 0x7f5f14101530 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346421559 0 0) 0x7f5f00003200 con 0x7f5f14107e00 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f141204f0 con 0x7f5f14107e00 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5f0c001af0 con 0x7f5f14107e00 2024-09-10T18:53:54.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152863197 0 0) 0x7f5f1411f2f0 con 0x7f5f141042a0 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f00003200 con 0x7f5f141042a0 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685337341 0 0) 0x7f5f1411e130 con 0x7f5f14101530 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f1411f2f0 con 0x7f5f14101530 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5f04001a30 con 0x7f5f14101530 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 585963126 0 0) 0x7f5f141204f0 con 0x7f5f14107e00 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 >> v1:172.21.15.83:6790/0 conn(0x7f5f14101530 legacy=0x7f5f14065b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 >> v1:172.21.15.165:6789/0 conn(0x7f5f141042a0 legacy=0x7f5f14066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f141216f0 con 0x7f5f14107e00 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f141206e0 con 0x7f5f14107e00 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 10 client.14835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f14120bb0 con 0x7f5f14107e00 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 librados: init done 2024-09-10T18:53:54.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5f0c001d60 con 0x7f5f14107e00 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5f0c003450 con 0x7f5f14107e00 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef8005040 con 0x7f5f14107e00 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.253+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5f0c0021f0 con 0x7f5f14107e00 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(116..116 src has 59..116) v4 ==== 6284+0+0 (unknown 1249206604 0 0) 0x7f5f0c087850 con 0x7f5f14107e00 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 10 client.14835.objecter ms_dispatch 0x7f5f14000b90 osd_map(116..116 src has 59..116) v4 2024-09-10T18:53:54.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 3 client.14835.objecter handle_osd_map got epochs [116,116] > 0 2024-09-10T18:53:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 3 client.14835.objecter handle_osd_map decoding full epoch 116 2024-09-10T18:53:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 20 client.14835.objecter dump_active .. 0 homeless 2024-09-10T18:53:54.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.257+0000 7f5f09ffb640 10 client.14835.objecter ms_handle_connect 0x7f5f00071710 2024-09-10T18:53:54.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.265+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5f0c054520 con 0x7f5f14107e00 2024-09-10T18:53:54.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:54.356 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:54.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:53:54.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:54.482 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:54.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.489+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f5ef80052f0 con 0x7f5f14107e00 2024-09-10T18:53:54.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:54.617 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:54.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:54.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:54.746 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:54.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:54.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:54.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:54.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.937+0000 7f5f09ffb640 1 -- 172.21.15.83:0/95383989 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v117) v1 ==== 113+0+0 (unknown 1796078789 0 0) 0x7f5f0c053ec0 con 0x7f5f14107e00 2024-09-10T18:53:54.941 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 10 librados: watch_flush enter 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 10 librados: watch_flush exit 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 20 client.14835.objecter shutdown clearing up homeless session... 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 10 client.14835.objecter successfully canceled tick 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5f00071710 legacy=0x7f5f00073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 >> v1:172.21.15.83:6789/0 conn(0x7f5f14107e00 legacy=0x7f5f1411c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 shutdown_connections 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 >> 172.21.15.83:0/95383989 conn(0x7f5f140f7070 msgr2=0x7f5f140fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 shutdown_connections 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 -- 172.21.15.83:0/95383989 wait complete. 2024-09-10T18:53:54.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:53:54.941+0000 7f5f1b77b640 1 librados: shutdown 2024-09-10T18:53:54.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: cluster 2024-09-10T18:53:53.096167+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: cluster 2024-09-10T18:53:53.099078+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: cluster 2024-09-10T18:53:53.916264+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: audit 2024-09-10T18:53:53.923779+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: cluster 2024-09-10T18:53:53.923862+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[18629]: audit 2024-09-10T18:53:54.493573+0000 mon.a (mon.0) 879 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:53:54.998 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: cluster 2024-09-10T18:53:53.096167+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:53:54.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: cluster 2024-09-10T18:53:53.099078+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:53:54.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: cluster 2024-09-10T18:53:53.916264+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:53:54.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: audit 2024-09-10T18:53:53.923779+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:53:54.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: cluster 2024-09-10T18:53:53.923862+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-10T18:53:54.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:54 smithi083 bash[32381]: audit 2024-09-10T18:53:54.493573+0000 mon.a (mon.0) 879 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:53:55.013 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: cluster 2024-09-10T18:53:53.096167+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:53:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: cluster 2024-09-10T18:53:53.099078+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:53:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: cluster 2024-09-10T18:53:53.916264+0000 mon.a (mon.0) 876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:53:55.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: audit 2024-09-10T18:53:53.923779+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.83:0/2030836995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:53:55.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: cluster 2024-09-10T18:53:53.923862+0000 mon.a (mon.0) 878 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-09-10T18:53:55.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:54 smithi165 bash[22591]: audit 2024-09-10T18:53:54.493573+0000 mon.a (mon.0) 879 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T18:53:55.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:55.141 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:55.244 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:55.269 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:55.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:55.408 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:55.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:55.543 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:55.647 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:55.672 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:55.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:55.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:55.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:55.928 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:55.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T18:53:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:55.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:55.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:53:56.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:56.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:56.199 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:56.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:55 smithi165 bash[22591]: cluster 2024-09-10T18:53:54.547387+0000 mgr.y (mgr.24526) 170 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:56.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:55 smithi165 bash[22591]: audit 2024-09-10T18:53:54.938467+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:53:56.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:55 smithi165 bash[22591]: cluster 2024-09-10T18:53:54.939064+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-10T18:53:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:56.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[18629]: cluster 2024-09-10T18:53:54.547387+0000 mgr.y (mgr.24526) 170 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:56.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[18629]: audit 2024-09-10T18:53:54.938467+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:53:56.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[18629]: cluster 2024-09-10T18:53:54.939064+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-10T18:53:56.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[32381]: cluster 2024-09-10T18:53:54.547387+0000 mgr.y (mgr.24526) 170 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:53:56.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[32381]: audit 2024-09-10T18:53:54.938467+0000 mon.a (mon.0) 880 : audit [INF] from='client.? v1:172.21.15.83:0/95383989' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T18:53:56.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:55 smithi083 bash[32381]: cluster 2024-09-10T18:53:54.939064+0000 mon.a (mon.0) 881 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-09-10T18:53:56.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.338 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:56.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.476 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:56.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:56.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:56.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:56.743 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:56.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:56.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:56.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.022 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:57.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:53:57.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:57.149 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:57.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:57.274 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:57.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.413 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:57.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:57.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:57.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:57.668 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:57.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:57.794 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:57.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:53:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:53:57.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:57.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:58.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:58.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:57 smithi083 bash[18629]: cluster 2024-09-10T18:53:56.548159+0000 mgr.y (mgr.24526) 171 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:58.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:57 smithi083 bash[32381]: cluster 2024-09-10T18:53:56.548159+0000 mgr.y (mgr.24526) 171 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:58.060 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:53:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:53:58.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:58.177 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:57 smithi165 bash[22591]: cluster 2024-09-10T18:53:56.548159+0000 mgr.y (mgr.24526) 171 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:53:58.197 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:58.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:58.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:58.324 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:53:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:53:58.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:58.466 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:58.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:58.603 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:53:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:53:58.706 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:58.730 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:53:58.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:58.865 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:53:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:53:58.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:58.978 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:58 smithi083 bash[18629]: audit 2024-09-10T18:53:57.603665+0000 mgr.y (mgr.24526) 172 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:58.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:58 smithi083 bash[32381]: audit 2024-09-10T18:53:57.603665+0000 mgr.y (mgr.24526) 172 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:58.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:58.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:59.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:58 smithi165 bash[22591]: audit 2024-09-10T18:53:57.603665+0000 mgr.y (mgr.24526) 172 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:53:59.131 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:59.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:59.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:59.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:59.395 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:53:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:53:59.452 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:53:59.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:59.523 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:53:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:53:59.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.658 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:59.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.797 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:53:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:53:59.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:59.924 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:53:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:59.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:00.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:00.049 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:00.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.175 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[18629]: cluster 2024-09-10T18:53:58.548556+0000 mgr.y (mgr.24526) 173 : cluster [DBG] pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:00.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[18629]: audit 2024-09-10T18:53:59.557819+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:00.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[18629]: audit 2024-09-10T18:53:59.559683+0000 mon.c (mon.2) 95 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:00.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[32381]: cluster 2024-09-10T18:53:58.548556+0000 mgr.y (mgr.24526) 173 : cluster [DBG] pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:00.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[32381]: audit 2024-09-10T18:53:59.557819+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:00.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:53:59 smithi083 bash[32381]: audit 2024-09-10T18:53:59.559683+0000 mon.c (mon.2) 95 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:00.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.306 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:00.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:59 smithi165 bash[22591]: cluster 2024-09-10T18:53:58.548556+0000 mgr.y (mgr.24526) 173 : cluster [DBG] pgmap v140: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:00.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:59 smithi165 bash[22591]: audit 2024-09-10T18:53:59.557819+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:00.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:53:59 smithi165 bash[22591]: audit 2024-09-10T18:53:59.559683+0000 mon.c (mon.2) 95 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:00.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.432 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:00.467 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:00.468 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:00.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:00.569 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:00.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:00.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:00.830 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:00.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:00.968 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:00.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:01.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:01.099 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:01.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:01.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:01.362 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:01.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:01.480 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:01.501 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:01.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:01.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:01.631 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:01.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:01.770 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:01.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:01.896 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:01.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:01.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:02.023 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:01 smithi165 bash[22591]: cluster 2024-09-10T18:54:00.549329+0000 mgr.y (mgr.24526) 174 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:02.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:02.160 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:02.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:01 smithi083 bash[18629]: cluster 2024-09-10T18:54:00.549329+0000 mgr.y (mgr.24526) 174 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:02.181 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:02 smithi083 bash[32381]: cluster 2024-09-10T18:54:00.549329+0000 mgr.y (mgr.24526) 174 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:02.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:02.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:02.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:02.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:02.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:02.571 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:02.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:02.709 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:02.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:02.834 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:02.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:02.963 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:02.964 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:02] "GET /metrics HTTP/1.1" 200 220223 "" "Prometheus/2.43.0" 2024-09-10T18:54:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:02.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:03.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.094 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:03.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:03.223 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:03.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:03.362 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:03.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:03.488 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:03.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:03.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:03.753 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:03.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:03.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:03.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:03.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:04.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:03 smithi165 bash[22591]: cluster 2024-09-10T18:54:02.549731+0000 mgr.y (mgr.24526) 175 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:04.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:04.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:03 smithi083 bash[32381]: cluster 2024-09-10T18:54:02.549731+0000 mgr.y (mgr.24526) 175 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:04.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:03 smithi083 bash[18629]: cluster 2024-09-10T18:54:02.549731+0000 mgr.y (mgr.24526) 175 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:04.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:04.149 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:04.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:04.279 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:04.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:04.408 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:04.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:04.547 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:04.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:04.683 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:04.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:04.824 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:04.928 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:04.958 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:04.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:05.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:05.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:05.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:05.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:05.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:05.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:05.365 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:05.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:05.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:05.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:05.742 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:05.767 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:05.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:05.900 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:05.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:06.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.015 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:05 smithi083 bash[18629]: cluster 2024-09-10T18:54:04.550130+0000 mgr.y (mgr.24526) 176 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:06.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:05 smithi083 bash[32381]: cluster 2024-09-10T18:54:04.550130+0000 mgr.y (mgr.24526) 176 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:06.028 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:06.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:06.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:06.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:06.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:05 smithi165 bash[22591]: cluster 2024-09-10T18:54:04.550130+0000 mgr.y (mgr.24526) 176 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:06.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:06.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.442 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:06.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:06.572 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:06.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.700 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:06.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:06.836 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:06.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:06.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:06.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:07.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:07.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:07.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:07.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:07.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:07.382 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:07.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:07.516 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:07.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:07.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:07.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:07.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:07.840 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:54:07.840 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:54:07.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:07.905 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:07.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:07.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:08.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.067 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:08.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:08.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:07 smithi165 bash[22591]: cluster 2024-09-10T18:54:06.550885+0000 mgr.y (mgr.24526) 177 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:08.207 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:08.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:07 smithi083 bash[18629]: cluster 2024-09-10T18:54:06.550885+0000 mgr.y (mgr.24526) 177 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:08.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:07 smithi083 bash[32381]: cluster 2024-09-10T18:54:06.550885+0000 mgr.y (mgr.24526) 177 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:08.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:08.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:08.472 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:08.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:08.598 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:08.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.737 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:08.840 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:08.868 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:08.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:08.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.006 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:09.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:09.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:09.121 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:09 smithi083 bash[32381]: audit 2024-09-10T18:54:07.614092+0000 mgr.y (mgr.24526) 178 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:09.121 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:08 smithi083 bash[18629]: audit 2024-09-10T18:54:07.614092+0000 mgr.y (mgr.24526) 178 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:09.132 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:09.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:09.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:09.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:09.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:09.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:08 smithi165 bash[22591]: audit 2024-09-10T18:54:07.614092+0000 mgr.y (mgr.24526) 178 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:09.397 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:09.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:09.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:09.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:09.668 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:09.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:09.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:09.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:09.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:09.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:10.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:10.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:09 smithi083 bash[18629]: cluster 2024-09-10T18:54:08.551474+0000 mgr.y (mgr.24526) 179 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:54:10.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:54:10.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:09 smithi083 bash[32381]: cluster 2024-09-10T18:54:08.551474+0000 mgr.y (mgr.24526) 179 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:54:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:10.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:10.197 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:10.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:10.314 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:09 smithi165 bash[22591]: cluster 2024-09-10T18:54:08.551474+0000 mgr.y (mgr.24526) 179 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T18:54:10.333 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:10.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:10.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:10.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:10.604 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:10.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:10.734 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:10.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:10.873 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:10.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:10.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:11.004 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:11.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:11 smithi165 bash[22591]: cluster 2024-09-10T18:54:09.846443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:11 smithi165 bash[22591]: cluster 2024-09-10T18:54:09.846532+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:11 smithi165 bash[22591]: cluster 2024-09-10T18:54:09.986607+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-10T18:54:11.144 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:11.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.282 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:11.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[18629]: cluster 2024-09-10T18:54:09.846443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:11.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[18629]: cluster 2024-09-10T18:54:09.846532+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:11.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[18629]: cluster 2024-09-10T18:54:09.986607+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-10T18:54:11.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[32381]: cluster 2024-09-10T18:54:09.846443+0000 osd.1 (osd.1) 111 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:11.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[32381]: cluster 2024-09-10T18:54:09.846532+0000 osd.1 (osd.1) 112 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:11.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:11 smithi083 bash[32381]: cluster 2024-09-10T18:54:09.986607+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-09-10T18:54:11.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.399 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:11.420 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:11.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:11.559 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:11.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- 172.21.15.83:0/1843912332 >> v1:172.21.15.83:6790/0 conn(0x7f10e410c230 legacy=0x7f10e410e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- 172.21.15.83:0/1843912332 shutdown_connections 2024-09-10T18:54:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- 172.21.15.83:0/1843912332 >> 172.21.15.83:0/1843912332 conn(0x7f10e406b990 msgr2=0x7f10e406bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- 172.21.15.83:0/1843912332 shutdown_connections 2024-09-10T18:54:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- 172.21.15.83:0/1843912332 wait complete. 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 librados: starting msgr at 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 librados: starting objecter 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 Processor -- start 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- start start 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 librados: setting wanted keys 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 librados: calling monclient init 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e412adf0 con 0x7f10e410fd90 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e412bf90 con 0x7f10e4109450 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e9f71640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e412d130 con 0x7f10e410c230 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.664+0000 7f10e37fe640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f10e4109450 0x7f10e4108870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:41132/0 (socket says 172.21.15.83:41132) 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e37fe640 1 -- 172.21.15.83:0/2596403154 learned_addr learned my addr 172.21.15.83:0/2596403154 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter ms_handle_connect 0x7f10e4109450 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter ms_handle_connect 0x7f10e410c230 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter ms_handle_connect 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191510063 0 0) 0x7f10e412d130 con 0x7f10e410c230 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10cc005530 con 0x7f10e410c230 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592941143 0 0) 0x7f10e412adf0 con 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e412d130 con 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696644890 0 0) 0x7f10e412bf90 con 0x7f10e4109450 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e412adf0 con 0x7f10e4109450 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557735234 0 0) 0x7f10e412d130 con 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc006ca0 con 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f10d4001720 con 0x7f10e410fd90 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 427366796 0 0) 0x7f10cc005530 con 0x7f10e410c230 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e412d130 con 0x7f10e410c230 2024-09-10T18:54:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f10dc001890 con 0x7f10e410c230 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893739825 0 0) 0x7f10e412adf0 con 0x7f10e4109450 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10cc0074b0 con 0x7f10e4109450 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1893748991 0 0) 0x7f10cc006ca0 con 0x7f10e410fd90 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 >> v1:172.21.15.83:6790/0 conn(0x7f10e410c230 legacy=0x7f10e4125fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 >> v1:172.21.15.165:6789/0 conn(0x7f10e4109450 legacy=0x7f10e4108870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e412e2d0 con 0x7f10e410fd90 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e412c180 con 0x7f10e410fd90 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e9f71640 10 client.14838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e412c5d0 con 0x7f10e410fd90 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e9f71640 1 librados: init done 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f10d40020d0 con 0x7f10e410fd90 2024-09-10T18:54:11.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f10d4003080 con 0x7f10e410fd90 2024-09-10T18:54:11.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/4091785262 >> v1:172.21.15.83:6789/0 conn(0x7fcfd006e740 legacy=0x7fcfd0116660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/4091785262 shutdown_connections 2024-09-10T18:54:11.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/4091785262 >> 172.21.15.83:0/4091785262 conn(0x7fcfd006b990 msgr2=0x7fcfd006bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:11.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.668+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d0005040 con 0x7f10e410fd90 2024-09-10T18:54:11.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f10d40031e0 con 0x7f10e410fd90 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(118..118 src has 59..118) v4 ==== 6284+0+0 (unknown 1351630741 0 0) 0x7f10d4087330 con 0x7f10e410fd90 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/4091785262 shutdown_connections 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/4091785262 wait complete. 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 librados: starting msgr at 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 librados: starting objecter 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 Processor -- start 2024-09-10T18:54:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 10 client.14838.objecter ms_dispatch 0x7f10e4000b90 osd_map(118..118 src has 59..118) v4 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 3 client.14838.objecter handle_osd_map got epochs [118,118] > 0 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 3 client.14838.objecter handle_osd_map decoding full epoch 118 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7f10cbfff640 20 client.14838.objecter dump_active .. 0 homeless 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- start start 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 librados: setting wanted keys 2024-09-10T18:54:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 librados: calling monclient init 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfd0121110 con 0x7fcfd0117d80 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfd012c230 con 0x7fcfd006e740 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd7aa9640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcfd012d410 con 0x7fcfd006dc60 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd6046640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fcfd0117d80 0x7fcfd0129b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59364/0 (socket says 172.21.15.83:59364) 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.672+0000 7fcfd6046640 1 -- 172.21.15.83:0/480987290 learned_addr learned my addr 172.21.15.83:0/480987290 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter ms_handle_connect 0x7fcfd0117d80 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter ms_handle_connect 0x7fcfd006dc60 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter ms_handle_connect 0x7fcfd006e740 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323508034 0 0) 0x7fcfd012c230 con 0x7fcfd006e740 2024-09-10T18:54:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfbc005580 con 0x7fcfd006e740 2024-09-10T18:54:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486480691 0 0) 0x7fcfd012d410 con 0x7fcfd006dc60 2024-09-10T18:54:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfd012c230 con 0x7fcfd006dc60 2024-09-10T18:54:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7f10cbfff640 10 client.14838.objecter ms_handle_connect 0x7f10cc071dd0 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931101698 0 0) 0x7fcfd012c230 con 0x7fcfd006dc60 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfd012d410 con 0x7fcfd006dc60 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcfcc0016b0 con 0x7fcfd006dc60 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214864919 0 0) 0x7fcfbc005580 con 0x7fcfd006e740 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfd012c230 con 0x7fcfd006e740 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132309544 0 0) 0x7fcfd012d410 con 0x7fcfd006dc60 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 >> v1:172.21.15.165:6789/0 conn(0x7fcfd006e740 legacy=0x7fcfd01263d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 >> v1:172.21.15.83:6789/0 conn(0x7fcfd0117d80 legacy=0x7fcfd0129b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd012e630 con 0x7fcfd006dc60 2024-09-10T18:54:11.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd012d600 con 0x7fcfd006dc60 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfd7aa9640 10 client.24764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd012da50 con 0x7fcfd006dc60 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfd7aa9640 1 librados: init done 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fcfcc001d70 con 0x7fcfd006dc60 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcfcc0030d0 con 0x7fcfd006dc60 2024-09-10T18:54:11.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.676+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fcfcc0194f0 con 0x7fcfd006dc60 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(118..118 src has 59..118) v4 ==== 6284+0+0 (unknown 1351630741 0 0) 0x7fcfcc087360 con 0x7fcfd006dc60 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfc67fc640 10 client.24764.objecter ms_dispatch 0x7fcfd0000b90 osd_map(118..118 src has 59..118) v4 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfc67fc640 3 client.24764.objecter handle_osd_map got epochs [118,118] > 0 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfc67fc640 3 client.24764.objecter handle_osd_map decoding full epoch 118 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfc67fc640 20 client.24764.objecter dump_active .. 0 homeless 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb4005010 con 0x7fcfd006dc60 2024-09-10T18:54:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.680+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f10d4053b20 con 0x7f10e410fd90 2024-09-10T18:54:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.684+0000 7fcfc67fc640 10 client.24764.objecter ms_handle_connect 0x7fcfbc071870 2024-09-10T18:54:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.688+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fcfcc054a70 con 0x7fcfd006dc60 2024-09-10T18:54:11.693 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:11.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.700+0000 7fb9091c6640 1 -- 172.21.15.83:0/525389819 >> v1:172.21.15.83:6789/0 conn(0x7fb904101530 legacy=0x7fb904101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.700+0000 7fb9091c6640 1 -- 172.21.15.83:0/525389819 shutdown_connections 2024-09-10T18:54:11.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.700+0000 7fb9091c6640 1 -- 172.21.15.83:0/525389819 >> 172.21.15.83:0/525389819 conn(0x7fb9040f7070 msgr2=0x7fb9040f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:11.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- 172.21.15.83:0/525389819 shutdown_connections 2024-09-10T18:54:11.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- 172.21.15.83:0/525389819 wait complete. 2024-09-10T18:54:11.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 librados: starting msgr at 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 librados: starting objecter 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 Processor -- start 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- start start 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 librados: setting wanted keys 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 librados: calling monclient init 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb90411c290 con 0x7fb904101530 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb90411d430 con 0x7fb904107e00 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb90411e610 con 0x7fb9041042a0 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb902d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb904101530 0x7fb904107170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59388/0 (socket says 172.21.15.83:59388) 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb90259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb9041042a0 0x7fb904117440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55820/0 (socket says 172.21.15.83:55820) 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb902d9d640 1 -- 172.21.15.83:0/4144328313 learned_addr learned my addr 172.21.15.83:0/4144328313 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb9041042a0 2024-09-10T18:54:11.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb904101530 2024-09-10T18:54:11.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb904107e00 2024-09-10T18:54:11.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393455409 0 0) 0x7fb90411e610 con 0x7fb9041042a0 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ec003120 con 0x7fb9041042a0 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879292848 0 0) 0x7fb90411c290 con 0x7fb904101530 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90411e610 con 0x7fb904101530 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91722626 0 0) 0x7fb90411d430 con 0x7fb904107e00 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90411c290 con 0x7fb904107e00 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 310889835 0 0) 0x7fb90411e610 con 0x7fb904101530 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb90411d430 con 0x7fb904101530 2024-09-10T18:54:11.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb8f4002eb0 con 0x7fb904101530 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534010824 0 0) 0x7fb8ec003120 con 0x7fb9041042a0 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb90411e610 con 0x7fb9041042a0 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb8f80017b0 con 0x7fb9041042a0 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591459921 0 0) 0x7fb90411e610 con 0x7fb9041042a0 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.165:6789/0 conn(0x7fb904107e00 legacy=0x7fb90411ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6789/0 conn(0x7fb904101530 legacy=0x7fb904107170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90411f810 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90411d620 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 10 client.24770.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90411dab0 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.704+0000 7fb9091c6640 1 librados: init done 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb9040fcc30 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb8f8001e30 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb8f8005920 con 0x7fb9041042a0 2024-09-10T18:54:11.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fb8f801bde0 con 0x7fb9041042a0 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(118..118 src has 59..118) v4 ==== 6284+0+0 (unknown 1351630741 0 0) 0x7fb8f808ab70 con 0x7fb9041042a0 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 osd_map(118..118 src has 59..118) v4 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 3 client.24770.objecter handle_osd_map got epochs [118,118] > 0 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 3 client.24770.objecter handle_osd_map decoding full epoch 118 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 20 client.24770.objecter dump_active .. 0 homeless 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb8ec071560 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.708+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v118) v1 ==== 54+0+16 (unknown 27550155 0 2420000192) 0x7fb8f80577e0 con 0x7fb9041042a0 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 10 client.24770.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 1 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 1 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 0 1 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 4 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 10 client.24770.objecter _send_command 1 2024-09-10T18:54:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb90404bf60 con 0x7fb90411c420 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb90411c420 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb90404bf60 con 0x7fb90411c420 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 1 = system:0 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 4 2024-09-10T18:54:11.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.712+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 0 1 2024-09-10T18:54:11.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 10 client.24770.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 2 2024-09-10T18:54:11.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 2 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90410d9a0 osd=0 3 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 0 2 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 4 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 10 client.24770.objecter _send_command 2 2024-09-10T18:54:11.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.736+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb9041068c0 con 0x7fb90411c420 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb9041068c0 con 0x7fb90411c420 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 4 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 0 2 2024-09-10T18:54:11.744 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:54:11.745 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:54:11.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 10 client.24770.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 3 2024-09-10T18:54:11.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 1 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 4 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 10 client.24770.objecter _send_command 3 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb90402a8d0 con 0x7fb904109780 2024-09-10T18:54:11.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.740+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb904109780 2024-09-10T18:54:11.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.744+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb90402a8d0 con 0x7fb904109780 2024-09-10T18:54:11.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.744+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:54:11.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.744+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 3 = system:0 2024-09-10T18:54:11.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.744+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 4 2024-09-10T18:54:11.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.744+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 1 3 2024-09-10T18:54:11.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 10 client.24770.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 4 2024-09-10T18:54:11.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 4 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90411d180 osd=1 3 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 1 4 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 4 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 10 client.24770.objecter _send_command 4 2024-09-10T18:54:11.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.768+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb90402acb0 con 0x7fb904109780 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb90402acb0 con 0x7fb904109780 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 4 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 1 4 2024-09-10T18:54:11.774 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 10 client.24770.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 5 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 5 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 2 5 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 4 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 10 client.24770.objecter _send_command 5 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb90405e760 con 0x7fb90405e310 2024-09-10T18:54:11.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb90405e310 2024-09-10T18:54:11.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb90405e760 con 0x7fb90405e310 2024-09-10T18:54:11.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:54:11.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 5 = system:0 2024-09-10T18:54:11.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 4 2024-09-10T18:54:11.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.772+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 2 5 2024-09-10T18:54:11.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 10 client.24770.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 6 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 6 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90402ae20 osd=2 3 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 2 6 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 4 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 10 client.24770.objecter _send_command 6 2024-09-10T18:54:11.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.796+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb90405eb10 con 0x7fb90405e310 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb90405eb10 con 0x7fb90405e310 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 4 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 2 6 2024-09-10T18:54:11.804 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 10 client.24770.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 7 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 7 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 3 7 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 4 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 10 client.24770.objecter _send_command 7 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9041226b0 con 0x7fb90405df00 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.800+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb90405df00 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.804+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb9041226b0 con 0x7fb90405df00 2024-09-10T18:54:11.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.804+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:54:11.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.804+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 7 = system:0 2024-09-10T18:54:11.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.804+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 4 2024-09-10T18:54:11.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.804+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 3 7 2024-09-10T18:54:11.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 10 client.24770.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 8 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 8 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb90405dd90 osd=3 3 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 3 8 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 4 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 10 client.24770.objecter _send_command 8 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.828+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb904122960 con 0x7fb90405df00 2024-09-10T18:54:11.832 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb904122960 con 0x7fb90405df00 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 4 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 3 8 2024-09-10T18:54:11.835 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 10 client.24770.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 9 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 9 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 4 9 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 4 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 10 client.24770.objecter _send_command 9 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb904126fa0 con 0x7fb904126a20 2024-09-10T18:54:11.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb904126a20 2024-09-10T18:54:11.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb904126fa0 con 0x7fb904126a20 2024-09-10T18:54:11.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:54:11.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 9 = system:0 2024-09-10T18:54:11.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 4 2024-09-10T18:54:11.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.832+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 4 9 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 10 client.24770.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 10 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 10 2024-09-10T18:54:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904122960 osd=4 3 2024-09-10T18:54:11.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 4 10 2024-09-10T18:54:11.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 4 2024-09-10T18:54:11.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 10 client.24770.objecter _send_command 10 2024-09-10T18:54:11.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb904126fa0 con 0x7fb904126a20 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb904126fa0 con 0x7fb904126a20 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 4 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.860+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 4 10 2024-09-10T18:54:11.866 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:54:11.867 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:54:11.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 10 client.24770.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 11 2024-09-10T18:54:11.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 11 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 5 11 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 4 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 10 client.24770.objecter _send_command 11 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb904133400 con 0x7fb90412f8d0 2024-09-10T18:54:11.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb90412f8d0 2024-09-10T18:54:11.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb904133400 con 0x7fb90412f8d0 2024-09-10T18:54:11.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:54:11.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 11 = system:0 2024-09-10T18:54:11.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 4 2024-09-10T18:54:11.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.864+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 5 11 2024-09-10T18:54:11.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.872+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f10d00052f0 con 0x7f10e410fd90 2024-09-10T18:54:11.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 10 client.24770.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 12 2024-09-10T18:54:11.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 12 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904126fa0 osd=5 3 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 5 12 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 4 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 10 client.24770.objecter _send_command 12 2024-09-10T18:54:11.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.892+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb904134ca0 con 0x7fb90412f8d0 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb904134ca0 con 0x7fb90412f8d0 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 4 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 5 12 2024-09-10T18:54:11.898 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 10 client.24770.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 13 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 13 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 6 13 2024-09-10T18:54:11.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 4 2024-09-10T18:54:11.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 10 client.24770.objecter _send_command 13 2024-09-10T18:54:11.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb904135830 con 0x7fb904135320 2024-09-10T18:54:11.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb904135320 2024-09-10T18:54:11.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb904135830 con 0x7fb904135320 2024-09-10T18:54:11.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:54:11.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 13 = system:0 2024-09-10T18:54:11.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 4 2024-09-10T18:54:11.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.896+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 6 13 2024-09-10T18:54:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 10 client.24770.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 14 2024-09-10T18:54:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 14 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904134ca0 osd=6 3 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 6 14 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 4 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 10 client.24770.objecter _send_command 14 2024-09-10T18:54:11.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.924+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb904135830 con 0x7fb904135320 2024-09-10T18:54:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb904135830 con 0x7fb904135320 2024-09-10T18:54:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 4 2024-09-10T18:54:11.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 6 14 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 10 client.24770.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 15 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 15 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 7 15 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 4 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 10 client.24770.objecter _send_command 15 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb90413bbf0 con 0x7fb9041359d0 2024-09-10T18:54:11.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 10 client.24770.objecter ms_handle_connect 0x7fb9041359d0 2024-09-10T18:54:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb90413bbf0 con 0x7fb9041359d0 2024-09-10T18:54:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:54:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 15 = system:0 2024-09-10T18:54:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 4 2024-09-10T18:54:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.928+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 7 15 2024-09-10T18:54:11.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 10 client.24770.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign -1 16 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 20 client.24770.objecter _get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_remove -1 16 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 20 client.24770.objecter get_session s=0x7fb904135830 osd=7 3 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 15 client.24770.objecter _session_command_op_assign 7 16 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 4 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 10 client.24770.objecter _send_command 16 2024-09-10T18:54:11.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.952+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb90413d490 con 0x7fb9041359d0 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb8eb7fe640 1 -- 172.21.15.83:0/4144328313 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fb90413d490 con 0x7fb9041359d0 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb8eb7fe640 10 client.24770.objecter ms_dispatch 0x7fb904000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb8eb7fe640 10 client.24770.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb8eb7fe640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 4 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb8eb7fe640 15 client.24770.objecter _session_command_op_remove 7 16 2024-09-10T18:54:11.961 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:54:11.962 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:54:11.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 librados: watch_flush enter 2024-09-10T18:54:11.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 librados: watch_flush exit 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.0 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6801/2535527483 conn(0x7fb90411c420 legacy=0x7fb90411c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90410d9a0 osd=0 1 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.1 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6805/2309624637 conn(0x7fb904109780 legacy=0x7fb904109bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90411d180 osd=1 1 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.2 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6809/386094220 conn(0x7fb90405e310 legacy=0x7fb904124b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90402ae20 osd=2 1 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.3 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6813/1279688021 conn(0x7fb90405df00 legacy=0x7fb9041293f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb90405dd90 osd=3 1 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.4 2024-09-10T18:54:11.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb90359e640 1 -- 172.21.15.83:0/4144328313 reap_dead start 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.165:6800/3225515265 conn(0x7fb904126a20 legacy=0x7fb90412dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb90359e640 1 -- 172.21.15.83:0/4144328313 reap_dead start 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904122960 osd=4 1 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.5 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.165:6804/571173777 conn(0x7fb90412f8d0 legacy=0x7fb904131cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904126fa0 osd=5 1 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.6 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.165:6808/1317922936 conn(0x7fb904135320 legacy=0x7fb904133400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904134ca0 osd=6 1 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter close_session for osd.7 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.165:6812/1610151650 conn(0x7fb9041359d0 legacy=0x7fb90413a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter put_session s=0x7fb904135830 osd=7 1 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 20 client.24770.objecter shutdown clearing up homeless session... 2024-09-10T18:54:11.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.956+0000 7fb9091c6640 10 client.24770.objecter successfully canceled tick 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6800/3071793735 conn(0x7fb8ec071560 legacy=0x7fb8ec073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> v1:172.21.15.83:6790/0 conn(0x7fb9041042a0 legacy=0x7fb904117440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb90359e640 1 -- 172.21.15.83:0/4144328313 reap_dead start 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 shutdown_connections 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 >> 172.21.15.83:0/4144328313 conn(0x7fb9040f7070 msgr2=0x7fb90410aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 shutdown_connections 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 -- 172.21.15.83:0/4144328313 wait complete. 2024-09-10T18:54:11.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.960+0000 7fb9091c6640 1 librados: shutdown 2024-09-10T18:54:11.967 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:11.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:11.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:11.992+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fcfb40052c0 con 0x7fcfd006dc60 2024-09-10T18:54:12.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.004+0000 7f10cbfff640 1 -- 172.21.15.83:0/2596403154 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v119) v1 ==== 113+0+0 (unknown 1823060822 0 0) 0x7f10d4053940 con 0x7f10e410fd90 2024-09-10T18:54:12.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.004+0000 7fcfc67fc640 1 -- 172.21.15.83:0/480987290 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v119) v1 ==== 97+0+0 (unknown 1344235883 0 0) 0x7fcfcc054890 con 0x7fcfd006dc60 2024-09-10T18:54:12.011 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T18:54:12.011 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T18:54:12.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 10 librados: watch_flush enter 2024-09-10T18:54:12.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 10 librados: watch_flush exit 2024-09-10T18:54:12.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 20 client.14838.objecter shutdown clearing up homeless session... 2024-09-10T18:54:12.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 10 client.14838.objecter successfully canceled tick 2024-09-10T18:54:12.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 >> v1:172.21.15.83:6800/3071793735 conn(0x7f10cc071dd0 legacy=0x7f10cc074280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:12.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 >> v1:172.21.15.83:6789/0 conn(0x7f10e410fd90 legacy=0x7f10e41296d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:12.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 shutdown_connections 2024-09-10T18:54:12.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.008+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 >> 172.21.15.83:0/2596403154 conn(0x7f10e406b990 msgr2=0x7f10e406e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:12.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 10 librados: watch_flush enter 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 10 librados: watch_flush exit 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 20 client.24764.objecter shutdown clearing up homeless session... 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 10 client.24764.objecter successfully canceled tick 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 >> v1:172.21.15.83:6800/3071793735 conn(0x7fcfbc071870 legacy=0x7fcfbc073d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 >> v1:172.21.15.83:6790/0 conn(0x7fcfd006dc60 legacy=0x7fcfd01205a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 shutdown_connections 2024-09-10T18:54:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 >> 172.21.15.83:0/480987290 conn(0x7fcfd006b990 msgr2=0x7fcfd01181b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:12.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 shutdown_connections 2024-09-10T18:54:12.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 -- 172.21.15.83:0/480987290 wait complete. 2024-09-10T18:54:12.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7fcfd7aa9640 1 librados: shutdown 2024-09-10T18:54:12.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 shutdown_connections 2024-09-10T18:54:12.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7f10e9f71640 1 -- 172.21.15.83:0/2596403154 wait complete. 2024-09-10T18:54:12.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:12.012+0000 7f10e9f71640 1 librados: shutdown 2024-09-10T18:54:12.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:12.091 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:12.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:12.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:12 smithi165 bash[22591]: cluster 2024-09-10T18:54:10.552844+0000 mgr.y (mgr.24526) 180 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:12.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:12 smithi165 bash[22591]: audit 2024-09-10T18:54:11.711981+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.83:0/4144328313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:12.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:12 smithi165 bash[22591]: audit 2024-09-10T18:54:11.874797+0000 mon.a (mon.0) 884 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:54:12.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:12 smithi165 bash[22591]: audit 2024-09-10T18:54:11.997161+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.83:0/480987290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:12 smithi165 bash[22591]: audit 2024-09-10T18:54:11.997735+0000 mon.a (mon.0) 885 : audit [INF] from='client.24764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:12.259 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18629]: cluster 2024-09-10T18:54:10.552844+0000 mgr.y (mgr.24526) 180 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18629]: audit 2024-09-10T18:54:11.711981+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.83:0/4144328313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18629]: audit 2024-09-10T18:54:11.874797+0000 mon.a (mon.0) 884 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18629]: audit 2024-09-10T18:54:11.997161+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.83:0/480987290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18629]: audit 2024-09-10T18:54:11.997735+0000 mon.a (mon.0) 885 : audit [INF] from='client.24764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[32381]: cluster 2024-09-10T18:54:10.552844+0000 mgr.y (mgr.24526) 180 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[32381]: audit 2024-09-10T18:54:11.711981+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.83:0/4144328313' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[32381]: audit 2024-09-10T18:54:11.874797+0000 mon.a (mon.0) 884 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T18:54:12.260 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[32381]: audit 2024-09-10T18:54:11.997161+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.83:0/480987290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.261 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[32381]: audit 2024-09-10T18:54:11.997735+0000 mon.a (mon.0) 885 : audit [INF] from='client.24764 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:54:12.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:12.365 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:12.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:12.502 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:12.606 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:12.635 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:12.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:12.772 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:12.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:12.913 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:12.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:13.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:13.017 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:12] "GET /metrics HTTP/1.1" 200 220223 "" "Prometheus/2.43.0" 2024-09-10T18:54:13.047 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:13.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:13.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:13.183 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:13.287 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[18629]: cluster 2024-09-10T18:54:12.003082+0000 mon.a (mon.0) 886 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[18629]: audit 2024-09-10T18:54:12.007561+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:54:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[18629]: audit 2024-09-10T18:54:12.007719+0000 mon.a (mon.0) 888 : audit [INF] from='client.24764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:54:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[18629]: cluster 2024-09-10T18:54:12.007765+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-10T18:54:13.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[32381]: cluster 2024-09-10T18:54:12.003082+0000 mon.a (mon.0) 886 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[32381]: audit 2024-09-10T18:54:12.007561+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:54:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[32381]: audit 2024-09-10T18:54:12.007719+0000 mon.a (mon.0) 888 : audit [INF] from='client.24764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:54:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:13 smithi083 bash[32381]: cluster 2024-09-10T18:54:12.007765+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-10T18:54:13.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:13 smithi165 bash[22591]: cluster 2024-09-10T18:54:12.003082+0000 mon.a (mon.0) 886 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:13 smithi165 bash[22591]: audit 2024-09-10T18:54:12.007561+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.83:0/2596403154' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T18:54:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:13 smithi165 bash[22591]: audit 2024-09-10T18:54:12.007719+0000 mon.a (mon.0) 888 : audit [INF] from='client.24764 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:54:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:13 smithi165 bash[22591]: cluster 2024-09-10T18:54:12.007765+0000 mon.a (mon.0) 889 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-09-10T18:54:13.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:13.442 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:13.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:13.569 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:13.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:13.701 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T18:54:13.704 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:13.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:13.842 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:13.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:13.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:13.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:14.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:14.088 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:14 smithi165 bash[22591]: cluster 2024-09-10T18:54:12.553544+0000 mgr.y (mgr.24526) 181 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:14.110 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:14.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:14.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:14 smithi083 bash[18629]: cluster 2024-09-10T18:54:12.553544+0000 mgr.y (mgr.24526) 181 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:14.222 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:14 smithi083 bash[32381]: cluster 2024-09-10T18:54:12.553544+0000 mgr.y (mgr.24526) 181 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:14.235 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:14.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:14.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:14.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:14.506 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:14.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:14.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:14.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:14.901 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:14.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:15.005 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:15.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.156 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:15.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.270 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:15 smithi083 bash[18629]: audit 2024-09-10T18:54:14.554135+0000 mon.c (mon.2) 98 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:15.270 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:15 smithi083 bash[32381]: audit 2024-09-10T18:54:14.554135+0000 mon.c (mon.2) 98 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:15.282 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:15.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:15 smithi165 bash[22591]: audit 2024-09-10T18:54:14.554135+0000 mon.c (mon.2) 98 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:15.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.408 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:15.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:15.535 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:15.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:15.661 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:15.704 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:54:15.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:54:15.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:15.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:15.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:15.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:16.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:16 smithi083 bash[18629]: cluster 2024-09-10T18:54:14.554332+0000 mgr.y (mgr.24526) 182 : cluster [DBG] pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:16.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:16 smithi083 bash[32381]: cluster 2024-09-10T18:54:14.554332+0000 mgr.y (mgr.24526) 182 : cluster [DBG] pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:16.162 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.188 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:16.205 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:16.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:16.311 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:16.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:16 smithi165 bash[22591]: cluster 2024-09-10T18:54:14.554332+0000 mgr.y (mgr.24526) 182 : cluster [DBG] pgmap v150: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:16.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:16.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:16.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.582 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:16.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:16.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:16.721 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:16.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:16.855 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:16.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:16.992 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:16.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:17.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:17.133 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:17.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:17.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:17.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:17.395 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:17.498 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:17.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:17.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:17.668 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:17.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:17.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:17.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:18.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:18.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:18.169 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:18.200 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:18.237 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:18.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:18.314 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:18 smithi083 bash[18629]: cluster 2024-09-10T18:54:16.556250+0000 mgr.y (mgr.24526) 183 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:18.314 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:18 smithi083 bash[32381]: cluster 2024-09-10T18:54:16.556250+0000 mgr.y (mgr.24526) 183 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:18.336 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:18.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:18 smithi165 bash[22591]: cluster 2024-09-10T18:54:16.556250+0000 mgr.y (mgr.24526) 183 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:18.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:18.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:18.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:18.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:18.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:18.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:18.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:18.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:18.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:19.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:19.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:19.160 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:19.264 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.267 INFO:tasks.workunit.client.0.smithi083.stderr:Note: switching to 'fad5356a94e6044ce1062d748daa5a0f2ed53a7d'. 2024-09-10T18:54:19.267 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: git switch -c 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:Or undo this operation with: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: git switch - 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.268 INFO:tasks.workunit.client.0.smithi083.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T18:54:19.269 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-09-10T18:54:19.269 INFO:tasks.workunit.client.0.smithi083.stderr:HEAD is now at fad5356a94e qa/upgrade: change pacific/octopus upgrade tests to use published image 2024-09-10T18:54:19.278 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T18:54:19.291 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:19.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:19 smithi165 bash[22591]: audit 2024-09-10T18:54:17.625409+0000 mgr.y (mgr.24526) 184 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:19.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:19 smithi083 bash[18629]: audit 2024-09-10T18:54:17.625409+0000 mgr.y (mgr.24526) 184 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:19.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:19 smithi083 bash[32381]: audit 2024-09-10T18:54:17.625409+0000 mgr.y (mgr.24526) 184 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:19.428 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:19.436 INFO:tasks.workunit.client.0.smithi083.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T18:54:19.439 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T18:54:19.439 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T18:54:19.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.566 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:19.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:19.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:19.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:19.845 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:19.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:19.983 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:19.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:20.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:20 smithi165 bash[22591]: cluster 2024-09-10T18:54:18.556956+0000 mgr.y (mgr.24526) 185 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:20 smithi165 bash[22591]: cluster 2024-09-10T18:54:19.491193+0000 osd.2 (osd.2) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:20.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:20 smithi165 bash[22591]: cluster 2024-09-10T18:54:19.491305+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:20.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:20.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:20.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:20.252 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[18629]: cluster 2024-09-10T18:54:18.556956+0000 mgr.y (mgr.24526) 185 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[18629]: cluster 2024-09-10T18:54:19.491193+0000 osd.2 (osd.2) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:20.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[18629]: cluster 2024-09-10T18:54:19.491305+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[32381]: cluster 2024-09-10T18:54:18.556956+0000 mgr.y (mgr.24526) 185 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[32381]: cluster 2024-09-10T18:54:19.491193+0000 osd.2 (osd.2) 59 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:20 smithi083 bash[32381]: cluster 2024-09-10T18:54:19.491305+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:20.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:20.391 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:20.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:20.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:20.528 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:20.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:20.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:20.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:20.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:20.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:21.043 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.078 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:21.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:21.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:21.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:21 smithi083 bash[18629]: cluster 2024-09-10T18:54:20.072477+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-10T18:54:21.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:21 smithi083 bash[32381]: cluster 2024-09-10T18:54:20.072477+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-10T18:54:21.320 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:21.357 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:21 smithi165 bash[22591]: cluster 2024-09-10T18:54:20.072477+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-09-10T18:54:21.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.492 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:21.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:21.634 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:21.636 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:21.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:21.771 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:21.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.972 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:21.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:22.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:22.107 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:22.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:22.246 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:22.350 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:22 smithi165 bash[22591]: cluster 2024-09-10T18:54:20.558336+0000 mgr.y (mgr.24526) 186 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:22.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:22 smithi083 bash[18629]: cluster 2024-09-10T18:54:20.558336+0000 mgr.y (mgr.24526) 186 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:22.364 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:22 smithi083 bash[32381]: cluster 2024-09-10T18:54:20.558336+0000 mgr.y (mgr.24526) 186 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:22.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:22.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:22.524 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:22.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:22.660 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:22.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:22.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:22.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:22.936 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:22.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:23.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:22] "GET /metrics HTTP/1.1" 200 220184 "" "Prometheus/2.43.0" 2024-09-10T18:54:23.068 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:23.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.206 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:23.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:23.343 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:23 smithi165 bash[22591]: cluster 2024-09-10T18:54:21.598268+0000 osd.4 (osd.4) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:23 smithi165 bash[22591]: cluster 2024-09-10T18:54:21.598387+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:23.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:23 smithi165 bash[22591]: cluster 2024-09-10T18:54:22.084657+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-10T18:54:23.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:23.476 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[18629]: cluster 2024-09-10T18:54:21.598268+0000 osd.4 (osd.4) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:23.476 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[18629]: cluster 2024-09-10T18:54:21.598387+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:23.476 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[18629]: cluster 2024-09-10T18:54:22.084657+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-10T18:54:23.476 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[32381]: cluster 2024-09-10T18:54:21.598268+0000 osd.4 (osd.4) 129 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:23.477 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[32381]: cluster 2024-09-10T18:54:21.598387+0000 osd.4 (osd.4) 130 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:23.477 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:23 smithi083 bash[32381]: cluster 2024-09-10T18:54:22.084657+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-09-10T18:54:23.483 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:23.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.621 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:23.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:23.759 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:23.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:23.897 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:23.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:24.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:24.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:24.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:24 smithi083 bash[18629]: cluster 2024-09-10T18:54:22.559054+0000 mgr.y (mgr.24526) 187 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:24.149 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:24 smithi083 bash[32381]: cluster 2024-09-10T18:54:22.559054+0000 mgr.y (mgr.24526) 187 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:24.170 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:24.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:24.288 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:24 smithi165 bash[22591]: cluster 2024-09-10T18:54:22.559054+0000 mgr.y (mgr.24526) 187 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:24.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:24.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:24.450 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:24.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.587 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:24.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:24.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:24.736 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T18:54:24.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:24.861 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:24.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.965 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:24.966 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:24.991 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:24.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:25.094 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:25.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:25.231 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:25.263 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:25.323 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T18:54:25.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:25.400 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:25.453 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T18:54:25.456 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T18:54:25.456 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T18:54:25.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:25.537 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:25.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:25.656 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T18:54:25.660 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-10T18:54:25.660 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T18:54:25.665 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-10T18:54:25.666 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-10T18:54:25.666 DEBUG:teuthology.orchestra.run.smithi083:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fad5356a94e6044ce1062d748daa5a0f2ed53a7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-10T18:54:25.677 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:+ parallel=1 2024-09-10T18:54:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '' = --serial ']' 2024-09-10T18:54:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:+ color= 2024-09-10T18:54:25.681 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -t 1 ']' 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:+ declare -A pids 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_aio 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_aio' 2024-09-10T18:54:25.682 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_aio 2024-09-10T18:54:25.683 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_aio 2024-09-10T18:54:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77635 2024-09-10T18:54:25.685 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_aio on pid 77635' 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stdout:test api_aio on pid 77635 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77635 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_aio_pp 2024-09-10T18:54:25.686 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_aio_pp' 2024-09-10T18:54:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.687 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_aio_pp 2024-09-10T18:54:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.688 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_aio 2024-09-10T18:54:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-10T18:54:25.689 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_aio: /' 2024-09-10T18:54:25.690 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_aio_pp 2024-09-10T18:54:25.690 INFO:tasks.workunit.client.0.smithi083.stdout:test api_aio_pp on pid 77643 2024-09-10T18:54:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77643 2024-09-10T18:54:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_aio_pp on pid 77643' 2024-09-10T18:54:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77643 2024-09-10T18:54:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.691 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-10T18:54:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_io 2024-09-10T18:54:25.692 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_io' 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_io 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.693 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_aio_pp 2024-09-10T18:54:25.694 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-10T18:54:25.694 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-10T18:54:25.694 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_io 2024-09-10T18:54:25.694 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77651 2024-09-10T18:54:25.694 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_io on pid 77651' 2024-09-10T18:54:25.695 INFO:tasks.workunit.client.0.smithi083.stdout:test api_io on pid 77651 2024-09-10T18:54:25.695 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77651 2024-09-10T18:54:25.695 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.695 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.696 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-10T18:54:25.696 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_io_pp 2024-09-10T18:54:25.696 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_io_pp' 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_io_pp 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_io 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_io.log 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.697 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_io: /' 2024-09-10T18:54:25.700 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_io_pp 2024-09-10T18:54:25.701 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77659 2024-09-10T18:54:25.701 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_io_pp on pid 77659' 2024-09-10T18:54:25.701 INFO:tasks.workunit.client.0.smithi083.stdout:test api_io_pp on pid 77659 2024-09-10T18:54:25.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77659 2024-09-10T18:54:25.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-10T18:54:25.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_asio 2024-09-10T18:54:25.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_asio' 2024-09-10T18:54:25.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_asio 2024-09-10T18:54:25.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_io_pp 2024-09-10T18:54:25.705 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-10T18:54:25.705 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-10T18:54:25.705 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_asio 2024-09-10T18:54:25.706 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77667 2024-09-10T18:54:25.706 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_asio on pid 77667' 2024-09-10T18:54:25.706 INFO:tasks.workunit.client.0.smithi083.stdout:test api_asio on pid 77667 2024-09-10T18:54:25.706 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77667 2024-09-10T18:54:25.707 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.707 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.707 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-10T18:54:25.707 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_list 2024-09-10T18:54:25.707 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_list' 2024-09-10T18:54:25.708 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_list 2024-09-10T18:54:25.708 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_asio: /' 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_asio 2024-09-10T18:54:25.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-10T18:54:25.711 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_list 2024-09-10T18:54:25.711 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77675 2024-09-10T18:54:25.711 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_list on pid 77675' 2024-09-10T18:54:25.711 INFO:tasks.workunit.client.0.smithi083.stdout:test api_list on pid 77675 2024-09-10T18:54:25.712 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77675 2024-09-10T18:54:25.712 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_lock 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_lock' 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_lock 2024-09-10T18:54:25.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_list 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_list.log 2024-09-10T18:54:25.714 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_list: /' 2024-09-10T18:54:25.717 INFO:tasks.workunit.client.0.smithi083.stdout:test api_lock on pid 77683 2024-09-10T18:54:25.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_lock 2024-09-10T18:54:25.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77683 2024-09-10T18:54:25.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_lock on pid 77683' 2024-09-10T18:54:25.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77683 2024-09-10T18:54:25.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-10T18:54:25.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_lock_pp 2024-09-10T18:54:25.720 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_lock_pp' 2024-09-10T18:54:25.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_lock_pp 2024-09-10T18:54:25.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.721 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.723 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.723 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.723 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.724 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_lock 2024-09-10T18:54:25.724 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_lock_pp 2024-09-10T18:54:25.724 INFO:tasks.workunit.client.0.smithi083.stdout:test api_lock_pp on pid 77698 2024-09-10T18:54:25.725 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77698 2024-09-10T18:54:25.725 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_lock_pp on pid 77698' 2024-09-10T18:54:25.725 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77698 2024-09-10T18:54:25.725 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.725 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.726 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-10T18:54:25.726 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_lock: /' 2024-09-10T18:54:25.730 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-10T18:54:25.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_misc 2024-09-10T18:54:25.731 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_misc' 2024-09-10T18:54:25.733 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.733 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.733 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.733 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.734 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_lock_pp 2024-09-10T18:54:25.734 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-10T18:54:25.734 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-10T18:54:25.734 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.735 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_misc 2024-09-10T18:54:25.738 INFO:tasks.workunit.client.0.smithi083.stdout:test api_misc on pid 77736 2024-09-10T18:54:25.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_misc 2024-09-10T18:54:25.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77736 2024-09-10T18:54:25.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_misc on pid 77736' 2024-09-10T18:54:25.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77736 2024-09-10T18:54:25.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.742 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-10T18:54:25.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_misc_pp 2024-09-10T18:54:25.742 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_misc_pp' 2024-09-10T18:54:25.748 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.748 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.748 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.749 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.749 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_misc 2024-09-10T18:54:25.749 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-10T18:54:25.749 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_misc: /' 2024-09-10T18:54:25.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_misc_pp 2024-09-10T18:54:25.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stdout:test api_misc_pp on pid 77781 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_misc_pp 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77781 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_misc_pp on pid 77781' 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77781 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.761 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.762 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-10T18:54:25.764 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.764 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.764 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.764 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.765 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_tier_pp 2024-09-10T18:54:25.765 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_tier_pp' 2024-09-10T18:54:25.765 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_misc_pp 2024-09-10T18:54:25.765 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-10T18:54:25.766 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-10T18:54:25.768 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_tier_pp 2024-09-10T18:54:25.769 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.773 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_tier_pp 2024-09-10T18:54:25.773 INFO:tasks.workunit.client.0.smithi083.stdout:test api_tier_pp on pid 77823 2024-09-10T18:54:25.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77823 2024-09-10T18:54:25.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_tier_pp on pid 77823' 2024-09-10T18:54:25.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77823 2024-09-10T18:54:25.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.775 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-10T18:54:25.777 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.777 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.777 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.777 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.778 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_tier_pp 2024-09-10T18:54:25.778 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-10T18:54:25.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:25.782 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_pool 2024-09-10T18:54:25.783 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_pool' 2024-09-10T18:54:25.790 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-10T18:54:25.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_pool 2024-09-10T18:54:25.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.797 INFO:tasks.workunit.client.0.smithi083.stdout:test api_pool on pid 77860 2024-09-10T18:54:25.797 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_pool 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77860 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_pool on pid 77860' 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77860 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-10T18:54:25.798 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_snapshots 2024-09-10T18:54:25.802 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_snapshots' 2024-09-10T18:54:25.802 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.803 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.803 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.803 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.803 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_pool: /' 2024-09-10T18:54:25.803 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-10T18:54:25.804 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_pool 2024-09-10T18:54:25.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_snapshots 2024-09-10T18:54:25.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:25.818 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_snapshots 2024-09-10T18:54:25.818 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77893 2024-09-10T18:54:25.818 INFO:tasks.workunit.client.0.smithi083.stdout:test api_snapshots on pid 77893 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_snapshots on pid 77893' 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77893 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_snapshots_pp 2024-09-10T18:54:25.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_snapshots_pp' 2024-09-10T18:54:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:25.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:25.822 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-10T18:54:25.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-10T18:54:25.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-10T18:54:25.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_snapshots 2024-09-10T18:54:25.838 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_snapshots_pp 2024-09-10T18:54:25.838 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_snapshots_pp 2024-09-10T18:54:25.839 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77933 2024-09-10T18:54:25.839 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_snapshots_pp on pid 77933' 2024-09-10T18:54:25.839 INFO:tasks.workunit.client.0.smithi083.stdout:test api_snapshots_pp on pid 77933 2024-09-10T18:54:25.839 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77933 2024-09-10T18:54:25.840 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.840 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.842 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-10T18:54:25.846 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_stat 2024-09-10T18:54:25.846 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_stat' 2024-09-10T18:54:25.846 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.847 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.847 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.847 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.847 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-10T18:54:25.848 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-10T18:54:25.848 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-10T18:54:25.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_stat 2024-09-10T18:54:25.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_stat 2024-09-10T18:54:25.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77949 2024-09-10T18:54:25.850 INFO:tasks.workunit.client.0.smithi083.stdout:test api_stat on pid 77949 2024-09-10T18:54:25.850 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_stat on pid 77949' 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77949 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_stat_pp 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_stat_pp' 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-10T18:54:25.851 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_stat_pp 2024-09-10T18:54:25.855 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.855 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.855 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.855 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.856 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_stat 2024-09-10T18:54:25.856 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-10T18:54:25.856 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_stat: /' 2024-09-10T18:54:25.858 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.859 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_stat_pp 2024-09-10T18:54:25.860 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77971 2024-09-10T18:54:25.860 INFO:tasks.workunit.client.0.smithi083.stdout:test api_stat_pp on pid 77971 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_stat_pp on pid 77971' 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77971 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_watch_notify 2024-09-10T18:54:25.861 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_watch_notify' 2024-09-10T18:54:25.862 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.862 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_watch_notify 2024-09-10T18:54:25.868 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_watch_notify 2024-09-10T18:54:25.868 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77984 2024-09-10T18:54:25.868 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_watch_notify on pid 77984' 2024-09-10T18:54:25.868 INFO:tasks.workunit.client.0.smithi083.stdout:test api_watch_notify on pid 77984 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77984 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_stat_pp 2024-09-10T18:54:25.870 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-10T18:54:25.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_watch_notify_pp 2024-09-10T18:54:25.871 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_watch_notify_pp' 2024-09-10T18:54:25.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_watch_notify_pp 2024-09-10T18:54:25.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_watch_notify 2024-09-10T18:54:25.873 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-10T18:54:25.873 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-10T18:54:25.873 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_watch_notify_pp 2024-09-10T18:54:25.873 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77999 2024-09-10T18:54:25.873 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_watch_notify_pp on pid 77999' 2024-09-10T18:54:25.874 INFO:tasks.workunit.client.0.smithi083.stdout:test api_watch_notify_pp on pid 77999 2024-09-10T18:54:25.874 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=77999 2024-09-10T18:54:25.874 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.874 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.875 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-10T18:54:25.876 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.876 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.876 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.876 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.876 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-10T18:54:25.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-10T18:54:25.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cmd 2024-09-10T18:54:25.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-10T18:54:25.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cmd' 2024-09-10T18:54:25.879 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.885 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cmd 2024-09-10T18:54:25.887 INFO:tasks.workunit.client.0.smithi083.stdout:test api_cmd on pid 78018 2024-09-10T18:54:25.887 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cmd 2024-09-10T18:54:25.887 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78018 2024-09-10T18:54:25.887 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cmd on pid 78018' 2024-09-10T18:54:25.888 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78018 2024-09-10T18:54:25.888 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.888 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.888 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cmd_pp 2024-09-10T18:54:25.888 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cmd_pp' 2024-09-10T18:54:25.889 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cmd 2024-09-10T18:54:25.892 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-10T18:54:25.894 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cmd_pp 2024-09-10T18:54:25.894 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.897 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cmd: /' 2024-09-10T18:54:25.900 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cmd_pp 2024-09-10T18:54:25.900 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78040 2024-09-10T18:54:25.900 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cmd_pp on pid 78040' 2024-09-10T18:54:25.900 INFO:tasks.workunit.client.0.smithi083.stdout:test api_cmd_pp on pid 78040 2024-09-10T18:54:25.901 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78040 2024-09-10T18:54:25.901 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.901 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.902 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-10T18:54:25.903 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_service 2024-09-10T18:54:25.904 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_service' 2024-09-10T18:54:25.907 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.907 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_service 2024-09-10T18:54:25.908 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-10T18:54:25.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_service 2024-09-10T18:54:25.919 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78079 2024-09-10T18:54:25.919 INFO:tasks.workunit.client.0.smithi083.stdout:test api_service on pid 78079 2024-09-10T18:54:25.921 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_service on pid 78079' 2024-09-10T18:54:25.921 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78079 2024-09-10T18:54:25.921 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.921 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.922 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_service_pp 2024-09-10T18:54:25.922 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_service_pp' 2024-09-10T18:54:25.922 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-10T18:54:25.922 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_service_pp 2024-09-10T18:54:25.922 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:25.924 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_service_pp 2024-09-10T18:54:25.924 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78088 2024-09-10T18:54:25.925 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_service_pp on pid 78088' 2024-09-10T18:54:25.925 INFO:tasks.workunit.client.0.smithi083.stdout:test api_service_pp on pid 78088 2024-09-10T18:54:25.929 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78088 2024-09-10T18:54:25.930 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.930 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-10T18:54:25.930 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.930 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_c_write_operations 2024-09-10T18:54:25.930 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_c_write_operations' 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.931 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_service 2024-09-10T18:54:25.932 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_c_write_operations 2024-09-10T18:54:25.932 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_service: /' 2024-09-10T18:54:25.932 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_service.log 2024-09-10T18:54:25.932 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.932 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.933 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.933 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.933 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-10T18:54:25.933 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_service_pp 2024-09-10T18:54:25.933 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-10T18:54:25.935 INFO:tasks.workunit.client.0.smithi083.stdout:test api_c_write_operations on pid 78109 2024-09-10T18:54:25.936 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_c_write_operations 2024-09-10T18:54:25.936 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78109 2024-09-10T18:54:25.936 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_c_write_operations on pid 78109' 2024-09-10T18:54:25.936 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78109 2024-09-10T18:54:25.936 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_c_read_operations 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_c_read_operations' 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_c_read_operations 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.938 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-10T18:54:25.939 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-10T18:54:25.940 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-10T18:54:25.942 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_c_read_operations 2024-09-10T18:54:25.942 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78141 2024-09-10T18:54:25.942 INFO:tasks.workunit.client.0.smithi083.stdout:test api_c_read_operations on pid 78141 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_c_read_operations on pid 78141' 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78141 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cls_remote_reads 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cls_remote_reads' 2024-09-10T18:54:25.943 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-10T18:54:25.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cls_remote_reads 2024-09-10T18:54:25.951 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.951 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.952 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.952 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.952 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-10T18:54:25.952 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-10T18:54:25.952 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-10T18:54:25.955 INFO:tasks.workunit.client.0.smithi083.stdout:test api_cls_remote_reads on pid 78160 2024-09-10T18:54:25.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cls_remote_reads 2024-09-10T18:54:25.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78160 2024-09-10T18:54:25.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cls_remote_reads on pid 78160' 2024-09-10T18:54:25.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78160 2024-09-10T18:54:25.956 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.956 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.961 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-10T18:54:25.964 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.964 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.964 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.964 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.965 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s list_parallel 2024-09-10T18:54:25.965 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' list_parallel' 2024-09-10T18:54:25.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo list_parallel 2024-09-10T18:54:25.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:25.972 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-10T18:54:25.975 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=list_parallel 2024-09-10T18:54:25.975 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78186 2024-09-10T18:54:25.975 INFO:tasks.workunit.client.0.smithi083.stdout:test list_parallel on pid 78186 2024-09-10T18:54:25.976 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test list_parallel on pid 78186' 2024-09-10T18:54:25.976 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78186 2024-09-10T18:54:25.976 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:25.976 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:25.978 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-10T18:54:25.978 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-10T18:54:25.981 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-10T18:54:25.983 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:25.984 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.984 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:25.984 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:25.986 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s open_pools_parallel 2024-09-10T18:54:25.986 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' open_pools_parallel' 2024-09-10T18:54:25.987 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_list_parallel 2024-09-10T18:54:25.994 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo open_pools_parallel 2024-09-10T18:54:25.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-10T18:54:25.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ list_parallel: /' 2024-09-10T18:54:25.995 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:26.006 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=open_pools_parallel 2024-09-10T18:54:26.006 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:26.007 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78272 2024-09-10T18:54:26.007 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test open_pools_parallel on pid 78272' 2024-09-10T18:54:26.007 INFO:tasks.workunit.client.0.smithi083.stdout:test open_pools_parallel on pid 78272 2024-09-10T18:54:26.008 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78272 2024-09-10T18:54:26.008 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s delete_pools_parallel 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' delete_pools_parallel' 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-09-10T18:54:26.009 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo delete_pools_parallel 2024-09-10T18:54:26.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=delete_pools_parallel 2024-09-10T18:54:26.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78280 2024-09-10T18:54:26.013 INFO:tasks.workunit.client.0.smithi083.stdout:test delete_pools_parallel on pid 78280 2024-09-10T18:54:26.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test delete_pools_parallel on pid 78280' 2024-09-10T18:54:26.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=78280 2024-09-10T18:54:26.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ ret=0 2024-09-10T18:54:26.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77635 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77635 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:26.015 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-10T18:54:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:26.017 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-10T18:54:26.017 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -z '' ']' 2024-09-10T18:54:26.026 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-10T18:54:26.026 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:26.026 INFO:tasks.workunit.client.0.smithi083.stderr:+ case $- in 2024-09-10T18:54:26.026 INFO:tasks.workunit.client.0.smithi083.stderr:+ return 2024-09-10T18:54:26.031 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-10T18:54:26.031 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-10T18:54:26.032 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-10T18:54:26.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:26.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:26.284 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:26.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:26.421 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:26.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:26.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:26.560 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:26.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:26.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:26.818 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:26.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:26 smithi165 bash[22591]: cluster 2024-09-10T18:54:24.559737+0000 mgr.y (mgr.24526) 188 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:26.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:26 smithi165 bash[22591]: cluster 2024-09-10T18:54:24.927883+0000 osd.3 (osd.3) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:26.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:26 smithi165 bash[22591]: cluster 2024-09-10T18:54:24.928030+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:26.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:26 smithi165 bash[22591]: cluster 2024-09-10T18:54:25.116627+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-10T18:54:26.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[18629]: cluster 2024-09-10T18:54:24.559737+0000 mgr.y (mgr.24526) 188 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[18629]: cluster 2024-09-10T18:54:24.927883+0000 osd.3 (osd.3) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[18629]: cluster 2024-09-10T18:54:24.928030+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[18629]: cluster 2024-09-10T18:54:25.116627+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[32381]: cluster 2024-09-10T18:54:24.559737+0000 mgr.y (mgr.24526) 188 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[32381]: cluster 2024-09-10T18:54:24.927883+0000 osd.3 (osd.3) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[32381]: cluster 2024-09-10T18:54:24.928030+0000 osd.3 (osd.3) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:26.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:26 smithi083 bash[32381]: cluster 2024-09-10T18:54:25.116627+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-09-10T18:54:26.957 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:26.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:27.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:27.096 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:27.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:27.234 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:27.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:27.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:27.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:27.510 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:27.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:27.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:27.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:27.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:27.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:27.782 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:27.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:27.920 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:27.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:28.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:28.055 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:28.159 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:28.186 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:28.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:28.317 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:28.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:28.446 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:28.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:28.580 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:28.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:28.717 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:28.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:28.854 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:28.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:28.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:28.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:29.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.121 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:29.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.253 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:29.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:29.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:29.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:29.532 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:29.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:29.662 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:29.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:29.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:29.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:29.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:30.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:30.068 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:30.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:30.208 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:30.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:30.345 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:30.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:30.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:30.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:30.619 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:30.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:30.757 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 20, 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 600, 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [ 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24404.0:96 4.3 4:c609908c:::notify.5:head [watch watch cookie 93844394783344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:22.075558+0000", 2024-09-10T18:54:30.787 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 308.70552920199998, 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0023988740000000001, 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24404", 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2284258763", 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 96 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075558+0000", 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075558+0000", 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075556+0000", 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.789 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075577+0000", 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1157000000000001e-05 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075606+0000", 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.8711e-05 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075614+0000", 2024-09-10T18:54:30.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.8939999999999993e-06 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.075974+0000", 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00035967300000000002 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.076101+0000", 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000127704 2024-09-10T18:54:30.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.076816+0000", 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00071503300000000001 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.076880+0000", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.3503999999999994e-05 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077048+0000", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016765000000000001 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077452+0000", 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00040433299999999999 2024-09-10T18:54:30.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077817+0000", 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00036550999999999999 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077849+0000", 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.2045000000000002e-05 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.794 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077887+0000", 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.7092999999999999e-05 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.077957+0000", 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.0523000000000002e-05 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:26 3.1b 3:db9d130d:gc::gc.24:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.099883+0000", 2024-09-10T18:54:30.795 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.681204256, 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0026548280000000001, 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 26 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.796 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099883+0000", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099883+0000", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099880+0000", 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:54:30.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099896+0000", 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5350999999999999e-05 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099920+0000", 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.4885e-05 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.099932+0000", 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1274000000000001e-05 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.100060+0000", 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000128567 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.799 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.100344+0000", 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00028343299999999998 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.101010+0000", 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00066639800000000001 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.101838+0000", 2024-09-10T18:54:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00082840599999999996 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.102400+0000", 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00056160100000000001 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.102429+0000", 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.9054e-05 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.801 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.102474+0000", 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5275000000000003e-05 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.102538+0000", 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.3548999999999998e-05 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:36 3.17 3:e9187ab8:reshard::reshard.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.122995+0000", 2024-09-10T18:54:30.802 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.65809190900001, 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0046736479999999999, 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 36 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.122995+0000", 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.122995+0000", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.122993+0000", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123002+0000", 2024-09-10T18:54:30.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6469999999999993e-06 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123023+0000", 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.0937e-05 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123032+0000", 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.3590000000000008e-06 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123162+0000", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000130455 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123460+0000", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00029807800000000001 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.123904+0000", 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00044327 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.124627+0000", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00072341799999999998 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.125264+0000", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00063677799999999995 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.127410+0000", 2024-09-10T18:54:30.807 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002145812 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.127553+0000", 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00014306900000000001 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.127669+0000", 2024-09-10T18:54:30.808 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000116455 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:47 3.17 3:e9187ab8:reshard::reshard.0000000000:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.131002+0000", 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.65008515900001, 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0019381890000000001, 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.809 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 47 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.810 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131002+0000", 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131002+0000", 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.130999+0000", 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:54:30.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131013+0000", 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3913999999999999e-05 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.812 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131046+0000", 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3259000000000001e-05 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.813 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131056+0000", 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2629999999999992e-06 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131149+0000", 2024-09-10T18:54:30.814 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3101000000000002e-05 2024-09-10T18:54:30.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.815 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.815 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131250+0000", 2024-09-10T18:54:30.815 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000101523 2024-09-10T18:54:30.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.816 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.816 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:54:30.816 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.131439+0000", 2024-09-10T18:54:30.816 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000188915 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.132123+0000", 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000683915 2024-09-10T18:54:30.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.818 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.818 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.132431+0000", 2024-09-10T18:54:30.818 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030773299999999998 2024-09-10T18:54:30.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.132844+0000", 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000413361 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.819 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.132902+0000", 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8026999999999999e-05 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.132940+0000", 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8243e-05 2024-09-10T18:54:30.820 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:95 3.6 3:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.158373+0000", 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.62271402300001, 2024-09-10T18:54:30.821 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0020369049999999999, 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.822 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 95 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158373+0000", 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158373+0000", 2024-09-10T18:54:30.823 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158371+0000", 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158380+0000", 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.0499999999999997e-06 2024-09-10T18:54:30.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158395+0000", 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4290999999999999e-05 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158399+0000", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6040000000000003e-06 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158464+0000", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.4305000000000002e-05 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158599+0000", 2024-09-10T18:54:30.826 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000135239 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.158789+0000", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000190593 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.159218+0000", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00042900700000000002 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.159447+0000", 2024-09-10T18:54:30.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000228719 2024-09-10T18:54:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.160309+0000", 2024-09-10T18:54:30.828 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00086143000000000005 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.160363+0000", 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.4874999999999999e-05 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.829 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.160410+0000", 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6876999999999999e-05 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:299 3.1b 3:db9d130d:gc::gc.24:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.830 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.269977+0000", 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.511110067, 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0024624080000000001, 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 299 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.831 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.269977+0000", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.269977+0000", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.269976+0000", 2024-09-10T18:54:30.832 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999986 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.269982+0000", 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2580000000000001e-06 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.269995+0000", 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2605e-05 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.833 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.270000+0000", 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8760000000000004e-06 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.270039+0000", 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.9768999999999998e-05 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.834 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.270102+0000", 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2797000000000002e-05 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.270332+0000", 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00023019100000000001 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.270781+0000", 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00044821699999999999 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.271362+0000", 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00058154099999999998 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.272364+0000", 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0010019899999999999 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.272399+0000", 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.4591000000000001e-05 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.272440+0000", 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.0960000000000001e-05 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.837 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:310 3.1f 3:fde4dd55:gc::gc.26:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.279106+0000", 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.501981312, 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002650258, 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.838 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.839 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.839 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.839 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 310 2024-09-10T18:54:30.839 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.839 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279106+0000", 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.840 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279106+0000", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279103+0000", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279116+0000", 2024-09-10T18:54:30.841 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2074e-05 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279135+0000", 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9142000000000001e-05 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279141+0000", 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.3459999999999999e-06 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279204+0000", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2514000000000005e-05 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279296+0000", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2263999999999996e-05 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.279558+0000", 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00026246899999999998 2024-09-10T18:54:30.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.280356+0000", 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00079818100000000002 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.280462+0000", 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010596500000000001 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.844 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.281702+0000", 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001239595 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.281733+0000", 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.0885e-05 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.281756+0000", 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3408e-05 2024-09-10T18:54:30.845 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:375 3.17 3:ec529b31:::obj_delete_at_hint.0000000050:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.341159+0000", 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.439928565, 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0024839039999999999, 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.846 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 375 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341159+0000", 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341159+0000", 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.847 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341156+0000", 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341167+0000", 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1336e-05 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.848 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341190+0000", 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3326e-05 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341198+0000", 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.0590000000000003e-06 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341296+0000", 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.7620000000000004e-05 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.341602+0000", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030624899999999997 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.342155+0000", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00055297199999999999 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.342751+0000", 2024-09-10T18:54:30.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00059548200000000004 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.343073+0000", 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00032271999999999997 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.343498+0000", 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00042469600000000001 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.343577+0000", 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.9151000000000002e-05 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.343643+0000", 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.5431000000000004e-05 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:376 3.17 3:ec529b31:::obj_delete_at_hint.0000000050:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.852 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.344790+0000", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.436297309, 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0019515279999999999, 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 376 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344790+0000", 2024-09-10T18:54:30.853 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344790+0000", 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344788+0000", 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.854 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344797+0000", 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7649999999999999e-06 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344815+0000", 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8144000000000001e-05 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344824+0000", 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.6309999999999992e-06 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.855 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.344912+0000", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8027000000000003e-05 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.345133+0000", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000221585 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.345611+0000", 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00047772199999999998 2024-09-10T18:54:30.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.346129+0000", 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00051783000000000005 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.857 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.346634+0000", 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00050535200000000004 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.346668+0000", 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3544999999999998e-05 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.346714+0000", 2024-09-10T18:54:30.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6579999999999998e-05 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.346742+0000", 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.7284000000000002e-05 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:381 3.6 3:6192870f:::obj_delete_at_hint.0000000053:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.357597+0000", 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.42349007500002, 2024-09-10T18:54:30.859 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021410180000000002, 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 381 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357597+0000", 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.860 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357597+0000", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357593+0000", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357612+0000", 2024-09-10T18:54:30.861 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9011000000000001e-05 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357634+0000", 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1832e-05 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357644+0000", 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.7180000000000008e-06 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357733+0000", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8974999999999994e-05 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.357919+0000", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000186047 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.358229+0000", 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030976300000000001 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.358699+0000", 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00047005400000000002 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.359288+0000", 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00058954600000000004 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.864 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.359609+0000", 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00032054700000000002 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.359686+0000", 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.7157999999999995e-05 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.359738+0000", 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.2380999999999999e-05 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:423 3.12 3:4b3bdbb4:::obj_delete_at_hint.0000000074:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.460760+0000", 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.32032745200002, 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021766239999999998, 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 423 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.866 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460760+0000", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460760+0000", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460757+0000", 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:54:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460769+0000", 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1996e-05 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460791+0000", 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1991999999999999e-05 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460799+0000", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.011e-06 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.460910+0000", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000110874 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.461135+0000", 2024-09-10T18:54:30.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00022529000000000001 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,7", 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.461510+0000", 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000374741 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.462175+0000", 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00066454999999999997 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.462623+0000", 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00044869000000000002 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.462815+0000", 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019138799999999999 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.871 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.462883+0000", 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8256000000000005e-05 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.462937+0000", 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.3705999999999997e-05 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:441 3.6 3:64735f45:::obj_delete_at_hint.0000000083:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.508918+0000", 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.27216901200001, 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001997548, 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 441 2024-09-10T18:54:30.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508918+0000", 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508918+0000", 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.874 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508914+0000", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999952 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508933+0000", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8961999999999999e-05 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508966+0000", 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3708000000000003e-05 2024-09-10T18:54:30.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.508988+0000", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1129e-05 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.509088+0000", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.9901000000000005e-05 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.509299+0000", 2024-09-10T18:54:30.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021107 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.509627+0000", 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00032817399999999998 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.510296+0000", 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00066956199999999996 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.510745+0000", 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000448524 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.510798+0000", 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.2849000000000002e-05 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.878 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.510848+0000", 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.0439999999999998e-05 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.510916+0000", 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7801999999999999e-05 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:442 3.6 3:64735f45:::obj_delete_at_hint.0000000083:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.511862+0000", 2024-09-10T18:54:30.879 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.26922576200002, 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0019914360000000001, 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 442 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511862+0000", 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511862+0000", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511860+0000", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511869+0000", 2024-09-10T18:54:30.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.5689999999999994e-06 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511888+0000", 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8981e-05 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.511895+0000", 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.4070000000000004e-06 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.512004+0000", 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010807 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.512222+0000", 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021869500000000001 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.512671+0000", 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000448906 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.513325+0000", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000654264 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.513686+0000", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000360825 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.513752+0000", 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.5730999999999998e-05 2024-09-10T18:54:30.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.513808+0000", 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.5943000000000003e-05 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.513853+0000", 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5160999999999997e-05 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:445 3.1 3:80d23011:::obj_delete_at_hint.0000000085:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.519646+0000", 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.26144163800001, 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002260668, 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.886 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 445 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519646+0000", 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519646+0000", 2024-09-10T18:54:30.887 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519644+0000", 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519653+0000", 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.8050000000000001e-06 2024-09-10T18:54:30.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519672+0000", 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9014000000000001e-05 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519682+0000", 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3500000000000003e-06 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.889 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.519818+0000", 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013582199999999999 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.520137+0000", 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031908099999999998 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,4", 2024-09-10T18:54:30.890 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.520646+0000", 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00050934199999999998 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.521356+0000", 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00070949299999999995 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.521728+0000", 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00037219599999999999 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.891 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.521824+0000", 2024-09-10T18:54:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6001000000000005e-05 2024-09-10T18:54:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.892 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.521867+0000", 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.3149000000000003e-05 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.521906+0000", 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.9496999999999998e-05 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.893 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:511 3.1 3:86cc8a85:::obj_delete_at_hint.0000000118:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.684246+0000", 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.09684098399998, 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021885989999999998, 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 511 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.894 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684246+0000", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684246+0000", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684242+0000", 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.895 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684259+0000", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6905000000000002e-05 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684294+0000", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.4601999999999998e-05 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684319+0000", 2024-09-10T18:54:30.896 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5256999999999999e-05 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684416+0000", 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6419e-05 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684613+0000", 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019761400000000001 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.897 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,4", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.684929+0000", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031593600000000001 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.685457+0000", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00052756099999999996 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.685928+0000", 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00047146299999999998 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.898 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.686277+0000", 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00034878600000000002 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.686377+0000", 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010032800000000001 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.899 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.686435+0000", 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7704999999999997e-05 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.14493.0:512 3.1 3:86cc8a85:::obj_delete_at_hint.0000000118:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:25.687462+0000", 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 305.09362491000002, 2024-09-10T18:54:30.900 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021690020000000002, 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.14493", 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/703687015", 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 512 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687462+0000", 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.901 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687462+0000", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687460+0000", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687474+0000", 2024-09-10T18:54:30.902 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4419999999999999e-05 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687499+0000", 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.4542e-05 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687508+0000", 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7879999999999995e-06 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.903 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687607+0000", 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.9356000000000002e-05 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.687757+0000", 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00014980899999999999 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 3,4", 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.688062+0000", 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030501600000000002 2024-09-10T18:54:30.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.688609+0000", 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00054689599999999995 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.689003+0000", 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000394046 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.689457+0000", 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000453743 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.689569+0000", 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000112372 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:25.689631+0000", 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2580000000000001e-05 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDScrubReserve(2.1 REQUEST e95)", 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:50:25.889431+0000", 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 244.891656426, 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0028319360000000002, 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.907 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889431+0000", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889431+0000", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889430+0000", 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:54:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889436+0000", 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.4060000000000004e-06 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889462+0000", 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6003000000000002e-05 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.909 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889478+0000", 2024-09-10T18:54:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5786000000000001e-05 2024-09-10T18:54:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.910 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.911 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889602+0000", 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000124797 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.889618+0000", 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5993000000000001e-05 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.912 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.892148+0000", 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0025299670000000002 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.892181+0000", 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3030000000000001e-05 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:50:25.892263+0000", 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.1481000000000001e-05 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.913 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:930 4.3 4:c609908c:::notify.5:head [watch ping cookie 94430446498256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:54:16.030398+0000", 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 14.750689116, 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031878100000000002, 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 930 2024-09-10T18:54:30.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030398+0000", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030398+0000", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030395+0000", 2024-09-10T18:54:30.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030409+0000", 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3753e-05 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030434+0000", 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5477e-05 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.916 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030444+0000", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.7990000000000005e-06 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030521+0000", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.6672e-05 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030614+0000", 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2928000000000002e-05 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:16.030717+0000", 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010318399999999999 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:939 4.3 4:c609908c:::notify.5:head [watch ping cookie 94430446498256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:54:21.034599+0000", 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 9.7464888090000006, 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00083800300000000003, 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.918 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 939 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034599+0000", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.919 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034599+0000", 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034593+0000", 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034615+0000", 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1937e-05 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.920 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034674+0000", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8820999999999998e-05 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034697+0000", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.2816999999999999e-05 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.034946+0000", 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024919699999999999 2024-09-10T18:54:30.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.035160+0000", 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021410900000000001 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:21.035437+0000", 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00027659599999999999 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:948 4.3 4:c609908c:::notify.5:head [watch ping cookie 94430446498256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:54:26.038690+0000", 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 4.7423972800000005, 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00036471399999999998, 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 948 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.923 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038690+0000", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038690+0000", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038688+0000", 2024-09-10T18:54:30.924 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999986 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038695+0000", 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.6649999999999997e-06 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038707+0000", 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2146e-05 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.925 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038715+0000", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.8199999999999997e-06 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038801+0000", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.5817999999999997e-05 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.038921+0000", 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011963700000000001 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:54:30.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:26.039055+0000", 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013426399999999999 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:54:30.927 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:54:30.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:30.960 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:30.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:31.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:31.100 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:31.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:31.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:31.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:31.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:31.371 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:31.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:31.509 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:31.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:31.614 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:31.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:31.771 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:31.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[32381]: audit 2024-09-10T18:54:25.913528+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[32381]: audit 2024-09-10T18:54:25.957425+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.83:0/2349694779' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[32381]: audit 2024-09-10T18:54:26.634775+0000 mon.a (mon.0) 893 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[18629]: audit 2024-09-10T18:54:25.913528+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[18629]: audit 2024-09-10T18:54:25.957425+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.83:0/2349694779' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T18:54:31.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:31 smithi083 bash[18629]: audit 2024-09-10T18:54:26.634775+0000 mon.a (mon.0) 893 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:31.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:32.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:32.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:31 smithi165 bash[22591]: audit 2024-09-10T18:54:25.913528+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:31 smithi165 bash[22591]: audit 2024-09-10T18:54:25.957425+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.83:0/2349694779' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T18:54:32.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:31 smithi165 bash[22591]: audit 2024-09-10T18:54:26.634775+0000 mon.a (mon.0) 893 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:32.038 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:32.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:32.170 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:32.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:32.306 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:32.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:32.443 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:32.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:32.582 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:32.685 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:32.717 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:32.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:32.850 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:32.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:32.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:32.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:32.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:33.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:32] "GET /metrics HTTP/1.1" 200 224786 "" "Prometheus/2.43.0" 2024-09-10T18:54:33.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:33.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:33.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:33.262 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:33.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:33.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:33.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:33.529 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:33.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:33.666 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:33.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:33.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:54:33.794 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:33.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:33.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:33.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:34.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:34.062 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:34.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:34.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.335 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3365143580 >> v1:172.21.15.83:6790/0 conn(0x7fc25c10b1a0 legacy=0x7fc25c10d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3365143580 shutdown_connections 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3365143580 >> 172.21.15.83:0/3365143580 conn(0x7fc25c075290 msgr2=0x7fc25c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3365143580 shutdown_connections 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3365143580 wait complete. 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 librados: starting msgr at 2024-09-10T18:54:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 librados: starting objecter 2024-09-10T18:54:34.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 Processor -- start 2024-09-10T18:54:34.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- start start 2024-09-10T18:54:34.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 librados: setting wanted keys 2024-09-10T18:54:34.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 librados: calling monclient init 2024-09-10T18:54:34.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c10aef0 con 0x7fc25c0fb370 2024-09-10T18:54:34.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c11f5d0 con 0x7fc25c10b1a0 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c1207d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc2614b5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc25c0fb370 0x7fc25c10a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51370/0 (socket says 172.21.15.83:51370) 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc2614b5640 1 -- 172.21.15.83:0/3193816989 learned_addr learned my addr 172.21.15.83:0/3193816989 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter ms_handle_connect 0x7fc25c0fb370 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter ms_handle_connect 0x7fc25c0fbe50 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter ms_handle_connect 0x7fc25c10b1a0 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610649749 0 0) 0x7fc25c1207d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2440054b0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485073015 0 0) 0x7fc25c11f5d0 con 0x7fc25c10b1a0 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c1207d0 con 0x7fc25c10b1a0 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4193644376 0 0) 0x7fc2440054b0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c11f5d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc250001880 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187365804 0 0) 0x7fc25c1207d0 con 0x7fc25c10b1a0 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2440054b0 con 0x7fc25c10b1a0 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc254003160 con 0x7fc25c10b1a0 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 182367071 0 0) 0x7fc25c11f5d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 >> v1:172.21.15.165:6789/0 conn(0x7fc25c10b1a0 legacy=0x7fc25c11cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 >> v1:172.21.15.83:6789/0 conn(0x7fc25c0fb370 legacy=0x7fc25c10a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c1219d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c1209c0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 10 client.25106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c120e10 con 0x7fc25c0fbe50 2024-09-10T18:54:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc263719640 1 librados: init done 2024-09-10T18:54:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc250001fd0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.360+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc250003230 con 0x7fc25c0fbe50 2024-09-10T18:54:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc248005010 con 0x7fc25c0fbe50 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc2500196d0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(124..124 src has 69..124) v4 ==== 11491+0+0 (unknown 1157140271 0 0) 0x7fc250088ae0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 10 client.25106.objecter ms_dispatch 0x7fc25c000b90 osd_map(124..124 src has 69..124) v4 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 3 client.25106.objecter handle_osd_map got epochs [124,124] > 0 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 3 client.25106.objecter handle_osd_map decoding full epoch 124 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 20 client.25106.objecter dump_active .. 0 homeless 2024-09-10T18:54:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.364+0000 7fc259ffb640 10 client.25106.objecter ms_handle_connect 0x7fc244071600 2024-09-10T18:54:34.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.372+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc250053e80 con 0x7fc25c0fbe50 2024-09-10T18:54:34.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:34.468 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:34.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:34.556+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fc2480052c0 con 0x7fc25c0fbe50 2024-09-10T18:54:34.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.605 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:34.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:34.742 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:34.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:34.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:34.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:34.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:35.004 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:35.107 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:35.138 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:35.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.253 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-10T18:54:35.253 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] Global test environment set-up. 2024-09-10T18:54:35.253 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5040 ms) 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.RegisterLate (32 ms) 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-09-10T18:54:35.254 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 5m), 8 in (since 6m) 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: flags noscrub,nodeep-scrub 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 6 pools, 132 pgs 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 132 active+clean 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 639 B/s rd, 0 op/s rd, 0 op/s wr 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.255 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 5m), 8 in (since 6m) 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: flags noscrub,nodeep-scrub 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T18:54:35.256 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 6 pools, 132 pgs 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 132 active+clean 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 1.5 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-09-10T18:54:35.257 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: flags noscrub,nodeep-scrub 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 6 pools, 132 pgs 2024-09-10T18:54:35.258 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T18:54:35.259 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:54:35.259 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 132 active+clean 2024-09-10T18:54:35.259 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:54:35.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:35.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:35.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:35.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:35.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:35.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.664 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:35.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.712+0000 7f0955232640 1 -- 172.21.15.83:0/3684874148 >> v1:172.21.15.83:6790/0 conn(0x7f09501116e0 legacy=0x7f0950113b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.712+0000 7f0955232640 1 -- 172.21.15.83:0/3684874148 shutdown_connections 2024-09-10T18:54:35.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.712+0000 7f0955232640 1 -- 172.21.15.83:0/3684874148 >> 172.21.15.83:0/3684874148 conn(0x7f09500692f0 msgr2=0x7f095006b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- 172.21.15.83:0/3684874148 shutdown_connections 2024-09-10T18:54:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- 172.21.15.83:0/3684874148 wait complete. 2024-09-10T18:54:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 librados: starting msgr at 2024-09-10T18:54:35.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 librados: starting objecter 2024-09-10T18:54:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 Processor -- start 2024-09-10T18:54:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- start start 2024-09-10T18:54:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 librados: setting wanted keys 2024-09-10T18:54:35.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 librados: calling monclient init 2024-09-10T18:54:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095010a700 con 0x7f095010ada0 2024-09-10T18:54:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0950125b70 con 0x7f095010db80 2024-09-10T18:54:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0950126d70 con 0x7f09501116e0 2024-09-10T18:54:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f094ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f095010ada0 0x7f0950109b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51424/0 (socket says 172.21.15.83:51424) 2024-09-10T18:54:35.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f094ed9d640 1 -- 172.21.15.83:0/4243312384 learned_addr learned my addr 172.21.15.83:0/4243312384 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter ms_handle_connect 0x7f095010ada0 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter ms_handle_connect 0x7f095010db80 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter ms_handle_connect 0x7f09501116e0 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339710002 0 0) 0x7f0950126d70 con 0x7f09501116e0 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0938003720 con 0x7f09501116e0 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313938701 0 0) 0x7f0950125b70 con 0x7f095010db80 2024-09-10T18:54:35.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0950126d70 con 0x7f095010db80 2024-09-10T18:54:35.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762286888 0 0) 0x7f0950126d70 con 0x7f095010db80 2024-09-10T18:54:35.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0950125b70 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0940001d90 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281098547 0 0) 0x7f0938003720 con 0x7f09501116e0 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0950126d70 con 0x7f09501116e0 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f09440015d0 con 0x7f09501116e0 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2107931894 0 0) 0x7f0950125b70 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 >> v1:172.21.15.83:6790/0 conn(0x7f09501116e0 legacy=0x7f0950123440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 >> v1:172.21.15.83:6789/0 conn(0x7f095010ada0 legacy=0x7f0950109b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950127f70 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0950125d60 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 10 client.25120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0950126270 con 0x7f095010db80 2024-09-10T18:54:35.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f0955232640 1 librados: init done 2024-09-10T18:54:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0940002780 con 0x7f095010db80 2024-09-10T18:54:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.716+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0940003540 con 0x7f095010db80 2024-09-10T18:54:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c005010 con 0x7f095010db80 2024-09-10T18:54:35.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f0940002320 con 0x7f095010db80 2024-09-10T18:54:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(124..124 src has 69..124) v4 ==== 11491+0+0 (unknown 1157140271 0 0) 0x7f0940088c00 con 0x7f095010db80 2024-09-10T18:54:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f09377fe640 10 client.25120.objecter ms_dispatch 0x7f0950000b90 osd_map(124..124 src has 69..124) v4 2024-09-10T18:54:35.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f09377fe640 3 client.25120.objecter handle_osd_map got epochs [124,124] > 0 2024-09-10T18:54:35.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.720+0000 7f09377fe640 3 client.25120.objecter handle_osd_map decoding full epoch 124 2024-09-10T18:54:35.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f09377fe640 20 client.25120.objecter dump_active .. 0 homeless 2024-09-10T18:54:35.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f09377fe640 10 client.25120.objecter ms_handle_connect 0x7f0938071560 2024-09-10T18:54:35.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/2357191571 >> v1:172.21.15.83:6790/0 conn(0x7f899c107f00 legacy=0x7f899c10a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/2357191571 shutdown_connections 2024-09-10T18:54:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/2357191571 >> 172.21.15.83:0/2357191571 conn(0x7f899c074a20 msgr2=0x7f899c0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/2357191571 shutdown_connections 2024-09-10T18:54:35.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/2357191571 wait complete. 2024-09-10T18:54:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 librados: starting msgr at 2024-09-10T18:54:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 librados: starting objecter 2024-09-10T18:54:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 Processor -- start 2024-09-10T18:54:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- start start 2024-09-10T18:54:35.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 librados: setting wanted keys 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 librados: calling monclient init 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f899c115790 con 0x7f899c1042e0 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f899c11f840 con 0x7f899c107f00 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f899c120a20 con 0x7f899c1015f0 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a1bfa640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f899c1015f0 0x7f899c114c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:43004/0 (socket says 172.21.15.83:43004) 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a1bfa640 1 -- 172.21.15.83:0/363382150 learned_addr learned my addr 172.21.15.83:0/363382150 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a13f9640 1 --1- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6789/0 conn(0x7f899c1042e0 0x7f899c1199e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51434/0 (socket says 172.21.15.83:51434) 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter ms_handle_connect 0x7f899c1015f0 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter ms_handle_connect 0x7f899c1042e0 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496656830 0 0) 0x7f899c120a20 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89880054b0 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter ms_handle_connect 0x7f899c107f00 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345218613 0 0) 0x7f89880054b0 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f899c120a20 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8998001940 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3507857445 0 0) 0x7f899c120a20 con 0x7f899c1015f0 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.165:6789/0 conn(0x7f899c107f00 legacy=0x7f899c11d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6789/0 conn(0x7f899c1042e0 legacy=0x7f899c1199e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c121c00 con 0x7f899c1015f0 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899c11e890 con 0x7f899c1015f0 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 10 client.25124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f899c11ed20 con 0x7f899c1015f0 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89a3e5e640 1 librados: init done 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8998001d10 con 0x7f899c1015f0 2024-09-10T18:54:35.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.724+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8998005c40 con 0x7f899c1015f0 2024-09-10T18:54:35.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8980004ff0 con 0x7f899c1015f0 2024-09-10T18:54:35.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f8998002a40 con 0x7f899c1015f0 2024-09-10T18:54:35.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0940054440 con 0x7f095010db80 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(124..124 src has 69..124) v4 ==== 11491+0+0 (unknown 1157140271 0 0) 0x7f899808b480 con 0x7f899c1015f0 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 osd_map(124..124 src has 69..124) v4 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 3 client.25124.objecter handle_osd_map got epochs [124,124] > 0 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 3 client.25124.objecter handle_osd_map decoding full epoch 124 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 20 client.25124.objecter dump_active .. 0 homeless 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v124) v1 ==== 54+0+16 (unknown 1125300689 0 2420000192) 0x7f899808d8a0 con 0x7f899c1015f0 2024-09-10T18:54:35.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.728+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f8988071460 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 1 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 1 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 0 1 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 4 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 10 client.25124.objecter _send_command 1 2024-09-10T18:54:35.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980004a60 con 0x7f8980004610 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f8980004610 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980004a60 con 0x7f8980004610 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 10 client.25124.objecter _finish_command 1 = system:0 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 4 2024-09-10T18:54:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.736+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 0 1 2024-09-10T18:54:35.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 2 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 2 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980005290 osd=0 3 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 0 2 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 4 2024-09-10T18:54:35.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 10 client.25124.objecter _send_command 2 2024-09-10T18:54:35.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.772+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8980004e40 con 0x7f8980004610 2024-09-10T18:54:35.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8980004e40 con 0x7f8980004610 2024-09-10T18:54:35.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 10 client.25124.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 4 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 0 2 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 3 2024-09-10T18:54:35.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 1 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 4 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 10 client.25124.objecter _send_command 3 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980002350 con 0x7f8980004150 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f8980004150 2024-09-10T18:54:35.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980002350 con 0x7f8980004150 2024-09-10T18:54:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:54:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 10 client.25124.objecter _finish_command 3 = system:0 2024-09-10T18:54:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 4 2024-09-10T18:54:35.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.776+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 1 3 2024-09-10T18:54:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 4 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 4 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980003fe0 osd=1 3 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 1 4 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 4 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 10 client.25124.objecter _send_command 4 2024-09-10T18:54:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.796+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f89800018f0 con 0x7f8980004150 2024-09-10T18:54:35.802 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:35.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89800018f0 con 0x7f8980004150 2024-09-10T18:54:35.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 10 client.25124.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 4 2024-09-10T18:54:35.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 1 4 2024-09-10T18:54:35.803 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:54:35.803 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:54:35.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 5 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 5 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 2 5 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 4 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 10 client.25124.objecter _send_command 5 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980003d40 con 0x7f898000d350 2024-09-10T18:54:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f898000d350 2024-09-10T18:54:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980003d40 con 0x7f898000d350 2024-09-10T18:54:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:54:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 10 client.25124.objecter _finish_command 5 = system:0 2024-09-10T18:54:35.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 4 2024-09-10T18:54:35.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.800+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 2 5 2024-09-10T18:54:35.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:35.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 6 2024-09-10T18:54:35.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 6 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f8980003620 osd=2 3 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 2 6 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 4 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 10 client.25124.objecter _send_command 6 2024-09-10T18:54:35.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f898000e6c0 con 0x7f898000d350 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f898000e6c0 con 0x7f898000d350 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 10 client.25124.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 4 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 2 6 2024-09-10T18:54:35.825 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 7 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 7 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 3 7 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 4 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 10 client.25124.objecter _send_command 7 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980015aa0 con 0x7f8980011fa0 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.820+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f8980011fa0 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.824+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980015aa0 con 0x7f8980011fa0 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.824+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.824+0000 7f89927fc640 10 client.25124.objecter _finish_command 7 = system:0 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.824+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 4 2024-09-10T18:54:35.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.824+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 3 7 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 8 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 8 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f89800032e0 osd=3 3 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 3 8 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 4 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 10 client.25124.objecter _send_command 8 2024-09-10T18:54:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.836+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f898000e1b0 con 0x7f8980011fa0 2024-09-10T18:54:35.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f898000e1b0 con 0x7f8980011fa0 2024-09-10T18:54:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 10 client.25124.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 4 2024-09-10T18:54:35.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 3 8 2024-09-10T18:54:35.843 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 9 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 9 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 4 9 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 4 2024-09-10T18:54:35.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 10 client.25124.objecter _send_command 9 2024-09-10T18:54:35.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f898000e150 con 0x7f898000cea0 2024-09-10T18:54:35.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f898000cea0 2024-09-10T18:54:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f898000e150 con 0x7f898000cea0 2024-09-10T18:54:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:54:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 10 client.25124.objecter _finish_command 9 = system:0 2024-09-10T18:54:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 4 2024-09-10T18:54:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.840+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 4 9 2024-09-10T18:54:35.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 10 2024-09-10T18:54:35.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 10 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898000df70 osd=4 3 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 4 10 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 4 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 10 client.25124.objecter _send_command 10 2024-09-10T18:54:35.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.876+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f898000e150 con 0x7f898000cea0 2024-09-10T18:54:35.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f898000e150 con 0x7f898000cea0 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 10 client.25124.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 4 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 4 10 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:54:35.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 11 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 11 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 5 11 2024-09-10T18:54:35.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 4 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 10 client.25124.objecter _send_command 11 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f898001c6b0 con 0x7f898001a9d0 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.880+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f898001a9d0 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.884+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f898001c6b0 con 0x7f898001a9d0 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.884+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:54:35.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.884+0000 7f89927fc640 10 client.25124.objecter _finish_command 11 = system:0 2024-09-10T18:54:35.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.884+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 4 2024-09-10T18:54:35.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.884+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 5 11 2024-09-10T18:54:35.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 12 2024-09-10T18:54:35.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 12 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898000e150 osd=5 3 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 5 12 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 4 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 10 client.25124.objecter _send_command 12 2024-09-10T18:54:35.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.904+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f898001c670 con 0x7f898001a9d0 2024-09-10T18:54:35.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f898001c670 con 0x7f898001a9d0 2024-09-10T18:54:35.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 10 client.25124.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 4 2024-09-10T18:54:35.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 5 12 2024-09-10T18:54:35.912 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:54:35.913 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 13 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 13 2024-09-10T18:54:35.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 6 13 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 4 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 10 client.25124.objecter _send_command 13 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980022af0 con 0x7f898001c880 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f898001c880 2024-09-10T18:54:35.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f093c0052c0 con 0x7f095010db80 2024-09-10T18:54:35.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980022af0 con 0x7f898001c880 2024-09-10T18:54:35.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:54:35.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 10 client.25124.objecter _finish_command 13 = system:0 2024-09-10T18:54:35.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 4 2024-09-10T18:54:35.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.908+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 6 13 2024-09-10T18:54:35.926 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:35.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 14 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 14 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f898001c670 osd=6 3 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 6 14 2024-09-10T18:54:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 4 2024-09-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 10 client.25124.objecter _send_command 14 2024-09-10T18:54:35.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.924+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f89800243d0 con 0x7f898001c880 2024-09-10T18:54:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:35.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:35.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f89800243d0 con 0x7f898001c880 2024-09-10T18:54:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 10 client.25124.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 4 2024-09-10T18:54:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 6 14 2024-09-10T18:54:35.932 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:54:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 15 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 15 2024-09-10T18:54:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 7 15 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 4 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 10 client.25124.objecter _send_command 15 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8980024ef0 con 0x7f8980024a80 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.928+0000 7f89927fc640 10 client.25124.objecter ms_handle_connect 0x7f8980024a80 2024-09-10T18:54:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.932+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f8980024ef0 con 0x7f8980024a80 2024-09-10T18:54:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.932+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:54:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.932+0000 7f89927fc640 10 client.25124.objecter _finish_command 15 = system:0 2024-09-10T18:54:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.932+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 4 2024-09-10T18:54:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.932+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 7 15 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 10 client.25124.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign -1 16 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 20 client.25124.objecter _get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_remove -1 16 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 20 client.25124.objecter get_session s=0x7f89800241f0 osd=7 3 2024-09-10T18:54:35.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 15 client.25124.objecter _session_command_op_assign 7 16 2024-09-10T18:54:35.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 4 2024-09-10T18:54:35.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 10 client.25124.objecter _send_command 16 2024-09-10T18:54:35.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.964+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8980024ef0 con 0x7f8980024a80 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89927fc640 1 -- 172.21.15.83:0/363382150 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f8980024ef0 con 0x7f8980024a80 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89927fc640 10 client.25124.objecter ms_dispatch 0x7f899c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89927fc640 10 client.25124.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89927fc640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 4 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89927fc640 15 client.25124.objecter _session_command_op_remove 7 16 2024-09-10T18:54:35.973 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 10 librados: watch_flush enter 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 10 librados: watch_flush exit 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.0 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6801/2535527483 conn(0x7f8980004610 legacy=0x7f8980002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980005290 osd=0 1 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.1 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6805/2309624637 conn(0x7f8980004150 legacy=0x7f898000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003fe0 osd=1 1 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.2 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6809/386094220 conn(0x7f898000d350 legacy=0x7f898000d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.968+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f8980003620 osd=2 1 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.3 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6813/1279688021 conn(0x7f8980011fa0 legacy=0x7f8980014380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800032e0 osd=3 1 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.4 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.165:6800/3225515265 conn(0x7f898000cea0 legacy=0x7f8980018da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000df70 osd=4 1 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.5 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.165:6804/571173777 conn(0x7f898001a9d0 legacy=0x7f898001ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a23fb640 1 -- 172.21.15.83:0/363382150 reap_dead start 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a23fb640 1 -- 172.21.15.83:0/363382150 reap_dead start 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a23fb640 1 -- 172.21.15.83:0/363382150 reap_dead start 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898000e150 osd=5 1 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.6 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.165:6808/1317922936 conn(0x7f898001c880 legacy=0x7f8980021290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f898001c670 osd=6 1 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter close_session for osd.7 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.165:6812/1610151650 conn(0x7f8980024a80 legacy=0x7f8980022af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter put_session s=0x7f89800241f0 osd=7 1 2024-09-10T18:54:35.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 20 client.25124.objecter shutdown clearing up homeless session... 2024-09-10T18:54:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 10 client.25124.objecter successfully canceled tick 2024-09-10T18:54:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8988071460 legacy=0x7f8988073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> v1:172.21.15.83:6790/0 conn(0x7f899c1015f0 legacy=0x7f899c114c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 shutdown_connections 2024-09-10T18:54:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 >> 172.21.15.83:0/363382150 conn(0x7f899c074a20 msgr2=0x7f899c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 shutdown_connections 2024-09-10T18:54:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 -- 172.21.15.83:0/363382150 wait complete. 2024-09-10T18:54:35.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:35.972+0000 7f89a3e5e640 1 librados: shutdown 2024-09-10T18:54:36.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:36.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:36.192 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:36.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:36.327 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:36.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: cluster 2024-09-10T18:54:26.561115+0000 mgr.y (mgr.24526) 189 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:26.641981+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3652997888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:27.627363+0000 mgr.y (mgr.24526) 190 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: cluster 2024-09-10T18:54:28.561821+0000 mgr.y (mgr.24526) 191 : cluster [DBG] pgmap v161: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:29.553869+0000 mon.c (mon.2) 102 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: cluster 2024-09-10T18:54:30.563488+0000 mgr.y (mgr.24526) 192 : cluster [DBG] pgmap v162: 196 pgs: 42 creating+peering, 22 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:31.190569+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.83:0/2884082795' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: cluster 2024-09-10T18:54:31.679370+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-10T18:54:36.439 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:31.692270+0000 mon.a (mon.0) 895 : audit [INF] from='client.24788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.440 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:36 smithi165 bash[22591]: audit 2024-09-10T18:54:31.695906+0000 mon.a (mon.0) 896 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.462 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:36.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: cluster 2024-09-10T18:54:26.561115+0000 mgr.y (mgr.24526) 189 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:26.641981+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3652997888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:27.627363+0000 mgr.y (mgr.24526) 190 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: cluster 2024-09-10T18:54:28.561821+0000 mgr.y (mgr.24526) 191 : cluster [DBG] pgmap v161: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:29.553869+0000 mon.c (mon.2) 102 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: cluster 2024-09-10T18:54:30.563488+0000 mgr.y (mgr.24526) 192 : cluster [DBG] pgmap v162: 196 pgs: 42 creating+peering, 22 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:31.190569+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.83:0/2884082795' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: cluster 2024-09-10T18:54:31.679370+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:31.692270+0000 mon.a (mon.0) 895 : audit [INF] from='client.24788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[18629]: audit 2024-09-10T18:54:31.695906+0000 mon.a (mon.0) 896 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: cluster 2024-09-10T18:54:26.561115+0000 mgr.y (mgr.24526) 189 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:26.641981+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3652997888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:27.627363+0000 mgr.y (mgr.24526) 190 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: cluster 2024-09-10T18:54:28.561821+0000 mgr.y (mgr.24526) 191 : cluster [DBG] pgmap v161: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 789 B/s rd, 0 op/s 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:29.553869+0000 mon.c (mon.2) 102 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: cluster 2024-09-10T18:54:30.563488+0000 mgr.y (mgr.24526) 192 : cluster [DBG] pgmap v162: 196 pgs: 42 creating+peering, 22 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:31.190569+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.83:0/2884082795' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: cluster 2024-09-10T18:54:31.679370+0000 mon.a (mon.0) 894 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:31.692270+0000 mon.a (mon.0) 895 : audit [INF] from='client.24788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:36 smithi083 bash[32381]: audit 2024-09-10T18:54:31.695906+0000 mon.a (mon.0) 896 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:36.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:36.593 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:36.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:36.732 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:36.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:36.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:36.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:37.001 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:37.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:37.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:37.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:37.272 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:37.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.400+0000 7f09377fe640 1 -- 172.21.15.83:0/4243312384 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v126) v1 ==== 91+0+0 (unknown 676959713 0 0) 0x7f0940053de0 con 0x7f095010db80 2024-09-10T18:54:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T18:54:37.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7fc259ffb640 1 -- 172.21.15.83:0/3193816989 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v126) v1 ==== 113+0+0 (unknown 1972986681 0 0) 0x7fc250053ca0 con 0x7fc25c0fbe50 2024-09-10T18:54:37.409 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T18:54:37.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 10 librados: watch_flush enter 2024-09-10T18:54:37.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 10 librados: watch_flush exit 2024-09-10T18:54:37.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 20 client.25120.objecter shutdown clearing up homeless session... 2024-09-10T18:54:37.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 10 client.25120.objecter successfully canceled tick 2024-09-10T18:54:37.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 >> v1:172.21.15.83:6800/3071793735 conn(0x7f0938071560 legacy=0x7f0938073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:37.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 >> v1:172.21.15.165:6789/0 conn(0x7f095010db80 legacy=0x7f095011fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:37.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 shutdown_connections 2024-09-10T18:54:37.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.404+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 >> 172.21.15.83:0/4243312384 conn(0x7f09500692f0 msgr2=0x7f0950113b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:37.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 shutdown_connections 2024-09-10T18:54:37.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 10 librados: watch_flush enter 2024-09-10T18:54:37.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 10 librados: watch_flush exit 2024-09-10T18:54:37.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 20 client.25106.objecter shutdown clearing up homeless session... 2024-09-10T18:54:37.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 10 client.25106.objecter successfully canceled tick 2024-09-10T18:54:37.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc244071600 legacy=0x7fc244073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 >> v1:172.21.15.83:6790/0 conn(0x7fc25c0fbe50 legacy=0x7fc25c119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 shutdown_connections 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 >> 172.21.15.83:0/3193816989 conn(0x7fc25c075290 msgr2=0x7fc25c0f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7f0955232640 1 -- 172.21.15.83:0/4243312384 wait complete. 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 shutdown_connections 2024-09-10T18:54:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7fc263719640 1 -- 172.21.15.83:0/3193816989 wait complete. 2024-09-10T18:54:37.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.412+0000 7fc263719640 1 librados: shutdown 2024-09-10T18:54:37.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:37.408+0000 7f0955232640 1 librados: shutdown 2024-09-10T18:54:37.431 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:37.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.709848+0000 mon.b (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.83:0/3913324152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.710479+0000 mon.b (mon.1) 37 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.710729+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.83:0/2511920983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.711191+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.83:0/4063647720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.816926+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.83:0/4207984406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.819475+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3856020767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.819911+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.83:0/3888037774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.820171+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.83:0/3523853163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:31.822873+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:32.564679+0000 mgr.y (mgr.24526) 193 : cluster [DBG] pgmap v164: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:33.219117+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.83:0/4093293606' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:33.351068+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:34.561069+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3193816989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:34.565357+0000 mgr.y (mgr.24526) 194 : cluster [DBG] pgmap v165: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:35.247326+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.83:0/3757481578' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:35.732693+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.83:0/363382150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:35.910298+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.83:0/4243312384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:36.356205+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:36.356727+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.363536+0000 mon.a (mon.0) 899 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.364344+0000 mon.a (mon.0) 900 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.372892+0000 mon.a (mon.0) 901 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.373227+0000 mon.a (mon.0) 902 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.373523+0000 mon.a (mon.0) 903 : audit [INF] from='client.24809 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.373851+0000 mon.a (mon.0) 904 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.374106+0000 mon.a (mon.0) 905 : audit [INF] from='client.24821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.374512+0000 mon.a (mon.0) 906 : audit [INF] from='client.24791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.374801+0000 mon.a (mon.0) 907 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.387630+0000 mon.a (mon.0) 908 : audit [INF] from='client.24788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.709848+0000 mon.b (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.83:0/3913324152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.710479+0000 mon.b (mon.1) 37 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.710729+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.83:0/2511920983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.711191+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.83:0/4063647720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.816926+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.83:0/4207984406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.819475+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3856020767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.819911+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.83:0/3888037774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.820171+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.83:0/3523853163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:31.822873+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:32.564679+0000 mgr.y (mgr.24526) 193 : cluster [DBG] pgmap v164: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:33.219117+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.83:0/4093293606' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:33.351068+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:34.561069+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3193816989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:34.565357+0000 mgr.y (mgr.24526) 194 : cluster [DBG] pgmap v165: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:35.247326+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.83:0/3757481578' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:35.732693+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.83:0/363382150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:35.910298+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.83:0/4243312384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:36.356205+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:36.356727+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:37.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.363536+0000 mon.a (mon.0) 899 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.364344+0000 mon.a (mon.0) 900 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.372892+0000 mon.a (mon.0) 901 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.373227+0000 mon.a (mon.0) 902 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.373523+0000 mon.a (mon.0) 903 : audit [INF] from='client.24809 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.373851+0000 mon.a (mon.0) 904 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.374106+0000 mon.a (mon.0) 905 : audit [INF] from='client.24821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.374512+0000 mon.a (mon.0) 906 : audit [INF] from='client.24791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.374801+0000 mon.a (mon.0) 907 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.387630+0000 mon.a (mon.0) 908 : audit [INF] from='client.24788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.388122+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:36.389536+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.391140+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.83:0/772315144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.391662+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.83:0/1631994152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.391974+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.83:0/3150570370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.392412+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.83:0/4263382328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.392646+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3310890798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.392952+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3183531288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.393387+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.83:0/3747125052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.393801+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.83:0/1488322349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.394107+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.83:0/4243639857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.402373+0000 mon.a (mon.0) 911 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.402633+0000 mon.a (mon.0) 912 : audit [INF] from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.402926+0000 mon.a (mon.0) 913 : audit [INF] from='client.25106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.404478+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.405476+0000 mon.a (mon.0) 914 : audit [INF] from='client.24866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.405820+0000 mon.a (mon.0) 915 : audit [INF] from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.406085+0000 mon.a (mon.0) 916 : audit [INF] from='client.24950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.406436+0000 mon.a (mon.0) 917 : audit [INF] from='client.24911 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.406666+0000 mon.a (mon.0) 918 : audit [INF] from='client.24857 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.406954+0000 mon.a (mon.0) 919 : audit [INF] from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: cluster 2024-09-10T18:54:36.566776+0000 mgr.y (mgr.24526) 195 : cluster [DBG] pgmap v167: 932 pgs: 303 creating+peering, 497 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.849305+0000 mon.a (mon.0) 920 : audit [INF] from='client.24851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.849990+0000 mon.a (mon.0) 921 : audit [INF] from='client.24917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.850418+0000 mon.a (mon.0) 922 : audit [INF] from='client.24956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:36.850837+0000 mon.a (mon.0) 923 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[18629]: audit 2024-09-10T18:54:37.280642+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/1187811648' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.388122+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:36.389536+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.391140+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.83:0/772315144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.391662+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.83:0/1631994152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.391974+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.83:0/3150570370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.392412+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.83:0/4263382328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.392646+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3310890798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.392952+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3183531288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.393387+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.83:0/3747125052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.393801+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.83:0/1488322349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.394107+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.83:0/4243639857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.402373+0000 mon.a (mon.0) 911 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.402633+0000 mon.a (mon.0) 912 : audit [INF] from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.402926+0000 mon.a (mon.0) 913 : audit [INF] from='client.25106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.404478+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.405476+0000 mon.a (mon.0) 914 : audit [INF] from='client.24866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.405820+0000 mon.a (mon.0) 915 : audit [INF] from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.406085+0000 mon.a (mon.0) 916 : audit [INF] from='client.24950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.406436+0000 mon.a (mon.0) 917 : audit [INF] from='client.24911 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.406666+0000 mon.a (mon.0) 918 : audit [INF] from='client.24857 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.406954+0000 mon.a (mon.0) 919 : audit [INF] from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: cluster 2024-09-10T18:54:36.566776+0000 mgr.y (mgr.24526) 195 : cluster [DBG] pgmap v167: 932 pgs: 303 creating+peering, 497 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.849305+0000 mon.a (mon.0) 920 : audit [INF] from='client.24851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.849990+0000 mon.a (mon.0) 921 : audit [INF] from='client.24917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.850418+0000 mon.a (mon.0) 922 : audit [INF] from='client.24956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:36.850837+0000 mon.a (mon.0) 923 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:37 smithi083 bash[32381]: audit 2024-09-10T18:54:37.280642+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/1187811648' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.574 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:37.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:37.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:37.780 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-10T18:54:37.780 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] Global test environment set-up. 2024-09-10T18:54:37.780 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-10T18:54:37.780 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (30 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-10T18:54:37.781 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] 12 tests from AsioRados (51 ms total) 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] Global test environment tear-down 2024-09-10T18:54:37.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (12029 ms total) 2024-09-10T18:54:37.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-10T18:54:37.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:37.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:37.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.709848+0000 mon.b (mon.1) 36 : audit [INF] from='client.? v1:172.21.15.83:0/3913324152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.710479+0000 mon.b (mon.1) 37 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.710729+0000 mon.b (mon.1) 38 : audit [INF] from='client.? v1:172.21.15.83:0/2511920983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.711191+0000 mon.b (mon.1) 39 : audit [INF] from='client.? v1:172.21.15.83:0/4063647720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.816926+0000 mon.c (mon.2) 104 : audit [INF] from='client.? v1:172.21.15.83:0/4207984406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.819475+0000 mon.c (mon.2) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3856020767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.819911+0000 mon.c (mon.2) 106 : audit [INF] from='client.? v1:172.21.15.83:0/3888037774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.820171+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.83:0/3523853163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:31.822873+0000 mon.c (mon.2) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:32.564679+0000 mgr.y (mgr.24526) 193 : cluster [DBG] pgmap v164: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:33.219117+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.83:0/4093293606' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:33.351068+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:34.561069+0000 mon.c (mon.2) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3193816989' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:34.565357+0000 mgr.y (mgr.24526) 194 : cluster [DBG] pgmap v165: 580 pgs: 42 creating+peering, 406 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:35.247326+0000 mon.b (mon.1) 40 : audit [DBG] from='client.? v1:172.21.15.83:0/3757481578' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:35.732693+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? v1:172.21.15.83:0/363382150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:37.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:35.910298+0000 mon.b (mon.1) 41 : audit [INF] from='client.? v1:172.21.15.83:0/4243312384' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:36.356205+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:36.356727+0000 mon.a (mon.0) 898 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.363536+0000 mon.a (mon.0) 899 : audit [INF] from='client.24796 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.364344+0000 mon.a (mon.0) 900 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.372892+0000 mon.a (mon.0) 901 : audit [INF] from='client.24839 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.373227+0000 mon.a (mon.0) 902 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.373523+0000 mon.a (mon.0) 903 : audit [INF] from='client.24809 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.373851+0000 mon.a (mon.0) 904 : audit [INF] from='client.24883 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.374106+0000 mon.a (mon.0) 905 : audit [INF] from='client.24821 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.374512+0000 mon.a (mon.0) 906 : audit [INF] from='client.24791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.374801+0000 mon.a (mon.0) 907 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.387630+0000 mon.a (mon.0) 908 : audit [INF] from='client.24788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi083-77648-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.388122+0000 mon.a (mon.0) 909 : audit [INF] from='client.? v1:172.21.15.83:0/1049522368' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi083-77640-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:36.389536+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.391140+0000 mon.c (mon.2) 113 : audit [INF] from='client.? v1:172.21.15.83:0/772315144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.391662+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.83:0/1631994152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.391974+0000 mon.c (mon.2) 115 : audit [INF] from='client.? v1:172.21.15.83:0/3150570370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.392412+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.83:0/4263382328' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.392646+0000 mon.c (mon.2) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3310890798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.392952+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3183531288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.393387+0000 mon.c (mon.2) 119 : audit [INF] from='client.? v1:172.21.15.83:0/3747125052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.393801+0000 mon.c (mon.2) 120 : audit [INF] from='client.? v1:172.21.15.83:0/1488322349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.394107+0000 mon.c (mon.2) 121 : audit [INF] from='client.? v1:172.21.15.83:0/4243639857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.402373+0000 mon.a (mon.0) 911 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.402633+0000 mon.a (mon.0) 912 : audit [INF] from='client.25120 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.402926+0000 mon.a (mon.0) 913 : audit [INF] from='client.25106 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.404478+0000 mon.c (mon.2) 122 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.405476+0000 mon.a (mon.0) 914 : audit [INF] from='client.24866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.405820+0000 mon.a (mon.0) 915 : audit [INF] from='client.24845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.406085+0000 mon.a (mon.0) 916 : audit [INF] from='client.24950 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.406436+0000 mon.a (mon.0) 917 : audit [INF] from='client.24911 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.406666+0000 mon.a (mon.0) 918 : audit [INF] from='client.24857 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.406954+0000 mon.a (mon.0) 919 : audit [INF] from='client.24929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: cluster 2024-09-10T18:54:36.566776+0000 mgr.y (mgr.24526) 195 : cluster [DBG] pgmap v167: 932 pgs: 303 creating+peering, 497 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.849305+0000 mon.a (mon.0) 920 : audit [INF] from='client.24851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.849990+0000 mon.a (mon.0) 921 : audit [INF] from='client.24917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.850418+0000 mon.a (mon.0) 922 : audit [INF] from='client.24956 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:37.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:36.850837+0000 mon.a (mon.0) 923 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:37.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:37 smithi165 bash[22591]: audit 2024-09-10T18:54:37.280642+0000 mon.a (mon.0) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/1187811648' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:37.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:37.984 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:37.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:38.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:38.110 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:38.207 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-10T18:54:38.207 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] Global test environment set-up. 2024-09-10T18:54:38.207 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-10T18:54:38.207 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjects (327 ms) 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (42 ms) 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo2 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo3 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo4 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo5 2024-09-10T18:54:38.208 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo6,foo7 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo7 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo6 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo4 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo5 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo7 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo6 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo1 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo1 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo2 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo3 2024-09-10T18:54:38.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (138 ms) 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 1 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 10 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 13 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 7 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 14 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 0 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 15 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 11 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 5 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 8 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 6 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 3 0 2024-09-10T18:54:38.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 4 0 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 12 0 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 9 0 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2 0 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (110 ms) 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: x cursor=MIN 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=1 cursor=16:02547ec2:::1:head 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=10 cursor=16:52ea6a34:::10:head 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=13 cursor=16:566253c9:::13:head 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=7 cursor=16:5c6b0b28:::7:head 2024-09-10T18:54:38.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=14 cursor=16:62a1935d:::14:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=0 cursor=16:6cac518f:::0:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=15 cursor=16:863748b0:::15:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=11 cursor=16:89d3ae78:::11:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=5 cursor=16:b29083e3:::5:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=8 cursor=16:bd63b0f1:::8:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=6 cursor=16:c4fdafeb:::6:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=3 cursor=16:cfc208b3:::3:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=4 cursor=16:d83876eb:::4:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=12 cursor=16:de5d7c5f:::12:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=9 cursor=16:e960b815:::9:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=2 cursor=16:f905c69b:::2:head 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-10T18:54:38.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=1 cursor=16:02547ec2:::1:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=10 cursor=16:52ea6a34:::10:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=13 cursor=16:566253c9:::13:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:566253c9:::13:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=7 cursor=16:5c6b0b28:::7:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-09-10T18:54:38.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-09-10T18:54:38.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:38.236 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:38.291 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=14 cursor=16:62a1935d:::14:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=0 cursor=16:6cac518f:::0:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=15 cursor=16:863748b0:::15:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:863748b0:::15:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=11 cursor=16:89d3ae78:::11:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=5 cursor=16:b29083e3:::5:head 2024-09-10T18:54:38.292 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=8 cursor=16:bd63b0f1:::8:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=6 cursor=16:c4fdafeb:::6:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=3 cursor=16:cfc208b3:::3:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=4 cursor=16:d83876eb:::4:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-09-10T18:54:38.293 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=12 cursor=16:de5d7c5f:::12:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=9 cursor=16:e960b815:::9:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:e960b815:::9:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=2 cursor=16:f905c69b:::2:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:d83876eb:::4:head expected=16:d83876eb:::4:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=4 expected=4 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-09-10T18:54:38.294 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:cfc208b3:::3:head expected=16:cfc208b3:::3:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=3 expected=3 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:f905c69b:::2:head expected=16:f905c69b:::2:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=2 expected=2 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:e960b815:::9:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:e960b815:::9:head expected=16:e960b815:::9:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=9 expected=9 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:de5d7c5f:::12:head expected=16:de5d7c5f:::12:head 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-09-10T18:54:38.295 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=12 expected=12 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:89d3ae78:::11:head expected=16:89d3ae78:::11:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=11 expected=11 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:c4fdafeb:::6:head expected=16:c4fdafeb:::6:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=6 expected=6 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:bd63b0f1:::8:head expected=16:bd63b0f1:::8:head 2024-09-10T18:54:38.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-09-10T18:54:38.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:38.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:38.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry= list_parallel: process_1_[78278]: starting. 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: creating pool ceph_test_rados_list_parallel.smithi083-78207 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: created object 0... 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: created object 25... 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: created object 49... 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: finishing. 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[78278]: shutting down. 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: starting. 2024-09-10T18:54:38.571 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: listing objects. 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: listed object 0... 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: listed object 25... 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: saw 50 objects 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[78279]: shutting down. 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: starting. 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: creating pool ceph_test_rados_list_parallel.smithi083-78207 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: created object 0... 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: created object 25... 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: created object 49... 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: finishing. 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[79954]: shutting down. 2024-09-10T18:54:38.572 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: starting. 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: listing objects. 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: listed object 0... 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: listed object 25... 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: saw 43 objects 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[79955]: shutting down. 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: starting. 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: removed 25 objects... 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: removed half of the objects 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: removed 50 objects... 2024-09-10T18:54:38.573 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: removed 50 objects 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[79956]: shutting down. 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: starting. 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: creating pool ceph_test_rados_list_parallel.smithi083-78207 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: created object 0... 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: created object 25... 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: created object 49... 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: finishing. 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[80020]: shutting down. 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.574 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: starting. 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: listing objects. 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: listed object 0... 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: listed object 25... 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: listed object 50... 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: saw 57 objects 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[80021]: shutting down. 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: starting. 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: added 25 objects... 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: added half of the objects 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: added 50 objects... 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: added 50 objects 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[80022]: shutting down. 2024-09-10T18:54:38.575 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.576 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:38.646 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:38.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:38.779 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:38.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: starting. 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: creating pool ceph_test_rados_list_parallel.smithi083-78207 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: created object 0... 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: created object 25... 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: created object 49... 2024-09-10T18:54:38.856 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: finishing. 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[80086]: shutting down. 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: starting. 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: listing objects. 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: listed object 0... 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: listed object 25... 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: listed object 50... 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: listed object 75... 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: saw 97 objects 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[80087]: shutting down. 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.857 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: starting. 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: removed 25 objects... 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: removed half of the objects 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: removed 50 objects... 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: removed 50 objects 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[80090]: shutting down. 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: starting. 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: added 25 objects... 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: added half of the objects 2024-09-10T18:54:38.858 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: added 50 objects... 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: added 50 objects 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[80088]: shutting down. 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: starting. 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: added 25 objects... 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: added half of the objects 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: added 50 objects... 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: added 50 objects 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[80089]: shutting down. 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.859 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: starting. 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: creating pool ceph_test_rados_list_parallel.smithi083-78207 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: created object 0... 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: created object 25... 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: created object 49... 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: finishing. 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[80175]: shutting down. 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:38.860 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: starting. 2024-09-10T18:54:38.861 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listing objects. 2024-09-10T18:54:38.861 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 0... 2024-09-10T18:54:38.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:38.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:39.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:39.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:39.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: cluster 2024-09-10T18:54:37.384421+0000 mon.a (mon.0) 925 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:39.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396287+0000 mon.a (mon.0) 926 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396400+0000 mon.a (mon.0) 927 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396509+0000 mon.a (mon.0) 928 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396663+0000 mon.a (mon.0) 929 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396759+0000 mon.a (mon.0) 930 : audit [INF] from='client.24809 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396834+0000 mon.a (mon.0) 931 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.396922+0000 mon.a (mon.0) 932 : audit [INF] from='client.24821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397025+0000 mon.a (mon.0) 933 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397117+0000 mon.a (mon.0) 934 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397280+0000 mon.a (mon.0) 935 : audit [INF] from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397358+0000 mon.a (mon.0) 936 : audit [INF] from='client.25106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397450+0000 mon.a (mon.0) 937 : audit [INF] from='client.24866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397530+0000 mon.a (mon.0) 938 : audit [INF] from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397608+0000 mon.a (mon.0) 939 : audit [INF] from='client.24950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397713+0000 mon.a (mon.0) 940 : audit [INF] from='client.24911 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397805+0000 mon.a (mon.0) 941 : audit [INF] from='client.24857 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397882+0000 mon.a (mon.0) 942 : audit [INF] from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.397983+0000 mon.a (mon.0) 943 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.398259+0000 mon.a (mon.0) 944 : audit [INF] from='client.24917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.398364+0000 mon.a (mon.0) 945 : audit [INF] from='client.24956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.398451+0000 mon.a (mon.0) 946 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: cluster 2024-09-10T18:54:37.398559+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.409818+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[18629]: audit 2024-09-10T18:54:37.449956+0000 mon.a (mon.0) 948 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: cluster 2024-09-10T18:54:37.384421+0000 mon.a (mon.0) 925 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396287+0000 mon.a (mon.0) 926 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396400+0000 mon.a (mon.0) 927 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396509+0000 mon.a (mon.0) 928 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396663+0000 mon.a (mon.0) 929 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396759+0000 mon.a (mon.0) 930 : audit [INF] from='client.24809 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396834+0000 mon.a (mon.0) 931 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.396922+0000 mon.a (mon.0) 932 : audit [INF] from='client.24821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397025+0000 mon.a (mon.0) 933 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397117+0000 mon.a (mon.0) 934 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397280+0000 mon.a (mon.0) 935 : audit [INF] from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397358+0000 mon.a (mon.0) 936 : audit [INF] from='client.25106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397450+0000 mon.a (mon.0) 937 : audit [INF] from='client.24866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397530+0000 mon.a (mon.0) 938 : audit [INF] from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397608+0000 mon.a (mon.0) 939 : audit [INF] from='client.24950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397713+0000 mon.a (mon.0) 940 : audit [INF] from='client.24911 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397805+0000 mon.a (mon.0) 941 : audit [INF] from='client.24857 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397882+0000 mon.a (mon.0) 942 : audit [INF] from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.397983+0000 mon.a (mon.0) 943 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.398259+0000 mon.a (mon.0) 944 : audit [INF] from='client.24917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.398364+0000 mon.a (mon.0) 945 : audit [INF] from='client.24956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.398451+0000 mon.a (mon.0) 946 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: cluster 2024-09-10T18:54:37.398559+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.409818+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:38 smithi083 bash[32381]: audit 2024-09-10T18:54:37.449956+0000 mon.a (mon.0) 948 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:39.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: cluster 2024-09-10T18:54:37.384421+0000 mon.a (mon.0) 925 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396287+0000 mon.a (mon.0) 926 : audit [INF] from='client.24796 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi083-77664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396400+0000 mon.a (mon.0) 927 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396509+0000 mon.a (mon.0) 928 : audit [INF] from='client.24839 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396663+0000 mon.a (mon.0) 929 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi083-77992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396759+0000 mon.a (mon.0) 930 : audit [INF] from='client.24809 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi083-77695-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396834+0000 mon.a (mon.0) 931 : audit [INF] from='client.24883 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi083-77986-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.396922+0000 mon.a (mon.0) 932 : audit [INF] from='client.24821 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi083-77715-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397025+0000 mon.a (mon.0) 933 : audit [INF] from='client.24791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi083-77656-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397117+0000 mon.a (mon.0) 934 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi083-77680-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.283 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397280+0000 mon.a (mon.0) 935 : audit [INF] from='client.25120 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397358+0000 mon.a (mon.0) 936 : audit [INF] from='client.25106 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397450+0000 mon.a (mon.0) 937 : audit [INF] from='client.24866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi083-77961-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397530+0000 mon.a (mon.0) 938 : audit [INF] from='client.24845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi083-77828-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397608+0000 mon.a (mon.0) 939 : audit [INF] from='client.24950 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397713+0000 mon.a (mon.0) 940 : audit [INF] from='client.24911 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397805+0000 mon.a (mon.0) 941 : audit [INF] from='client.24857 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi083-77943-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397882+0000 mon.a (mon.0) 942 : audit [INF] from='client.24929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.397983+0000 mon.a (mon.0) 943 : audit [INF] from='client.24851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi083-77907-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.398259+0000 mon.a (mon.0) 944 : audit [INF] from='client.24917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi083-78157-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.398364+0000 mon.a (mon.0) 945 : audit [INF] from='client.24956 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78166-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.398451+0000 mon.a (mon.0) 946 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: cluster 2024-09-10T18:54:37.398559+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-09-10T18:54:39.284 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.409818+0000 mon.c (mon.2) 123 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.285 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:38 smithi165 bash[22591]: audit 2024-09-10T18:54:37.449956+0000 mon.a (mon.0) 948 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:39.303 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:39.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:39.444 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:39.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:39.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:39.707 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:39.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:39.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:39.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:39.939 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 2 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (13923 ms) 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (13923 ms total) 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (13923 ms total) 2024-09-10T18:54:39.940 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-10T18:54:39.958 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (263 ms) 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (9 ms) 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-10T18:54:39.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (45 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (39 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (7 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (11 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-10T18:54:39.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (8 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (8 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (17 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (17 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (14 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-09-10T18:54:39.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (468 ms total) 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (13986 ms total) 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout:5... 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 50... 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 75... 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 100... 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: listed object 125... 2024-09-10T18:54:39.962 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: saw 150 objects 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[80176]: shutting down. 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: starting. 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: added 25 objects... 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: added half of the objects 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: added 50 objects... 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: added 50 objects 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[80177]: shutting down. 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.963 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******* SUCCESS ********** 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-10T18:54:39.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (38 ms) 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (47 ms) 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (13955 ms) 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (14040 ms total) 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (14040 ms total) 2024-09-10T18:54:39.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-10T18:54:40.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:40.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:40.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:40.175 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:40.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T18:54:40.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:40.313 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:40.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:40.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:37.638094+0000 mgr.y (mgr.24526) 196 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:37.837059+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:37.846979+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.149881+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.150081+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.200815+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.203092+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.254043+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.257386+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:54:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:38.469082+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.568348+0000 mgr.y (mgr.24526) 197 : cluster [DBG] pgmap v170: 932 pgs: 218 creating+peering, 582 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.678794+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.682999+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.800453+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.802977+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:38.952695+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:38.952922+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.83:0/4293857375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:38.953256+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:38.994082+0000 mon.a (mon.0) 950 : audit [INF] from='client.25142 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:38.994387+0000 mon.a (mon.0) 951 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.308241+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.83:0/3529073591' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:37.638094+0000 mgr.y (mgr.24526) 196 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:37.837059+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:37.846979+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.149881+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.150081+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.200815+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.203092+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.254043+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.257386+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:38.469082+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.568348+0000 mgr.y (mgr.24526) 197 : cluster [DBG] pgmap v170: 932 pgs: 218 creating+peering, 582 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.678794+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:54:40.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.682999+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.800453+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.802977+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:38.952695+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:38.952922+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.83:0/4293857375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:38.953256+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:38.994082+0000 mon.a (mon.0) 950 : audit [INF] from='client.25142 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:38.994387+0000 mon.a (mon.0) 951 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.308241+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.83:0/3529073591' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:40.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.470801+0000 mon.c (mon.2) 128 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.931808+0000 mon.a (mon.0) 952 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.932301+0000 mon.a (mon.0) 953 : audit [INF] from='client.25142 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.932415+0000 mon.a (mon.0) 954 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.933428+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.933589+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.947069+0000 mon.a (mon.0) 956 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[18629]: audit 2024-09-10T18:54:39.953548+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.83:0/2332893702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.470801+0000 mon.c (mon.2) 128 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.931808+0000 mon.a (mon.0) 952 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.932301+0000 mon.a (mon.0) 953 : audit [INF] from='client.25142 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.932415+0000 mon.a (mon.0) 954 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.933428+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.933589+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.947069+0000 mon.a (mon.0) 956 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:40 smithi083 bash[32381]: audit 2024-09-10T18:54:39.953548+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.83:0/2332893702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:40.579 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:37.638094+0000 mgr.y (mgr.24526) 196 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:37.837059+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:37.846979+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.149881+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.150081+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:40.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.200815+0000 osd.4 (osd.4) 131 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.203092+0000 osd.4 (osd.4) 132 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.254043+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.257386+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:38.469082+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.568348+0000 mgr.y (mgr.24526) 197 : cluster [DBG] pgmap v170: 932 pgs: 218 creating+peering, 582 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.678794+0000 osd.2 (osd.2) 61 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.682999+0000 osd.2 (osd.2) 62 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.800453+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.802977+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:38.952695+0000 mon.a (mon.0) 949 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:38.952922+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.83:0/4293857375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:38.953256+0000 mon.c (mon.2) 126 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:38.994082+0000 mon.a (mon.0) 950 : audit [INF] from='client.25142 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:38.994387+0000 mon.a (mon.0) 951 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.308241+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.83:0/3529073591' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.470801+0000 mon.c (mon.2) 128 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.931808+0000 mon.a (mon.0) 952 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.932301+0000 mon.a (mon.0) 953 : audit [INF] from='client.25142 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.932415+0000 mon.a (mon.0) 954 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.933428+0000 mon.b (mon.1) 42 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.933589+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.947069+0000 mon.a (mon.0) 956 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:40 smithi165 bash[22591]: audit 2024-09-10T18:54:39.953548+0000 mon.c (mon.2) 129 : audit [INF] from='client.? v1:172.21.15.83:0/2332893702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:40.638 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:40.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:40.718 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:40.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:40.856 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:40.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:40.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:40.997 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:41.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:41.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:41.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:41.268 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:41.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:41.402 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:41.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:41.527 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:41.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:41.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:41.667 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:41.771 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:41.804 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:41.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:41.919 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (54 ms) 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (55 ms) 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (13937 ms) 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-10T18:54:41.920 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:38.952922+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/4293857375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:38.953256+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:38.994082+0000 mon.a [INF] from='client.25142 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:38.994387+0000 mon.a [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.931808+0000 mon.a [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi083-78004-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.932301+0000 mon.a [INF] from='client.25142 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi083-77640-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.932415+0000 mon.a [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.933428+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.947069+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.953548+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2332893702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:39.964045+0000 mon.a [INF] from='client.25247 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:41.921 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.493339+0000 client.admin [INF] onexx 2024-09-10T18:54:41.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:41.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.238997+0000 osd.5 (osd.5) 109 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.242038+0000 osd.5 (osd.5) 110 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.635192+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.637750+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.798193+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.d scrub starts 2024-09-10T18:54:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:39.804046+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.d scrub ok 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:39.964045+0000 mon.a (mon.0) 957 : audit [INF] from='client.25247 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.221760+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.224951+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.479246+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.493339+0000 client.admin (client.14901) 0 : cluster [INF] onexx 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.494281+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.494341+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.494638+0000 mon.a (mon.0) 958 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.496781+0000 mon.a (mon.0) 959 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.498271+0000 mon.a (mon.0) 960 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.498447+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.498971+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.499422+0000 mon.a (mon.0) 961 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.500327+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.501333+0000 mon.a (mon.0) 962 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.960030+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.960443+0000 mon.a (mon.0) 964 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[18629]: audit 2024-09-10T18:54:40.977004+0000 mon.a (mon.0) 965 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.238997+0000 osd.5 (osd.5) 109 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.242038+0000 osd.5 (osd.5) 110 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:54:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.635192+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.637750+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.798193+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.d scrub starts 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:39.804046+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.d scrub ok 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:39.964045+0000 mon.a (mon.0) 957 : audit [INF] from='client.25247 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.221760+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.224951+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.479246+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.493339+0000 client.admin (client.14901) 0 : cluster [INF] onexx 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.494281+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.494341+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.494638+0000 mon.a (mon.0) 958 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.496781+0000 mon.a (mon.0) 959 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.498271+0000 mon.a (mon.0) 960 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.498447+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.498971+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.499422+0000 mon.a (mon.0) 961 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.500327+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.501333+0000 mon.a (mon.0) 962 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.960030+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.960443+0000 mon.a (mon.0) 964 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:41 smithi083 bash[32381]: audit 2024-09-10T18:54:40.977004+0000 mon.a (mon.0) 965 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:42.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:42.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:42.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:54:42.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:42.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.238997+0000 osd.5 (osd.5) 109 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.242038+0000 osd.5 (osd.5) 110 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.635192+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.637750+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.798193+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.d scrub starts 2024-09-10T18:54:42.356 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:39.804046+0000 osd.7 (osd.7) 120 : cluster [DBG] 3.d scrub ok 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:39.964045+0000 mon.a (mon.0) 957 : audit [INF] from='client.25247 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.221760+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.224951+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.479246+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.493339+0000 client.admin (client.14901) 0 : cluster [INF] onexx 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.494281+0000 mon.b (mon.1) 43 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.494341+0000 mon.c (mon.2) 131 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.494638+0000 mon.a (mon.0) 958 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.496781+0000 mon.a (mon.0) 959 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.498271+0000 mon.a (mon.0) 960 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.498447+0000 mon.b (mon.1) 44 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.498971+0000 mon.c (mon.2) 132 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.499422+0000 mon.a (mon.0) 961 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:42.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.500327+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.501333+0000 mon.a (mon.0) 962 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:42.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.960030+0000 mon.a (mon.0) 963 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:42.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.960443+0000 mon.a (mon.0) 964 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:42.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:41 smithi165 bash[22591]: audit 2024-09-10T18:54:40.977004+0000 mon.a (mon.0) 965 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:42.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:42.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:42.503 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:42.606 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:42.631 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:42.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:42.746 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:42] "GET /metrics HTTP/1.1" 200 224786 "" "Prometheus/2.43.0" 2024-09-10T18:54:42.766 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:42.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:42.894 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:42.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:42.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:43.030 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:43.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:43.169 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:43.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:43.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:43.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:43.416 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.519546+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.522445+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.572697+0000 mgr.y (mgr.24526) 198 : cluster [DBG] pgmap v172: 652 pgs: 96 creating+peering, 556 active+clean; 594 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 39 KiB/s wr, 520 op/s 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.845662+0000 osd.7 (osd.7) 121 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:40.848286+0000 osd.7 (osd.7) 122 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:40.977298+0000 mon.a (mon.0) 966 : audit [INF] from='client.25247 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:40.977976+0000 mon.a (mon.0) 967 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:40.982831+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:40.998521+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.007634+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.078448+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.109180+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.112528+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.184331+0000 osd.5 (osd.5) 111 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.188034+0000 osd.5 (osd.5) 112 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.191475+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.194448+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:54:43.418 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.470131+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.473512+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.480909+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.700702+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.702310+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.703425+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.704589+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.704722+0000 mgr.y (mgr.24526) 199 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.705678+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.706132+0000 mgr.y (mgr.24526) 200 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.706796+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.707349+0000 mgr.y (mgr.24526) 201 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.708027+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.708420+0000 mgr.y (mgr.24526) 202 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.709044+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.709528+0000 mgr.y (mgr.24526) 203 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.710051+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.710710+0000 mgr.y (mgr.24526) 204 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.711098+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.711909+0000 mgr.y (mgr.24526) 205 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.712919+0000 mgr.y (mgr.24526) 206 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.713909+0000 mgr.y (mgr.24526) 207 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.714980+0000 mgr.y (mgr.24526) 208 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.924897+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.925457+0000 client.admin (client.14901) 0 : cluster [INF] twoxx 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.925672+0000 mon.a (mon.0) 971 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.927842+0000 mon.a (mon.0) 972 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.928984+0000 mon.a (mon.0) 973 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.931965+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.932232+0000 mon.a (mon.0) 975 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.932580+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.932786+0000 mon.a (mon.0) 977 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.421 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.937925+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:41.940526+0000 mon.a (mon.0) 978 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:42.387709+0000 mon.a (mon.0) 979 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:42.388004+0000 mon.a (mon.0) 980 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:43 smithi165 bash[22591]: audit 2024-09-10T18:54:42.389118+0000 mon.a (mon.0) 981 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.432 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:43.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.519546+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.522445+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.572697+0000 mgr.y (mgr.24526) 198 : cluster [DBG] pgmap v172: 652 pgs: 96 creating+peering, 556 active+clean; 594 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 39 KiB/s wr, 520 op/s 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.845662+0000 osd.7 (osd.7) 121 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:40.848286+0000 osd.7 (osd.7) 122 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:40.977298+0000 mon.a (mon.0) 966 : audit [INF] from='client.25247 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:40.977976+0000 mon.a (mon.0) 967 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:40.982831+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:40.998521+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.007634+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.078448+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.109180+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.112528+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.184331+0000 osd.5 (osd.5) 111 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.188034+0000 osd.5 (osd.5) 112 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.191475+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.194448+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.470131+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-10T18:54:43.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.519546+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.522445+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.572697+0000 mgr.y (mgr.24526) 198 : cluster [DBG] pgmap v172: 652 pgs: 96 creating+peering, 556 active+clean; 594 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 39 KiB/s wr, 520 op/s 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.845662+0000 osd.7 (osd.7) 121 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:40.848286+0000 osd.7 (osd.7) 122 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:40.977298+0000 mon.a (mon.0) 966 : audit [INF] from='client.25247 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:40.977976+0000 mon.a (mon.0) 967 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:40.982831+0000 mon.b (mon.1) 45 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:40.998521+0000 mon.c (mon.2) 134 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.007634+0000 mon.c (mon.2) 135 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.078448+0000 mon.b (mon.1) 46 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.109180+0000 mon.b (mon.1) 47 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.112528+0000 mon.b (mon.1) 48 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.184331+0000 osd.5 (osd.5) 111 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:54:43.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.188034+0000 osd.5 (osd.5) 112 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:54:43.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.191475+0000 osd.4 (osd.4) 133 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:54:43.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.194448+0000 osd.4 (osd.4) 134 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:54:43.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.470131+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-09-10T18:54:43.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.473512+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:43.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.480909+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:43.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.700702+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.702310+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.703425+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.704589+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.704722+0000 mgr.y (mgr.24526) 199 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.705678+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.706132+0000 mgr.y (mgr.24526) 200 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.706796+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.546 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.707349+0000 mgr.y (mgr.24526) 201 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.708027+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.708420+0000 mgr.y (mgr.24526) 202 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.709044+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.709528+0000 mgr.y (mgr.24526) 203 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.710051+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.710710+0000 mgr.y (mgr.24526) 204 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.473512+0000 mon.a (mon.0) 969 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:43.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.480909+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:43.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.700702+0000 mon.b (mon.1) 49 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.702310+0000 mon.b (mon.1) 50 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.703425+0000 mon.b (mon.1) 51 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.704589+0000 mon.b (mon.1) 52 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.704722+0000 mgr.y (mgr.24526) 199 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.705678+0000 mon.b (mon.1) 53 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.549 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.706132+0000 mgr.y (mgr.24526) 200 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.706796+0000 mon.b (mon.1) 54 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.707349+0000 mgr.y (mgr.24526) 201 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.708027+0000 mon.b (mon.1) 55 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.708420+0000 mgr.y (mgr.24526) 202 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.709044+0000 mon.b (mon.1) 56 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.709528+0000 mgr.y (mgr.24526) 203 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:54:43.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.710051+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.551 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.710710+0000 mgr.y (mgr.24526) 204 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.711098+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.711909+0000 mgr.y (mgr.24526) 205 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.712919+0000 mgr.y (mgr.24526) 206 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.713909+0000 mgr.y (mgr.24526) 207 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.714980+0000 mgr.y (mgr.24526) 208 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.552 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.924897+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T18:54:43.553 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.925457+0000 client.admin (client.14901) 0 : cluster [INF] twoxx 2024-09-10T18:54:43.553 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.925672+0000 mon.a (mon.0) 971 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:43.553 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.927842+0000 mon.a (mon.0) 972 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.553 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.928984+0000 mon.a (mon.0) 973 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.553 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.931965+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.932232+0000 mon.a (mon.0) 975 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.932580+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.932786+0000 mon.a (mon.0) 977 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.937925+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:41.940526+0000 mon.a (mon.0) 978 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:42.387709+0000 mon.a (mon.0) 979 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.554 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:42.388004+0000 mon.a (mon.0) 980 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.555 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[32381]: audit 2024-09-10T18:54:42.389118+0000 mon.a (mon.0) 981 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.711098+0000 mon.b (mon.1) 58 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.711909+0000 mgr.y (mgr.24526) 205 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:54:43.555 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.712919+0000 mgr.y (mgr.24526) 206 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.713909+0000 mgr.y (mgr.24526) 207 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.714980+0000 mgr.y (mgr.24526) 208 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.924897+0000 mon.a (mon.0) 970 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.925457+0000 client.admin (client.14901) 0 : cluster [INF] twoxx 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.925672+0000 mon.a (mon.0) 971 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.927842+0000 mon.a (mon.0) 972 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:43.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.928984+0000 mon.a (mon.0) 973 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.931965+0000 mon.a (mon.0) 974 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.932232+0000 mon.a (mon.0) 975 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:43.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.932580+0000 mon.a (mon.0) 976 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T18:54:43.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.932786+0000 mon.a (mon.0) 977 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:43.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.937925+0000 mon.b (mon.1) 59 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:41.940526+0000 mon.a (mon.0) 978 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:42.387709+0000 mon.a (mon.0) 979 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:43.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:42.388004+0000 mon.a (mon.0) 980 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:43.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:43 smithi083 bash[18629]: audit 2024-09-10T18:54:42.389118+0000 mon.a (mon.0) 981 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:43.567 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:43.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:43.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:43.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:43.830 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:43.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:43.967 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:43.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:44.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:44.101 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:44.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:44.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:44.334 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:44.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.367 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:44.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:44.505 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:44.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.611 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:44.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:44.771 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:44.778 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.494281+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Lib open_pools_parallel: process_1_[78317]: starting. 2024-09-10T18:54:44.778 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: creating pool ceph_test_rados_open_pools_parallel.smithi083-78283 2024-09-10T18:54:44.778 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: created object 0... 2024-09-10T18:54:44.778 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: created object 25... 2024-09-10T18:54:44.778 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: created object 49... 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: finishing. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[78317]: shutting down. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[78318]: starting. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[78318]: rados_pool_create. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[78318]: rados_ioctx_create. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[78318]: shutting down. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: starting. 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: creating pool ceph_test_rados_open_pools_parallel.smithi083-78283 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: created object 0... 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: created object 25... 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: created object 49... 2024-09-10T18:54:44.779 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: finishing. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[80360]: shutting down. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[80361]: starting. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[80361]: rados_pool_create. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[80361]: rados_ioctx_create. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[80361]: shutting down. 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-09-10T18:54:44.780 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.557452+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:41.561199+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.412743+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.482957+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:42.573764+0000 mgr.y (mgr.24526) 209 : cluster [DBG] pgmap v175: 580 pgs: 160 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 22 KiB/s wr, 485 op/s 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.848259+0000 mon.b (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:44.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.850027+0000 mon.a (mon.0) 982 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.850116+0000 mon.a (mon.0) 983 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.850155+0000 mon.a (mon.0) 984 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.850252+0000 mon.a (mon.0) 985 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.854278+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:42.854617+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.271522+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.271931+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.272299+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.274224+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.297764+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.298621+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.299585+0000 mon.a (mon.0) 989 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.302591+0000 mon.a (mon.0) 990 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.302787+0000 mon.a (mon.0) 991 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.313730+0000 osd.1 (osd.1) 117 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:54:44.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.316169+0000 osd.1 (osd.1) 118 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:54:44.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:44 smithi165 bash[22591]: audit 2024-09-10T18:54:43.484397+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.557452+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:41.561199+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.412743+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.482957+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:42.573764+0000 mgr.y (mgr.24526) 209 : cluster [DBG] pgmap v175: 580 pgs: 160 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 22 KiB/s wr, 485 op/s 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.848259+0000 mon.b (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.850027+0000 mon.a (mon.0) 982 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.888 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.850116+0000 mon.a (mon.0) 983 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.850155+0000 mon.a (mon.0) 984 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.850252+0000 mon.a (mon.0) 985 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.854278+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:42.854617+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.271522+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.271931+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.557452+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.e scrub starts 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:41.561199+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.e scrub ok 2024-09-10T18:54:44.889 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.412743+0000 mon.b (mon.1) 60 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.482957+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:42.573764+0000 mgr.y (mgr.24526) 209 : cluster [DBG] pgmap v175: 580 pgs: 160 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 22 KiB/s wr, 485 op/s 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.848259+0000 mon.b (mon.1) 61 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.850027+0000 mon.a (mon.0) 982 : audit [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.850116+0000 mon.a (mon.0) 983 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.850155+0000 mon.a (mon.0) 984 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.850252+0000 mon.a (mon.0) 985 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.854278+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:42.854617+0000 mon.c (mon.2) 139 : audit [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.271522+0000 mon.b (mon.1) 62 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.271931+0000 mon.b (mon.1) 63 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.272299+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.274224+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-10T18:54:44.890 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.297764+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.298621+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.299585+0000 mon.a (mon.0) 989 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.302591+0000 mon.a (mon.0) 990 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.302787+0000 mon.a (mon.0) 991 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.313730+0000 osd.1 (osd.1) 117 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.316169+0000 osd.1 (osd.1) 118 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[18629]: audit 2024-09-10T18:54:43.484397+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.272299+0000 mon.b (mon.1) 64 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.274224+0000 mon.a (mon.0) 986 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.297764+0000 mon.a (mon.0) 987 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.298621+0000 mon.a (mon.0) 988 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.299585+0000 mon.a (mon.0) 989 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:44.891 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.302591+0000 mon.a (mon.0) 990 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:44.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.302787+0000 mon.a (mon.0) 991 : audit [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:44.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.313730+0000 osd.1 (osd.1) 117 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:54:44.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.316169+0000 osd.1 (osd.1) 118 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:54:44.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:44 smithi083 bash[32381]: audit 2024-09-10T18:54:43.484397+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:44.909 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:44.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:44.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:45.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:45.046 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:45.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:45.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:45.308 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: starting. 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: creating pool ceph_test_rados_delete_pools_parallel.smithi083-78300 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: created object 0... 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: created object 25... 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: created object 49... 2024-09-10T18:54:45.338 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: finishing. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[78341]: shutting down. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[78343]: starting. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[78343]: deleting pool ceph_test_rados_delete_pools_parallel.smithi083-78300 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[78343]: shutting down. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: starting. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: creating pool ceph_test_rados_delete_pools_parallel.smithi083-78300 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: created object 0... 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: created object 25... 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: created object 49... 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: finishing. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[80520]: shutting down. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: starting. 2024-09-10T18:54:45.339 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: listing objects. 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: listed object 0... 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: listed object 25... 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: saw 50 objects 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[80522]: shutting down. 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[80521]: starting. 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[80521]: deleting pool ceph_test_rados_delete_pools_parallel.smithi083-78300 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[80521]: shutting down. 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-09-10T18:54:45.340 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout:RadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.494341+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.494638+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.496781+0000 mon.a [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.498271+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.498447+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.498971+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.499422+0000 mon.a [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.500327+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.501333+0000 mon.a [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-77961-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.960030+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:45.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.960443+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:40.977004+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.412743+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.848259+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.850027+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/3849935910' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:45.444 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:45.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:45.575 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:45.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:45.708 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.850116+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.850155+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-77656-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.850252+0000 mon.a [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi083-77648-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.854278+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:42.854617+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2875213838' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:45.805 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.271522+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.271931+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.272299+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.297764+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.298621+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.299585+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.302591+0000 mon.a [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.302787+0000 mon.a [INF] from='client.25136 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.749861+0000 mon.a [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:43.760458+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.183935+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.184612+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.185325+0000 mon.a [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:45.806 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.185568+0000 mon.a [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.186852+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.187493+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/1506777304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.678462+0000 mon.a [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.678836+0000 mon.a [INF] from='client.25340 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.679202+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.681552+0000 mon.a [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.681672+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.682276+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.683689+0000 mon.a [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.683854+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.684180+0000 mon.a [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.684459+0000 mon.a [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.807 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.689546+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.689866+0000 mon.a [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.690169+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.758840+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.759071+0000 mon.a [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.759184+0000 mon.a [INF] from='client.25340 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:44.778434+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/1086960894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.206792+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.207453+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208045+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208419+0000 mon.a [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208582+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208597+0000 mon.a [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:45.808 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208862+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:45.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:45.857 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:45.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:45.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:45.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:42.546149+0000 osd.5 (osd.5) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:42.546311+0000 osd.5 (osd.5) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:42.662533+0000 osd.6 (osd.6) 81 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:42.665732+0000 osd.6 (osd.6) 82 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.281137+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.293262+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:54:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.641262+0000 osd.6 (osd.6) 83 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:43.645955+0000 osd.6 (osd.6) 84 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:43.749861+0000 mon.a (mon.0) 992 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:43.760458+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.183935+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.184612+0000 mon.a (mon.0) 993 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.185325+0000 mon.a (mon.0) 994 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.185568+0000 mon.a (mon.0) 995 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.186852+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.187493+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.83:0/1506777304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.485957+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:46.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.594654+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:42.546149+0000 osd.5 (osd.5) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:42.546311+0000 osd.5 (osd.5) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:42.662533+0000 osd.6 (osd.6) 81 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:42.665732+0000 osd.6 (osd.6) 82 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.281137+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.293262+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.641262+0000 osd.6 (osd.6) 83 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:43.645955+0000 osd.6 (osd.6) 84 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:43.749861+0000 mon.a (mon.0) 992 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:43.760458+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.183935+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.184612+0000 mon.a (mon.0) 993 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.185325+0000 mon.a (mon.0) 994 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.185568+0000 mon.a (mon.0) 995 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.186852+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.187493+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.83:0/1506777304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.485957+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.594654+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.678462+0000 mon.a (mon.0) 997 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.678836+0000 mon.a (mon.0) 998 : audit [INF] from='client.25340 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.679202+0000 mon.a (mon.0) 999 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.681552+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.681672+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.682276+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.683689+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.683854+0000 mon.a (mon.0) 1002 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.684180+0000 mon.a (mon.0) 1003 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.684459+0000 mon.a (mon.0) 1004 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.685361+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.689546+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.689866+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.690169+0000 mon.a (mon.0) 1007 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.678462+0000 mon.a (mon.0) 997 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.678836+0000 mon.a (mon.0) 998 : audit [INF] from='client.25340 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.679202+0000 mon.a (mon.0) 999 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.681552+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.681672+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.682276+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.683689+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.683854+0000 mon.a (mon.0) 1002 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.684180+0000 mon.a (mon.0) 1003 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.684459+0000 mon.a (mon.0) 1004 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.685361+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.689546+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.689866+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.690169+0000 mon.a (mon.0) 1007 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.758840+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:46.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.759071+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.759184+0000 mon.a (mon.0) 1010 : audit [INF] from='client.25340 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:44.778434+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.83:0/1086960894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.206792+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.207453+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208045+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208419+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208582+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208597+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208862+0000 mon.a (mon.0) 1013 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.208981+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.047 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.209105+0000 mon.a (mon.0) 1015 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.758840+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.759071+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.759184+0000 mon.a (mon.0) 1010 : audit [INF] from='client.25340 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:44.778434+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.83:0/1086960894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.206792+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.207453+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208045+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208419+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208582+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208597+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208862+0000 mon.a (mon.0) 1013 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:46.048 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.208981+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.209105+0000 mon.a (mon.0) 1015 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: cluster 2024-09-10T18:54:45.209210+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.212199+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25346 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.228653+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[32381]: audit 2024-09-10T18:54:45.229415+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: cluster 2024-09-10T18:54:45.209210+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.212199+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25346 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.228653+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T18:54:46.049 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:45 smithi083 bash[18629]: audit 2024-09-10T18:54:45.229415+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:46.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:42.546149+0000 osd.5 (osd.5) 113 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:54:46.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:42.546311+0000 osd.5 (osd.5) 114 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:42.662533+0000 osd.6 (osd.6) 81 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:42.665732+0000 osd.6 (osd.6) 82 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.281137+0000 osd.4 (osd.4) 135 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.293262+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.641262+0000 osd.6 (osd.6) 83 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:43.645955+0000 osd.6 (osd.6) 84 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:43.749861+0000 mon.a (mon.0) 992 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-77961-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:43.760458+0000 mon.b (mon.1) 65 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.183935+0000 mon.b (mon.1) 66 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.184612+0000 mon.a (mon.0) 993 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.185325+0000 mon.a (mon.0) 994 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.185568+0000 mon.a (mon.0) 995 : audit [INF] from='client.25136 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi083-77648-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T18:54:46.110 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.186852+0000 mon.c (mon.2) 141 : audit [INF] from='client.? v1:172.21.15.83:0/2936416874' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.187493+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.83:0/1506777304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.485957+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.594654+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.678462+0000 mon.a (mon.0) 997 : audit [INF] from='client.24875 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]: dispatch 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.678836+0000 mon.a (mon.0) 998 : audit [INF] from='client.25340 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.679202+0000 mon.a (mon.0) 999 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.681552+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.681672+0000 mon.b (mon.1) 67 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.682276+0000 mon.b (mon.1) 68 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.683689+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.683854+0000 mon.a (mon.0) 1002 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.684180+0000 mon.a (mon.0) 1003 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.684459+0000 mon.a (mon.0) 1004 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.685361+0000 mon.c (mon.2) 144 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.689546+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.689866+0000 mon.a (mon.0) 1006 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.690169+0000 mon.a (mon.0) 1007 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.758840+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-77986-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.759071+0000 mon.a (mon.0) 1009 : audit [INF] from='client.24875 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi083-78004-1"}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.759184+0000 mon.a (mon.0) 1010 : audit [INF] from='client.25340 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:44.778434+0000 mon.c (mon.2) 145 : audit [INF] from='client.? v1:172.21.15.83:0/1086960894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.206792+0000 mon.b (mon.1) 69 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.207453+0000 mon.b (mon.1) 70 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208045+0000 mon.b (mon.1) 71 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208419+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208582+0000 mon.b (mon.1) 72 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208597+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208862+0000 mon.a (mon.0) 1013 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.208981+0000 mon.a (mon.0) 1014 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.209105+0000 mon.a (mon.0) 1015 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: cluster 2024-09-10T18:54:45.209210+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-09-10T18:54:46.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.212199+0000 mon.a (mon.0) 1017 : audit [INF] from='client.25346 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.228653+0000 mon.b (mon.1) 73 : audit [DBG] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T18:54:46.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:45 smithi165 bash[22591]: audit 2024-09-10T18:54:45.229415+0000 mon.b (mon.1) 74 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:46.129 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:46.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:46.267 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:46.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:46.405 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:46.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:46.535 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:46.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:46.644 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:46.671 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:46.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:46.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:46.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:46.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:46.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:47.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.247085+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.250313+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.574894+0000 mgr.y (mgr.24526) 210 : cluster [DBG] pgmap v177: 652 pgs: 232 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 19 KiB/s wr, 428 op/s 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.924715+0000 osd.7 (osd.7) 123 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:44.930592+0000 osd.7 (osd.7) 124 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:45.298336+0000 client.admin (client.14901) 0 : cluster [INF] threexx 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:45.487595+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:45.796022+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:45.810509+0000 mon.a (mon.0) 1019 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:45.810621+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25346 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:45.812151+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:45.814672+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:45.853475+0000 osd.0 (osd.0) 125 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:54:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:45.922248+0000 osd.0 (osd.0) 126 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.259312+0000 mon.a (mon.0) 1022 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.296368+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.299616+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.489243+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.676427+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.247085+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.250313+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.574894+0000 mgr.y (mgr.24526) 210 : cluster [DBG] pgmap v177: 652 pgs: 232 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 19 KiB/s wr, 428 op/s 2024-09-10T18:54:47.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.924715+0000 osd.7 (osd.7) 123 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:44.930592+0000 osd.7 (osd.7) 124 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:45.298336+0000 client.admin (client.14901) 0 : cluster [INF] threexx 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:45.487595+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:45.796022+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:45.810509+0000 mon.a (mon.0) 1019 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:45.810621+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25346 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:45.812151+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:45.814672+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:45.853475+0000 osd.0 (osd.0) 125 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:45.922248+0000 osd.0 (osd.0) 126 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.259312+0000 mon.a (mon.0) 1022 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.296368+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.299616+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:54:47.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.489243+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.676427+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.676796+0000 mon.a (mon.0) 1024 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.676971+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.677266+0000 mon.a (mon.0) 1026 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[18629]: audit 2024-09-10T18:54:46.677554+0000 mon.a (mon.0) 1027 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.676796+0000 mon.a (mon.0) 1024 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.676971+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:47.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.677266+0000 mon.a (mon.0) 1026 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:47.046 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:46 smithi083 bash[32381]: audit 2024-09-10T18:54:46.677554+0000 mon.a (mon.0) 1027 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.247085+0000 osd.4 (osd.4) 137 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.250313+0000 osd.4 (osd.4) 138 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.574894+0000 mgr.y (mgr.24526) 210 : cluster [DBG] pgmap v177: 652 pgs: 232 unknown, 42 creating+peering, 378 active+clean; 586 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.6 KiB/s rd, 19 KiB/s wr, 428 op/s 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.924715+0000 osd.7 (osd.7) 123 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:44.930592+0000 osd.7 (osd.7) 124 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:45.298336+0000 client.admin (client.14901) 0 : cluster [INF] threexx 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:45.487595+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:45.796022+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T18:54:47.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:45.810509+0000 mon.a (mon.0) 1019 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:45.810621+0000 mon.a (mon.0) 1020 : audit [INF] from='client.25346 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:45.812151+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:45.814672+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:45.853475+0000 osd.0 (osd.0) 125 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:45.922248+0000 osd.0 (osd.0) 126 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.259312+0000 mon.a (mon.0) 1022 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.296368+0000 osd.1 (osd.1) 119 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.299616+0000 osd.1 (osd.1) 120 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.489243+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.676427+0000 mon.a (mon.0) 1023 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.676796+0000 mon.a (mon.0) 1024 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:47.060 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.676971+0000 mon.a (mon.0) 1025 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:47.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.677266+0000 mon.a (mon.0) 1026 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:47.061 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:46 smithi165 bash[22591]: audit 2024-09-10T18:54:46.677554+0000 mon.a (mon.0) 1027 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:47.076 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:47.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:47.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:47.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:47.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:47.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:47.477 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:47.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:47.611 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:47.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:47.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:47.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:47.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:47.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:47.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:47.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:48.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:48.157 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:48.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.296 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:48.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:48.429 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:48.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:48.616 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:54:48.616 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:54:48.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:48.700 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:48.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.827552+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.833244+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.833548+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.83:0/2991572135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.833798+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.83:0/1210858150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.895406+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:46.931477+0000 mon.c (mon.2) 153 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:47.272351+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:47.274686+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:47.386449+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:47.387558+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.447927+0000 osd.2 (osd.2) 67 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:47.490758+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.103120+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2"}]: dispatch 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.103792+0000 mon.a (mon.0) 1030 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.103968+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-10T18:54:48.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.104063+0000 mon.b (mon.1) 77 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.104342+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.105357+0000 client.admin (client.14901) 0 : cluster [INF] fourxx 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.105951+0000 mon.a (mon.0) 1033 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.106952+0000 mon.a (mon.0) 1034 : cluster [WRN] pool 'PoolQuotaPP_smithi083-77648-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.110519+0000 mon.a (mon.0) 1035 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.114031+0000 mon.a (mon.0) 1036 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[32381]: audit 2024-09-10T18:54:48.114426+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.827552+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.833244+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.833548+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.83:0/2991572135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.833798+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.83:0/1210858150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.895406+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:46.931477+0000 mon.c (mon.2) 153 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:47.272351+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:47.274686+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:47.386449+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:47.387558+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.447927+0000 osd.2 (osd.2) 67 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:47.490758+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.103120+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2"}]: dispatch 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.103792+0000 mon.a (mon.0) 1030 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.103968+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.104063+0000 mon.b (mon.1) 77 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.104342+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.105357+0000 client.admin (client.14901) 0 : cluster [INF] fourxx 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.105951+0000 mon.a (mon.0) 1033 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.106952+0000 mon.a (mon.0) 1034 : cluster [WRN] pool 'PoolQuotaPP_smithi083-77648-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.110519+0000 mon.a (mon.0) 1035 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.114031+0000 mon.a (mon.0) 1036 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:48 smithi083 bash[18629]: audit 2024-09-10T18:54:48.114426+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T18:54:48.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.827552+0000 mon.a (mon.0) 1028 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.833244+0000 mon.c (mon.2) 149 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.833548+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.83:0/2991572135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.833798+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.83:0/1210858150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.895406+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:46.931477+0000 mon.c (mon.2) 153 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:47.272351+0000 mon.b (mon.1) 75 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:47.274686+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:47.386449+0000 mon.c (mon.2) 154 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:47.387558+0000 mon.c (mon.2) 155 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.447927+0000 osd.2 (osd.2) 67 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:47.490758+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.103120+0000 mon.b (mon.1) 76 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2"}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.103792+0000 mon.a (mon.0) 1030 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.103968+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.104063+0000 mon.b (mon.1) 77 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.104342+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.105357+0000 client.admin (client.14901) 0 : cluster [INF] fourxx 2024-09-10T18:54:48.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.105951+0000 mon.a (mon.0) 1033 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:48.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.106952+0000 mon.a (mon.0) 1034 : cluster [WRN] pool 'PoolQuotaPP_smithi083-77648-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T18:54:48.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.110519+0000 mon.a (mon.0) 1035 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T18:54:48.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.114031+0000 mon.a (mon.0) 1036 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:48.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:48 smithi165 bash[22591]: audit 2024-09-10T18:54:48.114426+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T18:54:48.839 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:48.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:48.976 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:48.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:49.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:49.112 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:49.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:49.239 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:49.315 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.208981+0000 mon.a [INF] from='client.25336 ' api_io_pp: Running main() from gmock_main.cc 2024-09-10T18:54:49.315 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-10T18:54:49.315 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: seed 77664 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (320 ms) 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (12 ms) 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (8 ms) 2024-09-10T18:54:49.316 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (6 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (8 ms) 2024-09-10T18:54:49.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (20 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (7 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (15 ms) 2024-09-10T18:54:49.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (9 ms) 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (451 ms total) 2024-09-10T18:54:49.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1159 ms) 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (15 ms) 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (7 ms) 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-10T18:54:49.320 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (7 ms) 2024-09-10T18:54:49.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:49.373 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:49.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:49.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:49.508 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:49.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:49.644 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:49.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:49.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:49.748 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.262096+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:54:49.748 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.264842+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.438891+0000 osd.3 (osd.3) 137 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.488743+0000 osd.3 (osd.3) 138 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:46.575926+0000 mgr.y (mgr.24526) 211 : cluster [DBG] pgmap v180: 604 pgs: 40 creating+peering, 40 unknown, 524 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 54 op/s 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.176366+0000 osd.5 (osd.5) 115 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.180187+0000 osd.5 (osd.5) 116 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.233630+0000 osd.4 (osd.4) 141 : cluster [DBG] 2.f scrub starts 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.237277+0000 osd.4 (osd.4) 142 : cluster [DBG] 2.f scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.323551+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.a scrub starts 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.326275+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.a scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.390855+0000 mgr.y (mgr.24526) 212 : cluster [DBG] pgmap v182: 660 pgs: 40 creating+peering, 104 unknown, 516 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 41 MiB/s wr, 60 op/s 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:47.505762+0000 osd.2 (osd.2) 68 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:47.646524+0000 mgr.y (mgr.24526) 213 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:49.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.121133+0000 mon.a (mon.0) 1038 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.121251+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.121394+0000 mon.a (mon.0) 1040 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.121982+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.124217+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.125653+0000 mon.a (mon.0) 1043 : audit [INF] from='client.25370 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.262096+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.264842+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:54:49.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.438891+0000 osd.3 (osd.3) 137 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.488743+0000 osd.3 (osd.3) 138 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:46.575926+0000 mgr.y (mgr.24526) 211 : cluster [DBG] pgmap v180: 604 pgs: 40 creating+peering, 40 unknown, 524 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 54 op/s 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.176366+0000 osd.5 (osd.5) 115 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.180187+0000 osd.5 (osd.5) 116 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.233630+0000 osd.4 (osd.4) 141 : cluster [DBG] 2.f scrub starts 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.237277+0000 osd.4 (osd.4) 142 : cluster [DBG] 2.f scrub ok 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.323551+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.a scrub starts 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.326275+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.a scrub ok 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.390855+0000 mgr.y (mgr.24526) 212 : cluster [DBG] pgmap v182: 660 pgs: 40 creating+peering, 104 unknown, 516 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 41 MiB/s wr, 60 op/s 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:47.505762+0000 osd.2 (osd.2) 68 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:47.646524+0000 mgr.y (mgr.24526) 213 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:49.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.121133+0000 mon.a (mon.0) 1038 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.121251+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.121394+0000 mon.a (mon.0) 1040 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.121982+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.124217+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.125653+0000 mon.a (mon.0) 1043 : audit [INF] from='client.25370 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.130689+0000 mon.a (mon.0) 1044 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.374674+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:48.377946+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.492153+0000 mon.c (mon.2) 157 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:49.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.578049+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.580091+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.580428+0000 mon.a (mon.0) 1047 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:48.585153+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.116789+0000 mon.a (mon.0) 1049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.127884+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.128296+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.128543+0000 mon.a (mon.0) 1052 : audit [INF] from='client.25370 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.128874+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.129079+0000 mon.a (mon.0) 1054 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.129437+0000 mon.a (mon.0) 1055 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.131090+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-10T18:54:49.753 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.133630+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.130689+0000 mon.a (mon.0) 1044 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.374674+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:48.377946+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.492153+0000 mon.c (mon.2) 157 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.578049+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.580091+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.580428+0000 mon.a (mon.0) 1047 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:48.585153+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.116789+0000 mon.a (mon.0) 1049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:49.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.127884+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.128296+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.128543+0000 mon.a (mon.0) 1052 : audit [INF] from='client.25370 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.128874+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.129079+0000 mon.a (mon.0) 1054 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.129437+0000 mon.a (mon.0) 1055 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.131090+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.133630+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.134574+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:49.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.136666+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.137287+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.138108+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.138333+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.155083+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.155278+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.155555+0000 mon.a (mon.0) 1060 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[18629]: audit 2024-09-10T18:54:49.155797+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.134574+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.136666+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.137287+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.138108+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.138333+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.155083+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.155278+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.155555+0000 mon.a (mon.0) 1060 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:49 smithi083 bash[32381]: audit 2024-09-10T18:54:49.155797+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:49.781 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:49.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:49.919 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:49.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:50.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:50.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.262096+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.2 scrub starts 2024-09-10T18:54:50.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.264842+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.2 scrub ok 2024-09-10T18:54:50.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.438891+0000 osd.3 (osd.3) 137 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:54:50.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.488743+0000 osd.3 (osd.3) 138 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:54:50.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:46.575926+0000 mgr.y (mgr.24526) 211 : cluster [DBG] pgmap v180: 604 pgs: 40 creating+peering, 40 unknown, 524 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 54 op/s 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.176366+0000 osd.5 (osd.5) 115 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.180187+0000 osd.5 (osd.5) 116 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.233630+0000 osd.4 (osd.4) 141 : cluster [DBG] 2.f scrub starts 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.237277+0000 osd.4 (osd.4) 142 : cluster [DBG] 2.f scrub ok 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.323551+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.a scrub starts 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.326275+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.a scrub ok 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.390855+0000 mgr.y (mgr.24526) 212 : cluster [DBG] pgmap v182: 660 pgs: 40 creating+peering, 104 unknown, 516 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 41 MiB/s wr, 60 op/s 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:47.505762+0000 osd.2 (osd.2) 68 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:47.646524+0000 mgr.y (mgr.24526) 213 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:54:50.037 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.121133+0000 mon.a (mon.0) 1038 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.121251+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.121394+0000 mon.a (mon.0) 1040 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.121982+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.124217+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25358 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.125653+0000 mon.a (mon.0) 1043 : audit [INF] from='client.25370 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.038 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.130689+0000 mon.a (mon.0) 1044 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]: dispatch 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.374674+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:48.377946+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.492153+0000 mon.c (mon.2) 157 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.578049+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.580091+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.580428+0000 mon.a (mon.0) 1047 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:48.585153+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.116789+0000 mon.a (mon.0) 1049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.127884+0000 mon.a (mon.0) 1050 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.128296+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25358 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.128543+0000 mon.a (mon.0) 1052 : audit [INF] from='client.25370 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:50.039 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.128874+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pg_num": 4}]': finished 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.129079+0000 mon.a (mon.0) 1054 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-4-cache", "mode": "writeback"}]': finished 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.129437+0000 mon.a (mon.0) 1055 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.131090+0000 mon.a (mon.0) 1056 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.133630+0000 mon.b (mon.1) 78 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.134574+0000 mon.c (mon.2) 158 : audit [INF] from='client.? v1:172.21.15.83:0/174340361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.136666+0000 mon.b (mon.1) 79 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.137287+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.138108+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25283 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.138333+0000 mon.c (mon.2) 159 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.155083+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.155278+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.155555+0000 mon.a (mon.0) 1060 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:54:50.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:49 smithi165 bash[22591]: audit 2024-09-10T18:54:49.155797+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:54:50.055 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:50.144 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN api_stat: Running main() from gmock_main.cc 2024-09-10T18:54:50.144 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-09-10T18:54:50.144 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] Global test environment set-up. 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.Stat (238 ms) 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.StatNS (83 ms) 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (2 ms) 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStat (327 ms total) 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-10T18:54:50.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1078 ms) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (7 ms) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1087 ms total) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] Global test environment tear-down 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (24267 ms total) 2024-09-10T18:54:50.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ PASSED ] 8 tests. 2024-09-10T18:54:50.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:50.183 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:50.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.319 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:50.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:50.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:50.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout:entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-77695-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.209105+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.212199+0000 mon.a [INF] from='client.25346 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.229415+0000 mon.b [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.298336+0000 client.admin [INF] threexx 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.796022+0000 mon.a [INF] from='client.? v1:172.21.15.83:0/2677208122' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.810509+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-77656-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:50.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.810621+0000 mon.a [INF] from='client.25346 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi083-77640-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:45.814672+0000 mon.c [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.259312+0000 mon.a [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.676427+0000 mon.a [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-77715-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.676796+0000 mon.a [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-4", "overlaypool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.676971+0000 mon.a [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-77664-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.677266+0000 mon.a [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-77695-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:50.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-09-10T18:54:46.677554+0000 mon.a [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi083-77768-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10630 ms) 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (24676 ms total) 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (24676 ms total) 2024-09-10T18:54:50.583 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-10T18:54:50.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:50.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:50.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:50.864 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:50.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:50.990 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:50.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:51.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:51.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.210709+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.280447+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:54:51.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.283005+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.288911+0000 mon.b (mon.1) 82 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.453172+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.456158+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.493508+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.620511+0000 mon.a (mon.0) 1062 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.634012+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.637017+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.640628+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.641530+0000 mon.b (mon.1) 84 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.643340+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.643789+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.644598+0000 mon.a (mon.0) 1065 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.106 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:49.645488+0000 mon.a (mon.0) 1066 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.135420+0000 mon.a (mon.0) 1067 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.135995+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.136259+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.136781+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.136804+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.138963+0000 mon.a (mon.0) 1071 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:54:51.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.139150+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]': finished 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.139230+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.139288+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.141596+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.142764+0000 mon.c (mon.2) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3254835101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.143469+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.143757+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.144009+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.144311+0000 mon.b (mon.1) 89 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.144359+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.145588+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.146046+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.108 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.155600+0000 mon.a (mon.0) 1078 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.156445+0000 mon.a (mon.0) 1079 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.156872+0000 mon.a (mon.0) 1080 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.157214+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.157446+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.109 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[22591]: audit 2024-09-10T18:54:50.495112+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.127 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: Running main() from gmock_main.cc 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] Global test environment set-up. 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (274 ms) 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-09-10T18:54:51.145 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (57 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (11 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (8 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (6 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-10T18:54:51.146 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (9 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RmXattr (18 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 14 tests from LibRadosIo (413 ms total) 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-10T18:54:51.147 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1070 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (11 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (7 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (10 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (8 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (6 ms) 2024-09-10T18:54:51.148 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (7 ms) 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (14 ms) 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1143 ms total) 2024-09-10T18:54:51.149 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: 2024-09-10T18:54:51.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.210709+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.280447+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.283005+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.288911+0000 mon.b (mon.1) 82 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.453172+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.456158+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.493508+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.620511+0000 mon.a (mon.0) 1062 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.634012+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.637017+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.640628+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.641530+0000 mon.b (mon.1) 84 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.643340+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.643789+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.644598+0000 mon.a (mon.0) 1065 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:49.645488+0000 mon.a (mon.0) 1066 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.135420+0000 mon.a (mon.0) 1067 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.135995+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.136259+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.136781+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.136804+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.138963+0000 mon.a (mon.0) 1071 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.139150+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.139230+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.139288+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.141596+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.151 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.142764+0000 mon.c (mon.2) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3254835101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.143469+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.143757+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.144009+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.144311+0000 mon.b (mon.1) 89 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.144359+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.145588+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.146046+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.155600+0000 mon.a (mon.0) 1078 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.156445+0000 mon.a (mon.0) 1079 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.156872+0000 mon.a (mon.0) 1080 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.157214+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.157446+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[18629]: audit 2024-09-10T18:54:50.495112+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.210709+0000 mon.b (mon.1) 81 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.280447+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.283005+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.288911+0000 mon.b (mon.1) 82 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.453172+0000 osd.2 (osd.2) 69 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.456158+0000 osd.2 (osd.2) 70 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:54:51.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.493508+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.620511+0000 mon.a (mon.0) 1062 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.634012+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.637017+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.640628+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.641530+0000 mon.b (mon.1) 84 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.643340+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.643789+0000 mon.a (mon.0) 1064 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.644598+0000 mon.a (mon.0) 1065 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:49.645488+0000 mon.a (mon.0) 1066 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.135420+0000 mon.a (mon.0) 1067 : audit [INF] from='client.25283 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-77961-6"}]': finished 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.135995+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.136259+0000 mon.a (mon.0) 1069 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:51.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.136781+0000 mon.a (mon.0) 1070 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.136804+0000 mon.b (mon.1) 85 : audit [INF] from='client.? v1:172.21.15.83:0/4106131438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.138963+0000 mon.a (mon.0) 1071 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.139150+0000 mon.a (mon.0) 1072 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-4"}]': finished 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.139230+0000 mon.a (mon.0) 1073 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.139288+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.141596+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.142764+0000 mon.c (mon.2) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3254835101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.143469+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.143757+0000 mon.b (mon.1) 87 : audit [INF] from='client.? v1:172.21.15.83:0/3983676625' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.144009+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.144311+0000 mon.b (mon.1) 89 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.144359+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]: dispatch 2024-09-10T18:54:51.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.145588+0000 mon.a (mon.0) 1076 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.146046+0000 mon.a (mon.0) 1077 : audit [INF] from='client.25379 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.155600+0000 mon.a (mon.0) 1078 : audit [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.156445+0000 mon.a (mon.0) 1079 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.156872+0000 mon.a (mon.0) 1080 : audit [INF] from='client.25150 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.157214+0000 mon.a (mon.0) 1081 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.157446+0000 mon.a (mon.0) 1082 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:51.156 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:51 smithi083 bash[32381]: audit 2024-09-10T18:54:50.495112+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:51.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] Global test environment tear-down 2024-09-10T18:54:51.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [==========] 24 tests from 2 test suites ran. (25433 ms total) 2024-09-10T18:54:51.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ PASSED ] 24 tests. 2024-09-10T18:54:51.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:51.256 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:51.317 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T18:54:51.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:51.390 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:51.493 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:51.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: seed 77986 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (279 ms) 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-10T18:54:51.608 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (6 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (4 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (28 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (318 ms total) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1668 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1231 ms) 2024-09-10T18:54:51.609 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (11 ms) 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (43 ms) 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2953 ms total) 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (25720 ms total) 2024-09-10T18:54:51.610 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-10T18:54:51.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:51.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:51.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:51.767 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:54:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:54:51.787 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:51.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:51.927 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:51.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:52.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:52.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout:] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (9 ms) 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (9 ms) 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (6 ms) 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (10 ms) 2024-09-10T18:54:52.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (9 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (7 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (6 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (12 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (6 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (6 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-09-10T18:54:52.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1281 ms total) 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (26444 ms total) 2024-09-10T18:54:52.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-10T18:54:52.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:52.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:52.314 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:49.391654+0000 mgr.y (mgr.24526) 214 : cluster [DBG] pgmap v185: 608 pgs: 124 unknown, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.205456+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.207841+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.313494+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.316759+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.681663+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.a scrub starts 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:50.687151+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.a scrub ok 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.126338+0000 mon.a (mon.0) 1083 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.133778+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.133868+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.133961+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.134108+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:52.315 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.134175+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.134262+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.134332+0000 mon.a (mon.0) 1090 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.134795+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.135049+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.141700+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.438175+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.f scrub starts 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.441646+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.f scrub ok 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.496712+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.595155+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.596430+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.83:0/2706020620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.596718+0000 mon.a (mon.0) 1092 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.597199+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1090827878' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.597937+0000 mon.a (mon.0) 1093 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.598734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.604818+0000 mon.a (mon.0) 1095 : audit [INF] from='client.25391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:52 smithi165 bash[22591]: audit 2024-09-10T18:54:51.605250+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.333 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:52.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:52.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:52.529 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:49.391654+0000 mgr.y (mgr.24526) 214 : cluster [DBG] pgmap v185: 608 pgs: 124 unknown, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.205456+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.207841+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.313494+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.316759+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.681663+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.a scrub starts 2024-09-10T18:54:52.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:50.687151+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.a scrub ok 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.126338+0000 mon.a (mon.0) 1083 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.133778+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.133868+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.133961+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.134108+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.134175+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.134262+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.134332+0000 mon.a (mon.0) 1090 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.134795+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.712 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.135049+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:49.391654+0000 mgr.y (mgr.24526) 214 : cluster [DBG] pgmap v185: 608 pgs: 124 unknown, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.205456+0000 osd.5 (osd.5) 117 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.207841+0000 osd.5 (osd.5) 118 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.313494+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.316759+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.681663+0000 osd.6 (osd.6) 87 : cluster [DBG] 3.a scrub starts 2024-09-10T18:54:52.713 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:50.687151+0000 osd.6 (osd.6) 88 : cluster [DBG] 3.a scrub ok 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.126338+0000 mon.a (mon.0) 1083 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.133778+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? v1:172.21.15.83:0/1370785971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-77986-7"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.133868+0000 mon.a (mon.0) 1085 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.133961+0000 mon.a (mon.0) 1086 : audit [INF] from='client.25379 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi083-77640-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.134108+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-77656-16"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.134175+0000 mon.a (mon.0) 1088 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.134262+0000 mon.a (mon.0) 1089 : audit [INF] from='client.25150 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-4", "tierpool": "test-rados-api-smithi083-77828-4-cache"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.134332+0000 mon.a (mon.0) 1090 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.134795+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.83:0/610201929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.135049+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.83:0/2883396504' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.141700+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.438175+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.f scrub starts 2024-09-10T18:54:52.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.441646+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.f scrub ok 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.496712+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.595155+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.596430+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.83:0/2706020620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.596718+0000 mon.a (mon.0) 1092 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.597199+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1090827878' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.597937+0000 mon.a (mon.0) 1093 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.598734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.604818+0000 mon.a (mon.0) 1095 : audit [INF] from='client.25391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18629]: audit 2024-09-10T18:54:51.605250+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.141700+0000 mon.c (mon.2) 164 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.438175+0000 osd.1 (osd.1) 125 : cluster [DBG] 4.f scrub starts 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.441646+0000 osd.1 (osd.1) 126 : cluster [DBG] 4.f scrub ok 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.496712+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.595155+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.596430+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.83:0/2706020620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.596718+0000 mon.a (mon.0) 1092 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.597199+0000 mon.c (mon.2) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1090827878' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.597937+0000 mon.a (mon.0) 1093 : audit [INF] from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.598734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.25330 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.604818+0000 mon.a (mon.0) 1095 : audit [INF] from='client.25391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[32381]: audit 2024-09-10T18:54:51.605250+0000 mon.a (mon.0) 1096 : audit [INF] from='client.25400 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:52.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:52.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:52.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:54:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:54:52] "GET /metrics HTTP/1.1" 200 261175 "" "Prometheus/2.43.0" 2024-09-10T18:54:52.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:52.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:52.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:53.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:53.041 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:53.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:53.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:53.176 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:53.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:53.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:53.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.392318+0000 mgr.y (mgr.24526) 215 : cluster [DBG] pgmap v188: 600 pgs: 64 unknown, 32 creating+peering, 504 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T18:54:53.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.923274+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:51.926924+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.132052+0000 mon.a (mon.0) 1097 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.146966+0000 mon.a (mon.0) 1098 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.147120+0000 mon.a (mon.0) 1099 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.147297+0000 mon.a (mon.0) 1100 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.147409+0000 mon.a (mon.0) 1101 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.147686+0000 mon.a (mon.0) 1102 : audit [INF] from='client.25391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.147869+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.148141+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.149324+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.83:0/3392350921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.150637+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.152855+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.162264+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.162510+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.162758+0000 mon.a (mon.0) 1107 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.398785+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.411406+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[18629]: audit 2024-09-10T18:54:52.498410+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.392318+0000 mgr.y (mgr.24526) 215 : cluster [DBG] pgmap v188: 600 pgs: 64 unknown, 32 creating+peering, 504 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T18:54:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.923274+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:51.926924+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.132052+0000 mon.a (mon.0) 1097 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.146966+0000 mon.a (mon.0) 1098 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.147120+0000 mon.a (mon.0) 1099 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.147297+0000 mon.a (mon.0) 1100 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.147409+0000 mon.a (mon.0) 1101 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.147686+0000 mon.a (mon.0) 1102 : audit [INF] from='client.25391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.147869+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.148141+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.149324+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.83:0/3392350921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.150637+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.152855+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.162264+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.162510+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.162758+0000 mon.a (mon.0) 1107 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.398785+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.411406+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:53 smithi083 bash[32381]: audit 2024-09-10T18:54:52.498410+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.392318+0000 mgr.y (mgr.24526) 215 : cluster [DBG] pgmap v188: 600 pgs: 64 unknown, 32 creating+peering, 504 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T18:54:53.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.923274+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:51.926924+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.132052+0000 mon.a (mon.0) 1097 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.146966+0000 mon.a (mon.0) 1098 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi083-77992-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.147120+0000 mon.a (mon.0) 1099 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "mode": "writeback"}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.147297+0000 mon.a (mon.0) 1100 : audit [INF] from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-77664-22"}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.147409+0000 mon.a (mon.0) 1101 : audit [INF] from='client.25330 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi083-77768-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.147686+0000 mon.a (mon.0) 1102 : audit [INF] from='client.25391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.147869+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25400 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.148141+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.149324+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.83:0/3392350921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.150637+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.152855+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.162264+0000 mon.a (mon.0) 1105 : audit [INF] from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:53.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.162510+0000 mon.a (mon.0) 1106 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:53.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.162758+0000 mon.a (mon.0) 1107 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:53.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.398785+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:54:53.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.411406+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:54:53.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:53 smithi165 bash[22591]: audit 2024-09-10T18:54:52.498410+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:53.313 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:53.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:53.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:53.449 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:53.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:53.553 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:53.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:53.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:53.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:54.051 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:54.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:54.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:54.359 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:54.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:54.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.327351+0000 osd.4 (osd.4) 147 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.331087+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.119342+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.119414+0000 mon.a (mon.0) 1109 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.119481+0000 mon.a (mon.0) 1110 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.119601+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.130572+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.83:0/2884398523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.138918+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.139288+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.157036+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.157388+0000 mon.a (mon.0) 1113 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.157692+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[18629]: audit 2024-09-10T18:54:53.499494+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:54.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.327351+0000 osd.4 (osd.4) 147 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:54:54.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.331087+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:54:54.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.119342+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:54.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.119414+0000 mon.a (mon.0) 1109 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:54.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.119481+0000 mon.a (mon.0) 1110 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.119601+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.130572+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.83:0/2884398523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.138918+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.139288+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.157036+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.157388+0000 mon.a (mon.0) 1113 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.157692+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:54 smithi083 bash[32381]: audit 2024-09-10T18:54:53.499494+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: Running main() from gmock_main.cc 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] Global test environment set-up. 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-10T18:54:54.439 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (297 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockShared (35 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1008 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1008 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (6 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.Unlock (6 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (8 ms) 2024-09-10T18:54:54.440 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (5 ms) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLock (2373 ms total) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1094 ms) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6 ms) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1047 ms) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1019 ms) 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-10T18:54:54.441 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (9 ms) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (6 ms) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (11 ms) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3196 ms total) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] Global test environment tear-down 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (28700 ms total) 2024-09-10T18:54:54.442 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-10T18:54:54.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:54.496 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: seed 77715 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (269 ms) 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (4 ms) 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-10T18:54:54.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1008 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1009 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (6 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (8 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2312 ms total) 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-10T18:54:54.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1091 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (5 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1067 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1008 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (7 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (9 ms) 2024-09-10T18:54:54.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3200 ms total) 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (28744 ms total) 2024-09-10T18:54:54.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-10T18:54:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:54.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:54.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:54.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.327351+0000 osd.4 (osd.4) 147 : cluster [DBG] 4.1 scrub starts 2024-09-10T18:54:54.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.331087+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.1 scrub ok 2024-09-10T18:54:54.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.119342+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi083-77907-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:54.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.119414+0000 mon.a (mon.0) 1109 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:54.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.119481+0000 mon.a (mon.0) 1110 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.119601+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.130572+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.83:0/2884398523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.138918+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.83:0/3556761510' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.139288+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.83:0/4186865771' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.157036+0000 mon.a (mon.0) 1112 : audit [INF] from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.157388+0000 mon.a (mon.0) 1113 : audit [INF] from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.157692+0000 mon.a (mon.0) 1114 : audit [INF] from='client.25336 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]: dispatch 2024-09-10T18:54:54.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:54 smithi165 bash[22591]: audit 2024-09-10T18:54:53.499494+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:54.974 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:55.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:55.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:55.306 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:55.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:54:55.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:55.437 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:54:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:55.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:54:55.466 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.573507+0000 osd.3 (osd.3) 139 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:52.576227+0000 osd.3 (osd.3) 140 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.225630+0000 osd.5 (osd.5) 119 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.253644+0000 osd.5 (osd.5) 120 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.307609+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.310473+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.392882+0000 mgr.y (mgr.24526) 216 : cluster [DBG] pgmap v191: 496 pgs: 72 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.957431+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.b scrub starts 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:53.960188+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.b scrub ok 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.385487+0000 mon.a (mon.0) 1115 : audit [INF] from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.467 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.385596+0000 mon.a (mon.0) 1116 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1196 ms) 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (11 ms) 2024-09-10T18:54:55.489 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1207 ms total) 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (62 ms) 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5395 ms) 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (22 ms) 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (37 ms) 2024-09-10T18:54:55.490 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472351879216 notify_id 597000454146 notifier_gid 25352 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (41 ms) 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472351898288 notify_id 597000454146 notifier_gid 25352 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (20 ms) 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472351901712 notify_id 597000454147 notifier_gid 25352 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (19 ms) 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472351901712 notify_id 597000454144 notifier_gid 25352 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (24 ms) 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363128048 notify_id 597000454147 notifier_gid 25352 2024-09-10T18:54:55.491 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (9 ms) 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363128048 notify_id 597000454148 notifier_gid 25352 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (12 ms) 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: trying... 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363128048 notify_id 597000454145 notifier_gid 25352 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: timed out 2024-09-10T18:54:55.492 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-09-10T18:54:55.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:55.669 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:55.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:55.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:55.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.385656+0000 mon.a (mon.0) 1117 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:54.385920+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.388613+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.83:0/3723057016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.390551+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1128392704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.395762+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.404790+0000 mon.a (mon.0) 1119 : audit [INF] from='client.26204 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.405268+0000 mon.a (mon.0) 1120 : audit [INF] from='client.26107 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.405565+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:54.417977+0000 mon.a (mon.0) 1122 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:54.500671+0000 mon.c (mon.2) 171 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:54.953318+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:54.957309+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:54:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.360926+0000 mon.a (mon.0) 1123 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.367994+0000 mon.a (mon.0) 1124 : audit [INF] from='client.26204 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.368118+0000 mon.a (mon.0) 1125 : audit [INF] from='client.26107 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.368197+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.368316+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.383011+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.383169+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.399510+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[18629]: audit 2024-09-10T18:54:55.399914+0000 mon.a (mon.0) 1129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.573507+0000 osd.3 (osd.3) 139 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:52.576227+0000 osd.3 (osd.3) 140 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.225630+0000 osd.5 (osd.5) 119 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.253644+0000 osd.5 (osd.5) 120 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.307609+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.310473+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.392882+0000 mgr.y (mgr.24526) 216 : cluster [DBG] pgmap v191: 496 pgs: 72 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.957431+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.b scrub starts 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:53.960188+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.b scrub ok 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.385487+0000 mon.a (mon.0) 1115 : audit [INF] from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.385596+0000 mon.a (mon.0) 1116 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.385656+0000 mon.a (mon.0) 1117 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:54.385920+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.388613+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.83:0/3723057016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.390551+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1128392704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.395762+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.404790+0000 mon.a (mon.0) 1119 : audit [INF] from='client.26204 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.405268+0000 mon.a (mon.0) 1120 : audit [INF] from='client.26107 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.405565+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:54.417977+0000 mon.a (mon.0) 1122 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:54.500671+0000 mon.c (mon.2) 171 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:54.953318+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:54.957309+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:54:55.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.360926+0000 mon.a (mon.0) 1123 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.367994+0000 mon.a (mon.0) 1124 : audit [INF] from='client.26204 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.368118+0000 mon.a (mon.0) 1125 : audit [INF] from='client.26107 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.368197+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.368316+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.383011+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.383169+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.399510+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:55 smithi083 bash[32381]: audit 2024-09-10T18:54:55.399914+0000 mon.a (mon.0) 1129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.573507+0000 osd.3 (osd.3) 139 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:54:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:52.576227+0000 osd.3 (osd.3) 140 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.225630+0000 osd.5 (osd.5) 119 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.253644+0000 osd.5 (osd.5) 120 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.307609+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.310473+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.392882+0000 mgr.y (mgr.24526) 216 : cluster [DBG] pgmap v191: 496 pgs: 72 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.957431+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.b scrub starts 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:53.960188+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.b scrub ok 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.385487+0000 mon.a (mon.0) 1115 : audit [INF] from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi083-77640-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.385596+0000 mon.a (mon.0) 1116 : audit [INF] from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-77715-10"}]': finished 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.385656+0000 mon.a (mon.0) 1117 : audit [INF] from='client.25336 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-77695-10"}]': finished 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:54.385920+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-09-10T18:54:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.388613+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.83:0/3723057016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.390551+0000 mon.c (mon.2) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1128392704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.395762+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.404790+0000 mon.a (mon.0) 1119 : audit [INF] from='client.26204 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.405268+0000 mon.a (mon.0) 1120 : audit [INF] from='client.26107 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.405565+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:54.417977+0000 mon.a (mon.0) 1122 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:54.500671+0000 mon.c (mon.2) 171 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:54.953318+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:54.957309+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.360926+0000 mon.a (mon.0) 1123 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.367994+0000 mon.a (mon.0) 1124 : audit [INF] from='client.26204 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.368118+0000 mon.a (mon.0) 1125 : audit [INF] from='client.26107 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:55.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.368197+0000 mon.a (mon.0) 1126 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:55.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.368316+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-09-10T18:54:55.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.383011+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.83:0/258167267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.383169+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:55.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.399510+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25375 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]: dispatch 2024-09-10T18:54:55.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:55 smithi165 bash[22591]: audit 2024-09-10T18:54:55.399914+0000 mon.a (mon.0) 1129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:56.263 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:56.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:56.334 INFO:tasks.workunit.client.0.smithi083.stdout: a api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-10T18:54:56.334 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-10T18:54:56.334 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-10T18:54:56.334 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-10T18:54:56.334 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5030 ms) 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (74 ms) 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20022 ms) 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: attempt 0 of 20 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: failed to deregister: 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: { 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "epoch": 19, 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "modified": "2024-09-10T18:54:51.391891+0000", 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "services": { 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "laundry": { 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "daemons": { 2024-09-10T18:54:56.335 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "summary": "", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "close-test-pid78097": { 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_epoch": 19, 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_stamp": "2024-09-10T18:54:51.128905+0000", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "gid": 25406, 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "addr": "172.21.15.83:0/3765880583", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "metadata": { 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "arch": "x86_64", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_release": "quincy", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:54:56.336 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version_short": "17.2.7-1606-g5be57395", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro": "ubuntu", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_description": "Ubuntu 20.04.4 LTS", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_version": "20.04", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "foo": "bar", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "hostname": "smithi083", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_description": "#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_version": "5.4.0-195-generic", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_swap_kb": "2097148", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_total_kb": "32754944", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "os": "Linux", 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "this": "that" 2024-09-10T18:54:56.337 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: }, 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "task_status": {} 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: }, 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "pid78093": { 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_epoch": 18, 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_stamp": "2024-09-10T18:54:39.361423+0000", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "gid": 25259, 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "addr": "172.21.15.83:0/801074688", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "metadata": { 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "arch": "x86_64", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_release": "quincy", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version_short": "17.2.7-1606-g5be57395", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-10T18:54:56.338 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro": "ubuntu", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_description": "Ubuntu 20.04.4 LTS", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_version": "20.04", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "foo": "bar", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "hostname": "smithi083", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_description": "#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_version": "5.4.0-195-generic", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_swap_kb": "2097148", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_total_kb": "32754944", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "os": "Linux", 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "this": "that" 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: }, 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "task_status": {} 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:54:56.339 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: }, 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "rgw": { 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "daemons": { 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "summary": "", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "14493": { 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_epoch": 13, 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "start_stamp": "2024-09-10T18:49:25.207072+0000", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "gid": 14493, 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "addr": "172.21.15.83:0/703687015", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "metadata": { 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "arch": "x86_64", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_release": "quincy", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:54:56.340 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "ceph_version_short": "17.2.7-1606-g5be57395", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "container_hostname": "smithi083", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro": "centos", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "distro_version": "9", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "frontend_type#0": "beast", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "hostname": "smithi083", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "id": "foo.a", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_description": "#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "kernel_version": "5.4.0-195-generic", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_swap_kb": "2097148", 2024-09-10T18:54:56.341 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "mem_total_kb": "32754944", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "num_handles": "1", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "os": "Linux", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "pid": "6", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "realm_id": "", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "realm_name": "", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "zone_id": "ef939013-895d-4fcb-a130-e9a8938d4e31", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "zone_name": "default", 2024-09-10T18:54:56.342 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "zonegroup_id": "348ee464-2af6-469c-a2f0-1b47a5a7c71b", 2024-09-10T18:54:56.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:56.525 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:56.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:56.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:56.665 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:56.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:56.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "zonegroup_name api_watch_notify: Running main() from gmock_main.cc 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T18:54:56.782 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (332 ms) 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94153128865584 err -107 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (18 ms) 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94153128896128 err -107 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (8 ms) 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879297 cookie 94153128916256 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (13 ms) 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-10T18:54:56.783 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879298 cookie 94153128916256 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (11 ms) 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879297 cookie 94153129038688 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (9 ms) 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879299 cookie 94153128924736 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879299 cookie 94153128926224 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (23 ms) 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 541165879300 cookie 94153128924736 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 545460846597 cookie 94153128924736 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3012 ms) 2024-09-10T18:54:56.784 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94153128924736 err -107 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 24865 notify_id 571230650375 cookie 94153128924736 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5011 ms) 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94153128924736 err -107 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1011 ms) 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9448 ms total) 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-10T18:54:56.785 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1105 ms) 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1105 ms total) 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (30899 ms total) 2024-09-10T18:54:56.786 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-10T18:54:56.800 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:54:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:56.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:54:56.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.479202+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[18629]: audit 2024-09-10T18:54:55.506747+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.507162+0000 osd.1 (osd.1) 130 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[18629]: audit 2024-09-10T18:54:56.309786+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.83:0/830212394' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[18629]: audit 2024-09-10T18:54:56.511597+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.479202+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[32381]: audit 2024-09-10T18:54:55.506747+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.507162+0000 osd.1 (osd.1) 130 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[32381]: audit 2024-09-10T18:54:56.309786+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.83:0/830212394' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:54:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:56 smithi083 bash[32381]: audit 2024-09-10T18:54:56.511597+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:56 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.479202+0000 osd.1 (osd.1) 129 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:54:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:56 smithi165 bash[22591]: audit 2024-09-10T18:54:55.506747+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:56 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.507162+0000 osd.1 (osd.1) 130 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:54:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:56 smithi165 bash[22591]: audit 2024-09-10T18:54:56.309786+0000 mon.b (mon.1) 102 : audit [DBG] from='client.? v1:172.21.15.83:0/830212394' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:54:57.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:56 smithi165 bash[22591]: audit 2024-09-10T18:54:56.511597+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:57.499 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:57.549 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:54:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:57.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:57.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:57.924 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:57.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:58.034 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:58.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.051+0000 7f9821360640 1 -- 172.21.15.83:0/1575391707 >> v1:172.21.15.165:6789/0 conn(0x7f981c06df30 legacy=0x7f981c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.055+0000 7f9821360640 1 -- 172.21.15.83:0/1575391707 shutdown_connections 2024-09-10T18:54:58.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.055+0000 7f9821360640 1 -- 172.21.15.83:0/1575391707 >> 172.21.15.83:0/1575391707 conn(0x7f981c068750 msgr2=0x7f981c068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.055+0000 7f9821360640 1 -- 172.21.15.83:0/1575391707 shutdown_connections 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.055+0000 7f9821360640 1 -- 172.21.15.83:0/1575391707 wait complete. 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 librados: starting msgr at 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 librados: starting objecter 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 Processor -- start 2024-09-10T18:54:58.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 -- start start 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 librados: setting wanted keys 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 librados: calling monclient init 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f981c1174a0 con 0x7f981c06df30 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f981c123870 con 0x7f981c117d80 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9821360640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f981c124a10 con 0x7f981c10b850 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9820b5f640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f981c117d80 0x7f981c121140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:42646/0 (socket says 172.21.15.83:42646) 2024-09-10T18:54:58.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.059+0000 7f9820b5f640 1 -- 172.21.15.83:0/3975122953 learned_addr learned my addr 172.21.15.83:0/3975122953 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter ms_handle_connect 0x7f981c117d80 2024-09-10T18:54:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter ms_handle_connect 0x7f981c06df30 2024-09-10T18:54:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:58.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter ms_handle_connect 0x7f981c10b850 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786162953 0 0) 0x7f981c123870 con 0x7f981c117d80 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98040054b0 con 0x7f981c117d80 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116957816 0 0) 0x7f981c124a10 con 0x7f981c10b850 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.067+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f981c123870 con 0x7f981c10b850 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3876652598 0 0) 0x7f981c123870 con 0x7f981c10b850 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f981c124a10 con 0x7f981c10b850 2024-09-10T18:54:58.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f980c001780 con 0x7f981c10b850 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1115383619 0 0) 0x7f981c124a10 con 0x7f981c10b850 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.165:6789/0 conn(0x7f981c117d80 legacy=0x7f981c121140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6789/0 conn(0x7f981c06df30 legacy=0x7f981c111ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c125bf0 con 0x7f981c10b850 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c1228c0 con 0x7f981c10b850 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9821360640 10 client.31370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:58.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c122d10 con 0x7f981c10b850 2024-09-10T18:54:58.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f9821360640 1 librados: init done 2024-09-10T18:54:58.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.071+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f981c06f390 con 0x7f981c10b850 2024-09-10T18:54:58.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.075+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f980c002dc0 con 0x7f981c10b850 2024-09-10T18:54:58.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.075+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f980c0028d0 con 0x7f981c10b850 2024-09-10T18:54:58.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.075+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f980c019490 con 0x7f981c10b850 2024-09-10T18:54:58.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(143..143 src has 69..143) v4 ==== 10369+0+0 (unknown 2366051450 0 0) 0x7f980c089280 con 0x7f981c10b850 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 osd_map(143..143 src has 69..143) v4 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 3 client.31370.objecter handle_osd_map got epochs [143,143] > 0 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 3 client.31370.objecter handle_osd_map decoding full epoch 143 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 20 client.31370.objecter dump_active .. 0 homeless 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v143) v1 ==== 54+0+16 (unknown 2123765116 0 2420000192) 0x7f980c0895e0 con 0x7f981c10b850 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9821360640 10 client.31370.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 1 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.079+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c125360 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 1 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 0 1 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 4 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 10 client.31370.objecter _send_command 1 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c105800 con 0x7f981c125360 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c105800 con 0x7f981c125360 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9818ff9640 10 client.31370.objecter _finish_command 1 = system:0 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 4 2024-09-10T18:54:58.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.083+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 0 1 2024-09-10T18:54:58.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:54:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:54:58.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.111+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3400391779 >> v1:172.21.15.165:6789/0 conn(0x7f72b810f520 legacy=0x7f72b81119b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.111+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3400391779 shutdown_connections 2024-09-10T18:54:58.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.111+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3400391779 >> 172.21.15.83:0/3400391779 conn(0x7f72b80723b0 msgr2=0x7f72b80747c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:58.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.111+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f9804071a40 2024-09-10T18:54:58.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.111+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3400391779 shutdown_connections 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3400391779 wait complete. 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 librados: starting msgr at 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 librados: starting objecter 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 Processor -- start 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 -- start start 2024-09-10T18:54:58.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 librados: setting wanted keys 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 librados: calling monclient init 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72b8108560 con 0x7f72b810b9c0 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72b811b240 con 0x7f72b8108be0 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bf6d0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72b811c440 con 0x7f72b810f520 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bd46c640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f72b8108be0 0x7f72b8107a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:42758/0 (socket says 172.21.15.83:42758) 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72bd46c640 1 -- 172.21.15.83:0/3111658027 learned_addr learned my addr 172.21.15.83:0/3111658027 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72adffb640 10 client.?.objecter ms_handle_connect 0x7f72b8108be0 2024-09-10T18:54:58.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.119+0000 7f72adffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:58.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118713423 0 0) 0x7f72b811b240 con 0x7f72b8108be0 2024-09-10T18:54:58.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a40054b0 con 0x7f72b8108be0 2024-09-10T18:54:58.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 10 client.?.objecter ms_handle_connect 0x7f72b810f520 2024-09-10T18:54:58.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:58.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848241277 0 0) 0x7f72a40054b0 con 0x7f72b8108be0 2024-09-10T18:54:58.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72b811b240 con 0x7f72b8108be0 2024-09-10T18:54:58.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f72b4001980 con 0x7f72b8108be0 2024-09-10T18:54:58.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469371900 0 0) 0x7f72b811b240 con 0x7f72b8108be0 2024-09-10T18:54:58.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 >> v1:172.21.15.83:6790/0 conn(0x7f72b810f520 legacy=0x7f72b8118b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 >> v1:172.21.15.83:6789/0 conn(0x7f72b810b9c0 legacy=0x7f72b807cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b811d640 con 0x7f72b8108be0 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b811c630 con 0x7f72b8108be0 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72bf6d0640 10 client.31468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b811cac0 con 0x7f72b8108be0 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72bf6d0640 1 librados: init done 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f72b4001cc0 con 0x7f72b8108be0 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.123+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f72b4005bd0 con 0x7f72b8108be0 2024-09-10T18:54:58.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f72b4002270 con 0x7f72b8108be0 2024-09-10T18:54:58.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 10 client.31468.objecter ms_handle_connect 0x7f72a40716d0 2024-09-10T18:54:58.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(143..143 src has 69..143) v4 ==== 10369+0+0 (unknown 2366051450 0 0) 0x7f72b408afb0 con 0x7f72b8108be0 2024-09-10T18:54:58.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 10 client.31468.objecter ms_dispatch 0x7f72b8000b90 osd_map(143..143 src has 69..143) v4 2024-09-10T18:54:58.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 3 client.31468.objecter handle_osd_map got epochs [143,143] > 0 2024-09-10T18:54:58.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c005010 con 0x7f72b8108be0 2024-09-10T18:54:58.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 3 client.31468.objecter handle_osd_map decoding full epoch 143 2024-09-10T18:54:58.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.127+0000 7f72adffb640 20 client.31468.objecter dump_active .. 0 homeless 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.139+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f72b4056c30 con 0x7f72b8108be0 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 10 client.31370.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 2 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 2 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c10b100 osd=0 3 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 0 2 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 4 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 10 client.31370.objecter _send_command 2 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.143+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c04bf60 con 0x7f981c125360 2024-09-10T18:54:58.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.147+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c04bf60 con 0x7f981c125360 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.147+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.147+0000 7f9818ff9640 10 client.31370.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.147+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 4 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.147+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 0 2 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:54:58.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 10 client.31370.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 1 3 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 4 2024-09-10T18:54:58.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 10 client.31370.objecter _send_command 3 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.151+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c125990 con 0x7f981c0298c0 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c0298c0 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c125990 con 0x7f981c0298c0 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 10 client.31370.objecter _finish_command 3 = system:0 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 4 2024-09-10T18:54:58.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.159+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 1 3 2024-09-10T18:54:58.226 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 10 client.31370.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 4 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 4 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c11e8f0 osd=1 3 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 1 4 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 4 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 10 client.31370.objecter _send_command 4 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.211+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c02ab60 con 0x7f981c0298c0 2024-09-10T18:54:58.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.219+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c02ab60 con 0x7f981c0298c0 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.219+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.219+0000 7f9818ff9640 10 client.31370.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.219+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 4 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.219+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 1 4 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 10 client.31370.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 5 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 5 2024-09-10T18:54:58.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 2 5 2024-09-10T18:54:58.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 4 2024-09-10T18:54:58.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 10 client.31370.objecter _send_command 5 2024-09-10T18:54:58.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c152150 con 0x7f981c12c540 2024-09-10T18:54:58.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.223+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c12c540 2024-09-10T18:54:58.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.227+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c152150 con 0x7f981c12c540 2024-09-10T18:54:58.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.227+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:54:58.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.227+0000 7f9818ff9640 10 client.31370.objecter _finish_command 5 = system:0 2024-09-10T18:54:58.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.227+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 4 2024-09-10T18:54:58.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.227+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 2 5 2024-09-10T18:54:58.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 10 client.31370.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 6 2024-09-10T18:54:58.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 6 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c124350 osd=2 3 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 2 6 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 4 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 10 client.31370.objecter _send_command 6 2024-09-10T18:54:58.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.263+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c02a5e0 con 0x7f981c12c540 2024-09-10T18:54:58.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.267+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c02a5e0 con 0x7f981c12c540 2024-09-10T18:54:58.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.267+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.267+0000 7f9818ff9640 10 client.31370.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.267+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 4 2024-09-10T18:54:58.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.267+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 2 6 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 10 client.31370.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 7 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 7 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 3 7 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 4 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 10 client.31370.objecter _send_command 7 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c156b70 con 0x7f981c05e820 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c05e820 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c156b70 con 0x7f981c05e820 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 10 client.31370.objecter _finish_command 7 = system:0 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 4 2024-09-10T18:54:58.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.271+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 3 7 2024-09-10T18:54:58.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 10 client.31370.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 8 2024-09-10T18:54:58.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 8 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c02a9f0 osd=3 3 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 3 8 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 4 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 10 client.31370.objecter _send_command 8 2024-09-10T18:54:58.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.343+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c05dd90 con 0x7f981c05e820 2024-09-10T18:54:58.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.359+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c05dd90 con 0x7f981c05e820 2024-09-10T18:54:58.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.359+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.359+0000 7f9818ff9640 10 client.31370.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.359+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 4 2024-09-10T18:54:58.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.359+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 3 8 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.363+0000 7f9821360640 10 client.31370.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.363+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 9 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.363+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.363+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 9 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 4 9 2024-09-10T18:54:58.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 4 2024-09-10T18:54:58.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 10 client.31370.objecter _send_command 9 2024-09-10T18:54:58.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.367+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c05dcc0 con 0x7f981c152f90 2024-09-10T18:54:58.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.371+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c152f90 2024-09-10T18:54:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.379+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c05dcc0 con 0x7f981c152f90 2024-09-10T18:54:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.379+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:54:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.379+0000 7f9818ff9640 10 client.31370.objecter _finish_command 9 = system:0 2024-09-10T18:54:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.379+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 4 2024-09-10T18:54:58.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.379+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 4 9 2024-09-10T18:54:58.406 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:58.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:58.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 10 client.31370.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign -1 10 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 20 client.31370.objecter _get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 15 client.31370.objecter _session_command_op_remove -1 10 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 20 client.31370.objecter get_session s=0x7f981c05db50 osd=4 3 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 15 client.31370.objecter _session_command_op_assign 4 10 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 4 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 10 client.31370.objecter _send_command 10 2024-09-10T18:54:58.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.415+0000 7f9821360640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c158570 con 0x7f981c152f90 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c158570 con 0x7f981c152f90 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f9818ff9640 10 client.31370.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 4 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 4 10 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 10 client.31370.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 11 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 11 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 5 11 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 4 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 10 client.31370.objecter _send_command 11 2024-09-10T18:54:58.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.423+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c1590a0 con 0x7f981c158c20 2024-09-10T18:54:58.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c158c20 2024-09-10T18:54:58.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c1590a0 con 0x7f981c158c20 2024-09-10T18:54:58.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:54:58.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 10 client.31370.objecter _finish_command 11 = system:0 2024-09-10T18:54:58.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 4 2024-09-10T18:54:58.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.435+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 5 11 2024-09-10T18:54:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 10 client.31370.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 12 2024-09-10T18:54:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 12 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c158570 osd=5 3 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 5 12 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 4 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 10 client.31370.objecter _send_command 12 2024-09-10T18:54:58.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.487+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c1590a0 con 0x7f981c158c20 2024-09-10T18:54:58.504 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c1590a0 con 0x7f981c158c20 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f9818ff9640 10 client.31370.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 4 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 5 12 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 10 client.31370.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 13 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 13 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 6 13 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 4 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 10 client.31370.objecter _send_command 13 2024-09-10T18:54:58.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.499+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c126e90 con 0x7f981c126a60 2024-09-10T18:54:58.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.507+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c126a60 2024-09-10T18:54:58.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.511+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c126e90 con 0x7f981c126a60 2024-09-10T18:54:58.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.511+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:54:58.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.511+0000 7f9818ff9640 10 client.31370.objecter _finish_command 13 = system:0 2024-09-10T18:54:58.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.511+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 4 2024-09-10T18:54:58.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.511+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 6 13 2024-09-10T18:54:58.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 10 client.31370.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 14 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 14 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c1590a0 osd=6 3 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 6 14 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 4 2024-09-10T18:54:58.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 10 client.31370.objecter _send_command 14 2024-09-10T18:54:58.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.539+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c126e90 con 0x7f981c126a60 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c126e90 con 0x7f981c126a60 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 10 client.31370.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 4 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 6 14 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:54:58.549 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 10 client.31370.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 15 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 15 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 7 15 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 4 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 10 client.31370.objecter _send_command 15 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981c168130 con 0x7f981c127070 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f9818ff9640 10 client.31370.objecter ms_handle_connect 0x7f981c127070 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.543+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f729c0052c0 con 0x7f72b8108be0 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.547+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f981c168130 con 0x7f981c127070 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.547+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:54:58.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.547+0000 7f9818ff9640 10 client.31370.objecter _finish_command 15 = system:0 2024-09-10T18:54:58.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.547+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 4 2024-09-10T18:54:58.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.547+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 7 15 2024-09-10T18:54:58.553 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:54:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:58.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 10 client.31370.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign -1 16 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 20 client.31370.objecter _get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 15 client.31370.objecter _session_command_op_remove -1 16 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 20 client.31370.objecter get_session s=0x7f981c126e90 osd=7 3 2024-09-10T18:54:58.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 15 client.31370.objecter _session_command_op_assign 7 16 2024-09-10T18:54:58.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 4 2024-09-10T18:54:58.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 10 client.31370.objecter _send_command 16 2024-09-10T18:54:58.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.559+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f981c1699d0 con 0x7f981c127070 2024-09-10T18:54:58.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9818ff9640 1 -- 172.21.15.83:0/3975122953 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f981c1699d0 con 0x7f981c127070 2024-09-10T18:54:58.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9818ff9640 10 client.31370.objecter ms_dispatch 0x7f981c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:54:58.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9818ff9640 10 client.31370.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:54:58.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9818ff9640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 4 2024-09-10T18:54:58.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9818ff9640 15 client.31370.objecter _session_command_op_remove 7 16 2024-09-10T18:54:58.568 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 librados: watch_flush enter 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 librados: watch_flush exit 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.0 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6801/2535527483 conn(0x7f981c125360 legacy=0x7f981c123a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c10b100 osd=0 1 2024-09-10T18:54:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.1 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6805/2309624637 conn(0x7f981c0298c0 legacy=0x7f981c029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c11e8f0 osd=1 1 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.2 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6809/386094220 conn(0x7f981c12c540 legacy=0x7f981c12e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c124350 osd=2 1 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.3 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6813/1279688021 conn(0x7f981c05e820 legacy=0x7f981c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c02a9f0 osd=3 1 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.4 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.165:6800/3225515265 conn(0x7f981c152f90 legacy=0x7f981c157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c05db50 osd=4 1 2024-09-10T18:54:58.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.5 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.165:6804/571173777 conn(0x7f981c158c20 legacy=0x7f981c125e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c158570 osd=5 1 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9820b5f640 1 -- 172.21.15.83:0/3975122953 reap_dead start 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.6 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.165:6808/1317922936 conn(0x7f981c126a60 legacy=0x7f981c162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9820b5f640 1 -- 172.21.15.83:0/3975122953 reap_dead start 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c1590a0 osd=6 1 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter close_session for osd.7 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.165:6812/1610151650 conn(0x7f981c127070 legacy=0x7f981c166900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9820b5f640 1 -- 172.21.15.83:0/3975122953 reap_dead start 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter put_session s=0x7f981c126e90 osd=7 1 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 20 client.31370.objecter shutdown clearing up homeless session... 2024-09-10T18:54:58.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f9820b5f640 1 -- 172.21.15.83:0/3975122953 reap_dead start 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 10 client.31370.objecter successfully canceled tick 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9804071a40 legacy=0x7f9804073ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> v1:172.21.15.83:6790/0 conn(0x7f981c10b850 legacy=0x7f981c1125d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 shutdown_connections 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 >> 172.21.15.83:0/3975122953 conn(0x7f981c068750 msgr2=0x7f981c06af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 shutdown_connections 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 -- 172.21.15.83:0/3975122953 wait complete. 2024-09-10T18:54:58.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:58.563+0000 7f98027fc640 1 librados: shutdown 2024-09-10T18:54:58.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:58.687 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:58.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:58.789 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:58.821 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:54:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:58.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:54:58.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:58.962 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:58.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:59.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:59.098 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:54:59.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:59.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:54:59.203 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:59.230 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:54:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:59.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:54:59.334 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:59.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.347+0000 7ff083b49640 1 -- 172.21.15.83:0/1133830191 >> v1:172.21.15.165:6789/0 conn(0x7ff07c107e30 legacy=0x7ff07c10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:59.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.347+0000 7ff083b49640 1 -- 172.21.15.83:0/1133830191 shutdown_connections 2024-09-10T18:54:59.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.347+0000 7ff083b49640 1 -- 172.21.15.83:0/1133830191 >> 172.21.15.83:0/1133830191 conn(0x7ff07c075290 msgr2=0x7ff07c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:54:59.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.351+0000 7ff083b49640 1 -- 172.21.15.83:0/1133830191 shutdown_connections 2024-09-10T18:54:59.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.351+0000 7ff083b49640 1 -- 172.21.15.83:0/1133830191 wait complete. 2024-09-10T18:54:59.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 librados: starting msgr at 2024-09-10T18:54:59.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 librados: starting objecter 2024-09-10T18:54:59.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 Processor -- start 2024-09-10T18:54:59.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 -- start start 2024-09-10T18:54:59.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 librados: setting wanted keys 2024-09-10T18:54:59.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.355+0000 7ff083b49640 1 librados: calling monclient init 2024-09-10T18:54:59.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff083b49640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff07c1122d0 con 0x7ff07c1042a0 2024-09-10T18:54:59.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0820e6640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7ff07c107e30 0x7ff07c119cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:44060/0 (socket says 172.21.15.83:44060) 2024-09-10T18:54:59.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0820e6640 1 -- 172.21.15.83:0/1488512329 learned_addr learned my addr 172.21.15.83:0/1488512329 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:54:59.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff07c11b3d0 con 0x7ff07c107e30 2024-09-10T18:54:59.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff07c11b560 con 0x7ff07c1014e0 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 10 client.?.objecter ms_handle_connect 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 10 client.?.objecter ms_handle_connect 0x7ff07c1042a0 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715776873 0 0) 0x7ff07c11b3d0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680034e0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094902344 0 0) 0x7ff0680034e0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.359+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff07c11b3d0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.363+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff074002df0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.363+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1093737881 0 0) 0x7ff07c11b3d0 con 0x7ff07c107e30 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.363+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 >> v1:172.21.15.83:6790/0 conn(0x7ff07c1014e0 legacy=0x7ff07c111720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:59.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.363+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 >> v1:172.21.15.83:6789/0 conn(0x7ff07c1042a0 legacy=0x7ff07c116580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c11c700 con 0x7ff07c107e30 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff0727fc640 10 client.?.objecter ms_handle_connect 0x7ff07c1014e0 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c11b750 con 0x7ff07c107e30 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff083b49640 10 client.31915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c11bbe0 con 0x7ff07c107e30 2024-09-10T18:54:59.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff083b49640 1 librados: init done 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c0fcc60 con 0x7ff07c107e30 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.375+0000 7ff0727fc640 10 client.31915.objecter resend_mon_ops 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.375+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff074001ea0 con 0x7ff07c107e30 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.375+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff074003c10 con 0x7ff07c107e30 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.375+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff074003e80 con 0x7ff07c107e30 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.375+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(145..145 src has 69..145) v4 ==== 11850+0+0 (unknown 920772507 0 0) 0x7ff07408c940 con 0x7ff07c107e30 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.383+0000 7ff0727fc640 10 client.31915.objecter ms_dispatch 0x7ff07c000b90 osd_map(145..145 src has 69..145) v4 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.383+0000 7ff0727fc640 3 client.31915.objecter handle_osd_map got epochs [145,145] > 0 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.383+0000 7ff0727fc640 3 client.31915.objecter handle_osd_map decoding full epoch 145 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.383+0000 7ff0727fc640 20 client.31915.objecter dump_active .. 0 homeless 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.383+0000 7ff0727fc640 10 client.31915.objecter ms_handle_connect 0x7ff068071880 2024-09-10T18:54:59.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.387+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff074058070 con 0x7ff07c107e30 2024-09-10T18:54:59.533 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.306305+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.309635+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.393551+0000 mgr.y (mgr.24526) 217 : cluster [DBG] pgmap v194: 456 pgs: 128 unknown, 328 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.743937+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.748066+0000 osd.6 (osd.6) 90 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.965479+0000 osd.7 (osd.7) 125 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:55.969110+0000 osd.7 (osd.7) 126 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.476195+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.479663+0000 osd.1 (osd.1) 132 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:56.746058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:56.747364+0000 mon.a (mon.0) 1131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.747748+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-10T18:54:59.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:56.753318+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.83:0/4009038338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:56.758451+0000 mon.a (mon.0) 1133 : audit [INF] from='client.28627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:56.760391+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.83:0/704955493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:57.504164+0000 mon.a (mon.0) 1134 : audit [INF] from='client.28622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:57.514530+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[18629]: audit 2024-09-10T18:54:58.080087+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.83:0/3975122953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.306305+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.309635+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.393551+0000 mgr.y (mgr.24526) 217 : cluster [DBG] pgmap v194: 456 pgs: 128 unknown, 328 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.743937+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.748066+0000 osd.6 (osd.6) 90 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:54:59.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.965479+0000 osd.7 (osd.7) 125 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:55.969110+0000 osd.7 (osd.7) 126 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.476195+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.479663+0000 osd.1 (osd.1) 132 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:56.746058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:56.747364+0000 mon.a (mon.0) 1131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.747748+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:56.753318+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.83:0/4009038338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:56.758451+0000 mon.a (mon.0) 1133 : audit [INF] from='client.28627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:56.760391+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.83:0/704955493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:57.504164+0000 mon.a (mon.0) 1134 : audit [INF] from='client.28622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:57.514530+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:59.536 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:54:59 smithi083 bash[32381]: audit 2024-09-10T18:54:58.080087+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.83:0/3975122953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:59.554 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:54:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:59.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.306305+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.309635+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.393551+0000 mgr.y (mgr.24526) 217 : cluster [DBG] pgmap v194: 456 pgs: 128 unknown, 328 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.743937+0000 osd.6 (osd.6) 89 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.748066+0000 osd.6 (osd.6) 90 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.965479+0000 osd.7 (osd.7) 125 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:55.969110+0000 osd.7 (osd.7) 126 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:54:59.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.476195+0000 osd.1 (osd.1) 131 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.479663+0000 osd.1 (osd.1) 132 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:56.746058+0000 mon.a (mon.0) 1130 : audit [INF] from='client.25375 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi083-77992-12"}]': finished 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:56.747364+0000 mon.a (mon.0) 1131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.747748+0000 mon.a (mon.0) 1132 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:56.753318+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.83:0/4009038338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:56.758451+0000 mon.a (mon.0) 1133 : audit [INF] from='client.28627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:56.760391+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.83:0/704955493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:57.504164+0000 mon.a (mon.0) 1134 : audit [INF] from='client.28622 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:57.514530+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:54:59.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:54:59 smithi165 bash[22591]: audit 2024-09-10T18:54:58.080087+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.83:0/3975122953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:54:59.659 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:59.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:54:59.771+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7ff07c04bf60 con 0x7ff07c107e30 2024-09-10T18:54:59.983 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:54:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:54:59.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:00.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:00.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:00.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:00.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:56.313684+0000 mgr.y (mgr.24526) 218 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.894912+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.899382+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.951379+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:56.956605+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.394261+0000 mgr.y (mgr.24526) 219 : cluster [DBG] pgmap v196: 424 pgs: 32 creating+peering, 1 active+clean+scrubbing, 64 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T18:55:00.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:57.654342+0000 mgr.y (mgr.24526) 220 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.675998+0000 osd.3 (osd.3) 141 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.678476+0000 osd.3 (osd.3) 142 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.708276+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.a scrub starts 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.710804+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.a scrub ok 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.928327+0000 osd.0 (osd.0) 135 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:57.933665+0000 osd.0 (osd.0) 136 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.247794+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.a scrub starts 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.251782+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.a scrub ok 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.277225+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:58.309908+0000 mon.a (mon.0) 1135 : audit [INF] from='client.28627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:58.322413+0000 mon.a (mon.0) 1136 : audit [INF] from='client.28622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.323029+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:58.324091+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.423993+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.434928+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.c scrub starts 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.446699+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.c scrub ok 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:58.518997+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:58.546581+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3111658027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.702417+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:58.705778+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.274721+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.e scrub starts 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.278675+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.e scrub ok 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.325935+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.351313+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.83:0/1004801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.362203+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.83:0/2317553034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.395243+0000 mgr.y (mgr.24526) 221 : cluster [DBG] pgmap v199: 552 pgs: 32 creating+peering, 1 active+clean+scrubbing, 192 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.401560+0000 mon.a (mon.0) 1139 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.502445+0000 mon.a (mon.0) 1140 : audit [INF] from='client.31468 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.510770+0000 mon.a (mon.0) 1141 : audit [INF] from='client.30538 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.513064+0000 mon.a (mon.0) 1142 : audit [INF] from='client.30361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.522474+0000 mon.c (mon.2) 178 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.562471+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.565397+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.775043+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1488512329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: audit 2024-09-10T18:54:59.779457+0000 mon.a (mon.0) 1144 : audit [INF] from='client.31915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.922455+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-10T18:55:00.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:00 smithi165 bash[22591]: cluster 2024-09-10T18:54:59.926307+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-10T18:55:00.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.363+0000 7ff0727fc640 1 -- 172.21.15.83:0/1488512329 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v146) v1 ==== 113+0+0 (unknown 1235834447 0 0) 0x7ff074057a10 con 0x7ff07c107e30 2024-09-10T18:55:00.369 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T18:55:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 10 librados: watch_flush enter 2024-09-10T18:55:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 10 librados: watch_flush exit 2024-09-10T18:55:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 20 client.31915.objecter shutdown clearing up homeless session... 2024-09-10T18:55:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 10 client.31915.objecter successfully canceled tick 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff068071880 legacy=0x7ff068073d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 >> v1:172.21.15.165:6789/0 conn(0x7ff07c107e30 legacy=0x7ff07c119cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 shutdown_connections 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 >> 172.21.15.83:0/1488512329 conn(0x7ff07c075290 msgr2=0x7ff07c0f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.363+0000 7f72adffb640 1 -- 172.21.15.83:0/3111658027 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v146) v1 ==== 101+0+0 (unknown 4109755981 0 0) 0x7f72b40565d0 con 0x7f72b8108be0 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 10 librados: watch_flush enter 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 10 librados: watch_flush exit 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 20 client.31468.objecter shutdown clearing up homeless session... 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 10 client.31468.objecter successfully canceled tick 2024-09-10T18:55:00.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 >> v1:172.21.15.83:6800/3071793735 conn(0x7f72a40716d0 legacy=0x7f72a4073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 >> v1:172.21.15.165:6789/0 conn(0x7f72b8108be0 legacy=0x7f72b8107a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 shutdown_connections 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 >> 172.21.15.83:0/3111658027 conn(0x7f72b80723b0 msgr2=0x7f72b8111970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 shutdown_connections 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 -- 172.21.15.83:0/3111658027 wait complete. 2024-09-10T18:55:00.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.367+0000 7f72bf6d0640 1 librados: shutdown 2024-09-10T18:55:00.382 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-09-10T18:55:00.382 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 1.5 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-09-10T18:55:00.382 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.382 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.382 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2 pool(s) do not have an application enabled 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: flags noscrub,nodeep-scrub 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.383 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 8 pools, 196 pgs 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 11.224% pgs unknown 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 21.429% pgs not active 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 132 active+clean 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 42 creating+peering 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 22 unknown 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.384 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 707a4ada-6fa3-11ef-bcea-c7b262605968 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: nodeep-scrub flag(s) set 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 23 pool(s) do not have an application enabled 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: y(active, since 4m), standbys: x 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: flags nodeep-scrub 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-10T18:55:00.385 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 31 pools, 932 pgs 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 53.326% pgs unknown 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 32.511% pgs not active 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 497 unknown 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 303 creating+peering 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 132 active+clean 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-09-10T18:55:00.386 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.StatusFormat (8316 ms) 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.Status (21029 ms) 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] 4 tests from LibRadosService (34417 ms total) 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] Global test environment tear-down 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [==========] 4 tests from 1 test suite ran. (34417 ms total) 2024-09-10T18:55:00.387 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ PASSED ] 4 tests. 2024-09-10T18:55:00.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.379+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 shutdown_connections 2024-09-10T18:55:00.388 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.383+0000 7ff083b49640 1 -- 172.21.15.83:0/1488512329 wait complete. 2024-09-10T18:55:00.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:00.383+0000 7ff083b49640 1 librados: shutdown 2024-09-10T18:55:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:00.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:00.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:56.313684+0000 mgr.y (mgr.24526) 218 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.894912+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.899382+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.951379+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:56.956605+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.394261+0000 mgr.y (mgr.24526) 219 : cluster [DBG] pgmap v196: 424 pgs: 32 creating+peering, 1 active+clean+scrubbing, 64 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:57.654342+0000 mgr.y (mgr.24526) 220 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.675998+0000 osd.3 (osd.3) 141 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:55:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.678476+0000 osd.3 (osd.3) 142 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.708276+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.a scrub starts 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.710804+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.a scrub ok 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.928327+0000 osd.0 (osd.0) 135 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:57.933665+0000 osd.0 (osd.0) 136 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.247794+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.a scrub starts 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.251782+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.a scrub ok 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.277225+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:58.309908+0000 mon.a (mon.0) 1135 : audit [INF] from='client.28627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:58.322413+0000 mon.a (mon.0) 1136 : audit [INF] from='client.28622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.323029+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:58.324091+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.423993+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:55:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.434928+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.c scrub starts 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.446699+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.c scrub ok 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:58.518997+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:58.546581+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3111658027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.702417+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:58.705778+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.274721+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.e scrub starts 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.278675+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.e scrub ok 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.325935+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.351313+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.83:0/1004801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.362203+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.83:0/2317553034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.395243+0000 mgr.y (mgr.24526) 221 : cluster [DBG] pgmap v199: 552 pgs: 32 creating+peering, 1 active+clean+scrubbing, 192 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.401560+0000 mon.a (mon.0) 1139 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.502445+0000 mon.a (mon.0) 1140 : audit [INF] from='client.31468 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.510770+0000 mon.a (mon.0) 1141 : audit [INF] from='client.30538 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.513064+0000 mon.a (mon.0) 1142 : audit [INF] from='client.30361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.522474+0000 mon.c (mon.2) 178 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.562471+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.565397+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.775043+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1488512329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: audit 2024-09-10T18:54:59.779457+0000 mon.a (mon.0) 1144 : audit [INF] from='client.31915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.922455+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[18629]: cluster 2024-09-10T18:54:59.926307+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:56.313684+0000 mgr.y (mgr.24526) 218 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.894912+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:55:00.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.899382+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.951379+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:56.956605+0000 osd.7 (osd.7) 128 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.394261+0000 mgr.y (mgr.24526) 219 : cluster [DBG] pgmap v196: 424 pgs: 32 creating+peering, 1 active+clean+scrubbing, 64 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:57.654342+0000 mgr.y (mgr.24526) 220 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.675998+0000 osd.3 (osd.3) 141 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.678476+0000 osd.3 (osd.3) 142 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.708276+0000 osd.6 (osd.6) 91 : cluster [DBG] 4.a scrub starts 2024-09-10T18:55:00.524 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.710804+0000 osd.6 (osd.6) 92 : cluster [DBG] 4.a scrub ok 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.928327+0000 osd.0 (osd.0) 135 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:57.933665+0000 osd.0 (osd.0) 136 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.247794+0000 osd.5 (osd.5) 121 : cluster [DBG] 6.a scrub starts 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.251782+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.a scrub ok 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.277225+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:58.309908+0000 mon.a (mon.0) 1135 : audit [INF] from='client.28627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi083-77943-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:58.322413+0000 mon.a (mon.0) 1136 : audit [INF] from='client.28622 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi083-77640-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.323029+0000 mon.a (mon.0) 1137 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:58.324091+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.423993+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.434928+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.c scrub starts 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.446699+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.c scrub ok 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:58.518997+0000 mon.c (mon.2) 177 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:58.546581+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.83:0/3111658027' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.702417+0000 osd.3 (osd.3) 143 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:58.705778+0000 osd.3 (osd.3) 144 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.274721+0000 osd.4 (osd.4) 155 : cluster [DBG] 4.e scrub starts 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.278675+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.e scrub ok 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.325935+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.351313+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.83:0/1004801166' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.362203+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.83:0/2317553034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.395243+0000 mgr.y (mgr.24526) 221 : cluster [DBG] pgmap v199: 552 pgs: 32 creating+peering, 1 active+clean+scrubbing, 192 unknown, 327 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.401560+0000 mon.a (mon.0) 1139 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.502445+0000 mon.a (mon.0) 1140 : audit [INF] from='client.31468 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.510770+0000 mon.a (mon.0) 1141 : audit [INF] from='client.30538 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.513064+0000 mon.a (mon.0) 1142 : audit [INF] from='client.30361 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.522474+0000 mon.c (mon.2) 178 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:00.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.562471+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:00.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.565397+0000 mon.c (mon.2) 179 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:00.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.775043+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.83:0/1488512329' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: audit 2024-09-10T18:54:59.779457+0000 mon.a (mon.0) 1144 : audit [INF] from='client.31915 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T18:55:00.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.922455+0000 osd.7 (osd.7) 129 : cluster [DBG] 2.b scrub starts 2024-09-10T18:55:00.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:00 smithi083 bash[32381]: cluster 2024-09-10T18:54:59.926307+0000 osd.7 (osd.7) 130 : cluster [DBG] 2.b scrub ok 2024-09-10T18:55:00.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:00.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:00.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:00.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:00.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:00.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:01.447 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:01.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:01.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:01.629 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.321776+0000 mon.a (mon.0) 1145 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.322283+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.335963+0000 mon.a (mon.0) 1147 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.336080+0000 mon.a (mon.0) 1148 : audit [INF] from='client.31468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.336206+0000 mon.a (mon.0) 1149 : audit [INF] from='client.30538 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.336301+0000 mon.a (mon.0) 1150 : audit [INF] from='client.30361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.336369+0000 mon.a (mon.0) 1151 : audit [INF] from='client.31915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.336430+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.479320+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.484434+0000 mon.a (mon.0) 1153 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[18629]: audit 2024-09-10T18:55:00.525630+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.321776+0000 mon.a (mon.0) 1145 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.322283+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.335963+0000 mon.a (mon.0) 1147 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.336080+0000 mon.a (mon.0) 1148 : audit [INF] from='client.31468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:01.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.336206+0000 mon.a (mon.0) 1149 : audit [INF] from='client.30538 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.336301+0000 mon.a (mon.0) 1150 : audit [INF] from='client.30361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.336369+0000 mon.a (mon.0) 1151 : audit [INF] from='client.31915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.336430+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.479320+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.484434+0000 mon.a (mon.0) 1153 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:01.944 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:01 smithi083 bash[32381]: audit 2024-09-10T18:55:00.525630+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:01.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:55:02.084 INFO:tasks.workunit.client.0.smithi083.stdout:": "default" 2024-09-10T18:55:02.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: }, 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: "task_status": {} 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: } 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: attempt 1 of 20 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10980 ms) 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (36106 ms total) 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: 2024-09-10T18:55:02.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-10T18:55:02.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (36106 ms total) 2024-09-10T18:55:02.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.321776+0000 mon.a (mon.0) 1145 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.322283+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.335963+0000 mon.a (mon.0) 1147 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.336080+0000 mon.a (mon.0) 1148 : audit [INF] from='client.31468 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.336206+0000 mon.a (mon.0) 1149 : audit [INF] from='client.30538 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.336301+0000 mon.a (mon.0) 1150 : audit [INF] from='client.30361 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.336369+0000 mon.a (mon.0) 1151 : audit [INF] from='client.31915 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.336430+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.479320+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:02.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.484434+0000 mon.a (mon.0) 1153 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:02.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:01 smithi165 bash[22591]: audit 2024-09-10T18:55:00.525630+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:02.406 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:02.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:02.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:02.570 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:02.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:02.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:02.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:02.705 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:02.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:02.844 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:02.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:02.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T18:55:02.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:03.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:02] "GET /metrics HTTP/1.1" 200 254544 "" "Prometheus/2.43.0" 2024-09-10T18:55:03.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:03.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:03.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:03.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:03.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:03.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:03.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.195826+0000 osd.5 (osd.5) 123 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:55:03.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.199562+0000 osd.5 (osd.5) 124 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:55:03.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.299488+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.e scrub starts 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.303583+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.e scrub ok 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.909943+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:00.912154+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.415680+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.419326+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.530024+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.605657+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.608816+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.870576+0000 mon.a (mon.0) 1154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.870680+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.879967+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.83:0/1437774969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.880197+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.881493+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.83:0/2788138944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.886432+0000 mon.a (mon.0) 1156 : audit [INF] from='client.33169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.886641+0000 mon.a (mon.0) 1157 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:01.886928+0000 mon.a (mon.0) 1158 : audit [INF] from='client.33280 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:02.055474+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? v1:172.21.15.83:0/4179187246' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:03.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:03 smithi165 bash[22591]: audit 2024-09-10T18:55:02.536337+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.659 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:03.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:03.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:03.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.195826+0000 osd.5 (osd.5) 123 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:55:03.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.199562+0000 osd.5 (osd.5) 124 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:55:03.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.299488+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.e scrub starts 2024-09-10T18:55:03.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.303583+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.e scrub ok 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.909943+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:00.912154+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.415680+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.419326+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.530024+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.605657+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.608816+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.870576+0000 mon.a (mon.0) 1154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.870680+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.879967+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.83:0/1437774969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.880197+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.881493+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.83:0/2788138944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.886432+0000 mon.a (mon.0) 1156 : audit [INF] from='client.33169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.886641+0000 mon.a (mon.0) 1157 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:01.886928+0000 mon.a (mon.0) 1158 : audit [INF] from='client.33280 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:02.055474+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? v1:172.21.15.83:0/4179187246' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[18629]: audit 2024-09-10T18:55:02.536337+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.195826+0000 osd.5 (osd.5) 123 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.199562+0000 osd.5 (osd.5) 124 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.299488+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.e scrub starts 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.303583+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.e scrub ok 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.909943+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:00.912154+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.415680+0000 osd.2 (osd.2) 73 : cluster [DBG] 6.f scrub starts 2024-09-10T18:55:03.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.419326+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.530024+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.605657+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.608816+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.870576+0000 mon.a (mon.0) 1154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.870680+0000 mon.a (mon.0) 1155 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.879967+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.83:0/1437774969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.880197+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.881493+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.83:0/2788138944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.886432+0000 mon.a (mon.0) 1156 : audit [INF] from='client.33169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.886641+0000 mon.a (mon.0) 1157 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:01.886928+0000 mon.a (mon.0) 1158 : audit [INF] from='client.33280 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:02.055474+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? v1:172.21.15.83:0/4179187246' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:03.776 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:03 smithi083 bash[32381]: audit 2024-09-10T18:55:02.536337+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:03.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:03.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:03.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:03.925 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:03.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:04.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:04.224 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:04.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:04.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:04.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.400118+0000 mgr.y (mgr.24526) 222 : cluster [DBG] pgmap v201: 488 pgs: 2 creating+activating, 486 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.746279+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.b scrub starts 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:01.757801+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.b scrub ok 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:02.060071+0000 mgr.y (mgr.24526) 223 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.185518+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.d scrub starts 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.188826+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.d scrub ok 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.387801+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.390410+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.563968+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.566857+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:55:04.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.944515+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:02.948200+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.225295+0000 osd.4 (osd.4) 159 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.229133+0000 osd.5 (osd.5) 127 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.230408+0000 osd.4 (osd.4) 160 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.232905+0000 osd.5 (osd.5) 128 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.400770+0000 mgr.y (mgr.24526) 224 : cluster [DBG] pgmap v203: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.443665+0000 mon.a (mon.0) 1159 : audit [INF] from='client.33169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.443732+0000 mon.a (mon.0) 1160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.443799+0000 mon.a (mon.0) 1161 : audit [INF] from='client.33280 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: cluster 2024-09-10T18:55:03.444004+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.448650+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.83:0/466281865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.449109+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.83:0/594637325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.465604+0000 mon.a (mon.0) 1163 : audit [INF] from='client.35008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.465993+0000 mon.a (mon.0) 1164 : audit [INF] from='client.35014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.538533+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:04.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:03.614751+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[18629]: audit 2024-09-10T18:55:04.051283+0000 mon.a (mon.0) 1165 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.400118+0000 mgr.y (mgr.24526) 222 : cluster [DBG] pgmap v201: 488 pgs: 2 creating+activating, 486 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.746279+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.b scrub starts 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:01.757801+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.b scrub ok 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:02.060071+0000 mgr.y (mgr.24526) 223 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.185518+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.d scrub starts 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.188826+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.d scrub ok 2024-09-10T18:55:04.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.387801+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.390410+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.563968+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.566857+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.944515+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:02.948200+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.225295+0000 osd.4 (osd.4) 159 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.229133+0000 osd.5 (osd.5) 127 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.230408+0000 osd.4 (osd.4) 160 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.232905+0000 osd.5 (osd.5) 128 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.400770+0000 mgr.y (mgr.24526) 224 : cluster [DBG] pgmap v203: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.443665+0000 mon.a (mon.0) 1159 : audit [INF] from='client.33169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.443732+0000 mon.a (mon.0) 1160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.443799+0000 mon.a (mon.0) 1161 : audit [INF] from='client.33280 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: cluster 2024-09-10T18:55:03.444004+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.448650+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.83:0/466281865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.449109+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.83:0/594637325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.465604+0000 mon.a (mon.0) 1163 : audit [INF] from='client.35008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.465993+0000 mon.a (mon.0) 1164 : audit [INF] from='client.35014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.538533+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:03.614751+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.717 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:04 smithi083 bash[32381]: audit 2024-09-10T18:55:04.051283+0000 mon.a (mon.0) 1165 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.400118+0000 mgr.y (mgr.24526) 222 : cluster [DBG] pgmap v201: 488 pgs: 2 creating+activating, 486 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:55:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.746279+0000 osd.3 (osd.3) 145 : cluster [DBG] 6.b scrub starts 2024-09-10T18:55:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:01.757801+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.b scrub ok 2024-09-10T18:55:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:02.060071+0000 mgr.y (mgr.24526) 223 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.185518+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.d scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.188826+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.d scrub ok 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.387801+0000 osd.2 (osd.2) 75 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.390410+0000 osd.2 (osd.2) 76 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.563968+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.566857+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.944515+0000 osd.7 (osd.7) 131 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:02.948200+0000 osd.7 (osd.7) 132 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.225295+0000 osd.4 (osd.4) 159 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.229133+0000 osd.5 (osd.5) 127 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.230408+0000 osd.4 (osd.4) 160 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:55:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.232905+0000 osd.5 (osd.5) 128 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.400770+0000 mgr.y (mgr.24526) 224 : cluster [DBG] pgmap v203: 488 pgs: 64 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1000 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.443665+0000 mon.a (mon.0) 1159 : audit [INF] from='client.33169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.443732+0000 mon.a (mon.0) 1160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.443799+0000 mon.a (mon.0) 1161 : audit [INF] from='client.33280 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi083-77640-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: cluster 2024-09-10T18:55:03.444004+0000 mon.a (mon.0) 1162 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.448650+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.83:0/466281865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.449109+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.83:0/594637325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.465604+0000 mon.a (mon.0) 1163 : audit [INF] from='client.35008 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.465993+0000 mon.a (mon.0) 1164 : audit [INF] from='client.35014 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.538533+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:03.614751+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:04 smithi165 bash[22591]: audit 2024-09-10T18:55:04.051283+0000 mon.a (mon.0) 1165 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:04.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:55:05.102 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:05.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:05.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:05.240 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:55:05.240 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T18:55:05.241 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T18:55:05.241 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T18:55:05.241 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:55:05.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:05.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:05.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.224205+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.228340+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.444403+0000 mon.a (mon.0) 1166 : audit [INF] from='client.35008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.444488+0000 mon.a (mon.0) 1167 : audit [INF] from='client.35014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.444748+0000 mon.a (mon.0) 1168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.444810+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.454949+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.448 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.465341+0000 mon.a (mon.0) 1170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.525255+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.525335+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.542637+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.557508+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: audit 2024-09-10T18:55:04.558745+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.575612+0000 osd.6 (osd.6) 97 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.579034+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.762818+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[32381]: cluster 2024-09-10T18:55:04.765150+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.224205+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:55:05.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.228340+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.444403+0000 mon.a (mon.0) 1166 : audit [INF] from='client.35008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.444488+0000 mon.a (mon.0) 1167 : audit [INF] from='client.35014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.444748+0000 mon.a (mon.0) 1168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.444810+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.454949+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.465341+0000 mon.a (mon.0) 1170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.525255+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.525335+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.542637+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.557508+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: audit 2024-09-10T18:55:04.558745+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.575612+0000 osd.6 (osd.6) 97 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:55:05.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.579034+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:55:05.451 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.762818+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:55:05.451 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:05 smithi083 bash[18629]: cluster 2024-09-10T18:55:04.765150+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-10T18:55:05.501 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (14004 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (5263 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4925 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3002 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3609 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (5130 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3615 ms) 2024-09-10T18:55:05.502 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (39548 ms total) 2024-09-10T18:55:05.503 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: 2024-09-10T18:55:05.503 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-10T18:55:05.503 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (39548 ms total) 2024-09-10T18:55:05.503 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-10T18:55:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.224205+0000 osd.4 (osd.4) 161 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:55:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.228340+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:55:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.444403+0000 mon.a (mon.0) 1166 : audit [INF] from='client.35008 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi083-77648-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.444488+0000 mon.a (mon.0) 1167 : audit [INF] from='client.35014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-78130-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.444748+0000 mon.a (mon.0) 1168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.444810+0000 mon.a (mon.0) 1169 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.454949+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.465341+0000 mon.a (mon.0) 1170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]: dispatch 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.525255+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.525335+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.542637+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.557508+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: audit 2024-09-10T18:55:04.558745+0000 mon.a (mon.0) 1171 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]: dispatch 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.575612+0000 osd.6 (osd.6) 97 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.579034+0000 osd.6 (osd.6) 98 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.762818+0000 osd.3 (osd.3) 147 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:55:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:05 smithi165 bash[22591]: cluster 2024-09-10T18:55:04.765150+0000 osd.3 (osd.3) 148 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:55:05.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:05.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:05.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:05.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:05.954 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:05.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:06.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:06.090 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:06.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:06.193 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:06.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:06.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:06.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.401792+0000 mgr.y (mgr.24526) 225 : cluster [DBG] pgmap v206: 552 pgs: 128 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1011 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.441286+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.460480+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.460589+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.477322+0000 mon.a (mon.0) 1173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.477619+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]': finished 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.477830+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.484294+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3804266509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.526832+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.531704+0000 mon.a (mon.0) 1176 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.532548+0000 mon.a (mon.0) 1177 : audit [INF] from='client.37549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.548720+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.597190+0000 osd.6 (osd.6) 101 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.604331+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.629702+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.641087+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: audit 2024-09-10T18:55:05.659871+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.897065+0000 osd.7 (osd.7) 135 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[18629]: cluster 2024-09-10T18:55:05.902130+0000 osd.7 (osd.7) 136 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.401792+0000 mgr.y (mgr.24526) 225 : cluster [DBG] pgmap v206: 552 pgs: 128 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1011 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.441286+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.460480+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.460589+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:06.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.477322+0000 mon.a (mon.0) 1173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.477619+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]': finished 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.477830+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.484294+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3804266509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.526832+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.531704+0000 mon.a (mon.0) 1176 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.532548+0000 mon.a (mon.0) 1177 : audit [INF] from='client.37549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.548720+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.597190+0000 osd.6 (osd.6) 101 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.604331+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.629702+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.641087+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: audit 2024-09-10T18:55:05.659871+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:06.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.897065+0000 osd.7 (osd.7) 135 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:55:06.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:06 smithi083 bash[32381]: cluster 2024-09-10T18:55:05.902130+0000 osd.7 (osd.7) 136 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:55:06.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.401792+0000 mgr.y (mgr.24526) 225 : cluster [DBG] pgmap v206: 552 pgs: 128 unknown, 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1011 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.441286+0000 mon.a (mon.0) 1172 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.460480+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.460589+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.477322+0000 mon.a (mon.0) 1173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-7"}]': finished 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.477619+0000 mon.a (mon.0) 1174 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4"}]': finished 2024-09-10T18:55:06.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.477830+0000 mon.a (mon.0) 1175 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.484294+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.83:0/3804266509' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.526832+0000 mon.c (mon.2) 186 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.531704+0000 mon.a (mon.0) 1176 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.532548+0000 mon.a (mon.0) 1177 : audit [INF] from='client.37549 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.548720+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.597190+0000 osd.6 (osd.6) 101 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.604331+0000 osd.6 (osd.6) 102 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.629702+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.641087+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: audit 2024-09-10T18:55:05.659871+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.897065+0000 osd.7 (osd.7) 135 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:55:06.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:06 smithi165 bash[22591]: cluster 2024-09-10T18:55:05.902130+0000 osd.7 (osd.7) 136 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:55:06.878 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:06.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:07.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:07.554 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:07.558 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:07.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:07.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:08.039 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: cluster 2024-09-10T18:55:06.195190+0000 osd.4 (osd.4) 163 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: cluster 2024-09-10T18:55:06.199846+0000 osd.4 (osd.4) 164 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: cluster 2024-09-10T18:55:06.467225+0000 mon.a (mon.0) 1181 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.474413+0000 mon.a (mon.0) 1182 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.474562+0000 mon.a (mon.0) 1183 : audit [INF] from='client.37549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.474788+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: cluster 2024-09-10T18:55:06.475094+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.477709+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.491547+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.83:0/1161757064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.499166+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.503833+0000 mon.a (mon.0) 1187 : audit [INF] from='client.38674 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.504467+0000 mon.a (mon.0) 1188 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:06.551669+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[18629]: audit 2024-09-10T18:55:07.557178+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: cluster 2024-09-10T18:55:06.195190+0000 osd.4 (osd.4) 163 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: cluster 2024-09-10T18:55:06.199846+0000 osd.4 (osd.4) 164 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: cluster 2024-09-10T18:55:06.467225+0000 mon.a (mon.0) 1181 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.474413+0000 mon.a (mon.0) 1182 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.474562+0000 mon.a (mon.0) 1183 : audit [INF] from='client.37549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.474788+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: cluster 2024-09-10T18:55:06.475094+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:55:08.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.477709+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.491547+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.83:0/1161757064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.499166+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.503833+0000 mon.a (mon.0) 1187 : audit [INF] from='client.38674 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.504467+0000 mon.a (mon.0) 1188 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:06.551669+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:07 smithi083 bash[32381]: audit 2024-09-10T18:55:07.557178+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: cluster 2024-09-10T18:55:06.195190+0000 osd.4 (osd.4) 163 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: cluster 2024-09-10T18:55:06.199846+0000 osd.4 (osd.4) 164 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: cluster 2024-09-10T18:55:06.467225+0000 mon.a (mon.0) 1181 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.474413+0000 mon.a (mon.0) 1182 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6"}]': finished 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.474562+0000 mon.a (mon.0) 1183 : audit [INF] from='client.37549 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi083-77640-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.474788+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: cluster 2024-09-10T18:55:06.475094+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.477709+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.491547+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.83:0/1161757064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.499166+0000 mon.c (mon.2) 188 : audit [INF] from='client.? v1:172.21.15.83:0/2035315468' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.503833+0000 mon.a (mon.0) 1187 : audit [INF] from='client.38674 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.504467+0000 mon.a (mon.0) 1188 : audit [INF] from='client.25352 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T18:55:08.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:06.551669+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:07 smithi165 bash[22591]: audit 2024-09-10T18:55:07.557178+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:08.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:08.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:08.484 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:08.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:08.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:08.877 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:08.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:09.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:09.013 INFO:tasks.workunit.client.0.smithi083.stdout:pi_watch_notify_pp: flushed 2024-09-10T18:55:09.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3013 ms) 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: trying... 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363131168 notify_id 609885356034 notifier_gid 25352 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: timed out 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushed 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3020 ms) 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: List watches 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 2024-09-10T18:55:09.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363131168 notify_id 618475290630 notifier_gid 25352 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 done 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: watch_check 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: unwatch2 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: done 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3012 ms) 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: List watches 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94472363131168 notify_id 627065225223 notifier_gid 25352 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 done 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: watch_check 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: unwatch2 2024-09-10T18:55:09.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: done 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3031 ms) 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17717 ms total) 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (43124 ms total) 2024-09-10T18:55:09.016 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-10T18:55:09.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:09.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:09.175 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:07.219850+0000 osd.5 (osd.5) 129 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:07.224825+0000 osd.5 (osd.5) 130 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:07.402441+0000 mgr.y (mgr.24526) 226 : cluster [DBG] pgmap v209: 424 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 359 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:07.666363+0000 mgr.y (mgr.24526) 227 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:07.832964+0000 osd.7 (osd.7) 137 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:07.836139+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.559314+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:09.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.672723+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.673761+0000 mon.a (mon.0) 1189 : audit [INF] from='client.38674 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.673851+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.674543+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.675714+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.83:0/45913645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.686486+0000 mon.a (mon.0) 1192 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[18629]: audit 2024-09-10T18:55:08.704937+0000 mon.a (mon.0) 1193 : audit [INF] from='client.39853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:07.219850+0000 osd.5 (osd.5) 129 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:07.224825+0000 osd.5 (osd.5) 130 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:07.402441+0000 mgr.y (mgr.24526) 226 : cluster [DBG] pgmap v209: 424 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 359 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:55:09.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:07.666363+0000 mgr.y (mgr.24526) 227 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:07.832964+0000 osd.7 (osd.7) 137 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:07.836139+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.559314+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.672723+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.673761+0000 mon.a (mon.0) 1189 : audit [INF] from='client.38674 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.673851+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.674543+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.675714+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.83:0/45913645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.686486+0000 mon.a (mon.0) 1192 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:08 smithi083 bash[32381]: audit 2024-09-10T18:55:08.704937+0000 mon.a (mon.0) 1193 : audit [INF] from='client.39853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:09.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:09.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:07.219850+0000 osd.5 (osd.5) 129 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:55:09.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:07.224825+0000 osd.5 (osd.5) 130 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:55:09.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:07.402441+0000 mgr.y (mgr.24526) 226 : cluster [DBG] pgmap v209: 424 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 359 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:55:09.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:07.666363+0000 mgr.y (mgr.24526) 227 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:07.832964+0000 osd.7 (osd.7) 137 : cluster [DBG] 6.3 scrub starts 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:07.836139+0000 osd.7 (osd.7) 138 : cluster [DBG] 6.3 scrub ok 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.559314+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.672723+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.673761+0000 mon.a (mon.0) 1189 : audit [INF] from='client.38674 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi083-77648-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.673851+0000 mon.a (mon.0) 1190 : audit [INF] from='client.25352 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi083-78004-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.674543+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.675714+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.83:0/45913645' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.686486+0000 mon.a (mon.0) 1192 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.296 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:08 smithi165 bash[22591]: audit 2024-09-10T18:55:08.704937+0000 mon.a (mon.0) 1193 : audit [INF] from='client.39853 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:09.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:09.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.456 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:09.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.590 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:09.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:09.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:09.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.862 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:09.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.994 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:09.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:10.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.195821+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.199413+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.606773+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.609695+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: audit 2024-09-10T18:55:08.944765+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: audit 2024-09-10T18:55:08.944901+0000 mon.a (mon.0) 1195 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: audit 2024-09-10T18:55:08.945011+0000 mon.a (mon.0) 1196 : audit [INF] from='client.39853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: cluster 2024-09-10T18:55:08.945088+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:55:10.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: audit 2024-09-10T18:55:08.995278+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:10 smithi165 bash[22591]: audit 2024-09-10T18:55:09.509111+0000 mon.a (mon.0) 1198 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.137 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:10.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:10.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:10.347 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:10.360 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.195821+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T18:55:10.360 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.199413+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.606773+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.609695+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: audit 2024-09-10T18:55:08.944765+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: audit 2024-09-10T18:55:08.944901+0000 mon.a (mon.0) 1195 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: audit 2024-09-10T18:55:08.945011+0000 mon.a (mon.0) 1196 : audit [INF] from='client.39853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.361 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: cluster 2024-09-10T18:55:08.945088+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: audit 2024-09-10T18:55:08.995278+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[32381]: audit 2024-09-10T18:55:09.509111+0000 mon.a (mon.0) 1198 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.195821+0000 osd.5 (osd.5) 131 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.199413+0000 osd.5 (osd.5) 132 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.606773+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.609695+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: audit 2024-09-10T18:55:08.944765+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi083-77907-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:10.362 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: audit 2024-09-10T18:55:08.944901+0000 mon.a (mon.0) 1195 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: audit 2024-09-10T18:55:08.945011+0000 mon.a (mon.0) 1196 : audit [INF] from='client.39853 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:10.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: cluster 2024-09-10T18:55:08.945088+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-09-10T18:55:10.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: audit 2024-09-10T18:55:08.995278+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.363 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:10 smithi083 bash[18629]: audit 2024-09-10T18:55:09.509111+0000 mon.a (mon.0) 1198 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:10.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:10.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:10.598 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:10.938 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:10.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:11.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:11.177 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:09.166320+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:09.166427+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:09.170029+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:09.170731+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:09.403310+0000 mgr.y (mgr.24526) 228 : cluster [DBG] pgmap v212: 396 pgs: 1 active+clean+snaptrim, 72 unknown, 323 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:09.562491+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.098900+0000 mon.a (mon.0) 1199 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.119496+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.120173+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.83:0/1322748831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:10.175128+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:10.179120+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.582957+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: cluster 2024-09-10T18:55:10.717739+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.737707+0000 mon.a (mon.0) 1201 : audit [INF] from='client.42494 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[32381]: audit 2024-09-10T18:55:10.746181+0000 mon.a (mon.0) 1202 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:09.166320+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:09.166427+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:09.170029+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:09.170731+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:09.403310+0000 mgr.y (mgr.24526) 228 : cluster [DBG] pgmap v212: 396 pgs: 1 active+clean+snaptrim, 72 unknown, 323 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:09.562491+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.098900+0000 mon.a (mon.0) 1199 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.119496+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.120173+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.83:0/1322748831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.179 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:10.175128+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:55:11.180 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:10.179120+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:55:11.180 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.582957+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.180 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: cluster 2024-09-10T18:55:10.717739+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:55:11.180 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.737707+0000 mon.a (mon.0) 1201 : audit [INF] from='client.42494 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.180 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:11 smithi083 bash[18629]: audit 2024-09-10T18:55:10.746181+0000 mon.a (mon.0) 1202 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:09.166320+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:09.166427+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.d scrub starts 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:09.170029+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.d scrub ok 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:09.170731+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:09.403310+0000 mgr.y (mgr.24526) 228 : cluster [DBG] pgmap v212: 396 pgs: 1 active+clean+snaptrim, 72 unknown, 323 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:11.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:09.562491+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.098900+0000 mon.a (mon.0) 1199 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.119496+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.120173+0000 mon.c (mon.2) 193 : audit [INF] from='client.? v1:172.21.15.83:0/1322748831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:10.175128+0000 osd.5 (osd.5) 135 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:10.179120+0000 osd.5 (osd.5) 136 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.582957+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: cluster 2024-09-10T18:55:10.717739+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.737707+0000 mon.a (mon.0) 1201 : audit [INF] from='client.42494 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:11.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[22591]: audit 2024-09-10T18:55:10.746181+0000 mon.a (mon.0) 1202 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:11.593 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:11.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:11.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:11.753 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:11.771 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:11.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:11.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:11.909 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:11.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:12.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:12.038 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:12.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:12.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:12.294 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:12.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:12.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: cluster 2024-09-10T18:55:10.694847+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:55:12.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: cluster 2024-09-10T18:55:10.697550+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:55:12.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.114313+0000 mon.a (mon.0) 1203 : audit [INF] from='client.42494 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:12.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.114614+0000 mon.a (mon.0) 1204 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:12.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: cluster 2024-09-10T18:55:11.114928+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:55:12.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.121594+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.83:0/3758889687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.127723+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.129746+0000 mon.a (mon.0) 1206 : audit [INF] from='client.42491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.143995+0000 mon.a (mon.0) 1207 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:11.590535+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:12.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: cluster 2024-09-10T18:55:11.598154+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:55:12.367 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: cluster 2024-09-10T18:55:11.602333+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:55:12.368 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.071101+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:12.368 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.078823+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:12.368 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.081630+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:12.368 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.085596+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:12.368 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.089557+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:12.369 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.092503+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:12.369 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.093399+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:12.369 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:12 smithi165 bash[22591]: audit 2024-09-10T18:55:12.095758+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:12.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: cluster 2024-09-10T18:55:10.694847+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:55:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: cluster 2024-09-10T18:55:10.697550+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:55:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.114313+0000 mon.a (mon.0) 1203 : audit [INF] from='client.42494 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.114614+0000 mon.a (mon.0) 1204 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: cluster 2024-09-10T18:55:11.114928+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.121594+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.83:0/3758889687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.127723+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.129746+0000 mon.a (mon.0) 1206 : audit [INF] from='client.42491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.143995+0000 mon.a (mon.0) 1207 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:11.590535+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: cluster 2024-09-10T18:55:11.598154+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: cluster 2024-09-10T18:55:11.602333+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.071101+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.078823+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:12.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.081630+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.085596+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.089557+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.092503+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.093399+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18629]: audit 2024-09-10T18:55:12.095758+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: cluster 2024-09-10T18:55:10.694847+0000 osd.3 (osd.3) 149 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: cluster 2024-09-10T18:55:10.697550+0000 osd.3 (osd.3) 150 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.114313+0000 mon.a (mon.0) 1203 : audit [INF] from='client.42494 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi083-77648-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.114614+0000 mon.a (mon.0) 1204 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:12.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: cluster 2024-09-10T18:55:11.114928+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.121594+0000 mon.c (mon.2) 195 : audit [INF] from='client.? v1:172.21.15.83:0/3758889687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.127723+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.129746+0000 mon.a (mon.0) 1206 : audit [INF] from='client.42491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.143995+0000 mon.a (mon.0) 1207 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]: dispatch 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:11.590535+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: cluster 2024-09-10T18:55:11.598154+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: cluster 2024-09-10T18:55:11.602333+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:55:12.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.071101+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.078823+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.081630+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.085596+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.089557+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.092503+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.093399+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:12.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[32381]: audit 2024-09-10T18:55:12.095758+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:12.951 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:12.955 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:12] "GET /metrics HTTP/1.1" 200 254544 "" "Prometheus/2.43.0" 2024-09-10T18:55:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:12.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:13.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:13.115 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:13.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:13.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:13.577 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:13.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:13.595 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: cluster 2024-09-10T18:55:11.404025+0000 mgr.y (mgr.24526) 229 : cluster [DBG] pgmap v215: 460 pgs: 16 creating+peering, 48 unknown, 396 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:13.595 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.078469+0000 mgr.y (mgr.24526) 230 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:13.595 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.084639+0000 mgr.y (mgr.24526) 231 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.086273+0000 mgr.y (mgr.24526) 232 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.090310+0000 mgr.y (mgr.24526) 233 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.094304+0000 mgr.y (mgr.24526) 234 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.096431+0000 mgr.y (mgr.24526) 235 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.097505+0000 mgr.y (mgr.24526) 236 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.102322+0000 mgr.y (mgr.24526) 237 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.102954+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.106708+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: cluster 2024-09-10T18:55:12.107077+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: cluster 2024-09-10T18:55:12.108139+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.109551+0000 mgr.y (mgr.24526) 238 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.114293+0000 mgr.y (mgr.24526) 239 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.131565+0000 mon.a (mon.0) 1210 : audit [INF] from='client.42491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.131771+0000 mon.a (mon.0) 1211 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]': finished 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: cluster 2024-09-10T18:55:12.131978+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.362566+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.367937+0000 mon.a (mon.0) 1213 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.597 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[18629]: audit 2024-09-10T18:55:12.593352+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: cluster 2024-09-10T18:55:11.404025+0000 mgr.y (mgr.24526) 229 : cluster [DBG] pgmap v215: 460 pgs: 16 creating+peering, 48 unknown, 396 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.078469+0000 mgr.y (mgr.24526) 230 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.084639+0000 mgr.y (mgr.24526) 231 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.086273+0000 mgr.y (mgr.24526) 232 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.090310+0000 mgr.y (mgr.24526) 233 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.094304+0000 mgr.y (mgr.24526) 234 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.096431+0000 mgr.y (mgr.24526) 235 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.097505+0000 mgr.y (mgr.24526) 236 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.102322+0000 mgr.y (mgr.24526) 237 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:13.598 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.102954+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.106708+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: cluster 2024-09-10T18:55:12.107077+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: cluster 2024-09-10T18:55:12.108139+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.109551+0000 mgr.y (mgr.24526) 238 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.114293+0000 mgr.y (mgr.24526) 239 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.131565+0000 mon.a (mon.0) 1210 : audit [INF] from='client.42491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.131771+0000 mon.a (mon.0) 1211 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]': finished 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: cluster 2024-09-10T18:55:12.131978+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.362566+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.367937+0000 mon.a (mon.0) 1213 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.599 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:13 smithi083 bash[32381]: audit 2024-09-10T18:55:12.593352+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:13.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: cluster 2024-09-10T18:55:11.404025+0000 mgr.y (mgr.24526) 229 : cluster [DBG] pgmap v215: 460 pgs: 16 creating+peering, 48 unknown, 396 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:13.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.078469+0000 mgr.y (mgr.24526) 230 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.0"}]: dispatch 2024-09-10T18:55:13.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.084639+0000 mgr.y (mgr.24526) 231 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.1"}]: dispatch 2024-09-10T18:55:13.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.086273+0000 mgr.y (mgr.24526) 232 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.2"}]: dispatch 2024-09-10T18:55:13.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.090310+0000 mgr.y (mgr.24526) 233 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.3"}]: dispatch 2024-09-10T18:55:13.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.094304+0000 mgr.y (mgr.24526) 234 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.4"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.096431+0000 mgr.y (mgr.24526) 235 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.5"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.097505+0000 mgr.y (mgr.24526) 236 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.6"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.102322+0000 mgr.y (mgr.24526) 237 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.7"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.102954+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.106708+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: cluster 2024-09-10T18:55:12.107077+0000 mon.a (mon.0) 1208 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: cluster 2024-09-10T18:55:12.108139+0000 mon.a (mon.0) 1209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:13.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.109551+0000 mgr.y (mgr.24526) 238 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.8"}]: dispatch 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.114293+0000 mgr.y (mgr.24526) 239 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "11.9"}]: dispatch 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.131565+0000 mon.a (mon.0) 1210 : audit [INF] from='client.42491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi083-77640-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.131771+0000 mon.a (mon.0) 1211 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-9", "mode": "writeback"}]': finished 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: cluster 2024-09-10T18:55:12.131978+0000 mon.a (mon.0) 1212 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.362566+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.367937+0000 mon.a (mon.0) 1213 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:13.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:13 smithi165 bash[22591]: audit 2024-09-10T18:55:12.593352+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:13.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:13.727 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:13.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:13.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:14.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:14.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:14.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:14.422 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:14.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:14.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:14.785 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:14.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:14.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: audit 2024-09-10T18:55:13.322191+0000 mon.a (mon.0) 1214 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: audit 2024-09-10T18:55:13.330701+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: audit 2024-09-10T18:55:13.338954+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: audit 2024-09-10T18:55:13.594391+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: cluster 2024-09-10T18:55:13.914442+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:55:14.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[18629]: audit 2024-09-10T18:55:13.930004+0000 mon.a (mon.0) 1216 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: audit 2024-09-10T18:55:13.322191+0000 mon.a (mon.0) 1214 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: audit 2024-09-10T18:55:13.330701+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: audit 2024-09-10T18:55:13.338954+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: audit 2024-09-10T18:55:13.594391+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: cluster 2024-09-10T18:55:13.914442+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:55:14.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:14 smithi083 bash[32381]: audit 2024-09-10T18:55:13.930004+0000 mon.a (mon.0) 1216 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:14.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:14.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:15.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: audit 2024-09-10T18:55:13.322191+0000 mon.a (mon.0) 1214 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: audit 2024-09-10T18:55:13.330701+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: audit 2024-09-10T18:55:13.338954+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: audit 2024-09-10T18:55:13.594391+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: cluster 2024-09-10T18:55:13.914442+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-09-10T18:55:15.076 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:14 smithi165 bash[22591]: audit 2024-09-10T18:55:13.930004+0000 mon.a (mon.0) 1216 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.095 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:15.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:15.235 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:15.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.369 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:15.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:15.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:15.504 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:15.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:15.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:15.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:15.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:13.404599+0000 mgr.y (mgr.24526) 240 : cluster [DBG] pgmap v218: 396 pgs: 32 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T18:55:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:13.649752+0000 osd.3 (osd.3) 151 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:55:15.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:13.693926+0000 osd.3 (osd.3) 152 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:13.945748+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.83:0/204687692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.012499+0000 osd.0 (osd.0) 139 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.143271+0000 osd.0 (osd.0) 140 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.531038+0000 osd.6 (osd.6) 107 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.556418+0000 osd.2 (osd.2) 77 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.603090+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.607451+0000 osd.6 (osd.6) 108 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.645459+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.653893+0000 mon.a (mon.0) 1218 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.655049+0000 mon.a (mon.0) 1219 : audit [INF] from='client.46267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.672556+0000 osd.2 (osd.2) 78 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.685852+0000 mon.a (mon.0) 1220 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:15.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.701757+0000 mon.c (mon.2) 200 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.931347+0000 mon.a (mon.0) 1221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.939130+0000 mon.a (mon.0) 1222 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.939272+0000 mon.a (mon.0) 1223 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.939384+0000 mon.a (mon.0) 1224 : audit [INF] from='client.46267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:14.939439+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:14.965151+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.83:0/93947423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.126717+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.131684+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:15.377852+0000 mon.a (mon.0) 1226 : audit [INF] from='client.46817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:15 smithi165 bash[22591]: audit 2024-09-10T18:55:15.604547+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:15.912 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:15.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:13.404599+0000 mgr.y (mgr.24526) 240 : cluster [DBG] pgmap v218: 396 pgs: 32 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T18:55:15.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:13.649752+0000 osd.3 (osd.3) 151 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:55:15.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:13.693926+0000 osd.3 (osd.3) 152 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:55:15.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:13.945748+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.83:0/204687692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.012499+0000 osd.0 (osd.0) 139 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.143271+0000 osd.0 (osd.0) 140 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.531038+0000 osd.6 (osd.6) 107 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.556418+0000 osd.2 (osd.2) 77 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.603090+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.607451+0000 osd.6 (osd.6) 108 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.645459+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.653893+0000 mon.a (mon.0) 1218 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.655049+0000 mon.a (mon.0) 1219 : audit [INF] from='client.46267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.672556+0000 osd.2 (osd.2) 78 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.685852+0000 mon.a (mon.0) 1220 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.701757+0000 mon.c (mon.2) 200 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.931347+0000 mon.a (mon.0) 1221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:15.915 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.939130+0000 mon.a (mon.0) 1222 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.939272+0000 mon.a (mon.0) 1223 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.939384+0000 mon.a (mon.0) 1224 : audit [INF] from='client.46267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:14.939439+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:14.965151+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.83:0/93947423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.126717+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.131684+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:15.377852+0000 mon.a (mon.0) 1226 : audit [INF] from='client.46817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[18629]: audit 2024-09-10T18:55:15.604547+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:13.404599+0000 mgr.y (mgr.24526) 240 : cluster [DBG] pgmap v218: 396 pgs: 32 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:13.649752+0000 osd.3 (osd.3) 151 : cluster [DBG] 11.9 deep-scrub starts 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:13.693926+0000 osd.3 (osd.3) 152 : cluster [DBG] 11.9 deep-scrub ok 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:13.945748+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.83:0/204687692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.012499+0000 osd.0 (osd.0) 139 : cluster [DBG] 11.6 deep-scrub starts 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.143271+0000 osd.0 (osd.0) 140 : cluster [DBG] 11.6 deep-scrub ok 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.531038+0000 osd.6 (osd.6) 107 : cluster [DBG] 11.3 deep-scrub starts 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.556418+0000 osd.2 (osd.2) 77 : cluster [DBG] 11.2 deep-scrub starts 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.603090+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.607451+0000 osd.6 (osd.6) 108 : cluster [DBG] 11.3 deep-scrub ok 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.645459+0000 mon.a (mon.0) 1217 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.653893+0000 mon.a (mon.0) 1218 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]: dispatch 2024-09-10T18:55:15.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.655049+0000 mon.a (mon.0) 1219 : audit [INF] from='client.46267 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.672556+0000 osd.2 (osd.2) 78 : cluster [DBG] 11.2 deep-scrub ok 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.685852+0000 mon.a (mon.0) 1220 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.701757+0000 mon.c (mon.2) 200 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.931347+0000 mon.a (mon.0) 1221 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.939130+0000 mon.a (mon.0) 1222 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77877-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.939272+0000 mon.a (mon.0) 1223 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-9"}]': finished 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.939384+0000 mon.a (mon.0) 1224 : audit [INF] from='client.46267 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi083-77648-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:14.939439+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:14.965151+0000 mon.c (mon.2) 201 : audit [INF] from='client.? v1:172.21.15.83:0/93947423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.126717+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.131684+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:15.377852+0000 mon.a (mon.0) 1226 : audit [INF] from='client.46817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:15.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:15 smithi083 bash[32381]: audit 2024-09-10T18:55:15.604547+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:15.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] Global test environment set-up. 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-10T18:55:15.975 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:54:25.824+0000 7f0a7465e6c0 -1 failed for service _ceph-mon._tcp 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:54:25.824+0000 7f0a7465e6c0 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (50 ms) 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5010 ms) 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5060 ms total) 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x559fda6744b0 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x559fda678cb0 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 2 aios 2024-09-10T18:55:15.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x559fda6744b0 2024-09-10T18:55:15.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x559fda678cb0 2024-09-10T18:55:15.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: done. 2024-09-10T18:55:15.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (10193 ms) 2024-09-10T18:55:15.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (10193 ms total) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (1 ms) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.Exec (5 ms) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6 ms) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6919 ms) 2024-09-10T18:55:15.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22862 ms) 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 8 tests from LibRadosMisc (29799 ms total) 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] Global test environment tear-down 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (50183 ms total) 2024-09-10T18:55:15.982 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-10T18:55:16.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.044 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:16.089 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:16.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.183 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:16.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.322 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:16.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:16.461 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:16.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.597 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:16.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:16.733 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:16.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:16.868 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:16.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:16.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:17.000 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:17.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.405526+0000 mgr.y (mgr.24526) 241 : cluster [DBG] pgmap v221: 460 pgs: 96 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.646357+0000 osd.3 (osd.3) 153 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.650500+0000 osd.3 (osd.3) 154 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: audit 2024-09-10T18:55:15.953495+0000 mon.a (mon.0) 1227 : audit [INF] from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:15.953872+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: audit 2024-09-10T18:55:15.957639+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: audit 2024-09-10T18:55:15.964737+0000 mon.a (mon.0) 1229 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.114895+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:55:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.118637+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:55:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.314280+0000 osd.1 (osd.1) 133 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:55:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.317587+0000 osd.1 (osd.1) 134 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:55:17.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:16 smithi165 bash[22591]: audit 2024-09-10T18:55:16.605860+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:17.147 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:17.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:17.289 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.405526+0000 mgr.y (mgr.24526) 241 : cluster [DBG] pgmap v221: 460 pgs: 96 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.646357+0000 osd.3 (osd.3) 153 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.650500+0000 osd.3 (osd.3) 154 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: audit 2024-09-10T18:55:15.953495+0000 mon.a (mon.0) 1227 : audit [INF] from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:15.953872+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: audit 2024-09-10T18:55:15.957639+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: audit 2024-09-10T18:55:15.964737+0000 mon.a (mon.0) 1229 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.114895+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:55:17.315 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.118637+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.314280+0000 osd.1 (osd.1) 133 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.317587+0000 osd.1 (osd.1) 134 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[18629]: audit 2024-09-10T18:55:16.605860+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.405526+0000 mgr.y (mgr.24526) 241 : cluster [DBG] pgmap v221: 460 pgs: 96 unknown, 364 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.646357+0000 osd.3 (osd.3) 153 : cluster [DBG] 11.8 deep-scrub starts 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.650500+0000 osd.3 (osd.3) 154 : cluster [DBG] 11.8 deep-scrub ok 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: audit 2024-09-10T18:55:15.953495+0000 mon.a (mon.0) 1227 : audit [INF] from='client.46817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi083-77640-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:15.953872+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: audit 2024-09-10T18:55:15.957639+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: audit 2024-09-10T18:55:15.964737+0000 mon.a (mon.0) 1229 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T18:55:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.114895+0000 osd.4 (osd.4) 169 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:55:17.317 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.118637+0000 osd.4 (osd.4) 170 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:55:17.317 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.314280+0000 osd.1 (osd.1) 133 : cluster [DBG] 11.0 deep-scrub starts 2024-09-10T18:55:17.317 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.317587+0000 osd.1 (osd.1) 134 : cluster [DBG] 11.0 deep-scrub ok 2024-09-10T18:55:17.317 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:16 smithi083 bash[32381]: audit 2024-09-10T18:55:16.605860+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:17.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:17.420 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:17.524 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:17.554 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:17.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:17.693 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2019148345 >> v1:172.21.15.83:6790/0 conn(0x7fa9b0101530 legacy=0x7fa9b0101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2019148345 shutdown_connections 2024-09-10T18:55:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2019148345 >> 172.21.15.83:0/2019148345 conn(0x7fa9b00f70b0 msgr2=0x7fa9b00f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2019148345 shutdown_connections 2024-09-10T18:55:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2019148345 wait complete. 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 librados: starting msgr at 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 librados: starting objecter 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 Processor -- start 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- start start 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 librados: setting wanted keys 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 librados: calling monclient init 2024-09-10T18:55:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9b011c290 con 0x7fa9b0101530 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9b011d430 con 0x7fa9b0107e00 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9b011e610 con 0x7fa9b01042a0 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9affff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9b0101530 0x7fa9b01071e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36970/0 (socket says 172.21.15.83:36970) 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9af7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa9b01042a0 0x7fa9b0117440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45498/0 (socket says 172.21.15.83:45498) 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9affff640 1 -- 172.21.15.83:0/2065090936 learned_addr learned my addr 172.21.15.83:0/2065090936 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter ms_handle_connect 0x7fa9b01042a0 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter ms_handle_connect 0x7fa9b0101530 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830797222 0 0) 0x7fa9b011e610 con 0x7fa9b01042a0 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa99c003720 con 0x7fa9b01042a0 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226173628 0 0) 0x7fa9b011c290 con 0x7fa9b0101530 2024-09-10T18:55:17.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b011e610 con 0x7fa9b0101530 2024-09-10T18:55:17.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter ms_handle_connect 0x7fa9b0107e00 2024-09-10T18:55:17.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153908407 0 0) 0x7fa99c003720 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b011c290 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9980019d0 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673806141 0 0) 0x7fa9b011c290 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.165:6789/0 conn(0x7fa9b0107e00 legacy=0x7fa9b011ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6789/0 conn(0x7fa9b0101530 legacy=0x7fa9b01071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b011f810 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b011c480 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa998002120 con 0x7fa9b01042a0 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 10 client.48545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa998005a80 con 0x7fa9b01042a0 2024-09-10T18:55:17.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b011c8d0 con 0x7fa9b01042a0 2024-09-10T18:55:17.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.779+0000 7fa9b6710640 1 librados: init done 2024-09-10T18:55:17.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa9b00fcc30 con 0x7fa9b01042a0 2024-09-10T18:55:17.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa998001d10 con 0x7fa9b01042a0 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(161..161 src has 69..161) v4 ==== 10777+0+0 (unknown 4278337427 0 0) 0x7fa99808b0d0 con 0x7fa9b01042a0 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 osd_map(161..161 src has 69..161) v4 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 3 client.48545.objecter handle_osd_map got epochs [161,161] > 0 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 3 client.48545.objecter handle_osd_map decoding full epoch 161 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 20 client.48545.objecter dump_active .. 0 homeless 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v161) v1 ==== 54+0+16 (unknown 1338815507 0 2420000192) 0x7fa99808d490 con 0x7fa9b01042a0 2024-09-10T18:55:17.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa99c075500 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 10 client.48545.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 1 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 1 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 0 1 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 4 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 10 client.48545.objecter _send_command 1 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.783+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b0106680 con 0x7fa9b011e7a0 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b011e7a0 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b0106680 con 0x7fa9b011e7a0 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 10 client.48545.objecter _finish_command 1 = system:0 2024-09-10T18:55:17.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 4 2024-09-10T18:55:17.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.787+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 0 1 2024-09-10T18:55:17.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- 172.21.15.83:0/597573710 >> v1:172.21.15.83:6790/0 conn(0x7f5b30105370 legacy=0x7f5b30107750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- 172.21.15.83:0/597573710 shutdown_connections 2024-09-10T18:55:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- 172.21.15.83:0/597573710 >> 172.21.15.83:0/597573710 conn(0x7f5b300f81d0 msgr2=0x7f5b300fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- 172.21.15.83:0/597573710 shutdown_connections 2024-09-10T18:55:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- 172.21.15.83:0/597573710 wait complete. 2024-09-10T18:55:17.796 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 librados: starting msgr at 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 librados: starting objecter 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 Processor -- start 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- start start 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 librados: setting wanted keys 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 librados: calling monclient init 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b30101b80 con 0x7f5b30105370 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b30101d30 con 0x7f5b30108f00 2024-09-10T18:55:17.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b373c2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b30101f20 con 0x7f5b301025b0 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b3515e640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5b301025b0 0x7f5b30101380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45516/0 (socket says 172.21.15.83:45516) 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b3515e640 1 -- 172.21.15.83:0/362110219 learned_addr learned my addr 172.21.15.83:0/362110219 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b3495d640 1 --1- 172.21.15.83:0/362110219 >> v1:172.21.15.83:6789/0 conn(0x7f5b30105370 0x7f5b30119920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36988/0 (socket says 172.21.15.83:36988) 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter ms_handle_connect 0x7f5b30105370 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter ms_handle_connect 0x7f5b301025b0 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920022065 0 0) 0x7f5b30101f20 con 0x7f5b301025b0 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c003120 con 0x7f5b301025b0 2024-09-10T18:55:17.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter ms_handle_connect 0x7f5b30108f00 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926594541 0 0) 0x7f5b30101b80 con 0x7f5b30105370 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30101f20 con 0x7f5b30105370 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427165268 0 0) 0x7f5b30101d30 con 0x7f5b30108f00 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30101b80 con 0x7f5b30108f00 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16659335 0 0) 0x7f5b1c003120 con 0x7f5b301025b0 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b30101d30 con 0x7f5b301025b0 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5b2c002cd0 con 0x7f5b301025b0 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 578521012 0 0) 0x7f5b30101f20 con 0x7f5b30105370 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b1c003120 con 0x7f5b30105370 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.791+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5b20002130 con 0x7f5b30105370 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1151898076 0 0) 0x7f5b30101d30 con 0x7f5b301025b0 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 >> v1:172.21.15.165:6789/0 conn(0x7f5b30108f00 legacy=0x7f5b3011d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 >> v1:172.21.15.83:6789/0 conn(0x7f5b30105370 legacy=0x7f5b30119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b301217a0 con 0x7f5b301025b0 2024-09-10T18:55:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3011e7d0 con 0x7f5b301025b0 2024-09-10T18:55:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b373c2640 10 client.48551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3011ec20 con 0x7f5b301025b0 2024-09-10T18:55:17.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b373c2640 1 librados: init done 2024-09-10T18:55:17.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5b2c001ad0 con 0x7f5b301025b0 2024-09-10T18:55:17.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5b2c005e50 con 0x7f5b301025b0 2024-09-10T18:55:17.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b14005010 con 0x7f5b301025b0 2024-09-10T18:55:17.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5b2c001670 con 0x7f5b301025b0 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(161..161 src has 69..161) v4 ==== 10777+0+0 (unknown 4278337427 0 0) 0x7f5b2c08b4d0 con 0x7f5b301025b0 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 10 client.48551.objecter ms_dispatch 0x7f5b30000b90 osd_map(161..161 src has 69..161) v4 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 3 client.48551.objecter handle_osd_map got epochs [161,161] > 0 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 3 client.48551.objecter handle_osd_map decoding full epoch 161 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.795+0000 7f5b25ffb640 20 client.48551.objecter dump_active .. 0 homeless 2024-09-10T18:55:17.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.799+0000 7f5b25ffb640 10 client.48551.objecter ms_handle_connect 0x7f5b1c071670 2024-09-10T18:55:17.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.799+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5b2c056fb0 con 0x7f5b301025b0 2024-09-10T18:55:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 10 client.48545.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 2 2024-09-10T18:55:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 2 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b010d9a0 osd=0 3 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 0 2 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 4 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 10 client.48545.objecter _send_command 2 2024-09-10T18:55:17.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.811+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b011f280 con 0x7fa9b011e7a0 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b011f280 con 0x7fa9b011e7a0 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 10 client.48545.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 4 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 0 2 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:55:17.820 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 10 client.48545.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 3 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 3 2024-09-10T18:55:17.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 1 3 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 4 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 10 client.48545.objecter _send_command 3 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b010a280 con 0x7fa9b01097a0 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b01097a0 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:17.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b010a280 con 0x7fa9b01097a0 2024-09-10T18:55:17.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:55:17.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 10 client.48545.objecter _finish_command 3 = system:0 2024-09-10T18:55:17.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 4 2024-09-10T18:55:17.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.815+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 1 3 2024-09-10T18:55:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 10 client.48545.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 4 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 4 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b011f630 osd=1 3 2024-09-10T18:55:17.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 1 4 2024-09-10T18:55:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 4 2024-09-10T18:55:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 10 client.48545.objecter _send_command 4 2024-09-10T18:55:17.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b002ab10 con 0x7fa9b01097a0 2024-09-10T18:55:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b002ab10 con 0x7fa9b01097a0 2024-09-10T18:55:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 10 client.48545.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 4 2024-09-10T18:55:17.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 1 4 2024-09-10T18:55:17.842 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:55:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:55:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 10 client.48545.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 5 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 5 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 2 5 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 4 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 10 client.48545.objecter _send_command 5 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b0125b30 con 0x7fa9b0122030 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.835+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b0122030 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.839+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b0125b30 con 0x7fa9b0122030 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.839+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.839+0000 7fa9acff9640 10 client.48545.objecter _finish_command 5 = system:0 2024-09-10T18:55:17.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.839+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 4 2024-09-10T18:55:17.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.839+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 2 5 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 10 client.48545.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 6 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 6 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b01219b0 osd=2 3 2024-09-10T18:55:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 2 6 2024-09-10T18:55:17.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 4 2024-09-10T18:55:17.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 10 client.48545.objecter _send_command 6 2024-09-10T18:55:17.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.851+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b011cdc0 con 0x7fa9b0122030 2024-09-10T18:55:17.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b011cdc0 con 0x7fa9b0122030 2024-09-10T18:55:17.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 10 client.48545.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 4 2024-09-10T18:55:17.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 2 6 2024-09-10T18:55:17.859 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:55:17.859 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:55:17.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 10 client.48545.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 7 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 7 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 3 7 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 4 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 10 client.48545.objecter _send_command 7 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b005e990 con 0x7fa9b0126e20 2024-09-10T18:55:17.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b0126e20 2024-09-10T18:55:17.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b005e990 con 0x7fa9b0126e20 2024-09-10T18:55:17.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:55:17.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 10 client.48545.objecter _finish_command 7 = system:0 2024-09-10T18:55:17.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 4 2024-09-10T18:55:17.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.855+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 3 7 2024-09-10T18:55:17.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 10 client.48545.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 8 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 8 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005e310 osd=3 3 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 3 8 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 4 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 10 client.48545.objecter _send_command 8 2024-09-10T18:55:17.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.875+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b005e990 con 0x7fa9b0126e20 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b005e990 con 0x7fa9b0126e20 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 10 client.48545.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 4 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 3 8 2024-09-10T18:55:17.883 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 10 client.48545.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 9 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 9 2024-09-10T18:55:17.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 4 9 2024-09-10T18:55:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 4 2024-09-10T18:55:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 10 client.48545.objecter _send_command 9 2024-09-10T18:55:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b0126930 con 0x7fa9b0126460 2024-09-10T18:55:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.879+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b0126460 2024-09-10T18:55:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.883+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b0126930 con 0x7fa9b0126460 2024-09-10T18:55:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.883+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:55:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.883+0000 7fa9acff9640 10 client.48545.objecter _finish_command 9 = system:0 2024-09-10T18:55:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.883+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 4 2024-09-10T18:55:17.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.883+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 4 9 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 10 client.48545.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 10 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 10 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005e990 osd=4 3 2024-09-10T18:55:17.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 4 10 2024-09-10T18:55:17.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 4 2024-09-10T18:55:17.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 10 client.48545.objecter _send_command 10 2024-09-10T18:55:17.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.899+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b0126930 con 0x7fa9b0126460 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b0126930 con 0x7fa9b0126460 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9acff9640 10 client.48545.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 4 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 4 10 2024-09-10T18:55:17.909 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 10 client.48545.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 11 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 11 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 5 11 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 4 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 10 client.48545.objecter _send_command 11 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.903+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b01331e0 con 0x7fa9b012f6e0 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b012f6e0 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b01331e0 con 0x7fa9b012f6e0 2024-09-10T18:55:17.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:55:17.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 10 client.48545.objecter _finish_command 11 = system:0 2024-09-10T18:55:17.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 4 2024-09-10T18:55:17.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.907+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 5 11 2024-09-10T18:55:17.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 10 client.48545.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 12 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 12 2024-09-10T18:55:17.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b0126b70 osd=5 3 2024-09-10T18:55:17.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 5 12 2024-09-10T18:55:17.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 4 2024-09-10T18:55:17.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 10 client.48545.objecter _send_command 12 2024-09-10T18:55:17.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.919+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b005dbc0 con 0x7fa9b012f6e0 2024-09-10T18:55:17.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b005dbc0 con 0x7fa9b012f6e0 2024-09-10T18:55:17.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9acff9640 10 client.48545.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 4 2024-09-10T18:55:17.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 5 12 2024-09-10T18:55:17.929 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 10 client.48545.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 13 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 13 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 6 13 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 4 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 10 client.48545.objecter _send_command 13 2024-09-10T18:55:17.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.923+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b012b5d0 con 0x7fa9b012b0e0 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b012b0e0 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b012b5d0 con 0x7fa9b012b0e0 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 10 client.48545.objecter _finish_command 13 = system:0 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 4 2024-09-10T18:55:17.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.927+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 6 13 2024-09-10T18:55:17.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 10 client.48545.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 14 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 14 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b005dbc0 osd=6 3 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 6 14 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 4 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 10 client.48545.objecter _send_command 14 2024-09-10T18:55:17.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.943+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b012b5d0 con 0x7fa9b012b0e0 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b012b5d0 con 0x7fa9b012b0e0 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9acff9640 10 client.48545.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 4 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 6 14 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:55:17.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 10 client.48545.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 15 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 15 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 7 15 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 4 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 10 client.48545.objecter _send_command 15 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.947+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b012b860 con 0x7fa9b0138170 2024-09-10T18:55:17.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 10 client.48545.objecter ms_handle_connect 0x7fa9b0138170 2024-09-10T18:55:17.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa9b012b860 con 0x7fa9b0138170 2024-09-10T18:55:17.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:55:17.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 10 client.48545.objecter _finish_command 15 = system:0 2024-09-10T18:55:17.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 4 2024-09-10T18:55:17.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.951+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 7 15 2024-09-10T18:55:17.964 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:17.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:17.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 10 client.48545.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:17.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign -1 16 2024-09-10T18:55:17.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 20 client.48545.objecter _get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_remove -1 16 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 20 client.48545.objecter get_session s=0x7fa9b012b5d0 osd=7 3 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 15 client.48545.objecter _session_command_op_assign 7 16 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 4 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 10 client.48545.objecter _send_command 16 2024-09-10T18:55:17.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.983+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa9b013bc70 con 0x7fa9b0138170 2024-09-10T18:55:17.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.987+0000 7fa9acff9640 1 -- 172.21.15.83:0/2065090936 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa9b013bc70 con 0x7fa9b0138170 2024-09-10T18:55:17.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.987+0000 7fa9acff9640 10 client.48545.objecter ms_dispatch 0x7fa9b0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:17.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.987+0000 7fa9acff9640 10 client.48545.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:17.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.987+0000 7fa9acff9640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 4 2024-09-10T18:55:17.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.987+0000 7fa9acff9640 15 client.48545.objecter _session_command_op_remove 7 16 2024-09-10T18:55:17.995 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:17.995 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:55:17.996 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 librados: watch_flush enter 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 librados: watch_flush exit 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.0 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6801/2535527483 conn(0x7fa9b011e7a0 legacy=0x7fa9b011eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b010d9a0 osd=0 1 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.1 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6805/2309624637 conn(0x7fa9b01097a0 legacy=0x7fa9b0109b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b011f630 osd=1 1 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.2 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6809/386094220 conn(0x7fa9b0122030 legacy=0x7fa9b0124410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b4cad640 1 -- 172.21.15.83:0/2065090936 reap_dead start 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b01219b0 osd=2 1 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.3 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6813/1279688021 conn(0x7fa9b0126e20 legacy=0x7fa9b0129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e310 osd=3 1 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.4 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.165:6800/3225515265 conn(0x7fa9b0126460 legacy=0x7fa9b012dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005e990 osd=4 1 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.5 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.165:6804/571173777 conn(0x7fa9b012f6e0 legacy=0x7fa9b0131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b0126b70 osd=5 1 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.6 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.165:6808/1317922936 conn(0x7fa9b012b0e0 legacy=0x7fa9b01331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b005dbc0 osd=6 1 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter close_session for osd.7 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.165:6812/1610151650 conn(0x7fa9b0138170 legacy=0x7fa9b013a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b4cad640 1 -- 172.21.15.83:0/2065090936 reap_dead start 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter put_session s=0x7fa9b012b5d0 osd=7 1 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 20 client.48545.objecter shutdown clearing up homeless session... 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 10 client.48545.objecter successfully canceled tick 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa99c075500 legacy=0x7fa99c0779b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> v1:172.21.15.83:6790/0 conn(0x7fa9b01042a0 legacy=0x7fa9b0117440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:17.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 shutdown_connections 2024-09-10T18:55:18.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.527037+0000 osd.6 (osd.6) 109 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:55:18.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.532305+0000 osd.6 (osd.6) 110 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.649659+0000 osd.7 (osd.7) 139 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.656755+0000 osd.7 (osd.7) 140 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.961078+0000 mon.a (mon.0) 1230 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]': finished 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.962388+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.962641+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.963597+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.963845+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.83:0/2999817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.974974+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48533 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.981938+0000 mon.a (mon.0) 1233 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:16.982992+0000 mon.a (mon.0) 1234 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:16.991883+0000 osd.0 (osd.0) 141 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:55:18.001 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.607415+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.788886+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.83:0/2065090936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.956176+0000 mon.a (mon.0) 1235 : audit [INF] from='client.48533 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.961868+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.962074+0000 mon.a (mon.0) 1236 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.962663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.963438+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.966740+0000 mon.a (mon.0) 1239 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:17 smithi165 bash[22591]: audit 2024-09-10T18:55:17.978578+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.991+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 >> 172.21.15.83:0/2065090936 conn(0x7fa9b00f70b0 msgr2=0x7fa9b00fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:18.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.995+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 shutdown_connections 2024-09-10T18:55:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.995+0000 7fa9b6710640 1 -- 172.21.15.83:0/2065090936 wait complete. 2024-09-10T18:55:18.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:17.995+0000 7fa9b6710640 1 librados: shutdown 2024-09-10T18:55:18.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.035+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f5b140052c0 con 0x7f5b301025b0 2024-09-10T18:55:18.074 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:18.106 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.527037+0000 osd.6 (osd.6) 109 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.532305+0000 osd.6 (osd.6) 110 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.649659+0000 osd.7 (osd.7) 139 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.656755+0000 osd.7 (osd.7) 140 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.961078+0000 mon.a (mon.0) 1230 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]': finished 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.962388+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.962641+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.963597+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.963845+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.83:0/2999817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.974974+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48533 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.981938+0000 mon.a (mon.0) 1233 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:16.982992+0000 mon.a (mon.0) 1234 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:16.991883+0000 osd.0 (osd.0) 141 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.607415+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.788886+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.83:0/2065090936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.956176+0000 mon.a (mon.0) 1235 : audit [INF] from='client.48533 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.961868+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.962074+0000 mon.a (mon.0) 1236 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.962663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.963438+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:55:18.207 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.966740+0000 mon.a (mon.0) 1239 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[18629]: audit 2024-09-10T18:55:17.978578+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.527037+0000 osd.6 (osd.6) 109 : cluster [DBG] 11.1 deep-scrub starts 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.532305+0000 osd.6 (osd.6) 110 : cluster [DBG] 11.1 deep-scrub ok 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.649659+0000 osd.7 (osd.7) 139 : cluster [DBG] 11.4 deep-scrub starts 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.656755+0000 osd.7 (osd.7) 140 : cluster [DBG] 11.4 deep-scrub ok 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.961078+0000 mon.a (mon.0) 1230 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache", "force_nonempty":""}]': finished 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.962388+0000 mon.a (mon.0) 1231 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.962641+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.208 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.963597+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.963845+0000 mon.c (mon.2) 204 : audit [INF] from='client.? v1:172.21.15.83:0/2999817424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.974974+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48533 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.981938+0000 mon.a (mon.0) 1233 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:16.982992+0000 mon.a (mon.0) 1234 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:16.991883+0000 osd.0 (osd.0) 141 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.607415+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.788886+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.83:0/2065090936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.956176+0000 mon.a (mon.0) 1235 : audit [INF] from='client.48533 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.961868+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.83:0/4154205751' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.962074+0000 mon.a (mon.0) 1236 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77877-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.962663+0000 mon.a (mon.0) 1237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.963438+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-09-10T18:55:18.209 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.966740+0000 mon.a (mon.0) 1239 : audit [INF] from='client.46213 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]: dispatch 2024-09-10T18:55:18.210 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:17 smithi083 bash[32381]: audit 2024-09-10T18:55:17.978578+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:18.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:18.242 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:18.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:18.379 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:18.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:18.517 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:18.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:18.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:18.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:18.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:18.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:18.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.931+0000 7f5b25ffb640 1 -- 172.21.15.83:0/362110219 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v163) v1 ==== 113+0+0 (unknown 1241385513 0 0) 0x7f5b2c056950 con 0x7f5b301025b0 2024-09-10T18:55:18.938 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T18:55:18.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 10 librados: watch_flush enter 2024-09-10T18:55:18.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 10 librados: watch_flush exit 2024-09-10T18:55:18.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 20 client.48551.objecter shutdown clearing up homeless session... 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 10 client.48551.objecter successfully canceled tick 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5b1c071670 legacy=0x7f5b1c073b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 >> v1:172.21.15.83:6790/0 conn(0x7f5b301025b0 legacy=0x7f5b30101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 shutdown_connections 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 >> 172.21.15.83:0/362110219 conn(0x7f5b300f81d0 msgr2=0x7f5b30109380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 shutdown_connections 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 -- 172.21.15.83:0/362110219 wait complete. 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:18.939+0000 7f5b373c2640 1 librados: shutdown 2024-09-10T18:55:18.944 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:18.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:19.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:19.083 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:19.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:19.218 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:19.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:19.355 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:19.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:19.490 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:19.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.145179+0000 osd.0 (osd.0) 142 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.406438+0000 mgr.y (mgr.24526) 242 : cluster [DBG] pgmap v224: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.559201+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.563524+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:17.674458+0000 mgr.y (mgr.24526) 243 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:17.983771+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.040592+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.83:0/362110219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.040925+0000 mon.a (mon.0) 1242 : audit [INF] from='client.48551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.609254+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.936494+0000 mon.a (mon.0) 1243 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]': finished 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.936714+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:19.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: audit 2024-09-10T18:55:18.937140+0000 mon.a (mon.0) 1245 : audit [INF] from='client.48551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:55:19.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:19 smithi165 bash[22591]: cluster 2024-09-10T18:55:18.937215+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:55:19.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:19.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:19.766 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.145179+0000 osd.0 (osd.0) 142 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.406438+0000 mgr.y (mgr.24526) 242 : cluster [DBG] pgmap v224: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.559201+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.563524+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:17.674458+0000 mgr.y (mgr.24526) 243 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:17.983771+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.040592+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.83:0/362110219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.040925+0000 mon.a (mon.0) 1242 : audit [INF] from='client.48551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.609254+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.936494+0000 mon.a (mon.0) 1243 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]': finished 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.936714+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: audit 2024-09-10T18:55:18.937140+0000 mon.a (mon.0) 1245 : audit [INF] from='client.48551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[18629]: cluster 2024-09-10T18:55:18.937215+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.145179+0000 osd.0 (osd.0) 142 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.406438+0000 mgr.y (mgr.24526) 242 : cluster [DBG] pgmap v224: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.559201+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:55:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.563524+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:17.674458+0000 mgr.y (mgr.24526) 243 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:17.983771+0000 mon.a (mon.0) 1241 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.040592+0000 mon.c (mon.2) 207 : audit [INF] from='client.? v1:172.21.15.83:0/362110219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.040925+0000 mon.a (mon.0) 1242 : audit [INF] from='client.48551 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.609254+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.936494+0000 mon.a (mon.0) 1243 : audit [INF] from='client.46213 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77877-10", "tierpool":"test-rados-api-smithi083-77877-10-cache"}]': finished 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.936714+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? v1:172.21.15.83:0/2113007527' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi083-77640-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: audit 2024-09-10T18:55:18.937140+0000 mon.a (mon.0) 1245 : audit [INF] from='client.48551 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T18:55:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:19 smithi083 bash[32381]: cluster 2024-09-10T18:55:18.937215+0000 mon.a (mon.0) 1246 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-09-10T18:55:19.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:19.905 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:19.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:20.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:20.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:20.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.310 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:20.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:20.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:20.578 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:20.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:20.715 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:18.993142+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: cluster 2024-09-10T18:55:19.293392+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: cluster 2024-09-10T18:55:19.296665+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: cluster 2024-09-10T18:55:19.407429+0000 mgr.y (mgr.24526) 244 : cluster [DBG] pgmap v227: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.425224+0000 mon.a (mon.0) 1247 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.611039+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.942595+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.944210+0000 mon.a (mon.0) 1248 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: cluster 2024-09-10T18:55:19.944301+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.948840+0000 mon.a (mon.0) 1250 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.948953+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3840819660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.949190+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[18629]: audit 2024-09-10T18:55:19.949814+0000 mon.a (mon.0) 1252 : audit [INF] from='client.48557 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:18.993142+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: cluster 2024-09-10T18:55:19.293392+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: cluster 2024-09-10T18:55:19.296665+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: cluster 2024-09-10T18:55:19.407429+0000 mgr.y (mgr.24526) 244 : cluster [DBG] pgmap v227: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.425224+0000 mon.a (mon.0) 1247 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.611039+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.942595+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.944210+0000 mon.a (mon.0) 1248 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: cluster 2024-09-10T18:55:19.944301+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.948840+0000 mon.a (mon.0) 1250 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.948953+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3840819660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.949190+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:20.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:20 smithi083 bash[32381]: audit 2024-09-10T18:55:19.949814+0000 mon.a (mon.0) 1252 : audit [INF] from='client.48557 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:18.993142+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: cluster 2024-09-10T18:55:19.293392+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: cluster 2024-09-10T18:55:19.296665+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: cluster 2024-09-10T18:55:19.407429+0000 mgr.y (mgr.24526) 244 : cluster [DBG] pgmap v227: 428 pgs: 1 active+clean+snaptrim, 96 unknown, 331 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.425224+0000 mon.a (mon.0) 1247 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.611039+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:20.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.942595+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.944210+0000 mon.a (mon.0) 1248 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: cluster 2024-09-10T18:55:19.944301+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.948840+0000 mon.a (mon.0) 1250 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.948953+0000 mon.c (mon.2) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3840819660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.949190+0000 mon.a (mon.0) 1251 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:20.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:20 smithi165 bash[22591]: audit 2024-09-10T18:55:19.949814+0000 mon.a (mon.0) 1252 : audit [INF] from='client.48557 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:20.851 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:20.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: Running main() from gmock_main.cc 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] Global test environment set-up. 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolList (13143 ms) 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3901 ms) 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2961 ms) 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4340 ms) 2024-09-10T18:55:20.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2994 ms) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (7193 ms) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6164 ms) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5635 ms) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8816 ms) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] 9 tests from LibRadosPools (55147 ms total) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] Global test environment tear-down 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (55147 ms total) 2024-09-10T18:55:20.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-10T18:55:20.991 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:20.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:21.094 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:21.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:21.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:21.267 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:21.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:21.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:21.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:21.543 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:21.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.612777+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.951486+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.952387+0000 mon.a (mon.0) 1253 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.952681+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.953004+0000 mon.a (mon.0) 1255 : audit [INF] from='client.48557 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: cluster 2024-09-10T18:55:20.953500+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.958402+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3228302036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.961937+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48563 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.962303+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:21.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: audit 2024-09-10T18:55:20.962656+0000 mon.a (mon.0) 1259 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.009975+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:55:21.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:21 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.011717+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:55:21.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:21.677 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:21.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.612777+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.951486+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.952387+0000 mon.a (mon.0) 1253 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.952681+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.953004+0000 mon.a (mon.0) 1255 : audit [INF] from='client.48557 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: cluster 2024-09-10T18:55:20.953500+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.958402+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3228302036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.961937+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48563 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.962303+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: audit 2024-09-10T18:55:20.962656+0000 mon.a (mon.0) 1259 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.009975+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:55:21.699 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.011717+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.612777+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.951486+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.952387+0000 mon.a (mon.0) 1253 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.952681+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.953004+0000 mon.a (mon.0) 1255 : audit [INF] from='client.48557 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi083-77648-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: cluster 2024-09-10T18:55:20.953500+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.958402+0000 mon.c (mon.2) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3228302036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.961937+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48563 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.962303+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: audit 2024-09-10T18:55:20.962656+0000 mon.a (mon.0) 1259 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]: dispatch 2024-09-10T18:55:21.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.009975+0000 osd.0 (osd.0) 143 : cluster [DBG] 2.13 scrub starts 2024-09-10T18:55:21.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:21 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.011717+0000 osd.0 (osd.0) 144 : cluster [DBG] 2.13 scrub ok 2024-09-10T18:55:21.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:21.811 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:21.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T18:55:21.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:21.950 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:21.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:22.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:22.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:22.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:22.230 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:22.336 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:22.369 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:22.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:22.509 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.065278+0000 osd.5 (osd.5) 137 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.118926+0000 osd.5 (osd.5) 138 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: audit 2024-09-10T18:55:21.614523+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.945026+0000 mon.a (mon.0) 1260 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: audit 2024-09-10T18:55:21.956871+0000 mon.a (mon.0) 1261 : audit [INF] from='client.48563 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:22.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: audit 2024-09-10T18:55:21.957032+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:22.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: audit 2024-09-10T18:55:21.957132+0000 mon.a (mon.0) 1263 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]': finished 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.957548+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.065278+0000 osd.5 (osd.5) 137 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.118926+0000 osd.5 (osd.5) 138 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: audit 2024-09-10T18:55:21.614523+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:22.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.945026+0000 mon.a (mon.0) 1260 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:22.626 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: audit 2024-09-10T18:55:21.956871+0000 mon.a (mon.0) 1261 : audit [INF] from='client.48563 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:22.626 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: audit 2024-09-10T18:55:21.957032+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:22.626 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: audit 2024-09-10T18:55:21.957132+0000 mon.a (mon.0) 1263 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]': finished 2024-09-10T18:55:22.626 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.957548+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:55:22.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:22.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:22.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.065278+0000 osd.5 (osd.5) 137 : cluster [DBG] 11.7 deep-scrub starts 2024-09-10T18:55:22.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.118926+0000 osd.5 (osd.5) 138 : cluster [DBG] 11.7 deep-scrub ok 2024-09-10T18:55:22.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: audit 2024-09-10T18:55:21.614523+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:22.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.945026+0000 mon.a (mon.0) 1260 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: audit 2024-09-10T18:55:21.956871+0000 mon.a (mon.0) 1261 : audit [INF] from='client.48563 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi083-77640-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: audit 2024-09-10T18:55:21.957032+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? v1:172.21.15.83:0/251513516' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi083-77907-9"}]': finished 2024-09-10T18:55:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: audit 2024-09-10T18:55:21.957132+0000 mon.a (mon.0) 1263 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-11", "mode": "writeback"}]': finished 2024-09-10T18:55:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:22 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.957548+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-09-10T18:55:22.783 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:22.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:22.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:22.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:22.986 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:22] "GET /metrics HTTP/1.1" 200 245051 "" "Prometheus/2.43.0" 2024-09-10T18:55:23.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:23.051 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:23.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:23.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/155459883 >> v1:172.21.15.83:6790/0 conn(0x7fc980108f40 legacy=0x7fc98010b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:23.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/155459883 shutdown_connections 2024-09-10T18:55:23.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/155459883 >> 172.21.15.83:0/155459883 conn(0x7fc980075290 msgr2=0x7fc9800f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:23.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/155459883 shutdown_connections 2024-09-10T18:55:23.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/155459883 wait complete. 2024-09-10T18:55:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 librados: starting msgr at 2024-09-10T18:55:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 librados: starting objecter 2024-09-10T18:55:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 Processor -- start 2024-09-10T18:55:23.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- start start 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 librados: setting wanted keys 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 librados: calling monclient init 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc98011e890 con 0x7fc980108f40 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc98011fa50 con 0x7fc9800fd610 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc980120c50 con 0x7fc980107880 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc9859df640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc980107880 0x7fc98011d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37640/0 (socket says 172.21.15.83:37640) 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc9859df640 1 -- 172.21.15.83:0/487504229 learned_addr learned my addr 172.21.15.83:0/487504229 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter ms_handle_connect 0x7fc980108f40 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter ms_handle_connect 0x7fc980107880 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178214928 0 0) 0x7fc980120c50 con 0x7fc980107880 2024-09-10T18:55:23.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc96c0054b0 con 0x7fc980107880 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter ms_handle_connect 0x7fc9800fd610 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103897955 0 0) 0x7fc98011fa50 con 0x7fc9800fd610 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc980120c50 con 0x7fc9800fd610 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148473839 0 0) 0x7fc96c0054b0 con 0x7fc980107880 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc98011fa50 con 0x7fc980107880 2024-09-10T18:55:23.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc978001de0 con 0x7fc980107880 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3997117041 0 0) 0x7fc98011fa50 con 0x7fc980107880 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 >> v1:172.21.15.165:6789/0 conn(0x7fc9800fd610 legacy=0x7fc980107170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 >> v1:172.21.15.83:6789/0 conn(0x7fc980108f40 legacy=0x7fc980119980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980121e50 con 0x7fc980107880 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc980120e40 con 0x7fc980107880 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 10 client.48584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:23.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9801212d0 con 0x7fc980107880 2024-09-10T18:55:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc987442640 1 librados: init done 2024-09-10T18:55:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.123+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc978003360 con 0x7fc980107880 2024-09-10T18:55:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc978003ba0 con 0x7fc980107880 2024-09-10T18:55:23.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc964005010 con 0x7fc980107880 2024-09-10T18:55:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc978003d00 con 0x7fc980107880 2024-09-10T18:55:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(167..167 src has 69..167) v4 ==== 9573+0+0 (unknown 2615540010 0 0) 0x7fc97808b1b0 con 0x7fc980107880 2024-09-10T18:55:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 10 client.48584.objecter ms_dispatch 0x7fc980000b90 osd_map(167..167 src has 69..167) v4 2024-09-10T18:55:23.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 3 client.48584.objecter handle_osd_map got epochs [167,167] > 0 2024-09-10T18:55:23.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 3 client.48584.objecter handle_osd_map decoding full epoch 167 2024-09-10T18:55:23.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.127+0000 7fc975ffb640 20 client.48584.objecter dump_active .. 0 homeless 2024-09-10T18:55:23.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.131+0000 7fc975ffb640 10 client.48584.objecter ms_handle_connect 0x7fc96c0714a0 2024-09-10T18:55:23.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.135+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc978057170 con 0x7fc980107880 2024-09-10T18:55:23.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:55:23.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:23.191 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:23.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:23.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:23.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:23.323+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fc964005840 con 0x7fc980107880 2024-09-10T18:55:23.682 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:23.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:23.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: cluster 2024-09-10T18:55:21.408475+0000 mgr.y (mgr.24526) 245 : cluster [DBG] pgmap v230: 388 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.011744+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: cluster 2024-09-10T18:55:22.058995+0000 osd.5 (osd.5) 139 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: cluster 2024-09-10T18:55:22.061909+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.445003+0000 mon.a (mon.0) 1265 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.445922+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.455548+0000 mon.a (mon.0) 1266 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.456154+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.456496+0000 mon.a (mon.0) 1267 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.615846+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.963525+0000 mon.a (mon.0) 1268 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: cluster 2024-09-10T18:55:22.964658+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.966646+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.968846+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[18629]: audit 2024-09-10T18:55:22.969258+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: cluster 2024-09-10T18:55:21.408475+0000 mgr.y (mgr.24526) 245 : cluster [DBG] pgmap v230: 388 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.011744+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: cluster 2024-09-10T18:55:22.058995+0000 osd.5 (osd.5) 139 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: cluster 2024-09-10T18:55:22.061909+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.445003+0000 mon.a (mon.0) 1265 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.445922+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.455548+0000 mon.a (mon.0) 1266 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.456154+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.456496+0000 mon.a (mon.0) 1267 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.615846+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.963525+0000 mon.a (mon.0) 1268 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: cluster 2024-09-10T18:55:22.964658+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.966646+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.968846+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:23 smithi083 bash[32381]: audit 2024-09-10T18:55:22.969258+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:23.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: cluster 2024-09-10T18:55:21.408475+0000 mgr.y (mgr.24526) 245 : cluster [DBG] pgmap v230: 388 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.011744+0000 mon.c (mon.2) 214 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: cluster 2024-09-10T18:55:22.058995+0000 osd.5 (osd.5) 139 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: cluster 2024-09-10T18:55:22.061909+0000 osd.5 (osd.5) 140 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.445003+0000 mon.a (mon.0) 1265 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.445922+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.455548+0000 mon.a (mon.0) 1266 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.456154+0000 mon.c (mon.2) 216 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.456496+0000 mon.a (mon.0) 1267 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.615846+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.963525+0000 mon.a (mon.0) 1268 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: cluster 2024-09-10T18:55:22.964658+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.966646+0000 mon.c (mon.2) 218 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.968846+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:23.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:23 smithi165 bash[22591]: audit 2024-09-10T18:55:22.969258+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:23.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:23.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:23.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:24.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:24.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:24.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.254 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:24.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.379+0000 7fc975ffb640 1 -- 172.21.15.83:0/487504229 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v168) v1 ==== 87+0+0 (unknown 1331338901 0 0) 0x7fc978056b10 con 0x7fc980107880 2024-09-10T18:55:24.384 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.383+0000 7fc987442640 10 librados: watch_flush enter 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.383+0000 7fc987442640 10 librados: watch_flush exit 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.383+0000 7fc987442640 20 client.48584.objecter shutdown clearing up homeless session... 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.383+0000 7fc987442640 10 client.48584.objecter successfully canceled tick 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.387+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc96c0714a0 legacy=0x7fc96c073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.387+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 >> v1:172.21.15.83:6790/0 conn(0x7fc980107880 legacy=0x7fc98011d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.391+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 shutdown_connections 2024-09-10T18:55:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.391+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 >> 172.21.15.83:0/487504229 conn(0x7fc980075290 msgr2=0x7fc9800f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.391+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 shutdown_connections 2024-09-10T18:55:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.391+0000 7fc987442640 1 -- 172.21.15.83:0/487504229 wait complete. 2024-09-10T18:55:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:24.391+0000 7fc987442640 1 librados: shutdown 2024-09-10T18:55:24.396 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:24.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:24.532 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: cluster 2024-09-10T18:55:22.966187+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:55:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.018870+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.328419+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.83:0/487504229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: cluster 2024-09-10T18:55:23.409241+0000 mgr.y (mgr.24526) 246 : cluster [DBG] pgmap v233: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.457751+0000 mon.a (mon.0) 1272 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.458583+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.617631+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: cluster 2024-09-10T18:55:23.928526+0000 mon.a (mon.0) 1274 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: cluster 2024-09-10T18:55:23.931381+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:23.943018+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.381246+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.382191+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.83:0/3672459727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.383204+0000 mon.a (mon.0) 1277 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.383281+0000 mon.a (mon.0) 1278 : audit [INF] from='client.48584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: cluster 2024-09-10T18:55:24.383361+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.393094+0000 mon.a (mon.0) 1280 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:24 smithi165 bash[22591]: audit 2024-09-10T18:55:24.393661+0000 mon.a (mon.0) 1281 : audit [INF] from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:24.668 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: cluster 2024-09-10T18:55:22.966187+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.018870+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.328419+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.83:0/487504229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: cluster 2024-09-10T18:55:23.409241+0000 mgr.y (mgr.24526) 246 : cluster [DBG] pgmap v233: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.457751+0000 mon.a (mon.0) 1272 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.458583+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.617631+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: cluster 2024-09-10T18:55:23.928526+0000 mon.a (mon.0) 1274 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: cluster 2024-09-10T18:55:23.931381+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:23.943018+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:24.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.381246+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.382191+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.83:0/3672459727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.383204+0000 mon.a (mon.0) 1277 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.383281+0000 mon.a (mon.0) 1278 : audit [INF] from='client.48584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: cluster 2024-09-10T18:55:24.383361+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.393094+0000 mon.a (mon.0) 1280 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[32381]: audit 2024-09-10T18:55:24.393661+0000 mon.a (mon.0) 1281 : audit [INF] from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: cluster 2024-09-10T18:55:22.966187+0000 osd.0 (osd.0) 145 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.018870+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.328419+0000 mon.c (mon.2) 219 : audit [INF] from='client.? v1:172.21.15.83:0/487504229' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: cluster 2024-09-10T18:55:23.409241+0000 mgr.y (mgr.24526) 246 : cluster [DBG] pgmap v233: 356 pgs: 32 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 257 KiB/s wr, 3 op/s 2024-09-10T18:55:24.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.457751+0000 mon.a (mon.0) 1272 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.458583+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48584 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.617631+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: cluster 2024-09-10T18:55:23.928526+0000 mon.a (mon.0) 1274 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: cluster 2024-09-10T18:55:23.931381+0000 mon.a (mon.0) 1275 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:23.943018+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? v1:172.21.15.83:0/4143902700' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi083-77648-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.381246+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.382191+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.83:0/3672459727' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.383204+0000 mon.a (mon.0) 1277 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.383281+0000 mon.a (mon.0) 1278 : audit [INF] from='client.48584 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: cluster 2024-09-10T18:55:24.383361+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.393094+0000 mon.a (mon.0) 1280 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]: dispatch 2024-09-10T18:55:24.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:24 smithi083 bash[18629]: audit 2024-09-10T18:55:24.393661+0000 mon.a (mon.0) 1281 : audit [INF] from='client.49108 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:24.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.808 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:24.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:24.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:24.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:25.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:25.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:25.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:25.231 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:25.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:25.369 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:25.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:25.500 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:25.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:25.637 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:25.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: audit 2024-09-10T18:55:24.619633+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: cluster 2024-09-10T18:55:24.936047+0000 mon.a (mon.0) 1282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: audit 2024-09-10T18:55:24.951292+0000 mon.a (mon.0) 1283 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: audit 2024-09-10T18:55:24.951762+0000 mon.a (mon.0) 1284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: audit 2024-09-10T18:55:24.951888+0000 mon.a (mon.0) 1285 : audit [INF] from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[18629]: cluster 2024-09-10T18:55:24.952414+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: audit 2024-09-10T18:55:24.619633+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: cluster 2024-09-10T18:55:24.936047+0000 mon.a (mon.0) 1282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: audit 2024-09-10T18:55:24.951292+0000 mon.a (mon.0) 1283 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: audit 2024-09-10T18:55:24.951762+0000 mon.a (mon.0) 1284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: audit 2024-09-10T18:55:24.951888+0000 mon.a (mon.0) 1285 : audit [INF] from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:25.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:25 smithi083 bash[32381]: cluster 2024-09-10T18:55:24.952414+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:55:25.765 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:25.811 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:55:25.811 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: audit 2024-09-10T18:55:24.619633+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: cluster 2024-09-10T18:55:24.936047+0000 mon.a (mon.0) 1282 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: audit 2024-09-10T18:55:24.951292+0000 mon.a (mon.0) 1283 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi083-77907-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: audit 2024-09-10T18:55:24.951762+0000 mon.a (mon.0) 1284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-11"}]': finished 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: audit 2024-09-10T18:55:24.951888+0000 mon.a (mon.0) 1285 : audit [INF] from='client.49108 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi083-77640-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:25.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:25 smithi165 bash[22591]: cluster 2024-09-10T18:55:24.952414+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-09-10T18:55:25.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:25.900 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:25.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:26.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.035 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:26.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:26.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.298 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:26.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:26.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:26.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:26.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:26.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:26.704 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:26.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:26.837 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:26.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T18:55:26.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.981 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:26.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:27.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:27.098 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: cluster 2024-09-10T18:55:24.898565+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:27.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: cluster 2024-09-10T18:55:24.898704+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:27.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: audit 2024-09-10T18:55:25.621976+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:27.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: cluster 2024-09-10T18:55:25.954160+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:55:27.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: audit 2024-09-10T18:55:25.961567+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.83:0/1498622670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.099 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:26 smithi165 bash[22591]: audit 2024-09-10T18:55:25.963094+0000 mon.a (mon.0) 1288 : audit [INF] from='client.48590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.120 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:27.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:27.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: cluster 2024-09-10T18:55:24.898565+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: cluster 2024-09-10T18:55:24.898704+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: audit 2024-09-10T18:55:25.621976+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: cluster 2024-09-10T18:55:25.954160+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: audit 2024-09-10T18:55:25.961567+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.83:0/1498622670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[18629]: audit 2024-09-10T18:55:25.963094+0000 mon.a (mon.0) 1288 : audit [INF] from='client.48590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: cluster 2024-09-10T18:55:24.898565+0000 osd.1 (osd.1) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: cluster 2024-09-10T18:55:24.898704+0000 osd.1 (osd.1) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:27.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: audit 2024-09-10T18:55:25.621976+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: cluster 2024-09-10T18:55:25.954160+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-09-10T18:55:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: audit 2024-09-10T18:55:25.961567+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.83:0/1498622670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:26 smithi083 bash[32381]: audit 2024-09-10T18:55:25.963094+0000 mon.a (mon.0) 1288 : audit [INF] from='client.48590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:27.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:27.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:27.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:27.538 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:27.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:27.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:27.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:55:27.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:27.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:27.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:27.939 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:27.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:28.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: cluster 2024-09-10T18:55:25.410097+0000 mgr.y (mgr.24526) 247 : cluster [DBG] pgmap v236: 364 pgs: 40 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.623611+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: cluster 2024-09-10T18:55:26.688327+0000 osd.7 (osd.7) 141 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: cluster 2024-09-10T18:55:26.691566+0000 osd.7 (osd.7) 142 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.959674+0000 mon.a (mon.0) 1289 : audit [INF] from='client.48590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: cluster 2024-09-10T18:55:26.959892+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.969938+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.83:0/2925330191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.970192+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.970336+0000 mon.a (mon.0) 1291 : audit [INF] from='client.48593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:26.977048+0000 mon.a (mon.0) 1292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[18629]: audit 2024-09-10T18:55:27.625208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: cluster 2024-09-10T18:55:25.410097+0000 mgr.y (mgr.24526) 247 : cluster [DBG] pgmap v236: 364 pgs: 40 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.623611+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: cluster 2024-09-10T18:55:26.688327+0000 osd.7 (osd.7) 141 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: cluster 2024-09-10T18:55:26.691566+0000 osd.7 (osd.7) 142 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.959674+0000 mon.a (mon.0) 1289 : audit [INF] from='client.48590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: cluster 2024-09-10T18:55:26.959892+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.969938+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.83:0/2925330191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.970192+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.970336+0000 mon.a (mon.0) 1291 : audit [INF] from='client.48593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:26.977048+0000 mon.a (mon.0) 1292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:27 smithi083 bash[32381]: audit 2024-09-10T18:55:27.625208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.043 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:28.074 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:28.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: cluster 2024-09-10T18:55:25.410097+0000 mgr.y (mgr.24526) 247 : cluster [DBG] pgmap v236: 364 pgs: 40 unknown, 1 active+clean+snaptrim, 323 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.623611+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: cluster 2024-09-10T18:55:26.688327+0000 osd.7 (osd.7) 141 : cluster [DBG] 11.5 deep-scrub starts 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: cluster 2024-09-10T18:55:26.691566+0000 osd.7 (osd.7) 142 : cluster [DBG] 11.5 deep-scrub ok 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.959674+0000 mon.a (mon.0) 1289 : audit [INF] from='client.48590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi083-77648-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: cluster 2024-09-10T18:55:26.959892+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.969938+0000 mon.c (mon.2) 225 : audit [INF] from='client.? v1:172.21.15.83:0/2925330191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.970192+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.970336+0000 mon.a (mon.0) 1291 : audit [INF] from='client.48593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:26.977048+0000 mon.a (mon.0) 1292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:28.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:27 smithi165 bash[22591]: audit 2024-09-10T18:55:27.625208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:28.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:28.203 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:28.203 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:28.209 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:28.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:28.346 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:28.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:28.485 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:28.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.624 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:28.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.760 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:28.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.897 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:28.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:29.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:29.009 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: cluster 2024-09-10T18:55:27.411092+0000 mgr.y (mgr.24526) 248 : cluster [DBG] pgmap v239: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:29.009 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:27.682514+0000 mgr.y (mgr.24526) 249 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:27.965521+0000 mon.a (mon.0) 1293 : audit [INF] from='client.48593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:27.965845+0000 mon.a (mon.0) 1294 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: cluster 2024-09-10T18:55:27.966472+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:28.626762+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: cluster 2024-09-10T18:55:28.934003+0000 mon.a (mon.0) 1296 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: cluster 2024-09-10T18:55:28.950501+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:28.953321+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3820701333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.010 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:28 smithi165 bash[22591]: audit 2024-09-10T18:55:28.954157+0000 mon.a (mon.0) 1298 : audit [INF] from='client.48599 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.028 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:29.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:29.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:29.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: cluster 2024-09-10T18:55:27.411092+0000 mgr.y (mgr.24526) 248 : cluster [DBG] pgmap v239: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:27.682514+0000 mgr.y (mgr.24526) 249 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:27.965521+0000 mon.a (mon.0) 1293 : audit [INF] from='client.48593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:27.965845+0000 mon.a (mon.0) 1294 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: cluster 2024-09-10T18:55:27.966472+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:55:29.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:28.626762+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: cluster 2024-09-10T18:55:28.934003+0000 mon.a (mon.0) 1296 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: cluster 2024-09-10T18:55:28.950501+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:28.953321+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3820701333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[18629]: audit 2024-09-10T18:55:28.954157+0000 mon.a (mon.0) 1298 : audit [INF] from='client.48599 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: cluster 2024-09-10T18:55:27.411092+0000 mgr.y (mgr.24526) 248 : cluster [DBG] pgmap v239: 396 pgs: 8 creating+peering, 96 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:27.682514+0000 mgr.y (mgr.24526) 249 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:27.965521+0000 mon.a (mon.0) 1293 : audit [INF] from='client.48593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi083-77640-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:27.965845+0000 mon.a (mon.0) 1294 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: cluster 2024-09-10T18:55:27.966472+0000 mon.a (mon.0) 1295 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-09-10T18:55:29.276 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:28.626762+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:29.277 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: cluster 2024-09-10T18:55:28.934003+0000 mon.a (mon.0) 1296 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:29.277 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: cluster 2024-09-10T18:55:28.950501+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-09-10T18:55:29.277 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:28.953321+0000 mon.c (mon.2) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3820701333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.277 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:28 smithi083 bash[32381]: audit 2024-09-10T18:55:28.954157+0000 mon.a (mon.0) 1298 : audit [INF] from='client.48599 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:29.297 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:29.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:29.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:29.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:29.568 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:29.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:29.706 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:29.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.842 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:29.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.983 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:29.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:30.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:30.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:30.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:30.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:30.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:30.387 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:30.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:30.524 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:30.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: cluster 2024-09-10T18:55:29.412001+0000 mgr.y (mgr.24526) 250 : cluster [DBG] pgmap v242: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: audit 2024-09-10T18:55:29.562814+0000 mon.a (mon.0) 1299 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: audit 2024-09-10T18:55:29.565083+0000 mon.c (mon.2) 229 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: audit 2024-09-10T18:55:29.628443+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: audit 2024-09-10T18:55:29.956661+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48599 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: cluster 2024-09-10T18:55:29.957703+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:55:30.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:30 smithi165 bash[22591]: audit 2024-09-10T18:55:29.962956+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:30.661 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:30.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: cluster 2024-09-10T18:55:29.412001+0000 mgr.y (mgr.24526) 250 : cluster [DBG] pgmap v242: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:30.749 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: audit 2024-09-10T18:55:29.562814+0000 mon.a (mon.0) 1299 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: audit 2024-09-10T18:55:29.565083+0000 mon.c (mon.2) 229 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: audit 2024-09-10T18:55:29.628443+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: audit 2024-09-10T18:55:29.956661+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48599 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: cluster 2024-09-10T18:55:29.957703+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[18629]: audit 2024-09-10T18:55:29.962956+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: cluster 2024-09-10T18:55:29.412001+0000 mgr.y (mgr.24526) 250 : cluster [DBG] pgmap v242: 364 pgs: 8 creating+peering, 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: audit 2024-09-10T18:55:29.562814+0000 mon.a (mon.0) 1299 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: audit 2024-09-10T18:55:29.565083+0000 mon.c (mon.2) 229 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: audit 2024-09-10T18:55:29.628443+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: audit 2024-09-10T18:55:29.956661+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48599 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi083-77648-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:30.750 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: cluster 2024-09-10T18:55:29.957703+0000 mon.a (mon.0) 1301 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-09-10T18:55:30.751 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:30 smithi083 bash[32381]: audit 2024-09-10T18:55:29.962956+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:30.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:30.795 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:30.899 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:30.937 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:30.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:31.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:31.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:31.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:31.206 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:31.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:31.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:31.471 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:31.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:31.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:31.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:31.748 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:31.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:31 smithi165 bash[22591]: audit 2024-09-10T18:55:30.629931+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:31.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:31 smithi165 bash[22591]: audit 2024-09-10T18:55:30.959021+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:31.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:31 smithi165 bash[22591]: cluster 2024-09-10T18:55:30.959267+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:55:31.864 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:31.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:31.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:31.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:32.009 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:32.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[32381]: audit 2024-09-10T18:55:30.629931+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:32.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[32381]: audit 2024-09-10T18:55:30.959021+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:32.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[32381]: cluster 2024-09-10T18:55:30.959267+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:55:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[18629]: audit 2024-09-10T18:55:30.629931+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[18629]: audit 2024-09-10T18:55:30.959021+0000 mon.a (mon.0) 1303 : audit [INF] from='client.? v1:172.21.15.83:0/371010594' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi083-77640-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:31 smithi083 bash[18629]: cluster 2024-09-10T18:55:30.959267+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-09-10T18:55:32.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:32.146 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:32.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:32.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:32.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:32.418 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:32.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:32.550 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:32.645 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:32.645 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:32.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:32.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:32.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:32.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:32] "GET /metrics HTTP/1.1" 200 240378 "" "Prometheus/2.43.0" 2024-09-10T18:55:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: cluster 2024-09-10T18:55:31.412943+0000 mgr.y (mgr.24526) 251 : cluster [DBG] pgmap v245: 364 pgs: 32 creating+peering, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: audit 2024-09-10T18:55:31.631774+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: cluster 2024-09-10T18:55:31.967422+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:55:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: audit 2024-09-10T18:55:31.971968+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: audit 2024-09-10T18:55:32.231847+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: audit 2024-09-10T18:55:32.445229+0000 mon.a (mon.0) 1307 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: cluster 2024-09-10T18:55:32.540444+0000 osd.2 (osd.2) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[18629]: cluster 2024-09-10T18:55:32.541456+0000 osd.2 (osd.2) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:32.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: cluster 2024-09-10T18:55:31.412943+0000 mgr.y (mgr.24526) 251 : cluster [DBG] pgmap v245: 364 pgs: 32 creating+peering, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:32.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: audit 2024-09-10T18:55:31.631774+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:32.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: cluster 2024-09-10T18:55:31.967422+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:55:32.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: audit 2024-09-10T18:55:31.971968+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:32.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: audit 2024-09-10T18:55:32.231847+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: audit 2024-09-10T18:55:32.445229+0000 mon.a (mon.0) 1307 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: cluster 2024-09-10T18:55:32.540444+0000 osd.2 (osd.2) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:32.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:32 smithi083 bash[32381]: cluster 2024-09-10T18:55:32.541456+0000 osd.2 (osd.2) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:32.815 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: cluster 2024-09-10T18:55:31.412943+0000 mgr.y (mgr.24526) 251 : cluster [DBG] pgmap v245: 364 pgs: 32 creating+peering, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: audit 2024-09-10T18:55:31.631774+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: cluster 2024-09-10T18:55:31.967422+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: audit 2024-09-10T18:55:31.971968+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: audit 2024-09-10T18:55:32.231847+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: audit 2024-09-10T18:55:32.445229+0000 mon.a (mon.0) 1307 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:55:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: cluster 2024-09-10T18:55:32.540444+0000 osd.2 (osd.2) 79 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:32 smithi165 bash[22591]: cluster 2024-09-10T18:55:32.541456+0000 osd.2 (osd.2) 80 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:32.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:32.953 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:32.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:33.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.090 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:33.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:33.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:33.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:33.365 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:33.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:33.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:33.640 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:33.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.633494+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.970971+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.971523+0000 mon.a (mon.0) 1309 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: cluster 2024-09-10T18:55:32.971620+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.973577+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.974623+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[32381]: audit 2024-09-10T18:55:32.985381+0000 mon.a (mon.0) 1312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.633494+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.970971+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.971523+0000 mon.a (mon.0) 1309 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:33.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: cluster 2024-09-10T18:55:32.971620+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:55:33.757 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.973577+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.757 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.974623+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:33.757 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:33 smithi083 bash[18629]: audit 2024-09-10T18:55:32.985381+0000 mon.a (mon.0) 1312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.633494+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.970971+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? v1:172.21.15.83:0/1036124459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi083-77648-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.971523+0000 mon.a (mon.0) 1309 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: cluster 2024-09-10T18:55:32.971620+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.973577+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.974623+0000 mon.a (mon.0) 1311 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:33.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:33 smithi165 bash[22591]: audit 2024-09-10T18:55:32.985381+0000 mon.a (mon.0) 1312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:55:33.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:33.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:33.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:34.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.048 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:34.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:34.181 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:34.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.321 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:34.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.458 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:34.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.591 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:34.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.730 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:34.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:34.847 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: cluster 2024-09-10T18:55:33.413937+0000 mgr.y (mgr.24526) 252 : cluster [DBG] pgmap v248: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:34.847 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.635359+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:34.847 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: cluster 2024-09-10T18:55:33.939593+0000 mon.a (mon.0) 1313 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.950393+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.950423+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.950708+0000 mon.a (mon.0) 1315 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: cluster 2024-09-10T18:55:33.951092+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.957987+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.958586+0000 mon.a (mon.0) 1317 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.848 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[32381]: audit 2024-09-10T18:55:33.962833+0000 mon.a (mon.0) 1318 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: cluster 2024-09-10T18:55:33.413937+0000 mgr.y (mgr.24526) 252 : cluster [DBG] pgmap v248: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.635359+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: cluster 2024-09-10T18:55:33.939593+0000 mon.a (mon.0) 1313 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.950393+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.950423+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.950708+0000 mon.a (mon.0) 1315 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:55:34.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: cluster 2024-09-10T18:55:33.951092+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:55:34.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.957987+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.958586+0000 mon.a (mon.0) 1317 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:34 smithi083 bash[18629]: audit 2024-09-10T18:55:33.962833+0000 mon.a (mon.0) 1318 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: cluster 2024-09-10T18:55:33.413937+0000 mgr.y (mgr.24526) 252 : cluster [DBG] pgmap v248: 396 pgs: 64 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 81 KiB/s wr, 77 op/s 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.635359+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: cluster 2024-09-10T18:55:33.939593+0000 mon.a (mon.0) 1313 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.950393+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? v1:172.21.15.83:0/1537411188' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi083-77640-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.950423+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.950708+0000 mon.a (mon.0) 1315 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: cluster 2024-09-10T18:55:33.951092+0000 mon.a (mon.0) 1316 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-09-10T18:55:34.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.957987+0000 mon.c (mon.2) 235 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.958586+0000 mon.a (mon.0) 1317 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]: dispatch 2024-09-10T18:55:34.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:34 smithi165 bash[22591]: audit 2024-09-10T18:55:33.962833+0000 mon.a (mon.0) 1318 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:34.866 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:34.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:34.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:35.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:35.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:35.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:35.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:35.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:35.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:35.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:35.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:35.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:35.668 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:35.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:35.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3336391354 >> v1:172.21.15.83:6790/0 conn(0x7f059c1042e0 legacy=0x7f059c1066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3336391354 shutdown_connections 2024-09-10T18:55:35.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3336391354 >> 172.21.15.83:0/3336391354 conn(0x7f059c074a40 msgr2=0x7f059c0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:35.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3336391354 shutdown_connections 2024-09-10T18:55:35.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3336391354 wait complete. 2024-09-10T18:55:35.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 librados: starting msgr at 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 librados: starting objecter 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 Processor -- start 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- start start 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 librados: setting wanted keys 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 librados: calling monclient init 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f059c11bf30 con 0x7f059c1042e0 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f059c11d0f0 con 0x7f059c101570 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a3bb2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f059c11e2f0 con 0x7f059c107e40 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a214f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f059c107e40 0x7f059c11a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55068/0 (socket says 172.21.15.83:55068) 2024-09-10T18:55:35.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05a214f640 1 -- 172.21.15.83:0/3165951893 learned_addr learned my addr 172.21.15.83:0/3165951893 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter ms_handle_connect 0x7f059c1042e0 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter ms_handle_connect 0x7f059c107e40 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter ms_handle_connect 0x7f059c101570 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308643282 0 0) 0x7f059c11bf30 con 0x7f059c1042e0 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0588003120 con 0x7f059c1042e0 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327478331 0 0) 0x7f059c11e2f0 con 0x7f059c107e40 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059c11bf30 con 0x7f059c107e40 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1726721610 0 0) 0x7f059c11bf30 con 0x7f059c107e40 2024-09-10T18:55:35.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f059c11e2f0 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0594001e20 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.778+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4130677070 0 0) 0x7f0588003120 con 0x7f059c1042e0 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f059c11bf30 con 0x7f059c1042e0 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f058c0015d0 con 0x7f059c1042e0 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 34194703 0 0) 0x7f059c11e2f0 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 >> v1:172.21.15.165:6789/0 conn(0x7f059c101570 legacy=0x7f059c0ff460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 >> v1:172.21.15.83:6789/0 conn(0x7f059c1042e0 legacy=0x7f059c0ffb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c11f4f0 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c11e4e0 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05a3bb2640 10 client.48626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c11e930 con 0x7f059c107e40 2024-09-10T18:55:35.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05a3bb2640 1 librados: init done 2024-09-10T18:55:35.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.637219+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:35.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: cluster 2024-09-10T18:55:34.945670+0000 mon.a (mon.0) 1319 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:35.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.956809+0000 mon.a (mon.0) 1320 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]': finished 2024-09-10T18:55:35.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.957053+0000 mon.a (mon.0) 1321 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:35.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: cluster 2024-09-10T18:55:34.957444+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:55:35.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.960692+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:35.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.960750+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:35.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:35 smithi165 bash[22591]: audit 2024-09-10T18:55:34.974613+0000 mon.a (mon.0) 1324 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0594005070 con 0x7f059c107e40 2024-09-10T18:55:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0594005de0 con 0x7f059c107e40 2024-09-10T18:55:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c0fcc80 con 0x7f059c107e40 2024-09-10T18:55:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f059401c2e0 con 0x7f059c107e40 2024-09-10T18:55:35.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(179..179 src has 69..179) v4 ==== 9325+0+0 (unknown 2066703989 0 0) 0x7f05940022f0 con 0x7f059c107e40 2024-09-10T18:55:35.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 10 client.48626.objecter ms_dispatch 0x7f059c000b90 osd_map(179..179 src has 69..179) v4 2024-09-10T18:55:35.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 3 client.48626.objecter handle_osd_map got epochs [179,179] > 0 2024-09-10T18:55:35.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 3 client.48626.objecter handle_osd_map decoding full epoch 179 2024-09-10T18:55:35.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 20 client.48626.objecter dump_active .. 0 homeless 2024-09-10T18:55:35.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.782+0000 7f05927fc640 10 client.48626.objecter ms_handle_connect 0x7f05880759f0 2024-09-10T18:55:35.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.790+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0594057d00 con 0x7f059c107e40 2024-09-10T18:55:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- 172.21.15.83:0/4084779156 >> v1:172.21.15.83:6789/0 conn(0x7f1cf0102670 legacy=0x7f1cf0102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- 172.21.15.83:0/4084779156 shutdown_connections 2024-09-10T18:55:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- 172.21.15.83:0/4084779156 >> 172.21.15.83:0/4084779156 conn(0x7f1cf0074a20 msgr2=0x7f1cf00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- 172.21.15.83:0/4084779156 shutdown_connections 2024-09-10T18:55:35.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- 172.21.15.83:0/4084779156 wait complete. 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 librados: starting msgr at 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 librados: starting objecter 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 Processor -- start 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 -- start start 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 librados: setting wanted keys 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.798+0000 7f1cf6c72640 1 librados: calling monclient init 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cf011e200 con 0x7f1cf0102670 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cf011f3c0 con 0x7f1cf01053a0 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cf01205c0 con 0x7f1cf0108f60 2024-09-10T18:55:35.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf520f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f1cf0108f60 0x7f1cf011c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55090/0 (socket says 172.21.15.83:55090) 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf520f640 1 -- 172.21.15.83:0/2571981098 learned_addr learned my addr 172.21.15.83:0/2571981098 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf0102670 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf01053a0 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter ms_handle_connect 0x7f1cf0108f60 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369225265 0 0) 0x7f1cf011e200 con 0x7f1cf0102670 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd80034e0 con 0x7f1cf0102670 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940791212 0 0) 0x7f1cf011f3c0 con 0x7f1cf01053a0 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf011e200 con 0x7f1cf01053a0 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533290231 0 0) 0x7f1cf01205c0 con 0x7f1cf0108f60 2024-09-10T18:55:35.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf011f3c0 con 0x7f1cf0108f60 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2311629696 0 0) 0x7f1cd80034e0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cf01205c0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1ce00020f0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348748158 0 0) 0x7f1cf011e200 con 0x7f1cf01053a0 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cd80034e0 con 0x7f1cf01053a0 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1ce40018c0 con 0x7f1cf01053a0 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1869531649 0 0) 0x7f1cf01205c0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6790/0 conn(0x7f1cf0108f60 legacy=0x7f1cf011c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.165:6789/0 conn(0x7f1cf01053a0 legacy=0x7f1cf0107720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf01217c0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf011f5b0 con 0x7f1cf0102670 2024-09-10T18:55:35.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 10 client.24642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:35.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf011fa80 con 0x7f1cf0102670 2024-09-10T18:55:35.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 librados: init done 2024-09-10T18:55:35.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1ce0002300 con 0x7f1cf0102670 2024-09-10T18:55:35.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1ce0003510 con 0x7f1cf0102670 2024-09-10T18:55:35.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1cdc004ff0 con 0x7f1cf0102670 2024-09-10T18:55:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.814 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:35.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:35.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.802+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1ce0002af0 con 0x7f1cf0102670 2024-09-10T18:55:35.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(179..179 src has 69..179) v4 ==== 9325+0+0 (unknown 2066703989 0 0) 0x7f1ce0088450 con 0x7f1cf0102670 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 osd_map(179..179 src has 69..179) v4 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 3 client.24642.objecter handle_osd_map got epochs [179,179] > 0 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 3 client.24642.objecter handle_osd_map decoding full epoch 179 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 20 client.24642.objecter dump_active .. 0 homeless 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v179) v1 ==== 54+0+16 (unknown 4107056081 0 2420000192) 0x7f1ce008a840 con 0x7f1cf0102670 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cd80714e0 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 1 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 1 2024-09-10T18:55:35.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 0 1 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 4 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 10 client.24642.objecter _send_command 1 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc004a60 con 0x7f1cdc004610 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc004610 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc004a60 con 0x7f1cdc004610 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 1 = system:0 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 4 2024-09-10T18:55:35.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.806+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 0 1 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 2 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 2 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc005290 osd=0 3 2024-09-10T18:55:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 0 2 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 4 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 10 client.24642.objecter _send_command 2 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc004e40 con 0x7f1cdc004610 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc004e40 con 0x7f1cdc004610 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 4 2024-09-10T18:55:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 0 2 2024-09-10T18:55:35.833 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:55:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:55:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 1 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 4 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 10 client.24642.objecter _send_command 3 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc002350 con 0x7f1cdc004150 2024-09-10T18:55:35.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.826+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc004150 2024-09-10T18:55:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.830+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc002350 con 0x7f1cdc004150 2024-09-10T18:55:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.830+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:55:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.830+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 3 = system:0 2024-09-10T18:55:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.830+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 4 2024-09-10T18:55:35.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.830+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 1 3 2024-09-10T18:55:35.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:35.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 4 2024-09-10T18:55:35.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 4 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc003fe0 osd=1 3 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 1 4 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 4 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 10 client.24642.objecter _send_command 4 2024-09-10T18:55:35.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.854+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc0018f0 con 0x7f1cdc004150 2024-09-10T18:55:35.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc0018f0 con 0x7f1cdc004150 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 4 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 1 4 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:55:35.866 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 5 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 5 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 2 5 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 4 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 10 client.24642.objecter _send_command 5 2024-09-10T18:55:35.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.858+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc003d40 con 0x7f1cdc00d350 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc00d350 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc003d40 con 0x7f1cdc00d350 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 5 = system:0 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 4 2024-09-10T18:55:35.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.862+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 2 5 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 6 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 6 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc003620 osd=2 3 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 2 6 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 4 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 10 client.24642.objecter _send_command 6 2024-09-10T18:55:35.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc00e6c0 con 0x7f1cdc00d350 2024-09-10T18:55:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc00e6c0 con 0x7f1cdc00d350 2024-09-10T18:55:35.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:35.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:35.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 4 2024-09-10T18:55:35.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.890+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 2 6 2024-09-10T18:55:35.898 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:55:35.899 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:55:35.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 7 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 7 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 3 7 2024-09-10T18:55:35.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 4 2024-09-10T18:55:35.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 10 client.24642.objecter _send_command 7 2024-09-10T18:55:35.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc015aa0 con 0x7f1cdc011fa0 2024-09-10T18:55:35.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc011fa0 2024-09-10T18:55:35.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc015aa0 con 0x7f1cdc011fa0 2024-09-10T18:55:35.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:55:35.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 7 = system:0 2024-09-10T18:55:35.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 4 2024-09-10T18:55:35.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.894+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 3 7 2024-09-10T18:55:35.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:35.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:35.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 8 2024-09-10T18:55:35.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 8 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc0032e0 osd=3 3 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 3 8 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 4 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 10 client.24642.objecter _send_command 8 2024-09-10T18:55:35.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.922+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc00e140 con 0x7f1cdc011fa0 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc00e140 con 0x7f1cdc011fa0 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 4 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 3 8 2024-09-10T18:55:35.932 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:55:35.933 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:55:35.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 9 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 9 2024-09-10T18:55:35.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 4 9 2024-09-10T18:55:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 4 2024-09-10T18:55:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 10 client.24642.objecter _send_command 9 2024-09-10T18:55:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc00e070 con 0x7f1cdc00cea0 2024-09-10T18:55:35.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.926+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc00cea0 2024-09-10T18:55:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.930+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc00e070 con 0x7f1cdc00cea0 2024-09-10T18:55:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.930+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:55:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.930+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 9 = system:0 2024-09-10T18:55:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.930+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 4 2024-09-10T18:55:35.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.930+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 4 9 2024-09-10T18:55:35.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:35.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 10 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 10 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc00df00 osd=4 3 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 4 10 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 4 2024-09-10T18:55:35.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 10 client.24642.objecter _send_command 10 2024-09-10T18:55:35.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.946+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc01a4c0 con 0x7f1cdc00cea0 2024-09-10T18:55:35.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc01a4c0 con 0x7f1cdc00cea0 2024-09-10T18:55:35.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:35.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:35.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 4 2024-09-10T18:55:35.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 4 10 2024-09-10T18:55:35.959 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:55:35.960 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:55:35.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:35.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 11 2024-09-10T18:55:35.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 11 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 5 11 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 4 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 10 client.24642.objecter _send_command 11 2024-09-10T18:55:35.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.954+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc01e830 con 0x7f1cdc01ab40 2024-09-10T18:55:35.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.958+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc01ab40 2024-09-10T18:55:35.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-10T18:55:35.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-09-10T18:55:35.976 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (3534 ms) 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3199 ms) 2024-09-10T18:55:35.977 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2645 ms) 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (3309 ms) 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (12687 ms total) 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5240 ms) 2024-09-10T18:55:35.978 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (6139 ms) 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (11379 ms total) 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2680 ms) 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2792 ms) 2024-09-10T18:55:35.979 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2042 ms) 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1956 ms) 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9470 ms total) 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-10T18:55:35.980 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4018 ms) 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4025 ms) 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8043 ms total) 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (70135 ms total) 2024-09-10T18:55:35.981 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-09-10T18:55:35.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.970+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc01e830 con 0x7f1cdc01ab40 2024-09-10T18:55:35.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.970+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:55:35.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.970+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 11 = system:0 2024-09-10T18:55:35.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.970+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 4 2024-09-10T18:55:35.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.970+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 5 11 2024-09-10T18:55:36.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 12 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 12 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc01a4c0 osd=5 3 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 5 12 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 4 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 10 client.24642.objecter _send_command 12 2024-09-10T18:55:36.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:35.998+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc0200d0 con 0x7f1cdc01ab40 2024-09-10T18:55:36.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc0200d0 con 0x7f1cdc01ab40 2024-09-10T18:55:36.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:36.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:36.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 4 2024-09-10T18:55:36.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 5 12 2024-09-10T18:55:36.007 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:55:36.007 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 13 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 13 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 6 13 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 4 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 10 client.24642.objecter _send_command 13 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc020c30 con 0x7f1cdc020750 2024-09-10T18:55:36.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc020750 2024-09-10T18:55:36.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc020c30 con 0x7f1cdc020750 2024-09-10T18:55:36.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:55:36.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 13 = system:0 2024-09-10T18:55:36.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 4 2024-09-10T18:55:36.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.002+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 6 13 2024-09-10T18:55:36.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.637219+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:36.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: cluster 2024-09-10T18:55:34.945670+0000 mon.a (mon.0) 1319 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:36.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.956809+0000 mon.a (mon.0) 1320 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]': finished 2024-09-10T18:55:36.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.957053+0000 mon.a (mon.0) 1321 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:36.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: cluster 2024-09-10T18:55:34.957444+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.960692+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.960750+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[18629]: audit 2024-09-10T18:55:34.974613+0000 mon.a (mon.0) 1324 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.637219+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: cluster 2024-09-10T18:55:34.945670+0000 mon.a (mon.0) 1319 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.956809+0000 mon.a (mon.0) 1320 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-13", "mode": "writeback"}]': finished 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.957053+0000 mon.a (mon.0) 1321 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: cluster 2024-09-10T18:55:34.957444+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.960692+0000 mon.c (mon.2) 237 : audit [INF] from='client.? v1:172.21.15.83:0/1009673753' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.960750+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:35 smithi083 bash[32381]: audit 2024-09-10T18:55:34.974613+0000 mon.a (mon.0) 1324 : audit [INF] from='client.48572 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]: dispatch 2024-09-10T18:55:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 14 2024-09-10T18:55:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 14 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc0200d0 osd=6 3 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 6 14 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 4 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 10 client.24642.objecter _send_command 14 2024-09-10T18:55:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.030+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc020c30 con 0x7f1cdc020750 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc020c30 con 0x7f1cdc020750 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 4 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 6 14 2024-09-10T18:55:36.039 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 15 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 15 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 7 15 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 4 2024-09-10T18:55:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 10 client.24642.objecter _send_command 15 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cdc01fd00 con 0x7f1cdc020e70 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 10 client.24642.objecter ms_handle_connect 0x7f1cdc020e70 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1cdc01fd00 con 0x7f1cdc020e70 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 15 = system:0 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 4 2024-09-10T18:55:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.034+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 7 15 2024-09-10T18:55:36.051 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 10 client.24642.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign -1 16 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 20 client.24642.objecter _get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_remove -1 16 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 20 client.24642.objecter get_session s=0x7f1cdc020c30 osd=7 3 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 15 client.24642.objecter _session_command_op_assign 7 16 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 4 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 10 client.24642.objecter _send_command 16 2024-09-10T18:55:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.058+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1cdc01fd00 con 0x7f1cdc020e70 2024-09-10T18:55:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1ced7fa640 1 -- 172.21.15.83:0/2571981098 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1cdc01fd00 con 0x7f1cdc020e70 2024-09-10T18:55:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1ced7fa640 10 client.24642.objecter ms_dispatch 0x7f1cf0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1ced7fa640 10 client.24642.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1ced7fa640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 4 2024-09-10T18:55:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1ced7fa640 15 client.24642.objecter _session_command_op_remove 7 16 2024-09-10T18:55:36.069 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:55:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:55:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 librados: watch_flush enter 2024-09-10T18:55:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 librados: watch_flush exit 2024-09-10T18:55:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.0 2024-09-10T18:55:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6801/2535527483 conn(0x7f1cdc004610 legacy=0x7f1cdc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc005290 osd=0 1 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.1 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6805/2309624637 conn(0x7f1cdc004150 legacy=0x7f1cdc00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003fe0 osd=1 1 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.2 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6809/386094220 conn(0x7f1cdc00d350 legacy=0x7f1cdc00d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc003620 osd=2 1 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.3 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6813/1279688021 conn(0x7f1cdc011fa0 legacy=0x7f1cdc014380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf520f640 1 -- 172.21.15.83:0/2571981098 reap_dead start 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0032e0 osd=3 1 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.4 2024-09-10T18:55:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.165:6800/3225515265 conn(0x7f1cdc00cea0 legacy=0x7f1cdc018da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc00df00 osd=4 1 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.5 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.165:6804/571173777 conn(0x7f1cdc01ab40 legacy=0x7f1cdc01cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc01a4c0 osd=5 1 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.6 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.165:6808/1317922936 conn(0x7f1cdc020750 legacy=0x7f1cdc01e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc0200d0 osd=6 1 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter close_session for osd.7 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.165:6812/1610151650 conn(0x7f1cdc020e70 legacy=0x7f1cdc01f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter put_session s=0x7f1cdc020c30 osd=7 1 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 20 client.24642.objecter shutdown clearing up homeless session... 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 10 client.24642.objecter successfully canceled tick 2024-09-10T18:55:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf520f640 1 -- 172.21.15.83:0/2571981098 reap_dead start 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1cd80714e0 legacy=0x7f1cd8073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> v1:172.21.15.83:6789/0 conn(0x7f1cf0102670 legacy=0x7f1cf0107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 shutdown_connections 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.062+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 >> 172.21.15.83:0/2571981098 conn(0x7f1cf0074a20 msgr2=0x7f1cf00f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.066+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 shutdown_connections 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.066+0000 7f1cf6c72640 1 -- 172.21.15.83:0/2571981098 wait complete. 2024-09-10T18:55:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.066+0000 7f1cf6c72640 1 librados: shutdown 2024-09-10T18:55:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.086+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f059c04bf60 con 0x7f059c107e40 2024-09-10T18:55:36.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:36.212 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:36.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:36.347 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:36.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:36.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:36.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:36.617 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:36.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:36.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: cluster 2024-09-10T18:55:35.414864+0000 mgr.y (mgr.24526) 253 : cluster [DBG] pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.638892+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: cluster 2024-09-10T18:55:35.714972+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: cluster 2024-09-10T18:55:35.715794+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.810274+0000 mon.a (mon.0) 1325 : audit [DBG] from='client.? v1:172.21.15.83:0/2571981098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.962357+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.962753+0000 mon.a (mon.0) 1327 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: cluster 2024-09-10T18:55:35.963016+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.973524+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.83:0/993055759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:35.974304+0000 mon.a (mon.0) 1329 : audit [INF] from='client.48620 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.092408+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3165951893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.093257+0000 mon.a (mon.0) 1330 : audit [INF] from='client.48626 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.124595+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.125831+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.126953+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.128037+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.128678+0000 mgr.y (mgr.24526) 254 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: cluster 2024-09-10T18:55:35.414864+0000 mgr.y (mgr.24526) 253 : cluster [DBG] pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.638892+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: cluster 2024-09-10T18:55:35.714972+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: cluster 2024-09-10T18:55:35.715794+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.810274+0000 mon.a (mon.0) 1325 : audit [DBG] from='client.? v1:172.21.15.83:0/2571981098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.962357+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.962753+0000 mon.a (mon.0) 1327 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: cluster 2024-09-10T18:55:35.963016+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:55:36.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.973524+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.83:0/993055759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:35.974304+0000 mon.a (mon.0) 1329 : audit [INF] from='client.48620 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.092408+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3165951893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.093257+0000 mon.a (mon.0) 1330 : audit [INF] from='client.48626 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.124595+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.125831+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.126953+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.128037+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.128678+0000 mgr.y (mgr.24526) 254 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.129050+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.129827+0000 mgr.y (mgr.24526) 255 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.130037+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.131012+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.131019+0000 mgr.y (mgr.24526) 256 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.131940+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.132013+0000 mgr.y (mgr.24526) 257 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.132954+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.133067+0000 mgr.y (mgr.24526) 258 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.133952+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.134014+0000 mgr.y (mgr.24526) 259 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.135000+0000 mgr.y (mgr.24526) 260 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.135941+0000 mgr.y (mgr.24526) 261 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.136948+0000 mgr.y (mgr.24526) 262 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[18629]: audit 2024-09-10T18:55:36.137947+0000 mgr.y (mgr.24526) 263 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.129050+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.129827+0000 mgr.y (mgr.24526) 255 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.130037+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.131012+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.131019+0000 mgr.y (mgr.24526) 256 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.131940+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.132013+0000 mgr.y (mgr.24526) 257 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.132954+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.133067+0000 mgr.y (mgr.24526) 258 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.133952+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.134014+0000 mgr.y (mgr.24526) 259 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.135000+0000 mgr.y (mgr.24526) 260 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.135941+0000 mgr.y (mgr.24526) 261 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.136948+0000 mgr.y (mgr.24526) 262 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:36 smithi083 bash[32381]: audit 2024-09-10T18:55:36.137947+0000 mgr.y (mgr.24526) 263 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.752 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:36.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: cluster 2024-09-10T18:55:35.414864+0000 mgr.y (mgr.24526) 253 : cluster [DBG] pgmap v251: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:36.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.638892+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:36.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: cluster 2024-09-10T18:55:35.714972+0000 osd.3 (osd.3) 155 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:36.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: cluster 2024-09-10T18:55:35.715794+0000 osd.3 (osd.3) 156 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:36.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.810274+0000 mon.a (mon.0) 1325 : audit [DBG] from='client.? v1:172.21.15.83:0/2571981098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.962357+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? v1:172.21.15.83:0/2583724390' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi083-77648-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.962753+0000 mon.a (mon.0) 1327 : audit [INF] from='client.48572 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi083-77907-14"}]': finished 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: cluster 2024-09-10T18:55:35.963016+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.973524+0000 mon.c (mon.2) 239 : audit [INF] from='client.? v1:172.21.15.83:0/993055759' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:35.974304+0000 mon.a (mon.0) 1329 : audit [INF] from='client.48620 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.092408+0000 mon.c (mon.2) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3165951893' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.093257+0000 mon.a (mon.0) 1330 : audit [INF] from='client.48626 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.124595+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.125831+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.126953+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.128037+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.128678+0000 mgr.y (mgr.24526) 254 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.0"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.129050+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.129827+0000 mgr.y (mgr.24526) 255 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.1"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.130037+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.131012+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.131019+0000 mgr.y (mgr.24526) 256 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.2"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.131940+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.132013+0000 mgr.y (mgr.24526) 257 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.3"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.132954+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.133067+0000 mgr.y (mgr.24526) 258 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.4"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.133952+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.134014+0000 mgr.y (mgr.24526) 259 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.5"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.135000+0000 mgr.y (mgr.24526) 260 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.6"}]: dispatch 2024-09-10T18:55:36.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.135941+0000 mgr.y (mgr.24526) 261 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.7"}]: dispatch 2024-09-10T18:55:36.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.136948+0000 mgr.y (mgr.24526) 262 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.8"}]: dispatch 2024-09-10T18:55:36.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:36 smithi165 bash[22591]: audit 2024-09-10T18:55:36.137947+0000 mgr.y (mgr.24526) 263 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "97.9"}]: dispatch 2024-09-10T18:55:36.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:36.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:36.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.966+0000 7f05927fc640 1 -- 172.21.15.83:0/3165951893 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v181) v1 ==== 113+0+0 (unknown 2412877517 0 0) 0x7f05940576a0 con 0x7f059c107e40 2024-09-10T18:55:36.976 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T18:55:36.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.974+0000 7f05a3bb2640 10 librados: watch_flush enter 2024-09-10T18:55:36.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 10 librados: watch_flush exit 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 20 client.48626.objecter shutdown clearing up homeless session... 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 10 client.48626.objecter successfully canceled tick 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 >> v1:172.21.15.83:6800/3071793735 conn(0x7f05880759f0 legacy=0x7f0588077ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 >> v1:172.21.15.83:6790/0 conn(0x7f059c107e40 legacy=0x7f059c11a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 shutdown_connections 2024-09-10T18:55:36.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 >> 172.21.15.83:0/3165951893 conn(0x7f059c074a40 msgr2=0x7f059c0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:36.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 shutdown_connections 2024-09-10T18:55:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.978+0000 7f05a3bb2640 1 -- 172.21.15.83:0/3165951893 wait complete. 2024-09-10T18:55:36.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:36.982+0000 7f05a3bb2640 1 librados: shutdown 2024-09-10T18:55:36.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:37.018 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:37.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:37.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:37.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:37.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:37.423 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:37.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:37.560 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:37.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: audit 2024-09-10T18:55:36.640300+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: cluster 2024-09-10T18:55:36.726481+0000 osd.3 (osd.3) 157 : cluster [DBG] 97.6 scrub starts 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: cluster 2024-09-10T18:55:36.733466+0000 osd.3 (osd.3) 158 : cluster [DBG] 97.6 scrub ok 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: audit 2024-09-10T18:55:36.969706+0000 mon.a (mon.0) 1331 : audit [INF] from='client.48620 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: audit 2024-09-10T18:55:36.969957+0000 mon.a (mon.0) 1332 : audit [INF] from='client.48626 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:55:37.678 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[32381]: cluster 2024-09-10T18:55:36.970023+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: audit 2024-09-10T18:55:36.640300+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: cluster 2024-09-10T18:55:36.726481+0000 osd.3 (osd.3) 157 : cluster [DBG] 97.6 scrub starts 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: cluster 2024-09-10T18:55:36.733466+0000 osd.3 (osd.3) 158 : cluster [DBG] 97.6 scrub ok 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: audit 2024-09-10T18:55:36.969706+0000 mon.a (mon.0) 1331 : audit [INF] from='client.48620 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: audit 2024-09-10T18:55:36.969957+0000 mon.a (mon.0) 1332 : audit [INF] from='client.48626 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:55:37.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:37 smithi083 bash[18629]: cluster 2024-09-10T18:55:36.970023+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:55:37.699 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:37.802 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:37.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:37.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: audit 2024-09-10T18:55:36.640300+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: cluster 2024-09-10T18:55:36.726481+0000 osd.3 (osd.3) 157 : cluster [DBG] 97.6 scrub starts 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: cluster 2024-09-10T18:55:36.733466+0000 osd.3 (osd.3) 158 : cluster [DBG] 97.6 scrub ok 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: audit 2024-09-10T18:55:36.969706+0000 mon.a (mon.0) 1331 : audit [INF] from='client.48620 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi083-77640-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: audit 2024-09-10T18:55:36.969957+0000 mon.a (mon.0) 1332 : audit [INF] from='client.48626 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T18:55:37.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:37 smithi165 bash[22591]: cluster 2024-09-10T18:55:36.970023+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-09-10T18:55:37.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.968 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:37.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:38.073 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.099 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:38.203 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:38.229 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:38.332 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:38.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:38.490 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:38.593 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:38.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: cluster 2024-09-10T18:55:37.415802+0000 mgr.y (mgr.24526) 264 : cluster [DBG] pgmap v254: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 30 KiB/s rd, 44 KiB/s wr, 95 op/s 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: audit 2024-09-10T18:55:37.641978+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: audit 2024-09-10T18:55:37.693839+0000 mgr.y (mgr.24526) 265 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: cluster 2024-09-10T18:55:37.977092+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: audit 2024-09-10T18:55:37.987560+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.83:0/3697731758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:38 smithi165 bash[22591]: audit 2024-09-10T18:55:37.998920+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: cluster 2024-09-10T18:55:37.415802+0000 mgr.y (mgr.24526) 264 : cluster [DBG] pgmap v254: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 30 KiB/s rd, 44 KiB/s wr, 95 op/s 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: audit 2024-09-10T18:55:37.641978+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: audit 2024-09-10T18:55:37.693839+0000 mgr.y (mgr.24526) 265 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: cluster 2024-09-10T18:55:37.977092+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: audit 2024-09-10T18:55:37.987560+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.83:0/3697731758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[18629]: audit 2024-09-10T18:55:37.998920+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: cluster 2024-09-10T18:55:37.415802+0000 mgr.y (mgr.24526) 264 : cluster [DBG] pgmap v254: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 30 KiB/s rd, 44 KiB/s wr, 95 op/s 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: audit 2024-09-10T18:55:37.641978+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: audit 2024-09-10T18:55:37.693839+0000 mgr.y (mgr.24526) 265 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: cluster 2024-09-10T18:55:37.977092+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: audit 2024-09-10T18:55:37.987560+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.83:0/3697731758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:38 smithi083 bash[32381]: audit 2024-09-10T18:55:37.998920+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:38.895 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:38.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:38.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:39.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T18:55:39.048 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:39.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:39.186 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:39.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:39.325 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:39.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:39.460 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:39.564 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:39.598 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:39.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: audit 2024-09-10T18:55:38.643647+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.744005+0000 osd.2 (osd.2) 81 : cluster [DBG] 97.7 scrub starts 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.755498+0000 osd.2 (osd.2) 82 : cluster [DBG] 97.7 scrub ok 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.764258+0000 osd.7 (osd.7) 143 : cluster [DBG] 97.1 scrub starts 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.772360+0000 osd.7 (osd.7) 144 : cluster [DBG] 97.1 scrub ok 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.943608+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:39.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: audit 2024-09-10T18:55:38.996489+0000 mon.a (mon.0) 1337 : audit [INF] from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:38.996648+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: audit 2024-09-10T18:55:39.023933+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.045802+0000 osd.0 (osd.0) 146 : cluster [DBG] 97.2 scrub starts 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.054637+0000 osd.0 (osd.0) 147 : cluster [DBG] 97.2 scrub ok 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.114426+0000 osd.4 (osd.4) 171 : cluster [DBG] 97.9 scrub starts 2024-09-10T18:55:39.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:39 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.117588+0000 osd.4 (osd.4) 172 : cluster [DBG] 97.9 scrub ok 2024-09-10T18:55:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.741 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:39.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:39.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:39.856 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: audit 2024-09-10T18:55:38.643647+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:39.856 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.744005+0000 osd.2 (osd.2) 81 : cluster [DBG] 97.7 scrub starts 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.755498+0000 osd.2 (osd.2) 82 : cluster [DBG] 97.7 scrub ok 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.764258+0000 osd.7 (osd.7) 143 : cluster [DBG] 97.1 scrub starts 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.772360+0000 osd.7 (osd.7) 144 : cluster [DBG] 97.1 scrub ok 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.943608+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: audit 2024-09-10T18:55:38.996489+0000 mon.a (mon.0) 1337 : audit [INF] from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:38.996648+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: audit 2024-09-10T18:55:39.023933+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.045802+0000 osd.0 (osd.0) 146 : cluster [DBG] 97.2 scrub starts 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.054637+0000 osd.0 (osd.0) 147 : cluster [DBG] 97.2 scrub ok 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.114426+0000 osd.4 (osd.4) 171 : cluster [DBG] 97.9 scrub starts 2024-09-10T18:55:39.857 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.117588+0000 osd.4 (osd.4) 172 : cluster [DBG] 97.9 scrub ok 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: audit 2024-09-10T18:55:38.643647+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.744005+0000 osd.2 (osd.2) 81 : cluster [DBG] 97.7 scrub starts 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.755498+0000 osd.2 (osd.2) 82 : cluster [DBG] 97.7 scrub ok 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.764258+0000 osd.7 (osd.7) 143 : cluster [DBG] 97.1 scrub starts 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.772360+0000 osd.7 (osd.7) 144 : cluster [DBG] 97.1 scrub ok 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.943608+0000 mon.a (mon.0) 1336 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: audit 2024-09-10T18:55:38.996489+0000 mon.a (mon.0) 1337 : audit [INF] from='client.49168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi083-77648-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:38.996648+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: audit 2024-09-10T18:55:39.023933+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.045802+0000 osd.0 (osd.0) 146 : cluster [DBG] 97.2 scrub starts 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.054637+0000 osd.0 (osd.0) 147 : cluster [DBG] 97.2 scrub ok 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.114426+0000 osd.4 (osd.4) 171 : cluster [DBG] 97.9 scrub starts 2024-09-10T18:55:39.858 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:39 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.117588+0000 osd.4 (osd.4) 172 : cluster [DBG] 97.9 scrub ok 2024-09-10T18:55:39.876 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:39.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:39.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:40.032 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:40.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:40.161 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:40.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:40.294 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:40.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:40.430 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:40.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:40.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:40.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:40.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:40.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:40.844 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:40.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:55:40.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.614720+0000 osd.6 (osd.6) 113 : cluster [DBG] 97.4 scrub starts 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.621974+0000 osd.6 (osd.6) 114 : cluster [DBG] 97.4 scrub ok 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: audit 2024-09-10T18:55:39.645312+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: audit 2024-09-10T18:55:40.001820+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: cluster 2024-09-10T18:55:40.002301+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:55:40.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[18629]: audit 2024-09-10T18:55:40.646977+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.614720+0000 osd.6 (osd.6) 113 : cluster [DBG] 97.4 scrub starts 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.621974+0000 osd.6 (osd.6) 114 : cluster [DBG] 97.4 scrub ok 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: audit 2024-09-10T18:55:39.645312+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: audit 2024-09-10T18:55:40.001820+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: cluster 2024-09-10T18:55:40.002301+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:55:40.959 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:40 smithi083 bash[32381]: audit 2024-09-10T18:55:40.646977+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:40.977 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:40.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:41.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:41.108 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:41.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:41.239 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:41.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.614720+0000 osd.6 (osd.6) 113 : cluster [DBG] 97.4 scrub starts 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.621974+0000 osd.6 (osd.6) 114 : cluster [DBG] 97.4 scrub ok 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: audit 2024-09-10T18:55:39.645312+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: audit 2024-09-10T18:55:40.001820+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? v1:172.21.15.83:0/732379560' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi083-77640-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: cluster 2024-09-10T18:55:40.002301+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-09-10T18:55:41.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:40 smithi165 bash[22591]: audit 2024-09-10T18:55:40.646977+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:41.372 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:41.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:41.508 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:41.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:41.625 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:41.644 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:41.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:41.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:41.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:41.883 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:55:41.883 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T18:55:41.883 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T18:55:41.883 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:55:41.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- 172.21.15.83:0/3082277898 >> v1:172.21.15.83:6790/0 conn(0x7fe2d8107e30 legacy=0x7fe2d810a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:41.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- 172.21.15.83:0/3082277898 shutdown_connections 2024-09-10T18:55:41.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- 172.21.15.83:0/3082277898 >> 172.21.15.83:0/3082277898 conn(0x7fe2d8075290 msgr2=0x7fe2d80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:41.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- 172.21.15.83:0/3082277898 shutdown_connections 2024-09-10T18:55:41.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- 172.21.15.83:0/3082277898 wait complete. 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 librados: starting msgr at 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 librados: starting objecter 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 Processor -- start 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- start start 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 librados: setting wanted keys 2024-09-10T18:55:41.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 librados: calling monclient init 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2d811af60 con 0x7fe2d8107e30 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2d811c160 con 0x7fe2d81014e0 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2de8a0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2d811d360 con 0x7fe2d81042a0 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d77fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fe2d81042a0 0x7fe2d80fddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55202/0 (socket says 172.21.15.83:55202) 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d77fe640 1 -- 172.21.15.83:0/409092360 learned_addr learned my addr 172.21.15.83:0/409092360 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2dce3d640 1 --1- 172.21.15.83:0/409092360 >> v1:172.21.15.83:6789/0 conn(0x7fe2d8107e30 0x7fe2d8119660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:43506/0 (socket says 172.21.15.83:43506) 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter ms_handle_connect 0x7fe2d8107e30 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter ms_handle_connect 0x7fe2d81042a0 2024-09-10T18:55:41.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter ms_handle_connect 0x7fe2d81014e0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170393616 0 0) 0x7fe2d811af60 con 0x7fe2d8107e30 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c0003120 con 0x7fe2d8107e30 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713733783 0 0) 0x7fe2d811d360 con 0x7fe2d81042a0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d811af60 con 0x7fe2d81042a0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393380358 0 0) 0x7fe2d811c160 con 0x7fe2d81014e0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d811d360 con 0x7fe2d81014e0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.878+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3150529956 0 0) 0x7fe2c0003120 con 0x7fe2d8107e30 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d811c160 con 0x7fe2d8107e30 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540970719 0 0) 0x7fe2d811af60 con 0x7fe2d81042a0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c0003120 con 0x7fe2d81042a0 2024-09-10T18:55:41.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe2d0003060 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe2cc001840 con 0x7fe2d81042a0 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3713843121 0 0) 0x7fe2d811c160 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 >> v1:172.21.15.83:6790/0 conn(0x7fe2d81042a0 legacy=0x7fe2d80fddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 >> v1:172.21.15.165:6789/0 conn(0x7fe2d81014e0 legacy=0x7fe2d80fd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d811e560 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d811d550 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2de8a0640 10 client.24675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d811da20 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2de8a0640 1 librados: init done 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe2d0001580 con 0x7fe2d8107e30 2024-09-10T18:55:41.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe2d00037b0 con 0x7fe2d8107e30 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2c4005010 con 0x7fe2d8107e30 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.882+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe2d0002250 con 0x7fe2d8107e30 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(185..185 src has 69..185) v4 ==== 9181+0+0 (unknown 3245181594 0 0) 0x7fe2d008ad50 con 0x7fe2d8107e30 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 10 client.24675.objecter ms_dispatch 0x7fe2d8000b90 osd_map(185..185 src has 69..185) v4 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 3 client.24675.objecter handle_osd_map got epochs [185,185] > 0 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 3 client.24675.objecter handle_osd_map decoding full epoch 185 2024-09-10T18:55:41.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 20 client.24675.objecter dump_active .. 0 homeless 2024-09-10T18:55:41.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.886+0000 7fe2d4ff9640 10 client.24675.objecter ms_handle_connect 0x7fe2c0071620 2024-09-10T18:55:41.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:41.890+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe2d0056e70 con 0x7fe2d8107e30 2024-09-10T18:55:41.915 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:41.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:42.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: cluster 2024-09-10T18:55:39.416803+0000 mgr.y (mgr.24526) 266 : cluster [DBG] pgmap v257: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 32 KiB/s rd, 50 KiB/s wr, 96 op/s 2024-09-10T18:55:42.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: audit 2024-09-10T18:55:41.031233+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.83:0/202356046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: cluster 2024-09-10T18:55:41.031358+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: audit 2024-09-10T18:55:41.047796+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: audit 2024-09-10T18:55:41.048635+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: cluster 2024-09-10T18:55:41.400446+0000 osd.1 (osd.1) 139 : cluster [DBG] 97.8 scrub starts 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: cluster 2024-09-10T18:55:41.405577+0000 osd.1 (osd.1) 140 : cluster [DBG] 97.8 scrub ok 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[32381]: audit 2024-09-10T18:55:41.648441+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: cluster 2024-09-10T18:55:39.416803+0000 mgr.y (mgr.24526) 266 : cluster [DBG] pgmap v257: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 32 KiB/s rd, 50 KiB/s wr, 96 op/s 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: audit 2024-09-10T18:55:41.031233+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.83:0/202356046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: cluster 2024-09-10T18:55:41.031358+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:55:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: audit 2024-09-10T18:55:41.047796+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: audit 2024-09-10T18:55:41.048635+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: cluster 2024-09-10T18:55:41.400446+0000 osd.1 (osd.1) 139 : cluster [DBG] 97.8 scrub starts 2024-09-10T18:55:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: cluster 2024-09-10T18:55:41.405577+0000 osd.1 (osd.1) 140 : cluster [DBG] 97.8 scrub ok 2024-09-10T18:55:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:41 smithi083 bash[18629]: audit 2024-09-10T18:55:41.648441+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:42.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:42.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:42.118+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe2c40052c0 con 0x7fe2d8107e30 2024-09-10T18:55:42.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: cluster 2024-09-10T18:55:39.416803+0000 mgr.y (mgr.24526) 266 : cluster [DBG] pgmap v257: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 32 KiB/s rd, 50 KiB/s wr, 96 op/s 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: audit 2024-09-10T18:55:41.031233+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.83:0/202356046' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: cluster 2024-09-10T18:55:41.031358+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: audit 2024-09-10T18:55:41.047796+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49186 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: audit 2024-09-10T18:55:41.048635+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: cluster 2024-09-10T18:55:41.400446+0000 osd.1 (osd.1) 139 : cluster [DBG] 97.8 scrub starts 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: cluster 2024-09-10T18:55:41.405577+0000 osd.1 (osd.1) 140 : cluster [DBG] 97.8 scrub ok 2024-09-10T18:55:42.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:41 smithi165 bash[22591]: audit 2024-09-10T18:55:41.648441+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:42.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:42.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:42.328 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:42.433 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:42.465 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:42.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:42.604 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:42.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:42.741 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:42.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:42] "GET /metrics HTTP/1.1" 200 240378 "" "Prometheus/2.43.0" 2024-09-10T18:55:42.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:42.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:42.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:42.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T18:55:42.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.016 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:43.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.038+0000 7fe2d4ff9640 1 -- 172.21.15.83:0/409092360 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v187) v1 ==== 97+0+0 (unknown 585868610 0 0) 0x7fe2d0056810 con 0x7fe2d8107e30 2024-09-10T18:55:43.050 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T18:55:43.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.042+0000 7fe2de8a0640 10 librados: watch_flush enter 2024-09-10T18:55:43.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.046+0000 7fe2de8a0640 10 librados: watch_flush exit 2024-09-10T18:55:43.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.046+0000 7fe2de8a0640 20 client.24675.objecter shutdown clearing up homeless session... 2024-09-10T18:55:43.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.046+0000 7fe2de8a0640 10 client.24675.objecter successfully canceled tick 2024-09-10T18:55:43.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.054+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe2c0071620 legacy=0x7fe2c0073ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:43.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.054+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 >> v1:172.21.15.83:6789/0 conn(0x7fe2d8107e30 legacy=0x7fe2d8119660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:43.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.054+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 shutdown_connections 2024-09-10T18:55:43.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.054+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 >> 172.21.15.83:0/409092360 conn(0x7fe2d8075290 msgr2=0x7fe2d80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:43.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.062+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 shutdown_connections 2024-09-10T18:55:43.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.062+0000 7fe2de8a0640 1 -- 172.21.15.83:0/409092360 wait complete. 2024-09-10T18:55:43.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:43.062+0000 7fe2de8a0640 1 librados: shutdown 2024-09-10T18:55:43.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.122 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: cluster 2024-09-10T18:55:41.417532+0000 mgr.y (mgr.24526) 267 : cluster [DBG] pgmap v260: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.042047+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.042284+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: cluster 2024-09-10T18:55:42.042840+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.053450+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3168604002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.054049+0000 mon.a (mon.0) 1348 : audit [INF] from='client.48659 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.123614+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.126625+0000 mon.b (mon.1) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.127769+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.132139+0000 mon.a (mon.0) 1350 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.123 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[18629]: audit 2024-09-10T18:55:42.649874+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: cluster 2024-09-10T18:55:41.417532+0000 mgr.y (mgr.24526) 267 : cluster [DBG] pgmap v260: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.042047+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.042284+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: cluster 2024-09-10T18:55:42.042840+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.053450+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3168604002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.054049+0000 mon.a (mon.0) 1348 : audit [INF] from='client.48659 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.123614+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:43.124 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.126625+0000 mon.b (mon.1) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:55:43.125 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.127769+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.125 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.132139+0000 mon.a (mon.0) 1350 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.125 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:43 smithi083 bash[32381]: audit 2024-09-10T18:55:42.649874+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:43.146 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:43.250 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:43.281 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: cluster 2024-09-10T18:55:41.417532+0000 mgr.y (mgr.24526) 267 : cluster [DBG] pgmap v260: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.042047+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49186 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77943-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.042284+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? v1:172.21.15.83:0/2928542468' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: cluster 2024-09-10T18:55:42.042840+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.053450+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3168604002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.054049+0000 mon.a (mon.0) 1348 : audit [INF] from='client.48659 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.123614+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:55:43.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.126625+0000 mon.b (mon.1) 165 : audit [DBG] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T18:55:43.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.127769+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.132139+0000 mon.a (mon.0) 1350 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:43.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:43 smithi165 bash[22591]: audit 2024-09-10T18:55:42.649874+0000 mon.c (mon.2) 248 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:43.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:43.417 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:43.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.547 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:43.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.680 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:43.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:43.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:43.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:44.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:44.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:44.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:44.244 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:44.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:44.377 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:44.479 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:44.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:44.514 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:44.618 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.036232+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.042690+0000 mon.a (mon.0) 1352 : audit [INF] from='client.48659 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.042869+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.042924+0000 mon.a (mon.0) 1354 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.042975+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.064059+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2"}]: dispatch 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.064682+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.073851+0000 mon.a (mon.0) 1356 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.651589+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.946983+0000 mon.a (mon.0) 1357 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:44.631 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.962460+0000 mon.a (mon.0) 1358 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.962922+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.967799+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.968639+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3486686660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.971001+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.972723+0000 mon.a (mon.0) 1360 : audit [INF] from='client.48680 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[32381]: audit 2024-09-10T18:55:43.981157+0000 mon.a (mon.0) 1361 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.036232+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.042690+0000 mon.a (mon.0) 1352 : audit [INF] from='client.48659 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.042869+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:44.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.042924+0000 mon.a (mon.0) 1354 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.042975+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.064059+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2"}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.064682+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.073851+0000 mon.a (mon.0) 1356 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.651589+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.946983+0000 mon.a (mon.0) 1357 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.962460+0000 mon.a (mon.0) 1358 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.962922+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.967799+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.968639+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3486686660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.971001+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.972723+0000 mon.a (mon.0) 1360 : audit [INF] from='client.48680 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:44 smithi083 bash[18629]: audit 2024-09-10T18:55:43.981157+0000 mon.a (mon.0) 1361 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.647 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:44.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:44.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.036232+0000 mon.a (mon.0) 1351 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.042690+0000 mon.a (mon.0) 1352 : audit [INF] from='client.48659 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi083-77640-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.042869+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? v1:172.21.15.83:0/409092360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.042924+0000 mon.a (mon.0) 1354 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.042975+0000 mon.a (mon.0) 1355 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.064059+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2"}]: dispatch 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.064682+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.073851+0000 mon.a (mon.0) 1356 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.651589+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.946983+0000 mon.a (mon.0) 1357 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.962460+0000 mon.a (mon.0) 1358 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi083-77788-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.962922+0000 mon.a (mon.0) 1359 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.967799+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.968639+0000 mon.c (mon.2) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3486686660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.971001+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.972723+0000 mon.a (mon.0) 1360 : audit [INF] from='client.48680 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:44.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:44 smithi165 bash[22591]: audit 2024-09-10T18:55:43.981157+0000 mon.a (mon.0) 1361 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T18:55:44.787 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:44.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:44.926 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:44.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:45.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:45.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:45.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:45.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:45.471 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:45.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.186606+0000 osd.4 (osd.4) 173 : cluster [DBG] 97.3 scrub starts 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.193701+0000 osd.4 (osd.4) 174 : cluster [DBG] 97.3 scrub ok 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:43.418378+0000 mgr.y (mgr.24526) 268 : cluster [DBG] pgmap v263: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.564101+0000 mon.a (mon.0) 1362 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.569406+0000 mon.c (mon.2) 251 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.653814+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:44.679509+0000 osd.6 (osd.6) 115 : cluster [DBG] 97.0 scrub starts 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:44.685940+0000 osd.6 (osd.6) 116 : cluster [DBG] 97.0 scrub ok 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.966916+0000 mon.a (mon.0) 1363 : audit [INF] from='client.48680 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.967127+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.967224+0000 mon.a (mon.0) 1364 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: cluster 2024-09-10T18:55:44.967982+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.974229+0000 mon.a (mon.0) 1366 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:44.975033+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.032918+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.033275+0000 mon.a (mon.0) 1368 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.035069+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.035415+0000 mon.a (mon.0) 1369 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.036047+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.589 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:45 smithi165 bash[22591]: audit 2024-09-10T18:55:45.036436+0000 mon.a (mon.0) 1370 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.610 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:45.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.746 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.186606+0000 osd.4 (osd.4) 173 : cluster [DBG] 97.3 scrub starts 2024-09-10T18:55:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.193701+0000 osd.4 (osd.4) 174 : cluster [DBG] 97.3 scrub ok 2024-09-10T18:55:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:43.418378+0000 mgr.y (mgr.24526) 268 : cluster [DBG] pgmap v263: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.564101+0000 mon.a (mon.0) 1362 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.569406+0000 mon.c (mon.2) 251 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.653814+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:44.679509+0000 osd.6 (osd.6) 115 : cluster [DBG] 97.0 scrub starts 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:44.685940+0000 osd.6 (osd.6) 116 : cluster [DBG] 97.0 scrub ok 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.966916+0000 mon.a (mon.0) 1363 : audit [INF] from='client.48680 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.967127+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.967224+0000 mon.a (mon.0) 1364 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: cluster 2024-09-10T18:55:44.967982+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.974229+0000 mon.a (mon.0) 1366 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:44.975033+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.032918+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.033275+0000 mon.a (mon.0) 1368 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.035069+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.186606+0000 osd.4 (osd.4) 173 : cluster [DBG] 97.3 scrub starts 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.193701+0000 osd.4 (osd.4) 174 : cluster [DBG] 97.3 scrub ok 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:43.418378+0000 mgr.y (mgr.24526) 268 : cluster [DBG] pgmap v263: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 25 KiB/s rd, 28 op/s 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.564101+0000 mon.a (mon.0) 1362 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.569406+0000 mon.c (mon.2) 251 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.653814+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:44.679509+0000 osd.6 (osd.6) 115 : cluster [DBG] 97.0 scrub starts 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:44.685940+0000 osd.6 (osd.6) 116 : cluster [DBG] 97.0 scrub ok 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.966916+0000 mon.a (mon.0) 1363 : audit [INF] from='client.48680 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi083-77648-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.967127+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.967224+0000 mon.a (mon.0) 1364 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: cluster 2024-09-10T18:55:44.967982+0000 mon.a (mon.0) 1365 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.974229+0000 mon.a (mon.0) 1366 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:44.975033+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.032918+0000 mon.c (mon.2) 253 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.033275+0000 mon.a (mon.0) 1368 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.035069+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.035415+0000 mon.a (mon.0) 1369 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.036047+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[32381]: audit 2024-09-10T18:55:45.036436+0000 mon.a (mon.0) 1370 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.035415+0000 mon.a (mon.0) 1369 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:45.796 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.036047+0000 mon.c (mon.2) 255 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.796 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:45 smithi083 bash[18629]: audit 2024-09-10T18:55:45.036436+0000 mon.a (mon.0) 1370 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:45.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:45.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:45.990 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:46.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:55:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:46.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:46.177 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:46.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:46.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:46.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:46.447 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:46.531 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:46.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:46.576 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:46.578 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: cluster 2024-09-10T18:55:45.636006+0000 osd.5 (osd.5) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:46.578 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: cluster 2024-09-10T18:55:45.637523+0000 osd.5 (osd.5) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:46.578 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.655182+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:46.578 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.966986+0000 mon.a (mon.0) 1371 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:55:46.578 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.967154+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.967706+0000 mon.a (mon.0) 1373 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: cluster 2024-09-10T18:55:45.967890+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.970652+0000 mon.c (mon.2) 257 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.971672+0000 mon.a (mon.0) 1375 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.972275+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.579 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[18629]: audit 2024-09-10T18:55:45.988663+0000 mon.a (mon.0) 1376 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: cluster 2024-09-10T18:55:45.636006+0000 osd.5 (osd.5) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: cluster 2024-09-10T18:55:45.637523+0000 osd.5 (osd.5) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.655182+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.966986+0000 mon.a (mon.0) 1371 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.967154+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:46.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.967706+0000 mon.a (mon.0) 1373 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:46.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: cluster 2024-09-10T18:55:45.967890+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:55:46.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.970652+0000 mon.c (mon.2) 257 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.971672+0000 mon.a (mon.0) 1375 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.972275+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:46 smithi083 bash[32381]: audit 2024-09-10T18:55:45.988663+0000 mon.a (mon.0) 1376 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: cluster 2024-09-10T18:55:45.636006+0000 osd.5 (osd.5) 141 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: cluster 2024-09-10T18:55:45.637523+0000 osd.5 (osd.5) 142 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.655182+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.966986+0000 mon.a (mon.0) 1371 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T18:55:46.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.967154+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? v1:172.21.15.83:0/1632292978' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77640-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.967706+0000 mon.a (mon.0) 1373 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: cluster 2024-09-10T18:55:45.967890+0000 mon.a (mon.0) 1374 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.970652+0000 mon.c (mon.2) 257 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.971672+0000 mon.a (mon.0) 1375 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.972275+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.83:0/1434403536' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:46 smithi165 bash[22591]: audit 2024-09-10T18:55:45.988663+0000 mon.a (mon.0) 1376 : audit [INF] from='client.24829 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T18:55:46.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:46.717 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:46.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:46.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:46.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:46.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:47.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.132 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:47.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:47.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:47.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.397 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:47.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:47.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:47.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:47.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:47.738 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:47.769 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:47.780 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:47.800 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:47.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: cluster 2024-09-10T18:55:45.419318+0000 mgr.y (mgr.24526) 269 : cluster [DBG] pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: audit 2024-09-10T18:55:46.657156+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: audit 2024-09-10T18:55:47.421490+0000 mon.a (mon.0) 1377 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: cluster 2024-09-10T18:55:47.421744+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: audit 2024-09-10T18:55:47.423031+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3826671504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[32381]: audit 2024-09-10T18:55:47.424030+0000 mon.a (mon.0) 1379 : audit [INF] from='client.48710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: cluster 2024-09-10T18:55:45.419318+0000 mgr.y (mgr.24526) 269 : cluster [DBG] pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: audit 2024-09-10T18:55:46.657156+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: audit 2024-09-10T18:55:47.421490+0000 mon.a (mon.0) 1377 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: cluster 2024-09-10T18:55:47.421744+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:55:47.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: audit 2024-09-10T18:55:47.423031+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3826671504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:47.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:47 smithi083 bash[18629]: audit 2024-09-10T18:55:47.424030+0000 mon.a (mon.0) 1379 : audit [INF] from='client.48710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:47.936 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:47.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-10T18:55:47.999 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: seed 77788 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (280 ms) 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (38 ms) 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (15 ms) 2024-09-10T18:55:48.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (12 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (7 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (16 ms) 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-10T18:55:48.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (9 ms) 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2898 ms) 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (573 ms) 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: done waiting, doing copies 2024-09-10T18:55:48.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: done waiting 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60804 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5295 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (2 ms) 2024-09-10T18:55:48.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-10T18:55:48.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-10T18:55:48.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:48.059 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: cluster 2024-09-10T18:55:45.419318+0000 mgr.y (mgr.24526) 269 : cluster [DBG] pgmap v266: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: audit 2024-09-10T18:55:46.657156+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: audit 2024-09-10T18:55:47.421490+0000 mon.a (mon.0) 1377 : audit [INF] from='client.24829 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi083-77788-1","app":"app1","key":"key1"}]': finished 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: cluster 2024-09-10T18:55:47.421744+0000 mon.a (mon.0) 1378 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: audit 2024-09-10T18:55:47.423031+0000 mon.c (mon.2) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3826671504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:48.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:47 smithi165 bash[22591]: audit 2024-09-10T18:55:47.424030+0000 mon.a (mon.0) 1379 : audit [INF] from='client.48710 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:48.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:48.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:48.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:48.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:48.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:48.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:48.469 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:48.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:48.608 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:48.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:48.746 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:48.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:48.878 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:48.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:48.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:49.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: cluster 2024-09-10T18:55:47.420167+0000 mgr.y (mgr.24526) 270 : cluster [DBG] pgmap v269: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:49.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.658783+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.705328+0000 mgr.y (mgr.24526) 271 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:49.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.981653+0000 mon.a (mon.0) 1380 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:55:49.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.981808+0000 mon.a (mon.0) 1381 : audit [INF] from='client.48710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: cluster 2024-09-10T18:55:47.982129+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.991338+0000 mon.c (mon.2) 261 : audit [INF] from='client.? v1:172.21.15.83:0/262730917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:47.992006+0000 mon.a (mon.0) 1383 : audit [INF] from='client.48716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:48.022143+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:48.024226+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:48.024626+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:48.627188+0000 mon.c (mon.2) 262 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:55:49.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:48 smithi165 bash[22591]: audit 2024-09-10T18:55:48.660053+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.144 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: cluster 2024-09-10T18:55:47.420167+0000 mgr.y (mgr.24526) 270 : cluster [DBG] pgmap v269: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.658783+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.705328+0000 mgr.y (mgr.24526) 271 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.981653+0000 mon.a (mon.0) 1380 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.981808+0000 mon.a (mon.0) 1381 : audit [INF] from='client.48710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: cluster 2024-09-10T18:55:47.982129+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.991338+0000 mon.c (mon.2) 261 : audit [INF] from='client.? v1:172.21.15.83:0/262730917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:47.992006+0000 mon.a (mon.0) 1383 : audit [INF] from='client.48716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:48.022143+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:48.024226+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:48.024626+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:48.627188+0000 mon.c (mon.2) 262 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[18629]: audit 2024-09-10T18:55:48.660053+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: cluster 2024-09-10T18:55:47.420167+0000 mgr.y (mgr.24526) 270 : cluster [DBG] pgmap v269: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.658783+0000 mon.c (mon.2) 260 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.705328+0000 mgr.y (mgr.24526) 271 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.981653+0000 mon.a (mon.0) 1380 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi083-77943-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:55:49.154 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.981808+0000 mon.a (mon.0) 1381 : audit [INF] from='client.48710 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi083-77648-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: cluster 2024-09-10T18:55:47.982129+0000 mon.a (mon.0) 1382 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.991338+0000 mon.c (mon.2) 261 : audit [INF] from='client.? v1:172.21.15.83:0/262730917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:47.992006+0000 mon.a (mon.0) 1383 : audit [INF] from='client.48716 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:48.022143+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:48.024226+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:48.024626+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:48.627188+0000 mon.c (mon.2) 262 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:55:49.155 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:48 smithi083 bash[32381]: audit 2024-09-10T18:55:48.660053+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:49.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:49.307 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:49.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:49.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:49.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.580 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:49.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:49.717 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:49.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:49.857 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:49.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:49.870 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: cluster 2024-09-10T18:55:48.951186+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:49.870 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:48.989482+0000 mon.a (mon.0) 1388 : audit [INF] from='client.48716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:49.870 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:48.991691+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: cluster 2024-09-10T18:55:48.992005+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:49.003689+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:49.184199+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:49.185469+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: cluster 2024-09-10T18:55:49.187968+0000 mgr.y (mgr.24526) 272 : cluster [DBG] pgmap v272: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: cluster 2024-09-10T18:55:49.188312+0000 mgr.y (mgr.24526) 273 : cluster [DBG] pgmap v273: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:49.491204+0000 mon.a (mon.0) 1392 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:49.871 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[32381]: audit 2024-09-10T18:55:49.661756+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:49.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: cluster 2024-09-10T18:55:48.951186+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:49.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:48.989482+0000 mon.a (mon.0) 1388 : audit [INF] from='client.48716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:49.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:48.991691+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:49.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: cluster 2024-09-10T18:55:48.992005+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:55:49.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:49.003689+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:49.184199+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:49.185469+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: cluster 2024-09-10T18:55:49.187968+0000 mgr.y (mgr.24526) 272 : cluster [DBG] pgmap v272: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: cluster 2024-09-10T18:55:49.188312+0000 mgr.y (mgr.24526) 273 : cluster [DBG] pgmap v273: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:49.491204+0000 mon.a (mon.0) 1392 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:49.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:49 smithi165 bash[22591]: audit 2024-09-10T18:55:49.661756+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:49.998 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:50.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:50.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:50.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:50.275 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:50.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: cluster 2024-09-10T18:55:48.951186+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:48.989482+0000 mon.a (mon.0) 1388 : audit [INF] from='client.48716 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi083-77640-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:48.991691+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77788-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: cluster 2024-09-10T18:55:48.992005+0000 mon.a (mon.0) 1390 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:49.003689+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:49.184199+0000 mon.c (mon.2) 264 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:49.185469+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: cluster 2024-09-10T18:55:49.187968+0000 mgr.y (mgr.24526) 272 : cluster [DBG] pgmap v272: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: cluster 2024-09-10T18:55:49.188312+0000 mgr.y (mgr.24526) 273 : cluster [DBG] pgmap v273: 300 pgs: 40 unknown, 260 active+clean; 592 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:49.491204+0000 mon.a (mon.0) 1392 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:55:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:49 smithi083 bash[18629]: audit 2024-09-10T18:55:49.661756+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:50.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:50.413 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:50.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:50.549 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:50.577 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:55:50.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:50.686 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:50.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:50.825 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:50.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: cluster 2024-09-10T18:55:49.805763+0000 osd.3 (osd.3) 159 : cluster [DBG] 97.5 scrub starts 2024-09-10T18:55:50.914 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: cluster 2024-09-10T18:55:49.811580+0000 osd.3 (osd.3) 160 : cluster [DBG] 97.5 scrub ok 2024-09-10T18:55:50.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:50.965 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:50.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:51.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: cluster 2024-09-10T18:55:49.805763+0000 osd.3 (osd.3) 159 : cluster [DBG] 97.5 scrub starts 2024-09-10T18:55:51.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: cluster 2024-09-10T18:55:49.811580+0000 osd.3 (osd.3) 160 : cluster [DBG] 97.5 scrub ok 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: cluster 2024-09-10T18:55:49.982076+0000 mon.a (mon.0) 1393 : cluster [WRN] Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: cluster 2024-09-10T18:55:50.415740+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: audit 2024-09-10T18:55:50.416970+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.83:0/2327708310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[32381]: audit 2024-09-10T18:55:50.418855+0000 mon.a (mon.0) 1395 : audit [INF] from='client.48728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: cluster 2024-09-10T18:55:49.982076+0000 mon.a (mon.0) 1393 : cluster [WRN] Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: cluster 2024-09-10T18:55:50.415740+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: audit 2024-09-10T18:55:50.416970+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.83:0/2327708310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:50 smithi083 bash[18629]: audit 2024-09-10T18:55:50.418855+0000 mon.a (mon.0) 1395 : audit [INF] from='client.48728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:51.096 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: cluster 2024-09-10T18:55:49.805763+0000 osd.3 (osd.3) 159 : cluster [DBG] 97.5 scrub starts 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: cluster 2024-09-10T18:55:49.811580+0000 osd.3 (osd.3) 160 : cluster [DBG] 97.5 scrub ok 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: cluster 2024-09-10T18:55:49.982076+0000 mon.a (mon.0) 1393 : cluster [WRN] Health check update: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: cluster 2024-09-10T18:55:50.415740+0000 mon.a (mon.0) 1394 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: audit 2024-09-10T18:55:50.416970+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.83:0/2327708310' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:50 smithi165 bash[22591]: audit 2024-09-10T18:55:50.418855+0000 mon.a (mon.0) 1395 : audit [INF] from='client.48728 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:51.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:51.228 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:51.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:51.363 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:51.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:51.497 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:51.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:51.633 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:51.635 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:55:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:51.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:51.770 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:51.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:51.907 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:51.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:52.010 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:52.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.175 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:50.663344+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:51.005260+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:51.006016+0000 mon.a (mon.0) 1397 : audit [INF] from='client.48728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: cluster 2024-09-10T18:55:51.006246+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:51.012739+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3415774298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: cluster 2024-09-10T18:55:51.189101+0000 mgr.y (mgr.24526) 274 : cluster [DBG] pgmap v276: 340 pgs: 40 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 319 B/s wr, 1 op/s 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:51.423304+0000 mon.a (mon.0) 1399 : audit [INF] from='client.48734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[32381]: audit 2024-09-10T18:55:51.664991+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:50.663344+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.176 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:51.005260+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:51.006016+0000 mon.a (mon.0) 1397 : audit [INF] from='client.48728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: cluster 2024-09-10T18:55:51.006246+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:51.012739+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3415774298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: cluster 2024-09-10T18:55:51.189101+0000 mgr.y (mgr.24526) 274 : cluster [DBG] pgmap v276: 340 pgs: 40 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 319 B/s wr, 1 op/s 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:51.423304+0000 mon.a (mon.0) 1399 : audit [INF] from='client.48734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:51 smithi083 bash[18629]: audit 2024-09-10T18:55:51.664991+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.193 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:52.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:50.663344+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:51.005260+0000 mon.a (mon.0) 1396 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77788-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:51.006016+0000 mon.a (mon.0) 1397 : audit [INF] from='client.48728 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi083-77648-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: cluster 2024-09-10T18:55:51.006246+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:51.012739+0000 mon.c (mon.2) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3415774298' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: cluster 2024-09-10T18:55:51.189101+0000 mgr.y (mgr.24526) 274 : cluster [DBG] pgmap v276: 340 pgs: 40 unknown, 32 creating+peering, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 319 B/s wr, 1 op/s 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:51.423304+0000 mon.a (mon.0) 1399 : audit [INF] from='client.48734 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:52.310 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:51 smithi165 bash[22591]: audit 2024-09-10T18:55:51.664991+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:52.329 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:52.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.462 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:52.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:52.597 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:52.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:52.734 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:52.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:55:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:55:52] "GET /metrics HTTP/1.1" 200 238268 "" "Prometheus/2.43.0" 2024-09-10T18:55:52.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:52.867 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:52.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:52.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:53.002 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:53.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: cluster 2024-09-10T18:55:51.753791+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: cluster 2024-09-10T18:55:51.755209+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: audit 2024-09-10T18:55:52.016493+0000 mon.a (mon.0) 1400 : audit [INF] from='client.48734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: cluster 2024-09-10T18:55:52.016860+0000 mon.a (mon.0) 1401 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: audit 2024-09-10T18:55:52.051200+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:53.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[18629]: audit 2024-09-10T18:55:52.666897+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: cluster 2024-09-10T18:55:51.753791+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: cluster 2024-09-10T18:55:51.755209+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: audit 2024-09-10T18:55:52.016493+0000 mon.a (mon.0) 1400 : audit [INF] from='client.48734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: cluster 2024-09-10T18:55:52.016860+0000 mon.a (mon.0) 1401 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: audit 2024-09-10T18:55:52.051200+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:53.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:53 smithi083 bash[32381]: audit 2024-09-10T18:55:52.666897+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:53.144 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:53.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- 172.21.15.83:0/3374376587 >> v1:172.21.15.83:6790/0 conn(0x7feee0100c70 legacy=0x7feee0101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:53.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- 172.21.15.83:0/3374376587 shutdown_connections 2024-09-10T18:55:53.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- 172.21.15.83:0/3374376587 >> 172.21.15.83:0/3374376587 conn(0x7feee0074a20 msgr2=0x7feee00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:53.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- 172.21.15.83:0/3374376587 shutdown_connections 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- 172.21.15.83:0/3374376587 wait complete. 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 librados: starting msgr at 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 librados: starting objecter 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 Processor -- start 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- start start 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 librados: setting wanted keys 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 librados: calling monclient init 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee0109870 con 0x7feee01075b0 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee0109a60 con 0x7feee0100c70 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feee6d25640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee0120790 con 0x7feee0103a50 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedffff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7feee0103a50 0x7feee0108210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58680/0 (socket says 172.21.15.83:58680) 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedffff640 1 -- 172.21.15.83:0/2443463624 learned_addr learned my addr 172.21.15.83:0/2443463624 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:53.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter ms_handle_connect 0x7feee0103a50 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144745439 0 0) 0x7feee0120790 con 0x7feee0103a50 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc003720 con 0x7feee0103a50 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter ms_handle_connect 0x7feee01075b0 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter ms_handle_connect 0x7feee0100c70 2024-09-10T18:55:53.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:53.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573216527 0 0) 0x7feecc003720 con 0x7feee0103a50 2024-09-10T18:55:53.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee0120790 con 0x7feee0103a50 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7feed0001940 con 0x7feee0103a50 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149464106 0 0) 0x7feee0109a60 con 0x7feee0100c70 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.210+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc003720 con 0x7feee0100c70 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2466445614 0 0) 0x7feee0120790 con 0x7feee0103a50 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 >> v1:172.21.15.165:6789/0 conn(0x7feee0100c70 legacy=0x7feee0107b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:53.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 >> v1:172.21.15.83:6789/0 conn(0x7feee01075b0 legacy=0x7feee011d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0121970 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0120980 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feee6d25640 10 client.48752.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee0120e50 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feee6d25640 1 librados: init done 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7feed0001c40 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7feed0005b60 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec0005010 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7feed0002970 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(197..197 src has 69..197) v4 ==== 10495+0+0 (unknown 2731726808 0 0) 0x7feed008b070 con 0x7feee0103a50 2024-09-10T18:55:53.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 10 client.48752.objecter ms_dispatch 0x7feee0000b90 osd_map(197..197 src has 69..197) v4 2024-09-10T18:55:53.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 3 client.48752.objecter handle_osd_map got epochs [197,197] > 0 2024-09-10T18:55:53.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 3 client.48752.objecter handle_osd_map decoding full epoch 197 2024-09-10T18:55:53.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 20 client.48752.objecter dump_active .. 0 homeless 2024-09-10T18:55:53.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.214+0000 7feedd7fa640 10 client.48752.objecter ms_handle_connect 0x7feecc071600 2024-09-10T18:55:53.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.222+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7feed0056c90 con 0x7feee0103a50 2024-09-10T18:55:53.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:53.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: cluster 2024-09-10T18:55:51.753791+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: cluster 2024-09-10T18:55:51.755209+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: audit 2024-09-10T18:55:52.016493+0000 mon.a (mon.0) 1400 : audit [INF] from='client.48734 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi083-77640-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: cluster 2024-09-10T18:55:52.016860+0000 mon.a (mon.0) 1401 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: audit 2024-09-10T18:55:52.051200+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:53.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:53 smithi165 bash[22591]: audit 2024-09-10T18:55:52.666897+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:53.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:53.408 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.406+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7feec00052c0 con 0x7feee0103a50 2024-09-10T18:55:53.512 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:53.537 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:53.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:53.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:53.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:53.809 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:55:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:55:53.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:53.947 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:53.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.962+0000 7feedd7fa640 1 -- 172.21.15.83:0/2443463624 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v198) v1 ==== 113+0+0 (unknown 2039636264 0 0) 0x7feed0056630 con 0x7feee0103a50 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 10 librados: watch_flush enter 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 10 librados: watch_flush exit 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 20 client.48752.objecter shutdown clearing up homeless session... 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 10 client.48752.objecter successfully canceled tick 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 >> v1:172.21.15.83:6800/3071793735 conn(0x7feecc071600 legacy=0x7feecc073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.974+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 >> v1:172.21.15.83:6790/0 conn(0x7feee0103a50 legacy=0x7feee0108210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.982+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 shutdown_connections 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.982+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 >> 172.21.15.83:0/2443463624 conn(0x7feee0074a20 msgr2=0x7feee010a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.986+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 shutdown_connections 2024-09-10T18:55:53.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.986+0000 7feee6d25640 1 -- 172.21.15.83:0/2443463624 wait complete. 2024-09-10T18:55:53.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:53.986+0000 7feee6d25640 1 librados: shutdown 2024-09-10T18:55:54.097 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:54.120 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:55:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:55:54.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:54.237 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.019354+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: cluster 2024-09-10T18:55:53.019565+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.046040+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.050002+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: cluster 2024-09-10T18:55:53.190030+0000 mgr.y (mgr.24526) 275 : cluster [DBG] pgmap v279: 340 pgs: 72 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.412879+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.83:0/2443463624' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.495574+0000 mon.a (mon.0) 1407 : audit [INF] from='client.48752 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.668693+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: cluster 2024-09-10T18:55:53.952242+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.967506+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.967633+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.967855+0000 mon.a (mon.0) 1411 : audit [INF] from='client.48752 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: cluster 2024-09-10T18:55:53.967922+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:55:54.238 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:54 smithi165 bash[22591]: audit 2024-09-10T18:55:53.977161+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.258 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.019354+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: cluster 2024-09-10T18:55:53.019565+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.046040+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.050002+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: cluster 2024-09-10T18:55:53.190030+0000 mgr.y (mgr.24526) 275 : cluster [DBG] pgmap v279: 340 pgs: 72 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.412879+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.83:0/2443463624' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.495574+0000 mon.a (mon.0) 1407 : audit [INF] from='client.48752 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.668693+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: cluster 2024-09-10T18:55:53.952242+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.967506+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.967633+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.967855+0000 mon.a (mon.0) 1411 : audit [INF] from='client.48752 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: cluster 2024-09-10T18:55:53.967922+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[18629]: audit 2024-09-10T18:55:53.977161+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.019354+0000 mon.a (mon.0) 1403 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: cluster 2024-09-10T18:55:53.019565+0000 mon.a (mon.0) 1404 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.046040+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.050002+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: cluster 2024-09-10T18:55:53.190030+0000 mgr.y (mgr.24526) 275 : cluster [DBG] pgmap v279: 340 pgs: 72 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.412879+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.83:0/2443463624' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.495574+0000 mon.a (mon.0) 1407 : audit [INF] from='client.48752 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.668693+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: cluster 2024-09-10T18:55:53.952242+0000 mon.a (mon.0) 1408 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.967506+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? v1:172.21.15.83:0/4101386107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi083-77648-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.967633+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77788-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.967855+0000 mon.a (mon.0) 1411 : audit [INF] from='client.48752 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: cluster 2024-09-10T18:55:53.967922+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-09-10T18:55:54.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:54 smithi083 bash[32381]: audit 2024-09-10T18:55:53.977161+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:54.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:54.397 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:54.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:54.534 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:54.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:54.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:54.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:54.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:54.909 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:54.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:54.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:55.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:55.078 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:55.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:55.216 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:55.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:55.346 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:55.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:55.471 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:55.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[18629]: audit 2024-09-10T18:55:54.670472+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[18629]: audit 2024-09-10T18:55:54.967498+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[18629]: cluster 2024-09-10T18:55:54.968031+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[32381]: audit 2024-09-10T18:55:54.670472+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[32381]: audit 2024-09-10T18:55:54.967498+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:55.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:55 smithi083 bash[32381]: cluster 2024-09-10T18:55:54.968031+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:55:55.607 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:55.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:55.741 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:55.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:55.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:55 smithi165 bash[22591]: audit 2024-09-10T18:55:54.670472+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:55.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:55 smithi165 bash[22591]: audit 2024-09-10T18:55:54.967498+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? v1:172.21.15.83:0/469892016' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi083-77640-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:55.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:55 smithi165 bash[22591]: cluster 2024-09-10T18:55:54.968031+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-09-10T18:55:55.877 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:55.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:55.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:56.022 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:56.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:56.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:56.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.431 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:56.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:56.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T18:55:56.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: cluster 2024-09-10T18:55:54.417587+0000 osd.0 (osd.0) 148 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:56.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: cluster 2024-09-10T18:55:54.418668+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:56.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: cluster 2024-09-10T18:55:55.190985+0000 mgr.y (mgr.24526) 276 : cluster [DBG] pgmap v282: 308 pgs: 40 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:55:56.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: audit 2024-09-10T18:55:55.672236+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:56.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: cluster 2024-09-10T18:55:55.975189+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:55:56.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: audit 2024-09-10T18:55:55.976845+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.83:0/1858205069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: audit 2024-09-10T18:55:55.981847+0000 mon.a (mon.0) 1417 : audit [INF] from='client.48755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.686 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:56 smithi165 bash[22591]: audit 2024-09-10T18:55:55.998103+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:56.703 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:56.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: cluster 2024-09-10T18:55:54.417587+0000 osd.0 (osd.0) 148 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:56.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: cluster 2024-09-10T18:55:54.418668+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: cluster 2024-09-10T18:55:55.190985+0000 mgr.y (mgr.24526) 276 : cluster [DBG] pgmap v282: 308 pgs: 40 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: audit 2024-09-10T18:55:55.672236+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: cluster 2024-09-10T18:55:55.975189+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: audit 2024-09-10T18:55:55.976845+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.83:0/1858205069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: audit 2024-09-10T18:55:55.981847+0000 mon.a (mon.0) 1417 : audit [INF] from='client.48755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[32381]: audit 2024-09-10T18:55:55.998103+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: cluster 2024-09-10T18:55:54.417587+0000 osd.0 (osd.0) 148 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: cluster 2024-09-10T18:55:54.418668+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: cluster 2024-09-10T18:55:55.190985+0000 mgr.y (mgr.24526) 276 : cluster [DBG] pgmap v282: 308 pgs: 40 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:55:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: audit 2024-09-10T18:55:55.672236+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: cluster 2024-09-10T18:55:55.975189+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-09-10T18:55:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: audit 2024-09-10T18:55:55.976845+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.83:0/1858205069' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: audit 2024-09-10T18:55:55.981847+0000 mon.a (mon.0) 1417 : audit [INF] from='client.48755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:56 smithi083 bash[18629]: audit 2024-09-10T18:55:55.998103+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:56.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.839 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:56.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.980 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:56.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:57.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:57.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:57.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:57.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.401 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:57.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:57.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.674087+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:57.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.982704+0000 mon.a (mon.0) 1419 : audit [INF] from='client.48755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:57.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.983348+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:57.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: cluster 2024-09-10T18:55:56.983928+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:55:57.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.985683+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.83:0/624361872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.986320+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:57.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: audit 2024-09-10T18:55:56.987675+0000 mon.a (mon.0) 1423 : audit [INF] from='client.48761 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[22591]: cluster 2024-09-10T18:55:57.192004+0000 mgr.y (mgr.24526) 277 : cluster [DBG] pgmap v285: 332 pgs: 1 active+clean+snaptrim, 64 unknown, 267 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:57.539 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:57.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:57.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:57.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:57.790 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:55:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:55:57.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.674087+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.982704+0000 mon.a (mon.0) 1419 : audit [INF] from='client.48755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.983348+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: cluster 2024-09-10T18:55:56.983928+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.985683+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.83:0/624361872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.986320+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: audit 2024-09-10T18:55:56.987675+0000 mon.a (mon.0) 1423 : audit [INF] from='client.48761 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[18629]: cluster 2024-09-10T18:55:57.192004+0000 mgr.y (mgr.24526) 277 : cluster [DBG] pgmap v285: 332 pgs: 1 active+clean+snaptrim, 64 unknown, 267 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.674087+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.982704+0000 mon.a (mon.0) 1419 : audit [INF] from='client.48755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi083-77648-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.983348+0000 mon.a (mon.0) 1420 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: cluster 2024-09-10T18:55:56.983928+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-09-10T18:55:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.985683+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.83:0/624361872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.986320+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]: dispatch 2024-09-10T18:55:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: audit 2024-09-10T18:55:56.987675+0000 mon.a (mon.0) 1423 : audit [INF] from='client.48761 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:55:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:57 smithi083 bash[32381]: cluster 2024-09-10T18:55:57.192004+0000 mgr.y (mgr.24526) 277 : cluster [DBG] pgmap v285: 332 pgs: 1 active+clean+snaptrim, 64 unknown, 267 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T18:55:57.810 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:57.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:55:57.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:57.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:55:58.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.084 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:58.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:58.225 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:55:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:55:58.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.363 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:58.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:58.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:58.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:58.634 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:58.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:58.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.764 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:58.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:58.903 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:58.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:58.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- 172.21.15.83:0/796127301 >> v1:172.21.15.83:6790/0 conn(0x7ff3201014e0 legacy=0x7ff3201018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:58.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- 172.21.15.83:0/796127301 shutdown_connections 2024-09-10T18:55:58.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- 172.21.15.83:0/796127301 >> 172.21.15.83:0/796127301 conn(0x7ff320075290 msgr2=0x7ff3200f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:58.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- 172.21.15.83:0/796127301 shutdown_connections 2024-09-10T18:55:58.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- 172.21.15.83:0/796127301 wait complete. 2024-09-10T18:55:58.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 librados: starting msgr at 2024-09-10T18:55:58.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 librados: starting objecter 2024-09-10T18:55:58.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 Processor -- start 2024-09-10T18:55:58.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- start start 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 librados: setting wanted keys 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 librados: calling monclient init 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff32011bfc0 con 0x7ff3201014e0 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff32011d180 con 0x7ff320107e30 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff324b0e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff32011e380 con 0x7ff3201042a0 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff31dd9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff3201042a0 0x7ff3200fd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58790/0 (socket says 172.21.15.83:58790) 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff31dd9b640 1 -- 172.21.15.83:0/958394822 learned_addr learned my addr 172.21.15.83:0/958394822 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.978+0000 7ff31e59c640 1 --1- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6789/0 conn(0x7ff3201014e0 0x7ff3200fd050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34518/0 (socket says 172.21.15.83:34518) 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter ms_handle_connect 0x7ff3201042a0 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter ms_handle_connect 0x7ff3201014e0 2024-09-10T18:55:58.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:58.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter ms_handle_connect 0x7ff320107e30 2024-09-10T18:55:58.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.982+0000 7ff316ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:55:58.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.986+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018922025 0 0) 0x7ff32011bfc0 con 0x7ff3201014e0 2024-09-10T18:55:58.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.986+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30c003720 con 0x7ff3201014e0 2024-09-10T18:55:58.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.986+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924785768 0 0) 0x7ff32011e380 con 0x7ff3201042a0 2024-09-10T18:55:58.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.986+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff32011bfc0 con 0x7ff3201042a0 2024-09-10T18:55:58.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.986+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1707301049 0 0) 0x7ff32011bfc0 con 0x7ff3201042a0 2024-09-10T18:55:58.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff32011e380 con 0x7ff3201042a0 2024-09-10T18:55:58.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff318001cc0 con 0x7ff3201042a0 2024-09-10T18:55:58.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3702284810 0 0) 0x7ff32011e380 con 0x7ff3201042a0 2024-09-10T18:55:58.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.165:6789/0 conn(0x7ff320107e30 legacy=0x7ff32011a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:58.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6789/0 conn(0x7ff3201014e0 legacy=0x7ff3200fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:58.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32011f580 con 0x7ff3201042a0 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff324b0e640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32011c1b0 con 0x7ff3201042a0 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff324b0e640 10 client.48776.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff324b0e640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32011c680 con 0x7ff3201042a0 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff324b0e640 1 librados: init done 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff3180023c0 con 0x7ff3201042a0 2024-09-10T18:55:58.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.990+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff318005dd0 con 0x7ff3201042a0 2024-09-10T18:55:58.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff318001f60 con 0x7ff3201042a0 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(203..203 src has 69..203) v4 ==== 9013+0+0 (unknown 265179285 0 0) 0x7ff31808ac60 con 0x7ff3201042a0 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 osd_map(203..203 src has 69..203) v4 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 3 client.48776.objecter handle_osd_map got epochs [203,203] > 0 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 3 client.48776.objecter handle_osd_map decoding full epoch 203 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 20 client.48776.objecter dump_active .. 0 homeless 2024-09-10T18:55:59.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.994+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff30c0716b0 2024-09-10T18:55:59.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff324b0e640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff300004ff0 con 0x7ff3201042a0 2024-09-10T18:55:59.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v203) v1 ==== 54+0+16 (unknown 3851782266 0 2420000192) 0x7ff318056e30 con 0x7ff3201042a0 2024-09-10T18:55:59.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 10 client.48776.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 1 2024-09-10T18:55:59.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 1 2024-09-10T18:55:59.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 0 1 2024-09-10T18:55:59.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 4 2024-09-10T18:55:59.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 10 client.48776.objecter _send_command 1 2024-09-10T18:55:59.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:58.998+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff304006cc0 con 0x7ff304002f60 2024-09-10T18:55:59.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:59.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:59.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: audit 2024-09-10T18:55:57.675704+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: audit 2024-09-10T18:55:57.987185+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: audit 2024-09-10T18:55:57.987288+0000 mon.a (mon.0) 1425 : audit [INF] from='client.48761 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: cluster 2024-09-10T18:55:57.987602+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: audit 2024-09-10T18:55:57.992867+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[18629]: audit 2024-09-10T18:55:57.993403+0000 mon.a (mon.0) 1427 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: audit 2024-09-10T18:55:57.675704+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:59.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: audit 2024-09-10T18:55:57.987185+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:59.112 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: audit 2024-09-10T18:55:57.987288+0000 mon.a (mon.0) 1425 : audit [INF] from='client.48761 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:59.112 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: cluster 2024-09-10T18:55:57.987602+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:55:59.112 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: audit 2024-09-10T18:55:57.992867+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.112 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:58 smithi083 bash[32381]: audit 2024-09-10T18:55:57.993403+0000 mon.a (mon.0) 1427 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.114+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff304002f60 2024-09-10T18:55:59.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.118+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff304006cc0 con 0x7ff304002f60 2024-09-10T18:55:59.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.118+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:55:59.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.118+0000 7ff316ffd640 10 client.48776.objecter _finish_command 1 = system:0 2024-09-10T18:55:59.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.118+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 4 2024-09-10T18:55:59.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.118+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 0 1 2024-09-10T18:55:59.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 10 client.48776.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 2 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 2 2024-09-10T18:55:59.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304002870 osd=0 3 2024-09-10T18:55:59.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 0 2 2024-09-10T18:55:59.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 4 2024-09-10T18:55:59.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 10 client.48776.objecter _send_command 2 2024-09-10T18:55:59.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff3040011e0 con 0x7ff304002f60 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff3040011e0 con 0x7ff304002f60 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 10 client.48776.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 4 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 0 2 2024-09-10T18:55:59.145 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 10 client.48776.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 3 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 3 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 1 3 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 4 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 10 client.48776.objecter _send_command 3 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3040079f0 con 0x7ff304006e60 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.138+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff304006e60 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.142+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff3040079f0 con 0x7ff304006e60 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.142+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.142+0000 7ff316ffd640 10 client.48776.objecter _finish_command 3 = system:0 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.142+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 4 2024-09-10T18:55:59.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.142+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 1 3 2024-09-10T18:55:59.157 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:59.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 10 client.48776.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 4 2024-09-10T18:55:59.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 4 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304002610 osd=1 3 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 1 4 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 4 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 10 client.48776.objecter _send_command 4 2024-09-10T18:55:59.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.154+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff304008350 con 0x7ff304006e60 2024-09-10T18:55:59.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff304008350 con 0x7ff304006e60 2024-09-10T18:55:59.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 10 client.48776.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 4 2024-09-10T18:55:59.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 1 4 2024-09-10T18:55:59.164 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:55:59.164 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:55:59.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 10 client.48776.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 5 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 5 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 2 5 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 4 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 10 client.48776.objecter _send_command 5 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff30400e870 con 0x7ff30400cc80 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff30400cc80 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff30400e870 con 0x7ff30400cc80 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 10 client.48776.objecter _finish_command 5 = system:0 2024-09-10T18:55:59.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 4 2024-09-10T18:55:59.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.158+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 2 5 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 10 client.48776.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 6 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 6 2024-09-10T18:55:59.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304008cd0 osd=2 3 2024-09-10T18:55:59.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 2 6 2024-09-10T18:55:59.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 4 2024-09-10T18:55:59.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 10 client.48776.objecter _send_command 6 2024-09-10T18:55:59.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.170+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff30400ed60 con 0x7ff30400cc80 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff30400ed60 con 0x7ff30400cc80 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 10 client.48776.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 4 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 2 6 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:55:59.179 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 10 client.48776.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 7 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 7 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 3 7 2024-09-10T18:55:59.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 4 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 10 client.48776.objecter _send_command 7 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff304002280 con 0x7ff304007f40 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff304007f40 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff304002280 con 0x7ff304007f40 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 10 client.48776.objecter _finish_command 7 = system:0 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 4 2024-09-10T18:55:59.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.174+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 3 7 2024-09-10T18:55:59.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 10 client.48776.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 8 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 8 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304007dd0 osd=3 3 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 3 8 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 4 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 10 client.48776.objecter _send_command 8 2024-09-10T18:55:59.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff304001640 con 0x7ff304007f40 2024-09-10T18:55:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff304001640 con 0x7ff304007f40 2024-09-10T18:55:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 10 client.48776.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 4 2024-09-10T18:55:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 3 8 2024-09-10T18:55:59.197 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:55:59.197 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 10 client.48776.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 9 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 9 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 4 9 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 4 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 10 client.48776.objecter _send_command 9 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3040161e0 con 0x7ff3040017b0 2024-09-10T18:55:59.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.190+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff3040017b0 2024-09-10T18:55:59.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.194+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff3040161e0 con 0x7ff3040017b0 2024-09-10T18:55:59.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.194+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:55:59.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.194+0000 7ff316ffd640 10 client.48776.objecter _finish_command 9 = system:0 2024-09-10T18:55:59.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.194+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 4 2024-09-10T18:55:59.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.194+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 4 9 2024-09-10T18:55:59.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 10 client.48776.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 10 2024-09-10T18:55:59.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 10 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304001640 osd=4 3 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 4 10 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 4 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 10 client.48776.objecter _send_command 10 2024-09-10T18:55:59.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.210+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff3040161e0 con 0x7ff3040017b0 2024-09-10T18:55:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff3040161e0 con 0x7ff3040017b0 2024-09-10T18:55:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff316ffd640 10 client.48776.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 4 2024-09-10T18:55:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 4 10 2024-09-10T18:55:59.222 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:55:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:55:59.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 10 client.48776.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 11 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 11 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 5 11 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 4 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 10 client.48776.objecter _send_command 11 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.214+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff304022680 con 0x7ff30401ea50 2024-09-10T18:55:59.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff30401ea50 2024-09-10T18:55:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff304022680 con 0x7ff30401ea50 2024-09-10T18:55:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:55:59.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 10 client.48776.objecter _finish_command 11 = system:0 2024-09-10T18:55:59.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 4 2024-09-10T18:55:59.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.218+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 5 11 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 10 client.48776.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 12 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 12 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff3040161e0 osd=5 3 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 5 12 2024-09-10T18:55:59.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 4 2024-09-10T18:55:59.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 10 client.48776.objecter _send_command 12 2024-09-10T18:55:59.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.238+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff304022640 con 0x7ff30401ea50 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff304022640 con 0x7ff30401ea50 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 10 client.48776.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 4 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 5 12 2024-09-10T18:55:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 10 client.48776.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 13 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 13 2024-09-10T18:55:59.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 6 13 2024-09-10T18:55:59.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 4 2024-09-10T18:55:59.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 10 client.48776.objecter _send_command 13 2024-09-10T18:55:59.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff304023990 con 0x7ff304022cf0 2024-09-10T18:55:59.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.242+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff304022cf0 2024-09-10T18:55:59.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.246+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff304023990 con 0x7ff304022cf0 2024-09-10T18:55:59.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.246+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:55:59.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.246+0000 7ff316ffd640 10 client.48776.objecter _finish_command 13 = system:0 2024-09-10T18:55:59.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.246+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 4 2024-09-10T18:55:59.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.246+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 6 13 2024-09-10T18:55:59.259 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 10 client.48776.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 14 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 14 2024-09-10T18:55:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304022640 osd=6 3 2024-09-10T18:55:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 6 14 2024-09-10T18:55:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 4 2024-09-10T18:55:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 10 client.48776.objecter _send_command 14 2024-09-10T18:55:59.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.258+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff304023990 con 0x7ff304022cf0 2024-09-10T18:55:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff304023990 con 0x7ff304022cf0 2024-09-10T18:55:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff316ffd640 10 client.48776.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 4 2024-09-10T18:55:59.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 6 14 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 10 client.48776.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 15 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 15 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 7 15 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 4 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 10 client.48776.objecter _send_command 15 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.262+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff30402af50 con 0x7ff304027450 2024-09-10T18:55:59.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 10 client.48776.objecter ms_handle_connect 0x7ff304027450 2024-09-10T18:55:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff30402af50 con 0x7ff304027450 2024-09-10T18:55:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:55:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 10 client.48776.objecter _finish_command 15 = system:0 2024-09-10T18:55:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 4 2024-09-10T18:55:59.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.266+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 7 15 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 10 client.48776.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign -1 16 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 20 client.48776.objecter _get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_remove -1 16 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 20 client.48776.objecter get_session s=0x7ff304023bd0 osd=7 3 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 15 client.48776.objecter _session_command_op_assign 7 16 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 4 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 10 client.48776.objecter _send_command 16 2024-09-10T18:55:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.282+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff30402af50 con 0x7ff304027450 2024-09-10T18:55:59.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:55:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:55:59.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff316ffd640 1 -- 172.21.15.83:0/958394822 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff30402af50 con 0x7ff304027450 2024-09-10T18:55:59.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff316ffd640 10 client.48776.objecter ms_dispatch 0x7ff320000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:55:59.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff316ffd640 10 client.48776.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:55:59.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff316ffd640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 4 2024-09-10T18:55:59.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff316ffd640 15 client.48776.objecter _session_command_op_remove 7 16 2024-09-10T18:55:59.292 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 librados: watch_flush enter 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 librados: watch_flush exit 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.0 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6801/2535527483 conn(0x7ff304002f60 legacy=0x7ff304005410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002870 osd=0 1 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.1 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6805/2309624637 conn(0x7ff304006e60 legacy=0x7ff3040072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304002610 osd=1 1 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.2 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6809/386094220 conn(0x7ff30400cc80 legacy=0x7ff30400d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff31ed9d640 1 -- 172.21.15.83:0/958394822 reap_dead start 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304008cd0 osd=2 1 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.3 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6813/1279688021 conn(0x7ff304007f40 legacy=0x7ff304018570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304007dd0 osd=3 1 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.4 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.165:6800/3225515265 conn(0x7ff3040017b0 legacy=0x7ff30401ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304001640 osd=4 1 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.5 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.165:6804/571173777 conn(0x7ff30401ea50 legacy=0x7ff304020e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff3040161e0 osd=5 1 2024-09-10T18:55:59.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.6 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.165:6808/1317922936 conn(0x7ff304022cf0 legacy=0x7ff304023120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304022640 osd=6 1 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter close_session for osd.7 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff304027450 legacy=0x7ff304029830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter put_session s=0x7ff304023bd0 osd=7 1 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 20 client.48776.objecter shutdown clearing up homeless session... 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 10 client.48776.objecter successfully canceled tick 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff31ed9d640 1 -- 172.21.15.83:0/958394822 reap_dead start 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff30c0716b0 legacy=0x7ff30c073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> v1:172.21.15.83:6790/0 conn(0x7ff3201042a0 legacy=0x7ff3200fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 shutdown_connections 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 >> 172.21.15.83:0/958394822 conn(0x7ff320075290 msgr2=0x7ff320064740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 shutdown_connections 2024-09-10T18:55:59.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 -- 172.21.15.83:0/958394822 wait complete. 2024-09-10T18:55:59.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:55:59.286+0000 7ff314ff9640 1 librados: shutdown 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: audit 2024-09-10T18:55:57.675704+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: audit 2024-09-10T18:55:57.987185+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? v1:172.21.15.83:0/4260234155' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77788-24"}]': finished 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: audit 2024-09-10T18:55:57.987288+0000 mon.a (mon.0) 1425 : audit [INF] from='client.48761 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi083-77640-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: cluster 2024-09-10T18:55:57.987602+0000 mon.a (mon.0) 1426 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: audit 2024-09-10T18:55:57.992867+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:58 smithi165 bash[22591]: audit 2024-09-10T18:55:57.993403+0000 mon.a (mon.0) 1427 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:55:59.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.416 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:59.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.552 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:55:59.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:59.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:55:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:55:59.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:59.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:55:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:55:59.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.961 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:55:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:55:59.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots_pp: Running main() from gmock_main.cc 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: seed 77943 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (3545 ms) 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-10T18:56:00.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3188 ms) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2645 ms) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3309 ms) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4248 ms) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (16935 ms total) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5212 ms) 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-10T18:56:00.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4421 ms) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7010 ms) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3986 ms) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2055 ms) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi083-77943-7 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17959 ms) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (40643 ms total) 2024-09-10T18:56:00.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4998 ms) 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5007 ms total) 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3013 ms) 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-10T18:56:00.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2016 ms) 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1948 ms) 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2008 ms) 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8985 ms total) 2024-09-10T18:56:00.023 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-09-10T18:56:00.064 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:00.064 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:56:00.065 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:56:00.087 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:00.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:57.714638+0000 mgr.y (mgr.24526) 278 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:00.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.677541+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: cluster 2024-09-10T18:55:58.972525+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:00.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.990020+0000 mon.a (mon.0) 1429 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: cluster 2024-09-10T18:55:58.990368+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.993583+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.994056+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.83:0/3286556077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.994899+0000 mon.a (mon.0) 1431 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.995241+0000 mon.a (mon.0) 1432 : audit [INF] from='client.48767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:58.996831+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.83:0/1113318450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:59.001597+0000 mon.a (mon.0) 1433 : audit [INF] from='client.49294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:59.002737+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.83:0/958394822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: cluster 2024-09-10T18:55:59.192817+0000 mgr.y (mgr.24526) 279 : cluster [DBG] pgmap v288: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:59.557568+0000 mon.c (mon.2) 285 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:00.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:55:59 smithi165 bash[22591]: audit 2024-09-10T18:55:59.678984+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:00.216 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:57.714638+0000 mgr.y (mgr.24526) 278 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.677541+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: cluster 2024-09-10T18:55:58.972525+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.990020+0000 mon.a (mon.0) 1429 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: cluster 2024-09-10T18:55:58.990368+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.993583+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.994056+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.83:0/3286556077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.994899+0000 mon.a (mon.0) 1431 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.995241+0000 mon.a (mon.0) 1432 : audit [INF] from='client.48767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:58.996831+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.83:0/1113318450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:59.001597+0000 mon.a (mon.0) 1433 : audit [INF] from='client.49294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:59.002737+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.83:0/958394822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: cluster 2024-09-10T18:55:59.192817+0000 mgr.y (mgr.24526) 279 : cluster [DBG] pgmap v288: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:59.557568+0000 mon.c (mon.2) 285 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[18629]: audit 2024-09-10T18:55:59.678984+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:57.714638+0000 mgr.y (mgr.24526) 278 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.677541+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: cluster 2024-09-10T18:55:58.972525+0000 mon.a (mon.0) 1428 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.990020+0000 mon.a (mon.0) 1429 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: cluster 2024-09-10T18:55:58.990368+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.993583+0000 mon.c (mon.2) 282 : audit [INF] from='client.? v1:172.21.15.83:0/2936751915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.994056+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.83:0/3286556077' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.994899+0000 mon.a (mon.0) 1431 : audit [INF] from='client.48707 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.995241+0000 mon.a (mon.0) 1432 : audit [INF] from='client.48767 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:58.996831+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.83:0/1113318450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:59.001597+0000 mon.a (mon.0) 1433 : audit [INF] from='client.49294 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:59.002737+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.83:0/958394822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: cluster 2024-09-10T18:55:59.192817+0000 mgr.y (mgr.24526) 279 : cluster [DBG] pgmap v288: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:59.557568+0000 mon.c (mon.2) 285 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:00.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:55:59 smithi083 bash[32381]: audit 2024-09-10T18:55:59.678984+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:00.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:00.357 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:00.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:00.494 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:00.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:00.632 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:00.736 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:00.771 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:00.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:00.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:00.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:01.010 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:01.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:01.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.008407+0000 mon.a (mon.0) 1434 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.008551+0000 mon.a (mon.0) 1435 : audit [INF] from='client.48767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.008820+0000 mon.a (mon.0) 1436 : audit [INF] from='client.49294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: cluster 2024-09-10T18:56:00.008976+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.019382+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.065324+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.067197+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.067583+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:01.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[18629]: audit 2024-09-10T18:56:00.680537+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.008407+0000 mon.a (mon.0) 1434 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.008551+0000 mon.a (mon.0) 1435 : audit [INF] from='client.48767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.008820+0000 mon.a (mon.0) 1436 : audit [INF] from='client.49294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: cluster 2024-09-10T18:56:00.008976+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.019382+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.065324+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.067197+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.067583+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:01.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:01 smithi083 bash[32381]: audit 2024-09-10T18:56:00.680537+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:01.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:01.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.008407+0000 mon.a (mon.0) 1434 : audit [INF] from='client.48707 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi083-77943-15"}]': finished 2024-09-10T18:56:01.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.008551+0000 mon.a (mon.0) 1435 : audit [INF] from='client.48767 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi083-77648-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.008820+0000 mon.a (mon.0) 1436 : audit [INF] from='client.49294 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi083-77788-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: cluster 2024-09-10T18:56:00.008976+0000 mon.a (mon.0) 1437 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.019382+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.065324+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.067197+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.067583+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:01.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:00 smithi165 bash[22591]: audit 2024-09-10T18:56:00.680537+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:01.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:01.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:01.480 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:01.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:01.614 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:01.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:01.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:01.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:01.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:01.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:01.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:01.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:01.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.025 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:02.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:02.159 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:02.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:02.298 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:02.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T18:56:02.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:02.429 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:02.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 -- 172.21.15.83:0/2668278020 >> v1:172.21.15.83:6789/0 conn(0x7f5e54101540 legacy=0x7f5e54101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 -- 172.21.15.83:0/2668278020 shutdown_connections 2024-09-10T18:56:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 -- 172.21.15.83:0/2668278020 >> 172.21.15.83:0/2668278020 conn(0x7f5e54074a20 msgr2=0x7f5e540f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 -- 172.21.15.83:0/2668278020 shutdown_connections 2024-09-10T18:56:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 -- 172.21.15.83:0/2668278020 wait complete. 2024-09-10T18:56:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 librados: starting msgr at 2024-09-10T18:56:02.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.570+0000 7f5e5c264640 1 librados: starting objecter 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 Processor -- start 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- start start 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 librados: setting wanted keys 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 librados: calling monclient init 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e5411c160 con 0x7f5e541042b0 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e5411d320 con 0x7f5e54107e40 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e5411e520 con 0x7f5e54101540 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5a000640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5e54101540 0x7f5e540fd240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58884/0 (socket says 172.21.15.83:58884) 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5a000640 1 -- 172.21.15.83:0/3069936974 learned_addr learned my addr 172.21.15.83:0/3069936974 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e597ff640 1 --1- 172.21.15.83:0/3069936974 >> v1:172.21.15.83:6789/0 conn(0x7f5e541042b0 0x7f5e540fd950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34630/0 (socket says 172.21.15.83:34630) 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter ms_handle_connect 0x7f5e541042b0 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:02.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter ms_handle_connect 0x7f5e54101540 2024-09-10T18:56:02.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707934055 0 0) 0x7f5e5411c160 con 0x7f5e541042b0 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e40003720 con 0x7f5e541042b0 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter ms_handle_connect 0x7f5e54107e40 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063841500 0 0) 0x7f5e5411e520 con 0x7f5e54101540 2024-09-10T18:56:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5411c160 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330117834 0 0) 0x7f5e5411d320 con 0x7f5e54107e40 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5411e520 con 0x7f5e54107e40 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396783448 0 0) 0x7f5e40003720 con 0x7f5e541042b0 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e5411d320 con 0x7f5e541042b0 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1896110744 0 0) 0x7f5e5411c160 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e40003720 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e50001c60 con 0x7f5e541042b0 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e48002bb0 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 31774993 0 0) 0x7f5e40003720 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 >> v1:172.21.15.165:6789/0 conn(0x7f5e54107e40 legacy=0x7f5e5411a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 >> v1:172.21.15.83:6789/0 conn(0x7f5e541042b0 legacy=0x7f5e540fd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5411f720 con 0x7f5e54101540 2024-09-10T18:56:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5411e710 con 0x7f5e54101540 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 10 client.48812.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5411ebe0 con 0x7f5e54101540 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 librados: init done 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5e480024b0 con 0x7f5e54101540 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e48003700 con 0x7f5e54101540 2024-09-10T18:56:02.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.574+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e540fccb0 con 0x7f5e54101540 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5e48002050 con 0x7f5e54101540 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(206..206 src has 69..206) v4 ==== 9298+0+0 (unknown 1421204519 0 0) 0x7f5e4808ae20 con 0x7f5e54101540 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 10 client.48812.objecter ms_dispatch 0x7f5e54000b90 osd_map(206..206 src has 69..206) v4 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 3 client.48812.objecter handle_osd_map got epochs [206,206] > 0 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 3 client.48812.objecter handle_osd_map decoding full epoch 206 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 20 client.48812.objecter dump_active .. 0 homeless 2024-09-10T18:56:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.578+0000 7f5e467fc640 10 client.48812.objecter ms_handle_connect 0x7f5e40071480 2024-09-10T18:56:02.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.586+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5e48056ef0 con 0x7f5e54101540 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: audit 2024-09-10T18:56:01.015017+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: audit 2024-09-10T18:56:01.015384+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: cluster 2024-09-10T18:56:01.015501+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: audit 2024-09-10T18:56:01.026637+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: cluster 2024-09-10T18:56:01.193416+0000 mgr.y (mgr.24526) 280 : cluster [DBG] pgmap v291: 292 pgs: 32 creating+peering, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:02.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:02 smithi165 bash[22591]: audit 2024-09-10T18:56:01.682273+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:02.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: audit 2024-09-10T18:56:01.015017+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: audit 2024-09-10T18:56:01.015384+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: cluster 2024-09-10T18:56:01.015501+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: audit 2024-09-10T18:56:01.026637+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: cluster 2024-09-10T18:56:01.193416+0000 mgr.y (mgr.24526) 280 : cluster [DBG] pgmap v291: 292 pgs: 32 creating+peering, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[32381]: audit 2024-09-10T18:56:01.682273+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: audit 2024-09-10T18:56:01.015017+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.83:0/863234074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi083-77640-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:02.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: audit 2024-09-10T18:56:01.015384+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:02.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: cluster 2024-09-10T18:56:01.015501+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-09-10T18:56:02.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: audit 2024-09-10T18:56:01.026637+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:02.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: cluster 2024-09-10T18:56:01.193416+0000 mgr.y (mgr.24526) 280 : cluster [DBG] pgmap v291: 292 pgs: 32 creating+peering, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:02.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18629]: audit 2024-09-10T18:56:01.682273+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:02.693 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:02.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:02.766+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5e541075d0 con 0x7f5e54101540 2024-09-10T18:56:02.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:02] "GET /metrics HTTP/1.1" 200 235852 "" "Prometheus/2.43.0" 2024-09-10T18:56:02.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.828 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:02.832 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:02.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:02.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:02.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:03.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.026+0000 7f5e467fc640 1 -- 172.21.15.83:0/3069936974 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v207) v1 ==== 91+0+0 (unknown 335838478 0 0) 0x7f5e48056890 con 0x7f5e54101540 2024-09-10T18:56:03.030 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T18:56:03.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 10 librados: watch_flush enter 2024-09-10T18:56:03.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 10 librados: watch_flush exit 2024-09-10T18:56:03.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 20 client.48812.objecter shutdown clearing up homeless session... 2024-09-10T18:56:03.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 10 client.48812.objecter successfully canceled tick 2024-09-10T18:56:03.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5e40071480 legacy=0x7f5e40073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 >> v1:172.21.15.83:6790/0 conn(0x7f5e54101540 legacy=0x7f5e540fd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 shutdown_connections 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 >> 172.21.15.83:0/3069936974 conn(0x7f5e54074a20 msgr2=0x7f5e54072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 shutdown_connections 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 -- 172.21.15.83:0/3069936974 wait complete. 2024-09-10T18:56:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:03.034+0000 7f5e5c264640 1 librados: shutdown 2024-09-10T18:56:03.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:03.102 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:03.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:03.238 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:03.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:03.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:03.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:03.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:03.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.459828+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.83:0/3832577762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:02.460916+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.462011+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.83:0/1539942396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.463675+0000 mon.a (mon.0) 1447 : audit [INF] from='client.48794 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.474437+0000 mon.a (mon.0) 1448 : audit [INF] from='client.49315 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.489041+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.489455+0000 mon.a (mon.0) 1449 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.490952+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.491310+0000 mon.a (mon.0) 1450 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.491778+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.492122+0000 mon.a (mon.0) 1451 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.684003+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.768279+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3069936974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:02.768883+0000 mon.a (mon.0) 1452 : audit [INF] from='client.48812 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.015553+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.459828+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.83:0/3832577762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:02.460916+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.462011+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.83:0/1539942396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.463675+0000 mon.a (mon.0) 1447 : audit [INF] from='client.48794 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.474437+0000 mon.a (mon.0) 1448 : audit [INF] from='client.49315 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.489041+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.489455+0000 mon.a (mon.0) 1449 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.490952+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.491310+0000 mon.a (mon.0) 1450 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.491778+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.492122+0000 mon.a (mon.0) 1451 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.684003+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.768279+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3069936974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:02.768883+0000 mon.a (mon.0) 1452 : audit [INF] from='client.48812 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.015553+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.024807+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:03.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.025162+0000 mon.a (mon.0) 1455 : audit [INF] from='client.48794 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.025297+0000 mon.a (mon.0) 1456 : audit [INF] from='client.49315 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.025447+0000 mon.a (mon.0) 1457 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.025538+0000 mon.a (mon.0) 1458 : audit [INF] from='client.48812 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.025637+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.031220+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: audit 2024-09-10T18:56:03.042414+0000 mon.a (mon.0) 1460 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.063221+0000 osd.4 (osd.4) 177 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.094360+0000 osd.4 (osd.4) 178 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.194355+0000 mgr.y (mgr.24526) 281 : cluster [DBG] pgmap v294: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:03.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.024807+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.025162+0000 mon.a (mon.0) 1455 : audit [INF] from='client.48794 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.025297+0000 mon.a (mon.0) 1456 : audit [INF] from='client.49315 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.025447+0000 mon.a (mon.0) 1457 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.025538+0000 mon.a (mon.0) 1458 : audit [INF] from='client.48812 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.025637+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.031220+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: audit 2024-09-10T18:56:03.042414+0000 mon.a (mon.0) 1460 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.063221+0000 osd.4 (osd.4) 177 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.094360+0000 osd.4 (osd.4) 178 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:56:03.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:03 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.194355+0000 mgr.y (mgr.24526) 281 : cluster [DBG] pgmap v294: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:03.641 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:03.745 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:03.778 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:03.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.459828+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.83:0/3832577762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:02.460916+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-09-10T18:56:03.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.462011+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.83:0/1539942396' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.463675+0000 mon.a (mon.0) 1447 : audit [INF] from='client.48794 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.474437+0000 mon.a (mon.0) 1448 : audit [INF] from='client.49315 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:03.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.489041+0000 mon.c (mon.2) 290 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.489455+0000 mon.a (mon.0) 1449 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.490952+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.491310+0000 mon.a (mon.0) 1450 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.491778+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.492122+0000 mon.a (mon.0) 1451 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.684003+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.768279+0000 mon.c (mon.2) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3069936974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:02.768883+0000 mon.a (mon.0) 1452 : audit [INF] from='client.48812 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:56:03.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.015553+0000 mon.a (mon.0) 1453 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:03.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.024807+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:03.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.025162+0000 mon.a (mon.0) 1455 : audit [INF] from='client.48794 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi083-77788-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.025297+0000 mon.a (mon.0) 1456 : audit [INF] from='client.49315 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi083-77648-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:03.868 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.025447+0000 mon.a (mon.0) 1457 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi083-77640-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.025538+0000 mon.a (mon.0) 1458 : audit [INF] from='client.48812 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.025637+0000 mon.a (mon.0) 1459 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.031220+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: audit 2024-09-10T18:56:03.042414+0000 mon.a (mon.0) 1460 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.063221+0000 osd.4 (osd.4) 177 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:56:03.869 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.094360+0000 osd.4 (osd.4) 178 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:56:03.870 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:03 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.194355+0000 mgr.y (mgr.24526) 281 : cluster [DBG] pgmap v294: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:03.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:03.911 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:04.015 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:04.039 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:04.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:04.177 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:04.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:04.313 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:04.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:56:04.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:04.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:04.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:04.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:04.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:04.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:04.841 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:04.944 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:04.976 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:04.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:05.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:05.110 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:05.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:05.244 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:05.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.374 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:05.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.216394+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.219760+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.324741+0000 osd.5 (osd.5) 143 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.329279+0000 osd.5 (osd.5) 144 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.525744+0000 osd.1 (osd.1) 141 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.530733+0000 osd.1 (osd.1) 142 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.592500+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.595635+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: audit 2024-09-10T18:56:03.685571+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.696594+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.700014+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:56:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:03.974888+0000 mon.a (mon.0) 1461 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.018777+0000 osd.4 (osd.4) 179 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.020962+0000 osd.4 (osd.4) 180 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.428114+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.216394+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.219760+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.324741+0000 osd.5 (osd.5) 143 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.329279+0000 osd.5 (osd.5) 144 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.525744+0000 osd.1 (osd.1) 141 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.530733+0000 osd.1 (osd.1) 142 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.592500+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.595635+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:56:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: audit 2024-09-10T18:56:03.685571+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.696594+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.700014+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:03.974888+0000 mon.a (mon.0) 1461 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.018777+0000 osd.4 (osd.4) 179 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.020962+0000 osd.4 (osd.4) 180 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:56:05.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:05 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.428114+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:56:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.216394+0000 osd.0 (osd.0) 150 : cluster [DBG] 6.10 scrub starts 2024-09-10T18:56:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.219760+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.10 scrub ok 2024-09-10T18:56:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.324741+0000 osd.5 (osd.5) 143 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:56:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.329279+0000 osd.5 (osd.5) 144 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:56:05.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.525744+0000 osd.1 (osd.1) 141 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.530733+0000 osd.1 (osd.1) 142 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.592500+0000 osd.7 (osd.7) 145 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.595635+0000 osd.7 (osd.7) 146 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: audit 2024-09-10T18:56:03.685571+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.696594+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.700014+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:03.974888+0000 mon.a (mon.0) 1461 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.018777+0000 osd.4 (osd.4) 179 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.020962+0000 osd.4 (osd.4) 180 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:56:05.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:05 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.428114+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-09-10T18:56:05.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.647 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:05.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:05.787 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:05.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:05.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:05.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:06.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:06.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:06.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:06.189 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:06.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:06.322 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:06.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.466 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:06.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.352426+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.491566+0000 osd.1 (osd.1) 143 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.495321+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.551749+0000 osd.7 (osd.7) 147 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.556569+0000 osd.7 (osd.7) 148 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:04.687186+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.729149+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.730572+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:06.582 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.757919+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1d scrub starts 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:04.761856+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1d scrub ok 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.195237+0000 mgr.y (mgr.24526) 282 : cluster [DBG] pgmap v296: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.254455+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.257231+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:05.315057+0000 mon.a (mon.0) 1463 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.316016+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:05.319360+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:05.320325+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3339294164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:05.321027+0000 mon.a (mon.0) 1466 : audit [INF] from='client.48824 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.536485+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.539457+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: audit 2024-09-10T18:56:05.688749+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.984995+0000 osd.4 (osd.4) 181 : cluster [DBG] 16.19 scrub starts 2024-09-10T18:56:06.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:06 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.988898+0000 osd.4 (osd.4) 182 : cluster [DBG] 16.19 scrub ok 2024-09-10T18:56:06.602 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:06.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.352426+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:56:06.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.491566+0000 osd.1 (osd.1) 143 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:56:06.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.495321+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:56:06.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.551749+0000 osd.7 (osd.7) 147 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:56:06.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.556569+0000 osd.7 (osd.7) 148 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:04.687186+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.729149+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.730572+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.757919+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1d scrub starts 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:04.761856+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1d scrub ok 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.195237+0000 mgr.y (mgr.24526) 282 : cluster [DBG] pgmap v296: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.254455+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.257231+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:05.315057+0000 mon.a (mon.0) 1463 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.316016+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:56:06.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:05.319360+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:05.320325+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3339294164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:05.321027+0000 mon.a (mon.0) 1466 : audit [INF] from='client.48824 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.536485+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.539457+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: audit 2024-09-10T18:56:05.688749+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.984995+0000 osd.4 (osd.4) 181 : cluster [DBG] 16.19 scrub starts 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.988898+0000 osd.4 (osd.4) 182 : cluster [DBG] 16.19 scrub ok 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.352426+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:56:06.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.491566+0000 osd.1 (osd.1) 143 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.495321+0000 osd.1 (osd.1) 144 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.551749+0000 osd.7 (osd.7) 147 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.556569+0000 osd.7 (osd.7) 148 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:04.687186+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.729149+0000 osd.7 (osd.7) 149 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.730572+0000 osd.7 (osd.7) 150 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.757919+0000 osd.2 (osd.2) 83 : cluster [DBG] 16.1d scrub starts 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:04.761856+0000 osd.2 (osd.2) 84 : cluster [DBG] 16.1d scrub ok 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.195237+0000 mgr.y (mgr.24526) 282 : cluster [DBG] pgmap v296: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.254455+0000 osd.0 (osd.0) 152 : cluster [DBG] 4.16 scrub starts 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.257231+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.16 scrub ok 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:05.315057+0000 mon.a (mon.0) 1463 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi083-77640-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:06.707 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.316016+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:05.319360+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:05.320325+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3339294164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:05.321027+0000 mon.a (mon.0) 1466 : audit [INF] from='client.48824 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.536485+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.539457+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: audit 2024-09-10T18:56:05.688749+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.984995+0000 osd.4 (osd.4) 181 : cluster [DBG] 16.19 scrub starts 2024-09-10T18:56:06.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:06 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.988898+0000 osd.4 (osd.4) 182 : cluster [DBG] 16.19 scrub ok 2024-09-10T18:56:06.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:06.738 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:06.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.874 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:06.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:06.978 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:07.011 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:07.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:07.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:56:07.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:07.281 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:07.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:07.410 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:07.513 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:07.538 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.648063+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:05.650361+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: audit 2024-09-10T18:56:06.320544+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: audit 2024-09-10T18:56:06.320619+0000 mon.a (mon.0) 1468 : audit [INF] from='client.48824 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.320856+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.360951+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:07.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.361548+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.513102+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.517069+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.524368+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:06.527698+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: audit 2024-09-10T18:56:06.690468+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:07.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.196020+0000 mgr.y (mgr.24526) 283 : cluster [DBG] pgmap v299: 340 pgs: 72 creating+peering, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-09-10T18:56:07.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:07.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:07.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.648063+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:05.650361+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: audit 2024-09-10T18:56:06.320544+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: audit 2024-09-10T18:56:06.320619+0000 mon.a (mon.0) 1468 : audit [INF] from='client.48824 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.320856+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.360951+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.361548+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.513102+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.517069+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.524368+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:06.527698+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: audit 2024-09-10T18:56:06.690468+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.196020+0000 mgr.y (mgr.24526) 283 : cluster [DBG] pgmap v299: 340 pgs: 72 creating+peering, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.648063+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:05.650361+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: audit 2024-09-10T18:56:06.320544+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? v1:172.21.15.83:0/1011657977' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi083-77788-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: audit 2024-09-10T18:56:06.320619+0000 mon.a (mon.0) 1468 : audit [INF] from='client.48824 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi083-77648-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.320856+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.360951+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.361548+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.513102+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.517069+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.524368+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:06.527698+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:56:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: audit 2024-09-10T18:56:06.690468+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:07.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:07 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.196020+0000 mgr.y (mgr.24526) 283 : cluster [DBG] pgmap v299: 340 pgs: 72 creating+peering, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.8 KiB/s wr, 1 op/s 2024-09-10T18:56:07.806 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:07.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:07.923 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:07.944 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:07.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:08.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:08.080 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:08.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:08.215 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:08.318 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:08.345 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:08.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.433+0000 7f226543c640 1 -- 172.21.15.83:0/2683313605 >> v1:172.21.15.83:6789/0 conn(0x7f22601014e0 legacy=0x7f22601018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:08.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.433+0000 7f226543c640 1 -- 172.21.15.83:0/2683313605 shutdown_connections 2024-09-10T18:56:08.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.433+0000 7f226543c640 1 -- 172.21.15.83:0/2683313605 >> 172.21.15.83:0/2683313605 conn(0x7f2260075290 msgr2=0x7f22600f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:08.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.433+0000 7f226543c640 1 -- 172.21.15.83:0/2683313605 shutdown_connections 2024-09-10T18:56:08.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.433+0000 7f226543c640 1 -- 172.21.15.83:0/2683313605 wait complete. 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 librados: starting msgr at 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 librados: starting objecter 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 Processor -- start 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- start start 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 librados: setting wanted keys 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 librados: calling monclient init 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2260100b50 con 0x7f22601042a0 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2260100ce0 con 0x7f2260107e30 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2260100ed0 con 0x7f22601014e0 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f225effd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f22601014e0 0x7f2260100350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:41764/0 (socket says 172.21.15.83:41764) 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f225effd640 1 -- 172.21.15.83:0/3698527958 learned_addr learned my addr 172.21.15.83:0/3698527958 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:08.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter ms_handle_connect 0x7f22601014e0 2024-09-10T18:56:08.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:08.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter ms_handle_connect 0x7f22601042a0 2024-09-10T18:56:08.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter ms_handle_connect 0x7f2260107e30 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346739067 0 0) 0x7f2260100b50 con 0x7f22601042a0 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c003120 con 0x7f22601042a0 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256947816 0 0) 0x7f2260100ed0 con 0x7f22601014e0 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260100b50 con 0x7f22601014e0 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185672494 0 0) 0x7f2260100ce0 con 0x7f2260107e30 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260100ed0 con 0x7f2260107e30 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4036869860 0 0) 0x7f2260100b50 con 0x7f22601014e0 2024-09-10T18:56:08.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2260100ce0 con 0x7f22601014e0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2258001790 con 0x7f22601014e0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358197384 0 0) 0x7f224c003120 con 0x7f22601042a0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2260100b50 con 0x7f22601042a0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f22480018c0 con 0x7f22601042a0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007123995 0 0) 0x7f2260100ed0 con 0x7f2260107e30 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f224c003120 con 0x7f2260107e30 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f22500019f0 con 0x7f2260107e30 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1225295391 0 0) 0x7f2260100ce0 con 0x7f22601014e0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 >> v1:172.21.15.165:6789/0 conn(0x7f2260107e30 legacy=0x7f226011cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 >> v1:172.21.15.83:6789/0 conn(0x7f22601042a0 legacy=0x7f2260119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260121720 con 0x7f22601014e0 2024-09-10T18:56:08.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226011e710 con 0x7f22601014e0 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 10 client.48839.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f226011ebe0 con 0x7f22601014e0 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.437+0000 7f226543c640 1 librados: init done 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2258003210 con 0x7f22601014e0 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2258003a70 con 0x7f22601014e0 2024-09-10T18:56:08.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240005010 con 0x7f22601014e0 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2258003cb0 con 0x7f22601014e0 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(212..212 src has 69..212) v4 ==== 9874+0+0 (unknown 923953261 0 0) 0x7f225808b310 con 0x7f22601014e0 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 10 client.48839.objecter ms_dispatch 0x7f2260000b90 osd_map(212..212 src has 69..212) v4 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 3 client.48839.objecter handle_osd_map got epochs [212,212] > 0 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 3 client.48839.objecter handle_osd_map decoding full epoch 212 2024-09-10T18:56:08.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 20 client.48839.objecter dump_active .. 0 homeless 2024-09-10T18:56:08.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.441+0000 7f22577fe640 10 client.48839.objecter ms_handle_connect 0x7f224c071880 2024-09-10T18:56:08.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:08.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.445+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2258002780 con 0x7f22601014e0 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.261840+0000 osd.0 (osd.0) 154 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.266114+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.325310+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.327587+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.336433+0000 mon.a (mon.0) 1471 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.365671+0000 mon.c (mon.2) 302 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.512452+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:56:08.458 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: cluster 2024-09-10T18:56:07.514479+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.692055+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.725947+0000 mgr.y (mgr.24526) 284 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.794235+0000 mon.a (mon.0) 1472 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.795186+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.795663+0000 mon.a (mon.0) 1473 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.796475+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[18629]: audit 2024-09-10T18:56:07.796976+0000 mon.a (mon.0) 1474 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.261840+0000 osd.0 (osd.0) 154 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.266114+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.325310+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:56:08.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.327587+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.336433+0000 mon.a (mon.0) 1471 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.365671+0000 mon.c (mon.2) 302 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.512452+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: cluster 2024-09-10T18:56:07.514479+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.692055+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.725947+0000 mgr.y (mgr.24526) 284 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.794235+0000 mon.a (mon.0) 1472 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.795186+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.795663+0000 mon.a (mon.0) 1473 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.796475+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:08 smithi083 bash[32381]: audit 2024-09-10T18:56:07.796976+0000 mon.a (mon.0) 1474 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.473 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:08.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:08.609 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:08.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.261840+0000 osd.0 (osd.0) 154 : cluster [DBG] 3.1b scrub starts 2024-09-10T18:56:08.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.266114+0000 osd.0 (osd.0) 155 : cluster [DBG] 3.1b scrub ok 2024-09-10T18:56:08.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.325310+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-09-10T18:56:08.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.327587+0000 mon.c (mon.2) 301 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.336433+0000 mon.a (mon.0) 1471 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.365671+0000 mon.c (mon.2) 302 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.512452+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: cluster 2024-09-10T18:56:07.514479+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.692055+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.725947+0000 mgr.y (mgr.24526) 284 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.794235+0000 mon.a (mon.0) 1472 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.795186+0000 mon.c (mon.2) 304 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.795663+0000 mon.a (mon.0) 1473 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.796475+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:08 smithi165 bash[22591]: audit 2024-09-10T18:56:07.796976+0000 mon.a (mon.0) 1474 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:08.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:08.637+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f22400052c0 con 0x7f22601014e0 2024-09-10T18:56:08.711 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:08.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:08.740 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:08.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:08.876 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:08.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:08.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.008 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:09.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:09.145 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:09.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:09.280 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:09.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.408 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:09.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:09.542 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:09.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:09.672 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:09.776 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:09.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.797+0000 7f22577fe640 1 -- 172.21.15.83:0/3698527958 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v213) v1 ==== 113+0+0 (unknown 1280454613 0 0) 0x7f2258056b40 con 0x7f22601014e0 2024-09-10T18:56:09.800 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 10 librados: watch_flush enter 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 10 librados: watch_flush exit 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 20 client.48839.objecter shutdown clearing up homeless session... 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 10 client.48839.objecter successfully canceled tick 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 >> v1:172.21.15.83:6800/3071793735 conn(0x7f224c071880 legacy=0x7f224c073d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:09.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 >> v1:172.21.15.83:6790/0 conn(0x7f22601014e0 legacy=0x7f2260100350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:09.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 shutdown_connections 2024-09-10T18:56:09.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.801+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 >> 172.21.15.83:0/3698527958 conn(0x7f2260075290 msgr2=0x7f2260064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:09.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.805+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 shutdown_connections 2024-09-10T18:56:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.805+0000 7f226543c640 1 -- 172.21.15.83:0/3698527958 wait complete. 2024-09-10T18:56:09.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:09.805+0000 7f226543c640 1 librados: shutdown 2024-09-10T18:56:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp api_aio: Running main() from gmock_main.cc 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] Global test environment set-up. 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-10T18:56:09.812 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.TooBig (11723 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (5432 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (6284 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3013 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3264 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4953 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4100 ms) 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-10T18:56:09.813 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (4228 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4655 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3612 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2985 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3020 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2994 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2992 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.Flush (3015 ms) 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-10T18:56:09.814 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3006 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3011 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3059 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2930 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3453 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2992 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2602 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2960 ms) 2024-09-10T18:56:09.815 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3012 ms) 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3471 ms) 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 25 tests from LibRadosAio (96766 ms total) 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-10T18:56:09.816 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7346 ms) 2024-09-10T18:56:09.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.947 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:09.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.347548+0000 mon.a (mon.0) 1475 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.347744+0000 mon.a (mon.0) 1476 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: cluster 2024-09-10T18:56:08.347941+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.355925+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.361038+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.361982+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.363308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.364629+0000 mon.a (mon.0) 1480 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: cluster 2024-09-10T18:56:08.568498+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: cluster 2024-09-10T18:56:08.571876+0000 osd.1 (osd.1) 150 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.638708+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3698527958' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.639229+0000 mon.a (mon.0) 1481 : audit [INF] from='client.48839 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: audit 2024-09-10T18:56:08.693694+0000 mon.c (mon.2) 309 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: cluster 2024-09-10T18:56:08.978758+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.197004+0000 mgr.y (mgr.24526) 285 : cluster [DBG] pgmap v302: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.347548+0000 mon.a (mon.0) 1475 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.347744+0000 mon.a (mon.0) 1476 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: cluster 2024-09-10T18:56:08.347941+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.355925+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.361038+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.361982+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.363308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.364629+0000 mon.a (mon.0) 1480 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: cluster 2024-09-10T18:56:08.568498+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: cluster 2024-09-10T18:56:08.571876+0000 osd.1 (osd.1) 150 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.638708+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3698527958' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.639229+0000 mon.a (mon.0) 1481 : audit [INF] from='client.48839 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: audit 2024-09-10T18:56:08.693694+0000 mon.c (mon.2) 309 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:10.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: cluster 2024-09-10T18:56:08.978758+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:10.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:09 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.197004+0000 mgr.y (mgr.24526) 285 : cluster [DBG] pgmap v302: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-10T18:56:10.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:10.081 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.347548+0000 mon.a (mon.0) 1475 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.347744+0000 mon.a (mon.0) 1476 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: cluster 2024-09-10T18:56:08.347941+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-09-10T18:56:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.355925+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:10.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.361038+0000 mon.c (mon.2) 306 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.361982+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.83:0/300577746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.363308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.364629+0000 mon.a (mon.0) 1480 : audit [INF] from='client.48806 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: cluster 2024-09-10T18:56:08.568498+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: cluster 2024-09-10T18:56:08.571876+0000 osd.1 (osd.1) 150 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.638708+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3698527958' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.639229+0000 mon.a (mon.0) 1481 : audit [INF] from='client.48839 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: audit 2024-09-10T18:56:08.693694+0000 mon.c (mon.2) 309 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: cluster 2024-09-10T18:56:08.978758+0000 mon.a (mon.0) 1482 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:10.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:09 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.197004+0000 mgr.y (mgr.24526) 285 : cluster [DBG] pgmap v302: 300 pgs: 32 unknown, 268 active+clean; 608 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-09-10T18:56:10.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:10.221 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:10.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:10.359 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:10.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:10.496 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:10.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:10.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:10.632 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:10.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:10.766 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:10.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.354425+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:10.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.382482+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.384184+0000 osd.5 (osd.5) 147 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.614517+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.619556+0000 osd.1 (osd.1) 152 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.695187+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.700926+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.6 scrub starts 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.704445+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.6 scrub ok 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.763302+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.799451+0000 mon.a (mon.0) 1484 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.799690+0000 mon.a (mon.0) 1485 : audit [INF] from='client.48839 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:09.799787+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.810892+0000 mon.a (mon.0) 1487 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.830854+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.831172+0000 mon.a (mon.0) 1488 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.833203+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.833475+0000 mon.a (mon.0) 1489 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.834010+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:09.834349+0000 mon.a (mon.0) 1490 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.356242+0000 mon.a (mon.0) 1491 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.356461+0000 mon.a (mon.0) 1492 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.356696+0000 mon.a (mon.0) 1493 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.356961+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.359601+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.364461+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.366083+0000 mon.a (mon.0) 1495 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:10.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:10 smithi165 bash[22591]: audit 2024-09-10T18:56:10.367304+0000 mon.a (mon.0) 1496 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:10.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:10.902 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:10.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:11.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:11.042 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:11.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.159 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.354425+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.382482+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.384184+0000 osd.5 (osd.5) 147 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.614517+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.619556+0000 osd.1 (osd.1) 152 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.695187+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.700926+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.6 scrub starts 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.704445+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.6 scrub ok 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.763302+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.799451+0000 mon.a (mon.0) 1484 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:11.160 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.799690+0000 mon.a (mon.0) 1485 : audit [INF] from='client.48839 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:09.799787+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.810892+0000 mon.a (mon.0) 1487 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.830854+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.831172+0000 mon.a (mon.0) 1488 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.833203+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.833475+0000 mon.a (mon.0) 1489 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.834010+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.354425+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? v1:172.21.15.83:0/1538409934' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi083-77648-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.382482+0000 osd.5 (osd.5) 146 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:56:11.161 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.384184+0000 osd.5 (osd.5) 147 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.614517+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.619556+0000 osd.1 (osd.1) 152 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.695187+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.700926+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.6 scrub starts 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.704445+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.6 scrub ok 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.763302+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.799451+0000 mon.a (mon.0) 1484 : audit [INF] from='client.48806 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi083-77640-26"}]': finished 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.799690+0000 mon.a (mon.0) 1485 : audit [INF] from='client.48839 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:09.799787+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.810892+0000 mon.a (mon.0) 1487 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.830854+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.831172+0000 mon.a (mon.0) 1488 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.833203+0000 mon.c (mon.2) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.162 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.833475+0000 mon.a (mon.0) 1489 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.834010+0000 mon.c (mon.2) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:09.834349+0000 mon.a (mon.0) 1490 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.356242+0000 mon.a (mon.0) 1491 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.356461+0000 mon.a (mon.0) 1492 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.356696+0000 mon.a (mon.0) 1493 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.356961+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.359601+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.364461+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.366083+0000 mon.a (mon.0) 1495 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.163 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[32381]: audit 2024-09-10T18:56:10.367304+0000 mon.a (mon.0) 1496 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:09.834349+0000 mon.a (mon.0) 1490 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.356242+0000 mon.a (mon.0) 1491 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi083-77788-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.356461+0000 mon.a (mon.0) 1492 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.356696+0000 mon.a (mon.0) 1493 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi083-77640-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.356961+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.359601+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.364461+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.366083+0000 mon.a (mon.0) 1495 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:11.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:10 smithi083 bash[18629]: audit 2024-09-10T18:56:10.367304+0000 mon.a (mon.0) 1496 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]: dispatch 2024-09-10T18:56:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:11.282 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:11.317 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:11.334 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T18:56:11.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:56:11.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:11.450 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:11.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:11.578 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:11.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:11.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.716 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:11.753 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:11.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:11.841 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:11.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:11.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.202909+0000 osd.0 (osd.0) 156 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:56:11.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.206467+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:56:11.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: audit 2024-09-10T18:56:10.696597+0000 mon.c (mon.2) 315 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.718756+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:56:11.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.722494+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.747845+0000 osd.6 (osd.6) 121 : cluster [DBG] 12.17 scrub starts 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.750405+0000 osd.6 (osd.6) 122 : cluster [DBG] 12.17 scrub ok 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.955753+0000 osd.4 (osd.4) 183 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:10.959633+0000 osd.4 (osd.4) 184 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:11.199369+0000 mgr.y (mgr.24526) 286 : cluster [DBG] pgmap v305: 276 pgs: 8 creating+peering, 56 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 196 active+clean; 615 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:11.351209+0000 mon.a (mon.0) 1497 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: audit 2024-09-10T18:56:11.698141+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: audit 2024-09-10T18:56:11.817929+0000 mon.a (mon.0) 1498 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: cluster 2024-09-10T18:56:11.818186+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: audit 2024-09-10T18:56:11.819598+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.83:0/960092535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[32381]: audit 2024-09-10T18:56:11.820622+0000 mon.a (mon.0) 1500 : audit [INF] from='client.48851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.202909+0000 osd.0 (osd.0) 156 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:56:11.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.206467+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: audit 2024-09-10T18:56:10.696597+0000 mon.c (mon.2) 315 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.718756+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.722494+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.747845+0000 osd.6 (osd.6) 121 : cluster [DBG] 12.17 scrub starts 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.750405+0000 osd.6 (osd.6) 122 : cluster [DBG] 12.17 scrub ok 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.955753+0000 osd.4 (osd.4) 183 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:10.959633+0000 osd.4 (osd.4) 184 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:11.199369+0000 mgr.y (mgr.24526) 286 : cluster [DBG] pgmap v305: 276 pgs: 8 creating+peering, 56 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 196 active+clean; 615 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:11.351209+0000 mon.a (mon.0) 1497 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:11.957 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: audit 2024-09-10T18:56:11.698141+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:11.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: audit 2024-09-10T18:56:11.817929+0000 mon.a (mon.0) 1498 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:56:11.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: cluster 2024-09-10T18:56:11.818186+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:56:11.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: audit 2024-09-10T18:56:11.819598+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.83:0/960092535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:11.958 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:11 smithi083 bash[18629]: audit 2024-09-10T18:56:11.820622+0000 mon.a (mon.0) 1500 : audit [INF] from='client.48851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:11.974 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:11.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:12.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:12.090 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.202909+0000 osd.0 (osd.0) 156 : cluster [DBG] 2.1f scrub starts 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.206467+0000 osd.0 (osd.0) 157 : cluster [DBG] 2.1f scrub ok 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: audit 2024-09-10T18:56:10.696597+0000 mon.c (mon.2) 315 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.718756+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.722494+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.747845+0000 osd.6 (osd.6) 121 : cluster [DBG] 12.17 scrub starts 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.750405+0000 osd.6 (osd.6) 122 : cluster [DBG] 12.17 scrub ok 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.955753+0000 osd.4 (osd.4) 183 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:10.959633+0000 osd.4 (osd.4) 184 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:11.199369+0000 mgr.y (mgr.24526) 286 : cluster [DBG] pgmap v305: 276 pgs: 8 creating+peering, 56 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 196 active+clean; 615 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:11.351209+0000 mon.a (mon.0) 1497 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: audit 2024-09-10T18:56:11.698141+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: audit 2024-09-10T18:56:11.817929+0000 mon.a (mon.0) 1498 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-13"}]': finished 2024-09-10T18:56:12.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: cluster 2024-09-10T18:56:11.818186+0000 mon.a (mon.0) 1499 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-09-10T18:56:12.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: audit 2024-09-10T18:56:11.819598+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.83:0/960092535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:12.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:11 smithi165 bash[22591]: audit 2024-09-10T18:56:11.820622+0000 mon.a (mon.0) 1500 : audit [INF] from='client.48851 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:12.110 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:12.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:12.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:12.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:12.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:12.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:12.532 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:12.636 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:12.669 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:12.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:12.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:12.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:12.916 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:12] "GET /metrics HTTP/1.1" 200 235852 "" "Prometheus/2.43.0" 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: cluster 2024-09-10T18:56:11.171014+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: cluster 2024-09-10T18:56:11.174381+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: audit 2024-09-10T18:56:12.366820+0000 mon.a (mon.0) 1501 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: audit 2024-09-10T18:56:12.367061+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: cluster 2024-09-10T18:56:12.367231+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: audit 2024-09-10T18:56:12.372016+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: audit 2024-09-10T18:56:12.379117+0000 mon.a (mon.0) 1504 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:12.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[18629]: audit 2024-09-10T18:56:12.699778+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: cluster 2024-09-10T18:56:11.171014+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: cluster 2024-09-10T18:56:11.174381+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: audit 2024-09-10T18:56:12.366820+0000 mon.a (mon.0) 1501 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: audit 2024-09-10T18:56:12.367061+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: cluster 2024-09-10T18:56:12.367231+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: audit 2024-09-10T18:56:12.372016+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: audit 2024-09-10T18:56:12.379117+0000 mon.a (mon.0) 1504 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:12.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:12 smithi083 bash[32381]: audit 2024-09-10T18:56:12.699778+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:12.936 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:12.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:13.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:13.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: cluster 2024-09-10T18:56:11.171014+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.0 scrub starts 2024-09-10T18:56:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: cluster 2024-09-10T18:56:11.174381+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.0 scrub ok 2024-09-10T18:56:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: audit 2024-09-10T18:56:12.366820+0000 mon.a (mon.0) 1501 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi083-77640-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: audit 2024-09-10T18:56:12.367061+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48851 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi083-77648-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:13.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: cluster 2024-09-10T18:56:12.367231+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-09-10T18:56:13.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: audit 2024-09-10T18:56:12.372016+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:13.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: audit 2024-09-10T18:56:12.379117+0000 mon.a (mon.0) 1504 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:13.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:12 smithi165 bash[22591]: audit 2024-09-10T18:56:12.699778+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:13.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.211 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:13.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:13.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.475 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:13.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:13.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.749 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:13.838 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:13.850 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.871 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:13.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:13.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForCompl (69981 ms total) 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (164 ms) 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (164 ms total) 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-10T18:56:14.019 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (39 ms) 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (42 ms total) 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (71 ms) 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-09-10T18:56:14.020 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (75 ms total) 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (23 ms) 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (9 ms) 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (33 ms total) 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-10T18:56:14.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1018 ms) 2024-09-10T18:56:14.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1018 ms total) 2024-09-10T18:56:14.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-09-10T18:56:14.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-10T18:56:14.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (108216 ms total) 2024-09-10T18:56:14.022 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-10T18:56:14.022 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.200251+0000 mgr.y (mgr.24526) 287 : cluster [DBG] pgmap v308: 276 pgs: 40 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T18:56:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: audit 2024-09-10T18:56:13.370610+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.371337+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:56:14.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: audit 2024-09-10T18:56:13.375591+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: audit 2024-09-10T18:56:13.380069+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: audit 2024-09-10T18:56:13.390808+0000 mon.a (mon.0) 1507 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[32381]: audit 2024-09-10T18:56:13.391400+0000 mon.a (mon.0) 1508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.200251+0000 mgr.y (mgr.24526) 287 : cluster [DBG] pgmap v308: 276 pgs: 40 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: audit 2024-09-10T18:56:13.370610+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.371337+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: audit 2024-09-10T18:56:13.375591+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: audit 2024-09-10T18:56:13.380069+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: audit 2024-09-10T18:56:13.390808+0000 mon.a (mon.0) 1507 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:13 smithi083 bash[18629]: audit 2024-09-10T18:56:13.391400+0000 mon.a (mon.0) 1508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.200251+0000 mgr.y (mgr.24526) 287 : cluster [DBG] pgmap v308: 276 pgs: 40 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T18:56:14.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: audit 2024-09-10T18:56:13.370610+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:14.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.371337+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-09-10T18:56:14.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: audit 2024-09-10T18:56:13.375591+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.83:0/2020160585' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: audit 2024-09-10T18:56:13.380069+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: audit 2024-09-10T18:56:13.390808+0000 mon.a (mon.0) 1507 : audit [INF] from='client.48833 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]: dispatch 2024-09-10T18:56:14.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:13 smithi165 bash[22591]: audit 2024-09-10T18:56:13.391400+0000 mon.a (mon.0) 1508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:14.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:14.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.300 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:14.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:14.435 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:14.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:14.575 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:14.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:14.714 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:14.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.855 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:14.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:14.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:14.994 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:14.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:15.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:12.677183+0000 osd.2 (osd.2) 85 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:12.679930+0000 osd.2 (osd.2) 86 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.406332+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.c scrub starts 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.408412+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.c scrub ok 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.646623+0000 osd.2 (osd.2) 87 : cluster [DBG] 12.a scrub starts 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.649450+0000 osd.2 (osd.2) 88 : cluster [DBG] 12.a scrub ok 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:13.701116+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.112 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.750271+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.752978+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.956797+0000 osd.4 (osd.4) 185 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.960290+0000 osd.4 (osd.4) 186 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:13.996372+0000 mon.a (mon.0) 1509 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:13.996505+0000 mon.a (mon.0) 1510 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: cluster 2024-09-10T18:56:13.996694+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:13.999593+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.83:0/3398247217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.007474+0000 mon.a (mon.0) 1512 : audit [INF] from='client.49375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.007918+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.013683+0000 mon.a (mon.0) 1513 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.564795+0000 mon.a (mon.0) 1514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:15.113 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.568468+0000 mon.c (mon.2) 323 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[18629]: audit 2024-09-10T18:56:14.702829+0000 mon.c (mon.2) 324 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:12.677183+0000 osd.2 (osd.2) 85 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:12.679930+0000 osd.2 (osd.2) 86 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.406332+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.c scrub starts 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.408412+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.c scrub ok 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.646623+0000 osd.2 (osd.2) 87 : cluster [DBG] 12.a scrub starts 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.649450+0000 osd.2 (osd.2) 88 : cluster [DBG] 12.a scrub ok 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:13.701116+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.750271+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.752978+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.956797+0000 osd.4 (osd.4) 185 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:56:15.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.960290+0000 osd.4 (osd.4) 186 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:13.996372+0000 mon.a (mon.0) 1509 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:13.996505+0000 mon.a (mon.0) 1510 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: cluster 2024-09-10T18:56:13.996694+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:13.999593+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.83:0/3398247217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.007474+0000 mon.a (mon.0) 1512 : audit [INF] from='client.49375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.007918+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.013683+0000 mon.a (mon.0) 1513 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.564795+0000 mon.a (mon.0) 1514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.568468+0000 mon.c (mon.2) 323 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:15.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:14 smithi083 bash[32381]: audit 2024-09-10T18:56:14.702829+0000 mon.c (mon.2) 324 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:12.677183+0000 osd.2 (osd.2) 85 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:12.679930+0000 osd.2 (osd.2) 86 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.406332+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.c scrub starts 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.408412+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.c scrub ok 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.646623+0000 osd.2 (osd.2) 87 : cluster [DBG] 12.a scrub starts 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.649450+0000 osd.2 (osd.2) 88 : cluster [DBG] 12.a scrub ok 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:13.701116+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.750271+0000 osd.3 (osd.3) 167 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.752978+0000 osd.3 (osd.3) 168 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.956797+0000 osd.4 (osd.4) 185 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.960290+0000 osd.4 (osd.4) 186 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:56:15.116 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:13.996372+0000 mon.a (mon.0) 1509 : audit [INF] from='client.48833 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi083-77788-36"}]': finished 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:13.996505+0000 mon.a (mon.0) 1510 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: cluster 2024-09-10T18:56:13.996694+0000 mon.a (mon.0) 1511 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:13.999593+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.83:0/3398247217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.007474+0000 mon.a (mon.0) 1512 : audit [INF] from='client.49375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.007918+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.013683+0000 mon.a (mon.0) 1513 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.564795+0000 mon.a (mon.0) 1514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.568468+0000 mon.c (mon.2) 323 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:15.117 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:14 smithi165 bash[22591]: audit 2024-09-10T18:56:14.702829+0000 mon.c (mon.2) 324 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:15.134 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:15.237 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:15.269 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:15.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:15.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:15.508 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:15.542 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:15.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:15.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:15.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:15.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:15.868 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:15.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:15.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:15.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:15.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:16.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.551193+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.554660+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.604567+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.1c scrub starts 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.607921+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.1c scrub ok 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.793249+0000 osd.3 (osd.3) 169 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:14.796849+0000 osd.3 (osd.3) 170 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.000356+0000 mon.a (mon.0) 1515 : audit [INF] from='client.49375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.000539+0000 mon.a (mon.0) 1516 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.000755+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:56:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.003791+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.005117+0000 mon.a (mon.0) 1518 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.014339+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.018763+0000 mon.a (mon.0) 1519 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.201202+0000 mgr.y (mgr.24526) 288 : cluster [DBG] pgmap v312: 300 pgs: 64 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.645259+0000 osd.1 (osd.1) 153 : cluster [DBG] 16.13 scrub starts 2024-09-10T18:56:16.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.650671+0000 osd.1 (osd.1) 154 : cluster [DBG] 16.13 scrub ok 2024-09-10T18:56:16.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[32381]: audit 2024-09-10T18:56:15.704346+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:16.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.551193+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:56:16.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.554660+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:56:16.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.604567+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.1c scrub starts 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.607921+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.1c scrub ok 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.793249+0000 osd.3 (osd.3) 169 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:14.796849+0000 osd.3 (osd.3) 170 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.000356+0000 mon.a (mon.0) 1515 : audit [INF] from='client.49375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.000539+0000 mon.a (mon.0) 1516 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.000755+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:56:16.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.003791+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.005117+0000 mon.a (mon.0) 1518 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.014339+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.018763+0000 mon.a (mon.0) 1519 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.201202+0000 mgr.y (mgr.24526) 288 : cluster [DBG] pgmap v312: 300 pgs: 64 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.645259+0000 osd.1 (osd.1) 153 : cluster [DBG] 16.13 scrub starts 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.650671+0000 osd.1 (osd.1) 154 : cluster [DBG] 16.13 scrub ok 2024-09-10T18:56:16.045 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:16 smithi083 bash[18629]: audit 2024-09-10T18:56:15.704346+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:16.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.083 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:16.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.551193+0000 osd.7 (osd.7) 155 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:56:16.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.554660+0000 osd.7 (osd.7) 156 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:56:16.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.604567+0000 osd.2 (osd.2) 89 : cluster [DBG] 16.1c scrub starts 2024-09-10T18:56:16.200 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.607921+0000 osd.2 (osd.2) 90 : cluster [DBG] 16.1c scrub ok 2024-09-10T18:56:16.201 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.793249+0000 osd.3 (osd.3) 169 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:56:16.201 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:14.796849+0000 osd.3 (osd.3) 170 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:56:16.201 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.000356+0000 mon.a (mon.0) 1515 : audit [INF] from='client.49375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi083-77648-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:16.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.000539+0000 mon.a (mon.0) 1516 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:16.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.000755+0000 mon.a (mon.0) 1517 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-09-10T18:56:16.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.003791+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3434406830' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.005117+0000 mon.a (mon.0) 1518 : audit [INF] from='client.48845 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]: dispatch 2024-09-10T18:56:16.202 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.014339+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.018763+0000 mon.a (mon.0) 1519 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:16.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.201202+0000 mgr.y (mgr.24526) 288 : cluster [DBG] pgmap v312: 300 pgs: 64 unknown, 32 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 196 active+clean; 613 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:16.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.645259+0000 osd.1 (osd.1) 153 : cluster [DBG] 16.13 scrub starts 2024-09-10T18:56:16.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.650671+0000 osd.1 (osd.1) 154 : cluster [DBG] 16.13 scrub ok 2024-09-10T18:56:16.203 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:16 smithi165 bash[22591]: audit 2024-09-10T18:56:15.704346+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:16.219 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:16.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.357 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:16.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:16.492 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:16.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:16.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:16.768 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:16.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:16.904 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:16.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:17.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:17.035 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:17.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:17.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:17.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.240666+0000 osd.0 (osd.0) 160 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:56:17.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.245297+0000 osd.0 (osd.0) 161 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.523392+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.f scrub starts 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.528454+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.f scrub ok 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.652986+0000 osd.2 (osd.2) 91 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.655050+0000 osd.2 (osd.2) 92 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:15.993887+0000 mon.a (mon.0) 1520 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.001409+0000 mon.a (mon.0) 1521 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.002015+0000 mon.a (mon.0) 1522 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:16.002181+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.005531+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.011191+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.019805+0000 mon.a (mon.0) 1525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.044243+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.291 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.475938+0000 mon.a (mon.0) 1526 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.476538+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.476882+0000 mon.a (mon.0) 1527 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.477371+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.477670+0000 mon.a (mon.0) 1528 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:16.654982+0000 osd.1 (osd.1) 155 : cluster [DBG] 16.b scrub starts 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: cluster 2024-09-10T18:56:16.658930+0000 osd.1 (osd.1) 156 : cluster [DBG] 16.b scrub ok 2024-09-10T18:56:17.292 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[22591]: audit 2024-09-10T18:56:16.705991+0000 mon.c (mon.2) 330 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:17.310 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.240666+0000 osd.0 (osd.0) 160 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.245297+0000 osd.0 (osd.0) 161 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.523392+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.f scrub starts 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.528454+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.f scrub ok 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.652986+0000 osd.2 (osd.2) 91 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.655050+0000 osd.2 (osd.2) 92 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:15.993887+0000 mon.a (mon.0) 1520 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.001409+0000 mon.a (mon.0) 1521 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.002015+0000 mon.a (mon.0) 1522 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:17.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:16.002181+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.005531+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.011191+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.019805+0000 mon.a (mon.0) 1525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.044243+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.475938+0000 mon.a (mon.0) 1526 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.476538+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.476882+0000 mon.a (mon.0) 1527 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.477371+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.477670+0000 mon.a (mon.0) 1528 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:16.654982+0000 osd.1 (osd.1) 155 : cluster [DBG] 16.b scrub starts 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: cluster 2024-09-10T18:56:16.658930+0000 osd.1 (osd.1) 156 : cluster [DBG] 16.b scrub ok 2024-09-10T18:56:17.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[32381]: audit 2024-09-10T18:56:16.705991+0000 mon.c (mon.2) 330 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.240666+0000 osd.0 (osd.0) 160 : cluster [DBG] 3.1f scrub starts 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.245297+0000 osd.0 (osd.0) 161 : cluster [DBG] 3.1f scrub ok 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.523392+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.f scrub starts 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.528454+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.f scrub ok 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.652986+0000 osd.2 (osd.2) 91 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.655050+0000 osd.2 (osd.2) 92 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:15.993887+0000 mon.a (mon.0) 1520 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.001409+0000 mon.a (mon.0) 1521 : audit [INF] from='client.48845 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi083-77640-27"}]': finished 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.002015+0000 mon.a (mon.0) 1522 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:16.002181+0000 mon.a (mon.0) 1523 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-09-10T18:56:17.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.005531+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.011191+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.019805+0000 mon.a (mon.0) 1525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.044243+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.475938+0000 mon.a (mon.0) 1526 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.476538+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.476882+0000 mon.a (mon.0) 1527 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.477371+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.477670+0000 mon.a (mon.0) 1528 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:16.654982+0000 osd.1 (osd.1) 155 : cluster [DBG] 16.b scrub starts 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: cluster 2024-09-10T18:56:16.658930+0000 osd.1 (osd.1) 156 : cluster [DBG] 16.b scrub ok 2024-09-10T18:56:17.405 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:17 smithi083 bash[18629]: audit 2024-09-10T18:56:16.705991+0000 mon.c (mon.2) 330 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:17.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:17.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:17.583 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:17.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:17.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:17.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:17.854 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:17.866 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:56:17.866 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:56:17.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.992 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:17.994 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:17.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:18.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:18.125 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:18.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:18.262 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:18.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:18.393 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout:: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4327 ms) 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4459 ms) 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3186 ms) 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11972 ms total) 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-09-10T18:56:18.471 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-10T18:56:18.472 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (112612 ms total) 2024-09-10T18:56:18.472 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-10T18:56:18.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:18.525 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:18.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.008721+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.010192+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.010391+0000 mon.a (mon.0) 1530 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:18.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.010866+0000 mon.a (mon.0) 1531 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:18.527 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.010999+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.012020+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.013002+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.013478+0000 mon.a (mon.0) 1534 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.014502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.015459+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.202155+0000 mgr.y (mgr.24526) 289 : cluster [DBG] pgmap v315: 292 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.670530+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.673927+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:17.707601+0000 mon.c (mon.2) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.001173+0000 mon.a (mon.0) 1537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:18.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:18 smithi165 bash[22591]: audit 2024-09-10T18:56:18.010435+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:18.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:18.665 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.008721+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.010192+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.010391+0000 mon.a (mon.0) 1530 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.010866+0000 mon.a (mon.0) 1531 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.010999+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.012020+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.013002+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.013478+0000 mon.a (mon.0) 1534 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.014502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.015459+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.202155+0000 mgr.y (mgr.24526) 289 : cluster [DBG] pgmap v315: 292 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.670530+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.673927+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:17.707601+0000 mon.c (mon.2) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.001173+0000 mon.a (mon.0) 1537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[18629]: audit 2024-09-10T18:56:18.010435+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.008721+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.010192+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.010391+0000 mon.a (mon.0) 1530 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:18.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.010866+0000 mon.a (mon.0) 1531 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi083-77640-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.010999+0000 mon.a (mon.0) 1532 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.012020+0000 mon.c (mon.2) 331 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.013002+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.013478+0000 mon.a (mon.0) 1534 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.014502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.015459+0000 mon.a (mon.0) 1536 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.202155+0000 mgr.y (mgr.24526) 289 : cluster [DBG] pgmap v315: 292 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.670530+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.673927+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:17.707601+0000 mon.c (mon.2) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.001173+0000 mon.a (mon.0) 1537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:18.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:18 smithi083 bash[32381]: audit 2024-09-10T18:56:18.010435+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? v1:172.21.15.83:0/1216054038' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi083-77943-20"}]': finished 2024-09-10T18:56:18.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:18.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:18.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:18.935 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:18.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:19.026 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-10T18:56:19.026 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (11707 ms) 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19354 ms) 2024-09-10T18:56:19.027 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8750 ms) 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3450 ms) 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3200 ms) 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3791 ms) 2024-09-10T18:56:19.028 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2980 ms) 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3024 ms) 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3008 ms) 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3007 ms) 2024-09-10T18:56:19.029 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2987 ms) 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2995 ms) 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3017 ms) 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3031 ms) 2024-09-10T18:56:19.030 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3045 ms) 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2921 ms) 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3040 ms) 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3006 ms) 2024-09-10T18:56:19.031 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-10T18:56:19.032 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2958 ms) 2024-09-10T18:56:19.032 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T18:56:19.032 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3013 ms) 2024-09-10T18:56:19.032 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3036 ms) 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3403 ms) 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2898 ms) 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-10T18:56:19.033 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3035 ms) 2024-09-10T18:56:19.034 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-10T18:56:19.034 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3012 ms) 2024-09-10T18:56:19.034 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-10T18:56:19.034 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2644 ms) 2024-09-10T18:56:19.034 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-10T18:56:19.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.061 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:19.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.197 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:19.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:19.328 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:19.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:19.465 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:19.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:19.600 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:19.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.487132+0000 osd.7 (osd.7) 159 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:56:19.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.491703+0000 osd.7 (osd.7) 160 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:56:19.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.668171+0000 osd.2 (osd.2) 93 : cluster [DBG] 16.e scrub starts 2024-09-10T18:56:19.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.672013+0000 osd.2 (osd.2) 94 : cluster [DBG] 16.e scrub ok 2024-09-10T18:56:19.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: audit 2024-09-10T18:56:17.737080+0000 mgr.y (mgr.24526) 290 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.864343+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.867211+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.926333+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:17.929809+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: audit 2024-09-10T18:56:18.456505+0000 mon.a (mon.0) 1539 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]': finished 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: audit 2024-09-10T18:56:18.456748+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.456993+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: audit 2024-09-10T18:56:18.709319+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: audit 2024-09-10T18:56:19.013420+0000 mon.a (mon.0) 1542 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:19.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:19 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.013747+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:56:19.734 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.487132+0000 osd.7 (osd.7) 159 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:56:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.491703+0000 osd.7 (osd.7) 160 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.668171+0000 osd.2 (osd.2) 93 : cluster [DBG] 16.e scrub starts 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.672013+0000 osd.2 (osd.2) 94 : cluster [DBG] 16.e scrub ok 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: audit 2024-09-10T18:56:17.737080+0000 mgr.y (mgr.24526) 290 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.864343+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.867211+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.926333+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:56:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:17.929809+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: audit 2024-09-10T18:56:18.456505+0000 mon.a (mon.0) 1539 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]': finished 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: audit 2024-09-10T18:56:18.456748+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.456993+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: audit 2024-09-10T18:56:18.709319+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: audit 2024-09-10T18:56:19.013420+0000 mon.a (mon.0) 1542 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.013747+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.487132+0000 osd.7 (osd.7) 159 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.491703+0000 osd.7 (osd.7) 160 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.668171+0000 osd.2 (osd.2) 93 : cluster [DBG] 16.e scrub starts 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.672013+0000 osd.2 (osd.2) 94 : cluster [DBG] 16.e scrub ok 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: audit 2024-09-10T18:56:17.737080+0000 mgr.y (mgr.24526) 290 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.864343+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.867211+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.926333+0000 osd.4 (osd.4) 187 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:17.929809+0000 osd.4 (osd.4) 188 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: audit 2024-09-10T18:56:18.456505+0000 mon.a (mon.0) 1539 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-15", "mode": "writeback"}]': finished 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: audit 2024-09-10T18:56:18.456748+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.83:0/2507115064' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi083-77648-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.456993+0000 mon.a (mon.0) 1541 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: audit 2024-09-10T18:56:18.709319+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:19.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: audit 2024-09-10T18:56:19.013420+0000 mon.a (mon.0) 1542 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi083-77640-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:19.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:19 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.013747+0000 mon.a (mon.0) 1543 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-09-10T18:56:19.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:19.869 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:19.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.004 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:20.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.142 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:20.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.281 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:20.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.417 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:20.520 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.265407+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:56:20.555 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:20.646 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T18:56:20.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:20.694 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:20.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.810 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.265407+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.268804+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.882352+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:18.886644+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:19.058015+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.203001+0000 mgr.y (mgr.24526) 291 : cluster [DBG] pgmap v318: 268 pgs: 8 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.444694+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.446989+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:19.479283+0000 mon.a (mon.0) 1544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:19.711096+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.763253+0000 osd.6 (osd.6) 123 : cluster [DBG] 12.f scrub starts 2024-09-10T18:56:20.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.766826+0000 osd.6 (osd.6) 124 : cluster [DBG] 12.f scrub ok 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.926851+0000 osd.3 (osd.3) 175 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:19.931404+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:20.016020+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:20.017390+0000 mon.a (mon.0) 1545 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: cluster 2024-09-10T18:56:20.017494+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:20.018927+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.83:0/280424955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:20.020868+0000 mon.a (mon.0) 1547 : audit [INF] from='client.48875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:20 smithi165 bash[22591]: audit 2024-09-10T18:56:20.032017+0000 mon.a (mon.0) 1548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.268804+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.882352+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:18.886644+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:19.058015+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.203001+0000 mgr.y (mgr.24526) 291 : cluster [DBG] pgmap v318: 268 pgs: 8 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.444694+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.446989+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:19.479283+0000 mon.a (mon.0) 1544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:19.711096+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.763253+0000 osd.6 (osd.6) 123 : cluster [DBG] 12.f scrub starts 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.766826+0000 osd.6 (osd.6) 124 : cluster [DBG] 12.f scrub ok 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.926851+0000 osd.3 (osd.3) 175 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:19.931404+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:56:20.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:20.016020+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:20.017390+0000 mon.a (mon.0) 1545 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: cluster 2024-09-10T18:56:20.017494+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:20.018927+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.83:0/280424955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:20.020868+0000 mon.a (mon.0) 1547 : audit [INF] from='client.48875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[18629]: audit 2024-09-10T18:56:20.032017+0000 mon.a (mon.0) 1548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.265407+0000 osd.0 (osd.0) 162 : cluster [DBG] 4.1e scrub starts 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.268804+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.1e scrub ok 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.882352+0000 osd.3 (osd.3) 173 : cluster [DBG] 4.0 scrub starts 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:18.886644+0000 osd.3 (osd.3) 174 : cluster [DBG] 4.0 scrub ok 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:19.058015+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.203001+0000 mgr.y (mgr.24526) 291 : cluster [DBG] pgmap v318: 268 pgs: 8 unknown, 16 active+clean+snaptrim_wait, 16 active+clean+snaptrim, 228 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:20.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.444694+0000 osd.7 (osd.7) 161 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.446989+0000 osd.7 (osd.7) 162 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:19.479283+0000 mon.a (mon.0) 1544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:19.711096+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.763253+0000 osd.6 (osd.6) 123 : cluster [DBG] 12.f scrub starts 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.766826+0000 osd.6 (osd.6) 124 : cluster [DBG] 12.f scrub ok 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.926851+0000 osd.3 (osd.3) 175 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:19.931404+0000 osd.3 (osd.3) 176 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:20.016020+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:20.017390+0000 mon.a (mon.0) 1545 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: cluster 2024-09-10T18:56:20.017494+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:20.018927+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.83:0/280424955' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:20.020868+0000 mon.a (mon.0) 1547 : audit [INF] from='client.48875 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:20.824 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:20 smithi083 bash[32381]: audit 2024-09-10T18:56:20.032017+0000 mon.a (mon.0) 1548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]: dispatch 2024-09-10T18:56:20.832 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:20.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:20.974 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:20.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:21.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:21.104 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:21.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:21.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:21.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:21.376 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:21.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:21.488 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:21.511 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:21.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:21.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: cluster 2024-09-10T18:56:20.678026+0000 osd.1 (osd.1) 159 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: cluster 2024-09-10T18:56:20.681238+0000 osd.1 (osd.1) 160 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: audit 2024-09-10T18:56:20.712892+0000 mon.c (mon.2) 336 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.011166+0000 mon.a (mon.0) 1549 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: audit 2024-09-10T18:56:21.020744+0000 mon.a (mon.0) 1550 : audit [INF] from='client.48875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: audit 2024-09-10T18:56:21.020966+0000 mon.a (mon.0) 1551 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.021434+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: audit 2024-09-10T18:56:21.030201+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: audit 2024-09-10T18:56:21.031070+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.203882+0000 mgr.y (mgr.24526) 292 : cluster [DBG] pgmap v321: 292 pgs: 32 creating+peering, 37 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 208 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: cluster 2024-09-10T18:56:20.678026+0000 osd.1 (osd.1) 159 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: cluster 2024-09-10T18:56:20.681238+0000 osd.1 (osd.1) 160 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: audit 2024-09-10T18:56:20.712892+0000 mon.c (mon.2) 336 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.011166+0000 mon.a (mon.0) 1549 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: audit 2024-09-10T18:56:21.020744+0000 mon.a (mon.0) 1550 : audit [INF] from='client.48875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: audit 2024-09-10T18:56:21.020966+0000 mon.a (mon.0) 1551 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.021434+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: audit 2024-09-10T18:56:21.030201+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: audit 2024-09-10T18:56:21.031070+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:21 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.203882+0000 mgr.y (mgr.24526) 292 : cluster [DBG] pgmap v321: 292 pgs: 32 creating+peering, 37 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 208 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:21.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:21.784 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: cluster 2024-09-10T18:56:20.678026+0000 osd.1 (osd.1) 159 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: cluster 2024-09-10T18:56:20.681238+0000 osd.1 (osd.1) 160 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: audit 2024-09-10T18:56:20.712892+0000 mon.c (mon.2) 336 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.011166+0000 mon.a (mon.0) 1549 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: audit 2024-09-10T18:56:21.020744+0000 mon.a (mon.0) 1550 : audit [INF] from='client.48875 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi083-77648-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: audit 2024-09-10T18:56:21.020966+0000 mon.a (mon.0) 1551 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-15"}]': finished 2024-09-10T18:56:21.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.021434+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-09-10T18:56:21.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: audit 2024-09-10T18:56:21.030201+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: audit 2024-09-10T18:56:21.031070+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:21.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:21 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.203882+0000 mgr.y (mgr.24526) 292 : cluster [DBG] pgmap v321: 292 pgs: 32 creating+peering, 37 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 208 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:21.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:21.920 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:21.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:22.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:22.055 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:22.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.188 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:22.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:22.329 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:22.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:22.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:22.588 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:22.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:22.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:22.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:22] "GET /metrics HTTP/1.1" 200 235882 "" "Prometheus/2.43.0" 2024-09-10T18:56:22.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.851 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:22.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.344449+0000 osd.5 (osd.5) 150 : cluster [DBG] 16.11 scrub starts 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.348978+0000 osd.5 (osd.5) 151 : cluster [DBG] 16.11 scrub ok 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.647619+0000 osd.1 (osd.1) 161 : cluster [DBG] 16.9 scrub starts 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.650508+0000 osd.1 (osd.1) 162 : cluster [DBG] 16.9 scrub ok 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: audit 2024-09-10T18:56:21.714703+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:22.015012+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: audit 2024-09-10T18:56:22.024019+0000 mon.a (mon.0) 1555 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: cluster 2024-09-10T18:56:22.024498+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: audit 2024-09-10T18:56:22.027363+0000 mon.c (mon.2) 339 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:22.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:22 smithi165 bash[22591]: audit 2024-09-10T18:56:22.478108+0000 mon.a (mon.0) 1557 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:22.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:22.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:23.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:23.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.344449+0000 osd.5 (osd.5) 150 : cluster [DBG] 16.11 scrub starts 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.348978+0000 osd.5 (osd.5) 151 : cluster [DBG] 16.11 scrub ok 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.647619+0000 osd.1 (osd.1) 161 : cluster [DBG] 16.9 scrub starts 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.650508+0000 osd.1 (osd.1) 162 : cluster [DBG] 16.9 scrub ok 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: audit 2024-09-10T18:56:21.714703+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:22.015012+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: audit 2024-09-10T18:56:22.024019+0000 mon.a (mon.0) 1555 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: cluster 2024-09-10T18:56:22.024498+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: audit 2024-09-10T18:56:22.027363+0000 mon.c (mon.2) 339 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[18629]: audit 2024-09-10T18:56:22.478108+0000 mon.a (mon.0) 1557 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.344449+0000 osd.5 (osd.5) 150 : cluster [DBG] 16.11 scrub starts 2024-09-10T18:56:23.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.348978+0000 osd.5 (osd.5) 151 : cluster [DBG] 16.11 scrub ok 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.647619+0000 osd.1 (osd.1) 161 : cluster [DBG] 16.9 scrub starts 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.650508+0000 osd.1 (osd.1) 162 : cluster [DBG] 16.9 scrub ok 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: audit 2024-09-10T18:56:21.714703+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:22.015012+0000 mon.a (mon.0) 1554 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: audit 2024-09-10T18:56:22.024019+0000 mon.a (mon.0) 1555 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: cluster 2024-09-10T18:56:22.024498+0000 mon.a (mon.0) 1556 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: audit 2024-09-10T18:56:22.027363+0000 mon.c (mon.2) 339 : audit [INF] from='client.? v1:172.21.15.83:0/2624174910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:23.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:22 smithi083 bash[32381]: audit 2024-09-10T18:56:22.478108+0000 mon.a (mon.0) 1557 : audit [INF] from='client.48869 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]: dispatch 2024-09-10T18:56:23.123 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:23.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.259 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:23.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:23.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:23.529 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:23.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.662 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:23.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:23.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:23.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.934 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:23.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:24.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:24.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.200 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.504693+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:21.507823+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:22.508511+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:22.716400+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:22.940780+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:22.945163+0000 osd.3 (osd.3) 178 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.044804+0000 mon.a (mon.0) 1558 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:23.045078+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.045349+0000 mon.b (mon.1) 184 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.047739+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.83:0/3829606491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.050294+0000 mon.a (mon.0) 1560 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.052157+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49399 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.201 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.073335+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.073642+0000 mon.a (mon.0) 1562 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.074817+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.075076+0000 mon.a (mon.0) 1563 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.075564+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.504693+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:21.507823+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:22.508511+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:22.716400+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:22.940780+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:22.945163+0000 osd.3 (osd.3) 178 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:56:24.202 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.044804+0000 mon.a (mon.0) 1558 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:23.045078+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.045349+0000 mon.b (mon.1) 184 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.047739+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.83:0/3829606491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.050294+0000 mon.a (mon.0) 1560 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.052157+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49399 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.073335+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.073642+0000 mon.a (mon.0) 1562 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.074817+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.075076+0000 mon.a (mon.0) 1563 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.075564+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.075841+0000 mon.a (mon.0) 1564 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.203 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: cluster 2024-09-10T18:56:23.204837+0000 mgr.y (mgr.24526) 293 : cluster [DBG] pgmap v324: 292 pgs: 64 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:24.204 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[32381]: audit 2024-09-10T18:56:23.717983+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.075841+0000 mon.a (mon.0) 1564 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: cluster 2024-09-10T18:56:23.204837+0000 mgr.y (mgr.24526) 293 : cluster [DBG] pgmap v324: 292 pgs: 64 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:24.204 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:23 smithi083 bash[18629]: audit 2024-09-10T18:56:23.717983+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:24.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.504693+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:21.507823+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:22.508511+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:22.716400+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:22.940780+0000 osd.3 (osd.3) 177 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:22.945163+0000 osd.3 (osd.3) 178 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.044804+0000 mon.a (mon.0) 1558 : audit [INF] from='client.48869 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi083-77640-28"}]': finished 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:23.045078+0000 mon.a (mon.0) 1559 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.045349+0000 mon.b (mon.1) 184 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.047739+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.83:0/3829606491' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.320 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.050294+0000 mon.a (mon.0) 1560 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.052157+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49399 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.073335+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.073642+0000 mon.a (mon.0) 1562 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.074817+0000 mon.c (mon.2) 342 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.075076+0000 mon.a (mon.0) 1563 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.075564+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.075841+0000 mon.a (mon.0) 1564 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: cluster 2024-09-10T18:56:23.204837+0000 mgr.y (mgr.24526) 293 : cluster [DBG] pgmap v324: 292 pgs: 64 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:56:24.321 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:23 smithi165 bash[22591]: audit 2024-09-10T18:56:23.717983+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:24.339 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:24.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:24.580 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:24.613 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:24.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:24.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:24.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:24.882 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:24.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:24.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:25.027 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:25.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:25.155 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:25.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- 172.21.15.83:0/3338269314 >> v1:172.21.15.83:6790/0 conn(0x7fa4b01075b0 legacy=0x7fa4b0109a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- 172.21.15.83:0/3338269314 shutdown_connections 2024-09-10T18:56:25.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- 172.21.15.83:0/3338269314 >> 172.21.15.83:0/3338269314 conn(0x7fa4b0074a20 msgr2=0x7fa4b00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:25.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- 172.21.15.83:0/3338269314 shutdown_connections 2024-09-10T18:56:25.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- 172.21.15.83:0/3338269314 wait complete. 2024-09-10T18:56:25.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 librados: starting msgr at 2024-09-10T18:56:25.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 librados: starting objecter 2024-09-10T18:56:25.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 Processor -- start 2024-09-10T18:56:25.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- start start 2024-09-10T18:56:25.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 librados: setting wanted keys 2024-09-10T18:56:25.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 librados: calling monclient init 2024-09-10T18:56:25.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4b0100620 con 0x7fa4b01075b0 2024-09-10T18:56:25.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4b011f740 con 0x7fa4b0100c70 2024-09-10T18:56:25.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b7f67640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4b01208e0 con 0x7fa4b0103a50 2024-09-10T18:56:25.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b5502640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa4b0103a50 0x7fa4b00fd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51774/0 (socket says 172.21.15.83:51774) 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4b5502640 1 -- 172.21.15.83:0/1146584415 learned_addr learned my addr 172.21.15.83:0/1146584415 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter ms_handle_connect 0x7fa4b0103a50 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951624266 0 0) 0x7fa4b01208e0 con 0x7fa4b0103a50 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c003720 con 0x7fa4b0103a50 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter ms_handle_connect 0x7fa4b0100c70 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter ms_handle_connect 0x7fa4b01075b0 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.173+0000 7fa4a67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208706192 0 0) 0x7fa4b011f740 con 0x7fa4b0100c70 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b01208e0 con 0x7fa4b0100c70 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234384534 0 0) 0x7fa49c003720 con 0x7fa4b0103a50 2024-09-10T18:56:25.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4b011f740 con 0x7fa4b0103a50 2024-09-10T18:56:25.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa4a0002950 con 0x7fa4b0103a50 2024-09-10T18:56:25.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269826060 0 0) 0x7fa4b01208e0 con 0x7fa4b0100c70 2024-09-10T18:56:25.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa49c003720 con 0x7fa4b0100c70 2024-09-10T18:56:25.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa4ac0017f0 con 0x7fa4b0100c70 2024-09-10T18:56:25.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 653268813 0 0) 0x7fa4b011f740 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.165:6789/0 conn(0x7fa4b0100c70 legacy=0x7fa4b00fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6789/0 conn(0x7fa4b01075b0 legacy=0x7fa4b011d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0121ae0 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b011f930 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4b7f67640 10 client.48893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b011fdc0 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4b7f67640 1 librados: init done 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa4a0002ce0 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa4a0005af0 con 0x7fa4b0103a50 2024-09-10T18:56:25.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa494004ff0 con 0x7fa4b0103a50 2024-09-10T18:56:25.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.177+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa4a001bf10 con 0x7fa4b0103a50 2024-09-10T18:56:25.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(229..229 src has 69..229) v4 ==== 8668+0+0 (unknown 1405589075 0 0) 0x7fa4a008a840 con 0x7fa4b0103a50 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 osd_map(229..229 src has 69..229) v4 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 3 client.48893.objecter handle_osd_map got epochs [229,229] > 0 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 3 client.48893.objecter handle_osd_map decoding full epoch 229 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 20 client.48893.objecter dump_active .. 0 homeless 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v229) v1 ==== 54+0+16 (unknown 3570346261 0 2420000192) 0x7fa4a008ac40 con 0x7fa4b0103a50 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa49c071570 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 1 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 1 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 0 1 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 4 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 10 client.48893.objecter _send_command 1 2024-09-10T18:56:25.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa494004a60 con 0x7fa494004610 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa494004610 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa494004a60 con 0x7fa494004610 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 1 = system:0 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 4 2024-09-10T18:56:25.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.181+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 0 1 2024-09-10T18:56:25.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 2 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 2 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494005290 osd=0 3 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 0 2 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 4 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 10 client.48893.objecter _send_command 2 2024-09-10T18:56:25.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.197+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa494004e40 con 0x7fa494004610 2024-09-10T18:56:25.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa494004e40 con 0x7fa494004610 2024-09-10T18:56:25.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 4 2024-09-10T18:56:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 0 2 2024-09-10T18:56:25.203 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:56:25.203 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 1 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 4 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 10 client.48893.objecter _send_command 3 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa494002370 con 0x7fa494004110 2024-09-10T18:56:25.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa494004110 2024-09-10T18:56:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa494002370 con 0x7fa494004110 2024-09-10T18:56:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:56:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 3 = system:0 2024-09-10T18:56:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 4 2024-09-10T18:56:25.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.201+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 1 3 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 4 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 4 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494003fa0 osd=1 3 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 1 4 2024-09-10T18:56:25.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 4 2024-09-10T18:56:25.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 10 client.48893.objecter _send_command 4 2024-09-10T18:56:25.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.217+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa4940018f0 con 0x7fa494004110 2024-09-10T18:56:25.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa4940018f0 con 0x7fa494004110 2024-09-10T18:56:25.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 4 2024-09-10T18:56:25.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 1 4 2024-09-10T18:56:25.223 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 5 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 5 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 2 5 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 4 2024-09-10T18:56:25.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 10 client.48893.objecter _send_command 5 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa494003d90 con 0x7fa49400d350 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa49400d350 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa494003d90 con 0x7fa49400d350 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 5 = system:0 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 4 2024-09-10T18:56:25.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.221+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 2 5 2024-09-10T18:56:25.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 6 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 6 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa4940035e0 osd=2 3 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 2 6 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 4 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 10 client.48893.objecter _send_command 6 2024-09-10T18:56:25.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.245+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa49400e6f0 con 0x7fa49400d350 2024-09-10T18:56:25.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa49400e6f0 con 0x7fa49400d350 2024-09-10T18:56:25.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 4 2024-09-10T18:56:25.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 2 6 2024-09-10T18:56:25.251 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:56:25.251 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 7 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 7 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 3 7 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 4 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 10 client.48893.objecter _send_command 7 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa494015ac0 con 0x7fa494011fc0 2024-09-10T18:56:25.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa494011fc0 2024-09-10T18:56:25.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa494015ac0 con 0x7fa494011fc0 2024-09-10T18:56:25.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:56:25.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 7 = system:0 2024-09-10T18:56:25.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 4 2024-09-10T18:56:25.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.249+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 3 7 2024-09-10T18:56:25.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 8 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 8 2024-09-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494003310 osd=3 3 2024-09-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 3 8 2024-09-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 4 2024-09-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 10 client.48893.objecter _send_command 8 2024-09-10T18:56:25.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa49400e170 con 0x7fa494011fc0 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa49400e170 con 0x7fa494011fc0 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.269+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 4 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 3 8 2024-09-10T18:56:25.274 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 9 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 9 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 4 9 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 4 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 10 client.48893.objecter _send_command 9 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa49400e0d0 con 0x7fa49400cea0 2024-09-10T18:56:25.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa49400cea0 2024-09-10T18:56:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa49400e0d0 con 0x7fa49400cea0 2024-09-10T18:56:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:56:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 9 = system:0 2024-09-10T18:56:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 4 2024-09-10T18:56:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.273+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 4 9 2024-09-10T18:56:25.289 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 10 2024-09-10T18:56:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 10 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49400df30 osd=4 3 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 4 10 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 4 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 10 client.48893.objecter _send_command 10 2024-09-10T18:56:25.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.293+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa49401a4e0 con 0x7fa49400cea0 2024-09-10T18:56:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa49401a4e0 con 0x7fa49400cea0 2024-09-10T18:56:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 4 2024-09-10T18:56:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 4 10 2024-09-10T18:56:25.299 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:56:25.299 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 11 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 11 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 5 11 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 4 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 10 client.48893.objecter _send_command 11 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa49401c750 con 0x7fa49401ab90 2024-09-10T18:56:25.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa49401ab90 2024-09-10T18:56:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa49401c750 con 0x7fa49401ab90 2024-09-10T18:56:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:56:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 11 = system:0 2024-09-10T18:56:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 4 2024-09-10T18:56:25.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.297+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 5 11 2024-09-10T18:56:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 12 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 12 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49401a4e0 osd=5 3 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 5 12 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 4 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 10 client.48893.objecter _send_command 12 2024-09-10T18:56:25.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.317+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa49401c750 con 0x7fa49401ab90 2024-09-10T18:56:25.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa49401c750 con 0x7fa49401ab90 2024-09-10T18:56:25.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 4 2024-09-10T18:56:25.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 5 12 2024-09-10T18:56:25.323 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:56:25.325 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:56:25.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 13 2024-09-10T18:56:25.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 13 2024-09-10T18:56:25.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 6 13 2024-09-10T18:56:25.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 4 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 10 client.48893.objecter _send_command 13 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa494022da0 con 0x7fa49401f2a0 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa49401f2a0 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa494022da0 con 0x7fa49401f2a0 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:56:25.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 13 = system:0 2024-09-10T18:56:25.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 4 2024-09-10T18:56:25.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.321+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 6 13 2024-09-10T18:56:25.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 14 2024-09-10T18:56:25.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 14 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa49401c750 osd=6 3 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 6 14 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 4 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 10 client.48893.objecter _send_command 14 2024-09-10T18:56:25.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.337+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa494022da0 con 0x7fa49401f2a0 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa494022da0 con 0x7fa49401f2a0 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 4 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 6 14 2024-09-10T18:56:25.343 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 15 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 15 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 7 15 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 4 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 10 client.48893.objecter _send_command 15 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4940270a0 con 0x7fa494023420 2024-09-10T18:56:25.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 10 client.48893.objecter ms_handle_connect 0x7fa494023420 2024-09-10T18:56:25.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa4940270a0 con 0x7fa494023420 2024-09-10T18:56:25.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:56:25.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 15 = system:0 2024-09-10T18:56:25.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.341+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 4 2024-09-10T18:56:25.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.345+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 7 15 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 10 client.48893.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign -1 16 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 20 client.48893.objecter _get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_remove -1 16 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 20 client.48893.objecter get_session s=0x7fa494022da0 osd=7 3 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 15 client.48893.objecter _session_command_op_assign 7 16 2024-09-10T18:56:25.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 4 2024-09-10T18:56:25.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 10 client.48893.objecter _send_command 16 2024-09-10T18:56:25.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.357+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa494027060 con 0x7fa494023420 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4a67fc640 1 -- 172.21.15.83:0/1146584415 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa494027060 con 0x7fa494023420 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4a67fc640 10 client.48893.objecter ms_dispatch 0x7fa4b0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4a67fc640 10 client.48893.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4a67fc640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 4 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4a67fc640 15 client.48893.objecter _session_command_op_remove 7 16 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:56:25.363 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:56:25.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 librados: watch_flush enter 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 librados: watch_flush exit 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.0 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6801/2535527483 conn(0x7fa494004610 legacy=0x7fa494002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494005290 osd=0 1 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.1 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6805/2309624637 conn(0x7fa494004110 legacy=0x7fa49400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003fa0 osd=1 1 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.2 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6809/386094220 conn(0x7fa49400d350 legacy=0x7fa49400d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa4940035e0 osd=2 1 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.3 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6813/1279688021 conn(0x7fa494011fc0 legacy=0x7fa4940143a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494003310 osd=3 1 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.4 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.165:6800/3225515265 conn(0x7fa49400cea0 legacy=0x7fa494018dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49400df30 osd=4 1 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.5 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.165:6804/571173777 conn(0x7fa49401ab90 legacy=0x7fa49401afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401a4e0 osd=5 1 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.6 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.165:6808/1317922936 conn(0x7fa49401f2a0 legacy=0x7fa494021680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa49401c750 osd=6 1 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter close_session for osd.7 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.165:6812/1610151650 conn(0x7fa494023420 legacy=0x7fa494025800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter put_session s=0x7fa494022da0 osd=7 1 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 20 client.48893.objecter shutdown clearing up homeless session... 2024-09-10T18:56:25.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 10 client.48893.objecter successfully canceled tick 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b6504640 1 -- 172.21.15.83:0/1146584415 reap_dead start 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b6504640 1 -- 172.21.15.83:0/1146584415 reap_dead start 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b6504640 1 -- 172.21.15.83:0/1146584415 reap_dead start 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b6504640 1 -- 172.21.15.83:0/1146584415 reap_dead start 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b6504640 1 -- 172.21.15.83:0/1146584415 reap_dead start 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa49c071570 legacy=0x7fa49c073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> v1:172.21.15.83:6790/0 conn(0x7fa4b0103a50 legacy=0x7fa4b00fd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 shutdown_connections 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 >> 172.21.15.83:0/1146584415 conn(0x7fa4b0074a20 msgr2=0x7fa4b010ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 shutdown_connections 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 -- 172.21.15.83:0/1146584415 wait complete. 2024-09-10T18:56:25.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:25.361+0000 7fa4b7f67640 1 librados: shutdown 2024-09-10T18:56:25.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:25.422 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:25.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: cluster 2024-09-10T18:56:23.955572+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: cluster 2024-09-10T18:56:23.959160+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.001196+0000 mon.a (mon.0) 1565 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.001599+0000 mon.a (mon.0) 1566 : audit [INF] from='client.49399 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.002055+0000 mon.a (mon.0) 1567 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: cluster 2024-09-10T18:56:24.002159+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.003956+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.005044+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.039351+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.057276+0000 mon.a (mon.0) 1570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[18629]: audit 2024-09-10T18:56:24.719577+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: cluster 2024-09-10T18:56:23.955572+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: cluster 2024-09-10T18:56:23.959160+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.001196+0000 mon.a (mon.0) 1565 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.001599+0000 mon.a (mon.0) 1566 : audit [INF] from='client.49399 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.002055+0000 mon.a (mon.0) 1567 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: cluster 2024-09-10T18:56:24.002159+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.003956+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.005044+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.039351+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.057276+0000 mon.a (mon.0) 1570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:25 smithi083 bash[32381]: audit 2024-09-10T18:56:24.719577+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:25.559 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:25.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: cluster 2024-09-10T18:56:23.955572+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: cluster 2024-09-10T18:56:23.959160+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.001196+0000 mon.a (mon.0) 1565 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.001599+0000 mon.a (mon.0) 1566 : audit [INF] from='client.49399 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi083-77648-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.002055+0000 mon.a (mon.0) 1567 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi083-77640-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:25.677 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: cluster 2024-09-10T18:56:24.002159+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-09-10T18:56:25.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.003956+0000 mon.c (mon.2) 345 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.005044+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:25.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.039351+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.057276+0000 mon.a (mon.0) 1570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:25.678 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:25 smithi165 bash[22591]: audit 2024-09-10T18:56:24.719577+0000 mon.c (mon.2) 346 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:25.697 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:25.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:25.838 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:25.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:25.974 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:25.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:26.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.101 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:26.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 -- 172.21.15.83:0/3787214902 >> v1:172.21.15.83:6789/0 conn(0x7f8d2c1077c0 legacy=0x7f8d2c109c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:26.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 -- 172.21.15.83:0/3787214902 shutdown_connections 2024-09-10T18:56:26.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 -- 172.21.15.83:0/3787214902 >> 172.21.15.83:0/3787214902 conn(0x7f8d2c074c70 msgr2=0x7f8d2c0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:26.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 -- 172.21.15.83:0/3787214902 shutdown_connections 2024-09-10T18:56:26.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 -- 172.21.15.83:0/3787214902 wait complete. 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 librados: starting msgr at 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.213+0000 7f8d30db4640 1 librados: starting objecter 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 Processor -- start 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- start start 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 librados: setting wanted keys 2024-09-10T18:56:26.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 librados: calling monclient init 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d2c11c550 con 0x7f8d2c1077c0 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d2c11d6f0 con 0x7f8d2c103c60 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d2c11e890 con 0x7f8d2c100e80 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d2a59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f8d2c100e80 0x7f8d2c0fec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51798/0 (socket says 172.21.15.83:51798) 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d2a59c640 1 -- 172.21.15.83:0/4109126974 learned_addr learned my addr 172.21.15.83:0/4109126974 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:26.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter ms_handle_connect 0x7f8d2c100e80 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter ms_handle_connect 0x7f8d2c1077c0 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter ms_handle_connect 0x7f8d2c103c60 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452717262 0 0) 0x7f8d2c11c550 con 0x7f8d2c1077c0 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d14003120 con 0x7f8d2c1077c0 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841030710 0 0) 0x7f8d2c11e890 con 0x7f8d2c100e80 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c11c550 con 0x7f8d2c100e80 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501808807 0 0) 0x7f8d2c11d6f0 con 0x7f8d2c103c60 2024-09-10T18:56:26.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d2c11e890 con 0x7f8d2c103c60 2024-09-10T18:56:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1022168299 0 0) 0x7f8d14003120 con 0x7f8d2c1077c0 2024-09-10T18:56:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d2c11d6f0 con 0x7f8d2c1077c0 2024-09-10T18:56:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990657594 0 0) 0x7f8d2c11c550 con 0x7f8d2c100e80 2024-09-10T18:56:26.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d14003120 con 0x7f8d2c100e80 2024-09-10T18:56:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8d24001950 con 0x7f8d2c1077c0 2024-09-10T18:56:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8d1c0028d0 con 0x7f8d2c100e80 2024-09-10T18:56:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165178345 0 0) 0x7f8d2c11d6f0 con 0x7f8d2c1077c0 2024-09-10T18:56:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 >> v1:172.21.15.83:6790/0 conn(0x7f8d2c100e80 legacy=0x7f8d2c0fec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:26.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 >> v1:172.21.15.165:6789/0 conn(0x7f8d2c103c60 legacy=0x7f8d2c0ff370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c11fa30 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c11c740 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 10 client.24882.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c11cb90 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d30db4640 1 librados: init done 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8d24001eb0 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.217+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8d24003970 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c0fc5f0 con 0x7f8d2c1077c0 2024-09-10T18:56:26.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f8d24002f70 con 0x7f8d2c1077c0 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(230..230 src has 69..230) v4 ==== 9453+0+0 (unknown 3671462042 0 0) 0x7f8d24088850 con 0x7f8d2c1077c0 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 10 client.24882.objecter ms_dispatch 0x7f8d2c000b90 osd_map(230..230 src has 69..230) v4 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 3 client.24882.objecter handle_osd_map got epochs [230,230] > 0 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 3 client.24882.objecter handle_osd_map decoding full epoch 230 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 20 client.24882.objecter dump_active .. 0 homeless 2024-09-10T18:56:26.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.221+0000 7f8d1affd640 10 client.24882.objecter ms_handle_connect 0x7f8d14071510 2024-09-10T18:56:26.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.229+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8d24054880 con 0x7f8d2c1077c0 2024-09-10T18:56:26.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:26.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.379 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:26.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:26.413+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8d2c064430 con 0x7f8d2c1077c0 2024-09-10T18:56:26.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:24.938417+0000 osd.3 (osd.3) 181 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:24.942058+0000 osd.3 (osd.3) 182 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:25.182252+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.83:0/1146584415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.183072+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.185880+0000 osd.0 (osd.0) 165 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.205458+0000 mgr.y (mgr.24526) 294 : cluster [DBG] pgmap v327: 260 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.265943+0000 osd.5 (osd.5) 152 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.269216+0000 osd.5 (osd.5) 153 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:56:26.496 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:25.457223+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:25.459358+0000 mon.a (mon.0) 1571 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:25.459734+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:25.464709+0000 mon.a (mon.0) 1573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:25.721080+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.013128+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.015353+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.015583+0000 mon.a (mon.0) 1575 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.015998+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.017980+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.83:0/1250914240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.018584+0000 mon.a (mon.0) 1577 : audit [INF] from='client.48899 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.019095+0000 mon.a (mon.0) 1578 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:26 smithi165 bash[22591]: audit 2024-09-10T18:56:26.418017+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:56:26.509 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:26.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:26.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:24.938417+0000 osd.3 (osd.3) 181 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:56:26.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:24.942058+0000 osd.3 (osd.3) 182 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:56:26.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:25.182252+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.83:0/1146584415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:26.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.183072+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:56:26.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.185880+0000 osd.0 (osd.0) 165 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.205458+0000 mgr.y (mgr.24526) 294 : cluster [DBG] pgmap v327: 260 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.265943+0000 osd.5 (osd.5) 152 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.269216+0000 osd.5 (osd.5) 153 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:25.457223+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:25.459358+0000 mon.a (mon.0) 1571 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:25.459734+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:25.464709+0000 mon.a (mon.0) 1573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:25.721080+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.013128+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.015353+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.015583+0000 mon.a (mon.0) 1575 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.015998+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:56:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.017980+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.83:0/1250914240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:24.938417+0000 osd.3 (osd.3) 181 : cluster [DBG] 2.1b scrub starts 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:24.942058+0000 osd.3 (osd.3) 182 : cluster [DBG] 2.1b scrub ok 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:25.182252+0000 mon.c (mon.2) 347 : audit [DBG] from='client.? v1:172.21.15.83:0/1146584415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.183072+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.16 scrub starts 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.185880+0000 osd.0 (osd.0) 165 : cluster [DBG] 6.16 scrub ok 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.205458+0000 mgr.y (mgr.24526) 294 : cluster [DBG] pgmap v327: 260 pgs: 32 unknown, 16 active+clean+snaptrim_wait, 15 active+clean+snaptrim, 197 active+clean; 588 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.265943+0000 osd.5 (osd.5) 152 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.269216+0000 osd.5 (osd.5) 153 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:56:26.771 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:25.457223+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:25.459358+0000 mon.a (mon.0) 1571 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:25.459734+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:25.464709+0000 mon.a (mon.0) 1573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:25.721080+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.013128+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.015353+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi083-77640-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.015583+0000 mon.a (mon.0) 1575 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.015998+0000 mon.a (mon.0) 1576 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.017980+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.83:0/1250914240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.772 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.018584+0000 mon.a (mon.0) 1577 : audit [INF] from='client.48899 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.019095+0000 mon.a (mon.0) 1578 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.773 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[32381]: audit 2024-09-10T18:56:26.418017+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:56:26.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.018584+0000 mon.a (mon.0) 1577 : audit [INF] from='client.48899 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:26.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.019095+0000 mon.a (mon.0) 1578 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]: dispatch 2024-09-10T18:56:26.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:26 smithi083 bash[18629]: audit 2024-09-10T18:56:26.418017+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:56:26.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:26.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:26.926 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:26.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:27.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.021+0000 7f8d1affd640 1 -- 172.21.15.83:0/4109126974 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v231) v1 ==== 101+0+0 (unknown 715160524 0 0) 0x7f8d24001550 con 0x7f8d2c1077c0 2024-09-10T18:56:27.025 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T18:56:27.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 10 librados: watch_flush enter 2024-09-10T18:56:27.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 10 librados: watch_flush exit 2024-09-10T18:56:27.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 20 client.24882.objecter shutdown clearing up homeless session... 2024-09-10T18:56:27.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 10 client.24882.objecter successfully canceled tick 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8d14071510 legacy=0x7f8d140739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 >> v1:172.21.15.83:6789/0 conn(0x7f8d2c1077c0 legacy=0x7f8d2c11ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 shutdown_connections 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 >> 172.21.15.83:0/4109126974 conn(0x7f8d2c074c70 msgr2=0x7f8d2c10a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 shutdown_connections 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 -- 172.21.15.83:0/4109126974 wait complete. 2024-09-10T18:56:27.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.025+0000 7f8d30db4640 1 librados: shutdown 2024-09-10T18:56:27.029 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:27.069 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:27.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:27.210 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:27.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:27.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:27.484 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:27.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:27.621 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:27.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.185431+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:56:27.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.190436+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:26.722527+0000 mon.c (mon.2) 350 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.888074+0000 osd.4 (osd.4) 189 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.891398+0000 osd.4 (osd.4) 190 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.009177+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.009260+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:27.019298+0000 mon.a (mon.0) 1582 : audit [INF] from='client.48899 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:27.019445+0000 mon.a (mon.0) 1583 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]': finished 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:27.020847+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.020930+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:27.144966+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: audit 2024-09-10T18:56:27.149766+0000 mon.a (mon.0) 1586 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.206719+0000 mgr.y (mgr.24526) 295 : cluster [DBG] pgmap v330: 300 pgs: 40 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.185431+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.190436+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:26.722527+0000 mon.c (mon.2) 350 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.888074+0000 osd.4 (osd.4) 189 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.891398+0000 osd.4 (osd.4) 190 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.009177+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:27.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.009260+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:27.019298+0000 mon.a (mon.0) 1582 : audit [INF] from='client.48899 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:27.019445+0000 mon.a (mon.0) 1583 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]': finished 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:27.020847+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.020930+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:27.144966+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: audit 2024-09-10T18:56:27.149766+0000 mon.a (mon.0) 1586 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:27 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.206719+0000 mgr.y (mgr.24526) 295 : cluster [DBG] pgmap v330: 300 pgs: 40 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:27.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:27.749 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.185431+0000 osd.0 (osd.0) 166 : cluster [DBG] 3.12 scrub starts 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.190436+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.12 scrub ok 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:26.722527+0000 mon.c (mon.2) 350 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.888074+0000 osd.4 (osd.4) 189 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.891398+0000 osd.4 (osd.4) 190 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.009177+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.009260+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:27.019298+0000 mon.a (mon.0) 1582 : audit [INF] from='client.48899 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi083-77648-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:27.019445+0000 mon.a (mon.0) 1583 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-17", "mode": "writeback"}]': finished 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:27.020847+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? v1:172.21.15.83:0/4109126974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.020930+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:27.144966+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: audit 2024-09-10T18:56:27.149766+0000 mon.a (mon.0) 1586 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:27.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.206719+0000 mgr.y (mgr.24526) 295 : cluster [DBG] pgmap v330: 300 pgs: 40 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:27.751 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:27.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.833+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2480460352 >> v1:172.21.15.83:6790/0 conn(0x7f02d41053a0 legacy=0x7f02d4107780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.833+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2480460352 shutdown_connections 2024-09-10T18:56:27.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.833+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2480460352 >> 172.21.15.83:0/2480460352 conn(0x7f02d4074a20 msgr2=0x7f02d40f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:27.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2480460352 shutdown_connections 2024-09-10T18:56:27.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2480460352 wait complete. 2024-09-10T18:56:27.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 librados: starting msgr at 2024-09-10T18:56:27.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 librados: starting objecter 2024-09-10T18:56:27.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 Processor -- start 2024-09-10T18:56:27.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- start start 2024-09-10T18:56:27.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 librados: setting wanted keys 2024-09-10T18:56:27.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 librados: calling monclient init 2024-09-10T18:56:27.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d4101dc0 con 0x7f02d4108f60 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d2ffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f02d41053a0 0x7f02d4119960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51822/0 (socket says 172.21.15.83:51822) 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d2ffd640 1 -- 172.21.15.83:0/2428558554 learned_addr learned my addr 172.21.15.83:0/2428558554 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f02d4108f60 0x7f02d411d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34050/0 (socket says 172.21.15.83:34050) 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d4101f70 con 0x7f02d4102670 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02d41207d0 con 0x7f02d41053a0 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter ms_handle_connect 0x7f02d4108f60 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter ms_handle_connect 0x7f02d41053a0 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter ms_handle_connect 0x7f02d4102670 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675684832 0 0) 0x7f02d41207d0 con 0x7f02d41053a0 2024-09-10T18:56:27.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c0003720 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835115850 0 0) 0x7f02d4101dc0 con 0x7f02d4108f60 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d41207d0 con 0x7f02d4108f60 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120784524 0 0) 0x7f02d4101f70 con 0x7f02d4102670 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d4101dc0 con 0x7f02d4102670 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851069553 0 0) 0x7f02c0003720 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d4101f70 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f02cc0017b0 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046925452 0 0) 0x7f02d4101f70 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 >> v1:172.21.15.165:6789/0 conn(0x7f02d4102670 legacy=0x7f02d4101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 >> v1:172.21.15.83:6789/0 conn(0x7f02d4108f60 legacy=0x7f02d411d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4121970 con 0x7f02d41053a0 2024-09-10T18:56:27.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.837+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d41209c0 con 0x7f02d41053a0 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02d9bbf640 10 client.48911.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d4120e90 con 0x7f02d41053a0 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02d9bbf640 1 librados: init done 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f02cc002780 con 0x7f02d41053a0 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f02cc003970 con 0x7f02d41053a0 2024-09-10T18:56:27.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b4005010 con 0x7f02d41053a0 2024-09-10T18:56:27.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f02cc003ad0 con 0x7f02d41053a0 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(231..231 src has 69..231) v4 ==== 9466+0+0 (unknown 152155911 0 0) 0x7f02cc08b150 con 0x7f02d41053a0 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 10 client.48911.objecter ms_dispatch 0x7f02d4000b90 osd_map(231..231 src has 69..231) v4 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 3 client.48911.objecter handle_osd_map got epochs [231,231] > 0 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.841+0000 7f02cbfff640 3 client.48911.objecter handle_osd_map decoding full epoch 231 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.845+0000 7f02cbfff640 20 client.48911.objecter dump_active .. 0 homeless 2024-09-10T18:56:27.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.845+0000 7f02cbfff640 10 client.48911.objecter ms_handle_connect 0x7f02c0071710 2024-09-10T18:56:27.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:27.849+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f02cc057150 con 0x7f02d41053a0 2024-09-10T18:56:27.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:27.878 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:27.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:27.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:28.015 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:28.017 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:28.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.037+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02b4000b90 con 0x7f02c0071710 2024-09-10T18:56:28.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.065+0000 7f02cbfff640 1 -- 172.21.15.83:0/2428558554 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+704410 (unknown 2965378022 0 2354284301) 0x7f02b4000b90 con 0x7f02c0071710 2024-09-10T18:56:28.081 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:56:28.086 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 10 librados: watch_flush enter 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 10 librados: watch_flush exit 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 20 client.48911.objecter shutdown clearing up homeless session... 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 10 client.48911.objecter successfully canceled tick 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 >> v1:172.21.15.83:6800/3071793735 conn(0x7f02c0071710 legacy=0x7f02c0073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 >> v1:172.21.15.83:6790/0 conn(0x7f02d41053a0 legacy=0x7f02d4119960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 shutdown_connections 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 >> 172.21.15.83:0/2428558554 conn(0x7f02d4074a20 msgr2=0x7f02d40f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 shutdown_connections 2024-09-10T18:56:28.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 -- 172.21.15.83:0/2428558554 wait complete. 2024-09-10T18:56:28.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:28.073+0000 7f02d9bbf640 1 librados: shutdown 2024-09-10T18:56:28.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:28.149 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:28.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:28.286 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:28.381 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:28.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:28.413 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:28.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:28.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.773936+0000 osd.2 (osd.2) 95 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: cluster 2024-09-10T18:56:26.777244+0000 osd.2 (osd.2) 96 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:27.724236+0000 mon.c (mon.2) 351 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.011742+0000 mon.a (mon.0) 1587 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:28.026407+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:28.028846+0000 mon.a (mon.0) 1588 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.028964+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:28.031973+0000 mon.a (mon.0) 1590 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:28.032009+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[18629]: audit 2024-09-10T18:56:28.032401+0000 mon.a (mon.0) 1591 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.773936+0000 osd.2 (osd.2) 95 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: cluster 2024-09-10T18:56:26.777244+0000 osd.2 (osd.2) 96 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:27.724236+0000 mon.c (mon.2) 351 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.011742+0000 mon.a (mon.0) 1587 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:28.026407+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:28.028846+0000 mon.a (mon.0) 1588 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.028964+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:28.031973+0000 mon.a (mon.0) 1590 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:28.032009+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:28 smithi083 bash[32381]: audit 2024-09-10T18:56:28.032401+0000 mon.a (mon.0) 1591 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.545 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.773936+0000 osd.2 (osd.2) 95 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: cluster 2024-09-10T18:56:26.777244+0000 osd.2 (osd.2) 96 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:27.724236+0000 mon.c (mon.2) 351 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.011742+0000 mon.a (mon.0) 1587 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:28.026407+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:28.028846+0000 mon.a (mon.0) 1588 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.028964+0000 mon.a (mon.0) 1589 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:28.031973+0000 mon.a (mon.0) 1590 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]: dispatch 2024-09-10T18:56:28.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:28.032009+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.548 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:28 smithi165 bash[22591]: audit 2024-09-10T18:56:28.032401+0000 mon.a (mon.0) 1591 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:28.649 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:28.681 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:28.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:28.816 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:28.920 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:28.952 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:28.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:29.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:29.079 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:29.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:29.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:29.235 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:56:29.271 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-10T18:56:28.023907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601262,"num_objects":491,"num_object_clones":171,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":1370,"num_read_kb":1140,"num_write":1163,"num_write_kb":2248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2147,"ondisk_log_size":2147,"up":780,"acting":780,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":764,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2454340,"kb_used_data":29676,"kb_used_omap":13,"kb_used_meta":2424498,"kb_avail":747506876,"statfs":{"total":767960285184,"available":765447041024,"internally_reserved":0,"allocated":30388224,"data_stored":15901400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2482686656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-416,"num_objects":-40,"num_object_clones":-41,"num_object_copies":-120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-40,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029602"},"pg_stats":[{"pgid":"141.1f","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.018944+0000","last_change":"2024-09-10T18:56:24.018165+0000","last_active":"2024-09-10T18:56:24.018944+0000","last_peered":"2024-09-10T18:56:24.018944+0000","last_clean":"2024-09-10T18:56:24.018944+0000","last_became_active":"2024-09-10T18:56:24.017888+0000","last_became_peered":"2024-09-10T18:56:24.017888+0000","last_unstale":"2024-09-10T18:56:24.018944+0000","last_undegraded":"2024-09-10T18:56:24.018944+0000","last_fullsized":"2024-09-10T18:56:24.018944+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:28.696592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"141.1e","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.030766+0000","last_change":"2024-09-10T18:56:24.029940+0000","last_active":"2024-09-10T18:56:24.030766+0000","last_peered":"2024-09-10T18:56:24.030766+0000","last_clean":"2024-09-10T18:56:24.030766+0000","last_became_active":"2024-09-10T18:56:24.029617+0000","last_became_peered":"2024-09-10T18:56:24.029617+0000","last_unstale":"2024-09-10T18:56:24.030766+0000","last_undegraded":"2024-09-10T18:56:24.030766+0000","last_fullsized":"2024-09-10T18:56:24.030766+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.734697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"141.1d","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.029017+0000","last_change":"2024-09-10T18:56:24.015367+0000","last_active":"2024-09-10T18:56:24.029017+0000","last_peered":"2024-09-10T18:56:24.029017+0000","last_clean":"2024-09-10T18:56:24.029017+0000","last_became_active":"2024-09-10T18:56:24.014976+0000","last_became_peered":"2024-09-10T18:56:24.014976+0000","last_unstale":"2024-09-10T18:56:24.029017+0000","last_undegraded":"2024-09-10T18:56:24.029017+0000","last_fullsized":"2024-09-10T18:56:24.029017+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:28.218699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.1c","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.054399+0000","last_change":"2024-09-10T18:56:24.054021+0000","last_active":"2024-09-10T18:56:24.054399+0000","last_peered":"2024-09-10T18:56:24.054399+0000","last_clean":"2024-09-10T18:56:24.054399+0000","last_became_active":"2024-09-10T18:56:24.053627+0000","last_became_peered":"2024-09-10T18:56:24.053627+0000","last_unstale":"2024-09-10T18:56:24.054399+0000","last_undegraded":"2024-09-10T18:56:24.054399+0000","last_fullsized":"2024-09-10T18:56:24.054399+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:38.672183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"141.1b","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017755+0000","last_change":"2024-09-10T18:56:24.015777+0000","last_active":"2024-09-10T18:56:24.017755+0000","last_peered":"2024-09-10T18:56:24.017755+0000","last_clean":"2024-09-10T18:56:24.017755+0000","last_became_active":"2024-09-10T18:56:24.015460+0000","last_became_peered":"2024-09-10T18:56:24.015460+0000","last_unstale":"2024-09-10T18:56:24.017755+0000","last_undegraded":"2024-09-10T18:56:24.017755+0000","last_fullsized":"2024-09-10T18:56:24.017755+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.099714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"141.1a","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.036698+0000","last_change":"2024-09-10T18:56:24.036008+0000","last_active":"2024-09-10T18:56:24.036698+0000","last_peered":"2024-09-10T18:56:24.036698+0000","last_clean":"2024-09-10T18:56:24.036698+0000","last_became_active":"2024-09-10T18:56:24.035555+0000","last_became_peered":"2024-09-10T18:56:24.035555+0000","last_unstale":"2024-09-10T18:56:24.036698+0000","last_undegraded":"2024-09-10T18:56:24.036698+0000","last_fullsized":"2024-09-10T18:56:24.036698+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.801017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"141.19","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.056841+0000","last_change":"2024-09-10T18:56:24.038353+0000","last_active":"2024-09-10T18:56:24.056841+0000","last_peered":"2024-09-10T18:56:24.056841+0000","last_clean":"2024-09-10T18:56:24.056841+0000","last_became_active":"2024-09-10T18:56:24.038025+0000","last_became_peered":"2024-09-10T18:56:24.038025+0000","last_unstale":"2024-09-10T18:56:24.056841+0000","last_undegraded":"2024-09-10T18:56:24.056841+0000","last_fullsized":"2024-09-10T18:56:24.056841+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.187105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"141.18","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.028783+0000","last_change":"2024-09-10T18:56:24.026971+0000","last_active":"2024-09-10T18:56:24.028783+0000","last_peered":"2024-09-10T18:56:24.028783+0000","last_clean":"2024-09-10T18:56:24.028783+0000","last_became_active":"2024-09-10T18:56:24.026621+0000","last_became_peered":"2024-09-10T18:56:24.026621+0000","last_unstale":"2024-09-10T18:56:24.028783+0000","last_undegraded":"2024-09-10T18:56:24.028783+0000","last_fullsized":"2024-09-10T18:56:24.028783+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.353370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"141.17","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.039049+0000","last_change":"2024-09-10T18:56:24.034013+0000","last_active":"2024-09-10T18:56:24.039049+0000","last_peered":"2024-09-10T18:56:24.039049+0000","last_clean":"2024-09-10T18:56:24.039049+0000","last_became_active":"2024-09-10T18:56:24.033720+0000","last_became_peered":"2024-09-10T18:56:24.033720+0000","last_unstale":"2024-09-10T18:56:24.039049+0000","last_undegraded":"2024-09-10T18:56:24.039049+0000","last_fullsized":"2024-09-10T18:56:24.039049+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.644318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"141.16","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.035775+0000","last_change":"2024-09-10T18:56:24.033807+0000","last_active":"2024-09-10T18:56:24.035775+0000","last_peered":"2024-09-10T18:56:24.035775+0000","last_clean":"2024-09-10T18:56:24.035775+0000","last_became_active":"2024-09-10T18:56:24.033574+0000","last_became_peered":"2024-09-10T18:56:24.033574+0000","last_unstale":"2024-09-10T18:56:24.035775+0000","last_undegraded":"2024-09-10T18:56:24.035775+0000","last_fullsized":"2024-09-10T18:56:24.035775+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.421426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"141.15","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.055676+0000","last_change":"2024-09-10T18:56:24.040465+0000","last_active":"2024-09-10T18:56:24.055676+0000","last_peered":"2024-09-10T18:56:24.055676+0000","last_clean":"2024-09-10T18:56:24.055676+0000","last_became_active":"2024-09-10T18:56:24.040203+0000","last_became_peered":"2024-09-10T18:56:24.040203+0000","last_unstale":"2024-09-10T18:56:24.055676+0000","last_undegraded":"2024-09-10T18:56:24.055676+0000","last_fullsized":"2024-09-10T18:56:24.055676+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"141.14","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.039552+0000","last_change":"2024-09-10T18:56:24.034286+0000","last_active":"2024-09-10T18:56:24.039552+0000","last_peered":"2024-09-10T18:56:24.039552+0000","last_clean":"2024-09-10T18:56:24.039552+0000","last_became_active":"2024-09-10T18:56:24.034060+0000","last_became_peered":"2024-09-10T18:56:24.034060+0000","last_unstale":"2024-09-10T18:56:24.039552+0000","last_undegraded":"2024-09-10T18:56:24.039552+0000","last_fullsized":"2024-09-10T18:56:24.039552+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.457463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.13","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.026762+0000","last_change":"2024-09-10T18:56:24.024220+0000","last_active":"2024-09-10T18:56:24.026762+0000","last_peered":"2024-09-10T18:56:24.026762+0000","last_clean":"2024-09-10T18:56:24.026762+0000","last_became_active":"2024-09-10T18:56:24.023832+0000","last_became_peered":"2024-09-10T18:56:24.023832+0000","last_unstale":"2024-09-10T18:56:24.026762+0000","last_undegraded":"2024-09-10T18:56:24.026762+0000","last_fullsized":"2024-09-10T18:56:24.026762+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.551549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"141.12","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.040003+0000","last_change":"2024-09-10T18:56:24.033069+0000","last_active":"2024-09-10T18:56:24.040003+0000","last_peered":"2024-09-10T18:56:24.040003+0000","last_clean":"2024-09-10T18:56:24.040003+0000","last_became_active":"2024-09-10T18:56:24.032738+0000","last_became_peered":"2024-09-10T18:56:24.032738+0000","last_unstale":"2024-09-10T18:56:24.040003+0000","last_undegraded":"2024-09-10T18:56:24.040003+0000","last_fullsized":"2024-09-10T18:56:24.040003+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:24.577159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"141.11","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.054725+0000","last_change":"2024-09-10T18:56:24.054253+0000","last_active":"2024-09-10T18:56:24.054725+0000","last_peered":"2024-09-10T18:56:24.054725+0000","last_clean":"2024-09-10T18:56:24.054725+0000","last_became_active":"2024-09-10T18:56:24.053888+0000","last_became_peered":"2024-09-10T18:56:24.053888+0000","last_unstale":"2024-09-10T18:56:24.054725+0000","last_undegraded":"2024-09-10T18:56:24.054725+0000","last_fullsized":"2024-09-10T18:56:24.054725+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:49.816740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"141.10","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.058997+0000","last_change":"2024-09-10T18:56:24.044847+0000","last_active":"2024-09-10T18:56:24.058997+0000","last_peered":"2024-09-10T18:56:24.058997+0000","last_clean":"2024-09-10T18:56:24.058997+0000","last_became_active":"2024-09-10T18:56:24.044040+0000","last_became_peered":"2024-09-10T18:56:24.044040+0000","last_unstale":"2024-09-10T18:56:24.058997+0000","last_undegraded":"2024-09-10T18:56:24.058997+0000","last_fullsized":"2024-09-10T18:56:24.058997+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.619020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"141.f","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.039894+0000","last_change":"2024-09-10T18:56:24.033174+0000","last_active":"2024-09-10T18:56:24.039894+0000","last_peered":"2024-09-10T18:56:24.039894+0000","last_clean":"2024-09-10T18:56:24.039894+0000","last_became_active":"2024-09-10T18:56:24.032775+0000","last_became_peered":"2024-09-10T18:56:24.032775+0000","last_unstale":"2024-09-10T18:56:24.039894+0000","last_undegraded":"2024-09-10T18:56:24.039894+0000","last_fullsized":"2024-09-10T18:56:24.039894+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.500639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"141.e","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.039646+0000","last_change":"2024-09-10T18:56:24.034909+0000","last_active":"2024-09-10T18:56:24.039646+0000","last_peered":"2024-09-10T18:56:24.039646+0000","last_clean":"2024-09-10T18:56:24.039646+0000","last_became_active":"2024-09-10T18:56:24.034637+0000","last_became_peered":"2024-09-10T18:56:24.034637+0000","last_unstale":"2024-09-10T18:56:24.039646+0000","last_undegraded":"2024-09-10T18:56:24.039646+0000","last_fullsized":"2024-09-10T18:56:24.039646+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.342554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.d","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.026760+0000","last_change":"2024-09-10T18:56:24.024108+0000","last_active":"2024-09-10T18:56:24.026760+0000","last_peered":"2024-09-10T18:56:24.026760+0000","last_clean":"2024-09-10T18:56:24.026760+0000","last_became_active":"2024-09-10T18:56:24.023842+0000","last_became_peered":"2024-09-10T18:56:24.023842+0000","last_unstale":"2024-09-10T18:56:24.026760+0000","last_undegraded":"2024-09-10T18:56:24.026760+0000","last_fullsized":"2024-09-10T18:56:24.026760+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.353026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"141.c","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.056004+0000","last_change":"2024-09-10T18:56:24.038494+0000","last_active":"2024-09-10T18:56:24.056004+0000","last_peered":"2024-09-10T18:56:24.056004+0000","last_clean":"2024-09-10T18:56:24.056004+0000","last_became_active":"2024-09-10T18:56:24.038228+0000","last_became_peered":"2024-09-10T18:56:24.038228+0000","last_unstale":"2024-09-10T18:56:24.056004+0000","last_undegraded":"2024-09-10T18:56:24.056004+0000","last_fullsized":"2024-09-10T18:56:24.056004+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:24.882537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"141.b","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.027739+0000","last_change":"2024-09-10T18:56:24.026452+0000","last_active":"2024-09-10T18:56:24.027739+0000","last_peered":"2024-09-10T18:56:24.027739+0000","last_clean":"2024-09-10T18:56:24.027739+0000","last_became_active":"2024-09-10T18:56:24.024683+0000","last_became_peered":"2024-09-10T18:56:24.024683+0000","last_unstale":"2024-09-10T18:56:24.027739+0000","last_undegraded":"2024-09-10T18:56:24.027739+0000","last_fullsized":"2024-09-10T18:56:24.027739+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.050943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"141.a","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.028571+0000","last_change":"2024-09-10T18:56:24.025922+0000","last_active":"2024-09-10T18:56:24.028571+0000","last_peered":"2024-09-10T18:56:24.028571+0000","last_clean":"2024-09-10T18:56:24.028571+0000","last_became_active":"2024-09-10T18:56:24.025078+0000","last_became_peered":"2024-09-10T18:56:24.025078+0000","last_unstale":"2024-09-10T18:56:24.028571+0000","last_undegraded":"2024-09-10T18:56:24.028571+0000","last_fullsized":"2024-09-10T18:56:24.028571+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.115789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"141.9","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.038894+0000","last_change":"2024-09-10T18:56:24.030057+0000","last_active":"2024-09-10T18:56:24.038894+0000","last_peered":"2024-09-10T18:56:24.038894+0000","last_clean":"2024-09-10T18:56:24.038894+0000","last_became_active":"2024-09-10T18:56:24.028646+0000","last_became_peered":"2024-09-10T18:56:24.028646+0000","last_unstale":"2024-09-10T18:56:24.038894+0000","last_undegraded":"2024-09-10T18:56:24.038894+0000","last_fullsized":"2024-09-10T18:56:24.038894+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:49.466438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"141.8","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.026837+0000","last_change":"2024-09-10T18:56:24.024783+0000","last_active":"2024-09-10T18:56:24.026837+0000","last_peered":"2024-09-10T18:56:24.026837+0000","last_clean":"2024-09-10T18:56:24.026837+0000","last_became_active":"2024-09-10T18:56:24.023283+0000","last_became_peered":"2024-09-10T18:56:24.023283+0000","last_unstale":"2024-09-10T18:56:24.026837+0000","last_undegraded":"2024-09-10T18:56:24.026837+0000","last_fullsized":"2024-09-10T18:56:24.026837+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.930663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"141.7","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.061930+0000","last_change":"2024-09-10T18:56:24.057066+0000","last_active":"2024-09-10T18:56:24.061930+0000","last_peered":"2024-09-10T18:56:24.061930+0000","last_clean":"2024-09-10T18:56:24.061930+0000","last_became_active":"2024-09-10T18:56:24.056728+0000","last_became_peered":"2024-09-10T18:56:24.056728+0000","last_unstale":"2024-09-10T18:56:24.061930+0000","last_undegraded":"2024-09-10T18:56:24.061930+0000","last_fullsized":"2024-09-10T18:56:24.061930+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.212391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"141.6","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.030441+0000","last_change":"2024-09-10T18:56:24.029864+0000","last_active":"2024-09-10T18:56:24.030441+0000","last_peered":"2024-09-10T18:56:24.030441+0000","last_clean":"2024-09-10T18:56:24.030441+0000","last_became_active":"2024-09-10T18:56:24.029598+0000","last_became_peered":"2024-09-10T18:56:24.029598+0000","last_unstale":"2024-09-10T18:56:24.030441+0000","last_undegraded":"2024-09-10T18:56:24.030441+0000","last_fullsized":"2024-09-10T18:56:24.030441+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.723737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"141.5","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.031264+0000","last_change":"2024-09-10T18:56:24.022982+0000","last_active":"2024-09-10T18:56:24.031264+0000","last_peered":"2024-09-10T18:56:24.031264+0000","last_clean":"2024-09-10T18:56:24.031264+0000","last_became_active":"2024-09-10T18:56:24.022704+0000","last_became_peered":"2024-09-10T18:56:24.022704+0000","last_unstale":"2024-09-10T18:56:24.031264+0000","last_undegraded":"2024-09-10T18:56:24.031264+0000","last_fullsized":"2024-09-10T18:56:24.031264+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.219694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.4","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.039098+0000","last_change":"2024-09-10T18:56:24.035881+0000","last_active":"2024-09-10T18:56:24.039098+0000","last_peered":"2024-09-10T18:56:24.039098+0000","last_clean":"2024-09-10T18:56:24.039098+0000","last_became_active":"2024-09-10T18:56:24.035570+0000","last_became_peered":"2024-09-10T18:56:24.035570+0000","last_unstale":"2024-09-10T18:56:24.039098+0000","last_undegraded":"2024-09-10T18:56:24.039098+0000","last_fullsized":"2024-09-10T18:56:24.039098+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.088059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"141.3","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.028061+0000","last_change":"2024-09-10T18:56:24.014123+0000","last_active":"2024-09-10T18:56:24.028061+0000","last_peered":"2024-09-10T18:56:24.028061+0000","last_clean":"2024-09-10T18:56:24.028061+0000","last_became_active":"2024-09-10T18:56:24.013755+0000","last_became_peered":"2024-09-10T18:56:24.013755+0000","last_unstale":"2024-09-10T18:56:24.028061+0000","last_undegraded":"2024-09-10T18:56:24.028061+0000","last_fullsized":"2024-09-10T18:56:24.028061+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.448013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.2","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.056084+0000","last_change":"2024-09-10T18:56:24.038735+0000","last_active":"2024-09-10T18:56:24.056084+0000","last_peered":"2024-09-10T18:56:24.056084+0000","last_clean":"2024-09-10T18:56:24.056084+0000","last_became_active":"2024-09-10T18:56:24.032440+0000","last_became_peered":"2024-09-10T18:56:24.032440+0000","last_unstale":"2024-09-10T18:56:24.056084+0000","last_undegraded":"2024-09-10T18:56:24.056084+0000","last_fullsized":"2024-09-10T18:56:24.056084+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.340640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"141.1","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.028292+0000","last_change":"2024-09-10T18:56:24.018102+0000","last_active":"2024-09-10T18:56:24.028292+0000","last_peered":"2024-09-10T18:56:24.028292+0000","last_clean":"2024-09-10T18:56:24.028292+0000","last_became_active":"2024-09-10T18:56:24.017593+0000","last_became_peered":"2024-09-10T18:56:24.017593+0000","last_unstale":"2024-09-10T18:56:24.028292+0000","last_undegraded":"2024-09-10T18:56:24.028292+0000","last_fullsized":"2024-09-10T18:56:24.028292+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.125092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"141.0","version":"0'0","reported_seq":10,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.058959+0000","last_change":"2024-09-10T18:56:24.038871+0000","last_active":"2024-09-10T18:56:24.058959+0000","last_peered":"2024-09-10T18:56:24.058959+0000","last_clean":"2024-09-10T18:56:24.058959+0000","last_became_active":"2024-09-10T18:56:24.037734+0000","last_became_peered":"2024-09-10T18:56:24.037734+0000","last_unstale":"2024-09-10T18:56:24.058959+0000","last_undegraded":"2024-09-10T18:56:24.058959+0000","last_fullsized":"2024-09-10T18:56:24.058959+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:23.033176+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.033176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.057332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":150,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023666+0000","last_change":"2024-09-10T18:56:21.043075+0000","last_active":"2024-09-10T18:56:24.023666+0000","last_peered":"2024-09-10T18:56:24.023666+0000","last_clean":"2024-09-10T18:56:24.023666+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:56:24.023666+0000","last_undegraded":"2024-09-10T18:56:24.023666+0000","last_fullsized":"2024-09-10T18:56:24.023666+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015847356,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.1e","version":"226'51","reported_seq":163,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.447792+0000","last_change":"2024-09-10T18:56:24.447792+0000","last_active":"2024-09-10T18:56:24.447792+0000","last_peered":"2024-09-10T18:56:24.447792+0000","last_clean":"2024-09-10T18:56:24.447792+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:56:24.447792+0000","last_undegraded":"2024-09-10T18:56:24.447792+0000","last_fullsized":"2024-09-10T18:56:24.447792+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42490645199999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.1d","version":"226'15","reported_seq":129,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.521757+0000","last_change":"2024-09-10T18:56:24.521757+0000","last_active":"2024-09-10T18:56:24.521757+0000","last_peered":"2024-09-10T18:56:24.521757+0000","last_clean":"2024-09-10T18:56:24.521757+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:56:24.521757+0000","last_undegraded":"2024-09-10T18:56:24.521757+0000","last_fullsized":"2024-09-10T18:56:24.521757+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52334676700000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"64.1c","version":"224'39","reported_seq":137,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.016336+0000","last_change":"2024-09-10T18:56:22.515990+0000","last_active":"2024-09-10T18:56:24.016336+0000","last_peered":"2024-09-10T18:56:24.016336+0000","last_clean":"2024-09-10T18:56:24.016336+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:56:24.016336+0000","last_undegraded":"2024-09-10T18:56:24.016336+0000","last_fullsized":"2024-09-10T18:56:24.016336+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.490148316,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"64.1b","version":"224'52","reported_seq":150,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.028547+0000","last_change":"2024-09-10T18:56:22.510703+0000","last_active":"2024-09-10T18:56:24.028547+0000","last_peered":"2024-09-10T18:56:24.028547+0000","last_clean":"2024-09-10T18:56:24.028547+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:56:24.028547+0000","last_undegraded":"2024-09-10T18:56:24.028547+0000","last_fullsized":"2024-09-10T18:56:24.028547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.475657553,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":143,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.527874+0000","last_change":"2024-09-10T18:56:24.521848+0000","last_active":"2024-09-10T18:56:24.527874+0000","last_peered":"2024-09-10T18:56:24.527874+0000","last_clean":"2024-09-10T18:56:24.527874+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:56:24.527874+0000","last_undegraded":"2024-09-10T18:56:24.527874+0000","last_fullsized":"2024-09-10T18:56:24.527874+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47216588399999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"64.19","version":"224'17","reported_seq":137,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003770+0000","last_change":"2024-09-10T18:56:22.445191+0000","last_active":"2024-09-10T18:56:24.003770+0000","last_peered":"2024-09-10T18:56:24.003770+0000","last_clean":"2024-09-10T18:56:24.003770+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:56:24.003770+0000","last_undegraded":"2024-09-10T18:56:24.003770+0000","last_fullsized":"2024-09-10T18:56:24.003770+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41938524700000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.18","version":"228'78","reported_seq":170,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.416504+0000","last_change":"2024-09-10T18:56:20.405508+0000","last_active":"2024-09-10T18:56:24.416504+0000","last_peered":"2024-09-10T18:56:24.416504+0000","last_clean":"2024-09-10T18:56:24.416504+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:56:24.416504+0000","last_undegraded":"2024-09-10T18:56:24.416504+0000","last_fullsized":"2024-09-10T18:56:24.416504+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015499471000000001,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"64.17","version":"224'15","reported_seq":124,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.018794+0000","last_change":"2024-09-10T18:56:22.431917+0000","last_active":"2024-09-10T18:56:24.018794+0000","last_peered":"2024-09-10T18:56:24.018794+0000","last_clean":"2024-09-10T18:56:24.018794+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:56:24.018794+0000","last_undegraded":"2024-09-10T18:56:24.018794+0000","last_fullsized":"2024-09-10T18:56:24.018794+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.393125441,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"64.16","version":"228'15","reported_seq":124,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.531850+0000","last_change":"2024-09-10T18:56:24.527577+0000","last_active":"2024-09-10T18:56:24.531850+0000","last_peered":"2024-09-10T18:56:24.531850+0000","last_clean":"2024-09-10T18:56:24.531850+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:56:24.531850+0000","last_undegraded":"2024-09-10T18:56:24.531850+0000","last_fullsized":"2024-09-10T18:56:24.531850+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38899260000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"64.15","version":"228'61","reported_seq":161,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.536575+0000","last_change":"2024-09-10T18:56:24.532281+0000","last_active":"2024-09-10T18:56:24.536575+0000","last_peered":"2024-09-10T18:56:24.536575+0000","last_clean":"2024-09-10T18:56:24.536575+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:56:24.536575+0000","last_undegraded":"2024-09-10T18:56:24.536575+0000","last_fullsized":"2024-09-10T18:56:24.536575+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.48352263899999998,"stat_sum":{"num_bytes":115,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"64.14","version":"228'73","reported_seq":177,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.534874+0000","last_change":"2024-09-10T18:56:22.515926+0000","last_active":"2024-09-10T18:56:24.534874+0000","last_peered":"2024-09-10T18:56:24.534874+0000","last_clean":"2024-09-10T18:56:24.534874+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:56:24.534874+0000","last_undegraded":"2024-09-10T18:56:24.534874+0000","last_fullsized":"2024-09-10T18:56:24.534874+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.374902071,"stat_sum":{"num_bytes":181,"num_objects":17,"num_object_clones":11,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.13","version":"146'1","reported_seq":115,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:23.999723+0000","last_change":"2024-09-10T18:56:22.516505+0000","last_active":"2024-09-10T18:56:23.999723+0000","last_peered":"2024-09-10T18:56:23.999723+0000","last_clean":"2024-09-10T18:56:23.999723+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:56:23.999723+0000","last_undegraded":"2024-09-10T18:56:23.999723+0000","last_fullsized":"2024-09-10T18:56:23.999723+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003303276,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"64.12","version":"224'39","reported_seq":137,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.022904+0000","last_change":"2024-09-10T18:56:22.434244+0000","last_active":"2024-09-10T18:56:24.022904+0000","last_peered":"2024-09-10T18:56:24.022904+0000","last_clean":"2024-09-10T18:56:24.022904+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:56:24.022904+0000","last_undegraded":"2024-09-10T18:56:24.022904+0000","last_fullsized":"2024-09-10T18:56:24.022904+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018308125000000001,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"64.11","version":"226'51","reported_seq":164,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.435208+0000","last_change":"2024-09-10T18:56:24.435208+0000","last_active":"2024-09-10T18:56:24.435208+0000","last_peered":"2024-09-10T18:56:24.435208+0000","last_clean":"2024-09-10T18:56:24.435208+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:56:24.435208+0000","last_undegraded":"2024-09-10T18:56:24.435208+0000","last_fullsized":"2024-09-10T18:56:24.435208+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42675023899999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.10","version":"224'34","reported_seq":146,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003138+0000","last_change":"2024-09-10T18:56:22.445588+0000","last_active":"2024-09-10T18:56:24.003138+0000","last_peered":"2024-09-10T18:56:24.003138+0000","last_clean":"2024-09-10T18:56:24.003138+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:56:24.003138+0000","last_undegraded":"2024-09-10T18:56:24.003138+0000","last_fullsized":"2024-09-10T18:56:24.003138+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.422252707,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.f","version":"228'34","reported_seq":137,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.038720+0000","last_change":"2024-09-10T18:56:16.421708+0000","last_active":"2024-09-10T18:56:24.038720+0000","last_peered":"2024-09-10T18:56:24.038720+0000","last_clean":"2024-09-10T18:56:24.038720+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:56:24.038720+0000","last_undegraded":"2024-09-10T18:56:24.038720+0000","last_fullsized":"2024-09-10T18:56:24.038720+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0093610889999999995,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.e","version":"228'91","reported_seq":182,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.507000+0000","last_change":"2024-09-10T18:56:18.491572+0000","last_active":"2024-09-10T18:56:24.507000+0000","last_peered":"2024-09-10T18:56:24.507000+0000","last_clean":"2024-09-10T18:56:24.507000+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:56:24.507000+0000","last_undegraded":"2024-09-10T18:56:24.507000+0000","last_fullsized":"2024-09-10T18:56:24.507000+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.4054166299999999,"stat_sum":{"num_bytes":231,"num_objects":21,"num_object_clones":14,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"64.d","version":"220'17","reported_seq":138,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008278+0000","last_change":"2024-09-10T18:56:19.035413+0000","last_active":"2024-09-10T18:56:24.008278+0000","last_peered":"2024-09-10T18:56:24.008278+0000","last_clean":"2024-09-10T18:56:24.008278+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:56:24.008278+0000","last_undegraded":"2024-09-10T18:56:24.008278+0000","last_fullsized":"2024-09-10T18:56:24.008278+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010244544,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"},{"start":"c","length":"1"}]},{"pgid":"64.c","version":"220'16","reported_seq":123,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.031407+0000","last_change":"2024-09-10T18:56:18.426152+0000","last_active":"2024-09-10T18:56:24.031407+0000","last_peered":"2024-09-10T18:56:24.031407+0000","last_clean":"2024-09-10T18:56:24.031407+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:56:24.031407+0000","last_undegraded":"2024-09-10T18:56:24.031407+0000","last_fullsized":"2024-09-10T18:56:24.031407+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42065873399999998,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.b","version":"225'30","reported_seq":138,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.527470+0000","last_change":"2024-09-10T18:56:24.527470+0000","last_active":"2024-09-10T18:56:24.527470+0000","last_peered":"2024-09-10T18:56:24.527470+0000","last_clean":"2024-09-10T18:56:24.527470+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:56:24.527470+0000","last_undegraded":"2024-09-10T18:56:24.527470+0000","last_fullsized":"2024-09-10T18:56:24.527470+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49870747100000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"64.a","version":"226'15","reported_seq":128,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.531171+0000","last_change":"2024-09-10T18:56:24.531171+0000","last_active":"2024-09-10T18:56:24.531171+0000","last_peered":"2024-09-10T18:56:24.531171+0000","last_clean":"2024-09-10T18:56:24.531171+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:56:24.531171+0000","last_undegraded":"2024-09-10T18:56:24.531171+0000","last_fullsized":"2024-09-10T18:56:24.531171+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53279201799999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.532067+0000","last_change":"2024-09-10T18:56:24.532067+0000","last_active":"2024-09-10T18:56:24.532067+0000","last_peered":"2024-09-10T18:56:24.532067+0000","last_clean":"2024-09-10T18:56:24.532067+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:56:24.532067+0000","last_undegraded":"2024-09-10T18:56:24.532067+0000","last_fullsized":"2024-09-10T18:56:24.532067+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53048183599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"64.8","version":"228'45","reported_seq":149,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.438101+0000","last_change":"2024-09-10T18:56:22.431952+0000","last_active":"2024-09-10T18:56:24.438101+0000","last_peered":"2024-09-10T18:56:24.438101+0000","last_clean":"2024-09-10T18:56:24.438101+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:56:24.438101+0000","last_undegraded":"2024-09-10T18:56:24.438101+0000","last_fullsized":"2024-09-10T18:56:24.438101+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.39501242800000003,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"64.7","version":"228'103","reported_seq":204,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.536194+0000","last_change":"2024-09-10T18:56:24.532732+0000","last_active":"2024-09-10T18:56:24.536194+0000","last_peered":"2024-09-10T18:56:24.536194+0000","last_clean":"2024-09-10T18:56:24.536194+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:56:24.536194+0000","last_undegraded":"2024-09-10T18:56:24.536194+0000","last_fullsized":"2024-09-10T18:56:24.536194+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.486354097,"stat_sum":{"num_bytes":374,"num_objects":34,"num_object_clones":25,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.6","version":"228'63","reported_seq":160,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.527485+0000","last_change":"2024-09-10T18:56:22.511458+0000","last_active":"2024-09-10T18:56:24.527485+0000","last_peered":"2024-09-10T18:56:24.527485+0000","last_clean":"2024-09-10T18:56:24.527485+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:56:24.527485+0000","last_undegraded":"2024-09-10T18:56:24.527485+0000","last_fullsized":"2024-09-10T18:56:24.527485+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38538839200000002,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.5","version":"222'33","reported_seq":134,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.025126+0000","last_change":"2024-09-10T18:56:20.405472+0000","last_active":"2024-09-10T18:56:24.025126+0000","last_peered":"2024-09-10T18:56:24.025126+0000","last_clean":"2024-09-10T18:56:24.025126+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:56:24.025126+0000","last_undegraded":"2024-09-10T18:56:24.025126+0000","last_fullsized":"2024-09-10T18:56:24.025126+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.383550273,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.4","version":"224'78","reported_seq":176,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:23.999053+0000","last_change":"2024-09-10T18:56:22.512841+0000","last_active":"2024-09-10T18:56:23.999053+0000","last_peered":"2024-09-10T18:56:23.999053+0000","last_clean":"2024-09-10T18:56:23.999053+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:56:23.999053+0000","last_undegraded":"2024-09-10T18:56:23.999053+0000","last_fullsized":"2024-09-10T18:56:23.999053+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47377526800000003,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"64.3","version":"228'86","reported_seq":185,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.450462+0000","last_change":"2024-09-10T18:56:22.441185+0000","last_active":"2024-09-10T18:56:24.450462+0000","last_peered":"2024-09-10T18:56:24.450462+0000","last_clean":"2024-09-10T18:56:24.450462+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:56:24.450462+0000","last_undegraded":"2024-09-10T18:56:24.450462+0000","last_fullsized":"2024-09-10T18:56:24.450462+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35997798399999997,"stat_sum":{"num_bytes":154,"num_objects":14,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"64.2","version":"222'30","reported_seq":136,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.010449+0000","last_change":"2024-09-10T18:56:20.428987+0000","last_active":"2024-09-10T18:56:24.010449+0000","last_peered":"2024-09-10T18:56:24.010449+0000","last_clean":"2024-09-10T18:56:24.010449+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:56:24.010449+0000","last_undegraded":"2024-09-10T18:56:24.010449+0000","last_fullsized":"2024-09-10T18:56:24.010449+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40359896099999998,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"64.1","version":"228'53","reported_seq":161,"reported_epoch":228,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:24.442785+0000","last_change":"2024-09-10T18:56:22.434287+0000","last_active":"2024-09-10T18:56:24.442785+0000","last_peered":"2024-09-10T18:56:24.442785+0000","last_clean":"2024-09-10T18:56:24.442785+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:56:24.442785+0000","last_undegraded":"2024-09-10T18:56:24.442785+0000","last_fullsized":"2024-09-10T18:56:24.442785+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.39582951,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"64.0","version":"226'78","reported_seq":173,"reported_epoch":228,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:24.531693+0000","last_change":"2024-09-10T18:56:24.531693+0000","last_active":"2024-09-10T18:56:24.531693+0000","last_peered":"2024-09-10T18:56:24.531693+0000","last_clean":"2024-09-10T18:56:24.531693+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:56:24.531693+0000","last_undegraded":"2024-09-10T18:56:24.531693+0000","last_fullsized":"2024-09-10T18:56:24.531693+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53389596500000003,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007251+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:56:24.007251+0000","last_peered":"2024-09-10T18:56:24.007251+0000","last_clean":"2024-09-10T18:56:24.007251+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:56:24.007251+0000","last_undegraded":"2024-09-10T18:56:24.007251+0000","last_fullsized":"2024-09-10T18:56:24.007251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.508785+0000","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":207,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024239+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:56:24.024239+0000","last_peered":"2024-09-10T18:56:24.024239+0000","last_clean":"2024-09-10T18:56:24.024239+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:56:24.024239+0000","last_undegraded":"2024-09-10T18:56:24.024239+0000","last_fullsized":"2024-09-10T18:56:24.024239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:10.569242+0000","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":201,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013245+0000","last_change":"2024-09-10T18:53:11.768415+0000","last_active":"2024-09-10T18:56:24.013245+0000","last_peered":"2024-09-10T18:56:24.013245+0000","last_clean":"2024-09-10T18:56:24.013245+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:56:24.013245+0000","last_undegraded":"2024-09-10T18:56:24.013245+0000","last_fullsized":"2024-09-10T18:56:24.013245+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:53:11.768254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:53:11.768254+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048933630000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006553+0000","last_change":"2024-09-10T18:53:43.759497+0000","last_active":"2024-09-10T18:56:24.006553+0000","last_peered":"2024-09-10T18:56:24.006553+0000","last_clean":"2024-09-10T18:56:24.006553+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:56:24.006553+0000","last_undegraded":"2024-09-10T18:56:24.006553+0000","last_fullsized":"2024-09-10T18:56:24.006553+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:43.759401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:53:43.759401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026132320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.025139+0000","last_change":"2024-09-10T18:54:37.589336+0000","last_active":"2024-09-10T18:56:24.025139+0000","last_peered":"2024-09-10T18:56:24.025139+0000","last_clean":"2024-09-10T18:56:24.025139+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:56:24.025139+0000","last_undegraded":"2024-09-10T18:56:24.025139+0000","last_fullsized":"2024-09-10T18:56:24.025139+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"126'5","reported_seq":118,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016539+0000","last_change":"2024-09-10T18:54:37.707587+0000","last_active":"2024-09-10T18:56:24.016539+0000","last_peered":"2024-09-10T18:56:24.016539+0000","last_clean":"2024-09-10T18:56:24.016539+0000","last_became_active":"2024-09-10T18:54:37.707288+0000","last_became_peered":"2024-09-10T18:54:37.707288+0000","last_unstale":"2024-09-10T18:56:24.016539+0000","last_undegraded":"2024-09-10T18:56:24.016539+0000","last_fullsized":"2024-09-10T18:56:24.016539+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":208,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016362+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:56:24.016362+0000","last_peered":"2024-09-10T18:56:24.016362+0000","last_clean":"2024-09-10T18:56:24.016362+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:56:24.016362+0000","last_undegraded":"2024-09-10T18:56:24.016362+0000","last_fullsized":"2024-09-10T18:56:24.016362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":224,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023956+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:56:24.023956+0000","last_peered":"2024-09-10T18:56:24.023956+0000","last_clean":"2024-09-10T18:56:24.023956+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:56:24.023956+0000","last_undegraded":"2024-09-10T18:56:24.023956+0000","last_fullsized":"2024-09-10T18:56:24.023956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.632174+0000","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004463+0000","last_change":"2024-09-10T18:54:41.194586+0000","last_active":"2024-09-10T18:56:24.004463+0000","last_peered":"2024-09-10T18:56:24.004463+0000","last_clean":"2024-09-10T18:56:24.004463+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:56:24.004463+0000","last_undegraded":"2024-09-10T18:56:24.004463+0000","last_fullsized":"2024-09-10T18:56:24.004463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:41.194455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.194455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002893928,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008232+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:56:24.008232+0000","last_peered":"2024-09-10T18:56:24.008232+0000","last_clean":"2024-09-10T18:56:24.008232+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:56:24.008232+0000","last_undegraded":"2024-09-10T18:56:24.008232+0000","last_fullsized":"2024-09-10T18:56:24.008232+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:37.265515+0000","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013287+0000","last_change":"2024-09-10T18:54:37.650137+0000","last_active":"2024-09-10T18:56:24.013287+0000","last_peered":"2024-09-10T18:56:24.013287+0000","last_clean":"2024-09-10T18:56:24.013287+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:56:24.013287+0000","last_undegraded":"2024-09-10T18:56:24.013287+0000","last_fullsized":"2024-09-10T18:56:24.013287+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.033185+0000","last_change":"2024-09-10T18:54:37.610296+0000","last_active":"2024-09-10T18:56:24.033185+0000","last_peered":"2024-09-10T18:56:24.033185+0000","last_clean":"2024-09-10T18:56:24.033185+0000","last_became_active":"2024-09-10T18:54:37.597613+0000","last_became_peered":"2024-09-10T18:54:37.597613+0000","last_unstale":"2024-09-10T18:56:24.033185+0000","last_undegraded":"2024-09-10T18:56:24.033185+0000","last_fullsized":"2024-09-10T18:56:24.033185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":205,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.998472+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:56:23.998472+0000","last_peered":"2024-09-10T18:56:23.998472+0000","last_clean":"2024-09-10T18:56:23.998472+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:56:23.998472+0000","last_undegraded":"2024-09-10T18:56:23.998472+0000","last_fullsized":"2024-09-10T18:56:23.998472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:25.677268+0000","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":226,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.032622+0000","last_change":"2024-09-10T18:54:41.188135+0000","last_active":"2024-09-10T18:56:24.032622+0000","last_peered":"2024-09-10T18:56:24.032622+0000","last_clean":"2024-09-10T18:56:24.032622+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:56:24.032622+0000","last_undegraded":"2024-09-10T18:56:24.032622+0000","last_fullsized":"2024-09-10T18:56:24.032622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:54:41.188040+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.188040+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036526110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009647+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:56:24.009647+0000","last_peered":"2024-09-10T18:56:24.009647+0000","last_clean":"2024-09-10T18:56:24.009647+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:56:24.009647+0000","last_undegraded":"2024-09-10T18:56:24.009647+0000","last_fullsized":"2024-09-10T18:56:24.009647+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.192711+0000","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.025418+0000","last_change":"2024-09-10T18:54:39.242238+0000","last_active":"2024-09-10T18:56:24.025418+0000","last_peered":"2024-09-10T18:56:24.025418+0000","last_clean":"2024-09-10T18:56:24.025418+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:56:24.025418+0000","last_undegraded":"2024-09-10T18:56:24.025418+0000","last_fullsized":"2024-09-10T18:56:24.025418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:39.242050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:39.242050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002953027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.028863+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:56:24.028863+0000","last_peered":"2024-09-10T18:56:24.028863+0000","last_clean":"2024-09-10T18:56:24.028863+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:56:24.028863+0000","last_undegraded":"2024-09-10T18:56:24.028863+0000","last_fullsized":"2024-09-10T18:56:24.028863+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.799711+0000","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024348+0000","last_change":"2024-09-10T18:54:37.549144+0000","last_active":"2024-09-10T18:56:24.024348+0000","last_peered":"2024-09-10T18:56:24.024348+0000","last_clean":"2024-09-10T18:56:24.024348+0000","last_became_active":"2024-09-10T18:54:37.543738+0000","last_became_peered":"2024-09-10T18:54:37.543738+0000","last_unstale":"2024-09-10T18:56:24.024348+0000","last_undegraded":"2024-09-10T18:56:24.024348+0000","last_fullsized":"2024-09-10T18:56:24.024348+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":198,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999798+0000","last_change":"2024-09-10T18:56:22.517441+0000","last_active":"2024-09-10T18:56:23.999798+0000","last_peered":"2024-09-10T18:56:23.999798+0000","last_clean":"2024-09-10T18:56:23.999798+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:56:23.999798+0000","last_undegraded":"2024-09-10T18:56:23.999798+0000","last_fullsized":"2024-09-10T18:56:23.999798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:52.078168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:52.078168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029805980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":232,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.025020+0000","last_change":"2024-09-10T18:54:50.207933+0000","last_active":"2024-09-10T18:56:24.025020+0000","last_peered":"2024-09-10T18:56:24.025020+0000","last_clean":"2024-09-10T18:56:24.025020+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:56:24.025020+0000","last_undegraded":"2024-09-10T18:56:24.025020+0000","last_fullsized":"2024-09-10T18:56:24.025020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:54:50.207847+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.207847+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023224109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009558+0000","last_change":"2024-09-10T18:54:55.309898+0000","last_active":"2024-09-10T18:56:24.009558+0000","last_peered":"2024-09-10T18:56:24.009558+0000","last_clean":"2024-09-10T18:56:24.009558+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:56:24.009558+0000","last_undegraded":"2024-09-10T18:56:24.009558+0000","last_fullsized":"2024-09-10T18:56:24.009558+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.309689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.309689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003220512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":201,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009334+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:56:24.009334+0000","last_peered":"2024-09-10T18:56:24.009334+0000","last_clean":"2024-09-10T18:56:24.009334+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:56:24.009334+0000","last_undegraded":"2024-09-10T18:56:24.009334+0000","last_fullsized":"2024-09-10T18:56:24.009334+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:28.316337+0000","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010364+0000","last_change":"2024-09-10T18:54:37.639946+0000","last_active":"2024-09-10T18:56:24.010364+0000","last_peered":"2024-09-10T18:56:24.010364+0000","last_clean":"2024-09-10T18:56:24.010364+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:56:24.010364+0000","last_undegraded":"2024-09-10T18:56:24.010364+0000","last_fullsized":"2024-09-10T18:56:24.010364+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004131+0000","last_change":"2024-09-10T18:56:17.672189+0000","last_active":"2024-09-10T18:56:24.004131+0000","last_peered":"2024-09-10T18:56:24.004131+0000","last_clean":"2024-09-10T18:56:24.004131+0000","last_became_active":"2024-09-10T18:54:37.653532+0000","last_became_peered":"2024-09-10T18:54:37.653532+0000","last_unstale":"2024-09-10T18:56:24.004131+0000","last_undegraded":"2024-09-10T18:56:24.004131+0000","last_fullsized":"2024-09-10T18:56:24.004131+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.745658+0000","scrub_duration":0.0037230840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":235,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008273+0000","last_change":"2024-09-10T18:54:50.687363+0000","last_active":"2024-09-10T18:56:24.008273+0000","last_peered":"2024-09-10T18:56:24.008273+0000","last_clean":"2024-09-10T18:56:24.008273+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:56:24.008273+0000","last_undegraded":"2024-09-10T18:56:24.008273+0000","last_fullsized":"2024-09-10T18:56:24.008273+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:54:50.687166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.687166+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053731969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":195,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.000139+0000","last_change":"2024-09-10T18:54:59.926609+0000","last_active":"2024-09-10T18:56:24.000139+0000","last_peered":"2024-09-10T18:56:24.000139+0000","last_clean":"2024-09-10T18:56:24.000139+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:56:24.000139+0000","last_undegraded":"2024-09-10T18:56:24.000139+0000","last_fullsized":"2024-09-10T18:56:24.000139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:59.926330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.926330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035578459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":300,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008610+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:56:24.008610+0000","last_peered":"2024-09-10T18:56:24.008610+0000","last_clean":"2024-09-10T18:56:24.008610+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:56:24.008610+0000","last_undegraded":"2024-09-10T18:56:24.008610+0000","last_fullsized":"2024-09-10T18:56:24.008610+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:31.292414+0000","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003665+0000","last_change":"2024-09-10T18:55:01.419421+0000","last_active":"2024-09-10T18:56:24.003665+0000","last_peered":"2024-09-10T18:56:24.003665+0000","last_clean":"2024-09-10T18:56:24.003665+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:56:24.003665+0000","last_undegraded":"2024-09-10T18:56:24.003665+0000","last_fullsized":"2024-09-10T18:56:24.003665+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.419333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.419333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035926959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016363+0000","last_change":"2024-09-10T18:54:37.708467+0000","last_active":"2024-09-10T18:56:24.016363+0000","last_peered":"2024-09-10T18:56:24.016363+0000","last_clean":"2024-09-10T18:56:24.016363+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:56:24.016363+0000","last_undegraded":"2024-09-10T18:56:24.016363+0000","last_fullsized":"2024-09-10T18:56:24.016363+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008570+0000","last_change":"2024-09-10T18:56:05.989083+0000","last_active":"2024-09-10T18:56:24.008570+0000","last_peered":"2024-09-10T18:56:24.008570+0000","last_clean":"2024-09-10T18:56:24.008570+0000","last_became_active":"2024-09-10T18:54:37.569079+0000","last_became_peered":"2024-09-10T18:54:37.569079+0000","last_unstale":"2024-09-10T18:56:24.008570+0000","last_undegraded":"2024-09-10T18:56:24.008570+0000","last_fullsized":"2024-09-10T18:56:24.008570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.988916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.988916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:07.493116+0000","scrub_duration":0.0038149899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":216,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010760+0000","last_change":"2024-09-10T18:53:19.842367+0000","last_active":"2024-09-10T18:56:24.010760+0000","last_peered":"2024-09-10T18:56:24.010760+0000","last_clean":"2024-09-10T18:56:24.010760+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:56:24.010760+0000","last_undegraded":"2024-09-10T18:56:24.010760+0000","last_fullsized":"2024-09-10T18:56:24.010760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:53:19.842173+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:53:19.842173+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044794129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.015592+0000","last_change":"2024-09-10T18:54:47.326356+0000","last_active":"2024-09-10T18:56:24.015592+0000","last_peered":"2024-09-10T18:56:24.015592+0000","last_clean":"2024-09-10T18:56:24.015592+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:56:24.015592+0000","last_undegraded":"2024-09-10T18:56:24.015592+0000","last_fullsized":"2024-09-10T18:56:24.015592+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:47.326280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:54:47.326280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026845079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.002579+0000","last_change":"2024-09-10T18:54:50.317064+0000","last_active":"2024-09-10T18:56:24.002579+0000","last_peered":"2024-09-10T18:56:24.002579+0000","last_clean":"2024-09-10T18:56:24.002579+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:56:24.002579+0000","last_undegraded":"2024-09-10T18:56:24.002579+0000","last_fullsized":"2024-09-10T18:56:24.002579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:50.316782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.316782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003031536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004217+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:56:24.004217+0000","last_peered":"2024-09-10T18:56:24.004217+0000","last_clean":"2024-09-10T18:56:24.004217+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:56:24.004217+0000","last_undegraded":"2024-09-10T18:56:24.004217+0000","last_fullsized":"2024-09-10T18:56:24.004217+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:26.081764+0000","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009002+0000","last_change":"2024-09-10T18:54:37.600609+0000","last_active":"2024-09-10T18:56:24.009002+0000","last_peered":"2024-09-10T18:56:24.009002+0000","last_clean":"2024-09-10T18:56:24.009002+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:56:24.009002+0000","last_undegraded":"2024-09-10T18:56:24.009002+0000","last_fullsized":"2024-09-10T18:56:24.009002+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.18","version":"126'2","reported_seq":114,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022671+0000","last_change":"2024-09-10T18:54:37.671736+0000","last_active":"2024-09-10T18:56:24.022671+0000","last_peered":"2024-09-10T18:56:24.022671+0000","last_clean":"2024-09-10T18:56:24.022671+0000","last_became_active":"2024-09-10T18:54:37.671088+0000","last_became_peered":"2024-09-10T18:54:37.671088+0000","last_unstale":"2024-09-10T18:56:24.022671+0000","last_undegraded":"2024-09-10T18:56:24.022671+0000","last_fullsized":"2024-09-10T18:56:24.022671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":221,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007078+0000","last_change":"2024-09-10T18:53:22.802458+0000","last_active":"2024-09-10T18:56:24.007078+0000","last_peered":"2024-09-10T18:56:24.007078+0000","last_clean":"2024-09-10T18:56:24.007078+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:56:24.007078+0000","last_undegraded":"2024-09-10T18:56:24.007078+0000","last_fullsized":"2024-09-10T18:56:24.007078+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:53:22.802388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:53:22.802388+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035152899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":196,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.027826+0000","last_change":"2024-09-10T18:54:38.257497+0000","last_active":"2024-09-10T18:56:24.027826+0000","last_peered":"2024-09-10T18:56:24.027826+0000","last_clean":"2024-09-10T18:56:24.027826+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:56:24.027826+0000","last_undegraded":"2024-09-10T18:56:24.027826+0000","last_fullsized":"2024-09-10T18:56:24.027826+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:38.257394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:38.257394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032838559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.012908+0000","last_change":"2024-09-10T18:54:51.441795+0000","last_active":"2024-09-10T18:56:24.012908+0000","last_peered":"2024-09-10T18:56:24.012908+0000","last_clean":"2024-09-10T18:56:24.012908+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:56:24.012908+0000","last_undegraded":"2024-09-10T18:56:24.012908+0000","last_fullsized":"2024-09-10T18:56:24.012908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.441655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.441655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022907+0000","last_change":"2024-09-10T18:55:02.188929+0000","last_active":"2024-09-10T18:56:24.022907+0000","last_peered":"2024-09-10T18:56:24.022907+0000","last_clean":"2024-09-10T18:56:24.022907+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:56:24.022907+0000","last_undegraded":"2024-09-10T18:56:24.022907+0000","last_fullsized":"2024-09-10T18:56:24.022907+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032689020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.012983+0000","last_change":"2024-09-10T18:56:17.674109+0000","last_active":"2024-09-10T18:56:24.012983+0000","last_peered":"2024-09-10T18:56:24.012983+0000","last_clean":"2024-09-10T18:56:24.012983+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:56:24.012983+0000","last_undegraded":"2024-09-10T18:56:24.012983+0000","last_fullsized":"2024-09-10T18:56:24.012983+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.673943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.673943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:22.872965+0000","scrub_duration":0.003315691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1b","version":"126'10","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022939+0000","last_change":"2024-09-10T18:54:37.523299+0000","last_active":"2024-09-10T18:56:24.022939+0000","last_peered":"2024-09-10T18:56:24.022939+0000","last_clean":"2024-09-10T18:56:24.022939+0000","last_became_active":"2024-09-10T18:54:37.522926+0000","last_became_peered":"2024-09-10T18:54:37.522926+0000","last_unstale":"2024-09-10T18:56:24.022939+0000","last_undegraded":"2024-09-10T18:56:24.022939+0000","last_fullsized":"2024-09-10T18:56:24.022939+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":234,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010612+0000","last_change":"2024-09-10T18:55:03.230735+0000","last_active":"2024-09-10T18:56:24.010612+0000","last_peered":"2024-09-10T18:56:24.010612+0000","last_clean":"2024-09-10T18:56:24.010612+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:56:24.010612+0000","last_undegraded":"2024-09-10T18:56:24.010612+0000","last_fullsized":"2024-09-10T18:56:24.010612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:55:03.230435+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.230435+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048033199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":205,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.994900+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:56:23.994900+0000","last_peered":"2024-09-10T18:56:23.994900+0000","last_clean":"2024-09-10T18:56:23.994900+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:56:23.994900+0000","last_undegraded":"2024-09-10T18:56:23.994900+0000","last_fullsized":"2024-09-10T18:56:23.994900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:20.015637+0000","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":300,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004392+0000","last_change":"2024-09-10T18:54:59.278861+0000","last_active":"2024-09-10T18:56:24.004392+0000","last_peered":"2024-09-10T18:56:24.004392+0000","last_clean":"2024-09-10T18:56:24.004392+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:56:24.004392+0000","last_undegraded":"2024-09-10T18:56:24.004392+0000","last_fullsized":"2024-09-10T18:56:24.004392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:54:59.278694+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.278694+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038345380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.011865+0000","last_change":"2024-09-10T18:53:52.680873+0000","last_active":"2024-09-10T18:56:24.011865+0000","last_peered":"2024-09-10T18:56:24.011865+0000","last_clean":"2024-09-10T18:56:24.011865+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:56:24.011865+0000","last_undegraded":"2024-09-10T18:56:24.011865+0000","last_fullsized":"2024-09-10T18:56:24.011865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:53:52.680777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:53:52.680777+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033357439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.020851+0000","last_change":"2024-09-10T18:54:37.680764+0000","last_active":"2024-09-10T18:56:24.020851+0000","last_peered":"2024-09-10T18:56:24.020851+0000","last_clean":"2024-09-10T18:56:24.020851+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:56:24.020851+0000","last_undegraded":"2024-09-10T18:56:24.020851+0000","last_fullsized":"2024-09-10T18:56:24.020851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"126'5","reported_seq":118,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007108+0000","last_change":"2024-09-10T18:54:37.686458+0000","last_active":"2024-09-10T18:56:24.007108+0000","last_peered":"2024-09-10T18:56:24.007108+0000","last_clean":"2024-09-10T18:56:24.007108+0000","last_became_active":"2024-09-10T18:54:37.686146+0000","last_became_peered":"2024-09-10T18:54:37.686146+0000","last_unstale":"2024-09-10T18:56:24.007108+0000","last_undegraded":"2024-09-10T18:56:24.007108+0000","last_fullsized":"2024-09-10T18:56:24.007108+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":213,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023358+0000","last_change":"2024-09-10T18:54:37.847063+0000","last_active":"2024-09-10T18:56:24.023358+0000","last_peered":"2024-09-10T18:56:24.023358+0000","last_clean":"2024-09-10T18:56:24.023358+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:56:24.023358+0000","last_undegraded":"2024-09-10T18:56:24.023358+0000","last_fullsized":"2024-09-10T18:56:24.023358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:54:37.846984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:37.846984+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098700820000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.019245+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:56:24.019245+0000","last_peered":"2024-09-10T18:56:24.019245+0000","last_clean":"2024-09-10T18:56:24.019245+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:56:24.019245+0000","last_undegraded":"2024-09-10T18:56:24.019245+0000","last_fullsized":"2024-09-10T18:56:24.019245+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008966+0000","last_change":"2024-09-10T18:54:52.331412+0000","last_active":"2024-09-10T18:56:24.008966+0000","last_peered":"2024-09-10T18:56:24.008966+0000","last_clean":"2024-09-10T18:56:24.008966+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:56:24.008966+0000","last_undegraded":"2024-09-10T18:56:24.008966+0000","last_fullsized":"2024-09-10T18:56:24.008966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:52.331111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:52.331111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034919479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":615,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-10T18:56:25.001936+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:56:25.001936+0000","last_peered":"2024-09-10T18:56:25.001936+0000","last_clean":"2024-09-10T18:56:25.001936+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:56:25.001936+0000","last_undegraded":"2024-09-10T18:56:25.001936+0000","last_fullsized":"2024-09-10T18:56:25.001936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.052626+0000","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":403,"num_read_kb":398,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.996182+0000","last_change":"2024-09-10T18:55:07.836217+0000","last_active":"2024-09-10T18:56:23.996182+0000","last_peered":"2024-09-10T18:56:23.996182+0000","last_clean":"2024-09-10T18:56:23.996182+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:56:23.996182+0000","last_undegraded":"2024-09-10T18:56:23.996182+0000","last_fullsized":"2024-09-10T18:56:23.996182+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:07.836145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:23.121359+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.836145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:31.908992+0000","scrub_duration":0.0031616470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.001288+0000","last_change":"2024-09-10T18:56:12.680019+0000","last_active":"2024-09-10T18:56:24.001288+0000","last_peered":"2024-09-10T18:56:24.001288+0000","last_clean":"2024-09-10T18:56:24.001288+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:56:24.001288+0000","last_undegraded":"2024-09-10T18:56:24.001288+0000","last_fullsized":"2024-09-10T18:56:24.001288+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:12.679938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:12.679938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.716839+0000","scrub_duration":0.002716511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023851+0000","last_change":"2024-09-10T18:54:37.644991+0000","last_active":"2024-09-10T18:56:24.023851+0000","last_peered":"2024-09-10T18:56:24.023851+0000","last_clean":"2024-09-10T18:56:24.023851+0000","last_became_active":"2024-09-10T18:54:37.631506+0000","last_became_peered":"2024-09-10T18:54:37.631506+0000","last_unstale":"2024-09-10T18:56:24.023851+0000","last_undegraded":"2024-09-10T18:56:24.023851+0000","last_fullsized":"2024-09-10T18:56:24.023851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":233,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017321+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:56:24.017321+0000","last_peered":"2024-09-10T18:56:24.017321+0000","last_clean":"2024-09-10T18:56:24.017321+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:56:24.017321+0000","last_undegraded":"2024-09-10T18:56:24.017321+0000","last_fullsized":"2024-09-10T18:56:24.017321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.286896+0000","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.021232+0000","last_change":"2024-09-10T18:54:55.507250+0000","last_active":"2024-09-10T18:56:24.021232+0000","last_peered":"2024-09-10T18:56:24.021232+0000","last_clean":"2024-09-10T18:56:24.021232+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:56:24.021232+0000","last_undegraded":"2024-09-10T18:56:24.021232+0000","last_fullsized":"2024-09-10T18:56:24.021232+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.507168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.507168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027901786000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":289,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006919+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:56:24.006919+0000","last_peered":"2024-09-10T18:56:24.006919+0000","last_clean":"2024-09-10T18:56:24.006919+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:56:24.006919+0000","last_undegraded":"2024-09-10T18:56:24.006919+0000","last_fullsized":"2024-09-10T18:56:24.006919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.264937+0000","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":211,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008808+0000","last_change":"2024-09-10T18:54:49.283135+0000","last_active":"2024-09-10T18:56:24.008808+0000","last_peered":"2024-09-10T18:56:24.008808+0000","last_clean":"2024-09-10T18:56:24.008808+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:56:24.008808+0000","last_undegraded":"2024-09-10T18:56:24.008808+0000","last_fullsized":"2024-09-10T18:56:24.008808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:54:49.283014+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.283014+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024850369999999998,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008690+0000","last_change":"2024-09-10T18:54:46.264917+0000","last_active":"2024-09-10T18:56:24.008690+0000","last_peered":"2024-09-10T18:56:24.008690+0000","last_clean":"2024-09-10T18:56:24.008690+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:56:24.008690+0000","last_undegraded":"2024-09-10T18:56:24.008690+0000","last_fullsized":"2024-09-10T18:56:24.008690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:46.264846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:46.264846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027169630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006809+0000","last_change":"2024-09-10T18:54:37.726541+0000","last_active":"2024-09-10T18:56:24.006809+0000","last_peered":"2024-09-10T18:56:24.006809+0000","last_clean":"2024-09-10T18:56:24.006809+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:56:24.006809+0000","last_undegraded":"2024-09-10T18:56:24.006809+0000","last_fullsized":"2024-09-10T18:56:24.006809+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"126'2","reported_seq":114,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013419+0000","last_change":"2024-09-10T18:54:37.677688+0000","last_active":"2024-09-10T18:56:24.013419+0000","last_peered":"2024-09-10T18:56:24.013419+0000","last_clean":"2024-09-10T18:56:24.013419+0000","last_became_active":"2024-09-10T18:54:37.677049+0000","last_became_peered":"2024-09-10T18:54:37.677049+0000","last_unstale":"2024-09-10T18:56:24.013419+0000","last_undegraded":"2024-09-10T18:56:24.013419+0000","last_fullsized":"2024-09-10T18:56:24.013419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":349,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017234+0000","last_change":"2024-09-10T18:56:03.530880+0000","last_active":"2024-09-10T18:56:24.017234+0000","last_peered":"2024-09-10T18:56:24.017234+0000","last_clean":"2024-09-10T18:56:24.017234+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:56:24.017234+0000","last_undegraded":"2024-09-10T18:56:24.017234+0000","last_fullsized":"2024-09-10T18:56:24.017234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:56:03.530741+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.530741+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:04.774205+0000","scrub_duration":0.0049201490000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":205,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999408+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:56:23.999408+0000","last_peered":"2024-09-10T18:56:23.999408+0000","last_clean":"2024-09-10T18:56:23.999408+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:56:23.999408+0000","last_undegraded":"2024-09-10T18:56:23.999408+0000","last_fullsized":"2024-09-10T18:56:23.999408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.253426+0000","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":299,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013374+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:56:24.013374+0000","last_peered":"2024-09-10T18:56:24.013374+0000","last_clean":"2024-09-10T18:56:24.013374+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:56:24.013374+0000","last_undegraded":"2024-09-10T18:56:24.013374+0000","last_fullsized":"2024-09-10T18:56:24.013374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:34.440175+0000","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":210,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006943+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:56:24.006943+0000","last_peered":"2024-09-10T18:56:24.006943+0000","last_clean":"2024-09-10T18:56:24.006943+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:56:24.006943+0000","last_undegraded":"2024-09-10T18:56:24.006943+0000","last_fullsized":"2024-09-10T18:56:24.006943+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":190,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.027567+0000","last_change":"2024-09-10T18:54:48.378098+0000","last_active":"2024-09-10T18:56:24.027567+0000","last_peered":"2024-09-10T18:56:24.027567+0000","last_clean":"2024-09-10T18:56:24.027567+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:56:24.027567+0000","last_undegraded":"2024-09-10T18:56:24.027567+0000","last_fullsized":"2024-09-10T18:56:24.027567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:48.377955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:48.377955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031838769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010989+0000","last_change":"2024-09-10T18:54:37.710475+0000","last_active":"2024-09-10T18:56:24.010989+0000","last_peered":"2024-09-10T18:56:24.010989+0000","last_clean":"2024-09-10T18:56:24.010989+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:56:24.010989+0000","last_undegraded":"2024-09-10T18:56:24.010989+0000","last_fullsized":"2024-09-10T18:56:24.010989+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"126'5","reported_seq":117,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006247+0000","last_change":"2024-09-10T18:54:37.589535+0000","last_active":"2024-09-10T18:56:24.006247+0000","last_peered":"2024-09-10T18:56:24.006247+0000","last_clean":"2024-09-10T18:56:24.006247+0000","last_became_active":"2024-09-10T18:54:37.589242+0000","last_became_peered":"2024-09-10T18:54:37.589242+0000","last_unstale":"2024-09-10T18:56:24.006247+0000","last_undegraded":"2024-09-10T18:56:24.006247+0000","last_fullsized":"2024-09-10T18:56:24.006247+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":226,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024774+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:56:24.024774+0000","last_peered":"2024-09-10T18:56:24.024774+0000","last_clean":"2024-09-10T18:56:24.024774+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:56:24.024774+0000","last_undegraded":"2024-09-10T18:56:24.024774+0000","last_fullsized":"2024-09-10T18:56:24.024774+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017337+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:56:24.017337+0000","last_peered":"2024-09-10T18:56:24.017337+0000","last_clean":"2024-09-10T18:56:24.017337+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:56:24.017337+0000","last_undegraded":"2024-09-10T18:56:24.017337+0000","last_fullsized":"2024-09-10T18:56:24.017337+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.138878+0000","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.015323+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:56:24.015323+0000","last_peered":"2024-09-10T18:56:24.015323+0000","last_clean":"2024-09-10T18:56:24.015323+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:56:24.015323+0000","last_undegraded":"2024-09-10T18:56:24.015323+0000","last_fullsized":"2024-09-10T18:56:24.015323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:29.391815+0000","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":201,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009646+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:56:24.009646+0000","last_peered":"2024-09-10T18:56:24.009646+0000","last_clean":"2024-09-10T18:56:24.009646+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:56:24.009646+0000","last_undegraded":"2024-09-10T18:56:24.009646+0000","last_fullsized":"2024-09-10T18:56:24.009646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.883147+0000","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.998472+0000","last_change":"2024-09-10T18:56:13.649580+0000","last_active":"2024-09-10T18:56:23.998472+0000","last_peered":"2024-09-10T18:56:23.998472+0000","last_clean":"2024-09-10T18:56:23.998472+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:56:23.998472+0000","last_undegraded":"2024-09-10T18:56:23.998472+0000","last_fullsized":"2024-09-10T18:56:23.998472+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.649458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.649458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.256522+0000","scrub_duration":0.0027534759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"126'5","reported_seq":118,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016489+0000","last_change":"2024-09-10T18:54:37.679859+0000","last_active":"2024-09-10T18:56:24.016489+0000","last_peered":"2024-09-10T18:56:24.016489+0000","last_clean":"2024-09-10T18:56:24.016489+0000","last_became_active":"2024-09-10T18:54:37.675561+0000","last_became_peered":"2024-09-10T18:54:37.675561+0000","last_unstale":"2024-09-10T18:56:24.016489+0000","last_undegraded":"2024-09-10T18:56:24.016489+0000","last_fullsized":"2024-09-10T18:56:24.016489+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":242,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010060+0000","last_change":"2024-09-10T18:55:15.131835+0000","last_active":"2024-09-10T18:56:24.010060+0000","last_peered":"2024-09-10T18:56:24.010060+0000","last_clean":"2024-09-10T18:56:24.010060+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:56:24.010060+0000","last_undegraded":"2024-09-10T18:56:24.010060+0000","last_fullsized":"2024-09-10T18:56:24.010060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:55:15.131693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:15.131693+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048796719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024682+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:56:24.024682+0000","last_peered":"2024-09-10T18:56:24.024682+0000","last_clean":"2024-09-10T18:56:24.024682+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:56:24.024682+0000","last_undegraded":"2024-09-10T18:56:24.024682+0000","last_fullsized":"2024-09-10T18:56:24.024682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017117+0000","last_change":"2024-09-10T18:56:04.495501+0000","last_active":"2024-09-10T18:56:24.017117+0000","last_peered":"2024-09-10T18:56:24.017117+0000","last_clean":"2024-09-10T18:56:24.017117+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:56:24.017117+0000","last_undegraded":"2024-09-10T18:56:24.017117+0000","last_fullsized":"2024-09-10T18:56:24.017117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.495334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.495334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:06.519347+0000","scrub_duration":0.003611179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013074+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:56:24.013074+0000","last_peered":"2024-09-10T18:56:24.013074+0000","last_clean":"2024-09-10T18:56:24.013074+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:56:24.013074+0000","last_undegraded":"2024-09-10T18:56:24.013074+0000","last_fullsized":"2024-09-10T18:56:24.013074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.503223+0000","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013256+0000","last_change":"2024-09-10T18:54:37.645323+0000","last_active":"2024-09-10T18:56:24.013256+0000","last_peered":"2024-09-10T18:56:24.013256+0000","last_clean":"2024-09-10T18:56:24.013256+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:56:24.013256+0000","last_undegraded":"2024-09-10T18:56:24.013256+0000","last_fullsized":"2024-09-10T18:56:24.013256+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999521+0000","last_change":"2024-09-10T18:54:37.562045+0000","last_active":"2024-09-10T18:56:23.999521+0000","last_peered":"2024-09-10T18:56:23.999521+0000","last_clean":"2024-09-10T18:56:23.999521+0000","last_became_active":"2024-09-10T18:54:37.561404+0000","last_became_peered":"2024-09-10T18:54:37.561404+0000","last_unstale":"2024-09-10T18:56:23.999521+0000","last_undegraded":"2024-09-10T18:56:23.999521+0000","last_fullsized":"2024-09-10T18:56:23.999521+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":244,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.029952+0000","last_change":"2024-09-10T18:54:56.479766+0000","last_active":"2024-09-10T18:56:24.029952+0000","last_peered":"2024-09-10T18:56:24.029952+0000","last_clean":"2024-09-10T18:56:24.029952+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:56:24.029952+0000","last_undegraded":"2024-09-10T18:56:24.029952+0000","last_fullsized":"2024-09-10T18:56:24.029952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:54:56.479670+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.479670+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003399188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":216,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.001147+0000","last_change":"2024-09-10T18:56:15.655135+0000","last_active":"2024-09-10T18:56:24.001147+0000","last_peered":"2024-09-10T18:56:24.001147+0000","last_clean":"2024-09-10T18:56:24.001147+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:56:24.001147+0000","last_undegraded":"2024-09-10T18:56:24.001147+0000","last_fullsized":"2024-09-10T18:56:24.001147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:15.655055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.655055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.233854+0000","scrub_duration":0.0020254439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013057+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:56:24.013057+0000","last_peered":"2024-09-10T18:56:24.013057+0000","last_clean":"2024-09-10T18:56:24.013057+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:56:24.013057+0000","last_undegraded":"2024-09-10T18:56:24.013057+0000","last_fullsized":"2024-09-10T18:56:24.013057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.219400+0000","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.998714+0000","last_change":"2024-09-10T18:56:06.517243+0000","last_active":"2024-09-10T18:56:23.998714+0000","last_peered":"2024-09-10T18:56:23.998714+0000","last_clean":"2024-09-10T18:56:23.998714+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:56:23.998714+0000","last_undegraded":"2024-09-10T18:56:23.998714+0000","last_fullsized":"2024-09-10T18:56:23.998714+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.517079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.517079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:07.534844+0000","scrub_duration":0.0038953730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.018863+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:56:24.018863+0000","last_peered":"2024-09-10T18:56:24.018863+0000","last_clean":"2024-09-10T18:56:24.018863+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:56:24.018863+0000","last_undegraded":"2024-09-10T18:56:24.018863+0000","last_fullsized":"2024-09-10T18:56:24.018863+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.237861+0000","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"126'9","reported_seq":131,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.029094+0000","last_change":"2024-09-10T18:56:15.650829+0000","last_active":"2024-09-10T18:56:24.029094+0000","last_peered":"2024-09-10T18:56:24.029094+0000","last_clean":"2024-09-10T18:56:24.029094+0000","last_became_active":"2024-09-10T18:54:37.676077+0000","last_became_peered":"2024-09-10T18:54:37.676077+0000","last_unstale":"2024-09-10T18:56:24.029094+0000","last_undegraded":"2024-09-10T18:56:24.029094+0000","last_fullsized":"2024-09-10T18:56:24.029094+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:15.650681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.650681+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.012078+0000","scrub_duration":0.0053199110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":226,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022533+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:56:24.022533+0000","last_peered":"2024-09-10T18:56:24.022533+0000","last_clean":"2024-09-10T18:56:24.022533+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:56:24.022533+0000","last_undegraded":"2024-09-10T18:56:24.022533+0000","last_fullsized":"2024-09-10T18:56:24.022533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":205,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.993949+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:56:23.993949+0000","last_peered":"2024-09-10T18:56:23.993949+0000","last_clean":"2024-09-10T18:56:23.993949+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:56:23.993949+0000","last_undegraded":"2024-09-10T18:56:23.993949+0000","last_fullsized":"2024-09-10T18:56:23.993949+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.405645+0000","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":203,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023440+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:56:24.023440+0000","last_peered":"2024-09-10T18:56:24.023440+0000","last_clean":"2024-09-10T18:56:24.023440+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:56:24.023440+0000","last_undegraded":"2024-09-10T18:56:24.023440+0000","last_fullsized":"2024-09-10T18:56:24.023440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016829+0000","last_change":"2024-09-10T18:55:19.296815+0000","last_active":"2024-09-10T18:56:24.016829+0000","last_peered":"2024-09-10T18:56:24.016829+0000","last_clean":"2024-09-10T18:56:24.016829+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:56:24.016829+0000","last_undegraded":"2024-09-10T18:56:24.016829+0000","last_fullsized":"2024-09-10T18:56:24.016829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:19.296674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:55:19.296674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003182767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023673+0000","last_change":"2024-09-10T18:54:37.579617+0000","last_active":"2024-09-10T18:56:24.023673+0000","last_peered":"2024-09-10T18:56:24.023673+0000","last_clean":"2024-09-10T18:56:24.023673+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:56:24.023673+0000","last_undegraded":"2024-09-10T18:56:24.023673+0000","last_fullsized":"2024-09-10T18:56:24.023673+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008582+0000","last_change":"2024-09-10T18:54:37.530238+0000","last_active":"2024-09-10T18:56:24.008582+0000","last_peered":"2024-09-10T18:56:24.008582+0000","last_clean":"2024-09-10T18:56:24.008582+0000","last_became_active":"2024-09-10T18:54:37.529892+0000","last_became_peered":"2024-09-10T18:54:37.529892+0000","last_unstale":"2024-09-10T18:56:24.008582+0000","last_undegraded":"2024-09-10T18:56:24.008582+0000","last_fullsized":"2024-09-10T18:56:24.008582+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":216,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007561+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:56:24.007561+0000","last_peered":"2024-09-10T18:56:24.007561+0000","last_clean":"2024-09-10T18:56:24.007561+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:56:24.007561+0000","last_undegraded":"2024-09-10T18:56:24.007561+0000","last_fullsized":"2024-09-10T18:56:24.007561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.212086+0000","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024993+0000","last_change":"2024-09-10T18:55:03.233057+0000","last_active":"2024-09-10T18:56:24.024993+0000","last_peered":"2024-09-10T18:56:24.024993+0000","last_clean":"2024-09-10T18:56:24.024993+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:56:24.024993+0000","last_undegraded":"2024-09-10T18:56:24.024993+0000","last_fullsized":"2024-09-10T18:56:24.024993+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:03.232923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.232923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036625059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":253,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.994182+0000","last_change":"2024-09-10T18:56:04.556751+0000","last_active":"2024-09-10T18:56:23.994182+0000","last_peered":"2024-09-10T18:56:23.994182+0000","last_clean":"2024-09-10T18:56:23.994182+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:56:23.994182+0000","last_undegraded":"2024-09-10T18:56:23.994182+0000","last_fullsized":"2024-09-10T18:56:23.994182+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:56:04.556586+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.556586+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.342255+0000","scrub_duration":0.0046952210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016764+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:56:24.016764+0000","last_peered":"2024-09-10T18:56:24.016764+0000","last_clean":"2024-09-10T18:56:24.016764+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:56:24.016764+0000","last_undegraded":"2024-09-10T18:56:24.016764+0000","last_fullsized":"2024-09-10T18:56:24.016764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024470+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:56:24.024470+0000","last_peered":"2024-09-10T18:56:24.024470+0000","last_clean":"2024-09-10T18:56:24.024470+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:56:24.024470+0000","last_undegraded":"2024-09-10T18:56:24.024470+0000","last_fullsized":"2024-09-10T18:56:24.024470+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:13.987563+0000","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999137+0000","last_change":"2024-09-10T18:54:37.598514+0000","last_active":"2024-09-10T18:56:23.999137+0000","last_peered":"2024-09-10T18:56:23.999137+0000","last_clean":"2024-09-10T18:56:23.999137+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:56:23.999137+0000","last_undegraded":"2024-09-10T18:56:23.999137+0000","last_fullsized":"2024-09-10T18:56:23.999137+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.11","version":"126'7","reported_seq":130,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.025234+0000","last_change":"2024-09-10T18:56:21.349180+0000","last_active":"2024-09-10T18:56:24.025234+0000","last_peered":"2024-09-10T18:56:24.025234+0000","last_clean":"2024-09-10T18:56:24.025234+0000","last_became_active":"2024-09-10T18:54:37.602669+0000","last_became_peered":"2024-09-10T18:54:37.602669+0000","last_unstale":"2024-09-10T18:56:24.025234+0000","last_undegraded":"2024-09-10T18:56:24.025234+0000","last_fullsized":"2024-09-10T18:56:24.025234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:21.348988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.348988+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.217619+0000","scrub_duration":0.0044214550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":221,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999283+0000","last_change":"2024-09-10T18:54:39.804211+0000","last_active":"2024-09-10T18:56:23.999283+0000","last_peered":"2024-09-10T18:56:23.999283+0000","last_clean":"2024-09-10T18:56:23.999283+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:56:23.999283+0000","last_undegraded":"2024-09-10T18:56:23.999283+0000","last_fullsized":"2024-09-10T18:56:23.999283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:54:39.804056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:39.804056+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057418790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.005852+0000","last_change":"2024-09-10T18:54:58.446820+0000","last_active":"2024-09-10T18:56:24.005852+0000","last_peered":"2024-09-10T18:56:24.005852+0000","last_clean":"2024-09-10T18:56:24.005852+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:56:24.005852+0000","last_undegraded":"2024-09-10T18:56:24.005852+0000","last_fullsized":"2024-09-10T18:56:24.005852+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.446707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.446707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011714456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008357+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:56:24.008357+0000","last_peered":"2024-09-10T18:56:24.008357+0000","last_clean":"2024-09-10T18:56:24.008357+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:56:24.008357+0000","last_undegraded":"2024-09-10T18:56:24.008357+0000","last_fullsized":"2024-09-10T18:56:24.008357+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":189,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.995119+0000","last_change":"2024-09-10T18:54:38.803113+0000","last_active":"2024-09-10T18:56:23.995119+0000","last_peered":"2024-09-10T18:56:23.995119+0000","last_clean":"2024-09-10T18:56:23.995119+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:56:23.995119+0000","last_undegraded":"2024-09-10T18:56:23.995119+0000","last_fullsized":"2024-09-10T18:56:23.995119+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:38.802987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:38.802987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024544110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008074+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:56:24.008074+0000","last_peered":"2024-09-10T18:56:24.008074+0000","last_clean":"2024-09-10T18:56:24.008074+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:56:24.008074+0000","last_undegraded":"2024-09-10T18:56:24.008074+0000","last_fullsized":"2024-09-10T18:56:24.008074+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.438790+0000","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.995966+0000","last_change":"2024-09-10T18:54:37.525700+0000","last_active":"2024-09-10T18:56:23.995966+0000","last_peered":"2024-09-10T18:56:23.995966+0000","last_clean":"2024-09-10T18:56:23.995966+0000","last_became_active":"2024-09-10T18:54:37.525432+0000","last_became_peered":"2024-09-10T18:54:37.525432+0000","last_unstale":"2024-09-10T18:56:23.995966+0000","last_undegraded":"2024-09-10T18:56:23.995966+0000","last_fullsized":"2024-09-10T18:56:23.995966+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":224,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.032735+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:56:24.032735+0000","last_peered":"2024-09-10T18:56:24.032735+0000","last_clean":"2024-09-10T18:56:24.032735+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:56:24.032735+0000","last_undegraded":"2024-09-10T18:56:24.032735+0000","last_fullsized":"2024-09-10T18:56:24.032735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.198556+0000","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":196,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022577+0000","last_change":"2024-09-10T18:54:40.225129+0000","last_active":"2024-09-10T18:56:24.022577+0000","last_peered":"2024-09-10T18:56:24.022577+0000","last_clean":"2024-09-10T18:56:24.022577+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:56:24.022577+0000","last_undegraded":"2024-09-10T18:56:24.022577+0000","last_fullsized":"2024-09-10T18:56:24.022577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:40.224966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.224966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003056895,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013198+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:56:24.013198+0000","last_peered":"2024-09-10T18:56:24.013198+0000","last_clean":"2024-09-10T18:56:24.013198+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:56:24.013198+0000","last_undegraded":"2024-09-10T18:56:24.013198+0000","last_fullsized":"2024-09-10T18:56:24.013198+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":191,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.012984+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:56:24.012984+0000","last_peered":"2024-09-10T18:56:24.012984+0000","last_clean":"2024-09-10T18:56:24.012984+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:56:24.012984+0000","last_undegraded":"2024-09-10T18:56:24.012984+0000","last_fullsized":"2024-09-10T18:56:24.012984+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023509+0000","last_change":"2024-09-10T18:54:37.671776+0000","last_active":"2024-09-10T18:56:24.023509+0000","last_peered":"2024-09-10T18:56:24.023509+0000","last_clean":"2024-09-10T18:56:24.023509+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:56:24.023509+0000","last_undegraded":"2024-09-10T18:56:24.023509+0000","last_fullsized":"2024-09-10T18:56:24.023509+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"126'5","reported_seq":118,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.025787+0000","last_change":"2024-09-10T18:54:37.699169+0000","last_active":"2024-09-10T18:56:24.025787+0000","last_peered":"2024-09-10T18:56:24.025787+0000","last_clean":"2024-09-10T18:56:24.025787+0000","last_became_active":"2024-09-10T18:54:37.698688+0000","last_became_peered":"2024-09-10T18:54:37.698688+0000","last_unstale":"2024-09-10T18:56:24.025787+0000","last_undegraded":"2024-09-10T18:56:24.025787+0000","last_fullsized":"2024-09-10T18:56:24.025787+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":235,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.996743+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:56:23.996743+0000","last_peered":"2024-09-10T18:56:23.996743+0000","last_clean":"2024-09-10T18:56:23.996743+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:56:23.996743+0000","last_undegraded":"2024-09-10T18:56:23.996743+0000","last_fullsized":"2024-09-10T18:56:23.996743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.067234+0000","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":218,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003567+0000","last_change":"2024-09-10T18:54:41.561362+0000","last_active":"2024-09-10T18:56:24.003567+0000","last_peered":"2024-09-10T18:56:24.003567+0000","last_clean":"2024-09-10T18:56:24.003567+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:56:24.003567+0000","last_undegraded":"2024-09-10T18:56:24.003567+0000","last_fullsized":"2024-09-10T18:56:24.003567+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:54:41.561214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.561214+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036670589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022714+0000","last_change":"2024-09-10T18:55:00.199821+0000","last_active":"2024-09-10T18:56:24.022714+0000","last_peered":"2024-09-10T18:56:24.022714+0000","last_clean":"2024-09-10T18:56:24.022714+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:56:24.022714+0000","last_undegraded":"2024-09-10T18:56:24.022714+0000","last_fullsized":"2024-09-10T18:56:24.022714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.199571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.199571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036539939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023992+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:56:24.023992+0000","last_peered":"2024-09-10T18:56:24.023992+0000","last_clean":"2024-09-10T18:56:24.023992+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:56:24.023992+0000","last_undegraded":"2024-09-10T18:56:24.023992+0000","last_fullsized":"2024-09-10T18:56:24.023992+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004217+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:56:24.004217+0000","last_peered":"2024-09-10T18:56:24.004217+0000","last_clean":"2024-09-10T18:56:24.004217+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:56:24.004217+0000","last_undegraded":"2024-09-10T18:56:24.004217+0000","last_fullsized":"2024-09-10T18:56:24.004217+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:06.585634+0000","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.005937+0000","last_change":"2024-09-10T18:56:14.608089+0000","last_active":"2024-09-10T18:56:24.005937+0000","last_peered":"2024-09-10T18:56:24.005937+0000","last_clean":"2024-09-10T18:56:24.005937+0000","last_became_active":"2024-09-10T18:54:37.592760+0000","last_became_peered":"2024-09-10T18:54:37.592760+0000","last_unstale":"2024-09-10T18:56:24.005937+0000","last_undegraded":"2024-09-10T18:56:24.005937+0000","last_fullsized":"2024-09-10T18:56:24.005937+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.607936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.607936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.345720+0000","scrub_duration":0.0032677309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":212,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.997602+0000","last_change":"2024-09-10T18:53:40.127694+0000","last_active":"2024-09-10T18:56:23.997602+0000","last_peered":"2024-09-10T18:56:23.997602+0000","last_clean":"2024-09-10T18:56:23.997602+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:56:23.997602+0000","last_undegraded":"2024-09-10T18:56:23.997602+0000","last_fullsized":"2024-09-10T18:56:23.997602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:53:40.127483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:53:40.127483+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049170749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":247,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003671+0000","last_change":"2024-09-10T18:54:47.237451+0000","last_active":"2024-09-10T18:56:24.003671+0000","last_peered":"2024-09-10T18:56:24.003671+0000","last_clean":"2024-09-10T18:56:24.003671+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:56:24.003671+0000","last_undegraded":"2024-09-10T18:56:24.003671+0000","last_fullsized":"2024-09-10T18:56:24.003671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:54:47.237287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:47.237287+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035410250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":299,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010462+0000","last_change":"2024-09-10T18:55:04.228572+0000","last_active":"2024-09-10T18:56:24.010462+0000","last_peered":"2024-09-10T18:56:24.010462+0000","last_clean":"2024-09-10T18:56:24.010462+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:56:24.010462+0000","last_undegraded":"2024-09-10T18:56:24.010462+0000","last_fullsized":"2024-09-10T18:56:24.010462+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:04.228360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.228360+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039590529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010335+0000","last_change":"2024-09-10T18:55:01.757908+0000","last_active":"2024-09-10T18:56:24.010335+0000","last_peered":"2024-09-10T18:56:24.010335+0000","last_clean":"2024-09-10T18:56:24.010335+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:56:24.010335+0000","last_undegraded":"2024-09-10T18:56:24.010335+0000","last_fullsized":"2024-09-10T18:56:24.010335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.757807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.757807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011468136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":126,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006403+0000","last_change":"2024-09-10T18:54:37.602006+0000","last_active":"2024-09-10T18:56:24.006403+0000","last_peered":"2024-09-10T18:56:24.006403+0000","last_clean":"2024-09-10T18:56:24.006403+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:56:24.006403+0000","last_undegraded":"2024-09-10T18:56:24.006403+0000","last_fullsized":"2024-09-10T18:56:24.006403+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1d","version":"126'7","reported_seq":129,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003086+0000","last_change":"2024-09-10T18:56:04.762019+0000","last_active":"2024-09-10T18:56:24.003086+0000","last_peered":"2024-09-10T18:56:24.003086+0000","last_clean":"2024-09-10T18:56:24.003086+0000","last_became_active":"2024-09-10T18:54:37.592557+0000","last_became_peered":"2024-09-10T18:54:37.592557+0000","last_unstale":"2024-09-10T18:56:24.003086+0000","last_undegraded":"2024-09-10T18:56:24.003086+0000","last_fullsized":"2024-09-10T18:56:24.003086+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:04.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.761871+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.599357+0000","scrub_duration":0.003828508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":222,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.996611+0000","last_change":"2024-09-10T18:55:05.902412+0000","last_active":"2024-09-10T18:56:23.996611+0000","last_peered":"2024-09-10T18:56:23.996611+0000","last_clean":"2024-09-10T18:56:23.996611+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:56:23.996611+0000","last_undegraded":"2024-09-10T18:56:23.996611+0000","last_fullsized":"2024-09-10T18:56:23.996611+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:05.902154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.902154+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047960349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999141+0000","last_change":"2024-09-10T18:54:49.456319+0000","last_active":"2024-09-10T18:56:23.999141+0000","last_peered":"2024-09-10T18:56:23.999141+0000","last_clean":"2024-09-10T18:56:23.999141+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:56:23.999141+0000","last_undegraded":"2024-09-10T18:56:23.999141+0000","last_fullsized":"2024-09-10T18:56:23.999141+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.456167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.456167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002905819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":203,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023509+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:56:24.023509+0000","last_peered":"2024-09-10T18:56:24.023509+0000","last_clean":"2024-09-10T18:56:24.023509+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:56:24.023509+0000","last_undegraded":"2024-09-10T18:56:24.023509+0000","last_fullsized":"2024-09-10T18:56:24.023509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.259079+0000","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.005328+0000","last_change":"2024-09-10T18:55:02.390515+0000","last_active":"2024-09-10T18:56:24.005328+0000","last_peered":"2024-09-10T18:56:24.005328+0000","last_clean":"2024-09-10T18:56:24.005328+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:56:24.005328+0000","last_undegraded":"2024-09-10T18:56:24.005328+0000","last_fullsized":"2024-09-10T18:56:24.005328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.390417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.390417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:26.267881+0000","scrub_duration":0.002554265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004594+0000","last_change":"2024-09-10T18:54:37.614987+0000","last_active":"2024-09-10T18:56:24.004594+0000","last_peered":"2024-09-10T18:56:24.004594+0000","last_clean":"2024-09-10T18:56:24.004594+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:56:24.004594+0000","last_undegraded":"2024-09-10T18:56:24.004594+0000","last_fullsized":"2024-09-10T18:56:24.004594+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007961+0000","last_change":"2024-09-10T18:54:37.577908+0000","last_active":"2024-09-10T18:56:24.007961+0000","last_peered":"2024-09-10T18:56:24.007961+0000","last_clean":"2024-09-10T18:56:24.007961+0000","last_became_active":"2024-09-10T18:54:37.566860+0000","last_became_peered":"2024-09-10T18:54:37.566860+0000","last_unstale":"2024-09-10T18:56:24.007961+0000","last_undegraded":"2024-09-10T18:56:24.007961+0000","last_fullsized":"2024-09-10T18:56:24.007961+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":212,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008202+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:56:24.008202+0000","last_peered":"2024-09-10T18:56:24.008202+0000","last_clean":"2024-09-10T18:56:24.008202+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:56:24.008202+0000","last_undegraded":"2024-09-10T18:56:24.008202+0000","last_fullsized":"2024-09-10T18:56:24.008202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006961+0000","last_change":"2024-09-10T18:54:49.637132+0000","last_active":"2024-09-10T18:56:24.006961+0000","last_peered":"2024-09-10T18:56:24.006961+0000","last_clean":"2024-09-10T18:56:24.006961+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:56:24.006961+0000","last_undegraded":"2024-09-10T18:56:24.006961+0000","last_fullsized":"2024-09-10T18:56:24.006961+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.637024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.637024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029426489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017245+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:56:24.017245+0000","last_peered":"2024-09-10T18:56:24.017245+0000","last_clean":"2024-09-10T18:56:24.017245+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:56:24.017245+0000","last_undegraded":"2024-09-10T18:56:24.017245+0000","last_fullsized":"2024-09-10T18:56:24.017245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.998597+0000","last_change":"2024-09-10T18:56:03.595788+0000","last_active":"2024-09-10T18:56:23.998597+0000","last_peered":"2024-09-10T18:56:23.998597+0000","last_clean":"2024-09-10T18:56:23.998597+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:56:23.998597+0000","last_undegraded":"2024-09-10T18:56:23.998597+0000","last_fullsized":"2024-09-10T18:56:23.998597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.595643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.595643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.278652+0000","scrub_duration":0.0030518839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.018777+0000","last_change":"2024-09-10T18:54:37.711456+0000","last_active":"2024-09-10T18:56:24.018777+0000","last_peered":"2024-09-10T18:56:24.018777+0000","last_clean":"2024-09-10T18:56:24.018777+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:56:24.018777+0000","last_undegraded":"2024-09-10T18:56:24.018777+0000","last_fullsized":"2024-09-10T18:56:24.018777+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"126'5","reported_seq":118,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022716+0000","last_change":"2024-09-10T18:54:37.509472+0000","last_active":"2024-09-10T18:56:24.022716+0000","last_peered":"2024-09-10T18:56:24.022716+0000","last_clean":"2024-09-10T18:56:24.022716+0000","last_became_active":"2024-09-10T18:54:37.507583+0000","last_became_peered":"2024-09-10T18:54:37.507583+0000","last_unstale":"2024-09-10T18:56:24.022716+0000","last_undegraded":"2024-09-10T18:56:24.022716+0000","last_fullsized":"2024-09-10T18:56:24.022716+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":217,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999971+0000","last_change":"2024-09-10T18:54:40.848354+0000","last_active":"2024-09-10T18:56:23.999971+0000","last_peered":"2024-09-10T18:56:23.999971+0000","last_clean":"2024-09-10T18:56:23.999971+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:56:23.999971+0000","last_undegraded":"2024-09-10T18:56:23.999971+0000","last_fullsized":"2024-09-10T18:56:23.999971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:54:40.848291+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.848291+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025906409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024693+0000","last_change":"2024-09-10T18:56:04.366386+0000","last_active":"2024-09-10T18:56:24.024693+0000","last_peered":"2024-09-10T18:56:24.024693+0000","last_clean":"2024-09-10T18:56:24.024693+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:56:24.024693+0000","last_undegraded":"2024-09-10T18:56:24.024693+0000","last_fullsized":"2024-09-10T18:56:24.024693+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:34.552199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:34.552199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032492139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":292,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.012261+0000","last_change":"2024-09-10T18:54:40.522553+0000","last_active":"2024-09-10T18:56:24.012261+0000","last_peered":"2024-09-10T18:56:24.012261+0000","last_clean":"2024-09-10T18:56:24.012261+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:56:24.012261+0000","last_undegraded":"2024-09-10T18:56:24.012261+0000","last_fullsized":"2024-09-10T18:56:24.012261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:40.522460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.522460+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027852699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":201,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-10T18:56:25.186092+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:56:25.186092+0000","last_peered":"2024-09-10T18:56:25.186092+0000","last_clean":"2024-09-10T18:56:25.186092+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:56:25.186092+0000","last_undegraded":"2024-09-10T18:56:25.186092+0000","last_fullsized":"2024-09-10T18:56:25.186092+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.837926+0000","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013629+0000","last_change":"2024-09-10T18:54:37.646414+0000","last_active":"2024-09-10T18:56:24.013629+0000","last_peered":"2024-09-10T18:56:24.013629+0000","last_clean":"2024-09-10T18:56:24.013629+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:56:24.013629+0000","last_undegraded":"2024-09-10T18:56:24.013629+0000","last_fullsized":"2024-09-10T18:56:24.013629+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"126'5","reported_seq":117,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013273+0000","last_change":"2024-09-10T18:54:37.626608+0000","last_active":"2024-09-10T18:56:24.013273+0000","last_peered":"2024-09-10T18:56:24.013273+0000","last_clean":"2024-09-10T18:56:24.013273+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:56:24.013273+0000","last_undegraded":"2024-09-10T18:56:24.013273+0000","last_fullsized":"2024-09-10T18:56:24.013273+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":211,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013125+0000","last_change":"2024-09-10T18:53:43.850799+0000","last_active":"2024-09-10T18:56:24.013125+0000","last_peered":"2024-09-10T18:56:24.013125+0000","last_clean":"2024-09-10T18:56:24.013125+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:56:24.013125+0000","last_undegraded":"2024-09-10T18:56:24.013125+0000","last_fullsized":"2024-09-10T18:56:24.013125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:53:43.850612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:53:43.850612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047610880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":207,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022756+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:56:24.022756+0000","last_peered":"2024-09-10T18:56:24.022756+0000","last_clean":"2024-09-10T18:56:24.022756+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:56:24.022756+0000","last_undegraded":"2024-09-10T18:56:24.022756+0000","last_fullsized":"2024-09-10T18:56:24.022756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":287,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024576+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:56:24.024576+0000","last_peered":"2024-09-10T18:56:24.024576+0000","last_clean":"2024-09-10T18:56:24.024576+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:56:24.024576+0000","last_undegraded":"2024-09-10T18:56:24.024576+0000","last_fullsized":"2024-09-10T18:56:24.024576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.003217+0000","last_change":"2024-09-10T18:54:53.310725+0000","last_active":"2024-09-10T18:56:24.003217+0000","last_peered":"2024-09-10T18:56:24.003217+0000","last_clean":"2024-09-10T18:56:24.003217+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:56:24.003217+0000","last_undegraded":"2024-09-10T18:56:24.003217+0000","last_fullsized":"2024-09-10T18:56:24.003217+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.310489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.310489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026931849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023757+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:56:24.023757+0000","last_peered":"2024-09-10T18:56:24.023757+0000","last_clean":"2024-09-10T18:56:24.023757+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:56:24.023757+0000","last_undegraded":"2024-09-10T18:56:24.023757+0000","last_fullsized":"2024-09-10T18:56:24.023757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"126'5","reported_seq":117,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.001356+0000","last_change":"2024-09-10T18:54:37.613960+0000","last_active":"2024-09-10T18:56:24.001356+0000","last_peered":"2024-09-10T18:56:24.001356+0000","last_clean":"2024-09-10T18:56:24.001356+0000","last_became_active":"2024-09-10T18:54:37.610866+0000","last_became_peered":"2024-09-10T18:54:37.610866+0000","last_unstale":"2024-09-10T18:56:24.001356+0000","last_undegraded":"2024-09-10T18:56:24.001356+0000","last_fullsized":"2024-09-10T18:56:24.001356+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":219,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.998938+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:56:23.998938+0000","last_peered":"2024-09-10T18:56:23.998938+0000","last_clean":"2024-09-10T18:56:23.998938+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:56:23.998938+0000","last_undegraded":"2024-09-10T18:56:23.998938+0000","last_fullsized":"2024-09-10T18:56:23.998938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.661931+0000","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009550+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:56:24.009550+0000","last_peered":"2024-09-10T18:56:24.009550+0000","last_clean":"2024-09-10T18:56:24.009550+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:56:24.009550+0000","last_undegraded":"2024-09-10T18:56:24.009550+0000","last_fullsized":"2024-09-10T18:56:24.009550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:26.721892+0000","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023825+0000","last_change":"2024-09-10T18:55:10.179358+0000","last_active":"2024-09-10T18:56:24.023825+0000","last_peered":"2024-09-10T18:56:24.023825+0000","last_clean":"2024-09-10T18:56:24.023825+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:56:24.023825+0000","last_undegraded":"2024-09-10T18:56:24.023825+0000","last_fullsized":"2024-09-10T18:56:24.023825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003833071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":204,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009253+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:56:24.009253+0000","last_peered":"2024-09-10T18:56:24.009253+0000","last_clean":"2024-09-10T18:56:24.009253+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:56:24.009253+0000","last_undegraded":"2024-09-10T18:56:24.009253+0000","last_fullsized":"2024-09-10T18:56:24.009253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.752093+0000","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009641+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:56:24.009641+0000","last_peered":"2024-09-10T18:56:24.009641+0000","last_clean":"2024-09-10T18:56:24.009641+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:56:24.009641+0000","last_undegraded":"2024-09-10T18:56:24.009641+0000","last_fullsized":"2024-09-10T18:56:24.009641+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.510275+0000","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"126'9","reported_seq":132,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017081+0000","last_change":"2024-09-10T18:56:09.704635+0000","last_active":"2024-09-10T18:56:24.017081+0000","last_peered":"2024-09-10T18:56:24.017081+0000","last_clean":"2024-09-10T18:56:24.017081+0000","last_became_active":"2024-09-10T18:54:37.646743+0000","last_became_peered":"2024-09-10T18:54:37.646743+0000","last_unstale":"2024-09-10T18:56:24.017081+0000","last_undegraded":"2024-09-10T18:56:24.017081+0000","last_fullsized":"2024-09-10T18:56:24.017081+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.575925+0000","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":226,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.002793+0000","last_change":"2024-09-10T18:55:06.199960+0000","last_active":"2024-09-10T18:56:24.002793+0000","last_peered":"2024-09-10T18:56:24.002793+0000","last_clean":"2024-09-10T18:56:24.002793+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:56:24.002793+0000","last_undegraded":"2024-09-10T18:56:24.002793+0000","last_fullsized":"2024-09-10T18:56:24.002793+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:55:06.199853+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:55:06.199853+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045914980000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022485+0000","last_change":"2024-09-10T18:54:46.299783+0000","last_active":"2024-09-10T18:56:24.022485+0000","last_peered":"2024-09-10T18:56:24.022485+0000","last_clean":"2024-09-10T18:56:24.022485+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:56:24.022485+0000","last_undegraded":"2024-09-10T18:56:24.022485+0000","last_fullsized":"2024-09-10T18:56:24.022485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:46.299630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:46.299630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00314569,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009830+0000","last_change":"2024-09-10T18:54:43.293336+0000","last_active":"2024-09-10T18:56:24.009830+0000","last_peered":"2024-09-10T18:56:24.009830+0000","last_clean":"2024-09-10T18:56:24.009830+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:56:24.009830+0000","last_undegraded":"2024-09-10T18:56:24.009830+0000","last_fullsized":"2024-09-10T18:56:24.009830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:43.293266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:54:43.293266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012102872000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007133+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:56:24.007133+0000","last_peered":"2024-09-10T18:56:24.007133+0000","last_clean":"2024-09-10T18:56:24.007133+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:56:24.007133+0000","last_undegraded":"2024-09-10T18:56:24.007133+0000","last_fullsized":"2024-09-10T18:56:24.007133+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.361501+0000","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009741+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:56:24.009741+0000","last_peered":"2024-09-10T18:56:24.009741+0000","last_clean":"2024-09-10T18:56:24.009741+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:56:24.009741+0000","last_undegraded":"2024-09-10T18:56:24.009741+0000","last_fullsized":"2024-09-10T18:56:24.009741+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.443415+0000","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017366+0000","last_change":"2024-09-10T18:54:37.725751+0000","last_active":"2024-09-10T18:56:24.017366+0000","last_peered":"2024-09-10T18:56:24.017366+0000","last_clean":"2024-09-10T18:56:24.017366+0000","last_became_active":"2024-09-10T18:54:37.716564+0000","last_became_peered":"2024-09-10T18:54:37.716564+0000","last_unstale":"2024-09-10T18:56:24.017366+0000","last_undegraded":"2024-09-10T18:56:24.017366+0000","last_fullsized":"2024-09-10T18:56:24.017366+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":207,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.024044+0000","last_change":"2024-09-10T18:54:54.957426+0000","last_active":"2024-09-10T18:56:24.024044+0000","last_peered":"2024-09-10T18:56:24.024044+0000","last_clean":"2024-09-10T18:56:24.024044+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:56:24.024044+0000","last_undegraded":"2024-09-10T18:56:24.024044+0000","last_fullsized":"2024-09-10T18:56:24.024044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:54:54.957316+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:54:54.957316+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039293499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023929+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:56:24.023929+0000","last_peered":"2024-09-10T18:56:24.023929+0000","last_clean":"2024-09-10T18:56:24.023929+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:56:24.023929+0000","last_undegraded":"2024-09-10T18:56:24.023929+0000","last_fullsized":"2024-09-10T18:56:24.023929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:26.005607+0000","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006718+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:56:24.006718+0000","last_peered":"2024-09-10T18:56:24.006718+0000","last_clean":"2024-09-10T18:56:24.006718+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:56:24.006718+0000","last_undegraded":"2024-09-10T18:56:24.006718+0000","last_fullsized":"2024-09-10T18:56:24.006718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.718506+0000","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":199,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.994602+0000","last_change":"2024-09-10T18:55:02.948472+0000","last_active":"2024-09-10T18:56:23.994602+0000","last_peered":"2024-09-10T18:56:23.994602+0000","last_clean":"2024-09-10T18:56:23.994602+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:56:23.994602+0000","last_undegraded":"2024-09-10T18:56:23.994602+0000","last_fullsized":"2024-09-10T18:56:23.994602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.948218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.948218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:31.308219+0000","scrub_duration":0.0034990780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.029846+0000","last_change":"2024-09-10T18:54:37.614702+0000","last_active":"2024-09-10T18:56:24.029846+0000","last_peered":"2024-09-10T18:56:24.029846+0000","last_clean":"2024-09-10T18:56:24.029846+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:56:24.029846+0000","last_undegraded":"2024-09-10T18:56:24.029846+0000","last_fullsized":"2024-09-10T18:56:24.029846+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.012373+0000","last_change":"2024-09-10T18:54:37.661524+0000","last_active":"2024-09-10T18:56:24.012373+0000","last_peered":"2024-09-10T18:56:24.012373+0000","last_clean":"2024-09-10T18:56:24.012373+0000","last_became_active":"2024-09-10T18:54:37.657896+0000","last_became_peered":"2024-09-10T18:54:37.657896+0000","last_unstale":"2024-09-10T18:56:24.012373+0000","last_undegraded":"2024-09-10T18:56:24.012373+0000","last_fullsized":"2024-09-10T18:56:24.012373+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":220,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023780+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:56:24.023780+0000","last_peered":"2024-09-10T18:56:24.023780+0000","last_clean":"2024-09-10T18:56:24.023780+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:56:24.023780+0000","last_undegraded":"2024-09-10T18:56:24.023780+0000","last_fullsized":"2024-09-10T18:56:24.023780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.059487+0000","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.019075+0000","last_change":"2024-09-10T18:55:08.609833+0000","last_active":"2024-09-10T18:56:24.019075+0000","last_peered":"2024-09-10T18:56:24.019075+0000","last_clean":"2024-09-10T18:56:24.019075+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:56:24.019075+0000","last_undegraded":"2024-09-10T18:56:24.019075+0000","last_fullsized":"2024-09-10T18:56:24.019075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.609702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.609702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006011+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:56:24.006011+0000","last_peered":"2024-09-10T18:56:24.006011+0000","last_clean":"2024-09-10T18:56:24.006011+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:56:24.006011+0000","last_undegraded":"2024-09-10T18:56:24.006011+0000","last_fullsized":"2024-09-10T18:56:24.006011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.486395+0000","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.010543+0000","last_change":"2024-09-10T18:53:50.696340+0000","last_active":"2024-09-10T18:56:24.010543+0000","last_peered":"2024-09-10T18:56:24.010543+0000","last_clean":"2024-09-10T18:56:24.010543+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:56:24.010543+0000","last_undegraded":"2024-09-10T18:56:24.010543+0000","last_fullsized":"2024-09-10T18:56:24.010543+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:50.696244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:53:50.696244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002749755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013321+0000","last_change":"2024-09-10T18:54:37.573584+0000","last_active":"2024-09-10T18:56:24.013321+0000","last_peered":"2024-09-10T18:56:24.013321+0000","last_clean":"2024-09-10T18:56:24.013321+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:56:24.013321+0000","last_undegraded":"2024-09-10T18:56:24.013321+0000","last_fullsized":"2024-09-10T18:56:24.013321+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":112,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022605+0000","last_change":"2024-09-10T18:54:37.570864+0000","last_active":"2024-09-10T18:56:24.022605+0000","last_peered":"2024-09-10T18:56:24.022605+0000","last_clean":"2024-09-10T18:56:24.022605+0000","last_became_active":"2024-09-10T18:54:37.565268+0000","last_became_peered":"2024-09-10T18:54:37.565268+0000","last_unstale":"2024-09-10T18:56:24.022605+0000","last_undegraded":"2024-09-10T18:56:24.022605+0000","last_fullsized":"2024-09-10T18:56:24.022605+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":193,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999411+0000","last_change":"2024-09-10T18:54:56.956889+0000","last_active":"2024-09-10T18:56:23.999411+0000","last_peered":"2024-09-10T18:56:23.999411+0000","last_clean":"2024-09-10T18:56:23.999411+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:56:23.999411+0000","last_undegraded":"2024-09-10T18:56:23.999411+0000","last_fullsized":"2024-09-10T18:56:23.999411+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:54:56.956621+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.956621+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050516290000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":229,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.027647+0000","last_change":"2024-09-10T18:54:52.411600+0000","last_active":"2024-09-10T18:56:24.027647+0000","last_peered":"2024-09-10T18:56:24.027647+0000","last_clean":"2024-09-10T18:56:24.027647+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:56:24.027647+0000","last_undegraded":"2024-09-10T18:56:24.027647+0000","last_fullsized":"2024-09-10T18:56:24.027647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:54:52.411417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:52.411417+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012519529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":206,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022908+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:56:24.022908+0000","last_peered":"2024-09-10T18:56:24.022908+0000","last_clean":"2024-09-10T18:56:24.022908+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:56:24.022908+0000","last_undegraded":"2024-09-10T18:56:24.022908+0000","last_fullsized":"2024-09-10T18:56:24.022908+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":203,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.023242+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:56:24.023242+0000","last_peered":"2024-09-10T18:56:24.023242+0000","last_clean":"2024-09-10T18:56:24.023242+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:56:24.023242+0000","last_undegraded":"2024-09-10T18:56:24.023242+0000","last_fullsized":"2024-09-10T18:56:24.023242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.517080+0000","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017170+0000","last_change":"2024-09-10T18:54:37.702704+0000","last_active":"2024-09-10T18:56:24.017170+0000","last_peered":"2024-09-10T18:56:24.017170+0000","last_clean":"2024-09-10T18:56:24.017170+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:56:24.017170+0000","last_undegraded":"2024-09-10T18:56:24.017170+0000","last_fullsized":"2024-09-10T18:56:24.017170+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"126'10","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:23.999522+0000","last_change":"2024-09-10T18:54:37.562844+0000","last_active":"2024-09-10T18:56:23.999522+0000","last_peered":"2024-09-10T18:56:23.999522+0000","last_clean":"2024-09-10T18:56:23.999522+0000","last_became_active":"2024-09-10T18:54:37.562584+0000","last_became_peered":"2024-09-10T18:54:37.562584+0000","last_unstale":"2024-09-10T18:56:23.999522+0000","last_undegraded":"2024-09-10T18:56:23.999522+0000","last_fullsized":"2024-09-10T18:56:23.999522+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":200,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022159+0000","last_change":"2024-09-10T18:56:09.619685+0000","last_active":"2024-09-10T18:56:24.022159+0000","last_peered":"2024-09-10T18:56:24.022159+0000","last_clean":"2024-09-10T18:56:24.022159+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:56:24.022159+0000","last_undegraded":"2024-09-10T18:56:24.022159+0000","last_fullsized":"2024-09-10T18:56:24.022159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.619567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.619567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.792631+0000","scrub_duration":0.0049369640000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":212,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.007028+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:56:24.007028+0000","last_peered":"2024-09-10T18:56:24.007028+0000","last_clean":"2024-09-10T18:56:24.007028+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:56:24.007028+0000","last_undegraded":"2024-09-10T18:56:24.007028+0000","last_fullsized":"2024-09-10T18:56:24.007028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":198,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.006775+0000","last_change":"2024-09-10T18:54:57.678574+0000","last_active":"2024-09-10T18:56:24.006775+0000","last_peered":"2024-09-10T18:56:24.006775+0000","last_clean":"2024-09-10T18:56:24.006775+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:56:24.006775+0000","last_undegraded":"2024-09-10T18:56:24.006775+0000","last_fullsized":"2024-09-10T18:56:24.006775+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.678482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.678482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024273350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":314,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.018978+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:56:24.018978+0000","last_peered":"2024-09-10T18:56:24.018978+0000","last_clean":"2024-09-10T18:56:24.018978+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:56:24.018978+0000","last_undegraded":"2024-09-10T18:56:24.018978+0000","last_fullsized":"2024-09-10T18:56:24.018978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":123,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.019161+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:56:24.019161+0000","last_peered":"2024-09-10T18:56:24.019161+0000","last_clean":"2024-09-10T18:56:24.019161+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:56:24.019161+0000","last_undegraded":"2024-09-10T18:56:24.019161+0000","last_fullsized":"2024-09-10T18:56:24.019161+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:38.976604+0000","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"126'2","reported_seq":124,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.029047+0000","last_change":"2024-09-10T18:56:16.659103+0000","last_active":"2024-09-10T18:56:24.029047+0000","last_peered":"2024-09-10T18:56:24.029047+0000","last_clean":"2024-09-10T18:56:24.029047+0000","last_became_active":"2024-09-10T18:54:37.639260+0000","last_became_peered":"2024-09-10T18:54:37.639260+0000","last_unstale":"2024-09-10T18:56:24.029047+0000","last_undegraded":"2024-09-10T18:56:24.029047+0000","last_fullsized":"2024-09-10T18:56:24.029047+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.376270+0000","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.004131+0000","last_change":"2024-09-10T18:54:44.250455+0000","last_active":"2024-09-10T18:56:24.004131+0000","last_peered":"2024-09-10T18:56:24.004131+0000","last_clean":"2024-09-10T18:56:24.004131+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:56:24.004131+0000","last_undegraded":"2024-09-10T18:56:24.004131+0000","last_fullsized":"2024-09-10T18:56:24.004131+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:44.250321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:54:44.250321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031387210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":215,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.008501+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:56:24.008501+0000","last_peered":"2024-09-10T18:56:24.008501+0000","last_clean":"2024-09-10T18:56:24.008501+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:56:24.008501+0000","last_undegraded":"2024-09-10T18:56:24.008501+0000","last_fullsized":"2024-09-10T18:56:24.008501+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:56:30.819641+0000","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":215,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.022388+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:56:24.022388+0000","last_peered":"2024-09-10T18:56:24.022388+0000","last_clean":"2024-09-10T18:56:24.022388+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:56:24.022388+0000","last_undegraded":"2024-09-10T18:56:24.022388+0000","last_fullsized":"2024-09-10T18:56:24.022388+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.550858+0000","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":202,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.005649+0000","last_change":"2024-09-10T18:54:38.683107+0000","last_active":"2024-09-10T18:56:24.005649+0000","last_peered":"2024-09-10T18:56:24.005649+0000","last_clean":"2024-09-10T18:56:24.005649+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:56:24.005649+0000","last_undegraded":"2024-09-10T18:56:24.005649+0000","last_fullsized":"2024-09-10T18:56:24.005649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:38.683009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:54:38.683009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041384250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":123,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-10T18:56:25.269527+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:56:25.269527+0000","last_peered":"2024-09-10T18:56:25.269527+0000","last_clean":"2024-09-10T18:56:25.269527+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:56:25.269527+0000","last_undegraded":"2024-09-10T18:56:25.269527+0000","last_fullsized":"2024-09-10T18:56:25.269527+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.803420+0000","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":114,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.013406+0000","last_change":"2024-09-10T18:54:37.660686+0000","last_active":"2024-09-10T18:56:24.013406+0000","last_peered":"2024-09-10T18:56:24.013406+0000","last_clean":"2024-09-10T18:56:24.013406+0000","last_became_active":"2024-09-10T18:54:37.660393+0000","last_became_peered":"2024-09-10T18:54:37.660393+0000","last_unstale":"2024-09-10T18:56:24.013406+0000","last_undegraded":"2024-09-10T18:56:24.013406+0000","last_fullsized":"2024-09-10T18:56:24.013406+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":192,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.017171+0000","last_change":"2024-09-10T18:53:42.782981+0000","last_active":"2024-09-10T18:56:24.017171+0000","last_peered":"2024-09-10T18:56:24.017171+0000","last_clean":"2024-09-10T18:56:24.017171+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:56:24.017171+0000","last_undegraded":"2024-09-10T18:56:24.017171+0000","last_fullsized":"2024-09-10T18:56:24.017171+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:42.782870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:53:42.782870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0085960840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":208,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.942380+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:56:24.942380+0000","last_peered":"2024-09-10T18:56:24.942380+0000","last_clean":"2024-09-10T18:56:24.942380+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:56:24.942380+0000","last_undegraded":"2024-09-10T18:56:24.942380+0000","last_fullsized":"2024-09-10T18:56:24.942380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.997928+0000","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":222,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.009799+0000","last_change":"2024-09-10T18:54:58.424151+0000","last_active":"2024-09-10T18:56:24.009799+0000","last_peered":"2024-09-10T18:56:24.009799+0000","last_clean":"2024-09-10T18:56:24.009799+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:56:24.009799+0000","last_undegraded":"2024-09-10T18:56:24.009799+0000","last_fullsized":"2024-09-10T18:56:24.009799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.424008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.424008+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.146668564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":194,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016649+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:56:24.016649+0000","last_peered":"2024-09-10T18:56:24.016649+0000","last_clean":"2024-09-10T18:56:24.016649+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:56:24.016649+0000","last_undegraded":"2024-09-10T18:56:24.016649+0000","last_fullsized":"2024-09-10T18:56:24.016649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.684081+0000","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":113,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.016275+0000","last_change":"2024-09-10T18:54:37.707157+0000","last_active":"2024-09-10T18:56:24.016275+0000","last_peered":"2024-09-10T18:56:24.016275+0000","last_clean":"2024-09-10T18:56:24.016275+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:56:24.016275+0000","last_undegraded":"2024-09-10T18:56:24.016275+0000","last_fullsized":"2024-09-10T18:56:24.016275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":122,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-10T18:56:24.027871+0000","last_change":"2024-09-10T18:56:21.650671+0000","last_active":"2024-09-10T18:56:24.027871+0000","last_peered":"2024-09-10T18:56:24.027871+0000","last_clean":"2024-09-10T18:56:24.027871+0000","last_became_active":"2024-09-10T18:54:37.583272+0000","last_became_peered":"2024-09-10T18:54:37.583272+0000","last_unstale":"2024-09-10T18:56:24.027871+0000","last_undegraded":"2024-09-10T18:56:24.027871+0000","last_fullsized":"2024-09-10T18:56:24.027871+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.232910+0000","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":141,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":3021,"num_objects":278,"num_object_clones":171,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":513,"num_write_kb":512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3416064,"data_stored":9063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1386,"ondisk_log_size":1386,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":403,"num_read_kb":398,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972449,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305632,"kb_used_data":4120,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439520,"statfs":{"total":95995035648,"available":95682068480,"internally_reserved":0,"allocated":4218880,"data_stored":2355046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.90500000000000003,"15min":0.92500000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.91600000000000004,"15min":0.92500000000000004},"min":{"1min":0.442,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.216,"5min":1.554,"15min":1.554},"last":0.88700000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.244,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96699999999999997,"15min":1.0269999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.36499999999999999},"max":{"1min":1.742,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.1279999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.038,"15min":1.0549999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47699999999999998},"max":{"1min":2.9199999999999999,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98999999999999999,"15min":0.97899999999999998},"min":{"1min":0.59799999999999998,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":1.2270000000000001,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.2889999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.97899999999999998,"15min":1.01},"min":{"1min":0.51400000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.042},{"interface":"front","average":{"1min":1.052,"5min":0.99299999999999999,"15min":0.97099999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.3050000000000002,"5min":3.3050000000000002,"15min":3.3050000000000002},"last":1.014}]},{"osd":4,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.996,"15min":1.0189999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.443},"max":{"1min":1.2809999999999999,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":1.339},{"interface":"front","average":{"1min":1.044,"5min":1.046,"15min":1.026},"min":{"1min":0.55500000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.5349999999999999,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.1830000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98199999999999998,"15min":0.99099999999999999},"min":{"1min":0.70599999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.26,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.379},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.67900000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.357,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.4930000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.64500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.383,"5min":2.2429999999999999,"15min":2.2749999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1850000000000001},"min":{"1min":0.73399999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.4390000000000001,"5min":2.0070000000000001,"15min":2.536},"last":1.4630000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299499,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309612,"kb_used_data":3940,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435540,"statfs":{"total":95995035648,"available":95677992960,"internally_reserved":0,"allocated":4034560,"data_stored":2353209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.85999999999999999,"15min":0.88400000000000001},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.121},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.97399999999999998,"15min":0.94499999999999995},"min":{"1min":0.45900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.9970000000000001,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.179,"15min":0.98299999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.633,"5min":18.494,"15min":18.494},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.88700000000000001,"15min":0.90400000000000003},"min":{"1min":0.55000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.02,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.89700000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.0269999999999999,"15min":0.94899999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0760000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.87,"5min":0.91600000000000004,"15min":0.92300000000000004},"min":{"1min":0.57899999999999996,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.095}]},{"osd":3,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0189999999999999,"15min":0.96699999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.97999999999999998,"15min":0.97399999999999998},"min":{"1min":0.68100000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.696,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.0609999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.97399999999999998,"15min":0.89000000000000001},"min":{"1min":0.66200000000000003,"5min":0.56699999999999995,"15min":0.52000000000000002},"max":{"1min":1.4239999999999999,"5min":1.712,"15min":1.712},"last":0.999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0189999999999999},"min":{"1min":0.68100000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.411,"5min":2.202,"15min":2.202},"last":1.032}]},{"osd":5,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.042,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.53500000000000003,"15min":0.47499999999999998},"max":{"1min":1.4430000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.141,"5min":1.1160000000000001,"15min":1.095},"min":{"1min":0.754,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.659,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.86399999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:55:46 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0580000000000001,"15min":1.044},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.333,"5min":1.516,"15min":1.573},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81299999999999994,"5min":0.68500000000000005,"15min":0.66900000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.149}]}]},{"osd":5,"up_from":43,"seq":184683593845,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305132,"kb_used_data":3612,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440020,"statfs":{"total":95995035648,"available":95682580480,"internally_reserved":0,"allocated":3698688,"data_stored":1774315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95699999999999996,"15min":0.878},"min":{"1min":0.50800000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.0739999999999998,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":1.121},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.94499999999999995,"15min":0.88400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.4379999999999999,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":1.1579999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.357,"15min":1.0189999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.0409999999999999,"5min":31.548999999999999,"15min":31.548999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.345,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.7729999999999997,"5min":31.847000000000001,"15min":31.847000000000001},"last":0.92600000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.605,"15min":1.127},"min":{"1min":0.53900000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":5.9809999999999999,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.0569999999999999,"15min":0.94799999999999995},"min":{"1min":0.61199999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":4.6950000000000003,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":1.2370000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.998,"15min":0.93899999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3140000000000001,"5min":5.9809999999999999,"15min":5.9809999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.6339999999999999,"15min":1.1679999999999999},"min":{"1min":0.66300000000000003,"5min":0.60999999999999999,"15min":0.57399999999999995},"max":{"1min":4.6550000000000002,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.3300000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.95999999999999996,"15min":0.83399999999999996},"min":{"1min":0.76500000000000001,"5min":0.59599999999999997,"15min":0.439},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":1.087},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.04,"15min":0.92100000000000004},"min":{"1min":0.755,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.042}]},{"osd":6,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.1759999999999999,"15min":1.0680000000000001},"min":{"1min":0.71999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":6.2149999999999999},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.845,"5min":1.246,"15min":1.107},"min":{"1min":0.79300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"last":1.371}]},{"osd":7,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.2110000000000001,"15min":1.121},"min":{"1min":0.89300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.1140000000000008,"5min":9.1140000000000008,"15min":9.1140000000000008},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.774,"5min":1.226,"15min":1.198},"min":{"1min":0.88100000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":6.532,"5min":6.532,"15min":6.532},"last":1.2889999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855489,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304828,"kb_used_data":3292,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440324,"statfs":{"total":95995035648,"available":95682891776,"internally_reserved":0,"allocated":3371008,"data_stored":1763071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82799999999999996,"15min":0.84199999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.113,"5min":1.546,"15min":1.546},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.85499999999999998},"min":{"1min":0.58999999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2290000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.83999999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.873},"min":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.266,"5min":1.623,"15min":1.623},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92800000000000005,"15min":0.878},"min":{"1min":0.49299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.3340000000000001,"5min":1.575,"15min":1.575},"last":1.0409999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95999999999999996,"15min":0.90400000000000003},"min":{"1min":0.59899999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2569999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96099999999999997,"15min":0.90200000000000002},"min":{"1min":0.66400000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.325,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.64700000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.91300000000000003},"min":{"1min":0.53800000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2829999999999999,"5min":1.4730000000000001,"15min":1.714},"last":1.179},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.97699999999999998,"15min":0.93500000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55600000000000005},"max":{"1min":1.329,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.78700000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.96099999999999997,"15min":0.88},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.52100000000000002},"max":{"1min":1.4790000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0389999999999999,"15min":0.93300000000000005},"min":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4610000000000001,"5min":1.528,"15min":1.7909999999999999},"last":1.0840000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0309999999999999,"15min":0.95599999999999996},"min":{"1min":0.66600000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.464,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":1.337},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.143,"15min":1.0840000000000001},"min":{"1min":0.77200000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.4670000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.147,"15min":1.165},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6200000000000001,"5min":2.008,"15min":2.008},"last":1.4219999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.70599999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.6120000000000001},"last":1.21}]}]},{"osd":3,"up_from":29,"seq":124554051724,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305068,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440084,"statfs":{"total":95995035648,"available":95682646016,"internally_reserved":0,"allocated":3641344,"data_stored":1763583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.109,"15min":0.78100000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":1.0229999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.627,"5min":1.1879999999999999,"15min":0.872},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.0009999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.2929999999999999,"15min":0.88600000000000001},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.97399999999999998,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.024,"15min":0.83899999999999997},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.95699999999999996,"5min":15.189,"15min":15.189},"last":0.71799999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.3089999999999999,"15min":0.96499999999999997},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.329,"5min":26.558,"15min":26.558},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.2330000000000001,"15min":0.91600000000000004},"min":{"1min":0.38300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.256,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.68999999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.238,"15min":1.079},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.4019999999999999,"5min":10.359999999999999,"15min":10.359999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.3360000000000001,"15min":1.165},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.9580000000000002,"5min":10.391,"15min":10.391},"last":1.0269999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:56:06 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.385,"15min":1.1930000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":9.2750000000000004,"5min":17.535,"15min":17.535},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0600000000000001,"15min":1.0069999999999999},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.747}]},{"osd":6,"last update":"Tue Sep 10 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.286,"15min":1.204},"min":{"1min":0.71099999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":9.3610000000000007,"5min":10.51,"15min":10.51},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.506,"5min":1.127,"15min":1.0900000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":10.773999999999999,"5min":10.773999999999999,"15min":10.773999999999999},"last":0.86499999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.3280000000000001,"15min":1.1870000000000001},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":5.3109999999999999,"5min":18.312999999999999,"15min":18.312999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.3899999999999999,"15min":1.2929999999999999},"min":{"1min":0.66500000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":9.3840000000000003,"5min":10.611000000000001,"15min":10.611000000000001},"last":0.88800000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378773,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309340,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435812,"statfs":{"total":95995035648,"available":95678271488,"internally_reserved":0,"allocated":3731456,"data_stored":1764588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":0.83999999999999997,"15min":0.70299999999999996},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":9.516,"5min":9.516,"15min":9.516},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.357,"5min":0.871,"15min":0.76500000000000001},"min":{"1min":0.32800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.577999999999999,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.79500000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.74199999999999999,"15min":0.69399999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":5.516,"5min":5.516,"15min":5.516},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.286,"5min":0.90400000000000003,"15min":0.79100000000000004},"min":{"1min":0.37,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":9.0649999999999995,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.85999999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91000000000000003,"15min":0.85099999999999998},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.3579999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.546,"5min":0.91700000000000004,"15min":0.84099999999999997},"min":{"1min":0.499,"5min":0.434,"15min":0.434},"max":{"1min":15.736000000000001,"5min":15.736000000000001,"15min":15.736000000000001},"last":1.139}]},{"osd":4,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.1950000000000001,"15min":1.038},"min":{"1min":0.77200000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":21.571999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1699999999999999,"15min":1.1060000000000001},"min":{"1min":0.621,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.377,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.97499999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1160000000000001,"15min":1.111},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.2720000000000002,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":1.2749999999999999},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.2430000000000001,"15min":1.0820000000000001},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":21.654,"5min":21.654,"15min":21.654},"last":1.429}]},{"osd":6,"last update":"Tue Sep 10 18:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2210000000000001,"15min":1.1930000000000001},"min":{"1min":0.66900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.956,"5min":1.2450000000000001,"15min":1.125},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":15.971,"5min":15.971,"15min":15.971},"last":0.91400000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.2310000000000001,"15min":1.2150000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":15.025,"5min":15.025,"15min":15.025},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.927,"5min":1.2789999999999999,"15min":1.171},"min":{"1min":0.65800000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.816000000000001,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.256}]}]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305096,"kb_used_data":3560,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440056,"statfs":{"total":95995035648,"available":95682617344,"internally_reserved":0,"allocated":3645440,"data_stored":1763953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:32 2024","interfaces":[{"interface":"back","average":{"1min":2.4969999999999999,"5min":1.036,"15min":0.73999999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":32.677999999999997,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.6880000000000002,"5min":1.109,"15min":0.81000000000000005},"min":{"1min":0.376,"5min":0.253,"15min":0.253},"max":{"1min":32.398000000000003,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.89400000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:55:32 2024","interfaces":[{"interface":"back","average":{"1min":2.75,"5min":1.1080000000000001,"15min":0.80700000000000005},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":26.097000000000001,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":3.2909999999999999,"5min":1.2969999999999999,"15min":0.92300000000000004},"min":{"1min":0.39000000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":32.363,"5min":32.363,"15min":32.363},"last":1.4370000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.1930000000000001,"15min":0.94699999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.2649999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":1.29},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.4910000000000001,"15min":0.98499999999999999},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89600000000000002,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.76300000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1599999999999999,"15min":1.0529999999999999},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3320000000000001,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.6299999999999999,"15min":1.1359999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.014}]},{"osd":5,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.3700000000000001,"5min":1.2869999999999999,"15min":1.0980000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":26.050999999999998,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.056},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.2929999999999999,"15min":1.1830000000000001},"min":{"1min":0.69199999999999995,"5min":0.63100000000000001,"15min":0.61199999999999999},"max":{"1min":12.84,"5min":12.84,"15min":12.84},"last":1.6719999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.891,"5min":1.4670000000000001,"15min":1.206},"min":{"1min":0.78200000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":25.608000000000001,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.5289999999999999},{"interface":"front","average":{"1min":3.0990000000000002,"5min":1.458,"15min":1.202},"min":{"1min":0.622,"5min":0.622,"15min":0.41499999999999998},"max":{"1min":33.643000000000001,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.3340000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.4670000000000001,"15min":1.304},"min":{"1min":0.83999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.565999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.722},{"interface":"front","average":{"1min":3.3900000000000001,"5min":1.5629999999999999,"15min":1.2849999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.48399999999999999},"max":{"1min":33.673999999999999,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.3979999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309632,"kb_used_data":3952,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435520,"statfs":{"total":95995035648,"available":95677972480,"internally_reserved":0,"allocated":4046848,"data_stored":2363635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":0.77700000000000002,"15min":0.64900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.4659999999999993,"5min":8.4659999999999993,"15min":8.4659999999999993},"last":0.625},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.88300000000000001,"15min":0.73099999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":10.894,"5min":10.894,"15min":10.894},"last":0.72699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.5920000000000001,"15min":0.98699999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.2789999999999999,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.214,"15min":0.873},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.0640000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.0529999999999999,"15min":0.84999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0669999999999999,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.872},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.006},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3280000000000001,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.80100000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:56:19 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.131,"15min":1.014},"min":{"1min":0.69299999999999995,"5min":0.5,"15min":0.5},"max":{"1min":5.5369999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.149,"15min":1.0760000000000001},"min":{"1min":0.71399999999999997,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":4.2389999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.085}]},{"osd":5,"last update":"Tue Sep 10 18:56:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2390000000000001,"15min":1.0860000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.7989999999999999,"5min":10.048,"15min":10.048},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1339999999999999,"15min":1.1000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.4339999999999999,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":1.1419999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.161,"15min":1.1240000000000001},"min":{"1min":0.81699999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.9550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.997},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.3140000000000001,"15min":1.179},"min":{"1min":0.68400000000000005,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":11.894,"5min":11.894,"15min":11.894},"last":1.246}]},{"osd":7,"last update":"Tue Sep 10 18:55:48 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1850000000000001,"15min":1.1699999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.266,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1950000000000001,"15min":1.2749999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.202}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":1297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":1546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":1434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":1249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":141,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:56:29.284 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:56:29.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:29.356 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:29.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:29.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.200218+0000 osd.0 (osd.0) 168 : cluster [DBG] 16.0 scrub starts 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.205236+0000 osd.0 (osd.0) 169 : cluster [DBG] 16.0 scrub ok 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.544688+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.e scrub starts 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.549931+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.e scrub ok 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:27.748222+0000 mgr.y (mgr.24526) 296 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:29.608 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.786372+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.17 scrub starts 2024-09-10T18:56:29.609 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.790047+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.17 scrub ok 2024-09-10T18:56:29.609 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.915631+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:56:29.609 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:27.920573+0000 osd.3 (osd.3) 184 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:56:29.609 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:28.039855+0000 mgr.y (mgr.24526) 297 : audit [DBG] from='client.48911 v1:172.21.15.83:0/2428558554' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.200218+0000 osd.0 (osd.0) 168 : cluster [DBG] 16.0 scrub starts 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.205236+0000 osd.0 (osd.0) 169 : cluster [DBG] 16.0 scrub ok 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.544688+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.e scrub starts 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.549931+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.e scrub ok 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:27.748222+0000 mgr.y (mgr.24526) 296 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.786372+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.17 scrub starts 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.790047+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.17 scrub ok 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.915631+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:27.920573+0000 osd.3 (osd.3) 184 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:28.039855+0000 mgr.y (mgr.24526) 297 : audit [DBG] from='client.48911 v1:172.21.15.83:0/2428558554' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:29.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.572911+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.577576+0000 osd.7 (osd.7) 169 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.690906+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.a scrub starts 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.694230+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.a scrub ok 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:28.725902+0000 mon.c (mon.2) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.023283+0000 mon.a (mon.0) 1592 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.033915+0000 mon.a (mon.0) 1593 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.034129+0000 mon.a (mon.0) 1594 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.034432+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.035589+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.036378+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.038811+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: audit 2024-09-10T18:56:29.039713+0000 mon.a (mon.0) 1597 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:29 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.207638+0000 mgr.y (mgr.24526) 298 : cluster [DBG] pgmap v333: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:29.618 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:29.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:29.754 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:29.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:29.889 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:29.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:29.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.572911+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.577576+0000 osd.7 (osd.7) 169 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.690906+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.a scrub starts 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.694230+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.a scrub ok 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:28.725902+0000 mon.c (mon.2) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.023283+0000 mon.a (mon.0) 1592 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.033915+0000 mon.a (mon.0) 1593 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.034129+0000 mon.a (mon.0) 1594 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.034432+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:56:29.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.035589+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.036378+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.038811+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: audit 2024-09-10T18:56:29.039713+0000 mon.a (mon.0) 1597 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.207638+0000 mgr.y (mgr.24526) 298 : cluster [DBG] pgmap v333: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.200218+0000 osd.0 (osd.0) 168 : cluster [DBG] 16.0 scrub starts 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.205236+0000 osd.0 (osd.0) 169 : cluster [DBG] 16.0 scrub ok 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.544688+0000 osd.7 (osd.7) 166 : cluster [DBG] 3.e scrub starts 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.549931+0000 osd.7 (osd.7) 167 : cluster [DBG] 3.e scrub ok 2024-09-10T18:56:29.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:27.748222+0000 mgr.y (mgr.24526) 296 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.786372+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.17 scrub starts 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.790047+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.17 scrub ok 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.915631+0000 osd.3 (osd.3) 183 : cluster [DBG] 3.1e scrub starts 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:27.920573+0000 osd.3 (osd.3) 184 : cluster [DBG] 3.1e scrub ok 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:28.039855+0000 mgr.y (mgr.24526) 297 : audit [DBG] from='client.48911 v1:172.21.15.83:0/2428558554' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.572911+0000 osd.7 (osd.7) 168 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.577576+0000 osd.7 (osd.7) 169 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.690906+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.a scrub starts 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.694230+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.a scrub ok 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:28.725902+0000 mon.c (mon.2) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.023283+0000 mon.a (mon.0) 1592 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.033915+0000 mon.a (mon.0) 1593 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-17"}]': finished 2024-09-10T18:56:29.989 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.034129+0000 mon.a (mon.0) 1594 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.034432+0000 mon.a (mon.0) 1595 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.035589+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.83:0/599324055' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.036378+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48887 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]: dispatch 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.038811+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: audit 2024-09-10T18:56:29.039713+0000 mon.a (mon.0) 1597 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:29.990 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:29 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.207638+0000 mgr.y (mgr.24526) 298 : cluster [DBG] pgmap v333: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:29.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:30.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:30.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:30.152 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:30.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:30.280 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:30.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:30.407 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:30.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:30.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:30.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:30.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.789022+0000 osd.2 (osd.2) 99 : cluster [DBG] 16.a scrub starts 2024-09-10T18:56:30.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: cluster 2024-09-10T18:56:28.793048+0000 osd.2 (osd.2) 100 : cluster [DBG] 16.a scrub ok 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: audit 2024-09-10T18:56:29.564019+0000 mon.a (mon.0) 1598 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: audit 2024-09-10T18:56:29.566058+0000 mon.c (mon.2) 356 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: audit 2024-09-10T18:56:29.727712+0000 mon.c (mon.2) 357 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: audit 2024-09-10T18:56:30.033337+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: audit 2024-09-10T18:56:30.033509+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:30.655 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:30 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.033798+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:56:30.672 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:30.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.789022+0000 osd.2 (osd.2) 99 : cluster [DBG] 16.a scrub starts 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: cluster 2024-09-10T18:56:28.793048+0000 osd.2 (osd.2) 100 : cluster [DBG] 16.a scrub ok 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: audit 2024-09-10T18:56:29.564019+0000 mon.a (mon.0) 1598 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: audit 2024-09-10T18:56:29.566058+0000 mon.c (mon.2) 356 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: audit 2024-09-10T18:56:29.727712+0000 mon.c (mon.2) 357 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: audit 2024-09-10T18:56:30.033337+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:30.785 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: audit 2024-09-10T18:56:30.033509+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.033798+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.789022+0000 osd.2 (osd.2) 99 : cluster [DBG] 16.a scrub starts 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: cluster 2024-09-10T18:56:28.793048+0000 osd.2 (osd.2) 100 : cluster [DBG] 16.a scrub ok 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: audit 2024-09-10T18:56:29.564019+0000 mon.a (mon.0) 1598 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: audit 2024-09-10T18:56:29.566058+0000 mon.c (mon.2) 356 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: audit 2024-09-10T18:56:29.727712+0000 mon.c (mon.2) 357 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:30.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: audit 2024-09-10T18:56:30.033337+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48887 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi083-77640-29"}]': finished 2024-09-10T18:56:30.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: audit 2024-09-10T18:56:30.033509+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi083-77648-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:30.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:30 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.033798+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-09-10T18:56:30.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:30.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:30.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:30.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:31.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:31.069 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:31.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:31.202 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:31.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:31.336 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:31.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:31.473 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:31.559 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:31.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:31.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:31.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:31.747 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.223632+0000 osd.0 (osd.0) 170 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.227183+0000 osd.0 (osd.0) 171 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.922870+0000 osd.3 (osd.3) 185 : cluster [DBG] 3.b scrub starts 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:29.927454+0000 osd.3 (osd.3) 186 : cluster [DBG] 3.b scrub ok 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.062635+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.092399+0000 mon.b (mon.1) 191 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.187986+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.188514+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.494086+0000 osd.7 (osd.7) 170 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.496121+0000 osd.7 (osd.7) 171 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.552605+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.554312+0000 mon.a (mon.0) 1603 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.556090+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.558207+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.561432+0000 mon.a (mon.0) 1604 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.562283+0000 mon.a (mon.0) 1605 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.682110+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub starts 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.684579+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.d scrub ok 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:30.729128+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.912452+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.223632+0000 osd.0 (osd.0) 170 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.227183+0000 osd.0 (osd.0) 171 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.922870+0000 osd.3 (osd.3) 185 : cluster [DBG] 3.b scrub starts 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:29.927454+0000 osd.3 (osd.3) 186 : cluster [DBG] 3.b scrub ok 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.062635+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.092399+0000 mon.b (mon.1) 191 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.187986+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.188514+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.494086+0000 osd.7 (osd.7) 170 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.496121+0000 osd.7 (osd.7) 171 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.552605+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.554312+0000 mon.a (mon.0) 1603 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.556090+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.558207+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.561432+0000 mon.a (mon.0) 1604 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.562283+0000 mon.a (mon.0) 1605 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.682110+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub starts 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.684579+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.d scrub ok 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:30.729128+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.912452+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:30.915371+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:56:31.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.052393+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.054816+0000 mon.a (mon.0) 1606 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]': finished 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.054987+0000 mon.a (mon.0) 1607 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.055080+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.058946+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.061896+0000 mon.a (mon.0) 1609 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: audit 2024-09-10T18:56:31.066782+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.208562+0000 mgr.y (mgr.24526) 299 : cluster [DBG] pgmap v336: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 KiB/s wr, 171 op/s 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:30.915371+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.052393+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.054816+0000 mon.a (mon.0) 1606 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]': finished 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.054987+0000 mon.a (mon.0) 1607 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:31.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.055080+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:56:31.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.058946+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.061896+0000 mon.a (mon.0) 1609 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: audit 2024-09-10T18:56:31.066782+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:31 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.208562+0000 mgr.y (mgr.24526) 299 : cluster [DBG] pgmap v336: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 KiB/s wr, 171 op/s 2024-09-10T18:56:31.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:31.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.223632+0000 osd.0 (osd.0) 170 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.227183+0000 osd.0 (osd.0) 171 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.922870+0000 osd.3 (osd.3) 185 : cluster [DBG] 3.b scrub starts 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:29.927454+0000 osd.3 (osd.3) 186 : cluster [DBG] 3.b scrub ok 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.062635+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.83:0/1893320135' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.092399+0000 mon.b (mon.1) 191 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.187986+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.188514+0000 osd.1 (osd.1) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.494086+0000 osd.7 (osd.7) 170 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.496121+0000 osd.7 (osd.7) 171 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.552605+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48917 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T18:56:31.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.554312+0000 mon.a (mon.0) 1603 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.556090+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.558207+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.561432+0000 mon.a (mon.0) 1604 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.562283+0000 mon.a (mon.0) 1605 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.682110+0000 osd.1 (osd.1) 167 : cluster [DBG] 2.d scrub starts 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.684579+0000 osd.1 (osd.1) 168 : cluster [DBG] 2.d scrub ok 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:30.729128+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.912452+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:30.915371+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.052393+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.054816+0000 mon.a (mon.0) 1606 : audit [INF] from='client.48917 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi083-77648-33", "var": "eio", "val": "true"}]': finished 2024-09-10T18:56:31.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.054987+0000 mon.a (mon.0) 1607 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi083-77640-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:31.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.055080+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-09-10T18:56:31.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.058946+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.061896+0000 mon.a (mon.0) 1609 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:31.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: audit 2024-09-10T18:56:31.066782+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:31.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:31 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.208562+0000 mgr.y (mgr.24526) 299 : cluster [DBG] pgmap v336: 292 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 234 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 KiB/s wr, 171 op/s 2024-09-10T18:56:31.878 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:31.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:31.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.016 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:32.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:32.149 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:32.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:32.287 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:32.390 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:32.423 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:32.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.563 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:32.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:32.699 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:32.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.837 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:32.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:32.950 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:32] "GET /metrics HTTP/1.1" 200 233436 "" "Prometheus/2.43.0" 2024-09-10T18:56:32.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:32.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:33.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:33.097 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:33.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.247779+0000 osd.0 (osd.0) 172 : cluster [DBG] 12.e scrub starts 2024-09-10T18:56:33.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.251937+0000 osd.0 (osd.0) 173 : cluster [DBG] 12.e scrub ok 2024-09-10T18:56:33.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.689626+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:56:33.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.692333+0000 osd.1 (osd.1) 170 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: audit 2024-09-10T18:56:31.730394+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.860908+0000 osd.6 (osd.6) 125 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:31.864323+0000 osd.6 (osd.6) 126 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: audit 2024-09-10T18:56:32.061443+0000 mon.a (mon.0) 1611 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: audit 2024-09-10T18:56:32.068699+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.514025+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: audit 2024-09-10T18:56:32.516486+0000 mon.a (mon.0) 1613 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:32 smithi165 bash[22591]: audit 2024-09-10T18:56:32.732009+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.247779+0000 osd.0 (osd.0) 172 : cluster [DBG] 12.e scrub starts 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.251937+0000 osd.0 (osd.0) 173 : cluster [DBG] 12.e scrub ok 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.689626+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.692333+0000 osd.1 (osd.1) 170 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: audit 2024-09-10T18:56:31.730394+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.860908+0000 osd.6 (osd.6) 125 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:31.864323+0000 osd.6 (osd.6) 126 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: audit 2024-09-10T18:56:32.061443+0000 mon.a (mon.0) 1611 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:33.232 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: audit 2024-09-10T18:56:32.068699+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.514025+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: audit 2024-09-10T18:56:32.516486+0000 mon.a (mon.0) 1613 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[18629]: audit 2024-09-10T18:56:32.732009+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.247779+0000 osd.0 (osd.0) 172 : cluster [DBG] 12.e scrub starts 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.251937+0000 osd.0 (osd.0) 173 : cluster [DBG] 12.e scrub ok 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.689626+0000 osd.1 (osd.1) 169 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.692333+0000 osd.1 (osd.1) 170 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: audit 2024-09-10T18:56:31.730394+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.860908+0000 osd.6 (osd.6) 125 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:31.864323+0000 osd.6 (osd.6) 126 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: audit 2024-09-10T18:56:32.061443+0000 mon.a (mon.0) 1611 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:33.233 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: audit 2024-09-10T18:56:32.068699+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.234 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.514025+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-09-10T18:56:33.234 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: audit 2024-09-10T18:56:32.516486+0000 mon.a (mon.0) 1613 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:33.234 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:32 smithi083 bash[32381]: audit 2024-09-10T18:56:32.732009+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:33.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:33.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:33.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:33.418 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:56:33.418 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T18:56:33.419 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T18:56:33.419 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:56:33.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:33.509 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:33.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:33.644 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:33.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:33.782 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:33.805 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:33.806 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:33.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:33.919 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:33.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:34.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:34.031 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.256956+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:56:34.031 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.705159+0000 osd.1 (osd.1) 171 : cluster [DBG] 12.18 scrub starts 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.708448+0000 osd.1 (osd.1) 172 : cluster [DBG] 12.18 scrub ok 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.792556+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 deep-scrub starts 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.795917+0000 osd.4 (osd.4) 192 : cluster [DBG] 16.2 deep-scrub ok 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.985252+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:32.989633+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.063932+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:34.032 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.064120+0000 mon.a (mon.0) 1615 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.064323+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.065245+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.067508+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3886170435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.069366+0000 mon.a (mon.0) 1617 : audit [INF] from='client.48929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.070076+0000 mon.a (mon.0) 1618 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.209500+0000 mgr.y (mgr.24526) 300 : cluster [DBG] pgmap v339: 300 pgs: 72 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.676487+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:56:34.033 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.679668+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:56:34.034 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[18629]: audit 2024-09-10T18:56:33.733547+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:34.034 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.256956+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:56:34.034 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.705159+0000 osd.1 (osd.1) 171 : cluster [DBG] 12.18 scrub starts 2024-09-10T18:56:34.034 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.708448+0000 osd.1 (osd.1) 172 : cluster [DBG] 12.18 scrub ok 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.792556+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 deep-scrub starts 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.795917+0000 osd.4 (osd.4) 192 : cluster [DBG] 16.2 deep-scrub ok 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.985252+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:32.989633+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.063932+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.064120+0000 mon.a (mon.0) 1615 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.064323+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:56:34.035 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.065245+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.067508+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3886170435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.069366+0000 mon.a (mon.0) 1617 : audit [INF] from='client.48929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.070076+0000 mon.a (mon.0) 1618 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.209500+0000 mgr.y (mgr.24526) 300 : cluster [DBG] pgmap v339: 300 pgs: 72 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.676487+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.679668+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:56:34.036 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:33 smithi083 bash[32381]: audit 2024-09-10T18:56:33.733547+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:34.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:34.153 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:34.180 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:34.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.256956+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.705159+0000 osd.1 (osd.1) 171 : cluster [DBG] 12.18 scrub starts 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.708448+0000 osd.1 (osd.1) 172 : cluster [DBG] 12.18 scrub ok 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.792556+0000 osd.4 (osd.4) 191 : cluster [DBG] 16.2 deep-scrub starts 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.795917+0000 osd.4 (osd.4) 192 : cluster [DBG] 16.2 deep-scrub ok 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.985252+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:32.989633+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.063932+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi083-77640-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:34.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.064120+0000 mon.a (mon.0) 1615 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.064323+0000 mon.a (mon.0) 1616 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.065245+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.067508+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3886170435' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.069366+0000 mon.a (mon.0) 1617 : audit [INF] from='client.48929 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.070076+0000 mon.a (mon.0) 1618 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.209500+0000 mgr.y (mgr.24526) 300 : cluster [DBG] pgmap v339: 300 pgs: 72 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.676487+0000 osd.1 (osd.1) 173 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.679668+0000 osd.1 (osd.1) 174 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:56:34.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:33 smithi165 bash[22591]: audit 2024-09-10T18:56:33.733547+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:34.316 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:34.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:34.454 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:34.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:34.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T18:56:34.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:34.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:34.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:34.820 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:34.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:34.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:34.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:34.989 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:34.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:35.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:35.125 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:35.228 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:35.242 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.220920+0000 osd.5 (osd.5) 155 : cluster [DBG] 16.1f deep-scrub starts 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.227218+0000 osd.5 (osd.5) 156 : cluster [DBG] 16.1f deep-scrub ok 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.801375+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.805758+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.960262+0000 osd.3 (osd.3) 191 : cluster [DBG] 6.1f deep-scrub starts 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:33.964094+0000 osd.3 (osd.3) 192 : cluster [DBG] 6.1f deep-scrub ok 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: audit 2024-09-10T18:56:34.009468+0000 mon.a (mon.0) 1619 : audit [INF] from='client.48929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: audit 2024-09-10T18:56:34.009647+0000 mon.a (mon.0) 1620 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.009986+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: audit 2024-09-10T18:56:34.021969+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: audit 2024-09-10T18:56:34.026350+0000 mon.a (mon.0) 1622 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.057481+0000 mon.a (mon.0) 1623 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:35.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.654926+0000 osd.1 (osd.1) 175 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.658563+0000 osd.1 (osd.1) 176 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[32381]: audit 2024-09-10T18:56:34.735001+0000 mon.c (mon.2) 364 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.220920+0000 osd.5 (osd.5) 155 : cluster [DBG] 16.1f deep-scrub starts 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.227218+0000 osd.5 (osd.5) 156 : cluster [DBG] 16.1f deep-scrub ok 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.801375+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.805758+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.960262+0000 osd.3 (osd.3) 191 : cluster [DBG] 6.1f deep-scrub starts 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:33.964094+0000 osd.3 (osd.3) 192 : cluster [DBG] 6.1f deep-scrub ok 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: audit 2024-09-10T18:56:34.009468+0000 mon.a (mon.0) 1619 : audit [INF] from='client.48929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: audit 2024-09-10T18:56:34.009647+0000 mon.a (mon.0) 1620 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:35.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.009986+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: audit 2024-09-10T18:56:34.021969+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: audit 2024-09-10T18:56:34.026350+0000 mon.a (mon.0) 1622 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.057481+0000 mon.a (mon.0) 1623 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.654926+0000 osd.1 (osd.1) 175 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.658563+0000 osd.1 (osd.1) 176 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:56:35.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:35 smithi083 bash[18629]: audit 2024-09-10T18:56:34.735001+0000 mon.c (mon.2) 364 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:35.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:35.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.220920+0000 osd.5 (osd.5) 155 : cluster [DBG] 16.1f deep-scrub starts 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.227218+0000 osd.5 (osd.5) 156 : cluster [DBG] 16.1f deep-scrub ok 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.801375+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.805758+0000 osd.4 (osd.4) 194 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.960262+0000 osd.3 (osd.3) 191 : cluster [DBG] 6.1f deep-scrub starts 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:33.964094+0000 osd.3 (osd.3) 192 : cluster [DBG] 6.1f deep-scrub ok 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: audit 2024-09-10T18:56:34.009468+0000 mon.a (mon.0) 1619 : audit [INF] from='client.48929 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi083-77648-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: audit 2024-09-10T18:56:34.009647+0000 mon.a (mon.0) 1620 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.009986+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: audit 2024-09-10T18:56:34.021969+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: audit 2024-09-10T18:56:34.026350+0000 mon.a (mon.0) 1622 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]: dispatch 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.057481+0000 mon.a (mon.0) 1623 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.654926+0000 osd.1 (osd.1) 175 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.658563+0000 osd.1 (osd.1) 176 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:56:35.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:35 smithi165 bash[22591]: audit 2024-09-10T18:56:34.735001+0000 mon.c (mon.2) 364 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:35.391 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:35.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:35.527 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:35.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:35.666 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:35.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:35.804 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:35.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:35.941 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:35.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:36.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:36.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.210801+0000 osd.0 (osd.0) 174 : cluster [DBG] 12.c scrub starts 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.214292+0000 osd.0 (osd.0) 175 : cluster [DBG] 12.c scrub ok 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.849973+0000 osd.4 (osd.4) 195 : cluster [DBG] 2.f scrub starts 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.853821+0000 osd.4 (osd.4) 196 : cluster [DBG] 2.f scrub ok 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.921502+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:34.925664+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.004791+0000 mon.a (mon.0) 1624 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:36.182 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.013638+0000 mon.a (mon.0) 1625 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]': finished 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.013935+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.015693+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.028494+0000 mon.a (mon.0) 1627 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.074369+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.210310+0000 mgr.y (mgr.24526) 301 : cluster [DBG] pgmap v342: 260 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.489969+0000 mon.a (mon.0) 1628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:36 smithi165 bash[22591]: audit 2024-09-10T18:56:35.736224+0000 mon.c (mon.2) 365 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:36.205 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.210801+0000 osd.0 (osd.0) 174 : cluster [DBG] 12.c scrub starts 2024-09-10T18:56:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.214292+0000 osd.0 (osd.0) 175 : cluster [DBG] 12.c scrub ok 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.849973+0000 osd.4 (osd.4) 195 : cluster [DBG] 2.f scrub starts 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.853821+0000 osd.4 (osd.4) 196 : cluster [DBG] 2.f scrub ok 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.921502+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:34.925664+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.004791+0000 mon.a (mon.0) 1624 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:36.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.013638+0000 mon.a (mon.0) 1625 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]': finished 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.013935+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.015693+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.028494+0000 mon.a (mon.0) 1627 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.074369+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.210310+0000 mgr.y (mgr.24526) 301 : cluster [DBG] pgmap v342: 260 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:36.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.489969+0000 mon.a (mon.0) 1628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[18629]: audit 2024-09-10T18:56:35.736224+0000 mon.c (mon.2) 365 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:36.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.210801+0000 osd.0 (osd.0) 174 : cluster [DBG] 12.c scrub starts 2024-09-10T18:56:36.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.214292+0000 osd.0 (osd.0) 175 : cluster [DBG] 12.c scrub ok 2024-09-10T18:56:36.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.849973+0000 osd.4 (osd.4) 195 : cluster [DBG] 2.f scrub starts 2024-09-10T18:56:36.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.853821+0000 osd.4 (osd.4) 196 : cluster [DBG] 2.f scrub ok 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.921502+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:34.925664+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.004791+0000 mon.a (mon.0) 1624 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.013638+0000 mon.a (mon.0) 1625 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-19", "mode": "writeback"}]': finished 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.013935+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.015693+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.028494+0000 mon.a (mon.0) 1627 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.074369+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.210310+0000 mgr.y (mgr.24526) 301 : cluster [DBG] pgmap v342: 260 pgs: 32 unknown, 15 active+clean+snaptrim_wait, 11 active+clean+snaptrim, 202 active+clean; 587 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.489969+0000 mon.a (mon.0) 1628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:36.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:36 smithi083 bash[32381]: audit 2024-09-10T18:56:35.736224+0000 mon.c (mon.2) 365 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:36.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:36.344 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:36.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:36.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:36.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:36.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:36.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:36.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:36.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:36.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:36.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:37.019 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:37.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:37.157 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:37.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:37.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.179132+0000 osd.0 (osd.0) 176 : cluster [DBG] 12.3 scrub starts 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.182593+0000 osd.0 (osd.0) 177 : cluster [DBG] 12.3 scrub ok 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.233172+0000 osd.5 (osd.5) 157 : cluster [DBG] 16.1b scrub starts 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.237557+0000 osd.5 (osd.5) 158 : cluster [DBG] 16.1b scrub ok 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.878109+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.881520+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.988590+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c scrub starts 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:35.993962+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.c scrub ok 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.037077+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.037442+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.038922+0000 mon.a (mon.0) 1629 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.039029+0000 mon.a (mon.0) 1630 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.039106+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:56:37.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.041279+0000 mon.a (mon.0) 1632 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.042365+0000 mon.a (mon.0) 1633 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.044459+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.83:0/4233398667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.051079+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.267504+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.272226+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[18629]: audit 2024-09-10T18:56:36.737761+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.179132+0000 osd.0 (osd.0) 176 : cluster [DBG] 12.3 scrub starts 2024-09-10T18:56:37.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.182593+0000 osd.0 (osd.0) 177 : cluster [DBG] 12.3 scrub ok 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.233172+0000 osd.5 (osd.5) 157 : cluster [DBG] 16.1b scrub starts 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.237557+0000 osd.5 (osd.5) 158 : cluster [DBG] 16.1b scrub ok 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.878109+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.881520+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.988590+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c scrub starts 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:35.993962+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.c scrub ok 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.037077+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.037442+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.038922+0000 mon.a (mon.0) 1629 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.039029+0000 mon.a (mon.0) 1630 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.039106+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.041279+0000 mon.a (mon.0) 1632 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.295 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.042365+0000 mon.a (mon.0) 1633 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.044459+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.83:0/4233398667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.051079+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.267504+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:56:37.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.272226+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:56:37.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:37 smithi083 bash[32381]: audit 2024-09-10T18:56:36.737761+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:37.296 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:37.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.179132+0000 osd.0 (osd.0) 176 : cluster [DBG] 12.3 scrub starts 2024-09-10T18:56:37.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.182593+0000 osd.0 (osd.0) 177 : cluster [DBG] 12.3 scrub ok 2024-09-10T18:56:37.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.233172+0000 osd.5 (osd.5) 157 : cluster [DBG] 16.1b scrub starts 2024-09-10T18:56:37.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.237557+0000 osd.5 (osd.5) 158 : cluster [DBG] 16.1b scrub ok 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.878109+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.881520+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.988590+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.c scrub starts 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:35.993962+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.c scrub ok 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.037077+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.83:0/910408963' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.037442+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.038922+0000 mon.a (mon.0) 1629 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.039029+0000 mon.a (mon.0) 1630 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.039106+0000 mon.a (mon.0) 1631 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.041279+0000 mon.a (mon.0) 1632 : audit [INF] from='client.49441 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]: dispatch 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.042365+0000 mon.a (mon.0) 1633 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]: dispatch 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.044459+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.83:0/4233398667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.051079+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:37.300 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.267504+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:56:37.300 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.272226+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:56:37.300 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[22591]: audit 2024-09-10T18:56:36.737761+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:37.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:37.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:37.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:37.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:37.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:37.706 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:37.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:37.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:37.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:37.970 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:37.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:38.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.578637+0000 osd.7 (osd.7) 172 : cluster [DBG] 3.d scrub starts 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.583827+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.d scrub ok 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.824240+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.828196+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.852925+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.c scrub starts 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:36.856178+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2024-09-10T18:56:38.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.032371+0000 mon.a (mon.0) 1635 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.053642+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.053791+0000 mon.a (mon.0) 1637 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.054238+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.054362+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.091162+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.092591+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.093037+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:38.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:38.085 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.578637+0000 osd.7 (osd.7) 172 : cluster [DBG] 3.d scrub starts 2024-09-10T18:56:38.085 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.583827+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.d scrub ok 2024-09-10T18:56:38.085 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.824240+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-09-10T18:56:38.085 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.828196+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-09-10T18:56:38.085 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.852925+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.c scrub starts 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:36.856178+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.032371+0000 mon.a (mon.0) 1635 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.053642+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.053791+0000 mon.a (mon.0) 1637 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.054238+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.054362+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.091162+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.092591+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.093037+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.211207+0000 mgr.y (mgr.24526) 302 : cluster [DBG] pgmap v345: 292 pgs: 32 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 240 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.279425+0000 osd.5 (osd.5) 161 : cluster [DBG] 16.3 scrub starts 2024-09-10T18:56:38.086 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.282569+0000 osd.5 (osd.5) 162 : cluster [DBG] 16.3 scrub ok 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.585816+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.589191+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[18629]: audit 2024-09-10T18:56:37.739399+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.578637+0000 osd.7 (osd.7) 172 : cluster [DBG] 3.d scrub starts 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.583827+0000 osd.7 (osd.7) 173 : cluster [DBG] 3.d scrub ok 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.824240+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.828196+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.852925+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.c scrub starts 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:36.856178+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.032371+0000 mon.a (mon.0) 1635 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:38.087 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.053642+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49441 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi083-77640-30"}]': finished 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.053791+0000 mon.a (mon.0) 1637 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-19"}]': finished 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.054238+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi083-77648-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.054362+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.091162+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.092591+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.093037+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.211207+0000 mgr.y (mgr.24526) 302 : cluster [DBG] pgmap v345: 292 pgs: 32 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 240 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.279425+0000 osd.5 (osd.5) 161 : cluster [DBG] 16.3 scrub starts 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.282569+0000 osd.5 (osd.5) 162 : cluster [DBG] 16.3 scrub ok 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.585816+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.589191+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:56:38.088 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:38 smithi083 bash[32381]: audit 2024-09-10T18:56:37.739399+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:38.097 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:38.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:38.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:38.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:38.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.211207+0000 mgr.y (mgr.24526) 302 : cluster [DBG] pgmap v345: 292 pgs: 32 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 240 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T18:56:38.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.279425+0000 osd.5 (osd.5) 161 : cluster [DBG] 16.3 scrub starts 2024-09-10T18:56:38.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.282569+0000 osd.5 (osd.5) 162 : cluster [DBG] 16.3 scrub ok 2024-09-10T18:56:38.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.585816+0000 osd.1 (osd.1) 177 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:56:38.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.589191+0000 osd.1 (osd.1) 178 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:56:38.355 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:38 smithi165 bash[22591]: audit 2024-09-10T18:56:37.739399+0000 mon.c (mon.2) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:38.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:38.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:38.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:38.633 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:38.736 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:38.768 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:38.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:38.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:38.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:39.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:39.041 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:39.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:39.174 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:39.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:39.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:39.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:39.447 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:39.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:39.564 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: audit 2024-09-10T18:56:37.750552+0000 mgr.y (mgr.24526) 303 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.790173+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:37.795801+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.052549+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.054690+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: audit 2024-09-10T18:56:38.058412+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.058535+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: audit 2024-09-10T18:56:38.059142+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.291056+0000 osd.5 (osd.5) 163 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.294873+0000 osd.5 (osd.5) 164 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:56:39.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.592813+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:56:39.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.595426+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:56:39.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:39 smithi165 bash[22591]: audit 2024-09-10T18:56:38.741056+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:39.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:39.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:39.722 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: audit 2024-09-10T18:56:37.750552+0000 mgr.y (mgr.24526) 303 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.790173+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:37.795801+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.052549+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.054690+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: audit 2024-09-10T18:56:38.058412+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:39.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.058535+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: audit 2024-09-10T18:56:38.059142+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.291056+0000 osd.5 (osd.5) 163 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.294873+0000 osd.5 (osd.5) 164 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.592813+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.595426+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[18629]: audit 2024-09-10T18:56:38.741056+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: audit 2024-09-10T18:56:37.750552+0000 mgr.y (mgr.24526) 303 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.790173+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:37.795801+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-09-10T18:56:39.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.052549+0000 osd.3 (osd.3) 195 : cluster [DBG] 6.13 scrub starts 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.054690+0000 osd.3 (osd.3) 196 : cluster [DBG] 6.13 scrub ok 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: audit 2024-09-10T18:56:38.058412+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi083-77640-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.058535+0000 mon.a (mon.0) 1644 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: audit 2024-09-10T18:56:38.059142+0000 mon.a (mon.0) 1645 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.291056+0000 osd.5 (osd.5) 163 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.294873+0000 osd.5 (osd.5) 164 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.592813+0000 osd.1 (osd.1) 179 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.595426+0000 osd.1 (osd.1) 180 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:56:39.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:39 smithi083 bash[32381]: audit 2024-09-10T18:56:38.741056+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:39.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:39.858 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:39.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:39.960 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:39.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:40.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:40.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:40.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:40.231 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:40.252 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:40.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:40.387 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:40.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:40.526 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:40.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:40.662 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:40.707 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:40.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.777930+0000 osd.6 (osd.6) 131 : cluster [DBG] 12.5 deep-scrub starts 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:38.781159+0000 osd.6 (osd.6) 132 : cluster [DBG] 12.5 deep-scrub ok 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.212057+0000 mgr.y (mgr.24526) 304 : cluster [DBG] pgmap v348: 292 pgs: 64 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 208 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:39.504625+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.505976+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:39.507318+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.83:0/1872315293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:39.510949+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:39.522756+0000 mon.a (mon.0) 1648 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.626169+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.f scrub starts 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.629717+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.f scrub ok 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:39.742588+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.784788+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d deep-scrub starts 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.791828+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d deep-scrub ok 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.059586+0000 mon.a (mon.0) 1649 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:40.076689+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:40.076872+0000 mon.a (mon.0) 1651 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:40.077069+0000 mon.a (mon.0) 1652 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.077143+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.777930+0000 osd.6 (osd.6) 131 : cluster [DBG] 12.5 deep-scrub starts 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:38.781159+0000 osd.6 (osd.6) 132 : cluster [DBG] 12.5 deep-scrub ok 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.212057+0000 mgr.y (mgr.24526) 304 : cluster [DBG] pgmap v348: 292 pgs: 64 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 208 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:39.504625+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.505976+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:39.507318+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.83:0/1872315293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:39.510949+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:39.522756+0000 mon.a (mon.0) 1648 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.626169+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.f scrub starts 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.629717+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.f scrub ok 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:39.742588+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:40.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.784788+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d deep-scrub starts 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.791828+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d deep-scrub ok 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.059586+0000 mon.a (mon.0) 1649 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:40.076689+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:40.076872+0000 mon.a (mon.0) 1651 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:40.077069+0000 mon.a (mon.0) 1652 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.077143+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:40.106254+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[18629]: audit 2024-09-10T18:56:40.110467+0000 mon.a (mon.0) 1654 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:40.106254+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:40 smithi083 bash[32381]: audit 2024-09-10T18:56:40.110467+0000 mon.a (mon.0) 1654 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:40.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.777930+0000 osd.6 (osd.6) 131 : cluster [DBG] 12.5 deep-scrub starts 2024-09-10T18:56:40.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:38.781159+0000 osd.6 (osd.6) 132 : cluster [DBG] 12.5 deep-scrub ok 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.212057+0000 mgr.y (mgr.24526) 304 : cluster [DBG] pgmap v348: 292 pgs: 64 unknown, 11 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 208 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:39.504625+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.505976+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:39.507318+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.83:0/1872315293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:39.510949+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:39.522756+0000 mon.a (mon.0) 1648 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.626169+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.f scrub starts 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.629717+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.f scrub ok 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:39.742588+0000 mon.c (mon.2) 370 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.784788+0000 osd.6 (osd.6) 133 : cluster [DBG] 16.d deep-scrub starts 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.791828+0000 osd.6 (osd.6) 134 : cluster [DBG] 16.d deep-scrub ok 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.059586+0000 mon.a (mon.0) 1649 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:40.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:40.076689+0000 mon.a (mon.0) 1650 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi083-77640-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:40.076872+0000 mon.a (mon.0) 1651 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:40.077069+0000 mon.a (mon.0) 1652 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.077143+0000 mon.a (mon.0) 1653 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-09-10T18:56:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:40.106254+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:40 smithi165 bash[22591]: audit 2024-09-10T18:56:40.110467+0000 mon.a (mon.0) 1654 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:40.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:40.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:41.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:41.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:41.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:41.343 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:41.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:41.480 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:41.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:41.600 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:41.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.166138+0000 osd.0 (osd.0) 178 : cluster [DBG] 16.14 scrub starts 2024-09-10T18:56:41.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.169439+0000 osd.0 (osd.0) 179 : cluster [DBG] 16.14 scrub ok 2024-09-10T18:56:41.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.801247+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.805456+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.943726+0000 osd.4 (osd.4) 199 : cluster [DBG] 64.d deep-scrub starts 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:39.950156+0000 osd.4 (osd.4) 200 : cluster [DBG] 64.d deep-scrub ok 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.173853+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.177235+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:40.744168+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.952399+0000 osd.4 (osd.4) 201 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:40.956316+0000 osd.4 (osd.4) 202 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:41.079563+0000 mon.a (mon.0) 1655 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:41.079708+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:41.081882+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:41.087014+0000 mon.a (mon.0) 1657 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:41.092730+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.83:0/1228802550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: audit 2024-09-10T18:56:41.093246+0000 mon.a (mon.0) 1658 : audit [INF] from='client.48953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:41 smithi165 bash[22591]: cluster 2024-09-10T18:56:41.213048+0000 mgr.y (mgr.24526) 305 : cluster [DBG] pgmap v351: 332 pgs: 32 unknown, 8 creating+peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:41.621 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:41.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.758 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:41.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:41.876 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.166138+0000 osd.0 (osd.0) 178 : cluster [DBG] 16.14 scrub starts 2024-09-10T18:56:41.876 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.169439+0000 osd.0 (osd.0) 179 : cluster [DBG] 16.14 scrub ok 2024-09-10T18:56:41.876 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.801247+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.805456+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.943726+0000 osd.4 (osd.4) 199 : cluster [DBG] 64.d deep-scrub starts 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:39.950156+0000 osd.4 (osd.4) 200 : cluster [DBG] 64.d deep-scrub ok 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.173853+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.177235+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:40.744168+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.952399+0000 osd.4 (osd.4) 201 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:40.956316+0000 osd.4 (osd.4) 202 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:41.079563+0000 mon.a (mon.0) 1655 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:41.079708+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:41.081882+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:41.087014+0000 mon.a (mon.0) 1657 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:41.092730+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.83:0/1228802550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: audit 2024-09-10T18:56:41.093246+0000 mon.a (mon.0) 1658 : audit [INF] from='client.48953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[32381]: cluster 2024-09-10T18:56:41.213048+0000 mgr.y (mgr.24526) 305 : cluster [DBG] pgmap v351: 332 pgs: 32 unknown, 8 creating+peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.166138+0000 osd.0 (osd.0) 178 : cluster [DBG] 16.14 scrub starts 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.169439+0000 osd.0 (osd.0) 179 : cluster [DBG] 16.14 scrub ok 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.801247+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.1 scrub starts 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.805456+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.1 scrub ok 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.943726+0000 osd.4 (osd.4) 199 : cluster [DBG] 64.d deep-scrub starts 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:39.950156+0000 osd.4 (osd.4) 200 : cluster [DBG] 64.d deep-scrub ok 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.173853+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.5 scrub starts 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.177235+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.5 scrub ok 2024-09-10T18:56:41.878 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:40.744168+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.952399+0000 osd.4 (osd.4) 201 : cluster [DBG] 4.18 scrub starts 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:40.956316+0000 osd.4 (osd.4) 202 : cluster [DBG] 4.18 scrub ok 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:41.079563+0000 mon.a (mon.0) 1655 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:41.079708+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:41.081882+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:41.087014+0000 mon.a (mon.0) 1657 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:41.092730+0000 mon.c (mon.2) 372 : audit [INF] from='client.? v1:172.21.15.83:0/1228802550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: audit 2024-09-10T18:56:41.093246+0000 mon.a (mon.0) 1658 : audit [INF] from='client.48953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:41.879 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:41 smithi083 bash[18629]: cluster 2024-09-10T18:56:41.213048+0000 mgr.y (mgr.24526) 305 : cluster [DBG] pgmap v351: 332 pgs: 32 unknown, 8 creating+peering, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T18:56:41.898 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:41.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:42.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:42.034 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:42.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:42.167 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:42.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.303 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:42.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:42.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.569 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:42.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:42.705 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:42.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.821 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:42] "GET /metrics HTTP/1.1" 200 233436 "" "Prometheus/2.43.0" 2024-09-10T18:56:42.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: cluster 2024-09-10T18:56:41.532417+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:56:42.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: cluster 2024-09-10T18:56:41.536535+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:56:42.821 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:41.745786+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:42.088758+0000 mon.a (mon.0) 1659 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:42.089077+0000 mon.a (mon.0) 1660 : audit [INF] from='client.48953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.089247+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:42.092327+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:42.097158+0000 mon.a (mon.0) 1662 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[18629]: audit 2024-09-10T18:56:42.098113+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: cluster 2024-09-10T18:56:41.532417+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: cluster 2024-09-10T18:56:41.536535+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:41.745786+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:42.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:42.088758+0000 mon.a (mon.0) 1659 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:42.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:42.089077+0000 mon.a (mon.0) 1660 : audit [INF] from='client.48953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:42.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.089247+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:56:42.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:42.092327+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:42.097158+0000 mon.a (mon.0) 1662 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.823 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:42 smithi083 bash[32381]: audit 2024-09-10T18:56:42.098113+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:42.840 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: cluster 2024-09-10T18:56:41.532417+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: cluster 2024-09-10T18:56:41.536535+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:41.745786+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:42.088758+0000 mon.a (mon.0) 1659 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:42.089077+0000 mon.a (mon.0) 1660 : audit [INF] from='client.48953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.089247+0000 mon.a (mon.0) 1661 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:42.092327+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:42.097158+0000 mon.a (mon.0) 1662 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]: dispatch 2024-09-10T18:56:42.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:42 smithi165 bash[22591]: audit 2024-09-10T18:56:42.098113+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:42.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:42.975 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:42.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:43.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:43.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:43.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:43.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.386 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:43.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:43.524 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:43.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:43.661 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:43.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:43.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.358811+0000 osd.5 (osd.5) 165 : cluster [DBG] 16.c scrub starts 2024-09-10T18:56:43.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.362069+0000 osd.5 (osd.5) 166 : cluster [DBG] 16.c scrub ok 2024-09-10T18:56:43.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.564224+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:56:43.775 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.568035+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:42.747469+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.754006+0000 osd.6 (osd.6) 135 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.757615+0000 osd.6 (osd.6) 136 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:42.999435+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.003301+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.084225+0000 mon.a (mon.0) 1664 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.096707+0000 mon.a (mon.0) 1665 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]': finished 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.096832+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.096931+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.098494+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.100302+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.83:0/475656385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.100962+0000 mon.a (mon.0) 1669 : audit [INF] from='client.48959 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.135085+0000 osd.3 (osd.3) 197 : cluster [DBG] 12.12 scrub starts 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.136899+0000 osd.3 (osd.3) 198 : cluster [DBG] 12.12 scrub ok 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.203279+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: audit 2024-09-10T18:56:43.207760+0000 mon.a (mon.0) 1670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.214125+0000 mgr.y (mgr.24526) 306 : cluster [DBG] pgmap v354: 356 pgs: 64 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.358811+0000 osd.5 (osd.5) 165 : cluster [DBG] 16.c scrub starts 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.362069+0000 osd.5 (osd.5) 166 : cluster [DBG] 16.c scrub ok 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.564224+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.568035+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:42.747469+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:43.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.754006+0000 osd.6 (osd.6) 135 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.757615+0000 osd.6 (osd.6) 136 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:42.999435+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.003301+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.084225+0000 mon.a (mon.0) 1664 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.096707+0000 mon.a (mon.0) 1665 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]': finished 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.096832+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.096931+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.098494+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.100302+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.83:0/475656385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.100962+0000 mon.a (mon.0) 1669 : audit [INF] from='client.48959 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.135085+0000 osd.3 (osd.3) 197 : cluster [DBG] 12.12 scrub starts 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.136899+0000 osd.3 (osd.3) 198 : cluster [DBG] 12.12 scrub ok 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.203279+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: audit 2024-09-10T18:56:43.207760+0000 mon.a (mon.0) 1670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:43 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.214125+0000 mgr.y (mgr.24526) 306 : cluster [DBG] pgmap v354: 356 pgs: 64 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:56:43.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:43.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.358811+0000 osd.5 (osd.5) 165 : cluster [DBG] 16.c scrub starts 2024-09-10T18:56:43.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.362069+0000 osd.5 (osd.5) 166 : cluster [DBG] 16.c scrub ok 2024-09-10T18:56:43.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.564224+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:56:43.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.568035+0000 osd.7 (osd.7) 177 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:42.747469+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.754006+0000 osd.6 (osd.6) 135 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.757615+0000 osd.6 (osd.6) 136 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:42.999435+0000 osd.4 (osd.4) 203 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.003301+0000 osd.4 (osd.4) 204 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.084225+0000 mon.a (mon.0) 1664 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.096707+0000 mon.a (mon.0) 1665 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-21", "mode": "writeback"}]': finished 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.096832+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.096931+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.098494+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]: dispatch 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.100302+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.83:0/475656385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.100962+0000 mon.a (mon.0) 1669 : audit [INF] from='client.48959 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:43.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.135085+0000 osd.3 (osd.3) 197 : cluster [DBG] 12.12 scrub starts 2024-09-10T18:56:43.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.136899+0000 osd.3 (osd.3) 198 : cluster [DBG] 12.12 scrub ok 2024-09-10T18:56:43.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.203279+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: audit 2024-09-10T18:56:43.207760+0000 mon.a (mon.0) 1670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:43.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:43 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.214125+0000 mgr.y (mgr.24526) 306 : cluster [DBG] pgmap v354: 356 pgs: 64 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:56:43.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.933 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:43.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:44.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:44.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:44.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:44.212 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:44.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:44.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 -- 172.21.15.83:0/2608981466 >> v1:172.21.15.83:6790/0 conn(0x7f0a04103a50 legacy=0x7f0a04105e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 -- 172.21.15.83:0/2608981466 shutdown_connections 2024-09-10T18:56:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 -- 172.21.15.83:0/2608981466 >> 172.21.15.83:0/2608981466 conn(0x7f0a04074a20 msgr2=0x7f0a040f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 -- 172.21.15.83:0/2608981466 shutdown_connections 2024-09-10T18:56:44.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 -- 172.21.15.83:0/2608981466 wait complete. 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 librados: starting msgr at 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 librados: starting objecter 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.437+0000 7f0a08f32640 1 Processor -- start 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- start start 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 librados: setting wanted keys 2024-09-10T18:56:44.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 librados: calling monclient init 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a04071a80 con 0x7f0a041075b0 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a04071c70 con 0x7f0a04100c70 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a0411e360 con 0x7f0a04103a50 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09f9d9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0a04103a50 0x7f0a040ffc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50992/0 (socket says 172.21.15.83:50992) 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09f9d9b640 1 -- 172.21.15.83:0/3568574319 learned_addr learned my addr 172.21.15.83:0/3568574319 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter ms_handle_connect 0x7f0a04103a50 2024-09-10T18:56:44.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter ms_handle_connect 0x7f0a041075b0 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280425808 0 0) 0x7f0a0411e360 con 0x7f0a04103a50 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f0003120 con 0x7f0a04103a50 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter ms_handle_connect 0x7f0a04100c70 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540537866 0 0) 0x7f0a04071a80 con 0x7f0a041075b0 2024-09-10T18:56:44.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0411e360 con 0x7f0a041075b0 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957526279 0 0) 0x7f0a04071c70 con 0x7f0a04100c70 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a04071a80 con 0x7f0a04100c70 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951928181 0 0) 0x7f09f0003120 con 0x7f0a04103a50 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a04071c70 con 0x7f0a04103a50 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f09ec001610 con 0x7f0a04103a50 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 134864361 0 0) 0x7f0a0411e360 con 0x7f0a041075b0 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09f0003120 con 0x7f0a041075b0 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082258452 0 0) 0x7f0a04071c70 con 0x7f0a04103a50 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.165:6789/0 conn(0x7f0a04100c70 legacy=0x7f0a040ff4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6789/0 conn(0x7f0a041075b0 legacy=0x7f0a0411ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0411f560 con 0x7f0a04103a50 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0411e550 con 0x7f0a04103a50 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 10 client.48971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0411e9a0 con 0x7f0a04103a50 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 librados: init done 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f09ec001230 con 0x7f0a04103a50 2024-09-10T18:56:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f09ec0036a0 con 0x7f0a04103a50 2024-09-10T18:56:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.441+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f09e4004ff0 con 0x7f0a04103a50 2024-09-10T18:56:44.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f09ec001870 con 0x7f0a04103a50 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(248..248 src has 69..248) v4 ==== 9502+0+0 (unknown 1634520928 0 0) 0x7f09ec08ac90 con 0x7f0a04103a50 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 osd_map(248..248 src has 69..248) v4 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 3 client.48971.objecter handle_osd_map got epochs [248,248] > 0 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 3 client.48971.objecter handle_osd_map decoding full epoch 248 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 20 client.48971.objecter dump_active .. 0 homeless 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09f0071810 2024-09-10T18:56:44.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v248) v1 ==== 54+0+16 (unknown 203594432 0 2420000192) 0x7f09ec08b050 con 0x7f0a04103a50 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 10 client.48971.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 1 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 1 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 0 1 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 10 client.48971.objecter _send_command 1 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4005290 con 0x7f09e4004610 2024-09-10T18:56:44.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e4004610 2024-09-10T18:56:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4005290 con 0x7f09e4004610 2024-09-10T18:56:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:56:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 1 = system:0 2024-09-10T18:56:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T18:56:44.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.445+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 0 1 2024-09-10T18:56:44.451 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 10 client.48971.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 2 2024-09-10T18:56:44.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 2 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4005810 osd=0 3 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 0 2 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 10 client.48971.objecter _send_command 2 2024-09-10T18:56:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.465+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4004c60 con 0x7f09e4004610 2024-09-10T18:56:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4004c60 con 0x7f09e4004610 2024-09-10T18:56:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T18:56:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 0 2 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 10 client.48971.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 3 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 1 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 10 client.48971.objecter _send_command 3 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4003b10 con 0x7f09e40035e0 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e40035e0 2024-09-10T18:56:44.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4003b10 con 0x7f09e40035e0 2024-09-10T18:56:44.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:56:44.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 3 = system:0 2024-09-10T18:56:44.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T18:56:44.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.469+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 1 3 2024-09-10T18:56:44.476 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.489+0000 7f0a08f32640 10 client.48971.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.489+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 4 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.489+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 4 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 1 4 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T18:56:44.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 10 client.48971.objecter _send_command 4 2024-09-10T18:56:44.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4003060 con 0x7f09e40035e0 2024-09-10T18:56:44.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4003060 con 0x7f09e40035e0 2024-09-10T18:56:44.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T18:56:44.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 1 4 2024-09-10T18:56:44.497 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 10 client.48971.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 5 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 5 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 2 5 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 4 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 10 client.48971.objecter _send_command 5 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4002350 con 0x7f09e400d350 2024-09-10T18:56:44.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.493+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e400d350 2024-09-10T18:56:44.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.497+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4002350 con 0x7f09e400d350 2024-09-10T18:56:44.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.497+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:56:44.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.497+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 5 = system:0 2024-09-10T18:56:44.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.497+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 4 2024-09-10T18:56:44.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.497+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 2 5 2024-09-10T18:56:44.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 10 client.48971.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 6 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 6 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4001b60 osd=2 3 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 2 6 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 4 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 10 client.48971.objecter _send_command 6 2024-09-10T18:56:44.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.509+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4001650 con 0x7f09e400d350 2024-09-10T18:56:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4001650 con 0x7f09e400d350 2024-09-10T18:56:44.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 4 2024-09-10T18:56:44.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 2 6 2024-09-10T18:56:44.516 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 10 client.48971.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 7 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 7 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 3 7 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 4 2024-09-10T18:56:44.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 10 client.48971.objecter _send_command 7 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e400d140 con 0x7f09e40163e0 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.513+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e40163e0 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.517+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e400d140 con 0x7f09e40163e0 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.517+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.517+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 7 = system:0 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.517+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 4 2024-09-10T18:56:44.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.517+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 3 7 2024-09-10T18:56:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 10 client.48971.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 8 2024-09-10T18:56:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 8 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c990 osd=3 3 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 3 8 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 4 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 10 client.48971.objecter _send_command 8 2024-09-10T18:56:44.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.537+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e400c410 con 0x7f09e40163e0 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e400c410 con 0x7f09e40163e0 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 4 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 3 8 2024-09-10T18:56:44.544 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 10 client.48971.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 9 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 9 2024-09-10T18:56:44.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 4 9 2024-09-10T18:56:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 4 2024-09-10T18:56:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 10 client.48971.objecter _send_command 9 2024-09-10T18:56:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e400c340 con 0x7f09e4011b20 2024-09-10T18:56:44.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.541+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e4011b20 2024-09-10T18:56:44.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.545+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e400c340 con 0x7f09e4011b20 2024-09-10T18:56:44.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.545+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:56:44.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.545+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 9 = system:0 2024-09-10T18:56:44.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.545+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 4 2024-09-10T18:56:44.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.545+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 4 9 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 10 client.48971.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 10 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 10 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c1d0 osd=4 3 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 4 10 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 4 2024-09-10T18:56:44.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 10 client.48971.objecter _send_command 10 2024-09-10T18:56:44.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.565+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e400c340 con 0x7f09e4011b20 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e400c340 con 0x7f09e4011b20 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 4 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 4 10 2024-09-10T18:56:44.572 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 10 client.48971.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 11 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 11 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 5 11 2024-09-10T18:56:44.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 4 2024-09-10T18:56:44.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 10 client.48971.objecter _send_command 11 2024-09-10T18:56:44.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e401c2f0 con 0x7f09e401bcb0 2024-09-10T18:56:44.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.569+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e401bcb0 2024-09-10T18:56:44.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.573+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e401c2f0 con 0x7f09e401bcb0 2024-09-10T18:56:44.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.573+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:56:44.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.573+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 11 = system:0 2024-09-10T18:56:44.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.573+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 4 2024-09-10T18:56:44.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.573+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 5 11 2024-09-10T18:56:44.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 10 client.48971.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 12 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 12 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e400c340 osd=5 3 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 5 12 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 4 2024-09-10T18:56:44.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 10 client.48971.objecter _send_command 12 2024-09-10T18:56:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.593+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e401c2b0 con 0x7f09e401bcb0 2024-09-10T18:56:44.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e401c2b0 con 0x7f09e401bcb0 2024-09-10T18:56:44.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 4 2024-09-10T18:56:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 5 12 2024-09-10T18:56:44.602 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:56:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:56:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 10 client.48971.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 13 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 13 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 6 13 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 4 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 10 client.48971.objecter _send_command 13 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.597+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4026f20 con 0x7f09e40233e0 2024-09-10T18:56:44.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e40233e0 2024-09-10T18:56:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4026f20 con 0x7f09e40233e0 2024-09-10T18:56:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:56:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 13 = system:0 2024-09-10T18:56:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 4 2024-09-10T18:56:44.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.601+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 6 13 2024-09-10T18:56:44.608 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:44.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:44.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 10 client.48971.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 14 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 14 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e401c2b0 osd=6 3 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 6 14 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 4 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 10 client.48971.objecter _send_command 14 2024-09-10T18:56:44.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.617+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4026ee0 con 0x7f09e40233e0 2024-09-10T18:56:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4026ee0 con 0x7f09e40233e0 2024-09-10T18:56:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 4 2024-09-10T18:56:44.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 6 14 2024-09-10T18:56:44.625 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 10 client.48971.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 15 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 15 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 7 15 2024-09-10T18:56:44.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 4 2024-09-10T18:56:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 10 client.48971.objecter _send_command 15 2024-09-10T18:56:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e40282f0 con 0x7f09e4027610 2024-09-10T18:56:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.621+0000 7f09fb7fe640 10 client.48971.objecter ms_handle_connect 0x7f09e4027610 2024-09-10T18:56:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.625+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e40282f0 con 0x7f09e4027610 2024-09-10T18:56:44.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.625+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:56:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.625+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 15 = system:0 2024-09-10T18:56:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.625+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 4 2024-09-10T18:56:44.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.625+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 7 15 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 10 client.48971.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign -1 16 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 20 client.48971.objecter _get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_remove -1 16 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 20 client.48971.objecter get_session s=0x7f09e4026f20 osd=7 3 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 15 client.48971.objecter _session_command_op_assign 7 16 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 4 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 10 client.48971.objecter _send_command 16 2024-09-10T18:56:44.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.641+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e40282f0 con 0x7f09e4027610 2024-09-10T18:56:44.647 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:56:44.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f09fb7fe640 1 -- 172.21.15.83:0/3568574319 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e40282f0 con 0x7f09e4027610 2024-09-10T18:56:44.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f09fb7fe640 10 client.48971.objecter ms_dispatch 0x7f0a04000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f09fb7fe640 10 client.48971.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f09fb7fe640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 4 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f09fb7fe640 15 client.48971.objecter _session_command_op_remove 7 16 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 librados: watch_flush enter 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 librados: watch_flush exit 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.0 2024-09-10T18:56:44.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6801/2535527483 conn(0x7f09e4004610 legacy=0x7f09e4002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4005810 osd=0 1 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.1 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6805/2309624637 conn(0x7f09e40035e0 legacy=0x7f09e400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4004e10 osd=1 1 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.2 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6809/386094220 conn(0x7f09e400d350 legacy=0x7f09e400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a02d9d640 1 -- 172.21.15.83:0/3568574319 reap_dead start 2024-09-10T18:56:44.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4001b60 osd=2 1 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.3 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6813/1279688021 conn(0x7f09e40163e0 legacy=0x7f09e40187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c990 osd=3 1 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.4 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.165:6800/3225515265 conn(0x7f09e4011b20 legacy=0x7f09e401b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c1d0 osd=4 1 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.5 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.165:6804/571173777 conn(0x7f09e401bcb0 legacy=0x7f09e40217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e400c340 osd=5 1 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.6 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.165:6808/1317922936 conn(0x7f09e40233e0 legacy=0x7f09e40257c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e401c2b0 osd=6 1 2024-09-10T18:56:44.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter close_session for osd.7 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.165:6812/1610151650 conn(0x7f09e4027610 legacy=0x7f09e4027a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter put_session s=0x7f09e4026f20 osd=7 1 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 20 client.48971.objecter shutdown clearing up homeless session... 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 10 client.48971.objecter successfully canceled tick 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a02d9d640 1 -- 172.21.15.83:0/3568574319 reap_dead start 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6800/3071793735 conn(0x7f09f0071810 legacy=0x7f09f0073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> v1:172.21.15.83:6790/0 conn(0x7f0a04103a50 legacy=0x7f0a040ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 shutdown_connections 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 >> 172.21.15.83:0/3568574319 conn(0x7f0a04074a20 msgr2=0x7f0a04071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 shutdown_connections 2024-09-10T18:56:44.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 -- 172.21.15.83:0/3568574319 wait complete. 2024-09-10T18:56:44.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.645+0000 7f0a08f32640 1 librados: shutdown 2024-09-10T18:56:44.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:44.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.370787+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-09-10T18:56:44.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.374337+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-09-10T18:56:44.724 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.612255+0000 osd.1 (osd.1) 183 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: cluster 2024-09-10T18:56:43.615978+0000 osd.1 (osd.1) 184 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:43.749061+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.015979+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.016174+0000 mon.a (mon.0) 1672 : audit [INF] from='client.48959 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.016521+0000 mon.a (mon.0) 1673 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.016642+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.018060+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.022291+0000 mon.a (mon.0) 1675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.040014+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.041758+0000 mon.a (mon.0) 1676 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.052911+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.053572+0000 mon.a (mon.0) 1677 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.725 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.056251+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.057730+0000 mon.a (mon.0) 1678 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.446343+0000 mon.c (mon.2) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3568574319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.564391+0000 mon.a (mon.0) 1679 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[18629]: audit 2024-09-10T18:56:44.567546+0000 mon.c (mon.2) 381 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.370787+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.374337+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.612255+0000 osd.1 (osd.1) 183 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: cluster 2024-09-10T18:56:43.615978+0000 osd.1 (osd.1) 184 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:43.749061+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.015979+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:44.726 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.016174+0000 mon.a (mon.0) 1672 : audit [INF] from='client.48959 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.016521+0000 mon.a (mon.0) 1673 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.016642+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.018060+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.022291+0000 mon.a (mon.0) 1675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.040014+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.041758+0000 mon.a (mon.0) 1676 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.052911+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.053572+0000 mon.a (mon.0) 1677 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.056251+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.057730+0000 mon.a (mon.0) 1678 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.446343+0000 mon.c (mon.2) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3568574319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.564391+0000 mon.a (mon.0) 1679 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:44.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:44 smithi083 bash[32381]: audit 2024-09-10T18:56:44.567546+0000 mon.c (mon.2) 381 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:44.743 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:44.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/1148717935 >> v1:172.21.15.83:6789/0 conn(0x7fcb78107e00 legacy=0x7fcb7810a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/1148717935 shutdown_connections 2024-09-10T18:56:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/1148717935 >> 172.21.15.83:0/1148717935 conn(0x7fcb780f70b0 msgr2=0x7fcb780f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/1148717935 shutdown_connections 2024-09-10T18:56:44.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/1148717935 wait complete. 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 librados: starting msgr at 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 librados: starting objecter 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 Processor -- start 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- start start 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 librados: setting wanted keys 2024-09-10T18:56:44.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 librados: calling monclient init 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcb7811afb0 con 0x7fcb78107e00 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcb7811c1b0 con 0x7fcb781042a0 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7f4c2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcb7811d3b0 con 0x7fcb78101530 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7d25e640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fcb78101530 0x7fcb78104810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51010/0 (socket says 172.21.15.83:51010) 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb7d25e640 1 -- 172.21.15.83:0/3618730659 learned_addr learned my addr 172.21.15.83:0/3618730659 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter ms_handle_connect 0x7fcb78101530 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter ms_handle_connect 0x7fcb78107e00 2024-09-10T18:56:44.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869473353 0 0) 0x7fcb7811d3b0 con 0x7fcb78101530 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb64003720 con 0x7fcb78101530 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter ms_handle_connect 0x7fcb781042a0 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995360839 0 0) 0x7fcb7811afb0 con 0x7fcb78107e00 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7811d3b0 con 0x7fcb78107e00 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410031464 0 0) 0x7fcb7811c1b0 con 0x7fcb781042a0 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb7811afb0 con 0x7fcb781042a0 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3935724789 0 0) 0x7fcb64003720 con 0x7fcb78101530 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb7811c1b0 con 0x7fcb78101530 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.757+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcb74001940 con 0x7fcb78101530 2024-09-10T18:56:44.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2224334847 0 0) 0x7fcb7811d3b0 con 0x7fcb78107e00 2024-09-10T18:56:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb64003720 con 0x7fcb78107e00 2024-09-10T18:56:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3810095641 0 0) 0x7fcb7811c1b0 con 0x7fcb78101530 2024-09-10T18:56:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 >> v1:172.21.15.165:6789/0 conn(0x7fcb781042a0 legacy=0x7fcb78104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 >> v1:172.21.15.83:6789/0 conn(0x7fcb78107e00 legacy=0x7fcb781196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7811e5b0 con 0x7fcb78101530 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7811b1a0 con 0x7fcb78101530 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fcb74001bb0 con 0x7fcb78101530 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcb74005b20 con 0x7fcb78101530 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb7f4c2640 10 client.48977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7811b5f0 con 0x7fcb78101530 2024-09-10T18:56:44.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb7f4c2640 1 librados: init done 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb780fcc30 con 0x7fcb78101530 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.761+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fcb7401bf40 con 0x7fcb78101530 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(248..248 src has 69..248) v4 ==== 9502+0+0 (unknown 1634520928 0 0) 0x7fcb74088780 con 0x7fcb78101530 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 10 client.48977.objecter ms_dispatch 0x7fcb78000b90 osd_map(248..248 src has 69..248) v4 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 3 client.48977.objecter handle_osd_map got epochs [248,248] > 0 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 3 client.48977.objecter handle_osd_map decoding full epoch 248 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 20 client.48977.objecter dump_active .. 0 homeless 2024-09-10T18:56:44.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.765+0000 7fcb6dffb640 10 client.48977.objecter ms_handle_connect 0x7fcb64071430 2024-09-10T18:56:44.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.769+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fcb74056cc0 con 0x7fcb78101530 2024-09-10T18:56:44.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.370787+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.d scrub starts 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.374337+0000 osd.5 (osd.5) 168 : cluster [DBG] 6.d scrub ok 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.612255+0000 osd.1 (osd.1) 183 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: cluster 2024-09-10T18:56:43.615978+0000 osd.1 (osd.1) 184 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:43.749061+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.015979+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? v1:172.21.15.83:0/1970984313' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi083-77640-31"}]': finished 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.016174+0000 mon.a (mon.0) 1672 : audit [INF] from='client.48959 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi083-77648-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.016521+0000 mon.a (mon.0) 1673 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:44.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.016642+0000 mon.a (mon.0) 1674 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.018060+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.022291+0000 mon.a (mon.0) 1675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.040014+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.041758+0000 mon.a (mon.0) 1676 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.052911+0000 mon.c (mon.2) 378 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.053572+0000 mon.a (mon.0) 1677 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.056251+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.057730+0000 mon.a (mon.0) 1678 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.446343+0000 mon.c (mon.2) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3568574319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.564391+0000 mon.a (mon.0) 1679 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:44.861 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:44 smithi165 bash[22591]: audit 2024-09-10T18:56:44.567546+0000 mon.c (mon.2) 381 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:56:44.879 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:44.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:44.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:44.961+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fcb7804bf60 con 0x7fcb78101530 2024-09-10T18:56:44.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:45.010 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:45.029 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.021+0000 7fcb6dffb640 1 -- 172.21.15.83:0/3618730659 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v249) v1 ==== 87+0+0 (unknown 2605768417 0 0) 0x7fcb74056660 con 0x7fcb78101530 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 10 librados: watch_flush enter 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 10 librados: watch_flush exit 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 20 client.48977.objecter shutdown clearing up homeless session... 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 10 client.48977.objecter successfully canceled tick 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 >> v1:172.21.15.83:6800/3071793735 conn(0x7fcb64071430 legacy=0x7fcb640738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.029+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 >> v1:172.21.15.83:6790/0 conn(0x7fcb78101530 legacy=0x7fcb78104810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.033+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 shutdown_connections 2024-09-10T18:56:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.033+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 >> 172.21.15.83:0/3618730659 conn(0x7fcb780f70b0 msgr2=0x7fcb780fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.033+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 shutdown_connections 2024-09-10T18:56:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.033+0000 7fcb7f4c2640 1 -- 172.21.15.83:0/3618730659 wait complete. 2024-09-10T18:56:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:45.033+0000 7fcb7f4c2640 1 librados: shutdown 2024-09-10T18:56:45.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:45.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:45.278 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:45.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:45.414 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:45.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:45.552 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:45.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:45.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.185506+0000 osd.0 (osd.0) 182 : cluster [DBG] 16.15 scrub starts 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.189225+0000 osd.0 (osd.0) 183 : cluster [DBG] 16.15 scrub ok 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.571738+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub starts 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.575347+0000 osd.7 (osd.7) 179 : cluster [DBG] 16.1e scrub ok 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.581194+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.598122+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:56:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.679681+0000 osd.2 (osd.2) 107 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.683509+0000 osd.2 (osd.2) 108 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:44.750229+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:44.965889+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3618730659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:44.966332+0000 mon.a (mon.0) 1680 : audit [INF] from='client.48977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.974966+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:44.980245+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:45.010222+0000 mon.a (mon.0) 1681 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:45.010271+0000 mon.a (mon.0) 1682 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:45.023258+0000 mon.a (mon.0) 1683 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:45.023347+0000 mon.a (mon.0) 1684 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:45.023424+0000 mon.a (mon.0) 1685 : audit [INF] from='client.48977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:45.023468+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:56:45.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:45.025169+0000 mon.c (mon.2) 384 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: audit 2024-09-10T18:56:45.025889+0000 mon.a (mon.0) 1687 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[18629]: cluster 2024-09-10T18:56:45.215054+0000 mgr.y (mgr.24526) 307 : cluster [DBG] pgmap v357: 324 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.185506+0000 osd.0 (osd.0) 182 : cluster [DBG] 16.15 scrub starts 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.189225+0000 osd.0 (osd.0) 183 : cluster [DBG] 16.15 scrub ok 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.571738+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub starts 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.575347+0000 osd.7 (osd.7) 179 : cluster [DBG] 16.1e scrub ok 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.581194+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.598122+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.679681+0000 osd.2 (osd.2) 107 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.683509+0000 osd.2 (osd.2) 108 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:56:45.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:44.750229+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:44.965889+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3618730659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:44.966332+0000 mon.a (mon.0) 1680 : audit [INF] from='client.48977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.974966+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:44.980245+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:45.010222+0000 mon.a (mon.0) 1681 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:45.010271+0000 mon.a (mon.0) 1682 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:45.023258+0000 mon.a (mon.0) 1683 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:45.023347+0000 mon.a (mon.0) 1684 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:45.023424+0000 mon.a (mon.0) 1685 : audit [INF] from='client.48977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:45.023468+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:45.025169+0000 mon.c (mon.2) 384 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: audit 2024-09-10T18:56:45.025889+0000 mon.a (mon.0) 1687 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:45 smithi083 bash[32381]: cluster 2024-09-10T18:56:45.215054+0000 mgr.y (mgr.24526) 307 : cluster [DBG] pgmap v357: 324 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:45.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:45.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.185506+0000 osd.0 (osd.0) 182 : cluster [DBG] 16.15 scrub starts 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.189225+0000 osd.0 (osd.0) 183 : cluster [DBG] 16.15 scrub ok 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.571738+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub starts 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.575347+0000 osd.7 (osd.7) 179 : cluster [DBG] 16.1e scrub ok 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.581194+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-09-10T18:56:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.598122+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.679681+0000 osd.2 (osd.2) 107 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.683509+0000 osd.2 (osd.2) 108 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:44.750229+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:44.965889+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3618730659' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:44.966332+0000 mon.a (mon.0) 1680 : audit [INF] from='client.48977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.974966+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.9 scrub starts 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:44.980245+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:45.010222+0000 mon.a (mon.0) 1681 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:45.010271+0000 mon.a (mon.0) 1682 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:45.023258+0000 mon.a (mon.0) 1683 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-21"}]': finished 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:45.023347+0000 mon.a (mon.0) 1684 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi083-77640-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:45.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:45.023424+0000 mon.a (mon.0) 1685 : audit [INF] from='client.48977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:56:45.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:45.023468+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-09-10T18:56:45.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:45.025169+0000 mon.c (mon.2) 384 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: audit 2024-09-10T18:56:45.025889+0000 mon.a (mon.0) 1687 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:45.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:45 smithi165 bash[22591]: cluster 2024-09-10T18:56:45.215054+0000 mgr.y (mgr.24526) 307 : cluster [DBG] pgmap v357: 324 pgs: 32 unknown, 5 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 279 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:45.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:45.965 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:45.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:46.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:46.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:46.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.356 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:46.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.479 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:46.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.606 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:46.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:46.745 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:46.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:46.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:46.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:46.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:46.997 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:46 smithi165 bash[22591]: audit 2024-09-10T18:56:45.751446+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:46.997 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:46 smithi165 bash[22591]: cluster 2024-09-10T18:56:46.014160+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:46.997 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:46 smithi165 bash[22591]: cluster 2024-09-10T18:56:46.470638+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:56:47.018 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:47.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- 172.21.15.83:0/384810381 >> v1:172.21.15.83:6790/0 conn(0x7f94901040d0 legacy=0x7f94901064b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- 172.21.15.83:0/384810381 shutdown_connections 2024-09-10T18:56:47.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- 172.21.15.83:0/384810381 >> 172.21.15.83:0/384810381 conn(0x7f94900f6f20 msgr2=0x7f94900f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:47.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- 172.21.15.83:0/384810381 shutdown_connections 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- 172.21.15.83:0/384810381 wait complete. 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 librados: starting msgr at 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 librados: starting objecter 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 Processor -- start 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- start start 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 librados: setting wanted keys 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 librados: calling monclient init 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f949011b140 con 0x7f9490107c30 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f949011c340 con 0x7f94901040d0 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948ffff640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f949011d540 con 0x7f94901012f0 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948effd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f94901012f0 0x7f9490100460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51040/0 (socket says 172.21.15.83:51040) 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f948effd640 1 -- 172.21.15.83:0/828642128 learned_addr learned my addr 172.21.15.83:0/828642128 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter ms_handle_connect 0x7f94901012f0 2024-09-10T18:56:47.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter ms_handle_connect 0x7f9490107c30 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584157698 0 0) 0x7f949011d540 con 0x7f94901012f0 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9478003720 con 0x7f94901012f0 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter ms_handle_connect 0x7f94901040d0 2024-09-10T18:56:47.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:56:47.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145928019 0 0) 0x7f949011c340 con 0x7f94901040d0 2024-09-10T18:56:47.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.129+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f949011d540 con 0x7f94901040d0 2024-09-10T18:56:47.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747010339 0 0) 0x7f9478003720 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f949011c340 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9480001940 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3553426264 0 0) 0x7f949011c340 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 >> v1:172.21.15.165:6789/0 conn(0x7f94901040d0 legacy=0x7f9490116260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 >> v1:172.21.15.83:6789/0 conn(0x7f9490107c30 legacy=0x7f9490119990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949011e740 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949011b330 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f948ffff640 10 client.48989.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f949011b7c0 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f948ffff640 1 librados: init done 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9480002420 con 0x7f94901012f0 2024-09-10T18:56:47.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9480005af0 con 0x7f94901012f0 2024-09-10T18:56:47.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.133+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c005010 con 0x7f94901012f0 2024-09-10T18:56:47.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9480002010 con 0x7f94901012f0 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(251..251 src has 69..251) v4 ==== 9040+0+0 (unknown 809644968 0 0) 0x7f948008aa80 con 0x7f94901012f0 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 10 client.48989.objecter ms_dispatch 0x7f9490000b90 osd_map(251..251 src has 69..251) v4 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 3 client.48989.objecter handle_osd_map got epochs [251,251] > 0 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 3 client.48989.objecter handle_osd_map decoding full epoch 251 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 20 client.48989.objecter dump_active .. 0 homeless 2024-09-10T18:56:47.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.137+0000 7f94777fe640 10 client.48989.objecter ms_handle_connect 0x7f9478073700 2024-09-10T18:56:47.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.141+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f94800567d0 con 0x7f94901012f0 2024-09-10T18:56:47.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[18629]: audit 2024-09-10T18:56:45.751446+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[18629]: cluster 2024-09-10T18:56:46.014160+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[18629]: cluster 2024-09-10T18:56:46.470638+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[32381]: audit 2024-09-10T18:56:45.751446+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[32381]: cluster 2024-09-10T18:56:46.014160+0000 mon.a (mon.0) 1688 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:47.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:46 smithi083 bash[32381]: cluster 2024-09-10T18:56:46.470638+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-09-10T18:56:47.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.290 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:47.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.341+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f947c000b90 con 0x7f9478073700 2024-09-10T18:56:47.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.361+0000 7f94777fe640 1 -- 172.21.15.83:0/828642128 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+721223 (unknown 2965378022 0 3021260141) 0x7f947c000b90 con 0x7f9478073700 2024-09-10T18:56:47.368 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:56:47.381 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:56:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.369+0000 7f948ffff640 10 librados: watch_flush enter 2024-09-10T18:56:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.369+0000 7f948ffff640 10 librados: watch_flush exit 2024-09-10T18:56:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.369+0000 7f948ffff640 20 client.48989.objecter shutdown clearing up homeless session... 2024-09-10T18:56:47.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.369+0000 7f948ffff640 10 client.48989.objecter successfully canceled tick 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9478073700 legacy=0x7f9478075bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 >> v1:172.21.15.83:6790/0 conn(0x7f94901012f0 legacy=0x7f9490100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 shutdown_connections 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 >> 172.21.15.83:0/828642128 conn(0x7f94900f6f20 msgr2=0x7f94901080b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 shutdown_connections 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 -- 172.21.15.83:0/828642128 wait complete. 2024-09-10T18:56:47.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:56:47.373+0000 7f948ffff640 1 librados: shutdown 2024-09-10T18:56:47.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:47.415 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:47.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:47.553 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:47.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:47.684 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:47.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:47.864 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:47.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:47.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:48.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:48.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:48.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: audit 2024-09-10T18:56:46.753095+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: audit 2024-09-10T18:56:47.025422+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: cluster 2024-09-10T18:56:47.025767+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: audit 2024-09-10T18:56:47.044198+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: audit 2024-09-10T18:56:47.049226+0000 mon.a (mon.0) 1692 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: cluster 2024-09-10T18:56:47.215908+0000 mgr.y (mgr.24526) 308 : cluster [DBG] pgmap v360: 268 pgs: 40 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[18629]: audit 2024-09-10T18:56:47.754492+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: audit 2024-09-10T18:56:46.753095+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: audit 2024-09-10T18:56:47.025422+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: cluster 2024-09-10T18:56:47.025767+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: audit 2024-09-10T18:56:47.044198+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: audit 2024-09-10T18:56:47.049226+0000 mon.a (mon.0) 1692 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: cluster 2024-09-10T18:56:47.215908+0000 mgr.y (mgr.24526) 308 : cluster [DBG] pgmap v360: 268 pgs: 40 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:48.116 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:47 smithi083 bash[32381]: audit 2024-09-10T18:56:47.754492+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.192 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: audit 2024-09-10T18:56:46.753095+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.192 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: audit 2024-09-10T18:56:47.025422+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi083-77640-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:48.192 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: cluster 2024-09-10T18:56:47.025767+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-09-10T18:56:48.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: audit 2024-09-10T18:56:47.044198+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: audit 2024-09-10T18:56:47.049226+0000 mon.a (mon.0) 1692 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:48.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: cluster 2024-09-10T18:56:47.215908+0000 mgr.y (mgr.24526) 308 : cluster [DBG] pgmap v360: 268 pgs: 40 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:48.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:47 smithi165 bash[22591]: audit 2024-09-10T18:56:47.754492+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:48.214 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:48.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:48.353 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:48.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:48.489 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:48.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:56:48.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.624 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:48.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.759 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:48.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:48.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:48.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:49.031 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:49.045 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: audit 2024-09-10T18:56:47.344179+0000 mgr.y (mgr.24526) 309 : audit [DBG] from='client.48989 v1:172.21.15.83:0/828642128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:49.045 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: audit 2024-09-10T18:56:47.761864+0000 mgr.y (mgr.24526) 310 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:49.045 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: audit 2024-09-10T18:56:48.035804+0000 mon.a (mon.0) 1693 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:49.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: cluster 2024-09-10T18:56:48.036119+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:56:49.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: audit 2024-09-10T18:56:48.049029+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:49.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:49 smithi165 bash[22591]: audit 2024-09-10T18:56:48.755766+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:49.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: audit 2024-09-10T18:56:47.344179+0000 mgr.y (mgr.24526) 309 : audit [DBG] from='client.48989 v1:172.21.15.83:0/828642128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:49.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: audit 2024-09-10T18:56:47.761864+0000 mgr.y (mgr.24526) 310 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:49.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: audit 2024-09-10T18:56:48.035804+0000 mon.a (mon.0) 1693 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:49.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: cluster 2024-09-10T18:56:48.036119+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:56:49.114 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: audit 2024-09-10T18:56:48.049029+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[32381]: audit 2024-09-10T18:56:48.755766+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: audit 2024-09-10T18:56:47.344179+0000 mgr.y (mgr.24526) 309 : audit [DBG] from='client.48989 v1:172.21.15.83:0/828642128' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: audit 2024-09-10T18:56:47.761864+0000 mgr.y (mgr.24526) 310 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: audit 2024-09-10T18:56:48.035804+0000 mon.a (mon.0) 1693 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: cluster 2024-09-10T18:56:48.036119+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: audit 2024-09-10T18:56:48.049029+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:49.115 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:49 smithi083 bash[18629]: audit 2024-09-10T18:56:48.755766+0000 mon.c (mon.2) 388 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:49.116 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-10T18:56:47.215360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599656,"num_objects":347,"num_object_clones":24,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":1392,"num_read_kb":1160,"num_write":1168,"num_write_kb":2252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2449,"ondisk_log_size":2449,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":976,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2484632,"kb_used_data":30808,"kb_used_omap":13,"kb_used_meta":2453618,"kb_avail":747476584,"statfs":{"total":767960285184,"available":765416022016,"internally_reserved":0,"allocated":31547392,"data_stored":17622351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2512505536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-348,"num_objects":-20,"num_object_clones":-20,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-20,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001942"},"pg_stats":[{"pgid":"156.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"155.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"156.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:56:47.015808+0000","last_change":"2024-09-10T18:56:47.015808+0000","last_active":"2024-09-10T18:56:47.015808+0000","last_peered":"2024-09-10T18:56:47.015808+0000","last_clean":"2024-09-10T18:56:47.015808+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:56:47.015808+0000","last_undegraded":"2024-09-10T18:56:47.015808+0000","last_fullsized":"2024-09-10T18:56:47.015808+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:47.015808+0000","last_clean_scrub_stamp":"2024-09-10T18:56:47.015808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":172,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059482+0000","last_change":"2024-09-10T18:56:21.043075+0000","last_active":"2024-09-10T18:56:44.059482+0000","last_peered":"2024-09-10T18:56:44.059482+0000","last_clean":"2024-09-10T18:56:44.059482+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:56:44.059482+0000","last_undegraded":"2024-09-10T18:56:44.059482+0000","last_fullsized":"2024-09-10T18:56:44.059482+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015847356,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1e","version":"226'51","reported_seq":185,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056948+0000","last_change":"2024-09-10T18:56:24.447792+0000","last_active":"2024-09-10T18:56:44.056948+0000","last_peered":"2024-09-10T18:56:44.056948+0000","last_clean":"2024-09-10T18:56:44.056948+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:56:44.056948+0000","last_undegraded":"2024-09-10T18:56:44.056948+0000","last_fullsized":"2024-09-10T18:56:44.056948+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42490645199999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1d","version":"232'17","reported_seq":160,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023347+0000","last_change":"2024-09-10T18:56:32.550234+0000","last_active":"2024-09-10T18:56:44.023347+0000","last_peered":"2024-09-10T18:56:44.023347+0000","last_clean":"2024-09-10T18:56:44.023347+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:56:44.023347+0000","last_undegraded":"2024-09-10T18:56:44.023347+0000","last_fullsized":"2024-09-10T18:56:44.023347+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.46456055299999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1c","version":"244'51","reported_seq":190,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057179+0000","last_change":"2024-09-10T18:56:43.616106+0000","last_active":"2024-09-10T18:56:44.057179+0000","last_peered":"2024-09-10T18:56:44.057179+0000","last_clean":"2024-09-10T18:56:44.057179+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:56:44.057179+0000","last_undegraded":"2024-09-10T18:56:44.057179+0000","last_fullsized":"2024-09-10T18:56:44.057179+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.052158+0000","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.46841269000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1b","version":"240'68","reported_seq":195,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056132+0000","last_change":"2024-09-10T18:56:38.573446+0000","last_active":"2024-09-10T18:56:44.056132+0000","last_peered":"2024-09-10T18:56:44.056132+0000","last_clean":"2024-09-10T18:56:44.056132+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:56:44.056132+0000","last_undegraded":"2024-09-10T18:56:44.056132+0000","last_fullsized":"2024-09-10T18:56:44.056132+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50611468500000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":171,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025071+0000","last_change":"2024-09-10T18:56:30.534607+0000","last_active":"2024-09-10T18:56:44.025071+0000","last_peered":"2024-09-10T18:56:44.025071+0000","last_clean":"2024-09-10T18:56:44.025071+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:56:44.025071+0000","last_undegraded":"2024-09-10T18:56:44.025071+0000","last_fullsized":"2024-09-10T18:56:44.025071+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47585997400000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.19","version":"224'17","reported_seq":159,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045248+0000","last_change":"2024-09-10T18:56:22.445191+0000","last_active":"2024-09-10T18:56:44.045248+0000","last_peered":"2024-09-10T18:56:44.045248+0000","last_clean":"2024-09-10T18:56:44.045248+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:56:44.045248+0000","last_undegraded":"2024-09-10T18:56:44.045248+0000","last_fullsized":"2024-09-10T18:56:44.045248+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41938524700000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.18","version":"247'92","reported_seq":212,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:44.022808+0000","last_change":"2024-09-10T18:56:38.466392+0000","last_active":"2024-09-10T18:56:44.022808+0000","last_peered":"2024-09-10T18:56:44.022808+0000","last_clean":"2024-09-10T18:56:44.022808+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:56:44.022808+0000","last_undegraded":"2024-09-10T18:56:44.022808+0000","last_fullsized":"2024-09-10T18:56:44.022808+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.170223342,"stat_sum":{"num_bytes":140,"num_objects":13,"num_object_clones":6,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"64.17","version":"230'17","reported_seq":155,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033122+0000","last_change":"2024-09-10T18:56:32.455122+0000","last_active":"2024-09-10T18:56:44.033122+0000","last_peered":"2024-09-10T18:56:44.033122+0000","last_clean":"2024-09-10T18:56:44.033122+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:56:44.033122+0000","last_undegraded":"2024-09-10T18:56:44.033122+0000","last_fullsized":"2024-09-10T18:56:44.033122+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37027679899999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.16","version":"236'17","reported_seq":158,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056742+0000","last_change":"2024-09-10T18:56:38.572460+0000","last_active":"2024-09-10T18:56:44.056742+0000","last_peered":"2024-09-10T18:56:44.056742+0000","last_clean":"2024-09-10T18:56:44.056742+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:56:44.056742+0000","last_undegraded":"2024-09-10T18:56:44.056742+0000","last_fullsized":"2024-09-10T18:56:44.056742+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50398346900000002,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.15","version":"240'74","reported_seq":203,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.583994+0000","last_change":"2024-09-10T18:56:44.583994+0000","last_active":"2024-09-10T18:56:44.583994+0000","last_peered":"2024-09-10T18:56:44.583994+0000","last_clean":"2024-09-10T18:56:44.583994+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:56:44.583994+0000","last_undegraded":"2024-09-10T18:56:44.583994+0000","last_fullsized":"2024-09-10T18:56:44.583994+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53863357000000001,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.14","version":"246'95","reported_seq":231,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.595941+0000","last_change":"2024-09-10T18:56:44.595941+0000","last_active":"2024-09-10T18:56:44.595941+0000","last_peered":"2024-09-10T18:56:44.595941+0000","last_clean":"2024-09-10T18:56:44.595941+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:56:44.595941+0000","last_undegraded":"2024-09-10T18:56:44.595941+0000","last_fullsized":"2024-09-10T18:56:44.595941+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.53849196300000002,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.13","version":"146'1","reported_seq":144,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049416+0000","last_change":"2024-09-10T18:56:36.567133+0000","last_active":"2024-09-10T18:56:44.049416+0000","last_peered":"2024-09-10T18:56:44.049416+0000","last_clean":"2024-09-10T18:56:44.049416+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:56:44.049416+0000","last_undegraded":"2024-09-10T18:56:44.049416+0000","last_fullsized":"2024-09-10T18:56:44.049416+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51540947800000003,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.12","version":"248'51","reported_seq":174,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:44.478887+0000","last_change":"2024-09-10T18:56:42.472975+0000","last_active":"2024-09-10T18:56:44.478887+0000","last_peered":"2024-09-10T18:56:44.478887+0000","last_clean":"2024-09-10T18:56:44.478887+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:56:44.478887+0000","last_undegraded":"2024-09-10T18:56:44.478887+0000","last_fullsized":"2024-09-10T18:56:44.478887+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40688358000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"64.11","version":"226'51","reported_seq":186,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015133+0000","last_change":"2024-09-10T18:56:24.435208+0000","last_active":"2024-09-10T18:56:44.015133+0000","last_peered":"2024-09-10T18:56:44.015133+0000","last_clean":"2024-09-10T18:56:44.015133+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:56:44.015133+0000","last_undegraded":"2024-09-10T18:56:44.015133+0000","last_fullsized":"2024-09-10T18:56:44.015133+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42675023899999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.10","version":"224'34","reported_seq":168,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.016891+0000","last_change":"2024-09-10T18:56:22.445588+0000","last_active":"2024-09-10T18:56:44.016891+0000","last_peered":"2024-09-10T18:56:44.016891+0000","last_clean":"2024-09-10T18:56:44.016891+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:56:44.016891+0000","last_undegraded":"2024-09-10T18:56:44.016891+0000","last_fullsized":"2024-09-10T18:56:44.016891+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.422252707,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.f","version":"244'47","reported_seq":180,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:44.025116+0000","last_change":"2024-09-10T18:56:42.472973+0000","last_active":"2024-09-10T18:56:44.025116+0000","last_peered":"2024-09-10T18:56:44.025116+0000","last_clean":"2024-09-10T18:56:44.025116+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:56:44.025116+0000","last_undegraded":"2024-09-10T18:56:44.025116+0000","last_fullsized":"2024-09-10T18:56:44.025116+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37884067999999999,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"64.e","version":"246'119","reported_seq":240,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.549501+0000","last_change":"2024-09-10T18:56:44.549501+0000","last_active":"2024-09-10T18:56:44.549501+0000","last_peered":"2024-09-10T18:56:44.549501+0000","last_clean":"2024-09-10T18:56:44.549501+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:56:44.549501+0000","last_undegraded":"2024-09-10T18:56:44.549501+0000","last_fullsized":"2024-09-10T18:56:44.549501+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51804771500000002,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.d","version":"243'18","reported_seq":171,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.021977+0000","last_change":"2024-09-10T18:56:39.950486+0000","last_active":"2024-09-10T18:56:44.021977+0000","last_peered":"2024-09-10T18:56:44.021977+0000","last_clean":"2024-09-10T18:56:44.021977+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:56:44.021977+0000","last_undegraded":"2024-09-10T18:56:44.021977+0000","last_fullsized":"2024-09-10T18:56:44.021977+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.276748+0000","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.010244544,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.c","version":"242'20","reported_seq":157,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:44.023009+0000","last_change":"2024-09-10T18:56:40.443694+0000","last_active":"2024-09-10T18:56:44.023009+0000","last_peered":"2024-09-10T18:56:44.023009+0000","last_clean":"2024-09-10T18:56:44.023009+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:56:44.023009+0000","last_undegraded":"2024-09-10T18:56:44.023009+0000","last_fullsized":"2024-09-10T18:56:44.023009+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.353884792,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"64.b","version":"247'35","reported_seq":171,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056334+0000","last_change":"2024-09-10T18:56:38.571987+0000","last_active":"2024-09-10T18:56:44.056334+0000","last_peered":"2024-09-10T18:56:44.056334+0000","last_clean":"2024-09-10T18:56:44.056334+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:56:44.056334+0000","last_undegraded":"2024-09-10T18:56:44.056334+0000","last_fullsized":"2024-09-10T18:56:44.056334+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50470740199999997,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.a","version":"232'17","reported_seq":159,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044176+0000","last_change":"2024-09-10T18:56:36.567876+0000","last_active":"2024-09-10T18:56:44.044176+0000","last_peered":"2024-09-10T18:56:44.044176+0000","last_clean":"2024-09-10T18:56:44.044176+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:56:44.044176+0000","last_undegraded":"2024-09-10T18:56:44.044176+0000","last_fullsized":"2024-09-10T18:56:44.044176+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51642296899999995,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049242+0000","last_change":"2024-09-10T18:56:38.570909+0000","last_active":"2024-09-10T18:56:44.049242+0000","last_peered":"2024-09-10T18:56:44.049242+0000","last_clean":"2024-09-10T18:56:44.049242+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:56:44.049242+0000","last_undegraded":"2024-09-10T18:56:44.049242+0000","last_fullsized":"2024-09-10T18:56:44.049242+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.093015560999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.8","version":"234'51","reported_seq":186,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.027767+0000","last_change":"2024-09-10T18:56:32.455863+0000","last_active":"2024-09-10T18:56:44.027767+0000","last_peered":"2024-09-10T18:56:44.027767+0000","last_clean":"2024-09-10T18:56:44.027767+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:56:44.027767+0000","last_undegraded":"2024-09-10T18:56:44.027767+0000","last_fullsized":"2024-09-10T18:56:44.027767+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38419820900000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.7","version":"248'135","reported_seq":255,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:44.598524+0000","last_change":"2024-09-10T18:56:36.568238+0000","last_active":"2024-09-10T18:56:44.598524+0000","last_peered":"2024-09-10T18:56:44.598524+0000","last_clean":"2024-09-10T18:56:44.598524+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:56:44.598524+0000","last_undegraded":"2024-09-10T18:56:44.598524+0000","last_fullsized":"2024-09-10T18:56:44.598524+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.516314618,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":9,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"244'85","reported_seq":214,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025860+0000","last_change":"2024-09-10T18:56:42.556240+0000","last_active":"2024-09-10T18:56:44.025860+0000","last_peered":"2024-09-10T18:56:44.025860+0000","last_clean":"2024-09-10T18:56:44.025860+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:56:44.025860+0000","last_undegraded":"2024-09-10T18:56:44.025860+0000","last_fullsized":"2024-09-10T18:56:44.025860+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44957488800000001,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.5","version":"246'45","reported_seq":174,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-10T18:56:44.451079+0000","last_change":"2024-09-10T18:56:44.451079+0000","last_active":"2024-09-10T18:56:44.451079+0000","last_peered":"2024-09-10T18:56:44.451079+0000","last_clean":"2024-09-10T18:56:44.451079+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:56:44.451079+0000","last_undegraded":"2024-09-10T18:56:44.451079+0000","last_fullsized":"2024-09-10T18:56:44.451079+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42538504700000002,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"64.4","version":"240'102","reported_seq":227,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024557+0000","last_change":"2024-09-10T18:56:38.565192+0000","last_active":"2024-09-10T18:56:44.024557+0000","last_peered":"2024-09-10T18:56:44.024557+0000","last_clean":"2024-09-10T18:56:44.024557+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:56:44.024557+0000","last_undegraded":"2024-09-10T18:56:44.024557+0000","last_fullsized":"2024-09-10T18:56:44.024557+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11330530699999999,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.3","version":"236'102","reported_seq":229,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048198+0000","last_change":"2024-09-10T18:56:34.466051+0000","last_active":"2024-09-10T18:56:44.048198+0000","last_peered":"2024-09-10T18:56:44.048198+0000","last_clean":"2024-09-10T18:56:44.048198+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:56:44.048198+0000","last_undegraded":"2024-09-10T18:56:44.048198+0000","last_fullsized":"2024-09-10T18:56:44.048198+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44840992000000002,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.2","version":"230'34","reported_seq":168,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029167+0000","last_change":"2024-09-10T18:56:32.454632+0000","last_active":"2024-09-10T18:56:44.029167+0000","last_peered":"2024-09-10T18:56:44.029167+0000","last_clean":"2024-09-10T18:56:44.029167+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:56:44.029167+0000","last_undegraded":"2024-09-10T18:56:44.029167+0000","last_fullsized":"2024-09-10T18:56:44.029167+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37498961600000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"64.1","version":"248'65","reported_seq":200,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:56:44.455227+0000","last_change":"2024-09-10T18:56:44.451210+0000","last_active":"2024-09-10T18:56:44.455227+0000","last_peered":"2024-09-10T18:56:44.455227+0000","last_clean":"2024-09-10T18:56:44.455227+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:56:44.455227+0000","last_undegraded":"2024-09-10T18:56:44.455227+0000","last_fullsized":"2024-09-10T18:56:44.455227+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.141586712,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"64.0","version":"246'102","reported_seq":234,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.683699+0000","last_change":"2024-09-10T18:56:44.683613+0000","last_active":"2024-09-10T18:56:44.683699+0000","last_peered":"2024-09-10T18:56:44.683699+0000","last_clean":"2024-09-10T18:56:44.683699+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:56:44.683699+0000","last_undegraded":"2024-09-10T18:56:44.683699+0000","last_fullsized":"2024-09-10T18:56:44.683699+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:56:44.683516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.683516+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.401562+0000","scrub_duration":0.0037708920000000001,"objects_trimmed":0,"snaptrim_duration":0.53312375300000003,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":212,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047682+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:56:44.047682+0000","last_peered":"2024-09-10T18:56:44.047682+0000","last_clean":"2024-09-10T18:56:44.047682+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:56:44.047682+0000","last_undegraded":"2024-09-10T18:56:44.047682+0000","last_fullsized":"2024-09-10T18:56:44.047682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.508785+0000","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":227,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059587+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:56:44.059587+0000","last_peered":"2024-09-10T18:56:44.059587+0000","last_clean":"2024-09-10T18:56:44.059587+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:56:44.059587+0000","last_undegraded":"2024-09-10T18:56:44.059587+0000","last_fullsized":"2024-09-10T18:56:44.059587+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:10.569242+0000","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":231,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.051011+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:56:44.051011+0000","last_peered":"2024-09-10T18:56:44.051011+0000","last_clean":"2024-09-10T18:56:44.051011+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:56:44.051011+0000","last_undegraded":"2024-09-10T18:56:44.051011+0000","last_fullsized":"2024-09-10T18:56:44.051011+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.485909+0000","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.046545+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:56:44.046545+0000","last_peered":"2024-09-10T18:56:44.046545+0000","last_clean":"2024-09-10T18:56:44.046545+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:56:44.046545+0000","last_undegraded":"2024-09-10T18:56:44.046545+0000","last_fullsized":"2024-09-10T18:56:44.046545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.646246+0000","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023627+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:56:44.023627+0000","last_peered":"2024-09-10T18:56:44.023627+0000","last_clean":"2024-09-10T18:56:44.023627+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:56:44.023627+0000","last_undegraded":"2024-09-10T18:56:44.023627+0000","last_fullsized":"2024-09-10T18:56:44.023627+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.539352+0000","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"243'6","reported_seq":149,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.031538+0000","last_change":"2024-09-10T18:56:39.792005+0000","last_active":"2024-09-10T18:56:44.031538+0000","last_peered":"2024-09-10T18:56:44.031538+0000","last_clean":"2024-09-10T18:56:44.031538+0000","last_became_active":"2024-09-10T18:54:37.707288+0000","last_became_peered":"2024-09-10T18:54:37.707288+0000","last_unstale":"2024-09-10T18:56:44.031538+0000","last_undegraded":"2024-09-10T18:56:44.031538+0000","last_fullsized":"2024-09-10T18:56:44.031538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'6","last_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_deep_scrub":"243'6","last_deep_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.791846+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:07.646797+0000","scrub_duration":0.0069143870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":228,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048207+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:56:44.048207+0000","last_peered":"2024-09-10T18:56:44.048207+0000","last_clean":"2024-09-10T18:56:44.048207+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:56:44.048207+0000","last_undegraded":"2024-09-10T18:56:44.048207+0000","last_fullsized":"2024-09-10T18:56:44.048207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":244,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059315+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:56:44.059315+0000","last_peered":"2024-09-10T18:56:44.059315+0000","last_clean":"2024-09-10T18:56:44.059315+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:56:44.059315+0000","last_undegraded":"2024-09-10T18:56:44.059315+0000","last_fullsized":"2024-09-10T18:56:44.059315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.632174+0000","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":234,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015578+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:56:44.015578+0000","last_peered":"2024-09-10T18:56:44.015578+0000","last_clean":"2024-09-10T18:56:44.015578+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:56:44.015578+0000","last_undegraded":"2024-09-10T18:56:44.015578+0000","last_fullsized":"2024-09-10T18:56:44.015578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.405877+0000","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.016287+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:56:44.016287+0000","last_peered":"2024-09-10T18:56:44.016287+0000","last_clean":"2024-09-10T18:56:44.016287+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:56:44.016287+0000","last_undegraded":"2024-09-10T18:56:44.016287+0000","last_fullsized":"2024-09-10T18:56:44.016287+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":132,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.053690+0000","last_change":"2024-09-10T18:54:37.650137+0000","last_active":"2024-09-10T18:56:44.053690+0000","last_peered":"2024-09-10T18:56:44.053690+0000","last_clean":"2024-09-10T18:56:44.053690+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:56:44.053690+0000","last_undegraded":"2024-09-10T18:56:44.053690+0000","last_fullsized":"2024-09-10T18:56:44.053690+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023172+0000","last_change":"2024-09-10T18:56:42.362259+0000","last_active":"2024-09-10T18:56:44.023172+0000","last_peered":"2024-09-10T18:56:44.023172+0000","last_clean":"2024-09-10T18:56:44.023172+0000","last_became_active":"2024-09-10T18:54:37.597613+0000","last_became_peered":"2024-09-10T18:54:37.597613+0000","last_unstale":"2024-09-10T18:56:44.023172+0000","last_undegraded":"2024-09-10T18:56:44.023172+0000","last_fullsized":"2024-09-10T18:56:44.023172+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.362084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.362084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.874685+0000","scrub_duration":0.0031795769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":225,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023593+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:56:44.023593+0000","last_peered":"2024-09-10T18:56:44.023593+0000","last_clean":"2024-09-10T18:56:44.023593+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:56:44.023593+0000","last_undegraded":"2024-09-10T18:56:44.023593+0000","last_fullsized":"2024-09-10T18:56:44.023593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:25.677268+0000","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":246,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022632+0000","last_change":"2024-09-10T18:54:41.188135+0000","last_active":"2024-09-10T18:56:44.022632+0000","last_peered":"2024-09-10T18:56:44.022632+0000","last_clean":"2024-09-10T18:56:44.022632+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:56:44.022632+0000","last_undegraded":"2024-09-10T18:56:44.022632+0000","last_fullsized":"2024-09-10T18:56:44.022632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:54:41.188040+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.188040+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036526110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.016971+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:56:44.016971+0000","last_peered":"2024-09-10T18:56:44.016971+0000","last_clean":"2024-09-10T18:56:44.016971+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:56:44.016971+0000","last_undegraded":"2024-09-10T18:56:44.016971+0000","last_fullsized":"2024-09-10T18:56:44.016971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.192711+0000","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":213,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023738+0000","last_change":"2024-09-10T18:56:32.263959+0000","last_active":"2024-09-10T18:56:44.023738+0000","last_peered":"2024-09-10T18:56:44.023738+0000","last_clean":"2024-09-10T18:56:44.023738+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:56:44.023738+0000","last_undegraded":"2024-09-10T18:56:44.023738+0000","last_fullsized":"2024-09-10T18:56:44.023738+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:39.242050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:39.242050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002953027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056516+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:56:44.056516+0000","last_peered":"2024-09-10T18:56:44.056516+0000","last_clean":"2024-09-10T18:56:44.056516+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:56:44.056516+0000","last_undegraded":"2024-09-10T18:56:44.056516+0000","last_fullsized":"2024-09-10T18:56:44.056516+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.799711+0000","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":132,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059721+0000","last_change":"2024-09-10T18:54:37.549144+0000","last_active":"2024-09-10T18:56:44.059721+0000","last_peered":"2024-09-10T18:56:44.059721+0000","last_clean":"2024-09-10T18:56:44.059721+0000","last_became_active":"2024-09-10T18:54:37.543738+0000","last_became_peered":"2024-09-10T18:54:37.543738+0000","last_unstale":"2024-09-10T18:56:44.059721+0000","last_undegraded":"2024-09-10T18:56:44.059721+0000","last_fullsized":"2024-09-10T18:56:44.059721+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":218,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.026469+0000","last_change":"2024-09-10T18:56:22.517441+0000","last_active":"2024-09-10T18:56:44.026469+0000","last_peered":"2024-09-10T18:56:44.026469+0000","last_clean":"2024-09-10T18:56:44.026469+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:56:44.026469+0000","last_undegraded":"2024-09-10T18:56:44.026469+0000","last_fullsized":"2024-09-10T18:56:44.026469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:52.078168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:52.078168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029805980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":262,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025283+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:56:44.025283+0000","last_peered":"2024-09-10T18:56:44.025283+0000","last_clean":"2024-09-10T18:56:44.025283+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:56:44.025283+0000","last_undegraded":"2024-09-10T18:56:44.025283+0000","last_fullsized":"2024-09-10T18:56:44.025283+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.744708+0000","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017200+0000","last_change":"2024-09-10T18:54:55.309898+0000","last_active":"2024-09-10T18:56:44.017200+0000","last_peered":"2024-09-10T18:56:44.017200+0000","last_clean":"2024-09-10T18:56:44.017200+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:56:44.017200+0000","last_undegraded":"2024-09-10T18:56:44.017200+0000","last_fullsized":"2024-09-10T18:56:44.017200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.309689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.309689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003220512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":221,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.042847+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:56:44.042847+0000","last_peered":"2024-09-10T18:56:44.042847+0000","last_clean":"2024-09-10T18:56:44.042847+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:56:44.042847+0000","last_undegraded":"2024-09-10T18:56:44.042847+0000","last_fullsized":"2024-09-10T18:56:44.042847+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047600+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:56:44.047600+0000","last_peered":"2024-09-10T18:56:44.047600+0000","last_clean":"2024-09-10T18:56:44.047600+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:56:44.047600+0000","last_undegraded":"2024-09-10T18:56:44.047600+0000","last_fullsized":"2024-09-10T18:56:44.047600+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.879768+0000","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.050688+0000","last_change":"2024-09-10T18:56:17.672189+0000","last_active":"2024-09-10T18:56:44.050688+0000","last_peered":"2024-09-10T18:56:44.050688+0000","last_clean":"2024-09-10T18:56:44.050688+0000","last_became_active":"2024-09-10T18:54:37.653532+0000","last_became_peered":"2024-09-10T18:54:37.653532+0000","last_unstale":"2024-09-10T18:56:44.050688+0000","last_undegraded":"2024-09-10T18:56:44.050688+0000","last_fullsized":"2024-09-10T18:56:44.050688+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.745658+0000","scrub_duration":0.0037230840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":255,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029431+0000","last_change":"2024-09-10T18:54:50.687363+0000","last_active":"2024-09-10T18:56:44.029431+0000","last_peered":"2024-09-10T18:56:44.029431+0000","last_clean":"2024-09-10T18:56:44.029431+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:56:44.029431+0000","last_undegraded":"2024-09-10T18:56:44.029431+0000","last_fullsized":"2024-09-10T18:56:44.029431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:54:50.687166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.687166+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053731969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":215,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023159+0000","last_change":"2024-09-10T18:54:59.926609+0000","last_active":"2024-09-10T18:56:44.023159+0000","last_peered":"2024-09-10T18:56:44.023159+0000","last_clean":"2024-09-10T18:56:44.023159+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:56:44.023159+0000","last_undegraded":"2024-09-10T18:56:44.023159+0000","last_fullsized":"2024-09-10T18:56:44.023159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:59.926330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.926330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035578459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":324,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.020842+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:56:44.020842+0000","last_peered":"2024-09-10T18:56:44.020842+0000","last_clean":"2024-09-10T18:56:44.020842+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:56:44.020842+0000","last_undegraded":"2024-09-10T18:56:44.020842+0000","last_fullsized":"2024-09-10T18:56:44.020842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.050927+0000","last_change":"2024-09-10T18:55:01.419421+0000","last_active":"2024-09-10T18:56:44.050927+0000","last_peered":"2024-09-10T18:56:44.050927+0000","last_clean":"2024-09-10T18:56:44.050927+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:56:44.050927+0000","last_undegraded":"2024-09-10T18:56:44.050927+0000","last_fullsized":"2024-09-10T18:56:44.050927+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.419333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.419333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035926959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.031378+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:56:44.031378+0000","last_peered":"2024-09-10T18:56:44.031378+0000","last_clean":"2024-09-10T18:56:44.031378+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:56:44.031378+0000","last_undegraded":"2024-09-10T18:56:44.031378+0000","last_fullsized":"2024-09-10T18:56:44.031378+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.697752+0000","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015649+0000","last_change":"2024-09-10T18:56:05.989083+0000","last_active":"2024-09-10T18:56:44.015649+0000","last_peered":"2024-09-10T18:56:44.015649+0000","last_clean":"2024-09-10T18:56:44.015649+0000","last_became_active":"2024-09-10T18:54:37.569079+0000","last_became_peered":"2024-09-10T18:54:37.569079+0000","last_unstale":"2024-09-10T18:56:44.015649+0000","last_undegraded":"2024-09-10T18:56:44.015649+0000","last_fullsized":"2024-09-10T18:56:44.015649+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.988916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.988916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:07.493116+0000","scrub_duration":0.0038149899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":246,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048409+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:56:44.048409+0000","last_peered":"2024-09-10T18:56:44.048409+0000","last_clean":"2024-09-10T18:56:44.048409+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:56:44.048409+0000","last_undegraded":"2024-09-10T18:56:44.048409+0000","last_fullsized":"2024-09-10T18:56:44.048409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.663745+0000","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":236,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.052141+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:56:44.052141+0000","last_peered":"2024-09-10T18:56:44.052141+0000","last_clean":"2024-09-10T18:56:44.052141+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:56:44.052141+0000","last_undegraded":"2024-09-10T18:56:44.052141+0000","last_fullsized":"2024-09-10T18:56:44.052141+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.015829+0000","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":234,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.019253+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:56:44.019253+0000","last_peered":"2024-09-10T18:56:44.019253+0000","last_clean":"2024-09-10T18:56:44.019253+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:56:44.019253+0000","last_undegraded":"2024-09-10T18:56:44.019253+0000","last_fullsized":"2024-09-10T18:56:44.019253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.670897+0000","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.020375+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:56:44.020375+0000","last_peered":"2024-09-10T18:56:44.020375+0000","last_clean":"2024-09-10T18:56:44.020375+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:56:44.020375+0000","last_undegraded":"2024-09-10T18:56:44.020375+0000","last_fullsized":"2024-09-10T18:56:44.020375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.028165+0000","last_change":"2024-09-10T18:54:37.600609+0000","last_active":"2024-09-10T18:56:44.028165+0000","last_peered":"2024-09-10T18:56:44.028165+0000","last_clean":"2024-09-10T18:56:44.028165+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:56:44.028165+0000","last_undegraded":"2024-09-10T18:56:44.028165+0000","last_fullsized":"2024-09-10T18:56:44.028165+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.18","version":"126'2","reported_seq":134,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.053606+0000","last_change":"2024-09-10T18:54:37.671736+0000","last_active":"2024-09-10T18:56:44.053606+0000","last_peered":"2024-09-10T18:56:44.053606+0000","last_clean":"2024-09-10T18:56:44.053606+0000","last_became_active":"2024-09-10T18:54:37.671088+0000","last_became_peered":"2024-09-10T18:54:37.671088+0000","last_unstale":"2024-09-10T18:56:44.053606+0000","last_undegraded":"2024-09-10T18:56:44.053606+0000","last_fullsized":"2024-09-10T18:56:44.053606+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":251,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044444+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:56:44.044444+0000","last_peered":"2024-09-10T18:56:44.044444+0000","last_clean":"2024-09-10T18:56:44.044444+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:56:44.044444+0000","last_undegraded":"2024-09-10T18:56:44.044444+0000","last_fullsized":"2024-09-10T18:56:44.044444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.658076+0000","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049570+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:56:44.049570+0000","last_peered":"2024-09-10T18:56:44.049570+0000","last_clean":"2024-09-10T18:56:44.049570+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:56:44.049570+0000","last_undegraded":"2024-09-10T18:56:44.049570+0000","last_fullsized":"2024-09-10T18:56:44.049570+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.935684+0000","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":232,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057795+0000","last_change":"2024-09-10T18:56:39.629880+0000","last_active":"2024-09-10T18:56:44.057795+0000","last_peered":"2024-09-10T18:56:44.057795+0000","last_clean":"2024-09-10T18:56:44.057795+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:56:44.057795+0000","last_undegraded":"2024-09-10T18:56:44.057795+0000","last_fullsized":"2024-09-10T18:56:44.057795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:39.629726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.629726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.147581+0000","scrub_duration":0.0034450290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":230,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024978+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:56:44.024978+0000","last_peered":"2024-09-10T18:56:44.024978+0000","last_clean":"2024-09-10T18:56:44.024978+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:56:44.024978+0000","last_undegraded":"2024-09-10T18:56:44.024978+0000","last_fullsized":"2024-09-10T18:56:44.024978+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.752916+0000","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.060826+0000","last_change":"2024-09-10T18:56:17.674109+0000","last_active":"2024-09-10T18:56:44.060826+0000","last_peered":"2024-09-10T18:56:44.060826+0000","last_clean":"2024-09-10T18:56:44.060826+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:56:44.060826+0000","last_undegraded":"2024-09-10T18:56:44.060826+0000","last_fullsized":"2024-09-10T18:56:44.060826+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.673943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.673943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:22.872965+0000","scrub_duration":0.003315691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1b","version":"126'10","reported_seq":153,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.026542+0000","last_change":"2024-09-10T18:56:35.237751+0000","last_active":"2024-09-10T18:56:44.026542+0000","last_peered":"2024-09-10T18:56:44.026542+0000","last_clean":"2024-09-10T18:56:44.026542+0000","last_became_active":"2024-09-10T18:54:37.522926+0000","last_became_peered":"2024-09-10T18:54:37.522926+0000","last_unstale":"2024-09-10T18:56:44.026542+0000","last_undegraded":"2024-09-10T18:56:44.026542+0000","last_fullsized":"2024-09-10T18:56:44.026542+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:35.237574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.237574+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:57.596228+0000","scrub_duration":0.0042776200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":264,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.980617+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:56:44.980617+0000","last_peered":"2024-09-10T18:56:44.980617+0000","last_clean":"2024-09-10T18:56:44.980617+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:56:44.980617+0000","last_undegraded":"2024-09-10T18:56:44.980617+0000","last_fullsized":"2024-09-10T18:56:44.980617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.690508+0000","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":225,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022580+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:56:44.022580+0000","last_peered":"2024-09-10T18:56:44.022580+0000","last_clean":"2024-09-10T18:56:44.022580+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:56:44.022580+0000","last_undegraded":"2024-09-10T18:56:44.022580+0000","last_fullsized":"2024-09-10T18:56:44.022580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:20.015637+0000","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":324,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.020563+0000","last_change":"2024-09-10T18:54:59.278861+0000","last_active":"2024-09-10T18:56:44.020563+0000","last_peered":"2024-09-10T18:56:44.020563+0000","last_clean":"2024-09-10T18:56:44.020563+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:56:44.020563+0000","last_undegraded":"2024-09-10T18:56:44.020563+0000","last_fullsized":"2024-09-10T18:56:44.020563+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:54:59.278694+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.278694+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038345380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049627+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:56:44.049627+0000","last_peered":"2024-09-10T18:56:44.049627+0000","last_clean":"2024-09-10T18:56:44.049627+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:56:44.049627+0000","last_undegraded":"2024-09-10T18:56:44.049627+0000","last_fullsized":"2024-09-10T18:56:44.049627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.519413+0000","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057011+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:56:44.057011+0000","last_peered":"2024-09-10T18:56:44.057011+0000","last_clean":"2024-09-10T18:56:44.057011+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:56:44.057011+0000","last_undegraded":"2024-09-10T18:56:44.057011+0000","last_fullsized":"2024-09-10T18:56:44.057011+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.543952+0000","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"126'5","reported_seq":138,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.046547+0000","last_change":"2024-09-10T18:54:37.686458+0000","last_active":"2024-09-10T18:56:44.046547+0000","last_peered":"2024-09-10T18:56:44.046547+0000","last_clean":"2024-09-10T18:56:44.046547+0000","last_became_active":"2024-09-10T18:54:37.686146+0000","last_became_peered":"2024-09-10T18:54:37.686146+0000","last_unstale":"2024-09-10T18:56:44.046547+0000","last_undegraded":"2024-09-10T18:56:44.046547+0000","last_fullsized":"2024-09-10T18:56:44.046547+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":233,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057838+0000","last_change":"2024-09-10T18:54:37.847063+0000","last_active":"2024-09-10T18:56:44.057838+0000","last_peered":"2024-09-10T18:56:44.057838+0000","last_clean":"2024-09-10T18:56:44.057838+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:56:44.057838+0000","last_undegraded":"2024-09-10T18:56:44.057838+0000","last_fullsized":"2024-09-10T18:56:44.057838+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:54:37.846984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:37.846984+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098700820000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033818+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:56:44.033818+0000","last_peered":"2024-09-10T18:56:44.033818+0000","last_clean":"2024-09-10T18:56:44.033818+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:56:44.033818+0000","last_undegraded":"2024-09-10T18:56:44.033818+0000","last_fullsized":"2024-09-10T18:56:44.033818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015844+0000","last_change":"2024-09-10T18:54:52.331412+0000","last_active":"2024-09-10T18:56:44.015844+0000","last_peered":"2024-09-10T18:56:44.015844+0000","last_clean":"2024-09-10T18:56:44.015844+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:56:44.015844+0000","last_undegraded":"2024-09-10T18:56:44.015844+0000","last_fullsized":"2024-09-10T18:56:44.015844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:52.331111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:52.331111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034919479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":655,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-09-10T18:56:45.019691+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:56:45.019691+0000","last_peered":"2024-09-10T18:56:45.019691+0000","last_clean":"2024-09-10T18:56:45.019691+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:56:45.019691+0000","last_undegraded":"2024-09-10T18:56:45.019691+0000","last_fullsized":"2024-09-10T18:56:45.019691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.052626+0000","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":423,"num_read_kb":418,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.021874+0000","last_change":"2024-09-10T18:55:07.836217+0000","last_active":"2024-09-10T18:56:44.021874+0000","last_peered":"2024-09-10T18:56:44.021874+0000","last_clean":"2024-09-10T18:56:44.021874+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:56:44.021874+0000","last_undegraded":"2024-09-10T18:56:44.021874+0000","last_fullsized":"2024-09-10T18:56:44.021874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:07.836145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:23.121359+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.836145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031616470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048884+0000","last_change":"2024-09-10T18:56:12.680019+0000","last_active":"2024-09-10T18:56:44.048884+0000","last_peered":"2024-09-10T18:56:44.048884+0000","last_clean":"2024-09-10T18:56:44.048884+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:56:44.048884+0000","last_undegraded":"2024-09-10T18:56:44.048884+0000","last_fullsized":"2024-09-10T18:56:44.048884+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:12.679938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:12.679938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.716839+0000","scrub_duration":0.002716511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.189490+0000","last_change":"2024-09-10T18:56:44.189392+0000","last_active":"2024-09-10T18:56:44.189490+0000","last_peered":"2024-09-10T18:56:44.189490+0000","last_clean":"2024-09-10T18:56:44.189490+0000","last_became_active":"2024-09-10T18:54:37.631506+0000","last_became_peered":"2024-09-10T18:54:37.631506+0000","last_unstale":"2024-09-10T18:56:44.189490+0000","last_undegraded":"2024-09-10T18:56:44.189490+0000","last_fullsized":"2024-09-10T18:56:44.189490+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:44.189240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.189240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:49.319075+0000","scrub_duration":0.0036310299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":253,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048716+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:56:44.048716+0000","last_peered":"2024-09-10T18:56:44.048716+0000","last_clean":"2024-09-10T18:56:44.048716+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:56:44.048716+0000","last_undegraded":"2024-09-10T18:56:44.048716+0000","last_fullsized":"2024-09-10T18:56:44.048716+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.286896+0000","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":236,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.060620+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:56:44.060620+0000","last_peered":"2024-09-10T18:56:44.060620+0000","last_clean":"2024-09-10T18:56:44.060620+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:56:44.060620+0000","last_undegraded":"2024-09-10T18:56:44.060620+0000","last_fullsized":"2024-09-10T18:56:44.060620+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:59.107812+0000","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":313,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043813+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:56:44.043813+0000","last_peered":"2024-09-10T18:56:44.043813+0000","last_clean":"2024-09-10T18:56:44.043813+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:56:44.043813+0000","last_undegraded":"2024-09-10T18:56:44.043813+0000","last_fullsized":"2024-09-10T18:56:44.043813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.264937+0000","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":241,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015732+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:56:44.015732+0000","last_peered":"2024-09-10T18:56:44.015732+0000","last_clean":"2024-09-10T18:56:44.015732+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:56:44.015732+0000","last_undegraded":"2024-09-10T18:56:44.015732+0000","last_fullsized":"2024-09-10T18:56:44.015732+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:00.852108+0000","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.016624+0000","last_change":"2024-09-10T18:54:46.264917+0000","last_active":"2024-09-10T18:56:44.016624+0000","last_peered":"2024-09-10T18:56:44.016624+0000","last_clean":"2024-09-10T18:56:44.016624+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:56:44.016624+0000","last_undegraded":"2024-09-10T18:56:44.016624+0000","last_fullsized":"2024-09-10T18:56:44.016624+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:46.264846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:46.264846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027169630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043686+0000","last_change":"2024-09-10T18:54:37.726541+0000","last_active":"2024-09-10T18:56:44.043686+0000","last_peered":"2024-09-10T18:56:44.043686+0000","last_clean":"2024-09-10T18:56:44.043686+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:56:44.043686+0000","last_undegraded":"2024-09-10T18:56:44.043686+0000","last_fullsized":"2024-09-10T18:56:44.043686+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"126'2","reported_seq":144,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056829+0000","last_change":"2024-09-10T18:56:39.169695+0000","last_active":"2024-09-10T18:56:44.056829+0000","last_peered":"2024-09-10T18:56:44.056829+0000","last_clean":"2024-09-10T18:56:44.056829+0000","last_became_active":"2024-09-10T18:54:37.677049+0000","last_became_peered":"2024-09-10T18:54:37.677049+0000","last_unstale":"2024-09-10T18:56:44.056829+0000","last_undegraded":"2024-09-10T18:56:44.056829+0000","last_fullsized":"2024-09-10T18:56:44.056829+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:39.169459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.169459+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.615905+0000","scrub_duration":0.0030948830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":373,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061451+0000","last_change":"2024-09-10T18:56:03.530880+0000","last_active":"2024-09-10T18:56:44.061451+0000","last_peered":"2024-09-10T18:56:44.061451+0000","last_clean":"2024-09-10T18:56:44.061451+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:56:44.061451+0000","last_undegraded":"2024-09-10T18:56:44.061451+0000","last_fullsized":"2024-09-10T18:56:44.061451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:56:03.530741+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.530741+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:04.774205+0000","scrub_duration":0.0049201490000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":225,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024259+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:56:44.024259+0000","last_peered":"2024-09-10T18:56:44.024259+0000","last_clean":"2024-09-10T18:56:44.024259+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:56:44.024259+0000","last_undegraded":"2024-09-10T18:56:44.024259+0000","last_fullsized":"2024-09-10T18:56:44.024259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.253426+0000","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":323,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056010+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:56:44.056010+0000","last_peered":"2024-09-10T18:56:44.056010+0000","last_clean":"2024-09-10T18:56:44.056010+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:56:44.056010+0000","last_undegraded":"2024-09-10T18:56:44.056010+0000","last_fullsized":"2024-09-10T18:56:44.056010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":230,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029295+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:56:44.029295+0000","last_peered":"2024-09-10T18:56:44.029295+0000","last_clean":"2024-09-10T18:56:44.029295+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:56:44.029295+0000","last_undegraded":"2024-09-10T18:56:44.029295+0000","last_fullsized":"2024-09-10T18:56:44.029295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.051727+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:56:44.051727+0000","last_peered":"2024-09-10T18:56:44.051727+0000","last_clean":"2024-09-10T18:56:44.051727+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:56:44.051727+0000","last_undegraded":"2024-09-10T18:56:44.051727+0000","last_fullsized":"2024-09-10T18:56:44.051727+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.876489+0000","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048620+0000","last_change":"2024-09-10T18:54:37.710475+0000","last_active":"2024-09-10T18:56:44.048620+0000","last_peered":"2024-09-10T18:56:44.048620+0000","last_clean":"2024-09-10T18:56:44.048620+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:56:44.048620+0000","last_undegraded":"2024-09-10T18:56:44.048620+0000","last_fullsized":"2024-09-10T18:56:44.048620+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"126'5","reported_seq":147,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.051104+0000","last_change":"2024-09-10T18:56:27.790241+0000","last_active":"2024-09-10T18:56:44.051104+0000","last_peered":"2024-09-10T18:56:44.051104+0000","last_clean":"2024-09-10T18:56:44.051104+0000","last_became_active":"2024-09-10T18:54:37.589242+0000","last_became_peered":"2024-09-10T18:54:37.589242+0000","last_unstale":"2024-09-10T18:56:44.051104+0000","last_undegraded":"2024-09-10T18:56:44.051104+0000","last_fullsized":"2024-09-10T18:56:44.051104+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.790063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.790063+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.541752+0000","scrub_duration":0.0035948629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":246,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025365+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:56:44.025365+0000","last_peered":"2024-09-10T18:56:44.025365+0000","last_clean":"2024-09-10T18:56:44.025365+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:56:44.025365+0000","last_undegraded":"2024-09-10T18:56:44.025365+0000","last_fullsized":"2024-09-10T18:56:44.025365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061558+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:56:44.061558+0000","last_peered":"2024-09-10T18:56:44.061558+0000","last_clean":"2024-09-10T18:56:44.061558+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:56:44.061558+0000","last_undegraded":"2024-09-10T18:56:44.061558+0000","last_fullsized":"2024-09-10T18:56:44.061558+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.138878+0000","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.051900+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:56:44.051900+0000","last_peered":"2024-09-10T18:56:44.051900+0000","last_clean":"2024-09-10T18:56:44.051900+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:56:44.051900+0000","last_undegraded":"2024-09-10T18:56:44.051900+0000","last_fullsized":"2024-09-10T18:56:44.051900+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:29.391815+0000","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":221,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043350+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:56:44.043350+0000","last_peered":"2024-09-10T18:56:44.043350+0000","last_clean":"2024-09-10T18:56:44.043350+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:56:44.043350+0000","last_undegraded":"2024-09-10T18:56:44.043350+0000","last_fullsized":"2024-09-10T18:56:44.043350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.883147+0000","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015406+0000","last_change":"2024-09-10T18:56:13.649580+0000","last_active":"2024-09-10T18:56:44.015406+0000","last_peered":"2024-09-10T18:56:44.015406+0000","last_clean":"2024-09-10T18:56:44.015406+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:56:44.015406+0000","last_undegraded":"2024-09-10T18:56:44.015406+0000","last_fullsized":"2024-09-10T18:56:44.015406+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.649458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.649458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.256522+0000","scrub_duration":0.0027534759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"126'5","reported_seq":138,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.050889+0000","last_change":"2024-09-10T18:54:37.679859+0000","last_active":"2024-09-10T18:56:44.050889+0000","last_peered":"2024-09-10T18:56:44.050889+0000","last_clean":"2024-09-10T18:56:44.050889+0000","last_became_active":"2024-09-10T18:54:37.675561+0000","last_became_peered":"2024-09-10T18:54:37.675561+0000","last_unstale":"2024-09-10T18:56:44.050889+0000","last_undegraded":"2024-09-10T18:56:44.050889+0000","last_fullsized":"2024-09-10T18:56:44.050889+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":262,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017518+0000","last_change":"2024-09-10T18:55:15.131835+0000","last_active":"2024-09-10T18:56:44.017518+0000","last_peered":"2024-09-10T18:56:44.017518+0000","last_clean":"2024-09-10T18:56:44.017518+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:56:44.017518+0000","last_undegraded":"2024-09-10T18:56:44.017518+0000","last_fullsized":"2024-09-10T18:56:44.017518+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:55:15.131693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:15.131693+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048796719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029889+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:56:44.029889+0000","last_peered":"2024-09-10T18:56:44.029889+0000","last_clean":"2024-09-10T18:56:44.029889+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:56:44.029889+0000","last_undegraded":"2024-09-10T18:56:44.029889+0000","last_fullsized":"2024-09-10T18:56:44.029889+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061317+0000","last_change":"2024-09-10T18:56:04.495501+0000","last_active":"2024-09-10T18:56:44.061317+0000","last_peered":"2024-09-10T18:56:44.061317+0000","last_clean":"2024-09-10T18:56:44.061317+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:56:44.061317+0000","last_undegraded":"2024-09-10T18:56:44.061317+0000","last_fullsized":"2024-09-10T18:56:44.061317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.495334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.495334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:06.519347+0000","scrub_duration":0.003611179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":212,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047529+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:56:44.047529+0000","last_peered":"2024-09-10T18:56:44.047529+0000","last_clean":"2024-09-10T18:56:44.047529+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:56:44.047529+0000","last_undegraded":"2024-09-10T18:56:44.047529+0000","last_fullsized":"2024-09-10T18:56:44.047529+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.503223+0000","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056222+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:56:44.056222+0000","last_peered":"2024-09-10T18:56:44.056222+0000","last_clean":"2024-09-10T18:56:44.056222+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:56:44.056222+0000","last_undegraded":"2024-09-10T18:56:44.056222+0000","last_fullsized":"2024-09-10T18:56:44.056222+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.215962+0000","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.054511+0000","last_change":"2024-09-10T18:56:37.795971+0000","last_active":"2024-09-10T18:56:44.054511+0000","last_peered":"2024-09-10T18:56:44.054511+0000","last_clean":"2024-09-10T18:56:44.054511+0000","last_became_active":"2024-09-10T18:54:37.561404+0000","last_became_peered":"2024-09-10T18:54:37.561404+0000","last_unstale":"2024-09-10T18:56:44.054511+0000","last_undegraded":"2024-09-10T18:56:44.054511+0000","last_fullsized":"2024-09-10T18:56:44.054511+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.795811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.795811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.466240+0000","scrub_duration":0.0055362429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":264,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.046727+0000","last_change":"2024-09-10T18:54:56.479766+0000","last_active":"2024-09-10T18:56:44.046727+0000","last_peered":"2024-09-10T18:56:44.046727+0000","last_clean":"2024-09-10T18:56:44.046727+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:56:44.046727+0000","last_undegraded":"2024-09-10T18:56:44.046727+0000","last_fullsized":"2024-09-10T18:56:44.046727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:54:56.479670+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.479670+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003399188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":236,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048751+0000","last_change":"2024-09-10T18:56:15.655135+0000","last_active":"2024-09-10T18:56:44.048751+0000","last_peered":"2024-09-10T18:56:44.048751+0000","last_clean":"2024-09-10T18:56:44.048751+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:56:44.048751+0000","last_undegraded":"2024-09-10T18:56:44.048751+0000","last_fullsized":"2024-09-10T18:56:44.048751+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:15.655055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.655055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.233854+0000","scrub_duration":0.0020254439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.060773+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:56:44.060773+0000","last_peered":"2024-09-10T18:56:44.060773+0000","last_clean":"2024-09-10T18:56:44.060773+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:56:44.060773+0000","last_undegraded":"2024-09-10T18:56:44.060773+0000","last_fullsized":"2024-09-10T18:56:44.060773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.219400+0000","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023788+0000","last_change":"2024-09-10T18:56:06.517243+0000","last_active":"2024-09-10T18:56:44.023788+0000","last_peered":"2024-09-10T18:56:44.023788+0000","last_clean":"2024-09-10T18:56:44.023788+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:56:44.023788+0000","last_undegraded":"2024-09-10T18:56:44.023788+0000","last_fullsized":"2024-09-10T18:56:44.023788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.517079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.517079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:07.534844+0000","scrub_duration":0.0038953730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033434+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:56:44.033434+0000","last_peered":"2024-09-10T18:56:44.033434+0000","last_clean":"2024-09-10T18:56:44.033434+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:56:44.033434+0000","last_undegraded":"2024-09-10T18:56:44.033434+0000","last_fullsized":"2024-09-10T18:56:44.033434+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.237861+0000","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"126'9","reported_seq":151,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056816+0000","last_change":"2024-09-10T18:56:15.650829+0000","last_active":"2024-09-10T18:56:44.056816+0000","last_peered":"2024-09-10T18:56:44.056816+0000","last_clean":"2024-09-10T18:56:44.056816+0000","last_became_active":"2024-09-10T18:54:37.676077+0000","last_became_peered":"2024-09-10T18:54:37.676077+0000","last_unstale":"2024-09-10T18:56:44.056816+0000","last_undegraded":"2024-09-10T18:56:44.056816+0000","last_fullsized":"2024-09-10T18:56:44.056816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:15.650681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.650681+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.012078+0000","scrub_duration":0.0053199110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":246,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043294+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:56:44.043294+0000","last_peered":"2024-09-10T18:56:44.043294+0000","last_clean":"2024-09-10T18:56:44.043294+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:56:44.043294+0000","last_undegraded":"2024-09-10T18:56:44.043294+0000","last_fullsized":"2024-09-10T18:56:44.043294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":225,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.021165+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:56:44.021165+0000","last_peered":"2024-09-10T18:56:44.021165+0000","last_clean":"2024-09-10T18:56:44.021165+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:56:44.021165+0000","last_undegraded":"2024-09-10T18:56:44.021165+0000","last_fullsized":"2024-09-10T18:56:44.021165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.405645+0000","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057879+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:56:44.057879+0000","last_peered":"2024-09-10T18:56:44.057879+0000","last_clean":"2024-09-10T18:56:44.057879+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:56:44.057879+0000","last_undegraded":"2024-09-10T18:56:44.057879+0000","last_fullsized":"2024-09-10T18:56:44.057879+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.060992+0000","last_change":"2024-09-10T18:55:19.296815+0000","last_active":"2024-09-10T18:56:44.060992+0000","last_peered":"2024-09-10T18:56:44.060992+0000","last_clean":"2024-09-10T18:56:44.060992+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:56:44.060992+0000","last_undegraded":"2024-09-10T18:56:44.060992+0000","last_fullsized":"2024-09-10T18:56:44.060992+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:19.296674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:55:19.296674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003182767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057930+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:56:44.057930+0000","last_peered":"2024-09-10T18:56:44.057930+0000","last_clean":"2024-09-10T18:56:44.057930+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:56:44.057930+0000","last_undegraded":"2024-09-10T18:56:44.057930+0000","last_fullsized":"2024-09-10T18:56:44.057930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.407178+0000","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029782+0000","last_change":"2024-09-10T18:54:37.530238+0000","last_active":"2024-09-10T18:56:44.029782+0000","last_peered":"2024-09-10T18:56:44.029782+0000","last_clean":"2024-09-10T18:56:44.029782+0000","last_became_active":"2024-09-10T18:54:37.529892+0000","last_became_peered":"2024-09-10T18:54:37.529892+0000","last_unstale":"2024-09-10T18:56:44.029782+0000","last_undegraded":"2024-09-10T18:56:44.029782+0000","last_fullsized":"2024-09-10T18:56:44.029782+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":236,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047740+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:56:44.047740+0000","last_peered":"2024-09-10T18:56:44.047740+0000","last_clean":"2024-09-10T18:56:44.047740+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:56:44.047740+0000","last_undegraded":"2024-09-10T18:56:44.047740+0000","last_fullsized":"2024-09-10T18:56:44.047740+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.212086+0000","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.026738+0000","last_change":"2024-09-10T18:55:03.233057+0000","last_active":"2024-09-10T18:56:44.026738+0000","last_peered":"2024-09-10T18:56:44.026738+0000","last_clean":"2024-09-10T18:56:44.026738+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:56:44.026738+0000","last_undegraded":"2024-09-10T18:56:44.026738+0000","last_fullsized":"2024-09-10T18:56:44.026738+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:03.232923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.232923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036625059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":273,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.021336+0000","last_change":"2024-09-10T18:56:04.556751+0000","last_active":"2024-09-10T18:56:44.021336+0000","last_peered":"2024-09-10T18:56:44.021336+0000","last_clean":"2024-09-10T18:56:44.021336+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:56:44.021336+0000","last_undegraded":"2024-09-10T18:56:44.021336+0000","last_fullsized":"2024-09-10T18:56:44.021336+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:56:04.556586+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.556586+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.342255+0000","scrub_duration":0.0046952210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.031708+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:56:44.031708+0000","last_peered":"2024-09-10T18:56:44.031708+0000","last_clean":"2024-09-10T18:56:44.031708+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:56:44.031708+0000","last_undegraded":"2024-09-10T18:56:44.031708+0000","last_fullsized":"2024-09-10T18:56:44.031708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024403+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:56:44.024403+0000","last_peered":"2024-09-10T18:56:44.024403+0000","last_clean":"2024-09-10T18:56:44.024403+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:56:44.024403+0000","last_undegraded":"2024-09-10T18:56:44.024403+0000","last_fullsized":"2024-09-10T18:56:44.024403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:13.987563+0000","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024083+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:56:44.024083+0000","last_peered":"2024-09-10T18:56:44.024083+0000","last_clean":"2024-09-10T18:56:44.024083+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:56:44.024083+0000","last_undegraded":"2024-09-10T18:56:44.024083+0000","last_fullsized":"2024-09-10T18:56:44.024083+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.745484+0000","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.11","version":"126'7","reported_seq":150,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023634+0000","last_change":"2024-09-10T18:56:21.349180+0000","last_active":"2024-09-10T18:56:44.023634+0000","last_peered":"2024-09-10T18:56:44.023634+0000","last_clean":"2024-09-10T18:56:44.023634+0000","last_became_active":"2024-09-10T18:54:37.602669+0000","last_became_peered":"2024-09-10T18:54:37.602669+0000","last_unstale":"2024-09-10T18:56:44.023634+0000","last_undegraded":"2024-09-10T18:56:44.023634+0000","last_fullsized":"2024-09-10T18:56:44.023634+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:21.348988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.348988+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.217619+0000","scrub_duration":0.0044214550000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":251,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024178+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:56:44.024178+0000","last_peered":"2024-09-10T18:56:44.024178+0000","last_clean":"2024-09-10T18:56:44.024178+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:56:44.024178+0000","last_undegraded":"2024-09-10T18:56:44.024178+0000","last_fullsized":"2024-09-10T18:56:44.024178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.212518+0000","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045053+0000","last_change":"2024-09-10T18:54:58.446820+0000","last_active":"2024-09-10T18:56:44.045053+0000","last_peered":"2024-09-10T18:56:44.045053+0000","last_clean":"2024-09-10T18:56:44.045053+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:56:44.045053+0000","last_undegraded":"2024-09-10T18:56:44.045053+0000","last_fullsized":"2024-09-10T18:56:44.045053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.446707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.446707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011714456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029569+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:56:44.029569+0000","last_peered":"2024-09-10T18:56:44.029569+0000","last_clean":"2024-09-10T18:56:44.029569+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:56:44.029569+0000","last_undegraded":"2024-09-10T18:56:44.029569+0000","last_fullsized":"2024-09-10T18:56:44.029569+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022673+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:56:44.022673+0000","last_peered":"2024-09-10T18:56:44.022673+0000","last_clean":"2024-09-10T18:56:44.022673+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:56:44.022673+0000","last_undegraded":"2024-09-10T18:56:44.022673+0000","last_fullsized":"2024-09-10T18:56:44.022673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.916290+0000","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015877+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:56:44.015877+0000","last_peered":"2024-09-10T18:56:44.015877+0000","last_clean":"2024-09-10T18:56:44.015877+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:56:44.015877+0000","last_undegraded":"2024-09-10T18:56:44.015877+0000","last_fullsized":"2024-09-10T18:56:44.015877+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.438790+0000","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.575741+0000","last_change":"2024-09-10T18:56:44.575541+0000","last_active":"2024-09-10T18:56:44.575741+0000","last_peered":"2024-09-10T18:56:44.575741+0000","last_clean":"2024-09-10T18:56:44.575741+0000","last_became_active":"2024-09-10T18:54:37.525432+0000","last_became_peered":"2024-09-10T18:54:37.525432+0000","last_unstale":"2024-09-10T18:56:44.575741+0000","last_undegraded":"2024-09-10T18:56:44.575741+0000","last_fullsized":"2024-09-10T18:56:44.575741+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:44.575356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.575356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:02.690486+0000","scrub_duration":0.0034995669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":244,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022883+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:56:44.022883+0000","last_peered":"2024-09-10T18:56:44.022883+0000","last_clean":"2024-09-10T18:56:44.022883+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:56:44.022883+0000","last_undegraded":"2024-09-10T18:56:44.022883+0000","last_fullsized":"2024-09-10T18:56:44.022883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.198556+0000","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061763+0000","last_change":"2024-09-10T18:56:30.684729+0000","last_active":"2024-09-10T18:56:44.061763+0000","last_peered":"2024-09-10T18:56:44.061763+0000","last_clean":"2024-09-10T18:56:44.061763+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:56:44.061763+0000","last_undegraded":"2024-09-10T18:56:44.061763+0000","last_fullsized":"2024-09-10T18:56:44.061763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.684589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.093891+0000","scrub_duration":0.0023375330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":213,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.055619+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:56:44.055619+0000","last_peered":"2024-09-10T18:56:44.055619+0000","last_clean":"2024-09-10T18:56:44.055619+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:56:44.055619+0000","last_undegraded":"2024-09-10T18:56:44.055619+0000","last_fullsized":"2024-09-10T18:56:44.055619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":211,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043159+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:56:44.043159+0000","last_peered":"2024-09-10T18:56:44.043159+0000","last_clean":"2024-09-10T18:56:44.043159+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:56:44.043159+0000","last_undegraded":"2024-09-10T18:56:44.043159+0000","last_fullsized":"2024-09-10T18:56:44.043159+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.055701+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:56:44.055701+0000","last_peered":"2024-09-10T18:56:44.055701+0000","last_clean":"2024-09-10T18:56:44.055701+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:56:44.055701+0000","last_undegraded":"2024-09-10T18:56:44.055701+0000","last_fullsized":"2024-09-10T18:56:44.055701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.383629+0000","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"237'6","reported_seq":149,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.026231+0000","last_change":"2024-09-10T18:56:33.227360+0000","last_active":"2024-09-10T18:56:44.026231+0000","last_peered":"2024-09-10T18:56:44.026231+0000","last_clean":"2024-09-10T18:56:44.026231+0000","last_became_active":"2024-09-10T18:54:37.698688+0000","last_became_peered":"2024-09-10T18:54:37.698688+0000","last_unstale":"2024-09-10T18:56:44.026231+0000","last_undegraded":"2024-09-10T18:56:44.026231+0000","last_fullsized":"2024-09-10T18:56:44.026231+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'6","last_scrub_stamp":"2024-09-10T18:56:33.227233+0000","last_deep_scrub":"237'6","last_deep_scrub_stamp":"2024-09-10T18:56:33.227233+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.227233+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.165107+0000","scrub_duration":0.0062020080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":255,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022540+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:56:44.022540+0000","last_peered":"2024-09-10T18:56:44.022540+0000","last_clean":"2024-09-10T18:56:44.022540+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:56:44.022540+0000","last_undegraded":"2024-09-10T18:56:44.022540+0000","last_fullsized":"2024-09-10T18:56:44.022540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.067234+0000","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":248,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.050635+0000","last_change":"2024-09-10T18:56:36.828365+0000","last_active":"2024-09-10T18:56:44.050635+0000","last_peered":"2024-09-10T18:56:44.050635+0000","last_clean":"2024-09-10T18:56:44.050635+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:56:44.050635+0000","last_undegraded":"2024-09-10T18:56:44.050635+0000","last_fullsized":"2024-09-10T18:56:44.050635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:56:36.828207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.828207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.829596+0000","scrub_duration":0.0038610699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.021943+0000","last_change":"2024-09-10T18:55:00.199821+0000","last_active":"2024-09-10T18:56:44.021943+0000","last_peered":"2024-09-10T18:56:44.021943+0000","last_clean":"2024-09-10T18:56:44.021943+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:56:44.021943+0000","last_undegraded":"2024-09-10T18:56:44.021943+0000","last_fullsized":"2024-09-10T18:56:44.021943+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.199571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.199571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036539939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029645+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:56:44.029645+0000","last_peered":"2024-09-10T18:56:44.029645+0000","last_clean":"2024-09-10T18:56:44.029645+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:56:44.029645+0000","last_undegraded":"2024-09-10T18:56:44.029645+0000","last_fullsized":"2024-09-10T18:56:44.029645+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015204+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:56:44.015204+0000","last_peered":"2024-09-10T18:56:44.015204+0000","last_clean":"2024-09-10T18:56:44.015204+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:56:44.015204+0000","last_undegraded":"2024-09-10T18:56:44.015204+0000","last_fullsized":"2024-09-10T18:56:44.015204+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:06.585634+0000","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045147+0000","last_change":"2024-09-10T18:56:14.608089+0000","last_active":"2024-09-10T18:56:44.045147+0000","last_peered":"2024-09-10T18:56:44.045147+0000","last_clean":"2024-09-10T18:56:44.045147+0000","last_became_active":"2024-09-10T18:54:37.592760+0000","last_became_peered":"2024-09-10T18:54:37.592760+0000","last_unstale":"2024-09-10T18:56:44.045147+0000","last_undegraded":"2024-09-10T18:56:44.045147+0000","last_fullsized":"2024-09-10T18:56:44.045147+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.607936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.607936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.345720+0000","scrub_duration":0.0032677309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":242,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.027138+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:56:44.027138+0000","last_peered":"2024-09-10T18:56:44.027138+0000","last_clean":"2024-09-10T18:56:44.027138+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:56:44.027138+0000","last_undegraded":"2024-09-10T18:56:44.027138+0000","last_fullsized":"2024-09-10T18:56:44.027138+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.960254+0000","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":277,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.020167+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:56:44.020167+0000","last_peered":"2024-09-10T18:56:44.020167+0000","last_clean":"2024-09-10T18:56:44.020167+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:56:44.020167+0000","last_undegraded":"2024-09-10T18:56:44.020167+0000","last_fullsized":"2024-09-10T18:56:44.020167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.123931+0000","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":323,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.016213+0000","last_change":"2024-09-10T18:55:04.228572+0000","last_active":"2024-09-10T18:56:44.016213+0000","last_peered":"2024-09-10T18:56:44.016213+0000","last_clean":"2024-09-10T18:56:44.016213+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:56:44.016213+0000","last_undegraded":"2024-09-10T18:56:44.016213+0000","last_fullsized":"2024-09-10T18:56:44.016213+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:04.228360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.228360+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039590529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":212,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047866+0000","last_change":"2024-09-10T18:55:01.757908+0000","last_active":"2024-09-10T18:56:44.047866+0000","last_peered":"2024-09-10T18:56:44.047866+0000","last_clean":"2024-09-10T18:56:44.047866+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:56:44.047866+0000","last_undegraded":"2024-09-10T18:56:44.047866+0000","last_fullsized":"2024-09-10T18:56:44.047866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.757807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.757807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011468136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":156,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.030327+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:56:44.030327+0000","last_peered":"2024-09-10T18:56:44.030327+0000","last_clean":"2024-09-10T18:56:44.030327+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:56:44.030327+0000","last_undegraded":"2024-09-10T18:56:44.030327+0000","last_fullsized":"2024-09-10T18:56:44.030327+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.921825+0000","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1d","version":"126'7","reported_seq":149,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045883+0000","last_change":"2024-09-10T18:56:04.762019+0000","last_active":"2024-09-10T18:56:44.045883+0000","last_peered":"2024-09-10T18:56:44.045883+0000","last_clean":"2024-09-10T18:56:44.045883+0000","last_became_active":"2024-09-10T18:54:37.592557+0000","last_became_peered":"2024-09-10T18:54:37.592557+0000","last_unstale":"2024-09-10T18:56:44.045883+0000","last_undegraded":"2024-09-10T18:56:44.045883+0000","last_fullsized":"2024-09-10T18:56:44.045883+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:04.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.761871+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.599357+0000","scrub_duration":0.003828508,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":242,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022151+0000","last_change":"2024-09-10T18:55:05.902412+0000","last_active":"2024-09-10T18:56:44.022151+0000","last_peered":"2024-09-10T18:56:44.022151+0000","last_clean":"2024-09-10T18:56:44.022151+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:56:44.022151+0000","last_undegraded":"2024-09-10T18:56:44.022151+0000","last_fullsized":"2024-09-10T18:56:44.022151+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:05.902154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.902154+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047960349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.054428+0000","last_change":"2024-09-10T18:54:49.456319+0000","last_active":"2024-09-10T18:56:44.054428+0000","last_peered":"2024-09-10T18:56:44.054428+0000","last_clean":"2024-09-10T18:56:44.054428+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:56:44.054428+0000","last_undegraded":"2024-09-10T18:56:44.054428+0000","last_fullsized":"2024-09-10T18:56:44.054428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.456167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.456167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002905819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057792+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:56:44.057792+0000","last_peered":"2024-09-10T18:56:44.057792+0000","last_clean":"2024-09-10T18:56:44.057792+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:56:44.057792+0000","last_undegraded":"2024-09-10T18:56:44.057792+0000","last_fullsized":"2024-09-10T18:56:44.057792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.259079+0000","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044474+0000","last_change":"2024-09-10T18:55:02.390515+0000","last_active":"2024-09-10T18:56:44.044474+0000","last_peered":"2024-09-10T18:56:44.044474+0000","last_clean":"2024-09-10T18:56:44.044474+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:56:44.044474+0000","last_undegraded":"2024-09-10T18:56:44.044474+0000","last_fullsized":"2024-09-10T18:56:44.044474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.390417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.390417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002554265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.020930+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:56:44.020930+0000","last_peered":"2024-09-10T18:56:44.020930+0000","last_clean":"2024-09-10T18:56:44.020930+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:56:44.020930+0000","last_undegraded":"2024-09-10T18:56:44.020930+0000","last_fullsized":"2024-09-10T18:56:44.020930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.580079+0000","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.015647+0000","last_change":"2024-09-10T18:56:32.796094+0000","last_active":"2024-09-10T18:56:44.015647+0000","last_peered":"2024-09-10T18:56:44.015647+0000","last_clean":"2024-09-10T18:56:44.015647+0000","last_became_active":"2024-09-10T18:54:37.566860+0000","last_became_peered":"2024-09-10T18:54:37.566860+0000","last_unstale":"2024-09-10T18:56:44.015647+0000","last_undegraded":"2024-09-10T18:56:44.015647+0000","last_fullsized":"2024-09-10T18:56:44.015647+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.795927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.610135+0000","scrub_duration":0.0032690250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":232,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.028432+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:56:44.028432+0000","last_peered":"2024-09-10T18:56:44.028432+0000","last_clean":"2024-09-10T18:56:44.028432+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:56:44.028432+0000","last_undegraded":"2024-09-10T18:56:44.028432+0000","last_fullsized":"2024-09-10T18:56:44.028432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.030796+0000","last_change":"2024-09-10T18:54:49.637132+0000","last_active":"2024-09-10T18:56:44.030796+0000","last_peered":"2024-09-10T18:56:44.030796+0000","last_clean":"2024-09-10T18:56:44.030796+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:56:44.030796+0000","last_undegraded":"2024-09-10T18:56:44.030796+0000","last_fullsized":"2024-09-10T18:56:44.030796+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.637024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.637024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029426489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":214,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048196+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:56:44.048196+0000","last_peered":"2024-09-10T18:56:44.048196+0000","last_clean":"2024-09-10T18:56:44.048196+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:56:44.048196+0000","last_undegraded":"2024-09-10T18:56:44.048196+0000","last_fullsized":"2024-09-10T18:56:44.048196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023677+0000","last_change":"2024-09-10T18:56:03.595788+0000","last_active":"2024-09-10T18:56:44.023677+0000","last_peered":"2024-09-10T18:56:44.023677+0000","last_clean":"2024-09-10T18:56:44.023677+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:56:44.023677+0000","last_undegraded":"2024-09-10T18:56:44.023677+0000","last_fullsized":"2024-09-10T18:56:44.023677+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.595643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.595643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.278652+0000","scrub_duration":0.0030518839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033356+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:56:44.033356+0000","last_peered":"2024-09-10T18:56:44.033356+0000","last_clean":"2024-09-10T18:56:44.033356+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:56:44.033356+0000","last_undegraded":"2024-09-10T18:56:44.033356+0000","last_fullsized":"2024-09-10T18:56:44.033356+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.043389+0000","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"126'5","reported_seq":148,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.026447+0000","last_change":"2024-09-10T18:56:37.282686+0000","last_active":"2024-09-10T18:56:44.026447+0000","last_peered":"2024-09-10T18:56:44.026447+0000","last_clean":"2024-09-10T18:56:44.026447+0000","last_became_active":"2024-09-10T18:54:37.507583+0000","last_became_peered":"2024-09-10T18:54:37.507583+0000","last_unstale":"2024-09-10T18:56:44.026447+0000","last_undegraded":"2024-09-10T18:56:44.026447+0000","last_fullsized":"2024-09-10T18:56:44.026447+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:37.282578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.282578+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.873328+0000","scrub_duration":0.0030887559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":247,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022968+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:56:44.022968+0000","last_peered":"2024-09-10T18:56:44.022968+0000","last_clean":"2024-09-10T18:56:44.022968+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:56:44.022968+0000","last_undegraded":"2024-09-10T18:56:44.022968+0000","last_fullsized":"2024-09-10T18:56:44.022968+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.936795+0000","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029939+0000","last_change":"2024-09-10T18:56:04.366386+0000","last_active":"2024-09-10T18:56:44.029939+0000","last_peered":"2024-09-10T18:56:44.029939+0000","last_clean":"2024-09-10T18:56:44.029939+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:56:44.029939+0000","last_undegraded":"2024-09-10T18:56:44.029939+0000","last_fullsized":"2024-09-10T18:56:44.029939+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:34.552199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:34.552199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032492139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":316,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049940+0000","last_change":"2024-09-10T18:54:40.522553+0000","last_active":"2024-09-10T18:56:44.049940+0000","last_peered":"2024-09-10T18:56:44.049940+0000","last_clean":"2024-09-10T18:56:44.049940+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:56:44.049940+0000","last_undegraded":"2024-09-10T18:56:44.049940+0000","last_fullsized":"2024-09-10T18:56:44.049940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:40.522460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.522460+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027852699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":221,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057587+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:56:44.057587+0000","last_peered":"2024-09-10T18:56:44.057587+0000","last_clean":"2024-09-10T18:56:44.057587+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:56:44.057587+0000","last_undegraded":"2024-09-10T18:56:44.057587+0000","last_fullsized":"2024-09-10T18:56:44.057587+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.837926+0000","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057019+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:56:44.057019+0000","last_peered":"2024-09-10T18:56:44.057019+0000","last_clean":"2024-09-10T18:56:44.057019+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:56:44.057019+0000","last_undegraded":"2024-09-10T18:56:44.057019+0000","last_fullsized":"2024-09-10T18:56:44.057019+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.533486+0000","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"126'5","reported_seq":147,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.053439+0000","last_change":"2024-09-10T18:56:27.205415+0000","last_active":"2024-09-10T18:56:44.053439+0000","last_peered":"2024-09-10T18:56:44.053439+0000","last_clean":"2024-09-10T18:56:44.053439+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:56:44.053439+0000","last_undegraded":"2024-09-10T18:56:44.053439+0000","last_fullsized":"2024-09-10T18:56:44.053439+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.205258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.205258+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.814923+0000","scrub_duration":0.0049140169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":241,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.053141+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:56:44.053141+0000","last_peered":"2024-09-10T18:56:44.053141+0000","last_clean":"2024-09-10T18:56:44.053141+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:56:44.053141+0000","last_undegraded":"2024-09-10T18:56:44.053141+0000","last_fullsized":"2024-09-10T18:56:44.053141+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.489013+0000","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":227,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056575+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:56:44.056575+0000","last_peered":"2024-09-10T18:56:44.056575+0000","last_clean":"2024-09-10T18:56:44.056575+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:56:44.056575+0000","last_undegraded":"2024-09-10T18:56:44.056575+0000","last_fullsized":"2024-09-10T18:56:44.056575+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":311,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025196+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:56:44.025196+0000","last_peered":"2024-09-10T18:56:44.025196+0000","last_clean":"2024-09-10T18:56:44.025196+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:56:44.025196+0000","last_undegraded":"2024-09-10T18:56:44.025196+0000","last_fullsized":"2024-09-10T18:56:44.025196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.019882+0000","last_change":"2024-09-10T18:54:53.310725+0000","last_active":"2024-09-10T18:56:44.019882+0000","last_peered":"2024-09-10T18:56:44.019882+0000","last_clean":"2024-09-10T18:56:44.019882+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:56:44.019882+0000","last_undegraded":"2024-09-10T18:56:44.019882+0000","last_fullsized":"2024-09-10T18:56:44.019882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.310489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.310489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026931849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":132,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059071+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:56:44.059071+0000","last_peered":"2024-09-10T18:56:44.059071+0000","last_clean":"2024-09-10T18:56:44.059071+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:56:44.059071+0000","last_undegraded":"2024-09-10T18:56:44.059071+0000","last_fullsized":"2024-09-10T18:56:44.059071+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"126'5","reported_seq":147,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048974+0000","last_change":"2024-09-10T18:56:39.805638+0000","last_active":"2024-09-10T18:56:44.048974+0000","last_peered":"2024-09-10T18:56:44.048974+0000","last_clean":"2024-09-10T18:56:44.048974+0000","last_became_active":"2024-09-10T18:54:37.610866+0000","last_became_peered":"2024-09-10T18:54:37.610866+0000","last_unstale":"2024-09-10T18:56:44.048974+0000","last_undegraded":"2024-09-10T18:56:44.048974+0000","last_fullsized":"2024-09-10T18:56:44.048974+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:39.805468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.805468+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.334135+0000","scrub_duration":0.0040878859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":239,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023952+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:56:44.023952+0000","last_peered":"2024-09-10T18:56:44.023952+0000","last_clean":"2024-09-10T18:56:44.023952+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:56:44.023952+0000","last_undegraded":"2024-09-10T18:56:44.023952+0000","last_fullsized":"2024-09-10T18:56:44.023952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.661931+0000","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.028442+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:56:44.028442+0000","last_peered":"2024-09-10T18:56:44.028442+0000","last_clean":"2024-09-10T18:56:44.028442+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:56:44.028442+0000","last_undegraded":"2024-09-10T18:56:44.028442+0000","last_fullsized":"2024-09-10T18:56:44.028442+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":213,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029499+0000","last_change":"2024-09-10T18:55:10.179358+0000","last_active":"2024-09-10T18:56:44.029499+0000","last_peered":"2024-09-10T18:56:44.029499+0000","last_clean":"2024-09-10T18:56:44.029499+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:56:44.029499+0000","last_undegraded":"2024-09-10T18:56:44.029499+0000","last_fullsized":"2024-09-10T18:56:44.029499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003833071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.042645+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:56:44.042645+0000","last_peered":"2024-09-10T18:56:44.042645+0000","last_clean":"2024-09-10T18:56:44.042645+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:56:44.042645+0000","last_undegraded":"2024-09-10T18:56:44.042645+0000","last_fullsized":"2024-09-10T18:56:44.042645+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.752093+0000","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017192+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:56:44.017192+0000","last_peered":"2024-09-10T18:56:44.017192+0000","last_clean":"2024-09-10T18:56:44.017192+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:56:44.017192+0000","last_undegraded":"2024-09-10T18:56:44.017192+0000","last_fullsized":"2024-09-10T18:56:44.017192+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.510275+0000","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"126'9","reported_seq":152,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.032142+0000","last_change":"2024-09-10T18:56:09.704635+0000","last_active":"2024-09-10T18:56:44.032142+0000","last_peered":"2024-09-10T18:56:44.032142+0000","last_clean":"2024-09-10T18:56:44.032142+0000","last_became_active":"2024-09-10T18:54:37.646743+0000","last_became_peered":"2024-09-10T18:54:37.646743+0000","last_unstale":"2024-09-10T18:56:44.032142+0000","last_undegraded":"2024-09-10T18:56:44.032142+0000","last_fullsized":"2024-09-10T18:56:44.032142+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.575925+0000","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":246,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.019486+0000","last_change":"2024-09-10T18:55:06.199960+0000","last_active":"2024-09-10T18:56:44.019486+0000","last_peered":"2024-09-10T18:56:44.019486+0000","last_clean":"2024-09-10T18:56:44.019486+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:56:44.019486+0000","last_undegraded":"2024-09-10T18:56:44.019486+0000","last_fullsized":"2024-09-10T18:56:44.019486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:55:06.199853+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:55:06.199853+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045914980000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":236,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061697+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:56:44.061697+0000","last_peered":"2024-09-10T18:56:44.061697+0000","last_clean":"2024-09-10T18:56:44.061697+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:56:44.061697+0000","last_undegraded":"2024-09-10T18:56:44.061697+0000","last_fullsized":"2024-09-10T18:56:44.061697+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.423650+0000","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":224,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017295+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:56:44.017295+0000","last_peered":"2024-09-10T18:56:44.017295+0000","last_clean":"2024-09-10T18:56:44.017295+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:56:44.017295+0000","last_undegraded":"2024-09-10T18:56:44.017295+0000","last_fullsized":"2024-09-10T18:56:44.017295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.667997+0000","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045616+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:56:44.045616+0000","last_peered":"2024-09-10T18:56:44.045616+0000","last_clean":"2024-09-10T18:56:44.045616+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:56:44.045616+0000","last_undegraded":"2024-09-10T18:56:44.045616+0000","last_fullsized":"2024-09-10T18:56:44.045616+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.361501+0000","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017154+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:56:44.017154+0000","last_peered":"2024-09-10T18:56:44.017154+0000","last_clean":"2024-09-10T18:56:44.017154+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:56:44.017154+0000","last_undegraded":"2024-09-10T18:56:44.017154+0000","last_fullsized":"2024-09-10T18:56:44.017154+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.443415+0000","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048796+0000","last_change":"2024-09-10T18:54:37.725751+0000","last_active":"2024-09-10T18:56:44.048796+0000","last_peered":"2024-09-10T18:56:44.048796+0000","last_clean":"2024-09-10T18:56:44.048796+0000","last_became_active":"2024-09-10T18:54:37.716564+0000","last_became_peered":"2024-09-10T18:54:37.716564+0000","last_unstale":"2024-09-10T18:56:44.048796+0000","last_undegraded":"2024-09-10T18:56:44.048796+0000","last_fullsized":"2024-09-10T18:56:44.048796+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":227,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.059398+0000","last_change":"2024-09-10T18:54:54.957426+0000","last_active":"2024-09-10T18:56:44.059398+0000","last_peered":"2024-09-10T18:56:44.059398+0000","last_clean":"2024-09-10T18:56:44.059398+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:56:44.059398+0000","last_undegraded":"2024-09-10T18:56:44.059398+0000","last_fullsized":"2024-09-10T18:56:44.059398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:54:54.957316+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:54:54.957316+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039293499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029157+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:56:44.029157+0000","last_peered":"2024-09-10T18:56:44.029157+0000","last_clean":"2024-09-10T18:56:44.029157+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:56:44.029157+0000","last_undegraded":"2024-09-10T18:56:44.029157+0000","last_fullsized":"2024-09-10T18:56:44.029157+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":214,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043537+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:56:44.043537+0000","last_peered":"2024-09-10T18:56:44.043537+0000","last_clean":"2024-09-10T18:56:44.043537+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:56:44.043537+0000","last_undegraded":"2024-09-10T18:56:44.043537+0000","last_fullsized":"2024-09-10T18:56:44.043537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.718506+0000","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":219,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025204+0000","last_change":"2024-09-10T18:55:02.948472+0000","last_active":"2024-09-10T18:56:44.025204+0000","last_peered":"2024-09-10T18:56:44.025204+0000","last_clean":"2024-09-10T18:56:44.025204+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:56:44.025204+0000","last_undegraded":"2024-09-10T18:56:44.025204+0000","last_fullsized":"2024-09-10T18:56:44.025204+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.948218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.948218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034990780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.046640+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:56:44.046640+0000","last_peered":"2024-09-10T18:56:44.046640+0000","last_clean":"2024-09-10T18:56:44.046640+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:56:44.046640+0000","last_undegraded":"2024-09-10T18:56:44.046640+0000","last_fullsized":"2024-09-10T18:56:44.046640+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.479269+0000","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":133,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.050277+0000","last_change":"2024-09-10T18:54:37.661524+0000","last_active":"2024-09-10T18:56:44.050277+0000","last_peered":"2024-09-10T18:56:44.050277+0000","last_clean":"2024-09-10T18:56:44.050277+0000","last_became_active":"2024-09-10T18:54:37.657896+0000","last_became_peered":"2024-09-10T18:54:37.657896+0000","last_unstale":"2024-09-10T18:56:44.050277+0000","last_undegraded":"2024-09-10T18:56:44.050277+0000","last_fullsized":"2024-09-10T18:56:44.050277+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":240,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.028979+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:56:44.028979+0000","last_peered":"2024-09-10T18:56:44.028979+0000","last_clean":"2024-09-10T18:56:44.028979+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:56:44.028979+0000","last_undegraded":"2024-09-10T18:56:44.028979+0000","last_fullsized":"2024-09-10T18:56:44.028979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.059487+0000","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033742+0000","last_change":"2024-09-10T18:55:08.609833+0000","last_active":"2024-09-10T18:56:44.033742+0000","last_peered":"2024-09-10T18:56:44.033742+0000","last_clean":"2024-09-10T18:56:44.033742+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:56:44.033742+0000","last_undegraded":"2024-09-10T18:56:44.033742+0000","last_fullsized":"2024-09-10T18:56:44.033742+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.609702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.609702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":214,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.045951+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:56:44.045951+0000","last_peered":"2024-09-10T18:56:44.045951+0000","last_clean":"2024-09-10T18:56:44.045951+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:56:44.045951+0000","last_undegraded":"2024-09-10T18:56:44.045951+0000","last_fullsized":"2024-09-10T18:56:44.045951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.486395+0000","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048018+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:56:44.048018+0000","last_peered":"2024-09-10T18:56:44.048018+0000","last_clean":"2024-09-10T18:56:44.048018+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:56:44.048018+0000","last_undegraded":"2024-09-10T18:56:44.048018+0000","last_fullsized":"2024-09-10T18:56:44.048018+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.320880+0000","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":132,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.043540+0000","last_change":"2024-09-10T18:54:37.573584+0000","last_active":"2024-09-10T18:56:44.043540+0000","last_peered":"2024-09-10T18:56:44.043540+0000","last_clean":"2024-09-10T18:56:44.043540+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:56:44.043540+0000","last_undegraded":"2024-09-10T18:56:44.043540+0000","last_fullsized":"2024-09-10T18:56:44.043540+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.058760+0000","last_change":"2024-09-10T18:56:40.177375+0000","last_active":"2024-09-10T18:56:44.058760+0000","last_peered":"2024-09-10T18:56:44.058760+0000","last_clean":"2024-09-10T18:56:44.058760+0000","last_became_active":"2024-09-10T18:54:37.565268+0000","last_became_peered":"2024-09-10T18:54:37.565268+0000","last_unstale":"2024-09-10T18:56:44.058760+0000","last_undegraded":"2024-09-10T18:56:44.058760+0000","last_fullsized":"2024-09-10T18:56:44.058760+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.177249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.177249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.578420+0000","scrub_duration":0.0033094600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.024315+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:56:44.024315+0000","last_peered":"2024-09-10T18:56:44.024315+0000","last_clean":"2024-09-10T18:56:44.024315+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:56:44.024315+0000","last_undegraded":"2024-09-10T18:56:44.024315+0000","last_fullsized":"2024-09-10T18:56:44.024315+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.123736+0000","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":259,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.598362+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:56:44.598362+0000","last_peered":"2024-09-10T18:56:44.598362+0000","last_clean":"2024-09-10T18:56:44.598362+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:56:44.598362+0000","last_undegraded":"2024-09-10T18:56:44.598362+0000","last_fullsized":"2024-09-10T18:56:44.598362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.905881+0000","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":226,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.022229+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:56:44.022229+0000","last_peered":"2024-09-10T18:56:44.022229+0000","last_clean":"2024-09-10T18:56:44.022229+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:56:44.022229+0000","last_undegraded":"2024-09-10T18:56:44.022229+0000","last_fullsized":"2024-09-10T18:56:44.022229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.057718+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:56:44.057718+0000","last_peered":"2024-09-10T18:56:44.057718+0000","last_clean":"2024-09-10T18:56:44.057718+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:56:44.057718+0000","last_undegraded":"2024-09-10T18:56:44.057718+0000","last_fullsized":"2024-09-10T18:56:44.057718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.517080+0000","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.032345+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:56:44.032345+0000","last_peered":"2024-09-10T18:56:44.032345+0000","last_clean":"2024-09-10T18:56:44.032345+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:56:44.032345+0000","last_undegraded":"2024-09-10T18:56:44.032345+0000","last_fullsized":"2024-09-10T18:56:44.032345+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.150161+0000","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"126'10","reported_seq":152,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044337+0000","last_change":"2024-09-10T18:56:28.793230+0000","last_active":"2024-09-10T18:56:44.044337+0000","last_peered":"2024-09-10T18:56:44.044337+0000","last_clean":"2024-09-10T18:56:44.044337+0000","last_became_active":"2024-09-10T18:54:37.562584+0000","last_became_peered":"2024-09-10T18:54:37.562584+0000","last_unstale":"2024-09-10T18:56:44.044337+0000","last_undegraded":"2024-09-10T18:56:44.044337+0000","last_fullsized":"2024-09-10T18:56:44.044337+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:28.793065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.793065+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.828376+0000","scrub_duration":0.0039084699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":220,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.061333+0000","last_change":"2024-09-10T18:56:09.619685+0000","last_active":"2024-09-10T18:56:44.061333+0000","last_peered":"2024-09-10T18:56:44.061333+0000","last_clean":"2024-09-10T18:56:44.061333+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:56:44.061333+0000","last_undegraded":"2024-09-10T18:56:44.061333+0000","last_fullsized":"2024-09-10T18:56:44.061333+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.619567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.619567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.792631+0000","scrub_duration":0.0049369640000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":232,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044037+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:56:44.044037+0000","last_peered":"2024-09-10T18:56:44.044037+0000","last_clean":"2024-09-10T18:56:44.044037+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:56:44.044037+0000","last_undegraded":"2024-09-10T18:56:44.044037+0000","last_fullsized":"2024-09-10T18:56:44.044037+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":218,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.046698+0000","last_change":"2024-09-10T18:54:57.678574+0000","last_active":"2024-09-10T18:56:44.046698+0000","last_peered":"2024-09-10T18:56:44.046698+0000","last_clean":"2024-09-10T18:56:44.046698+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:56:44.046698+0000","last_undegraded":"2024-09-10T18:56:44.046698+0000","last_fullsized":"2024-09-10T18:56:44.046698+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.678482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.678482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024273350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":338,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033665+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:56:44.033665+0000","last_peered":"2024-09-10T18:56:44.033665+0000","last_clean":"2024-09-10T18:56:44.033665+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:56:44.033665+0000","last_undegraded":"2024-09-10T18:56:44.033665+0000","last_fullsized":"2024-09-10T18:56:44.033665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.033579+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:56:44.033579+0000","last_peered":"2024-09-10T18:56:44.033579+0000","last_clean":"2024-09-10T18:56:44.033579+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:56:44.033579+0000","last_undegraded":"2024-09-10T18:56:44.033579+0000","last_fullsized":"2024-09-10T18:56:44.033579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:38.976604+0000","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"126'2","reported_seq":144,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056755+0000","last_change":"2024-09-10T18:56:16.659103+0000","last_active":"2024-09-10T18:56:44.056755+0000","last_peered":"2024-09-10T18:56:44.056755+0000","last_clean":"2024-09-10T18:56:44.056755+0000","last_became_active":"2024-09-10T18:54:37.639260+0000","last_became_peered":"2024-09-10T18:54:37.639260+0000","last_unstale":"2024-09-10T18:56:44.056755+0000","last_undegraded":"2024-09-10T18:56:44.056755+0000","last_fullsized":"2024-09-10T18:56:44.056755+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.376270+0000","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.025649+0000","last_change":"2024-09-10T18:54:44.250455+0000","last_active":"2024-09-10T18:56:44.025649+0000","last_peered":"2024-09-10T18:56:44.025649+0000","last_clean":"2024-09-10T18:56:44.025649+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:56:44.025649+0000","last_undegraded":"2024-09-10T18:56:44.025649+0000","last_fullsized":"2024-09-10T18:56:44.025649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:44.250321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:54:44.250321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031387210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":235,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.029708+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:56:44.029708+0000","last_peered":"2024-09-10T18:56:44.029708+0000","last_clean":"2024-09-10T18:56:44.029708+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:56:44.029708+0000","last_undegraded":"2024-09-10T18:56:44.029708+0000","last_fullsized":"2024-09-10T18:56:44.029708+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":235,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.056199+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:56:44.056199+0000","last_peered":"2024-09-10T18:56:44.056199+0000","last_clean":"2024-09-10T18:56:44.056199+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:56:44.056199+0000","last_undegraded":"2024-09-10T18:56:44.056199+0000","last_fullsized":"2024-09-10T18:56:44.056199+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.550858+0000","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":232,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.044790+0000","last_change":"2024-09-10T18:56:26.777537+0000","last_active":"2024-09-10T18:56:44.044790+0000","last_peered":"2024-09-10T18:56:44.044790+0000","last_clean":"2024-09-10T18:56:44.044790+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:56:44.044790+0000","last_undegraded":"2024-09-10T18:56:44.044790+0000","last_fullsized":"2024-09-10T18:56:44.044790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.777271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.777271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.531064+0000","scrub_duration":0.0030242540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.023546+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:56:44.023546+0000","last_peered":"2024-09-10T18:56:44.023546+0000","last_clean":"2024-09-10T18:56:44.023546+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:56:44.023546+0000","last_undegraded":"2024-09-10T18:56:44.023546+0000","last_fullsized":"2024-09-10T18:56:44.023546+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.803420+0000","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":134,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.053530+0000","last_change":"2024-09-10T18:54:37.660686+0000","last_active":"2024-09-10T18:56:44.053530+0000","last_peered":"2024-09-10T18:56:44.053530+0000","last_clean":"2024-09-10T18:56:44.053530+0000","last_became_active":"2024-09-10T18:54:37.660393+0000","last_became_peered":"2024-09-10T18:54:37.660393+0000","last_unstale":"2024-09-10T18:56:44.053530+0000","last_undegraded":"2024-09-10T18:56:44.053530+0000","last_fullsized":"2024-09-10T18:56:44.053530+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":222,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047976+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:56:44.047976+0000","last_peered":"2024-09-10T18:56:44.047976+0000","last_clean":"2024-09-10T18:56:44.047976+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:56:44.047976+0000","last_undegraded":"2024-09-10T18:56:44.047976+0000","last_fullsized":"2024-09-10T18:56:44.047976+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.573240+0000","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":228,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.048545+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:56:44.048545+0000","last_peered":"2024-09-10T18:56:44.048545+0000","last_clean":"2024-09-10T18:56:44.048545+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:56:44.048545+0000","last_undegraded":"2024-09-10T18:56:44.048545+0000","last_fullsized":"2024-09-10T18:56:44.048545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.997928+0000","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":242,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.017360+0000","last_change":"2024-09-10T18:54:58.424151+0000","last_active":"2024-09-10T18:56:44.017360+0000","last_peered":"2024-09-10T18:56:44.017360+0000","last_clean":"2024-09-10T18:56:44.017360+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:56:44.017360+0000","last_undegraded":"2024-09-10T18:56:44.017360+0000","last_fullsized":"2024-09-10T18:56:44.017360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.424008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.424008+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.146668564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":214,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.047640+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:56:44.047640+0000","last_peered":"2024-09-10T18:56:44.047640+0000","last_clean":"2024-09-10T18:56:44.047640+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:56:44.047640+0000","last_undegraded":"2024-09-10T18:56:44.047640+0000","last_fullsized":"2024-09-10T18:56:44.047640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.684081+0000","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":143,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.031328+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:56:44.031328+0000","last_peered":"2024-09-10T18:56:44.031328+0000","last_clean":"2024-09-10T18:56:44.031328+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:56:44.031328+0000","last_undegraded":"2024-09-10T18:56:44.031328+0000","last_fullsized":"2024-09-10T18:56:44.031328+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.050482+0000","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":142,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-09-10T18:56:44.049648+0000","last_change":"2024-09-10T18:56:21.650671+0000","last_active":"2024-09-10T18:56:44.049648+0000","last_peered":"2024-09-10T18:56:44.049648+0000","last_clean":"2024-09-10T18:56:44.049648+0000","last_became_active":"2024-09-10T18:54:37.583272+0000","last_became_peered":"2024-09-10T18:54:37.583272+0000","last_unstale":"2024-09-10T18:56:44.049648+0000","last_undegraded":"2024-09-10T18:56:44.049648+0000","last_fullsized":"2024-09-10T18:56:44.049648+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.232910+0000","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":156,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":155,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":1415,"num_objects":134,"num_object_clones":24,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":518,"num_write_kb":516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1634304,"data_stored":4245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1686,"ondisk_log_size":1686,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":423,"num_read_kb":418,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972453,"num_pgs":126,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309920,"kb_used_data":4232,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435232,"statfs":{"total":95995035648,"available":95677677568,"internally_reserved":0,"allocated":4333568,"data_stored":2568778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.90500000000000003,"15min":0.92500000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.91600000000000004,"15min":0.92500000000000004},"min":{"1min":0.442,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.216,"5min":1.554,"15min":1.554},"last":1.2430000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.872,"15min":0.89300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.244,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96699999999999997,"15min":1.0269999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.36499999999999999},"max":{"1min":1.742,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.075}]},{"osd":2,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.038,"15min":1.0549999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.47699999999999998},"max":{"1min":2.9199999999999999,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98999999999999999,"15min":0.97899999999999998},"min":{"1min":0.59799999999999998,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":1.2270000000000001,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.1990000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.97899999999999998,"15min":1.01},"min":{"1min":0.51400000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.8780000000000001,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.052,"5min":0.99299999999999999,"15min":0.97099999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":3.3050000000000002,"5min":3.3050000000000002,"15min":3.3050000000000002},"last":0.93600000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.996,"15min":1.0189999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.443},"max":{"1min":1.2809999999999999,"5min":2.1989999999999998,"15min":2.1989999999999998},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.044,"5min":1.046,"15min":1.026},"min":{"1min":0.55500000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.5349999999999999,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":1.296}]},{"osd":5,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98199999999999998,"15min":0.99099999999999999},"min":{"1min":0.70599999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.26,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":1.008},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1459999999999999,"15min":1.1819999999999999},"min":{"1min":0.67900000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.357,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.165}]},{"osd":6,"last update":"Tue Sep 10 18:55:43 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.1830000000000001,"15min":1.234},"min":{"1min":0.64500000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.383,"5min":2.2429999999999999,"15min":2.2749999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.1850000000000001},"min":{"1min":0.73399999999999999,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.4390000000000001,"5min":2.0070000000000001,"15min":2.536},"last":1.3400000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299503,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313980,"kb_used_data":4132,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93431172,"statfs":{"total":95995035648,"available":95673520128,"internally_reserved":0,"allocated":4231168,"data_stored":2567164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.85999999999999999,"15min":0.88400000000000001},"min":{"1min":0.44500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.97399999999999998,"15min":0.94499999999999995},"min":{"1min":0.45900000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.9970000000000001,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.179,"15min":0.98299999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.633,"5min":18.494,"15min":18.494},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.88700000000000001,"15min":0.90400000000000003},"min":{"1min":0.55000000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.02,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.79700000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.0269999999999999,"15min":0.94899999999999995},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0760000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.093},{"interface":"front","average":{"1min":0.87,"5min":0.91600000000000004,"15min":0.92300000000000004},"min":{"1min":0.57899999999999996,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":1.1479999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0189999999999999,"15min":0.96699999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.278,"5min":2.278,"15min":2.278},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.97999999999999998,"15min":0.97399999999999998},"min":{"1min":0.68100000000000005,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.696,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":0.98399999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.97399999999999998,"15min":0.89000000000000001},"min":{"1min":0.66200000000000003,"5min":0.56699999999999995,"15min":0.52000000000000002},"max":{"1min":1.4239999999999999,"5min":1.712,"15min":1.712},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0609999999999999,"15min":1.0189999999999999},"min":{"1min":0.68100000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.411,"5min":2.202,"15min":2.202},"last":1.429}]},{"osd":5,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.042,"15min":0.98799999999999999},"min":{"1min":0.751,"5min":0.53500000000000003,"15min":0.47499999999999998},"max":{"1min":1.4430000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.415},{"interface":"front","average":{"1min":1.141,"5min":1.1160000000000001,"15min":1.095},"min":{"1min":0.754,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.659,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.4670000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:55:46 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0580000000000001,"15min":1.044},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.333,"5min":1.516,"15min":1.573},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.177,"15min":1.1479999999999999},"min":{"1min":0.81299999999999994,"5min":0.68500000000000005,"15min":0.66900000000000004},"max":{"1min":1.77,"5min":1.77,"15min":1.77},"last":1.4510000000000001}]}]},{"osd":5,"up_from":43,"seq":184683593849,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309392,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435760,"statfs":{"total":95995035648,"available":95678218240,"internally_reserved":0,"allocated":3784704,"data_stored":1987803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95699999999999996,"15min":0.878},"min":{"1min":0.50800000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.0739999999999998,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.94499999999999995,"15min":0.88400000000000001},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.4379999999999999,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.92000000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.357,"15min":1.0189999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":3.0409999999999999,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.081},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.345,"15min":1.036},"min":{"1min":0.54600000000000004,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":4.7729999999999997,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.1379999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.605,"15min":1.127},"min":{"1min":0.53900000000000003,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":5.9809999999999999,"5min":61.347000000000001,"15min":61.347000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.0569999999999999,"15min":0.94799999999999995},"min":{"1min":0.61199999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":4.6950000000000003,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":1.161}]},{"osd":3,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.998,"15min":0.93899999999999995},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.3140000000000001,"5min":5.9809999999999999,"15min":5.9809999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.6339999999999999,"15min":1.1679999999999999},"min":{"1min":0.66300000000000003,"5min":0.60999999999999999,"15min":0.57399999999999995},"max":{"1min":4.6550000000000002,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.2310000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:56:10 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":0.95999999999999996,"15min":0.83399999999999996},"min":{"1min":0.76500000000000001,"5min":0.59599999999999997,"15min":0.439},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":2.9220000000000002},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.04,"15min":0.92100000000000004},"min":{"1min":0.755,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.722,"5min":1.1759999999999999,"15min":1.0680000000000001},"min":{"1min":0.71999999999999997,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":6.2149999999999999,"5min":6.2149999999999999,"15min":6.2149999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.246,"15min":1.107},"min":{"1min":0.79300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":7.3520000000000003,"5min":7.3520000000000003,"15min":7.3520000000000003},"last":0.81599999999999995}]},{"osd":7,"last update":"Tue Sep 10 18:55:51 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.2110000000000001,"15min":1.121},"min":{"1min":0.89300000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":9.1140000000000008,"5min":9.1140000000000008,"15min":9.1140000000000008},"last":1.028},{"interface":"front","average":{"1min":1.774,"5min":1.226,"15min":1.198},"min":{"1min":0.88100000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":6.532,"5min":6.532,"15min":6.532},"last":0.94799999999999995}]}]},{"osd":4,"up_from":35,"seq":150323855493,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313388,"kb_used_data":3548,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93431764,"statfs":{"total":95995035648,"available":95674126336,"internally_reserved":0,"allocated":3633152,"data_stored":1977085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82799999999999996,"15min":0.84199999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.113,"5min":1.546,"15min":1.546},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.85499999999999998},"min":{"1min":0.58999999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2290000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.0920000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.873},"min":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.266,"5min":1.623,"15min":1.623},"last":0.879},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92800000000000005,"15min":0.878},"min":{"1min":0.49299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.3340000000000001,"5min":1.575,"15min":1.575},"last":1.294}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95999999999999996,"15min":0.90400000000000003},"min":{"1min":0.59899999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2569999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96099999999999997,"15min":0.90200000000000002},"min":{"1min":0.66400000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.325,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.84399999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.91300000000000003},"min":{"1min":0.53800000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2829999999999999,"5min":1.4730000000000001,"15min":1.714},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.97699999999999998,"15min":0.93500000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55600000000000005},"max":{"1min":1.329,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.147}]},{"osd":5,"last update":"Tue Sep 10 18:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.96099999999999997,"15min":0.88},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.52100000000000002},"max":{"1min":1.4790000000000001,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0389999999999999,"15min":0.93300000000000005},"min":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.4610000000000001,"5min":1.528,"15min":1.7909999999999999},"last":1.1859999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0309999999999999,"15min":0.95599999999999996},"min":{"1min":0.66600000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.464,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.143,"15min":1.0840000000000001},"min":{"1min":0.77200000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":1.349}]},{"osd":7,"last update":"Tue Sep 10 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.147,"15min":1.165},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.6200000000000001,"5min":2.008,"15min":2.008},"last":1.262},{"interface":"front","average":{"1min":1.177,"5min":1.149,"15min":1.1659999999999999},"min":{"1min":0.70599999999999996,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.538,"5min":1.538,"15min":1.6120000000000001},"last":1.248}]}]},{"osd":3,"up_from":29,"seq":124554051728,"num_pgs":137,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305216,"kb_used_data":3680,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439936,"statfs":{"total":95995035648,"available":95682494464,"internally_reserved":0,"allocated":3768320,"data_stored":1988249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.109,"15min":0.78100000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":1.0229999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.627,"5min":1.1879999999999999,"15min":0.872},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.0009999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.2929999999999999,"15min":0.88600000000000001},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.97399999999999998,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.024,"15min":0.83899999999999997},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.95699999999999996,"5min":15.189,"15min":15.189},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.3089999999999999,"15min":0.96499999999999997},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.329,"5min":26.558,"15min":26.558},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.2330000000000001,"15min":0.91600000000000004},"min":{"1min":0.38300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.256,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.78500000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.238,"15min":1.079},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.4019999999999999,"5min":10.359999999999999,"15min":10.359999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.3360000000000001,"15min":1.165},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.9580000000000002,"5min":10.391,"15min":10.391},"last":1}]},{"osd":5,"last update":"Tue Sep 10 18:56:06 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.385,"15min":1.1930000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":9.2750000000000004,"5min":17.535,"15min":17.535},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.0600000000000001,"15min":1.0069999999999999},"min":{"1min":0.51100000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":5.3369999999999997,"5min":5.3369999999999997,"15min":5.3369999999999997},"last":0.81499999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:55:58 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.286,"15min":1.204},"min":{"1min":0.71099999999999997,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":9.3610000000000007,"5min":10.51,"15min":10.51},"last":1.044},{"interface":"front","average":{"1min":1.506,"5min":1.127,"15min":1.0900000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":10.773999999999999,"5min":10.773999999999999,"15min":10.773999999999999},"last":0.95299999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:55:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.3280000000000001,"15min":1.1870000000000001},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":5.3109999999999999,"5min":18.312999999999999,"15min":18.312999999999999},"last":0.872},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.3899999999999999,"15min":1.2929999999999999},"min":{"1min":0.66500000000000004,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":9.3840000000000003,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.087}]}]},{"osd":2,"up_from":19,"seq":81604378777,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313548,"kb_used_data":3708,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93431604,"statfs":{"total":95995035648,"available":95673962496,"internally_reserved":0,"allocated":3796992,"data_stored":1978180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.84099999999999997,"15min":0.70799999999999996},"min":{"1min":0.32200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84999999999999998,"5min":9.516,"15min":9.516},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.82799999999999996,"15min":0.76200000000000001},"min":{"1min":0.25800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0620000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.94499999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.36699999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82999999999999996,"5min":5.516,"15min":5.516},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.88900000000000001,"15min":0.79400000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.052,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.92100000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91000000000000003,"15min":0.85099999999999998},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.3579999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.546,"5min":0.91700000000000004,"15min":0.84099999999999997},"min":{"1min":0.499,"5min":0.434,"15min":0.434},"max":{"1min":15.736000000000001,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.1950000000000001,"15min":1.038},"min":{"1min":0.77200000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":21.571999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1699999999999999,"15min":1.1060000000000001},"min":{"1min":0.621,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.377,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.337}]},{"osd":5,"last update":"Tue Sep 10 18:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1160000000000001,"15min":1.111},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.2720000000000002,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":1.4490000000000001},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.2430000000000001,"15min":1.0820000000000001},"min":{"1min":0.70499999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":21.654,"5min":21.654,"15min":21.654},"last":1.222}]},{"osd":6,"last update":"Tue Sep 10 18:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.2210000000000001,"15min":1.1930000000000001},"min":{"1min":0.66900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.2450000000000001,"15min":1.125},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":15.971,"5min":15.971,"15min":15.971},"last":1.0149999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:55:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.2310000000000001,"15min":1.2150000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":15.025,"5min":15.025,"15min":15.025},"last":1.399},{"interface":"front","average":{"1min":1.927,"5min":1.2789999999999999,"15min":1.171},"min":{"1min":0.65800000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.816000000000001,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.173}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":138,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309336,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435816,"statfs":{"total":95995035648,"available":95678275584,"internally_reserved":0,"allocated":3743744,"data_stored":1977527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0309999999999999,"15min":0.749},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.90500000000000003,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.095,"15min":0.81699999999999995},"min":{"1min":0.30399999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.0529999999999999,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.53700000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1040000000000001,"15min":0.81200000000000006},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1299999999999999,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":0.93000000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4370000000000001,"5min":32.363,"15min":32.363},"last":0.60999999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.1930000000000001,"15min":0.94699999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.2649999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.4910000000000001,"15min":0.98499999999999999},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89600000000000002,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1599999999999999,"15min":1.0529999999999999},"min":{"1min":0.48199999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3320000000000001,"5min":12.808999999999999,"15min":12.808999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.6299999999999999,"15min":1.1359999999999999},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.3009999999999999,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.80000000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.3700000000000001,"5min":1.2869999999999999,"15min":1.0980000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":26.050999999999998,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.1419999999999999},{"interface":"front","average":{"1min":2.0609999999999999,"5min":1.2929999999999999,"15min":1.1830000000000001},"min":{"1min":0.69199999999999995,"5min":0.63100000000000001,"15min":0.61199999999999999},"max":{"1min":12.84,"5min":12.84,"15min":12.84},"last":1.0269999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.891,"5min":1.4670000000000001,"15min":1.206},"min":{"1min":0.78200000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":25.608000000000001,"5min":25.608000000000001,"15min":25.608000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":3.0990000000000002,"5min":1.458,"15min":1.202},"min":{"1min":0.622,"5min":0.622,"15min":0.41499999999999998},"max":{"1min":33.643000000000001,"5min":33.643000000000001,"15min":33.643000000000001},"last":0.95799999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.4670000000000001,"15min":1.304},"min":{"1min":0.83999999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":25.565999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":3.3900000000000001,"5min":1.5629999999999999,"15min":1.2849999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.48399999999999999},"max":{"1min":33.673999999999999,"5min":33.673999999999999,"15min":33.673999999999999},"last":0.91300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738544,"num_pgs":131,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309852,"kb_used_data":4156,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435300,"statfs":{"total":95995035648,"available":95677747200,"internally_reserved":0,"allocated":4255744,"data_stored":2577565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:55:44 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":0.77700000000000002,"15min":0.64900000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.4659999999999993,"5min":8.4659999999999993,"15min":8.4659999999999993},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.88300000000000001,"15min":0.73099999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":10.894,"5min":10.894,"15min":10.894},"last":0.746}]},{"osd":2,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.5920000000000001,"15min":0.98699999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.2789999999999999,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.214,"15min":0.873},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.0640000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.88}]},{"osd":3,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.0529999999999999,"15min":0.84999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0669999999999999,"5min":21.620999999999999,"15min":21.620999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.006},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3280000000000001,"5min":45.765000000000001,"15min":45.765000000000001},"last":1.159}]},{"osd":4,"last update":"Tue Sep 10 18:56:19 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.131,"15min":1.014},"min":{"1min":0.69299999999999995,"5min":0.5,"15min":0.5},"max":{"1min":5.5369999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.149,"15min":1.0760000000000001},"min":{"1min":0.71399999999999997,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":4.2389999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.4990000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:56:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.2390000000000001,"15min":1.0860000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":2.7989999999999999,"5min":10.048,"15min":10.048},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1339999999999999,"15min":1.1000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.4339999999999999,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":1.635}]},{"osd":6,"last update":"Tue Sep 10 18:55:53 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.161,"15min":1.1240000000000001},"min":{"1min":0.81699999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":3.9550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.3140000000000001,"15min":1.179},"min":{"1min":0.68400000000000005,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":11.894,"5min":11.894,"15min":11.894},"last":1.5660000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:55:48 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1850000000000001,"15min":1.1699999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.266,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.675},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1950000000000001,"15min":1.2749999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.976,"5min":3.976,"15min":3.976},"last":1.732}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:56:49.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-10T18:56:49.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:49.205 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:49.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:49.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:49.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:49.481 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:49.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:49.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:56:49.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:49.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:49.751 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:49.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:49.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:49.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:49.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:50.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:50.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.038808+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: cluster 2024-09-10T18:56:49.038994+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.041124+0000 mon.c (mon.2) 389 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.050044+0000 mon.a (mon.0) 1698 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.070469+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.074448+0000 mon.a (mon.0) 1699 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: cluster 2024-09-10T18:56:49.216521+0000 mgr.y (mgr.24526) 311 : cluster [DBG] pgmap v363: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.538484+0000 mon.c (mon.2) 390 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.756536+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.978235+0000 mon.c (mon.2) 392 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:56:50.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.979685+0000 mon.c (mon.2) 393 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[18629]: audit 2024-09-10T18:56:49.992320+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.038808+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: cluster 2024-09-10T18:56:49.038994+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.041124+0000 mon.c (mon.2) 389 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.050044+0000 mon.a (mon.0) 1698 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.070469+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.074448+0000 mon.a (mon.0) 1699 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: cluster 2024-09-10T18:56:49.216521+0000 mgr.y (mgr.24526) 311 : cluster [DBG] pgmap v363: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.538484+0000 mon.c (mon.2) 390 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.756536+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:50.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.978235+0000 mon.c (mon.2) 392 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:56:50.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.979685+0000 mon.c (mon.2) 393 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:56:50.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:50 smithi083 bash[32381]: audit 2024-09-10T18:56:49.992320+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:50.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.038808+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? v1:172.21.15.83:0/1541046648' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: cluster 2024-09-10T18:56:49.038994+0000 mon.a (mon.0) 1697 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.041124+0000 mon.c (mon.2) 389 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.050044+0000 mon.a (mon.0) 1698 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.070469+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.074448+0000 mon.a (mon.0) 1699 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: cluster 2024-09-10T18:56:49.216521+0000 mgr.y (mgr.24526) 311 : cluster [DBG] pgmap v363: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:56:50.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.538484+0000 mon.c (mon.2) 390 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:50.271 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.756536+0000 mon.c (mon.2) 391 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:50.271 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.978235+0000 mon.c (mon.2) 392 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:56:50.271 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.979685+0000 mon.c (mon.2) 393 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:56:50.271 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:50 smithi165 bash[22591]: audit 2024-09-10T18:56:49.992320+0000 mon.a (mon.0) 1700 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:56:50.290 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:50.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:50.427 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:50.461 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:56:50.461 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:56:50.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:50.563 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:50.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:50.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:50.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.839 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:50.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:50.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:50.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:51.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:51.099 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:51.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:51.236 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: cluster 2024-09-10T18:56:49.982354+0000 mgr.y (mgr.24526) 312 : cluster [DBG] pgmap v364: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: cluster 2024-09-10T18:56:50.043969+0000 mon.a (mon.0) 1701 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.056146+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.057565+0000 mon.a (mon.0) 1702 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.057901+0000 mon.a (mon.0) 1703 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:51.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: cluster 2024-09-10T18:56:50.058122+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.059586+0000 mon.c (mon.2) 394 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.060073+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.83:0/1949657653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.061727+0000 mon.a (mon.0) 1705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.062090+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.062413+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48995 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:50.758009+0000 mon.c (mon.2) 396 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:51.058562+0000 mon.a (mon.0) 1708 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:51.058662+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: audit 2024-09-10T18:56:51.058760+0000 mon.a (mon.0) 1710 : audit [INF] from='client.48995 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[18629]: cluster 2024-09-10T18:56:51.059058+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:56:51.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: cluster 2024-09-10T18:56:49.982354+0000 mgr.y (mgr.24526) 312 : cluster [DBG] pgmap v364: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: cluster 2024-09-10T18:56:50.043969+0000 mon.a (mon.0) 1701 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.056146+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.057565+0000 mon.a (mon.0) 1702 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.057901+0000 mon.a (mon.0) 1703 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: cluster 2024-09-10T18:56:50.058122+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.059586+0000 mon.c (mon.2) 394 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.060073+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.83:0/1949657653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.061727+0000 mon.a (mon.0) 1705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.062090+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.062413+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48995 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:50.758009+0000 mon.c (mon.2) 396 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:51.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:51.058562+0000 mon.a (mon.0) 1708 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:51.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:51.058662+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: audit 2024-09-10T18:56:51.058760+0000 mon.a (mon.0) 1710 : audit [INF] from='client.48995 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:51.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:51 smithi083 bash[32381]: cluster 2024-09-10T18:56:51.059058+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:56:51.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T18:56:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: cluster 2024-09-10T18:56:49.982354+0000 mgr.y (mgr.24526) 312 : cluster [DBG] pgmap v364: 292 pgs: 64 unknown, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 221 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:56:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: cluster 2024-09-10T18:56:50.043969+0000 mon.a (mon.0) 1701 : cluster [INF] Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-10T18:56:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.056146+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.057565+0000 mon.a (mon.0) 1702 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.057901+0000 mon.a (mon.0) 1703 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: cluster 2024-09-10T18:56:50.058122+0000 mon.a (mon.0) 1704 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.059586+0000 mon.c (mon.2) 394 : audit [INF] from='client.? v1:172.21.15.83:0/4095671902' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.060073+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.83:0/1949657653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.061727+0000 mon.a (mon.0) 1705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.062090+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48965 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.062413+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48995 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:50.758009+0000 mon.c (mon.2) 396 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:51.058562+0000 mon.a (mon.0) 1708 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:51.058662+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48965 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi083-77640-32"}]': finished 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: audit 2024-09-10T18:56:51.058760+0000 mon.a (mon.0) 1710 : audit [INF] from='client.48995 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[22591]: cluster 2024-09-10T18:56:51.059058+0000 mon.a (mon.0) 1711 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-09-10T18:56:51.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:51.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.507 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:51.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:51.624 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:56:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:56:51.645 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:51.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:51.785 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:51.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.922 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:51.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:52.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:52.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:52.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:52.196 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:52.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:52.337 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:52.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.061785+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.066991+0000 mon.a (mon.0) 1712 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.097693+0000 mon.c (mon.2) 397 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.098067+0000 mon.a (mon.0) 1713 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.099497+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.099834+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.100428+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.100722+0000 mon.a (mon.0) 1715 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:51.759669+0000 mon.c (mon.2) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: cluster 2024-09-10T18:56:52.051894+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:52.063841+0000 mon.a (mon.0) 1717 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]': finished 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:52.064088+0000 mon.a (mon.0) 1718 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:52.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: cluster 2024-09-10T18:56:52.064536+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:52.067624+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:52.068462+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18629]: audit 2024-09-10T18:56:52.069111+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.061785+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.066991+0000 mon.a (mon.0) 1712 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.097693+0000 mon.c (mon.2) 397 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.098067+0000 mon.a (mon.0) 1713 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.099497+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.099834+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.100428+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.100722+0000 mon.a (mon.0) 1715 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:51.759669+0000 mon.c (mon.2) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: cluster 2024-09-10T18:56:52.051894+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:52.063841+0000 mon.a (mon.0) 1717 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]': finished 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:52.064088+0000 mon.a (mon.0) 1718 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: cluster 2024-09-10T18:56:52.064536+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:52.067624+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:52.068462+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:52.341 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[32381]: audit 2024-09-10T18:56:52.069111+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.061785+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.066991+0000 mon.a (mon.0) 1712 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.097693+0000 mon.c (mon.2) 397 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.098067+0000 mon.a (mon.0) 1713 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.099497+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.099834+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.100428+0000 mon.c (mon.2) 399 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.100722+0000 mon.a (mon.0) 1715 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:51.759669+0000 mon.c (mon.2) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: cluster 2024-09-10T18:56:52.051894+0000 mon.a (mon.0) 1716 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:52.063841+0000 mon.a (mon.0) 1717 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-23", "mode": "writeback"}]': finished 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:52.064088+0000 mon.a (mon.0) 1718 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi083-77640-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: cluster 2024-09-10T18:56:52.064536+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:52.067624+0000 mon.c (mon.2) 401 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:52.068462+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:52.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:52 smithi165 bash[22591]: audit 2024-09-10T18:56:52.069111+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:52.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:52.477 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:52.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:52.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:52.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:52.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:52.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:52.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:52.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:52.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:56:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:56:52] "GET /metrics HTTP/1.1" 200 238334 "" "Prometheus/2.43.0" 2024-09-10T18:56:53.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.162 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:53.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.302 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:53.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:53.436 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:53.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:53 smithi165 bash[22591]: cluster 2024-09-10T18:56:51.983166+0000 mgr.y (mgr.24526) 313 : cluster [DBG] pgmap v367: 324 pgs: 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:53.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:53 smithi165 bash[22591]: audit 2024-09-10T18:56:52.761349+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:53.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:53 smithi165 bash[22591]: audit 2024-09-10T18:56:53.065599+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:53.573 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:53.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.714 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[18629]: cluster 2024-09-10T18:56:51.983166+0000 mgr.y (mgr.24526) 313 : cluster [DBG] pgmap v367: 324 pgs: 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[18629]: audit 2024-09-10T18:56:52.761349+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[18629]: audit 2024-09-10T18:56:53.065599+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[32381]: cluster 2024-09-10T18:56:51.983166+0000 mgr.y (mgr.24526) 313 : cluster [DBG] pgmap v367: 324 pgs: 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[32381]: audit 2024-09-10T18:56:52.761349+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:53 smithi083 bash[32381]: audit 2024-09-10T18:56:53.065599+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? v1:172.21.15.83:0/3569841005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:53.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:53.863 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:53.864 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:53.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:53.982 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:53.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:54.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:54.116 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:54.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:54.253 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:54.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:54.391 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:54.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:54.523 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:54.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:54.641 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: cluster 2024-09-10T18:56:53.502306+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:56:54.641 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:53.521600+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.641 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:53.525624+0000 mon.a (mon.0) 1724 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.641 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:53.762997+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: cluster 2024-09-10T18:56:53.983731+0000 mgr.y (mgr.24526) 314 : cluster [DBG] pgmap v370: 356 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:54.023156+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:54.023513+0000 mon.a (mon.0) 1726 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: cluster 2024-09-10T18:56:54.023746+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:54.028231+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:54.030873+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:54.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:54 smithi165 bash[22591]: audit 2024-09-10T18:56:54.041149+0000 mon.a (mon.0) 1729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.661 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:54.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: cluster 2024-09-10T18:56:53.502306+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:53.521600+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:53.525624+0000 mon.a (mon.0) 1724 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:53.762997+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: cluster 2024-09-10T18:56:53.983731+0000 mgr.y (mgr.24526) 314 : cluster [DBG] pgmap v370: 356 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:54.023156+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:54.023513+0000 mon.a (mon.0) 1726 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: cluster 2024-09-10T18:56:54.023746+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:56:54.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:54.028231+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:54.030873+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[18629]: audit 2024-09-10T18:56:54.041149+0000 mon.a (mon.0) 1729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: cluster 2024-09-10T18:56:53.502306+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:53.521600+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:53.525624+0000 mon.a (mon.0) 1724 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:53.762997+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: cluster 2024-09-10T18:56:53.983731+0000 mgr.y (mgr.24526) 314 : cluster [DBG] pgmap v370: 356 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 285 active+clean; 586 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:54.023156+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi083-77640-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:54.023513+0000 mon.a (mon.0) 1726 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: cluster 2024-09-10T18:56:54.023746+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:54.028231+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:54.030873+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:54.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:54 smithi083 bash[32381]: audit 2024-09-10T18:56:54.041149+0000 mon.a (mon.0) 1729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]: dispatch 2024-09-10T18:56:54.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:54.899 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:54.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:54.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:55.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:55.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:55.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:55.203 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:55.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.341 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:55.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.478 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:55.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:55 smithi165 bash[22591]: audit 2024-09-10T18:56:54.764674+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:55.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:55 smithi165 bash[22591]: cluster 2024-09-10T18:56:55.016288+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:55.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:55 smithi165 bash[22591]: audit 2024-09-10T18:56:55.025860+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:55.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:55 smithi165 bash[22591]: audit 2024-09-10T18:56:55.026024+0000 mon.a (mon.0) 1732 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:55.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:55 smithi165 bash[22591]: cluster 2024-09-10T18:56:55.026128+0000 mon.a (mon.0) 1733 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:56:55.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[18629]: audit 2024-09-10T18:56:54.764674+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[18629]: cluster 2024-09-10T18:56:55.016288+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[18629]: audit 2024-09-10T18:56:55.025860+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[18629]: audit 2024-09-10T18:56:55.026024+0000 mon.a (mon.0) 1732 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[18629]: cluster 2024-09-10T18:56:55.026128+0000 mon.a (mon.0) 1733 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[32381]: audit 2024-09-10T18:56:54.764674+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[32381]: cluster 2024-09-10T18:56:55.016288+0000 mon.a (mon.0) 1730 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:56:55.594 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[32381]: audit 2024-09-10T18:56:55.025860+0000 mon.a (mon.0) 1731 : audit [INF] from='client.? v1:172.21.15.83:0/3516908431' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:55.595 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[32381]: audit 2024-09-10T18:56:55.026024+0000 mon.a (mon.0) 1732 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-23"}]': finished 2024-09-10T18:56:55.595 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:55 smithi083 bash[32381]: cluster 2024-09-10T18:56:55.026128+0000 mon.a (mon.0) 1733 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-09-10T18:56:55.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:55.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:55.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:55.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:55.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:55.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:55.892 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:56:55.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:56.134 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:56.160 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:56.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.296 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:56.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:56.433 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:56.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:54.724383+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:54.724808+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:55.551538+0000 osd.5 (osd.5) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:55.551919+0000 osd.5 (osd.5) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: audit 2024-09-10T18:56:55.766421+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:55.986961+0000 mgr.y (mgr.24526) 315 : cluster [DBG] pgmap v373: 396 pgs: 3 active+clean+snaptrim, 393 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: cluster 2024-09-10T18:56:56.027062+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: audit 2024-09-10T18:56:56.038640+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: audit 2024-09-10T18:56:56.047005+0000 mon.a (mon.0) 1735 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[18629]: audit 2024-09-10T18:56:56.047408+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:54.724383+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:54.724808+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:55.551538+0000 osd.5 (osd.5) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:55.551919+0000 osd.5 (osd.5) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: audit 2024-09-10T18:56:55.766421+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:55.986961+0000 mgr.y (mgr.24526) 315 : cluster [DBG] pgmap v373: 396 pgs: 3 active+clean+snaptrim, 393 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: cluster 2024-09-10T18:56:56.027062+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: audit 2024-09-10T18:56:56.038640+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: audit 2024-09-10T18:56:56.047005+0000 mon.a (mon.0) 1735 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:56 smithi083 bash[32381]: audit 2024-09-10T18:56:56.047408+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:56.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:56.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:56.813 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:56.849 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:54.724383+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:54.724808+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:55.551538+0000 osd.5 (osd.5) 169 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:55.551919+0000 osd.5 (osd.5) 170 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: audit 2024-09-10T18:56:55.766421+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:55.986961+0000 mgr.y (mgr.24526) 315 : cluster [DBG] pgmap v373: 396 pgs: 3 active+clean+snaptrim, 393 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: cluster 2024-09-10T18:56:56.027062+0000 mon.a (mon.0) 1734 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: audit 2024-09-10T18:56:56.038640+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: audit 2024-09-10T18:56:56.047005+0000 mon.a (mon.0) 1735 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:56 smithi165 bash[22591]: audit 2024-09-10T18:56:56.047408+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:56.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:56.984 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:56.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:57.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:57.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:57.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:57.247 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:57.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:56:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:56:57.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:57.520 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:56:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:56:57.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:57.660 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:57.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: cluster 2024-09-10T18:56:56.522930+0000 mon.a (mon.0) 1737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:56.768192+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.035919+0000 mon.a (mon.0) 1738 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.036159+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: cluster 2024-09-10T18:56:57.036423+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.037228+0000 mon.c (mon.2) 408 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.039338+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.043378+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[18629]: audit 2024-09-10T18:56:57.052527+0000 mon.a (mon.0) 1742 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: cluster 2024-09-10T18:56:56.522930+0000 mon.a (mon.0) 1737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:56.768192+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.035919+0000 mon.a (mon.0) 1738 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.036159+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: cluster 2024-09-10T18:56:57.036423+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.037228+0000 mon.c (mon.2) 408 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.039338+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.043378+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:57 smithi083 bash[32381]: audit 2024-09-10T18:56:57.052527+0000 mon.a (mon.0) 1742 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:56:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:56:57.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: cluster 2024-09-10T18:56:56.522930+0000 mon.a (mon.0) 1737 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:56.768192+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.035919+0000 mon.a (mon.0) 1738 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.036159+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? v1:172.21.15.83:0/3833866740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi083-77648-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: cluster 2024-09-10T18:56:57.036423+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.037228+0000 mon.c (mon.2) 408 : audit [INF] from='client.? v1:172.21.15.83:0/2737878222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.039338+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.043378+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49001 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]: dispatch 2024-09-10T18:56:57.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:57 smithi165 bash[22591]: audit 2024-09-10T18:56:57.052527+0000 mon.a (mon.0) 1742 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:56:57.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:57.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:57.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:58.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:58.070 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:58.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.206 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:58.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.343 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:58.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:58.482 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:58.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.620 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:58.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:58.761 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:57.765977+0000 mgr.y (mgr.24526) 316 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:57.769975+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: cluster 2024-09-10T18:56:57.987622+0000 mgr.y (mgr.24526) 317 : cluster [DBG] pgmap v376: 420 pgs: 64 unknown, 3 active+clean+snaptrim, 353 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.036810+0000 mon.a (mon.0) 1743 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.037058+0000 mon.a (mon.0) 1744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: cluster 2024-09-10T18:56:58.037232+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.077583+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.078046+0000 mon.a (mon.0) 1746 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.079648+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.080306+0000 mon.a (mon.0) 1747 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.083443+0000 mon.c (mon.2) 412 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[18629]: audit 2024-09-10T18:56:58.083854+0000 mon.a (mon.0) 1748 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:57.765977+0000 mgr.y (mgr.24526) 316 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:57.769975+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: cluster 2024-09-10T18:56:57.987622+0000 mgr.y (mgr.24526) 317 : cluster [DBG] pgmap v376: 420 pgs: 64 unknown, 3 active+clean+snaptrim, 353 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.036810+0000 mon.a (mon.0) 1743 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.037058+0000 mon.a (mon.0) 1744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: cluster 2024-09-10T18:56:58.037232+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.077583+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.078046+0000 mon.a (mon.0) 1746 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.079648+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.080306+0000 mon.a (mon.0) 1747 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.083443+0000 mon.c (mon.2) 412 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:58 smithi083 bash[32381]: audit 2024-09-10T18:56:58.083854+0000 mon.a (mon.0) 1748 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:57.765977+0000 mgr.y (mgr.24526) 316 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:57.769975+0000 mon.c (mon.2) 409 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: cluster 2024-09-10T18:56:57.987622+0000 mgr.y (mgr.24526) 317 : cluster [DBG] pgmap v376: 420 pgs: 64 unknown, 3 active+clean+snaptrim, 353 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.036810+0000 mon.a (mon.0) 1743 : audit [INF] from='client.49001 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi083-77640-33"}]': finished 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.037058+0000 mon.a (mon.0) 1744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: cluster 2024-09-10T18:56:58.037232+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.077583+0000 mon.c (mon.2) 410 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.078046+0000 mon.a (mon.0) 1746 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.079648+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.080306+0000 mon.a (mon.0) 1747 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:58.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.083443+0000 mon.c (mon.2) 412 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:58 smithi165 bash[22591]: audit 2024-09-10T18:56:58.083854+0000 mon.a (mon.0) 1748 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:56:58.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:58.897 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:58.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:59.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.044 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:59.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:59.183 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:56:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:56:59.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:59.321 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:56:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:56:59.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:59.457 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:56:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:58.772021+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: cluster 2024-09-10T18:56:58.809764+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: cluster 2024-09-10T18:56:58.810256+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:59.026504+0000 mon.a (mon.0) 1749 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: cluster 2024-09-10T18:56:59.026731+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:56:59.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:59.029400+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: cluster 2024-09-10T18:56:59.033145+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: cluster 2024-09-10T18:56:59.033437+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:59.041635+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:59.049378+0000 mon.a (mon.0) 1751 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.546 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[18629]: audit 2024-09-10T18:56:59.049907+0000 mon.a (mon.0) 1752 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.585 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:59.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:59.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:58.772021+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: cluster 2024-09-10T18:56:58.809764+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: cluster 2024-09-10T18:56:58.810256+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:59.026504+0000 mon.a (mon.0) 1749 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: cluster 2024-09-10T18:56:59.026731+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:59.029400+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: cluster 2024-09-10T18:56:59.033145+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.703 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: cluster 2024-09-10T18:56:59.033437+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:59.041635+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:59.049378+0000 mon.a (mon.0) 1751 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.704 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:56:59 smithi165 bash[22591]: audit 2024-09-10T18:56:59.049907+0000 mon.a (mon.0) 1752 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.725 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:56:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:56:59.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.840 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:58.772021+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:56:59.840 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: cluster 2024-09-10T18:56:58.809764+0000 osd.3 (osd.3) 199 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.840 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: cluster 2024-09-10T18:56:58.810256+0000 osd.3 (osd.3) 200 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:59.026504+0000 mon.a (mon.0) 1749 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi083-77640-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: cluster 2024-09-10T18:56:59.026731+0000 mon.a (mon.0) 1750 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:59.029400+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: cluster 2024-09-10T18:56:59.033145+0000 osd.4 (osd.4) 207 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: cluster 2024-09-10T18:56:59.033437+0000 osd.4 (osd.4) 208 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:59.041635+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:59.049378+0000 mon.a (mon.0) 1751 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:56:59.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:56:59 smithi083 bash[32381]: audit 2024-09-10T18:56:59.049907+0000 mon.a (mon.0) 1752 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:56:59.862 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:56:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:59.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:56:59.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:00.000 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:00.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:00.141 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:00.244 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:00.276 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:00.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:00.414 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:00.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:00.551 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:00.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:00.690 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:00.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:56:59.565975+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:56:59.570651+0000 mon.c (mon.2) 415 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:56:59.773741+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: cluster 2024-09-10T18:56:59.988493+0000 mgr.y (mgr.24526) 318 : cluster [DBG] pgmap v379: 356 pgs: 32 unknown, 3 active+clean+snaptrim, 321 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:57:00.042932+0000 mon.a (mon.0) 1754 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:57:00.044893+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: cluster 2024-09-10T18:57:00.481966+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[18629]: audit 2024-09-10T18:57:00.482873+0000 mon.a (mon.0) 1756 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:56:59.565975+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:56:59.570651+0000 mon.c (mon.2) 415 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:56:59.773741+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: cluster 2024-09-10T18:56:59.988493+0000 mgr.y (mgr.24526) 318 : cluster [DBG] pgmap v379: 356 pgs: 32 unknown, 3 active+clean+snaptrim, 321 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:57:00.042932+0000 mon.a (mon.0) 1754 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:57:00.044893+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: cluster 2024-09-10T18:57:00.481966+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:57:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:00 smithi083 bash[32381]: audit 2024-09-10T18:57:00.482873+0000 mon.a (mon.0) 1756 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:00.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:56:59.565975+0000 mon.a (mon.0) 1753 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:00.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:56:59.570651+0000 mon.c (mon.2) 415 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:00.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:56:59.773741+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:00.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: cluster 2024-09-10T18:56:59.988493+0000 mgr.y (mgr.24526) 318 : cluster [DBG] pgmap v379: 356 pgs: 32 unknown, 3 active+clean+snaptrim, 321 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:00.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:57:00.042932+0000 mon.a (mon.0) 1754 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:00.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:57:00.044893+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: cluster 2024-09-10T18:57:00.481966+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-09-10T18:57:00.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:00 smithi165 bash[22591]: audit 2024-09-10T18:57:00.482873+0000 mon.a (mon.0) 1756 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:00.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:00.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:00.962 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:00.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:01.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:01.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:01.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:01.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:01.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:01.363 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:01.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:01.479 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:01.501 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:01.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:01.640 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:01.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:01.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: audit 2024-09-10T18:57:00.775337+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: audit 2024-09-10T18:57:01.044792+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: audit 2024-09-10T18:57:01.045634+0000 mon.a (mon.0) 1757 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: audit 2024-09-10T18:57:01.045769+0000 mon.a (mon.0) 1758 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: cluster 2024-09-10T18:57:01.045906+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:57:01.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:01 smithi165 bash[22591]: audit 2024-09-10T18:57:01.051180+0000 mon.a (mon.0) 1760 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:01.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: audit 2024-09-10T18:57:00.775337+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:01.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: audit 2024-09-10T18:57:01.044792+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: audit 2024-09-10T18:57:01.045634+0000 mon.a (mon.0) 1757 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:01.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: audit 2024-09-10T18:57:01.045769+0000 mon.a (mon.0) 1758 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: cluster 2024-09-10T18:57:01.045906+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[32381]: audit 2024-09-10T18:57:01.051180+0000 mon.a (mon.0) 1760 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: audit 2024-09-10T18:57:00.775337+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: audit 2024-09-10T18:57:01.044792+0000 mon.b (mon.1) 219 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: audit 2024-09-10T18:57:01.045634+0000 mon.a (mon.0) 1757 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi083-77640-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: audit 2024-09-10T18:57:01.045769+0000 mon.a (mon.0) 1758 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: cluster 2024-09-10T18:57:01.045906+0000 mon.a (mon.0) 1759 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-09-10T18:57:01.895 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:01 smithi083 bash[18629]: audit 2024-09-10T18:57:01.051180+0000 mon.a (mon.0) 1760 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]: dispatch 2024-09-10T18:57:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:01.916 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:01.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:02.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:02.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:02.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:02.201 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:02.305 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:02.335 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:02.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:02.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:02.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:02.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:02.712 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:02.749 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:02.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: audit 2024-09-10T18:57:01.776637+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: cluster 2024-09-10T18:57:01.989443+0000 mgr.y (mgr.24526) 319 : cluster [DBG] pgmap v382: 300 pgs: 8 unknown, 292 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: cluster 2024-09-10T18:57:02.039334+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: audit 2024-09-10T18:57:02.048986+0000 mon.a (mon.0) 1762 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]': finished 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: cluster 2024-09-10T18:57:02.049874+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: audit 2024-09-10T18:57:02.102766+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:02 smithi165 bash[22591]: audit 2024-09-10T18:57:02.514671+0000 mon.a (mon.0) 1764 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.882 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:02] "GET /metrics HTTP/1.1" 200 238314 "" "Prometheus/2.43.0" 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: audit 2024-09-10T18:57:01.776637+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: cluster 2024-09-10T18:57:01.989443+0000 mgr.y (mgr.24526) 319 : cluster [DBG] pgmap v382: 300 pgs: 8 unknown, 292 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: cluster 2024-09-10T18:57:02.039334+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: audit 2024-09-10T18:57:02.048986+0000 mon.a (mon.0) 1762 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]': finished 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: cluster 2024-09-10T18:57:02.049874+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: audit 2024-09-10T18:57:02.102766+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[18629]: audit 2024-09-10T18:57:02.514671+0000 mon.a (mon.0) 1764 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: audit 2024-09-10T18:57:01.776637+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: cluster 2024-09-10T18:57:01.989443+0000 mgr.y (mgr.24526) 319 : cluster [DBG] pgmap v382: 300 pgs: 8 unknown, 292 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: cluster 2024-09-10T18:57:02.039334+0000 mon.a (mon.0) 1761 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: audit 2024-09-10T18:57:02.048986+0000 mon.a (mon.0) 1762 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-25", "mode": "writeback"}]': finished 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: cluster 2024-09-10T18:57:02.049874+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: audit 2024-09-10T18:57:02.102766+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.884 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:02 smithi083 bash[32381]: audit 2024-09-10T18:57:02.514671+0000 mon.a (mon.0) 1764 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:02.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:02.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:02.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:03.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:03.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:03.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.164 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:03.268 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:03.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:03.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.439 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:03.542 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:03.574 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:03.677 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.711 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:03.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:03.848 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:03.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: cluster 2024-09-10T18:57:02.573231+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:02.778022+0000 mon.c (mon.2) 419 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.051651+0000 mon.a (mon.0) 1766 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: cluster 2024-09-10T18:57:03.051943+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.054364+0000 mon.c (mon.2) 420 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.056520+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.83:0/1906485925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.057207+0000 mon.a (mon.0) 1768 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.058837+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.060967+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:03.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:03 smithi165 bash[22591]: audit 2024-09-10T18:57:03.063196+0000 mon.a (mon.0) 1770 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:03.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:03.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:03.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: cluster 2024-09-10T18:57:02.573231+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:02.778022+0000 mon.c (mon.2) 419 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.051651+0000 mon.a (mon.0) 1766 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: cluster 2024-09-10T18:57:03.051943+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.054364+0000 mon.c (mon.2) 420 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.056520+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.83:0/1906485925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.057207+0000 mon.a (mon.0) 1768 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.058837+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.060967+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[18629]: audit 2024-09-10T18:57:03.063196+0000 mon.a (mon.0) 1770 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: cluster 2024-09-10T18:57:02.573231+0000 mon.a (mon.0) 1765 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:02.778022+0000 mon.c (mon.2) 419 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.051651+0000 mon.a (mon.0) 1766 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: cluster 2024-09-10T18:57:03.051943+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.054364+0000 mon.c (mon.2) 420 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.056520+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.83:0/1906485925' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.057207+0000 mon.a (mon.0) 1768 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.058837+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.060967+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49546 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:03 smithi083 bash[32381]: audit 2024-09-10T18:57:03.063196+0000 mon.a (mon.0) 1770 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]: dispatch 2024-09-10T18:57:04.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:04.119 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:04.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:04.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:04.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:04.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:04.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:04.539 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:04.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:04.673 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:04.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:04.811 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: cluster 2024-09-10T18:57:03.523006+0000 osd.0 (osd.0) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: cluster 2024-09-10T18:57:03.523381+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:03.779611+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: cluster 2024-09-10T18:57:03.990434+0000 mgr.y (mgr.24526) 320 : cluster [DBG] pgmap v385: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: cluster 2024-09-10T18:57:04.018813+0000 mon.a (mon.0) 1771 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:04.034082+0000 mon.a (mon.0) 1772 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:04.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:04.034643+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:04.034786+0000 mon.a (mon.0) 1774 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: cluster 2024-09-10T18:57:04.034930+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:57:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:04.037842+0000 mon.c (mon.2) 422 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:04 smithi165 bash[22591]: audit 2024-09-10T18:57:04.046595+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:04.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: cluster 2024-09-10T18:57:03.523006+0000 osd.0 (osd.0) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:04.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: cluster 2024-09-10T18:57:03.523381+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:04.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:03.779611+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:04.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: cluster 2024-09-10T18:57:03.990434+0000 mgr.y (mgr.24526) 320 : cluster [DBG] pgmap v385: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: cluster 2024-09-10T18:57:04.018813+0000 mon.a (mon.0) 1771 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:04.034082+0000 mon.a (mon.0) 1772 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:04.034643+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:04.034786+0000 mon.a (mon.0) 1774 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: cluster 2024-09-10T18:57:04.034930+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:04.037842+0000 mon.c (mon.2) 422 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[32381]: audit 2024-09-10T18:57:04.046595+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: cluster 2024-09-10T18:57:03.523006+0000 osd.0 (osd.0) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: cluster 2024-09-10T18:57:03.523381+0000 osd.0 (osd.0) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:03.779611+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:04.929 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: cluster 2024-09-10T18:57:03.990434+0000 mgr.y (mgr.24526) 320 : cluster [DBG] pgmap v385: 292 pgs: 32 unknown, 260 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: cluster 2024-09-10T18:57:04.018813+0000 mon.a (mon.0) 1771 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:04.034082+0000 mon.a (mon.0) 1772 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:04.034643+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49546 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi083-77648-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:04.034786+0000 mon.a (mon.0) 1774 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-25"}]': finished 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: cluster 2024-09-10T18:57:04.034930+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:04.037842+0000 mon.c (mon.2) 422 : audit [INF] from='client.? v1:172.21.15.83:0/1799998036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.930 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:04 smithi083 bash[18629]: audit 2024-09-10T18:57:04.046595+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49025 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]: dispatch 2024-09-10T18:57:04.947 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:04.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:05.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:05.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:05.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:05.227 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:05.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:05.368 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:05.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:05.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:05.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:05.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/277180672 >> v1:172.21.15.83:6790/0 conn(0x7f1500105450 legacy=0x7f1500107830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/277180672 shutdown_connections 2024-09-10T18:57:05.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/277180672 >> 172.21.15.83:0/277180672 conn(0x7f1500074a20 msgr2=0x7f15000f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:05.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/277180672 shutdown_connections 2024-09-10T18:57:05.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/277180672 wait complete. 2024-09-10T18:57:05.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 librados: starting msgr at 2024-09-10T18:57:05.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 librados: starting objecter 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 Processor -- start 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- start start 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 librados: setting wanted keys 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 librados: calling monclient init 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15000fd9d0 con 0x7f1500101590 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15000fdb80 con 0x7f1500105450 2024-09-10T18:57:05.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1500120840 con 0x7f1500108fb0 2024-09-10T18:57:05.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14ff7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1500101590 0x7f15000fd1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53212/0 (socket says 172.21.15.83:53212) 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14ff7fe640 1 -- 172.21.15.83:0/2517598000 learned_addr learned my addr 172.21.15.83:0/2517598000 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter ms_handle_connect 0x7f1500101590 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter ms_handle_connect 0x7f1500108fb0 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter ms_handle_connect 0x7f1500105450 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282524340 0 0) 0x7f1500120840 con 0x7f1500108fb0 2024-09-10T18:57:05.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003720 con 0x7f1500108fb0 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195828908 0 0) 0x7f15000fdb80 con 0x7f1500105450 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1500120840 con 0x7f1500105450 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755503551 0 0) 0x7f14e8003720 con 0x7f1500108fb0 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15000fdb80 con 0x7f1500108fb0 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f14f8001af0 con 0x7f1500108fb0 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813005408 0 0) 0x7f1500120840 con 0x7f1500105450 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14e8003720 con 0x7f1500105450 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f14f00015d0 con 0x7f1500105450 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203206902 0 0) 0x7f15000fdb80 con 0x7f1500108fb0 2024-09-10T18:57:05.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 >> v1:172.21.15.165:6789/0 conn(0x7f1500105450 legacy=0x7f15001199d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 >> v1:172.21.15.83:6789/0 conn(0x7f1500101590 legacy=0x7f15000fd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15001219e0 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1500120a30 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 10 client.49049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1500120f40 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f150623a640 1 librados: init done 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f14f80021c0 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.624+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f14f80034d0 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f14f80198f0 con 0x7f1500108fb0 2024-09-10T18:57:05.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec005010 con 0x7f1500108fb0 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(269..269 src has 69..269) v4 ==== 7901+0+0 (unknown 420541792 0 0) 0x7f14f8087e70 con 0x7f1500108fb0 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 10 client.49049.objecter ms_dispatch 0x7f1500000b90 osd_map(269..269 src has 69..269) v4 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 3 client.49049.objecter handle_osd_map got epochs [269,269] > 0 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 3 client.49049.objecter handle_osd_map decoding full epoch 269 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 20 client.49049.objecter dump_active .. 0 homeless 2024-09-10T18:57:05.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.628+0000 7f14e7fff640 10 client.49049.objecter ms_handle_connect 0x7f14e8071580 2024-09-10T18:57:05.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.632+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f14f8054050 con 0x7f1500108fb0 2024-09-10T18:57:05.639 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:05.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:05.778 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.864+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f14ec005840 con 0x7f1500108fb0 2024-09-10T18:57:05.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f14e7fff640 1 -- 172.21.15.83:0/2517598000 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2524 (unknown 3226119948 0 3843948470) 0x7f14f8053e70 con 0x7f1500108fb0 2024-09-10T18:57:05.871 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:57:05.871 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":959,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":269,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994114,"num_in_osds":8,"osd_in_since":1725994087,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":260},{"state_name":"unknown","count":32}],"num_pgs":292,"num_pools":11,"num_objects":328,"data_bytes":599420,"bytes_used":2546257920,"bytes_avail":765414027264,"bytes_total":767960285184,"unknown_pgs_ratio":0.10958904027938843,"read_bytes_sec":1279,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-10T18:56:01.193087+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":13,"start_stamp":"2024-09-10T18:49:25.207072+0000","gid":14493,"addr":"172.21.15.83:0/703687015","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)","ceph_version_short":"17.2.7-1606-g5be57395","container_hostname":"smithi083","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi083","id":"foo.a","kernel_description":"#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024","kernel_version":"5.4.0-195-generic","mem_swap_kb":"2097148","mem_total_kb":"32754944","num_handles":"1","os":"Linux","pid":"6","realm_id":"","realm_name":"","zone_id":"ef939013-895d-4fcb-a130-e9a8938d4e31","zone_name":"default","zonegroup_id":"348ee464-2af6-469c-a2f0-1b47a5a7c71b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T18:57:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 10 librados: watch_flush enter 2024-09-10T18:57:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 10 librados: watch_flush exit 2024-09-10T18:57:05.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 20 client.49049.objecter shutdown clearing up homeless session... 2024-09-10T18:57:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 10 client.49049.objecter successfully canceled tick 2024-09-10T18:57:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 >> v1:172.21.15.83:6800/3071793735 conn(0x7f14e8071580 legacy=0x7f14e8073a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 >> v1:172.21.15.83:6790/0 conn(0x7f1500108fb0 legacy=0x7f150011d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 shutdown_connections 2024-09-10T18:57:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 >> 172.21.15.83:0/2517598000 conn(0x7f1500074a20 msgr2=0x7f15000f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 shutdown_connections 2024-09-10T18:57:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 -- 172.21.15.83:0/2517598000 wait complete. 2024-09-10T18:57:05.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:05.868+0000 7f150623a640 1 librados: shutdown 2024-09-10T18:57:05.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:05.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:05.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:06.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:05 smithi165 bash[22591]: audit 2024-09-10T18:57:04.781186+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:06.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:05 smithi165 bash[22591]: audit 2024-09-10T18:57:05.051292+0000 mon.a (mon.0) 1777 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:06.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:05 smithi165 bash[22591]: cluster 2024-09-10T18:57:05.051397+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:57:06.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:05 smithi165 bash[22591]: audit 2024-09-10T18:57:05.088371+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:05 smithi165 bash[22591]: audit 2024-09-10T18:57:05.499764+0000 mon.a (mon.0) 1779 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:06.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:06.195 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:06.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[18629]: audit 2024-09-10T18:57:04.781186+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:06.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[18629]: audit 2024-09-10T18:57:05.051292+0000 mon.a (mon.0) 1777 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:06.255 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[18629]: cluster 2024-09-10T18:57:05.051397+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[18629]: audit 2024-09-10T18:57:05.088371+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[18629]: audit 2024-09-10T18:57:05.499764+0000 mon.a (mon.0) 1779 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[32381]: audit 2024-09-10T18:57:04.781186+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[32381]: audit 2024-09-10T18:57:05.051292+0000 mon.a (mon.0) 1777 : audit [INF] from='client.49025 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi083-77640-34"}]': finished 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[32381]: cluster 2024-09-10T18:57:05.051397+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[32381]: audit 2024-09-10T18:57:05.088371+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.256 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:05 smithi083 bash[32381]: audit 2024-09-10T18:57:05.499764+0000 mon.a (mon.0) 1779 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:06.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:06.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.610 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:06.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:06.747 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:06.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- 172.21.15.83:0/1586164075 >> v1:172.21.15.83:6789/0 conn(0x7f6e600fbed0 legacy=0x7f6e60109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:06.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- 172.21.15.83:0/1586164075 shutdown_connections 2024-09-10T18:57:06.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- 172.21.15.83:0/1586164075 >> 172.21.15.83:0/1586164075 conn(0x7f6e60075290 msgr2=0x7f6e600f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:06.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- 172.21.15.83:0/1586164075 shutdown_connections 2024-09-10T18:57:06.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- 172.21.15.83:0/1586164075 wait complete. 2024-09-10T18:57:06.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 librados: starting msgr at 2024-09-10T18:57:06.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 librados: starting objecter 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 Processor -- start 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- start start 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 librados: setting wanted keys 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 librados: calling monclient init 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e6011e070 con 0x7f6e600fbed0 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e6011f230 con 0x7f6e600fb3f0 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e661db640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e60120430 con 0x7f6e6010b220 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e5effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6e600fbed0 0x7f6e601078f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53234/0 (socket says 172.21.15.83:53234) 2024-09-10T18:57:06.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e5effd640 1 -- 172.21.15.83:0/2853297582 learned_addr learned my addr 172.21.15.83:0/2853297582 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter ms_handle_connect 0x7f6e600fbed0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter ms_handle_connect 0x7f6e6010b220 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76146623 0 0) 0x7f6e6011e070 con 0x7f6e600fbed0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e48003200 con 0x7f6e600fbed0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter ms_handle_connect 0x7f6e600fb3f0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220822055 0 0) 0x7f6e60120430 con 0x7f6e6010b220 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.788+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6011e070 con 0x7f6e6010b220 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327783067 0 0) 0x7f6e6011f230 con 0x7f6e600fb3f0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e60120430 con 0x7f6e600fb3f0 2024-09-10T18:57:06.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951083974 0 0) 0x7f6e6011e070 con 0x7f6e6010b220 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6011f230 con 0x7f6e6010b220 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6e50001840 con 0x7f6e6010b220 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436203506 0 0) 0x7f6e48003200 con 0x7f6e600fbed0 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6011e070 con 0x7f6e600fbed0 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6e54001350 con 0x7f6e600fbed0 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4218846821 0 0) 0x7f6e6011f230 con 0x7f6e6010b220 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 >> v1:172.21.15.165:6789/0 conn(0x7f6e600fb3f0 legacy=0x7f6e601071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:06.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 >> v1:172.21.15.83:6789/0 conn(0x7f6e600fbed0 legacy=0x7f6e601078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e60121630 con 0x7f6e6010b220 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e60120620 con 0x7f6e6010b220 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e661db640 10 client.49055.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6e50001e00 con 0x7f6e6010b220 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e60120a70 con 0x7f6e6010b220 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e661db640 1 librados: init done 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6e500030d0 con 0x7f6e6010b220 2024-09-10T18:57:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f6e50003230 con 0x7f6e6010b220 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.792+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e60065740 con 0x7f6e6010b220 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(270..270 src has 69..270) v4 ==== 8895+0+0 (unknown 1196702659 0 0) 0x7f6e50088cd0 con 0x7f6e6010b220 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 10 client.49055.objecter ms_dispatch 0x7f6e60000b90 osd_map(270..270 src has 69..270) v4 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 3 client.49055.objecter handle_osd_map got epochs [270,270] > 0 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 3 client.49055.objecter handle_osd_map decoding full epoch 270 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 20 client.49055.objecter dump_active .. 0 homeless 2024-09-10T18:57:06.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.796+0000 7f6e47fff640 10 client.49055.objecter ms_handle_connect 0x7f6e480716e0 2024-09-10T18:57:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.800+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f6e50054ab0 con 0x7f6e6010b220 2024-09-10T18:57:06.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:06.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:06.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:06.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.503476+0000 mon.c (mon.2) 425 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.782784+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:06.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.870296+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.83:0/2517598000' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:06.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.933622+0000 mon.a (mon.0) 1780 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:06.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:06.976+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6e6004bf60 con 0x7f6e6010b220 2024-09-10T18:57:06.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:07.019 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.503476+0000 mon.c (mon.2) 425 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.782784+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.870296+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.83:0/2517598000' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.933622+0000 mon.a (mon.0) 1780 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.934574+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:05.940143+0000 mon.a (mon.0) 1781 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: cluster 2024-09-10T18:57:05.992549+0000 mgr.y (mgr.24526) 321 : cluster [DBG] pgmap v388: 228 pgs: 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.055132+0000 mon.a (mon.0) 1782 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: cluster 2024-09-10T18:57:06.055279+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.057062+0000 mon.c (mon.2) 429 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.059029+0000 mon.a (mon.0) 1784 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.060774+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.83:0/1020925830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.065182+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49043 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.068193+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.072357+0000 mon.a (mon.0) 1786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:06 smithi165 bash[22591]: audit 2024-09-10T18:57:06.784129+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:07.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:07.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:07.285 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.934574+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:05.940143+0000 mon.a (mon.0) 1781 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: cluster 2024-09-10T18:57:05.992549+0000 mgr.y (mgr.24526) 321 : cluster [DBG] pgmap v388: 228 pgs: 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.055132+0000 mon.a (mon.0) 1782 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: cluster 2024-09-10T18:57:06.055279+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.057062+0000 mon.c (mon.2) 429 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.059029+0000 mon.a (mon.0) 1784 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.060774+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.83:0/1020925830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.065182+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49043 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.068193+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.072357+0000 mon.a (mon.0) 1786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[32381]: audit 2024-09-10T18:57:06.784129+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.503476+0000 mon.c (mon.2) 425 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.782784+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.870296+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.83:0/2517598000' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.933622+0000 mon.a (mon.0) 1780 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.934574+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:05.940143+0000 mon.a (mon.0) 1781 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: cluster 2024-09-10T18:57:05.992549+0000 mgr.y (mgr.24526) 321 : cluster [DBG] pgmap v388: 228 pgs: 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.055132+0000 mon.a (mon.0) 1782 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi083-77640-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: cluster 2024-09-10T18:57:06.055279+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-09-10T18:57:07.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.057062+0000 mon.c (mon.2) 429 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.059029+0000 mon.a (mon.0) 1784 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.060774+0000 mon.c (mon.2) 430 : audit [INF] from='client.? v1:172.21.15.83:0/1020925830' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.065182+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49043 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.068193+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.072357+0000 mon.a (mon.0) 1786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:07.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:06 smithi083 bash[18629]: audit 2024-09-10T18:57:06.784129+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:07.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:07.425 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:07.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e47fff640 1 -- 172.21.15.83:0/2853297582 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v271) v1 ==== 97+0+0 (unknown 1548146969 0 0) 0x7f6e500548d0 con 0x7f6e6010b220 2024-09-10T18:57:07.503 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T18:57:07.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 10 librados: watch_flush enter 2024-09-10T18:57:07.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 10 librados: watch_flush exit 2024-09-10T18:57:07.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 20 client.49055.objecter shutdown clearing up homeless session... 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 10 client.49055.objecter successfully canceled tick 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 >> v1:172.21.15.83:6800/3071793735 conn(0x7f6e480716e0 legacy=0x7f6e48073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.500+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 >> v1:172.21.15.83:6790/0 conn(0x7f6e6010b220 legacy=0x7f6e6011c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.504+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 shutdown_connections 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.504+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 >> 172.21.15.83:0/2853297582 conn(0x7f6e60075290 msgr2=0x7f6e600f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.504+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 shutdown_connections 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.504+0000 7f6e661db640 1 -- 172.21.15.83:0/2853297582 wait complete. 2024-09-10T18:57:07.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.504+0000 7f6e661db640 1 librados: shutdown 2024-09-10T18:57:07.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:07.560 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:07.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:07.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:07.699 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:07.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.780+0000 7f4f78845640 1 -- 172.21.15.83:0/2924078231 >> v1:172.21.15.83:6790/0 conn(0x7f4f741075b0 legacy=0x7f4f74109a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- 172.21.15.83:0/2924078231 shutdown_connections 2024-09-10T18:57:07.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- 172.21.15.83:0/2924078231 >> 172.21.15.83:0/2924078231 conn(0x7f4f74074a20 msgr2=0x7f4f740f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:07.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- 172.21.15.83:0/2924078231 shutdown_connections 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- 172.21.15.83:0/2924078231 wait complete. 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 librados: starting msgr at 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 librados: starting objecter 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 Processor -- start 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- start start 2024-09-10T18:57:07.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 librados: setting wanted keys 2024-09-10T18:57:07.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 librados: calling monclient init 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f74100520 con 0x7f4f741075b0 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f741006d0 con 0x7f4f74103a50 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f7259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f4f74100c70 0x7f4f740fc840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59848/0 (socket says 172.21.15.83:59848) 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f7259c640 1 -- 172.21.15.83:0/2892718786 learned_addr learned my addr 172.21.15.83:0/2892718786 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f7411c380 con 0x7f4f74100c70 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter ms_handle_connect 0x7f4f74100c70 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828107750 0 0) 0x7f4f7411c380 con 0x7f4f74100c70 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f5c0034e0 con 0x7f4f74100c70 2024-09-10T18:57:07.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter ms_handle_connect 0x7f4f741075b0 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter ms_handle_connect 0x7f4f74103a50 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859168297 0 0) 0x7f4f74100520 con 0x7f4f741075b0 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f7411c380 con 0x7f4f741075b0 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392590542 0 0) 0x7f4f741006d0 con 0x7f4f74103a50 2024-09-10T18:57:07.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74100520 con 0x7f4f74103a50 2024-09-10T18:57:07.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2553808904 0 0) 0x7f4f5c0034e0 con 0x7f4f74100c70 2024-09-10T18:57:07.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f741006d0 con 0x7f4f74100c70 2024-09-10T18:57:07.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4f64001940 con 0x7f4f74100c70 2024-09-10T18:57:07.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1901618304 0 0) 0x7f4f7411c380 con 0x7f4f741075b0 2024-09-10T18:57:07.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f5c0034e0 con 0x7f4f741075b0 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4f6c0017b0 con 0x7f4f741075b0 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619239706 0 0) 0x7f4f74100520 con 0x7f4f74103a50 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f7411c380 con 0x7f4f74103a50 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4f68001990 con 0x7f4f74103a50 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470493469 0 0) 0x7f4f741006d0 con 0x7f4f74100c70 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.165:6789/0 conn(0x7f4f74103a50 legacy=0x7f4f740fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6789/0 conn(0x7f4f741075b0 legacy=0x7f4f74118c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.784+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7411d580 con 0x7f4f74100c70 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7411c570 con 0x7f4f74100c70 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f78845640 10 client.49061.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7411ca40 con 0x7f4f74100c70 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f78845640 1 librados: init done 2024-09-10T18:57:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4f64001cd0 con 0x7f4f74100c70 2024-09-10T18:57:07.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4f64005b60 con 0x7f4f74100c70 2024-09-10T18:57:07.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4f54004ff0 con 0x7f4f74100c70 2024-09-10T18:57:07.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4f6401bf80 con 0x7f4f74100c70 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(271..271 src has 69..271) v4 ==== 9026+0+0 (unknown 264083904 0 0) 0x7f4f6408a930 con 0x7f4f74100c70 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 osd_map(271..271 src has 69..271) v4 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 3 client.49061.objecter handle_osd_map got epochs [271,271] > 0 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.788+0000 7f4f62ffd640 3 client.49061.objecter handle_osd_map decoding full epoch 271 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 20 client.49061.objecter dump_active .. 0 homeless 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v271) v1 ==== 54+0+16 (unknown 3541901219 0 2420000192) 0x7f4f6408ace0 con 0x7f4f74100c70 2024-09-10T18:57:07.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f5c073a30 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 10 client.49061.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 1 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 1 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 0 1 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 4 2024-09-10T18:57:07.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 10 client.49061.objecter _send_command 1 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f54004a60 con 0x7f4f54004610 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f54004610 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f54004a60 con 0x7f4f54004610 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 1 = system:0 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 4 2024-09-10T18:57:07.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.792+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 0 1 2024-09-10T18:57:07.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 10 client.49061.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 2 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 2 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54005290 osd=0 3 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 0 2 2024-09-10T18:57:07.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 4 2024-09-10T18:57:07.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 10 client.49061.objecter _send_command 2 2024-09-10T18:57:07.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.808+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f54004e40 con 0x7f4f54004610 2024-09-10T18:57:07.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f54004e40 con 0x7f4f54004610 2024-09-10T18:57:07.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 4 2024-09-10T18:57:07.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 0 2 2024-09-10T18:57:07.815 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 10 client.49061.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 1 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 4 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 10 client.49061.objecter _send_command 3 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f540022f0 con 0x7f4f54004110 2024-09-10T18:57:07.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f54004110 2024-09-10T18:57:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f540022f0 con 0x7f4f54004110 2024-09-10T18:57:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:57:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 3 = system:0 2024-09-10T18:57:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 4 2024-09-10T18:57:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.812+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 1 3 2024-09-10T18:57:07.827 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:07.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 10 client.49061.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 4 2024-09-10T18:57:07.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 4 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54003fa0 osd=1 3 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 1 4 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 4 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 10 client.49061.objecter _send_command 4 2024-09-10T18:57:07.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.828+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f540018f0 con 0x7f4f54004110 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f540018f0 con 0x7f4f54004110 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 4 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 1 4 2024-09-10T18:57:07.835 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:57:07.836 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:57:07.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 10 client.49061.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 5 2024-09-10T18:57:07.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 5 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 2 5 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 4 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 10 client.49061.objecter _send_command 5 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f54003dc0 con 0x7f4f5400d350 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f5400d350 2024-09-10T18:57:07.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f54003dc0 con 0x7f4f5400d350 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 5 = system:0 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 4 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.832+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 2 5 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 10 client.49061.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 6 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 6 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f540035e0 osd=2 3 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 2 6 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 4 2024-09-10T18:57:07.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 10 client.49061.objecter _send_command 6 2024-09-10T18:57:07.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.844+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f540030d0 con 0x7f4f5400d350 2024-09-10T18:57:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f540030d0 con 0x7f4f5400d350 2024-09-10T18:57:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 4 2024-09-10T18:57:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 2 6 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 10 client.49061.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 7 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 7 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 3 7 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 4 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 10 client.49061.objecter _send_command 7 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f5400d100 con 0x7f4f54011fd0 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.848+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f54011fd0 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.852+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f5400d100 con 0x7f4f54011fd0 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.852+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:57:07.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.852+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 7 = system:0 2024-09-10T18:57:07.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.852+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 4 2024-09-10T18:57:07.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.852+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 3 7 2024-09-10T18:57:07.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 10 client.49061.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 8 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 8 2024-09-10T18:57:07.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5400c990 osd=3 3 2024-09-10T18:57:07.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 3 8 2024-09-10T18:57:07.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 4 2024-09-10T18:57:07.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 10 client.49061.objecter _send_command 8 2024-09-10T18:57:07.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.868+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f54013320 con 0x7f4f54011fd0 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f54013320 con 0x7f4f54011fd0 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 4 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 3 8 2024-09-10T18:57:07.875 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 10 client.49061.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 9 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 9 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 4 9 2024-09-10T18:57:07.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 4 2024-09-10T18:57:07.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 10 client.49061.objecter _send_command 9 2024-09-10T18:57:07.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f5401a230 con 0x7f4f54016730 2024-09-10T18:57:07.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.872+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f54016730 2024-09-10T18:57:07.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.876+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f5401a230 con 0x7f4f54016730 2024-09-10T18:57:07.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.876+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:57:07.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.876+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 9 = system:0 2024-09-10T18:57:07.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.876+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 4 2024-09-10T18:57:07.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.876+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 4 9 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 10 client.49061.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 10 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 10 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54012da0 osd=4 3 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 4 10 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 4 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 10 client.49061.objecter _send_command 10 2024-09-10T18:57:07.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.892+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f5401a230 con 0x7f4f54016730 2024-09-10T18:57:07.901 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:57:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f5401a230 con 0x7f4f54016730 2024-09-10T18:57:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 4 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 4 10 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 10 client.49061.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 11 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 11 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 5 11 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 4 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 10 client.49061.objecter _send_command 11 2024-09-10T18:57:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.896+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f5401c550 con 0x7f4f5401a8b0 2024-09-10T18:57:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f5401a8b0 2024-09-10T18:57:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f5401c550 con 0x7f4f5401a8b0 2024-09-10T18:57:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:57:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 11 = system:0 2024-09-10T18:57:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 4 2024-09-10T18:57:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.900+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 5 11 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 10 client.49061.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 12 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 12 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5401a230 osd=5 3 2024-09-10T18:57:07.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 5 12 2024-09-10T18:57:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 4 2024-09-10T18:57:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 10 client.49061.objecter _send_command 12 2024-09-10T18:57:07.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.920+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f5401c550 con 0x7f4f5401a8b0 2024-09-10T18:57:07.928 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:57:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f5401c550 con 0x7f4f5401a8b0 2024-09-10T18:57:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 4 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 5 12 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 10 client.49061.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 13 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 13 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 6 13 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 4 2024-09-10T18:57:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 10 client.49061.objecter _send_command 13 2024-09-10T18:57:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f54026ea0 con 0x7f4f5401c820 2024-09-10T18:57:07.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:07.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.924+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f5401c820 2024-09-10T18:57:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.928+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f54026ea0 con 0x7f4f5401c820 2024-09-10T18:57:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.928+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:57:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.928+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 13 = system:0 2024-09-10T18:57:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.928+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 4 2024-09-10T18:57:07.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.928+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 6 13 2024-09-10T18:57:07.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:06.980670+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.83:0/2853297582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:07.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:06.981133+0000 mon.a (mon.0) 1787 : audit [INF] from='client.49055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:07.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: cluster 2024-09-10T18:57:07.049680+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:07.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.501555+0000 mon.a (mon.0) 1789 : audit [INF] from='client.49043 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:07.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.502068+0000 mon.a (mon.0) 1790 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:07.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.502322+0000 mon.a (mon.0) 1791 : audit [INF] from='client.49055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 10 client.49061.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 14 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 14 2024-09-10T18:57:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f5401c550 osd=6 3 2024-09-10T18:57:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 6 14 2024-09-10T18:57:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 4 2024-09-10T18:57:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 10 client.49061.objecter _send_command 14 2024-09-10T18:57:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.952+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f54012bd0 con 0x7f4f5401c820 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f54012bd0 con 0x7f4f5401c820 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 4 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 6 14 2024-09-10T18:57:07.961 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 10 client.49061.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 15 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 15 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 7 15 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 4 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 10 client.49061.objecter _send_command 15 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.956+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f54011b20 con 0x7f4f54011610 2024-09-10T18:57:07.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 10 client.49061.objecter ms_handle_connect 0x7f4f54011610 2024-09-10T18:57:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4f54011b20 con 0x7f4f54011610 2024-09-10T18:57:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:57:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 15 = system:0 2024-09-10T18:57:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 4 2024-09-10T18:57:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.960+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 7 15 2024-09-10T18:57:07.966 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:07.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 10 client.49061.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign -1 16 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 20 client.49061.objecter _get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 15 client.49061.objecter _session_command_op_remove -1 16 2024-09-10T18:57:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 20 client.49061.objecter get_session s=0x7f4f54012bd0 osd=7 3 2024-09-10T18:57:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 15 client.49061.objecter _session_command_op_assign 7 16 2024-09-10T18:57:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 4 2024-09-10T18:57:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 10 client.49061.objecter _send_command 16 2024-09-10T18:57:07.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4f54011b20 con 0x7f4f54011610 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f62ffd640 1 -- 172.21.15.83:0/2892718786 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4f54011b20 con 0x7f4f54011610 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f62ffd640 10 client.49061.objecter ms_dispatch 0x7f4f74000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f62ffd640 10 client.49061.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f62ffd640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 4 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.980+0000 7f4f62ffd640 15 client.49061.objecter _session_command_op_remove 7 16 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:57:07.986 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 librados: watch_flush enter 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 librados: watch_flush exit 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.0 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6801/2535527483 conn(0x7f4f54004610 legacy=0x7f4f54002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54005290 osd=0 1 2024-09-10T18:57:07.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.1 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6805/2309624637 conn(0x7f4f54004110 legacy=0x7f4f5400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54003fa0 osd=1 1 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.2 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6809/386094220 conn(0x7f4f5400d350 legacy=0x7f4f5400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f540035e0 osd=2 1 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.3 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6813/1279688021 conn(0x7f4f54011fd0 legacy=0x7f4f54012450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5400c990 osd=3 1 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.4 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.165:6800/3225515265 conn(0x7f4f54016730 legacy=0x7f4f54018b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012da0 osd=4 1 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.5 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.165:6804/571173777 conn(0x7f4f5401a8b0 legacy=0x7f4f5401ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401a230 osd=5 1 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.6 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.165:6808/1317922936 conn(0x7f4f5401c820 legacy=0x7f4f540255f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f5401c550 osd=6 1 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter close_session for osd.7 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.165:6812/1610151650 conn(0x7f4f54011610 legacy=0x7f4f54026ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter put_session s=0x7f4f54012bd0 osd=7 1 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 20 client.49061.objecter shutdown clearing up homeless session... 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 10 client.49061.objecter successfully canceled tick 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f72d9d640 1 -- 172.21.15.83:0/2892718786 reap_dead start 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6800/3071793735 conn(0x7f4f5c073a30 legacy=0x7f4f5c075ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f72d9d640 1 -- 172.21.15.83:0/2892718786 reap_dead start 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> v1:172.21.15.83:6790/0 conn(0x7f4f74100c70 legacy=0x7f4f740fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f72d9d640 1 -- 172.21.15.83:0/2892718786 reap_dead start 2024-09-10T18:57:07.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f72d9d640 1 -- 172.21.15.83:0/2892718786 reap_dead start 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f72d9d640 1 -- 172.21.15.83:0/2892718786 reap_dead start 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 shutdown_connections 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 >> 172.21.15.83:0/2892718786 conn(0x7f4f74074a20 msgr2=0x7f4f7410a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 shutdown_connections 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 -- 172.21.15.83:0/2892718786 wait complete. 2024-09-10T18:57:07.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:07.984+0000 7f4f78845640 1 librados: shutdown 2024-09-10T18:57:08.009 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T18:57:08.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:57:08.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:08.104 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:08.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:06.980670+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.83:0/2853297582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:06.981133+0000 mon.a (mon.0) 1787 : audit [INF] from='client.49055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: cluster 2024-09-10T18:57:07.049680+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.501555+0000 mon.a (mon.0) 1789 : audit [INF] from='client.49043 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.502068+0000 mon.a (mon.0) 1790 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.502322+0000 mon.a (mon.0) 1791 : audit [INF] from='client.49055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: cluster 2024-09-10T18:57:07.502507+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.785018+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:08.220 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.785821+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.786141+0000 mon.a (mon.0) 1793 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:07 smithi165 bash[22591]: audit 2024-09-10T18:57:07.793336+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? v1:172.21.15.83:0/2892718786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:08.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:08.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: cluster 2024-09-10T18:57:07.502507+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.785018+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.785821+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.786141+0000 mon.a (mon.0) 1793 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[32381]: audit 2024-09-10T18:57:07.793336+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? v1:172.21.15.83:0/2892718786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:06.980670+0000 mon.c (mon.2) 432 : audit [INF] from='client.? v1:172.21.15.83:0/2853297582' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:06.981133+0000 mon.a (mon.0) 1787 : audit [INF] from='client.49055 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: cluster 2024-09-10T18:57:07.049680+0000 mon.a (mon.0) 1788 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.501555+0000 mon.a (mon.0) 1789 : audit [INF] from='client.49043 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi083-77648-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.502068+0000 mon.a (mon.0) 1790 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.502322+0000 mon.a (mon.0) 1791 : audit [INF] from='client.49055 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:08.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: cluster 2024-09-10T18:57:07.502507+0000 mon.a (mon.0) 1792 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-09-10T18:57:08.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.785018+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:08.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.785821+0000 mon.c (mon.2) 434 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.786141+0000 mon.a (mon.0) 1793 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:57:08.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:07 smithi083 bash[18629]: audit 2024-09-10T18:57:07.793336+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? v1:172.21.15.83:0/2892718786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:08.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:08.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:08.513 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:08.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:08.646 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:08.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:08.782 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:08.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:08.919 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:08.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:09.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:09.057 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:09.072 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:07.777229+0000 mgr.y (mgr.24526) 322 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:09.073 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: cluster 2024-09-10T18:57:07.993294+0000 mgr.y (mgr.24526) 323 : cluster [DBG] pgmap v391: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:09.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:07.777229+0000 mgr.y (mgr.24526) 322 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: cluster 2024-09-10T18:57:07.993294+0000 mgr.y (mgr.24526) 323 : cluster [DBG] pgmap v391: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.063446+0000 mon.a (mon.0) 1794 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.063714+0000 mon.a (mon.0) 1795 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: cluster 2024-09-10T18:57:08.063877+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.064770+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.066354+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.074929+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.075620+0000 mon.a (mon.0) 1797 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.097012+0000 mon.b (mon.1) 224 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.100973+0000 mon.a (mon.0) 1798 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.110512+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.110895+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.111877+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.112194+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.112680+0000 mon.c (mon.2) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: audit 2024-09-10T18:57:08.112980+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:09 smithi165 bash[22591]: cluster 2024-09-10T18:57:09.020897+0000 mon.a (mon.0) 1802 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:09.192 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.063446+0000 mon.a (mon.0) 1794 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.063714+0000 mon.a (mon.0) 1795 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: cluster 2024-09-10T18:57:08.063877+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.064770+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.066354+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.074929+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.075620+0000 mon.a (mon.0) 1797 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.097012+0000 mon.b (mon.1) 224 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.100973+0000 mon.a (mon.0) 1798 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.110512+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.110895+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.111877+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.112194+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.112680+0000 mon.c (mon.2) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: audit 2024-09-10T18:57:08.112980+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[32381]: cluster 2024-09-10T18:57:09.020897+0000 mon.a (mon.0) 1802 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:07.777229+0000 mgr.y (mgr.24526) 322 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: cluster 2024-09-10T18:57:07.993294+0000 mgr.y (mgr.24526) 323 : cluster [DBG] pgmap v391: 292 pgs: 64 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.063446+0000 mon.a (mon.0) 1794 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi083-77640-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:09.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.063714+0000 mon.a (mon.0) 1795 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: cluster 2024-09-10T18:57:08.063877+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.064770+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.066354+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.074929+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.075620+0000 mon.a (mon.0) 1797 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.097012+0000 mon.b (mon.1) 224 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.100973+0000 mon.a (mon.0) 1798 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.110512+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.110895+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.111877+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.112194+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:09.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.112680+0000 mon.c (mon.2) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: audit 2024-09-10T18:57:08.112980+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:09.294 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:09 smithi083 bash[18629]: cluster 2024-09-10T18:57:09.020897+0000 mon.a (mon.0) 1802 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:09.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.331 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:09.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.469 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:09.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:09.601 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:09.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:09.742 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:09.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:09.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:09.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:09.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:09.901 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 20, 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 600, 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [ 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24404.0:87 4.0 4:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:22.050694+0000", 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 467.84473234000001, 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016360855000000001, 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.902 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24404", 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2284258763", 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 87 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050694+0000", 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.903 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050694+0000", 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050692+0000", 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.904 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050699+0000", 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.7570000000000004e-06 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050710+0000", 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0433e-05 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.905 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050717+0000", 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8820000000000003e-06 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.050999+0000", 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00028208299999999998 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051007+0000", 2024-09-10T18:57:09.906 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.9000000000000006e-06 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063436+0000", 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012429739 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063616+0000", 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00017962100000000001 2024-09-10T18:57:09.907 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 0,7", 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063726+0000", 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010985299999999999 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.064148+0000", 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00042238300000000002 2024-09-10T18:57:09.908 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.066272+0000", 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021238680000000001 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.066987+0000", 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00071509000000000004 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.909 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.067025+0000", 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8186000000000003e-05 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.067055+0000", 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.9204e-05 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.910 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:98 4.0 4:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:49:22.051624+0000", 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 467.84380168600001, 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.015318925000000001, 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T18:57:09.911 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 98 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051624+0000", 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051624+0000", 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.912 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051623+0000", 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999986 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.913 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051630+0000", 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.0899999999999999e-06 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051643+0000", 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3372e-05 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.914 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051652+0000", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.2979999999999992e-06 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051757+0000", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000105349 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.051764+0000", 2024-09-10T18:57:09.915 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7279999999999998e-06 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.062971+0000", 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011207142 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063065+0000", 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.4207000000000001e-05 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.916 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 0,7", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063179+0000", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000114089 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.063941+0000", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00076218800000000001 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.065587+0000", 2024-09-10T18:57:09.917 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001645281 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.066856+0000", 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0012696859999999999 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.066908+0000", 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.1768999999999999e-05 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.918 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:49:22.066943+0000", 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.5349999999999999e-05 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.25402.0:10 60.1a e144/139 60:58f8f05b:test-rados-api-smithi083-77907-7::foo:head v 144'6, mlcod=142'3)", 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:54:58.374682+0000", 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 131.52074403699999, 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011969221, 2024-09-10T18:57:09.919 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374682+0000", 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374682+0000", 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.920 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374682+0000", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374687+0000", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6629999999999998e-06 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374693+0000", 2024-09-10T18:57:09.921 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.9159999999999998e-06 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.374700+0000", 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.4019999999999999e-06 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.375052+0000", 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00035222699999999999 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.922 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.375072+0000", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9627e-05 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.386619+0000", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011546898999999999 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:54:58.386651+0000", 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.2759999999999998e-05 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.923 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.28627.0:3 66.c e146/143 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head v 146'1, mlcod=0'0)", 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:00.406030+0000", 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 129.48939638799999, 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011510092, 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.924 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406030+0000", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406030+0000", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406029+0000", 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.999999 2024-09-10T18:57:09.925 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406038+0000", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.8009999999999998e-06 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406049+0000", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0971e-05 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.406057+0000", 2024-09-10T18:57:09.926 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.2880000000000004e-06 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.407933+0000", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0018766379999999999 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.407976+0000", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.2994000000000002e-05 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417500+0000", 2024-09-10T18:57:09.927 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0095235690000000008 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417540+0000", 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.9852999999999999e-05 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.30361.0:2 69.16 e146/145 69:68f45781:::oid_SimpleWritePP_:head v 146'1, mlcod=0'0)", 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:00.417647+0000", 2024-09-10T18:57:09.928 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 129.47777877799999, 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.021410669, 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417647+0000", 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417647+0000", 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.929 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417647+0000", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417657+0000", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0457e-05 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417663+0000", 2024-09-10T18:57:09.930 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6330000000000001e-06 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.417669+0000", 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.767e-06 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.418825+0000", 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001156444 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.931 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.418837+0000", 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2181000000000001e-05 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.439002+0000", 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.020164341999999998 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.439058+0000", 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6252999999999998e-05 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.932 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.30538.0:2 70.15 70:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:00.422131+0000", 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 129.47329544499999, 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.018678259999999999, 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.30538", 2024-09-10T18:57:09.933 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1004801166", 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422131+0000", 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422131+0000", 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.934 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422129+0000", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422138+0000", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.0159999999999997e-06 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422151+0000", 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3550000000000001e-05 2024-09-10T18:57:09.935 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422168+0000", 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.7047000000000001e-05 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422336+0000", 2024-09-10T18:57:09.936 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000167689 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422459+0000", 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000123003 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.422589+0000", 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013050799999999999 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.937 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.426528+0000", 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0039386409999999997 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.440710+0000", 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.014181885999999999 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.938 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.440717+0000", 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.3780000000000003e-06 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.440770+0000", 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.2664000000000003e-05 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.939 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.440809+0000", 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8949999999999998e-05 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.30538.0:4 70.15 70:ad551702:::test:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:00.447268+0000", 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 129.44815797699999, 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.014213753000000001, 2024-09-10T18:57:09.940 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.30538", 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1004801166", 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 4 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447268+0000", 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.941 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447268+0000", 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.942 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447266+0000", 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447274+0000", 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.3219999999999993e-06 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.943 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447288+0000", 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3672000000000001e-05 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447296+0000", 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8950000000000004e-06 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447345+0000", 2024-09-10T18:57:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8914e-05 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447433+0000", 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7861999999999993e-05 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 4,7", 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.447552+0000", 2024-09-10T18:57:09.945 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000118477 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.451515+0000", 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0039633209999999997 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.454355+0000", 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0028404469999999999 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.946 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.461383+0000", 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0070272850000000003 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.461440+0000", 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7568999999999997e-05 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.947 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:00.461482+0000", 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.1581000000000001e-05 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(osd.7.0:302 66.c e148/143 66:37fc6798:test-rados-api-smithi083-77943-8::foo:3 v 148'5, mlcod=146'1)", 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:03.499220+0000", 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 126.396206244, 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.014512208, 2024-09-10T18:57:09.948 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499220+0000", 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499220+0000", 2024-09-10T18:57:09.949 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499219+0000", 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.950 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499234+0000", 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4595e-05 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499240+0000", 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7459999999999997e-06 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.951 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499246+0000", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2179999999999996e-06 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499391+0000", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000145282 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.499410+0000", 2024-09-10T18:57:09.952 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8831e-05 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.513707+0000", 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.014296843 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.513732+0000", 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5103999999999999e-05 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.953 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(osd.2.0:172 60.b e148/139 60:d3e5ddd7:test-rados-api-smithi083-77907-8::foo:7 v 148'6, mlcod=147'3)", 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:03.518015+0000", 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 126.377410946, 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.015830969, 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518015+0000", 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.954 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518015+0000", 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518015+0000", 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.955 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518021+0000", 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.1310000000000003e-06 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518027+0000", 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7300000000000002e-06 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.518033+0000", 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.618e-06 2024-09-10T18:57:09.956 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.519111+0000", 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0010771349999999999 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.519131+0000", 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9984e-05 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.957 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.533821+0000", 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.014690498 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:03.533846+0000", 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5126999999999998e-05 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.958 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.28627.0:9 66.c e149/143 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head v 149'6, mlcod=147'3)", 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.503272+0000", 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 125.392154076, 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.010690333999999999, 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503272+0000", 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.959 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503272+0000", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503272+0000", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503278+0000", 2024-09-10T18:57:09.960 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.331e-06 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503287+0000", 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7080000000000003e-06 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503294+0000", 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8940000000000004e-06 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.961 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.504527+0000", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0012337870000000001 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.504547+0000", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9199999999999999e-05 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.513934+0000", 2024-09-10T18:57:09.962 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0093879159999999996 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.513962+0000", 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.792e-05 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.25402.0:19 60.b e149/139 60:d3e5ddd7:test-rados-api-smithi083-77907-8::foo:head v 149'7, mlcod=147'4)", 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.503495+0000", 2024-09-10T18:57:09.963 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 125.391931281, 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.021876761000000002, 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503495+0000", 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503495+0000", 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.964 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503494+0000", 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503509+0000", 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4335000000000001e-05 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.965 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503516+0000", 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.3039999999999999e-06 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.503521+0000", 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.7709999999999997e-06 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.512955+0000", 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.009433861 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.966 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.512976+0000", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.0806999999999999e-05 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.525344+0000", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012368601 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.525372+0000", 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.7484e-05 2024-09-10T18:57:09.967 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.35008.0:2 73.1f e149/148 73:fd8eaea7:nspace::oid_SimpleWritePP_:head v 149'1, mlcod=0'0)", 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.527476+0000", 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 125.36794999, 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016058308, 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.968 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527476+0000", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527476+0000", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527476+0000", 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.969 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527486+0000", 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.005e-05 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527491+0000", 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.378e-06 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.970 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.527497+0000", 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7130000000000002e-06 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.537382+0000", 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0098853220000000002 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.537399+0000", 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6430999999999999e-05 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.971 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.542989+0000", 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0055899699999999997 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.543534+0000", 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00054577200000000003 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.972 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:110 78.1 78:8731ae67:test-rados-api-smithi083-77828-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156)", 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:12.206901+0000", 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 117.688525333, 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.036265566999999999, 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 110 2024-09-10T18:57:09.973 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206901+0000", 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206901+0000", 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.974 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206899+0000", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206906+0000", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.5299999999999999e-06 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206918+0000", 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1939999999999999e-05 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.975 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.206925+0000", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7560000000000002e-06 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.207094+0000", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016877299999999999 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.207315+0000", 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00022086799999999999 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.976 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.232814+0000", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.025499404 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.232905+0000", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.0853000000000006e-05 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.238471+0000", 2024-09-10T18:57:09.977 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0055657730000000004 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.238563+0000", 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2020000000000003e-05 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243166+0000", 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0046036899999999997 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.978 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDECSubOpWrite(80.3s2 161/153 ECSubWrite(tid=192, reqid=osd.2.0:192, at_version=161'7, trim_to=0'0, roll_forward_to=160'5))", 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:16.994957+0000", 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 112.90046938499999, 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.13913235600000001, 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "reached pg", 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.994957+0000", 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.979 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.994957+0000", 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.994956+0000", 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.980 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.994995+0000", 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8824e-05 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.995004+0000", 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3060000000000003e-06 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.995011+0000", 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7240000000000003e-06 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.981 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.130433+0000", 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.13542227000000001 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.130446+0000", 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2999999999999999e-05 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:57:09.982 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.133142+0000", 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002695266 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.134089+0000", 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00094747400000000002 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24663.0:2 107.13 107:cbb9903e:::test_obj:head [writefull 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T18:57:09.983 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:42.059503+0000", 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 87.835922608999994, 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.028445076999999999, 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24663", 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2928542468", 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.984 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059503+0000", 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059503+0000", 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059501+0000", 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.985 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059510+0000", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.2220000000000002e-06 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059520+0000", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0708e-05 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059527+0000", 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.455e-06 2024-09-10T18:57:09.986 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059579+0000", 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.2206999999999997e-05 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.059587+0000", 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.2800000000000003e-06 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.987 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.071340+0000", 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011752534 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.071443+0000", 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010286200000000001 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 5,6", 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.071571+0000", 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00012857999999999999 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.988 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.074033+0000", 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0024619519999999999 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.076118+0000", 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0020844549999999998 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.989 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.087866+0000", 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011748177 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.087911+0000", 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5077999999999998e-05 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:42.087949+0000", 2024-09-10T18:57:09.990 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.7710999999999999e-05 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.48680.0:2 110.16 110:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:44.983105+0000", 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 84.912320945999994, 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.017541879, 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.991 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.48680", 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3486686660", 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983105+0000", 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.992 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983105+0000", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983103+0000", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983112+0000", 2024-09-10T18:57:09.993 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2599999999999994e-06 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983123+0000", 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1342000000000001e-05 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.983132+0000", 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.4530000000000002e-06 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.994 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.984539+0000", 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001406884 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.984547+0000", 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8829999999999995e-06 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.994522+0000", 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0099741900000000008 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.995 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.994646+0000", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00012467000000000001 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 5,7", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.995138+0000", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00049133900000000003 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.997168+0000", 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002029959 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.996 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:44.999371+0000", 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002203843 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:45.000465+0000", 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001093412 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:09.997 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:45.000608+0000", 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000143449 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:45.000647+0000", 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8714999999999998e-05 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.24732.0:2 118.11 e198/196 118:8e20cd32:test-rados-api-smithi083-77788-26::foo:head v 198'1, mlcod=0'0)", 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:54.003436+0000", 2024-09-10T18:57:09.998 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 75.891990312000004, 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.033438748999999997, 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.003436+0000", 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:09.999 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.003436+0000", 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.003434+0000", 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.027613+0000", 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.024179830999999999 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.000 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.027636+0000", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.2793999999999999e-05 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.027647+0000", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0626e-05 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.027704+0000", 2024-09-10T18:57:10.001 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7311000000000001e-05 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.027744+0000", 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.9915e-05 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036808+0000", 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0090643760000000007 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.002 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036875+0000", 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.6109e-05 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDECSubOpWrite(113.6s2 198/192 ECSubWrite(tid=504, reqid=client.48707.0:10, at_version=198'4, trim_to=0'0, roll_forward_to=198'3))", 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:54.009101+0000", 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 75.886325002000007, 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.013318767, 2024-09-10T18:57:10.003 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "reached pg", 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009101+0000", 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009101+0000", 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009099+0000", 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009131+0000", 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.1912999999999998e-05 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.005 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009145+0000", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4197e-05 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009157+0000", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2052000000000001e-05 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009214+0000", 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6833999999999998e-05 2024-09-10T18:57:10.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.009222+0000", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.5109999999999997e-06 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.022369+0000", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.013147290000000001 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.022420+0000", 2024-09-10T18:57:10.007 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.1041999999999997e-05 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.48707.0:13 113.4s0 113:2ce8ec92:test-rados-api-smithi083-77943-19::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:54.998324+0000", 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 74.897101733, 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016913634, 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:10.008 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.48707", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2936751915", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 13 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998324+0000", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998324+0000", 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998322+0000", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998333+0000", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1396999999999999e-05 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998347+0000", 2024-09-10T18:57:10.010 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3675e-05 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.998356+0000", 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.5310000000000007e-06 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.004257+0000", 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0059004979999999997 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.011 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.006101+0000", 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001843803 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:57:10.012 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.006768+0000", 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00066722799999999996 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.008920+0000", 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021521140000000001 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.015171+0000", 2024-09-10T18:57:10.013 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0062508040000000004 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:55.015238+0000", 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7340000000000002e-05 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.014 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24909.0:1 152.6s0 152:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:56:40.096957+0000", 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 29.798468902, 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.003376549, 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24909", 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1970984313", 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.015 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096957+0000", 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096957+0000", 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:57:10.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096954+0000", 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096964+0000", 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.5810000000000003e-06 2024-09-10T18:57:10.017 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096977+0000", 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2918999999999999e-05 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.096986+0000", 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.6889999999999993e-06 2024-09-10T18:57:10.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.097039+0000", 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.3338999999999999e-05 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:40.097043+0000", 2024-09-10T18:57:10.019 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.2760000000000002e-06 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.094867+0000", 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.997824033 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.094997+0000", 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000129441 2024-09-10T18:57:10.020 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.095321+0000", 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00032456199999999997 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.096604+0000", 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001283078 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.021 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.100285+0000", 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0036808570000000001 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:41.100334+0000", 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8610999999999999e-05 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:10.022 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.023 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:57:10.023 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:57:10.023 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:57:10.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:10.059 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:10.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:10.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:10.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:10.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.063364+0000 mon.a (mon.0) 1803 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:57:10.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.063669+0000 mon.a (mon.0) 1804 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:10.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.063762+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:10.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: cluster 2024-09-10T18:57:09.063851+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:57:10.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.065580+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.066285+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.066470+0000 mon.a (mon.0) 1807 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.076952+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.081475+0000 mon.a (mon.0) 1808 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.994933+0000 mon.c (mon.2) 444 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[18629]: audit 2024-09-10T18:57:09.997112+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:10.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T18:57:10.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.063364+0000 mon.a (mon.0) 1803 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.063669+0000 mon.a (mon.0) 1804 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.063762+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: cluster 2024-09-10T18:57:09.063851+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.065580+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.066285+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.066470+0000 mon.a (mon.0) 1807 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.076952+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:10.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.081475+0000 mon.a (mon.0) 1808 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.994933+0000 mon.c (mon.2) 444 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:10 smithi165 bash[22591]: audit 2024-09-10T18:57:09.997112+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.063364+0000 mon.a (mon.0) 1803 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.063669+0000 mon.a (mon.0) 1804 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.063762+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: cluster 2024-09-10T18:57:09.063851+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.065580+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.066285+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.802 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.066470+0000 mon.a (mon.0) 1807 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:10.802 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.076952+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:10.802 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.081475+0000 mon.a (mon.0) 1808 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:10.802 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.994933+0000 mon.c (mon.2) 444 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.802 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:10 smithi083 bash[32381]: audit 2024-09-10T18:57:09.997112+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T18:57:10.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:10.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.927 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:10.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:11.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:11.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:11.200 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:11.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:11.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.476 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:11.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:11.613 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:11.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:57:11.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:11.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: cluster 2024-09-10T18:57:09.994158+0000 mgr.y (mgr.24526) 324 : cluster [DBG] pgmap v394: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:11.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.078326+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.507536+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.509600+0000 mon.a (mon.0) 1810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.509871+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: cluster 2024-09-10T18:57:10.510107+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.511474+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.523731+0000 mon.a (mon.0) 1813 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:10.525851+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: cluster 2024-09-10T18:57:11.063768+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.075426+0000 mon.a (mon.0) 1816 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.075826+0000 mon.a (mon.0) 1817 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]': finished 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.075941+0000 mon.a (mon.0) 1818 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:11.731 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: cluster 2024-09-10T18:57:11.076037+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:57:11.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.079943+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.085153+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.732 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[22591]: audit 2024-09-10T18:57:11.085443+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.732 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:11.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: cluster 2024-09-10T18:57:09.994158+0000 mgr.y (mgr.24526) 324 : cluster [DBG] pgmap v394: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.078326+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.507536+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.509600+0000 mon.a (mon.0) 1810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.509871+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: cluster 2024-09-10T18:57:10.510107+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.511474+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.523731+0000 mon.a (mon.0) 1813 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:10.525851+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: cluster 2024-09-10T18:57:11.063768+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.075426+0000 mon.a (mon.0) 1816 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.075826+0000 mon.a (mon.0) 1817 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]': finished 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.075941+0000 mon.a (mon.0) 1818 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: cluster 2024-09-10T18:57:11.076037+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.079943+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.085153+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[32381]: audit 2024-09-10T18:57:11.085443+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: cluster 2024-09-10T18:57:09.994158+0000 mgr.y (mgr.24526) 324 : cluster [DBG] pgmap v394: 268 pgs: 40 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:11.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.078326+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.507536+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.509600+0000 mon.a (mon.0) 1810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.509871+0000 mon.a (mon.0) 1811 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: cluster 2024-09-10T18:57:10.510107+0000 mon.a (mon.0) 1812 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.511474+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.523731+0000 mon.a (mon.0) 1813 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]: dispatch 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:10.525851+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: cluster 2024-09-10T18:57:11.063768+0000 mon.a (mon.0) 1815 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.075426+0000 mon.a (mon.0) 1816 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.075826+0000 mon.a (mon.0) 1817 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-27", "mode": "writeback"}]': finished 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.075941+0000 mon.a (mon.0) 1818 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:11.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: cluster 2024-09-10T18:57:11.076037+0000 mon.a (mon.0) 1819 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-09-10T18:57:11.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.079943+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.83:0/2502095084' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.085153+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:11.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:11 smithi083 bash[18629]: audit 2024-09-10T18:57:11.085443+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49040 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]: dispatch 2024-09-10T18:57:11.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:57:11.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.885 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:11.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:11.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:12.015 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:12.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:12.138 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:12.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:12.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:12.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:12.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:12.549 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:12.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.686 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:12.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:12.828 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:12.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:12.945 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:12] "GET /metrics HTTP/1.1" 200 238314 "" "Prometheus/2.43.0" 2024-09-10T18:57:12.964 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:12.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:13.068 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:13.100 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:13.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:13.227 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:13.236 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: cluster 2024-09-10T18:57:11.995420+0000 mgr.y (mgr.24526) 325 : cluster [DBG] pgmap v397: 268 pgs: 8 unknown, 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.079857+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: cluster 2024-09-10T18:57:12.079991+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.090474+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.115277+0000 mon.c (mon.2) 450 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.115610+0000 mon.a (mon.0) 1823 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.116701+0000 mon.c (mon.2) 451 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.117006+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.117053+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.117468+0000 mon.c (mon.2) 452 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.117756+0000 mon.a (mon.0) 1825 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: audit 2024-09-10T18:57:12.120941+0000 mon.a (mon.0) 1826 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[32381]: cluster 2024-09-10T18:57:12.515454+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: cluster 2024-09-10T18:57:11.995420+0000 mgr.y (mgr.24526) 325 : cluster [DBG] pgmap v397: 268 pgs: 8 unknown, 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.079857+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: cluster 2024-09-10T18:57:12.079991+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.090474+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.115277+0000 mon.c (mon.2) 450 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.115610+0000 mon.a (mon.0) 1823 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.116701+0000 mon.c (mon.2) 451 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.117006+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.117053+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.117468+0000 mon.c (mon.2) 452 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.117756+0000 mon.a (mon.0) 1825 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: audit 2024-09-10T18:57:12.120941+0000 mon.a (mon.0) 1826 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:13 smithi083 bash[18629]: cluster 2024-09-10T18:57:12.515454+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:57:13.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: cluster 2024-09-10T18:57:11.995420+0000 mgr.y (mgr.24526) 325 : cluster [DBG] pgmap v397: 268 pgs: 8 unknown, 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.079857+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49040 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi083-77640-35"}]': finished 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: cluster 2024-09-10T18:57:12.079991+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.090474+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.115277+0000 mon.c (mon.2) 450 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.115610+0000 mon.a (mon.0) 1823 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.116701+0000 mon.c (mon.2) 451 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.117006+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.117053+0000 mon.b (mon.1) 227 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.117468+0000 mon.c (mon.2) 452 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.117756+0000 mon.a (mon.0) 1825 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: audit 2024-09-10T18:57:12.120941+0000 mon.a (mon.0) 1826 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:13.354 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:13 smithi165 bash[22591]: cluster 2024-09-10T18:57:12.515454+0000 mon.a (mon.0) 1827 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-10T18:57:13.374 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:13.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:13.508 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:13.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:13.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:13.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:13.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:13.915 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:13.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:14.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:14.063 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:14.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:14.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:14.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:14.337 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:14.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:14.474 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:14.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:14.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.084984+0000 mon.a (mon.0) 1828 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.085207+0000 mon.a (mon.0) 1829 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: cluster 2024-09-10T18:57:13.085298+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.086177+0000 mon.b (mon.1) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.086508+0000 mon.c (mon.2) 453 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.089541+0000 mon.a (mon.0) 1831 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.089706+0000 mon.c (mon.2) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.096601+0000 mon.a (mon.0) 1832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.721 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.097053+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:13.097128+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: cluster 2024-09-10T18:57:14.022447+0000 mon.a (mon.0) 1834 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: cluster 2024-09-10T18:57:14.024549+0000 mon.a (mon.0) 1835 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:14.722 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:14 smithi165 bash[22591]: audit 2024-09-10T18:57:14.482311+0000 mon.a (mon.0) 1836 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:14.738 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.084984+0000 mon.a (mon.0) 1828 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.085207+0000 mon.a (mon.0) 1829 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: cluster 2024-09-10T18:57:13.085298+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.086177+0000 mon.b (mon.1) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.086508+0000 mon.c (mon.2) 453 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.089541+0000 mon.a (mon.0) 1831 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.089706+0000 mon.c (mon.2) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.744 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.096601+0000 mon.a (mon.0) 1832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.097053+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:13.097128+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: cluster 2024-09-10T18:57:14.022447+0000 mon.a (mon.0) 1834 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: cluster 2024-09-10T18:57:14.024549+0000 mon.a (mon.0) 1835 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[32381]: audit 2024-09-10T18:57:14.482311+0000 mon.a (mon.0) 1836 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.084984+0000 mon.a (mon.0) 1828 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.085207+0000 mon.a (mon.0) 1829 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: cluster 2024-09-10T18:57:13.085298+0000 mon.a (mon.0) 1830 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-09-10T18:57:14.745 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.086177+0000 mon.b (mon.1) 228 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.086508+0000 mon.c (mon.2) 453 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.089541+0000 mon.a (mon.0) 1831 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.089706+0000 mon.c (mon.2) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.096601+0000 mon.a (mon.0) 1832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.097053+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:13.097128+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: cluster 2024-09-10T18:57:14.022447+0000 mon.a (mon.0) 1834 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: cluster 2024-09-10T18:57:14.024549+0000 mon.a (mon.0) 1835 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:14.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:14 smithi083 bash[18629]: audit 2024-09-10T18:57:14.482311+0000 mon.a (mon.0) 1836 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-27"}]': finished 2024-09-10T18:57:14.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:14.877 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:14.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:14.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.007 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:15.111 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:15.144 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:15.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:15.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:15.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.417 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:15.487 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:15.498 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: cluster 2024-09-10T18:57:13.521004+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: cluster 2024-09-10T18:57:13.521271+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: cluster 2024-09-10T18:57:13.996374+0000 mgr.y (mgr.24526) 326 : cluster [DBG] pgmap v400: 260 pgs: 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.098112+0000 mon.c (mon.2) 456 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.482513+0000 mon.a (mon.0) 1837 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: cluster 2024-09-10T18:57:14.482655+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.483724+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.490413+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.565975+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:14.568563+0000 mon.c (mon.2) 458 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:15.499 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.038650+0000 mon.a (mon.0) 1841 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: cluster 2024-09-10T18:57:13.521004+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: cluster 2024-09-10T18:57:13.521271+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: cluster 2024-09-10T18:57:13.996374+0000 mgr.y (mgr.24526) 326 : cluster [DBG] pgmap v400: 260 pgs: 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.098112+0000 mon.c (mon.2) 456 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.482513+0000 mon.a (mon.0) 1837 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: cluster 2024-09-10T18:57:14.482655+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.483724+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.490413+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.565975+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:14.568563+0000 mon.c (mon.2) 458 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.038650+0000 mon.a (mon.0) 1841 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.038798+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: cluster 2024-09-10T18:57:15.038935+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.074524+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.099111+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.471225+0000 mon.a (mon.0) 1844 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.472413+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.472990+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.501 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.473819+0000 mon.c (mon.2) 462 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.502 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[18629]: audit 2024-09-10T18:57:15.474380+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:15.556 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:15.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: cluster 2024-09-10T18:57:13.521004+0000 osd.6 (osd.6) 137 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: cluster 2024-09-10T18:57:13.521271+0000 osd.6 (osd.6) 138 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: cluster 2024-09-10T18:57:13.996374+0000 mgr.y (mgr.24526) 326 : cluster [DBG] pgmap v400: 260 pgs: 21 peering, 239 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.098112+0000 mon.c (mon.2) 456 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.482513+0000 mon.a (mon.0) 1837 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: cluster 2024-09-10T18:57:14.482655+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.483724+0000 mon.c (mon.2) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3732018111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.490413+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49067 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]: dispatch 2024-09-10T18:57:15.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.565975+0000 mon.a (mon.0) 1840 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:14.568563+0000 mon.c (mon.2) 458 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.038650+0000 mon.a (mon.0) 1841 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi083-77640-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.038798+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: cluster 2024-09-10T18:57:15.038935+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.074524+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.099111+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.471225+0000 mon.a (mon.0) 1844 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.472413+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.472990+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.473819+0000 mon.c (mon.2) 462 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.676 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:15 smithi165 bash[22591]: audit 2024-09-10T18:57:15.474380+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.688 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:15.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.038798+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49067 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-46"}]': finished 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: cluster 2024-09-10T18:57:15.038935+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.074524+0000 mon.c (mon.2) 459 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.099111+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.471225+0000 mon.a (mon.0) 1844 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.472413+0000 mon.c (mon.2) 461 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.472990+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.473819+0000 mon.c (mon.2) 462 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:15 smithi083 bash[32381]: audit 2024-09-10T18:57:15.474380+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:15.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.824 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:15.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:15.962 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:15.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:16.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:16.087 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:16.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:16.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:16.225 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:16.276 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:16.276 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:16.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:16.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:16.498 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:16.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: cluster 2024-09-10T18:57:15.998844+0000 mgr.y (mgr.24526) 327 : cluster [DBG] pgmap v403: 236 pgs: 8 creating+peering, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:15.999609+0000 mon.c (mon.2) 463 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.001938+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.042038+0000 mon.a (mon.0) 1848 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.042501+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: cluster 2024-09-10T18:57:16.042742+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.043565+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.044122+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.045621+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.049825+0000 mon.a (mon.0) 1852 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[18629]: audit 2024-09-10T18:57:16.100219+0000 mon.c (mon.2) 465 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: cluster 2024-09-10T18:57:15.998844+0000 mgr.y (mgr.24526) 327 : cluster [DBG] pgmap v403: 236 pgs: 8 creating+peering, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:15.999609+0000 mon.c (mon.2) 463 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.001938+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.042038+0000 mon.a (mon.0) 1848 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.042501+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: cluster 2024-09-10T18:57:16.042742+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.043565+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.044122+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.045621+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.049825+0000 mon.a (mon.0) 1852 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:16 smithi083 bash[32381]: audit 2024-09-10T18:57:16.100219+0000 mon.c (mon.2) 465 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:16.635 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:16.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.769 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: cluster 2024-09-10T18:57:15.998844+0000 mgr.y (mgr.24526) 327 : cluster [DBG] pgmap v403: 236 pgs: 8 creating+peering, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:15.999609+0000 mon.c (mon.2) 463 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.001938+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.042038+0000 mon.a (mon.0) 1848 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi083-77648-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.042501+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: cluster 2024-09-10T18:57:16.042742+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.043565+0000 mon.c (mon.2) 464 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.044122+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:16.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.045621+0000 mon.b (mon.1) 229 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.049825+0000 mon.a (mon.0) 1852 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:16.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:16 smithi165 bash[22591]: audit 2024-09-10T18:57:16.100219+0000 mon.c (mon.2) 465 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:16.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:16.905 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:16.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:17.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:17.048 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:17.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:17.182 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:17.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:17.324 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:17.428 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:17.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.593 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:17.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: cluster 2024-09-10T18:57:16.494713+0000 mon.a (mon.0) 1853 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: audit 2024-09-10T18:57:17.101384+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: audit 2024-09-10T18:57:17.487393+0000 mon.a (mon.0) 1854 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: cluster 2024-09-10T18:57:17.487562+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: audit 2024-09-10T18:57:17.489970+0000 mon.c (mon.2) 467 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.709 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[22591]: audit 2024-09-10T18:57:17.490849+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.729 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:17.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: cluster 2024-09-10T18:57:16.494713+0000 mon.a (mon.0) 1853 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-10T18:57:17.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: audit 2024-09-10T18:57:17.101384+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:17.745 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: audit 2024-09-10T18:57:17.487393+0000 mon.a (mon.0) 1854 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: cluster 2024-09-10T18:57:17.487562+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: audit 2024-09-10T18:57:17.489970+0000 mon.c (mon.2) 467 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[32381]: audit 2024-09-10T18:57:17.490849+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: cluster 2024-09-10T18:57:16.494713+0000 mon.a (mon.0) 1853 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: audit 2024-09-10T18:57:17.101384+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: audit 2024-09-10T18:57:17.487393+0000 mon.a (mon.0) 1854 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: cluster 2024-09-10T18:57:17.487562+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: audit 2024-09-10T18:57:17.489970+0000 mon.c (mon.2) 467 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.746 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:17 smithi083 bash[18629]: audit 2024-09-10T18:57:17.490849+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:17.833 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.866 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:17.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:17.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:18.001 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:18.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:18.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:18.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:18.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.274 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:18.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.404 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- 172.21.15.83:0/513207541 >> v1:172.21.15.83:6790/0 conn(0x7fd164075520 legacy=0x7fd1640779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- 172.21.15.83:0/513207541 shutdown_connections 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- 172.21.15.83:0/513207541 >> 172.21.15.83:0/513207541 conn(0x7fd16406b990 msgr2=0x7fd16406bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- 172.21.15.83:0/513207541 shutdown_connections 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- 172.21.15.83:0/513207541 wait complete. 2024-09-10T18:57:18.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 librados: starting msgr at 2024-09-10T18:57:18.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 librados: starting objecter 2024-09-10T18:57:18.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 Processor -- start 2024-09-10T18:57:18.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- start start 2024-09-10T18:57:18.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 librados: setting wanted keys 2024-09-10T18:57:18.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 librados: calling monclient init 2024-09-10T18:57:18.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1641084d0 con 0x7fd164075520 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1641086c0 con 0x7fd164071950 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd169795640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd164135a40 con 0x7fd164108bd0 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd1627fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd164075520 0x7fd16407f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:39538/0 (socket says 172.21.15.83:39538) 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd1637fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd164108bd0 0x7fd16407ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44438/0 (socket says 172.21.15.83:44438) 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd1627fc640 1 -- 172.21.15.83:0/3322922868 learned_addr learned my addr 172.21.15.83:0/3322922868 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter ms_handle_connect 0x7fd164075520 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter ms_handle_connect 0x7fd164108bd0 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter ms_handle_connect 0x7fd164071950 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:18.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124408849 0 0) 0x7fd164135a40 con 0x7fd164108bd0 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd14c0034c0 con 0x7fd164108bd0 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056296630 0 0) 0x7fd1641084d0 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164135a40 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965681726 0 0) 0x7fd1641086c0 con 0x7fd164071950 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1641084d0 con 0x7fd164071950 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000510999 0 0) 0x7fd164135a40 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1641086c0 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd154001960 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222414042 0 0) 0x7fd14c0034c0 con 0x7fd164108bd0 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd164135a40 con 0x7fd164108bd0 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd158002d90 con 0x7fd164108bd0 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.480+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 934163353 0 0) 0x7fd1641086c0 con 0x7fd164075520 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 >> v1:172.21.15.83:6790/0 conn(0x7fd164108bd0 legacy=0x7fd16407ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:18.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 >> v1:172.21.15.165:6789/0 conn(0x7fd164071950 legacy=0x7fd164107b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164136c60 con 0x7fd164075520 2024-09-10T18:57:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164135c30 con 0x7fd164075520 2024-09-10T18:57:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd169795640 10 client.25017.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:18.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164136080 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd169795640 1 librados: init done 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd1540020a0 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd154005af0 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd150005010 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd154001c40 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(282..282 src has 69..282) v4 ==== 9234+0+0 (unknown 99643597 0 0) 0x7fd15408ab50 con 0x7fd164075520 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 10 client.25017.objecter ms_dispatch 0x7fd164000b90 osd_map(282..282 src has 69..282) v4 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 3 client.25017.objecter handle_osd_map got epochs [282,282] > 0 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 3 client.25017.objecter handle_osd_map decoding full epoch 282 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 20 client.25017.objecter dump_active .. 0 homeless 2024-09-10T18:57:18.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.484+0000 7fd14b7fe640 10 client.25017.objecter ms_handle_connect 0x7fd14c075b90 2024-09-10T18:57:18.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.492+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd154056c60 con 0x7fd164075520 2024-09-10T18:57:18.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.534 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:18.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.664+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd150000b90 con 0x7fd14c075b90 2024-09-10T18:57:18.672 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:18.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.684+0000 7fd14b7fe640 1 -- 172.21.15.83:0/3322922868 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+717515 (unknown 2965378022 0 2001519484) 0x7fd150000b90 con 0x7fd14c075b90 2024-09-10T18:57:18.700 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:57:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 10 librados: watch_flush enter 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 10 librados: watch_flush exit 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 20 client.25017.objecter shutdown clearing up homeless session... 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 10 client.25017.objecter successfully canceled tick 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd14c075b90 legacy=0x7fd14c078040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 >> v1:172.21.15.83:6789/0 conn(0x7fd164075520 legacy=0x7fd16407f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 shutdown_connections 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 >> 172.21.15.83:0/3322922868 conn(0x7fd16406b990 msgr2=0x7fd16406bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 shutdown_connections 2024-09-10T18:57:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 -- 172.21.15.83:0/3322922868 wait complete. 2024-09-10T18:57:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:18.692+0000 7fd169795640 1 librados: shutdown 2024-09-10T18:57:18.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:18.813 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:18.917 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.950 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:18.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:19.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:17.784626+0000 mgr.y (mgr.24526) 328 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: cluster 2024-09-10T18:57:17.999672+0000 mgr.y (mgr.24526) 329 : cluster [DBG] pgmap v406: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:18.048491+0000 mon.a (mon.0) 1857 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:18.048844+0000 mon.a (mon.0) 1858 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: cluster 2024-09-10T18:57:18.048943+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:18.052894+0000 mon.c (mon.2) 468 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.081 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:18.053500+0000 mon.a (mon.0) 1860 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.082 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:18.102509+0000 mon.c (mon.2) 469 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:19.082 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: cluster 2024-09-10T18:57:19.026475+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:19.082 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: audit 2024-09-10T18:57:19.038376+0000 mon.a (mon.0) 1862 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.082 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:19 smithi165 bash[22591]: cluster 2024-09-10T18:57:19.038675+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:57:19.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:19.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:17.784626+0000 mgr.y (mgr.24526) 328 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: cluster 2024-09-10T18:57:17.999672+0000 mgr.y (mgr.24526) 329 : cluster [DBG] pgmap v406: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:18.048491+0000 mon.a (mon.0) 1857 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:18.048844+0000 mon.a (mon.0) 1858 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: cluster 2024-09-10T18:57:18.048943+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:18.052894+0000 mon.c (mon.2) 468 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.205 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:18.053500+0000 mon.a (mon.0) 1860 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:18.102509+0000 mon.c (mon.2) 469 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: cluster 2024-09-10T18:57:19.026475+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: audit 2024-09-10T18:57:19.038376+0000 mon.a (mon.0) 1862 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[18629]: cluster 2024-09-10T18:57:19.038675+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:17.784626+0000 mgr.y (mgr.24526) 328 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: cluster 2024-09-10T18:57:17.999672+0000 mgr.y (mgr.24526) 329 : cluster [DBG] pgmap v406: 260 pgs: 32 unknown, 228 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 12 op/s; 319 B/s, 4 objects/s recovering 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:18.048491+0000 mon.a (mon.0) 1857 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi083-77648-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:18.048844+0000 mon.a (mon.0) 1858 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: cluster 2024-09-10T18:57:18.048943+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:18.052894+0000 mon.c (mon.2) 468 : audit [INF] from='client.? v1:172.21.15.83:0/2994450680' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.206 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:18.053500+0000 mon.a (mon.0) 1860 : audit [INF] from='client.49073 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]: dispatch 2024-09-10T18:57:19.207 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:18.102509+0000 mon.c (mon.2) 469 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:19.207 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: cluster 2024-09-10T18:57:19.026475+0000 mon.a (mon.0) 1861 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:19.207 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: audit 2024-09-10T18:57:19.038376+0000 mon.a (mon.0) 1862 : audit [INF] from='client.49073 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi083-77640-36"}]': finished 2024-09-10T18:57:19.207 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:19 smithi083 bash[32381]: cluster 2024-09-10T18:57:19.038675+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-09-10T18:57:19.225 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:19.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:19.364 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:19.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:19.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:19.607 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:19.639 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:19.715 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:57:19.715 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:57:19.715 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T18:57:19.715 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:57:19.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:19.776 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:19.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:19.912 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:19.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:20.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:20.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:20.075 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-10T18:57:18.043643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599307,"num_objects":330,"num_object_clones":1,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":1427,"num_read_kb":1190,"num_write":1181,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":591612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2534,"ondisk_log_size":2534,"up":681,"acting":681,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":794,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2476336,"kb_used_data":34984,"kb_used_omap":13,"kb_used_meta":2441138,"kb_avail":747484880,"statfs":{"total":767960285184,"available":765424517120,"internally_reserved":0,"allocated":35823616,"data_stored":20286864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2499726016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":736,"num_objects":1,"num_object_clones":-4,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":6,"num_read_kb":5,"num_write":40,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010702"},"pg_stats":[{"pgid":"171.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"172.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:18.038656+0000","last_change":"2024-09-10T18:57:18.038656+0000","last_active":"2024-09-10T18:57:18.038656+0000","last_peered":"2024-09-10T18:57:18.038656+0000","last_clean":"2024-09-10T18:57:18.038656+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:18.038656+0000","last_undegraded":"2024-09-10T18:57:18.038656+0000","last_fullsized":"2024-09-10T18:57:18.038656+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:18.038656+0000","last_clean_scrub_stamp":"2024-09-10T18:57:18.038656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"171.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:16.030814+0000","last_change":"2024-09-10T18:57:16.030814+0000","last_active":"2024-09-10T18:57:16.030814+0000","last_peered":"2024-09-10T18:57:16.030814+0000","last_clean":"2024-09-10T18:57:16.030814+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:16.030814+0000","last_undegraded":"2024-09-10T18:57:16.030814+0000","last_fullsized":"2024-09-10T18:57:16.030814+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:16.030814+0000","last_clean_scrub_stamp":"2024-09-10T18:57:16.030814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":212,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045014+0000","last_change":"2024-09-10T18:57:12.088164+0000","last_active":"2024-09-10T18:57:14.045014+0000","last_peered":"2024-09-10T18:57:14.045014+0000","last_clean":"2024-09-10T18:57:14.045014+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:57:14.045014+0000","last_undegraded":"2024-09-10T18:57:14.045014+0000","last_fullsized":"2024-09-10T18:57:14.045014+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001429844,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1e","version":"257'56","reported_seq":232,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043021+0000","last_change":"2024-09-10T18:57:12.087749+0000","last_active":"2024-09-10T18:57:14.043021+0000","last_peered":"2024-09-10T18:57:14.043021+0000","last_clean":"2024-09-10T18:57:14.043021+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:57:14.043021+0000","last_undegraded":"2024-09-10T18:57:14.043021+0000","last_fullsized":"2024-09-10T18:57:14.043021+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013064730000000001,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1d","version":"276'22","reported_seq":207,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499051+0000","last_change":"2024-09-10T18:57:14.088852+0000","last_active":"2024-09-10T18:57:14.499051+0000","last_peered":"2024-09-10T18:57:14.499051+0000","last_clean":"2024-09-10T18:57:14.499051+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:57:14.499051+0000","last_undegraded":"2024-09-10T18:57:14.499051+0000","last_fullsized":"2024-09-10T18:57:14.499051+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.66291335799999995,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1c","version":"244'51","reported_seq":230,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089093+0000","last_change":"2024-09-10T18:57:12.101919+0000","last_active":"2024-09-10T18:57:14.089093+0000","last_peered":"2024-09-10T18:57:14.089093+0000","last_clean":"2024-09-10T18:57:14.089093+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:57:14.089093+0000","last_undegraded":"2024-09-10T18:57:14.089093+0000","last_fullsized":"2024-09-10T18:57:14.089093+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.052158+0000","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.0015668699999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1b","version":"240'68","reported_seq":235,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088680+0000","last_change":"2024-09-10T18:57:12.101541+0000","last_active":"2024-09-10T18:57:14.088680+0000","last_peered":"2024-09-10T18:57:14.088680+0000","last_clean":"2024-09-10T18:57:14.088680+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:57:14.088680+0000","last_undegraded":"2024-09-10T18:57:14.088680+0000","last_fullsized":"2024-09-10T18:57:14.088680+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001356674,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":211,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498228+0000","last_change":"2024-09-10T18:57:12.082611+0000","last_active":"2024-09-10T18:57:14.498228+0000","last_peered":"2024-09-10T18:57:14.498228+0000","last_clean":"2024-09-10T18:57:14.498228+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:57:14.498228+0000","last_undegraded":"2024-09-10T18:57:14.498228+0000","last_fullsized":"2024-09-10T18:57:14.498228+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031669459999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.19","version":"257'22","reported_seq":205,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.038794+0000","last_change":"2024-09-10T18:57:12.083160+0000","last_active":"2024-09-10T18:57:14.038794+0000","last_peered":"2024-09-10T18:57:14.038794+0000","last_clean":"2024-09-10T18:57:14.038794+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:57:14.038794+0000","last_undegraded":"2024-09-10T18:57:14.038794+0000","last_fullsized":"2024-09-10T18:57:14.038794+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001622898,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.18","version":"254'104","reported_seq":268,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041927+0000","last_change":"2024-09-10T18:57:12.080852+0000","last_active":"2024-09-10T18:57:14.041927+0000","last_peered":"2024-09-10T18:57:14.041927+0000","last_clean":"2024-09-10T18:57:14.041927+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:57:14.041927+0000","last_undegraded":"2024-09-10T18:57:14.041927+0000","last_fullsized":"2024-09-10T18:57:14.041927+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049831699999999997,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.17","version":"230'17","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505775+0000","last_change":"2024-09-10T18:57:12.096526+0000","last_active":"2024-09-10T18:57:14.505775+0000","last_peered":"2024-09-10T18:57:14.505775+0000","last_clean":"2024-09-10T18:57:14.505775+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:57:14.505775+0000","last_undegraded":"2024-09-10T18:57:14.505775+0000","last_fullsized":"2024-09-10T18:57:14.505775+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040168770000000003,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.16","version":"276'22","reported_seq":202,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.104706+0000","last_change":"2024-09-10T18:57:14.104706+0000","last_active":"2024-09-10T18:57:14.104706+0000","last_peered":"2024-09-10T18:57:14.104706+0000","last_clean":"2024-09-10T18:57:14.104706+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:57:14.104706+0000","last_undegraded":"2024-09-10T18:57:14.104706+0000","last_fullsized":"2024-09-10T18:57:14.104706+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01497361,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.15","version":"240'74","reported_seq":243,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497035+0000","last_change":"2024-09-10T18:57:12.094250+0000","last_active":"2024-09-10T18:57:14.497035+0000","last_peered":"2024-09-10T18:57:14.497035+0000","last_clean":"2024-09-10T18:57:14.497035+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:57:14.497035+0000","last_undegraded":"2024-09-10T18:57:14.497035+0000","last_fullsized":"2024-09-10T18:57:14.497035+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020446689999999998,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.14","version":"246'95","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089102+0000","last_change":"2024-09-10T18:57:12.102146+0000","last_active":"2024-09-10T18:57:14.089102+0000","last_peered":"2024-09-10T18:57:14.089102+0000","last_clean":"2024-09-10T18:57:14.089102+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:57:14.089102+0000","last_undegraded":"2024-09-10T18:57:14.089102+0000","last_fullsized":"2024-09-10T18:57:14.089102+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001757273,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.13","version":"146'1","reported_seq":184,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.494668+0000","last_change":"2024-09-10T18:57:12.085898+0000","last_active":"2024-09-10T18:57:14.494668+0000","last_peered":"2024-09-10T18:57:14.494668+0000","last_clean":"2024-09-10T18:57:14.494668+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:57:14.494668+0000","last_undegraded":"2024-09-10T18:57:14.494668+0000","last_fullsized":"2024-09-10T18:57:14.494668+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089313700000000001,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.12","version":"248'51","reported_seq":220,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042325+0000","last_change":"2024-09-10T18:57:12.082570+0000","last_active":"2024-09-10T18:57:14.042325+0000","last_peered":"2024-09-10T18:57:14.042325+0000","last_clean":"2024-09-10T18:57:14.042325+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:57:14.042325+0000","last_undegraded":"2024-09-10T18:57:14.042325+0000","last_fullsized":"2024-09-10T18:57:14.042325+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014724779999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.11","version":"226'51","reported_seq":226,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029049+0000","last_change":"2024-09-10T18:57:12.075224+0000","last_active":"2024-09-10T18:57:14.029049+0000","last_peered":"2024-09-10T18:57:14.029049+0000","last_clean":"2024-09-10T18:57:14.029049+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:57:14.029049+0000","last_undegraded":"2024-09-10T18:57:14.029049+0000","last_fullsized":"2024-09-10T18:57:14.029049+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024845969999999998,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.10","version":"224'34","reported_seq":208,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.036782+0000","last_change":"2024-09-10T18:57:12.082312+0000","last_active":"2024-09-10T18:57:14.036782+0000","last_peered":"2024-09-10T18:57:14.036782+0000","last_clean":"2024-09-10T18:57:14.036782+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:57:14.036782+0000","last_undegraded":"2024-09-10T18:57:14.036782+0000","last_fullsized":"2024-09-10T18:57:14.036782+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031434950000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.f","version":"276'63","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.103644+0000","last_change":"2024-09-10T18:57:14.103644+0000","last_active":"2024-09-10T18:57:14.103644+0000","last_peered":"2024-09-10T18:57:14.103644+0000","last_clean":"2024-09-10T18:57:14.103644+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:57:14.103644+0000","last_undegraded":"2024-09-10T18:57:14.103644+0000","last_fullsized":"2024-09-10T18:57:14.103644+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.059560770999999998,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.e","version":"246'119","reported_seq":280,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505725+0000","last_change":"2024-09-10T18:57:12.096269+0000","last_active":"2024-09-10T18:57:14.505725+0000","last_peered":"2024-09-10T18:57:14.505725+0000","last_clean":"2024-09-10T18:57:14.505725+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:57:14.505725+0000","last_undegraded":"2024-09-10T18:57:14.505725+0000","last_fullsized":"2024-09-10T18:57:14.505725+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040092849999999996,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.d","version":"243'18","reported_seq":211,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031493+0000","last_change":"2024-09-10T18:57:12.075689+0000","last_active":"2024-09-10T18:57:14.031493+0000","last_peered":"2024-09-10T18:57:14.031493+0000","last_clean":"2024-09-10T18:57:14.031493+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:57:14.031493+0000","last_undegraded":"2024-09-10T18:57:14.031493+0000","last_fullsized":"2024-09-10T18:57:14.031493+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.276748+0000","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.00088231900000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.c","version":"252'22","reported_seq":206,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042737+0000","last_change":"2024-09-10T18:57:12.086788+0000","last_active":"2024-09-10T18:57:14.042737+0000","last_peered":"2024-09-10T18:57:14.042737+0000","last_clean":"2024-09-10T18:57:14.042737+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:57:14.042737+0000","last_undegraded":"2024-09-10T18:57:14.042737+0000","last_fullsized":"2024-09-10T18:57:14.042737+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000435341,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.b","version":"247'35","reported_seq":211,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088653+0000","last_change":"2024-09-10T18:57:12.102105+0000","last_active":"2024-09-10T18:57:14.088653+0000","last_peered":"2024-09-10T18:57:14.088653+0000","last_clean":"2024-09-10T18:57:14.088653+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:57:14.088653+0000","last_undegraded":"2024-09-10T18:57:14.088653+0000","last_fullsized":"2024-09-10T18:57:14.088653+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018049310000000001,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.a","version":"232'17","reported_seq":199,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495761+0000","last_change":"2024-09-10T18:57:12.099079+0000","last_active":"2024-09-10T18:57:14.495761+0000","last_peered":"2024-09-10T18:57:14.495761+0000","last_clean":"2024-09-10T18:57:14.495761+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:57:14.495761+0000","last_undegraded":"2024-09-10T18:57:14.495761+0000","last_fullsized":"2024-09-10T18:57:14.495761+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059614639999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":182,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495487+0000","last_change":"2024-09-10T18:57:12.100522+0000","last_active":"2024-09-10T18:57:14.495487+0000","last_peered":"2024-09-10T18:57:14.495487+0000","last_clean":"2024-09-10T18:57:14.495487+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:57:14.495487+0000","last_undegraded":"2024-09-10T18:57:14.495487+0000","last_fullsized":"2024-09-10T18:57:14.495487+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070255990000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.8","version":"234'51","reported_seq":226,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.502973+0000","last_change":"2024-09-10T18:57:12.090483+0000","last_active":"2024-09-10T18:57:14.502973+0000","last_peered":"2024-09-10T18:57:14.502973+0000","last_clean":"2024-09-10T18:57:14.502973+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:57:14.502973+0000","last_undegraded":"2024-09-10T18:57:14.502973+0000","last_fullsized":"2024-09-10T18:57:14.502973+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098166500000000005,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.7","version":"276'158","reported_seq":323,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497720+0000","last_change":"2024-09-10T18:57:14.106035+0000","last_active":"2024-09-10T18:57:14.497720+0000","last_peered":"2024-09-10T18:57:14.497720+0000","last_clean":"2024-09-10T18:57:14.497720+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:57:14.497720+0000","last_undegraded":"2024-09-10T18:57:14.497720+0000","last_fullsized":"2024-09-10T18:57:14.497720+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0179852030000001,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.6","version":"244'85","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499521+0000","last_change":"2024-09-10T18:57:12.087056+0000","last_active":"2024-09-10T18:57:14.499521+0000","last_peered":"2024-09-10T18:57:14.499521+0000","last_clean":"2024-09-10T18:57:14.499521+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:57:14.499521+0000","last_undegraded":"2024-09-10T18:57:14.499521+0000","last_fullsized":"2024-09-10T18:57:14.499521+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001432352,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.5","version":"258'51","reported_seq":226,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044100+0000","last_change":"2024-09-10T18:57:12.085846+0000","last_active":"2024-09-10T18:57:14.044100+0000","last_peered":"2024-09-10T18:57:14.044100+0000","last_clean":"2024-09-10T18:57:14.044100+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:57:14.044100+0000","last_undegraded":"2024-09-10T18:57:14.044100+0000","last_fullsized":"2024-09-10T18:57:14.044100+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011614399999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.4","version":"257'107","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499965+0000","last_change":"2024-09-10T18:57:12.084966+0000","last_active":"2024-09-10T18:57:14.499965+0000","last_peered":"2024-09-10T18:57:14.499965+0000","last_clean":"2024-09-10T18:57:14.499965+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:57:14.499965+0000","last_undegraded":"2024-09-10T18:57:14.499965+0000","last_fullsized":"2024-09-10T18:57:14.499965+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035549599999999998,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.3","version":"236'102","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041513+0000","last_change":"2024-09-10T18:57:12.086641+0000","last_active":"2024-09-10T18:57:14.041513+0000","last_peered":"2024-09-10T18:57:14.041513+0000","last_clean":"2024-09-10T18:57:14.041513+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:57:14.041513+0000","last_undegraded":"2024-09-10T18:57:14.041513+0000","last_fullsized":"2024-09-10T18:57:14.041513+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017589960000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.2","version":"230'34","reported_seq":208,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.503965+0000","last_change":"2024-09-10T18:57:12.094964+0000","last_active":"2024-09-10T18:57:14.503965+0000","last_peered":"2024-09-10T18:57:14.503965+0000","last_clean":"2024-09-10T18:57:14.503965+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:57:14.503965+0000","last_undegraded":"2024-09-10T18:57:14.503965+0000","last_fullsized":"2024-09-10T18:57:14.503965+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030363909999999998,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.1","version":"263'72","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042309+0000","last_change":"2024-09-10T18:57:12.082640+0000","last_active":"2024-09-10T18:57:14.042309+0000","last_peered":"2024-09-10T18:57:14.042309+0000","last_clean":"2024-09-10T18:57:14.042309+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:57:14.042309+0000","last_undegraded":"2024-09-10T18:57:14.042309+0000","last_fullsized":"2024-09-10T18:57:14.042309+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014933259999999999,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"64.0","version":"246'102","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.494181+0000","last_change":"2024-09-10T18:57:12.089979+0000","last_active":"2024-09-10T18:57:14.494181+0000","last_peered":"2024-09-10T18:57:14.494181+0000","last_clean":"2024-09-10T18:57:14.494181+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:57:14.494181+0000","last_undegraded":"2024-09-10T18:57:14.494181+0000","last_fullsized":"2024-09-10T18:57:14.494181+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:56:44.683516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.683516+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.401562+0000","scrub_duration":0.0037708920000000001,"objects_trimmed":0,"snaptrim_duration":0.0020552420000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041179+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:57:14.041179+0000","last_peered":"2024-09-10T18:57:14.041179+0000","last_clean":"2024-09-10T18:57:14.041179+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:57:14.041179+0000","last_undegraded":"2024-09-10T18:57:14.041179+0000","last_fullsized":"2024-09-10T18:57:14.041179+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.508785+0000","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045412+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:57:14.045412+0000","last_peered":"2024-09-10T18:57:14.045412+0000","last_clean":"2024-09-10T18:57:14.045412+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:57:14.045412+0000","last_undegraded":"2024-09-10T18:57:14.045412+0000","last_fullsized":"2024-09-10T18:57:14.045412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":261,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043133+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:57:14.043133+0000","last_peered":"2024-09-10T18:57:14.043133+0000","last_clean":"2024-09-10T18:57:14.043133+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:57:14.043133+0000","last_undegraded":"2024-09-10T18:57:14.043133+0000","last_fullsized":"2024-09-10T18:57:14.043133+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.485909+0000","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039906+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:57:14.039906+0000","last_peered":"2024-09-10T18:57:14.039906+0000","last_clean":"2024-09-10T18:57:14.039906+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:57:14.039906+0000","last_undegraded":"2024-09-10T18:57:14.039906+0000","last_fullsized":"2024-09-10T18:57:14.039906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.646246+0000","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041999+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:57:14.041999+0000","last_peered":"2024-09-10T18:57:14.041999+0000","last_clean":"2024-09-10T18:57:14.041999+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:57:14.041999+0000","last_undegraded":"2024-09-10T18:57:14.041999+0000","last_fullsized":"2024-09-10T18:57:14.041999+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.539352+0000","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"243'6","reported_seq":19,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040230+0000","last_change":"2024-09-10T18:57:11.110363+0000","last_active":"2024-09-10T18:57:14.040230+0000","last_peered":"2024-09-10T18:57:14.040230+0000","last_clean":"2024-09-10T18:57:14.040230+0000","last_became_active":"2024-09-10T18:57:11.096232+0000","last_became_peered":"2024-09-10T18:57:11.096232+0000","last_unstale":"2024-09-10T18:57:14.040230+0000","last_undegraded":"2024-09-10T18:57:14.040230+0000","last_fullsized":"2024-09-10T18:57:14.040230+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'6","last_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_deep_scrub":"243'6","last_deep_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.791846+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:59.502541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":258,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043676+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:57:14.043676+0000","last_peered":"2024-09-10T18:57:14.043676+0000","last_clean":"2024-09-10T18:57:14.043676+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:57:14.043676+0000","last_undegraded":"2024-09-10T18:57:14.043676+0000","last_fullsized":"2024-09-10T18:57:14.043676+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045169+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:57:14.045169+0000","last_peered":"2024-09-10T18:57:14.045169+0000","last_clean":"2024-09-10T18:57:14.045169+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:57:14.045169+0000","last_undegraded":"2024-09-10T18:57:14.045169+0000","last_fullsized":"2024-09-10T18:57:14.045169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.632174+0000","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":264,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029094+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:57:14.029094+0000","last_peered":"2024-09-10T18:57:14.029094+0000","last_clean":"2024-09-10T18:57:14.029094+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:57:14.029094+0000","last_undegraded":"2024-09-10T18:57:14.029094+0000","last_fullsized":"2024-09-10T18:57:14.029094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.405877+0000","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030879+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:57:14.030879+0000","last_peered":"2024-09-10T18:57:14.030879+0000","last_clean":"2024-09-10T18:57:14.030879+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:57:14.030879+0000","last_undegraded":"2024-09-10T18:57:14.030879+0000","last_fullsized":"2024-09-10T18:57:14.030879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":162,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039399+0000","last_change":"2024-09-10T18:54:37.650137+0000","last_active":"2024-09-10T18:57:14.039399+0000","last_peered":"2024-09-10T18:57:14.039399+0000","last_clean":"2024-09-10T18:57:14.039399+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:57:14.039399+0000","last_undegraded":"2024-09-10T18:57:14.039399+0000","last_fullsized":"2024-09-10T18:57:14.039399+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040553+0000","last_change":"2024-09-10T18:57:11.095499+0000","last_active":"2024-09-10T18:57:14.040553+0000","last_peered":"2024-09-10T18:57:14.040553+0000","last_clean":"2024-09-10T18:57:14.040553+0000","last_became_active":"2024-09-10T18:57:11.094328+0000","last_became_peered":"2024-09-10T18:57:11.094328+0000","last_unstale":"2024-09-10T18:57:14.040553+0000","last_undegraded":"2024-09-10T18:57:14.040553+0000","last_fullsized":"2024-09-10T18:57:14.040553+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.362084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.362084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.428098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":255,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499047+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:57:14.499047+0000","last_peered":"2024-09-10T18:57:14.499047+0000","last_clean":"2024-09-10T18:57:14.499047+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:57:14.499047+0000","last_undegraded":"2024-09-10T18:57:14.499047+0000","last_fullsized":"2024-09-10T18:57:14.499047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:25.677268+0000","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043324+0000","last_change":"2024-09-10T18:54:41.188135+0000","last_active":"2024-09-10T18:57:14.043324+0000","last_peered":"2024-09-10T18:57:14.043324+0000","last_clean":"2024-09-10T18:57:14.043324+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:57:14.043324+0000","last_undegraded":"2024-09-10T18:57:14.043324+0000","last_fullsized":"2024-09-10T18:57:14.043324+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:54:41.188040+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.188040+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036526110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031470+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:57:14.031470+0000","last_peered":"2024-09-10T18:57:14.031470+0000","last_clean":"2024-09-10T18:57:14.031470+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:57:14.031470+0000","last_undegraded":"2024-09-10T18:57:14.031470+0000","last_fullsized":"2024-09-10T18:57:14.031470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.192711+0000","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":243,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042123+0000","last_change":"2024-09-10T18:56:32.263959+0000","last_active":"2024-09-10T18:57:14.042123+0000","last_peered":"2024-09-10T18:57:14.042123+0000","last_clean":"2024-09-10T18:57:14.042123+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:57:14.042123+0000","last_undegraded":"2024-09-10T18:57:14.042123+0000","last_fullsized":"2024-09-10T18:57:14.042123+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:39.242050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:39.242050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002953027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088821+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:57:14.088821+0000","last_peered":"2024-09-10T18:57:14.088821+0000","last_clean":"2024-09-10T18:57:14.088821+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:57:14.088821+0000","last_undegraded":"2024-09-10T18:57:14.088821+0000","last_fullsized":"2024-09-10T18:57:14.088821+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.799711+0000","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042712+0000","last_change":"2024-09-10T18:57:11.097093+0000","last_active":"2024-09-10T18:57:14.042712+0000","last_peered":"2024-09-10T18:57:14.042712+0000","last_clean":"2024-09-10T18:57:14.042712+0000","last_became_active":"2024-09-10T18:57:11.096680+0000","last_became_peered":"2024-09-10T18:57:11.096680+0000","last_unstale":"2024-09-10T18:57:14.042712+0000","last_undegraded":"2024-09-10T18:57:14.042712+0000","last_fullsized":"2024-09-10T18:57:14.042712+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500650+0000","last_change":"2024-09-10T18:56:22.517441+0000","last_active":"2024-09-10T18:57:14.500650+0000","last_peered":"2024-09-10T18:57:14.500650+0000","last_clean":"2024-09-10T18:57:14.500650+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:57:14.500650+0000","last_undegraded":"2024-09-10T18:57:14.500650+0000","last_fullsized":"2024-09-10T18:57:14.500650+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:52.078168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:52.078168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029805980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":292,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043638+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:57:14.043638+0000","last_peered":"2024-09-10T18:57:14.043638+0000","last_clean":"2024-09-10T18:57:14.043638+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:57:14.043638+0000","last_undegraded":"2024-09-10T18:57:14.043638+0000","last_fullsized":"2024-09-10T18:57:14.043638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.744708+0000","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031740+0000","last_change":"2024-09-10T18:54:55.309898+0000","last_active":"2024-09-10T18:57:14.031740+0000","last_peered":"2024-09-10T18:57:14.031740+0000","last_clean":"2024-09-10T18:57:14.031740+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:57:14.031740+0000","last_undegraded":"2024-09-10T18:57:14.031740+0000","last_fullsized":"2024-09-10T18:57:14.031740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.309689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.309689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003220512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037644+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:57:14.037644+0000","last_peered":"2024-09-10T18:57:14.037644+0000","last_clean":"2024-09-10T18:57:14.037644+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:57:14.037644+0000","last_undegraded":"2024-09-10T18:57:14.037644+0000","last_fullsized":"2024-09-10T18:57:14.037644+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042167+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:57:14.042167+0000","last_peered":"2024-09-10T18:57:14.042167+0000","last_clean":"2024-09-10T18:57:14.042167+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:57:14.042167+0000","last_undegraded":"2024-09-10T18:57:14.042167+0000","last_fullsized":"2024-09-10T18:57:14.042167+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.879768+0000","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505550+0000","last_change":"2024-09-10T18:57:11.089669+0000","last_active":"2024-09-10T18:57:14.505550+0000","last_peered":"2024-09-10T18:57:14.505550+0000","last_clean":"2024-09-10T18:57:14.505550+0000","last_became_active":"2024-09-10T18:57:11.089348+0000","last_became_peered":"2024-09-10T18:57:11.089348+0000","last_unstale":"2024-09-10T18:57:14.505550+0000","last_undegraded":"2024-09-10T18:57:14.505550+0000","last_fullsized":"2024-09-10T18:57:14.505550+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:20.858492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.504161+0000","last_change":"2024-09-10T18:54:50.687363+0000","last_active":"2024-09-10T18:57:14.504161+0000","last_peered":"2024-09-10T18:57:14.504161+0000","last_clean":"2024-09-10T18:57:14.504161+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:57:14.504161+0000","last_undegraded":"2024-09-10T18:57:14.504161+0000","last_fullsized":"2024-09-10T18:57:14.504161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:54:50.687166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.687166+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053731969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499051+0000","last_change":"2024-09-10T18:54:59.926609+0000","last_active":"2024-09-10T18:57:14.499051+0000","last_peered":"2024-09-10T18:57:14.499051+0000","last_clean":"2024-09-10T18:57:14.499051+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:57:14.499051+0000","last_undegraded":"2024-09-10T18:57:14.499051+0000","last_fullsized":"2024-09-10T18:57:14.499051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:59.926330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.926330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035578459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":360,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032538+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:57:14.032538+0000","last_peered":"2024-09-10T18:57:14.032538+0000","last_clean":"2024-09-10T18:57:14.032538+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:57:14.032538+0000","last_undegraded":"2024-09-10T18:57:14.032538+0000","last_fullsized":"2024-09-10T18:57:14.032538+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498031+0000","last_change":"2024-09-10T18:55:01.419421+0000","last_active":"2024-09-10T18:57:14.498031+0000","last_peered":"2024-09-10T18:57:14.498031+0000","last_clean":"2024-09-10T18:57:14.498031+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:57:14.498031+0000","last_undegraded":"2024-09-10T18:57:14.498031+0000","last_fullsized":"2024-09-10T18:57:14.498031+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.419333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.419333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035926959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505216+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:57:14.505216+0000","last_peered":"2024-09-10T18:57:14.505216+0000","last_clean":"2024-09-10T18:57:14.505216+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:57:14.505216+0000","last_undegraded":"2024-09-10T18:57:14.505216+0000","last_fullsized":"2024-09-10T18:57:14.505216+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.697752+0000","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.087985+0000","last_change":"2024-09-10T18:57:11.087464+0000","last_active":"2024-09-10T18:57:14.087985+0000","last_peered":"2024-09-10T18:57:14.087985+0000","last_clean":"2024-09-10T18:57:14.087985+0000","last_became_active":"2024-09-10T18:57:11.087104+0000","last_became_peered":"2024-09-10T18:57:11.087104+0000","last_unstale":"2024-09-10T18:57:14.087985+0000","last_undegraded":"2024-09-10T18:57:14.087985+0000","last_fullsized":"2024-09-10T18:57:14.087985+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.988916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.988916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.112231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041610+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:57:14.041610+0000","last_peered":"2024-09-10T18:57:14.041610+0000","last_clean":"2024-09-10T18:57:14.041610+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:57:14.041610+0000","last_undegraded":"2024-09-10T18:57:14.041610+0000","last_fullsized":"2024-09-10T18:57:14.041610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.663745+0000","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089510+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:57:14.089510+0000","last_peered":"2024-09-10T18:57:14.089510+0000","last_clean":"2024-09-10T18:57:14.089510+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:57:14.089510+0000","last_undegraded":"2024-09-10T18:57:14.089510+0000","last_fullsized":"2024-09-10T18:57:14.089510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.015829+0000","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":264,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030886+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:57:14.030886+0000","last_peered":"2024-09-10T18:57:14.030886+0000","last_clean":"2024-09-10T18:57:14.030886+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:57:14.030886+0000","last_undegraded":"2024-09-10T18:57:14.030886+0000","last_fullsized":"2024-09-10T18:57:14.030886+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.670897+0000","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032363+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:57:14.032363+0000","last_peered":"2024-09-10T18:57:14.032363+0000","last_clean":"2024-09-10T18:57:14.032363+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:57:14.032363+0000","last_undegraded":"2024-09-10T18:57:14.032363+0000","last_fullsized":"2024-09-10T18:57:14.032363+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":163,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.504751+0000","last_change":"2024-09-10T18:54:37.600609+0000","last_active":"2024-09-10T18:57:14.504751+0000","last_peered":"2024-09-10T18:57:14.504751+0000","last_clean":"2024-09-10T18:57:14.504751+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:57:14.504751+0000","last_undegraded":"2024-09-10T18:57:14.504751+0000","last_fullsized":"2024-09-10T18:57:14.504751+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.18","version":"126'2","reported_seq":179,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039728+0000","last_change":"2024-09-10T18:57:11.108239+0000","last_active":"2024-09-10T18:57:14.039728+0000","last_peered":"2024-09-10T18:57:14.039728+0000","last_clean":"2024-09-10T18:57:14.039728+0000","last_became_active":"2024-09-10T18:57:11.086021+0000","last_became_peered":"2024-09-10T18:57:11.086021+0000","last_unstale":"2024-09-10T18:57:14.039728+0000","last_undegraded":"2024-09-10T18:57:14.039728+0000","last_fullsized":"2024-09-10T18:57:14.039728+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":281,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037901+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:57:14.037901+0000","last_peered":"2024-09-10T18:57:14.037901+0000","last_clean":"2024-09-10T18:57:14.037901+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:57:14.037901+0000","last_undegraded":"2024-09-10T18:57:14.037901+0000","last_fullsized":"2024-09-10T18:57:14.037901+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.658076+0000","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088345+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:57:14.088345+0000","last_peered":"2024-09-10T18:57:14.088345+0000","last_clean":"2024-09-10T18:57:14.088345+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:57:14.088345+0000","last_undegraded":"2024-09-10T18:57:14.088345+0000","last_fullsized":"2024-09-10T18:57:14.088345+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.935684+0000","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.087265+0000","last_change":"2024-09-10T18:56:39.629880+0000","last_active":"2024-09-10T18:57:14.087265+0000","last_peered":"2024-09-10T18:57:14.087265+0000","last_clean":"2024-09-10T18:57:14.087265+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:57:14.087265+0000","last_undegraded":"2024-09-10T18:57:14.087265+0000","last_fullsized":"2024-09-10T18:57:14.087265+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:39.629726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.629726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.147581+0000","scrub_duration":0.0034450290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":260,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043180+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:57:14.043180+0000","last_peered":"2024-09-10T18:57:14.043180+0000","last_clean":"2024-09-10T18:57:14.043180+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:57:14.043180+0000","last_undegraded":"2024-09-10T18:57:14.043180+0000","last_fullsized":"2024-09-10T18:57:14.043180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.752916+0000","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.090179+0000","last_change":"2024-09-10T18:56:17.674109+0000","last_active":"2024-09-10T18:57:14.090179+0000","last_peered":"2024-09-10T18:57:14.090179+0000","last_clean":"2024-09-10T18:57:14.090179+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:57:14.090179+0000","last_undegraded":"2024-09-10T18:57:14.090179+0000","last_fullsized":"2024-09-10T18:57:14.090179+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.673943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.673943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:22.872965+0000","scrub_duration":0.003315691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1b","version":"126'10","reported_seq":199,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042705+0000","last_change":"2024-09-10T18:57:11.095438+0000","last_active":"2024-09-10T18:57:14.042705+0000","last_peered":"2024-09-10T18:57:14.042705+0000","last_clean":"2024-09-10T18:57:14.042705+0000","last_became_active":"2024-09-10T18:57:11.083353+0000","last_became_peered":"2024-09-10T18:57:11.083353+0000","last_unstale":"2024-09-10T18:57:14.042705+0000","last_undegraded":"2024-09-10T18:57:14.042705+0000","last_fullsized":"2024-09-10T18:57:14.042705+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:35.237574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.237574+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.321698+0000","scrub_duration":0.0042776200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":294,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030014+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:57:14.030014+0000","last_peered":"2024-09-10T18:57:14.030014+0000","last_clean":"2024-09-10T18:57:14.030014+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:57:14.030014+0000","last_undegraded":"2024-09-10T18:57:14.030014+0000","last_fullsized":"2024-09-10T18:57:14.030014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.690508+0000","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":255,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498311+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:57:14.498311+0000","last_peered":"2024-09-10T18:57:14.498311+0000","last_clean":"2024-09-10T18:57:14.498311+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:57:14.498311+0000","last_undegraded":"2024-09-10T18:57:14.498311+0000","last_fullsized":"2024-09-10T18:57:14.498311+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:20.015637+0000","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":360,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032168+0000","last_change":"2024-09-10T18:54:59.278861+0000","last_active":"2024-09-10T18:57:14.032168+0000","last_peered":"2024-09-10T18:57:14.032168+0000","last_clean":"2024-09-10T18:57:14.032168+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:57:14.032168+0000","last_undegraded":"2024-09-10T18:57:14.032168+0000","last_fullsized":"2024-09-10T18:57:14.032168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:54:59.278694+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.278694+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038345380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043379+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:57:14.043379+0000","last_peered":"2024-09-10T18:57:14.043379+0000","last_clean":"2024-09-10T18:57:14.043379+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:57:14.043379+0000","last_undegraded":"2024-09-10T18:57:14.043379+0000","last_fullsized":"2024-09-10T18:57:14.043379+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.519413+0000","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089925+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:57:14.089925+0000","last_peered":"2024-09-10T18:57:14.089925+0000","last_clean":"2024-09-10T18:57:14.089925+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:57:14.089925+0000","last_undegraded":"2024-09-10T18:57:14.089925+0000","last_fullsized":"2024-09-10T18:57:14.089925+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.543952+0000","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"126'5","reported_seq":21,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495598+0000","last_change":"2024-09-10T18:57:11.101605+0000","last_active":"2024-09-10T18:57:14.495598+0000","last_peered":"2024-09-10T18:57:14.495598+0000","last_clean":"2024-09-10T18:57:14.495598+0000","last_became_active":"2024-09-10T18:57:11.090251+0000","last_became_peered":"2024-09-10T18:57:11.090251+0000","last_unstale":"2024-09-10T18:57:14.495598+0000","last_undegraded":"2024-09-10T18:57:14.495598+0000","last_fullsized":"2024-09-10T18:57:14.495598+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043519+0000","last_change":"2024-09-10T18:54:37.847063+0000","last_active":"2024-09-10T18:57:14.043519+0000","last_peered":"2024-09-10T18:57:14.043519+0000","last_clean":"2024-09-10T18:57:14.043519+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:57:14.043519+0000","last_undegraded":"2024-09-10T18:57:14.043519+0000","last_fullsized":"2024-09-10T18:57:14.043519+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:54:37.846984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:37.846984+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098700820000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.506320+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:57:14.506320+0000","last_peered":"2024-09-10T18:57:14.506320+0000","last_clean":"2024-09-10T18:57:14.506320+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:57:14.506320+0000","last_undegraded":"2024-09-10T18:57:14.506320+0000","last_fullsized":"2024-09-10T18:57:14.506320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029629+0000","last_change":"2024-09-10T18:54:52.331412+0000","last_active":"2024-09-10T18:57:14.029629+0000","last_peered":"2024-09-10T18:57:14.029629+0000","last_clean":"2024-09-10T18:57:14.029629+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:57:14.029629+0000","last_undegraded":"2024-09-10T18:57:14.029629+0000","last_fullsized":"2024-09-10T18:57:14.029629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:52.331111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:52.331111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034919479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":715,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T18:57:15.040484+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:57:15.040484+0000","last_peered":"2024-09-10T18:57:15.040484+0000","last_clean":"2024-09-10T18:57:15.040484+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:57:15.040484+0000","last_undegraded":"2024-09-10T18:57:15.040484+0000","last_fullsized":"2024-09-10T18:57:15.040484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.052626+0000","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498161+0000","last_change":"2024-09-10T18:55:07.836217+0000","last_active":"2024-09-10T18:57:14.498161+0000","last_peered":"2024-09-10T18:57:14.498161+0000","last_clean":"2024-09-10T18:57:14.498161+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:57:14.498161+0000","last_undegraded":"2024-09-10T18:57:14.498161+0000","last_fullsized":"2024-09-10T18:57:14.498161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:07.836145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:23.121359+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.836145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031616470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495385+0000","last_change":"2024-09-10T18:56:12.680019+0000","last_active":"2024-09-10T18:57:14.495385+0000","last_peered":"2024-09-10T18:57:14.495385+0000","last_clean":"2024-09-10T18:57:14.495385+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:57:14.495385+0000","last_undegraded":"2024-09-10T18:57:14.495385+0000","last_fullsized":"2024-09-10T18:57:14.495385+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:12.679938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:12.679938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.716839+0000","scrub_duration":0.002716511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":182,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042067+0000","last_change":"2024-09-10T18:57:11.094446+0000","last_active":"2024-09-10T18:57:14.042067+0000","last_peered":"2024-09-10T18:57:14.042067+0000","last_clean":"2024-09-10T18:57:14.042067+0000","last_became_active":"2024-09-10T18:57:11.093990+0000","last_became_peered":"2024-09-10T18:57:11.093990+0000","last_unstale":"2024-09-10T18:57:14.042067+0000","last_undegraded":"2024-09-10T18:57:14.042067+0000","last_fullsized":"2024-09-10T18:57:14.042067+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:44.189240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.189240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.841047+0000","scrub_duration":0.0036310299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":283,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045127+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:57:14.045127+0000","last_peered":"2024-09-10T18:57:14.045127+0000","last_clean":"2024-09-10T18:57:14.045127+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:57:14.045127+0000","last_undegraded":"2024-09-10T18:57:14.045127+0000","last_fullsized":"2024-09-10T18:57:14.045127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.286896+0000","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.090164+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:57:14.090164+0000","last_peered":"2024-09-10T18:57:14.090164+0000","last_clean":"2024-09-10T18:57:14.090164+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:57:14.090164+0000","last_undegraded":"2024-09-10T18:57:14.090164+0000","last_fullsized":"2024-09-10T18:57:14.090164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:59.107812+0000","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":349,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037537+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:57:14.037537+0000","last_peered":"2024-09-10T18:57:14.037537+0000","last_clean":"2024-09-10T18:57:14.037537+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:57:14.037537+0000","last_undegraded":"2024-09-10T18:57:14.037537+0000","last_fullsized":"2024-09-10T18:57:14.037537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.264937+0000","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029343+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:57:14.029343+0000","last_peered":"2024-09-10T18:57:14.029343+0000","last_clean":"2024-09-10T18:57:14.029343+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:57:14.029343+0000","last_undegraded":"2024-09-10T18:57:14.029343+0000","last_fullsized":"2024-09-10T18:57:14.029343+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:00.852108+0000","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030757+0000","last_change":"2024-09-10T18:54:46.264917+0000","last_active":"2024-09-10T18:57:14.030757+0000","last_peered":"2024-09-10T18:57:14.030757+0000","last_clean":"2024-09-10T18:57:14.030757+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:57:14.030757+0000","last_undegraded":"2024-09-10T18:57:14.030757+0000","last_fullsized":"2024-09-10T18:57:14.030757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:46.264846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:46.264846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027169630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":163,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037150+0000","last_change":"2024-09-10T18:54:37.726541+0000","last_active":"2024-09-10T18:57:14.037150+0000","last_peered":"2024-09-10T18:57:14.037150+0000","last_clean":"2024-09-10T18:57:14.037150+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:57:14.037150+0000","last_undegraded":"2024-09-10T18:57:14.037150+0000","last_fullsized":"2024-09-10T18:57:14.037150+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"126'2","reported_seq":30,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044406+0000","last_change":"2024-09-10T18:57:11.104522+0000","last_active":"2024-09-10T18:57:14.044406+0000","last_peered":"2024-09-10T18:57:14.044406+0000","last_clean":"2024-09-10T18:57:14.044406+0000","last_became_active":"2024-09-10T18:57:11.092794+0000","last_became_peered":"2024-09-10T18:57:11.092794+0000","last_unstale":"2024-09-10T18:57:14.044406+0000","last_undegraded":"2024-09-10T18:57:14.044406+0000","last_fullsized":"2024-09-10T18:57:14.044406+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:39.169459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.169459+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.829162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":409,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089882+0000","last_change":"2024-09-10T18:56:03.530880+0000","last_active":"2024-09-10T18:57:14.089882+0000","last_peered":"2024-09-10T18:57:14.089882+0000","last_clean":"2024-09-10T18:57:14.089882+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:57:14.089882+0000","last_undegraded":"2024-09-10T18:57:14.089882+0000","last_fullsized":"2024-09-10T18:57:14.089882+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:56:03.530741+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.530741+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049201490000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":255,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500520+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:57:14.500520+0000","last_peered":"2024-09-10T18:57:14.500520+0000","last_clean":"2024-09-10T18:57:14.500520+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:57:14.500520+0000","last_undegraded":"2024-09-10T18:57:14.500520+0000","last_fullsized":"2024-09-10T18:57:14.500520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.253426+0000","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":359,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040970+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:57:14.040970+0000","last_peered":"2024-09-10T18:57:14.040970+0000","last_clean":"2024-09-10T18:57:14.040970+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:57:14.040970+0000","last_undegraded":"2024-09-10T18:57:14.040970+0000","last_fullsized":"2024-09-10T18:57:14.040970+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":260,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.502308+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:57:14.502308+0000","last_peered":"2024-09-10T18:57:14.502308+0000","last_clean":"2024-09-10T18:57:14.502308+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:57:14.502308+0000","last_undegraded":"2024-09-10T18:57:14.502308+0000","last_fullsized":"2024-09-10T18:57:14.502308+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088083+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:57:14.088083+0000","last_peered":"2024-09-10T18:57:14.088083+0000","last_clean":"2024-09-10T18:57:14.088083+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:57:14.088083+0000","last_undegraded":"2024-09-10T18:57:14.088083+0000","last_fullsized":"2024-09-10T18:57:14.088083+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.876489+0000","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":163,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041821+0000","last_change":"2024-09-10T18:54:37.710475+0000","last_active":"2024-09-10T18:57:14.041821+0000","last_peered":"2024-09-10T18:57:14.041821+0000","last_clean":"2024-09-10T18:57:14.041821+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:57:14.041821+0000","last_undegraded":"2024-09-10T18:57:14.041821+0000","last_fullsized":"2024-09-10T18:57:14.041821+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"126'5","reported_seq":18,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044151+0000","last_change":"2024-09-10T18:57:11.102477+0000","last_active":"2024-09-10T18:57:14.044151+0000","last_peered":"2024-09-10T18:57:14.044151+0000","last_clean":"2024-09-10T18:57:14.044151+0000","last_became_active":"2024-09-10T18:57:11.096848+0000","last_became_peered":"2024-09-10T18:57:11.096848+0000","last_unstale":"2024-09-10T18:57:14.044151+0000","last_undegraded":"2024-09-10T18:57:14.044151+0000","last_fullsized":"2024-09-10T18:57:14.044151+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.790063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.790063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.819892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043975+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:57:14.043975+0000","last_peered":"2024-09-10T18:57:14.043975+0000","last_clean":"2024-09-10T18:57:14.043975+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:57:14.043975+0000","last_undegraded":"2024-09-10T18:57:14.043975+0000","last_fullsized":"2024-09-10T18:57:14.043975+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089965+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:57:14.089965+0000","last_peered":"2024-09-10T18:57:14.089965+0000","last_clean":"2024-09-10T18:57:14.089965+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:57:14.089965+0000","last_undegraded":"2024-09-10T18:57:14.089965+0000","last_fullsized":"2024-09-10T18:57:14.089965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.138878+0000","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.086316+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:57:14.086316+0000","last_peered":"2024-09-10T18:57:14.086316+0000","last_clean":"2024-09-10T18:57:14.086316+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:57:14.086316+0000","last_undegraded":"2024-09-10T18:57:14.086316+0000","last_fullsized":"2024-09-10T18:57:14.086316+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:29.391815+0000","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.038138+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:57:14.038138+0000","last_peered":"2024-09-10T18:57:14.038138+0000","last_clean":"2024-09-10T18:57:14.038138+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:57:14.038138+0000","last_undegraded":"2024-09-10T18:57:14.038138+0000","last_fullsized":"2024-09-10T18:57:14.038138+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.883147+0000","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495821+0000","last_change":"2024-09-10T18:56:13.649580+0000","last_active":"2024-09-10T18:57:14.495821+0000","last_peered":"2024-09-10T18:57:14.495821+0000","last_clean":"2024-09-10T18:57:14.495821+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:57:14.495821+0000","last_undegraded":"2024-09-10T18:57:14.495821+0000","last_fullsized":"2024-09-10T18:57:14.495821+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.649458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.649458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.256522+0000","scrub_duration":0.0027534759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"126'5","reported_seq":35,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.506047+0000","last_change":"2024-09-10T18:57:11.096564+0000","last_active":"2024-09-10T18:57:14.506047+0000","last_peered":"2024-09-10T18:57:14.506047+0000","last_clean":"2024-09-10T18:57:14.506047+0000","last_became_active":"2024-09-10T18:57:11.088957+0000","last_became_peered":"2024-09-10T18:57:11.088957+0000","last_unstale":"2024-09-10T18:57:14.506047+0000","last_undegraded":"2024-09-10T18:57:14.506047+0000","last_fullsized":"2024-09-10T18:57:14.506047+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":292,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032269+0000","last_change":"2024-09-10T18:55:15.131835+0000","last_active":"2024-09-10T18:57:14.032269+0000","last_peered":"2024-09-10T18:57:14.032269+0000","last_clean":"2024-09-10T18:57:14.032269+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:57:14.032269+0000","last_undegraded":"2024-09-10T18:57:14.032269+0000","last_fullsized":"2024-09-10T18:57:14.032269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:55:15.131693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:15.131693+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048796719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043198+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:57:14.043198+0000","last_peered":"2024-09-10T18:57:14.043198+0000","last_clean":"2024-09-10T18:57:14.043198+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:57:14.043198+0000","last_undegraded":"2024-09-10T18:57:14.043198+0000","last_fullsized":"2024-09-10T18:57:14.043198+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.086735+0000","last_change":"2024-09-10T18:56:04.495501+0000","last_active":"2024-09-10T18:57:14.086735+0000","last_peered":"2024-09-10T18:57:14.086735+0000","last_clean":"2024-09-10T18:57:14.086735+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:57:14.086735+0000","last_undegraded":"2024-09-10T18:57:14.086735+0000","last_fullsized":"2024-09-10T18:57:14.086735+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.495334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.495334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003611179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043022+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:57:14.043022+0000","last_peered":"2024-09-10T18:57:14.043022+0000","last_clean":"2024-09-10T18:57:14.043022+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:57:14.043022+0000","last_undegraded":"2024-09-10T18:57:14.043022+0000","last_fullsized":"2024-09-10T18:57:14.043022+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.503223+0000","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039453+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:57:14.039453+0000","last_peered":"2024-09-10T18:57:14.039453+0000","last_clean":"2024-09-10T18:57:14.039453+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:57:14.039453+0000","last_undegraded":"2024-09-10T18:57:14.039453+0000","last_fullsized":"2024-09-10T18:57:14.039453+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.215962+0000","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":17,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037377+0000","last_change":"2024-09-10T18:57:11.091444+0000","last_active":"2024-09-10T18:57:14.037377+0000","last_peered":"2024-09-10T18:57:14.037377+0000","last_clean":"2024-09-10T18:57:14.037377+0000","last_became_active":"2024-09-10T18:57:11.091111+0000","last_became_peered":"2024-09-10T18:57:11.091111+0000","last_unstale":"2024-09-10T18:57:14.037377+0000","last_undegraded":"2024-09-10T18:57:14.037377+0000","last_fullsized":"2024-09-10T18:57:14.037377+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.795811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.795811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.217254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":294,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.087823+0000","last_change":"2024-09-10T18:54:56.479766+0000","last_active":"2024-09-10T18:57:14.087823+0000","last_peered":"2024-09-10T18:57:14.087823+0000","last_clean":"2024-09-10T18:57:14.087823+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:57:14.087823+0000","last_undegraded":"2024-09-10T18:57:14.087823+0000","last_fullsized":"2024-09-10T18:57:14.087823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:54:56.479670+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.479670+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003399188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495329+0000","last_change":"2024-09-10T18:56:15.655135+0000","last_active":"2024-09-10T18:57:14.495329+0000","last_peered":"2024-09-10T18:57:14.495329+0000","last_clean":"2024-09-10T18:57:14.495329+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:57:14.495329+0000","last_undegraded":"2024-09-10T18:57:14.495329+0000","last_fullsized":"2024-09-10T18:57:14.495329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:15.655055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.655055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.233854+0000","scrub_duration":0.0020254439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.087011+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:57:14.087011+0000","last_peered":"2024-09-10T18:57:14.087011+0000","last_clean":"2024-09-10T18:57:14.087011+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:57:14.087011+0000","last_undegraded":"2024-09-10T18:57:14.087011+0000","last_fullsized":"2024-09-10T18:57:14.087011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.219400+0000","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499436+0000","last_change":"2024-09-10T18:56:06.517243+0000","last_active":"2024-09-10T18:57:14.499436+0000","last_peered":"2024-09-10T18:57:14.499436+0000","last_clean":"2024-09-10T18:57:14.499436+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:57:14.499436+0000","last_undegraded":"2024-09-10T18:57:14.499436+0000","last_fullsized":"2024-09-10T18:57:14.499436+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.517079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.517079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038953730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505868+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:57:14.505868+0000","last_peered":"2024-09-10T18:57:14.505868+0000","last_clean":"2024-09-10T18:57:14.505868+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:57:14.505868+0000","last_undegraded":"2024-09-10T18:57:14.505868+0000","last_fullsized":"2024-09-10T18:57:14.505868+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.237861+0000","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"126'9","reported_seq":29,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042361+0000","last_change":"2024-09-10T18:57:11.102250+0000","last_active":"2024-09-10T18:57:14.042361+0000","last_peered":"2024-09-10T18:57:14.042361+0000","last_clean":"2024-09-10T18:57:14.042361+0000","last_became_active":"2024-09-10T18:57:11.081753+0000","last_became_peered":"2024-09-10T18:57:11.081753+0000","last_unstale":"2024-09-10T18:57:14.042361+0000","last_undegraded":"2024-09-10T18:57:14.042361+0000","last_fullsized":"2024-09-10T18:57:14.042361+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:15.650681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.650681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.780843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.038281+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:57:14.038281+0000","last_peered":"2024-09-10T18:57:14.038281+0000","last_clean":"2024-09-10T18:57:14.038281+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:57:14.038281+0000","last_undegraded":"2024-09-10T18:57:14.038281+0000","last_fullsized":"2024-09-10T18:57:14.038281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":255,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497846+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:57:14.497846+0000","last_peered":"2024-09-10T18:57:14.497846+0000","last_clean":"2024-09-10T18:57:14.497846+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:57:14.497846+0000","last_undegraded":"2024-09-10T18:57:14.497846+0000","last_fullsized":"2024-09-10T18:57:14.497846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.405645+0000","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043605+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:57:14.043605+0000","last_peered":"2024-09-10T18:57:14.043605+0000","last_clean":"2024-09-10T18:57:14.043605+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:57:14.043605+0000","last_undegraded":"2024-09-10T18:57:14.043605+0000","last_fullsized":"2024-09-10T18:57:14.043605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089486+0000","last_change":"2024-09-10T18:55:19.296815+0000","last_active":"2024-09-10T18:57:14.089486+0000","last_peered":"2024-09-10T18:57:14.089486+0000","last_clean":"2024-09-10T18:57:14.089486+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:57:14.089486+0000","last_undegraded":"2024-09-10T18:57:14.089486+0000","last_fullsized":"2024-09-10T18:57:14.089486+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:19.296674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:55:19.296674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003182767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043679+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:57:14.043679+0000","last_peered":"2024-09-10T18:57:14.043679+0000","last_clean":"2024-09-10T18:57:14.043679+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:57:14.043679+0000","last_undegraded":"2024-09-10T18:57:14.043679+0000","last_fullsized":"2024-09-10T18:57:14.043679+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.407178+0000","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029625+0000","last_change":"2024-09-10T18:57:11.086152+0000","last_active":"2024-09-10T18:57:14.029625+0000","last_peered":"2024-09-10T18:57:14.029625+0000","last_clean":"2024-09-10T18:57:14.029625+0000","last_became_active":"2024-09-10T18:57:11.085788+0000","last_became_peered":"2024-09-10T18:57:11.085788+0000","last_unstale":"2024-09-10T18:57:14.029625+0000","last_undegraded":"2024-09-10T18:57:14.029625+0000","last_fullsized":"2024-09-10T18:57:14.029625+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042306+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:57:14.042306+0000","last_peered":"2024-09-10T18:57:14.042306+0000","last_clean":"2024-09-10T18:57:14.042306+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:57:14.042306+0000","last_undegraded":"2024-09-10T18:57:14.042306+0000","last_fullsized":"2024-09-10T18:57:14.042306+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.212086+0000","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043045+0000","last_change":"2024-09-10T18:55:03.233057+0000","last_active":"2024-09-10T18:57:14.043045+0000","last_peered":"2024-09-10T18:57:14.043045+0000","last_clean":"2024-09-10T18:57:14.043045+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:57:14.043045+0000","last_undegraded":"2024-09-10T18:57:14.043045+0000","last_fullsized":"2024-09-10T18:57:14.043045+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:03.232923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.232923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036625059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":303,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497992+0000","last_change":"2024-09-10T18:56:04.556751+0000","last_active":"2024-09-10T18:57:14.497992+0000","last_peered":"2024-09-10T18:57:14.497992+0000","last_clean":"2024-09-10T18:57:14.497992+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:57:14.497992+0000","last_undegraded":"2024-09-10T18:57:14.497992+0000","last_fullsized":"2024-09-10T18:57:14.497992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:56:04.556586+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.556586+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.342255+0000","scrub_duration":0.0046952210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.502284+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:57:14.502284+0000","last_peered":"2024-09-10T18:57:14.502284+0000","last_clean":"2024-09-10T18:57:14.502284+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:57:14.502284+0000","last_undegraded":"2024-09-10T18:57:14.502284+0000","last_fullsized":"2024-09-10T18:57:14.502284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043557+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:57:14.043557+0000","last_peered":"2024-09-10T18:57:14.043557+0000","last_clean":"2024-09-10T18:57:14.043557+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:57:14.043557+0000","last_undegraded":"2024-09-10T18:57:14.043557+0000","last_fullsized":"2024-09-10T18:57:14.043557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500094+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:57:14.500094+0000","last_peered":"2024-09-10T18:57:14.500094+0000","last_clean":"2024-09-10T18:57:14.500094+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:57:14.500094+0000","last_undegraded":"2024-09-10T18:57:14.500094+0000","last_fullsized":"2024-09-10T18:57:14.500094+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.745484+0000","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.11","version":"126'7","reported_seq":22,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.493637+0000","last_change":"2024-09-10T18:57:11.106310+0000","last_active":"2024-09-10T18:57:14.493637+0000","last_peered":"2024-09-10T18:57:14.493637+0000","last_clean":"2024-09-10T18:57:14.493637+0000","last_became_active":"2024-09-10T18:57:11.087842+0000","last_became_peered":"2024-09-10T18:57:11.087842+0000","last_unstale":"2024-09-10T18:57:14.493637+0000","last_undegraded":"2024-09-10T18:57:14.493637+0000","last_fullsized":"2024-09-10T18:57:14.493637+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:21.348988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.348988+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.354400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":281,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500325+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:57:14.500325+0000","last_peered":"2024-09-10T18:57:14.500325+0000","last_clean":"2024-09-10T18:57:14.500325+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:57:14.500325+0000","last_undegraded":"2024-09-10T18:57:14.500325+0000","last_fullsized":"2024-09-10T18:57:14.500325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.212518+0000","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497063+0000","last_change":"2024-09-10T18:54:58.446820+0000","last_active":"2024-09-10T18:57:14.497063+0000","last_peered":"2024-09-10T18:57:14.497063+0000","last_clean":"2024-09-10T18:57:14.497063+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:57:14.497063+0000","last_undegraded":"2024-09-10T18:57:14.497063+0000","last_fullsized":"2024-09-10T18:57:14.497063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.446707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.446707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011714456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.504149+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:57:14.504149+0000","last_peered":"2024-09-10T18:57:14.504149+0000","last_clean":"2024-09-10T18:57:14.504149+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:57:14.504149+0000","last_undegraded":"2024-09-10T18:57:14.504149+0000","last_fullsized":"2024-09-10T18:57:14.504149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498492+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:57:14.498492+0000","last_peered":"2024-09-10T18:57:14.498492+0000","last_clean":"2024-09-10T18:57:14.498492+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:57:14.498492+0000","last_undegraded":"2024-09-10T18:57:14.498492+0000","last_fullsized":"2024-09-10T18:57:14.498492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.916290+0000","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030350+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:57:14.030350+0000","last_peered":"2024-09-10T18:57:14.030350+0000","last_clean":"2024-09-10T18:57:14.030350+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:57:14.030350+0000","last_undegraded":"2024-09-10T18:57:14.030350+0000","last_fullsized":"2024-09-10T18:57:14.030350+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.438790+0000","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505442+0000","last_change":"2024-09-10T18:57:11.088938+0000","last_active":"2024-09-10T18:57:14.505442+0000","last_peered":"2024-09-10T18:57:14.505442+0000","last_clean":"2024-09-10T18:57:14.505442+0000","last_became_active":"2024-09-10T18:57:11.088572+0000","last_became_peered":"2024-09-10T18:57:11.088572+0000","last_unstale":"2024-09-10T18:57:14.505442+0000","last_undegraded":"2024-09-10T18:57:14.505442+0000","last_fullsized":"2024-09-10T18:57:14.505442+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:44.575356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.575356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.190115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043481+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:57:14.043481+0000","last_peered":"2024-09-10T18:57:14.043481+0000","last_clean":"2024-09-10T18:57:14.043481+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:57:14.043481+0000","last_undegraded":"2024-09-10T18:57:14.043481+0000","last_fullsized":"2024-09-10T18:57:14.043481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.198556+0000","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089825+0000","last_change":"2024-09-10T18:56:30.684729+0000","last_active":"2024-09-10T18:57:14.089825+0000","last_peered":"2024-09-10T18:57:14.089825+0000","last_clean":"2024-09-10T18:57:14.089825+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:57:14.089825+0000","last_undegraded":"2024-09-10T18:57:14.089825+0000","last_fullsized":"2024-09-10T18:57:14.089825+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.684589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.093891+0000","scrub_duration":0.0023375330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":243,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040762+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:57:14.040762+0000","last_peered":"2024-09-10T18:57:14.040762+0000","last_clean":"2024-09-10T18:57:14.040762+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:57:14.040762+0000","last_undegraded":"2024-09-10T18:57:14.040762+0000","last_fullsized":"2024-09-10T18:57:14.040762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":241,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.038121+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:57:14.038121+0000","last_peered":"2024-09-10T18:57:14.038121+0000","last_clean":"2024-09-10T18:57:14.038121+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:57:14.038121+0000","last_undegraded":"2024-09-10T18:57:14.038121+0000","last_fullsized":"2024-09-10T18:57:14.038121+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040866+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:57:14.040866+0000","last_peered":"2024-09-10T18:57:14.040866+0000","last_clean":"2024-09-10T18:57:14.040866+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:57:14.040866+0000","last_undegraded":"2024-09-10T18:57:14.040866+0000","last_fullsized":"2024-09-10T18:57:14.040866+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.383629+0000","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500026+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:57:14.500026+0000","last_peered":"2024-09-10T18:57:14.500026+0000","last_clean":"2024-09-10T18:57:14.500026+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:57:14.500026+0000","last_undegraded":"2024-09-10T18:57:14.500026+0000","last_fullsized":"2024-09-10T18:57:14.500026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.067234+0000","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":278,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.497592+0000","last_change":"2024-09-10T18:56:36.828365+0000","last_active":"2024-09-10T18:57:14.497592+0000","last_peered":"2024-09-10T18:57:14.497592+0000","last_clean":"2024-09-10T18:57:14.497592+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:57:14.497592+0000","last_undegraded":"2024-09-10T18:57:14.497592+0000","last_fullsized":"2024-09-10T18:57:14.497592+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:56:36.828207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.828207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.829596+0000","scrub_duration":0.0038610699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042093+0000","last_change":"2024-09-10T18:55:00.199821+0000","last_active":"2024-09-10T18:57:14.042093+0000","last_peered":"2024-09-10T18:57:14.042093+0000","last_clean":"2024-09-10T18:57:14.042093+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:57:14.042093+0000","last_undegraded":"2024-09-10T18:57:14.042093+0000","last_fullsized":"2024-09-10T18:57:14.042093+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.199571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.199571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036539939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042721+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:57:14.042721+0000","last_peered":"2024-09-10T18:57:14.042721+0000","last_clean":"2024-09-10T18:57:14.042721+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:57:14.042721+0000","last_undegraded":"2024-09-10T18:57:14.042721+0000","last_fullsized":"2024-09-10T18:57:14.042721+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.028818+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:57:14.028818+0000","last_peered":"2024-09-10T18:57:14.028818+0000","last_clean":"2024-09-10T18:57:14.028818+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:57:14.028818+0000","last_undegraded":"2024-09-10T18:57:14.028818+0000","last_fullsized":"2024-09-10T18:57:14.028818+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":14,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042410+0000","last_change":"2024-09-10T18:57:11.088532+0000","last_active":"2024-09-10T18:57:14.042410+0000","last_peered":"2024-09-10T18:57:14.042410+0000","last_clean":"2024-09-10T18:57:14.042410+0000","last_became_active":"2024-09-10T18:57:11.088183+0000","last_became_peered":"2024-09-10T18:57:11.088183+0000","last_unstale":"2024-09-10T18:57:14.042410+0000","last_undegraded":"2024-09-10T18:57:14.042410+0000","last_fullsized":"2024-09-10T18:57:14.042410+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.607936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.607936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:35.543153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499763+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:57:14.499763+0000","last_peered":"2024-09-10T18:57:14.499763+0000","last_clean":"2024-09-10T18:57:14.499763+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:57:14.499763+0000","last_undegraded":"2024-09-10T18:57:14.499763+0000","last_fullsized":"2024-09-10T18:57:14.499763+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.960254+0000","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":307,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.033524+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:57:14.033524+0000","last_peered":"2024-09-10T18:57:14.033524+0000","last_clean":"2024-09-10T18:57:14.033524+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:57:14.033524+0000","last_undegraded":"2024-09-10T18:57:14.033524+0000","last_fullsized":"2024-09-10T18:57:14.033524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.123931+0000","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":359,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.029858+0000","last_change":"2024-09-10T18:55:04.228572+0000","last_active":"2024-09-10T18:57:14.029858+0000","last_peered":"2024-09-10T18:57:14.029858+0000","last_clean":"2024-09-10T18:57:14.029858+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:57:14.029858+0000","last_undegraded":"2024-09-10T18:57:14.029858+0000","last_fullsized":"2024-09-10T18:57:14.029858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:04.228360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.228360+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039590529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041264+0000","last_change":"2024-09-10T18:55:01.757908+0000","last_active":"2024-09-10T18:57:14.041264+0000","last_peered":"2024-09-10T18:57:14.041264+0000","last_clean":"2024-09-10T18:57:14.041264+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:57:14.041264+0000","last_undegraded":"2024-09-10T18:57:14.041264+0000","last_fullsized":"2024-09-10T18:57:14.041264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.757807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.757807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011468136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":186,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.503748+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:57:14.503748+0000","last_peered":"2024-09-10T18:57:14.503748+0000","last_clean":"2024-09-10T18:57:14.503748+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:57:14.503748+0000","last_undegraded":"2024-09-10T18:57:14.503748+0000","last_fullsized":"2024-09-10T18:57:14.503748+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.921825+0000","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1d","version":"126'7","reported_seq":25,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040107+0000","last_change":"2024-09-10T18:57:11.107177+0000","last_active":"2024-09-10T18:57:14.040107+0000","last_peered":"2024-09-10T18:57:14.040107+0000","last_clean":"2024-09-10T18:57:14.040107+0000","last_became_active":"2024-09-10T18:57:11.086953+0000","last_became_peered":"2024-09-10T18:57:11.086953+0000","last_unstale":"2024-09-10T18:57:14.040107+0000","last_undegraded":"2024-09-10T18:57:14.040107+0000","last_fullsized":"2024-09-10T18:57:14.040107+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:04.761871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.761871+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498733+0000","last_change":"2024-09-10T18:55:05.902412+0000","last_active":"2024-09-10T18:57:14.498733+0000","last_peered":"2024-09-10T18:57:14.498733+0000","last_clean":"2024-09-10T18:57:14.498733+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:57:14.498733+0000","last_undegraded":"2024-09-10T18:57:14.498733+0000","last_fullsized":"2024-09-10T18:57:14.498733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:05.902154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.902154+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047960349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.494383+0000","last_change":"2024-09-10T18:54:49.456319+0000","last_active":"2024-09-10T18:57:14.494383+0000","last_peered":"2024-09-10T18:57:14.494383+0000","last_clean":"2024-09-10T18:57:14.494383+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:57:14.494383+0000","last_undegraded":"2024-09-10T18:57:14.494383+0000","last_fullsized":"2024-09-10T18:57:14.494383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.456167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.456167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002905819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043381+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:57:14.043381+0000","last_peered":"2024-09-10T18:57:14.043381+0000","last_clean":"2024-09-10T18:57:14.043381+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:57:14.043381+0000","last_undegraded":"2024-09-10T18:57:14.043381+0000","last_fullsized":"2024-09-10T18:57:14.043381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:23.259079+0000","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.496816+0000","last_change":"2024-09-10T18:55:02.390515+0000","last_active":"2024-09-10T18:57:14.496816+0000","last_peered":"2024-09-10T18:57:14.496816+0000","last_clean":"2024-09-10T18:57:14.496816+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:57:14.496816+0000","last_undegraded":"2024-09-10T18:57:14.496816+0000","last_fullsized":"2024-09-10T18:57:14.496816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.390417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.390417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002554265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032740+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:57:14.032740+0000","last_peered":"2024-09-10T18:57:14.032740+0000","last_clean":"2024-09-10T18:57:14.032740+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:57:14.032740+0000","last_undegraded":"2024-09-10T18:57:14.032740+0000","last_fullsized":"2024-09-10T18:57:14.032740+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.580079+0000","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.030186+0000","last_change":"2024-09-10T18:56:32.796094+0000","last_active":"2024-09-10T18:57:14.030186+0000","last_peered":"2024-09-10T18:57:14.030186+0000","last_clean":"2024-09-10T18:57:14.030186+0000","last_became_active":"2024-09-10T18:54:37.566860+0000","last_became_peered":"2024-09-10T18:54:37.566860+0000","last_unstale":"2024-09-10T18:57:14.030186+0000","last_undegraded":"2024-09-10T18:57:14.030186+0000","last_fullsized":"2024-09-10T18:57:14.030186+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.795927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.610135+0000","scrub_duration":0.0032690250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.503196+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:57:14.503196+0000","last_peered":"2024-09-10T18:57:14.503196+0000","last_clean":"2024-09-10T18:57:14.503196+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:57:14.503196+0000","last_undegraded":"2024-09-10T18:57:14.503196+0000","last_fullsized":"2024-09-10T18:57:14.503196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.504281+0000","last_change":"2024-09-10T18:54:49.637132+0000","last_active":"2024-09-10T18:57:14.504281+0000","last_peered":"2024-09-10T18:57:14.504281+0000","last_clean":"2024-09-10T18:57:14.504281+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:57:14.504281+0000","last_undegraded":"2024-09-10T18:57:14.504281+0000","last_fullsized":"2024-09-10T18:57:14.504281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.637024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.637024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029426489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045049+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:57:14.045049+0000","last_peered":"2024-09-10T18:57:14.045049+0000","last_clean":"2024-09-10T18:57:14.045049+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:57:14.045049+0000","last_undegraded":"2024-09-10T18:57:14.045049+0000","last_fullsized":"2024-09-10T18:57:14.045049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499417+0000","last_change":"2024-09-10T18:56:03.595788+0000","last_active":"2024-09-10T18:57:14.499417+0000","last_peered":"2024-09-10T18:57:14.499417+0000","last_clean":"2024-09-10T18:57:14.499417+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:57:14.499417+0000","last_undegraded":"2024-09-10T18:57:14.499417+0000","last_fullsized":"2024-09-10T18:57:14.499417+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.595643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.595643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.278652+0000","scrub_duration":0.0030518839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505736+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:57:14.505736+0000","last_peered":"2024-09-10T18:57:14.505736+0000","last_clean":"2024-09-10T18:57:14.505736+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:57:14.505736+0000","last_undegraded":"2024-09-10T18:57:14.505736+0000","last_fullsized":"2024-09-10T18:57:14.505736+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.043389+0000","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"126'5","reported_seq":178,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042570+0000","last_change":"2024-09-10T18:56:37.282686+0000","last_active":"2024-09-10T18:57:14.042570+0000","last_peered":"2024-09-10T18:57:14.042570+0000","last_clean":"2024-09-10T18:57:14.042570+0000","last_became_active":"2024-09-10T18:54:37.507583+0000","last_became_peered":"2024-09-10T18:54:37.507583+0000","last_unstale":"2024-09-10T18:57:14.042570+0000","last_undegraded":"2024-09-10T18:57:14.042570+0000","last_fullsized":"2024-09-10T18:57:14.042570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:37.282578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.282578+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.873328+0000","scrub_duration":0.0030887559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":277,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498791+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:57:14.498791+0000","last_peered":"2024-09-10T18:57:14.498791+0000","last_clean":"2024-09-10T18:57:14.498791+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:57:14.498791+0000","last_undegraded":"2024-09-10T18:57:14.498791+0000","last_fullsized":"2024-09-10T18:57:14.498791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.936795+0000","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043344+0000","last_change":"2024-09-10T18:56:04.366386+0000","last_active":"2024-09-10T18:57:14.043344+0000","last_peered":"2024-09-10T18:57:14.043344+0000","last_clean":"2024-09-10T18:57:14.043344+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:57:14.043344+0000","last_undegraded":"2024-09-10T18:57:14.043344+0000","last_fullsized":"2024-09-10T18:57:14.043344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:53:34.552199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:53:34.552199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032492139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":352,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043999+0000","last_change":"2024-09-10T18:54:40.522553+0000","last_active":"2024-09-10T18:57:14.043999+0000","last_peered":"2024-09-10T18:57:14.043999+0000","last_clean":"2024-09-10T18:57:14.043999+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:57:14.043999+0000","last_undegraded":"2024-09-10T18:57:14.043999+0000","last_fullsized":"2024-09-10T18:57:14.043999+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:40.522460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.522460+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027852699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043220+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:57:14.043220+0000","last_peered":"2024-09-10T18:57:14.043220+0000","last_clean":"2024-09-10T18:57:14.043220+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:57:14.043220+0000","last_undegraded":"2024-09-10T18:57:14.043220+0000","last_fullsized":"2024-09-10T18:57:14.043220+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.837926+0000","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039955+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:57:14.039955+0000","last_peered":"2024-09-10T18:57:14.039955+0000","last_clean":"2024-09-10T18:57:14.039955+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:57:14.039955+0000","last_undegraded":"2024-09-10T18:57:14.039955+0000","last_fullsized":"2024-09-10T18:57:14.039955+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.533486+0000","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"126'5","reported_seq":177,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040379+0000","last_change":"2024-09-10T18:56:27.205415+0000","last_active":"2024-09-10T18:57:14.040379+0000","last_peered":"2024-09-10T18:57:14.040379+0000","last_clean":"2024-09-10T18:57:14.040379+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:57:14.040379+0000","last_undegraded":"2024-09-10T18:57:14.040379+0000","last_fullsized":"2024-09-10T18:57:14.040379+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.205258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.205258+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.814923+0000","scrub_duration":0.0049140169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039256+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:57:14.039256+0000","last_peered":"2024-09-10T18:57:14.039256+0000","last_clean":"2024-09-10T18:57:14.039256+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:57:14.039256+0000","last_undegraded":"2024-09-10T18:57:14.039256+0000","last_fullsized":"2024-09-10T18:57:14.039256+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.489013+0000","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041983+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:57:14.041983+0000","last_peered":"2024-09-10T18:57:14.041983+0000","last_clean":"2024-09-10T18:57:14.041983+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:57:14.041983+0000","last_undegraded":"2024-09-10T18:57:14.041983+0000","last_fullsized":"2024-09-10T18:57:14.041983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":347,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043496+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:57:14.043496+0000","last_peered":"2024-09-10T18:57:14.043496+0000","last_clean":"2024-09-10T18:57:14.043496+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:57:14.043496+0000","last_undegraded":"2024-09-10T18:57:14.043496+0000","last_fullsized":"2024-09-10T18:57:14.043496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032933+0000","last_change":"2024-09-10T18:54:53.310725+0000","last_active":"2024-09-10T18:57:14.032933+0000","last_peered":"2024-09-10T18:57:14.032933+0000","last_clean":"2024-09-10T18:57:14.032933+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:57:14.032933+0000","last_undegraded":"2024-09-10T18:57:14.032933+0000","last_fullsized":"2024-09-10T18:57:14.032933+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.310489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.310489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026931849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":162,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041800+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:57:14.041800+0000","last_peered":"2024-09-10T18:57:14.041800+0000","last_clean":"2024-09-10T18:57:14.041800+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:57:14.041800+0000","last_undegraded":"2024-09-10T18:57:14.041800+0000","last_fullsized":"2024-09-10T18:57:14.041800+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"126'5","reported_seq":177,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.495437+0000","last_change":"2024-09-10T18:56:39.805638+0000","last_active":"2024-09-10T18:57:14.495437+0000","last_peered":"2024-09-10T18:57:14.495437+0000","last_clean":"2024-09-10T18:57:14.495437+0000","last_became_active":"2024-09-10T18:54:37.610866+0000","last_became_peered":"2024-09-10T18:54:37.610866+0000","last_unstale":"2024-09-10T18:57:14.495437+0000","last_undegraded":"2024-09-10T18:57:14.495437+0000","last_fullsized":"2024-09-10T18:57:14.495437+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:39.805468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.805468+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.334135+0000","scrub_duration":0.0040878859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.499739+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:57:14.499739+0000","last_peered":"2024-09-10T18:57:14.499739+0000","last_clean":"2024-09-10T18:57:14.499739+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:57:14.499739+0000","last_undegraded":"2024-09-10T18:57:14.499739+0000","last_fullsized":"2024-09-10T18:57:14.499739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:21.661931+0000","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505012+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:57:14.505012+0000","last_peered":"2024-09-10T18:57:14.505012+0000","last_clean":"2024-09-10T18:57:14.505012+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:57:14.505012+0000","last_undegraded":"2024-09-10T18:57:14.505012+0000","last_fullsized":"2024-09-10T18:57:14.505012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":243,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042519+0000","last_change":"2024-09-10T18:55:10.179358+0000","last_active":"2024-09-10T18:57:14.042519+0000","last_peered":"2024-09-10T18:57:14.042519+0000","last_clean":"2024-09-10T18:57:14.042519+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:57:14.042519+0000","last_undegraded":"2024-09-10T18:57:14.042519+0000","last_fullsized":"2024-09-10T18:57:14.042519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003833071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037608+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:57:14.037608+0000","last_peered":"2024-09-10T18:57:14.037608+0000","last_clean":"2024-09-10T18:57:14.037608+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:57:14.037608+0000","last_undegraded":"2024-09-10T18:57:14.037608+0000","last_fullsized":"2024-09-10T18:57:14.037608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.752093+0000","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031969+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:57:14.031969+0000","last_peered":"2024-09-10T18:57:14.031969+0000","last_clean":"2024-09-10T18:57:14.031969+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:57:14.031969+0000","last_undegraded":"2024-09-10T18:57:14.031969+0000","last_fullsized":"2024-09-10T18:57:14.031969+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:15.510275+0000","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"126'9","reported_seq":182,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505735+0000","last_change":"2024-09-10T18:56:09.704635+0000","last_active":"2024-09-10T18:57:14.505735+0000","last_peered":"2024-09-10T18:57:14.505735+0000","last_clean":"2024-09-10T18:57:14.505735+0000","last_became_active":"2024-09-10T18:54:37.646743+0000","last_became_peered":"2024-09-10T18:54:37.646743+0000","last_unstale":"2024-09-10T18:57:14.505735+0000","last_undegraded":"2024-09-10T18:57:14.505735+0000","last_fullsized":"2024-09-10T18:57:14.505735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.575925+0000","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031161+0000","last_change":"2024-09-10T18:55:06.199960+0000","last_active":"2024-09-10T18:57:14.031161+0000","last_peered":"2024-09-10T18:57:14.031161+0000","last_clean":"2024-09-10T18:57:14.031161+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:57:14.031161+0000","last_undegraded":"2024-09-10T18:57:14.031161+0000","last_fullsized":"2024-09-10T18:57:14.031161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:55:06.199853+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:55:06.199853+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045914980000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089749+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:57:14.089749+0000","last_peered":"2024-09-10T18:57:14.089749+0000","last_clean":"2024-09-10T18:57:14.089749+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:57:14.089749+0000","last_undegraded":"2024-09-10T18:57:14.089749+0000","last_fullsized":"2024-09-10T18:57:14.089749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.423650+0000","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031889+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:57:14.031889+0000","last_peered":"2024-09-10T18:57:14.031889+0000","last_clean":"2024-09-10T18:57:14.031889+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:57:14.031889+0000","last_undegraded":"2024-09-10T18:57:14.031889+0000","last_fullsized":"2024-09-10T18:57:14.031889+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.667997+0000","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039193+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:57:14.039193+0000","last_peered":"2024-09-10T18:57:14.039193+0000","last_clean":"2024-09-10T18:57:14.039193+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:57:14.039193+0000","last_undegraded":"2024-09-10T18:57:14.039193+0000","last_fullsized":"2024-09-10T18:57:14.039193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.361501+0000","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031761+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:57:14.031761+0000","last_peered":"2024-09-10T18:57:14.031761+0000","last_clean":"2024-09-10T18:57:14.031761+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:57:14.031761+0000","last_undegraded":"2024-09-10T18:57:14.031761+0000","last_fullsized":"2024-09-10T18:57:14.031761+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.443415+0000","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":163,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045163+0000","last_change":"2024-09-10T18:54:37.725751+0000","last_active":"2024-09-10T18:57:14.045163+0000","last_peered":"2024-09-10T18:57:14.045163+0000","last_clean":"2024-09-10T18:57:14.045163+0000","last_became_active":"2024-09-10T18:54:37.716564+0000","last_became_peered":"2024-09-10T18:54:37.716564+0000","last_unstale":"2024-09-10T18:57:14.045163+0000","last_undegraded":"2024-09-10T18:57:14.045163+0000","last_fullsized":"2024-09-10T18:57:14.045163+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.045305+0000","last_change":"2024-09-10T18:54:54.957426+0000","last_active":"2024-09-10T18:57:14.045305+0000","last_peered":"2024-09-10T18:57:14.045305+0000","last_clean":"2024-09-10T18:57:14.045305+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:57:14.045305+0000","last_undegraded":"2024-09-10T18:57:14.045305+0000","last_fullsized":"2024-09-10T18:57:14.045305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:54:54.957316+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:54:54.957316+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039293499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044092+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:57:14.044092+0000","last_peered":"2024-09-10T18:57:14.044092+0000","last_clean":"2024-09-10T18:57:14.044092+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:57:14.044092+0000","last_undegraded":"2024-09-10T18:57:14.044092+0000","last_fullsized":"2024-09-10T18:57:14.044092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.036794+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:57:14.036794+0000","last_peered":"2024-09-10T18:57:14.036794+0000","last_clean":"2024-09-10T18:57:14.036794+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:57:14.036794+0000","last_undegraded":"2024-09-10T18:57:14.036794+0000","last_fullsized":"2024-09-10T18:57:14.036794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:26.718506+0000","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.498164+0000","last_change":"2024-09-10T18:55:02.948472+0000","last_active":"2024-09-10T18:57:14.498164+0000","last_peered":"2024-09-10T18:57:14.498164+0000","last_clean":"2024-09-10T18:57:14.498164+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:57:14.498164+0000","last_undegraded":"2024-09-10T18:57:14.498164+0000","last_fullsized":"2024-09-10T18:57:14.498164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.948218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.948218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034990780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.087734+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:57:14.087734+0000","last_peered":"2024-09-10T18:57:14.087734+0000","last_clean":"2024-09-10T18:57:14.087734+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:57:14.087734+0000","last_undegraded":"2024-09-10T18:57:14.087734+0000","last_fullsized":"2024-09-10T18:57:14.087734+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.479269+0000","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":163,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044314+0000","last_change":"2024-09-10T18:54:37.661524+0000","last_active":"2024-09-10T18:57:14.044314+0000","last_peered":"2024-09-10T18:57:14.044314+0000","last_clean":"2024-09-10T18:57:14.044314+0000","last_became_active":"2024-09-10T18:54:37.657896+0000","last_became_peered":"2024-09-10T18:54:37.657896+0000","last_unstale":"2024-09-10T18:57:14.044314+0000","last_undegraded":"2024-09-10T18:57:14.044314+0000","last_fullsized":"2024-09-10T18:57:14.044314+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":270,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043994+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:57:14.043994+0000","last_peered":"2024-09-10T18:57:14.043994+0000","last_clean":"2024-09-10T18:57:14.043994+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:57:14.043994+0000","last_undegraded":"2024-09-10T18:57:14.043994+0000","last_fullsized":"2024-09-10T18:57:14.043994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.059487+0000","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.506186+0000","last_change":"2024-09-10T18:55:08.609833+0000","last_active":"2024-09-10T18:57:14.506186+0000","last_peered":"2024-09-10T18:57:14.506186+0000","last_clean":"2024-09-10T18:57:14.506186+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:57:14.506186+0000","last_undegraded":"2024-09-10T18:57:14.506186+0000","last_fullsized":"2024-09-10T18:57:14.506186+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.609702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.609702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039391+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:57:14.039391+0000","last_peered":"2024-09-10T18:57:14.039391+0000","last_clean":"2024-09-10T18:57:14.039391+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:57:14.039391+0000","last_undegraded":"2024-09-10T18:57:14.039391+0000","last_fullsized":"2024-09-10T18:57:14.039391+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.486395+0000","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041405+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:57:14.041405+0000","last_peered":"2024-09-10T18:57:14.041405+0000","last_clean":"2024-09-10T18:57:14.041405+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:57:14.041405+0000","last_undegraded":"2024-09-10T18:57:14.041405+0000","last_fullsized":"2024-09-10T18:57:14.041405+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.320880+0000","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":162,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.038645+0000","last_change":"2024-09-10T18:54:37.573584+0000","last_active":"2024-09-10T18:57:14.038645+0000","last_peered":"2024-09-10T18:57:14.038645+0000","last_clean":"2024-09-10T18:57:14.038645+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:57:14.038645+0000","last_undegraded":"2024-09-10T18:57:14.038645+0000","last_fullsized":"2024-09-10T18:57:14.038645+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041025+0000","last_change":"2024-09-10T18:56:40.177375+0000","last_active":"2024-09-10T18:57:14.041025+0000","last_peered":"2024-09-10T18:57:14.041025+0000","last_clean":"2024-09-10T18:57:14.041025+0000","last_became_active":"2024-09-10T18:54:37.565268+0000","last_became_peered":"2024-09-10T18:54:37.565268+0000","last_unstale":"2024-09-10T18:57:14.041025+0000","last_undegraded":"2024-09-10T18:57:14.041025+0000","last_fullsized":"2024-09-10T18:57:14.041025+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.177249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.177249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.578420+0000","scrub_duration":0.0033094600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.500330+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:57:14.500330+0000","last_peered":"2024-09-10T18:57:14.500330+0000","last_clean":"2024-09-10T18:57:14.500330+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:57:14.500330+0000","last_undegraded":"2024-09-10T18:57:14.500330+0000","last_fullsized":"2024-09-10T18:57:14.500330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.123736+0000","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088165+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:57:14.088165+0000","last_peered":"2024-09-10T18:57:14.088165+0000","last_clean":"2024-09-10T18:57:14.088165+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:57:14.088165+0000","last_undegraded":"2024-09-10T18:57:14.088165+0000","last_fullsized":"2024-09-10T18:57:14.088165+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.905881+0000","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042307+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:57:14.042307+0000","last_peered":"2024-09-10T18:57:14.042307+0000","last_clean":"2024-09-10T18:57:14.042307+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:57:14.042307+0000","last_undegraded":"2024-09-10T18:57:14.042307+0000","last_fullsized":"2024-09-10T18:57:14.042307+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043296+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:57:14.043296+0000","last_peered":"2024-09-10T18:57:14.043296+0000","last_clean":"2024-09-10T18:57:14.043296+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:57:14.043296+0000","last_undegraded":"2024-09-10T18:57:14.043296+0000","last_fullsized":"2024-09-10T18:57:14.043296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.517080+0000","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505838+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:57:14.505838+0000","last_peered":"2024-09-10T18:57:14.505838+0000","last_clean":"2024-09-10T18:57:14.505838+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:57:14.505838+0000","last_undegraded":"2024-09-10T18:57:14.505838+0000","last_fullsized":"2024-09-10T18:57:14.505838+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.150161+0000","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"126'10","reported_seq":182,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.496021+0000","last_change":"2024-09-10T18:56:28.793230+0000","last_active":"2024-09-10T18:57:14.496021+0000","last_peered":"2024-09-10T18:57:14.496021+0000","last_clean":"2024-09-10T18:57:14.496021+0000","last_became_active":"2024-09-10T18:54:37.562584+0000","last_became_peered":"2024-09-10T18:54:37.562584+0000","last_unstale":"2024-09-10T18:57:14.496021+0000","last_undegraded":"2024-09-10T18:57:14.496021+0000","last_fullsized":"2024-09-10T18:57:14.496021+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:28.793065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.793065+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.828376+0000","scrub_duration":0.0039084699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.089617+0000","last_change":"2024-09-10T18:56:09.619685+0000","last_active":"2024-09-10T18:57:14.089617+0000","last_peered":"2024-09-10T18:57:14.089617+0000","last_clean":"2024-09-10T18:57:14.089617+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:57:14.089617+0000","last_undegraded":"2024-09-10T18:57:14.089617+0000","last_fullsized":"2024-09-10T18:57:14.089617+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.619567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.619567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:16.792631+0000","scrub_duration":0.0049369640000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.037990+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:57:14.037990+0000","last_peered":"2024-09-10T18:57:14.037990+0000","last_clean":"2024-09-10T18:57:14.037990+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:57:14.037990+0000","last_undegraded":"2024-09-10T18:57:14.037990+0000","last_fullsized":"2024-09-10T18:57:14.037990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.040121+0000","last_change":"2024-09-10T18:54:57.678574+0000","last_active":"2024-09-10T18:57:14.040121+0000","last_peered":"2024-09-10T18:57:14.040121+0000","last_clean":"2024-09-10T18:57:14.040121+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:57:14.040121+0000","last_undegraded":"2024-09-10T18:57:14.040121+0000","last_fullsized":"2024-09-10T18:57:14.040121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.678482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.678482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024273350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":374,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.506190+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:57:14.506190+0000","last_peered":"2024-09-10T18:57:14.506190+0000","last_clean":"2024-09-10T18:57:14.506190+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:57:14.506190+0000","last_undegraded":"2024-09-10T18:57:14.506190+0000","last_fullsized":"2024-09-10T18:57:14.506190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.506024+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:57:14.506024+0000","last_peered":"2024-09-10T18:57:14.506024+0000","last_clean":"2024-09-10T18:57:14.506024+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:57:14.506024+0000","last_undegraded":"2024-09-10T18:57:14.506024+0000","last_fullsized":"2024-09-10T18:57:14.506024+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:38.976604+0000","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"126'2","reported_seq":19,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.042452+0000","last_change":"2024-09-10T18:57:11.095107+0000","last_active":"2024-09-10T18:57:14.042452+0000","last_peered":"2024-09-10T18:57:14.042452+0000","last_clean":"2024-09-10T18:57:14.042452+0000","last_became_active":"2024-09-10T18:57:11.082286+0000","last_became_peered":"2024-09-10T18:57:11.082286+0000","last_unstale":"2024-09-10T18:57:14.042452+0000","last_undegraded":"2024-09-10T18:57:14.042452+0000","last_fullsized":"2024-09-10T18:57:14.042452+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:35.765725+0000","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.031847+0000","last_change":"2024-09-10T18:54:44.250455+0000","last_active":"2024-09-10T18:57:14.031847+0000","last_peered":"2024-09-10T18:57:14.031847+0000","last_clean":"2024-09-10T18:57:14.031847+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:57:14.031847+0000","last_undegraded":"2024-09-10T18:57:14.031847+0000","last_fullsized":"2024-09-10T18:57:14.031847+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:44.250321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:54:44.250321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031387210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.504358+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:57:14.504358+0000","last_peered":"2024-09-10T18:57:14.504358+0000","last_clean":"2024-09-10T18:57:14.504358+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:57:14.504358+0000","last_undegraded":"2024-09-10T18:57:14.504358+0000","last_fullsized":"2024-09-10T18:57:14.504358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041416+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:57:14.041416+0000","last_peered":"2024-09-10T18:57:14.041416+0000","last_clean":"2024-09-10T18:57:14.041416+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:57:14.041416+0000","last_undegraded":"2024-09-10T18:57:14.041416+0000","last_fullsized":"2024-09-10T18:57:14.041416+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:17.550858+0000","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.493417+0000","last_change":"2024-09-10T18:56:26.777537+0000","last_active":"2024-09-10T18:57:14.493417+0000","last_peered":"2024-09-10T18:57:14.493417+0000","last_clean":"2024-09-10T18:57:14.493417+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:57:14.493417+0000","last_undegraded":"2024-09-10T18:57:14.493417+0000","last_fullsized":"2024-09-10T18:57:14.493417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.777271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.777271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.531064+0000","scrub_duration":0.0030242540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043875+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:57:14.043875+0000","last_peered":"2024-09-10T18:57:14.043875+0000","last_clean":"2024-09-10T18:57:14.043875+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:57:14.043875+0000","last_undegraded":"2024-09-10T18:57:14.043875+0000","last_fullsized":"2024-09-10T18:57:14.043875+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.803420+0000","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":164,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.039640+0000","last_change":"2024-09-10T18:54:37.660686+0000","last_active":"2024-09-10T18:57:14.039640+0000","last_peered":"2024-09-10T18:57:14.039640+0000","last_clean":"2024-09-10T18:57:14.039640+0000","last_became_active":"2024-09-10T18:54:37.660393+0000","last_became_peered":"2024-09-10T18:54:37.660393+0000","last_unstale":"2024-09-10T18:57:14.039640+0000","last_undegraded":"2024-09-10T18:57:14.039640+0000","last_fullsized":"2024-09-10T18:57:14.039640+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":252,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.044950+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:57:14.044950+0000","last_peered":"2024-09-10T18:57:14.044950+0000","last_clean":"2024-09-10T18:57:14.044950+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:57:14.044950+0000","last_undegraded":"2024-09-10T18:57:14.044950+0000","last_fullsized":"2024-09-10T18:57:14.044950+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.573240+0000","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":258,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.041743+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:57:14.041743+0000","last_peered":"2024-09-10T18:57:14.041743+0000","last_clean":"2024-09-10T18:57:14.041743+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:57:14.041743+0000","last_undegraded":"2024-09-10T18:57:14.041743+0000","last_fullsized":"2024-09-10T18:57:14.041743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.997928+0000","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.032214+0000","last_change":"2024-09-10T18:54:58.424151+0000","last_active":"2024-09-10T18:57:14.032214+0000","last_peered":"2024-09-10T18:57:14.032214+0000","last_clean":"2024-09-10T18:57:14.032214+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:57:14.032214+0000","last_undegraded":"2024-09-10T18:57:14.032214+0000","last_fullsized":"2024-09-10T18:57:14.032214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.424008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.424008+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.146668564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.043997+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:57:14.043997+0000","last_peered":"2024-09-10T18:57:14.043997+0000","last_clean":"2024-09-10T18:57:14.043997+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:57:14.043997+0000","last_undegraded":"2024-09-10T18:57:14.043997+0000","last_fullsized":"2024-09-10T18:57:14.043997+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:18.684081+0000","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":173,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.505154+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:57:14.505154+0000","last_peered":"2024-09-10T18:57:14.505154+0000","last_clean":"2024-09-10T18:57:14.505154+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:57:14.505154+0000","last_undegraded":"2024-09-10T18:57:14.505154+0000","last_fullsized":"2024-09-10T18:57:14.505154+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.050482+0000","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":172,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T18:57:14.088460+0000","last_change":"2024-09-10T18:56:21.650671+0000","last_active":"2024-09-10T18:57:14.088460+0000","last_peered":"2024-09-10T18:57:14.088460+0000","last_clean":"2024-09-10T18:57:14.088460+0000","last_became_active":"2024-09-10T18:54:37.583272+0000","last_became_peered":"2024-09-10T18:54:37.583272+0000","last_unstale":"2024-09-10T18:57:14.088460+0000","last_undegraded":"2024-09-10T18:57:14.088460+0000","last_fullsized":"2024-09-10T18:57:14.088460+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.232910+0000","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":172,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":171,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":1194,"num_objects":118,"num_object_clones":1,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":94,"num_read_kb":1,"num_write":536,"num_write_kb":532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1437696,"data_stored":3582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1777,"ondisk_log_size":1777,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":453,"num_read_kb":448,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":31,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":55,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972459,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310440,"kb_used_data":4752,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93434712,"statfs":{"total":95995035648,"available":95677145088,"internally_reserved":0,"allocated":4866048,"data_stored":2901513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89700000000000002,"15min":0.91900000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2150000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.92000000000000004},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2430000000000001,"5min":1.554,"15min":1.554},"last":1.0209999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.88800000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3640000000000001,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96099999999999997,"15min":1.0189999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.286,"5min":1.742,"15min":1.8280000000000001},"last":1.0840000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.022,"15min":1.046},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.421,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98199999999999998,"15min":0.97899999999999998},"min":{"1min":0.51400000000000001,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":1.395,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1,"15min":1.01},"min":{"1min":0.56799999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.54,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":0.745},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.97999999999999998,"15min":0.96799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1719999999999999,"5min":3.3050000000000002,"15min":3.3050000000000002},"last":0.83599999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0229999999999999,"15min":1.022},"min":{"1min":0.499,"5min":0.499,"15min":0.443},"max":{"1min":1.4219999999999999,"5min":1.556,"15min":2.1989999999999998},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.0660000000000001,"15min":1.0369999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.669,"5min":1.669,"15min":2.1120000000000001},"last":0.89200000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98499999999999999,"15min":0.99199999999999999},"min":{"1min":0.51700000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.613,"5min":1.613,"15min":2.0819999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.163,"15min":1.1799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.514,"5min":1.514,"15min":2.1419999999999999},"last":1.4490000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1950000000000001,"15min":1.232},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.569,"5min":1.569,"15min":2.2749999999999999},"last":1.514},{"interface":"front","average":{"1min":1.248,"5min":1.1830000000000001,"15min":1.1879999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.53100000000000003},"max":{"1min":1.637,"5min":1.637,"15min":2.536},"last":1.2969999999999999}]}]},{"osd":6,"up_from":52,"seq":223338299509,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314540,"kb_used_data":4692,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430612,"statfs":{"total":95995035648,"available":95672946688,"internally_reserved":0,"allocated":4804608,"data_stored":2900606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.502,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3799999999999999,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.96299999999999997,"15min":0.93899999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3580000000000001,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.97299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1830000000000001,"15min":0.98799999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3089999999999999,"5min":18.494,"15min":18.494},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.871,"15min":0.90000000000000002},"min":{"1min":0.57199999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.2110000000000001,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.77500000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0149999999999999,"15min":0.95199999999999996},"min":{"1min":0.66900000000000004,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.321,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.1579999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.91700000000000004,"15min":0.92300000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2470000000000001,"5min":1.988,"15min":1.988},"last":0.73699999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0069999999999999,"15min":0.96999999999999997},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2989999999999999,"5min":2.278,"15min":2.278},"last":1.198},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.96899999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.121,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":0.94099999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.97499999999999998,"15min":0.90000000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.3560000000000001,"5min":1.712,"15min":1.712},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0640000000000001,"15min":1.0249999999999999},"min":{"1min":0.67700000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.429,"5min":2.202,"15min":2.202},"last":0.89500000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.042,"15min":0.99299999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.4690000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.131,"15min":1.099},"min":{"1min":0.73299999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.4670000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.125}]},{"osd":7,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.401,"5min":1.401,"15min":1.573},"last":1.046},{"interface":"front","average":{"1min":1.107,"5min":1.1619999999999999,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4510000000000001,"5min":1.77,"15min":1.77},"last":1.0720000000000001}]}]},{"osd":5,"up_from":43,"seq":184683593855,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309896,"kb_used_data":4200,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435256,"statfs":{"total":95995035648,"available":95677702144,"internally_reserved":0,"allocated":4300800,"data_stored":2320991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.0820000000000001,"15min":0.92900000000000005},"min":{"1min":0.41599999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.617},{"interface":"front","average":{"1min":1.292,"5min":1.024,"15min":0.91300000000000003},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0519999999999996,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.73899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4369999999999998,"5min":1.6579999999999999,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":15.824,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.427,"15min":1.069},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.0839999999999996,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.276}]},{"osd":2,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":1.8899999999999999,"15min":1.2250000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":20.536000000000001,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.452},{"interface":"front","average":{"1min":1.383,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.60299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":5.2489999999999997,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":1.3029999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.1279999999999999,"15min":0.98699999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5979999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.7609999999999999,"15min":1.218},"min":{"1min":0.66400000000000003,"5min":0.60999999999999999,"15min":0.57399999999999995},"max":{"1min":7.7679999999999998,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.254}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.147,"15min":0.90600000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":13.005000000000001,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.544,"5min":1.1659999999999999,"15min":0.96899999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":7.6299999999999999,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.252,"15min":1.0920000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.3520000000000001,"15min":1.143},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.47199999999999998},"max":{"1min":9.6760000000000002,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":0.66000000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.268,"15min":1.131},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.016,"5min":9.1140000000000008,"15min":9.1140000000000008},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2789999999999999,"15min":1.204},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":4.7949999999999999,"5min":6.532,"15min":6.532},"last":0.78800000000000003}]}]},{"osd":4,"up_from":35,"seq":150323855499,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301452,"kb_used_data":4092,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443700,"statfs":{"total":95995035648,"available":95686348800,"internally_reserved":0,"allocated":4190208,"data_stored":2310603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.82799999999999996,"15min":0.84199999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.113,"5min":1.546,"15min":1.546},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.89400000000000002,"15min":0.85499999999999998},"min":{"1min":0.58999999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2290000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.4750000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85299999999999998,"15min":0.873},"min":{"1min":0.52200000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.266,"5min":1.623,"15min":1.623},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92800000000000005,"15min":0.878},"min":{"1min":0.49299999999999999,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.3340000000000001,"5min":1.575,"15min":1.575},"last":1.2150000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95999999999999996,"15min":0.90400000000000003},"min":{"1min":0.59899999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2569999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96099999999999997,"15min":0.90200000000000002},"min":{"1min":0.66400000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.325,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.90800000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95399999999999996,"15min":0.91300000000000003},"min":{"1min":0.53800000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.2829999999999999,"5min":1.4730000000000001,"15min":1.714},"last":1.5489999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.97699999999999998,"15min":0.93500000000000005},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55600000000000005},"max":{"1min":1.329,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.85999999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96199999999999997,"15min":0.88900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.3939999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.999,"5min":1.026,"15min":0.94399999999999995},"min":{"1min":0.65100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.256,"5min":1.528,"15min":1.7909999999999999},"last":1.097}]},{"osd":6,"last update":"Tue Sep 10 18:57:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0349999999999999,"15min":0.96899999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.276},{"interface":"front","average":{"1min":1.079,"5min":1.135,"15min":1.0860000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4670000000000001,"5min":1.581,"15min":1.581},"last":1.3300000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1359999999999999,"15min":1.1619999999999999},"min":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.516,"5min":1.6200000000000001,"15min":2.008},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.137}]}]},{"osd":3,"up_from":29,"seq":124554051734,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309864,"kb_used_data":4176,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435288,"statfs":{"total":95995035648,"available":95677734912,"internally_reserved":0,"allocated":4276224,"data_stored":2310249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.109,"15min":0.78100000000000003},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":1.0229999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.627,"5min":1.1879999999999999,"15min":0.872},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.0009999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":1.194}]},{"osd":1,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":1.2929999999999999,"15min":0.88600000000000001},"min":{"1min":0.32100000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.97399999999999998,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.024,"15min":0.83899999999999997},"min":{"1min":0.36199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.95699999999999996,"5min":15.189,"15min":15.189},"last":0.96399999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.3089999999999999,"15min":0.96499999999999997},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.329,"5min":26.558,"15min":26.558},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":1.2330000000000001,"15min":0.91600000000000004},"min":{"1min":0.38300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.256,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.89200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:56:11 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.238,"15min":1.079},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.4019999999999999,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.26},{"interface":"front","average":{"1min":1.194,"5min":1.3360000000000001,"15min":1.165},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":2.9580000000000002,"5min":10.391,"15min":10.391},"last":1.2989999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.2130000000000001},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":8.6679999999999993,"5min":17.535,"15min":17.535},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.123,"15min":1.028},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":8.7880000000000003,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.81899999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:56:59 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.2709999999999999,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.5289999999999999,"5min":10.51,"15min":10.51},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.135,"15min":1.0920000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.8279999999999998,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.097}]},{"osd":7,"last update":"Tue Sep 10 18:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.2989999999999999,"15min":1.175},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4570000000000001,"5min":18.312999999999999,"15min":18.312999999999999},"last":1.05},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4319999999999999,"15min":1.306},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":7.298,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.2170000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378783,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301552,"kb_used_data":4192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443600,"statfs":{"total":95995035648,"available":95686246400,"internally_reserved":0,"allocated":4292608,"data_stored":2311182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.84099999999999997,"15min":0.70799999999999996},"min":{"1min":0.32200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84999999999999998,"5min":9.516,"15min":9.516},"last":0.628},{"interface":"front","average":{"1min":0.623,"5min":0.82799999999999996,"15min":0.76200000000000001},"min":{"1min":0.25800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0620000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.81899999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.36699999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82999999999999996,"5min":5.516,"15min":5.516},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.88900000000000001,"15min":0.79400000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.052,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.78000000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.91000000000000003,"15min":0.85099999999999998},"min":{"1min":0.36599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.3579999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.546,"5min":0.91700000000000004,"15min":0.84099999999999997},"min":{"1min":0.499,"5min":0.434,"15min":0.434},"max":{"1min":15.736000000000001,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.93300000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:56:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.1950000000000001,"15min":1.038},"min":{"1min":0.77200000000000002,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":21.571999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1699999999999999,"15min":1.1060000000000001},"min":{"1min":0.621,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.377,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.96399999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.113,"15min":1.109},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.605,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.214,"15min":1.085},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.429,"5min":21.654,"15min":21.654},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.206,"15min":1.1870000000000001},"min":{"1min":0.622,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5009999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.252,"15min":1.129},"min":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.5669999999999999,"5min":15.971,"15min":15.971},"last":1.04}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.21,"15min":1.2070000000000001},"min":{"1min":0.65800000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.399,"5min":15.025,"15min":15.025},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.2869999999999999,"15min":1.173},"min":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.0299999999999998,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.1339999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314016,"kb_used_data":4160,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93431136,"statfs":{"total":95995035648,"available":95673483264,"internally_reserved":0,"allocated":4259840,"data_stored":2320805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0309999999999999,"15min":0.749},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.90500000000000003,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.095,"15min":0.81699999999999995},"min":{"1min":0.30399999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.0529999999999999,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1040000000000001,"15min":0.81200000000000006},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1299999999999999,"5min":26.097000000000001,"15min":26.097000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":0.93000000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4370000000000001,"5min":32.363,"15min":32.363},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.1930000000000001,"15min":0.94699999999999995},"min":{"1min":0.42299999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.2649999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":1.4910000000000001,"15min":0.98499999999999999},"min":{"1min":0.34699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.89600000000000002,"5min":60.999000000000002,"15min":60.999000000000002},"last":1.0629999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1819999999999999,"15min":1.0669999999999999},"min":{"1min":0.47399999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.5379999999999998,"5min":12.808999999999999,"15min":12.808999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.6639999999999999,"15min":1.1619999999999999},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":6.5430000000000001,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.21}]},{"osd":5,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.2789999999999999,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":2.2719999999999998,"5min":26.050999999999998,"15min":26.050999999999998},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.3220000000000001,"15min":1.1919999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":4.5869999999999997,"5min":12.84,"15min":12.84},"last":0.73899999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.528,"15min":1.232},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":8.1760000000000002,"5min":25.608000000000001,"15min":25.608000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.4450000000000001,"15min":1.1990000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.486,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.1479999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.486,"15min":1.306},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":3.6499999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.5409999999999999,"15min":1.278},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.5740000000000001,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.27}]}]},{"osd":0,"up_from":8,"seq":34359738550,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314576,"kb_used_data":4720,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430576,"statfs":{"total":95995035648,"available":95672909824,"internally_reserved":0,"allocated":4833280,"data_stored":2910915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:56:49 2024","interfaces":[{"interface":"back","average":{"1min":4.8890000000000002,"5min":1.631,"15min":0.94099999999999995},"min":{"1min":0.34000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":80.710999999999999,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":4.9489999999999998,"5min":1.714,"15min":1.024},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":86.009,"5min":86.009,"15min":86.009},"last":0.54000000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.5920000000000001,"15min":0.98699999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.2789999999999999,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.214,"15min":0.873},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.0640000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.82099999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.0529999999999999,"15min":0.84999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0669999999999999,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.006},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.3280000000000001,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.89200000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:56:19 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.131,"15min":1.014},"min":{"1min":0.69299999999999995,"5min":0.5,"15min":0.5},"max":{"1min":5.5369999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.268,"5min":1.149,"15min":1.0760000000000001},"min":{"1min":0.71399999999999997,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":4.2389999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.206}]},{"osd":5,"last update":"Tue Sep 10 18:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.246,"15min":1.0940000000000001},"min":{"1min":0.71199999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.776,"5min":10.048,"15min":10.048},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.1319999999999999,"15min":1.1000000000000001},"min":{"1min":0.75700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.635,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:56:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.1220000000000001},"min":{"1min":0.69099999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.3200000000000001,"15min":1.1819999999999999},"min":{"1min":0.66400000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.696,"5min":11.894,"15min":11.894},"last":1.407}]},{"osd":7,"last update":"Tue Sep 10 18:56:49 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1890000000000001,"15min":1.171},"min":{"1min":0.66500000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.675,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.321,"15min":1.3049999999999999},"min":{"1min":0.75900000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":8.2840000000000007,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":0.86099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:57:20.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:18.666530+0000 mgr.y (mgr.24526) 330 : audit [DBG] from='client.25017 v1:172.21.15.83:0/3322922868' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.061506+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.086320+0000 mon.a (mon.0) 1864 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.091678+0000 mon.c (mon.2) 470 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.092069+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.094757+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.095157+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.095705+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.096037+0000 mon.a (mon.0) 1867 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:19.103560+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.001233+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.002062+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.181 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.049650+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:18.666530+0000 mgr.y (mgr.24526) 330 : audit [DBG] from='client.25017 v1:172.21.15.83:0/3322922868' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.061506+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.086320+0000 mon.a (mon.0) 1864 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.091678+0000 mon.c (mon.2) 470 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.092069+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.094757+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.095157+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.095705+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.096037+0000 mon.a (mon.0) 1867 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:19.103560+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.001233+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.002062+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.049650+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.050114+0000 mon.a (mon.0) 1869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.050230+0000 mon.a (mon.0) 1870 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.050305+0000 mon.a (mon.0) 1871 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: cluster 2024-09-10T18:57:20.050371+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.053977+0000 mon.a (mon.0) 1873 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.054428+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.054745+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.055027+0000 mon.a (mon.0) 1874 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[18629]: audit 2024-09-10T18:57:20.055685+0000 mon.a (mon.0) 1875 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.050114+0000 mon.a (mon.0) 1869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.050230+0000 mon.a (mon.0) 1870 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.050305+0000 mon.a (mon.0) 1871 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: cluster 2024-09-10T18:57:20.050371+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.053977+0000 mon.a (mon.0) 1873 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.054428+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.054745+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.055027+0000 mon.a (mon.0) 1874 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.184 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:20 smithi083 bash[32381]: audit 2024-09-10T18:57:20.055685+0000 mon.a (mon.0) 1875 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.200 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:20.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:20.338 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:18.666530+0000 mgr.y (mgr.24526) 330 : audit [DBG] from='client.25017 v1:172.21.15.83:0/3322922868' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.061506+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.086320+0000 mon.a (mon.0) 1864 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.091678+0000 mon.c (mon.2) 470 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.092069+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.094757+0000 mon.c (mon.2) 471 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.095157+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.095705+0000 mon.c (mon.2) 472 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.096037+0000 mon.a (mon.0) 1867 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:19.103560+0000 mon.c (mon.2) 473 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.001233+0000 mon.c (mon.2) 474 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.002062+0000 mon.a (mon.0) 1868 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.049650+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.050114+0000 mon.a (mon.0) 1869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.050230+0000 mon.a (mon.0) 1870 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi083-77640-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.050305+0000 mon.a (mon.0) 1871 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: cluster 2024-09-10T18:57:20.050371+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-09-10T18:57:20.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.053977+0000 mon.a (mon.0) 1873 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:20.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.054428+0000 mon.c (mon.2) 475 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.054745+0000 mon.c (mon.2) 476 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.055027+0000 mon.a (mon.0) 1874 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:20.366 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:20 smithi165 bash[22591]: audit 2024-09-10T18:57:20.055685+0000 mon.a (mon.0) 1875 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:20.441 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T18:57:20.475 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:20.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.610 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:20.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:20.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:20.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:20.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:20.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:20.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:21.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:21.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:21.160 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:21.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:21.295 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:21.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:21.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:21.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:21.566 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:21.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:21.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:21.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[18629]: cluster 2024-09-10T18:57:20.000497+0000 mgr.y (mgr.24526) 331 : cluster [DBG] pgmap v409: 267 pgs: 40 unknown, 227 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:21.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[18629]: audit 2024-09-10T18:57:20.104498+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:21.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[18629]: audit 2024-09-10T18:57:21.058562+0000 mon.a (mon.0) 1876 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[32381]: cluster 2024-09-10T18:57:20.000497+0000 mgr.y (mgr.24526) 331 : cluster [DBG] pgmap v409: 267 pgs: 40 unknown, 227 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[32381]: audit 2024-09-10T18:57:20.104498+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:21 smithi083 bash[32381]: audit 2024-09-10T18:57:21.058562+0000 mon.a (mon.0) 1876 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:21.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:21.845 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:21.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:21 smithi165 bash[22591]: cluster 2024-09-10T18:57:20.000497+0000 mgr.y (mgr.24526) 331 : cluster [DBG] pgmap v409: 267 pgs: 40 unknown, 227 active+clean; 585 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:21 smithi165 bash[22591]: audit 2024-09-10T18:57:20.104498+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:21.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:21 smithi165 bash[22591]: audit 2024-09-10T18:57:21.058562+0000 mon.a (mon.0) 1876 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:21.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:21.982 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:21.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:22.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:22.111 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:22.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:22.246 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:22.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:22.380 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:22.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:22.520 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:22.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:22.637 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.057008+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.637 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.105632+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.637 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.496629+0000 mon.a (mon.0) 1877 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.638 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: cluster 2024-09-10T18:57:21.496897+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:57:22.638 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.497521+0000 mon.c (mon.2) 479 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.638 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.498284+0000 mon.a (mon.0) 1879 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.638 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:21.498532+0000 mon.a (mon.0) 1880 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.638 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: cluster 2024-09-10T18:57:22.001540+0000 mgr.y (mgr.24526) 332 : cluster [DBG] pgmap v412: 259 pgs: 2 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: cluster 2024-09-10T18:57:22.050574+0000 mon.a (mon.0) 1881 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.061028+0000 mon.a (mon.0) 1882 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.061247+0000 mon.a (mon.0) 1883 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]': finished 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.061396+0000 mon.a (mon.0) 1884 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: cluster 2024-09-10T18:57:22.061563+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:57:22.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.090546+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.092510+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.092951+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:22.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18629]: audit 2024-09-10T18:57:22.106842+0000 mon.c (mon.2) 480 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.641 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.057008+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.641 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.105632+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.641 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.496629+0000 mon.a (mon.0) 1877 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.641 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: cluster 2024-09-10T18:57:21.496897+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:57:22.641 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.497521+0000 mon.c (mon.2) 479 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.498284+0000 mon.a (mon.0) 1879 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:21.498532+0000 mon.a (mon.0) 1880 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: cluster 2024-09-10T18:57:22.001540+0000 mgr.y (mgr.24526) 332 : cluster [DBG] pgmap v412: 259 pgs: 2 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T18:57:22.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: cluster 2024-09-10T18:57:22.050574+0000 mon.a (mon.0) 1881 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:22.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.061028+0000 mon.a (mon.0) 1882 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.061247+0000 mon.a (mon.0) 1883 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]': finished 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.061396+0000 mon.a (mon.0) 1884 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: cluster 2024-09-10T18:57:22.061563+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.090546+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.092510+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.092951+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:22.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[32381]: audit 2024-09-10T18:57:22.106842+0000 mon.c (mon.2) 480 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:22.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:22.775 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.057008+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.775 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.105632+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.496629+0000 mon.a (mon.0) 1877 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: cluster 2024-09-10T18:57:21.496897+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.497521+0000 mon.c (mon.2) 479 : audit [INF] from='client.? v1:172.21.15.83:0/2768920774' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.498284+0000 mon.a (mon.0) 1879 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]: dispatch 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:21.498532+0000 mon.a (mon.0) 1880 : audit [INF] from='client.49079 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]: dispatch 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: cluster 2024-09-10T18:57:22.001540+0000 mgr.y (mgr.24526) 332 : cluster [DBG] pgmap v412: 259 pgs: 2 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: cluster 2024-09-10T18:57:22.050574+0000 mon.a (mon.0) 1881 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.061028+0000 mon.a (mon.0) 1882 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi083-77640-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.061247+0000 mon.a (mon.0) 1883 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-29", "mode": "writeback"}]': finished 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.061396+0000 mon.a (mon.0) 1884 : audit [INF] from='client.49079 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi083-77648-47"}]': finished 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: cluster 2024-09-10T18:57:22.061563+0000 mon.a (mon.0) 1885 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.090546+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.092510+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:22.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.092951+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:22.777 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:22 smithi165 bash[22591]: audit 2024-09-10T18:57:22.106842+0000 mon.c (mon.2) 480 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:22.793 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:22.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:22.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:22.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:23.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:22] "GET /metrics HTTP/1.1" 200 231054 "" "Prometheus/2.43.0" 2024-09-10T18:57:23.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:23.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:57:23.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:23.205 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:23.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:23.335 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:23.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:23.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:23.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:23.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:23.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:23.754 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:23.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:23.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:23.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:23.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/1143190660 >> v1:172.21.15.83:6790/0 conn(0x7fcd800fd8d0 legacy=0x7fcd800fb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/1143190660 shutdown_connections 2024-09-10T18:57:23.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/1143190660 >> 172.21.15.83:0/1143190660 conn(0x7fcd80074a20 msgr2=0x7fcd800f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:23.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/1143190660 shutdown_connections 2024-09-10T18:57:23.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/1143190660 wait complete. 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 librados: starting msgr at 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 librados: starting objecter 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 Processor -- start 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- start start 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 librados: setting wanted keys 2024-09-10T18:57:23.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 librados: calling monclient init 2024-09-10T18:57:23.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd8011e260 con 0x7fcd8010b2b0 2024-09-10T18:57:23.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd851c3640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fcd800fd440 0x7fcd80107700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50182/0 (socket says 172.21.15.83:50182) 2024-09-10T18:57:23.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd8011f460 con 0x7fcd800fd8d0 2024-09-10T18:57:23.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd851c3640 1 -- 172.21.15.83:0/3105305930 learned_addr learned my addr 172.21.15.83:0/3105305930 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:23.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd75ffb640 10 client.?.objecter ms_handle_connect 0x7fcd800fd440 2024-09-10T18:57:23.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd75ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd80120660 con 0x7fcd800fd440 2024-09-10T18:57:23.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd75ffb640 10 client.?.objecter ms_handle_connect 0x7fcd8010b2b0 2024-09-10T18:57:23.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd75ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.940+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582571627 0 0) 0x7fcd8011e260 con 0x7fcd8010b2b0 2024-09-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd68003120 con 0x7fcd8010b2b0 2024-09-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 10 client.?.objecter ms_handle_connect 0x7fcd800fd8d0 2024-09-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237412164 0 0) 0x7fcd80120660 con 0x7fcd800fd440 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8011e260 con 0x7fcd800fd440 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230805064 0 0) 0x7fcd8011f460 con 0x7fcd800fd8d0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80120660 con 0x7fcd800fd8d0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3189749807 0 0) 0x7fcd68003120 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd8011f460 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcd700019f0 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3463661935 0 0) 0x7fcd8011f460 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 >> v1:172.21.15.83:6790/0 conn(0x7fcd800fd440 legacy=0x7fcd80107700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 >> v1:172.21.15.165:6789/0 conn(0x7fcd800fd8d0 legacy=0x7fcd80107e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80121860 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8011e450 con 0x7fcd8010b2b0 2024-09-10T18:57:23.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd87427640 10 client.25035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8011e8a0 con 0x7fcd8010b2b0 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd87427640 1 librados: init done 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fcd70002260 con 0x7fcd8010b2b0 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcd700030e0 con 0x7fcd8010b2b0 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c005010 con 0x7fcd8010b2b0 2024-09-10T18:57:23.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.944+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fcd70001e00 con 0x7fcd8010b2b0 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(287..287 src has 69..287) v4 ==== 9345+0+0 (unknown 17317698 0 0) 0x7fcd70088140 con 0x7fcd8010b2b0 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 10 client.25035.objecter ms_dispatch 0x7fcd80000b90 osd_map(287..287 src has 69..287) v4 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 3 client.25035.objecter handle_osd_map got epochs [287,287] > 0 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 3 client.25035.objecter handle_osd_map decoding full epoch 287 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 20 client.25035.objecter dump_active .. 0 homeless 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7fcd75ffb640 10 client.25035.objecter ms_handle_connect 0x7fcd68071550 2024-09-10T18:57:23.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7f481241b640 1 -- 172.21.15.83:0/4185728600 >> v1:172.21.15.83:6790/0 conn(0x7f480c0fd610 legacy=0x7f480c0fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7f481241b640 1 -- 172.21.15.83:0/4185728600 shutdown_connections 2024-09-10T18:57:23.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.948+0000 7f481241b640 1 -- 172.21.15.83:0/4185728600 >> 172.21.15.83:0/4185728600 conn(0x7f480c075290 msgr2=0x7f480c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:23.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- 172.21.15.83:0/4185728600 shutdown_connections 2024-09-10T18:57:23.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- 172.21.15.83:0/4185728600 wait complete. 2024-09-10T18:57:23.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 librados: starting msgr at 2024-09-10T18:57:23.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 librados: starting objecter 2024-09-10T18:57:23.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 Processor -- start 2024-09-10T18:57:23.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- start start 2024-09-10T18:57:23.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 librados: setting wanted keys 2024-09-10T18:57:23.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 librados: calling monclient init 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f480c0fd3c0 con 0x7f480c108f40 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f480c11fbb0 con 0x7f480c11b0a0 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f480c120dd0 con 0x7f480c0fd610 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f480bfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f480c0fd610 0x7f480c1072e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50202/0 (socket says 172.21.15.83:50202) 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f480bfff640 1 -- 172.21.15.83:0/2254290967 learned_addr learned my addr 172.21.15.83:0/2254290967 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter ms_handle_connect 0x7f480c0fd610 2024-09-10T18:57:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter ms_handle_connect 0x7f480c108f40 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter ms_handle_connect 0x7f480c11b0a0 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610066428 0 0) 0x7f480c120dd0 con 0x7f480c0fd610 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f8003120 con 0x7f480c0fd610 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128962503 0 0) 0x7f480c11fbb0 con 0x7f480c11b0a0 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c120dd0 con 0x7f480c11b0a0 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480067789 0 0) 0x7f47f8003120 con 0x7f480c0fd610 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f480c11fbb0 con 0x7f480c0fd610 2024-09-10T18:57:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f47f4002b30 con 0x7f480c0fd610 2024-09-10T18:57:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2871781863 0 0) 0x7f480c11fbb0 con 0x7f480c0fd610 2024-09-10T18:57:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.165:6789/0 conn(0x7f480c11b0a0 legacy=0x7f480c11d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6789/0 conn(0x7f480c108f40 legacy=0x7f480c119980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c121fd0 con 0x7f480c0fd610 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480c11ec00 con 0x7f480c0fd610 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 10 client.49109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f480c11f090 con 0x7f480c0fd610 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f481241b640 1 librados: init done 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f47f4002d50 con 0x7f480c0fd610 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f47f4003760 con 0x7f480c0fd610 2024-09-10T18:57:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.952+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fcd700541c0 con 0x7fcd8010b2b0 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f47ec004ff0 con 0x7f480c0fd610 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f47f4002320 con 0x7f480c0fd610 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(287..287 src has 69..287) v4 ==== 9345+0+0 (unknown 17317698 0 0) 0x7f47f408ad00 con 0x7f480c0fd610 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 osd_map(287..287 src has 69..287) v4 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 3 client.49109.objecter handle_osd_map got epochs [287,287] > 0 2024-09-10T18:57:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 3 client.49109.objecter handle_osd_map decoding full epoch 287 2024-09-10T18:57:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 20 client.49109.objecter dump_active .. 0 homeless 2024-09-10T18:57:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v287) v1 ==== 54+0+16 (unknown 2110158131 0 2420000192) 0x7f47f408b0e0 con 0x7f480c0fd610 2024-09-10T18:57:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47f8071590 2024-09-10T18:57:23.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 10 client.49109.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 1 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 1 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 0 1 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 4 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 10 client.49109.objecter _send_command 1 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec005290 con 0x7f47ec004610 2024-09-10T18:57:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.956+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec004610 2024-09-10T18:57:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.960+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec005290 con 0x7f47ec004610 2024-09-10T18:57:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.960+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:57:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.960+0000 7f48097fa640 10 client.49109.objecter _finish_command 1 = system:0 2024-09-10T18:57:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.960+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 4 2024-09-10T18:57:23.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.960+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 0 1 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 10 client.49109.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 2 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 2 2024-09-10T18:57:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec005810 osd=0 3 2024-09-10T18:57:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 0 2 2024-09-10T18:57:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 4 2024-09-10T18:57:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 10 client.49109.objecter _send_command 2 2024-09-10T18:57:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec004c60 con 0x7f47ec004610 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec004c60 con 0x7f47ec004610 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 10 client.49109.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 4 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 0 2 2024-09-10T18:57:23.980 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 10 client.49109.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 3 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 3 2024-09-10T18:57:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 1 3 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 4 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 10 client.49109.objecter _send_command 3 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec003b50 con 0x7f47ec003620 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec003620 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec003b50 con 0x7f47ec003620 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:57:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 10 client.49109.objecter _finish_command 3 = system:0 2024-09-10T18:57:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 4 2024-09-10T18:57:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.976+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 1 3 2024-09-10T18:57:23.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 10 client.49109.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 4 2024-09-10T18:57:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:24.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 4 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec004e10 osd=1 3 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 1 4 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 4 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 10 client.49109.objecter _send_command 4 2024-09-10T18:57:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:23.996+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec003110 con 0x7f47ec003620 2024-09-10T18:57:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec003110 con 0x7f47ec003620 2024-09-10T18:57:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 10 client.49109.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 4 2024-09-10T18:57:24.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 1 4 2024-09-10T18:57:24.004 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:57:24.004 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 10 client.49109.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 5 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 5 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 2 5 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 4 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 10 client.49109.objecter _send_command 5 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec0022d0 con 0x7f47ec00d350 2024-09-10T18:57:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.000+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec00d350 2024-09-10T18:57:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.004+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec0022d0 con 0x7f47ec00d350 2024-09-10T18:57:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.004+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:57:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.004+0000 7f48097fa640 10 client.49109.objecter _finish_command 5 = system:0 2024-09-10T18:57:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.004+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 4 2024-09-10T18:57:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.004+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 2 5 2024-09-10T18:57:24.019 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:24.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 10 client.49109.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 6 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 6 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec001b60 osd=2 3 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 2 6 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 4 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 10 client.49109.objecter _send_command 6 2024-09-10T18:57:24.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.020+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec0015e0 con 0x7f47ec00d350 2024-09-10T18:57:24.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec0015e0 con 0x7f47ec00d350 2024-09-10T18:57:24.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 10 client.49109.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.028 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 4 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 2 6 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 10 client.49109.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 7 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 7 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 3 7 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 4 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 10 client.49109.objecter _send_command 7 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec00d140 con 0x7f47ec011fd0 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.024+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec011fd0 2024-09-10T18:57:24.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.028+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec00d140 con 0x7f47ec011fd0 2024-09-10T18:57:24.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.028+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:57:24.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.028+0000 7f48097fa640 10 client.49109.objecter _finish_command 7 = system:0 2024-09-10T18:57:24.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.028+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 4 2024-09-10T18:57:24.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.028+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 3 7 2024-09-10T18:57:24.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 10 client.49109.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 8 2024-09-10T18:57:24.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 8 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c990 osd=3 3 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 3 8 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 4 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 10 client.49109.objecter _send_command 8 2024-09-10T18:57:24.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec00c480 con 0x7f47ec011fd0 2024-09-10T18:57:24.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec00c480 con 0x7f47ec011fd0 2024-09-10T18:57:24.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f48097fa640 10 client.49109.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 4 2024-09-10T18:57:24.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.048+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 3 8 2024-09-10T18:57:24.054 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 10 client.49109.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 9 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 9 2024-09-10T18:57:24.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 4 9 2024-09-10T18:57:24.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 4 2024-09-10T18:57:24.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 10 client.49109.objecter _send_command 9 2024-09-10T18:57:24.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.052+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec00c420 con 0x7f47ec011b20 2024-09-10T18:57:24.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec011b20 2024-09-10T18:57:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec00c420 con 0x7f47ec011b20 2024-09-10T18:57:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:57:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 10 client.49109.objecter _finish_command 9 = system:0 2024-09-10T18:57:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 4 2024-09-10T18:57:24.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.056+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 4 9 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 10 client.49109.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 10 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 10 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c240 osd=4 3 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 4 10 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 4 2024-09-10T18:57:24.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 10 client.49109.objecter _send_command 10 2024-09-10T18:57:24.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.072+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec00c420 con 0x7f47ec011b20 2024-09-10T18:57:24.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec00c420 con 0x7f47ec011b20 2024-09-10T18:57:24.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 10 client.49109.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 4 2024-09-10T18:57:24.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 4 10 2024-09-10T18:57:24.080 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:57:24.080 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 10 client.49109.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 11 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 11 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 5 11 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 4 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 10 client.49109.objecter _send_command 11 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec017ea0 con 0x7f47ec0178b0 2024-09-10T18:57:24.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec0178b0 2024-09-10T18:57:24.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec017ea0 con 0x7f47ec0178b0 2024-09-10T18:57:24.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:57:24.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 10 client.49109.objecter _finish_command 11 = system:0 2024-09-10T18:57:24.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 4 2024-09-10T18:57:24.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.076+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 5 11 2024-09-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 10 client.49109.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 12 2024-09-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 12 2024-09-10T18:57:24.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec00c420 osd=5 3 2024-09-10T18:57:24.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 5 12 2024-09-10T18:57:24.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 4 2024-09-10T18:57:24.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 10 client.49109.objecter _send_command 12 2024-09-10T18:57:24.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.096+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec017e60 con 0x7f47ec0178b0 2024-09-10T18:57:24.103 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:57:24.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec017e60 con 0x7f47ec0178b0 2024-09-10T18:57:24.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 10 client.49109.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 4 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 5 12 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 10 client.49109.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 13 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 13 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 6 13 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 4 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 10 client.49109.objecter _send_command 13 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec022ad0 con 0x7f47ec01efd0 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.100+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec01efd0 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.104+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec022ad0 con 0x7f47ec01efd0 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.104+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.104+0000 7f48097fa640 10 client.49109.objecter _finish_command 13 = system:0 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.104+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 4 2024-09-10T18:57:24.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.104+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 6 13 2024-09-10T18:57:24.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 10 client.49109.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 14 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 14 2024-09-10T18:57:24.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec017ea0 osd=6 3 2024-09-10T18:57:24.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 6 14 2024-09-10T18:57:24.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 4 2024-09-10T18:57:24.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 10 client.49109.objecter _send_command 14 2024-09-10T18:57:24.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec022ad0 con 0x7f47ec01efd0 2024-09-10T18:57:24.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec022ad0 con 0x7f47ec01efd0 2024-09-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f48097fa640 10 client.49109.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 4 2024-09-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.132+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 6 14 2024-09-10T18:57:24.138 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 10 client.49109.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 15 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 15 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 7 15 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 4 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 10 client.49109.objecter _send_command 15 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47ec024e20 con 0x7f47ec023150 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 10 client.49109.objecter ms_handle_connect 0x7f47ec023150 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f47ec024e20 con 0x7f47ec023150 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 10 client.49109.objecter _finish_command 15 = system:0 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 4 2024-09-10T18:57:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.136+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 7 15 2024-09-10T18:57:24.144 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 10 client.49109.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign -1 16 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 20 client.49109.objecter _get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 15 client.49109.objecter _session_command_op_remove -1 16 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 20 client.49109.objecter get_session s=0x7f47ec022ad0 osd=7 3 2024-09-10T18:57:24.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 15 client.49109.objecter _session_command_op_assign 7 16 2024-09-10T18:57:24.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 4 2024-09-10T18:57:24.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 10 client.49109.objecter _send_command 16 2024-09-10T18:57:24.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.152+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47ec024e20 con 0x7f47ec023150 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48097fa640 1 -- 172.21.15.83:0/2254290967 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47ec024e20 con 0x7f47ec023150 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48097fa640 10 client.49109.objecter ms_dispatch 0x7f480c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48097fa640 10 client.49109.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48097fa640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 4 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48097fa640 15 client.49109.objecter _session_command_op_remove 7 16 2024-09-10T18:57:24.161 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:57:24.162 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:57:24.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 librados: watch_flush enter 2024-09-10T18:57:24.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 librados: watch_flush exit 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.0 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6801/2535527483 conn(0x7f47ec004610 legacy=0x7f47ec002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec005810 osd=0 1 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.1 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6805/2309624637 conn(0x7f47ec003620 legacy=0x7f47ec00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec004e10 osd=1 1 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.2 2024-09-10T18:57:24.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6809/386094220 conn(0x7f47ec00d350 legacy=0x7f47ec00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec001b60 osd=2 1 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.3 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6813/1279688021 conn(0x7f47ec011fd0 legacy=0x7f47ec0143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c990 osd=3 1 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.4 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.165:6800/3225515265 conn(0x7f47ec011b20 legacy=0x7f47ec016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c240 osd=4 1 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.5 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.165:6804/571173777 conn(0x7f47ec0178b0 legacy=0x7f47ec01d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec00c420 osd=5 1 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.6 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.165:6808/1317922936 conn(0x7f47ec01efd0 legacy=0x7f47ec0213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec017ea0 osd=6 1 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter close_session for osd.7 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.165:6812/1610151650 conn(0x7f47ec023150 legacy=0x7f47ec0235a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter put_session s=0x7f47ec022ad0 osd=7 1 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 20 client.49109.objecter shutdown clearing up homeless session... 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 10 client.49109.objecter successfully canceled tick 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48109b8640 1 -- 172.21.15.83:0/2254290967 reap_dead start 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48109b8640 1 -- 172.21.15.83:0/2254290967 reap_dead start 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48109b8640 1 -- 172.21.15.83:0/2254290967 reap_dead start 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48109b8640 1 -- 172.21.15.83:0/2254290967 reap_dead start 2024-09-10T18:57:24.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f48109b8640 1 -- 172.21.15.83:0/2254290967 reap_dead start 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6800/3071793735 conn(0x7f47f8071590 legacy=0x7f47f8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.156+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> v1:172.21.15.83:6790/0 conn(0x7f480c0fd610 legacy=0x7f480c1072e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.160+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 shutdown_connections 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.160+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 >> 172.21.15.83:0/2254290967 conn(0x7f480c075290 msgr2=0x7f480c10bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.160+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 shutdown_connections 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.160+0000 7f481241b640 1 -- 172.21.15.83:0/2254290967 wait complete. 2024-09-10T18:57:24.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.160+0000 7f481241b640 1 librados: shutdown 2024-09-10T18:57:24.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:24.228+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fcd6c0052c0 con 0x7fcd8010b2b0 2024-09-10T18:57:24.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:24.279 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:24.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:24.414 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:24.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:24.550 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:24.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:24.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:23.067982+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:24.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: cluster 2024-09-10T18:57:23.068051+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:23.069112+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:23.107811+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:23.959056+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.83:0/2254290967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:24.002668+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[32381]: audit 2024-09-10T18:57:24.003834+0000 mon.a (mon.0) 1892 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:23.067982+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:24.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: cluster 2024-09-10T18:57:23.068051+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:57:24.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:23.069112+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:24.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:23.107811+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:24.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:23.959056+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.83:0/2254290967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:24.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:24.002668+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:24 smithi083 bash[18629]: audit 2024-09-10T18:57:24.003834+0000 mon.a (mon.0) 1892 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:24.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:23.067982+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi083-77648-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: cluster 2024-09-10T18:57:23.068051+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:23.069112+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:23.107811+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:23.959056+0000 mon.c (mon.2) 482 : audit [DBG] from='client.? v1:172.21.15.83:0/2254290967' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:24.002668+0000 mon.c (mon.2) 483 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:24 smithi165 bash[22591]: audit 2024-09-10T18:57:24.003834+0000 mon.a (mon.0) 1892 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-09-10T18:57:24.828 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:24.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:24.967 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:24.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:25.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.048+0000 7fcd75ffb640 1 -- 172.21.15.83:0/3105305930 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v289) v1 ==== 91+0+0 (unknown 1582930839 0 0) 0x7fcd70053b60 con 0x7fcd8010b2b0 2024-09-10T18:57:25.054 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T18:57:25.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 10 librados: watch_flush enter 2024-09-10T18:57:25.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 10 librados: watch_flush exit 2024-09-10T18:57:25.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 20 client.25035.objecter shutdown clearing up homeless session... 2024-09-10T18:57:25.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 10 client.25035.objecter successfully canceled tick 2024-09-10T18:57:25.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 >> v1:172.21.15.83:6800/3071793735 conn(0x7fcd68071550 legacy=0x7fcd68073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:25.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 >> v1:172.21.15.83:6789/0 conn(0x7fcd8010b2b0 legacy=0x7fcd8011c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:25.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 shutdown_connections 2024-09-10T18:57:25.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.056+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 >> 172.21.15.83:0/3105305930 conn(0x7fcd80074a20 msgr2=0x7fcd800f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:25.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.060+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 shutdown_connections 2024-09-10T18:57:25.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.060+0000 7fcd87427640 1 -- 172.21.15.83:0/3105305930 wait complete. 2024-09-10T18:57:25.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:25.060+0000 7fcd87427640 1 librados: shutdown 2024-09-10T18:57:25.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:25.092 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:25.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:25.230 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:25.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:25.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:25.371 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:25.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:25.509 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:25.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: cluster 2024-09-10T18:57:24.002173+0000 mgr.y (mgr.24526) 333 : cluster [DBG] pgmap v415: 266 pgs: 8 unknown, 1 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:24.108978+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:24.504746+0000 mon.a (mon.0) 1893 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: cluster 2024-09-10T18:57:24.504928+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: cluster 2024-09-10T18:57:24.506015+0000 mon.a (mon.0) 1895 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:24.506020+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:24.507997+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:24.516589+0000 mon.a (mon.0) 1897 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.039313+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.051829+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.052095+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:57:25.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.052304+0000 mon.a (mon.0) 1901 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.052845+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.059707+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.062947+0000 mon.a (mon.0) 1903 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.110003+0000 mon.c (mon.2) 487 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.110683+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:25 smithi165 bash[22591]: audit 2024-09-10T18:57:25.114601+0000 mon.a (mon.0) 1904 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.647 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:25.649 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: cluster 2024-09-10T18:57:24.002173+0000 mgr.y (mgr.24526) 333 : cluster [DBG] pgmap v415: 266 pgs: 8 unknown, 1 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:25.649 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:24.108978+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.649 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:24.504746+0000 mon.a (mon.0) 1893 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-09-10T18:57:25.649 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: cluster 2024-09-10T18:57:24.504928+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:57:25.649 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: cluster 2024-09-10T18:57:24.506015+0000 mon.a (mon.0) 1895 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:24.506020+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:24.507997+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:24.516589+0000 mon.a (mon.0) 1897 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.039313+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.051829+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.052095+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.052304+0000 mon.a (mon.0) 1901 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.052845+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.059707+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.062947+0000 mon.a (mon.0) 1903 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.110003+0000 mon.c (mon.2) 487 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.110683+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.650 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[18629]: audit 2024-09-10T18:57:25.114601+0000 mon.a (mon.0) 1904 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: cluster 2024-09-10T18:57:24.002173+0000 mgr.y (mgr.24526) 333 : cluster [DBG] pgmap v415: 266 pgs: 8 unknown, 1 peering, 257 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:24.108978+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:24.504746+0000 mon.a (mon.0) 1893 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: cluster 2024-09-10T18:57:24.504928+0000 mon.a (mon.0) 1894 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: cluster 2024-09-10T18:57:24.506015+0000 mon.a (mon.0) 1895 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:24.506020+0000 mon.c (mon.2) 485 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:24.507997+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:24.516589+0000 mon.a (mon.0) 1897 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.651 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.039313+0000 mon.a (mon.0) 1898 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.051829+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi083-77648-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.052095+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? v1:172.21.15.83:0/3105305930' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.052304+0000 mon.a (mon.0) 1901 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.052845+0000 mon.a (mon.0) 1902 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.059707+0000 mon.c (mon.2) 486 : audit [INF] from='client.? v1:172.21.15.83:0/2112792776' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.062947+0000 mon.a (mon.0) 1903 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]: dispatch 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.110003+0000 mon.c (mon.2) 487 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.110683+0000 mon.b (mon.1) 233 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:25 smithi083 bash[32381]: audit 2024-09-10T18:57:25.114601+0000 mon.a (mon.0) 1904 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:25.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:25.788 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:25.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:25.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:25.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:26.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:26.073 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:26.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:26.211 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:26.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.350 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:26.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:26.488 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:26.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.514413+0000 osd.7 (osd.7) 180 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.517205+0000 osd.7 (osd.7) 181 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.628920+0000 osd.1 (osd.1) 187 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:25.633361+0000 osd.1 (osd.1) 188 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.004710+0000 mgr.y (mgr.24526) 334 : cluster [DBG] pgmap v418: 266 pgs: 8 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 26 op/s 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.052249+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.052605+0000 mon.a (mon.0) 1905 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.053013+0000 mon.a (mon.0) 1906 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:26.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.053127+0000 mon.a (mon.0) 1907 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.056629+0000 mon.a (mon.0) 1908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.091301+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.091578+0000 mon.a (mon.0) 1909 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.093075+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.093400+0000 mon.a (mon.0) 1910 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.093907+0000 mon.c (mon.2) 490 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.094220+0000 mon.a (mon.0) 1911 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:26 smithi165 bash[22591]: audit 2024-09-10T18:57:26.111071+0000 mon.c (mon.2) 491 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:26.626 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:26.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.514413+0000 osd.7 (osd.7) 180 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.517205+0000 osd.7 (osd.7) 181 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.628920+0000 osd.1 (osd.1) 187 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:25.633361+0000 osd.1 (osd.1) 188 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.004710+0000 mgr.y (mgr.24526) 334 : cluster [DBG] pgmap v418: 266 pgs: 8 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 26 op/s 2024-09-10T18:57:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.052249+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.052605+0000 mon.a (mon.0) 1905 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.053013+0000 mon.a (mon.0) 1906 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.053127+0000 mon.a (mon.0) 1907 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.056629+0000 mon.a (mon.0) 1908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.091301+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.091578+0000 mon.a (mon.0) 1909 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.093075+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.093400+0000 mon.a (mon.0) 1910 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.093907+0000 mon.c (mon.2) 490 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.094220+0000 mon.a (mon.0) 1911 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[18629]: audit 2024-09-10T18:57:26.111071+0000 mon.c (mon.2) 491 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.514413+0000 osd.7 (osd.7) 180 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.517205+0000 osd.7 (osd.7) 181 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.628920+0000 osd.1 (osd.1) 187 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:25.633361+0000 osd.1 (osd.1) 188 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.004710+0000 mgr.y (mgr.24526) 334 : cluster [DBG] pgmap v418: 266 pgs: 8 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 26 op/s 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.052249+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.052605+0000 mon.a (mon.0) 1905 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi083-77640-37"}]': finished 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.053013+0000 mon.a (mon.0) 1906 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.053127+0000 mon.a (mon.0) 1907 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.056629+0000 mon.a (mon.0) 1908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.091301+0000 mon.c (mon.2) 488 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.091578+0000 mon.a (mon.0) 1909 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.093075+0000 mon.c (mon.2) 489 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.093400+0000 mon.a (mon.0) 1910 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:26.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.093907+0000 mon.c (mon.2) 490 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.094220+0000 mon.a (mon.0) 1911 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:26.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:26 smithi083 bash[32381]: audit 2024-09-10T18:57:26.111071+0000 mon.c (mon.2) 491 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:26.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.897 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:26.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:27.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:27.029 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:27.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:27.164 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:27.268 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:27.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:57:27.306 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:27.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:27.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:27.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:27.582 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:27.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.494418+0000 osd.7 (osd.7) 182 : cluster [DBG] 64.1d scrub starts 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.499066+0000 osd.7 (osd.7) 183 : cluster [DBG] 64.1d scrub ok 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.646637+0000 osd.0 (osd.0) 186 : cluster [DBG] 16.18 scrub starts 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:26.650380+0000 osd.0 (osd.0) 187 : cluster [DBG] 16.18 scrub ok 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:27.046022+0000 mon.a (mon.0) 1912 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:27.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.058163+0000 mon.a (mon.0) 1913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.058505+0000 mon.a (mon.0) 1914 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: cluster 2024-09-10T18:57:27.058664+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.063509+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.064060+0000 mon.c (mon.2) 492 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.065113+0000 mon.a (mon.0) 1917 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[22591]: audit 2024-09-10T18:57:27.112171+0000 mon.c (mon.2) 493 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:27.716 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.494418+0000 osd.7 (osd.7) 182 : cluster [DBG] 64.1d scrub starts 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.499066+0000 osd.7 (osd.7) 183 : cluster [DBG] 64.1d scrub ok 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.646637+0000 osd.0 (osd.0) 186 : cluster [DBG] 16.18 scrub starts 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:26.650380+0000 osd.0 (osd.0) 187 : cluster [DBG] 16.18 scrub ok 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:27.046022+0000 mon.a (mon.0) 1912 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.058163+0000 mon.a (mon.0) 1913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.058505+0000 mon.a (mon.0) 1914 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: cluster 2024-09-10T18:57:27.058664+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.063509+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.064060+0000 mon.c (mon.2) 492 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.065113+0000 mon.a (mon.0) 1917 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[18629]: audit 2024-09-10T18:57:27.112171+0000 mon.c (mon.2) 493 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.494418+0000 osd.7 (osd.7) 182 : cluster [DBG] 64.1d scrub starts 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.499066+0000 osd.7 (osd.7) 183 : cluster [DBG] 64.1d scrub ok 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.646637+0000 osd.0 (osd.0) 186 : cluster [DBG] 16.18 scrub starts 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:26.650380+0000 osd.0 (osd.0) 187 : cluster [DBG] 16.18 scrub ok 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:27.046022+0000 mon.a (mon.0) 1912 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.058163+0000 mon.a (mon.0) 1913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-29"}]': finished 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.058505+0000 mon.a (mon.0) 1914 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi083-77640-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: cluster 2024-09-10T18:57:27.058664+0000 mon.a (mon.0) 1915 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-09-10T18:57:27.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.063509+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:27.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.064060+0000 mon.c (mon.2) 492 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.065113+0000 mon.a (mon.0) 1917 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:27.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:27 smithi083 bash[32381]: audit 2024-09-10T18:57:27.112171+0000 mon.c (mon.2) 493 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:27.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:27.852 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:27.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:57:27.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:27.992 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:27.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:27.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:28.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:28.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:28.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:28.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:28.253 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:28.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:28.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:28.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:28.526 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:28.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:28.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:28.759 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:28.791 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:28.824 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:28.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:28.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:28.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:28.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:28.945 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T18:57:28.946 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:57:29.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:29.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: cluster 2024-09-10T18:57:27.955715+0000 osd.7 (osd.7) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: cluster 2024-09-10T18:57:27.956070+0000 osd.7 (osd.7) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.005847+0000 mon.c (mon.2) 494 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.007803+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.058665+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.113514+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.510600+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.510764+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[18629]: audit 2024-09-10T18:57:28.511981+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: cluster 2024-09-10T18:57:27.955715+0000 osd.7 (osd.7) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: cluster 2024-09-10T18:57:27.956070+0000 osd.7 (osd.7) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.005847+0000 mon.c (mon.2) 494 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.007803+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.058665+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.113514+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.510600+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.510764+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:57:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:28 smithi083 bash[32381]: audit 2024-09-10T18:57:28.511981+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:29.063 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:29.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:29.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:29.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:29.337 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: cluster 2024-09-10T18:57:27.955715+0000 osd.7 (osd.7) 184 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: cluster 2024-09-10T18:57:27.956070+0000 osd.7 (osd.7) 185 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.005847+0000 mon.c (mon.2) 494 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.007803+0000 mon.a (mon.0) 1918 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.058665+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.113514+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.510600+0000 mon.a (mon.0) 1920 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.510764+0000 mon.a (mon.0) 1921 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-09-10T18:57:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:28 smithi165 bash[22591]: audit 2024-09-10T18:57:28.511981+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]: dispatch 2024-09-10T18:57:29.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:29.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:29.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T18:57:29.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:29.749 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:29.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.882 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:29.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:29.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:27.794357+0000 mgr.y (mgr.24526) 335 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.005352+0000 mgr.y (mgr.24526) 336 : cluster [DBG] pgmap v421: 257 pgs: 1 active+clean+snaptrim, 1 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1022 KiB/s wr, 25 op/s 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.503573+0000 osd.7 (osd.7) 186 : cluster [DBG] 64.4 scrub starts 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.508803+0000 osd.7 (osd.7) 187 : cluster [DBG] 64.4 scrub ok 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.774075+0000 osd.2 (osd.2) 111 : cluster [DBG] 16.1a scrub starts 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:28.776679+0000 osd.2 (osd.2) 112 : cluster [DBG] 16.1a scrub ok 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.064185+0000 mon.a (mon.0) 1923 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:29.999 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.064424+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.064650+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.068567+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.078191+0000 mon.a (mon.0) 1926 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.092257+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.093672+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.094003+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.103006+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.f scrub starts 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.105048+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.f scrub ok 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.114759+0000 mon.c (mon.2) 496 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.567512+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:30.000 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:29 smithi165 bash[22591]: audit 2024-09-10T18:57:29.570141+0000 mon.c (mon.2) 497 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:30.016 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:27.794357+0000 mgr.y (mgr.24526) 335 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.005352+0000 mgr.y (mgr.24526) 336 : cluster [DBG] pgmap v421: 257 pgs: 1 active+clean+snaptrim, 1 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1022 KiB/s wr, 25 op/s 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.503573+0000 osd.7 (osd.7) 186 : cluster [DBG] 64.4 scrub starts 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.508803+0000 osd.7 (osd.7) 187 : cluster [DBG] 64.4 scrub ok 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.774075+0000 osd.2 (osd.2) 111 : cluster [DBG] 16.1a scrub starts 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:28.776679+0000 osd.2 (osd.2) 112 : cluster [DBG] 16.1a scrub ok 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.064185+0000 mon.a (mon.0) 1923 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.064424+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:30.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.064650+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.068567+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.078191+0000 mon.a (mon.0) 1926 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.092257+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.093672+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.094003+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.103006+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.f scrub starts 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.105048+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.f scrub ok 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.114759+0000 mon.c (mon.2) 496 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.567512+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[32381]: audit 2024-09-10T18:57:29.570141+0000 mon.c (mon.2) 497 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:27.794357+0000 mgr.y (mgr.24526) 335 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.005352+0000 mgr.y (mgr.24526) 336 : cluster [DBG] pgmap v421: 257 pgs: 1 active+clean+snaptrim, 1 clean+premerge+peered, 255 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1022 KiB/s wr, 25 op/s 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.503573+0000 osd.7 (osd.7) 186 : cluster [DBG] 64.4 scrub starts 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.508803+0000 osd.7 (osd.7) 187 : cluster [DBG] 64.4 scrub ok 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.774075+0000 osd.2 (osd.2) 111 : cluster [DBG] 16.1a scrub starts 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:28.776679+0000 osd.2 (osd.2) 112 : cluster [DBG] 16.1a scrub ok 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.064185+0000 mon.a (mon.0) 1923 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi083-77640-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.064424+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? v1:172.21.15.83:0/3264741116' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi083-77648-48"}]': finished 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.064650+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.068567+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.078191+0000 mon.a (mon.0) 1926 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.092257+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.093672+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.094003+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:30.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.103006+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.f scrub starts 2024-09-10T18:57:30.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.105048+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.f scrub ok 2024-09-10T18:57:30.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.114759+0000 mon.c (mon.2) 496 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:30.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.567512+0000 mon.a (mon.0) 1930 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:30.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:29 smithi083 bash[18629]: audit 2024-09-10T18:57:29.570141+0000 mon.c (mon.2) 497 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:30.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:30.149 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:30.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:30.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.284 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:30.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:30.413 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:30.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:30.549 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:30.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.688 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:30.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:30.823 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:30.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:30.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:31.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.088 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:31.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:31.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:31.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:31.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:31.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:31.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.635 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:31.636 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.477143+0000 osd.7 (osd.7) 188 : cluster [DBG] 64.6 scrub starts 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.481877+0000 osd.7 (osd.7) 189 : cluster [DBG] 64.6 scrub ok 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.697412+0000 osd.4 (osd.4) 209 : cluster [DBG] 16.12 scrub starts 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:29.700919+0000 osd.4 (osd.4) 210 : cluster [DBG] 16.12 scrub ok 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.006316+0000 mgr.y (mgr.24526) 337 : cluster [DBG] pgmap v424: 265 pgs: 40 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 223 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.066862+0000 mon.a (mon.0) 1931 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.067073+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.067151+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.069518+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.638 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.072763+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.073444+0000 mon.a (mon.0) 1935 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: audit 2024-09-10T18:57:30.115845+0000 mon.c (mon.2) 498 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.245187+0000 osd.5 (osd.5) 171 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.249110+0000 osd.5 (osd.5) 172 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.610186+0000 osd.1 (osd.1) 189 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.615927+0000 osd.1 (osd.1) 190 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:57:31.639 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:31 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.926402+0000 mon.a (mon.0) 1936 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:31.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:31.773 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.477143+0000 osd.7 (osd.7) 188 : cluster [DBG] 64.6 scrub starts 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.481877+0000 osd.7 (osd.7) 189 : cluster [DBG] 64.6 scrub ok 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.697412+0000 osd.4 (osd.4) 209 : cluster [DBG] 16.12 scrub starts 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:29.700919+0000 osd.4 (osd.4) 210 : cluster [DBG] 16.12 scrub ok 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.006316+0000 mgr.y (mgr.24526) 337 : cluster [DBG] pgmap v424: 265 pgs: 40 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 223 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.066862+0000 mon.a (mon.0) 1931 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:31.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.067073+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.067151+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.069518+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.072763+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.073444+0000 mon.a (mon.0) 1935 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: audit 2024-09-10T18:57:30.115845+0000 mon.c (mon.2) 498 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.245187+0000 osd.5 (osd.5) 171 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.249110+0000 osd.5 (osd.5) 172 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.610186+0000 osd.1 (osd.1) 189 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.615927+0000 osd.1 (osd.1) 190 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:57:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.926402+0000 mon.a (mon.0) 1936 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.477143+0000 osd.7 (osd.7) 188 : cluster [DBG] 64.6 scrub starts 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.481877+0000 osd.7 (osd.7) 189 : cluster [DBG] 64.6 scrub ok 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.697412+0000 osd.4 (osd.4) 209 : cluster [DBG] 16.12 scrub starts 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:29.700919+0000 osd.4 (osd.4) 210 : cluster [DBG] 16.12 scrub ok 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.006316+0000 mgr.y (mgr.24526) 337 : cluster [DBG] pgmap v424: 265 pgs: 40 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 223 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.066862+0000 mon.a (mon.0) 1931 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.067073+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi083-77648-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.067151+0000 mon.a (mon.0) 1933 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.069518+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.072763+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.073444+0000 mon.a (mon.0) 1935 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: audit 2024-09-10T18:57:30.115845+0000 mon.c (mon.2) 498 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.245187+0000 osd.5 (osd.5) 171 : cluster [DBG] 2.12 scrub starts 2024-09-10T18:57:31.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.249110+0000 osd.5 (osd.5) 172 : cluster [DBG] 2.12 scrub ok 2024-09-10T18:57:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.610186+0000 osd.1 (osd.1) 189 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:57:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.615927+0000 osd.1 (osd.1) 190 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:57:31.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:31 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.926402+0000 mon.a (mon.0) 1936 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:31.869 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:31.870 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:31.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:31.899 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:31.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:32.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:32.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:32.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:32.170 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:32.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:32.305 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:32.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:32.443 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:32.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:32.578 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:32.682 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.796942+0000 osd.2 (osd.2) 113 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:30.800236+0000 osd.2 (osd.2) 114 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.105439+0000 osd.3 (osd.3) 203 : cluster [DBG] 16.4 scrub starts 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.116954+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.273655+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.277076+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.517213+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.690 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.518930+0000 mon.a (mon.0) 1937 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.519385+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.520606+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.521538+0000 osd.6 (osd.6) 139 : cluster [DBG] 16.16 scrub starts 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.521562+0000 mon.a (mon.0) 1939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.530681+0000 osd.6 (osd.6) 140 : cluster [DBG] 16.16 scrub ok 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:31.531813+0000 mon.a (mon.0) 1940 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.684331+0000 osd.0 (osd.0) 188 : cluster [DBG] 12.10 scrub starts 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.687863+0000 osd.0 (osd.0) 189 : cluster [DBG] 12.10 scrub ok 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.713265+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:31.717494+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.007152+0000 mgr.y (mgr.24526) 338 : cluster [DBG] pgmap v427: 256 pgs: 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:32.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.007773+0000 mon.c (mon.2) 501 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.796942+0000 osd.2 (osd.2) 113 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:30.800236+0000 osd.2 (osd.2) 114 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.105439+0000 osd.3 (osd.3) 203 : cluster [DBG] 16.4 scrub starts 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.116954+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.273655+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.277076+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.517213+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.518930+0000 mon.a (mon.0) 1937 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:32.692 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.519385+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.520606+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.521538+0000 osd.6 (osd.6) 139 : cluster [DBG] 16.16 scrub starts 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.521562+0000 mon.a (mon.0) 1939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.530681+0000 osd.6 (osd.6) 140 : cluster [DBG] 16.16 scrub ok 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:31.531813+0000 mon.a (mon.0) 1940 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.684331+0000 osd.0 (osd.0) 188 : cluster [DBG] 12.10 scrub starts 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.687863+0000 osd.0 (osd.0) 189 : cluster [DBG] 12.10 scrub ok 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.713265+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:31.717494+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.007152+0000 mgr.y (mgr.24526) 338 : cluster [DBG] pgmap v427: 256 pgs: 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.007773+0000 mon.c (mon.2) 501 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.010068+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.074935+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.075023+0000 mon.a (mon.0) 1943 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.075187+0000 mon.a (mon.0) 1944 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.075253+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.075312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.077267+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.077395+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.077804+0000 mon.a (mon.0) 1947 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.081288+0000 mon.a (mon.0) 1948 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[32381]: audit 2024-09-10T18:57:32.118433+0000 mon.c (mon.2) 503 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.010068+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.074935+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.075023+0000 mon.a (mon.0) 1943 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.075187+0000 mon.a (mon.0) 1944 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.075253+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.075312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.077267+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.077395+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.077804+0000 mon.a (mon.0) 1947 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.081288+0000 mon.a (mon.0) 1948 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18629]: audit 2024-09-10T18:57:32.118433+0000 mon.c (mon.2) 503 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:32.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:32.858 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.796942+0000 osd.2 (osd.2) 113 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:30.800236+0000 osd.2 (osd.2) 114 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.105439+0000 osd.3 (osd.3) 203 : cluster [DBG] 16.4 scrub starts 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.116954+0000 mon.c (mon.2) 499 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.273655+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub starts 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.277076+0000 osd.5 (osd.5) 174 : cluster [DBG] 6.19 scrub ok 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.517213+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.518930+0000 mon.a (mon.0) 1937 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.519385+0000 mon.a (mon.0) 1938 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.520606+0000 mon.c (mon.2) 500 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.521538+0000 osd.6 (osd.6) 139 : cluster [DBG] 16.16 scrub starts 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.521562+0000 mon.a (mon.0) 1939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.530681+0000 osd.6 (osd.6) 140 : cluster [DBG] 16.16 scrub ok 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:31.531813+0000 mon.a (mon.0) 1940 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.684331+0000 osd.0 (osd.0) 188 : cluster [DBG] 12.10 scrub starts 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.687863+0000 osd.0 (osd.0) 189 : cluster [DBG] 12.10 scrub ok 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.713265+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.9 scrub starts 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:31.717494+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.9 scrub ok 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.007152+0000 mgr.y (mgr.24526) 338 : cluster [DBG] pgmap v427: 256 pgs: 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.007773+0000 mon.c (mon.2) 501 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.010068+0000 mon.a (mon.0) 1941 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-09-10T18:57:32.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.074935+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi083-77648-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.075023+0000 mon.a (mon.0) 1943 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.075187+0000 mon.a (mon.0) 1944 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.075253+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.075312+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.077267+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.077395+0000 mon.c (mon.2) 502 : audit [INF] from='client.? v1:172.21.15.83:0/1825483525' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.077804+0000 mon.a (mon.0) 1947 : audit [INF] from='client.49115 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]: dispatch 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.081288+0000 mon.a (mon.0) 1948 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]: dispatch 2024-09-10T18:57:32.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:32 smithi165 bash[22591]: audit 2024-09-10T18:57:32.118433+0000 mon.c (mon.2) 503 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:32.962 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:32.973 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:32] "GET /metrics HTTP/1.1" 200 233523 "" "Prometheus/2.43.0" 2024-09-10T18:57:32.995 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:32.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:33.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:33.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:33.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.397 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:33.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:33.532 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:33.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.231764+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.232141+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.261612+0000 osd.5 (osd.5) 175 : cluster [DBG] 64.18 scrub starts 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.266999+0000 osd.5 (osd.5) 176 : cluster [DBG] 64.18 scrub ok 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.760320+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.e scrub starts 2024-09-10T18:57:33.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.764551+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.e scrub ok 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:33.068681+0000 mon.a (mon.0) 1949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.078607+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.078703+0000 mon.a (mon.0) 1951 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]': finished 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: cluster 2024-09-10T18:57:33.078831+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.109229+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.111247+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.111755+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.119606+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.236718+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.649 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:33 smithi165 bash[22591]: audit 2024-09-10T18:57:33.240985+0000 mon.a (mon.0) 1956 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.662 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:33.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.799 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:33.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.231764+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.232141+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.261612+0000 osd.5 (osd.5) 175 : cluster [DBG] 64.18 scrub starts 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.266999+0000 osd.5 (osd.5) 176 : cluster [DBG] 64.18 scrub ok 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.760320+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.e scrub starts 2024-09-10T18:57:33.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.764551+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.e scrub ok 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:33.068681+0000 mon.a (mon.0) 1949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.078607+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.078703+0000 mon.a (mon.0) 1951 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]': finished 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: cluster 2024-09-10T18:57:33.078831+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.109229+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.111247+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.111755+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.119606+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.236718+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[18629]: audit 2024-09-10T18:57:33.240985+0000 mon.a (mon.0) 1956 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.231764+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.232141+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.261612+0000 osd.5 (osd.5) 175 : cluster [DBG] 64.18 scrub starts 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.266999+0000 osd.5 (osd.5) 176 : cluster [DBG] 64.18 scrub ok 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.760320+0000 osd.4 (osd.4) 213 : cluster [DBG] 4.e scrub starts 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.764551+0000 osd.4 (osd.4) 214 : cluster [DBG] 4.e scrub ok 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:33.068681+0000 mon.a (mon.0) 1949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.078607+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49115 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi083-77640-38"}]': finished 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.078703+0000 mon.a (mon.0) 1951 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-31", "mode": "writeback"}]': finished 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: cluster 2024-09-10T18:57:33.078831+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.109229+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.111247+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.111755+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:33.919 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.119606+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:33.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.236718+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:33 smithi083 bash[32381]: audit 2024-09-10T18:57:33.240985+0000 mon.a (mon.0) 1956 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:33.938 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:33.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:34.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:34.070 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:34.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:34.208 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:34.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:34.347 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:34.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- 172.21.15.83:0/102927257 >> v1:172.21.15.83:6790/0 conn(0x7f40d40fd610 legacy=0x7f40d40fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- 172.21.15.83:0/102927257 shutdown_connections 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- 172.21.15.83:0/102927257 >> 172.21.15.83:0/102927257 conn(0x7f40d4075290 msgr2=0x7f40d40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- 172.21.15.83:0/102927257 shutdown_connections 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- 172.21.15.83:0/102927257 wait complete. 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 librados: starting msgr at 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 librados: starting objecter 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 Processor -- start 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- start start 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 librados: setting wanted keys 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 librados: calling monclient init 2024-09-10T18:57:34.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40d411e6c0 con 0x7f40d41140c0 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40d411f880 con 0x7f40d4108f40 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.415+0000 7f40da339640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40d4120a40 con 0x7f40d40fd610 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d3fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f40d40fd610 0x7f40d4113790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47768/0 (socket says 172.21.15.83:47768) 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d3fff640 1 -- 172.21.15.83:0/3076336184 learned_addr learned my addr 172.21.15.83:0/3076336184 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d88d6640 1 --1- 172.21.15.83:0/3076336184 >> v1:172.21.15.83:6789/0 conn(0x7f40d41140c0 0x7f40d411cde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44442/0 (socket says 172.21.15.83:44442) 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter ms_handle_connect 0x7f40d40fd610 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter ms_handle_connect 0x7f40d41140c0 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter ms_handle_connect 0x7f40d4108f40 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206962491 0 0) 0x7f40d4120a40 con 0x7f40d40fd610 2024-09-10T18:57:34.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c00038e0 con 0x7f40d40fd610 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342237035 0 0) 0x7f40d411e6c0 con 0x7f40d41140c0 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d4120a40 con 0x7f40d41140c0 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156629738 0 0) 0x7f40d411f880 con 0x7f40d4108f40 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d411e6c0 con 0x7f40d4108f40 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3551935701 0 0) 0x7f40c00038e0 con 0x7f40d40fd610 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40d411f880 con 0x7f40d40fd610 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590680162 0 0) 0x7f40d4120a40 con 0x7f40d41140c0 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40c00038e0 con 0x7f40d41140c0 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f40bc002c70 con 0x7f40d40fd610 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f40cc001bc0 con 0x7f40d41140c0 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4045999908 0 0) 0x7f40d411f880 con 0x7f40d40fd610 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 >> v1:172.21.15.165:6789/0 conn(0x7f40d4108f40 legacy=0x7f40d41196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:34.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 >> v1:172.21.15.83:6789/0 conn(0x7f40d41140c0 legacy=0x7f40d411cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d4121c40 con 0x7f40d40fd610 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d411e8b0 con 0x7f40d40fd610 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40da339640 10 client.49130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d411ed00 con 0x7f40d40fd610 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40da339640 1 librados: init done 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f40bc002360 con 0x7f40d40fd610 2024-09-10T18:57:34.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.419+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f40bc003620 con 0x7f40d40fd610 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b4005010 con 0x7f40d40fd610 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f40bc001f00 con 0x7f40d40fd610 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(298..298 src has 69..298) v4 ==== 8991+0+0 (unknown 215838532 0 0) 0x7f40bc08acc0 con 0x7f40d40fd610 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 10 client.49130.objecter ms_dispatch 0x7f40d4000b90 osd_map(298..298 src has 69..298) v4 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 3 client.49130.objecter handle_osd_map got epochs [298,298] > 0 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 3 client.49130.objecter handle_osd_map decoding full epoch 298 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 20 client.49130.objecter dump_active .. 0 homeless 2024-09-10T18:57:34.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.423+0000 7f40d0ff9640 10 client.49130.objecter ms_handle_connect 0x7f40c00717a0 2024-09-10T18:57:34.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.431+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f40bc056ec0 con 0x7f40d40fd610 2024-09-10T18:57:34.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:34.469 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:34.479 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:34.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.790964+0000 osd.2 (osd.2) 115 : cluster [DBG] 64.a scrub starts 2024-09-10T18:57:34.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:32.795138+0000 osd.2 (osd.2) 116 : cluster [DBG] 64.a scrub ok 2024-09-10T18:57:34.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:33.404601+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:57:34.576 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:33.408645+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.007894+0000 mgr.y (mgr.24526) 339 : cluster [DBG] pgmap v430: 264 pgs: 8 unknown, 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.044194+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.045622+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.045780+0000 mon.a (mon.0) 1958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.045885+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.047773+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.051291+0000 mon.a (mon.0) 1961 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.059606+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:34.577 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:34 smithi165 bash[22591]: audit 2024-09-10T18:57:34.120830+0000 mon.c (mon.2) 505 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:34.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:34.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:34.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.615+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40b4000b90 con 0x7f40c00717a0 2024-09-10T18:57:34.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.635+0000 7f40d0ff9640 1 -- 172.21.15.83:0/3076336184 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+693282 (unknown 2965378022 0 3110977087) 0x7f40b4000b90 con 0x7f40c00717a0 2024-09-10T18:57:34.650 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:57:34.655 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:57:34.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 10 librados: watch_flush enter 2024-09-10T18:57:34.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 10 librados: watch_flush exit 2024-09-10T18:57:34.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 20 client.49130.objecter shutdown clearing up homeless session... 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 10 client.49130.objecter successfully canceled tick 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 >> v1:172.21.15.83:6800/3071793735 conn(0x7f40c00717a0 legacy=0x7f40c0073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 >> v1:172.21.15.83:6790/0 conn(0x7f40d40fd610 legacy=0x7f40d4113790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 shutdown_connections 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 >> 172.21.15.83:0/3076336184 conn(0x7f40d4075290 msgr2=0x7f40d40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 shutdown_connections 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 -- 172.21.15.83:0/3076336184 wait complete. 2024-09-10T18:57:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:34.647+0000 7f40da339640 1 librados: shutdown 2024-09-10T18:57:34.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:34.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:34.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.790964+0000 osd.2 (osd.2) 115 : cluster [DBG] 64.a scrub starts 2024-09-10T18:57:34.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:32.795138+0000 osd.2 (osd.2) 116 : cluster [DBG] 64.a scrub ok 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:33.404601+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:33.408645+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.007894+0000 mgr.y (mgr.24526) 339 : cluster [DBG] pgmap v430: 264 pgs: 8 unknown, 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.044194+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.045622+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.045780+0000 mon.a (mon.0) 1958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.045885+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.047773+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.051291+0000 mon.a (mon.0) 1961 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.059606+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:34.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[18629]: audit 2024-09-10T18:57:34.120830+0000 mon.c (mon.2) 505 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.790964+0000 osd.2 (osd.2) 115 : cluster [DBG] 64.a scrub starts 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:32.795138+0000 osd.2 (osd.2) 116 : cluster [DBG] 64.a scrub ok 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:33.404601+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.11 scrub starts 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:33.408645+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.11 scrub ok 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.007894+0000 mgr.y (mgr.24526) 339 : cluster [DBG] pgmap v430: 264 pgs: 8 unknown, 1 peering, 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.044194+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.045622+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi083-77640-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.045780+0000 mon.a (mon.0) 1958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.045885+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.047773+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.051291+0000 mon.a (mon.0) 1961 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]: dispatch 2024-09-10T18:57:34.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.059606+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:34.783 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:34 smithi083 bash[32381]: audit 2024-09-10T18:57:34.120830+0000 mon.c (mon.2) 505 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:34.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:34.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:34.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:34.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:35.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:35.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:35.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:35.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:35.301 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:35.405 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:35.440 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:35.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:35.574 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:35.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:35.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.242014+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.244341+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.375233+0000 osd.7 (osd.7) 192 : cluster [DBG] 2.b scrub starts 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.378973+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub ok 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: audit 2024-09-10T18:57:34.621500+0000 mgr.y (mgr.24526) 340 : audit [DBG] from='client.49130 v1:172.21.15.83:0/3076336184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.040332+0000 mon.a (mon.0) 1963 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: audit 2024-09-10T18:57:35.122369+0000 mon.c (mon.2) 506 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: audit 2024-09-10T18:57:35.494687+0000 mon.a (mon.0) 1964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:35.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: audit 2024-09-10T18:57:35.494932+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.495145+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[18629]: audit 2024-09-10T18:57:35.497557+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.242014+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.244341+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.375233+0000 osd.7 (osd.7) 192 : cluster [DBG] 2.b scrub starts 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.378973+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub ok 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: audit 2024-09-10T18:57:34.621500+0000 mgr.y (mgr.24526) 340 : audit [DBG] from='client.49130 v1:172.21.15.83:0/3076336184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.040332+0000 mon.a (mon.0) 1963 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:35.719 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: audit 2024-09-10T18:57:35.122369+0000 mon.c (mon.2) 506 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:35.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: audit 2024-09-10T18:57:35.494687+0000 mon.a (mon.0) 1964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:35.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: audit 2024-09-10T18:57:35.494932+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:35.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.495145+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:57:35.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:35 smithi083 bash[32381]: audit 2024-09-10T18:57:35.497557+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.242014+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.244341+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.375233+0000 osd.7 (osd.7) 192 : cluster [DBG] 2.b scrub starts 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.378973+0000 osd.7 (osd.7) 193 : cluster [DBG] 2.b scrub ok 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: audit 2024-09-10T18:57:34.621500+0000 mgr.y (mgr.24526) 340 : audit [DBG] from='client.49130 v1:172.21.15.83:0/3076336184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.040332+0000 mon.a (mon.0) 1963 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: audit 2024-09-10T18:57:35.122369+0000 mon.c (mon.2) 506 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: audit 2024-09-10T18:57:35.494687+0000 mon.a (mon.0) 1964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-31"}]': finished 2024-09-10T18:57:35.808 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: audit 2024-09-10T18:57:35.494932+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:35.809 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.495145+0000 mon.a (mon.0) 1966 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-09-10T18:57:35.809 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:35 smithi165 bash[22591]: audit 2024-09-10T18:57:35.497557+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]: dispatch 2024-09-10T18:57:35.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:35.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:35.983 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:35.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:36.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:36.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:36.229 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-10T18:57:34.040651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793483,"num_objects":330,"num_object_clones":1,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":1442,"num_read_kb":1204,"num_write":1179,"num_write_kb":10455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":591484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2519,"ondisk_log_size":2519,"up":765,"acting":765,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":757,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2503224,"kb_used_data":49400,"kb_used_omap":13,"kb_used_meta":2453618,"kb_avail":747457992,"statfs":{"total":767960285184,"available":765396983808,"internally_reserved":0,"allocated":50585600,"data_stored":34211957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2512505536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4194432,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-3,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.978430"},"pg_stats":[{"pgid":"176.17","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090409+0000","last_change":"2024-09-10T18:57:30.090409+0000","last_active":"2024-09-10T18:57:30.090409+0000","last_peered":"2024-09-10T18:57:30.090409+0000","last_clean":"2024-09-10T18:57:30.090409+0000","last_became_active":"2024-09-10T18:57:30.090112+0000","last_became_peered":"2024-09-10T18:57:30.090112+0000","last_unstale":"2024-09-10T18:57:30.090409+0000","last_undegraded":"2024-09-10T18:57:30.090409+0000","last_fullsized":"2024-09-10T18:57:30.090409+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:46.512515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"176.16","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.085582+0000","last_change":"2024-09-10T18:57:30.085582+0000","last_active":"2024-09-10T18:57:30.085582+0000","last_peered":"2024-09-10T18:57:30.085582+0000","last_clean":"2024-09-10T18:57:30.085582+0000","last_became_active":"2024-09-10T18:57:30.085246+0000","last_became_peered":"2024-09-10T18:57:30.085246+0000","last_unstale":"2024-09-10T18:57:30.085582+0000","last_undegraded":"2024-09-10T18:57:30.085582+0000","last_fullsized":"2024-09-10T18:57:30.085582+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:38.742430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"176.15","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.091491+0000","last_change":"2024-09-10T18:57:30.091491+0000","last_active":"2024-09-10T18:57:30.091491+0000","last_peered":"2024-09-10T18:57:30.091491+0000","last_clean":"2024-09-10T18:57:30.091491+0000","last_became_active":"2024-09-10T18:57:30.091142+0000","last_became_peered":"2024-09-10T18:57:30.091142+0000","last_unstale":"2024-09-10T18:57:30.091491+0000","last_undegraded":"2024-09-10T18:57:30.091491+0000","last_fullsized":"2024-09-10T18:57:30.091491+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.102593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.14","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.085982+0000","last_change":"2024-09-10T18:57:30.085982+0000","last_active":"2024-09-10T18:57:30.085982+0000","last_peered":"2024-09-10T18:57:30.085982+0000","last_clean":"2024-09-10T18:57:30.085982+0000","last_became_active":"2024-09-10T18:57:30.085377+0000","last_became_peered":"2024-09-10T18:57:30.085377+0000","last_unstale":"2024-09-10T18:57:30.085982+0000","last_undegraded":"2024-09-10T18:57:30.085982+0000","last_fullsized":"2024-09-10T18:57:30.085982+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.392239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.13","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.092633+0000","last_change":"2024-09-10T18:57:30.092633+0000","last_active":"2024-09-10T18:57:30.092633+0000","last_peered":"2024-09-10T18:57:30.092633+0000","last_clean":"2024-09-10T18:57:30.092633+0000","last_became_active":"2024-09-10T18:57:30.092348+0000","last_became_peered":"2024-09-10T18:57:30.092348+0000","last_unstale":"2024-09-10T18:57:30.092633+0000","last_undegraded":"2024-09-10T18:57:30.092633+0000","last_fullsized":"2024-09-10T18:57:30.092633+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.584385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.12","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.086300+0000","last_change":"2024-09-10T18:57:30.086300+0000","last_active":"2024-09-10T18:57:30.086300+0000","last_peered":"2024-09-10T18:57:30.086300+0000","last_clean":"2024-09-10T18:57:30.086300+0000","last_became_active":"2024-09-10T18:57:30.085905+0000","last_became_peered":"2024-09-10T18:57:30.085905+0000","last_unstale":"2024-09-10T18:57:30.086300+0000","last_undegraded":"2024-09-10T18:57:30.086300+0000","last_fullsized":"2024-09-10T18:57:30.086300+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:39.440965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.11","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.091744+0000","last_change":"2024-09-10T18:57:30.091744+0000","last_active":"2024-09-10T18:57:30.091744+0000","last_peered":"2024-09-10T18:57:30.091744+0000","last_clean":"2024-09-10T18:57:30.091744+0000","last_became_active":"2024-09-10T18:57:30.091176+0000","last_became_peered":"2024-09-10T18:57:30.091176+0000","last_unstale":"2024-09-10T18:57:30.091744+0000","last_undegraded":"2024-09-10T18:57:30.091744+0000","last_fullsized":"2024-09-10T18:57:30.091744+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:38.262964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.10","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090272+0000","last_change":"2024-09-10T18:57:30.090272+0000","last_active":"2024-09-10T18:57:30.090272+0000","last_peered":"2024-09-10T18:57:30.090272+0000","last_clean":"2024-09-10T18:57:30.090272+0000","last_became_active":"2024-09-10T18:57:30.089957+0000","last_became_peered":"2024-09-10T18:57:30.089957+0000","last_unstale":"2024-09-10T18:57:30.090272+0000","last_undegraded":"2024-09-10T18:57:30.090272+0000","last_fullsized":"2024-09-10T18:57:30.090272+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.895108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.f","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.088424+0000","last_change":"2024-09-10T18:57:30.088424+0000","last_active":"2024-09-10T18:57:30.088424+0000","last_peered":"2024-09-10T18:57:30.088424+0000","last_clean":"2024-09-10T18:57:30.088424+0000","last_became_active":"2024-09-10T18:57:30.088135+0000","last_became_peered":"2024-09-10T18:57:30.088135+0000","last_unstale":"2024-09-10T18:57:30.088424+0000","last_undegraded":"2024-09-10T18:57:30.088424+0000","last_fullsized":"2024-09-10T18:57:30.088424+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:36.453457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.e","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.086571+0000","last_change":"2024-09-10T18:57:30.086571+0000","last_active":"2024-09-10T18:57:30.086571+0000","last_peered":"2024-09-10T18:57:30.086571+0000","last_clean":"2024-09-10T18:57:30.086571+0000","last_became_active":"2024-09-10T18:57:30.086063+0000","last_became_peered":"2024-09-10T18:57:30.086063+0000","last_unstale":"2024-09-10T18:57:30.086571+0000","last_undegraded":"2024-09-10T18:57:30.086571+0000","last_fullsized":"2024-09-10T18:57:30.086571+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:29.086775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.d","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.081670+0000","last_change":"2024-09-10T18:57:30.081670+0000","last_active":"2024-09-10T18:57:30.081670+0000","last_peered":"2024-09-10T18:57:30.081670+0000","last_clean":"2024-09-10T18:57:30.081670+0000","last_became_active":"2024-09-10T18:57:30.081317+0000","last_became_peered":"2024-09-10T18:57:30.081317+0000","last_unstale":"2024-09-10T18:57:30.081670+0000","last_undegraded":"2024-09-10T18:57:30.081670+0000","last_fullsized":"2024-09-10T18:57:30.081670+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.179495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.c","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090270+0000","last_change":"2024-09-10T18:57:30.090270+0000","last_active":"2024-09-10T18:57:30.090270+0000","last_peered":"2024-09-10T18:57:30.090270+0000","last_clean":"2024-09-10T18:57:30.090270+0000","last_became_active":"2024-09-10T18:57:30.090024+0000","last_became_peered":"2024-09-10T18:57:30.090024+0000","last_unstale":"2024-09-10T18:57:30.090270+0000","last_undegraded":"2024-09-10T18:57:30.090270+0000","last_fullsized":"2024-09-10T18:57:30.090270+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:39.823445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.b","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.087517+0000","last_change":"2024-09-10T18:57:30.087517+0000","last_active":"2024-09-10T18:57:30.087517+0000","last_peered":"2024-09-10T18:57:30.087517+0000","last_clean":"2024-09-10T18:57:30.087517+0000","last_became_active":"2024-09-10T18:57:30.087245+0000","last_became_peered":"2024-09-10T18:57:30.087245+0000","last_unstale":"2024-09-10T18:57:30.087517+0000","last_undegraded":"2024-09-10T18:57:30.087517+0000","last_fullsized":"2024-09-10T18:57:30.087517+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:42.333810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"176.a","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.093922+0000","last_change":"2024-09-10T18:57:30.093922+0000","last_active":"2024-09-10T18:57:30.093922+0000","last_peered":"2024-09-10T18:57:30.093922+0000","last_clean":"2024-09-10T18:57:30.093922+0000","last_became_active":"2024-09-10T18:57:30.093662+0000","last_became_peered":"2024-09-10T18:57:30.093662+0000","last_unstale":"2024-09-10T18:57:30.093922+0000","last_undegraded":"2024-09-10T18:57:30.093922+0000","last_fullsized":"2024-09-10T18:57:30.093922+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.406625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.9","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.091020+0000","last_change":"2024-09-10T18:57:30.091020+0000","last_active":"2024-09-10T18:57:30.091020+0000","last_peered":"2024-09-10T18:57:30.091020+0000","last_clean":"2024-09-10T18:57:30.091020+0000","last_became_active":"2024-09-10T18:57:30.090708+0000","last_became_peered":"2024-09-10T18:57:30.090708+0000","last_unstale":"2024-09-10T18:57:30.091020+0000","last_undegraded":"2024-09-10T18:57:30.091020+0000","last_fullsized":"2024-09-10T18:57:30.091020+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:42.572274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.8","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090444+0000","last_change":"2024-09-10T18:57:30.090444+0000","last_active":"2024-09-10T18:57:30.090444+0000","last_peered":"2024-09-10T18:57:30.090444+0000","last_clean":"2024-09-10T18:57:30.090444+0000","last_became_active":"2024-09-10T18:57:30.090139+0000","last_became_peered":"2024-09-10T18:57:30.090139+0000","last_unstale":"2024-09-10T18:57:30.090444+0000","last_undegraded":"2024-09-10T18:57:30.090444+0000","last_fullsized":"2024-09-10T18:57:30.090444+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.655747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.7","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090760+0000","last_change":"2024-09-10T18:57:30.090760+0000","last_active":"2024-09-10T18:57:30.090760+0000","last_peered":"2024-09-10T18:57:30.090760+0000","last_clean":"2024-09-10T18:57:30.090760+0000","last_became_active":"2024-09-10T18:57:30.090257+0000","last_became_peered":"2024-09-10T18:57:30.090257+0000","last_unstale":"2024-09-10T18:57:30.090760+0000","last_undegraded":"2024-09-10T18:57:30.090760+0000","last_fullsized":"2024-09-10T18:57:30.090760+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.693113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.6","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.092247+0000","last_change":"2024-09-10T18:57:30.092247+0000","last_active":"2024-09-10T18:57:30.092247+0000","last_peered":"2024-09-10T18:57:30.092247+0000","last_clean":"2024-09-10T18:57:30.092247+0000","last_became_active":"2024-09-10T18:57:30.091782+0000","last_became_peered":"2024-09-10T18:57:30.091782+0000","last_unstale":"2024-09-10T18:57:30.092247+0000","last_undegraded":"2024-09-10T18:57:30.092247+0000","last_fullsized":"2024-09-10T18:57:30.092247+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:42.021663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.5","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.086787+0000","last_change":"2024-09-10T18:57:30.086787+0000","last_active":"2024-09-10T18:57:30.086787+0000","last_peered":"2024-09-10T18:57:30.086787+0000","last_clean":"2024-09-10T18:57:30.086787+0000","last_became_active":"2024-09-10T18:57:30.086379+0000","last_became_peered":"2024-09-10T18:57:30.086379+0000","last_unstale":"2024-09-10T18:57:30.086787+0000","last_undegraded":"2024-09-10T18:57:30.086787+0000","last_fullsized":"2024-09-10T18:57:30.086787+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.427833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"176.4","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.090109+0000","last_change":"2024-09-10T18:57:30.090109+0000","last_active":"2024-09-10T18:57:30.090109+0000","last_peered":"2024-09-10T18:57:30.090109+0000","last_clean":"2024-09-10T18:57:30.090109+0000","last_became_active":"2024-09-10T18:57:30.089625+0000","last_became_peered":"2024-09-10T18:57:30.089625+0000","last_unstale":"2024-09-10T18:57:30.090109+0000","last_undegraded":"2024-09-10T18:57:30.090109+0000","last_fullsized":"2024-09-10T18:57:30.090109+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:44.789902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.3","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.089700+0000","last_change":"2024-09-10T18:57:30.089700+0000","last_active":"2024-09-10T18:57:30.089700+0000","last_peered":"2024-09-10T18:57:30.089700+0000","last_clean":"2024-09-10T18:57:30.089700+0000","last_became_active":"2024-09-10T18:57:30.089433+0000","last_became_peered":"2024-09-10T18:57:30.089433+0000","last_unstale":"2024-09-10T18:57:30.089700+0000","last_undegraded":"2024-09-10T18:57:30.089700+0000","last_fullsized":"2024-09-10T18:57:30.089700+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.359213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"176.2","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.088020+0000","last_change":"2024-09-10T18:57:30.088020+0000","last_active":"2024-09-10T18:57:30.088020+0000","last_peered":"2024-09-10T18:57:30.088020+0000","last_clean":"2024-09-10T18:57:30.088020+0000","last_became_active":"2024-09-10T18:57:30.087600+0000","last_became_peered":"2024-09-10T18:57:30.087600+0000","last_unstale":"2024-09-10T18:57:30.088020+0000","last_undegraded":"2024-09-10T18:57:30.088020+0000","last_fullsized":"2024-09-10T18:57:30.088020+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:33.169846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.1","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.088333+0000","last_change":"2024-09-10T18:57:30.088333+0000","last_active":"2024-09-10T18:57:30.088333+0000","last_peered":"2024-09-10T18:57:30.088333+0000","last_clean":"2024-09-10T18:57:30.088333+0000","last_became_active":"2024-09-10T18:57:30.087972+0000","last_became_peered":"2024-09-10T18:57:30.087972+0000","last_unstale":"2024-09-10T18:57:30.088333+0000","last_undegraded":"2024-09-10T18:57:30.088333+0000","last_fullsized":"2024-09-10T18:57:30.088333+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:30.735730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.0","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.088011+0000","last_change":"2024-09-10T18:57:30.088011+0000","last_active":"2024-09-10T18:57:30.088011+0000","last_peered":"2024-09-10T18:57:30.088011+0000","last_clean":"2024-09-10T18:57:30.088011+0000","last_became_active":"2024-09-10T18:57:30.087626+0000","last_became_peered":"2024-09-10T18:57:30.087626+0000","last_unstale":"2024-09-10T18:57:30.088011+0000","last_undegraded":"2024-09-10T18:57:30.088011+0000","last_fullsized":"2024-09-10T18:57:30.088011+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.596352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.18","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.092021+0000","last_change":"2024-09-10T18:57:30.092021+0000","last_active":"2024-09-10T18:57:30.092021+0000","last_peered":"2024-09-10T18:57:30.092021+0000","last_clean":"2024-09-10T18:57:30.092021+0000","last_became_active":"2024-09-10T18:57:30.091423+0000","last_became_peered":"2024-09-10T18:57:30.091423+0000","last_unstale":"2024-09-10T18:57:30.092021+0000","last_undegraded":"2024-09-10T18:57:30.092021+0000","last_fullsized":"2024-09-10T18:57:30.092021+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.304518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.19","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.086986+0000","last_change":"2024-09-10T18:57:30.086986+0000","last_active":"2024-09-10T18:57:30.086986+0000","last_peered":"2024-09-10T18:57:30.086986+0000","last_clean":"2024-09-10T18:57:30.086986+0000","last_became_active":"2024-09-10T18:57:30.086615+0000","last_became_peered":"2024-09-10T18:57:30.086615+0000","last_unstale":"2024-09-10T18:57:30.086986+0000","last_undegraded":"2024-09-10T18:57:30.086986+0000","last_fullsized":"2024-09-10T18:57:30.086986+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.728468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"176.1a","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.089305+0000","last_change":"2024-09-10T18:57:30.089305+0000","last_active":"2024-09-10T18:57:30.089305+0000","last_peered":"2024-09-10T18:57:30.089305+0000","last_clean":"2024-09-10T18:57:30.089305+0000","last_became_active":"2024-09-10T18:57:30.087057+0000","last_became_peered":"2024-09-10T18:57:30.087057+0000","last_unstale":"2024-09-10T18:57:30.089305+0000","last_undegraded":"2024-09-10T18:57:30.089305+0000","last_fullsized":"2024-09-10T18:57:30.089305+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.413923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"176.1b","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.096553+0000","last_change":"2024-09-10T18:57:30.096553+0000","last_active":"2024-09-10T18:57:30.096553+0000","last_peered":"2024-09-10T18:57:30.096553+0000","last_clean":"2024-09-10T18:57:30.096553+0000","last_became_active":"2024-09-10T18:57:30.096082+0000","last_became_peered":"2024-09-10T18:57:30.096082+0000","last_unstale":"2024-09-10T18:57:30.096553+0000","last_undegraded":"2024-09-10T18:57:30.096553+0000","last_fullsized":"2024-09-10T18:57:30.096553+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.600109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"176.1c","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.091313+0000","last_change":"2024-09-10T18:57:30.091313+0000","last_active":"2024-09-10T18:57:30.091313+0000","last_peered":"2024-09-10T18:57:30.091313+0000","last_clean":"2024-09-10T18:57:30.091313+0000","last_became_active":"2024-09-10T18:57:30.091007+0000","last_became_peered":"2024-09-10T18:57:30.091007+0000","last_unstale":"2024-09-10T18:57:30.091313+0000","last_undegraded":"2024-09-10T18:57:30.091313+0000","last_fullsized":"2024-09-10T18:57:30.091313+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.241744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"176.1d","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.084333+0000","last_change":"2024-09-10T18:57:30.084333+0000","last_active":"2024-09-10T18:57:30.084333+0000","last_peered":"2024-09-10T18:57:30.084333+0000","last_clean":"2024-09-10T18:57:30.084333+0000","last_became_active":"2024-09-10T18:57:30.083577+0000","last_became_peered":"2024-09-10T18:57:30.083577+0000","last_unstale":"2024-09-10T18:57:30.084333+0000","last_undegraded":"2024-09-10T18:57:30.084333+0000","last_fullsized":"2024-09-10T18:57:30.084333+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:41.275637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"176.1e","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.088678+0000","last_change":"2024-09-10T18:57:30.088678+0000","last_active":"2024-09-10T18:57:30.088678+0000","last_peered":"2024-09-10T18:57:30.088678+0000","last_clean":"2024-09-10T18:57:30.088678+0000","last_became_active":"2024-09-10T18:57:30.088376+0000","last_became_peered":"2024-09-10T18:57:30.088376+0000","last_unstale":"2024-09-10T18:57:30.088678+0000","last_undegraded":"2024-09-10T18:57:30.088678+0000","last_fullsized":"2024-09-10T18:57:30.088678+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.948564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"176.1f","version":"0'0","reported_seq":9,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.089425+0000","last_change":"2024-09-10T18:57:30.089425+0000","last_active":"2024-09-10T18:57:30.089425+0000","last_peered":"2024-09-10T18:57:30.089425+0000","last_clean":"2024-09-10T18:57:30.089425+0000","last_became_active":"2024-09-10T18:57:30.088921+0000","last_became_peered":"2024-09-10T18:57:30.088921+0000","last_unstale":"2024-09-10T18:57:30.089425+0000","last_undegraded":"2024-09-10T18:57:30.089425+0000","last_fullsized":"2024-09-10T18:57:30.089425+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:29.053545+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.053545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:49.234100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":236,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.077751+0000","last_change":"2024-09-10T18:57:25.062746+0000","last_active":"2024-09-10T18:57:29.077751+0000","last_peered":"2024-09-10T18:57:29.077751+0000","last_clean":"2024-09-10T18:57:29.077751+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:57:29.077751+0000","last_undegraded":"2024-09-10T18:57:29.077751+0000","last_fullsized":"2024-09-10T18:57:29.077751+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027169109999999998,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1e","version":"257'56","reported_seq":256,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078054+0000","last_change":"2024-09-10T18:57:25.061059+0000","last_active":"2024-09-10T18:57:29.078054+0000","last_peered":"2024-09-10T18:57:29.078054+0000","last_clean":"2024-09-10T18:57:29.078054+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:57:29.078054+0000","last_undegraded":"2024-09-10T18:57:29.078054+0000","last_fullsized":"2024-09-10T18:57:29.078054+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027151649999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1d","version":"276'22","reported_seq":240,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071372+0000","last_change":"2024-09-10T18:57:26.499259+0000","last_active":"2024-09-10T18:57:29.071372+0000","last_peered":"2024-09-10T18:57:29.071372+0000","last_clean":"2024-09-10T18:57:29.071372+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:57:29.071372+0000","last_undegraded":"2024-09-10T18:57:29.071372+0000","last_fullsized":"2024-09-10T18:57:29.071372+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.247847+0000","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0023609540000000002,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1c","version":"244'51","reported_seq":253,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075064+0000","last_change":"2024-09-10T18:57:25.058709+0000","last_active":"2024-09-10T18:57:29.075064+0000","last_peered":"2024-09-10T18:57:29.075064+0000","last_clean":"2024-09-10T18:57:29.075064+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:57:29.075064+0000","last_undegraded":"2024-09-10T18:57:29.075064+0000","last_fullsized":"2024-09-10T18:57:29.075064+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.052158+0000","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.00060113899999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1b","version":"240'68","reported_seq":268,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.616203+0000","last_change":"2024-09-10T18:57:30.616122+0000","last_active":"2024-09-10T18:57:30.616203+0000","last_peered":"2024-09-10T18:57:30.616203+0000","last_clean":"2024-09-10T18:57:30.616203+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:57:30.616203+0000","last_undegraded":"2024-09-10T18:57:30.616203+0000","last_fullsized":"2024-09-10T18:57:30.616203+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:57:30.615944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.615944+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:46.447187+0000","scrub_duration":0.005614539,"objects_trimmed":0,"snaptrim_duration":0.0034883840000000002,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1a","version":"228'34","reported_seq":234,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068098+0000","last_change":"2024-09-10T18:57:25.069425+0000","last_active":"2024-09-10T18:57:29.068098+0000","last_peered":"2024-09-10T18:57:29.068098+0000","last_clean":"2024-09-10T18:57:29.068098+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:57:29.068098+0000","last_undegraded":"2024-09-10T18:57:29.068098+0000","last_fullsized":"2024-09-10T18:57:29.068098+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046434629999999996,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.19","version":"257'22","reported_seq":229,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066100+0000","last_change":"2024-09-10T18:57:25.056021+0000","last_active":"2024-09-10T18:57:29.066100+0000","last_peered":"2024-09-10T18:57:29.066100+0000","last_clean":"2024-09-10T18:57:29.066100+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:57:29.066100+0000","last_undegraded":"2024-09-10T18:57:29.066100+0000","last_fullsized":"2024-09-10T18:57:29.066100+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029485689999999998,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.18","version":"254'104","reported_seq":292,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066747+0000","last_change":"2024-09-10T18:57:25.054181+0000","last_active":"2024-09-10T18:57:29.066747+0000","last_peered":"2024-09-10T18:57:29.066747+0000","last_clean":"2024-09-10T18:57:29.066747+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:57:29.066747+0000","last_undegraded":"2024-09-10T18:57:29.066747+0000","last_fullsized":"2024-09-10T18:57:29.066747+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043912200000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.17","version":"230'17","reported_seq":219,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073326+0000","last_change":"2024-09-10T18:57:25.063055+0000","last_active":"2024-09-10T18:57:29.073326+0000","last_peered":"2024-09-10T18:57:29.073326+0000","last_clean":"2024-09-10T18:57:29.073326+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:57:29.073326+0000","last_undegraded":"2024-09-10T18:57:29.073326+0000","last_fullsized":"2024-09-10T18:57:29.073326+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059160039999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.16","version":"276'22","reported_seq":225,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074112+0000","last_change":"2024-09-10T18:57:25.069652+0000","last_active":"2024-09-10T18:57:29.074112+0000","last_peered":"2024-09-10T18:57:29.074112+0000","last_clean":"2024-09-10T18:57:29.074112+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:57:29.074112+0000","last_undegraded":"2024-09-10T18:57:29.074112+0000","last_fullsized":"2024-09-10T18:57:29.074112+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032364239999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.15","version":"240'74","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074652+0000","last_change":"2024-09-10T18:57:25.070991+0000","last_active":"2024-09-10T18:57:29.074652+0000","last_peered":"2024-09-10T18:57:29.074652+0000","last_clean":"2024-09-10T18:57:29.074652+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:57:29.074652+0000","last_undegraded":"2024-09-10T18:57:29.074652+0000","last_fullsized":"2024-09-10T18:57:29.074652+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035917190000000002,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.14","version":"246'95","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075153+0000","last_change":"2024-09-10T18:57:25.633449+0000","last_active":"2024-09-10T18:57:29.075153+0000","last_peered":"2024-09-10T18:57:29.075153+0000","last_clean":"2024-09-10T18:57:29.075153+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:57:29.075153+0000","last_undegraded":"2024-09-10T18:57:29.075153+0000","last_fullsized":"2024-09-10T18:57:29.075153+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'95","last_scrub_stamp":"2024-09-10T18:57:25.633366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.633366+0000","objects_scrubbed":6,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:31.276059+0000","scrub_duration":0.0044083560000000004,"objects_trimmed":0,"snaptrim_duration":0.00052456400000000004,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.13","version":"146'1","reported_seq":208,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074254+0000","last_change":"2024-09-10T18:57:25.068778+0000","last_active":"2024-09-10T18:57:29.074254+0000","last_peered":"2024-09-10T18:57:29.074254+0000","last_clean":"2024-09-10T18:57:29.074254+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:57:29.074254+0000","last_undegraded":"2024-09-10T18:57:29.074254+0000","last_fullsized":"2024-09-10T18:57:29.074254+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002800444,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.12","version":"248'51","reported_seq":244,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067552+0000","last_change":"2024-09-10T18:57:25.055409+0000","last_active":"2024-09-10T18:57:29.067552+0000","last_peered":"2024-09-10T18:57:29.067552+0000","last_clean":"2024-09-10T18:57:29.067552+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:57:29.067552+0000","last_undegraded":"2024-09-10T18:57:29.067552+0000","last_fullsized":"2024-09-10T18:57:29.067552+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094359100000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.11","version":"226'51","reported_seq":250,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060307+0000","last_change":"2024-09-10T18:57:25.045130+0000","last_active":"2024-09-10T18:57:29.060307+0000","last_peered":"2024-09-10T18:57:29.060307+0000","last_clean":"2024-09-10T18:57:29.060307+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:57:29.060307+0000","last_undegraded":"2024-09-10T18:57:29.060307+0000","last_fullsized":"2024-09-10T18:57:29.060307+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001622641,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.10","version":"224'34","reported_seq":232,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.065833+0000","last_change":"2024-09-10T18:57:25.052334+0000","last_active":"2024-09-10T18:57:29.065833+0000","last_peered":"2024-09-10T18:57:29.065833+0000","last_clean":"2024-09-10T18:57:29.065833+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:57:29.065833+0000","last_undegraded":"2024-09-10T18:57:29.065833+0000","last_fullsized":"2024-09-10T18:57:29.065833+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002159617,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.f","version":"276'63","reported_seq":264,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075063+0000","last_change":"2024-09-10T18:57:25.062024+0000","last_active":"2024-09-10T18:57:29.075063+0000","last_peered":"2024-09-10T18:57:29.075063+0000","last_clean":"2024-09-10T18:57:29.075063+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:57:29.075063+0000","last_undegraded":"2024-09-10T18:57:29.075063+0000","last_fullsized":"2024-09-10T18:57:29.075063+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012067110000000001,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.e","version":"246'119","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074413+0000","last_change":"2024-09-10T18:57:25.062809+0000","last_active":"2024-09-10T18:57:29.074413+0000","last_peered":"2024-09-10T18:57:29.074413+0000","last_clean":"2024-09-10T18:57:29.074413+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:57:29.074413+0000","last_undegraded":"2024-09-10T18:57:29.074413+0000","last_fullsized":"2024-09-10T18:57:29.074413+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060342670000000003,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.d","version":"243'18","reported_seq":235,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063443+0000","last_change":"2024-09-10T18:57:25.051636+0000","last_active":"2024-09-10T18:57:29.063443+0000","last_peered":"2024-09-10T18:57:29.063443+0000","last_clean":"2024-09-10T18:57:29.063443+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:57:29.063443+0000","last_undegraded":"2024-09-10T18:57:29.063443+0000","last_fullsized":"2024-09-10T18:57:29.063443+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.276748+0000","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.0019838450000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.c","version":"252'22","reported_seq":230,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070988+0000","last_change":"2024-09-10T18:57:25.056955+0000","last_active":"2024-09-10T18:57:29.070988+0000","last_peered":"2024-09-10T18:57:29.070988+0000","last_clean":"2024-09-10T18:57:29.070988+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:57:29.070988+0000","last_undegraded":"2024-09-10T18:57:29.070988+0000","last_fullsized":"2024-09-10T18:57:29.070988+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032927400000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.b","version":"247'35","reported_seq":234,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074112+0000","last_change":"2024-09-10T18:57:25.054300+0000","last_active":"2024-09-10T18:57:29.074112+0000","last_peered":"2024-09-10T18:57:29.074112+0000","last_clean":"2024-09-10T18:57:29.074112+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:57:29.074112+0000","last_undegraded":"2024-09-10T18:57:29.074112+0000","last_fullsized":"2024-09-10T18:57:29.074112+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039818630000000004,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.a","version":"232'17","reported_seq":223,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073892+0000","last_change":"2024-09-10T18:57:25.068118+0000","last_active":"2024-09-10T18:57:29.073892+0000","last_peered":"2024-09-10T18:57:29.073892+0000","last_clean":"2024-09-10T18:57:29.073892+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:57:29.073892+0000","last_undegraded":"2024-09-10T18:57:29.073892+0000","last_fullsized":"2024-09-10T18:57:29.073892+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002263138,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.9","version":"0'0","reported_seq":206,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074336+0000","last_change":"2024-09-10T18:57:25.068382+0000","last_active":"2024-09-10T18:57:29.074336+0000","last_peered":"2024-09-10T18:57:29.074336+0000","last_clean":"2024-09-10T18:57:29.074336+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:57:29.074336+0000","last_undegraded":"2024-09-10T18:57:29.074336+0000","last_fullsized":"2024-09-10T18:57:29.074336+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024363800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.8","version":"234'51","reported_seq":250,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071919+0000","last_change":"2024-09-10T18:57:25.056745+0000","last_active":"2024-09-10T18:57:29.071919+0000","last_peered":"2024-09-10T18:57:29.071919+0000","last_clean":"2024-09-10T18:57:29.071919+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:57:29.071919+0000","last_undegraded":"2024-09-10T18:57:29.071919+0000","last_fullsized":"2024-09-10T18:57:29.071919+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062324299999999997,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.7","version":"276'158","reported_seq":347,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076021+0000","last_change":"2024-09-10T18:57:25.070632+0000","last_active":"2024-09-10T18:57:29.076021+0000","last_peered":"2024-09-10T18:57:29.076021+0000","last_clean":"2024-09-10T18:57:29.076021+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:57:29.076021+0000","last_undegraded":"2024-09-10T18:57:29.076021+0000","last_fullsized":"2024-09-10T18:57:29.076021+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034175,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.6","version":"290'93","reported_seq":346,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.482167+0000","last_change":"2024-09-10T18:57:29.482043+0000","last_active":"2024-09-10T18:57:29.482167+0000","last_peered":"2024-09-10T18:57:29.482167+0000","last_clean":"2024-09-10T18:57:29.482167+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:57:29.482167+0000","last_undegraded":"2024-09-10T18:57:29.482167+0000","last_fullsized":"2024-09-10T18:57:29.482167+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'93","last_scrub_stamp":"2024-09-10T18:57:29.481888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.481888+0000","objects_scrubbed":6,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.178966+0000","scrub_duration":0.004612952,"objects_trimmed":0,"snaptrim_duration":0.053520075,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.5","version":"258'51","reported_seq":250,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072045+0000","last_change":"2024-09-10T18:57:25.060287+0000","last_active":"2024-09-10T18:57:29.072045+0000","last_peered":"2024-09-10T18:57:29.072045+0000","last_clean":"2024-09-10T18:57:29.072045+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:57:29.072045+0000","last_undegraded":"2024-09-10T18:57:29.072045+0000","last_fullsized":"2024-09-10T18:57:29.072045+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053275700000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.4","version":"257'107","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070580+0000","last_change":"2024-09-10T18:57:28.509015+0000","last_active":"2024-09-10T18:57:29.070580+0000","last_peered":"2024-09-10T18:57:29.070580+0000","last_clean":"2024-09-10T18:57:29.070580+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:57:29.070580+0000","last_undegraded":"2024-09-10T18:57:29.070580+0000","last_fullsized":"2024-09-10T18:57:29.070580+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.441046+0000","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0050976069999999997,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.3","version":"236'102","reported_seq":293,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067591+0000","last_change":"2024-09-10T18:57:25.061798+0000","last_active":"2024-09-10T18:57:29.067591+0000","last_peered":"2024-09-10T18:57:29.067591+0000","last_clean":"2024-09-10T18:57:29.067591+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:57:29.067591+0000","last_undegraded":"2024-09-10T18:57:29.067591+0000","last_fullsized":"2024-09-10T18:57:29.067591+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029166359999999998,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.2","version":"230'34","reported_seq":232,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073249+0000","last_change":"2024-09-10T18:57:25.059185+0000","last_active":"2024-09-10T18:57:29.073249+0000","last_peered":"2024-09-10T18:57:29.073249+0000","last_clean":"2024-09-10T18:57:29.073249+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:57:29.073249+0000","last_undegraded":"2024-09-10T18:57:29.073249+0000","last_fullsized":"2024-09-10T18:57:29.073249+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082116250000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.1","version":"263'72","reported_seq":277,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068184+0000","last_change":"2024-09-10T18:57:25.054484+0000","last_active":"2024-09-10T18:57:29.068184+0000","last_peered":"2024-09-10T18:57:29.068184+0000","last_clean":"2024-09-10T18:57:29.068184+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:57:29.068184+0000","last_undegraded":"2024-09-10T18:57:29.068184+0000","last_fullsized":"2024-09-10T18:57:29.068184+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044783900000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"64.0","version":"246'102","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072807+0000","last_change":"2024-09-10T18:57:25.067789+0000","last_active":"2024-09-10T18:57:29.072807+0000","last_peered":"2024-09-10T18:57:29.072807+0000","last_clean":"2024-09-10T18:57:29.072807+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:57:29.072807+0000","last_undegraded":"2024-09-10T18:57:29.072807+0000","last_fullsized":"2024-09-10T18:57:29.072807+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:56:44.683516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.683516+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.401562+0000","scrub_duration":0.0037708920000000001,"objects_trimmed":0,"snaptrim_duration":0.002552459,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13","length":"2"}]},{"pgid":"6.1b","version":"0'0","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067218+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:57:29.067218+0000","last_peered":"2024-09-10T18:57:29.067218+0000","last_clean":"2024-09-10T18:57:29.067218+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:57:29.067218+0000","last_undegraded":"2024-09-10T18:57:29.067218+0000","last_fullsized":"2024-09-10T18:57:29.067218+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":272,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078989+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:57:29.078989+0000","last_peered":"2024-09-10T18:57:29.078989+0000","last_clean":"2024-09-10T18:57:29.078989+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:57:29.078989+0000","last_undegraded":"2024-09-10T18:57:29.078989+0000","last_fullsized":"2024-09-10T18:57:29.078989+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":276,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068873+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:57:29.068873+0000","last_peered":"2024-09-10T18:57:29.068873+0000","last_clean":"2024-09-10T18:57:29.068873+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:57:29.068873+0000","last_undegraded":"2024-09-10T18:57:29.068873+0000","last_fullsized":"2024-09-10T18:57:29.068873+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.485909+0000","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068031+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:57:29.068031+0000","last_peered":"2024-09-10T18:57:29.068031+0000","last_clean":"2024-09-10T18:57:29.068031+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:57:29.068031+0000","last_undegraded":"2024-09-10T18:57:29.068031+0000","last_fullsized":"2024-09-10T18:57:29.068031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.646246+0000","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067478+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:57:29.067478+0000","last_peered":"2024-09-10T18:57:29.067478+0000","last_clean":"2024-09-10T18:57:29.067478+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:57:29.067478+0000","last_undegraded":"2024-09-10T18:57:29.067478+0000","last_fullsized":"2024-09-10T18:57:29.067478+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.539352+0000","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"243'6","reported_seq":54,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069805+0000","last_change":"2024-09-10T18:57:27.072912+0000","last_active":"2024-09-10T18:57:29.069805+0000","last_peered":"2024-09-10T18:57:29.069805+0000","last_clean":"2024-09-10T18:57:29.069805+0000","last_became_active":"2024-09-10T18:57:27.072565+0000","last_became_peered":"2024-09-10T18:57:27.072565+0000","last_unstale":"2024-09-10T18:57:29.069805+0000","last_undegraded":"2024-09-10T18:57:29.069805+0000","last_fullsized":"2024-09-10T18:57:29.069805+0000","mapping_epoch":274,"log_start":"243'6","ondisk_log_start":"243'6","created":124,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'6","last_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_deep_scrub":"243'6","last_deep_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.791846+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":13,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:57.446964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":273,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069482+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:57:29.069482+0000","last_peered":"2024-09-10T18:57:29.069482+0000","last_clean":"2024-09-10T18:57:29.069482+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:57:29.069482+0000","last_undegraded":"2024-09-10T18:57:29.069482+0000","last_fullsized":"2024-09-10T18:57:29.069482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078174+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:57:29.078174+0000","last_peered":"2024-09-10T18:57:29.078174+0000","last_clean":"2024-09-10T18:57:29.078174+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:57:29.078174+0000","last_undegraded":"2024-09-10T18:57:29.078174+0000","last_fullsized":"2024-09-10T18:57:29.078174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":279,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060039+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:57:29.060039+0000","last_peered":"2024-09-10T18:57:29.060039+0000","last_clean":"2024-09-10T18:57:29.060039+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:57:29.060039+0000","last_undegraded":"2024-09-10T18:57:29.060039+0000","last_fullsized":"2024-09-10T18:57:29.060039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.405877+0000","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060700+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:57:29.060700+0000","last_peered":"2024-09-10T18:57:29.060700+0000","last_clean":"2024-09-10T18:57:29.060700+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:57:29.060700+0000","last_undegraded":"2024-09-10T18:57:29.060700+0000","last_fullsized":"2024-09-10T18:57:29.060700+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":177,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066272+0000","last_change":"2024-09-10T18:54:37.650137+0000","last_active":"2024-09-10T18:57:29.066272+0000","last_peered":"2024-09-10T18:57:29.066272+0000","last_clean":"2024-09-10T18:57:29.066272+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:57:29.066272+0000","last_undegraded":"2024-09-10T18:57:29.066272+0000","last_fullsized":"2024-09-10T18:57:29.066272+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":41,"reported_epoch":294,"state":"peering","last_fresh":"2024-09-10T18:57:30.074370+0000","last_change":"2024-09-10T18:57:30.074772+0000","last_active":"2024-09-10T18:57:28.067361+0000","last_peered":"2024-09-10T18:57:29.079036+0000","last_clean":"2024-09-10T18:57:29.079036+0000","last_became_active":"2024-09-10T18:57:11.094328+0000","last_became_peered":"2024-09-10T18:57:29.075674+0000","last_unstale":"2024-09-10T18:57:30.074370+0000","last_undegraded":"2024-09-10T18:57:30.074370+0000","last_fullsized":"2024-09-10T18:57:30.074370+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.362084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.362084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.348997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071350+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:57:29.071350+0000","last_peered":"2024-09-10T18:57:29.071350+0000","last_clean":"2024-09-10T18:57:29.071350+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:57:29.071350+0000","last_undegraded":"2024-09-10T18:57:29.071350+0000","last_fullsized":"2024-09-10T18:57:29.071350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072181+0000","last_change":"2024-09-10T18:54:41.188135+0000","last_active":"2024-09-10T18:57:29.072181+0000","last_peered":"2024-09-10T18:57:29.072181+0000","last_clean":"2024-09-10T18:57:29.072181+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:57:29.072181+0000","last_undegraded":"2024-09-10T18:57:29.072181+0000","last_fullsized":"2024-09-10T18:57:29.072181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:54:41.188040+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:54:41.188040+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036526110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061536+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:57:29.061536+0000","last_peered":"2024-09-10T18:57:29.061536+0000","last_clean":"2024-09-10T18:57:29.061536+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:57:29.061536+0000","last_undegraded":"2024-09-10T18:57:29.061536+0000","last_fullsized":"2024-09-10T18:57:29.061536+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.192711+0000","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":258,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067966+0000","last_change":"2024-09-10T18:56:32.263959+0000","last_active":"2024-09-10T18:57:29.067966+0000","last_peered":"2024-09-10T18:57:29.067966+0000","last_clean":"2024-09-10T18:57:29.067966+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:57:29.067966+0000","last_undegraded":"2024-09-10T18:57:29.067966+0000","last_fullsized":"2024-09-10T18:57:29.067966+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:39.242050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:39.242050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002953027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074320+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:57:29.074320+0000","last_peered":"2024-09-10T18:57:29.074320+0000","last_clean":"2024-09-10T18:57:29.074320+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:57:29.074320+0000","last_undegraded":"2024-09-10T18:57:29.074320+0000","last_fullsized":"2024-09-10T18:57:29.074320+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.799711+0000","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"237'6","reported_seq":59,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.105169+0000","last_change":"2024-09-10T18:57:29.105121+0000","last_active":"2024-09-10T18:57:29.105169+0000","last_peered":"2024-09-10T18:57:29.105169+0000","last_clean":"2024-09-10T18:57:29.105169+0000","last_became_active":"2024-09-10T18:57:19.076695+0000","last_became_peered":"2024-09-10T18:57:19.076695+0000","last_unstale":"2024-09-10T18:57:29.105169+0000","last_undegraded":"2024-09-10T18:57:29.105169+0000","last_fullsized":"2024-09-10T18:57:29.105169+0000","mapping_epoch":274,"log_start":"237'6","ondisk_log_start":"237'6","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'6","last_scrub_stamp":"2024-09-10T18:57:29.105052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.105052+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":6,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.677376+0000","scrub_duration":0.0020046370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":272,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071503+0000","last_change":"2024-09-10T18:57:25.517299+0000","last_active":"2024-09-10T18:57:29.071503+0000","last_peered":"2024-09-10T18:57:29.071503+0000","last_clean":"2024-09-10T18:57:29.071503+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:57:29.071503+0000","last_undegraded":"2024-09-10T18:57:29.071503+0000","last_fullsized":"2024-09-10T18:57:29.071503+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:25.517210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.517210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:26.377413+0000","scrub_duration":0.0027489659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071815+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:57:29.071815+0000","last_peered":"2024-09-10T18:57:29.071815+0000","last_clean":"2024-09-10T18:57:29.071815+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:57:29.071815+0000","last_undegraded":"2024-09-10T18:57:29.071815+0000","last_fullsized":"2024-09-10T18:57:29.071815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:40.744708+0000","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061085+0000","last_change":"2024-09-10T18:54:55.309898+0000","last_active":"2024-09-10T18:57:29.061085+0000","last_peered":"2024-09-10T18:57:29.061085+0000","last_clean":"2024-09-10T18:57:29.061085+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:57:29.061085+0000","last_undegraded":"2024-09-10T18:57:29.061085+0000","last_fullsized":"2024-09-10T18:57:29.061085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.309689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.309689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003220512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063828+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:57:29.063828+0000","last_peered":"2024-09-10T18:57:29.063828+0000","last_clean":"2024-09-10T18:57:29.063828+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:57:29.063828+0000","last_undegraded":"2024-09-10T18:57:29.063828+0000","last_fullsized":"2024-09-10T18:57:29.063828+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067420+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:57:29.067420+0000","last_peered":"2024-09-10T18:57:29.067420+0000","last_clean":"2024-09-10T18:57:29.067420+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:57:29.067420+0000","last_undegraded":"2024-09-10T18:57:29.067420+0000","last_fullsized":"2024-09-10T18:57:29.067420+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.879768+0000","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":49,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074364+0000","last_change":"2024-09-10T18:57:23.457770+0000","last_active":"2024-09-10T18:57:29.074364+0000","last_peered":"2024-09-10T18:57:29.074364+0000","last_clean":"2024-09-10T18:57:29.074364+0000","last_became_active":"2024-09-10T18:57:23.457438+0000","last_became_peered":"2024-09-10T18:57:23.457438+0000","last_unstale":"2024-09-10T18:57:29.074364+0000","last_undegraded":"2024-09-10T18:57:29.074364+0000","last_fullsized":"2024-09-10T18:57:29.074364+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":300,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073545+0000","last_change":"2024-09-10T18:54:50.687363+0000","last_active":"2024-09-10T18:57:29.073545+0000","last_peered":"2024-09-10T18:57:29.073545+0000","last_clean":"2024-09-10T18:57:29.073545+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:57:29.073545+0000","last_undegraded":"2024-09-10T18:57:29.073545+0000","last_fullsized":"2024-09-10T18:57:29.073545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:54:50.687166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:50.687166+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053731969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069334+0000","last_change":"2024-09-10T18:54:59.926609+0000","last_active":"2024-09-10T18:57:29.069334+0000","last_peered":"2024-09-10T18:57:29.069334+0000","last_clean":"2024-09-10T18:57:29.069334+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:57:29.069334+0000","last_undegraded":"2024-09-10T18:57:29.069334+0000","last_fullsized":"2024-09-10T18:57:29.069334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:59.926330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.926330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035578459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":378,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.062834+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:57:29.062834+0000","last_peered":"2024-09-10T18:57:29.062834+0000","last_clean":"2024-09-10T18:57:29.062834+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:57:29.062834+0000","last_undegraded":"2024-09-10T18:57:29.062834+0000","last_fullsized":"2024-09-10T18:57:29.062834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":265,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076591+0000","last_change":"2024-09-10T18:55:01.419421+0000","last_active":"2024-09-10T18:57:29.076591+0000","last_peered":"2024-09-10T18:57:29.076591+0000","last_clean":"2024-09-10T18:57:29.076591+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:57:29.076591+0000","last_undegraded":"2024-09-10T18:57:29.076591+0000","last_fullsized":"2024-09-10T18:57:29.076591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.419333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.419333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035926959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073512+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:57:29.073512+0000","last_peered":"2024-09-10T18:57:29.073512+0000","last_clean":"2024-09-10T18:57:29.073512+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:57:29.073512+0000","last_undegraded":"2024-09-10T18:57:29.073512+0000","last_fullsized":"2024-09-10T18:57:29.073512+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.697752+0000","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":32,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071654+0000","last_change":"2024-09-10T18:57:11.087464+0000","last_active":"2024-09-10T18:57:29.071654+0000","last_peered":"2024-09-10T18:57:29.071654+0000","last_clean":"2024-09-10T18:57:29.071654+0000","last_became_active":"2024-09-10T18:57:11.087104+0000","last_became_peered":"2024-09-10T18:57:11.087104+0000","last_unstale":"2024-09-10T18:57:29.071654+0000","last_undegraded":"2024-09-10T18:57:29.071654+0000","last_fullsized":"2024-09-10T18:57:29.071654+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.988916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.988916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067607+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:57:29.067607+0000","last_peered":"2024-09-10T18:57:29.067607+0000","last_clean":"2024-09-10T18:57:29.067607+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:57:29.067607+0000","last_undegraded":"2024-09-10T18:57:29.067607+0000","last_fullsized":"2024-09-10T18:57:29.067607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.663745+0000","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075200+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:57:29.075200+0000","last_peered":"2024-09-10T18:57:29.075200+0000","last_clean":"2024-09-10T18:57:29.075200+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:57:29.075200+0000","last_undegraded":"2024-09-10T18:57:29.075200+0000","last_fullsized":"2024-09-10T18:57:29.075200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.015829+0000","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":279,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061018+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:57:29.061018+0000","last_peered":"2024-09-10T18:57:29.061018+0000","last_clean":"2024-09-10T18:57:29.061018+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:57:29.061018+0000","last_undegraded":"2024-09-10T18:57:29.061018+0000","last_fullsized":"2024-09-10T18:57:29.061018+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.670897+0000","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.064429+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:57:29.064429+0000","last_peered":"2024-09-10T18:57:29.064429+0000","last_clean":"2024-09-10T18:57:29.064429+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:57:29.064429+0000","last_undegraded":"2024-09-10T18:57:29.064429+0000","last_fullsized":"2024-09-10T18:57:29.064429+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":178,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073966+0000","last_change":"2024-09-10T18:54:37.600609+0000","last_active":"2024-09-10T18:57:29.073966+0000","last_peered":"2024-09-10T18:57:29.073966+0000","last_clean":"2024-09-10T18:57:29.073966+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:57:29.073966+0000","last_undegraded":"2024-09-10T18:57:29.073966+0000","last_fullsized":"2024-09-10T18:57:29.073966+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.18","version":"126'2","reported_seq":204,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066596+0000","last_change":"2024-09-10T18:57:26.650589+0000","last_active":"2024-09-10T18:57:29.066596+0000","last_peered":"2024-09-10T18:57:29.066596+0000","last_clean":"2024-09-10T18:57:29.066596+0000","last_became_active":"2024-09-10T18:57:11.086021+0000","last_became_peered":"2024-09-10T18:57:11.086021+0000","last_unstale":"2024-09-10T18:57:29.066596+0000","last_undegraded":"2024-09-10T18:57:29.066596+0000","last_fullsized":"2024-09-10T18:57:29.066596+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:57:26.650396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.650396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.736604+0000","scrub_duration":0.003633781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":296,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066653+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:57:29.066653+0000","last_peered":"2024-09-10T18:57:29.066653+0000","last_clean":"2024-09-10T18:57:29.066653+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:57:29.066653+0000","last_undegraded":"2024-09-10T18:57:29.066653+0000","last_fullsized":"2024-09-10T18:57:29.066653+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.658076+0000","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":270,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072177+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:57:29.072177+0000","last_peered":"2024-09-10T18:57:29.072177+0000","last_clean":"2024-09-10T18:57:29.072177+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:57:29.072177+0000","last_undegraded":"2024-09-10T18:57:29.072177+0000","last_fullsized":"2024-09-10T18:57:29.072177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.935684+0000","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":276,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.077618+0000","last_change":"2024-09-10T18:56:39.629880+0000","last_active":"2024-09-10T18:57:29.077618+0000","last_peered":"2024-09-10T18:57:29.077618+0000","last_clean":"2024-09-10T18:57:29.077618+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:57:29.077618+0000","last_undegraded":"2024-09-10T18:57:29.077618+0000","last_fullsized":"2024-09-10T18:57:29.077618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:39.629726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.629726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.147581+0000","scrub_duration":0.0034450290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":275,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071488+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:57:29.071488+0000","last_peered":"2024-09-10T18:57:29.071488+0000","last_clean":"2024-09-10T18:57:29.071488+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:57:29.071488+0000","last_undegraded":"2024-09-10T18:57:29.071488+0000","last_fullsized":"2024-09-10T18:57:29.071488+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.752916+0000","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.077706+0000","last_change":"2024-09-10T18:56:17.674109+0000","last_active":"2024-09-10T18:57:29.077706+0000","last_peered":"2024-09-10T18:57:29.077706+0000","last_clean":"2024-09-10T18:57:29.077706+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:57:29.077706+0000","last_undegraded":"2024-09-10T18:57:29.077706+0000","last_fullsized":"2024-09-10T18:57:29.077706+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.673943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.673943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003315691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":309,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060977+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:57:29.060977+0000","last_peered":"2024-09-10T18:57:29.060977+0000","last_clean":"2024-09-10T18:57:29.060977+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:57:29.060977+0000","last_undegraded":"2024-09-10T18:57:29.060977+0000","last_fullsized":"2024-09-10T18:57:29.060977+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.690508+0000","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066265+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:57:29.066265+0000","last_peered":"2024-09-10T18:57:29.066265+0000","last_clean":"2024-09-10T18:57:29.066265+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:57:29.066265+0000","last_undegraded":"2024-09-10T18:57:29.066265+0000","last_fullsized":"2024-09-10T18:57:29.066265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":378,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.064675+0000","last_change":"2024-09-10T18:54:59.278861+0000","last_active":"2024-09-10T18:57:29.064675+0000","last_peered":"2024-09-10T18:57:29.064675+0000","last_clean":"2024-09-10T18:57:29.064675+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:57:29.064675+0000","last_undegraded":"2024-09-10T18:57:29.064675+0000","last_fullsized":"2024-09-10T18:57:29.064675+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:54:59.278694+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:54:59.278694+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038345380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":268,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068457+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:57:29.068457+0000","last_peered":"2024-09-10T18:57:29.068457+0000","last_clean":"2024-09-10T18:57:29.068457+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:57:29.068457+0000","last_undegraded":"2024-09-10T18:57:29.068457+0000","last_fullsized":"2024-09-10T18:57:29.068457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.519413+0000","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076005+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:57:29.076005+0000","last_peered":"2024-09-10T18:57:29.076005+0000","last_clean":"2024-09-10T18:57:29.076005+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:57:29.076005+0000","last_undegraded":"2024-09-10T18:57:29.076005+0000","last_fullsized":"2024-09-10T18:57:29.076005+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:48.543952+0000","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1a","version":"126'5","reported_seq":46,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073753+0000","last_change":"2024-09-10T18:57:28.776742+0000","last_active":"2024-09-10T18:57:29.073753+0000","last_peered":"2024-09-10T18:57:29.073753+0000","last_clean":"2024-09-10T18:57:29.073753+0000","last_became_active":"2024-09-10T18:57:11.090251+0000","last_became_peered":"2024-09-10T18:57:11.090251+0000","last_unstale":"2024-09-10T18:57:29.073753+0000","last_undegraded":"2024-09-10T18:57:29.073753+0000","last_fullsized":"2024-09-10T18:57:29.073753+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:57:28.776683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.776683+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.451489+0000","scrub_duration":0.0025894619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":278,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078556+0000","last_change":"2024-09-10T18:54:37.847063+0000","last_active":"2024-09-10T18:57:29.078556+0000","last_peered":"2024-09-10T18:57:29.078556+0000","last_clean":"2024-09-10T18:57:29.078556+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:57:29.078556+0000","last_undegraded":"2024-09-10T18:57:29.078556+0000","last_fullsized":"2024-09-10T18:57:29.078556+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:54:37.846984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:37.846984+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098700820000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074746+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:57:29.074746+0000","last_peered":"2024-09-10T18:57:29.074746+0000","last_clean":"2024-09-10T18:57:29.074746+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:57:29.074746+0000","last_undegraded":"2024-09-10T18:57:29.074746+0000","last_fullsized":"2024-09-10T18:57:29.074746+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060508+0000","last_change":"2024-09-10T18:54:52.331412+0000","last_active":"2024-09-10T18:57:29.060508+0000","last_peered":"2024-09-10T18:57:29.060508+0000","last_clean":"2024-09-10T18:57:29.060508+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:57:29.060508+0000","last_undegraded":"2024-09-10T18:57:29.060508+0000","last_fullsized":"2024-09-10T18:57:29.060508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:52.331111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:52.331111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034919479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":744,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.710668+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:57:29.710668+0000","last_peered":"2024-09-10T18:57:29.710668+0000","last_clean":"2024-09-10T18:57:29.710668+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:57:29.710668+0000","last_undegraded":"2024-09-10T18:57:29.710668+0000","last_fullsized":"2024-09-10T18:57:29.710668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":467,"num_read_kb":462,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068546+0000","last_change":"2024-09-10T18:55:07.836217+0000","last_active":"2024-09-10T18:57:29.068546+0000","last_peered":"2024-09-10T18:57:29.068546+0000","last_clean":"2024-09-10T18:57:29.068546+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:57:29.068546+0000","last_undegraded":"2024-09-10T18:57:29.068546+0000","last_fullsized":"2024-09-10T18:57:29.068546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:07.836145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:23.121359+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.836145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031616470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074131+0000","last_change":"2024-09-10T18:56:12.680019+0000","last_active":"2024-09-10T18:57:29.074131+0000","last_peered":"2024-09-10T18:57:29.074131+0000","last_clean":"2024-09-10T18:57:29.074131+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:57:29.074131+0000","last_undegraded":"2024-09-10T18:57:29.074131+0000","last_fullsized":"2024-09-10T18:57:29.074131+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:12.679938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:12.679938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002716511,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":197,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076539+0000","last_change":"2024-09-10T18:57:11.094446+0000","last_active":"2024-09-10T18:57:29.076539+0000","last_peered":"2024-09-10T18:57:29.076539+0000","last_clean":"2024-09-10T18:57:29.076539+0000","last_became_active":"2024-09-10T18:57:11.093990+0000","last_became_peered":"2024-09-10T18:57:11.093990+0000","last_unstale":"2024-09-10T18:57:29.076539+0000","last_undegraded":"2024-09-10T18:57:29.076539+0000","last_fullsized":"2024-09-10T18:57:29.076539+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:44.189240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.189240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.841047+0000","scrub_duration":0.0036310299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":298,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069530+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:57:29.069530+0000","last_peered":"2024-09-10T18:57:29.069530+0000","last_clean":"2024-09-10T18:57:29.069530+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:57:29.069530+0000","last_undegraded":"2024-09-10T18:57:29.069530+0000","last_fullsized":"2024-09-10T18:57:29.069530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:42.286896+0000","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076413+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:57:29.076413+0000","last_peered":"2024-09-10T18:57:29.076413+0000","last_clean":"2024-09-10T18:57:29.076413+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:57:29.076413+0000","last_undegraded":"2024-09-10T18:57:29.076413+0000","last_fullsized":"2024-09-10T18:57:29.076413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:59.107812+0000","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":367,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066416+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:57:29.066416+0000","last_peered":"2024-09-10T18:57:29.066416+0000","last_clean":"2024-09-10T18:57:29.066416+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:57:29.066416+0000","last_undegraded":"2024-09-10T18:57:29.066416+0000","last_fullsized":"2024-09-10T18:57:29.066416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.264937+0000","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060364+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:57:29.060364+0000","last_peered":"2024-09-10T18:57:29.060364+0000","last_clean":"2024-09-10T18:57:29.060364+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:57:29.060364+0000","last_undegraded":"2024-09-10T18:57:29.060364+0000","last_fullsized":"2024-09-10T18:57:29.060364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:00.852108+0000","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060903+0000","last_change":"2024-09-10T18:54:46.264917+0000","last_active":"2024-09-10T18:57:29.060903+0000","last_peered":"2024-09-10T18:57:29.060903+0000","last_clean":"2024-09-10T18:57:29.060903+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:57:29.060903+0000","last_undegraded":"2024-09-10T18:57:29.060903+0000","last_fullsized":"2024-09-10T18:57:29.060903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:46.264846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:46.264846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027169630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":178,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066300+0000","last_change":"2024-09-10T18:54:37.726541+0000","last_active":"2024-09-10T18:57:29.066300+0000","last_peered":"2024-09-10T18:57:29.066300+0000","last_clean":"2024-09-10T18:57:29.066300+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:57:29.066300+0000","last_undegraded":"2024-09-10T18:57:29.066300+0000","last_fullsized":"2024-09-10T18:57:29.066300+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"126'2","reported_seq":45,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069868+0000","last_change":"2024-09-10T18:57:11.104522+0000","last_active":"2024-09-10T18:57:29.069868+0000","last_peered":"2024-09-10T18:57:29.069868+0000","last_clean":"2024-09-10T18:57:29.069868+0000","last_became_active":"2024-09-10T18:57:11.092794+0000","last_became_peered":"2024-09-10T18:57:11.092794+0000","last_unstale":"2024-09-10T18:57:29.069868+0000","last_undegraded":"2024-09-10T18:57:29.069868+0000","last_fullsized":"2024-09-10T18:57:29.069868+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:39.169459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.169459+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.829162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":426,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075741+0000","last_change":"2024-09-10T18:56:03.530880+0000","last_active":"2024-09-10T18:57:29.075741+0000","last_peered":"2024-09-10T18:57:29.075741+0000","last_clean":"2024-09-10T18:57:29.075741+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:57:29.075741+0000","last_undegraded":"2024-09-10T18:57:29.075741+0000","last_fullsized":"2024-09-10T18:57:29.075741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:56:03.530741+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.530741+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049201490000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072032+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:57:29.072032+0000","last_peered":"2024-09-10T18:57:29.072032+0000","last_clean":"2024-09-10T18:57:29.072032+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:57:29.072032+0000","last_undegraded":"2024-09-10T18:57:29.072032+0000","last_fullsized":"2024-09-10T18:57:29.072032+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.253426+0000","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":377,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067745+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:57:29.067745+0000","last_peered":"2024-09-10T18:57:29.067745+0000","last_clean":"2024-09-10T18:57:29.067745+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:57:29.067745+0000","last_undegraded":"2024-09-10T18:57:29.067745+0000","last_fullsized":"2024-09-10T18:57:29.067745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":275,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073359+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:57:29.073359+0000","last_peered":"2024-09-10T18:57:29.073359+0000","last_clean":"2024-09-10T18:57:29.073359+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:57:29.073359+0000","last_undegraded":"2024-09-10T18:57:29.073359+0000","last_fullsized":"2024-09-10T18:57:29.073359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071768+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:57:29.071768+0000","last_peered":"2024-09-10T18:57:29.071768+0000","last_clean":"2024-09-10T18:57:29.071768+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:57:29.071768+0000","last_undegraded":"2024-09-10T18:57:29.071768+0000","last_fullsized":"2024-09-10T18:57:29.071768+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.876489+0000","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":178,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067808+0000","last_change":"2024-09-10T18:54:37.710475+0000","last_active":"2024-09-10T18:57:29.067808+0000","last_peered":"2024-09-10T18:57:29.067808+0000","last_clean":"2024-09-10T18:57:29.067808+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:57:29.067808+0000","last_undegraded":"2024-09-10T18:57:29.067808+0000","last_fullsized":"2024-09-10T18:57:29.067808+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.17","version":"126'5","reported_seq":33,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068959+0000","last_change":"2024-09-10T18:57:11.102477+0000","last_active":"2024-09-10T18:57:29.068959+0000","last_peered":"2024-09-10T18:57:29.068959+0000","last_clean":"2024-09-10T18:57:29.068959+0000","last_became_active":"2024-09-10T18:57:11.096848+0000","last_became_peered":"2024-09-10T18:57:11.096848+0000","last_unstale":"2024-09-10T18:57:29.068959+0000","last_undegraded":"2024-09-10T18:57:29.068959+0000","last_fullsized":"2024-09-10T18:57:29.068959+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.790063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.790063+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.819892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071901+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:57:29.071901+0000","last_peered":"2024-09-10T18:57:29.071901+0000","last_clean":"2024-09-10T18:57:29.071901+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:57:29.071901+0000","last_undegraded":"2024-09-10T18:57:29.071901+0000","last_fullsized":"2024-09-10T18:57:29.071901+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":270,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075813+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:57:29.075813+0000","last_peered":"2024-09-10T18:57:29.075813+0000","last_clean":"2024-09-10T18:57:29.075813+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:57:29.075813+0000","last_undegraded":"2024-09-10T18:57:29.075813+0000","last_fullsized":"2024-09-10T18:57:29.075813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.138878+0000","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074926+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:57:29.074926+0000","last_peered":"2024-09-10T18:57:29.074926+0000","last_clean":"2024-09-10T18:57:29.074926+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:57:29.074926+0000","last_undegraded":"2024-09-10T18:57:29.074926+0000","last_fullsized":"2024-09-10T18:57:29.074926+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.064119+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:57:29.064119+0000","last_peered":"2024-09-10T18:57:29.064119+0000","last_clean":"2024-09-10T18:57:29.064119+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:57:29.064119+0000","last_undegraded":"2024-09-10T18:57:29.064119+0000","last_fullsized":"2024-09-10T18:57:29.064119+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074008+0000","last_change":"2024-09-10T18:56:13.649580+0000","last_active":"2024-09-10T18:57:29.074008+0000","last_peered":"2024-09-10T18:57:29.074008+0000","last_clean":"2024-09-10T18:57:29.074008+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:57:29.074008+0000","last_undegraded":"2024-09-10T18:57:29.074008+0000","last_fullsized":"2024-09-10T18:57:29.074008+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.649458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.649458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027534759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"126'5","reported_seq":50,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074845+0000","last_change":"2024-09-10T18:57:11.096564+0000","last_active":"2024-09-10T18:57:29.074845+0000","last_peered":"2024-09-10T18:57:29.074845+0000","last_clean":"2024-09-10T18:57:29.074845+0000","last_became_active":"2024-09-10T18:57:11.088957+0000","last_became_peered":"2024-09-10T18:57:11.088957+0000","last_unstale":"2024-09-10T18:57:29.074845+0000","last_undegraded":"2024-09-10T18:57:29.074845+0000","last_fullsized":"2024-09-10T18:57:29.074845+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":307,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061847+0000","last_change":"2024-09-10T18:55:15.131835+0000","last_active":"2024-09-10T18:57:29.061847+0000","last_peered":"2024-09-10T18:57:29.061847+0000","last_clean":"2024-09-10T18:57:29.061847+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:57:29.061847+0000","last_undegraded":"2024-09-10T18:57:29.061847+0000","last_fullsized":"2024-09-10T18:57:29.061847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:55:15.131693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:15.131693+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048796719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068362+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:57:29.068362+0000","last_peered":"2024-09-10T18:57:29.068362+0000","last_clean":"2024-09-10T18:57:29.068362+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:57:29.068362+0000","last_undegraded":"2024-09-10T18:57:29.068362+0000","last_fullsized":"2024-09-10T18:57:29.068362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075664+0000","last_change":"2024-09-10T18:56:04.495501+0000","last_active":"2024-09-10T18:57:29.075664+0000","last_peered":"2024-09-10T18:57:29.075664+0000","last_clean":"2024-09-10T18:57:29.075664+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:57:29.075664+0000","last_undegraded":"2024-09-10T18:57:29.075664+0000","last_fullsized":"2024-09-10T18:57:29.075664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.495334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.495334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003611179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068817+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:57:29.068817+0000","last_peered":"2024-09-10T18:57:29.068817+0000","last_clean":"2024-09-10T18:57:29.068817+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:57:29.068817+0000","last_undegraded":"2024-09-10T18:57:29.068817+0000","last_fullsized":"2024-09-10T18:57:29.068817+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069022+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:57:29.069022+0000","last_peered":"2024-09-10T18:57:29.069022+0000","last_clean":"2024-09-10T18:57:29.069022+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:57:29.069022+0000","last_undegraded":"2024-09-10T18:57:29.069022+0000","last_fullsized":"2024-09-10T18:57:29.069022+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.215962+0000","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":32,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.062784+0000","last_change":"2024-09-10T18:57:11.091444+0000","last_active":"2024-09-10T18:57:29.062784+0000","last_peered":"2024-09-10T18:57:29.062784+0000","last_clean":"2024-09-10T18:57:29.062784+0000","last_became_active":"2024-09-10T18:57:11.091111+0000","last_became_peered":"2024-09-10T18:57:11.091111+0000","last_unstale":"2024-09-10T18:57:29.062784+0000","last_undegraded":"2024-09-10T18:57:29.062784+0000","last_fullsized":"2024-09-10T18:57:29.062784+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.795811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.795811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.217254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":308,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070384+0000","last_change":"2024-09-10T18:54:56.479766+0000","last_active":"2024-09-10T18:57:29.070384+0000","last_peered":"2024-09-10T18:57:29.070384+0000","last_clean":"2024-09-10T18:57:29.070384+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:57:29.070384+0000","last_undegraded":"2024-09-10T18:57:29.070384+0000","last_fullsized":"2024-09-10T18:57:29.070384+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:54:56.479670+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.479670+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003399188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":281,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073988+0000","last_change":"2024-09-10T18:56:15.655135+0000","last_active":"2024-09-10T18:57:29.073988+0000","last_peered":"2024-09-10T18:57:29.073988+0000","last_clean":"2024-09-10T18:57:29.073988+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:57:29.073988+0000","last_undegraded":"2024-09-10T18:57:29.073988+0000","last_fullsized":"2024-09-10T18:57:29.073988+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:15.655055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.655055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.233854+0000","scrub_duration":0.0020254439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.077789+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:57:29.077789+0000","last_peered":"2024-09-10T18:57:29.077789+0000","last_clean":"2024-09-10T18:57:29.077789+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:57:29.077789+0000","last_undegraded":"2024-09-10T18:57:29.077789+0000","last_fullsized":"2024-09-10T18:57:29.077789+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:31.219400+0000","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071583+0000","last_change":"2024-09-10T18:56:06.517243+0000","last_active":"2024-09-10T18:57:29.071583+0000","last_peered":"2024-09-10T18:57:29.071583+0000","last_clean":"2024-09-10T18:57:29.071583+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:57:29.071583+0000","last_undegraded":"2024-09-10T18:57:29.071583+0000","last_fullsized":"2024-09-10T18:57:29.071583+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.517079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.517079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038953730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073513+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:57:29.073513+0000","last_peered":"2024-09-10T18:57:29.073513+0000","last_clean":"2024-09-10T18:57:29.073513+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:57:29.073513+0000","last_undegraded":"2024-09-10T18:57:29.073513+0000","last_fullsized":"2024-09-10T18:57:29.073513+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.237861+0000","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"126'9","reported_seq":44,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069891+0000","last_change":"2024-09-10T18:57:11.102250+0000","last_active":"2024-09-10T18:57:29.069891+0000","last_peered":"2024-09-10T18:57:29.069891+0000","last_clean":"2024-09-10T18:57:29.069891+0000","last_became_active":"2024-09-10T18:57:11.081753+0000","last_became_peered":"2024-09-10T18:57:11.081753+0000","last_unstale":"2024-09-10T18:57:29.069891+0000","last_undegraded":"2024-09-10T18:57:29.069891+0000","last_fullsized":"2024-09-10T18:57:29.069891+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:15.650681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.650681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.062787+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:57:29.062787+0000","last_peered":"2024-09-10T18:57:29.062787+0000","last_clean":"2024-09-10T18:57:29.062787+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:57:29.062787+0000","last_undegraded":"2024-09-10T18:57:29.062787+0000","last_fullsized":"2024-09-10T18:57:29.062787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.065859+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:57:29.065859+0000","last_peered":"2024-09-10T18:57:29.065859+0000","last_clean":"2024-09-10T18:57:29.065859+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:57:29.065859+0000","last_undegraded":"2024-09-10T18:57:29.065859+0000","last_fullsized":"2024-09-10T18:57:29.065859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.405645+0000","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":268,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078646+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:57:29.078646+0000","last_peered":"2024-09-10T18:57:29.078646+0000","last_clean":"2024-09-10T18:57:29.078646+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:57:29.078646+0000","last_undegraded":"2024-09-10T18:57:29.078646+0000","last_fullsized":"2024-09-10T18:57:29.078646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":264,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075385+0000","last_change":"2024-09-10T18:55:19.296815+0000","last_active":"2024-09-10T18:57:29.075385+0000","last_peered":"2024-09-10T18:57:29.075385+0000","last_clean":"2024-09-10T18:57:29.075385+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:57:29.075385+0000","last_undegraded":"2024-09-10T18:57:29.075385+0000","last_fullsized":"2024-09-10T18:57:29.075385+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:19.296674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:55:19.296674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003182767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078724+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:57:29.078724+0000","last_peered":"2024-09-10T18:57:29.078724+0000","last_clean":"2024-09-10T18:57:29.078724+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:57:29.078724+0000","last_undegraded":"2024-09-10T18:57:29.078724+0000","last_fullsized":"2024-09-10T18:57:29.078724+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:45.407178+0000","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":39,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.701264+0000","last_change":"2024-09-10T18:57:29.701100+0000","last_active":"2024-09-10T18:57:29.701264+0000","last_peered":"2024-09-10T18:57:29.701264+0000","last_clean":"2024-09-10T18:57:29.701264+0000","last_became_active":"2024-09-10T18:57:11.085788+0000","last_became_peered":"2024-09-10T18:57:11.085788+0000","last_unstale":"2024-09-10T18:57:29.701264+0000","last_undegraded":"2024-09-10T18:57:29.701264+0000","last_fullsized":"2024-09-10T18:57:29.701264+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.700929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.700929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:49.024524+0000","scrub_duration":0.003421678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":281,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066928+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:57:29.066928+0000","last_peered":"2024-09-10T18:57:29.066928+0000","last_clean":"2024-09-10T18:57:29.066928+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:57:29.066928+0000","last_undegraded":"2024-09-10T18:57:29.066928+0000","last_fullsized":"2024-09-10T18:57:29.066928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.212086+0000","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070636+0000","last_change":"2024-09-10T18:55:03.233057+0000","last_active":"2024-09-10T18:57:29.070636+0000","last_peered":"2024-09-10T18:57:29.070636+0000","last_clean":"2024-09-10T18:57:29.070636+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:57:29.070636+0000","last_undegraded":"2024-09-10T18:57:29.070636+0000","last_fullsized":"2024-09-10T18:57:29.070636+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:03.232923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.232923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036625059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066002+0000","last_change":"2024-09-10T18:56:04.556751+0000","last_active":"2024-09-10T18:57:29.066002+0000","last_peered":"2024-09-10T18:57:29.066002+0000","last_clean":"2024-09-10T18:57:29.066002+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:57:29.066002+0000","last_undegraded":"2024-09-10T18:57:29.066002+0000","last_fullsized":"2024-09-10T18:57:29.066002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:56:04.556586+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.556586+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046952210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073822+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:57:29.073822+0000","last_peered":"2024-09-10T18:57:29.073822+0000","last_clean":"2024-09-10T18:57:29.073822+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:57:29.073822+0000","last_undegraded":"2024-09-10T18:57:29.073822+0000","last_fullsized":"2024-09-10T18:57:29.073822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":265,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074372+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:57:29.074372+0000","last_peered":"2024-09-10T18:57:29.074372+0000","last_clean":"2024-09-10T18:57:29.074372+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:57:29.074372+0000","last_undegraded":"2024-09-10T18:57:29.074372+0000","last_fullsized":"2024-09-10T18:57:29.074372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071875+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:57:29.071875+0000","last_peered":"2024-09-10T18:57:29.071875+0000","last_clean":"2024-09-10T18:57:29.071875+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:57:29.071875+0000","last_undegraded":"2024-09-10T18:57:29.071875+0000","last_fullsized":"2024-09-10T18:57:29.071875+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.745484+0000","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.11","version":"126'7","reported_seq":37,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073746+0000","last_change":"2024-09-10T18:57:11.106310+0000","last_active":"2024-09-10T18:57:29.073746+0000","last_peered":"2024-09-10T18:57:29.073746+0000","last_clean":"2024-09-10T18:57:29.073746+0000","last_became_active":"2024-09-10T18:57:11.087842+0000","last_became_peered":"2024-09-10T18:57:11.087842+0000","last_unstale":"2024-09-10T18:57:29.073746+0000","last_undegraded":"2024-09-10T18:57:29.073746+0000","last_fullsized":"2024-09-10T18:57:29.073746+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:21.348988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.348988+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":295,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071957+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:57:29.071957+0000","last_peered":"2024-09-10T18:57:29.071957+0000","last_clean":"2024-09-10T18:57:29.071957+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:57:29.071957+0000","last_undegraded":"2024-09-10T18:57:29.071957+0000","last_fullsized":"2024-09-10T18:57:29.071957+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:50.212518+0000","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075813+0000","last_change":"2024-09-10T18:54:58.446820+0000","last_active":"2024-09-10T18:57:29.075813+0000","last_peered":"2024-09-10T18:57:29.075813+0000","last_clean":"2024-09-10T18:57:29.075813+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:57:29.075813+0000","last_undegraded":"2024-09-10T18:57:29.075813+0000","last_fullsized":"2024-09-10T18:57:29.075813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.446707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.446707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011714456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073642+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:57:29.073642+0000","last_peered":"2024-09-10T18:57:29.073642+0000","last_clean":"2024-09-10T18:57:29.073642+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:57:29.073642+0000","last_undegraded":"2024-09-10T18:57:29.073642+0000","last_fullsized":"2024-09-10T18:57:29.073642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066333+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:57:29.066333+0000","last_peered":"2024-09-10T18:57:29.066333+0000","last_clean":"2024-09-10T18:57:29.066333+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:57:29.066333+0000","last_undegraded":"2024-09-10T18:57:29.066333+0000","last_fullsized":"2024-09-10T18:57:29.066333+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.916290+0000","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060689+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:57:29.060689+0000","last_peered":"2024-09-10T18:57:29.060689+0000","last_clean":"2024-09-10T18:57:29.060689+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:57:29.060689+0000","last_undegraded":"2024-09-10T18:57:29.060689+0000","last_fullsized":"2024-09-10T18:57:29.060689+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":289,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071400+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:57:29.071400+0000","last_peered":"2024-09-10T18:57:29.071400+0000","last_clean":"2024-09-10T18:57:29.071400+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:57:29.071400+0000","last_undegraded":"2024-09-10T18:57:29.071400+0000","last_fullsized":"2024-09-10T18:57:29.071400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":270,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074994+0000","last_change":"2024-09-10T18:56:30.684729+0000","last_active":"2024-09-10T18:57:29.074994+0000","last_peered":"2024-09-10T18:57:29.074994+0000","last_clean":"2024-09-10T18:57:29.074994+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:57:29.074994+0000","last_undegraded":"2024-09-10T18:57:29.074994+0000","last_fullsized":"2024-09-10T18:57:29.074994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.684589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.093891+0000","scrub_duration":0.0023375330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":258,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.062522+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:57:29.062522+0000","last_peered":"2024-09-10T18:57:29.062522+0000","last_clean":"2024-09-10T18:57:29.062522+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:57:29.062522+0000","last_undegraded":"2024-09-10T18:57:29.062522+0000","last_fullsized":"2024-09-10T18:57:29.062522+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":256,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.062561+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:57:29.062561+0000","last_peered":"2024-09-10T18:57:29.062561+0000","last_clean":"2024-09-10T18:57:29.062561+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:57:29.062561+0000","last_undegraded":"2024-09-10T18:57:29.062561+0000","last_fullsized":"2024-09-10T18:57:29.062561+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067653+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:57:29.067653+0000","last_peered":"2024-09-10T18:57:29.067653+0000","last_clean":"2024-09-10T18:57:29.067653+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:57:29.067653+0000","last_undegraded":"2024-09-10T18:57:29.067653+0000","last_fullsized":"2024-09-10T18:57:29.067653+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.383629+0000","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":299,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073174+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:57:29.073174+0000","last_peered":"2024-09-10T18:57:29.073174+0000","last_clean":"2024-09-10T18:57:29.073174+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:57:29.073174+0000","last_undegraded":"2024-09-10T18:57:29.073174+0000","last_fullsized":"2024-09-10T18:57:29.073174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.067234+0000","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":293,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075898+0000","last_change":"2024-09-10T18:56:36.828365+0000","last_active":"2024-09-10T18:57:29.075898+0000","last_peered":"2024-09-10T18:57:29.075898+0000","last_clean":"2024-09-10T18:57:29.075898+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:57:29.075898+0000","last_undegraded":"2024-09-10T18:57:29.075898+0000","last_fullsized":"2024-09-10T18:57:29.075898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:56:36.828207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.828207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:41.829596+0000","scrub_duration":0.0038610699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066966+0000","last_change":"2024-09-10T18:55:00.199821+0000","last_active":"2024-09-10T18:57:29.066966+0000","last_peered":"2024-09-10T18:57:29.066966+0000","last_clean":"2024-09-10T18:57:29.066966+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:57:29.066966+0000","last_undegraded":"2024-09-10T18:57:29.066966+0000","last_fullsized":"2024-09-10T18:57:29.066966+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.199571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.199571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036539939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":265,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068460+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:57:29.068460+0000","last_peered":"2024-09-10T18:57:29.068460+0000","last_clean":"2024-09-10T18:57:29.068460+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:57:29.068460+0000","last_undegraded":"2024-09-10T18:57:29.068460+0000","last_fullsized":"2024-09-10T18:57:29.068460+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060034+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:57:29.060034+0000","last_peered":"2024-09-10T18:57:29.060034+0000","last_clean":"2024-09-10T18:57:29.060034+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:57:29.060034+0000","last_undegraded":"2024-09-10T18:57:29.060034+0000","last_fullsized":"2024-09-10T18:57:29.060034+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072486+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:57:29.072486+0000","last_peered":"2024-09-10T18:57:29.072486+0000","last_clean":"2024-09-10T18:57:29.072486+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:57:29.072486+0000","last_undegraded":"2024-09-10T18:57:29.072486+0000","last_fullsized":"2024-09-10T18:57:29.072486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:34.960254+0000","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":322,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066970+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:57:29.066970+0000","last_peered":"2024-09-10T18:57:29.066970+0000","last_clean":"2024-09-10T18:57:29.066970+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:57:29.066970+0000","last_undegraded":"2024-09-10T18:57:29.066970+0000","last_fullsized":"2024-09-10T18:57:29.066970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.123931+0000","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":377,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060733+0000","last_change":"2024-09-10T18:55:04.228572+0000","last_active":"2024-09-10T18:57:29.060733+0000","last_peered":"2024-09-10T18:57:29.060733+0000","last_clean":"2024-09-10T18:57:29.060733+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:57:29.060733+0000","last_undegraded":"2024-09-10T18:57:29.060733+0000","last_fullsized":"2024-09-10T18:57:29.060733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:04.228360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.228360+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039590529999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":257,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067325+0000","last_change":"2024-09-10T18:55:01.757908+0000","last_active":"2024-09-10T18:57:29.067325+0000","last_peered":"2024-09-10T18:57:29.067325+0000","last_clean":"2024-09-10T18:57:29.067325+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:57:29.067325+0000","last_undegraded":"2024-09-10T18:57:29.067325+0000","last_fullsized":"2024-09-10T18:57:29.067325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.757807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.757807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011468136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":201,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072591+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:57:29.072591+0000","last_peered":"2024-09-10T18:57:29.072591+0000","last_clean":"2024-09-10T18:57:29.072591+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:57:29.072591+0000","last_undegraded":"2024-09-10T18:57:29.072591+0000","last_fullsized":"2024-09-10T18:57:29.072591+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.921825+0000","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067524+0000","last_change":"2024-09-10T18:55:05.902412+0000","last_active":"2024-09-10T18:57:29.067524+0000","last_peered":"2024-09-10T18:57:29.067524+0000","last_clean":"2024-09-10T18:57:29.067524+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:57:29.067524+0000","last_undegraded":"2024-09-10T18:57:29.067524+0000","last_fullsized":"2024-09-10T18:57:29.067524+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:55:05.902154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.902154+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047960349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073258+0000","last_change":"2024-09-10T18:54:49.456319+0000","last_active":"2024-09-10T18:57:29.073258+0000","last_peered":"2024-09-10T18:57:29.073258+0000","last_clean":"2024-09-10T18:57:29.073258+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:57:29.073258+0000","last_undegraded":"2024-09-10T18:57:29.073258+0000","last_fullsized":"2024-09-10T18:57:29.073258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.456167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.456167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002905819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":268,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078432+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:57:29.078432+0000","last_peered":"2024-09-10T18:57:29.078432+0000","last_clean":"2024-09-10T18:57:29.078432+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:57:29.078432+0000","last_undegraded":"2024-09-10T18:57:29.078432+0000","last_fullsized":"2024-09-10T18:57:29.078432+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":265,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075370+0000","last_change":"2024-09-10T18:55:02.390515+0000","last_active":"2024-09-10T18:57:29.075370+0000","last_peered":"2024-09-10T18:57:29.075370+0000","last_clean":"2024-09-10T18:57:29.075370+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:57:29.075370+0000","last_undegraded":"2024-09-10T18:57:29.075370+0000","last_fullsized":"2024-09-10T18:57:29.075370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.390417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.390417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002554265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.065417+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:57:29.065417+0000","last_peered":"2024-09-10T18:57:29.065417+0000","last_clean":"2024-09-10T18:57:29.065417+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:57:29.065417+0000","last_undegraded":"2024-09-10T18:57:29.065417+0000","last_fullsized":"2024-09-10T18:57:29.065417+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.580079+0000","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.060503+0000","last_change":"2024-09-10T18:56:32.796094+0000","last_active":"2024-09-10T18:57:29.060503+0000","last_peered":"2024-09-10T18:57:29.060503+0000","last_clean":"2024-09-10T18:57:29.060503+0000","last_became_active":"2024-09-10T18:54:37.566860+0000","last_became_peered":"2024-09-10T18:54:37.566860+0000","last_unstale":"2024-09-10T18:57:29.060503+0000","last_undegraded":"2024-09-10T18:57:29.060503+0000","last_fullsized":"2024-09-10T18:57:29.060503+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.795927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:47.610135+0000","scrub_duration":0.0032690250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":277,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072441+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:57:29.072441+0000","last_peered":"2024-09-10T18:57:29.072441+0000","last_clean":"2024-09-10T18:57:29.072441+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:57:29.072441+0000","last_undegraded":"2024-09-10T18:57:29.072441+0000","last_fullsized":"2024-09-10T18:57:29.072441+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074551+0000","last_change":"2024-09-10T18:54:49.637132+0000","last_active":"2024-09-10T18:57:29.074551+0000","last_peered":"2024-09-10T18:57:29.074551+0000","last_clean":"2024-09-10T18:57:29.074551+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:57:29.074551+0000","last_undegraded":"2024-09-10T18:57:29.074551+0000","last_fullsized":"2024-09-10T18:57:29.074551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:49.637024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:49.637024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029426489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069392+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:57:29.069392+0000","last_peered":"2024-09-10T18:57:29.069392+0000","last_clean":"2024-09-10T18:57:29.069392+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:57:29.069392+0000","last_undegraded":"2024-09-10T18:57:29.069392+0000","last_fullsized":"2024-09-10T18:57:29.069392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071474+0000","last_change":"2024-09-10T18:56:03.595788+0000","last_active":"2024-09-10T18:57:29.071474+0000","last_peered":"2024-09-10T18:57:29.071474+0000","last_clean":"2024-09-10T18:57:29.071474+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:57:29.071474+0000","last_undegraded":"2024-09-10T18:57:29.071474+0000","last_fullsized":"2024-09-10T18:57:29.071474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.595643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.595643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030518839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073417+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:57:29.073417+0000","last_peered":"2024-09-10T18:57:29.073417+0000","last_clean":"2024-09-10T18:57:29.073417+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:57:29.073417+0000","last_undegraded":"2024-09-10T18:57:29.073417+0000","last_fullsized":"2024-09-10T18:57:29.073417+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:44.043389+0000","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"126'5","reported_seq":193,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070154+0000","last_change":"2024-09-10T18:56:37.282686+0000","last_active":"2024-09-10T18:57:29.070154+0000","last_peered":"2024-09-10T18:57:29.070154+0000","last_clean":"2024-09-10T18:57:29.070154+0000","last_became_active":"2024-09-10T18:54:37.507583+0000","last_became_peered":"2024-09-10T18:54:37.507583+0000","last_unstale":"2024-09-10T18:57:29.070154+0000","last_undegraded":"2024-09-10T18:57:29.070154+0000","last_fullsized":"2024-09-10T18:57:29.070154+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:37.282578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.282578+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.873328+0000","scrub_duration":0.0030887559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069044+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:57:29.069044+0000","last_peered":"2024-09-10T18:57:29.069044+0000","last_clean":"2024-09-10T18:57:29.069044+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:57:29.069044+0000","last_undegraded":"2024-09-10T18:57:29.069044+0000","last_fullsized":"2024-09-10T18:57:29.069044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.936795+0000","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":274,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-09-10T18:57:30.249495+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:57:30.249495+0000","last_peered":"2024-09-10T18:57:30.249495+0000","last_clean":"2024-09-10T18:57:30.249495+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:57:30.249495+0000","last_undegraded":"2024-09-10T18:57:30.249495+0000","last_fullsized":"2024-09-10T18:57:30.249495+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.607986+0000","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":370,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069176+0000","last_change":"2024-09-10T18:54:40.522553+0000","last_active":"2024-09-10T18:57:29.069176+0000","last_peered":"2024-09-10T18:57:29.069176+0000","last_clean":"2024-09-10T18:57:29.069176+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:57:29.069176+0000","last_undegraded":"2024-09-10T18:57:29.069176+0000","last_fullsized":"2024-09-10T18:57:29.069176+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:40.522460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:40.522460+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027852699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":266,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078264+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:57:29.078264+0000","last_peered":"2024-09-10T18:57:29.078264+0000","last_clean":"2024-09-10T18:57:29.078264+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:57:29.078264+0000","last_undegraded":"2024-09-10T18:57:29.078264+0000","last_fullsized":"2024-09-10T18:57:29.078264+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.837926+0000","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069639+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:57:29.069639+0000","last_peered":"2024-09-10T18:57:29.069639+0000","last_clean":"2024-09-10T18:57:29.069639+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:57:29.069639+0000","last_undegraded":"2024-09-10T18:57:29.069639+0000","last_fullsized":"2024-09-10T18:57:29.069639+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.533486+0000","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"126'5","reported_seq":192,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066389+0000","last_change":"2024-09-10T18:56:27.205415+0000","last_active":"2024-09-10T18:57:29.066389+0000","last_peered":"2024-09-10T18:57:29.066389+0000","last_clean":"2024-09-10T18:57:29.066389+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:57:29.066389+0000","last_undegraded":"2024-09-10T18:57:29.066389+0000","last_fullsized":"2024-09-10T18:57:29.066389+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.205258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.205258+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:36.814923+0000","scrub_duration":0.0049140169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063067+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:57:29.063067+0000","last_peered":"2024-09-10T18:57:29.063067+0000","last_clean":"2024-09-10T18:57:29.063067+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:57:29.063067+0000","last_undegraded":"2024-09-10T18:57:29.063067+0000","last_fullsized":"2024-09-10T18:57:29.063067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":272,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068226+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:57:29.068226+0000","last_peered":"2024-09-10T18:57:29.068226+0000","last_clean":"2024-09-10T18:57:29.068226+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:57:29.068226+0000","last_undegraded":"2024-09-10T18:57:29.068226+0000","last_fullsized":"2024-09-10T18:57:29.068226+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":365,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071717+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:57:29.071717+0000","last_peered":"2024-09-10T18:57:29.071717+0000","last_clean":"2024-09-10T18:57:29.071717+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:57:29.071717+0000","last_undegraded":"2024-09-10T18:57:29.071717+0000","last_fullsized":"2024-09-10T18:57:29.071717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066813+0000","last_change":"2024-09-10T18:54:53.310725+0000","last_active":"2024-09-10T18:57:29.066813+0000","last_peered":"2024-09-10T18:57:29.066813+0000","last_clean":"2024-09-10T18:57:29.066813+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:57:29.066813+0000","last_undegraded":"2024-09-10T18:57:29.066813+0000","last_fullsized":"2024-09-10T18:57:29.066813+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.310489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.310489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026931849999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":177,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.076421+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:57:29.076421+0000","last_peered":"2024-09-10T18:57:29.076421+0000","last_clean":"2024-09-10T18:57:29.076421+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:57:29.076421+0000","last_undegraded":"2024-09-10T18:57:29.076421+0000","last_fullsized":"2024-09-10T18:57:29.076421+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"126'5","reported_seq":192,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074242+0000","last_change":"2024-09-10T18:56:39.805638+0000","last_active":"2024-09-10T18:57:29.074242+0000","last_peered":"2024-09-10T18:57:29.074242+0000","last_clean":"2024-09-10T18:57:29.074242+0000","last_became_active":"2024-09-10T18:54:37.610866+0000","last_became_peered":"2024-09-10T18:54:37.610866+0000","last_unstale":"2024-09-10T18:57:29.074242+0000","last_undegraded":"2024-09-10T18:57:29.074242+0000","last_fullsized":"2024-09-10T18:57:29.074242+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:39.805468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.805468+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.334135+0000","scrub_duration":0.0040878859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":283,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072121+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:57:29.072121+0000","last_peered":"2024-09-10T18:57:29.072121+0000","last_clean":"2024-09-10T18:57:29.072121+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:57:29.072121+0000","last_undegraded":"2024-09-10T18:57:29.072121+0000","last_fullsized":"2024-09-10T18:57:29.072121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074612+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:57:29.074612+0000","last_peered":"2024-09-10T18:57:29.074612+0000","last_clean":"2024-09-10T18:57:29.074612+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:57:29.074612+0000","last_undegraded":"2024-09-10T18:57:29.074612+0000","last_fullsized":"2024-09-10T18:57:29.074612+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":258,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067870+0000","last_change":"2024-09-10T18:55:10.179358+0000","last_active":"2024-09-10T18:57:29.067870+0000","last_peered":"2024-09-10T18:57:29.067870+0000","last_clean":"2024-09-10T18:57:29.067870+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:57:29.067870+0000","last_undegraded":"2024-09-10T18:57:29.067870+0000","last_fullsized":"2024-09-10T18:57:29.067870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.179141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.179141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003833071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063421+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:57:29.063421+0000","last_peered":"2024-09-10T18:57:29.063421+0000","last_clean":"2024-09-10T18:57:29.063421+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:57:29.063421+0000","last_undegraded":"2024-09-10T18:57:29.063421+0000","last_fullsized":"2024-09-10T18:57:29.063421+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061087+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:57:29.061087+0000","last_peered":"2024-09-10T18:57:29.061087+0000","last_clean":"2024-09-10T18:57:29.061087+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:57:29.061087+0000","last_undegraded":"2024-09-10T18:57:29.061087+0000","last_fullsized":"2024-09-10T18:57:29.061087+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"126'9","reported_seq":197,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074494+0000","last_change":"2024-09-10T18:56:09.704635+0000","last_active":"2024-09-10T18:57:29.074494+0000","last_peered":"2024-09-10T18:57:29.074494+0000","last_clean":"2024-09-10T18:57:29.074494+0000","last_became_active":"2024-09-10T18:54:37.646743+0000","last_became_peered":"2024-09-10T18:54:37.646743+0000","last_unstale":"2024-09-10T18:57:29.074494+0000","last_undegraded":"2024-09-10T18:57:29.074494+0000","last_fullsized":"2024-09-10T18:57:29.074494+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:30.575925+0000","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":291,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063140+0000","last_change":"2024-09-10T18:55:06.199960+0000","last_active":"2024-09-10T18:57:29.063140+0000","last_peered":"2024-09-10T18:57:29.063140+0000","last_clean":"2024-09-10T18:57:29.063140+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:57:29.063140+0000","last_undegraded":"2024-09-10T18:57:29.063140+0000","last_fullsized":"2024-09-10T18:57:29.063140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:55:06.199853+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:55:06.199853+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045914980000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074947+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:57:29.074947+0000","last_peered":"2024-09-10T18:57:29.074947+0000","last_clean":"2024-09-10T18:57:29.074947+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:57:29.074947+0000","last_undegraded":"2024-09-10T18:57:29.074947+0000","last_fullsized":"2024-09-10T18:57:29.074947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.423650+0000","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":269,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061640+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:57:29.061640+0000","last_peered":"2024-09-10T18:57:29.061640+0000","last_clean":"2024-09-10T18:57:29.061640+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:57:29.061640+0000","last_undegraded":"2024-09-10T18:57:29.061640+0000","last_fullsized":"2024-09-10T18:57:29.061640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:53.667997+0000","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067670+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:57:29.067670+0000","last_peered":"2024-09-10T18:57:29.067670+0000","last_clean":"2024-09-10T18:57:29.067670+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:57:29.067670+0000","last_undegraded":"2024-09-10T18:57:29.067670+0000","last_fullsized":"2024-09-10T18:57:29.067670+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:39.361501+0000","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061810+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:57:29.061810+0000","last_peered":"2024-09-10T18:57:29.061810+0000","last_clean":"2024-09-10T18:57:29.061810+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:57:29.061810+0000","last_undegraded":"2024-09-10T18:57:29.061810+0000","last_fullsized":"2024-09-10T18:57:29.061810+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:32.443415+0000","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":178,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069575+0000","last_change":"2024-09-10T18:54:37.725751+0000","last_active":"2024-09-10T18:57:29.069575+0000","last_peered":"2024-09-10T18:57:29.069575+0000","last_clean":"2024-09-10T18:57:29.069575+0000","last_became_active":"2024-09-10T18:54:37.716564+0000","last_became_peered":"2024-09-10T18:54:37.716564+0000","last_unstale":"2024-09-10T18:57:29.069575+0000","last_undegraded":"2024-09-10T18:57:29.069575+0000","last_fullsized":"2024-09-10T18:57:29.069575+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":272,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078308+0000","last_change":"2024-09-10T18:54:54.957426+0000","last_active":"2024-09-10T18:57:29.078308+0000","last_peered":"2024-09-10T18:57:29.078308+0000","last_clean":"2024-09-10T18:57:29.078308+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:57:29.078308+0000","last_undegraded":"2024-09-10T18:57:29.078308+0000","last_fullsized":"2024-09-10T18:57:29.078308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:54:54.957316+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:54:54.957316+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039293499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074036+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:57:29.074036+0000","last_peered":"2024-09-10T18:57:29.074036+0000","last_clean":"2024-09-10T18:57:29.074036+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:57:29.074036+0000","last_undegraded":"2024-09-10T18:57:29.074036+0000","last_fullsized":"2024-09-10T18:57:29.074036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066180+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:57:29.066180+0000","last_peered":"2024-09-10T18:57:29.066180+0000","last_clean":"2024-09-10T18:57:29.066180+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:57:29.066180+0000","last_undegraded":"2024-09-10T18:57:29.066180+0000","last_fullsized":"2024-09-10T18:57:29.066180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068115+0000","last_change":"2024-09-10T18:55:02.948472+0000","last_active":"2024-09-10T18:57:29.068115+0000","last_peered":"2024-09-10T18:57:29.068115+0000","last_clean":"2024-09-10T18:57:29.068115+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:57:29.068115+0000","last_undegraded":"2024-09-10T18:57:29.068115+0000","last_fullsized":"2024-09-10T18:57:29.068115+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.948218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.948218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034990780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070269+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:57:29.070269+0000","last_peered":"2024-09-10T18:57:29.070269+0000","last_clean":"2024-09-10T18:57:29.070269+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:57:29.070269+0000","last_undegraded":"2024-09-10T18:57:29.070269+0000","last_fullsized":"2024-09-10T18:57:29.070269+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.479269+0000","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":178,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069785+0000","last_change":"2024-09-10T18:54:37.661524+0000","last_active":"2024-09-10T18:57:29.069785+0000","last_peered":"2024-09-10T18:57:29.069785+0000","last_clean":"2024-09-10T18:57:29.069785+0000","last_became_active":"2024-09-10T18:54:37.657896+0000","last_became_peered":"2024-09-10T18:54:37.657896+0000","last_unstale":"2024-09-10T18:57:29.069785+0000","last_undegraded":"2024-09-10T18:57:29.069785+0000","last_fullsized":"2024-09-10T18:57:29.069785+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":285,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073904+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:57:29.073904+0000","last_peered":"2024-09-10T18:57:29.073904+0000","last_clean":"2024-09-10T18:57:29.073904+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:57:29.073904+0000","last_undegraded":"2024-09-10T18:57:29.073904+0000","last_fullsized":"2024-09-10T18:57:29.073904+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:33.059487+0000","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074092+0000","last_change":"2024-09-10T18:55:08.609833+0000","last_active":"2024-09-10T18:57:29.074092+0000","last_peered":"2024-09-10T18:57:29.074092+0000","last_clean":"2024-09-10T18:57:29.074092+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:57:29.074092+0000","last_undegraded":"2024-09-10T18:57:29.074092+0000","last_fullsized":"2024-09-10T18:57:29.074092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.609702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.609702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067775+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:57:29.067775+0000","last_peered":"2024-09-10T18:57:29.067775+0000","last_clean":"2024-09-10T18:57:29.067775+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:57:29.067775+0000","last_undegraded":"2024-09-10T18:57:29.067775+0000","last_fullsized":"2024-09-10T18:57:29.067775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.486395+0000","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067520+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:57:29.067520+0000","last_peered":"2024-09-10T18:57:29.067520+0000","last_clean":"2024-09-10T18:57:29.067520+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:57:29.067520+0000","last_undegraded":"2024-09-10T18:57:29.067520+0000","last_fullsized":"2024-09-10T18:57:29.067520+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.320880+0000","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":177,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.063425+0000","last_change":"2024-09-10T18:54:37.573584+0000","last_active":"2024-09-10T18:57:29.063425+0000","last_peered":"2024-09-10T18:57:29.063425+0000","last_clean":"2024-09-10T18:57:29.063425+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:57:29.063425+0000","last_undegraded":"2024-09-10T18:57:29.063425+0000","last_fullsized":"2024-09-10T18:57:29.063425+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":187,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070960+0000","last_change":"2024-09-10T18:56:40.177375+0000","last_active":"2024-09-10T18:57:29.070960+0000","last_peered":"2024-09-10T18:57:29.070960+0000","last_clean":"2024-09-10T18:57:29.070960+0000","last_became_active":"2024-09-10T18:54:37.565268+0000","last_became_peered":"2024-09-10T18:54:37.565268+0000","last_unstale":"2024-09-10T18:57:29.070960+0000","last_undegraded":"2024-09-10T18:57:29.070960+0000","last_fullsized":"2024-09-10T18:57:29.070960+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.177249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.177249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.578420+0000","scrub_duration":0.0033094600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070801+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:57:29.070801+0000","last_peered":"2024-09-10T18:57:29.070801+0000","last_clean":"2024-09-10T18:57:29.070801+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:57:29.070801+0000","last_undegraded":"2024-09-10T18:57:29.070801+0000","last_fullsized":"2024-09-10T18:57:29.070801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.123736+0000","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":303,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071887+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:57:29.071887+0000","last_peered":"2024-09-10T18:57:29.071887+0000","last_clean":"2024-09-10T18:57:29.071887+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:57:29.071887+0000","last_undegraded":"2024-09-10T18:57:29.071887+0000","last_fullsized":"2024-09-10T18:57:29.071887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.905881+0000","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":271,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067184+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:57:29.067184+0000","last_peered":"2024-09-10T18:57:29.067184+0000","last_clean":"2024-09-10T18:57:29.067184+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:57:29.067184+0000","last_undegraded":"2024-09-10T18:57:29.067184+0000","last_fullsized":"2024-09-10T18:57:29.067184+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":268,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.078326+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:57:29.078326+0000","last_peered":"2024-09-10T18:57:29.078326+0000","last_clean":"2024-09-10T18:57:29.078326+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:57:29.078326+0000","last_undegraded":"2024-09-10T18:57:29.078326+0000","last_fullsized":"2024-09-10T18:57:29.078326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.517080+0000","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074512+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:57:29.074512+0000","last_peered":"2024-09-10T18:57:29.074512+0000","last_clean":"2024-09-10T18:57:29.074512+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:57:29.074512+0000","last_undegraded":"2024-09-10T18:57:29.074512+0000","last_fullsized":"2024-09-10T18:57:29.074512+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.150161+0000","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"126'10","reported_seq":197,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074324+0000","last_change":"2024-09-10T18:56:28.793230+0000","last_active":"2024-09-10T18:57:29.074324+0000","last_peered":"2024-09-10T18:57:29.074324+0000","last_clean":"2024-09-10T18:57:29.074324+0000","last_became_active":"2024-09-10T18:54:37.562584+0000","last_became_peered":"2024-09-10T18:54:37.562584+0000","last_unstale":"2024-09-10T18:57:29.074324+0000","last_undegraded":"2024-09-10T18:57:29.074324+0000","last_fullsized":"2024-09-10T18:57:29.074324+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:28.793065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.793065+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:54.828376+0000","scrub_duration":0.0039084699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074729+0000","last_change":"2024-09-10T18:56:09.619685+0000","last_active":"2024-09-10T18:57:29.074729+0000","last_peered":"2024-09-10T18:57:29.074729+0000","last_clean":"2024-09-10T18:57:29.074729+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:57:29.074729+0000","last_undegraded":"2024-09-10T18:57:29.074729+0000","last_fullsized":"2024-09-10T18:57:29.074729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.619567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.619567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049369640000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":277,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066567+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:57:29.066567+0000","last_peered":"2024-09-10T18:57:29.066567+0000","last_clean":"2024-09-10T18:57:29.066567+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:57:29.066567+0000","last_undegraded":"2024-09-10T18:57:29.066567+0000","last_fullsized":"2024-09-10T18:57:29.066567+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":263,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068147+0000","last_change":"2024-09-10T18:54:57.678574+0000","last_active":"2024-09-10T18:57:29.068147+0000","last_peered":"2024-09-10T18:57:29.068147+0000","last_clean":"2024-09-10T18:57:29.068147+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:57:29.068147+0000","last_undegraded":"2024-09-10T18:57:29.068147+0000","last_fullsized":"2024-09-10T18:57:29.068147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.678482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.678482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024273350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":392,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.074004+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:57:29.074004+0000","last_peered":"2024-09-10T18:57:29.074004+0000","last_clean":"2024-09-10T18:57:29.074004+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:57:29.074004+0000","last_undegraded":"2024-09-10T18:57:29.074004+0000","last_fullsized":"2024-09-10T18:57:29.074004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073915+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:57:29.073915+0000","last_peered":"2024-09-10T18:57:29.073915+0000","last_clean":"2024-09-10T18:57:29.073915+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:57:29.073915+0000","last_undegraded":"2024-09-10T18:57:29.073915+0000","last_fullsized":"2024-09-10T18:57:29.073915+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:38.976604+0000","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"126'2","reported_seq":34,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.070000+0000","last_change":"2024-09-10T18:57:11.095107+0000","last_active":"2024-09-10T18:57:29.070000+0000","last_peered":"2024-09-10T18:57:29.070000+0000","last_clean":"2024-09-10T18:57:29.070000+0000","last_became_active":"2024-09-10T18:57:11.082286+0000","last_became_peered":"2024-09-10T18:57:11.082286+0000","last_unstale":"2024-09-10T18:57:29.070000+0000","last_undegraded":"2024-09-10T18:57:29.070000+0000","last_fullsized":"2024-09-10T18:57:29.070000+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:35.765725+0000","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.064188+0000","last_change":"2024-09-10T18:54:44.250455+0000","last_active":"2024-09-10T18:57:29.064188+0000","last_peered":"2024-09-10T18:57:29.064188+0000","last_clean":"2024-09-10T18:57:29.064188+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:57:29.064188+0000","last_undegraded":"2024-09-10T18:57:29.064188+0000","last_fullsized":"2024-09-10T18:57:29.064188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:44.250321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:54:44.250321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031387210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073794+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:57:29.073794+0000","last_peered":"2024-09-10T18:57:29.073794+0000","last_clean":"2024-09-10T18:57:29.073794+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:57:29.073794+0000","last_undegraded":"2024-09-10T18:57:29.073794+0000","last_fullsized":"2024-09-10T18:57:29.073794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":280,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067956+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:57:29.067956+0000","last_peered":"2024-09-10T18:57:29.067956+0000","last_clean":"2024-09-10T18:57:29.067956+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:57:29.067956+0000","last_undegraded":"2024-09-10T18:57:29.067956+0000","last_fullsized":"2024-09-10T18:57:29.067956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":277,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.075611+0000","last_change":"2024-09-10T18:56:26.777537+0000","last_active":"2024-09-10T18:57:29.075611+0000","last_peered":"2024-09-10T18:57:29.075611+0000","last_clean":"2024-09-10T18:57:29.075611+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:57:29.075611+0000","last_undegraded":"2024-09-10T18:57:29.075611+0000","last_fullsized":"2024-09-10T18:57:29.075611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.777271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.777271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:37.531064+0000","scrub_duration":0.0030242540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.071681+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:57:29.071681+0000","last_peered":"2024-09-10T18:57:29.071681+0000","last_clean":"2024-09-10T18:57:29.071681+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:57:29.071681+0000","last_undegraded":"2024-09-10T18:57:29.071681+0000","last_fullsized":"2024-09-10T18:57:29.071681+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.803420+0000","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":179,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.066510+0000","last_change":"2024-09-10T18:54:37.660686+0000","last_active":"2024-09-10T18:57:29.066510+0000","last_peered":"2024-09-10T18:57:29.066510+0000","last_clean":"2024-09-10T18:57:29.066510+0000","last_became_active":"2024-09-10T18:54:37.660393+0000","last_became_peered":"2024-09-10T18:54:37.660393+0000","last_unstale":"2024-09-10T18:57:29.066510+0000","last_undegraded":"2024-09-10T18:57:29.066510+0000","last_fullsized":"2024-09-10T18:57:29.066510+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":267,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.069260+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:57:29.069260+0000","last_peered":"2024-09-10T18:57:29.069260+0000","last_clean":"2024-09-10T18:57:29.069260+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:57:29.069260+0000","last_undegraded":"2024-09-10T18:57:29.069260+0000","last_fullsized":"2024-09-10T18:57:29.069260+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.573240+0000","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":273,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.067724+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:57:29.067724+0000","last_peered":"2024-09-10T18:57:29.067724+0000","last_clean":"2024-09-10T18:57:29.067724+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:57:29.067724+0000","last_undegraded":"2024-09-10T18:57:29.067724+0000","last_fullsized":"2024-09-10T18:57:29.067724+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:51.997928+0000","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":287,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.061497+0000","last_change":"2024-09-10T18:54:58.424151+0000","last_active":"2024-09-10T18:57:29.061497+0000","last_peered":"2024-09-10T18:57:29.061497+0000","last_clean":"2024-09-10T18:57:29.061497+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:57:29.061497+0000","last_undegraded":"2024-09-10T18:57:29.061497+0000","last_fullsized":"2024-09-10T18:57:29.061497+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.424008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.424008+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.146668564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":259,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.068688+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:57:29.068688+0000","last_peered":"2024-09-10T18:57:29.068688+0000","last_clean":"2024-09-10T18:57:29.068688+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:57:29.068688+0000","last_undegraded":"2024-09-10T18:57:29.068688+0000","last_fullsized":"2024-09-10T18:57:29.068688+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":188,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.073408+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:57:29.073408+0000","last_peered":"2024-09-10T18:57:29.073408+0000","last_clean":"2024-09-10T18:57:29.073408+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:57:29.073408+0000","last_undegraded":"2024-09-10T18:57:29.073408+0000","last_fullsized":"2024-09-10T18:57:29.073408+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:52.050482+0000","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":186,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-09-10T18:57:29.072282+0000","last_change":"2024-09-10T18:56:21.650671+0000","last_active":"2024-09-10T18:57:29.072282+0000","last_peered":"2024-09-10T18:57:29.072282+0000","last_clean":"2024-09-10T18:57:29.072282+0000","last_became_active":"2024-09-10T18:54:37.583272+0000","last_became_peered":"2024-09-10T18:54:37.583272+0000","last_unstale":"2024-09-10T18:57:29.072282+0000","last_undegraded":"2024-09-10T18:57:29.072282+0000","last_fullsized":"2024-09-10T18:57:29.072282+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:43.232910+0000","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":176,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":4195498,"num_objects":119,"num_object_clones":1,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":95,"num_read_kb":1,"num_write":539,"num_write_kb":8725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14020608,"data_stored":12586494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1785,"ondisk_log_size":1785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":467,"num_read_kb":462,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":27,"stat_sum":{"num_bytes":1792,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":81,"acting":81,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972462,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318988,"kb_used_data":9132,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93426164,"statfs":{"total":95995035648,"available":95668391936,"internally_reserved":0,"allocated":9351168,"data_stored":7275521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.89700000000000002,"15min":0.91900000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2150000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.92000000000000004},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.2430000000000001,"5min":1.554,"15min":1.554},"last":0.876}]},{"osd":1,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.88800000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3640000000000001,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96099999999999997,"15min":1.0189999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.286,"5min":1.742,"15min":1.8280000000000001},"last":0.97499999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.022,"15min":1.046},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.421,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.98199999999999998,"15min":0.97899999999999998},"min":{"1min":0.51400000000000001,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":1.395,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.034}]},{"osd":3,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1,"15min":1.01},"min":{"1min":0.56799999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.54,"5min":2.8780000000000001,"15min":2.8780000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.97999999999999998,"15min":0.96799999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1719999999999999,"5min":3.3050000000000002,"15min":3.3050000000000002},"last":0.77600000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0229999999999999,"15min":1.022},"min":{"1min":0.499,"5min":0.499,"15min":0.443},"max":{"1min":1.4219999999999999,"5min":1.556,"15min":2.1989999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.0660000000000001,"15min":1.0369999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.669,"5min":1.669,"15min":2.1120000000000001},"last":0.94899999999999995}]},{"osd":5,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.98499999999999999,"15min":0.99199999999999999},"min":{"1min":0.51700000000000002,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.613,"5min":1.613,"15min":2.0819999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.163,"15min":1.1799999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.514,"5min":1.514,"15min":2.1419999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.1950000000000001,"15min":1.232},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.569,"5min":1.569,"15min":2.2749999999999999},"last":1.163},{"interface":"front","average":{"1min":1.248,"5min":1.1830000000000001,"15min":1.1879999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.53100000000000003},"max":{"1min":1.637,"5min":1.637,"15min":2.536},"last":1.1020000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299512,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318964,"kb_used_data":4956,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93426188,"statfs":{"total":95995035648,"available":95668416512,"internally_reserved":0,"allocated":5074944,"data_stored":3065617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.86699999999999999,"15min":0.88300000000000001},"min":{"1min":0.502,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3799999999999999,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.96299999999999997,"15min":0.93899999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3580000000000001,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.91900000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.1830000000000001,"15min":0.98799999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.3089999999999999,"5min":18.494,"15min":18.494},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.871,"15min":0.90000000000000002},"min":{"1min":0.57199999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.2110000000000001,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.85299999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0149999999999999,"15min":0.95199999999999996},"min":{"1min":0.66900000000000004,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.321,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.91700000000000004,"15min":0.92300000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.2470000000000001,"5min":1.988,"15min":1.988},"last":0.72499999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.0069999999999999,"15min":0.96999999999999997},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2989999999999999,"5min":2.278,"15min":2.278},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.97299999999999998,"15min":0.96899999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.121,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":0.94399999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.97499999999999998,"15min":0.90000000000000002},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.3560000000000001,"5min":1.712,"15min":1.712},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0640000000000001,"15min":1.0249999999999999},"min":{"1min":0.67700000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.429,"5min":2.202,"15min":2.202},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Sep 10 18:56:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.042,"15min":0.99299999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.4690000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.131,"15min":1.099},"min":{"1min":0.73299999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.4670000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.77000000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:56:47 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.78900000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1.401,"5min":1.401,"15min":1.573},"last":1.149},{"interface":"front","average":{"1min":1.107,"5min":1.1619999999999999,"15min":1.1459999999999999},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.4510000000000001,"5min":1.77,"15min":1.77},"last":1.0609999999999999}]}]},{"osd":5,"up_from":43,"seq":184683593858,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318416,"kb_used_data":8560,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93426736,"statfs":{"total":95995035648,"available":95668977664,"internally_reserved":0,"allocated":8765440,"data_stored":6684138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.0820000000000001,"15min":0.92900000000000005},"min":{"1min":0.41599999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.034},{"interface":"front","average":{"1min":1.292,"5min":1.024,"15min":0.91300000000000003},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0519999999999996,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.96699999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4369999999999998,"5min":1.6579999999999999,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":15.824,"5min":31.548999999999999,"15min":31.548999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.361,"5min":1.427,"15min":1.069},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.0839999999999996,"5min":31.847000000000001,"15min":31.847000000000001},"last":0.83099999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":1.8899999999999999,"15min":1.2250000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":20.536000000000001,"5min":61.347000000000001,"15min":61.347000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.383,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.60299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":5.2489999999999997,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.64000000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.1279999999999999,"15min":0.98699999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5979999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.139},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.7609999999999999,"15min":1.218},"min":{"1min":0.66400000000000003,"5min":0.60999999999999999,"15min":0.57399999999999995},"max":{"1min":7.7679999999999998,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.117}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.147,"15min":0.90600000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":13.005000000000001,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.544,"5min":1.1659999999999999,"15min":0.96899999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":7.6299999999999999,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.002}]},{"osd":6,"last update":"Tue Sep 10 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.252,"15min":1.0920000000000001},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.3520000000000001,"15min":1.143},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.47199999999999998},"max":{"1min":9.6760000000000002,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":0.78800000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.268,"15min":1.131},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.016,"5min":9.1140000000000008,"15min":9.1140000000000008},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.2789999999999999,"15min":1.204},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":4.7949999999999999,"5min":6.532,"15min":6.532},"last":0.89700000000000002}]}]},{"osd":4,"up_from":35,"seq":150323855502,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301708,"kb_used_data":4356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443444,"statfs":{"total":95995035648,"available":95686086656,"internally_reserved":0,"allocated":4460544,"data_stored":2475614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81000000000000005,"15min":0.84099999999999997},"min":{"1min":0.505,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3320000000000001,"5min":1.546,"15min":1.546},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.89500000000000002,"15min":0.86299999999999999},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.4750000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.002}]},{"osd":1,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.85099999999999998,"15min":0.878},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.93700000000000006,"15min":0.88900000000000001},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.323,"5min":1.575,"15min":1.575},"last":0.85099999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.4179999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.90800000000000003},"min":{"1min":0.60099999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.482,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":1.125}]},{"osd":3,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94299999999999995,"15min":0.91800000000000004},"min":{"1min":0.64400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.714},"last":0.876},{"interface":"front","average":{"1min":1.006,"5min":0.96799999999999997,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.093}]},{"osd":5,"last update":"Tue Sep 10 18:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96199999999999997,"15min":0.88900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.3939999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.044},{"interface":"front","average":{"1min":0.999,"5min":1.026,"15min":0.94399999999999995},"min":{"1min":0.65100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.256,"5min":1.528,"15min":1.7909999999999999},"last":1.196}]},{"osd":6,"last update":"Tue Sep 10 18:57:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0349999999999999,"15min":0.96899999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.135,"15min":1.0860000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4670000000000001,"5min":1.581,"15min":1.581},"last":1.294}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1359999999999999,"15min":1.1619999999999999},"min":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.516,"5min":1.6200000000000001,"15min":2.008},"last":1.27},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.3240000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051737,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310156,"kb_used_data":4460,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93434996,"statfs":{"total":95995035648,"available":95677435904,"internally_reserved":0,"allocated":4567040,"data_stored":2489953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2150000000000001,"15min":0.82199999999999995},"min":{"1min":0.38600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":8.6180000000000003,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.002,"5min":1.2410000000000001,"15min":0.89100000000000001},"min":{"1min":0.441,"5min":0.252,"15min":0.252},"max":{"1min":5.3479999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.69399999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.3819999999999999,"15min":0.91600000000000004},"min":{"1min":0.41799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.3879999999999999,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.022,"5min":1.0600000000000001,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.0549999999999997,"5min":15.189,"15min":15.189},"last":1.0469999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.3660000000000001,"15min":0.98699999999999999},"min":{"1min":0.502,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1920000000000002,"5min":26.558,"15min":26.558},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.2410000000000001,"15min":0.91900000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.3199999999999998,"5min":25.536999999999999,"15min":25.536999999999999},"last":1.1060000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.3049999999999999,"15min":1.111},"min":{"1min":0.73799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":7.3550000000000004,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.486},{"interface":"front","average":{"1min":1.522,"5min":1.411,"15min":1.198},"min":{"1min":0.79000000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":7.4189999999999996,"5min":10.391,"15min":10.391},"last":1.53}]},{"osd":5,"last update":"Tue Sep 10 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.2130000000000001},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":8.6679999999999993,"5min":17.535,"15min":17.535},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.123,"15min":1.028},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":8.7880000000000003,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":1.21}]},{"osd":6,"last update":"Tue Sep 10 18:56:59 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.2709999999999999,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.5289999999999999,"5min":10.51,"15min":10.51},"last":1.431},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.135,"15min":1.0920000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.8279999999999998,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.2350000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.2989999999999999,"15min":1.175},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.4570000000000001,"5min":18.312999999999999,"15min":18.312999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4319999999999999,"15min":1.306},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":7.298,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.3799999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378786,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305912,"kb_used_data":8552,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93439240,"statfs":{"total":95995035648,"available":95681781760,"internally_reserved":0,"allocated":8757248,"data_stored":6670497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.84099999999999997,"15min":0.70799999999999996},"min":{"1min":0.32200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.84999999999999998,"5min":9.516,"15min":9.516},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.82799999999999996,"15min":0.76200000000000001},"min":{"1min":0.25800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0620000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.61499999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:56:35 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.72199999999999998,"15min":0.69299999999999995},"min":{"1min":0.36699999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.82999999999999996,"5min":5.516,"15min":5.516},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.88900000000000001,"15min":0.79400000000000004},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.052,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.52300000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.89400000000000002,"15min":0.85399999999999998},"min":{"1min":0.52500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.139,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.84599999999999997},"min":{"1min":0.53800000000000003,"5min":0.434,"15min":0.434},"max":{"1min":2.0539999999999998,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.77300000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.171,"15min":1.0389999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2909999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1519999999999999,"15min":1.1040000000000001},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4379999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.80100000000000005}]},{"osd":5,"last update":"Tue Sep 10 18:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.113,"15min":1.109},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.605,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.214,"15min":1.085},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.429,"5min":21.654,"15min":21.654},"last":1.0509999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.206,"15min":1.1870000000000001},"min":{"1min":0.622,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5009999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.093},{"interface":"front","average":{"1min":1.117,"5min":1.252,"15min":1.129},"min":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.5669999999999999,"5min":15.971,"15min":15.971},"last":1.1100000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.21,"15min":1.2070000000000001},"min":{"1min":0.65800000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.399,"5min":15.025,"15min":15.025},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.147,"5min":1.2869999999999999,"15min":1.173},"min":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.0299999999999998,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.028}]}]},{"osd":1,"up_from":13,"seq":55834575021,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314268,"kb_used_data":4420,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430884,"statfs":{"total":95995035648,"available":95673225216,"internally_reserved":0,"allocated":4526080,"data_stored":2485552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0309999999999999,"15min":0.749},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.90500000000000003,"5min":32.677999999999997,"15min":32.677999999999997},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.095,"15min":0.81699999999999995},"min":{"1min":0.30399999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.0529999999999999,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.755}]},{"osd":2,"last update":"Tue Sep 10 18:56:34 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1040000000000001,"15min":0.81200000000000006},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1299999999999999,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":0.93000000000000005},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.4370000000000001,"5min":32.363,"15min":32.363},"last":1.006}]},{"osd":3,"last update":"Tue Sep 10 18:57:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.208,"15min":0.95899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7360000000000002,"5min":25.704000000000001,"15min":25.704000000000001},"last":1.091},{"interface":"front","average":{"1min":0.754,"5min":1.476,"15min":0.98799999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1890000000000001,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1819999999999999,"15min":1.0669999999999999},"min":{"1min":0.47399999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.5379999999999998,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.262},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.6639999999999999,"15min":1.1619999999999999},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":6.5430000000000001,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.325}]},{"osd":5,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.2789999999999999,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":2.2719999999999998,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.202},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.3220000000000001,"15min":1.1919999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":4.5869999999999997,"5min":12.84,"15min":12.84},"last":1.363}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.528,"15min":1.232},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":8.1760000000000002,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.454},{"interface":"front","average":{"1min":1.036,"5min":1.4450000000000001,"15min":1.1990000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.486,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.2210000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.486,"15min":1.306},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":3.6499999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.411},{"interface":"front","average":{"1min":1.048,"5min":1.5409999999999999,"15min":1.278},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.5740000000000001,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.1699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738553,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314812,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430340,"statfs":{"total":95995035648,"available":95672668160,"internally_reserved":0,"allocated":5083136,"data_stored":3065065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:56:49 2024","interfaces":[{"interface":"back","average":{"1min":4.8890000000000002,"5min":1.631,"15min":0.94099999999999995},"min":{"1min":0.34000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":80.710999999999999,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":4.9489999999999998,"5min":1.714,"15min":1.024},"min":{"1min":0.41999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":86.009,"5min":86.009,"15min":86.009},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.601,"15min":0.99299999999999999},"min":{"1min":0.32900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":1.474,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.2,"15min":0.88300000000000001},"min":{"1min":0.48299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.2450000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0820000000000001,"15min":0.86399999999999999},"min":{"1min":0.38500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3680000000000003,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.45,"15min":1.0129999999999999},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.0019999999999998,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.754}]},{"osd":4,"last update":"Tue Sep 10 18:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.1160000000000001,"15min":1.0189999999999999},"min":{"1min":0.72399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.3799999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.369},{"interface":"front","average":{"1min":1.022,"5min":1.143,"15min":1.0760000000000001},"min":{"1min":0.63100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4990000000000001,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.3580000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.246,"15min":1.0940000000000001},"min":{"1min":0.71199999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.776,"5min":10.048,"15min":10.048},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.1319999999999999,"15min":1.1000000000000001},"min":{"1min":0.75700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.635,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:56:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.1220000000000001},"min":{"1min":0.69099999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.089},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.3200000000000001,"15min":1.1819999999999999},"min":{"1min":0.66400000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.696,"5min":11.894,"15min":11.894},"last":1.1339999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:56:49 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.1890000000000001,"15min":1.171},"min":{"1min":0.66500000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.675,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.107},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.321,"15min":1.3049999999999999},"min":{"1min":0.75900000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":8.2840000000000007,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.0109999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4411392,"data_stored":4194848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":4194840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":4194808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":176,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:57:36.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-10T18:57:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:36.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:36.377 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:36.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:36.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:36.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:36.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:36.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.756088+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-10T18:57:36.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:34.758986+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-10T18:57:36.766 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.266737+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.279429+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.345015+0000 osd.7 (osd.7) 194 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.352703+0000 osd.7 (osd.7) 195 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.619908+0000 osd.6 (osd.6) 141 : cluster [DBG] 64.8 scrub starts 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.624382+0000 osd.6 (osd.6) 142 : cluster [DBG] 64.8 scrub ok 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.807717+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:35.811962+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.010023+0000 mgr.y (mgr.24526) 341 : cluster [DBG] pgmap v433: 255 pgs: 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.010590+0000 mon.c (mon.2) 507 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.012541+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.767 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.055329+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.056257+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.056519+0000 mon.a (mon.0) 1971 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.056735+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.124473+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.156606+0000 osd.3 (osd.3) 204 : cluster [DBG] 16.7 scrub starts 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.160269+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub ok 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.533334+0000 mon.c (mon.2) 509 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.533758+0000 mon.a (mon.0) 1973 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.534676+0000 mon.c (mon.2) 510 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.535186+0000 mon.a (mon.0) 1974 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.535963+0000 mon.c (mon.2) 511 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:36 smithi165 bash[22591]: audit 2024-09-10T18:57:36.536525+0000 mon.a (mon.0) 1975 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:36.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.756088+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:34.758986+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.266737+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.279429+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.345015+0000 osd.7 (osd.7) 194 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:57:36.813 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.352703+0000 osd.7 (osd.7) 195 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.619908+0000 osd.6 (osd.6) 141 : cluster [DBG] 64.8 scrub starts 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.624382+0000 osd.6 (osd.6) 142 : cluster [DBG] 64.8 scrub ok 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.807717+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:35.811962+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.010023+0000 mgr.y (mgr.24526) 341 : cluster [DBG] pgmap v433: 255 pgs: 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.010590+0000 mon.c (mon.2) 507 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.012541+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.055329+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.056257+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.056519+0000 mon.a (mon.0) 1971 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.056735+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.124473+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.156606+0000 osd.3 (osd.3) 204 : cluster [DBG] 16.7 scrub starts 2024-09-10T18:57:36.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.160269+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub ok 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.533334+0000 mon.c (mon.2) 509 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.533758+0000 mon.a (mon.0) 1973 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.534676+0000 mon.c (mon.2) 510 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.535186+0000 mon.a (mon.0) 1974 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.535963+0000 mon.c (mon.2) 511 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[32381]: audit 2024-09-10T18:57:36.536525+0000 mon.a (mon.0) 1975 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.756088+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.c scrub starts 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:34.758986+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.c scrub ok 2024-09-10T18:57:36.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.266737+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.1c scrub starts 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.279429+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.1c scrub ok 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.345015+0000 osd.7 (osd.7) 194 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.352703+0000 osd.7 (osd.7) 195 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.619908+0000 osd.6 (osd.6) 141 : cluster [DBG] 64.8 scrub starts 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.624382+0000 osd.6 (osd.6) 142 : cluster [DBG] 64.8 scrub ok 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.807717+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:35.811962+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.010023+0000 mgr.y (mgr.24526) 341 : cluster [DBG] pgmap v433: 255 pgs: 255 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.010590+0000 mon.c (mon.2) 507 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.012541+0000 mon.a (mon.0) 1968 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.055329+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi083-77640-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.056257+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? v1:172.21.15.83:0/1714781422' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi083-77648-49"}]': finished 2024-09-10T18:57:36.816 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.056519+0000 mon.a (mon.0) 1971 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.056735+0000 mon.a (mon.0) 1972 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.124473+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.156606+0000 osd.3 (osd.3) 204 : cluster [DBG] 16.7 scrub starts 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.160269+0000 osd.3 (osd.3) 205 : cluster [DBG] 16.7 scrub ok 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.533334+0000 mon.c (mon.2) 509 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.533758+0000 mon.a (mon.0) 1973 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.534676+0000 mon.c (mon.2) 510 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.535186+0000 mon.a (mon.0) 1974 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.535963+0000 mon.c (mon.2) 511 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.817 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:36 smithi083 bash[18629]: audit 2024-09-10T18:57:36.536525+0000 mon.a (mon.0) 1975 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:36.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:36.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:37.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:37.061 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:37.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:37.207 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:37.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:37.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:37.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:37.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:37.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:37.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.382796+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:57:37.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.386140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:57:37.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.555790+0000 mon.a (mon.0) 1976 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:37.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.582925+0000 osd.0 (osd.0) 190 : cluster [DBG] 16.8 scrub starts 2024-09-10T18:57:37.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.585725+0000 osd.0 (osd.0) 191 : cluster [DBG] 16.8 scrub ok 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.064878+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.065254+0000 mon.a (mon.0) 1977 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.065447+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.068115+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.069819+0000 mon.a (mon.0) 1979 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.070480+0000 mon.a (mon.0) 1980 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[18629]: audit 2024-09-10T18:57:37.125825+0000 mon.c (mon.2) 513 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.382796+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.386140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.555790+0000 mon.a (mon.0) 1976 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.582925+0000 osd.0 (osd.0) 190 : cluster [DBG] 16.8 scrub starts 2024-09-10T18:57:37.601 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.585725+0000 osd.0 (osd.0) 191 : cluster [DBG] 16.8 scrub ok 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.064878+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.065254+0000 mon.a (mon.0) 1977 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.065447+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.068115+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.069819+0000 mon.a (mon.0) 1979 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.070480+0000 mon.a (mon.0) 1980 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.602 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:37 smithi083 bash[32381]: audit 2024-09-10T18:57:37.125825+0000 mon.c (mon.2) 513 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:37.621 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:37.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.382796+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.12 scrub starts 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.386140+0000 osd.7 (osd.7) 197 : cluster [DBG] 6.12 scrub ok 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.555790+0000 mon.a (mon.0) 1976 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.582925+0000 osd.0 (osd.0) 190 : cluster [DBG] 16.8 scrub starts 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.585725+0000 osd.0 (osd.0) 191 : cluster [DBG] 16.8 scrub ok 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.064878+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.065254+0000 mon.a (mon.0) 1977 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi083-77648-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:37.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.065447+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-09-10T18:57:37.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.068115+0000 mon.c (mon.2) 512 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.069819+0000 mon.a (mon.0) 1979 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:37.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.070480+0000 mon.a (mon.0) 1980 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:37.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[22591]: audit 2024-09-10T18:57:37.125825+0000 mon.c (mon.2) 513 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:37.758 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:37.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:37.894 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:37.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:37.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:38.026 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T18:57:38.026 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:57:38.030 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:38.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:38.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:38.165 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:38.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:38.305 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:38.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:38.437 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:38.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:38.677 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.836458+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.f scrub starts 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:36.839969+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.f scrub ok 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.576628+0000 osd.6 (osd.6) 143 : cluster [DBG] 12.4 scrub starts 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.580100+0000 osd.6 (osd.6) 144 : cluster [DBG] 12.4 scrub ok 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: audit 2024-09-10T18:57:37.802223+0000 mgr.y (mgr.24526) 342 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.855368+0000 osd.4 (osd.4) 217 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.858573+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:57:38.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.011159+0000 mgr.y (mgr.24526) 343 : cluster [DBG] pgmap v436: 263 pgs: 40 unknown, 223 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: audit 2024-09-10T18:57:38.506886+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: audit 2024-09-10T18:57:38.508110+0000 mon.a (mon.0) 1981 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.508411+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: audit 2024-09-10T18:57:38.510141+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[32381]: audit 2024-09-10T18:57:38.514364+0000 mon.a (mon.0) 1984 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.836458+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.f scrub starts 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:36.839969+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.f scrub ok 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.576628+0000 osd.6 (osd.6) 143 : cluster [DBG] 12.4 scrub starts 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.580100+0000 osd.6 (osd.6) 144 : cluster [DBG] 12.4 scrub ok 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: audit 2024-09-10T18:57:37.802223+0000 mgr.y (mgr.24526) 342 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:38.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.855368+0000 osd.4 (osd.4) 217 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.858573+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.011159+0000 mgr.y (mgr.24526) 343 : cluster [DBG] pgmap v436: 263 pgs: 40 unknown, 223 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: audit 2024-09-10T18:57:38.506886+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: audit 2024-09-10T18:57:38.508110+0000 mon.a (mon.0) 1981 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.508411+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: audit 2024-09-10T18:57:38.510141+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:38.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:38 smithi083 bash[18629]: audit 2024-09-10T18:57:38.514364+0000 mon.a (mon.0) 1984 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.711 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:38.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T18:57:38.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:38.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.836458+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.f scrub starts 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:36.839969+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.f scrub ok 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.576628+0000 osd.6 (osd.6) 143 : cluster [DBG] 12.4 scrub starts 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.580100+0000 osd.6 (osd.6) 144 : cluster [DBG] 12.4 scrub ok 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: audit 2024-09-10T18:57:37.802223+0000 mgr.y (mgr.24526) 342 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.855368+0000 osd.4 (osd.4) 217 : cluster [DBG] 6.1e scrub starts 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.858573+0000 osd.4 (osd.4) 218 : cluster [DBG] 6.1e scrub ok 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.011159+0000 mgr.y (mgr.24526) 343 : cluster [DBG] pgmap v436: 263 pgs: 40 unknown, 223 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: audit 2024-09-10T18:57:38.506886+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: audit 2024-09-10T18:57:38.508110+0000 mon.a (mon.0) 1981 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.508411+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: audit 2024-09-10T18:57:38.510141+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:38.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:38 smithi165 bash[22591]: audit 2024-09-10T18:57:38.514364+0000 mon.a (mon.0) 1984 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:38.846 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:38.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:38.978 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:38.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:39.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:39.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:39.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:39.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:39.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:39.379 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:39.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:39.516 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:39.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:39.652 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:39.757 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:39.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.833388+0000 osd.2 (osd.2) 121 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:37.839190+0000 osd.2 (osd.2) 122 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.612445+0000 osd.6 (osd.6) 145 : cluster [DBG] 64.e scrub starts 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.617194+0000 osd.6 (osd.6) 146 : cluster [DBG] 64.e scrub ok 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.840018+0000 osd.2 (osd.2) 123 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:38.843599+0000 osd.2 (osd.2) 124 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.068469+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.069649+0000 mon.a (mon.0) 1985 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.069925+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.070068+0000 mon.a (mon.0) 1987 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:39.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.070235+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:57:39.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.071857+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:39.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: audit 2024-09-10T18:57:39.074026+0000 mon.a (mon.0) 1990 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:39.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.150283+0000 osd.3 (osd.3) 206 : cluster [DBG] 12.b scrub starts 2024-09-10T18:57:39.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:39 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.153556+0000 osd.3 (osd.3) 207 : cluster [DBG] 12.b scrub ok 2024-09-10T18:57:39.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:39.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:39.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:40.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.833388+0000 osd.2 (osd.2) 121 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:37.839190+0000 osd.2 (osd.2) 122 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.612445+0000 osd.6 (osd.6) 145 : cluster [DBG] 64.e scrub starts 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.617194+0000 osd.6 (osd.6) 146 : cluster [DBG] 64.e scrub ok 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.840018+0000 osd.2 (osd.2) 123 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:38.843599+0000 osd.2 (osd.2) 124 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.068469+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.069649+0000 mon.a (mon.0) 1985 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.069925+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.070068+0000 mon.a (mon.0) 1987 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.070235+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.071857+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: audit 2024-09-10T18:57:39.074026+0000 mon.a (mon.0) 1990 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.150283+0000 osd.3 (osd.3) 206 : cluster [DBG] 12.b scrub starts 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.153556+0000 osd.3 (osd.3) 207 : cluster [DBG] 12.b scrub ok 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.833388+0000 osd.2 (osd.2) 121 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:37.839190+0000 osd.2 (osd.2) 122 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.612445+0000 osd.6 (osd.6) 145 : cluster [DBG] 64.e scrub starts 2024-09-10T18:57:40.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.617194+0000 osd.6 (osd.6) 146 : cluster [DBG] 64.e scrub ok 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.840018+0000 osd.2 (osd.2) 123 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:38.843599+0000 osd.2 (osd.2) 124 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.068469+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.069649+0000 mon.a (mon.0) 1985 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi083-77648-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.069925+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.070068+0000 mon.a (mon.0) 1987 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.070235+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.071857+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]: dispatch 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: audit 2024-09-10T18:57:39.074026+0000 mon.a (mon.0) 1990 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.150283+0000 osd.3 (osd.3) 206 : cluster [DBG] 12.b scrub starts 2024-09-10T18:57:40.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:39 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.153556+0000 osd.3 (osd.3) 207 : cluster [DBG] 12.b scrub ok 2024-09-10T18:57:40.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:40.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:40.204 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:40.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:40.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/2308972619 >> v1:172.21.15.83:6789/0 conn(0x7f15b00fd600 legacy=0x7f15b00fd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/2308972619 shutdown_connections 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/2308972619 >> 172.21.15.83:0/2308972619 conn(0x7f15b0075290 msgr2=0x7f15b00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/2308972619 shutdown_connections 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/2308972619 wait complete. 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 librados: starting msgr at 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 librados: starting objecter 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 Processor -- start 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- start start 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 librados: setting wanted keys 2024-09-10T18:57:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 librados: calling monclient init 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15b011c0c0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15b011d280 con 0x7f15b00fd600 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15b011e480 con 0x7f15b0108f30 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f15b00fe1d0 0x7f15b0104920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44494/0 (socket says 172.21.15.83:44494) 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a7fff640 1 -- 172.21.15.83:0/3118551563 learned_addr learned my addr 172.21.15.83:0/3118551563 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter ms_handle_connect 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739604758 0 0) 0x7f15b011c0c0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c003720 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter ms_handle_connect 0x7f15b00fd600 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753822519 0 0) 0x7f159c003720 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15b011c0c0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1598002950 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908149411 0 0) 0x7f15b011d280 con 0x7f15b00fd600 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c003720 con 0x7f15b00fd600 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3546545228 0 0) 0x7f15b011c0c0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 >> v1:172.21.15.83:6790/0 conn(0x7f15b0108f30 legacy=0x7f15b011a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 >> v1:172.21.15.165:6789/0 conn(0x7f15b00fd600 legacy=0x7f15b0104210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b011f680 con 0x7f15b00fe1d0 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter ms_handle_connect 0x7f15b0108f30 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1598002c90 con 0x7f15b00fe1d0 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1598005ba0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b011d470 con 0x7f15b00fe1d0 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 10 client.25080.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b011d8c0 con 0x7f15b00fe1d0 2024-09-10T18:57:40.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.391+0000 7f15b49d9640 1 librados: init done 2024-09-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b0065740 con 0x7f15b00fe1d0 2024-09-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/2010051622 >> v1:172.21.15.83:6790/0 conn(0x7ff42010f680 legacy=0x7ff420111a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/2010051622 shutdown_connections 2024-09-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/2010051622 >> 172.21.15.83:0/2010051622 conn(0x7ff4200fa370 msgr2=0x7ff4200fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:40.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/2010051622 shutdown_connections 2024-09-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/2010051622 wait complete. 2024-09-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1598001660 con 0x7f15b00fe1d0 2024-09-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 librados: starting msgr at 2024-09-10T18:57:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 librados: starting objecter 2024-09-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(304..304 src has 69..304) v4 ==== 9125+0+0 (unknown 2107270570 0 0) 0x7f159808ab30 con 0x7f15b00fe1d0 2024-09-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 10 client.25080.objecter ms_dispatch 0x7f15b0000b90 osd_map(304..304 src has 69..304) v4 2024-09-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 3 client.25080.objecter handle_osd_map got epochs [304,304] > 0 2024-09-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 3 client.25080.objecter handle_osd_map decoding full epoch 304 2024-09-10T18:57:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 Processor -- start 2024-09-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 20 client.25080.objecter dump_active .. 0 homeless 2024-09-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- start start 2024-09-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 librados: setting wanted keys 2024-09-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 librados: calling monclient init 2024-09-10T18:57:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7f15a6ffd640 10 client.25080.objecter ms_handle_connect 0x7f159c071610 2024-09-10T18:57:40.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff42010f3a0 con 0x7ff420071c40 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff420123a70 con 0x7ff42010f680 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff420124c10 con 0x7ff420072720 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff425ae8640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff420071c40 0x7ff42010e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:44520/0 (socket says 172.21.15.83:44520) 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff425ae8640 1 -- 172.21.15.83:0/1962598025 learned_addr learned my addr 172.21.15.83:0/1962598025 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter ms_handle_connect 0x7ff420071c40 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter ms_handle_connect 0x7ff420072720 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497474084 0 0) 0x7ff420124c10 con 0x7ff420072720 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40c0054b0 con 0x7ff420072720 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter ms_handle_connect 0x7ff42010f680 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:40.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203781848 0 0) 0x7ff40c0054b0 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff420124c10 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff410001880 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817680459 0 0) 0x7ff420123a70 con 0x7ff42010f680 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40c0054b0 con 0x7ff42010f680 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4146367905 0 0) 0x7ff420124c10 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.165:6789/0 conn(0x7ff42010f680 legacy=0x7ff420121340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6789/0 conn(0x7ff420071c40 legacy=0x7ff42010e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420124da0 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff420125050 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 10 client.49151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff420125520 con 0x7ff420072720 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff427d4c640 1 librados: init done 2024-09-10T18:57:40.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff410002050 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.395+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff4100032a0 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff404004ff0 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff410001bf0 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(304..304 src has 69..304) v4 ==== 9125+0+0 (unknown 2107270570 0 0) 0x7ff4100880b0 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 osd_map(304..304 src has 69..304) v4 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 3 client.49151.objecter handle_osd_map got epochs [304,304] > 0 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 3 client.49151.objecter handle_osd_map decoding full epoch 304 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 20 client.49151.objecter dump_active .. 0 homeless 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v304) v1 ==== 54+0+16 (unknown 1181836021 0 2420000192) 0x7ff41008a4c0 con 0x7ff420072720 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff40c071530 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 10 client.49151.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 1 2024-09-10T18:57:40.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 1 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 0 1 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 4 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 10 client.49151.objecter _send_command 1 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff404004a60 con 0x7ff404004610 2024-09-10T18:57:40.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.399+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404004610 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1598056c90 con 0x7f15b00fe1d0 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff404004a60 con 0x7ff404004610 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7ff4167fc640 10 client.49151.objecter _finish_command 1 = system:0 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 4 2024-09-10T18:57:40.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.403+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 0 1 2024-09-10T18:57:40.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 10 client.49151.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 2 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 2 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404005290 osd=0 3 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 0 2 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 4 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 10 client.49151.objecter _send_command 2 2024-09-10T18:57:40.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.415+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff404004e40 con 0x7ff404004610 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff404004e40 con 0x7ff404004610 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 10 client.49151.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 4 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 0 2 2024-09-10T18:57:40.424 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 10 client.49151.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 1 3 2024-09-10T18:57:40.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 4 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 10 client.49151.objecter _send_command 3 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4040022f0 con 0x7ff404004110 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404004110 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4040022f0 con 0x7ff404004110 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 10 client.49151.objecter _finish_command 3 = system:0 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 4 2024-09-10T18:57:40.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.419+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 1 3 2024-09-10T18:57:40.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 10 client.49151.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 4 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 4 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404003fa0 osd=1 3 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 1 4 2024-09-10T18:57:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 4 2024-09-10T18:57:40.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 10 client.49151.objecter _send_command 4 2024-09-10T18:57:40.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.435+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff4040018f0 con 0x7ff404004110 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4040018f0 con 0x7ff404004110 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 10 client.49151.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 4 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 1 4 2024-09-10T18:57:40.443 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 10 client.49151.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 5 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 5 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 2 5 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 4 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 10 client.49151.objecter _send_command 5 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff404003e30 con 0x7ff40400d350 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff40400d350 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff404003e30 con 0x7ff40400d350 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:57:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 10 client.49151.objecter _finish_command 5 = system:0 2024-09-10T18:57:40.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 4 2024-09-10T18:57:40.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.439+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 2 5 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 10 client.49151.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 6 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 6 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff4040035e0 osd=2 3 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 2 6 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 4 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 10 client.49151.objecter _send_command 6 2024-09-10T18:57:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff4040030d0 con 0x7ff40400d350 2024-09-10T18:57:40.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4040030d0 con 0x7ff40400d350 2024-09-10T18:57:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 10 client.49151.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 4 2024-09-10T18:57:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 2 6 2024-09-10T18:57:40.461 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 10 client.49151.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 7 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 7 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 3 7 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 4 2024-09-10T18:57:40.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 10 client.49151.objecter _send_command 7 2024-09-10T18:57:40.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff40400d100 con 0x7ff404010e50 2024-09-10T18:57:40.463 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:40.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.455+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404010e50 2024-09-10T18:57:40.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.459+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff40400d100 con 0x7ff404010e50 2024-09-10T18:57:40.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.459+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:57:40.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.459+0000 7ff4167fc640 10 client.49151.objecter _finish_command 7 = system:0 2024-09-10T18:57:40.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.459+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 4 2024-09-10T18:57:40.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.459+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 3 7 2024-09-10T18:57:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 10 client.49151.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 8 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 8 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40400c990 osd=3 3 2024-09-10T18:57:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 3 8 2024-09-10T18:57:40.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 4 2024-09-10T18:57:40.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 10 client.49151.objecter _send_command 8 2024-09-10T18:57:40.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.475+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff404012130 con 0x7ff404010e50 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff404012130 con 0x7ff404010e50 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 10 client.49151.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 4 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 3 8 2024-09-10T18:57:40.483 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 10 client.49151.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 9 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 9 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 4 9 2024-09-10T18:57:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 4 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 10 client.49151.objecter _send_command 9 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff40401a290 con 0x7ff404016790 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404016790 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff40401a290 con 0x7ff404016790 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:57:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 10 client.49151.objecter _finish_command 9 = system:0 2024-09-10T18:57:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 4 2024-09-10T18:57:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.479+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 4 9 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 10 client.49151.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 10 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 10 2024-09-10T18:57:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404011bb0 osd=4 3 2024-09-10T18:57:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 4 10 2024-09-10T18:57:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 4 2024-09-10T18:57:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 10 client.49151.objecter _send_command 10 2024-09-10T18:57:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.499+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff40401a290 con 0x7ff404016790 2024-09-10T18:57:40.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff40401a290 con 0x7ff404016790 2024-09-10T18:57:40.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff4167fc640 10 client.49151.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 4 2024-09-10T18:57:40.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 4 10 2024-09-10T18:57:40.510 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:57:40.510 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:57:40.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 10 client.49151.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 11 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 11 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 5 11 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 4 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 10 client.49151.objecter _send_command 11 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.503+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff40401e610 con 0x7ff40401a950 2024-09-10T18:57:40.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff40401a950 2024-09-10T18:57:40.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff40401e610 con 0x7ff40401a950 2024-09-10T18:57:40.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:57:40.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 10 client.49151.objecter _finish_command 11 = system:0 2024-09-10T18:57:40.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 4 2024-09-10T18:57:40.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.507+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 5 11 2024-09-10T18:57:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 10 client.49151.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 12 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 12 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40401a290 osd=5 3 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 5 12 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 4 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 10 client.49151.objecter _send_command 12 2024-09-10T18:57:40.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.527+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff40401feb0 con 0x7ff40401a950 2024-09-10T18:57:40.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff40401feb0 con 0x7ff40401a950 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 10 client.49151.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 4 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 5 12 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:57:40.536 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 10 client.49151.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 13 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 13 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 6 13 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 4 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 10 client.49151.objecter _send_command 13 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff404020a60 con 0x7ff404020530 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.531+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404020530 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.535+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff404020a60 con 0x7ff404020530 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.535+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.535+0000 7ff4167fc640 10 client.49151.objecter _finish_command 13 = system:0 2024-09-10T18:57:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.535+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 4 2024-09-10T18:57:40.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.535+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 6 13 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 10 client.49151.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 14 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 14 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff40401feb0 osd=6 3 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 6 14 2024-09-10T18:57:40.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 4 2024-09-10T18:57:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 10 client.49151.objecter _send_command 14 2024-09-10T18:57:40.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.559+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff404020a60 con 0x7ff404020530 2024-09-10T18:57:40.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff404020a60 con 0x7ff404020530 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff4167fc640 10 client.49151.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 4 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 6 14 2024-09-10T18:57:40.569 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 10 client.49151.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 15 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 15 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 7 15 2024-09-10T18:57:40.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 4 2024-09-10T18:57:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 10 client.49151.objecter _send_command 15 2024-09-10T18:57:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.563+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff404026ee0 con 0x7ff404020c50 2024-09-10T18:57:40.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 10 client.49151.objecter ms_handle_connect 0x7ff404020c50 2024-09-10T18:57:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff404026ee0 con 0x7ff404020c50 2024-09-10T18:57:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:57:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 10 client.49151.objecter _finish_command 15 = system:0 2024-09-10T18:57:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 4 2024-09-10T18:57:40.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.567+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 7 15 2024-09-10T18:57:40.597 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 10 client.49151.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign -1 16 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 20 client.49151.objecter _get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_remove -1 16 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 20 client.49151.objecter get_session s=0x7ff404020a60 osd=7 3 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 15 client.49151.objecter _session_command_op_assign 7 16 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 4 2024-09-10T18:57:40.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 10 client.49151.objecter _send_command 16 2024-09-10T18:57:40.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff404026ee0 con 0x7ff404020c50 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff4167fc640 1 -- 172.21.15.83:0/1962598025 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff404026ee0 con 0x7ff404020c50 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff4167fc640 10 client.49151.objecter ms_dispatch 0x7ff420000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff4167fc640 10 client.49151.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff4167fc640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 4 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.595+0000 7ff4167fc640 15 client.49151.objecter _session_command_op_remove 7 16 2024-09-10T18:57:40.602 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:57:40.603 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:57:40.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 librados: watch_flush enter 2024-09-10T18:57:40.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 librados: watch_flush exit 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.0 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6801/2535527483 conn(0x7ff404004610 legacy=0x7ff404002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404005290 osd=0 1 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.1 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6805/2309624637 conn(0x7ff404004110 legacy=0x7ff40400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404003fa0 osd=1 1 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.2 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6809/386094220 conn(0x7ff40400d350 legacy=0x7ff40400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff4040035e0 osd=2 1 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.3 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6813/1279688021 conn(0x7ff404010e50 legacy=0x7ff404011260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40400c990 osd=3 1 2024-09-10T18:57:40.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.4 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.165:6800/3225515265 conn(0x7ff404016790 legacy=0x7ff404018b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404011bb0 osd=4 1 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.5 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.165:6804/571173777 conn(0x7ff40401a950 legacy=0x7ff40401cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401a290 osd=5 1 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.6 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.165:6808/1317922936 conn(0x7ff404020530 legacy=0x7ff40401e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff40401feb0 osd=6 1 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter close_session for osd.7 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff404020c50 legacy=0x7ff404025630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter put_session s=0x7ff404020a60 osd=7 1 2024-09-10T18:57:40.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 20 client.49151.objecter shutdown clearing up homeless session... 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 10 client.49151.objecter successfully canceled tick 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff40c071530 legacy=0x7ff40c0739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> v1:172.21.15.83:6790/0 conn(0x7ff420072720 legacy=0x7ff42011dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff4262e9640 1 -- 172.21.15.83:0/1962598025 reap_dead start 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 shutdown_connections 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 >> 172.21.15.83:0/1962598025 conn(0x7ff4200fa370 msgr2=0x7ff420111a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 shutdown_connections 2024-09-10T18:57:40.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 -- 172.21.15.83:0/1962598025 wait complete. 2024-09-10T18:57:40.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.599+0000 7ff427d4c640 1 librados: shutdown 2024-09-10T18:57:40.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:40.627+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f15b004bf60 con 0x7f15b00fe1d0 2024-09-10T18:57:40.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:40.714 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.204620+0000 osd.5 (osd.5) 181 : cluster [DBG] 64.f scrub starts 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.210766+0000 osd.5 (osd.5) 182 : cluster [DBG] 64.f scrub ok 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.572802+0000 osd.0 (osd.0) 192 : cluster [DBG] 12.19 scrub starts 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.579823+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.19 scrub ok 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.589412+0000 osd.1 (osd.1) 193 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.593095+0000 osd.1 (osd.1) 194 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.804364+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:39.807934+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:40.012057+0000 mgr.y (mgr.24526) 344 : cluster [DBG] pgmap v439: 262 pgs: 40 unknown, 222 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.012816+0000 mon.c (mon.2) 514 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.015162+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.074626+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.076960+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.077049+0000 mon.a (mon.0) 1993 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.077233+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: cluster 2024-09-10T18:57:40.077298+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.079247+0000 mon.a (mon.0) 1996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.102128+0000 mon.c (mon.2) 515 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.102466+0000 mon.a (mon.0) 1997 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.103249+0000 mon.c (mon.2) 516 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.103506+0000 mon.a (mon.0) 1998 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.103987+0000 mon.c (mon.2) 517 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.104251+0000 mon.a (mon.0) 1999 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.716 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[32381]: audit 2024-09-10T18:57:40.404274+0000 mon.c (mon.2) 518 : audit [DBG] from='client.? v1:172.21.15.83:0/1962598025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.204620+0000 osd.5 (osd.5) 181 : cluster [DBG] 64.f scrub starts 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.210766+0000 osd.5 (osd.5) 182 : cluster [DBG] 64.f scrub ok 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.572802+0000 osd.0 (osd.0) 192 : cluster [DBG] 12.19 scrub starts 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.579823+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.19 scrub ok 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.589412+0000 osd.1 (osd.1) 193 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.593095+0000 osd.1 (osd.1) 194 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.804364+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:39.807934+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:57:40.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:40.012057+0000 mgr.y (mgr.24526) 344 : cluster [DBG] pgmap v439: 262 pgs: 40 unknown, 222 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.012816+0000 mon.c (mon.2) 514 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.015162+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.074626+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.076960+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.077049+0000 mon.a (mon.0) 1993 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.077233+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: cluster 2024-09-10T18:57:40.077298+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.079247+0000 mon.a (mon.0) 1996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.102128+0000 mon.c (mon.2) 515 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.102466+0000 mon.a (mon.0) 1997 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.103249+0000 mon.c (mon.2) 516 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.103506+0000 mon.a (mon.0) 1998 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.103987+0000 mon.c (mon.2) 517 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.104251+0000 mon.a (mon.0) 1999 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:40 smithi083 bash[18629]: audit 2024-09-10T18:57:40.404274+0000 mon.c (mon.2) 518 : audit [DBG] from='client.? v1:172.21.15.83:0/1962598025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:40.732 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:40.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:40.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.204620+0000 osd.5 (osd.5) 181 : cluster [DBG] 64.f scrub starts 2024-09-10T18:57:40.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.210766+0000 osd.5 (osd.5) 182 : cluster [DBG] 64.f scrub ok 2024-09-10T18:57:40.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.572802+0000 osd.0 (osd.0) 192 : cluster [DBG] 12.19 scrub starts 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.579823+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.19 scrub ok 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.589412+0000 osd.1 (osd.1) 193 : cluster [DBG] 3.0 scrub starts 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.593095+0000 osd.1 (osd.1) 194 : cluster [DBG] 3.0 scrub ok 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.804364+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:39.807934+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:40.012057+0000 mgr.y (mgr.24526) 344 : cluster [DBG] pgmap v439: 262 pgs: 40 unknown, 222 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.012816+0000 mon.c (mon.2) 514 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.015162+0000 mon.a (mon.0) 1991 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.074626+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.076960+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? v1:172.21.15.83:0/3346335030' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi083-77640-39"}]': finished 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.077049+0000 mon.a (mon.0) 1993 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:40.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.077233+0000 mon.a (mon.0) 1994 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: cluster 2024-09-10T18:57:40.077298+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.079247+0000 mon.a (mon.0) 1996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.102128+0000 mon.c (mon.2) 515 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.102466+0000 mon.a (mon.0) 1997 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.103249+0000 mon.c (mon.2) 516 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.103506+0000 mon.a (mon.0) 1998 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.103987+0000 mon.c (mon.2) 517 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.104251+0000 mon.a (mon.0) 1999 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:40.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:40 smithi165 bash[22591]: audit 2024-09-10T18:57:40.404274+0000 mon.c (mon.2) 518 : audit [DBG] from='client.? v1:172.21.15.83:0/1962598025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:40.869 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:40.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:40.973 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:41.007 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:41.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.075+0000 7f15a6ffd640 1 -- 172.21.15.83:0/3118551563 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v305) v1 ==== 101+0+0 (unknown 669787839 0 0) 0x7f1598056630 con 0x7f15b00fe1d0 2024-09-10T18:57:41.082 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T18:57:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 10 librados: watch_flush enter 2024-09-10T18:57:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 10 librados: watch_flush exit 2024-09-10T18:57:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 20 client.25080.objecter shutdown clearing up homeless session... 2024-09-10T18:57:41.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 10 client.25080.objecter successfully canceled tick 2024-09-10T18:57:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 >> v1:172.21.15.83:6800/3071793735 conn(0x7f159c071610 legacy=0x7f159c073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.079+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 >> v1:172.21.15.83:6789/0 conn(0x7f15b00fe1d0 legacy=0x7f15b0104920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:41.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.083+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 shutdown_connections 2024-09-10T18:57:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.083+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 >> 172.21.15.83:0/3118551563 conn(0x7f15b0075290 msgr2=0x7f15b00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.083+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 shutdown_connections 2024-09-10T18:57:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.083+0000 7f15b49d9640 1 -- 172.21.15.83:0/3118551563 wait complete. 2024-09-10T18:57:41.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:41.083+0000 7f15b49d9640 1 librados: shutdown 2024-09-10T18:57:41.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:41.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:41.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:41.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:41.405 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:41.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:41.603 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:41.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:41.742 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:41.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:41.859 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:40.634322+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.066630+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.066689+0000 mon.a (mon.0) 2002 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.077706+0000 mon.a (mon.0) 2003 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]': finished 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.077993+0000 mon.a (mon.0) 2004 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.078260+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.078348+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.080477+0000 mon.c (mon.2) 519 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.081107+0000 mon.a (mon.0) 2007 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.084529+0000 mon.c (mon.2) 520 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.860 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.085191+0000 mon.a (mon.0) 2008 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.199900+0000 osd.3 (osd.3) 208 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.203082+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.300922+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[18629]: audit 2024-09-10T18:57:41.305010+0000 mon.a (mon.0) 2009 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:40.634322+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.066630+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.066689+0000 mon.a (mon.0) 2002 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.077706+0000 mon.a (mon.0) 2003 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]': finished 2024-09-10T18:57:41.861 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.077993+0000 mon.a (mon.0) 2004 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.078260+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.078348+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.080477+0000 mon.c (mon.2) 519 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.081107+0000 mon.a (mon.0) 2007 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.084529+0000 mon.c (mon.2) 520 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.085191+0000 mon.a (mon.0) 2008 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.199900+0000 osd.3 (osd.3) 208 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.203082+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.300922+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.862 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:41 smithi083 bash[32381]: audit 2024-09-10T18:57:41.305010+0000 mon.a (mon.0) 2009 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:40.634322+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.066630+0000 mon.a (mon.0) 2001 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.066689+0000 mon.a (mon.0) 2002 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.077706+0000 mon.a (mon.0) 2003 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-33", "mode": "writeback"}]': finished 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.077993+0000 mon.a (mon.0) 2004 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi083-77640-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.078260+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.83:0/3118551563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.078348+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.080477+0000 mon.c (mon.2) 519 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.081107+0000 mon.a (mon.0) 2007 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:41.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.084529+0000 mon.c (mon.2) 520 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.085191+0000 mon.a (mon.0) 2008 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:41.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.199900+0000 osd.3 (osd.3) 208 : cluster [DBG] 4.14 scrub starts 2024-09-10T18:57:41.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.203082+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.14 scrub ok 2024-09-10T18:57:41.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.300922+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:41 smithi165 bash[22591]: audit 2024-09-10T18:57:41.305010+0000 mon.a (mon.0) 2009 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:41.879 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:41.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:41.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.018 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:42.122 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.145 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:42.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:42.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:42.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.415 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:42.518 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:42.546 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:42.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:42.681 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:42.694 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:42.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:42.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:42.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:42.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:42] "GET /metrics HTTP/1.1" 200 233523 "" "Prometheus/2.43.0" 2024-09-10T18:57:43.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.766939+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.1 deep-scrub starts 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.770507+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.1 deep-scrub ok 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.883384+0000 osd.2 (osd.2) 127 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:41.887489+0000 osd.2 (osd.2) 128 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.012685+0000 mgr.y (mgr.24526) 345 : cluster [DBG] pgmap v442: 254 pgs: 2 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.524895+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.081 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.526578+0000 mon.a (mon.0) 2010 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.526841+0000 mon.a (mon.0) 2011 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.527001+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.527588+0000 mon.c (mon.2) 521 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.529435+0000 mon.a (mon.0) 2013 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: audit 2024-09-10T18:57:42.529809+0000 mon.a (mon.0) 2014 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.585991+0000 mon.a (mon.0) 2015 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.644788+0000 osd.1 (osd.1) 195 : cluster [DBG] 64.16 scrub starts 2024-09-10T18:57:43.082 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.649277+0000 osd.1 (osd.1) 196 : cluster [DBG] 64.16 scrub ok 2024-09-10T18:57:43.098 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:43.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.766939+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.1 deep-scrub starts 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.770507+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.1 deep-scrub ok 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.883384+0000 osd.2 (osd.2) 127 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:41.887489+0000 osd.2 (osd.2) 128 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.012685+0000 mgr.y (mgr.24526) 345 : cluster [DBG] pgmap v442: 254 pgs: 2 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.524895+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.526578+0000 mon.a (mon.0) 2010 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.526841+0000 mon.a (mon.0) 2011 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.527001+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.527588+0000 mon.c (mon.2) 521 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.529435+0000 mon.a (mon.0) 2013 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: audit 2024-09-10T18:57:42.529809+0000 mon.a (mon.0) 2014 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.585991+0000 mon.a (mon.0) 2015 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.644788+0000 osd.1 (osd.1) 195 : cluster [DBG] 64.16 scrub starts 2024-09-10T18:57:43.219 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:43 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.649277+0000 osd.1 (osd.1) 196 : cluster [DBG] 64.16 scrub ok 2024-09-10T18:57:43.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:43.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:43.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:43.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.766939+0000 osd.4 (osd.4) 219 : cluster [DBG] 4.1 deep-scrub starts 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.770507+0000 osd.4 (osd.4) 220 : cluster [DBG] 4.1 deep-scrub ok 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.883384+0000 osd.2 (osd.2) 127 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:41.887489+0000 osd.2 (osd.2) 128 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.012685+0000 mgr.y (mgr.24526) 345 : cluster [DBG] pgmap v442: 254 pgs: 2 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:43.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.524895+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.526578+0000 mon.a (mon.0) 2010 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.526841+0000 mon.a (mon.0) 2011 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.527001+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.527588+0000 mon.c (mon.2) 521 : audit [INF] from='client.? v1:172.21.15.83:0/3492996066' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.529435+0000 mon.a (mon.0) 2013 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]: dispatch 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: audit 2024-09-10T18:57:42.529809+0000 mon.a (mon.0) 2014 : audit [INF] from='client.49136 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]: dispatch 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.585991+0000 mon.a (mon.0) 2015 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.644788+0000 osd.1 (osd.1) 195 : cluster [DBG] 64.16 scrub starts 2024-09-10T18:57:43.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:43 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.649277+0000 osd.1 (osd.1) 196 : cluster [DBG] 64.16 scrub ok 2024-09-10T18:57:43.502 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:43.512 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:43.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:57:43.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:43.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:43.784 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:43.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.920 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:43.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:44.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.058 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:44.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.197 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:44.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:57:44.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:44.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:44.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:44.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:44.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:44.606 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.599654+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:57:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.602441+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.895889+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:42.899667+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:43.076608+0000 mon.a (mon.0) 2016 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:43.091642+0000 mon.a (mon.0) 2017 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:43.091789+0000 mon.a (mon.0) 2018 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:43.091859+0000 mon.a (mon.0) 2019 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:43.091923+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:44.014329+0000 mon.c (mon.2) 522 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:44.017130+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:44.057633+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: cluster 2024-09-10T18:57:44.057793+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:57:44.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:44 smithi165 bash[22591]: audit 2024-09-10T18:57:44.065050+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:44.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:44.745 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.599654+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.602441+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.895889+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:42.899667+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:43.076608+0000 mon.a (mon.0) 2016 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:43.091642+0000 mon.a (mon.0) 2017 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:43.091789+0000 mon.a (mon.0) 2018 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:43.091859+0000 mon.a (mon.0) 2019 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:43.091923+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:44.014329+0000 mon.c (mon.2) 522 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:44.017130+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:44.057633+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: cluster 2024-09-10T18:57:44.057793+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[18629]: audit 2024-09-10T18:57:44.065050+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.599654+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.11 scrub starts 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.602441+0000 osd.6 (osd.6) 148 : cluster [DBG] 2.11 scrub ok 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.895889+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:42.899667+0000 osd.2 (osd.2) 130 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:43.076608+0000 mon.a (mon.0) 2016 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:43.091642+0000 mon.a (mon.0) 2017 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi083-77640-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:43.091789+0000 mon.a (mon.0) 2018 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-33"}]': finished 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:43.091859+0000 mon.a (mon.0) 2019 : audit [INF] from='client.49136 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi083-77648-50"}]': finished 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:43.091923+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:44.014329+0000 mon.c (mon.2) 522 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:44.017130+0000 mon.a (mon.0) 2021 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:44.057633+0000 mon.a (mon.0) 2022 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: cluster 2024-09-10T18:57:44.057793+0000 mon.a (mon.0) 2023 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-09-10T18:57:44.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:44 smithi083 bash[32381]: audit 2024-09-10T18:57:44.065050+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:44.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:44.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:44.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:45.018 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:45.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:45.150 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:45.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:45.288 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:45.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:45.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:45.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:45.559 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:45.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:45.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:43.912008+0000 osd.2 (osd.2) 131 : cluster [DBG] 12.a scrub starts 2024-09-10T18:57:45.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:43.914725+0000 osd.2 (osd.2) 132 : cluster [DBG] 12.a scrub ok 2024-09-10T18:57:45.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:44.013624+0000 mgr.y (mgr.24526) 346 : cluster [DBG] pgmap v445: 261 pgs: 8 unknown, 1 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:45.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:44.566498+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:44.569062+0000 mon.c (mon.2) 523 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:44.609293+0000 osd.1 (osd.1) 197 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:44.612039+0000 osd.1 (osd.1) 198 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:45.058609+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:45.059202+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: cluster 2024-09-10T18:57:45.059798+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:45.061814+0000 mon.c (mon.2) 524 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:45.064792+0000 mon.a (mon.0) 2028 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[32381]: audit 2024-09-10T18:57:45.067572+0000 mon.a (mon.0) 2029 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:43.912008+0000 osd.2 (osd.2) 131 : cluster [DBG] 12.a scrub starts 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:43.914725+0000 osd.2 (osd.2) 132 : cluster [DBG] 12.a scrub ok 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:44.013624+0000 mgr.y (mgr.24526) 346 : cluster [DBG] pgmap v445: 261 pgs: 8 unknown, 1 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:44.566498+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:44.569062+0000 mon.c (mon.2) 523 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:44.609293+0000 osd.1 (osd.1) 197 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:44.612039+0000 osd.1 (osd.1) 198 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:45.058609+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:45.059202+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: cluster 2024-09-10T18:57:45.059798+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:45.061814+0000 mon.c (mon.2) 524 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:45.064792+0000 mon.a (mon.0) 2028 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.677 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:45 smithi083 bash[18629]: audit 2024-09-10T18:57:45.067572+0000 mon.a (mon.0) 2029 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.694 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:45.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:43.912008+0000 osd.2 (osd.2) 131 : cluster [DBG] 12.a scrub starts 2024-09-10T18:57:45.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:43.914725+0000 osd.2 (osd.2) 132 : cluster [DBG] 12.a scrub ok 2024-09-10T18:57:45.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:44.013624+0000 mgr.y (mgr.24526) 346 : cluster [DBG] pgmap v445: 261 pgs: 8 unknown, 1 peering, 252 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:57:45.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:44.566498+0000 mon.a (mon.0) 2025 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:44.569062+0000 mon.c (mon.2) 523 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:44.609293+0000 osd.1 (osd.1) 197 : cluster [DBG] 6.4 scrub starts 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:44.612039+0000 osd.1 (osd.1) 198 : cluster [DBG] 6.4 scrub ok 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:45.058609+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:45.059202+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? v1:172.21.15.83:0/3398916501' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi083-77648-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: cluster 2024-09-10T18:57:45.059798+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:45.061814+0000 mon.c (mon.2) 524 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:45.064792+0000 mon.a (mon.0) 2028 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:45.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:45 smithi165 bash[22591]: audit 2024-09-10T18:57:45.067572+0000 mon.a (mon.0) 2029 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:45.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:45.828 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:45.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:45.968 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:45.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:46.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:46.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:46.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:46.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:46.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:46.377 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:46.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:46.517 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:46.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:46.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:46.784 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 -- 172.21.15.83:0/1764170949 >> v1:172.21.15.83:6790/0 conn(0x7fa64c1075a0 legacy=0x7fa64c109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 -- 172.21.15.83:0/1764170949 shutdown_connections 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 -- 172.21.15.83:0/1764170949 >> 172.21.15.83:0/1764170949 conn(0x7fa64c0fa370 msgr2=0x7fa64c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 -- 172.21.15.83:0/1764170949 shutdown_connections 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 -- 172.21.15.83:0/1764170949 wait complete. 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 librados: starting msgr at 2024-09-10T18:57:46.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 librados: starting objecter 2024-09-10T18:57:46.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.847+0000 7fa651b08640 1 Processor -- start 2024-09-10T18:57:46.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- start start 2024-09-10T18:57:46.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 librados: setting wanted keys 2024-09-10T18:57:46.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 librados: calling monclient init 2024-09-10T18:57:46.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa64c11e240 con 0x7fa64c10b130 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa64c11f400 con 0x7fa64c1075a0 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa64c120600 con 0x7fa64c104810 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa64b7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa64c104810 0x7fa64c1007b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:53854/0 (socket says 172.21.15.83:53854) 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa64bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa64c10b130 0x7fa64c11c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36084/0 (socket says 172.21.15.83:36084) 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa64b7fe640 1 -- 172.21.15.83:0/2130696131 learned_addr learned my addr 172.21.15.83:0/2130696131 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter ms_handle_connect 0x7fa64c10b130 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter ms_handle_connect 0x7fa64c104810 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter ms_handle_connect 0x7fa64c1075a0 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:46.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709574046 0 0) 0x7fa64c11e240 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6340034e0 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283103883 0 0) 0x7fa64c120600 con 0x7fa64c104810 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c11e240 con 0x7fa64c104810 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226666303 0 0) 0x7fa6340034e0 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c120600 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa644001c40 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533776803 0 0) 0x7fa64c11f400 con 0x7fa64c1075a0 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6340034e0 con 0x7fa64c1075a0 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1083230057 0 0) 0x7fa64c11e240 con 0x7fa64c104810 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c11f400 con 0x7fa64c104810 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa6400019d0 con 0x7fa64c104810 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3238621000 0 0) 0x7fa64c120600 con 0x7fa64c10b130 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 >> v1:172.21.15.83:6790/0 conn(0x7fa64c104810 legacy=0x7fa64c1007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:46.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 >> v1:172.21.15.165:6789/0 conn(0x7fa64c1075a0 legacy=0x7fa64c100ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c121800 con 0x7fa64c10b130 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c1207f0 con 0x7fa64c10b130 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 10 client.25098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c120c40 con 0x7fa64c10b130 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa651b08640 1 librados: init done 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa6440022c0 con 0x7fa64c10b130 2024-09-10T18:57:46.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.851+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa644005d90 con 0x7fa64c10b130 2024-09-10T18:57:46.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa638005010 con 0x7fa64c10b130 2024-09-10T18:57:46.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa644001e60 con 0x7fa64c10b130 2024-09-10T18:57:46.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(310..310 src has 69..310) v4 ==== 8449+0+0 (unknown 582421951 0 0) 0x7fa64408aa70 con 0x7fa64c10b130 2024-09-10T18:57:46.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 10 client.25098.objecter ms_dispatch 0x7fa64c000b90 osd_map(310..310 src has 69..310) v4 2024-09-10T18:57:46.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 3 client.25098.objecter handle_osd_map got epochs [310,310] > 0 2024-09-10T18:57:46.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 3 client.25098.objecter handle_osd_map decoding full epoch 310 2024-09-10T18:57:46.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 20 client.25098.objecter dump_active .. 0 homeless 2024-09-10T18:57:46.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.855+0000 7fa633fff640 10 client.25098.objecter ms_handle_connect 0x7fa634071770 2024-09-10T18:57:46.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:46.863+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa644056e90 con 0x7fa64c10b130 2024-09-10T18:57:46.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:46.912 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:46.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:47.015 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:47.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:47.085 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:57:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:57:47.085 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T18:57:47.085 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:57:47.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.115+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa638005840 con 0x7fa64c10b130 2024-09-10T18:57:47.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.115+0000 7fa633fff640 1 -- 172.21.15.83:0/2130696131 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2497 (unknown 3226119948 0 2331141763) 0x7fa644056830 con 0x7fa64c10b130 2024-09-10T18:57:47.121 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:57:47.121 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1001,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":311,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994114,"num_in_osds":8,"osd_in_since":1725994087,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":251},{"state_name":"creating+peering","count":32},{"state_name":"clean+premerge+peered","count":2}],"num_pgs":285,"num_pools":11,"num_objects":333,"data_bytes":4793747,"bytes_used":2561220608,"bytes_avail":765399064576,"bytes_total":767960285184,"inactive_pgs_ratio":0.11929824203252792,"read_bytes_sec":1535,"write_bytes_sec":511,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-10T18:56:01.193087+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":13,"start_stamp":"2024-09-10T18:49:25.207072+0000","gid":14493,"addr":"172.21.15.83:0/703687015","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)","ceph_version_short":"17.2.7-1606-g5be57395","container_hostname":"smithi083","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi083","id":"foo.a","kernel_description":"#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024","kernel_version":"5.4.0-195-generic","mem_swap_kb":"2097148","mem_total_kb":"32754944","num_handles":"1","os":"Linux","pid":"6","realm_id":"","realm_name":"","zone_id":"ef939013-895d-4fcb-a130-e9a8938d4e31","zone_name":"default","zonegroup_id":"348ee464-2af6-469c-a2f0-1b47a5a7c71b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T18:57:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 10 librados: watch_flush enter 2024-09-10T18:57:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 10 librados: watch_flush exit 2024-09-10T18:57:47.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 20 client.25098.objecter shutdown clearing up homeless session... 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 10 client.25098.objecter successfully canceled tick 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa634071770 legacy=0x7fa634073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 >> v1:172.21.15.83:6789/0 conn(0x7fa64c10b130 legacy=0x7fa64c11c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 shutdown_connections 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 >> 172.21.15.83:0/2130696131 conn(0x7fa64c0fa370 msgr2=0x7fa64c10b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 shutdown_connections 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 -- 172.21.15.83:0/2130696131 wait complete. 2024-09-10T18:57:47.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:47.119+0000 7fa651b08640 1 librados: shutdown 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: cluster 2024-09-10T18:57:46.016453+0000 mgr.y (mgr.24526) 347 : cluster [DBG] pgmap v448: 285 pgs: 32 creating+peering, 2 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.061243+0000 mon.a (mon.0) 2030 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.061470+0000 mon.a (mon.0) 2031 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: cluster 2024-09-10T18:57:46.061970+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.066141+0000 mon.c (mon.2) 525 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.066252+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.070167+0000 mon.a (mon.0) 2033 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.071317+0000 mon.a (mon.0) 2034 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.149 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.104951+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.106456+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.150 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[18629]: audit 2024-09-10T18:57:46.106780+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:47.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:47.150 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: cluster 2024-09-10T18:57:46.016453+0000 mgr.y (mgr.24526) 347 : cluster [DBG] pgmap v448: 285 pgs: 32 creating+peering, 2 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.061243+0000 mon.a (mon.0) 2030 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.061470+0000 mon.a (mon.0) 2031 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: cluster 2024-09-10T18:57:46.061970+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.066141+0000 mon.c (mon.2) 525 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.066252+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.070167+0000 mon.a (mon.0) 2033 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.071317+0000 mon.a (mon.0) 2034 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.104951+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.106456+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:47 smithi083 bash[32381]: audit 2024-09-10T18:57:46.106780+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: cluster 2024-09-10T18:57:46.016453+0000 mgr.y (mgr.24526) 347 : cluster [DBG] pgmap v448: 285 pgs: 32 creating+peering, 2 clean+premerge+peered, 251 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.061243+0000 mon.a (mon.0) 2030 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.061470+0000 mon.a (mon.0) 2031 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: cluster 2024-09-10T18:57:46.061970+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.066141+0000 mon.c (mon.2) 525 : audit [INF] from='client.? v1:172.21.15.83:0/3965626681' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.066252+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.070167+0000 mon.a (mon.0) 2033 : audit [INF] from='client.49142 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.071317+0000 mon.a (mon.0) 2034 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.104951+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.165 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.106456+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:47.166 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[22591]: audit 2024-09-10T18:57:46.106780+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:47.184 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:47.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:47.318 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:47.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:47.455 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:47.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:47.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:47.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:47.728 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:47.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T18:57:47.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:47.845 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:47.860 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:47.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:47.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:47.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:48.052 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T18:57:48.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:57:48.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:48.134 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:48.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:48.272 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:48.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:48.407 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:48.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: cluster 2024-09-10T18:57:46.897538+0000 osd.2 (osd.2) 133 : cluster [DBG] 16.11 deep-scrub starts 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: cluster 2024-09-10T18:57:46.904894+0000 osd.2 (osd.2) 134 : cluster [DBG] 16.11 deep-scrub ok 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.066509+0000 mon.a (mon.0) 2038 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.066672+0000 mon.a (mon.0) 2039 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.066894+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.067004+0000 mon.b (mon.1) 249 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: cluster 2024-09-10T18:57:47.067006+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.071876+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:48.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.072688+0000 mon.a (mon.0) 2043 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.097403+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.098846+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.099223+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:47.120660+0000 mon.a (mon.0) 2047 : audit [DBG] from='client.? v1:172.21.15.83:0/2130696131' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: cluster 2024-09-10T18:57:47.181437+0000 osd.3 (osd.3) 210 : cluster [DBG] 12.8 scrub starts 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: cluster 2024-09-10T18:57:47.189435+0000 osd.3 (osd.3) 211 : cluster [DBG] 12.8 scrub ok 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:48.018041+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[32381]: audit 2024-09-10T18:57:48.020418+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: cluster 2024-09-10T18:57:46.897538+0000 osd.2 (osd.2) 133 : cluster [DBG] 16.11 deep-scrub starts 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: cluster 2024-09-10T18:57:46.904894+0000 osd.2 (osd.2) 134 : cluster [DBG] 16.11 deep-scrub ok 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.066509+0000 mon.a (mon.0) 2038 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.066672+0000 mon.a (mon.0) 2039 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.066894+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.067004+0000 mon.b (mon.1) 249 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: cluster 2024-09-10T18:57:47.067006+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.071876+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.072688+0000 mon.a (mon.0) 2043 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.097403+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.098846+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.099223+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:47.120660+0000 mon.a (mon.0) 2047 : audit [DBG] from='client.? v1:172.21.15.83:0/2130696131' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:48.516 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: cluster 2024-09-10T18:57:47.181437+0000 osd.3 (osd.3) 210 : cluster [DBG] 12.8 scrub starts 2024-09-10T18:57:48.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: cluster 2024-09-10T18:57:47.189435+0000 osd.3 (osd.3) 211 : cluster [DBG] 12.8 scrub ok 2024-09-10T18:57:48.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:48.018041+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.517 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:48 smithi165 bash[22591]: audit 2024-09-10T18:57:48.020418+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:48.531 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:48.540 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:48.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:48.680 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:48.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: cluster 2024-09-10T18:57:46.897538+0000 osd.2 (osd.2) 133 : cluster [DBG] 16.11 deep-scrub starts 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: cluster 2024-09-10T18:57:46.904894+0000 osd.2 (osd.2) 134 : cluster [DBG] 16.11 deep-scrub ok 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.066509+0000 mon.a (mon.0) 2038 : audit [INF] from='client.49142 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi083-77640-40"}]': finished 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.066672+0000 mon.a (mon.0) 2039 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.066894+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.067004+0000 mon.b (mon.1) 249 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: cluster 2024-09-10T18:57:47.067006+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.071876+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.072688+0000 mon.a (mon.0) 2043 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.097403+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.098846+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.099223+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:47.120660+0000 mon.a (mon.0) 2047 : audit [DBG] from='client.? v1:172.21.15.83:0/2130696131' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: cluster 2024-09-10T18:57:47.181437+0000 osd.3 (osd.3) 210 : cluster [DBG] 12.8 scrub starts 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: cluster 2024-09-10T18:57:47.189435+0000 osd.3 (osd.3) 211 : cluster [DBG] 12.8 scrub ok 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:48.018041+0000 mon.c (mon.2) 526 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:48 smithi083 bash[18629]: audit 2024-09-10T18:57:48.020418+0000 mon.a (mon.0) 2048 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-10T18:57:48.818 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:48.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:48.956 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:48.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:49.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:49.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:49.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:49.225 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:49.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:49.359 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:49.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:49.495 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:49.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:47.813539+0000 mgr.y (mgr.24526) 348 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:47.834496+0000 osd.4 (osd.4) 221 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:47.839106+0000 osd.4 (osd.4) 222 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.017403+0000 mgr.y (mgr.24526) 349 : cluster [DBG] pgmap v451: 252 pgs: 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.143927+0000 osd.5 (osd.5) 183 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.147111+0000 osd.5 (osd.5) 184 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.506295+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.613 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.508459+0000 mon.a (mon.0) 2049 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.508568+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.508641+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.508732+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.509649+0000 mon.a (mon.0) 2053 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.510303+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:48.516655+0000 mon.a (mon.0) 2055 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.728848+0000 osd.1 (osd.1) 199 : cluster [DBG] 64.b scrub starts 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:48.733711+0000 osd.1 (osd.1) 200 : cluster [DBG] 64.b scrub ok 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.063089+0000 mon.a (mon.0) 2056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:49.614 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:47.813539+0000 mgr.y (mgr.24526) 348 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:47.834496+0000 osd.4 (osd.4) 221 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:47.839106+0000 osd.4 (osd.4) 222 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.017403+0000 mgr.y (mgr.24526) 349 : cluster [DBG] pgmap v451: 252 pgs: 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.143927+0000 osd.5 (osd.5) 183 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.147111+0000 osd.5 (osd.5) 184 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.506295+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.508459+0000 mon.a (mon.0) 2049 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.508568+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.508641+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.508732+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.509649+0000 mon.a (mon.0) 2053 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.510303+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:49.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:48.516655+0000 mon.a (mon.0) 2055 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.728848+0000 osd.1 (osd.1) 199 : cluster [DBG] 64.b scrub starts 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:48.733711+0000 osd.1 (osd.1) 200 : cluster [DBG] 64.b scrub ok 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.063089+0000 mon.a (mon.0) 2056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:49.069827+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: audit 2024-09-10T18:57:49.504691+0000 mon.a (mon.0) 2058 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]': finished 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.504969+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:49.069827+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: audit 2024-09-10T18:57:49.504691+0000 mon.a (mon.0) 2058 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]': finished 2024-09-10T18:57:49.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:49 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.504969+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:57:49.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:49.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:49.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:47.813539+0000 mgr.y (mgr.24526) 348 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:49.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:47.834496+0000 osd.4 (osd.4) 221 : cluster [DBG] 3.14 scrub starts 2024-09-10T18:57:49.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:47.839106+0000 osd.4 (osd.4) 222 : cluster [DBG] 3.14 scrub ok 2024-09-10T18:57:49.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.017403+0000 mgr.y (mgr.24526) 349 : cluster [DBG] pgmap v451: 252 pgs: 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:57:49.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.143927+0000 osd.5 (osd.5) 183 : cluster [DBG] 4.12 scrub starts 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.147111+0000 osd.5 (osd.5) 184 : cluster [DBG] 4.12 scrub ok 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.506295+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.508459+0000 mon.a (mon.0) 2049 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.508568+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi083-77640-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.508641+0000 mon.a (mon.0) 2051 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.508732+0000 mon.a (mon.0) 2052 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.509649+0000 mon.a (mon.0) 2053 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.510303+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:48.516655+0000 mon.a (mon.0) 2055 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]: dispatch 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.728848+0000 osd.1 (osd.1) 199 : cluster [DBG] 64.b scrub starts 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:48.733711+0000 osd.1 (osd.1) 200 : cluster [DBG] 64.b scrub ok 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.063089+0000 mon.a (mon.0) 2056 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:49.069827+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi083-77648-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: audit 2024-09-10T18:57:49.504691+0000 mon.a (mon.0) 2058 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-35", "mode": "writeback"}]': finished 2024-09-10T18:57:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:49 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.504969+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-09-10T18:57:49.760 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:49.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:49.892 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:49.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:49.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:50.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:50.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:50.160 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:50.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:50.291 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:50.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:50.417 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:50.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:50.523 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.382345+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.386394+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.528594+0000 osd.6 (osd.6) 149 : cluster [DBG] 64.17 scrub starts 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.532058+0000 osd.6 (osd.6) 150 : cluster [DBG] 64.17 scrub ok 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.018118+0000 mgr.y (mgr.24526) 350 : cluster [DBG] pgmap v454: 260 pgs: 8 unknown, 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: audit 2024-09-10T18:57:50.033988+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: audit 2024-09-10T18:57:50.075999+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.076582+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.171034+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.b scrub starts 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.173396+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.b scrub ok 2024-09-10T18:57:50.524 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: audit 2024-09-10T18:57:50.501809+0000 mon.c (mon.2) 528 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:50.525 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:50 smithi165 bash[22591]: audit 2024-09-10T18:57:50.503200+0000 mon.c (mon.2) 529 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:50.541 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:50.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:50.681 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:50.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:50.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.382345+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.386394+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.528594+0000 osd.6 (osd.6) 149 : cluster [DBG] 64.17 scrub starts 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.532058+0000 osd.6 (osd.6) 150 : cluster [DBG] 64.17 scrub ok 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.018118+0000 mgr.y (mgr.24526) 350 : cluster [DBG] pgmap v454: 260 pgs: 8 unknown, 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: audit 2024-09-10T18:57:50.033988+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: audit 2024-09-10T18:57:50.075999+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.076582+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.171034+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.b scrub starts 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.173396+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.b scrub ok 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: audit 2024-09-10T18:57:50.501809+0000 mon.c (mon.2) 528 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[18629]: audit 2024-09-10T18:57:50.503200+0000 mon.c (mon.2) 529 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.382345+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.3 deep-scrub starts 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.386394+0000 osd.7 (osd.7) 199 : cluster [DBG] 6.3 deep-scrub ok 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.528594+0000 osd.6 (osd.6) 149 : cluster [DBG] 64.17 scrub starts 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.532058+0000 osd.6 (osd.6) 150 : cluster [DBG] 64.17 scrub ok 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.018118+0000 mgr.y (mgr.24526) 350 : cluster [DBG] pgmap v454: 260 pgs: 8 unknown, 32 creating+peering, 1 clean+premerge+peered, 219 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: audit 2024-09-10T18:57:50.033988+0000 mon.c (mon.2) 527 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: audit 2024-09-10T18:57:50.075999+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi083-77640-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.076582+0000 mon.a (mon.0) 2061 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.171034+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.b scrub starts 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.173396+0000 osd.3 (osd.3) 213 : cluster [DBG] 6.b scrub ok 2024-09-10T18:57:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: audit 2024-09-10T18:57:50.501809+0000 mon.c (mon.2) 528 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:50 smithi083 bash[32381]: audit 2024-09-10T18:57:50.503200+0000 mon.c (mon.2) 529 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:50.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:50.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:50.953 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:50.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:51.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:51.103 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:51.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:51.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:51.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:51.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:51.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:51.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:51.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.916241+0000 osd.2 (osd.2) 135 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: cluster 2024-09-10T18:57:49.919779+0000 osd.2 (osd.2) 136 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: audit 2024-09-10T18:57:50.513035+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.788829+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.2 deep-scrub starts 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.792288+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.2 deep-scrub ok 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.077149+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: audit 2024-09-10T18:57:51.083333+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: audit 2024-09-10T18:57:51.157867+0000 mon.b (mon.1) 251 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[22591]: audit 2024-09-10T18:57:51.161960+0000 mon.a (mon.0) 2065 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.616 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:57:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:57:51.646 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:51.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:51.763 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.916241+0000 osd.2 (osd.2) 135 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: cluster 2024-09-10T18:57:49.919779+0000 osd.2 (osd.2) 136 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: audit 2024-09-10T18:57:50.513035+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.788829+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.2 deep-scrub starts 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.792288+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.2 deep-scrub ok 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.077149+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: audit 2024-09-10T18:57:51.083333+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: audit 2024-09-10T18:57:51.157867+0000 mon.b (mon.1) 251 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[18629]: audit 2024-09-10T18:57:51.161960+0000 mon.a (mon.0) 2065 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.916241+0000 osd.2 (osd.2) 135 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: cluster 2024-09-10T18:57:49.919779+0000 osd.2 (osd.2) 136 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:57:51.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: audit 2024-09-10T18:57:50.513035+0000 mon.a (mon.0) 2062 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.788829+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.2 deep-scrub starts 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.792288+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.2 deep-scrub ok 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.077149+0000 mon.a (mon.0) 2063 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: audit 2024-09-10T18:57:51.083333+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: audit 2024-09-10T18:57:51.157867+0000 mon.b (mon.1) 251 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:51 smithi083 bash[32381]: audit 2024-09-10T18:57:51.161960+0000 mon.a (mon.0) 2065 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:51.780 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:51.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:51.919 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:51.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:52.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.051 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:52.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.187 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:52.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:52.329 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:52.433 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:52.468 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:52.525 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:52.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.746453+0000 osd.1 (osd.1) 201 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:50.752275+0000 osd.1 (osd.1) 202 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.409422+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.412690+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.494579+0000 osd.6 (osd.6) 151 : cluster [DBG] 64.2 scrub starts 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.498956+0000 osd.6 (osd.6) 152 : cluster [DBG] 64.2 scrub ok 2024-09-10T18:57:52.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.779217+0000 osd.4 (osd.4) 225 : cluster [DBG] 64.11 scrub starts 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:51.784171+0000 osd.4 (osd.4) 226 : cluster [DBG] 64.11 scrub ok 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.019056+0000 mgr.y (mgr.24526) 351 : cluster [DBG] pgmap v457: 259 pgs: 8 creating+peering, 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.019671+0000 mon.c (mon.2) 530 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.021965+0000 mon.a (mon.0) 2066 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.083621+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.083853+0000 mon.a (mon.0) 2068 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.084112+0000 mon.a (mon.0) 2069 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.084276+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.084946+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.086078+0000 osd.3 (osd.3) 214 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.086692+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:52.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.092206+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.095625+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[32381]: audit 2024-09-10T18:57:52.099495+0000 mon.a (mon.0) 2073 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.746453+0000 osd.1 (osd.1) 201 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:50.752275+0000 osd.1 (osd.1) 202 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.409422+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.412690+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.494579+0000 osd.6 (osd.6) 151 : cluster [DBG] 64.2 scrub starts 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.498956+0000 osd.6 (osd.6) 152 : cluster [DBG] 64.2 scrub ok 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.779217+0000 osd.4 (osd.4) 225 : cluster [DBG] 64.11 scrub starts 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:51.784171+0000 osd.4 (osd.4) 226 : cluster [DBG] 64.11 scrub ok 2024-09-10T18:57:52.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.019056+0000 mgr.y (mgr.24526) 351 : cluster [DBG] pgmap v457: 259 pgs: 8 creating+peering, 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.019671+0000 mon.c (mon.2) 530 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.021965+0000 mon.a (mon.0) 2066 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.083621+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.083853+0000 mon.a (mon.0) 2068 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.084112+0000 mon.a (mon.0) 2069 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.084276+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.084946+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.086078+0000 osd.3 (osd.3) 214 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.086692+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.092206+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:57:52.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.095625+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18629]: audit 2024-09-10T18:57:52.099495+0000 mon.a (mon.0) 2073 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.604 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:52.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.741 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:52.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.746453+0000 osd.1 (osd.1) 201 : cluster [DBG] 3.4 scrub starts 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:50.752275+0000 osd.1 (osd.1) 202 : cluster [DBG] 3.4 scrub ok 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.409422+0000 osd.7 (osd.7) 200 : cluster [DBG] 6.5 scrub starts 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.412690+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.5 scrub ok 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.494579+0000 osd.6 (osd.6) 151 : cluster [DBG] 64.2 scrub starts 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.498956+0000 osd.6 (osd.6) 152 : cluster [DBG] 64.2 scrub ok 2024-09-10T18:57:52.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.779217+0000 osd.4 (osd.4) 225 : cluster [DBG] 64.11 scrub starts 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:51.784171+0000 osd.4 (osd.4) 226 : cluster [DBG] 64.11 scrub ok 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.019056+0000 mgr.y (mgr.24526) 351 : cluster [DBG] pgmap v457: 259 pgs: 8 creating+peering, 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.019671+0000 mon.c (mon.2) 530 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.021965+0000 mon.a (mon.0) 2066 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.083621+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.083853+0000 mon.a (mon.0) 2068 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.084112+0000 mon.a (mon.0) 2069 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.084276+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.084946+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]: dispatch 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.086078+0000 osd.3 (osd.3) 214 : cluster [DBG] 2.19 scrub starts 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.086692+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.092206+0000 osd.3 (osd.3) 215 : cluster [DBG] 2.19 scrub ok 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.095625+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:52 smithi165 bash[22591]: audit 2024-09-10T18:57:52.099495+0000 mon.a (mon.0) 2073 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:52.877 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:52.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:52.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:52.992 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:57:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:57:52] "GET /metrics HTTP/1.1" 200 233519 "" "Prometheus/2.43.0" 2024-09-10T18:57:53.012 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:53.116 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:53.145 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:53.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:53.281 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:53.382 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:53.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:53.417 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:53.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:53.555 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:53.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:53.692 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:53.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:53.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:53.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:53.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:54.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:54.070 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAiete 2024-09-10T18:57:54.070 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6210 ms) 2024-09-10T18:57:54.070 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-10T18:57:54.070 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7029 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7004 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7007 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6959 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7044 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6974 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7019 ms) 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-10T18:57:54.071 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7028 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6964 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7022 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7012 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6995 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7001 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6986 ms) 2024-09-10T18:57:54.072 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (111602 ms total) 2024-09-10T18:57:54.073 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: 2024-09-10T18:57:54.073 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] Global test environment tear-down 2024-09-10T18:57:54.073 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (208368 ms total) 2024-09-10T18:57:54.073 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ PASSED ] 41 tests. 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78160 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78160 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77698 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77698 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78141 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78141 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78040 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78040 2024-09-10T18:57:54.075 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77651 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77651 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77860 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77860 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77984 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77984 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77823 2024-09-10T18:57:54.076 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77823 2024-09-10T18:57:54.095 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:54.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:54.233 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.369429+0000 osd.7 (osd.7) 202 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.371555+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.756325+0000 osd.4 (osd.4) 227 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:52.759678+0000 osd.4 (osd.4) 228 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.084955+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.085059+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.085200+0000 mon.a (mon.0) 2076 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:54.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.085316+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.090286+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.110893+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.112930+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:53.113353+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.134226+0000 osd.3 (osd.3) 216 : cluster [DBG] 64.19 scrub starts 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.142335+0000 osd.3 (osd.3) 217 : cluster [DBG] 64.19 scrub ok 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.047518+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:54.063757+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:54.063911+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.064367+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:57:54.236 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:54 smithi165 bash[22591]: audit 2024-09-10T18:57:54.065273+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:54.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:54.371 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:54.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:54.487 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.369429+0000 osd.7 (osd.7) 202 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.371555+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.756325+0000 osd.4 (osd.4) 227 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:52.759678+0000 osd.4 (osd.4) 228 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.084955+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.085059+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.085200+0000 mon.a (mon.0) 2076 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.085316+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.090286+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.110893+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.112930+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:53.113353+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:54.488 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.134226+0000 osd.3 (osd.3) 216 : cluster [DBG] 64.19 scrub starts 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.142335+0000 osd.3 (osd.3) 217 : cluster [DBG] 64.19 scrub ok 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.047518+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:54.063757+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:54.063911+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.064367+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[18629]: audit 2024-09-10T18:57:54.065273+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.369429+0000 osd.7 (osd.7) 202 : cluster [DBG] 6.15 scrub starts 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.371555+0000 osd.7 (osd.7) 203 : cluster [DBG] 6.15 scrub ok 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.756325+0000 osd.4 (osd.4) 227 : cluster [DBG] 6.17 scrub starts 2024-09-10T18:57:54.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:52.759678+0000 osd.4 (osd.4) 228 : cluster [DBG] 6.17 scrub ok 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.084955+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? v1:172.21.15.83:0/2516189614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi083-77648-52"}]': finished 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.085059+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.085200+0000 mon.a (mon.0) 2076 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.085316+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.090286+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]: dispatch 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.110893+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.112930+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:53.113353+0000 mon.a (mon.0) 2081 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.134226+0000 osd.3 (osd.3) 216 : cluster [DBG] 64.19 scrub starts 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.142335+0000 osd.3 (osd.3) 217 : cluster [DBG] 64.19 scrub ok 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.047518+0000 mon.a (mon.0) 2082 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:54.063757+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? v1:172.21.15.83:0/3641805745' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi083-77640-41"}]': finished 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:54.063911+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:57:54.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.064367+0000 mon.a (mon.0) 2085 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-09-10T18:57:54.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:54 smithi083 bash[32381]: audit 2024-09-10T18:57:54.065273+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:54.506 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:54.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:54.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:54.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:54.784 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:54.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:54.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:54.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:55.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:55.059 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:55.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:55.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:55.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:55.335 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:55.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.470 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:55.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.603 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:55.706 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:55.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.662215+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:57:55.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.666857+0000 osd.0 (osd.0) 195 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:57:55.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.714704+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.718333+0000 osd.4 (osd.4) 230 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.791768+0000 osd.1 (osd.1) 203 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:53.795212+0000 osd.1 (osd.1) 204 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.019986+0000 mgr.y (mgr.24526) 352 : cluster [DBG] pgmap v460: 251 pgs: 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: audit 2024-09-10T18:57:54.103801+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[18629]: audit 2024-09-10T18:57:54.107790+0000 mon.a (mon.0) 2087 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.662215+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.666857+0000 osd.0 (osd.0) 195 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.714704+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.718333+0000 osd.4 (osd.4) 230 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:57:55.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.791768+0000 osd.1 (osd.1) 203 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:57:55.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:53.795212+0000 osd.1 (osd.1) 204 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:57:55.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.019986+0000 mgr.y (mgr.24526) 352 : cluster [DBG] pgmap v460: 251 pgs: 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:55.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: audit 2024-09-10T18:57:54.103801+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:55 smithi083 bash[32381]: audit 2024-09-10T18:57:54.107790+0000 mon.a (mon.0) 2087 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.739 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.662215+0000 osd.0 (osd.0) 194 : cluster [DBG] 3.6 scrub starts 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.666857+0000 osd.0 (osd.0) 195 : cluster [DBG] 3.6 scrub ok 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.714704+0000 osd.4 (osd.4) 229 : cluster [DBG] 4.1a scrub starts 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.718333+0000 osd.4 (osd.4) 230 : cluster [DBG] 4.1a scrub ok 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.791768+0000 osd.1 (osd.1) 203 : cluster [DBG] 4.4 scrub starts 2024-09-10T18:57:55.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:53.795212+0000 osd.1 (osd.1) 204 : cluster [DBG] 4.4 scrub ok 2024-09-10T18:57:55.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.019986+0000 mgr.y (mgr.24526) 352 : cluster [DBG] pgmap v460: 251 pgs: 1 peering, 250 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T18:57:55.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: audit 2024-09-10T18:57:54.103801+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:55 smithi165 bash[22591]: audit 2024-09-10T18:57:54.107790+0000 mon.a (mon.0) 2087 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:57:55.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.872 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:55.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:55.975 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.003 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:56.045 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:57:56.045 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:57:56.045 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T18:57:56.045 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T18:57:56.045 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:57:56.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:56.134 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:56.237 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:56.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- 172.21.15.83:0/991888149 >> v1:172.21.15.83:6790/0 conn(0x7f0c6c0fbeb0 legacy=0x7f0c6c109a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- 172.21.15.83:0/991888149 shutdown_connections 2024-09-10T18:57:56.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- 172.21.15.83:0/991888149 >> 172.21.15.83:0/991888149 conn(0x7f0c6c075290 msgr2=0x7f0c6c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:56.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- 172.21.15.83:0/991888149 shutdown_connections 2024-09-10T18:57:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- 172.21.15.83:0/991888149 wait complete. 2024-09-10T18:57:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 librados: starting msgr at 2024-09-10T18:57:56.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 librados: starting objecter 2024-09-10T18:57:56.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 Processor -- start 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- start start 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 librados: setting wanted keys 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 librados: calling monclient init 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0c6c10aed0 con 0x7f0c6c0fb3d0 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0c6c11f5d0 con 0x7f0c6c0fbeb0 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c745a1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0c6c120770 con 0x7f0c6c10b200 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c7233d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0c6c0fb3d0 0x7f0c6c10a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38712/0 (socket says 172.21.15.83:38712) 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c7233d640 1 -- 172.21.15.83:0/3925826915 learned_addr learned my addr 172.21.15.83:0/3925826915 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c72b3e640 1 --1- 172.21.15.83:0/3925826915 >> v1:172.21.15.83:6790/0 conn(0x7f0c6c10b200 0x7f0c6c11cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39150/0 (socket says 172.21.15.83:39150) 2024-09-10T18:57:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter ms_handle_connect 0x7f0c6c0fb3d0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter ms_handle_connect 0x7f0c6c10b200 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter ms_handle_connect 0x7f0c6c0fbeb0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208854178 0 0) 0x7f0c6c10aed0 con 0x7f0c6c0fb3d0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c580034e0 con 0x7f0c6c0fb3d0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119615479 0 0) 0x7f0c6c120770 con 0x7f0c6c10b200 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c10aed0 con 0x7f0c6c10b200 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218998268 0 0) 0x7f0c6c11f5d0 con 0x7f0c6c0fbeb0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c120770 con 0x7f0c6c0fbeb0 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351363256 0 0) 0x7f0c6c10aed0 con 0x7f0c6c10b200 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c6c11f5d0 con 0x7f0c6c10b200 2024-09-10T18:57:56.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.367+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0c64001b70 con 0x7f0c6c10b200 2024-09-10T18:57:56.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2226218569 0 0) 0x7f0c6c11f5d0 con 0x7f0c6c10b200 2024-09-10T18:57:56.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 >> v1:172.21.15.165:6789/0 conn(0x7f0c6c0fbeb0 legacy=0x7f0c6c119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 >> v1:172.21.15.83:6789/0 conn(0x7f0c6c0fb3d0 legacy=0x7f0c6c10a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c121970 con 0x7f0c6c10b200 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c120960 con 0x7f0c6c10b200 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c745a1640 10 client.49187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c120db0 con 0x7f0c6c10b200 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c745a1640 1 librados: init done 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0c640021f0 con 0x7f0c6c10b200 2024-09-10T18:57:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0c64005d20 con 0x7f0c6c10b200 2024-09-10T18:57:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.371+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f0c64001d90 con 0x7f0c6c10b200 2024-09-10T18:57:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c50005010 con 0x7f0c6c10b200 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(320..320 src has 69..320) v4 ==== 9193+0+0 (unknown 2792219906 0 0) 0x7f0c6408ace0 con 0x7f0c6c10b200 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 10 client.49187.objecter ms_dispatch 0x7f0c6c000b90 osd_map(320..320 src has 69..320) v4 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 3 client.49187.objecter handle_osd_map got epochs [320,320] > 0 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 3 client.49187.objecter handle_osd_map decoding full epoch 320 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 20 client.49187.objecter dump_active .. 0 homeless 2024-09-10T18:57:56.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.375+0000 7f0c62ffd640 10 client.49187.objecter ms_handle_connect 0x7f0c58079be0 2024-09-10T18:57:56.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.383+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0c64056e10 con 0x7f0c6c10b200 2024-09-10T18:57:56.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:56.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.527 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:56.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- 172.21.15.83:0/4131626471 >> v1:172.21.15.83:6790/0 conn(0x7fdf20101530 legacy=0x7fdf20101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- 172.21.15.83:0/4131626471 shutdown_connections 2024-09-10T18:57:56.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- 172.21.15.83:0/4131626471 >> 172.21.15.83:0/4131626471 conn(0x7fdf200f70b0 msgr2=0x7fdf200f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:56.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- 172.21.15.83:0/4131626471 shutdown_connections 2024-09-10T18:57:56.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- 172.21.15.83:0/4131626471 wait complete. 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 librados: starting msgr at 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 librados: starting objecter 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 Processor -- start 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- start start 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 librados: setting wanted keys 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 librados: calling monclient init 2024-09-10T18:57:56.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf20107b30 con 0x7fdf201042a0 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.535+0000 7fdf269ee640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf2011f690 con 0x7fdf20107e00 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf20120830 con 0x7fdf20101530 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1ffff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fdf20101530 0x7fdf20107270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39162/0 (socket says 172.21.15.83:39162) 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1ffff640 1 -- 172.21.15.83:0/1165368822 learned_addr learned my addr 172.21.15.83:0/1165368822 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter ms_handle_connect 0x7fdf20101530 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter ms_handle_connect 0x7fdf201042a0 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter ms_handle_connect 0x7fdf20107e00 2024-09-10T18:57:56.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711825460 0 0) 0x7fdf20120830 con 0x7fdf20101530 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0c0034e0 con 0x7fdf20101530 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634245239 0 0) 0x7fdf20107b30 con 0x7fdf201042a0 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf20120830 con 0x7fdf201042a0 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886447328 0 0) 0x7fdf2011f690 con 0x7fdf20107e00 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf20107b30 con 0x7fdf20107e00 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287064104 0 0) 0x7fdf20120830 con 0x7fdf201042a0 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf2011f690 con 0x7fdf201042a0 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdf080028f0 con 0x7fdf201042a0 2024-09-10T18:57:56.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644886104 0 0) 0x7fdf0c0034e0 con 0x7fdf20101530 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf20120830 con 0x7fdf20101530 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdf10002050 con 0x7fdf20101530 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2081282177 0 0) 0x7fdf20107b30 con 0x7fdf20107e00 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0c0034e0 con 0x7fdf20107e00 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdf18001b80 con 0x7fdf20107e00 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3811995397 0 0) 0x7fdf2011f690 con 0x7fdf201042a0 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6790/0 conn(0x7fdf20101530 legacy=0x7fdf20107270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.165:6789/0 conn(0x7fdf20107e00 legacy=0x7fdf2011cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf201219d0 con 0x7fdf201042a0 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2011e6e0 con 0x7fdf201042a0 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fdf08002b60 con 0x7fdf201042a0 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdf080031f0 con 0x7fdf201042a0 2024-09-10T18:57:56.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 10 client.25122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:57:56.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2011eb30 con 0x7fdf201042a0 2024-09-10T18:57:56.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 1 librados: init done 2024-09-10T18:57:56.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.539+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdf200fcc30 con 0x7fdf201042a0 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fdf08001560 con 0x7fdf201042a0 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(320..320 src has 69..320) v4 ==== 9193+0+0 (unknown 2792219906 0 0) 0x7fdf080882a0 con 0x7fdf201042a0 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 osd_map(320..320 src has 69..320) v4 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 3 client.25122.objecter handle_osd_map got epochs [320,320] > 0 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 3 client.25122.objecter handle_osd_map decoding full epoch 320 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 20 client.25122.objecter dump_active .. 0 homeless 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v320) v1 ==== 54+0+16 (unknown 4582422 0 2420000192) 0x7fdf0808a660 con 0x7fdf201042a0 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf0c071750 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 10 client.25122.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 1 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 1 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 0 1 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 4 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 10 client.25122.objecter _send_command 1 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf2004bf60 con 0x7fdf2011f820 2024-09-10T18:57:56.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf2011f820 2024-09-10T18:57:56.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf2004bf60 con 0x7fdf2011f820 2024-09-10T18:57:56.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:57:56.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 1 = system:0 2024-09-10T18:57:56.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 4 2024-09-10T18:57:56.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.543+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 0 1 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 10 client.25122.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 2 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 2 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf201098b0 osd=0 3 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 0 2 2024-09-10T18:57:56.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 4 2024-09-10T18:57:56.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 10 client.25122.objecter _send_command 2 2024-09-10T18:57:56.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf2010d8b0 con 0x7fdf2011f820 2024-09-10T18:57:56.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf2010d8b0 con 0x7fdf2011f820 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 4 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 0 2 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:57:56.570 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 10 client.25122.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 1 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 4 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 10 client.25122.objecter _send_command 3 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.563+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf2002a8d0 con 0x7fdf200f7b00 2024-09-10T18:57:56.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf200f7b00 2024-09-10T18:57:56.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf2002a8d0 con 0x7fdf200f7b00 2024-09-10T18:57:56.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:57:56.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 3 = system:0 2024-09-10T18:57:56.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 4 2024-09-10T18:57:56.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.567+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 1 3 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 10 client.25122.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 4 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 4 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf200f7480 osd=1 3 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 1 4 2024-09-10T18:57:56.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 4 2024-09-10T18:57:56.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 10 client.25122.objecter _send_command 4 2024-09-10T18:57:56.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf2002acb0 con 0x7fdf200f7b00 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf2002acb0 con 0x7fdf200f7b00 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 4 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 1 4 2024-09-10T18:57:56.589 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 10 client.25122.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 5 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 5 2024-09-10T18:57:56.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 2 5 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 4 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 10 client.25122.objecter _send_command 5 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.583+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf2011f190 con 0x7fdf2005e820 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf2005e820 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf2011f190 con 0x7fdf2005e820 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 5 = system:0 2024-09-10T18:57:56.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 4 2024-09-10T18:57:56.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.587+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 2 5 2024-09-10T18:57:56.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.595+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f0c500052c0 con 0x7f0c6c10b200 2024-09-10T18:57:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 10 client.25122.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 6 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 6 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2011f020 osd=2 3 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 2 6 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 4 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 10 client.25122.objecter _send_command 6 2024-09-10T18:57:56.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf2012a430 con 0x7fdf2005e820 2024-09-10T18:57:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf2012a430 con 0x7fdf2005e820 2024-09-10T18:57:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 4 2024-09-10T18:57:56.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 2 6 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 10 client.25122.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 7 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 7 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 3 7 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 4 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 10 client.25122.objecter _send_command 7 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf20126850 con 0x7fdf2005df00 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.607+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf2005df00 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.611+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf20126850 con 0x7fdf2005df00 2024-09-10T18:57:56.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.611+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:57:56.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.611+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 7 = system:0 2024-09-10T18:57:56.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.611+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 4 2024-09-10T18:57:56.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.611+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 3 7 2024-09-10T18:57:56.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 10 client.25122.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 8 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 8 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2005dd90 osd=3 3 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 3 8 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 4 2024-09-10T18:57:56.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 10 client.25122.objecter _send_command 8 2024-09-10T18:57:56.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf2012ee50 con 0x7fdf2005df00 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf2012ee50 con 0x7fdf2005df00 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 4 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 3 8 2024-09-10T18:57:56.634 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.627+0000 7fdf269ee640 10 client.25122.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 9 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 9 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 4 9 2024-09-10T18:57:56.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 4 2024-09-10T18:57:56.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 10 client.25122.objecter _send_command 9 2024-09-10T18:57:56.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf20131180 con 0x7fdf2012f4d0 2024-09-10T18:57:56.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf2012f4d0 2024-09-10T18:57:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf20131180 con 0x7fdf2012f4d0 2024-09-10T18:57:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:57:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 9 = system:0 2024-09-10T18:57:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 4 2024-09-10T18:57:56.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.631+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 4 9 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 10 client.25122.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 10 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 10 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf2012ee50 osd=4 3 2024-09-10T18:57:56.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 4 10 2024-09-10T18:57:56.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 4 2024-09-10T18:57:56.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 10 client.25122.objecter _send_command 10 2024-09-10T18:57:56.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.647+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf20131180 con 0x7fdf2012f4d0 2024-09-10T18:57:56.657 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:57:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf20131180 con 0x7fdf2012f4d0 2024-09-10T18:57:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 4 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 4 10 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 10 client.25122.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 11 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 11 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 5 11 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 4 2024-09-10T18:57:56.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 10 client.25122.objecter _send_command 11 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.651+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf201376a0 con 0x7fdf20131440 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf20131440 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf201376a0 con 0x7fdf20131440 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 11 = system:0 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 4 2024-09-10T18:57:56.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.655+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 5 11 2024-09-10T18:57:56.667 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:56.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 10 client.25122.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 12 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 12 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20131180 osd=5 3 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 5 12 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 4 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 10 client.25122.objecter _send_command 12 2024-09-10T18:57:56.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.671+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf20138f40 con 0x7fdf20131440 2024-09-10T18:57:56.681 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf20138f40 con 0x7fdf20131440 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 4 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 5 12 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 10 client.25122.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 13 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 13 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 6 13 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 4 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 10 client.25122.objecter _send_command 13 2024-09-10T18:57:56.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.675+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf20139a90 con 0x7fdf201395c0 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf201395c0 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf20139a90 con 0x7fdf201395c0 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 13 = system:0 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 4 2024-09-10T18:57:56.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.679+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 6 13 2024-09-10T18:57:56.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 10 client.25122.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 14 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 14 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20138f40 osd=6 3 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 6 14 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 4 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 10 client.25122.objecter _send_command 14 2024-09-10T18:57:56.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.699+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf20139cb0 con 0x7fdf201395c0 2024-09-10T18:57:56.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf20139cb0 con 0x7fdf201395c0 2024-09-10T18:57:56.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 4 2024-09-10T18:57:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 6 14 2024-09-10T18:57:56.707 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:57:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:57:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 10 client.25122.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:57:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 15 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 15 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 7 15 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 4 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 10 client.25122.objecter _send_command 15 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf20126a10 con 0x7fdf201382c0 2024-09-10T18:57:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 10 client.25122.objecter ms_handle_connect 0x7fdf201382c0 2024-09-10T18:57:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fdf20126a10 con 0x7fdf201382c0 2024-09-10T18:57:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:57:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 15 = system:0 2024-09-10T18:57:56.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 4 2024-09-10T18:57:56.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.703+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 7 15 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 10 client.25122.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign -1 16 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 20 client.25122.objecter _get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_remove -1 16 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 20 client.25122.objecter get_session s=0x7fdf20139cb0 osd=7 3 2024-09-10T18:57:56.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 15 client.25122.objecter _session_command_op_assign 7 16 2024-09-10T18:57:56.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 4 2024-09-10T18:57:56.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 10 client.25122.objecter _send_command 16 2024-09-10T18:57:56.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.727+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdf20126a10 con 0x7fdf201382c0 2024-09-10T18:57:56.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf1cff9640 1 -- 172.21.15.83:0/1165368822 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fdf20126a10 con 0x7fdf201382c0 2024-09-10T18:57:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf1cff9640 10 client.25122.objecter ms_dispatch 0x7fdf20000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:57:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf1cff9640 10 client.25122.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:57:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf1cff9640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 4 2024-09-10T18:57:56.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf1cff9640 15 client.25122.objecter _session_command_op_remove 7 16 2024-09-10T18:57:56.735 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:57:56.736 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:57:56.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 librados: watch_flush enter 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 librados: watch_flush exit 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.0 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6801/2535527483 conn(0x7fdf2011f820 legacy=0x7fdf2011fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf201098b0 osd=0 1 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.1 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6805/2309624637 conn(0x7fdf200f7b00 legacy=0x7fdf200f7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf200f7480 osd=1 1 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.2 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6809/386094220 conn(0x7fdf2005e820 legacy=0x7fdf20128d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2011f020 osd=2 1 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.3 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6813/1279688021 conn(0x7fdf2005df00 legacy=0x7fdf2012d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2005dd90 osd=3 1 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf24f8b640 1 -- 172.21.15.83:0/1165368822 reap_dead start 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.4 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.165:6800/3225515265 conn(0x7fdf2012f4d0 legacy=0x7fdf2012f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf24f8b640 1 -- 172.21.15.83:0/1165368822 reap_dead start 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf2012ee50 osd=4 1 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.5 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.165:6804/571173777 conn(0x7fdf20131440 legacy=0x7fdf20135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20131180 osd=5 1 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.6 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.165:6808/1317922936 conn(0x7fdf201395c0 legacy=0x7fdf201376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20138f40 osd=6 1 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter close_session for osd.7 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.165:6812/1610151650 conn(0x7fdf201382c0 legacy=0x7fdf2013f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter put_session s=0x7fdf20139cb0 osd=7 1 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 20 client.25122.objecter shutdown clearing up homeless session... 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 10 client.25122.objecter successfully canceled tick 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6800/3071793735 conn(0x7fdf0c071750 legacy=0x7fdf0c073c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> v1:172.21.15.83:6789/0 conn(0x7fdf201042a0 legacy=0x7fdf20119830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf24f8b640 1 -- 172.21.15.83:0/1165368822 reap_dead start 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 shutdown_connections 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 >> 172.21.15.83:0/1165368822 conn(0x7fdf200f70b0 msgr2=0x7fdf2010b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 shutdown_connections 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 -- 172.21.15.83:0/1165368822 wait complete. 2024-09-10T18:57:56.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:56.731+0000 7fdf269ee640 1 librados: shutdown 2024-09-10T18:57:56.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.665218+0000 osd.4 (osd.4) 231 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:57:56.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.669847+0000 osd.4 (osd.4) 232 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:57:56.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.818456+0000 osd.1 (osd.1) 205 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:57:56.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:54.822313+0000 osd.1 (osd.1) 206 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.378358+0000 osd.6 (osd.6) 153 : cluster [DBG] 3.a scrub starts 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.387881+0000 osd.6 (osd.6) 154 : cluster [DBG] 3.a scrub ok 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:55.494324+0000 mon.b (mon.1) 254 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:55.496480+0000 mon.a (mon.0) 2088 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.496614+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:55.498083+0000 mon.a (mon.0) 2090 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.661212+0000 osd.0 (osd.0) 196 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.666012+0000 osd.0 (osd.0) 197 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.021648+0000 mgr.y (mgr.24526) 353 : cluster [DBG] pgmap v463: 250 pgs: 2 active+clean+snaptrim, 248 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.022113+0000 mon.c (mon.2) 531 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.023652+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.047848+0000 mon.c (mon.2) 532 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.055880+0000 mon.a (mon.0) 2092 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.066780+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.066974+0000 mon.a (mon.0) 2094 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:56.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: audit 2024-09-10T18:57:56.067116+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:57:56.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:56 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.067211+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:57:56.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.665218+0000 osd.4 (osd.4) 231 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.669847+0000 osd.4 (osd.4) 232 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.818456+0000 osd.1 (osd.1) 205 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:54.822313+0000 osd.1 (osd.1) 206 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.378358+0000 osd.6 (osd.6) 153 : cluster [DBG] 3.a scrub starts 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.387881+0000 osd.6 (osd.6) 154 : cluster [DBG] 3.a scrub ok 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:55.494324+0000 mon.b (mon.1) 254 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:55.496480+0000 mon.a (mon.0) 2088 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.496614+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:57:56.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:55.498083+0000 mon.a (mon.0) 2090 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.661212+0000 osd.0 (osd.0) 196 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.666012+0000 osd.0 (osd.0) 197 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.021648+0000 mgr.y (mgr.24526) 353 : cluster [DBG] pgmap v463: 250 pgs: 2 active+clean+snaptrim, 248 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.022113+0000 mon.c (mon.2) 531 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.023652+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.047848+0000 mon.c (mon.2) 532 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.055880+0000 mon.a (mon.0) 2092 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.066780+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:56.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.066974+0000 mon.a (mon.0) 2094 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.665218+0000 osd.4 (osd.4) 231 : cluster [DBG] 3.3 scrub starts 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.669847+0000 osd.4 (osd.4) 232 : cluster [DBG] 3.3 scrub ok 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.818456+0000 osd.1 (osd.1) 205 : cluster [DBG] 6.1d scrub starts 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:54.822313+0000 osd.1 (osd.1) 206 : cluster [DBG] 6.1d scrub ok 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.378358+0000 osd.6 (osd.6) 153 : cluster [DBG] 3.a scrub starts 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.387881+0000 osd.6 (osd.6) 154 : cluster [DBG] 3.a scrub ok 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:55.494324+0000 mon.b (mon.1) 254 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:55.496480+0000 mon.a (mon.0) 2088 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.496614+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:55.498083+0000 mon.a (mon.0) 2090 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]: dispatch 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.661212+0000 osd.0 (osd.0) 196 : cluster [DBG] 3.17 scrub starts 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.666012+0000 osd.0 (osd.0) 197 : cluster [DBG] 3.17 scrub ok 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.021648+0000 mgr.y (mgr.24526) 353 : cluster [DBG] pgmap v463: 250 pgs: 2 active+clean+snaptrim, 248 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.022113+0000 mon.c (mon.2) 531 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.023652+0000 mon.a (mon.0) 2091 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.047848+0000 mon.c (mon.2) 532 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.055880+0000 mon.a (mon.0) 2092 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.066780+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi083-77648-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.066974+0000 mon.a (mon.0) 2094 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-35"}]': finished 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: audit 2024-09-10T18:57:56.067116+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.067211+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: audit 2024-09-10T18:57:56.067116+0000 mon.a (mon.0) 2095 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-10T18:57:56.767 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:56 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.067211+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-09-10T18:57:56.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:56.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:56.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:56.932 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:56.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:57.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:57.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.067+0000 7f0c62ffd640 1 -- 172.21.15.83:0/3925826915 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v321) v1 ==== 87+0+0 (unknown 4212529106 0 0) 0x7f0c640567b0 con 0x7f0c6c10b200 2024-09-10T18:57:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T18:57:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 10 librados: watch_flush enter 2024-09-10T18:57:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 10 librados: watch_flush exit 2024-09-10T18:57:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 20 client.49187.objecter shutdown clearing up homeless session... 2024-09-10T18:57:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 10 client.49187.objecter successfully canceled tick 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 >> v1:172.21.15.83:6800/3071793735 conn(0x7f0c58079be0 legacy=0x7f0c5807c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 >> v1:172.21.15.83:6790/0 conn(0x7f0c6c10b200 legacy=0x7f0c6c11cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 shutdown_connections 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 >> 172.21.15.83:0/3925826915 conn(0x7f0c6c075290 msgr2=0x7f0c6c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 shutdown_connections 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 -- 172.21.15.83:0/3925826915 wait complete. 2024-09-10T18:57:57.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:57:57.071+0000 7f0c745a1640 1 librados: shutdown 2024-09-10T18:57:57.078 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:57.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:57.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:57.321 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:57.356 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:57.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:57.503 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:57.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.983238+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:55.986720+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.123170+0000 osd.5 (osd.5) 185 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.125874+0000 osd.5 (osd.5) 186 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: audit 2024-09-10T18:57:56.546572+0000 mon.a (mon.0) 2097 : audit [DBG] from='client.? v1:172.21.15.83:0/1165368822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: audit 2024-09-10T18:57:56.599236+0000 mon.c (mon.2) 533 : audit [INF] from='client.? v1:172.21.15.83:0/3925826915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: audit 2024-09-10T18:57:56.599786+0000 mon.a (mon.0) 2098 : audit [INF] from='client.49187 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: audit 2024-09-10T18:57:57.049244+0000 mon.c (mon.2) 534 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:57.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:57.643 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:57.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:57.760 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:57.058528+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:57.760 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: audit 2024-09-10T18:57:57.067136+0000 mon.a (mon.0) 2100 : audit [INF] from='client.49187 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:57:57.760 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[18629]: cluster 2024-09-10T18:57:57.067342+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:57:57.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.983238+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:55.986720+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.123170+0000 osd.5 (osd.5) 185 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.125874+0000 osd.5 (osd.5) 186 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: audit 2024-09-10T18:57:56.546572+0000 mon.a (mon.0) 2097 : audit [DBG] from='client.? v1:172.21.15.83:0/1165368822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: audit 2024-09-10T18:57:56.599236+0000 mon.c (mon.2) 533 : audit [INF] from='client.? v1:172.21.15.83:0/3925826915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: audit 2024-09-10T18:57:56.599786+0000 mon.a (mon.0) 2098 : audit [INF] from='client.49187 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: audit 2024-09-10T18:57:57.049244+0000 mon.c (mon.2) 534 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:57.058528+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: audit 2024-09-10T18:57:57.067136+0000 mon.a (mon.0) 2100 : audit [INF] from='client.49187 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:57:57.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:57 smithi083 bash[32381]: cluster 2024-09-10T18:57:57.067342+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:57:57.779 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.983238+0000 osd.2 (osd.2) 137 : cluster [DBG] 2.1 scrub starts 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:55.986720+0000 osd.2 (osd.2) 138 : cluster [DBG] 2.1 scrub ok 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.123170+0000 osd.5 (osd.5) 185 : cluster [DBG] 2.3 scrub starts 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.125874+0000 osd.5 (osd.5) 186 : cluster [DBG] 2.3 scrub ok 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: audit 2024-09-10T18:57:56.546572+0000 mon.a (mon.0) 2097 : audit [DBG] from='client.? v1:172.21.15.83:0/1165368822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: audit 2024-09-10T18:57:56.599236+0000 mon.c (mon.2) 533 : audit [INF] from='client.? v1:172.21.15.83:0/3925826915' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: audit 2024-09-10T18:57:56.599786+0000 mon.a (mon.0) 2098 : audit [INF] from='client.49187 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:57:57.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: audit 2024-09-10T18:57:57.049244+0000 mon.c (mon.2) 534 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:57.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:57.058528+0000 mon.a (mon.0) 2099 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:57:57.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: audit 2024-09-10T18:57:57.067136+0000 mon.a (mon.0) 2100 : audit [INF] from='client.49187 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:57:57.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[22591]: cluster 2024-09-10T18:57:57.067342+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-09-10T18:57:57.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:57.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:57.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:58.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:58.053 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:58.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:57:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:57:58.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:58.188 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:58.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:58.327 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:57:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:57:58.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:58.464 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:57:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:57:58.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:58.599 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:57:58.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.347313+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.350989+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.370435+0000 osd.7 (osd.7) 204 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.374906+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.823654+0000 osd.1 (osd.1) 207 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:57:58.717 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:56.826947+0000 osd.1 (osd.1) 208 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: audit 2024-09-10T18:57:57.824649+0000 mgr.y (mgr.24526) 354 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:58.022543+0000 mgr.y (mgr.24526) 355 : cluster [DBG] pgmap v466: 226 pgs: 8 unknown, 1 active+clean+snaptrim, 217 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: audit 2024-09-10T18:57:58.050811+0000 mon.c (mon.2) 535 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: audit 2024-09-10T18:57:58.070673+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: cluster 2024-09-10T18:57:58.071309+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: audit 2024-09-10T18:57:58.075433+0000 mon.a (mon.0) 2103 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.718 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:58 smithi165 bash[22591]: audit 2024-09-10T18:57:58.075708+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:58.738 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.347313+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.350989+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.370435+0000 osd.7 (osd.7) 204 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.374906+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.823654+0000 osd.1 (osd.1) 207 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:56.826947+0000 osd.1 (osd.1) 208 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:57:58.785 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: audit 2024-09-10T18:57:57.824649+0000 mgr.y (mgr.24526) 354 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:58.022543+0000 mgr.y (mgr.24526) 355 : cluster [DBG] pgmap v466: 226 pgs: 8 unknown, 1 active+clean+snaptrim, 217 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: audit 2024-09-10T18:57:58.050811+0000 mon.c (mon.2) 535 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: audit 2024-09-10T18:57:58.070673+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: cluster 2024-09-10T18:57:58.071309+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: audit 2024-09-10T18:57:58.075433+0000 mon.a (mon.0) 2103 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[32381]: audit 2024-09-10T18:57:58.075708+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.347313+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.17 scrub starts 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.350989+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.17 scrub ok 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.370435+0000 osd.7 (osd.7) 204 : cluster [DBG] 1.0 scrub starts 2024-09-10T18:57:58.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.374906+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub ok 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.823654+0000 osd.1 (osd.1) 207 : cluster [DBG] 12.7 scrub starts 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:56.826947+0000 osd.1 (osd.1) 208 : cluster [DBG] 12.7 scrub ok 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: audit 2024-09-10T18:57:57.824649+0000 mgr.y (mgr.24526) 354 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:58.022543+0000 mgr.y (mgr.24526) 355 : cluster [DBG] pgmap v466: 226 pgs: 8 unknown, 1 active+clean+snaptrim, 217 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: audit 2024-09-10T18:57:58.050811+0000 mon.c (mon.2) 535 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: audit 2024-09-10T18:57:58.070673+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: cluster 2024-09-10T18:57:58.071309+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: audit 2024-09-10T18:57:58.075433+0000 mon.a (mon.0) 2103 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:57:58.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:58 smithi083 bash[18629]: audit 2024-09-10T18:57:58.075708+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:58.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:58.875 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:58.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:58.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:59.014 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:57:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:57:59.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:59.143 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:59.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:59.280 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:59.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:59.416 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:57:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:57:59.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:59.550 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:57:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:57:59.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:59.684 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:57:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:57:59.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:59.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.052297+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:59.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.075845+0000 mon.a (mon.0) 2105 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:59.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.076053+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: cluster 2024-09-10T18:57:59.076422+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.077177+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.078131+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[32381]: audit 2024-09-10T18:57:59.086807+0000 mon.a (mon.0) 2109 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.052297+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.075845+0000 mon.a (mon.0) 2105 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.076053+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: cluster 2024-09-10T18:57:59.076422+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.077177+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.078131+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:57:59 smithi083 bash[18629]: audit 2024-09-10T18:57:59.086807+0000 mon.a (mon.0) 2109 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.052297+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.075845+0000 mon.a (mon.0) 2105 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.076053+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: cluster 2024-09-10T18:57:59.076422+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.077177+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.078131+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]: dispatch 2024-09-10T18:57:59.801 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:57:59 smithi165 bash[22591]: audit 2024-09-10T18:57:59.086807+0000 mon.a (mon.0) 2109 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:57:59.822 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:57:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:57:59.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:59.960 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:57:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:57:59.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:00.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:00.102 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:00.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:00.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:00.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:00.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- 172.21.15.83:0/52600741 >> v1:172.21.15.83:6789/0 conn(0x7f83201075a0 legacy=0x7f8320109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:00.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- 172.21.15.83:0/52600741 shutdown_connections 2024-09-10T18:58:00.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- 172.21.15.83:0/52600741 >> 172.21.15.83:0/52600741 conn(0x7f83200fa370 msgr2=0x7f83200fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:00.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- 172.21.15.83:0/52600741 shutdown_connections 2024-09-10T18:58:00.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- 172.21.15.83:0/52600741 wait complete. 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 librados: starting msgr at 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 librados: starting objecter 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 Processor -- start 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 -- start start 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 librados: setting wanted keys 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.271+0000 7f8325d1c640 1 librados: calling monclient init 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8320104090 con 0x7f8320104810 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f832011f650 con 0x7f83201075a0 2024-09-10T18:58:00.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f83201207f0 con 0x7f832010b130 2024-09-10T18:58:00.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f831ffff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f832010b130 0x7f832011cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39202/0 (socket says 172.21.15.83:39202) 2024-09-10T18:58:00.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f831ffff640 1 -- 172.21.15.83:0/2352347524 learned_addr learned my addr 172.21.15.83:0/2352347524 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:00.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f831f7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8320104810 0x7f83201034e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38772/0 (socket says 172.21.15.83:38772) 2024-09-10T18:58:00.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter ms_handle_connect 0x7f832010b130 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter ms_handle_connect 0x7f8320104810 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter ms_handle_connect 0x7f83201075a0 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520704966 0 0) 0x7f83201207f0 con 0x7f832010b130 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8308003720 con 0x7f832010b130 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038633130 0 0) 0x7f8320104090 con 0x7f8320104810 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83201207f0 con 0x7f8320104810 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840206490 0 0) 0x7f832011f650 con 0x7f83201075a0 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320104090 con 0x7f83201075a0 2024-09-10T18:58:00.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692987811 0 0) 0x7f8308003720 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f832011f650 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8318001900 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2568132591 0 0) 0x7f83201207f0 con 0x7f8320104810 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8308003720 con 0x7f8320104810 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f83140018c0 con 0x7f8320104810 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3749776578 0 0) 0x7f832011f650 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 >> v1:172.21.15.165:6789/0 conn(0x7f83201075a0 legacy=0x7f83201197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 >> v1:172.21.15.83:6789/0 conn(0x7f8320104810 legacy=0x7f83201034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83201219d0 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832011e6a0 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 10 client.49205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f832011eb70 con 0x7f832010b130 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8325d1c640 1 librados: init done 2024-09-10T18:58:00.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f83180023e0 con 0x7f832010b130 2024-09-10T18:58:00.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.275+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8318005a80 con 0x7f832010b130 2024-09-10T18:58:00.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c005010 con 0x7f832010b130 2024-09-10T18:58:00.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f8318001f80 con 0x7f832010b130 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(324..324 src has 69..324) v4 ==== 8406+0+0 (unknown 3748573371 0 0) 0x7f831808a8d0 con 0x7f832010b130 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 10 client.49205.objecter ms_dispatch 0x7f8320000b90 osd_map(324..324 src has 69..324) v4 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 3 client.49205.objecter handle_osd_map got epochs [324,324] > 0 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 3 client.49205.objecter handle_osd_map decoding full epoch 324 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 20 client.49205.objecter dump_active .. 0 homeless 2024-09-10T18:58:00.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.279+0000 7f8307fff640 10 client.49205.objecter ms_handle_connect 0x7f8308071680 2024-09-10T18:58:00.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.287+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8318056d00 con 0x7f832010b130 2024-09-10T18:58:00.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:00.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.455+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f830c000b90 con 0x7f8308071680 2024-09-10T18:58:00.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:00.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.475+0000 7f8307fff640 1 -- 172.21.15.83:0/2352347524 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+673739 (unknown 2965378022 0 3641236680) 0x7f830c000b90 con 0x7f8308071680 2024-09-10T18:58:00.487 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:58:00.493 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 10 librados: watch_flush enter 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 10 librados: watch_flush exit 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 20 client.49205.objecter shutdown clearing up homeless session... 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 10 client.49205.objecter successfully canceled tick 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8308071680 legacy=0x7f8308073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 >> v1:172.21.15.83:6790/0 conn(0x7f832010b130 legacy=0x7f832011cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 shutdown_connections 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 >> 172.21.15.83:0/2352347524 conn(0x7f83200fa370 msgr2=0x7f832010b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:00.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 shutdown_connections 2024-09-10T18:58:00.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 -- 172.21.15.83:0/2352347524 wait complete. 2024-09-10T18:58:00.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:00.483+0000 7f8325d1c640 1 librados: shutdown 2024-09-10T18:58:00.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:00.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:00.620 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:57:59.570006+0000 mon.a (mon.0) 2110 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:00.620 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:57:59.572453+0000 mon.c (mon.2) 537 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: cluster 2024-09-10T18:57:59.766848+0000 osd.5 (osd.5) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: cluster 2024-09-10T18:57:59.767136+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:00.023369+0000 mgr.y (mgr.24526) 356 : cluster [DBG] pgmap v469: 249 pgs: 32 unknown, 1 active+clean+snaptrim, 216 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.024150+0000 mon.c (mon.2) 538 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.027065+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.053772+0000 mon.c (mon.2) 539 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.076968+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.077415+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.077539+0000 mon.a (mon.0) 2113 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.077681+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:00.077791+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.080836+0000 mon.a (mon.0) 2116 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.621 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.103227+0000 mon.c (mon.2) 540 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.103558+0000 mon.a (mon.0) 2117 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:57:59.570006+0000 mon.a (mon.0) 2110 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:57:59.572453+0000 mon.c (mon.2) 537 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: cluster 2024-09-10T18:57:59.766848+0000 osd.5 (osd.5) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: cluster 2024-09-10T18:57:59.767136+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:00.023369+0000 mgr.y (mgr.24526) 356 : cluster [DBG] pgmap v469: 249 pgs: 32 unknown, 1 active+clean+snaptrim, 216 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.024150+0000 mon.c (mon.2) 538 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.027065+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.053772+0000 mon.c (mon.2) 539 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:00.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.076968+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.077415+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.077539+0000 mon.a (mon.0) 2113 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.077681+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:00.077791+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.080836+0000 mon.a (mon.0) 2116 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.103227+0000 mon.c (mon.2) 540 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.103558+0000 mon.a (mon.0) 2117 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.104749+0000 mon.c (mon.2) 541 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.105030+0000 mon.a (mon.0) 2118 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.105599+0000 mon.c (mon.2) 542 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[18629]: audit 2024-09-10T18:58:00.105916+0000 mon.a (mon.0) 2119 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.104749+0000 mon.c (mon.2) 541 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.105030+0000 mon.a (mon.0) 2118 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.105599+0000 mon.c (mon.2) 542 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:00 smithi083 bash[32381]: audit 2024-09-10T18:58:00.105916+0000 mon.a (mon.0) 2119 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.638 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:00.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:58:00.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:58:00.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:00.772 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:57:59.570006+0000 mon.a (mon.0) 2110 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:57:59.572453+0000 mon.c (mon.2) 537 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: cluster 2024-09-10T18:57:59.766848+0000 osd.5 (osd.5) 187 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: cluster 2024-09-10T18:57:59.767136+0000 osd.5 (osd.5) 188 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:00.023369+0000 mgr.y (mgr.24526) 356 : cluster [DBG] pgmap v469: 249 pgs: 32 unknown, 1 active+clean+snaptrim, 216 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.024150+0000 mon.c (mon.2) 538 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.027065+0000 mon.a (mon.0) 2111 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-10T18:58:00.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.053772+0000 mon.c (mon.2) 539 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.076968+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.077415+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? v1:172.21.15.83:0/3788743919' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi083-77648-53"}]': finished 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.077539+0000 mon.a (mon.0) 2113 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.077681+0000 mon.a (mon.0) 2114 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:00.077791+0000 mon.a (mon.0) 2115 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.080836+0000 mon.a (mon.0) 2116 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.103227+0000 mon.c (mon.2) 540 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.103558+0000 mon.a (mon.0) 2117 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.104749+0000 mon.c (mon.2) 541 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.105030+0000 mon.a (mon.0) 2118 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.105599+0000 mon.c (mon.2) 542 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:00 smithi165 bash[22591]: audit 2024-09-10T18:58:00.105916+0000 mon.a (mon.0) 2119 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:00.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:00.897 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:00.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:01.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:01.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:01.166 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:01.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:01.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:01.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:01.441 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:01.506 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-10T18:58:00.075124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4793749,"num_objects":335,"num_object_clones":2,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":1471,"num_read_kb":1232,"num_write":1195,"num_write_kb":10466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":591740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2491,"ondisk_log_size":2491,"up":651,"acting":651,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":751,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2490120,"kb_used_data":52952,"kb_used_omap":13,"kb_used_meta":2436978,"kb_avail":747471096,"statfs":{"total":767960285184,"available":765410402304,"internally_reserved":0,"allocated":54222848,"data_stored":36386457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2495466176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2,"num_objects":-2,"num_object_clones":-1,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-3,"num_write_kb":-3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-3,"num_flush_kb":-3,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013271"},"pg_stats":[{"pgid":"187.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"187.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:57:58.060769+0000","last_change":"2024-09-10T18:57:58.060769+0000","last_active":"2024-09-10T18:57:58.060769+0000","last_peered":"2024-09-10T18:57:58.060769+0000","last_clean":"2024-09-10T18:57:58.060769+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:57:58.060769+0000","last_undegraded":"2024-09-10T18:57:58.060769+0000","last_fullsized":"2024-09-10T18:57:58.060769+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:58.060769+0000","last_clean_scrub_stamp":"2024-09-10T18:57:58.060769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":266,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.092083+0000","last_change":"2024-09-10T18:57:54.092083+0000","last_active":"2024-09-10T18:57:54.092083+0000","last_peered":"2024-09-10T18:57:54.092083+0000","last_clean":"2024-09-10T18:57:54.092083+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:57:54.092083+0000","last_undegraded":"2024-09-10T18:57:54.092083+0000","last_fullsized":"2024-09-10T18:57:54.092083+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.1e","version":"318'64","reported_seq":293,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2024-09-10T18:57:54.094421+0000","last_change":"2024-09-10T18:57:54.090539+0000","last_active":"2024-09-10T18:57:54.094421+0000","last_peered":"2024-09-10T18:57:54.094421+0000","last_clean":"2024-09-10T18:57:54.094421+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:57:54.094421+0000","last_undegraded":"2024-09-10T18:57:54.094421+0000","last_fullsized":"2024-09-10T18:57:54.094421+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027151649999999999,"stat_sum":{"num_bytes":40,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"276'22","reported_seq":270,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.056252+0000","last_change":"2024-09-10T18:57:54.056252+0000","last_active":"2024-09-10T18:57:54.056252+0000","last_peered":"2024-09-10T18:57:54.056252+0000","last_clean":"2024-09-10T18:57:54.056252+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:57:54.056252+0000","last_undegraded":"2024-09-10T18:57:54.056252+0000","last_fullsized":"2024-09-10T18:57:54.056252+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.247847+0000","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.1c","version":"244'51","reported_seq":283,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097920+0000","last_change":"2024-09-10T18:57:54.097920+0000","last_active":"2024-09-10T18:57:54.097920+0000","last_peered":"2024-09-10T18:57:54.097920+0000","last_clean":"2024-09-10T18:57:54.097920+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:57:54.097920+0000","last_undegraded":"2024-09-10T18:57:54.097920+0000","last_fullsized":"2024-09-10T18:57:54.097920+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.052158+0000","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.1b","version":"240'68","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097343+0000","last_change":"2024-09-10T18:57:54.097343+0000","last_active":"2024-09-10T18:57:54.097343+0000","last_peered":"2024-09-10T18:57:54.097343+0000","last_clean":"2024-09-10T18:57:54.097343+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:57:54.097343+0000","last_undegraded":"2024-09-10T18:57:54.097343+0000","last_fullsized":"2024-09-10T18:57:54.097343+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:57:30.615944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.615944+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:46.447187+0000","scrub_duration":0.005614539,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":274,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.056894+0000","last_change":"2024-09-10T18:57:54.056894+0000","last_active":"2024-09-10T18:57:54.056894+0000","last_peered":"2024-09-10T18:57:54.056894+0000","last_clean":"2024-09-10T18:57:54.056894+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:57:54.056894+0000","last_undegraded":"2024-09-10T18:57:54.056894+0000","last_fullsized":"2024-09-10T18:57:54.056894+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:57:35.352719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.352719+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.462559+0000","scrub_duration":0.0074091110000000003,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.19","version":"257'22","reported_seq":269,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061737+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T18:57:54.061737+0000","last_peered":"2024-09-10T18:57:54.061737+0000","last_clean":"2024-09-10T18:57:54.061737+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:57:54.061737+0000","last_undegraded":"2024-09-10T18:57:54.061737+0000","last_fullsized":"2024-09-10T18:57:54.061737+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.319797+0000","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.18","version":"254'104","reported_seq":331,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071086+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T18:57:54.071086+0000","last_peered":"2024-09-10T18:57:54.071086+0000","last_clean":"2024-09-10T18:57:54.071086+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:57:54.071086+0000","last_undegraded":"2024-09-10T18:57:54.071086+0000","last_fullsized":"2024-09-10T18:57:54.071086+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.932946+0000","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.17","version":"230'17","reported_seq":259,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.074533+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T18:57:54.074533+0000","last_peered":"2024-09-10T18:57:54.074533+0000","last_clean":"2024-09-10T18:57:54.074533+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:57:54.074533+0000","last_undegraded":"2024-09-10T18:57:54.074533+0000","last_fullsized":"2024-09-10T18:57:54.074533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:17.416599+0000","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.16","version":"276'22","reported_seq":265,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.099018+0000","last_change":"2024-09-10T18:57:54.099018+0000","last_active":"2024-09-10T18:57:54.099018+0000","last_peered":"2024-09-10T18:57:54.099018+0000","last_clean":"2024-09-10T18:57:54.099018+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:57:54.099018+0000","last_undegraded":"2024-09-10T18:57:54.099018+0000","last_fullsized":"2024-09-10T18:57:54.099018+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:42.649292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.649292+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.495881+0000","scrub_duration":0.004377405,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.15","version":"240'74","reported_seq":307,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065259+0000","last_change":"2024-09-10T18:57:54.065259+0000","last_active":"2024-09-10T18:57:54.065259+0000","last_peered":"2024-09-10T18:57:54.065259+0000","last_clean":"2024-09-10T18:57:54.065259+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:57:54.065259+0000","last_undegraded":"2024-09-10T18:57:54.065259+0000","last_fullsized":"2024-09-10T18:57:54.065259+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'74","last_scrub_stamp":"2024-09-10T18:57:41.887503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.887503+0000","objects_scrubbed":6,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:47.295134+0000","scrub_duration":0.0039986459999999998,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.14","version":"246'95","reported_seq":334,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.098423+0000","last_change":"2024-09-10T18:57:54.098423+0000","last_active":"2024-09-10T18:57:54.098423+0000","last_peered":"2024-09-10T18:57:54.098423+0000","last_clean":"2024-09-10T18:57:54.098423+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:57:54.098423+0000","last_undegraded":"2024-09-10T18:57:54.098423+0000","last_fullsized":"2024-09-10T18:57:54.098423+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'95","last_scrub_stamp":"2024-09-10T18:57:25.633366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.633366+0000","objects_scrubbed":6,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:31.276059+0000","scrub_duration":0.0044083560000000004,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.13","version":"146'1","reported_seq":248,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.062883+0000","last_change":"2024-09-10T18:57:54.062883+0000","last_active":"2024-09-10T18:57:54.062883+0000","last_peered":"2024-09-10T18:57:54.062883+0000","last_clean":"2024-09-10T18:57:54.062883+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:57:54.062883+0000","last_undegraded":"2024-09-10T18:57:54.062883+0000","last_fullsized":"2024-09-10T18:57:54.062883+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:57:38.843609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.843609+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.556193+0000","scrub_duration":0.003475601,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.12","version":"248'51","reported_seq":273,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072598+0000","last_change":"2024-09-10T18:57:54.072598+0000","last_active":"2024-09-10T18:57:54.072598+0000","last_peered":"2024-09-10T18:57:54.072598+0000","last_clean":"2024-09-10T18:57:54.072598+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:57:54.072598+0000","last_undegraded":"2024-09-10T18:57:54.072598+0000","last_fullsized":"2024-09-10T18:57:54.072598+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.11","version":"297'52","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.062338+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T18:57:54.062338+0000","last_peered":"2024-09-10T18:57:54.062338+0000","last_clean":"2024-09-10T18:57:54.062338+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:57:54.062338+0000","last_undegraded":"2024-09-10T18:57:54.062338+0000","last_fullsized":"2024-09-10T18:57:54.062338+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:04.071589+0000","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.10","version":"224'34","reported_seq":262,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064626+0000","last_change":"2024-09-10T18:57:54.064626+0000","last_active":"2024-09-10T18:57:54.064626+0000","last_peered":"2024-09-10T18:57:54.064626+0000","last_clean":"2024-09-10T18:57:54.064626+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:57:54.064626+0000","last_undegraded":"2024-09-10T18:57:54.064626+0000","last_fullsized":"2024-09-10T18:57:54.064626+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.f","version":"276'63","reported_seq":303,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.078499+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T18:57:54.078499+0000","last_peered":"2024-09-10T18:57:54.078499+0000","last_clean":"2024-09-10T18:57:54.078499+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:57:54.078499+0000","last_undegraded":"2024-09-10T18:57:54.078499+0000","last_fullsized":"2024-09-10T18:57:54.078499+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.262512+0000","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.e","version":"246'119","reported_seq":344,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.082546+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T18:57:54.082546+0000","last_peered":"2024-09-10T18:57:54.082546+0000","last_clean":"2024-09-10T18:57:54.082546+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:57:54.082546+0000","last_undegraded":"2024-09-10T18:57:54.082546+0000","last_fullsized":"2024-09-10T18:57:54.082546+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.422085+0000","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.d","version":"243'18","reported_seq":265,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066253+0000","last_change":"2024-09-10T18:57:54.066253+0000","last_active":"2024-09-10T18:57:54.066253+0000","last_peered":"2024-09-10T18:57:54.066253+0000","last_clean":"2024-09-10T18:57:54.066253+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:57:54.066253+0000","last_undegraded":"2024-09-10T18:57:54.066253+0000","last_fullsized":"2024-09-10T18:57:54.066253+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.c","version":"252'22","reported_seq":259,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072778+0000","last_change":"2024-09-10T18:57:54.072778+0000","last_active":"2024-09-10T18:57:54.072778+0000","last_peered":"2024-09-10T18:57:54.072778+0000","last_clean":"2024-09-10T18:57:54.072778+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:57:54.072778+0000","last_undegraded":"2024-09-10T18:57:54.072778+0000","last_fullsized":"2024-09-10T18:57:54.072778+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.b","version":"247'35","reported_seq":274,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097877+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T18:57:54.097877+0000","last_peered":"2024-09-10T18:57:54.097877+0000","last_clean":"2024-09-10T18:57:54.097877+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:57:54.097877+0000","last_undegraded":"2024-09-10T18:57:54.097877+0000","last_fullsized":"2024-09-10T18:57:54.097877+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.729128+0000","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.a","version":"232'17","reported_seq":263,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061755+0000","last_change":"2024-09-10T18:57:54.061755+0000","last_active":"2024-09-10T18:57:54.061755+0000","last_peered":"2024-09-10T18:57:54.061755+0000","last_clean":"2024-09-10T18:57:54.061755+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:57:54.061755+0000","last_undegraded":"2024-09-10T18:57:54.061755+0000","last_fullsized":"2024-09-10T18:57:54.061755+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:57:32.795149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.795149+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:35.820364+0000","scrub_duration":0.0040623889999999996,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":246,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063413+0000","last_change":"2024-09-10T18:57:54.063413+0000","last_active":"2024-09-10T18:57:54.063413+0000","last_peered":"2024-09-10T18:57:54.063413+0000","last_clean":"2024-09-10T18:57:54.063413+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:57:54.063413+0000","last_undegraded":"2024-09-10T18:57:54.063413+0000","last_fullsized":"2024-09-10T18:57:54.063413+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.800250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.800250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.514815+0000","scrub_duration":0.0031899250000000001,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.8","version":"234'51","reported_seq":290,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066122+0000","last_change":"2024-09-10T18:57:54.066122+0000","last_active":"2024-09-10T18:57:54.066122+0000","last_peered":"2024-09-10T18:57:54.066122+0000","last_clean":"2024-09-10T18:57:54.066122+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:57:54.066122+0000","last_undegraded":"2024-09-10T18:57:54.066122+0000","last_fullsized":"2024-09-10T18:57:54.066122+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T18:57:35.624392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.624392+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.488899+0000","scrub_duration":0.004362795,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.7","version":"276'158","reported_seq":387,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068278+0000","last_change":"2024-09-10T18:57:54.068278+0000","last_active":"2024-09-10T18:57:54.068278+0000","last_peered":"2024-09-10T18:57:54.068278+0000","last_clean":"2024-09-10T18:57:54.068278+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:57:54.068278+0000","last_undegraded":"2024-09-10T18:57:54.068278+0000","last_fullsized":"2024-09-10T18:57:54.068278+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'158","last_scrub_stamp":"2024-09-10T18:57:37.839200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.839200+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.802432+0000","scrub_duration":0.0057174490000000003,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.6","version":"290'93","reported_seq":376,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058056+0000","last_change":"2024-09-10T18:57:54.058056+0000","last_active":"2024-09-10T18:57:54.058056+0000","last_peered":"2024-09-10T18:57:54.058056+0000","last_clean":"2024-09-10T18:57:54.058056+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:57:54.058056+0000","last_undegraded":"2024-09-10T18:57:54.058056+0000","last_fullsized":"2024-09-10T18:57:54.058056+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'93","last_scrub_stamp":"2024-09-10T18:57:29.481888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.481888+0000","objects_scrubbed":6,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.178966+0000","scrub_duration":0.004612952,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.5","version":"305'53","reported_seq":283,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.078767+0000","last_change":"2024-09-10T18:57:54.078767+0000","last_active":"2024-09-10T18:57:54.078767+0000","last_peered":"2024-09-10T18:57:54.078767+0000","last_clean":"2024-09-10T18:57:54.078767+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:57:54.078767+0000","last_undegraded":"2024-09-10T18:57:54.078767+0000","last_fullsized":"2024-09-10T18:57:54.078767+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.4","version":"257'107","reported_seq":337,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060629+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T18:57:54.060629+0000","last_peered":"2024-09-10T18:57:54.060629+0000","last_clean":"2024-09-10T18:57:54.060629+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:57:54.060629+0000","last_undegraded":"2024-09-10T18:57:54.060629+0000","last_fullsized":"2024-09-10T18:57:54.060629+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.441046+0000","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.3","version":"236'102","reported_seq":323,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065977+0000","last_change":"2024-09-10T18:57:54.065977+0000","last_active":"2024-09-10T18:57:54.065977+0000","last_peered":"2024-09-10T18:57:54.065977+0000","last_clean":"2024-09-10T18:57:54.065977+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:57:54.065977+0000","last_undegraded":"2024-09-10T18:57:54.065977+0000","last_fullsized":"2024-09-10T18:57:54.065977+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.2","version":"230'34","reported_seq":272,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068000+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T18:57:54.068000+0000","last_peered":"2024-09-10T18:57:54.068000+0000","last_clean":"2024-09-10T18:57:54.068000+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:57:54.068000+0000","last_undegraded":"2024-09-10T18:57:54.068000+0000","last_fullsized":"2024-09-10T18:57:54.068000+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.440202+0000","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.1","version":"263'72","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071923+0000","last_change":"2024-09-10T18:57:54.071923+0000","last_active":"2024-09-10T18:57:54.071923+0000","last_peered":"2024-09-10T18:57:54.071923+0000","last_clean":"2024-09-10T18:57:54.071923+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:57:54.071923+0000","last_undegraded":"2024-09-10T18:57:54.071923+0000","last_fullsized":"2024-09-10T18:57:54.071923+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"64.0","version":"246'102","reported_seq":328,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060094+0000","last_change":"2024-09-10T18:57:54.060094+0000","last_active":"2024-09-10T18:57:54.060094+0000","last_peered":"2024-09-10T18:57:54.060094+0000","last_clean":"2024-09-10T18:57:54.060094+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:57:54.060094+0000","last_undegraded":"2024-09-10T18:57:54.060094+0000","last_fullsized":"2024-09-10T18:57:54.060094+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:56:44.683516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.683516+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:13.401562+0000","scrub_duration":0.0037708920000000001,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":282,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066820+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:57:54.066820+0000","last_peered":"2024-09-10T18:57:54.066820+0000","last_clean":"2024-09-10T18:57:54.066820+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:57:54.066820+0000","last_undegraded":"2024-09-10T18:57:54.066820+0000","last_fullsized":"2024-09-10T18:57:54.066820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":297,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090810+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:57:54.090810+0000","last_peered":"2024-09-10T18:57:54.090810+0000","last_clean":"2024-09-10T18:57:54.090810+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:57:54.090810+0000","last_undegraded":"2024-09-10T18:57:54.090810+0000","last_fullsized":"2024-09-10T18:57:54.090810+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":301,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061921+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:57:54.061921+0000","last_peered":"2024-09-10T18:57:54.061921+0000","last_clean":"2024-09-10T18:57:54.061921+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:57:54.061921+0000","last_undegraded":"2024-09-10T18:57:54.061921+0000","last_fullsized":"2024-09-10T18:57:54.061921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059665+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:57:54.059665+0000","last_peered":"2024-09-10T18:57:54.059665+0000","last_clean":"2024-09-10T18:57:54.059665+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:57:54.059665+0000","last_undegraded":"2024-09-10T18:57:54.059665+0000","last_fullsized":"2024-09-10T18:57:54.059665+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.070467+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:57:54.070467+0000","last_peered":"2024-09-10T18:57:54.070467+0000","last_clean":"2024-09-10T18:57:54.070467+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:57:54.070467+0000","last_undegraded":"2024-09-10T18:57:54.070467+0000","last_fullsized":"2024-09-10T18:57:54.070467+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"243'6","reported_seq":79,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090754+0000","last_change":"2024-09-10T18:57:27.072912+0000","last_active":"2024-09-10T18:57:54.090754+0000","last_peered":"2024-09-10T18:57:54.090754+0000","last_clean":"2024-09-10T18:57:54.090754+0000","last_became_active":"2024-09-10T18:57:27.072565+0000","last_became_peered":"2024-09-10T18:57:27.072565+0000","last_unstale":"2024-09-10T18:57:54.090754+0000","last_undegraded":"2024-09-10T18:57:54.090754+0000","last_fullsized":"2024-09-10T18:57:54.090754+0000","mapping_epoch":274,"log_start":"243'6","ondisk_log_start":"243'6","created":124,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'6","last_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_deep_scrub":"243'6","last_deep_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.791846+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":13,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:57.446964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.062453+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:57:54.062453+0000","last_peered":"2024-09-10T18:57:54.062453+0000","last_clean":"2024-09-10T18:57:54.062453+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:57:54.062453+0000","last_undegraded":"2024-09-10T18:57:54.062453+0000","last_fullsized":"2024-09-10T18:57:54.062453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":314,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090582+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:57:54.090582+0000","last_peered":"2024-09-10T18:57:54.090582+0000","last_clean":"2024-09-10T18:57:54.090582+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:57:54.090582+0000","last_undegraded":"2024-09-10T18:57:54.090582+0000","last_fullsized":"2024-09-10T18:57:54.090582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":304,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060495+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:57:54.060495+0000","last_peered":"2024-09-10T18:57:54.060495+0000","last_clean":"2024-09-10T18:57:54.060495+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:57:54.060495+0000","last_undegraded":"2024-09-10T18:57:54.060495+0000","last_fullsized":"2024-09-10T18:57:54.060495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.405877+0000","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064097+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:57:54.064097+0000","last_peered":"2024-09-10T18:57:54.064097+0000","last_clean":"2024-09-10T18:57:54.064097+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:57:54.064097+0000","last_undegraded":"2024-09-10T18:57:54.064097+0000","last_fullsized":"2024-09-10T18:57:54.064097+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088413+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T18:57:54.088413+0000","last_peered":"2024-09-10T18:57:54.088413+0000","last_clean":"2024-09-10T18:57:54.088413+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:57:54.088413+0000","last_undegraded":"2024-09-10T18:57:54.088413+0000","last_fullsized":"2024-09-10T18:57:54.088413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.279016+0000","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":74,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059381+0000","last_change":"2024-09-10T18:57:31.113969+0000","last_active":"2024-09-10T18:57:54.059381+0000","last_peered":"2024-09-10T18:57:54.059381+0000","last_clean":"2024-09-10T18:57:54.059381+0000","last_became_active":"2024-09-10T18:57:31.113373+0000","last_became_peered":"2024-09-10T18:57:31.113373+0000","last_unstale":"2024-09-10T18:57:54.059381+0000","last_undegraded":"2024-09-10T18:57:54.059381+0000","last_fullsized":"2024-09-10T18:57:54.059381+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.362084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.362084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.259984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054453+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:57:54.054453+0000","last_peered":"2024-09-10T18:57:54.054453+0000","last_clean":"2024-09-10T18:57:54.054453+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:57:54.054453+0000","last_undegraded":"2024-09-10T18:57:54.054453+0000","last_fullsized":"2024-09-10T18:57:54.054453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":325,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072336+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T18:57:54.072336+0000","last_peered":"2024-09-10T18:57:54.072336+0000","last_clean":"2024-09-10T18:57:54.072336+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:57:54.072336+0000","last_undegraded":"2024-09-10T18:57:54.072336+0000","last_fullsized":"2024-09-10T18:57:54.072336+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:42.997043+0000","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064005+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:57:54.064005+0000","last_peered":"2024-09-10T18:57:54.064005+0000","last_clean":"2024-09-10T18:57:54.064005+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:57:54.064005+0000","last_undegraded":"2024-09-10T18:57:54.064005+0000","last_fullsized":"2024-09-10T18:57:54.064005+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.070722+0000","last_change":"2024-09-10T18:57:31.277231+0000","last_active":"2024-09-10T18:57:54.070722+0000","last_peered":"2024-09-10T18:57:54.070722+0000","last_clean":"2024-09-10T18:57:54.070722+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:57:54.070722+0000","last_undegraded":"2024-09-10T18:57:54.070722+0000","last_fullsized":"2024-09-10T18:57:54.070722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.277085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.277085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:39.184999+0000","scrub_duration":0.003344746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":211,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.096830+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:57:54.096830+0000","last_peered":"2024-09-10T18:57:54.096830+0000","last_clean":"2024-09-10T18:57:54.096830+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:57:54.096830+0000","last_undegraded":"2024-09-10T18:57:54.096830+0000","last_fullsized":"2024-09-10T18:57:54.096830+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"237'6","reported_seq":84,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061319+0000","last_change":"2024-09-10T18:57:29.105121+0000","last_active":"2024-09-10T18:57:54.061319+0000","last_peered":"2024-09-10T18:57:54.061319+0000","last_clean":"2024-09-10T18:57:54.061319+0000","last_became_active":"2024-09-10T18:57:19.076695+0000","last_became_peered":"2024-09-10T18:57:19.076695+0000","last_unstale":"2024-09-10T18:57:54.061319+0000","last_undegraded":"2024-09-10T18:57:54.061319+0000","last_fullsized":"2024-09-10T18:57:54.061319+0000","mapping_epoch":274,"log_start":"237'6","ondisk_log_start":"237'6","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'6","last_scrub_stamp":"2024-09-10T18:57:29.105052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.105052+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":6,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.677376+0000","scrub_duration":0.0020046370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":297,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.057393+0000","last_change":"2024-09-10T18:57:25.517299+0000","last_active":"2024-09-10T18:57:54.057393+0000","last_peered":"2024-09-10T18:57:54.057393+0000","last_clean":"2024-09-10T18:57:54.057393+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:57:54.057393+0000","last_undegraded":"2024-09-10T18:57:54.057393+0000","last_fullsized":"2024-09-10T18:57:54.057393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:25.517210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.517210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:26.377413+0000","scrub_duration":0.0027489659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":331,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.073938+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:57:54.073938+0000","last_peered":"2024-09-10T18:57:54.073938+0000","last_clean":"2024-09-10T18:57:54.073938+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:57:54.073938+0000","last_undegraded":"2024-09-10T18:57:54.073938+0000","last_fullsized":"2024-09-10T18:57:54.073938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":304,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064608+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T18:57:54.064608+0000","last_peered":"2024-09-10T18:57:54.064608+0000","last_clean":"2024-09-10T18:57:54.064608+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:57:54.064608+0000","last_undegraded":"2024-09-10T18:57:54.064608+0000","last_fullsized":"2024-09-10T18:57:54.064608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.804733+0000","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087608+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:57:54.087608+0000","last_peered":"2024-09-10T18:57:54.087608+0000","last_clean":"2024-09-10T18:57:54.087608+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:57:54.087608+0000","last_undegraded":"2024-09-10T18:57:54.087608+0000","last_fullsized":"2024-09-10T18:57:54.087608+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058297+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:57:54.058297+0000","last_peered":"2024-09-10T18:57:54.058297+0000","last_clean":"2024-09-10T18:57:54.058297+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:57:54.058297+0000","last_undegraded":"2024-09-10T18:57:54.058297+0000","last_fullsized":"2024-09-10T18:57:54.058297+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":74,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.074200+0000","last_change":"2024-09-10T18:57:23.457770+0000","last_active":"2024-09-10T18:57:54.074200+0000","last_peered":"2024-09-10T18:57:54.074200+0000","last_clean":"2024-09-10T18:57:54.074200+0000","last_became_active":"2024-09-10T18:57:23.457438+0000","last_became_peered":"2024-09-10T18:57:23.457438+0000","last_unstale":"2024-09-10T18:57:54.074200+0000","last_undegraded":"2024-09-10T18:57:54.074200+0000","last_fullsized":"2024-09-10T18:57:54.074200+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":335,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-10T18:57:55.388043+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T18:57:55.388043+0000","last_peered":"2024-09-10T18:57:55.388043+0000","last_clean":"2024-09-10T18:57:55.388043+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:57:55.388043+0000","last_undegraded":"2024-09-10T18:57:55.388043+0000","last_fullsized":"2024-09-10T18:57:55.388043+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.716139+0000","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054801+0000","last_change":"2024-09-10T18:57:34.379150+0000","last_active":"2024-09-10T18:57:54.054801+0000","last_peered":"2024-09-10T18:57:54.054801+0000","last_clean":"2024-09-10T18:57:54.054801+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:57:54.054801+0000","last_undegraded":"2024-09-10T18:57:54.054801+0000","last_fullsized":"2024-09-10T18:57:54.054801+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.378985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.378985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.427844+0000","scrub_duration":0.00364804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":408,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065305+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:57:54.065305+0000","last_peered":"2024-09-10T18:57:54.065305+0000","last_clean":"2024-09-10T18:57:54.065305+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:57:54.065305+0000","last_undegraded":"2024-09-10T18:57:54.065305+0000","last_fullsized":"2024-09-10T18:57:54.065305+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":300,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067267+0000","last_change":"2024-09-10T18:57:36.840129+0000","last_active":"2024-09-10T18:57:54.067267+0000","last_peered":"2024-09-10T18:57:54.067267+0000","last_clean":"2024-09-10T18:57:54.067267+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:57:54.067267+0000","last_undegraded":"2024-09-10T18:57:54.067267+0000","last_fullsized":"2024-09-10T18:57:54.067267+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.839979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.839979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:44.823438+0000","scrub_duration":0.0034210669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067266+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:57:54.067266+0000","last_peered":"2024-09-10T18:57:54.067266+0000","last_clean":"2024-09-10T18:57:54.067266+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:57:54.067266+0000","last_undegraded":"2024-09-10T18:57:54.067266+0000","last_fullsized":"2024-09-10T18:57:54.067266+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.697752+0000","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":316,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064801+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:57:54.064801+0000","last_peered":"2024-09-10T18:57:54.064801+0000","last_clean":"2024-09-10T18:57:54.064801+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:57:54.064801+0000","last_undegraded":"2024-09-10T18:57:54.064801+0000","last_fullsized":"2024-09-10T18:57:54.064801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":305,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.096497+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:57:54.096497+0000","last_peered":"2024-09-10T18:57:54.096497+0000","last_clean":"2024-09-10T18:57:54.096497+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:57:54.096497+0000","last_undegraded":"2024-09-10T18:57:54.096497+0000","last_fullsized":"2024-09-10T18:57:54.096497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":304,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063899+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:57:54.063899+0000","last_peered":"2024-09-10T18:57:54.063899+0000","last_clean":"2024-09-10T18:57:54.063899+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:57:54.063899+0000","last_undegraded":"2024-09-10T18:57:54.063899+0000","last_fullsized":"2024-09-10T18:57:54.063899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:04.670897+0000","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064532+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:57:54.064532+0000","last_peered":"2024-09-10T18:57:54.064532+0000","last_clean":"2024-09-10T18:57:54.064532+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:57:54.064532+0000","last_undegraded":"2024-09-10T18:57:54.064532+0000","last_fullsized":"2024-09-10T18:57:54.064532+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067417+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T18:57:54.067417+0000","last_peered":"2024-09-10T18:57:54.067417+0000","last_clean":"2024-09-10T18:57:54.067417+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:57:54.067417+0000","last_undegraded":"2024-09-10T18:57:54.067417+0000","last_fullsized":"2024-09-10T18:57:54.067417+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.592874+0000","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":321,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058818+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:57:54.058818+0000","last_peered":"2024-09-10T18:57:54.058818+0000","last_clean":"2024-09-10T18:57:54.058818+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:57:54.058818+0000","last_undegraded":"2024-09-10T18:57:54.058818+0000","last_fullsized":"2024-09-10T18:57:54.058818+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.096621+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:57:54.096621+0000","last_peered":"2024-09-10T18:57:54.096621+0000","last_clean":"2024-09-10T18:57:54.096621+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:57:54.096621+0000","last_undegraded":"2024-09-10T18:57:54.096621+0000","last_fullsized":"2024-09-10T18:57:54.096621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":301,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.094613+0000","last_change":"2024-09-10T18:56:39.629880+0000","last_active":"2024-09-10T18:57:54.094613+0000","last_peered":"2024-09-10T18:57:54.094613+0000","last_clean":"2024-09-10T18:57:54.094613+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:57:54.094613+0000","last_undegraded":"2024-09-10T18:57:54.094613+0000","last_fullsized":"2024-09-10T18:57:54.094613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:39.629726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.629726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034450290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":299,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.073642+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:57:54.073642+0000","last_peered":"2024-09-10T18:57:54.073642+0000","last_clean":"2024-09-10T18:57:54.073642+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:57:54.073642+0000","last_undegraded":"2024-09-10T18:57:54.073642+0000","last_fullsized":"2024-09-10T18:57:54.073642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":211,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.098867+0000","last_change":"2024-09-10T18:56:17.674109+0000","last_active":"2024-09-10T18:57:54.098867+0000","last_peered":"2024-09-10T18:57:54.098867+0000","last_clean":"2024-09-10T18:57:54.098867+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:57:54.098867+0000","last_undegraded":"2024-09-10T18:57:54.098867+0000","last_fullsized":"2024-09-10T18:57:54.098867+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.673943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.673943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003315691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":334,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063388+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:57:54.063388+0000","last_peered":"2024-09-10T18:57:54.063388+0000","last_clean":"2024-09-10T18:57:54.063388+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:57:54.063388+0000","last_undegraded":"2024-09-10T18:57:54.063388+0000","last_fullsized":"2024-09-10T18:57:54.063388+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.053558+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:57:54.053558+0000","last_peered":"2024-09-10T18:57:54.053558+0000","last_clean":"2024-09-10T18:57:54.053558+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:57:54.053558+0000","last_undegraded":"2024-09-10T18:57:54.053558+0000","last_fullsized":"2024-09-10T18:57:54.053558+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":418,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064579+0000","last_change":"2024-09-10T18:57:32.764761+0000","last_active":"2024-09-10T18:57:54.064579+0000","last_peered":"2024-09-10T18:57:54.064579+0000","last_clean":"2024-09-10T18:57:54.064579+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:57:54.064579+0000","last_undegraded":"2024-09-10T18:57:54.064579+0000","last_fullsized":"2024-09-10T18:57:54.064579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:57:32.764561+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.764561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:41.529379+0000","scrub_duration":0.0041273020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":293,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059806+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:57:54.059806+0000","last_peered":"2024-09-10T18:57:54.059806+0000","last_clean":"2024-09-10T18:57:54.059806+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:57:54.059806+0000","last_undegraded":"2024-09-10T18:57:54.059806+0000","last_fullsized":"2024-09-10T18:57:54.059806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:55.519413+0000","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":211,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097818+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:57:54.097818+0000","last_peered":"2024-09-10T18:57:54.097818+0000","last_clean":"2024-09-10T18:57:54.097818+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:57:54.097818+0000","last_undegraded":"2024-09-10T18:57:54.097818+0000","last_fullsized":"2024-09-10T18:57:54.097818+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":313,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090427+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T18:57:54.090427+0000","last_peered":"2024-09-10T18:57:54.090427+0000","last_clean":"2024-09-10T18:57:54.090427+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:57:54.090427+0000","last_undegraded":"2024-09-10T18:57:54.090427+0000","last_fullsized":"2024-09-10T18:57:54.090427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.981000+0000","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":296,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068134+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:57:54.068134+0000","last_peered":"2024-09-10T18:57:54.068134+0000","last_clean":"2024-09-10T18:57:54.068134+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:57:54.068134+0000","last_undegraded":"2024-09-10T18:57:54.068134+0000","last_fullsized":"2024-09-10T18:57:54.068134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":304,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061408+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T18:57:54.061408+0000","last_peered":"2024-09-10T18:57:54.061408+0000","last_clean":"2024-09-10T18:57:54.061408+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:57:54.061408+0000","last_undegraded":"2024-09-10T18:57:54.061408+0000","last_fullsized":"2024-09-10T18:57:54.061408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.482905+0000","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":794,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.766307+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:57:54.766307+0000","last_peered":"2024-09-10T18:57:54.766307+0000","last_clean":"2024-09-10T18:57:54.766307+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:57:54.766307+0000","last_undegraded":"2024-09-10T18:57:54.766307+0000","last_fullsized":"2024-09-10T18:57:54.766307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":492,"num_read_kb":487,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054193+0000","last_change":"2024-09-10T18:57:49.386576+0000","last_active":"2024-09-10T18:57:54.054193+0000","last_peered":"2024-09-10T18:57:54.054193+0000","last_clean":"2024-09-10T18:57:54.054193+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:57:54.054193+0000","last_undegraded":"2024-09-10T18:57:54.054193+0000","last_fullsized":"2024-09-10T18:57:54.054193+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.386404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.605455+0000","scrub_duration":0.003967628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":222,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061830+0000","last_change":"2024-09-10T18:57:49.919967+0000","last_active":"2024-09-10T18:57:54.061830+0000","last_peered":"2024-09-10T18:57:54.061830+0000","last_clean":"2024-09-10T18:57:54.061830+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:57:54.061830+0000","last_undegraded":"2024-09-10T18:57:54.061830+0000","last_fullsized":"2024-09-10T18:57:54.061830+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.919794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.919794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.849294+0000","scrub_duration":0.0034551590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":323,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063477+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:57:54.063477+0000","last_peered":"2024-09-10T18:57:54.063477+0000","last_clean":"2024-09-10T18:57:54.063477+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:57:54.063477+0000","last_undegraded":"2024-09-10T18:57:54.063477+0000","last_fullsized":"2024-09-10T18:57:54.063477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":305,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.098173+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:57:54.098173+0000","last_peered":"2024-09-10T18:57:54.098173+0000","last_clean":"2024-09-10T18:57:54.098173+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:57:54.098173+0000","last_undegraded":"2024-09-10T18:57:54.098173+0000","last_fullsized":"2024-09-10T18:57:54.098173+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:59.107812+0000","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":397,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058309+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:57:54.058309+0000","last_peered":"2024-09-10T18:57:54.058309+0000","last_clean":"2024-09-10T18:57:54.058309+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:57:54.058309+0000","last_undegraded":"2024-09-10T18:57:54.058309+0000","last_fullsized":"2024-09-10T18:57:54.058309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":311,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061309+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:57:54.061309+0000","last_peered":"2024-09-10T18:57:54.061309+0000","last_clean":"2024-09-10T18:57:54.061309+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:57:54.061309+0000","last_undegraded":"2024-09-10T18:57:54.061309+0000","last_fullsized":"2024-09-10T18:57:54.061309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:00.852108+0000","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065056+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T18:57:54.065056+0000","last_peered":"2024-09-10T18:57:54.065056+0000","last_clean":"2024-09-10T18:57:54.065056+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:57:54.065056+0000","last_undegraded":"2024-09-10T18:57:54.065056+0000","last_fullsized":"2024-09-10T18:57:54.065056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:19.820859+0000","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058102+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T18:57:54.058102+0000","last_peered":"2024-09-10T18:57:54.058102+0000","last_clean":"2024-09-10T18:57:54.058102+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:57:54.058102+0000","last_undegraded":"2024-09-10T18:57:54.058102+0000","last_fullsized":"2024-09-10T18:57:54.058102+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.921143+0000","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"126'2","reported_seq":70,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060655+0000","last_change":"2024-09-10T18:57:11.104522+0000","last_active":"2024-09-10T18:57:54.060655+0000","last_peered":"2024-09-10T18:57:54.060655+0000","last_clean":"2024-09-10T18:57:54.060655+0000","last_became_active":"2024-09-10T18:57:11.092794+0000","last_became_peered":"2024-09-10T18:57:11.092794+0000","last_unstale":"2024-09-10T18:57:54.060655+0000","last_undegraded":"2024-09-10T18:57:54.060655+0000","last_fullsized":"2024-09-10T18:57:54.060655+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:39.169459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.169459+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":466,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097252+0000","last_change":"2024-09-10T18:57:50.752463+0000","last_active":"2024-09-10T18:57:54.097252+0000","last_peered":"2024-09-10T18:57:54.097252+0000","last_clean":"2024-09-10T18:57:54.097252+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:57:54.097252+0000","last_undegraded":"2024-09-10T18:57:54.097252+0000","last_fullsized":"2024-09-10T18:57:54.097252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:57:50.752292+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.752292+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.294597+0000","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.055452+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:57:54.055452+0000","last_peered":"2024-09-10T18:57:54.055452+0000","last_clean":"2024-09-10T18:57:54.055452+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:57:54.055452+0000","last_undegraded":"2024-09-10T18:57:54.055452+0000","last_fullsized":"2024-09-10T18:57:54.055452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":407,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087330+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:57:54.087330+0000","last_peered":"2024-09-10T18:57:54.087330+0000","last_clean":"2024-09-10T18:57:54.087330+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:57:54.087330+0000","last_undegraded":"2024-09-10T18:57:54.087330+0000","last_fullsized":"2024-09-10T18:57:54.087330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":300,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066322+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:57:54.066322+0000","last_peered":"2024-09-10T18:57:54.066322+0000","last_clean":"2024-09-10T18:57:54.066322+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:57:54.066322+0000","last_undegraded":"2024-09-10T18:57:54.066322+0000","last_fullsized":"2024-09-10T18:57:54.066322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":289,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.095980+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:57:54.095980+0000","last_peered":"2024-09-10T18:57:54.095980+0000","last_clean":"2024-09-10T18:57:54.095980+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:57:54.095980+0000","last_undegraded":"2024-09-10T18:57:54.095980+0000","last_fullsized":"2024-09-10T18:57:54.095980+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065069+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T18:57:54.065069+0000","last_peered":"2024-09-10T18:57:54.065069+0000","last_clean":"2024-09-10T18:57:54.065069+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:57:54.065069+0000","last_undegraded":"2024-09-10T18:57:54.065069+0000","last_fullsized":"2024-09-10T18:57:54.065069+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.040014+0000","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":315,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.074036+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:57:54.074036+0000","last_peered":"2024-09-10T18:57:54.074036+0000","last_clean":"2024-09-10T18:57:54.074036+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:57:54.074036+0000","last_undegraded":"2024-09-10T18:57:54.074036+0000","last_fullsized":"2024-09-10T18:57:54.074036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097341+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:57:54.097341+0000","last_peered":"2024-09-10T18:57:54.097341+0000","last_clean":"2024-09-10T18:57:54.097341+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:57:54.097341+0000","last_undegraded":"2024-09-10T18:57:54.097341+0000","last_fullsized":"2024-09-10T18:57:54.097341+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.093640+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:57:54.093640+0000","last_peered":"2024-09-10T18:57:54.093640+0000","last_clean":"2024-09-10T18:57:54.093640+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:57:54.093640+0000","last_undegraded":"2024-09-10T18:57:54.093640+0000","last_fullsized":"2024-09-10T18:57:54.093640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088138+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:57:54.088138+0000","last_peered":"2024-09-10T18:57:54.088138+0000","last_clean":"2024-09-10T18:57:54.088138+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:57:54.088138+0000","last_undegraded":"2024-09-10T18:57:54.088138+0000","last_fullsized":"2024-09-10T18:57:54.088138+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":222,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059999+0000","last_change":"2024-09-10T18:57:43.914840+0000","last_active":"2024-09-10T18:57:54.059999+0000","last_peered":"2024-09-10T18:57:54.059999+0000","last_clean":"2024-09-10T18:57:54.059999+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:57:54.059999+0000","last_undegraded":"2024-09-10T18:57:54.059999+0000","last_fullsized":"2024-09-10T18:57:54.059999+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:43.914731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:43.914731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.644461+0000","scrub_duration":0.0026766530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":342,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.670392+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T18:57:54.670392+0000","last_peered":"2024-09-10T18:57:54.670392+0000","last_clean":"2024-09-10T18:57:54.670392+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:57:54.670392+0000","last_undegraded":"2024-09-10T18:57:54.670392+0000","last_fullsized":"2024-09-10T18:57:54.670392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:08.776283+0000","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072256+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:57:54.072256+0000","last_peered":"2024-09-10T18:57:54.072256+0000","last_clean":"2024-09-10T18:57:54.072256+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:57:54.072256+0000","last_undegraded":"2024-09-10T18:57:54.072256+0000","last_fullsized":"2024-09-10T18:57:54.072256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":301,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.093810+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T18:57:54.093810+0000","last_peered":"2024-09-10T18:57:54.093810+0000","last_clean":"2024-09-10T18:57:54.093810+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:57:54.093810+0000","last_undegraded":"2024-09-10T18:57:54.093810+0000","last_fullsized":"2024-09-10T18:57:54.093810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.242393+0000","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":282,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061820+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:57:54.061820+0000","last_peered":"2024-09-10T18:57:54.061820+0000","last_clean":"2024-09-10T18:57:54.061820+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:57:54.061820+0000","last_undegraded":"2024-09-10T18:57:54.061820+0000","last_fullsized":"2024-09-10T18:57:54.061820+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.089234+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:57:54.089234+0000","last_peered":"2024-09-10T18:57:54.089234+0000","last_clean":"2024-09-10T18:57:54.089234+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:57:54.089234+0000","last_undegraded":"2024-09-10T18:57:54.089234+0000","last_fullsized":"2024-09-10T18:57:54.089234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":57,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087433+0000","last_change":"2024-09-10T18:57:11.091444+0000","last_active":"2024-09-10T18:57:54.087433+0000","last_peered":"2024-09-10T18:57:54.087433+0000","last_clean":"2024-09-10T18:57:54.087433+0000","last_became_active":"2024-09-10T18:57:11.091111+0000","last_became_peered":"2024-09-10T18:57:11.091111+0000","last_unstale":"2024-09-10T18:57:54.087433+0000","last_undegraded":"2024-09-10T18:57:54.087433+0000","last_fullsized":"2024-09-10T18:57:54.087433+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.795811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.795811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.217254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":343,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.094949+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T18:57:54.094949+0000","last_peered":"2024-09-10T18:57:54.094949+0000","last_clean":"2024-09-10T18:57:54.094949+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:57:54.094949+0000","last_undegraded":"2024-09-10T18:57:54.094949+0000","last_fullsized":"2024-09-10T18:57:54.094949+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:05.333581+0000","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.061554+0000","last_change":"2024-09-10T18:56:15.655135+0000","last_active":"2024-09-10T18:57:54.061554+0000","last_peered":"2024-09-10T18:57:54.061554+0000","last_clean":"2024-09-10T18:57:54.061554+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:57:54.061554+0000","last_undegraded":"2024-09-10T18:57:54.061554+0000","last_fullsized":"2024-09-10T18:57:54.061554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:15.655055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.655055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020254439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.098977+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:57:54.098977+0000","last_peered":"2024-09-10T18:57:54.098977+0000","last_clean":"2024-09-10T18:57:54.098977+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:57:54.098977+0000","last_undegraded":"2024-09-10T18:57:54.098977+0000","last_fullsized":"2024-09-10T18:57:54.098977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054716+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T18:57:54.054716+0000","last_peered":"2024-09-10T18:57:54.054716+0000","last_clean":"2024-09-10T18:57:54.054716+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:57:54.054716+0000","last_undegraded":"2024-09-10T18:57:54.054716+0000","last_fullsized":"2024-09-10T18:57:54.054716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.701730+0000","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067920+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:57:54.067920+0000","last_peered":"2024-09-10T18:57:54.067920+0000","last_clean":"2024-09-10T18:57:54.067920+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:57:54.067920+0000","last_undegraded":"2024-09-10T18:57:54.067920+0000","last_fullsized":"2024-09-10T18:57:54.067920+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"126'9","reported_seq":68,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071407+0000","last_change":"2024-09-10T18:57:11.102250+0000","last_active":"2024-09-10T18:57:54.071407+0000","last_peered":"2024-09-10T18:57:54.071407+0000","last_clean":"2024-09-10T18:57:54.071407+0000","last_became_active":"2024-09-10T18:57:11.081753+0000","last_became_peered":"2024-09-10T18:57:11.081753+0000","last_unstale":"2024-09-10T18:57:54.071407+0000","last_undegraded":"2024-09-10T18:57:54.071407+0000","last_fullsized":"2024-09-10T18:57:54.071407+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:15.650681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.650681+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":316,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087669+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:57:54.087669+0000","last_peered":"2024-09-10T18:57:54.087669+0000","last_clean":"2024-09-10T18:57:54.087669+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:57:54.087669+0000","last_undegraded":"2024-09-10T18:57:54.087669+0000","last_fullsized":"2024-09-10T18:57:54.087669+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.053201+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:57:54.053201+0000","last_peered":"2024-09-10T18:57:54.053201+0000","last_clean":"2024-09-10T18:57:54.053201+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:57:54.053201+0000","last_undegraded":"2024-09-10T18:57:54.053201+0000","last_fullsized":"2024-09-10T18:57:54.053201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":293,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.086641+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:57:54.086641+0000","last_peered":"2024-09-10T18:57:54.086641+0000","last_clean":"2024-09-10T18:57:54.086641+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:57:54.086641+0000","last_undegraded":"2024-09-10T18:57:54.086641+0000","last_fullsized":"2024-09-10T18:57:54.086641+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":299,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097061+0000","last_change":"2024-09-10T18:57:44.612113+0000","last_active":"2024-09-10T18:57:54.097061+0000","last_peered":"2024-09-10T18:57:54.097061+0000","last_clean":"2024-09-10T18:57:54.097061+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:57:54.097061+0000","last_undegraded":"2024-09-10T18:57:54.097061+0000","last_fullsized":"2024-09-10T18:57:54.097061+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:44.612043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:57:44.612043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.176921+0000","scrub_duration":0.0026996279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090537+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:57:54.090537+0000","last_peered":"2024-09-10T18:57:54.090537+0000","last_clean":"2024-09-10T18:57:54.090537+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:57:54.090537+0000","last_undegraded":"2024-09-10T18:57:54.090537+0000","last_fullsized":"2024-09-10T18:57:54.090537+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":64,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063526+0000","last_change":"2024-09-10T18:57:29.701100+0000","last_active":"2024-09-10T18:57:54.063526+0000","last_peered":"2024-09-10T18:57:54.063526+0000","last_clean":"2024-09-10T18:57:54.063526+0000","last_became_active":"2024-09-10T18:57:11.085788+0000","last_became_peered":"2024-09-10T18:57:11.085788+0000","last_unstale":"2024-09-10T18:57:54.063526+0000","last_undegraded":"2024-09-10T18:57:54.063526+0000","last_fullsized":"2024-09-10T18:57:54.063526+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:29.700929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.700929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:49.024524+0000","scrub_duration":0.003421678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058471+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:57:54.058471+0000","last_peered":"2024-09-10T18:57:54.058471+0000","last_clean":"2024-09-10T18:57:54.058471+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:57:54.058471+0000","last_undegraded":"2024-09-10T18:57:54.058471+0000","last_fullsized":"2024-09-10T18:57:54.058471+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071678+0000","last_change":"2024-09-10T18:55:03.233057+0000","last_active":"2024-09-10T18:57:54.071678+0000","last_peered":"2024-09-10T18:57:54.071678+0000","last_clean":"2024-09-10T18:57:54.071678+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:57:54.071678+0000","last_undegraded":"2024-09-10T18:57:54.071678+0000","last_fullsized":"2024-09-10T18:57:54.071678+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:03.232923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:55:03.232923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036625059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":342,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.053280+0000","last_change":"2024-09-10T18:56:04.556751+0000","last_active":"2024-09-10T18:57:54.053280+0000","last_peered":"2024-09-10T18:57:54.053280+0000","last_clean":"2024-09-10T18:57:54.053280+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:57:54.053280+0000","last_undegraded":"2024-09-10T18:57:54.053280+0000","last_fullsized":"2024-09-10T18:57:54.053280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:56:04.556586+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.556586+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046952210000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067596+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:57:54.067596+0000","last_peered":"2024-09-10T18:57:54.067596+0000","last_clean":"2024-09-10T18:57:54.067596+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:57:54.067596+0000","last_undegraded":"2024-09-10T18:57:54.067596+0000","last_fullsized":"2024-09-10T18:57:54.067596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":289,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.076456+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:57:54.076456+0000","last_peered":"2024-09-10T18:57:54.076456+0000","last_clean":"2024-09-10T18:57:54.076456+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:57:54.076456+0000","last_undegraded":"2024-09-10T18:57:54.076456+0000","last_fullsized":"2024-09-10T18:57:54.076456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.055210+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:57:54.055210+0000","last_peered":"2024-09-10T18:57:54.055210+0000","last_clean":"2024-09-10T18:57:54.055210+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:57:54.055210+0000","last_undegraded":"2024-09-10T18:57:54.055210+0000","last_fullsized":"2024-09-10T18:57:54.055210+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.11","version":"310'8","reported_seq":73,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059807+0000","last_change":"2024-09-10T18:57:46.905106+0000","last_active":"2024-09-10T18:57:54.059807+0000","last_peered":"2024-09-10T18:57:54.059807+0000","last_clean":"2024-09-10T18:57:54.059807+0000","last_became_active":"2024-09-10T18:57:11.087842+0000","last_became_peered":"2024-09-10T18:57:11.087842+0000","last_unstale":"2024-09-10T18:57:54.059807+0000","last_undegraded":"2024-09-10T18:57:54.059807+0000","last_fullsized":"2024-09-10T18:57:54.059807+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'8","last_scrub_stamp":"2024-09-10T18:57:46.904909+0000","last_deep_scrub":"310'8","last_deep_scrub_stamp":"2024-09-10T18:57:46.904909+0000","last_clean_scrub_stamp":"2024-09-10T18:57:46.904909+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.741858+0000","scrub_duration":0.007209315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":320,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.055345+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:57:54.055345+0000","last_peered":"2024-09-10T18:57:54.055345+0000","last_clean":"2024-09-10T18:57:54.055345+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:57:54.055345+0000","last_undegraded":"2024-09-10T18:57:54.055345+0000","last_fullsized":"2024-09-10T18:57:54.055345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059904+0000","last_change":"2024-09-10T18:57:34.759089+0000","last_active":"2024-09-10T18:57:54.059904+0000","last_peered":"2024-09-10T18:57:54.059904+0000","last_clean":"2024-09-10T18:57:54.059904+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:57:54.059904+0000","last_undegraded":"2024-09-10T18:57:54.059904+0000","last_fullsized":"2024-09-10T18:57:54.059904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.758993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.758993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:36.134527+0000","scrub_duration":0.0028569620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066813+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:57:54.066813+0000","last_peered":"2024-09-10T18:57:54.066813+0000","last_clean":"2024-09-10T18:57:54.066813+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:57:54.066813+0000","last_undegraded":"2024-09-10T18:57:54.066813+0000","last_fullsized":"2024-09-10T18:57:54.066813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":288,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.053671+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:57:54.053671+0000","last_peered":"2024-09-10T18:57:54.053671+0000","last_clean":"2024-09-10T18:57:54.053671+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:57:54.053671+0000","last_undegraded":"2024-09-10T18:57:54.053671+0000","last_fullsized":"2024-09-10T18:57:54.053671+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064116+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:57:54.064116+0000","last_peered":"2024-09-10T18:57:54.064116+0000","last_clean":"2024-09-10T18:57:54.064116+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:57:54.064116+0000","last_undegraded":"2024-09-10T18:57:54.064116+0000","last_fullsized":"2024-09-10T18:57:54.064116+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":313,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072575+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:57:54.072575+0000","last_peered":"2024-09-10T18:57:54.072575+0000","last_clean":"2024-09-10T18:57:54.072575+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:57:54.072575+0000","last_undegraded":"2024-09-10T18:57:54.072575+0000","last_fullsized":"2024-09-10T18:57:54.072575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097960+0000","last_change":"2024-09-10T18:56:30.684729+0000","last_active":"2024-09-10T18:57:54.097960+0000","last_peered":"2024-09-10T18:57:54.097960+0000","last_clean":"2024-09-10T18:57:54.097960+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:57:54.097960+0000","last_undegraded":"2024-09-10T18:57:54.097960+0000","last_fullsized":"2024-09-10T18:57:54.097960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.684589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023375330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":283,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087180+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:57:54.087180+0000","last_peered":"2024-09-10T18:57:54.087180+0000","last_clean":"2024-09-10T18:57:54.087180+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:57:54.087180+0000","last_undegraded":"2024-09-10T18:57:54.087180+0000","last_fullsized":"2024-09-10T18:57:54.087180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":281,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087633+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:57:54.087633+0000","last_peered":"2024-09-10T18:57:54.087633+0000","last_clean":"2024-09-10T18:57:54.087633+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:57:54.087633+0000","last_undegraded":"2024-09-10T18:57:54.087633+0000","last_fullsized":"2024-09-10T18:57:54.087633+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088987+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:57:54.088987+0000","last_peered":"2024-09-10T18:57:54.088987+0000","last_clean":"2024-09-10T18:57:54.088987+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:57:54.088987+0000","last_undegraded":"2024-09-10T18:57:54.088987+0000","last_fullsized":"2024-09-10T18:57:54.088987+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.383629+0000","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":324,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.056212+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:57:54.056212+0000","last_peered":"2024-09-10T18:57:54.056212+0000","last_clean":"2024-09-10T18:57:54.056212+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:57:54.056212+0000","last_undegraded":"2024-09-10T18:57:54.056212+0000","last_fullsized":"2024-09-10T18:57:54.056212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":318,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066621+0000","last_change":"2024-09-10T18:56:36.828365+0000","last_active":"2024-09-10T18:57:54.066621+0000","last_peered":"2024-09-10T18:57:54.066621+0000","last_clean":"2024-09-10T18:57:54.066621+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:57:54.066621+0000","last_undegraded":"2024-09-10T18:57:54.066621+0000","last_fullsized":"2024-09-10T18:57:54.066621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:56:36.828207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.828207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038610699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":301,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.070612+0000","last_change":"2024-09-10T18:57:34.244420+0000","last_active":"2024-09-10T18:57:54.070612+0000","last_peered":"2024-09-10T18:57:54.070612+0000","last_clean":"2024-09-10T18:57:54.070612+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:57:54.070612+0000","last_undegraded":"2024-09-10T18:57:54.070612+0000","last_fullsized":"2024-09-10T18:57:54.070612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.244346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.244346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:36.724795+0000","scrub_duration":0.00228093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":289,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071909+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:57:54.071909+0000","last_peered":"2024-09-10T18:57:54.071909+0000","last_clean":"2024-09-10T18:57:54.071909+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:57:54.071909+0000","last_undegraded":"2024-09-10T18:57:54.071909+0000","last_fullsized":"2024-09-10T18:57:54.071909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060384+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:57:54.060384+0000","last_peered":"2024-09-10T18:57:54.060384+0000","last_clean":"2024-09-10T18:57:54.060384+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:57:54.060384+0000","last_undegraded":"2024-09-10T18:57:54.060384+0000","last_fullsized":"2024-09-10T18:57:54.060384+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":311,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.056552+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:57:54.056552+0000","last_peered":"2024-09-10T18:57:54.056552+0000","last_clean":"2024-09-10T18:57:54.056552+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:57:54.056552+0000","last_undegraded":"2024-09-10T18:57:54.056552+0000","last_fullsized":"2024-09-10T18:57:54.056552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":347,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065604+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:57:54.065604+0000","last_peered":"2024-09-10T18:57:54.065604+0000","last_clean":"2024-09-10T18:57:54.065604+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:57:54.065604+0000","last_undegraded":"2024-09-10T18:57:54.065604+0000","last_fullsized":"2024-09-10T18:57:54.065604+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.123931+0000","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":417,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.062545+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T18:57:54.062545+0000","last_peered":"2024-09-10T18:57:54.062545+0000","last_clean":"2024-09-10T18:57:54.062545+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:57:54.062545+0000","last_undegraded":"2024-09-10T18:57:54.062545+0000","last_fullsized":"2024-09-10T18:57:54.062545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.279543+0000","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064340+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T18:57:54.064340+0000","last_peered":"2024-09-10T18:57:54.064340+0000","last_clean":"2024-09-10T18:57:54.064340+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:57:54.064340+0000","last_undegraded":"2024-09-10T18:57:54.064340+0000","last_fullsized":"2024-09-10T18:57:54.064340+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.083086+0000","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":226,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065240+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:57:54.065240+0000","last_peered":"2024-09-10T18:57:54.065240+0000","last_clean":"2024-09-10T18:57:54.065240+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:57:54.065240+0000","last_undegraded":"2024-09-10T18:57:54.065240+0000","last_fullsized":"2024-09-10T18:57:54.065240+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.921825+0000","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":321,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.053881+0000","last_change":"2024-09-10T18:57:33.408816+0000","last_active":"2024-09-10T18:57:54.053881+0000","last_peered":"2024-09-10T18:57:54.053881+0000","last_clean":"2024-09-10T18:57:54.053881+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:57:54.053881+0000","last_undegraded":"2024-09-10T18:57:54.053881+0000","last_fullsized":"2024-09-10T18:57:54.053881+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:33.408655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:33.408655+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.644071+0000","scrub_duration":0.0039294170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058426+0000","last_change":"2024-09-10T18:57:39.808114+0000","last_active":"2024-09-10T18:57:54.058426+0000","last_peered":"2024-09-10T18:57:54.058426+0000","last_clean":"2024-09-10T18:57:54.058426+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:57:54.058426+0000","last_undegraded":"2024-09-10T18:57:54.058426+0000","last_fullsized":"2024-09-10T18:57:54.058426+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.807949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.807949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.276337+0000","scrub_duration":0.0034718420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":293,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087195+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:57:54.087195+0000","last_peered":"2024-09-10T18:57:54.087195+0000","last_clean":"2024-09-10T18:57:54.087195+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:57:54.087195+0000","last_undegraded":"2024-09-10T18:57:54.087195+0000","last_fullsized":"2024-09-10T18:57:54.087195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":300,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058642+0000","last_change":"2024-09-10T18:57:42.899838+0000","last_active":"2024-09-10T18:57:54.058642+0000","last_peered":"2024-09-10T18:57:54.058642+0000","last_clean":"2024-09-10T18:57:54.058642+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:57:54.058642+0000","last_undegraded":"2024-09-10T18:57:54.058642+0000","last_fullsized":"2024-09-10T18:57:54.058642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.899684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.899684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.010497+0000","scrub_duration":0.0036750730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065194+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:57:54.065194+0000","last_peered":"2024-09-10T18:57:54.065194+0000","last_clean":"2024-09-10T18:57:54.065194+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:57:54.065194+0000","last_undegraded":"2024-09-10T18:57:54.065194+0000","last_fullsized":"2024-09-10T18:57:54.065194+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063661+0000","last_change":"2024-09-10T18:56:32.796094+0000","last_active":"2024-09-10T18:57:54.063661+0000","last_peered":"2024-09-10T18:57:54.063661+0000","last_clean":"2024-09-10T18:57:54.063661+0000","last_became_active":"2024-09-10T18:54:37.566860+0000","last_became_peered":"2024-09-10T18:54:37.566860+0000","last_unstale":"2024-09-10T18:57:54.063661+0000","last_undegraded":"2024-09-10T18:57:54.063661+0000","last_fullsized":"2024-09-10T18:57:54.063661+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.795927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032690250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064491+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:57:54.064491+0000","last_peered":"2024-09-10T18:57:54.064491+0000","last_clean":"2024-09-10T18:57:54.064491+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:57:54.064491+0000","last_undegraded":"2024-09-10T18:57:54.064491+0000","last_fullsized":"2024-09-10T18:57:54.064491+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":306,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065824+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T18:57:54.065824+0000","last_peered":"2024-09-10T18:57:54.065824+0000","last_clean":"2024-09-10T18:57:54.065824+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:57:54.065824+0000","last_undegraded":"2024-09-10T18:57:54.065824+0000","last_fullsized":"2024-09-10T18:57:54.065824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.849115+0000","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":284,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063236+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:57:54.063236+0000","last_peered":"2024-09-10T18:57:54.063236+0000","last_clean":"2024-09-10T18:57:54.063236+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:57:54.063236+0000","last_undegraded":"2024-09-10T18:57:54.063236+0000","last_fullsized":"2024-09-10T18:57:54.063236+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054872+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T18:57:54.054872+0000","last_peered":"2024-09-10T18:57:54.054872+0000","last_clean":"2024-09-10T18:57:54.054872+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:57:54.054872+0000","last_undegraded":"2024-09-10T18:57:54.054872+0000","last_fullsized":"2024-09-10T18:57:54.054872+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:13.562040+0000","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067916+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:57:54.067916+0000","last_peered":"2024-09-10T18:57:54.067916+0000","last_clean":"2024-09-10T18:57:54.067916+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:57:54.067916+0000","last_undegraded":"2024-09-10T18:57:54.067916+0000","last_fullsized":"2024-09-10T18:57:54.067916+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"126'5","reported_seq":217,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071539+0000","last_change":"2024-09-10T18:56:37.282686+0000","last_active":"2024-09-10T18:57:54.071539+0000","last_peered":"2024-09-10T18:57:54.071539+0000","last_clean":"2024-09-10T18:57:54.071539+0000","last_became_active":"2024-09-10T18:54:37.507583+0000","last_became_peered":"2024-09-10T18:54:37.507583+0000","last_unstale":"2024-09-10T18:57:54.071539+0000","last_undegraded":"2024-09-10T18:57:54.071539+0000","last_fullsized":"2024-09-10T18:57:54.071539+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:37.282578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.282578+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.873328+0000","scrub_duration":0.0030887559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":316,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.054634+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:57:54.054634+0000","last_peered":"2024-09-10T18:57:54.054634+0000","last_clean":"2024-09-10T18:57:54.054634+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:57:54.054634+0000","last_undegraded":"2024-09-10T18:57:54.054634+0000","last_fullsized":"2024-09-10T18:57:54.054634+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.072358+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:57:54.072358+0000","last_peered":"2024-09-10T18:57:54.072358+0000","last_clean":"2024-09-10T18:57:54.072358+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:57:54.072358+0000","last_undegraded":"2024-09-10T18:57:54.072358+0000","last_fullsized":"2024-09-10T18:57:54.072358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.607986+0000","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":410,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060222+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T18:57:54.060222+0000","last_peered":"2024-09-10T18:57:54.060222+0000","last_clean":"2024-09-10T18:57:54.060222+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:57:54.060222+0000","last_undegraded":"2024-09-10T18:57:54.060222+0000","last_fullsized":"2024-09-10T18:57:54.060222+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:44.667849+0000","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":291,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090280+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:57:54.090280+0000","last_peered":"2024-09-10T18:57:54.090280+0000","last_clean":"2024-09-10T18:57:54.090280+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:57:54.090280+0000","last_undegraded":"2024-09-10T18:57:54.090280+0000","last_fullsized":"2024-09-10T18:57:54.090280+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.089565+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:57:54.089565+0000","last_peered":"2024-09-10T18:57:54.089565+0000","last_clean":"2024-09-10T18:57:54.089565+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:57:54.089565+0000","last_undegraded":"2024-09-10T18:57:54.089565+0000","last_fullsized":"2024-09-10T18:57:54.089565+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.533486+0000","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"126'5","reported_seq":217,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088506+0000","last_change":"2024-09-10T18:56:27.205415+0000","last_active":"2024-09-10T18:57:54.088506+0000","last_peered":"2024-09-10T18:57:54.088506+0000","last_clean":"2024-09-10T18:57:54.088506+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:57:54.088506+0000","last_undegraded":"2024-09-10T18:57:54.088506+0000","last_fullsized":"2024-09-10T18:57:54.088506+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.205258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.205258+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049140169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":311,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088728+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:57:54.088728+0000","last_peered":"2024-09-10T18:57:54.088728+0000","last_clean":"2024-09-10T18:57:54.088728+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:57:54.088728+0000","last_undegraded":"2024-09-10T18:57:54.088728+0000","last_fullsized":"2024-09-10T18:57:54.088728+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":297,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.089401+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:57:54.089401+0000","last_peered":"2024-09-10T18:57:54.089401+0000","last_clean":"2024-09-10T18:57:54.089401+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:57:54.089401+0000","last_undegraded":"2024-09-10T18:57:54.089401+0000","last_fullsized":"2024-09-10T18:57:54.089401+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":394,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.073849+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:57:54.073849+0000","last_peered":"2024-09-10T18:57:54.073849+0000","last_clean":"2024-09-10T18:57:54.073849+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:57:54.073849+0000","last_undegraded":"2024-09-10T18:57:54.073849+0000","last_fullsized":"2024-09-10T18:57:54.073849+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065111+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T18:57:54.065111+0000","last_peered":"2024-09-10T18:57:54.065111+0000","last_clean":"2024-09-10T18:57:54.065111+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:57:54.065111+0000","last_undegraded":"2024-09-10T18:57:54.065111+0000","last_fullsized":"2024-09-10T18:57:54.065111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.237854+0000","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":202,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.091026+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:57:54.091026+0000","last_peered":"2024-09-10T18:57:54.091026+0000","last_clean":"2024-09-10T18:57:54.091026+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:57:54.091026+0000","last_undegraded":"2024-09-10T18:57:54.091026+0000","last_fullsized":"2024-09-10T18:57:54.091026+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"126'5","reported_seq":217,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.062018+0000","last_change":"2024-09-10T18:56:39.805638+0000","last_active":"2024-09-10T18:57:54.062018+0000","last_peered":"2024-09-10T18:57:54.062018+0000","last_clean":"2024-09-10T18:57:54.062018+0000","last_became_active":"2024-09-10T18:54:37.610866+0000","last_became_peered":"2024-09-10T18:54:37.610866+0000","last_unstale":"2024-09-10T18:57:54.062018+0000","last_undegraded":"2024-09-10T18:57:54.062018+0000","last_fullsized":"2024-09-10T18:57:54.062018+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:39.805468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.805468+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.334135+0000","scrub_duration":0.0040878859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":308,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.055037+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:57:54.055037+0000","last_peered":"2024-09-10T18:57:54.055037+0000","last_clean":"2024-09-10T18:57:54.055037+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:57:54.055037+0000","last_undegraded":"2024-09-10T18:57:54.055037+0000","last_fullsized":"2024-09-10T18:57:54.055037+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":296,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067770+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:57:54.067770+0000","last_peered":"2024-09-10T18:57:54.067770+0000","last_clean":"2024-09-10T18:57:54.067770+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:57:54.067770+0000","last_undegraded":"2024-09-10T18:57:54.067770+0000","last_fullsized":"2024-09-10T18:57:54.067770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071536+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T18:57:54.071536+0000","last_peered":"2024-09-10T18:57:54.071536+0000","last_clean":"2024-09-10T18:57:54.071536+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:57:54.071536+0000","last_undegraded":"2024-09-10T18:57:54.071536+0000","last_fullsized":"2024-09-10T18:57:54.071536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.840825+0000","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087556+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:57:54.087556+0000","last_peered":"2024-09-10T18:57:54.087556+0000","last_clean":"2024-09-10T18:57:54.087556+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:57:54.087556+0000","last_undegraded":"2024-09-10T18:57:54.087556+0000","last_fullsized":"2024-09-10T18:57:54.087556+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066435+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:57:54.066435+0000","last_peered":"2024-09-10T18:57:54.066435+0000","last_clean":"2024-09-10T18:57:54.066435+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:57:54.066435+0000","last_undegraded":"2024-09-10T18:57:54.066435+0000","last_fullsized":"2024-09-10T18:57:54.066435+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"126'9","reported_seq":242,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-09-10T18:57:55.067344+0000","last_change":"2024-09-10T18:57:55.067344+0000","last_active":"2024-09-10T18:57:55.067344+0000","last_peered":"2024-09-10T18:57:55.067344+0000","last_clean":"2024-09-10T18:57:55.067344+0000","last_became_active":"2024-09-10T18:57:55.066835+0000","last_became_peered":"2024-09-10T18:57:55.066835+0000","last_unstale":"2024-09-10T18:57:55.067344+0000","last_undegraded":"2024-09-10T18:57:55.067344+0000","last_fullsized":"2024-09-10T18:57:55.067344+0000","mapping_epoch":124,"log_start":"126'9","ondisk_log_start":"126'9","created":124,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":14,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":326,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064122+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T18:57:54.064122+0000","last_peered":"2024-09-10T18:57:54.064122+0000","last_clean":"2024-09-10T18:57:54.064122+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:57:54.064122+0000","last_undegraded":"2024-09-10T18:57:54.064122+0000","last_fullsized":"2024-09-10T18:57:54.064122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.974665+0000","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":305,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.097867+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:57:54.097867+0000","last_peered":"2024-09-10T18:57:54.097867+0000","last_clean":"2024-09-10T18:57:54.097867+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:57:54.097867+0000","last_undegraded":"2024-09-10T18:57:54.097867+0000","last_fullsized":"2024-09-10T18:57:54.097867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.423650+0000","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":294,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064286+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:57:54.064286+0000","last_peered":"2024-09-10T18:57:54.064286+0000","last_clean":"2024-09-10T18:57:54.064286+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:57:54.064286+0000","last_undegraded":"2024-09-10T18:57:54.064286+0000","last_fullsized":"2024-09-10T18:57:54.064286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058563+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:57:54.058563+0000","last_peered":"2024-09-10T18:57:54.058563+0000","last_clean":"2024-09-10T18:57:54.058563+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:57:54.058563+0000","last_undegraded":"2024-09-10T18:57:54.058563+0000","last_fullsized":"2024-09-10T18:57:54.058563+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064120+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:57:54.064120+0000","last_peered":"2024-09-10T18:57:54.064120+0000","last_clean":"2024-09-10T18:57:54.064120+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:57:54.064120+0000","last_undegraded":"2024-09-10T18:57:54.064120+0000","last_fullsized":"2024-09-10T18:57:54.064120+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"126'5","reported_seq":233,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063578+0000","last_change":"2024-09-10T18:57:51.092320+0000","last_active":"2024-09-10T18:57:54.063578+0000","last_peered":"2024-09-10T18:57:54.063578+0000","last_clean":"2024-09-10T18:57:54.063578+0000","last_became_active":"2024-09-10T18:57:51.091993+0000","last_became_peered":"2024-09-10T18:57:51.091993+0000","last_unstale":"2024-09-10T18:57:54.063578+0000","last_undegraded":"2024-09-10T18:57:54.063578+0000","last_fullsized":"2024-09-10T18:57:54.063578+0000","mapping_epoch":124,"log_start":"126'5","ondisk_log_start":"126'5","created":124,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.160276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.160276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":5,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:58.768933+0000","scrub_duration":0.0035864650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":297,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.091093+0000","last_change":"2024-09-10T18:54:54.957426+0000","last_active":"2024-09-10T18:57:54.091093+0000","last_peered":"2024-09-10T18:57:54.091093+0000","last_clean":"2024-09-10T18:57:54.091093+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:57:54.091093+0000","last_undegraded":"2024-09-10T18:57:54.091093+0000","last_fullsized":"2024-09-10T18:57:54.091093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:54:54.957316+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:54:54.957316+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039293499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.076234+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:57:54.076234+0000","last_peered":"2024-09-10T18:57:54.076234+0000","last_clean":"2024-09-10T18:57:54.076234+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:57:54.076234+0000","last_undegraded":"2024-09-10T18:57:54.076234+0000","last_fullsized":"2024-09-10T18:57:54.076234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":284,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.057940+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:57:54.057940+0000","last_peered":"2024-09-10T18:57:54.057940+0000","last_clean":"2024-09-10T18:57:54.057940+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:57:54.057940+0000","last_undegraded":"2024-09-10T18:57:54.057940+0000","last_fullsized":"2024-09-10T18:57:54.057940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.055232+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T18:57:54.055232+0000","last_peered":"2024-09-10T18:57:54.055232+0000","last_clean":"2024-09-10T18:57:54.055232+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:57:54.055232+0000","last_undegraded":"2024-09-10T18:57:54.055232+0000","last_fullsized":"2024-09-10T18:57:54.055232+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.521548+0000","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":211,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.094955+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:57:54.094955+0000","last_peered":"2024-09-10T18:57:54.094955+0000","last_clean":"2024-09-10T18:57:54.094955+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:57:54.094955+0000","last_undegraded":"2024-09-10T18:57:54.094955+0000","last_fullsized":"2024-09-10T18:57:54.094955+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.479269+0000","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":206,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060536+0000","last_change":"2024-09-10T18:57:31.111871+0000","last_active":"2024-09-10T18:57:54.060536+0000","last_peered":"2024-09-10T18:57:54.060536+0000","last_clean":"2024-09-10T18:57:54.060536+0000","last_became_active":"2024-09-10T18:54:37.657896+0000","last_became_peered":"2024-09-10T18:54:37.657896+0000","last_unstale":"2024-09-10T18:57:54.060536+0000","last_undegraded":"2024-09-10T18:57:54.060536+0000","last_fullsized":"2024-09-10T18:57:54.060536+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":309,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.076098+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:57:54.076098+0000","last_peered":"2024-09-10T18:57:54.076098+0000","last_clean":"2024-09-10T18:57:54.076098+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:57:54.076098+0000","last_undegraded":"2024-09-10T18:57:54.076098+0000","last_fullsized":"2024-09-10T18:57:54.076098+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":296,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068136+0000","last_change":"2024-09-10T18:55:08.609833+0000","last_active":"2024-09-10T18:57:54.068136+0000","last_peered":"2024-09-10T18:57:54.068136+0000","last_clean":"2024-09-10T18:57:54.068136+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:57:54.068136+0000","last_undegraded":"2024-09-10T18:57:54.068136+0000","last_fullsized":"2024-09-10T18:57:54.068136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.609702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.609702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":284,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058781+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:57:54.058781+0000","last_peered":"2024-09-10T18:57:54.058781+0000","last_clean":"2024-09-10T18:57:54.058781+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:57:54.058781+0000","last_undegraded":"2024-09-10T18:57:54.058781+0000","last_fullsized":"2024-09-10T18:57:54.058781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064498+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:57:54.064498+0000","last_peered":"2024-09-10T18:57:54.064498+0000","last_clean":"2024-09-10T18:57:54.064498+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:57:54.064498+0000","last_undegraded":"2024-09-10T18:57:54.064498+0000","last_fullsized":"2024-09-10T18:57:54.064498+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:56.320880+0000","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087944+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T18:57:54.087944+0000","last_peered":"2024-09-10T18:57:54.087944+0000","last_clean":"2024-09-10T18:57:54.087944+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:57:54.087944+0000","last_undegraded":"2024-09-10T18:57:54.087944+0000","last_fullsized":"2024-09-10T18:57:54.087944+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.553089+0000","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.090666+0000","last_change":"2024-09-10T18:56:40.177375+0000","last_active":"2024-09-10T18:57:54.090666+0000","last_peered":"2024-09-10T18:57:54.090666+0000","last_clean":"2024-09-10T18:57:54.090666+0000","last_became_active":"2024-09-10T18:54:37.565268+0000","last_became_peered":"2024-09-10T18:54:37.565268+0000","last_unstale":"2024-09-10T18:57:54.090666+0000","last_undegraded":"2024-09-10T18:57:54.090666+0000","last_fullsized":"2024-09-10T18:57:54.090666+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.177249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.177249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.578420+0000","scrub_duration":0.0033094600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.056767+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:57:54.056767+0000","last_peered":"2024-09-10T18:57:54.056767+0000","last_clean":"2024-09-10T18:57:54.056767+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:57:54.056767+0000","last_undegraded":"2024-09-10T18:57:54.056767+0000","last_fullsized":"2024-09-10T18:57:54.056767+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:57:58.123736+0000","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":328,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.096073+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:57:54.096073+0000","last_peered":"2024-09-10T18:57:54.096073+0000","last_clean":"2024-09-10T18:57:54.096073+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:57:54.096073+0000","last_undegraded":"2024-09-10T18:57:54.096073+0000","last_fullsized":"2024-09-10T18:57:54.096073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:03.905881+0000","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":295,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.070741+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:57:54.070741+0000","last_peered":"2024-09-10T18:57:54.070741+0000","last_clean":"2024-09-10T18:57:54.070741+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:57:54.070741+0000","last_undegraded":"2024-09-10T18:57:54.070741+0000","last_fullsized":"2024-09-10T18:57:54.070741+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":293,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.087015+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:57:54.087015+0000","last_peered":"2024-09-10T18:57:54.087015+0000","last_clean":"2024-09-10T18:57:54.087015+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:57:54.087015+0000","last_undegraded":"2024-09-10T18:57:54.087015+0000","last_fullsized":"2024-09-10T18:57:54.087015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.073792+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:57:54.073792+0000","last_peered":"2024-09-10T18:57:54.073792+0000","last_clean":"2024-09-10T18:57:54.073792+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:57:54.073792+0000","last_undegraded":"2024-09-10T18:57:54.073792+0000","last_fullsized":"2024-09-10T18:57:54.073792+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:01.150161+0000","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"126'10","reported_seq":242,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060868+0000","last_change":"2024-09-10T18:57:39.096358+0000","last_active":"2024-09-10T18:57:54.060868+0000","last_peered":"2024-09-10T18:57:54.060868+0000","last_clean":"2024-09-10T18:57:54.060868+0000","last_became_active":"2024-09-10T18:57:39.095796+0000","last_became_peered":"2024-09-10T18:57:39.095796+0000","last_unstale":"2024-09-10T18:57:54.060868+0000","last_undegraded":"2024-09-10T18:57:54.060868+0000","last_fullsized":"2024-09-10T18:57:54.060868+0000","mapping_epoch":124,"log_start":"126'10","ondisk_log_start":"126'10","created":124,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:28.793065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.793065+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":15,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039084699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":299,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.822588+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T18:57:54.822588+0000","last_peered":"2024-09-10T18:57:54.822588+0000","last_clean":"2024-09-10T18:57:54.822588+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:57:54.822588+0000","last_undegraded":"2024-09-10T18:57:54.822588+0000","last_fullsized":"2024-09-10T18:57:54.822588+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.514634+0000","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.058620+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:57:54.058620+0000","last_peered":"2024-09-10T18:57:54.058620+0000","last_clean":"2024-09-10T18:57:54.058620+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:57:54.058620+0000","last_undegraded":"2024-09-10T18:57:54.058620+0000","last_fullsized":"2024-09-10T18:57:54.058620+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060068+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T18:57:54.060068+0000","last_peered":"2024-09-10T18:57:54.060068+0000","last_clean":"2024-09-10T18:57:54.060068+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:57:54.060068+0000","last_undegraded":"2024-09-10T18:57:54.060068+0000","last_fullsized":"2024-09-10T18:57:54.060068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.346480+0000","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":422,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068043+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:57:54.068043+0000","last_peered":"2024-09-10T18:57:54.068043+0000","last_clean":"2024-09-10T18:57:54.068043+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:57:54.068043+0000","last_undegraded":"2024-09-10T18:57:54.068043+0000","last_fullsized":"2024-09-10T18:57:54.068043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.068030+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:57:54.068030+0000","last_peered":"2024-09-10T18:57:54.068030+0000","last_clean":"2024-09-10T18:57:54.068030+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:57:54.068030+0000","last_undegraded":"2024-09-10T18:57:54.068030+0000","last_fullsized":"2024-09-10T18:57:54.068030+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"126'10","reported_seq":78,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.071470+0000","last_change":"2024-09-10T18:57:35.056450+0000","last_active":"2024-09-10T18:57:54.071470+0000","last_peered":"2024-09-10T18:57:54.071470+0000","last_clean":"2024-09-10T18:57:54.071470+0000","last_became_active":"2024-09-10T18:57:35.056095+0000","last_became_peered":"2024-09-10T18:57:35.056095+0000","last_unstale":"2024-09-10T18:57:54.071470+0000","last_undegraded":"2024-09-10T18:57:54.071470+0000","last_fullsized":"2024-09-10T18:57:54.071470+0000","mapping_epoch":274,"log_start":"126'10","ondisk_log_start":"126'10","created":124,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":12,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064462+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T18:57:54.064462+0000","last_peered":"2024-09-10T18:57:54.064462+0000","last_clean":"2024-09-10T18:57:54.064462+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:57:54.064462+0000","last_undegraded":"2024-09-10T18:57:54.064462+0000","last_fullsized":"2024-09-10T18:57:54.064462+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.795787+0000","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":305,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.066914+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:57:54.066914+0000","last_peered":"2024-09-10T18:57:54.066914+0000","last_clean":"2024-09-10T18:57:54.066914+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:57:54.066914+0000","last_undegraded":"2024-09-10T18:57:54.066914+0000","last_fullsized":"2024-09-10T18:57:54.066914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":305,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.089077+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:57:54.089077+0000","last_peered":"2024-09-10T18:57:54.089077+0000","last_clean":"2024-09-10T18:57:54.089077+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:57:54.089077+0000","last_undegraded":"2024-09-10T18:57:54.089077+0000","last_fullsized":"2024-09-10T18:57:54.089077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":302,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.059257+0000","last_change":"2024-09-10T18:56:26.777537+0000","last_active":"2024-09-10T18:57:54.059257+0000","last_peered":"2024-09-10T18:57:54.059257+0000","last_clean":"2024-09-10T18:57:54.059257+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:57:54.059257+0000","last_undegraded":"2024-09-10T18:57:54.059257+0000","last_fullsized":"2024-09-10T18:57:54.059257+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.777271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.777271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030242540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":212,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.073228+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:57:54.073228+0000","last_peered":"2024-09-10T18:57:54.073228+0000","last_clean":"2024-09-10T18:57:54.073228+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:57:54.073228+0000","last_undegraded":"2024-09-10T18:57:54.073228+0000","last_fullsized":"2024-09-10T18:57:54.073228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":234,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.088544+0000","last_change":"2024-09-10T18:57:47.078409+0000","last_active":"2024-09-10T18:57:54.088544+0000","last_peered":"2024-09-10T18:57:54.088544+0000","last_clean":"2024-09-10T18:57:54.088544+0000","last_became_active":"2024-09-10T18:57:47.077956+0000","last_became_peered":"2024-09-10T18:57:47.077956+0000","last_unstale":"2024-09-10T18:57:54.088544+0000","last_undegraded":"2024-09-10T18:57:54.088544+0000","last_fullsized":"2024-09-10T18:57:54.088544+0000","mapping_epoch":124,"log_start":"126'2","ondisk_log_start":"126'2","created":124,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:57:36.585730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.585730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.361630+0000","scrub_duration":0.0027517190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.063087+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:57:54.063087+0000","last_peered":"2024-09-10T18:57:54.063087+0000","last_clean":"2024-09-10T18:57:54.063087+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:57:54.063087+0000","last_undegraded":"2024-09-10T18:57:54.063087+0000","last_fullsized":"2024-09-10T18:57:54.063087+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":298,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.064975+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:57:54.064975+0000","last_peered":"2024-09-10T18:57:54.064975+0000","last_clean":"2024-09-10T18:57:54.064975+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:57:54.064975+0000","last_undegraded":"2024-09-10T18:57:54.064975+0000","last_fullsized":"2024-09-10T18:57:54.064975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":322,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.065251+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T18:57:54.065251+0000","last_peered":"2024-09-10T18:57:54.065251+0000","last_clean":"2024-09-10T18:57:54.065251+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:57:54.065251+0000","last_undegraded":"2024-09-10T18:57:54.065251+0000","last_fullsized":"2024-09-10T18:57:54.065251+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.200134+0000","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":284,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.060962+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:57:54.060962+0000","last_peered":"2024-09-10T18:57:54.060962+0000","last_clean":"2024-09-10T18:57:54.060962+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:57:54.060962+0000","last_undegraded":"2024-09-10T18:57:54.060962+0000","last_fullsized":"2024-09-10T18:57:54.060962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":213,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.067221+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:57:54.067221+0000","last_peered":"2024-09-10T18:57:54.067221+0000","last_clean":"2024-09-10T18:57:54.067221+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:57:54.067221+0000","last_undegraded":"2024-09-10T18:57:54.067221+0000","last_fullsized":"2024-09-10T18:57:54.067221+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":231,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-09-10T18:57:54.096710+0000","last_change":"2024-09-10T18:57:43.108161+0000","last_active":"2024-09-10T18:57:54.096710+0000","last_peered":"2024-09-10T18:57:54.096710+0000","last_clean":"2024-09-10T18:57:54.096710+0000","last_became_active":"2024-09-10T18:57:43.107719+0000","last_became_peered":"2024-09-10T18:57:43.107719+0000","last_unstale":"2024-09-10T18:57:54.096710+0000","last_undegraded":"2024-09-10T18:57:54.096710+0000","last_fullsized":"2024-09-10T18:57:54.096710+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":187,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":4195508,"num_objects":122,"num_object_clones":2,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":122,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":545,"num_write_kb":8730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14057472,"data_stored":12586524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1796,"ondisk_log_size":1796,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":492,"num_read_kb":487,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":21,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34,"ondisk_log_size":34,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972467,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319416,"kb_used_data":9560,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425736,"statfs":{"total":95995035648,"available":95667953664,"internally_reserved":0,"allocated":9789440,"data_stored":7535445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88200000000000001,"15min":0.91400000000000003},"min":{"1min":0.53300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.3420000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.92900000000000005,"15min":0.93400000000000005},"min":{"1min":0.58999999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.155}]},{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.875,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.98499999999999999,"15min":1.02},"min":{"1min":0.72499999999999998,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.3360000000000001,"5min":1.742,"15min":1.8280000000000001},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0309999999999999,"15min":1.0389999999999999},"min":{"1min":0.70699999999999996,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3580000000000001,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":1.365},{"interface":"front","average":{"1min":1.111,"5min":1.012,"15min":0.98699999999999999},"min":{"1min":0.57399999999999995,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":1.0660000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.0369999999999999,"15min":1.018},"min":{"1min":0.73199999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":4.2080000000000002,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.66500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":1.004}]},{"osd":4,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.04,"15min":1.0189999999999999},"min":{"1min":0.67800000000000005,"5min":0.499,"15min":0.443},"max":{"1min":1.2789999999999999,"5min":1.556,"15min":2.1989999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.1160000000000001,"15min":1.054},"min":{"1min":0.78500000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.2400000000000002,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":0.85099999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.026,"15min":1.0049999999999999},"min":{"1min":0.59799999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.181,"5min":1.175,"15min":1.1779999999999999},"min":{"1min":0.59699999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":2.1419999999999999},"last":1.113}]},{"osd":6,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.71099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.591,"5min":1.591,"15min":2.2749999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2090000000000001,"15min":1.196},"min":{"1min":0.86099999999999999,"5min":0.59199999999999997,"15min":0.53100000000000003},"max":{"1min":4.1310000000000002,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":0.92600000000000005}]}]},{"osd":6,"up_from":52,"seq":223338299517,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302772,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442380,"statfs":{"total":95995035648,"available":95684997120,"internally_reserved":0,"allocated":5541888,"data_stored":3350972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.872,"15min":0.88400000000000001},"min":{"1min":0.58399999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.032,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.57799999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1779999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.0069999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.196,"15min":0.997},"min":{"1min":0.56699999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9530000000000001,"5min":18.494,"15min":18.494},"last":1.117},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.871,"15min":0.90200000000000002},"min":{"1min":0.73099999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.244,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.96499999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0249999999999999,"15min":0.95899999999999996},"min":{"1min":0.56399999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.2430000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.92300000000000004,"15min":0.92400000000000004},"min":{"1min":0.54300000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.3300000000000001,"5min":1.988,"15min":1.988},"last":0.72699999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0329999999999999,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.234,"5min":2.278,"15min":2.278},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.98299999999999998,"15min":0.97399999999999998},"min":{"1min":0.73899999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.7529999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.0800000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.97599999999999998,"15min":0.90400000000000003},"min":{"1min":0.58999999999999997,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.452,"5min":1.712,"15min":1.712},"last":0.879},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.099,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.2250000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.999},"min":{"1min":0.67900000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.516,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.165},{"interface":"front","average":{"1min":1.252,"5min":1.163,"15min":1.1100000000000001},"min":{"1min":0.77000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.1669999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.2809999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.0960000000000001,"15min":1.0600000000000001},"min":{"1min":0.90200000000000002,"5min":0.503,"15min":0.503},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.175,"15min":1.153},"min":{"1min":0.93100000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.254}]}]},{"osd":5,"up_from":43,"seq":184683593863,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318840,"kb_used_data":9000,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93426312,"statfs":{"total":95995035648,"available":95668543488,"internally_reserved":0,"allocated":9216000,"data_stored":6954808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.0820000000000001,"15min":0.92900000000000005},"min":{"1min":0.41599999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.3949999999999996,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.292,"5min":1.024,"15min":0.91300000000000003},"min":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":5.0519999999999996,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":1.1140000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4369999999999998,"5min":1.6579999999999999,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":15.824,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.008},{"interface":"front","average":{"1min":1.361,"5min":1.427,"15min":1.069},"min":{"1min":0.55100000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.0839999999999996,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.0529999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":1.8899999999999999,"15min":1.2250000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":20.536000000000001,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.1339999999999999,"15min":0.98199999999999998},"min":{"1min":0.60299999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":5.2489999999999997,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.1279999999999999,"15min":0.98699999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.5979999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.75},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.7609999999999999,"15min":1.218},"min":{"1min":0.66400000000000003,"5min":0.60999999999999999,"15min":0.57399999999999995},"max":{"1min":7.7679999999999998,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.135}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.147,"15min":0.90600000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":13.005000000000001,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.544,"5min":1.1659999999999999,"15min":0.96899999999999997},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":7.6299999999999999,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.091}]},{"osd":6,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.3660000000000001,"15min":1.133},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":7.5380000000000003,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.278},{"interface":"front","average":{"1min":1.355,"5min":1.407,"15min":1.165},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.47199999999999998},"max":{"1min":4.0709999999999997,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.2450000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":1.49,"15min":1.2010000000000001},"min":{"1min":0.68500000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":15.613,"5min":15.613,"15min":15.613},"last":1.179},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.51,"15min":1.2749999999999999},"min":{"1min":0.72899999999999998,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":16.965,"5min":16.965,"15min":16.965},"last":0.94399999999999995}]}]},{"osd":4,"up_from":35,"seq":150323855507,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306336,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438816,"statfs":{"total":95995035648,"available":95681347584,"internally_reserved":0,"allocated":4931584,"data_stored":2750241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81000000000000005,"15min":0.84099999999999997},"min":{"1min":0.505,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3320000000000001,"5min":1.546,"15min":1.546},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.89500000000000002,"15min":0.86299999999999999},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.4750000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.996}]},{"osd":1,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.85099999999999998,"15min":0.878},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.2},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.93700000000000006,"15min":0.88900000000000001},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.323,"5min":1.575,"15min":1.575},"last":1.032}]},{"osd":2,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.4179999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.90800000000000003},"min":{"1min":0.60099999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.482,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.92600000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94299999999999995,"15min":0.91800000000000004},"min":{"1min":0.64400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.714},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.006,"5min":0.96799999999999997,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.054}]},{"osd":5,"last update":"Tue Sep 10 18:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96199999999999997,"15min":0.88900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.3939999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.999,"5min":1.026,"15min":0.94399999999999995},"min":{"1min":0.65100000000000002,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.256,"5min":1.528,"15min":1.7909999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:57:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0349999999999999,"15min":0.96899999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.135,"15min":1.0860000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.4670000000000001,"5min":1.581,"15min":1.581},"last":1.1120000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1359999999999999,"15min":1.1619999999999999},"min":{"1min":0.64100000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.516,"5min":1.6200000000000001,"15min":2.008},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1559999999999999,"15min":1.165},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.5149999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051742,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314748,"kb_used_data":4908,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430404,"statfs":{"total":95995035648,"available":95672733696,"internally_reserved":0,"allocated":5025792,"data_stored":2760625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2150000000000001,"15min":0.82199999999999995},"min":{"1min":0.38600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":8.6180000000000003,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.002,"5min":1.2410000000000001,"15min":0.89100000000000001},"min":{"1min":0.441,"5min":0.252,"15min":0.252},"max":{"1min":5.3479999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.63500000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.3819999999999999,"15min":0.91600000000000004},"min":{"1min":0.41799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.3879999999999999,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.022,"5min":1.0600000000000001,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.0549999999999997,"5min":15.189,"15min":15.189},"last":0.99399999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.3660000000000001,"15min":0.98699999999999999},"min":{"1min":0.502,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1920000000000002,"5min":26.558,"15min":26.558},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.2410000000000001,"15min":0.91900000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.3199999999999998,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.3049999999999999,"15min":1.111},"min":{"1min":0.73799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":7.3550000000000004,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.383},{"interface":"front","average":{"1min":1.522,"5min":1.411,"15min":1.198},"min":{"1min":0.79000000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":7.4189999999999996,"5min":10.391,"15min":10.391},"last":1.421}]},{"osd":5,"last update":"Tue Sep 10 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.4379999999999999,"15min":1.2130000000000001},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":8.6679999999999993,"5min":17.535,"15min":17.535},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.123,"15min":1.028},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":8.7880000000000003,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.82999999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:56:59 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.2709999999999999,"15min":1.2010000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.5289999999999999,"5min":10.51,"15min":10.51},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.135,"15min":1.0920000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.8279999999999998,"5min":10.773999999999999,"15min":10.773999999999999},"last":0.88700000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:57:53 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.3220000000000001,"15min":1.179},"min":{"1min":0.60699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.214,"5min":18.312999999999999,"15min":18.312999999999999},"last":1.139},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.4470000000000001,"15min":1.306},"min":{"1min":0.77500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.0270000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.339}]}]},{"osd":2,"up_from":19,"seq":81604378791,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310528,"kb_used_data":9000,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93434624,"statfs":{"total":95995035648,"available":95677054976,"internally_reserved":0,"allocated":9216000,"data_stored":6945114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.84199999999999997,"15min":0.71399999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.5629999999999999,"5min":9.516,"15min":9.516},"last":0.625},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81200000000000006,"15min":0.76400000000000001},"min":{"1min":0.32400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.2350000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.92900000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.75900000000000001,"15min":0.70999999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.2380000000000004,"5min":5.516,"15min":5.516},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.84799999999999998,"15min":0.79900000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.26,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.89700000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.89400000000000002,"15min":0.85399999999999998},"min":{"1min":0.52500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.139,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.84599999999999997},"min":{"1min":0.53800000000000003,"5min":0.434,"15min":0.434},"max":{"1min":2.0539999999999998,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.79100000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.171,"15min":1.0389999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2909999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.147},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1519999999999999,"15min":1.1040000000000001},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4379999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.0549999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:57:02 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.113,"15min":1.109},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.605,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":1.171},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.214,"15min":1.085},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.429,"5min":21.654,"15min":21.654},"last":1.079}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.206,"15min":1.1870000000000001},"min":{"1min":0.622,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5009999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.252,"15min":1.129},"min":{"1min":0.52100000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.5669999999999999,"5min":15.971,"15min":15.971},"last":1.1180000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.21,"15min":1.2070000000000001},"min":{"1min":0.65800000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.399,"5min":15.025,"15min":15.025},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.2869999999999999,"15min":1.173},"min":{"1min":0.57699999999999996,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.0299999999999998,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.022}]}]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302212,"kb_used_data":4844,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442940,"statfs":{"total":95995035648,"available":95685570560,"internally_reserved":0,"allocated":4960256,"data_stored":2749570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.028,"15min":0.76500000000000001},"min":{"1min":0.34200000000000003,"5min":0.246,"15min":0.246},"max":{"1min":2.238,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.87,"5min":1.1060000000000001,"15min":0.83599999999999997},"min":{"1min":0.32900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.8250000000000002,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.58499999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.117,"15min":0.83099999999999996},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.3109999999999999,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.3,"15min":0.94999999999999996},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.488,"5min":32.363,"15min":32.363},"last":0.83199999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:57:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.208,"15min":0.95899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7360000000000002,"5min":25.704000000000001,"15min":25.704000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.754,"5min":1.476,"15min":0.98799999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1890000000000001,"5min":60.999000000000002,"15min":60.999000000000002},"last":6.0099999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:57:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.1819999999999999,"15min":1.0669999999999999},"min":{"1min":0.47399999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.5379999999999998,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.6639999999999999,"15min":1.1619999999999999},"min":{"1min":0.45400000000000001,"5min":0.378,"15min":0.378},"max":{"1min":6.5430000000000001,"5min":60.529000000000003,"15min":60.529000000000003},"last":6.181}]},{"osd":5,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.2789999999999999,"15min":1.0980000000000001},"min":{"1min":0.55900000000000005,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":2.2719999999999998,"5min":26.050999999999998,"15min":26.050999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.3220000000000001,"15min":1.1919999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":4.5869999999999997,"5min":12.84,"15min":12.84},"last":1.052}]},{"osd":6,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.528,"15min":1.232},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":8.1760000000000002,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.4450000000000001,"15min":1.1990000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.486,"5min":33.643000000000001,"15min":33.643000000000001},"last":6.1219999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:56:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.486,"15min":1.306},"min":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":3.6499999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.5409999999999999,"15min":1.278},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.5740000000000001,"5min":33.673999999999999,"15min":33.673999999999999},"last":6.2599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315268,"kb_used_data":5412,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93429884,"statfs":{"total":95995035648,"available":95672201216,"internally_reserved":0,"allocated":5541888,"data_stored":3339682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.6240000000000001,"15min":0.94899999999999995},"min":{"1min":0.39100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0920000000000001,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.716,"15min":1.038},"min":{"1min":0.46000000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.2629999999999999,"5min":86.009,"15min":86.009},"last":0.48099999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.601,"15min":0.99299999999999999},"min":{"1min":0.32900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":1.474,"5min":36.682000000000002,"15min":36.682000000000002},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.2,"15min":0.88300000000000001},"min":{"1min":0.48299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.2450000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.72599999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0820000000000001,"15min":0.86399999999999999},"min":{"1min":0.38500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3680000000000003,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.45,"15min":1.0129999999999999},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.0019999999999998,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.84099999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.1160000000000001,"15min":1.0189999999999999},"min":{"1min":0.72399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.3799999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.143,"15min":1.0760000000000001},"min":{"1min":0.63100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4990000000000001,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.0489999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:57:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.246,"15min":1.0940000000000001},"min":{"1min":0.71199999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.776,"5min":10.048,"15min":10.048},"last":1.21},{"interface":"front","average":{"1min":1.069,"5min":1.1319999999999999,"15min":1.1000000000000001},"min":{"1min":0.75700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.635,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.80600000000000005}]},{"osd":6,"last update":"Tue Sep 10 18:56:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1579999999999999,"15min":1.1220000000000001},"min":{"1min":0.69099999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.4550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.3200000000000001,"15min":1.1819999999999999},"min":{"1min":0.66400000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.696,"5min":11.894,"15min":11.894},"last":1.1639999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1659999999999999,"15min":1.167},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.512,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3260000000000001,"15min":1.3029999999999999},"min":{"1min":0.80100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.3780000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.1220000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4411392,"data_stored":4194848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4411392,"data_stored":4194848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4399104,"data_stored":4194808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:58:01.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.583 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:01.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:01.645 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:01.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:00.459974+0000 mgr.y (mgr.24526) 357 : audit [DBG] from='client.49205 v1:172.21.15.83:0/2352347524' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:01.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:00.565829+0000 mon.a (mon.0) 2120 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:01.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.055357+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.082027+0000 mon.a (mon.0) 2121 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.082217+0000 mon.a (mon.0) 2122 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:01.082344+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.082577+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.085015+0000 mon.c (mon.2) 544 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.085771+0000 mon.a (mon.0) 2124 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:01 smithi165 bash[22591]: audit 2024-09-10T18:58:01.086457+0000 mon.a (mon.0) 2125 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:00.459974+0000 mgr.y (mgr.24526) 357 : audit [DBG] from='client.49205 v1:172.21.15.83:0/2352347524' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:00.565829+0000 mon.a (mon.0) 2120 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.055357+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.082027+0000 mon.a (mon.0) 2121 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:01.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.082217+0000 mon.a (mon.0) 2122 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:01.082344+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.082577+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.085015+0000 mon.c (mon.2) 544 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.085771+0000 mon.a (mon.0) 2124 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[32381]: audit 2024-09-10T18:58:01.086457+0000 mon.a (mon.0) 2125 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:00.459974+0000 mgr.y (mgr.24526) 357 : audit [DBG] from='client.49205 v1:172.21.15.83:0/2352347524' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:00.565829+0000 mon.a (mon.0) 2120 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.055357+0000 mon.c (mon.2) 543 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.082027+0000 mon.a (mon.0) 2121 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.082217+0000 mon.a (mon.0) 2122 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi083-77648-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:01.082344+0000 mon.a (mon.0) 2123 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-09-10T18:58:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.082577+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.085015+0000 mon.c (mon.2) 544 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.085771+0000 mon.a (mon.0) 2124 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:01 smithi083 bash[18629]: audit 2024-09-10T18:58:01.086457+0000 mon.a (mon.0) 2125 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]: dispatch 2024-09-10T18:58:01.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.885 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:01.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:01.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.018 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:02.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:02.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:02.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:02.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:02.425 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:02.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.552 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:02.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:02.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:02.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:02.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:02 smithi165 bash[22591]: cluster 2024-09-10T18:58:02.024316+0000 mgr.y (mgr.24526) 358 : cluster [DBG] pgmap v472: 249 pgs: 2 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T18:58:02.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:02 smithi165 bash[22591]: audit 2024-09-10T18:58:02.057086+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:02.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:02 smithi165 bash[22591]: cluster 2024-09-10T18:58:02.075210+0000 mon.a (mon.0) 2126 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:02.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:02 smithi165 bash[22591]: audit 2024-09-10T18:58:02.518746+0000 mon.a (mon.0) 2127 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]': finished 2024-09-10T18:58:02.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:02 smithi165 bash[22591]: cluster 2024-09-10T18:58:02.518844+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:58:02.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:02.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:02.945 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:02] "GET /metrics HTTP/1.1" 200 231116 "" "Prometheus/2.43.0" 2024-09-10T18:58:02.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[32381]: cluster 2024-09-10T18:58:02.024316+0000 mgr.y (mgr.24526) 358 : cluster [DBG] pgmap v472: 249 pgs: 2 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[32381]: audit 2024-09-10T18:58:02.057086+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[32381]: cluster 2024-09-10T18:58:02.075210+0000 mon.a (mon.0) 2126 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[32381]: audit 2024-09-10T18:58:02.518746+0000 mon.a (mon.0) 2127 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]': finished 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[32381]: cluster 2024-09-10T18:58:02.518844+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18629]: cluster 2024-09-10T18:58:02.024316+0000 mgr.y (mgr.24526) 358 : cluster [DBG] pgmap v472: 249 pgs: 2 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18629]: audit 2024-09-10T18:58:02.057086+0000 mon.c (mon.2) 545 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18629]: cluster 2024-09-10T18:58:02.075210+0000 mon.a (mon.0) 2126 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18629]: audit 2024-09-10T18:58:02.518746+0000 mon.a (mon.0) 2127 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-37", "mode": "writeback"}]': finished 2024-09-10T18:58:02.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:02 smithi083 bash[18629]: cluster 2024-09-10T18:58:02.518844+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-09-10T18:58:02.963 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:02.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:03.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:03.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:03.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:03.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:03.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:03.380 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:03.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:03.516 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:03.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:03.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:03.767 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:02.581138+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.767 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:02.584976+0000 mon.a (mon.0) 2129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:03.058848+0000 mon.c (mon.2) 546 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:03.087006+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:03.088519+0000 mon.a (mon.0) 2130 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:03.088705+0000 mon.a (mon.0) 2131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: cluster 2024-09-10T18:58:03.088833+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[18629]: audit 2024-09-10T18:58:03.091083+0000 mon.a (mon.0) 2133 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:02.581138+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:02.584976+0000 mon.a (mon.0) 2129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:03.058848+0000 mon.c (mon.2) 546 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:03.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:03.087006+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:03.088519+0000 mon.a (mon.0) 2130 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:03.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:03.088705+0000 mon.a (mon.0) 2131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:03.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: cluster 2024-09-10T18:58:03.088833+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:58:03.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:03 smithi083 bash[32381]: audit 2024-09-10T18:58:03.091083+0000 mon.a (mon.0) 2133 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:03.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:03.839 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:02.581138+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.839 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:02.584976+0000 mon.a (mon.0) 2129 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:03.058848+0000 mon.c (mon.2) 546 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:03.087006+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:03.088519+0000 mon.a (mon.0) 2130 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi083-77648-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:03.088705+0000 mon.a (mon.0) 2131 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: cluster 2024-09-10T18:58:03.088833+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-09-10T18:58:03.840 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:03 smithi165 bash[22591]: audit 2024-09-10T18:58:03.091083+0000 mon.a (mon.0) 2133 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]: dispatch 2024-09-10T18:58:03.892 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:03.926 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:03.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:04.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:04.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:04.208 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:04.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:04.345 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:04.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:04.483 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:04.584 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:58:04.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:04.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: cluster 2024-09-10T18:58:04.025219+0000 mgr.y (mgr.24526) 359 : cluster [DBG] pgmap v475: 256 pgs: 8 unknown, 1 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-09-10T18:58:04.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: audit 2024-09-10T18:58:04.025950+0000 mon.c (mon.2) 547 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: audit 2024-09-10T18:58:04.028256+0000 mon.a (mon.0) 2134 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: cluster 2024-09-10T18:58:04.055067+0000 mon.a (mon.0) 2135 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:04.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: audit 2024-09-10T18:58:04.064708+0000 mon.c (mon.2) 548 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:04.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: audit 2024-09-10T18:58:04.064933+0000 mon.a (mon.0) 2136 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:04.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: audit 2024-09-10T18:58:04.065434+0000 mon.a (mon.0) 2137 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-10T18:58:04.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:04 smithi165 bash[22591]: cluster 2024-09-10T18:58:04.065649+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:58:04.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:04.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:04.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: cluster 2024-09-10T18:58:04.025219+0000 mgr.y (mgr.24526) 359 : cluster [DBG] pgmap v475: 256 pgs: 8 unknown, 1 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-09-10T18:58:04.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: audit 2024-09-10T18:58:04.025950+0000 mon.c (mon.2) 547 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: audit 2024-09-10T18:58:04.028256+0000 mon.a (mon.0) 2134 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: cluster 2024-09-10T18:58:04.055067+0000 mon.a (mon.0) 2135 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:04.727 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: audit 2024-09-10T18:58:04.064708+0000 mon.c (mon.2) 548 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: audit 2024-09-10T18:58:04.064933+0000 mon.a (mon.0) 2136 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: audit 2024-09-10T18:58:04.065434+0000 mon.a (mon.0) 2137 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[18629]: cluster 2024-09-10T18:58:04.065649+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: cluster 2024-09-10T18:58:04.025219+0000 mgr.y (mgr.24526) 359 : cluster [DBG] pgmap v475: 256 pgs: 8 unknown, 1 peering, 247 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: audit 2024-09-10T18:58:04.025950+0000 mon.c (mon.2) 547 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: audit 2024-09-10T18:58:04.028256+0000 mon.a (mon.0) 2134 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: cluster 2024-09-10T18:58:04.055067+0000 mon.a (mon.0) 2135 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: audit 2024-09-10T18:58:04.064708+0000 mon.c (mon.2) 548 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: audit 2024-09-10T18:58:04.064933+0000 mon.a (mon.0) 2136 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-37"}]': finished 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: audit 2024-09-10T18:58:04.065434+0000 mon.a (mon.0) 2137 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-10T18:58:04.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:04 smithi083 bash[32381]: cluster 2024-09-10T18:58:04.065649+0000 mon.a (mon.0) 2138 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-09-10T18:58:04.748 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:04.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:04.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:04.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:04.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:05.022 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:05.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:05.161 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T18:58:05.161 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:58:05.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.285 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:05.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:05.411 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:05.451 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:05.452 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:05.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:05.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:05.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:05 smithi165 bash[22591]: audit 2024-09-10T18:58:05.066473+0000 mon.c (mon.2) 549 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:05.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:05 smithi165 bash[22591]: cluster 2024-09-10T18:58:05.077844+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:58:05.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:05 smithi165 bash[22591]: audit 2024-09-10T18:58:05.085599+0000 mon.c (mon.2) 550 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:05 smithi165 bash[22591]: audit 2024-09-10T18:58:05.086252+0000 mon.a (mon.0) 2140 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:05.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:05.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[32381]: audit 2024-09-10T18:58:05.066473+0000 mon.c (mon.2) 549 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:05.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[32381]: cluster 2024-09-10T18:58:05.077844+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[32381]: audit 2024-09-10T18:58:05.085599+0000 mon.c (mon.2) 550 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[32381]: audit 2024-09-10T18:58:05.086252+0000 mon.a (mon.0) 2140 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[18629]: audit 2024-09-10T18:58:05.066473+0000 mon.c (mon.2) 549 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[18629]: cluster 2024-09-10T18:58:05.077844+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[18629]: audit 2024-09-10T18:58:05.085599+0000 mon.c (mon.2) 550 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:05 smithi083 bash[18629]: audit 2024-09-10T18:58:05.086252+0000 mon.a (mon.0) 2140 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:05.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:05.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T18:58:05.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:05.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:06.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:06.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:06.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:06.224 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:06.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:06.363 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:06.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:06.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:06.505 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:06.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:06.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:06.634 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:06.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: cluster 2024-09-10T18:58:06.027517+0000 mgr.y (mgr.24526) 360 : cluster [DBG] pgmap v478: 216 pgs: 2 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.068246+0000 mon.c (mon.2) 551 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.081426+0000 mon.a (mon.0) 2141 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.081585+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: cluster 2024-09-10T18:58:06.081696+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.085769+0000 mon.a (mon.0) 2143 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.085846+0000 mon.c (mon.2) 552 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:06.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:06 smithi165 bash[22591]: audit 2024-09-10T18:58:06.086349+0000 mon.a (mon.0) 2144 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:06.770 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:06.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:06.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:06.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:07.010 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:07.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: cluster 2024-09-10T18:58:06.027517+0000 mgr.y (mgr.24526) 360 : cluster [DBG] pgmap v478: 216 pgs: 2 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:07.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.068246+0000 mon.c (mon.2) 551 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:07.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.081426+0000 mon.a (mon.0) 2141 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:07.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.081585+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:07.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: cluster 2024-09-10T18:58:06.081696+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:58:07.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.085769+0000 mon.a (mon.0) 2143 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:07.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.085846+0000 mon.c (mon.2) 552 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:07.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[32381]: audit 2024-09-10T18:58:06.086349+0000 mon.a (mon.0) 2144 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:07.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: cluster 2024-09-10T18:58:06.027517+0000 mgr.y (mgr.24526) 360 : cluster [DBG] pgmap v478: 216 pgs: 2 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:07.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.068246+0000 mon.c (mon.2) 551 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:07.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.081426+0000 mon.a (mon.0) 2141 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:07.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.081585+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:07.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: cluster 2024-09-10T18:58:06.081696+0000 mon.a (mon.0) 2142 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-09-10T18:58:07.026 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.085769+0000 mon.a (mon.0) 2143 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:07.026 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.085846+0000 mon.c (mon.2) 552 : audit [INF] from='client.? v1:172.21.15.83:0/2376077129' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:07.026 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:06 smithi083 bash[18629]: audit 2024-09-10T18:58:06.086349+0000 mon.a (mon.0) 2144 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]: dispatch 2024-09-10T18:58:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:07.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:07.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:07.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:07.590 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:07.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:07.708 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: cluster 2024-09-10T18:58:06.596990+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.069904+0000 mon.c (mon.2) 553 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.086030+0000 mon.a (mon.0) 2146 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.086240+0000 mon.a (mon.0) 2147 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: cluster 2024-09-10T18:58:07.086577+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.086943+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.092113+0000 mon.a (mon.0) 2149 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.120741+0000 mon.c (mon.2) 554 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.121082+0000 mon.a (mon.0) 2150 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.122362+0000 mon.c (mon.2) 555 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.122682+0000 mon.a (mon.0) 2151 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.123195+0000 mon.c (mon.2) 556 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[18629]: audit 2024-09-10T18:58:07.123504+0000 mon.a (mon.0) 2152 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: cluster 2024-09-10T18:58:06.596990+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.069904+0000 mon.c (mon.2) 553 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.086030+0000 mon.a (mon.0) 2146 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.086240+0000 mon.a (mon.0) 2147 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: cluster 2024-09-10T18:58:07.086577+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.086943+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.092113+0000 mon.a (mon.0) 2149 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.120741+0000 mon.c (mon.2) 554 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.121082+0000 mon.a (mon.0) 2150 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.710 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.122362+0000 mon.c (mon.2) 555 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.122682+0000 mon.a (mon.0) 2151 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.123195+0000 mon.c (mon.2) 556 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.711 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:07 smithi083 bash[32381]: audit 2024-09-10T18:58:07.123504+0000 mon.a (mon.0) 2152 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.721 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:07.825 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:07.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:07.863 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: cluster 2024-09-10T18:58:06.596990+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.069904+0000 mon.c (mon.2) 553 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.086030+0000 mon.a (mon.0) 2146 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.086240+0000 mon.a (mon.0) 2147 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi083-77648-54"}]': finished 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: cluster 2024-09-10T18:58:07.086577+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.086943+0000 mon.b (mon.1) 262 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.092113+0000 mon.a (mon.0) 2149 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.120741+0000 mon.c (mon.2) 554 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.121082+0000 mon.a (mon.0) 2150 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.122362+0000 mon.c (mon.2) 555 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.122682+0000 mon.a (mon.0) 2151 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:07.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.123195+0000 mon.c (mon.2) 556 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:07 smithi165 bash[22591]: audit 2024-09-10T18:58:07.123504+0000 mon.a (mon.0) 2152 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:07.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:07.991 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:07.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:08.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.120 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:08.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:08.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:08.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.533 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:08.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:08.663 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:08.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:07.835982+0000 mgr.y (mgr.24526) 361 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: cluster 2024-09-10T18:58:07.863106+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: cluster 2024-09-10T18:58:07.863502+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: cluster 2024-09-10T18:58:08.028334+0000 mgr.y (mgr.24526) 362 : cluster [DBG] pgmap v481: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.028974+0000 mon.c (mon.2) 557 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.031318+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.071298+0000 mon.c (mon.2) 558 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.086639+0000 mon.a (mon.0) 2154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:08.780 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.086764+0000 mon.a (mon.0) 2155 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.086931+0000 mon.a (mon.0) 2156 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: cluster 2024-09-10T18:58:08.087005+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.089405+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.089868+0000 mon.c (mon.2) 559 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.090662+0000 mon.a (mon.0) 2158 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[32381]: audit 2024-09-10T18:58:08.093479+0000 mon.a (mon.0) 2159 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:07.835982+0000 mgr.y (mgr.24526) 361 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: cluster 2024-09-10T18:58:07.863106+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: cluster 2024-09-10T18:58:07.863502+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:08.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: cluster 2024-09-10T18:58:08.028334+0000 mgr.y (mgr.24526) 362 : cluster [DBG] pgmap v481: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.028974+0000 mon.c (mon.2) 557 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.031318+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.071298+0000 mon.c (mon.2) 558 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.086639+0000 mon.a (mon.0) 2154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.086764+0000 mon.a (mon.0) 2155 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.086931+0000 mon.a (mon.0) 2156 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: cluster 2024-09-10T18:58:08.087005+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.089405+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.089868+0000 mon.c (mon.2) 559 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.090662+0000 mon.a (mon.0) 2158 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.782 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:08 smithi083 bash[18629]: audit 2024-09-10T18:58:08.093479+0000 mon.a (mon.0) 2159 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:07.835982+0000 mgr.y (mgr.24526) 361 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: cluster 2024-09-10T18:58:07.863106+0000 osd.2 (osd.2) 139 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: cluster 2024-09-10T18:58:07.863502+0000 osd.2 (osd.2) 140 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: cluster 2024-09-10T18:58:08.028334+0000 mgr.y (mgr.24526) 362 : cluster [DBG] pgmap v481: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.028974+0000 mon.c (mon.2) 557 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.031318+0000 mon.a (mon.0) 2153 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.071298+0000 mon.c (mon.2) 558 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.086639+0000 mon.a (mon.0) 2154 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.086764+0000 mon.a (mon.0) 2155 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi083-77648-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.086931+0000 mon.a (mon.0) 2156 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: cluster 2024-09-10T18:58:08.087005+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-09-10T18:58:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.089405+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.089868+0000 mon.c (mon.2) 559 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.090662+0000 mon.a (mon.0) 2158 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:08.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:08 smithi165 bash[22591]: audit 2024-09-10T18:58:08.093479+0000 mon.a (mon.0) 2159 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:08.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:08.939 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:08.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:09.043 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:09.074 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:09.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:09.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:09.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:09.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:09.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:09.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:09.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:09.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:09.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:09.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: cluster 2024-09-10T18:58:08.607678+0000 osd.3 (osd.3) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: cluster 2024-09-10T18:58:08.608193+0000 osd.3 (osd.3) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: audit 2024-09-10T18:58:09.072530+0000 mon.c (mon.2) 560 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: audit 2024-09-10T18:58:09.524281+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: audit 2024-09-10T18:58:09.526162+0000 mon.a (mon.0) 2160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: cluster 2024-09-10T18:58:09.526424+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:58:09.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:09 smithi165 bash[22591]: audit 2024-09-10T18:58:09.528245+0000 mon.a (mon.0) 2162 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:09.761 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:09.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:09.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:09.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: cluster 2024-09-10T18:58:08.607678+0000 osd.3 (osd.3) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: cluster 2024-09-10T18:58:08.608193+0000 osd.3 (osd.3) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: audit 2024-09-10T18:58:09.072530+0000 mon.c (mon.2) 560 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: audit 2024-09-10T18:58:09.524281+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: audit 2024-09-10T18:58:09.526162+0000 mon.a (mon.0) 2160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: cluster 2024-09-10T18:58:09.526424+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:58:09.900 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[32381]: audit 2024-09-10T18:58:09.528245+0000 mon.a (mon.0) 2162 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: cluster 2024-09-10T18:58:08.607678+0000 osd.3 (osd.3) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: cluster 2024-09-10T18:58:08.608193+0000 osd.3 (osd.3) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: audit 2024-09-10T18:58:09.072530+0000 mon.c (mon.2) 560 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: audit 2024-09-10T18:58:09.524281+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: audit 2024-09-10T18:58:09.526162+0000 mon.a (mon.0) 2160 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: cluster 2024-09-10T18:58:09.526424+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-09-10T18:58:09.901 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:09 smithi083 bash[18629]: audit 2024-09-10T18:58:09.528245+0000 mon.a (mon.0) 2162 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]: dispatch 2024-09-10T18:58:10.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:10.033 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:10.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:10.170 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:10.191 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:10.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.300 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:10.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.439 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:10.542 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:10.655 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:10.677 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:10.707 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:10.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: cluster 2024-09-10T18:58:10.029370+0000 mgr.y (mgr.24526) 363 : cluster [DBG] pgmap v484: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:10.708 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: audit 2024-09-10T18:58:10.074348+0000 mon.c (mon.2) 561 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: cluster 2024-09-10T18:58:10.084754+0000 mon.a (mon.0) 2163 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: audit 2024-09-10T18:58:10.098293+0000 mon.a (mon.0) 2164 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: audit 2024-09-10T18:58:10.098841+0000 mon.a (mon.0) 2165 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]': finished 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: cluster 2024-09-10T18:58:10.099031+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: audit 2024-09-10T18:58:10.211231+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[32381]: audit 2024-09-10T18:58:10.214986+0000 mon.a (mon.0) 2167 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:10.709 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: cluster 2024-09-10T18:58:10.029370+0000 mgr.y (mgr.24526) 363 : cluster [DBG] pgmap v484: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: audit 2024-09-10T18:58:10.074348+0000 mon.c (mon.2) 561 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: cluster 2024-09-10T18:58:10.084754+0000 mon.a (mon.0) 2163 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: audit 2024-09-10T18:58:10.098293+0000 mon.a (mon.0) 2164 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: audit 2024-09-10T18:58:10.098841+0000 mon.a (mon.0) 2165 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]': finished 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: cluster 2024-09-10T18:58:10.099031+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: audit 2024-09-10T18:58:10.211231+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:10.710 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:10 smithi083 bash[18629]: audit 2024-09-10T18:58:10.214986+0000 mon.a (mon.0) 2167 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:10.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:10.836 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:10.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:10.976 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:10.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:10.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: cluster 2024-09-10T18:58:10.029370+0000 mgr.y (mgr.24526) 363 : cluster [DBG] pgmap v484: 247 pgs: 32 unknown, 1 clean+premerge+peered, 214 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:10.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: audit 2024-09-10T18:58:10.074348+0000 mon.c (mon.2) 561 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:10.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: cluster 2024-09-10T18:58:10.084754+0000 mon.a (mon.0) 2163 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:10.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: audit 2024-09-10T18:58:10.098293+0000 mon.a (mon.0) 2164 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi083-77648-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:10.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: audit 2024-09-10T18:58:10.098841+0000 mon.a (mon.0) 2165 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-39", "mode": "writeback"}]': finished 2024-09-10T18:58:10.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: cluster 2024-09-10T18:58:10.099031+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-09-10T18:58:10.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: audit 2024-09-10T18:58:10.211231+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:10.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:10 smithi165 bash[22591]: audit 2024-09-10T18:58:10.214986+0000 mon.a (mon.0) 2167 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:11.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:11.112 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:11.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:11.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:11.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.380 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:11.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:11.519 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:11.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:11.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:11.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[18629]: audit 2024-09-10T18:58:11.075855+0000 mon.c (mon.2) 562 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[18629]: audit 2024-09-10T18:58:11.098873+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[18629]: audit 2024-09-10T18:58:11.100915+0000 mon.a (mon.0) 2168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[18629]: cluster 2024-09-10T18:58:11.101005+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[18629]: audit 2024-09-10T18:58:11.103258+0000 mon.a (mon.0) 2170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[32381]: audit 2024-09-10T18:58:11.075855+0000 mon.c (mon.2) 562 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[32381]: audit 2024-09-10T18:58:11.098873+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[32381]: audit 2024-09-10T18:58:11.100915+0000 mon.a (mon.0) 2168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[32381]: cluster 2024-09-10T18:58:11.101005+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:58:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:11 smithi083 bash[32381]: audit 2024-09-10T18:58:11.103258+0000 mon.a (mon.0) 2170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:11.796 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:11.899 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:11.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:11.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:12.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:12.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[22591]: audit 2024-09-10T18:58:11.075855+0000 mon.c (mon.2) 562 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:12.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[22591]: audit 2024-09-10T18:58:11.098873+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:12.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[22591]: audit 2024-09-10T18:58:11.100915+0000 mon.a (mon.0) 2168 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:58:12.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[22591]: cluster 2024-09-10T18:58:11.101005+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-09-10T18:58:12.046 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:11 smithi165 bash[22591]: audit 2024-09-10T18:58:11.103258+0000 mon.a (mon.0) 2170 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]: dispatch 2024-09-10T18:58:12.075 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:12.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:12.207 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:12.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:12.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:12.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:12.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:12.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:12.616 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:12.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:12.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:12.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: cluster 2024-09-10T18:58:12.030268+0000 mgr.y (mgr.24526) 364 : cluster [DBG] pgmap v487: 254 pgs: 8 creating+peering, 1 peering, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 83 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.030926+0000 mon.c (mon.2) 563 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.033220+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.077284+0000 mon.c (mon.2) 564 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: cluster 2024-09-10T18:58:12.091470+0000 mon.a (mon.0) 2172 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.103323+0000 mon.a (mon.0) 2173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.103957+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-10T18:58:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: cluster 2024-09-10T18:58:12.104183+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:58:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.105610+0000 mon.c (mon.2) 565 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:12 smithi165 bash[22591]: audit 2024-09-10T18:58:12.106023+0000 mon.a (mon.0) 2176 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:12.891 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:12] "GET /metrics HTTP/1.1" 200 231116 "" "Prometheus/2.43.0" 2024-09-10T18:58:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:12.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: cluster 2024-09-10T18:58:12.030268+0000 mgr.y (mgr.24526) 364 : cluster [DBG] pgmap v487: 254 pgs: 8 creating+peering, 1 peering, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 83 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.030926+0000 mon.c (mon.2) 563 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.033220+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.077284+0000 mon.c (mon.2) 564 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: cluster 2024-09-10T18:58:12.091470+0000 mon.a (mon.0) 2172 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:12.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.103323+0000 mon.a (mon.0) 2173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.103957+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: cluster 2024-09-10T18:58:12.104183+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.105610+0000 mon.c (mon.2) 565 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[18629]: audit 2024-09-10T18:58:12.106023+0000 mon.a (mon.0) 2176 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: cluster 2024-09-10T18:58:12.030268+0000 mgr.y (mgr.24526) 364 : cluster [DBG] pgmap v487: 254 pgs: 8 creating+peering, 1 peering, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 83 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.030926+0000 mon.c (mon.2) 563 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.033220+0000 mon.a (mon.0) 2171 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.077284+0000 mon.c (mon.2) 564 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: cluster 2024-09-10T18:58:12.091470+0000 mon.a (mon.0) 2172 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.103323+0000 mon.a (mon.0) 2173 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-39"}]': finished 2024-09-10T18:58:12.893 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.103957+0000 mon.a (mon.0) 2174 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-10T18:58:12.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: cluster 2024-09-10T18:58:12.104183+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-09-10T18:58:12.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.105610+0000 mon.c (mon.2) 565 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.894 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:12 smithi083 bash[32381]: audit 2024-09-10T18:58:12.106023+0000 mon.a (mon.0) 2176 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:12.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:13.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: seed 77828 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-10T18:58:13.117 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (322 ms) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (15363 ms) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (37 ms) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (15722 ms total) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (9752 ms) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (9450 ms) 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-10T18:58:13.118 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10002 ms) 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [4] 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [5,4] 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting some heads 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrubs... 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46419 ms) 2024-09-10T18:58:13.119 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9651 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8010 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8023 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8408 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9560 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9023 ms) 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-10T18:58:13.120 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9984 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13471 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7547 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8003 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13014 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8012 ms) 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-10T18:58:13.121 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8030 ms) 2024-09-10T18:58:13.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:13.156 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:13.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:13.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:13.427 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:13.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:13.562 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:13.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7fd8d1c44640 1 -- 172.21.15.83:0/3532143934 >> v1:172.21.15.83:6789/0 conn(0x7fd8cc1042a0 legacy=0x7fd8cc106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7fd8d1c44640 1 -- 172.21.15.83:0/3532143934 shutdown_connections 2024-09-10T18:58:13.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7fd8d1c44640 1 -- 172.21.15.83:0/3532143934 >> 172.21.15.83:0/3532143934 conn(0x7fd8cc075290 msgr2=0x7fd8cc0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7ff319387640 1 -- 172.21.15.83:0/80002318 >> v1:172.21.15.83:6790/0 conn(0x7ff314108f40 legacy=0x7ff31410b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7ff319387640 1 -- 172.21.15.83:0/80002318 shutdown_connections 2024-09-10T18:58:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7ff319387640 1 -- 172.21.15.83:0/80002318 >> 172.21.15.83:0/80002318 conn(0x7ff314075290 msgr2=0x7ff3140f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7fd8d1c44640 1 -- 172.21.15.83:0/3532143934 shutdown_connections 2024-09-10T18:58:13.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7fd8d1c44640 1 -- 172.21.15.83:0/3532143934 wait complete. 2024-09-10T18:58:13.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7ff319387640 1 -- 172.21.15.83:0/80002318 shutdown_connections 2024-09-10T18:58:13.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.590+0000 7ff319387640 1 -- 172.21.15.83:0/80002318 wait complete. 2024-09-10T18:58:13.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 librados: starting msgr at 2024-09-10T18:58:13.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 librados: starting objecter 2024-09-10T18:58:13.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 Processor -- start 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 -- start start 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 librados: setting wanted keys 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 librados: calling monclient init 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 librados: starting msgr at 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 librados: starting objecter 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 Processor -- start 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 -- start start 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 librados: setting wanted keys 2024-09-10T18:58:13.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 librados: calling monclient init 2024-09-10T18:58:13.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8cc11e220 con 0x7fd8cc1014e0 2024-09-10T18:58:13.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8cc11f3e0 con 0x7fd8cc107e30 2024-09-10T18:58:13.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8cc1205e0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8caffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd8cc1042a0 0x7fd8cc0fd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58100/0 (socket says 172.21.15.83:58100) 2024-09-10T18:58:13.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8caffd640 1 -- 172.21.15.83:0/4292587372 learned_addr learned my addr 172.21.15.83:0/4292587372 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff31411e140 con 0x7ff314108f40 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff31411f2e0 con 0x7ff3140fd640 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff319387640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff3141204c0 con 0x7ff3140fe210 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff3127fc640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff3140fe210 0x7ff314107580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58116/0 (socket says 172.21.15.83:58116) 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff3127fc640 1 -- 172.21.15.83:0/327544357 learned_addr learned my addr 172.21.15.83:0/327544357 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter ms_handle_connect 0x7fd8cc1014e0 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter ms_handle_connect 0x7fd8cc1042a0 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter ms_handle_connect 0x7fd8cc107e30 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff3140fe210 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff314108f40 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter ms_handle_connect 0x7ff3140fd640 2024-09-10T18:58:13.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315689048 0 0) 0x7fd8cc11e220 con 0x7fd8cc1014e0 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b80038b0 con 0x7fd8cc1014e0 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060961802 0 0) 0x7fd8cc1205e0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8cc11e220 con 0x7fd8cc1042a0 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932005890 0 0) 0x7fd8cc11f3e0 con 0x7fd8cc107e30 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8cc1205e0 con 0x7fd8cc107e30 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832961285 0 0) 0x7ff3141204c0 con 0x7ff3140fe210 2024-09-10T18:58:13.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc0034e0 con 0x7ff3140fe210 2024-09-10T18:58:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106034248 0 0) 0x7fd8cc11e220 con 0x7fd8cc1042a0 2024-09-10T18:58:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480661942 0 0) 0x7ff31411e140 con 0x7ff314108f40 2024-09-10T18:58:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3141204c0 con 0x7ff314108f40 2024-09-10T18:58:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077144614 0 0) 0x7ff31411f2e0 con 0x7ff3140fd640 2024-09-10T18:58:13.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31411e140 con 0x7ff3140fd640 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8cc11f3e0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4129663621 0 0) 0x7fd8cc1205e0 con 0x7fd8cc107e30 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8cc11e220 con 0x7fd8cc107e30 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364603488 0 0) 0x7fd8b80038b0 con 0x7fd8cc1014e0 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8cc1205e0 con 0x7fd8cc1014e0 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd8b4002df0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd8c4001bc0 con 0x7fd8cc107e30 2024-09-10T18:58:13.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd8bc001860 con 0x7fd8cc1014e0 2024-09-10T18:58:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309731970 0 0) 0x7ff3141204c0 con 0x7ff314108f40 2024-09-10T18:58:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31411f2e0 con 0x7ff314108f40 2024-09-10T18:58:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3237179168 0 0) 0x7fd8cc11f3e0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff30c0022d0 con 0x7ff314108f40 2024-09-10T18:58:13.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3401703623 0 0) 0x7ff31411e140 con 0x7ff3140fd640 2024-09-10T18:58:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3141204c0 con 0x7ff3140fd640 2024-09-10T18:58:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 >> v1:172.21.15.165:6789/0 conn(0x7fd8cc107e30 legacy=0x7fd8cc11c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 >> v1:172.21.15.83:6789/0 conn(0x7fd8cc1014e0 legacy=0x7fd8cc0fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731896637 0 0) 0x7ff2fc0034e0 con 0x7ff3140fe210 2024-09-10T18:58:13.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff31411e140 con 0x7ff3140fe210 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff304001760 con 0x7ff3140fd640 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff308001a40 con 0x7ff3140fe210 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208471593 0 0) 0x7ff31411f2e0 con 0x7ff314108f40 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc1217e0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc11e410 con 0x7fd8cc1042a0 2024-09-10T18:58:13.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd8b4002340 con 0x7fd8cc1042a0 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6790/0 conn(0x7ff3140fe210 legacy=0x7ff314107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.165:6789/0 conn(0x7ff3140fd640 legacy=0x7ff314106e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.594+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3141216a0 con 0x7ff314108f40 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31411e330 con 0x7ff314108f40 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff319387640 10 client.25152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31411e780 con 0x7ff314108f40 2024-09-10T18:58:13.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff319387640 1 librados: init done 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff30c002680 con 0x7ff314108f40 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff30c003640 con 0x7ff314108f40 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff300004ff0 con 0x7ff314108f40 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff30c001c00 con 0x7ff314108f40 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd8b40035f0 con 0x7fd8cc1042a0 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.598+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd8b4003890 con 0x7fd8cc1042a0 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8d1c44640 10 client.49220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:13.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc11e860 con 0x7fd8cc1042a0 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8d1c44640 1 librados: init done 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(337..337 src has 69..337) v4 ==== 8088+0+0 (unknown 1695776725 0 0) 0x7ff30c088080 con 0x7ff314108f40 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 osd_map(337..337 src has 69..337) v4 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 3 client.25152.objecter handle_osd_map got epochs [337,337] > 0 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 3 client.25152.objecter handle_osd_map decoding full epoch 337 2024-09-10T18:58:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 20 client.25152.objecter dump_active .. 0 homeless 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v337) v1 ==== 54+0+16 (unknown 2759012399 0 2420000192) 0x7ff30c0883f0 con 0x7ff314108f40 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 10 client.25152.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 1 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 1 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 0 1 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 4 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 10 client.25152.objecter _send_command 1 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff300005290 con 0x7ff300004610 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff2fc071770 2024-09-10T18:58:13.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300004610 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(337..337 src has 69..337) v4 ==== 8088+0+0 (unknown 1695776725 0 0) 0x7fd8b408b760 con 0x7fd8cc1042a0 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 10 client.49220.objecter ms_dispatch 0x7fd8cc000b90 osd_map(337..337 src has 69..337) v4 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 3 client.49220.objecter handle_osd_map got epochs [337,337] > 0 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 3 client.49220.objecter handle_osd_map decoding full epoch 337 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 20 client.49220.objecter dump_active .. 0 homeless 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8c3fff640 10 client.49220.objecter ms_handle_connect 0x7fd8b8071a10 2024-09-10T18:58:13.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc0fcc60 con 0x7fd8cc1042a0 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff300005290 con 0x7ff300004610 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 1 = system:0 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 4 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.602+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 0 1 2024-09-10T18:58:13.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.606+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd8b4057e30 con 0x7fd8cc1042a0 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 10 client.25152.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 2 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 2 2024-09-10T18:58:13.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300005810 osd=0 3 2024-09-10T18:58:13.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 0 2 2024-09-10T18:58:13.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 4 2024-09-10T18:58:13.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 10 client.25152.objecter _send_command 2 2024-09-10T18:58:13.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300004c60 con 0x7ff300004610 2024-09-10T18:58:13.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300004c60 con 0x7ff300004610 2024-09-10T18:58:13.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 4 2024-09-10T18:58:13.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 0 2 2024-09-10T18:58:13.625 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 10 client.25152.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 1 3 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 4 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 10 client.25152.objecter _send_command 3 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff300003b10 con 0x7ff3000035e0 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff3000035e0 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff300003b10 con 0x7ff3000035e0 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 3 = system:0 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 4 2024-09-10T18:58:13.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.618+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 1 3 2024-09-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 10 client.25152.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 4 2024-09-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 4 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300004e10 osd=1 3 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 1 4 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 4 2024-09-10T18:58:13.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 10 client.25152.objecter _send_command 4 2024-09-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300003060 con 0x7ff3000035e0 2024-09-10T18:58:13.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300003060 con 0x7ff3000035e0 2024-09-10T18:58:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 4 2024-09-10T18:58:13.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 1 4 2024-09-10T18:58:13.647 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:58:13.648 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:58:13.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 10 client.25152.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 5 2024-09-10T18:58:13.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 5 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 2 5 2024-09-10T18:58:13.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 4 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 10 client.25152.objecter _send_command 5 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.638+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff300002350 con 0x7ff30000d350 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff30000d350 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff300002350 con 0x7ff30000d350 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:58:13.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 5 = system:0 2024-09-10T18:58:13.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 4 2024-09-10T18:58:13.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.642+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 2 5 2024-09-10T18:58:13.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 10 client.25152.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 6 2024-09-10T18:58:13.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 6 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300001b60 osd=2 3 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 2 6 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 4 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 10 client.25152.objecter _send_command 6 2024-09-10T18:58:13.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.654+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300001650 con 0x7ff30000d350 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300001650 con 0x7ff30000d350 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 4 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 2 6 2024-09-10T18:58:13.664 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 10 client.25152.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 7 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 7 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 3 7 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 4 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 10 client.25152.objecter _send_command 7 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff30000d140 con 0x7ff300011fd0 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300011fd0 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff30000d140 con 0x7ff300011fd0 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 7 = system:0 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 4 2024-09-10T18:58:13.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.658+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 3 7 2024-09-10T18:58:13.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:13.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 10 client.25152.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 8 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 8 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000c990 osd=3 3 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 3 8 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 4 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 10 client.25152.objecter _send_command 8 2024-09-10T18:58:13.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff30000d140 con 0x7ff300011fd0 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff30000d140 con 0x7ff300011fd0 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 4 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 3 8 2024-09-10T18:58:13.680 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 10 client.25152.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 9 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 9 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 4 9 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 4 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 10 client.25152.objecter _send_command 9 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff30000c5a0 con 0x7ff300011b20 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.674+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300011b20 2024-09-10T18:58:13.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.678+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff30000c5a0 con 0x7ff300011b20 2024-09-10T18:58:13.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.678+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:58:13.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.678+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 9 = system:0 2024-09-10T18:58:13.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.678+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 4 2024-09-10T18:58:13.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.678+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 4 9 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 10 client.25152.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 10 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 10 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000d140 osd=4 3 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 4 10 2024-09-10T18:58:13.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 4 2024-09-10T18:58:13.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 10 client.25152.objecter _send_command 10 2024-09-10T18:58:13.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.690+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff30000c5a0 con 0x7ff300011b20 2024-09-10T18:58:13.698 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff30000c5a0 con 0x7ff300011b20 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 4 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 4 10 2024-09-10T18:58:13.699 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 10 client.25152.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 11 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 11 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 5 11 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 4 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 10 client.25152.objecter _send_command 11 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff300017d20 con 0x7ff300017870 2024-09-10T18:58:13.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300017870 2024-09-10T18:58:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff300017d20 con 0x7ff300017870 2024-09-10T18:58:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:58:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 11 = system:0 2024-09-10T18:58:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 4 2024-09-10T18:58:13.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.694+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 5 11 2024-09-10T18:58:13.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:13.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 10 client.25152.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 12 2024-09-10T18:58:13.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 12 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff30000c5a0 osd=5 3 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 5 12 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 4 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 10 client.25152.objecter _send_command 12 2024-09-10T18:58:13.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.710+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300017d20 con 0x7ff300017870 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300017d20 con 0x7ff300017870 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 4 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 5 12 2024-09-10T18:58:13.720 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 10 client.25152.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 13 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 13 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 6 13 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 4 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 10 client.25152.objecter _send_command 13 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3000228e0 con 0x7ff300017f40 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.714+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300017f40 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.718+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff3000228e0 con 0x7ff300017f40 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.718+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.718+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 13 = system:0 2024-09-10T18:58:13.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.718+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 4 2024-09-10T18:58:13.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.718+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 6 13 2024-09-10T18:58:13.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 10 client.25152.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 14 2024-09-10T18:58:13.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 14 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300017d20 osd=6 3 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 6 14 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 4 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 10 client.25152.objecter _send_command 14 2024-09-10T18:58:13.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.738+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300024180 con 0x7ff300017f40 2024-09-10T18:58:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300024180 con 0x7ff300017f40 2024-09-10T18:58:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 4 2024-09-10T18:58:13.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 6 14 2024-09-10T18:58:13.749 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:58:13.749 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 10 client.25152.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 15 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 15 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 7 15 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 4 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 10 client.25152.objecter _send_command 15 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff300024d90 con 0x7ff300024860 2024-09-10T18:58:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.742+0000 7ff2fb7fe640 10 client.25152.objecter ms_handle_connect 0x7ff300024860 2024-09-10T18:58:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.746+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff300024d90 con 0x7ff300024860 2024-09-10T18:58:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.746+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:58:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.746+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 15 = system:0 2024-09-10T18:58:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.746+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 4 2024-09-10T18:58:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.746+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 7 15 2024-09-10T18:58:13.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 10 client.25152.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:13.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign -1 16 2024-09-10T18:58:13.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 20 client.25152.objecter _get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 15 client.25152.objecter _session_command_op_remove -1 16 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 20 client.25152.objecter get_session s=0x7ff300024180 osd=7 3 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 15 client.25152.objecter _session_command_op_assign 7 16 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 4 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 10 client.25152.objecter _send_command 16 2024-09-10T18:58:13.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.762+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff300024d90 con 0x7ff300024860 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff2fb7fe640 1 -- 172.21.15.83:0/327544357 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff300024d90 con 0x7ff300024860 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff2fb7fe640 10 client.25152.objecter ms_dispatch 0x7ff314000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff2fb7fe640 10 client.25152.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff2fb7fe640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 4 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff2fb7fe640 15 client.25152.objecter _session_command_op_remove 7 16 2024-09-10T18:58:13.772 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:58:13.773 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 librados: watch_flush enter 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 librados: watch_flush exit 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.0 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6801/2535527483 conn(0x7ff300004610 legacy=0x7ff300002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300005810 osd=0 1 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.1 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6805/2309624637 conn(0x7ff3000035e0 legacy=0x7ff30000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300004e10 osd=1 1 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.2 2024-09-10T18:58:13.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6809/386094220 conn(0x7ff30000d350 legacy=0x7ff30000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300001b60 osd=2 1 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.3 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6813/1279688021 conn(0x7ff300011fd0 legacy=0x7ff3000143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c990 osd=3 1 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.4 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.165:6800/3225515265 conn(0x7ff300011b20 legacy=0x7ff300016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000d140 osd=4 1 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.5 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.165:6804/571173777 conn(0x7ff300017870 legacy=0x7ff30001d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff3137fe640 1 -- 172.21.15.83:0/327544357 reap_dead start 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff3137fe640 1 -- 172.21.15.83:0/327544357 reap_dead start 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff30000c5a0 osd=5 1 2024-09-10T18:58:13.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.6 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.165:6808/1317922936 conn(0x7ff300017f40 legacy=0x7ff300021060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300017d20 osd=6 1 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter close_session for osd.7 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff300024860 legacy=0x7ff3000228e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter put_session s=0x7ff300024180 osd=7 1 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 20 client.25152.objecter shutdown clearing up homeless session... 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 10 client.25152.objecter successfully canceled tick 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff3137fe640 1 -- 172.21.15.83:0/327544357 reap_dead start 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff2fc071770 legacy=0x7ff2fc073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> v1:172.21.15.83:6789/0 conn(0x7ff314108f40 legacy=0x7ff31411c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 shutdown_connections 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 >> 172.21.15.83:0/327544357 conn(0x7ff314075290 msgr2=0x7ff3140f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:13.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 shutdown_connections 2024-09-10T18:58:13.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 -- 172.21.15.83:0/327544357 wait complete. 2024-09-10T18:58:13.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.766+0000 7ff319387640 1 librados: shutdown 2024-09-10T18:58:13.799 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: cluster 2024-09-10T18:58:12.633995+0000 mon.a (mon.0) 2177 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:13.799 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: audit 2024-09-10T18:58:13.078668+0000 mon.c (mon.2) 566 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: audit 2024-09-10T18:58:13.111879+0000 mon.a (mon.0) 2178 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: cluster 2024-09-10T18:58:13.112151+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: audit 2024-09-10T18:58:13.113693+0000 mon.c (mon.2) 567 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: audit 2024-09-10T18:58:13.116448+0000 mon.a (mon.0) 2180 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[18629]: audit 2024-09-10T18:58:13.604317+0000 mon.a (mon.0) 2181 : audit [DBG] from='client.? v1:172.21.15.83:0/327544357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: cluster 2024-09-10T18:58:12.633995+0000 mon.a (mon.0) 2177 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: audit 2024-09-10T18:58:13.078668+0000 mon.c (mon.2) 566 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: audit 2024-09-10T18:58:13.111879+0000 mon.a (mon.0) 2178 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: cluster 2024-09-10T18:58:13.112151+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:58:13.800 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: audit 2024-09-10T18:58:13.113693+0000 mon.c (mon.2) 567 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: audit 2024-09-10T18:58:13.116448+0000 mon.a (mon.0) 2180 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.801 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:13 smithi083 bash[32381]: audit 2024-09-10T18:58:13.604317+0000 mon.a (mon.0) 2181 : audit [DBG] from='client.? v1:172.21.15.83:0/327544357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:13.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:13.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:13.810+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd8cc04bf60 con 0x7fd8cc1042a0 2024-09-10T18:58:13.840 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:13.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: cluster 2024-09-10T18:58:12.633995+0000 mon.a (mon.0) 2177 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: audit 2024-09-10T18:58:13.078668+0000 mon.c (mon.2) 566 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: audit 2024-09-10T18:58:13.111879+0000 mon.a (mon.0) 2178 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: cluster 2024-09-10T18:58:13.112151+0000 mon.a (mon.0) 2179 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: audit 2024-09-10T18:58:13.113693+0000 mon.c (mon.2) 567 : audit [INF] from='client.? v1:172.21.15.83:0/613988482' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: audit 2024-09-10T18:58:13.116448+0000 mon.a (mon.0) 2180 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]: dispatch 2024-09-10T18:58:13.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:13 smithi165 bash[22591]: audit 2024-09-10T18:58:13.604317+0000 mon.a (mon.0) 2181 : audit [DBG] from='client.? v1:172.21.15.83:0/327544357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:13.977 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:13.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:14.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.066+0000 7fd8c3fff640 1 -- 172.21.15.83:0/4292587372 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v338) v1 ==== 97+0+0 (unknown 101859782 0 0) 0x7fd8b40577d0 con 0x7fd8cc1042a0 2024-09-10T18:58:14.074 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T18:58:14.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 10 librados: watch_flush enter 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 10 librados: watch_flush exit 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 20 client.49220.objecter shutdown clearing up homeless session... 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 10 client.49220.objecter successfully canceled tick 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd8b8071a10 legacy=0x7fd8b8073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 >> v1:172.21.15.83:6790/0 conn(0x7fd8cc1042a0 legacy=0x7fd8cc0fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 shutdown_connections 2024-09-10T18:58:14.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.082+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 >> 172.21.15.83:0/4292587372 conn(0x7fd8cc075290 msgr2=0x7fd8cc0f9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:14.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.086+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 shutdown_connections 2024-09-10T18:58:14.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.094+0000 7fd8d1c44640 1 -- 172.21.15.83:0/4292587372 wait complete. 2024-09-10T18:58:14.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:14.094+0000 7fd8d1c44640 1 librados: shutdown 2024-09-10T18:58:14.115 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:14.197 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:58:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:58:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T18:58:14.197 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:58:14.218 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:14.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:14.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:14.383 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:14.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.518 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:14.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.652 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:14.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:14.792 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:14.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:13.815336+0000 mon.c (mon.2) 568 : audit [INF] from='client.? v1:172.21.15.83:0/4292587372' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:13.815829+0000 mon.a (mon.0) 2182 : audit [INF] from='client.49220 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: cluster 2024-09-10T18:58:14.031276+0000 mgr.y (mgr.24526) 365 : cluster [DBG] pgmap v490: 214 pgs: 1 peering, 213 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 978 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: cluster 2024-09-10T18:58:14.061374+0000 mon.a (mon.0) 2183 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.072201+0000 mon.a (mon.0) 2184 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.072386+0000 mon.a (mon.0) 2185 : audit [INF] from='client.49220 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: cluster 2024-09-10T18:58:14.072467+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.072936+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.077834+0000 mon.a (mon.0) 2187 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.084734+0000 mon.c (mon.2) 569 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.108018+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:14.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.110486+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:14.909 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.110975+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:14.909 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:14 smithi165 bash[22591]: audit 2024-09-10T18:58:14.561768+0000 mon.c (mon.2) 570 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:14.927 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:14.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:14.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T18:58:15.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:13.815336+0000 mon.c (mon.2) 568 : audit [INF] from='client.? v1:172.21.15.83:0/4292587372' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:13.815829+0000 mon.a (mon.0) 2182 : audit [INF] from='client.49220 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: cluster 2024-09-10T18:58:14.031276+0000 mgr.y (mgr.24526) 365 : cluster [DBG] pgmap v490: 214 pgs: 1 peering, 213 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 978 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: cluster 2024-09-10T18:58:14.061374+0000 mon.a (mon.0) 2183 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.072201+0000 mon.a (mon.0) 2184 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.072386+0000 mon.a (mon.0) 2185 : audit [INF] from='client.49220 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: cluster 2024-09-10T18:58:14.072467+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.072936+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.077834+0000 mon.a (mon.0) 2187 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:15.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.084734+0000 mon.c (mon.2) 569 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.108018+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.110486+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.110975+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[32381]: audit 2024-09-10T18:58:14.561768+0000 mon.c (mon.2) 570 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:13.815336+0000 mon.c (mon.2) 568 : audit [INF] from='client.? v1:172.21.15.83:0/4292587372' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:13.815829+0000 mon.a (mon.0) 2182 : audit [INF] from='client.49220 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: cluster 2024-09-10T18:58:14.031276+0000 mgr.y (mgr.24526) 365 : cluster [DBG] pgmap v490: 214 pgs: 1 peering, 213 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 978 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: cluster 2024-09-10T18:58:14.061374+0000 mon.a (mon.0) 2183 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.072201+0000 mon.a (mon.0) 2184 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi083-77648-55"}]': finished 2024-09-10T18:58:15.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.072386+0000 mon.a (mon.0) 2185 : audit [INF] from='client.49220 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: cluster 2024-09-10T18:58:14.072467+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.072936+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.077834+0000 mon.a (mon.0) 2187 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.084734+0000 mon.c (mon.2) 569 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.108018+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.110486+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.110975+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:15.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:14 smithi083 bash[18629]: audit 2024-09-10T18:58:14.561768+0000 mon.c (mon.2) 570 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:15.056 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:15.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:15.184 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:15.287 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:15.318 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:15.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:15.457 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:15.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:15.595 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:15.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:15.736 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:15.840 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:15.872 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:15.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:15.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.009 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:16.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.146 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:16.250 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:16.279 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:16.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:16.411 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:16.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:16.546 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:16.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:16.684 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:16.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.076743+0000 mon.a (mon.0) 2191 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.076978+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: cluster 2024-09-10T18:58:15.077043+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.083112+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.085750+0000 mon.c (mon.2) 571 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.086055+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:15.090429+0000 mon.a (mon.0) 2195 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:16.034255+0000 mon.c (mon.2) 572 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[18629]: audit 2024-09-10T18:58:16.036406+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.076743+0000 mon.a (mon.0) 2191 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.076978+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: cluster 2024-09-10T18:58:15.077043+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.083112+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.085750+0000 mon.c (mon.2) 571 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.086055+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:15.090429+0000 mon.a (mon.0) 2195 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:16.034255+0000 mon.c (mon.2) 572 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:16 smithi083 bash[32381]: audit 2024-09-10T18:58:16.036406+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.076743+0000 mon.a (mon.0) 2191 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.076978+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi083-77648-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: cluster 2024-09-10T18:58:15.077043+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.083112+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.085750+0000 mon.c (mon.2) 571 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.086055+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:15.090429+0000 mon.a (mon.0) 2195 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:16.034255+0000 mon.c (mon.2) 572 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:16 smithi165 bash[22591]: audit 2024-09-10T18:58:16.036406+0000 mon.a (mon.0) 2196 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-10T18:58:16.823 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:16.928 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:16.958 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:16.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:17.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.100 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:17.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:17.240 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:17.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:17.379 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:17.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.510 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:17.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- 172.21.15.83:0/3486296699 >> v1:172.21.15.83:6789/0 conn(0x7ff980100c60 legacy=0x7ff980101030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- 172.21.15.83:0/3486296699 shutdown_connections 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- 172.21.15.83:0/3486296699 >> 172.21.15.83:0/3486296699 conn(0x7ff980074a40 msgr2=0x7ff9800f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- 172.21.15.83:0/3486296699 shutdown_connections 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- 172.21.15.83:0/3486296699 wait complete. 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 librados: starting msgr at 2024-09-10T18:58:17.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 librados: starting objecter 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 Processor -- start 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- start start 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 librados: setting wanted keys 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 librados: calling monclient init 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff980119e90 con 0x7ff9801075a0 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff98011b090 con 0x7ff980100c60 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff987328640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff98011c290 con 0x7ff980103a40 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff9858c5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff9801075a0 0x7ff980118590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60794/0 (socket says 172.21.15.83:60794) 2024-09-10T18:58:17.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff9848c3640 1 --1- 172.21.15.83:0/3157946003 >> v1:172.21.15.83:6790/0 conn(0x7ff980103a40 0x7ff980114be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58146/0 (socket says 172.21.15.83:58146) 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff9858c5640 1 -- 172.21.15.83:0/3157946003 learned_addr learned my addr 172.21.15.83:0/3157946003 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter ms_handle_connect 0x7ff980103a40 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter ms_handle_connect 0x7ff9801075a0 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter ms_handle_connect 0x7ff980100c60 2024-09-10T18:58:17.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:17.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224179405 0 0) 0x7ff980119e90 con 0x7ff9801075a0 2024-09-10T18:58:17.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.510+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c003120 con 0x7ff9801075a0 2024-09-10T18:58:17.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.514+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2850503051 0 0) 0x7ff96c003120 con 0x7ff9801075a0 2024-09-10T18:58:17.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.514+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff980119e90 con 0x7ff9801075a0 2024-09-10T18:58:17.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.514+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff978001d70 con 0x7ff9801075a0 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2079000358 0 0) 0x7ff980119e90 con 0x7ff9801075a0 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 >> v1:172.21.15.83:6790/0 conn(0x7ff980103a40 legacy=0x7ff980114be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 >> v1:172.21.15.165:6789/0 conn(0x7ff980100c60 legacy=0x7ff980100140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98011f6a0 con 0x7ff9801075a0 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98011a080 con 0x7ff9801075a0 2024-09-10T18:58:17.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff987328640 10 client.25164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:17.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98011a510 con 0x7ff9801075a0 2024-09-10T18:58:17.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff987328640 1 librados: init done 2024-09-10T18:58:17.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9800fc3d0 con 0x7ff9801075a0 2024-09-10T18:58:17.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff9780026a0 con 0x7ff9801075a0 2024-09-10T18:58:17.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.518+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff978005ca0 con 0x7ff9801075a0 2024-09-10T18:58:17.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff97801c0c0 con 0x7ff9801075a0 2024-09-10T18:58:17.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(341..341 src has 69..341) v4 ==== 9157+0+0 (unknown 2563968617 0 0) 0x7ff97808ad40 con 0x7ff9801075a0 2024-09-10T18:58:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 10 client.25164.objecter ms_dispatch 0x7ff980000b90 osd_map(341..341 src has 69..341) v4 2024-09-10T18:58:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 3 client.25164.objecter handle_osd_map got epochs [341,341] > 0 2024-09-10T18:58:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 3 client.25164.objecter handle_osd_map decoding full epoch 341 2024-09-10T18:58:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 20 client.25164.objecter dump_active .. 0 homeless 2024-09-10T18:58:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.522+0000 7ff975ffb640 10 client.25164.objecter ms_handle_connect 0x7ff96c071790 2024-09-10T18:58:17.533 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.526+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff978056e80 con 0x7ff9801075a0 2024-09-10T18:58:17.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: cluster 2024-09-10T18:58:15.681112+0000 osd.4 (osd.4) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: cluster 2024-09-10T18:58:15.681385+0000 osd.4 (osd.4) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: cluster 2024-09-10T18:58:16.033580+0000 mgr.y (mgr.24526) 366 : cluster [DBG] pgmap v493: 245 pgs: 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:16.086908+0000 mon.c (mon.2) 573 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:16.512816+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.627 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:16.514204+0000 mon.a (mon.0) 2197 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:16.514294+0000 mon.a (mon.0) 2198 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: cluster 2024-09-10T18:58:16.514343+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:16.525728+0000 mon.a (mon.0) 2200 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.083578+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.084001+0000 mon.a (mon.0) 2202 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: cluster 2024-09-10T18:58:17.084103+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.086675+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.090371+0000 mon.c (mon.2) 574 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.090550+0000 mon.a (mon.0) 2204 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.091797+0000 mon.c (mon.2) 575 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.628 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[32381]: audit 2024-09-10T18:58:17.095124+0000 mon.a (mon.0) 2205 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: cluster 2024-09-10T18:58:15.681112+0000 osd.4 (osd.4) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: cluster 2024-09-10T18:58:15.681385+0000 osd.4 (osd.4) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: cluster 2024-09-10T18:58:16.033580+0000 mgr.y (mgr.24526) 366 : cluster [DBG] pgmap v493: 245 pgs: 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:16.086908+0000 mon.c (mon.2) 573 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:16.512816+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:16.514204+0000 mon.a (mon.0) 2197 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:16.514294+0000 mon.a (mon.0) 2198 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: cluster 2024-09-10T18:58:16.514343+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:16.525728+0000 mon.a (mon.0) 2200 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.083578+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.084001+0000 mon.a (mon.0) 2202 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: cluster 2024-09-10T18:58:17.084103+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:58:17.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.086675+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.630 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.090371+0000 mon.c (mon.2) 574 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.630 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.090550+0000 mon.a (mon.0) 2204 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.630 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.091797+0000 mon.c (mon.2) 575 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.630 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:17 smithi083 bash[18629]: audit 2024-09-10T18:58:17.095124+0000 mon.a (mon.0) 2205 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:17.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.702+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff98010c8b0 con 0x7ff96c071790 2024-09-10T18:58:17.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.730+0000 7ff975ffb640 1 -- 172.21.15.83:0/3157946003 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+687824 (unknown 2965378022 0 3742829543) 0x7ff98010c8b0 con 0x7ff96c071790 2024-09-10T18:58:17.742 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:58:17.745 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 10 librados: watch_flush enter 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 10 librados: watch_flush exit 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 20 client.25164.objecter shutdown clearing up homeless session... 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 10 client.25164.objecter successfully canceled tick 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff96c071790 legacy=0x7ff96c073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:17.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 >> v1:172.21.15.83:6789/0 conn(0x7ff9801075a0 legacy=0x7ff980118590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:17.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 shutdown_connections 2024-09-10T18:58:17.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 >> 172.21.15.83:0/3157946003 conn(0x7ff980074a40 msgr2=0x7ff98010a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:17.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 shutdown_connections 2024-09-10T18:58:17.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 -- 172.21.15.83:0/3157946003 wait complete. 2024-09-10T18:58:17.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:17.738+0000 7ff987328640 1 librados: shutdown 2024-09-10T18:58:17.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:17.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: cluster 2024-09-10T18:58:15.681112+0000 osd.4 (osd.4) 233 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: cluster 2024-09-10T18:58:15.681385+0000 osd.4 (osd.4) 234 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: cluster 2024-09-10T18:58:16.033580+0000 mgr.y (mgr.24526) 366 : cluster [DBG] pgmap v493: 245 pgs: 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:16.086908+0000 mon.c (mon.2) 573 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:16.512816+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:16.514204+0000 mon.a (mon.0) 2197 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:16.514294+0000 mon.a (mon.0) 2198 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: cluster 2024-09-10T18:58:16.514343+0000 mon.a (mon.0) 2199 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2024-09-10T18:58:17.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:16.525728+0000 mon.a (mon.0) 2200 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.083578+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi083-77648-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.084001+0000 mon.a (mon.0) 2202 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: cluster 2024-09-10T18:58:17.084103+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.086675+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.090371+0000 mon.c (mon.2) 574 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.090550+0000 mon.a (mon.0) 2204 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.091797+0000 mon.c (mon.2) 575 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[22591]: audit 2024-09-10T18:58:17.095124+0000 mon.a (mon.0) 2205 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T18:58:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:17.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:58:17.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:58:17.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:17.918 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:17.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:18.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:18.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:18.058 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:18.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:18.192 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:18.296 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:18.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:18.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:18.471 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:18.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:18.594 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-10T18:58:17.077048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8793765,"num_objects":337,"num_object_clones":2,"num_object_copies":1011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":1491,"num_read_kb":1252,"num_write":1233,"num_write_kb":14394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":591740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2492,"ondisk_log_size":2492,"up":735,"acting":735,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":732,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2504608,"kb_used_data":67400,"kb_used_omap":13,"kb_used_meta":2436978,"kb_avail":747456608,"statfs":{"total":767960285184,"available":765395566592,"internally_reserved":0,"allocated":69017600,"data_stored":50011847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2495466176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.971573"},"pg_stats":[{"pgid":"192.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"192.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:58:17.070512+0000","last_change":"2024-09-10T18:58:17.070512+0000","last_active":"2024-09-10T18:58:17.070512+0000","last_peered":"2024-09-10T18:58:17.070512+0000","last_clean":"2024-09-10T18:58:17.070512+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:58:17.070512+0000","last_undegraded":"2024-09-10T18:58:17.070512+0000","last_fullsized":"2024-09-10T18:58:17.070512+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:17.070512+0000","last_clean_scrub_stamp":"2024-09-10T18:58:17.070512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.1f","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.093201+0000","last_change":"2024-09-10T18:58:15.093201+0000","last_active":"2024-09-10T18:58:15.093201+0000","last_peered":"2024-09-10T18:58:15.093201+0000","last_clean":"2024-09-10T18:58:15.093201+0000","last_became_active":"2024-09-10T18:58:15.092929+0000","last_became_peered":"2024-09-10T18:58:15.092929+0000","last_unstale":"2024-09-10T18:58:15.093201+0000","last_undegraded":"2024-09-10T18:58:15.093201+0000","last_fullsized":"2024-09-10T18:58:15.093201+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.659735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.1e","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.099694+0000","last_change":"2024-09-10T18:58:15.099694+0000","last_active":"2024-09-10T18:58:15.099694+0000","last_peered":"2024-09-10T18:58:15.099694+0000","last_clean":"2024-09-10T18:58:15.099694+0000","last_became_active":"2024-09-10T18:58:15.099180+0000","last_became_peered":"2024-09-10T18:58:15.099180+0000","last_unstale":"2024-09-10T18:58:15.099694+0000","last_undegraded":"2024-09-10T18:58:15.099694+0000","last_fullsized":"2024-09-10T18:58:15.099694+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:25.090101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"191.1d","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.097898+0000","last_change":"2024-09-10T18:58:15.097898+0000","last_active":"2024-09-10T18:58:15.097898+0000","last_peered":"2024-09-10T18:58:15.097898+0000","last_clean":"2024-09-10T18:58:15.097898+0000","last_became_active":"2024-09-10T18:58:15.097622+0000","last_became_peered":"2024-09-10T18:58:15.097622+0000","last_unstale":"2024-09-10T18:58:15.097898+0000","last_undegraded":"2024-09-10T18:58:15.097898+0000","last_fullsized":"2024-09-10T18:58:15.097898+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.889263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"191.1c","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.091533+0000","last_change":"2024-09-10T18:58:15.091533+0000","last_active":"2024-09-10T18:58:15.091533+0000","last_peered":"2024-09-10T18:58:15.091533+0000","last_clean":"2024-09-10T18:58:15.091533+0000","last_became_active":"2024-09-10T18:58:15.091254+0000","last_became_peered":"2024-09-10T18:58:15.091254+0000","last_unstale":"2024-09-10T18:58:15.091533+0000","last_undegraded":"2024-09-10T18:58:15.091533+0000","last_fullsized":"2024-09-10T18:58:15.091533+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:21.839667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.1b","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.093949+0000","last_change":"2024-09-10T18:58:15.093949+0000","last_active":"2024-09-10T18:58:15.093949+0000","last_peered":"2024-09-10T18:58:15.093949+0000","last_clean":"2024-09-10T18:58:15.093949+0000","last_became_active":"2024-09-10T18:58:15.092201+0000","last_became_peered":"2024-09-10T18:58:15.092201+0000","last_unstale":"2024-09-10T18:58:15.093949+0000","last_undegraded":"2024-09-10T18:58:15.093949+0000","last_fullsized":"2024-09-10T18:58:15.093949+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:37.366887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.1a","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.095373+0000","last_change":"2024-09-10T18:58:15.095373+0000","last_active":"2024-09-10T18:58:15.095373+0000","last_peered":"2024-09-10T18:58:15.095373+0000","last_clean":"2024-09-10T18:58:15.095373+0000","last_became_active":"2024-09-10T18:58:15.094842+0000","last_became_peered":"2024-09-10T18:58:15.094842+0000","last_unstale":"2024-09-10T18:58:15.095373+0000","last_undegraded":"2024-09-10T18:58:15.095373+0000","last_fullsized":"2024-09-10T18:58:15.095373+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:34.732808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.19","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.088476+0000","last_change":"2024-09-10T18:58:15.088476+0000","last_active":"2024-09-10T18:58:15.088476+0000","last_peered":"2024-09-10T18:58:15.088476+0000","last_clean":"2024-09-10T18:58:15.088476+0000","last_became_active":"2024-09-10T18:58:15.088199+0000","last_became_peered":"2024-09-10T18:58:15.088199+0000","last_unstale":"2024-09-10T18:58:15.088476+0000","last_undegraded":"2024-09-10T18:58:15.088476+0000","last_fullsized":"2024-09-10T18:58:15.088476+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.474546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.18","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.084220+0000","last_change":"2024-09-10T18:58:15.084220+0000","last_active":"2024-09-10T18:58:15.084220+0000","last_peered":"2024-09-10T18:58:15.084220+0000","last_clean":"2024-09-10T18:58:15.084220+0000","last_became_active":"2024-09-10T18:58:15.083933+0000","last_became_peered":"2024-09-10T18:58:15.083933+0000","last_unstale":"2024-09-10T18:58:15.084220+0000","last_undegraded":"2024-09-10T18:58:15.084220+0000","last_fullsized":"2024-09-10T18:58:15.084220+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:34.748346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.17","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.100367+0000","last_change":"2024-09-10T18:58:15.100367+0000","last_active":"2024-09-10T18:58:15.100367+0000","last_peered":"2024-09-10T18:58:15.100367+0000","last_clean":"2024-09-10T18:58:15.100367+0000","last_became_active":"2024-09-10T18:58:15.100050+0000","last_became_peered":"2024-09-10T18:58:15.100050+0000","last_unstale":"2024-09-10T18:58:15.100367+0000","last_undegraded":"2024-09-10T18:58:15.100367+0000","last_fullsized":"2024-09-10T18:58:15.100367+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.214462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"191.16","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.100221+0000","last_change":"2024-09-10T18:58:15.100221+0000","last_active":"2024-09-10T18:58:15.100221+0000","last_peered":"2024-09-10T18:58:15.100221+0000","last_clean":"2024-09-10T18:58:15.100221+0000","last_became_active":"2024-09-10T18:58:15.099918+0000","last_became_peered":"2024-09-10T18:58:15.099918+0000","last_unstale":"2024-09-10T18:58:15.100221+0000","last_undegraded":"2024-09-10T18:58:15.100221+0000","last_fullsized":"2024-09-10T18:58:15.100221+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:38.399129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"191.15","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.092544+0000","last_change":"2024-09-10T18:58:15.092544+0000","last_active":"2024-09-10T18:58:15.092544+0000","last_peered":"2024-09-10T18:58:15.092544+0000","last_clean":"2024-09-10T18:58:15.092544+0000","last_became_active":"2024-09-10T18:58:15.092171+0000","last_became_peered":"2024-09-10T18:58:15.092171+0000","last_unstale":"2024-09-10T18:58:15.092544+0000","last_undegraded":"2024-09-10T18:58:15.092544+0000","last_fullsized":"2024-09-10T18:58:15.092544+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:36.555296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.14","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.090117+0000","last_change":"2024-09-10T18:58:15.090117+0000","last_active":"2024-09-10T18:58:15.090117+0000","last_peered":"2024-09-10T18:58:15.090117+0000","last_clean":"2024-09-10T18:58:15.090117+0000","last_became_active":"2024-09-10T18:58:15.088855+0000","last_became_peered":"2024-09-10T18:58:15.088855+0000","last_unstale":"2024-09-10T18:58:15.090117+0000","last_undegraded":"2024-09-10T18:58:15.090117+0000","last_fullsized":"2024-09-10T18:58:15.090117+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:25.965704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.13","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.092213+0000","last_change":"2024-09-10T18:58:15.092213+0000","last_active":"2024-09-10T18:58:15.092213+0000","last_peered":"2024-09-10T18:58:15.092213+0000","last_clean":"2024-09-10T18:58:15.092213+0000","last_became_active":"2024-09-10T18:58:15.091901+0000","last_became_peered":"2024-09-10T18:58:15.091901+0000","last_unstale":"2024-09-10T18:58:15.092213+0000","last_undegraded":"2024-09-10T18:58:15.092213+0000","last_fullsized":"2024-09-10T18:58:15.092213+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:14.637010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.12","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.087122+0000","last_change":"2024-09-10T18:58:15.087122+0000","last_active":"2024-09-10T18:58:15.087122+0000","last_peered":"2024-09-10T18:58:15.087122+0000","last_clean":"2024-09-10T18:58:15.087122+0000","last_became_active":"2024-09-10T18:58:15.085869+0000","last_became_peered":"2024-09-10T18:58:15.085869+0000","last_unstale":"2024-09-10T18:58:15.087122+0000","last_undegraded":"2024-09-10T18:58:15.087122+0000","last_fullsized":"2024-09-10T18:58:15.087122+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:18.164702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.11","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.101651+0000","last_change":"2024-09-10T18:58:15.101651+0000","last_active":"2024-09-10T18:58:15.101651+0000","last_peered":"2024-09-10T18:58:15.101651+0000","last_clean":"2024-09-10T18:58:15.101651+0000","last_became_active":"2024-09-10T18:58:15.101303+0000","last_became_peered":"2024-09-10T18:58:15.101303+0000","last_unstale":"2024-09-10T18:58:15.101651+0000","last_undegraded":"2024-09-10T18:58:15.101651+0000","last_fullsized":"2024-09-10T18:58:15.101651+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:30.792668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"191.10","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.100128+0000","last_change":"2024-09-10T18:58:15.100128+0000","last_active":"2024-09-10T18:58:15.100128+0000","last_peered":"2024-09-10T18:58:15.100128+0000","last_clean":"2024-09-10T18:58:15.100128+0000","last_became_active":"2024-09-10T18:58:15.099832+0000","last_became_peered":"2024-09-10T18:58:15.099832+0000","last_unstale":"2024-09-10T18:58:15.100128+0000","last_undegraded":"2024-09-10T18:58:15.100128+0000","last_fullsized":"2024-09-10T18:58:15.100128+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.435662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"191.f","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.093257+0000","last_change":"2024-09-10T18:58:15.093257+0000","last_active":"2024-09-10T18:58:15.093257+0000","last_peered":"2024-09-10T18:58:15.093257+0000","last_clean":"2024-09-10T18:58:15.093257+0000","last_became_active":"2024-09-10T18:58:15.092989+0000","last_became_peered":"2024-09-10T18:58:15.092989+0000","last_unstale":"2024-09-10T18:58:15.093257+0000","last_undegraded":"2024-09-10T18:58:15.093257+0000","last_fullsized":"2024-09-10T18:58:15.093257+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.374511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.e","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.079350+0000","last_change":"2024-09-10T18:58:15.079350+0000","last_active":"2024-09-10T18:58:15.079350+0000","last_peered":"2024-09-10T18:58:15.079350+0000","last_clean":"2024-09-10T18:58:15.079350+0000","last_became_active":"2024-09-10T18:58:15.079022+0000","last_became_peered":"2024-09-10T18:58:15.079022+0000","last_unstale":"2024-09-10T18:58:15.079350+0000","last_undegraded":"2024-09-10T18:58:15.079350+0000","last_fullsized":"2024-09-10T18:58:15.079350+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.461209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.d","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.101770+0000","last_change":"2024-09-10T18:58:15.101770+0000","last_active":"2024-09-10T18:58:15.101770+0000","last_peered":"2024-09-10T18:58:15.101770+0000","last_clean":"2024-09-10T18:58:15.101770+0000","last_became_active":"2024-09-10T18:58:15.101455+0000","last_became_peered":"2024-09-10T18:58:15.101455+0000","last_unstale":"2024-09-10T18:58:15.101770+0000","last_undegraded":"2024-09-10T18:58:15.101770+0000","last_fullsized":"2024-09-10T18:58:15.101770+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.312644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"191.c","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.084933+0000","last_change":"2024-09-10T18:58:15.084933+0000","last_active":"2024-09-10T18:58:15.084933+0000","last_peered":"2024-09-10T18:58:15.084933+0000","last_clean":"2024-09-10T18:58:15.084933+0000","last_became_active":"2024-09-10T18:58:15.084581+0000","last_became_peered":"2024-09-10T18:58:15.084581+0000","last_unstale":"2024-09-10T18:58:15.084933+0000","last_undegraded":"2024-09-10T18:58:15.084933+0000","last_fullsized":"2024-09-10T18:58:15.084933+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.335177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"191.b","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.095750+0000","last_change":"2024-09-10T18:58:15.095750+0000","last_active":"2024-09-10T18:58:15.095750+0000","last_peered":"2024-09-10T18:58:15.095750+0000","last_clean":"2024-09-10T18:58:15.095750+0000","last_became_active":"2024-09-10T18:58:15.095376+0000","last_became_peered":"2024-09-10T18:58:15.095376+0000","last_unstale":"2024-09-10T18:58:15.095750+0000","last_undegraded":"2024-09-10T18:58:15.095750+0000","last_fullsized":"2024-09-10T18:58:15.095750+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:14.634789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.a","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.089874+0000","last_change":"2024-09-10T18:58:15.089874+0000","last_active":"2024-09-10T18:58:15.089874+0000","last_peered":"2024-09-10T18:58:15.089874+0000","last_clean":"2024-09-10T18:58:15.089874+0000","last_became_active":"2024-09-10T18:58:15.089573+0000","last_became_peered":"2024-09-10T18:58:15.089573+0000","last_unstale":"2024-09-10T18:58:15.089874+0000","last_undegraded":"2024-09-10T18:58:15.089874+0000","last_fullsized":"2024-09-10T18:58:15.089874+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.311059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"191.9","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.095133+0000","last_change":"2024-09-10T18:58:15.095133+0000","last_active":"2024-09-10T18:58:15.095133+0000","last_peered":"2024-09-10T18:58:15.095133+0000","last_clean":"2024-09-10T18:58:15.095133+0000","last_became_active":"2024-09-10T18:58:15.094830+0000","last_became_peered":"2024-09-10T18:58:15.094830+0000","last_unstale":"2024-09-10T18:58:15.095133+0000","last_undegraded":"2024-09-10T18:58:15.095133+0000","last_fullsized":"2024-09-10T18:58:15.095133+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:38.410757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.8","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.088724+0000","last_change":"2024-09-10T18:58:15.088724+0000","last_active":"2024-09-10T18:58:15.088724+0000","last_peered":"2024-09-10T18:58:15.088724+0000","last_clean":"2024-09-10T18:58:15.088724+0000","last_became_active":"2024-09-10T18:58:15.088436+0000","last_became_peered":"2024-09-10T18:58:15.088436+0000","last_unstale":"2024-09-10T18:58:15.088724+0000","last_undegraded":"2024-09-10T18:58:15.088724+0000","last_fullsized":"2024-09-10T18:58:15.088724+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.717254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"191.7","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.091052+0000","last_change":"2024-09-10T18:58:15.091052+0000","last_active":"2024-09-10T18:58:15.091052+0000","last_peered":"2024-09-10T18:58:15.091052+0000","last_clean":"2024-09-10T18:58:15.091052+0000","last_became_active":"2024-09-10T18:58:15.090707+0000","last_became_peered":"2024-09-10T18:58:15.090707+0000","last_unstale":"2024-09-10T18:58:15.091052+0000","last_undegraded":"2024-09-10T18:58:15.091052+0000","last_fullsized":"2024-09-10T18:58:15.091052+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:32.389090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.6","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.098657+0000","last_change":"2024-09-10T18:58:15.098657+0000","last_active":"2024-09-10T18:58:15.098657+0000","last_peered":"2024-09-10T18:58:15.098657+0000","last_clean":"2024-09-10T18:58:15.098657+0000","last_became_active":"2024-09-10T18:58:15.098015+0000","last_became_peered":"2024-09-10T18:58:15.098015+0000","last_unstale":"2024-09-10T18:58:15.098657+0000","last_undegraded":"2024-09-10T18:58:15.098657+0000","last_fullsized":"2024-09-10T18:58:15.098657+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:38.788833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"191.5","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.086978+0000","last_change":"2024-09-10T18:58:15.086978+0000","last_active":"2024-09-10T18:58:15.086978+0000","last_peered":"2024-09-10T18:58:15.086978+0000","last_clean":"2024-09-10T18:58:15.086978+0000","last_became_active":"2024-09-10T18:58:15.086721+0000","last_became_peered":"2024-09-10T18:58:15.086721+0000","last_unstale":"2024-09-10T18:58:15.086978+0000","last_undegraded":"2024-09-10T18:58:15.086978+0000","last_fullsized":"2024-09-10T18:58:15.086978+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:25.361003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"191.4","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.087907+0000","last_change":"2024-09-10T18:58:15.087907+0000","last_active":"2024-09-10T18:58:15.087907+0000","last_peered":"2024-09-10T18:58:15.087907+0000","last_clean":"2024-09-10T18:58:15.087907+0000","last_became_active":"2024-09-10T18:58:15.087532+0000","last_became_peered":"2024-09-10T18:58:15.087532+0000","last_unstale":"2024-09-10T18:58:15.087907+0000","last_undegraded":"2024-09-10T18:58:15.087907+0000","last_fullsized":"2024-09-10T18:58:15.087907+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:35.023416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.3","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.092482+0000","last_change":"2024-09-10T18:58:15.092482+0000","last_active":"2024-09-10T18:58:15.092482+0000","last_peered":"2024-09-10T18:58:15.092482+0000","last_clean":"2024-09-10T18:58:15.092482+0000","last_became_active":"2024-09-10T18:58:15.089850+0000","last_became_peered":"2024-09-10T18:58:15.089850+0000","last_unstale":"2024-09-10T18:58:15.092482+0000","last_undegraded":"2024-09-10T18:58:15.092482+0000","last_fullsized":"2024-09-10T18:58:15.092482+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:34.208921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"191.2","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.083534+0000","last_change":"2024-09-10T18:58:15.083534+0000","last_active":"2024-09-10T18:58:15.083534+0000","last_peered":"2024-09-10T18:58:15.083534+0000","last_clean":"2024-09-10T18:58:15.083534+0000","last_became_active":"2024-09-10T18:58:15.083023+0000","last_became_peered":"2024-09-10T18:58:15.083023+0000","last_unstale":"2024-09-10T18:58:15.083534+0000","last_undegraded":"2024-09-10T18:58:15.083534+0000","last_fullsized":"2024-09-10T18:58:15.083534+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:43.807453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.1","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.084771+0000","last_change":"2024-09-10T18:58:15.084771+0000","last_active":"2024-09-10T18:58:15.084771+0000","last_peered":"2024-09-10T18:58:15.084771+0000","last_clean":"2024-09-10T18:58:15.084771+0000","last_became_active":"2024-09-10T18:58:15.084538+0000","last_became_peered":"2024-09-10T18:58:15.084538+0000","last_unstale":"2024-09-10T18:58:15.084771+0000","last_undegraded":"2024-09-10T18:58:15.084771+0000","last_fullsized":"2024-09-10T18:58:15.084771+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:29.759078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"191.0","version":"0'0","reported_seq":9,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.099192+0000","last_change":"2024-09-10T18:58:15.099192+0000","last_active":"2024-09-10T18:58:15.099192+0000","last_peered":"2024-09-10T18:58:15.099192+0000","last_clean":"2024-09-10T18:58:15.099192+0000","last_became_active":"2024-09-10T18:58:15.098794+0000","last_became_peered":"2024-09-10T18:58:15.098794+0000","last_unstale":"2024-09-10T18:58:15.099192+0000","last_undegraded":"2024-09-10T18:58:15.099192+0000","last_fullsized":"2024-09-10T18:58:15.099192+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":338,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:14.059778+0000","last_clean_scrub_stamp":"2024-09-10T18:58:14.059778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:35.911483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":286,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091906+0000","last_change":"2024-09-10T18:57:54.092083+0000","last_active":"2024-09-10T18:58:14.091906+0000","last_peered":"2024-09-10T18:58:14.091906+0000","last_clean":"2024-09-10T18:58:14.091906+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:58:14.091906+0000","last_undegraded":"2024-09-10T18:58:14.091906+0000","last_fullsized":"2024-09-10T18:58:14.091906+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"318'64","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091722+0000","last_change":"2024-09-10T18:57:56.094903+0000","last_active":"2024-09-10T18:58:14.091722+0000","last_peered":"2024-09-10T18:58:14.091722+0000","last_clean":"2024-09-10T18:58:14.091722+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:58:14.091722+0000","last_undegraded":"2024-09-10T18:58:14.091722+0000","last_fullsized":"2024-09-10T18:58:14.091722+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016940303,"stat_sum":{"num_bytes":40,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"276'22","reported_seq":291,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075226+0000","last_change":"2024-09-10T18:57:54.056252+0000","last_active":"2024-09-10T18:58:14.075226+0000","last_peered":"2024-09-10T18:58:14.075226+0000","last_clean":"2024-09-10T18:58:14.075226+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:58:14.075226+0000","last_undegraded":"2024-09-10T18:58:14.075226+0000","last_fullsized":"2024-09-10T18:58:14.075226+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.247847+0000","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087052+0000","last_change":"2024-09-10T18:57:54.097920+0000","last_active":"2024-09-10T18:58:14.087052+0000","last_peered":"2024-09-10T18:58:14.087052+0000","last_clean":"2024-09-10T18:58:14.087052+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:58:14.087052+0000","last_undegraded":"2024-09-10T18:58:14.087052+0000","last_fullsized":"2024-09-10T18:58:14.087052+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":319,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080498+0000","last_change":"2024-09-10T18:57:54.097343+0000","last_active":"2024-09-10T18:58:14.080498+0000","last_peered":"2024-09-10T18:58:14.080498+0000","last_clean":"2024-09-10T18:58:14.080498+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:58:14.080498+0000","last_undegraded":"2024-09-10T18:58:14.080498+0000","last_fullsized":"2024-09-10T18:58:14.080498+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:57:30.615944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.615944+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:46.447187+0000","scrub_duration":0.005614539,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":295,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074319+0000","last_change":"2024-09-10T18:57:54.056894+0000","last_active":"2024-09-10T18:58:14.074319+0000","last_peered":"2024-09-10T18:58:14.074319+0000","last_clean":"2024-09-10T18:58:14.074319+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:58:14.074319+0000","last_undegraded":"2024-09-10T18:58:14.074319+0000","last_fullsized":"2024-09-10T18:58:14.074319+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:57:35.352719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.352719+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.462559+0000","scrub_duration":0.0074091110000000003,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"257'22","reported_seq":290,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072604+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T18:58:14.072604+0000","last_peered":"2024-09-10T18:58:14.072604+0000","last_clean":"2024-09-10T18:58:14.072604+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:58:14.072604+0000","last_undegraded":"2024-09-10T18:58:14.072604+0000","last_fullsized":"2024-09-10T18:58:14.072604+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.319797+0000","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"254'104","reported_seq":351,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067030+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T18:58:14.067030+0000","last_peered":"2024-09-10T18:58:14.067030+0000","last_clean":"2024-09-10T18:58:14.067030+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:58:14.067030+0000","last_undegraded":"2024-09-10T18:58:14.067030+0000","last_fullsized":"2024-09-10T18:58:14.067030+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.932946+0000","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"230'17","reported_seq":279,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080763+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T18:58:14.080763+0000","last_peered":"2024-09-10T18:58:14.080763+0000","last_clean":"2024-09-10T18:58:14.080763+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:58:14.080763+0000","last_undegraded":"2024-09-10T18:58:14.080763+0000","last_fullsized":"2024-09-10T18:58:14.080763+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:17.416599+0000","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"276'22","reported_seq":286,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090277+0000","last_change":"2024-09-10T18:57:54.099018+0000","last_active":"2024-09-10T18:58:14.090277+0000","last_peered":"2024-09-10T18:58:14.090277+0000","last_clean":"2024-09-10T18:58:14.090277+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:58:14.090277+0000","last_undegraded":"2024-09-10T18:58:14.090277+0000","last_fullsized":"2024-09-10T18:58:14.090277+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:42.649292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.649292+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.495881+0000","scrub_duration":0.004377405,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"240'74","reported_seq":328,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.077037+0000","last_change":"2024-09-10T18:57:54.065259+0000","last_active":"2024-09-10T18:58:14.077037+0000","last_peered":"2024-09-10T18:58:14.077037+0000","last_clean":"2024-09-10T18:58:14.077037+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:58:14.077037+0000","last_undegraded":"2024-09-10T18:58:14.077037+0000","last_fullsized":"2024-09-10T18:58:14.077037+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'74","last_scrub_stamp":"2024-09-10T18:57:41.887503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.887503+0000","objects_scrubbed":6,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:47.295134+0000","scrub_duration":0.0039986459999999998,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"246'95","reported_seq":355,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087142+0000","last_change":"2024-09-10T18:57:54.098423+0000","last_active":"2024-09-10T18:58:14.087142+0000","last_peered":"2024-09-10T18:58:14.087142+0000","last_clean":"2024-09-10T18:58:14.087142+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:58:14.087142+0000","last_undegraded":"2024-09-10T18:58:14.087142+0000","last_fullsized":"2024-09-10T18:58:14.087142+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'95","last_scrub_stamp":"2024-09-10T18:57:25.633366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.633366+0000","objects_scrubbed":6,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:31.276059+0000","scrub_duration":0.0044083560000000004,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"146'1","reported_seq":269,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076538+0000","last_change":"2024-09-10T18:57:54.062883+0000","last_active":"2024-09-10T18:58:14.076538+0000","last_peered":"2024-09-10T18:58:14.076538+0000","last_clean":"2024-09-10T18:58:14.076538+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:58:14.076538+0000","last_undegraded":"2024-09-10T18:58:14.076538+0000","last_fullsized":"2024-09-10T18:58:14.076538+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:57:38.843609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.843609+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.556193+0000","scrub_duration":0.003475601,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":293,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067547+0000","last_change":"2024-09-10T18:57:54.072598+0000","last_active":"2024-09-10T18:58:14.067547+0000","last_peered":"2024-09-10T18:58:14.067547+0000","last_clean":"2024-09-10T18:58:14.067547+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:58:14.067547+0000","last_undegraded":"2024-09-10T18:58:14.067547+0000","last_fullsized":"2024-09-10T18:58:14.067547+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"297'52","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073721+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T18:58:14.073721+0000","last_peered":"2024-09-10T18:58:14.073721+0000","last_clean":"2024-09-10T18:58:14.073721+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:58:14.073721+0000","last_undegraded":"2024-09-10T18:58:14.073721+0000","last_fullsized":"2024-09-10T18:58:14.073721+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:04.071589+0000","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"224'34","reported_seq":283,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071297+0000","last_change":"2024-09-10T18:57:54.064626+0000","last_active":"2024-09-10T18:58:14.071297+0000","last_peered":"2024-09-10T18:58:14.071297+0000","last_clean":"2024-09-10T18:58:14.071297+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:58:14.071297+0000","last_undegraded":"2024-09-10T18:58:14.071297+0000","last_fullsized":"2024-09-10T18:58:14.071297+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"326'68","reported_seq":328,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080117+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T18:58:14.080117+0000","last_peered":"2024-09-10T18:58:14.080117+0000","last_clean":"2024-09-10T18:58:14.080117+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:58:14.080117+0000","last_undegraded":"2024-09-10T18:58:14.080117+0000","last_fullsized":"2024-09-10T18:58:14.080117+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.262512+0000","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"246'119","reported_seq":364,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079640+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T18:58:14.079640+0000","last_peered":"2024-09-10T18:58:14.079640+0000","last_clean":"2024-09-10T18:58:14.079640+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:58:14.079640+0000","last_undegraded":"2024-09-10T18:58:14.079640+0000","last_fullsized":"2024-09-10T18:58:14.079640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.422085+0000","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"243'18","reported_seq":286,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074891+0000","last_change":"2024-09-10T18:57:54.066253+0000","last_active":"2024-09-10T18:58:14.074891+0000","last_peered":"2024-09-10T18:58:14.074891+0000","last_clean":"2024-09-10T18:58:14.074891+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:58:14.074891+0000","last_undegraded":"2024-09-10T18:58:14.074891+0000","last_fullsized":"2024-09-10T18:58:14.074891+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"252'22","reported_seq":279,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068269+0000","last_change":"2024-09-10T18:57:54.072778+0000","last_active":"2024-09-10T18:58:14.068269+0000","last_peered":"2024-09-10T18:58:14.068269+0000","last_clean":"2024-09-10T18:58:14.068269+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:58:14.068269+0000","last_undegraded":"2024-09-10T18:58:14.068269+0000","last_fullsized":"2024-09-10T18:58:14.068269+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":296,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080527+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T18:58:14.080527+0000","last_peered":"2024-09-10T18:58:14.080527+0000","last_clean":"2024-09-10T18:58:14.080527+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:58:14.080527+0000","last_undegraded":"2024-09-10T18:58:14.080527+0000","last_fullsized":"2024-09-10T18:58:14.080527+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.729128+0000","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"232'17","reported_seq":284,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076505+0000","last_change":"2024-09-10T18:57:54.061755+0000","last_active":"2024-09-10T18:58:14.076505+0000","last_peered":"2024-09-10T18:58:14.076505+0000","last_clean":"2024-09-10T18:58:14.076505+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:58:14.076505+0000","last_undegraded":"2024-09-10T18:58:14.076505+0000","last_fullsized":"2024-09-10T18:58:14.076505+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:57:32.795149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.795149+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:35.820364+0000","scrub_duration":0.0040623889999999996,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":267,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.083832+0000","last_change":"2024-09-10T18:57:54.063413+0000","last_active":"2024-09-10T18:58:14.083832+0000","last_peered":"2024-09-10T18:58:14.083832+0000","last_clean":"2024-09-10T18:58:14.083832+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:58:14.083832+0000","last_undegraded":"2024-09-10T18:58:14.083832+0000","last_fullsized":"2024-09-10T18:58:14.083832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.800250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.800250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.514815+0000","scrub_duration":0.0031899250000000001,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":310,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.070549+0000","last_change":"2024-09-10T18:57:54.066122+0000","last_active":"2024-09-10T18:58:14.070549+0000","last_peered":"2024-09-10T18:58:14.070549+0000","last_clean":"2024-09-10T18:58:14.070549+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:58:14.070549+0000","last_undegraded":"2024-09-10T18:58:14.070549+0000","last_fullsized":"2024-09-10T18:58:14.070549+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T18:57:35.624392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.624392+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.488899+0000","scrub_duration":0.004362795,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"276'158","reported_seq":408,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.084830+0000","last_change":"2024-09-10T18:57:54.068278+0000","last_active":"2024-09-10T18:58:14.084830+0000","last_peered":"2024-09-10T18:58:14.084830+0000","last_clean":"2024-09-10T18:58:14.084830+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:58:14.084830+0000","last_undegraded":"2024-09-10T18:58:14.084830+0000","last_fullsized":"2024-09-10T18:58:14.084830+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'158","last_scrub_stamp":"2024-09-10T18:57:37.839200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.839200+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.802432+0000","scrub_duration":0.0057174490000000003,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"290'93","reported_seq":397,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076118+0000","last_change":"2024-09-10T18:57:54.058056+0000","last_active":"2024-09-10T18:58:14.076118+0000","last_peered":"2024-09-10T18:58:14.076118+0000","last_clean":"2024-09-10T18:58:14.076118+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:58:14.076118+0000","last_undegraded":"2024-09-10T18:58:14.076118+0000","last_fullsized":"2024-09-10T18:58:14.076118+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'93","last_scrub_stamp":"2024-09-10T18:57:29.481888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.481888+0000","objects_scrubbed":6,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.178966+0000","scrub_duration":0.004612952,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":303,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080124+0000","last_change":"2024-09-10T18:57:54.078767+0000","last_active":"2024-09-10T18:58:14.080124+0000","last_peered":"2024-09-10T18:58:14.080124+0000","last_clean":"2024-09-10T18:58:14.080124+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:58:14.080124+0000","last_undegraded":"2024-09-10T18:58:14.080124+0000","last_fullsized":"2024-09-10T18:58:14.080124+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"257'107","reported_seq":358,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075089+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T18:58:14.075089+0000","last_peered":"2024-09-10T18:58:14.075089+0000","last_clean":"2024-09-10T18:58:14.075089+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:58:14.075089+0000","last_undegraded":"2024-09-10T18:58:14.075089+0000","last_fullsized":"2024-09-10T18:58:14.075089+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.441046+0000","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"236'102","reported_seq":344,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073577+0000","last_change":"2024-09-10T18:57:54.065977+0000","last_active":"2024-09-10T18:58:14.073577+0000","last_peered":"2024-09-10T18:58:14.073577+0000","last_clean":"2024-09-10T18:58:14.073577+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:58:14.073577+0000","last_undegraded":"2024-09-10T18:58:14.073577+0000","last_fullsized":"2024-09-10T18:58:14.073577+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":292,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073905+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T18:58:14.073905+0000","last_peered":"2024-09-10T18:58:14.073905+0000","last_clean":"2024-09-10T18:58:14.073905+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:58:14.073905+0000","last_undegraded":"2024-09-10T18:58:14.073905+0000","last_fullsized":"2024-09-10T18:58:14.073905+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.440202+0000","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"263'72","reported_seq":326,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067348+0000","last_change":"2024-09-10T18:57:54.071923+0000","last_active":"2024-09-10T18:58:14.067348+0000","last_peered":"2024-09-10T18:58:14.067348+0000","last_clean":"2024-09-10T18:58:14.067348+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:58:14.067348+0000","last_undegraded":"2024-09-10T18:58:14.067348+0000","last_fullsized":"2024-09-10T18:58:14.067348+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":349,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075928+0000","last_change":"2024-09-10T18:57:54.060094+0000","last_active":"2024-09-10T18:58:14.075928+0000","last_peered":"2024-09-10T18:58:14.075928+0000","last_clean":"2024-09-10T18:58:14.075928+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:58:14.075928+0000","last_undegraded":"2024-09-10T18:58:14.075928+0000","last_fullsized":"2024-09-10T18:58:14.075928+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:56:44.683516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.683516+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708920000000001,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":302,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072300+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:58:14.072300+0000","last_peered":"2024-09-10T18:58:14.072300+0000","last_clean":"2024-09-10T18:58:14.072300+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:58:14.072300+0000","last_undegraded":"2024-09-10T18:58:14.072300+0000","last_fullsized":"2024-09-10T18:58:14.072300+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":316,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092468+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:58:14.092468+0000","last_peered":"2024-09-10T18:58:14.092468+0000","last_clean":"2024-09-10T18:58:14.092468+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:58:14.092468+0000","last_undegraded":"2024-09-10T18:58:14.092468+0000","last_fullsized":"2024-09-10T18:58:14.092468+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":321,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080238+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:58:14.080238+0000","last_peered":"2024-09-10T18:58:14.080238+0000","last_clean":"2024-09-10T18:58:14.080238+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:58:14.080238+0000","last_undegraded":"2024-09-10T18:58:14.080238+0000","last_fullsized":"2024-09-10T18:58:14.080238+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074878+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:58:14.074878+0000","last_peered":"2024-09-10T18:58:14.074878+0000","last_clean":"2024-09-10T18:58:14.074878+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:58:14.074878+0000","last_undegraded":"2024-09-10T18:58:14.074878+0000","last_fullsized":"2024-09-10T18:58:14.074878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.066839+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:58:14.066839+0000","last_peered":"2024-09-10T18:58:14.066839+0000","last_clean":"2024-09-10T18:58:14.066839+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:58:14.066839+0000","last_undegraded":"2024-09-10T18:58:14.066839+0000","last_fullsized":"2024-09-10T18:58:14.066839+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.d","version":"319'10","reported_seq":102,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091319+0000","last_change":"2024-09-10T18:57:27.072912+0000","last_active":"2024-09-10T18:58:14.091319+0000","last_peered":"2024-09-10T18:58:14.091319+0000","last_clean":"2024-09-10T18:58:14.091319+0000","last_became_active":"2024-09-10T18:57:27.072565+0000","last_became_peered":"2024-09-10T18:57:27.072565+0000","last_unstale":"2024-09-10T18:58:14.091319+0000","last_undegraded":"2024-09-10T18:58:14.091319+0000","last_fullsized":"2024-09-10T18:58:14.091319+0000","mapping_epoch":274,"log_start":"243'6","ondisk_log_start":"243'6","created":124,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'6","last_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_deep_scrub":"243'6","last_deep_scrub_stamp":"2024-09-10T18:56:39.791846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.791846+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081205+0000","last_change":"2024-09-10T18:55:04.765245+0000","last_active":"2024-09-10T18:58:14.081205+0000","last_peered":"2024-09-10T18:58:14.081205+0000","last_clean":"2024-09-10T18:58:14.081205+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:58:14.081205+0000","last_undegraded":"2024-09-10T18:58:14.081205+0000","last_fullsized":"2024-09-10T18:58:14.081205+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.765156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.765156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022801589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":333,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092174+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:58:14.092174+0000","last_peered":"2024-09-10T18:58:14.092174+0000","last_clean":"2024-09-10T18:58:14.092174+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:58:14.092174+0000","last_undegraded":"2024-09-10T18:58:14.092174+0000","last_fullsized":"2024-09-10T18:58:14.092174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073418+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:58:14.073418+0000","last_peered":"2024-09-10T18:58:14.073418+0000","last_clean":"2024-09-10T18:58:14.073418+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:58:14.073418+0000","last_undegraded":"2024-09-10T18:58:14.073418+0000","last_fullsized":"2024-09-10T18:58:14.073418+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074940+0000","last_change":"2024-09-10T18:55:16.118902+0000","last_active":"2024-09-10T18:58:14.074940+0000","last_peered":"2024-09-10T18:58:14.074940+0000","last_clean":"2024-09-10T18:58:14.074940+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:58:14.074940+0000","last_undegraded":"2024-09-10T18:58:14.074940+0000","last_fullsized":"2024-09-10T18:58:14.074940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:16.118660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:16.118660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035251430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088551+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T18:58:14.088551+0000","last_peered":"2024-09-10T18:58:14.088551+0000","last_clean":"2024-09-10T18:58:14.088551+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:58:14.088551+0000","last_undegraded":"2024-09-10T18:58:14.088551+0000","last_fullsized":"2024-09-10T18:58:14.088551+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:37.279016+0000","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":94,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074242+0000","last_change":"2024-09-10T18:57:31.113969+0000","last_active":"2024-09-10T18:58:14.074242+0000","last_peered":"2024-09-10T18:58:14.074242+0000","last_clean":"2024-09-10T18:58:14.074242+0000","last_became_active":"2024-09-10T18:57:31.113373+0000","last_became_peered":"2024-09-10T18:57:31.113373+0000","last_unstale":"2024-09-10T18:58:14.074242+0000","last_undegraded":"2024-09-10T18:58:14.074242+0000","last_fullsized":"2024-09-10T18:58:14.074242+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.362084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.362084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075336+0000","last_change":"2024-09-10T18:56:07.514585+0000","last_active":"2024-09-10T18:58:14.075336+0000","last_peered":"2024-09-10T18:58:14.075336+0000","last_clean":"2024-09-10T18:58:14.075336+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:58:14.075336+0000","last_undegraded":"2024-09-10T18:58:14.075336+0000","last_fullsized":"2024-09-10T18:58:14.075336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:07.514489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.514489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019759069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":344,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068641+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T18:58:14.068641+0000","last_peered":"2024-09-10T18:58:14.068641+0000","last_clean":"2024-09-10T18:58:14.068641+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:58:14.068641+0000","last_undegraded":"2024-09-10T18:58:14.068641+0000","last_fullsized":"2024-09-10T18:58:14.068641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:42.997043+0000","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074911+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:58:14.074911+0000","last_peered":"2024-09-10T18:58:14.074911+0000","last_clean":"2024-09-10T18:58:14.074911+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:58:14.074911+0000","last_undegraded":"2024-09-10T18:58:14.074911+0000","last_fullsized":"2024-09-10T18:58:14.074911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067023+0000","last_change":"2024-09-10T18:57:31.277231+0000","last_active":"2024-09-10T18:58:14.067023+0000","last_peered":"2024-09-10T18:58:14.067023+0000","last_clean":"2024-09-10T18:58:14.067023+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:58:14.067023+0000","last_undegraded":"2024-09-10T18:58:14.067023+0000","last_fullsized":"2024-09-10T18:58:14.067023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.277085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.277085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:39.184999+0000","scrub_duration":0.003344746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080793+0000","last_change":"2024-09-10T18:56:20.681395+0000","last_active":"2024-09-10T18:58:14.080793+0000","last_peered":"2024-09-10T18:58:14.080793+0000","last_clean":"2024-09-10T18:58:14.080793+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:58:14.080793+0000","last_undegraded":"2024-09-10T18:58:14.080793+0000","last_fullsized":"2024-09-10T18:58:14.080793+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:20.681251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:20.681251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003109845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"319'8","reported_seq":106,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081387+0000","last_change":"2024-09-10T18:57:29.105121+0000","last_active":"2024-09-10T18:58:14.081387+0000","last_peered":"2024-09-10T18:58:14.081387+0000","last_clean":"2024-09-10T18:58:14.081387+0000","last_became_active":"2024-09-10T18:57:19.076695+0000","last_became_peered":"2024-09-10T18:57:19.076695+0000","last_unstale":"2024-09-10T18:58:14.081387+0000","last_undegraded":"2024-09-10T18:58:14.081387+0000","last_fullsized":"2024-09-10T18:58:14.081387+0000","mapping_epoch":274,"log_start":"237'6","ondisk_log_start":"237'6","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'6","last_scrub_stamp":"2024-09-10T18:57:29.105052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.105052+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":6,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.677376+0000","scrub_duration":0.0020046370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":317,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075563+0000","last_change":"2024-09-10T18:57:25.517299+0000","last_active":"2024-09-10T18:58:14.075563+0000","last_peered":"2024-09-10T18:58:14.075563+0000","last_clean":"2024-09-10T18:58:14.075563+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:58:14.075563+0000","last_undegraded":"2024-09-10T18:58:14.075563+0000","last_fullsized":"2024-09-10T18:58:14.075563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:25.517210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.517210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:26.377413+0000","scrub_duration":0.0027489659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":350,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079883+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:58:14.079883+0000","last_peered":"2024-09-10T18:58:14.079883+0000","last_clean":"2024-09-10T18:58:14.079883+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:58:14.079883+0000","last_undegraded":"2024-09-10T18:58:14.079883+0000","last_fullsized":"2024-09-10T18:58:14.079883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075246+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T18:58:14.075246+0000","last_peered":"2024-09-10T18:58:14.075246+0000","last_clean":"2024-09-10T18:58:14.075246+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:58:14.075246+0000","last_undegraded":"2024-09-10T18:58:14.075246+0000","last_fullsized":"2024-09-10T18:58:14.075246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.804733+0000","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":310,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088004+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:58:14.088004+0000","last_peered":"2024-09-10T18:58:14.088004+0000","last_clean":"2024-09-10T18:58:14.088004+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:58:14.088004+0000","last_undegraded":"2024-09-10T18:58:14.088004+0000","last_fullsized":"2024-09-10T18:58:14.088004+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071658+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:58:14.071658+0000","last_peered":"2024-09-10T18:58:14.071658+0000","last_clean":"2024-09-10T18:58:14.071658+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:58:14.071658+0000","last_undegraded":"2024-09-10T18:58:14.071658+0000","last_fullsized":"2024-09-10T18:58:14.071658+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":93,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079485+0000","last_change":"2024-09-10T18:57:23.457770+0000","last_active":"2024-09-10T18:58:14.079485+0000","last_peered":"2024-09-10T18:58:14.079485+0000","last_clean":"2024-09-10T18:58:14.079485+0000","last_became_active":"2024-09-10T18:57:23.457438+0000","last_became_peered":"2024-09-10T18:57:23.457438+0000","last_unstale":"2024-09-10T18:58:14.079485+0000","last_undegraded":"2024-09-10T18:58:14.079485+0000","last_fullsized":"2024-09-10T18:58:14.079485+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.672023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.672023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":354,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074287+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T18:58:14.074287+0000","last_peered":"2024-09-10T18:58:14.074287+0000","last_clean":"2024-09-10T18:58:14.074287+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:58:14.074287+0000","last_undegraded":"2024-09-10T18:58:14.074287+0000","last_fullsized":"2024-09-10T18:58:14.074287+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.716139+0000","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074682+0000","last_change":"2024-09-10T18:57:34.379150+0000","last_active":"2024-09-10T18:58:14.074682+0000","last_peered":"2024-09-10T18:58:14.074682+0000","last_clean":"2024-09-10T18:58:14.074682+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:58:14.074682+0000","last_undegraded":"2024-09-10T18:58:14.074682+0000","last_fullsized":"2024-09-10T18:58:14.074682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.378985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.378985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.427844+0000","scrub_duration":0.00364804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":432,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075321+0000","last_change":"2024-09-10T18:55:09.170194+0000","last_active":"2024-09-10T18:58:14.075321+0000","last_peered":"2024-09-10T18:58:14.075321+0000","last_clean":"2024-09-10T18:58:14.075321+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:58:14.075321+0000","last_undegraded":"2024-09-10T18:58:14.075321+0000","last_fullsized":"2024-09-10T18:58:14.075321+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:55:09.170040+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170040+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034920670000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":320,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.085213+0000","last_change":"2024-09-10T18:57:36.840129+0000","last_active":"2024-09-10T18:58:14.085213+0000","last_peered":"2024-09-10T18:58:14.085213+0000","last_clean":"2024-09-10T18:58:14.085213+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:58:14.085213+0000","last_undegraded":"2024-09-10T18:58:14.085213+0000","last_fullsized":"2024-09-10T18:58:14.085213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.839979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.839979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:44.823438+0000","scrub_duration":0.0034210669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078825+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:58:14.078825+0000","last_peered":"2024-09-10T18:58:14.078825+0000","last_clean":"2024-09-10T18:58:14.078825+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:58:14.078825+0000","last_undegraded":"2024-09-10T18:58:14.078825+0000","last_fullsized":"2024-09-10T18:58:14.078825+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":336,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073695+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:58:14.073695+0000","last_peered":"2024-09-10T18:58:14.073695+0000","last_clean":"2024-09-10T18:58:14.073695+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:58:14.073695+0000","last_undegraded":"2024-09-10T18:58:14.073695+0000","last_fullsized":"2024-09-10T18:58:14.073695+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081039+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:58:14.081039+0000","last_peered":"2024-09-10T18:58:14.081039+0000","last_clean":"2024-09-10T18:58:14.081039+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:58:14.081039+0000","last_undegraded":"2024-09-10T18:58:14.081039+0000","last_fullsized":"2024-09-10T18:58:14.081039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074788+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:58:14.074788+0000","last_peered":"2024-09-10T18:58:14.074788+0000","last_clean":"2024-09-10T18:58:14.074788+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:58:14.074788+0000","last_undegraded":"2024-09-10T18:58:14.074788+0000","last_fullsized":"2024-09-10T18:58:14.074788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075963+0000","last_change":"2024-09-10T18:55:00.303776+0000","last_active":"2024-09-10T18:58:14.075963+0000","last_peered":"2024-09-10T18:58:14.075963+0000","last_clean":"2024-09-10T18:58:14.075963+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:58:14.075963+0000","last_undegraded":"2024-09-10T18:58:14.075963+0000","last_fullsized":"2024-09-10T18:58:14.075963+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.303598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.303598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039581699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.077995+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T18:58:14.077995+0000","last_peered":"2024-09-10T18:58:14.077995+0000","last_clean":"2024-09-10T18:58:14.077995+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:58:14.077995+0000","last_undegraded":"2024-09-10T18:58:14.077995+0000","last_fullsized":"2024-09-10T18:58:14.077995+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.592874+0000","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":341,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071861+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:58:14.071861+0000","last_peered":"2024-09-10T18:58:14.071861+0000","last_clean":"2024-09-10T18:58:14.071861+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:58:14.071861+0000","last_undegraded":"2024-09-10T18:58:14.071861+0000","last_fullsized":"2024-09-10T18:58:14.071861+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079268+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:58:14.079268+0000","last_peered":"2024-09-10T18:58:14.079268+0000","last_clean":"2024-09-10T18:58:14.079268+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:58:14.079268+0000","last_undegraded":"2024-09-10T18:58:14.079268+0000","last_fullsized":"2024-09-10T18:58:14.079268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":321,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090246+0000","last_change":"2024-09-10T18:56:39.629880+0000","last_active":"2024-09-10T18:58:14.090246+0000","last_peered":"2024-09-10T18:58:14.090246+0000","last_clean":"2024-09-10T18:58:14.090246+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:58:14.090246+0000","last_undegraded":"2024-09-10T18:58:14.090246+0000","last_fullsized":"2024-09-10T18:58:14.090246+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:39.629726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.629726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034450290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079582+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:58:14.079582+0000","last_peered":"2024-09-10T18:58:14.079582+0000","last_clean":"2024-09-10T18:58:14.079582+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:58:14.079582+0000","last_undegraded":"2024-09-10T18:58:14.079582+0000","last_fullsized":"2024-09-10T18:58:14.079582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":241,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090351+0000","last_change":"2024-09-10T18:57:56.827061+0000","last_active":"2024-09-10T18:58:14.090351+0000","last_peered":"2024-09-10T18:58:14.090351+0000","last_clean":"2024-09-10T18:58:14.090351+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:58:14.090351+0000","last_undegraded":"2024-09-10T18:58:14.090351+0000","last_fullsized":"2024-09-10T18:58:14.090351+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.826954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.826954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.327774+0000","scrub_duration":0.0032518439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":354,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074320+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:58:14.074320+0000","last_peered":"2024-09-10T18:58:14.074320+0000","last_clean":"2024-09-10T18:58:14.074320+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:58:14.074320+0000","last_undegraded":"2024-09-10T18:58:14.074320+0000","last_fullsized":"2024-09-10T18:58:14.074320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072502+0000","last_change":"2024-09-10T18:56:14.554734+0000","last_active":"2024-09-10T18:58:14.072502+0000","last_peered":"2024-09-10T18:58:14.072502+0000","last_clean":"2024-09-10T18:58:14.072502+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:58:14.072502+0000","last_undegraded":"2024-09-10T18:58:14.072502+0000","last_fullsized":"2024-09-10T18:58:14.072502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:14.554664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.554664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034514569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":442,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075984+0000","last_change":"2024-09-10T18:57:32.764761+0000","last_active":"2024-09-10T18:58:14.075984+0000","last_peered":"2024-09-10T18:58:14.075984+0000","last_clean":"2024-09-10T18:58:14.075984+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:58:14.075984+0000","last_undegraded":"2024-09-10T18:58:14.075984+0000","last_fullsized":"2024-09-10T18:58:14.075984+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:57:32.764561+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.764561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:41.529379+0000","scrub_duration":0.0041273020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":313,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078403+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:58:14.078403+0000","last_peered":"2024-09-10T18:58:14.078403+0000","last_clean":"2024-09-10T18:58:14.078403+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:58:14.078403+0000","last_undegraded":"2024-09-10T18:58:14.078403+0000","last_fullsized":"2024-09-10T18:58:14.078403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090429+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:58:14.090429+0000","last_peered":"2024-09-10T18:58:14.090429+0000","last_clean":"2024-09-10T18:58:14.090429+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:58:14.090429+0000","last_undegraded":"2024-09-10T18:58:14.090429+0000","last_fullsized":"2024-09-10T18:58:14.090429+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":332,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092057+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T18:58:14.092057+0000","last_peered":"2024-09-10T18:58:14.092057+0000","last_clean":"2024-09-10T18:58:14.092057+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:58:14.092057+0000","last_undegraded":"2024-09-10T18:58:14.092057+0000","last_fullsized":"2024-09-10T18:58:14.092057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.981000+0000","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081690+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:58:14.081690+0000","last_peered":"2024-09-10T18:58:14.081690+0000","last_clean":"2024-09-10T18:58:14.081690+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:58:14.081690+0000","last_undegraded":"2024-09-10T18:58:14.081690+0000","last_fullsized":"2024-09-10T18:58:14.081690+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074039+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T18:58:14.074039+0000","last_peered":"2024-09-10T18:58:14.074039+0000","last_clean":"2024-09-10T18:58:14.074039+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:58:14.074039+0000","last_undegraded":"2024-09-10T18:58:14.074039+0000","last_fullsized":"2024-09-10T18:58:14.074039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.482905+0000","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":834,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.812511+0000","last_change":"2024-09-10T18:56:14.796998+0000","last_active":"2024-09-10T18:58:14.812511+0000","last_peered":"2024-09-10T18:58:14.812511+0000","last_clean":"2024-09-10T18:58:14.812511+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:58:14.812511+0000","last_undegraded":"2024-09-10T18:58:14.812511+0000","last_fullsized":"2024-09-10T18:58:14.812511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:56:14.796858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:14.796858+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035079629999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":512,"num_read_kb":507,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073609+0000","last_change":"2024-09-10T18:57:49.386576+0000","last_active":"2024-09-10T18:58:14.073609+0000","last_peered":"2024-09-10T18:58:14.073609+0000","last_clean":"2024-09-10T18:58:14.073609+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:58:14.073609+0000","last_undegraded":"2024-09-10T18:58:14.073609+0000","last_fullsized":"2024-09-10T18:58:14.073609+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.386404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.605455+0000","scrub_duration":0.003967628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":242,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082983+0000","last_change":"2024-09-10T18:57:49.919967+0000","last_active":"2024-09-10T18:58:14.082983+0000","last_peered":"2024-09-10T18:58:14.082983+0000","last_clean":"2024-09-10T18:58:14.082983+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:58:14.082983+0000","last_undegraded":"2024-09-10T18:58:14.082983+0000","last_fullsized":"2024-09-10T18:58:14.082983+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.919794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.919794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.849294+0000","scrub_duration":0.0034551590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":343,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082161+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:58:14.082161+0000","last_peered":"2024-09-10T18:58:14.082161+0000","last_clean":"2024-09-10T18:58:14.082161+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:58:14.082161+0000","last_undegraded":"2024-09-10T18:58:14.082161+0000","last_fullsized":"2024-09-10T18:58:14.082161+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090906+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:58:14.090906+0000","last_peered":"2024-09-10T18:58:14.090906+0000","last_clean":"2024-09-10T18:58:14.090906+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:58:14.090906+0000","last_undegraded":"2024-09-10T18:58:14.090906+0000","last_fullsized":"2024-09-10T18:58:14.090906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":421,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071544+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:58:14.071544+0000","last_peered":"2024-09-10T18:58:14.071544+0000","last_clean":"2024-09-10T18:58:14.071544+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:58:14.071544+0000","last_undegraded":"2024-09-10T18:58:14.071544+0000","last_fullsized":"2024-09-10T18:58:14.071544+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":331,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073881+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:58:14.073881+0000","last_peered":"2024-09-10T18:58:14.073881+0000","last_clean":"2024-09-10T18:58:14.073881+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:58:14.073881+0000","last_undegraded":"2024-09-10T18:58:14.073881+0000","last_fullsized":"2024-09-10T18:58:14.073881+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075147+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T18:58:14.075147+0000","last_peered":"2024-09-10T18:58:14.075147+0000","last_clean":"2024-09-10T18:58:14.075147+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:58:14.075147+0000","last_undegraded":"2024-09-10T18:58:14.075147+0000","last_fullsized":"2024-09-10T18:58:14.075147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:19.820859+0000","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071591+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T18:58:14.071591+0000","last_peered":"2024-09-10T18:58:14.071591+0000","last_clean":"2024-09-10T18:58:14.071591+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:58:14.071591+0000","last_undegraded":"2024-09-10T18:58:14.071591+0000","last_fullsized":"2024-09-10T18:58:14.071591+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.921143+0000","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":490,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087712+0000","last_change":"2024-09-10T18:57:50.752463+0000","last_active":"2024-09-10T18:58:14.087712+0000","last_peered":"2024-09-10T18:58:14.087712+0000","last_clean":"2024-09-10T18:58:14.087712+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:58:14.087712+0000","last_undegraded":"2024-09-10T18:58:14.087712+0000","last_fullsized":"2024-09-10T18:58:14.087712+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:57:50.752292+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.752292+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.294597+0000","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075898+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:58:14.075898+0000","last_peered":"2024-09-10T18:58:14.075898+0000","last_clean":"2024-09-10T18:58:14.075898+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:58:14.075898+0000","last_undegraded":"2024-09-10T18:58:14.075898+0000","last_fullsized":"2024-09-10T18:58:14.075898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":430,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090281+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:58:14.090281+0000","last_peered":"2024-09-10T18:58:14.090281+0000","last_clean":"2024-09-10T18:58:14.090281+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:58:14.090281+0000","last_undegraded":"2024-09-10T18:58:14.090281+0000","last_fullsized":"2024-09-10T18:58:14.090281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":319,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073994+0000","last_change":"2024-09-10T18:55:05.604470+0000","last_active":"2024-09-10T18:58:14.073994+0000","last_peered":"2024-09-10T18:58:14.073994+0000","last_clean":"2024-09-10T18:58:14.073994+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:58:14.073994+0000","last_undegraded":"2024-09-10T18:58:14.073994+0000","last_fullsized":"2024-09-10T18:58:14.073994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:55:05.604339+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:55:05.604339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070590669999999996,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":309,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078882+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:58:14.078882+0000","last_peered":"2024-09-10T18:58:14.078882+0000","last_clean":"2024-09-10T18:58:14.078882+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:58:14.078882+0000","last_undegraded":"2024-09-10T18:58:14.078882+0000","last_fullsized":"2024-09-10T18:58:14.078882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074046+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T18:58:14.074046+0000","last_peered":"2024-09-10T18:58:14.074046+0000","last_clean":"2024-09-10T18:58:14.074046+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:58:14.074046+0000","last_undegraded":"2024-09-10T18:58:14.074046+0000","last_fullsized":"2024-09-10T18:58:14.074046+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.040014+0000","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":334,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079982+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:58:14.079982+0000","last_peered":"2024-09-10T18:58:14.079982+0000","last_clean":"2024-09-10T18:58:14.079982+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:58:14.079982+0000","last_undegraded":"2024-09-10T18:58:14.079982+0000","last_fullsized":"2024-09-10T18:58:14.079982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087805+0000","last_change":"2024-09-10T18:56:05.539583+0000","last_active":"2024-09-10T18:58:14.087805+0000","last_peered":"2024-09-10T18:58:14.087805+0000","last_clean":"2024-09-10T18:58:14.087805+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:58:14.087805+0000","last_undegraded":"2024-09-10T18:58:14.087805+0000","last_fullsized":"2024-09-10T18:58:14.087805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.539465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.539465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029377980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080711+0000","last_change":"2024-09-10T18:56:08.572029+0000","last_active":"2024-09-10T18:58:14.080711+0000","last_peered":"2024-09-10T18:58:14.080711+0000","last_clean":"2024-09-10T18:58:14.080711+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:58:14.080711+0000","last_undegraded":"2024-09-10T18:58:14.080711+0000","last_fullsized":"2024-09-10T18:58:14.080711+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:08.571885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:56:08.571885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032956679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":310,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088271+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:58:14.088271+0000","last_peered":"2024-09-10T18:58:14.088271+0000","last_clean":"2024-09-10T18:58:14.088271+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:58:14.088271+0000","last_undegraded":"2024-09-10T18:58:14.088271+0000","last_fullsized":"2024-09-10T18:58:14.088271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":242,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076629+0000","last_change":"2024-09-10T18:57:43.914840+0000","last_active":"2024-09-10T18:58:14.076629+0000","last_peered":"2024-09-10T18:58:14.076629+0000","last_clean":"2024-09-10T18:58:14.076629+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:58:14.076629+0000","last_undegraded":"2024-09-10T18:58:14.076629+0000","last_fullsized":"2024-09-10T18:58:14.076629+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:43.914731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:43.914731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.644461+0000","scrub_duration":0.0026766530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":362,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075260+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T18:58:14.075260+0000","last_peered":"2024-09-10T18:58:14.075260+0000","last_clean":"2024-09-10T18:58:14.075260+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:58:14.075260+0000","last_undegraded":"2024-09-10T18:58:14.075260+0000","last_fullsized":"2024-09-10T18:58:14.075260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:08.776283+0000","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078888+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:58:14.078888+0000","last_peered":"2024-09-10T18:58:14.078888+0000","last_clean":"2024-09-10T18:58:14.078888+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:58:14.078888+0000","last_undegraded":"2024-09-10T18:58:14.078888+0000","last_fullsized":"2024-09-10T18:58:14.078888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":321,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087630+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T18:58:14.087630+0000","last_peered":"2024-09-10T18:58:14.087630+0000","last_clean":"2024-09-10T18:58:14.087630+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:58:14.087630+0000","last_undegraded":"2024-09-10T18:58:14.087630+0000","last_fullsized":"2024-09-10T18:58:14.087630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.242393+0000","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":302,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080075+0000","last_change":"2024-09-10T18:56:05.650498+0000","last_active":"2024-09-10T18:58:14.080075+0000","last_peered":"2024-09-10T18:58:14.080075+0000","last_clean":"2024-09-10T18:58:14.080075+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:58:14.080075+0000","last_undegraded":"2024-09-10T18:58:14.080075+0000","last_fullsized":"2024-09-10T18:58:14.080075+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.650373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.650373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022361669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091106+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:58:14.091106+0000","last_peered":"2024-09-10T18:58:14.091106+0000","last_clean":"2024-09-10T18:58:14.091106+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:58:14.091106+0000","last_undegraded":"2024-09-10T18:58:14.091106+0000","last_fullsized":"2024-09-10T18:58:14.091106+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":76,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087838+0000","last_change":"2024-09-10T18:57:11.091444+0000","last_active":"2024-09-10T18:58:14.087838+0000","last_peered":"2024-09-10T18:58:14.087838+0000","last_clean":"2024-09-10T18:58:14.087838+0000","last_became_active":"2024-09-10T18:57:11.091111+0000","last_became_peered":"2024-09-10T18:57:11.091111+0000","last_unstale":"2024-09-10T18:58:14.087838+0000","last_undegraded":"2024-09-10T18:58:14.087838+0000","last_fullsized":"2024-09-10T18:58:14.087838+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":275,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.795811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.795811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":363,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079040+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T18:58:14.079040+0000","last_peered":"2024-09-10T18:58:14.079040+0000","last_clean":"2024-09-10T18:58:14.079040+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:58:14.079040+0000","last_undegraded":"2024-09-10T18:58:14.079040+0000","last_fullsized":"2024-09-10T18:58:14.079040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:05.333581+0000","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":336,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082848+0000","last_change":"2024-09-10T18:57:55.986832+0000","last_active":"2024-09-10T18:58:14.082848+0000","last_peered":"2024-09-10T18:58:14.082848+0000","last_clean":"2024-09-10T18:58:14.082848+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:58:14.082848+0000","last_undegraded":"2024-09-10T18:58:14.082848+0000","last_fullsized":"2024-09-10T18:58:14.082848+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:55.986726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.986726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:23.114093+0000","scrub_duration":0.0034130330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090453+0000","last_change":"2024-09-10T18:56:06.527877+0000","last_active":"2024-09-10T18:58:14.090453+0000","last_peered":"2024-09-10T18:58:14.090453+0000","last_clean":"2024-09-10T18:58:14.090453+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:58:14.090453+0000","last_undegraded":"2024-09-10T18:58:14.090453+0000","last_fullsized":"2024-09-10T18:58:14.090453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:06.527707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:06.527707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00323098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075559+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T18:58:14.075559+0000","last_peered":"2024-09-10T18:58:14.075559+0000","last_clean":"2024-09-10T18:58:14.075559+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:58:14.075559+0000","last_undegraded":"2024-09-10T18:58:14.075559+0000","last_fullsized":"2024-09-10T18:58:14.075559+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.701730+0000","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080917+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:58:14.080917+0000","last_peered":"2024-09-10T18:58:14.080917+0000","last_clean":"2024-09-10T18:58:14.080917+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:58:14.080917+0000","last_undegraded":"2024-09-10T18:58:14.080917+0000","last_fullsized":"2024-09-10T18:58:14.080917+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":335,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088300+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:58:14.088300+0000","last_peered":"2024-09-10T18:58:14.088300+0000","last_clean":"2024-09-10T18:58:14.088300+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:58:14.088300+0000","last_undegraded":"2024-09-10T18:58:14.088300+0000","last_fullsized":"2024-09-10T18:58:14.088300+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072134+0000","last_change":"2024-09-10T18:56:21.508009+0000","last_active":"2024-09-10T18:58:14.072134+0000","last_peered":"2024-09-10T18:58:14.072134+0000","last_clean":"2024-09-10T18:58:14.072134+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:58:14.072134+0000","last_undegraded":"2024-09-10T18:58:14.072134+0000","last_fullsized":"2024-09-10T18:58:14.072134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.507845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.507845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030676980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092102+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:58:14.092102+0000","last_peered":"2024-09-10T18:58:14.092102+0000","last_clean":"2024-09-10T18:58:14.092102+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:58:14.092102+0000","last_undegraded":"2024-09-10T18:58:14.092102+0000","last_fullsized":"2024-09-10T18:58:14.092102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":319,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087362+0000","last_change":"2024-09-10T18:57:44.612113+0000","last_active":"2024-09-10T18:58:14.087362+0000","last_peered":"2024-09-10T18:58:14.087362+0000","last_clean":"2024-09-10T18:58:14.087362+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:58:14.087362+0000","last_undegraded":"2024-09-10T18:58:14.087362+0000","last_fullsized":"2024-09-10T18:58:14.087362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:44.612043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:57:44.612043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.176921+0000","scrub_duration":0.0026996279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092135+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:58:14.092135+0000","last_peered":"2024-09-10T18:58:14.092135+0000","last_clean":"2024-09-10T18:58:14.092135+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:58:14.092135+0000","last_undegraded":"2024-09-10T18:58:14.092135+0000","last_fullsized":"2024-09-10T18:58:14.092135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":326,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071792+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:58:14.071792+0000","last_peered":"2024-09-10T18:58:14.071792+0000","last_clean":"2024-09-10T18:58:14.071792+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:58:14.071792+0000","last_undegraded":"2024-09-10T18:58:14.071792+0000","last_fullsized":"2024-09-10T18:58:14.071792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080483+0000","last_change":"2024-09-10T18:57:56.125998+0000","last_active":"2024-09-10T18:58:14.080483+0000","last_peered":"2024-09-10T18:58:14.080483+0000","last_clean":"2024-09-10T18:58:14.080483+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:58:14.080483+0000","last_undegraded":"2024-09-10T18:58:14.080483+0000","last_fullsized":"2024-09-10T18:58:14.080483+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.125881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.125881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.508448+0000","scrub_duration":0.0026334969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":372,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072268+0000","last_change":"2024-09-10T18:57:56.375088+0000","last_active":"2024-09-10T18:58:14.072268+0000","last_peered":"2024-09-10T18:58:14.072268+0000","last_clean":"2024-09-10T18:58:14.072268+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:58:14.072268+0000","last_undegraded":"2024-09-10T18:58:14.072268+0000","last_fullsized":"2024-09-10T18:58:14.072268+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:57:56.374923+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.374923+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:58.365310+0000","scrub_duration":0.0043604359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079237+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:58:14.079237+0000","last_peered":"2024-09-10T18:58:14.079237+0000","last_clean":"2024-09-10T18:58:14.079237+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:58:14.079237+0000","last_undegraded":"2024-09-10T18:58:14.079237+0000","last_fullsized":"2024-09-10T18:58:14.079237+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":308,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068599+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:58:14.068599+0000","last_peered":"2024-09-10T18:58:14.068599+0000","last_clean":"2024-09-10T18:58:14.068599+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:58:14.068599+0000","last_undegraded":"2024-09-10T18:58:14.068599+0000","last_fullsized":"2024-09-10T18:58:14.068599+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075816+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:58:14.075816+0000","last_peered":"2024-09-10T18:58:14.075816+0000","last_clean":"2024-09-10T18:58:14.075816+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:58:14.075816+0000","last_undegraded":"2024-09-10T18:58:14.075816+0000","last_fullsized":"2024-09-10T18:58:14.075816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":340,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075859+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:58:14.075859+0000","last_peered":"2024-09-10T18:58:14.075859+0000","last_clean":"2024-09-10T18:58:14.075859+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:58:14.075859+0000","last_undegraded":"2024-09-10T18:58:14.075859+0000","last_fullsized":"2024-09-10T18:58:14.075859+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":326,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082711+0000","last_change":"2024-09-10T18:57:34.759089+0000","last_active":"2024-09-10T18:58:14.082711+0000","last_peered":"2024-09-10T18:58:14.082711+0000","last_clean":"2024-09-10T18:58:14.082711+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:58:14.082711+0000","last_undegraded":"2024-09-10T18:58:14.082711+0000","last_fullsized":"2024-09-10T18:58:14.082711+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.758993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.758993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:36.134527+0000","scrub_duration":0.0028569620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074426+0000","last_change":"2024-09-10T18:54:57.710979+0000","last_active":"2024-09-10T18:58:14.074426+0000","last_peered":"2024-09-10T18:58:14.074426+0000","last_clean":"2024-09-10T18:58:14.074426+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:58:14.074426+0000","last_undegraded":"2024-09-10T18:58:14.074426+0000","last_fullsized":"2024-09-10T18:58:14.074426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:57.710814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.710814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024229130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":308,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072566+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:58:14.072566+0000","last_peered":"2024-09-10T18:58:14.072566+0000","last_clean":"2024-09-10T18:58:14.072566+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:58:14.072566+0000","last_undegraded":"2024-09-10T18:58:14.072566+0000","last_fullsized":"2024-09-10T18:58:14.072566+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074757+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:58:14.074757+0000","last_peered":"2024-09-10T18:58:14.074757+0000","last_clean":"2024-09-10T18:58:14.074757+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:58:14.074757+0000","last_undegraded":"2024-09-10T18:58:14.074757+0000","last_fullsized":"2024-09-10T18:58:14.074757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":332,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068813+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:58:14.068813+0000","last_peered":"2024-09-10T18:58:14.068813+0000","last_clean":"2024-09-10T18:58:14.068813+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:58:14.068813+0000","last_undegraded":"2024-09-10T18:58:14.068813+0000","last_fullsized":"2024-09-10T18:58:14.068813+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.089263+0000","last_change":"2024-09-10T18:56:30.684729+0000","last_active":"2024-09-10T18:58:14.089263+0000","last_peered":"2024-09-10T18:58:14.089263+0000","last_clean":"2024-09-10T18:58:14.089263+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:58:14.089263+0000","last_undegraded":"2024-09-10T18:58:14.089263+0000","last_fullsized":"2024-09-10T18:58:14.089263+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.684589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.684589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023375330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":302,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090078+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:58:14.090078+0000","last_peered":"2024-09-10T18:58:14.090078+0000","last_clean":"2024-09-10T18:58:14.090078+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:58:14.090078+0000","last_undegraded":"2024-09-10T18:58:14.090078+0000","last_fullsized":"2024-09-10T18:58:14.090078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":300,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088187+0000","last_change":"2024-09-10T18:54:51.927199+0000","last_active":"2024-09-10T18:58:14.088187+0000","last_peered":"2024-09-10T18:58:14.088187+0000","last_clean":"2024-09-10T18:58:14.088187+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:58:14.088187+0000","last_undegraded":"2024-09-10T18:58:14.088187+0000","last_fullsized":"2024-09-10T18:58:14.088187+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:51.926947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:54:51.926947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003458273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090169+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:58:14.090169+0000","last_peered":"2024-09-10T18:58:14.090169+0000","last_clean":"2024-09-10T18:58:14.090169+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:58:14.090169+0000","last_undegraded":"2024-09-10T18:58:14.090169+0000","last_fullsized":"2024-09-10T18:58:14.090169+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":344,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076524+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:58:14.076524+0000","last_peered":"2024-09-10T18:58:14.076524+0000","last_clean":"2024-09-10T18:58:14.076524+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:58:14.076524+0000","last_undegraded":"2024-09-10T18:58:14.076524+0000","last_fullsized":"2024-09-10T18:58:14.076524+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":338,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.084635+0000","last_change":"2024-09-10T18:56:36.828365+0000","last_active":"2024-09-10T18:58:14.084635+0000","last_peered":"2024-09-10T18:58:14.084635+0000","last_clean":"2024-09-10T18:58:14.084635+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:58:14.084635+0000","last_undegraded":"2024-09-10T18:58:14.084635+0000","last_fullsized":"2024-09-10T18:58:14.084635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:56:36.828207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.828207+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038610699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":320,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067182+0000","last_change":"2024-09-10T18:57:34.244420+0000","last_active":"2024-09-10T18:58:14.067182+0000","last_peered":"2024-09-10T18:58:14.067182+0000","last_clean":"2024-09-10T18:58:14.067182+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:58:14.067182+0000","last_undegraded":"2024-09-10T18:58:14.067182+0000","last_fullsized":"2024-09-10T18:58:14.067182+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.244346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.244346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:36.724795+0000","scrub_duration":0.00228093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":308,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078627+0000","last_change":"2024-09-10T18:54:58.251951+0000","last_active":"2024-09-10T18:58:14.078627+0000","last_peered":"2024-09-10T18:58:14.078627+0000","last_clean":"2024-09-10T18:58:14.078627+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:58:14.078627+0000","last_undegraded":"2024-09-10T18:58:14.078627+0000","last_fullsized":"2024-09-10T18:58:14.078627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:58.251796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.251796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039082370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073156+0000","last_change":"2024-09-10T18:56:04.021031+0000","last_active":"2024-09-10T18:58:14.073156+0000","last_peered":"2024-09-10T18:58:14.073156+0000","last_clean":"2024-09-10T18:58:14.073156+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:58:14.073156+0000","last_undegraded":"2024-09-10T18:58:14.073156+0000","last_fullsized":"2024-09-10T18:58:14.073156+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:04.020966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:04.020966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002153622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":331,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076641+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:58:14.076641+0000","last_peered":"2024-09-10T18:58:14.076641+0000","last_clean":"2024-09-10T18:58:14.076641+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:58:14.076641+0000","last_undegraded":"2024-09-10T18:58:14.076641+0000","last_fullsized":"2024-09-10T18:58:14.076641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":367,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076342+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:58:14.076342+0000","last_peered":"2024-09-10T18:58:14.076342+0000","last_clean":"2024-09-10T18:58:14.076342+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:58:14.076342+0000","last_undegraded":"2024-09-10T18:58:14.076342+0000","last_fullsized":"2024-09-10T18:58:14.076342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":441,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074196+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T18:58:14.074196+0000","last_peered":"2024-09-10T18:58:14.074196+0000","last_clean":"2024-09-10T18:58:14.074196+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:58:14.074196+0000","last_undegraded":"2024-09-10T18:58:14.074196+0000","last_fullsized":"2024-09-10T18:58:14.074196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.279543+0000","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072505+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T18:58:14.072505+0000","last_peered":"2024-09-10T18:58:14.072505+0000","last_clean":"2024-09-10T18:58:14.072505+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:58:14.072505+0000","last_undegraded":"2024-09-10T18:58:14.072505+0000","last_fullsized":"2024-09-10T18:58:14.072505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.083086+0000","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":245,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072959+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:58:14.072959+0000","last_peered":"2024-09-10T18:58:14.072959+0000","last_clean":"2024-09-10T18:58:14.072959+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:58:14.072959+0000","last_undegraded":"2024-09-10T18:58:14.072959+0000","last_fullsized":"2024-09-10T18:58:14.072959+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":341,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072823+0000","last_change":"2024-09-10T18:57:33.408816+0000","last_active":"2024-09-10T18:58:14.072823+0000","last_peered":"2024-09-10T18:58:14.072823+0000","last_clean":"2024-09-10T18:58:14.072823+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:58:14.072823+0000","last_undegraded":"2024-09-10T18:58:14.072823+0000","last_fullsized":"2024-09-10T18:58:14.072823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:33.408655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:33.408655+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.644071+0000","scrub_duration":0.0039294170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":326,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076008+0000","last_change":"2024-09-10T18:57:39.808114+0000","last_active":"2024-09-10T18:58:14.076008+0000","last_peered":"2024-09-10T18:58:14.076008+0000","last_clean":"2024-09-10T18:58:14.076008+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:58:14.076008+0000","last_undegraded":"2024-09-10T18:58:14.076008+0000","last_fullsized":"2024-09-10T18:58:14.076008+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.807949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.807949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:43.276337+0000","scrub_duration":0.0034718420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092035+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:58:14.092035+0000","last_peered":"2024-09-10T18:58:14.092035+0000","last_clean":"2024-09-10T18:58:14.092035+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:58:14.092035+0000","last_undegraded":"2024-09-10T18:58:14.092035+0000","last_fullsized":"2024-09-10T18:58:14.092035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":320,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076873+0000","last_change":"2024-09-10T18:57:42.899838+0000","last_active":"2024-09-10T18:58:14.076873+0000","last_peered":"2024-09-10T18:58:14.076873+0000","last_clean":"2024-09-10T18:58:14.076873+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:58:14.076873+0000","last_undegraded":"2024-09-10T18:58:14.076873+0000","last_fullsized":"2024-09-10T18:58:14.076873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.899684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.899684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.010497+0000","scrub_duration":0.0036750730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076184+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:58:14.076184+0000","last_peered":"2024-09-10T18:58:14.076184+0000","last_clean":"2024-09-10T18:58:14.076184+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:58:14.076184+0000","last_undegraded":"2024-09-10T18:58:14.076184+0000","last_fullsized":"2024-09-10T18:58:14.076184+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":253,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074657+0000","last_change":"2024-09-10T18:58:11.107535+0000","last_active":"2024-09-10T18:58:14.074657+0000","last_peered":"2024-09-10T18:58:14.074657+0000","last_clean":"2024-09-10T18:58:14.074657+0000","last_became_active":"2024-09-10T18:58:11.106177+0000","last_became_peered":"2024-09-10T18:58:11.106177+0000","last_unstale":"2024-09-10T18:58:14.074657+0000","last_undegraded":"2024-09-10T18:58:14.074657+0000","last_fullsized":"2024-09-10T18:58:14.074657+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":335,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:32.795927+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.795927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032690250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":321,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071553+0000","last_change":"2024-09-10T18:55:11.602636+0000","last_active":"2024-09-10T18:58:14.071553+0000","last_peered":"2024-09-10T18:58:14.071553+0000","last_clean":"2024-09-10T18:58:14.071553+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:58:14.071553+0000","last_undegraded":"2024-09-10T18:58:14.071553+0000","last_fullsized":"2024-09-10T18:58:14.071553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:55:11.602358+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:55:11.602358+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039124090000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073451+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T18:58:14.073451+0000","last_peered":"2024-09-10T18:58:14.073451+0000","last_clean":"2024-09-10T18:58:14.073451+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:58:14.073451+0000","last_undegraded":"2024-09-10T18:58:14.073451+0000","last_fullsized":"2024-09-10T18:58:14.073451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.849115+0000","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082065+0000","last_change":"2024-09-10T18:55:10.697660+0000","last_active":"2024-09-10T18:58:14.082065+0000","last_peered":"2024-09-10T18:58:14.082065+0000","last_clean":"2024-09-10T18:58:14.082065+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:58:14.082065+0000","last_undegraded":"2024-09-10T18:58:14.082065+0000","last_fullsized":"2024-09-10T18:58:14.082065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:10.697558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:55:10.697558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002638226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075461+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T18:58:14.075461+0000","last_peered":"2024-09-10T18:58:14.075461+0000","last_clean":"2024-09-10T18:58:14.075461+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:58:14.075461+0000","last_undegraded":"2024-09-10T18:58:14.075461+0000","last_fullsized":"2024-09-10T18:58:14.075461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:13.562040+0000","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.080847+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:58:14.080847+0000","last_peered":"2024-09-10T18:58:14.080847+0000","last_clean":"2024-09-10T18:58:14.080847+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:58:14.080847+0000","last_undegraded":"2024-09-10T18:58:14.080847+0000","last_fullsized":"2024-09-10T18:58:14.080847+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3","version":"319'11","reported_seq":258,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078669+0000","last_change":"2024-09-10T18:58:07.105537+0000","last_active":"2024-09-10T18:58:14.078669+0000","last_peered":"2024-09-10T18:58:14.078669+0000","last_clean":"2024-09-10T18:58:14.078669+0000","last_became_active":"2024-09-10T18:58:07.105078+0000","last_became_peered":"2024-09-10T18:58:07.105078+0000","last_unstale":"2024-09-10T18:58:14.078669+0000","last_undegraded":"2024-09-10T18:58:14.078669+0000","last_fullsized":"2024-09-10T18:58:14.078669+0000","mapping_epoch":124,"log_start":"319'11","ondisk_log_start":"319'11","created":124,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:37.282578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.282578+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030887559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":336,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074155+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:58:14.074155+0000","last_peered":"2024-09-10T18:58:14.074155+0000","last_clean":"2024-09-10T18:58:14.074155+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:58:14.074155+0000","last_undegraded":"2024-09-10T18:58:14.074155+0000","last_fullsized":"2024-09-10T18:58:14.074155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":317,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078963+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:58:14.078963+0000","last_peered":"2024-09-10T18:58:14.078963+0000","last_clean":"2024-09-10T18:58:14.078963+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:58:14.078963+0000","last_undegraded":"2024-09-10T18:58:14.078963+0000","last_fullsized":"2024-09-10T18:58:14.078963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:40.607986+0000","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":434,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078593+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T18:58:14.078593+0000","last_peered":"2024-09-10T18:58:14.078593+0000","last_clean":"2024-09-10T18:58:14.078593+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:58:14.078593+0000","last_undegraded":"2024-09-10T18:58:14.078593+0000","last_fullsized":"2024-09-10T18:58:14.078593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:44.667849+0000","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":310,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091960+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:58:14.091960+0000","last_peered":"2024-09-10T18:58:14.091960+0000","last_clean":"2024-09-10T18:58:14.091960+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:58:14.091960+0000","last_undegraded":"2024-09-10T18:58:14.091960+0000","last_fullsized":"2024-09-10T18:58:14.091960+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091201+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:58:14.091201+0000","last_peered":"2024-09-10T18:58:14.091201+0000","last_clean":"2024-09-10T18:58:14.091201+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:58:14.091201+0000","last_undegraded":"2024-09-10T18:58:14.091201+0000","last_fullsized":"2024-09-10T18:58:14.091201+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"319'7","reported_seq":238,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088616+0000","last_change":"2024-09-10T18:56:27.205415+0000","last_active":"2024-09-10T18:58:14.088616+0000","last_peered":"2024-09-10T18:58:14.088616+0000","last_clean":"2024-09-10T18:58:14.088616+0000","last_became_active":"2024-09-10T18:54:37.606267+0000","last_became_peered":"2024-09-10T18:54:37.606267+0000","last_unstale":"2024-09-10T18:58:14.088616+0000","last_undegraded":"2024-09-10T18:58:14.088616+0000","last_fullsized":"2024-09-10T18:58:14.088616+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:27.205258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.205258+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049140169999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":330,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.089799+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:58:14.089799+0000","last_peered":"2024-09-10T18:58:14.089799+0000","last_clean":"2024-09-10T18:58:14.089799+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:58:14.089799+0000","last_undegraded":"2024-09-10T18:58:14.089799+0000","last_fullsized":"2024-09-10T18:58:14.089799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":316,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090770+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:58:14.090770+0000","last_peered":"2024-09-10T18:58:14.090770+0000","last_clean":"2024-09-10T18:58:14.090770+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:58:14.090770+0000","last_undegraded":"2024-09-10T18:58:14.090770+0000","last_fullsized":"2024-09-10T18:58:14.090770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":417,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079792+0000","last_change":"2024-09-10T18:54:53.253758+0000","last_active":"2024-09-10T18:58:14.079792+0000","last_peered":"2024-09-10T18:58:14.079792+0000","last_clean":"2024-09-10T18:58:14.079792+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:58:14.079792+0000","last_undegraded":"2024-09-10T18:58:14.079792+0000","last_fullsized":"2024-09-10T18:58:14.079792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:54:53.253659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.253659+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027928524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.076206+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T18:58:14.076206+0000","last_peered":"2024-09-10T18:58:14.076206+0000","last_clean":"2024-09-10T18:58:14.076206+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:58:14.076206+0000","last_undegraded":"2024-09-10T18:58:14.076206+0000","last_fullsized":"2024-09-10T18:58:14.076206+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.237854+0000","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":221,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091790+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:58:14.091790+0000","last_peered":"2024-09-10T18:58:14.091790+0000","last_clean":"2024-09-10T18:58:14.091790+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:58:14.091790+0000","last_undegraded":"2024-09-10T18:58:14.091790+0000","last_fullsized":"2024-09-10T18:58:14.091790+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1","version":"319'10","reported_seq":259,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-09-10T18:58:15.097903+0000","last_change":"2024-09-10T18:58:15.097903+0000","last_active":"2024-09-10T18:58:15.097903+0000","last_peered":"2024-09-10T18:58:15.097903+0000","last_clean":"2024-09-10T18:58:15.097903+0000","last_became_active":"2024-09-10T18:58:15.097586+0000","last_became_peered":"2024-09-10T18:58:15.097586+0000","last_unstale":"2024-09-10T18:58:15.097903+0000","last_undegraded":"2024-09-10T18:58:15.097903+0000","last_fullsized":"2024-09-10T18:58:15.097903+0000","mapping_epoch":124,"log_start":"319'10","ondisk_log_start":"319'10","created":124,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'5","last_scrub_stamp":"2024-09-10T18:56:39.805468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.805468+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":17,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040878859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":328,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075745+0000","last_change":"2024-09-10T18:56:17.491891+0000","last_active":"2024-09-10T18:58:14.075745+0000","last_peered":"2024-09-10T18:58:14.075745+0000","last_clean":"2024-09-10T18:58:14.075745+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:58:14.075745+0000","last_undegraded":"2024-09-10T18:58:14.075745+0000","last_fullsized":"2024-09-10T18:58:14.075745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:17.491714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.491714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044606300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":315,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078425+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:58:14.078425+0000","last_peered":"2024-09-10T18:58:14.078425+0000","last_clean":"2024-09-10T18:58:14.078425+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:58:14.078425+0000","last_undegraded":"2024-09-10T18:58:14.078425+0000","last_fullsized":"2024-09-10T18:58:14.078425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":311,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067796+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T18:58:14.067796+0000","last_peered":"2024-09-10T18:58:14.067796+0000","last_clean":"2024-09-10T18:58:14.067796+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:58:14.067796+0000","last_undegraded":"2024-09-10T18:58:14.067796+0000","last_fullsized":"2024-09-10T18:58:14.067796+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.840825+0000","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":313,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.087923+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:58:14.087923+0000","last_peered":"2024-09-10T18:58:14.087923+0000","last_clean":"2024-09-10T18:58:14.087923+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:58:14.087923+0000","last_undegraded":"2024-09-10T18:58:14.087923+0000","last_fullsized":"2024-09-10T18:58:14.087923+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075351+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:58:14.075351+0000","last_peered":"2024-09-10T18:58:14.075351+0000","last_clean":"2024-09-10T18:58:14.075351+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:58:14.075351+0000","last_undegraded":"2024-09-10T18:58:14.075351+0000","last_fullsized":"2024-09-10T18:58:14.075351+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.6","version":"319'13","reported_seq":265,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079784+0000","last_change":"2024-09-10T18:57:55.067344+0000","last_active":"2024-09-10T18:58:14.079784+0000","last_peered":"2024-09-10T18:58:14.079784+0000","last_clean":"2024-09-10T18:58:14.079784+0000","last_became_active":"2024-09-10T18:57:55.066835+0000","last_became_peered":"2024-09-10T18:57:55.066835+0000","last_unstale":"2024-09-10T18:58:14.079784+0000","last_undegraded":"2024-09-10T18:58:14.079784+0000","last_fullsized":"2024-09-10T18:58:14.079784+0000","mapping_epoch":124,"log_start":"126'9","ondisk_log_start":"126'9","created":124,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'9","last_scrub_stamp":"2024-09-10T18:56:09.704460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.704460+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":14,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003389736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":346,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075058+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T18:58:14.075058+0000","last_peered":"2024-09-10T18:58:14.075058+0000","last_clean":"2024-09-10T18:58:14.075058+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:58:14.075058+0000","last_undegraded":"2024-09-10T18:58:14.075058+0000","last_fullsized":"2024-09-10T18:58:14.075058+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.974665+0000","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.089191+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:58:14.089191+0000","last_peered":"2024-09-10T18:58:14.089191+0000","last_clean":"2024-09-10T18:58:14.089191+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:58:14.089191+0000","last_undegraded":"2024-09-10T18:58:14.089191+0000","last_fullsized":"2024-09-10T18:58:14.089191+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075157+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:58:14.075157+0000","last_peered":"2024-09-10T18:58:14.075157+0000","last_clean":"2024-09-10T18:58:14.075157+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:58:14.075157+0000","last_undegraded":"2024-09-10T18:58:14.075157+0000","last_fullsized":"2024-09-10T18:58:14.075157+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075185+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:58:14.075185+0000","last_peered":"2024-09-10T18:58:14.075185+0000","last_clean":"2024-09-10T18:58:14.075185+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:58:14.075185+0000","last_undegraded":"2024-09-10T18:58:14.075185+0000","last_fullsized":"2024-09-10T18:58:14.075185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":233,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075030+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:58:14.075030+0000","last_peered":"2024-09-10T18:58:14.075030+0000","last_clean":"2024-09-10T18:58:14.075030+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:58:14.075030+0000","last_undegraded":"2024-09-10T18:58:14.075030+0000","last_fullsized":"2024-09-10T18:58:14.075030+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"319'7","reported_seq":255,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.082459+0000","last_change":"2024-09-10T18:57:51.092320+0000","last_active":"2024-09-10T18:58:14.082459+0000","last_peered":"2024-09-10T18:58:14.082459+0000","last_clean":"2024-09-10T18:58:14.082459+0000","last_became_active":"2024-09-10T18:57:51.091993+0000","last_became_peered":"2024-09-10T18:57:51.091993+0000","last_unstale":"2024-09-10T18:58:14.082459+0000","last_undegraded":"2024-09-10T18:58:14.082459+0000","last_fullsized":"2024-09-10T18:58:14.082459+0000","mapping_epoch":124,"log_start":"126'5","ondisk_log_start":"126'5","created":124,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.160276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.160276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":5,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:58.768933+0000","scrub_duration":0.0035864650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":326,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.092274+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T18:58:14.092274+0000","last_peered":"2024-09-10T18:58:14.092274+0000","last_clean":"2024-09-10T18:58:14.092274+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:58:14.092274+0000","last_undegraded":"2024-09-10T18:58:14.092274+0000","last_fullsized":"2024-09-10T18:58:14.092274+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.849844+0000","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068863+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:58:14.068863+0000","last_peered":"2024-09-10T18:58:14.068863+0000","last_clean":"2024-09-10T18:58:14.068863+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:58:14.068863+0000","last_undegraded":"2024-09-10T18:58:14.068863+0000","last_fullsized":"2024-09-10T18:58:14.068863+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.071421+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:58:14.071421+0000","last_peered":"2024-09-10T18:58:14.071421+0000","last_clean":"2024-09-10T18:58:14.071421+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:58:14.071421+0000","last_undegraded":"2024-09-10T18:58:14.071421+0000","last_fullsized":"2024-09-10T18:58:14.071421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074403+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T18:58:14.074403+0000","last_peered":"2024-09-10T18:58:14.074403+0000","last_clean":"2024-09-10T18:58:14.074403+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:58:14.074403+0000","last_undegraded":"2024-09-10T18:58:14.074403+0000","last_fullsized":"2024-09-10T18:58:14.074403+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.521548+0000","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078884+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:58:14.078884+0000","last_peered":"2024-09-10T18:58:14.078884+0000","last_clean":"2024-09-10T18:58:14.078884+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:58:14.078884+0000","last_undegraded":"2024-09-10T18:58:14.078884+0000","last_fullsized":"2024-09-10T18:58:14.078884+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"126'2","reported_seq":246,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078771+0000","last_change":"2024-09-10T18:58:03.098960+0000","last_active":"2024-09-10T18:58:14.078771+0000","last_peered":"2024-09-10T18:58:14.078771+0000","last_clean":"2024-09-10T18:58:14.078771+0000","last_became_active":"2024-09-10T18:58:03.098470+0000","last_became_peered":"2024-09-10T18:58:03.098470+0000","last_unstale":"2024-09-10T18:58:14.078771+0000","last_undegraded":"2024-09-10T18:58:14.078771+0000","last_fullsized":"2024-09-10T18:58:14.078771+0000","mapping_epoch":124,"log_start":"126'2","ondisk_log_start":"126'2","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":2,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":328,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.068708+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:58:14.068708+0000","last_peered":"2024-09-10T18:58:14.068708+0000","last_clean":"2024-09-10T18:58:14.068708+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:58:14.068708+0000","last_undegraded":"2024-09-10T18:58:14.068708+0000","last_fullsized":"2024-09-10T18:58:14.068708+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":325,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081613+0000","last_change":"2024-09-10T18:57:56.351163+0000","last_active":"2024-09-10T18:58:14.081613+0000","last_peered":"2024-09-10T18:58:14.081613+0000","last_clean":"2024-09-10T18:58:14.081613+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:58:14.081613+0000","last_undegraded":"2024-09-10T18:58:14.081613+0000","last_fullsized":"2024-09-10T18:58:14.081613+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.351005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.351005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:59.113542+0000","scrub_duration":0.0035909499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074374+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:58:14.074374+0000","last_peered":"2024-09-10T18:58:14.074374+0000","last_clean":"2024-09-10T18:58:14.074374+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:58:14.074374+0000","last_undegraded":"2024-09-10T18:58:14.074374+0000","last_fullsized":"2024-09-10T18:58:14.074374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072930+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:58:14.072930+0000","last_peered":"2024-09-10T18:58:14.072930+0000","last_clean":"2024-09-10T18:58:14.072930+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:58:14.072930+0000","last_undegraded":"2024-09-10T18:58:14.072930+0000","last_fullsized":"2024-09-10T18:58:14.072930+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088766+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T18:58:14.088766+0000","last_peered":"2024-09-10T18:58:14.088766+0000","last_clean":"2024-09-10T18:58:14.088766+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:58:14.088766+0000","last_undegraded":"2024-09-10T18:58:14.088766+0000","last_fullsized":"2024-09-10T18:58:14.088766+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.553089+0000","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":251,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091638+0000","last_change":"2024-09-10T18:57:59.089772+0000","last_active":"2024-09-10T18:58:14.091638+0000","last_peered":"2024-09-10T18:58:14.091638+0000","last_clean":"2024-09-10T18:58:14.091638+0000","last_became_active":"2024-09-10T18:57:59.089458+0000","last_became_peered":"2024-09-10T18:57:59.089458+0000","last_unstale":"2024-09-10T18:58:14.091638+0000","last_undegraded":"2024-09-10T18:58:14.091638+0000","last_fullsized":"2024-09-10T18:58:14.091638+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.177249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.177249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033094600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075285+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:58:14.075285+0000","last_peered":"2024-09-10T18:58:14.075285+0000","last_clean":"2024-09-10T18:58:14.075285+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:58:14.075285+0000","last_undegraded":"2024-09-10T18:58:14.075285+0000","last_fullsized":"2024-09-10T18:58:14.075285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":348,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078997+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:58:14.078997+0000","last_peered":"2024-09-10T18:58:14.078997+0000","last_clean":"2024-09-10T18:58:14.078997+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:58:14.078997+0000","last_undegraded":"2024-09-10T18:58:14.078997+0000","last_fullsized":"2024-09-10T18:58:14.078997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":314,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067268+0000","last_change":"2024-09-10T18:55:09.170922+0000","last_active":"2024-09-10T18:58:14.067268+0000","last_peered":"2024-09-10T18:58:14.067268+0000","last_clean":"2024-09-10T18:58:14.067268+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:58:14.067268+0000","last_undegraded":"2024-09-10T18:58:14.067268+0000","last_fullsized":"2024-09-10T18:58:14.067268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:09.170742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:09.170742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043047989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.091977+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:58:14.091977+0000","last_peered":"2024-09-10T18:58:14.091977+0000","last_clean":"2024-09-10T18:58:14.091977+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:58:14.091977+0000","last_undegraded":"2024-09-10T18:58:14.091977+0000","last_fullsized":"2024-09-10T18:58:14.091977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079898+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:58:14.079898+0000","last_peered":"2024-09-10T18:58:14.079898+0000","last_clean":"2024-09-10T18:58:14.079898+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:58:14.079898+0000","last_undegraded":"2024-09-10T18:58:14.079898+0000","last_fullsized":"2024-09-10T18:58:14.079898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.a","version":"319'16","reported_seq":268,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.077128+0000","last_change":"2024-09-10T18:57:39.096358+0000","last_active":"2024-09-10T18:58:14.077128+0000","last_peered":"2024-09-10T18:58:14.077128+0000","last_clean":"2024-09-10T18:58:14.077128+0000","last_became_active":"2024-09-10T18:57:39.095796+0000","last_became_peered":"2024-09-10T18:57:39.095796+0000","last_unstale":"2024-09-10T18:58:14.077128+0000","last_undegraded":"2024-09-10T18:58:14.077128+0000","last_fullsized":"2024-09-10T18:58:14.077128+0000","mapping_epoch":124,"log_start":"126'10","ondisk_log_start":"126'10","created":124,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'10","last_scrub_stamp":"2024-09-10T18:56:28.793065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.793065+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":15,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039084699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":319,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088903+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T18:58:14.088903+0000","last_peered":"2024-09-10T18:58:14.088903+0000","last_clean":"2024-09-10T18:58:14.088903+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:58:14.088903+0000","last_undegraded":"2024-09-10T18:58:14.088903+0000","last_fullsized":"2024-09-10T18:58:14.088903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.514634+0000","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.072072+0000","last_change":"2024-09-10T18:54:58.705901+0000","last_active":"2024-09-10T18:58:14.072072+0000","last_peered":"2024-09-10T18:58:14.072072+0000","last_clean":"2024-09-10T18:58:14.072072+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:58:14.072072+0000","last_undegraded":"2024-09-10T18:58:14.072072+0000","last_fullsized":"2024-09-10T18:58:14.072072+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:54:58.705784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:58.705784+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033198820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075041+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T18:58:14.075041+0000","last_peered":"2024-09-10T18:58:14.075041+0000","last_clean":"2024-09-10T18:58:14.075041+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:58:14.075041+0000","last_undegraded":"2024-09-10T18:58:14.075041+0000","last_fullsized":"2024-09-10T18:58:14.075041+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.346480+0000","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":445,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081538+0000","last_change":"2024-09-10T18:54:55.748354+0000","last_active":"2024-09-10T18:58:14.081538+0000","last_peered":"2024-09-10T18:58:14.081538+0000","last_clean":"2024-09-10T18:58:14.081538+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:58:14.081538+0000","last_undegraded":"2024-09-10T18:58:14.081538+0000","last_fullsized":"2024-09-10T18:58:14.081538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:54:55.748077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.748077+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040024409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081452+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:58:14.081452+0000","last_peered":"2024-09-10T18:58:14.081452+0000","last_clean":"2024-09-10T18:58:14.081452+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:58:14.081452+0000","last_undegraded":"2024-09-10T18:58:14.081452+0000","last_fullsized":"2024-09-10T18:58:14.081452+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.b","version":"319'14","reported_seq":101,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.067727+0000","last_change":"2024-09-10T18:57:35.056450+0000","last_active":"2024-09-10T18:58:14.067727+0000","last_peered":"2024-09-10T18:58:14.067727+0000","last_clean":"2024-09-10T18:58:14.067727+0000","last_became_active":"2024-09-10T18:57:35.056095+0000","last_became_peered":"2024-09-10T18:57:35.056095+0000","last_unstale":"2024-09-10T18:58:14.067727+0000","last_undegraded":"2024-09-10T18:58:14.067727+0000","last_fullsized":"2024-09-10T18:58:14.067727+0000","mapping_epoch":274,"log_start":"126'10","ondisk_log_start":"126'10","created":124,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:56:16.658945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:16.658945+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":12,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038241360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075906+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T18:58:14.075906+0000","last_peered":"2024-09-10T18:58:14.075906+0000","last_clean":"2024-09-10T18:58:14.075906+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:58:14.075906+0000","last_undegraded":"2024-09-10T18:58:14.075906+0000","last_fullsized":"2024-09-10T18:58:14.075906+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.795787+0000","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.074542+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:58:14.074542+0000","last_peered":"2024-09-10T18:58:14.074542+0000","last_clean":"2024-09-10T18:58:14.074542+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:58:14.074542+0000","last_undegraded":"2024-09-10T18:58:14.074542+0000","last_fullsized":"2024-09-10T18:58:14.074542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":324,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.090501+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:58:14.090501+0000","last_peered":"2024-09-10T18:58:14.090501+0000","last_clean":"2024-09-10T18:58:14.090501+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:58:14.090501+0000","last_undegraded":"2024-09-10T18:58:14.090501+0000","last_fullsized":"2024-09-10T18:58:14.090501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":322,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.077164+0000","last_change":"2024-09-10T18:56:26.777537+0000","last_active":"2024-09-10T18:58:14.077164+0000","last_peered":"2024-09-10T18:58:14.077164+0000","last_clean":"2024-09-10T18:58:14.077164+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:58:14.077164+0000","last_undegraded":"2024-09-10T18:58:14.077164+0000","last_fullsized":"2024-09-10T18:58:14.077164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.777271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.777271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030242540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":231,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078319+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:58:14.078319+0000","last_peered":"2024-09-10T18:58:14.078319+0000","last_clean":"2024-09-10T18:58:14.078319+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:58:14.078319+0000","last_undegraded":"2024-09-10T18:58:14.078319+0000","last_fullsized":"2024-09-10T18:58:14.078319+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.8","version":"126'2","reported_seq":253,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.088711+0000","last_change":"2024-09-10T18:57:47.078409+0000","last_active":"2024-09-10T18:58:14.088711+0000","last_peered":"2024-09-10T18:58:14.088711+0000","last_clean":"2024-09-10T18:58:14.088711+0000","last_became_active":"2024-09-10T18:57:47.077956+0000","last_became_peered":"2024-09-10T18:57:47.077956+0000","last_unstale":"2024-09-10T18:58:14.088711+0000","last_undegraded":"2024-09-10T18:58:14.088711+0000","last_fullsized":"2024-09-10T18:58:14.088711+0000","mapping_epoch":124,"log_start":"126'2","ondisk_log_start":"126'2","created":124,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'2","last_scrub_stamp":"2024-09-10T18:57:36.585730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.585730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.361630+0000","scrub_duration":0.0027517190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":312,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.081980+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:58:14.081980+0000","last_peered":"2024-09-10T18:58:14.081980+0000","last_clean":"2024-09-10T18:58:14.081980+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:58:14.081980+0000","last_undegraded":"2024-09-10T18:58:14.081980+0000","last_fullsized":"2024-09-10T18:58:14.081980+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":318,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.073876+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:58:14.073876+0000","last_peered":"2024-09-10T18:58:14.073876+0000","last_clean":"2024-09-10T18:58:14.073876+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:58:14.073876+0000","last_undegraded":"2024-09-10T18:58:14.073876+0000","last_fullsized":"2024-09-10T18:58:14.073876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":342,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.075548+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T18:58:14.075548+0000","last_peered":"2024-09-10T18:58:14.075548+0000","last_clean":"2024-09-10T18:58:14.075548+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:58:14.075548+0000","last_undegraded":"2024-09-10T18:58:14.075548+0000","last_fullsized":"2024-09-10T18:58:14.075548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.200134+0000","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":304,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079066+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:58:14.079066+0000","last_peered":"2024-09-10T18:58:14.079066+0000","last_clean":"2024-09-10T18:58:14.079066+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:58:14.079066+0000","last_undegraded":"2024-09-10T18:58:14.079066+0000","last_fullsized":"2024-09-10T18:58:14.079066+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":232,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.078691+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:58:14.078691+0000","last_peered":"2024-09-10T18:58:14.078691+0000","last_clean":"2024-09-10T18:58:14.078691+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:58:14.078691+0000","last_undegraded":"2024-09-10T18:58:14.078691+0000","last_fullsized":"2024-09-10T18:58:14.078691+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":251,"reported_epoch":338,"state":"active+clean","last_fresh":"2024-09-10T18:58:14.079518+0000","last_change":"2024-09-10T18:57:43.108161+0000","last_active":"2024-09-10T18:58:14.079518+0000","last_peered":"2024-09-10T18:58:14.079518+0000","last_clean":"2024-09-10T18:58:14.079518+0000","last_became_active":"2024-09-10T18:57:43.107719+0000","last_became_peered":"2024-09-10T18:57:43.107719+0000","last_unstale":"2024-09-10T18:58:14.079518+0000","last_undegraded":"2024-09-10T18:58:14.079518+0000","last_fullsized":"2024-09-10T18:58:14.079518+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:21.650515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:21.650515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027691759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":192,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":191,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195524,"num_objects":124,"num_object_clones":2,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26075136,"data_stored":24586572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1802,"ondisk_log_size":1802,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":512,"num_read_kb":507,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3},{"poolid":16,"num_pg":17,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":51,"acting":51,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972471,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323672,"kb_used_data":13816,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93421480,"statfs":{"total":95995035648,"available":95663595520,"internally_reserved":0,"allocated":14147584,"data_stored":11747386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88200000000000001,"15min":0.91400000000000003},"min":{"1min":0.53300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.3420000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.92900000000000005,"15min":0.93400000000000005},"min":{"1min":0.58999999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.83999999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.875,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.98499999999999999,"15min":1.02},"min":{"1min":0.72499999999999998,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.3360000000000001,"5min":1.742,"15min":1.8280000000000001},"last":1.0600000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0309999999999999,"15min":1.0389999999999999},"min":{"1min":0.70699999999999996,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3580000000000001,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":1.2},{"interface":"front","average":{"1min":1.111,"5min":1.012,"15min":0.98699999999999999},"min":{"1min":0.57399999999999995,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":1.254}]},{"osd":3,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.0369999999999999,"15min":1.018},"min":{"1min":0.73199999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":4.2080000000000002,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":1.153},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.66500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.90400000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.04,"15min":1.0189999999999999},"min":{"1min":0.67800000000000005,"5min":0.499,"15min":0.443},"max":{"1min":1.2789999999999999,"5min":1.556,"15min":2.1989999999999998},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.1160000000000001,"15min":1.054},"min":{"1min":0.78500000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.2400000000000002,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.4890000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.026,"15min":1.0049999999999999},"min":{"1min":0.59799999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":1.115},{"interface":"front","average":{"1min":1.181,"5min":1.175,"15min":1.1779999999999999},"min":{"1min":0.59699999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":2.1419999999999999},"last":1.3520000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.71099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.591,"5min":1.591,"15min":2.2749999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2090000000000001,"15min":1.196},"min":{"1min":0.86099999999999999,"5min":0.59199999999999997,"15min":0.53100000000000003},"max":{"1min":4.1310000000000002,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.6910000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299521,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303128,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442024,"statfs":{"total":95995035648,"available":95684632576,"internally_reserved":0,"allocated":5890048,"data_stored":3552167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.872,"15min":0.88400000000000001},"min":{"1min":0.58399999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.032,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.57799999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1779999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.1699999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.196,"15min":0.997},"min":{"1min":0.56699999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9530000000000001,"5min":18.494,"15min":18.494},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.871,"15min":0.90200000000000002},"min":{"1min":0.73099999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.244,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":1.139}]},{"osd":2,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0249999999999999,"15min":0.95899999999999996},"min":{"1min":0.56399999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.2430000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.92300000000000004,"15min":0.92400000000000004},"min":{"1min":0.54300000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.3300000000000001,"5min":1.988,"15min":1.988},"last":1.0820000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0329999999999999,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.234,"5min":2.278,"15min":2.278},"last":0.876},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.98299999999999998,"15min":0.97399999999999998},"min":{"1min":0.73899999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.7529999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.97599999999999998,"15min":0.90400000000000003},"min":{"1min":0.58999999999999997,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.452,"5min":1.712,"15min":1.712},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.099,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.01}]},{"osd":5,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.999},"min":{"1min":0.67900000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.516,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.266},{"interface":"front","average":{"1min":1.252,"5min":1.163,"15min":1.1100000000000001},"min":{"1min":0.77000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.1669999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.2}]},{"osd":7,"last update":"Tue Sep 10 18:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.0960000000000001,"15min":1.0600000000000001},"min":{"1min":0.90200000000000002,"5min":0.503,"15min":0.503},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.175,"15min":1.153},"min":{"1min":0.93100000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.3280000000000001}]}]},{"osd":5,"up_from":43,"seq":184683593867,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319200,"kb_used_data":9344,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93425952,"statfs":{"total":95995035648,"available":95668174848,"internally_reserved":0,"allocated":9568256,"data_stored":7156011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.093,"15min":0.94399999999999995},"min":{"1min":0.59799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1539999999999999,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.878,"5min":0.998,"15min":0.91400000000000003},"min":{"1min":0.621,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.212,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.86599999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.6679999999999999,"15min":1.1339999999999999},"min":{"1min":0.63800000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.3140000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.421,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1659999999999999,"5min":31.847000000000001,"15min":31.847000000000001},"last":0.97699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.8999999999999999,"15min":1.2370000000000001},"min":{"1min":0.51000000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.752,"5min":61.347000000000001,"15min":61.347000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":1,"5min":1.135,"15min":0.98999999999999999},"min":{"1min":0.64000000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2450000000000001,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.79200000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1319999999999999,"15min":0.998},"min":{"1min":0.56299999999999994,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8520000000000001,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.177,"5min":1.798,"15min":1.236},"min":{"1min":0.83999999999999997,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":2.3820000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":0.83999999999999997}]},{"osd":4,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1830000000000001,"15min":0.92900000000000005},"min":{"1min":0.68300000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6519999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.071},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2250000000000001,"15min":0.998},"min":{"1min":0.77200000000000002,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":4.1970000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.3660000000000001,"15min":1.133},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":7.5380000000000003,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.355,"5min":1.407,"15min":1.165},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.47199999999999998},"max":{"1min":4.0709999999999997,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.0249999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":1.49,"15min":1.2010000000000001},"min":{"1min":0.68500000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":15.613,"5min":15.613,"15min":15.613},"last":1.1279999999999999},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.51,"15min":1.2749999999999999},"min":{"1min":0.72899999999999998,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":16.965,"5min":16.965,"15min":16.965},"last":0.95399999999999996}]}]},{"osd":4,"up_from":35,"seq":150323855511,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306688,"kb_used_data":5152,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438464,"statfs":{"total":95995035648,"available":95680987136,"internally_reserved":0,"allocated":5275648,"data_stored":2952704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.81000000000000005,"15min":0.84099999999999997},"min":{"1min":0.505,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3320000000000001,"5min":1.546,"15min":1.546},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.89500000000000002,"15min":0.86299999999999999},"min":{"1min":0.52900000000000003,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.4750000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.2090000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.85099999999999998,"15min":0.878},"min":{"1min":0.54700000000000004,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.97099999999999997,"5min":0.93700000000000006,"15min":0.88900000000000001},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.323,"5min":1.575,"15min":1.575},"last":0.745}]},{"osd":2,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94599999999999995,"15min":0.90700000000000003},"min":{"1min":0.58799999999999997,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.4179999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.048},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.90800000000000003},"min":{"1min":0.60099999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.482,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":1.365}]},{"osd":3,"last update":"Tue Sep 10 18:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.94299999999999995,"15min":0.91800000000000004},"min":{"1min":0.64400000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.714},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.006,"5min":0.96799999999999997,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.137}]},{"osd":5,"last update":"Tue Sep 10 18:58:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.97499999999999998,"15min":0.89900000000000002},"min":{"1min":0.58099999999999996,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.262,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.038,"15min":0.95999999999999996},"min":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.377,"5min":1.4610000000000001,"15min":1.7909999999999999},"last":1.4139999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0489999999999999,"15min":0.97999999999999998},"min":{"1min":0.69899999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.762,"5min":1.994,"15min":1.994},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.129,"15min":1.095},"min":{"1min":0.78700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.24}]},{"osd":7,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1379999999999999,"15min":1.1619999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.585,"5min":1.6200000000000001,"15min":2.008},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.173,"5min":1.1559999999999999,"15min":1.1639999999999999},"min":{"1min":0.68100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6830000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.272}]}]},{"osd":3,"up_from":29,"seq":124554051746,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315104,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93430048,"statfs":{"total":95995035648,"available":95672369152,"internally_reserved":0,"allocated":5373952,"data_stored":2961820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.2150000000000001,"15min":0.82199999999999995},"min":{"1min":0.38600000000000001,"5min":0.222,"15min":0.222},"max":{"1min":8.6180000000000003,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.2410000000000001,"15min":0.89100000000000001},"min":{"1min":0.441,"5min":0.252,"15min":0.252},"max":{"1min":5.3479999999999999,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.81000000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.3819999999999999,"15min":0.91600000000000004},"min":{"1min":0.41799999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.3879999999999999,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.62},{"interface":"front","average":{"1min":1.022,"5min":1.0600000000000001,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.0549999999999997,"5min":15.189,"15min":15.189},"last":1.0649999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.3660000000000001,"15min":0.98699999999999999},"min":{"1min":0.502,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1920000000000002,"5min":26.558,"15min":26.558},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.2410000000000001,"15min":0.91900000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.3199999999999998,"5min":25.536999999999999,"15min":25.536999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:57:16 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.3049999999999999,"15min":1.111},"min":{"1min":0.73799999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":7.3550000000000004,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.522,"5min":1.411,"15min":1.198},"min":{"1min":0.79000000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":7.4189999999999996,"5min":10.391,"15min":10.391},"last":1.3480000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:58:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.4279999999999999,"15min":1.212},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.355,"5min":17.535,"15min":17.535},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.105,"15min":1.0329999999999999},"min":{"1min":0.60399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.04,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.98999999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:58:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.288,"15min":1.2030000000000001},"min":{"1min":0.84099999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.9550000000000001,"5min":10.51,"15min":10.51},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.1439999999999999,"15min":1.0920000000000001},"min":{"1min":0.63100000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3769999999999998,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.131}]},{"osd":7,"last update":"Tue Sep 10 18:57:53 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.3220000000000001,"15min":1.179},"min":{"1min":0.60699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.214,"5min":18.312999999999999,"15min":18.312999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.4470000000000001,"15min":1.306},"min":{"1min":0.77500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.0270000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.3899999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378795,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314764,"kb_used_data":13244,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93430388,"statfs":{"total":95995035648,"available":95672717312,"internally_reserved":0,"allocated":13561856,"data_stored":11147585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.84199999999999997,"15min":0.71399999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.5629999999999999,"5min":9.516,"15min":9.516},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81200000000000006,"15min":0.76400000000000001},"min":{"1min":0.32400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.2350000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.72899999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.75900000000000001,"15min":0.70999999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.2380000000000004,"5min":5.516,"15min":5.516},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.84799999999999998,"15min":0.79900000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.26,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.628}]},{"osd":3,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.89400000000000002,"15min":0.85399999999999998},"min":{"1min":0.52500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.139,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.84599999999999997},"min":{"1min":0.53800000000000003,"5min":0.434,"15min":0.434},"max":{"1min":2.0539999999999998,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.79300000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.171,"15min":1.0389999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.2909999999999999,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.103},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.1519999999999999,"15min":1.1040000000000001},"min":{"1min":0.55100000000000005,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.4379999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.02}]},{"osd":5,"last update":"Tue Sep 10 18:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0860000000000001,"15min":1.101},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.458,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.198,"15min":1.0860000000000001},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.321,"5min":21.654,"15min":21.654},"last":0.95099999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.177,"15min":1.177},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.5089999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.262,"15min":1.1359999999999999},"min":{"1min":0.57899999999999996,"5min":0.505,"15min":0.505},"max":{"1min":3.367,"5min":15.971,"15min":15.971},"last":0.98799999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1819999999999999,"15min":1.1950000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":15.025,"15min":15.025},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.284,"15min":1.179},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.4239999999999999,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.0720000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575030,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306452,"kb_used_data":9092,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93438700,"statfs":{"total":95995035648,"available":95681228800,"internally_reserved":0,"allocated":9310208,"data_stored":6952035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.028,"15min":0.76500000000000001},"min":{"1min":0.34200000000000003,"5min":0.246,"15min":0.246},"max":{"1min":2.238,"5min":32.677999999999997,"15min":32.677999999999997},"last":1.5469999999999999},{"interface":"front","average":{"1min":0.87,"5min":1.1060000000000001,"15min":0.83599999999999997},"min":{"1min":0.32900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":3.8250000000000002,"5min":32.398000000000003,"15min":32.398000000000003},"last":2.7200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.117,"15min":0.83099999999999996},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.3109999999999999,"5min":26.097000000000001,"15min":26.097000000000001},"last":1.5149999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.3,"15min":0.94999999999999996},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.488,"5min":32.363,"15min":32.363},"last":4.0890000000000004}]},{"osd":3,"last update":"Tue Sep 10 18:57:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.208,"15min":0.95899999999999996},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.7360000000000002,"5min":25.704000000000001,"15min":25.704000000000001},"last":3.1469999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.476,"15min":0.98799999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1890000000000001,"5min":60.999000000000002,"15min":60.999000000000002},"last":2.6589999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:58:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.234,"15min":1.0900000000000001},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.762,"5min":12.808999999999999,"15min":12.808999999999999},"last":2.762},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.748,"15min":1.202},"min":{"1min":0.56000000000000005,"5min":0.378,"15min":0.378},"max":{"1min":6.181,"5min":60.529000000000003,"15min":60.529000000000003},"last":1.7090000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.2869999999999999,"15min":1.1040000000000001},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":2.194,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.3069999999999999,"15min":1.1930000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.2719999999999998,"5min":12.84,"15min":12.84},"last":2.7949999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.528,"15min":1.244},"min":{"1min":0.59399999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.3279999999999998,"5min":25.608000000000001,"15min":25.608000000000001},"last":3.121},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.526,"15min":1.2270000000000001},"min":{"1min":0.63300000000000001,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":6.1219999999999999,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.9359999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.51,"15min":1.3140000000000001},"min":{"1min":0.66200000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.4849999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":3.0880000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.617,"15min":1.3029999999999999},"min":{"1min":0.71499999999999997,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":6.2599999999999998,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.839}]}]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315600,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93429552,"statfs":{"total":95995035648,"available":95671861248,"internally_reserved":0,"allocated":5890048,"data_stored":3542139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.6240000000000001,"15min":0.94899999999999995},"min":{"1min":0.39100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0920000000000001,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.716,"15min":1.038},"min":{"1min":0.46000000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.2629999999999999,"5min":86.009,"15min":86.009},"last":0.61199999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.601,"15min":0.99299999999999999},"min":{"1min":0.32900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":1.474,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.2,"15min":0.88300000000000001},"min":{"1min":0.48299999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.2450000000000001,"5min":24.614000000000001,"15min":24.614000000000001},"last":1.105}]},{"osd":3,"last update":"Tue Sep 10 18:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.0820000000000001,"15min":0.86399999999999999},"min":{"1min":0.38500000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3680000000000003,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.45,"15min":1.0129999999999999},"min":{"1min":0.39800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.0019999999999998,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:57:19 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.1160000000000001,"15min":1.0189999999999999},"min":{"1min":0.72399999999999998,"5min":0.5,"15min":0.5},"max":{"1min":1.3799999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.032},{"interface":"front","average":{"1min":1.022,"5min":1.143,"15min":1.0760000000000001},"min":{"1min":0.63100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.4990000000000001,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.1879999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:58:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.268,"15min":1.1060000000000001},"min":{"1min":0.85799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":2.609,"5min":10.048,"15min":10.048},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.101,"15min":1.0980000000000001},"min":{"1min":0.80600000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3400000000000001,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":1.1339999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:57:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1599999999999999,"15min":1.131},"min":{"1min":0.81299999999999994,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.4550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.3089999999999999,"15min":1.1890000000000001},"min":{"1min":0.86199999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.4929999999999999,"5min":11.894,"15min":11.894},"last":0.95099999999999996}]},{"osd":7,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1659999999999999,"15min":1.167},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.512,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3260000000000001,"15min":1.3029999999999999},"min":{"1min":0.80100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.3780000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.2410000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4182016,"data_stored":4000468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4194856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8194816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":191,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:58:18.606 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-10T18:58:18.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:18.713 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:18.747 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:18.850 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:18.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:18.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:18.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:19.023 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:19.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:19.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:19.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:19.281 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:17.706567+0000 mgr.y (mgr.24526) 367 : audit [DBG] from='client.25164 v1:172.21.15.83:0/3157946003' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:17.847396+0000 mgr.y (mgr.24526) 368 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: cluster 2024-09-10T18:58:18.034297+0000 mgr.y (mgr.24526) 369 : cluster [DBG] pgmap v496: 253 pgs: 8 unknown, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.082622+0000 mon.a (mon.0) 2206 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.082841+0000 mon.a (mon.0) 2207 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: cluster 2024-09-10T18:58:18.083154+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.088768+0000 mon.b (mon.1) 271 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.090347+0000 mon.c (mon.2) 576 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.091798+0000 mon.c (mon.2) 577 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.092904+0000 mon.c (mon.2) 578 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.093095+0000 mon.a (mon.0) 2209 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: audit 2024-09-10T18:58:18.093980+0000 mon.a (mon.0) 2210 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[18629]: cluster 2024-09-10T18:58:18.518458+0000 mon.a (mon.0) 2211 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:17.706567+0000 mgr.y (mgr.24526) 367 : audit [DBG] from='client.25164 v1:172.21.15.83:0/3157946003' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:17.847396+0000 mgr.y (mgr.24526) 368 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: cluster 2024-09-10T18:58:18.034297+0000 mgr.y (mgr.24526) 369 : cluster [DBG] pgmap v496: 253 pgs: 8 unknown, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.082622+0000 mon.a (mon.0) 2206 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.082841+0000 mon.a (mon.0) 2207 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: cluster 2024-09-10T18:58:18.083154+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:58:19.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.088768+0000 mon.b (mon.1) 271 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.090347+0000 mon.c (mon.2) 576 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.091798+0000 mon.c (mon.2) 577 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.092904+0000 mon.c (mon.2) 578 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.093095+0000 mon.a (mon.0) 2209 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: audit 2024-09-10T18:58:18.093980+0000 mon.a (mon.0) 2210 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:19 smithi083 bash[32381]: cluster 2024-09-10T18:58:18.518458+0000 mon.a (mon.0) 2211 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:19.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:17.706567+0000 mgr.y (mgr.24526) 367 : audit [DBG] from='client.25164 v1:172.21.15.83:0/3157946003' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:17.847396+0000 mgr.y (mgr.24526) 368 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: cluster 2024-09-10T18:58:18.034297+0000 mgr.y (mgr.24526) 369 : cluster [DBG] pgmap v496: 253 pgs: 8 unknown, 245 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.082622+0000 mon.a (mon.0) 2206 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.082841+0000 mon.a (mon.0) 2207 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pg_num","val":"11"}]': finished 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: cluster 2024-09-10T18:58:18.083154+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.088768+0000 mon.b (mon.1) 271 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.090347+0000 mon.c (mon.2) 576 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.091798+0000 mon.c (mon.2) 577 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.092904+0000 mon.c (mon.2) 578 : audit [INF] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.093095+0000 mon.a (mon.0) 2209 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: audit 2024-09-10T18:58:18.093980+0000 mon.a (mon.0) 2210 : audit [INF] from='client.24818 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T18:58:19.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:19 smithi165 bash[22591]: cluster 2024-09-10T18:58:18.518458+0000 mon.a (mon.0) 2211 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:19.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:19.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:19.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:19.563 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:19.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:19.694 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:19.773 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:19.799 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:19.829 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:19.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:19.969 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:19.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:20.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:20.102 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:19.090548+0000 mon.a (mon.0) 2212 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:19.090623+0000 mon.a (mon.0) 2213 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: cluster 2024-09-10T18:58:19.090829+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:19.093328+0000 mon.c (mon.2) 579 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:19.094169+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:20.035763+0000 mon.c (mon.2) 580 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[18629]: audit 2024-09-10T18:58:20.037899+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:20.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:20.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:19.090548+0000 mon.a (mon.0) 2212 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T18:58:20.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:19.090623+0000 mon.a (mon.0) 2213 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:58:20.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: cluster 2024-09-10T18:58:19.090829+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:58:20.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:19.093328+0000 mon.c (mon.2) 579 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:20.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:19.094169+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:20.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:20.035763+0000 mon.c (mon.2) 580 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:20 smithi165 bash[22591]: audit 2024-09-10T18:58:20.037899+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:20.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:20.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:20.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:20.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:20.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:19.090548+0000 mon.a (mon.0) 2212 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:19.090623+0000 mon.a (mon.0) 2213 : audit [INF] from='client.24818 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi083-77680-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: cluster 2024-09-10T18:58:19.090829+0000 mon.a (mon.0) 2214 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:19.093328+0000 mon.c (mon.2) 579 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:19.094169+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:20.035763+0000 mon.c (mon.2) 580 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:20 smithi083 bash[32381]: audit 2024-09-10T18:58:20.037899+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-10T18:58:20.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:20.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:20.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:20.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:20.784 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:20.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:20.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:20.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:21.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:21.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:21.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:21.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: cluster 2024-09-10T18:58:20.035053+0000 mgr.y (mgr.24526) 370 : cluster [DBG] pgmap v499: 244 pgs: 244 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:21.180 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.096763+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.096825+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: cluster 2024-09-10T18:58:20.096908+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.097823+0000 mon.c (mon.2) 581 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.098378+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.132407+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.133871+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.136314+0000 mon.a (mon.0) 2221 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.181 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[22591]: audit 2024-09-10T18:58:20.137490+0000 mon.a (mon.0) 2222 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.200 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:21.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:21.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:21.335 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:21.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: cluster 2024-09-10T18:58:20.035053+0000 mgr.y (mgr.24526) 370 : cluster [DBG] pgmap v499: 244 pgs: 244 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.096763+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.096825+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: cluster 2024-09-10T18:58:20.096908+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.097823+0000 mon.c (mon.2) 581 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.098378+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:21.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.132407+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.133871+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.136314+0000 mon.a (mon.0) 2221 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[18629]: audit 2024-09-10T18:58:20.137490+0000 mon.a (mon.0) 2222 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: cluster 2024-09-10T18:58:20.035053+0000 mgr.y (mgr.24526) 370 : cluster [DBG] pgmap v499: 244 pgs: 244 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.096763+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.096825+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: cluster 2024-09-10T18:58:20.096908+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.097823+0000 mon.c (mon.2) 581 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.098378+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]: dispatch 2024-09-10T18:58:21.453 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.132407+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.454 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.133871+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.454 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.136314+0000 mon.a (mon.0) 2221 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:21.454 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:21 smithi083 bash[32381]: audit 2024-09-10T18:58:20.137490+0000 mon.a (mon.0) 2222 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]: dispatch 2024-09-10T18:58:21.473 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:21.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:21.609 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:21.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:21.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:21.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:21.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:21.882 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:21.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:21.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:22.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:22.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:22.144 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: cluster 2024-09-10T18:58:20.528609+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:22.144 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: cluster 2024-09-10T18:58:20.528967+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:22.144 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.103496+0000 mon.c (mon.2) 582 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.103994+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.104366+0000 mon.a (mon.0) 2224 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]': finished 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: cluster 2024-09-10T18:58:21.104525+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.135851+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.137571+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[32381]: audit 2024-09-10T18:58:21.137992+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: cluster 2024-09-10T18:58:20.528609+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: cluster 2024-09-10T18:58:20.528967+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.103496+0000 mon.c (mon.2) 582 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:22.145 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.103994+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:22.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.104366+0000 mon.a (mon.0) 2224 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]': finished 2024-09-10T18:58:22.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: cluster 2024-09-10T18:58:21.104525+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:58:22.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.135851+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.137571+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.146 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18629]: audit 2024-09-10T18:58:21.137992+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:22.161 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:22.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:22.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: cluster 2024-09-10T18:58:20.528609+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: cluster 2024-09-10T18:58:20.528967+0000 osd.0 (osd.0) 199 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:22.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.103496+0000 mon.c (mon.2) 582 : audit [DBG] from='client.? v1:172.21.15.83:0/1232835119' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.103994+0000 mon.a (mon.0) 2223 : audit [INF] from='client.? v1:172.21.15.83:0/4044906782' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi083-77648-56"}]': finished 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.104366+0000 mon.a (mon.0) 2224 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-41"}]': finished 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: cluster 2024-09-10T18:58:21.104525+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.135851+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.137571+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:22.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:22 smithi165 bash[22591]: audit 2024-09-10T18:58:21.137992+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:22.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:22.422 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:22.525 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:22.558 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:22.657 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:22.660 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:22.683 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:22.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:22.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:22] "GET /metrics HTTP/1.1" 200 231112 "" "Prometheus/2.43.0" 2024-09-10T18:58:22.820 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:22.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:22.959 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:22.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:22.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:23.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:23.096 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:23.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:23.230 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:23.272 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:58:23.272 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:58:23.272 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T18:58:23.272 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T18:58:23.272 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:58:23.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:23.344 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:23 smithi165 bash[22591]: cluster 2024-09-10T18:58:22.036046+0000 mgr.y (mgr.24526) 371 : cluster [DBG] pgmap v502: 244 pgs: 2 peering, 242 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:23.344 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:23 smithi165 bash[22591]: audit 2024-09-10T18:58:22.128854+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:23.344 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:23 smithi165 bash[22591]: cluster 2024-09-10T18:58:22.128965+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:58:23.344 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:23 smithi165 bash[22591]: audit 2024-09-10T18:58:22.130946+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:23.365 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:23.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[18629]: cluster 2024-09-10T18:58:22.036046+0000 mgr.y (mgr.24526) 371 : cluster [DBG] pgmap v502: 244 pgs: 2 peering, 242 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[18629]: audit 2024-09-10T18:58:22.128854+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[18629]: cluster 2024-09-10T18:58:22.128965+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:58:23.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[18629]: audit 2024-09-10T18:58:22.130946+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:23.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[32381]: cluster 2024-09-10T18:58:22.036046+0000 mgr.y (mgr.24526) 371 : cluster [DBG] pgmap v502: 244 pgs: 2 peering, 242 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:23.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[32381]: audit 2024-09-10T18:58:22.128854+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi083-77648-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:23.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[32381]: cluster 2024-09-10T18:58:22.128965+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2024-09-10T18:58:23.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:23 smithi083 bash[32381]: audit 2024-09-10T18:58:22.130946+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:23.501 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:23.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:23.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:23.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:23.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:23.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:23.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:23.899 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:23.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:23.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T18:58:24.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:24.032 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:24.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:24.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:24.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:24.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:24.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:24.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:23.587456+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: cluster 2024-09-10T18:58:23.588553+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:58:24.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:23.591245+0000 mon.a (mon.0) 2233 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.037689+0000 mon.c (mon.2) 583 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.040020+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: cluster 2024-09-10T18:58:24.066264+0000 mon.a (mon.0) 2235 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.081902+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.082109+0000 mon.a (mon.0) 2237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.082319+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: cluster 2024-09-10T18:58:24.082403+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.085288+0000 mon.b (mon.1) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.086019+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.089692+0000 mon.a (mon.0) 2240 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.117331+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.119118+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[32381]: audit 2024-09-10T18:58:24.119474+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:23.587456+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: cluster 2024-09-10T18:58:23.588553+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:23.591245+0000 mon.a (mon.0) 2233 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.037689+0000 mon.c (mon.2) 583 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.040020+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: cluster 2024-09-10T18:58:24.066264+0000 mon.a (mon.0) 2235 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.081902+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.082109+0000 mon.a (mon.0) 2237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.082319+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-10T18:58:24.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: cluster 2024-09-10T18:58:24.082403+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.085288+0000 mon.b (mon.1) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.086019+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.089692+0000 mon.a (mon.0) 2240 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.117331+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.119118+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.415 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:24 smithi083 bash[18629]: audit 2024-09-10T18:58:24.119474+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:24.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:24.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:24.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:24.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:23.587456+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: cluster 2024-09-10T18:58:23.588553+0000 mon.a (mon.0) 2232 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:23.591245+0000 mon.a (mon.0) 2233 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.037689+0000 mon.c (mon.2) 583 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.040020+0000 mon.a (mon.0) 2234 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: cluster 2024-09-10T18:58:24.066264+0000 mon.a (mon.0) 2235 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.081902+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi083-77648-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.082109+0000 mon.a (mon.0) 2237 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.082319+0000 mon.a (mon.0) 2238 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi083-77680-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: cluster 2024-09-10T18:58:24.082403+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.085288+0000 mon.b (mon.1) 275 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.086019+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.089692+0000 mon.a (mon.0) 2240 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.117331+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.119118+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:24.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:24 smithi165 bash[22591]: audit 2024-09-10T18:58:24.119474+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:24.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:24.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:24.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:24.844 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:24.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:24.982 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:24.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:24.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:25.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:25.112 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:25.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:25.250 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:25.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: cluster 2024-09-10T18:58:24.037004+0000 mgr.y (mgr.24526) 372 : cluster [DBG] pgmap v505: 243 pgs: 32 unknown, 1 peering, 210 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: audit 2024-09-10T18:58:25.082547+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: audit 2024-09-10T18:58:25.083419+0000 mon.a (mon.0) 2244 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: audit 2024-09-10T18:58:25.083718+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: cluster 2024-09-10T18:58:25.083794+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:58:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: audit 2024-09-10T18:58:25.084486+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:25.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:25 smithi165 bash[22591]: audit 2024-09-10T18:58:25.086589+0000 mon.a (mon.0) 2248 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.379 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: cluster 2024-09-10T18:58:24.037004+0000 mgr.y (mgr.24526) 372 : cluster [DBG] pgmap v505: 243 pgs: 32 unknown, 1 peering, 210 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: audit 2024-09-10T18:58:25.082547+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: audit 2024-09-10T18:58:25.083419+0000 mon.a (mon.0) 2244 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: audit 2024-09-10T18:58:25.083718+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: cluster 2024-09-10T18:58:25.083794+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:58:25.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: audit 2024-09-10T18:58:25.084486+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[32381]: audit 2024-09-10T18:58:25.086589+0000 mon.a (mon.0) 2248 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: cluster 2024-09-10T18:58:24.037004+0000 mgr.y (mgr.24526) 372 : cluster [DBG] pgmap v505: 243 pgs: 32 unknown, 1 peering, 210 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: audit 2024-09-10T18:58:25.082547+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: audit 2024-09-10T18:58:25.083419+0000 mon.a (mon.0) 2244 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: audit 2024-09-10T18:58:25.083718+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi083-77680-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:25.441 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: cluster 2024-09-10T18:58:25.083794+0000 mon.a (mon.0) 2246 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2024-09-10T18:58:25.442 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: audit 2024-09-10T18:58:25.084486+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:25.442 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:25 smithi083 bash[18629]: audit 2024-09-10T18:58:25.086589+0000 mon.a (mon.0) 2248 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T18:58:25.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:25.507 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:25.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:25.646 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:25.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:25.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:25.781 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:25.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:25.918 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:25.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:26.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:26.054 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:26.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:26.187 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:26.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:26.325 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:26.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:26.457 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:26.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:26.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:26.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:26.725 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:26.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:26.865 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:26.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:26.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:27.005 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:27.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: cluster 2024-09-10T18:58:25.495925+0000 osd.6 (osd.6) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: cluster 2024-09-10T18:58:25.496191+0000 osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: cluster 2024-09-10T18:58:26.039135+0000 mgr.y (mgr.24526) 373 : cluster [DBG] pgmap v508: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: audit 2024-09-10T18:58:26.522756+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: audit 2024-09-10T18:58:26.523502+0000 mon.a (mon.0) 2249 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: cluster 2024-09-10T18:58:26.523775+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: audit 2024-09-10T18:58:26.525448+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[22591]: audit 2024-09-10T18:58:26.534164+0000 mon.a (mon.0) 2252 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.131 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:27.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:27.268 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:27.279 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: cluster 2024-09-10T18:58:25.495925+0000 osd.6 (osd.6) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:27.279 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: cluster 2024-09-10T18:58:25.496191+0000 osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: cluster 2024-09-10T18:58:26.039135+0000 mgr.y (mgr.24526) 373 : cluster [DBG] pgmap v508: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: audit 2024-09-10T18:58:26.522756+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: audit 2024-09-10T18:58:26.523502+0000 mon.a (mon.0) 2249 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: cluster 2024-09-10T18:58:26.523775+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: audit 2024-09-10T18:58:26.525448+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[18629]: audit 2024-09-10T18:58:26.534164+0000 mon.a (mon.0) 2252 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: cluster 2024-09-10T18:58:25.495925+0000 osd.6 (osd.6) 157 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: cluster 2024-09-10T18:58:25.496191+0000 osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:27.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: cluster 2024-09-10T18:58:26.039135+0000 mgr.y (mgr.24526) 373 : cluster [DBG] pgmap v508: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T18:58:27.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: audit 2024-09-10T18:58:26.522756+0000 mon.b (mon.1) 278 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: audit 2024-09-10T18:58:26.523502+0000 mon.a (mon.0) 2249 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T18:58:27.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: cluster 2024-09-10T18:58:26.523775+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2024-09-10T18:58:27.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: audit 2024-09-10T18:58:26.525448+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:27.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:27 smithi083 bash[32381]: audit 2024-09-10T18:58:26.534164+0000 mon.a (mon.0) 2252 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:27.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:27.402 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:27.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:27.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:27.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:27.676 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:27.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:27.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:27.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:27.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:27.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:27.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:28.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:28.083 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:28.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:28.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetR8 expected=8 2024-09-10T18:58:28.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-09-10T18:58:28.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:b29083e3:::5:head expected=16:b29083e3:::5:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=5 expected=5 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:566253c9:::13:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:566253c9:::13:head expected=16:566253c9:::13:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=13 expected=13 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:02547ec2:::1:head expected=16:02547ec2:::1:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=1 expected=1 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:863748b0:::15:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:863748b0:::15:head expected=16:863748b0:::15:head 2024-09-10T18:58:28.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=15 expected=15 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:6cac518f:::0:head expected=16:6cac518f:::0:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=0 expected=0 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:62a1935d:::14:head expected=16:62a1935d:::14:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=14 expected=14 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:5c6b0b28:::7:head expected=16:5c6b0b28:::7:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=7 expected=7 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-09-10T18:58:28.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=16:52ea6a34:::10:head expected=16:52ea6a34:::10:head 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=10 expected=10 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (367 ms) 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (197576 ms) 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 0/5 -> MIN 16:33333333::::head 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 1/5 -> 16:33333333::::head 16:66666666::::head 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 2/5 -> 16:66666666::::head 16:99999999::::head 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 3/5 -> 16:99999999::::head 16:cccccccc::::head 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 4/5 -> 16:cccccccc::::head MAX 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (27178 ms) 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 7 tests from LibRadosList (225738 ms total) 2024-09-10T18:58:28.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1042 ms) 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo2 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo3 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo4 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo5 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo6,foo7 2024-09-10T18:58:28.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo7 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo6 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo4 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo5 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo7 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo6 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo1 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo1 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo2 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo3 2024-09-10T18:58:28.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (56 ms) 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.093010+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.093095+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.093153+0000 mon.a (mon.0) 2255 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: cluster 2024-09-10T18:58:27.093224+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.093337+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.093867+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:28.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:28 smithi165 bash[22591]: audit 2024-09-10T18:58:27.097124+0000 mon.a (mon.0) 2258 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.093010+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.093095+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.093153+0000 mon.a (mon.0) 2255 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: cluster 2024-09-10T18:58:27.093224+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.093337+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.093867+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[18629]: audit 2024-09-10T18:58:27.097124+0000 mon.a (mon.0) 2258 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.093010+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi083-77680-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.093095+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.093153+0000 mon.a (mon.0) 2255 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: cluster 2024-09-10T18:58:27.093224+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2024-09-10T18:58:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.093337+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.093867+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]: dispatch 2024-09-10T18:58:28.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:28 smithi083 bash[32381]: audit 2024-09-10T18:58:27.097124+0000 mon.a (mon.0) 2258 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T18:58:28.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:28.342 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:28.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:28.478 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:28.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:28.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListECead 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: ok, hit_set contains 191:602f83fe:::foo:head 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9020 ms) 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg_num = 32 2024-09-10T18:58:28.619 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 0 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 1 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 2 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 3 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 4 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 5 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 6 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 7 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 8 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 9 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 10 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 11 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 12 ls 1725994709,0 2024-09-10T18:58:28.620 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 13 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 14 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 15 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 16 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 17 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 18 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 19 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 20 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 21 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 22 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 23 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 24 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 25 ls 1725994709,0 2024-09-10T18:58:28.621 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 26 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 27 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 28 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 29 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 30 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 31 ls 1725994709,0 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg_num = 32 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6cac518f:::0:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:02547ec2:::1:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f905c69b:::2:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:cfc208b3:::3:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d83876eb:::4:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b29083e3:::5:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c4fdafeb:::6:head 2024-09-10T18:58:28.622 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:5c6b0b28:::7:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:bd63b0f1:::8:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e960b815:::9:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:52ea6a34:::10:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:89d3ae78:::11:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:de5d7c5f:::12:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:566253c9:::13:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:62a1935d:::14:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:863748b0:::15:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3958e169:::16:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4d4dabf9:::17:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:8391935d:::18:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:28883081:::19:head 2024-09-10T18:58:28.623 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:69259c59:::20:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4bdb80b7:::21:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a11c5d71:::22:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:271af37b:::23:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:95b121be:::24:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:58d1031b:::25:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:0a050783:::26:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c709704c:::27:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:cbe56eaf:::28:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:86b4b162:::29:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:70d89383:::30:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:dd450c7c:::31:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6d5729b1:::32:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c388f3fb:::33:head 2024-09-10T18:58:28.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:56cfea31:::34:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9dbc1bf7:::35:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:40b74ccd:::36:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4d5aaf42:::37:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:920f362c:::38:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6cc53222:::39:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9cad833f:::40:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:1ea84d41:::41:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c4480ef6:::42:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a694361e:::43:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d1bd33e9:::44:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ddc2cd5d:::45:head 2024-09-10T18:58:28.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:2b782207:::46:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7b187fca:::47:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:90ecdf6f:::48:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a5ed95fe:::49:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ea0eaa55:::50:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f33ef17b:::51:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a0d1b2f6:::52:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:60c5229e:::53:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:edcbc575:::54:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:102cf253:::55:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:efb7fb0b:::56:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:50d0a326:::57:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d4dc5daf:::58:head 2024-09-10T18:58:28.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3a130462:::59:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ec87ed71:::60:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d5bc9454:::61:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3ddfe313:::62:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7c2816b9:::63:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:47e00e4d:::64:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c6410c18:::65:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b48ed237:::66:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:cd63ad31:::67:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b179e92b:::68:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:0d9f741a:::69:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6d3352ae:::70:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c6d5c19e:::71:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:bc4729c3:::72:head 2024-09-10T18:58:28.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:77e930b9:::73:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:0abeecfd:::74:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b7c37e15:::75:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b6378398:::76:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:02bd68de:::77:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:cc795d2d:::78:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:630d4fea:::79:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e0d29ef5:::80:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:fd6f13d2:::81:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:606461d5:::82:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:eadbdc43:::83:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:8761d0bb:::84:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9ef0186f:::85:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e0d41294:::86:head 2024-09-10T18:58:28.628 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:961de695:::87:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:1423148f:::88:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:633a8fa2:::89:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a8653809:::90:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3dac8b33:::91:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:35aad435:::92:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f6dcc343:::93:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:dbbdad87:::94:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:1cb48ce0:::95:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:03cd461c:::96:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:17a4ea99:::97:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9993c9a7:::98:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6394211c:::99:head 2024-09-10T18:58:28.629 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:94c7ae57:::100:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6fdee5bb:::101:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9a477fd1:::102:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:eb850916:::103:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:affc56b9:::104:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b42dc814:::105:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f319f8f0:::106:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9a40b9de:::107:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:8b524f28:::108:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e3de589f:::109:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:90f90a5b:::110:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a7b4f1d7:::111:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:af51766e:::112:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b6f90bd1:::113:head 2024-09-10T18:58:28.630 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e0261208:::114:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c9569ef7:::115:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:61bebe50:::116:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:fe93412b:::117:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d3d38bee:::118:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3100ba0c:::119:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d0560ada:::120:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f0ea8b35:::121:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:766f231a:::122:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a07a2582:::123:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:bd7c6b3a:::124:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:fb2ddaff:::125:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4408e1fe:::126:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ee1df7a7:::127:head 2024-09-10T18:58:28.631 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c3002909:::128:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4f48ffa9:::129:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:edf38733:::130:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c08425c0:::131:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:5f902d98:::132:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:41ea2c93:::133:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:813cee13:::134:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:0131818d:::135:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:26ba5a85:::136:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:381b8a5a:::137:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:28797e47:::138:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:bfca7f22:::139:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:36807075:::140:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:80b03975:::141:head 2024-09-10T18:58:28.632 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:5c15709b:::142:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f39ea15e:::143:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ea992956:::144:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:48887b1c:::145:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:9f24a9dd:::146:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:987f100b:::147:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d2dd3581:::148:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7fed1808:::149:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c80b70e9:::150:head 2024-09-10T18:58:28.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:85ed90f9:::151:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:36428b24:::152:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d044c34a:::153:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7c18bf58:::154:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d1c21232:::155:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a7a3c575:::156:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:87da0633:::157:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d5ac3822:::158:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3f20522d:::159:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6ca26563:::160:head 2024-09-10T18:58:28.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:532ce135:::161:head 2024-09-10T18:58:28.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:c78863e6:::162:head 2024-09-10T18:58:28.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:28.740 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:28.751 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:28.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:28.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:28.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:28.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:29.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:29.157 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:29.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:29.294 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:27.856836+0000 mgr.y (mgr.24526) 374 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: cluster 2024-09-10T18:58:28.039968+0000 mgr.y (mgr.24526) 375 : cluster [DBG] pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.102953+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.103083+0000 mon.a (mon.0) 2260 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T18:58:29.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: cluster 2024-09-10T18:58:28.103706+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.137237+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.139128+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.139658+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.615447+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.936097+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.938151+0000 mon.b (mon.1) 282 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.940087+0000 mon.a (mon.0) 2265 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: audit 2024-09-10T18:58:28.941966+0000 mon.a (mon.0) 2266 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:29 smithi165 bash[22591]: cluster 2024-09-10T18:58:29.070701+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:29.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.428 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:29.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:27.856836+0000 mgr.y (mgr.24526) 374 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:29.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: cluster 2024-09-10T18:58:28.039968+0000 mgr.y (mgr.24526) 375 : cluster [DBG] pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:29.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.102953+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:29.550 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.103083+0000 mon.a (mon.0) 2260 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: cluster 2024-09-10T18:58:28.103706+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.137237+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.139128+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.139658+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.615447+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.936097+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.938151+0000 mon.b (mon.1) 282 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.940087+0000 mon.a (mon.0) 2265 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: audit 2024-09-10T18:58:28.941966+0000 mon.a (mon.0) 2266 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[32381]: cluster 2024-09-10T18:58:29.070701+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:27.856836+0000 mgr.y (mgr.24526) 374 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: cluster 2024-09-10T18:58:28.039968+0000 mgr.y (mgr.24526) 375 : cluster [DBG] pgmap v511: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.102953+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? v1:172.21.15.83:0/1833223569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi083-77648-57"}]': finished 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.103083+0000 mon.a (mon.0) 2260 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: cluster 2024-09-10T18:58:28.103706+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.137237+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.139128+0000 mon.a (mon.0) 2263 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.139658+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.615447+0000 mon.b (mon.1) 280 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi083-77828-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.936097+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.938151+0000 mon.b (mon.1) 282 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.552 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.940087+0000 mon.a (mon.0) 2265 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:29.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: audit 2024-09-10T18:58:28.941966+0000 mon.a (mon.0) 2266 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]: dispatch 2024-09-10T18:58:29.553 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:29 smithi083 bash[18629]: cluster 2024-09-10T18:58:29.070701+0000 mon.a (mon.0) 2267 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:29.565 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:29.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.701 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:29.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.840 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:29.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.977 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:29.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:30.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:30.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:30.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:30.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:30.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:30.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:30.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:30.509 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:30.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:30.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:30.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: audit 2024-09-10T18:58:29.138780+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: audit 2024-09-10T18:58:29.138942+0000 mon.a (mon.0) 2269 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]': finished 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: cluster 2024-09-10T18:58:29.139041+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: audit 2024-09-10T18:58:29.140671+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: audit 2024-09-10T18:58:29.141399+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:30.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:30 smithi165 bash[22591]: audit 2024-09-10T18:58:29.562609+0000 mon.c (mon.2) 584 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:30.782 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/3753198288 >> v1:172.21.15.83:6790/0 conn(0x7f2ac40fe1b0 legacy=0x7f2ac4107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:30.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/3753198288 shutdown_connections 2024-09-10T18:58:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/3753198288 >> 172.21.15.83:0/3753198288 conn(0x7f2ac4075290 msgr2=0x7f2ac40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/3753198288 shutdown_connections 2024-09-10T18:58:30.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/3753198288 wait complete. 2024-09-10T18:58:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:30.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 librados: starting msgr at 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 librados: starting objecter 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 Processor -- start 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- start start 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 librados: setting wanted keys 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 librados: calling monclient init 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac411c730 con 0x7f2ac4108f40 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac411d8d0 con 0x7f2ac41131e0 2024-09-10T18:58:30.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2ac9e3f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f2ac40fd610 0x7f2ac41128e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49972/0 (socket says 172.21.15.83:49972) 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2ac9e3f640 1 -- 172.21.15.83:0/1930806301 learned_addr learned my addr 172.21.15.83:0/1930806301 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2aba7fc640 10 client.?.objecter ms_handle_connect 0x7f2ac4108f40 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2aba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2aba7fc640 10 client.?.objecter ms_handle_connect 0x7f2ac41131e0 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2aba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.778+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac411da60 con 0x7f2ac40fd610 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 10 client.?.objecter ms_handle_connect 0x7f2ac40fd610 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805978039 0 0) 0x7f2ac411d8d0 con 0x7f2ac41131e0 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0003720 con 0x7f2ac41131e0 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041175349 0 0) 0x7f2ac411da60 con 0x7f2ac40fd610 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac411d8d0 con 0x7f2ac40fd610 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67118636 0 0) 0x7f2ab0003720 con 0x7f2ac41131e0 2024-09-10T18:58:30.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac411da60 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ac0001770 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327715255 0 0) 0x7f2ac411d8d0 con 0x7f2ac40fd610 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ab0003720 con 0x7f2ac40fd610 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2abc002d50 con 0x7f2ac40fd610 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1778423481 0 0) 0x7f2ac411da60 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 >> v1:172.21.15.83:6790/0 conn(0x7f2ac40fd610 legacy=0x7f2ac41128e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 >> v1:172.21.15.83:6789/0 conn(0x7f2ac4108f40 legacy=0x7f2ac4117720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac411dc30 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac411dee0 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2acc0a3640 10 client.49765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac411e330 con 0x7f2ac41131e0 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2acc0a3640 1 librados: init done 2024-09-10T18:58:30.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2ac0001df0 con 0x7f2ac41131e0 2024-09-10T18:58:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ac00030d0 con 0x7f2ac41131e0 2024-09-10T18:58:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa8005010 con 0x7f2ac41131e0 2024-09-10T18:58:30.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.782+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2ac0001990 con 0x7f2ac41131e0 2024-09-10T18:58:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: audit 2024-09-10T18:58:29.138780+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: audit 2024-09-10T18:58:29.138942+0000 mon.a (mon.0) 2269 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]': finished 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: cluster 2024-09-10T18:58:29.139041+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: audit 2024-09-10T18:58:29.140671+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: audit 2024-09-10T18:58:29.141399+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[32381]: audit 2024-09-10T18:58:29.562609+0000 mon.c (mon.2) 584 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: audit 2024-09-10T18:58:29.138780+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi083-77648-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: audit 2024-09-10T18:58:29.138942+0000 mon.a (mon.0) 2269 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-43"}]': finished 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: cluster 2024-09-10T18:58:29.139041+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: audit 2024-09-10T18:58:29.140671+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: audit 2024-09-10T18:58:29.141399+0000 mon.a (mon.0) 2272 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:30.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:30 smithi083 bash[18629]: audit 2024-09-10T18:58:29.562609+0000 mon.c (mon.2) 584 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(354..354 src has 69..354) v4 ==== 8073+0+0 (unknown 3710825790 0 0) 0x7f2ac0087b70 con 0x7f2ac41131e0 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 10 client.49765.objecter ms_dispatch 0x7f2ac4000b90 osd_map(354..354 src has 69..354) v4 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 3 client.49765.objecter handle_osd_map got epochs [354,354] > 0 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 3 client.49765.objecter handle_osd_map decoding full epoch 354 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 20 client.49765.objecter dump_active .. 0 homeless 2024-09-10T18:58:30.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.786+0000 7f2aba7fc640 10 client.49765.objecter ms_handle_connect 0x7f2ab0071610 2024-09-10T18:58:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:30.790+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2ac0053c60 con 0x7f2ac41131e0 2024-09-10T18:58:30.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:30.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:30.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:31.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.002+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f2aa80052c0 con 0x7f2ac41131e0 2024-09-10T18:58:31.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.041 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- 172.21.15.83:0/1560069294 >> v1:172.21.15.83:6790/0 conn(0x7f88581042a0 legacy=0x7f8858106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- 172.21.15.83:0/1560069294 shutdown_connections 2024-09-10T18:58:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- 172.21.15.83:0/1560069294 >> 172.21.15.83:0/1560069294 conn(0x7f8858075290 msgr2=0x7f88580f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- 172.21.15.83:0/1560069294 shutdown_connections 2024-09-10T18:58:31.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- 172.21.15.83:0/1560069294 wait complete. 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 librados: starting msgr at 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 librados: starting objecter 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 Processor -- start 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- start start 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 librados: setting wanted keys 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 librados: calling monclient init 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f885810a250 con 0x7f8858107e30 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f885811f6c0 con 0x7f88581014e0 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885fdff640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8858120860 con 0x7f88581042a0 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885d39a640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f88581042a0 0x7f8858119860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49982/0 (socket says 172.21.15.83:49982) 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f885d39a640 1 -- 172.21.15.83:0/3830773023 learned_addr learned my addr 172.21.15.83:0/3830773023 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter ms_handle_connect 0x7f88581042a0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter ms_handle_connect 0x7f8858107e30 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285972872 0 0) 0x7f8858120860 con 0x7f88581042a0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88440034e0 con 0x7f88581042a0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter ms_handle_connect 0x7f88581014e0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604923889 0 0) 0x7f885810a250 con 0x7f8858107e30 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8858120860 con 0x7f8858107e30 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871090753 0 0) 0x7f885811f6c0 con 0x7f88581014e0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.066+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f885810a250 con 0x7f88581014e0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3434330812 0 0) 0x7f88440034e0 con 0x7f88581042a0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f885811f6c0 con 0x7f88581042a0 2024-09-10T18:58:31.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8848002a70 con 0x7f88581042a0 2024-09-10T18:58:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2553903120 0 0) 0x7f885810a250 con 0x7f88581014e0 2024-09-10T18:58:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88440034e0 con 0x7f88581014e0 2024-09-10T18:58:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f88540018c0 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651593632 0 0) 0x7f8858120860 con 0x7f8858107e30 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f885810a250 con 0x7f8858107e30 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8850001bc0 con 0x7f8858107e30 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 446953121 0 0) 0x7f88440034e0 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 >> v1:172.21.15.83:6790/0 conn(0x7f88581042a0 legacy=0x7f8858119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 >> v1:172.21.15.83:6789/0 conn(0x7f8858107e30 legacy=0x7f885811cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8858121a00 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885811e710 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f885fdff640 10 client.49771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f885811eb60 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f885fdff640 1 librados: init done 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8854001f80 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8854003150 con 0x7f88581014e0 2024-09-10T18:58:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c005010 con 0x7f88581014e0 2024-09-10T18:58:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f8854001b20 con 0x7f88581014e0 2024-09-10T18:58:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(354..354 src has 69..354) v4 ==== 8073+0+0 (unknown 3710825790 0 0) 0x7f8854087c10 con 0x7f88581014e0 2024-09-10T18:58:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 10 client.49771.objecter ms_dispatch 0x7f8858000b90 osd_map(354..354 src has 69..354) v4 2024-09-10T18:58:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 3 client.49771.objecter handle_osd_map got epochs [354,354] > 0 2024-09-10T18:58:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.070+0000 7f884e7fc640 3 client.49771.objecter handle_osd_map decoding full epoch 354 2024-09-10T18:58:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.074+0000 7f884e7fc640 20 client.49771.objecter dump_active .. 0 homeless 2024-09-10T18:58:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.074+0000 7f884e7fc640 10 client.49771.objecter ms_handle_connect 0x7f8844071760 2024-09-10T18:58:31.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.082+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8854054180 con 0x7f88581014e0 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- 172.21.15.83:0/945880654 >> v1:172.21.15.83:6789/0 conn(0x7f2fc81044e0 legacy=0x7f2fc81068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- 172.21.15.83:0/945880654 shutdown_connections 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- 172.21.15.83:0/945880654 >> 172.21.15.83:0/945880654 conn(0x7f2fc8074a40 msgr2=0x7f2fc80f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- 172.21.15.83:0/945880654 shutdown_connections 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- 172.21.15.83:0/945880654 wait complete. 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 librados: starting msgr at 2024-09-10T18:58:31.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 librados: starting objecter 2024-09-10T18:58:31.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 Processor -- start 2024-09-10T18:58:31.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- start start 2024-09-10T18:58:31.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 librados: setting wanted keys 2024-09-10T18:58:31.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 librados: calling monclient init 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc811e5b0 con 0x7f2fc8108070 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc811f7b0 con 0x7f2fc8101750 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fce911640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc81209b0 con 0x7f2fc81044e0 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fbf7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f2fc81044e0 0x7f2fc80fdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49998/0 (socket says 172.21.15.83:49998) 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fbf7fe640 1 -- 172.21.15.83:0/3844182165 learned_addr learned my addr 172.21.15.83:0/3844182165 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter ms_handle_connect 0x7f2fc8108070 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter ms_handle_connect 0x7f2fc81044e0 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter ms_handle_connect 0x7f2fc8101750 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:31.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362042319 0 0) 0x7f2fc811e5b0 con 0x7f2fc8108070 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4003720 con 0x7f2fc8108070 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912193599 0 0) 0x7f2fc811f7b0 con 0x7f2fc8101750 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc811e5b0 con 0x7f2fc8101750 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757045679 0 0) 0x7f2fc81209b0 con 0x7f2fc81044e0 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc811f7b0 con 0x7f2fc81044e0 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3797800083 0 0) 0x7f2fb4003720 con 0x7f2fc8108070 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc81209b0 con 0x7f2fc8108070 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.090+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2fc0001b80 con 0x7f2fc8108070 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 451522598 0 0) 0x7f2fc811e5b0 con 0x7f2fc8101750 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb4003720 con 0x7f2fc8101750 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2fb8001990 con 0x7f2fc8101750 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1174722292 0 0) 0x7f2fc811f7b0 con 0x7f2fc81044e0 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc811e5b0 con 0x7f2fc81044e0 2024-09-10T18:58:31.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2715706326 0 0) 0x7f2fc81209b0 con 0x7f2fc8108070 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6790/0 conn(0x7f2fc81044e0 legacy=0x7f2fc80fdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.165:6789/0 conn(0x7f2fc8101750 legacy=0x7f2fc80fd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8121bb0 con 0x7f2fc8108070 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8120ba0 con 0x7f2fc8108070 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fce911640 10 client.25194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc8121070 con 0x7f2fc8108070 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fce911640 1 librados: init done 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2fc0002250 con 0x7f2fc8108070 2024-09-10T18:58:31.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2fc0003540 con 0x7f2fc8108070 2024-09-10T18:58:31.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2fa8004ff0 con 0x7f2fc8108070 2024-09-10T18:58:31.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2fc00199e0 con 0x7f2fc8108070 2024-09-10T18:58:31.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(354..354 src has 69..354) v4 ==== 8073+0+0 (unknown 3710825790 0 0) 0x7f2fc00880a0 con 0x7f2fc8108070 2024-09-10T18:58:31.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 osd_map(354..354 src has 69..354) v4 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 3 client.25194.objecter handle_osd_map got epochs [354,354] > 0 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.094+0000 7f2fc57fa640 3 client.25194.objecter handle_osd_map decoding full epoch 354 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 20 client.25194.objecter dump_active .. 0 homeless 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v354) v1 ==== 54+0+16 (unknown 1304699541 0 2420000192) 0x7f2fc008a4c0 con 0x7f2fc8108070 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fb4071930 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 10 client.25194.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 1 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 1 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 0 1 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 4 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 10 client.25194.objecter _send_command 1 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa8004a60 con 0x7f2fa8004610 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8004610 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa8004a60 con 0x7f2fa8004610 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 1 = system:0 2024-09-10T18:58:31.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 4 2024-09-10T18:58:31.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.098+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 0 1 2024-09-10T18:58:31.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 10 client.25194.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 2 2024-09-10T18:58:31.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 2 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8005290 osd=0 3 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 0 2 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 4 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 10 client.25194.objecter _send_command 2 2024-09-10T18:58:31.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.114+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa8004e40 con 0x7f2fa8004610 2024-09-10T18:58:31.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa8004e40 con 0x7f2fa8004610 2024-09-10T18:58:31.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 4 2024-09-10T18:58:31.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 0 2 2024-09-10T18:58:31.124 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:58:31.124 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:58:31.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 10 client.25194.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 1 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 4 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 10 client.25194.objecter _send_command 3 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa80022f0 con 0x7f2fa8004110 2024-09-10T18:58:31.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8004110 2024-09-10T18:58:31.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa80022f0 con 0x7f2fa8004110 2024-09-10T18:58:31.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:58:31.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 3 = system:0 2024-09-10T18:58:31.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 4 2024-09-10T18:58:31.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.118+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 1 3 2024-09-10T18:58:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 10 client.25194.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 4 2024-09-10T18:58:31.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 4 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8003fa0 osd=1 3 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 1 4 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 4 2024-09-10T18:58:31.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 10 client.25194.objecter _send_command 4 2024-09-10T18:58:31.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.134+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa80018f0 con 0x7f2fa8004110 2024-09-10T18:58:31.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:31.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.146+0000 7f2aba7fc640 1 -- 172.21.15.83:0/1930806301 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v355) v1 ==== 91+0+0 (unknown 4011939362 0 0) 0x7f2ac0053a80 con 0x7f2ac41131e0 2024-09-10T18:58:31.153 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T18:58:31.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa80018f0 con 0x7f2fa8004110 2024-09-10T18:58:31.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 4 2024-09-10T18:58:31.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 1 4 2024-09-10T18:58:31.161 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:58:31.161 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 10 librados: watch_flush enter 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 10 librados: watch_flush exit 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 20 client.49765.objecter shutdown clearing up homeless session... 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 10 client.49765.objecter successfully canceled tick 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2ab0071610 legacy=0x7f2ab0073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.150+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 >> v1:172.21.15.165:6789/0 conn(0x7f2ac41131e0 legacy=0x7f2ac411ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.158+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 shutdown_connections 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.158+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 >> 172.21.15.83:0/1930806301 conn(0x7f2ac4075290 msgr2=0x7f2ac40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.158+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 shutdown_connections 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.158+0000 7f2acc0a3640 1 -- 172.21.15.83:0/1930806301 wait complete. 2024-09-10T18:58:31.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.158+0000 7f2acc0a3640 1 librados: shutdown 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 10 client.25194.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 5 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 5 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 2 5 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 4 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 10 client.25194.objecter _send_command 5 2024-09-10T18:58:31.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa8003e30 con 0x7f2fa800d350 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa800d350 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa8003e30 con 0x7f2fa800d350 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 5 = system:0 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 4 2024-09-10T18:58:31.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.162+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 2 5 2024-09-10T18:58:31.169 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:31.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 10 client.25194.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 6 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 6 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa80035e0 osd=2 3 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 2 6 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 4 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 10 client.25194.objecter _send_command 6 2024-09-10T18:58:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa8003060 con 0x7f2fa800d350 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa8003060 con 0x7f2fa800d350 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 4 2024-09-10T18:58:31.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 2 6 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 10 client.25194.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 7 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 7 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 3 7 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 4 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 10 client.25194.objecter _send_command 7 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa800d010 con 0x7f2fa8010e80 2024-09-10T18:58:31.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.182+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8010e80 2024-09-10T18:58:31.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.186+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa800d010 con 0x7f2fa8010e80 2024-09-10T18:58:31.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.186+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:58:31.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.186+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 7 = system:0 2024-09-10T18:58:31.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.186+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 4 2024-09-10T18:58:31.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.186+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 3 7 2024-09-10T18:58:31.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 10 client.25194.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 8 2024-09-10T18:58:31.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 8 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa800c990 osd=3 3 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 3 8 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 4 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 10 client.25194.objecter _send_command 8 2024-09-10T18:58:31.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.198+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa80132e0 con 0x7f2fa8010e80 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa80132e0 con 0x7f2fa8010e80 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 4 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 3 8 2024-09-10T18:58:31.207 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 10 client.25194.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 9 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 9 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 4 9 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 4 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 10 client.25194.objecter _send_command 9 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa801a290 con 0x7f2fa8016790 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8016790 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa801a290 con 0x7f2fa8016790 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:58:31.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 9 = system:0 2024-09-10T18:58:31.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 4 2024-09-10T18:58:31.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.202+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 4 9 2024-09-10T18:58:31.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 10 client.25194.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 10 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 10 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8012b20 osd=4 3 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 4 10 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 4 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 10 client.25194.objecter _send_command 10 2024-09-10T18:58:31.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.230+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa800c410 con 0x7f2fa8016790 2024-09-10T18:58:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa800c410 con 0x7f2fa8016790 2024-09-10T18:58:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 4 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 4 10 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 10 client.25194.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 11 2024-09-10T18:58:31.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 11 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 5 11 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 4 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 10 client.25194.objecter _send_command 11 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa800c340 con 0x7f2fa8012670 2024-09-10T18:58:31.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.234+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8012670 2024-09-10T18:58:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.238+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa800c340 con 0x7f2fa8012670 2024-09-10T18:58:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.238+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:58:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.238+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 11 = system:0 2024-09-10T18:58:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.238+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 4 2024-09-10T18:58:31.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.238+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 5 11 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 10 client.25194.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 12 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 12 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa800c1d0 osd=5 3 2024-09-10T18:58:31.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 5 12 2024-09-10T18:58:31.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 4 2024-09-10T18:58:31.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 10 client.25194.objecter _send_command 12 2024-09-10T18:58:31.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.258+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa801eb40 con 0x7f2fa8012670 2024-09-10T18:58:31.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa801eb40 con 0x7f2fa8012670 2024-09-10T18:58:31.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.267 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:58:31.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 4 2024-09-10T18:58:31.268 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:58:31.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 5 12 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 10 client.25194.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 13 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 13 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 6 13 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 4 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 10 client.25194.objecter _send_command 13 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa8022e80 con 0x7f2fa801f1c0 2024-09-10T18:58:31.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa801f1c0 2024-09-10T18:58:31.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa8022e80 con 0x7f2fa801f1c0 2024-09-10T18:58:31.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:58:31.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 13 = system:0 2024-09-10T18:58:31.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 4 2024-09-10T18:58:31.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.262+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 6 13 2024-09-10T18:58:31.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:31.296 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 10 client.25194.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 14 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 14 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa801eb40 osd=6 3 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 6 14 2024-09-10T18:58:31.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 4 2024-09-10T18:58:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 10 client.25194.objecter _send_command 14 2024-09-10T18:58:31.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.298+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa8024720 con 0x7f2fa801f1c0 2024-09-10T18:58:31.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa8024720 con 0x7f2fa801f1c0 2024-09-10T18:58:31.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 4 2024-09-10T18:58:31.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 6 14 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 10 client.25194.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 15 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 15 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 7 15 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 4 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 10 client.25194.objecter _send_command 15 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fa80252f0 con 0x7f2fa8024de0 2024-09-10T18:58:31.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.302+0000 7f2fc57fa640 10 client.25194.objecter ms_handle_connect 0x7f2fa8024de0 2024-09-10T18:58:31.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.306+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2fa80252f0 con 0x7f2fa8024de0 2024-09-10T18:58:31.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.306+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:58:31.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.306+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 15 = system:0 2024-09-10T18:58:31.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.306+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 4 2024-09-10T18:58:31.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.306+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 7 15 2024-09-10T18:58:31.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 10 client.25194.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:58:31.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign -1 16 2024-09-10T18:58:31.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 20 client.25194.objecter _get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 15 client.25194.objecter _session_command_op_remove -1 16 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 20 client.25194.objecter get_session s=0x7f2fa8024720 osd=7 3 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 15 client.25194.objecter _session_command_op_assign 7 16 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 4 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 10 client.25194.objecter _send_command 16 2024-09-10T18:58:31.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.326+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2fa80252f0 con 0x7f2fa8024de0 2024-09-10T18:58:31.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fc57fa640 1 -- 172.21.15.83:0/3844182165 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2fa80252f0 con 0x7f2fa8024de0 2024-09-10T18:58:31.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fc57fa640 10 client.25194.objecter ms_dispatch 0x7f2fc8000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:31.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fc57fa640 10 client.25194.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:31.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fc57fa640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 4 2024-09-10T18:58:31.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fc57fa640 15 client.25194.objecter _session_command_op_remove 7 16 2024-09-10T18:58:31.338 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 10 librados: watch_flush enter 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 10 librados: watch_flush exit 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 10 client.25194.objecter close_session for osd.0 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6801/2535527483 conn(0x7f2fa8004610 legacy=0x7f2fa8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8005290 osd=0 1 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 10 client.25194.objecter close_session for osd.1 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.330+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6805/2309624637 conn(0x7f2fa8004110 legacy=0x7f2fa800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8003fa0 osd=1 1 2024-09-10T18:58:31.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.2 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6809/386094220 conn(0x7f2fa800d350 legacy=0x7f2fa800f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa80035e0 osd=2 1 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.3 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fcceae640 1 -- 172.21.15.83:0/3844182165 reap_dead start 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6813/1279688021 conn(0x7f2fa8010e80 legacy=0x7f2fa8011290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c990 osd=3 1 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.4 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.165:6800/3225515265 conn(0x7f2fa8016790 legacy=0x7f2fa8018b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8012b20 osd=4 1 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.5 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.165:6804/571173777 conn(0x7f2fa8012670 legacy=0x7f2fa801d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa800c1d0 osd=5 1 2024-09-10T18:58:31.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.6 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.165:6808/1317922936 conn(0x7f2fa801f1c0 legacy=0x7f2fa8021600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa801eb40 osd=6 1 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter close_session for osd.7 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.165:6812/1610151650 conn(0x7f2fa8024de0 legacy=0x7f2fa8022e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter put_session s=0x7f2fa8024720 osd=7 1 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 20 client.25194.objecter shutdown clearing up homeless session... 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 10 client.25194.objecter successfully canceled tick 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2fb4071930 legacy=0x7f2fb4073de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> v1:172.21.15.83:6789/0 conn(0x7f2fc8108070 legacy=0x7f2fc811ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fcceae640 1 -- 172.21.15.83:0/3844182165 reap_dead start 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fcceae640 1 -- 172.21.15.83:0/3844182165 reap_dead start 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 shutdown_connections 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 >> 172.21.15.83:0/3844182165 conn(0x7f2fc8074a40 msgr2=0x7f2fc80f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:31.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 shutdown_connections 2024-09-10T18:58:31.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 -- 172.21.15.83:0/3844182165 wait complete. 2024-09-10T18:58:31.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.334+0000 7f2fce911640 1 librados: shutdown 2024-09-10T18:58:31.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f883c005840 con 0x7f88581014e0 2024-09-10T18:58:31.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f884e7fc640 1 -- 172.21.15.83:0/3830773023 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2392 (unknown 3226119948 0 84919154) 0x7f8854053b20 con 0x7f88581014e0 2024-09-10T18:58:31.371 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:58:31.371 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1045,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":355,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994114,"num_in_osds":8,"osd_in_since":1725994087,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":228}],"num_pgs":228,"num_pools":9,"num_objects":321,"data_bytes":8791717,"bytes_used":2570104832,"bytes_avail":765390180352,"bytes_total":767960285184},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-10T18:56:01.193087+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":13,"start_stamp":"2024-09-10T18:49:25.207072+0000","gid":14493,"addr":"172.21.15.83:0/703687015","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)","ceph_version_short":"17.2.7-1606-g5be57395","container_hostname":"smithi083","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi083","id":"foo.a","kernel_description":"#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024","kernel_version":"5.4.0-195-generic","mem_swap_kb":"2097148","mem_total_kb":"32754944","num_handles":"1","os":"Linux","pid":"6","realm_id":"","realm_name":"","zone_id":"ef939013-895d-4fcb-a130-e9a8938d4e31","zone_name":"default","zonegroup_id":"348ee464-2af6-469c-a2f0-1b47a5a7c71b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T18:58:31.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 10 librados: watch_flush enter 2024-09-10T18:58:31.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 10 librados: watch_flush exit 2024-09-10T18:58:31.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 20 client.49771.objecter shutdown clearing up homeless session... 2024-09-10T18:58:31.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 10 client.49771.objecter successfully canceled tick 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8844071760 legacy=0x7f8844073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 >> v1:172.21.15.165:6789/0 conn(0x7f88581014e0 legacy=0x7f88581005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 shutdown_connections 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.366+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 >> 172.21.15.83:0/3830773023 conn(0x7f8858075290 msgr2=0x7f8858064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.370+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 shutdown_connections 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.370+0000 7f885fdff640 1 -- 172.21.15.83:0/3830773023 wait complete. 2024-09-10T18:58:31.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:31.370+0000 7f885fdff640 1 librados: shutdown 2024-09-10T18:58:31.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.427 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:31.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:31.562 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:31.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:31.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: cluster 2024-09-10T18:58:30.040804+0000 mgr.y (mgr.24526) 376 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:30.584153+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: cluster 2024-09-10T18:58:30.584246+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:30.585157+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.006472+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.83:0/1930806301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.065744+0000 mon.a (mon.0) 2276 : audit [INF] from='client.49765 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.101533+0000 mon.a (mon.0) 2277 : audit [DBG] from='client.? v1:172.21.15.83:0/3844182165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: cluster 2024-09-10T18:58:31.137195+0000 mon.a (mon.0) 2278 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.151890+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.151974+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.152062+0000 mon.a (mon.0) 2281 : audit [INF] from='client.49765 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: cluster 2024-09-10T18:58:31.152138+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.154847+0000 mon.b (mon.1) 284 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[22591]: audit 2024-09-10T18:58:31.160364+0000 mon.a (mon.0) 2283 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:31.816 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:31.834 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:31.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:31.968 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:31.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: cluster 2024-09-10T18:58:30.040804+0000 mgr.y (mgr.24526) 376 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:30.584153+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: cluster 2024-09-10T18:58:30.584246+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-10T18:58:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:30.585157+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.006472+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.83:0/1930806301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.065744+0000 mon.a (mon.0) 2276 : audit [INF] from='client.49765 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.101533+0000 mon.a (mon.0) 2277 : audit [DBG] from='client.? v1:172.21.15.83:0/3844182165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: cluster 2024-09-10T18:58:31.137195+0000 mon.a (mon.0) 2278 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.151890+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.151974+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.152062+0000 mon.a (mon.0) 2281 : audit [INF] from='client.49765 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: cluster 2024-09-10T18:58:31.152138+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.154847+0000 mon.b (mon.1) 284 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[18629]: audit 2024-09-10T18:58:31.160364+0000 mon.a (mon.0) 2283 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: cluster 2024-09-10T18:58:30.040804+0000 mgr.y (mgr.24526) 376 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:30.584153+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: cluster 2024-09-10T18:58:30.584246+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:30.585157+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]: dispatch 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.006472+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.83:0/1930806301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.065744+0000 mon.a (mon.0) 2276 : audit [INF] from='client.49765 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.101533+0000 mon.a (mon.0) 2277 : audit [DBG] from='client.? v1:172.21.15.83:0/3844182165' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: cluster 2024-09-10T18:58:31.137195+0000 mon.a (mon.0) 2278 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.151890+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi083-77648-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.151974+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? v1:172.21.15.83:0/3151444780' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi083-77680-2"}]': finished 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.152062+0000 mon.a (mon.0) 2281 : audit [INF] from='client.49765 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: cluster 2024-09-10T18:58:31.152138+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e355: 8 total, 8 up, 8 in 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.154847+0000 mon.b (mon.1) 284 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:31 smithi083 bash[32381]: audit 2024-09-10T18:58:31.160364+0000 mon.a (mon.0) 2283 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:32.105 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:32.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:32.235 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:32.288 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T18:58:32.288 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:58:32.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.370 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:32.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:32.508 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:32.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:32.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:31.368054+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? v1:172.21.15.83:0/3830773023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:58:32.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:31.391677+0000 osd.5 (osd.5) 189 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:58:32.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:31.395062+0000 osd.5 (osd.5) 190 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.041650+0000 mgr.y (mgr.24526) 377 : cluster [DBG] pgmap v517: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:32.151042+0000 mon.a (mon.0) 2284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.151334+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:32.152793+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:32.155174+0000 mon.c (mon.2) 585 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:32.155742+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: audit 2024-09-10T18:58:32.157710+0000 mon.a (mon.0) 2287 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.225530+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:58:32.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.229881+0000 osd.3 (osd.3) 221 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:31.368054+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? v1:172.21.15.83:0/3830773023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:31.391677+0000 osd.5 (osd.5) 189 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:31.395062+0000 osd.5 (osd.5) 190 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.041650+0000 mgr.y (mgr.24526) 377 : cluster [DBG] pgmap v517: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:32.151042+0000 mon.a (mon.0) 2284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.151334+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:32.152793+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:32.155174+0000 mon.c (mon.2) 585 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:32.155742+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: audit 2024-09-10T18:58:32.157710+0000 mon.a (mon.0) 2287 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.225530+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:58:32.707 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.229881+0000 osd.3 (osd.3) 221 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:58:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:31.368054+0000 mon.b (mon.1) 285 : audit [DBG] from='client.? v1:172.21.15.83:0/3830773023' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:58:32.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:31.391677+0000 osd.5 (osd.5) 189 : cluster [DBG] 2.18 scrub starts 2024-09-10T18:58:32.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.775 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:32.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:31.395062+0000 osd.5 (osd.5) 190 : cluster [DBG] 2.18 scrub ok 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.041650+0000 mgr.y (mgr.24526) 377 : cluster [DBG] pgmap v517: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:32.151042+0000 mon.a (mon.0) 2284 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.151334+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e356: 8 total, 8 up, 8 in 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:32.152793+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:32.155174+0000 mon.c (mon.2) 585 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:32.155742+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: audit 2024-09-10T18:58:32.157710+0000 mon.a (mon.0) 2287 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.225530+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.17 scrub starts 2024-09-10T18:58:32.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:32 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.229881+0000 osd.3 (osd.3) 221 : cluster [DBG] 4.17 scrub ok 2024-09-10T18:58:32.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:32.890 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:32] "GET /metrics HTTP/1.1" 200 228718 "" "Prometheus/2.43.0" 2024-09-10T18:58:32.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:32.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:33.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:33.046 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:33.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T18:58:33.150 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:33.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:33.321 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:33.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.456 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:33.559 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:33.591 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:33.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:33.730 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.202487+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.226760+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.364225+0000 osd.5 (osd.5) 191 : cluster [DBG] 64.1 scrub starts 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.368592+0000 osd.5 (osd.5) 192 : cluster [DBG] 64.1 scrub ok 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.929164+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:32.931692+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.156238+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.156480+0000 mon.a (mon.0) 2289 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:33.156754+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.160066+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.161284+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.164083+0000 mon.a (mon.0) 2292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.184758+0000 mon.c (mon.2) 586 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: audit 2024-09-10T18:58:33.185197+0000 mon.a (mon.0) 2293 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:33.222855+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[18629]: cluster 2024-09-10T18:58:33.225295+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.202487+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.226760+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:58:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.364225+0000 osd.5 (osd.5) 191 : cluster [DBG] 64.1 scrub starts 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.368592+0000 osd.5 (osd.5) 192 : cluster [DBG] 64.1 scrub ok 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.929164+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:32.931692+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.156238+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.156480+0000 mon.a (mon.0) 2289 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:33.156754+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.160066+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.161284+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.164083+0000 mon.a (mon.0) 2292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.184758+0000 mon.c (mon.2) 586 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: audit 2024-09-10T18:58:33.185197+0000 mon.a (mon.0) 2293 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:33.222855+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:58:33.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:33 smithi083 bash[32381]: cluster 2024-09-10T18:58:33.225295+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:58:33.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.864 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:33.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:33.907 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T18:58:33.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:58:33.968 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:33.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:34.093 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7462ddf6:::16.ListObjectsStart 2024-09-10T18:58:34.093 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 1 0 2024-09-10T18:58:34.093 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 10 0 2024-09-10T18:58:34.093 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 13 0 2024-09-10T18:58:34.093 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 7 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 14 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 0 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 15 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 11 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 5 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 8 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 6 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 3 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 4 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 12 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 9 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2 0 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T18:58:34.094 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (90 ms) 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 3 tests from LibRadosListEC (1188 ms total) 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2934 ms) 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 1 test from LibRadosListNP (2934 ms total) 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] Global test environment tear-down 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [==========] 11 tests from 3 test suites ran. (248344 ms total) 2024-09-10T18:58:34.095 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ PASSED ] 11 tests. 2024-09-10T18:58:34.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.202487+0000 osd.7 (osd.7) 206 : cluster [DBG] 2.8 scrub starts 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.226760+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.8 scrub ok 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.364225+0000 osd.5 (osd.5) 191 : cluster [DBG] 64.1 scrub starts 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.368592+0000 osd.5 (osd.5) 192 : cluster [DBG] 64.1 scrub ok 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.929164+0000 osd.2 (osd.2) 141 : cluster [DBG] 4.1c scrub starts 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:32.931692+0000 osd.2 (osd.2) 142 : cluster [DBG] 4.1c scrub ok 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.156238+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77680-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.156480+0000 mon.a (mon.0) 2289 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:33.156754+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e357: 8 total, 8 up, 8 in 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.160066+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.161284+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.164083+0000 mon.a (mon.0) 2292 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.184758+0000 mon.c (mon.2) 586 : audit [INF] from='client.? v1:172.21.15.83:0/1282296492' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: audit 2024-09-10T18:58:33.185197+0000 mon.a (mon.0) 2293 : audit [INF] from='client.49271 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:33.222855+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.1e scrub starts 2024-09-10T18:58:34.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:33 smithi165 bash[22591]: cluster 2024-09-10T18:58:33.225295+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.1e scrub ok 2024-09-10T18:58:34.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:34.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:34.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:34.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:34.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.531 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:34.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.660 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:34.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:34.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.042648+0000 mgr.y (mgr.24526) 378 : cluster [DBG] pgmap v520: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:34.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.073243+0000 mon.a (mon.0) 2294 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:34.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.082703+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:34.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.082944+0000 mon.a (mon.0) 2296 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.083105+0000 mon.a (mon.0) 2297 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.083327+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.085376+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.087937+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: audit 2024-09-10T18:58:34.089564+0000 mon.a (mon.0) 2300 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.261422+0000 osd.3 (osd.3) 224 : cluster [DBG] 64.3 scrub starts 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.265452+0000 osd.3 (osd.3) 225 : cluster [DBG] 64.3 scrub ok 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.042648+0000 mgr.y (mgr.24526) 378 : cluster [DBG] pgmap v520: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.073243+0000 mon.a (mon.0) 2294 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.082703+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.082944+0000 mon.a (mon.0) 2296 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.083105+0000 mon.a (mon.0) 2297 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.083327+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.085376+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.087937+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: audit 2024-09-10T18:58:34.089564+0000 mon.a (mon.0) 2300 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.261422+0000 osd.3 (osd.3) 224 : cluster [DBG] 64.3 scrub starts 2024-09-10T18:58:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:34 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.265452+0000 osd.3 (osd.3) 225 : cluster [DBG] 64.3 scrub ok 2024-09-10T18:58:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:34.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:58:34.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.935 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:34.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:35.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:35.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.042648+0000 mgr.y (mgr.24526) 378 : cluster [DBG] pgmap v520: 260 pgs: 64 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:35.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.073243+0000 mon.a (mon.0) 2294 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.082703+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.082944+0000 mon.a (mon.0) 2296 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.083105+0000 mon.a (mon.0) 2297 : audit [INF] from='client.49271 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi083-77680-3","pool2":"test-rados-api-smithi083-77680-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.083327+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e358: 8 total, 8 up, 8 in 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.085376+0000 mon.b (mon.1) 288 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.087937+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]: dispatch 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: audit 2024-09-10T18:58:34.089564+0000 mon.a (mon.0) 2300 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.261422+0000 osd.3 (osd.3) 224 : cluster [DBG] 64.3 scrub starts 2024-09-10T18:58:35.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:34 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.265452+0000 osd.3 (osd.3) 225 : cluster [DBG] 64.3 scrub ok 2024-09-10T18:58:35.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:35.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:35.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:35.337 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:35.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:35.471 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:35.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:35.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:35.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:35.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:35.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:35.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:35.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:58:35.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:35.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:35.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:36.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.629904+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.e scrub starts 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.635166+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.e scrub ok 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.808605+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.812213+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.966485+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.e scrub starts 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:34.970157+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.e scrub ok 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.083397+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.083519+0000 mon.a (mon.0) 2302 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T18:58:36.139 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: cluster 2024-09-10T18:58:35.083638+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.086649+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.090438+0000 mon.a (mon.0) 2304 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.114852+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.115600+0000 mon.a (mon.0) 2306 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[18629]: audit 2024-09-10T18:58:35.116021+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.629904+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.e scrub starts 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.635166+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.e scrub ok 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.808605+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.812213+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:58:36.140 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.966485+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.e scrub starts 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:34.970157+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.e scrub ok 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.083397+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.083519+0000 mon.a (mon.0) 2302 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: cluster 2024-09-10T18:58:35.083638+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.086649+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.090438+0000 mon.a (mon.0) 2304 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.114852+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.115600+0000 mon.a (mon.0) 2306 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:36 smithi083 bash[32381]: audit 2024-09-10T18:58:35.116021+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:36.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:36.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.629904+0000 osd.4 (osd.4) 235 : cluster [DBG] 6.e scrub starts 2024-09-10T18:58:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.635166+0000 osd.4 (osd.4) 236 : cluster [DBG] 6.e scrub ok 2024-09-10T18:58:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.808605+0000 osd.1 (osd.1) 209 : cluster [DBG] 4.2 scrub starts 2024-09-10T18:58:36.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.812213+0000 osd.1 (osd.1) 210 : cluster [DBG] 4.2 scrub ok 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.966485+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.e scrub starts 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:34.970157+0000 osd.2 (osd.2) 144 : cluster [DBG] 2.e scrub ok 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.083397+0000 mon.a (mon.0) 2301 : audit [INF] from='client.? v1:172.21.15.83:0/4209688971' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi083-77648-58"}]': finished 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.083519+0000 mon.a (mon.0) 2302 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: cluster 2024-09-10T18:58:35.083638+0000 mon.a (mon.0) 2303 : cluster [DBG] osdmap e359: 8 total, 8 up, 8 in 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.086649+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.090438+0000 mon.a (mon.0) 2304 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.114852+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.115600+0000 mon.a (mon.0) 2306 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:36.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:36 smithi165 bash[22591]: audit 2024-09-10T18:58:35.116021+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:36.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:36.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:36.569 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:36.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.708 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:36.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:36.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:36.985 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:36.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:37.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:37.115 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:37.218 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:37.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:37.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:37.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:35.835557+0000 osd.1 (osd.1) 211 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:58:37.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:35.839224+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:58:37.358 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.013492+0000 osd.2 (osd.2) 145 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.017828+0000 osd.2 (osd.2) 146 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.044317+0000 mgr.y (mgr.24526) 379 : cluster [DBG] pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: audit 2024-09-10T18:58:36.093680+0000 mon.a (mon.0) 2308 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: audit 2024-09-10T18:58:36.093758+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.093798+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: audit 2024-09-10T18:58:36.094243+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: audit 2024-09-10T18:58:36.094871+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:37.359 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[22591]: audit 2024-09-10T18:58:36.097752+0000 mon.a (mon.0) 2312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.370 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:37.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:37.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:37.507 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:37.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:35.835557+0000 osd.1 (osd.1) 211 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:35.839224+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.013492+0000 osd.2 (osd.2) 145 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.017828+0000 osd.2 (osd.2) 146 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.044317+0000 mgr.y (mgr.24526) 379 : cluster [DBG] pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: audit 2024-09-10T18:58:36.093680+0000 mon.a (mon.0) 2308 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: audit 2024-09-10T18:58:36.093758+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.093798+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: audit 2024-09-10T18:58:36.094243+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: audit 2024-09-10T18:58:36.094871+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[18629]: audit 2024-09-10T18:58:36.097752+0000 mon.a (mon.0) 2312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:35.835557+0000 osd.1 (osd.1) 211 : cluster [DBG] 2.4 scrub starts 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:35.839224+0000 osd.1 (osd.1) 212 : cluster [DBG] 2.4 scrub ok 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.013492+0000 osd.2 (osd.2) 145 : cluster [DBG] 64.0 scrub starts 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.017828+0000 osd.2 (osd.2) 146 : cluster [DBG] 64.0 scrub ok 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.044317+0000 mgr.y (mgr.24526) 379 : cluster [DBG] pgmap v523: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: audit 2024-09-10T18:58:36.093680+0000 mon.a (mon.0) 2308 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: audit 2024-09-10T18:58:36.093758+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.093798+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e360: 8 total, 8 up, 8 in 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: audit 2024-09-10T18:58:36.094243+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: audit 2024-09-10T18:58:36.094871+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:37 smithi083 bash[32381]: audit 2024-09-10T18:58:36.097752+0000 mon.a (mon.0) 2312 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T18:58:37.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:37.642 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:37.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:37.774 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:37.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:37.894 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:37.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:37.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:37.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:38.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:38.069 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:38.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:38.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.676485+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.d scrub starts 2024-09-10T18:58:38.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.681303+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.d scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.847876+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:36.851353+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.059392+0000 osd.2 (osd.2) 147 : cluster [DBG] 64.a scrub starts 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.063457+0000 osd.2 (osd.2) 148 : cluster [DBG] 64.a scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.302232+0000 osd.3 (osd.3) 226 : cluster [DBG] 64.10 scrub starts 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.311798+0000 osd.3 (osd.3) 227 : cluster [DBG] 64.10 scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.371619+0000 osd.5 (osd.5) 193 : cluster [DBG] 6.a scrub starts 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.374005+0000 osd.5 (osd.5) 194 : cluster [DBG] 6.a scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.446600+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.451558+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: audit 2024-09-10T18:58:37.535907+0000 mon.a (mon.0) 2313 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T18:58:38.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:38 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.536106+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-10T18:58:38.203 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:38.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:38.341 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:38.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.478 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.676485+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.d scrub starts 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.681303+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.d scrub ok 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.847876+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:36.851353+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.059392+0000 osd.2 (osd.2) 147 : cluster [DBG] 64.a scrub starts 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.063457+0000 osd.2 (osd.2) 148 : cluster [DBG] 64.a scrub ok 2024-09-10T18:58:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.302232+0000 osd.3 (osd.3) 226 : cluster [DBG] 64.10 scrub starts 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.311798+0000 osd.3 (osd.3) 227 : cluster [DBG] 64.10 scrub ok 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.371619+0000 osd.5 (osd.5) 193 : cluster [DBG] 6.a scrub starts 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.374005+0000 osd.5 (osd.5) 194 : cluster [DBG] 6.a scrub ok 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.446600+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.451558+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: audit 2024-09-10T18:58:37.535907+0000 mon.a (mon.0) 2313 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.536106+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.676485+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.d scrub starts 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.681303+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.d scrub ok 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.847876+0000 osd.1 (osd.1) 213 : cluster [DBG] 4.7 scrub starts 2024-09-10T18:58:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:36.851353+0000 osd.1 (osd.1) 214 : cluster [DBG] 4.7 scrub ok 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.059392+0000 osd.2 (osd.2) 147 : cluster [DBG] 64.a scrub starts 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.063457+0000 osd.2 (osd.2) 148 : cluster [DBG] 64.a scrub ok 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.302232+0000 osd.3 (osd.3) 226 : cluster [DBG] 64.10 scrub starts 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.311798+0000 osd.3 (osd.3) 227 : cluster [DBG] 64.10 scrub ok 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.371619+0000 osd.5 (osd.5) 193 : cluster [DBG] 6.a scrub starts 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.374005+0000 osd.5 (osd.5) 194 : cluster [DBG] 6.a scrub ok 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.446600+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.2 scrub starts 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.451558+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.2 scrub ok 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: audit 2024-09-10T18:58:37.535907+0000 mon.a (mon.0) 2313 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T18:58:38.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:38 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.536106+0000 mon.a (mon.0) 2314 : cluster [DBG] osdmap e361: 8 total, 8 up, 8 in 2024-09-10T18:58:38.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:38.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:38.731 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:38.731 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:38.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:38.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.890 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:38.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:38.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:39.030 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:39.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:39.163 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:39.181 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.679917+0000 osd.4 (osd.4) 239 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:58:39.181 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:37.683066+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: audit 2024-09-10T18:58:37.862099+0000 mgr.y (mgr.24526) 380 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.045119+0000 mgr.y (mgr.24526) 381 : cluster [DBG] pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.060381+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c scrub starts 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.064257+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c scrub ok 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: audit 2024-09-10T18:58:38.118488+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.118580+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.288548+0000 osd.3 (osd.3) 228 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.293656+0000 osd.3 (osd.3) 229 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.382624+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.383048+0000 osd.1 (osd.1) 216 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:39.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.429357+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.434328+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.075876+0000 mon.a (mon.0) 2317 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.679917+0000 osd.4 (osd.4) 239 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:37.683066+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: audit 2024-09-10T18:58:37.862099+0000 mgr.y (mgr.24526) 380 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.045119+0000 mgr.y (mgr.24526) 381 : cluster [DBG] pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.060381+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c scrub starts 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.064257+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c scrub ok 2024-09-10T18:58:39.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: audit 2024-09-10T18:58:38.118488+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.118580+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.288548+0000 osd.3 (osd.3) 228 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.293656+0000 osd.3 (osd.3) 229 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.382624+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.383048+0000 osd.1 (osd.1) 216 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.429357+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.434328+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:58:39.184 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:39 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.075876+0000 mon.a (mon.0) 2317 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:39.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:39.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.679917+0000 osd.4 (osd.4) 239 : cluster [DBG] 6.1a scrub starts 2024-09-10T18:58:39.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:37.683066+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.1a scrub ok 2024-09-10T18:58:39.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: audit 2024-09-10T18:58:37.862099+0000 mgr.y (mgr.24526) 380 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:39.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.045119+0000 mgr.y (mgr.24526) 381 : cluster [DBG] pgmap v526: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:39.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.060381+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.c scrub starts 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.064257+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.c scrub ok 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: audit 2024-09-10T18:58:38.118488+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi083-77648-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.118580+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e362: 8 total, 8 up, 8 in 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.288548+0000 osd.3 (osd.3) 228 : cluster [DBG] 3.18 scrub starts 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.293656+0000 osd.3 (osd.3) 229 : cluster [DBG] 3.18 scrub ok 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.382624+0000 osd.1 (osd.1) 215 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.383048+0000 osd.1 (osd.1) 216 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.429357+0000 osd.6 (osd.6) 161 : cluster [DBG] 4.1f scrub starts 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.434328+0000 osd.6 (osd.6) 162 : cluster [DBG] 4.1f scrub ok 2024-09-10T18:58:39.270 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:39 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.075876+0000 mon.a (mon.0) 2317 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:39.298 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:39.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:39.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:39.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:39.704 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:39.745 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:39.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.843 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:39.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:39.982 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:39.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:40.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:40.109 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:40.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.147885+0000 osd.7 (osd.7) 208 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.152064+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.636125+0000 osd.4 (osd.4) 241 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.640059+0000 osd.4 (osd.4) 242 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.860577+0000 osd.1 (osd.1) 217 : cluster [DBG] 2.d scrub starts 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:38.864351+0000 osd.1 (osd.1) 218 : cluster [DBG] 2.d scrub ok 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.152021+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.314293+0000 osd.3 (osd.3) 230 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.317471+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.377228+0000 osd.5 (osd.5) 195 : cluster [DBG] 64.12 scrub starts 2024-09-10T18:58:40.227 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.380536+0000 osd.5 (osd.5) 196 : cluster [DBG] 64.12 scrub ok 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.147885+0000 osd.7 (osd.7) 208 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.152064+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.636125+0000 osd.4 (osd.4) 241 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.640059+0000 osd.4 (osd.4) 242 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.860577+0000 osd.1 (osd.1) 217 : cluster [DBG] 2.d scrub starts 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:38.864351+0000 osd.1 (osd.1) 218 : cluster [DBG] 2.d scrub ok 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.152021+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.314293+0000 osd.3 (osd.3) 230 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.317471+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.377228+0000 osd.5 (osd.5) 195 : cluster [DBG] 64.12 scrub starts 2024-09-10T18:58:40.228 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:40 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.380536+0000 osd.5 (osd.5) 196 : cluster [DBG] 64.12 scrub ok 2024-09-10T18:58:40.243 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:40.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.147885+0000 osd.7 (osd.7) 208 : cluster [DBG] 3.15 scrub starts 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.152064+0000 osd.7 (osd.7) 209 : cluster [DBG] 3.15 scrub ok 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.636125+0000 osd.4 (osd.4) 241 : cluster [DBG] 12.0 scrub starts 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.640059+0000 osd.4 (osd.4) 242 : cluster [DBG] 12.0 scrub ok 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.860577+0000 osd.1 (osd.1) 217 : cluster [DBG] 2.d scrub starts 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:38.864351+0000 osd.1 (osd.1) 218 : cluster [DBG] 2.d scrub ok 2024-09-10T18:58:40.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.152021+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e363: 8 total, 8 up, 8 in 2024-09-10T18:58:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.314293+0000 osd.3 (osd.3) 230 : cluster [DBG] 6.6 scrub starts 2024-09-10T18:58:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.317471+0000 osd.3 (osd.3) 231 : cluster [DBG] 6.6 scrub ok 2024-09-10T18:58:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.377228+0000 osd.5 (osd.5) 195 : cluster [DBG] 64.12 scrub starts 2024-09-10T18:58:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:40 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.380536+0000 osd.5 (osd.5) 196 : cluster [DBG] 64.12 scrub ok 2024-09-10T18:58:40.381 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:40.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:40.516 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:40.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:40.657 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:40.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:40.794 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:40.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:40.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:40.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:41.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:41.066 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:41.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:41.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:41.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.852802+0000 osd.0 (osd.0) 200 : cluster [DBG] 64.1e scrub starts 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.853140+0000 osd.1 (osd.1) 219 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.856726+0000 osd.1 (osd.1) 220 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:39.857671+0000 osd.0 (osd.0) 201 : cluster [DBG] 64.1e scrub ok 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.046073+0000 mgr.y (mgr.24526) 382 : cluster [DBG] pgmap v529: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.162433+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-10T18:58:41.332 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: audit 2024-09-10T18:58:40.168693+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.399858+0000 osd.5 (osd.5) 197 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.404792+0000 osd.5 (osd.5) 198 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.424159+0000 osd.6 (osd.6) 163 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.428147+0000 osd.6 (osd.6) 164 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.901895+0000 osd.7 (osd.7) 210 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[32381]: cluster 2024-09-10T18:58:40.902233+0000 osd.7 (osd.7) 211 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.852802+0000 osd.0 (osd.0) 200 : cluster [DBG] 64.1e scrub starts 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.853140+0000 osd.1 (osd.1) 219 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.856726+0000 osd.1 (osd.1) 220 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:58:41.333 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:39.857671+0000 osd.0 (osd.0) 201 : cluster [DBG] 64.1e scrub ok 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.046073+0000 mgr.y (mgr.24526) 382 : cluster [DBG] pgmap v529: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.162433+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: audit 2024-09-10T18:58:40.168693+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.399858+0000 osd.5 (osd.5) 197 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.404792+0000 osd.5 (osd.5) 198 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.424159+0000 osd.6 (osd.6) 163 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.428147+0000 osd.6 (osd.6) 164 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.901895+0000 osd.7 (osd.7) 210 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:41.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:41 smithi083 bash[18629]: cluster 2024-09-10T18:58:40.902233+0000 osd.7 (osd.7) 211 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:41.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:41.381 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T18:58:41.382 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:58:41.423 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:41.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.852802+0000 osd.0 (osd.0) 200 : cluster [DBG] 64.1e scrub starts 2024-09-10T18:58:41.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.853140+0000 osd.1 (osd.1) 219 : cluster [DBG] 12.13 scrub starts 2024-09-10T18:58:41.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.856726+0000 osd.1 (osd.1) 220 : cluster [DBG] 12.13 scrub ok 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:39.857671+0000 osd.0 (osd.0) 201 : cluster [DBG] 64.1e scrub ok 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.046073+0000 mgr.y (mgr.24526) 382 : cluster [DBG] pgmap v529: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.162433+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e364: 8 total, 8 up, 8 in 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: audit 2024-09-10T18:58:40.168693+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.399858+0000 osd.5 (osd.5) 197 : cluster [DBG] 4.15 scrub starts 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.404792+0000 osd.5 (osd.5) 198 : cluster [DBG] 4.15 scrub ok 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.424159+0000 osd.6 (osd.6) 163 : cluster [DBG] 3.10 scrub starts 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.428147+0000 osd.6 (osd.6) 164 : cluster [DBG] 3.10 scrub ok 2024-09-10T18:58:41.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.901895+0000 osd.7 (osd.7) 210 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:58:41.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:41 smithi165 bash[22591]: cluster 2024-09-10T18:58:40.902233+0000 osd.7 (osd.7) 211 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:58:41.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:41.476 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:41.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:41.609 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:41.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:41.745 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:41.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:41.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:41.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:41.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:42.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T18:58:42.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:42.151 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout:o.RoundTripAppendPP (3003 ms) 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3009 ms) 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3430 ms) 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2570 ms) 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: setting pool EIO 2024-09-10T18:58:42.189 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: max_success 1293, min_failed 1294 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4484 ms) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2508 ms) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (129316 ms total) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3042 ms) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8967 ms) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15022 ms) 2024-09-10T18:58:42.190 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3008 ms) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30039 ms total) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3003 ms) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3003 ms total) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14001 ms) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7004 ms) 2024-09-10T18:58:42.191 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7440 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6589 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2971 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7020 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6988 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7016 ms) 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-10T18:58:42.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6979 ms) 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7035 ms) 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7005 ms) 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6975 ms) 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-10T18:58:42.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7097 ms) 2024-09-10T18:58:42.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:42.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:42 smithi165 bash[22591]: audit 2024-09-10T18:58:41.174944+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:42.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:42 smithi165 bash[22591]: cluster 2024-09-10T18:58:41.175014+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-10T18:58:42.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:42 smithi165 bash[22591]: audit 2024-09-10T18:58:41.177944+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:42.268 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:42 smithi165 bash[22591]: cluster 2024-09-10T18:58:41.231408+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:58:42.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:42 smithi165 bash[22591]: cluster 2024-09-10T18:58:41.241348+0000 osd.3 (osd.3) 233 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:58:42.288 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:42.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.419 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:42.458 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[32381]: audit 2024-09-10T18:58:41.174944+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[32381]: cluster 2024-09-10T18:58:41.175014+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[32381]: audit 2024-09-10T18:58:41.177944+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[32381]: cluster 2024-09-10T18:58:41.231408+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[32381]: cluster 2024-09-10T18:58:41.241348+0000 osd.3 (osd.3) 233 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18629]: audit 2024-09-10T18:58:41.174944+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18629]: cluster 2024-09-10T18:58:41.175014+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e365: 8 total, 8 up, 8 in 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18629]: audit 2024-09-10T18:58:41.177944+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]: dispatch 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18629]: cluster 2024-09-10T18:58:41.231408+0000 osd.3 (osd.3) 232 : cluster [DBG] 5.0 scrub starts 2024-09-10T18:58:42.459 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18629]: cluster 2024-09-10T18:58:41.241348+0000 osd.3 (osd.3) 233 : cluster [DBG] 5.0 scrub ok 2024-09-10T18:58:42.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.558 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:42.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:42.696 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:42.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:42] "GET /metrics HTTP/1.1" 200 228718 "" "Prometheus/2.43.0" 2024-09-10T18:58:42.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:42.834 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:42.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.973 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:42.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:43.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:43.108 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:43.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:43.231 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:43.334 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:43 smithi165 bash[22591]: cluster 2024-09-10T18:58:42.046794+0000 mgr.y (mgr.24526) 383 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:43 smithi165 bash[22591]: audit 2024-09-10T18:58:42.184202+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:43 smithi165 bash[22591]: cluster 2024-09-10T18:58:42.184260+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-10T18:58:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:43 smithi165 bash[22591]: cluster 2024-09-10T18:58:42.483148+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub starts 2024-09-10T18:58:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:43 smithi165 bash[22591]: cluster 2024-09-10T18:58:42.486490+0000 osd.6 (osd.6) 166 : cluster [DBG] 4.a scrub ok 2024-09-10T18:58:43.369 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:43.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:43.499 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[18629]: cluster 2024-09-10T18:58:42.046794+0000 mgr.y (mgr.24526) 383 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[18629]: audit 2024-09-10T18:58:42.184202+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[18629]: cluster 2024-09-10T18:58:42.184260+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[18629]: cluster 2024-09-10T18:58:42.483148+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub starts 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[18629]: cluster 2024-09-10T18:58:42.486490+0000 osd.6 (osd.6) 166 : cluster [DBG] 4.a scrub ok 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[32381]: cluster 2024-09-10T18:58:42.046794+0000 mgr.y (mgr.24526) 383 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[32381]: audit 2024-09-10T18:58:42.184202+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? v1:172.21.15.83:0/2421542018' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi083-77648-59"}]': finished 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[32381]: cluster 2024-09-10T18:58:42.184260+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e366: 8 total, 8 up, 8 in 2024-09-10T18:58:43.500 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[32381]: cluster 2024-09-10T18:58:42.483148+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub starts 2024-09-10T18:58:43.501 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:43 smithi083 bash[32381]: cluster 2024-09-10T18:58:42.486490+0000 osd.6 (osd.6) 166 : cluster [DBG] 4.a scrub ok 2024-09-10T18:58:43.509 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:43.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:43.643 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:43.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:43.785 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:43.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:43.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:43.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:44.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:44.060 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:44.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:44.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.154582+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.158072+0000 osd.7 (osd.7) 213 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.199383+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: audit 2024-09-10T18:58:43.201180+0000 mon.c (mon.2) 587 : audit [INF] from='client.? v1:172.21.15.83:0/2776807341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: audit 2024-09-10T18:58:43.202286+0000 mon.a (mon.0) 2327 : audit [INF] from='client.49283 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: audit 2024-09-10T18:58:44.091163+0000 mon.a (mon.0) 2328 : audit [INF] from='client.49283 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.091378+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-10T18:58:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.154582+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.158072+0000 osd.7 (osd.7) 213 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.199383+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: audit 2024-09-10T18:58:43.201180+0000 mon.c (mon.2) 587 : audit [INF] from='client.? v1:172.21.15.83:0/2776807341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: audit 2024-09-10T18:58:43.202286+0000 mon.a (mon.0) 2327 : audit [INF] from='client.49283 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: audit 2024-09-10T18:58:44.091163+0000 mon.a (mon.0) 2328 : audit [INF] from='client.49283 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:44.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:44 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.091378+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-10T18:58:44.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:44.333 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:44.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:44.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.154582+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub starts 2024-09-10T18:58:44.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.158072+0000 osd.7 (osd.7) 213 : cluster [DBG] 2.1d scrub ok 2024-09-10T18:58:44.449 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.199383+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e367: 8 total, 8 up, 8 in 2024-09-10T18:58:44.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: audit 2024-09-10T18:58:43.201180+0000 mon.c (mon.2) 587 : audit [INF] from='client.? v1:172.21.15.83:0/2776807341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: audit 2024-09-10T18:58:43.202286+0000 mon.a (mon.0) 2327 : audit [INF] from='client.49283 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:44.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: audit 2024-09-10T18:58:44.091163+0000 mon.a (mon.0) 2328 : audit [INF] from='client.49283 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi083-77648-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:44.450 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:44 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.091378+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e368: 8 total, 8 up, 8 in 2024-09-10T18:58:44.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:44.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:44.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:44.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:44.753 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:44.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:44.891 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:44.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:44.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.031 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:45.134 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.160 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:45.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.277 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.836887+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-09-10T18:58:45.277 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.840223+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.989662+0000 osd.2 (osd.2) 151 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:43.993815+0000 osd.2 (osd.2) 152 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.047338+0000 mgr.y (mgr.24526) 384 : cluster [DBG] pgmap v535: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.188881+0000 osd.7 (osd.7) 214 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.192845+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.194550+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.431025+0000 osd.5 (osd.5) 199 : cluster [DBG] 64.5 scrub starts 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.435717+0000 osd.5 (osd.5) 200 : cluster [DBG] 64.5 scrub ok 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: audit 2024-09-10T18:58:44.561740+0000 mon.c (mon.2) 588 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.844539+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:44.847772+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: cluster 2024-09-10T18:58:45.106924+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-10T18:58:45.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: audit 2024-09-10T18:58:45.131647+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: audit 2024-09-10T18:58:45.133632+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[18629]: audit 2024-09-10T18:58:45.134077+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.836887+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.840223+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.989662+0000 osd.2 (osd.2) 151 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:43.993815+0000 osd.2 (osd.2) 152 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.047338+0000 mgr.y (mgr.24526) 384 : cluster [DBG] pgmap v535: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.188881+0000 osd.7 (osd.7) 214 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.192845+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.194550+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.431025+0000 osd.5 (osd.5) 199 : cluster [DBG] 64.5 scrub starts 2024-09-10T18:58:45.279 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.435717+0000 osd.5 (osd.5) 200 : cluster [DBG] 64.5 scrub ok 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: audit 2024-09-10T18:58:44.561740+0000 mon.c (mon.2) 588 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.844539+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:44.847772+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: cluster 2024-09-10T18:58:45.106924+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: audit 2024-09-10T18:58:45.131647+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: audit 2024-09-10T18:58:45.133632+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.280 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:45 smithi083 bash[32381]: audit 2024-09-10T18:58:45.134077+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:45.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:45.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:45.424 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:45.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.564 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.836887+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.f scrub starts 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.840223+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.f scrub ok 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.989662+0000 osd.2 (osd.2) 151 : cluster [DBG] 64.13 scrub starts 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:43.993815+0000 osd.2 (osd.2) 152 : cluster [DBG] 64.13 scrub ok 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.047338+0000 mgr.y (mgr.24526) 384 : cluster [DBG] pgmap v535: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.188881+0000 osd.7 (osd.7) 214 : cluster [DBG] 2.0 scrub starts 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.192845+0000 osd.7 (osd.7) 215 : cluster [DBG] 2.0 scrub ok 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.194550+0000 mon.a (mon.0) 2330 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.431025+0000 osd.5 (osd.5) 199 : cluster [DBG] 64.5 scrub starts 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.435717+0000 osd.5 (osd.5) 200 : cluster [DBG] 64.5 scrub ok 2024-09-10T18:58:45.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: audit 2024-09-10T18:58:44.561740+0000 mon.c (mon.2) 588 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.844539+0000 osd.0 (osd.0) 202 : cluster [DBG] 6.9 scrub starts 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:44.847772+0000 osd.0 (osd.0) 203 : cluster [DBG] 6.9 scrub ok 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: cluster 2024-09-10T18:58:45.106924+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e369: 8 total, 8 up, 8 in 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: audit 2024-09-10T18:58:45.131647+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: audit 2024-09-10T18:58:45.133632+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:45 smithi165 bash[22591]: audit 2024-09-10T18:58:45.134077+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:45.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:45.698 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:45.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:45.835 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:45.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:45.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:46.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:46.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:46.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:46.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:46.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:46.384 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:46.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:46.522 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:46.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:46.656 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:46.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:46.796 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:46.836 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:46.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:46.932 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:46.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:47.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:47.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:47.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:47.202 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:47.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.202+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/98388575 >> v1:172.21.15.83:6790/0 conn(0x7fa0a8102660 legacy=0x7fa0a8102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.202+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/98388575 shutdown_connections 2024-09-10T18:58:47.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.202+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/98388575 >> 172.21.15.83:0/98388575 conn(0x7fa0a8074a20 msgr2=0x7fa0a80f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:47.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.202+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/98388575 shutdown_connections 2024-09-10T18:58:47.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.202+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/98388575 wait complete. 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 librados: starting msgr at 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 librados: starting objecter 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 Processor -- start 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- start start 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 librados: setting wanted keys 2024-09-10T18:58:47.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 librados: calling monclient init 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0a8101f70 con 0x7fa0a8108f30 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0a811f7b0 con 0x7fa0a8102660 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0a8120950 con 0x7fa0a81053d0 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0a7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa0a8108f30 0x7fa0a811d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45110/0 (socket says 172.21.15.83:45110) 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0a7fff640 1 -- 172.21.15.83:0/793915967 learned_addr learned my addr 172.21.15.83:0/793915967 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0a6ffd640 1 --1- 172.21.15.83:0/793915967 >> v1:172.21.15.83:6790/0 conn(0x7fa0a81053d0 0x7fa0a8119950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59660/0 (socket says 172.21.15.83:59660) 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter ms_handle_connect 0x7fa0a8108f30 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter ms_handle_connect 0x7fa0a81053d0 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter ms_handle_connect 0x7fa0a8102660 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716527139 0 0) 0x7fa0a8120950 con 0x7fa0a81053d0 2024-09-10T18:58:47.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090003720 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347625394 0 0) 0x7fa0a811f7b0 con 0x7fa0a8102660 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a8120950 con 0x7fa0a8102660 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362655197 0 0) 0x7fa090003720 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0a811f7b0 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa09c0018c0 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 674940199 0 0) 0x7fa0a811f7b0 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 >> v1:172.21.15.165:6789/0 conn(0x7fa0a8102660 legacy=0x7fa0a8101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 >> v1:172.21.15.83:6789/0 conn(0x7fa0a8108f30 legacy=0x7fa0a811d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8121af0 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a8120b40 con 0x7fa0a81053d0 2024-09-10T18:58:47.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 10 client.49295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8120f90 con 0x7fa0a81053d0 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 librados: init done 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa09c002ca0 con 0x7fa0a81053d0 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa09c005a80 con 0x7fa0a81053d0 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa094005010 con 0x7fa0a81053d0 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.206+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa09c01bea0 con 0x7fa0a81053d0 2024-09-10T18:58:47.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(371..371 src has 69..371) v4 ==== 8384+0+0 (unknown 4100539519 0 0) 0x7fa09c08a750 con 0x7fa0a81053d0 2024-09-10T18:58:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 10 client.49295.objecter ms_dispatch 0x7fa0a8000b90 osd_map(371..371 src has 69..371) v4 2024-09-10T18:58:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 3 client.49295.objecter handle_osd_map got epochs [371,371] > 0 2024-09-10T18:58:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 3 client.49295.objecter handle_osd_map decoding full epoch 371 2024-09-10T18:58:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 20 client.49295.objecter dump_active .. 0 homeless 2024-09-10T18:58:47.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.210+0000 7fa08ffff640 10 client.49295.objecter ms_handle_connect 0x7fa090071320 2024-09-10T18:58:47.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.222+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa09c056b90 con 0x7fa0a81053d0 2024-09-10T18:58:47.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:47.335 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:47.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.406+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa0940052c0 con 0x7fa0a81053d0 2024-09-10T18:58:47.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:47.471 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:47.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:45.867821+0000 osd.1 (osd.1) 223 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:45.871297+0000 osd.1 (osd.1) 224 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.049550+0000 mgr.y (mgr.24526) 385 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: audit 2024-09-10T18:58:46.108145+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.108719+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: audit 2024-09-10T18:58:46.110105+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:47.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.431700+0000 osd.5 (osd.5) 201 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:58:47.588 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.437391+0000 osd.5 (osd.5) 202 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:58:47.603 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- 172.21.15.83:0/285479504 >> v1:172.21.15.83:6790/0 conn(0x7ff614107580 legacy=0x7ff614109960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- 172.21.15.83:0/285479504 shutdown_connections 2024-09-10T18:58:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- 172.21.15.83:0/285479504 >> 172.21.15.83:0/285479504 conn(0x7ff6140fa370 msgr2=0x7ff6140fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- 172.21.15.83:0/285479504 shutdown_connections 2024-09-10T18:58:47.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- 172.21.15.83:0/285479504 wait complete. 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 librados: starting msgr at 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 librados: starting objecter 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 Processor -- start 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- start start 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 librados: setting wanted keys 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 librados: calling monclient init 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6141040f0 con 0x7ff614107580 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff61411f5e0 con 0x7ff61410b110 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff618ab5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6141207e0 con 0x7ff6141047f0 2024-09-10T18:58:47.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff611d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff614107580 0x7ff614119720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45124/0 (socket says 172.21.15.83:45124) 2024-09-10T18:58:47.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff611d9b640 1 -- 172.21.15.83:0/2766688520 learned_addr learned my addr 172.21.15.83:0/2766688520 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:47.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff61259c640 1 --1- 172.21.15.83:0/2766688520 >> v1:172.21.15.83:6790/0 conn(0x7ff6141047f0 0x7ff6141034e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59670/0 (socket says 172.21.15.83:59670) 2024-09-10T18:58:47.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter ms_handle_connect 0x7ff614107580 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter ms_handle_connect 0x7ff6141047f0 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter ms_handle_connect 0x7ff61410b110 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286410520 0 0) 0x7ff6141207e0 con 0x7ff6141047f0 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc003720 con 0x7ff6141047f0 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482952186 0 0) 0x7ff614112480 con 0x7ff614107580 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6141207e0 con 0x7ff614107580 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177874010 0 0) 0x7ff61411f5e0 con 0x7ff61410b110 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff614112480 con 0x7ff61410b110 2024-09-10T18:58:47.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4260648013 0 0) 0x7ff6141207e0 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff61411f5e0 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff608001430 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798119760 0 0) 0x7ff614112480 con 0x7ff61410b110 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6141207e0 con 0x7ff61410b110 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff60c001b80 con 0x7ff61410b110 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3277563957 0 0) 0x7ff5fc003720 con 0x7ff6141047f0 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff614112480 con 0x7ff6141047f0 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff604001840 con 0x7ff6141047f0 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.626+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 158054882 0 0) 0x7ff61411f5e0 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 >> v1:172.21.15.83:6790/0 conn(0x7ff6141047f0 legacy=0x7ff6141034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 >> v1:172.21.15.165:6789/0 conn(0x7ff61410b110 legacy=0x7ff61411ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6141219e0 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61411e5d0 con 0x7ff614107580 2024-09-10T18:58:47.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff618ab5640 10 client.25224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:47.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61411ea20 con 0x7ff614107580 2024-09-10T18:58:47.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff618ab5640 1 librados: init done 2024-09-10T18:58:47.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6140fff10 con 0x7ff614107580 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/1052257812 >> v1:172.21.15.83:6790/0 conn(0x7f47641025a0 legacy=0x7f4764102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/1052257812 shutdown_connections 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/1052257812 >> 172.21.15.83:0/1052257812 conn(0x7f47640f81a0 msgr2=0x7f47640fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/1052257812 shutdown_connections 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/1052257812 wait complete. 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff6080011e0 con 0x7ff614107580 2024-09-10T18:58:47.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff6080033a0 con 0x7ff614107580 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 librados: starting msgr at 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 librados: starting objecter 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 Processor -- start 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- start start 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 librados: setting wanted keys 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 librados: calling monclient init 2024-09-10T18:58:47.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47640ff340 con 0x7f47641025a0 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f476411d3f0 con 0x7f4764108ef0 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f476259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f4764105360 0x7f4764117590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59676/0 (socket says 172.21.15.83:59676) 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f476411e590 con 0x7f4764105360 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f476259c640 1 -- 172.21.15.83:0/2771272008 learned_addr learned my addr 172.21.15.83:0/2771272008 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff6080035f0 con 0x7ff614107580 2024-09-10T18:58:47.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f474b7fe640 10 client.?.objecter ms_handle_connect 0x7f4764105360 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f474b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f474b7fe640 10 client.?.objecter ms_handle_connect 0x7f47641025a0 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.630+0000 7f474b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986548778 0 0) 0x7f476411e590 con 0x7f4764105360 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f474c0034e0 con 0x7f4764105360 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(371..371 src has 69..371) v4 ==== 8384+0+0 (unknown 4100539519 0 0) 0x7ff60808a760 con 0x7ff614107580 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 10 client.25224.objecter ms_dispatch 0x7ff614000b90 osd_map(371..371 src has 69..371) v4 2024-09-10T18:58:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 3 client.25224.objecter handle_osd_map got epochs [371,371] > 0 2024-09-10T18:58:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 3 client.25224.objecter handle_osd_map decoding full epoch 371 2024-09-10T18:58:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 20 client.25224.objecter dump_active .. 0 homeless 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 10 client.?.objecter ms_handle_connect 0x7f4764108ef0 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270695792 0 0) 0x7f474c0034e0 con 0x7f4764105360 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f476411e590 con 0x7f4764105360 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4758002990 con 0x7f4764105360 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038595406 0 0) 0x7f476411d3f0 con 0x7f4764108ef0 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f474c0034e0 con 0x7f4764108ef0 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3606208475 0 0) 0x7f476411e590 con 0x7f4764105360 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.165:6789/0 conn(0x7f4764108ef0 legacy=0x7f476411acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6789/0 conn(0x7f47641025a0 legacy=0x7f47640fe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476411f770 con 0x7f4764105360 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476411c440 con 0x7f4764105360 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f47691b6640 10 client.49307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f476411c890 con 0x7f4764105360 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f47691b6640 1 librados: init done 2024-09-10T18:58:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7ff602ffd640 10 client.25224.objecter ms_handle_connect 0x7ff5fc0716a0 2024-09-10T18:58:47.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4758002d60 con 0x7f4764105360 2024-09-10T18:58:47.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4758005c40 con 0x7f4764105360 2024-09-10T18:58:47.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4758001eb0 con 0x7f4764105360 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(371..371 src has 69..371) v4 ==== 8384+0+0 (unknown 4100539519 0 0) 0x7f475808a800 con 0x7f4764105360 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 osd_map(371..371 src has 69..371) v4 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 3 client.49307.objecter handle_osd_map got epochs [371,371] > 0 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 3 client.49307.objecter handle_osd_map decoding full epoch 371 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 20 client.49307.objecter dump_active .. 0 homeless 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.634+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f474c0715a0 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4750004ff0 con 0x7f4764105360 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v371) v1 ==== 54+0+16 (unknown 3925016236 0 2420000192) 0x7f4758056c60 con 0x7f4764105360 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 10 client.49307.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 1 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 1 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 0 1 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 4 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 10 client.49307.objecter _send_command 1 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750005290 con 0x7f4750001b90 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f4750001b90 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff608057930 con 0x7ff614107580 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4750005290 con 0x7f4750001b90 2024-09-10T18:58:47.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:58:47.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 10 client.49307.objecter _finish_command 1 = system:0 2024-09-10T18:58:47.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 4 2024-09-10T18:58:47.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.638+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 0 1 2024-09-10T18:58:47.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 10 client.49307.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 2 2024-09-10T18:58:47.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 2 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750005810 osd=0 3 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 0 2 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 4 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 10 client.49307.objecter _send_command 2 2024-09-10T18:58:47.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.654+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750002200 con 0x7f4750001b90 2024-09-10T18:58:47.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4750002200 con 0x7f4750001b90 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 10 client.49307.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 4 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 0 2 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:58:47.660 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:58:47.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 10 client.49307.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 1 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 4 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 10 client.49307.objecter _send_command 3 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750003b40 con 0x7f47500035e0 2024-09-10T18:58:47.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f47500035e0 2024-09-10T18:58:47.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4750003b40 con 0x7f47500035e0 2024-09-10T18:58:47.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:58:47.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 10 client.49307.objecter _finish_command 3 = system:0 2024-09-10T18:58:47.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 4 2024-09-10T18:58:47.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.658+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 1 3 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 10 client.49307.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 4 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 4 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f47500023b0 osd=1 3 2024-09-10T18:58:47.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 1 4 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 4 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 10 client.49307.objecter _send_command 4 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f47500030d0 con 0x7f47500035e0 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47500030d0 con 0x7f47500035e0 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 10 client.49307.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 4 2024-09-10T18:58:47.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 1 4 2024-09-10T18:58:47.677 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:58:47.677 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:58:47.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 10 client.49307.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 5 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 5 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 2 5 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 4 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 10 client.49307.objecter _send_command 5 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750004df0 con 0x7f475000d350 2024-09-10T18:58:47.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.674+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f475000d350 2024-09-10T18:58:47.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.678+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4750004df0 con 0x7f475000d350 2024-09-10T18:58:47.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.678+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:58:47.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.678+0000 7f474b7fe640 10 client.49307.objecter _finish_command 5 = system:0 2024-09-10T18:58:47.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.678+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 4 2024-09-10T18:58:47.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.678+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 2 5 2024-09-10T18:58:47.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 10 client.49307.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 6 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 6 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750004610 osd=2 3 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 2 6 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 4 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 10 client.49307.objecter _send_command 6 2024-09-10T18:58:47.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.702+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750004100 con 0x7f475000d350 2024-09-10T18:58:47.706 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4750004100 con 0x7f475000d350 2024-09-10T18:58:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 10 client.49307.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 4 2024-09-10T18:58:47.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 2 6 2024-09-10T18:58:47.708 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 10 client.49307.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 7 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 7 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 3 7 2024-09-10T18:58:47.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 4 2024-09-10T18:58:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 10 client.49307.objecter _send_command 7 2024-09-10T18:58:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475000d080 con 0x7f4750011fd0 2024-09-10T18:58:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f4750011fd0 2024-09-10T18:58:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f475000d080 con 0x7f4750011fd0 2024-09-10T18:58:47.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:58:47.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 10 client.49307.objecter _finish_command 7 = system:0 2024-09-10T18:58:47.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 4 2024-09-10T18:58:47.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.706+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 3 7 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:45.867821+0000 osd.1 (osd.1) 223 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:45.871297+0000 osd.1 (osd.1) 224 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.049550+0000 mgr.y (mgr.24526) 385 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: audit 2024-09-10T18:58:46.108145+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.108719+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-10T18:58:47.719 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: audit 2024-09-10T18:58:46.110105+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.431700+0000 osd.5 (osd.5) 201 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.437391+0000 osd.5 (osd.5) 202 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:45.867821+0000 osd.1 (osd.1) 223 : cluster [DBG] 6.1 scrub starts 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:45.871297+0000 osd.1 (osd.1) 224 : cluster [DBG] 6.1 scrub ok 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.049550+0000 mgr.y (mgr.24526) 385 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: audit 2024-09-10T18:58:46.108145+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi083-77648-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.108719+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e370: 8 total, 8 up, 8 in 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: audit 2024-09-10T18:58:46.110105+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:47.720 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.431700+0000 osd.5 (osd.5) 201 : cluster [DBG] 3.5 scrub starts 2024-09-10T18:58:47.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:47 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.437391+0000 osd.5 (osd.5) 202 : cluster [DBG] 3.5 scrub ok 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 10 client.49307.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 8 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 8 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f475000c990 osd=3 3 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 3 8 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 4 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 10 client.49307.objecter _send_command 8 2024-09-10T18:58:47.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.734+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f47500132f0 con 0x7f4750011fd0 2024-09-10T18:58:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:47.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f47500132f0 con 0x7f4750011fd0 2024-09-10T18:58:47.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 10 client.49307.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 4 2024-09-10T18:58:47.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 3 8 2024-09-10T18:58:47.740 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 10 client.49307.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 9 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 9 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 4 9 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 4 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 10 client.49307.objecter _send_command 9 2024-09-10T18:58:47.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475001e640 con 0x7f475001ab40 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.738+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f475001ab40 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.742+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f475001e640 con 0x7f475001ab40 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.742+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.742+0000 7f474b7fe640 10 client.49307.objecter _finish_command 9 = system:0 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.742+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 4 2024-09-10T18:58:47.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.742+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 4 9 2024-09-10T18:58:47.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 10 client.49307.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 10 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 10 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f47500132f0 osd=4 3 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 4 10 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 4 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 10 client.49307.objecter _send_command 10 2024-09-10T18:58:47.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.766+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f475001e640 con 0x7f475001ab40 2024-09-10T18:58:47.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f475001e640 con 0x7f475001ab40 2024-09-10T18:58:47.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 10 client.49307.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 4 2024-09-10T18:58:47.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 4 10 2024-09-10T18:58:47.773 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:58:47.773 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:58:47.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 10 client.49307.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 11 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 11 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 5 11 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 4 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 10 client.49307.objecter _send_command 11 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750022990 con 0x7f475001ed00 2024-09-10T18:58:47.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.770+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f475001ed00 2024-09-10T18:58:47.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.774+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4750022990 con 0x7f475001ed00 2024-09-10T18:58:47.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.774+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:58:47.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.774+0000 7f474b7fe640 10 client.49307.objecter _finish_command 11 = system:0 2024-09-10T18:58:47.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.774+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 4 2024-09-10T18:58:47.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.774+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 5 11 2024-09-10T18:58:47.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 10 client.49307.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 12 2024-09-10T18:58:47.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 12 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f475001e640 osd=5 3 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 5 12 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 4 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 10 client.49307.objecter _send_command 12 2024-09-10T18:58:47.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.798+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750024230 con 0x7f475001ed00 2024-09-10T18:58:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.802+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4750024230 con 0x7f475001ed00 2024-09-10T18:58:47.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.802+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.802+0000 7f474b7fe640 10 client.49307.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.802+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 4 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.802+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 5 12 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:58:47.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 10 client.49307.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 13 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 13 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 6 13 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 4 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 10 client.49307.objecter _send_command 13 2024-09-10T18:58:47.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750024e00 con 0x7f47500248f0 2024-09-10T18:58:47.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f47500248f0 2024-09-10T18:58:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4750024e00 con 0x7f47500248f0 2024-09-10T18:58:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:58:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 10 client.49307.objecter _finish_command 13 = system:0 2024-09-10T18:58:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 4 2024-09-10T18:58:47.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.806+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 6 13 2024-09-10T18:58:47.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 10 client.49307.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 14 2024-09-10T18:58:47.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 14 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750024230 osd=6 3 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 6 14 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 4 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 10 client.49307.objecter _send_command 14 2024-09-10T18:58:47.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.826+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4750024e00 con 0x7f47500248f0 2024-09-10T18:58:47.832 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:58:47.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4750024e00 con 0x7f47500248f0 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 10 client.49307.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 4 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 6 14 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 10 client.49307.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 15 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 15 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 7 15 2024-09-10T18:58:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 4 2024-09-10T18:58:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 10 client.49307.objecter _send_command 15 2024-09-10T18:58:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475002b200 con 0x7f4750024ff0 2024-09-10T18:58:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.830+0000 7f474b7fe640 10 client.49307.objecter ms_handle_connect 0x7f4750024ff0 2024-09-10T18:58:47.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.834+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f475002b200 con 0x7f4750024ff0 2024-09-10T18:58:47.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.834+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:58:47.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.834+0000 7f474b7fe640 10 client.49307.objecter _finish_command 15 = system:0 2024-09-10T18:58:47.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.834+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 4 2024-09-10T18:58:47.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.834+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 7 15 2024-09-10T18:58:47.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 10 client.49307.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign -1 16 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 20 client.49307.objecter _get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 15 client.49307.objecter _session_command_op_remove -1 16 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 20 client.49307.objecter get_session s=0x7f4750024e00 osd=7 3 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 15 client.49307.objecter _session_command_op_assign 7 16 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 4 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 10 client.49307.objecter _send_command 16 2024-09-10T18:58:47.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.846+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f475000c240 con 0x7f4750024ff0 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.850+0000 7f474b7fe640 1 -- 172.21.15.83:0/2771272008 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f475000c240 con 0x7f4750024ff0 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.850+0000 7f474b7fe640 10 client.49307.objecter ms_dispatch 0x7f4764000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.850+0000 7f474b7fe640 10 client.49307.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.850+0000 7f474b7fe640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 4 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.850+0000 7f474b7fe640 15 client.49307.objecter _session_command_op_remove 7 16 2024-09-10T18:58:47.854 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:58:47.855 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 librados: watch_flush enter 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 librados: watch_flush exit 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.0 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6801/2535527483 conn(0x7f4750001b90 legacy=0x7f4750002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750005810 osd=0 1 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.1 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6805/2309624637 conn(0x7f47500035e0 legacy=0x7f475000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500023b0 osd=1 1 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.2 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6809/386094220 conn(0x7f475000d350 legacy=0x7f475000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750004610 osd=2 1 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.3 2024-09-10T18:58:47.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6813/1279688021 conn(0x7f4750011fd0 legacy=0x7f4750012420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475000c990 osd=3 1 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.4 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.165:6800/3225515265 conn(0x7f475001ab40 legacy=0x7f475001cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f47500132f0 osd=4 1 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.5 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.165:6804/571173777 conn(0x7f475001ed00 legacy=0x7f4750021140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f475001e640 osd=5 1 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.6 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.165:6808/1317922936 conn(0x7f47500248f0 legacy=0x7f4750022990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024230 osd=6 1 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter close_session for osd.7 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.165:6812/1610151650 conn(0x7f4750024ff0 legacy=0x7f47500299b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter put_session s=0x7f4750024e00 osd=7 1 2024-09-10T18:58:47.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 20 client.49307.objecter shutdown clearing up homeless session... 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 10 client.49307.objecter successfully canceled tick 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f476359e640 1 -- 172.21.15.83:0/2771272008 reap_dead start 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6800/3071793735 conn(0x7f474c0715a0 legacy=0x7f474c073a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f476359e640 1 -- 172.21.15.83:0/2771272008 reap_dead start 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> v1:172.21.15.83:6790/0 conn(0x7f4764105360 legacy=0x7f4764117590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f476359e640 1 -- 172.21.15.83:0/2771272008 reap_dead start 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f476359e640 1 -- 172.21.15.83:0/2771272008 reap_dead start 2024-09-10T18:58:47.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.854+0000 7f476359e640 1 -- 172.21.15.83:0/2771272008 reap_dead start 2024-09-10T18:58:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.858+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 shutdown_connections 2024-09-10T18:58:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.858+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 >> 172.21.15.83:0/2771272008 conn(0x7f47640f81a0 msgr2=0x7f4764109350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.858+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 shutdown_connections 2024-09-10T18:58:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.858+0000 7f47691b6640 1 -- 172.21.15.83:0/2771272008 wait complete. 2024-09-10T18:58:47.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.858+0000 7f47691b6640 1 librados: shutdown 2024-09-10T18:58:47.864 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:47.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.866+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff614110d00 con 0x7ff5fc0716a0 2024-09-10T18:58:47.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.886+0000 7ff602ffd640 1 -- 172.21.15.83:0/2766688520 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+624759 (unknown 2965378022 0 1204620578) 0x7ff614110d00 con 0x7ff5fc0716a0 2024-09-10T18:58:47.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 10 librados: watch_flush enter 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 10 librados: watch_flush exit 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 20 client.25224.objecter shutdown clearing up homeless session... 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 10 client.25224.objecter successfully canceled tick 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff5fc0716a0 legacy=0x7ff5fc073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 >> v1:172.21.15.83:6789/0 conn(0x7ff614107580 legacy=0x7ff614119720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 shutdown_connections 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 >> 172.21.15.83:0/2766688520 conn(0x7ff6140fa370 msgr2=0x7ff61410b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:47.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 shutdown_connections 2024-09-10T18:58:47.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 -- 172.21.15.83:0/2766688520 wait complete. 2024-09-10T18:58:47.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:47.894+0000 7ff618ab5640 1 librados: shutdown 2024-09-10T18:58:47.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:47.996 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:47.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:48.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:48.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.114+0000 7fa08ffff640 1 -- 172.21.15.83:0/793915967 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v372) v1 ==== 101+0+0 (unknown 2643912090 0 0) 0x7fa09c056530 con 0x7fa0a81053d0 2024-09-10T18:58:48.122 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T18:58:48.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.118+0000 7fa0ae0a7640 10 librados: watch_flush enter 2024-09-10T18:58:48.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.122+0000 7fa0ae0a7640 10 librados: watch_flush exit 2024-09-10T18:58:48.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.122+0000 7fa0ae0a7640 20 client.49295.objecter shutdown clearing up homeless session... 2024-09-10T18:58:48.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.122+0000 7fa0ae0a7640 10 client.49295.objecter successfully canceled tick 2024-09-10T18:58:48.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.122+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa090071320 legacy=0x7fa0900737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:48.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.122+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 >> v1:172.21.15.83:6790/0 conn(0x7fa0a81053d0 legacy=0x7fa0a8119950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:48.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.126+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 shutdown_connections 2024-09-10T18:58:48.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.126+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 >> 172.21.15.83:0/793915967 conn(0x7fa0a8074a20 msgr2=0x7fa0a810c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:48.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.126+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 shutdown_connections 2024-09-10T18:58:48.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.126+0000 7fa0ae0a7640 1 -- 172.21.15.83:0/793915967 wait complete. 2024-09-10T18:58:48.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:58:48.126+0000 7fa0ae0a7640 1 librados: shutdown 2024-09-10T18:58:48.132 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:48.218 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:48.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:48.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.401 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:48.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:48.536 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:48.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:48.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:48.776 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.497805+0000 osd.4 (osd.4) 243 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.501500+0000 osd.4 (osd.4) 244 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.950710+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:46.953600+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.171581+0000 osd.7 (osd.7) 216 : cluster [DBG] 3.e scrub starts 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.177731+0000 osd.7 (osd.7) 217 : cluster [DBG] 3.e scrub ok 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:47.410413+0000 mon.c (mon.2) 589 : audit [INF] from='client.? v1:172.21.15.83:0/793915967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.789 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.471053+0000 osd.5 (osd.5) 203 : cluster [DBG] 64.c scrub starts 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.475276+0000 osd.5 (osd.5) 204 : cluster [DBG] 64.c scrub ok 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.517636+0000 osd.4 (osd.4) 245 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.523088+0000 osd.4 (osd.4) 246 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.552263+0000 osd.6 (osd.6) 167 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.555622+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.557521+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:47.558800+0000 mon.a (mon.0) 2339 : audit [INF] from='client.49295 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:47.639226+0000 mon.c (mon.2) 590 : audit [DBG] from='client.? v1:172.21.15.83:0/2771272008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:47.869337+0000 mgr.y (mgr.24526) 386 : audit [DBG] from='client.25224 v1:172.21.15.83:0/2766688520' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:47.889672+0000 mgr.y (mgr.24526) 387 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.050405+0000 mgr.y (mgr.24526) 388 : cluster [DBG] pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.104282+0000 mon.a (mon.0) 2340 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:48.112664+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: audit 2024-09-10T18:58:48.112883+0000 mon.a (mon.0) 2342 : audit [INF] from='client.49295 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.113017+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.159963+0000 osd.3 (osd.3) 234 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.167648+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.497805+0000 osd.4 (osd.4) 243 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.501500+0000 osd.4 (osd.4) 244 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.950710+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:46.953600+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.171581+0000 osd.7 (osd.7) 216 : cluster [DBG] 3.e scrub starts 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.177731+0000 osd.7 (osd.7) 217 : cluster [DBG] 3.e scrub ok 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:47.410413+0000 mon.c (mon.2) 589 : audit [INF] from='client.? v1:172.21.15.83:0/793915967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.471053+0000 osd.5 (osd.5) 203 : cluster [DBG] 64.c scrub starts 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.475276+0000 osd.5 (osd.5) 204 : cluster [DBG] 64.c scrub ok 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.517636+0000 osd.4 (osd.4) 245 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.523088+0000 osd.4 (osd.4) 246 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.552263+0000 osd.6 (osd.6) 167 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.555622+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.557521+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:47.558800+0000 mon.a (mon.0) 2339 : audit [INF] from='client.49295 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:47.639226+0000 mon.c (mon.2) 590 : audit [DBG] from='client.? v1:172.21.15.83:0/2771272008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:47.869337+0000 mgr.y (mgr.24526) 386 : audit [DBG] from='client.25224 v1:172.21.15.83:0/2766688520' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:47.889672+0000 mgr.y (mgr.24526) 387 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.050405+0000 mgr.y (mgr.24526) 388 : cluster [DBG] pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.104282+0000 mon.a (mon.0) 2340 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:58:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:48.112664+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:48.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: audit 2024-09-10T18:58:48.112883+0000 mon.a (mon.0) 2342 : audit [INF] from='client.49295 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:48.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.113017+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-10T18:58:48.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.159963+0000 osd.3 (osd.3) 234 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:58:48.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:48 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.167648+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:58:48.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.497805+0000 osd.4 (osd.4) 243 : cluster [DBG] 12.1a scrub starts 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.501500+0000 osd.4 (osd.4) 244 : cluster [DBG] 12.1a scrub ok 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.950710+0000 osd.2 (osd.2) 153 : cluster [DBG] 2.10 scrub starts 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:46.953600+0000 osd.2 (osd.2) 154 : cluster [DBG] 2.10 scrub ok 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.171581+0000 osd.7 (osd.7) 216 : cluster [DBG] 3.e scrub starts 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.177731+0000 osd.7 (osd.7) 217 : cluster [DBG] 3.e scrub ok 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:47.410413+0000 mon.c (mon.2) 589 : audit [INF] from='client.? v1:172.21.15.83:0/793915967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.471053+0000 osd.5 (osd.5) 203 : cluster [DBG] 64.c scrub starts 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.475276+0000 osd.5 (osd.5) 204 : cluster [DBG] 64.c scrub ok 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.517636+0000 osd.4 (osd.4) 245 : cluster [DBG] 12.2 scrub starts 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.523088+0000 osd.4 (osd.4) 246 : cluster [DBG] 12.2 scrub ok 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.552263+0000 osd.6 (osd.6) 167 : cluster [DBG] 4.5 scrub starts 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.555622+0000 osd.6 (osd.6) 168 : cluster [DBG] 4.5 scrub ok 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.557521+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e371: 8 total, 8 up, 8 in 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:47.558800+0000 mon.a (mon.0) 2339 : audit [INF] from='client.49295 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:47.639226+0000 mon.c (mon.2) 590 : audit [DBG] from='client.? v1:172.21.15.83:0/2771272008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:47.869337+0000 mgr.y (mgr.24526) 386 : audit [DBG] from='client.25224 v1:172.21.15.83:0/2766688520' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:47.889672+0000 mgr.y (mgr.24526) 387 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.050405+0000 mgr.y (mgr.24526) 388 : cluster [DBG] pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.104282+0000 mon.a (mon.0) 2340 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T18:58:48.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:48.112664+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi083-77648-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:48.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: audit 2024-09-10T18:58:48.112883+0000 mon.a (mon.0) 2342 : audit [INF] from='client.49295 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T18:58:48.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.113017+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e372: 8 total, 8 up, 8 in 2024-09-10T18:58:48.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.159963+0000 osd.3 (osd.3) 234 : cluster [DBG] 6.1b scrub starts 2024-09-10T18:58:48.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:48 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.167648+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.1b scrub ok 2024-09-10T18:58:48.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.944 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:48.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:49.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:49.081 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:49.185 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:49.216 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:49.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:49.351 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:49.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:49.489 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:49.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:49.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:49.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.869266+0000 osd.1 (osd.1) 225 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:47.871587+0000 osd.1 (osd.1) 226 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.213887+0000 osd.7 (osd.7) 218 : cluster [DBG] 3.f scrub starts 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:48.217403+0000 osd.7 (osd.7) 219 : cluster [DBG] 3.f scrub ok 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.117979+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.194638+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:58:49.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.198218+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.869266+0000 osd.1 (osd.1) 225 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:47.871587+0000 osd.1 (osd.1) 226 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.213887+0000 osd.7 (osd.7) 218 : cluster [DBG] 3.f scrub starts 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:48.217403+0000 osd.7 (osd.7) 219 : cluster [DBG] 3.f scrub ok 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.117979+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.194638+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:58:49.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:49 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.198218+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:58:49.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.869266+0000 osd.1 (osd.1) 225 : cluster [DBG] 12.6 scrub starts 2024-09-10T18:58:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:47.871587+0000 osd.1 (osd.1) 226 : cluster [DBG] 12.6 scrub ok 2024-09-10T18:58:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.213887+0000 osd.7 (osd.7) 218 : cluster [DBG] 3.f scrub starts 2024-09-10T18:58:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:48.217403+0000 osd.7 (osd.7) 219 : cluster [DBG] 3.f scrub ok 2024-09-10T18:58:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.117979+0000 mon.a (mon.0) 2344 : cluster [DBG] osdmap e373: 8 total, 8 up, 8 in 2024-09-10T18:58:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.194638+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.1d scrub starts 2024-09-10T18:58:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:49 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.198218+0000 osd.3 (osd.3) 237 : cluster [DBG] 4.1d scrub ok 2024-09-10T18:58:49.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:49.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:49.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:50.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:50.030 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:50.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:50.157 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:50.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.293 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:50.380 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":540,"stamp":"2024-09-10T18:58:47.105230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791886,"num_objects":324,"num_object_clones":2,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":0,"num_read":1521,"num_read_kb":1282,"num_write":1112,"num_write_kb":14331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2474,"ondisk_log_size":2474,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":709,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2483396,"kb_used_data":71180,"kb_used_omap":13,"kb_used_meta":2412018,"kb_avail":747477820,"statfs":{"total":767960285184,"available":765417287680,"internally_reserved":0,"allocated":72888320,"data_stored":52364250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2469907136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":84,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.909428"},"pg_stats":[{"pgid":"197.1f","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.102726+0000","last_change":"2024-09-10T18:58:32.209818+0000","last_active":"2024-09-10T18:58:44.102726+0000","last_peered":"2024-09-10T18:58:44.102726+0000","last_clean":"2024-09-10T18:58:44.102726+0000","last_became_active":"2024-09-10T18:58:32.209541+0000","last_became_peered":"2024-09-10T18:58:32.209541+0000","last_unstale":"2024-09-10T18:58:44.102726+0000","last_undegraded":"2024-09-10T18:58:44.102726+0000","last_fullsized":"2024-09-10T18:58:44.102726+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.609092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.1e","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094192+0000","last_change":"2024-09-10T18:58:32.218303+0000","last_active":"2024-09-10T18:58:44.094192+0000","last_peered":"2024-09-10T18:58:44.094192+0000","last_clean":"2024-09-10T18:58:44.094192+0000","last_became_active":"2024-09-10T18:58:32.215570+0000","last_became_peered":"2024-09-10T18:58:32.215570+0000","last_unstale":"2024-09-10T18:58:44.094192+0000","last_undegraded":"2024-09-10T18:58:44.094192+0000","last_fullsized":"2024-09-10T18:58:44.094192+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:32.522586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.1d","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119662+0000","last_change":"2024-09-10T18:58:32.215879+0000","last_active":"2024-09-10T18:58:44.119662+0000","last_peered":"2024-09-10T18:58:44.119662+0000","last_clean":"2024-09-10T18:58:44.119662+0000","last_became_active":"2024-09-10T18:58:32.215608+0000","last_became_peered":"2024-09-10T18:58:32.215608+0000","last_unstale":"2024-09-10T18:58:44.119662+0000","last_undegraded":"2024-09-10T18:58:44.119662+0000","last_fullsized":"2024-09-10T18:58:44.119662+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:59.182393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.1c","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108647+0000","last_change":"2024-09-10T18:58:32.222831+0000","last_active":"2024-09-10T18:58:44.108647+0000","last_peered":"2024-09-10T18:58:44.108647+0000","last_clean":"2024-09-10T18:58:44.108647+0000","last_became_active":"2024-09-10T18:58:32.222225+0000","last_became_peered":"2024-09-10T18:58:32.222225+0000","last_unstale":"2024-09-10T18:58:44.108647+0000","last_undegraded":"2024-09-10T18:58:44.108647+0000","last_fullsized":"2024-09-10T18:58:44.108647+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.939283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.1b","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100623+0000","last_change":"2024-09-10T18:58:32.220645+0000","last_active":"2024-09-10T18:58:44.100623+0000","last_peered":"2024-09-10T18:58:44.100623+0000","last_clean":"2024-09-10T18:58:44.100623+0000","last_became_active":"2024-09-10T18:58:32.220311+0000","last_became_peered":"2024-09-10T18:58:32.220311+0000","last_unstale":"2024-09-10T18:58:44.100623+0000","last_undegraded":"2024-09-10T18:58:44.100623+0000","last_fullsized":"2024-09-10T18:58:44.100623+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:57.514536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.1a","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113084+0000","last_change":"2024-09-10T18:58:32.218402+0000","last_active":"2024-09-10T18:58:44.113084+0000","last_peered":"2024-09-10T18:58:44.113084+0000","last_clean":"2024-09-10T18:58:44.113084+0000","last_became_active":"2024-09-10T18:58:32.218126+0000","last_became_peered":"2024-09-10T18:58:32.218126+0000","last_unstale":"2024-09-10T18:58:44.113084+0000","last_undegraded":"2024-09-10T18:58:44.113084+0000","last_fullsized":"2024-09-10T18:58:44.113084+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:59.491361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.19","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.112722+0000","last_change":"2024-09-10T18:58:32.226836+0000","last_active":"2024-09-10T18:58:44.112722+0000","last_peered":"2024-09-10T18:58:44.112722+0000","last_clean":"2024-09-10T18:58:44.112722+0000","last_became_active":"2024-09-10T18:58:32.226300+0000","last_became_peered":"2024-09-10T18:58:32.226300+0000","last_unstale":"2024-09-10T18:58:44.112722+0000","last_undegraded":"2024-09-10T18:58:44.112722+0000","last_fullsized":"2024-09-10T18:58:44.112722+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:32.275111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.18","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099311+0000","last_change":"2024-09-10T18:58:32.159799+0000","last_active":"2024-09-10T18:58:44.099311+0000","last_peered":"2024-09-10T18:58:44.099311+0000","last_clean":"2024-09-10T18:58:44.099311+0000","last_became_active":"2024-09-10T18:58:32.159504+0000","last_became_peered":"2024-09-10T18:58:32.159504+0000","last_unstale":"2024-09-10T18:58:44.099311+0000","last_undegraded":"2024-09-10T18:58:44.099311+0000","last_fullsized":"2024-09-10T18:58:44.099311+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.074997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.17","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109856+0000","last_change":"2024-09-10T18:58:32.223285+0000","last_active":"2024-09-10T18:58:44.109856+0000","last_peered":"2024-09-10T18:58:44.109856+0000","last_clean":"2024-09-10T18:58:44.109856+0000","last_became_active":"2024-09-10T18:58:32.222950+0000","last_became_peered":"2024-09-10T18:58:32.222950+0000","last_unstale":"2024-09-10T18:58:44.109856+0000","last_undegraded":"2024-09-10T18:58:44.109856+0000","last_fullsized":"2024-09-10T18:58:44.109856+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.507191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.16","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094282+0000","last_change":"2024-09-10T18:58:32.210931+0000","last_active":"2024-09-10T18:58:44.094282+0000","last_peered":"2024-09-10T18:58:44.094282+0000","last_clean":"2024-09-10T18:58:44.094282+0000","last_became_active":"2024-09-10T18:58:32.208122+0000","last_became_peered":"2024-09-10T18:58:32.208122+0000","last_unstale":"2024-09-10T18:58:44.094282+0000","last_undegraded":"2024-09-10T18:58:44.094282+0000","last_fullsized":"2024-09-10T18:58:44.094282+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.683324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.15","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.104003+0000","last_change":"2024-09-10T18:58:32.205802+0000","last_active":"2024-09-10T18:58:44.104003+0000","last_peered":"2024-09-10T18:58:44.104003+0000","last_clean":"2024-09-10T18:58:44.104003+0000","last_became_active":"2024-09-10T18:58:32.205506+0000","last_became_peered":"2024-09-10T18:58:32.205506+0000","last_unstale":"2024-09-10T18:58:44.104003+0000","last_undegraded":"2024-09-10T18:58:44.104003+0000","last_fullsized":"2024-09-10T18:58:44.104003+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.916350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"197.14","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088055+0000","last_change":"2024-09-10T18:58:32.203188+0000","last_active":"2024-09-10T18:58:44.088055+0000","last_peered":"2024-09-10T18:58:44.088055+0000","last_clean":"2024-09-10T18:58:44.088055+0000","last_became_active":"2024-09-10T18:58:32.202693+0000","last_became_peered":"2024-09-10T18:58:32.202693+0000","last_unstale":"2024-09-10T18:58:44.088055+0000","last_undegraded":"2024-09-10T18:58:44.088055+0000","last_fullsized":"2024-09-10T18:58:44.088055+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:57.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.13","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100735+0000","last_change":"2024-09-10T18:58:32.208359+0000","last_active":"2024-09-10T18:58:44.100735+0000","last_peered":"2024-09-10T18:58:44.100735+0000","last_clean":"2024-09-10T18:58:44.100735+0000","last_became_active":"2024-09-10T18:58:32.207636+0000","last_became_peered":"2024-09-10T18:58:32.207636+0000","last_unstale":"2024-09-10T18:58:44.100735+0000","last_undegraded":"2024-09-10T18:58:44.100735+0000","last_fullsized":"2024-09-10T18:58:44.100735+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:46.977627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.12","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113106+0000","last_change":"2024-09-10T18:58:32.224858+0000","last_active":"2024-09-10T18:58:44.113106+0000","last_peered":"2024-09-10T18:58:44.113106+0000","last_clean":"2024-09-10T18:58:44.113106+0000","last_became_active":"2024-09-10T18:58:32.224594+0000","last_became_peered":"2024-09-10T18:58:32.224594+0000","last_unstale":"2024-09-10T18:58:44.113106+0000","last_undegraded":"2024-09-10T18:58:44.113106+0000","last_fullsized":"2024-09-10T18:58:44.113106+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.429155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.11","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107486+0000","last_change":"2024-09-10T18:58:32.209964+0000","last_active":"2024-09-10T18:58:44.107486+0000","last_peered":"2024-09-10T18:58:44.107486+0000","last_clean":"2024-09-10T18:58:44.107486+0000","last_became_active":"2024-09-10T18:58:32.209626+0000","last_became_peered":"2024-09-10T18:58:32.209626+0000","last_unstale":"2024-09-10T18:58:44.107486+0000","last_undegraded":"2024-09-10T18:58:44.107486+0000","last_fullsized":"2024-09-10T18:58:44.107486+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.211998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.10","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107274+0000","last_change":"2024-09-10T18:58:32.203825+0000","last_active":"2024-09-10T18:58:44.107274+0000","last_peered":"2024-09-10T18:58:44.107274+0000","last_clean":"2024-09-10T18:58:44.107274+0000","last_became_active":"2024-09-10T18:58:32.203013+0000","last_became_peered":"2024-09-10T18:58:32.203013+0000","last_unstale":"2024-09-10T18:58:44.107274+0000","last_undegraded":"2024-09-10T18:58:44.107274+0000","last_fullsized":"2024-09-10T18:58:44.107274+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.160349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.f","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089670+0000","last_change":"2024-09-10T18:58:32.207494+0000","last_active":"2024-09-10T18:58:44.089670+0000","last_peered":"2024-09-10T18:58:44.089670+0000","last_clean":"2024-09-10T18:58:44.089670+0000","last_became_active":"2024-09-10T18:58:32.207128+0000","last_became_peered":"2024-09-10T18:58:32.207128+0000","last_unstale":"2024-09-10T18:58:44.089670+0000","last_undegraded":"2024-09-10T18:58:44.089670+0000","last_fullsized":"2024-09-10T18:58:44.089670+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:53.939321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.e","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110586+0000","last_change":"2024-09-10T18:58:32.205950+0000","last_active":"2024-09-10T18:58:44.110586+0000","last_peered":"2024-09-10T18:58:44.110586+0000","last_clean":"2024-09-10T18:58:44.110586+0000","last_became_active":"2024-09-10T18:58:32.205469+0000","last_became_peered":"2024-09-10T18:58:32.205469+0000","last_unstale":"2024-09-10T18:58:44.110586+0000","last_undegraded":"2024-09-10T18:58:44.110586+0000","last_fullsized":"2024-09-10T18:58:44.110586+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.318782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"197.d","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121076+0000","last_change":"2024-09-10T18:58:32.225543+0000","last_active":"2024-09-10T18:58:44.121076+0000","last_peered":"2024-09-10T18:58:44.121076+0000","last_clean":"2024-09-10T18:58:44.121076+0000","last_became_active":"2024-09-10T18:58:32.225283+0000","last_became_peered":"2024-09-10T18:58:32.225283+0000","last_unstale":"2024-09-10T18:58:44.121076+0000","last_undegraded":"2024-09-10T18:58:44.121076+0000","last_fullsized":"2024-09-10T18:58:44.121076+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.964345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.c","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092407+0000","last_change":"2024-09-10T18:58:32.222309+0000","last_active":"2024-09-10T18:58:44.092407+0000","last_peered":"2024-09-10T18:58:44.092407+0000","last_clean":"2024-09-10T18:58:44.092407+0000","last_became_active":"2024-09-10T18:58:32.222025+0000","last_became_peered":"2024-09-10T18:58:32.222025+0000","last_unstale":"2024-09-10T18:58:44.092407+0000","last_undegraded":"2024-09-10T18:58:44.092407+0000","last_fullsized":"2024-09-10T18:58:44.092407+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.198023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.b","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108578+0000","last_change":"2024-09-10T18:58:32.224764+0000","last_active":"2024-09-10T18:58:44.108578+0000","last_peered":"2024-09-10T18:58:44.108578+0000","last_clean":"2024-09-10T18:58:44.108578+0000","last_became_active":"2024-09-10T18:58:32.224462+0000","last_became_peered":"2024-09-10T18:58:32.224462+0000","last_unstale":"2024-09-10T18:58:44.108578+0000","last_undegraded":"2024-09-10T18:58:44.108578+0000","last_fullsized":"2024-09-10T18:58:44.108578+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.606907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.a","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119171+0000","last_change":"2024-09-10T18:58:32.216536+0000","last_active":"2024-09-10T18:58:44.119171+0000","last_peered":"2024-09-10T18:58:44.119171+0000","last_clean":"2024-09-10T18:58:44.119171+0000","last_became_active":"2024-09-10T18:58:32.216275+0000","last_became_peered":"2024-09-10T18:58:32.216275+0000","last_unstale":"2024-09-10T18:58:44.119171+0000","last_undegraded":"2024-09-10T18:58:44.119171+0000","last_fullsized":"2024-09-10T18:58:44.119171+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:34.565264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.9","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089715+0000","last_change":"2024-09-10T18:58:32.207530+0000","last_active":"2024-09-10T18:58:44.089715+0000","last_peered":"2024-09-10T18:58:44.089715+0000","last_clean":"2024-09-10T18:58:44.089715+0000","last_became_active":"2024-09-10T18:58:32.207198+0000","last_became_peered":"2024-09-10T18:58:32.207198+0000","last_unstale":"2024-09-10T18:58:44.089715+0000","last_undegraded":"2024-09-10T18:58:44.089715+0000","last_fullsized":"2024-09-10T18:58:44.089715+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.090723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"197.8","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092784+0000","last_change":"2024-09-10T18:58:32.209026+0000","last_active":"2024-09-10T18:58:44.092784+0000","last_peered":"2024-09-10T18:58:44.092784+0000","last_clean":"2024-09-10T18:58:44.092784+0000","last_became_active":"2024-09-10T18:58:32.208595+0000","last_became_peered":"2024-09-10T18:58:32.208595+0000","last_unstale":"2024-09-10T18:58:44.092784+0000","last_undegraded":"2024-09-10T18:58:44.092784+0000","last_fullsized":"2024-09-10T18:58:44.092784+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.622621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"197.6","version":"369'11","reported_seq":33,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-10T18:58:45.403366+0000","last_change":"2024-09-10T18:58:32.215028+0000","last_active":"2024-09-10T18:58:45.403366+0000","last_peered":"2024-09-10T18:58:45.403366+0000","last_clean":"2024-09-10T18:58:45.403366+0000","last_became_active":"2024-09-10T18:58:32.214766+0000","last_became_peered":"2024-09-10T18:58:32.214766+0000","last_unstale":"2024-09-10T18:58:45.403366+0000","last_undegraded":"2024-09-10T18:58:45.403366+0000","last_fullsized":"2024-09-10T18:58:45.403366+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:33.796612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.7","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103209+0000","last_change":"2024-09-10T18:58:32.163739+0000","last_active":"2024-09-10T18:58:44.103209+0000","last_peered":"2024-09-10T18:58:44.103209+0000","last_clean":"2024-09-10T18:58:44.103209+0000","last_became_active":"2024-09-10T18:58:32.163106+0000","last_became_peered":"2024-09-10T18:58:32.163106+0000","last_unstale":"2024-09-10T18:58:44.103209+0000","last_undegraded":"2024-09-10T18:58:44.103209+0000","last_fullsized":"2024-09-10T18:58:44.103209+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.530351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.4","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093233+0000","last_change":"2024-09-10T18:58:32.216890+0000","last_active":"2024-09-10T18:58:44.093233+0000","last_peered":"2024-09-10T18:58:44.093233+0000","last_clean":"2024-09-10T18:58:44.093233+0000","last_became_active":"2024-09-10T18:58:32.216619+0000","last_became_peered":"2024-09-10T18:58:32.216619+0000","last_unstale":"2024-09-10T18:58:44.093233+0000","last_undegraded":"2024-09-10T18:58:44.093233+0000","last_fullsized":"2024-09-10T18:58:44.093233+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:38.567833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"197.5","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096461+0000","last_change":"2024-09-10T18:58:32.224875+0000","last_active":"2024-09-10T18:58:44.096461+0000","last_peered":"2024-09-10T18:58:44.096461+0000","last_clean":"2024-09-10T18:58:44.096461+0000","last_became_active":"2024-09-10T18:58:32.224557+0000","last_became_peered":"2024-09-10T18:58:32.224557+0000","last_unstale":"2024-09-10T18:58:44.096461+0000","last_undegraded":"2024-09-10T18:58:44.096461+0000","last_fullsized":"2024-09-10T18:58:44.096461+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:44.368237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"197.2","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101032+0000","last_change":"2024-09-10T18:58:32.209971+0000","last_active":"2024-09-10T18:58:44.101032+0000","last_peered":"2024-09-10T18:58:44.101032+0000","last_clean":"2024-09-10T18:58:44.101032+0000","last_became_active":"2024-09-10T18:58:32.209461+0000","last_became_peered":"2024-09-10T18:58:32.209461+0000","last_unstale":"2024-09-10T18:58:44.101032+0000","last_undegraded":"2024-09-10T18:58:44.101032+0000","last_fullsized":"2024-09-10T18:58:44.101032+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.118720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"197.3","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119647+0000","last_change":"2024-09-10T18:58:32.221041+0000","last_active":"2024-09-10T18:58:44.119647+0000","last_peered":"2024-09-10T18:58:44.119647+0000","last_clean":"2024-09-10T18:58:44.119647+0000","last_became_active":"2024-09-10T18:58:32.220513+0000","last_became_peered":"2024-09-10T18:58:32.220513+0000","last_unstale":"2024-09-10T18:58:44.119647+0000","last_undegraded":"2024-09-10T18:58:44.119647+0000","last_fullsized":"2024-09-10T18:58:44.119647+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.662338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"197.0","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107853+0000","last_change":"2024-09-10T18:58:32.223044+0000","last_active":"2024-09-10T18:58:44.107853+0000","last_peered":"2024-09-10T18:58:44.107853+0000","last_clean":"2024-09-10T18:58:44.107853+0000","last_became_active":"2024-09-10T18:58:32.222717+0000","last_became_peered":"2024-09-10T18:58:32.222717+0000","last_unstale":"2024-09-10T18:58:44.107853+0000","last_undegraded":"2024-09-10T18:58:44.107853+0000","last_fullsized":"2024-09-10T18:58:44.107853+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:47.957427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"197.1","version":"0'0","reported_seq":22,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096299+0000","last_change":"2024-09-10T18:58:32.210403+0000","last_active":"2024-09-10T18:58:44.096299+0000","last_peered":"2024-09-10T18:58:44.096299+0000","last_clean":"2024-09-10T18:58:44.096299+0000","last_became_active":"2024-09-10T18:58:32.210042+0000","last_became_peered":"2024-09-10T18:58:32.210042+0000","last_unstale":"2024-09-10T18:58:44.096299+0000","last_undegraded":"2024-09-10T18:58:44.096299+0000","last_fullsized":"2024-09-10T18:58:44.096299+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":355,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:31.136009+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.136009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.330551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1f","version":"223'39","reported_seq":316,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121382+0000","last_change":"2024-09-10T18:57:54.092083+0000","last_active":"2024-09-10T18:58:44.121382+0000","last_peered":"2024-09-10T18:58:44.121382+0000","last_clean":"2024-09-10T18:58:44.121382+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:58:44.121382+0000","last_undegraded":"2024-09-10T18:58:44.121382+0000","last_fullsized":"2024-09-10T18:58:44.121382+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"318'64","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121183+0000","last_change":"2024-09-10T18:58:39.857881+0000","last_active":"2024-09-10T18:58:44.121183+0000","last_peered":"2024-09-10T18:58:44.121183+0000","last_clean":"2024-09-10T18:58:44.121183+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:58:44.121183+0000","last_undegraded":"2024-09-10T18:58:44.121183+0000","last_fullsized":"2024-09-10T18:58:44.121183+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.880941+0000","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.016940303,"stat_sum":{"num_bytes":40,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"276'22","reported_seq":321,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095550+0000","last_change":"2024-09-10T18:57:54.056252+0000","last_active":"2024-09-10T18:58:44.095550+0000","last_peered":"2024-09-10T18:58:44.095550+0000","last_clean":"2024-09-10T18:58:44.095550+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:58:44.095550+0000","last_undegraded":"2024-09-10T18:58:44.095550+0000","last_fullsized":"2024-09-10T18:58:44.095550+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":334,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108780+0000","last_change":"2024-09-10T18:57:54.097920+0000","last_active":"2024-09-10T18:58:44.108780+0000","last_peered":"2024-09-10T18:58:44.108780+0000","last_clean":"2024-09-10T18:58:44.108780+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:58:44.108780+0000","last_undegraded":"2024-09-10T18:58:44.108780+0000","last_fullsized":"2024-09-10T18:58:44.108780+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:56:43.615986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.615986+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003646903,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":349,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108677+0000","last_change":"2024-09-10T18:57:54.097343+0000","last_active":"2024-09-10T18:58:44.108677+0000","last_peered":"2024-09-10T18:58:44.108677+0000","last_clean":"2024-09-10T18:58:44.108677+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:58:44.108677+0000","last_undegraded":"2024-09-10T18:58:44.108677+0000","last_fullsized":"2024-09-10T18:58:44.108677+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:57:30.615944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.615944+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:46.447187+0000","scrub_duration":0.005614539,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":325,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.086773+0000","last_change":"2024-09-10T18:57:54.056894+0000","last_active":"2024-09-10T18:58:44.086773+0000","last_peered":"2024-09-10T18:58:44.086773+0000","last_clean":"2024-09-10T18:58:44.086773+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:58:44.086773+0000","last_undegraded":"2024-09-10T18:58:44.086773+0000","last_fullsized":"2024-09-10T18:58:44.086773+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:57:35.352719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.352719+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074091110000000003,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"257'22","reported_seq":320,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088487+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T18:58:44.088487+0000","last_peered":"2024-09-10T18:58:44.088487+0000","last_clean":"2024-09-10T18:58:44.088487+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:58:44.088487+0000","last_undegraded":"2024-09-10T18:58:44.088487+0000","last_fullsized":"2024-09-10T18:58:44.088487+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.319797+0000","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"254'104","reported_seq":381,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095538+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T18:58:44.095538+0000","last_peered":"2024-09-10T18:58:44.095538+0000","last_clean":"2024-09-10T18:58:44.095538+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:58:44.095538+0000","last_undegraded":"2024-09-10T18:58:44.095538+0000","last_fullsized":"2024-09-10T18:58:44.095538+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:45.932946+0000","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"230'17","reported_seq":309,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.098789+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T18:58:44.098789+0000","last_peered":"2024-09-10T18:58:44.098789+0000","last_clean":"2024-09-10T18:58:44.098789+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:58:44.098789+0000","last_undegraded":"2024-09-10T18:58:44.098789+0000","last_fullsized":"2024-09-10T18:58:44.098789+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:17.416599+0000","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"276'22","reported_seq":316,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109528+0000","last_change":"2024-09-10T18:57:54.099018+0000","last_active":"2024-09-10T18:58:44.109528+0000","last_peered":"2024-09-10T18:58:44.109528+0000","last_clean":"2024-09-10T18:58:44.109528+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:58:44.109528+0000","last_undegraded":"2024-09-10T18:58:44.109528+0000","last_fullsized":"2024-09-10T18:58:44.109528+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:42.649292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.649292+0000","objects_scrubbed":1,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:48.495881+0000","scrub_duration":0.004377405,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"240'74","reported_seq":358,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.091563+0000","last_change":"2024-09-10T18:57:54.065259+0000","last_active":"2024-09-10T18:58:44.091563+0000","last_peered":"2024-09-10T18:58:44.091563+0000","last_clean":"2024-09-10T18:58:44.091563+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:58:44.091563+0000","last_undegraded":"2024-09-10T18:58:44.091563+0000","last_fullsized":"2024-09-10T18:58:44.091563+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'74","last_scrub_stamp":"2024-09-10T18:57:41.887503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.887503+0000","objects_scrubbed":6,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:47.295134+0000","scrub_duration":0.0039986459999999998,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":52,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"246'95","reported_seq":385,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108855+0000","last_change":"2024-09-10T18:57:54.098423+0000","last_active":"2024-09-10T18:58:44.108855+0000","last_peered":"2024-09-10T18:58:44.108855+0000","last_clean":"2024-09-10T18:58:44.108855+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:58:44.108855+0000","last_undegraded":"2024-09-10T18:58:44.108855+0000","last_fullsized":"2024-09-10T18:58:44.108855+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'95","last_scrub_stamp":"2024-09-10T18:57:25.633366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.633366+0000","objects_scrubbed":6,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044083560000000004,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":29,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"146'1","reported_seq":309,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088391+0000","last_change":"2024-09-10T18:58:43.994005+0000","last_active":"2024-09-10T18:58:44.088391+0000","last_peered":"2024-09-10T18:58:44.088391+0000","last_clean":"2024-09-10T18:58:44.088391+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:58:44.088391+0000","last_undegraded":"2024-09-10T18:58:44.088391+0000","last_fullsized":"2024-09-10T18:58:44.088391+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:58:43.993832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.993832+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.350619+0000","scrub_duration":0.0040278960000000004,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":333,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.097373+0000","last_change":"2024-09-10T18:58:39.380635+0000","last_active":"2024-09-10T18:58:44.097373+0000","last_peered":"2024-09-10T18:58:44.097373+0000","last_clean":"2024-09-10T18:58:44.097373+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:58:44.097373+0000","last_undegraded":"2024-09-10T18:58:44.097373+0000","last_fullsized":"2024-09-10T18:58:44.097373+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.953708+0000","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"297'52","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100075+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T18:58:44.100075+0000","last_peered":"2024-09-10T18:58:44.100075+0000","last_clean":"2024-09-10T18:58:44.100075+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:58:44.100075+0000","last_undegraded":"2024-09-10T18:58:44.100075+0000","last_fullsized":"2024-09-10T18:58:44.100075+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:04.071589+0000","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"224'34","reported_seq":323,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089404+0000","last_change":"2024-09-10T18:58:37.312137+0000","last_active":"2024-09-10T18:58:44.089404+0000","last_peered":"2024-09-10T18:58:44.089404+0000","last_clean":"2024-09-10T18:58:44.089404+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:58:44.089404+0000","last_undegraded":"2024-09-10T18:58:44.089404+0000","last_fullsized":"2024-09-10T18:58:44.089404+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.542659+0000","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"326'68","reported_seq":358,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.111450+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T18:58:44.111450+0000","last_peered":"2024-09-10T18:58:44.111450+0000","last_clean":"2024-09-10T18:58:44.111450+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:58:44.111450+0000","last_undegraded":"2024-09-10T18:58:44.111450+0000","last_fullsized":"2024-09-10T18:58:44.111450+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.262512+0000","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"246'119","reported_seq":394,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095112+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T18:58:44.095112+0000","last_peered":"2024-09-10T18:58:44.095112+0000","last_clean":"2024-09-10T18:58:44.095112+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:58:44.095112+0000","last_undegraded":"2024-09-10T18:58:44.095112+0000","last_fullsized":"2024-09-10T18:58:44.095112+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.422085+0000","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"243'18","reported_seq":316,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.102188+0000","last_change":"2024-09-10T18:57:54.066253+0000","last_active":"2024-09-10T18:58:44.102188+0000","last_peered":"2024-09-10T18:58:44.102188+0000","last_clean":"2024-09-10T18:58:44.102188+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:58:44.102188+0000","last_undegraded":"2024-09-10T18:58:44.102188+0000","last_fullsized":"2024-09-10T18:58:44.102188+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'18","last_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:56:39.950183+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061437230000000002,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"252'22","reported_seq":309,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101306+0000","last_change":"2024-09-10T18:57:54.072778+0000","last_active":"2024-09-10T18:58:44.101306+0000","last_peered":"2024-09-10T18:58:44.101306+0000","last_clean":"2024-09-10T18:58:44.101306+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:58:44.101306+0000","last_undegraded":"2024-09-10T18:58:44.101306+0000","last_fullsized":"2024-09-10T18:58:44.101306+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:54:55.359884+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":326,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108763+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T18:58:44.108763+0000","last_peered":"2024-09-10T18:58:44.108763+0000","last_clean":"2024-09-10T18:58:44.108763+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:58:44.108763+0000","last_undegraded":"2024-09-10T18:58:44.108763+0000","last_fullsized":"2024-09-10T18:58:44.108763+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.729128+0000","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"232'17","reported_seq":324,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087862+0000","last_change":"2024-09-10T18:58:37.063649+0000","last_active":"2024-09-10T18:58:44.087862+0000","last_peered":"2024-09-10T18:58:44.087862+0000","last_clean":"2024-09-10T18:58:44.087862+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:58:44.087862+0000","last_undegraded":"2024-09-10T18:58:44.087862+0000","last_fullsized":"2024-09-10T18:58:44.087862+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:58:37.063474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.063474+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.880086+0000","scrub_duration":0.003955554,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":297,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087646+0000","last_change":"2024-09-10T18:57:54.063413+0000","last_active":"2024-09-10T18:58:44.087646+0000","last_peered":"2024-09-10T18:58:44.087646+0000","last_clean":"2024-09-10T18:58:44.087646+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:58:44.087646+0000","last_undegraded":"2024-09-10T18:58:44.087646+0000","last_fullsized":"2024-09-10T18:58:44.087646+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.800250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.800250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:50.514815+0000","scrub_duration":0.0031899250000000001,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":340,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092690+0000","last_change":"2024-09-10T18:57:54.066122+0000","last_active":"2024-09-10T18:58:44.092690+0000","last_peered":"2024-09-10T18:58:44.092690+0000","last_clean":"2024-09-10T18:58:44.092690+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:58:44.092690+0000","last_undegraded":"2024-09-10T18:58:44.092690+0000","last_fullsized":"2024-09-10T18:58:44.092690+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T18:57:35.624392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.624392+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004362795,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"276'158","reported_seq":438,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092088+0000","last_change":"2024-09-10T18:57:54.068278+0000","last_active":"2024-09-10T18:58:44.092088+0000","last_peered":"2024-09-10T18:58:44.092088+0000","last_clean":"2024-09-10T18:58:44.092088+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:58:44.092088+0000","last_undegraded":"2024-09-10T18:58:44.092088+0000","last_fullsized":"2024-09-10T18:58:44.092088+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'158","last_scrub_stamp":"2024-09-10T18:57:37.839200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.839200+0000","objects_scrubbed":10,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057174490000000003,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":47,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"290'93","reported_seq":427,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088700+0000","last_change":"2024-09-10T18:57:54.058056+0000","last_active":"2024-09-10T18:58:44.088700+0000","last_peered":"2024-09-10T18:58:44.088700+0000","last_clean":"2024-09-10T18:58:44.088700+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:58:44.088700+0000","last_undegraded":"2024-09-10T18:58:44.088700+0000","last_fullsized":"2024-09-10T18:58:44.088700+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'93","last_scrub_stamp":"2024-09-10T18:57:29.481888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:29.481888+0000","objects_scrubbed":6,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004612952,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":343,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.436020+0000","last_change":"2024-09-10T18:58:44.435895+0000","last_active":"2024-09-10T18:58:44.436020+0000","last_peered":"2024-09-10T18:58:44.436020+0000","last_clean":"2024-09-10T18:58:44.436020+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:58:44.436020+0000","last_undegraded":"2024-09-10T18:58:44.436020+0000","last_fullsized":"2024-09-10T18:58:44.436020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T18:58:44.435728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.435728+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.065189+0000","scrub_duration":0.0045739450000000003,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"257'107","reported_seq":388,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088337+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T18:58:44.088337+0000","last_peered":"2024-09-10T18:58:44.088337+0000","last_clean":"2024-09-10T18:58:44.088337+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:58:44.088337+0000","last_undegraded":"2024-09-10T18:58:44.088337+0000","last_fullsized":"2024-09-10T18:58:44.088337+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:53.441046+0000","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"236'102","reported_seq":384,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092076+0000","last_change":"2024-09-10T18:58:34.265530+0000","last_active":"2024-09-10T18:58:44.092076+0000","last_peered":"2024-09-10T18:58:44.092076+0000","last_clean":"2024-09-10T18:58:44.092076+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:58:44.092076+0000","last_undegraded":"2024-09-10T18:58:44.092076+0000","last_fullsized":"2024-09-10T18:58:44.092076+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:50.420507+0000","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":322,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093663+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T18:58:44.093663+0000","last_peered":"2024-09-10T18:58:44.093663+0000","last_clean":"2024-09-10T18:58:44.093663+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:58:44.093663+0000","last_undegraded":"2024-09-10T18:58:44.093663+0000","last_fullsized":"2024-09-10T18:58:44.093663+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.440202+0000","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"263'72","reported_seq":366,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.097140+0000","last_change":"2024-09-10T18:58:32.368725+0000","last_active":"2024-09-10T18:58:44.097140+0000","last_peered":"2024-09-10T18:58:44.097140+0000","last_clean":"2024-09-10T18:58:44.097140+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:58:44.097140+0000","last_undegraded":"2024-09-10T18:58:44.097140+0000","last_fullsized":"2024-09-10T18:58:44.097140+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'72","last_scrub_stamp":"2024-09-10T18:58:32.368604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.368604+0000","objects_scrubbed":7,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:57.712897+0000","scrub_duration":0.0042920229999999998,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":389,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087792+0000","last_change":"2024-09-10T18:58:36.017962+0000","last_active":"2024-09-10T18:58:44.087792+0000","last_peered":"2024-09-10T18:58:44.087792+0000","last_clean":"2024-09-10T18:58:44.087792+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:58:44.087792+0000","last_undegraded":"2024-09-10T18:58:44.087792+0000","last_fullsized":"2024-09-10T18:58:44.087792+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:58:36.017841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.017841+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.872443+0000","scrub_duration":0.0042543119999999997,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":332,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.091755+0000","last_change":"2024-09-10T18:56:10.722647+0000","last_active":"2024-09-10T18:58:44.091755+0000","last_peered":"2024-09-10T18:58:44.091755+0000","last_clean":"2024-09-10T18:58:44.091755+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:58:44.091755+0000","last_undegraded":"2024-09-10T18:58:44.091755+0000","last_fullsized":"2024-09-10T18:58:44.091755+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.722510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.722510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036579450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":346,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121903+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:58:44.121903+0000","last_peered":"2024-09-10T18:58:44.121903+0000","last_clean":"2024-09-10T18:58:44.121903+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:58:44.121903+0000","last_undegraded":"2024-09-10T18:58:44.121903+0000","last_fullsized":"2024-09-10T18:58:44.121903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":351,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094505+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:58:44.094505+0000","last_peered":"2024-09-10T18:58:44.094505+0000","last_clean":"2024-09-10T18:58:44.094505+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:58:44.094505+0000","last_undegraded":"2024-09-10T18:58:44.094505+0000","last_fullsized":"2024-09-10T18:58:44.094505+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089371+0000","last_change":"2024-09-10T18:56:30.915545+0000","last_active":"2024-09-10T18:58:44.089371+0000","last_peered":"2024-09-10T18:58:44.089371+0000","last_clean":"2024-09-10T18:58:44.089371+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:58:44.089371+0000","last_undegraded":"2024-09-10T18:58:44.089371+0000","last_fullsized":"2024-09-10T18:58:44.089371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.915384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.915384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002820814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096739+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:58:44.096739+0000","last_peered":"2024-09-10T18:58:44.096739+0000","last_clean":"2024-09-10T18:58:44.096739+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:58:44.096739+0000","last_undegraded":"2024-09-10T18:58:44.096739+0000","last_fullsized":"2024-09-10T18:58:44.096739+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":358,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094896+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T18:58:44.094896+0000","last_peered":"2024-09-10T18:58:44.094896+0000","last_clean":"2024-09-10T18:58:44.094896+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:58:44.094896+0000","last_undegraded":"2024-09-10T18:58:44.094896+0000","last_fullsized":"2024-09-10T18:58:44.094896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.048099+0000","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":363,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121585+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:58:44.121585+0000","last_peered":"2024-09-10T18:58:44.121585+0000","last_clean":"2024-09-10T18:58:44.121585+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:58:44.121585+0000","last_undegraded":"2024-09-10T18:58:44.121585+0000","last_fullsized":"2024-09-10T18:58:44.121585+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099674+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:58:44.099674+0000","last_peered":"2024-09-10T18:58:44.099674+0000","last_clean":"2024-09-10T18:58:44.099674+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:58:44.099674+0000","last_undegraded":"2024-09-10T18:58:44.099674+0000","last_fullsized":"2024-09-10T18:58:44.099674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100579+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T18:58:44.100579+0000","last_peered":"2024-09-10T18:58:44.100579+0000","last_clean":"2024-09-10T18:58:44.100579+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:58:44.100579+0000","last_undegraded":"2024-09-10T18:58:44.100579+0000","last_fullsized":"2024-09-10T18:58:44.100579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:46.825797+0000","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119242+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T18:58:44.119242+0000","last_peered":"2024-09-10T18:58:44.119242+0000","last_clean":"2024-09-10T18:58:44.119242+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:58:44.119242+0000","last_undegraded":"2024-09-10T18:58:44.119242+0000","last_fullsized":"2024-09-10T18:58:44.119242+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095695+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T18:58:44.095695+0000","last_peered":"2024-09-10T18:58:44.095695+0000","last_clean":"2024-09-10T18:58:44.095695+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:58:44.095695+0000","last_undegraded":"2024-09-10T18:58:44.095695+0000","last_fullsized":"2024-09-10T18:58:44.095695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.053990+0000","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":374,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101650+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T18:58:44.101650+0000","last_peered":"2024-09-10T18:58:44.101650+0000","last_clean":"2024-09-10T18:58:44.101650+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:58:44.101650+0000","last_undegraded":"2024-09-10T18:58:44.101650+0000","last_fullsized":"2024-09-10T18:58:44.101650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101249+0000","last_change":"2024-09-10T18:56:17.929975+0000","last_active":"2024-09-10T18:58:44.101249+0000","last_peered":"2024-09-10T18:58:44.101249+0000","last_clean":"2024-09-10T18:58:44.101249+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:58:44.101249+0000","last_undegraded":"2024-09-10T18:58:44.101249+0000","last_fullsized":"2024-09-10T18:58:44.101249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.929825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.929825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033803790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":341,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.097035+0000","last_change":"2024-09-10T18:57:31.277231+0000","last_active":"2024-09-10T18:58:44.097035+0000","last_peered":"2024-09-10T18:58:44.097035+0000","last_clean":"2024-09-10T18:58:44.097035+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:58:44.097035+0000","last_undegraded":"2024-09-10T18:58:44.097035+0000","last_fullsized":"2024-09-10T18:58:44.097035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.277085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.277085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003344746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":271,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108960+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T18:58:44.108960+0000","last_peered":"2024-09-10T18:58:44.108960+0000","last_clean":"2024-09-10T18:58:44.108960+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:58:44.108960+0000","last_undegraded":"2024-09-10T18:58:44.108960+0000","last_fullsized":"2024-09-10T18:58:44.108960+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:59.115727+0000","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":347,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088723+0000","last_change":"2024-09-10T18:57:25.517299+0000","last_active":"2024-09-10T18:58:44.088723+0000","last_peered":"2024-09-10T18:58:44.088723+0000","last_clean":"2024-09-10T18:58:44.088723+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:58:44.088723+0000","last_undegraded":"2024-09-10T18:58:44.088723+0000","last_fullsized":"2024-09-10T18:58:44.088723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:25.517210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:25.517210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027489659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":380,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110474+0000","last_change":"2024-09-10T18:56:36.272399+0000","last_active":"2024-09-10T18:58:44.110474+0000","last_peered":"2024-09-10T18:58:44.110474+0000","last_clean":"2024-09-10T18:58:44.110474+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:58:44.110474+0000","last_undegraded":"2024-09-10T18:58:44.110474+0000","last_fullsized":"2024-09-10T18:58:44.110474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:56:36.272235+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.272235+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046176539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100920+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T18:58:44.100920+0000","last_peered":"2024-09-10T18:58:44.100920+0000","last_clean":"2024-09-10T18:58:44.100920+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:58:44.100920+0000","last_undegraded":"2024-09-10T18:58:44.100920+0000","last_fullsized":"2024-09-10T18:58:44.100920+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.804733+0000","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":340,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.112853+0000","last_change":"2024-09-10T18:55:00.912262+0000","last_active":"2024-09-10T18:58:44.112853+0000","last_peered":"2024-09-10T18:58:44.112853+0000","last_clean":"2024-09-10T18:58:44.112853+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:58:44.112853+0000","last_undegraded":"2024-09-10T18:58:44.112853+0000","last_fullsized":"2024-09-10T18:58:44.112853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:00.912161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:55:00.912161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021549210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.090479+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:58:44.090479+0000","last_peered":"2024-09-10T18:58:44.090479+0000","last_clean":"2024-09-10T18:58:44.090479+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:58:44.090479+0000","last_undegraded":"2024-09-10T18:58:44.090479+0000","last_fullsized":"2024-09-10T18:58:44.090479+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":384,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093825+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T18:58:44.093825+0000","last_peered":"2024-09-10T18:58:44.093825+0000","last_clean":"2024-09-10T18:58:44.093825+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:58:44.093825+0000","last_undegraded":"2024-09-10T18:58:44.093825+0000","last_fullsized":"2024-09-10T18:58:44.093825+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.716139+0000","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088295+0000","last_change":"2024-09-10T18:57:34.379150+0000","last_active":"2024-09-10T18:58:44.088295+0000","last_peered":"2024-09-10T18:58:44.088295+0000","last_clean":"2024-09-10T18:58:44.088295+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:58:44.088295+0000","last_undegraded":"2024-09-10T18:58:44.088295+0000","last_fullsized":"2024-09-10T18:58:44.088295+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.378985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.378985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00364804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":478,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.102437+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T18:58:44.102437+0000","last_peered":"2024-09-10T18:58:44.102437+0000","last_clean":"2024-09-10T18:58:44.102437+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:58:44.102437+0000","last_undegraded":"2024-09-10T18:58:44.102437+0000","last_fullsized":"2024-09-10T18:58:44.102437+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.009195+0000","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092347+0000","last_change":"2024-09-10T18:57:36.840129+0000","last_active":"2024-09-10T18:58:44.092347+0000","last_peered":"2024-09-10T18:58:44.092347+0000","last_clean":"2024-09-10T18:58:44.092347+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:58:44.092347+0000","last_undegraded":"2024-09-10T18:58:44.092347+0000","last_fullsized":"2024-09-10T18:58:44.092347+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.839979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.839979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034210669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095273+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:58:44.095273+0000","last_peered":"2024-09-10T18:58:44.095273+0000","last_clean":"2024-09-10T18:58:44.095273+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:58:44.095273+0000","last_undegraded":"2024-09-10T18:58:44.095273+0000","last_fullsized":"2024-09-10T18:58:44.095273+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":366,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092181+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:58:44.092181+0000","last_peered":"2024-09-10T18:58:44.092181+0000","last_clean":"2024-09-10T18:58:44.092181+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:58:44.092181+0000","last_undegraded":"2024-09-10T18:58:44.092181+0000","last_fullsized":"2024-09-10T18:58:44.092181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108294+0000","last_change":"2024-09-10T18:56:28.694362+0000","last_active":"2024-09-10T18:58:44.108294+0000","last_peered":"2024-09-10T18:58:44.108294+0000","last_clean":"2024-09-10T18:58:44.108294+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:58:44.108294+0000","last_undegraded":"2024-09-10T18:58:44.108294+0000","last_fullsized":"2024-09-10T18:58:44.108294+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:28.694238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.492707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.694238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003238482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101857+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:58:44.101857+0000","last_peered":"2024-09-10T18:58:44.101857+0000","last_clean":"2024-09-10T18:58:44.101857+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:58:44.101857+0000","last_undegraded":"2024-09-10T18:58:44.101857+0000","last_fullsized":"2024-09-10T18:58:44.101857+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103172+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T18:58:44.103172+0000","last_peered":"2024-09-10T18:58:44.103172+0000","last_clean":"2024-09-10T18:58:44.103172+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:58:44.103172+0000","last_undegraded":"2024-09-10T18:58:44.103172+0000","last_fullsized":"2024-09-10T18:58:44.103172+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.012955+0000","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093361+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T18:58:44.093361+0000","last_peered":"2024-09-10T18:58:44.093361+0000","last_clean":"2024-09-10T18:58:44.093361+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:58:44.093361+0000","last_undegraded":"2024-09-10T18:58:44.093361+0000","last_fullsized":"2024-09-10T18:58:44.093361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.592874+0000","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":371,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.090192+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:58:44.090192+0000","last_peered":"2024-09-10T18:58:44.090192+0000","last_clean":"2024-09-10T18:58:44.090192+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:58:44.090192+0000","last_undegraded":"2024-09-10T18:58:44.090192+0000","last_fullsized":"2024-09-10T18:58:44.090192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":345,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107896+0000","last_change":"2024-09-10T18:56:31.692428+0000","last_active":"2024-09-10T18:58:44.107896+0000","last_peered":"2024-09-10T18:58:44.107896+0000","last_clean":"2024-09-10T18:58:44.107896+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:58:44.107896+0000","last_undegraded":"2024-09-10T18:58:44.107896+0000","last_fullsized":"2024-09-10T18:58:44.107896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.692339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.692339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026691089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":361,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110072+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T18:58:44.110072+0000","last_peered":"2024-09-10T18:58:44.110072+0000","last_clean":"2024-09-10T18:58:44.110072+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:58:44.110072+0000","last_undegraded":"2024-09-10T18:58:44.110072+0000","last_fullsized":"2024-09-10T18:58:44.110072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.611116+0000","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.106028+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:58:44.106028+0000","last_peered":"2024-09-10T18:58:44.106028+0000","last_clean":"2024-09-10T18:58:44.106028+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:58:44.106028+0000","last_undegraded":"2024-09-10T18:58:44.106028+0000","last_fullsized":"2024-09-10T18:58:44.106028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":271,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110075+0000","last_change":"2024-09-10T18:57:56.827061+0000","last_active":"2024-09-10T18:58:44.110075+0000","last_peered":"2024-09-10T18:58:44.110075+0000","last_clean":"2024-09-10T18:58:44.110075+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:58:44.110075+0000","last_undegraded":"2024-09-10T18:58:44.110075+0000","last_fullsized":"2024-09-10T18:58:44.110075+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.826954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.826954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.327774+0000","scrub_duration":0.0032518439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":384,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100899+0000","last_change":"2024-09-10T18:56:44.980434+0000","last_active":"2024-09-10T18:58:44.100899+0000","last_peered":"2024-09-10T18:58:44.100899+0000","last_clean":"2024-09-10T18:58:44.100899+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:58:44.100899+0000","last_undegraded":"2024-09-10T18:58:44.100899+0000","last_fullsized":"2024-09-10T18:58:44.100899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:56:44.980260+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:50:20.372707+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.980260+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051507660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.085965+0000","last_change":"2024-09-10T18:58:32.226883+0000","last_active":"2024-09-10T18:58:44.085965+0000","last_peered":"2024-09-10T18:58:44.085965+0000","last_clean":"2024-09-10T18:58:44.085965+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:58:44.085965+0000","last_undegraded":"2024-09-10T18:58:44.085965+0000","last_fullsized":"2024-09-10T18:58:44.085965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.226771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.226771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.723944+0000","scrub_duration":0.024188028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":478,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103511+0000","last_change":"2024-09-10T18:57:32.764761+0000","last_active":"2024-09-10T18:58:44.103511+0000","last_peered":"2024-09-10T18:58:44.103511+0000","last_clean":"2024-09-10T18:58:44.103511+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:58:44.103511+0000","last_undegraded":"2024-09-10T18:58:44.103511+0000","last_fullsized":"2024-09-10T18:58:44.103511+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:57:32.764561+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.764561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041273020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":343,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092258+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:58:44.092258+0000","last_peered":"2024-09-10T18:58:44.092258+0000","last_clean":"2024-09-10T18:58:44.092258+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:58:44.092258+0000","last_undegraded":"2024-09-10T18:58:44.092258+0000","last_fullsized":"2024-09-10T18:58:44.092258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109676+0000","last_change":"2024-09-10T18:56:34.658727+0000","last_active":"2024-09-10T18:58:44.109676+0000","last_peered":"2024-09-10T18:58:44.109676+0000","last_clean":"2024-09-10T18:58:44.109676+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:58:44.109676+0000","last_undegraded":"2024-09-10T18:58:44.109676+0000","last_fullsized":"2024-09-10T18:58:44.109676+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.658578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.658578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035512220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":362,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121637+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T18:58:44.121637+0000","last_peered":"2024-09-10T18:58:44.121637+0000","last_clean":"2024-09-10T18:58:44.121637+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:58:44.121637+0000","last_undegraded":"2024-09-10T18:58:44.121637+0000","last_fullsized":"2024-09-10T18:58:44.121637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.981000+0000","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":345,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099331+0000","last_change":"2024-09-10T18:55:01.608966+0000","last_active":"2024-09-10T18:58:44.099331+0000","last_peered":"2024-09-10T18:58:44.099331+0000","last_clean":"2024-09-10T18:58:44.099331+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:58:44.099331+0000","last_undegraded":"2024-09-10T18:58:44.099331+0000","last_fullsized":"2024-09-10T18:58:44.099331+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:01.608837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:55:01.608837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029996240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100512+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T18:58:44.100512+0000","last_peered":"2024-09-10T18:58:44.100512+0000","last_clean":"2024-09-10T18:58:44.100512+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:58:44.100512+0000","last_undegraded":"2024-09-10T18:58:44.100512+0000","last_fullsized":"2024-09-10T18:58:44.100512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":904,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.878387+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T18:58:44.878387+0000","last_peered":"2024-09-10T18:58:44.878387+0000","last_clean":"2024-09-10T18:58:44.878387+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:58:44.878387+0000","last_undegraded":"2024-09-10T18:58:44.878387+0000","last_fullsized":"2024-09-10T18:58:44.878387+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.772656+0000","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":542,"num_read_kb":537,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087448+0000","last_change":"2024-09-10T18:57:49.386576+0000","last_active":"2024-09-10T18:58:44.087448+0000","last_peered":"2024-09-10T18:58:44.087448+0000","last_clean":"2024-09-10T18:58:44.087448+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:58:44.087448+0000","last_undegraded":"2024-09-10T18:58:44.087448+0000","last_fullsized":"2024-09-10T18:58:44.087448+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.386404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.605455+0000","scrub_duration":0.003967628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":272,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087541+0000","last_change":"2024-09-10T18:57:49.919967+0000","last_active":"2024-09-10T18:58:44.087541+0000","last_peered":"2024-09-10T18:58:44.087541+0000","last_clean":"2024-09-10T18:58:44.087541+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:58:44.087541+0000","last_undegraded":"2024-09-10T18:58:44.087541+0000","last_fullsized":"2024-09-10T18:58:44.087541+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.919794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.919794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.849294+0000","scrub_duration":0.0034551590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":373,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095258+0000","last_change":"2024-09-10T18:56:22.945334+0000","last_active":"2024-09-10T18:58:44.095258+0000","last_peered":"2024-09-10T18:58:44.095258+0000","last_clean":"2024-09-10T18:58:44.095258+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:58:44.095258+0000","last_undegraded":"2024-09-10T18:58:44.095258+0000","last_fullsized":"2024-09-10T18:58:44.095258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:56:22.945181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:22.945181+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004257416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109897+0000","last_change":"2024-09-10T18:56:38.595582+0000","last_active":"2024-09-10T18:58:44.109897+0000","last_peered":"2024-09-10T18:58:44.109897+0000","last_clean":"2024-09-10T18:58:44.109897+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:58:44.109897+0000","last_undegraded":"2024-09-10T18:58:44.109897+0000","last_fullsized":"2024-09-10T18:58:44.109897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.595440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.595440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002497249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":457,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089922+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:58:44.089922+0000","last_peered":"2024-09-10T18:58:44.089922+0000","last_clean":"2024-09-10T18:58:44.089922+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:58:44.089922+0000","last_undegraded":"2024-09-10T18:58:44.089922+0000","last_fullsized":"2024-09-10T18:58:44.089922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":361,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100239+0000","last_change":"2024-09-10T18:56:33.805943+0000","last_active":"2024-09-10T18:58:44.100239+0000","last_peered":"2024-09-10T18:58:44.100239+0000","last_clean":"2024-09-10T18:58:44.100239+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:58:44.100239+0000","last_undegraded":"2024-09-10T18:58:44.100239+0000","last_fullsized":"2024-09-10T18:58:44.100239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:56:33.805774+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.805774+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042693669999999996,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100746+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T18:58:44.100746+0000","last_peered":"2024-09-10T18:58:44.100746+0000","last_clean":"2024-09-10T18:58:44.100746+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:58:44.100746+0000","last_undegraded":"2024-09-10T18:58:44.100746+0000","last_fullsized":"2024-09-10T18:58:44.100746+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:19.820859+0000","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089822+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T18:58:44.089822+0000","last_peered":"2024-09-10T18:58:44.089822+0000","last_clean":"2024-09-10T18:58:44.089822+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:58:44.089822+0000","last_undegraded":"2024-09-10T18:58:44.089822+0000","last_fullsized":"2024-09-10T18:58:44.089822+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.921143+0000","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":526,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109203+0000","last_change":"2024-09-10T18:57:50.752463+0000","last_active":"2024-09-10T18:58:44.109203+0000","last_peered":"2024-09-10T18:58:44.109203+0000","last_clean":"2024-09-10T18:58:44.109203+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:58:44.109203+0000","last_undegraded":"2024-09-10T18:58:44.109203+0000","last_fullsized":"2024-09-10T18:58:44.109203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:57:50.752292+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.752292+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:51.294597+0000","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096298+0000","last_change":"2024-09-10T18:56:19.447059+0000","last_active":"2024-09-10T18:58:44.096298+0000","last_peered":"2024-09-10T18:58:44.096298+0000","last_clean":"2024-09-10T18:58:44.096298+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:58:44.096298+0000","last_undegraded":"2024-09-10T18:58:44.096298+0000","last_fullsized":"2024-09-10T18:58:44.096298+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.446994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.446994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022874689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":466,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119259+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:58:44.119259+0000","last_peered":"2024-09-10T18:58:44.119259+0000","last_clean":"2024-09-10T18:58:44.119259+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:58:44.119259+0000","last_undegraded":"2024-09-10T18:58:44.119259+0000","last_fullsized":"2024-09-10T18:58:44.119259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":359,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093716+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T18:58:44.093716+0000","last_peered":"2024-09-10T18:58:44.093716+0000","last_clean":"2024-09-10T18:58:44.093716+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:58:44.093716+0000","last_undegraded":"2024-09-10T18:58:44.093716+0000","last_fullsized":"2024-09-10T18:58:44.093716+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.942357+0000","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":339,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107593+0000","last_change":"2024-09-10T18:56:33.679822+0000","last_active":"2024-09-10T18:58:44.107593+0000","last_peered":"2024-09-10T18:58:44.107593+0000","last_clean":"2024-09-10T18:58:44.107593+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:58:44.107593+0000","last_undegraded":"2024-09-10T18:58:44.107593+0000","last_fullsized":"2024-09-10T18:58:44.107593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.679682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.679682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003099456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092428+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T18:58:44.092428+0000","last_peered":"2024-09-10T18:58:44.092428+0000","last_clean":"2024-09-10T18:58:44.092428+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:58:44.092428+0000","last_undegraded":"2024-09-10T18:58:44.092428+0000","last_fullsized":"2024-09-10T18:58:44.092428+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.040014+0000","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":364,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110521+0000","last_change":"2024-09-10T18:55:07.225010+0000","last_active":"2024-09-10T18:58:44.110521+0000","last_peered":"2024-09-10T18:58:44.110521+0000","last_clean":"2024-09-10T18:58:44.110521+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:58:44.110521+0000","last_undegraded":"2024-09-10T18:58:44.110521+0000","last_fullsized":"2024-09-10T18:58:44.110521+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:55:07.224841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:55:07.224841+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004841845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109285+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T18:58:44.109285+0000","last_peered":"2024-09-10T18:58:44.109285+0000","last_clean":"2024-09-10T18:58:44.109285+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:58:44.109285+0000","last_undegraded":"2024-09-10T18:58:44.109285+0000","last_fullsized":"2024-09-10T18:58:44.109285+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.994362+0000","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":351,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108050+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T18:58:44.108050+0000","last_peered":"2024-09-10T18:58:44.108050+0000","last_clean":"2024-09-10T18:58:44.108050+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:58:44.108050+0000","last_undegraded":"2024-09-10T18:58:44.108050+0000","last_fullsized":"2024-09-10T18:58:44.108050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.143473+0000","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":340,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113197+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:58:44.113197+0000","last_peered":"2024-09-10T18:58:44.113197+0000","last_clean":"2024-09-10T18:58:44.113197+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:58:44.113197+0000","last_undegraded":"2024-09-10T18:58:44.113197+0000","last_fullsized":"2024-09-10T18:58:44.113197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":272,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087964+0000","last_change":"2024-09-10T18:57:43.914840+0000","last_active":"2024-09-10T18:58:44.087964+0000","last_peered":"2024-09-10T18:58:44.087964+0000","last_clean":"2024-09-10T18:58:44.087964+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:58:44.087964+0000","last_undegraded":"2024-09-10T18:58:44.087964+0000","last_fullsized":"2024-09-10T18:58:44.087964+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:43.914731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:43.914731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.644461+0000","scrub_duration":0.0026766530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":392,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101442+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T18:58:44.101442+0000","last_peered":"2024-09-10T18:58:44.101442+0000","last_clean":"2024-09-10T18:58:44.101442+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:58:44.101442+0000","last_undegraded":"2024-09-10T18:58:44.101442+0000","last_fullsized":"2024-09-10T18:58:44.101442+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:08.776283+0000","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.098264+0000","last_change":"2024-09-10T18:55:08.199611+0000","last_active":"2024-09-10T18:58:44.098264+0000","last_peered":"2024-09-10T18:58:44.098264+0000","last_clean":"2024-09-10T18:58:44.098264+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:58:44.098264+0000","last_undegraded":"2024-09-10T18:58:44.098264+0000","last_fullsized":"2024-09-10T18:58:44.098264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:55:08.199434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034980150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":351,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109109+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T18:58:44.109109+0000","last_peered":"2024-09-10T18:58:44.109109+0000","last_clean":"2024-09-10T18:58:44.109109+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:58:44.109109+0000","last_undegraded":"2024-09-10T18:58:44.109109+0000","last_fullsized":"2024-09-10T18:58:44.109109+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.242393+0000","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.090423+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T18:58:44.090423+0000","last_peered":"2024-09-10T18:58:44.090423+0000","last_clean":"2024-09-10T18:58:44.090423+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:58:44.090423+0000","last_undegraded":"2024-09-10T18:58:44.090423+0000","last_fullsized":"2024-09-10T18:58:44.090423+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.784439+0000","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119723+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:58:44.119723+0000","last_peered":"2024-09-10T18:58:44.119723+0000","last_clean":"2024-09-10T18:58:44.119723+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:58:44.119723+0000","last_undegraded":"2024-09-10T18:58:44.119723+0000","last_fullsized":"2024-09-10T18:58:44.119723+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":393,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107768+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T18:58:44.107768+0000","last_peered":"2024-09-10T18:58:44.107768+0000","last_clean":"2024-09-10T18:58:44.107768+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:58:44.107768+0000","last_undegraded":"2024-09-10T18:58:44.107768+0000","last_fullsized":"2024-09-10T18:58:44.107768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:05.333581+0000","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":366,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.087374+0000","last_change":"2024-09-10T18:57:55.986832+0000","last_active":"2024-09-10T18:58:44.087374+0000","last_peered":"2024-09-10T18:58:44.087374+0000","last_clean":"2024-09-10T18:58:44.087374+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:58:44.087374+0000","last_undegraded":"2024-09-10T18:58:44.087374+0000","last_fullsized":"2024-09-10T18:58:44.087374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:55.986726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.986726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:23.114093+0000","scrub_duration":0.0034130330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":351,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110156+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T18:58:44.110156+0000","last_peered":"2024-09-10T18:58:44.110156+0000","last_clean":"2024-09-10T18:58:44.110156+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:58:44.110156+0000","last_undegraded":"2024-09-10T18:58:44.110156+0000","last_fullsized":"2024-09-10T18:58:44.110156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.829268+0000","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095854+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T18:58:44.095854+0000","last_peered":"2024-09-10T18:58:44.095854+0000","last_clean":"2024-09-10T18:58:44.095854+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:58:44.095854+0000","last_undegraded":"2024-09-10T18:58:44.095854+0000","last_fullsized":"2024-09-10T18:58:44.095854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.701730+0000","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.098958+0000","last_change":"2024-09-10T18:56:19.766995+0000","last_active":"2024-09-10T18:58:44.098958+0000","last_peered":"2024-09-10T18:58:44.098958+0000","last_clean":"2024-09-10T18:58:44.098958+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:58:44.098958+0000","last_undegraded":"2024-09-10T18:58:44.098958+0000","last_fullsized":"2024-09-10T18:58:44.098958+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:19.766841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.766841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034891050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":365,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113269+0000","last_change":"2024-09-10T18:54:57.933802+0000","last_active":"2024-09-10T18:58:44.113269+0000","last_peered":"2024-09-10T18:58:44.113269+0000","last_clean":"2024-09-10T18:58:44.113269+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:58:44.113269+0000","last_undegraded":"2024-09-10T18:58:44.113269+0000","last_fullsized":"2024-09-10T18:58:44.113269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:54:57.933672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:54:57.933672+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052588060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.193150+0000","last_change":"2024-09-10T18:58:44.193043+0000","last_active":"2024-09-10T18:58:44.193150+0000","last_peered":"2024-09-10T18:58:44.193150+0000","last_clean":"2024-09-10T18:58:44.193150+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:58:44.193150+0000","last_undegraded":"2024-09-10T18:58:44.193150+0000","last_fullsized":"2024-09-10T18:58:44.193150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.192853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.192853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.060471+0000","scrub_duration":0.0038691179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121659+0000","last_change":"2024-09-10T18:54:56.899471+0000","last_active":"2024-09-10T18:58:44.121659+0000","last_peered":"2024-09-10T18:58:44.121659+0000","last_clean":"2024-09-10T18:58:44.121659+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:58:44.121659+0000","last_undegraded":"2024-09-10T18:58:44.121659+0000","last_fullsized":"2024-09-10T18:58:44.121659+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:56.899388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:56.899388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044077070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":349,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.108963+0000","last_change":"2024-09-10T18:57:44.612113+0000","last_active":"2024-09-10T18:58:44.108963+0000","last_peered":"2024-09-10T18:58:44.108963+0000","last_clean":"2024-09-10T18:58:44.108963+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:58:44.108963+0000","last_undegraded":"2024-09-10T18:58:44.108963+0000","last_fullsized":"2024-09-10T18:58:44.108963+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:44.612043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:57:44.612043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:54.176921+0000","scrub_duration":0.0026996279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121741+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:58:44.121741+0000","last_peered":"2024-09-10T18:58:44.121741+0000","last_clean":"2024-09-10T18:58:44.121741+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:58:44.121741+0000","last_undegraded":"2024-09-10T18:58:44.121741+0000","last_fullsized":"2024-09-10T18:58:44.121741+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":356,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.090656+0000","last_change":"2024-09-10T18:56:19.931557+0000","last_active":"2024-09-10T18:58:44.090656+0000","last_peered":"2024-09-10T18:58:44.090656+0000","last_clean":"2024-09-10T18:58:44.090656+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:58:44.090656+0000","last_undegraded":"2024-09-10T18:58:44.090656+0000","last_fullsized":"2024-09-10T18:58:44.090656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:56:19.931418+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:56:19.931418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044625940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099999+0000","last_change":"2024-09-10T18:57:56.125998+0000","last_active":"2024-09-10T18:58:44.099999+0000","last_peered":"2024-09-10T18:58:44.099999+0000","last_clean":"2024-09-10T18:58:44.099999+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:58:44.099999+0000","last_undegraded":"2024-09-10T18:58:44.099999+0000","last_fullsized":"2024-09-10T18:58:44.099999+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.125881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.125881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.508448+0000","scrub_duration":0.0026334969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":402,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.085498+0000","last_change":"2024-09-10T18:57:56.375088+0000","last_active":"2024-09-10T18:58:44.085498+0000","last_peered":"2024-09-10T18:58:44.085498+0000","last_clean":"2024-09-10T18:58:44.085498+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:58:44.085498+0000","last_undegraded":"2024-09-10T18:58:44.085498+0000","last_fullsized":"2024-09-10T18:58:44.085498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:57:56.374923+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.374923+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:58.365310+0000","scrub_duration":0.0043604359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":341,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095530+0000","last_change":"2024-09-10T18:55:04.579191+0000","last_active":"2024-09-10T18:58:44.095530+0000","last_peered":"2024-09-10T18:58:44.095530+0000","last_clean":"2024-09-10T18:58:44.095530+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:58:44.095530+0000","last_undegraded":"2024-09-10T18:58:44.095530+0000","last_fullsized":"2024-09-10T18:58:44.095530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:04.579043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:55:04.579043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033398640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":338,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110815+0000","last_change":"2024-09-10T18:56:09.384252+0000","last_active":"2024-09-10T18:58:44.110815+0000","last_peered":"2024-09-10T18:58:44.110815+0000","last_clean":"2024-09-10T18:58:44.110815+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:58:44.110815+0000","last_undegraded":"2024-09-10T18:58:44.110815+0000","last_fullsized":"2024-09-10T18:58:44.110815+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:09.384188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:09.384188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016707029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096141+0000","last_change":"2024-09-10T18:56:30.496188+0000","last_active":"2024-09-10T18:58:44.096141+0000","last_peered":"2024-09-10T18:58:44.096141+0000","last_clean":"2024-09-10T18:58:44.096141+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:58:44.096141+0000","last_undegraded":"2024-09-10T18:58:44.096141+0000","last_fullsized":"2024-09-10T18:58:44.096141+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:56:30.496126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020101300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"77'17","reported_seq":370,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096223+0000","last_change":"2024-09-10T18:56:36.583984+0000","last_active":"2024-09-10T18:58:44.096223+0000","last_peered":"2024-09-10T18:58:44.096223+0000","last_clean":"2024-09-10T18:58:44.096223+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:58:44.096223+0000","last_undegraded":"2024-09-10T18:58:44.096223+0000","last_fullsized":"2024-09-10T18:58:44.096223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'17","last_scrub_stamp":"2024-09-10T18:56:36.583836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.583836+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050879319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":366,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089602+0000","last_change":"2024-09-10T18:58:38.064448+0000","last_active":"2024-09-10T18:58:44.089602+0000","last_peered":"2024-09-10T18:58:44.089602+0000","last_clean":"2024-09-10T18:58:44.089602+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:58:44.089602+0000","last_undegraded":"2024-09-10T18:58:44.089602+0000","last_fullsized":"2024-09-10T18:58:44.089602+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.064267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.064267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.065185+0000","scrub_duration":0.0037942050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":351,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093917+0000","last_change":"2024-09-10T18:58:42.486651+0000","last_active":"2024-09-10T18:58:44.093917+0000","last_peered":"2024-09-10T18:58:44.093917+0000","last_clean":"2024-09-10T18:58:44.093917+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:58:44.093917+0000","last_undegraded":"2024-09-10T18:58:44.093917+0000","last_fullsized":"2024-09-10T18:58:44.093917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:42.486498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:42.486498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:47.324824+0000","scrub_duration":0.0032582879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":338,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.085923+0000","last_change":"2024-09-10T18:56:41.536711+0000","last_active":"2024-09-10T18:58:44.085923+0000","last_peered":"2024-09-10T18:58:44.085923+0000","last_clean":"2024-09-10T18:58:44.085923+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:58:44.085923+0000","last_undegraded":"2024-09-10T18:58:44.085923+0000","last_fullsized":"2024-09-10T18:58:44.085923+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:41.536545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:41.536545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003993295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100362+0000","last_change":"2024-09-10T18:56:10.959799+0000","last_active":"2024-09-10T18:58:44.100362+0000","last_peered":"2024-09-10T18:58:44.100362+0000","last_clean":"2024-09-10T18:58:44.100362+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:58:44.100362+0000","last_undegraded":"2024-09-10T18:58:44.100362+0000","last_fullsized":"2024-09-10T18:58:44.100362+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.959642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.959642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037938,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":362,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101798+0000","last_change":"2024-09-10T18:56:13.408480+0000","last_active":"2024-09-10T18:58:44.101798+0000","last_peered":"2024-09-10T18:58:44.101798+0000","last_clean":"2024-09-10T18:58:44.101798+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:58:44.101798+0000","last_undegraded":"2024-09-10T18:58:44.101798+0000","last_fullsized":"2024-09-10T18:58:44.101798+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:56:13.408417+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.408417+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002040924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109984+0000","last_change":"2024-09-10T18:58:38.864525+0000","last_active":"2024-09-10T18:58:44.109984+0000","last_peered":"2024-09-10T18:58:44.109984+0000","last_clean":"2024-09-10T18:58:44.109984+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:58:44.109984+0000","last_undegraded":"2024-09-10T18:58:44.109984+0000","last_fullsized":"2024-09-10T18:58:44.109984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.864360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.864360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.640967+0000","scrub_duration":0.00368761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":332,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113539+0000","last_change":"2024-09-10T18:54:53.960288+0000","last_active":"2024-09-10T18:58:44.113539+0000","last_peered":"2024-09-10T18:58:44.113539+0000","last_clean":"2024-09-10T18:58:44.113539+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:58:44.113539+0000","last_undegraded":"2024-09-10T18:58:44.113539+0000","last_fullsized":"2024-09-10T18:58:44.113539+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:53.960194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:54:53.960194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027047870000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":340,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.848045+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T18:58:44.848045+0000","last_peered":"2024-09-10T18:58:44.848045+0000","last_clean":"2024-09-10T18:58:44.848045+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:58:44.848045+0000","last_undegraded":"2024-09-10T18:58:44.848045+0000","last_fullsized":"2024-09-10T18:58:44.848045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:03.423360+0000","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119273+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:58:44.119273+0000","last_peered":"2024-09-10T18:58:44.119273+0000","last_clean":"2024-09-10T18:58:44.119273+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:58:44.119273+0000","last_undegraded":"2024-09-10T18:58:44.119273+0000","last_fullsized":"2024-09-10T18:58:44.119273+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":374,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089815+0000","last_change":"2024-09-10T18:56:15.528640+0000","last_active":"2024-09-10T18:58:44.089815+0000","last_peered":"2024-09-10T18:58:44.089815+0000","last_clean":"2024-09-10T18:58:44.089815+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:58:44.089815+0000","last_undegraded":"2024-09-10T18:58:44.089815+0000","last_fullsized":"2024-09-10T18:58:44.089815+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:56:15.528474+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.528474+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049455280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":378,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092404+0000","last_change":"2024-09-10T18:58:34.970321+0000","last_active":"2024-09-10T18:58:44.092404+0000","last_peered":"2024-09-10T18:58:44.092404+0000","last_clean":"2024-09-10T18:58:44.092404+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:58:44.092404+0000","last_undegraded":"2024-09-10T18:58:44.092404+0000","last_fullsized":"2024-09-10T18:58:44.092404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:34.970167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.970167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.762548+0000","scrub_duration":0.0036040859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":350,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095701+0000","last_change":"2024-09-10T18:57:34.244420+0000","last_active":"2024-09-10T18:58:44.095701+0000","last_peered":"2024-09-10T18:58:44.095701+0000","last_clean":"2024-09-10T18:58:44.095701+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:58:44.095701+0000","last_undegraded":"2024-09-10T18:58:44.095701+0000","last_fullsized":"2024-09-10T18:58:44.095701+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.244346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.244346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00228093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.097854+0000","last_change":"2024-09-10T18:58:37.374134+0000","last_active":"2024-09-10T18:58:44.097854+0000","last_peered":"2024-09-10T18:58:44.097854+0000","last_clean":"2024-09-10T18:58:44.097854+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:58:44.097854+0000","last_undegraded":"2024-09-10T18:58:44.097854+0000","last_fullsized":"2024-09-10T18:58:44.097854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.374014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.374014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.692319+0000","scrub_duration":0.0023318560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":273,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099526+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T18:58:44.099526+0000","last_peered":"2024-09-10T18:58:44.099526+0000","last_clean":"2024-09-10T18:58:44.099526+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:58:44.099526+0000","last_undegraded":"2024-09-10T18:58:44.099526+0000","last_fullsized":"2024-09-10T18:58:44.099526+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.117442+0000","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":361,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089213+0000","last_change":"2024-09-10T18:56:27.550110+0000","last_active":"2024-09-10T18:58:44.089213+0000","last_peered":"2024-09-10T18:58:44.089213+0000","last_clean":"2024-09-10T18:58:44.089213+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:58:44.089213+0000","last_undegraded":"2024-09-10T18:58:44.089213+0000","last_fullsized":"2024-09-10T18:58:44.089213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:27.549946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.549946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051279189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":397,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103293+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:58:44.103293+0000","last_peered":"2024-09-10T18:58:44.103293+0000","last_clean":"2024-09-10T18:58:44.103293+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:58:44.103293+0000","last_undegraded":"2024-09-10T18:58:44.103293+0000","last_fullsized":"2024-09-10T18:58:44.103293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":477,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.100696+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T18:58:44.100696+0000","last_peered":"2024-09-10T18:58:44.100696+0000","last_clean":"2024-09-10T18:58:44.100696+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:58:44.100696+0000","last_undegraded":"2024-09-10T18:58:44.100696+0000","last_fullsized":"2024-09-10T18:58:44.100696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:01.279543+0000","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.091244+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T18:58:44.091244+0000","last_peered":"2024-09-10T18:58:44.091244+0000","last_clean":"2024-09-10T18:58:44.091244+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:58:44.091244+0000","last_undegraded":"2024-09-10T18:58:44.091244+0000","last_fullsized":"2024-09-10T18:58:44.091244+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.083086+0000","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":275,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093046+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:58:44.093046+0000","last_peered":"2024-09-10T18:58:44.093046+0000","last_clean":"2024-09-10T18:58:44.093046+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:58:44.093046+0000","last_undegraded":"2024-09-10T18:58:44.093046+0000","last_fullsized":"2024-09-10T18:58:44.093046+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":371,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.086646+0000","last_change":"2024-09-10T18:57:33.408816+0000","last_active":"2024-09-10T18:58:44.086646+0000","last_peered":"2024-09-10T18:58:44.086646+0000","last_clean":"2024-09-10T18:58:44.086646+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:58:44.086646+0000","last_undegraded":"2024-09-10T18:58:44.086646+0000","last_fullsized":"2024-09-10T18:58:44.086646+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:33.408655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:33.408655+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039294170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":356,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088795+0000","last_change":"2024-09-10T18:57:39.808114+0000","last_active":"2024-09-10T18:58:44.088795+0000","last_peered":"2024-09-10T18:58:44.088795+0000","last_clean":"2024-09-10T18:58:44.088795+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:58:44.088795+0000","last_undegraded":"2024-09-10T18:58:44.088795+0000","last_fullsized":"2024-09-10T18:58:44.088795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.807949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.807949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034718420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121548+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:58:44.121548+0000","last_peered":"2024-09-10T18:58:44.121548+0000","last_clean":"2024-09-10T18:58:44.121548+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:58:44.121548+0000","last_undegraded":"2024-09-10T18:58:44.121548+0000","last_fullsized":"2024-09-10T18:58:44.121548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":350,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089081+0000","last_change":"2024-09-10T18:57:42.899838+0000","last_active":"2024-09-10T18:58:44.089081+0000","last_peered":"2024-09-10T18:58:44.089081+0000","last_clean":"2024-09-10T18:58:44.089081+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:58:44.089081+0000","last_undegraded":"2024-09-10T18:58:44.089081+0000","last_fullsized":"2024-09-10T18:58:44.089081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.899684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.899684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.010497+0000","scrub_duration":0.0036750730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103732+0000","last_change":"2024-09-10T18:56:26.891562+0000","last_active":"2024-09-10T18:58:44.103732+0000","last_peered":"2024-09-10T18:58:44.103732+0000","last_clean":"2024-09-10T18:58:44.103732+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:58:44.103732+0000","last_undegraded":"2024-09-10T18:58:44.103732+0000","last_fullsized":"2024-09-10T18:58:44.103732+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:26.891413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.891413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003243409,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":361,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094125+0000","last_change":"2024-09-10T18:58:40.428324+0000","last_active":"2024-09-10T18:58:44.094125+0000","last_peered":"2024-09-10T18:58:44.094125+0000","last_clean":"2024-09-10T18:58:44.094125+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:58:44.094125+0000","last_undegraded":"2024-09-10T18:58:44.094125+0000","last_fullsized":"2024-09-10T18:58:44.094125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:58:40.428157+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.428157+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.549402+0000","scrub_duration":0.003853726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093547+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T18:58:44.093547+0000","last_peered":"2024-09-10T18:58:44.093547+0000","last_clean":"2024-09-10T18:58:44.093547+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:58:44.093547+0000","last_undegraded":"2024-09-10T18:58:44.093547+0000","last_fullsized":"2024-09-10T18:58:44.093547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.849115+0000","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095085+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T18:58:44.095085+0000","last_peered":"2024-09-10T18:58:44.095085+0000","last_clean":"2024-09-10T18:58:44.095085+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:58:44.095085+0000","last_undegraded":"2024-09-10T18:58:44.095085+0000","last_fullsized":"2024-09-10T18:58:44.095085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.425187+0000","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095778+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T18:58:44.095778+0000","last_peered":"2024-09-10T18:58:44.095778+0000","last_clean":"2024-09-10T18:58:44.095778+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:58:44.095778+0000","last_undegraded":"2024-09-10T18:58:44.095778+0000","last_fullsized":"2024-09-10T18:58:44.095778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:13.562040+0000","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.098874+0000","last_change":"2024-09-10T18:56:35.881687+0000","last_active":"2024-09-10T18:58:44.098874+0000","last_peered":"2024-09-10T18:58:44.098874+0000","last_clean":"2024-09-10T18:58:44.098874+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:58:44.098874+0000","last_undegraded":"2024-09-10T18:58:44.098874+0000","last_fullsized":"2024-09-10T18:58:44.098874+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.881530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.881530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033229869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":366,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088147+0000","last_change":"2024-09-10T18:56:28.577747+0000","last_active":"2024-09-10T18:58:44.088147+0000","last_peered":"2024-09-10T18:58:44.088147+0000","last_clean":"2024-09-10T18:58:44.088147+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:58:44.088147+0000","last_undegraded":"2024-09-10T18:58:44.088147+0000","last_fullsized":"2024-09-10T18:58:44.088147+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:56:28.577591+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:56:28.577591+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045494740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":347,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.098383+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:58:44.098383+0000","last_peered":"2024-09-10T18:58:44.098383+0000","last_clean":"2024-09-10T18:58:44.098383+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:58:44.098383+0000","last_undegraded":"2024-09-10T18:58:44.098383+0000","last_fullsized":"2024-09-10T18:58:44.098383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":470,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.093692+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T18:58:44.093692+0000","last_peered":"2024-09-10T18:58:44.093692+0000","last_clean":"2024-09-10T18:58:44.093692+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:58:44.093692+0000","last_undegraded":"2024-09-10T18:58:44.093692+0000","last_fullsized":"2024-09-10T18:58:44.093692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":340,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121361+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:58:44.121361+0000","last_peered":"2024-09-10T18:58:44.121361+0000","last_clean":"2024-09-10T18:58:44.121361+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:58:44.121361+0000","last_undegraded":"2024-09-10T18:58:44.121361+0000","last_fullsized":"2024-09-10T18:58:44.121361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119930+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:58:44.119930+0000","last_peered":"2024-09-10T18:58:44.119930+0000","last_clean":"2024-09-10T18:58:44.119930+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:58:44.119930+0000","last_undegraded":"2024-09-10T18:58:44.119930+0000","last_fullsized":"2024-09-10T18:58:44.119930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":360,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119469+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:58:44.119469+0000","last_peered":"2024-09-10T18:58:44.119469+0000","last_clean":"2024-09-10T18:58:44.119469+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:58:44.119469+0000","last_undegraded":"2024-09-10T18:58:44.119469+0000","last_fullsized":"2024-09-10T18:58:44.119469+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":346,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.119809+0000","last_change":"2024-09-10T18:55:21.011819+0000","last_active":"2024-09-10T18:58:44.119809+0000","last_peered":"2024-09-10T18:58:44.119809+0000","last_clean":"2024-09-10T18:58:44.119809+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:58:44.119809+0000","last_undegraded":"2024-09-10T18:58:44.119809+0000","last_fullsized":"2024-09-10T18:58:44.119809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:21.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.575153+0000","last_clean_scrub_stamp":"2024-09-10T18:55:21.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016879950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":463,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.110383+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T18:58:44.110383+0000","last_peered":"2024-09-10T18:58:44.110383+0000","last_clean":"2024-09-10T18:58:44.110383+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:58:44.110383+0000","last_undegraded":"2024-09-10T18:58:44.110383+0000","last_fullsized":"2024-09-10T18:58:44.110383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.132106+0000","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103057+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T18:58:44.103057+0000","last_peered":"2024-09-10T18:58:44.103057+0000","last_clean":"2024-09-10T18:58:44.103057+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:58:44.103057+0000","last_undegraded":"2024-09-10T18:58:44.103057+0000","last_fullsized":"2024-09-10T18:58:44.103057+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.237854+0000","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":251,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121338+0000","last_change":"2024-09-10T18:54:37.631768+0000","last_active":"2024-09-10T18:58:44.121338+0000","last_peered":"2024-09-10T18:58:44.121338+0000","last_clean":"2024-09-10T18:58:44.121338+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:58:44.121338+0000","last_undegraded":"2024-09-10T18:58:44.121338+0000","last_fullsized":"2024-09-10T18:58:44.121338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:54:31.683713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":368,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.096003+0000","last_change":"2024-09-10T18:58:38.152259+0000","last_active":"2024-09-10T18:58:44.096003+0000","last_peered":"2024-09-10T18:58:44.096003+0000","last_clean":"2024-09-10T18:58:44.096003+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:58:44.096003+0000","last_undegraded":"2024-09-10T18:58:44.096003+0000","last_fullsized":"2024-09-10T18:58:44.096003+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.152079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.152079+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.945644+0000","scrub_duration":0.0040090350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":345,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094467+0000","last_change":"2024-09-10T18:55:02.566942+0000","last_active":"2024-09-10T18:58:44.094467+0000","last_peered":"2024-09-10T18:58:44.094467+0000","last_clean":"2024-09-10T18:58:44.094467+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:58:44.094467+0000","last_undegraded":"2024-09-10T18:58:44.094467+0000","last_fullsized":"2024-09-10T18:58:44.094467+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:02.566861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:02.566861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002853408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":341,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.097659+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T18:58:44.097659+0000","last_peered":"2024-09-10T18:58:44.097659+0000","last_clean":"2024-09-10T18:58:44.097659+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:58:44.097659+0000","last_undegraded":"2024-09-10T18:58:44.097659+0000","last_fullsized":"2024-09-10T18:58:44.097659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.840825+0000","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":343,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.112761+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:58:44.112761+0000","last_peered":"2024-09-10T18:58:44.112761+0000","last_clean":"2024-09-10T18:58:44.112761+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:58:44.112761+0000","last_undegraded":"2024-09-10T18:58:44.112761+0000","last_fullsized":"2024-09-10T18:58:44.112761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101053+0000","last_change":"2024-09-10T18:56:13.960463+0000","last_active":"2024-09-10T18:58:44.101053+0000","last_peered":"2024-09-10T18:58:44.101053+0000","last_clean":"2024-09-10T18:58:44.101053+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:58:44.101053+0000","last_undegraded":"2024-09-10T18:58:44.101053+0000","last_fullsized":"2024-09-10T18:58:44.101053+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.960307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.960307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034054630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":376,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101970+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T18:58:44.101970+0000","last_peered":"2024-09-10T18:58:44.101970+0000","last_clean":"2024-09-10T18:58:44.101970+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:58:44.101970+0000","last_undegraded":"2024-09-10T18:58:44.101970+0000","last_fullsized":"2024-09-10T18:58:44.101970+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.974665+0000","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109737+0000","last_change":"2024-09-10T18:56:37.589413+0000","last_active":"2024-09-10T18:58:44.109737+0000","last_peered":"2024-09-10T18:58:44.109737+0000","last_clean":"2024-09-10T18:58:44.109737+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:58:44.109737+0000","last_undegraded":"2024-09-10T18:58:44.109737+0000","last_fullsized":"2024-09-10T18:58:44.109737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:37.589206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:37.589206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032412069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101312+0000","last_change":"2024-09-10T18:56:43.003568+0000","last_active":"2024-09-10T18:58:44.101312+0000","last_peered":"2024-09-10T18:58:44.101312+0000","last_clean":"2024-09-10T18:58:44.101312+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:58:44.101312+0000","last_undegraded":"2024-09-10T18:58:44.101312+0000","last_fullsized":"2024-09-10T18:58:44.101312+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.003325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.003325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036735230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088798+0000","last_change":"2024-09-10T18:56:23.959328+0000","last_active":"2024-09-10T18:58:44.088798+0000","last_peered":"2024-09-10T18:58:44.088798+0000","last_clean":"2024-09-10T18:58:44.088798+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:58:44.088798+0000","last_undegraded":"2024-09-10T18:58:44.088798+0000","last_fullsized":"2024-09-10T18:58:44.088798+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":263,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101265+0000","last_change":"2024-09-10T18:56:03.094486+0000","last_active":"2024-09-10T18:58:44.101265+0000","last_peered":"2024-09-10T18:58:44.101265+0000","last_clean":"2024-09-10T18:58:44.101265+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:58:44.101265+0000","last_undegraded":"2024-09-10T18:58:44.101265+0000","last_fullsized":"2024-09-10T18:58:44.101265+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.094374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.094374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.031077224000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":356,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121679+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T18:58:44.121679+0000","last_peered":"2024-09-10T18:58:44.121679+0000","last_clean":"2024-09-10T18:58:44.121679+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:58:44.121679+0000","last_undegraded":"2024-09-10T18:58:44.121679+0000","last_fullsized":"2024-09-10T18:58:44.121679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.849844+0000","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":344,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.111352+0000","last_change":"2024-09-10T18:55:22.062030+0000","last_active":"2024-09-10T18:58:44.111352+0000","last_peered":"2024-09-10T18:58:44.111352+0000","last_clean":"2024-09-10T18:58:44.111352+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:58:44.111352+0000","last_undegraded":"2024-09-10T18:58:44.111352+0000","last_fullsized":"2024-09-10T18:58:44.111352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:55:22.061917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:55:22.061917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028660370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":334,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089725+0000","last_change":"2024-09-10T18:56:17.867315+0000","last_active":"2024-09-10T18:58:44.089725+0000","last_peered":"2024-09-10T18:58:44.089725+0000","last_clean":"2024-09-10T18:58:44.089725+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:58:44.089725+0000","last_undegraded":"2024-09-10T18:58:44.089725+0000","last_fullsized":"2024-09-10T18:58:44.089725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.086878+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T18:58:44.086878+0000","last_peered":"2024-09-10T18:58:44.086878+0000","last_clean":"2024-09-10T18:58:44.086878+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:58:44.086878+0000","last_undegraded":"2024-09-10T18:58:44.086878+0000","last_fullsized":"2024-09-10T18:58:44.086878+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:55.521548+0000","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107685+0000","last_change":"2024-09-10T18:56:32.708619+0000","last_active":"2024-09-10T18:58:44.107685+0000","last_peered":"2024-09-10T18:58:44.107685+0000","last_clean":"2024-09-10T18:58:44.107685+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:58:44.107685+0000","last_undegraded":"2024-09-10T18:58:44.107685+0000","last_fullsized":"2024-09-10T18:58:44.107685+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:32.708464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.708464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032020400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":358,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.111121+0000","last_change":"2024-09-10T18:56:03.329458+0000","last_active":"2024-09-10T18:58:44.111121+0000","last_peered":"2024-09-10T18:58:44.111121+0000","last_clean":"2024-09-10T18:58:44.111121+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:58:44.111121+0000","last_undegraded":"2024-09-10T18:58:44.111121+0000","last_fullsized":"2024-09-10T18:58:44.111121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:03.329289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.329289+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044308079999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":355,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099259+0000","last_change":"2024-09-10T18:57:56.351163+0000","last_active":"2024-09-10T18:58:44.099259+0000","last_peered":"2024-09-10T18:58:44.099259+0000","last_clean":"2024-09-10T18:58:44.099259+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:58:44.099259+0000","last_undegraded":"2024-09-10T18:58:44.099259+0000","last_fullsized":"2024-09-10T18:58:44.099259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.351005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.351005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:59.113542+0000","scrub_duration":0.0035909499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":334,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088960+0000","last_change":"2024-09-10T18:56:13.753059+0000","last_active":"2024-09-10T18:58:44.088960+0000","last_peered":"2024-09-10T18:58:44.088960+0000","last_clean":"2024-09-10T18:58:44.088960+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:58:44.088960+0000","last_undegraded":"2024-09-10T18:58:44.088960+0000","last_fullsized":"2024-09-10T18:58:44.088960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:13.752982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:13.752982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026827040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.091986+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:58:44.091986+0000","last_peered":"2024-09-10T18:58:44.091986+0000","last_clean":"2024-09-10T18:58:44.091986+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:58:44.091986+0000","last_undegraded":"2024-09-10T18:58:44.091986+0000","last_fullsized":"2024-09-10T18:58:44.091986+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.113419+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T18:58:44.113419+0000","last_peered":"2024-09-10T18:58:44.113419+0000","last_clean":"2024-09-10T18:58:44.113419+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:58:44.113419+0000","last_undegraded":"2024-09-10T18:58:44.113419+0000","last_fullsized":"2024-09-10T18:58:44.113419+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:02.553089+0000","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.088541+0000","last_change":"2024-09-10T18:56:42.568208+0000","last_active":"2024-09-10T18:58:44.088541+0000","last_peered":"2024-09-10T18:58:44.088541+0000","last_clean":"2024-09-10T18:58:44.088541+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:58:44.088541+0000","last_undegraded":"2024-09-10T18:58:44.088541+0000","last_fullsized":"2024-09-10T18:58:44.088541+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:56:42.568045+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.568045+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036986330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"77'15","reported_seq":378,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.107837+0000","last_change":"2024-09-10T18:56:44.598271+0000","last_active":"2024-09-10T18:58:44.107837+0000","last_peered":"2024-09-10T18:58:44.107837+0000","last_clean":"2024-09-10T18:58:44.107837+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:58:44.107837+0000","last_undegraded":"2024-09-10T18:58:44.107837+0000","last_fullsized":"2024-09-10T18:58:44.107837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:44.598130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:44.598130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016863953000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095856+0000","last_change":"2024-09-10T18:58:31.395234+0000","last_active":"2024-09-10T18:58:44.095856+0000","last_peered":"2024-09-10T18:58:44.095856+0000","last_clean":"2024-09-10T18:58:44.095856+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:58:44.095856+0000","last_undegraded":"2024-09-10T18:58:44.095856+0000","last_fullsized":"2024-09-10T18:58:44.095856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.395072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.395072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:33.507413+0000","scrub_duration":0.0032990430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.121442+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:58:44.121442+0000","last_peered":"2024-09-10T18:58:44.121442+0000","last_clean":"2024-09-10T18:58:44.121442+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:58:44.121442+0000","last_undegraded":"2024-09-10T18:58:44.121442+0000","last_fullsized":"2024-09-10T18:58:44.121442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095228+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:58:44.095228+0000","last_peered":"2024-09-10T18:58:44.095228+0000","last_clean":"2024-09-10T18:58:44.095228+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:58:44.095228+0000","last_undegraded":"2024-09-10T18:58:44.095228+0000","last_fullsized":"2024-09-10T18:58:44.095228+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":349,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.109352+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T18:58:44.109352+0000","last_peered":"2024-09-10T18:58:44.109352+0000","last_clean":"2024-09-10T18:58:44.109352+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:58:44.109352+0000","last_undegraded":"2024-09-10T18:58:44.109352+0000","last_fullsized":"2024-09-10T18:58:44.109352+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:56.514634+0000","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":362,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.090307+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T18:58:44.090307+0000","last_peered":"2024-09-10T18:58:44.090307+0000","last_clean":"2024-09-10T18:58:44.090307+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:58:44.090307+0000","last_undegraded":"2024-09-10T18:58:44.090307+0000","last_fullsized":"2024-09-10T18:58:44.090307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:53.283437+0000","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089915+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T18:58:44.089915+0000","last_peered":"2024-09-10T18:58:44.089915+0000","last_clean":"2024-09-10T18:58:44.089915+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:58:44.089915+0000","last_undegraded":"2024-09-10T18:58:44.089915+0000","last_fullsized":"2024-09-10T18:58:44.089915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.346480+0000","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":491,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099179+0000","last_change":"2024-09-10T18:58:38.434610+0000","last_active":"2024-09-10T18:58:44.099179+0000","last_peered":"2024-09-10T18:58:44.099179+0000","last_clean":"2024-09-10T18:58:44.099179+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:58:44.099179+0000","last_undegraded":"2024-09-10T18:58:44.099179+0000","last_fullsized":"2024-09-10T18:58:44.099179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:58:38.434348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.434348+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:43.611964+0000","scrub_duration":0.0046950070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.099094+0000","last_change":"2024-09-10T18:56:10.750481+0000","last_active":"2024-09-10T18:58:44.099094+0000","last_peered":"2024-09-10T18:58:44.099094+0000","last_clean":"2024-09-10T18:58:44.099094+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:58:44.099094+0000","last_undegraded":"2024-09-10T18:58:44.099094+0000","last_fullsized":"2024-09-10T18:58:44.099094+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.750410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.750410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002546988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.103058+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T18:58:44.103058+0000","last_peered":"2024-09-10T18:58:44.103058+0000","last_clean":"2024-09-10T18:58:44.103058+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:58:44.103058+0000","last_undegraded":"2024-09-10T18:58:44.103058+0000","last_fullsized":"2024-09-10T18:58:44.103058+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:03.795787+0000","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094084+0000","last_change":"2024-09-10T18:55:17.563724+0000","last_active":"2024-09-10T18:58:44.094084+0000","last_peered":"2024-09-10T18:58:44.094084+0000","last_clean":"2024-09-10T18:58:44.094084+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:58:44.094084+0000","last_undegraded":"2024-09-10T18:58:44.094084+0000","last_fullsized":"2024-09-10T18:58:44.094084+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:55:17.563541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.563541+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042049610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":354,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.120089+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:58:44.120089+0000","last_peered":"2024-09-10T18:58:44.120089+0000","last_clean":"2024-09-10T18:58:44.120089+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:58:44.120089+0000","last_undegraded":"2024-09-10T18:58:44.120089+0000","last_fullsized":"2024-09-10T18:58:44.120089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":362,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.089377+0000","last_change":"2024-09-10T18:58:32.931801+0000","last_active":"2024-09-10T18:58:44.089377+0000","last_peered":"2024-09-10T18:58:44.089377+0000","last_clean":"2024-09-10T18:58:44.089377+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:58:44.089377+0000","last_undegraded":"2024-09-10T18:58:44.089377+0000","last_fullsized":"2024-09-10T18:58:44.089377+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.931700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.931700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:37.655765+0000","scrub_duration":0.002489907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":261,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.102022+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:58:44.102022+0000","last_peered":"2024-09-10T18:58:44.102022+0000","last_clean":"2024-09-10T18:58:44.102022+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:58:44.102022+0000","last_undegraded":"2024-09-10T18:58:44.102022+0000","last_fullsized":"2024-09-10T18:58:44.102022+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094969+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:58:44.094969+0000","last_peered":"2024-09-10T18:58:44.094969+0000","last_clean":"2024-09-10T18:58:44.094969+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:58:44.094969+0000","last_undegraded":"2024-09-10T18:58:44.094969+0000","last_fullsized":"2024-09-10T18:58:44.094969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":348,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.092335+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:58:44.092335+0000","last_peered":"2024-09-10T18:58:44.092335+0000","last_clean":"2024-09-10T18:58:44.092335+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:58:44.092335+0000","last_undegraded":"2024-09-10T18:58:44.092335+0000","last_fullsized":"2024-09-10T18:58:44.092335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":372,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.101153+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T18:58:44.101153+0000","last_peered":"2024-09-10T18:58:44.101153+0000","last_clean":"2024-09-10T18:58:44.101153+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:58:44.101153+0000","last_undegraded":"2024-09-10T18:58:44.101153+0000","last_fullsized":"2024-09-10T18:58:44.101153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:58:57.200134+0000","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":334,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.094329+0000","last_change":"2024-09-10T18:56:03.700191+0000","last_active":"2024-09-10T18:58:44.094329+0000","last_peered":"2024-09-10T18:58:44.094329+0000","last_clean":"2024-09-10T18:58:44.094329+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:58:44.094329+0000","last_undegraded":"2024-09-10T18:58:44.094329+0000","last_fullsized":"2024-09-10T18:58:44.094329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.700032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.700032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033275370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":262,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-10T18:58:44.095191+0000","last_change":"2024-09-10T18:56:42.757775+0000","last_active":"2024-09-10T18:58:44.095191+0000","last_peered":"2024-09-10T18:58:44.095191+0000","last_clean":"2024-09-10T18:58:44.095191+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:58:44.095191+0000","last_undegraded":"2024-09-10T18:58:44.095191+0000","last_fullsized":"2024-09-10T18:58:44.095191+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:42.757625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:42.757625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003526834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":197,"num_pg":32,"stat_sum":{"num_bytes":169,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":2,"num_bytes_hit_set_archive":168,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195524,"num_objects":124,"num_object_clones":2,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26075136,"data_stored":24586572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1802,"ondisk_log_size":1802,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":542,"num_read_kb":537,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972477,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311648,"kb_used_data":14280,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93433504,"statfs":{"total":95995035648,"available":95675908096,"internally_reserved":0,"allocated":14622720,"data_stored":12032711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88200000000000001,"15min":0.91400000000000003},"min":{"1min":0.53300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.3420000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.92900000000000005,"15min":0.93400000000000005},"min":{"1min":0.58999999999999997,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.7000000000000002,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":0.875,"15min":0.89900000000000002},"min":{"1min":0.47299999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.98499999999999999,"15min":1.02},"min":{"1min":0.72499999999999998,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.3360000000000001,"5min":1.742,"15min":1.8280000000000001},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.0309999999999999,"15min":1.0389999999999999},"min":{"1min":0.70699999999999996,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.3580000000000001,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.012,"15min":0.98699999999999999},"min":{"1min":0.57399999999999995,"5min":0.499,"15min":0.48799999999999999},"max":{"1min":4.1749999999999998,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.0369999999999999,"15min":1.018},"min":{"1min":0.73199999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":4.2080000000000002,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.66500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.66400000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.04,"15min":1.0189999999999999},"min":{"1min":0.67800000000000005,"5min":0.499,"15min":0.443},"max":{"1min":1.2789999999999999,"5min":1.556,"15min":2.1989999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.1160000000000001,"15min":1.054},"min":{"1min":0.78500000000000003,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":4.2400000000000002,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.1759999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.026,"15min":1.0049999999999999},"min":{"1min":0.59799999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":5.4710000000000001,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.175,"15min":1.1779999999999999},"min":{"1min":0.59699999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.5329999999999999,"5min":1.5329999999999999,"15min":2.1419999999999999},"last":0.91200000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.71099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.591,"5min":1.591,"15min":2.2749999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.2090000000000001,"15min":1.196},"min":{"1min":0.86099999999999999,"5min":0.59199999999999997,"15min":0.53100000000000003},"max":{"1min":4.1310000000000002,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":0.85199999999999998}]}]},{"osd":6,"up_from":52,"seq":223338299527,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307736,"kb_used_data":6200,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437416,"statfs":{"total":95995035648,"available":95679913984,"internally_reserved":0,"allocated":6348800,"data_stored":3845082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.872,"15min":0.88400000000000001},"min":{"1min":0.58399999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.032,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.93899999999999995,"15min":0.93700000000000006},"min":{"1min":0.57799999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.1779999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.0549999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.196,"15min":0.997},"min":{"1min":0.56699999999999995,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.9530000000000001,"5min":18.494,"15min":18.494},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.871,"15min":0.90200000000000002},"min":{"1min":0.73099999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.244,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":1.325}]},{"osd":2,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0249999999999999,"15min":0.95899999999999996},"min":{"1min":0.56399999999999995,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.2430000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.92300000000000004,"15min":0.92400000000000004},"min":{"1min":0.54300000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.3300000000000001,"5min":1.988,"15min":1.988},"last":1.1439999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0329999999999999,"15min":0.97899999999999998},"min":{"1min":0.627,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.234,"5min":2.278,"15min":2.278},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.98299999999999998,"15min":0.97399999999999998},"min":{"1min":0.73899999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.7529999999999999,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":0.91000000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.97599999999999998,"15min":0.90400000000000003},"min":{"1min":0.58999999999999997,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.452,"5min":1.712,"15min":1.712},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.099,"15min":1.036},"min":{"1min":0.66700000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.302}]},{"osd":5,"last update":"Tue Sep 10 18:57:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.999},"min":{"1min":0.67900000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.516,"5min":1.7210000000000001,"15min":1.7210000000000001},"last":1.28},{"interface":"front","average":{"1min":1.252,"5min":1.163,"15min":1.1100000000000001},"min":{"1min":0.77000000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.1669999999999998,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.2609999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:57:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.0960000000000001,"15min":1.0600000000000001},"min":{"1min":0.90200000000000002,"5min":0.503,"15min":0.503},"max":{"1min":2.069,"5min":2.069,"15min":2.069},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.175,"15min":1.153},"min":{"1min":0.93100000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.794,"5min":1.794,"15min":1.794},"last":1.518}]}]},{"osd":5,"up_from":43,"seq":184683593873,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307180,"kb_used_data":9804,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93437972,"statfs":{"total":95995035648,"available":95680483328,"internally_reserved":0,"allocated":10039296,"data_stored":7449310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.093,"15min":0.94399999999999995},"min":{"1min":0.59799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1539999999999999,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.998,"15min":0.91400000000000003},"min":{"1min":0.621,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.212,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.876}]},{"osd":1,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.6679999999999999,"15min":1.1339999999999999},"min":{"1min":0.63800000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.3140000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.421,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1659999999999999,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.0940000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.8999999999999999,"15min":1.2370000000000001},"min":{"1min":0.51000000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.752,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1,"5min":1.135,"15min":0.98999999999999999},"min":{"1min":0.64000000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2450000000000001,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":1.0469999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1319999999999999,"15min":0.998},"min":{"1min":0.56299999999999994,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8520000000000001,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.798,"15min":1.236},"min":{"1min":0.83999999999999997,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":2.3820000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":0.75900000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1830000000000001,"15min":0.92900000000000005},"min":{"1min":0.68300000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6519999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.252},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2250000000000001,"15min":0.998},"min":{"1min":0.77200000000000002,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":4.1970000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.2290000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.3660000000000001,"15min":1.133},"min":{"1min":0.63400000000000001,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":7.5380000000000003,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.153},{"interface":"front","average":{"1min":1.355,"5min":1.407,"15min":1.165},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.47199999999999998},"max":{"1min":4.0709999999999997,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.177}]},{"osd":7,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":1.49,"15min":1.2010000000000001},"min":{"1min":0.68500000000000005,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":15.613,"5min":15.613,"15min":15.613},"last":1.3},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.51,"15min":1.2749999999999999},"min":{"1min":0.72899999999999998,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":16.965,"5min":16.965,"15min":16.965},"last":1.325}]}]},{"osd":4,"up_from":35,"seq":150323855517,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307176,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437976,"statfs":{"total":95995035648,"available":95680487424,"internally_reserved":0,"allocated":5791744,"data_stored":3256162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79400000000000004,"15min":0.83999999999999997},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0469999999999999,"5min":1.546,"15min":1.546},"last":1.179},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.58899999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.258,"5min":1.667,"15min":1.7070000000000001},"last":0.90400000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85299999999999998,"15min":0.874},"min":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.90800000000000003,"15min":0.89100000000000001},"min":{"1min":0.60699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1919999999999999,"5min":1.575,"15min":1.575},"last":1.2869999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.93000000000000005,"15min":0.91500000000000004},"min":{"1min":0.54600000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2350000000000001,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.93400000000000005,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.497,"15min":0.497},"max":{"1min":1.365,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":1.3140000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.65200000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1339999999999999,"5min":1.5489999999999999,"15min":1.714},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":0.94499999999999995},"min":{"1min":0.67000000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.137,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.357}]},{"osd":5,"last update":"Tue Sep 10 18:58:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.97499999999999998,"15min":0.89900000000000002},"min":{"1min":0.58099999999999996,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.262,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.038,"15min":0.95999999999999996},"min":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.377,"5min":1.4610000000000001,"15min":1.7909999999999999},"last":1.4430000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.0489999999999999,"15min":0.97999999999999998},"min":{"1min":0.69899999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.762,"5min":1.994,"15min":1.994},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.165,"5min":1.129,"15min":1.095},"min":{"1min":0.78700000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.575}]},{"osd":7,"last update":"Tue Sep 10 18:57:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1379999999999999,"15min":1.1619999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.585,"5min":1.6200000000000001,"15min":2.008},"last":1.55},{"interface":"front","average":{"1min":1.173,"5min":1.1559999999999999,"15min":1.1639999999999999},"min":{"1min":0.68100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.6830000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.4770000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051752,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315588,"kb_used_data":5732,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93429564,"statfs":{"total":95995035648,"available":95671873536,"internally_reserved":0,"allocated":5869568,"data_stored":3255887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.222,"15min":0.82399999999999995},"min":{"1min":0.36099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":1.026,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.75,"5min":1.242,"15min":0.89400000000000002},"min":{"1min":0.38800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":1.5700000000000001,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.53200000000000003}]},{"osd":1,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":1.3839999999999999,"15min":0.91700000000000004},"min":{"1min":0.48099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.802,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.0449999999999999,"15min":0.86499999999999999},"min":{"1min":0.39900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.075,"5min":15.189,"15min":15.189},"last":0.85699999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.355,"15min":0.99099999999999999},"min":{"1min":0.41899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1599999999999999,"5min":26.558,"15min":26.558},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.25,"15min":0.92900000000000005},"min":{"1min":0.378,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.6760000000000002,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.73899999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:58:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.335,"15min":1.1279999999999999},"min":{"1min":0.74199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.2469999999999999,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.095},{"interface":"front","average":{"1min":1.274,"5min":1.4330000000000001,"15min":1.2150000000000001},"min":{"1min":0.67000000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.2850000000000001,"5min":10.391,"15min":10.391},"last":1.236}]},{"osd":5,"last update":"Tue Sep 10 18:58:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.4279999999999999,"15min":1.212},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.355,"5min":17.535,"15min":17.535},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.105,"15min":1.0329999999999999},"min":{"1min":0.60399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.04,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.91700000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:58:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.288,"15min":1.2030000000000001},"min":{"1min":0.84099999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.9550000000000001,"5min":10.51,"15min":10.51},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.1439999999999999,"15min":1.0920000000000001},"min":{"1min":0.63100000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.3769999999999998,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.1140000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:57:53 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.3220000000000001,"15min":1.179},"min":{"1min":0.60699999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.214,"5min":18.312999999999999,"15min":18.312999999999999},"last":1.139},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.4470000000000001,"15min":1.306},"min":{"1min":0.77500000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":2.0270000000000001,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.028}]}]},{"osd":2,"up_from":19,"seq":81604378801,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319388,"kb_used_data":13716,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93425764,"statfs":{"total":95995035648,"available":95667982336,"internally_reserved":0,"allocated":14045184,"data_stored":11441917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.83699999999999997,"15min":0.71599999999999997},"min":{"1min":0.35799999999999998,"5min":0.219,"15min":0.219},"max":{"1min":1.2,"5min":9.516,"15min":9.516},"last":0.625},{"interface":"front","average":{"1min":0.75,"5min":0.81799999999999995,"15min":0.76900000000000002},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0600000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":1.0600000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.75600000000000001,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1339999999999999,"5min":5.516,"15min":5.516},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.83599999999999997,"15min":0.80400000000000005},"min":{"1min":0.41599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.048,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.876,"15min":0.85099999999999998},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0349999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.91000000000000003,"15min":0.84699999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.115,"5min":15.736000000000001,"15min":15.736000000000001},"last":1.194}]},{"osd":4,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.151,"15min":1.0449999999999999},"min":{"1min":0.629,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.835,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.421},{"interface":"front","average":{"1min":1.087,"5min":1.1459999999999999,"15min":1.107},"min":{"1min":0.76100000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8560000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.24}]},{"osd":5,"last update":"Tue Sep 10 18:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0860000000000001,"15min":1.101},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.458,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":1.093},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.198,"15min":1.0860000000000001},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.321,"5min":21.654,"15min":21.654},"last":1.9870000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.177,"15min":1.177},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.5089999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.262,"15min":1.1359999999999999},"min":{"1min":0.57899999999999996,"5min":0.505,"15min":0.505},"max":{"1min":3.367,"5min":15.971,"15min":15.971},"last":1.4530000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:58:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1819999999999999,"15min":1.1950000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.363,"5min":15.025,"15min":15.025},"last":1.905},{"interface":"front","average":{"1min":1.196,"5min":1.284,"15min":1.179},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":3.4239999999999999,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.304}]}]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311084,"kb_used_data":9564,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93434068,"statfs":{"total":95995035648,"available":95676485632,"internally_reserved":0,"allocated":9793536,"data_stored":7246582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.0409999999999999,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.877,"5min":32.677999999999997,"15min":32.677999999999997},"last":1.079},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.135,"15min":0.85899999999999999},"min":{"1min":0.41399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.7200000000000002,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.59299999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.141,"15min":0.84699999999999998},"min":{"1min":0.27200000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.048,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3480000000000001,"15min":0.97899999999999998},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.0890000000000004,"5min":32.363,"15min":32.363},"last":0.68000000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.23,"15min":0.97399999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.1469999999999998,"5min":25.704000000000001,"15min":25.704000000000001},"last":0.747},{"interface":"front","average":{"1min":1.2,"5min":1.5580000000000001,"15min":1.02},"min":{"1min":0.39500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.0099999999999998,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.82099999999999995}]},{"osd":4,"last update":"Tue Sep 10 18:58:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.234,"15min":1.0900000000000001},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.762,"5min":12.808999999999999,"15min":12.808999999999999},"last":0.996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.748,"15min":1.202},"min":{"1min":0.56000000000000005,"5min":0.378,"15min":0.378},"max":{"1min":6.181,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.96999999999999997}]},{"osd":5,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.2869999999999999,"15min":1.1040000000000001},"min":{"1min":0.52200000000000002,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":2.194,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.3069999999999999,"15min":1.1930000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.2719999999999998,"5min":12.84,"15min":12.84},"last":1.0549999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.528,"15min":1.244},"min":{"1min":0.59399999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.3279999999999998,"5min":25.608000000000001,"15min":25.608000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.526,"15min":1.2270000000000001},"min":{"1min":0.63300000000000001,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":6.1219999999999999,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.1870000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:57:59 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.51,"15min":1.3140000000000001},"min":{"1min":0.66200000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.4849999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.617,"15min":1.3029999999999999},"min":{"1min":0.71499999999999997,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":6.2599999999999998,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.1080000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303596,"kb_used_data":6228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93441556,"statfs":{"total":95995035648,"available":95684153344,"internally_reserved":0,"allocated":6377472,"data_stored":3836599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.6240000000000001,"15min":0.94899999999999995},"min":{"1min":0.39100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.0920000000000001,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.716,"15min":1.038},"min":{"1min":0.46000000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.2629999999999999,"5min":86.009,"15min":86.009},"last":0.67900000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.6299999999999999,"15min":1.0109999999999999},"min":{"1min":0.54400000000000004,"5min":0.216,"15min":0.216},"max":{"1min":3.427,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.62},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1970000000000001,"15min":0.89500000000000002},"min":{"1min":0.61899999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.105,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.76700000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.095,"15min":0.88},"min":{"1min":0.53800000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.246,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.4730000000000001,"15min":1.0269999999999999},"min":{"1min":0.51300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.012,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.91200000000000003}]},{"osd":4,"last update":"Tue Sep 10 18:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.143,"15min":1.0349999999999999},"min":{"1min":0.70099999999999996,"5min":0.5,"15min":0.5},"max":{"1min":3.3279999999999998,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.266,"5min":1.1579999999999999,"15min":1.093},"min":{"1min":0.85499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":3.419,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.0409999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:58:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.268,"15min":1.1060000000000001},"min":{"1min":0.85799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":2.609,"5min":10.048,"15min":10.048},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.101,"15min":1.0980000000000001},"min":{"1min":0.80600000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.3400000000000001,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.89200000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:57:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.1599999999999999,"15min":1.131},"min":{"1min":0.81299999999999994,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2.4550000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.202,"5min":1.3089999999999999,"15min":1.1890000000000001},"min":{"1min":0.86199999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.4929999999999999,"5min":11.894,"15min":11.894},"last":1.069}]},{"osd":7,"last update":"Tue Sep 10 18:57:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1659999999999999,"15min":1.167},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.512,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.004},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3260000000000001,"15min":1.3029999999999999},"min":{"1min":0.80100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.3780000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":0.93999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4182016,"data_stored":4000468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8417280,"data_stored":8194864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4194856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8194816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":197,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:58:50.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:50.401 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:58:50.401 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T18:58:50.401 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T18:58:50.401 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T18:58:50.401 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:58:50.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:50.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.565 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:50.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.699 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:50.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.513198+0000 osd.4 (osd.4) 247 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.517264+0000 osd.4 (osd.4) 248 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:49.610642+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:49.612353+0000 mon.b (mon.1) 292 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:49.614480+0000 mon.a (mon.0) 2345 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:49.616059+0000 mon.a (mon.0) 2346 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.912188+0000 osd.0 (osd.0) 204 : cluster [DBG] 12.1d scrub starts 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:49.915781+0000 osd.0 (osd.0) 205 : cluster [DBG] 12.1d scrub ok 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.051216+0000 mgr.y (mgr.24526) 389 : cluster [DBG] pgmap v544: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:50.119705+0000 mon.a (mon.0) 2347 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]': finished 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.119928+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:50.123249+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[18629]: audit 2024-09-10T18:58:50.555325+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.513198+0000 osd.4 (osd.4) 247 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.517264+0000 osd.4 (osd.4) 248 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:49.610642+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:49.612353+0000 mon.b (mon.1) 292 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:49.614480+0000 mon.a (mon.0) 2345 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:49.616059+0000 mon.a (mon.0) 2346 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.912188+0000 osd.0 (osd.0) 204 : cluster [DBG] 12.1d scrub starts 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:49.915781+0000 osd.0 (osd.0) 205 : cluster [DBG] 12.1d scrub ok 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.051216+0000 mgr.y (mgr.24526) 389 : cluster [DBG] pgmap v544: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:50.119705+0000 mon.a (mon.0) 2347 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]': finished 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.119928+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-10T18:58:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:50.123249+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:50 smithi083 bash[32381]: audit 2024-09-10T18:58:50.555325+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:50.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:50.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.513198+0000 osd.4 (osd.4) 247 : cluster [DBG] 12.1b scrub starts 2024-09-10T18:58:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.517264+0000 osd.4 (osd.4) 248 : cluster [DBG] 12.1b scrub ok 2024-09-10T18:58:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:49.610642+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:49.612353+0000 mon.b (mon.1) 292 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:49.614480+0000 mon.a (mon.0) 2345 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:49.616059+0000 mon.a (mon.0) 2346 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]: dispatch 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.912188+0000 osd.0 (osd.0) 204 : cluster [DBG] 12.1d scrub starts 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:49.915781+0000 osd.0 (osd.0) 205 : cluster [DBG] 12.1d scrub ok 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.051216+0000 mgr.y (mgr.24526) 389 : cluster [DBG] pgmap v544: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:50.119705+0000 mon.a (mon.0) 2347 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-45"}]': finished 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.119928+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e374: 8 total, 8 up, 8 in 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:50.123249+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:50.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:50 smithi165 bash[22591]: audit 2024-09-10T18:58:50.555325+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:50.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.962 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:50.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:51.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:51.099 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:51.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T18:58:51.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:51.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:51.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:51.367 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:51.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:51.508 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:51.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.205744+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.208988+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.571634+0000 mon.a (mon.0) 2350 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.595097+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:50.597316+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: audit 2024-09-10T18:58:51.013284+0000 mon.c (mon.2) 592 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: audit 2024-09-10T18:58:51.014846+0000 mon.c (mon.2) 593 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: audit 2024-09-10T18:58:51.027281+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: audit 2024-09-10T18:58:51.127966+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:51.128362+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-10T18:58:51.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: audit 2024-09-10T18:58:51.129823+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[18629]: cluster 2024-09-10T18:58:51.194615+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.205744+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.208988+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.571634+0000 mon.a (mon.0) 2350 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.595097+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:50.597316+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: audit 2024-09-10T18:58:51.013284+0000 mon.c (mon.2) 592 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: audit 2024-09-10T18:58:51.014846+0000 mon.c (mon.2) 593 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: audit 2024-09-10T18:58:51.027281+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: audit 2024-09-10T18:58:51.127966+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:51.625 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:51.128362+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-10T18:58:51.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: audit 2024-09-10T18:58:51.129823+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:51.626 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:51 smithi083 bash[32381]: cluster 2024-09-10T18:58:51.194615+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:58:51.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:51.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:51.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.205744+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub starts 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.208988+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.5 scrub ok 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.571634+0000 mon.a (mon.0) 2350 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.595097+0000 osd.6 (osd.6) 169 : cluster [DBG] 2.7 scrub starts 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:50.597316+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.7 scrub ok 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: audit 2024-09-10T18:58:51.013284+0000 mon.c (mon.2) 592 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: audit 2024-09-10T18:58:51.014846+0000 mon.c (mon.2) 593 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: audit 2024-09-10T18:58:51.027281+0000 mon.a (mon.0) 2351 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: audit 2024-09-10T18:58:51.127966+0000 mon.a (mon.0) 2352 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:51.128362+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e375: 8 total, 8 up, 8 in 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: audit 2024-09-10T18:58:51.129823+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]: dispatch 2024-09-10T18:58:51.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[22591]: cluster 2024-09-10T18:58:51.194615+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:58:51.762 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:58:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:58:51.781 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:51.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:51.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:51.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:52.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:52.064 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:52.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.197 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:52.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:58:52.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:52.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:52.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T18:58:52.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.464 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:52.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:52.600 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:52.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:52.717 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:58:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:58:52] "GET /metrics HTTP/1.1" 200 231183 "" "Prometheus/2.43.0" 2024-09-10T18:58:52.739 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:52.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.878 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:52.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:52.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:53.018 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:53.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:53.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:53.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:53.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:51.903988+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:51.907734+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.052004+0000 mgr.y (mgr.24526) 390 : cluster [DBG] pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.126247+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.126638+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.126821+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.130788+0000 mon.a (mon.0) 2357 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.155961+0000 mon.c (mon.2) 594 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.156351+0000 mon.a (mon.0) 2358 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.157835+0000 mon.c (mon.2) 595 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.158147+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.272 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.158727+0000 mon.c (mon.2) 596 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: audit 2024-09-10T18:58:52.159035+0000 mon.a (mon.0) 2360 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.229724+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.232894+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.444277+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.447531+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:51.903988+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:51.907734+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.052004+0000 mgr.y (mgr.24526) 390 : cluster [DBG] pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:53.273 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.126247+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.126638+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.126821+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.130788+0000 mon.a (mon.0) 2357 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.155961+0000 mon.c (mon.2) 594 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.156351+0000 mon.a (mon.0) 2358 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.157835+0000 mon.c (mon.2) 595 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.158147+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.158727+0000 mon.c (mon.2) 596 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: audit 2024-09-10T18:58:52.159035+0000 mon.a (mon.0) 2360 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.229724+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.232894+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.444277+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:58:53.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:53 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.447531+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:58:53.291 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:53.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:58:53.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:51.903988+0000 osd.1 (osd.1) 227 : cluster [DBG] 2.9 scrub starts 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:51.907734+0000 osd.1 (osd.1) 228 : cluster [DBG] 2.9 scrub ok 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.052004+0000 mgr.y (mgr.24526) 390 : cluster [DBG] pgmap v547: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.126247+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.126638+0000 mon.a (mon.0) 2355 : audit [INF] from='client.? v1:172.21.15.83:0/1886183667' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi083-77648-61"}]': finished 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.126821+0000 mon.a (mon.0) 2356 : cluster [DBG] osdmap e376: 8 total, 8 up, 8 in 2024-09-10T18:58:53.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.130788+0000 mon.a (mon.0) 2357 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.155961+0000 mon.c (mon.2) 594 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.156351+0000 mon.a (mon.0) 2358 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.157835+0000 mon.c (mon.2) 595 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.158147+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.158727+0000 mon.c (mon.2) 596 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: audit 2024-09-10T18:58:52.159035+0000 mon.a (mon.0) 2360 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.229724+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.19 scrub starts 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.232894+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.19 scrub ok 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.444277+0000 osd.5 (osd.5) 205 : cluster [DBG] 2.2 scrub starts 2024-09-10T18:58:53.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:53 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.447531+0000 osd.5 (osd.5) 206 : cluster [DBG] 2.2 scrub ok 2024-09-10T18:58:53.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:53.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:53.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:53.561 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:53.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:53.696 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:53.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:53.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:53.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:53.969 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:53.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:54.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:54.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:54.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:54.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:54.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:54.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:54.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:54.520 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:54.624 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:54.653 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:54.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.241992+0000 osd.7 (osd.7) 222 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.253360+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.878851+0000 osd.0 (osd.0) 206 : cluster [DBG] 64.1f scrub starts 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.880784+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.881830+0000 osd.0 (osd.0) 207 : cluster [DBG] 64.1f scrub ok 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:52.884073+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.131110+0000 mon.a (mon.0) 2361 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.131691+0000 mon.a (mon.0) 2362 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.131955+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.147640+0000 mon.c (mon.2) 597 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.153076+0000 mon.a (mon.0) 2364 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.771 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.186607+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: audit 2024-09-10T18:58:53.190385+0000 mon.a (mon.0) 2365 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.456317+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:58:54.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:54 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.460789+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.241992+0000 osd.7 (osd.7) 222 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.253360+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.878851+0000 osd.0 (osd.0) 206 : cluster [DBG] 64.1f scrub starts 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.880784+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.881830+0000 osd.0 (osd.0) 207 : cluster [DBG] 64.1f scrub ok 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:52.884073+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.131110+0000 mon.a (mon.0) 2361 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.131691+0000 mon.a (mon.0) 2362 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.131955+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.147640+0000 mon.c (mon.2) 597 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.153076+0000 mon.a (mon.0) 2364 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.186607+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: audit 2024-09-10T18:58:53.190385+0000 mon.a (mon.0) 2365 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.456317+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.460789+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:58:54.792 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.241992+0000 osd.7 (osd.7) 222 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.253360+0000 osd.7 (osd.7) 223 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.878851+0000 osd.0 (osd.0) 206 : cluster [DBG] 64.1f scrub starts 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.880784+0000 osd.1 (osd.1) 229 : cluster [DBG] 2.a deep-scrub starts 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.881830+0000 osd.0 (osd.0) 207 : cluster [DBG] 64.1f scrub ok 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:52.884073+0000 osd.1 (osd.1) 230 : cluster [DBG] 2.a deep-scrub ok 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.131110+0000 mon.a (mon.0) 2361 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.131691+0000 mon.a (mon.0) 2362 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.131955+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e377: 8 total, 8 up, 8 in 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.147640+0000 mon.c (mon.2) 597 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.153076+0000 mon.a (mon.0) 2364 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.186607+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: audit 2024-09-10T18:58:53.190385+0000 mon.a (mon.0) 2365 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:58:54.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.456317+0000 osd.5 (osd.5) 207 : cluster [DBG] 2.16 scrub starts 2024-09-10T18:58:54.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:54 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.460789+0000 osd.5 (osd.5) 208 : cluster [DBG] 2.16 scrub ok 2024-09-10T18:58:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:54.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:54.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:54.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:55.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:55.058 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:55.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:55.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:55.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:55.327 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:55.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:55.464 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:55.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:55.601 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:55.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:55.738 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.492688+0000 osd.4 (osd.4) 249 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.495263+0000 osd.4 (osd.4) 250 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.946056+0000 osd.2 (osd.2) 155 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:53.953833+0000 osd.2 (osd.2) 156 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.052815+0000 mgr.y (mgr.24526) 391 : cluster [DBG] pgmap v550: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.228261+0000 osd.7 (osd.7) 224 : cluster [DBG] 12.d scrub starts 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.237576+0000 osd.7 (osd.7) 225 : cluster [DBG] 12.d scrub ok 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:54.525251+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:54.526703+0000 mon.a (mon.0) 2366 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.526861+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:54.538075+0000 mon.a (mon.0) 2368 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.910100+0000 osd.0 (osd.0) 208 : cluster [DBG] 4.b scrub starts 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.918303+0000 osd.0 (osd.0) 209 : cluster [DBG] 4.b scrub ok 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:55.098668+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:55.099382+0000 mon.a (mon.0) 2369 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:55.099672+0000 mon.a (mon.0) 2370 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.099859+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[18629]: audit 2024-09-10T18:58:55.102523+0000 mon.a (mon.0) 2372 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.492688+0000 osd.4 (osd.4) 249 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.495263+0000 osd.4 (osd.4) 250 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.946056+0000 osd.2 (osd.2) 155 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:53.953833+0000 osd.2 (osd.2) 156 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:58:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.052815+0000 mgr.y (mgr.24526) 391 : cluster [DBG] pgmap v550: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.228261+0000 osd.7 (osd.7) 224 : cluster [DBG] 12.d scrub starts 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.237576+0000 osd.7 (osd.7) 225 : cluster [DBG] 12.d scrub ok 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:54.525251+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:54.526703+0000 mon.a (mon.0) 2366 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.526861+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:54.538075+0000 mon.a (mon.0) 2368 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.910100+0000 osd.0 (osd.0) 208 : cluster [DBG] 4.b scrub starts 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.918303+0000 osd.0 (osd.0) 209 : cluster [DBG] 4.b scrub ok 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:55.098668+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:55.099382+0000 mon.a (mon.0) 2369 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:55.099672+0000 mon.a (mon.0) 2370 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.099859+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-10T18:58:55.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:55 smithi083 bash[32381]: audit 2024-09-10T18:58:55.102523+0000 mon.a (mon.0) 2372 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.492688+0000 osd.4 (osd.4) 249 : cluster [DBG] 4.1b scrub starts 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.495263+0000 osd.4 (osd.4) 250 : cluster [DBG] 4.1b scrub ok 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.946056+0000 osd.2 (osd.2) 155 : cluster [DBG] 64.7 scrub starts 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:53.953833+0000 osd.2 (osd.2) 156 : cluster [DBG] 64.7 scrub ok 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.052815+0000 mgr.y (mgr.24526) 391 : cluster [DBG] pgmap v550: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.228261+0000 osd.7 (osd.7) 224 : cluster [DBG] 12.d scrub starts 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.237576+0000 osd.7 (osd.7) 225 : cluster [DBG] 12.d scrub ok 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:54.525251+0000 mon.b (mon.1) 295 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:54.526703+0000 mon.a (mon.0) 2366 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.526861+0000 mon.a (mon.0) 2367 : cluster [DBG] osdmap e378: 8 total, 8 up, 8 in 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:54.538075+0000 mon.a (mon.0) 2368 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.910100+0000 osd.0 (osd.0) 208 : cluster [DBG] 4.b scrub starts 2024-09-10T18:58:55.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.918303+0000 osd.0 (osd.0) 209 : cluster [DBG] 4.b scrub ok 2024-09-10T18:58:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:55.098668+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:55.099382+0000 mon.a (mon.0) 2369 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi083-77648-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:55.099672+0000 mon.a (mon.0) 2370 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:58:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.099859+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e379: 8 total, 8 up, 8 in 2024-09-10T18:58:55.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:55 smithi165 bash[22591]: audit 2024-09-10T18:58:55.102523+0000 mon.a (mon.0) 2372 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]: dispatch 2024-09-10T18:58:55.873 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:55.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:55.886 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:55.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:56.011 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:56.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:56.135 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:56.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.271 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:56.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.413 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:56.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:56.553 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:56.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:56.690 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:56.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.837992+0000 osd.1 (osd.1) 231 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:58:56.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:54.847253+0000 osd.1 (osd.1) 232 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:58:56.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.256801+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:58:56.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.261612+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.636501+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.639768+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.917445+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.922508+0000 osd.0 (osd.0) 211 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.054232+0000 mgr.y (mgr.24526) 392 : cluster [DBG] pgmap v553: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.091608+0000 mon.a (mon.0) 2373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: audit 2024-09-10T18:58:56.102328+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: audit 2024-09-10T18:58:56.103141+0000 mon.a (mon.0) 2374 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]': finished 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.103534+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-10T18:58:56.807 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:56 smithi165 bash[22591]: audit 2024-09-10T18:58:56.106436+0000 mon.a (mon.0) 2376 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:56.825 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:58:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:58:56.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:56.964 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:56.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.837992+0000 osd.1 (osd.1) 231 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:54.847253+0000 osd.1 (osd.1) 232 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.256801+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.261612+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.636501+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.639768+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.917445+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.922508+0000 osd.0 (osd.0) 211 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:58:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.054232+0000 mgr.y (mgr.24526) 392 : cluster [DBG] pgmap v553: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.091608+0000 mon.a (mon.0) 2373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: audit 2024-09-10T18:58:56.102328+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: audit 2024-09-10T18:58:56.103141+0000 mon.a (mon.0) 2374 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]': finished 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.103534+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[18629]: audit 2024-09-10T18:58:56.106436+0000 mon.a (mon.0) 2376 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.837992+0000 osd.1 (osd.1) 231 : cluster [DBG] 64.1c scrub starts 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:54.847253+0000 osd.1 (osd.1) 232 : cluster [DBG] 64.1c scrub ok 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.256801+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub starts 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.261612+0000 osd.7 (osd.7) 227 : cluster [DBG] 6.8 scrub ok 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.636501+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.14 scrub starts 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.639768+0000 osd.6 (osd.6) 172 : cluster [DBG] 2.14 scrub ok 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.917445+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.1 scrub starts 2024-09-10T18:58:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.922508+0000 osd.0 (osd.0) 211 : cluster [DBG] 3.1 scrub ok 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.054232+0000 mgr.y (mgr.24526) 392 : cluster [DBG] pgmap v553: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.091608+0000 mon.a (mon.0) 2373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: audit 2024-09-10T18:58:56.102328+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: audit 2024-09-10T18:58:56.103141+0000 mon.a (mon.0) 2374 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-47", "mode": "writeback"}]': finished 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.103534+0000 mon.a (mon.0) 2375 : cluster [DBG] osdmap e380: 8 total, 8 up, 8 in 2024-09-10T18:58:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:56 smithi083 bash[32381]: audit 2024-09-10T18:58:56.106436+0000 mon.a (mon.0) 2376 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:58:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:57.046 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:57.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:57.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:57.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:57.246 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:57.349 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:58:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:58:57.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.517 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:57.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:57.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.807382+0000 osd.1 (osd.1) 233 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:55.811175+0000 osd.1 (osd.1) 234 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.285852+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.293007+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.445438+0000 osd.4 (osd.4) 251 : cluster [DBG] 64.d scrub starts 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.450164+0000 osd.4 (osd.4) 252 : cluster [DBG] 64.d scrub ok 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:56.532437+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: audit 2024-09-10T18:58:57.102404+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: audit 2024-09-10T18:58:57.103613+0000 mon.a (mon.0) 2378 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.103869+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: audit 2024-09-10T18:58:57.106380+0000 mon.a (mon.0) 2380 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: audit 2024-09-10T18:58:57.110347+0000 mon.c (mon.2) 598 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: audit 2024-09-10T18:58:57.110800+0000 mon.a (mon.0) 2381 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.256054+0000 osd.3 (osd.3) 241 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.807382+0000 osd.1 (osd.1) 233 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:55.811175+0000 osd.1 (osd.1) 234 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.285852+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.293007+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.445438+0000 osd.4 (osd.4) 251 : cluster [DBG] 64.d scrub starts 2024-09-10T18:58:57.777 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.450164+0000 osd.4 (osd.4) 252 : cluster [DBG] 64.d scrub ok 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:56.532437+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: audit 2024-09-10T18:58:57.102404+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: audit 2024-09-10T18:58:57.103613+0000 mon.a (mon.0) 2378 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.103869+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: audit 2024-09-10T18:58:57.106380+0000 mon.a (mon.0) 2380 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: audit 2024-09-10T18:58:57.110347+0000 mon.c (mon.2) 598 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: audit 2024-09-10T18:58:57.110800+0000 mon.a (mon.0) 2381 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:57 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.256054+0000 osd.3 (osd.3) 241 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:58:57.791 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.807382+0000 osd.1 (osd.1) 233 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T18:58:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:55.811175+0000 osd.1 (osd.1) 234 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T18:58:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.285852+0000 osd.7 (osd.7) 228 : cluster [DBG] 3.13 scrub starts 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.293007+0000 osd.7 (osd.7) 229 : cluster [DBG] 3.13 scrub ok 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.445438+0000 osd.4 (osd.4) 251 : cluster [DBG] 64.d scrub starts 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.450164+0000 osd.4 (osd.4) 252 : cluster [DBG] 64.d scrub ok 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:56.532437+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: audit 2024-09-10T18:58:57.102404+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: audit 2024-09-10T18:58:57.103613+0000 mon.a (mon.0) 2378 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.103869+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e381: 8 total, 8 up, 8 in 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: audit 2024-09-10T18:58:57.106380+0000 mon.a (mon.0) 2380 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: audit 2024-09-10T18:58:57.110347+0000 mon.c (mon.2) 598 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: audit 2024-09-10T18:58:57.110800+0000 mon.a (mon.0) 2381 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:57.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.256054+0000 osd.3 (osd.3) 241 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:58:57.894 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.926 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:58:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:57.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:58:58.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:58.060 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:58.063 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:58.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.202 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:58.227 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:58:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:58:58.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.340 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:58.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.476 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:58.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:58.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.753 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:58.857 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:58:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:58.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:58:58.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:59.029 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:59.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.435422+0000 osd.4 (osd.4) 253 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.446513+0000 osd.4 (osd.4) 254 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.765379+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:57.770120+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:57.900984+0000 mgr.y (mgr.24526) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.055076+0000 mgr.y (mgr.24526) 394 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.104239+0000 mon.a (mon.0) 2382 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.104381+0000 mon.a (mon.0) 2383 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.104491+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.105441+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.108479+0000 mon.c (mon.2) 599 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.141 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.109350+0000 mon.a (mon.0) 2385 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: audit 2024-09-10T18:58:58.109777+0000 mon.a (mon.0) 2386 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.274776+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.281996+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.435422+0000 osd.4 (osd.4) 253 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.446513+0000 osd.4 (osd.4) 254 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.765379+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:57.770120+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:57.900984+0000 mgr.y (mgr.24526) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:59.142 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.055076+0000 mgr.y (mgr.24526) 394 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.104239+0000 mon.a (mon.0) 2382 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.104381+0000 mon.a (mon.0) 2383 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.104491+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.105441+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.108479+0000 mon.c (mon.2) 599 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.109350+0000 mon.a (mon.0) 2385 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: audit 2024-09-10T18:58:58.109777+0000 mon.a (mon.0) 2386 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.274776+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:58:59.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:58:59 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.281996+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:58:59.158 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:58:59.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.435422+0000 osd.4 (osd.4) 253 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.446513+0000 osd.4 (osd.4) 254 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.765379+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.6 scrub starts 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:57.770120+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.6 scrub ok 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:57.900984+0000 mgr.y (mgr.24526) 393 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.055076+0000 mgr.y (mgr.24526) 394 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.104239+0000 mon.a (mon.0) 2382 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.104381+0000 mon.a (mon.0) 2383 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:58:59.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.104491+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e382: 8 total, 8 up, 8 in 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.105441+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.108479+0000 mon.c (mon.2) 599 : audit [INF] from='client.? v1:172.21.15.83:0/1989053279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.109350+0000 mon.a (mon.0) 2385 : audit [INF] from='client.49313 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]: dispatch 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: audit 2024-09-10T18:58:58.109777+0000 mon.a (mon.0) 2386 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.274776+0000 osd.3 (osd.3) 242 : cluster [DBG] 3.2 scrub starts 2024-09-10T18:58:59.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:58:59 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.281996+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.2 scrub ok 2024-09-10T18:58:59.290 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:59.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:59.383 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T18:58:59.383 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-10T18:58:59.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-10T18:58:59.383 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-10T18:58:59.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.28627.0:5 66.c 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:01.890647+0000", 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.48405482999999, 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10262373599999999, 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.28627", 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4009038338", 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890647+0000", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890647+0000", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890645+0000", 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T18:58:59.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890655+0000", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0265000000000001e-05 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890667+0000", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890676+0000", 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7560000000000005e-06 2024-09-10T18:58:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898312+0000", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076363990000000003 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898418+0000", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105642 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898609+0000", 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019089899999999999 2024-09-10T18:58:59.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.908199+0000", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095898189999999994 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.908363+0000", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016451399999999999 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993116+0000", 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.084753172000000002 2024-09-10T18:58:59.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993218+0000", 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102069 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993271+0000", 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2924999999999999e-05 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.33280.0:2 72.6 72:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-10T18:58:59.389 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:03.454746+0000", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 235.91995662100001, 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.079654984999999998, 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.33280", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/2788138944", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454746+0000", 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454746+0000", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454743+0000", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454753+0000", 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0618999999999999e-05 2024-09-10T18:58:59.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454767+0000", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3655000000000001e-05 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454776+0000", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7560000000000005e-06 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.470739+0000", 2024-09-10T18:58:59.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015963356000000001 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.470746+0000", 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-10T18:58:59.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.482837+0000", 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012090386999999999 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.482846+0000", 2024-09-10T18:58:59.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0299999999999999e-06 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520469+0000", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037623707999999999 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520651+0000", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018204199999999999 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520847+0000", 2024-09-10T18:58:59.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000195231 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.521329+0000", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048234599999999998 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.521779+0000", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045026799999999998 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534312+0000", 2024-09-10T18:58:59.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012532585000000001 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534363+0000", 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1341000000000002e-05 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534401+0000", 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7416000000000002e-05 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.28627.0:9 66.c 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e149)", 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:04.450461+0000", 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.924240911, 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073743234000000005, 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.28627", 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4009038338", 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 9 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450461+0000", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450461+0000", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450458+0000", 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450472+0000", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4127e-05 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450504+0000", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1578999999999997e-05 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450519+0000", 2024-09-10T18:58:59.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450673+0000", 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015369099999999999 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450689+0000", 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6147999999999999e-05 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471156+0000", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020466640000000001 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471261+0000", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104817 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,5", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471420+0000", 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015986900000000001 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.479203+0000", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077824490000000003 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.487328+0000", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081247579999999993 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524074+0000", 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036746094999999999 2024-09-10T18:58:59.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524141+0000", 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6767999999999995e-05 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524205+0000", 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4077000000000003e-05 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35008.0:2 73.1f 73:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:04.484368+0000", 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 234.890334741, 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058343063000000001, 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35008", 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/466281865", 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T18:58:59.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484368+0000", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484368+0000", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484365+0000", 2024-09-10T18:58:59.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484378+0000", 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2665e-05 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484390+0000", 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2615e-05 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484415+0000", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4341999999999998e-05 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484466+0000", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1864999999999999e-05 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484475+0000", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3219999999999993e-06 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510456+0000", 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025981364 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510580+0000", 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123545 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510696+0000", 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116555 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.511041+0000", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034510399999999999 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.540117+0000", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029075515 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542572+0000", 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024546239999999999 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542665+0000", 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3058000000000002e-05 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542711+0000", 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6054999999999999e-05 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:224 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.737589+0000", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.63711348699999, 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027100756, 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737589+0000", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737589+0000", 2024-09-10T18:58:59.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737584+0000", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746765+0000", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091805819999999996 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746788+0000", 2024-09-10T18:58:59.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2648000000000001e-05 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746795+0000", 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8380000000000002e-06 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746877+0000", 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1819999999999999e-05 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746921+0000", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3693000000000003e-05 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.764625+0000", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017703855000000001 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.764690+0000", 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4727999999999994e-05 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.417 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:225 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.765592+0000", 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.60910989600001, 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024732403, 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765592+0000", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765592+0000", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765590+0000", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770186+0000", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045959460000000001 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770237+0000", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1035000000000002e-05 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770249+0000", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510999999999999e-05 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.771257+0000", 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010075220000000001 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.771286+0000", 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9459999999999999e-05 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790249+0000", 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018962372000000002 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790325+0000", 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5965999999999997e-05 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:226 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.790974+0000", 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.58372816599999, 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018381779000000001, 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790974+0000", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790974+0000", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790972+0000", 2024-09-10T18:58:59.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795203+0000", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042301700000000001 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795236+0000", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2966000000000003e-05 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795245+0000", 2024-09-10T18:58:59.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8549999999999997e-06 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795347+0000", 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010155299999999999 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795377+0000", 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9929000000000001e-05 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.809306+0000", 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013929182 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.809356+0000", 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9796999999999998e-05 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.6.0:339 11.3 e155/124 -13:c5d8c2df:::temp_11.3_0_24313_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.833266+0000", 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.541436586, 2024-09-10T18:58:59.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021768810999999999, 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833266+0000", 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833266+0000", 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833265+0000", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838111+0000", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004845388 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838167+0000", 2024-09-10T18:58:59.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.643e-05 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838186+0000", 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9040000000000001e-05 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838286+0000", 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9642000000000006e-05 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838324+0000", 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8491999999999997e-05 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.854981+0000", 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016656823000000001 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.855035+0000", 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3582000000000003e-05 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.6.0:340 11.3 e155/124 -13:c5d8c2df:::temp_11.3_0_24313_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.868202+0000", 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.506500421, 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02096164, 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.431 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868202+0000", 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868202+0000", 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868198+0000", 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873323+0000", 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051244960000000001 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873366+0000", 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3547000000000002e-05 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873386+0000", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9375e-05 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873465+0000", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9726000000000002e-05 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873487+0000", 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2079e-05 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.889103+0000", 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015615891999999999 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.889164+0000", 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0149000000000003e-05 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.435 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.6.0:341 11.3 e155/124 -13:c5d8c2df:::temp_11.3_0_24313_1:head v 0'0, mlcod=0'0)", 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.897795+0000", 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 227.476907775, 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018094632999999999, 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.897795+0000", 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.897795+0000", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.897791+0000", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.902189+0000", 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043985539999999998 2024-09-10T18:58:59.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.902234+0000", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5244000000000003e-05 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.902246+0000", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1191e-05 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.902303+0000", 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7423000000000002e-05 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.902326+0000", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2799999999999999e-05 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.915822+0000", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013496107 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.915889+0000", 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7212999999999995e-05 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.439 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.24699.0:2 111.6 111:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:45.971302+0000", 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 193.40340015699999, 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.063178365, 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.24699", 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1632292978", 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.440 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971302+0000", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971302+0000", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971299+0000", 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T18:58:59.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971309+0000", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6299999999999993e-06 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971321+0000", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2330999999999999e-05 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971330+0000", 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9430000000000006e-06 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971392+0000", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2025000000000003e-05 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971399+0000", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7089999999999996e-06 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.992622+0000", 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021222477 2024-09-10T18:58:59.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.992630+0000", 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3049999999999993e-06 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024190+0000", 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031560438000000003 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024336+0000", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145344 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024507+0000", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017166499999999999 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.025077+0000", 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056921300000000005 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.027024+0000", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00194736 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034341+0000", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073167370000000002 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034401+0000", 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0884999999999997e-05 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034481+0000", 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9021000000000001e-05 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.24732.0:2 118.11 118:8e20cd32:test-rados-api-smithi083-77788-26::foo:head [writefull 0~8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:53.983682+0000", 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 185.39102022899999, 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058042643999999997, 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.24732", 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4260234155", 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983682+0000", 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983682+0000", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983680+0000", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995043+0000", 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011363009 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995068+0000", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4885e-05 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995078+0000", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0448999999999999e-05 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995241+0000", 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016265999999999999 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995489+0000", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024785900000000001 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995841+0000", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035254999999999998 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.013679+0000", 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017837493999999999 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.033265+0000", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019586587999999999 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041553+0000", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082874439999999997 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041675+0000", 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012207 2024-09-10T18:58:59.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041725+0000", 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9957000000000003e-05 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:1538 141.3 141:c3978ec1:test-rados-api-smithi083-77828-18::bar:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:27.038867+0000", 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 152.33583521, 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032955234999999999, 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T18:58:59.453 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1538 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038867+0000", 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038867+0000", 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038864+0000", 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038873+0000", 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038888+0000", 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.45e-05 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038894+0000", 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4840000000000001e-06 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.039184+0000", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028970999999999999 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.039192+0000", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5950000000000001e-06 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.041143+0000", 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001951709 2024-09-10T18:58:59.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.041357+0000", 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021345700000000001 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.061665+0000", 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020307819000000001 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.061814+0000", 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014969000000000001 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.071822+0000", 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010007920999999999 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.48947.0:2 151.1e 151:7cdf4dc8:::oid_XattrsRoundTripPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:40.085858+0000", 2024-09-10T18:58:59.459 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 139.28884465100001, 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028557097, 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.48947", 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1872315293", 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085858+0000", 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085858+0000", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085855+0000", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085865+0000", 2024-09-10T18:58:59.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0526000000000001e-05 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085881+0000", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5911999999999999e-05 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085889+0000", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7910000000000005e-06 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.091064+0000", 2024-09-10T18:58:59.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051752680000000002 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.091074+0000", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4010000000000003e-06 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112086+0000", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021012005 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112224+0000", 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013811800000000001 2024-09-10T18:58:59.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112386+0000", 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016208100000000001 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112991+0000", 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060498400000000003 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114306+0000", 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013147180000000001 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114352+0000", 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6020000000000003e-05 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114388+0000", 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6556e-05 2024-09-10T18:58:59.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114415+0000", 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6568e-05 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.48947.0:4 151.1e 151:7cdf4dc8:::oid_XattrsRoundTripPP_:head [setxattr attr1 (12) in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:42.091646+0000", 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 137.283055875, 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023530008000000002, 2024-09-10T18:58:59.466 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.48947", 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1872315293", 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091646+0000", 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091646+0000", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091643+0000", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091656+0000", 2024-09-10T18:58:59.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510000000000001e-05 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091680+0000", 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4394999999999999e-05 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091690+0000", 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9850000000000001e-06 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091784+0000", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4003000000000001e-05 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091904+0000", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119968 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.092079+0000", 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017448299999999999 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.092809+0000", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073068899999999997 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.108846+0000", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016036594000000001 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115065+0000", 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062191290000000003 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115133+0000", 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8429000000000005e-05 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115176+0000", 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3093999999999998e-05 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2189 64.1d 64:b8b5cecb:test-rados-api-smithi083-77828-28::foo:head [writefull 0~5] snapc 11=[11] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-10T18:58:59.472 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:08.065256+0000", 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 111.30944606200001, 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020636341999999998, 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2189 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065256+0000", 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065256+0000", 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065253+0000", 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065267+0000", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3886999999999999e-05 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065284+0000", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7597000000000001e-05 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065295+0000", 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-10T18:58:59.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065387+0000", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2594000000000001e-05 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065399+0000", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2310000000000001e-05 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074391+0000", 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089920130000000001 2024-09-10T18:58:59.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074498+0000", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010673100000000001 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074689+0000", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019112299999999999 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.077280+0000", 2024-09-10T18:58:59.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025910450000000002 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.078598+0000", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013178689999999999 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085819+0000", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072206199999999996 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085860+0000", 2024-09-10T18:58:59.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0979000000000002e-05 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085893+0000", 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.273e-05 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2299 64.6 64:646b25c2:test-rados-api-smithi083-77828-30::foo:head [writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:17.494930+0000", 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 101.87977254, 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021475062, 2024-09-10T18:58:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2299 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494930+0000", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494930+0000", 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494925+0000", 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501000+0000", 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060746059999999998 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501046+0000", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6699000000000001e-05 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501063+0000", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6350000000000001e-05 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501166+0000", 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103325 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501335+0000", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000168992 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501499+0000", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016421 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.512384+0000", 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010885018999999999 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.515420+0000", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030353709999999998 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516115+0000", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069553900000000001 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516284+0000", 2024-09-10T18:58:59.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016939599999999999 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516405+0000", 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012037200000000001 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2302 64.6 64:646b25c2:test-rados-api-smithi083-77828-30::foo:head [writefull 0~5] snapc 14=[14,13] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:19.038678+0000", 2024-09-10T18:58:59.485 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.336024017, 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022007105999999999, 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2302 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038678+0000", 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038678+0000", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038674+0000", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038695+0000", 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1912999999999998e-05 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038747+0000", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1486000000000001e-05 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038769+0000", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1739999999999999e-05 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038950+0000", 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018124100000000001 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038977+0000", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6874000000000001e-05 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052474+0000", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013497055000000001 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052577+0000", 2024-09-10T18:58:59.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103589 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052780+0000", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020221099999999999 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.054276+0000", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014968329999999999 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.055530+0000", 2024-09-10T18:58:59.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012535039999999999 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060595+0000", 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050655250000000004 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060644+0000", 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8114e-05 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060685+0000", 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1786999999999998e-05 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.25170.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:58:24.092116+0000", 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.282586232, 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009413197, 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.25170", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1833223569", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092116+0000", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092116+0000", 2024-09-10T18:58:59.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092114+0000", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092122+0000", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4670000000000004e-06 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092135+0000", 2024-09-10T18:58:59.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.308e-05 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092144+0000", 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3909999999999998e-06 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.093221+0000", 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010772430000000001 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.093225+0000", 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1699999999999999e-06 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.097615+0000", 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.004389607 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.097806+0000", 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019083199999999999 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.098187+0000", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0003814 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.098953+0000", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076624899999999999 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.101477+0000", 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025242659999999998 2024-09-10T18:58:59.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.101529+0000", 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1836999999999999e-05 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.25176.0:1 195.6s0 195:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:58:27.093700+0000", 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.281002479000001, 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0292627759999999, 2024-09-10T18:58:59.498 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.25176", 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3151444780", 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093700+0000", 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093700+0000", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093695+0000", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093714+0000", 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9188999999999999e-05 2024-09-10T18:58:59.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093743+0000", 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8665000000000001e-05 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093760+0000", 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7371e-05 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.096045+0000", 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002284861 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.096052+0000", 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5370000000000003e-06 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121277+0000", 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0252254999999999 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.502 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121413+0000", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013598500000000001 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121815+0000", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040199300000000001 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122598+0000", 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078350900000000005 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122881+0000", 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028243400000000003 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122963+0000", 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1678e-05 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T18:58:59.504 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T18:58:59.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:59.536 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:58:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:58:59.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:59.674 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:58:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:58:59.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:59.811 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:58:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:58:59.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:59.948 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:58:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:58:59.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:00.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:00.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T18:59:00.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:00.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:00.209 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.243701+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.248211+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.771809+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:58.776316+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.101802+0000 mon.a (mon.0) 2387 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.109316+0000 mon.a (mon.0) 2388 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.109495+0000 mon.a (mon.0) 2389 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.109626+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.111867+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.124554+0000 mon.a (mon.0) 2391 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.144639+0000 mon.c (mon.2) 600 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.144980+0000 mon.a (mon.0) 2392 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.146487+0000 mon.c (mon.2) 601 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.146866+0000 mon.a (mon.0) 2393 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.147455+0000 mon.c (mon.2) 602 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.147786+0000 mon.a (mon.0) 2394 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.313242+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.323523+0000 osd.3 (osd.3) 245 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.450317+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.453897+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.570084+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: audit 2024-09-10T18:58:59.572612+0000 mon.c (mon.2) 603 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.656873+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:59:00.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.660383+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.243701+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.248211+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.771809+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:58.776316+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.101802+0000 mon.a (mon.0) 2387 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.109316+0000 mon.a (mon.0) 2388 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.109495+0000 mon.a (mon.0) 2389 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.109626+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-10T18:59:00.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.111867+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.124554+0000 mon.a (mon.0) 2391 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.144639+0000 mon.c (mon.2) 600 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.144980+0000 mon.a (mon.0) 2392 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.146487+0000 mon.c (mon.2) 601 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.146866+0000 mon.a (mon.0) 2393 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.147455+0000 mon.c (mon.2) 602 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.147786+0000 mon.a (mon.0) 2394 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.313242+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.323523+0000 osd.3 (osd.3) 245 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.450317+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.453897+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.570084+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: audit 2024-09-10T18:58:59.572612+0000 mon.c (mon.2) 603 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.656873+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:59:00.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:00 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.660383+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:59:00.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.327 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.243701+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1c scrub starts 2024-09-10T18:59:00.327 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.248211+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.1c scrub ok 2024-09-10T18:59:00.327 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.771809+0000 osd.1 (osd.1) 237 : cluster [DBG] 2.15 scrub starts 2024-09-10T18:59:00.327 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:58.776316+0000 osd.1 (osd.1) 238 : cluster [DBG] 2.15 scrub ok 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.101802+0000 mon.a (mon.0) 2387 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.109316+0000 mon.a (mon.0) 2388 : audit [INF] from='client.49313 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi083-77648-62"}]': finished 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.109495+0000 mon.a (mon.0) 2389 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.109626+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e383: 8 total, 8 up, 8 in 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.111867+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.124554+0000 mon.a (mon.0) 2391 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.144639+0000 mon.c (mon.2) 600 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.144980+0000 mon.a (mon.0) 2392 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.146487+0000 mon.c (mon.2) 601 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.146866+0000 mon.a (mon.0) 2393 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.147455+0000 mon.c (mon.2) 602 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.147786+0000 mon.a (mon.0) 2394 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.313242+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.7 scrub starts 2024-09-10T18:59:00.328 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.323523+0000 osd.3 (osd.3) 245 : cluster [DBG] 3.7 scrub ok 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.450317+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.7 scrub starts 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.453897+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.7 scrub ok 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.570084+0000 mon.a (mon.0) 2395 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: audit 2024-09-10T18:58:59.572612+0000 mon.c (mon.2) 603 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.656873+0000 osd.6 (osd.6) 173 : cluster [DBG] 2.1a scrub starts 2024-09-10T18:59:00.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:00 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.660383+0000 osd.6 (osd.6) 174 : cluster [DBG] 2.1a scrub ok 2024-09-10T18:59:00.346 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:00.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:00.485 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:00.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:00.622 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:00.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.761 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:00.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.900 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:00.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:01.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.038 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:01.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:01.166 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:01.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:01.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.432 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:01.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:01.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:01.550 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:01.574 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:01.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:01.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.478852+0000 osd.4 (osd.4) 255 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:59:01.691 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.481785+0000 osd.4 (osd.4) 256 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.776851+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.19 deep-scrub starts 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.788472+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.19 deep-scrub ok 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.926808+0000 osd.0 (osd.0) 212 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:58:59.930300+0000 osd.0 (osd.0) 213 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.055872+0000 mgr.y (mgr.24526) 395 : cluster [DBG] pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.132658+0000 mon.a (mon.0) 2396 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.132740+0000 mon.a (mon.0) 2397 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.133035+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.133089+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.692 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.134726+0000 mon.c (mon.2) 604 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.135221+0000 mon.a (mon.0) 2399 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: audit 2024-09-10T18:59:00.136689+0000 mon.a (mon.0) 2400 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.640763+0000 osd.6 (osd.6) 175 : cluster [DBG] 12.17 deep-scrub starts 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.644288+0000 osd.6 (osd.6) 176 : cluster [DBG] 12.17 deep-scrub ok 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.478852+0000 osd.4 (osd.4) 255 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.481785+0000 osd.4 (osd.4) 256 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.776851+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.19 deep-scrub starts 2024-09-10T18:59:01.693 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.788472+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.19 deep-scrub ok 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.926808+0000 osd.0 (osd.0) 212 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:58:59.930300+0000 osd.0 (osd.0) 213 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.055872+0000 mgr.y (mgr.24526) 395 : cluster [DBG] pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.132658+0000 mon.a (mon.0) 2396 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.132740+0000 mon.a (mon.0) 2397 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.133035+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.133089+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.134726+0000 mon.c (mon.2) 604 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.135221+0000 mon.a (mon.0) 2399 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: audit 2024-09-10T18:59:00.136689+0000 mon.a (mon.0) 2400 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.694 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.640763+0000 osd.6 (osd.6) 175 : cluster [DBG] 12.17 deep-scrub starts 2024-09-10T18:59:01.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:01 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.644288+0000 osd.6 (osd.6) 176 : cluster [DBG] 12.17 deep-scrub ok 2024-09-10T18:59:01.711 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:01.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.478852+0000 osd.4 (osd.4) 255 : cluster [DBG] 12.1e scrub starts 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.481785+0000 osd.4 (osd.4) 256 : cluster [DBG] 12.1e scrub ok 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.776851+0000 osd.1 (osd.1) 239 : cluster [DBG] 3.19 deep-scrub starts 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.788472+0000 osd.1 (osd.1) 240 : cluster [DBG] 3.19 deep-scrub ok 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.926808+0000 osd.0 (osd.0) 212 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:58:59.930300+0000 osd.0 (osd.0) 213 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T18:59:01.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.055872+0000 mgr.y (mgr.24526) 395 : cluster [DBG] pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.132658+0000 mon.a (mon.0) 2396 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.132740+0000 mon.a (mon.0) 2397 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi083-77648-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.133035+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e384: 8 total, 8 up, 8 in 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.133089+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.134726+0000 mon.c (mon.2) 604 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.135221+0000 mon.a (mon.0) 2399 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: audit 2024-09-10T18:59:00.136689+0000 mon.a (mon.0) 2400 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.640763+0000 osd.6 (osd.6) 175 : cluster [DBG] 12.17 deep-scrub starts 2024-09-10T18:59:01.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:01 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.644288+0000 osd.6 (osd.6) 176 : cluster [DBG] 12.17 deep-scrub ok 2024-09-10T18:59:01.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:01.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:01.985 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:01.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:02.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:02.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:02.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:02.262 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:02.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:02.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:02.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:02.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:02.656 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.903171+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T18:59:02.656 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:00.907031+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T18:59:02.656 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.272385+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:59:02.656 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.274332+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:59:02.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.484421+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.c scrub starts 2024-09-10T18:59:02.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.489047+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.c scrub ok 2024-09-10T18:59:02.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:01.586303+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:01.587304+0000 mon.a (mon.0) 2401 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.587490+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:01.600666+0000 mon.a (mon.0) 2403 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.660584+0000 osd.6 (osd.6) 177 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.663635+0000 osd.6 (osd.6) 178 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.885756+0000 osd.2 (osd.2) 159 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:59:02.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.892210+0000 osd.2 (osd.2) 160 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.056759+0000 mgr.y (mgr.24526) 396 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:02.140343+0000 mon.a (mon.0) 2404 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:02.140783+0000 mon.a (mon.0) 2405 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.142609+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:02.181590+0000 mon.b (mon.1) 303 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18629]: audit 2024-09-10T18:59:02.185284+0000 mon.a (mon.0) 2407 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.660 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.903171+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T18:59:02.660 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:00.907031+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T18:59:02.660 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.272385+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:59:02.660 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.274332+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:59:02.660 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.484421+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.c scrub starts 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.489047+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.c scrub ok 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:01.586303+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:01.587304+0000 mon.a (mon.0) 2401 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.587490+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:01.600666+0000 mon.a (mon.0) 2403 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.660584+0000 osd.6 (osd.6) 177 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.663635+0000 osd.6 (osd.6) 178 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.885756+0000 osd.2 (osd.2) 159 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.892210+0000 osd.2 (osd.2) 160 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.056759+0000 mgr.y (mgr.24526) 396 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:02.140343+0000 mon.a (mon.0) 2404 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:02.662 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:02.140783+0000 mon.a (mon.0) 2405 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T18:59:02.663 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.142609+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-10T18:59:02.663 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:02.181590+0000 mon.b (mon.1) 303 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.663 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[32381]: audit 2024-09-10T18:59:02.185284+0000 mon.a (mon.0) 2407 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.677 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:02.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:02.813 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.903171+0000 osd.2 (osd.2) 157 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:00.907031+0000 osd.2 (osd.2) 158 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.272385+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.11 scrub starts 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.274332+0000 osd.3 (osd.3) 247 : cluster [DBG] 4.11 scrub ok 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.484421+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.c scrub starts 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.489047+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.c scrub ok 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:01.586303+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:01.587304+0000 mon.a (mon.0) 2401 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.587490+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e385: 8 total, 8 up, 8 in 2024-09-10T18:59:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:01.600666+0000 mon.a (mon.0) 2403 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.660584+0000 osd.6 (osd.6) 177 : cluster [DBG] 12.1f scrub starts 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.663635+0000 osd.6 (osd.6) 178 : cluster [DBG] 12.1f scrub ok 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.885756+0000 osd.2 (osd.2) 159 : cluster [DBG] 64.15 scrub starts 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.892210+0000 osd.2 (osd.2) 160 : cluster [DBG] 64.15 scrub ok 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.056759+0000 mgr.y (mgr.24526) 396 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:02.140343+0000 mon.a (mon.0) 2404 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi083-77648-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:02.140783+0000 mon.a (mon.0) 2405 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.142609+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e386: 8 total, 8 up, 8 in 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:02.181590+0000 mon.b (mon.1) 303 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:02 smithi165 bash[22591]: audit 2024-09-10T18:59:02.185284+0000 mon.a (mon.0) 2407 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:02.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:02.951 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:02.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:03.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:03.071 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:02] "GET /metrics HTTP/1.1" 200 228750 "" "Prometheus/2.43.0" 2024-09-10T18:59:03.085 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:03.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:03.222 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:03.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:03.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:03.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:03.496 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:03.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:03.629 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:03.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.787995+0000 osd.1 (osd.1) 241 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:01.793662+0000 osd.1 (osd.1) 242 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.440678+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.444091+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.709161+0000 osd.6 (osd.6) 179 : cluster [DBG] 12.f scrub starts 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.712824+0000 osd.6 (osd.6) 180 : cluster [DBG] 12.f scrub ok 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: audit 2024-09-10T18:59:03.144528+0000 mon.a (mon.0) 2408 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:03.745 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: audit 2024-09-10T18:59:03.144559+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.144677+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-10T18:59:03.746 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:03 smithi165 bash[22591]: audit 2024-09-10T18:59:03.148847+0000 mon.a (mon.0) 2410 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:03.767 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:03.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:03.905 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:03.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:04.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.787995+0000 osd.1 (osd.1) 241 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:01.793662+0000 osd.1 (osd.1) 242 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.440678+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.444091+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.709161+0000 osd.6 (osd.6) 179 : cluster [DBG] 12.f scrub starts 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.712824+0000 osd.6 (osd.6) 180 : cluster [DBG] 12.f scrub ok 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: audit 2024-09-10T18:59:03.144528+0000 mon.a (mon.0) 2408 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: audit 2024-09-10T18:59:03.144559+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.144677+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[18629]: audit 2024-09-10T18:59:03.148847+0000 mon.a (mon.0) 2410 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.787995+0000 osd.1 (osd.1) 241 : cluster [DBG] 64.14 scrub starts 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:01.793662+0000 osd.1 (osd.1) 242 : cluster [DBG] 64.14 scrub ok 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.440678+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.13 scrub starts 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.444091+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.13 scrub ok 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.709161+0000 osd.6 (osd.6) 179 : cluster [DBG] 12.f scrub starts 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.712824+0000 osd.6 (osd.6) 180 : cluster [DBG] 12.f scrub ok 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: audit 2024-09-10T18:59:03.144528+0000 mon.a (mon.0) 2408 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: audit 2024-09-10T18:59:03.144559+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.023 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.144677+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e387: 8 total, 8 up, 8 in 2024-09-10T18:59:04.024 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:03 smithi083 bash[32381]: audit 2024-09-10T18:59:03.148847+0000 mon.a (mon.0) 2410 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.041 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:04.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:04.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:04.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:04.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:04.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:04.445 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:04.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:04.580 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.764132+0000 osd.1 (osd.1) 243 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:02.768796+0000 osd.1 (osd.1) 244 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.208566+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.215031+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.717690+0000 osd.1 (osd.1) 245 : cluster [DBG] 64.16 deep-scrub starts 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.724801+0000 osd.1 (osd.1) 246 : cluster [DBG] 64.16 deep-scrub ok 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.759174+0000 osd.6 (osd.6) 181 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:59:04.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.762729+0000 osd.6 (osd.6) 182 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.823187+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.826234+0000 osd.0 (osd.0) 215 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.921037+0000 osd.2 (osd.2) 161 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:03.924159+0000 osd.2 (osd.2) 162 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.057730+0000 mgr.y (mgr.24526) 397 : cluster [DBG] pgmap v565: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.090137+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.106776+0000 mon.a (mon.0) 2412 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.107011+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.111890+0000 mon.c (mon.2) 605 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.112396+0000 mon.a (mon.0) 2414 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.157608+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.159261+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.161373+0000 mon.a (mon.0) 2415 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.624 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[32381]: audit 2024-09-10T18:59:04.162801+0000 mon.a (mon.0) 2416 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:04.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:04.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.764132+0000 osd.1 (osd.1) 243 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:02.768796+0000 osd.1 (osd.1) 244 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.208566+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.215031+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.717690+0000 osd.1 (osd.1) 245 : cluster [DBG] 64.16 deep-scrub starts 2024-09-10T18:59:04.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.724801+0000 osd.1 (osd.1) 246 : cluster [DBG] 64.16 deep-scrub ok 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.759174+0000 osd.6 (osd.6) 181 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.762729+0000 osd.6 (osd.6) 182 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.823187+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.826234+0000 osd.0 (osd.0) 215 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.921037+0000 osd.2 (osd.2) 161 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:03.924159+0000 osd.2 (osd.2) 162 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.057730+0000 mgr.y (mgr.24526) 397 : cluster [DBG] pgmap v565: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.090137+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.106776+0000 mon.a (mon.0) 2412 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.107011+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.111890+0000 mon.c (mon.2) 605 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.112396+0000 mon.a (mon.0) 2414 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.157608+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.159261+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.161373+0000 mon.a (mon.0) 2415 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:04 smithi165 bash[22591]: audit 2024-09-10T18:59:04.162801+0000 mon.a (mon.0) 2416 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.852 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:04.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.764132+0000 osd.1 (osd.1) 243 : cluster [DBG] 64.1b scrub starts 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:02.768796+0000 osd.1 (osd.1) 244 : cluster [DBG] 64.1b scrub ok 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.208566+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.1c scrub starts 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.215031+0000 osd.7 (osd.7) 233 : cluster [DBG] 2.1c scrub ok 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.717690+0000 osd.1 (osd.1) 245 : cluster [DBG] 64.16 deep-scrub starts 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.724801+0000 osd.1 (osd.1) 246 : cluster [DBG] 64.16 deep-scrub ok 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.759174+0000 osd.6 (osd.6) 181 : cluster [DBG] 12.15 scrub starts 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.762729+0000 osd.6 (osd.6) 182 : cluster [DBG] 12.15 scrub ok 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.823187+0000 osd.0 (osd.0) 214 : cluster [DBG] 4.6 scrub starts 2024-09-10T18:59:04.970 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.826234+0000 osd.0 (osd.0) 215 : cluster [DBG] 4.6 scrub ok 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.921037+0000 osd.2 (osd.2) 161 : cluster [DBG] 64.9 scrub starts 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:03.924159+0000 osd.2 (osd.2) 162 : cluster [DBG] 64.9 scrub ok 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.057730+0000 mgr.y (mgr.24526) 397 : cluster [DBG] pgmap v565: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.090137+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.106776+0000 mon.a (mon.0) 2412 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]': finished 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.107011+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e388: 8 total, 8 up, 8 in 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.111890+0000 mon.c (mon.2) 605 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.112396+0000 mon.a (mon.0) 2414 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.157608+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.159261+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.161373+0000 mon.a (mon.0) 2415 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:04.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:04 smithi083 bash[18629]: audit 2024-09-10T18:59:04.162801+0000 mon.a (mon.0) 2416 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-47"}]: dispatch 2024-09-10T18:59:04.990 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:04.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:05.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:05.120 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:05.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:05.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:05.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:05.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:05.531 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:05.628 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.190563+0000 osd.7 (osd.7) 234 : cluster [DBG] 64.6 deep-scrub starts 2024-09-10T18:59:05.628 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.213290+0000 osd.7 (osd.7) 235 : cluster [DBG] 64.6 deep-scrub ok 2024-09-10T18:59:05.629 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.447571+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:59:05.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:05.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:05.804 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:05.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.921 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.451811+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.500351+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.505029+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.784703+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:04.788711+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: audit 2024-09-10T18:59:05.105245+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: cluster 2024-09-10T18:59:05.105472+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: audit 2024-09-10T18:59:05.107296+0000 mon.c (mon.2) 606 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[18629]: audit 2024-09-10T18:59:05.108506+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.190563+0000 osd.7 (osd.7) 234 : cluster [DBG] 64.6 deep-scrub starts 2024-09-10T18:59:05.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.213290+0000 osd.7 (osd.7) 235 : cluster [DBG] 64.6 deep-scrub ok 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.447571+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.451811+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.500351+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.505029+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.784703+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:04.788711+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: audit 2024-09-10T18:59:05.105245+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: cluster 2024-09-10T18:59:05.105472+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: audit 2024-09-10T18:59:05.107296+0000 mon.c (mon.2) 606 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:05.923 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:05 smithi083 bash[32381]: audit 2024-09-10T18:59:05.108506+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:05.941 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:05.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:06.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.190563+0000 osd.7 (osd.7) 234 : cluster [DBG] 64.6 deep-scrub starts 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.213290+0000 osd.7 (osd.7) 235 : cluster [DBG] 64.6 deep-scrub ok 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.447571+0000 osd.4 (osd.4) 259 : cluster [DBG] 5.1 scrub starts 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.451811+0000 osd.4 (osd.4) 260 : cluster [DBG] 5.1 scrub ok 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.500351+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.16 scrub starts 2024-09-10T18:59:06.056 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.505029+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.16 scrub ok 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.784703+0000 osd.0 (osd.0) 216 : cluster [DBG] 6.18 scrub starts 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:04.788711+0000 osd.0 (osd.0) 217 : cluster [DBG] 6.18 scrub ok 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: audit 2024-09-10T18:59:05.105245+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: cluster 2024-09-10T18:59:05.105472+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e389: 8 total, 8 up, 8 in 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: audit 2024-09-10T18:59:05.107296+0000 mon.c (mon.2) 606 : audit [INF] from='client.? v1:172.21.15.83:0/204315265' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:06.057 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:05 smithi165 bash[22591]: audit 2024-09-10T18:59:05.108506+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]: dispatch 2024-09-10T18:59:06.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:06.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:06.212 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:06.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:06.349 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:06.453 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.487 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:06.590 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.622 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1225130385 >> v1:172.21.15.83:6789/0 conn(0x7faa04107e30 legacy=0x7faa0410a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1225130385 shutdown_connections 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1225130385 >> 172.21.15.83:0/1225130385 conn(0x7faa04074a20 msgr2=0x7faa040f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1225130385 shutdown_connections 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1225130385 wait complete. 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 librados: starting msgr at 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 librados: starting objecter 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 Processor -- start 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- start start 2024-09-10T18:59:06.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 librados: setting wanted keys 2024-09-10T18:59:06.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 librados: calling monclient init 2024-09-10T18:59:06.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa0411e430 con 0x7faa04107e30 2024-09-10T18:59:06.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa0411f5f0 con 0x7faa041042d0 2024-09-10T18:59:06.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faa041207f0 con 0x7faa041014f0 2024-09-10T18:59:06.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0a32f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7faa041014f0 0x7faa040fac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:54820/0 (socket says 172.21.15.83:54820) 2024-09-10T18:59:06.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0a32f640 1 -- 172.21.15.83:0/1621818676 learned_addr learned my addr 172.21.15.83:0/1621818676 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter ms_handle_connect 0x7faa04107e30 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter ms_handle_connect 0x7faa041014f0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102474316 0 0) 0x7faa0411e430 con 0x7faa04107e30 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003720 con 0x7faa04107e30 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter ms_handle_connect 0x7faa041042d0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967319218 0 0) 0x7faa041207f0 con 0x7faa041014f0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa0411e430 con 0x7faa041014f0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830366313 0 0) 0x7faa0411f5f0 con 0x7faa041042d0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa041207f0 con 0x7faa041042d0 2024-09-10T18:59:06.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094194718 0 0) 0x7fa9f0003720 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa0411f5f0 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2503711814 0 0) 0x7faa0411e430 con 0x7faa041014f0 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9f0003720 con 0x7faa041014f0 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9fc0023d0 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7faa00002890 con 0x7faa041014f0 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250067635 0 0) 0x7faa0411f5f0 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 >> v1:172.21.15.83:6790/0 conn(0x7faa041014f0 legacy=0x7faa040fac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 >> v1:172.21.15.165:6789/0 conn(0x7faa041042d0 legacy=0x7faa040fb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa041219f0 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa041209e0 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 10 client.25251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa04120e30 con 0x7faa04107e30 2024-09-10T18:59:06.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7faa0c593640 1 librados: init done 2024-09-10T18:59:06.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa9fc003120 con 0x7faa04107e30 2024-09-10T18:59:06.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.633+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9fc0037d0 con 0x7faa04107e30 2024-09-10T18:59:06.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8005010 con 0x7faa04107e30 2024-09-10T18:59:06.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa9fc001b70 con 0x7faa04107e30 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(390..390 src has 69..390) v4 ==== 7975+0+0 (unknown 1447480461 0 0) 0x7fa9fc088170 con 0x7faa04107e30 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 10 client.25251.objecter ms_dispatch 0x7faa04000b90 osd_map(390..390 src has 69..390) v4 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 3 client.25251.objecter handle_osd_map got epochs [390,390] > 0 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 3 client.25251.objecter handle_osd_map decoding full epoch 390 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 20 client.25251.objecter dump_active .. 0 homeless 2024-09-10T18:59:06.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.637+0000 7fa9faffd640 10 client.25251.objecter ms_handle_connect 0x7fa9f00717f0 2024-09-10T18:59:06.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.641+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa9fc054750 con 0x7faa04107e30 2024-09-10T18:59:06.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: cluster 2024-09-10T18:59:05.510076+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:59:06.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: cluster 2024-09-10T18:59:05.516052+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:59:06.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: cluster 2024-09-10T18:59:06.059827+0000 mgr.y (mgr.24526) 398 : cluster [DBG] pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.107781+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: cluster 2024-09-10T18:59:06.108056+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.109519+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.114053+0000 mon.a (mon.0) 2422 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.140307+0000 mon.c (mon.2) 607 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.140741+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.142378+0000 mon.c (mon.2) 608 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.142768+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.143337+0000 mon.c (mon.2) 609 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[32381]: audit 2024-09-10T18:59:06.143684+0000 mon.a (mon.0) 2425 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:06.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: cluster 2024-09-10T18:59:05.510076+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: cluster 2024-09-10T18:59:05.516052+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: cluster 2024-09-10T18:59:06.059827+0000 mgr.y (mgr.24526) 398 : cluster [DBG] pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.107781+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: cluster 2024-09-10T18:59:06.108056+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.109519+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.114053+0000 mon.a (mon.0) 2422 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.140307+0000 mon.c (mon.2) 607 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.140741+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.142378+0000 mon.c (mon.2) 608 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.142768+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:06.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.143337+0000 mon.c (mon.2) 609 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:06.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:06 smithi083 bash[18629]: audit 2024-09-10T18:59:06.143684+0000 mon.a (mon.0) 2425 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:06.758 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:06.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:06.825+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa9e8005840 con 0x7faa04107e30 2024-09-10T18:59:06.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:06.893 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:06.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:06.986 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:06.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:07.030 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: cluster 2024-09-10T18:59:05.510076+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.1d scrub starts 2024-09-10T18:59:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: cluster 2024-09-10T18:59:05.516052+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.1d scrub ok 2024-09-10T18:59:07.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: cluster 2024-09-10T18:59:06.059827+0000 mgr.y (mgr.24526) 398 : cluster [DBG] pgmap v568: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.107781+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi083-77648-63"}]': finished 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: cluster 2024-09-10T18:59:06.108056+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e390: 8 total, 8 up, 8 in 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.109519+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.114053+0000 mon.a (mon.0) 2422 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.140307+0000 mon.c (mon.2) 607 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.140741+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.142378+0000 mon.c (mon.2) 608 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.142768+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.143337+0000 mon.c (mon.2) 609 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:07.115 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:06 smithi165 bash[22591]: audit 2024-09-10T18:59:06.143684+0000 mon.a (mon.0) 2425 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.121+0000 7fa9faffd640 1 -- 172.21.15.83:0/1621818676 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v391) v1 ==== 87+0+0 (unknown 4156345429 0 0) 0x7fa9fc0540f0 con 0x7faa04107e30 2024-09-10T18:59:07.126 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T18:59:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.125+0000 7faa0c593640 10 librados: watch_flush enter 2024-09-10T18:59:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.125+0000 7faa0c593640 10 librados: watch_flush exit 2024-09-10T18:59:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.125+0000 7faa0c593640 20 client.25251.objecter shutdown clearing up homeless session... 2024-09-10T18:59:07.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:07.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.125+0000 7faa0c593640 10 client.25251.objecter successfully canceled tick 2024-09-10T18:59:07.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa9f00717f0 legacy=0x7fa9f0073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 >> v1:172.21.15.83:6789/0 conn(0x7faa04107e30 legacy=0x7faa0411cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 shutdown_connections 2024-09-10T18:59:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 >> 172.21.15.83:0/1621818676 conn(0x7faa04074a20 msgr2=0x7faa0410b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 shutdown_connections 2024-09-10T18:59:07.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.133+0000 7faa0c593640 1 -- 172.21.15.83:0/1621818676 wait complete. 2024-09-10T18:59:07.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7faa0c593640 1 librados: shutdown 2024-09-10T18:59:07.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- 172.21.15.83:0/314948697 >> v1:172.21.15.83:6789/0 conn(0x7f37f0117d80 legacy=0x7f37f011a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- 172.21.15.83:0/314948697 shutdown_connections 2024-09-10T18:59:07.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- 172.21.15.83:0/314948697 >> 172.21.15.83:0/314948697 conn(0x7f37f006b990 msgr2=0x7f37f006bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:07.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- 172.21.15.83:0/314948697 shutdown_connections 2024-09-10T18:59:07.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- 172.21.15.83:0/314948697 wait complete. 2024-09-10T18:59:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 librados: starting msgr at 2024-09-10T18:59:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 librados: starting objecter 2024-09-10T18:59:07.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 Processor -- start 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 -- start start 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 librados: setting wanted keys 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.137+0000 7f37f7752640 1 librados: calling monclient init 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f7752640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37f012b1b0 con 0x7f37f006e740 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f7752640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37f012c350 con 0x7f37f006dc60 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f7752640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37f012d4f0 con 0x7f37f0117d80 2024-09-10T18:59:07.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f4ced640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f37f006e740 0x7f37f0126360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55154/0 (socket says 172.21.15.83:55154) 2024-09-10T18:59:07.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f4ced640 1 -- 172.21.15.83:0/2794473793 learned_addr learned my addr 172.21.15.83:0/2794473793 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:07.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f006e740 2024-09-10T18:59:07.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824586748 0 0) 0x7f37f012b1b0 con 0x7f37f006e740 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc005530 con 0x7f37f006e740 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f0117d80 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter ms_handle_connect 0x7f37f006dc60 2024-09-10T18:59:07.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036307824 0 0) 0x7f37dc005530 con 0x7f37f006e740 2024-09-10T18:59:07.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f012b1b0 con 0x7f37f006e740 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228295419 0 0) 0x7f37f012c350 con 0x7f37f006dc60 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc005530 con 0x7f37f006dc60 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f37e8002c80 con 0x7f37f006e740 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3345369338 0 0) 0x7f37f012b1b0 con 0x7f37f006e740 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6790/0 conn(0x7f37f0117d80 legacy=0x7f37f0129a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f15b099d640 1 -- 172.21.15.83:0/2708872189 >> v1:172.21.15.165:6789/0 conn(0x7f15ac117d80 legacy=0x7f15ac11a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f15aa7fc640 1 -- 172.21.15.83:0/2708872189 <== mon.1 v1:172.21.15.165:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f15a4003060 con 0x7f15ac117d80 2024-09-10T18:59:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f15b099d640 1 -- 172.21.15.83:0/2708872189 shutdown_connections 2024-09-10T18:59:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f15b099d640 1 -- 172.21.15.83:0/2708872189 >> 172.21.15.83:0/2708872189 conn(0x7f15ac068750 msgr2=0x7f15ac068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.165:6789/0 conn(0x7f37f006dc60 legacy=0x7f37f0108aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- 172.21.15.83:0/2708872189 shutdown_connections 2024-09-10T18:59:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f012e690 con 0x7f37f006e740 2024-09-10T18:59:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- 172.21.15.83:0/2708872189 wait complete. 2024-09-10T18:59:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f012d6e0 con 0x7f37f006e740 2024-09-10T18:59:07.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 librados: starting msgr at 2024-09-10T18:59:07.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 librados: starting objecter 2024-09-10T18:59:07.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 Processor -- start 2024-09-10T18:59:07.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f37e8003000 con 0x7f37f006e740 2024-09-10T18:59:07.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.141+0000 7f37f7752640 10 client.25260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- start start 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 librados: setting wanted keys 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 librados: calling monclient init 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15ac12aae0 con 0x7f15ac06df30 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15ac12bca0 con 0x7f15ac10b850 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15b099d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15ac12cea0 con 0x7f15ac117d80 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15ab7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f15ac06df30 0x7f15ac107930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55162/0 (socket says 172.21.15.83:55162) 2024-09-10T18:59:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f15ab7fe640 1 -- 172.21.15.83:0/3528958378 learned_addr learned my addr 172.21.15.83:0/3528958378 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter ms_handle_connect 0x7f15ac06df30 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter ms_handle_connect 0x7f15ac10b850 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894285545 0 0) 0x7f15ac12aae0 con 0x7f15ac06df30 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1594003200 con 0x7f15ac06df30 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524615581 0 0) 0x7f15ac12bca0 con 0x7f15ac10b850 2024-09-10T18:59:07.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ac12aae0 con 0x7f15ac10b850 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f37e8005e40 con 0x7f37f006e740 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f012db30 con 0x7f37f006e740 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 1 librados: init done 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f37e80060d0 con 0x7f37f006e740 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f37f006f350 con 0x7f37f006e740 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(391..391 src has 69..391) v4 ==== 8245+0+0 (unknown 971246096 0 0) 0x7f37e808aa20 con 0x7f37f006e740 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 osd_map(391..391 src has 69..391) v4 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 3 client.25260.objecter handle_osd_map got epochs [391,391] > 0 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 3 client.25260.objecter handle_osd_map decoding full epoch 391 2024-09-10T18:59:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 20 client.25260.objecter dump_active .. 0 homeless 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777765095 0 0) 0x7f1594003200 con 0x7f15ac06df30 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15ac12bca0 con 0x7f15ac06df30 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter ms_handle_connect 0x7f15ac117d80 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390745998 0 0) 0x7f15ac12aae0 con 0x7f15ac10b850 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v391) v1 ==== 54+0+16 (unknown 3774239582 0 2420000192) 0x7f37e808b020 con 0x7f37f006e740 2024-09-10T18:59:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1594003200 con 0x7f15ac10b850 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f159c0029d0 con 0x7f15ac06df30 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f15a0001720 con 0x7f15ac10b850 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37dc071500 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 10 client.25260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 1 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 1 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1661524873 0 0) 0x7f15ac12bca0 con 0x7f15ac06df30 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 >> v1:172.21.15.83:6790/0 conn(0x7f15ac117d80 legacy=0x7f15ac1291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 0 1 2024-09-10T18:59:07.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 4 2024-09-10T18:59:07.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 10 client.25260.objecter _send_command 1 2024-09-10T18:59:07.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.145+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f00298c0 con 0x7f37f012c9f0 2024-09-10T18:59:07.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 >> v1:172.21.15.165:6789/0 conn(0x7f15ac10b850 legacy=0x7f15ac108040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac12e0a0 con 0x7f15ac06df30 2024-09-10T18:59:07.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f012c9f0 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ac12be90 con 0x7f15ac06df30 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f15b099d640 10 client.25263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ac12c320 con 0x7f15ac06df30 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f15b099d640 1 librados: init done 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f159c001870 con 0x7f15ac06df30 2024-09-10T18:59:07.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f00298c0 con 0x7f37f012c9f0 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 1 = system:0 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 4 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 0 1 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f159c005b20 con 0x7f15ac06df30 2024-09-10T18:59:07.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598005010 con 0x7f15ac06df30 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.149+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f159c005cc0 con 0x7f15ac06df30 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(391..391 src has 69..391) v4 ==== 8245+0+0 (unknown 971246096 0 0) 0x7f159c08b4d0 con 0x7f15ac06df30 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 10 client.25263.objecter ms_dispatch 0x7f15ac000b90 osd_map(391..391 src has 69..391) v4 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 3 client.25263.objecter handle_osd_map got epochs [391,391] > 0 2024-09-10T18:59:07.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 3 client.25263.objecter handle_osd_map decoding full epoch 391 2024-09-10T18:59:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:07.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 20 client.25263.objecter dump_active .. 0 homeless 2024-09-10T18:59:07.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.157+0000 7f1593fff640 10 client.25263.objecter ms_handle_connect 0x7f15940718e0 2024-09-10T18:59:07.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.161+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f159c0575a0 con 0x7f15ac06df30 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 10 client.25260.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 2 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 2 2024-09-10T18:59:07.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f011d880 osd=0 3 2024-09-10T18:59:07.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 0 2 2024-09-10T18:59:07.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 4 2024-09-10T18:59:07.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 10 client.25260.objecter _send_command 2 2024-09-10T18:59:07.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f002abb0 con 0x7f37f012c9f0 2024-09-10T18:59:07.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f002abb0 con 0x7f37f012c9f0 2024-09-10T18:59:07.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 4 2024-09-10T18:59:07.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 0 2 2024-09-10T18:59:07.174 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 10 client.25260.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 1 3 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 4 2024-09-10T18:59:07.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 10 client.25260.objecter _send_command 3 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.169+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f0119890 con 0x7f37f012e1d0 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f012e1d0 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f0119890 con 0x7f37f012e1d0 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 3 = system:0 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 4 2024-09-10T18:59:07.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.173+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 1 3 2024-09-10T18:59:07.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 10 client.25260.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 4 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 4 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f012e060 osd=1 3 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 1 4 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 4 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 10 client.25260.objecter _send_command 4 2024-09-10T18:59:07.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f005ea60 con 0x7f37f012e1d0 2024-09-10T18:59:07.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f005ea60 con 0x7f37f012e1d0 2024-09-10T18:59:07.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 4 2024-09-10T18:59:07.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 1 4 2024-09-10T18:59:07.202 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:59:07.202 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:59:07.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 10 client.25260.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 5 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 5 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 2 5 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 4 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 10 client.25260.objecter _send_command 5 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.197+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f002a700 con 0x7f37f005df00 2024-09-10T18:59:07.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f005df00 2024-09-10T18:59:07.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f002a700 con 0x7f37f005df00 2024-09-10T18:59:07.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:59:07.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 5 = system:0 2024-09-10T18:59:07.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 4 2024-09-10T18:59:07.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.201+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 2 5 2024-09-10T18:59:07.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 10 client.25260.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 6 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 6 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f005dd90 osd=2 3 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 2 6 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 4 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 10 client.25260.objecter _send_command 6 2024-09-10T18:59:07.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.221+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f01370f0 con 0x7f37f005df00 2024-09-10T18:59:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f01370f0 con 0x7f37f005df00 2024-09-10T18:59:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 4 2024-09-10T18:59:07.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 2 6 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 10 client.25260.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 7 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 7 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 3 7 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 4 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 10 client.25260.objecter _send_command 7 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f0133510 con 0x7f37f0029de0 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f0029de0 2024-09-10T18:59:07.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f0133510 con 0x7f37f0029de0 2024-09-10T18:59:07.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:59:07.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 7 = system:0 2024-09-10T18:59:07.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 4 2024-09-10T18:59:07.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.225+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 3 7 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 10 client.25260.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 8 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 8 2024-09-10T18:59:07.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0029c70 osd=3 3 2024-09-10T18:59:07.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 3 8 2024-09-10T18:59:07.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 4 2024-09-10T18:59:07.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 10 client.25260.objecter _send_command 8 2024-09-10T18:59:07.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f0133510 con 0x7f37f0029de0 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f0133510 con 0x7f37f0029de0 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 4 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 3 8 2024-09-10T18:59:07.249 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:59:07.250 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:59:07.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 10 client.25260.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 9 2024-09-10T18:59:07.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 9 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 4 9 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 4 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 10 client.25260.objecter _send_command 9 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.245+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f013fbc0 con 0x7f37f013c020 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f013c020 2024-09-10T18:59:07.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f013fbc0 con 0x7f37f013c020 2024-09-10T18:59:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:59:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 9 = system:0 2024-09-10T18:59:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 4 2024-09-10T18:59:07.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.249+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 4 9 2024-09-10T18:59:07.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:07.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 10 client.25260.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 10 2024-09-10T18:59:07.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 10 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0133510 osd=4 3 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 4 10 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 4 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 10 client.25260.objecter _send_command 10 2024-09-10T18:59:07.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.269+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f0141460 con 0x7f37f013c020 2024-09-10T18:59:07.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f0141460 con 0x7f37f013c020 2024-09-10T18:59:07.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 4 2024-09-10T18:59:07.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 4 10 2024-09-10T18:59:07.275 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 10 client.25260.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 11 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 11 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 5 11 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 4 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 10 client.25260.objecter _send_command 11 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f0141fe0 con 0x7f37f0141b10 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f0141b10 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f0141fe0 con 0x7f37f0141b10 2024-09-10T18:59:07.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:59:07.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 11 = system:0 2024-09-10T18:59:07.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 4 2024-09-10T18:59:07.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.273+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 5 11 2024-09-10T18:59:07.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 10 client.25260.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 12 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 12 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0141460 osd=5 3 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 5 12 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 4 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 10 client.25260.objecter _send_command 12 2024-09-10T18:59:07.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.289+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f0141fe0 con 0x7f37f0141b10 2024-09-10T18:59:07.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f0141fe0 con 0x7f37f0141b10 2024-09-10T18:59:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 4 2024-09-10T18:59:07.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 5 12 2024-09-10T18:59:07.297 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:59:07.297 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 10 client.25260.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 13 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 13 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 6 13 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 4 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 10 client.25260.objecter _send_command 13 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f01423b0 con 0x7f37f0140980 2024-09-10T18:59:07.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.293+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f0140980 2024-09-10T18:59:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.297+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f01423b0 con 0x7f37f0140980 2024-09-10T18:59:07.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.297+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:59:07.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.297+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 13 = system:0 2024-09-10T18:59:07.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.297+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 4 2024-09-10T18:59:07.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.297+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 6 13 2024-09-10T18:59:07.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 10 client.25260.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 14 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 14 2024-09-10T18:59:07.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0141fe0 osd=6 3 2024-09-10T18:59:07.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 6 14 2024-09-10T18:59:07.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 4 2024-09-10T18:59:07.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 10 client.25260.objecter _send_command 14 2024-09-10T18:59:07.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.309+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f0140e00 con 0x7f37f0140980 2024-09-10T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f0140e00 con 0x7f37f0140980 2024-09-10T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 4 2024-09-10T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 6 14 2024-09-10T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:59:07.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 10 client.25260.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 15 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 15 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 7 15 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 4 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 10 client.25260.objecter _send_command 15 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f37f014c9c0 con 0x7f37f0140f70 2024-09-10T18:59:07.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.313+0000 7f37e5ffb640 10 client.25260.objecter ms_handle_connect 0x7f37f0140f70 2024-09-10T18:59:07.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.317+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f37f014c9c0 con 0x7f37f0140f70 2024-09-10T18:59:07.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.317+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:59:07.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.317+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 15 = system:0 2024-09-10T18:59:07.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.317+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 4 2024-09-10T18:59:07.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.317+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 7 15 2024-09-10T18:59:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 10 client.25260.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign -1 16 2024-09-10T18:59:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 20 client.25260.objecter _get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 15 client.25260.objecter _session_command_op_remove -1 16 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 20 client.25260.objecter get_session s=0x7f37f0140e00 osd=7 3 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 15 client.25260.objecter _session_command_op_assign 7 16 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 4 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 10 client.25260.objecter _send_command 16 2024-09-10T18:59:07.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.329+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f37f014c9c0 con 0x7f37f0140f70 2024-09-10T18:59:07.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37e5ffb640 1 -- 172.21.15.83:0/2794473793 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f37f014c9c0 con 0x7f37f0140f70 2024-09-10T18:59:07.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37e5ffb640 10 client.25260.objecter ms_dispatch 0x7f37f0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:07.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37e5ffb640 10 client.25260.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:07.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37e5ffb640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 4 2024-09-10T18:59:07.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37e5ffb640 15 client.25260.objecter _session_command_op_remove 7 16 2024-09-10T18:59:07.337 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 librados: watch_flush enter 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 librados: watch_flush exit 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.0 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6801/2535527483 conn(0x7f37f012c9f0 legacy=0x7f37f012cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f011d880 osd=0 1 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.1 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6805/2309624637 conn(0x7f37f012e1d0 legacy=0x7f37f0118170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f012e060 osd=1 1 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.2 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6809/386094220 conn(0x7f37f005df00 legacy=0x7f37f01359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f005dd90 osd=2 1 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.3 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6813/1279688021 conn(0x7f37f0029de0 legacy=0x7f37f013a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0029c70 osd=3 1 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.4 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.165:6800/3225515265 conn(0x7f37f013c020 legacy=0x7f37f013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0133510 osd=4 1 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.5 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.165:6804/571173777 conn(0x7f37f0141b10 legacy=0x7f37f013fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141460 osd=5 1 2024-09-10T18:59:07.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.6 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.165:6808/1317922936 conn(0x7f37f0140980 legacy=0x7f37f0147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0141fe0 osd=6 1 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter close_session for osd.7 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.165:6812/1610151650 conn(0x7f37f0140f70 legacy=0x7f37f014b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter put_session s=0x7f37f0140e00 osd=7 1 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 20 client.25260.objecter shutdown clearing up homeless session... 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 10 client.25260.objecter successfully canceled tick 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f5cef640 1 -- 172.21.15.83:0/2794473793 reap_dead start 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f5cef640 1 -- 172.21.15.83:0/2794473793 reap_dead start 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f5cef640 1 -- 172.21.15.83:0/2794473793 reap_dead start 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f5cef640 1 -- 172.21.15.83:0/2794473793 reap_dead start 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f5cef640 1 -- 172.21.15.83:0/2794473793 reap_dead start 2024-09-10T18:59:07.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6800/3071793735 conn(0x7f37dc071500 legacy=0x7f37dc0739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> v1:172.21.15.83:6789/0 conn(0x7f37f006e740 legacy=0x7f37f0126360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 shutdown_connections 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 >> 172.21.15.83:0/2794473793 conn(0x7f37f006b990 msgr2=0x7f37f006c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 shutdown_connections 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 -- 172.21.15.83:0/2794473793 wait complete. 2024-09-10T18:59:07.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.333+0000 7f37f7752640 1 librados: shutdown 2024-09-10T18:59:07.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.341+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1598000b90 con 0x7f15940718e0 2024-09-10T18:59:07.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.369+0000 7f1593fff640 1 -- 172.21.15.83:0/3528958378 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+620316 (unknown 2965378022 0 429925038) 0x7f1598000b90 con 0x7f15940718e0 2024-09-10T18:59:07.381 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:59:07.389 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 10 librados: watch_flush enter 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 10 librados: watch_flush exit 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 20 client.25263.objecter shutdown clearing up homeless session... 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 10 client.25263.objecter successfully canceled tick 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 >> v1:172.21.15.83:6800/3071793735 conn(0x7f15940718e0 legacy=0x7f1594073d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 >> v1:172.21.15.83:6789/0 conn(0x7f15ac06df30 legacy=0x7f15ac107930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 shutdown_connections 2024-09-10T18:59:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 >> 172.21.15.83:0/3528958378 conn(0x7f15ac068750 msgr2=0x7f15ac06b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 shutdown_connections 2024-09-10T18:59:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 -- 172.21.15.83:0/3528958378 wait complete. 2024-09-10T18:59:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:07.377+0000 7f15b099d640 1 librados: shutdown 2024-09-10T18:59:07.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:07.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:07.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:07.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:07.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:07.708 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:07.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:07.849 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:07.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:06.828800+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: cluster 2024-09-10T18:59:06.976352+0000 osd.2 (osd.2) 163 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: cluster 2024-09-10T18:59:06.979587+0000 osd.2 (osd.2) 164 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: cluster 2024-09-10T18:59:07.101828+0000 mon.a (mon.0) 2427 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.111840+0000 mon.a (mon.0) 2428 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.112016+0000 mon.a (mon.0) 2429 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.112096+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: cluster 2024-09-10T18:59:07.112140+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.126131+0000 mon.c (mon.2) 610 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.128598+0000 mon.a (mon.0) 2432 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.135138+0000 mon.b (mon.1) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:07.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.139994+0000 mon.a (mon.0) 2433 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:07.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[22591]: audit 2024-09-10T18:59:07.149045+0000 mon.a (mon.0) 2434 : audit [DBG] from='client.? v1:172.21.15.83:0/2794473793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:07.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:07.985 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:07.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:07.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:06.828800+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: cluster 2024-09-10T18:59:06.976352+0000 osd.2 (osd.2) 163 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: cluster 2024-09-10T18:59:06.979587+0000 osd.2 (osd.2) 164 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: cluster 2024-09-10T18:59:07.101828+0000 mon.a (mon.0) 2427 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.111840+0000 mon.a (mon.0) 2428 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.112016+0000 mon.a (mon.0) 2429 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.112096+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: cluster 2024-09-10T18:59:07.112140+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.126131+0000 mon.c (mon.2) 610 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.128598+0000 mon.a (mon.0) 2432 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.135138+0000 mon.b (mon.1) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.139994+0000 mon.a (mon.0) 2433 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[18629]: audit 2024-09-10T18:59:07.149045+0000 mon.a (mon.0) 2434 : audit [DBG] from='client.? v1:172.21.15.83:0/2794473793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:06.828800+0000 mon.a (mon.0) 2426 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: cluster 2024-09-10T18:59:06.976352+0000 osd.2 (osd.2) 163 : cluster [DBG] 12.9 scrub starts 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: cluster 2024-09-10T18:59:06.979587+0000 osd.2 (osd.2) 164 : cluster [DBG] 12.9 scrub ok 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: cluster 2024-09-10T18:59:07.101828+0000 mon.a (mon.0) 2427 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.111840+0000 mon.a (mon.0) 2428 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.112016+0000 mon.a (mon.0) 2429 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi083-77648-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.112096+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? v1:172.21.15.83:0/1621818676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: cluster 2024-09-10T18:59:07.112140+0000 mon.a (mon.0) 2431 : cluster [DBG] osdmap e391: 8 total, 8 up, 8 in 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.126131+0000 mon.c (mon.2) 610 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.128598+0000 mon.a (mon.0) 2432 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.135138+0000 mon.b (mon.1) 308 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.139994+0000 mon.a (mon.0) 2433 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:07 smithi083 bash[32381]: audit 2024-09-10T18:59:07.149045+0000 mon.a (mon.0) 2434 : audit [DBG] from='client.? v1:172.21.15.83:0/2794473793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:08.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:08.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:08.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:08.251 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:08.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:08.363 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:08.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:08.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:08.558 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:08.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:08.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:08.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:08.940 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:08.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:09.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:09.112 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:09.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:09.251 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:09.300 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: audit 2024-09-10T18:59:07.346539+0000 mgr.y (mgr.24526) 399 : audit [DBG] from='client.25263 v1:172.21.15.83:0/3528958378' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:09.300 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: audit 2024-09-10T18:59:07.912408+0000 mgr.y (mgr.24526) 400 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: cluster 2024-09-10T18:59:08.060597+0000 mgr.y (mgr.24526) 401 : cluster [DBG] pgmap v571: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: audit 2024-09-10T18:59:08.566112+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: audit 2024-09-10T18:59:08.567760+0000 mon.a (mon.0) 2435 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: cluster 2024-09-10T18:59:08.567934+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[18629]: audit 2024-09-10T18:59:08.569809+0000 mon.a (mon.0) 2437 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: audit 2024-09-10T18:59:07.346539+0000 mgr.y (mgr.24526) 399 : audit [DBG] from='client.25263 v1:172.21.15.83:0/3528958378' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: audit 2024-09-10T18:59:07.912408+0000 mgr.y (mgr.24526) 400 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:09.301 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: cluster 2024-09-10T18:59:08.060597+0000 mgr.y (mgr.24526) 401 : cluster [DBG] pgmap v571: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:09.302 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: audit 2024-09-10T18:59:08.566112+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.302 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: audit 2024-09-10T18:59:08.567760+0000 mon.a (mon.0) 2435 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:09.302 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: cluster 2024-09-10T18:59:08.567934+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-10T18:59:09.302 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:09 smithi083 bash[32381]: audit 2024-09-10T18:59:08.569809+0000 mon.a (mon.0) 2437 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: audit 2024-09-10T18:59:07.346539+0000 mgr.y (mgr.24526) 399 : audit [DBG] from='client.25263 v1:172.21.15.83:0/3528958378' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: audit 2024-09-10T18:59:07.912408+0000 mgr.y (mgr.24526) 400 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: cluster 2024-09-10T18:59:08.060597+0000 mgr.y (mgr.24526) 401 : cluster [DBG] pgmap v571: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: audit 2024-09-10T18:59:08.566112+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: audit 2024-09-10T18:59:08.567760+0000 mon.a (mon.0) 2435 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: cluster 2024-09-10T18:59:08.567934+0000 mon.a (mon.0) 2436 : cluster [DBG] osdmap e392: 8 total, 8 up, 8 in 2024-09-10T18:59:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:09 smithi165 bash[22591]: audit 2024-09-10T18:59:08.569809+0000 mon.a (mon.0) 2437 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:09.387 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:09.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:09.526 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:09.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:09.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:09.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:09.794 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:09.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:09.929 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:09.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:10.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:10.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:10.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:10.191 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[18629]: audit 2024-09-10T18:59:09.130163+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[18629]: audit 2024-09-10T18:59:09.130835+0000 mon.a (mon.0) 2438 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[18629]: audit 2024-09-10T18:59:09.130909+0000 mon.a (mon.0) 2439 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[18629]: cluster 2024-09-10T18:59:09.131017+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[18629]: audit 2024-09-10T18:59:09.133878+0000 mon.a (mon.0) 2441 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[32381]: audit 2024-09-10T18:59:09.130163+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[32381]: audit 2024-09-10T18:59:09.130835+0000 mon.a (mon.0) 2438 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:10.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[32381]: audit 2024-09-10T18:59:09.130909+0000 mon.a (mon.0) 2439 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:10.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[32381]: cluster 2024-09-10T18:59:09.131017+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-10T18:59:10.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:10 smithi083 bash[32381]: audit 2024-09-10T18:59:09.133878+0000 mon.a (mon.0) 2441 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.293 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:10.326 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:10.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:10.468 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":570,"stamp":"2024-09-10T18:59:07.111453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791877,"num_objects":341,"num_object_clones":2,"num_object_copies":1023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":1544,"num_read_kb":1304,"num_write":1121,"num_write_kb":14340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2505,"ondisk_log_size":2505,"up":588,"acting":588,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":687,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2477800,"kb_used_data":73912,"kb_used_omap":13,"kb_used_meta":2403698,"kb_avail":747483416,"statfs":{"total":767960285184,"available":765423017984,"internally_reserved":0,"allocated":75685888,"data_stored":53852307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2461387456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.968780"},"pg_stats":[{"pgid":"205.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"205.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T18:59:06.097952+0000","last_change":"2024-09-10T18:59:06.097952+0000","last_active":"2024-09-10T18:59:06.097952+0000","last_peered":"2024-09-10T18:59:06.097952+0000","last_clean":"2024-09-10T18:59:06.097952+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T18:59:06.097952+0000","last_undegraded":"2024-09-10T18:59:06.097952+0000","last_fullsized":"2024-09-10T18:59:06.097952+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:06.097952+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.097952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":347,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559961+0000","last_change":"2024-09-10T18:58:52.881919+0000","last_active":"2024-09-10T18:59:03.559961+0000","last_peered":"2024-09-10T18:59:03.559961+0000","last_clean":"2024-09-10T18:59:03.559961+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:59:03.559961+0000","last_undegraded":"2024-09-10T18:59:03.559961+0000","last_fullsized":"2024-09-10T18:59:03.559961+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.087038+0000","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"377'67","reported_seq":377,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559290+0000","last_change":"2024-09-10T18:58:39.857881+0000","last_active":"2024-09-10T18:59:03.559290+0000","last_peered":"2024-09-10T18:59:03.559290+0000","last_clean":"2024-09-10T18:59:03.559290+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:59:03.559290+0000","last_undegraded":"2024-09-10T18:59:03.559290+0000","last_fullsized":"2024-09-10T18:59:03.559290+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.880941+0000","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.016940303,"stat_sum":{"num_bytes":64,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"377'23","reported_seq":341,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213639+0000","last_change":"2024-09-10T18:57:54.056252+0000","last_active":"2024-09-10T18:59:03.213639+0000","last_peered":"2024-09-10T18:59:03.213639+0000","last_clean":"2024-09-10T18:59:03.213639+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:59:03.213639+0000","last_undegraded":"2024-09-10T18:59:03.213639+0000","last_fullsized":"2024-09-10T18:59:03.213639+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":364,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121932+0000","last_change":"2024-09-10T18:58:54.847481+0000","last_active":"2024-09-10T18:59:04.121932+0000","last_peered":"2024-09-10T18:59:04.121932+0000","last_clean":"2024-09-10T18:59:04.121932+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:59:04.121932+0000","last_undegraded":"2024-09-10T18:59:04.121932+0000","last_fullsized":"2024-09-10T18:59:04.121932+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:58:54.847269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.847269+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.700907+0000","scrub_duration":0.0091152980000000008,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":379,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121774+0000","last_change":"2024-09-10T18:59:02.768985+0000","last_active":"2024-09-10T18:59:04.121774+0000","last_peered":"2024-09-10T18:59:04.121774+0000","last_clean":"2024-09-10T18:59:04.121774+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:59:04.121774+0000","last_undegraded":"2024-09-10T18:59:04.121774+0000","last_fullsized":"2024-09-10T18:59:04.121774+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.764846+0000","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":344,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.212666+0000","last_change":"2024-09-10T18:57:54.056894+0000","last_active":"2024-09-10T18:59:03.212666+0000","last_peered":"2024-09-10T18:59:03.212666+0000","last_clean":"2024-09-10T18:59:03.212666+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:59:03.212666+0000","last_undegraded":"2024-09-10T18:59:03.212666+0000","last_fullsized":"2024-09-10T18:59:03.212666+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:57:35.352719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.352719+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074091110000000003,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"377'23","reported_seq":341,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.108302+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T18:59:04.108302+0000","last_peered":"2024-09-10T18:59:04.108302+0000","last_clean":"2024-09-10T18:59:04.108302+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:59:04.108302+0000","last_undegraded":"2024-09-10T18:59:04.108302+0000","last_fullsized":"2024-09-10T18:59:04.108302+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"254'104","reported_seq":400,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.131292+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T18:59:04.131292+0000","last_peered":"2024-09-10T18:59:04.131292+0000","last_clean":"2024-09-10T18:59:04.131292+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:59:04.131292+0000","last_undegraded":"2024-09-10T18:59:04.131292+0000","last_fullsized":"2024-09-10T18:59:04.131292+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"377'19","reported_seq":331,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153164+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T18:59:04.153164+0000","last_peered":"2024-09-10T18:59:04.153164+0000","last_clean":"2024-09-10T18:59:04.153164+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:59:04.153164+0000","last_undegraded":"2024-09-10T18:59:04.153164+0000","last_fullsized":"2024-09-10T18:59:04.153164+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:17.416599+0000","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"387'24","reported_seq":348,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122318+0000","last_change":"2024-09-10T18:59:03.724989+0000","last_active":"2024-09-10T18:59:04.122318+0000","last_peered":"2024-09-10T18:59:04.122318+0000","last_clean":"2024-09-10T18:59:04.122318+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:59:04.122318+0000","last_undegraded":"2024-09-10T18:59:04.122318+0000","last_fullsized":"2024-09-10T18:59:04.122318+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.707234+0000","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"377'75","reported_seq":389,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109790+0000","last_change":"2024-09-10T18:59:01.892397+0000","last_active":"2024-09-10T18:59:04.109790+0000","last_peered":"2024-09-10T18:59:04.109790+0000","last_clean":"2024-09-10T18:59:04.109790+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:59:04.109790+0000","last_undegraded":"2024-09-10T18:59:04.109790+0000","last_fullsized":"2024-09-10T18:59:04.109790+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'75","last_scrub_stamp":"2024-09-10T18:59:01.892227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.892227+0000","objects_scrubbed":7,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.595320+0000","scrub_duration":0.0063234340000000002,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":60,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"377'97","reported_seq":417,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122054+0000","last_change":"2024-09-10T18:59:01.793871+0000","last_active":"2024-09-10T18:59:04.122054+0000","last_peered":"2024-09-10T18:59:04.122054+0000","last_clean":"2024-09-10T18:59:04.122054+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:59:04.122054+0000","last_undegraded":"2024-09-10T18:59:04.122054+0000","last_fullsized":"2024-09-10T18:59:04.122054+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.183490+0000","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"146'1","reported_seq":329,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.108599+0000","last_change":"2024-09-10T18:58:43.994005+0000","last_active":"2024-09-10T18:59:04.108599+0000","last_peered":"2024-09-10T18:59:04.108599+0000","last_clean":"2024-09-10T18:59:04.108599+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:59:04.108599+0000","last_undegraded":"2024-09-10T18:59:04.108599+0000","last_fullsized":"2024-09-10T18:59:04.108599+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:58:43.993832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.993832+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.350619+0000","scrub_duration":0.0040278960000000004,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":352,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.131819+0000","last_change":"2024-09-10T18:58:39.380635+0000","last_active":"2024-09-10T18:59:04.131819+0000","last_peered":"2024-09-10T18:59:04.131819+0000","last_clean":"2024-09-10T18:59:04.131819+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:59:04.131819+0000","last_undegraded":"2024-09-10T18:59:04.131819+0000","last_fullsized":"2024-09-10T18:59:04.131819+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.953708+0000","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"297'52","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.096919+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T18:59:04.096919+0000","last_peered":"2024-09-10T18:59:04.096919+0000","last_clean":"2024-09-10T18:59:04.096919+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:59:04.096919+0000","last_undegraded":"2024-09-10T18:59:04.096919+0000","last_fullsized":"2024-09-10T18:59:04.096919+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":344,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105156+0000","last_change":"2024-09-10T18:58:37.312137+0000","last_active":"2024-09-10T18:59:04.105156+0000","last_peered":"2024-09-10T18:59:04.105156+0000","last_clean":"2024-09-10T18:59:04.105156+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:59:04.105156+0000","last_undegraded":"2024-09-10T18:59:04.105156+0000","last_fullsized":"2024-09-10T18:59:04.105156+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.542659+0000","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"326'68","reported_seq":377,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134735+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T18:59:04.134735+0000","last_peered":"2024-09-10T18:59:04.134735+0000","last_clean":"2024-09-10T18:59:04.134735+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:59:04.134735+0000","last_undegraded":"2024-09-10T18:59:04.134735+0000","last_fullsized":"2024-09-10T18:59:04.134735+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"246'119","reported_seq":414,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153119+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T18:59:04.153119+0000","last_peered":"2024-09-10T18:59:04.153119+0000","last_clean":"2024-09-10T18:59:04.153119+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:59:04.153119+0000","last_undegraded":"2024-09-10T18:59:04.153119+0000","last_fullsized":"2024-09-10T18:59:04.153119+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"377'20","reported_seq":348,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.099080+0000","last_change":"2024-09-10T18:58:56.450425+0000","last_active":"2024-09-10T18:59:04.099080+0000","last_peered":"2024-09-10T18:59:04.099080+0000","last_clean":"2024-09-10T18:59:04.099080+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:59:04.099080+0000","last_undegraded":"2024-09-10T18:59:04.099080+0000","last_fullsized":"2024-09-10T18:59:04.099080+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:25.313261+0000","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"377'23","reported_seq":339,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132505+0000","last_change":"2024-09-10T18:58:47.475482+0000","last_active":"2024-09-10T18:59:04.132505+0000","last_peered":"2024-09-10T18:59:04.132505+0000","last_clean":"2024-09-10T18:59:04.132505+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:59:04.132505+0000","last_undegraded":"2024-09-10T18:59:04.132505+0000","last_fullsized":"2024-09-10T18:59:04.132505+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'22","last_scrub_stamp":"2024-09-10T18:58:47.475285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.475285+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.029410+0000","scrub_duration":0.0041187790000000004,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":346,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121902+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T18:59:04.121902+0000","last_peered":"2024-09-10T18:59:04.121902+0000","last_clean":"2024-09-10T18:59:04.121902+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:59:04.121902+0000","last_undegraded":"2024-09-10T18:59:04.121902+0000","last_fullsized":"2024-09-10T18:59:04.121902+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.729128+0000","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"232'17","reported_seq":344,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.107109+0000","last_change":"2024-09-10T18:58:37.063649+0000","last_active":"2024-09-10T18:59:04.107109+0000","last_peered":"2024-09-10T18:59:04.107109+0000","last_clean":"2024-09-10T18:59:04.107109+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:59:04.107109+0000","last_undegraded":"2024-09-10T18:59:04.107109+0000","last_fullsized":"2024-09-10T18:59:04.107109+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:58:37.063474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.063474+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.880086+0000","scrub_duration":0.003955554,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":327,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105788+0000","last_change":"2024-09-10T18:59:03.924359+0000","last_active":"2024-09-10T18:59:04.105788+0000","last_peered":"2024-09-10T18:59:04.105788+0000","last_clean":"2024-09-10T18:59:04.105788+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:59:04.105788+0000","last_undegraded":"2024-09-10T18:59:04.105788+0000","last_fullsized":"2024-09-10T18:59:04.105788+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.924176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.924176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:06.750994+0000","scrub_duration":0.003004348,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":360,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.151426+0000","last_change":"2024-09-10T18:57:54.066122+0000","last_active":"2024-09-10T18:59:04.151426+0000","last_peered":"2024-09-10T18:59:04.151426+0000","last_clean":"2024-09-10T18:59:04.151426+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:59:04.151426+0000","last_undegraded":"2024-09-10T18:59:04.151426+0000","last_fullsized":"2024-09-10T18:59:04.151426+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T18:57:35.624392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.624392+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004362795,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"377'159","reported_seq":473,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111047+0000","last_change":"2024-09-10T18:58:53.954029+0000","last_active":"2024-09-10T18:59:04.111047+0000","last_peered":"2024-09-10T18:59:04.111047+0000","last_clean":"2024-09-10T18:59:04.111047+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:59:04.111047+0000","last_undegraded":"2024-09-10T18:59:04.111047+0000","last_fullsized":"2024-09-10T18:59:04.111047+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'159","last_scrub_stamp":"2024-09-10T18:58:53.953850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.953850+0000","objects_scrubbed":11,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.670681+0000","scrub_duration":0.0076573049999999997,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":12,"num_read_kb":2,"num_write":48,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"388'99","reported_seq":462,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.213480+0000","last_change":"2024-09-10T18:59:04.213412+0000","last_active":"2024-09-10T18:59:04.213480+0000","last_peered":"2024-09-10T18:59:04.213480+0000","last_clean":"2024-09-10T18:59:04.213480+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:59:04.213480+0000","last_undegraded":"2024-09-10T18:59:04.213480+0000","last_fullsized":"2024-09-10T18:59:04.213480+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.213299+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.577134+0000","scrub_duration":0.022641919,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133728+0000","last_change":"2024-09-10T18:58:44.435895+0000","last_active":"2024-09-10T18:59:04.133728+0000","last_peered":"2024-09-10T18:59:04.133728+0000","last_clean":"2024-09-10T18:59:04.133728+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:59:04.133728+0000","last_undegraded":"2024-09-10T18:59:04.133728+0000","last_fullsized":"2024-09-10T18:59:04.133728+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T18:58:44.435728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.435728+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.065189+0000","scrub_duration":0.0045739450000000003,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"257'107","reported_seq":407,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213506+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T18:59:03.213506+0000","last_peered":"2024-09-10T18:59:03.213506+0000","last_clean":"2024-09-10T18:59:03.213506+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:59:03.213506+0000","last_undegraded":"2024-09-10T18:59:03.213506+0000","last_fullsized":"2024-09-10T18:59:03.213506+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"377'103","reported_seq":405,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111558+0000","last_change":"2024-09-10T18:58:34.265530+0000","last_active":"2024-09-10T18:59:04.111558+0000","last_peered":"2024-09-10T18:59:04.111558+0000","last_clean":"2024-09-10T18:59:04.111558+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:59:04.111558+0000","last_undegraded":"2024-09-10T18:59:04.111558+0000","last_fullsized":"2024-09-10T18:59:04.111558+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:50.420507+0000","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":342,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152841+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T18:59:04.152841+0000","last_peered":"2024-09-10T18:59:04.152841+0000","last_clean":"2024-09-10T18:59:04.152841+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:59:04.152841+0000","last_undegraded":"2024-09-10T18:59:04.152841+0000","last_fullsized":"2024-09-10T18:59:04.152841+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.440202+0000","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"377'73","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132056+0000","last_change":"2024-09-10T18:58:32.368725+0000","last_active":"2024-09-10T18:59:04.132056+0000","last_peered":"2024-09-10T18:59:04.132056+0000","last_clean":"2024-09-10T18:59:04.132056+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:59:04.132056+0000","last_undegraded":"2024-09-10T18:59:04.132056+0000","last_fullsized":"2024-09-10T18:59:04.132056+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'72","last_scrub_stamp":"2024-09-10T18:58:32.368604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.368604+0000","objects_scrubbed":7,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:57.712897+0000","scrub_duration":0.0042920229999999998,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":73,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":409,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.104794+0000","last_change":"2024-09-10T18:58:36.017962+0000","last_active":"2024-09-10T18:59:04.104794+0000","last_peered":"2024-09-10T18:59:04.104794+0000","last_clean":"2024-09-10T18:59:04.104794+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:59:04.104794+0000","last_undegraded":"2024-09-10T18:59:04.104794+0000","last_fullsized":"2024-09-10T18:59:04.104794+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:58:36.017841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.017841+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.872443+0000","scrub_duration":0.0042543119999999997,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109103+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T18:59:04.109103+0000","last_peered":"2024-09-10T18:59:04.109103+0000","last_clean":"2024-09-10T18:59:04.109103+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:59:04.109103+0000","last_undegraded":"2024-09-10T18:59:04.109103+0000","last_fullsized":"2024-09-10T18:59:04.109103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.261172+0000","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":365,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560719+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:59:03.560719+0000","last_peered":"2024-09-10T18:59:03.560719+0000","last_clean":"2024-09-10T18:59:03.560719+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:59:03.560719+0000","last_undegraded":"2024-09-10T18:59:03.560719+0000","last_fullsized":"2024-09-10T18:59:03.560719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"77'10","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.112014+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:59:04.112014+0000","last_peered":"2024-09-10T18:59:04.112014+0000","last_clean":"2024-09-10T18:59:04.112014+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:59:04.112014+0000","last_undegraded":"2024-09-10T18:59:04.112014+0000","last_fullsized":"2024-09-10T18:59:04.112014+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109741+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T18:59:04.109741+0000","last_peered":"2024-09-10T18:59:04.109741+0000","last_clean":"2024-09-10T18:59:04.109741+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:59:04.109741+0000","last_undegraded":"2024-09-10T18:59:04.109741+0000","last_fullsized":"2024-09-10T18:59:04.109741+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:04.994389+0000","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":280,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.131636+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:59:04.131636+0000","last_peered":"2024-09-10T18:59:04.131636+0000","last_clean":"2024-09-10T18:59:04.131636+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:59:04.131636+0000","last_undegraded":"2024-09-10T18:59:04.131636+0000","last_fullsized":"2024-09-10T18:59:04.131636+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":378,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.115843+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T18:59:04.115843+0000","last_peered":"2024-09-10T18:59:04.115843+0000","last_clean":"2024-09-10T18:59:04.115843+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:59:04.115843+0000","last_undegraded":"2024-09-10T18:59:04.115843+0000","last_fullsized":"2024-09-10T18:59:04.115843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.048099+0000","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'11","reported_seq":382,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560421+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:59:03.560421+0000","last_peered":"2024-09-10T18:59:03.560421+0000","last_clean":"2024-09-10T18:59:03.560421+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:59:03.560421+0000","last_undegraded":"2024-09-10T18:59:03.560421+0000","last_fullsized":"2024-09-10T18:59:03.560421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097735+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:59:04.097735+0000","last_peered":"2024-09-10T18:59:04.097735+0000","last_clean":"2024-09-10T18:59:04.097735+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:59:04.097735+0000","last_undegraded":"2024-09-10T18:59:04.097735+0000","last_fullsized":"2024-09-10T18:59:04.097735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":372,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097586+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T18:59:04.097586+0000","last_peered":"2024-09-10T18:59:04.097586+0000","last_clean":"2024-09-10T18:59:04.097586+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:59:04.097586+0000","last_undegraded":"2024-09-10T18:59:04.097586+0000","last_fullsized":"2024-09-10T18:59:04.097586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:46.825797+0000","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559624+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T18:59:03.559624+0000","last_peered":"2024-09-10T18:59:03.559624+0000","last_clean":"2024-09-10T18:59:03.559624+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:59:03.559624+0000","last_undegraded":"2024-09-10T18:59:03.559624+0000","last_fullsized":"2024-09-10T18:59:03.559624+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":373,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213593+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T18:59:03.213593+0000","last_peered":"2024-09-10T18:59:03.213593+0000","last_clean":"2024-09-10T18:59:03.213593+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:59:03.213593+0000","last_undegraded":"2024-09-10T18:59:03.213593+0000","last_fullsized":"2024-09-10T18:59:03.213593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.053990+0000","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"108'22","reported_seq":393,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134191+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T18:59:04.134191+0000","last_peered":"2024-09-10T18:59:04.134191+0000","last_clean":"2024-09-10T18:59:04.134191+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:59:04.134191+0000","last_undegraded":"2024-09-10T18:59:04.134191+0000","last_fullsized":"2024-09-10T18:59:04.134191+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097624+0000","last_change":"2024-09-10T18:58:53.495436+0000","last_active":"2024-09-10T18:59:04.097624+0000","last_peered":"2024-09-10T18:59:04.097624+0000","last_clean":"2024-09-10T18:59:04.097624+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:59:04.097624+0000","last_undegraded":"2024-09-10T18:59:04.097624+0000","last_fullsized":"2024-09-10T18:59:04.097624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.495273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.495273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.149497+0000","scrub_duration":0.0024582229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":360,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132171+0000","last_change":"2024-09-10T18:57:31.277231+0000","last_active":"2024-09-10T18:59:04.132171+0000","last_peered":"2024-09-10T18:59:04.132171+0000","last_clean":"2024-09-10T18:59:04.132171+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:59:04.132171+0000","last_undegraded":"2024-09-10T18:59:04.132171+0000","last_fullsized":"2024-09-10T18:59:04.132171+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.277085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.277085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003344746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":291,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122236+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T18:59:04.122236+0000","last_peered":"2024-09-10T18:59:04.122236+0000","last_clean":"2024-09-10T18:59:04.122236+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:59:04.122236+0000","last_undegraded":"2024-09-10T18:59:04.122236+0000","last_fullsized":"2024-09-10T18:59:04.122236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:59.115727+0000","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":376,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.217037+0000","last_change":"2024-09-10T18:59:03.215137+0000","last_active":"2024-09-10T18:59:03.217037+0000","last_peered":"2024-09-10T18:59:03.217037+0000","last_clean":"2024-09-10T18:59:03.217037+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:59:03.217037+0000","last_undegraded":"2024-09-10T18:59:03.217037+0000","last_fullsized":"2024-09-10T18:59:03.217037+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.215039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.215039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:32.858523+0000","scrub_duration":0.0064108910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"95'17","reported_seq":409,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:05.516421+0000","last_change":"2024-09-10T18:59:05.516222+0000","last_active":"2024-09-10T18:59:05.516421+0000","last_peered":"2024-09-10T18:59:05.516421+0000","last_clean":"2024-09-10T18:59:05.516421+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:59:05.516421+0000","last_undegraded":"2024-09-10T18:59:05.516421+0000","last_fullsized":"2024-09-10T18:59:05.516421+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:59:05.516062+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:05.516062+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.578286+0000","scrub_duration":0.0058657070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097847+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T18:59:04.097847+0000","last_peered":"2024-09-10T18:59:04.097847+0000","last_clean":"2024-09-10T18:59:04.097847+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:59:04.097847+0000","last_undegraded":"2024-09-10T18:59:04.097847+0000","last_fullsized":"2024-09-10T18:59:04.097847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:16.804733+0000","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":369,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.789098+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T18:59:04.789098+0000","last_peered":"2024-09-10T18:59:04.789098+0000","last_clean":"2024-09-10T18:59:04.789098+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:59:04.789098+0000","last_undegraded":"2024-09-10T18:59:04.789098+0000","last_fullsized":"2024-09-10T18:59:04.789098+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.122348+0000","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":283,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.108399+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:59:04.108399+0000","last_peered":"2024-09-10T18:59:04.108399+0000","last_clean":"2024-09-10T18:59:04.108399+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:59:04.108399+0000","last_undegraded":"2024-09-10T18:59:04.108399+0000","last_fullsized":"2024-09-10T18:59:04.108399+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"77'19","reported_seq":404,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153122+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T18:59:04.153122+0000","last_peered":"2024-09-10T18:59:04.153122+0000","last_clean":"2024-09-10T18:59:04.153122+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:59:04.153122+0000","last_undegraded":"2024-09-10T18:59:04.153122+0000","last_fullsized":"2024-09-10T18:59:04.153122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.716139+0000","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":363,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.217137+0000","last_change":"2024-09-10T18:57:34.379150+0000","last_active":"2024-09-10T18:59:03.217137+0000","last_peered":"2024-09-10T18:59:03.217137+0000","last_clean":"2024-09-10T18:59:03.217137+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:59:03.217137+0000","last_undegraded":"2024-09-10T18:59:03.217137+0000","last_fullsized":"2024-09-10T18:59:03.217137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.378985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.378985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00364804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":502,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100330+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T18:59:04.100330+0000","last_peered":"2024-09-10T18:59:04.100330+0000","last_clean":"2024-09-10T18:59:04.100330+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:59:04.100330+0000","last_undegraded":"2024-09-10T18:59:04.100330+0000","last_fullsized":"2024-09-10T18:59:04.100330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.009195+0000","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":380,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.115937+0000","last_change":"2024-09-10T18:59:00.907296+0000","last_active":"2024-09-10T18:59:04.115937+0000","last_peered":"2024-09-10T18:59:04.115937+0000","last_clean":"2024-09-10T18:59:04.115937+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:59:04.115937+0000","last_undegraded":"2024-09-10T18:59:04.115937+0000","last_fullsized":"2024-09-10T18:59:04.115937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.907056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.861464+0000","scrub_duration":0.0036581080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":282,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153046+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:59:04.153046+0000","last_peered":"2024-09-10T18:59:04.153046+0000","last_clean":"2024-09-10T18:59:04.153046+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:59:04.153046+0000","last_undegraded":"2024-09-10T18:59:04.153046+0000","last_fullsized":"2024-09-10T18:59:04.153046+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"95'13","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111714+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:59:04.111714+0000","last_peered":"2024-09-10T18:59:04.111714+0000","last_clean":"2024-09-10T18:59:04.111714+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:59:04.111714+0000","last_undegraded":"2024-09-10T18:59:04.111714+0000","last_fullsized":"2024-09-10T18:59:04.111714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":385,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121649+0000","last_change":"2024-09-10T18:58:52.884147+0000","last_active":"2024-09-10T18:59:04.121649+0000","last_peered":"2024-09-10T18:59:04.121649+0000","last_clean":"2024-09-10T18:59:04.121649+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:59:04.121649+0000","last_undegraded":"2024-09-10T18:59:04.121649+0000","last_fullsized":"2024-09-10T18:59:04.121649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.884077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.965200+0000","scrub_duration":0.0032688370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097873+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:59:04.097873+0000","last_peered":"2024-09-10T18:59:04.097873+0000","last_clean":"2024-09-10T18:59:04.097873+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:59:04.097873+0000","last_undegraded":"2024-09-10T18:59:04.097873+0000","last_fullsized":"2024-09-10T18:59:04.097873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":372,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100045+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T18:59:04.100045+0000","last_peered":"2024-09-10T18:59:04.100045+0000","last_clean":"2024-09-10T18:59:04.100045+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:59:04.100045+0000","last_undegraded":"2024-09-10T18:59:04.100045+0000","last_fullsized":"2024-09-10T18:59:04.100045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.012955+0000","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":282,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152169+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T18:59:04.152169+0000","last_peered":"2024-09-10T18:59:04.152169+0000","last_clean":"2024-09-10T18:59:04.152169+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:59:04.152169+0000","last_undegraded":"2024-09-10T18:59:04.152169+0000","last_fullsized":"2024-09-10T18:59:04.152169+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"77'15","reported_seq":391,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.106372+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:59:04.106372+0000","last_peered":"2024-09-10T18:59:04.106372+0000","last_clean":"2024-09-10T18:59:04.106372+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:59:04.106372+0000","last_undegraded":"2024-09-10T18:59:04.106372+0000","last_fullsized":"2024-09-10T18:59:04.106372+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121450+0000","last_change":"2024-09-10T18:58:51.907906+0000","last_active":"2024-09-10T18:59:04.121450+0000","last_peered":"2024-09-10T18:59:04.121450+0000","last_clean":"2024-09-10T18:59:04.121450+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:59:04.121450+0000","last_undegraded":"2024-09-10T18:59:04.121450+0000","last_fullsized":"2024-09-10T18:59:04.121450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:51.907749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:51.907749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.097418+0000","scrub_duration":0.0036594549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":381,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122770+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T18:59:04.122770+0000","last_peered":"2024-09-10T18:59:04.122770+0000","last_clean":"2024-09-10T18:59:04.122770+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:59:04.122770+0000","last_undegraded":"2024-09-10T18:59:04.122770+0000","last_fullsized":"2024-09-10T18:59:04.122770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.611116+0000","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":367,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133195+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:59:04.133195+0000","last_peered":"2024-09-10T18:59:04.133195+0000","last_clean":"2024-09-10T18:59:04.133195+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:59:04.133195+0000","last_undegraded":"2024-09-10T18:59:04.133195+0000","last_fullsized":"2024-09-10T18:59:04.133195+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":291,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122910+0000","last_change":"2024-09-10T18:57:56.827061+0000","last_active":"2024-09-10T18:59:04.122910+0000","last_peered":"2024-09-10T18:59:04.122910+0000","last_clean":"2024-09-10T18:59:04.122910+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:59:04.122910+0000","last_undegraded":"2024-09-10T18:59:04.122910+0000","last_fullsized":"2024-09-10T18:59:04.122910+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.826954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.826954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.327774+0000","scrub_duration":0.0032518439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'17","reported_seq":414,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098700+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T18:59:04.098700+0000","last_peered":"2024-09-10T18:59:04.098700+0000","last_clean":"2024-09-10T18:59:04.098700+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:59:04.098700+0000","last_undegraded":"2024-09-10T18:59:04.098700+0000","last_fullsized":"2024-09-10T18:59:04.098700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:27.341520+0000","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":373,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213195+0000","last_change":"2024-09-10T18:58:32.226883+0000","last_active":"2024-09-10T18:59:03.213195+0000","last_peered":"2024-09-10T18:59:03.213195+0000","last_clean":"2024-09-10T18:59:03.213195+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:59:03.213195+0000","last_undegraded":"2024-09-10T18:59:03.213195+0000","last_fullsized":"2024-09-10T18:59:03.213195+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.226771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.226771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.723944+0000","scrub_duration":0.024188028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":502,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100299+0000","last_change":"2024-09-10T18:57:32.764761+0000","last_active":"2024-09-10T18:59:04.100299+0000","last_peered":"2024-09-10T18:59:04.100299+0000","last_clean":"2024-09-10T18:59:04.100299+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:59:04.100299+0000","last_undegraded":"2024-09-10T18:59:04.100299+0000","last_fullsized":"2024-09-10T18:59:04.100299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:57:32.764561+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.764561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041273020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":363,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111652+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:59:04.111652+0000","last_peered":"2024-09-10T18:59:04.111652+0000","last_clean":"2024-09-10T18:59:04.111652+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:59:04.111652+0000","last_undegraded":"2024-09-10T18:59:04.111652+0000","last_fullsized":"2024-09-10T18:59:04.111652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":291,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122901+0000","last_change":"2024-09-10T18:58:47.871654+0000","last_active":"2024-09-10T18:59:04.122901+0000","last_peered":"2024-09-10T18:59:04.122901+0000","last_clean":"2024-09-10T18:59:04.122901+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:59:04.122901+0000","last_undegraded":"2024-09-10T18:59:04.122901+0000","last_fullsized":"2024-09-10T18:59:04.122901+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.871591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.871591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.791278+0000","scrub_duration":0.0023117910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"77'12","reported_seq":381,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560307+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T18:59:03.560307+0000","last_peered":"2024-09-10T18:59:03.560307+0000","last_clean":"2024-09-10T18:59:03.560307+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:59:03.560307+0000","last_undegraded":"2024-09-10T18:59:03.560307+0000","last_fullsized":"2024-09-10T18:59:03.560307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:11.981000+0000","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.154091+0000","last_change":"2024-09-10T18:58:50.597419+0000","last_active":"2024-09-10T18:59:04.154091+0000","last_peered":"2024-09-10T18:59:04.154091+0000","last_clean":"2024-09-10T18:59:04.154091+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:59:04.154091+0000","last_undegraded":"2024-09-10T18:59:04.154091+0000","last_fullsized":"2024-09-10T18:59:04.154091+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.597321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.597321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.919633+0000","scrub_duration":0.0021838199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098318+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T18:59:04.098318+0000","last_peered":"2024-09-10T18:59:04.098318+0000","last_clean":"2024-09-10T18:59:04.098318+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:59:04.098318+0000","last_undegraded":"2024-09-10T18:59:04.098318+0000","last_fullsized":"2024-09-10T18:59:04.098318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":944,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.922318+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T18:59:04.922318+0000","last_peered":"2024-09-10T18:59:04.922318+0000","last_clean":"2024-09-10T18:59:04.922318+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:59:04.922318+0000","last_undegraded":"2024-09-10T18:59:04.922318+0000","last_fullsized":"2024-09-10T18:59:04.922318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.772656+0000","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":562,"num_read_kb":557,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":367,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213341+0000","last_change":"2024-09-10T18:57:49.386576+0000","last_active":"2024-09-10T18:59:03.213341+0000","last_peered":"2024-09-10T18:59:03.213341+0000","last_clean":"2024-09-10T18:59:03.213341+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:59:03.213341+0000","last_undegraded":"2024-09-10T18:59:03.213341+0000","last_fullsized":"2024-09-10T18:59:03.213341+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.386404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003967628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105573+0000","last_change":"2024-09-10T18:57:49.919967+0000","last_active":"2024-09-10T18:59:04.105573+0000","last_peered":"2024-09-10T18:59:04.105573+0000","last_clean":"2024-09-10T18:59:04.105573+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:59:04.105573+0000","last_undegraded":"2024-09-10T18:59:04.105573+0000","last_fullsized":"2024-09-10T18:59:04.105573+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.919794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.919794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.849294+0000","scrub_duration":0.0034551590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"77'13","reported_seq":403,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.117034+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T18:59:04.117034+0000","last_peered":"2024-09-10T18:59:04.117034+0000","last_clean":"2024-09-10T18:59:04.117034+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:59:04.117034+0000","last_undegraded":"2024-09-10T18:59:04.117034+0000","last_fullsized":"2024-09-10T18:59:04.117034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.664434+0000","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":385,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.123339+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T18:59:04.123339+0000","last_peered":"2024-09-10T18:59:04.123339+0000","last_clean":"2024-09-10T18:59:04.123339+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:59:04.123339+0000","last_undegraded":"2024-09-10T18:59:04.123339+0000","last_fullsized":"2024-09-10T18:59:04.123339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.410089+0000","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":481,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105918+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:59:04.105918+0000","last_peered":"2024-09-10T18:59:04.105918+0000","last_clean":"2024-09-10T18:59:04.105918+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:59:04.105918+0000","last_undegraded":"2024-09-10T18:59:04.105918+0000","last_fullsized":"2024-09-10T18:59:04.105918+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":391,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.452121+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T18:59:04.452121+0000","last_peered":"2024-09-10T18:59:04.452121+0000","last_clean":"2024-09-10T18:59:04.452121+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:59:04.452121+0000","last_undegraded":"2024-09-10T18:59:04.452121+0000","last_fullsized":"2024-09-10T18:59:04.452121+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.789834+0000","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":372,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097926+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T18:59:04.097926+0000","last_peered":"2024-09-10T18:59:04.097926+0000","last_clean":"2024-09-10T18:59:04.097926+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:59:04.097926+0000","last_undegraded":"2024-09-10T18:59:04.097926+0000","last_fullsized":"2024-09-10T18:59:04.097926+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:19.820859+0000","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":283,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105768+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T18:59:04.105768+0000","last_peered":"2024-09-10T18:59:04.105768+0000","last_clean":"2024-09-10T18:59:04.105768+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:59:04.105768+0000","last_undegraded":"2024-09-10T18:59:04.105768+0000","last_fullsized":"2024-09-10T18:59:04.105768+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"113'40","reported_seq":550,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122692+0000","last_change":"2024-09-10T18:57:50.752463+0000","last_active":"2024-09-10T18:59:04.122692+0000","last_peered":"2024-09-10T18:59:04.122692+0000","last_clean":"2024-09-10T18:59:04.122692+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:59:04.122692+0000","last_undegraded":"2024-09-10T18:59:04.122692+0000","last_fullsized":"2024-09-10T18:59:04.122692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:57:50.752292+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.752292+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":373,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.217164+0000","last_change":"2024-09-10T18:58:50.209102+0000","last_active":"2024-09-10T18:59:03.217164+0000","last_peered":"2024-09-10T18:59:03.217164+0000","last_clean":"2024-09-10T18:59:03.217164+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:59:03.217164+0000","last_undegraded":"2024-09-10T18:59:03.217164+0000","last_fullsized":"2024-09-10T18:59:03.217164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.208993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.208993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.944132+0000","scrub_duration":0.003212126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":489,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559802+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:59:03.559802+0000","last_peered":"2024-09-10T18:59:03.559802+0000","last_clean":"2024-09-10T18:59:03.559802+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:59:03.559802+0000","last_undegraded":"2024-09-10T18:59:03.559802+0000","last_fullsized":"2024-09-10T18:59:03.559802+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":379,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152939+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T18:59:04.152939+0000","last_peered":"2024-09-10T18:59:04.152939+0000","last_clean":"2024-09-10T18:59:04.152939+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:59:04.152939+0000","last_undegraded":"2024-09-10T18:59:04.152939+0000","last_fullsized":"2024-09-10T18:59:04.152939+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.942357+0000","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":369,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121125+0000","last_change":"2024-09-10T18:58:45.871468+0000","last_active":"2024-09-10T18:59:04.121125+0000","last_peered":"2024-09-10T18:59:04.121125+0000","last_clean":"2024-09-10T18:59:04.121125+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:59:04.121125+0000","last_undegraded":"2024-09-10T18:59:04.121125+0000","last_fullsized":"2024-09-10T18:59:04.121125+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:45.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:45.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.946492+0000","scrub_duration":0.0033902770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":283,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.112097+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T18:59:04.112097+0000","last_peered":"2024-09-10T18:59:04.112097+0000","last_clean":"2024-09-10T18:59:04.112097+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:59:04.112097+0000","last_undegraded":"2024-09-10T18:59:04.112097+0000","last_fullsized":"2024-09-10T18:59:04.112097+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"77'14","reported_seq":393,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133548+0000","last_change":"2024-09-10T18:58:46.437660+0000","last_active":"2024-09-10T18:59:04.133548+0000","last_peered":"2024-09-10T18:59:04.133548+0000","last_clean":"2024-09-10T18:59:04.133548+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:59:04.133548+0000","last_undegraded":"2024-09-10T18:59:04.133548+0000","last_fullsized":"2024-09-10T18:59:04.133548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:46.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.437410+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.595024+0000","scrub_duration":0.0054784669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122688+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T18:59:04.122688+0000","last_peered":"2024-09-10T18:59:04.122688+0000","last_clean":"2024-09-10T18:59:04.122688+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:59:04.122688+0000","last_undegraded":"2024-09-10T18:59:04.122688+0000","last_fullsized":"2024-09-10T18:59:04.122688+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.994362+0000","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121448+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T18:59:04.121448+0000","last_peered":"2024-09-10T18:59:04.121448+0000","last_clean":"2024-09-10T18:59:04.121448+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:59:04.121448+0000","last_undegraded":"2024-09-10T18:59:04.121448+0000","last_fullsized":"2024-09-10T18:59:04.121448+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.143473+0000","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":359,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.558799+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:59:03.558799+0000","last_peered":"2024-09-10T18:59:03.558799+0000","last_clean":"2024-09-10T18:59:03.558799+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:59:03.558799+0000","last_undegraded":"2024-09-10T18:59:03.558799+0000","last_fullsized":"2024-09-10T18:59:03.558799+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.107233+0000","last_change":"2024-09-10T18:57:43.914840+0000","last_active":"2024-09-10T18:59:04.107233+0000","last_peered":"2024-09-10T18:59:04.107233+0000","last_clean":"2024-09-10T18:59:04.107233+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:59:04.107233+0000","last_undegraded":"2024-09-10T18:59:04.107233+0000","last_fullsized":"2024-09-10T18:59:04.107233+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:43.914731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:43.914731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.644461+0000","scrub_duration":0.0026766530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"77'19","reported_seq":412,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.099436+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T18:59:04.099436+0000","last_peered":"2024-09-10T18:59:04.099436+0000","last_clean":"2024-09-10T18:59:04.099436+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:59:04.099436+0000","last_undegraded":"2024-09-10T18:59:04.099436+0000","last_fullsized":"2024-09-10T18:59:04.099436+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:08.776283+0000","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":373,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133127+0000","last_change":"2024-09-10T18:58:52.447688+0000","last_active":"2024-09-10T18:59:04.133127+0000","last_peered":"2024-09-10T18:59:04.133127+0000","last_clean":"2024-09-10T18:59:04.133127+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:59:04.133127+0000","last_undegraded":"2024-09-10T18:59:04.133127+0000","last_fullsized":"2024-09-10T18:59:04.133127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.447547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.447547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.724802+0000","scrub_duration":0.0031696559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122568+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T18:59:04.122568+0000","last_peered":"2024-09-10T18:59:04.122568+0000","last_clean":"2024-09-10T18:59:04.122568+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:59:04.122568+0000","last_undegraded":"2024-09-10T18:59:04.122568+0000","last_fullsized":"2024-09-10T18:59:04.122568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.242393+0000","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111867+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T18:59:04.111867+0000","last_peered":"2024-09-10T18:59:04.111867+0000","last_clean":"2024-09-10T18:59:04.111867+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:59:04.111867+0000","last_undegraded":"2024-09-10T18:59:04.111867+0000","last_fullsized":"2024-09-10T18:59:04.111867+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.784439+0000","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559378+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:59:03.559378+0000","last_peered":"2024-09-10T18:59:03.559378+0000","last_clean":"2024-09-10T18:59:03.559378+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:59:03.559378+0000","last_undegraded":"2024-09-10T18:59:03.559378+0000","last_fullsized":"2024-09-10T18:59:03.559378+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"104'26","reported_seq":413,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121233+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T18:59:04.121233+0000","last_peered":"2024-09-10T18:59:04.121233+0000","last_clean":"2024-09-10T18:59:04.121233+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:59:04.121233+0000","last_undegraded":"2024-09-10T18:59:04.121233+0000","last_fullsized":"2024-09-10T18:59:04.121233+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:05.333581+0000","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105263+0000","last_change":"2024-09-10T18:57:55.986832+0000","last_active":"2024-09-10T18:59:04.105263+0000","last_peered":"2024-09-10T18:59:04.105263+0000","last_clean":"2024-09-10T18:59:04.105263+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:59:04.105263+0000","last_undegraded":"2024-09-10T18:59:04.105263+0000","last_fullsized":"2024-09-10T18:59:04.105263+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:55.986726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.986726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:23.114093+0000","scrub_duration":0.0034130330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.123044+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T18:59:04.123044+0000","last_peered":"2024-09-10T18:59:04.123044+0000","last_clean":"2024-09-10T18:59:04.123044+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:59:04.123044+0000","last_undegraded":"2024-09-10T18:59:04.123044+0000","last_fullsized":"2024-09-10T18:59:04.123044+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.829268+0000","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":367,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214337+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T18:59:03.214337+0000","last_peered":"2024-09-10T18:59:03.214337+0000","last_clean":"2024-09-10T18:59:03.214337+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:59:03.214337+0000","last_undegraded":"2024-09-10T18:59:03.214337+0000","last_fullsized":"2024-09-10T18:59:03.214337+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153702+0000","last_change":"2024-09-10T18:59:02.713009+0000","last_active":"2024-09-10T18:59:04.153702+0000","last_peered":"2024-09-10T18:59:04.153702+0000","last_clean":"2024-09-10T18:59:04.153702+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:59:04.153702+0000","last_undegraded":"2024-09-10T18:59:04.153702+0000","last_fullsized":"2024-09-10T18:59:04.153702+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.712839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.712839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:04.351359+0000","scrub_duration":0.0035464870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"77'14","reported_seq":394,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.558618+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T18:59:03.558618+0000","last_peered":"2024-09-10T18:59:03.558618+0000","last_clean":"2024-09-10T18:59:03.558618+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:59:03.558618+0000","last_undegraded":"2024-09-10T18:59:03.558618+0000","last_fullsized":"2024-09-10T18:59:03.558618+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.541418+0000","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":373,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.212460+0000","last_change":"2024-09-10T18:58:44.193043+0000","last_active":"2024-09-10T18:59:03.212460+0000","last_peered":"2024-09-10T18:59:03.212460+0000","last_clean":"2024-09-10T18:59:03.212460+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:59:03.212460+0000","last_undegraded":"2024-09-10T18:59:03.212460+0000","last_fullsized":"2024-09-10T18:59:03.212460+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.192853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.192853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.060471+0000","scrub_duration":0.0038691179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":371,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.826417+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T18:59:03.826417+0000","last_peered":"2024-09-10T18:59:03.826417+0000","last_clean":"2024-09-10T18:59:03.826417+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:59:03.826417+0000","last_undegraded":"2024-09-10T18:59:03.826417+0000","last_fullsized":"2024-09-10T18:59:03.826417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.522110+0000","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":369,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122249+0000","last_change":"2024-09-10T18:57:44.612113+0000","last_active":"2024-09-10T18:59:04.122249+0000","last_peered":"2024-09-10T18:59:04.122249+0000","last_clean":"2024-09-10T18:59:04.122249+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:59:04.122249+0000","last_undegraded":"2024-09-10T18:59:04.122249+0000","last_fullsized":"2024-09-10T18:59:04.122249+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:44.612043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:57:44.612043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026996279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560745+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:59:03.560745+0000","last_peered":"2024-09-10T18:59:03.560745+0000","last_clean":"2024-09-10T18:59:03.560745+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:59:03.560745+0000","last_undegraded":"2024-09-10T18:59:03.560745+0000","last_fullsized":"2024-09-10T18:59:03.560745+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"109'15","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.108526+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T18:59:04.108526+0000","last_peered":"2024-09-10T18:59:04.108526+0000","last_clean":"2024-09-10T18:59:04.108526+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:59:04.108526+0000","last_undegraded":"2024-09-10T18:59:04.108526+0000","last_fullsized":"2024-09-10T18:59:04.108526+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.648653+0000","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":373,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133288+0000","last_change":"2024-09-10T18:57:56.125998+0000","last_active":"2024-09-10T18:59:04.133288+0000","last_peered":"2024-09-10T18:59:04.133288+0000","last_clean":"2024-09-10T18:59:04.133288+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:59:04.133288+0000","last_undegraded":"2024-09-10T18:59:04.133288+0000","last_fullsized":"2024-09-10T18:59:04.133288+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.125881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.125881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026334969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":421,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.212719+0000","last_change":"2024-09-10T18:57:56.375088+0000","last_active":"2024-09-10T18:59:03.212719+0000","last_peered":"2024-09-10T18:59:03.212719+0000","last_clean":"2024-09-10T18:59:03.212719+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:59:03.212719+0000","last_undegraded":"2024-09-10T18:59:03.212719+0000","last_fullsized":"2024-09-10T18:59:03.212719+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:57:56.374923+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.374923+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043604359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153737+0000","last_change":"2024-09-10T18:58:47.555840+0000","last_active":"2024-09-10T18:59:04.153737+0000","last_peered":"2024-09-10T18:59:04.153737+0000","last_clean":"2024-09-10T18:59:04.153737+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:59:04.153737+0000","last_undegraded":"2024-09-10T18:59:04.153737+0000","last_fullsized":"2024-09-10T18:59:04.153737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.555637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.555637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.468673+0000","scrub_duration":0.0032502030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":367,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134445+0000","last_change":"2024-09-10T18:58:59.454163+0000","last_active":"2024-09-10T18:59:04.134445+0000","last_peered":"2024-09-10T18:59:04.134445+0000","last_clean":"2024-09-10T18:59:04.134445+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:59:04.134445+0000","last_undegraded":"2024-09-10T18:59:04.134445+0000","last_fullsized":"2024-09-10T18:59:04.134445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.565579+0000","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":291,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.215035+0000","last_change":"2024-09-10T18:58:54.237739+0000","last_active":"2024-09-10T18:59:03.215035+0000","last_peered":"2024-09-10T18:59:03.215035+0000","last_clean":"2024-09-10T18:59:03.215035+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:59:03.215035+0000","last_undegraded":"2024-09-10T18:59:03.215035+0000","last_fullsized":"2024-09-10T18:59:03.215035+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.237585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.237585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.680207+0000","scrub_duration":0.0092150240000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"376'25","reported_seq":407,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.217064+0000","last_change":"2024-09-10T18:58:52.253613+0000","last_active":"2024-09-10T18:59:03.217064+0000","last_peered":"2024-09-10T18:59:03.217064+0000","last_clean":"2024-09-10T18:59:03.217064+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:59:03.217064+0000","last_undegraded":"2024-09-10T18:59:03.217064+0000","last_fullsized":"2024-09-10T18:59:03.217064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'25","last_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.253376+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.678764+0000","scrub_duration":0.011205784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.110159+0000","last_change":"2024-09-10T18:58:38.064448+0000","last_active":"2024-09-10T18:59:04.110159+0000","last_peered":"2024-09-10T18:59:04.110159+0000","last_clean":"2024-09-10T18:59:04.110159+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:59:04.110159+0000","last_undegraded":"2024-09-10T18:59:04.110159+0000","last_fullsized":"2024-09-10T18:59:04.110159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.064267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.064267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.065185+0000","scrub_duration":0.0037942050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":371,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153077+0000","last_change":"2024-09-10T18:58:42.486651+0000","last_active":"2024-09-10T18:59:04.153077+0000","last_peered":"2024-09-10T18:59:04.153077+0000","last_clean":"2024-09-10T18:59:04.153077+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:59:04.153077+0000","last_undegraded":"2024-09-10T18:59:04.153077+0000","last_fullsized":"2024-09-10T18:59:04.153077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:42.486498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:42.486498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:47.324824+0000","scrub_duration":0.0032582879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":367,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213366+0000","last_change":"2024-09-10T18:58:55.261885+0000","last_active":"2024-09-10T18:59:03.213366+0000","last_peered":"2024-09-10T18:59:03.213366+0000","last_clean":"2024-09-10T18:59:03.213366+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:59:03.213366+0000","last_undegraded":"2024-09-10T18:59:03.213366+0000","last_fullsized":"2024-09-10T18:59:03.213366+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.261635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.261635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.433144+0000","scrub_duration":0.0046216190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":293,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097356+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T18:59:04.097356+0000","last_peered":"2024-09-10T18:59:04.097356+0000","last_clean":"2024-09-10T18:59:04.097356+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:59:04.097356+0000","last_undegraded":"2024-09-10T18:59:04.097356+0000","last_fullsized":"2024-09-10T18:59:04.097356+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.666116+0000","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"115'15","reported_seq":391,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134409+0000","last_change":"2024-09-10T18:59:01.489222+0000","last_active":"2024-09-10T18:59:04.134409+0000","last_peered":"2024-09-10T18:59:04.134409+0000","last_clean":"2024-09-10T18:59:04.134409+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:59:04.134409+0000","last_undegraded":"2024-09-10T18:59:04.134409+0000","last_fullsized":"2024-09-10T18:59:04.134409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:59:01.489057+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.489057+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:21.754146+0000","scrub_duration":0.0045114960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.123244+0000","last_change":"2024-09-10T18:58:38.864525+0000","last_active":"2024-09-10T18:59:04.123244+0000","last_peered":"2024-09-10T18:59:04.123244+0000","last_clean":"2024-09-10T18:59:04.123244+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:59:04.123244+0000","last_undegraded":"2024-09-10T18:59:04.123244+0000","last_fullsized":"2024-09-10T18:59:04.123244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.864360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.864360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.640967+0000","scrub_duration":0.00368761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":361,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559559+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T18:59:03.559559+0000","last_peered":"2024-09-10T18:59:03.559559+0000","last_clean":"2024-09-10T18:59:03.559559+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:59:03.559559+0000","last_undegraded":"2024-09-10T18:59:03.559559+0000","last_fullsized":"2024-09-10T18:59:03.559559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.575228+0000","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":359,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.558383+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T18:59:03.558383+0000","last_peered":"2024-09-10T18:59:03.558383+0000","last_clean":"2024-09-10T18:59:03.558383+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:59:03.558383+0000","last_undegraded":"2024-09-10T18:59:03.558383+0000","last_fullsized":"2024-09-10T18:59:03.558383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:03.423360+0000","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559935+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:59:03.559935+0000","last_peered":"2024-09-10T18:59:03.559935+0000","last_clean":"2024-09-10T18:59:03.559935+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:59:03.559935+0000","last_undegraded":"2024-09-10T18:59:03.559935+0000","last_fullsized":"2024-09-10T18:59:03.559935+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"104'22","reported_seq":403,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214735+0000","last_change":"2024-09-10T18:58:48.217515+0000","last_active":"2024-09-10T18:59:03.214735+0000","last_peered":"2024-09-10T18:59:03.214735+0000","last_clean":"2024-09-10T18:59:03.214735+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:59:03.214735+0000","last_undegraded":"2024-09-10T18:59:03.214735+0000","last_fullsized":"2024-09-10T18:59:03.214735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:58:48.217409+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.217409+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.394749+0000","scrub_duration":0.0034746339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":398,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.115628+0000","last_change":"2024-09-10T18:58:34.970321+0000","last_active":"2024-09-10T18:59:04.115628+0000","last_peered":"2024-09-10T18:59:04.115628+0000","last_clean":"2024-09-10T18:59:04.115628+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:59:04.115628+0000","last_undegraded":"2024-09-10T18:59:04.115628+0000","last_fullsized":"2024-09-10T18:59:04.115628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:34.970167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.970167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.762548+0000","scrub_duration":0.0036040859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":369,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.131633+0000","last_change":"2024-09-10T18:57:34.244420+0000","last_active":"2024-09-10T18:59:04.131633+0000","last_peered":"2024-09-10T18:59:04.131633+0000","last_clean":"2024-09-10T18:59:04.131633+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:59:04.131633+0000","last_undegraded":"2024-09-10T18:59:04.131633+0000","last_fullsized":"2024-09-10T18:59:04.131633+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.244346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.244346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00228093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":367,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132689+0000","last_change":"2024-09-10T18:58:37.374134+0000","last_active":"2024-09-10T18:59:04.132689+0000","last_peered":"2024-09-10T18:59:04.132689+0000","last_clean":"2024-09-10T18:59:04.132689+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:59:04.132689+0000","last_undegraded":"2024-09-10T18:59:04.132689+0000","last_fullsized":"2024-09-10T18:59:04.132689+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.374014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.374014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.692319+0000","scrub_duration":0.0023318560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":293,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097113+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T18:59:04.097113+0000","last_peered":"2024-09-10T18:59:04.097113+0000","last_clean":"2024-09-10T18:59:04.097113+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:59:04.097113+0000","last_undegraded":"2024-09-10T18:59:04.097113+0000","last_fullsized":"2024-09-10T18:59:04.097113+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.117442+0000","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"77'11","reported_seq":390,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214555+0000","last_change":"2024-09-10T18:58:47.178004+0000","last_active":"2024-09-10T18:59:03.214555+0000","last_peered":"2024-09-10T18:59:03.214555+0000","last_clean":"2024-09-10T18:59:03.214555+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:59:03.214555+0000","last_undegraded":"2024-09-10T18:59:03.214555+0000","last_fullsized":"2024-09-10T18:59:03.214555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:47.177751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.177751+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:03.310286+0000","scrub_duration":0.0059440730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":417,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100683+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:59:04.100683+0000","last_peered":"2024-09-10T18:59:04.100683+0000","last_clean":"2024-09-10T18:59:04.100683+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:59:04.100683+0000","last_undegraded":"2024-09-10T18:59:04.100683+0000","last_fullsized":"2024-09-10T18:59:04.100683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":501,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098191+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T18:59:04.098191+0000","last_peered":"2024-09-10T18:59:04.098191+0000","last_clean":"2024-09-10T18:59:04.098191+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:59:04.098191+0000","last_undegraded":"2024-09-10T18:59:04.098191+0000","last_fullsized":"2024-09-10T18:59:04.098191+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111342+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T18:59:04.111342+0000","last_peered":"2024-09-10T18:59:04.111342+0000","last_clean":"2024-09-10T18:59:04.111342+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:59:04.111342+0000","last_undegraded":"2024-09-10T18:59:04.111342+0000","last_fullsized":"2024-09-10T18:59:04.111342+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":295,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.151683+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:59:04.151683+0000","last_peered":"2024-09-10T18:59:04.151683+0000","last_clean":"2024-09-10T18:59:04.151683+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:59:04.151683+0000","last_undegraded":"2024-09-10T18:59:04.151683+0000","last_fullsized":"2024-09-10T18:59:04.151683+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"77'11","reported_seq":390,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.213170+0000","last_change":"2024-09-10T18:57:33.408816+0000","last_active":"2024-09-10T18:59:03.213170+0000","last_peered":"2024-09-10T18:59:03.213170+0000","last_clean":"2024-09-10T18:59:03.213170+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:59:03.213170+0000","last_undegraded":"2024-09-10T18:59:03.213170+0000","last_fullsized":"2024-09-10T18:59:03.213170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:33.408655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:33.408655+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039294170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":386,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105680+0000","last_change":"2024-09-10T18:58:46.953749+0000","last_active":"2024-09-10T18:59:04.105680+0000","last_peered":"2024-09-10T18:59:04.105680+0000","last_clean":"2024-09-10T18:59:04.105680+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:59:04.105680+0000","last_undegraded":"2024-09-10T18:59:04.105680+0000","last_fullsized":"2024-09-10T18:59:04.105680+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.953608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.953608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.755421+0000","scrub_duration":0.0028191259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":361,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560715+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:59:03.560715+0000","last_peered":"2024-09-10T18:59:03.560715+0000","last_clean":"2024-09-10T18:59:03.560715+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:59:03.560715+0000","last_undegraded":"2024-09-10T18:59:03.560715+0000","last_fullsized":"2024-09-10T18:59:03.560715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":370,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109518+0000","last_change":"2024-09-10T18:57:42.899838+0000","last_active":"2024-09-10T18:59:04.109518+0000","last_peered":"2024-09-10T18:59:04.109518+0000","last_clean":"2024-09-10T18:59:04.109518+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:59:04.109518+0000","last_undegraded":"2024-09-10T18:59:04.109518+0000","last_fullsized":"2024-09-10T18:59:04.109518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.899684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.899684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:10.010497+0000","scrub_duration":0.0036750730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":293,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100816+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T18:59:04.100816+0000","last_peered":"2024-09-10T18:59:04.100816+0000","last_clean":"2024-09-10T18:59:04.100816+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:59:04.100816+0000","last_undegraded":"2024-09-10T18:59:04.100816+0000","last_fullsized":"2024-09-10T18:59:04.100816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.954334+0000","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"113'6","reported_seq":381,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.151827+0000","last_change":"2024-09-10T18:58:40.428324+0000","last_active":"2024-09-10T18:59:04.151827+0000","last_peered":"2024-09-10T18:59:04.151827+0000","last_clean":"2024-09-10T18:59:04.151827+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:59:04.151827+0000","last_undegraded":"2024-09-10T18:59:04.151827+0000","last_fullsized":"2024-09-10T18:59:04.151827+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:58:40.428157+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.428157+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.549402+0000","scrub_duration":0.003853726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152362+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T18:59:04.152362+0000","last_peered":"2024-09-10T18:59:04.152362+0000","last_clean":"2024-09-10T18:59:04.152362+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:59:04.152362+0000","last_undegraded":"2024-09-10T18:59:04.152362+0000","last_fullsized":"2024-09-10T18:59:04.152362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":364,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.116888+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T18:59:04.116888+0000","last_peered":"2024-09-10T18:59:04.116888+0000","last_clean":"2024-09-10T18:59:04.116888+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:59:04.116888+0000","last_undegraded":"2024-09-10T18:59:04.116888+0000","last_fullsized":"2024-09-10T18:59:04.116888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.425187+0000","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":367,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.217115+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T18:59:03.217115+0000","last_peered":"2024-09-10T18:59:03.217115+0000","last_clean":"2024-09-10T18:59:03.217115+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:59:03.217115+0000","last_undegraded":"2024-09-10T18:59:03.217115+0000","last_fullsized":"2024-09-10T18:59:03.217115+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:13.562040+0000","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153855+0000","last_change":"2024-09-10T18:59:01.663804+0000","last_active":"2024-09-10T18:59:04.153855+0000","last_peered":"2024-09-10T18:59:04.153855+0000","last_clean":"2024-09-10T18:59:04.153855+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:59:04.153855+0000","last_undegraded":"2024-09-10T18:59:04.153855+0000","last_fullsized":"2024-09-10T18:59:04.153855+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.663643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.663643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.528103+0000","scrub_duration":0.0029754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"108'16","reported_seq":395,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214110+0000","last_change":"2024-09-10T18:58:56.293280+0000","last_active":"2024-09-10T18:59:03.214110+0000","last_peered":"2024-09-10T18:59:03.214110+0000","last_clean":"2024-09-10T18:59:03.214110+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:59:03.214110+0000","last_undegraded":"2024-09-10T18:59:03.214110+0000","last_fullsized":"2024-09-10T18:59:03.214110+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:58:56.293027+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.293027+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.342426+0000","scrub_duration":0.0069050090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":366,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133249+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:59:04.133249+0000","last_peered":"2024-09-10T18:59:04.133249+0000","last_clean":"2024-09-10T18:59:04.133249+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:59:04.133249+0000","last_undegraded":"2024-09-10T18:59:04.133249+0000","last_fullsized":"2024-09-10T18:59:04.133249+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":494,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111996+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T18:59:04.111996+0000","last_peered":"2024-09-10T18:59:04.111996+0000","last_clean":"2024-09-10T18:59:04.111996+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:59:04.111996+0000","last_undegraded":"2024-09-10T18:59:04.111996+0000","last_fullsized":"2024-09-10T18:59:04.111996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":359,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559640+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:59:03.559640+0000","last_peered":"2024-09-10T18:59:03.559640+0000","last_clean":"2024-09-10T18:59:03.559640+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:59:03.559640+0000","last_undegraded":"2024-09-10T18:59:03.559640+0000","last_fullsized":"2024-09-10T18:59:03.559640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560128+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:59:03.560128+0000","last_peered":"2024-09-10T18:59:03.560128+0000","last_clean":"2024-09-10T18:59:03.560128+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:59:03.560128+0000","last_undegraded":"2024-09-10T18:59:03.560128+0000","last_fullsized":"2024-09-10T18:59:03.560128+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":379,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559288+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:59:03.559288+0000","last_peered":"2024-09-10T18:59:03.559288+0000","last_clean":"2024-09-10T18:59:03.559288+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:59:03.559288+0000","last_undegraded":"2024-09-10T18:59:03.559288+0000","last_fullsized":"2024-09-10T18:59:03.559288+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":375,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560856+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T18:59:03.560856+0000","last_peered":"2024-09-10T18:59:03.560856+0000","last_clean":"2024-09-10T18:59:03.560856+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:59:03.560856+0000","last_undegraded":"2024-09-10T18:59:03.560856+0000","last_fullsized":"2024-09-10T18:59:03.560856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:19.026489+0000","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":486,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.133392+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T18:59:04.133392+0000","last_peered":"2024-09-10T18:59:04.133392+0000","last_clean":"2024-09-10T18:59:04.133392+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:59:04.133392+0000","last_undegraded":"2024-09-10T18:59:04.133392+0000","last_fullsized":"2024-09-10T18:59:04.133392+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.132106+0000","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":372,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.100215+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T18:59:04.100215+0000","last_peered":"2024-09-10T18:59:04.100215+0000","last_clean":"2024-09-10T18:59:04.100215+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:59:04.100215+0000","last_undegraded":"2024-09-10T18:59:04.100215+0000","last_fullsized":"2024-09-10T18:59:04.100215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560149+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T18:59:03.560149+0000","last_peered":"2024-09-10T18:59:03.560149+0000","last_clean":"2024-09-10T18:59:03.560149+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:59:03.560149+0000","last_undegraded":"2024-09-10T18:59:03.560149+0000","last_fullsized":"2024-09-10T18:59:03.560149+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.987912+0000","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"77'9","reported_seq":387,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214699+0000","last_change":"2024-09-10T18:58:38.152259+0000","last_active":"2024-09-10T18:59:03.214699+0000","last_peered":"2024-09-10T18:59:03.214699+0000","last_clean":"2024-09-10T18:59:03.214699+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:59:03.214699+0000","last_undegraded":"2024-09-10T18:59:03.214699+0000","last_fullsized":"2024-09-10T18:59:03.214699+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.152079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.152079+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.945644+0000","scrub_duration":0.0040090350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152816+0000","last_change":"2024-09-10T18:58:55.639930+0000","last_active":"2024-09-10T18:59:04.152816+0000","last_peered":"2024-09-10T18:59:04.152816+0000","last_clean":"2024-09-10T18:59:04.152816+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:59:04.152816+0000","last_undegraded":"2024-09-10T18:59:04.152816+0000","last_fullsized":"2024-09-10T18:59:04.152816+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.639777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.639777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.180410+0000","scrub_duration":0.003186637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":360,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132337+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T18:59:04.132337+0000","last_peered":"2024-09-10T18:59:04.132337+0000","last_clean":"2024-09-10T18:59:04.132337+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:59:04.132337+0000","last_undegraded":"2024-09-10T18:59:04.132337+0000","last_fullsized":"2024-09-10T18:59:04.132337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:15.840825+0000","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":362,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.558336+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:59:03.558336+0000","last_peered":"2024-09-10T18:59:03.558336+0000","last_clean":"2024-09-10T18:59:03.558336+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:59:03.558336+0000","last_undegraded":"2024-09-10T18:59:03.558336+0000","last_fullsized":"2024-09-10T18:59:03.558336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":293,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098069+0000","last_change":"2024-09-10T18:58:46.501591+0000","last_active":"2024-09-10T18:59:04.098069+0000","last_peered":"2024-09-10T18:59:04.098069+0000","last_clean":"2024-09-10T18:59:04.098069+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:59:04.098069+0000","last_undegraded":"2024-09-10T18:59:04.098069+0000","last_fullsized":"2024-09-10T18:59:04.098069+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.501508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.732556+0000","scrub_duration":0.0036484099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"115'15","reported_seq":396,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098252+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T18:59:04.098252+0000","last_peered":"2024-09-10T18:59:04.098252+0000","last_clean":"2024-09-10T18:59:04.098252+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:59:04.098252+0000","last_undegraded":"2024-09-10T18:59:04.098252+0000","last_fullsized":"2024-09-10T18:59:04.098252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":385,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.123056+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T18:59:04.123056+0000","last_peered":"2024-09-10T18:59:04.123056+0000","last_clean":"2024-09-10T18:59:04.123056+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:59:04.123056+0000","last_undegraded":"2024-09-10T18:59:04.123056+0000","last_fullsized":"2024-09-10T18:59:04.123056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:17.284809+0000","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":374,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.099193+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T18:59:04.099193+0000","last_peered":"2024-09-10T18:59:04.099193+0000","last_clean":"2024-09-10T18:59:04.099193+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:59:04.099193+0000","last_undegraded":"2024-09-10T18:59:04.099193+0000","last_fullsized":"2024-09-10T18:59:04.099193+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.811604+0000","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":365,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.108733+0000","last_change":"2024-09-10T18:58:57.260069+0000","last_active":"2024-09-10T18:59:04.108733+0000","last_peered":"2024-09-10T18:59:04.108733+0000","last_clean":"2024-09-10T18:59:04.108733+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:59:04.108733+0000","last_undegraded":"2024-09-10T18:59:04.108733+0000","last_fullsized":"2024-09-10T18:59:04.108733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":293,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.097869+0000","last_change":"2024-09-10T18:58:49.517454+0000","last_active":"2024-09-10T18:59:04.097869+0000","last_peered":"2024-09-10T18:59:04.097869+0000","last_clean":"2024-09-10T18:59:04.097869+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:59:04.097869+0000","last_undegraded":"2024-09-10T18:59:04.097869+0000","last_fullsized":"2024-09-10T18:59:04.097869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.517273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.517273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.037358+0000","scrub_duration":0.0039870039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"108'9","reported_seq":375,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560444+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T18:59:03.560444+0000","last_peered":"2024-09-10T18:59:03.560444+0000","last_clean":"2024-09-10T18:59:03.560444+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:59:03.560444+0000","last_undegraded":"2024-09-10T18:59:03.560444+0000","last_fullsized":"2024-09-10T18:59:03.560444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:12.849844+0000","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":373,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134281+0000","last_change":"2024-09-10T18:58:53.461054+0000","last_active":"2024-09-10T18:59:04.134281+0000","last_peered":"2024-09-10T18:59:04.134281+0000","last_clean":"2024-09-10T18:59:04.134281+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:59:04.134281+0000","last_undegraded":"2024-09-10T18:59:04.134281+0000","last_fullsized":"2024-09-10T18:59:04.134281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.460812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.460812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:07.134909+0000","scrub_duration":0.0042777029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":357,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.105617+0000","last_change":"2024-09-10T18:58:51.198860+0000","last_active":"2024-09-10T18:59:04.105617+0000","last_peered":"2024-09-10T18:59:04.105617+0000","last_clean":"2024-09-10T18:59:04.105617+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:59:04.105617+0000","last_undegraded":"2024-09-10T18:59:04.105617+0000","last_fullsized":"2024-09-10T18:59:04.105617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":367,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.212776+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T18:59:03.212776+0000","last_peered":"2024-09-10T18:59:03.212776+0000","last_clean":"2024-09-10T18:59:03.212776+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:59:03.212776+0000","last_undegraded":"2024-09-10T18:59:03.212776+0000","last_fullsized":"2024-09-10T18:59:03.212776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":291,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121377+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T18:59:04.121377+0000","last_peered":"2024-09-10T18:59:04.121377+0000","last_clean":"2024-09-10T18:59:04.121377+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:59:04.121377+0000","last_undegraded":"2024-09-10T18:59:04.121377+0000","last_fullsized":"2024-09-10T18:59:04.121377+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.299236+0000","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'9","reported_seq":387,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.505329+0000","last_change":"2024-09-10T18:59:04.505215+0000","last_active":"2024-09-10T18:59:04.505329+0000","last_peered":"2024-09-10T18:59:04.505329+0000","last_clean":"2024-09-10T18:59:04.505329+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:59:04.505329+0000","last_undegraded":"2024-09-10T18:59:04.505329+0000","last_fullsized":"2024-09-10T18:59:04.505329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:59:04.505046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.505046+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.388122+0000","scrub_duration":0.0045725100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":375,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.154005+0000","last_change":"2024-09-10T18:57:56.351163+0000","last_active":"2024-09-10T18:59:04.154005+0000","last_peered":"2024-09-10T18:59:04.154005+0000","last_clean":"2024-09-10T18:59:04.154005+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:59:04.154005+0000","last_undegraded":"2024-09-10T18:59:04.154005+0000","last_fullsized":"2024-09-10T18:59:04.154005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.351005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.351005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035909499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":364,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109104+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T18:59:04.109104+0000","last_peered":"2024-09-10T18:59:04.109104+0000","last_clean":"2024-09-10T18:59:04.109104+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:59:04.109104+0000","last_undegraded":"2024-09-10T18:59:04.109104+0000","last_fullsized":"2024-09-10T18:59:04.109104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.764329+0000","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111442+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:59:04.111442+0000","last_peered":"2024-09-10T18:59:04.111442+0000","last_clean":"2024-09-10T18:59:04.111442+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:59:04.111442+0000","last_undegraded":"2024-09-10T18:59:04.111442+0000","last_fullsized":"2024-09-10T18:59:04.111442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":280,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559015+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T18:59:03.559015+0000","last_peered":"2024-09-10T18:59:03.559015+0000","last_clean":"2024-09-10T18:59:03.559015+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:59:03.559015+0000","last_undegraded":"2024-09-10T18:59:03.559015+0000","last_fullsized":"2024-09-10T18:59:03.559015+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":371,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.214103+0000","last_change":"2024-09-10T18:58:58.248456+0000","last_active":"2024-09-10T18:59:03.214103+0000","last_peered":"2024-09-10T18:59:03.214103+0000","last_clean":"2024-09-10T18:59:03.214103+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:59:03.214103+0000","last_undegraded":"2024-09-10T18:59:03.214103+0000","last_fullsized":"2024-09-10T18:59:03.214103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.067159+0000","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"383'22","reported_seq":415,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.121350+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T18:59:04.121350+0000","last_peered":"2024-09-10T18:59:04.121350+0000","last_clean":"2024-09-10T18:59:04.121350+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:59:04.121350+0000","last_undegraded":"2024-09-10T18:59:04.121350+0000","last_fullsized":"2024-09-10T18:59:04.121350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.312723+0000","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":373,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.132040+0000","last_change":"2024-09-10T18:58:31.395234+0000","last_active":"2024-09-10T18:59:04.132040+0000","last_peered":"2024-09-10T18:59:04.132040+0000","last_clean":"2024-09-10T18:59:04.132040+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:59:04.132040+0000","last_undegraded":"2024-09-10T18:59:04.132040+0000","last_fullsized":"2024-09-10T18:59:04.132040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.395072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.395072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:33.507413+0000","scrub_duration":0.0032990430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":361,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.559869+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:59:03.559869+0000","last_peered":"2024-09-10T18:59:03.559869+0000","last_clean":"2024-09-10T18:59:03.559869+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:59:03.559869+0000","last_undegraded":"2024-09-10T18:59:03.559869+0000","last_fullsized":"2024-09-10T18:59:03.559869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":282,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153304+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:59:04.153304+0000","last_peered":"2024-09-10T18:59:04.153304+0000","last_clean":"2024-09-10T18:59:04.153304+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:59:04.153304+0000","last_undegraded":"2024-09-10T18:59:04.153304+0000","last_fullsized":"2024-09-10T18:59:04.153304+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":369,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.122585+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T18:59:04.122585+0000","last_peered":"2024-09-10T18:59:04.122585+0000","last_clean":"2024-09-10T18:59:04.122585+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:59:04.122585+0000","last_undegraded":"2024-09-10T18:59:04.122585+0000","last_fullsized":"2024-09-10T18:59:04.122585+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"77'9","reported_seq":382,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.106183+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T18:59:04.106183+0000","last_peered":"2024-09-10T18:59:04.106183+0000","last_clean":"2024-09-10T18:59:04.106183+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:59:04.106183+0000","last_undegraded":"2024-09-10T18:59:04.106183+0000","last_fullsized":"2024-09-10T18:59:04.106183+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:53.283437+0000","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":368,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.110027+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T18:59:04.110027+0000","last_peered":"2024-09-10T18:59:04.110027+0000","last_clean":"2024-09-10T18:59:04.110027+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:59:04.110027+0000","last_undegraded":"2024-09-10T18:59:04.110027+0000","last_fullsized":"2024-09-10T18:59:04.110027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:06.346480+0000","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":515,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.154012+0000","last_change":"2024-09-10T18:58:38.434610+0000","last_active":"2024-09-10T18:59:04.154012+0000","last_peered":"2024-09-10T18:59:04.154012+0000","last_clean":"2024-09-10T18:59:04.154012+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:59:04.154012+0000","last_undegraded":"2024-09-10T18:59:04.154012+0000","last_fullsized":"2024-09-10T18:59:04.154012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:58:38.434348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.434348+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:43.611964+0000","scrub_duration":0.0046950070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153928+0000","last_change":"2024-09-10T18:59:00.644451+0000","last_active":"2024-09-10T18:59:04.153928+0000","last_peered":"2024-09-10T18:59:04.153928+0000","last_clean":"2024-09-10T18:59:04.153928+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:59:04.153928+0000","last_undegraded":"2024-09-10T18:59:04.153928+0000","last_fullsized":"2024-09-10T18:59:04.153928+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.644299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.665454+0000","scrub_duration":0.00344357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":372,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.099854+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T18:59:04.099854+0000","last_peered":"2024-09-10T18:59:04.099854+0000","last_clean":"2024-09-10T18:59:04.099854+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:59:04.099854+0000","last_undegraded":"2024-09-10T18:59:04.099854+0000","last_fullsized":"2024-09-10T18:59:04.099854+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":384,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.153269+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T18:59:04.153269+0000","last_peered":"2024-09-10T18:59:04.153269+0000","last_clean":"2024-09-10T18:59:04.153269+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:59:04.153269+0000","last_undegraded":"2024-09-10T18:59:04.153269+0000","last_fullsized":"2024-09-10T18:59:04.153269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.205170+0000","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"77'5","reported_seq":373,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-10T18:59:03.560486+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:59:03.560486+0000","last_peered":"2024-09-10T18:59:03.560486+0000","last_clean":"2024-09-10T18:59:03.560486+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:59:03.560486+0000","last_undegraded":"2024-09-10T18:59:03.560486+0000","last_fullsized":"2024-09-10T18:59:03.560486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":382,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.109817+0000","last_change":"2024-09-10T18:58:32.931801+0000","last_active":"2024-09-10T18:59:04.109817+0000","last_peered":"2024-09-10T18:59:04.109817+0000","last_clean":"2024-09-10T18:59:04.109817+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:59:04.109817+0000","last_undegraded":"2024-09-10T18:59:04.109817+0000","last_fullsized":"2024-09-10T18:59:04.109817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.931700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.931700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:37.655765+0000","scrub_duration":0.002489907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":280,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.134857+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:59:04.134857+0000","last_peered":"2024-09-10T18:59:04.134857+0000","last_clean":"2024-09-10T18:59:04.134857+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:59:04.134857+0000","last_undegraded":"2024-09-10T18:59:04.134857+0000","last_fullsized":"2024-09-10T18:59:04.134857+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":362,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.116759+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:59:04.116759+0000","last_peered":"2024-09-10T18:59:04.116759+0000","last_clean":"2024-09-10T18:59:04.116759+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:59:04.116759+0000","last_undegraded":"2024-09-10T18:59:04.116759+0000","last_fullsized":"2024-09-10T18:59:04.116759+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":368,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111906+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:59:04.111906+0000","last_peered":"2024-09-10T18:59:04.111906+0000","last_clean":"2024-09-10T18:59:04.111906+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:59:04.111906+0000","last_undegraded":"2024-09-10T18:59:04.111906+0000","last_fullsized":"2024-09-10T18:59:04.111906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"77'9","reported_seq":392,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.098153+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T18:59:04.098153+0000","last_peered":"2024-09-10T18:59:04.098153+0000","last_clean":"2024-09-10T18:59:04.098153+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:59:04.098153+0000","last_undegraded":"2024-09-10T18:59:04.098153+0000","last_fullsized":"2024-09-10T18:59:04.098153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":364,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.111932+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T18:59:04.111932+0000","last_peered":"2024-09-10T18:59:04.111932+0000","last_clean":"2024-09-10T18:59:04.111932+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:59:04.111932+0000","last_undegraded":"2024-09-10T18:59:04.111932+0000","last_fullsized":"2024-09-10T18:59:04.111932+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.078660+0000","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":292,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-10T18:59:04.152891+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T18:59:04.152891+0000","last_peered":"2024-09-10T18:59:04.152891+0000","last_clean":"2024-09-10T18:59:04.152891+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:59:04.152891+0000","last_undegraded":"2024-09-10T18:59:04.152891+0000","last_fullsized":"2024-09-10T18:59:04.152891+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.897723+0000","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":205,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195684,"num_objects":144,"num_object_clones":2,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":102,"num_read_kb":6,"num_write":571,"num_write_kb":12662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26320896,"data_stored":24587052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1829,"ondisk_log_size":1829,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":562,"num_read_kb":557,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":479,"ondisk_log_size":479,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972481,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311980,"kb_used_data":14620,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93433172,"statfs":{"total":95995035648,"available":95675568128,"internally_reserved":0,"allocated":14970880,"data_stored":12220983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.85799999999999998,"15min":0.91200000000000003},"min":{"1min":0.52200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.2929999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.91200000000000003,"15min":0.92900000000000005},"min":{"1min":0.498,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3839999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.018}]},{"osd":1,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86099999999999999,"15min":0.89600000000000002},"min":{"1min":0.52400000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1870000000000001,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.96399999999999997,"15min":1.0129999999999999},"min":{"1min":0.54400000000000004,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.4119999999999999,"5min":1.742,"15min":1.8280000000000001},"last":0.84499999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.61099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.365,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.004,"5min":0.999,"15min":0.98899999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.355,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.042,"15min":1.0189999999999999},"min":{"1min":0.65400000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.323,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":1.115},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.98899999999999999,"15min":0.97699999999999998},"min":{"1min":0.628,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.256,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.875}]},{"osd":4,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0469999999999999,"15min":1.0189999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4279999999999999,"5min":1.556,"15min":2.1989999999999998},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.125,"15min":1.0589999999999999},"min":{"1min":0.61799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4890000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":0.81399999999999995}]},{"osd":5,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.012,"15min":0.997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.161,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.099,"5min":1.159,"15min":1.171},"min":{"1min":0.51000000000000001,"5min":0.505,"15min":0.505},"max":{"1min":1.49,"5min":1.5329999999999999,"15min":2.1419999999999999},"last":1.216}]},{"osd":6,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2030000000000001,"15min":1.2230000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5680000000000001,"5min":1.591,"15min":2.2749999999999999},"last":1.046},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.204,"15min":1.194},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.6910000000000001,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":0.90200000000000002}]}]},{"osd":6,"up_from":52,"seq":223338299531,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308052,"kb_used_data":6532,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437100,"statfs":{"total":95995035648,"available":95679590400,"internally_reserved":0,"allocated":6688768,"data_stored":4024756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.87,"15min":0.88600000000000001},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.8839999999999999,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.006,"15min":0.95699999999999996},"min":{"1min":0.57699999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.7309999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.0780000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.212,"15min":1.0149999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.2410000000000001,"5min":18.494,"15min":18.494},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.022,"5min":0.90800000000000003,"15min":0.90900000000000003},"min":{"1min":0.70199999999999996,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.7689999999999999,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.97599999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0289999999999999,"15min":0.95899999999999996},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3440000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.244,"5min":0.97799999999999998,"15min":0.94499999999999995},"min":{"1min":0.65700000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.3320000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.05,"15min":0.99299999999999999},"min":{"1min":0.64600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1949999999999998,"5min":2.278,"15min":2.278},"last":1.256},{"interface":"front","average":{"1min":1.087,"5min":1.014,"15min":0.98199999999999998},"min":{"1min":0.78700000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.7090000000000001,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.2849999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.044,"15min":0.93400000000000005},"min":{"1min":0.67600000000000005,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":6.008,"5min":6.008,"15min":6.008},"last":1.548},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.135,"15min":1.054},"min":{"1min":0.745,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.3039999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.4430000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.115,"15min":1.028},"min":{"1min":0.82599999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":5.9560000000000004,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.623},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.194,"15min":1.125},"min":{"1min":0.74299999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":3.5169999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.532}]},{"osd":7,"last update":"Tue Sep 10 18:58:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.103,"15min":1.0649999999999999},"min":{"1min":0.76000000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.3520000000000001,"5min":2.069,"15min":2.069},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.264,"15min":1.1859999999999999},"min":{"1min":0.79100000000000004,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.758}]}]},{"osd":5,"up_from":43,"seq":184683593877,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307500,"kb_used_data":10140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93437652,"statfs":{"total":95995035648,"available":95680155648,"internally_reserved":0,"allocated":10383360,"data_stored":7628992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.093,"15min":0.94399999999999995},"min":{"1min":0.59799999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.1539999999999999,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.04},{"interface":"front","average":{"1min":0.878,"5min":0.998,"15min":0.91400000000000003},"min":{"1min":0.621,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.212,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.71899999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.6679999999999999,"15min":1.1339999999999999},"min":{"1min":0.63800000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.3140000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.421,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.1659999999999999,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.135}]},{"osd":2,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.8999999999999999,"15min":1.2370000000000001},"min":{"1min":0.51000000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.752,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1,"5min":1.135,"15min":0.98999999999999999},"min":{"1min":0.64000000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":2.2450000000000001,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.95999999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1319999999999999,"15min":0.998},"min":{"1min":0.56299999999999994,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.8520000000000001,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.232},{"interface":"front","average":{"1min":1.177,"5min":1.798,"15min":1.236},"min":{"1min":0.83999999999999997,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":2.3820000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.105}]},{"osd":4,"last update":"Tue Sep 10 18:58:14 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.1830000000000001,"15min":0.92900000000000005},"min":{"1min":0.68300000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6519999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2250000000000001,"15min":0.998},"min":{"1min":0.77200000000000002,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":4.1970000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.0700000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.399,"15min":1.149},"min":{"1min":0.68400000000000005,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":5.2009999999999996,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.4350000000000001,"15min":1.1830000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.47199999999999998},"max":{"1min":5.1769999999999996,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.206}]},{"osd":7,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.5529999999999999,"15min":1.22},"min":{"1min":0.81200000000000006,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.7210000000000001,"5min":15.613,"15min":15.613},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.282,"5min":1.54,"15min":1.2809999999999999},"min":{"1min":0.75700000000000001,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":4.2480000000000002,"5min":16.965,"15min":16.965},"last":0.91800000000000004}]}]},{"osd":4,"up_from":35,"seq":150323855521,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311684,"kb_used_data":5988,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93433468,"statfs":{"total":95995035648,"available":95675871232,"internally_reserved":0,"allocated":6131712,"data_stored":3444001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.79400000000000004,"15min":0.83999999999999997},"min":{"1min":0.54400000000000004,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0469999999999999,"5min":1.546,"15min":1.546},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.58899999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.258,"5min":1.667,"15min":1.7070000000000001},"last":1.2390000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.85299999999999998,"15min":0.874},"min":{"1min":0.53700000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.2,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.026},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.90800000000000003,"15min":0.89100000000000001},"min":{"1min":0.60699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.1919999999999999,"5min":1.575,"15min":1.575},"last":0.84999999999999998}]},{"osd":2,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.93000000000000005,"15min":0.91500000000000004},"min":{"1min":0.54600000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.2350000000000001,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.119},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.93400000000000005,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.497,"15min":0.497},"max":{"1min":1.365,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":1.089}]},{"osd":3,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.65200000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1339999999999999,"5min":1.5489999999999999,"15min":1.714},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.95399999999999996,"15min":0.94499999999999995},"min":{"1min":0.67000000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.137,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.147}]},{"osd":5,"last update":"Tue Sep 10 18:58:10 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.97499999999999998,"15min":0.89900000000000002},"min":{"1min":0.58099999999999996,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.262,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.038,"15min":0.95999999999999996},"min":{"1min":0.61099999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.377,"5min":1.4610000000000001,"15min":1.7909999999999999},"last":1.1679999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.038,"15min":0.98499999999999999},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.196},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.127,"15min":1.1020000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.2150000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1459999999999999,"15min":1.159},"min":{"1min":0.73899999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.629,"5min":1.629,"15min":2.008},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1419999999999999,"15min":1.1579999999999999},"min":{"1min":0.58499999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5489999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.327}]}]},{"osd":3,"up_from":29,"seq":124554051756,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303472,"kb_used_data":6104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93441680,"statfs":{"total":95995035648,"available":95684280320,"internally_reserved":0,"allocated":6250496,"data_stored":3443951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.222,"15min":0.82399999999999995},"min":{"1min":0.36099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":1.026,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.75,"5min":1.242,"15min":0.89400000000000002},"min":{"1min":0.38800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":1.5700000000000001,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.48299999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":1.3839999999999999,"15min":0.91700000000000004},"min":{"1min":0.48099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.802,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.0449999999999999,"15min":0.86499999999999999},"min":{"1min":0.39900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.075,"5min":15.189,"15min":15.189},"last":0.71299999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:58:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":1.355,"15min":0.99099999999999999},"min":{"1min":0.41899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1599999999999999,"5min":26.558,"15min":26.558},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.25,"15min":0.92900000000000005},"min":{"1min":0.378,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.6760000000000002,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.59799999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:58:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.335,"15min":1.1279999999999999},"min":{"1min":0.74199999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":3.2469999999999999,"5min":10.359999999999999,"15min":10.359999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.274,"5min":1.4330000000000001,"15min":1.2150000000000001},"min":{"1min":0.67000000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":3.2850000000000001,"5min":10.391,"15min":10.391},"last":1.069}]},{"osd":5,"last update":"Tue Sep 10 18:58:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.4279999999999999,"15min":1.212},"min":{"1min":0.73799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.355,"5min":17.535,"15min":17.535},"last":0.874},{"interface":"front","average":{"1min":1.042,"5min":1.105,"15min":1.0329999999999999},"min":{"1min":0.60399999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":2.04,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.66800000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.3,"15min":1.2030000000000001},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6919999999999999,"5min":10.51,"15min":10.51},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1399999999999999,"15min":1.0900000000000001},"min":{"1min":0.63,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5820000000000001,"5min":10.773999999999999,"15min":10.773999999999999},"last":0.74199999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.331,"15min":1.177},"min":{"1min":0.621,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7270000000000001,"5min":18.312999999999999,"15min":18.312999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.4319999999999999,"15min":1.3},"min":{"1min":0.70099999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.575,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.129}]}]},{"osd":2,"up_from":19,"seq":81604378805,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319732,"kb_used_data":14044,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93425420,"statfs":{"total":95995035648,"available":95667630080,"internally_reserved":0,"allocated":14381056,"data_stored":11630020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.83699999999999997,"15min":0.71599999999999997},"min":{"1min":0.35799999999999998,"5min":0.219,"15min":0.219},"max":{"1min":1.2,"5min":9.516,"15min":9.516},"last":0.442},{"interface":"front","average":{"1min":0.75,"5min":0.81799999999999995,"15min":0.76900000000000002},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0600000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.36099999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.75600000000000001,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1339999999999999,"5min":5.516,"15min":5.516},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.83599999999999997,"15min":0.80400000000000005},"min":{"1min":0.41599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.048,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.876,"15min":0.85099999999999998},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0349999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.503},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.91000000000000003,"15min":0.84699999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.115,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:58:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.151,"15min":1.0449999999999999},"min":{"1min":0.629,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.835,"5min":21.571999999999999,"15min":21.571999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.087,"5min":1.1459999999999999,"15min":1.107},"min":{"1min":0.76100000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.8560000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.88700000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:58:07 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0860000000000001,"15min":1.101},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.458,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.198,"15min":1.0860000000000001},"min":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.321,"5min":21.654,"15min":21.654},"last":0.755}]},{"osd":6,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":0.64400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.5609999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2370000000000001,"15min":1.1299999999999999},"min":{"1min":0.63,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":15.971,"15min":15.971},"last":1.052}]},{"osd":7,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.59699999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.905,"5min":15.025,"15min":15.025},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.331,"15min":1.1990000000000001},"min":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":6.851,"5min":14.816000000000001,"15min":14.816000000000001},"last":0.98099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311448,"kb_used_data":9912,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433704,"statfs":{"total":95995035648,"available":95676112896,"internally_reserved":0,"allocated":10149888,"data_stored":7434870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.0409999999999999,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.877,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.135,"15min":0.85899999999999999},"min":{"1min":0.41399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.7200000000000002,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.44}]},{"osd":2,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.141,"15min":0.84699999999999998},"min":{"1min":0.27200000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.048,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3480000000000001,"15min":0.97899999999999998},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.0890000000000004,"5min":32.363,"15min":32.363},"last":0.39400000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.23,"15min":0.97399999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.1469999999999998,"5min":25.704000000000001,"15min":25.704000000000001},"last":0.496},{"interface":"front","average":{"1min":1.2,"5min":1.5580000000000001,"15min":1.02},"min":{"1min":0.39500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.0099999999999998,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.55200000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:58:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.234,"15min":1.0900000000000001},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.762,"5min":12.808999999999999,"15min":12.808999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.748,"15min":1.202},"min":{"1min":0.56000000000000005,"5min":0.378,"15min":0.378},"max":{"1min":6.181,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.84699999999999998}]},{"osd":5,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.284,"15min":1.1020000000000001},"min":{"1min":0.61899999999999999,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":1.4790000000000001,"5min":26.050999999999998,"15min":26.050999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.3180000000000001,"15min":1.1970000000000001},"min":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7949999999999999,"5min":12.84,"15min":12.84},"last":0.83399999999999996}]},{"osd":6,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.5409999999999999,"15min":1.254},"min":{"1min":0.67900000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":3.121,"5min":25.608000000000001,"15min":25.608000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.123,"5min":1.5369999999999999,"15min":1.23},"min":{"1min":0.79400000000000004,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.9359999999999999,"5min":33.643000000000001,"15min":33.643000000000001},"last":0.89300000000000002}]},{"osd":7,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.5229999999999999,"15min":1.3180000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0880000000000001,"5min":25.565999999999999,"15min":25.565999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.607,"15min":1.304},"min":{"1min":0.66100000000000003,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.839,"5min":33.673999999999999,"15min":33.673999999999999},"last":0.96599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303932,"kb_used_data":6572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93441220,"statfs":{"total":95995035648,"available":95683809280,"internally_reserved":0,"allocated":6729728,"data_stored":4024734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.649,"15min":0.96199999999999997},"min":{"1min":0.45700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.028,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.746,"15min":1.0620000000000001},"min":{"1min":0.48099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6850000000000001,"5min":86.009,"15min":86.009},"last":0.71999999999999997}]},{"osd":2,"last update":"Tue Sep 10 18:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.6299999999999999,"15min":1.0109999999999999},"min":{"1min":0.54400000000000004,"5min":0.216,"15min":0.216},"max":{"1min":3.427,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1970000000000001,"15min":0.89500000000000002},"min":{"1min":0.61899999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.105,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.81299999999999994}]},{"osd":3,"last update":"Tue Sep 10 18:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.095,"15min":0.88},"min":{"1min":0.53800000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.246,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.4730000000000001,"15min":1.0269999999999999},"min":{"1min":0.51300000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.012,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.58399999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:58:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.143,"15min":1.0349999999999999},"min":{"1min":0.70099999999999996,"5min":0.5,"15min":0.5},"max":{"1min":3.3279999999999998,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.266,"5min":1.1579999999999999,"15min":1.093},"min":{"1min":0.85499999999999998,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":3.419,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.0049999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2629999999999999,"15min":1.111},"min":{"1min":0.749,"5min":0.442,"15min":0.442},"max":{"1min":1.288,"5min":10.048,"15min":10.048},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.087,"15min":1.097},"min":{"1min":0.63,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.337,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":1.0820000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:59:00 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1599999999999999,"15min":1.1319999999999999},"min":{"1min":0.75900000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5580000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.3240000000000001,"15min":1.1930000000000001},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5740000000000001,"5min":11.894,"15min":11.894},"last":1.1739999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:58:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1890000000000001,"15min":1.1699999999999999},"min":{"1min":0.90900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6899999999999999,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.3200000000000001,"15min":1.2949999999999999},"min":{"1min":0.84499999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.7310000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4210688,"data_stored":4000524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8437760,"data_stored":8194904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4448256,"data_stored":4194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8421376,"data_stored":8194856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:59:10.478 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph status --format=json 2024-09-10T18:59:10.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:10.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:10 smithi165 bash[22591]: audit 2024-09-10T18:59:09.130163+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:10 smithi165 bash[22591]: audit 2024-09-10T18:59:09.130835+0000 mon.a (mon.0) 2438 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi083-77648-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:10.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:10 smithi165 bash[22591]: audit 2024-09-10T18:59:09.130909+0000 mon.a (mon.0) 2439 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:10.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:10 smithi165 bash[22591]: cluster 2024-09-10T18:59:09.131017+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e393: 8 total, 8 up, 8 in 2024-09-10T18:59:10.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:10 smithi165 bash[22591]: audit 2024-09-10T18:59:09.133878+0000 mon.a (mon.0) 2441 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]: dispatch 2024-09-10T18:59:10.597 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:10.701 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:10.736 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:10.839 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:10.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:10.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:10.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.010 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:11.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:11.150 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:11.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[22591]: cluster 2024-09-10T18:59:10.061396+0000 mgr.y (mgr.24526) 402 : cluster [DBG] pgmap v574: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:11.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[22591]: cluster 2024-09-10T18:59:10.126411+0000 mon.a (mon.0) 2442 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:11.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[22591]: cluster 2024-09-10T18:59:10.128106+0000 mon.a (mon.0) 2443 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:11.265 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[22591]: audit 2024-09-10T18:59:10.144900+0000 mon.a (mon.0) 2444 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]': finished 2024-09-10T18:59:11.266 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[22591]: cluster 2024-09-10T18:59:10.145302+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2024-09-10T18:59:11.285 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:11.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:11.402 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[18629]: cluster 2024-09-10T18:59:10.061396+0000 mgr.y (mgr.24526) 402 : cluster [DBG] pgmap v574: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[18629]: cluster 2024-09-10T18:59:10.126411+0000 mon.a (mon.0) 2442 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[18629]: cluster 2024-09-10T18:59:10.128106+0000 mon.a (mon.0) 2443 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[18629]: audit 2024-09-10T18:59:10.144900+0000 mon.a (mon.0) 2444 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]': finished 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[18629]: cluster 2024-09-10T18:59:10.145302+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[32381]: cluster 2024-09-10T18:59:10.061396+0000 mgr.y (mgr.24526) 402 : cluster [DBG] pgmap v574: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[32381]: cluster 2024-09-10T18:59:10.126411+0000 mon.a (mon.0) 2442 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[32381]: cluster 2024-09-10T18:59:10.128106+0000 mon.a (mon.0) 2443 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[32381]: audit 2024-09-10T18:59:10.144900+0000 mon.a (mon.0) 2444 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-49", "mode": "readproxy"}]': finished 2024-09-10T18:59:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:11 smithi083 bash[32381]: cluster 2024-09-10T18:59:10.145302+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e394: 8 total, 8 up, 8 in 2024-09-10T18:59:11.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:11.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:11.541 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:11.564 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:11.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:11.697 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:11.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.834 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:11.880 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T18:59:11.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:11.972 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:11.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:12.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.102 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:12.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:12.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:12.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:12.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:12.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:12.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:59:12.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:12 smithi165 bash[22591]: cluster 2024-09-10T18:59:11.145418+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2024-09-10T18:59:12.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:12 smithi165 bash[22591]: audit 2024-09-10T18:59:11.152038+0000 mon.c (mon.2) 611 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:12 smithi165 bash[22591]: audit 2024-09-10T18:59:11.156966+0000 mon.a (mon.0) 2447 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[18629]: cluster 2024-09-10T18:59:11.145418+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2024-09-10T18:59:12.492 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[18629]: audit 2024-09-10T18:59:11.152038+0000 mon.c (mon.2) 611 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[18629]: audit 2024-09-10T18:59:11.156966+0000 mon.a (mon.0) 2447 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[32381]: cluster 2024-09-10T18:59:11.145418+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e395: 8 total, 8 up, 8 in 2024-09-10T18:59:12.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[32381]: audit 2024-09-10T18:59:11.152038+0000 mon.c (mon.2) 611 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.493 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[32381]: audit 2024-09-10T18:59:11.156966+0000 mon.a (mon.0) 2447 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:12.510 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:12.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:12.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:12.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.762 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:12] "GET /metrics HTTP/1.1" 200 228750 "" "Prometheus/2.43.0" 2024-09-10T18:59:12.779 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:12.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.915 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:12.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:13.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:13.054 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:13.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:13.188 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[18629]: cluster 2024-09-10T18:59:12.061960+0000 mgr.y (mgr.24526) 403 : cluster [DBG] pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[18629]: audit 2024-09-10T18:59:12.166920+0000 mon.a (mon.0) 2448 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:13.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[18629]: cluster 2024-09-10T18:59:12.167032+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2024-09-10T18:59:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[18629]: audit 2024-09-10T18:59:12.170993+0000 mon.c (mon.2) 612 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[18629]: audit 2024-09-10T18:59:12.171875+0000 mon.a (mon.0) 2450 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:13.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[32381]: cluster 2024-09-10T18:59:12.061960+0000 mgr.y (mgr.24526) 403 : cluster [DBG] pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:13.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[32381]: audit 2024-09-10T18:59:12.166920+0000 mon.a (mon.0) 2448 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:13.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[32381]: cluster 2024-09-10T18:59:12.167032+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2024-09-10T18:59:13.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[32381]: audit 2024-09-10T18:59:12.170993+0000 mon.c (mon.2) 612 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:13 smithi083 bash[32381]: audit 2024-09-10T18:59:12.171875+0000 mon.a (mon.0) 2450 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.314 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:13.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:13.422 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T18:59:13.422 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T18:59:13.448 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:13.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:13.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:13 smithi165 bash[22591]: cluster 2024-09-10T18:59:12.061960+0000 mgr.y (mgr.24526) 403 : cluster [DBG] pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:13 smithi165 bash[22591]: audit 2024-09-10T18:59:12.166920+0000 mon.a (mon.0) 2448 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:13 smithi165 bash[22591]: cluster 2024-09-10T18:59:12.167032+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e396: 8 total, 8 up, 8 in 2024-09-10T18:59:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:13 smithi165 bash[22591]: audit 2024-09-10T18:59:12.170993+0000 mon.c (mon.2) 612 : audit [INF] from='client.? v1:172.21.15.83:0/4271809472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:13 smithi165 bash[22591]: audit 2024-09-10T18:59:12.171875+0000 mon.a (mon.0) 2450 : audit [INF] from='client.49325 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]: dispatch 2024-09-10T18:59:13.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:13.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:13.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:13.870 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:13.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:13.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.005 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:14.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.136 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:14.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T18:59:14.266 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:14.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:13.171402+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:14.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: cluster 2024-09-10T18:59:13.171485+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2024-09-10T18:59:14.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:13.204922+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:13.207303+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:13.207849+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:14.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:14.103217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:14.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: cluster 2024-09-10T18:59:14.103330+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2024-09-10T18:59:14.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:14 smithi165 bash[22591]: audit 2024-09-10T18:59:14.105384+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:14.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:14.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.540 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:13.171402+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:14.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: cluster 2024-09-10T18:59:13.171485+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:13.204922+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:13.207303+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:13.207849+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:14.103217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: cluster 2024-09-10T18:59:14.103330+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[18629]: audit 2024-09-10T18:59:14.105384+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:13.171402+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49325 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi083-77648-64"}]': finished 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: cluster 2024-09-10T18:59:13.171485+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e397: 8 total, 8 up, 8 in 2024-09-10T18:59:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:13.204922+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:13.207303+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:13.207849+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:14.103217+0000 mon.a (mon.0) 2456 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi083-77648-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: cluster 2024-09-10T18:59:14.103330+0000 mon.a (mon.0) 2457 : cluster [DBG] osdmap e398: 8 total, 8 up, 8 in 2024-09-10T18:59:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:14 smithi083 bash[32381]: audit 2024-09-10T18:59:14.105384+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:14.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.676 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:14.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:14.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:14.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:15.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:15.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:15.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:15.211 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:15.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:15.348 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:15.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:15.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:15.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:15.621 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:15.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:15.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:15 smithi165 bash[22591]: cluster 2024-09-10T18:59:14.062760+0000 mgr.y (mgr.24526) 404 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:15.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:15 smithi165 bash[22591]: cluster 2024-09-10T18:59:14.263370+0000 osd.3 (osd.3) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:15.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:15 smithi165 bash[22591]: cluster 2024-09-10T18:59:14.263789+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:15.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:15 smithi165 bash[22591]: audit 2024-09-10T18:59:14.569991+0000 mon.a (mon.0) 2459 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:15.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:15 smithi165 bash[22591]: audit 2024-09-10T18:59:14.572827+0000 mon.c (mon.2) 613 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:15.758 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:15.758 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[32381]: cluster 2024-09-10T18:59:14.062760+0000 mgr.y (mgr.24526) 404 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[32381]: cluster 2024-09-10T18:59:14.263370+0000 osd.3 (osd.3) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[32381]: cluster 2024-09-10T18:59:14.263789+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[32381]: audit 2024-09-10T18:59:14.569991+0000 mon.a (mon.0) 2459 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[32381]: audit 2024-09-10T18:59:14.572827+0000 mon.c (mon.2) 613 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[18629]: cluster 2024-09-10T18:59:14.062760+0000 mgr.y (mgr.24526) 404 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T18:59:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[18629]: cluster 2024-09-10T18:59:14.263370+0000 osd.3 (osd.3) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[18629]: cluster 2024-09-10T18:59:14.263789+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[18629]: audit 2024-09-10T18:59:14.569991+0000 mon.a (mon.0) 2459 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:15 smithi083 bash[18629]: audit 2024-09-10T18:59:14.572827+0000 mon.c (mon.2) 613 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:15.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:15.893 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:15.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:15.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:15.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:16.033 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:16.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:16.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:16.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:16.286 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:16.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:16.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:16.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:16.562 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:16.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:16.699 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:16.778 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:16.779 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:16.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:16.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:16 smithi165 bash[22591]: cluster 2024-09-10T18:59:15.549934+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2024-09-10T18:59:16.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:16 smithi165 bash[22591]: cluster 2024-09-10T18:59:16.064903+0000 mgr.y (mgr.24526) 405 : cluster [DBG] pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:16.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:16 smithi165 bash[22591]: audit 2024-09-10T18:59:16.111352+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:16.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:16 smithi165 bash[22591]: cluster 2024-09-10T18:59:16.111460+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2024-09-10T18:59:16.837 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:16.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[18629]: cluster 2024-09-10T18:59:15.549934+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[18629]: cluster 2024-09-10T18:59:16.064903+0000 mgr.y (mgr.24526) 405 : cluster [DBG] pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[18629]: audit 2024-09-10T18:59:16.111352+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[18629]: cluster 2024-09-10T18:59:16.111460+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[32381]: cluster 2024-09-10T18:59:15.549934+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e399: 8 total, 8 up, 8 in 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[32381]: cluster 2024-09-10T18:59:16.064903+0000 mgr.y (mgr.24526) 405 : cluster [DBG] pgmap v583: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[32381]: audit 2024-09-10T18:59:16.111352+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi083-77648-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:16.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:16 smithi083 bash[32381]: cluster 2024-09-10T18:59:16.111460+0000 mon.a (mon.0) 2462 : cluster [DBG] osdmap e400: 8 total, 8 up, 8 in 2024-09-10T18:59:16.974 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:16.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:17.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:17.117 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:17.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:17.256 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:17.359 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:17.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:17.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:17.529 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:17.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:17.664 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:17.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:17.800 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:17.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:17.940 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:17.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:18.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.076 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:18.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:18.191 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:18 smithi083 bash[32381]: cluster 2024-09-10T18:59:17.115033+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2024-09-10T18:59:18.192 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:18 smithi083 bash[18629]: cluster 2024-09-10T18:59:17.115033+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2024-09-10T18:59:18.213 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:18.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.330 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:18 smithi165 bash[22591]: cluster 2024-09-10T18:59:17.115033+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e401: 8 total, 8 up, 8 in 2024-09-10T18:59:18.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:18.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:18.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:18.478 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:18.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:18.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:18.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:18.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:18.809 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:18.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.886 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:18.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:18.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:19.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:19.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:19.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:19.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:19.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:19.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:19.414 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:19 smithi165 bash[22591]: audit 2024-09-10T18:59:17.923828+0000 mgr.y (mgr.24526) 406 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:19.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:19 smithi165 bash[22591]: cluster 2024-09-10T18:59:18.065758+0000 mgr.y (mgr.24526) 407 : cluster [DBG] pgmap v586: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:19.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:19 smithi165 bash[22591]: cluster 2024-09-10T18:59:18.106605+0000 mon.a (mon.0) 2464 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:19.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:19 smithi165 bash[22591]: cluster 2024-09-10T18:59:18.128959+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2024-09-10T18:59:19.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:19 smithi165 bash[22591]: audit 2024-09-10T18:59:18.130098+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:19.433 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:19.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:19.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[32381]: audit 2024-09-10T18:59:17.923828+0000 mgr.y (mgr.24526) 406 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:19.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[32381]: cluster 2024-09-10T18:59:18.065758+0000 mgr.y (mgr.24526) 407 : cluster [DBG] pgmap v586: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:19.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[32381]: cluster 2024-09-10T18:59:18.106605+0000 mon.a (mon.0) 2464 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:19.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[32381]: cluster 2024-09-10T18:59:18.128959+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[32381]: audit 2024-09-10T18:59:18.130098+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[18629]: audit 2024-09-10T18:59:17.923828+0000 mgr.y (mgr.24526) 406 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[18629]: cluster 2024-09-10T18:59:18.065758+0000 mgr.y (mgr.24526) 407 : cluster [DBG] pgmap v586: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[18629]: cluster 2024-09-10T18:59:18.106605+0000 mon.a (mon.0) 2464 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[18629]: cluster 2024-09-10T18:59:18.128959+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e402: 8 total, 8 up, 8 in 2024-09-10T18:59:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:19 smithi083 bash[18629]: audit 2024-09-10T18:59:18.130098+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:19.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:19.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:19.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:19.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:19.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:19.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:19.978 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:20.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:20.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:20.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:20.253 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:20.334 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[32381]: audit 2024-09-10T18:59:19.152572+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:20.334 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[32381]: cluster 2024-09-10T18:59:19.152643+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2024-09-10T18:59:20.334 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[32381]: audit 2024-09-10T18:59:19.157217+0000 mon.a (mon.0) 2469 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:20.334 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[18629]: audit 2024-09-10T18:59:19.152572+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:20.335 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[18629]: cluster 2024-09-10T18:59:19.152643+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2024-09-10T18:59:20.335 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:20 smithi083 bash[18629]: audit 2024-09-10T18:59:19.157217+0000 mon.a (mon.0) 2469 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:20.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:20 smithi165 bash[22591]: audit 2024-09-10T18:59:19.152572+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:20 smithi165 bash[22591]: cluster 2024-09-10T18:59:19.152643+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e403: 8 total, 8 up, 8 in 2024-09-10T18:59:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:20 smithi165 bash[22591]: audit 2024-09-10T18:59:19.157217+0000 mon.a (mon.0) 2469 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]: dispatch 2024-09-10T18:59:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:20.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:20.531 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:20.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:20.667 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:20.771 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:20.809 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:20.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:20.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:20.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:21.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:21.078 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:21.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:21.211 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:21.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:21.349 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: cluster 2024-09-10T18:59:20.066580+0000 mgr.y (mgr.24526) 408 : cluster [DBG] pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.158363+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: cluster 2024-09-10T18:59:20.158645+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.186576+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.188382+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.188899+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.507650+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[22591]: audit 2024-09-10T18:59:20.511683+0000 mon.a (mon.0) 2475 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: cluster 2024-09-10T18:59:20.066580+0000 mgr.y (mgr.24526) 408 : cluster [DBG] pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.158363+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: cluster 2024-09-10T18:59:20.158645+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.186576+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.188382+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.188899+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.507650+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[32381]: audit 2024-09-10T18:59:20.511683+0000 mon.a (mon.0) 2475 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: cluster 2024-09-10T18:59:20.066580+0000 mgr.y (mgr.24526) 408 : cluster [DBG] pgmap v589: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.158363+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? v1:172.21.15.83:0/3243213600' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi083-77648-65"}]': finished 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: cluster 2024-09-10T18:59:20.158645+0000 mon.a (mon.0) 2471 : cluster [DBG] osdmap e404: 8 total, 8 up, 8 in 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.186576+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.188382+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.188899+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.507650+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:21 smithi083 bash[18629]: audit 2024-09-10T18:59:20.511683+0000 mon.a (mon.0) 2475 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:21.484 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:21.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:21.615 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:21.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:21.732 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:21.754 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:21.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:21.892 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:21.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:21.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:22.028 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:22.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:22.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:22.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:22.298 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:22.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:22.432 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:22.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:22.567 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:22.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:22.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: audit 2024-09-10T18:59:21.181886+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: audit 2024-09-10T18:59:21.182981+0000 mon.a (mon.0) 2476 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:22.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: audit 2024-09-10T18:59:21.183065+0000 mon.a (mon.0) 2477 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: cluster 2024-09-10T18:59:21.183134+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: audit 2024-09-10T18:59:21.184317+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18629]: audit 2024-09-10T18:59:21.185484+0000 mon.a (mon.0) 2480 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: audit 2024-09-10T18:59:21.181886+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: audit 2024-09-10T18:59:21.182981+0000 mon.a (mon.0) 2476 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: audit 2024-09-10T18:59:21.183065+0000 mon.a (mon.0) 2477 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: cluster 2024-09-10T18:59:21.183134+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: audit 2024-09-10T18:59:21.184317+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:22.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[32381]: audit 2024-09-10T18:59:21.185484+0000 mon.a (mon.0) 2480 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.696 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:22.799 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:22.832 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: audit 2024-09-10T18:59:21.181886+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: audit 2024-09-10T18:59:21.182981+0000 mon.a (mon.0) 2476 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi083-77648-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T18:59:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: audit 2024-09-10T18:59:21.183065+0000 mon.a (mon.0) 2477 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: cluster 2024-09-10T18:59:21.183134+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e405: 8 total, 8 up, 8 in 2024-09-10T18:59:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: audit 2024-09-10T18:59:21.184317+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:22.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:22 smithi165 bash[22591]: audit 2024-09-10T18:59:21.185484+0000 mon.a (mon.0) 2480 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:22.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:22.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:22] "GET /metrics HTTP/1.1" 200 228754 "" "Prometheus/2.43.0" 2024-09-10T18:59:22.972 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:22.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:23.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:23.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFull3:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:5d165639:::164:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f43765fc:::165:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b4c720e9:::166:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e694b040:::167:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:afa38db2:::168:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:77ba9f53:::169:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:87495034:::170:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7c96bf0e:::171:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:dbe346cc:::172:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:e943ec24:::173:head 2024-09-10T18:59:23.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f97a9c0c:::174:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:6f26e74d:::175:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4f95e106:::176:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:0e6f2f8f:::177:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:05db05f1:::178:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:38a78d66:::179:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:d095610b:::180:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:a1a9d709:::181:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:1e5d39db:::182:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:f7df4fb9:::183:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:03a7f161:::184:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ba70721e:::185:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:28e5662d:::186:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:973d52de:::187:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:4303eb1c:::188:head 2024-09-10T18:59:23.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:b990b48e:::189:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:29b8165b:::190:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:3547f197:::191:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:7e260936:::192:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:1abec7b1:::193:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:10fdda93:::194:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:15817eea:::195:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:770bab57:::196:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:ed9e13e7:::197:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:71471a8f:::198:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 193:10fb1d02:::199:head 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8448 ms) 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,0 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first is 1725994718 2024-09-10T18:59:23.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,1725994726,1725994727,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,1725994726,1725994727,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994718,1725994720,1725994721,1725994723,1725994724,1725994726,1725994727,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725994721,1725994723,1725994724,1725994726,1725994727,1725994729,1725994730,0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first now 1725994721, trimmed 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20544 ms) 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: foo0 2024-09-10T18:59:23.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-10T18:59:23.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13979 ms) 2024-09-10T18:59:23.198 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-10T18:59:23.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:23.234 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:23.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:23.369 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:23.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:23.506 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:23.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:23.646 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:23.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:23.785 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:22.067433+0000 mgr.y (mgr.24526) 409 : cluster [DBG] pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:22.178256+0000 mon.a (mon.0) 2481 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:22.638393+0000 mon.a (mon.0) 2482 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:22.638600+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2024-09-10T18:59:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:22.685696+0000 mon.b (mon.1) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:22.687114+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:22.689349+0000 mon.a (mon.0) 2484 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:22.690577+0000 mon.a (mon.0) 2485 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: audit 2024-09-10T18:59:23.192030+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:23.192241+0000 mon.a (mon.0) 2487 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:23.582477+0000 osd.2 (osd.2) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:23.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:23 smithi165 bash[22591]: cluster 2024-09-10T18:59:23.583333+0000 osd.2 (osd.2) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:23.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:23.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:24.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:22.067433+0000 mgr.y (mgr.24526) 409 : cluster [DBG] pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:22.178256+0000 mon.a (mon.0) 2481 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:22.638393+0000 mon.a (mon.0) 2482 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:22.638600+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:22.685696+0000 mon.b (mon.1) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:22.687114+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:22.689349+0000 mon.a (mon.0) 2484 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:22.690577+0000 mon.a (mon.0) 2485 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: audit 2024-09-10T18:59:23.192030+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:23.192241+0000 mon.a (mon.0) 2487 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2024-09-10T18:59:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:23.582477+0000 osd.2 (osd.2) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[18629]: cluster 2024-09-10T18:59:23.583333+0000 osd.2 (osd.2) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:22.067433+0000 mgr.y (mgr.24526) 409 : cluster [DBG] pgmap v592: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:22.178256+0000 mon.a (mon.0) 2481 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:22.638393+0000 mon.a (mon.0) 2482 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]': finished 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:22.638600+0000 mon.a (mon.0) 2483 : cluster [DBG] osdmap e406: 8 total, 8 up, 8 in 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:22.685696+0000 mon.b (mon.1) 313 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:22.687114+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:22.689349+0000 mon.a (mon.0) 2484 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:22.690577+0000 mon.a (mon.0) 2485 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-49"}]: dispatch 2024-09-10T18:59:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: audit 2024-09-10T18:59:23.192030+0000 mon.a (mon.0) 2486 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi083-77648-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:23.192241+0000 mon.a (mon.0) 2487 : cluster [DBG] osdmap e407: 8 total, 8 up, 8 in 2024-09-10T18:59:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:23.582477+0000 osd.2 (osd.2) 165 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:23 smithi083 bash[32381]: cluster 2024-09-10T18:59:23.583333+0000 osd.2 (osd.2) 166 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:24.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:24.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:24.194 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:24.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:24.330 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:24.433 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.461 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:24.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:24.594 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:24.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[32381]: cluster 2024-09-10T18:59:24.068284+0000 mgr.y (mgr.24526) 410 : cluster [DBG] pgmap v595: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:24.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[32381]: cluster 2024-09-10T18:59:24.099652+0000 mon.a (mon.0) 2488 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:24.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[32381]: cluster 2024-09-10T18:59:24.111263+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2024-09-10T18:59:24.721 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[32381]: audit 2024-09-10T18:59:24.112230+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[32381]: audit 2024-09-10T18:59:24.117450+0000 mon.a (mon.0) 2490 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[18629]: cluster 2024-09-10T18:59:24.068284+0000 mgr.y (mgr.24526) 410 : cluster [DBG] pgmap v595: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[18629]: cluster 2024-09-10T18:59:24.099652+0000 mon.a (mon.0) 2488 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[18629]: cluster 2024-09-10T18:59:24.111263+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[18629]: audit 2024-09-10T18:59:24.112230+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:24.722 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:24 smithi083 bash[18629]: audit 2024-09-10T18:59:24.117450+0000 mon.a (mon.0) 2490 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:24.734 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:24.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.875 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:24.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:24.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:24.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:24 smithi165 bash[22591]: cluster 2024-09-10T18:59:24.068284+0000 mgr.y (mgr.24526) 410 : cluster [DBG] pgmap v595: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T18:59:24.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:24 smithi165 bash[22591]: cluster 2024-09-10T18:59:24.099652+0000 mon.a (mon.0) 2488 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:24.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:24 smithi165 bash[22591]: cluster 2024-09-10T18:59:24.111263+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e408: 8 total, 8 up, 8 in 2024-09-10T18:59:24.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:24 smithi165 bash[22591]: audit 2024-09-10T18:59:24.112230+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:24.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:24 smithi165 bash[22591]: audit 2024-09-10T18:59:24.117450+0000 mon.a (mon.0) 2490 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:25.013 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:25.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:25.139 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:25.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:25.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:25.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:25.413 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:25.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:25.551 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:25.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.684 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:25.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: cluster 2024-09-10T18:59:24.505213+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:25.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: cluster 2024-09-10T18:59:24.505504+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:25.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: audit 2024-09-10T18:59:25.115311+0000 mon.a (mon.0) 2491 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: cluster 2024-09-10T18:59:25.117094+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: audit 2024-09-10T18:59:25.121033+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: audit 2024-09-10T18:59:25.274022+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[32381]: audit 2024-09-10T18:59:25.277569+0000 mon.a (mon.0) 2494 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: cluster 2024-09-10T18:59:24.505213+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: cluster 2024-09-10T18:59:24.505504+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: audit 2024-09-10T18:59:25.115311+0000 mon.a (mon.0) 2491 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: cluster 2024-09-10T18:59:25.117094+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2024-09-10T18:59:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: audit 2024-09-10T18:59:25.121033+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:25.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: audit 2024-09-10T18:59:25.274022+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:25 smithi083 bash[18629]: audit 2024-09-10T18:59:25.277569+0000 mon.a (mon.0) 2494 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.817 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:25.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: cluster 2024-09-10T18:59:24.505213+0000 osd.5 (osd.5) 217 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: cluster 2024-09-10T18:59:24.505504+0000 osd.5 (osd.5) 218 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: audit 2024-09-10T18:59:25.115311+0000 mon.a (mon.0) 2491 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: cluster 2024-09-10T18:59:25.117094+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e409: 8 total, 8 up, 8 in 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: audit 2024-09-10T18:59:25.121033+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:25.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: audit 2024-09-10T18:59:25.274022+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:25 smithi165 bash[22591]: audit 2024-09-10T18:59:25.277569+0000 mon.a (mon.0) 2494 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T18:59:25.953 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:25.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:26.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:26.083 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:26.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:26.220 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:26.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:26.353 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:26.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:26.491 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:26.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:26.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.673+0000 7fb4c7c82640 1 -- 172.21.15.83:0/2966759759 >> v1:172.21.15.83:6790/0 conn(0x7fb4c01015f0 legacy=0x7fb4c01019c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.673+0000 7fb4c7c82640 1 -- 172.21.15.83:0/2966759759 shutdown_connections 2024-09-10T18:59:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.673+0000 7fb4c7c82640 1 -- 172.21.15.83:0/2966759759 >> 172.21.15.83:0/2966759759 conn(0x7fb4c0074a20 msgr2=0x7fb4c00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.673+0000 7fb4c7c82640 1 -- 172.21.15.83:0/2966759759 shutdown_connections 2024-09-10T18:59:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.673+0000 7fb4c7c82640 1 -- 172.21.15.83:0/2966759759 wait complete. 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 librados: starting msgr at 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 librados: starting objecter 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 Processor -- start 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- start start 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 librados: setting wanted keys 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 librados: calling monclient init 2024-09-10T18:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4c00fde60 con 0x7fb4c0107f00 2024-09-10T18:59:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4c011f800 con 0x7fb4c01015f0 2024-09-10T18:59:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4c01209a0 con 0x7fb4c01042e0 2024-09-10T18:59:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c521d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb4c01042e0 0x7fb4c01199a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56802/0 (socket says 172.21.15.83:56802) 2024-09-10T18:59:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c621f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb4c0107f00 0x7fb4c011d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54794/0 (socket says 172.21.15.83:54794) 2024-09-10T18:59:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c521d640 1 -- 172.21.15.83:0/3026409237 learned_addr learned my addr 172.21.15.83:0/3026409237 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:26.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter ms_handle_connect 0x7fb4c0107f00 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter ms_handle_connect 0x7fb4c01042e0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter ms_handle_connect 0x7fb4c01015f0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640039206 0 0) 0x7fb4c00fde60 con 0x7fb4c0107f00 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a80034e0 con 0x7fb4c0107f00 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825466741 0 0) 0x7fb4c011f800 con 0x7fb4c01015f0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c00fde60 con 0x7fb4c01015f0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660308875 0 0) 0x7fb4c01209a0 con 0x7fb4c01042e0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c011f800 con 0x7fb4c01042e0 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068601439 0 0) 0x7fb4a80034e0 con 0x7fb4c0107f00 2024-09-10T18:59:26.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4c01209a0 con 0x7fb4c0107f00 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb4b00019d0 con 0x7fb4c0107f00 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408479374 0 0) 0x7fb4c011f800 con 0x7fb4c01042e0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4a80034e0 con 0x7fb4c01042e0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb4b40018b0 con 0x7fb4c01042e0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659258874 0 0) 0x7fb4c00fde60 con 0x7fb4c01015f0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4c011f800 con 0x7fb4c01015f0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb4b80017b0 con 0x7fb4c01015f0 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1729407207 0 0) 0x7fb4c01209a0 con 0x7fb4c0107f00 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 >> v1:172.21.15.83:6790/0 conn(0x7fb4c01042e0 legacy=0x7fb4c01199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 >> v1:172.21.15.165:6789/0 conn(0x7fb4c01015f0 legacy=0x7fb4c00fd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c0121b80 con 0x7fb4c0107f00 2024-09-10T18:59:26.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c0120b90 con 0x7fb4c0107f00 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 10 client.25281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c0120fe0 con 0x7fb4c0107f00 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4c7c82640 1 librados: init done 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb4b0002380 con 0x7fb4c0107f00 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.677+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb4b00059a0 con 0x7fb4c0107f00 2024-09-10T18:59:26.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0005010 con 0x7fb4c0107f00 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fb4b001be40 con 0x7fb4c0107f00 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(410..410 src has 69..410) v4 ==== 8103+0+0 (unknown 1921669347 0 0) 0x7fb4b008a550 con 0x7fb4c0107f00 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 10 client.25281.objecter ms_dispatch 0x7fb4c0000b90 osd_map(410..410 src has 69..410) v4 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 3 client.25281.objecter handle_osd_map got epochs [410,410] > 0 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 3 client.25281.objecter handle_osd_map decoding full epoch 410 2024-09-10T18:59:26.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 20 client.25281.objecter dump_active .. 0 homeless 2024-09-10T18:59:26.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.681+0000 7fb4be7fc640 10 client.25281.objecter ms_handle_connect 0x7fb4a8071820 2024-09-10T18:59:26.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.689+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fb4b0056ab0 con 0x7fb4c0107f00 2024-09-10T18:59:26.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:26.763 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:26.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:26.899 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:26.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:26.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.929+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb4a0005840 con 0x7fb4c0107f00 2024-09-10T18:59:26.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4be7fc640 1 -- 172.21.15.83:0/3026409237 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2474 (unknown 3226119948 0 2346968541) 0x7fb4b0056450 con 0x7fb4c0107f00 2024-09-10T18:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:59:26.934 INFO:teuthology.orchestra.run.smithi083.stdout:{"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1101,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":410,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994114,"num_in_osds":8,"osd_in_since":1725994087,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":228}],"num_pgs":228,"num_pools":9,"num_objects":346,"data_bytes":8791917,"bytes_used":2539905024,"bytes_avail":765420380160,"bytes_total":767960285184,"read_bytes_sec":13562,"write_bytes_sec":1023,"read_op_per_sec":13,"write_op_per_sec":8},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.83:8443/","prometheus":"http://172.21.15.83:9283/"}},"servicemap":{"epoch":24,"modified":"2024-09-10T18:56:01.193087+0000","services":{"rgw":{"daemons":{"summary":"","14493":{"start_epoch":13,"start_stamp":"2024-09-10T18:49:25.207072+0000","gid":14493,"addr":"172.21.15.83:0/703687015","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)","ceph_version_short":"17.2.7-1606-g5be57395","container_hostname":"smithi083","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi083","id":"foo.a","kernel_description":"#215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024","kernel_version":"5.4.0-195-generic","mem_swap_kb":"2097148","mem_total_kb":"32754944","num_handles":"1","os":"Linux","pid":"6","realm_id":"","realm_name":"","zone_id":"ef939013-895d-4fcb-a130-e9a8938d4e31","zone_name":"default","zonegroup_id":"348ee464-2af6-469c-a2f0-1b47a5a7c71b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-10T18:59:26.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 10 librados: watch_flush enter 2024-09-10T18:59:26.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 10 librados: watch_flush exit 2024-09-10T18:59:26.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 20 client.25281.objecter shutdown clearing up homeless session... 2024-09-10T18:59:26.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 10 client.25281.objecter successfully canceled tick 2024-09-10T18:59:26.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 >> v1:172.21.15.83:6800/3071793735 conn(0x7fb4a8071820 legacy=0x7fb4a8073cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 >> v1:172.21.15.83:6789/0 conn(0x7fb4c0107f00 legacy=0x7fb4c011d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 shutdown_connections 2024-09-10T18:59:26.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.933+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 >> 172.21.15.83:0/3026409237 conn(0x7fb4c0074a20 msgr2=0x7fb4c00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:26.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.937+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 shutdown_connections 2024-09-10T18:59:26.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.937+0000 7fb4a7fff640 1 -- 172.21.15.83:0/3026409237 wait complete. 2024-09-10T18:59:26.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:26.937+0000 7fb4a7fff640 1 librados: shutdown 2024-09-10T18:59:27.002 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:27.032 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:27.131 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ PP2 2024-09-10T18:59:27.131 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2924 ms) 2024-09-10T18:59:27.131 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-10T18:59:27.131 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7020 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6990 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6991 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7071 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6983 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6958 ms) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139060 ms total) 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-09-10T18:59:27.132 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-10T18:59:27.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (301419 ms total) 2024-09-10T18:59:27.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: cluster 2024-09-10T18:59:26.070333+0000 mgr.y (mgr.24526) 411 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.116992+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.118709+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.118797+0000 mon.a (mon.0) 2496 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: cluster 2024-09-10T18:59:26.118882+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.120046+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:27.133 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.120714+0000 mon.a (mon.0) 2499 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[18629]: audit 2024-09-10T18:59:26.934141+0000 mon.a (mon.0) 2500 : audit [DBG] from='client.? v1:172.21.15.83:0/3026409237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:59:27.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:27.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:27.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:27.290 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:27.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: cluster 2024-09-10T18:59:26.070333+0000 mgr.y (mgr.24526) 411 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.116992+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.118709+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.118797+0000 mon.a (mon.0) 2496 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: cluster 2024-09-10T18:59:26.118882+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2024-09-10T18:59:27.408 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.120046+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:27.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.120714+0000 mon.a (mon.0) 2499 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[22591]: audit 2024-09-10T18:59:26.934141+0000 mon.a (mon.0) 2500 : audit [DBG] from='client.? v1:172.21.15.83:0/3026409237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:59:27.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: cluster 2024-09-10T18:59:26.070333+0000 mgr.y (mgr.24526) 411 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.116992+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.118709+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.118797+0000 mon.a (mon.0) 2496 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: cluster 2024-09-10T18:59:26.118882+0000 mon.a (mon.0) 2497 : cluster [DBG] osdmap e410: 8 total, 8 up, 8 in 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.120046+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]: dispatch 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.120714+0000 mon.a (mon.0) 2499 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:27.505 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:27 smithi083 bash[32381]: audit 2024-09-10T18:59:26.934141+0000 mon.a (mon.0) 2500 : audit [DBG] from='client.? v1:172.21.15.83:0/3026409237' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-09-10T18:59:27.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:27.566 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:27.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:27.700 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:27.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:27.835 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:27.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:27.969 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:27.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:28.072 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:28.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:28.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: audit 2024-09-10T18:59:27.120166+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: audit 2024-09-10T18:59:27.121003+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:28.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: audit 2024-09-10T18:59:27.121117+0000 mon.a (mon.0) 2502 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:28.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: cluster 2024-09-10T18:59:27.121217+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2024-09-10T18:59:28.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: audit 2024-09-10T18:59:27.124102+0000 mon.a (mon.0) 2504 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.135 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[18629]: cluster 2024-09-10T18:59:28.115376+0000 mon.a (mon.0) 2505 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:28.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:28.242 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:28.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:28.363 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: audit 2024-09-10T18:59:27.120166+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: audit 2024-09-10T18:59:27.121003+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: audit 2024-09-10T18:59:27.121117+0000 mon.a (mon.0) 2502 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: cluster 2024-09-10T18:59:27.121217+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2024-09-10T18:59:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: audit 2024-09-10T18:59:27.124102+0000 mon.a (mon.0) 2504 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:28 smithi165 bash[22591]: cluster 2024-09-10T18:59:28.115376+0000 mon.a (mon.0) 2505 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:28.380 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:28.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:28.516 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: audit 2024-09-10T18:59:27.120166+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: audit 2024-09-10T18:59:27.121003+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? v1:172.21.15.83:0/4034623724' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi083-77648-66"}]': finished 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: audit 2024-09-10T18:59:27.121117+0000 mon.a (mon.0) 2502 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: cluster 2024-09-10T18:59:27.121217+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e411: 8 total, 8 up, 8 in 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: audit 2024-09-10T18:59:27.124102+0000 mon.a (mon.0) 2504 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]: dispatch 2024-09-10T18:59:28.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:28 smithi083 bash[32381]: cluster 2024-09-10T18:59:28.115376+0000 mon.a (mon.0) 2505 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T18:59:28.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:28.652 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:28.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:28.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:28.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:28.925 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:28.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:29.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:29.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:29.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:29.199 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:29.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:29.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: audit 2024-09-10T18:59:27.935187+0000 mgr.y (mgr.24526) 412 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:29.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: cluster 2024-09-10T18:59:28.071126+0000 mgr.y (mgr.24526) 413 : cluster [DBG] pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:29.316 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: audit 2024-09-10T18:59:28.128014+0000 mon.a (mon.0) 2506 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]': finished 2024-09-10T18:59:29.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: cluster 2024-09-10T18:59:28.128160+0000 mon.a (mon.0) 2507 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2024-09-10T18:59:29.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: audit 2024-09-10T18:59:28.185609+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.317 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:29 smithi165 bash[22591]: audit 2024-09-10T18:59:28.189401+0000 mon.a (mon.0) 2508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.336 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:29.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: audit 2024-09-10T18:59:27.935187+0000 mgr.y (mgr.24526) 412 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: cluster 2024-09-10T18:59:28.071126+0000 mgr.y (mgr.24526) 413 : cluster [DBG] pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: audit 2024-09-10T18:59:28.128014+0000 mon.a (mon.0) 2506 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]': finished 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: cluster 2024-09-10T18:59:28.128160+0000 mon.a (mon.0) 2507 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: audit 2024-09-10T18:59:28.185609+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[32381]: audit 2024-09-10T18:59:28.189401+0000 mon.a (mon.0) 2508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: audit 2024-09-10T18:59:27.935187+0000 mgr.y (mgr.24526) 412 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: cluster 2024-09-10T18:59:28.071126+0000 mgr.y (mgr.24526) 413 : cluster [DBG] pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 1023 B/s wr, 22 op/s 2024-09-10T18:59:29.452 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: audit 2024-09-10T18:59:28.128014+0000 mon.a (mon.0) 2506 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-51", "mode": "writeback"}]': finished 2024-09-10T18:59:29.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: cluster 2024-09-10T18:59:28.128160+0000 mon.a (mon.0) 2507 : cluster [DBG] osdmap e412: 8 total, 8 up, 8 in 2024-09-10T18:59:29.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: audit 2024-09-10T18:59:28.185609+0000 mon.b (mon.1) 319 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.453 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:29 smithi083 bash[18629]: audit 2024-09-10T18:59:28.189401+0000 mon.a (mon.0) 2508 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T18:59:29.472 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:29.544 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-10T18:59:29.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:59:29.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:29.608 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:29.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:29.747 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:29.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:29.888 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:29.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:29.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:30.024 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:30.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:30.163 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:30.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:30.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:30 smithi165 bash[22591]: audit 2024-09-10T18:59:29.172817+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:30 smithi165 bash[22591]: audit 2024-09-10T18:59:29.173549+0000 mon.a (mon.0) 2509 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:59:30.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:30 smithi165 bash[22591]: cluster 2024-09-10T18:59:29.173669+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2024-09-10T18:59:30.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:30 smithi165 bash[22591]: audit 2024-09-10T18:59:29.176267+0000 mon.a (mon.0) 2511 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:30 smithi165 bash[22591]: audit 2024-09-10T18:59:29.563129+0000 mon.c (mon.2) 614 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:30.304 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- 172.21.15.83:0/339507945 >> v1:172.21.15.83:6789/0 conn(0x7f2bf81075a0 legacy=0x7f2bf8109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- 172.21.15.83:0/339507945 shutdown_connections 2024-09-10T18:59:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- 172.21.15.83:0/339507945 >> 172.21.15.83:0/339507945 conn(0x7f2bf80fa3d0 msgr2=0x7f2bf80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- 172.21.15.83:0/339507945 shutdown_connections 2024-09-10T18:59:30.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- 172.21.15.83:0/339507945 wait complete. 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 librados: starting msgr at 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 librados: starting objecter 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 Processor -- start 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- start start 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 librados: setting wanted keys 2024-09-10T18:59:30.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 librados: calling monclient init 2024-09-10T18:59:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2bf811e590 con 0x7f2bf8104810 2024-09-10T18:59:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2bf811f730 con 0x7f2bf81075a0 2024-09-10T18:59:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bfdc69640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2bf8120910 con 0x7f2bf810b130 2024-09-10T18:59:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bf77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2bf8104810 0x7f2bf8103740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54808/0 (socket says 172.21.15.83:54808) 2024-09-10T18:59:30.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.329+0000 7f2bf77fe640 1 -- 172.21.15.83:0/1027337646 learned_addr learned my addr 172.21.15.83:0/1027337646 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:30.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter ms_handle_connect 0x7f2bf8104810 2024-09-10T18:59:30.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter ms_handle_connect 0x7f2bf810b130 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter ms_handle_connect 0x7f2bf81075a0 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95067518 0 0) 0x7f2bf811e590 con 0x7f2bf8104810 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be00054b0 con 0x7f2bf8104810 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563814452 0 0) 0x7f2bf8120910 con 0x7f2bf810b130 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf811e590 con 0x7f2bf810b130 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668165992 0 0) 0x7f2bf811f730 con 0x7f2bf81075a0 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf8120910 con 0x7f2bf81075a0 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1618754027 0 0) 0x7f2be00054b0 con 0x7f2bf8104810 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bf811f730 con 0x7f2bf8104810 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2bf0001cc0 con 0x7f2bf8104810 2024-09-10T18:59:30.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918927986 0 0) 0x7f2bf811f730 con 0x7f2bf8104810 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 >> v1:172.21.15.83:6790/0 conn(0x7f2bf810b130 legacy=0x7f2bf811ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 >> v1:172.21.15.165:6789/0 conn(0x7f2bf81075a0 legacy=0x7f2bf8119740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8121b30 con 0x7f2bf8104810 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf8120b00 con 0x7f2bf8104810 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bfdc69640 10 client.25287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf8120f50 con 0x7f2bf8104810 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bfdc69640 1 librados: init done 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2bf0002050 con 0x7f2bf8104810 2024-09-10T18:59:30.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2bf0005ee0 con 0x7f2bf8104810 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4005010 con 0x7f2bf8104810 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.333+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2bf00025f0 con 0x7f2bf8104810 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(414..414 src has 69..414) v4 ==== 7996+0+0 (unknown 1591115502 0 0) 0x7f2bf008a900 con 0x7f2bf8104810 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 10 client.25287.objecter ms_dispatch 0x7f2bf8000b90 osd_map(414..414 src has 69..414) v4 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 3 client.25287.objecter handle_osd_map got epochs [414,414] > 0 2024-09-10T18:59:30.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 3 client.25287.objecter handle_osd_map decoding full epoch 414 2024-09-10T18:59:30.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 20 client.25287.objecter dump_active .. 0 homeless 2024-09-10T18:59:30.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.337+0000 7f2bdffff640 10 client.25287.objecter ms_handle_connect 0x7f2be00716a0 2024-09-10T18:59:30.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.341+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2bf0056f00 con 0x7f2bf8104810 2024-09-10T18:59:30.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:30.421 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[32381]: audit 2024-09-10T18:59:29.172817+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[32381]: audit 2024-09-10T18:59:29.173549+0000 mon.a (mon.0) 2509 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[32381]: cluster 2024-09-10T18:59:29.173669+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[32381]: audit 2024-09-10T18:59:29.176267+0000 mon.a (mon.0) 2511 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[32381]: audit 2024-09-10T18:59:29.563129+0000 mon.c (mon.2) 614 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[18629]: audit 2024-09-10T18:59:29.172817+0000 mon.b (mon.1) 320 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[18629]: audit 2024-09-10T18:59:29.173549+0000 mon.a (mon.0) 2509 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[18629]: cluster 2024-09-10T18:59:29.173669+0000 mon.a (mon.0) 2510 : cluster [DBG] osdmap e413: 8 total, 8 up, 8 in 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[18629]: audit 2024-09-10T18:59:29.176267+0000 mon.a (mon.0) 2511 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T18:59:30.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:30 smithi083 bash[18629]: audit 2024-09-10T18:59:29.563129+0000 mon.c (mon.2) 614 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:30.434 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:30.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:30.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.541+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2be40052c0 con 0x7f2bf8104810 2024-09-10T18:59:30.563 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:30.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- 172.21.15.83:0/2592746285 >> v1:172.21.15.83:6789/0 conn(0x7fabc8108f40 legacy=0x7fabc810b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- 172.21.15.83:0/2592746285 shutdown_connections 2024-09-10T18:59:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- 172.21.15.83:0/2592746285 >> 172.21.15.83:0/2592746285 conn(0x7fabc8075290 msgr2=0x7fabc80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- 172.21.15.83:0/2592746285 shutdown_connections 2024-09-10T18:59:30.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- 172.21.15.83:0/2592746285 wait complete. 2024-09-10T18:59:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 librados: starting msgr at 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 librados: starting objecter 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 Processor -- start 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 -- start start 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 librados: setting wanted keys 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.561+0000 7fabce4fc640 1 librados: calling monclient init 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fabc8122b70 con 0x7fabc80fd610 2024-09-10T18:59:30.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fabc8123d30 con 0x7fabc811edc0 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fabc8124f30 con 0x7fabc8108f40 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fabc80fd610 0x7fabc8101e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54828/0 (socket says 172.21.15.83:54828) 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc7fff640 1 -- 172.21.15.83:0/3679328003 learned_addr learned my addr 172.21.15.83:0/3679328003 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter ms_handle_connect 0x7fabc80fd610 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter ms_handle_connect 0x7fabc8108f40 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071671439 0 0) 0x7fabc8122b70 con 0x7fabc80fd610 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb00039c0 con 0x7fabc80fd610 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302509601 0 0) 0x7fabc8124f30 con 0x7fabc8108f40 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8122b70 con 0x7fabc8108f40 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter ms_handle_connect 0x7fabc811edc0 2024-09-10T18:59:30.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376612517 0 0) 0x7fabb00039c0 con 0x7fabc80fd610 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc8124f30 con 0x7fabc80fd610 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fabb80019f0 con 0x7fabc80fd610 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4289244628 0 0) 0x7fabc8122b70 con 0x7fabc8108f40 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb00039c0 con 0x7fabc8108f40 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fabbc0017f0 con 0x7fabc8108f40 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956413381 0 0) 0x7fabc8123d30 con 0x7fabc811edc0 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc8122b70 con 0x7fabc811edc0 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1566560664 0 0) 0x7fabc8124f30 con 0x7fabc80fd610 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6790/0 conn(0x7fabc8108f40 legacy=0x7fabc8102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.165:6789/0 conn(0x7fabc811edc0 legacy=0x7fabc8121270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc8126130 con 0x7fabc80fd610 2024-09-10T18:59:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc8125120 con 0x7fabc80fd610 2024-09-10T18:59:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 10 client.25293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc8125570 con 0x7fabc80fd610 2024-09-10T18:59:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 librados: init done 2024-09-10T18:59:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fabb8002490 con 0x7fabc80fd610 2024-09-10T18:59:30.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fabb8005b80 con 0x7fabc80fd610 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fabb4004ff0 con 0x7fabc80fd610 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.565+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fabb801bfc0 con 0x7fabc80fd610 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(414..414 src has 69..414) v4 ==== 7996+0+0 (unknown 1591115502 0 0) 0x7fabb808a750 con 0x7fabc80fd610 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 osd_map(414..414 src has 69..414) v4 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 3 client.25293.objecter handle_osd_map got epochs [414,414] > 0 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 3 client.25293.objecter handle_osd_map decoding full epoch 414 2024-09-10T18:59:30.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 20 client.25293.objecter dump_active .. 0 homeless 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v414) v1 ==== 54+0+16 (unknown 330119727 0 2420000192) 0x7fabb808cb40 con 0x7fabc80fd610 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb0071720 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 10 client.25293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 1 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 1 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 0 1 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 4 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 10 client.25293.objecter _send_command 1 2024-09-10T18:59:30.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb4005290 con 0x7fabb4004610 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb4004610 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb4005290 con 0x7fabb4004610 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 1 = system:0 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 4 2024-09-10T18:59:30.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.569+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 0 1 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 10 client.25293.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 2 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 2 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4005810 osd=0 3 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 0 2 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 4 2024-09-10T18:59:30.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 10 client.25293.objecter _send_command 2 2024-09-10T18:59:30.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb4004cc0 con 0x7fabb4004610 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb4004cc0 con 0x7fabb4004610 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 4 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.585+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 0 2 2024-09-10T18:59:30.590 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 10 client.25293.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 1 3 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 4 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 10 client.25293.objecter _send_command 3 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb40022b0 con 0x7fabb4004110 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb4004110 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb40022b0 con 0x7fabb4004110 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 3 = system:0 2024-09-10T18:59:30.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 4 2024-09-10T18:59:30.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.589+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 1 3 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 10 client.25293.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 4 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 4 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4003fa0 osd=1 3 2024-09-10T18:59:30.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 1 4 2024-09-10T18:59:30.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 4 2024-09-10T18:59:30.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 10 client.25293.objecter _send_command 4 2024-09-10T18:59:30.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.609+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb4001890 con 0x7fabb4004110 2024-09-10T18:59:30.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb4001890 con 0x7fabb4004110 2024-09-10T18:59:30.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 4 2024-09-10T18:59:30.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 1 4 2024-09-10T18:59:30.616 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 10 client.25293.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 5 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 5 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 2 5 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 4 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 10 client.25293.objecter _send_command 5 2024-09-10T18:59:30.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb4003d90 con 0x7fabb400d350 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb400d350 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb4003d90 con 0x7fabb400d350 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 5 = system:0 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 4 2024-09-10T18:59:30.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.613+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 2 5 2024-09-10T18:59:30.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 10 client.25293.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 6 2024-09-10T18:59:30.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 6 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb40035e0 osd=2 3 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 2 6 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 4 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 10 client.25293.objecter _send_command 6 2024-09-10T18:59:30.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb40030d0 con 0x7fabb400d350 2024-09-10T18:59:30.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb40030d0 con 0x7fabb400d350 2024-09-10T18:59:30.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 4 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 2 6 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 10 client.25293.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 7 2024-09-10T18:59:30.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 7 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 3 7 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 4 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 10 client.25293.objecter _send_command 7 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.629+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb400d100 con 0x7fabb4011fd0 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb4011fd0 2024-09-10T18:59:30.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb400d100 con 0x7fabb4011fd0 2024-09-10T18:59:30.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:59:30.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 7 = system:0 2024-09-10T18:59:30.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 4 2024-09-10T18:59:30.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.633+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 3 7 2024-09-10T18:59:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 10 client.25293.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 8 2024-09-10T18:59:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 8 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb400c990 osd=3 3 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 3 8 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 4 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 10 client.25293.objecter _send_command 8 2024-09-10T18:59:30.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb40132d0 con 0x7fabb4011fd0 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb40132d0 con 0x7fabb4011fd0 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 4 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.653+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 3 8 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:59:30.658 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:59:30.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 10 client.25293.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 9 2024-09-10T18:59:30.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 9 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 4 9 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 4 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 10 client.25293.objecter _send_command 9 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb401a210 con 0x7fabb4016710 2024-09-10T18:59:30.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb4016710 2024-09-10T18:59:30.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb401a210 con 0x7fabb4016710 2024-09-10T18:59:30.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:59:30.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 9 = system:0 2024-09-10T18:59:30.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 4 2024-09-10T18:59:30.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.657+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 4 9 2024-09-10T18:59:30.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 10 client.25293.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 10 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 10 2024-09-10T18:59:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb40132d0 osd=4 3 2024-09-10T18:59:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 4 10 2024-09-10T18:59:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 4 2024-09-10T18:59:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 10 client.25293.objecter _send_command 10 2024-09-10T18:59:30.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.673+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb400c240 con 0x7fabb4016710 2024-09-10T18:59:30.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb400c240 con 0x7fabb4016710 2024-09-10T18:59:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 4 2024-09-10T18:59:30.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 4 10 2024-09-10T18:59:30.680 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 10 client.25293.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 11 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 11 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 5 11 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 4 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 10 client.25293.objecter _send_command 11 2024-09-10T18:59:30.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb4011b20 con 0x7fabb4011610 2024-09-10T18:59:30.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.677+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb4011610 2024-09-10T18:59:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.681+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb4011b20 con 0x7fabb4011610 2024-09-10T18:59:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.681+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:59:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.681+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 11 = system:0 2024-09-10T18:59:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.681+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 4 2024-09-10T18:59:30.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.681+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 5 11 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 10 client.25293.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 12 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 12 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb400c240 osd=5 3 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 5 12 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 4 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 10 client.25293.objecter _send_command 12 2024-09-10T18:59:30.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.693+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb4011b20 con 0x7fabb4011610 2024-09-10T18:59:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.697+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb4011b20 con 0x7fabb4011610 2024-09-10T18:59:30.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.697+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.697+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.697+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 4 2024-09-10T18:59:30.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.697+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 5 12 2024-09-10T18:59:30.701 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:59:30.701 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 10 client.25293.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 13 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 13 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 6 13 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 4 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 10 client.25293.objecter _send_command 13 2024-09-10T18:59:30.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb4011e20 con 0x7fabb401f140 2024-09-10T18:59:30.703 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb401f140 2024-09-10T18:59:30.704 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb4011e20 con 0x7fabb401f140 2024-09-10T18:59:30.704 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:59:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 13 = system:0 2024-09-10T18:59:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 4 2024-09-10T18:59:30.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.701+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 6 13 2024-09-10T18:59:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:30.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 10 client.25293.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 14 2024-09-10T18:59:30.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 14 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4011b20 osd=6 3 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 6 14 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 4 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 10 client.25293.objecter _send_command 14 2024-09-10T18:59:30.721 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.717+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb4024340 con 0x7fabb401f140 2024-09-10T18:59:30.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb4024340 con 0x7fabb401f140 2024-09-10T18:59:30.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 4 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 6 14 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:59:30.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 10 client.25293.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 15 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 15 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 7 15 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 4 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 10 client.25293.objecter _send_command 15 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fabb4024ed0 con 0x7fabb40249f0 2024-09-10T18:59:30.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.721+0000 7fabc4ff9640 10 client.25293.objecter ms_handle_connect 0x7fabb40249f0 2024-09-10T18:59:30.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.725+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fabb4024ed0 con 0x7fabb40249f0 2024-09-10T18:59:30.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.725+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:59:30.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.725+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 15 = system:0 2024-09-10T18:59:30.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.725+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 4 2024-09-10T18:59:30.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.725+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 7 15 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 10 client.25293.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign -1 16 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 20 client.25293.objecter _get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_remove -1 16 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 20 client.25293.objecter get_session s=0x7fabb4024340 osd=7 3 2024-09-10T18:59:30.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 15 client.25293.objecter _session_command_op_assign 7 16 2024-09-10T18:59:30.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 4 2024-09-10T18:59:30.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 10 client.25293.objecter _send_command 16 2024-09-10T18:59:30.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.741+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fabb4024ed0 con 0x7fabb40249f0 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabc4ff9640 1 -- 172.21.15.83:0/3679328003 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fabb4024ed0 con 0x7fabb40249f0 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabc4ff9640 10 client.25293.objecter ms_dispatch 0x7fabc8000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabc4ff9640 10 client.25293.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabc4ff9640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 4 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabc4ff9640 15 client.25293.objecter _session_command_op_remove 7 16 2024-09-10T18:59:30.747 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:59:30.748 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:59:30.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 librados: watch_flush enter 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 librados: watch_flush exit 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.0 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6801/2535527483 conn(0x7fabb4004610 legacy=0x7fabb4002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4005810 osd=0 1 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.1 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6805/2309624637 conn(0x7fabb4004110 legacy=0x7fabb400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4003fa0 osd=1 1 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.2 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6809/386094220 conn(0x7fabb400d350 legacy=0x7fabb400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40035e0 osd=2 1 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.3 2024-09-10T18:59:30.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6813/1279688021 conn(0x7fabb4011fd0 legacy=0x7fabb4012400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c990 osd=3 1 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.4 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.165:6800/3225515265 conn(0x7fabb4016710 legacy=0x7fabb4018af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb40132d0 osd=4 1 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.5 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.165:6804/571173777 conn(0x7fabb4011610 legacy=0x7fabb401b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb400c240 osd=5 1 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.6 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.165:6808/1317922936 conn(0x7fabb401f140 legacy=0x7fabb4021520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4011b20 osd=6 1 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter close_session for osd.7 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.165:6812/1610151650 conn(0x7fabb40249f0 legacy=0x7fabb4022c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter put_session s=0x7fabb4024340 osd=7 1 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 20 client.25293.objecter shutdown clearing up homeless session... 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 10 client.25293.objecter successfully canceled tick 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabcca99640 1 -- 172.21.15.83:0/3679328003 reap_dead start 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabcca99640 1 -- 172.21.15.83:0/3679328003 reap_dead start 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabcca99640 1 -- 172.21.15.83:0/3679328003 reap_dead start 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabcca99640 1 -- 172.21.15.83:0/3679328003 reap_dead start 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6800/3071793735 conn(0x7fabb0071720 legacy=0x7fabb0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabcca99640 1 -- 172.21.15.83:0/3679328003 reap_dead start 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> v1:172.21.15.83:6789/0 conn(0x7fabc80fd610 legacy=0x7fabc8101e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 shutdown_connections 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 >> 172.21.15.83:0/3679328003 conn(0x7fabc8075290 msgr2=0x7fabc80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:30.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 shutdown_connections 2024-09-10T18:59:30.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 -- 172.21.15.83:0/3679328003 wait complete. 2024-09-10T18:59:30.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:30.745+0000 7fabce4fc640 1 librados: shutdown 2024-09-10T18:59:30.799 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:59:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T18:59:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T18:59:30.800 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T18:59:30.800 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:59:30.807 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:30.836 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:30.940 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:30.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:31.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:31.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.185+0000 7f2bdffff640 1 -- 172.21.15.83:0/1027337646 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v415) v1 ==== 97+0+0 (unknown 3241076009 0 0) 0x7f2bf00568a0 con 0x7f2bf8104810 2024-09-10T18:59:31.189 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T18:59:31.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 10 librados: watch_flush enter 2024-09-10T18:59:31.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 10 librados: watch_flush exit 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 20 client.25287.objecter shutdown clearing up homeless session... 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 10 client.25287.objecter successfully canceled tick 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2be00716a0 legacy=0x7f2be0073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.189+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 >> v1:172.21.15.83:6789/0 conn(0x7f2bf8104810 legacy=0x7f2bf8103740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.193+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 shutdown_connections 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.193+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 >> 172.21.15.83:0/1027337646 conn(0x7f2bf80fa3d0 msgr2=0x7f2bf810b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:31.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.193+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 shutdown_connections 2024-09-10T18:59:31.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.193+0000 7f2bfdc69640 1 -- 172.21.15.83:0/1027337646 wait complete. 2024-09-10T18:59:31.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:31.193+0000 7f2bfdc69640 1 librados: shutdown 2024-09-10T18:59:31.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:31.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: cluster 2024-09-10T18:59:30.071910+0000 mgr.y (mgr.24526) 414 : cluster [DBG] pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:31.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: audit 2024-09-10T18:59:30.173827+0000 mon.a (mon.0) 2512 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:59:31.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: audit 2024-09-10T18:59:30.173960+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: cluster 2024-09-10T18:59:30.173970+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2024-09-10T18:59:31.222 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: audit 2024-09-10T18:59:30.183742+0000 mon.a (mon.0) 2514 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: audit 2024-09-10T18:59:30.542975+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[18629]: audit 2024-09-10T18:59:30.570005+0000 mon.a (mon.0) 2516 : audit [DBG] from='client.? v1:172.21.15.83:0/3679328003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: cluster 2024-09-10T18:59:30.071910+0000 mgr.y (mgr.24526) 414 : cluster [DBG] pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: audit 2024-09-10T18:59:30.173827+0000 mon.a (mon.0) 2512 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: audit 2024-09-10T18:59:30.173960+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: cluster 2024-09-10T18:59:30.173970+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: audit 2024-09-10T18:59:30.183742+0000 mon.a (mon.0) 2514 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: audit 2024-09-10T18:59:30.542975+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:59:31.223 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:31 smithi083 bash[32381]: audit 2024-09-10T18:59:30.570005+0000 mon.a (mon.0) 2516 : audit [DBG] from='client.? v1:172.21.15.83:0/3679328003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:31.235 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:31.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:31.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:31.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:31.492 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:31.492 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: cluster 2024-09-10T18:59:30.071910+0000 mgr.y (mgr.24526) 414 : cluster [DBG] pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: audit 2024-09-10T18:59:30.173827+0000 mon.a (mon.0) 2512 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: audit 2024-09-10T18:59:30.173960+0000 mon.b (mon.1) 321 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: cluster 2024-09-10T18:59:30.173970+0000 mon.a (mon.0) 2513 : cluster [DBG] osdmap e414: 8 total, 8 up, 8 in 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: audit 2024-09-10T18:59:30.183742+0000 mon.a (mon.0) 2514 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: audit 2024-09-10T18:59:30.542975+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T18:59:31.493 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:31 smithi165 bash[22591]: audit 2024-09-10T18:59:30.570005+0000 mon.a (mon.0) 2516 : audit [DBG] from='client.? v1:172.21.15.83:0/3679328003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:31.516 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:31.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:31.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:31.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:31.792 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:31.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:31.924 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:31.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:32.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:32.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:32.160 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:32.192 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:32.296 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:32.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: cluster 2024-09-10T18:59:31.177807+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: cluster 2024-09-10T18:59:31.177864+0000 mon.a (mon.0) 2518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: audit 2024-09-10T18:59:31.187683+0000 mon.b (mon.1) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: audit 2024-09-10T18:59:31.188424+0000 mon.a (mon.0) 2519 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: audit 2024-09-10T18:59:31.188682+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:59:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: cluster 2024-09-10T18:59:31.188839+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2024-09-10T18:59:32.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:32 smithi165 bash[22591]: audit 2024-09-10T18:59:31.197414+0000 mon.a (mon.0) 2522 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:32.470 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: cluster 2024-09-10T18:59:31.177807+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: cluster 2024-09-10T18:59:31.177864+0000 mon.a (mon.0) 2518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: audit 2024-09-10T18:59:31.187683+0000 mon.b (mon.1) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: audit 2024-09-10T18:59:31.188424+0000 mon.a (mon.0) 2519 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: audit 2024-09-10T18:59:31.188682+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: cluster 2024-09-10T18:59:31.188839+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18629]: audit 2024-09-10T18:59:31.197414+0000 mon.a (mon.0) 2522 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: cluster 2024-09-10T18:59:31.177807+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: cluster 2024-09-10T18:59:31.177864+0000 mon.a (mon.0) 2518 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: audit 2024-09-10T18:59:31.187683+0000 mon.b (mon.1) 322 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: audit 2024-09-10T18:59:31.188424+0000 mon.a (mon.0) 2519 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: audit 2024-09-10T18:59:31.188682+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? v1:172.21.15.83:0/1027337646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: cluster 2024-09-10T18:59:31.188839+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e415: 8 total, 8 up, 8 in 2024-09-10T18:59:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[32381]: audit 2024-09-10T18:59:31.197414+0000 mon.a (mon.0) 2522 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T18:59:32.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:32.608 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:32.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T18:59:32.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:32.740 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:32.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:32.867 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:32.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:32.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:32.984 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:32] "GET /metrics HTTP/1.1" 200 228772 "" "Prometheus/2.43.0" 2024-09-10T18:59:33.003 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:33.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:33.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:33.143 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:33.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[32381]: cluster 2024-09-10T18:59:32.072836+0000 mgr.y (mgr.24526) 415 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[32381]: audit 2024-09-10T18:59:32.200129+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[32381]: audit 2024-09-10T18:59:32.200767+0000 mon.a (mon.0) 2523 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[32381]: cluster 2024-09-10T18:59:32.200922+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[32381]: audit 2024-09-10T18:59:32.210075+0000 mon.a (mon.0) 2525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[18629]: cluster 2024-09-10T18:59:32.072836+0000 mgr.y (mgr.24526) 415 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[18629]: audit 2024-09-10T18:59:32.200129+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.257 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[18629]: audit 2024-09-10T18:59:32.200767+0000 mon.a (mon.0) 2523 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:33.258 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[18629]: cluster 2024-09-10T18:59:32.200922+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2024-09-10T18:59:33.258 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:33 smithi083 bash[18629]: audit 2024-09-10T18:59:32.210075+0000 mon.a (mon.0) 2525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:33.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:33.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:33 smithi165 bash[22591]: cluster 2024-09-10T18:59:32.072836+0000 mgr.y (mgr.24526) 415 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:33.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:33 smithi165 bash[22591]: audit 2024-09-10T18:59:32.200129+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:33 smithi165 bash[22591]: audit 2024-09-10T18:59:32.200767+0000 mon.a (mon.0) 2523 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T18:59:33.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:33 smithi165 bash[22591]: cluster 2024-09-10T18:59:32.200922+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e416: 8 total, 8 up, 8 in 2024-09-10T18:59:33.393 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:33 smithi165 bash[22591]: audit 2024-09-10T18:59:32.210075+0000 mon.a (mon.0) 2525 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T18:59:33.412 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:33.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:33.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:33.550 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:33.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:33.681 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:33.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:33.817 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:33.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:33.954 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:33.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:34.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:34.093 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:34.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:34.232 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:34.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:34.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:34.369 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:34.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:34.485 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:34 smithi165 bash[22591]: audit 2024-09-10T18:59:33.220352+0000 mon.a (mon.0) 2526 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T18:59:34.486 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:34 smithi165 bash[22591]: cluster 2024-09-10T18:59:33.220437+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2024-09-10T18:59:34.486 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:34 smithi165 bash[22591]: cluster 2024-09-10T18:59:34.115746+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2024-09-10T18:59:34.507 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:34.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[18629]: audit 2024-09-10T18:59:33.220352+0000 mon.a (mon.0) 2526 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T18:59:34.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[18629]: cluster 2024-09-10T18:59:33.220437+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2024-09-10T18:59:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[18629]: cluster 2024-09-10T18:59:34.115746+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2024-09-10T18:59:34.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[32381]: audit 2024-09-10T18:59:33.220352+0000 mon.a (mon.0) 2526 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T18:59:34.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[32381]: cluster 2024-09-10T18:59:33.220437+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e417: 8 total, 8 up, 8 in 2024-09-10T18:59:34.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:34 smithi083 bash[32381]: cluster 2024-09-10T18:59:34.115746+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e418: 8 total, 8 up, 8 in 2024-09-10T18:59:34.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:34.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:34.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T18:59:34.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T18:59:34.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:34.777 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:34.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:34.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:34.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:35.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:35.056 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:35.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:35.192 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:35.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:35.329 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:35.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:35.469 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:35.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:35 smithi083 bash[18629]: cluster 2024-09-10T18:59:34.073434+0000 mgr.y (mgr.24526) 416 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:35.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:35 smithi083 bash[18629]: cluster 2024-09-10T18:59:34.214814+0000 mon.a (mon.0) 2529 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T18:59:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:35 smithi083 bash[32381]: cluster 2024-09-10T18:59:34.073434+0000 mgr.y (mgr.24526) 416 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:35 smithi083 bash[32381]: cluster 2024-09-10T18:59:34.214814+0000 mon.a (mon.0) 2529 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T18:59:35.573 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:35.610 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:35 smithi165 bash[22591]: cluster 2024-09-10T18:59:34.073434+0000 mgr.y (mgr.24526) 416 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-10T18:59:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:35 smithi165 bash[22591]: cluster 2024-09-10T18:59:34.214814+0000 mon.a (mon.0) 2529 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T18:59:35.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:35.745 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:35.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:35.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:35.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:35.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:36.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:36.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:36.264 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:36.296 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:36.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.413 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:36 smithi165 bash[22591]: cluster 2024-09-10T18:59:35.847124+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:36.413 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:36 smithi165 bash[22591]: cluster 2024-09-10T18:59:35.847420+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:36.432 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:36.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:36.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:36 smithi083 bash[18629]: cluster 2024-09-10T18:59:35.847124+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:36.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:36 smithi083 bash[18629]: cluster 2024-09-10T18:59:35.847420+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:36.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:36 smithi083 bash[32381]: cluster 2024-09-10T18:59:35.847124+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:36 smithi083 bash[32381]: cluster 2024-09-10T18:59:35.847420+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:36.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:36.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:36.712 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:36.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.851 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:36.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.987 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:36.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:37.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:37.125 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:37.222 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:37.229 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:37.262 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:37.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:37.380 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:37 smithi083 bash[18629]: cluster 2024-09-10T18:59:36.075685+0000 mgr.y (mgr.24526) 417 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:37.380 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:37 smithi083 bash[18629]: cluster 2024-09-10T18:59:36.244438+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2024-09-10T18:59:37.380 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:37 smithi083 bash[32381]: cluster 2024-09-10T18:59:36.075685+0000 mgr.y (mgr.24526) 417 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:37.380 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:37 smithi083 bash[32381]: cluster 2024-09-10T18:59:36.244438+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2024-09-10T18:59:37.395 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:37.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:37 smithi165 bash[22591]: cluster 2024-09-10T18:59:36.075685+0000 mgr.y (mgr.24526) 417 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:37.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:37 smithi165 bash[22591]: cluster 2024-09-10T18:59:36.244438+0000 mon.a (mon.0) 2530 : cluster [DBG] osdmap e419: 8 total, 8 up, 8 in 2024-09-10T18:59:37.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:37.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.674 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:37.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:37.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:37.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.947 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:37.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:38.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:38.088 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:38.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:38.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:38.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:38.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:38.362 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:38.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:38.491 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:38.593 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:38.628 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:38.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:38.763 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:38.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:38.900 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:38.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:39.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:39.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:39.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:39.164 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:39.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:39.301 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:39 smithi165 bash[22591]: cluster 2024-09-10T18:59:37.481039+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:39 smithi165 bash[22591]: cluster 2024-09-10T18:59:37.481364+0000 osd.0 (osd.0) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:39 smithi165 bash[22591]: audit 2024-09-10T18:59:37.941354+0000 mgr.y (mgr.24526) 418 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:39 smithi165 bash[22591]: cluster 2024-09-10T18:59:38.076309+0000 mgr.y (mgr.24526) 419 : cluster [DBG] pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:39 smithi165 bash[22591]: cluster 2024-09-10T18:59:38.258847+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2024-09-10T18:59:39.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:39.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:39.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:39.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[18629]: cluster 2024-09-10T18:59:37.481039+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:39.547 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[18629]: cluster 2024-09-10T18:59:37.481364+0000 osd.0 (osd.0) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[18629]: audit 2024-09-10T18:59:37.941354+0000 mgr.y (mgr.24526) 418 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[18629]: cluster 2024-09-10T18:59:38.076309+0000 mgr.y (mgr.24526) 419 : cluster [DBG] pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[18629]: cluster 2024-09-10T18:59:38.258847+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[32381]: cluster 2024-09-10T18:59:37.481039+0000 osd.0 (osd.0) 218 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[32381]: cluster 2024-09-10T18:59:37.481364+0000 osd.0 (osd.0) 219 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[32381]: audit 2024-09-10T18:59:37.941354+0000 mgr.y (mgr.24526) 418 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[32381]: cluster 2024-09-10T18:59:38.076309+0000 mgr.y (mgr.24526) 419 : cluster [DBG] pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:39.548 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:39 smithi083 bash[32381]: cluster 2024-09-10T18:59:38.258847+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e420: 8 total, 8 up, 8 in 2024-09-10T18:59:39.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:39.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:39.713 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:39.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:39.849 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:39.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:39.981 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:39.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:40.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:40.118 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:40.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:40.255 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:40.333 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:40.333 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:40.358 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:40.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.521 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:40.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:40.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:40.797 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:40.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:40.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:40.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:41.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:41.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:41.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:41.210 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:41.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:41.346 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:41.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:41.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:41.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:41 smithi165 bash[22591]: cluster 2024-09-10T18:59:40.077152+0000 mgr.y (mgr.24526) 420 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:41.462 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:41.481 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:41.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:41 smithi083 bash[18629]: cluster 2024-09-10T18:59:40.077152+0000 mgr.y (mgr.24526) 420 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:41 smithi083 bash[32381]: cluster 2024-09-10T18:59:40.077152+0000 mgr.y (mgr.24526) 420 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:41.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:41.618 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:41.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:41.753 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:41.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:41.889 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:41.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:41.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:42.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:42.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:42.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:42.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:42.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:42.567 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:42.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:42.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:42.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.843 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:42.947 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.960 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:42] "GET /metrics HTTP/1.1" 200 228772 "" "Prometheus/2.43.0" 2024-09-10T18:59:42.982 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:42.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:43.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:43.121 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:43.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:43.260 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:43.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:43.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:43.514 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:43 smithi083 bash[18629]: cluster 2024-09-10T18:59:42.079135+0000 mgr.y (mgr.24526) 421 : cluster [DBG] pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:43.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:43 smithi083 bash[32381]: cluster 2024-09-10T18:59:42.079135+0000 mgr.y (mgr.24526) 421 : cluster [DBG] pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:43.531 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:43.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:43 smithi165 bash[22591]: cluster 2024-09-10T18:59:42.079135+0000 mgr.y (mgr.24526) 421 : cluster [DBG] pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:43.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:43.668 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:43.771 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:43.807 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:43.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:43.945 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:43.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:44.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:44.076 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.136+0000 7f3ad5106640 1 -- 172.21.15.83:0/3419287301 >> v1:172.21.15.83:6789/0 conn(0x7f3ad01042e0 legacy=0x7f3ad01066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.136+0000 7f3ad5106640 1 -- 172.21.15.83:0/3419287301 shutdown_connections 2024-09-10T18:59:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.136+0000 7f3ad5106640 1 -- 172.21.15.83:0/3419287301 >> 172.21.15.83:0/3419287301 conn(0x7f3ad0074a40 msgr2=0x7f3ad00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.136+0000 7f3ad5106640 1 -- 172.21.15.83:0/3419287301 shutdown_connections 2024-09-10T18:59:44.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.136+0000 7f3ad5106640 1 -- 172.21.15.83:0/3419287301 wait complete. 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 librados: starting msgr at 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 librados: starting objecter 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 Processor -- start 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- start start 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 librados: setting wanted keys 2024-09-10T18:59:44.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 librados: calling monclient init 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ad0100d60 con 0x7f3ad0107e40 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ad011f6d0 con 0x7f3ad01042e0 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ad0120870 con 0x7f3ad0101570 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3acf59e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f3ad0107e40 0x7f3ad011cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51948/0 (socket says 172.21.15.83:51948) 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3acf59e640 1 -- 172.21.15.83:0/3031909133 learned_addr learned my addr 172.21.15.83:0/3031909133 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad0107e40 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126348352 0 0) 0x7f3ad0100d60 con 0x7f3ad0107e40 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab8003120 con 0x7f3ad0107e40 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad01042e0 2024-09-10T18:59:44.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:44.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477758386 0 0) 0x7f3ad011f6d0 con 0x7f3ad01042e0 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad0100d60 con 0x7f3ad01042e0 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1424507810 0 0) 0x7f3ab8003120 con 0x7f3ad0107e40 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ad011f6d0 con 0x7f3ad0107e40 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3ac00018c0 con 0x7f3ad0107e40 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad0101570 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913146571 0 0) 0x7f3ad0120870 con 0x7f3ad0101570 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab8003120 con 0x7f3ad0101570 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3018266455 0 0) 0x7f3ad0100d60 con 0x7f3ad01042e0 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ad0120870 con 0x7f3ad01042e0 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3ac4002c90 con 0x7f3ad01042e0 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4191120666 0 0) 0x7f3ad011f6d0 con 0x7f3ad0107e40 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 >> v1:172.21.15.83:6790/0 conn(0x7f3ad0101570 legacy=0x7f3ad01001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:44.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 >> v1:172.21.15.165:6789/0 conn(0x7f3ad01042e0 legacy=0x7f3ad0119870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad0121a70 con 0x7f3ad0107e40 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad011f8c0 con 0x7f3ad0107e40 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 10 client.25299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad011fd10 con 0x7f3ad0107e40 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ad5106640 1 librados: init done 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f3ac0001c90 con 0x7f3ad0107e40 2024-09-10T18:59:44.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.140+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f3ac0005af0 con 0x7f3ad0107e40 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc005010 con 0x7f3ad0107e40 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f3ac00028e0 con 0x7f3ad0107e40 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(420..420 src has 69..420) v4 ==== 8022+0+0 (unknown 326640470 0 0) 0x7f3ac008a400 con 0x7f3ad0107e40 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 10 client.25299.objecter ms_dispatch 0x7f3ad0000b90 osd_map(420..420 src has 69..420) v4 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 3 client.25299.objecter handle_osd_map got epochs [420,420] > 0 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 3 client.25299.objecter handle_osd_map decoding full epoch 420 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 20 client.25299.objecter dump_active .. 0 homeless 2024-09-10T18:59:44.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.144+0000 7f3ab77fe640 10 client.25299.objecter ms_handle_connect 0x7f3ab8071510 2024-09-10T18:59:44.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.148+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f3ac0056a10 con 0x7f3ad0107e40 2024-09-10T18:59:44.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:44.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:44.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.332+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3abc000b90 con 0x7f3ab8071510 2024-09-10T18:59:44.343 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:44.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.352+0000 7f3ab77fe640 1 -- 172.21.15.83:0/3031909133 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+625623 (unknown 2965378022 0 1674846156) 0x7f3abc000b90 con 0x7f3ab8071510 2024-09-10T18:59:44.363 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T18:59:44.368 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 10 librados: watch_flush enter 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 10 librados: watch_flush exit 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 20 client.25299.objecter shutdown clearing up homeless session... 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 10 client.25299.objecter successfully canceled tick 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 >> v1:172.21.15.83:6800/3071793735 conn(0x7f3ab8071510 legacy=0x7f3ab80739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 >> v1:172.21.15.83:6789/0 conn(0x7f3ad0107e40 legacy=0x7f3ad011cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:44.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 shutdown_connections 2024-09-10T18:59:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 >> 172.21.15.83:0/3031909133 conn(0x7f3ad0074a40 msgr2=0x7f3ad00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 shutdown_connections 2024-09-10T18:59:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 -- 172.21.15.83:0/3031909133 wait complete. 2024-09-10T18:59:44.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:44.360+0000 7f3ad5106640 1 librados: shutdown 2024-09-10T18:59:44.384 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:44 smithi083 bash[18629]: audit 2024-09-10T18:59:43.247680+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.384 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:44 smithi083 bash[18629]: audit 2024-09-10T18:59:43.256044+0000 mon.a (mon.0) 2532 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:44 smithi083 bash[32381]: audit 2024-09-10T18:59:43.247680+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:44 smithi083 bash[32381]: audit 2024-09-10T18:59:43.256044+0000 mon.a (mon.0) 2532 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.446 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:44.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.599 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:44 smithi165 bash[22591]: audit 2024-09-10T18:59:43.247680+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.599 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:44 smithi165 bash[22591]: audit 2024-09-10T18:59:43.256044+0000 mon.a (mon.0) 2532 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:44.618 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:44.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:44.754 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:44.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.890 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:44.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:44.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.026 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:45.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:45.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:45.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:45.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:45.428 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: cluster 2024-09-10T18:59:44.079787+0000 mgr.y (mgr.24526) 422 : cluster [DBG] pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: audit 2024-09-10T18:59:44.267262+0000 mon.a (mon.0) 2533 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: cluster 2024-09-10T18:59:44.267322+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: audit 2024-09-10T18:59:44.267994+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: audit 2024-09-10T18:59:44.271523+0000 mon.a (mon.0) 2535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: audit 2024-09-10T18:59:44.571023+0000 mon.a (mon.0) 2536 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[18629]: audit 2024-09-10T18:59:44.573716+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: cluster 2024-09-10T18:59:44.079787+0000 mgr.y (mgr.24526) 422 : cluster [DBG] pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: audit 2024-09-10T18:59:44.267262+0000 mon.a (mon.0) 2533 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: cluster 2024-09-10T18:59:44.267322+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: audit 2024-09-10T18:59:44.267994+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: audit 2024-09-10T18:59:44.271523+0000 mon.a (mon.0) 2535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: audit 2024-09-10T18:59:44.571023+0000 mon.a (mon.0) 2536 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:45 smithi083 bash[32381]: audit 2024-09-10T18:59:44.573716+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:45.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: cluster 2024-09-10T18:59:44.079787+0000 mgr.y (mgr.24526) 422 : cluster [DBG] pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: audit 2024-09-10T18:59:44.267262+0000 mon.a (mon.0) 2533 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: cluster 2024-09-10T18:59:44.267322+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 8 in 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: audit 2024-09-10T18:59:44.267994+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: audit 2024-09-10T18:59:44.271523+0000 mon.a (mon.0) 2535 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: audit 2024-09-10T18:59:44.571023+0000 mon.a (mon.0) 2536 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:45.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:45 smithi165 bash[22591]: audit 2024-09-10T18:59:44.573716+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T18:59:45.567 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:45.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.701 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:45.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:45.837 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:45.940 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:45.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:45.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:46.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:46.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:46.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:46.244 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:46.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:44.336848+0000 mgr.y (mgr.24526) 423 : audit [DBG] from='client.25299 v1:172.21.15.83:0/3031909133' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: cluster 2024-09-10T18:59:45.223197+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: cluster 2024-09-10T18:59:45.223529+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:45.279076+0000 mon.a (mon.0) 2537 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: cluster 2024-09-10T18:59:45.279169+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:45.317702+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:45.319239+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:45.321338+0000 mon.a (mon.0) 2539 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.361 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:46 smithi165 bash[22591]: audit 2024-09-10T18:59:45.322736+0000 mon.a (mon.0) 2540 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.381 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:46.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:46.518 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:44.336848+0000 mgr.y (mgr.24526) 423 : audit [DBG] from='client.25299 v1:172.21.15.83:0/3031909133' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: cluster 2024-09-10T18:59:45.223197+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: cluster 2024-09-10T18:59:45.223529+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:45.279076+0000 mon.a (mon.0) 2537 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: cluster 2024-09-10T18:59:45.279169+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:45.317702+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:45.319239+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:45.321338+0000 mon.a (mon.0) 2539 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[18629]: audit 2024-09-10T18:59:45.322736+0000 mon.a (mon.0) 2540 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:44.336848+0000 mgr.y (mgr.24526) 423 : audit [DBG] from='client.25299 v1:172.21.15.83:0/3031909133' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: cluster 2024-09-10T18:59:45.223197+0000 osd.6 (osd.6) 183 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: cluster 2024-09-10T18:59:45.223529+0000 osd.6 (osd.6) 184 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:45.279076+0000 mon.a (mon.0) 2537 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]': finished 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: cluster 2024-09-10T18:59:45.279169+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e422: 8 total, 8 up, 8 in 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:45.317702+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:45.319239+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:45.321338+0000 mon.a (mon.0) 2539 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:46 smithi083 bash[32381]: audit 2024-09-10T18:59:45.322736+0000 mon.a (mon.0) 2540 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-51"}]: dispatch 2024-09-10T18:59:46.605 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":619,"stamp":"2024-09-10T18:59:44.263196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791933,"num_objects":348,"num_object_clones":2,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":2017,"num_read_kb":1773,"num_write":1416,"num_write_kb":14345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2804,"ondisk_log_size":2804,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2481532,"kb_used_data":77676,"kb_used_omap":13,"kb_used_meta":2403698,"kb_avail":747479684,"statfs":{"total":767960285184,"available":765419196416,"internally_reserved":0,"allocated":79540224,"data_stored":56153991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2461387456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.022945"},"pg_stats":[{"pgid":"209.1f","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247298+0000","last_change":"2024-09-10T18:59:25.264793+0000","last_active":"2024-09-10T18:59:36.247298+0000","last_peered":"2024-09-10T18:59:36.247298+0000","last_clean":"2024-09-10T18:59:36.247298+0000","last_became_active":"2024-09-10T18:59:25.264037+0000","last_became_peered":"2024-09-10T18:59:25.264037+0000","last_unstale":"2024-09-10T18:59:36.247298+0000","last_undegraded":"2024-09-10T18:59:36.247298+0000","last_fullsized":"2024-09-10T18:59:36.247298+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:39.938371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"209.1e","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889279+0000","last_change":"2024-09-10T18:59:25.256532+0000","last_active":"2024-09-10T18:59:36.889279+0000","last_peered":"2024-09-10T18:59:36.889279+0000","last_clean":"2024-09-10T18:59:36.889279+0000","last_became_active":"2024-09-10T18:59:25.256224+0000","last_became_peered":"2024-09-10T18:59:25.256224+0000","last_unstale":"2024-09-10T18:59:36.889279+0000","last_undegraded":"2024-09-10T18:59:36.889279+0000","last_fullsized":"2024-09-10T18:59:36.889279+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:44.632853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"209.1d","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243096+0000","last_change":"2024-09-10T18:59:25.259841+0000","last_active":"2024-09-10T18:59:36.243096+0000","last_peered":"2024-09-10T18:59:36.243096+0000","last_clean":"2024-09-10T18:59:36.243096+0000","last_became_active":"2024-09-10T18:59:25.259482+0000","last_became_peered":"2024-09-10T18:59:25.259482+0000","last_unstale":"2024-09-10T18:59:36.243096+0000","last_undegraded":"2024-09-10T18:59:36.243096+0000","last_fullsized":"2024-09-10T18:59:36.243096+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.390436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"209.1c","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246542+0000","last_change":"2024-09-10T18:59:25.251796+0000","last_active":"2024-09-10T18:59:36.246542+0000","last_peered":"2024-09-10T18:59:36.246542+0000","last_clean":"2024-09-10T18:59:36.246542+0000","last_became_active":"2024-09-10T18:59:25.251519+0000","last_became_peered":"2024-09-10T18:59:25.251519+0000","last_unstale":"2024-09-10T18:59:36.246542+0000","last_undegraded":"2024-09-10T18:59:36.246542+0000","last_fullsized":"2024-09-10T18:59:36.246542+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.982300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"209.1b","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926924+0000","last_change":"2024-09-10T18:59:25.258754+0000","last_active":"2024-09-10T18:59:36.926924+0000","last_peered":"2024-09-10T18:59:36.926924+0000","last_clean":"2024-09-10T18:59:36.926924+0000","last_became_active":"2024-09-10T18:59:25.258104+0000","last_became_peered":"2024-09-10T18:59:25.258104+0000","last_unstale":"2024-09-10T18:59:36.926924+0000","last_undegraded":"2024-09-10T18:59:36.926924+0000","last_fullsized":"2024-09-10T18:59:36.926924+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:47.956819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"209.1a","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922792+0000","last_change":"2024-09-10T18:59:25.268118+0000","last_active":"2024-09-10T18:59:36.922792+0000","last_peered":"2024-09-10T18:59:36.922792+0000","last_clean":"2024-09-10T18:59:36.922792+0000","last_became_active":"2024-09-10T18:59:25.267543+0000","last_became_peered":"2024-09-10T18:59:25.267543+0000","last_unstale":"2024-09-10T18:59:36.922792+0000","last_undegraded":"2024-09-10T18:59:36.922792+0000","last_fullsized":"2024-09-10T18:59:36.922792+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.880598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"209.19","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887498+0000","last_change":"2024-09-10T18:59:25.265498+0000","last_active":"2024-09-10T18:59:36.887498+0000","last_peered":"2024-09-10T18:59:36.887498+0000","last_clean":"2024-09-10T18:59:36.887498+0000","last_became_active":"2024-09-10T18:59:25.265150+0000","last_became_peered":"2024-09-10T18:59:25.265150+0000","last_unstale":"2024-09-10T18:59:36.887498+0000","last_undegraded":"2024-09-10T18:59:36.887498+0000","last_fullsized":"2024-09-10T18:59:36.887498+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.082118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"209.18","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245284+0000","last_change":"2024-09-10T18:59:25.261749+0000","last_active":"2024-09-10T18:59:36.245284+0000","last_peered":"2024-09-10T18:59:36.245284+0000","last_clean":"2024-09-10T18:59:36.245284+0000","last_became_active":"2024-09-10T18:59:25.261319+0000","last_became_peered":"2024-09-10T18:59:25.261319+0000","last_unstale":"2024-09-10T18:59:36.245284+0000","last_undegraded":"2024-09-10T18:59:36.245284+0000","last_fullsized":"2024-09-10T18:59:36.245284+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:51.233913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"209.17","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243408+0000","last_change":"2024-09-10T18:59:25.268053+0000","last_active":"2024-09-10T18:59:36.243408+0000","last_peered":"2024-09-10T18:59:36.243408+0000","last_clean":"2024-09-10T18:59:36.243408+0000","last_became_active":"2024-09-10T18:59:25.267739+0000","last_became_peered":"2024-09-10T18:59:25.267739+0000","last_unstale":"2024-09-10T18:59:36.243408+0000","last_undegraded":"2024-09-10T18:59:36.243408+0000","last_fullsized":"2024-09-10T18:59:36.243408+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:49.340117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"209.16","version":"0'0","reported_seq":19,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928909+0000","last_change":"2024-09-10T18:59:25.261916+0000","last_active":"2024-09-10T18:59:36.928909+0000","last_peered":"2024-09-10T18:59:36.928909+0000","last_clean":"2024-09-10T18:59:36.928909+0000","last_became_active":"2024-09-10T18:59:25.261616+0000","last_became_peered":"2024-09-10T18:59:25.261616+0000","last_unstale":"2024-09-10T18:59:36.928909+0000","last_undegraded":"2024-09-10T18:59:36.928909+0000","last_fullsized":"2024-09-10T18:59:36.928909+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.631774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"209.15","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923751+0000","last_change":"2024-09-10T18:59:25.266308+0000","last_active":"2024-09-10T18:59:36.923751+0000","last_peered":"2024-09-10T18:59:36.923751+0000","last_clean":"2024-09-10T18:59:36.923751+0000","last_became_active":"2024-09-10T18:59:25.265866+0000","last_became_peered":"2024-09-10T18:59:25.265866+0000","last_unstale":"2024-09-10T18:59:36.923751+0000","last_undegraded":"2024-09-10T18:59:36.923751+0000","last_fullsized":"2024-09-10T18:59:36.923751+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:40.485415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"209.14","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926468+0000","last_change":"2024-09-10T18:59:25.122524+0000","last_active":"2024-09-10T18:59:36.926468+0000","last_peered":"2024-09-10T18:59:36.926468+0000","last_clean":"2024-09-10T18:59:36.926468+0000","last_became_active":"2024-09-10T18:59:25.122186+0000","last_became_peered":"2024-09-10T18:59:25.122186+0000","last_unstale":"2024-09-10T18:59:36.926468+0000","last_undegraded":"2024-09-10T18:59:36.926468+0000","last_fullsized":"2024-09-10T18:59:36.926468+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.147364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"209.13","version":"412'2","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242014+0000","last_change":"2024-09-10T18:59:25.260268+0000","last_active":"2024-09-10T18:59:36.242014+0000","last_peered":"2024-09-10T18:59:36.242014+0000","last_clean":"2024-09-10T18:59:36.242014+0000","last_became_active":"2024-09-10T18:59:25.259923+0000","last_became_peered":"2024-09-10T18:59:25.259923+0000","last_unstale":"2024-09-10T18:59:36.242014+0000","last_undegraded":"2024-09-10T18:59:36.242014+0000","last_fullsized":"2024-09-10T18:59:36.242014+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:35.371868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"209.12","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245758+0000","last_change":"2024-09-10T18:59:25.262429+0000","last_active":"2024-09-10T18:59:36.245758+0000","last_peered":"2024-09-10T18:59:36.245758+0000","last_clean":"2024-09-10T18:59:36.245758+0000","last_became_active":"2024-09-10T18:59:25.262125+0000","last_became_peered":"2024-09-10T18:59:25.262125+0000","last_unstale":"2024-09-10T18:59:36.245758+0000","last_undegraded":"2024-09-10T18:59:36.245758+0000","last_fullsized":"2024-09-10T18:59:36.245758+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:48.604613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"209.11","version":"0'0","reported_seq":19,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927720+0000","last_change":"2024-09-10T18:59:25.126804+0000","last_active":"2024-09-10T18:59:36.927720+0000","last_peered":"2024-09-10T18:59:36.927720+0000","last_clean":"2024-09-10T18:59:36.927720+0000","last_became_active":"2024-09-10T18:59:25.126172+0000","last_became_peered":"2024-09-10T18:59:25.126172+0000","last_unstale":"2024-09-10T18:59:36.927720+0000","last_undegraded":"2024-09-10T18:59:36.927720+0000","last_fullsized":"2024-09-10T18:59:36.927720+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:37.466871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"209.10","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.886570+0000","last_change":"2024-09-10T18:59:25.262680+0000","last_active":"2024-09-10T18:59:36.886570+0000","last_peered":"2024-09-10T18:59:36.886570+0000","last_clean":"2024-09-10T18:59:36.886570+0000","last_became_active":"2024-09-10T18:59:25.262371+0000","last_became_peered":"2024-09-10T18:59:25.262371+0000","last_unstale":"2024-09-10T18:59:36.886570+0000","last_undegraded":"2024-09-10T18:59:36.886570+0000","last_fullsized":"2024-09-10T18:59:36.886570+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:41.770007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"209.f","version":"0'0","reported_seq":19,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929435+0000","last_change":"2024-09-10T18:59:25.260094+0000","last_active":"2024-09-10T18:59:36.929435+0000","last_peered":"2024-09-10T18:59:36.929435+0000","last_clean":"2024-09-10T18:59:36.929435+0000","last_became_active":"2024-09-10T18:59:25.259601+0000","last_became_peered":"2024-09-10T18:59:25.259601+0000","last_unstale":"2024-09-10T18:59:36.929435+0000","last_undegraded":"2024-09-10T18:59:36.929435+0000","last_fullsized":"2024-09-10T18:59:36.929435+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.479929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"209.e","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927318+0000","last_change":"2024-09-10T18:59:25.257730+0000","last_active":"2024-09-10T18:59:36.927318+0000","last_peered":"2024-09-10T18:59:36.927318+0000","last_clean":"2024-09-10T18:59:36.927318+0000","last_became_active":"2024-09-10T18:59:25.257222+0000","last_became_peered":"2024-09-10T18:59:25.257222+0000","last_unstale":"2024-09-10T18:59:36.927318+0000","last_undegraded":"2024-09-10T18:59:36.927318+0000","last_fullsized":"2024-09-10T18:59:36.927318+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.984103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"209.d","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242861+0000","last_change":"2024-09-10T18:59:25.268763+0000","last_active":"2024-09-10T18:59:36.242861+0000","last_peered":"2024-09-10T18:59:36.242861+0000","last_clean":"2024-09-10T18:59:36.242861+0000","last_became_active":"2024-09-10T18:59:25.268453+0000","last_became_peered":"2024-09-10T18:59:25.268453+0000","last_unstale":"2024-09-10T18:59:36.242861+0000","last_undegraded":"2024-09-10T18:59:36.242861+0000","last_fullsized":"2024-09-10T18:59:36.242861+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:27.942423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"209.c","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926525+0000","last_change":"2024-09-10T18:59:25.259765+0000","last_active":"2024-09-10T18:59:36.926525+0000","last_peered":"2024-09-10T18:59:36.926525+0000","last_clean":"2024-09-10T18:59:36.926525+0000","last_became_active":"2024-09-10T18:59:25.259401+0000","last_became_peered":"2024-09-10T18:59:25.259401+0000","last_unstale":"2024-09-10T18:59:36.926525+0000","last_undegraded":"2024-09-10T18:59:36.926525+0000","last_fullsized":"2024-09-10T18:59:36.926525+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:50.272500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"209.b","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926265+0000","last_change":"2024-09-10T18:59:25.263467+0000","last_active":"2024-09-10T18:59:36.926265+0000","last_peered":"2024-09-10T18:59:36.926265+0000","last_clean":"2024-09-10T18:59:36.926265+0000","last_became_active":"2024-09-10T18:59:25.262673+0000","last_became_peered":"2024-09-10T18:59:25.262673+0000","last_unstale":"2024-09-10T18:59:36.926265+0000","last_undegraded":"2024-09-10T18:59:36.926265+0000","last_fullsized":"2024-09-10T18:59:36.926265+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:52.544932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"209.a","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888215+0000","last_change":"2024-09-10T18:59:25.270711+0000","last_active":"2024-09-10T18:59:36.888215+0000","last_peered":"2024-09-10T18:59:36.888215+0000","last_clean":"2024-09-10T18:59:36.888215+0000","last_became_active":"2024-09-10T18:59:25.270115+0000","last_became_peered":"2024-09-10T18:59:25.270115+0000","last_unstale":"2024-09-10T18:59:36.888215+0000","last_undegraded":"2024-09-10T18:59:36.888215+0000","last_fullsized":"2024-09-10T18:59:36.888215+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:38.257203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"209.9","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240869+0000","last_change":"2024-09-10T18:59:25.267309+0000","last_active":"2024-09-10T18:59:36.240869+0000","last_peered":"2024-09-10T18:59:36.240869+0000","last_clean":"2024-09-10T18:59:36.240869+0000","last_became_active":"2024-09-10T18:59:25.263562+0000","last_became_peered":"2024-09-10T18:59:25.263562+0000","last_unstale":"2024-09-10T18:59:36.240869+0000","last_undegraded":"2024-09-10T18:59:36.240869+0000","last_fullsized":"2024-09-10T18:59:36.240869+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:47.100335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"209.8","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923150+0000","last_change":"2024-09-10T18:59:25.272190+0000","last_active":"2024-09-10T18:59:36.923150+0000","last_peered":"2024-09-10T18:59:36.923150+0000","last_clean":"2024-09-10T18:59:36.923150+0000","last_became_active":"2024-09-10T18:59:25.271898+0000","last_became_peered":"2024-09-10T18:59:25.271898+0000","last_unstale":"2024-09-10T18:59:36.923150+0000","last_undegraded":"2024-09-10T18:59:36.923150+0000","last_fullsized":"2024-09-10T18:59:36.923150+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:36.069805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"209.3","version":"412'2","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924746+0000","last_change":"2024-09-10T18:59:25.258972+0000","last_active":"2024-09-10T18:59:36.924746+0000","last_peered":"2024-09-10T18:59:36.924746+0000","last_clean":"2024-09-10T18:59:36.924746+0000","last_became_active":"2024-09-10T18:59:25.258656+0000","last_became_peered":"2024-09-10T18:59:25.258656+0000","last_unstale":"2024-09-10T18:59:36.924746+0000","last_undegraded":"2024-09-10T18:59:36.924746+0000","last_fullsized":"2024-09-10T18:59:36.924746+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:37.587873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"209.2","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922969+0000","last_change":"2024-09-10T18:59:25.267816+0000","last_active":"2024-09-10T18:59:36.922969+0000","last_peered":"2024-09-10T18:59:36.922969+0000","last_clean":"2024-09-10T18:59:36.922969+0000","last_became_active":"2024-09-10T18:59:25.266916+0000","last_became_peered":"2024-09-10T18:59:25.266916+0000","last_unstale":"2024-09-10T18:59:36.922969+0000","last_undegraded":"2024-09-10T18:59:36.922969+0000","last_fullsized":"2024-09-10T18:59:36.922969+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.977669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"209.1","version":"417'2","reported_seq":22,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928184+0000","last_change":"2024-09-10T18:59:25.259671+0000","last_active":"2024-09-10T18:59:36.928184+0000","last_peered":"2024-09-10T18:59:36.928184+0000","last_clean":"2024-09-10T18:59:36.928184+0000","last_became_active":"2024-09-10T18:59:25.257290+0000","last_became_peered":"2024-09-10T18:59:25.257290+0000","last_unstale":"2024-09-10T18:59:36.928184+0000","last_undegraded":"2024-09-10T18:59:36.928184+0000","last_fullsized":"2024-09-10T18:59:36.928184+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:52.011868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"209.0","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245285+0000","last_change":"2024-09-10T18:59:25.272385+0000","last_active":"2024-09-10T18:59:36.245285+0000","last_peered":"2024-09-10T18:59:36.245285+0000","last_clean":"2024-09-10T18:59:36.245285+0000","last_became_active":"2024-09-10T18:59:25.272099+0000","last_became_peered":"2024-09-10T18:59:25.272099+0000","last_unstale":"2024-09-10T18:59:36.245285+0000","last_undegraded":"2024-09-10T18:59:36.245285+0000","last_fullsized":"2024-09-10T18:59:36.245285+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:41.535432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"209.7","version":"417'2","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925079+0000","last_change":"2024-09-10T18:59:25.253121+0000","last_active":"2024-09-10T18:59:36.925079+0000","last_peered":"2024-09-10T18:59:36.925079+0000","last_clean":"2024-09-10T18:59:36.925079+0000","last_became_active":"2024-09-10T18:59:25.252168+0000","last_became_peered":"2024-09-10T18:59:25.252168+0000","last_unstale":"2024-09-10T18:59:36.925079+0000","last_undegraded":"2024-09-10T18:59:36.925079+0000","last_fullsized":"2024-09-10T18:59:36.925079+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:46.021292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"209.6","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925603+0000","last_change":"2024-09-10T18:59:25.265729+0000","last_active":"2024-09-10T18:59:36.925603+0000","last_peered":"2024-09-10T18:59:36.925603+0000","last_clean":"2024-09-10T18:59:36.925603+0000","last_became_active":"2024-09-10T18:59:25.265196+0000","last_became_peered":"2024-09-10T18:59:25.265196+0000","last_unstale":"2024-09-10T18:59:36.925603+0000","last_undegraded":"2024-09-10T18:59:36.925603+0000","last_fullsized":"2024-09-10T18:59:36.925603+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:38.722356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"209.5","version":"0'0","reported_seq":19,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928965+0000","last_change":"2024-09-10T18:59:25.261663+0000","last_active":"2024-09-10T18:59:36.928965+0000","last_peered":"2024-09-10T18:59:36.928965+0000","last_clean":"2024-09-10T18:59:36.928965+0000","last_became_active":"2024-09-10T18:59:25.261169+0000","last_became_peered":"2024-09-10T18:59:25.261169+0000","last_unstale":"2024-09-10T18:59:36.928965+0000","last_undegraded":"2024-09-10T18:59:36.928965+0000","last_fullsized":"2024-09-10T18:59:36.928965+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.978652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"209.4","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888362+0000","last_change":"2024-09-10T18:59:25.271875+0000","last_active":"2024-09-10T18:59:36.888362+0000","last_peered":"2024-09-10T18:59:36.888362+0000","last_clean":"2024-09-10T18:59:36.888362+0000","last_became_active":"2024-09-10T18:59:25.271315+0000","last_became_peered":"2024-09-10T18:59:25.271315+0000","last_unstale":"2024-09-10T18:59:36.888362+0000","last_undegraded":"2024-09-10T18:59:36.888362+0000","last_fullsized":"2024-09-10T18:59:36.888362+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":408,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:24.100401+0000","last_clean_scrub_stamp":"2024-09-10T18:59:24.100401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:53.727848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":379,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.886490+0000","last_change":"2024-09-10T18:58:52.881919+0000","last_active":"2024-09-10T18:59:36.886490+0000","last_peered":"2024-09-10T18:59:36.886490+0000","last_clean":"2024-09-10T18:59:36.886490+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T18:59:36.886490+0000","last_undegraded":"2024-09-10T18:59:36.886490+0000","last_fullsized":"2024-09-10T18:59:36.886490+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.087038+0000","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"377'67","reported_seq":409,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.886362+0000","last_change":"2024-09-10T18:58:39.857881+0000","last_active":"2024-09-10T18:59:36.886362+0000","last_peered":"2024-09-10T18:59:36.886362+0000","last_clean":"2024-09-10T18:59:36.886362+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T18:59:36.886362+0000","last_undegraded":"2024-09-10T18:59:36.886362+0000","last_fullsized":"2024-09-10T18:59:36.886362+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.880941+0000","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.016940303,"stat_sum":{"num_bytes":64,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"377'23","reported_seq":373,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926613+0000","last_change":"2024-09-10T18:57:54.056252+0000","last_active":"2024-09-10T18:59:36.926613+0000","last_peered":"2024-09-10T18:59:36.926613+0000","last_clean":"2024-09-10T18:59:36.926613+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T18:59:36.926613+0000","last_undegraded":"2024-09-10T18:59:36.926613+0000","last_fullsized":"2024-09-10T18:59:36.926613+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'22","last_scrub_stamp":"2024-09-10T18:57:26.499083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:26.499083+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045209559999999996,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242095+0000","last_change":"2024-09-10T18:58:54.847481+0000","last_active":"2024-09-10T18:59:36.242095+0000","last_peered":"2024-09-10T18:59:36.242095+0000","last_clean":"2024-09-10T18:59:36.242095+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T18:59:36.242095+0000","last_undegraded":"2024-09-10T18:59:36.242095+0000","last_fullsized":"2024-09-10T18:59:36.242095+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:58:54.847269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.847269+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.700907+0000","scrub_duration":0.0091152980000000008,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":410,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242141+0000","last_change":"2024-09-10T18:59:02.768985+0000","last_active":"2024-09-10T18:59:36.242141+0000","last_peered":"2024-09-10T18:59:36.242141+0000","last_clean":"2024-09-10T18:59:36.242141+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T18:59:36.242141+0000","last_undegraded":"2024-09-10T18:59:36.242141+0000","last_fullsized":"2024-09-10T18:59:36.242141+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.764846+0000","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":376,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.921746+0000","last_change":"2024-09-10T18:57:54.056894+0000","last_active":"2024-09-10T18:59:36.921746+0000","last_peered":"2024-09-10T18:59:36.921746+0000","last_clean":"2024-09-10T18:59:36.921746+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T18:59:36.921746+0000","last_undegraded":"2024-09-10T18:59:36.921746+0000","last_fullsized":"2024-09-10T18:59:36.921746+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:57:35.352719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.352719+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074091110000000003,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"391'24","reported_seq":374,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245442+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T18:59:36.245442+0000","last_peered":"2024-09-10T18:59:36.245442+0000","last_clean":"2024-09-10T18:59:36.245442+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T18:59:36.245442+0000","last_undegraded":"2024-09-10T18:59:36.245442+0000","last_fullsized":"2024-09-10T18:59:36.245442+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"254'104","reported_seq":430,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.237721+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T18:59:36.237721+0000","last_peered":"2024-09-10T18:59:36.237721+0000","last_clean":"2024-09-10T18:59:36.237721+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T18:59:36.237721+0000","last_undegraded":"2024-09-10T18:59:36.237721+0000","last_fullsized":"2024-09-10T18:59:36.237721+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"377'19","reported_seq":362,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.920578+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T18:59:36.920578+0000","last_peered":"2024-09-10T18:59:36.920578+0000","last_clean":"2024-09-10T18:59:36.920578+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T18:59:36.920578+0000","last_undegraded":"2024-09-10T18:59:36.920578+0000","last_fullsized":"2024-09-10T18:59:36.920578+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"387'24","reported_seq":379,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243258+0000","last_change":"2024-09-10T18:59:03.724989+0000","last_active":"2024-09-10T18:59:36.243258+0000","last_peered":"2024-09-10T18:59:36.243258+0000","last_clean":"2024-09-10T18:59:36.243258+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T18:59:36.243258+0000","last_undegraded":"2024-09-10T18:59:36.243258+0000","last_fullsized":"2024-09-10T18:59:36.243258+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.707234+0000","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"377'75","reported_seq":420,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922879+0000","last_change":"2024-09-10T18:59:01.892397+0000","last_active":"2024-09-10T18:59:36.922879+0000","last_peered":"2024-09-10T18:59:36.922879+0000","last_clean":"2024-09-10T18:59:36.922879+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T18:59:36.922879+0000","last_undegraded":"2024-09-10T18:59:36.922879+0000","last_fullsized":"2024-09-10T18:59:36.922879+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'75","last_scrub_stamp":"2024-09-10T18:59:01.892227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.892227+0000","objects_scrubbed":7,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.595320+0000","scrub_duration":0.0063234340000000002,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":60,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"377'97","reported_seq":448,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242125+0000","last_change":"2024-09-10T18:59:01.793871+0000","last_active":"2024-09-10T18:59:36.242125+0000","last_peered":"2024-09-10T18:59:36.242125+0000","last_clean":"2024-09-10T18:59:36.242125+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T18:59:36.242125+0000","last_undegraded":"2024-09-10T18:59:36.242125+0000","last_fullsized":"2024-09-10T18:59:36.242125+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.183490+0000","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":76,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":31,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"409'2","reported_seq":363,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924640+0000","last_change":"2024-09-10T18:58:43.994005+0000","last_active":"2024-09-10T18:59:36.924640+0000","last_peered":"2024-09-10T18:59:36.924640+0000","last_clean":"2024-09-10T18:59:36.924640+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T18:59:36.924640+0000","last_undegraded":"2024-09-10T18:59:36.924640+0000","last_fullsized":"2024-09-10T18:59:36.924640+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:58:43.993832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.993832+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.350619+0000","scrub_duration":0.0040278960000000004,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":382,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238038+0000","last_change":"2024-09-10T18:58:39.380635+0000","last_active":"2024-09-10T18:59:36.238038+0000","last_peered":"2024-09-10T18:59:36.238038+0000","last_clean":"2024-09-10T18:59:36.238038+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T18:59:36.238038+0000","last_undegraded":"2024-09-10T18:59:36.238038+0000","last_fullsized":"2024-09-10T18:59:36.238038+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.953708+0000","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"297'52","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925696+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T18:59:36.925696+0000","last_peered":"2024-09-10T18:59:36.925696+0000","last_clean":"2024-09-10T18:59:36.925696+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T18:59:36.925696+0000","last_undegraded":"2024-09-10T18:59:36.925696+0000","last_fullsized":"2024-09-10T18:59:36.925696+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":375,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245428+0000","last_change":"2024-09-10T18:58:37.312137+0000","last_active":"2024-09-10T18:59:36.245428+0000","last_peered":"2024-09-10T18:59:36.245428+0000","last_clean":"2024-09-10T18:59:36.245428+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T18:59:36.245428+0000","last_undegraded":"2024-09-10T18:59:36.245428+0000","last_fullsized":"2024-09-10T18:59:36.245428+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.542659+0000","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"326'68","reported_seq":407,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240797+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T18:59:36.240797+0000","last_peered":"2024-09-10T18:59:36.240797+0000","last_clean":"2024-09-10T18:59:36.240797+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T18:59:36.240797+0000","last_undegraded":"2024-09-10T18:59:36.240797+0000","last_fullsized":"2024-09-10T18:59:36.240797+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"246'119","reported_seq":445,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.920495+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T18:59:36.920495+0000","last_peered":"2024-09-10T18:59:36.920495+0000","last_clean":"2024-09-10T18:59:36.920495+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T18:59:36.920495+0000","last_undegraded":"2024-09-10T18:59:36.920495+0000","last_fullsized":"2024-09-10T18:59:36.920495+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"377'20","reported_seq":378,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927154+0000","last_change":"2024-09-10T18:58:56.450425+0000","last_active":"2024-09-10T18:59:36.927154+0000","last_peered":"2024-09-10T18:59:36.927154+0000","last_clean":"2024-09-10T18:59:36.927154+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T18:59:36.927154+0000","last_undegraded":"2024-09-10T18:59:36.927154+0000","last_fullsized":"2024-09-10T18:59:36.927154+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:25.313261+0000","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"377'23","reported_seq":369,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238493+0000","last_change":"2024-09-10T18:58:47.475482+0000","last_active":"2024-09-10T18:59:36.238493+0000","last_peered":"2024-09-10T18:59:36.238493+0000","last_clean":"2024-09-10T18:59:36.238493+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T18:59:36.238493+0000","last_undegraded":"2024-09-10T18:59:36.238493+0000","last_fullsized":"2024-09-10T18:59:36.238493+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'22","last_scrub_stamp":"2024-09-10T18:58:47.475285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.475285+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.029410+0000","scrub_duration":0.0041187790000000004,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":377,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242171+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T18:59:36.242171+0000","last_peered":"2024-09-10T18:59:36.242171+0000","last_clean":"2024-09-10T18:59:36.242171+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T18:59:36.242171+0000","last_undegraded":"2024-09-10T18:59:36.242171+0000","last_fullsized":"2024-09-10T18:59:36.242171+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"232'17","reported_seq":375,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923970+0000","last_change":"2024-09-10T18:58:37.063649+0000","last_active":"2024-09-10T18:59:36.923970+0000","last_peered":"2024-09-10T18:59:36.923970+0000","last_clean":"2024-09-10T18:59:36.923970+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T18:59:36.923970+0000","last_undegraded":"2024-09-10T18:59:36.923970+0000","last_fullsized":"2024-09-10T18:59:36.923970+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:58:37.063474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.063474+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.880086+0000","scrub_duration":0.003955554,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":358,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922720+0000","last_change":"2024-09-10T18:59:03.924359+0000","last_active":"2024-09-10T18:59:36.922720+0000","last_peered":"2024-09-10T18:59:36.922720+0000","last_clean":"2024-09-10T18:59:36.922720+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T18:59:36.922720+0000","last_undegraded":"2024-09-10T18:59:36.922720+0000","last_fullsized":"2024-09-10T18:59:36.922720+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.924176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.924176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:06.750994+0000","scrub_duration":0.003004348,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.920170+0000","last_change":"2024-09-10T18:57:54.066122+0000","last_active":"2024-09-10T18:59:36.920170+0000","last_peered":"2024-09-10T18:59:36.920170+0000","last_clean":"2024-09-10T18:59:36.920170+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T18:59:36.920170+0000","last_undegraded":"2024-09-10T18:59:36.920170+0000","last_fullsized":"2024-09-10T18:59:36.920170+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T18:57:35.624392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.624392+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004362795,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"409'160","reported_seq":507,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923327+0000","last_change":"2024-09-10T18:58:53.954029+0000","last_active":"2024-09-10T18:59:36.923327+0000","last_peered":"2024-09-10T18:59:36.923327+0000","last_clean":"2024-09-10T18:59:36.923327+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T18:59:36.923327+0000","last_undegraded":"2024-09-10T18:59:36.923327+0000","last_fullsized":"2024-09-10T18:59:36.923327+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'159","last_scrub_stamp":"2024-09-10T18:58:53.953850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.953850+0000","objects_scrubbed":11,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.670681+0000","scrub_duration":0.0076573049999999997,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"388'99","reported_seq":494,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.921902+0000","last_change":"2024-09-10T18:59:04.213412+0000","last_active":"2024-09-10T18:59:36.921902+0000","last_peered":"2024-09-10T18:59:36.921902+0000","last_clean":"2024-09-10T18:59:36.921902+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T18:59:36.921902+0000","last_undegraded":"2024-09-10T18:59:36.921902+0000","last_fullsized":"2024-09-10T18:59:36.921902+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.213299+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.577134+0000","scrub_duration":0.022641919,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240100+0000","last_change":"2024-09-10T18:58:44.435895+0000","last_active":"2024-09-10T18:59:36.240100+0000","last_peered":"2024-09-10T18:59:36.240100+0000","last_clean":"2024-09-10T18:59:36.240100+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T18:59:36.240100+0000","last_undegraded":"2024-09-10T18:59:36.240100+0000","last_fullsized":"2024-09-10T18:59:36.240100+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T18:58:44.435728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.435728+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.065189+0000","scrub_duration":0.0045739450000000003,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"257'107","reported_seq":439,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.921742+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T18:59:36.921742+0000","last_peered":"2024-09-10T18:59:36.921742+0000","last_clean":"2024-09-10T18:59:36.921742+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T18:59:36.921742+0000","last_undegraded":"2024-09-10T18:59:36.921742+0000","last_fullsized":"2024-09-10T18:59:36.921742+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"409'104","reported_seq":439,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246685+0000","last_change":"2024-09-10T18:58:34.265530+0000","last_active":"2024-09-10T18:59:36.246685+0000","last_peered":"2024-09-10T18:59:36.246685+0000","last_clean":"2024-09-10T18:59:36.246685+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T18:59:36.246685+0000","last_undegraded":"2024-09-10T18:59:36.246685+0000","last_fullsized":"2024-09-10T18:59:36.246685+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:50.420507+0000","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":373,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.920318+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T18:59:36.920318+0000","last_peered":"2024-09-10T18:59:36.920318+0000","last_clean":"2024-09-10T18:59:36.920318+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T18:59:36.920318+0000","last_undegraded":"2024-09-10T18:59:36.920318+0000","last_fullsized":"2024-09-10T18:59:36.920318+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"409'74","reported_seq":419,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238165+0000","last_change":"2024-09-10T18:58:32.368725+0000","last_active":"2024-09-10T18:59:36.238165+0000","last_peered":"2024-09-10T18:59:36.238165+0000","last_clean":"2024-09-10T18:59:36.238165+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T18:59:36.238165+0000","last_undegraded":"2024-09-10T18:59:36.238165+0000","last_fullsized":"2024-09-10T18:59:36.238165+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'72","last_scrub_stamp":"2024-09-10T18:58:32.368604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.368604+0000","objects_scrubbed":7,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:57.712897+0000","scrub_duration":0.0042920229999999998,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":440,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922719+0000","last_change":"2024-09-10T18:58:36.017962+0000","last_active":"2024-09-10T18:59:36.922719+0000","last_peered":"2024-09-10T18:59:36.922719+0000","last_clean":"2024-09-10T18:59:36.922719+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T18:59:36.922719+0000","last_undegraded":"2024-09-10T18:59:36.922719+0000","last_fullsized":"2024-09-10T18:59:36.922719+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:58:36.017841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.017841+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.872443+0000","scrub_duration":0.0042543119999999997,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246463+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T18:59:36.246463+0000","last_peered":"2024-09-10T18:59:36.246463+0000","last_clean":"2024-09-10T18:59:36.246463+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T18:59:36.246463+0000","last_undegraded":"2024-09-10T18:59:36.246463+0000","last_fullsized":"2024-09-10T18:59:36.246463+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.261172+0000","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889911+0000","last_change":"2024-09-10T18:56:10.206639+0000","last_active":"2024-09-10T18:59:36.889911+0000","last_peered":"2024-09-10T18:59:36.889911+0000","last_clean":"2024-09-10T18:59:36.889911+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T18:59:36.889911+0000","last_undegraded":"2024-09-10T18:59:36.889911+0000","last_fullsized":"2024-09-10T18:59:36.889911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:10.206481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:10.206481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034783700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":417,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247261+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T18:59:36.247261+0000","last_peered":"2024-09-10T18:59:36.247261+0000","last_clean":"2024-09-10T18:59:36.247261+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T18:59:36.247261+0000","last_undegraded":"2024-09-10T18:59:36.247261+0000","last_fullsized":"2024-09-10T18:59:36.247261+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246267+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T18:59:36.246267+0000","last_peered":"2024-09-10T18:59:36.246267+0000","last_clean":"2024-09-10T18:59:36.246267+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T18:59:36.246267+0000","last_undegraded":"2024-09-10T18:59:36.246267+0000","last_fullsized":"2024-09-10T18:59:36.246267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:04.994389+0000","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":310,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.237670+0000","last_change":"2024-09-10T18:56:38.295129+0000","last_active":"2024-09-10T18:59:36.237670+0000","last_peered":"2024-09-10T18:59:36.237670+0000","last_clean":"2024-09-10T18:59:36.237670+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T18:59:36.237670+0000","last_undegraded":"2024-09-10T18:59:36.237670+0000","last_fullsized":"2024-09-10T18:59:36.237670+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036585929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":409,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247674+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T18:59:36.247674+0000","last_peered":"2024-09-10T18:59:36.247674+0000","last_clean":"2024-09-10T18:59:36.247674+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T18:59:36.247674+0000","last_undegraded":"2024-09-10T18:59:36.247674+0000","last_fullsized":"2024-09-10T18:59:36.247674+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.048099+0000","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":426,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889717+0000","last_change":"2024-09-10T18:56:15.245471+0000","last_active":"2024-09-10T18:59:36.889717+0000","last_peered":"2024-09-10T18:59:36.889717+0000","last_clean":"2024-09-10T18:59:36.889717+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T18:59:36.889717+0000","last_undegraded":"2024-09-10T18:59:36.889717+0000","last_fullsized":"2024-09-10T18:59:36.889717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:15.245307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:15.245307+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045321099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926990+0000","last_change":"2024-09-10T18:56:40.956490+0000","last_active":"2024-09-10T18:59:36.926990+0000","last_peered":"2024-09-10T18:59:36.926990+0000","last_clean":"2024-09-10T18:59:36.926990+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T18:59:36.926990+0000","last_undegraded":"2024-09-10T18:59:36.926990+0000","last_fullsized":"2024-09-10T18:59:36.926990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:40.956331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:40.956331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037999560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928314+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T18:59:36.928314+0000","last_peered":"2024-09-10T18:59:36.928314+0000","last_clean":"2024-09-10T18:59:36.928314+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T18:59:36.928314+0000","last_undegraded":"2024-09-10T18:59:36.928314+0000","last_fullsized":"2024-09-10T18:59:36.928314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:46.825797+0000","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887857+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T18:59:36.887857+0000","last_peered":"2024-09-10T18:59:36.887857+0000","last_clean":"2024-09-10T18:59:36.887857+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T18:59:36.887857+0000","last_undegraded":"2024-09-10T18:59:36.887857+0000","last_fullsized":"2024-09-10T18:59:36.887857+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926754+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T18:59:36.926754+0000","last_peered":"2024-09-10T18:59:36.926754+0000","last_clean":"2024-09-10T18:59:36.926754+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T18:59:36.926754+0000","last_undegraded":"2024-09-10T18:59:36.926754+0000","last_fullsized":"2024-09-10T18:59:36.926754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.053990+0000","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":441,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239053+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T18:59:36.239053+0000","last_peered":"2024-09-10T18:59:36.239053+0000","last_clean":"2024-09-10T18:59:36.239053+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T18:59:36.239053+0000","last_undegraded":"2024-09-10T18:59:36.239053+0000","last_fullsized":"2024-09-10T18:59:36.239053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928476+0000","last_change":"2024-09-10T18:58:53.495436+0000","last_active":"2024-09-10T18:59:36.928476+0000","last_peered":"2024-09-10T18:59:36.928476+0000","last_clean":"2024-09-10T18:59:36.928476+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T18:59:36.928476+0000","last_undegraded":"2024-09-10T18:59:36.928476+0000","last_fullsized":"2024-09-10T18:59:36.928476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.495273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.495273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.149497+0000","scrub_duration":0.0024582229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.237903+0000","last_change":"2024-09-10T18:57:31.277231+0000","last_active":"2024-09-10T18:59:36.237903+0000","last_peered":"2024-09-10T18:59:36.237903+0000","last_clean":"2024-09-10T18:59:36.237903+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T18:59:36.237903+0000","last_undegraded":"2024-09-10T18:59:36.237903+0000","last_fullsized":"2024-09-10T18:59:36.237903+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.277085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.277085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003344746,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":322,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242375+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T18:59:36.242375+0000","last_peered":"2024-09-10T18:59:36.242375+0000","last_clean":"2024-09-10T18:59:36.242375+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T18:59:36.242375+0000","last_undegraded":"2024-09-10T18:59:36.242375+0000","last_fullsized":"2024-09-10T18:59:36.242375+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:59.115727+0000","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":408,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927321+0000","last_change":"2024-09-10T18:59:03.215137+0000","last_active":"2024-09-10T18:59:36.927321+0000","last_peered":"2024-09-10T18:59:36.927321+0000","last_clean":"2024-09-10T18:59:36.927321+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T18:59:36.927321+0000","last_undegraded":"2024-09-10T18:59:36.927321+0000","last_fullsized":"2024-09-10T18:59:36.927321+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.215039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.215039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:32.858523+0000","scrub_duration":0.0064108910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":448,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239673+0000","last_change":"2024-09-10T18:59:05.516222+0000","last_active":"2024-09-10T18:59:36.239673+0000","last_peered":"2024-09-10T18:59:36.239673+0000","last_clean":"2024-09-10T18:59:36.239673+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T18:59:36.239673+0000","last_undegraded":"2024-09-10T18:59:36.239673+0000","last_fullsized":"2024-09-10T18:59:36.239673+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:59:05.516062+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:05.516062+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.578286+0000","scrub_duration":0.0058657070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928413+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T18:59:36.928413+0000","last_peered":"2024-09-10T18:59:36.928413+0000","last_clean":"2024-09-10T18:59:36.928413+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T18:59:36.928413+0000","last_undegraded":"2024-09-10T18:59:36.928413+0000","last_fullsized":"2024-09-10T18:59:36.928413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":401,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.886995+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T18:59:36.886995+0000","last_peered":"2024-09-10T18:59:36.886995+0000","last_clean":"2024-09-10T18:59:36.886995+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T18:59:36.886995+0000","last_undegraded":"2024-09-10T18:59:36.886995+0000","last_fullsized":"2024-09-10T18:59:36.886995+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.122348+0000","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":314,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246011+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T18:59:36.246011+0000","last_peered":"2024-09-10T18:59:36.246011+0000","last_clean":"2024-09-10T18:59:36.246011+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T18:59:36.246011+0000","last_undegraded":"2024-09-10T18:59:36.246011+0000","last_fullsized":"2024-09-10T18:59:36.246011+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":459,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923927+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T18:59:36.923927+0000","last_peered":"2024-09-10T18:59:36.923927+0000","last_clean":"2024-09-10T18:59:36.923927+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T18:59:36.923927+0000","last_undegraded":"2024-09-10T18:59:36.923927+0000","last_fullsized":"2024-09-10T18:59:36.923927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927048+0000","last_change":"2024-09-10T18:57:34.379150+0000","last_active":"2024-09-10T18:59:36.927048+0000","last_peered":"2024-09-10T18:59:36.927048+0000","last_clean":"2024-09-10T18:59:36.927048+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T18:59:36.927048+0000","last_undegraded":"2024-09-10T18:59:36.927048+0000","last_fullsized":"2024-09-10T18:59:36.927048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.378985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.378985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00364804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":539,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929219+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T18:59:36.929219+0000","last_peered":"2024-09-10T18:59:36.929219+0000","last_clean":"2024-09-10T18:59:36.929219+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T18:59:36.929219+0000","last_undegraded":"2024-09-10T18:59:36.929219+0000","last_fullsized":"2024-09-10T18:59:36.929219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.009195+0000","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":411,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926633+0000","last_change":"2024-09-10T18:59:00.907296+0000","last_active":"2024-09-10T18:59:36.926633+0000","last_peered":"2024-09-10T18:59:36.926633+0000","last_clean":"2024-09-10T18:59:36.926633+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T18:59:36.926633+0000","last_undegraded":"2024-09-10T18:59:36.926633+0000","last_fullsized":"2024-09-10T18:59:36.926633+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.907056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.861464+0000","scrub_duration":0.0036581080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":313,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923715+0000","last_change":"2024-09-10T18:56:38.781327+0000","last_active":"2024-09-10T18:59:36.923715+0000","last_peered":"2024-09-10T18:59:36.923715+0000","last_clean":"2024-09-10T18:59:36.923715+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T18:59:36.923715+0000","last_undegraded":"2024-09-10T18:59:36.923715+0000","last_fullsized":"2024-09-10T18:59:36.923715+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003131722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":426,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246767+0000","last_change":"2024-09-10T18:56:29.927646+0000","last_active":"2024-09-10T18:59:36.246767+0000","last_peered":"2024-09-10T18:59:36.246767+0000","last_clean":"2024-09-10T18:59:36.246767+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T18:59:36.246767+0000","last_undegraded":"2024-09-10T18:59:36.246767+0000","last_fullsized":"2024-09-10T18:59:36.246767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'13","last_scrub_stamp":"2024-09-10T18:56:29.927465+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.927465+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044821779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":416,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242100+0000","last_change":"2024-09-10T18:58:52.884147+0000","last_active":"2024-09-10T18:59:36.242100+0000","last_peered":"2024-09-10T18:59:36.242100+0000","last_clean":"2024-09-10T18:59:36.242100+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T18:59:36.242100+0000","last_undegraded":"2024-09-10T18:59:36.242100+0000","last_fullsized":"2024-09-10T18:59:36.242100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.884077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.965200+0000","scrub_duration":0.0032688370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928736+0000","last_change":"2024-09-10T18:56:36.856348+0000","last_active":"2024-09-10T18:59:36.928736+0000","last_peered":"2024-09-10T18:59:36.928736+0000","last_clean":"2024-09-10T18:59:36.928736+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T18:59:36.928736+0000","last_undegraded":"2024-09-10T18:59:36.928736+0000","last_fullsized":"2024-09-10T18:59:36.928736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929209+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T18:59:36.929209+0000","last_peered":"2024-09-10T18:59:36.929209+0000","last_clean":"2024-09-10T18:59:36.929209+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T18:59:36.929209+0000","last_undegraded":"2024-09-10T18:59:36.929209+0000","last_fullsized":"2024-09-10T18:59:36.929209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:42.012955+0000","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":313,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923221+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T18:59:36.923221+0000","last_peered":"2024-09-10T18:59:36.923221+0000","last_clean":"2024-09-10T18:59:36.923221+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T18:59:36.923221+0000","last_undegraded":"2024-09-10T18:59:36.923221+0000","last_fullsized":"2024-09-10T18:59:36.923221+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":440,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246000+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T18:59:36.246000+0000","last_peered":"2024-09-10T18:59:36.246000+0000","last_clean":"2024-09-10T18:59:36.246000+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T18:59:36.246000+0000","last_undegraded":"2024-09-10T18:59:36.246000+0000","last_fullsized":"2024-09-10T18:59:36.246000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.241437+0000","last_change":"2024-09-10T18:58:51.907906+0000","last_active":"2024-09-10T18:59:36.241437+0000","last_peered":"2024-09-10T18:59:36.241437+0000","last_clean":"2024-09-10T18:59:36.241437+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T18:59:36.241437+0000","last_undegraded":"2024-09-10T18:59:36.241437+0000","last_fullsized":"2024-09-10T18:59:36.241437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:51.907749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:51.907749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.097418+0000","scrub_duration":0.0036594549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":412,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243792+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T18:59:36.243792+0000","last_peered":"2024-09-10T18:59:36.243792+0000","last_clean":"2024-09-10T18:59:36.243792+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T18:59:36.243792+0000","last_undegraded":"2024-09-10T18:59:36.243792+0000","last_fullsized":"2024-09-10T18:59:36.243792+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.611116+0000","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239440+0000","last_change":"2024-09-10T18:56:43.374520+0000","last_active":"2024-09-10T18:59:36.239440+0000","last_peered":"2024-09-10T18:59:36.239440+0000","last_clean":"2024-09-10T18:59:36.239440+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T18:59:36.239440+0000","last_undegraded":"2024-09-10T18:59:36.239440+0000","last_fullsized":"2024-09-10T18:59:36.239440+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.374347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.374347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034327590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":322,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243906+0000","last_change":"2024-09-10T18:57:56.827061+0000","last_active":"2024-09-10T18:59:36.243906+0000","last_peered":"2024-09-10T18:59:36.243906+0000","last_clean":"2024-09-10T18:59:36.243906+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T18:59:36.243906+0000","last_undegraded":"2024-09-10T18:59:36.243906+0000","last_fullsized":"2024-09-10T18:59:36.243906+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.826954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.826954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032518439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":453,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929548+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T18:59:36.929548+0000","last_peered":"2024-09-10T18:59:36.929548+0000","last_clean":"2024-09-10T18:59:36.929548+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T18:59:36.929548+0000","last_undegraded":"2024-09-10T18:59:36.929548+0000","last_fullsized":"2024-09-10T18:59:36.929548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:27.341520+0000","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923945+0000","last_change":"2024-09-10T18:58:32.226883+0000","last_active":"2024-09-10T18:59:36.923945+0000","last_peered":"2024-09-10T18:59:36.923945+0000","last_clean":"2024-09-10T18:59:36.923945+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T18:59:36.923945+0000","last_undegraded":"2024-09-10T18:59:36.923945+0000","last_fullsized":"2024-09-10T18:59:36.923945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.226771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.226771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.723944+0000","scrub_duration":0.024188028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":539,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929345+0000","last_change":"2024-09-10T18:57:32.764761+0000","last_active":"2024-09-10T18:59:36.929345+0000","last_peered":"2024-09-10T18:59:36.929345+0000","last_clean":"2024-09-10T18:59:36.929345+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T18:59:36.929345+0000","last_undegraded":"2024-09-10T18:59:36.929345+0000","last_fullsized":"2024-09-10T18:59:36.929345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T18:57:32.764561+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.764561+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041273020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":394,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246830+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T18:59:36.246830+0000","last_peered":"2024-09-10T18:59:36.246830+0000","last_clean":"2024-09-10T18:59:36.246830+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T18:59:36.246830+0000","last_undegraded":"2024-09-10T18:59:36.246830+0000","last_fullsized":"2024-09-10T18:59:36.246830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":322,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243440+0000","last_change":"2024-09-10T18:58:47.871654+0000","last_active":"2024-09-10T18:59:36.243440+0000","last_peered":"2024-09-10T18:59:36.243440+0000","last_clean":"2024-09-10T18:59:36.243440+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T18:59:36.243440+0000","last_undegraded":"2024-09-10T18:59:36.243440+0000","last_fullsized":"2024-09-10T18:59:36.243440+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.871591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.871591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.791278+0000","scrub_duration":0.0023117910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889752+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T18:59:36.889752+0000","last_peered":"2024-09-10T18:59:36.889752+0000","last_clean":"2024-09-10T18:59:36.889752+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T18:59:36.889752+0000","last_undegraded":"2024-09-10T18:59:36.889752+0000","last_fullsized":"2024-09-10T18:59:36.889752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925729+0000","last_change":"2024-09-10T18:58:50.597419+0000","last_active":"2024-09-10T18:59:36.925729+0000","last_peered":"2024-09-10T18:59:36.925729+0000","last_clean":"2024-09-10T18:59:36.925729+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T18:59:36.925729+0000","last_undegraded":"2024-09-10T18:59:36.925729+0000","last_fullsized":"2024-09-10T18:59:36.925729+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.597321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.597321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.919633+0000","scrub_duration":0.0021838199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928151+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T18:59:36.928151+0000","last_peered":"2024-09-10T18:59:36.928151+0000","last_clean":"2024-09-10T18:59:36.928151+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T18:59:36.928151+0000","last_undegraded":"2024-09-10T18:59:36.928151+0000","last_fullsized":"2024-09-10T18:59:36.928151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1010,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-09-10T18:59:39.000634+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T18:59:39.000634+0000","last_peered":"2024-09-10T18:59:39.000634+0000","last_clean":"2024-09-10T18:59:39.000634+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T18:59:39.000634+0000","last_undegraded":"2024-09-10T18:59:39.000634+0000","last_fullsized":"2024-09-10T18:59:39.000634+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.772656+0000","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":597,"num_read_kb":592,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926546+0000","last_change":"2024-09-10T18:57:49.386576+0000","last_active":"2024-09-10T18:59:36.926546+0000","last_peered":"2024-09-10T18:59:36.926546+0000","last_clean":"2024-09-10T18:59:36.926546+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T18:59:36.926546+0000","last_undegraded":"2024-09-10T18:59:36.926546+0000","last_fullsized":"2024-09-10T18:59:36.926546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.386404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003967628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":333,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924185+0000","last_change":"2024-09-10T18:59:06.979721+0000","last_active":"2024-09-10T18:59:36.924185+0000","last_peered":"2024-09-10T18:59:36.924185+0000","last_clean":"2024-09-10T18:59:36.924185+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T18:59:36.924185+0000","last_undegraded":"2024-09-10T18:59:36.924185+0000","last_fullsized":"2024-09-10T18:59:36.924185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.979599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.979599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.070088+0000","scrub_duration":0.0031493580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":440,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247635+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T18:59:36.247635+0000","last_peered":"2024-09-10T18:59:36.247635+0000","last_clean":"2024-09-10T18:59:36.247635+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T18:59:36.247635+0000","last_undegraded":"2024-09-10T18:59:36.247635+0000","last_fullsized":"2024-09-10T18:59:36.247635+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.664434+0000","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":416,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243872+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T18:59:36.243872+0000","last_peered":"2024-09-10T18:59:36.243872+0000","last_clean":"2024-09-10T18:59:36.243872+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T18:59:36.243872+0000","last_undegraded":"2024-09-10T18:59:36.243872+0000","last_fullsized":"2024-09-10T18:59:36.243872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:00.410089+0000","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":519,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.280203+0000","last_change":"2024-09-10T18:56:18.886830+0000","last_active":"2024-09-10T18:59:36.280203+0000","last_peered":"2024-09-10T18:59:36.280203+0000","last_clean":"2024-09-10T18:59:36.280203+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T18:59:36.280203+0000","last_undegraded":"2024-09-10T18:59:36.280203+0000","last_fullsized":"2024-09-10T18:59:36.280203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:56:18.886660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.886660+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041903139999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":421,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927911+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T18:59:36.927911+0000","last_peered":"2024-09-10T18:59:36.927911+0000","last_clean":"2024-09-10T18:59:36.927911+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T18:59:36.927911+0000","last_undegraded":"2024-09-10T18:59:36.927911+0000","last_fullsized":"2024-09-10T18:59:36.927911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.789834+0000","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928386+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T18:59:36.928386+0000","last_peered":"2024-09-10T18:59:36.928386+0000","last_clean":"2024-09-10T18:59:36.928386+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T18:59:36.928386+0000","last_undegraded":"2024-09-10T18:59:36.928386+0000","last_fullsized":"2024-09-10T18:59:36.928386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":314,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245729+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T18:59:36.245729+0000","last_peered":"2024-09-10T18:59:36.245729+0000","last_clean":"2024-09-10T18:59:36.245729+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T18:59:36.245729+0000","last_undegraded":"2024-09-10T18:59:36.245729+0000","last_fullsized":"2024-09-10T18:59:36.245729+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":606,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.279106+0000","last_change":"2024-09-10T18:57:50.752463+0000","last_active":"2024-09-10T18:59:36.279106+0000","last_peered":"2024-09-10T18:59:36.279106+0000","last_clean":"2024-09-10T18:59:36.279106+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T18:59:36.279106+0000","last_undegraded":"2024-09-10T18:59:36.279106+0000","last_fullsized":"2024-09-10T18:59:36.279106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'40","last_scrub_stamp":"2024-09-10T18:57:50.752292+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.752292+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927603+0000","last_change":"2024-09-10T18:58:50.209102+0000","last_active":"2024-09-10T18:59:36.927603+0000","last_peered":"2024-09-10T18:59:36.927603+0000","last_clean":"2024-09-10T18:59:36.927603+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T18:59:36.927603+0000","last_undegraded":"2024-09-10T18:59:36.927603+0000","last_fullsized":"2024-09-10T18:59:36.927603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.208993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.208993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.944132+0000","scrub_duration":0.003212126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":528,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888263+0000","last_change":"2024-09-10T18:55:17.145349+0000","last_active":"2024-09-10T18:59:36.888263+0000","last_peered":"2024-09-10T18:59:36.888263+0000","last_clean":"2024-09-10T18:59:36.888263+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T18:59:36.888263+0000","last_undegraded":"2024-09-10T18:59:36.888263+0000","last_fullsized":"2024-09-10T18:59:36.888263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:55:17.145187+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:55:17.145187+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.153177231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":410,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923567+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T18:59:36.923567+0000","last_peered":"2024-09-10T18:59:36.923567+0000","last_clean":"2024-09-10T18:59:36.923567+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T18:59:36.923567+0000","last_undegraded":"2024-09-10T18:59:36.923567+0000","last_fullsized":"2024-09-10T18:59:36.923567+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.942357+0000","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.241103+0000","last_change":"2024-09-10T18:58:45.871468+0000","last_active":"2024-09-10T18:59:36.241103+0000","last_peered":"2024-09-10T18:59:36.241103+0000","last_clean":"2024-09-10T18:59:36.241103+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T18:59:36.241103+0000","last_undegraded":"2024-09-10T18:59:36.241103+0000","last_fullsized":"2024-09-10T18:59:36.241103+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:45.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:45.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.946492+0000","scrub_duration":0.0033902770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":314,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246908+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T18:59:36.246908+0000","last_peered":"2024-09-10T18:59:36.246908+0000","last_clean":"2024-09-10T18:59:36.246908+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T18:59:36.246908+0000","last_undegraded":"2024-09-10T18:59:36.246908+0000","last_fullsized":"2024-09-10T18:59:36.246908+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":441,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239969+0000","last_change":"2024-09-10T18:58:46.437660+0000","last_active":"2024-09-10T18:59:36.239969+0000","last_peered":"2024-09-10T18:59:36.239969+0000","last_clean":"2024-09-10T18:59:36.239969+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T18:59:36.239969+0000","last_undegraded":"2024-09-10T18:59:36.239969+0000","last_fullsized":"2024-09-10T18:59:36.239969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:46.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.437410+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.595024+0000","scrub_duration":0.0054784669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242801+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T18:59:36.242801+0000","last_peered":"2024-09-10T18:59:36.242801+0000","last_clean":"2024-09-10T18:59:36.242801+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T18:59:36.242801+0000","last_undegraded":"2024-09-10T18:59:36.242801+0000","last_fullsized":"2024-09-10T18:59:36.242801+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:49.994362+0000","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.241858+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T18:59:36.241858+0000","last_peered":"2024-09-10T18:59:36.241858+0000","last_clean":"2024-09-10T18:59:36.241858+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T18:59:36.241858+0000","last_undegraded":"2024-09-10T18:59:36.241858+0000","last_fullsized":"2024-09-10T18:59:36.241858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.143473+0000","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887251+0000","last_change":"2024-09-10T18:56:11.174662+0000","last_active":"2024-09-10T18:59:36.887251+0000","last_peered":"2024-09-10T18:59:36.887251+0000","last_clean":"2024-09-10T18:59:36.887251+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T18:59:36.887251+0000","last_undegraded":"2024-09-10T18:59:36.887251+0000","last_fullsized":"2024-09-10T18:59:36.887251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:11.174407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T18:56:11.174407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031252239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924162+0000","last_change":"2024-09-10T18:57:43.914840+0000","last_active":"2024-09-10T18:59:36.924162+0000","last_peered":"2024-09-10T18:59:36.924162+0000","last_clean":"2024-09-10T18:59:36.924162+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T18:59:36.924162+0000","last_undegraded":"2024-09-10T18:59:36.924162+0000","last_fullsized":"2024-09-10T18:59:36.924162+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:43.914731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:43.914731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026766530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":457,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928710+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T18:59:36.928710+0000","last_peered":"2024-09-10T18:59:36.928710+0000","last_clean":"2024-09-10T18:59:36.928710+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T18:59:36.928710+0000","last_undegraded":"2024-09-10T18:59:36.928710+0000","last_fullsized":"2024-09-10T18:59:36.928710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238897+0000","last_change":"2024-09-10T18:58:52.447688+0000","last_active":"2024-09-10T18:59:36.238897+0000","last_peered":"2024-09-10T18:59:36.238897+0000","last_clean":"2024-09-10T18:59:36.238897+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T18:59:36.238897+0000","last_undegraded":"2024-09-10T18:59:36.238897+0000","last_fullsized":"2024-09-10T18:59:36.238897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.447547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.447547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.724802+0000","scrub_duration":0.0031696559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242642+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T18:59:36.242642+0000","last_peered":"2024-09-10T18:59:36.242642+0000","last_clean":"2024-09-10T18:59:36.242642+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T18:59:36.242642+0000","last_undegraded":"2024-09-10T18:59:36.242642+0000","last_fullsized":"2024-09-10T18:59:36.242642+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247226+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T18:59:36.247226+0000","last_peered":"2024-09-10T18:59:36.247226+0000","last_clean":"2024-09-10T18:59:36.247226+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T18:59:36.247226+0000","last_undegraded":"2024-09-10T18:59:36.247226+0000","last_fullsized":"2024-09-10T18:59:36.247226+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:56.784439+0000","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888549+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T18:59:36.888549+0000","last_peered":"2024-09-10T18:59:36.888549+0000","last_clean":"2024-09-10T18:59:36.888549+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T18:59:36.888549+0000","last_undegraded":"2024-09-10T18:59:36.888549+0000","last_fullsized":"2024-09-10T18:59:36.888549+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":462,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240996+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T18:59:36.240996+0000","last_peered":"2024-09-10T18:59:36.240996+0000","last_clean":"2024-09-10T18:59:36.240996+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T18:59:36.240996+0000","last_undegraded":"2024-09-10T18:59:36.240996+0000","last_fullsized":"2024-09-10T18:59:36.240996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":417,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923911+0000","last_change":"2024-09-10T18:57:55.986832+0000","last_active":"2024-09-10T18:59:36.923911+0000","last_peered":"2024-09-10T18:59:36.923911+0000","last_clean":"2024-09-10T18:59:36.923911+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T18:59:36.923911+0000","last_undegraded":"2024-09-10T18:59:36.923911+0000","last_fullsized":"2024-09-10T18:59:36.923911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:55.986726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.986726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034130330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.244003+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T18:59:36.244003+0000","last_peered":"2024-09-10T18:59:36.244003+0000","last_clean":"2024-09-10T18:59:36.244003+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T18:59:36.244003+0000","last_undegraded":"2024-09-10T18:59:36.244003+0000","last_fullsized":"2024-09-10T18:59:36.244003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:01.829268+0000","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927030+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T18:59:36.927030+0000","last_peered":"2024-09-10T18:59:36.927030+0000","last_clean":"2024-09-10T18:59:36.927030+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T18:59:36.927030+0000","last_undegraded":"2024-09-10T18:59:36.927030+0000","last_fullsized":"2024-09-10T18:59:36.927030+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925294+0000","last_change":"2024-09-10T18:59:02.713009+0000","last_active":"2024-09-10T18:59:36.925294+0000","last_peered":"2024-09-10T18:59:36.925294+0000","last_clean":"2024-09-10T18:59:36.925294+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T18:59:36.925294+0000","last_undegraded":"2024-09-10T18:59:36.925294+0000","last_fullsized":"2024-09-10T18:59:36.925294+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.712839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.712839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:04.351359+0000","scrub_duration":0.0035464870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":447,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887985+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T18:59:36.887985+0000","last_peered":"2024-09-10T18:59:36.887985+0000","last_clean":"2024-09-10T18:59:36.887985+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T18:59:36.887985+0000","last_undegraded":"2024-09-10T18:59:36.887985+0000","last_fullsized":"2024-09-10T18:59:36.887985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.541418+0000","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.922605+0000","last_change":"2024-09-10T18:58:44.193043+0000","last_active":"2024-09-10T18:59:36.922605+0000","last_peered":"2024-09-10T18:59:36.922605+0000","last_clean":"2024-09-10T18:59:36.922605+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T18:59:36.922605+0000","last_undegraded":"2024-09-10T18:59:36.922605+0000","last_fullsized":"2024-09-10T18:59:36.922605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.192853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.192853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:51.060471+0000","scrub_duration":0.0038691179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889781+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T18:59:36.889781+0000","last_peered":"2024-09-10T18:59:36.889781+0000","last_clean":"2024-09-10T18:59:36.889781+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T18:59:36.889781+0000","last_undegraded":"2024-09-10T18:59:36.889781+0000","last_fullsized":"2024-09-10T18:59:36.889781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.522110+0000","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.242401+0000","last_change":"2024-09-10T18:57:44.612113+0000","last_active":"2024-09-10T18:59:36.242401+0000","last_peered":"2024-09-10T18:59:36.242401+0000","last_clean":"2024-09-10T18:59:36.242401+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T18:59:36.242401+0000","last_undegraded":"2024-09-10T18:59:36.242401+0000","last_fullsized":"2024-09-10T18:59:36.242401+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:44.612043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T18:57:44.612043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026996279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889997+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T18:59:36.889997+0000","last_peered":"2024-09-10T18:59:36.889997+0000","last_clean":"2024-09-10T18:59:36.889997+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T18:59:36.889997+0000","last_undegraded":"2024-09-10T18:59:36.889997+0000","last_fullsized":"2024-09-10T18:59:36.889997+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":432,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246227+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T18:59:36.246227+0000","last_peered":"2024-09-10T18:59:36.246227+0000","last_clean":"2024-09-10T18:59:36.246227+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T18:59:36.246227+0000","last_undegraded":"2024-09-10T18:59:36.246227+0000","last_fullsized":"2024-09-10T18:59:36.246227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.648653+0000","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238418+0000","last_change":"2024-09-10T18:57:56.125998+0000","last_active":"2024-09-10T18:59:36.238418+0000","last_peered":"2024-09-10T18:59:36.238418+0000","last_clean":"2024-09-10T18:59:36.238418+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T18:59:36.238418+0000","last_undegraded":"2024-09-10T18:59:36.238418+0000","last_fullsized":"2024-09-10T18:59:36.238418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.125881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.125881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026334969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":453,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923145+0000","last_change":"2024-09-10T18:57:56.375088+0000","last_active":"2024-09-10T18:59:36.923145+0000","last_peered":"2024-09-10T18:59:36.923145+0000","last_clean":"2024-09-10T18:59:36.923145+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T18:59:36.923145+0000","last_undegraded":"2024-09-10T18:59:36.923145+0000","last_fullsized":"2024-09-10T18:59:36.923145+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:57:56.374923+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.374923+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043604359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925492+0000","last_change":"2024-09-10T18:58:47.555840+0000","last_active":"2024-09-10T18:59:36.925492+0000","last_peered":"2024-09-10T18:59:36.925492+0000","last_clean":"2024-09-10T18:59:36.925492+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T18:59:36.925492+0000","last_undegraded":"2024-09-10T18:59:36.925492+0000","last_fullsized":"2024-09-10T18:59:36.925492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.555637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.555637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.468673+0000","scrub_duration":0.0032502030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240318+0000","last_change":"2024-09-10T18:58:59.454163+0000","last_active":"2024-09-10T18:59:36.240318+0000","last_peered":"2024-09-10T18:59:36.240318+0000","last_clean":"2024-09-10T18:59:36.240318+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T18:59:36.240318+0000","last_undegraded":"2024-09-10T18:59:36.240318+0000","last_fullsized":"2024-09-10T18:59:36.240318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.565579+0000","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927454+0000","last_change":"2024-09-10T18:58:54.237739+0000","last_active":"2024-09-10T18:59:36.927454+0000","last_peered":"2024-09-10T18:59:36.927454+0000","last_clean":"2024-09-10T18:59:36.927454+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T18:59:36.927454+0000","last_undegraded":"2024-09-10T18:59:36.927454+0000","last_fullsized":"2024-09-10T18:59:36.927454+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.237585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.237585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.680207+0000","scrub_duration":0.0092150240000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":460,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927540+0000","last_change":"2024-09-10T18:58:52.253613+0000","last_active":"2024-09-10T18:59:36.927540+0000","last_peered":"2024-09-10T18:59:36.927540+0000","last_clean":"2024-09-10T18:59:36.927540+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T18:59:36.927540+0000","last_undegraded":"2024-09-10T18:59:36.927540+0000","last_fullsized":"2024-09-10T18:59:36.927540+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'25","last_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.253376+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.678764+0000","scrub_duration":0.011205784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":417,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925597+0000","last_change":"2024-09-10T18:58:38.064448+0000","last_active":"2024-09-10T18:59:36.925597+0000","last_peered":"2024-09-10T18:59:36.925597+0000","last_clean":"2024-09-10T18:59:36.925597+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T18:59:36.925597+0000","last_undegraded":"2024-09-10T18:59:36.925597+0000","last_fullsized":"2024-09-10T18:59:36.925597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.064267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.064267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:48.065185+0000","scrub_duration":0.0037942050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923977+0000","last_change":"2024-09-10T18:58:42.486651+0000","last_active":"2024-09-10T18:59:36.923977+0000","last_peered":"2024-09-10T18:59:36.923977+0000","last_clean":"2024-09-10T18:59:36.923977+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T18:59:36.923977+0000","last_undegraded":"2024-09-10T18:59:36.923977+0000","last_fullsized":"2024-09-10T18:59:36.923977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:42.486498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:42.486498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:47.324824+0000","scrub_duration":0.0032582879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924022+0000","last_change":"2024-09-10T18:58:55.261885+0000","last_active":"2024-09-10T18:59:36.924022+0000","last_peered":"2024-09-10T18:59:36.924022+0000","last_clean":"2024-09-10T18:59:36.924022+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T18:59:36.924022+0000","last_undegraded":"2024-09-10T18:59:36.924022+0000","last_fullsized":"2024-09-10T18:59:36.924022+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.261635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.261635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.433144+0000","scrub_duration":0.0046216190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928171+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T18:59:36.928171+0000","last_peered":"2024-09-10T18:59:36.928171+0000","last_clean":"2024-09-10T18:59:36.928171+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T18:59:36.928171+0000","last_undegraded":"2024-09-10T18:59:36.928171+0000","last_fullsized":"2024-09-10T18:59:36.928171+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.666116+0000","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":436,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239223+0000","last_change":"2024-09-10T18:59:01.489222+0000","last_active":"2024-09-10T18:59:36.239223+0000","last_peered":"2024-09-10T18:59:36.239223+0000","last_clean":"2024-09-10T18:59:36.239223+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T18:59:36.239223+0000","last_undegraded":"2024-09-10T18:59:36.239223+0000","last_fullsized":"2024-09-10T18:59:36.239223+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:59:01.489057+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.489057+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:21.754146+0000","scrub_duration":0.0045114960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243623+0000","last_change":"2024-09-10T18:58:38.864525+0000","last_active":"2024-09-10T18:59:36.243623+0000","last_peered":"2024-09-10T18:59:36.243623+0000","last_clean":"2024-09-10T18:59:36.243623+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T18:59:36.243623+0000","last_undegraded":"2024-09-10T18:59:36.243623+0000","last_fullsized":"2024-09-10T18:59:36.243623+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.864360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.864360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.640967+0000","scrub_duration":0.00368761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887755+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T18:59:36.887755+0000","last_peered":"2024-09-10T18:59:36.887755+0000","last_clean":"2024-09-10T18:59:36.887755+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T18:59:36.887755+0000","last_undegraded":"2024-09-10T18:59:36.887755+0000","last_fullsized":"2024-09-10T18:59:36.887755+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.575228+0000","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.887749+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T18:59:36.887749+0000","last_peered":"2024-09-10T18:59:36.887749+0000","last_clean":"2024-09-10T18:59:36.887749+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T18:59:36.887749+0000","last_undegraded":"2024-09-10T18:59:36.887749+0000","last_fullsized":"2024-09-10T18:59:36.887749+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:03.423360+0000","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888171+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T18:59:36.888171+0000","last_peered":"2024-09-10T18:59:36.888171+0000","last_clean":"2024-09-10T18:59:36.888171+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T18:59:36.888171+0000","last_undegraded":"2024-09-10T18:59:36.888171+0000","last_fullsized":"2024-09-10T18:59:36.888171+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":453,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927994+0000","last_change":"2024-09-10T18:58:48.217515+0000","last_active":"2024-09-10T18:59:36.927994+0000","last_peered":"2024-09-10T18:59:36.927994+0000","last_clean":"2024-09-10T18:59:36.927994+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T18:59:36.927994+0000","last_undegraded":"2024-09-10T18:59:36.927994+0000","last_fullsized":"2024-09-10T18:59:36.927994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:58:48.217409+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.217409+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:54.394749+0000","scrub_duration":0.0034746339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":429,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926632+0000","last_change":"2024-09-10T18:58:34.970321+0000","last_active":"2024-09-10T18:59:36.926632+0000","last_peered":"2024-09-10T18:59:36.926632+0000","last_clean":"2024-09-10T18:59:36.926632+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T18:59:36.926632+0000","last_undegraded":"2024-09-10T18:59:36.926632+0000","last_fullsized":"2024-09-10T18:59:36.926632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:34.970167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.970167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:39.762548+0000","scrub_duration":0.0036040859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238127+0000","last_change":"2024-09-10T18:57:34.244420+0000","last_active":"2024-09-10T18:59:36.238127+0000","last_peered":"2024-09-10T18:59:36.238127+0000","last_clean":"2024-09-10T18:59:36.238127+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T18:59:36.238127+0000","last_undegraded":"2024-09-10T18:59:36.238127+0000","last_fullsized":"2024-09-10T18:59:36.238127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:34.244346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T18:57:34.244346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00228093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238620+0000","last_change":"2024-09-10T18:58:37.374134+0000","last_active":"2024-09-10T18:59:36.238620+0000","last_peered":"2024-09-10T18:59:36.238620+0000","last_clean":"2024-09-10T18:59:36.238620+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T18:59:36.238620+0000","last_undegraded":"2024-09-10T18:59:36.238620+0000","last_fullsized":"2024-09-10T18:59:36.238620+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.374014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.374014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:41.692319+0000","scrub_duration":0.0023318560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926818+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T18:59:36.926818+0000","last_peered":"2024-09-10T18:59:36.926818+0000","last_clean":"2024-09-10T18:59:36.926818+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T18:59:36.926818+0000","last_undegraded":"2024-09-10T18:59:36.926818+0000","last_fullsized":"2024-09-10T18:59:36.926818+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.117442+0000","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":434,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927850+0000","last_change":"2024-09-10T18:58:47.178004+0000","last_active":"2024-09-10T18:59:36.927850+0000","last_peered":"2024-09-10T18:59:36.927850+0000","last_clean":"2024-09-10T18:59:36.927850+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T18:59:36.927850+0000","last_undegraded":"2024-09-10T18:59:36.927850+0000","last_fullsized":"2024-09-10T18:59:36.927850+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:47.177751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.177751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:03.310286+0000","scrub_duration":0.0059440730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":447,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929767+0000","last_change":"2024-09-10T18:56:34.853998+0000","last_active":"2024-09-10T18:59:36.929767+0000","last_peered":"2024-09-10T18:59:36.929767+0000","last_clean":"2024-09-10T18:59:36.929767+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T18:59:36.929767+0000","last_undegraded":"2024-09-10T18:59:36.929767+0000","last_fullsized":"2024-09-10T18:59:36.929767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:56:34.853831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.853831+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003732731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":538,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929190+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T18:59:36.929190+0000","last_peered":"2024-09-10T18:59:36.929190+0000","last_clean":"2024-09-10T18:59:36.929190+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T18:59:36.929190+0000","last_undegraded":"2024-09-10T18:59:36.929190+0000","last_fullsized":"2024-09-10T18:59:36.929190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246584+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T18:59:36.246584+0000","last_peered":"2024-09-10T18:59:36.246584+0000","last_clean":"2024-09-10T18:59:36.246584+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T18:59:36.246584+0000","last_undegraded":"2024-09-10T18:59:36.246584+0000","last_fullsized":"2024-09-10T18:59:36.246584+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":326,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.921905+0000","last_change":"2024-09-10T18:56:34.925841+0000","last_active":"2024-09-10T18:59:36.921905+0000","last_peered":"2024-09-10T18:59:36.921905+0000","last_clean":"2024-09-10T18:59:36.921905+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T18:59:36.921905+0000","last_undegraded":"2024-09-10T18:59:36.921905+0000","last_fullsized":"2024-09-10T18:59:36.921905+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:56:34.925675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.925675+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040564759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":434,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925306+0000","last_change":"2024-09-10T18:57:33.408816+0000","last_active":"2024-09-10T18:59:36.925306+0000","last_peered":"2024-09-10T18:59:36.925306+0000","last_clean":"2024-09-10T18:59:36.925306+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T18:59:36.925306+0000","last_undegraded":"2024-09-10T18:59:36.925306+0000","last_fullsized":"2024-09-10T18:59:36.925306+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:33.408655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:33.408655+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039294170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":417,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924134+0000","last_change":"2024-09-10T18:58:46.953749+0000","last_active":"2024-09-10T18:59:36.924134+0000","last_peered":"2024-09-10T18:59:36.924134+0000","last_clean":"2024-09-10T18:59:36.924134+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T18:59:36.924134+0000","last_undegraded":"2024-09-10T18:59:36.924134+0000","last_fullsized":"2024-09-10T18:59:36.924134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.953608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.953608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.755421+0000","scrub_duration":0.0028191259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889564+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T18:59:36.889564+0000","last_peered":"2024-09-10T18:59:36.889564+0000","last_clean":"2024-09-10T18:59:36.889564+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T18:59:36.889564+0000","last_undegraded":"2024-09-10T18:59:36.889564+0000","last_fullsized":"2024-09-10T18:59:36.889564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":401,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925101+0000","last_change":"2024-09-10T18:57:42.899838+0000","last_active":"2024-09-10T18:59:36.925101+0000","last_peered":"2024-09-10T18:59:36.925101+0000","last_clean":"2024-09-10T18:59:36.925101+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T18:59:36.925101+0000","last_undegraded":"2024-09-10T18:59:36.925101+0000","last_fullsized":"2024-09-10T18:59:36.925101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.899684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.899684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036750730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929504+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T18:59:36.929504+0000","last_peered":"2024-09-10T18:59:36.929504+0000","last_clean":"2024-09-10T18:59:36.929504+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T18:59:36.929504+0000","last_undegraded":"2024-09-10T18:59:36.929504+0000","last_fullsized":"2024-09-10T18:59:36.929504+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.954334+0000","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":418,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.921882+0000","last_change":"2024-09-10T18:58:40.428324+0000","last_active":"2024-09-10T18:59:36.921882+0000","last_peered":"2024-09-10T18:59:36.921882+0000","last_clean":"2024-09-10T18:59:36.921882+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T18:59:36.921882+0000","last_undegraded":"2024-09-10T18:59:36.921882+0000","last_fullsized":"2024-09-10T18:59:36.921882+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:58:40.428157+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.428157+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:40.549402+0000","scrub_duration":0.003853726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923368+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T18:59:36.923368+0000","last_peered":"2024-09-10T18:59:36.923368+0000","last_clean":"2024-09-10T18:59:36.923368+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T18:59:36.923368+0000","last_undegraded":"2024-09-10T18:59:36.923368+0000","last_fullsized":"2024-09-10T18:59:36.923368+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247575+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T18:59:36.247575+0000","last_peered":"2024-09-10T18:59:36.247575+0000","last_clean":"2024-09-10T18:59:36.247575+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T18:59:36.247575+0000","last_undegraded":"2024-09-10T18:59:36.247575+0000","last_fullsized":"2024-09-10T18:59:36.247575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.425187+0000","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926892+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T18:59:36.926892+0000","last_peered":"2024-09-10T18:59:36.926892+0000","last_clean":"2024-09-10T18:59:36.926892+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T18:59:36.926892+0000","last_undegraded":"2024-09-10T18:59:36.926892+0000","last_fullsized":"2024-09-10T18:59:36.926892+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925197+0000","last_change":"2024-09-10T18:59:01.663804+0000","last_active":"2024-09-10T18:59:36.925197+0000","last_peered":"2024-09-10T18:59:36.925197+0000","last_clean":"2024-09-10T18:59:36.925197+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T18:59:36.925197+0000","last_undegraded":"2024-09-10T18:59:36.925197+0000","last_fullsized":"2024-09-10T18:59:36.925197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.663643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.663643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.528103+0000","scrub_duration":0.0029754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":439,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.926905+0000","last_change":"2024-09-10T18:58:56.293280+0000","last_active":"2024-09-10T18:59:36.926905+0000","last_peered":"2024-09-10T18:59:36.926905+0000","last_clean":"2024-09-10T18:59:36.926905+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T18:59:36.926905+0000","last_undegraded":"2024-09-10T18:59:36.926905+0000","last_fullsized":"2024-09-10T18:59:36.926905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:58:56.293027+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.293027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:58.342426+0000","scrub_duration":0.0069050090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":396,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239004+0000","last_change":"2024-09-10T18:57:30.249373+0000","last_active":"2024-09-10T18:59:36.239004+0000","last_peered":"2024-09-10T18:59:36.239004+0000","last_clean":"2024-09-10T18:59:36.239004+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T18:59:36.239004+0000","last_undegraded":"2024-09-10T18:59:36.239004+0000","last_fullsized":"2024-09-10T18:59:36.239004+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:30.249133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:30.249133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003727419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":532,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.280150+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T18:59:36.280150+0000","last_peered":"2024-09-10T18:59:36.280150+0000","last_clean":"2024-09-10T18:59:36.280150+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T18:59:36.280150+0000","last_undegraded":"2024-09-10T18:59:36.280150+0000","last_fullsized":"2024-09-10T18:59:36.280150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889359+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T18:59:36.889359+0000","last_peered":"2024-09-10T18:59:36.889359+0000","last_clean":"2024-09-10T18:59:36.889359+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T18:59:36.889359+0000","last_undegraded":"2024-09-10T18:59:36.889359+0000","last_fullsized":"2024-09-10T18:59:36.889359+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889003+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T18:59:36.889003+0000","last_peered":"2024-09-10T18:59:36.889003+0000","last_clean":"2024-09-10T18:59:36.889003+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T18:59:36.889003+0000","last_undegraded":"2024-09-10T18:59:36.889003+0000","last_fullsized":"2024-09-10T18:59:36.889003+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":420,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888602+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T18:59:36.888602+0000","last_peered":"2024-09-10T18:59:36.888602+0000","last_clean":"2024-09-10T18:59:36.888602+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T18:59:36.888602+0000","last_undegraded":"2024-09-10T18:59:36.888602+0000","last_fullsized":"2024-09-10T18:59:36.888602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":407,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888855+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T18:59:36.888855+0000","last_peered":"2024-09-10T18:59:36.888855+0000","last_clean":"2024-09-10T18:59:36.888855+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T18:59:36.888855+0000","last_undegraded":"2024-09-10T18:59:36.888855+0000","last_fullsized":"2024-09-10T18:59:36.888855+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:19.026489+0000","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":523,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.277279+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T18:59:36.277279+0000","last_peered":"2024-09-10T18:59:36.277279+0000","last_clean":"2024-09-10T18:59:36.277279+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T18:59:36.277279+0000","last_undegraded":"2024-09-10T18:59:36.277279+0000","last_fullsized":"2024-09-10T18:59:36.277279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.132106+0000","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929661+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T18:59:36.929661+0000","last_peered":"2024-09-10T18:59:36.929661+0000","last_clean":"2024-09-10T18:59:36.929661+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T18:59:36.929661+0000","last_undegraded":"2024-09-10T18:59:36.929661+0000","last_fullsized":"2024-09-10T18:59:36.929661+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889470+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T18:59:36.889470+0000","last_peered":"2024-09-10T18:59:36.889470+0000","last_clean":"2024-09-10T18:59:36.889470+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T18:59:36.889470+0000","last_undegraded":"2024-09-10T18:59:36.889470+0000","last_fullsized":"2024-09-10T18:59:36.889470+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.987912+0000","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":428,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927232+0000","last_change":"2024-09-10T18:58:38.152259+0000","last_active":"2024-09-10T18:59:36.927232+0000","last_peered":"2024-09-10T18:59:36.927232+0000","last_clean":"2024-09-10T18:59:36.927232+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T18:59:36.927232+0000","last_undegraded":"2024-09-10T18:59:36.927232+0000","last_fullsized":"2024-09-10T18:59:36.927232+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.152079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.152079+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:55.945644+0000","scrub_duration":0.0040090350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923972+0000","last_change":"2024-09-10T18:58:55.639930+0000","last_active":"2024-09-10T18:59:36.923972+0000","last_peered":"2024-09-10T18:59:36.923972+0000","last_clean":"2024-09-10T18:59:36.923972+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T18:59:36.923972+0000","last_undegraded":"2024-09-10T18:59:36.923972+0000","last_fullsized":"2024-09-10T18:59:36.923972+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.639777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.639777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.180410+0000","scrub_duration":0.003186637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238319+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T18:59:36.238319+0000","last_peered":"2024-09-10T18:59:36.238319+0000","last_clean":"2024-09-10T18:59:36.238319+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T18:59:36.238319+0000","last_undegraded":"2024-09-10T18:59:36.238319+0000","last_fullsized":"2024-09-10T18:59:36.238319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":394,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.886833+0000","last_change":"2024-09-10T18:56:03.219928+0000","last_active":"2024-09-10T18:59:36.886833+0000","last_peered":"2024-09-10T18:59:36.886833+0000","last_clean":"2024-09-10T18:59:36.886833+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T18:59:36.886833+0000","last_undegraded":"2024-09-10T18:59:36.886833+0000","last_fullsized":"2024-09-10T18:59:36.886833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:03.219776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:03.219776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032877649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928502+0000","last_change":"2024-09-10T18:58:46.501591+0000","last_active":"2024-09-10T18:59:36.928502+0000","last_peered":"2024-09-10T18:59:36.928502+0000","last_clean":"2024-09-10T18:59:36.928502+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T18:59:36.928502+0000","last_undegraded":"2024-09-10T18:59:36.928502+0000","last_fullsized":"2024-09-10T18:59:36.928502+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.501508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.732556+0000","scrub_duration":0.0036484099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":441,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928910+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T18:59:36.928910+0000","last_peered":"2024-09-10T18:59:36.928910+0000","last_clean":"2024-09-10T18:59:36.928910+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T18:59:36.928910+0000","last_undegraded":"2024-09-10T18:59:36.928910+0000","last_fullsized":"2024-09-10T18:59:36.928910+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":416,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243529+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T18:59:36.243529+0000","last_peered":"2024-09-10T18:59:36.243529+0000","last_clean":"2024-09-10T18:59:36.243529+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T18:59:36.243529+0000","last_undegraded":"2024-09-10T18:59:36.243529+0000","last_fullsized":"2024-09-10T18:59:36.243529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:17.284809+0000","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928578+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T18:59:36.928578+0000","last_peered":"2024-09-10T18:59:36.928578+0000","last_clean":"2024-09-10T18:59:36.928578+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T18:59:36.928578+0000","last_undegraded":"2024-09-10T18:59:36.928578+0000","last_fullsized":"2024-09-10T18:59:36.928578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.811604+0000","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":396,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245636+0000","last_change":"2024-09-10T18:58:57.260069+0000","last_active":"2024-09-10T18:59:36.245636+0000","last_peered":"2024-09-10T18:59:36.245636+0000","last_clean":"2024-09-10T18:59:36.245636+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T18:59:36.245636+0000","last_undegraded":"2024-09-10T18:59:36.245636+0000","last_fullsized":"2024-09-10T18:59:36.245636+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:23.959177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:23.959177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034976370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928501+0000","last_change":"2024-09-10T18:58:49.517454+0000","last_active":"2024-09-10T18:59:36.928501+0000","last_peered":"2024-09-10T18:59:36.928501+0000","last_clean":"2024-09-10T18:59:36.928501+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T18:59:36.928501+0000","last_undegraded":"2024-09-10T18:59:36.928501+0000","last_fullsized":"2024-09-10T18:59:36.928501+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.517273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.517273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:52.037358+0000","scrub_duration":0.0039870039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":416,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889697+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T18:59:36.889697+0000","last_peered":"2024-09-10T18:59:36.889697+0000","last_clean":"2024-09-10T18:59:36.889697+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T18:59:36.889697+0000","last_undegraded":"2024-09-10T18:59:36.889697+0000","last_fullsized":"2024-09-10T18:59:36.889697+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240017+0000","last_change":"2024-09-10T18:58:53.461054+0000","last_active":"2024-09-10T18:59:36.240017+0000","last_peered":"2024-09-10T18:59:36.240017+0000","last_clean":"2024-09-10T18:59:36.240017+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T18:59:36.240017+0000","last_undegraded":"2024-09-10T18:59:36.240017+0000","last_fullsized":"2024-09-10T18:59:36.240017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.460812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.460812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:07.134909+0000","scrub_duration":0.0042777029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245598+0000","last_change":"2024-09-10T18:58:51.198860+0000","last_active":"2024-09-10T18:59:36.245598+0000","last_peered":"2024-09-10T18:59:36.245598+0000","last_clean":"2024-09-10T18:59:36.245598+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T18:59:36.245598+0000","last_undegraded":"2024-09-10T18:59:36.245598+0000","last_fullsized":"2024-09-10T18:59:36.245598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:17.867220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:56:17.867220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028211619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925417+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T18:59:36.925417+0000","last_peered":"2024-09-10T18:59:36.925417+0000","last_clean":"2024-09-10T18:59:36.925417+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T18:59:36.925417+0000","last_undegraded":"2024-09-10T18:59:36.925417+0000","last_fullsized":"2024-09-10T18:59:36.925417+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":322,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240883+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T18:59:36.240883+0000","last_peered":"2024-09-10T18:59:36.240883+0000","last_clean":"2024-09-10T18:59:36.240883+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T18:59:36.240883+0000","last_undegraded":"2024-09-10T18:59:36.240883+0000","last_fullsized":"2024-09-10T18:59:36.240883+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.299236+0000","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":426,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.240010+0000","last_change":"2024-09-10T18:59:04.505215+0000","last_active":"2024-09-10T18:59:36.240010+0000","last_peered":"2024-09-10T18:59:36.240010+0000","last_clean":"2024-09-10T18:59:36.240010+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T18:59:36.240010+0000","last_undegraded":"2024-09-10T18:59:36.240010+0000","last_fullsized":"2024-09-10T18:59:36.240010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:59:04.505046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.505046+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.388122+0000","scrub_duration":0.0045725100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925647+0000","last_change":"2024-09-10T18:57:56.351163+0000","last_active":"2024-09-10T18:59:36.925647+0000","last_peered":"2024-09-10T18:59:36.925647+0000","last_clean":"2024-09-10T18:59:36.925647+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T18:59:36.925647+0000","last_undegraded":"2024-09-10T18:59:36.925647+0000","last_fullsized":"2024-09-10T18:59:36.925647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.351005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.351005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035909499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245757+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T18:59:36.245757+0000","last_peered":"2024-09-10T18:59:36.245757+0000","last_clean":"2024-09-10T18:59:36.245757+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T18:59:36.245757+0000","last_undegraded":"2024-09-10T18:59:36.245757+0000","last_fullsized":"2024-09-10T18:59:36.245757+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.764329+0000","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246656+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T18:59:36.246656+0000","last_peered":"2024-09-10T18:59:36.246656+0000","last_clean":"2024-09-10T18:59:36.246656+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T18:59:36.246656+0000","last_undegraded":"2024-09-10T18:59:36.246656+0000","last_fullsized":"2024-09-10T18:59:36.246656+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":312,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888694+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T18:59:36.888694+0000","last_peered":"2024-09-10T18:59:36.888694+0000","last_clean":"2024-09-10T18:59:36.888694+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T18:59:36.888694+0000","last_undegraded":"2024-09-10T18:59:36.888694+0000","last_fullsized":"2024-09-10T18:59:36.888694+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.927025+0000","last_change":"2024-09-10T18:58:58.248456+0000","last_active":"2024-09-10T18:59:36.927025+0000","last_peered":"2024-09-10T18:59:36.927025+0000","last_clean":"2024-09-10T18:59:36.927025+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T18:59:36.927025+0000","last_undegraded":"2024-09-10T18:59:36.927025+0000","last_fullsized":"2024-09-10T18:59:36.927025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.067159+0000","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":464,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.241170+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T18:59:36.241170+0000","last_peered":"2024-09-10T18:59:36.241170+0000","last_clean":"2024-09-10T18:59:36.241170+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T18:59:36.241170+0000","last_undegraded":"2024-09-10T18:59:36.241170+0000","last_fullsized":"2024-09-10T18:59:36.241170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.312723+0000","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.238389+0000","last_change":"2024-09-10T18:58:31.395234+0000","last_active":"2024-09-10T18:59:36.238389+0000","last_peered":"2024-09-10T18:59:36.238389+0000","last_clean":"2024-09-10T18:59:36.238389+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T18:59:36.238389+0000","last_undegraded":"2024-09-10T18:59:36.238389+0000","last_fullsized":"2024-09-10T18:59:36.238389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.395072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.395072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032990430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.889516+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T18:59:36.889516+0000","last_peered":"2024-09-10T18:59:36.889516+0000","last_clean":"2024-09-10T18:59:36.889516+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T18:59:36.889516+0000","last_undegraded":"2024-09-10T18:59:36.889516+0000","last_fullsized":"2024-09-10T18:59:36.889516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":313,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924716+0000","last_change":"2024-09-10T18:56:31.864501+0000","last_active":"2024-09-10T18:59:36.924716+0000","last_peered":"2024-09-10T18:59:36.924716+0000","last_clean":"2024-09-10T18:59:36.924716+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T18:59:36.924716+0000","last_undegraded":"2024-09-10T18:59:36.924716+0000","last_fullsized":"2024-09-10T18:59:36.924716+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.864333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.864333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033311500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.243106+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T18:59:36.243106+0000","last_peered":"2024-09-10T18:59:36.243106+0000","last_clean":"2024-09-10T18:59:36.243106+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T18:59:36.243106+0000","last_undegraded":"2024-09-10T18:59:36.243106+0000","last_fullsized":"2024-09-10T18:59:36.243106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":422,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.245973+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T18:59:36.245973+0000","last_peered":"2024-09-10T18:59:36.245973+0000","last_clean":"2024-09-10T18:59:36.245973+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T18:59:36.245973+0000","last_undegraded":"2024-09-10T18:59:36.245973+0000","last_fullsized":"2024-09-10T18:59:36.245973+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:53.283437+0000","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246346+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T18:59:36.246346+0000","last_peered":"2024-09-10T18:59:36.246346+0000","last_clean":"2024-09-10T18:59:36.246346+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T18:59:36.246346+0000","last_undegraded":"2024-09-10T18:59:36.246346+0000","last_fullsized":"2024-09-10T18:59:36.246346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":553,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925561+0000","last_change":"2024-09-10T18:58:38.434610+0000","last_active":"2024-09-10T18:59:36.925561+0000","last_peered":"2024-09-10T18:59:36.925561+0000","last_clean":"2024-09-10T18:59:36.925561+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T18:59:36.925561+0000","last_undegraded":"2024-09-10T18:59:36.925561+0000","last_fullsized":"2024-09-10T18:59:36.925561+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:58:38.434348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.434348+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T18:59:43.611964+0000","scrub_duration":0.0046950070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925463+0000","last_change":"2024-09-10T18:59:00.644451+0000","last_active":"2024-09-10T18:59:36.925463+0000","last_peered":"2024-09-10T18:59:36.925463+0000","last_clean":"2024-09-10T18:59:36.925463+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T18:59:36.925463+0000","last_undegraded":"2024-09-10T18:59:36.925463+0000","last_fullsized":"2024-09-10T18:59:36.925463+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.644299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.665454+0000","scrub_duration":0.00344357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.929241+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T18:59:36.929241+0000","last_peered":"2024-09-10T18:59:36.929241+0000","last_clean":"2024-09-10T18:59:36.929241+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T18:59:36.929241+0000","last_undegraded":"2024-09-10T18:59:36.929241+0000","last_fullsized":"2024-09-10T18:59:36.929241+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":415,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.924379+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T18:59:36.924379+0000","last_peered":"2024-09-10T18:59:36.924379+0000","last_clean":"2024-09-10T18:59:36.924379+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T18:59:36.924379+0000","last_undegraded":"2024-09-10T18:59:36.924379+0000","last_fullsized":"2024-09-10T18:59:36.924379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.205170+0000","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":408,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.888590+0000","last_change":"2024-09-10T18:56:07.266301+0000","last_active":"2024-09-10T18:59:36.888590+0000","last_peered":"2024-09-10T18:59:36.888590+0000","last_clean":"2024-09-10T18:59:36.888590+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T18:59:36.888590+0000","last_undegraded":"2024-09-10T18:59:36.888590+0000","last_fullsized":"2024-09-10T18:59:36.888590+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'5","last_scrub_stamp":"2024-09-10T18:56:07.266130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:07.266130+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041803400000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":413,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.925384+0000","last_change":"2024-09-10T18:58:32.931801+0000","last_active":"2024-09-10T18:59:36.925384+0000","last_peered":"2024-09-10T18:59:36.925384+0000","last_clean":"2024-09-10T18:59:36.925384+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T18:59:36.925384+0000","last_undegraded":"2024-09-10T18:59:36.925384+0000","last_fullsized":"2024-09-10T18:59:36.925384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.931700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.931700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002489907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":310,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.239464+0000","last_change":"2024-09-10T18:56:25.269391+0000","last_active":"2024-09-10T18:59:36.239464+0000","last_peered":"2024-09-10T18:59:36.239464+0000","last_clean":"2024-09-10T18:59:36.239464+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T18:59:36.239464+0000","last_undegraded":"2024-09-10T18:59:36.239464+0000","last_fullsized":"2024-09-10T18:59:36.239464+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.269225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.269225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031845459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247512+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T18:59:36.247512+0000","last_peered":"2024-09-10T18:59:36.247512+0000","last_clean":"2024-09-10T18:59:36.247512+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T18:59:36.247512+0000","last_undegraded":"2024-09-10T18:59:36.247512+0000","last_fullsized":"2024-09-10T18:59:36.247512+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.246879+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T18:59:36.246879+0000","last_peered":"2024-09-10T18:59:36.246879+0000","last_clean":"2024-09-10T18:59:36.246879+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T18:59:36.246879+0000","last_undegraded":"2024-09-10T18:59:36.246879+0000","last_fullsized":"2024-09-10T18:59:36.246879+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.928864+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T18:59:36.928864+0000","last_peered":"2024-09-10T18:59:36.928864+0000","last_clean":"2024-09-10T18:59:36.928864+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T18:59:36.928864+0000","last_undegraded":"2024-09-10T18:59:36.928864+0000","last_fullsized":"2024-09-10T18:59:36.928864+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.247067+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T18:59:36.247067+0000","last_peered":"2024-09-10T18:59:36.247067+0000","last_clean":"2024-09-10T18:59:36.247067+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T18:59:36.247067+0000","last_undegraded":"2024-09-10T18:59:36.247067+0000","last_fullsized":"2024-09-10T18:59:36.247067+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:02.078660+0000","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":323,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T18:59:36.923527+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T18:59:36.923527+0000","last_peered":"2024-09-10T18:59:36.923527+0000","last_clean":"2024-09-10T18:59:36.923527+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T18:59:36.923527+0000","last_undegraded":"2024-09-10T18:59:36.923527+0000","last_fullsized":"2024-09-10T18:59:36.923527+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.897723+0000","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":209,"num_pg":32,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":48,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195724,"num_objects":149,"num_object_clones":2,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":107,"num_read_kb":7,"num_write":576,"num_write_kb":12667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26382336,"data_stored":24587172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1834,"ondisk_log_size":1834,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":597,"num_read_kb":592,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972488,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312472,"kb_used_data":15104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93432680,"statfs":{"total":95995035648,"available":95675064320,"internally_reserved":0,"allocated":15466496,"data_stored":12510798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.85799999999999998,"15min":0.91200000000000003},"min":{"1min":0.52200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.2929999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.873,"5min":0.91200000000000003,"15min":0.92900000000000005},"min":{"1min":0.498,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3839999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86099999999999999,"15min":0.89600000000000002},"min":{"1min":0.52400000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1870000000000001,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.96399999999999997,"15min":1.0129999999999999},"min":{"1min":0.54400000000000004,"5min":0.47799999999999998,"15min":0.36499999999999999},"max":{"1min":1.4119999999999999,"5min":1.742,"15min":1.8280000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.61099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.365,"5min":3.4990000000000001,"15min":3.4990000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.004,"5min":0.999,"15min":0.98899999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.355,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.64500000000000002}]},{"osd":3,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.042,"15min":1.0189999999999999},"min":{"1min":0.65400000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.323,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.98899999999999999,"15min":0.97699999999999998},"min":{"1min":0.628,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.256,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0469999999999999,"15min":1.0189999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4279999999999999,"5min":1.556,"15min":2.1989999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.125,"15min":1.0589999999999999},"min":{"1min":0.61799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.4890000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.012,"15min":0.997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.161,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.099,"5min":1.159,"15min":1.171},"min":{"1min":0.51000000000000001,"5min":0.505,"15min":0.505},"max":{"1min":1.49,"5min":1.5329999999999999,"15min":2.1419999999999999},"last":0.54400000000000004}]},{"osd":6,"last update":"Tue Sep 10 18:58:49 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.2030000000000001,"15min":1.2230000000000001},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5680000000000001,"5min":1.591,"15min":2.2749999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.204,"15min":1.194},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.6910000000000001,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":0.70599999999999996}]}]},{"osd":6,"up_from":52,"seq":223338299538,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308524,"kb_used_data":7004,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436628,"statfs":{"total":95995035648,"available":95679107072,"internally_reserved":0,"allocated":7172096,"data_stored":4314547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.87,"15min":0.88600000000000001},"min":{"1min":0.435,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.8839999999999999,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.006,"15min":0.95699999999999996},"min":{"1min":0.57699999999999996,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.7309999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":1.1990000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.212,"15min":1.0149999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":2.2410000000000001,"5min":18.494,"15min":18.494},"last":1.036},{"interface":"front","average":{"1min":1.022,"5min":0.90800000000000003,"15min":0.90900000000000003},"min":{"1min":0.70199999999999996,"5min":0.51400000000000001,"15min":0.505},"max":{"1min":1.7689999999999999,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.94099999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.0289999999999999,"15min":0.95899999999999996},"min":{"1min":0.57899999999999996,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3440000000000001,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":1.085},{"interface":"front","average":{"1min":1.244,"5min":0.97799999999999998,"15min":0.94499999999999995},"min":{"1min":0.65700000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":0.99199999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.05,"15min":0.99299999999999999},"min":{"1min":0.64600000000000002,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":2.1949999999999998,"5min":2.278,"15min":2.278},"last":1.252},{"interface":"front","average":{"1min":1.087,"5min":1.014,"15min":0.98199999999999998},"min":{"1min":0.78700000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.7090000000000001,"5min":3.4279999999999999,"15min":3.4279999999999999},"last":1.226}]},{"osd":4,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.044,"15min":0.93400000000000005},"min":{"1min":0.67600000000000005,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":6.008,"5min":6.008,"15min":6.008},"last":1.292},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.135,"15min":1.054},"min":{"1min":0.745,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":2.3039999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":0.89400000000000002}]},{"osd":5,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.115,"15min":1.028},"min":{"1min":0.82599999999999996,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":5.9560000000000004,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.194,"15min":1.125},"min":{"1min":0.74299999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":3.5169999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.3540000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:58:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.103,"15min":1.0649999999999999},"min":{"1min":0.76000000000000001,"5min":0.503,"15min":0.503},"max":{"1min":1.3520000000000001,"5min":2.069,"15min":2.069},"last":1.319},{"interface":"front","average":{"1min":1.631,"5min":1.264,"15min":1.1859999999999999},"min":{"1min":0.79100000000000004,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.391}]}]},{"osd":5,"up_from":43,"seq":184683593884,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307968,"kb_used_data":10608,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93437184,"statfs":{"total":95995035648,"available":95679676416,"internally_reserved":0,"allocated":10862592,"data_stored":7918775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1000000000000001,"15min":0.94699999999999995},"min":{"1min":0.52900000000000003,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.347,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.1870000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.012,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3160000000000001,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.95899999999999996}]},{"osd":1,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.6739999999999999,"15min":1.1399999999999999},"min":{"1min":0.67900000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0980000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.071},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.431,"15min":1.0760000000000001},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.135,"5min":31.847000000000001,"15min":31.847000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.8959999999999999,"15min":1.238},"min":{"1min":0.59299999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.22,"5min":61.347000000000001,"15min":61.347000000000001},"last":0.75},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.137,"15min":0.996},"min":{"1min":0.66100000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.23,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.99399999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1339999999999999,"15min":1},"min":{"1min":0.58799999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2769999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.159},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.792,"15min":1.2410000000000001},"min":{"1min":0.72699999999999998,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":1.2270000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":0.92000000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.167,"15min":0.94299999999999995},"min":{"1min":0.52800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2529999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.224,"15min":1.0109999999999999},"min":{"1min":0.67200000000000004,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.0329999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.399,"15min":1.149},"min":{"1min":0.68400000000000005,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":5.2009999999999996,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.4350000000000001,"15min":1.1830000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.47199999999999998},"max":{"1min":5.1769999999999996,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.2589999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.5529999999999999,"15min":1.22},"min":{"1min":0.81200000000000006,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":5.7210000000000001,"5min":15.613,"15min":15.613},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.54,"15min":1.2809999999999999},"min":{"1min":0.75700000000000001,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":4.2480000000000002,"5min":16.965,"15min":16.965},"last":1.2310000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855528,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312128,"kb_used_data":6440,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93433024,"statfs":{"total":95995035648,"available":95675416576,"internally_reserved":0,"allocated":6594560,"data_stored":3725442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.403,"5min":1.546,"15min":1.546},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.91600000000000004,"15min":0.88600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.403,"5min":1.667,"15min":1.7070000000000001},"last":1.6100000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.133,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.89700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2869999999999999,"5min":1.575,"15min":1.575},"last":1.0980000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.93999999999999995,"15min":0.92100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5209999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.4770000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95099999999999996,"15min":0.92400000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":1.274}]},{"osd":3,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.96199999999999997,"15min":0.93600000000000005},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.46300000000000002},"max":{"1min":4.2670000000000003,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.95399999999999996,"15min":0.95099999999999996},"min":{"1min":0.49099999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.3740000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.1399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.94899999999999995,"15min":0.90600000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.3899999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.05,"15min":0.97599999999999998},"min":{"1min":0.57399999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.5209999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.038,"15min":0.98499999999999999},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.127,"15min":1.1020000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.367}]},{"osd":7,"last update":"Tue Sep 10 18:58:57 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.1459999999999999,"15min":1.159},"min":{"1min":0.73899999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.629,"5min":1.629,"15min":2.008},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1419999999999999,"15min":1.1579999999999999},"min":{"1min":0.58499999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.5489999999999999,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":1.028}]}]},{"osd":3,"up_from":29,"seq":124554051763,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303928,"kb_used_data":6568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93441224,"statfs":{"total":95995035648,"available":95683813376,"internally_reserved":0,"allocated":6725632,"data_stored":3725416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.24,"15min":0.82499999999999996},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.86799999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.63,"5min":1.2589999999999999,"15min":0.88900000000000001},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":1.085,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.80400000000000005}]},{"osd":1,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.3919999999999999,"15min":0.91200000000000003},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.26500000000000001},"max":{"1min":0.88200000000000001,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.0740000000000001,"15min":0.871},"min":{"1min":0.52900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0009999999999999,"5min":15.189,"15min":15.189},"last":0.69699999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.381,"15min":0.99099999999999999},"min":{"1min":0.432,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.153,"5min":26.558,"15min":26.558},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.2809999999999999,"15min":0.93100000000000005},"min":{"1min":0.434,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.8540000000000001,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.3600000000000001,"15min":1.137},"min":{"1min":0.73099999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5,"5min":10.359999999999999,"15min":10.359999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.464,"15min":1.224},"min":{"1min":0.81000000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.633,"5min":10.391,"15min":10.391},"last":1.3799999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.46,"15min":1.2130000000000001},"min":{"1min":0.76500000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.498,"5min":17.535,"15min":17.535},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.995,"5min":1.119,"15min":1.034},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6819999999999999,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.73899999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.3,"15min":1.2030000000000001},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6919999999999999,"5min":10.51,"15min":10.51},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1399999999999999,"15min":1.0900000000000001},"min":{"1min":0.63,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5820000000000001,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.016}]},{"osd":7,"last update":"Tue Sep 10 18:58:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.331,"15min":1.177},"min":{"1min":0.621,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.7270000000000001,"5min":18.312999999999999,"15min":18.312999999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.4319999999999999,"15min":1.3},"min":{"1min":0.70099999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.575,"5min":10.611000000000001,"15min":10.611000000000001},"last":1.145}]}]},{"osd":2,"up_from":19,"seq":81604378812,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320200,"kb_used_data":14520,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93424952,"statfs":{"total":95995035648,"available":95667150848,"internally_reserved":0,"allocated":14868480,"data_stored":11919819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.83699999999999997,"15min":0.71599999999999997},"min":{"1min":0.35799999999999998,"5min":0.219,"15min":0.219},"max":{"1min":1.2,"5min":9.516,"15min":9.516},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.81799999999999995,"15min":0.76900000000000002},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.0600000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.64000000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:58:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.75600000000000001,"15min":0.71099999999999997},"min":{"1min":0.38500000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.1339999999999999,"5min":5.516,"15min":5.516},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.83599999999999997,"15min":0.80400000000000005},"min":{"1min":0.41599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.048,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.86799999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.85699999999999998},"min":{"1min":0.503,"5min":0.36599999999999999,"15min":0.34799999999999998},"max":{"1min":1.3859999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.91700000000000004,"15min":0.84299999999999997},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":1.194,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.70699999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2090000000000001,"15min":1.0629999999999999},"min":{"1min":0.49299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":3.851,"5min":21.571999999999999,"15min":21.571999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.675,"5min":1.2869999999999999,"15min":1.149},"min":{"1min":0.75900000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":10.881,"5min":10.881,"15min":10.881},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:59:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.60599999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2589999999999999,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.226,"5min":1.246,"15min":1.103},"min":{"1min":0.63900000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.4980000000000002,"5min":21.654,"15min":21.654},"last":1.0409999999999999}]},{"osd":6,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":0.64400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.5609999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2370000000000001,"15min":1.1299999999999999},"min":{"1min":0.63,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":15.971,"15min":15.971},"last":1.0720000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.59699999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.905,"5min":15.025,"15min":15.025},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.331,"15min":1.1990000000000001},"min":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":6.851,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.0029999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575047,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311904,"kb_used_data":10384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93433248,"statfs":{"total":95995035648,"available":95675645952,"internally_reserved":0,"allocated":10633216,"data_stored":7724661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":1.0409999999999999,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.877,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.135,"15min":0.85899999999999999},"min":{"1min":0.41399999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.7200000000000002,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.879}]},{"osd":2,"last update":"Tue Sep 10 18:58:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.141,"15min":0.84699999999999998},"min":{"1min":0.27200000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.048,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.3480000000000001,"15min":0.97899999999999998},"min":{"1min":0.36599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.0890000000000004,"5min":32.363,"15min":32.363},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.27,"15min":0.98199999999999998},"min":{"1min":0.496,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.7929999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.6000000000000001,"15min":1.0269999999999999},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.26600000000000001},"max":{"1min":1.871,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.75700000000000001}]},{"osd":4,"last update":"Tue Sep 10 18:59:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2470000000000001,"15min":1.0940000000000001},"min":{"1min":0.70599999999999996,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3220000000000001,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.038},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.7589999999999999,"15min":1.2090000000000001},"min":{"1min":0.72199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.544,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.95499999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.284,"15min":1.1020000000000001},"min":{"1min":0.61899999999999999,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":1.4790000000000001,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.3180000000000001,"15min":1.1970000000000001},"min":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7949999999999999,"5min":12.84,"15min":12.84},"last":1.3120000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.5409999999999999,"15min":1.254},"min":{"1min":0.67900000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":3.121,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.244},{"interface":"front","average":{"1min":1.123,"5min":1.5369999999999999,"15min":1.23},"min":{"1min":0.79400000000000004,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.9359999999999999,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.077}]},{"osd":7,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.5229999999999999,"15min":1.3180000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0880000000000001,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.607,"15min":1.304},"min":{"1min":0.66100000000000003,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.839,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.137}]}]},{"osd":0,"up_from":8,"seq":34359738579,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304408,"kb_used_data":7048,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93440744,"statfs":{"total":95995035648,"available":95683321856,"internally_reserved":0,"allocated":7217152,"data_stored":4314533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.649,"15min":0.96199999999999997},"min":{"1min":0.45700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.028,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.746,"15min":1.0620000000000001},"min":{"1min":0.48099999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.6850000000000001,"5min":86.009,"15min":86.009},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.6639999999999999,"15min":1.0209999999999999},"min":{"1min":0.60699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.593,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.2190000000000001,"15min":0.90600000000000003},"min":{"1min":0.57299999999999995,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.333,"5min":24.614000000000001,"15min":24.614000000000001},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1459999999999999,"15min":0.89300000000000002},"min":{"1min":0.48799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.379,"5min":21.620999999999999,"15min":21.620999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.5049999999999999,"15min":1.0349999999999999},"min":{"1min":0.57999999999999996,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.4019999999999999,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.85999999999999999}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1619999999999999,"15min":1.0429999999999999},"min":{"1min":0.71199999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.6839999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.012},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1739999999999999,"15min":1.099},"min":{"1min":0.70499999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.6439999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.82799999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2629999999999999,"15min":1.111},"min":{"1min":0.749,"5min":0.442,"15min":0.442},"max":{"1min":1.288,"5min":10.048,"15min":10.048},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.087,"15min":1.097},"min":{"1min":0.63,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.337,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:59:00 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1599999999999999,"15min":1.1319999999999999},"min":{"1min":0.75900000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5580000000000001,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.141,"5min":1.3240000000000001,"15min":1.1930000000000001},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.5740000000000001,"5min":11.894,"15min":11.894},"last":1.103}]},{"osd":7,"last update":"Tue Sep 10 18:58:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1890000000000001,"15min":1.1699999999999999},"min":{"1min":0.90900000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6899999999999999,"5min":6.2949999999999999,"15min":6.2949999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.3200000000000001,"15min":1.2949999999999999},"min":{"1min":0.84499999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.7310000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.034}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8445952,"data_stored":8194920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4452352,"data_stored":4194928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8433664,"data_stored":8194880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":209,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T18:59:46.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:46.650 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:46.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:46.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:46.772 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:46.774 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T18:59:46.774 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 20, 2024-09-10T18:59:46.774 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 600, 2024-09-10T18:59:46.774 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [ 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.35014.0:2 74.15 74:ad551702:::test:head [create,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.468675+0000", 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 282.30292328199999, 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.073929000999999994, 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.35014", 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/594637325", 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2 2024-09-10T18:59:46.775 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468675+0000", 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468675+0000", 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.776 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468672+0000", 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468683+0000", 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0550000000000001e-05 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.777 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468700+0000", 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6807e-05 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468709+0000", 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8899999999999996e-06 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468763+0000", 2024-09-10T18:59:46.778 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.4539000000000001e-05 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.468772+0000", 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2010000000000005e-06 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.523386+0000", 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.054613566000000002 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.779 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.523491+0000", 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000104848 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.523618+0000", 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00012721299999999999 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.524712+0000", 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001094329 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.780 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.525908+0000", 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0011953210000000001 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.542508+0000", 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016600597000000002 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.781 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.542561+0000", 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.3220000000000002e-05 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.542604+0000", 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.2678e-05 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.35014.0:4 74.15 74:ad551702:::test:head [cmpext 0~4,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T18:59:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.546215+0000", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 282.22538327000001, 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016032649, 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.35014", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/594637325", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 4 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546215+0000", 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.783 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546215+0000", 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546213+0000", 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.784 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546223+0000", 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6739999999999992e-06 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546236+0000", 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3004e-05 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546243+0000", 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.1010000000000003e-06 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.785 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546293+0000", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.0487e-05 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546376+0000", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.3149e-05 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.546547+0000", 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000170512 2024-09-10T18:59:46.786 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.550246+0000", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.003699375 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.550273+0000", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6962999999999999e-05 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.562128+0000", 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011854965 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.787 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.562188+0000", 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.9558000000000001e-05 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.562248+0000", 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.0055e-05 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.788 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.35014.0:6 74.15 74:ad551702:::test:head [cmpext 0~4,write 0~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.570230+0000", 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 282.20136859600001, 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012140022, 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.35014", 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/594637325", 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 6 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.789 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570230+0000", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570230+0000", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570226+0000", 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.790 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570241+0000", 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.508e-05 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570265+0000", 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3793000000000001e-05 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570274+0000", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3840000000000003e-06 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570329+0000", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.4861999999999998e-05 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.570425+0000", 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6058999999999996e-05 2024-09-10T18:59:46.792 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.582370+0000", 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011944362 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.35014.0:7 74.15 74:ad551702:::test:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:04.586220+0000", 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 282.18537815500002, 2024-09-10T18:59:46.793 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0097300939999999999, 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.35014", 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/594637325", 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 7 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586220+0000", 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.794 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586220+0000", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586217+0000", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586232+0000", 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4939999999999999e-05 2024-09-10T18:59:46.795 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586253+0000", 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1634e-05 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586261+0000", 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.6170000000000001e-06 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.796 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586310+0000", 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8998000000000001e-05 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586398+0000", 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7929000000000005e-05 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,4", 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.586514+0000", 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000116338 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.797 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.588217+0000", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001702627 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.590274+0000", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0020571679999999998 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.595852+0000", 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0055776250000000001 2024-09-10T18:59:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.595908+0000", 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.626e-05 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:04.595950+0000", 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.2318999999999999e-05 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.799 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.42494.0:2 82.3 82:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:11.180909+0000", 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 275.590688898, 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.029749075999999999, 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.42494", 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1322748831", 2024-09-10T18:59:46.800 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180909+0000", 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.801 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180909+0000", 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180907+0000", 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180916+0000", 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.1790000000000008e-06 2024-09-10T18:59:46.802 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180927+0000", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1725e-05 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.180935+0000", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.7549999999999994e-06 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.181038+0000", 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000103216 2024-09-10T18:59:46.803 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.181157+0000", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000118374 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 2,3", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.199563+0000", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.018406181000000001 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.200781+0000", 2024-09-10T18:59:46.804 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0012180190000000001 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.204242+0000", 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0034612459999999999 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.210559+0000", 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0063168570000000004 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.805 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.210617+0000", 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7599000000000002e-05 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.210659+0000", 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.1891000000000003e-05 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(osd.6.52:2 11.6 11:621664ec:test-rados-api-smithi083-77788-17::big2:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:59:46.806 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:11.850227+0000", 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 274.921371384, 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012173318000000001, 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850227+0000", 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.807 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850227+0000", 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850222+0000", 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850242+0000", 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9547000000000001e-05 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.808 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850270+0000", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.8464999999999999e-05 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850281+0000", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0096e-05 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850339+0000", 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7976999999999997e-05 2024-09-10T18:59:46.809 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.850436+0000", 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.7540000000000005e-05 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.862400+0000", 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011964216999999999 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.810 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(osd.6.52:3 11.6 11:621664ec:test-rados-api-smithi083-77788-17::big2:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:11.880613+0000", 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 274.89098531899998, 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00775802, 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880613+0000", 2024-09-10T18:59:46.811 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880613+0000", 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.812 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880609+0000", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880627+0000", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.7611000000000001e-05 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880653+0000", 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6821999999999998e-05 2024-09-10T18:59:46.813 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880664+0000", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0134e-05 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880719+0000", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.5515e-05 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.880812+0000", 2024-09-10T18:59:46.814 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2626000000000004e-05 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.888371+0000", 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0075593650000000002 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(osd.6.52:4 11.6 11:621664ec:test-rados-api-smithi083-77788-17::big2:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-09-10T18:59:46.815 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:11.908616+0000", 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 274.86298245500001, 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0083087379999999995, 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908616+0000", 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.816 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908616+0000", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908612+0000", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908634+0000", 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.2684000000000002e-05 2024-09-10T18:59:46.817 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908693+0000", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8659000000000002e-05 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908704+0000", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0565e-05 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908757+0000", 2024-09-10T18:59:46.818 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.3230999999999999e-05 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.908867+0000", 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011048900000000001 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:11.916925+0000", 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0080573350000000005 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.819 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:111 78.10 78:09ab0bed:test-rados-api-smithi083-77828-10::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156)", 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:12.243502+0000", 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 274.52809678099999, 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011976532999999999, 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "reached pg", 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 111 2024-09-10T18:59:46.820 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243502+0000", 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243502+0000", 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.821 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243498+0000", 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243512+0000", 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3818000000000001e-05 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243535+0000", 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3448999999999999e-05 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.822 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243545+0000", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.285e-06 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243610+0000", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.4999999999999994e-05 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.243864+0000", 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00025397499999999998 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.823 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.249229+0000", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0053648580000000001 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.249311+0000", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.2479999999999996e-05 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.255352+0000", 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0060411800000000002 2024-09-10T18:59:46.824 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:12.255478+0000", 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000125871 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.28627.0:51 66.18 66:1ff486be:test-rados-api-smithi083-77943-11::foo:head [write 0~1048576] snapc 10=[10] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:16.968742+0000", 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 269.80285587899999, 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.17914411999999999, 2024-09-10T18:59:46.825 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.28627", 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/4009038338", 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 51 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.968742+0000", 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.826 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.968742+0000", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.968740+0000", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.969520+0000", 2024-09-10T18:59:46.827 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00077990100000000005 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.971683+0000", 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021624999999999999 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.971693+0000", 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.944e-06 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.828 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.972214+0000", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00052166099999999998 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.972332+0000", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011751099999999999 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 4,6", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.972466+0000", 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000134203 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.829 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.977809+0000", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0053428820000000002 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.137291+0000", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.15948166899999999 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.147781+0000", 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.01049012 2024-09-10T18:59:46.830 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.147839+0000", 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8520999999999998e-05 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.147887+0000", 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.7376999999999997e-05 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.831 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDScrubReserve(4.3 GRANT e161)", 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:16.996668+0000", 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 269.77493076399998, 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.135073789, 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996668+0000", 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.832 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996668+0000", 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996667+0000", 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.833 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:46.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:46.834 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.834 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996671+0000", 2024-09-10T18:59:46.834 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8299999999999998e-06 2024-09-10T18:59:46.834 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.834 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996677+0000", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.998e-06 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.996684+0000", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.1119999999999998e-06 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131704+0000", 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.13502046500000001 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.835 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131733+0000", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.8707000000000001e-05 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131736+0000", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.4219999999999999e-06 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131741+0000", 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.835e-06 2024-09-10T18:59:46.836 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDScrubReserve(4.3 GRANT e161)", 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:16.997148+0000", 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 269.774450108, 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.13465908200000001, 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.997148+0000", 2024-09-10T18:59:46.837 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.997148+0000", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.997147+0000", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.999999 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:16.997152+0000", 2024-09-10T18:59:46.838 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8579999999999999e-06 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.130907+0000", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.133754872 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.130915+0000", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.7540000000000006e-06 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131778+0000", 2024-09-10T18:59:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00086280299999999999 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131792+0000", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4035e-05 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131794+0000", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5440000000000001e-06 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:17.131807+0000", 2024-09-10T18:59:46.840 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3101e-05 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:1061 97.d 97:b11a8b5b:test-rados-api-smithi083-77828-14::foo16:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e179)", 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:35.903606+0000", 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 250.86799234099999, 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0074202909999999999, 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:59:46.841 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1061 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903606+0000", 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903606+0000", 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.842 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903603+0000", 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903615+0000", 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2293000000000001e-05 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.843 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903642+0000", 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6747999999999999e-05 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903656+0000", 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3905e-05 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.903776+0000", 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000120041 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.844 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.904307+0000", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00053104499999999995 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.906631+0000", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002324123 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.906842+0000", 2024-09-10T18:59:46.845 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021118800000000001 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.907717+0000", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00087514500000000002 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.907911+0000", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000193592 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.911026+0000", 2024-09-10T18:59:46.846 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0031155559999999998 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:1068 97.12 97:4964f55c:test-rados-api-smithi083-77828-14::foo13:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e179)", 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:35.949997+0000", 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 250.82160098599999, 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0081837200000000002, 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:59:46.847 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1068 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.949997+0000", 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.949997+0000", 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.848 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.949994+0000", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.950006+0000", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1151999999999999e-05 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.950031+0000", 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5452e-05 2024-09-10T18:59:46.849 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.950041+0000", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.4720000000000001e-06 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.950152+0000", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000111505 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.950465+0000", 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031343799999999998 2024-09-10T18:59:46.850 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.953497+0000", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0030315609999999999 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.953687+0000", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019038700000000001 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.955572+0000", 2024-09-10T18:59:46.851 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001884232 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.955640+0000", 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8427000000000001e-05 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.956026+0000", 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00038631000000000001 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.852 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.956116+0000", 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.9324000000000001e-05 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:35.958181+0000", 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0020653669999999998 2024-09-10T18:59:46.853 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.48707.0:1 113.0s0 113:13de00cd:test-rados-api-smithi083-77943-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:48.000263+0000", 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 238.771335284, 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0413487939999999, 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.48707", 2024-09-10T18:59:46.854 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2936751915", 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000263+0000", 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000263+0000", 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.855 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000261+0000", 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000269+0000", 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.0669999999999992e-06 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.856 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000281+0000", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1464999999999999e-05 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.000288+0000", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.267e-06 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.001779+0000", 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0014913190000000001 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.857 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:48.001783+0000", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.7160000000000001e-06 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.036890+0000", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0351068210000001 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.037010+0000", 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000120316 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.858 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.037398+0000", 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00038759200000000003 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.039300+0000", 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001902644 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.859 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.041551+0000", 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0022507569999999999 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:49.041612+0000", 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.0897999999999999e-05 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.860 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24732.0:4 116.0s0 116:0e00b8e4:test-rados-api-smithi083-77788-26::foo.copy:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:54.049016+0000", 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 232.72258276400001, 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.047498298000000001, 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24732", 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/4260234155", 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 4 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.861 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049016+0000", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049016+0000", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049012+0000", 2024-09-10T18:59:46.862 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049024+0000", 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1984000000000001e-05 2024-09-10T18:59:46.863 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049043+0000", 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8184000000000001e-05 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049051+0000", 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.4849999999999992e-06 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.864 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049120+0000", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8398999999999993e-05 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.049253+0000", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000133738 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.096514+0000", 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0472606 2024-09-10T18:59:46.865 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.48887.0:1 143.6s0 143:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:56:26.070230+0000", 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 200.70136839200001, 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0101075349999999, 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.48887", 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/599324055", 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1 2024-09-10T18:59:46.866 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070230+0000", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070230+0000", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070228+0000", 2024-09-10T18:59:46.867 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070237+0000", 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3000000000000007e-06 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070252+0000", 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4752e-05 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.868 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070260+0000", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.3610000000000001e-06 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070329+0000", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8895999999999999e-05 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:26.070344+0000", 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5563999999999999e-05 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.869 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.076531+0000", 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0061867879999999 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.076621+0000", 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.0019000000000007e-05 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T18:59:46.870 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.077072+0000", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00045090800000000003 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.078104+0000", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001031967 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.080300+0000", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021963439999999998 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:56:27.080337+0000", 2024-09-10T18:59:46.871 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.7041e-05 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:2790 179.5 179:a77be4ab:test-rados-api-smithi083-77828-34::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:57:41.083925+0000", 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 125.68767328200001, 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.026126432000000002, 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:59:46.872 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2790 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083925+0000", 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083925+0000", 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.873 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083921+0000", 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083938+0000", 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.7413000000000001e-05 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.874 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083970+0000", 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.1454000000000001e-05 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.083981+0000", 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.135e-05 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.084061+0000", 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.9734000000000006e-05 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.875 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.084068+0000", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.588e-06 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.094802+0000", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0107338 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.094979+0000", 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000176596 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.876 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.107333+0000", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012353784 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.107420+0000", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.7150000000000004e-05 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 1,5", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.107530+0000", 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000110685 2024-09-10T18:59:46.877 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.108024+0000", 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00049399600000000002 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.108267+0000", 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024249499999999999 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.878 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.109971+0000", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001704283 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.110004+0000", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.2931e-05 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:41.110051+0000", 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.7397e-05 2024-09-10T18:59:46.879 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.27490.0:3169 64.1e 64:7e27adf5:test-rados-api-smithi083-77828-36::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e316)", 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:57:52.085795+0000", 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 114.685803745, 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.010900155999999999, 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.27490", 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 3169 2024-09-10T18:59:46.880 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085795+0000", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085795+0000", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085788+0000", 2024-09-10T18:59:46.881 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999938 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085813+0000", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.4615999999999999e-05 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085847+0000", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.4069000000000002e-05 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085865+0000", 2024-09-10T18:59:46.882 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8130000000000001e-05 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085976+0000", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000110441 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.085985+0000", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3889999999999994e-06 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.096390+0000", 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.010405368 2024-09-10T18:59:46.883 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.096480+0000", 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.9617000000000006e-05 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:52.096695+0000", 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000214668 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.884 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:1651 4.3 4:c609908c:::notify.5:head [watch ping cookie 94430446498256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:59:46.286871+0000", 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 0.48472691299999998, 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.012911077, 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T18:59:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1651 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286871+0000", 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286871+0000", 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.886 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286867+0000", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999952 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286886+0000", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9942999999999999e-05 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286921+0000", 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.4521999999999999e-05 2024-09-10T18:59:46.887 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.286943+0000", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.2103000000000002e-05 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.298482+0000", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.011538859 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.298594+0000", 2024-09-10T18:59:46.888 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000112235 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:59:46.299782+0000", 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0011883309999999999 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T18:59:46.889 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T18:59:46.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:46.999 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:47.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:47.139 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:47.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:47.283 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:47.386 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:47.417 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:47.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:47.533 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[18629]: cluster 2024-09-10T18:59:46.082286+0000 mgr.y (mgr.24526) 424 : cluster [DBG] pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:47.533 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[18629]: cluster 2024-09-10T18:59:46.274513+0000 mon.a (mon.0) 2541 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T18:59:47.533 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[18629]: cluster 2024-09-10T18:59:46.292616+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2024-09-10T18:59:47.534 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[32381]: cluster 2024-09-10T18:59:46.082286+0000 mgr.y (mgr.24526) 424 : cluster [DBG] pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:47.534 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[32381]: cluster 2024-09-10T18:59:46.274513+0000 mon.a (mon.0) 2541 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T18:59:47.534 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:47 smithi083 bash[32381]: cluster 2024-09-10T18:59:46.292616+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2024-09-10T18:59:47.554 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:47.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:47 smithi165 bash[22591]: cluster 2024-09-10T18:59:46.082286+0000 mgr.y (mgr.24526) 424 : cluster [DBG] pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T18:59:47.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:47 smithi165 bash[22591]: cluster 2024-09-10T18:59:46.274513+0000 mon.a (mon.0) 2541 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T18:59:47.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:47 smithi165 bash[22591]: cluster 2024-09-10T18:59:46.292616+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e423: 8 total, 8 up, 8 in 2024-09-10T18:59:47.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:47.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:47.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:47.825 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:47.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:47.963 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:47.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:48.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.100 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:48.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:48.237 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:48.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:48.364 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:48.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:48.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:48 smithi165 bash[22591]: audit 2024-09-10T18:59:47.298765+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:48 smithi165 bash[22591]: cluster 2024-09-10T18:59:47.300174+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2024-09-10T18:59:48.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:48 smithi165 bash[22591]: audit 2024-09-10T18:59:47.302682+0000 mon.a (mon.0) 2544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.493 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T18:59:48.505 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[18629]: audit 2024-09-10T18:59:47.298765+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[18629]: cluster 2024-09-10T18:59:47.300174+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[18629]: audit 2024-09-10T18:59:47.302682+0000 mon.a (mon.0) 2544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[32381]: audit 2024-09-10T18:59:47.298765+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[32381]: cluster 2024-09-10T18:59:47.300174+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e424: 8 total, 8 up, 8 in 2024-09-10T18:59:48.576 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:48 smithi083 bash[32381]: audit 2024-09-10T18:59:47.302682+0000 mon.a (mon.0) 2544 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:48.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:48.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.777 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:48.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:48.914 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:48.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:49.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:49.051 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:49.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:49.186 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:49.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:49.323 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:49.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:49.459 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:49.562 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:49.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T18:59:49.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:49.721 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:49.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:47.949887+0000 mgr.y (mgr.24526) 425 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: cluster 2024-09-10T18:59:48.083285+0000 mgr.y (mgr.24526) 426 : cluster [DBG] pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: cluster 2024-09-10T18:59:48.294400+0000 mon.a (mon.0) 2545 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:48.312210+0000 mon.a (mon.0) 2546 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: cluster 2024-09-10T18:59:48.312575+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:48.358355+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:48.359767+0000 mon.b (mon.1) 330 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:48.361951+0000 mon.a (mon.0) 2548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: audit 2024-09-10T18:59:48.363209+0000 mon.a (mon.0) 2549 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[18629]: cluster 2024-09-10T18:59:49.125196+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:47.949887+0000 mgr.y (mgr.24526) 425 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: cluster 2024-09-10T18:59:48.083285+0000 mgr.y (mgr.24526) 426 : cluster [DBG] pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: cluster 2024-09-10T18:59:48.294400+0000 mon.a (mon.0) 2545 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:48.312210+0000 mon.a (mon.0) 2546 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: cluster 2024-09-10T18:59:48.312575+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:48.358355+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:48.359767+0000 mon.b (mon.1) 330 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:48.361951+0000 mon.a (mon.0) 2548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: audit 2024-09-10T18:59:48.363209+0000 mon.a (mon.0) 2549 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:49 smithi083 bash[32381]: cluster 2024-09-10T18:59:49.125196+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2024-09-10T18:59:49.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:49.862 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:47.949887+0000 mgr.y (mgr.24526) 425 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: cluster 2024-09-10T18:59:48.083285+0000 mgr.y (mgr.24526) 426 : cluster [DBG] pgmap v624: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: cluster 2024-09-10T18:59:48.294400+0000 mon.a (mon.0) 2545 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:48.312210+0000 mon.a (mon.0) 2546 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: cluster 2024-09-10T18:59:48.312575+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e425: 8 total, 8 up, 8 in 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:48.358355+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:48.359767+0000 mon.b (mon.1) 330 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:48.361951+0000 mon.a (mon.0) 2548 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: audit 2024-09-10T18:59:48.363209+0000 mon.a (mon.0) 2549 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-53"}]: dispatch 2024-09-10T18:59:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:49 smithi165 bash[22591]: cluster 2024-09-10T18:59:49.125196+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e426: 8 total, 8 up, 8 in 2024-09-10T18:59:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:49.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:49.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:50.001 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:50.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:50.143 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:50.245 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:50.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:50.418 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:50.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:50.557 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:50.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:50.695 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:50.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:50.831 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:50.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3728843427 >> v1:172.21.15.83:6790/0 conn(0x7f33fc10caa0 legacy=0x7f33fc10ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3728843427 shutdown_connections 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3728843427 >> 172.21.15.83:0/3728843427 conn(0x7f33fc06b990 msgr2=0x7f33fc06bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3728843427 shutdown_connections 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3728843427 wait complete. 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 librados: starting msgr at 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 librados: starting objecter 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 Processor -- start 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- start start 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 librados: setting wanted keys 2024-09-10T18:59:50.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 librados: calling monclient init 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc109710 con 0x7f33fc110630 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc127ee0 con 0x7f33fc10caa0 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33fc1290e0 con 0x7f33fc109d30 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33fbfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f33fc109d30 0x7f33fc108b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35930/0 (socket says 172.21.15.83:35930) 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33fbfff640 1 -- 172.21.15.83:0/3782810308 learned_addr learned my addr 172.21.15.83:0/3782810308 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter ms_handle_connect 0x7f33fc109d30 2024-09-10T18:59:50.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter ms_handle_connect 0x7f33fc110630 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter ms_handle_connect 0x7f33fc10caa0 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321458896 0 0) 0x7f33fc1290e0 con 0x7f33fc109d30 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e80054b0 con 0x7f33fc109d30 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284442464 0 0) 0x7f33fc109710 con 0x7f33fc110630 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc1290e0 con 0x7f33fc110630 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380517829 0 0) 0x7f33fc127ee0 con 0x7f33fc10caa0 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc109710 con 0x7f33fc10caa0 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094230628 0 0) 0x7f33e80054b0 con 0x7f33fc109d30 2024-09-10T18:59:50.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33fc127ee0 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f33ec001900 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 616136979 0 0) 0x7f33fc1290e0 con 0x7f33fc110630 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e80054b0 con 0x7f33fc110630 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 939986434 0 0) 0x7f33fc127ee0 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.165:6789/0 conn(0x7f33fc10caa0 legacy=0x7f33fc122060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6789/0 conn(0x7f33fc110630 legacy=0x7f33fc125790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc12a2e0 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc126f10 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 10 client.49376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1273e0 con 0x7f33fc109d30 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f34024d3640 1 librados: init done 2024-09-10T18:59:50.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f33ec001c90 con 0x7f33fc109d30 2024-09-10T18:59:50.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.924+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f33ec005bd0 con 0x7f33fc109d30 2024-09-10T18:59:50.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f33dc004ff0 con 0x7f33fc109d30 2024-09-10T18:59:50.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f33ec01bff0 con 0x7f33fc109d30 2024-09-10T18:59:50.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- 172.21.15.83:0/1525717550 >> v1:172.21.15.83:6790/0 conn(0x7f1d8c1073c0 legacy=0x7f1d8c1097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(427..427 src has 69..427) v4 ==== 7975+0+0 (unknown 2320303674 0 0) 0x7f33ec08a5e0 con 0x7f33fc109d30 2024-09-10T18:59:50.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 osd_map(427..427 src has 69..427) v4 2024-09-10T18:59:50.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 3 client.49376.objecter handle_osd_map got epochs [427,427] > 0 2024-09-10T18:59:50.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 3 client.49376.objecter handle_osd_map decoding full epoch 427 2024-09-10T18:59:50.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- 172.21.15.83:0/1525717550 shutdown_connections 2024-09-10T18:59:50.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- 172.21.15.83:0/1525717550 >> 172.21.15.83:0/1525717550 conn(0x7f1d8c0fa200 msgr2=0x7f1d8c0fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:50.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 20 client.49376.objecter dump_active .. 0 homeless 2024-09-10T18:59:50.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v427) v1 ==== 54+0+16 (unknown 3296173923 0 2420000192) 0x7f33ec08c9d0 con 0x7f33fc109d30 2024-09-10T18:59:50.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- 172.21.15.83:0/1525717550 shutdown_connections 2024-09-10T18:59:50.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- 172.21.15.83:0/1525717550 wait complete. 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33e8071550 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 10 client.49376.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 1 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 1 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 0 1 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 4 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 10 client.49376.objecter _send_command 1 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc005290 con 0x7f33dc004610 2024-09-10T18:59:50.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 librados: starting msgr at 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 librados: starting objecter 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 Processor -- start 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 -- start start 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 librados: setting wanted keys 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f1d92d9d640 1 librados: calling monclient init 2024-09-10T18:59:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.928+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc004610 2024-09-10T18:59:50.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d8c075a90 con 0x7f1d8c104600 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d8c075c40 con 0x7f1d8c1073c0 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d8c124db0 con 0x7f1d8c10af50 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d9259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f1d8c10af50 0x7f1d8c075290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35950/0 (socket says 172.21.15.83:35950) 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d91d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d8c104600 0x7f1d8c1036c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:51982/0 (socket says 172.21.15.83:51982) 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d9259c640 1 -- 172.21.15.83:0/886341207 learned_addr learned my addr 172.21.15.83:0/886341207 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter ms_handle_connect 0x7f1d8c104600 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter ms_handle_connect 0x7f1d8c10af50 2024-09-10T18:59:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111699852 0 0) 0x7f1d8c075a90 con 0x7f1d8c104600 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d78003720 con 0x7f1d8c104600 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter ms_handle_connect 0x7f1d8c1073c0 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281281651 0 0) 0x7f1d8c124db0 con 0x7f1d8c10af50 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c075a90 con 0x7f1d8c10af50 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271097994 0 0) 0x7f1d8c075c40 con 0x7f1d8c1073c0 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c124db0 con 0x7f1d8c1073c0 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912419302 0 0) 0x7f1d78003720 con 0x7f1d8c104600 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d8c075c40 con 0x7f1d8c104600 2024-09-10T18:59:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d880018e0 con 0x7f1d8c104600 2024-09-10T18:59:50.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc005290 con 0x7f33dc004610 2024-09-10T18:59:50.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 1: 0 ) v1 2024-09-10T18:59:50.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f33f97fa640 10 client.49376.objecter _finish_command 1 = system:0 2024-09-10T18:59:50.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 4 2024-09-10T18:59:50.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 0 1 2024-09-10T18:59:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 907502341 0 0) 0x7f1d8c075a90 con 0x7f1d8c10af50 2024-09-10T18:59:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d78003720 con 0x7f1d8c10af50 2024-09-10T18:59:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d84001c80 con 0x7f1d8c10af50 2024-09-10T18:59:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 582101086 0 0) 0x7f1d8c075c40 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 >> v1:172.21.15.83:6790/0 conn(0x7f1d8c10af50 legacy=0x7f1d8c075290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 >> v1:172.21.15.165:6789/0 conn(0x7f1d8c1073c0 legacy=0x7f1d8c074b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c125f50 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c124fa0 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 10 client.25311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c1253f0 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 librados: init done 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1d88001fe0 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d880059a0 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d70005010 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.932+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1d88001b80 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(427..427 src has 69..427) v4 ==== 7975+0+0 (unknown 2320303674 0 0) 0x7f1d8808a510 con 0x7f1d8c104600 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 10 client.25311.objecter ms_dispatch 0x7f1d8c000b90 osd_map(427..427 src has 69..427) v4 2024-09-10T18:59:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 3 client.25311.objecter handle_osd_map got epochs [427,427] > 0 2024-09-10T18:59:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 3 client.25311.objecter handle_osd_map decoding full epoch 427 2024-09-10T18:59:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 20 client.25311.objecter dump_active .. 0 homeless 2024-09-10T18:59:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.936+0000 7f1d827fc640 10 client.25311.objecter ms_handle_connect 0x7f1d780716d0 2024-09-10T18:59:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.940+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1d880566d0 con 0x7f1d8c104600 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 10 client.49376.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 2 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 2 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc005810 osd=0 3 2024-09-10T18:59:50.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 0 2 2024-09-10T18:59:50.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 4 2024-09-10T18:59:50.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 10 client.49376.objecter _send_command 2 2024-09-10T18:59:50.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc004cc0 con 0x7f33dc004610 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc004cc0 con 0x7f33dc004610 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 10 client.49376.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 4 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 0 2 2024-09-10T18:59:50.952 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T18:59:50.953 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T18:59:50.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 10 client.49376.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:50.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 3 2024-09-10T18:59:50.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 1 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 4 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 10 client.49376.objecter _send_command 3 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc0022b0 con 0x7f33dc004110 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc004110 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc0022b0 con 0x7f33dc004110 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 3: 0 ) v1 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 10 client.49376.objecter _finish_command 3 = system:0 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 4 2024-09-10T18:59:50.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.948+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 1 3 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 10 client.49376.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 4 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 4 2024-09-10T18:59:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc003fa0 osd=1 3 2024-09-10T18:59:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 1 4 2024-09-10T18:59:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 4 2024-09-10T18:59:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 10 client.49376.objecter _send_command 4 2024-09-10T18:59:50.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.968+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc001890 con 0x7f33dc004110 2024-09-10T18:59:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc001890 con 0x7f33dc004110 2024-09-10T18:59:50.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 10 client.49376.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 4 2024-09-10T18:59:50.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 1 4 2024-09-10T18:59:50.975 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 10 client.49376.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 5 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 5 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 2 5 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 4 2024-09-10T18:59:50.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 10 client.49376.objecter _send_command 5 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc003d90 con 0x7f33dc00d350 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc00d350 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc003d90 con 0x7f33dc00d350 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 5: 0 ) v1 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 10 client.49376.objecter _finish_command 5 = system:0 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 4 2024-09-10T18:59:50.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.972+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 2 5 2024-09-10T18:59:50.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:50.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 10 client.49376.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 6 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 6 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0035e0 osd=2 3 2024-09-10T18:59:50.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 2 6 2024-09-10T18:59:50.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 4 2024-09-10T18:59:50.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 10 client.49376.objecter _send_command 6 2024-09-10T18:59:50.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc0030d0 con 0x7f33dc00d350 2024-09-10T18:59:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc0030d0 con 0x7f33dc00d350 2024-09-10T18:59:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 10 client.49376.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 4 2024-09-10T18:59:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 2 6 2024-09-10T18:59:50.998 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T18:59:50.998 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T18:59:50.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 10 client.49376.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:50.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 7 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 7 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 3 7 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 4 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 10 client.49376.objecter _send_command 7 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc00d0d0 con 0x7f33dc011fd0 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.992+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc011fd0 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.996+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc00d0d0 con 0x7f33dc011fd0 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.996+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 7: 0 ) v1 2024-09-10T18:59:50.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.996+0000 7f33f97fa640 10 client.49376.objecter _finish_command 7 = system:0 2024-09-10T18:59:51.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.996+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 4 2024-09-10T18:59:51.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:50.996+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 3 7 2024-09-10T18:59:51.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 10 client.49376.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:51.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 8 2024-09-10T18:59:51.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 8 2024-09-10T18:59:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc00c990 osd=3 3 2024-09-10T18:59:51.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 3 8 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 4 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 10 client.49376.objecter _send_command 8 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc0132d0 con 0x7f33dc011fd0 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc0132d0 con 0x7f33dc011fd0 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f33f97fa640 10 client.49376.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:51.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 4 2024-09-10T18:59:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 3 8 2024-09-10T18:59:51.014 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T18:59:51.014 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 10 client.49376.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 9 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 9 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 4 9 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 4 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 10 client.49376.objecter _send_command 9 2024-09-10T18:59:51.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.008+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc01a210 con 0x7f33dc016710 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc016710 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc01a210 con 0x7f33dc016710 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 9: 0 ) v1 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 10 client.49376.objecter _finish_command 9 = system:0 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 4 2024-09-10T18:59:51.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.012+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 4 9 2024-09-10T18:59:51.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 10 client.49376.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 10 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 10 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0132d0 osd=4 3 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 4 10 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 4 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 10 client.49376.objecter _send_command 10 2024-09-10T18:59:51.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.024+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc01a210 con 0x7f33dc016710 2024-09-10T18:59:51.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc01a210 con 0x7f33dc016710 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 10 client.49376.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 4 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 4 10 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T18:59:51.032 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 10 client.49376.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 11 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 11 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 5 11 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 4 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 10 client.49376.objecter _send_command 11 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc01e570 con 0x7f33dc01a890 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc01a890 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc01e570 con 0x7f33dc01a890 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 11: 0 ) v1 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 10 client.49376.objecter _finish_command 11 = system:0 2024-09-10T18:59:51.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 4 2024-09-10T18:59:51.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.028+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 5 11 2024-09-10T18:59:51.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 10 client.49376.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 12 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 12 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc01a210 osd=5 3 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 5 12 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 4 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 10 client.49376.objecter _send_command 12 2024-09-10T18:59:51.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc01fe10 con 0x7f33dc01a890 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc01fe10 con 0x7f33dc01a890 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f33f97fa640 10 client.49376.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 4 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 5 12 2024-09-10T18:59:51.057 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 10 client.49376.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 13 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 13 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 6 13 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 4 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 10 client.49376.objecter _send_command 13 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.052+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc0209d0 con 0x7f33dc0204c0 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc0204c0 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc0209d0 con 0x7f33dc0204c0 2024-09-10T18:59:51.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 13: 0 ) v1 2024-09-10T18:59:51.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 10 client.49376.objecter _finish_command 13 = system:0 2024-09-10T18:59:51.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 4 2024-09-10T18:59:51.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.056+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 6 13 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 10 client.49376.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 14 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 14 2024-09-10T18:59:51.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc01fe10 osd=6 3 2024-09-10T18:59:51.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 6 14 2024-09-10T18:59:51.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 4 2024-09-10T18:59:51.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 10 client.49376.objecter _send_command 14 2024-09-10T18:59:51.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.068+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc0209d0 con 0x7f33dc0204c0 2024-09-10T18:59:51.075 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc0209d0 con 0x7f33dc0204c0 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 10 client.49376.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 4 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 6 14 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 10 client.49376.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 15 2024-09-10T18:59:51.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 15 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 7 15 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 4 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 10 client.49376.objecter _send_command 15 2024-09-10T18:59:51.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33dc026df0 con 0x7f33dc020bb0 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 10 client.49376.objecter ms_handle_connect 0x7f33dc020bb0 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f33dc026df0 con 0x7f33dc020bb0 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 15: 0 ) v1 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 10 client.49376.objecter _finish_command 15 = system:0 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 4 2024-09-10T18:59:51.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.072+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 7 15 2024-09-10T18:59:51.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign -1 16 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter _get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 15 client.49376.objecter _session_command_op_remove -1 16 2024-09-10T18:59:51.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter get_session s=0x7f33dc0209d0 osd=7 3 2024-09-10T18:59:51.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 15 client.49376.objecter _session_command_op_assign 7 16 2024-09-10T18:59:51.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 4 2024-09-10T18:59:51.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter _send_command 16 2024-09-10T18:59:51.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33dc00c240 con 0x7f33dc020bb0 2024-09-10T18:59:51.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f33f97fa640 1 -- 172.21.15.83:0/3782810308 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f33dc00c240 con 0x7f33dc020bb0 2024-09-10T18:59:51.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f33f97fa640 10 client.49376.objecter ms_dispatch 0x7f33fc000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T18:59:51.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f33f97fa640 10 client.49376.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T18:59:51.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f33f97fa640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 4 2024-09-10T18:59:51.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f33f97fa640 15 client.49376.objecter _session_command_op_remove 7 16 2024-09-10T18:59:51.110 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T18:59:51.110 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 librados: watch_flush enter 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 librados: watch_flush exit 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter close_session for osd.0 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6801/2535527483 conn(0x7f33dc004610 legacy=0x7f33dc002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc005810 osd=0 1 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter close_session for osd.1 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6805/2309624637 conn(0x7f33dc004110 legacy=0x7f33dc00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc003fa0 osd=1 1 2024-09-10T18:59:51.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter close_session for osd.2 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6809/386094220 conn(0x7f33dc00d350 legacy=0x7f33dc00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0035e0 osd=2 1 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 10 client.49376.objecter close_session for osd.3 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f3400a70640 1 -- 172.21.15.83:0/3782810308 reap_dead start 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.104+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6813/1279688021 conn(0x7f33dc011fd0 legacy=0x7f33dc012400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc00c990 osd=3 1 2024-09-10T18:59:51.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 10 client.49376.objecter close_session for osd.4 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.165:6800/3225515265 conn(0x7f33dc016710 legacy=0x7f33dc018af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0132d0 osd=4 1 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 10 client.49376.objecter close_session for osd.5 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.165:6804/571173777 conn(0x7f33dc01a890 legacy=0x7f33dc01ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01a210 osd=5 1 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 10 client.49376.objecter close_session for osd.6 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.165:6808/1317922936 conn(0x7f33dc0204c0 legacy=0x7f33dc01e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc01fe10 osd=6 1 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 10 client.49376.objecter close_session for osd.7 2024-09-10T18:59:51.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.165:6812/1610151650 conn(0x7f33dc020bb0 legacy=0x7f33dc025570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter put_session s=0x7f33dc0209d0 osd=7 1 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 20 client.49376.objecter shutdown clearing up homeless session... 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 10 client.49376.objecter successfully canceled tick 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f3400a70640 1 -- 172.21.15.83:0/3782810308 reap_dead start 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6800/3071793735 conn(0x7f33e8071550 legacy=0x7f33e8073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> v1:172.21.15.83:6790/0 conn(0x7f33fc109d30 legacy=0x7f33fc108b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:51.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 shutdown_connections 2024-09-10T18:59:51.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 >> 172.21.15.83:0/3782810308 conn(0x7f33fc06b990 msgr2=0x7f33fc06c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:51.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 shutdown_connections 2024-09-10T18:59:51.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 -- 172.21.15.83:0/3782810308 wait complete. 2024-09-10T18:59:51.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.108+0000 7f34024d3640 1 librados: shutdown 2024-09-10T18:59:51.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:51.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:51.144+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f1d700052c0 con 0x7f1d8c104600 2024-09-10T18:59:51.221 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:51.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: cluster 2024-09-10T18:59:49.926167+0000 osd.7 (osd.7) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: cluster 2024-09-10T18:59:49.926529+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: cluster 2024-09-10T18:59:50.084087+0000 mgr.y (mgr.24526) 427 : cluster [DBG] pgmap v627: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: cluster 2024-09-10T18:59:50.131696+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: audit 2024-09-10T18:59:50.132304+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: audit 2024-09-10T18:59:50.136553+0000 mon.a (mon.0) 2552 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: audit 2024-09-10T18:59:50.931980+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? v1:172.21.15.83:0/3782810308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:51.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[22591]: audit 2024-09-10T18:59:51.081433+0000 mon.c (mon.2) 617 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:51.379 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:51.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:51.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: cluster 2024-09-10T18:59:49.926167+0000 osd.7 (osd.7) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:51.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: cluster 2024-09-10T18:59:49.926529+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:51.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: cluster 2024-09-10T18:59:50.084087+0000 mgr.y (mgr.24526) 427 : cluster [DBG] pgmap v627: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: cluster 2024-09-10T18:59:50.131696+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: audit 2024-09-10T18:59:50.132304+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: audit 2024-09-10T18:59:50.136553+0000 mon.a (mon.0) 2552 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: audit 2024-09-10T18:59:50.931980+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? v1:172.21.15.83:0/3782810308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[18629]: audit 2024-09-10T18:59:51.081433+0000 mon.c (mon.2) 617 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: cluster 2024-09-10T18:59:49.926167+0000 osd.7 (osd.7) 236 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: cluster 2024-09-10T18:59:49.926529+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: cluster 2024-09-10T18:59:50.084087+0000 mgr.y (mgr.24526) 427 : cluster [DBG] pgmap v627: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: cluster 2024-09-10T18:59:50.131696+0000 mon.a (mon.0) 2551 : cluster [DBG] osdmap e427: 8 total, 8 up, 8 in 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: audit 2024-09-10T18:59:50.132304+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: audit 2024-09-10T18:59:50.136553+0000 mon.a (mon.0) 2552 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:51.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: audit 2024-09-10T18:59:50.931980+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? v1:172.21.15.83:0/3782810308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T18:59:51.496 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:51 smithi083 bash[32381]: audit 2024-09-10T18:59:51.081433+0000 mon.c (mon.2) 617 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:51.511 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:51.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:51.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:51.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:51.773 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:51.863 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 18:59:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T18:59:51.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:51.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:51.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:52.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T18:59:52.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.132+0000 7f1d827fc640 1 -- 172.21.15.83:0/886341207 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v429) v1 ==== 91+0+0 (unknown 1824500944 0 0) 0x7f1d880564f0 con 0x7f1d8c104600 2024-09-10T18:59:52.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 10 librados: watch_flush enter 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 10 librados: watch_flush exit 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 20 client.25311.objecter shutdown clearing up homeless session... 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 10 client.25311.objecter successfully canceled tick 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1d780716d0 legacy=0x7f1d78073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 >> v1:172.21.15.83:6789/0 conn(0x7f1d8c104600 legacy=0x7f1d8c1036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 shutdown_connections 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.136+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 >> 172.21.15.83:0/886341207 conn(0x7f1d8c0fa200 msgr2=0x7f1d8c10b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.140+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 shutdown_connections 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.140+0000 7f1d92d9d640 1 -- 172.21.15.83:0/886341207 wait complete. 2024-09-10T18:59:52.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T18:59:52.140+0000 7f1d92d9d640 1 librados: shutdown 2024-09-10T18:59:52.172 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:52.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.131429+0000 mon.a (mon.0) 2553 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:52.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: cluster 2024-09-10T18:59:51.131739+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2024-09-10T18:59:52.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.132388+0000 mon.b (mon.1) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.156613+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.192614+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.193940+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.196200+0000 mon.a (mon.0) 2556 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.197337+0000 mon.a (mon.0) 2557 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.395710+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.403616+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.841130+0000 mon.c (mon.2) 618 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.842659+0000 mon.c (mon.2) 619 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:52.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[32381]: audit 2024-09-10T18:59:51.852794+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.131429+0000 mon.a (mon.0) 2553 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: cluster 2024-09-10T18:59:51.131739+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.132388+0000 mon.b (mon.1) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.156613+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.192614+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.193940+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.196200+0000 mon.a (mon.0) 2556 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.197337+0000 mon.a (mon.0) 2557 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.395710+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.403616+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.841130+0000 mon.c (mon.2) 618 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.842659+0000 mon.c (mon.2) 619 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:52.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18629]: audit 2024-09-10T18:59:51.852794+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.305 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:52.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.131429+0000 mon.a (mon.0) 2553 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: cluster 2024-09-10T18:59:51.131739+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e428: 8 total, 8 up, 8 in 2024-09-10T18:59:52.422 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.132388+0000 mon.b (mon.1) 332 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.156613+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.192614+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.193940+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.196200+0000 mon.a (mon.0) 2556 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.197337+0000 mon.a (mon.0) 2557 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-55"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.395710+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.403616+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.841130+0000 mon.c (mon.2) 618 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.842659+0000 mon.c (mon.2) 619 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:52.423 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:52 smithi165 bash[22591]: audit 2024-09-10T18:59:51.852794+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T18:59:52.441 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:52.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:52.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:52.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:52.711 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:52.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.827 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 18:59:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:18:59:52] "GET /metrics HTTP/1.1" 200 228869 "" "Prometheus/2.43.0" 2024-09-10T18:59:52.851 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:52.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.987 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:52.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:53.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:53.119 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:53.222 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:53.251 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:53.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:53 smithi165 bash[22591]: cluster 2024-09-10T18:59:52.085013+0000 mgr.y (mgr.24526) 428 : cluster [DBG] pgmap v630: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:53 smithi165 bash[22591]: cluster 2024-09-10T18:59:52.128021+0000 mon.a (mon.0) 2561 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:53 smithi165 bash[22591]: audit 2024-09-10T18:59:52.135347+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:59:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:53 smithi165 bash[22591]: cluster 2024-09-10T18:59:52.135480+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:59:53.383 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:53.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:53.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[18629]: cluster 2024-09-10T18:59:52.085013+0000 mgr.y (mgr.24526) 428 : cluster [DBG] pgmap v630: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:53.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[18629]: cluster 2024-09-10T18:59:52.128021+0000 mon.a (mon.0) 2561 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:53.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[18629]: audit 2024-09-10T18:59:52.135347+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:59:53.500 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[18629]: cluster 2024-09-10T18:59:52.135480+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:59:53.500 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[32381]: cluster 2024-09-10T18:59:52.085013+0000 mgr.y (mgr.24526) 428 : cluster [DBG] pgmap v630: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:53.501 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[32381]: cluster 2024-09-10T18:59:52.128021+0000 mon.a (mon.0) 2561 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T18:59:53.501 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[32381]: audit 2024-09-10T18:59:52.135347+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? v1:172.21.15.83:0/886341207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T18:59:53.501 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:53 smithi083 bash[32381]: cluster 2024-09-10T18:59:52.135480+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e429: 8 total, 8 up, 8 in 2024-09-10T18:59:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:53.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:53.614 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:53.625 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:53.659 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:53.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:53.797 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:53.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:53.936 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:53.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:54.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:54.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:54.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T18:59:54.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:54.204 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:54.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:54.338 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:54.441 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:54.454 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.162326+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: audit 2024-09-10T18:59:53.171023+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: audit 2024-09-10T18:59:53.175020+0000 mon.a (mon.0) 2565 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.177644+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.179198+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.414055+0000 osd.5 (osd.5) 219 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.417535+0000 osd.5 (osd.5) 220 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.691126+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.691530+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[32381]: cluster 2024-09-10T18:59:54.114231+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.162326+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:59:54.455 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: audit 2024-09-10T18:59:53.171023+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: audit 2024-09-10T18:59:53.175020+0000 mon.a (mon.0) 2565 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.177644+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.179198+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.414055+0000 osd.5 (osd.5) 219 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.417535+0000 osd.5 (osd.5) 220 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.691126+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.691530+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:54.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:54 smithi083 bash[18629]: cluster 2024-09-10T18:59:54.114231+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:54.476 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:54.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:54.579 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.162326+0000 mon.a (mon.0) 2564 : cluster [DBG] osdmap e430: 8 total, 8 up, 8 in 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: audit 2024-09-10T18:59:53.171023+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: audit 2024-09-10T18:59:53.175020+0000 mon.a (mon.0) 2565 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.177644+0000 osd.3 (osd.3) 250 : cluster [DBG] 4.10 scrub starts 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.179198+0000 osd.3 (osd.3) 251 : cluster [DBG] 4.10 scrub ok 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.414055+0000 osd.5 (osd.5) 219 : cluster [DBG] 12.11 scrub starts 2024-09-10T18:59:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.417535+0000 osd.5 (osd.5) 220 : cluster [DBG] 12.11 scrub ok 2024-09-10T18:59:54.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.691126+0000 osd.1 (osd.1) 247 : cluster [DBG] purged_snaps scrub starts 2024-09-10T18:59:54.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.691530+0000 osd.1 (osd.1) 248 : cluster [DBG] purged_snaps scrub ok 2024-09-10T18:59:54.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:54 smithi165 bash[22591]: cluster 2024-09-10T18:59:54.114231+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T18:59:54.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:54.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:54.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:54.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:54.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:54.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:54.969 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T18:59:54.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:55.022 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:55.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:55.151 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:55.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:55.286 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:55.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.986396+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:53.990019+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:54.085638+0000 mgr.y (mgr.24526) 429 : cluster [DBG] pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: audit 2024-09-10T18:59:54.175282+0000 mon.a (mon.0) 2567 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:54.175715+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: audit 2024-09-10T18:59:54.224792+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: audit 2024-09-10T18:59:54.226031+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: audit 2024-09-10T18:59:54.228336+0000 mon.a (mon.0) 2569 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: audit 2024-09-10T18:59:54.229381+0000 mon.a (mon.0) 2570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:54.412052+0000 osd.5 (osd.5) 221 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[18629]: cluster 2024-09-10T18:59:54.415289+0000 osd.5 (osd.5) 222 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:59:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.986396+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:53.990019+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:54.085638+0000 mgr.y (mgr.24526) 429 : cluster [DBG] pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: audit 2024-09-10T18:59:54.175282+0000 mon.a (mon.0) 2567 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:54.175715+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: audit 2024-09-10T18:59:54.224792+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: audit 2024-09-10T18:59:54.226031+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: audit 2024-09-10T18:59:54.228336+0000 mon.a (mon.0) 2569 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: audit 2024-09-10T18:59:54.229381+0000 mon.a (mon.0) 2570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:54.412052+0000 osd.5 (osd.5) 221 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:59:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:55 smithi083 bash[32381]: cluster 2024-09-10T18:59:54.415289+0000 osd.5 (osd.5) 222 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:59:55.389 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:55.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:55.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:55.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.986396+0000 osd.2 (osd.2) 167 : cluster [DBG] 6.14 scrub starts 2024-09-10T18:59:55.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:53.990019+0000 osd.2 (osd.2) 168 : cluster [DBG] 6.14 scrub ok 2024-09-10T18:59:55.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:54.085638+0000 mgr.y (mgr.24526) 429 : cluster [DBG] pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T18:59:55.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: audit 2024-09-10T18:59:54.175282+0000 mon.a (mon.0) 2567 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:54.175715+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e431: 8 total, 8 up, 8 in 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: audit 2024-09-10T18:59:54.224792+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: audit 2024-09-10T18:59:54.226031+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: audit 2024-09-10T18:59:54.228336+0000 mon.a (mon.0) 2569 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: audit 2024-09-10T18:59:54.229381+0000 mon.a (mon.0) 2570 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-57"}]: dispatch 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:54.412052+0000 osd.5 (osd.5) 221 : cluster [DBG] 12.14 scrub starts 2024-09-10T18:59:55.536 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:55 smithi165 bash[22591]: cluster 2024-09-10T18:59:54.415289+0000 osd.5 (osd.5) 222 : cluster [DBG] 12.14 scrub ok 2024-09-10T18:59:55.548 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:55.650 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:55.674 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:55.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:55.811 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:55.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:55.946 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:55.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:56.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:56.086 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:56.190 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:56.222 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:56.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:56.353 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:56.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:56.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[18629]: cluster 2024-09-10T18:59:55.147525+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:59:56.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[18629]: cluster 2024-09-10T18:59:55.150001+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:59:56.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[18629]: cluster 2024-09-10T18:59:55.173266+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:59:56.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[18629]: cluster 2024-09-10T18:59:55.513086+0000 osd.6 (osd.6) 185 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:59:56.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[18629]: cluster 2024-09-10T18:59:55.520636+0000 osd.6 (osd.6) 186 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:59:56.471 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[32381]: cluster 2024-09-10T18:59:55.147525+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:59:56.471 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[32381]: cluster 2024-09-10T18:59:55.150001+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:59:56.471 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[32381]: cluster 2024-09-10T18:59:55.173266+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:59:56.471 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[32381]: cluster 2024-09-10T18:59:55.513086+0000 osd.6 (osd.6) 185 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:59:56.471 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:56 smithi083 bash[32381]: cluster 2024-09-10T18:59:55.520636+0000 osd.6 (osd.6) 186 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:59:56.489 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:56.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:56.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:56 smithi165 bash[22591]: cluster 2024-09-10T18:59:55.147525+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.11 scrub starts 2024-09-10T18:59:56.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:56 smithi165 bash[22591]: cluster 2024-09-10T18:59:55.150001+0000 osd.3 (osd.3) 253 : cluster [DBG] 6.11 scrub ok 2024-09-10T18:59:56.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:56 smithi165 bash[22591]: cluster 2024-09-10T18:59:55.173266+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e432: 8 total, 8 up, 8 in 2024-09-10T18:59:56.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:56 smithi165 bash[22591]: cluster 2024-09-10T18:59:55.513086+0000 osd.6 (osd.6) 185 : cluster [DBG] 12.1 scrub starts 2024-09-10T18:59:56.606 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:56 smithi165 bash[22591]: cluster 2024-09-10T18:59:55.520636+0000 osd.6 (osd.6) 186 : cluster [DBG] 12.1 scrub ok 2024-09-10T18:59:56.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:56.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:56.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:56.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:56.898 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:56.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:57.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:57.037 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:57.141 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:57.174 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:57.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:56.086780+0000 mgr.y (mgr.24526) 430 : cluster [DBG] pgmap v636: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:56.103194+0000 osd.0 (osd.0) 220 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:56.106878+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:56.205102+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: audit 2024-09-10T18:59:56.208877+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: audit 2024-09-10T18:59:56.214087+0000 mon.a (mon.0) 2573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.028760+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.b scrub starts 2024-09-10T18:59:57.214 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.030793+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.b scrub ok 2024-09-10T18:59:57.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:57.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:56.086780+0000 mgr.y (mgr.24526) 430 : cluster [DBG] pgmap v636: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:57.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:56.103194+0000 osd.0 (osd.0) 220 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:59:57.289 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:56.106878+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:56.205102+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: audit 2024-09-10T18:59:56.208877+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: audit 2024-09-10T18:59:56.214087+0000 mon.a (mon.0) 2573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.028760+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.b scrub starts 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.030793+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.b scrub ok 2024-09-10T18:59:57.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:56.086780+0000 mgr.y (mgr.24526) 430 : cluster [DBG] pgmap v636: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:56.103194+0000 osd.0 (osd.0) 220 : cluster [DBG] 4.3 scrub starts 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:56.106878+0000 osd.0 (osd.0) 221 : cluster [DBG] 4.3 scrub ok 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:56.205102+0000 mon.a (mon.0) 2572 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: audit 2024-09-10T18:59:56.208877+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: audit 2024-09-10T18:59:56.214087+0000 mon.a (mon.0) 2573 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.028760+0000 osd.7 (osd.7) 238 : cluster [DBG] 2.b scrub starts 2024-09-10T18:59:57.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:57 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.030793+0000 osd.7 (osd.7) 239 : cluster [DBG] 2.b scrub ok 2024-09-10T18:59:57.308 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:57.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:57.445 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:57.548 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:57.585 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T18:59:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T18:59:57.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:57.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:57.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:57.858 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T18:59:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T18:59:57.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:57.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:57.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:58.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.131 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:58.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 18:59:57 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T18:59:58.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:58.256 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:58.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:58.365 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.090049+0000 osd.3 (osd.3) 254 : cluster [DBG] 3.b scrub starts 2024-09-10T18:59:58.365 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.094710+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.b scrub ok 2024-09-10T18:59:58.365 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.206614+0000 mon.a (mon.0) 2574 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:58.366 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.206892+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:59:58.396 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T18:59:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T18:59:58.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.090049+0000 osd.3 (osd.3) 254 : cluster [DBG] 3.b scrub starts 2024-09-10T18:59:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.094710+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.b scrub ok 2024-09-10T18:59:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.206614+0000 mon.a (mon.0) 2574 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.206892+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:59:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.214858+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.259579+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.261017+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.263149+0000 mon.a (mon.0) 2576 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: audit 2024-09-10T18:59:57.264471+0000 mon.a (mon.0) 2577 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.324766+0000 osd.5 (osd.5) 223 : cluster [DBG] 6.d scrub starts 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:57.328095+0000 osd.5 (osd.5) 224 : cluster [DBG] 6.d scrub ok 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.055830+0000 osd.7 (osd.7) 240 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:59:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:58 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.060181+0000 osd.7 (osd.7) 241 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:59:58.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.637 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.090049+0000 osd.3 (osd.3) 254 : cluster [DBG] 3.b scrub starts 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.094710+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.b scrub ok 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.206614+0000 mon.a (mon.0) 2574 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.206892+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.214858+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.259579+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.261017+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.263149+0000 mon.a (mon.0) 2576 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: audit 2024-09-10T18:59:57.264471+0000 mon.a (mon.0) 2577 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.324766+0000 osd.5 (osd.5) 223 : cluster [DBG] 6.d scrub starts 2024-09-10T18:59:58.638 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:57.328095+0000 osd.5 (osd.5) 224 : cluster [DBG] 6.d scrub ok 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.055830+0000 osd.7 (osd.7) 240 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.060181+0000 osd.7 (osd.7) 241 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.214858+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.259579+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.261017+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.263149+0000 mon.a (mon.0) 2576 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: audit 2024-09-10T18:59:57.264471+0000 mon.a (mon.0) 2577 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-59"}]: dispatch 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.324766+0000 osd.5 (osd.5) 223 : cluster [DBG] 6.d scrub starts 2024-09-10T18:59:58.639 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:57.328095+0000 osd.5 (osd.5) 224 : cluster [DBG] 6.d scrub ok 2024-09-10T18:59:58.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.055830+0000 osd.7 (osd.7) 240 : cluster [DBG] 64.1a scrub starts 2024-09-10T18:59:58.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:58 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.060181+0000 osd.7 (osd.7) 241 : cluster [DBG] 64.1a scrub ok 2024-09-10T18:59:58.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:58.697 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:58.752 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:58.782 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:58.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:58.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:58.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T18:59:59.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:59.058 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T18:59:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T18:59:59.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:59.187 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T18:59:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T18:59:59.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:59.324 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:59.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:59.463 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T18:59:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: audit 2024-09-10T18:59:57.952707+0000 mgr.y (mgr.24526) 431 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.059755+0000 osd.2 (osd.2) 169 : cluster [DBG] 12.a scrub starts 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.063310+0000 osd.2 (osd.2) 170 : cluster [DBG] 12.a scrub ok 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.087549+0000 mgr.y (mgr.24526) 432 : cluster [DBG] pgmap v639: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.226675+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.480663+0000 osd.6 (osd.6) 187 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:58.485223+0000 osd.6 (osd.6) 188 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: audit 2024-09-10T18:59:59.129655+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.130904+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:59:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[18629]: audit 2024-09-10T18:59:59.136482+0000 mon.a (mon.0) 2580 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: audit 2024-09-10T18:59:57.952707+0000 mgr.y (mgr.24526) 431 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.059755+0000 osd.2 (osd.2) 169 : cluster [DBG] 12.a scrub starts 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.063310+0000 osd.2 (osd.2) 170 : cluster [DBG] 12.a scrub ok 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.087549+0000 mgr.y (mgr.24526) 432 : cluster [DBG] pgmap v639: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.226675+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.480663+0000 osd.6 (osd.6) 187 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:58.485223+0000 osd.6 (osd.6) 188 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: audit 2024-09-10T18:59:59.129655+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.130904+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:59:59.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 18:59:59 smithi083 bash[32381]: audit 2024-09-10T18:59:59.136482+0000 mon.a (mon.0) 2580 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:59.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: audit 2024-09-10T18:59:57.952707+0000 mgr.y (mgr.24526) 431 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T18:59:59.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.059755+0000 osd.2 (osd.2) 169 : cluster [DBG] 12.a scrub starts 2024-09-10T18:59:59.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.063310+0000 osd.2 (osd.2) 170 : cluster [DBG] 12.a scrub ok 2024-09-10T18:59:59.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.087549+0000 mgr.y (mgr.24526) 432 : cluster [DBG] pgmap v639: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T18:59:59.580 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.226675+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-09-10T18:59:59.581 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.480663+0000 osd.6 (osd.6) 187 : cluster [DBG] 12.16 scrub starts 2024-09-10T18:59:59.581 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:58.485223+0000 osd.6 (osd.6) 188 : cluster [DBG] 12.16 scrub ok 2024-09-10T18:59:59.581 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: audit 2024-09-10T18:59:59.129655+0000 mon.b (mon.1) 342 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.581 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.130904+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-09-10T18:59:59.581 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 18:59:59 smithi165 bash[22591]: audit 2024-09-10T18:59:59.136482+0000 mon.a (mon.0) 2580 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:59.600 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T18:59:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T18:59:59.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:59.948 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:59.978 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T18:59:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:59:59.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:00.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:00.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:00.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:00.231 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.062141+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:00:00.231 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.066122+0000 osd.3 (osd.3) 257 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:00:00.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:00.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:00.384 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:00.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:00.501 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.062141+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.066122+0000 osd.3 (osd.3) 257 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.078534+0000 osd.2 (osd.2) 171 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.081265+0000 osd.2 (osd.2) 172 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.102487+0000 osd.0 (osd.0) 222 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.106054+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.416337+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.420980+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.483151+0000 osd.6 (osd.6) 189 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T18:59:59.486638+0000 osd.6 (osd.6) 190 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T18:59:59.572354+0000 mon.a (mon.0) 2581 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T18:59:59.574830+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:00.502 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.000146+0000 mon.a (mon.0) 2582 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.000204+0000 mon.a (mon.0) 2583 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.000259+0000 mon.a (mon.0) 2584 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.000293+0000 mon.a (mon.0) 2585 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.000328+0000 mon.a (mon.0) 2586 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T19:00:00.131016+0000 mon.a (mon.0) 2587 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.131278+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T19:00:00.142582+0000 mon.b (mon.1) 343 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T19:00:00.146199+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.503 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:00 smithi165 bash[22591]: audit 2024-09-10T19:00:00.149804+0000 mon.a (mon.0) 2589 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.521 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.062141+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.066122+0000 osd.3 (osd.3) 257 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.078534+0000 osd.2 (osd.2) 171 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.081265+0000 osd.2 (osd.2) 172 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.102487+0000 osd.0 (osd.0) 222 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.106054+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.416337+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.420980+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.483151+0000 osd.6 (osd.6) 189 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T18:59:59.486638+0000 osd.6 (osd.6) 190 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:00:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T18:59:59.572354+0000 mon.a (mon.0) 2581 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T18:59:59.574830+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.000146+0000 mon.a (mon.0) 2582 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.000204+0000 mon.a (mon.0) 2583 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.000259+0000 mon.a (mon.0) 2584 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.000293+0000 mon.a (mon.0) 2585 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.000328+0000 mon.a (mon.0) 2586 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T19:00:00.131016+0000 mon.a (mon.0) 2587 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.131278+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T19:00:00.142582+0000 mon.b (mon.1) 343 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T19:00:00.146199+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[18629]: audit 2024-09-10T19:00:00.149804+0000 mon.a (mon.0) 2589 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.078534+0000 osd.2 (osd.2) 171 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.081265+0000 osd.2 (osd.2) 172 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.102487+0000 osd.0 (osd.0) 222 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.106054+0000 osd.0 (osd.0) 223 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.416337+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.420980+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.483151+0000 osd.6 (osd.6) 189 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T18:59:59.486638+0000 osd.6 (osd.6) 190 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T18:59:59.572354+0000 mon.a (mon.0) 2581 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T18:59:59.574830+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.000146+0000 mon.a (mon.0) 2582 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.000204+0000 mon.a (mon.0) 2583 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.000259+0000 mon.a (mon.0) 2584 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.000293+0000 mon.a (mon.0) 2585 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.000328+0000 mon.a (mon.0) 2586 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T19:00:00.131016+0000 mon.a (mon.0) 2587 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.131278+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T19:00:00.142582+0000 mon.b (mon.1) 343 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T19:00:00.146199+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:00 smithi083 bash[32381]: audit 2024-09-10T19:00:00.149804+0000 mon.a (mon.0) 2589 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:00.624 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:00.656 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:00.760 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:00.796 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:00.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:00.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:00.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:01.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:01.073 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:01.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:01.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:01.343 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.053621+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.056587+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.088132+0000 mgr.y (mgr.24526) 433 : cluster [DBG] pgmap v642: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.094009+0000 osd.2 (osd.2) 173 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.097216+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.310059+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.313567+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:00:01.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:00.566883+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: audit 2024-09-10T19:00:01.139153+0000 mon.a (mon.0) 2591 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: cluster 2024-09-10T19:00:01.139377+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: audit 2024-09-10T19:00:01.179831+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: audit 2024-09-10T19:00:01.181529+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: audit 2024-09-10T19:00:01.183524+0000 mon.a (mon.0) 2593 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.053621+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.056587+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.088132+0000 mgr.y (mgr.24526) 433 : cluster [DBG] pgmap v642: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.094009+0000 osd.2 (osd.2) 173 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.097216+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.310059+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:00:01.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.313567+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:00.566883+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: audit 2024-09-10T19:00:01.139153+0000 mon.a (mon.0) 2591 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: cluster 2024-09-10T19:00:01.139377+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: audit 2024-09-10T19:00:01.179831+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: audit 2024-09-10T19:00:01.181529+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: audit 2024-09-10T19:00:01.183524+0000 mon.a (mon.0) 2593 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.373 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[32381]: audit 2024-09-10T19:00:01.185041+0000 mon.a (mon.0) 2594 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:01.481 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:01.482 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:01.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.053621+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:00:01.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.056587+0000 osd.0 (osd.0) 225 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:00:01.482 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.088132+0000 mgr.y (mgr.24526) 433 : cluster [DBG] pgmap v642: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.094009+0000 osd.2 (osd.2) 173 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.097216+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.310059+0000 osd.5 (osd.5) 225 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.313567+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:00.566883+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: audit 2024-09-10T19:00:01.139153+0000 mon.a (mon.0) 2591 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: cluster 2024-09-10T19:00:01.139377+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: audit 2024-09-10T19:00:01.179831+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: audit 2024-09-10T19:00:01.181529+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: audit 2024-09-10T19:00:01.183524+0000 mon.a (mon.0) 2593 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:01.483 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:01 smithi165 bash[22591]: audit 2024-09-10T19:00:01.185041+0000 mon.a (mon.0) 2594 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:01.532 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:01 smithi083 bash[18629]: audit 2024-09-10T19:00:01.185041+0000 mon.a (mon.0) 2594 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-61"}]: dispatch 2024-09-10T19:00:01.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:01.616 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:01.672 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:00:01.672 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:00:01.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:01.752 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:01.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:01.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:01.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:01.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:02.028 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:02.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:02.165 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:02.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:02.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:02.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:02.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:02.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:01.343336+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:00:02.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:01.347225+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:01.589487+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:01.594991+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.049991+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.055152+0000 osd.7 (osd.7) 243 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:01.343336+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:01.347225+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:01.589487+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:01.594991+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.049991+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:00:02.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.055152+0000 osd.7 (osd.7) 243 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:00:02.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:02.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:01.343336+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:01.347225+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:01.589487+0000 osd.1 (osd.1) 249 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:01.594991+0000 osd.1 (osd.1) 250 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.049991+0000 osd.7 (osd.7) 242 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:00:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:02 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.055152+0000 osd.7 (osd.7) 243 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:00:02.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:02.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:02.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:02.847 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:02.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:02.981 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:02.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:03.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:02] "GET /metrics HTTP/1.1" 200 228832 "" "Prometheus/2.43.0" 2024-09-10T19:00:03.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:03.107 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:03.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:03.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:03.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.047648+0000 osd.0 (osd.0) 226 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:00:03.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.051984+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:00:03.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.088719+0000 mgr.y (mgr.24526) 434 : cluster [DBG] pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:03.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.248169+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:00:03.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:02.306428+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.c scrub starts 2024-09-10T19:00:03.243 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.067638+0000 osd.7 (osd.7) 244 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:00:03.243 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:03 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.071895+0000 osd.7 (osd.7) 245 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:00:03.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.047648+0000 osd.0 (osd.0) 226 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.051984+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.088719+0000 mgr.y (mgr.24526) 434 : cluster [DBG] pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.248169+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:02.306428+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.c scrub starts 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.067638+0000 osd.7 (osd.7) 244 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:00:03.349 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.071895+0000 osd.7 (osd.7) 245 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.047648+0000 osd.0 (osd.0) 226 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.051984+0000 osd.0 (osd.0) 227 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.088719+0000 mgr.y (mgr.24526) 434 : cluster [DBG] pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.248169+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:02.306428+0000 osd.4 (osd.4) 267 : cluster [DBG] 4.c scrub starts 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.067638+0000 osd.7 (osd.7) 244 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:00:03.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:03 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.071895+0000 osd.7 (osd.7) 245 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:00:03.368 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:03.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:03.504 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:03.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:03.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:03.775 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:03.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:03.904 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:03.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:04.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:04.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:04.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:04.167 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:04.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:04.279 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.032505+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.036958+0000 osd.0 (osd.0) 229 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.206501+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.208459+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.254882+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: audit 2024-09-10T19:00:03.258260+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: audit 2024-09-10T19:00:03.265535+0000 mon.a (mon.0) 2597 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.398541+0000 osd.6 (osd.6) 191 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.402952+0000 osd.6 (osd.6) 192 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.608787+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:03.612209+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:00:04.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.103369+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:00:04.281 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:04 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.106648+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:00:04.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:04.397 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.032505+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.036958+0000 osd.0 (osd.0) 229 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.206501+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.208459+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.254882+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: audit 2024-09-10T19:00:03.258260+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: audit 2024-09-10T19:00:03.265535+0000 mon.a (mon.0) 2597 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.398541+0000 osd.6 (osd.6) 191 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.402952+0000 osd.6 (osd.6) 192 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.608787+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:03.612209+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.103369+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.106648+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.032505+0000 osd.0 (osd.0) 228 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.036958+0000 osd.0 (osd.0) 229 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.206501+0000 osd.5 (osd.5) 227 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.208459+0000 osd.5 (osd.5) 228 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.254882+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:00:04.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: audit 2024-09-10T19:00:03.258260+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: audit 2024-09-10T19:00:03.265535+0000 mon.a (mon.0) 2597 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.398541+0000 osd.6 (osd.6) 191 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.402952+0000 osd.6 (osd.6) 192 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.608787+0000 osd.1 (osd.1) 251 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:03.612209+0000 osd.1 (osd.1) 252 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.103369+0000 osd.7 (osd.7) 246 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:00:04.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:04 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.106648+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:00:04.434 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:04.538 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:04.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:00:04.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:04.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:04.839 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:04.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:04.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:04.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:05.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:05.107 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:05.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:05.241 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:05.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:05.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.057138+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.060361+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.089574+0000 mgr.y (mgr.24526) 435 : cluster [DBG] pgmap v648: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: audit 2024-09-10T19:00:04.262174+0000 mon.a (mon.0) 2598 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: cluster 2024-09-10T19:00:04.262327+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: audit 2024-09-10T19:00:04.262820+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: audit 2024-09-10T19:00:04.275909+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:05 smithi165 bash[22591]: audit 2024-09-10T19:00:04.280413+0000 mon.a (mon.0) 2600 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.371 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:05.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:05.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.057138+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.060361+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.089574+0000 mgr.y (mgr.24526) 435 : cluster [DBG] pgmap v648: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: audit 2024-09-10T19:00:04.262174+0000 mon.a (mon.0) 2598 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: cluster 2024-09-10T19:00:04.262327+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: audit 2024-09-10T19:00:04.262820+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: audit 2024-09-10T19:00:04.275909+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[32381]: audit 2024-09-10T19:00:04.280413+0000 mon.a (mon.0) 2600 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.057138+0000 osd.0 (osd.0) 230 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.060361+0000 osd.0 (osd.0) 231 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:00:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.089574+0000 mgr.y (mgr.24526) 435 : cluster [DBG] pgmap v648: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:00:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: audit 2024-09-10T19:00:04.262174+0000 mon.a (mon.0) 2598 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: cluster 2024-09-10T19:00:04.262327+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:00:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: audit 2024-09-10T19:00:04.262820+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: audit 2024-09-10T19:00:04.275909+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:05 smithi083 bash[18629]: audit 2024-09-10T19:00:04.280413+0000 mon.a (mon.0) 2600 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:05.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:05.645 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:05.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:05.781 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:05.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:05.916 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:06.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:06.056 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:06.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:06.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:06.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:06.323 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:06.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:06.426 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:05.192928+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:05.196337+0000 osd.5 (osd.5) 230 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:05.217099+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.e scrub starts 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:05.221180+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.e scrub ok 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: audit 2024-09-10T19:00:05.276932+0000 mon.a (mon.0) 2601 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:05.277071+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.085889+0000 osd.7 (osd.7) 248 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:00:06.437 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.091194+0000 osd.7 (osd.7) 249 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:05.192928+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:05.196337+0000 osd.5 (osd.5) 230 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:05.217099+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.e scrub starts 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:05.221180+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.e scrub ok 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: audit 2024-09-10T19:00:05.276932+0000 mon.a (mon.0) 2601 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:05.277071+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.085889+0000 osd.7 (osd.7) 248 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:00:06.438 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:06 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.091194+0000 osd.7 (osd.7) 249 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:00:06.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:06.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:06.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:05.192928+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:00:06.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:05.196337+0000 osd.5 (osd.5) 230 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:00:06.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:05.217099+0000 osd.4 (osd.4) 268 : cluster [DBG] 4.e scrub starts 2024-09-10T19:00:06.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:05.221180+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.e scrub ok 2024-09-10T19:00:06.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: audit 2024-09-10T19:00:05.276932+0000 mon.a (mon.0) 2601 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:06.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:05.277071+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:00:06.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.085889+0000 osd.7 (osd.7) 248 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:00:06.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:06 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.091194+0000 osd.7 (osd.7) 249 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:00:06.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:06.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:06.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:06.726 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:06.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:06.864 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:06.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:06.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:07.002 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:07.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:07.135 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:07.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:07.273 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:07.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.414 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:07.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:07.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.091878+0000 mgr.y (mgr.24526) 436 : cluster [DBG] pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.135894+0000 osd.2 (osd.2) 175 : cluster [DBG] 2.e scrub starts 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.139449+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub ok 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.270955+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[18629]: cluster 2024-09-10T19:00:06.285948+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.091878+0000 mgr.y (mgr.24526) 436 : cluster [DBG] pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.135894+0000 osd.2 (osd.2) 175 : cluster [DBG] 2.e scrub starts 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.139449+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub ok 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.270955+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:07 smithi083 bash[32381]: cluster 2024-09-10T19:00:06.285948+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:00:07.552 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:07.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.091878+0000 mgr.y (mgr.24526) 436 : cluster [DBG] pgmap v651: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:07.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.135894+0000 osd.2 (osd.2) 175 : cluster [DBG] 2.e scrub starts 2024-09-10T19:00:07.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.139449+0000 osd.2 (osd.2) 176 : cluster [DBG] 2.e scrub ok 2024-09-10T19:00:07.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.270955+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:07.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[22591]: cluster 2024-09-10T19:00:06.285948+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:00:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:07.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.681 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:07.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:07.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:07.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:07.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:08.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.032+0000 7fe6207ac640 1 -- 172.21.15.83:0/2574790951 >> v1:172.21.15.83:6789/0 conn(0x7fe618104290 legacy=0x7fe618106670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:08.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.032+0000 7fe6207ac640 1 -- 172.21.15.83:0/2574790951 shutdown_connections 2024-09-10T19:00:08.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.032+0000 7fe6207ac640 1 -- 172.21.15.83:0/2574790951 >> 172.21.15.83:0/2574790951 conn(0x7fe6180f70a0 msgr2=0x7fe6180f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:08.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.032+0000 7fe6207ac640 1 -- 172.21.15.83:0/2574790951 shutdown_connections 2024-09-10T19:00:08.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.032+0000 7fe6207ac640 1 -- 172.21.15.83:0/2574790951 wait complete. 2024-09-10T19:00:08.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 librados: starting msgr at 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 librados: starting objecter 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 Processor -- start 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- start start 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 librados: setting wanted keys 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 librados: calling monclient init 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe61811bfd0 con 0x7fe618101520 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe61811d1d0 con 0x7fe618104290 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe61e548640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe618101520 0x7fe618065db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58572/0 (socket says 172.21.15.83:58572) 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe61e548640 1 -- 172.21.15.83:0/2185456230 learned_addr learned my addr 172.21.15.83:0/2185456230 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:08.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe61811e3d0 con 0x7fe618107df0 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe618101520 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe618107df0 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter ms_handle_connect 0x7fe618104290 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253024306 0 0) 0x7fe61811bfd0 con 0x7fe618101520 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604003120 con 0x7fe618101520 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883499751 0 0) 0x7fe61811e3d0 con 0x7fe618107df0 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61811bfd0 con 0x7fe618107df0 2024-09-10T19:00:08.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845178304 0 0) 0x7fe61811d1d0 con 0x7fe618104290 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61811e3d0 con 0x7fe618104290 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669391530 0 0) 0x7fe61811bfd0 con 0x7fe618107df0 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe61811d1d0 con 0x7fe618107df0 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe610001d70 con 0x7fe618107df0 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518978192 0 0) 0x7fe604003120 con 0x7fe618101520 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe61811bfd0 con 0x7fe618101520 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe6140018c0 con 0x7fe618101520 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470644939 0 0) 0x7fe61811e3d0 con 0x7fe618104290 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe604003120 con 0x7fe618104290 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3240735971 0 0) 0x7fe61811d1d0 con 0x7fe618107df0 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 >> v1:172.21.15.165:6789/0 conn(0x7fe618104290 legacy=0x7fe618066500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 >> v1:172.21.15.83:6789/0 conn(0x7fe618101520 legacy=0x7fe618065db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61811f5d0 con 0x7fe618107df0 2024-09-10T19:00:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61811e5c0 con 0x7fe618107df0 2024-09-10T19:00:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 10 client.49388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61811ea90 con 0x7fe618107df0 2024-09-10T19:00:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe6207ac640 1 librados: init done 2024-09-10T19:00:08.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe610002660 con 0x7fe618107df0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.036+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe610003540 con 0x7fe618107df0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6180fcc20 con 0x7fe618107df0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe610003780 con 0x7fe618107df0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(444..444 src has 69..444) v4 ==== 8004+0+0 (unknown 2991003050 0 0) 0x7fe610087ff0 con 0x7fe618107df0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 10 client.49388.objecter ms_dispatch 0x7fe618000b90 osd_map(444..444 src has 69..444) v4 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 3 client.49388.objecter handle_osd_map got epochs [444,444] > 0 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 3 client.49388.objecter handle_osd_map decoding full epoch 444 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 20 client.49388.objecter dump_active .. 0 homeless 2024-09-10T19:00:08.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.040+0000 7fe60effd640 10 client.49388.objecter ms_handle_connect 0x7fe6040717b0 2024-09-10T19:00:08.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.044+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe610055390 con 0x7fe618107df0 2024-09-10T19:00:08.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:08.195 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:08.224 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:08.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.224+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6180ffdf0 con 0x7fe6040717b0 2024-09-10T19:00:08.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.244+0000 7fe60effd640 1 -- 172.21.15.83:0/2185456230 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+625047 (unknown 2965378022 0 1703462841) 0x7fe6180ffdf0 con 0x7fe6040717b0 2024-09-10T19:00:08.257 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:00:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 10 librados: watch_flush enter 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 10 librados: watch_flush exit 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 20 client.49388.objecter shutdown clearing up homeless session... 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 10 client.49388.objecter successfully canceled tick 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe6040717b0 legacy=0x7fe604073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 >> v1:172.21.15.83:6790/0 conn(0x7fe618107df0 legacy=0x7fe61811a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 shutdown_connections 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 >> 172.21.15.83:0/2185456230 conn(0x7fe6180f70a0 msgr2=0x7fe61810aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:08.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 shutdown_connections 2024-09-10T19:00:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 -- 172.21.15.83:0/2185456230 wait complete. 2024-09-10T19:00:08.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:08.252+0000 7fe6207ac640 1 librados: shutdown 2024-09-10T19:00:08.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:07 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:08.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.351 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:08.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:08.488 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:08.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[18629]: cluster 2024-09-10T19:00:07.293880+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:00:08.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[18629]: cluster 2024-09-10T19:00:07.320451+0000 osd.6 (osd.6) 193 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[18629]: cluster 2024-09-10T19:00:07.323354+0000 osd.6 (osd.6) 194 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[18629]: cluster 2024-09-10T19:00:08.098926+0000 osd.7 (osd.7) 250 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[18629]: cluster 2024-09-10T19:00:08.101893+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[32381]: cluster 2024-09-10T19:00:07.293880+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[32381]: cluster 2024-09-10T19:00:07.320451+0000 osd.6 (osd.6) 193 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[32381]: cluster 2024-09-10T19:00:07.323354+0000 osd.6 (osd.6) 194 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[32381]: cluster 2024-09-10T19:00:08.098926+0000 osd.7 (osd.7) 250 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:00:08.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:08 smithi083 bash[32381]: cluster 2024-09-10T19:00:08.101893+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:00:08.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:08 smithi165 bash[22591]: cluster 2024-09-10T19:00:07.293880+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:00:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:08 smithi165 bash[22591]: cluster 2024-09-10T19:00:07.320451+0000 osd.6 (osd.6) 193 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:00:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:08 smithi165 bash[22591]: cluster 2024-09-10T19:00:07.323354+0000 osd.6 (osd.6) 194 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:00:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:08 smithi165 bash[22591]: cluster 2024-09-10T19:00:08.098926+0000 osd.7 (osd.7) 250 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:00:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:08 smithi165 bash[22591]: cluster 2024-09-10T19:00:08.101893+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:00:08.624 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:08.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.759 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:08.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:08.895 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:08.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:08.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:09.032 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:09.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:09.160 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:09.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:09.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:09.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:09.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:09.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:09.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[18629]: audit 2024-09-10T19:00:07.961805+0000 mgr.y (mgr.24526) 437 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[18629]: cluster 2024-09-10T19:00:08.092597+0000 mgr.y (mgr.24526) 438 : cluster [DBG] pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[18629]: audit 2024-09-10T19:00:08.228656+0000 mgr.y (mgr.24526) 439 : audit [DBG] from='client.49388 v1:172.21.15.83:0/2185456230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[18629]: cluster 2024-09-10T19:00:08.307095+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[18629]: cluster 2024-09-10T19:00:09.134468+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[32381]: audit 2024-09-10T19:00:07.961805+0000 mgr.y (mgr.24526) 437 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[32381]: cluster 2024-09-10T19:00:08.092597+0000 mgr.y (mgr.24526) 438 : cluster [DBG] pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[32381]: audit 2024-09-10T19:00:08.228656+0000 mgr.y (mgr.24526) 439 : audit [DBG] from='client.49388 v1:172.21.15.83:0/2185456230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[32381]: cluster 2024-09-10T19:00:08.307095+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:00:09.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:09 smithi083 bash[32381]: cluster 2024-09-10T19:00:09.134468+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:00:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:09 smithi165 bash[22591]: audit 2024-09-10T19:00:07.961805+0000 mgr.y (mgr.24526) 437 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:09 smithi165 bash[22591]: cluster 2024-09-10T19:00:08.092597+0000 mgr.y (mgr.24526) 438 : cluster [DBG] pgmap v654: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:00:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:09 smithi165 bash[22591]: audit 2024-09-10T19:00:08.228656+0000 mgr.y (mgr.24526) 439 : audit [DBG] from='client.49388 v1:172.21.15.83:0/2185456230' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:09 smithi165 bash[22591]: cluster 2024-09-10T19:00:08.307095+0000 mon.a (mon.0) 2606 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:00:09.552 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:09 smithi165 bash[22591]: cluster 2024-09-10T19:00:09.134468+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:00:09.575 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:09.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.715 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:09.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.853 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:09.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.996 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:09.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:10.016 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-10T19:00:08.092137+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791990,"num_objects":360,"num_object_clones":2,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":360,"num_whiteouts":0,"num_read":2043,"num_read_kb":1797,"num_write":1435,"num_write_kb":14364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2829,"ondisk_log_size":2829,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":669,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2500480,"kb_used_data":79968,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":747460736,"statfs":{"total":767960285184,"available":765399793664,"internally_reserved":0,"allocated":81887232,"data_stored":57602804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.003178"},"pg_stats":[{"pgid":"215.1f","version":"442'2","reported_seq":12,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.294580+0000","last_change":"2024-09-10T19:00:04.268476+0000","last_active":"2024-09-10T19:00:05.294580+0000","last_peered":"2024-09-10T19:00:05.294580+0000","last_clean":"2024-09-10T19:00:05.294580+0000","last_became_active":"2024-09-10T19:00:04.267732+0000","last_became_peered":"2024-09-10T19:00:04.267732+0000","last_unstale":"2024-09-10T19:00:05.294580+0000","last_undegraded":"2024-09-10T19:00:05.294580+0000","last_fullsized":"2024-09-10T19:00:05.294580+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.164189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.1e","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.292449+0000","last_change":"2024-09-10T19:00:04.291945+0000","last_active":"2024-09-10T19:00:04.292449+0000","last_peered":"2024-09-10T19:00:04.292449+0000","last_clean":"2024-09-10T19:00:04.292449+0000","last_became_active":"2024-09-10T19:00:04.290965+0000","last_became_peered":"2024-09-10T19:00:04.290965+0000","last_unstale":"2024-09-10T19:00:04.292449+0000","last_undegraded":"2024-09-10T19:00:04.292449+0000","last_fullsized":"2024-09-10T19:00:04.292449+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.075695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.1d","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284613+0000","last_change":"2024-09-10T19:00:04.276088+0000","last_active":"2024-09-10T19:00:04.284613+0000","last_peered":"2024-09-10T19:00:04.284613+0000","last_clean":"2024-09-10T19:00:04.284613+0000","last_became_active":"2024-09-10T19:00:04.275821+0000","last_became_peered":"2024-09-10T19:00:04.275821+0000","last_unstale":"2024-09-10T19:00:04.284613+0000","last_undegraded":"2024-09-10T19:00:04.284613+0000","last_fullsized":"2024-09-10T19:00:04.284613+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.739652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.1c","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287320+0000","last_change":"2024-09-10T19:00:04.286817+0000","last_active":"2024-09-10T19:00:04.287320+0000","last_peered":"2024-09-10T19:00:04.287320+0000","last_clean":"2024-09-10T19:00:04.287320+0000","last_became_active":"2024-09-10T19:00:04.286469+0000","last_became_peered":"2024-09-10T19:00:04.286469+0000","last_unstale":"2024-09-10T19:00:04.287320+0000","last_undegraded":"2024-09-10T19:00:04.287320+0000","last_fullsized":"2024-09-10T19:00:04.287320+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.583907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1b","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.289032+0000","last_change":"2024-09-10T19:00:04.288725+0000","last_active":"2024-09-10T19:00:04.289032+0000","last_peered":"2024-09-10T19:00:04.289032+0000","last_clean":"2024-09-10T19:00:04.289032+0000","last_became_active":"2024-09-10T19:00:04.288361+0000","last_became_peered":"2024-09-10T19:00:04.288361+0000","last_unstale":"2024-09-10T19:00:04.289032+0000","last_undegraded":"2024-09-10T19:00:04.289032+0000","last_fullsized":"2024-09-10T19:00:04.289032+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.963316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1a","version":"442'1","reported_seq":11,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.297914+0000","last_change":"2024-09-10T19:00:04.275768+0000","last_active":"2024-09-10T19:00:05.297914+0000","last_peered":"2024-09-10T19:00:05.297914+0000","last_clean":"2024-09-10T19:00:05.297914+0000","last_became_active":"2024-09-10T19:00:04.275245+0000","last_became_peered":"2024-09-10T19:00:04.275245+0000","last_unstale":"2024-09-10T19:00:05.297914+0000","last_undegraded":"2024-09-10T19:00:05.297914+0000","last_fullsized":"2024-09-10T19:00:05.297914+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.008216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.19","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284260+0000","last_change":"2024-09-10T19:00:04.283513+0000","last_active":"2024-09-10T19:00:04.284260+0000","last_peered":"2024-09-10T19:00:04.284260+0000","last_clean":"2024-09-10T19:00:04.284260+0000","last_became_active":"2024-09-10T19:00:04.283166+0000","last_became_peered":"2024-09-10T19:00:04.283166+0000","last_unstale":"2024-09-10T19:00:04.284260+0000","last_undegraded":"2024-09-10T19:00:04.284260+0000","last_fullsized":"2024-09-10T19:00:04.284260+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.031130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.18","version":"442'3","reported_seq":14,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.303416+0000","last_change":"2024-09-10T19:00:04.280704+0000","last_active":"2024-09-10T19:00:05.303416+0000","last_peered":"2024-09-10T19:00:05.303416+0000","last_clean":"2024-09-10T19:00:05.303416+0000","last_became_active":"2024-09-10T19:00:04.280190+0000","last_became_peered":"2024-09-10T19:00:04.280190+0000","last_unstale":"2024-09-10T19:00:05.303416+0000","last_undegraded":"2024-09-10T19:00:05.303416+0000","last_fullsized":"2024-09-10T19:00:05.303416+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.454766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.17","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284684+0000","last_change":"2024-09-10T19:00:04.279347+0000","last_active":"2024-09-10T19:00:04.284684+0000","last_peered":"2024-09-10T19:00:04.284684+0000","last_clean":"2024-09-10T19:00:04.284684+0000","last_became_active":"2024-09-10T19:00:04.277956+0000","last_became_peered":"2024-09-10T19:00:04.277956+0000","last_unstale":"2024-09-10T19:00:04.284684+0000","last_undegraded":"2024-09-10T19:00:04.284684+0000","last_fullsized":"2024-09-10T19:00:04.284684+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.373100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.16","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287896+0000","last_change":"2024-09-10T19:00:04.287474+0000","last_active":"2024-09-10T19:00:04.287896+0000","last_peered":"2024-09-10T19:00:04.287896+0000","last_clean":"2024-09-10T19:00:04.287896+0000","last_became_active":"2024-09-10T19:00:04.287210+0000","last_became_peered":"2024-09-10T19:00:04.287210+0000","last_unstale":"2024-09-10T19:00:04.287896+0000","last_undegraded":"2024-09-10T19:00:04.287896+0000","last_fullsized":"2024-09-10T19:00:04.287896+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.857969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.15","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.290948+0000","last_change":"2024-09-10T19:00:04.287919+0000","last_active":"2024-09-10T19:00:04.290948+0000","last_peered":"2024-09-10T19:00:04.290948+0000","last_clean":"2024-09-10T19:00:04.290948+0000","last_became_active":"2024-09-10T19:00:04.287638+0000","last_became_peered":"2024-09-10T19:00:04.287638+0000","last_unstale":"2024-09-10T19:00:04.290948+0000","last_undegraded":"2024-09-10T19:00:04.290948+0000","last_fullsized":"2024-09-10T19:00:04.290948+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.14","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.277161+0000","last_change":"2024-09-10T19:00:04.275814+0000","last_active":"2024-09-10T19:00:04.277161+0000","last_peered":"2024-09-10T19:00:04.277161+0000","last_clean":"2024-09-10T19:00:04.277161+0000","last_became_active":"2024-09-10T19:00:04.275307+0000","last_became_peered":"2024-09-10T19:00:04.275307+0000","last_unstale":"2024-09-10T19:00:04.277161+0000","last_undegraded":"2024-09-10T19:00:04.277161+0000","last_fullsized":"2024-09-10T19:00:04.277161+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:03.936168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.13","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284411+0000","last_change":"2024-09-10T19:00:04.283918+0000","last_active":"2024-09-10T19:00:04.284411+0000","last_peered":"2024-09-10T19:00:04.284411+0000","last_clean":"2024-09-10T19:00:04.284411+0000","last_became_active":"2024-09-10T19:00:04.283556+0000","last_became_peered":"2024-09-10T19:00:04.283556+0000","last_unstale":"2024-09-10T19:00:04.284411+0000","last_undegraded":"2024-09-10T19:00:04.284411+0000","last_fullsized":"2024-09-10T19:00:04.284411+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.952005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.12","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.282328+0000","last_change":"2024-09-10T19:00:04.275283+0000","last_active":"2024-09-10T19:00:04.282328+0000","last_peered":"2024-09-10T19:00:04.282328+0000","last_clean":"2024-09-10T19:00:04.282328+0000","last_became_active":"2024-09-10T19:00:04.274956+0000","last_became_peered":"2024-09-10T19:00:04.274956+0000","last_unstale":"2024-09-10T19:00:04.282328+0000","last_undegraded":"2024-09-10T19:00:04.282328+0000","last_fullsized":"2024-09-10T19:00:04.282328+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.517350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.11","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287700+0000","last_change":"2024-09-10T19:00:04.287301+0000","last_active":"2024-09-10T19:00:04.287700+0000","last_peered":"2024-09-10T19:00:04.287700+0000","last_clean":"2024-09-10T19:00:04.287700+0000","last_became_active":"2024-09-10T19:00:04.287016+0000","last_became_peered":"2024-09-10T19:00:04.287016+0000","last_unstale":"2024-09-10T19:00:04.287700+0000","last_undegraded":"2024-09-10T19:00:04.287700+0000","last_fullsized":"2024-09-10T19:00:04.287700+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.617302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.10","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284445+0000","last_change":"2024-09-10T19:00:04.282956+0000","last_active":"2024-09-10T19:00:04.284445+0000","last_peered":"2024-09-10T19:00:04.284445+0000","last_clean":"2024-09-10T19:00:04.284445+0000","last_became_active":"2024-09-10T19:00:04.281332+0000","last_became_peered":"2024-09-10T19:00:04.281332+0000","last_unstale":"2024-09-10T19:00:04.284445+0000","last_undegraded":"2024-09-10T19:00:04.284445+0000","last_fullsized":"2024-09-10T19:00:04.284445+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:07.407596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.f","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.275693+0000","last_change":"2024-09-10T19:00:04.274491+0000","last_active":"2024-09-10T19:00:04.275693+0000","last_peered":"2024-09-10T19:00:04.275693+0000","last_clean":"2024-09-10T19:00:04.275693+0000","last_became_active":"2024-09-10T19:00:04.274224+0000","last_became_peered":"2024-09-10T19:00:04.274224+0000","last_unstale":"2024-09-10T19:00:04.275693+0000","last_undegraded":"2024-09-10T19:00:04.275693+0000","last_fullsized":"2024-09-10T19:00:04.275693+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.187213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"215.e","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.286613+0000","last_change":"2024-09-10T19:00:04.286254+0000","last_active":"2024-09-10T19:00:04.286613+0000","last_peered":"2024-09-10T19:00:04.286613+0000","last_clean":"2024-09-10T19:00:04.286613+0000","last_became_active":"2024-09-10T19:00:04.285951+0000","last_became_peered":"2024-09-10T19:00:04.285951+0000","last_unstale":"2024-09-10T19:00:04.286613+0000","last_undegraded":"2024-09-10T19:00:04.286613+0000","last_fullsized":"2024-09-10T19:00:04.286613+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.086526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.d","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284329+0000","last_change":"2024-09-10T19:00:04.276650+0000","last_active":"2024-09-10T19:00:04.284329+0000","last_peered":"2024-09-10T19:00:04.284329+0000","last_clean":"2024-09-10T19:00:04.284329+0000","last_became_active":"2024-09-10T19:00:04.276363+0000","last_became_peered":"2024-09-10T19:00:04.276363+0000","last_unstale":"2024-09-10T19:00:04.284329+0000","last_undegraded":"2024-09-10T19:00:04.284329+0000","last_fullsized":"2024-09-10T19:00:04.284329+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.699872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.c","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284952+0000","last_change":"2024-09-10T19:00:04.283152+0000","last_active":"2024-09-10T19:00:04.284952+0000","last_peered":"2024-09-10T19:00:04.284952+0000","last_clean":"2024-09-10T19:00:04.284952+0000","last_became_active":"2024-09-10T19:00:04.282635+0000","last_became_peered":"2024-09-10T19:00:04.282635+0000","last_unstale":"2024-09-10T19:00:04.284952+0000","last_undegraded":"2024-09-10T19:00:04.284952+0000","last_fullsized":"2024-09-10T19:00:04.284952+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.211376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.b","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.292501+0000","last_change":"2024-09-10T19:00:04.292003+0000","last_active":"2024-09-10T19:00:04.292501+0000","last_peered":"2024-09-10T19:00:04.292501+0000","last_clean":"2024-09-10T19:00:04.292501+0000","last_became_active":"2024-09-10T19:00:04.291749+0000","last_became_peered":"2024-09-10T19:00:04.291749+0000","last_unstale":"2024-09-10T19:00:04.292501+0000","last_undegraded":"2024-09-10T19:00:04.292501+0000","last_fullsized":"2024-09-10T19:00:04.292501+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.927530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.a","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.286230+0000","last_change":"2024-09-10T19:00:04.285065+0000","last_active":"2024-09-10T19:00:04.286230+0000","last_peered":"2024-09-10T19:00:04.286230+0000","last_clean":"2024-09-10T19:00:04.286230+0000","last_became_active":"2024-09-10T19:00:04.284826+0000","last_became_peered":"2024-09-10T19:00:04.284826+0000","last_unstale":"2024-09-10T19:00:04.286230+0000","last_undegraded":"2024-09-10T19:00:04.286230+0000","last_fullsized":"2024-09-10T19:00:04.286230+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.9","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274274+0000","last_change":"2024-09-10T19:00:04.270585+0000","last_active":"2024-09-10T19:00:04.274274+0000","last_peered":"2024-09-10T19:00:04.274274+0000","last_clean":"2024-09-10T19:00:04.274274+0000","last_became_active":"2024-09-10T19:00:04.270285+0000","last_became_peered":"2024-09-10T19:00:04.270285+0000","last_unstale":"2024-09-10T19:00:04.274274+0000","last_undegraded":"2024-09-10T19:00:04.274274+0000","last_fullsized":"2024-09-10T19:00:04.274274+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.542757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.8","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287520+0000","last_change":"2024-09-10T19:00:04.287128+0000","last_active":"2024-09-10T19:00:04.287520+0000","last_peered":"2024-09-10T19:00:04.287520+0000","last_clean":"2024-09-10T19:00:04.287520+0000","last_became_active":"2024-09-10T19:00:04.286817+0000","last_became_peered":"2024-09-10T19:00:04.286817+0000","last_unstale":"2024-09-10T19:00:04.287520+0000","last_undegraded":"2024-09-10T19:00:04.287520+0000","last_fullsized":"2024-09-10T19:00:04.287520+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.684928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.7","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.293412+0000","last_change":"2024-09-10T19:00:04.293140+0000","last_active":"2024-09-10T19:00:04.293412+0000","last_peered":"2024-09-10T19:00:04.293412+0000","last_clean":"2024-09-10T19:00:04.293412+0000","last_became_active":"2024-09-10T19:00:04.292824+0000","last_became_peered":"2024-09-10T19:00:04.292824+0000","last_unstale":"2024-09-10T19:00:04.293412+0000","last_undegraded":"2024-09-10T19:00:04.293412+0000","last_fullsized":"2024-09-10T19:00:04.293412+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.962055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.6","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.280055+0000","last_change":"2024-09-10T19:00:04.279607+0000","last_active":"2024-09-10T19:00:04.280055+0000","last_peered":"2024-09-10T19:00:04.280055+0000","last_clean":"2024-09-10T19:00:04.280055+0000","last_became_active":"2024-09-10T19:00:04.279164+0000","last_became_peered":"2024-09-10T19:00:04.279164+0000","last_unstale":"2024-09-10T19:00:04.280055+0000","last_undegraded":"2024-09-10T19:00:04.280055+0000","last_fullsized":"2024-09-10T19:00:04.280055+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.486237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.5","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.275719+0000","last_change":"2024-09-10T19:00:04.272438+0000","last_active":"2024-09-10T19:00:04.275719+0000","last_peered":"2024-09-10T19:00:04.275719+0000","last_clean":"2024-09-10T19:00:04.275719+0000","last_became_active":"2024-09-10T19:00:04.272020+0000","last_became_peered":"2024-09-10T19:00:04.272020+0000","last_unstale":"2024-09-10T19:00:04.275719+0000","last_undegraded":"2024-09-10T19:00:04.275719+0000","last_fullsized":"2024-09-10T19:00:04.275719+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.089634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.4","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284441+0000","last_change":"2024-09-10T19:00:04.276737+0000","last_active":"2024-09-10T19:00:04.284441+0000","last_peered":"2024-09-10T19:00:04.284441+0000","last_clean":"2024-09-10T19:00:04.284441+0000","last_became_active":"2024-09-10T19:00:04.276433+0000","last_became_peered":"2024-09-10T19:00:04.276433+0000","last_unstale":"2024-09-10T19:00:04.284441+0000","last_undegraded":"2024-09-10T19:00:04.284441+0000","last_fullsized":"2024-09-10T19:00:04.284441+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.159916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.3","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274314+0000","last_change":"2024-09-10T19:00:04.272390+0000","last_active":"2024-09-10T19:00:04.274314+0000","last_peered":"2024-09-10T19:00:04.274314+0000","last_clean":"2024-09-10T19:00:04.274314+0000","last_became_active":"2024-09-10T19:00:04.272053+0000","last_became_peered":"2024-09-10T19:00:04.272053+0000","last_unstale":"2024-09-10T19:00:04.274314+0000","last_undegraded":"2024-09-10T19:00:04.274314+0000","last_fullsized":"2024-09-10T19:00:04.274314+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.2","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.284308+0000","last_change":"2024-09-10T19:00:04.283670+0000","last_active":"2024-09-10T19:00:04.284308+0000","last_peered":"2024-09-10T19:00:04.284308+0000","last_clean":"2024-09-10T19:00:04.284308+0000","last_became_active":"2024-09-10T19:00:04.283381+0000","last_became_peered":"2024-09-10T19:00:04.283381+0000","last_unstale":"2024-09-10T19:00:04.284308+0000","last_undegraded":"2024-09-10T19:00:04.284308+0000","last_fullsized":"2024-09-10T19:00:04.284308+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.053377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287432+0000","last_change":"2024-09-10T19:00:04.287062+0000","last_active":"2024-09-10T19:00:04.287432+0000","last_peered":"2024-09-10T19:00:04.287432+0000","last_clean":"2024-09-10T19:00:04.287432+0000","last_became_active":"2024-09-10T19:00:04.286750+0000","last_became_peered":"2024-09-10T19:00:04.286750+0000","last_unstale":"2024-09-10T19:00:04.287432+0000","last_undegraded":"2024-09-10T19:00:04.287432+0000","last_fullsized":"2024-09-10T19:00:04.287432+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.558827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.0","version":"0'0","reported_seq":10,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278514+0000","last_change":"2024-09-10T19:00:04.277890+0000","last_active":"2024-09-10T19:00:04.278514+0000","last_peered":"2024-09-10T19:00:04.278514+0000","last_clean":"2024-09-10T19:00:04.278514+0000","last_became_active":"2024-09-10T19:00:04.277555+0000","last_became_peered":"2024-09-10T19:00:04.277555+0000","last_unstale":"2024-09-10T19:00:04.278514+0000","last_undegraded":"2024-09-10T19:00:04.278514+0000","last_fullsized":"2024-09-10T19:00:04.278514+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:27.693272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":401,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264218+0000","last_change":"2024-09-10T18:58:52.881919+0000","last_active":"2024-09-10T19:00:04.264218+0000","last_peered":"2024-09-10T19:00:04.264218+0000","last_clean":"2024-09-10T19:00:04.264218+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:00:04.264218+0000","last_undegraded":"2024-09-10T19:00:04.264218+0000","last_fullsized":"2024-09-10T19:00:04.264218+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.087038+0000","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.001733262,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"377'67","reported_seq":431,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263526+0000","last_change":"2024-09-10T18:58:39.857881+0000","last_active":"2024-09-10T19:00:04.263526+0000","last_peered":"2024-09-10T19:00:04.263526+0000","last_clean":"2024-09-10T19:00:04.263526+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:00:04.263526+0000","last_undegraded":"2024-09-10T19:00:04.263526+0000","last_fullsized":"2024-09-10T19:00:04.263526+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.016940303,"stat_sum":{"num_bytes":64,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"425'25","reported_seq":407,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265065+0000","last_change":"2024-09-10T19:00:03.072017+0000","last_active":"2024-09-10T19:00:04.265065+0000","last_peered":"2024-09-10T19:00:04.265065+0000","last_clean":"2024-09-10T19:00:04.265065+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:00:04.265065+0000","last_undegraded":"2024-09-10T19:00:04.265065+0000","last_fullsized":"2024-09-10T19:00:04.265065+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'25","last_scrub_stamp":"2024-09-10T19:00:03.071902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.071902+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.351800+0000","scrub_duration":0.0041116920000000001,"objects_trimmed":0,"snaptrim_duration":0.0019269700000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":417,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.282630+0000","last_change":"2024-09-10T18:58:54.847481+0000","last_active":"2024-09-10T19:00:04.282630+0000","last_peered":"2024-09-10T19:00:04.282630+0000","last_clean":"2024-09-10T19:00:04.282630+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:00:04.282630+0000","last_undegraded":"2024-09-10T19:00:04.282630+0000","last_fullsized":"2024-09-10T19:00:04.282630+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:58:54.847269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.847269+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091152980000000008,"objects_trimmed":0,"snaptrim_duration":0.0012188170000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":432,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.288802+0000","last_change":"2024-09-10T18:59:02.768985+0000","last_active":"2024-09-10T19:00:04.288802+0000","last_peered":"2024-09-10T19:00:04.288802+0000","last_clean":"2024-09-10T19:00:04.288802+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:00:04.288802+0000","last_undegraded":"2024-09-10T19:00:04.288802+0000","last_fullsized":"2024-09-10T19:00:04.288802+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.764846+0000","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.00084579999999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":408,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263758+0000","last_change":"2024-09-10T18:59:58.060460+0000","last_active":"2024-09-10T19:00:04.263758+0000","last_peered":"2024-09-10T19:00:04.263758+0000","last_clean":"2024-09-10T19:00:04.263758+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:00:04.263758+0000","last_undegraded":"2024-09-10T19:00:04.263758+0000","last_fullsized":"2024-09-10T19:00:04.263758+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:59:58.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.060206+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.833991+0000","scrub_duration":0.0042048110000000001,"objects_trimmed":0,"snaptrim_duration":0.0025643660000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"442'29","reported_seq":401,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.307831+0000","last_change":"2024-09-10T18:57:54.061737+0000","last_active":"2024-09-10T19:00:05.307831+0000","last_peered":"2024-09-10T19:00:05.307831+0000","last_clean":"2024-09-10T19:00:05.307831+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:00:05.307831+0000","last_undegraded":"2024-09-10T19:00:05.307831+0000","last_fullsized":"2024-09-10T19:00:05.307831+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080540149999999994,"objects_trimmed":0,"snaptrim_duration":0.0039749959999999997,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"254'104","reported_seq":451,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266645+0000","last_change":"2024-09-10T18:57:54.071086+0000","last_active":"2024-09-10T19:00:04.266645+0000","last_peered":"2024-09-10T19:00:04.266645+0000","last_clean":"2024-09-10T19:00:04.266645+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:00:04.266645+0000","last_undegraded":"2024-09-10T19:00:04.266645+0000","last_fullsized":"2024-09-10T19:00:04.266645+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T18:57:32.267016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:32.267016+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052553230000000001,"objects_trimmed":0,"snaptrim_duration":0.00069417100000000005,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"428'23","reported_seq":388,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268280+0000","last_change":"2024-09-10T18:57:54.074533+0000","last_active":"2024-09-10T19:00:04.268280+0000","last_peered":"2024-09-10T19:00:04.268280+0000","last_clean":"2024-09-10T19:00:04.268280+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:00:04.268280+0000","last_undegraded":"2024-09-10T19:00:04.268280+0000","last_fullsized":"2024-09-10T19:00:04.268280+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.0066870059999999997,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"387'24","reported_seq":401,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.286926+0000","last_change":"2024-09-10T18:59:03.724989+0000","last_active":"2024-09-10T19:00:04.286926+0000","last_peered":"2024-09-10T19:00:04.286926+0000","last_clean":"2024-09-10T19:00:04.286926+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:00:04.286926+0000","last_undegraded":"2024-09-10T19:00:04.286926+0000","last_fullsized":"2024-09-10T19:00:04.286926+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.707234+0000","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.001361021,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"428'79","reported_seq":447,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.280160+0000","last_change":"2024-09-10T18:59:01.892397+0000","last_active":"2024-09-10T19:00:04.280160+0000","last_peered":"2024-09-10T19:00:04.280160+0000","last_clean":"2024-09-10T19:00:04.280160+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:00:04.280160+0000","last_undegraded":"2024-09-10T19:00:04.280160+0000","last_fullsized":"2024-09-10T19:00:04.280160+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'75","last_scrub_stamp":"2024-09-10T18:59:01.892227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.892227+0000","objects_scrubbed":7,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.595320+0000","scrub_duration":0.0063234340000000002,"objects_trimmed":0,"snaptrim_duration":0.0019943130000000002,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"434'100","reported_seq":473,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.282716+0000","last_change":"2024-09-10T18:59:01.793871+0000","last_active":"2024-09-10T19:00:04.282716+0000","last_peered":"2024-09-10T19:00:04.282716+0000","last_clean":"2024-09-10T19:00:04.282716+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:00:04.282716+0000","last_undegraded":"2024-09-10T19:00:04.282716+0000","last_fullsized":"2024-09-10T19:00:04.282716+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.183490+0000","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.0015587590000000001,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"409'2","reported_seq":385,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.280924+0000","last_change":"2024-09-10T18:58:43.994005+0000","last_active":"2024-09-10T19:00:04.280924+0000","last_peered":"2024-09-10T19:00:04.280924+0000","last_clean":"2024-09-10T19:00:04.280924+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:00:04.280924+0000","last_undegraded":"2024-09-10T19:00:04.280924+0000","last_fullsized":"2024-09-10T19:00:04.280924+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'1","last_scrub_stamp":"2024-09-10T18:58:43.993832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.993832+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040278960000000004,"objects_trimmed":0,"snaptrim_duration":0.0025161630000000001,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":403,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267401+0000","last_change":"2024-09-10T18:58:39.380635+0000","last_active":"2024-09-10T19:00:04.267401+0000","last_peered":"2024-09-10T19:00:04.267401+0000","last_clean":"2024-09-10T19:00:04.267401+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:00:04.267401+0000","last_undegraded":"2024-09-10T19:00:04.267401+0000","last_fullsized":"2024-09-10T19:00:04.267401+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.953708+0000","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.001311361,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"297'52","reported_seq":414,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257388+0000","last_change":"2024-09-10T18:57:54.062338+0000","last_active":"2024-09-10T19:00:04.257388+0000","last_peered":"2024-09-10T19:00:04.257388+0000","last_clean":"2024-09-10T19:00:04.257388+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T19:00:04.257388+0000","last_undegraded":"2024-09-10T19:00:04.257388+0000","last_fullsized":"2024-09-10T19:00:04.257388+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.001141544,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":397,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262807+0000","last_change":"2024-09-10T18:58:37.312137+0000","last_active":"2024-09-10T19:00:04.262807+0000","last_peered":"2024-09-10T19:00:04.262807+0000","last_clean":"2024-09-10T19:00:04.262807+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:00:04.262807+0000","last_undegraded":"2024-09-10T19:00:04.262807+0000","last_fullsized":"2024-09-10T19:00:04.262807+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0054864759999999997,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"431'70","reported_seq":430,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266967+0000","last_change":"2024-09-10T18:57:54.078499+0000","last_active":"2024-09-10T19:00:04.266967+0000","last_peered":"2024-09-10T19:00:04.266967+0000","last_clean":"2024-09-10T19:00:04.266967+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:00:04.266967+0000","last_undegraded":"2024-09-10T19:00:04.266967+0000","last_fullsized":"2024-09-10T19:00:04.266967+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.0014433099999999999,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"246'119","reported_seq":466,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265898+0000","last_change":"2024-09-10T18:57:54.082546+0000","last_active":"2024-09-10T19:00:04.265898+0000","last_peered":"2024-09-10T19:00:04.265898+0000","last_clean":"2024-09-10T19:00:04.265898+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:00:04.265898+0000","last_undegraded":"2024-09-10T19:00:04.265898+0000","last_fullsized":"2024-09-10T19:00:04.265898+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T18:57:38.617203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:38.617203+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046630150000000004,"objects_trimmed":0,"snaptrim_duration":0.0089002290000000008,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"438'22","reported_seq":402,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258825+0000","last_change":"2024-09-10T18:58:56.450425+0000","last_active":"2024-09-10T19:00:04.258825+0000","last_peered":"2024-09-10T19:00:04.258825+0000","last_clean":"2024-09-10T19:00:04.258825+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T19:00:04.258825+0000","last_undegraded":"2024-09-10T19:00:04.258825+0000","last_fullsized":"2024-09-10T19:00:04.258825+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:25.313261+0000","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.0022571480000000001,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"377'23","reported_seq":390,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269575+0000","last_change":"2024-09-10T18:58:47.475482+0000","last_active":"2024-09-10T19:00:04.269575+0000","last_peered":"2024-09-10T19:00:04.269575+0000","last_clean":"2024-09-10T19:00:04.269575+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:00:04.269575+0000","last_undegraded":"2024-09-10T19:00:04.269575+0000","last_fullsized":"2024-09-10T19:00:04.269575+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'22","last_scrub_stamp":"2024-09-10T18:58:47.475285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.475285+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.029410+0000","scrub_duration":0.0041187790000000004,"objects_trimmed":0,"snaptrim_duration":0.001040362,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":399,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.288366+0000","last_change":"2024-09-10T18:57:54.097877+0000","last_active":"2024-09-10T19:00:04.288366+0000","last_peered":"2024-09-10T19:00:04.288366+0000","last_clean":"2024-09-10T19:00:04.288366+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:00:04.288366+0000","last_undegraded":"2024-09-10T19:00:04.288366+0000","last_fullsized":"2024-09-10T19:00:04.288366+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.001225788,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"434'20","reported_seq":400,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278715+0000","last_change":"2024-09-10T18:58:37.063649+0000","last_active":"2024-09-10T19:00:04.278715+0000","last_peered":"2024-09-10T19:00:04.278715+0000","last_clean":"2024-09-10T19:00:04.278715+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:00:04.278715+0000","last_undegraded":"2024-09-10T19:00:04.278715+0000","last_fullsized":"2024-09-10T19:00:04.278715+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:58:37.063474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.063474+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003955554,"objects_trimmed":0,"snaptrim_duration":0.002177216,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":380,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278584+0000","last_change":"2024-09-10T18:59:03.924359+0000","last_active":"2024-09-10T19:00:04.278584+0000","last_peered":"2024-09-10T19:00:04.278584+0000","last_clean":"2024-09-10T19:00:04.278584+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:00:04.278584+0000","last_undegraded":"2024-09-10T19:00:04.278584+0000","last_fullsized":"2024-09-10T19:00:04.278584+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.924176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.924176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:06.750994+0000","scrub_duration":0.003004348,"objects_trimmed":0,"snaptrim_duration":0.0011587170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":422,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270527+0000","last_change":"2024-09-10T19:00:03.403113+0000","last_active":"2024-09-10T19:00:04.270527+0000","last_peered":"2024-09-10T19:00:04.270527+0000","last_clean":"2024-09-10T19:00:04.270527+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:00:04.270527+0000","last_undegraded":"2024-09-10T19:00:04.270527+0000","last_fullsized":"2024-09-10T19:00:04.270527+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T19:00:03.402967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.402967+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.326121+0000","scrub_duration":0.0043031060000000001,"objects_trimmed":0,"snaptrim_duration":0.0021959010000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"409'160","reported_seq":529,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.281172+0000","last_change":"2024-09-10T18:58:53.954029+0000","last_active":"2024-09-10T19:00:04.281172+0000","last_peered":"2024-09-10T19:00:04.281172+0000","last_clean":"2024-09-10T19:00:04.281172+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:00:04.281172+0000","last_undegraded":"2024-09-10T19:00:04.281172+0000","last_fullsized":"2024-09-10T19:00:04.281172+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'159","last_scrub_stamp":"2024-09-10T18:58:53.953850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.953850+0000","objects_scrubbed":11,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.670681+0000","scrub_duration":0.0076573049999999997,"objects_trimmed":0,"snaptrim_duration":0.001287884,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"388'99","reported_seq":516,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266832+0000","last_change":"2024-09-10T18:59:04.213412+0000","last_active":"2024-09-10T19:00:04.266832+0000","last_peered":"2024-09-10T19:00:04.266832+0000","last_clean":"2024-09-10T19:00:04.266832+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:00:04.266832+0000","last_undegraded":"2024-09-10T19:00:04.266832+0000","last_fullsized":"2024-09-10T19:00:04.266832+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.213299+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.577134+0000","scrub_duration":0.022641919,"objects_trimmed":0,"snaptrim_duration":0.0028148299999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":413,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.271628+0000","last_change":"2024-09-10T18:58:44.435895+0000","last_active":"2024-09-10T19:00:04.271628+0000","last_peered":"2024-09-10T19:00:04.271628+0000","last_clean":"2024-09-10T19:00:04.271628+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:00:04.271628+0000","last_undegraded":"2024-09-10T19:00:04.271628+0000","last_fullsized":"2024-09-10T19:00:04.271628+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T18:58:44.435728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.435728+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045739450000000003,"objects_trimmed":0,"snaptrim_duration":0.004517236,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"257'107","reported_seq":461,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264419+0000","last_change":"2024-09-10T18:57:54.060629+0000","last_active":"2024-09-10T19:00:04.264419+0000","last_peered":"2024-09-10T19:00:04.264419+0000","last_clean":"2024-09-10T19:00:04.264419+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:00:04.264419+0000","last_undegraded":"2024-09-10T19:00:04.264419+0000","last_fullsized":"2024-09-10T19:00:04.264419+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T18:57:28.508821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:28.508821+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051100670000000003,"objects_trimmed":0,"snaptrim_duration":0.0033410190000000002,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"431'106","reported_seq":463,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265027+0000","last_change":"2024-09-10T18:58:34.265530+0000","last_active":"2024-09-10T19:00:04.265027+0000","last_peered":"2024-09-10T19:00:04.265027+0000","last_clean":"2024-09-10T19:00:04.265027+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:00:04.265027+0000","last_undegraded":"2024-09-10T19:00:04.265027+0000","last_fullsized":"2024-09-10T19:00:04.265027+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0013311620000000001,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":394,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274107+0000","last_change":"2024-09-10T18:57:54.068000+0000","last_active":"2024-09-10T19:00:04.274107+0000","last_peered":"2024-09-10T19:00:04.274107+0000","last_clean":"2024-09-10T19:00:04.274107+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:00:04.274107+0000","last_undegraded":"2024-09-10T19:00:04.274107+0000","last_fullsized":"2024-09-10T19:00:04.274107+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T18:57:51.498965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.498965+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042645950000000004,"objects_trimmed":0,"snaptrim_duration":0.0022769880000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"409'74","reported_seq":440,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269815+0000","last_change":"2024-09-10T18:58:32.368725+0000","last_active":"2024-09-10T19:00:04.269815+0000","last_peered":"2024-09-10T19:00:04.269815+0000","last_clean":"2024-09-10T19:00:04.269815+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:00:04.269815+0000","last_undegraded":"2024-09-10T19:00:04.269815+0000","last_fullsized":"2024-09-10T19:00:04.269815+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'72","last_scrub_stamp":"2024-09-10T18:58:32.368604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.368604+0000","objects_scrubbed":7,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042920229999999998,"objects_trimmed":0,"snaptrim_duration":0.00066729300000000001,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":462,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278575+0000","last_change":"2024-09-10T18:58:36.017962+0000","last_active":"2024-09-10T19:00:04.278575+0000","last_peered":"2024-09-10T19:00:04.278575+0000","last_clean":"2024-09-10T19:00:04.278575+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:00:04.278575+0000","last_undegraded":"2024-09-10T19:00:04.278575+0000","last_fullsized":"2024-09-10T19:00:04.278575+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T18:58:36.017841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.017841+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042543119999999997,"objects_trimmed":0,"snaptrim_duration":0.00254025,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264596+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T19:00:04.264596+0000","last_peered":"2024-09-10T19:00:04.264596+0000","last_clean":"2024-09-10T19:00:04.264596+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:00:04.264596+0000","last_undegraded":"2024-09-10T19:00:04.264596+0000","last_fullsized":"2024-09-10T19:00:04.264596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.261172+0000","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":429,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265022+0000","last_change":"2024-09-10T18:59:59.106249+0000","last_active":"2024-09-10T19:00:04.265022+0000","last_peered":"2024-09-10T19:00:04.265022+0000","last_clean":"2024-09-10T19:00:04.265022+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:00:04.265022+0000","last_undegraded":"2024-09-10T19:00:04.265022+0000","last_fullsized":"2024-09-10T19:00:04.265022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.106063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.106063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.167241+0000","scrub_duration":0.0034646490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":439,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261030+0000","last_change":"2024-09-10T18:56:27.920865+0000","last_active":"2024-09-10T19:00:04.261030+0000","last_peered":"2024-09-10T19:00:04.261030+0000","last_clean":"2024-09-10T19:00:04.261030+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:00:04.261030+0000","last_undegraded":"2024-09-10T19:00:04.261030+0000","last_fullsized":"2024-09-10T19:00:04.261030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'10","last_scrub_stamp":"2024-09-10T18:56:27.920598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:27.920598+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046713129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266545+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T19:00:04.266545+0000","last_peered":"2024-09-10T19:00:04.266545+0000","last_clean":"2024-09-10T19:00:04.266545+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:00:04.266545+0000","last_undegraded":"2024-09-10T19:00:04.266545+0000","last_fullsized":"2024-09-10T19:00:04.266545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":341,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269430+0000","last_change":"2024-09-10T18:59:53.417704+0000","last_active":"2024-09-10T19:00:04.269430+0000","last_peered":"2024-09-10T19:00:04.269430+0000","last_clean":"2024-09-10T19:00:04.269430+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:00:04.269430+0000","last_undegraded":"2024-09-10T19:00:04.269430+0000","last_fullsized":"2024-09-10T19:00:04.269430+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.417551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.417551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.108833+0000","scrub_duration":0.0034045260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":431,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261601+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T19:00:04.261601+0000","last_peered":"2024-09-10T19:00:04.261601+0000","last_clean":"2024-09-10T19:00:04.261601+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:00:04.261601+0000","last_undegraded":"2024-09-10T19:00:04.261601+0000","last_fullsized":"2024-09-10T19:00:04.261601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":458,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264408+0000","last_change":"2024-09-10T19:00:03.037143+0000","last_active":"2024-09-10T19:00:04.264408+0000","last_peered":"2024-09-10T19:00:04.264408+0000","last_clean":"2024-09-10T19:00:04.264408+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:00:04.264408+0000","last_undegraded":"2024-09-10T19:00:04.264408+0000","last_fullsized":"2024-09-10T19:00:04.264408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:03.036974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.036974+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:32.356994+0000","scrub_duration":0.0043269010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":436,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.256951+0000","last_change":"2024-09-10T19:00:01.347416+0000","last_active":"2024-09-10T19:00:04.256951+0000","last_peered":"2024-09-10T19:00:04.256951+0000","last_clean":"2024-09-10T19:00:04.256951+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:00:04.256951+0000","last_undegraded":"2024-09-10T19:00:04.256951+0000","last_fullsized":"2024-09-10T19:00:04.256951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:01.347241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.347241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.701874+0000","scrub_duration":0.003764398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257134+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T19:00:04.257134+0000","last_peered":"2024-09-10T19:00:04.257134+0000","last_clean":"2024-09-10T19:00:04.257134+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T19:00:04.257134+0000","last_undegraded":"2024-09-10T19:00:04.257134+0000","last_fullsized":"2024-09-10T19:00:04.257134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261410+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T19:00:04.261410+0000","last_peered":"2024-09-10T19:00:04.261410+0000","last_clean":"2024-09-10T19:00:04.261410+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:00:04.261410+0000","last_undegraded":"2024-09-10T19:00:04.261410+0000","last_fullsized":"2024-09-10T19:00:04.261410+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265246+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T19:00:04.265246+0000","last_peered":"2024-09-10T19:00:04.265246+0000","last_clean":"2024-09-10T19:00:04.265246+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:00:04.265246+0000","last_undegraded":"2024-09-10T19:00:04.265246+0000","last_fullsized":"2024-09-10T19:00:04.265246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":462,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269940+0000","last_change":"2024-09-10T18:57:35.279726+0000","last_active":"2024-09-10T19:00:04.269940+0000","last_peered":"2024-09-10T19:00:04.269940+0000","last_clean":"2024-09-10T19:00:04.269940+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:00:04.269940+0000","last_undegraded":"2024-09-10T19:00:04.269940+0000","last_fullsized":"2024-09-10T19:00:04.269940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'22","last_scrub_stamp":"2024-09-10T18:57:35.279454+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.279454+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012463752999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258045+0000","last_change":"2024-09-10T18:58:53.495436+0000","last_active":"2024-09-10T19:00:04.258045+0000","last_peered":"2024-09-10T19:00:04.258045+0000","last_clean":"2024-09-10T19:00:04.258045+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T19:00:04.258045+0000","last_undegraded":"2024-09-10T19:00:04.258045+0000","last_fullsized":"2024-09-10T19:00:04.258045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.495273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.495273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.149497+0000","scrub_duration":0.0024582229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269588+0000","last_change":"2024-09-10T19:00:03.208547+0000","last_active":"2024-09-10T19:00:04.269588+0000","last_peered":"2024-09-10T19:00:04.269588+0000","last_clean":"2024-09-10T19:00:04.269588+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:00:04.269588+0000","last_undegraded":"2024-09-10T19:00:04.269588+0000","last_fullsized":"2024-09-10T19:00:04.269588+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.208468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.208468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.413505+0000","scrub_duration":0.0019202379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.288559+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T19:00:04.288559+0000","last_peered":"2024-09-10T19:00:04.288559+0000","last_clean":"2024-09-10T19:00:04.288559+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:00:04.288559+0000","last_undegraded":"2024-09-10T19:00:04.288559+0000","last_fullsized":"2024-09-10T19:00:04.288559+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":430,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264350+0000","last_change":"2024-09-10T18:59:03.215137+0000","last_active":"2024-09-10T19:00:04.264350+0000","last_peered":"2024-09-10T19:00:04.264350+0000","last_clean":"2024-09-10T19:00:04.264350+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:00:04.264350+0000","last_undegraded":"2024-09-10T19:00:04.264350+0000","last_fullsized":"2024-09-10T19:00:04.264350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.215039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.215039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:32.858523+0000","scrub_duration":0.0064108910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":469,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.271425+0000","last_change":"2024-09-10T18:59:05.516222+0000","last_active":"2024-09-10T19:00:04.271425+0000","last_peered":"2024-09-10T19:00:04.271425+0000","last_clean":"2024-09-10T19:00:04.271425+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:00:04.271425+0000","last_undegraded":"2024-09-10T19:00:04.271425+0000","last_fullsized":"2024-09-10T19:00:04.271425+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:59:05.516062+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:05.516062+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.578286+0000","scrub_duration":0.0058657070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257423+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T19:00:04.257423+0000","last_peered":"2024-09-10T19:00:04.257423+0000","last_clean":"2024-09-10T19:00:04.257423+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T19:00:04.257423+0000","last_undegraded":"2024-09-10T19:00:04.257423+0000","last_fullsized":"2024-09-10T19:00:04.257423+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":423,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260348+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T19:00:04.260348+0000","last_peered":"2024-09-10T19:00:04.260348+0000","last_clean":"2024-09-10T19:00:04.260348+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:00:04.260348+0000","last_undegraded":"2024-09-10T19:00:04.260348+0000","last_fullsized":"2024-09-10T19:00:04.260348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.122348+0000","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":336,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267548+0000","last_change":"2024-09-10T18:56:43.136975+0000","last_active":"2024-09-10T19:00:04.267548+0000","last_peered":"2024-09-10T19:00:04.267548+0000","last_clean":"2024-09-10T19:00:04.267548+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:00:04.267548+0000","last_undegraded":"2024-09-10T19:00:04.267548+0000","last_fullsized":"2024-09-10T19:00:04.267548+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:43.136906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:43.136906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00178308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":480,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274325+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T19:00:04.274325+0000","last_peered":"2024-09-10T19:00:04.274325+0000","last_clean":"2024-09-10T19:00:04.274325+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:00:04.274325+0000","last_undegraded":"2024-09-10T19:00:04.274325+0000","last_fullsized":"2024-09-10T19:00:04.274325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265577+0000","last_change":"2024-09-10T18:59:57.030876+0000","last_active":"2024-09-10T19:00:04.265577+0000","last_peered":"2024-09-10T19:00:04.265577+0000","last_clean":"2024-09-10T19:00:04.265577+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:00:04.265577+0000","last_undegraded":"2024-09-10T19:00:04.265577+0000","last_fullsized":"2024-09-10T19:00:04.265577+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.030798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.030798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.038402+0000","scrub_duration":0.0020075510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":566,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259136+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T19:00:04.259136+0000","last_peered":"2024-09-10T19:00:04.259136+0000","last_clean":"2024-09-10T19:00:04.259136+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T19:00:04.259136+0000","last_undegraded":"2024-09-10T19:00:04.259136+0000","last_fullsized":"2024-09-10T19:00:04.259136+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":433,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.281711+0000","last_change":"2024-09-10T18:59:00.907296+0000","last_active":"2024-09-10T19:00:04.281711+0000","last_peered":"2024-09-10T19:00:04.281711+0000","last_clean":"2024-09-10T19:00:04.281711+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:00:04.281711+0000","last_undegraded":"2024-09-10T19:00:04.281711+0000","last_fullsized":"2024-09-10T19:00:04.281711+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.907056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.861464+0000","scrub_duration":0.0036581080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268974+0000","last_change":"2024-09-10T18:59:59.486802+0000","last_active":"2024-09-10T19:00:04.268974+0000","last_peered":"2024-09-10T19:00:04.268974+0000","last_clean":"2024-09-10T19:00:04.268974+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:00:04.268974+0000","last_undegraded":"2024-09-10T19:00:04.268974+0000","last_fullsized":"2024-09-10T19:00:04.268974+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.486653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.486653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:19.115655+0000","scrub_duration":0.003394902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":458,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265145+0000","last_change":"2024-09-10T18:59:57.094883+0000","last_active":"2024-09-10T19:00:04.265145+0000","last_peered":"2024-09-10T19:00:04.265145+0000","last_clean":"2024-09-10T19:00:04.265145+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:00:04.265145+0000","last_undegraded":"2024-09-10T19:00:04.265145+0000","last_fullsized":"2024-09-10T19:00:04.265145+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T18:59:57.094721+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.094721+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.024150+0000","scrub_duration":0.0045594349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":438,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.277794+0000","last_change":"2024-09-10T18:58:52.884147+0000","last_active":"2024-09-10T19:00:04.277794+0000","last_peered":"2024-09-10T19:00:04.277794+0000","last_clean":"2024-09-10T19:00:04.277794+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:00:04.277794+0000","last_undegraded":"2024-09-10T19:00:04.277794+0000","last_fullsized":"2024-09-10T19:00:04.277794+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.884077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032688370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":429,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259075+0000","last_change":"2024-09-10T19:00:02.315530+0000","last_active":"2024-09-10T19:00:04.259075+0000","last_peered":"2024-09-10T19:00:04.259075+0000","last_clean":"2024-09-10T19:00:04.259075+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T19:00:04.259075+0000","last_undegraded":"2024-09-10T19:00:04.259075+0000","last_fullsized":"2024-09-10T19:00:04.259075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259527+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T19:00:04.259527+0000","last_peered":"2024-09-10T19:00:04.259527+0000","last_clean":"2024-09-10T19:00:04.259527+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T19:00:04.259527+0000","last_undegraded":"2024-09-10T19:00:04.259527+0000","last_fullsized":"2024-09-10T19:00:04.259527+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270125+0000","last_change":"2024-09-10T18:57:37.580205+0000","last_active":"2024-09-10T19:00:04.270125+0000","last_peered":"2024-09-10T19:00:04.270125+0000","last_clean":"2024-09-10T19:00:04.270125+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:00:04.270125+0000","last_undegraded":"2024-09-10T19:00:04.270125+0000","last_fullsized":"2024-09-10T19:00:04.270125+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.580106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.580106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034251759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":462,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263453+0000","last_change":"2024-09-10T18:56:32.989712+0000","last_active":"2024-09-10T19:00:04.263453+0000","last_peered":"2024-09-10T19:00:04.263453+0000","last_clean":"2024-09-10T19:00:04.263453+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:00:04.263453+0000","last_undegraded":"2024-09-10T19:00:04.263453+0000","last_fullsized":"2024-09-10T19:00:04.263453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'15","last_scrub_stamp":"2024-09-10T18:56:32.989637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:32.989637+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043333410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":428,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.275163+0000","last_change":"2024-09-10T18:58:51.907906+0000","last_active":"2024-09-10T19:00:04.275163+0000","last_peered":"2024-09-10T19:00:04.275163+0000","last_clean":"2024-09-10T19:00:04.275163+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:00:04.275163+0000","last_undegraded":"2024-09-10T19:00:04.275163+0000","last_fullsized":"2024-09-10T19:00:04.275163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:51.907749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:51.907749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036594549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":434,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.289112+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T19:00:04.289112+0000","last_peered":"2024-09-10T19:00:04.289112+0000","last_clean":"2024-09-10T19:00:04.289112+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:00:04.289112+0000","last_undegraded":"2024-09-10T19:00:04.289112+0000","last_fullsized":"2024-09-10T19:00:04.289112+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":428,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.271130+0000","last_change":"2024-09-10T18:59:57.328269+0000","last_active":"2024-09-10T19:00:04.271130+0000","last_peered":"2024-09-10T19:00:04.271130+0000","last_clean":"2024-09-10T19:00:04.271130+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:00:04.271130+0000","last_undegraded":"2024-09-10T19:00:04.271130+0000","last_fullsized":"2024-09-10T19:00:04.271130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.328111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.328111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.067869+0000","scrub_duration":0.0032441140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.291191+0000","last_change":"2024-09-10T18:57:56.827061+0000","last_active":"2024-09-10T19:00:04.291191+0000","last_peered":"2024-09-10T19:00:04.291191+0000","last_clean":"2024-09-10T19:00:04.291191+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:00:04.291191+0000","last_undegraded":"2024-09-10T19:00:04.291191+0000","last_fullsized":"2024-09-10T19:00:04.291191+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.826954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.826954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032518439999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":475,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258266+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T19:00:04.258266+0000","last_peered":"2024-09-10T19:00:04.258266+0000","last_clean":"2024-09-10T19:00:04.258266+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:00:04.258266+0000","last_undegraded":"2024-09-10T19:00:04.258266+0000","last_fullsized":"2024-09-10T19:00:04.258266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:27.341520+0000","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264868+0000","last_change":"2024-09-10T18:58:32.226883+0000","last_active":"2024-09-10T19:00:04.264868+0000","last_peered":"2024-09-10T19:00:04.264868+0000","last_clean":"2024-09-10T19:00:04.264868+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:00:04.264868+0000","last_undegraded":"2024-09-10T19:00:04.264868+0000","last_fullsized":"2024-09-10T19:00:04.264868+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.226771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.226771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024188028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":576,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.221489+0000","last_change":"2024-09-10T19:00:05.221386+0000","last_active":"2024-09-10T19:00:05.221489+0000","last_peered":"2024-09-10T19:00:05.221489+0000","last_clean":"2024-09-10T19:00:05.221489+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T19:00:05.221489+0000","last_undegraded":"2024-09-10T19:00:05.221489+0000","last_fullsized":"2024-09-10T19:00:05.221489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T19:00:05.221191+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.221191+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.966409+0000","scrub_duration":0.0039722840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":416,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264020+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T19:00:04.264020+0000","last_peered":"2024-09-10T19:00:04.264020+0000","last_clean":"2024-09-10T19:00:04.264020+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:00:04.264020+0000","last_undegraded":"2024-09-10T19:00:04.264020+0000","last_fullsized":"2024-09-10T19:00:04.264020+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287079+0000","last_change":"2024-09-10T18:58:47.871654+0000","last_active":"2024-09-10T19:00:04.287079+0000","last_peered":"2024-09-10T19:00:04.287079+0000","last_clean":"2024-09-10T19:00:04.287079+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:00:04.287079+0000","last_undegraded":"2024-09-10T19:00:04.287079+0000","last_fullsized":"2024-09-10T19:00:04.287079+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.871591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.871591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023117910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":453,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264107+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T19:00:04.264107+0000","last_peered":"2024-09-10T19:00:04.264107+0000","last_clean":"2024-09-10T19:00:04.264107+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:00:04.264107+0000","last_undegraded":"2024-09-10T19:00:04.264107+0000","last_fullsized":"2024-09-10T19:00:04.264107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268778+0000","last_change":"2024-09-10T18:58:50.597419+0000","last_active":"2024-09-10T19:00:04.268778+0000","last_peered":"2024-09-10T19:00:04.268778+0000","last_clean":"2024-09-10T19:00:04.268778+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:00:04.268778+0000","last_undegraded":"2024-09-10T19:00:04.268778+0000","last_fullsized":"2024-09-10T19:00:04.268778+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.597321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.597321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021838199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257950+0000","last_change":"2024-09-10T18:57:41.770708+0000","last_active":"2024-09-10T19:00:04.257950+0000","last_peered":"2024-09-10T19:00:04.257950+0000","last_clean":"2024-09-10T19:00:04.257950+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T19:00:04.257950+0000","last_undegraded":"2024-09-10T19:00:04.257950+0000","last_fullsized":"2024-09-10T19:00:04.257950+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.770525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034795440000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1057,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.060596+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T19:00:05.060596+0000","last_peered":"2024-09-10T19:00:05.060596+0000","last_clean":"2024-09-10T19:00:05.060596+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:00:05.060596+0000","last_undegraded":"2024-09-10T19:00:05.060596+0000","last_fullsized":"2024-09-10T19:00:05.060596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":622,"num_read_kb":617,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":431,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265086+0000","last_change":"2024-09-10T19:00:04.106819+0000","last_active":"2024-09-10T19:00:04.265086+0000","last_peered":"2024-09-10T19:00:04.265086+0000","last_clean":"2024-09-10T19:00:04.265086+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:00:04.265086+0000","last_undegraded":"2024-09-10T19:00:04.265086+0000","last_fullsized":"2024-09-10T19:00:04.265086+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.106660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.106660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.468919+0000","scrub_duration":0.0031927650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":355,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278497+0000","last_change":"2024-09-10T18:59:06.979721+0000","last_active":"2024-09-10T19:00:04.278497+0000","last_peered":"2024-09-10T19:00:04.278497+0000","last_clean":"2024-09-10T19:00:04.278497+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:00:04.278497+0000","last_undegraded":"2024-09-10T19:00:04.278497+0000","last_fullsized":"2024-09-10T19:00:04.278497+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.979599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.979599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.070088+0000","scrub_duration":0.0031493580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":462,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262661+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T19:00:04.262661+0000","last_peered":"2024-09-10T19:00:04.262661+0000","last_clean":"2024-09-10T19:00:04.262661+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:00:04.262661+0000","last_undegraded":"2024-09-10T19:00:04.262661+0000","last_fullsized":"2024-09-10T19:00:04.262661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.664434+0000","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":438,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.287520+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T19:00:04.287520+0000","last_peered":"2024-09-10T19:00:04.287520+0000","last_clean":"2024-09-10T19:00:04.287520+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:00:04.287520+0000","last_undegraded":"2024-09-10T19:00:04.287520+0000","last_fullsized":"2024-09-10T19:00:04.287520+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":556,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263168+0000","last_change":"2024-09-10T18:59:59.066338+0000","last_active":"2024-09-10T19:00:04.263168+0000","last_peered":"2024-09-10T19:00:04.263168+0000","last_clean":"2024-09-10T19:00:04.263168+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:00:04.263168+0000","last_undegraded":"2024-09-10T19:00:04.263168+0000","last_fullsized":"2024-09-10T19:00:04.263168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:59:59.066136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.066136+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.679135+0000","scrub_duration":0.0038648509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":443,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257668+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T19:00:04.257668+0000","last_peered":"2024-09-10T19:00:04.257668+0000","last_clean":"2024-09-10T19:00:04.257668+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:00:04.257668+0000","last_undegraded":"2024-09-10T19:00:04.257668+0000","last_fullsized":"2024-09-10T19:00:04.257668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.789834+0000","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257268+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T19:00:04.257268+0000","last_peered":"2024-09-10T19:00:04.257268+0000","last_clean":"2024-09-10T19:00:04.257268+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T19:00:04.257268+0000","last_undegraded":"2024-09-10T19:00:04.257268+0000","last_fullsized":"2024-09-10T19:00:04.257268+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":336,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263092+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T19:00:04.263092+0000","last_peered":"2024-09-10T19:00:04.263092+0000","last_clean":"2024-09-10T19:00:04.263092+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:00:04.263092+0000","last_undegraded":"2024-09-10T19:00:04.263092+0000","last_fullsized":"2024-09-10T19:00:04.263092+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":643,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.283180+0000","last_change":"2024-09-10T19:00:01.595171+0000","last_active":"2024-09-10T19:00:04.283180+0000","last_peered":"2024-09-10T19:00:04.283180+0000","last_clean":"2024-09-10T19:00:04.283180+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:00:04.283180+0000","last_undegraded":"2024-09-10T19:00:04.283180+0000","last_fullsized":"2024-09-10T19:00:04.283180+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:00:01.595006+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.595006+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.951465+0000","scrub_duration":0.0053873050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265911+0000","last_change":"2024-09-10T18:58:50.209102+0000","last_active":"2024-09-10T19:00:04.265911+0000","last_peered":"2024-09-10T19:00:04.265911+0000","last_clean":"2024-09-10T19:00:04.265911+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:00:04.265911+0000","last_undegraded":"2024-09-10T19:00:04.265911+0000","last_fullsized":"2024-09-10T19:00:04.265911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.208993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.208993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003212126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":565,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261025+0000","last_change":"2024-09-10T18:59:56.107053+0000","last_active":"2024-09-10T19:00:04.261025+0000","last_peered":"2024-09-10T19:00:04.261025+0000","last_clean":"2024-09-10T19:00:04.261025+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:00:04.261025+0000","last_undegraded":"2024-09-10T19:00:04.261025+0000","last_fullsized":"2024-09-10T19:00:04.261025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:59:56.106896+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:59:56.106896+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.084878+0000","scrub_duration":0.0036065950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":431,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274186+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T19:00:04.274186+0000","last_peered":"2024-09-10T19:00:04.274186+0000","last_clean":"2024-09-10T19:00:04.274186+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:00:04.274186+0000","last_undegraded":"2024-09-10T19:00:04.274186+0000","last_fullsized":"2024-09-10T19:00:04.274186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":422,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268506+0000","last_change":"2024-09-10T18:58:45.871468+0000","last_active":"2024-09-10T19:00:04.268506+0000","last_peered":"2024-09-10T19:00:04.268506+0000","last_clean":"2024-09-10T19:00:04.268506+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:00:04.268506+0000","last_undegraded":"2024-09-10T19:00:04.268506+0000","last_fullsized":"2024-09-10T19:00:04.268506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:45.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:45.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033902770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":336,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265395+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T19:00:04.265395+0000","last_peered":"2024-09-10T19:00:04.265395+0000","last_clean":"2024-09-10T19:00:04.265395+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:00:04.265395+0000","last_undegraded":"2024-09-10T19:00:04.265395+0000","last_fullsized":"2024-09-10T19:00:04.265395+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":462,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.271492+0000","last_change":"2024-09-10T18:58:46.437660+0000","last_active":"2024-09-10T19:00:04.271492+0000","last_peered":"2024-09-10T19:00:04.271492+0000","last_clean":"2024-09-10T19:00:04.271492+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:00:04.271492+0000","last_undegraded":"2024-09-10T19:00:04.271492+0000","last_fullsized":"2024-09-10T19:00:04.271492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:46.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.437410+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.595024+0000","scrub_duration":0.0054784669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":428,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.288868+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T19:00:04.288868+0000","last_peered":"2024-09-10T19:00:04.288868+0000","last_clean":"2024-09-10T19:00:04.288868+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:00:04.288868+0000","last_undegraded":"2024-09-10T19:00:04.288868+0000","last_fullsized":"2024-09-10T19:00:04.288868+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.276150+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T19:00:04.276150+0000","last_peered":"2024-09-10T19:00:04.276150+0000","last_clean":"2024-09-10T19:00:04.276150+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:00:04.276150+0000","last_undegraded":"2024-09-10T19:00:04.276150+0000","last_fullsized":"2024-09-10T19:00:04.276150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":423,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260866+0000","last_change":"2024-09-10T19:00:00.056759+0000","last_active":"2024-09-10T19:00:04.260866+0000","last_peered":"2024-09-10T19:00:04.260866+0000","last_clean":"2024-09-10T19:00:04.260866+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:00:04.260866+0000","last_undegraded":"2024-09-10T19:00:04.260866+0000","last_fullsized":"2024-09-10T19:00:04.260866+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.056600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.056600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.057092+0000","scrub_duration":0.0028960119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":355,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278807+0000","last_change":"2024-09-10T18:59:58.063484+0000","last_active":"2024-09-10T19:00:04.278807+0000","last_peered":"2024-09-10T19:00:04.278807+0000","last_clean":"2024-09-10T19:00:04.278807+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:00:04.278807+0000","last_undegraded":"2024-09-10T19:00:04.278807+0000","last_fullsized":"2024-09-10T19:00:04.278807+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.063321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.063321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:58.931188+0000","scrub_duration":0.0034568020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":479,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258642+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T19:00:04.258642+0000","last_peered":"2024-09-10T19:00:04.258642+0000","last_clean":"2024-09-10T19:00:04.258642+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T19:00:04.258642+0000","last_undegraded":"2024-09-10T19:00:04.258642+0000","last_fullsized":"2024-09-10T19:00:04.258642+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267820+0000","last_change":"2024-09-10T18:58:52.447688+0000","last_active":"2024-09-10T19:00:04.267820+0000","last_peered":"2024-09-10T19:00:04.267820+0000","last_clean":"2024-09-10T19:00:04.267820+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:00:04.267820+0000","last_undegraded":"2024-09-10T19:00:04.267820+0000","last_fullsized":"2024-09-10T19:00:04.267820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.447547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.447547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.724802+0000","scrub_duration":0.0031696559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.283097+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T19:00:04.283097+0000","last_peered":"2024-09-10T19:00:04.283097+0000","last_clean":"2024-09-10T19:00:04.283097+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:00:04.283097+0000","last_undegraded":"2024-09-10T19:00:04.283097+0000","last_fullsized":"2024-09-10T19:00:04.283097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260904+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T19:00:04.260904+0000","last_peered":"2024-09-10T19:00:04.260904+0000","last_clean":"2024-09-10T19:00:04.260904+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:00:04.260904+0000","last_undegraded":"2024-09-10T19:00:04.260904+0000","last_fullsized":"2024-09-10T19:00:04.260904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261591+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T19:00:04.261591+0000","last_peered":"2024-09-10T19:00:04.261591+0000","last_clean":"2024-09-10T19:00:04.261591+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:00:04.261591+0000","last_undegraded":"2024-09-10T19:00:04.261591+0000","last_fullsized":"2024-09-10T19:00:04.261591+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":484,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267982+0000","last_change":"2024-09-10T18:57:39.593192+0000","last_active":"2024-09-10T19:00:04.267982+0000","last_peered":"2024-09-10T19:00:04.267982+0000","last_clean":"2024-09-10T19:00:04.267982+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:00:04.267982+0000","last_undegraded":"2024-09-10T19:00:04.267982+0000","last_fullsized":"2024-09-10T19:00:04.267982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'26","last_scrub_stamp":"2024-09-10T18:57:39.593100+0000","last_deep_scrub":"104'26","last_deep_scrub_stamp":"2024-09-10T18:51:58.430868+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.593100+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278393+0000","last_change":"2024-09-10T18:59:59.081414+0000","last_active":"2024-09-10T19:00:04.278393+0000","last_peered":"2024-09-10T19:00:04.278393+0000","last_clean":"2024-09-10T19:00:04.278393+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:00:04.278393+0000","last_undegraded":"2024-09-10T19:00:04.278393+0000","last_fullsized":"2024-09-10T19:00:04.278393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.081277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.081277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.759642+0000","scrub_duration":0.002648634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.291275+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T19:00:04.291275+0000","last_peered":"2024-09-10T19:00:04.291275+0000","last_clean":"2024-09-10T19:00:04.291275+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:00:04.291275+0000","last_undegraded":"2024-09-10T19:00:04.291275+0000","last_fullsized":"2024-09-10T19:00:04.291275+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265434+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T19:00:04.265434+0000","last_peered":"2024-09-10T19:00:04.265434+0000","last_clean":"2024-09-10T19:00:04.265434+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:00:04.265434+0000","last_undegraded":"2024-09-10T19:00:04.265434+0000","last_fullsized":"2024-09-10T19:00:04.265434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268432+0000","last_change":"2024-09-10T18:59:02.713009+0000","last_active":"2024-09-10T19:00:04.268432+0000","last_peered":"2024-09-10T19:00:04.268432+0000","last_clean":"2024-09-10T19:00:04.268432+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:00:04.268432+0000","last_undegraded":"2024-09-10T19:00:04.268432+0000","last_fullsized":"2024-09-10T19:00:04.268432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.712839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.712839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035464870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":469,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262137+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T19:00:04.262137+0000","last_peered":"2024-09-10T19:00:04.262137+0000","last_clean":"2024-09-10T19:00:04.262137+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:00:04.262137+0000","last_undegraded":"2024-09-10T19:00:04.262137+0000","last_fullsized":"2024-09-10T19:00:04.262137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.541418+0000","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264455+0000","last_change":"2024-09-10T18:58:44.193043+0000","last_active":"2024-09-10T19:00:04.264455+0000","last_peered":"2024-09-10T19:00:04.264455+0000","last_clean":"2024-09-10T19:00:04.264455+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:00:04.264455+0000","last_undegraded":"2024-09-10T19:00:04.264455+0000","last_fullsized":"2024-09-10T19:00:04.264455+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.192853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.192853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038691179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":425,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264219+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T19:00:04.264219+0000","last_peered":"2024-09-10T19:00:04.264219+0000","last_clean":"2024-09-10T19:00:04.264219+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:00:04.264219+0000","last_undegraded":"2024-09-10T19:00:04.264219+0000","last_fullsized":"2024-09-10T19:00:04.264219+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.522110+0000","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":432,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.282859+0000","last_change":"2024-09-10T19:00:03.612383+0000","last_active":"2024-09-10T19:00:04.282859+0000","last_peered":"2024-09-10T19:00:04.282859+0000","last_clean":"2024-09-10T19:00:04.282859+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:00:04.282859+0000","last_undegraded":"2024-09-10T19:00:04.282859+0000","last_fullsized":"2024-09-10T19:00:04.282859+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.612225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.557620+0000","scrub_duration":0.0033384840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264331+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T19:00:04.264331+0000","last_peered":"2024-09-10T19:00:04.264331+0000","last_clean":"2024-09-10T19:00:04.264331+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:00:04.264331+0000","last_undegraded":"2024-09-10T19:00:04.264331+0000","last_fullsized":"2024-09-10T19:00:04.264331+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":454,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267696+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T19:00:04.267696+0000","last_peered":"2024-09-10T19:00:04.267696+0000","last_clean":"2024-09-10T19:00:04.267696+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:00:04.267696+0000","last_undegraded":"2024-09-10T19:00:04.267696+0000","last_fullsized":"2024-09-10T19:00:04.267696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.648653+0000","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265445+0000","last_change":"2024-09-10T18:57:56.125998+0000","last_active":"2024-09-10T19:00:04.265445+0000","last_peered":"2024-09-10T19:00:04.265445+0000","last_clean":"2024-09-10T19:00:04.265445+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:00:04.265445+0000","last_undegraded":"2024-09-10T19:00:04.265445+0000","last_fullsized":"2024-09-10T19:00:04.265445+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.125881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:10.406977+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.125881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026334969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":475,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264470+0000","last_change":"2024-09-10T18:57:56.375088+0000","last_active":"2024-09-10T19:00:04.264470+0000","last_peered":"2024-09-10T19:00:04.264470+0000","last_clean":"2024-09-10T19:00:04.264470+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T19:00:04.264470+0000","last_undegraded":"2024-09-10T19:00:04.264470+0000","last_fullsized":"2024-09-10T19:00:04.264470+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T18:57:56.374923+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.374923+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043604359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":423,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.269214+0000","last_change":"2024-09-10T18:58:47.555840+0000","last_active":"2024-09-10T19:00:04.269214+0000","last_peered":"2024-09-10T19:00:04.269214+0000","last_clean":"2024-09-10T19:00:04.269214+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:00:04.269214+0000","last_undegraded":"2024-09-10T19:00:04.269214+0000","last_fullsized":"2024-09-10T19:00:04.269214+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.555637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.555637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.468673+0000","scrub_duration":0.0032502030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":418,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266417+0000","last_change":"2024-09-10T18:58:59.454163+0000","last_active":"2024-09-10T19:00:04.266417+0000","last_peered":"2024-09-10T19:00:04.266417+0000","last_clean":"2024-09-10T19:00:04.266417+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:00:04.266417+0000","last_undegraded":"2024-09-10T19:00:04.266417+0000","last_fullsized":"2024-09-10T19:00:04.266417+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.565579+0000","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265749+0000","last_change":"2024-09-10T18:58:54.237739+0000","last_active":"2024-09-10T19:00:04.265749+0000","last_peered":"2024-09-10T19:00:04.265749+0000","last_clean":"2024-09-10T19:00:04.265749+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:00:04.265749+0000","last_undegraded":"2024-09-10T19:00:04.265749+0000","last_fullsized":"2024-09-10T19:00:04.265749+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.237585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.237585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.680207+0000","scrub_duration":0.0092150240000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":482,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265829+0000","last_change":"2024-09-10T18:58:52.253613+0000","last_active":"2024-09-10T19:00:04.265829+0000","last_peered":"2024-09-10T19:00:04.265829+0000","last_clean":"2024-09-10T19:00:04.265829+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:00:04.265829+0000","last_undegraded":"2024-09-10T19:00:04.265829+0000","last_fullsized":"2024-09-10T19:00:04.265829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'25","last_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.253376+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:05.678764+0000","scrub_duration":0.011205784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":439,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.281027+0000","last_change":"2024-09-10T18:58:38.064448+0000","last_active":"2024-09-10T19:00:04.281027+0000","last_peered":"2024-09-10T19:00:04.281027+0000","last_clean":"2024-09-10T19:00:04.281027+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:00:04.281027+0000","last_undegraded":"2024-09-10T19:00:04.281027+0000","last_fullsized":"2024-09-10T19:00:04.281027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.064267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.064267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037942050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":423,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274406+0000","last_change":"2024-09-10T18:58:42.486651+0000","last_active":"2024-09-10T19:00:04.274406+0000","last_peered":"2024-09-10T19:00:04.274406+0000","last_clean":"2024-09-10T19:00:04.274406+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:00:04.274406+0000","last_undegraded":"2024-09-10T19:00:04.274406+0000","last_fullsized":"2024-09-10T19:00:04.274406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:42.486498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:42.486498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032582879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264935+0000","last_change":"2024-09-10T18:58:55.261885+0000","last_active":"2024-09-10T19:00:04.264935+0000","last_peered":"2024-09-10T19:00:04.264935+0000","last_clean":"2024-09-10T19:00:04.264935+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:00:04.264935+0000","last_undegraded":"2024-09-10T19:00:04.264935+0000","last_fullsized":"2024-09-10T19:00:04.264935+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.261635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.261635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.433144+0000","scrub_duration":0.0046216190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.256807+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T19:00:04.256807+0000","last_peered":"2024-09-10T19:00:04.256807+0000","last_clean":"2024-09-10T19:00:04.256807+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:00:04.256807+0000","last_undegraded":"2024-09-10T19:00:04.256807+0000","last_fullsized":"2024-09-10T19:00:04.256807+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.666116+0000","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":457,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270088+0000","last_change":"2024-09-10T18:59:01.489222+0000","last_active":"2024-09-10T19:00:04.270088+0000","last_peered":"2024-09-10T19:00:04.270088+0000","last_clean":"2024-09-10T19:00:04.270088+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:00:04.270088+0000","last_undegraded":"2024-09-10T19:00:04.270088+0000","last_fullsized":"2024-09-10T19:00:04.270088+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:59:01.489057+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.489057+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:21.754146+0000","scrub_duration":0.0045114960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":428,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.286898+0000","last_change":"2024-09-10T18:58:38.864525+0000","last_active":"2024-09-10T19:00:04.286898+0000","last_peered":"2024-09-10T19:00:04.286898+0000","last_clean":"2024-09-10T19:00:04.286898+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:00:04.286898+0000","last_undegraded":"2024-09-10T19:00:04.286898+0000","last_fullsized":"2024-09-10T19:00:04.286898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.864360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.864360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260775+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T19:00:04.260775+0000","last_peered":"2024-09-10T19:00:04.260775+0000","last_clean":"2024-09-10T19:00:04.260775+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:00:04.260775+0000","last_undegraded":"2024-09-10T19:00:04.260775+0000","last_fullsized":"2024-09-10T19:00:04.260775+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.575228+0000","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":413,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262014+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T19:00:04.262014+0000","last_peered":"2024-09-10T19:00:04.262014+0000","last_clean":"2024-09-10T19:00:04.262014+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:00:04.262014+0000","last_undegraded":"2024-09-10T19:00:04.262014+0000","last_fullsized":"2024-09-10T19:00:04.262014+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260903+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T19:00:04.260903+0000","last_peered":"2024-09-10T19:00:04.260903+0000","last_clean":"2024-09-10T19:00:04.260903+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:00:04.260903+0000","last_undegraded":"2024-09-10T19:00:04.260903+0000","last_fullsized":"2024-09-10T19:00:04.260903+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":475,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268150+0000","last_change":"2024-09-10T18:58:48.217515+0000","last_active":"2024-09-10T19:00:04.268150+0000","last_peered":"2024-09-10T19:00:04.268150+0000","last_clean":"2024-09-10T19:00:04.268150+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:00:04.268150+0000","last_undegraded":"2024-09-10T19:00:04.268150+0000","last_fullsized":"2024-09-10T19:00:04.268150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:58:48.217409+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.217409+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034746339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":451,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.283461+0000","last_change":"2024-09-10T18:58:34.970321+0000","last_active":"2024-09-10T19:00:04.283461+0000","last_peered":"2024-09-10T19:00:04.283461+0000","last_clean":"2024-09-10T19:00:04.283461+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:00:04.283461+0000","last_undegraded":"2024-09-10T19:00:04.283461+0000","last_fullsized":"2024-09-10T19:00:04.283461+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:34.970167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.970167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036040859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":430,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266761+0000","last_change":"2024-09-10T19:00:00.313745+0000","last_active":"2024-09-10T19:00:04.266761+0000","last_peered":"2024-09-10T19:00:04.266761+0000","last_clean":"2024-09-10T19:00:04.266761+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:00:04.266761+0000","last_undegraded":"2024-09-10T19:00:04.266761+0000","last_fullsized":"2024-09-10T19:00:04.266761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.313582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.313582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.500345+0000","scrub_duration":0.0034259580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":418,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267640+0000","last_change":"2024-09-10T18:58:37.374134+0000","last_active":"2024-09-10T19:00:04.267640+0000","last_peered":"2024-09-10T19:00:04.267640+0000","last_clean":"2024-09-10T19:00:04.267640+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:00:04.267640+0000","last_undegraded":"2024-09-10T19:00:04.267640+0000","last_fullsized":"2024-09-10T19:00:04.267640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.374014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.374014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023318560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.256718+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T19:00:04.256718+0000","last_peered":"2024-09-10T19:00:04.256718+0000","last_clean":"2024-09-10T19:00:04.256718+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T19:00:04.256718+0000","last_undegraded":"2024-09-10T19:00:04.256718+0000","last_fullsized":"2024-09-10T19:00:04.256718+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:08.117442+0000","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":456,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267323+0000","last_change":"2024-09-10T18:58:47.178004+0000","last_active":"2024-09-10T19:00:04.267323+0000","last_peered":"2024-09-10T19:00:04.267323+0000","last_clean":"2024-09-10T19:00:04.267323+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:00:04.267323+0000","last_undegraded":"2024-09-10T19:00:04.267323+0000","last_fullsized":"2024-09-10T19:00:04.267323+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:47.177751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.177751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059440730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":479,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260061+0000","last_change":"2024-09-10T18:59:59.421165+0000","last_active":"2024-09-10T19:00:04.260061+0000","last_peered":"2024-09-10T19:00:04.260061+0000","last_clean":"2024-09-10T19:00:04.260061+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:00:04.260061+0000","last_undegraded":"2024-09-10T19:00:04.260061+0000","last_fullsized":"2024-09-10T19:00:04.260061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:59:59.420998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.420998+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.365936+0000","scrub_duration":0.0045326000000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":565,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258083+0000","last_change":"2024-09-10T18:57:31.717665+0000","last_active":"2024-09-10T19:00:04.258083+0000","last_peered":"2024-09-10T19:00:04.258083+0000","last_clean":"2024-09-10T19:00:04.258083+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T19:00:04.258083+0000","last_undegraded":"2024-09-10T19:00:04.258083+0000","last_fullsized":"2024-09-10T19:00:04.258083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:31.717511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.717511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041214659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264701+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T19:00:04.264701+0000","last_peered":"2024-09-10T19:00:04.264701+0000","last_clean":"2024-09-10T19:00:04.264701+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:00:04.264701+0000","last_undegraded":"2024-09-10T19:00:04.264701+0000","last_fullsized":"2024-09-10T19:00:04.264701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":357,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.277343+0000","last_change":"2024-09-10T18:59:55.520903+0000","last_active":"2024-09-10T19:00:04.277343+0000","last_peered":"2024-09-10T19:00:04.277343+0000","last_clean":"2024-09-10T19:00:04.277343+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:00:04.277343+0000","last_undegraded":"2024-09-10T19:00:04.277343+0000","last_fullsized":"2024-09-10T19:00:04.277343+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:59:55.520655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.520655+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.622877+0000","scrub_duration":0.0073248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":466,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264496+0000","last_change":"2024-09-10T19:00:02.055337+0000","last_active":"2024-09-10T19:00:04.264496+0000","last_peered":"2024-09-10T19:00:04.264496+0000","last_clean":"2024-09-10T19:00:04.264496+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:00:04.264496+0000","last_undegraded":"2024-09-10T19:00:04.264496+0000","last_fullsized":"2024-09-10T19:00:04.264496+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:02.055169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.055169+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:18.062663+0000","scrub_duration":0.0050398609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":439,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.278822+0000","last_change":"2024-09-10T18:58:46.953749+0000","last_active":"2024-09-10T19:00:04.278822+0000","last_peered":"2024-09-10T19:00:04.278822+0000","last_clean":"2024-09-10T19:00:04.278822+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:00:04.278822+0000","last_undegraded":"2024-09-10T19:00:04.278822+0000","last_fullsized":"2024-09-10T19:00:04.278822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.953608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.953608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028191259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263992+0000","last_change":"2024-09-10T18:56:05.257380+0000","last_active":"2024-09-10T19:00:04.263992+0000","last_peered":"2024-09-10T19:00:04.263992+0000","last_clean":"2024-09-10T19:00:04.263992+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:00:04.263992+0000","last_undegraded":"2024-09-10T19:00:04.263992+0000","last_fullsized":"2024-09-10T19:00:04.263992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:05.257240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:05.257240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026860640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":433,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.280372+0000","last_change":"2024-09-10T18:59:53.990233+0000","last_active":"2024-09-10T19:00:04.280372+0000","last_peered":"2024-09-10T19:00:04.280372+0000","last_clean":"2024-09-10T19:00:04.280372+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:00:04.280372+0000","last_undegraded":"2024-09-10T19:00:04.280372+0000","last_fullsized":"2024-09-10T19:00:04.280372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.990034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.990034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:21.398651+0000","scrub_duration":0.00353419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259945+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T19:00:04.259945+0000","last_peered":"2024-09-10T19:00:04.259945+0000","last_clean":"2024-09-10T19:00:04.259945+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:00:04.259945+0000","last_undegraded":"2024-09-10T19:00:04.259945+0000","last_fullsized":"2024-09-10T19:00:04.259945+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:10.954334+0000","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":439,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270909+0000","last_change":"2024-09-10T18:58:40.428324+0000","last_active":"2024-09-10T19:00:04.270909+0000","last_peered":"2024-09-10T19:00:04.270909+0000","last_clean":"2024-09-10T19:00:04.270909+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:00:04.270909+0000","last_undegraded":"2024-09-10T19:00:04.270909+0000","last_fullsized":"2024-09-10T19:00:04.270909+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:58:40.428157+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.428157+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003853726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.277702+0000","last_change":"2024-09-10T18:57:42.602560+0000","last_active":"2024-09-10T19:00:04.277702+0000","last_peered":"2024-09-10T19:00:04.277702+0000","last_clean":"2024-09-10T19:00:04.277702+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:00:04.277702+0000","last_undegraded":"2024-09-10T19:00:04.277702+0000","last_fullsized":"2024-09-10T19:00:04.277702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:42.602448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:42.602448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027286200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":417,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262467+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T19:00:04.262467+0000","last_peered":"2024-09-10T19:00:04.262467+0000","last_clean":"2024-09-10T19:00:04.262467+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:00:04.262467+0000","last_undegraded":"2024-09-10T19:00:04.262467+0000","last_fullsized":"2024-09-10T19:00:04.262467+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265421+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T19:00:04.265421+0000","last_peered":"2024-09-10T19:00:04.265421+0000","last_clean":"2024-09-10T19:00:04.265421+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:00:04.265421+0000","last_undegraded":"2024-09-10T19:00:04.265421+0000","last_fullsized":"2024-09-10T19:00:04.265421+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268361+0000","last_change":"2024-09-10T18:59:01.663804+0000","last_active":"2024-09-10T19:00:04.268361+0000","last_peered":"2024-09-10T19:00:04.268361+0000","last_clean":"2024-09-10T19:00:04.268361+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:00:04.268361+0000","last_undegraded":"2024-09-10T19:00:04.268361+0000","last_fullsized":"2024-09-10T19:00:04.268361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.663643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.663643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.528103+0000","scrub_duration":0.0029754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":461,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265411+0000","last_change":"2024-09-10T18:58:56.293280+0000","last_active":"2024-09-10T19:00:04.265411+0000","last_peered":"2024-09-10T19:00:04.265411+0000","last_clean":"2024-09-10T19:00:04.265411+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:00:04.265411+0000","last_undegraded":"2024-09-10T19:00:04.265411+0000","last_fullsized":"2024-09-10T19:00:04.265411+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:58:56.293027+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.293027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069050090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:05.196616+0000","last_change":"2024-09-10T19:00:05.196515+0000","last_active":"2024-09-10T19:00:05.196616+0000","last_peered":"2024-09-10T19:00:05.196616+0000","last_clean":"2024-09-10T19:00:05.196616+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:00:05.196616+0000","last_undegraded":"2024-09-10T19:00:05.196616+0000","last_fullsized":"2024-09-10T19:00:05.196616+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:05.196346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.196346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.115412+0000","scrub_duration":0.0033251359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":559,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264556+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T19:00:04.264556+0000","last_peered":"2024-09-10T19:00:04.264556+0000","last_clean":"2024-09-10T19:00:04.264556+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:00:04.264556+0000","last_undegraded":"2024-09-10T19:00:04.264556+0000","last_fullsized":"2024-09-10T19:00:04.264556+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":413,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263663+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T19:00:04.263663+0000","last_peered":"2024-09-10T19:00:04.263663+0000","last_clean":"2024-09-10T19:00:04.263663+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:00:04.263663+0000","last_undegraded":"2024-09-10T19:00:04.263663+0000","last_fullsized":"2024-09-10T19:00:04.263663+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261898+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T19:00:04.261898+0000","last_peered":"2024-09-10T19:00:04.261898+0000","last_clean":"2024-09-10T19:00:04.261898+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:00:04.261898+0000","last_undegraded":"2024-09-10T19:00:04.261898+0000","last_fullsized":"2024-09-10T19:00:04.261898+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":442,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.260766+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T19:00:04.260766+0000","last_peered":"2024-09-10T19:00:04.260766+0000","last_clean":"2024-09-10T19:00:04.260766+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:00:04.260766+0000","last_undegraded":"2024-09-10T19:00:04.260766+0000","last_fullsized":"2024-09-10T19:00:04.260766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":429,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261762+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T19:00:04.261762+0000","last_peered":"2024-09-10T19:00:04.261762+0000","last_clean":"2024-09-10T19:00:04.261762+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:00:04.261762+0000","last_undegraded":"2024-09-10T19:00:04.261762+0000","last_fullsized":"2024-09-10T19:00:04.261762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:19.026489+0000","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":549,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.271295+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T19:00:04.271295+0000","last_peered":"2024-09-10T19:00:04.271295+0000","last_clean":"2024-09-10T19:00:04.271295+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:00:04.271295+0000","last_undegraded":"2024-09-10T19:00:04.271295+0000","last_fullsized":"2024-09-10T19:00:04.271295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.132106+0000","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259573+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T19:00:04.259573+0000","last_peered":"2024-09-10T19:00:04.259573+0000","last_clean":"2024-09-10T19:00:04.259573+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T19:00:04.259573+0000","last_undegraded":"2024-09-10T19:00:04.259573+0000","last_fullsized":"2024-09-10T19:00:04.259573+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263385+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T19:00:04.263385+0000","last_peered":"2024-09-10T19:00:04.263385+0000","last_clean":"2024-09-10T19:00:04.263385+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:00:04.263385+0000","last_undegraded":"2024-09-10T19:00:04.263385+0000","last_fullsized":"2024-09-10T19:00:04.263385+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:09.987912+0000","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":450,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270119+0000","last_change":"2024-09-10T18:58:38.152259+0000","last_active":"2024-09-10T19:00:04.270119+0000","last_peered":"2024-09-10T19:00:04.270119+0000","last_clean":"2024-09-10T19:00:04.270119+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:00:04.270119+0000","last_undegraded":"2024-09-10T19:00:04.270119+0000","last_fullsized":"2024-09-10T19:00:04.270119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.152079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.152079+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040090350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270540+0000","last_change":"2024-09-10T18:58:55.639930+0000","last_active":"2024-09-10T19:00:04.270540+0000","last_peered":"2024-09-10T19:00:04.270540+0000","last_clean":"2024-09-10T19:00:04.270540+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:00:04.270540+0000","last_undegraded":"2024-09-10T19:00:04.270540+0000","last_fullsized":"2024-09-10T19:00:04.270540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.639777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.639777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.180410+0000","scrub_duration":0.003186637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":411,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267538+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T19:00:04.267538+0000","last_peered":"2024-09-10T19:00:04.267538+0000","last_clean":"2024-09-10T19:00:04.267538+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:00:04.267538+0000","last_undegraded":"2024-09-10T19:00:04.267538+0000","last_fullsized":"2024-09-10T19:00:04.267538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261901+0000","last_change":"2024-09-10T19:00:04.060523+0000","last_active":"2024-09-10T19:00:04.261901+0000","last_peered":"2024-09-10T19:00:04.261901+0000","last_clean":"2024-09-10T19:00:04.261901+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:00:04.261901+0000","last_undegraded":"2024-09-10T19:00:04.261901+0000","last_fullsized":"2024-09-10T19:00:04.261901+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.060371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.060371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.162315+0000","scrub_duration":0.0031455099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257487+0000","last_change":"2024-09-10T18:58:46.501591+0000","last_active":"2024-09-10T19:00:04.257487+0000","last_peered":"2024-09-10T19:00:04.257487+0000","last_clean":"2024-09-10T19:00:04.257487+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:00:04.257487+0000","last_undegraded":"2024-09-10T19:00:04.257487+0000","last_fullsized":"2024-09-10T19:00:04.257487+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.501508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036484099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":463,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258935+0000","last_change":"2024-09-10T18:57:47.839278+0000","last_active":"2024-09-10T19:00:04.258935+0000","last_peered":"2024-09-10T19:00:04.258935+0000","last_clean":"2024-09-10T19:00:04.258935+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T19:00:04.258935+0000","last_undegraded":"2024-09-10T19:00:04.258935+0000","last_fullsized":"2024-09-10T19:00:04.258935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:57:47.839124+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.839124+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045027610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":438,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.286882+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T19:00:04.286882+0000","last_peered":"2024-09-10T19:00:04.286882+0000","last_clean":"2024-09-10T19:00:04.286882+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:00:04.286882+0000","last_undegraded":"2024-09-10T19:00:04.286882+0000","last_fullsized":"2024-09-10T19:00:04.286882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:17.284809+0000","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":426,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258325+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T19:00:04.258325+0000","last_peered":"2024-09-10T19:00:04.258325+0000","last_clean":"2024-09-10T19:00:04.258325+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:00:04.258325+0000","last_undegraded":"2024-09-10T19:00:04.258325+0000","last_fullsized":"2024-09-10T19:00:04.258325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:12.811604+0000","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":428,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265887+0000","last_change":"2024-09-10T18:59:55.150083+0000","last_active":"2024-09-10T19:00:04.265887+0000","last_peered":"2024-09-10T19:00:04.265887+0000","last_clean":"2024-09-10T19:00:04.265887+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:00:04.265887+0000","last_undegraded":"2024-09-10T19:00:04.265887+0000","last_fullsized":"2024-09-10T19:00:04.265887+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:55.150009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.150009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.988153+0000","scrub_duration":0.0024462989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":345,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.258050+0000","last_change":"2024-09-10T18:58:49.517454+0000","last_active":"2024-09-10T19:00:04.258050+0000","last_peered":"2024-09-10T19:00:04.258050+0000","last_clean":"2024-09-10T19:00:04.258050+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T19:00:04.258050+0000","last_undegraded":"2024-09-10T19:00:04.258050+0000","last_fullsized":"2024-09-10T19:00:04.258050+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.517273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.517273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039870039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":438,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264520+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T19:00:04.264520+0000","last_peered":"2024-09-10T19:00:04.264520+0000","last_clean":"2024-09-10T19:00:04.264520+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:00:04.264520+0000","last_undegraded":"2024-09-10T19:00:04.264520+0000","last_fullsized":"2024-09-10T19:00:04.264520+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266421+0000","last_change":"2024-09-10T18:58:53.461054+0000","last_active":"2024-09-10T19:00:04.266421+0000","last_peered":"2024-09-10T19:00:04.266421+0000","last_clean":"2024-09-10T19:00:04.266421+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:00:04.266421+0000","last_undegraded":"2024-09-10T19:00:04.266421+0000","last_fullsized":"2024-09-10T19:00:04.266421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.460812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.460812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:07.134909+0000","scrub_duration":0.0042777029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":420,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262988+0000","last_change":"2024-09-10T18:59:53.179273+0000","last_active":"2024-09-10T19:00:04.262988+0000","last_peered":"2024-09-10T19:00:04.262988+0000","last_clean":"2024-09-10T19:00:04.262988+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:00:04.262988+0000","last_undegraded":"2024-09-10T19:00:04.262988+0000","last_fullsized":"2024-09-10T19:00:04.262988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.179202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.179202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.193560+0000","scrub_duration":0.0015188129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263881+0000","last_change":"2024-09-10T18:57:36.386334+0000","last_active":"2024-09-10T19:00:04.263881+0000","last_peered":"2024-09-10T19:00:04.263881+0000","last_clean":"2024-09-10T19:00:04.263881+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:00:04.263881+0000","last_undegraded":"2024-09-10T19:00:04.263881+0000","last_fullsized":"2024-09-10T19:00:04.263881+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:36.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T18:57:36.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003234233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.267933+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T19:00:04.267933+0000","last_peered":"2024-09-10T19:00:04.267933+0000","last_clean":"2024-09-10T19:00:04.267933+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:00:04.267933+0000","last_undegraded":"2024-09-10T19:00:04.267933+0000","last_fullsized":"2024-09-10T19:00:04.267933+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.299236+0000","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":447,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266248+0000","last_change":"2024-09-10T18:59:04.505215+0000","last_active":"2024-09-10T19:00:04.266248+0000","last_peered":"2024-09-10T19:00:04.266248+0000","last_clean":"2024-09-10T19:00:04.266248+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:00:04.266248+0000","last_undegraded":"2024-09-10T19:00:04.266248+0000","last_fullsized":"2024-09-10T19:00:04.266248+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:59:04.505046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.505046+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.388122+0000","scrub_duration":0.0045725100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":427,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268712+0000","last_change":"2024-09-10T18:57:56.351163+0000","last_active":"2024-09-10T19:00:04.268712+0000","last_peered":"2024-09-10T19:00:04.268712+0000","last_clean":"2024-09-10T19:00:04.268712+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:00:04.268712+0000","last_undegraded":"2024-09-10T19:00:04.268712+0000","last_fullsized":"2024-09-10T19:00:04.268712+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:56.351005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:56.351005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035909499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":417,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266067+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T19:00:04.266067+0000","last_peered":"2024-09-10T19:00:04.266067+0000","last_clean":"2024-09-10T19:00:04.266067+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:00:04.266067+0000","last_undegraded":"2024-09-10T19:00:04.266067+0000","last_fullsized":"2024-09-10T19:00:04.266067+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.764329+0000","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264888+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T19:00:04.264888+0000","last_peered":"2024-09-10T19:00:04.264888+0000","last_clean":"2024-09-10T19:00:04.264888+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:00:04.264888+0000","last_undegraded":"2024-09-10T19:00:04.264888+0000","last_fullsized":"2024-09-10T19:00:04.264888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":334,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262930+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T19:00:04.262930+0000","last_peered":"2024-09-10T19:00:04.262930+0000","last_clean":"2024-09-10T19:00:04.262930+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:00:04.262930+0000","last_undegraded":"2024-09-10T19:00:04.262930+0000","last_fullsized":"2024-09-10T19:00:04.262930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":425,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264080+0000","last_change":"2024-09-10T18:58:58.248456+0000","last_active":"2024-09-10T19:00:04.264080+0000","last_peered":"2024-09-10T19:00:04.264080+0000","last_clean":"2024-09-10T19:00:04.264080+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:00:04.264080+0000","last_undegraded":"2024-09-10T19:00:04.264080+0000","last_fullsized":"2024-09-10T19:00:04.264080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.067159+0000","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":486,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268651+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T19:00:04.268651+0000","last_peered":"2024-09-10T19:00:04.268651+0000","last_clean":"2024-09-10T19:00:04.268651+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:00:04.268651+0000","last_undegraded":"2024-09-10T19:00:04.268651+0000","last_fullsized":"2024-09-10T19:00:04.268651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:11.312723+0000","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266901+0000","last_change":"2024-09-10T18:58:31.395234+0000","last_active":"2024-09-10T19:00:04.266901+0000","last_peered":"2024-09-10T19:00:04.266901+0000","last_clean":"2024-09-10T19:00:04.266901+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:00:04.266901+0000","last_undegraded":"2024-09-10T19:00:04.266901+0000","last_fullsized":"2024-09-10T19:00:04.266901+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.395072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.395072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032990430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263873+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T19:00:04.263873+0000","last_peered":"2024-09-10T19:00:04.263873+0000","last_clean":"2024-09-10T19:00:04.263873+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:00:04.263873+0000","last_undegraded":"2024-09-10T19:00:04.263873+0000","last_fullsized":"2024-09-10T19:00:04.263873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265983+0000","last_change":"2024-09-10T18:59:58.485524+0000","last_active":"2024-09-10T19:00:04.265983+0000","last_peered":"2024-09-10T19:00:04.265983+0000","last_clean":"2024-09-10T19:00:04.265983+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:00:04.265983+0000","last_undegraded":"2024-09-10T19:00:04.265983+0000","last_fullsized":"2024-09-10T19:00:04.265983+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.485240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.485240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.282886+0000","scrub_duration":0.0043729700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":422,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.283030+0000","last_change":"2024-09-10T18:57:54.822480+0000","last_active":"2024-09-10T19:00:04.283030+0000","last_peered":"2024-09-10T19:00:04.283030+0000","last_clean":"2024-09-10T19:00:04.283030+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:00:04.283030+0000","last_undegraded":"2024-09-10T19:00:04.283030+0000","last_fullsized":"2024-09-10T19:00:04.283030+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:54.822329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.822329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037549549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":444,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.263321+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T19:00:04.263321+0000","last_peered":"2024-09-10T19:00:04.263321+0000","last_clean":"2024-09-10T19:00:04.263321+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:00:04.263321+0000","last_undegraded":"2024-09-10T19:00:04.263321+0000","last_fullsized":"2024-09-10T19:00:04.263321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.266819+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T19:00:04.266819+0000","last_peered":"2024-09-10T19:00:04.266819+0000","last_clean":"2024-09-10T19:00:04.266819+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:00:04.266819+0000","last_undegraded":"2024-09-10T19:00:04.266819+0000","last_fullsized":"2024-09-10T19:00:04.266819+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":579,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268641+0000","last_change":"2024-09-10T18:58:38.434610+0000","last_active":"2024-09-10T19:00:04.268641+0000","last_peered":"2024-09-10T19:00:04.268641+0000","last_clean":"2024-09-10T19:00:04.268641+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:00:04.268641+0000","last_undegraded":"2024-09-10T19:00:04.268641+0000","last_fullsized":"2024-09-10T19:00:04.268641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:58:38.434348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.434348+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046950070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268562+0000","last_change":"2024-09-10T18:59:00.644451+0000","last_active":"2024-09-10T19:00:04.268562+0000","last_peered":"2024-09-10T19:00:04.268562+0000","last_clean":"2024-09-10T19:00:04.268562+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:00:04.268562+0000","last_undegraded":"2024-09-10T19:00:04.268562+0000","last_fullsized":"2024-09-10T19:00:04.268562+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.644299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:13.665454+0000","scrub_duration":0.00344357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":424,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.259439+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T19:00:04.259439+0000","last_peered":"2024-09-10T19:00:04.259439+0000","last_clean":"2024-09-10T19:00:04.259439+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:00:04.259439+0000","last_undegraded":"2024-09-10T19:00:04.259439+0000","last_fullsized":"2024-09-10T19:00:04.259439+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":436,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.274556+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T19:00:04.274556+0000","last_peered":"2024-09-10T19:00:04.274556+0000","last_clean":"2024-09-10T19:00:04.274556+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:00:04.274556+0000","last_undegraded":"2024-09-10T19:00:04.274556+0000","last_fullsized":"2024-09-10T19:00:04.274556+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.205170+0000","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":440,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.261368+0000","last_change":"2024-09-10T19:00:02.052175+0000","last_active":"2024-09-10T19:00:04.261368+0000","last_peered":"2024-09-10T19:00:04.261368+0000","last_clean":"2024-09-10T19:00:04.261368+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:00:04.261368+0000","last_undegraded":"2024-09-10T19:00:04.261368+0000","last_fullsized":"2024-09-10T19:00:04.261368+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:00:02.051995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.051995+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.499770+0000","scrub_duration":0.0042242440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":445,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.280609+0000","last_change":"2024-09-10T19:00:00.097374+0000","last_active":"2024-09-10T19:00:04.280609+0000","last_peered":"2024-09-10T19:00:04.280609+0000","last_clean":"2024-09-10T19:00:04.280609+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:00:04.280609+0000","last_undegraded":"2024-09-10T19:00:04.280609+0000","last_fullsized":"2024-09-10T19:00:04.280609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.097226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.097226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.731885+0000","scrub_duration":0.0030874499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":341,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.270506+0000","last_change":"2024-09-10T18:59:54.415459+0000","last_active":"2024-09-10T19:00:04.270506+0000","last_peered":"2024-09-10T19:00:04.270506+0000","last_clean":"2024-09-10T19:00:04.270506+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:00:04.270506+0000","last_undegraded":"2024-09-10T19:00:04.270506+0000","last_fullsized":"2024-09-10T19:00:04.270506+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:54.415297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:54.415297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:55.651756+0000","scrub_duration":0.0031427790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":415,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.262294+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T19:00:04.262294+0000","last_peered":"2024-09-10T19:00:04.262294+0000","last_clean":"2024-09-10T19:00:04.262294+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:00:04.262294+0000","last_undegraded":"2024-09-10T19:00:04.262294+0000","last_fullsized":"2024-09-10T19:00:04.262294+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":421,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.265342+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T19:00:04.265342+0000","last_peered":"2024-09-10T19:00:04.265342+0000","last_clean":"2024-09-10T19:00:04.265342+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:00:04.265342+0000","last_undegraded":"2024-09-10T19:00:04.265342+0000","last_fullsized":"2024-09-10T19:00:04.265342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":453,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.257743+0000","last_change":"2024-09-10T18:57:35.812113+0000","last_active":"2024-09-10T19:00:04.257743+0000","last_peered":"2024-09-10T19:00:04.257743+0000","last_clean":"2024-09-10T19:00:04.257743+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:00:04.257743+0000","last_undegraded":"2024-09-10T19:00:04.257743+0000","last_fullsized":"2024-09-10T19:00:04.257743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:57:35.811973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:35.811973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041510690000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":417,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.264843+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T19:00:04.264843+0000","last_peered":"2024-09-10T19:00:04.264843+0000","last_clean":"2024-09-10T19:00:04.264843+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:00:04.264843+0000","last_undegraded":"2024-09-10T19:00:04.264843+0000","last_fullsized":"2024-09-10T19:00:04.264843+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":344,"reported_epoch":441,"state":"active+clean","last_fresh":"2024-09-10T19:00:04.268908+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T19:00:04.268908+0000","last_peered":"2024-09-10T19:00:04.268908+0000","last_clean":"2024-09-10T19:00:04.268908+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:00:04.268908+0000","last_undegraded":"2024-09-10T19:00:04.268908+0000","last_fullsized":"2024-09-10T19:00:04.268908+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.897723+0000","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":215,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195783,"num_objects":159,"num_object_clones":2,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":593,"num_write_kb":12682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26468352,"data_stored":24587349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1861,"ondisk_log_size":1861,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":622,"num_read_kb":617,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972493,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316896,"kb_used_data":15376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93428256,"statfs":{"total":95995035648,"available":95670534144,"internally_reserved":0,"allocated":15745024,"data_stored":12689654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.3839999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92100000000000004,"15min":0.92400000000000004},"min":{"1min":0.50600000000000001,"5min":0.372,"15min":0.34699999999999998},"max":{"1min":1.2529999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.302}]},{"osd":1,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.208,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":1.004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.421,"5min":1.742,"15min":1.8280000000000001},"last":1.0620000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.014,"15min":1.0289999999999999},"min":{"1min":0.45000000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6739999999999999,"5min":2.9199999999999999,"15min":3.4990000000000001},"last":1.218},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0009999999999999,"15min":0.98999999999999999},"min":{"1min":0.64500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.97,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.69799999999999995}]},{"osd":3,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0629999999999999,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.9390000000000001,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":0.97599999999999998},"min":{"1min":0.56200000000000006,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.833,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.97499999999999998}]},{"osd":4,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.014,"15min":1.0149999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3700000000000001,"5min":1.4279999999999999,"15min":2.1989999999999998},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.046,"5min":1.127,"15min":1.0609999999999999},"min":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0310000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.012}]},{"osd":5,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.998,"15min":0.99199999999999999},"min":{"1min":0.56899999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7669999999999999,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1379999999999999,"15min":1.1639999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":2.1419999999999999},"last":1.131}]},{"osd":6,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.179,"15min":1.2150000000000001},"min":{"1min":0.60899999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.468,"5min":1.591,"15min":2.2749999999999999},"last":1.516},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2150000000000001,"15min":1.1950000000000001},"min":{"1min":0.70599999999999996,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":2.1560000000000001,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.161}]}]},{"osd":6,"up_from":52,"seq":223338299543,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312996,"kb_used_data":7308,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432156,"statfs":{"total":95995035648,"available":95674527744,"internally_reserved":0,"allocated":7483392,"data_stored":4502326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.88700000000000001,"15min":0.88400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36399999999999999},"max":{"1min":1.181,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.95899999999999996},"min":{"1min":0.46300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.276,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":0.99399999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.452,"5min":2.2410000000000001,"15min":18.494},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.88100000000000001,"15min":0.90500000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1679999999999999,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":0.90900000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94199999999999995,"15min":0.96499999999999997},"min":{"1min":0.47499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.288,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":1.034},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5309999999999999,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.157}]},{"osd":3,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.05,"15min":0.997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.827,"5min":2.278,"15min":2.278},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.002,"15min":0.98499999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.641,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":1.135}]},{"osd":4,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.028,"15min":0.94499999999999995},"min":{"1min":0.66500000000000004,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.782,"5min":6.008,"15min":6.008},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1319999999999999,"15min":1.0569999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42999999999999999},"max":{"1min":1.6100000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1220000000000001,"15min":1.0369999999999999},"min":{"1min":0.71099999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6739999999999999,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.111},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1890000000000001,"15min":1.125},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.708,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.1830000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.115,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.5840000000000001,"5min":2.069,"15min":2.069},"last":1.093},{"interface":"front","average":{"1min":1.292,"5min":1.2929999999999999,"15min":1.196},"min":{"1min":0.90600000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8680000000000001,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.218}]}]},{"osd":5,"up_from":43,"seq":184683593889,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312420,"kb_used_data":10900,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432732,"statfs":{"total":95995035648,"available":95675117568,"internally_reserved":0,"allocated":11161600,"data_stored":8097672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1000000000000001,"15min":0.94699999999999995},"min":{"1min":0.52900000000000003,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.347,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.012,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3160000000000001,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":0.96799999999999997}]},{"osd":1,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.6739999999999999,"15min":1.1399999999999999},"min":{"1min":0.67900000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0980000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.431,"15min":1.0760000000000001},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.135,"5min":31.847000000000001,"15min":31.847000000000001},"last":0.89600000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.8959999999999999,"15min":1.238},"min":{"1min":0.59299999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.22,"5min":61.347000000000001,"15min":61.347000000000001},"last":1.113},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.137,"15min":0.996},"min":{"1min":0.66100000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.23,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":0.93600000000000005}]},{"osd":3,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1339999999999999,"15min":1},"min":{"1min":0.58799999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2769999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.792,"15min":1.2410000000000001},"min":{"1min":0.72699999999999998,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":1.2270000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":0.76800000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.167,"15min":0.94299999999999995},"min":{"1min":0.52800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2529999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.008,"5min":1.224,"15min":1.0109999999999999},"min":{"1min":0.67200000000000004,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1}]},{"osd":6,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.395,"15min":1.1519999999999999},"min":{"1min":0.67000000000000004,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":1.3400000000000001,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.252},{"interface":"front","average":{"1min":1.105,"5min":1.4299999999999999,"15min":1.1879999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.47199999999999998},"max":{"1min":1.5760000000000001,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.171}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.532,"15min":1.2130000000000001},"min":{"1min":0.59699999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.3899999999999999,"5min":15.613,"15min":15.613},"last":1.03},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.544,"15min":1.2749999999999999},"min":{"1min":0.77300000000000002,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":1.538,"5min":16.965,"15min":16.965},"last":1.1950000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855533,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312408,"kb_used_data":6720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432744,"statfs":{"total":95995035648,"available":95675129856,"internally_reserved":0,"allocated":6881280,"data_stored":3904316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.403,"5min":1.546,"15min":1.546},"last":0.371},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.91600000000000004,"15min":0.88600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.403,"5min":1.667,"15min":1.7070000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.133,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.89700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2869999999999999,"5min":1.575,"15min":1.575},"last":0.55200000000000005}]},{"osd":2,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.93999999999999995,"15min":0.92100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5209999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95099999999999996,"15min":0.92400000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.96199999999999997,"15min":0.93600000000000005},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.46300000000000002},"max":{"1min":4.2670000000000003,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.95399999999999996,"15min":0.95099999999999996},"min":{"1min":0.49099999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.3740000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.70899999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:59:12 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.94899999999999995,"15min":0.90600000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.3899999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.05,"15min":0.97599999999999998},"min":{"1min":0.57399999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":0.52600000000000002}]},{"osd":6,"last update":"Tue Sep 10 18:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.038,"15min":0.98499999999999999},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.127,"15min":1.1020000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.73699999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.80100000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5880000000000001,"5min":1.629,"15min":2.008},"last":0.753},{"interface":"front","average":{"1min":1.325,"5min":1.1839999999999999,"15min":1.167},"min":{"1min":1,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.65900000000000003}]}]},{"osd":3,"up_from":29,"seq":124554051768,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304212,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93440940,"statfs":{"total":95995035648,"available":95683522560,"internally_reserved":0,"allocated":7016448,"data_stored":3904300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.24,"15min":0.82499999999999996},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.86799999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.63,"5min":1.2589999999999999,"15min":0.88900000000000001},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":1.085,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.61799999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.3919999999999999,"15min":0.91200000000000003},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.26500000000000001},"max":{"1min":0.88200000000000001,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.0740000000000001,"15min":0.871},"min":{"1min":0.52900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0009999999999999,"5min":15.189,"15min":15.189},"last":0.76400000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.381,"15min":0.99099999999999999},"min":{"1min":0.432,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.153,"5min":26.558,"15min":26.558},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.2809999999999999,"15min":0.93100000000000005},"min":{"1min":0.434,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.8540000000000001,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.68000000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.3600000000000001,"15min":1.137},"min":{"1min":0.73099999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5,"5min":10.359999999999999,"15min":10.359999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.464,"15min":1.224},"min":{"1min":0.81000000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.633,"5min":10.391,"15min":10.391},"last":0.995}]},{"osd":5,"last update":"Tue Sep 10 18:59:12 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.46,"15min":1.2130000000000001},"min":{"1min":0.76500000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.498,"5min":17.535,"15min":17.535},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.995,"5min":1.119,"15min":1.034},"min":{"1min":0.66800000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.6819999999999999,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.90800000000000003}]},{"osd":6,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.157,"5min":1.3,"15min":1.2030000000000001},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6919999999999999,"5min":10.51,"15min":10.51},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.1399999999999999,"15min":1.0900000000000001},"min":{"1min":0.63,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5820000000000001,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.1140000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:56 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.179},"min":{"1min":0.60199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":4.0229999999999997,"5min":5.3109999999999999,"15min":18.312999999999999},"last":0.874},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3919999999999999,"15min":1.3140000000000001},"min":{"1min":0.86399999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":4.5350000000000001,"5min":9.3840000000000003,"15min":10.611000000000001},"last":1.141}]}]},{"osd":2,"up_from":19,"seq":81604378817,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320500,"kb_used_data":14804,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93424652,"statfs":{"total":95995035648,"available":95666843648,"internally_reserved":0,"allocated":15159296,"data_stored":12098700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.84099999999999997,"15min":0.72099999999999997},"min":{"1min":0.36799999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.2749999999999999,"5min":9.516,"15min":9.516},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.84999999999999998,"15min":0.77900000000000003},"min":{"1min":0.36099999999999999,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.0760000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.66500000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.77700000000000002,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.0720000000000001,"5min":5.516,"15min":5.516},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85899999999999999,"15min":0.81200000000000006},"min":{"1min":0.54000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0820000000000001,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.97099999999999997}]},{"osd":3,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.85699999999999998},"min":{"1min":0.503,"5min":0.36599999999999999,"15min":0.34799999999999998},"max":{"1min":1.3859999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.91700000000000004,"15min":0.84299999999999997},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":1.194,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.80300000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2090000000000001,"15min":1.0629999999999999},"min":{"1min":0.49299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":3.851,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.675,"5min":1.2869999999999999,"15min":1.149},"min":{"1min":0.75900000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":10.881,"5min":10.881,"15min":10.881},"last":1.3160000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:59:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.069,"15min":1.0900000000000001},"min":{"1min":0.60599999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.2589999999999999,"5min":5.1959999999999997,"15min":5.1959999999999997},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.246,"15min":1.103},"min":{"1min":0.63900000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.4980000000000002,"5min":21.654,"15min":21.654},"last":0.93799999999999994}]},{"osd":6,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1719999999999999,"15min":1.169},"min":{"1min":0.64400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.5609999999999999,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2370000000000001,"15min":1.1299999999999999},"min":{"1min":0.63,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":15.971,"15min":15.971},"last":0.90500000000000003}]},{"osd":7,"last update":"Tue Sep 10 18:59:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"min":{"1min":0.59699999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.905,"5min":15.025,"15min":15.025},"last":1.165},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.331,"15min":1.1990000000000001},"min":{"1min":0.56499999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":6.851,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.0980000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312196,"kb_used_data":10676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432956,"statfs":{"total":95995035648,"available":95675346944,"internally_reserved":0,"allocated":10932224,"data_stored":7912421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.0649999999999999,"15min":0.79300000000000004},"min":{"1min":0.33900000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.079,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.87},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.1839999999999999,"15min":0.872},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":1.177,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.83199999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.181,"15min":0.85699999999999998},"min":{"1min":0.46000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.724,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.746,"5min":1.371,"15min":0.98399999999999999},"min":{"1min":0.39400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.03,"5min":32.363,"15min":32.363},"last":1.0389999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.27,"15min":0.98199999999999998},"min":{"1min":0.496,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.7929999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":1.117},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.6000000000000001,"15min":1.0269999999999999},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.26600000000000001},"max":{"1min":1.871,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.76800000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:59:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2470000000000001,"15min":1.0940000000000001},"min":{"1min":0.70599999999999996,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3220000000000001,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.3460000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.7589999999999999,"15min":1.2090000000000001},"min":{"1min":0.72199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.544,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.95499999999999996}]},{"osd":5,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.284,"15min":1.1020000000000001},"min":{"1min":0.61899999999999999,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":1.4790000000000001,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.3180000000000001,"15min":1.1970000000000001},"min":{"1min":0.53200000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.7949999999999999,"5min":12.84,"15min":12.84},"last":1.304}]},{"osd":6,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.5409999999999999,"15min":1.254},"min":{"1min":0.67900000000000005,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":3.121,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.246},{"interface":"front","average":{"1min":1.123,"5min":1.5369999999999999,"15min":1.23},"min":{"1min":0.79400000000000004,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.9359999999999999,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.1639999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.5229999999999999,"15min":1.3180000000000001},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.0880000000000001,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.607,"15min":1.304},"min":{"1min":0.66100000000000003,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":1.839,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.2250000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308852,"kb_used_data":7332,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436300,"statfs":{"total":95995035648,"available":95678771200,"internally_reserved":0,"allocated":7507968,"data_stored":4493415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.591,"15min":0.96899999999999997},"min":{"1min":0.32400000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.877,"5min":80.710999999999999,"15min":80.710999999999999},"last":6.758},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.7270000000000001,"15min":1.077},"min":{"1min":0.48699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.1779999999999999,"5min":86.009,"15min":86.009},"last":3.3109999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.6639999999999999,"15min":1.0209999999999999},"min":{"1min":0.60699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.593,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.2190000000000001,"15min":0.90600000000000003},"min":{"1min":0.57299999999999995,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.333,"5min":24.614000000000001,"15min":24.614000000000001},"last":3.2090000000000001}]},{"osd":3,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1459999999999999,"15min":0.89300000000000002},"min":{"1min":0.48799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.379,"5min":21.620999999999999,"15min":21.620999999999999},"last":4.4749999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.5049999999999999,"15min":1.0349999999999999},"min":{"1min":0.57999999999999996,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.4019999999999999,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.45700000000000002}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1619999999999999,"15min":1.0429999999999999},"min":{"1min":0.71199999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.6839999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":2.5190000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1739999999999999,"15min":1.099},"min":{"1min":0.70499999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.6439999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.74399999999999999}]},{"osd":5,"last update":"Tue Sep 10 18:59:04 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.2629999999999999,"15min":1.111},"min":{"1min":0.749,"5min":0.442,"15min":0.442},"max":{"1min":1.288,"5min":10.048,"15min":10.048},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.087,"15min":1.097},"min":{"1min":0.63,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.337,"5min":3.8959999999999999,"15min":3.8959999999999999},"last":0.82499999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:00:04 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1619999999999999,"15min":1.135},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56899999999999995},"max":{"1min":1.792,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2669999999999999,"15min":1.1970000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.804,"5min":11.894,"15min":11.894},"last":0.48999999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.149,"15min":1.167},"min":{"1min":0.80600000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5549999999999999,"5min":2.266,"15min":6.2949999999999999},"last":2.468},{"interface":"front","average":{"1min":1.25,"5min":1.3660000000000001,"15min":1.29},"min":{"1min":0.78400000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0129999999999999,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":0.85599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8194945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4194961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8441856,"data_stored":8194896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:00:10.026 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:00:10.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:00:10.027 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:00:10.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:10.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:10.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:10.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:10.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:10 smithi165 bash[22591]: cluster 2024-09-10T19:00:09.136105+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:00:10.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:10 smithi165 bash[22591]: cluster 2024-09-10T19:00:09.145224+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:00:10.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:10 smithi165 bash[22591]: cluster 2024-09-10T19:00:09.568300+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:00:10.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:10 smithi165 bash[22591]: cluster 2024-09-10T19:00:09.571831+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:00:10.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:10.408 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:10.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:10.547 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:10.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:10.660 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[18629]: cluster 2024-09-10T19:00:09.136105+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:00:10.660 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[18629]: cluster 2024-09-10T19:00:09.145224+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:00:10.660 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[18629]: cluster 2024-09-10T19:00:09.568300+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:00:10.660 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[18629]: cluster 2024-09-10T19:00:09.571831+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:00:10.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[32381]: cluster 2024-09-10T19:00:09.136105+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:00:10.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[32381]: cluster 2024-09-10T19:00:09.145224+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:00:10.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[32381]: cluster 2024-09-10T19:00:09.568300+0000 osd.1 (osd.1) 253 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:00:10.661 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:10 smithi083 bash[32381]: cluster 2024-09-10T19:00:09.571831+0000 osd.1 (osd.1) 254 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:00:10.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:10.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:10.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:10.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:10.954 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:10.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:11.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:11.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:11.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:11.231 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:11.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.093401+0000 mgr.y (mgr.24526) 440 : cluster [DBG] pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.222499+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.c scrub starts 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.225315+0000 osd.2 (osd.2) 178 : cluster [DBG] 2.c scrub ok 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.263053+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.266236+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.345306+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:00:11.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.348699+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:00:11.369 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:11.473 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:11.506 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:11.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:11.622 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:11.643 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:11.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:11.782 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.093401+0000 mgr.y (mgr.24526) 440 : cluster [DBG] pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.222499+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.c scrub starts 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.225315+0000 osd.2 (osd.2) 178 : cluster [DBG] 2.c scrub ok 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.263053+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.266236+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.345306+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.348699+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.093401+0000 mgr.y (mgr.24526) 440 : cluster [DBG] pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.222499+0000 osd.2 (osd.2) 177 : cluster [DBG] 2.c scrub starts 2024-09-10T19:00:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.225315+0000 osd.2 (osd.2) 178 : cluster [DBG] 2.c scrub ok 2024-09-10T19:00:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.263053+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:00:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.266236+0000 osd.4 (osd.4) 271 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:00:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.345306+0000 osd.6 (osd.6) 195 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:00:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:11 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.348699+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:00:11.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:11.921 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:11.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:12.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:12.060 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:12.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:12.192 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:12.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:12.326 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:12.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:12.503 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.934577+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:10.939862+0000 osd.3 (osd.3) 259 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:11.382788+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:11.386270+0000 osd.6 (osd.6) 198 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.001007+0000 osd.7 (osd.7) 252 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.005480+0000 osd.7 (osd.7) 253 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.095635+0000 mgr.y (mgr.24526) 441 : cluster [DBG] pgmap v658: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.125697+0000 osd.0 (osd.0) 232 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.128970+0000 osd.0 (osd.0) 233 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.934577+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:00:12.504 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:10.939862+0000 osd.3 (osd.3) 259 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:11.382788+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:11.386270+0000 osd.6 (osd.6) 198 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.001007+0000 osd.7 (osd.7) 252 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.005480+0000 osd.7 (osd.7) 253 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.095635+0000 mgr.y (mgr.24526) 441 : cluster [DBG] pgmap v658: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.125697+0000 osd.0 (osd.0) 232 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:00:12.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.128970+0000 osd.0 (osd.0) 233 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:00:12.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:12.599 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:12.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.934577+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:10.939862+0000 osd.3 (osd.3) 259 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:11.382788+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:11.386270+0000 osd.6 (osd.6) 198 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.001007+0000 osd.7 (osd.7) 252 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.005480+0000 osd.7 (osd.7) 253 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.095635+0000 mgr.y (mgr.24526) 441 : cluster [DBG] pgmap v658: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.125697+0000 osd.0 (osd.0) 232 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:00:12.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:12 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.128970+0000 osd.0 (osd.0) 233 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:00:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:12.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 -- 172.21.15.83:0/2880891909 >> v1:172.21.15.83:6789/0 conn(0x7fac601042a0 legacy=0x7fac60106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 -- 172.21.15.83:0/2880891909 shutdown_connections 2024-09-10T19:00:12.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 -- 172.21.15.83:0/2880891909 >> 172.21.15.83:0/2880891909 conn(0x7fac60075290 msgr2=0x7fac600f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:12.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 -- 172.21.15.83:0/2880891909 shutdown_connections 2024-09-10T19:00:12.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 -- 172.21.15.83:0/2880891909 wait complete. 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 librados: starting msgr at 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.615+0000 7fac67182640 1 librados: starting objecter 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 Processor -- start 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- start start 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 librados: setting wanted keys 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 librados: calling monclient init 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac6011e310 con 0x7fac601014e0 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac6011f510 con 0x7fac601042a0 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac60120710 con 0x7fac60107e30 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac64f1e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fac601014e0 0x7fac60065080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58592/0 (socket says 172.21.15.83:58592) 2024-09-10T19:00:12.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac64f1e640 1 -- 172.21.15.83:0/1760447166 learned_addr learned my addr 172.21.15.83:0/1760447166 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter ms_handle_connect 0x7fac601014e0 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter ms_handle_connect 0x7fac60107e30 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter ms_handle_connect 0x7fac601042a0 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789588268 0 0) 0x7fac60120710 con 0x7fac60107e30 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac500034e0 con 0x7fac60107e30 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922975153 0 0) 0x7fac6011e310 con 0x7fac601014e0 2024-09-10T19:00:12.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac60120710 con 0x7fac601014e0 2024-09-10T19:00:12.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855312251 0 0) 0x7fac6011f510 con 0x7fac601042a0 2024-09-10T19:00:12.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6011e310 con 0x7fac601042a0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2019403884 0 0) 0x7fac500034e0 con 0x7fac60107e30 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac6011f510 con 0x7fac60107e30 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac58001c40 con 0x7fac60107e30 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156321152 0 0) 0x7fac60120710 con 0x7fac601014e0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac500034e0 con 0x7fac601014e0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac54001900 con 0x7fac601014e0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2632038080 0 0) 0x7fac6011e310 con 0x7fac601042a0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac60120710 con 0x7fac601042a0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac4c0015d0 con 0x7fac601042a0 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1529539945 0 0) 0x7fac6011f510 con 0x7fac60107e30 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 >> v1:172.21.15.165:6789/0 conn(0x7fac601042a0 legacy=0x7fac60065790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 >> v1:172.21.15.83:6789/0 conn(0x7fac601014e0 legacy=0x7fac60065080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac60121910 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6011e500 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 10 client.49394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6011e9d0 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 librados: init done 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fac580025c0 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac58003470 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac44005010 con 0x7fac60107e30 2024-09-10T19:00:12.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.619+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fac58019890 con 0x7fac60107e30 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(446..446 src has 69..446) v4 ==== 8060+0+0 (unknown 1686100548 0 0) 0x7fac58087f90 con 0x7fac60107e30 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 10 client.49394.objecter ms_dispatch 0x7fac60000b90 osd_map(446..446 src has 69..446) v4 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 3 client.49394.objecter handle_osd_map got epochs [446,446] > 0 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 3 client.49394.objecter handle_osd_map decoding full epoch 446 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 20 client.49394.objecter dump_active .. 0 homeless 2024-09-10T19:00:12.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.623+0000 7fac5d7fa640 10 client.49394.objecter ms_handle_connect 0x7fac500717a0 2024-09-10T19:00:12.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.627+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fac58054510 con 0x7fac60107e30 2024-09-10T19:00:12.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 -- 172.21.15.83:0/3947586213 >> v1:172.21.15.83:6790/0 conn(0x7fd7dc104270 legacy=0x7fd7dc106650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 -- 172.21.15.83:0/3947586213 shutdown_connections 2024-09-10T19:00:12.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 -- 172.21.15.83:0/3947586213 >> 172.21.15.83:0/3947586213 conn(0x7fd7dc074af0 msgr2=0x7fd7dc0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:12.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 -- 172.21.15.83:0/3947586213 shutdown_connections 2024-09-10T19:00:12.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 -- 172.21.15.83:0/3947586213 wait complete. 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 librados: starting msgr at 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 librados: starting objecter 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.631+0000 7fd7e46a4640 1 Processor -- start 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- start start 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 librados: setting wanted keys 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 librados: calling monclient init 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7dc100de0 con 0x7fd7dc101500 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7dc11d2a0 con 0x7fd7dc107e00 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7dc11e480 con 0x7fd7dc104270 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e1c3f640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd7dc104270 0x7fd7dc117440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:43616/0 (socket says 172.21.15.83:43616) 2024-09-10T19:00:12.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e1c3f640 1 -- 172.21.15.83:0/642682167 learned_addr learned my addr 172.21.15.83:0/642682167 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e2440640 1 --1- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6789/0 conn(0x7fd7dc101500 0x7fd7dc100220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58620/0 (socket says 172.21.15.83:58620) 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter ms_handle_connect 0x7fd7dc104270 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter ms_handle_connect 0x7fd7dc101500 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499831928 0 0) 0x7fd7dc11e480 con 0x7fd7dc104270 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c4003120 con 0x7fd7dc104270 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter ms_handle_connect 0x7fd7dc107e00 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624408606 0 0) 0x7fd7dc100de0 con 0x7fd7dc101500 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc11e480 con 0x7fd7dc101500 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152638921 0 0) 0x7fd7dc11d2a0 con 0x7fd7dc107e00 2024-09-10T19:00:12.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc100de0 con 0x7fd7dc107e00 2024-09-10T19:00:12.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647225920 0 0) 0x7fd7c4003120 con 0x7fd7dc104270 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7dc11d2a0 con 0x7fd7dc104270 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd7d00015d0 con 0x7fd7dc104270 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2858529843 0 0) 0x7fd7dc11e480 con 0x7fd7dc101500 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c4003120 con 0x7fd7dc101500 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd7d4001eb0 con 0x7fd7dc101500 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573480174 0 0) 0x7fd7dc100de0 con 0x7fd7dc107e00 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7dc11e480 con 0x7fd7dc107e00 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd7cc001840 con 0x7fd7dc107e00 2024-09-10T19:00:12.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 349581408 0 0) 0x7fd7c4003120 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6790/0 conn(0x7fd7dc104270 legacy=0x7fd7dc117440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.165:6789/0 conn(0x7fd7dc107e00 legacy=0x7fd7dc11ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc11f660 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc11c2f0 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 10 client.25329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd7d40015f0 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc11c7c0 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 librados: init done 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd7d4005d90 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.635+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd7c8004ff0 con 0x7fd7dc101500 2024-09-10T19:00:12.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd7d4005fb0 con 0x7fd7dc101500 2024-09-10T19:00:12.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(446..446 src has 69..446) v4 ==== 8060+0+0 (unknown 1686100548 0 0) 0x7fd7d408a800 con 0x7fd7dc101500 2024-09-10T19:00:12.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 osd_map(446..446 src has 69..446) v4 2024-09-10T19:00:12.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 3 client.25329.objecter handle_osd_map got epochs [446,446] > 0 2024-09-10T19:00:12.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 3 client.25329.objecter handle_osd_map decoding full epoch 446 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 20 client.25329.objecter dump_active .. 0 homeless 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v446) v1 ==== 54+0+16 (unknown 1274518014 0 2420000192) 0x7fd7d408ac00 con 0x7fd7dc101500 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c4071620 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 1 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 1 2024-09-10T19:00:12.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 0 1 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 4 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 10 client.25329.objecter _send_command 1 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8005290 con 0x7fd7c8004610 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8004610 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8005290 con 0x7fd7c8004610 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 10 client.25329.objecter _finish_command 1 = system:0 2024-09-10T19:00:12.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 4 2024-09-10T19:00:12.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.639+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 0 1 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 2 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 2 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8005810 osd=0 3 2024-09-10T19:00:12.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 0 2 2024-09-10T19:00:12.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 4 2024-09-10T19:00:12.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 10 client.25329.objecter _send_command 2 2024-09-10T19:00:12.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.667+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8004cc0 con 0x7fd7c8004610 2024-09-10T19:00:12.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8004cc0 con 0x7fd7c8004610 2024-09-10T19:00:12.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 10 client.25329.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 4 2024-09-10T19:00:12.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 0 2 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 3 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 3 2024-09-10T19:00:12.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 1 3 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 4 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 10 client.25329.objecter _send_command 3 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c80022b0 con 0x7fd7c8004110 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.671+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8004110 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.675+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c80022b0 con 0x7fd7c8004110 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.675+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.675+0000 7fd7daffd640 10 client.25329.objecter _finish_command 3 = system:0 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.675+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 4 2024-09-10T19:00:12.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.675+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 1 3 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 4 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 4 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8003fa0 osd=1 3 2024-09-10T19:00:12.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 1 4 2024-09-10T19:00:12.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 4 2024-09-10T19:00:12.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 10 client.25329.objecter _send_command 4 2024-09-10T19:00:12.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.691+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8001820 con 0x7fd7c8004110 2024-09-10T19:00:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8001820 con 0x7fd7c8004110 2024-09-10T19:00:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 10 client.25329.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 4 2024-09-10T19:00:12.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 1 4 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 5 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 5 2024-09-10T19:00:12.700 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 2 5 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 4 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 10 client.25329.objecter _send_command 5 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8015060 con 0x7fd7c8003af0 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8003af0 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8015060 con 0x7fd7c8003af0 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 10 client.25329.objecter _finish_command 5 = system:0 2024-09-10T19:00:12.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 4 2024-09-10T19:00:12.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.695+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 2 5 2024-09-10T19:00:12.703 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:12.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 6 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 6 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8001bd0 osd=2 3 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 2 6 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 4 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 10 client.25329.objecter _send_command 6 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c80030d0 con 0x7fd7c8003af0 2024-09-10T19:00:12.728 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:12.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c80030d0 con 0x7fd7c8003af0 2024-09-10T19:00:12.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7daffd640 10 client.25329.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 4 2024-09-10T19:00:12.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 2 6 2024-09-10T19:00:12.730 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:00:12.730 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:00:12.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 7 2024-09-10T19:00:12.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 7 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 3 7 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 4 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 10 client.25329.objecter _send_command 7 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.723+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8011680 con 0x7fd7c8015200 2024-09-10T19:00:12.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8015200 2024-09-10T19:00:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:12.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8011680 con 0x7fd7c8015200 2024-09-10T19:00:12.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:00:12.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 10 client.25329.objecter _finish_command 7 = system:0 2024-09-10T19:00:12.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 4 2024-09-10T19:00:12.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.727+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 3 7 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 8 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 8 2024-09-10T19:00:12.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8010ed0 osd=3 3 2024-09-10T19:00:12.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 3 8 2024-09-10T19:00:12.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 4 2024-09-10T19:00:12.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 10 client.25329.objecter _send_command 8 2024-09-10T19:00:12.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8016530 con 0x7fd7c8015200 2024-09-10T19:00:12.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8016530 con 0x7fd7c8015200 2024-09-10T19:00:12.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7daffd640 10 client.25329.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 4 2024-09-10T19:00:12.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 3 8 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 9 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 9 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 4 9 2024-09-10T19:00:12.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 4 2024-09-10T19:00:12.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 10 client.25329.objecter _send_command 9 2024-09-10T19:00:12.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.743+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c801e610 con 0x7fd7c801ab10 2024-09-10T19:00:12.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c801ab10 2024-09-10T19:00:12.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c801e610 con 0x7fd7c801ab10 2024-09-10T19:00:12.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:00:12.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 10 client.25329.objecter _finish_command 9 = system:0 2024-09-10T19:00:12.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 4 2024-09-10T19:00:12.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.747+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 4 9 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 10 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 10 2024-09-10T19:00:12.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8016530 osd=4 3 2024-09-10T19:00:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 4 10 2024-09-10T19:00:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 4 2024-09-10T19:00:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 10 client.25329.objecter _send_command 10 2024-09-10T19:00:12.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.759+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c801fe80 con 0x7fd7c801ab10 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c801fe80 con 0x7fd7c801ab10 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7daffd640 10 client.25329.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 4 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 4 10 2024-09-10T19:00:12.770 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.763+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 11 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 11 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 5 11 2024-09-10T19:00:12.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 4 2024-09-10T19:00:12.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 10 client.25329.objecter _send_command 11 2024-09-10T19:00:12.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8020a40 con 0x7fd7c8020530 2024-09-10T19:00:12.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8020530 2024-09-10T19:00:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8020a40 con 0x7fd7c8020530 2024-09-10T19:00:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:00:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 10 client.25329.objecter _finish_command 11 = system:0 2024-09-10T19:00:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 4 2024-09-10T19:00:12.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.767+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 5 11 2024-09-10T19:00:12.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:12] "GET /metrics HTTP/1.1" 200 228832 "" "Prometheus/2.43.0" 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 12 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 12 2024-09-10T19:00:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c801fe80 osd=5 3 2024-09-10T19:00:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 5 12 2024-09-10T19:00:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 4 2024-09-10T19:00:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 10 client.25329.objecter _send_command 12 2024-09-10T19:00:12.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.787+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8020a40 con 0x7fd7c8020530 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8020a40 con 0x7fd7c8020530 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 10 client.25329.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 4 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 5 12 2024-09-10T19:00:12.795 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:00:12.796 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:00:12.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 13 2024-09-10T19:00:12.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 13 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 6 13 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 4 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 10 client.25329.objecter _send_command 13 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8026e30 con 0x7fd7c8020be0 2024-09-10T19:00:12.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.791+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8020be0 2024-09-10T19:00:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.795+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8026e30 con 0x7fd7c8020be0 2024-09-10T19:00:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.795+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:00:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.795+0000 7fd7daffd640 10 client.25329.objecter _finish_command 13 = system:0 2024-09-10T19:00:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.795+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 4 2024-09-10T19:00:12.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.795+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 6 13 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 14 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 14 2024-09-10T19:00:12.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8020a40 osd=6 3 2024-09-10T19:00:12.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 6 14 2024-09-10T19:00:12.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 4 2024-09-10T19:00:12.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 10 client.25329.objecter _send_command 14 2024-09-10T19:00:12.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.815+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8026e30 con 0x7fd7c8020be0 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8026e30 con 0x7fd7c8020be0 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 10 client.25329.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 4 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 6 14 2024-09-10T19:00:12.822 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:00:12.823 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:00:12.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:12.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 15 2024-09-10T19:00:12.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 15 2024-09-10T19:00:12.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 7 15 2024-09-10T19:00:12.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 4 2024-09-10T19:00:12.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 10 client.25329.objecter _send_command 15 2024-09-10T19:00:12.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c8028050 con 0x7fd7c8016e10 2024-09-10T19:00:12.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 10 client.25329.objecter ms_handle_connect 0x7fd7c8016e10 2024-09-10T19:00:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd7c8028050 con 0x7fd7c8016e10 2024-09-10T19:00:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:00:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 10 client.25329.objecter _finish_command 15 = system:0 2024-09-10T19:00:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 4 2024-09-10T19:00:12.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.819+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 7 15 2024-09-10T19:00:12.833 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:12.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.835+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fac440052c0 con 0x7fac60107e30 2024-09-10T19:00:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 10 client.25329.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:12.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign -1 16 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 20 client.25329.objecter _get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_remove -1 16 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 20 client.25329.objecter get_session s=0x7fd7c8026e30 osd=7 3 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 15 client.25329.objecter _session_command_op_assign 7 16 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 4 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 10 client.25329.objecter _send_command 16 2024-09-10T19:00:12.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.839+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd7c8028050 con 0x7fd7c8016e10 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.843+0000 7fd7daffd640 1 -- 172.21.15.83:0/642682167 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fd7c8028050 con 0x7fd7c8016e10 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.843+0000 7fd7daffd640 10 client.25329.objecter ms_dispatch 0x7fd7dc000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.843+0000 7fd7daffd640 10 client.25329.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.843+0000 7fd7daffd640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 4 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.843+0000 7fd7daffd640 15 client.25329.objecter _session_command_op_remove 7 16 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:00:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:00:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 librados: watch_flush enter 2024-09-10T19:00:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 librados: watch_flush exit 2024-09-10T19:00:12.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.0 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6801/2535527483 conn(0x7fd7c8004610 legacy=0x7fd7c8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8005810 osd=0 1 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.1 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6805/2309624637 conn(0x7fd7c8004110 legacy=0x7fd7c800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8003fa0 osd=1 1 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.2 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6809/386094220 conn(0x7fd7c8003af0 legacy=0x7fd7c80138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8001bd0 osd=2 1 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.3 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6813/1279688021 conn(0x7fd7c8015200 legacy=0x7fd7c8015610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8010ed0 osd=3 1 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.4 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.165:6800/3225515265 conn(0x7fd7c801ab10 legacy=0x7fd7c801cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8016530 osd=4 1 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.5 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.165:6804/571173777 conn(0x7fd7c8020530 legacy=0x7fd7c801e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c801fe80 osd=5 1 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.6 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.165:6808/1317922936 conn(0x7fd7c8020be0 legacy=0x7fd7c80255a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8020a40 osd=6 1 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter close_session for osd.7 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e2c41640 1 -- 172.21.15.83:0/642682167 reap_dead start 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.165:6812/1610151650 conn(0x7fd7c8016e10 legacy=0x7fd7c8017260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e2c41640 1 -- 172.21.15.83:0/642682167 reap_dead start 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter put_session s=0x7fd7c8026e30 osd=7 1 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 20 client.25329.objecter shutdown clearing up homeless session... 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 10 client.25329.objecter successfully canceled tick 2024-09-10T19:00:12.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e2c41640 1 -- 172.21.15.83:0/642682167 reap_dead start 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e2c41640 1 -- 172.21.15.83:0/642682167 reap_dead start 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd7c4071620 legacy=0x7fd7c4073ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> v1:172.21.15.83:6789/0 conn(0x7fd7dc101500 legacy=0x7fd7dc100220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 shutdown_connections 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 >> 172.21.15.83:0/642682167 conn(0x7fd7dc074af0 msgr2=0x7fd7dc10ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 shutdown_connections 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.847+0000 7fd7e46a4640 1 -- 172.21.15.83:0/642682167 wait complete. 2024-09-10T19:00:12.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:12.851+0000 7fd7e46a4640 1 librados: shutdown 2024-09-10T19:00:12.860 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:12.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:12.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:12.998 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:13.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:13.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:13.264 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:13.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.343+0000 7fac5d7fa640 1 -- 172.21.15.83:0/1760447166 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v447) v1 ==== 101+0+0 (unknown 2728965836 0 0) 0x7fac58053eb0 con 0x7fac60107e30 2024-09-10T19:00:13.348 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:00:13.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 10 librados: watch_flush enter 2024-09-10T19:00:13.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 10 librados: watch_flush exit 2024-09-10T19:00:13.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 20 client.49394.objecter shutdown clearing up homeless session... 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 10 client.49394.objecter successfully canceled tick 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 >> v1:172.21.15.83:6800/3071793735 conn(0x7fac500717a0 legacy=0x7fac50073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 >> v1:172.21.15.83:6790/0 conn(0x7fac60107e30 legacy=0x7fac6011ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 shutdown_connections 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 >> 172.21.15.83:0/1760447166 conn(0x7fac60075290 msgr2=0x7fac600f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 shutdown_connections 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 -- 172.21.15.83:0/1760447166 wait complete. 2024-09-10T19:00:13.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:13.347+0000 7fac67182640 1 librados: shutdown 2024-09-10T19:00:13.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.397482+0000 osd.6 (osd.6) 199 : cluster [DBG] 64.e scrub starts 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.403229+0000 osd.6 (osd.6) 200 : cluster [DBG] 64.e scrub ok 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: audit 2024-09-10T19:00:12.642748+0000 mon.a (mon.0) 2608 : audit [DBG] from='client.? v1:172.21.15.83:0/642682167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: audit 2024-09-10T19:00:12.842268+0000 mon.c (mon.2) 621 : audit [INF] from='client.? v1:172.21.15.83:0/1760447166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: audit 2024-09-10T19:00:12.842808+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.183628+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:00:13.371 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.397482+0000 osd.6 (osd.6) 199 : cluster [DBG] 64.e scrub starts 2024-09-10T19:00:13.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.403229+0000 osd.6 (osd.6) 200 : cluster [DBG] 64.e scrub ok 2024-09-10T19:00:13.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: audit 2024-09-10T19:00:12.642748+0000 mon.a (mon.0) 2608 : audit [DBG] from='client.? v1:172.21.15.83:0/642682167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:13.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: audit 2024-09-10T19:00:12.842268+0000 mon.c (mon.2) 621 : audit [INF] from='client.? v1:172.21.15.83:0/1760447166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: audit 2024-09-10T19:00:12.842808+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:13 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.183628+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:00:13.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:13.504 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.397482+0000 osd.6 (osd.6) 199 : cluster [DBG] 64.e scrub starts 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.403229+0000 osd.6 (osd.6) 200 : cluster [DBG] 64.e scrub ok 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: audit 2024-09-10T19:00:12.642748+0000 mon.a (mon.0) 2608 : audit [DBG] from='client.? v1:172.21.15.83:0/642682167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: audit 2024-09-10T19:00:12.842268+0000 mon.c (mon.2) 621 : audit [INF] from='client.? v1:172.21.15.83:0/1760447166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: audit 2024-09-10T19:00:12.842808+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:13.515 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:13 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.183628+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:00:13.536 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:13.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:13.667 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:13.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:13.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:13.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:13.936 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:13.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:14.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:14.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:14.212 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:14.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:14.347 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:14.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.483 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:14.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.901940+0000 osd.3 (osd.3) 260 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:12.905882+0000 osd.3 (osd.3) 261 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.166619+0000 osd.5 (osd.5) 233 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.172534+0000 osd.5 (osd.5) 234 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.188372+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:00:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.329901+0000 mon.a (mon.0) 2610 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: audit 2024-09-10T19:00:13.346386+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.346547+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.484091+0000 osd.1 (osd.1) 255 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:13.487810+0000 osd.1 (osd.1) 256 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.096515+0000 mgr.y (mgr.24526) 442 : cluster [DBG] pgmap v660: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.135870+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.147540+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:00:14.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:14 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.151851+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:00:14.621 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:14.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.901940+0000 osd.3 (osd.3) 260 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:12.905882+0000 osd.3 (osd.3) 261 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.166619+0000 osd.5 (osd.5) 233 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.172534+0000 osd.5 (osd.5) 234 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.188372+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.329901+0000 mon.a (mon.0) 2610 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:00:14.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: audit 2024-09-10T19:00:13.346386+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.346547+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.484091+0000 osd.1 (osd.1) 255 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:13.487810+0000 osd.1 (osd.1) 256 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.096515+0000 mgr.y (mgr.24526) 442 : cluster [DBG] pgmap v660: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.135870+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.147540+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.151851+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.901940+0000 osd.3 (osd.3) 260 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:12.905882+0000 osd.3 (osd.3) 261 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.166619+0000 osd.5 (osd.5) 233 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.172534+0000 osd.5 (osd.5) 234 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.188372+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:00:14.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.329901+0000 mon.a (mon.0) 2610 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: audit 2024-09-10T19:00:13.346386+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.346547+0000 mon.a (mon.0) 2612 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.484091+0000 osd.1 (osd.1) 255 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:13.487810+0000 osd.1 (osd.1) 256 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.096515+0000 mgr.y (mgr.24526) 442 : cluster [DBG] pgmap v660: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.135870+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.147540+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:00:14.740 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:14 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.151851+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:00:14.757 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:14.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:14.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:14.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:14.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:15.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:15.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:15.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:15.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:15.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:15.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:15.450 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:15.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:15.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:15.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:15.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.183632+0000 osd.2 (osd.2) 179 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.192501+0000 osd.2 (osd.2) 180 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.534277+0000 osd.1 (osd.1) 257 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.537754+0000 osd.1 (osd.1) 258 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: audit 2024-09-10T19:00:14.571980+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: audit 2024-09-10T19:00:14.574468+0000 mon.c (mon.2) 622 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.864679+0000 osd.3 (osd.3) 262 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:14.868367+0000 osd.3 (osd.3) 263 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.112926+0000 osd.4 (osd.4) 276 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.117369+0000 osd.4 (osd.4) 277 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.183632+0000 osd.2 (osd.2) 179 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:00:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.192501+0000 osd.2 (osd.2) 180 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.534277+0000 osd.1 (osd.1) 257 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.537754+0000 osd.1 (osd.1) 258 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: audit 2024-09-10T19:00:14.571980+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: audit 2024-09-10T19:00:14.574468+0000 mon.c (mon.2) 622 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.864679+0000 osd.3 (osd.3) 262 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:14.868367+0000 osd.3 (osd.3) 263 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.112926+0000 osd.4 (osd.4) 276 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:00:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:15 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.117369+0000 osd.4 (osd.4) 277 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:00:15.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:15.862 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.183632+0000 osd.2 (osd.2) 179 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.192501+0000 osd.2 (osd.2) 180 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.534277+0000 osd.1 (osd.1) 257 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.537754+0000 osd.1 (osd.1) 258 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: audit 2024-09-10T19:00:14.571980+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: audit 2024-09-10T19:00:14.574468+0000 mon.c (mon.2) 622 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.864679+0000 osd.3 (osd.3) 262 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:14.868367+0000 osd.3 (osd.3) 263 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.112926+0000 osd.4 (osd.4) 276 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:00:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:15 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.117369+0000 osd.4 (osd.4) 277 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:00:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:15.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:15.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:15.996 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:16.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:16.135 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:16.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:16.271 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:16.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:16.408 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:16.445 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:16.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:16.543 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:16.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:16.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.179210+0000 osd.2 (osd.2) 181 : cluster [DBG] 64.13 deep-scrub starts 2024-09-10T19:00:16.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.186478+0000 osd.2 (osd.2) 182 : cluster [DBG] 64.13 deep-scrub ok 2024-09-10T19:00:16.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.227726+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.230757+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.438716+0000 osd.6 (osd.6) 201 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:15.443260+0000 osd.6 (osd.6) 202 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.098792+0000 mgr.y (mgr.24526) 443 : cluster [DBG] pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 881 B/s wr, 3 op/s 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.125703+0000 osd.0 (osd.0) 234 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.130057+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.328373+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:16.660 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:16 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.328799+0000 osd.3 (osd.3) 265 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:16.679 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:16.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:16.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.179210+0000 osd.2 (osd.2) 181 : cluster [DBG] 64.13 deep-scrub starts 2024-09-10T19:00:16.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.186478+0000 osd.2 (osd.2) 182 : cluster [DBG] 64.13 deep-scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.227726+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.230757+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.438716+0000 osd.6 (osd.6) 201 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:15.443260+0000 osd.6 (osd.6) 202 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.098792+0000 mgr.y (mgr.24526) 443 : cluster [DBG] pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 881 B/s wr, 3 op/s 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.125703+0000 osd.0 (osd.0) 234 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.130057+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.328373+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.328799+0000 osd.3 (osd.3) 265 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.179210+0000 osd.2 (osd.2) 181 : cluster [DBG] 64.13 deep-scrub starts 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.186478+0000 osd.2 (osd.2) 182 : cluster [DBG] 64.13 deep-scrub ok 2024-09-10T19:00:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.227726+0000 osd.5 (osd.5) 235 : cluster [DBG] 2.3 deep-scrub starts 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.230757+0000 osd.5 (osd.5) 236 : cluster [DBG] 2.3 deep-scrub ok 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.438716+0000 osd.6 (osd.6) 201 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:15.443260+0000 osd.6 (osd.6) 202 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.098792+0000 mgr.y (mgr.24526) 443 : cluster [DBG] pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 881 B/s wr, 3 op/s 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.125703+0000 osd.0 (osd.0) 234 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.130057+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.328373+0000 osd.3 (osd.3) 264 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:16.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:16 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.328799+0000 osd.3 (osd.3) 265 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:16.816 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:16.919 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:16.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:00:16.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:00:16.954 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:16.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:16.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:17.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.085 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:17.136 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:00:17.137 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:00:17.137 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T19:00:17.137 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T19:00:17.137 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:00:17.188 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.215 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:17.318 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:17.350 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:17.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:17.487 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:17.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:17.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:17.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.468156+0000 osd.6 (osd.6) 203 : cluster [DBG] 3.a scrub starts 2024-09-10T19:00:17.735 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.473997+0000 osd.6 (osd.6) 204 : cluster [DBG] 3.a scrub ok 2024-09-10T19:00:17.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.585115+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:00:17.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.850526+0000 osd.3 (osd.3) 266 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:00:17.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:16.862570+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:00:17.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:17.046134+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:00:17.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[22591]: cluster 2024-09-10T19:00:17.050043+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:00:17.756 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:17.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:17.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:17.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.468156+0000 osd.6 (osd.6) 203 : cluster [DBG] 3.a scrub starts 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.473997+0000 osd.6 (osd.6) 204 : cluster [DBG] 3.a scrub ok 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.585115+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.850526+0000 osd.3 (osd.3) 266 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:16.862570+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:17.046134+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[18629]: cluster 2024-09-10T19:00:17.050043+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:00:17.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.468156+0000 osd.6 (osd.6) 203 : cluster [DBG] 3.a scrub starts 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.473997+0000 osd.6 (osd.6) 204 : cluster [DBG] 3.a scrub ok 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.585115+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.850526+0000 osd.3 (osd.3) 266 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:16.862570+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:17.046134+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:00:17.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:17 smithi083 bash[32381]: cluster 2024-09-10T19:00:17.050043+0000 osd.7 (osd.7) 255 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:00:17.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:18.012 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:17 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:18.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:18.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:18.165 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:18.268 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:18.304 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:18.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:18.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:18.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:18.578 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:18.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:18.715 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:17.843444+0000 osd.3 (osd.3) 268 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:17.847175+0000 osd.3 (osd.3) 269 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: audit 2024-09-10T19:00:17.973099+0000 mgr.y (mgr.24526) 444 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.072025+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.076455+0000 osd.4 (osd.4) 279 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.099589+0000 mgr.y (mgr.24526) 445 : cluster [DBG] pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.138111+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.141217+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:17.843444+0000 osd.3 (osd.3) 268 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:00:18.814 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:17.847175+0000 osd.3 (osd.3) 269 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: audit 2024-09-10T19:00:17.973099+0000 mgr.y (mgr.24526) 444 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.072025+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.076455+0000 osd.4 (osd.4) 279 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.099589+0000 mgr.y (mgr.24526) 445 : cluster [DBG] pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.138111+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:00:18.815 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:18 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.141217+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:00:18.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:00:18.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:18.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:17.843444+0000 osd.3 (osd.3) 268 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:00:18.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:17.847175+0000 osd.3 (osd.3) 269 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:00:18.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: audit 2024-09-10T19:00:17.973099+0000 mgr.y (mgr.24526) 444 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:18.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.072025+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:00:18.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.076455+0000 osd.4 (osd.4) 279 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:00:18.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.099589+0000 mgr.y (mgr.24526) 445 : cluster [DBG] pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:18.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.138111+0000 osd.0 (osd.0) 236 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:00:18.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:18 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.141217+0000 osd.0 (osd.0) 237 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:00:18.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:18.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:18.987 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:18.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:19.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:19.123 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:19.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:19.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:19.399 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:19.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:19.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:19.535 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:19.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.674 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:19.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.158118+0000 osd.2 (osd.2) 183 : cluster [DBG] 64.a scrub starts 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.161527+0000 osd.2 (osd.2) 184 : cluster [DBG] 64.a scrub ok 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.245482+0000 osd.5 (osd.5) 237 : cluster [DBG] 64.f scrub starts 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[18629]: cluster 2024-09-10T19:00:18.252208+0000 osd.5 (osd.5) 238 : cluster [DBG] 64.f scrub ok 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[18629]: cluster 2024-09-10T19:00:19.135781+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.158118+0000 osd.2 (osd.2) 183 : cluster [DBG] 64.a scrub starts 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.161527+0000 osd.2 (osd.2) 184 : cluster [DBG] 64.a scrub ok 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.245482+0000 osd.5 (osd.5) 237 : cluster [DBG] 64.f scrub starts 2024-09-10T19:00:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[32381]: cluster 2024-09-10T19:00:18.252208+0000 osd.5 (osd.5) 238 : cluster [DBG] 64.f scrub ok 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:19 smithi083 bash[32381]: cluster 2024-09-10T19:00:19.135781+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:19 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.158118+0000 osd.2 (osd.2) 183 : cluster [DBG] 64.a scrub starts 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:19 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.161527+0000 osd.2 (osd.2) 184 : cluster [DBG] 64.a scrub ok 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:19 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.245482+0000 osd.5 (osd.5) 237 : cluster [DBG] 64.f scrub starts 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:19 smithi165 bash[22591]: cluster 2024-09-10T19:00:18.252208+0000 osd.5 (osd.5) 238 : cluster [DBG] 64.f scrub ok 2024-09-10T19:00:19.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:19 smithi165 bash[22591]: cluster 2024-09-10T19:00:19.135781+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:00:19.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:19.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.952 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:19.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:20.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:20.089 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:20.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:20.193 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:20.220 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:20.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:20.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:20.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:20.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:20.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:20.630 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:20.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:20.769 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:20 smithi165 bash[22591]: cluster 2024-09-10T19:00:19.279755+0000 osd.5 (osd.5) 239 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:00:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:20 smithi165 bash[22591]: cluster 2024-09-10T19:00:19.288867+0000 osd.5 (osd.5) 240 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:00:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:20 smithi165 bash[22591]: cluster 2024-09-10T19:00:20.100400+0000 mgr.y (mgr.24526) 446 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:20 smithi165 bash[22591]: cluster 2024-09-10T19:00:20.157853+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:00:20.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:20.907 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:20.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:21.010 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:21.024 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[18629]: cluster 2024-09-10T19:00:19.279755+0000 osd.5 (osd.5) 239 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:00:21.024 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[18629]: cluster 2024-09-10T19:00:19.288867+0000 osd.5 (osd.5) 240 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[18629]: cluster 2024-09-10T19:00:20.100400+0000 mgr.y (mgr.24526) 446 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[18629]: cluster 2024-09-10T19:00:20.157853+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[32381]: cluster 2024-09-10T19:00:19.279755+0000 osd.5 (osd.5) 239 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[32381]: cluster 2024-09-10T19:00:19.288867+0000 osd.5 (osd.5) 240 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[32381]: cluster 2024-09-10T19:00:20.100400+0000 mgr.y (mgr.24526) 446 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:00:21.025 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:20 smithi083 bash[32381]: cluster 2024-09-10T19:00:20.157853+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:00:21.046 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:21.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:21.185 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:21.288 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:21.328 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:21.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:21.445 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:21.458 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:21.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- 172.21.15.83:0/522681725 >> v1:172.21.15.83:6789/0 conn(0x7f61c4072720 legacy=0x7f61c410df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- 172.21.15.83:0/522681725 shutdown_connections 2024-09-10T19:00:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- 172.21.15.83:0/522681725 >> 172.21.15.83:0/522681725 conn(0x7f61c40fa370 msgr2=0x7f61c40fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- 172.21.15.83:0/522681725 shutdown_connections 2024-09-10T19:00:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- 172.21.15.83:0/522681725 wait complete. 2024-09-10T19:00:21.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 librados: starting msgr at 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 librados: starting objecter 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 Processor -- start 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 -- start start 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 librados: setting wanted keys 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.487+0000 7f61ca965640 1 librados: calling monclient init 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c41229b0 con 0x7f61c4072720 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c4123b50 con 0x7f61c4071c40 2024-09-10T19:00:21.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c4124cf0 con 0x7f61c410f680 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c37fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f61c4072720 0x7f61c411db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45692/0 (socket says 172.21.15.83:45692) 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c37fe640 1 -- 172.21.15.83:0/3533912499 learned_addr learned my addr 172.21.15.83:0/3533912499 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter ms_handle_connect 0x7f61c4072720 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter ms_handle_connect 0x7f61c4071c40 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003050050 0 0) 0x7f61c41229b0 con 0x7f61c4072720 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac0034e0 con 0x7f61c4072720 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter ms_handle_connect 0x7f61c410f680 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:21.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639412437 0 0) 0x7f61c4123b50 con 0x7f61c4071c40 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c41229b0 con 0x7f61c4071c40 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482132291 0 0) 0x7f61c4124cf0 con 0x7f61c410f680 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c4123b50 con 0x7f61c410f680 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257641038 0 0) 0x7f61ac0034e0 con 0x7f61c4072720 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c4124cf0 con 0x7f61c4072720 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f61b8002050 con 0x7f61c4072720 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3366121150 0 0) 0x7f61c41229b0 con 0x7f61c4071c40 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61ac0034e0 con 0x7f61c4071c40 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f61bc002c70 con 0x7f61c4071c40 2024-09-10T19:00:21.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092844536 0 0) 0x7f61c4123b50 con 0x7f61c410f680 2024-09-10T19:00:21.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c41229b0 con 0x7f61c410f680 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f61b4001800 con 0x7f61c410f680 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4271598704 0 0) 0x7f61ac0034e0 con 0x7f61c4071c40 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 >> v1:172.21.15.83:6790/0 conn(0x7f61c410f680 legacy=0x7f61c4121290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 >> v1:172.21.15.83:6789/0 conn(0x7f61c4072720 legacy=0x7f61c411db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4125e90 con 0x7f61c4071c40 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c4123d40 con 0x7f61c4071c40 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 10 client.49915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c4124190 con 0x7f61c4071c40 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 librados: init done 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f61bc001a90 con 0x7f61c4071c40 2024-09-10T19:00:21.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f61bc0034d0 con 0x7f61c4071c40 2024-09-10T19:00:21.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.491+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b0005010 con 0x7f61c4071c40 2024-09-10T19:00:21.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f61bc001630 con 0x7f61c4071c40 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(451..451 src has 69..451) v4 ==== 8060+0+0 (unknown 3511510775 0 0) 0x7f61bc088090 con 0x7f61c4071c40 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 10 client.49915.objecter ms_dispatch 0x7f61c4000b90 osd_map(451..451 src has 69..451) v4 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 3 client.49915.objecter handle_osd_map got epochs [451,451] > 0 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 3 client.49915.objecter handle_osd_map decoding full epoch 451 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 20 client.49915.objecter dump_active .. 0 homeless 2024-09-10T19:00:21.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.495+0000 7f61c0ff9640 10 client.49915.objecter ms_handle_connect 0x7f61ac0716c0 2024-09-10T19:00:21.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.507+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f61bc054190 con 0x7f61c4071c40 2024-09-10T19:00:21.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:21.592 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:21.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.687+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61b0000b90 con 0x7f61ac0716c0 2024-09-10T19:00:21.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:21.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.715+0000 7f61c0ff9640 1 -- 172.21.15.83:0/3533912499 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+625984 (unknown 2965378022 0 1926132783) 0x7f61b0000b90 con 0x7f61ac0716c0 2024-09-10T19:00:21.726 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:00:21.729 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:00:21.733 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 10 librados: watch_flush enter 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 10 librados: watch_flush exit 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 20 client.49915.objecter shutdown clearing up homeless session... 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 10 client.49915.objecter successfully canceled tick 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 >> v1:172.21.15.83:6800/3071793735 conn(0x7f61ac0716c0 legacy=0x7f61ac073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:21.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 >> v1:172.21.15.165:6789/0 conn(0x7f61c4071c40 legacy=0x7f61c410e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 shutdown_connections 2024-09-10T19:00:21.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 >> 172.21.15.83:0/3533912499 conn(0x7f61c40fa370 msgr2=0x7f61c40fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:21.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 shutdown_connections 2024-09-10T19:00:21.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 -- 172.21.15.83:0/3533912499 wait complete. 2024-09-10T19:00:21.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:21.723+0000 7f61ca965640 1 librados: shutdown 2024-09-10T19:00:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:21.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:20.535958+0000 osd.6 (osd.6) 205 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:20.541034+0000 osd.6 (osd.6) 206 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.096522+0000 osd.4 (osd.4) 280 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.100049+0000 osd.4 (osd.4) 281 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.201729+0000 osd.0 (osd.0) 238 : cluster [DBG] 12.c scrub starts 2024-09-10T19:00:21.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:21 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.205169+0000 osd.0 (osd.0) 239 : cluster [DBG] 12.c scrub ok 2024-09-10T19:00:21.871 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:21.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:21.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:20.535958+0000 osd.6 (osd.6) 205 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:00:21.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:20.541034+0000 osd.6 (osd.6) 206 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:00:21.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.096522+0000 osd.4 (osd.4) 280 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:00:21.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.100049+0000 osd.4 (osd.4) 281 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:00:21.910 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.201729+0000 osd.0 (osd.0) 238 : cluster [DBG] 12.c scrub starts 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.205169+0000 osd.0 (osd.0) 239 : cluster [DBG] 12.c scrub ok 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:20.535958+0000 osd.6 (osd.6) 205 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:20.541034+0000 osd.6 (osd.6) 206 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.096522+0000 osd.4 (osd.4) 280 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.100049+0000 osd.4 (osd.4) 281 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.201729+0000 osd.0 (osd.0) 238 : cluster [DBG] 12.c scrub starts 2024-09-10T19:00:21.911 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:21 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.205169+0000 osd.0 (osd.0) 239 : cluster [DBG] 12.c scrub ok 2024-09-10T19:00:21.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:22.010 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:22.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:22.149 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:22.252 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:22.287 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:22.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:22.426 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:22.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:22.560 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-09-10T19:00:20.152512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792000,"num_objects":362,"num_object_clones":4,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":2060,"num_read_kb":1813,"num_write":1445,"num_write_kb":14370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2849,"ondisk_log_size":2849,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2501236,"kb_used_data":80700,"kb_used_omap":13,"kb_used_meta":2420338,"kb_avail":747459980,"statfs":{"total":767960285184,"available":765399019520,"internally_reserved":0,"allocated":82636800,"data_stored":58062178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2478426816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.021107"},"pg_stats":[{"pgid":"215.1f","version":"442'2","reported_seq":19,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339401+0000","last_change":"2024-09-10T19:00:04.268476+0000","last_active":"2024-09-10T19:00:13.339401+0000","last_peered":"2024-09-10T19:00:13.339401+0000","last_clean":"2024-09-10T19:00:13.339401+0000","last_became_active":"2024-09-10T19:00:04.267732+0000","last_became_peered":"2024-09-10T19:00:04.267732+0000","last_unstale":"2024-09-10T19:00:13.339401+0000","last_undegraded":"2024-09-10T19:00:13.339401+0000","last_fullsized":"2024-09-10T19:00:13.339401+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.164189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.1e","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477937+0000","last_change":"2024-09-10T19:00:04.291945+0000","last_active":"2024-09-10T19:00:13.477937+0000","last_peered":"2024-09-10T19:00:13.477937+0000","last_clean":"2024-09-10T19:00:13.477937+0000","last_became_active":"2024-09-10T19:00:04.290965+0000","last_became_peered":"2024-09-10T19:00:04.290965+0000","last_unstale":"2024-09-10T19:00:13.477937+0000","last_undegraded":"2024-09-10T19:00:13.477937+0000","last_fullsized":"2024-09-10T19:00:13.477937+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.075695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.1d","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349380+0000","last_change":"2024-09-10T19:00:04.276088+0000","last_active":"2024-09-10T19:00:13.349380+0000","last_peered":"2024-09-10T19:00:13.349380+0000","last_clean":"2024-09-10T19:00:13.349380+0000","last_became_active":"2024-09-10T19:00:04.275821+0000","last_became_peered":"2024-09-10T19:00:04.275821+0000","last_unstale":"2024-09-10T19:00:13.349380+0000","last_undegraded":"2024-09-10T19:00:13.349380+0000","last_fullsized":"2024-09-10T19:00:13.349380+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.739652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.1c","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474518+0000","last_change":"2024-09-10T19:00:04.286817+0000","last_active":"2024-09-10T19:00:13.474518+0000","last_peered":"2024-09-10T19:00:13.474518+0000","last_clean":"2024-09-10T19:00:13.474518+0000","last_became_active":"2024-09-10T19:00:04.286469+0000","last_became_peered":"2024-09-10T19:00:04.286469+0000","last_unstale":"2024-09-10T19:00:13.474518+0000","last_undegraded":"2024-09-10T19:00:13.474518+0000","last_fullsized":"2024-09-10T19:00:13.474518+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.583907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1b","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474244+0000","last_change":"2024-09-10T19:00:04.288725+0000","last_active":"2024-09-10T19:00:13.474244+0000","last_peered":"2024-09-10T19:00:13.474244+0000","last_clean":"2024-09-10T19:00:13.474244+0000","last_became_active":"2024-09-10T19:00:04.288361+0000","last_became_peered":"2024-09-10T19:00:04.288361+0000","last_unstale":"2024-09-10T19:00:13.474244+0000","last_undegraded":"2024-09-10T19:00:13.474244+0000","last_fullsized":"2024-09-10T19:00:13.474244+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.963316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1a","version":"445'3","reported_seq":19,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472203+0000","last_change":"2024-09-10T19:00:04.275768+0000","last_active":"2024-09-10T19:00:13.472203+0000","last_peered":"2024-09-10T19:00:13.472203+0000","last_clean":"2024-09-10T19:00:13.472203+0000","last_became_active":"2024-09-10T19:00:04.275245+0000","last_became_peered":"2024-09-10T19:00:04.275245+0000","last_unstale":"2024-09-10T19:00:13.472203+0000","last_undegraded":"2024-09-10T19:00:13.472203+0000","last_fullsized":"2024-09-10T19:00:13.472203+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.008216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.19","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473179+0000","last_change":"2024-09-10T19:00:04.283513+0000","last_active":"2024-09-10T19:00:13.473179+0000","last_peered":"2024-09-10T19:00:13.473179+0000","last_clean":"2024-09-10T19:00:13.473179+0000","last_became_active":"2024-09-10T19:00:04.283166+0000","last_became_peered":"2024-09-10T19:00:04.283166+0000","last_unstale":"2024-09-10T19:00:13.473179+0000","last_undegraded":"2024-09-10T19:00:13.473179+0000","last_fullsized":"2024-09-10T19:00:13.473179+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.031130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.18","version":"446'5","reported_seq":24,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470736+0000","last_change":"2024-09-10T19:00:04.280704+0000","last_active":"2024-09-10T19:00:13.470736+0000","last_peered":"2024-09-10T19:00:13.470736+0000","last_clean":"2024-09-10T19:00:13.470736+0000","last_became_active":"2024-09-10T19:00:04.280190+0000","last_became_peered":"2024-09-10T19:00:04.280190+0000","last_unstale":"2024-09-10T19:00:13.470736+0000","last_undegraded":"2024-09-10T19:00:13.470736+0000","last_fullsized":"2024-09-10T19:00:13.470736+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.454766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.17","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350401+0000","last_change":"2024-09-10T19:00:04.279347+0000","last_active":"2024-09-10T19:00:13.350401+0000","last_peered":"2024-09-10T19:00:13.350401+0000","last_clean":"2024-09-10T19:00:13.350401+0000","last_became_active":"2024-09-10T19:00:04.277956+0000","last_became_peered":"2024-09-10T19:00:04.277956+0000","last_unstale":"2024-09-10T19:00:13.350401+0000","last_undegraded":"2024-09-10T19:00:13.350401+0000","last_fullsized":"2024-09-10T19:00:13.350401+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.373100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.16","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413255+0000","last_change":"2024-09-10T19:00:04.287474+0000","last_active":"2024-09-10T19:00:13.413255+0000","last_peered":"2024-09-10T19:00:13.413255+0000","last_clean":"2024-09-10T19:00:13.413255+0000","last_became_active":"2024-09-10T19:00:04.287210+0000","last_became_peered":"2024-09-10T19:00:04.287210+0000","last_unstale":"2024-09-10T19:00:13.413255+0000","last_undegraded":"2024-09-10T19:00:13.413255+0000","last_fullsized":"2024-09-10T19:00:13.413255+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.857969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.15","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474260+0000","last_change":"2024-09-10T19:00:04.287919+0000","last_active":"2024-09-10T19:00:13.474260+0000","last_peered":"2024-09-10T19:00:13.474260+0000","last_clean":"2024-09-10T19:00:13.474260+0000","last_became_active":"2024-09-10T19:00:04.287638+0000","last_became_peered":"2024-09-10T19:00:04.287638+0000","last_unstale":"2024-09-10T19:00:13.474260+0000","last_undegraded":"2024-09-10T19:00:13.474260+0000","last_fullsized":"2024-09-10T19:00:13.474260+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.14","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.338451+0000","last_change":"2024-09-10T19:00:04.275814+0000","last_active":"2024-09-10T19:00:13.338451+0000","last_peered":"2024-09-10T19:00:13.338451+0000","last_clean":"2024-09-10T19:00:13.338451+0000","last_became_active":"2024-09-10T19:00:04.275307+0000","last_became_peered":"2024-09-10T19:00:04.275307+0000","last_unstale":"2024-09-10T19:00:13.338451+0000","last_undegraded":"2024-09-10T19:00:13.338451+0000","last_fullsized":"2024-09-10T19:00:13.338451+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:03.936168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.13","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412328+0000","last_change":"2024-09-10T19:00:04.283918+0000","last_active":"2024-09-10T19:00:13.412328+0000","last_peered":"2024-09-10T19:00:13.412328+0000","last_clean":"2024-09-10T19:00:13.412328+0000","last_became_active":"2024-09-10T19:00:04.283556+0000","last_became_peered":"2024-09-10T19:00:04.283556+0000","last_unstale":"2024-09-10T19:00:13.412328+0000","last_undegraded":"2024-09-10T19:00:13.412328+0000","last_fullsized":"2024-09-10T19:00:13.412328+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.952005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.12","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347480+0000","last_change":"2024-09-10T19:00:04.275283+0000","last_active":"2024-09-10T19:00:13.347480+0000","last_peered":"2024-09-10T19:00:13.347480+0000","last_clean":"2024-09-10T19:00:13.347480+0000","last_became_active":"2024-09-10T19:00:04.274956+0000","last_became_peered":"2024-09-10T19:00:04.274956+0000","last_unstale":"2024-09-10T19:00:13.347480+0000","last_undegraded":"2024-09-10T19:00:13.347480+0000","last_fullsized":"2024-09-10T19:00:13.347480+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.517350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.11","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345923+0000","last_change":"2024-09-10T19:00:04.287301+0000","last_active":"2024-09-10T19:00:13.345923+0000","last_peered":"2024-09-10T19:00:13.345923+0000","last_clean":"2024-09-10T19:00:13.345923+0000","last_became_active":"2024-09-10T19:00:04.287016+0000","last_became_peered":"2024-09-10T19:00:04.287016+0000","last_unstale":"2024-09-10T19:00:13.345923+0000","last_undegraded":"2024-09-10T19:00:13.345923+0000","last_fullsized":"2024-09-10T19:00:13.345923+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.617302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.10","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.411758+0000","last_change":"2024-09-10T19:00:04.282956+0000","last_active":"2024-09-10T19:00:13.411758+0000","last_peered":"2024-09-10T19:00:13.411758+0000","last_clean":"2024-09-10T19:00:13.411758+0000","last_became_active":"2024-09-10T19:00:04.281332+0000","last_became_peered":"2024-09-10T19:00:04.281332+0000","last_unstale":"2024-09-10T19:00:13.411758+0000","last_undegraded":"2024-09-10T19:00:13.411758+0000","last_fullsized":"2024-09-10T19:00:13.411758+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:07.407596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.f","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478577+0000","last_change":"2024-09-10T19:00:04.274491+0000","last_active":"2024-09-10T19:00:13.478577+0000","last_peered":"2024-09-10T19:00:13.478577+0000","last_clean":"2024-09-10T19:00:13.478577+0000","last_became_active":"2024-09-10T19:00:04.274224+0000","last_became_peered":"2024-09-10T19:00:04.274224+0000","last_unstale":"2024-09-10T19:00:13.478577+0000","last_undegraded":"2024-09-10T19:00:13.478577+0000","last_fullsized":"2024-09-10T19:00:13.478577+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.187213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"215.e","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413398+0000","last_change":"2024-09-10T19:00:04.286254+0000","last_active":"2024-09-10T19:00:13.413398+0000","last_peered":"2024-09-10T19:00:13.413398+0000","last_clean":"2024-09-10T19:00:13.413398+0000","last_became_active":"2024-09-10T19:00:04.285951+0000","last_became_peered":"2024-09-10T19:00:04.285951+0000","last_unstale":"2024-09-10T19:00:13.413398+0000","last_undegraded":"2024-09-10T19:00:13.413398+0000","last_fullsized":"2024-09-10T19:00:13.413398+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.086526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.d","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345938+0000","last_change":"2024-09-10T19:00:04.276650+0000","last_active":"2024-09-10T19:00:13.345938+0000","last_peered":"2024-09-10T19:00:13.345938+0000","last_clean":"2024-09-10T19:00:13.345938+0000","last_became_active":"2024-09-10T19:00:04.276363+0000","last_became_peered":"2024-09-10T19:00:04.276363+0000","last_unstale":"2024-09-10T19:00:13.345938+0000","last_undegraded":"2024-09-10T19:00:13.345938+0000","last_fullsized":"2024-09-10T19:00:13.345938+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.699872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.c","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349484+0000","last_change":"2024-09-10T19:00:04.283152+0000","last_active":"2024-09-10T19:00:13.349484+0000","last_peered":"2024-09-10T19:00:13.349484+0000","last_clean":"2024-09-10T19:00:13.349484+0000","last_became_active":"2024-09-10T19:00:04.282635+0000","last_became_peered":"2024-09-10T19:00:04.282635+0000","last_unstale":"2024-09-10T19:00:13.349484+0000","last_undegraded":"2024-09-10T19:00:13.349484+0000","last_fullsized":"2024-09-10T19:00:13.349484+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.211376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.b","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412608+0000","last_change":"2024-09-10T19:00:04.292003+0000","last_active":"2024-09-10T19:00:13.412608+0000","last_peered":"2024-09-10T19:00:13.412608+0000","last_clean":"2024-09-10T19:00:13.412608+0000","last_became_active":"2024-09-10T19:00:04.291749+0000","last_became_peered":"2024-09-10T19:00:04.291749+0000","last_unstale":"2024-09-10T19:00:13.412608+0000","last_undegraded":"2024-09-10T19:00:13.412608+0000","last_fullsized":"2024-09-10T19:00:13.412608+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.927530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.a","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412490+0000","last_change":"2024-09-10T19:00:04.285065+0000","last_active":"2024-09-10T19:00:13.412490+0000","last_peered":"2024-09-10T19:00:13.412490+0000","last_clean":"2024-09-10T19:00:13.412490+0000","last_became_active":"2024-09-10T19:00:04.284826+0000","last_became_peered":"2024-09-10T19:00:04.284826+0000","last_unstale":"2024-09-10T19:00:13.412490+0000","last_undegraded":"2024-09-10T19:00:13.412490+0000","last_fullsized":"2024-09-10T19:00:13.412490+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.9","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472712+0000","last_change":"2024-09-10T19:00:04.270585+0000","last_active":"2024-09-10T19:00:13.472712+0000","last_peered":"2024-09-10T19:00:13.472712+0000","last_clean":"2024-09-10T19:00:13.472712+0000","last_became_active":"2024-09-10T19:00:04.270285+0000","last_became_peered":"2024-09-10T19:00:04.270285+0000","last_unstale":"2024-09-10T19:00:13.472712+0000","last_undegraded":"2024-09-10T19:00:13.472712+0000","last_fullsized":"2024-09-10T19:00:13.472712+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.542757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.8","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.469949+0000","last_change":"2024-09-10T19:00:04.287128+0000","last_active":"2024-09-10T19:00:13.469949+0000","last_peered":"2024-09-10T19:00:13.469949+0000","last_clean":"2024-09-10T19:00:13.469949+0000","last_became_active":"2024-09-10T19:00:04.286817+0000","last_became_peered":"2024-09-10T19:00:04.286817+0000","last_unstale":"2024-09-10T19:00:13.469949+0000","last_undegraded":"2024-09-10T19:00:13.469949+0000","last_fullsized":"2024-09-10T19:00:13.469949+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.684928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.7","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413505+0000","last_change":"2024-09-10T19:00:04.293140+0000","last_active":"2024-09-10T19:00:13.413505+0000","last_peered":"2024-09-10T19:00:13.413505+0000","last_clean":"2024-09-10T19:00:13.413505+0000","last_became_active":"2024-09-10T19:00:04.292824+0000","last_became_peered":"2024-09-10T19:00:04.292824+0000","last_unstale":"2024-09-10T19:00:13.413505+0000","last_undegraded":"2024-09-10T19:00:13.413505+0000","last_fullsized":"2024-09-10T19:00:13.413505+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.962055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.6","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473758+0000","last_change":"2024-09-10T19:00:04.279607+0000","last_active":"2024-09-10T19:00:13.473758+0000","last_peered":"2024-09-10T19:00:13.473758+0000","last_clean":"2024-09-10T19:00:13.473758+0000","last_became_active":"2024-09-10T19:00:04.279164+0000","last_became_peered":"2024-09-10T19:00:04.279164+0000","last_unstale":"2024-09-10T19:00:13.473758+0000","last_undegraded":"2024-09-10T19:00:13.473758+0000","last_fullsized":"2024-09-10T19:00:13.473758+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.486237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.5","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473560+0000","last_change":"2024-09-10T19:00:04.272438+0000","last_active":"2024-09-10T19:00:13.473560+0000","last_peered":"2024-09-10T19:00:13.473560+0000","last_clean":"2024-09-10T19:00:13.473560+0000","last_became_active":"2024-09-10T19:00:04.272020+0000","last_became_peered":"2024-09-10T19:00:04.272020+0000","last_unstale":"2024-09-10T19:00:13.473560+0000","last_undegraded":"2024-09-10T19:00:13.473560+0000","last_fullsized":"2024-09-10T19:00:13.473560+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.089634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.4","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348445+0000","last_change":"2024-09-10T19:00:04.276737+0000","last_active":"2024-09-10T19:00:13.348445+0000","last_peered":"2024-09-10T19:00:13.348445+0000","last_clean":"2024-09-10T19:00:13.348445+0000","last_became_active":"2024-09-10T19:00:04.276433+0000","last_became_peered":"2024-09-10T19:00:04.276433+0000","last_unstale":"2024-09-10T19:00:13.348445+0000","last_undegraded":"2024-09-10T19:00:13.348445+0000","last_fullsized":"2024-09-10T19:00:13.348445+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.159916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.3","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337206+0000","last_change":"2024-09-10T19:00:04.272390+0000","last_active":"2024-09-10T19:00:13.337206+0000","last_peered":"2024-09-10T19:00:13.337206+0000","last_clean":"2024-09-10T19:00:13.337206+0000","last_became_active":"2024-09-10T19:00:04.272053+0000","last_became_peered":"2024-09-10T19:00:04.272053+0000","last_unstale":"2024-09-10T19:00:13.337206+0000","last_undegraded":"2024-09-10T19:00:13.337206+0000","last_fullsized":"2024-09-10T19:00:13.337206+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.2","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.471409+0000","last_change":"2024-09-10T19:00:04.283670+0000","last_active":"2024-09-10T19:00:13.471409+0000","last_peered":"2024-09-10T19:00:13.471409+0000","last_clean":"2024-09-10T19:00:13.471409+0000","last_became_active":"2024-09-10T19:00:04.283381+0000","last_became_peered":"2024-09-10T19:00:04.283381+0000","last_unstale":"2024-09-10T19:00:13.471409+0000","last_undegraded":"2024-09-10T19:00:13.471409+0000","last_fullsized":"2024-09-10T19:00:13.471409+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.053377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470197+0000","last_change":"2024-09-10T19:00:04.287062+0000","last_active":"2024-09-10T19:00:13.470197+0000","last_peered":"2024-09-10T19:00:13.470197+0000","last_clean":"2024-09-10T19:00:13.470197+0000","last_became_active":"2024-09-10T19:00:04.286750+0000","last_became_peered":"2024-09-10T19:00:04.286750+0000","last_unstale":"2024-09-10T19:00:13.470197+0000","last_undegraded":"2024-09-10T19:00:13.470197+0000","last_fullsized":"2024-09-10T19:00:13.470197+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.558827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.0","version":"0'0","reported_seq":16,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476217+0000","last_change":"2024-09-10T19:00:04.277890+0000","last_active":"2024-09-10T19:00:13.476217+0000","last_peered":"2024-09-10T19:00:13.476217+0000","last_clean":"2024-09-10T19:00:13.476217+0000","last_became_active":"2024-09-10T19:00:04.277555+0000","last_became_peered":"2024-09-10T19:00:04.277555+0000","last_unstale":"2024-09-10T19:00:13.476217+0000","last_undegraded":"2024-09-10T19:00:13.476217+0000","last_fullsized":"2024-09-10T19:00:13.476217+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:27.693272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":411,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350545+0000","last_change":"2024-09-10T19:00:09.133776+0000","last_active":"2024-09-10T19:00:13.350545+0000","last_peered":"2024-09-10T19:00:13.350545+0000","last_clean":"2024-09-10T19:00:13.350545+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:00:13.350545+0000","last_undegraded":"2024-09-10T19:00:13.350545+0000","last_fullsized":"2024-09-10T19:00:13.350545+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.087038+0000","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.001151228,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1e","version":"377'67","reported_seq":441,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348676+0000","last_change":"2024-09-10T19:00:09.133632+0000","last_active":"2024-09-10T19:00:13.348676+0000","last_peered":"2024-09-10T19:00:13.348676+0000","last_clean":"2024-09-10T19:00:13.348676+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:00:13.348676+0000","last_undegraded":"2024-09-10T19:00:13.348676+0000","last_fullsized":"2024-09-10T19:00:13.348676+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.001768403,"stat_sum":{"num_bytes":64,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1d","version":"425'25","reported_seq":417,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478039+0000","last_change":"2024-09-10T19:00:09.141997+0000","last_active":"2024-09-10T19:00:13.478039+0000","last_peered":"2024-09-10T19:00:13.478039+0000","last_clean":"2024-09-10T19:00:13.478039+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:00:13.478039+0000","last_undegraded":"2024-09-10T19:00:13.478039+0000","last_fullsized":"2024-09-10T19:00:13.478039+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'25","last_scrub_stamp":"2024-09-10T19:00:03.071902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.071902+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.351800+0000","scrub_duration":0.0041116920000000001,"objects_trimmed":0,"snaptrim_duration":0.00054085400000000001,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1c","version":"244'51","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474436+0000","last_change":"2024-09-10T19:00:09.154076+0000","last_active":"2024-09-10T19:00:13.474436+0000","last_peered":"2024-09-10T19:00:13.474436+0000","last_clean":"2024-09-10T19:00:13.474436+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:00:13.474436+0000","last_undegraded":"2024-09-10T19:00:13.474436+0000","last_fullsized":"2024-09-10T19:00:13.474436+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:58:54.847269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.847269+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091152980000000008,"objects_trimmed":0,"snaptrim_duration":0.001309983,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1b","version":"240'68","reported_seq":442,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473459+0000","last_change":"2024-09-10T19:00:09.153044+0000","last_active":"2024-09-10T19:00:13.473459+0000","last_peered":"2024-09-10T19:00:13.473459+0000","last_clean":"2024-09-10T19:00:13.473459+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:00:13.473459+0000","last_undegraded":"2024-09-10T19:00:13.473459+0000","last_fullsized":"2024-09-10T19:00:13.473459+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.764846+0000","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.00091902300000000004,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":418,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476722+0000","last_change":"2024-09-10T19:00:09.140420+0000","last_active":"2024-09-10T19:00:13.476722+0000","last_peered":"2024-09-10T19:00:13.476722+0000","last_clean":"2024-09-10T19:00:13.476722+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:00:13.476722+0000","last_undegraded":"2024-09-10T19:00:13.476722+0000","last_fullsized":"2024-09-10T19:00:13.476722+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:59:58.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.060206+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.833991+0000","scrub_duration":0.0042048110000000001,"objects_trimmed":0,"snaptrim_duration":0.000713731,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.19","version":"446'43","reported_seq":423,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345330+0000","last_change":"2024-09-10T19:00:11.141786+0000","last_active":"2024-09-10T19:00:13.345330+0000","last_peered":"2024-09-10T19:00:13.345330+0000","last_clean":"2024-09-10T19:00:13.345330+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:00:13.345330+0000","last_undegraded":"2024-09-10T19:00:13.345330+0000","last_fullsized":"2024-09-10T19:00:13.345330+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'22","last_scrub_stamp":"2024-09-10T18:57:53.142346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.142346+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080540149999999994,"objects_trimmed":1,"snaptrim_duration":2.0058510420000002,"stat_sum":{"num_bytes":58,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.18","version":"254'104","reported_seq":471,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470097+0000","last_change":"2024-09-10T19:00:13.172720+0000","last_active":"2024-09-10T19:00:13.470097+0000","last_peered":"2024-09-10T19:00:13.470097+0000","last_clean":"2024-09-10T19:00:13.470097+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:00:13.470097+0000","last_undegraded":"2024-09-10T19:00:13.470097+0000","last_fullsized":"2024-09-10T19:00:13.470097+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T19:00:13.172545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.172545+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.420509+0000","scrub_duration":0.0057922160000000002,"objects_trimmed":0,"snaptrim_duration":0.00054311599999999995,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.17","version":"428'23","reported_seq":398,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476465+0000","last_change":"2024-09-10T19:00:09.153066+0000","last_active":"2024-09-10T19:00:13.476465+0000","last_peered":"2024-09-10T19:00:13.476465+0000","last_clean":"2024-09-10T19:00:13.476465+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:00:13.476465+0000","last_undegraded":"2024-09-10T19:00:13.476465+0000","last_fullsized":"2024-09-10T19:00:13.476465+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'17","last_scrub_stamp":"2024-09-10T18:57:49.532065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:49.532065+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033982259999999999,"objects_trimmed":0,"snaptrim_duration":0.00059798700000000002,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.16","version":"387'24","reported_seq":411,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.479790+0000","last_change":"2024-09-10T19:00:09.154491+0000","last_active":"2024-09-10T19:00:13.479790+0000","last_peered":"2024-09-10T19:00:13.479790+0000","last_clean":"2024-09-10T19:00:13.479790+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:00:13.479790+0000","last_undegraded":"2024-09-10T19:00:13.479790+0000","last_fullsized":"2024-09-10T19:00:13.479790+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.707234+0000","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.00059059,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.15","version":"428'79","reported_seq":457,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412999+0000","last_change":"2024-09-10T19:00:09.246033+0000","last_active":"2024-09-10T19:00:13.412999+0000","last_peered":"2024-09-10T19:00:13.412999+0000","last_clean":"2024-09-10T19:00:13.412999+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:00:13.412999+0000","last_undegraded":"2024-09-10T19:00:13.412999+0000","last_fullsized":"2024-09-10T19:00:13.412999+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'75","last_scrub_stamp":"2024-09-10T18:59:01.892227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.892227+0000","objects_scrubbed":7,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063234340000000002,"objects_trimmed":0,"snaptrim_duration":0.0021460160000000002,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.14","version":"434'100","reported_seq":483,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474701+0000","last_change":"2024-09-10T19:00:09.154463+0000","last_active":"2024-09-10T19:00:13.474701+0000","last_peered":"2024-09-10T19:00:13.474701+0000","last_clean":"2024-09-10T19:00:13.474701+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:00:13.474701+0000","last_undegraded":"2024-09-10T19:00:13.474701+0000","last_fullsized":"2024-09-10T19:00:13.474701+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.001548511,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.13","version":"448'4","reported_seq":407,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:15.186862+0000","last_change":"2024-09-10T19:00:15.186686+0000","last_active":"2024-09-10T19:00:15.186862+0000","last_peered":"2024-09-10T19:00:15.186862+0000","last_clean":"2024-09-10T19:00:15.186862+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:00:15.186862+0000","last_undegraded":"2024-09-10T19:00:15.186862+0000","last_fullsized":"2024-09-10T19:00:15.186862+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'4","last_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_deep_scrub":"448'4","last_deep_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.186496+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:43.223489+0000","scrub_duration":0.0071427519999999996,"objects_trimmed":0,"snaptrim_duration":0.0027226429999999999,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.12","version":"248'51","reported_seq":413,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.471851+0000","last_change":"2024-09-10T19:00:09.139988+0000","last_active":"2024-09-10T19:00:13.471851+0000","last_peered":"2024-09-10T19:00:13.471851+0000","last_clean":"2024-09-10T19:00:13.471851+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:00:13.471851+0000","last_undegraded":"2024-09-10T19:00:13.471851+0000","last_fullsized":"2024-09-10T19:00:13.471851+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.0013440240000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.11","version":"297'52","reported_seq":424,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336266+0000","last_change":"2024-09-10T19:00:09.131993+0000","last_active":"2024-09-10T19:00:13.336266+0000","last_peered":"2024-09-10T19:00:13.336266+0000","last_clean":"2024-09-10T19:00:13.336266+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T19:00:13.336266+0000","last_undegraded":"2024-09-10T19:00:13.336266+0000","last_fullsized":"2024-09-10T19:00:13.336266+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'52","last_scrub_stamp":"2024-09-10T18:57:51.784182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.784182+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048360160000000003,"objects_trimmed":0,"snaptrim_duration":0.0018507490000000001,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.10","version":"377'35","reported_seq":407,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343422+0000","last_change":"2024-09-10T19:00:09.137517+0000","last_active":"2024-09-10T19:00:13.343422+0000","last_peered":"2024-09-10T19:00:13.343422+0000","last_clean":"2024-09-10T19:00:13.343422+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:00:13.343422+0000","last_undegraded":"2024-09-10T19:00:13.343422+0000","last_fullsized":"2024-09-10T19:00:13.343422+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0016697909999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.f","version":"431'70","reported_seq":440,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474486+0000","last_change":"2024-09-10T19:00:09.144327+0000","last_active":"2024-09-10T19:00:13.474486+0000","last_peered":"2024-09-10T19:00:13.474486+0000","last_clean":"2024-09-10T19:00:13.474486+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:00:13.474486+0000","last_undegraded":"2024-09-10T19:00:13.474486+0000","last_fullsized":"2024-09-10T19:00:13.474486+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'63","last_scrub_stamp":"2024-09-10T18:57:39.210783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.210783+0000","objects_scrubbed":5,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060217370000000001,"objects_trimmed":0,"snaptrim_duration":0.00039215099999999998,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.e","version":"246'119","reported_seq":486,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475681+0000","last_change":"2024-09-10T19:00:12.403421+0000","last_active":"2024-09-10T19:00:13.475681+0000","last_peered":"2024-09-10T19:00:13.475681+0000","last_clean":"2024-09-10T19:00:13.475681+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:00:13.475681+0000","last_undegraded":"2024-09-10T19:00:13.475681+0000","last_fullsized":"2024-09-10T19:00:13.475681+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T19:00:12.403246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.403246+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.629677+0000","scrub_duration":0.0056039189999999997,"objects_trimmed":0,"snaptrim_duration":0.0010794800000000001,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.d","version":"438'22","reported_seq":412,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.338659+0000","last_change":"2024-09-10T19:00:09.133730+0000","last_active":"2024-09-10T19:00:13.338659+0000","last_peered":"2024-09-10T19:00:13.338659+0000","last_clean":"2024-09-10T19:00:13.338659+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T19:00:13.338659+0000","last_undegraded":"2024-09-10T19:00:13.338659+0000","last_fullsized":"2024-09-10T19:00:13.338659+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:25.313261+0000","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.00096098099999999997,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.c","version":"377'23","reported_seq":400,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473196+0000","last_change":"2024-09-10T19:00:09.144104+0000","last_active":"2024-09-10T19:00:13.473196+0000","last_peered":"2024-09-10T19:00:13.473196+0000","last_clean":"2024-09-10T19:00:13.473196+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:00:13.473196+0000","last_undegraded":"2024-09-10T19:00:13.473196+0000","last_fullsized":"2024-09-10T19:00:13.473196+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'22","last_scrub_stamp":"2024-09-10T18:58:47.475285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.475285+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041187790000000004,"objects_trimmed":0,"snaptrim_duration":0.001023022,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.b","version":"334'36","reported_seq":409,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473476+0000","last_change":"2024-09-10T19:00:09.152937+0000","last_active":"2024-09-10T19:00:13.473476+0000","last_peered":"2024-09-10T19:00:13.473476+0000","last_clean":"2024-09-10T19:00:13.473476+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:00:13.473476+0000","last_undegraded":"2024-09-10T19:00:13.473476+0000","last_fullsized":"2024-09-10T19:00:13.473476+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'35","last_scrub_stamp":"2024-09-10T18:57:48.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.733723+0000","objects_scrubbed":3,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047698860000000001,"objects_trimmed":0,"snaptrim_duration":0.00091768399999999999,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.a","version":"434'20","reported_seq":410,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412334+0000","last_change":"2024-09-10T19:00:09.243782+0000","last_active":"2024-09-10T19:00:13.412334+0000","last_peered":"2024-09-10T19:00:13.412334+0000","last_clean":"2024-09-10T19:00:13.412334+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:00:13.412334+0000","last_undegraded":"2024-09-10T19:00:13.412334+0000","last_fullsized":"2024-09-10T19:00:13.412334+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'17","last_scrub_stamp":"2024-09-10T18:58:37.063474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.063474+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003955554,"objects_trimmed":0,"snaptrim_duration":0.001329872,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":390,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412679+0000","last_change":"2024-09-10T19:00:09.244557+0000","last_active":"2024-09-10T19:00:13.412679+0000","last_peered":"2024-09-10T19:00:13.412679+0000","last_clean":"2024-09-10T19:00:13.412679+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:00:13.412679+0000","last_undegraded":"2024-09-10T19:00:13.412679+0000","last_fullsized":"2024-09-10T19:00:13.412679+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.924176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.924176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003004348,"objects_trimmed":0,"snaptrim_duration":0.0019161549999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.8","version":"234'51","reported_seq":432,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472077+0000","last_change":"2024-09-10T19:00:09.152501+0000","last_active":"2024-09-10T19:00:13.472077+0000","last_peered":"2024-09-10T19:00:13.472077+0000","last_clean":"2024-09-10T19:00:13.472077+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:00:13.472077+0000","last_undegraded":"2024-09-10T19:00:13.472077+0000","last_fullsized":"2024-09-10T19:00:13.472077+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T19:00:03.402967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.402967+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.326121+0000","scrub_duration":0.0043031060000000001,"objects_trimmed":0,"snaptrim_duration":0.00082492700000000004,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.7","version":"409'160","reported_seq":539,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413761+0000","last_change":"2024-09-10T19:00:09.246905+0000","last_active":"2024-09-10T19:00:13.413761+0000","last_peered":"2024-09-10T19:00:13.413761+0000","last_clean":"2024-09-10T19:00:13.413761+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:00:13.413761+0000","last_undegraded":"2024-09-10T19:00:13.413761+0000","last_fullsized":"2024-09-10T19:00:13.413761+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'159","last_scrub_stamp":"2024-09-10T18:58:53.953850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.953850+0000","objects_scrubbed":11,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:14.670681+0000","scrub_duration":0.0076573049999999997,"objects_trimmed":0,"snaptrim_duration":0.0013596680000000001,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.6","version":"388'99","reported_seq":526,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478477+0000","last_change":"2024-09-10T19:00:09.142059+0000","last_active":"2024-09-10T19:00:13.478477+0000","last_peered":"2024-09-10T19:00:13.478477+0000","last_clean":"2024-09-10T19:00:13.478477+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:00:13.478477+0000","last_undegraded":"2024-09-10T19:00:13.478477+0000","last_fullsized":"2024-09-10T19:00:13.478477+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.213299+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022641919,"objects_trimmed":0,"snaptrim_duration":0.00038340999999999999,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.5","version":"305'53","reported_seq":423,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474939+0000","last_change":"2024-09-10T19:00:09.144898+0000","last_active":"2024-09-10T19:00:13.474939+0000","last_peered":"2024-09-10T19:00:13.474939+0000","last_clean":"2024-09-10T19:00:13.474939+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:00:13.474939+0000","last_undegraded":"2024-09-10T19:00:13.474939+0000","last_fullsized":"2024-09-10T19:00:13.474939+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T18:58:44.435728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.435728+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045739450000000003,"objects_trimmed":0,"snaptrim_duration":0.00047679800000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.4","version":"257'107","reported_seq":481,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477982+0000","last_change":"2024-09-10T19:00:09.141463+0000","last_active":"2024-09-10T19:00:13.477982+0000","last_peered":"2024-09-10T19:00:13.477982+0000","last_clean":"2024-09-10T19:00:13.477982+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:00:13.477982+0000","last_undegraded":"2024-09-10T19:00:13.477982+0000","last_fullsized":"2024-09-10T19:00:13.477982+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T19:00:06.091203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:06.091203+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.467989+0000","scrub_duration":0.0051957840000000002,"objects_trimmed":0,"snaptrim_duration":0.00046374099999999999,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.3","version":"431'106","reported_seq":473,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346638+0000","last_change":"2024-09-10T19:00:09.146145+0000","last_active":"2024-09-10T19:00:13.346638+0000","last_peered":"2024-09-10T19:00:13.346638+0000","last_clean":"2024-09-10T19:00:13.346638+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:00:13.346638+0000","last_undegraded":"2024-09-10T19:00:13.346638+0000","last_fullsized":"2024-09-10T19:00:13.346638+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0088077559999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.2","version":"230'34","reported_seq":414,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:15.443550+0000","last_change":"2024-09-10T19:00:15.443447+0000","last_active":"2024-09-10T19:00:15.443550+0000","last_peered":"2024-09-10T19:00:15.443550+0000","last_clean":"2024-09-10T19:00:15.443550+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:00:15.443550+0000","last_undegraded":"2024-09-10T19:00:15.443550+0000","last_fullsized":"2024-09-10T19:00:15.443550+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T19:00:15.443270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.443270+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.357357+0000","scrub_duration":0.0044317829999999999,"objects_trimmed":0,"snaptrim_duration":0.001131515,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.1","version":"409'74","reported_seq":450,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.471369+0000","last_change":"2024-09-10T19:00:09.139795+0000","last_active":"2024-09-10T19:00:13.471369+0000","last_peered":"2024-09-10T19:00:13.471369+0000","last_clean":"2024-09-10T19:00:13.471369+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:00:13.471369+0000","last_undegraded":"2024-09-10T19:00:13.471369+0000","last_fullsized":"2024-09-10T19:00:13.471369+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'72","last_scrub_stamp":"2024-09-10T18:58:32.368604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.368604+0000","objects_scrubbed":7,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042920229999999998,"objects_trimmed":0,"snaptrim_duration":0.00038332600000000002,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"64.0","version":"246'102","reported_seq":482,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:14.192634+0000","last_change":"2024-09-10T19:00:14.192590+0000","last_active":"2024-09-10T19:00:14.192634+0000","last_peered":"2024-09-10T19:00:14.192634+0000","last_clean":"2024-09-10T19:00:14.192634+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:00:14.192634+0000","last_undegraded":"2024-09-10T19:00:14.192634+0000","last_fullsized":"2024-09-10T19:00:14.192634+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T19:00:14.192507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.192507+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.168955+0000","scrub_duration":0.0088236210000000002,"objects_trimmed":0,"snaptrim_duration":0.0015602579999999999,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345875+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T19:00:13.345875+0000","last_peered":"2024-09-10T19:00:13.345875+0000","last_clean":"2024-09-10T19:00:13.345875+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:00:13.345875+0000","last_undegraded":"2024-09-10T19:00:13.345875+0000","last_fullsized":"2024-09-10T19:00:13.345875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":435,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.351387+0000","last_change":"2024-09-10T18:59:59.106249+0000","last_active":"2024-09-10T19:00:13.351387+0000","last_peered":"2024-09-10T19:00:13.351387+0000","last_clean":"2024-09-10T19:00:13.351387+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:00:13.351387+0000","last_undegraded":"2024-09-10T19:00:13.351387+0000","last_fullsized":"2024-09-10T19:00:13.351387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.106063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.106063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.167241+0000","scrub_duration":0.0034646490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":455,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349575+0000","last_change":"2024-09-10T19:00:10.940056+0000","last_active":"2024-09-10T19:00:13.349575+0000","last_peered":"2024-09-10T19:00:13.349575+0000","last_clean":"2024-09-10T19:00:13.349575+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:00:13.349575+0000","last_undegraded":"2024-09-10T19:00:13.349575+0000","last_fullsized":"2024-09-10T19:00:13.349575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:00:10.939877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.939877+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:33.784202+0000","scrub_duration":0.0051523760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349456+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T19:00:13.349456+0000","last_peered":"2024-09-10T19:00:13.349456+0000","last_clean":"2024-09-10T19:00:13.349456+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:00:13.349456+0000","last_undegraded":"2024-09-10T19:00:13.349456+0000","last_fullsized":"2024-09-10T19:00:13.349456+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":347,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470666+0000","last_change":"2024-09-10T18:59:53.417704+0000","last_active":"2024-09-10T19:00:13.470666+0000","last_peered":"2024-09-10T19:00:13.470666+0000","last_clean":"2024-09-10T19:00:13.470666+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:00:13.470666+0000","last_undegraded":"2024-09-10T19:00:13.470666+0000","last_fullsized":"2024-09-10T19:00:13.470666+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.417551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.417551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.108833+0000","scrub_duration":0.0034045260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":437,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349983+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T19:00:13.349983+0000","last_peered":"2024-09-10T19:00:13.349983+0000","last_clean":"2024-09-10T19:00:13.349983+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:00:13.349983+0000","last_undegraded":"2024-09-10T19:00:13.349983+0000","last_fullsized":"2024-09-10T19:00:13.349983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":464,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350807+0000","last_change":"2024-09-10T19:00:03.037143+0000","last_active":"2024-09-10T19:00:13.350807+0000","last_peered":"2024-09-10T19:00:13.350807+0000","last_clean":"2024-09-10T19:00:13.350807+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:00:13.350807+0000","last_undegraded":"2024-09-10T19:00:13.350807+0000","last_fullsized":"2024-09-10T19:00:13.350807+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:03.036974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.036974+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:32.356994+0000","scrub_duration":0.0043269010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":442,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336448+0000","last_change":"2024-09-10T19:00:01.347416+0000","last_active":"2024-09-10T19:00:13.336448+0000","last_peered":"2024-09-10T19:00:13.336448+0000","last_clean":"2024-09-10T19:00:13.336448+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:00:13.336448+0000","last_undegraded":"2024-09-10T19:00:13.336448+0000","last_fullsized":"2024-09-10T19:00:13.336448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:01.347241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.347241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.701874+0000","scrub_duration":0.003764398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336914+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T19:00:13.336914+0000","last_peered":"2024-09-10T19:00:13.336914+0000","last_clean":"2024-09-10T19:00:13.336914+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T19:00:13.336914+0000","last_undegraded":"2024-09-10T19:00:13.336914+0000","last_fullsized":"2024-09-10T19:00:13.336914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345060+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T19:00:13.345060+0000","last_peered":"2024-09-10T19:00:13.345060+0000","last_clean":"2024-09-10T19:00:13.345060+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:00:13.345060+0000","last_undegraded":"2024-09-10T19:00:13.345060+0000","last_fullsized":"2024-09-10T19:00:13.345060+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478036+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T19:00:13.478036+0000","last_peered":"2024-09-10T19:00:13.478036+0000","last_clean":"2024-09-10T19:00:13.478036+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:00:13.478036+0000","last_undegraded":"2024-09-10T19:00:13.478036+0000","last_fullsized":"2024-09-10T19:00:13.478036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":478,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473429+0000","last_change":"2024-09-10T19:00:09.145311+0000","last_active":"2024-09-10T19:00:13.473429+0000","last_peered":"2024-09-10T19:00:13.473429+0000","last_clean":"2024-09-10T19:00:13.473429+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:00:13.473429+0000","last_undegraded":"2024-09-10T19:00:13.473429+0000","last_fullsized":"2024-09-10T19:00:13.473429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:09.145229+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.145229+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:21.374619+0000","scrub_duration":0.0090468159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":432,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337588+0000","last_change":"2024-09-10T18:58:53.495436+0000","last_active":"2024-09-10T19:00:13.337588+0000","last_peered":"2024-09-10T19:00:13.337588+0000","last_clean":"2024-09-10T19:00:13.337588+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T19:00:13.337588+0000","last_undegraded":"2024-09-10T19:00:13.337588+0000","last_fullsized":"2024-09-10T19:00:13.337588+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.495273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.495273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024582229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.471026+0000","last_change":"2024-09-10T19:00:03.208547+0000","last_active":"2024-09-10T19:00:13.471026+0000","last_peered":"2024-09-10T19:00:13.471026+0000","last_clean":"2024-09-10T19:00:13.471026+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:00:13.471026+0000","last_undegraded":"2024-09-10T19:00:13.471026+0000","last_fullsized":"2024-09-10T19:00:13.471026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.208468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.208468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.413505+0000","scrub_duration":0.0019202379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473972+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T19:00:13.473972+0000","last_peered":"2024-09-10T19:00:13.473972+0000","last_clean":"2024-09-10T19:00:13.473972+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:00:13.473972+0000","last_undegraded":"2024-09-10T19:00:13.473972+0000","last_fullsized":"2024-09-10T19:00:13.473972+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":436,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478302+0000","last_change":"2024-09-10T18:59:03.215137+0000","last_active":"2024-09-10T19:00:13.478302+0000","last_peered":"2024-09-10T19:00:13.478302+0000","last_clean":"2024-09-10T19:00:13.478302+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:00:13.478302+0000","last_undegraded":"2024-09-10T19:00:13.478302+0000","last_fullsized":"2024-09-10T19:00:13.478302+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.215039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.215039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:32.858523+0000","scrub_duration":0.0064108910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":475,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474718+0000","last_change":"2024-09-10T18:59:05.516222+0000","last_active":"2024-09-10T19:00:13.474718+0000","last_peered":"2024-09-10T19:00:13.474718+0000","last_clean":"2024-09-10T19:00:13.474718+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:00:13.474718+0000","last_undegraded":"2024-09-10T19:00:13.474718+0000","last_fullsized":"2024-09-10T19:00:13.474718+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:59:05.516062+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:05.516062+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:30.578286+0000","scrub_duration":0.0058657070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":432,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337159+0000","last_change":"2024-09-10T18:57:53.718511+0000","last_active":"2024-09-10T19:00:13.337159+0000","last_peered":"2024-09-10T19:00:13.337159+0000","last_clean":"2024-09-10T19:00:13.337159+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T19:00:13.337159+0000","last_undegraded":"2024-09-10T19:00:13.337159+0000","last_fullsized":"2024-09-10T19:00:13.337159+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.718349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.718349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035135330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":429,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343509+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T19:00:13.343509+0000","last_peered":"2024-09-10T19:00:13.343509+0000","last_clean":"2024-09-10T19:00:13.343509+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:00:13.343509+0000","last_undegraded":"2024-09-10T19:00:13.343509+0000","last_fullsized":"2024-09-10T19:00:13.343509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":352,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:14.868614+0000","last_change":"2024-09-10T19:00:14.868536+0000","last_active":"2024-09-10T19:00:14.868614+0000","last_peered":"2024-09-10T19:00:14.868614+0000","last_clean":"2024-09-10T19:00:14.868614+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:00:14.868614+0000","last_undegraded":"2024-09-10T19:00:14.868614+0000","last_fullsized":"2024-09-10T19:00:14.868614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.868377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.868377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.646932+0000","scrub_duration":0.0036007919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":486,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472659+0000","last_change":"2024-09-10T18:57:55.387991+0000","last_active":"2024-09-10T19:00:13.472659+0000","last_peered":"2024-09-10T19:00:13.472659+0000","last_clean":"2024-09-10T19:00:13.472659+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:00:13.472659+0000","last_undegraded":"2024-09-10T19:00:13.472659+0000","last_fullsized":"2024-09-10T19:00:13.472659+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:55.387889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.387889+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094746150000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477828+0000","last_change":"2024-09-10T18:59:57.030876+0000","last_active":"2024-09-10T19:00:13.477828+0000","last_peered":"2024-09-10T19:00:13.477828+0000","last_clean":"2024-09-10T19:00:13.477828+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:00:13.477828+0000","last_undegraded":"2024-09-10T19:00:13.477828+0000","last_fullsized":"2024-09-10T19:00:13.477828+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.030798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.030798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.038402+0000","scrub_duration":0.0020075510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":574,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339114+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T19:00:13.339114+0000","last_peered":"2024-09-10T19:00:13.339114+0000","last_clean":"2024-09-10T19:00:13.339114+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T19:00:13.339114+0000","last_undegraded":"2024-09-10T19:00:13.339114+0000","last_fullsized":"2024-09-10T19:00:13.339114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":439,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.414164+0000","last_change":"2024-09-10T18:59:00.907296+0000","last_active":"2024-09-10T19:00:13.414164+0000","last_peered":"2024-09-10T19:00:13.414164+0000","last_clean":"2024-09-10T19:00:13.414164+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:00:13.414164+0000","last_undegraded":"2024-09-10T19:00:13.414164+0000","last_fullsized":"2024-09-10T19:00:13.414164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.907056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:29.861464+0000","scrub_duration":0.0036581080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477163+0000","last_change":"2024-09-10T18:59:59.486802+0000","last_active":"2024-09-10T19:00:13.477163+0000","last_peered":"2024-09-10T19:00:13.477163+0000","last_clean":"2024-09-10T19:00:13.477163+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:00:13.477163+0000","last_undegraded":"2024-09-10T19:00:13.477163+0000","last_fullsized":"2024-09-10T19:00:13.477163+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.486653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.486653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:19.115655+0000","scrub_duration":0.003394902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":464,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346780+0000","last_change":"2024-09-10T18:59:57.094883+0000","last_active":"2024-09-10T19:00:13.346780+0000","last_peered":"2024-09-10T19:00:13.346780+0000","last_clean":"2024-09-10T19:00:13.346780+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:00:13.346780+0000","last_undegraded":"2024-09-10T19:00:13.346780+0000","last_fullsized":"2024-09-10T19:00:13.346780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T18:59:57.094721+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.094721+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.024150+0000","scrub_duration":0.0045594349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":444,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473665+0000","last_change":"2024-09-10T18:58:52.884147+0000","last_active":"2024-09-10T19:00:13.473665+0000","last_peered":"2024-09-10T19:00:13.473665+0000","last_clean":"2024-09-10T19:00:13.473665+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:00:13.473665+0000","last_undegraded":"2024-09-10T19:00:13.473665+0000","last_fullsized":"2024-09-10T19:00:13.473665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.884077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032688370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":435,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.338348+0000","last_change":"2024-09-10T19:00:02.315530+0000","last_active":"2024-09-10T19:00:13.338348+0000","last_peered":"2024-09-10T19:00:13.338348+0000","last_clean":"2024-09-10T19:00:13.338348+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T19:00:13.338348+0000","last_undegraded":"2024-09-10T19:00:13.338348+0000","last_fullsized":"2024-09-10T19:00:13.338348+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339184+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T19:00:13.339184+0000","last_peered":"2024-09-10T19:00:13.339184+0000","last_clean":"2024-09-10T19:00:13.339184+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T19:00:13.339184+0000","last_undegraded":"2024-09-10T19:00:13.339184+0000","last_fullsized":"2024-09-10T19:00:13.339184+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475175+0000","last_change":"2024-09-10T19:00:07.323465+0000","last_active":"2024-09-10T19:00:13.475175+0000","last_peered":"2024-09-10T19:00:13.475175+0000","last_clean":"2024-09-10T19:00:13.475175+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:00:13.475175+0000","last_undegraded":"2024-09-10T19:00:13.475175+0000","last_fullsized":"2024-09-10T19:00:13.475175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:07.323364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:07.323364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.180738+0000","scrub_duration":0.0028330460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":478,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.344768+0000","last_change":"2024-09-10T19:00:12.906006+0000","last_active":"2024-09-10T19:00:13.344768+0000","last_peered":"2024-09-10T19:00:13.344768+0000","last_clean":"2024-09-10T19:00:13.344768+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:00:13.344768+0000","last_undegraded":"2024-09-10T19:00:13.344768+0000","last_fullsized":"2024-09-10T19:00:13.344768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:00:12.905889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.905889+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:42.533979+0000","scrub_duration":0.003872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":434,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473854+0000","last_change":"2024-09-10T18:58:51.907906+0000","last_active":"2024-09-10T19:00:13.473854+0000","last_peered":"2024-09-10T19:00:13.473854+0000","last_clean":"2024-09-10T19:00:13.473854+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:00:13.473854+0000","last_undegraded":"2024-09-10T19:00:13.473854+0000","last_fullsized":"2024-09-10T19:00:13.473854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:51.907749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:51.907749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036594549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":440,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.479515+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T19:00:13.479515+0000","last_peered":"2024-09-10T19:00:13.479515+0000","last_clean":"2024-09-10T19:00:13.479515+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:00:13.479515+0000","last_undegraded":"2024-09-10T19:00:13.479515+0000","last_fullsized":"2024-09-10T19:00:13.479515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":434,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473717+0000","last_change":"2024-09-10T18:59:57.328269+0000","last_active":"2024-09-10T19:00:13.473717+0000","last_peered":"2024-09-10T19:00:13.473717+0000","last_clean":"2024-09-10T19:00:13.473717+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:00:13.473717+0000","last_undegraded":"2024-09-10T19:00:13.473717+0000","last_fullsized":"2024-09-10T19:00:13.473717+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.328111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.328111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.067869+0000","scrub_duration":0.0032441140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":360,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:14.537996+0000","last_change":"2024-09-10T19:00:14.537922+0000","last_active":"2024-09-10T19:00:14.537996+0000","last_peered":"2024-09-10T19:00:14.537996+0000","last_clean":"2024-09-10T19:00:14.537996+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:00:14.537996+0000","last_undegraded":"2024-09-10T19:00:14.537996+0000","last_fullsized":"2024-09-10T19:00:14.537996+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.537765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.537765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:28.781636+0000","scrub_duration":0.0033849069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":481,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337203+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T19:00:13.337203+0000","last_peered":"2024-09-10T19:00:13.337203+0000","last_clean":"2024-09-10T19:00:13.337203+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:00:13.337203+0000","last_undegraded":"2024-09-10T19:00:13.337203+0000","last_fullsized":"2024-09-10T19:00:13.337203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:27.341520+0000","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477385+0000","last_change":"2024-09-10T18:58:32.226883+0000","last_active":"2024-09-10T19:00:13.477385+0000","last_peered":"2024-09-10T19:00:13.477385+0000","last_clean":"2024-09-10T19:00:13.477385+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:00:13.477385+0000","last_undegraded":"2024-09-10T19:00:13.477385+0000","last_fullsized":"2024-09-10T19:00:13.477385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.226771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.226771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024188028,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":584,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339336+0000","last_change":"2024-09-10T19:00:05.221386+0000","last_active":"2024-09-10T19:00:13.339336+0000","last_peered":"2024-09-10T19:00:13.339336+0000","last_clean":"2024-09-10T19:00:13.339336+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T19:00:13.339336+0000","last_undegraded":"2024-09-10T19:00:13.339336+0000","last_fullsized":"2024-09-10T19:00:13.339336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T19:00:05.221191+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.221191+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.966409+0000","scrub_duration":0.0039722840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":422,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350346+0000","last_change":"2024-09-10T18:56:35.994161+0000","last_active":"2024-09-10T19:00:13.350346+0000","last_peered":"2024-09-10T19:00:13.350346+0000","last_clean":"2024-09-10T19:00:13.350346+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:00:13.350346+0000","last_undegraded":"2024-09-10T19:00:13.350346+0000","last_fullsized":"2024-09-10T19:00:13.350346+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T18:56:35.993988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.993988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005131666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.479875+0000","last_change":"2024-09-10T18:58:47.871654+0000","last_active":"2024-09-10T19:00:13.479875+0000","last_peered":"2024-09-10T19:00:13.479875+0000","last_clean":"2024-09-10T19:00:13.479875+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:00:13.479875+0000","last_undegraded":"2024-09-10T19:00:13.479875+0000","last_fullsized":"2024-09-10T19:00:13.479875+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.871591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.871591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023117910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":459,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349364+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T19:00:13.349364+0000","last_peered":"2024-09-10T19:00:13.349364+0000","last_clean":"2024-09-10T19:00:13.349364+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:00:13.349364+0000","last_undegraded":"2024-09-10T19:00:13.349364+0000","last_fullsized":"2024-09-10T19:00:13.349364+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477509+0000","last_change":"2024-09-10T18:58:50.597419+0000","last_active":"2024-09-10T19:00:13.477509+0000","last_peered":"2024-09-10T19:00:13.477509+0000","last_clean":"2024-09-10T19:00:13.477509+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:00:13.477509+0000","last_undegraded":"2024-09-10T19:00:13.477509+0000","last_fullsized":"2024-09-10T19:00:13.477509+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.597321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.597321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021838199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":442,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336723+0000","last_change":"2024-09-10T19:00:10.266358+0000","last_active":"2024-09-10T19:00:13.336723+0000","last_peered":"2024-09-10T19:00:13.336723+0000","last_clean":"2024-09-10T19:00:13.336723+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T19:00:13.336723+0000","last_undegraded":"2024-09-10T19:00:13.336723+0000","last_fullsized":"2024-09-10T19:00:13.336723+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.266243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.266243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.094340+0000","scrub_duration":0.003123285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1073,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:14.135710+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T19:00:14.135710+0000","last_peered":"2024-09-10T19:00:14.135710+0000","last_clean":"2024-09-10T19:00:14.135710+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:00:14.135710+0000","last_undegraded":"2024-09-10T19:00:14.135710+0000","last_fullsized":"2024-09-10T19:00:14.135710+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":632,"num_read_kb":627,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":437,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477387+0000","last_change":"2024-09-10T19:00:04.106819+0000","last_active":"2024-09-10T19:00:13.477387+0000","last_peered":"2024-09-10T19:00:13.477387+0000","last_clean":"2024-09-10T19:00:13.477387+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:00:13.477387+0000","last_undegraded":"2024-09-10T19:00:13.477387+0000","last_fullsized":"2024-09-10T19:00:13.477387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.106660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.106660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.468919+0000","scrub_duration":0.0031927650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":361,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412508+0000","last_change":"2024-09-10T18:59:06.979721+0000","last_active":"2024-09-10T19:00:13.412508+0000","last_peered":"2024-09-10T19:00:13.412508+0000","last_clean":"2024-09-10T19:00:13.412508+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:00:13.412508+0000","last_undegraded":"2024-09-10T19:00:13.412508+0000","last_fullsized":"2024-09-10T19:00:13.412508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:06.979599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:06.979599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.070088+0000","scrub_duration":0.0031493580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":468,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349312+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T19:00:13.349312+0000","last_peered":"2024-09-10T19:00:13.349312+0000","last_clean":"2024-09-10T19:00:13.349312+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:00:13.349312+0000","last_undegraded":"2024-09-10T19:00:13.349312+0000","last_fullsized":"2024-09-10T19:00:13.349312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.664434+0000","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":444,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.480140+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T19:00:13.480140+0000","last_peered":"2024-09-10T19:00:13.480140+0000","last_clean":"2024-09-10T19:00:13.480140+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:00:13.480140+0000","last_undegraded":"2024-09-10T19:00:13.480140+0000","last_fullsized":"2024-09-10T19:00:13.480140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":564,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343899+0000","last_change":"2024-09-10T18:59:59.066338+0000","last_active":"2024-09-10T19:00:13.343899+0000","last_peered":"2024-09-10T19:00:13.343899+0000","last_clean":"2024-09-10T19:00:13.343899+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:00:13.343899+0000","last_undegraded":"2024-09-10T19:00:13.343899+0000","last_fullsized":"2024-09-10T19:00:13.343899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:59:59.066136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.066136+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.679135+0000","scrub_duration":0.0038648509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":449,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336592+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T19:00:13.336592+0000","last_peered":"2024-09-10T19:00:13.336592+0000","last_clean":"2024-09-10T19:00:13.336592+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:00:13.336592+0000","last_undegraded":"2024-09-10T19:00:13.336592+0000","last_fullsized":"2024-09-10T19:00:13.336592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.789834+0000","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337162+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T19:00:13.337162+0000","last_peered":"2024-09-10T19:00:13.337162+0000","last_clean":"2024-09-10T19:00:13.337162+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T19:00:13.337162+0000","last_undegraded":"2024-09-10T19:00:13.337162+0000","last_fullsized":"2024-09-10T19:00:13.337162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":342,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343874+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T19:00:13.343874+0000","last_peered":"2024-09-10T19:00:13.343874+0000","last_clean":"2024-09-10T19:00:13.343874+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:00:13.343874+0000","last_undegraded":"2024-09-10T19:00:13.343874+0000","last_fullsized":"2024-09-10T19:00:13.343874+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":651,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477883+0000","last_change":"2024-09-10T19:00:01.595171+0000","last_active":"2024-09-10T19:00:13.477883+0000","last_peered":"2024-09-10T19:00:13.477883+0000","last_clean":"2024-09-10T19:00:13.477883+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:00:13.477883+0000","last_undegraded":"2024-09-10T19:00:13.477883+0000","last_fullsized":"2024-09-10T19:00:13.477883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:00:01.595006+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.595006+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.951465+0000","scrub_duration":0.0053873050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478800+0000","last_change":"2024-09-10T18:58:50.209102+0000","last_active":"2024-09-10T19:00:13.478800+0000","last_peered":"2024-09-10T19:00:13.478800+0000","last_clean":"2024-09-10T19:00:13.478800+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:00:13.478800+0000","last_undegraded":"2024-09-10T19:00:13.478800+0000","last_fullsized":"2024-09-10T19:00:13.478800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.208993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.208993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003212126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":573,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346967+0000","last_change":"2024-09-10T18:59:56.107053+0000","last_active":"2024-09-10T19:00:13.346967+0000","last_peered":"2024-09-10T19:00:13.346967+0000","last_clean":"2024-09-10T19:00:13.346967+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:00:13.346967+0000","last_undegraded":"2024-09-10T19:00:13.346967+0000","last_fullsized":"2024-09-10T19:00:13.346967+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:59:56.106896+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:59:56.106896+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.084878+0000","scrub_duration":0.0036065950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":437,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472315+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T19:00:13.472315+0000","last_peered":"2024-09-10T19:00:13.472315+0000","last_clean":"2024-09-10T19:00:13.472315+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:00:13.472315+0000","last_undegraded":"2024-09-10T19:00:13.472315+0000","last_fullsized":"2024-09-10T19:00:13.472315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":428,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473469+0000","last_change":"2024-09-10T18:58:45.871468+0000","last_active":"2024-09-10T19:00:13.473469+0000","last_peered":"2024-09-10T19:00:13.473469+0000","last_clean":"2024-09-10T19:00:13.473469+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:00:13.473469+0000","last_undegraded":"2024-09-10T19:00:13.473469+0000","last_fullsized":"2024-09-10T19:00:13.473469+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:45.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:45.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033902770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":342,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347199+0000","last_change":"2024-09-10T18:57:39.153712+0000","last_active":"2024-09-10T19:00:13.347199+0000","last_peered":"2024-09-10T19:00:13.347199+0000","last_clean":"2024-09-10T19:00:13.347199+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:00:13.347199+0000","last_undegraded":"2024-09-10T19:00:13.347199+0000","last_fullsized":"2024-09-10T19:00:13.347199+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.153569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.153569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031939199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":468,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474784+0000","last_change":"2024-09-10T18:58:46.437660+0000","last_active":"2024-09-10T19:00:13.474784+0000","last_peered":"2024-09-10T19:00:13.474784+0000","last_clean":"2024-09-10T19:00:13.474784+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:00:13.474784+0000","last_undegraded":"2024-09-10T19:00:13.474784+0000","last_fullsized":"2024-09-10T19:00:13.474784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:46.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.437410+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054784669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":434,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.479312+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T19:00:13.479312+0000","last_peered":"2024-09-10T19:00:13.479312+0000","last_clean":"2024-09-10T19:00:13.479312+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:00:13.479312+0000","last_undegraded":"2024-09-10T19:00:13.479312+0000","last_fullsized":"2024-09-10T19:00:13.479312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473174+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T19:00:13.473174+0000","last_peered":"2024-09-10T19:00:13.473174+0000","last_clean":"2024-09-10T19:00:13.473174+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:00:13.473174+0000","last_undegraded":"2024-09-10T19:00:13.473174+0000","last_fullsized":"2024-09-10T19:00:13.473174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":429,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.344209+0000","last_change":"2024-09-10T19:00:00.056759+0000","last_active":"2024-09-10T19:00:13.344209+0000","last_peered":"2024-09-10T19:00:13.344209+0000","last_clean":"2024-09-10T19:00:13.344209+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:00:13.344209+0000","last_undegraded":"2024-09-10T19:00:13.344209+0000","last_fullsized":"2024-09-10T19:00:13.344209+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.056600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.056600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.057092+0000","scrub_duration":0.0028960119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":361,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412612+0000","last_change":"2024-09-10T18:59:58.063484+0000","last_active":"2024-09-10T19:00:13.412612+0000","last_peered":"2024-09-10T19:00:13.412612+0000","last_clean":"2024-09-10T19:00:13.412612+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:00:13.412612+0000","last_undegraded":"2024-09-10T19:00:13.412612+0000","last_fullsized":"2024-09-10T19:00:13.412612+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.063321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.063321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:58.931188+0000","scrub_duration":0.0034568020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":485,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.338036+0000","last_change":"2024-09-10T18:57:54.670140+0000","last_active":"2024-09-10T19:00:13.338036+0000","last_peered":"2024-09-10T19:00:13.338036+0000","last_clean":"2024-09-10T19:00:13.338036+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T19:00:13.338036+0000","last_undegraded":"2024-09-10T19:00:13.338036+0000","last_fullsized":"2024-09-10T19:00:13.338036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'19","last_scrub_stamp":"2024-09-10T18:57:54.669871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:54.669871+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044082740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472709+0000","last_change":"2024-09-10T18:58:52.447688+0000","last_active":"2024-09-10T19:00:13.472709+0000","last_peered":"2024-09-10T19:00:13.472709+0000","last_clean":"2024-09-10T19:00:13.472709+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:00:13.472709+0000","last_undegraded":"2024-09-10T19:00:13.472709+0000","last_fullsized":"2024-09-10T19:00:13.472709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.447547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.447547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.724802+0000","scrub_duration":0.0031696559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477201+0000","last_change":"2024-09-10T18:57:53.795372+0000","last_active":"2024-09-10T19:00:13.477201+0000","last_peered":"2024-09-10T19:00:13.477201+0000","last_clean":"2024-09-10T19:00:13.477201+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:00:13.477201+0000","last_undegraded":"2024-09-10T19:00:13.477201+0000","last_fullsized":"2024-09-10T19:00:13.477201+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:53.795221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.795221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033420210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349459+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T19:00:13.349459+0000","last_peered":"2024-09-10T19:00:13.349459+0000","last_clean":"2024-09-10T19:00:13.349459+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:00:13.349459+0000","last_undegraded":"2024-09-10T19:00:13.349459+0000","last_fullsized":"2024-09-10T19:00:13.349459+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348708+0000","last_change":"2024-09-10T18:56:34.214468+0000","last_active":"2024-09-10T19:00:13.348708+0000","last_peered":"2024-09-10T19:00:13.348708+0000","last_clean":"2024-09-10T19:00:13.348708+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:00:13.348708+0000","last_undegraded":"2024-09-10T19:00:13.348708+0000","last_fullsized":"2024-09-10T19:00:13.348708+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:34.214302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:34.214302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033637530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":500,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.488040+0000","last_change":"2024-09-10T19:00:13.487926+0000","last_active":"2024-09-10T19:00:13.488040+0000","last_peered":"2024-09-10T19:00:13.488040+0000","last_clean":"2024-09-10T19:00:13.488040+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:00:13.488040+0000","last_undegraded":"2024-09-10T19:00:13.488040+0000","last_fullsized":"2024-09-10T19:00:13.488040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.487819+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:18.610954+0000","scrub_duration":0.003630566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412216+0000","last_change":"2024-09-10T18:59:59.081414+0000","last_active":"2024-09-10T19:00:13.412216+0000","last_peered":"2024-09-10T19:00:13.412216+0000","last_clean":"2024-09-10T19:00:13.412216+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:00:13.412216+0000","last_undegraded":"2024-09-10T19:00:13.412216+0000","last_fullsized":"2024-09-10T19:00:13.412216+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.081277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.081277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.759642+0000","scrub_duration":0.002648634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.479628+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T19:00:13.479628+0000","last_peered":"2024-09-10T19:00:13.479628+0000","last_clean":"2024-09-10T19:00:13.479628+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:00:13.479628+0000","last_undegraded":"2024-09-10T19:00:13.479628+0000","last_fullsized":"2024-09-10T19:00:13.479628+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478256+0000","last_change":"2024-09-10T18:57:51.412856+0000","last_active":"2024-09-10T19:00:13.478256+0000","last_peered":"2024-09-10T19:00:13.478256+0000","last_clean":"2024-09-10T19:00:13.478256+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:00:13.478256+0000","last_undegraded":"2024-09-10T19:00:13.478256+0000","last_fullsized":"2024-09-10T19:00:13.478256+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:51.412699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:51.412699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003181985,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477084+0000","last_change":"2024-09-10T18:59:02.713009+0000","last_active":"2024-09-10T19:00:13.477084+0000","last_peered":"2024-09-10T19:00:13.477084+0000","last_clean":"2024-09-10T19:00:13.477084+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:00:13.477084+0000","last_undegraded":"2024-09-10T19:00:13.477084+0000","last_fullsized":"2024-09-10T19:00:13.477084+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.712839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.712839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035464870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":475,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345352+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T19:00:13.345352+0000","last_peered":"2024-09-10T19:00:13.345352+0000","last_clean":"2024-09-10T19:00:13.345352+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:00:13.345352+0000","last_undegraded":"2024-09-10T19:00:13.345352+0000","last_fullsized":"2024-09-10T19:00:13.345352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:20.541418+0000","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477015+0000","last_change":"2024-09-10T18:58:44.193043+0000","last_active":"2024-09-10T19:00:13.477015+0000","last_peered":"2024-09-10T19:00:13.477015+0000","last_clean":"2024-09-10T19:00:13.477015+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:00:13.477015+0000","last_undegraded":"2024-09-10T19:00:13.477015+0000","last_fullsized":"2024-09-10T19:00:13.477015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.192853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.192853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038691179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":431,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349572+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T19:00:13.349572+0000","last_peered":"2024-09-10T19:00:13.349572+0000","last_clean":"2024-09-10T19:00:13.349572+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:00:13.349572+0000","last_undegraded":"2024-09-10T19:00:13.349572+0000","last_fullsized":"2024-09-10T19:00:13.349572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.522110+0000","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":438,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475063+0000","last_change":"2024-09-10T19:00:03.612383+0000","last_active":"2024-09-10T19:00:13.475063+0000","last_peered":"2024-09-10T19:00:13.475063+0000","last_clean":"2024-09-10T19:00:13.475063+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:00:13.475063+0000","last_undegraded":"2024-09-10T19:00:13.475063+0000","last_fullsized":"2024-09-10T19:00:13.475063+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.612225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.557620+0000","scrub_duration":0.0033384840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349777+0000","last_change":"2024-09-10T18:56:31.252230+0000","last_active":"2024-09-10T19:00:13.349777+0000","last_peered":"2024-09-10T19:00:13.349777+0000","last_clean":"2024-09-10T19:00:13.349777+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:00:13.349777+0000","last_undegraded":"2024-09-10T19:00:13.349777+0000","last_fullsized":"2024-09-10T19:00:13.349777+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:31.251962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:31.251962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039444600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":460,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346252+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T19:00:13.346252+0000","last_peered":"2024-09-10T19:00:13.346252+0000","last_clean":"2024-09-10T19:00:13.346252+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:00:13.346252+0000","last_undegraded":"2024-09-10T19:00:13.346252+0000","last_fullsized":"2024-09-10T19:00:13.346252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":440,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:15.230953+0000","last_change":"2024-09-10T19:00:15.230885+0000","last_active":"2024-09-10T19:00:15.230953+0000","last_peered":"2024-09-10T19:00:15.230953+0000","last_clean":"2024-09-10T19:00:15.230953+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:00:15.230953+0000","last_undegraded":"2024-09-10T19:00:15.230953+0000","last_fullsized":"2024-09-10T19:00:15.230953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.230765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:38.515881+0000","scrub_duration":0.002962732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":491,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477077+0000","last_change":"2024-09-10T19:00:12.005658+0000","last_active":"2024-09-10T19:00:13.477077+0000","last_peered":"2024-09-10T19:00:13.477077+0000","last_clean":"2024-09-10T19:00:13.477077+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T19:00:13.477077+0000","last_undegraded":"2024-09-10T19:00:13.477077+0000","last_fullsized":"2024-09-10T19:00:13.477077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.658532+0000","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":429,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477685+0000","last_change":"2024-09-10T18:58:47.555840+0000","last_active":"2024-09-10T19:00:13.477685+0000","last_peered":"2024-09-10T19:00:13.477685+0000","last_clean":"2024-09-10T19:00:13.477685+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:00:13.477685+0000","last_undegraded":"2024-09-10T19:00:13.477685+0000","last_fullsized":"2024-09-10T19:00:13.477685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.555637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.555637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032502030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":424,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473998+0000","last_change":"2024-09-10T18:58:59.454163+0000","last_active":"2024-09-10T19:00:13.473998+0000","last_peered":"2024-09-10T19:00:13.473998+0000","last_clean":"2024-09-10T19:00:13.473998+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:00:13.473998+0000","last_undegraded":"2024-09-10T19:00:13.473998+0000","last_fullsized":"2024-09-10T19:00:13.473998+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.565579+0000","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478872+0000","last_change":"2024-09-10T18:58:54.237739+0000","last_active":"2024-09-10T19:00:13.478872+0000","last_peered":"2024-09-10T19:00:13.478872+0000","last_clean":"2024-09-10T19:00:13.478872+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:00:13.478872+0000","last_undegraded":"2024-09-10T19:00:13.478872+0000","last_fullsized":"2024-09-10T19:00:13.478872+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.237585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.237585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:15.680207+0000","scrub_duration":0.0092150240000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":488,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478608+0000","last_change":"2024-09-10T18:58:52.253613+0000","last_active":"2024-09-10T19:00:13.478608+0000","last_peered":"2024-09-10T19:00:13.478608+0000","last_clean":"2024-09-10T19:00:13.478608+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:00:13.478608+0000","last_undegraded":"2024-09-10T19:00:13.478608+0000","last_fullsized":"2024-09-10T19:00:13.478608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'25","last_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.253376+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011205784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":455,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413849+0000","last_change":"2024-09-10T19:00:10.225407+0000","last_active":"2024-09-10T19:00:13.413849+0000","last_peered":"2024-09-10T19:00:13.413849+0000","last_clean":"2024-09-10T19:00:13.413849+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:00:13.413849+0000","last_undegraded":"2024-09-10T19:00:13.413849+0000","last_fullsized":"2024-09-10T19:00:13.413849+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.225321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.225321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.536640+0000","scrub_duration":0.0027779139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":429,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472852+0000","last_change":"2024-09-10T18:58:42.486651+0000","last_active":"2024-09-10T19:00:13.472852+0000","last_peered":"2024-09-10T19:00:13.472852+0000","last_clean":"2024-09-10T19:00:13.472852+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:00:13.472852+0000","last_undegraded":"2024-09-10T19:00:13.472852+0000","last_fullsized":"2024-09-10T19:00:13.472852+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:42.486498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:42.486498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032582879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477425+0000","last_change":"2024-09-10T18:58:55.261885+0000","last_active":"2024-09-10T19:00:13.477425+0000","last_peered":"2024-09-10T19:00:13.477425+0000","last_clean":"2024-09-10T19:00:13.477425+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:00:13.477425+0000","last_undegraded":"2024-09-10T19:00:13.477425+0000","last_fullsized":"2024-09-10T19:00:13.477425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.261635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.261635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.433144+0000","scrub_duration":0.0046216190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336407+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T19:00:13.336407+0000","last_peered":"2024-09-10T19:00:13.336407+0000","last_clean":"2024-09-10T19:00:13.336407+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:00:13.336407+0000","last_undegraded":"2024-09-10T19:00:13.336407+0000","last_fullsized":"2024-09-10T19:00:13.336407+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":463,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473652+0000","last_change":"2024-09-10T18:59:01.489222+0000","last_active":"2024-09-10T19:00:13.473652+0000","last_peered":"2024-09-10T19:00:13.473652+0000","last_clean":"2024-09-10T19:00:13.473652+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:00:13.473652+0000","last_undegraded":"2024-09-10T19:00:13.473652+0000","last_fullsized":"2024-09-10T19:00:13.473652+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:59:01.489057+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.489057+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:21.754146+0000","scrub_duration":0.0045114960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":434,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477755+0000","last_change":"2024-09-10T18:58:38.864525+0000","last_active":"2024-09-10T19:00:13.477755+0000","last_peered":"2024-09-10T19:00:13.477755+0000","last_clean":"2024-09-10T19:00:13.477755+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:00:13.477755+0000","last_undegraded":"2024-09-10T19:00:13.477755+0000","last_fullsized":"2024-09-10T19:00:13.477755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.864360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.864360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346628+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T19:00:13.346628+0000","last_peered":"2024-09-10T19:00:13.346628+0000","last_clean":"2024-09-10T19:00:13.346628+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:00:13.346628+0000","last_undegraded":"2024-09-10T19:00:13.346628+0000","last_fullsized":"2024-09-10T19:00:13.346628+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":419,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.345154+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T19:00:13.345154+0000","last_peered":"2024-09-10T19:00:13.345154+0000","last_clean":"2024-09-10T19:00:13.345154+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:00:13.345154+0000","last_undegraded":"2024-09-10T19:00:13.345154+0000","last_fullsized":"2024-09-10T19:00:13.345154+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346943+0000","last_change":"2024-09-10T18:56:35.182748+0000","last_active":"2024-09-10T19:00:13.346943+0000","last_peered":"2024-09-10T19:00:13.346943+0000","last_clean":"2024-09-10T19:00:13.346943+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:00:13.346943+0000","last_undegraded":"2024-09-10T19:00:13.346943+0000","last_fullsized":"2024-09-10T19:00:13.346943+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:35.182606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:56:35.182606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033736399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":481,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478684+0000","last_change":"2024-09-10T18:58:48.217515+0000","last_active":"2024-09-10T19:00:13.478684+0000","last_peered":"2024-09-10T19:00:13.478684+0000","last_clean":"2024-09-10T19:00:13.478684+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:00:13.478684+0000","last_undegraded":"2024-09-10T19:00:13.478684+0000","last_fullsized":"2024-09-10T19:00:13.478684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'22","last_scrub_stamp":"2024-09-10T18:58:48.217409+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.217409+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034746339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":467,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.414161+0000","last_change":"2024-09-10T19:00:06.139605+0000","last_active":"2024-09-10T19:00:13.414161+0000","last_peered":"2024-09-10T19:00:13.414161+0000","last_clean":"2024-09-10T19:00:13.414161+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:00:13.414161+0000","last_undegraded":"2024-09-10T19:00:13.414161+0000","last_fullsized":"2024-09-10T19:00:13.414161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:00:06.139460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:06.139460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:08.785287+0000","scrub_duration":0.0034469779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":436,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470180+0000","last_change":"2024-09-10T19:00:00.313745+0000","last_active":"2024-09-10T19:00:13.470180+0000","last_peered":"2024-09-10T19:00:13.470180+0000","last_clean":"2024-09-10T19:00:13.470180+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:00:13.470180+0000","last_undegraded":"2024-09-10T19:00:13.470180+0000","last_fullsized":"2024-09-10T19:00:13.470180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.313582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.313582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.500345+0000","scrub_duration":0.0034259580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":424,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472383+0000","last_change":"2024-09-10T18:58:37.374134+0000","last_active":"2024-09-10T19:00:13.472383+0000","last_peered":"2024-09-10T19:00:13.472383+0000","last_clean":"2024-09-10T19:00:13.472383+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:00:13.472383+0000","last_undegraded":"2024-09-10T19:00:13.472383+0000","last_fullsized":"2024-09-10T19:00:13.472383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.374014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.374014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023318560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.336234+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T19:00:13.336234+0000","last_peered":"2024-09-10T19:00:13.336234+0000","last_clean":"2024-09-10T19:00:13.336234+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T19:00:13.336234+0000","last_undegraded":"2024-09-10T19:00:13.336234+0000","last_fullsized":"2024-09-10T19:00:13.336234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":462,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478863+0000","last_change":"2024-09-10T18:58:47.178004+0000","last_active":"2024-09-10T19:00:13.478863+0000","last_peered":"2024-09-10T19:00:13.478863+0000","last_clean":"2024-09-10T19:00:13.478863+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:00:13.478863+0000","last_undegraded":"2024-09-10T19:00:13.478863+0000","last_fullsized":"2024-09-10T19:00:13.478863+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:47.177751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.177751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059440730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":485,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.340007+0000","last_change":"2024-09-10T18:59:59.421165+0000","last_active":"2024-09-10T19:00:13.340007+0000","last_peered":"2024-09-10T19:00:13.340007+0000","last_clean":"2024-09-10T19:00:13.340007+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:00:13.340007+0000","last_undegraded":"2024-09-10T19:00:13.340007+0000","last_fullsized":"2024-09-10T19:00:13.340007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:59:59.420998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.420998+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.365936+0000","scrub_duration":0.0045326000000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":583,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:15.117673+0000","last_change":"2024-09-10T19:00:15.117567+0000","last_active":"2024-09-10T19:00:15.117673+0000","last_peered":"2024-09-10T19:00:15.117673+0000","last_clean":"2024-09-10T19:00:15.117673+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T19:00:15.117673+0000","last_undegraded":"2024-09-10T19:00:15.117673+0000","last_fullsized":"2024-09-10T19:00:15.117673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:15.117380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.117380+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.034441+0000","scrub_duration":0.004280982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346084+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T19:00:13.346084+0000","last_peered":"2024-09-10T19:00:13.346084+0000","last_clean":"2024-09-10T19:00:13.346084+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:00:13.346084+0000","last_undegraded":"2024-09-10T19:00:13.346084+0000","last_fullsized":"2024-09-10T19:00:13.346084+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":363,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473610+0000","last_change":"2024-09-10T18:59:55.520903+0000","last_active":"2024-09-10T19:00:13.473610+0000","last_peered":"2024-09-10T19:00:13.473610+0000","last_clean":"2024-09-10T19:00:13.473610+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:00:13.473610+0000","last_undegraded":"2024-09-10T19:00:13.473610+0000","last_fullsized":"2024-09-10T19:00:13.473610+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:59:55.520655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.520655+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.622877+0000","scrub_duration":0.0073248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":472,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477129+0000","last_change":"2024-09-10T19:00:02.055337+0000","last_active":"2024-09-10T19:00:13.477129+0000","last_peered":"2024-09-10T19:00:13.477129+0000","last_clean":"2024-09-10T19:00:13.477129+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:00:13.477129+0000","last_undegraded":"2024-09-10T19:00:13.477129+0000","last_fullsized":"2024-09-10T19:00:13.477129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:02.055169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.055169+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:18.062663+0000","scrub_duration":0.0050398609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":445,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412426+0000","last_change":"2024-09-10T18:58:46.953749+0000","last_active":"2024-09-10T19:00:13.412426+0000","last_peered":"2024-09-10T19:00:13.412426+0000","last_clean":"2024-09-10T19:00:13.412426+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:00:13.412426+0000","last_undegraded":"2024-09-10T19:00:13.412426+0000","last_fullsized":"2024-09-10T19:00:13.412426+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.953608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.953608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028191259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":431,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349140+0000","last_change":"2024-09-10T19:00:12.129140+0000","last_active":"2024-09-10T19:00:13.349140+0000","last_peered":"2024-09-10T19:00:13.349140+0000","last_clean":"2024-09-10T19:00:13.349140+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:00:13.349140+0000","last_undegraded":"2024-09-10T19:00:13.349140+0000","last_fullsized":"2024-09-10T19:00:13.349140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:12.128979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.128979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.750053+0000","scrub_duration":0.0031738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":439,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.412800+0000","last_change":"2024-09-10T18:59:53.990233+0000","last_active":"2024-09-10T19:00:13.412800+0000","last_peered":"2024-09-10T19:00:13.412800+0000","last_clean":"2024-09-10T19:00:13.412800+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:00:13.412800+0000","last_undegraded":"2024-09-10T19:00:13.412800+0000","last_fullsized":"2024-09-10T19:00:13.412800+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.990034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.990034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:21.398651+0000","scrub_duration":0.00353419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339622+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T19:00:13.339622+0000","last_peered":"2024-09-10T19:00:13.339622+0000","last_clean":"2024-09-10T19:00:13.339622+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:00:13.339622+0000","last_undegraded":"2024-09-10T19:00:13.339622+0000","last_fullsized":"2024-09-10T19:00:13.339622+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":445,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.471664+0000","last_change":"2024-09-10T18:58:40.428324+0000","last_active":"2024-09-10T19:00:13.471664+0000","last_peered":"2024-09-10T19:00:13.471664+0000","last_clean":"2024-09-10T19:00:13.471664+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:00:13.471664+0000","last_undegraded":"2024-09-10T19:00:13.471664+0000","last_fullsized":"2024-09-10T19:00:13.471664+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'6","last_scrub_stamp":"2024-09-10T18:58:40.428157+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.428157+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003853726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":443,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474270+0000","last_change":"2024-09-10T19:00:10.348880+0000","last_active":"2024-09-10T19:00:13.474270+0000","last_peered":"2024-09-10T19:00:13.474270+0000","last_clean":"2024-09-10T19:00:13.474270+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:00:13.474270+0000","last_undegraded":"2024-09-10T19:00:13.474270+0000","last_fullsized":"2024-09-10T19:00:13.474270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.348709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.348709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.066540+0000","scrub_duration":0.0033057529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":423,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348609+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T19:00:13.348609+0000","last_peered":"2024-09-10T19:00:13.348609+0000","last_clean":"2024-09-10T19:00:13.348609+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:00:13.348609+0000","last_undegraded":"2024-09-10T19:00:13.348609+0000","last_fullsized":"2024-09-10T19:00:13.348609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478147+0000","last_change":"2024-09-10T18:57:52.371666+0000","last_active":"2024-09-10T19:00:13.478147+0000","last_peered":"2024-09-10T19:00:13.478147+0000","last_clean":"2024-09-10T19:00:13.478147+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:00:13.478147+0000","last_undegraded":"2024-09-10T19:00:13.478147+0000","last_fullsized":"2024-09-10T19:00:13.478147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.371566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.371566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020730700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477009+0000","last_change":"2024-09-10T18:59:01.663804+0000","last_active":"2024-09-10T19:00:13.477009+0000","last_peered":"2024-09-10T19:00:13.477009+0000","last_clean":"2024-09-10T19:00:13.477009+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:00:13.477009+0000","last_undegraded":"2024-09-10T19:00:13.477009+0000","last_fullsized":"2024-09-10T19:00:13.477009+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.663643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.663643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:23.528103+0000","scrub_duration":0.0029754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":467,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477774+0000","last_change":"2024-09-10T18:58:56.293280+0000","last_active":"2024-09-10T19:00:13.477774+0000","last_peered":"2024-09-10T19:00:13.477774+0000","last_clean":"2024-09-10T19:00:13.477774+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:00:13.477774+0000","last_undegraded":"2024-09-10T19:00:13.477774+0000","last_fullsized":"2024-09-10T19:00:13.477774+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:58:56.293027+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.293027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069050090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472808+0000","last_change":"2024-09-10T19:00:05.196515+0000","last_active":"2024-09-10T19:00:13.472808+0000","last_peered":"2024-09-10T19:00:13.472808+0000","last_clean":"2024-09-10T19:00:13.472808+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:00:13.472808+0000","last_undegraded":"2024-09-10T19:00:13.472808+0000","last_fullsized":"2024-09-10T19:00:13.472808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:05.196346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.196346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.115412+0000","scrub_duration":0.0033251359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":567,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350635+0000","last_change":"2024-09-10T18:57:41.203238+0000","last_active":"2024-09-10T19:00:13.350635+0000","last_peered":"2024-09-10T19:00:13.350635+0000","last_clean":"2024-09-10T19:00:13.350635+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:00:13.350635+0000","last_undegraded":"2024-09-10T19:00:13.350635+0000","last_fullsized":"2024-09-10T19:00:13.350635+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:57:41.203090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:41.203090+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030866539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":419,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348885+0000","last_change":"2024-09-10T18:56:25.186030+0000","last_active":"2024-09-10T19:00:13.348885+0000","last_peered":"2024-09-10T19:00:13.348885+0000","last_clean":"2024-09-10T19:00:13.348885+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:00:13.348885+0000","last_undegraded":"2024-09-10T19:00:13.348885+0000","last_fullsized":"2024-09-10T19:00:13.348885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:25.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T18:56:25.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027064400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349009+0000","last_change":"2024-09-10T18:56:29.227362+0000","last_active":"2024-09-10T19:00:13.349009+0000","last_peered":"2024-09-10T19:00:13.349009+0000","last_clean":"2024-09-10T19:00:13.349009+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:00:13.349009+0000","last_undegraded":"2024-09-10T19:00:13.349009+0000","last_fullsized":"2024-09-10T19:00:13.349009+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:29.227199+0000","last_clean_scrub_stamp":"2024-09-10T18:56:29.227199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034231449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":448,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.344678+0000","last_change":"2024-09-10T18:56:26.190647+0000","last_active":"2024-09-10T19:00:13.344678+0000","last_peered":"2024-09-10T19:00:13.344678+0000","last_clean":"2024-09-10T19:00:13.344678+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:00:13.344678+0000","last_undegraded":"2024-09-10T19:00:13.344678+0000","last_fullsized":"2024-09-10T19:00:13.344678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:56:26.190452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:56:26.190452+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048783639999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":435,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348987+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T19:00:13.348987+0000","last_peered":"2024-09-10T19:00:13.348987+0000","last_clean":"2024-09-10T19:00:13.348987+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:00:13.348987+0000","last_undegraded":"2024-09-10T19:00:13.348987+0000","last_fullsized":"2024-09-10T19:00:13.348987+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:19.026489+0000","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":557,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474189+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T19:00:13.474189+0000","last_peered":"2024-09-10T19:00:13.474189+0000","last_clean":"2024-09-10T19:00:13.474189+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:00:13.474189+0000","last_undegraded":"2024-09-10T19:00:13.474189+0000","last_fullsized":"2024-09-10T19:00:13.474189+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339756+0000","last_change":"2024-09-10T18:57:52.759843+0000","last_active":"2024-09-10T19:00:13.339756+0000","last_peered":"2024-09-10T19:00:13.339756+0000","last_clean":"2024-09-10T19:00:13.339756+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T19:00:13.339756+0000","last_undegraded":"2024-09-10T19:00:13.339756+0000","last_fullsized":"2024-09-10T19:00:13.339756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.759687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.759687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003277631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349814+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T19:00:13.349814+0000","last_peered":"2024-09-10T19:00:13.349814+0000","last_clean":"2024-09-10T19:00:13.349814+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:00:13.349814+0000","last_undegraded":"2024-09-10T19:00:13.349814+0000","last_fullsized":"2024-09-10T19:00:13.349814+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":456,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478408+0000","last_change":"2024-09-10T18:58:38.152259+0000","last_active":"2024-09-10T19:00:13.478408+0000","last_peered":"2024-09-10T19:00:13.478408+0000","last_clean":"2024-09-10T19:00:13.478408+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:00:13.478408+0000","last_undegraded":"2024-09-10T19:00:13.478408+0000","last_fullsized":"2024-09-10T19:00:13.478408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.152079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.152079+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040090350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":433,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476082+0000","last_change":"2024-09-10T18:58:55.639930+0000","last_active":"2024-09-10T19:00:13.476082+0000","last_peered":"2024-09-10T19:00:13.476082+0000","last_clean":"2024-09-10T19:00:13.476082+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:00:13.476082+0000","last_undegraded":"2024-09-10T19:00:13.476082+0000","last_fullsized":"2024-09-10T19:00:13.476082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.639777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.639777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003186637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":417,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.472092+0000","last_change":"2024-09-10T18:57:48.147362+0000","last_active":"2024-09-10T19:00:13.472092+0000","last_peered":"2024-09-10T19:00:13.472092+0000","last_clean":"2024-09-10T19:00:13.472092+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:00:13.472092+0000","last_undegraded":"2024-09-10T19:00:13.472092+0000","last_fullsized":"2024-09-10T19:00:13.472092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:48.147121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:57:48.147121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030702189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":432,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343295+0000","last_change":"2024-09-10T19:00:04.060523+0000","last_active":"2024-09-10T19:00:13.343295+0000","last_peered":"2024-09-10T19:00:13.343295+0000","last_clean":"2024-09-10T19:00:13.343295+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:00:13.343295+0000","last_undegraded":"2024-09-10T19:00:13.343295+0000","last_fullsized":"2024-09-10T19:00:13.343295+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.060371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.060371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.162315+0000","scrub_duration":0.0031455099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337399+0000","last_change":"2024-09-10T18:58:46.501591+0000","last_active":"2024-09-10T19:00:13.337399+0000","last_peered":"2024-09-10T19:00:13.337399+0000","last_clean":"2024-09-10T19:00:13.337399+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:00:13.337399+0000","last_undegraded":"2024-09-10T19:00:13.337399+0000","last_fullsized":"2024-09-10T19:00:13.337399+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.501508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036484099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":479,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.338617+0000","last_change":"2024-09-10T19:00:13.188540+0000","last_active":"2024-09-10T19:00:13.338617+0000","last_peered":"2024-09-10T19:00:13.338617+0000","last_clean":"2024-09-10T19:00:13.338617+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T19:00:13.338617+0000","last_undegraded":"2024-09-10T19:00:13.338617+0000","last_fullsized":"2024-09-10T19:00:13.338617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:00:13.188382+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.188382+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.453734+0000","scrub_duration":0.0046400579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":444,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477537+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T19:00:13.477537+0000","last_peered":"2024-09-10T19:00:13.477537+0000","last_clean":"2024-09-10T19:00:13.477537+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:00:13.477537+0000","last_undegraded":"2024-09-10T19:00:13.477537+0000","last_fullsized":"2024-09-10T19:00:13.477537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:17.284809+0000","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":432,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337786+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T19:00:13.337786+0000","last_peered":"2024-09-10T19:00:13.337786+0000","last_clean":"2024-09-10T19:00:13.337786+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:00:13.337786+0000","last_undegraded":"2024-09-10T19:00:13.337786+0000","last_fullsized":"2024-09-10T19:00:13.337786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":434,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347747+0000","last_change":"2024-09-10T18:59:55.150083+0000","last_active":"2024-09-10T19:00:13.347747+0000","last_peered":"2024-09-10T19:00:13.347747+0000","last_clean":"2024-09-10T19:00:13.347747+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:00:13.347747+0000","last_undegraded":"2024-09-10T19:00:13.347747+0000","last_fullsized":"2024-09-10T19:00:13.347747+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:55.150009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.150009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.988153+0000","scrub_duration":0.0024462989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":351,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.337692+0000","last_change":"2024-09-10T18:58:49.517454+0000","last_active":"2024-09-10T19:00:13.337692+0000","last_peered":"2024-09-10T19:00:13.337692+0000","last_clean":"2024-09-10T19:00:13.337692+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T19:00:13.337692+0000","last_undegraded":"2024-09-10T19:00:13.337692+0000","last_fullsized":"2024-09-10T19:00:13.337692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.517273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.517273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039870039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":444,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.350921+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T19:00:13.350921+0000","last_peered":"2024-09-10T19:00:13.350921+0000","last_clean":"2024-09-10T19:00:13.350921+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:00:13.350921+0000","last_undegraded":"2024-09-10T19:00:13.350921+0000","last_fullsized":"2024-09-10T19:00:13.350921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475541+0000","last_change":"2024-09-10T18:58:53.461054+0000","last_active":"2024-09-10T19:00:13.475541+0000","last_peered":"2024-09-10T19:00:13.475541+0000","last_clean":"2024-09-10T19:00:13.475541+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:00:13.475541+0000","last_undegraded":"2024-09-10T19:00:13.475541+0000","last_fullsized":"2024-09-10T19:00:13.475541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.460812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.460812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042777029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":426,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.343631+0000","last_change":"2024-09-10T18:59:53.179273+0000","last_active":"2024-09-10T19:00:13.343631+0000","last_peered":"2024-09-10T19:00:13.343631+0000","last_clean":"2024-09-10T19:00:13.343631+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:00:13.343631+0000","last_undegraded":"2024-09-10T19:00:13.343631+0000","last_fullsized":"2024-09-10T19:00:13.343631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.179202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.179202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.193560+0000","scrub_duration":0.0015188129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":437,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476776+0000","last_change":"2024-09-10T19:00:08.102047+0000","last_active":"2024-09-10T19:00:13.476776+0000","last_peered":"2024-09-10T19:00:13.476776+0000","last_clean":"2024-09-10T19:00:13.476776+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:00:13.476776+0000","last_undegraded":"2024-09-10T19:00:13.476776+0000","last_fullsized":"2024-09-10T19:00:13.476776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:08.101901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:00:08.101901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.050832+0000","scrub_duration":0.0028849869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473219+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T19:00:13.473219+0000","last_peered":"2024-09-10T19:00:13.473219+0000","last_clean":"2024-09-10T19:00:13.473219+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:00:13.473219+0000","last_undegraded":"2024-09-10T19:00:13.473219+0000","last_fullsized":"2024-09-10T19:00:13.473219+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":453,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475347+0000","last_change":"2024-09-10T18:59:04.505215+0000","last_active":"2024-09-10T19:00:13.475347+0000","last_peered":"2024-09-10T19:00:13.475347+0000","last_clean":"2024-09-10T19:00:13.475347+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:00:13.475347+0000","last_undegraded":"2024-09-10T19:00:13.475347+0000","last_fullsized":"2024-09-10T19:00:13.475347+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:59:04.505046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.505046+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.388122+0000","scrub_duration":0.0045725100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":443,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477457+0000","last_change":"2024-09-10T19:00:11.386471+0000","last_active":"2024-09-10T19:00:13.477457+0000","last_peered":"2024-09-10T19:00:13.477457+0000","last_clean":"2024-09-10T19:00:13.477457+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:00:13.477457+0000","last_undegraded":"2024-09-10T19:00:13.477457+0000","last_fullsized":"2024-09-10T19:00:13.477457+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:11.386293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:11.386293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:34.331345+0000","scrub_duration":0.0033555310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":423,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348114+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T19:00:13.348114+0000","last_peered":"2024-09-10T19:00:13.348114+0000","last_clean":"2024-09-10T19:00:13.348114+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:00:13.348114+0000","last_undegraded":"2024-09-10T19:00:13.348114+0000","last_fullsized":"2024-09-10T19:00:13.348114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:16.764329+0000","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347335+0000","last_change":"2024-09-10T18:56:38.054771+0000","last_active":"2024-09-10T19:00:13.347335+0000","last_peered":"2024-09-10T19:00:13.347335+0000","last_clean":"2024-09-10T19:00:13.347335+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:00:13.347335+0000","last_undegraded":"2024-09-10T19:00:13.347335+0000","last_fullsized":"2024-09-10T19:00:13.347335+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:38.054694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:56:38.054694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002113593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":340,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.346103+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T19:00:13.346103+0000","last_peered":"2024-09-10T19:00:13.346103+0000","last_clean":"2024-09-10T19:00:13.346103+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:00:13.346103+0000","last_undegraded":"2024-09-10T19:00:13.346103+0000","last_fullsized":"2024-09-10T19:00:13.346103+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":431,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.478098+0000","last_change":"2024-09-10T18:58:58.248456+0000","last_active":"2024-09-10T19:00:13.478098+0000","last_peered":"2024-09-10T19:00:13.478098+0000","last_clean":"2024-09-10T19:00:13.478098+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:00:13.478098+0000","last_undegraded":"2024-09-10T19:00:13.478098+0000","last_fullsized":"2024-09-10T19:00:13.478098+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:22.067159+0000","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":492,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473381+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T19:00:13.473381+0000","last_peered":"2024-09-10T19:00:13.473381+0000","last_clean":"2024-09-10T19:00:13.473381+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:00:13.473381+0000","last_undegraded":"2024-09-10T19:00:13.473381+0000","last_fullsized":"2024-09-10T19:00:13.473381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.470355+0000","last_change":"2024-09-10T18:58:31.395234+0000","last_active":"2024-09-10T19:00:13.470355+0000","last_peered":"2024-09-10T19:00:13.470355+0000","last_clean":"2024-09-10T19:00:13.470355+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:00:13.470355+0000","last_undegraded":"2024-09-10T19:00:13.470355+0000","last_fullsized":"2024-09-10T19:00:13.470355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:31.395072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:31.395072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032990430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349938+0000","last_change":"2024-09-10T18:56:18.268979+0000","last_active":"2024-09-10T19:00:13.349938+0000","last_peered":"2024-09-10T19:00:13.349938+0000","last_clean":"2024-09-10T19:00:13.349938+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:00:13.349938+0000","last_undegraded":"2024-09-10T19:00:13.349938+0000","last_fullsized":"2024-09-10T19:00:13.349938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:18.268813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:18.268813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033086249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.475811+0000","last_change":"2024-09-10T18:59:58.485524+0000","last_active":"2024-09-10T19:00:13.475811+0000","last_peered":"2024-09-10T19:00:13.475811+0000","last_clean":"2024-09-10T19:00:13.475811+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:00:13.475811+0000","last_undegraded":"2024-09-10T19:00:13.475811+0000","last_fullsized":"2024-09-10T19:00:13.475811+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.485240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.485240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.282886+0000","scrub_duration":0.0043729700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":438,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476731+0000","last_change":"2024-09-10T19:00:09.572036+0000","last_active":"2024-09-10T19:00:13.476731+0000","last_peered":"2024-09-10T19:00:13.476731+0000","last_clean":"2024-09-10T19:00:13.476731+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:00:13.476731+0000","last_undegraded":"2024-09-10T19:00:13.476731+0000","last_fullsized":"2024-09-10T19:00:13.476731+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:09.571847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.571847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:28.433824+0000","scrub_duration":0.0034123030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":450,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.344545+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T19:00:13.344545+0000","last_peered":"2024-09-10T19:00:13.344545+0000","last_clean":"2024-09-10T19:00:13.344545+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:00:13.344545+0000","last_undegraded":"2024-09-10T19:00:13.344545+0000","last_fullsized":"2024-09-10T19:00:13.344545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.349641+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T19:00:13.349641+0000","last_peered":"2024-09-10T19:00:13.349641+0000","last_clean":"2024-09-10T19:00:13.349641+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:00:13.349641+0000","last_undegraded":"2024-09-10T19:00:13.349641+0000","last_fullsized":"2024-09-10T19:00:13.349641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":587,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477376+0000","last_change":"2024-09-10T18:58:38.434610+0000","last_active":"2024-09-10T19:00:13.477376+0000","last_peered":"2024-09-10T19:00:13.477376+0000","last_clean":"2024-09-10T19:00:13.477376+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:00:13.477376+0000","last_undegraded":"2024-09-10T19:00:13.477376+0000","last_fullsized":"2024-09-10T19:00:13.477376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T18:58:38.434348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.434348+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046950070000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.477233+0000","last_change":"2024-09-10T18:59:00.644451+0000","last_active":"2024-09-10T19:00:13.477233+0000","last_peered":"2024-09-10T19:00:13.477233+0000","last_clean":"2024-09-10T19:00:13.477233+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:00:13.477233+0000","last_undegraded":"2024-09-10T19:00:13.477233+0000","last_fullsized":"2024-09-10T19:00:13.477233+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.644299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00344357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":430,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.339124+0000","last_change":"2024-09-10T18:57:37.858662+0000","last_active":"2024-09-10T19:00:13.339124+0000","last_peered":"2024-09-10T19:00:13.339124+0000","last_clean":"2024-09-10T19:00:13.339124+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:00:13.339124+0000","last_undegraded":"2024-09-10T19:00:13.339124+0000","last_fullsized":"2024-09-10T19:00:13.339124+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:37.858577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T18:57:37.858577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031779339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":442,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.473221+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T19:00:13.473221+0000","last_peered":"2024-09-10T19:00:13.473221+0000","last_clean":"2024-09-10T19:00:13.473221+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:00:13.473221+0000","last_undegraded":"2024-09-10T19:00:13.473221+0000","last_fullsized":"2024-09-10T19:00:13.473221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:28.205170+0000","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":446,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347337+0000","last_change":"2024-09-10T19:00:02.052175+0000","last_active":"2024-09-10T19:00:13.347337+0000","last_peered":"2024-09-10T19:00:13.347337+0000","last_clean":"2024-09-10T19:00:13.347337+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:00:13.347337+0000","last_undegraded":"2024-09-10T19:00:13.347337+0000","last_fullsized":"2024-09-10T19:00:13.347337+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:00:02.051995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.051995+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.499770+0000","scrub_duration":0.0042242440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":451,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.413347+0000","last_change":"2024-09-10T19:00:00.097374+0000","last_active":"2024-09-10T19:00:13.413347+0000","last_peered":"2024-09-10T19:00:13.413347+0000","last_clean":"2024-09-10T19:00:13.413347+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:00:13.413347+0000","last_undegraded":"2024-09-10T19:00:13.413347+0000","last_fullsized":"2024-09-10T19:00:13.413347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.097226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.097226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.731885+0000","scrub_duration":0.0030874499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":347,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.474011+0000","last_change":"2024-09-10T18:59:54.415459+0000","last_active":"2024-09-10T19:00:13.474011+0000","last_peered":"2024-09-10T19:00:13.474011+0000","last_clean":"2024-09-10T19:00:13.474011+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:00:13.474011+0000","last_undegraded":"2024-09-10T19:00:13.474011+0000","last_fullsized":"2024-09-10T19:00:13.474011+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:54.415297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:54.415297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:55.651756+0000","scrub_duration":0.0031427790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":421,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.348495+0000","last_change":"2024-09-10T18:56:33.964274+0000","last_active":"2024-09-10T19:00:13.348495+0000","last_peered":"2024-09-10T19:00:13.348495+0000","last_clean":"2024-09-10T19:00:13.348495+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:00:13.348495+0000","last_undegraded":"2024-09-10T19:00:13.348495+0000","last_fullsized":"2024-09-10T19:00:13.348495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T18:56:33.964103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037373380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":427,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347061+0000","last_change":"2024-09-10T18:56:24.942275+0000","last_active":"2024-09-10T19:00:13.347061+0000","last_peered":"2024-09-10T19:00:13.347061+0000","last_clean":"2024-09-10T19:00:13.347061+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:00:13.347061+0000","last_undegraded":"2024-09-10T19:00:13.347061+0000","last_fullsized":"2024-09-10T19:00:13.347061+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:24.942076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T18:56:24.942076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035504619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":469,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-09-10T19:00:14.152160+0000","last_change":"2024-09-10T19:00:14.152011+0000","last_active":"2024-09-10T19:00:14.152160+0000","last_peered":"2024-09-10T19:00:14.152160+0000","last_clean":"2024-09-10T19:00:14.152160+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:00:14.152160+0000","last_undegraded":"2024-09-10T19:00:14.152160+0000","last_fullsized":"2024-09-10T19:00:14.152160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:14.151862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.151862+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.920562+0000","scrub_duration":0.0041793289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":423,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.347126+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T19:00:13.347126+0000","last_peered":"2024-09-10T19:00:13.347126+0000","last_clean":"2024-09-10T19:00:13.347126+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:00:13.347126+0000","last_undegraded":"2024-09-10T19:00:13.347126+0000","last_fullsized":"2024-09-10T19:00:13.347126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":350,"reported_epoch":447,"state":"active+clean","last_fresh":"2024-09-10T19:00:13.476427+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T19:00:13.476427+0000","last_peered":"2024-09-10T19:00:13.476427+0000","last_clean":"2024-09-10T19:00:13.476427+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:00:13.476427+0000","last_undegraded":"2024-09-10T19:00:13.476427+0000","last_fullsized":"2024-09-10T19:00:13.476427+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:31.897723+0000","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":215,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195793,"num_objects":161,"num_object_clones":4,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26505216,"data_stored":24587397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1877,"ondisk_log_size":1877,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":632,"num_read_kb":627,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972495,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317016,"kb_used_data":15488,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93428136,"statfs":{"total":95995035648,"available":95670411264,"internally_reserved":0,"allocated":15859712,"data_stored":12749322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.3839999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.024},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92100000000000004,"15min":0.92400000000000004},"min":{"1min":0.50600000000000001,"5min":0.372,"15min":0.34699999999999998},"max":{"1min":1.2529999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.79800000000000004}]},{"osd":1,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.208,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":1.004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.421,"5min":1.742,"15min":1.8280000000000001},"last":0.90200000000000002}]},{"osd":2,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.014,"15min":1.0289999999999999},"min":{"1min":0.45000000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6739999999999999,"5min":2.9199999999999999,"15min":3.4990000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0009999999999999,"15min":0.98999999999999999},"min":{"1min":0.64500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.97,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":1.0589999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0629999999999999,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.9390000000000001,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":1.266},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":0.97599999999999998},"min":{"1min":0.56200000000000006,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.833,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.92700000000000005}]},{"osd":4,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.014,"15min":1.0149999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3700000000000001,"5min":1.4279999999999999,"15min":2.1989999999999998},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.046,"5min":1.127,"15min":1.0609999999999999},"min":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0310000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.2010000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.998,"15min":0.99199999999999999},"min":{"1min":0.56899999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7669999999999999,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1379999999999999,"15min":1.1639999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":2.1419999999999999},"last":1.171}]},{"osd":6,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.179,"15min":1.2150000000000001},"min":{"1min":0.60899999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.468,"5min":1.591,"15min":2.2749999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2150000000000001,"15min":1.1950000000000001},"min":{"1min":0.70599999999999996,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":2.1560000000000001,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.325}]}]},{"osd":6,"up_from":52,"seq":223338299545,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313068,"kb_used_data":7380,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432084,"statfs":{"total":95995035648,"available":95674454016,"internally_reserved":0,"allocated":7557120,"data_stored":4553066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.88700000000000001,"15min":0.88400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36399999999999999},"max":{"1min":1.181,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.95899999999999996},"min":{"1min":0.46300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.276,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":1.2370000000000001}]},{"osd":1,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.452,"5min":2.2410000000000001,"15min":18.494},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.88100000000000001,"15min":0.90500000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1679999999999999,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":1.1830000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94199999999999995,"15min":0.96499999999999997},"min":{"1min":0.47499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.288,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5309999999999999,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.0449999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.05,"15min":0.997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.827,"5min":2.278,"15min":2.278},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.014,"5min":1.002,"15min":0.98499999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.641,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":0.82899999999999996}]},{"osd":4,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.028,"15min":0.94499999999999995},"min":{"1min":0.66500000000000004,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.782,"5min":6.008,"15min":6.008},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1319999999999999,"15min":1.0569999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42999999999999999},"max":{"1min":1.6100000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.214}]},{"osd":5,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1220000000000001,"15min":1.0369999999999999},"min":{"1min":0.71099999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6739999999999999,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1890000000000001,"15min":1.125},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.708,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.2609999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.115,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.5840000000000001,"5min":2.069,"15min":2.069},"last":1.01},{"interface":"front","average":{"1min":1.292,"5min":1.2929999999999999,"15min":1.196},"min":{"1min":0.90600000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8680000000000001,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.29}]}]},{"osd":5,"up_from":43,"seq":184683593891,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312516,"kb_used_data":10988,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432636,"statfs":{"total":95995035648,"available":95675019264,"internally_reserved":0,"allocated":11251712,"data_stored":8157306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.1000000000000001,"15min":0.94699999999999995},"min":{"1min":0.52900000000000003,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.347,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.88400000000000001,"5min":1.012,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.3160000000000001,"5min":7.3070000000000004,"15min":7.3070000000000004},"last":1.2709999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.6739999999999999,"15min":1.1399999999999999},"min":{"1min":0.67900000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0980000000000001,"5min":31.548999999999999,"15min":31.548999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":1.431,"15min":1.0760000000000001},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.135,"5min":31.847000000000001,"15min":31.847000000000001},"last":1.123}]},{"osd":2,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.8959999999999999,"15min":1.238},"min":{"1min":0.59299999999999997,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.22,"5min":61.347000000000001,"15min":61.347000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.137,"15min":0.996},"min":{"1min":0.66100000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.23,"5min":6.1600000000000001,"15min":6.1600000000000001},"last":1.04}]},{"osd":3,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.1339999999999999,"15min":1},"min":{"1min":0.58799999999999997,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2769999999999999,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.008},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.792,"15min":1.2410000000000001},"min":{"1min":0.72699999999999998,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":1.2270000000000001,"5min":53.173000000000002,"15min":53.173000000000002},"last":1.208}]},{"osd":4,"last update":"Tue Sep 10 18:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.167,"15min":0.94299999999999995},"min":{"1min":0.52800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.2529999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.224,"15min":1.0109999999999999},"min":{"1min":0.67200000000000004,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.2390000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.395,"15min":1.1519999999999999},"min":{"1min":0.67000000000000004,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":1.3400000000000001,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.4299999999999999,"15min":1.1879999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.47199999999999998},"max":{"1min":1.5760000000000001,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.3700000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.532,"15min":1.2130000000000001},"min":{"1min":0.59699999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.3899999999999999,"5min":15.613,"15min":15.613},"last":1.587},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.544,"15min":1.2749999999999999},"min":{"1min":0.77300000000000002,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":1.538,"5min":16.965,"15min":16.965},"last":1.165}]}]},{"osd":4,"up_from":35,"seq":150323855535,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312504,"kb_used_data":6816,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432648,"statfs":{"total":95995035648,"available":95675031552,"internally_reserved":0,"allocated":6979584,"data_stored":3963960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.83999999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.403,"5min":1.546,"15min":1.546},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.91600000000000004,"15min":0.88600000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.403,"5min":1.667,"15min":1.7070000000000001},"last":0.86899999999999999}]},{"osd":1,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84599999999999997,"15min":0.86899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.133,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.91200000000000003,"15min":0.89700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2869999999999999,"5min":1.575,"15min":1.575},"last":0.82899999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.93999999999999995,"15min":0.92100000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5209999999999999,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95099999999999996,"15min":0.92400000000000004},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.387,"5min":1.5840000000000001,"15min":1.7569999999999999},"last":0.997}]},{"osd":3,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.96199999999999997,"15min":0.93600000000000005},"min":{"1min":0.54900000000000004,"5min":0.53800000000000003,"15min":0.46300000000000002},"max":{"1min":4.2670000000000003,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":1.048},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.95399999999999996,"15min":0.95099999999999996},"min":{"1min":0.49099999999999999,"5min":0.442,"15min":0.442},"max":{"1min":1.3740000000000001,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.2350000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.96099999999999997,"15min":0.92100000000000004},"min":{"1min":0.67200000000000004,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.095,"5min":1.0669999999999999,"15min":0.99399999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5209999999999999,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.204}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0660000000000001,"15min":1.0009999999999999},"min":{"1min":0.60799999999999998,"5min":0.501,"15min":0.501},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.1160000000000001},"min":{"1min":0.73699999999999999,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7230000000000001,"5min":1.927,"15min":1.927},"last":1.169}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.80100000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5880000000000001,"5min":1.629,"15min":2.008},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.1839999999999999,"15min":1.167},"min":{"1min":1,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.304}]}]},{"osd":3,"up_from":29,"seq":124554051770,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304324,"kb_used_data":6956,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93440828,"statfs":{"total":95995035648,"available":95683407872,"internally_reserved":0,"allocated":7122944,"data_stored":3963958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.24,"15min":0.82499999999999996},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.86799999999999999,"5min":26.594000000000001,"15min":26.594000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63,"5min":1.2589999999999999,"15min":0.88900000000000001},"min":{"1min":0.372,"5min":0.252,"15min":0.252},"max":{"1min":1.085,"5min":24.204999999999998,"15min":24.204999999999998},"last":0.84999999999999998}]},{"osd":1,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.3919999999999999,"15min":0.91200000000000003},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.26500000000000001},"max":{"1min":0.88200000000000001,"5min":53.661999999999999,"15min":53.661999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.0740000000000001,"15min":0.871},"min":{"1min":0.52900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0009999999999999,"5min":15.189,"15min":15.189},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.381,"15min":0.99099999999999999},"min":{"1min":0.432,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.153,"5min":26.558,"15min":26.558},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.2809999999999999,"15min":0.93100000000000005},"min":{"1min":0.434,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.8540000000000001,"5min":25.536999999999999,"15min":25.536999999999999},"last":0.91800000000000004}]},{"osd":4,"last update":"Tue Sep 10 18:59:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.3600000000000001,"15min":1.137},"min":{"1min":0.73099999999999998,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.5,"5min":10.359999999999999,"15min":10.359999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.464,"15min":1.224},"min":{"1min":0.81000000000000005,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.633,"5min":10.391,"15min":10.391},"last":1.1519999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.70399999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.8009999999999999,"5min":9.2750000000000004,"15min":17.535},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.143,"15min":1.048},"min":{"1min":0.66900000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.6619999999999999,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":0.97599999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.204,"15min":1.1990000000000001},"min":{"1min":0.80200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.9159999999999999,"5min":9.3610000000000007,"15min":10.51},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1599999999999999,"15min":1.093},"min":{"1min":0.71499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9359999999999999,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:56 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.179},"min":{"1min":0.60199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":4.0229999999999997,"5min":5.3109999999999999,"15min":18.312999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3919999999999999,"15min":1.3140000000000001},"min":{"1min":0.86399999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":4.5350000000000001,"5min":9.3840000000000003,"15min":10.611000000000001},"last":1.2410000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378819,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320576,"kb_used_data":14888,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93424576,"statfs":{"total":95995035648,"available":95666765824,"internally_reserved":0,"allocated":15245312,"data_stored":12158332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.84099999999999997,"15min":0.72099999999999997},"min":{"1min":0.36799999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.2749999999999999,"5min":9.516,"15min":9.516},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.84999999999999998,"15min":0.77900000000000003},"min":{"1min":0.36099999999999999,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.0760000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.81399999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.77700000000000002,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.0720000000000001,"5min":5.516,"15min":5.516},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85899999999999999,"15min":0.81200000000000006},"min":{"1min":0.54000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0820000000000001,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.85799999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.91500000000000004,"15min":0.85699999999999998},"min":{"1min":0.503,"5min":0.36599999999999999,"15min":0.34799999999999998},"max":{"1min":1.3859999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.91700000000000004,"15min":0.84299999999999997},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":1.194,"5min":15.736000000000001,"15min":15.736000000000001},"last":1.294}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.2090000000000001,"15min":1.0629999999999999},"min":{"1min":0.49299999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":3.851,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.675,"5min":1.2869999999999999,"15min":1.149},"min":{"1min":0.75900000000000001,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":10.881,"5min":10.881,"15min":10.881},"last":1.214}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.054,"15min":1.0900000000000001},"min":{"1min":0.754,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8340000000000001,"5min":4.2720000000000002,"15min":5.1959999999999997},"last":1.034},{"interface":"front","average":{"1min":1.056,"5min":1.268,"15min":1.1080000000000001},"min":{"1min":0.69299999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.405,"5min":21.654,"15min":21.654},"last":1.405}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1950000000000001,"15min":1.1659999999999999},"min":{"1min":0.85099999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.2529999999999999,"15min":1.131},"min":{"1min":0.82799999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.3400000000000001,"5min":15.971,"15min":15.971},"last":1.1699999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2150000000000001,"15min":1.1830000000000001},"min":{"1min":0.88300000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.298,"5min":15.025,"15min":15.025},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.361,"15min":1.2},"min":{"1min":0.91800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3759999999999999,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.369}]}]},{"osd":1,"up_from":13,"seq":55834575054,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312268,"kb_used_data":10748,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432884,"statfs":{"total":95995035648,"available":95675273216,"internally_reserved":0,"allocated":11005952,"data_stored":7963161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.0649999999999999,"15min":0.79300000000000004},"min":{"1min":0.33900000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.079,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.1839999999999999,"15min":0.872},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":1.177,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.66000000000000003}]},{"osd":2,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.181,"15min":0.85699999999999998},"min":{"1min":0.46000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.724,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.746,"5min":1.371,"15min":0.98399999999999999},"min":{"1min":0.39400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.03,"5min":32.363,"15min":32.363},"last":0.99399999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.27,"15min":0.98199999999999998},"min":{"1min":0.496,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.7929999999999999,"5min":25.704000000000001,"15min":25.704000000000001},"last":1.131},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.6000000000000001,"15min":1.0269999999999999},"min":{"1min":0.42499999999999999,"5min":0.34699999999999998,"15min":0.26600000000000001},"max":{"1min":1.871,"5min":60.999000000000002,"15min":60.999000000000002},"last":0.625}]},{"osd":4,"last update":"Tue Sep 10 18:59:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.2470000000000001,"15min":1.0940000000000001},"min":{"1min":0.70599999999999996,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.3220000000000001,"5min":12.808999999999999,"15min":12.808999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.7589999999999999,"15min":1.2090000000000001},"min":{"1min":0.72199999999999998,"5min":0.378,"15min":0.378},"max":{"1min":1.544,"5min":60.529000000000003,"15min":60.529000000000003},"last":0.76600000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.3089999999999999,"15min":1.1040000000000001},"min":{"1min":0.78000000000000003,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":1.8160000000000001,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.385,"15min":1.204},"min":{"1min":0.84299999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.2229999999999999,"5min":12.84,"15min":12.84},"last":1.377}]},{"osd":6,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.615,"15min":1.2689999999999999},"min":{"1min":0.96999999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.1509999999999998,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.581,"15min":1.2350000000000001},"min":{"1min":0.76100000000000001,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.9890000000000001,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.0700000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6060000000000001,"15min":1.3260000000000001},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.2989999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.6699999999999999,"15min":1.3089999999999999},"min":{"1min":0.88500000000000001,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":2.0600000000000001,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.26}]}]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308964,"kb_used_data":7436,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436188,"statfs":{"total":95995035648,"available":95678656512,"internally_reserved":0,"allocated":7614464,"data_stored":4553073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.591,"15min":0.96899999999999997},"min":{"1min":0.32400000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.877,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.7270000000000001,"15min":1.077},"min":{"1min":0.48699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.1779999999999999,"5min":86.009,"15min":86.009},"last":0.86899999999999999}]},{"osd":2,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":1.6639999999999999,"15min":1.0209999999999999},"min":{"1min":0.60699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.593,"5min":36.682000000000002,"15min":36.682000000000002},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.2190000000000001,"15min":0.90600000000000003},"min":{"1min":0.57299999999999995,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.333,"5min":24.614000000000001,"15min":24.614000000000001},"last":1.151}]},{"osd":3,"last update":"Tue Sep 10 18:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.1459999999999999,"15min":0.89300000000000002},"min":{"1min":0.48799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.379,"5min":21.620999999999999,"15min":21.620999999999999},"last":1.04},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.5049999999999999,"15min":1.0349999999999999},"min":{"1min":0.57999999999999996,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.4019999999999999,"5min":45.765000000000001,"15min":45.765000000000001},"last":0.748}]},{"osd":4,"last update":"Tue Sep 10 18:59:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.1619999999999999,"15min":1.0429999999999999},"min":{"1min":0.71199999999999997,"5min":0.5,"15min":0.5},"max":{"1min":1.6839999999999999,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1739999999999999,"15min":1.099},"min":{"1min":0.70499999999999996,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.6439999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":0.71299999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:00:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.71799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.585,"5min":2.7989999999999999,"15min":10.048},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0600000000000001,"15min":1.0960000000000001},"min":{"1min":0.71599999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.496,"5min":1.635,"15min":3.8959999999999999},"last":1.2310000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:00:04 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1619999999999999,"15min":1.135},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56899999999999995},"max":{"1min":1.792,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2669999999999999,"15min":1.1970000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.804,"5min":11.894,"15min":11.894},"last":1.111}]},{"osd":7,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.149,"15min":1.167},"min":{"1min":0.80600000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5549999999999999,"5min":2.266,"15min":6.2949999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.25,"5min":1.3660000000000001,"15min":1.29},"min":{"1min":0.78400000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0129999999999999,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.268}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8194945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339968,"data_stored":761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4194961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":8194912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:00:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.570 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:22.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:22.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:22.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:22.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:22.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:22 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.561507+0000 osd.6 (osd.6) 207 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:00:22.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:22 smithi165 bash[22591]: cluster 2024-09-10T19:00:21.565357+0000 osd.6 (osd.6) 208 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:00:22.824 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:22 smithi165 bash[22591]: audit 2024-09-10T19:00:21.692997+0000 mgr.y (mgr.24526) 447 : audit [DBG] from='client.49915 v1:172.21.15.83:0/3533912499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:22.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:22 smithi165 bash[22591]: cluster 2024-09-10T19:00:22.102652+0000 mgr.y (mgr.24526) 448 : cluster [DBG] pgmap v668: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:22.847 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:22.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:22.964 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:22] "GET /metrics HTTP/1.1" 200 228892 "" "Prometheus/2.43.0" 2024-09-10T19:00:22.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.561507+0000 osd.6 (osd.6) 207 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:00:22.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[32381]: cluster 2024-09-10T19:00:21.565357+0000 osd.6 (osd.6) 208 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:00:22.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[32381]: audit 2024-09-10T19:00:21.692997+0000 mgr.y (mgr.24526) 447 : audit [DBG] from='client.49915 v1:172.21.15.83:0/3533912499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:22.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[32381]: cluster 2024-09-10T19:00:22.102652+0000 mgr.y (mgr.24526) 448 : cluster [DBG] pgmap v668: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:22.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.561507+0000 osd.6 (osd.6) 207 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:00:22.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[18629]: cluster 2024-09-10T19:00:21.565357+0000 osd.6 (osd.6) 208 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:00:22.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[18629]: audit 2024-09-10T19:00:21.692997+0000 mgr.y (mgr.24526) 447 : audit [DBG] from='client.49915 v1:172.21.15.83:0/3533912499' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:22.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:22 smithi083 bash[18629]: cluster 2024-09-10T19:00:22.102652+0000 mgr.y (mgr.24526) 448 : cluster [DBG] pgmap v668: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:22.985 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:22.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:23.046 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:23.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:23.115 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:23.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:23.249 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:23.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:23.386 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:23.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:23.525 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:23.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:23.663 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:23.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:23.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:22.880185+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.c scrub starts 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:22.885130+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.c scrub ok 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.086196+0000 osd.7 (osd.7) 256 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.089473+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.192537+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.196048+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.200776+0000 osd.0 (osd.0) 240 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:00:23.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:23 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.207038+0000 osd.0 (osd.0) 241 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:00:23.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:23.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:22.880185+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.c scrub starts 2024-09-10T19:00:23.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:22.885130+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.c scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.086196+0000 osd.7 (osd.7) 256 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.089473+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.192537+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.196048+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.200776+0000 osd.0 (osd.0) 240 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.207038+0000 osd.0 (osd.0) 241 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:22.880185+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.c scrub starts 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:22.885130+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.c scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.086196+0000 osd.7 (osd.7) 256 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.089473+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:00:23.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.192537+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:00:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.196048+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:00:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.200776+0000 osd.0 (osd.0) 240 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:00:23.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:23 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.207038+0000 osd.0 (osd.0) 241 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:00:23.937 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:23.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:24.040 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:24.068 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:24.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:24.203 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:24.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:24.340 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:24.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:24.474 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:24.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:24.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:24.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:24.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:24.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:24.888 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:24.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:24.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:25.021 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:25.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.157 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:25.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:25.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:25.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:25.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.835385+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:23.838869+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:24.100109+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:24.103710+0000 mgr.y (mgr.24526) 449 : cluster [DBG] pgmap v669: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:24.104047+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[32381]: cluster 2024-09-10T19:00:24.141544+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.835385+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:00:25.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:23.838869+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:00:25.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:24.100109+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:00:25.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:24.103710+0000 mgr.y (mgr.24526) 449 : cluster [DBG] pgmap v669: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:25.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:24.104047+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:00:25.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:25 smithi083 bash[18629]: cluster 2024-09-10T19:00:24.141544+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:00:25.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:25.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.835385+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:23.838869+0000 osd.3 (osd.3) 273 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:24.100109+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:24.103710+0000 mgr.y (mgr.24526) 449 : cluster [DBG] pgmap v669: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:24.104047+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:00:25.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:25 smithi165 bash[22591]: cluster 2024-09-10T19:00:24.141544+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:00:25.571 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:25.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:25.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:25.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.840 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:25.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:25.981 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:25.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:26.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:26.116 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:26.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:26.254 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:26.358 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.162724+0000 osd.4 (osd.4) 282 : cluster [DBG] 64.11 deep-scrub starts 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.163962+0000 osd.0 (osd.0) 242 : cluster [DBG] 12.e scrub starts 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.167438+0000 osd.0 (osd.0) 243 : cluster [DBG] 12.e scrub ok 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.174111+0000 osd.4 (osd.4) 283 : cluster [DBG] 64.11 deep-scrub ok 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.221438+0000 osd.2 (osd.2) 187 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.225112+0000 osd.2 (osd.2) 188 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.634418+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.637864+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.680624+0000 osd.6 (osd.6) 209 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:00:26.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:26 smithi165 bash[22591]: cluster 2024-09-10T19:00:25.684567+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:00:26.390 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:26.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.162724+0000 osd.4 (osd.4) 282 : cluster [DBG] 64.11 deep-scrub starts 2024-09-10T19:00:26.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.163962+0000 osd.0 (osd.0) 242 : cluster [DBG] 12.e scrub starts 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.167438+0000 osd.0 (osd.0) 243 : cluster [DBG] 12.e scrub ok 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.174111+0000 osd.4 (osd.4) 283 : cluster [DBG] 64.11 deep-scrub ok 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.221438+0000 osd.2 (osd.2) 187 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.225112+0000 osd.2 (osd.2) 188 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.634418+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.637864+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.680624+0000 osd.6 (osd.6) 209 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[32381]: cluster 2024-09-10T19:00:25.684567+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:00:26.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.162724+0000 osd.4 (osd.4) 282 : cluster [DBG] 64.11 deep-scrub starts 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.163962+0000 osd.0 (osd.0) 242 : cluster [DBG] 12.e scrub starts 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.167438+0000 osd.0 (osd.0) 243 : cluster [DBG] 12.e scrub ok 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.174111+0000 osd.4 (osd.4) 283 : cluster [DBG] 64.11 deep-scrub ok 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.221438+0000 osd.2 (osd.2) 187 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.225112+0000 osd.2 (osd.2) 188 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.634418+0000 osd.1 (osd.1) 259 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.637864+0000 osd.1 (osd.1) 260 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.680624+0000 osd.6 (osd.6) 209 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:00:26.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:26 smithi083 bash[18629]: cluster 2024-09-10T19:00:25.684567+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:00:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:26.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:26.532 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:26.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:26.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:26.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:26.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:26.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:26.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:26.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:26.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:27.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:27.080 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:27.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:27.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:27 smithi165 bash[22591]: cluster 2024-09-10T19:00:26.105804+0000 mgr.y (mgr.24526) 450 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:00:27.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:27 smithi165 bash[22591]: cluster 2024-09-10T19:00:26.111945+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.0 deep-scrub starts 2024-09-10T19:00:27.194 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:27 smithi165 bash[22591]: cluster 2024-09-10T19:00:26.116027+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 deep-scrub ok 2024-09-10T19:00:27.212 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:27.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:27.329 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[18629]: cluster 2024-09-10T19:00:26.105804+0000 mgr.y (mgr.24526) 450 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:00:27.329 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[18629]: cluster 2024-09-10T19:00:26.111945+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.0 deep-scrub starts 2024-09-10T19:00:27.329 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[18629]: cluster 2024-09-10T19:00:26.116027+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 deep-scrub ok 2024-09-10T19:00:27.329 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[32381]: cluster 2024-09-10T19:00:26.105804+0000 mgr.y (mgr.24526) 450 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:00:27.329 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[32381]: cluster 2024-09-10T19:00:26.111945+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.0 deep-scrub starts 2024-09-10T19:00:27.330 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:27 smithi083 bash[32381]: cluster 2024-09-10T19:00:26.116027+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 deep-scrub ok 2024-09-10T19:00:27.350 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:27.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:27.484 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:27.570 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:00:27.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:00:27.570 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:00:27.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:00:27.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:27.623 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:27.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:27.760 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:27.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:27.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:27.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:27.999 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.033 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:28.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:28.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:27 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:28.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:26.921312+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:00:28.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:26.925112+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:00:28.166 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:28.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:28.300 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:28.404 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.149468+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.f scrub starts 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.154690+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.f scrub ok 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.155640+0000 osd.5 (osd.5) 241 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.158984+0000 osd.5 (osd.5) 242 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.160348+0000 osd.4 (osd.4) 284 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.165223+0000 osd.4 (osd.4) 285 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.176126+0000 osd.2 (osd.2) 189 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:00:28.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[32381]: cluster 2024-09-10T19:00:27.181312+0000 osd.2 (osd.2) 190 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:26.921312+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:26.925112+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.149468+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.f scrub starts 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.154690+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.f scrub ok 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.155640+0000 osd.5 (osd.5) 241 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.158984+0000 osd.5 (osd.5) 242 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.160348+0000 osd.4 (osd.4) 284 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.165223+0000 osd.4 (osd.4) 285 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.176126+0000 osd.2 (osd.2) 189 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:00:28.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:28 smithi083 bash[18629]: cluster 2024-09-10T19:00:27.181312+0000 osd.2 (osd.2) 190 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:00:28.436 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:28.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:26.921312+0000 osd.3 (osd.3) 274 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:26.925112+0000 osd.3 (osd.3) 275 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.149468+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.f scrub starts 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.154690+0000 osd.7 (osd.7) 263 : cluster [DBG] 3.f scrub ok 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.155640+0000 osd.5 (osd.5) 241 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.158984+0000 osd.5 (osd.5) 242 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.160348+0000 osd.4 (osd.4) 284 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.165223+0000 osd.4 (osd.4) 285 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.176126+0000 osd.2 (osd.2) 189 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:00:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:28 smithi165 bash[22591]: cluster 2024-09-10T19:00:27.181312+0000 osd.2 (osd.2) 190 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:00:28.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:28.709 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:28.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.846 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:28.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:28.982 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:28.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:29.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:29.122 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:29.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:29.257 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:29.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[18629]: audit 2024-09-10T19:00:27.982876+0000 mgr.y (mgr.24526) 451 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[18629]: cluster 2024-09-10T19:00:28.106590+0000 mgr.y (mgr.24526) 452 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[18629]: cluster 2024-09-10T19:00:28.726249+0000 osd.1 (osd.1) 261 : cluster [DBG] 64.b scrub starts 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[18629]: cluster 2024-09-10T19:00:28.730892+0000 osd.1 (osd.1) 262 : cluster [DBG] 64.b scrub ok 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[18629]: cluster 2024-09-10T19:00:29.143459+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[32381]: audit 2024-09-10T19:00:27.982876+0000 mgr.y (mgr.24526) 451 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[32381]: cluster 2024-09-10T19:00:28.106590+0000 mgr.y (mgr.24526) 452 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[32381]: cluster 2024-09-10T19:00:28.726249+0000 osd.1 (osd.1) 261 : cluster [DBG] 64.b scrub starts 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[32381]: cluster 2024-09-10T19:00:28.730892+0000 osd.1 (osd.1) 262 : cluster [DBG] 64.b scrub ok 2024-09-10T19:00:29.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:29 smithi083 bash[32381]: cluster 2024-09-10T19:00:29.143459+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:00:29.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:29 smithi165 bash[22591]: audit 2024-09-10T19:00:27.982876+0000 mgr.y (mgr.24526) 451 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:29 smithi165 bash[22591]: cluster 2024-09-10T19:00:28.106590+0000 mgr.y (mgr.24526) 452 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:00:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:29 smithi165 bash[22591]: cluster 2024-09-10T19:00:28.726249+0000 osd.1 (osd.1) 261 : cluster [DBG] 64.b scrub starts 2024-09-10T19:00:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:29 smithi165 bash[22591]: cluster 2024-09-10T19:00:28.730892+0000 osd.1 (osd.1) 262 : cluster [DBG] 64.b scrub ok 2024-09-10T19:00:29.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:29 smithi165 bash[22591]: cluster 2024-09-10T19:00:29.143459+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:00:29.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:29.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:29.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:29.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:29.664 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:29.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:29.798 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:29.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:29.937 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:29.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:30.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:30.071 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:30.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:30.206 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:30.310 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[18629]: audit 2024-09-10T19:00:29.565033+0000 mon.c (mon.2) 623 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[18629]: cluster 2024-09-10T19:00:29.719593+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[18629]: cluster 2024-09-10T19:00:29.731721+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[32381]: audit 2024-09-10T19:00:29.565033+0000 mon.c (mon.2) 623 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[32381]: cluster 2024-09-10T19:00:29.719593+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:00:30.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:30 smithi083 bash[32381]: cluster 2024-09-10T19:00:29.731721+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:00:30.340 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:30.444 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:30.477 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:30.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:30.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:30 smithi165 bash[22591]: audit 2024-09-10T19:00:29.565033+0000 mon.c (mon.2) 623 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:30.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:30 smithi165 bash[22591]: cluster 2024-09-10T19:00:29.719593+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:00:30.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:30 smithi165 bash[22591]: cluster 2024-09-10T19:00:29.731721+0000 osd.6 (osd.6) 212 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:00:30.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:30.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:30.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:30.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:30.886 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:30.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:30.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:31.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:31.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:31.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:31.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:31.437 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:31.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.107386+0000 mgr.y (mgr.24526) 453 : cluster [DBG] pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:31.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.151038+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-09-10T19:00:31.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.155067+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-09-10T19:00:31.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.187974+0000 osd.2 (osd.2) 191 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:00:31.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.192752+0000 osd.2 (osd.2) 192 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:00:31.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.107386+0000 mgr.y (mgr.24526) 453 : cluster [DBG] pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:31.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.151038+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-09-10T19:00:31.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.155067+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-09-10T19:00:31.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.187974+0000 osd.2 (osd.2) 191 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:00:31.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:31 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.192752+0000 osd.2 (osd.2) 192 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:00:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:31.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.555 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:31.555 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.107386+0000 mgr.y (mgr.24526) 453 : cluster [DBG] pgmap v674: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:31.555 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.151038+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-09-10T19:00:31.555 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.155067+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-09-10T19:00:31.555 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.187974+0000 osd.2 (osd.2) 191 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:00:31.556 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:31 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.192752+0000 osd.2 (osd.2) 192 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:00:31.580 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:31.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:31.719 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:31.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:31.855 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:31.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:31.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- 172.21.15.83:0/3054280408 >> v1:172.21.15.83:6789/0 conn(0x7f0528108ef0 legacy=0x7f052810b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:31.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- 172.21.15.83:0/3054280408 shutdown_connections 2024-09-10T19:00:31.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- 172.21.15.83:0/3054280408 >> 172.21.15.83:0/3054280408 conn(0x7f05280f7090 msgr2=0x7f05280f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:31.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- 172.21.15.83:0/3054280408 shutdown_connections 2024-09-10T19:00:31.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- 172.21.15.83:0/3054280408 wait complete. 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 librados: starting msgr at 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 librados: starting objecter 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 Processor -- start 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- start start 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 librados: setting wanted keys 2024-09-10T19:00:31.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 librados: calling monclient init 2024-09-10T19:00:31.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f052811e810 con 0x7f05281077d0 2024-09-10T19:00:31.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f052811f9b0 con 0x7f05280fd5c0 2024-09-10T19:00:31.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052f9d4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0528120b90 con 0x7f0528108ef0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052df71640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f05281077d0 0x7f052811d060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57536/0 (socket says 172.21.15.83:57536) 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 learned_addr learned my addr 172.21.15.83:0/2604873198 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter ms_handle_connect 0x7f05281077d0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter ms_handle_connect 0x7f0528108ef0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter ms_handle_connect 0x7f05280fd5c0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401185196 0 0) 0x7f052811e810 con 0x7f05281077d0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514003720 con 0x7f05281077d0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152595955 0 0) 0x7f0528120b90 con 0x7f0528108ef0 2024-09-10T19:00:31.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052811e810 con 0x7f0528108ef0 2024-09-10T19:00:31.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926062361 0 0) 0x7f0514003720 con 0x7f05281077d0 2024-09-10T19:00:31.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0528120b90 con 0x7f05281077d0 2024-09-10T19:00:31.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746933978 0 0) 0x7f052811f9b0 con 0x7f05280fd5c0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514003720 con 0x7f05280fd5c0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200258074 0 0) 0x7f052811e810 con 0x7f0528108ef0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f052811f9b0 con 0x7f0528108ef0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.879+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f05200030d0 con 0x7f05281077d0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0524001840 con 0x7f0528108ef0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 355786252 0 0) 0x7f0528120b90 con 0x7f05281077d0 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6790/0 conn(0x7f0528108ef0 legacy=0x7f0528119930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:31.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.165:6789/0 conn(0x7f05280fd5c0 legacy=0x7f05281070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0528121db0 con 0x7f05281077d0 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0528120d80 con 0x7f05281077d0 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f052f9d4640 10 client.25341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05281211d0 con 0x7f05281077d0 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f052f9d4640 1 librados: init done 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0520001580 con 0x7f05281077d0 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0520003940 con 0x7f05281077d0 2024-09-10T19:00:31.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f050c004ff0 con 0x7f05281077d0 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f0520003ac0 con 0x7f05281077d0 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(454..454 src has 69..454) v4 ==== 8060+0+0 (unknown 1326074345 0 0) 0x7f052008abc0 con 0x7f05281077d0 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 osd_map(454..454 src has 69..454) v4 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 3 client.25341.objecter handle_osd_map got epochs [454,454] > 0 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.883+0000 7f051dffb640 3 client.25341.objecter handle_osd_map decoding full epoch 454 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 20 client.25341.objecter dump_active .. 0 homeless 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f0514071780 2024-09-10T19:00:31.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) v1 ==== 54+0+16 (unknown 1521394261 0 2420000192) 0x7f052008af70 con 0x7f05281077d0 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 10 client.25341.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 1 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 1 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 0 1 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 4 2024-09-10T19:00:31.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 10 client.25341.objecter _send_command 1 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c005290 con 0x7f050c004610 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c004610 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c005290 con 0x7f050c004610 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 10 client.25341.objecter _finish_command 1 = system:0 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 4 2024-09-10T19:00:31.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.887+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 0 1 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 10 client.25341.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 2 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 2 2024-09-10T19:00:31.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c005810 osd=0 3 2024-09-10T19:00:31.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 0 2 2024-09-10T19:00:31.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 4 2024-09-10T19:00:31.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 10 client.25341.objecter _send_command 2 2024-09-10T19:00:31.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.903+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c004cc0 con 0x7f050c004610 2024-09-10T19:00:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c004cc0 con 0x7f050c004610 2024-09-10T19:00:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 10 client.25341.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 4 2024-09-10T19:00:31.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 0 2 2024-09-10T19:00:31.913 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:00:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:00:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 10 client.25341.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 1 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 4 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 10 client.25341.objecter _send_command 3 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c0022b0 con 0x7f050c004110 2024-09-10T19:00:31.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.907+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c004110 2024-09-10T19:00:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.911+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c0022b0 con 0x7f050c004110 2024-09-10T19:00:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.911+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:00:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.911+0000 7f051dffb640 10 client.25341.objecter _finish_command 3 = system:0 2024-09-10T19:00:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.911+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 4 2024-09-10T19:00:31.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.911+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 1 3 2024-09-10T19:00:31.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 10 client.25341.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:31.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 4 2024-09-10T19:00:31.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 4 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c003fa0 osd=1 3 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 1 4 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 4 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 10 client.25341.objecter _send_command 4 2024-09-10T19:00:31.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.923+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c001890 con 0x7f050c004110 2024-09-10T19:00:31.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c001890 con 0x7f050c004110 2024-09-10T19:00:31.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:31.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f051dffb640 10 client.25341.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:31.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 4 2024-09-10T19:00:31.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 1 4 2024-09-10T19:00:31.933 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 10 client.25341.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 5 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 5 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 2 5 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 4 2024-09-10T19:00:31.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 10 client.25341.objecter _send_command 5 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.927+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c003df0 con 0x7f050c00d350 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c00d350 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c003df0 con 0x7f050c00d350 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 10 client.25341.objecter _finish_command 5 = system:0 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 4 2024-09-10T19:00:31.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.931+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 2 5 2024-09-10T19:00:31.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 10 client.25341.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 6 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 6 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c0035e0 osd=2 3 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 2 6 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 4 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 10 client.25341.objecter _send_command 6 2024-09-10T19:00:31.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c003060 con 0x7f050c00d350 2024-09-10T19:00:31.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c003060 con 0x7f050c00d350 2024-09-10T19:00:31.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:31.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f051dffb640 10 client.25341.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:31.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 4 2024-09-10T19:00:31.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 2 6 2024-09-10T19:00:31.958 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:00:31.958 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:00:31.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 10 client.25341.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 7 2024-09-10T19:00:31.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 7 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 3 7 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 4 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 10 client.25341.objecter _send_command 7 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.951+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c00d010 con 0x7f050c010e50 2024-09-10T19:00:31.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c010e50 2024-09-10T19:00:31.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:31.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c00d010 con 0x7f050c010e50 2024-09-10T19:00:31.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:00:31.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 10 client.25341.objecter _finish_command 7 = system:0 2024-09-10T19:00:31.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 4 2024-09-10T19:00:31.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.955+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 3 7 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 10 client.25341.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 8 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 8 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c00c990 osd=3 3 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 3 8 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 4 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 10 client.25341.objecter _send_command 8 2024-09-10T19:00:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.967+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c0132d0 con 0x7f050c010e50 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c0132d0 con 0x7f050c010e50 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 10 client.25341.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 4 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 3 8 2024-09-10T19:00:31.976 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 10 client.25341.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 9 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 9 2024-09-10T19:00:31.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 4 9 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 4 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 10 client.25341.objecter _send_command 9 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c013530 con 0x7f050c016760 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.971+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c016760 2024-09-10T19:00:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.975+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c013530 con 0x7f050c016760 2024-09-10T19:00:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.975+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:00:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.975+0000 7f051dffb640 10 client.25341.objecter _finish_command 9 = system:0 2024-09-10T19:00:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.975+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 4 2024-09-10T19:00:31.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.975+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 4 9 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 10 client.25341.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 10 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 10 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c0132d0 osd=4 3 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 4 10 2024-09-10T19:00:31.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 4 2024-09-10T19:00:31.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 10 client.25341.objecter _send_command 10 2024-09-10T19:00:31.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.987+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c013530 con 0x7f050c016760 2024-09-10T19:00:31.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c013530 con 0x7f050c016760 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f051dffb640 10 client.25341.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 4 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 4 10 2024-09-10T19:00:31.997 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:00:31.998 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:00:31.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 10 client.25341.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:31.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 11 2024-09-10T19:00:31.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 11 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 5 11 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 4 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 10 client.25341.objecter _send_command 11 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.991+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c01c610 con 0x7f050c01c120 2024-09-10T19:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c01c120 2024-09-10T19:00:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c01c610 con 0x7f050c01c120 2024-09-10T19:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 10 client.25341.objecter _finish_command 11 = system:0 2024-09-10T19:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 4 2024-09-10T19:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:31.995+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 5 11 2024-09-10T19:00:32.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:32.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 10 client.25341.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:32.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 12 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 12 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c013530 osd=5 3 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 5 12 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 4 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 10 client.25341.objecter _send_command 12 2024-09-10T19:00:32.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.011+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c01c610 con 0x7f050c01c120 2024-09-10T19:00:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c01c610 con 0x7f050c01c120 2024-09-10T19:00:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 10 client.25341.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:32.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 4 2024-09-10T19:00:32.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 5 12 2024-09-10T19:00:32.019 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:00:32.019 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:00:32.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 10 client.25341.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:32.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 13 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 13 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 6 13 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 4 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 10 client.25341.objecter _send_command 13 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c01c8e0 con 0x7f050c01f050 2024-09-10T19:00:32.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c01f050 2024-09-10T19:00:32.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c01c8e0 con 0x7f050c01f050 2024-09-10T19:00:32.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:00:32.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 10 client.25341.objecter _finish_command 13 = system:0 2024-09-10T19:00:32.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 4 2024-09-10T19:00:32.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.015+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 6 13 2024-09-10T19:00:32.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 10 client.25341.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:32.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 14 2024-09-10T19:00:32.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 14 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c01c610 osd=6 3 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 6 14 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 4 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 10 client.25341.objecter _send_command 14 2024-09-10T19:00:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.035+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c01c8e0 con 0x7f050c01f050 2024-09-10T19:00:32.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c01c8e0 con 0x7f050c01f050 2024-09-10T19:00:32.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:32.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 10 client.25341.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:32.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 4 2024-09-10T19:00:32.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 6 14 2024-09-10T19:00:32.045 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:00:32.045 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:00:32.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 10 client.25341.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 15 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 15 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 7 15 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 4 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 10 client.25341.objecter _send_command 15 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f050c01ae70 con 0x7f050c01aa20 2024-09-10T19:00:32.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.039+0000 7f051dffb640 10 client.25341.objecter ms_handle_connect 0x7f050c01aa20 2024-09-10T19:00:32.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.043+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f050c01ae70 con 0x7f050c01aa20 2024-09-10T19:00:32.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.043+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:00:32.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.043+0000 7f051dffb640 10 client.25341.objecter _finish_command 15 = system:0 2024-09-10T19:00:32.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.043+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 4 2024-09-10T19:00:32.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.043+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 7 15 2024-09-10T19:00:32.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 10 client.25341.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:00:32.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign -1 16 2024-09-10T19:00:32.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 20 client.25341.objecter _get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_remove -1 16 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 20 client.25341.objecter get_session s=0x7f050c01c8e0 osd=7 3 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 15 client.25341.objecter _session_command_op_assign 7 16 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 4 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 10 client.25341.objecter _send_command 16 2024-09-10T19:00:32.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.063+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f050c01ae70 con 0x7f050c01aa20 2024-09-10T19:00:32.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f051dffb640 1 -- 172.21.15.83:0/2604873198 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f050c01ae70 con 0x7f050c01aa20 2024-09-10T19:00:32.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f051dffb640 10 client.25341.objecter ms_dispatch 0x7f0528000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:32.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f051dffb640 10 client.25341.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:32.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f051dffb640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 4 2024-09-10T19:00:32.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f051dffb640 15 client.25341.objecter _session_command_op_remove 7 16 2024-09-10T19:00:32.073 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:00:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:00:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 librados: watch_flush enter 2024-09-10T19:00:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 librados: watch_flush exit 2024-09-10T19:00:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.0 2024-09-10T19:00:32.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6801/2535527483 conn(0x7f050c004610 legacy=0x7f050c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c005810 osd=0 1 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.1 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6805/2309624637 conn(0x7f050c004110 legacy=0x7f050c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c003fa0 osd=1 1 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.2 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6809/386094220 conn(0x7f050c00d350 legacy=0x7f050c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0035e0 osd=2 1 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.3 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6813/1279688021 conn(0x7f050c010e50 legacy=0x7f050c011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c00c990 osd=3 1 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.4 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.165:6800/3225515265 conn(0x7f050c016760 legacy=0x7f050c018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c0132d0 osd=4 1 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.5 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.165:6804/571173777 conn(0x7f050c01c120 legacy=0x7f050c011990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c013530 osd=5 1 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.6 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.165:6808/1317922936 conn(0x7f050c01f050 legacy=0x7f050c021430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c610 osd=6 1 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter close_session for osd.7 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.165:6812/1610151650 conn(0x7f050c01aa20 legacy=0x7f050c023cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter put_session s=0x7f050c01c8e0 osd=7 1 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 20 client.25341.objecter shutdown clearing up homeless session... 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 10 client.25341.objecter successfully canceled tick 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 reap_dead start 2024-09-10T19:00:32.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 reap_dead start 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 reap_dead start 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 reap_dead start 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052df71640 1 -- 172.21.15.83:0/2604873198 reap_dead start 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6800/3071793735 conn(0x7f0514071780 legacy=0x7f0514073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> v1:172.21.15.83:6789/0 conn(0x7f05281077d0 legacy=0x7f052811d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 shutdown_connections 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.067+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 >> 172.21.15.83:0/2604873198 conn(0x7f05280f7090 msgr2=0x7f05280f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.071+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 shutdown_connections 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.071+0000 7f052f9d4640 1 -- 172.21.15.83:0/2604873198 wait complete. 2024-09-10T19:00:32.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:32.071+0000 7f052f9d4640 1 librados: shutdown 2024-09-10T19:00:32.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:32.131 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:32.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:32.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.895742+0000 osd.3 (osd.3) 276 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:00:32.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:30.901190+0000 osd.3 (osd.3) 277 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:00:32.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.127048+0000 osd.5 (osd.5) 245 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:00:32.247 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.135470+0000 osd.5 (osd.5) 246 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:00:32.248 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.172026+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:00:32.248 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.174502+0000 osd.2 (osd.2) 193 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:00:32.248 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.183014+0000 osd.2 (osd.2) 194 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:00:32.248 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:32 smithi165 bash[22591]: audit 2024-09-10T19:00:31.889004+0000 mon.a (mon.0) 2621 : audit [DBG] from='client.? v1:172.21.15.83:0/2604873198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:32.270 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.895742+0000 osd.3 (osd.3) 276 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:30.901190+0000 osd.3 (osd.3) 277 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.127048+0000 osd.5 (osd.5) 245 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.135470+0000 osd.5 (osd.5) 246 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.172026+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.174502+0000 osd.2 (osd.2) 193 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.183014+0000 osd.2 (osd.2) 194 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:00:32.350 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18629]: audit 2024-09-10T19:00:31.889004+0000 mon.a (mon.0) 2621 : audit [DBG] from='client.? v1:172.21.15.83:0/2604873198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.895742+0000 osd.3 (osd.3) 276 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:30.901190+0000 osd.3 (osd.3) 277 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.127048+0000 osd.5 (osd.5) 245 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.135470+0000 osd.5 (osd.5) 246 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.172026+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.174502+0000 osd.2 (osd.2) 193 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.183014+0000 osd.2 (osd.2) 194 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:00:32.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[32381]: audit 2024-09-10T19:00:31.889004+0000 mon.a (mon.0) 2621 : audit [DBG] from='client.? v1:172.21.15.83:0/2604873198' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:32.351 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:32.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:32.407 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:32.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:32.547 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:32.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:32.556 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:00:32.556 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:00:32.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:32.685 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:32.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:32.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:32] "GET /metrics HTTP/1.1" 200 228896 "" "Prometheus/2.43.0" 2024-09-10T19:00:32.822 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:32.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:32.962 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:32.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:33.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.102 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:33.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:33.234 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:33.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.930431+0000 osd.3 (osd.3) 278 : cluster [DBG] 12.b scrub starts 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:31.933958+0000 osd.3 (osd.3) 279 : cluster [DBG] 12.b scrub ok 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:32.108394+0000 mgr.y (mgr.24526) 454 : cluster [DBG] pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:32.150480+0000 osd.0 (osd.0) 244 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:32.153908+0000 osd.0 (osd.0) 245 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:32.168677+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:00:33.350 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[32381]: cluster 2024-09-10T19:00:32.172993+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.930431+0000 osd.3 (osd.3) 278 : cluster [DBG] 12.b scrub starts 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:31.933958+0000 osd.3 (osd.3) 279 : cluster [DBG] 12.b scrub ok 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:32.108394+0000 mgr.y (mgr.24526) 454 : cluster [DBG] pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:32.150480+0000 osd.0 (osd.0) 244 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:32.153908+0000 osd.0 (osd.0) 245 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:32.168677+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:00:33.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:33 smithi083 bash[18629]: cluster 2024-09-10T19:00:32.172993+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:00:33.368 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:33.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.930431+0000 osd.3 (osd.3) 278 : cluster [DBG] 12.b scrub starts 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:31.933958+0000 osd.3 (osd.3) 279 : cluster [DBG] 12.b scrub ok 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:32.108394+0000 mgr.y (mgr.24526) 454 : cluster [DBG] pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:32.150480+0000 osd.0 (osd.0) 244 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:32.153908+0000 osd.0 (osd.0) 245 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:32.168677+0000 osd.7 (osd.7) 264 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:00:33.487 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:33 smithi165 bash[22591]: cluster 2024-09-10T19:00:32.172993+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:00:33.508 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:33.612 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.646 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:33.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.785 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:33.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.922 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:33.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- 172.21.15.83:0/2384744366 >> v1:172.21.15.83:6789/0 conn(0x7f55dc0701c0 legacy=0x7f55dc10e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- 172.21.15.83:0/2384744366 shutdown_connections 2024-09-10T19:00:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- 172.21.15.83:0/2384744366 >> 172.21.15.83:0/2384744366 conn(0x7f55dc0f81a0 msgr2=0x7f55dc0fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- 172.21.15.83:0/2384744366 shutdown_connections 2024-09-10T19:00:34.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- 172.21.15.83:0/2384744366 wait complete. 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 librados: starting msgr at 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 librados: starting objecter 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 Processor -- start 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- start start 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 librados: setting wanted keys 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 librados: calling monclient init 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55dc10f440 con 0x7f55dc10f7e0 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55dc1239e0 con 0x7f55dc0716e0 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e3447640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55dc124b80 con 0x7f55dc0701c0 2024-09-10T19:00:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e11e3640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f55dc0701c0 0x7f55dc10e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39630/0 (socket says 172.21.15.83:39630) 2024-09-10T19:00:34.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e11e3640 1 -- 172.21.15.83:0/3725847594 learned_addr learned my addr 172.21.15.83:0/3725847594 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:34.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55e19e4640 1 --1- 172.21.15.83:0/3725847594 >> v1:172.21.15.83:6789/0 conn(0x7f55dc10f7e0 0x7f55dc1212b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60530/0 (socket says 172.21.15.83:60530) 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter ms_handle_connect 0x7f55dc0701c0 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter ms_handle_connect 0x7f55dc10f7e0 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter ms_handle_connect 0x7f55dc0716e0 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63630952 0 0) 0x7f55dc10f440 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c8003720 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.003+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765990325 0 0) 0x7f55dc124b80 con 0x7f55dc0701c0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55dc10f440 con 0x7f55dc0701c0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569906907 0 0) 0x7f55dc1239e0 con 0x7f55dc0716e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55dc124b80 con 0x7f55dc0716e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1193618068 0 0) 0x7f55c8003720 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55dc1239e0 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f55d4001cd0 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953262671 0 0) 0x7f55dc124b80 con 0x7f55dc0716e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c8003720 con 0x7f55dc0716e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f55d8001430 con 0x7f55dc0716e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2458390501 0 0) 0x7f55dc1239e0 con 0x7f55dc10f7e0 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 >> v1:172.21.15.83:6790/0 conn(0x7f55dc0701c0 legacy=0x7f55dc10e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 >> v1:172.21.15.165:6789/0 conn(0x7f55dc0716e0 legacy=0x7f55dc11db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc125d60 con 0x7f55dc10f7e0 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc122a30 con 0x7f55dc10f7e0 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55e3447640 10 client.25347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc122e80 con 0x7f55dc10f7e0 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55e3447640 1 librados: init done 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f55d40022d0 con 0x7f55dc10f7e0 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f55d4005d60 con 0x7f55dc10f7e0 2024-09-10T19:00:34.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c0005010 con 0x7f55dc10f7e0 2024-09-10T19:00:34.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.007+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f55d4001e70 con 0x7f55dc10f7e0 2024-09-10T19:00:34.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(454..454 src has 69..454) v4 ==== 8060+0+0 (unknown 1326074345 0 0) 0x7f55d408a800 con 0x7f55dc10f7e0 2024-09-10T19:00:34.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 10 client.25347.objecter ms_dispatch 0x7f55dc000b90 osd_map(454..454 src has 69..454) v4 2024-09-10T19:00:34.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 3 client.25347.objecter handle_osd_map got epochs [454,454] > 0 2024-09-10T19:00:34.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 3 client.25347.objecter handle_osd_map decoding full epoch 454 2024-09-10T19:00:34.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 20 client.25347.objecter dump_active .. 0 homeless 2024-09-10T19:00:34.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.011+0000 7f55d1ffb640 10 client.25347.objecter ms_handle_connect 0x7f55c80714a0 2024-09-10T19:00:34.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.015+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f55d4056d80 con 0x7f55dc10f7e0 2024-09-10T19:00:34.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:34.051 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:34.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:34.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:34.223+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f55c0005840 con 0x7f55dc10f7e0 2024-09-10T19:00:34.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.089266+0000 osd.5 (osd.5) 247 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:00:34.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.093953+0000 osd.5 (osd.5) 248 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:00:34.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.112823+0000 osd.0 (osd.0) 246 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:00:34.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.116047+0000 osd.0 (osd.0) 247 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:00:34.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.137915+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.f scrub starts 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[32381]: cluster 2024-09-10T19:00:33.140930+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.f scrub ok 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.089266+0000 osd.5 (osd.5) 247 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.093953+0000 osd.5 (osd.5) 248 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.112823+0000 osd.0 (osd.0) 246 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.116047+0000 osd.0 (osd.0) 247 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.137915+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.f scrub starts 2024-09-10T19:00:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:34 smithi083 bash[18629]: cluster 2024-09-10T19:00:33.140930+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.f scrub ok 2024-09-10T19:00:34.318 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:34.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.456 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:00:34.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.089266+0000 osd.5 (osd.5) 247 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:00:34.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.093953+0000 osd.5 (osd.5) 248 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:00:34.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.112823+0000 osd.0 (osd.0) 246 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:00:34.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.116047+0000 osd.0 (osd.0) 247 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:00:34.460 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.137915+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.f scrub starts 2024-09-10T19:00:34.460 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:34 smithi165 bash[22591]: cluster 2024-09-10T19:00:33.140930+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.f scrub ok 2024-09-10T19:00:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:34.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:34.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:34.729 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:34.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.867 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:34.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:34.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:35.005 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:35.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:35.143 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:35.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:35.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.215+0000 7f55d1ffb640 1 -- 172.21.15.83:0/3725847594 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v455) v1 ==== 87+0+0 (unknown 3947668102 0 0) 0x7f55d4056720 con 0x7f55dc10f7e0 2024-09-10T19:00:35.221 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 10 librados: watch_flush enter 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 10 librados: watch_flush exit 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 20 client.25347.objecter shutdown clearing up homeless session... 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 10 client.25347.objecter successfully canceled tick 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 >> v1:172.21.15.83:6800/3071793735 conn(0x7f55c80714a0 legacy=0x7f55c8073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 >> v1:172.21.15.83:6789/0 conn(0x7f55dc10f7e0 legacy=0x7f55dc1212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 shutdown_connections 2024-09-10T19:00:35.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 >> 172.21.15.83:0/3725847594 conn(0x7f55dc0f81a0 msgr2=0x7f55dc0f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:35.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 shutdown_connections 2024-09-10T19:00:35.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 -- 172.21.15.83:0/3725847594 wait complete. 2024-09-10T19:00:35.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:35.219+0000 7f55e3447640 1 librados: shutdown 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[18629]: cluster 2024-09-10T19:00:34.109141+0000 mgr.y (mgr.24526) 455 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[18629]: audit 2024-09-10T19:00:34.230299+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[18629]: cluster 2024-09-10T19:00:34.706019+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.d scrub starts 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[18629]: cluster 2024-09-10T19:00:34.708424+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.d scrub ok 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[32381]: cluster 2024-09-10T19:00:34.109141+0000 mgr.y (mgr.24526) 455 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[32381]: audit 2024-09-10T19:00:34.230299+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[32381]: cluster 2024-09-10T19:00:34.706019+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.d scrub starts 2024-09-10T19:00:35.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:35 smithi083 bash[32381]: cluster 2024-09-10T19:00:34.708424+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.d scrub ok 2024-09-10T19:00:35.247 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:35.271 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:35.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:35.409 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:35.513 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:35.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:35 smithi165 bash[22591]: cluster 2024-09-10T19:00:34.109141+0000 mgr.y (mgr.24526) 455 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:00:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:35 smithi165 bash[22591]: audit 2024-09-10T19:00:34.230299+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:00:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:35 smithi165 bash[22591]: cluster 2024-09-10T19:00:34.706019+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.d scrub starts 2024-09-10T19:00:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:35 smithi165 bash[22591]: cluster 2024-09-10T19:00:34.708424+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.d scrub ok 2024-09-10T19:00:35.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:35.689 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:35.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:35.828 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:35.932 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:35.964 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:35.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:36.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:36.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:36.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:36.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:36.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:36.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:36.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:36.493 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[18629]: cluster 2024-09-10T19:00:35.203314+0000 mon.a (mon.0) 2623 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[18629]: audit 2024-09-10T19:00:35.215877+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[18629]: cluster 2024-09-10T19:00:35.216133+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[18629]: cluster 2024-09-10T19:00:35.252409+0000 osd.4 (osd.4) 286 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[18629]: cluster 2024-09-10T19:00:35.255350+0000 osd.4 (osd.4) 287 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[32381]: cluster 2024-09-10T19:00:35.203314+0000 mon.a (mon.0) 2623 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[32381]: audit 2024-09-10T19:00:35.215877+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[32381]: cluster 2024-09-10T19:00:35.216133+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[32381]: cluster 2024-09-10T19:00:35.252409+0000 osd.4 (osd.4) 286 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:00:36.494 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:36 smithi083 bash[32381]: cluster 2024-09-10T19:00:35.255350+0000 osd.4 (osd.4) 287 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:36 smithi165 bash[22591]: cluster 2024-09-10T19:00:35.203314+0000 mon.a (mon.0) 2623 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:36 smithi165 bash[22591]: audit 2024-09-10T19:00:35.215877+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? v1:172.21.15.83:0/3725847594' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:00:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:36 smithi165 bash[22591]: cluster 2024-09-10T19:00:35.216133+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:00:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:36 smithi165 bash[22591]: cluster 2024-09-10T19:00:35.252409+0000 osd.4 (osd.4) 286 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:00:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:36 smithi165 bash[22591]: cluster 2024-09-10T19:00:35.255350+0000 osd.4 (osd.4) 287 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:00:36.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:36.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:36.790 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:36.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:36.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:36.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:37.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:37.066 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:37.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:37.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:37 smithi083 bash[18629]: cluster 2024-09-10T19:00:36.110970+0000 mgr.y (mgr.24526) 456 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-10T19:00:37.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:37 smithi083 bash[32381]: cluster 2024-09-10T19:00:36.110970+0000 mgr.y (mgr.24526) 456 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-10T19:00:37.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:37.341 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:37.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:37.477 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:37.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:37 smithi165 bash[22591]: cluster 2024-09-10T19:00:36.110970+0000 mgr.y (mgr.24526) 456 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-09-10T19:00:37.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:37.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:37.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:37.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:37.750 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:37.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:37.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:37.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:37.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:38.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:38.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.142 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:37 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:38.162 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:38.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.298 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:38.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- 172.21.15.83:0/2616511853 >> v1:172.21.15.83:6790/0 conn(0x7fd7240fbe50 legacy=0x7fd7241099f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:38.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- 172.21.15.83:0/2616511853 shutdown_connections 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- 172.21.15.83:0/2616511853 >> 172.21.15.83:0/2616511853 conn(0x7fd724075290 msgr2=0x7fd7240f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- 172.21.15.83:0/2616511853 shutdown_connections 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- 172.21.15.83:0/2616511853 wait complete. 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 librados: starting msgr at 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 librados: starting objecter 2024-09-10T19:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 Processor -- start 2024-09-10T19:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- start start 2024-09-10T19:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 librados: setting wanted keys 2024-09-10T19:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 librados: calling monclient init 2024-09-10T19:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd72411e230 con 0x7fd7240fb370 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd72411f3f0 con 0x7fd72410b1a0 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd72ba75640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7241205f0 con 0x7fd7240fbe50 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd729811640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd7240fb370 0x7fd724107590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60540/0 (socket says 172.21.15.83:60540) 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd729811640 1 -- 172.21.15.83:0/2961613143 learned_addr learned my addr 172.21.15.83:0/2961613143 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd729010640 1 --1- 172.21.15.83:0/2961613143 >> v1:172.21.15.83:6790/0 conn(0x7fd7240fbe50 0x7fd724107ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39648/0 (socket says 172.21.15.83:39648) 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.359+0000 7fd7227fc640 10 client.?.objecter ms_handle_connect 0x7fd7240fbe50 2024-09-10T19:00:38.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.?.objecter ms_handle_connect 0x7fd7240fb370 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786999480 0 0) 0x7fd7241205f0 con 0x7fd7240fbe50 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd710003120 con 0x7fd7240fbe50 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736174831 0 0) 0x7fd72411e230 con 0x7fd7240fb370 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7241205f0 con 0x7fd7240fb370 2024-09-10T19:00:38.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.?.objecter ms_handle_connect 0x7fd72410b1a0 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918363232 0 0) 0x7fd72411f3f0 con 0x7fd72410b1a0 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd72411e230 con 0x7fd72410b1a0 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2435243491 0 0) 0x7fd7241205f0 con 0x7fd7240fb370 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd72411f3f0 con 0x7fd7240fb370 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4255461623 0 0) 0x7fd710003120 con 0x7fd7240fbe50 2024-09-10T19:00:38.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7241205f0 con 0x7fd7240fbe50 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd71c0019f0 con 0x7fd7240fb370 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd70c001950 con 0x7fd7240fbe50 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760960746 0 0) 0x7fd72411f3f0 con 0x7fd7240fb370 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 >> v1:172.21.15.83:6790/0 conn(0x7fd7240fbe50 legacy=0x7fd724107ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 >> v1:172.21.15.165:6789/0 conn(0x7fd72410b1a0 legacy=0x7fd72411c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7241217f0 con 0x7fd7240fb370 2024-09-10T19:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72411e420 con 0x7fd7240fb370 2024-09-10T19:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd72ba75640 10 client.25353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72411e8f0 con 0x7fd7240fb370 2024-09-10T19:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd72ba75640 1 librados: init done 2024-09-10T19:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd71c002440 con 0x7fd7240fb370 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd71c005a80 con 0x7fd7240fb370 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd704005010 con 0x7fd7240fb370 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd71c0025b0 con 0x7fd7240fb370 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(455..455 src has 69..455) v4 ==== 8032+0+0 (unknown 3838672855 0 0) 0x7fd71c08a680 con 0x7fd7240fb370 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 10 client.25353.objecter ms_dispatch 0x7fd724000b90 osd_map(455..455 src has 69..455) v4 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 3 client.25353.objecter handle_osd_map got epochs [455,455] > 0 2024-09-10T19:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 3 client.25353.objecter handle_osd_map decoding full epoch 455 2024-09-10T19:00:38.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.363+0000 7fd7227fc640 20 client.25353.objecter dump_active .. 0 homeless 2024-09-10T19:00:38.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.367+0000 7fd7227fc640 10 client.25353.objecter ms_handle_connect 0x7fd7100716f0 2024-09-10T19:00:38.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.371+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd71c0567a0 con 0x7fd7240fb370 2024-09-10T19:00:38.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:38.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:38.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:38.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:38.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.539+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd704000b90 con 0x7fd7100716f0 2024-09-10T19:00:38.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.563+0000 7fd7227fc640 1 -- 172.21.15.83:0/2961613143 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+626937 (unknown 2965378022 0 3082479272) 0x7fd704000b90 con 0x7fd7100716f0 2024-09-10T19:00:38.575 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:00:38.578 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.567+0000 7fd72ba75640 10 librados: watch_flush enter 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.567+0000 7fd72ba75640 10 librados: watch_flush exit 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.567+0000 7fd72ba75640 20 client.25353.objecter shutdown clearing up homeless session... 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.567+0000 7fd72ba75640 10 client.25353.objecter successfully canceled tick 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.567+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd7100716f0 legacy=0x7fd710073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 >> v1:172.21.15.83:6789/0 conn(0x7fd7240fb370 legacy=0x7fd724107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 shutdown_connections 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 >> 172.21.15.83:0/2961613143 conn(0x7fd724075290 msgr2=0x7fd7240f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:38.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 shutdown_connections 2024-09-10T19:00:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 -- 172.21.15.83:0/2961613143 wait complete. 2024-09-10T19:00:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:38.571+0000 7fd72ba75640 1 librados: shutdown 2024-09-10T19:00:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:38.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:38.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:38.716 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:38.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:00:38.854 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:38.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:38.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:38.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:39.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:39.133 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:39.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:39.269 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:39 smithi165 bash[22591]: audit 2024-09-10T19:00:37.993718+0000 mgr.y (mgr.24526) 457 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:39 smithi165 bash[22591]: cluster 2024-09-10T19:00:38.111738+0000 mgr.y (mgr.24526) 458 : cluster [DBG] pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:39.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:39 smithi165 bash[22591]: cluster 2024-09-10T19:00:39.148501+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:00:39.368 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2024-09-10T19:00:38.111244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8791988,"num_objects":359,"num_object_clones":2,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":2084,"num_read_kb":1837,"num_write":1447,"num_write_kb":14370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2859,"ondisk_log_size":2859,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2505856,"kb_used_data":81240,"kb_used_omap":13,"kb_used_meta":2424498,"kb_avail":747455360,"statfs":{"total":767960285184,"available":765394288640,"internally_reserved":0,"allocated":83189760,"data_stored":58464316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2482686656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004373"},"pg_stats":[{"pgid":"215.1f","version":"442'2","reported_seq":27,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168512+0000","last_change":"2024-09-10T19:00:04.268476+0000","last_active":"2024-09-10T19:00:31.168512+0000","last_peered":"2024-09-10T19:00:31.168512+0000","last_clean":"2024-09-10T19:00:31.168512+0000","last_became_active":"2024-09-10T19:00:04.267732+0000","last_became_peered":"2024-09-10T19:00:04.267732+0000","last_unstale":"2024-09-10T19:00:31.168512+0000","last_undegraded":"2024-09-10T19:00:31.168512+0000","last_fullsized":"2024-09-10T19:00:31.168512+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.164189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.1e","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435452+0000","last_change":"2024-09-10T19:00:04.291945+0000","last_active":"2024-09-10T19:00:31.435452+0000","last_peered":"2024-09-10T19:00:31.435452+0000","last_clean":"2024-09-10T19:00:31.435452+0000","last_became_active":"2024-09-10T19:00:04.290965+0000","last_became_peered":"2024-09-10T19:00:04.290965+0000","last_unstale":"2024-09-10T19:00:31.435452+0000","last_undegraded":"2024-09-10T19:00:31.435452+0000","last_fullsized":"2024-09-10T19:00:31.435452+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.075695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.1d","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180166+0000","last_change":"2024-09-10T19:00:04.276088+0000","last_active":"2024-09-10T19:00:31.180166+0000","last_peered":"2024-09-10T19:00:31.180166+0000","last_clean":"2024-09-10T19:00:31.180166+0000","last_became_active":"2024-09-10T19:00:04.275821+0000","last_became_peered":"2024-09-10T19:00:04.275821+0000","last_unstale":"2024-09-10T19:00:31.180166+0000","last_undegraded":"2024-09-10T19:00:31.180166+0000","last_fullsized":"2024-09-10T19:00:31.180166+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.739652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.1c","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174929+0000","last_change":"2024-09-10T19:00:04.286817+0000","last_active":"2024-09-10T19:00:31.174929+0000","last_peered":"2024-09-10T19:00:31.174929+0000","last_clean":"2024-09-10T19:00:31.174929+0000","last_became_active":"2024-09-10T19:00:04.286469+0000","last_became_peered":"2024-09-10T19:00:04.286469+0000","last_unstale":"2024-09-10T19:00:31.174929+0000","last_undegraded":"2024-09-10T19:00:31.174929+0000","last_fullsized":"2024-09-10T19:00:31.174929+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.583907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1b","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174795+0000","last_change":"2024-09-10T19:00:04.288725+0000","last_active":"2024-09-10T19:00:31.174795+0000","last_peered":"2024-09-10T19:00:31.174795+0000","last_clean":"2024-09-10T19:00:31.174795+0000","last_became_active":"2024-09-10T19:00:04.288361+0000","last_became_peered":"2024-09-10T19:00:04.288361+0000","last_unstale":"2024-09-10T19:00:31.174795+0000","last_undegraded":"2024-09-10T19:00:31.174795+0000","last_fullsized":"2024-09-10T19:00:31.174795+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.963316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1a","version":"451'4","reported_seq":28,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577888+0000","last_change":"2024-09-10T19:00:04.275768+0000","last_active":"2024-09-10T19:00:31.577888+0000","last_peered":"2024-09-10T19:00:31.577888+0000","last_clean":"2024-09-10T19:00:31.577888+0000","last_became_active":"2024-09-10T19:00:04.275245+0000","last_became_peered":"2024-09-10T19:00:04.275245+0000","last_unstale":"2024-09-10T19:00:31.577888+0000","last_undegraded":"2024-09-10T19:00:31.577888+0000","last_fullsized":"2024-09-10T19:00:31.577888+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.008216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.19","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174274+0000","last_change":"2024-09-10T19:00:04.283513+0000","last_active":"2024-09-10T19:00:31.174274+0000","last_peered":"2024-09-10T19:00:31.174274+0000","last_clean":"2024-09-10T19:00:31.174274+0000","last_became_active":"2024-09-10T19:00:04.283166+0000","last_became_peered":"2024-09-10T19:00:04.283166+0000","last_unstale":"2024-09-10T19:00:31.174274+0000","last_undegraded":"2024-09-10T19:00:31.174274+0000","last_fullsized":"2024-09-10T19:00:31.174274+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.031130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.18","version":"451'6","reported_seq":32,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174078+0000","last_change":"2024-09-10T19:00:04.280704+0000","last_active":"2024-09-10T19:00:31.174078+0000","last_peered":"2024-09-10T19:00:31.174078+0000","last_clean":"2024-09-10T19:00:31.174078+0000","last_became_active":"2024-09-10T19:00:04.280190+0000","last_became_peered":"2024-09-10T19:00:04.280190+0000","last_unstale":"2024-09-10T19:00:31.174078+0000","last_undegraded":"2024-09-10T19:00:31.174078+0000","last_fullsized":"2024-09-10T19:00:31.174078+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.454766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.17","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.182715+0000","last_change":"2024-09-10T19:00:04.279347+0000","last_active":"2024-09-10T19:00:31.182715+0000","last_peered":"2024-09-10T19:00:31.182715+0000","last_clean":"2024-09-10T19:00:31.182715+0000","last_became_active":"2024-09-10T19:00:04.277956+0000","last_became_peered":"2024-09-10T19:00:04.277956+0000","last_unstale":"2024-09-10T19:00:31.182715+0000","last_undegraded":"2024-09-10T19:00:31.182715+0000","last_fullsized":"2024-09-10T19:00:31.182715+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.373100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.16","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173015+0000","last_change":"2024-09-10T19:00:04.287474+0000","last_active":"2024-09-10T19:00:31.173015+0000","last_peered":"2024-09-10T19:00:31.173015+0000","last_clean":"2024-09-10T19:00:31.173015+0000","last_became_active":"2024-09-10T19:00:04.287210+0000","last_became_peered":"2024-09-10T19:00:04.287210+0000","last_unstale":"2024-09-10T19:00:31.173015+0000","last_undegraded":"2024-09-10T19:00:31.173015+0000","last_fullsized":"2024-09-10T19:00:31.173015+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.857969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.15","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.434967+0000","last_change":"2024-09-10T19:00:04.287919+0000","last_active":"2024-09-10T19:00:31.434967+0000","last_peered":"2024-09-10T19:00:31.434967+0000","last_clean":"2024-09-10T19:00:31.434967+0000","last_became_active":"2024-09-10T19:00:04.287638+0000","last_became_peered":"2024-09-10T19:00:04.287638+0000","last_unstale":"2024-09-10T19:00:31.434967+0000","last_undegraded":"2024-09-10T19:00:31.434967+0000","last_fullsized":"2024-09-10T19:00:31.434967+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.14","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167610+0000","last_change":"2024-09-10T19:00:04.275814+0000","last_active":"2024-09-10T19:00:31.167610+0000","last_peered":"2024-09-10T19:00:31.167610+0000","last_clean":"2024-09-10T19:00:31.167610+0000","last_became_active":"2024-09-10T19:00:04.275307+0000","last_became_peered":"2024-09-10T19:00:04.275307+0000","last_unstale":"2024-09-10T19:00:31.167610+0000","last_undegraded":"2024-09-10T19:00:31.167610+0000","last_fullsized":"2024-09-10T19:00:31.167610+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:03.936168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.13","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174106+0000","last_change":"2024-09-10T19:00:04.283918+0000","last_active":"2024-09-10T19:00:31.174106+0000","last_peered":"2024-09-10T19:00:31.174106+0000","last_clean":"2024-09-10T19:00:31.174106+0000","last_became_active":"2024-09-10T19:00:04.283556+0000","last_became_peered":"2024-09-10T19:00:04.283556+0000","last_unstale":"2024-09-10T19:00:31.174106+0000","last_undegraded":"2024-09-10T19:00:31.174106+0000","last_fullsized":"2024-09-10T19:00:31.174106+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.952005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.12","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.170666+0000","last_change":"2024-09-10T19:00:04.275283+0000","last_active":"2024-09-10T19:00:31.170666+0000","last_peered":"2024-09-10T19:00:31.170666+0000","last_clean":"2024-09-10T19:00:31.170666+0000","last_became_active":"2024-09-10T19:00:04.274956+0000","last_became_peered":"2024-09-10T19:00:04.274956+0000","last_unstale":"2024-09-10T19:00:31.170666+0000","last_undegraded":"2024-09-10T19:00:31.170666+0000","last_fullsized":"2024-09-10T19:00:31.170666+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.517350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.11","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178789+0000","last_change":"2024-09-10T19:00:04.287301+0000","last_active":"2024-09-10T19:00:31.178789+0000","last_peered":"2024-09-10T19:00:31.178789+0000","last_clean":"2024-09-10T19:00:31.178789+0000","last_became_active":"2024-09-10T19:00:04.287016+0000","last_became_peered":"2024-09-10T19:00:04.287016+0000","last_unstale":"2024-09-10T19:00:31.178789+0000","last_undegraded":"2024-09-10T19:00:31.178789+0000","last_fullsized":"2024-09-10T19:00:31.178789+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.617302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.10","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174747+0000","last_change":"2024-09-10T19:00:04.282956+0000","last_active":"2024-09-10T19:00:31.174747+0000","last_peered":"2024-09-10T19:00:31.174747+0000","last_clean":"2024-09-10T19:00:31.174747+0000","last_became_active":"2024-09-10T19:00:04.281332+0000","last_became_peered":"2024-09-10T19:00:04.281332+0000","last_unstale":"2024-09-10T19:00:31.174747+0000","last_undegraded":"2024-09-10T19:00:31.174747+0000","last_fullsized":"2024-09-10T19:00:31.174747+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:07.407596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.f","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179602+0000","last_change":"2024-09-10T19:00:04.274491+0000","last_active":"2024-09-10T19:00:31.179602+0000","last_peered":"2024-09-10T19:00:31.179602+0000","last_clean":"2024-09-10T19:00:31.179602+0000","last_became_active":"2024-09-10T19:00:04.274224+0000","last_became_peered":"2024-09-10T19:00:04.274224+0000","last_unstale":"2024-09-10T19:00:31.179602+0000","last_undegraded":"2024-09-10T19:00:31.179602+0000","last_fullsized":"2024-09-10T19:00:31.179602+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.187213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"215.e","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173128+0000","last_change":"2024-09-10T19:00:04.286254+0000","last_active":"2024-09-10T19:00:31.173128+0000","last_peered":"2024-09-10T19:00:31.173128+0000","last_clean":"2024-09-10T19:00:31.173128+0000","last_became_active":"2024-09-10T19:00:04.285951+0000","last_became_peered":"2024-09-10T19:00:04.285951+0000","last_unstale":"2024-09-10T19:00:31.173128+0000","last_undegraded":"2024-09-10T19:00:31.173128+0000","last_fullsized":"2024-09-10T19:00:31.173128+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.086526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.d","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177269+0000","last_change":"2024-09-10T19:00:04.276650+0000","last_active":"2024-09-10T19:00:31.177269+0000","last_peered":"2024-09-10T19:00:31.177269+0000","last_clean":"2024-09-10T19:00:31.177269+0000","last_became_active":"2024-09-10T19:00:04.276363+0000","last_became_peered":"2024-09-10T19:00:04.276363+0000","last_unstale":"2024-09-10T19:00:31.177269+0000","last_undegraded":"2024-09-10T19:00:31.177269+0000","last_fullsized":"2024-09-10T19:00:31.177269+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:14.699872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.c","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174066+0000","last_change":"2024-09-10T19:00:04.283152+0000","last_active":"2024-09-10T19:00:31.174066+0000","last_peered":"2024-09-10T19:00:31.174066+0000","last_clean":"2024-09-10T19:00:31.174066+0000","last_became_active":"2024-09-10T19:00:04.282635+0000","last_became_peered":"2024-09-10T19:00:04.282635+0000","last_unstale":"2024-09-10T19:00:31.174066+0000","last_undegraded":"2024-09-10T19:00:31.174066+0000","last_fullsized":"2024-09-10T19:00:31.174066+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.211376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"215.b","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174189+0000","last_change":"2024-09-10T19:00:04.292003+0000","last_active":"2024-09-10T19:00:31.174189+0000","last_peered":"2024-09-10T19:00:31.174189+0000","last_clean":"2024-09-10T19:00:31.174189+0000","last_became_active":"2024-09-10T19:00:04.291749+0000","last_became_peered":"2024-09-10T19:00:04.291749+0000","last_unstale":"2024-09-10T19:00:31.174189+0000","last_undegraded":"2024-09-10T19:00:31.174189+0000","last_fullsized":"2024-09-10T19:00:31.174189+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.927530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.a","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175649+0000","last_change":"2024-09-10T19:00:04.285065+0000","last_active":"2024-09-10T19:00:31.175649+0000","last_peered":"2024-09-10T19:00:31.175649+0000","last_clean":"2024-09-10T19:00:31.175649+0000","last_became_active":"2024-09-10T19:00:04.284826+0000","last_became_peered":"2024-09-10T19:00:04.284826+0000","last_unstale":"2024-09-10T19:00:31.175649+0000","last_undegraded":"2024-09-10T19:00:31.175649+0000","last_fullsized":"2024-09-10T19:00:31.175649+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.116268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.9","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.433433+0000","last_change":"2024-09-10T19:00:04.270585+0000","last_active":"2024-09-10T19:00:31.433433+0000","last_peered":"2024-09-10T19:00:31.433433+0000","last_clean":"2024-09-10T19:00:31.433433+0000","last_became_active":"2024-09-10T19:00:04.270285+0000","last_became_peered":"2024-09-10T19:00:04.270285+0000","last_unstale":"2024-09-10T19:00:31.433433+0000","last_undegraded":"2024-09-10T19:00:31.433433+0000","last_fullsized":"2024-09-10T19:00:31.433433+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.542757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"215.8","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.574813+0000","last_change":"2024-09-10T19:00:04.287128+0000","last_active":"2024-09-10T19:00:31.574813+0000","last_peered":"2024-09-10T19:00:31.574813+0000","last_clean":"2024-09-10T19:00:31.574813+0000","last_became_active":"2024-09-10T19:00:04.286817+0000","last_became_peered":"2024-09-10T19:00:04.286817+0000","last_unstale":"2024-09-10T19:00:31.574813+0000","last_undegraded":"2024-09-10T19:00:31.574813+0000","last_fullsized":"2024-09-10T19:00:31.574813+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:10.684928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.7","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175026+0000","last_change":"2024-09-10T19:00:04.293140+0000","last_active":"2024-09-10T19:00:31.175026+0000","last_peered":"2024-09-10T19:00:31.175026+0000","last_clean":"2024-09-10T19:00:31.175026+0000","last_became_active":"2024-09-10T19:00:04.292824+0000","last_became_peered":"2024-09-10T19:00:04.292824+0000","last_unstale":"2024-09-10T19:00:31.175026+0000","last_undegraded":"2024-09-10T19:00:31.175026+0000","last_fullsized":"2024-09-10T19:00:31.175026+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.962055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"215.6","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576952+0000","last_change":"2024-09-10T19:00:04.279607+0000","last_active":"2024-09-10T19:00:31.576952+0000","last_peered":"2024-09-10T19:00:31.576952+0000","last_clean":"2024-09-10T19:00:31.576952+0000","last_became_active":"2024-09-10T19:00:04.279164+0000","last_became_peered":"2024-09-10T19:00:04.279164+0000","last_unstale":"2024-09-10T19:00:31.576952+0000","last_undegraded":"2024-09-10T19:00:31.576952+0000","last_fullsized":"2024-09-10T19:00:31.576952+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.486237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.5","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576587+0000","last_change":"2024-09-10T19:00:04.272438+0000","last_active":"2024-09-10T19:00:31.576587+0000","last_peered":"2024-09-10T19:00:31.576587+0000","last_clean":"2024-09-10T19:00:31.576587+0000","last_became_active":"2024-09-10T19:00:04.272020+0000","last_became_peered":"2024-09-10T19:00:04.272020+0000","last_unstale":"2024-09-10T19:00:31.576587+0000","last_undegraded":"2024-09-10T19:00:31.576587+0000","last_fullsized":"2024-09-10T19:00:31.576587+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.089634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.4","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179173+0000","last_change":"2024-09-10T19:00:04.276737+0000","last_active":"2024-09-10T19:00:31.179173+0000","last_peered":"2024-09-10T19:00:31.179173+0000","last_clean":"2024-09-10T19:00:31.179173+0000","last_became_active":"2024-09-10T19:00:04.276433+0000","last_became_peered":"2024-09-10T19:00:04.276433+0000","last_unstale":"2024-09-10T19:00:31.179173+0000","last_undegraded":"2024-09-10T19:00:31.179173+0000","last_fullsized":"2024-09-10T19:00:31.179173+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.159916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"215.3","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166948+0000","last_change":"2024-09-10T19:00:04.272390+0000","last_active":"2024-09-10T19:00:31.166948+0000","last_peered":"2024-09-10T19:00:31.166948+0000","last_clean":"2024-09-10T19:00:31.166948+0000","last_became_active":"2024-09-10T19:00:04.272053+0000","last_became_peered":"2024-09-10T19:00:04.272053+0000","last_unstale":"2024-09-10T19:00:31.166948+0000","last_undegraded":"2024-09-10T19:00:31.166948+0000","last_fullsized":"2024-09-10T19:00:31.166948+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.331540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"215.2","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174659+0000","last_change":"2024-09-10T19:00:04.283670+0000","last_active":"2024-09-10T19:00:31.174659+0000","last_peered":"2024-09-10T19:00:31.174659+0000","last_clean":"2024-09-10T19:00:31.174659+0000","last_became_active":"2024-09-10T19:00:04.283381+0000","last_became_peered":"2024-09-10T19:00:04.283381+0000","last_unstale":"2024-09-10T19:00:31.174659+0000","last_undegraded":"2024-09-10T19:00:31.174659+0000","last_fullsized":"2024-09-10T19:00:31.174659+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.053377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"215.1","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575339+0000","last_change":"2024-09-10T19:00:04.287062+0000","last_active":"2024-09-10T19:00:31.575339+0000","last_peered":"2024-09-10T19:00:31.575339+0000","last_clean":"2024-09-10T19:00:31.575339+0000","last_became_active":"2024-09-10T19:00:04.286750+0000","last_became_peered":"2024-09-10T19:00:04.286750+0000","last_unstale":"2024-09-10T19:00:31.575339+0000","last_undegraded":"2024-09-10T19:00:31.575339+0000","last_fullsized":"2024-09-10T19:00:31.575339+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.558827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"215.0","version":"0'0","reported_seq":23,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180783+0000","last_change":"2024-09-10T19:00:04.277890+0000","last_active":"2024-09-10T19:00:31.180783+0000","last_peered":"2024-09-10T19:00:31.180783+0000","last_clean":"2024-09-10T19:00:31.180783+0000","last_became_active":"2024-09-10T19:00:04.277555+0000","last_became_peered":"2024-09-10T19:00:04.277555+0000","last_unstale":"2024-09-10T19:00:31.180783+0000","last_undegraded":"2024-09-10T19:00:31.180783+0000","last_fullsized":"2024-09-10T19:00:31.180783+0000","mapping_epoch":440,"log_start":"0'0","ondisk_log_start":"0'0","created":440,"last_epoch_clean":441,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:03.239097+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.239097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:27.693272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183736+0000","last_change":"2024-09-10T19:00:31.183736+0000","last_active":"2024-09-10T19:00:31.183736+0000","last_peered":"2024-09-10T19:00:31.183736+0000","last_clean":"2024-09-10T19:00:31.183736+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:00:31.183736+0000","last_undegraded":"2024-09-10T19:00:31.183736+0000","last_fullsized":"2024-09-10T19:00:31.183736+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.00083591500000000001,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1e","version":"377'67","reported_seq":458,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183961+0000","last_change":"2024-09-10T19:00:31.183961+0000","last_active":"2024-09-10T19:00:31.183961+0000","last_peered":"2024-09-10T19:00:31.183961+0000","last_clean":"2024-09-10T19:00:31.183961+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:00:31.183961+0000","last_undegraded":"2024-09-10T19:00:31.183961+0000","last_fullsized":"2024-09-10T19:00:31.183961+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'64","last_scrub_stamp":"2024-09-10T18:58:39.857682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.857682+0000","objects_scrubbed":6,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047380599999999997,"objects_trimmed":0,"snaptrim_duration":0.0033737160000000001,"stat_sum":{"num_bytes":64,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1d","version":"425'25","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180688+0000","last_change":"2024-09-10T19:00:31.180688+0000","last_active":"2024-09-10T19:00:31.180688+0000","last_peered":"2024-09-10T19:00:31.180688+0000","last_clean":"2024-09-10T19:00:31.180688+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:00:31.180688+0000","last_undegraded":"2024-09-10T19:00:31.180688+0000","last_fullsized":"2024-09-10T19:00:31.180688+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'25","last_scrub_stamp":"2024-09-10T19:00:03.071902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.071902+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.351800+0000","scrub_duration":0.0041116920000000001,"objects_trimmed":0,"snaptrim_duration":0.0020449829999999998,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1c","version":"244'51","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.438295+0000","last_change":"2024-09-10T19:00:31.438295+0000","last_active":"2024-09-10T19:00:31.438295+0000","last_peered":"2024-09-10T19:00:31.438295+0000","last_clean":"2024-09-10T19:00:31.438295+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:00:31.438295+0000","last_undegraded":"2024-09-10T19:00:31.438295+0000","last_fullsized":"2024-09-10T19:00:31.438295+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T18:58:54.847269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.847269+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091152980000000008,"objects_trimmed":0,"snaptrim_duration":0.000481294,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1b","version":"240'68","reported_seq":459,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436311+0000","last_change":"2024-09-10T19:00:31.436311+0000","last_active":"2024-09-10T19:00:31.436311+0000","last_peered":"2024-09-10T19:00:31.436311+0000","last_clean":"2024-09-10T19:00:31.436311+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:00:31.436311+0000","last_undegraded":"2024-09-10T19:00:31.436311+0000","last_fullsized":"2024-09-10T19:00:31.436311+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.0016072580000000001,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1a","version":"228'34","reported_seq":435,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181771+0000","last_change":"2024-09-10T19:00:31.181771+0000","last_active":"2024-09-10T19:00:31.181771+0000","last_peered":"2024-09-10T19:00:31.181771+0000","last_clean":"2024-09-10T19:00:31.181771+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:00:31.181771+0000","last_undegraded":"2024-09-10T19:00:31.181771+0000","last_fullsized":"2024-09-10T19:00:31.181771+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:59:58.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.060206+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.833991+0000","scrub_duration":0.0042048110000000001,"objects_trimmed":0,"snaptrim_duration":0.001942296,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.19","version":"454'47","reported_seq":456,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:33.179989+0000","last_change":"2024-09-10T19:00:33.179989+0000","last_active":"2024-09-10T19:00:33.179989+0000","last_peered":"2024-09-10T19:00:33.179989+0000","last_clean":"2024-09-10T19:00:33.179989+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:00:33.179989+0000","last_undegraded":"2024-09-10T19:00:33.179989+0000","last_fullsized":"2024-09-10T19:00:33.179989+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'45","last_scrub_stamp":"2024-09-10T19:00:30.901206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.901206+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:55.729039+0000","scrub_duration":0.0053195880000000001,"objects_trimmed":1,"snaptrim_duration":2.0093565710000001,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.18","version":"254'104","reported_seq":488,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575511+0000","last_change":"2024-09-10T19:00:31.575511+0000","last_active":"2024-09-10T19:00:31.575511+0000","last_peered":"2024-09-10T19:00:31.575511+0000","last_clean":"2024-09-10T19:00:31.575511+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:00:31.575511+0000","last_undegraded":"2024-09-10T19:00:31.575511+0000","last_fullsized":"2024-09-10T19:00:31.575511+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T19:00:13.172545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.172545+0000","objects_scrubbed":7,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.420509+0000","scrub_duration":0.0057922160000000002,"objects_trimmed":0,"snaptrim_duration":0.00054543799999999998,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.17","version":"428'23","reported_seq":425,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179632+0000","last_change":"2024-09-10T19:00:31.179632+0000","last_active":"2024-09-10T19:00:31.179632+0000","last_peered":"2024-09-10T19:00:31.179632+0000","last_clean":"2024-09-10T19:00:31.179632+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:00:31.179632+0000","last_undegraded":"2024-09-10T19:00:31.179632+0000","last_fullsized":"2024-09-10T19:00:31.179632+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'23","last_scrub_stamp":"2024-09-10T19:00:20.541045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:20.541045+0000","objects_scrubbed":4,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.871740+0000","scrub_duration":0.0049668359999999996,"objects_trimmed":0,"snaptrim_duration":0.0028546690000000002,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.16","version":"387'24","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.437906+0000","last_change":"2024-09-10T19:00:31.437906+0000","last_active":"2024-09-10T19:00:31.437906+0000","last_peered":"2024-09-10T19:00:31.437906+0000","last_clean":"2024-09-10T19:00:31.437906+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:00:31.437906+0000","last_undegraded":"2024-09-10T19:00:31.437906+0000","last_fullsized":"2024-09-10T19:00:31.437906+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.0022438419999999998,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.15","version":"428'79","reported_seq":484,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173482+0000","last_change":"2024-09-10T19:00:31.173482+0000","last_active":"2024-09-10T19:00:31.173482+0000","last_peered":"2024-09-10T19:00:31.173482+0000","last_clean":"2024-09-10T19:00:31.173482+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:00:31.173482+0000","last_undegraded":"2024-09-10T19:00:31.173482+0000","last_fullsized":"2024-09-10T19:00:31.173482+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'79","last_scrub_stamp":"2024-09-10T19:00:27.181327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.181327+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:35.433070+0000","scrub_duration":0.005077341,"objects_trimmed":0,"snaptrim_duration":0.001575438,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.14","version":"434'100","reported_seq":500,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.437580+0000","last_change":"2024-09-10T19:00:31.437580+0000","last_active":"2024-09-10T19:00:31.437580+0000","last_peered":"2024-09-10T19:00:31.437580+0000","last_clean":"2024-09-10T19:00:31.437580+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:00:31.437580+0000","last_undegraded":"2024-09-10T19:00:31.437580+0000","last_fullsized":"2024-09-10T19:00:31.437580+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.0022056889999999998,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.13","version":"448'4","reported_seq":424,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.176787+0000","last_change":"2024-09-10T19:00:31.176787+0000","last_active":"2024-09-10T19:00:31.176787+0000","last_peered":"2024-09-10T19:00:31.176787+0000","last_clean":"2024-09-10T19:00:31.176787+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:00:31.176787+0000","last_undegraded":"2024-09-10T19:00:31.176787+0000","last_fullsized":"2024-09-10T19:00:31.176787+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'4","last_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_deep_scrub":"448'4","last_deep_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.186496+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:43.223489+0000","scrub_duration":0.0071427519999999996,"objects_trimmed":0,"snaptrim_duration":0.0024511099999999998,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.12","version":"248'51","reported_seq":430,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578561+0000","last_change":"2024-09-10T19:00:31.578561+0000","last_active":"2024-09-10T19:00:31.578561+0000","last_peered":"2024-09-10T19:00:31.578561+0000","last_clean":"2024-09-10T19:00:31.578561+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:00:31.578561+0000","last_undegraded":"2024-09-10T19:00:31.578561+0000","last_fullsized":"2024-09-10T19:00:31.578561+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T18:58:39.380542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.380542+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032644879999999999,"objects_trimmed":0,"snaptrim_duration":0.0019282279999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.11","version":"452'56","reported_seq":455,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167229+0000","last_change":"2024-09-10T19:00:31.167229+0000","last_active":"2024-09-10T19:00:31.167229+0000","last_peered":"2024-09-10T19:00:31.167229+0000","last_clean":"2024-09-10T19:00:31.167229+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T19:00:31.167229+0000","last_undegraded":"2024-09-10T19:00:31.167229+0000","last_fullsized":"2024-09-10T19:00:31.167229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'56","last_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_deep_scrub":"452'56","last_deep_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.174126+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.989849+0000","scrub_duration":0.011240307,"objects_trimmed":0,"snaptrim_duration":0.001055891,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.10","version":"377'35","reported_seq":424,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174891+0000","last_change":"2024-09-10T19:00:31.174891+0000","last_active":"2024-09-10T19:00:31.174891+0000","last_peered":"2024-09-10T19:00:31.174891+0000","last_clean":"2024-09-10T19:00:31.174891+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:00:31.174891+0000","last_undegraded":"2024-09-10T19:00:31.174891+0000","last_fullsized":"2024-09-10T19:00:31.174891+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-09-10T18:58:37.311823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.311823+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092982399999999993,"objects_trimmed":0,"snaptrim_duration":0.0048756729999999996,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.f","version":"431'70","reported_seq":467,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577739+0000","last_change":"2024-09-10T19:00:31.577739+0000","last_active":"2024-09-10T19:00:31.577739+0000","last_peered":"2024-09-10T19:00:31.577739+0000","last_clean":"2024-09-10T19:00:31.577739+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:00:31.577739+0000","last_undegraded":"2024-09-10T19:00:31.577739+0000","last_fullsized":"2024-09-10T19:00:31.577739+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'70","last_scrub_stamp":"2024-09-10T19:00:18.252236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.252236+0000","objects_scrubbed":7,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:27.668280+0000","scrub_duration":0.0064748799999999997,"objects_trimmed":0,"snaptrim_duration":0.00047910800000000001,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.e","version":"246'119","reported_seq":503,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179348+0000","last_change":"2024-09-10T19:00:31.179348+0000","last_active":"2024-09-10T19:00:31.179348+0000","last_peered":"2024-09-10T19:00:31.179348+0000","last_clean":"2024-09-10T19:00:31.179348+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:00:31.179348+0000","last_undegraded":"2024-09-10T19:00:31.179348+0000","last_fullsized":"2024-09-10T19:00:31.179348+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T19:00:12.403246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.403246+0000","objects_scrubbed":7,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.629677+0000","scrub_duration":0.0056039189999999997,"objects_trimmed":0,"snaptrim_duration":0.0034957830000000001,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.d","version":"438'22","reported_seq":429,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168611+0000","last_change":"2024-09-10T19:00:31.168611+0000","last_active":"2024-09-10T19:00:31.168611+0000","last_peered":"2024-09-10T19:00:31.168611+0000","last_clean":"2024-09-10T19:00:31.168611+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T19:00:31.168611+0000","last_undegraded":"2024-09-10T19:00:31.168611+0000","last_fullsized":"2024-09-10T19:00:31.168611+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.001219143,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.c","version":"377'23","reported_seq":417,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578390+0000","last_change":"2024-09-10T19:00:31.578390+0000","last_active":"2024-09-10T19:00:31.578390+0000","last_peered":"2024-09-10T19:00:31.578390+0000","last_clean":"2024-09-10T19:00:31.578390+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:00:31.578390+0000","last_undegraded":"2024-09-10T19:00:31.578390+0000","last_fullsized":"2024-09-10T19:00:31.578390+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'22","last_scrub_stamp":"2024-09-10T18:58:47.475285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.475285+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041187790000000004,"objects_trimmed":0,"snaptrim_duration":0.001070466,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.b","version":"334'36","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.437120+0000","last_change":"2024-09-10T19:00:31.437120+0000","last_active":"2024-09-10T19:00:31.437120+0000","last_peered":"2024-09-10T19:00:31.437120+0000","last_clean":"2024-09-10T19:00:31.437120+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:00:31.437120+0000","last_undegraded":"2024-09-10T19:00:31.437120+0000","last_fullsized":"2024-09-10T19:00:31.437120+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'36","last_scrub_stamp":"2024-09-10T19:00:28.730904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:28.730904+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.146970+0000","scrub_duration":0.0045585440000000003,"objects_trimmed":0,"snaptrim_duration":0.00216029,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.a","version":"434'20","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177296+0000","last_change":"2024-09-10T19:00:31.177296+0000","last_active":"2024-09-10T19:00:31.177296+0000","last_peered":"2024-09-10T19:00:31.177296+0000","last_clean":"2024-09-10T19:00:31.177296+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:00:31.177296+0000","last_undegraded":"2024-09-10T19:00:31.177296+0000","last_fullsized":"2024-09-10T19:00:31.177296+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'20","last_scrub_stamp":"2024-09-10T19:00:18.161542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.161542+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.711115+0000","scrub_duration":0.0033164079999999999,"objects_trimmed":0,"snaptrim_duration":0.001467271,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.9","version":"0'0","reported_seq":417,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174020+0000","last_change":"2024-09-10T19:00:31.174020+0000","last_active":"2024-09-10T19:00:31.174020+0000","last_peered":"2024-09-10T19:00:31.174020+0000","last_clean":"2024-09-10T19:00:31.174020+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:00:31.174020+0000","last_undegraded":"2024-09-10T19:00:31.174020+0000","last_fullsized":"2024-09-10T19:00:31.174020+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.225132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.225132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:55.143010+0000","scrub_duration":0.0035652290000000001,"objects_trimmed":0,"snaptrim_duration":0.001714269,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.8","version":"234'51","reported_seq":449,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178915+0000","last_change":"2024-09-10T19:00:31.178915+0000","last_active":"2024-09-10T19:00:31.178915+0000","last_peered":"2024-09-10T19:00:31.178915+0000","last_clean":"2024-09-10T19:00:31.178915+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:00:31.178915+0000","last_undegraded":"2024-09-10T19:00:31.178915+0000","last_fullsized":"2024-09-10T19:00:31.178915+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T19:00:03.402967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.402967+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.326121+0000","scrub_duration":0.0043031060000000001,"objects_trimmed":0,"snaptrim_duration":0.0045812479999999996,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.7","version":"409'160","reported_seq":566,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177289+0000","last_change":"2024-09-10T19:00:31.177289+0000","last_active":"2024-09-10T19:00:31.177289+0000","last_peered":"2024-09-10T19:00:31.177289+0000","last_clean":"2024-09-10T19:00:31.177289+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:00:31.177289+0000","last_undegraded":"2024-09-10T19:00:31.177289+0000","last_fullsized":"2024-09-10T19:00:31.177289+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'160","last_scrub_stamp":"2024-09-10T19:00:30.192760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.192760+0000","objects_scrubbed":12,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.851352+0000","scrub_duration":0.0047442129999999997,"objects_trimmed":0,"snaptrim_duration":0.002136874,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.6","version":"388'99","reported_seq":543,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.182302+0000","last_change":"2024-09-10T19:00:31.182302+0000","last_active":"2024-09-10T19:00:31.182302+0000","last_peered":"2024-09-10T19:00:31.182302+0000","last_clean":"2024-09-10T19:00:31.182302+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:00:31.182302+0000","last_undegraded":"2024-09-10T19:00:31.182302+0000","last_fullsized":"2024-09-10T19:00:31.182302+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.213299+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022641919,"objects_trimmed":0,"snaptrim_duration":0.0019668480000000002,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.5","version":"305'53","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:33.094359+0000","last_change":"2024-09-10T19:00:33.094132+0000","last_active":"2024-09-10T19:00:33.094359+0000","last_peered":"2024-09-10T19:00:33.094359+0000","last_clean":"2024-09-10T19:00:33.094359+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:00:33.094359+0000","last_undegraded":"2024-09-10T19:00:33.094359+0000","last_fullsized":"2024-09-10T19:00:33.094359+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T19:00:33.093969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.093969+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:33.928384+0000","scrub_duration":0.0045796200000000004,"objects_trimmed":0,"snaptrim_duration":0.00084958699999999996,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.4","version":"257'107","reported_seq":498,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180125+0000","last_change":"2024-09-10T19:00:31.180125+0000","last_active":"2024-09-10T19:00:31.180125+0000","last_peered":"2024-09-10T19:00:31.180125+0000","last_clean":"2024-09-10T19:00:31.180125+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:00:31.180125+0000","last_undegraded":"2024-09-10T19:00:31.180125+0000","last_fullsized":"2024-09-10T19:00:31.180125+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T19:00:06.091203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:06.091203+0000","objects_scrubbed":7,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.467989+0000","scrub_duration":0.0051957840000000002,"objects_trimmed":0,"snaptrim_duration":0.001903835,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":32,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.3","version":"431'106","reported_seq":490,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177654+0000","last_change":"2024-09-10T19:00:31.177654+0000","last_active":"2024-09-10T19:00:31.177654+0000","last_peered":"2024-09-10T19:00:31.177654+0000","last_clean":"2024-09-10T19:00:31.177654+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:00:31.177654+0000","last_undegraded":"2024-09-10T19:00:31.177654+0000","last_fullsized":"2024-09-10T19:00:31.177654+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'102","last_scrub_stamp":"2024-09-10T18:58:34.265456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.265456+0000","objects_scrubbed":6,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040061309999999996,"objects_trimmed":0,"snaptrim_duration":0.0031593329999999998,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.2","version":"230'34","reported_seq":431,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178933+0000","last_change":"2024-09-10T19:00:31.178933+0000","last_active":"2024-09-10T19:00:31.178933+0000","last_peered":"2024-09-10T19:00:31.178933+0000","last_clean":"2024-09-10T19:00:31.178933+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:00:31.178933+0000","last_undegraded":"2024-09-10T19:00:31.178933+0000","last_fullsized":"2024-09-10T19:00:31.178933+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T19:00:15.443270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.443270+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.357357+0000","scrub_duration":0.0044317829999999999,"objects_trimmed":0,"snaptrim_duration":0.0037034139999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.1","version":"409'74","reported_seq":477,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576601+0000","last_change":"2024-09-10T19:00:31.576601+0000","last_active":"2024-09-10T19:00:31.576601+0000","last_peered":"2024-09-10T19:00:31.576601+0000","last_clean":"2024-09-10T19:00:31.576601+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:00:31.576601+0000","last_undegraded":"2024-09-10T19:00:31.576601+0000","last_fullsized":"2024-09-10T19:00:31.576601+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'74","last_scrub_stamp":"2024-09-10T19:00:31.135496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.135496+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.789352+0000","scrub_duration":0.0081214419999999996,"objects_trimmed":0,"snaptrim_duration":0.00076532300000000004,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"64.0","version":"246'102","reported_seq":499,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.176698+0000","last_change":"2024-09-10T19:00:31.176698+0000","last_active":"2024-09-10T19:00:31.176698+0000","last_peered":"2024-09-10T19:00:31.176698+0000","last_clean":"2024-09-10T19:00:31.176698+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:00:31.176698+0000","last_undegraded":"2024-09-10T19:00:31.176698+0000","last_fullsized":"2024-09-10T19:00:31.176698+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T19:00:14.192507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.192507+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.168955+0000","scrub_duration":0.0088236210000000002,"objects_trimmed":0,"snaptrim_duration":0.00167111,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.172745+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T19:00:31.172745+0000","last_peered":"2024-09-10T19:00:31.172745+0000","last_clean":"2024-09-10T19:00:31.172745+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:00:31.172745+0000","last_undegraded":"2024-09-10T19:00:31.172745+0000","last_fullsized":"2024-09-10T19:00:31.172745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":442,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183471+0000","last_change":"2024-09-10T18:59:59.106249+0000","last_active":"2024-09-10T19:00:31.183471+0000","last_peered":"2024-09-10T19:00:31.183471+0000","last_clean":"2024-09-10T19:00:31.183471+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:00:31.183471+0000","last_undegraded":"2024-09-10T19:00:31.183471+0000","last_fullsized":"2024-09-10T19:00:31.183471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.106063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.106063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:17.167241+0000","scrub_duration":0.0034646490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":462,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173854+0000","last_change":"2024-09-10T19:00:10.940056+0000","last_active":"2024-09-10T19:00:31.173854+0000","last_peered":"2024-09-10T19:00:31.173854+0000","last_clean":"2024-09-10T19:00:31.173854+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:00:31.173854+0000","last_undegraded":"2024-09-10T19:00:31.173854+0000","last_fullsized":"2024-09-10T19:00:31.173854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:00:10.939877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.939877+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:33.784202+0000","scrub_duration":0.0051523760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173260+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T19:00:31.173260+0000","last_peered":"2024-09-10T19:00:31.173260+0000","last_clean":"2024-09-10T19:00:31.173260+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:00:31.173260+0000","last_undegraded":"2024-09-10T19:00:31.173260+0000","last_fullsized":"2024-09-10T19:00:31.173260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":354,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575458+0000","last_change":"2024-09-10T18:59:53.417704+0000","last_active":"2024-09-10T19:00:31.575458+0000","last_peered":"2024-09-10T19:00:31.575458+0000","last_clean":"2024-09-10T19:00:31.575458+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:00:31.575458+0000","last_undegraded":"2024-09-10T19:00:31.575458+0000","last_fullsized":"2024-09-10T19:00:31.575458+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.417551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.417551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.108833+0000","scrub_duration":0.0034045260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174235+0000","last_change":"2024-09-10T18:58:33.225425+0000","last_active":"2024-09-10T19:00:31.174235+0000","last_peered":"2024-09-10T19:00:31.174235+0000","last_clean":"2024-09-10T19:00:31.174235+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:00:31.174235+0000","last_undegraded":"2024-09-10T19:00:31.174235+0000","last_fullsized":"2024-09-10T19:00:31.174235+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:33.225302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:33.225302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023814840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":471,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183150+0000","last_change":"2024-09-10T19:00:03.037143+0000","last_active":"2024-09-10T19:00:31.183150+0000","last_peered":"2024-09-10T19:00:31.183150+0000","last_clean":"2024-09-10T19:00:31.183150+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:00:31.183150+0000","last_undegraded":"2024-09-10T19:00:31.183150+0000","last_fullsized":"2024-09-10T19:00:31.183150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:03.036974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.036974+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:32.356994+0000","scrub_duration":0.0043269010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":449,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166286+0000","last_change":"2024-09-10T19:00:01.347416+0000","last_active":"2024-09-10T19:00:31.166286+0000","last_peered":"2024-09-10T19:00:31.166286+0000","last_clean":"2024-09-10T19:00:31.166286+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:00:31.166286+0000","last_undegraded":"2024-09-10T19:00:31.166286+0000","last_fullsized":"2024-09-10T19:00:31.166286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:01.347241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.347241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.701874+0000","scrub_duration":0.003764398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166609+0000","last_change":"2024-09-10T18:58:37.683235+0000","last_active":"2024-09-10T19:00:31.166609+0000","last_peered":"2024-09-10T19:00:31.166609+0000","last_clean":"2024-09-10T19:00:31.166609+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T19:00:31.166609+0000","last_undegraded":"2024-09-10T19:00:31.166609+0000","last_fullsized":"2024-09-10T19:00:31.166609+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:37.683082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.683082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030566450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":347,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178114+0000","last_change":"2024-09-10T18:57:31.688020+0000","last_active":"2024-09-10T19:00:31.178114+0000","last_peered":"2024-09-10T19:00:31.178114+0000","last_clean":"2024-09-10T19:00:31.178114+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:00:31.178114+0000","last_undegraded":"2024-09-10T19:00:31.178114+0000","last_fullsized":"2024-09-10T19:00:31.178114+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:31.687877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:31.687877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034471419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178869+0000","last_change":"2024-09-10T18:58:43.158260+0000","last_active":"2024-09-10T19:00:31.178869+0000","last_peered":"2024-09-10T19:00:31.178869+0000","last_clean":"2024-09-10T19:00:31.178869+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:00:31.178869+0000","last_undegraded":"2024-09-10T19:00:31.178869+0000","last_fullsized":"2024-09-10T19:00:31.178869+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.158082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.158082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033992950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":485,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577811+0000","last_change":"2024-09-10T19:00:09.145311+0000","last_active":"2024-09-10T19:00:31.577811+0000","last_peered":"2024-09-10T19:00:31.577811+0000","last_clean":"2024-09-10T19:00:31.577811+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:00:31.577811+0000","last_undegraded":"2024-09-10T19:00:31.577811+0000","last_fullsized":"2024-09-10T19:00:31.577811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:09.145229+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.145229+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:21.374619+0000","scrub_duration":0.0090468159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":439,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167341+0000","last_change":"2024-09-10T18:58:53.495436+0000","last_active":"2024-09-10T19:00:31.167341+0000","last_peered":"2024-09-10T19:00:31.167341+0000","last_clean":"2024-09-10T19:00:31.167341+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T19:00:31.167341+0000","last_undegraded":"2024-09-10T19:00:31.167341+0000","last_fullsized":"2024-09-10T19:00:31.167341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.495273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.495273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024582229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575627+0000","last_change":"2024-09-10T19:00:03.208547+0000","last_active":"2024-09-10T19:00:31.575627+0000","last_peered":"2024-09-10T19:00:31.575627+0000","last_clean":"2024-09-10T19:00:31.575627+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:00:31.575627+0000","last_undegraded":"2024-09-10T19:00:31.575627+0000","last_fullsized":"2024-09-10T19:00:31.575627+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.208468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.208468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:29.413505+0000","scrub_duration":0.0019202379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435068+0000","last_change":"2024-09-10T18:58:39.856907+0000","last_active":"2024-09-10T19:00:31.435068+0000","last_peered":"2024-09-10T19:00:31.435068+0000","last_clean":"2024-09-10T19:00:31.435068+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:00:31.435068+0000","last_undegraded":"2024-09-10T19:00:31.435068+0000","last_fullsized":"2024-09-10T19:00:31.435068+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.856740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.856740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034908270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":443,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179350+0000","last_change":"2024-09-10T18:59:03.215137+0000","last_active":"2024-09-10T19:00:31.179350+0000","last_peered":"2024-09-10T19:00:31.179350+0000","last_clean":"2024-09-10T19:00:31.179350+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:00:31.179350+0000","last_undegraded":"2024-09-10T19:00:31.179350+0000","last_fullsized":"2024-09-10T19:00:31.179350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.215039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.215039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064108910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":482,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577589+0000","last_change":"2024-09-10T18:59:05.516222+0000","last_active":"2024-09-10T19:00:31.577589+0000","last_peered":"2024-09-10T19:00:31.577589+0000","last_clean":"2024-09-10T19:00:31.577589+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:00:31.577589+0000","last_undegraded":"2024-09-10T19:00:31.577589+0000","last_fullsized":"2024-09-10T19:00:31.577589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'17","last_scrub_stamp":"2024-09-10T18:59:05.516062+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:05.516062+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058657070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":449,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:35.255777+0000","last_change":"2024-09-10T19:00:35.255523+0000","last_active":"2024-09-10T19:00:35.255777+0000","last_peered":"2024-09-10T19:00:35.255777+0000","last_clean":"2024-09-10T19:00:35.255777+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T19:00:35.255777+0000","last_undegraded":"2024-09-10T19:00:35.255777+0000","last_fullsized":"2024-09-10T19:00:35.255777+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:35.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:00:35.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:46.150006+0000","scrub_duration":0.0028367790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177082+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T19:00:31.177082+0000","last_peered":"2024-09-10T19:00:31.177082+0000","last_clean":"2024-09-10T19:00:31.177082+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:00:31.177082+0000","last_undegraded":"2024-09-10T19:00:31.177082+0000","last_fullsized":"2024-09-10T19:00:31.177082+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":359,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171391+0000","last_change":"2024-09-10T19:00:14.868536+0000","last_active":"2024-09-10T19:00:31.171391+0000","last_peered":"2024-09-10T19:00:31.171391+0000","last_clean":"2024-09-10T19:00:31.171391+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:00:31.171391+0000","last_undegraded":"2024-09-10T19:00:31.171391+0000","last_fullsized":"2024-09-10T19:00:31.171391+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.868377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.868377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.646932+0000","scrub_duration":0.0036007919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":503,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175479+0000","last_change":"2024-09-10T19:00:16.474150+0000","last_active":"2024-09-10T19:00:31.175479+0000","last_peered":"2024-09-10T19:00:31.175479+0000","last_clean":"2024-09-10T19:00:31.175479+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:00:31.175479+0000","last_undegraded":"2024-09-10T19:00:31.175479+0000","last_fullsized":"2024-09-10T19:00:31.175479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:00:16.474006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.474006+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.225107+0000","scrub_duration":0.0057543610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179005+0000","last_change":"2024-09-10T18:59:57.030876+0000","last_active":"2024-09-10T19:00:31.179005+0000","last_peered":"2024-09-10T19:00:31.179005+0000","last_clean":"2024-09-10T19:00:31.179005+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:00:31.179005+0000","last_undegraded":"2024-09-10T19:00:31.179005+0000","last_fullsized":"2024-09-10T19:00:31.179005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.030798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.030798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.038402+0000","scrub_duration":0.0020075510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":585,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.321230+0000","last_change":"2024-09-10T18:58:36.681485+0000","last_active":"2024-09-10T19:00:31.321230+0000","last_peered":"2024-09-10T19:00:31.321230+0000","last_clean":"2024-09-10T19:00:31.321230+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T19:00:31.321230+0000","last_undegraded":"2024-09-10T19:00:31.321230+0000","last_fullsized":"2024-09-10T19:00:31.321230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T18:58:36.681315+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.681315+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00471186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":456,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:33.141150+0000","last_change":"2024-09-10T19:00:33.141061+0000","last_active":"2024-09-10T19:00:33.141150+0000","last_peered":"2024-09-10T19:00:33.141150+0000","last_clean":"2024-09-10T19:00:33.141150+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:00:33.141150+0000","last_undegraded":"2024-09-10T19:00:33.141150+0000","last_fullsized":"2024-09-10T19:00:33.141150+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.140938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.140938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:02.048809+0000","scrub_duration":0.0029466620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177262+0000","last_change":"2024-09-10T18:59:59.486802+0000","last_active":"2024-09-10T19:00:31.177262+0000","last_peered":"2024-09-10T19:00:31.177262+0000","last_clean":"2024-09-10T19:00:31.177262+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:00:31.177262+0000","last_undegraded":"2024-09-10T19:00:31.177262+0000","last_fullsized":"2024-09-10T19:00:31.177262+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.486653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.486653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:19.115655+0000","scrub_duration":0.003394902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":471,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173691+0000","last_change":"2024-09-10T18:59:57.094883+0000","last_active":"2024-09-10T19:00:31.173691+0000","last_peered":"2024-09-10T19:00:31.173691+0000","last_clean":"2024-09-10T19:00:31.173691+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:00:31.173691+0000","last_undegraded":"2024-09-10T19:00:31.173691+0000","last_fullsized":"2024-09-10T19:00:31.173691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T18:59:57.094721+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.094721+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:13.024150+0000","scrub_duration":0.0045594349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":451,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.434667+0000","last_change":"2024-09-10T18:58:52.884147+0000","last_active":"2024-09-10T19:00:31.434667+0000","last_peered":"2024-09-10T19:00:31.434667+0000","last_clean":"2024-09-10T19:00:31.434667+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:00:31.434667+0000","last_undegraded":"2024-09-10T19:00:31.434667+0000","last_fullsized":"2024-09-10T19:00:31.434667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.884077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032688370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":442,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167290+0000","last_change":"2024-09-10T19:00:02.315530+0000","last_active":"2024-09-10T19:00:31.167290+0000","last_peered":"2024-09-10T19:00:31.167290+0000","last_clean":"2024-09-10T19:00:31.167290+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T19:00:31.167290+0000","last_undegraded":"2024-09-10T19:00:31.167290+0000","last_fullsized":"2024-09-10T19:00:31.167290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:56:36.856187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:56:36.856187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031585509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168291+0000","last_change":"2024-09-10T18:58:34.635361+0000","last_active":"2024-09-10T19:00:31.168291+0000","last_peered":"2024-09-10T19:00:31.168291+0000","last_clean":"2024-09-10T19:00:31.168291+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T19:00:31.168291+0000","last_undegraded":"2024-09-10T19:00:31.168291+0000","last_fullsized":"2024-09-10T19:00:31.168291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.635181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.635181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051477889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175201+0000","last_change":"2024-09-10T19:00:07.323465+0000","last_active":"2024-09-10T19:00:31.175201+0000","last_peered":"2024-09-10T19:00:31.175201+0000","last_clean":"2024-09-10T19:00:31.175201+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:00:31.175201+0000","last_undegraded":"2024-09-10T19:00:31.175201+0000","last_fullsized":"2024-09-10T19:00:31.175201+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:07.323364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:07.323364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.180738+0000","scrub_duration":0.0028330460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":485,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171304+0000","last_change":"2024-09-10T19:00:12.906006+0000","last_active":"2024-09-10T19:00:31.171304+0000","last_peered":"2024-09-10T19:00:31.171304+0000","last_clean":"2024-09-10T19:00:31.171304+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:00:31.171304+0000","last_undegraded":"2024-09-10T19:00:31.171304+0000","last_fullsized":"2024-09-10T19:00:31.171304+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:00:12.905889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.905889+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:42.533979+0000","scrub_duration":0.003872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":441,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.434062+0000","last_change":"2024-09-10T18:58:51.907906+0000","last_active":"2024-09-10T19:00:31.434062+0000","last_peered":"2024-09-10T19:00:31.434062+0000","last_clean":"2024-09-10T19:00:31.434062+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:00:31.434062+0000","last_undegraded":"2024-09-10T19:00:31.434062+0000","last_fullsized":"2024-09-10T19:00:31.434062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:51.907749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:51.907749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036594549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436217+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T19:00:31.436217+0000","last_peered":"2024-09-10T19:00:31.436217+0000","last_clean":"2024-09-10T19:00:31.436217+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:00:31.436217+0000","last_undegraded":"2024-09-10T19:00:31.436217+0000","last_fullsized":"2024-09-10T19:00:31.436217+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":441,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576790+0000","last_change":"2024-09-10T18:59:57.328269+0000","last_active":"2024-09-10T19:00:31.576790+0000","last_peered":"2024-09-10T19:00:31.576790+0000","last_clean":"2024-09-10T19:00:31.576790+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:00:31.576790+0000","last_undegraded":"2024-09-10T19:00:31.576790+0000","last_fullsized":"2024-09-10T19:00:31.576790+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.328111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.328111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.067869+0000","scrub_duration":0.0032441140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":367,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436281+0000","last_change":"2024-09-10T19:00:14.537922+0000","last_active":"2024-09-10T19:00:31.436281+0000","last_peered":"2024-09-10T19:00:31.436281+0000","last_clean":"2024-09-10T19:00:31.436281+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:00:31.436281+0000","last_undegraded":"2024-09-10T19:00:31.436281+0000","last_fullsized":"2024-09-10T19:00:31.436281+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.537765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.537765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:28.781636+0000","scrub_duration":0.0033849069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":488,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166783+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T19:00:31.166783+0000","last_peered":"2024-09-10T19:00:31.166783+0000","last_clean":"2024-09-10T19:00:31.166783+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:00:31.166783+0000","last_undegraded":"2024-09-10T19:00:31.166783+0000","last_fullsized":"2024-09-10T19:00:31.166783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181704+0000","last_change":"2024-09-10T19:00:24.104254+0000","last_active":"2024-09-10T19:00:31.181704+0000","last_peered":"2024-09-10T19:00:31.181704+0000","last_clean":"2024-09-10T19:00:31.181704+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:00:31.181704+0000","last_undegraded":"2024-09-10T19:00:31.181704+0000","last_fullsized":"2024-09-10T19:00:31.181704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:24.104066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:24.104066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.917926+0000","scrub_duration":0.0038382149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":595,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.321403+0000","last_change":"2024-09-10T19:00:05.221386+0000","last_active":"2024-09-10T19:00:31.321403+0000","last_peered":"2024-09-10T19:00:31.321403+0000","last_clean":"2024-09-10T19:00:31.321403+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T19:00:31.321403+0000","last_undegraded":"2024-09-10T19:00:31.321403+0000","last_fullsized":"2024-09-10T19:00:31.321403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T19:00:05.221191+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.221191+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.966409+0000","scrub_duration":0.0039722840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":439,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174017+0000","last_change":"2024-09-10T19:00:22.885319+0000","last_active":"2024-09-10T19:00:31.174017+0000","last_peered":"2024-09-10T19:00:31.174017+0000","last_clean":"2024-09-10T19:00:31.174017+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:00:31.174017+0000","last_undegraded":"2024-09-10T19:00:31.174017+0000","last_fullsized":"2024-09-10T19:00:31.174017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:00:22.885146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:22.885146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.180722+0000","scrub_duration":0.0048287809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435724+0000","last_change":"2024-09-10T18:58:47.871654+0000","last_active":"2024-09-10T19:00:31.435724+0000","last_peered":"2024-09-10T19:00:31.435724+0000","last_clean":"2024-09-10T19:00:31.435724+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:00:31.435724+0000","last_undegraded":"2024-09-10T19:00:31.435724+0000","last_fullsized":"2024-09-10T19:00:31.435724+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.871591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.871591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023117910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":466,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181436+0000","last_change":"2024-09-10T18:57:53.667021+0000","last_active":"2024-09-10T19:00:31.181436+0000","last_peered":"2024-09-10T19:00:31.181436+0000","last_clean":"2024-09-10T19:00:31.181436+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:00:31.181436+0000","last_undegraded":"2024-09-10T19:00:31.181436+0000","last_fullsized":"2024-09-10T19:00:31.181436+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'12","last_scrub_stamp":"2024-09-10T18:57:53.666867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:57:53.666867+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004555941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177551+0000","last_change":"2024-09-10T18:58:50.597419+0000","last_active":"2024-09-10T19:00:31.177551+0000","last_peered":"2024-09-10T19:00:31.177551+0000","last_clean":"2024-09-10T19:00:31.177551+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:00:31.177551+0000","last_undegraded":"2024-09-10T19:00:31.177551+0000","last_fullsized":"2024-09-10T19:00:31.177551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.597321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.597321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021838199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":449,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166563+0000","last_change":"2024-09-10T19:00:10.266358+0000","last_active":"2024-09-10T19:00:31.166563+0000","last_peered":"2024-09-10T19:00:31.166563+0000","last_clean":"2024-09-10T19:00:31.166563+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T19:00:31.166563+0000","last_undegraded":"2024-09-10T19:00:31.166563+0000","last_fullsized":"2024-09-10T19:00:31.166563+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.266243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.266243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.094340+0000","scrub_duration":0.003123285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1100,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:35.129947+0000","last_change":"2024-09-10T18:58:41.241436+0000","last_active":"2024-09-10T19:00:35.129947+0000","last_peered":"2024-09-10T19:00:35.129947+0000","last_clean":"2024-09-10T19:00:35.129947+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:00:35.129947+0000","last_undegraded":"2024-09-10T19:00:35.129947+0000","last_fullsized":"2024-09-10T19:00:35.129947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T18:58:41.241354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:41.241354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00989547,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":652,"num_read_kb":647,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178348+0000","last_change":"2024-09-10T19:00:04.106819+0000","last_active":"2024-09-10T19:00:31.178348+0000","last_peered":"2024-09-10T19:00:31.178348+0000","last_clean":"2024-09-10T19:00:31.178348+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:00:31.178348+0000","last_undegraded":"2024-09-10T19:00:31.178348+0000","last_fullsized":"2024-09-10T19:00:31.178348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.106660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.106660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:23.468919+0000","scrub_duration":0.0031927650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":378,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183206+0000","last_change":"2024-09-10T19:00:31.183145+0000","last_active":"2024-09-10T19:00:31.183206+0000","last_peered":"2024-09-10T19:00:31.183206+0000","last_clean":"2024-09-10T19:00:31.183206+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:00:31.183206+0000","last_undegraded":"2024-09-10T19:00:31.183206+0000","last_fullsized":"2024-09-10T19:00:31.183206+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:31.183027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.183027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.165521+0000","scrub_duration":0.0084730650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":475,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177285+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T19:00:31.177285+0000","last_peered":"2024-09-10T19:00:31.177285+0000","last_clean":"2024-09-10T19:00:31.177285+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:00:31.177285+0000","last_undegraded":"2024-09-10T19:00:31.177285+0000","last_fullsized":"2024-09-10T19:00:31.177285+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436465+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T19:00:31.436465+0000","last_peered":"2024-09-10T19:00:31.436465+0000","last_clean":"2024-09-10T19:00:31.436465+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:00:31.436465+0000","last_undegraded":"2024-09-10T19:00:31.436465+0000","last_fullsized":"2024-09-10T19:00:31.436465+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":575,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.324389+0000","last_change":"2024-09-10T18:59:59.066338+0000","last_active":"2024-09-10T19:00:31.324389+0000","last_peered":"2024-09-10T19:00:31.324389+0000","last_clean":"2024-09-10T19:00:31.324389+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:00:31.324389+0000","last_undegraded":"2024-09-10T19:00:31.324389+0000","last_fullsized":"2024-09-10T19:00:31.324389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:59:59.066136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.066136+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.679135+0000","scrub_duration":0.0038648509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":456,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166425+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T19:00:31.166425+0000","last_peered":"2024-09-10T19:00:31.166425+0000","last_clean":"2024-09-10T19:00:31.166425+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:00:31.166425+0000","last_undegraded":"2024-09-10T19:00:31.166425+0000","last_fullsized":"2024-09-10T19:00:31.166425+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166786+0000","last_change":"2024-09-10T18:57:50.792451+0000","last_active":"2024-09-10T19:00:31.166786+0000","last_peered":"2024-09-10T19:00:31.166786+0000","last_clean":"2024-09-10T19:00:31.166786+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T19:00:31.166786+0000","last_undegraded":"2024-09-10T19:00:31.166786+0000","last_fullsized":"2024-09-10T19:00:31.166786+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.792298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003365444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":349,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.170891+0000","last_change":"2024-09-10T18:57:47.189651+0000","last_active":"2024-09-10T19:00:31.170891+0000","last_peered":"2024-09-10T19:00:31.170891+0000","last_clean":"2024-09-10T19:00:31.170891+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:00:31.170891+0000","last_undegraded":"2024-09-10T19:00:31.170891+0000","last_fullsized":"2024-09-10T19:00:31.170891+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:47.189467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:47.189467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077946020000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":662,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436586+0000","last_change":"2024-09-10T19:00:01.595171+0000","last_active":"2024-09-10T19:00:31.436586+0000","last_peered":"2024-09-10T19:00:31.436586+0000","last_clean":"2024-09-10T19:00:31.436586+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:00:31.436586+0000","last_undegraded":"2024-09-10T19:00:31.436586+0000","last_fullsized":"2024-09-10T19:00:31.436586+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:00:01.595006+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.595006+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.951465+0000","scrub_duration":0.0053873050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180399+0000","last_change":"2024-09-10T18:58:50.209102+0000","last_active":"2024-09-10T19:00:31.180399+0000","last_peered":"2024-09-10T19:00:31.180399+0000","last_clean":"2024-09-10T19:00:31.180399+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:00:31.180399+0000","last_undegraded":"2024-09-10T19:00:31.180399+0000","last_fullsized":"2024-09-10T19:00:31.180399+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:50.208993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:50.208993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003212126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":584,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.323455+0000","last_change":"2024-09-10T18:59:56.107053+0000","last_active":"2024-09-10T19:00:31.323455+0000","last_peered":"2024-09-10T19:00:31.323455+0000","last_clean":"2024-09-10T19:00:31.323455+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:00:31.323455+0000","last_undegraded":"2024-09-10T19:00:31.323455+0000","last_fullsized":"2024-09-10T19:00:31.323455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:59:56.106896+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:59:56.106896+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.084878+0000","scrub_duration":0.0036065950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175269+0000","last_change":"2024-09-10T18:58:37.451840+0000","last_active":"2024-09-10T19:00:31.175269+0000","last_peered":"2024-09-10T19:00:31.175269+0000","last_clean":"2024-09-10T19:00:31.175269+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:00:31.175269+0000","last_undegraded":"2024-09-10T19:00:31.175269+0000","last_fullsized":"2024-09-10T19:00:31.175269+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T18:58:37.451583+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T18:58:37.451583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004712585,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":435,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.433710+0000","last_change":"2024-09-10T18:58:45.871468+0000","last_active":"2024-09-10T19:00:31.433710+0000","last_peered":"2024-09-10T19:00:31.433710+0000","last_clean":"2024-09-10T19:00:31.433710+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:00:31.433710+0000","last_undegraded":"2024-09-10T19:00:31.433710+0000","last_fullsized":"2024-09-10T19:00:31.433710+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:45.871308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:45.871308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033902770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":359,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.934413+0000","last_change":"2024-09-10T19:00:31.934239+0000","last_active":"2024-09-10T19:00:31.934413+0000","last_peered":"2024-09-10T19:00:31.934413+0000","last_clean":"2024-09-10T19:00:31.934413+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:00:31.934413+0000","last_undegraded":"2024-09-10T19:00:31.934413+0000","last_fullsized":"2024-09-10T19:00:31.934413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:31.933970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.933970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.584006+0000","scrub_duration":0.003362608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":475,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577799+0000","last_change":"2024-09-10T18:58:46.437660+0000","last_active":"2024-09-10T19:00:31.577799+0000","last_peered":"2024-09-10T19:00:31.577799+0000","last_clean":"2024-09-10T19:00:31.577799+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:00:31.577799+0000","last_undegraded":"2024-09-10T19:00:31.577799+0000","last_fullsized":"2024-09-10T19:00:31.577799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:46.437410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.437410+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054784669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":441,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436755+0000","last_change":"2024-09-10T18:58:35.839393+0000","last_active":"2024-09-10T19:00:31.436755+0000","last_peered":"2024-09-10T19:00:31.436755+0000","last_clean":"2024-09-10T19:00:31.436755+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:00:31.436755+0000","last_undegraded":"2024-09-10T19:00:31.436755+0000","last_fullsized":"2024-09-10T19:00:31.436755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:35.839238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:35.839238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035832669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.434303+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T19:00:31.434303+0000","last_peered":"2024-09-10T19:00:31.434303+0000","last_clean":"2024-09-10T19:00:31.434303+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:00:31.434303+0000","last_undegraded":"2024-09-10T19:00:31.434303+0000","last_fullsized":"2024-09-10T19:00:31.434303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177896+0000","last_change":"2024-09-10T19:00:00.056759+0000","last_active":"2024-09-10T19:00:31.177896+0000","last_peered":"2024-09-10T19:00:31.177896+0000","last_clean":"2024-09-10T19:00:31.177896+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:00:31.177896+0000","last_undegraded":"2024-09-10T19:00:31.177896+0000","last_fullsized":"2024-09-10T19:00:31.177896+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.056600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.056600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:11.057092+0000","scrub_duration":0.0028960119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":368,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175993+0000","last_change":"2024-09-10T18:59:58.063484+0000","last_active":"2024-09-10T19:00:31.175993+0000","last_peered":"2024-09-10T19:00:31.175993+0000","last_clean":"2024-09-10T19:00:31.175993+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:00:31.175993+0000","last_undegraded":"2024-09-10T19:00:31.175993+0000","last_fullsized":"2024-09-10T19:00:31.175993+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.063321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.063321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:58.931188+0000","scrub_duration":0.0034568020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":502,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168088+0000","last_change":"2024-09-10T19:00:27.165321+0000","last_active":"2024-09-10T19:00:31.168088+0000","last_peered":"2024-09-10T19:00:31.168088+0000","last_clean":"2024-09-10T19:00:31.168088+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T19:00:31.168088+0000","last_undegraded":"2024-09-10T19:00:31.168088+0000","last_fullsized":"2024-09-10T19:00:31.168088+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:00:27.165232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.165232+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:49.580033+0000","scrub_duration":0.0047657200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578310+0000","last_change":"2024-09-10T18:58:52.447688+0000","last_active":"2024-09-10T19:00:31.578310+0000","last_peered":"2024-09-10T19:00:31.578310+0000","last_clean":"2024-09-10T19:00:31.578310+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:00:31.578310+0000","last_undegraded":"2024-09-10T19:00:31.578310+0000","last_fullsized":"2024-09-10T19:00:31.578310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.447547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.447547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031696559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436475+0000","last_change":"2024-09-10T19:00:25.638042+0000","last_active":"2024-09-10T19:00:31.436475+0000","last_peered":"2024-09-10T19:00:31.436475+0000","last_clean":"2024-09-10T19:00:31.436475+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:00:31.436475+0000","last_undegraded":"2024-09-10T19:00:31.436475+0000","last_fullsized":"2024-09-10T19:00:31.436475+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.637874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.637874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.329382+0000","scrub_duration":0.0033414320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173768+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T19:00:31.173768+0000","last_peered":"2024-09-10T19:00:31.173768+0000","last_clean":"2024-09-10T19:00:31.173768+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:00:31.173768+0000","last_undegraded":"2024-09-10T19:00:31.173768+0000","last_fullsized":"2024-09-10T19:00:31.173768+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179264+0000","last_change":"2024-09-10T19:00:21.205302+0000","last_active":"2024-09-10T19:00:31.179264+0000","last_peered":"2024-09-10T19:00:31.179264+0000","last_clean":"2024-09-10T19:00:31.179264+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:00:31.179264+0000","last_undegraded":"2024-09-10T19:00:31.179264+0000","last_fullsized":"2024-09-10T19:00:31.179264+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.205176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.205176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:35.277081+0000","scrub_duration":0.0033764950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":507,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.433875+0000","last_change":"2024-09-10T19:00:13.487926+0000","last_active":"2024-09-10T19:00:31.433875+0000","last_peered":"2024-09-10T19:00:31.433875+0000","last_clean":"2024-09-10T19:00:31.433875+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:00:31.433875+0000","last_undegraded":"2024-09-10T19:00:31.433875+0000","last_fullsized":"2024-09-10T19:00:31.433875+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.487819+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:18.610954+0000","scrub_duration":0.003630566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171683+0000","last_change":"2024-09-10T18:59:59.081414+0000","last_active":"2024-09-10T19:00:31.171683+0000","last_peered":"2024-09-10T19:00:31.171683+0000","last_clean":"2024-09-10T19:00:31.171683+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:00:31.171683+0000","last_undegraded":"2024-09-10T19:00:31.171683+0000","last_fullsized":"2024-09-10T19:00:31.171683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.081277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.081277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.759642+0000","scrub_duration":0.002648634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.436316+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T19:00:31.436316+0000","last_peered":"2024-09-10T19:00:31.436316+0000","last_clean":"2024-09-10T19:00:31.436316+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:00:31.436316+0000","last_undegraded":"2024-09-10T19:00:31.436316+0000","last_fullsized":"2024-09-10T19:00:31.436316+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179600+0000","last_change":"2024-09-10T19:00:17.050264+0000","last_active":"2024-09-10T19:00:31.179600+0000","last_peered":"2024-09-10T19:00:31.179600+0000","last_clean":"2024-09-10T19:00:31.179600+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:00:31.179600+0000","last_undegraded":"2024-09-10T19:00:31.179600+0000","last_fullsized":"2024-09-10T19:00:31.179600+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.050058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.050058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.678802+0000","scrub_duration":0.0038046009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177028+0000","last_change":"2024-09-10T18:59:02.713009+0000","last_active":"2024-09-10T19:00:31.177028+0000","last_peered":"2024-09-10T19:00:31.177028+0000","last_clean":"2024-09-10T19:00:31.177028+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:00:31.177028+0000","last_undegraded":"2024-09-10T19:00:31.177028+0000","last_fullsized":"2024-09-10T19:00:31.177028+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.712839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.712839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035464870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":482,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178681+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T19:00:31.178681+0000","last_peered":"2024-09-10T19:00:31.178681+0000","last_clean":"2024-09-10T19:00:31.178681+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:00:31.178681+0000","last_undegraded":"2024-09-10T19:00:31.178681+0000","last_fullsized":"2024-09-10T19:00:31.178681+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181140+0000","last_change":"2024-09-10T19:00:26.116209+0000","last_active":"2024-09-10T19:00:31.181140+0000","last_peered":"2024-09-10T19:00:31.181140+0000","last_clean":"2024-09-10T19:00:31.181140+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:00:31.181140+0000","last_undegraded":"2024-09-10T19:00:31.181140+0000","last_fullsized":"2024-09-10T19:00:31.181140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.116042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.423133+0000","scrub_duration":0.0039421509999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181592+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T19:00:31.181592+0000","last_peered":"2024-09-10T19:00:31.181592+0000","last_clean":"2024-09-10T19:00:31.181592+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:00:31.181592+0000","last_undegraded":"2024-09-10T19:00:31.181592+0000","last_fullsized":"2024-09-10T19:00:31.181592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":445,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435806+0000","last_change":"2024-09-10T19:00:03.612383+0000","last_active":"2024-09-10T19:00:31.435806+0000","last_peered":"2024-09-10T19:00:31.435806+0000","last_clean":"2024-09-10T19:00:31.435806+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:00:31.435806+0000","last_undegraded":"2024-09-10T19:00:31.435806+0000","last_fullsized":"2024-09-10T19:00:31.435806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.612225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.557620+0000","scrub_duration":0.0033384840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181738+0000","last_change":"2024-09-10T19:00:25.167720+0000","last_active":"2024-09-10T19:00:31.181738+0000","last_peered":"2024-09-10T19:00:31.181738+0000","last_clean":"2024-09-10T19:00:31.181738+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:00:31.181738+0000","last_undegraded":"2024-09-10T19:00:31.181738+0000","last_fullsized":"2024-09-10T19:00:31.181738+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.167452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.167452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:47.782239+0000","scrub_duration":0.0033805139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":467,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171759+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T19:00:31.171759+0000","last_peered":"2024-09-10T19:00:31.171759+0000","last_clean":"2024-09-10T19:00:31.171759+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:00:31.171759+0000","last_undegraded":"2024-09-10T19:00:31.171759+0000","last_fullsized":"2024-09-10T19:00:31.171759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576047+0000","last_change":"2024-09-10T19:00:15.230885+0000","last_active":"2024-09-10T19:00:31.576047+0000","last_peered":"2024-09-10T19:00:31.576047+0000","last_clean":"2024-09-10T19:00:31.576047+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:00:31.576047+0000","last_undegraded":"2024-09-10T19:00:31.576047+0000","last_fullsized":"2024-09-10T19:00:31.576047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.230765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:38.515881+0000","scrub_duration":0.002962732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":498,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181219+0000","last_change":"2024-09-10T19:00:12.005658+0000","last_active":"2024-09-10T19:00:31.181219+0000","last_peered":"2024-09-10T19:00:31.181219+0000","last_clean":"2024-09-10T19:00:31.181219+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T19:00:31.181219+0000","last_undegraded":"2024-09-10T19:00:31.181219+0000","last_fullsized":"2024-09-10T19:00:31.181219+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.658532+0000","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177596+0000","last_change":"2024-09-10T18:58:47.555840+0000","last_active":"2024-09-10T19:00:31.177596+0000","last_peered":"2024-09-10T19:00:31.177596+0000","last_clean":"2024-09-10T19:00:31.177596+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:00:31.177596+0000","last_undegraded":"2024-09-10T19:00:31.177596+0000","last_fullsized":"2024-09-10T19:00:31.177596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.555637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.555637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032502030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":431,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.576839+0000","last_change":"2024-09-10T18:58:59.454163+0000","last_active":"2024-09-10T19:00:31.576839+0000","last_peered":"2024-09-10T19:00:31.576839+0000","last_clean":"2024-09-10T19:00:31.576839+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:00:31.576839+0000","last_undegraded":"2024-09-10T19:00:31.576839+0000","last_fullsized":"2024-09-10T19:00:31.576839+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180154+0000","last_change":"2024-09-10T18:58:54.237739+0000","last_active":"2024-09-10T19:00:31.180154+0000","last_peered":"2024-09-10T19:00:31.180154+0000","last_clean":"2024-09-10T19:00:31.180154+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:00:31.180154+0000","last_undegraded":"2024-09-10T19:00:31.180154+0000","last_fullsized":"2024-09-10T19:00:31.180154+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.237585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.237585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092150240000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":495,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180286+0000","last_change":"2024-09-10T18:58:52.253613+0000","last_active":"2024-09-10T19:00:31.180286+0000","last_peered":"2024-09-10T19:00:31.180286+0000","last_clean":"2024-09-10T19:00:31.180286+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:00:31.180286+0000","last_undegraded":"2024-09-10T19:00:31.180286+0000","last_fullsized":"2024-09-10T19:00:31.180286+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'25","last_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.253376+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011205784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":462,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174004+0000","last_change":"2024-09-10T19:00:10.225407+0000","last_active":"2024-09-10T19:00:31.174004+0000","last_peered":"2024-09-10T19:00:31.174004+0000","last_clean":"2024-09-10T19:00:31.174004+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:00:31.174004+0000","last_undegraded":"2024-09-10T19:00:31.174004+0000","last_fullsized":"2024-09-10T19:00:31.174004+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.225321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.225321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.536640+0000","scrub_duration":0.0027779139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":446,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175530+0000","last_change":"2024-09-10T19:00:29.731792+0000","last_active":"2024-09-10T19:00:31.175530+0000","last_peered":"2024-09-10T19:00:31.175530+0000","last_clean":"2024-09-10T19:00:31.175530+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:00:31.175530+0000","last_undegraded":"2024-09-10T19:00:31.175530+0000","last_fullsized":"2024-09-10T19:00:31.175530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:00:29.731727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.097458+0000","scrub_duration":0.012104484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181605+0000","last_change":"2024-09-10T18:58:55.261885+0000","last_active":"2024-09-10T19:00:31.181605+0000","last_peered":"2024-09-10T19:00:31.181605+0000","last_clean":"2024-09-10T19:00:31.181605+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:00:31.181605+0000","last_undegraded":"2024-09-10T19:00:31.181605+0000","last_fullsized":"2024-09-10T19:00:31.181605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.261635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.261635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046216190000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166434+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T19:00:31.166434+0000","last_peered":"2024-09-10T19:00:31.166434+0000","last_clean":"2024-09-10T19:00:31.166434+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:00:31.166434+0000","last_undegraded":"2024-09-10T19:00:31.166434+0000","last_fullsized":"2024-09-10T19:00:31.166434+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":470,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577994+0000","last_change":"2024-09-10T18:59:01.489222+0000","last_active":"2024-09-10T19:00:31.577994+0000","last_peered":"2024-09-10T19:00:31.577994+0000","last_clean":"2024-09-10T19:00:31.577994+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:00:31.577994+0000","last_undegraded":"2024-09-10T19:00:31.577994+0000","last_fullsized":"2024-09-10T19:00:31.577994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-09-10T18:59:01.489057+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.489057+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045114960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":451,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:34.708527+0000","last_change":"2024-09-10T19:00:34.708486+0000","last_active":"2024-09-10T19:00:34.708527+0000","last_peered":"2024-09-10T19:00:34.708527+0000","last_clean":"2024-09-10T19:00:34.708527+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:00:34.708527+0000","last_undegraded":"2024-09-10T19:00:34.708527+0000","last_fullsized":"2024-09-10T19:00:34.708527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:34.708428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:34.708428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.828851+0000","scrub_duration":0.0023757650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177492+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T19:00:31.177492+0000","last_peered":"2024-09-10T19:00:31.177492+0000","last_clean":"2024-09-10T19:00:31.177492+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:00:31.177492+0000","last_undegraded":"2024-09-10T19:00:31.177492+0000","last_fullsized":"2024-09-10T19:00:31.177492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":426,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178597+0000","last_change":"2024-09-10T18:58:44.847935+0000","last_active":"2024-09-10T19:00:31.178597+0000","last_peered":"2024-09-10T19:00:31.178597+0000","last_clean":"2024-09-10T19:00:31.178597+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:00:31.178597+0000","last_undegraded":"2024-09-10T19:00:31.178597+0000","last_fullsized":"2024-09-10T19:00:31.178597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:44.847782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T18:58:44.847782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003136481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:32.154232+0000","last_change":"2024-09-10T19:00:32.154082+0000","last_active":"2024-09-10T19:00:32.154232+0000","last_peered":"2024-09-10T19:00:32.154232+0000","last_clean":"2024-09-10T19:00:32.154232+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:00:32.154232+0000","last_undegraded":"2024-09-10T19:00:32.154232+0000","last_fullsized":"2024-09-10T19:00:32.154232+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:32.153923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.153923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.158934+0000","scrub_duration":0.003341105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":498,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181218+0000","last_change":"2024-09-10T19:00:27.155044+0000","last_active":"2024-09-10T19:00:31.181218+0000","last_peered":"2024-09-10T19:00:31.181218+0000","last_clean":"2024-09-10T19:00:31.181218+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:00:31.181218+0000","last_undegraded":"2024-09-10T19:00:31.181218+0000","last_fullsized":"2024-09-10T19:00:31.181218+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:27.154715+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.154715+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:32.274725+0000","scrub_duration":0.005022098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":474,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173979+0000","last_change":"2024-09-10T19:00:06.139605+0000","last_active":"2024-09-10T19:00:31.173979+0000","last_peered":"2024-09-10T19:00:31.173979+0000","last_clean":"2024-09-10T19:00:31.173979+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:00:31.173979+0000","last_undegraded":"2024-09-10T19:00:31.173979+0000","last_fullsized":"2024-09-10T19:00:31.173979+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:00:06.139460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:06.139460+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:08.785287+0000","scrub_duration":0.0034469779999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":443,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575037+0000","last_change":"2024-09-10T19:00:00.313745+0000","last_active":"2024-09-10T19:00:31.575037+0000","last_peered":"2024-09-10T19:00:31.575037+0000","last_clean":"2024-09-10T19:00:31.575037+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:00:31.575037+0000","last_undegraded":"2024-09-10T19:00:31.575037+0000","last_fullsized":"2024-09-10T19:00:31.575037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.313582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.313582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:15.500345+0000","scrub_duration":0.0034259580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":441,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578051+0000","last_change":"2024-09-10T19:00:30.155345+0000","last_active":"2024-09-10T19:00:31.578051+0000","last_peered":"2024-09-10T19:00:31.578051+0000","last_clean":"2024-09-10T19:00:31.578051+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:00:31.578051+0000","last_undegraded":"2024-09-10T19:00:31.578051+0000","last_fullsized":"2024-09-10T19:00:31.578051+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:30.155089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.155089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:42.013903+0000","scrub_duration":0.0038319109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166066+0000","last_change":"2024-09-10T18:58:38.640229+0000","last_active":"2024-09-10T19:00:31.166066+0000","last_peered":"2024-09-10T19:00:31.166066+0000","last_clean":"2024-09-10T19:00:31.166066+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T19:00:31.166066+0000","last_undegraded":"2024-09-10T19:00:31.166066+0000","last_fullsized":"2024-09-10T19:00:31.166066+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:38.640076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.640076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038603940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":469,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181918+0000","last_change":"2024-09-10T18:58:47.178004+0000","last_active":"2024-09-10T19:00:31.181918+0000","last_peered":"2024-09-10T19:00:31.181918+0000","last_clean":"2024-09-10T19:00:31.181918+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:00:31.181918+0000","last_undegraded":"2024-09-10T19:00:31.181918+0000","last_fullsized":"2024-09-10T19:00:31.181918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:47.177751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.177751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059440730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":492,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.169204+0000","last_change":"2024-09-10T18:59:59.421165+0000","last_active":"2024-09-10T19:00:31.169204+0000","last_peered":"2024-09-10T19:00:31.169204+0000","last_clean":"2024-09-10T19:00:31.169204+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:00:31.169204+0000","last_undegraded":"2024-09-10T19:00:31.169204+0000","last_fullsized":"2024-09-10T19:00:31.169204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:59:59.420998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.420998+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:22.365936+0000","scrub_duration":0.0045326000000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":594,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.321542+0000","last_change":"2024-09-10T19:00:15.117567+0000","last_active":"2024-09-10T19:00:31.321542+0000","last_peered":"2024-09-10T19:00:31.321542+0000","last_clean":"2024-09-10T19:00:31.321542+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T19:00:31.321542+0000","last_undegraded":"2024-09-10T19:00:31.321542+0000","last_fullsized":"2024-09-10T19:00:31.321542+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:15.117380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.117380+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.034441+0000","scrub_duration":0.004280982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":428,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.172888+0000","last_change":"2024-09-10T18:57:50.173500+0000","last_active":"2024-09-10T19:00:31.172888+0000","last_peered":"2024-09-10T19:00:31.172888+0000","last_clean":"2024-09-10T19:00:31.172888+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:00:31.172888+0000","last_undegraded":"2024-09-10T19:00:31.172888+0000","last_fullsized":"2024-09-10T19:00:31.172888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:50.173403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T18:57:50.173403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023035519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":370,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175203+0000","last_change":"2024-09-10T18:59:55.520903+0000","last_active":"2024-09-10T19:00:31.175203+0000","last_peered":"2024-09-10T19:00:31.175203+0000","last_clean":"2024-09-10T19:00:31.175203+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:00:31.175203+0000","last_undegraded":"2024-09-10T19:00:31.175203+0000","last_fullsized":"2024-09-10T19:00:31.175203+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:59:55.520655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.520655+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:06.622877+0000","scrub_duration":0.0073248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":479,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178659+0000","last_change":"2024-09-10T19:00:02.055337+0000","last_active":"2024-09-10T19:00:31.178659+0000","last_peered":"2024-09-10T19:00:31.178659+0000","last_clean":"2024-09-10T19:00:31.178659+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:00:31.178659+0000","last_undegraded":"2024-09-10T19:00:31.178659+0000","last_fullsized":"2024-09-10T19:00:31.178659+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:02.055169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.055169+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:18.062663+0000","scrub_duration":0.0050398609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":462,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175418+0000","last_change":"2024-09-10T19:00:23.196206+0000","last_active":"2024-09-10T19:00:31.175418+0000","last_peered":"2024-09-10T19:00:31.175418+0000","last_clean":"2024-09-10T19:00:31.175418+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:00:31.175418+0000","last_undegraded":"2024-09-10T19:00:31.175418+0000","last_fullsized":"2024-09-10T19:00:31.175418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:34.688462+0000","scrub_duration":0.0034334669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181244+0000","last_change":"2024-09-10T19:00:12.129140+0000","last_active":"2024-09-10T19:00:31.181244+0000","last_peered":"2024-09-10T19:00:31.181244+0000","last_clean":"2024-09-10T19:00:31.181244+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:00:31.181244+0000","last_undegraded":"2024-09-10T19:00:31.181244+0000","last_fullsized":"2024-09-10T19:00:31.181244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:12.128979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.128979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.750053+0000","scrub_duration":0.0031738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":446,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173535+0000","last_change":"2024-09-10T18:59:53.990233+0000","last_active":"2024-09-10T19:00:31.173535+0000","last_peered":"2024-09-10T19:00:31.173535+0000","last_clean":"2024-09-10T19:00:31.173535+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:00:31.173535+0000","last_undegraded":"2024-09-10T19:00:31.173535+0000","last_fullsized":"2024-09-10T19:00:31.173535+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.990034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.990034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:21.398651+0000","scrub_duration":0.00353419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168565+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T19:00:31.168565+0000","last_peered":"2024-09-10T19:00:31.168565+0000","last_clean":"2024-09-10T19:00:31.168565+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:00:31.168565+0000","last_undegraded":"2024-09-10T19:00:31.168565+0000","last_fullsized":"2024-09-10T19:00:31.168565+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":462,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174614+0000","last_change":"2024-09-10T19:00:21.565543+0000","last_active":"2024-09-10T19:00:31.174614+0000","last_peered":"2024-09-10T19:00:31.174614+0000","last_clean":"2024-09-10T19:00:31.174614+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:00:31.174614+0000","last_undegraded":"2024-09-10T19:00:31.174614+0000","last_fullsized":"2024-09-10T19:00:31.174614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:00:21.565373+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.565373+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:28.542472+0000","scrub_duration":0.0037082009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175590+0000","last_change":"2024-09-10T19:00:10.348880+0000","last_active":"2024-09-10T19:00:31.175590+0000","last_peered":"2024-09-10T19:00:31.175590+0000","last_clean":"2024-09-10T19:00:31.175590+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:00:31.175590+0000","last_undegraded":"2024-09-10T19:00:31.175590+0000","last_fullsized":"2024-09-10T19:00:31.175590+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.348709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.348709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.066540+0000","scrub_duration":0.0033057529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":430,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175184+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T19:00:31.175184+0000","last_peered":"2024-09-10T19:00:31.175184+0000","last_clean":"2024-09-10T19:00:31.175184+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:00:31.175184+0000","last_undegraded":"2024-09-10T19:00:31.175184+0000","last_fullsized":"2024-09-10T19:00:31.175184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179024+0000","last_change":"2024-09-10T19:00:23.089654+0000","last_active":"2024-09-10T19:00:31.179024+0000","last_peered":"2024-09-10T19:00:31.179024+0000","last_clean":"2024-09-10T19:00:31.179024+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:00:31.179024+0000","last_undegraded":"2024-09-10T19:00:31.179024+0000","last_fullsized":"2024-09-10T19:00:31.179024+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.089483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.089483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.384760+0000","scrub_duration":0.003188331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.176998+0000","last_change":"2024-09-10T18:59:01.663804+0000","last_active":"2024-09-10T19:00:31.176998+0000","last_peered":"2024-09-10T19:00:31.176998+0000","last_clean":"2024-09-10T19:00:31.176998+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:00:31.176998+0000","last_undegraded":"2024-09-10T19:00:31.176998+0000","last_fullsized":"2024-09-10T19:00:31.176998+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.663643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.663643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":474,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178843+0000","last_change":"2024-09-10T18:58:56.293280+0000","last_active":"2024-09-10T19:00:31.178843+0000","last_peered":"2024-09-10T19:00:31.178843+0000","last_clean":"2024-09-10T19:00:31.178843+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:00:31.178843+0000","last_undegraded":"2024-09-10T19:00:31.178843+0000","last_fullsized":"2024-09-10T19:00:31.178843+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'16","last_scrub_stamp":"2024-09-10T18:58:56.293027+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.293027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069050090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578325+0000","last_change":"2024-09-10T19:00:05.196515+0000","last_active":"2024-09-10T19:00:31.578325+0000","last_peered":"2024-09-10T19:00:31.578325+0000","last_clean":"2024-09-10T19:00:31.578325+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:00:31.578325+0000","last_undegraded":"2024-09-10T19:00:31.578325+0000","last_fullsized":"2024-09-10T19:00:31.578325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:05.196346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.196346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.115412+0000","scrub_duration":0.0033251359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":588,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.324223+0000","last_change":"2024-09-10T19:00:26.925255+0000","last_active":"2024-09-10T19:00:31.324223+0000","last_peered":"2024-09-10T19:00:31.324223+0000","last_clean":"2024-09-10T19:00:31.324223+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:00:31.324223+0000","last_undegraded":"2024-09-10T19:00:31.324223+0000","last_fullsized":"2024-09-10T19:00:31.324223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:26.925124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.925124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.740275+0000","scrub_duration":0.0037103370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180944+0000","last_change":"2024-09-10T19:00:18.141379+0000","last_active":"2024-09-10T19:00:31.180944+0000","last_peered":"2024-09-10T19:00:31.180944+0000","last_clean":"2024-09-10T19:00:31.180944+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:00:31.180944+0000","last_undegraded":"2024-09-10T19:00:31.180944+0000","last_fullsized":"2024-09-10T19:00:31.180944+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.141228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.141228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.638561+0000","scrub_duration":0.003005354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:33.116304+0000","last_change":"2024-09-10T19:00:33.116212+0000","last_active":"2024-09-10T19:00:33.116304+0000","last_peered":"2024-09-10T19:00:33.116304+0000","last_clean":"2024-09-10T19:00:33.116304+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:00:33.116304+0000","last_undegraded":"2024-09-10T19:00:33.116304+0000","last_fullsized":"2024-09-10T19:00:33.116304+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.116061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.408011+0000","scrub_duration":0.003119681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":465,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177751+0000","last_change":"2024-09-10T19:00:16.130225+0000","last_active":"2024-09-10T19:00:31.177751+0000","last_peered":"2024-09-10T19:00:31.177751+0000","last_clean":"2024-09-10T19:00:31.177751+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:00:31.177751+0000","last_undegraded":"2024-09-10T19:00:31.177751+0000","last_fullsized":"2024-09-10T19:00:31.177751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:16.130067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.130067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:25.570493+0000","scrub_duration":0.0042760100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":442,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179800+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T19:00:31.179800+0000","last_peered":"2024-09-10T19:00:31.179800+0000","last_clean":"2024-09-10T19:00:31.179800+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:00:31.179800+0000","last_undegraded":"2024-09-10T19:00:31.179800+0000","last_fullsized":"2024-09-10T19:00:31.179800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":568,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577398+0000","last_change":"2024-09-10T18:58:40.405050+0000","last_active":"2024-09-10T19:00:31.577398+0000","last_peered":"2024-09-10T19:00:31.577398+0000","last_clean":"2024-09-10T19:00:31.577398+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:00:31.577398+0000","last_undegraded":"2024-09-10T19:00:31.577398+0000","last_fullsized":"2024-09-10T19:00:31.577398+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:58:40.404810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:40.404810+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047181080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168996+0000","last_change":"2024-09-10T19:00:18.076688+0000","last_active":"2024-09-10T19:00:31.168996+0000","last_peered":"2024-09-10T19:00:31.168996+0000","last_clean":"2024-09-10T19:00:31.168996+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T19:00:31.168996+0000","last_undegraded":"2024-09-10T19:00:31.168996+0000","last_fullsized":"2024-09-10T19:00:31.168996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.076465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.076465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:33.575144+0000","scrub_duration":0.0043459409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":347,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.180549+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T19:00:31.180549+0000","last_peered":"2024-09-10T19:00:31.180549+0000","last_clean":"2024-09-10T19:00:31.180549+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:00:31.180549+0000","last_undegraded":"2024-09-10T19:00:31.180549+0000","last_fullsized":"2024-09-10T19:00:31.180549+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":473,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:32.173339+0000","last_change":"2024-09-10T19:00:32.173178+0000","last_active":"2024-09-10T19:00:32.173339+0000","last_peered":"2024-09-10T19:00:32.173339+0000","last_clean":"2024-09-10T19:00:32.173339+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:00:32.173339+0000","last_undegraded":"2024-09-10T19:00:32.173339+0000","last_fullsized":"2024-09-10T19:00:32.173339+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:32.173009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.173009+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:53.697500+0000","scrub_duration":0.0042085289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":440,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175508+0000","last_change":"2024-09-10T18:58:55.639930+0000","last_active":"2024-09-10T19:00:31.175508+0000","last_peered":"2024-09-10T19:00:31.175508+0000","last_clean":"2024-09-10T19:00:31.175508+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:00:31.175508+0000","last_undegraded":"2024-09-10T19:00:31.175508+0000","last_fullsized":"2024-09-10T19:00:31.175508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.639777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.639777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003186637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577853+0000","last_change":"2024-09-10T19:00:19.289035+0000","last_active":"2024-09-10T19:00:31.577853+0000","last_peered":"2024-09-10T19:00:31.577853+0000","last_clean":"2024-09-10T19:00:31.577853+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:00:31.577853+0000","last_undegraded":"2024-09-10T19:00:31.577853+0000","last_fullsized":"2024-09-10T19:00:31.577853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:19.288883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:19.288883+0000","last_clean_scrub_stamp":"2024-09-10T19:00:19.288883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:26.277775+0000","scrub_duration":0.0090246279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":439,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.176953+0000","last_change":"2024-09-10T19:00:04.060523+0000","last_active":"2024-09-10T19:00:31.176953+0000","last_peered":"2024-09-10T19:00:31.176953+0000","last_clean":"2024-09-10T19:00:31.176953+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:00:31.176953+0000","last_undegraded":"2024-09-10T19:00:31.176953+0000","last_fullsized":"2024-09-10T19:00:31.176953+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.060371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.060371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:04.162315+0000","scrub_duration":0.0031455099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.166919+0000","last_change":"2024-09-10T18:58:46.501591+0000","last_active":"2024-09-10T19:00:31.166919+0000","last_peered":"2024-09-10T19:00:31.166919+0000","last_clean":"2024-09-10T19:00:31.166919+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:00:31.166919+0000","last_undegraded":"2024-09-10T19:00:31.166919+0000","last_fullsized":"2024-09-10T19:00:31.166919+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:46.501508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:46.501508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036484099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":486,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167843+0000","last_change":"2024-09-10T19:00:13.188540+0000","last_active":"2024-09-10T19:00:31.167843+0000","last_peered":"2024-09-10T19:00:31.167843+0000","last_clean":"2024-09-10T19:00:31.167843+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T19:00:31.167843+0000","last_undegraded":"2024-09-10T19:00:31.167843+0000","last_fullsized":"2024-09-10T19:00:31.167843+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:00:13.188382+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.188382+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:20.453734+0000","scrub_duration":0.0046400579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":451,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435940+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T19:00:31.435940+0000","last_peered":"2024-09-10T19:00:31.435940+0000","last_clean":"2024-09-10T19:00:31.435940+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:00:31.435940+0000","last_undegraded":"2024-09-10T19:00:31.435940+0000","last_fullsized":"2024-09-10T19:00:31.435940+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":439,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167918+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T19:00:31.167918+0000","last_peered":"2024-09-10T19:00:31.167918+0000","last_clean":"2024-09-10T19:00:31.167918+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:00:31.167918+0000","last_undegraded":"2024-09-10T19:00:31.167918+0000","last_fullsized":"2024-09-10T19:00:31.167918+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":441,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171561+0000","last_change":"2024-09-10T18:59:55.150083+0000","last_active":"2024-09-10T19:00:31.171561+0000","last_peered":"2024-09-10T19:00:31.171561+0000","last_clean":"2024-09-10T19:00:31.171561+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:00:31.171561+0000","last_undegraded":"2024-09-10T19:00:31.171561+0000","last_fullsized":"2024-09-10T19:00:31.171561+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:55.150009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.150009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:05.988153+0000","scrub_duration":0.0024462989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":358,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167892+0000","last_change":"2024-09-10T18:58:49.517454+0000","last_active":"2024-09-10T19:00:31.167892+0000","last_peered":"2024-09-10T19:00:31.167892+0000","last_clean":"2024-09-10T19:00:31.167892+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T19:00:31.167892+0000","last_undegraded":"2024-09-10T19:00:31.167892+0000","last_fullsized":"2024-09-10T19:00:31.167892+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.517273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.517273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039870039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":451,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.183257+0000","last_change":"2024-09-10T18:57:55.666228+0000","last_active":"2024-09-10T19:00:31.183257+0000","last_peered":"2024-09-10T19:00:31.183257+0000","last_clean":"2024-09-10T19:00:31.183257+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:00:31.183257+0000","last_undegraded":"2024-09-10T19:00:31.183257+0000","last_fullsized":"2024-09-10T19:00:31.183257+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'9","last_scrub_stamp":"2024-09-10T18:57:55.666023+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T18:57:55.666023+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046888279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":437,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577621+0000","last_change":"2024-09-10T18:58:53.461054+0000","last_active":"2024-09-10T19:00:31.577621+0000","last_peered":"2024-09-10T19:00:31.577621+0000","last_clean":"2024-09-10T19:00:31.577621+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:00:31.577621+0000","last_undegraded":"2024-09-10T19:00:31.577621+0000","last_fullsized":"2024-09-10T19:00:31.577621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:53.460812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T18:58:53.460812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042777029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":433,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.170718+0000","last_change":"2024-09-10T18:59:53.179273+0000","last_active":"2024-09-10T19:00:31.170718+0000","last_peered":"2024-09-10T19:00:31.170718+0000","last_clean":"2024-09-10T19:00:31.170718+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:00:31.170718+0000","last_undegraded":"2024-09-10T19:00:31.170718+0000","last_fullsized":"2024-09-10T19:00:31.170718+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.179202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.179202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:09.193560+0000","scrub_duration":0.0015188129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.179831+0000","last_change":"2024-09-10T19:00:08.102047+0000","last_active":"2024-09-10T19:00:31.179831+0000","last_peered":"2024-09-10T19:00:31.179831+0000","last_clean":"2024-09-10T19:00:31.179831+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:00:31.179831+0000","last_undegraded":"2024-09-10T19:00:31.179831+0000","last_fullsized":"2024-09-10T19:00:31.179831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:08.101901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:00:08.101901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:24.050832+0000","scrub_duration":0.0028849869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.433796+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T19:00:31.433796+0000","last_peered":"2024-09-10T19:00:31.433796+0000","last_clean":"2024-09-10T19:00:31.433796+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:00:31.433796+0000","last_undegraded":"2024-09-10T19:00:31.433796+0000","last_fullsized":"2024-09-10T19:00:31.433796+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":460,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.577382+0000","last_change":"2024-09-10T18:59:04.505215+0000","last_active":"2024-09-10T19:00:31.577382+0000","last_peered":"2024-09-10T19:00:31.577382+0000","last_clean":"2024-09-10T19:00:31.577382+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:00:31.577382+0000","last_undegraded":"2024-09-10T19:00:31.577382+0000","last_fullsized":"2024-09-10T19:00:31.577382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:59:04.505046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.505046+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045725100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":450,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177416+0000","last_change":"2024-09-10T19:00:11.386471+0000","last_active":"2024-09-10T19:00:31.177416+0000","last_peered":"2024-09-10T19:00:31.177416+0000","last_clean":"2024-09-10T19:00:31.177416+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:00:31.177416+0000","last_undegraded":"2024-09-10T19:00:31.177416+0000","last_fullsized":"2024-09-10T19:00:31.177416+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:11.386293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:11.386293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:34.331345+0000","scrub_duration":0.0033555310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":430,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171917+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T19:00:31.171917+0000","last_peered":"2024-09-10T19:00:31.171917+0000","last_clean":"2024-09-10T19:00:31.171917+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:00:31.171917+0000","last_undegraded":"2024-09-10T19:00:31.171917+0000","last_fullsized":"2024-09-10T19:00:31.171917+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173038+0000","last_change":"2024-09-10T19:00:23.839038+0000","last_active":"2024-09-10T19:00:31.173038+0000","last_peered":"2024-09-10T19:00:31.173038+0000","last_clean":"2024-09-10T19:00:31.173038+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:00:31.173038+0000","last_undegraded":"2024-09-10T19:00:31.173038+0000","last_fullsized":"2024-09-10T19:00:31.173038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.838880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.838880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:52.392526+0000","scrub_duration":0.0033929009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":347,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178904+0000","last_change":"2024-09-10T18:57:39.580096+0000","last_active":"2024-09-10T19:00:31.178904+0000","last_peered":"2024-09-10T19:00:31.178904+0000","last_clean":"2024-09-10T19:00:31.178904+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:00:31.178904+0000","last_undegraded":"2024-09-10T19:00:31.178904+0000","last_fullsized":"2024-09-10T19:00:31.178904+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:39.579843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:57:39.579843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068212799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178796+0000","last_change":"2024-09-10T18:58:58.248456+0000","last_active":"2024-09-10T19:00:31.178796+0000","last_peered":"2024-09-10T19:00:31.178796+0000","last_clean":"2024-09-10T19:00:31.178796+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:00:31.178796+0000","last_undegraded":"2024-09-10T19:00:31.178796+0000","last_fullsized":"2024-09-10T19:00:31.178796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":499,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.433818+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T19:00:31.433818+0000","last_peered":"2024-09-10T19:00:31.433818+0000","last_clean":"2024-09-10T19:00:31.433818+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:00:31.433818+0000","last_undegraded":"2024-09-10T19:00:31.433818+0000","last_fullsized":"2024-09-10T19:00:31.433818+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.575156+0000","last_change":"2024-09-10T19:00:27.159256+0000","last_active":"2024-09-10T19:00:31.575156+0000","last_peered":"2024-09-10T19:00:31.575156+0000","last_clean":"2024-09-10T19:00:31.575156+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:00:31.575156+0000","last_undegraded":"2024-09-10T19:00:31.575156+0000","last_fullsized":"2024-09-10T19:00:31.575156+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:27.159007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.159007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:54.356014+0000","scrub_duration":0.0031545399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.181085+0000","last_change":"2024-09-10T19:00:23.207215+0000","last_active":"2024-09-10T19:00:31.181085+0000","last_peered":"2024-09-10T19:00:31.181085+0000","last_clean":"2024-09-10T19:00:31.181085+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:00:31.181085+0000","last_undegraded":"2024-09-10T19:00:31.181085+0000","last_fullsized":"2024-09-10T19:00:31.181085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.207050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.229672+0000","scrub_duration":0.0061363199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.176435+0000","last_change":"2024-09-10T18:59:58.485524+0000","last_active":"2024-09-10T19:00:31.176435+0000","last_peered":"2024-09-10T19:00:31.176435+0000","last_clean":"2024-09-10T19:00:31.176435+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:00:31.176435+0000","last_undegraded":"2024-09-10T19:00:31.176435+0000","last_fullsized":"2024-09-10T19:00:31.176435+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.485240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.485240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.282886+0000","scrub_duration":0.0043729700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":445,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.435379+0000","last_change":"2024-09-10T19:00:09.572036+0000","last_active":"2024-09-10T19:00:31.435379+0000","last_peered":"2024-09-10T19:00:31.435379+0000","last_clean":"2024-09-10T19:00:31.435379+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:00:31.435379+0000","last_undegraded":"2024-09-10T19:00:31.435379+0000","last_fullsized":"2024-09-10T19:00:31.435379+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:09.571847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.571847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:28.433824+0000","scrub_duration":0.0034123030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":457,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.171221+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T19:00:31.171221+0000","last_peered":"2024-09-10T19:00:31.171221+0000","last_clean":"2024-09-10T19:00:31.171221+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:00:31.171221+0000","last_undegraded":"2024-09-10T19:00:31.171221+0000","last_fullsized":"2024-09-10T19:00:31.171221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":434,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173681+0000","last_change":"2024-09-10T18:57:52.092326+0000","last_active":"2024-09-10T19:00:31.173681+0000","last_peered":"2024-09-10T19:00:31.173681+0000","last_clean":"2024-09-10T19:00:31.173681+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:00:31.173681+0000","last_undegraded":"2024-09-10T19:00:31.173681+0000","last_fullsized":"2024-09-10T19:00:31.173681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:57:52.092223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:57:52.092223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060382309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":608,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.320890+0000","last_change":"2024-09-10T19:00:25.684751+0000","last_active":"2024-09-10T19:00:31.320890+0000","last_peered":"2024-09-10T19:00:31.320890+0000","last_clean":"2024-09-10T19:00:31.320890+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:00:31.320890+0000","last_undegraded":"2024-09-10T19:00:31.320890+0000","last_fullsized":"2024-09-10T19:00:31.320890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:00:25.684577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.684577+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.541358+0000","scrub_duration":0.0038437710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177284+0000","last_change":"2024-09-10T18:59:00.644451+0000","last_active":"2024-09-10T19:00:31.177284+0000","last_peered":"2024-09-10T19:00:31.177284+0000","last_clean":"2024-09-10T19:00:31.177284+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:00:31.177284+0000","last_undegraded":"2024-09-10T19:00:31.177284+0000","last_fullsized":"2024-09-10T19:00:31.177284+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T18:59:00.644299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00344357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":447,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.168000+0000","last_change":"2024-09-10T19:00:21.100228+0000","last_active":"2024-09-10T19:00:31.168000+0000","last_peered":"2024-09-10T19:00:31.168000+0000","last_clean":"2024-09-10T19:00:31.168000+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:00:31.168000+0000","last_undegraded":"2024-09-10T19:00:31.168000+0000","last_fullsized":"2024-09-10T19:00:31.168000+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.100058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.100058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.121511+0000","scrub_duration":0.0034414139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":449,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175684+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T19:00:31.175684+0000","last_peered":"2024-09-10T19:00:31.175684+0000","last_clean":"2024-09-10T19:00:31.175684+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:00:31.175684+0000","last_undegraded":"2024-09-10T19:00:31.175684+0000","last_fullsized":"2024-09-10T19:00:31.175684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":453,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.178631+0000","last_change":"2024-09-10T19:00:02.052175+0000","last_active":"2024-09-10T19:00:31.178631+0000","last_peered":"2024-09-10T19:00:31.178631+0000","last_clean":"2024-09-10T19:00:31.178631+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:00:31.178631+0000","last_undegraded":"2024-09-10T19:00:31.178631+0000","last_fullsized":"2024-09-10T19:00:31.178631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:00:02.051995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.051995+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:30.499770+0000","scrub_duration":0.0042242440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":458,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173798+0000","last_change":"2024-09-10T19:00:00.097374+0000","last_active":"2024-09-10T19:00:31.173798+0000","last_peered":"2024-09-10T19:00:31.173798+0000","last_clean":"2024-09-10T19:00:31.173798+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:00:31.173798+0000","last_undegraded":"2024-09-10T19:00:31.173798+0000","last_fullsized":"2024-09-10T19:00:31.173798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.097226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.097226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:12.731885+0000","scrub_duration":0.0030874499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":354,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.578196+0000","last_change":"2024-09-10T18:59:54.415459+0000","last_active":"2024-09-10T19:00:31.578196+0000","last_peered":"2024-09-10T19:00:31.578196+0000","last_clean":"2024-09-10T19:00:31.578196+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:00:31.578196+0000","last_undegraded":"2024-09-10T19:00:31.578196+0000","last_fullsized":"2024-09-10T19:00:31.578196+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:54.415297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:54.415297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:00:55.651756+0000","scrub_duration":0.0031427790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":438,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.175038+0000","last_change":"2024-09-10T19:00:16.862735+0000","last_active":"2024-09-10T19:00:31.175038+0000","last_peered":"2024-09-10T19:00:31.175038+0000","last_clean":"2024-09-10T19:00:31.175038+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:00:31.175038+0000","last_undegraded":"2024-09-10T19:00:31.175038+0000","last_fullsized":"2024-09-10T19:00:31.175038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:16.862586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.862586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:27.431531+0000","scrub_duration":0.011943657999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":444,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.173820+0000","last_change":"2024-09-10T19:00:17.847339+0000","last_active":"2024-09-10T19:00:31.173820+0000","last_peered":"2024-09-10T19:00:31.173820+0000","last_clean":"2024-09-10T19:00:31.173820+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:00:31.173820+0000","last_undegraded":"2024-09-10T19:00:31.173820+0000","last_fullsized":"2024-09-10T19:00:31.173820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.847185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.847185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:36.511351+0000","scrub_duration":0.0036286880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":476,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.167019+0000","last_change":"2024-09-10T19:00:14.152011+0000","last_active":"2024-09-10T19:00:31.167019+0000","last_peered":"2024-09-10T19:00:31.167019+0000","last_clean":"2024-09-10T19:00:31.167019+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:00:31.167019+0000","last_undegraded":"2024-09-10T19:00:31.167019+0000","last_fullsized":"2024-09-10T19:00:31.167019+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:14.151862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.151862+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:16.920562+0000","scrub_duration":0.0041793289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":430,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.174504+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T19:00:31.174504+0000","last_peered":"2024-09-10T19:00:31.174504+0000","last_clean":"2024-09-10T19:00:31.174504+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:00:31.174504+0000","last_undegraded":"2024-09-10T19:00:31.174504+0000","last_fullsized":"2024-09-10T19:00:31.174504+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":357,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-09-10T19:00:31.177127+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T19:00:31.177127+0000","last_peered":"2024-09-10T19:00:31.177127+0000","last_clean":"2024-09-10T19:00:31.177127+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:00:31.177127+0000","last_undegraded":"2024-09-10T19:00:31.177127+0000","last_fullsized":"2024-09-10T19:00:31.177127+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":215,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":11,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195783,"num_objects":159,"num_object_clones":2,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":109,"num_read_kb":9,"num_write":599,"num_write_kb":12688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26480640,"data_stored":24587349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1885,"ondisk_log_size":1885,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":652,"num_read_kb":647,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972499,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317068,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93428084,"statfs":{"total":95995035648,"available":95670358016,"internally_reserved":0,"allocated":15921152,"data_stored":12799578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.87,"15min":0.90700000000000003},"min":{"1min":0.40300000000000002,"5min":0.311,"15min":0.311},"max":{"1min":1.3839999999999999,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":1.012},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.92100000000000004,"15min":0.92400000000000004},"min":{"1min":0.50600000000000001,"5min":0.372,"15min":0.34699999999999998},"max":{"1min":1.2529999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.056}]},{"osd":1,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.87,"15min":0.89000000000000001},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.208,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.97499999999999998,"15min":1.004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.421,"5min":1.742,"15min":1.8280000000000001},"last":1.266}]},{"osd":2,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.014,"15min":1.0289999999999999},"min":{"1min":0.45000000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.6739999999999999,"5min":2.9199999999999999,"15min":3.4990000000000001},"last":1.143},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.0009999999999999,"15min":0.98999999999999999},"min":{"1min":0.64500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.97,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.98099999999999998}]},{"osd":3,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0629999999999999,"15min":1.018},"min":{"1min":0.67900000000000005,"5min":0.51400000000000001,"15min":0.48299999999999998},"max":{"1min":1.9390000000000001,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.99099999999999999,"15min":0.97599999999999998},"min":{"1min":0.56200000000000006,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.833,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":1.099}]},{"osd":4,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.014,"15min":1.0149999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3700000000000001,"5min":1.4279999999999999,"15min":2.1989999999999998},"last":1.22},{"interface":"front","average":{"1min":1.046,"5min":1.127,"15min":1.0609999999999999},"min":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0310000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":0.78100000000000003}]},{"osd":5,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.998,"15min":0.99199999999999999},"min":{"1min":0.56899999999999995,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.7669999999999999,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1379999999999999,"15min":1.1639999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":2.1419999999999999},"last":1.3080000000000001}]},{"osd":6,"last update":"Tue Sep 10 18:59:49 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.179,"15min":1.2150000000000001},"min":{"1min":0.60899999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.468,"5min":1.591,"15min":2.2749999999999999},"last":1.357},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2150000000000001,"15min":1.1950000000000001},"min":{"1min":0.70599999999999996,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":2.1560000000000001,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.169}]}]},{"osd":6,"up_from":52,"seq":223338299549,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313120,"kb_used_data":7448,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432032,"statfs":{"total":95995035648,"available":95674400768,"internally_reserved":0,"allocated":7626752,"data_stored":4603338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.88700000000000001,"15min":0.88400000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.36399999999999999},"max":{"1min":1.181,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.95599999999999996,"15min":0.95899999999999996},"min":{"1min":0.46300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.276,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":0.998}]},{"osd":1,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.99299999999999999,"15min":1.0209999999999999},"min":{"1min":0.54100000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.452,"5min":2.2410000000000001,"15min":18.494},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.88100000000000001,"15min":0.90500000000000003},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.1679999999999999,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":0.76100000000000001}]},{"osd":2,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94199999999999995,"15min":0.96499999999999997},"min":{"1min":0.47499999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.288,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.98599999999999999,"15min":0.94499999999999995},"min":{"1min":0.59499999999999997,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.5309999999999999,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.0549999999999999}]},{"osd":3,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.05,"15min":0.997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.827,"5min":2.278,"15min":2.278},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.002,"15min":0.98499999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.641,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":1.022}]},{"osd":4,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.028,"15min":0.94499999999999995},"min":{"1min":0.66500000000000004,"5min":0.53000000000000003,"15min":0.52000000000000002},"max":{"1min":1.782,"5min":6.008,"15min":6.008},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.1319999999999999,"15min":1.0569999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.42999999999999999},"max":{"1min":1.6100000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.3400000000000001}]},{"osd":5,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1220000000000001,"15min":1.0369999999999999},"min":{"1min":0.71099999999999997,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.6739999999999999,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1890000000000001,"15min":1.125},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.708,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.2909999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.115,"15min":1.071},"min":{"1min":0.86299999999999999,"5min":0.503,"15min":0.503},"max":{"1min":1.5840000000000001,"5min":2.069,"15min":2.069},"last":1.131},{"interface":"front","average":{"1min":1.292,"5min":1.2929999999999999,"15min":1.196},"min":{"1min":0.90600000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8680000000000001,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.22}]}]},{"osd":5,"up_from":43,"seq":184683593895,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312580,"kb_used_data":11060,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432572,"statfs":{"total":95995035648,"available":95674953728,"internally_reserved":0,"allocated":11325440,"data_stored":8207580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.077,"15min":0.95799999999999996},"min":{"1min":0.68799999999999994,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.1870000000000001,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.96399999999999997,"15min":0.91700000000000004},"min":{"1min":0.65700000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.2709999999999999,"5min":5.0519999999999996,"15min":7.3070000000000004},"last":0.91400000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.2689999999999999,"15min":1.1479999999999999},"min":{"1min":0.64900000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3100000000000001,"5min":15.824,"15min":31.548999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.054,"15min":1.0780000000000001},"min":{"1min":0.65900000000000003,"5min":0.54600000000000004,"15min":0.48899999999999999},"max":{"1min":1.2,"5min":5.0839999999999996,"15min":31.847000000000001},"last":0.84399999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":1.272,"15min":1.2370000000000001},"min":{"1min":0.63300000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1719999999999999,"5min":20.536000000000001,"15min":61.347000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0980000000000001,"15min":1.006},"min":{"1min":0.68600000000000005,"5min":0.60299999999999998,"15min":0.53200000000000003},"max":{"1min":1.4350000000000001,"5min":5.2489999999999997,"15min":6.1600000000000001},"last":0.98199999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.0880000000000001,"15min":1.0029999999999999},"min":{"1min":0.68100000000000005,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.2110000000000001,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.216,"15min":1.248},"min":{"1min":0.76800000000000002,"5min":0.66300000000000003,"15min":0.57399999999999995},"max":{"1min":1.5109999999999999,"5min":7.7679999999999998,"15min":53.173000000000002},"last":1.0309999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:00:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.1619999999999999,"15min":0.95299999999999996},"min":{"1min":0.57599999999999996,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.3129999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.141,"15min":1.0169999999999999},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.2390000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":0.94299999999999995}]},{"osd":6,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.395,"15min":1.1519999999999999},"min":{"1min":0.67000000000000004,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":1.3400000000000001,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.052},{"interface":"front","average":{"1min":1.105,"5min":1.4299999999999999,"15min":1.1879999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.47199999999999998},"max":{"1min":1.5760000000000001,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":0.88900000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.532,"15min":1.2130000000000001},"min":{"1min":0.59699999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.3899999999999999,"5min":15.613,"15min":15.613},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.544,"15min":1.2749999999999999},"min":{"1min":0.77300000000000002,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":1.538,"5min":16.965,"15min":16.965},"last":1.012}]}]},{"osd":4,"up_from":35,"seq":150323855539,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312556,"kb_used_data":6884,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432596,"statfs":{"total":95995035648,"available":95674978304,"internally_reserved":0,"allocated":7049216,"data_stored":4014232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.82299999999999995,"15min":0.84299999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.3149999999999999,"5min":1.403,"15min":1.546},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.93500000000000005,"15min":0.89900000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.7070000000000001},"last":0.85899999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:00:23 2024","interfaces":[{"interface":"back","average":{"1min":2.4159999999999999,"5min":1.1679999999999999,"15min":0.97099999999999997},"min":{"1min":0.49299999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":27.716000000000001,"5min":27.716000000000001,"15min":27.716000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":2.3599999999999999,"5min":1.196,"15min":1},"min":{"1min":0.55200000000000005,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":27.93,"5min":27.93,"15min":27.93},"last":0.88500000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:00:23 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.93999999999999995,"15min":0.92800000000000005},"min":{"1min":0.54100000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4770000000000001,"5min":1.5209999999999999,"15min":1.5509999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.95799999999999996,"15min":0.93500000000000005},"min":{"1min":0.58399999999999996,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.3400000000000001,"5min":1.482,"15min":1.7569999999999999},"last":0.92200000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":0.995,"15min":0.95299999999999996},"min":{"1min":0.67800000000000005,"5min":0.53800000000000003,"15min":0.46300000000000002},"max":{"1min":1.429,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":1.071},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97199999999999998,"15min":0.96099999999999997},"min":{"1min":0.70899999999999996,"5min":0.442,"15min":0.442},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.95899999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.96099999999999997,"15min":0.92100000000000004},"min":{"1min":0.67200000000000004,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.095,"5min":1.0669999999999999,"15min":0.99399999999999999},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.5209999999999999,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.022}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0660000000000001,"15min":1.0009999999999999},"min":{"1min":0.60799999999999998,"5min":0.501,"15min":0.501},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.1619999999999999,"15min":1.1160000000000001},"min":{"1min":0.73699999999999999,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.7230000000000001,"5min":1.927,"15min":1.927},"last":1.0960000000000001}]},{"osd":7,"last update":"Tue Sep 10 18:59:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"min":{"1min":0.80100000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5880000000000001,"5min":1.629,"15min":2.008},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.1839999999999999,"15min":1.167},"min":{"1min":1,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.1559999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051774,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308540,"kb_used_data":7020,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436612,"statfs":{"total":95995035648,"available":95679090688,"internally_reserved":0,"allocated":7188480,"data_stored":4014216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.749,"15min":0.82499999999999996},"min":{"1min":0.373,"5min":0.222,"15min":0.222},"max":{"1min":0.91200000000000003,"5min":8.6180000000000003,"15min":26.594000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.74199999999999999,"15min":0.88800000000000001},"min":{"1min":0.48099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.86599999999999999,"5min":5.3479999999999999,"15min":24.204999999999998},"last":0.85299999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.76400000000000001,"15min":0.91000000000000003},"min":{"1min":0.39900000000000002,"5min":0.32100000000000001,"15min":0.26500000000000001},"max":{"1min":0.92500000000000004,"5min":5.3879999999999999,"15min":53.661999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.747,"5min":0.80800000000000005,"15min":0.875},"min":{"1min":0.48399999999999999,"5min":0.36199999999999999,"15min":0.27900000000000003},"max":{"1min":1.2270000000000001,"5min":4.0549999999999997,"15min":15.189},"last":1.0640000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85899999999999999,"15min":0.998},"min":{"1min":0.53000000000000003,"5min":0.39700000000000002,"15min":0.32200000000000001},"max":{"1min":1.0940000000000001,"5min":5.1920000000000002,"15min":26.558},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.80300000000000005,"15min":0.93500000000000005},"min":{"1min":0.55600000000000005,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.98899999999999999,"5min":3.6760000000000002,"15min":25.536999999999999},"last":0.91200000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:00:23 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.145,"15min":1.1399999999999999},"min":{"1min":0.753,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.3540000000000001,"5min":7.3550000000000004,"15min":10.359999999999999},"last":1.232},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.2669999999999999,"15min":1.234},"min":{"1min":0.88400000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.958,"5min":7.4189999999999996,"15min":10.391},"last":1.3}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.2290000000000001,"15min":1.218},"min":{"1min":0.70399999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.8009999999999999,"5min":9.2750000000000004,"15min":17.535},"last":1.163},{"interface":"front","average":{"1min":1.1850000000000001,"5min":1.143,"15min":1.048},"min":{"1min":0.66900000000000004,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":4.6619999999999999,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":1.143}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.204,"15min":1.1990000000000001},"min":{"1min":0.80200000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.9159999999999999,"5min":9.3610000000000007,"15min":10.51},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1599999999999999,"15min":1.093},"min":{"1min":0.71499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.9359999999999999,"5min":10.773999999999999,"15min":10.773999999999999},"last":1.3919999999999999}]},{"osd":7,"last update":"Tue Sep 10 18:59:56 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.1100000000000001,"15min":1.179},"min":{"1min":0.60199999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":4.0229999999999997,"5min":5.3109999999999999,"15min":18.312999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.3919999999999999,"15min":1.3140000000000001},"min":{"1min":0.86399999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":4.5350000000000001,"5min":9.3840000000000003,"15min":10.611000000000001},"last":1.421}]}]},{"osd":2,"up_from":19,"seq":81604378823,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320632,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93424520,"statfs":{"total":95995035648,"available":95666708480,"internally_reserved":0,"allocated":15319040,"data_stored":12208606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.84099999999999997,"15min":0.72099999999999997},"min":{"1min":0.36799999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.2749999999999999,"5min":9.516,"15min":9.516},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.84999999999999998,"15min":0.77900000000000003},"min":{"1min":0.36099999999999999,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.0760000000000001,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.81899999999999995}]},{"osd":1,"last update":"Tue Sep 10 18:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.77700000000000002,"15min":0.71599999999999997},"min":{"1min":0.50800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.0720000000000001,"5min":5.516,"15min":5.516},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.85899999999999999,"15min":0.81200000000000006},"min":{"1min":0.54000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.0820000000000001,"5min":9.0649999999999995,"15min":9.0649999999999995},"last":0.75900000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:00:21 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.86299999999999999,"15min":0.86499999999999999},"min":{"1min":0.54800000000000004,"5min":0.36599999999999999,"15min":0.34799999999999998},"max":{"1min":1.427,"5min":4.3579999999999997,"15min":5.1479999999999997},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.95699999999999996,"15min":0.84799999999999998},"min":{"1min":0.58599999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.3100000000000001,"5min":15.736000000000001,"15min":15.736000000000001},"last":0.94999999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:00:21 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.2370000000000001,"15min":1.0700000000000001},"min":{"1min":0.70599999999999996,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.4690000000000001,"5min":21.571999999999999,"15min":21.571999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.1899999999999999,"15min":1.1559999999999999},"min":{"1min":0.75900000000000001,"5min":0.55100000000000005,"15min":0.46000000000000002},"max":{"1min":1.589,"5min":10.881,"15min":10.881},"last":1.2629999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:00:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.054,"15min":1.0900000000000001},"min":{"1min":0.754,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.8340000000000001,"5min":4.2720000000000002,"15min":5.1959999999999997},"last":1.427},{"interface":"front","average":{"1min":1.056,"5min":1.268,"15min":1.1080000000000001},"min":{"1min":0.69299999999999995,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.405,"5min":21.654,"15min":21.654},"last":1.302}]},{"osd":6,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.1950000000000001,"15min":1.1659999999999999},"min":{"1min":0.85099999999999998,"5min":0.503,"15min":0.503},"max":{"1min":1.7,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.51},{"interface":"front","average":{"1min":1.069,"5min":1.2529999999999999,"15min":1.131},"min":{"1min":0.82799999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.3400000000000001,"5min":15.971,"15min":15.971},"last":1.3280000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:00:07 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.2150000000000001,"15min":1.1830000000000001},"min":{"1min":0.88300000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.298,"5min":15.025,"15min":15.025},"last":1.389},{"interface":"front","average":{"1min":1.123,"5min":1.361,"15min":1.2},"min":{"1min":0.91800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3759999999999999,"5min":14.816000000000001,"15min":14.816000000000001},"last":1.151}]}]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312348,"kb_used_data":10820,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432804,"statfs":{"total":95995035648,"available":95675191296,"internally_reserved":0,"allocated":11079680,"data_stored":8013435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.0649999999999999,"15min":0.79300000000000004},"min":{"1min":0.33900000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.079,"5min":32.677999999999997,"15min":32.677999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.76800000000000002,"5min":1.1839999999999999,"15min":0.872},"min":{"1min":0.44,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":1.177,"5min":32.398000000000003,"15min":32.398000000000003},"last":0.83099999999999996}]},{"osd":2,"last update":"Tue Sep 10 18:59:41 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.181,"15min":0.85699999999999998},"min":{"1min":0.46000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.724,"5min":26.097000000000001,"15min":26.097000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.746,"5min":1.371,"15min":0.98399999999999999},"min":{"1min":0.39400000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.03,"5min":32.363,"15min":32.363},"last":1.2669999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":0.95999999999999996,"15min":1.0029999999999999},"min":{"1min":0.72299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.1349999999999998,"5min":3.7360000000000002,"15min":25.704000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.84899999999999998,"15min":1.0329999999999999},"min":{"1min":0.52200000000000002,"5min":0.34699999999999998,"15min":0.26600000000000001},"max":{"1min":1.298,"5min":6.0099999999999998,"15min":60.999000000000002},"last":1.2410000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:00:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.143,"15min":1.109},"min":{"1min":0.83599999999999997,"5min":0.47399999999999998,"15min":0.40899999999999997},"max":{"1min":2.0760000000000001,"5min":3.5379999999999998,"15min":12.808999999999999},"last":1.498},{"interface":"front","average":{"1min":0.98599999999999999,"5min":1.1060000000000001,"15min":1.2190000000000001},"min":{"1min":0.74299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.9339999999999999,"5min":6.5430000000000001,"15min":60.529000000000003},"last":1.115}]},{"osd":5,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.3089999999999999,"15min":1.1040000000000001},"min":{"1min":0.78000000000000003,"5min":0.40600000000000003,"15min":0.38},"max":{"1min":1.8160000000000001,"5min":26.050999999999998,"15min":26.050999999999998},"last":1.151},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.385,"15min":1.204},"min":{"1min":0.84299999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.2229999999999999,"5min":12.84,"15min":12.84},"last":1.3480000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.615,"15min":1.2689999999999999},"min":{"1min":0.96999999999999997,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.1509999999999998,"5min":25.608000000000001,"15min":25.608000000000001},"last":1.427},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.581,"15min":1.2350000000000001},"min":{"1min":0.76100000000000001,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":1.9890000000000001,"5min":33.643000000000001,"15min":33.643000000000001},"last":1.3029999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:00:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6060000000000001,"15min":1.3260000000000001},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.2989999999999999,"5min":25.565999999999999,"15min":25.565999999999999},"last":1.593},{"interface":"front","average":{"1min":1.224,"5min":1.6699999999999999,"15min":1.3089999999999999},"min":{"1min":0.88500000000000001,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":2.0600000000000001,"5min":33.673999999999999,"15min":33.673999999999999},"last":1.1930000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309012,"kb_used_data":7500,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436140,"statfs":{"total":95995035648,"available":95678607360,"internally_reserved":0,"allocated":7680000,"data_stored":4603331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.591,"15min":0.96899999999999997},"min":{"1min":0.32400000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.877,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.7270000000000001,"15min":1.077},"min":{"1min":0.48699999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.1779999999999999,"5min":86.009,"15min":86.009},"last":0.72199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79900000000000004,"15min":1.0309999999999999},"min":{"1min":0.41399999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.1659999999999999,"5min":3.427,"15min":36.682000000000002},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.83499999999999996,"15min":0.92800000000000005},"min":{"1min":0.442,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":24.614000000000001},"last":0.84199999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.91300000000000003,"15min":0.92100000000000004},"min":{"1min":0.49299999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.4749999999999996,"5min":4.4749999999999996,"15min":21.620999999999999},"last":1.288},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86399999999999999,"15min":1.0429999999999999},"min":{"1min":0.45700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.2210000000000001,"5min":2.012,"15min":45.765000000000001},"last":1.0580000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:00:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.1200000000000001,"15min":1.0489999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.5},"max":{"1min":2.5190000000000001,"5min":5.5369999999999999,"15min":6.3630000000000004},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.1539999999999999,"15min":1.105},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51100000000000001},"max":{"1min":1.6499999999999999,"5min":4.2389999999999999,"15min":4.2389999999999999},"last":1.1240000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:00:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.105,"15min":1.1180000000000001},"min":{"1min":0.71799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":1.585,"5min":2.7989999999999999,"15min":10.048},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0600000000000001,"15min":1.0960000000000001},"min":{"1min":0.71599999999999997,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.496,"5min":1.635,"15min":3.8959999999999999},"last":1.216}]},{"osd":6,"last update":"Tue Sep 10 19:00:04 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.1619999999999999,"15min":1.135},"min":{"1min":0.755,"5min":0.69099999999999995,"15min":0.56899999999999995},"max":{"1min":1.792,"5min":3.9550000000000001,"15min":3.9550000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1599999999999999,"5min":1.2669999999999999,"15min":1.1970000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.804,"5min":11.894,"15min":11.894},"last":1.262}]},{"osd":7,"last update":"Tue Sep 10 18:59:53 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.149,"15min":1.167},"min":{"1min":0.80600000000000005,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.5549999999999999,"5min":2.266,"15min":6.2949999999999999},"last":1.155},{"interface":"front","average":{"1min":1.25,"5min":1.3660000000000001,"15min":1.29},"min":{"1min":0.78400000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.0129999999999999,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.321}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8194945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4468736,"data_stored":4194961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8445952,"data_stored":8194896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":215,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:00:39.379 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-10T19:00:39.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.415 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:39.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:39.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[18629]: audit 2024-09-10T19:00:37.993718+0000 mgr.y (mgr.24526) 457 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[18629]: cluster 2024-09-10T19:00:38.111738+0000 mgr.y (mgr.24526) 458 : cluster [DBG] pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[18629]: cluster 2024-09-10T19:00:39.148501+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[32381]: audit 2024-09-10T19:00:37.993718+0000 mgr.y (mgr.24526) 457 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[32381]: cluster 2024-09-10T19:00:38.111738+0000 mgr.y (mgr.24526) 458 : cluster [DBG] pgmap v680: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:39.529 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:39 smithi083 bash[32381]: cluster 2024-09-10T19:00:39.148501+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:00:39.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:39.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:39.680 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:39.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:39.818 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:39.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:39.954 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:39.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:40.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.084 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:40.187 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.219 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:40.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:40.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:40.494 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:40.495 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:40 smithi165 bash[22591]: audit 2024-09-10T19:00:38.544248+0000 mgr.y (mgr.24526) 459 : audit [DBG] from='client.25353 v1:172.21.15.83:0/2961613143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:40.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:40 smithi083 bash[18629]: audit 2024-09-10T19:00:38.544248+0000 mgr.y (mgr.24526) 459 : audit [DBG] from='client.25353 v1:172.21.15.83:0/2961613143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:40.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:40 smithi083 bash[32381]: audit 2024-09-10T19:00:38.544248+0000 mgr.y (mgr.24526) 459 : audit [DBG] from='client.25353 v1:172.21.15.83:0/2961613143' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:00:40.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:40.630 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:40.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:40.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:40.771 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:40.862 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:40.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:40.907 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:40.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:41.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:41.047 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:41.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: cluster 2024-09-10T19:00:39.739987+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: cluster 2024-09-10T19:00:39.740350+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: cluster 2024-09-10T19:00:40.112587+0000 mgr.y (mgr.24526) 460 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: cluster 2024-09-10T19:00:40.233551+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: audit 2024-09-10T19:00:41.215764+0000 mon.b (mon.1) 350 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: audit 2024-09-10T19:00:41.217711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: audit 2024-09-10T19:00:41.219406+0000 mon.a (mon.0) 2628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[32381]: audit 2024-09-10T19:00:41.221231+0000 mon.a (mon.0) 2629 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.294 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:41.327 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:41.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:41.468 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: cluster 2024-09-10T19:00:39.739987+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: cluster 2024-09-10T19:00:39.740350+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: cluster 2024-09-10T19:00:40.112587+0000 mgr.y (mgr.24526) 460 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: cluster 2024-09-10T19:00:40.233551+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: audit 2024-09-10T19:00:41.215764+0000 mon.b (mon.1) 350 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: audit 2024-09-10T19:00:41.217711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: audit 2024-09-10T19:00:41.219406+0000 mon.a (mon.0) 2628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:41 smithi083 bash[18629]: audit 2024-09-10T19:00:41.221231+0000 mon.a (mon.0) 2629 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.585 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:41.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: cluster 2024-09-10T19:00:39.739987+0000 osd.2 (osd.2) 197 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:41.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: cluster 2024-09-10T19:00:39.740350+0000 osd.2 (osd.2) 198 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: cluster 2024-09-10T19:00:40.112587+0000 mgr.y (mgr.24526) 460 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: cluster 2024-09-10T19:00:40.233551+0000 mon.a (mon.0) 2627 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: audit 2024-09-10T19:00:41.215764+0000 mon.b (mon.1) 350 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: audit 2024-09-10T19:00:41.217711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: audit 2024-09-10T19:00:41.219406+0000 mon.a (mon.0) 2628 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:41.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:41 smithi165 bash[22591]: audit 2024-09-10T19:00:41.221231+0000 mon.a (mon.0) 2629 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-63"}]: dispatch 2024-09-10T19:00:41.606 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:41.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.743 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:41.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:41.880 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:41.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:41.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:41.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:42.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:42.152 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:42.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.279 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:42.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:42.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:42.552 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:42.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.688 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:42.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:42.798 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:42] "GET /metrics HTTP/1.1" 200 228896 "" "Prometheus/2.43.0" 2024-09-10T19:00:42.827 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:42.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:42.965 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:42.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:43.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:43.101 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:43.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:43.239 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:43.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:43.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:43 smithi165 bash[22591]: cluster 2024-09-10T19:00:42.114850+0000 mgr.y (mgr.24526) 461 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:43.353 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:43 smithi165 bash[22591]: cluster 2024-09-10T19:00:42.250502+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:00:43.368 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:43.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:43.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:43 smithi083 bash[32381]: cluster 2024-09-10T19:00:42.114850+0000 mgr.y (mgr.24526) 461 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:43.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:43 smithi083 bash[32381]: cluster 2024-09-10T19:00:42.250502+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:00:43.486 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:43 smithi083 bash[18629]: cluster 2024-09-10T19:00:42.114850+0000 mgr.y (mgr.24526) 461 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:00:43.486 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:43 smithi083 bash[18629]: cluster 2024-09-10T19:00:42.250502+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:00:43.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:43.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:43.644 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:43.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:43.782 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:43.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:43.906 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:43.907 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:43.919 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:43.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:44.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:44.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:44.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.186 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:44.289 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:44.319 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:44.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: cluster 2024-09-10T19:00:43.252323+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:43.254266+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:43.265236+0000 mon.a (mon.0) 2632 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:44.152056+0000 mon.a (mon.0) 2633 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: cluster 2024-09-10T19:00:44.152274+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:44.152961+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:44.158302+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:44 smithi165 bash[22591]: audit 2024-09-10T19:00:44.161845+0000 mon.a (mon.0) 2635 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.454 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: cluster 2024-09-10T19:00:43.252323+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:43.254266+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:43.265236+0000 mon.a (mon.0) 2632 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:44.152056+0000 mon.a (mon.0) 2633 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: cluster 2024-09-10T19:00:44.152274+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:44.152961+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:44.158302+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[18629]: audit 2024-09-10T19:00:44.161845+0000 mon.a (mon.0) 2635 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: cluster 2024-09-10T19:00:43.252323+0000 mon.a (mon.0) 2631 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:43.254266+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:43.265236+0000 mon.a (mon.0) 2632 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:44.152056+0000 mon.a (mon.0) 2633 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: cluster 2024-09-10T19:00:44.152274+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:44.152961+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:44.158302+0000 mon.b (mon.1) 354 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:44 smithi083 bash[32381]: audit 2024-09-10T19:00:44.161845+0000 mon.a (mon.0) 2635 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:00:44.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:44.695 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:44.730 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:44.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:44.867 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:44.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:44.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:45.004 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:45.108 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:45.145 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:45.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:45.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:45.284 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:45.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: cluster 2024-09-10T19:00:44.115814+0000 mgr.y (mgr.24526) 462 : cluster [DBG] pgmap v687: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: cluster 2024-09-10T19:00:44.259267+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: audit 2024-09-10T19:00:44.572838+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: audit 2024-09-10T19:00:44.575799+0000 mon.c (mon.2) 624 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: audit 2024-09-10T19:00:45.153944+0000 mon.a (mon.0) 2638 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[18629]: cluster 2024-09-10T19:00:45.154103+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: cluster 2024-09-10T19:00:44.115814+0000 mgr.y (mgr.24526) 462 : cluster [DBG] pgmap v687: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: cluster 2024-09-10T19:00:44.259267+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: audit 2024-09-10T19:00:44.572838+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: audit 2024-09-10T19:00:44.575799+0000 mon.c (mon.2) 624 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: audit 2024-09-10T19:00:45.153944+0000 mon.a (mon.0) 2638 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:45 smithi083 bash[32381]: cluster 2024-09-10T19:00:45.154103+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:00:45.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.553 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: cluster 2024-09-10T19:00:44.115814+0000 mgr.y (mgr.24526) 462 : cluster [DBG] pgmap v687: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: cluster 2024-09-10T19:00:44.259267+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: audit 2024-09-10T19:00:44.572838+0000 mon.a (mon.0) 2637 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: audit 2024-09-10T19:00:44.575799+0000 mon.c (mon.2) 624 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: audit 2024-09-10T19:00:45.153944+0000 mon.a (mon.0) 2638 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:00:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:45 smithi165 bash[22591]: cluster 2024-09-10T19:00:45.154103+0000 mon.a (mon.0) 2639 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:00:45.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:45.683 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:45.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:45.819 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:45.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.951 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:45.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:46.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.089 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:46.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:46.220 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:46.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:46.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.489 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:46.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:46.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:46.762 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:46.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.901 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:46.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:47.005 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:47.031 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:47.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:47.176 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:47.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:47.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:47 smithi083 bash[18629]: cluster 2024-09-10T19:00:46.118067+0000 mgr.y (mgr.24526) 463 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:47.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:47 smithi083 bash[18629]: cluster 2024-09-10T19:00:46.157247+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:00:47.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:47 smithi083 bash[32381]: cluster 2024-09-10T19:00:46.118067+0000 mgr.y (mgr.24526) 463 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:47.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:47 smithi083 bash[32381]: cluster 2024-09-10T19:00:46.157247+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:00:47.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:47.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:47.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:47.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:47 smithi165 bash[22591]: cluster 2024-09-10T19:00:46.118067+0000 mgr.y (mgr.24526) 463 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:47.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:47 smithi165 bash[22591]: cluster 2024-09-10T19:00:46.157247+0000 mon.a (mon.0) 2640 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:00:47.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:47.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:47.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:47.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:47.723 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:47.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:47.857 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:47.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:47.992 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:47.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:48.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:48.109 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:47 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:48.126 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:48.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:48.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[18629]: cluster 2024-09-10T19:00:46.302508+0000 osd.4 (osd.4) 288 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:48.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[18629]: cluster 2024-09-10T19:00:46.302817+0000 osd.4 (osd.4) 289 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:48.243 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[18629]: cluster 2024-09-10T19:00:47.160172+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:00:48.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[32381]: cluster 2024-09-10T19:00:46.302508+0000 osd.4 (osd.4) 288 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:48.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[32381]: cluster 2024-09-10T19:00:46.302817+0000 osd.4 (osd.4) 289 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:48.243 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:48 smithi083 bash[32381]: cluster 2024-09-10T19:00:47.160172+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:00:48.259 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:48 smithi165 bash[22591]: cluster 2024-09-10T19:00:46.302508+0000 osd.4 (osd.4) 288 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:48 smithi165 bash[22591]: cluster 2024-09-10T19:00:46.302817+0000 osd.4 (osd.4) 289 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:48 smithi165 bash[22591]: cluster 2024-09-10T19:00:47.160172+0000 mon.a (mon.0) 2641 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:00:48.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:48.394 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:48.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:48.528 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:48.631 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:48.667 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:48.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:48.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:48.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:48.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:48.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:49.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:49.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:49.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:49.215 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:49.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:49.355 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:49.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[32381]: audit 2024-09-10T19:00:48.001759+0000 mgr.y (mgr.24526) 464 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[32381]: cluster 2024-09-10T19:00:48.118899+0000 mgr.y (mgr.24526) 465 : cluster [DBG] pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[32381]: cluster 2024-09-10T19:00:48.175849+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[32381]: cluster 2024-09-10T19:00:48.176367+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[32381]: cluster 2024-09-10T19:00:48.178689+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[18629]: audit 2024-09-10T19:00:48.001759+0000 mgr.y (mgr.24526) 464 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:49.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[18629]: cluster 2024-09-10T19:00:48.118899+0000 mgr.y (mgr.24526) 465 : cluster [DBG] pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-10T19:00:49.471 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[18629]: cluster 2024-09-10T19:00:48.175849+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:49.471 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[18629]: cluster 2024-09-10T19:00:48.176367+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:49.471 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:49 smithi083 bash[18629]: cluster 2024-09-10T19:00:48.178689+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:00:49.489 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:49.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:49.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:49 smithi165 bash[22591]: audit 2024-09-10T19:00:48.001759+0000 mgr.y (mgr.24526) 464 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:49.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:49 smithi165 bash[22591]: cluster 2024-09-10T19:00:48.118899+0000 mgr.y (mgr.24526) 465 : cluster [DBG] pgmap v693: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-09-10T19:00:49.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:49 smithi165 bash[22591]: cluster 2024-09-10T19:00:48.175849+0000 osd.5 (osd.5) 249 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:49.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:49 smithi165 bash[22591]: cluster 2024-09-10T19:00:48.176367+0000 osd.5 (osd.5) 250 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:49.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:49 smithi165 bash[22591]: cluster 2024-09-10T19:00:48.178689+0000 mon.a (mon.0) 2642 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:00:49.627 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:49.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:49.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:49.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:49.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:49.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.030 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:50.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:50.160 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:50.225 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:00:50.225 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:00:50.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:50.278 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:50 smithi083 bash[18629]: cluster 2024-09-10T19:00:49.217158+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:00:50.278 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:50 smithi083 bash[32381]: cluster 2024-09-10T19:00:49.217158+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:00:50.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:50.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:50 smithi165 bash[22591]: cluster 2024-09-10T19:00:49.217158+0000 mon.a (mon.0) 2643 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:00:50.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.428 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:50.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:50.565 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:50.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:50.704 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:50.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:50.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:50.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:50.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:51.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:51.116 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:51.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:51.251 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:51.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:51.388 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:51.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:51.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:51 smithi165 bash[22591]: cluster 2024-09-10T19:00:50.119646+0000 mgr.y (mgr.24526) 466 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:51.505 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:00:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:00:51.523 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:51.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:51 smithi083 bash[18629]: cluster 2024-09-10T19:00:50.119646+0000 mgr.y (mgr.24526) 466 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:51.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:51 smithi083 bash[32381]: cluster 2024-09-10T19:00:50.119646+0000 mgr.y (mgr.24526) 466 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:00:51.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:51.660 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:51.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:51.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:51.899 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:51.931 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:51.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:52.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:52.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:52.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:52.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:52.297 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:52.325 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:52 smithi165 bash[22591]: audit 2024-09-10T19:00:51.901191+0000 mon.c (mon.2) 625 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:52 smithi165 bash[22591]: audit 2024-09-10T19:00:52.212345+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:52.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:52.458 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:52.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:52 smithi083 bash[18629]: audit 2024-09-10T19:00:51.901191+0000 mon.c (mon.2) 625 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:52 smithi083 bash[18629]: audit 2024-09-10T19:00:52.212345+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:52 smithi083 bash[32381]: audit 2024-09-10T19:00:51.901191+0000 mon.c (mon.2) 625 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:52 smithi083 bash[32381]: audit 2024-09-10T19:00:52.212345+0000 mon.a (mon.0) 2644 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:52.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:52.586 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:52.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:52.714 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:52.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:52.848 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:52.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:52.986 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:52.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:00:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:00:52] "GET /metrics HTTP/1.1" 200 228862 "" "Prometheus/2.43.0" 2024-09-10T19:00:53.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:53.124 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:53.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:53.258 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:53.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:53.397 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:53.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: cluster 2024-09-10T19:00:52.121746+0000 mgr.y (mgr.24526) 467 : cluster [DBG] pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 686 B/s wr, 4 op/s 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.223661+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.296553+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.297094+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.298395+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.514 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.298906+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.300695+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.301206+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.302962+0000 mon.c (mon.2) 629 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.303478+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: cephadm 2024-09-10T19:00:52.304794+0000 mgr.y (mgr.24526) 468 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3293M 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.312175+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.679603+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.680174+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.682057+0000 mon.c (mon.2) 631 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.682582+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.684456+0000 mon.c (mon.2) 632 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.515 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.684918+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.686751+0000 mon.c (mon.2) 633 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.687571+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.696485+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.699655+0000 mon.c (mon.2) 634 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.700691+0000 mon.c (mon.2) 635 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[32381]: audit 2024-09-10T19:00:52.709359+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: cluster 2024-09-10T19:00:52.121746+0000 mgr.y (mgr.24526) 467 : cluster [DBG] pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 686 B/s wr, 4 op/s 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.223661+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.296553+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.516 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.297094+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.298395+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.298906+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.300695+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.301206+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.302962+0000 mon.c (mon.2) 629 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.303478+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: cephadm 2024-09-10T19:00:52.304794+0000 mgr.y (mgr.24526) 468 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3293M 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.312175+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.679603+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.680174+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.682057+0000 mon.c (mon.2) 631 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.682582+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.684456+0000 mon.c (mon.2) 632 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.684918+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.686751+0000 mon.c (mon.2) 633 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.687571+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.696485+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.699655+0000 mon.c (mon.2) 634 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:53.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.700691+0000 mon.c (mon.2) 635 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:53.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:53 smithi083 bash[18629]: audit 2024-09-10T19:00:52.709359+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.535 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: cluster 2024-09-10T19:00:52.121746+0000 mgr.y (mgr.24526) 467 : cluster [DBG] pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 686 B/s wr, 4 op/s 2024-09-10T19:00:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.223661+0000 mon.a (mon.0) 2645 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.296553+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.297094+0000 mon.a (mon.0) 2646 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.298395+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.298906+0000 mon.a (mon.0) 2647 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.300695+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.301206+0000 mon.a (mon.0) 2648 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.302962+0000 mon.c (mon.2) 629 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.303478+0000 mon.a (mon.0) 2649 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: cephadm 2024-09-10T19:00:52.304794+0000 mgr.y (mgr.24526) 468 : cephadm [INF] Adjusting osd_memory_target on smithi083 to 3293M 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.312175+0000 mon.a (mon.0) 2650 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.679603+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.680174+0000 mon.a (mon.0) 2651 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.682057+0000 mon.c (mon.2) 631 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.682582+0000 mon.a (mon.0) 2652 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.684456+0000 mon.c (mon.2) 632 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.684918+0000 mon.a (mon.0) 2653 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.686751+0000 mon.c (mon.2) 633 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.687571+0000 mon.a (mon.0) 2654 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.696485+0000 mon.a (mon.0) 2655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.699655+0000 mon.c (mon.2) 634 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.700691+0000 mon.c (mon.2) 635 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:53.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:53 smithi165 bash[22591]: audit 2024-09-10T19:00:52.709359+0000 mon.a (mon.0) 2656 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:00:53.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:53.668 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:53.691 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:00:53.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- 172.21.15.83:0/760704350 >> v1:172.21.15.83:6790/0 conn(0x7f562c108f70 legacy=0x7f562c10b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- 172.21.15.83:0/760704350 shutdown_connections 2024-09-10T19:00:53.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- 172.21.15.83:0/760704350 >> 172.21.15.83:0/760704350 conn(0x7f562c074a20 msgr2=0x7f562c0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:53.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- 172.21.15.83:0/760704350 shutdown_connections 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- 172.21.15.83:0/760704350 wait complete. 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 librados: starting msgr at 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 librados: starting objecter 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 Processor -- start 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 -- start start 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 librados: setting wanted keys 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.762+0000 7f5630ca9640 1 librados: calling monclient init 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f562c101e10 con 0x7f562c1053b0 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f562c101fc0 con 0x7f562c102680 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f562c1207d0 con 0x7f562c108f70 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f562ad9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f562c108f70 0x7f562c11d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58832/0 (socket says 172.21.15.83:58832) 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 learned_addr learned my addr 172.21.15.83:0/3648331443 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:53.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter ms_handle_connect 0x7f562c108f70 2024-09-10T19:00:53.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:53.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter ms_handle_connect 0x7f562c1053b0 2024-09-10T19:00:53.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:53.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter ms_handle_connect 0x7f562c102680 2024-09-10T19:00:53.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:53.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242647391 0 0) 0x7f562c101e10 con 0x7f562c1053b0 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56140034e0 con 0x7f562c1053b0 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455393836 0 0) 0x7f562c1207d0 con 0x7f562c108f70 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f562c101e10 con 0x7f562c108f70 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003666390 0 0) 0x7f562c101fc0 con 0x7f562c102680 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f562c1207d0 con 0x7f562c102680 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031398675 0 0) 0x7f562c1207d0 con 0x7f562c102680 2024-09-10T19:00:53.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562c101fc0 con 0x7f562c102680 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f56200018c0 con 0x7f562c102680 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295708393 0 0) 0x7f562c101e10 con 0x7f562c108f70 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562c1207d0 con 0x7f562c108f70 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5624002f10 con 0x7f562c108f70 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688747978 0 0) 0x7f56140034e0 con 0x7f562c1053b0 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562c101e10 con 0x7f562c1053b0 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f561c0018c0 con 0x7f562c1053b0 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3079320865 0 0) 0x7f562c101fc0 con 0x7f562c102680 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6790/0 conn(0x7f562c108f70 legacy=0x7f562c11d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6789/0 conn(0x7f562c1053b0 legacy=0x7f562c119960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f562c1219b0 con 0x7f562c102680 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1209c0 con 0x7f562c102680 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 10 client.49939.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c120e10 con 0x7f562c102680 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 librados: init done 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f56200021f0 con 0x7f562c102680 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f56200030d0 con 0x7f562c102680 2024-09-10T19:00:53.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.766+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f560c004ff0 con 0x7f562c102680 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5620001d90 con 0x7f562c102680 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(465..465 src has 69..465) v4 ==== 8032+0+0 (unknown 55565318 0 0) 0x7f5620087a50 con 0x7f562c102680 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 osd_map(465..465 src has 69..465) v4 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 3 client.49939.objecter handle_osd_map got epochs [465,465] > 0 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 3 client.49939.objecter handle_osd_map decoding full epoch 465 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 20 client.49939.objecter dump_active .. 0 homeless 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v465) v1 ==== 54+0+16 (unknown 3237333402 0 2420000192) 0x7f5620088060 con 0x7f562c102680 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f56140717d0 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 10 client.49939.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 1 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 1 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 0 1 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 4 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 10 client.49939.objecter _send_command 1 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c005290 con 0x7f560c004610 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c004610 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c005290 con 0x7f560c004610 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 10 client.49939.objecter _finish_command 1 = system:0 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 4 2024-09-10T19:00:53.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.770+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 0 1 2024-09-10T19:00:53.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 10 client.49939.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 2 2024-09-10T19:00:53.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 2 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c005810 osd=0 3 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 0 2 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 4 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 10 client.49939.objecter _send_command 2 2024-09-10T19:00:53.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c004c60 con 0x7f560c004610 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c004c60 con 0x7f560c004610 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 10 client.49939.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 4 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 0 2 2024-09-10T19:00:53.796 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 10 client.49939.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 3 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 3 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 1 3 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 4 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 10 client.49939.objecter _send_command 3 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c003b10 con 0x7f560c0035e0 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.790+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c0035e0 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.794+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c003b10 con 0x7f560c0035e0 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.794+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:00:53.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.794+0000 7f561affd640 10 client.49939.objecter _finish_command 3 = system:0 2024-09-10T19:00:53.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.794+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 4 2024-09-10T19:00:53.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.794+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 1 3 2024-09-10T19:00:53.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:53.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:53.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 10 client.49939.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 4 2024-09-10T19:00:53.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 4 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c004e10 osd=1 3 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 1 4 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 4 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 10 client.49939.objecter _send_command 4 2024-09-10T19:00:53.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c003060 con 0x7f560c0035e0 2024-09-10T19:00:53.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c003060 con 0x7f560c0035e0 2024-09-10T19:00:53.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f561affd640 10 client.49939.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 4 2024-09-10T19:00:53.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 1 4 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 10 client.49939.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 5 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 5 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 2 5 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 4 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 10 client.49939.objecter _send_command 5 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.814+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c0022b0 con 0x7f560c00d350 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c00d350 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c0022b0 con 0x7f560c00d350 2024-09-10T19:00:53.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:00:53.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 10 client.49939.objecter _finish_command 5 = system:0 2024-09-10T19:00:53.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 4 2024-09-10T19:00:53.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.818+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 2 5 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 10 client.49939.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 6 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 6 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c001b60 osd=2 3 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 2 6 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 4 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 10 client.49939.objecter _send_command 6 2024-09-10T19:00:53.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c001650 con 0x7f560c00d350 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c001650 con 0x7f560c00d350 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f561affd640 10 client.49939.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 4 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.846+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 2 6 2024-09-10T19:00:53.854 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 10 client.49939.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 7 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 7 2024-09-10T19:00:53.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 3 7 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 4 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 10 client.49939.objecter _send_command 7 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c00d140 con 0x7f560c011fd0 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c011fd0 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c00d140 con 0x7f560c011fd0 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:00:53.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 10 client.49939.objecter _finish_command 7 = system:0 2024-09-10T19:00:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 4 2024-09-10T19:00:53.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.850+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 3 7 2024-09-10T19:00:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 10 client.49939.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 8 2024-09-10T19:00:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 8 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c990 osd=3 3 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 3 8 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 4 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 10 client.49939.objecter _send_command 8 2024-09-10T19:00:53.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c00c410 con 0x7f560c011fd0 2024-09-10T19:00:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c00c410 con 0x7f560c011fd0 2024-09-10T19:00:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f561affd640 10 client.49939.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 4 2024-09-10T19:00:53.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 3 8 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 10 client.49939.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 9 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 9 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 4 9 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 4 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 10 client.49939.objecter _send_command 9 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.870+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c00c370 con 0x7f560c011b20 2024-09-10T19:00:53.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c011b20 2024-09-10T19:00:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c00c370 con 0x7f560c011b20 2024-09-10T19:00:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:00:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 10 client.49939.objecter _finish_command 9 = system:0 2024-09-10T19:00:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 4 2024-09-10T19:00:53.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.874+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 4 9 2024-09-10T19:00:53.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 10 client.49939.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 10 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 10 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c1d0 osd=4 3 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 4 10 2024-09-10T19:00:53.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 4 2024-09-10T19:00:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 10 client.49939.objecter _send_command 10 2024-09-10T19:00:53.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.898+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c00c370 con 0x7f560c011b20 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c00c370 con 0x7f560c011b20 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 10 client.49939.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 4 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 4 10 2024-09-10T19:00:53.908 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 10 client.49939.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 11 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 11 2024-09-10T19:00:53.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 5 11 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 4 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 10 client.49939.objecter _send_command 11 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c017ea0 con 0x7f560c0178b0 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.902+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c0178b0 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.906+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c017ea0 con 0x7f560c0178b0 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.906+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:00:53.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.906+0000 7f561affd640 10 client.49939.objecter _finish_command 11 = system:0 2024-09-10T19:00:53.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.906+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 4 2024-09-10T19:00:53.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.906+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 5 11 2024-09-10T19:00:53.932 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 10 client.49939.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 12 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 12 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c00c370 osd=5 3 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 5 12 2024-09-10T19:00:53.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 4 2024-09-10T19:00:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 10 client.49939.objecter _send_command 12 2024-09-10T19:00:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.926+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c017e60 con 0x7f560c0178b0 2024-09-10T19:00:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:53.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c017e60 con 0x7f560c0178b0 2024-09-10T19:00:53.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f561affd640 10 client.49939.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 4 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 5 12 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 10 client.49939.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 13 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 13 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 6 13 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 4 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 10 client.49939.objecter _send_command 13 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.930+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c022ad0 con 0x7f560c01efd0 2024-09-10T19:00:53.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c01efd0 2024-09-10T19:00:53.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c022ad0 con 0x7f560c01efd0 2024-09-10T19:00:53.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:00:53.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 10 client.49939.objecter _finish_command 13 = system:0 2024-09-10T19:00:53.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 4 2024-09-10T19:00:53.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.934+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 6 13 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 10 client.49939.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 14 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 14 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c017e60 osd=6 3 2024-09-10T19:00:53.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 6 14 2024-09-10T19:00:53.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 4 2024-09-10T19:00:53.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 10 client.49939.objecter _send_command 14 2024-09-10T19:00:53.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.954+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c022ad0 con 0x7f560c01efd0 2024-09-10T19:00:53.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c022ad0 con 0x7f560c01efd0 2024-09-10T19:00:53.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 10 client.49939.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 4 2024-09-10T19:00:53.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 6 14 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 10 client.49939.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 15 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 15 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 7 15 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 4 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 10 client.49939.objecter _send_command 15 2024-09-10T19:00:53.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560c026e40 con 0x7f560c023150 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.958+0000 7f561affd640 10 client.49939.objecter ms_handle_connect 0x7f560c023150 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.962+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f560c026e40 con 0x7f560c023150 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.962+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.962+0000 7f561affd640 10 client.49939.objecter _finish_command 15 = system:0 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.962+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 4 2024-09-10T19:00:53.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.962+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 7 15 2024-09-10T19:00:53.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 10 client.49939.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:00:53.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign -1 16 2024-09-10T19:00:53.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 20 client.49939.objecter _get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_remove -1 16 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 20 client.49939.objecter get_session s=0x7f560c022ad0 osd=7 3 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 15 client.49939.objecter _session_command_op_assign 7 16 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 4 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 10 client.49939.objecter _send_command 16 2024-09-10T19:00:53.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.982+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f560c0286e0 con 0x7f560c023150 2024-09-10T19:00:53.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f561affd640 1 -- 172.21.15.83:0/3648331443 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f560c0286e0 con 0x7f560c023150 2024-09-10T19:00:53.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f561affd640 10 client.49939.objecter ms_dispatch 0x7f562c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:00:53.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f561affd640 10 client.49939.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:00:53.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f561affd640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 4 2024-09-10T19:00:53.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f561affd640 15 client.49939.objecter _session_command_op_remove 7 16 2024-09-10T19:00:53.992 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:00:53.992 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:00:53.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 librados: watch_flush enter 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 librados: watch_flush exit 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.0 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6801/2535527483 conn(0x7f560c004610 legacy=0x7f560c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c005810 osd=0 1 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.1 2024-09-10T19:00:53.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6805/2309624637 conn(0x7f560c0035e0 legacy=0x7f560c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c004e10 osd=1 1 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.2 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6809/386094220 conn(0x7f560c00d350 legacy=0x7f560c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c001b60 osd=2 1 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.3 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6813/1279688021 conn(0x7f560c011fd0 legacy=0x7f560c0143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c990 osd=3 1 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.4 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.165:6800/3225515265 conn(0x7f560c011b20 legacy=0x7f560c016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c1d0 osd=4 1 2024-09-10T19:00:53.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.5 2024-09-10T19:00:53.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.165:6804/571173777 conn(0x7f560c0178b0 legacy=0x7f560c01d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c00c370 osd=5 1 2024-09-10T19:00:53.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.6 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.165:6808/1317922936 conn(0x7f560c01efd0 legacy=0x7f560c0213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c017e60 osd=6 1 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter close_session for osd.7 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.165:6812/1610151650 conn(0x7f560c023150 legacy=0x7f560c025590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter put_session s=0x7f560c022ad0 osd=7 1 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 20 client.49939.objecter shutdown clearing up homeless session... 2024-09-10T19:00:53.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 10 client.49939.objecter successfully canceled tick 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 reap_dead start 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 reap_dead start 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.83:6800/3071793735 conn(0x7f56140717d0 legacy=0x7f5614073c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> v1:172.21.15.165:6789/0 conn(0x7f562c102680 legacy=0x7f562c1015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 reap_dead start 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 reap_dead start 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f562ad9d640 1 -- 172.21.15.83:0/3648331443 reap_dead start 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 shutdown_connections 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 >> 172.21.15.83:0/3648331443 conn(0x7f562c074a20 msgr2=0x7f562c10c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 shutdown_connections 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 -- 172.21.15.83:0/3648331443 wait complete. 2024-09-10T19:00:53.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:53.986+0000 7f5630ca9640 1 librados: shutdown 2024-09-10T19:00:54.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:54.079 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:54.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:54.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:54.209 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:54.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:54.346 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:54.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:54.483 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:54.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[18629]: cephadm 2024-09-10T19:00:52.689462+0000 mgr.y (mgr.24526) 469 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3293M 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[18629]: audit 2024-09-10T19:00:53.771487+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.83:0/3648331443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[18629]: cluster 2024-09-10T19:00:54.146270+0000 mon.a (mon.0) 2657 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[18629]: cluster 2024-09-10T19:00:54.154691+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[32381]: cephadm 2024-09-10T19:00:52.689462+0000 mgr.y (mgr.24526) 469 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3293M 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[32381]: audit 2024-09-10T19:00:53.771487+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.83:0/3648331443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[32381]: cluster 2024-09-10T19:00:54.146270+0000 mon.a (mon.0) 2657 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:54.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:54 smithi083 bash[32381]: cluster 2024-09-10T19:00:54.154691+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:00:54.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:54 smithi165 bash[22591]: cephadm 2024-09-10T19:00:52.689462+0000 mgr.y (mgr.24526) 469 : cephadm [INF] Adjusting osd_memory_target on smithi165 to 3293M 2024-09-10T19:00:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:54 smithi165 bash[22591]: audit 2024-09-10T19:00:53.771487+0000 mon.b (mon.1) 355 : audit [DBG] from='client.? v1:172.21.15.83:0/3648331443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:00:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:54 smithi165 bash[22591]: cluster 2024-09-10T19:00:54.146270+0000 mon.a (mon.0) 2657 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:00:54.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:54 smithi165 bash[22591]: cluster 2024-09-10T19:00:54.154691+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:00:54.620 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:54.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:54.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:54.894 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:54.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:54.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.032 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:55.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:55.169 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:55.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.287 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:55 smithi165 bash[22591]: cluster 2024-09-10T19:00:54.122709+0000 mgr.y (mgr.24526) 470 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 587 B/s wr, 4 op/s 2024-09-10T19:00:55.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:55 smithi083 bash[18629]: cluster 2024-09-10T19:00:54.122709+0000 mgr.y (mgr.24526) 470 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 587 B/s wr, 4 op/s 2024-09-10T19:00:55.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:55 smithi083 bash[32381]: cluster 2024-09-10T19:00:54.122709+0000 mgr.y (mgr.24526) 470 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 587 B/s wr, 4 op/s 2024-09-10T19:00:55.309 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:55.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:55.446 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:55.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.581 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:55.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:00:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:00:55.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:55.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:55.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:55.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:55.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:56.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:56.122 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:56.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:56.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:56.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:56.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:56.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:56.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:56.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:56.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:56.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:56.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:56.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:56.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- 172.21.15.83:0/4001594622 >> v1:172.21.15.83:6790/0 conn(0x7f8314108ef0 legacy=0x7f831410b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- 172.21.15.83:0/4001594622 shutdown_connections 2024-09-10T19:00:56.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- 172.21.15.83:0/4001594622 >> 172.21.15.83:0/4001594622 conn(0x7f83140f81a0 msgr2=0x7f83140fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:56.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- 172.21.15.83:0/4001594622 shutdown_connections 2024-09-10T19:00:56.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- 172.21.15.83:0/4001594622 wait complete. 2024-09-10T19:00:56.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 librados: starting msgr at 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 librados: starting objecter 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 Processor -- start 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- start start 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 librados: setting wanted keys 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 librados: calling monclient init 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f831411c010 con 0x7f8314108ef0 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f831411d210 con 0x7f8314105360 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8319226640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f831411e410 con 0x7f83141025a0 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8312d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f83141025a0 0x7f83140fe770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58854/0 (socket says 172.21.15.83:58854) 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f8312d9d640 1 -- 172.21.15.83:0/823358612 learned_addr learned my addr 172.21.15.83:0/823358612 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:56.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter ms_handle_connect 0x7f83141025a0 2024-09-10T19:00:56.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter ms_handle_connect 0x7f8314105360 2024-09-10T19:00:56.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795104386 0 0) 0x7f831411e410 con 0x7f83141025a0 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8300003120 con 0x7f83141025a0 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter ms_handle_connect 0x7f8314108ef0 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819322127 0 0) 0x7f831411d210 con 0x7f8314105360 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831411e410 con 0x7f8314105360 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606098877 0 0) 0x7f831411c010 con 0x7f8314108ef0 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831411d210 con 0x7f8314108ef0 2024-09-10T19:00:56.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596216958 0 0) 0x7f8300003120 con 0x7f83141025a0 2024-09-10T19:00:56.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f831411c010 con 0x7f83141025a0 2024-09-10T19:00:56.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f83040018c0 con 0x7f83141025a0 2024-09-10T19:00:56.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040942753 0 0) 0x7f831411d210 con 0x7f8314108ef0 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8300003120 con 0x7f8314108ef0 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f830c0023d0 con 0x7f8314108ef0 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1379902681 0 0) 0x7f831411c010 con 0x7f83141025a0 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 >> v1:172.21.15.165:6789/0 conn(0x7f8314105360 legacy=0x7f83140fee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 >> v1:172.21.15.83:6789/0 conn(0x7f8314108ef0 legacy=0x7f831411a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.870+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831411f610 con 0x7f83141025a0 2024-09-10T19:00:56.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831411d400 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f8319226640 10 client.49436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f831411d850 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f8319226640 1 librados: init done 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8304002360 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f83040059a0 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f4005010 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f830401be80 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(466..466 src has 69..466) v4 ==== 8032+0+0 (unknown 3378952324 0 0) 0x7f830408a5e0 con 0x7f83141025a0 2024-09-10T19:00:56.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 10 client.49436.objecter ms_dispatch 0x7f8314000b90 osd_map(466..466 src has 69..466) v4 2024-09-10T19:00:56.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 3 client.49436.objecter handle_osd_map got epochs [466,466] > 0 2024-09-10T19:00:56.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 3 client.49436.objecter handle_osd_map decoding full epoch 466 2024-09-10T19:00:56.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.874+0000 7f830b7fe640 20 client.49436.objecter dump_active .. 0 homeless 2024-09-10T19:00:56.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.878+0000 7f830b7fe640 10 client.49436.objecter ms_handle_connect 0x7f83000716a0 2024-09-10T19:00:56.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.882+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f8304056760 con 0x7f83141025a0 2024-09-10T19:00:56.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:56.934 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:56.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3310055568 >> v1:172.21.15.83:6789/0 conn(0x7f66b80fb310 legacy=0x7f66b80fb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3310055568 shutdown_connections 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3310055568 >> 172.21.15.83:0/3310055568 conn(0x7f66b80f7060 msgr2=0x7f66b80f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3310055568 shutdown_connections 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3310055568 wait complete. 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 librados: starting msgr at 2024-09-10T19:00:56.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.966+0000 7f66bcb5b640 1 librados: starting objecter 2024-09-10T19:00:56.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 Processor -- start 2024-09-10T19:00:56.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- start start 2024-09-10T19:00:56.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 librados: setting wanted keys 2024-09-10T19:00:56.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 librados: calling monclient init 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b811e010 con 0x7f66b80fbdf0 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b811f1d0 con 0x7f66b80fb310 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f66b81203d0 con 0x7f66b810b140 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66b5d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f66b80fbdf0 0x7f66b81078d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38248/0 (socket says 172.21.15.83:38248) 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66b6d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f66b810b140 0x7f66b811c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:58882/0 (socket says 172.21.15.83:58882) 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66b5d9b640 1 -- 172.21.15.83:0/3548974608 learned_addr learned my addr 172.21.15.83:0/3548974608 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter ms_handle_connect 0x7f66b810b140 2024-09-10T19:00:56.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter ms_handle_connect 0x7f66b80fb310 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter ms_handle_connect 0x7f66b80fbdf0 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889918817 0 0) 0x7f66b811f1d0 con 0x7f66b80fb310 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a00039c0 con 0x7f66b80fb310 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343732780 0 0) 0x7f66b81203d0 con 0x7f66b810b140 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b811f1d0 con 0x7f66b810b140 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36344906 0 0) 0x7f66b811e010 con 0x7f66b80fbdf0 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b81203d0 con 0x7f66b80fbdf0 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274369862 0 0) 0x7f66a00039c0 con 0x7f66b80fb310 2024-09-10T19:00:56.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b811e010 con 0x7f66b80fb310 2024-09-10T19:00:56.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f66a8001550 con 0x7f66b80fb310 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4040174404 0 0) 0x7f66b811f1d0 con 0x7f66b810b140 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66a00039c0 con 0x7f66b810b140 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f66b0001b40 con 0x7f66b810b140 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1899143101 0 0) 0x7f66b81203d0 con 0x7f66b80fbdf0 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b811f1d0 con 0x7f66b80fbdf0 2024-09-10T19:00:56.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f66ac0018d0 con 0x7f66b80fbdf0 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1693847307 0 0) 0x7f66a00039c0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 >> v1:172.21.15.165:6789/0 conn(0x7f66b80fb310 legacy=0x7f66b81071c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 >> v1:172.21.15.83:6789/0 conn(0x7f66b80fbdf0 legacy=0x7f66b81078d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b81215d0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b811f3c0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 10 client.49442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b811f890 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f66b00025f0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 librados: init done 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f66b0005ca0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.970+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b80664b0 con 0x7f66b810b140 2024-09-10T19:00:56.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f66b0002190 con 0x7f66b810b140 2024-09-10T19:00:56.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(466..466 src has 69..466) v4 ==== 8032+0+0 (unknown 3378952324 0 0) 0x7f66b008a7f0 con 0x7f66b810b140 2024-09-10T19:00:56.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 10 client.49442.objecter ms_dispatch 0x7f66b8000b90 osd_map(466..466 src has 69..466) v4 2024-09-10T19:00:56.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 3 client.49442.objecter handle_osd_map got epochs [466,466] > 0 2024-09-10T19:00:56.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 3 client.49442.objecter handle_osd_map decoding full epoch 466 2024-09-10T19:00:56.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 20 client.49442.objecter dump_active .. 0 homeless 2024-09-10T19:00:56.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.974+0000 7f66a6ffd640 10 client.49442.objecter ms_handle_connect 0x7f66a00717e0 2024-09-10T19:00:56.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:56.982+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f66b0056dd0 con 0x7f66b810b140 2024-09-10T19:00:57.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:57.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.066+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7f82f4005840 con 0x7f83141025a0 2024-09-10T19:00:57.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:57.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.166+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f66b804bf60 con 0x7f66b810b140 2024-09-10T19:00:57.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:57.208 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:57.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.258+0000 7f830b7fe640 1 -- 172.21.15.83:0/823358612 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v467) v1 ==== 129+0+0 (unknown 462827411 0 0) 0x7f8304056580 con 0x7f83141025a0 2024-09-10T19:00:57.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.258+0000 7f66a6ffd640 1 -- 172.21.15.83:0/3548974608 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v467) v1 ==== 97+0+0 (unknown 1399043531 0 0) 0x7f66b0056770 con 0x7f66b810b140 2024-09-10T19:00:57.263 INFO:teuthology.orchestra.run.smithi083.stderr:set pool 1 pgp_num to 1 2024-09-10T19:00:57.263 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:00:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 10 librados: watch_flush enter 2024-09-10T19:00:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 10 librados: watch_flush exit 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 20 client.49436.objecter shutdown clearing up homeless session... 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 10 client.49436.objecter successfully canceled tick 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 >> v1:172.21.15.83:6800/3071793735 conn(0x7f83000716a0 legacy=0x7f8300073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 >> v1:172.21.15.83:6790/0 conn(0x7f83141025a0 legacy=0x7f83140fe770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 shutdown_connections 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 >> 172.21.15.83:0/823358612 conn(0x7f83140f81a0 msgr2=0x7f83140f8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 shutdown_connections 2024-09-10T19:00:57.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 -- 172.21.15.83:0/823358612 wait complete. 2024-09-10T19:00:57.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f8319226640 1 librados: shutdown 2024-09-10T19:00:57.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 10 librados: watch_flush enter 2024-09-10T19:00:57.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 10 librados: watch_flush exit 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 20 client.49442.objecter shutdown clearing up homeless session... 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 10 client.49442.objecter successfully canceled tick 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 >> v1:172.21.15.83:6800/3071793735 conn(0x7f66a00717e0 legacy=0x7f66a0073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 >> v1:172.21.15.83:6790/0 conn(0x7f66b810b140 legacy=0x7f66b811c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 shutdown_connections 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 >> 172.21.15.83:0/3548974608 conn(0x7f66b80f7060 msgr2=0x7f66b810d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 shutdown_connections 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 -- 172.21.15.83:0/3548974608 wait complete. 2024-09-10T19:00:57.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:00:57.262+0000 7f66bcb5b640 1 librados: shutdown 2024-09-10T19:00:57.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:57.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:57 smithi165 bash[22591]: cluster 2024-09-10T19:00:56.125115+0000 mgr.y (mgr.24526) 471 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 515 B/s wr, 4 op/s 2024-09-10T19:00:57.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:57 smithi165 bash[22591]: audit 2024-09-10T19:00:57.071058+0000 mon.c (mon.2) 636 : audit [INF] from='client.? v1:172.21.15.83:0/823358612' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:57 smithi165 bash[22591]: audit 2024-09-10T19:00:57.071562+0000 mon.a (mon.0) 2659 : audit [INF] from='client.49436 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:57 smithi165 bash[22591]: audit 2024-09-10T19:00:57.171272+0000 mon.c (mon.2) 637 : audit [INF] from='client.? v1:172.21.15.83:0/3548974608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:57 smithi165 bash[22591]: audit 2024-09-10T19:00:57.172048+0000 mon.a (mon.0) 2660 : audit [INF] from='client.49442 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:00:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:00:57.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:57.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[18629]: cluster 2024-09-10T19:00:56.125115+0000 mgr.y (mgr.24526) 471 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 515 B/s wr, 4 op/s 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[18629]: audit 2024-09-10T19:00:57.071058+0000 mon.c (mon.2) 636 : audit [INF] from='client.? v1:172.21.15.83:0/823358612' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[18629]: audit 2024-09-10T19:00:57.071562+0000 mon.a (mon.0) 2659 : audit [INF] from='client.49436 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[18629]: audit 2024-09-10T19:00:57.171272+0000 mon.c (mon.2) 637 : audit [INF] from='client.? v1:172.21.15.83:0/3548974608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[18629]: audit 2024-09-10T19:00:57.172048+0000 mon.a (mon.0) 2660 : audit [INF] from='client.49442 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[32381]: cluster 2024-09-10T19:00:56.125115+0000 mgr.y (mgr.24526) 471 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 515 B/s wr, 4 op/s 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[32381]: audit 2024-09-10T19:00:57.071058+0000 mon.c (mon.2) 636 : audit [INF] from='client.? v1:172.21.15.83:0/823358612' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[32381]: audit 2024-09-10T19:00:57.071562+0000 mon.a (mon.0) 2659 : audit [INF] from='client.49436 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:00:57.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[32381]: audit 2024-09-10T19:00:57.171272+0000 mon.c (mon.2) 637 : audit [INF] from='client.? v1:172.21.15.83:0/3548974608' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:57 smithi083 bash[32381]: audit 2024-09-10T19:00:57.172048+0000 mon.a (mon.0) 2660 : audit [INF] from='client.49442 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:00:57.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:57.617 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:57.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:57.756 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:57.860 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:57.895 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:57.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:57.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.032 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:58.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.172 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:58.267 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:00:58.270 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: cluster 2024-09-10T19:00:57.249531+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:58.270 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:57.258560+0000 mon.a (mon.0) 2662 : audit [INF] from='client.49436 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:00:58.270 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:57.258796+0000 mon.a (mon.0) 2663 : audit [INF] from='client.49442 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: cluster 2024-09-10T19:00:57.258915+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: cluster 2024-09-10T19:00:57.294287+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: cluster 2024-09-10T19:00:57.294559+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:58.232892+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:58.234878+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:58.236422+0000 mon.a (mon.0) 2665 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[32381]: audit 2024-09-10T19:00:58.238243+0000 mon.a (mon.0) 2666 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:58.301 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:00:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:00:58.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:58.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: cluster 2024-09-10T19:00:57.249531+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:57.258560+0000 mon.a (mon.0) 2662 : audit [INF] from='client.49436 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:57.258796+0000 mon.a (mon.0) 2663 : audit [INF] from='client.49442 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: cluster 2024-09-10T19:00:57.258915+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: cluster 2024-09-10T19:00:57.294287+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: cluster 2024-09-10T19:00:57.294559+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:58.232892+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:58.234878+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:58.236422+0000 mon.a (mon.0) 2665 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:58 smithi083 bash[18629]: audit 2024-09-10T19:00:58.238243+0000 mon.a (mon.0) 2666 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.575 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: cluster 2024-09-10T19:00:57.249531+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:57.258560+0000 mon.a (mon.0) 2662 : audit [INF] from='client.49436 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:57.258796+0000 mon.a (mon.0) 2663 : audit [INF] from='client.49442 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: cluster 2024-09-10T19:00:57.258915+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: cluster 2024-09-10T19:00:57.294287+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: cluster 2024-09-10T19:00:57.294559+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:58.232892+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:58.234878+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:58.236422+0000 mon.a (mon.0) 2665 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:00:58.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:58 smithi165 bash[22591]: audit 2024-09-10T19:00:58.238243+0000 mon.a (mon.0) 2666 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-65"}]: dispatch 2024-09-10T19:00:58.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:58.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.839 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:00:58.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:58.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:59.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:59.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:59.243 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:00:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:00:59.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:59.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:00:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:00:59.422 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:00:59.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:59.508 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:00:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[18629]: audit 2024-09-10T19:00:58.012985+0000 mgr.y (mgr.24526) 472 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[18629]: cluster 2024-09-10T19:00:58.125942+0000 mgr.y (mgr.24526) 473 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[18629]: cluster 2024-09-10T19:00:58.280128+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[32381]: audit 2024-09-10T19:00:58.012985+0000 mgr.y (mgr.24526) 472 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[32381]: cluster 2024-09-10T19:00:58.125942+0000 mgr.y (mgr.24526) 473 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:00:59.521 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:00:59 smithi083 bash[32381]: cluster 2024-09-10T19:00:58.280128+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:00:59.611 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:59 smithi165 bash[22591]: audit 2024-09-10T19:00:58.012985+0000 mgr.y (mgr.24526) 472 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:00:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:59 smithi165 bash[22591]: cluster 2024-09-10T19:00:58.125942+0000 mgr.y (mgr.24526) 473 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:00:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:00:59 smithi165 bash[22591]: cluster 2024-09-10T19:00:58.280128+0000 mon.a (mon.0) 2667 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:00:59.642 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:59.745 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.769 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:00:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:00:59.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:59.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:00:59.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:00:59.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:00.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:00.042 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:00.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:00.180 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:00.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:00.322 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:00.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:00.459 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:00.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:59.297538+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[18629]: audit 2024-09-10T19:00:59.565302+0000 mon.c (mon.2) 638 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:59.589739+0000 osd.0 (osd.0) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[18629]: cluster 2024-09-10T19:00:59.590016+0000 osd.0 (osd.0) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:59.297538+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[32381]: audit 2024-09-10T19:00:59.565302+0000 mon.c (mon.2) 638 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:59.589739+0000 osd.0 (osd.0) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:00 smithi083 bash[32381]: cluster 2024-09-10T19:00:59.590016+0000 osd.0 (osd.0) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:00.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:00.574 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:59.297538+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:01:00.574 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:00 smithi165 bash[22591]: audit 2024-09-10T19:00:59.565302+0000 mon.c (mon.2) 638 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:00.575 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:59.589739+0000 osd.0 (osd.0) 248 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:00.575 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:00 smithi165 bash[22591]: cluster 2024-09-10T19:00:59.590016+0000 osd.0 (osd.0) 249 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:00.596 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:00.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:00.735 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:00.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:00.872 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:00.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:00.920 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:01:00.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:01.011 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:01.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:01.137 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:01.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:01.326 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:01.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:01.446 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[18629]: cluster 2024-09-10T19:01:00.126715+0000 mgr.y (mgr.24526) 474 : cluster [DBG] pgmap v705: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:01.446 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[18629]: cluster 2024-09-10T19:01:00.306751+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:01:01.446 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[18629]: audit 2024-09-10T19:01:00.312312+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.446 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[18629]: audit 2024-09-10T19:01:00.318671+0000 mon.a (mon.0) 2670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.447 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[32381]: cluster 2024-09-10T19:01:00.126715+0000 mgr.y (mgr.24526) 474 : cluster [DBG] pgmap v705: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:01.447 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[32381]: cluster 2024-09-10T19:01:00.306751+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:01:01.447 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[32381]: audit 2024-09-10T19:01:00.312312+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.447 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:01 smithi083 bash[32381]: audit 2024-09-10T19:01:00.318671+0000 mon.a (mon.0) 2670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.466 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:01.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:01.583 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:01.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:01 smithi165 bash[22591]: cluster 2024-09-10T19:01:00.126715+0000 mgr.y (mgr.24526) 474 : cluster [DBG] pgmap v705: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:01.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:01 smithi165 bash[22591]: cluster 2024-09-10T19:01:00.306751+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:01:01.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:01 smithi165 bash[22591]: audit 2024-09-10T19:01:00.312312+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.583 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:01 smithi165 bash[22591]: audit 2024-09-10T19:01:00.318671+0000 mon.a (mon.0) 2670 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:01.604 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:01.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:01.740 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:01.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:01.875 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:01.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:01.978 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:02.011 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:02.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:02.147 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:02.250 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:02.283 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:02.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:02.419 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:02.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:02.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[18629]: audit 2024-09-10T19:01:01.312426+0000 mon.a (mon.0) 2671 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:02.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[18629]: cluster 2024-09-10T19:01:01.312801+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:01:02.534 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[18629]: audit 2024-09-10T19:01:01.316116+0000 mon.b (mon.1) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:02.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[18629]: cluster 2024-09-10T19:01:02.127628+0000 mgr.y (mgr.24526) 475 : cluster [DBG] pgmap v708: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:02.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[32381]: audit 2024-09-10T19:01:01.312426+0000 mon.a (mon.0) 2671 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:02.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[32381]: cluster 2024-09-10T19:01:01.312801+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:01:02.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[32381]: audit 2024-09-10T19:01:01.316116+0000 mon.b (mon.1) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:02.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[32381]: cluster 2024-09-10T19:01:02.127628+0000 mgr.y (mgr.24526) 475 : cluster [DBG] pgmap v708: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:02.552 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:02 smithi165 bash[22591]: audit 2024-09-10T19:01:01.312426+0000 mon.a (mon.0) 2671 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:02 smithi165 bash[22591]: cluster 2024-09-10T19:01:01.312801+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:01:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:02 smithi165 bash[22591]: audit 2024-09-10T19:01:01.316116+0000 mon.b (mon.1) 359 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:02.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:02 smithi165 bash[22591]: cluster 2024-09-10T19:01:02.127628+0000 mgr.y (mgr.24526) 475 : cluster [DBG] pgmap v708: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:02.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:02.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:02.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:02] "GET /metrics HTTP/1.1" 200 226404 "" "Prometheus/2.43.0" 2024-09-10T19:01:02.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:02.831 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:02.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:02.968 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:02.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:03.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:03.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:03.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:03.244 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:03.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:03.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:03.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:03.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:03.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:03.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: cluster 2024-09-10T19:01:02.259755+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: cluster 2024-09-10T19:01:02.260083+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: cluster 2024-09-10T19:01:02.340806+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: audit 2024-09-10T19:01:02.377532+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: audit 2024-09-10T19:01:02.379420+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: audit 2024-09-10T19:01:02.380947+0000 mon.a (mon.0) 2674 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:03 smithi165 bash[22591]: audit 2024-09-10T19:01:02.382799+0000 mon.a (mon.0) 2675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.643 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:03.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:03.779 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: cluster 2024-09-10T19:01:02.259755+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: cluster 2024-09-10T19:01:02.260083+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: cluster 2024-09-10T19:01:02.340806+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: audit 2024-09-10T19:01:02.377532+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: audit 2024-09-10T19:01:02.379420+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: audit 2024-09-10T19:01:02.380947+0000 mon.a (mon.0) 2674 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[18629]: audit 2024-09-10T19:01:02.382799+0000 mon.a (mon.0) 2675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: cluster 2024-09-10T19:01:02.259755+0000 osd.7 (osd.7) 266 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:03.781 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: cluster 2024-09-10T19:01:02.260083+0000 osd.7 (osd.7) 267 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:03.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: cluster 2024-09-10T19:01:02.340806+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:01:03.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: audit 2024-09-10T19:01:02.377532+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: audit 2024-09-10T19:01:02.379420+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: audit 2024-09-10T19:01:02.380947+0000 mon.a (mon.0) 2674 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:03.782 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:03 smithi083 bash[32381]: audit 2024-09-10T19:01:02.382799+0000 mon.a (mon.0) 2675 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-67"}]: dispatch 2024-09-10T19:01:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:03.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:03.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:03.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:03.936 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:01:03.936 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:01:03.936 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T19:01:03.936 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:01:04.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:04.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:04.184 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:04.286 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:04.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:04.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:04.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[18629]: cluster 2024-09-10T19:01:03.351614+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:01:04.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[18629]: cluster 2024-09-10T19:01:04.128402+0000 mgr.y (mgr.24526) 476 : cluster [DBG] pgmap v711: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:01:04.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[18629]: cluster 2024-09-10T19:01:04.162039+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:01:04.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[18629]: audit 2024-09-10T19:01:04.169667+0000 mon.b (mon.1) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[18629]: audit 2024-09-10T19:01:04.172816+0000 mon.a (mon.0) 2678 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[32381]: cluster 2024-09-10T19:01:03.351614+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[32381]: cluster 2024-09-10T19:01:04.128402+0000 mgr.y (mgr.24526) 476 : cluster [DBG] pgmap v711: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[32381]: cluster 2024-09-10T19:01:04.162039+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[32381]: audit 2024-09-10T19:01:04.169667+0000 mon.b (mon.1) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:04 smithi083 bash[32381]: audit 2024-09-10T19:01:04.172816+0000 mon.a (mon.0) 2678 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:04.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:04.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:04 smithi165 bash[22591]: cluster 2024-09-10T19:01:03.351614+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:01:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:04 smithi165 bash[22591]: cluster 2024-09-10T19:01:04.128402+0000 mgr.y (mgr.24526) 476 : cluster [DBG] pgmap v711: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:01:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:04 smithi165 bash[22591]: cluster 2024-09-10T19:01:04.162039+0000 mon.a (mon.0) 2677 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:01:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:04 smithi165 bash[22591]: audit 2024-09-10T19:01:04.169667+0000 mon.b (mon.1) 362 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:04 smithi165 bash[22591]: audit 2024-09-10T19:01:04.172816+0000 mon.a (mon.0) 2678 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:04.635 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:01:04.635 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:01:04.635 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T19:01:04.635 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T19:01:04.635 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:01:04.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:04.822 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:04.854 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:04.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:04.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:04.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:05.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:05.123 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:05.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:05.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:05.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:05.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:05.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:05.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:05.671 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:05.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:05.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:05.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:05.945 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:05.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:06.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:06.079 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.166103+0000 mon.a (mon.0) 2679 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: cluster 2024-09-10T19:01:05.166285+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.170606+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.211677+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.213065+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.214916+0000 mon.a (mon.0) 2681 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.177 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[18629]: audit 2024-09-10T19:01:05.216230+0000 mon.a (mon.0) 2682 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.166103+0000 mon.a (mon.0) 2679 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: cluster 2024-09-10T19:01:05.166285+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.170606+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.211677+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.213065+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.214916+0000 mon.a (mon.0) 2681 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.178 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:06 smithi083 bash[32381]: audit 2024-09-10T19:01:05.216230+0000 mon.a (mon.0) 2682 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:06.206 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:06.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:06.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:01:06.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:06.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:06.479 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:06.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.166103+0000 mon.a (mon.0) 2679 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: cluster 2024-09-10T19:01:05.166285+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.170606+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.211677+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.213065+0000 mon.b (mon.1) 365 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.214916+0000 mon.a (mon.0) 2681 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:06 smithi165 bash[22591]: audit 2024-09-10T19:01:05.216230+0000 mon.a (mon.0) 2682 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-69"}]: dispatch 2024-09-10T19:01:06.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:06.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:06.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:06.850 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:06.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:06.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:06.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:06.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:07.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:07.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:07.166 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:07.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:07.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:07.306 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:07.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:07.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:07 smithi083 bash[32381]: cluster 2024-09-10T19:01:06.130485+0000 mgr.y (mgr.24526) 477 : cluster [DBG] pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:01:07.422 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:07 smithi083 bash[32381]: cluster 2024-09-10T19:01:06.174831+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:01:07.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:07 smithi083 bash[18629]: cluster 2024-09-10T19:01:06.130485+0000 mgr.y (mgr.24526) 477 : cluster [DBG] pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:01:07.422 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:07 smithi083 bash[18629]: cluster 2024-09-10T19:01:06.174831+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:01:07.444 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:07.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:07.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:07 smithi165 bash[22591]: cluster 2024-09-10T19:01:06.130485+0000 mgr.y (mgr.24526) 477 : cluster [DBG] pgmap v714: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:01:07.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:07 smithi165 bash[22591]: cluster 2024-09-10T19:01:06.174831+0000 mon.a (mon.0) 2683 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:01:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:07.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:07.720 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:07.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:07.857 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:07.961 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:07.996 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:07.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:08.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:08.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:08.133 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:08.236 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[18629]: cluster 2024-09-10T19:01:07.188229+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:01:08.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[18629]: audit 2024-09-10T19:01:07.190915+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[18629]: audit 2024-09-10T19:01:07.196086+0000 mon.a (mon.0) 2685 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[32381]: cluster 2024-09-10T19:01:07.188229+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:01:08.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[32381]: audit 2024-09-10T19:01:07.190915+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:08 smithi083 bash[32381]: audit 2024-09-10T19:01:07.196086+0000 mon.a (mon.0) 2685 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.256 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:08.359 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.390 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:08.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:08.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:08 smithi165 bash[22591]: cluster 2024-09-10T19:01:07.188229+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:01:08.505 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:08 smithi165 bash[22591]: audit 2024-09-10T19:01:07.190915+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.506 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:08 smithi165 bash[22591]: audit 2024-09-10T19:01:07.196086+0000 mon.a (mon.0) 2685 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:08.526 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:08.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:08.664 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:08.767 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:08.793 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:08.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- 172.21.15.83:0/152195646 >> v1:172.21.15.83:6790/0 conn(0x7f0a00100c60 legacy=0x7f0a00101030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:08.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- 172.21.15.83:0/152195646 shutdown_connections 2024-09-10T19:01:08.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- 172.21.15.83:0/152195646 >> 172.21.15.83:0/152195646 conn(0x7f0a00074a40 msgr2=0x7f0a000f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:08.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- 172.21.15.83:0/152195646 shutdown_connections 2024-09-10T19:01:08.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- 172.21.15.83:0/152195646 wait complete. 2024-09-10T19:01:08.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 librados: starting msgr at 2024-09-10T19:01:08.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 librados: starting objecter 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 Processor -- start 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- start start 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 librados: setting wanted keys 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 librados: calling monclient init 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a001005e0 con 0x7f0a001075a0 2024-09-10T19:01:08.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a0011d340 con 0x7f0a00100c60 2024-09-10T19:01:08.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a08630640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a0011e4e0 con 0x7f0a00103a40 2024-09-10T19:01:08.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a05bcb640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0a00103a40 0x7f0a000ff150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:32848/0 (socket says 172.21.15.83:32848) 2024-09-10T19:01:08.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f0a05bcb640 1 -- 172.21.15.83:0/3433567134 learned_addr learned my addr 172.21.15.83:0/3433567134 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 10 client.?.objecter ms_handle_connect 0x7f0a001075a0 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 10 client.?.objecter ms_handle_connect 0x7f0a00100c60 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769732354 0 0) 0x7f0a001005e0 con 0x7f0a001075a0 2024-09-10T19:01:08.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.826+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003b70 con 0x7f0a001075a0 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 10 client.?.objecter ms_handle_connect 0x7f0a00103a40 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220105109 0 0) 0x7f0a0011d340 con 0x7f0a00100c60 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a001005e0 con 0x7f0a00100c60 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322083945 0 0) 0x7f0a0011e4e0 con 0x7f0a00103a40 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0011d340 con 0x7f0a00103a40 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1953283336 0 0) 0x7f09ec003b70 con 0x7f0a001075a0 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0011e4e0 con 0x7f0a001075a0 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f09f80019a0 con 0x7f0a001075a0 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1303580800 0 0) 0x7f0a0011d340 con 0x7f0a00103a40 2024-09-10T19:01:08.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09ec003b70 con 0x7f0a00103a40 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1758869548 0 0) 0x7f0a0011e4e0 con 0x7f0a001075a0 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6790/0 conn(0x7f0a00103a40 legacy=0x7f0a000ff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.165:6789/0 conn(0x7f0a00100c60 legacy=0x7f0a000fea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0011f6c0 con 0x7f0a001075a0 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0011d530 con 0x7f0a001075a0 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f0a08630640 10 client.25377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0011d980 con 0x7f0a001075a0 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f0a08630640 1 librados: init done 2024-09-10T19:01:08.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f09f8002580 con 0x7f0a001075a0 2024-09-10T19:01:08.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f09f8003460 con 0x7f0a001075a0 2024-09-10T19:01:08.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.830+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f09e4004ff0 con 0x7f0a001075a0 2024-09-10T19:01:08.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f09f8002120 con 0x7f0a001075a0 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(478..478 src has 69..478) v4 ==== 8004+0+0 (unknown 609717799 0 0) 0x7f09f8087d60 con 0x7f0a001075a0 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 osd_map(478..478 src has 69..478) v4 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 3 client.25377.objecter handle_osd_map got epochs [478,478] > 0 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 3 client.25377.objecter handle_osd_map decoding full epoch 478 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 20 client.25377.objecter dump_active .. 0 homeless 2024-09-10T19:01:08.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v478) v1 ==== 54+0+16 (unknown 2736291213 0 2420000192) 0x7f09f8088180 con 0x7f0a001075a0 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09ec071780 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 10 client.25377.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 1 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 1 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 0 1 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 10 client.25377.objecter _send_command 1 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4005290 con 0x7f09e4004610 2024-09-10T19:01:08.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e4004610 2024-09-10T19:01:08.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4005290 con 0x7f09e4004610 2024-09-10T19:01:08.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:01:08.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 1 = system:0 2024-09-10T19:01:08.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T19:01:08.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.834+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 0 1 2024-09-10T19:01:08.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 10 client.25377.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 2 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 2 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4005810 osd=0 3 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 0 2 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 10 client.25377.objecter _send_command 2 2024-09-10T19:01:08.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.850+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4004c60 con 0x7f09e4004610 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4004c60 con 0x7f09e4004610 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 4 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 0 2 2024-09-10T19:01:08.860 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:01:08.861 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:01:08.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 10 client.25377.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 3 2024-09-10T19:01:08.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 1 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 10 client.25377.objecter _send_command 3 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4003a50 con 0x7f09e40035e0 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e40035e0 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4003a50 con 0x7f09e40035e0 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 3 = system:0 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T19:01:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.854+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 1 3 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 10 client.25377.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 4 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 4 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4004e10 osd=1 3 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 1 4 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 10 client.25377.objecter _send_command 4 2024-09-10T19:01:08.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4003060 con 0x7f09e40035e0 2024-09-10T19:01:08.880 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:01:08.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4003060 con 0x7f09e40035e0 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 4 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 1 4 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 10 client.25377.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 5 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 5 2024-09-10T19:01:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 2 5 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 4 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 10 client.25377.objecter _send_command 5 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e40150d0 con 0x7f09e4002070 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e4002070 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e40150d0 con 0x7f09e4002070 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 5 = system:0 2024-09-10T19:01:08.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 4 2024-09-10T19:01:08.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.874+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 2 5 2024-09-10T19:01:08.897 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 10 client.25377.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 6 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 6 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4003410 osd=2 3 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 2 6 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 4 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 10 client.25377.objecter _send_command 6 2024-09-10T19:01:08.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.898+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e40015e0 con 0x7f09e4002070 2024-09-10T19:01:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e40015e0 con 0x7f09e4002070 2024-09-10T19:01:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 4 2024-09-10T19:01:08.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 2 6 2024-09-10T19:01:08.908 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:01:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:01:08.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 10 client.25377.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 7 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 7 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 3 7 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 4 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 10 client.25377.objecter _send_command 7 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4011550 con 0x7f09e4015270 2024-09-10T19:01:08.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e4015270 2024-09-10T19:01:08.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4011550 con 0x7f09e4015270 2024-09-10T19:01:08.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:01:08.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 7 = system:0 2024-09-10T19:01:08.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 4 2024-09-10T19:01:08.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.902+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 3 7 2024-09-10T19:01:08.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 10 client.25377.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 8 2024-09-10T19:01:08.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 8 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4010ed0 osd=3 3 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 3 8 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 4 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 10 client.25377.objecter _send_command 8 2024-09-10T19:01:08.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e40176f0 con 0x7f09e4015270 2024-09-10T19:01:08.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e40176f0 con 0x7f09e4015270 2024-09-10T19:01:08.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 4 2024-09-10T19:01:08.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 3 8 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 10 client.25377.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 9 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 9 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 4 9 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 4 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 10 client.25377.objecter _send_command 9 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.918+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e401e660 con 0x7f09e401ab60 2024-09-10T19:01:08.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e401ab60 2024-09-10T19:01:08.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:08.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e401e660 con 0x7f09e401ab60 2024-09-10T19:01:08.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:01:08.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 9 = system:0 2024-09-10T19:01:08.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 4 2024-09-10T19:01:08.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.922+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 4 9 2024-09-10T19:01:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:08.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 10 client.25377.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 10 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 10 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e40176f0 osd=4 3 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 4 10 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 4 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 10 client.25377.objecter _send_command 10 2024-09-10T19:01:08.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.938+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e401e660 con 0x7f09e401ab60 2024-09-10T19:01:08.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e401e660 con 0x7f09e401ab60 2024-09-10T19:01:08.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 4 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 4 10 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 10 client.25377.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 11 2024-09-10T19:01:08.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 11 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 5 11 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 4 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 10 client.25377.objecter _send_command 11 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4022980 con 0x7f09e401ece0 2024-09-10T19:01:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e401ece0 2024-09-10T19:01:08.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4022980 con 0x7f09e401ece0 2024-09-10T19:01:08.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:01:08.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 11 = system:0 2024-09-10T19:01:08.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 4 2024-09-10T19:01:08.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.942+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 5 11 2024-09-10T19:01:08.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 10 client.25377.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 12 2024-09-10T19:01:08.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 12 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e401e660 osd=5 3 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 5 12 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 4 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 10 client.25377.objecter _send_command 12 2024-09-10T19:01:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.962+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4022980 con 0x7f09e401ece0 2024-09-10T19:01:08.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4022980 con 0x7f09e401ece0 2024-09-10T19:01:08.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 4 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 5 12 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 10 client.25377.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 13 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 13 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 6 13 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 4 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 10 client.25377.objecter _send_command 13 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4024e30 con 0x7f09e40249e0 2024-09-10T19:01:08.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.966+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e40249e0 2024-09-10T19:01:08.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.970+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4024e30 con 0x7f09e40249e0 2024-09-10T19:01:08.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.970+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:01:08.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.970+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 13 = system:0 2024-09-10T19:01:08.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.970+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 4 2024-09-10T19:01:08.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.970+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 6 13 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 10 client.25377.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 14 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 14 2024-09-10T19:01:08.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4022980 osd=6 3 2024-09-10T19:01:08.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 6 14 2024-09-10T19:01:08.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 4 2024-09-10T19:01:08.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 10 client.25377.objecter _send_command 14 2024-09-10T19:01:08.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.982+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4024e30 con 0x7f09e40249e0 2024-09-10T19:01:08.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4024e30 con 0x7f09e40249e0 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 4 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 6 14 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:01:08.993 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 10 client.25377.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 15 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 15 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 7 15 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 4 2024-09-10T19:01:08.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 10 client.25377.objecter _send_command 15 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09e4017290 con 0x7f09e4024fa0 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.986+0000 7f09f2ffd640 10 client.25377.objecter ms_handle_connect 0x7f09e4024fa0 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.990+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f09e4017290 con 0x7f09e4024fa0 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.990+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.990+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 15 = system:0 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.990+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 4 2024-09-10T19:01:08.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:08.990+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 7 15 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 10 client.25377.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign -1 16 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 20 client.25377.objecter _get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 15 client.25377.objecter _session_command_op_remove -1 16 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 20 client.25377.objecter get_session s=0x7f09e4024e30 osd=7 3 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 15 client.25377.objecter _session_command_op_assign 7 16 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 4 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 10 client.25377.objecter _send_command 16 2024-09-10T19:01:09.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.010+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09e4017290 con 0x7f09e4024fa0 2024-09-10T19:01:09.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f09f2ffd640 1 -- 172.21.15.83:0/3433567134 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f09e4017290 con 0x7f09e4024fa0 2024-09-10T19:01:09.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f09f2ffd640 10 client.25377.objecter ms_dispatch 0x7f0a00000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:09.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f09f2ffd640 10 client.25377.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:09.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f09f2ffd640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 4 2024-09-10T19:01:09.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f09f2ffd640 15 client.25377.objecter _session_command_op_remove 7 16 2024-09-10T19:01:09.022 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 librados: watch_flush enter 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 librados: watch_flush exit 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.0 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6801/2535527483 conn(0x7f09e4004610 legacy=0x7f09e4002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4005810 osd=0 1 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.1 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6805/2309624637 conn(0x7f09e40035e0 legacy=0x7f09e400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4004e10 osd=1 1 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.2 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6809/386094220 conn(0x7f09e4002070 legacy=0x7f09e40138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4003410 osd=2 1 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.3 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6813/1279688021 conn(0x7f09e4015270 legacy=0x7f09e40156a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4010ed0 osd=3 1 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.4 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.165:6800/3225515265 conn(0x7f09e401ab60 legacy=0x7f09e401cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e40176f0 osd=4 1 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.5 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.165:6804/571173777 conn(0x7f09e401ece0 legacy=0x7f09e4021100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e401e660 osd=5 1 2024-09-10T19:01:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.6 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.165:6808/1317922936 conn(0x7f09e40249e0 legacy=0x7f09e4015db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4022980 osd=6 1 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter close_session for osd.7 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.165:6812/1610151650 conn(0x7f09e4024fa0 legacy=0x7f09e40169f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter put_session s=0x7f09e4024e30 osd=7 1 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 20 client.25377.objecter shutdown clearing up homeless session... 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 10 client.25377.objecter successfully canceled tick 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6800/3071793735 conn(0x7f09ec071780 legacy=0x7f09ec073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.014+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> v1:172.21.15.83:6789/0 conn(0x7f0a001075a0 legacy=0x7f0a0011ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a06bcd640 1 -- 172.21.15.83:0/3433567134 reap_dead start 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 shutdown_connections 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 >> 172.21.15.83:0/3433567134 conn(0x7f0a00074a40 msgr2=0x7f0a00070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 shutdown_connections 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a08630640 1 -- 172.21.15.83:0/3433567134 wait complete. 2024-09-10T19:01:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:09.018+0000 7f0a08630640 1 librados: shutdown 2024-09-10T19:01:09.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:09.064 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:09.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:09.203 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:09.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.024201+0000 mgr.y (mgr.24526) 478 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: cluster 2024-09-10T19:01:08.131168+0000 mgr.y (mgr.24526) 479 : cluster [DBG] pgmap v717: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: cluster 2024-09-10T19:01:08.181813+0000 mon.a (mon.0) 2686 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.203826+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.204271+0000 mon.a (mon.0) 2687 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: cluster 2024-09-10T19:01:08.204478+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.208953+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.214253+0000 mon.a (mon.0) 2689 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[18629]: audit 2024-09-10T19:01:08.836676+0000 mon.a (mon.0) 2690 : audit [DBG] from='client.? v1:172.21.15.83:0/3433567134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.024201+0000 mgr.y (mgr.24526) 478 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: cluster 2024-09-10T19:01:08.131168+0000 mgr.y (mgr.24526) 479 : cluster [DBG] pgmap v717: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: cluster 2024-09-10T19:01:08.181813+0000 mon.a (mon.0) 2686 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.203826+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.204271+0000 mon.a (mon.0) 2687 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: cluster 2024-09-10T19:01:08.204478+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.208953+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.214253+0000 mon.a (mon.0) 2689 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.324 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:09 smithi083 bash[32381]: audit 2024-09-10T19:01:08.836676+0000 mon.a (mon.0) 2690 : audit [DBG] from='client.? v1:172.21.15.83:0/3433567134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:09.336 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.024201+0000 mgr.y (mgr.24526) 478 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: cluster 2024-09-10T19:01:08.131168+0000 mgr.y (mgr.24526) 479 : cluster [DBG] pgmap v717: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: cluster 2024-09-10T19:01:08.181813+0000 mon.a (mon.0) 2686 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.203826+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.204271+0000 mon.a (mon.0) 2687 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: cluster 2024-09-10T19:01:08.204478+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.208953+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.214253+0000 mon.a (mon.0) 2689 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:01:09.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:09 smithi165 bash[22591]: audit 2024-09-10T19:01:08.836676+0000 mon.a (mon.0) 2690 : audit [DBG] from='client.? v1:172.21.15.83:0/3433567134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:09.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:09.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:09.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:09.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:09.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:09.751 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:09.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:09.885 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:09.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:09.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:10.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:10.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:10.164 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:10.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:10.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:10 smithi083 bash[18629]: audit 2024-09-10T19:01:09.203000+0000 mon.a (mon.0) 2691 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:01:10.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:10 smithi083 bash[18629]: cluster 2024-09-10T19:01:09.203279+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:01:10.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:10 smithi083 bash[32381]: audit 2024-09-10T19:01:09.203000+0000 mon.a (mon.0) 2691 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:01:10.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:10 smithi083 bash[32381]: cluster 2024-09-10T19:01:09.203279+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:01:10.296 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:10.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:10.437 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:10.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:10 smithi165 bash[22591]: audit 2024-09-10T19:01:09.203000+0000 mon.a (mon.0) 2691 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:01:10.553 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:10 smithi165 bash[22591]: cluster 2024-09-10T19:01:09.203279+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:01:10.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:10.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:10.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:10.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:10.842 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:10.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:10.975 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:10.982 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:10.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:11.086 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:11.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:11.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:11.388 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:11.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:11.503 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:11 smithi083 bash[18629]: cluster 2024-09-10T19:01:10.132050+0000 mgr.y (mgr.24526) 480 : cluster [DBG] pgmap v720: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:11.504 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:11 smithi083 bash[18629]: cluster 2024-09-10T19:01:10.221967+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:01:11.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:11 smithi083 bash[32381]: cluster 2024-09-10T19:01:10.132050+0000 mgr.y (mgr.24526) 480 : cluster [DBG] pgmap v720: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:11.504 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:11 smithi083 bash[32381]: cluster 2024-09-10T19:01:10.221967+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:01:11.524 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:11.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:11.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:11 smithi165 bash[22591]: cluster 2024-09-10T19:01:10.132050+0000 mgr.y (mgr.24526) 480 : cluster [DBG] pgmap v720: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:11.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:11 smithi165 bash[22591]: cluster 2024-09-10T19:01:10.221967+0000 mon.a (mon.0) 2693 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:01:11.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:11.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:11.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:11.802 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:11.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:11.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:11.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:11.970 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:12.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.081 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:12.185 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.220 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:12.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:12.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:12.493 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:12.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:01:12.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:12.629 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:12.732 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.760 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:12.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:12.873 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:12] "GET /metrics HTTP/1.1" 200 226404 "" "Prometheus/2.43.0" 2024-09-10T19:01:12.887 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:12.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:12.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:13.027 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:13.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.163 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:13.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:13.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.436 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:13.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[32381]: cluster 2024-09-10T19:01:12.134343+0000 mgr.y (mgr.24526) 481 : cluster [DBG] pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-10T19:01:13.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[32381]: cluster 2024-09-10T19:01:12.175624+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:13.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[32381]: cluster 2024-09-10T19:01:12.175933+0000 osd.6 (osd.6) 214 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:13.450 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[32381]: cluster 2024-09-10T19:01:12.233212+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:01:13.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[18629]: cluster 2024-09-10T19:01:12.134343+0000 mgr.y (mgr.24526) 481 : cluster [DBG] pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-10T19:01:13.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[18629]: cluster 2024-09-10T19:01:12.175624+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:13.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[18629]: cluster 2024-09-10T19:01:12.175933+0000 osd.6 (osd.6) 214 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:13.450 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:13 smithi083 bash[18629]: cluster 2024-09-10T19:01:12.233212+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:01:13.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.573 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:13.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:13.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:13 smithi165 bash[22591]: cluster 2024-09-10T19:01:12.134343+0000 mgr.y (mgr.24526) 481 : cluster [DBG] pgmap v722: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-09-10T19:01:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:13 smithi165 bash[22591]: cluster 2024-09-10T19:01:12.175624+0000 osd.6 (osd.6) 213 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:13 smithi165 bash[22591]: cluster 2024-09-10T19:01:12.175933+0000 osd.6 (osd.6) 214 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:13.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:13 smithi165 bash[22591]: cluster 2024-09-10T19:01:12.233212+0000 mon.a (mon.0) 2694 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:01:13.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:13.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:13.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:13.849 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:13.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:13.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:13.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:14.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:14.122 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:14.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.256 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:14.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:14.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:14 smithi165 bash[22591]: cluster 2024-09-10T19:01:14.155847+0000 mon.a (mon.0) 2695 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:14.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:14 smithi165 bash[22591]: cluster 2024-09-10T19:01:14.166397+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:01:14.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:14.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:14 smithi083 bash[18629]: cluster 2024-09-10T19:01:14.155847+0000 mon.a (mon.0) 2695 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:14.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:14 smithi083 bash[18629]: cluster 2024-09-10T19:01:14.166397+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:01:14.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:14 smithi083 bash[32381]: cluster 2024-09-10T19:01:14.155847+0000 mon.a (mon.0) 2695 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:14.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:14 smithi083 bash[32381]: cluster 2024-09-10T19:01:14.166397+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:01:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- 172.21.15.83:0/2782010950 >> v1:172.21.15.83:6790/0 conn(0x7fd0281042a0 legacy=0x7fd028106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- 172.21.15.83:0/2782010950 shutdown_connections 2024-09-10T19:01:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- 172.21.15.83:0/2782010950 >> 172.21.15.83:0/2782010950 conn(0x7fd0280f7070 msgr2=0x7fd0280f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- 172.21.15.83:0/2782010950 shutdown_connections 2024-09-10T19:01:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- 172.21.15.83:0/2782010950 wait complete. 2024-09-10T19:01:14.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 librados: starting msgr at 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 librados: starting objecter 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 Processor -- start 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- start start 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 librados: setting wanted keys 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 librados: calling monclient init 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd02811b280 con 0x7fd0281042a0 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd02811c420 con 0x7fd028107e00 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd02e3b3640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd02811d5c0 con 0x7fd028101530 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd027fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd028101530 0x7fd028106f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52726/0 (socket says 172.21.15.83:52726) 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd027fff640 1 -- 172.21.15.83:0/3680034989 learned_addr learned my addr 172.21.15.83:0/3680034989 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:14.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter ms_handle_connect 0x7fd028101530 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter ms_handle_connect 0x7fd0281042a0 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter ms_handle_connect 0x7fd028107e00 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849960463 0 0) 0x7fd02811d5c0 con 0x7fd028101530 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0100034e0 con 0x7fd028101530 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367040615 0 0) 0x7fd02811b280 con 0x7fd0281042a0 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.590+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02811d5c0 con 0x7fd0281042a0 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858005738 0 0) 0x7fd02811c420 con 0x7fd028107e00 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd02811b280 con 0x7fd028107e00 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668787357 0 0) 0x7fd0100034e0 con 0x7fd028101530 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd02811c420 con 0x7fd028101530 2024-09-10T19:01:14.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd018001980 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055744009 0 0) 0x7fd02811d5c0 con 0x7fd0281042a0 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0100034e0 con 0x7fd0281042a0 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd01c001430 con 0x7fd0281042a0 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 588205496 0 0) 0x7fd02811c420 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 >> v1:172.21.15.165:6789/0 conn(0x7fd028107e00 legacy=0x7fd028119b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 >> v1:172.21.15.83:6789/0 conn(0x7fd0281042a0 legacy=0x7fd028116430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02811e760 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02811b470 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd02e3b3640 10 client.49454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02811b8c0 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd02e3b3640 1 librados: init done 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd018002d00 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd018005ac0 con 0x7fd028101530 2024-09-10T19:01:14.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014005040 con 0x7fd028101530 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd018005d50 con 0x7fd028101530 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(482..482 src has 69..482) v4 ==== 8004+0+0 (unknown 3666674319 0 0) 0x7fd01808a4e0 con 0x7fd028101530 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 10 client.49454.objecter ms_dispatch 0x7fd028000b90 osd_map(482..482 src has 69..482) v4 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 3 client.49454.objecter handle_osd_map got epochs [482,482] > 0 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.594+0000 7fd024ff9640 3 client.49454.objecter handle_osd_map decoding full epoch 482 2024-09-10T19:01:14.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.598+0000 7fd024ff9640 20 client.49454.objecter dump_active .. 0 homeless 2024-09-10T19:01:14.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.598+0000 7fd024ff9640 10 client.49454.objecter ms_handle_connect 0x7fd0100716e0 2024-09-10T19:01:14.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.602+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd018056aa0 con 0x7fd028101530 2024-09-10T19:01:14.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.663 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:14.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.798 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:14.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:14.794+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fd0140052f0 con 0x7fd028101530 2024-09-10T19:01:14.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:14.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:15.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:15.062 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:15.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:15.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:15.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.246+0000 7fd024ff9640 1 -- 172.21.15.83:0/3680034989 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v483) v1 ==== 91+0+0 (unknown 2730579671 0 0) 0x7fd018056440 con 0x7fd028101530 2024-09-10T19:01:15.253 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 10 librados: watch_flush enter 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 10 librados: watch_flush exit 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 20 client.49454.objecter shutdown clearing up homeless session... 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 10 client.49454.objecter successfully canceled tick 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd0100716e0 legacy=0x7fd010073b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:15.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.250+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 >> v1:172.21.15.83:6790/0 conn(0x7fd028101530 legacy=0x7fd028106f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:15.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.254+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 shutdown_connections 2024-09-10T19:01:15.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.254+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 >> 172.21.15.83:0/3680034989 conn(0x7fd0280f7070 msgr2=0x7fd0280f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:15.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.254+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 shutdown_connections 2024-09-10T19:01:15.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.254+0000 7fd02e3b3640 1 -- 172.21.15.83:0/3680034989 wait complete. 2024-09-10T19:01:15.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:15.254+0000 7fd02e3b3640 1 librados: shutdown 2024-09-10T19:01:15.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:15.332 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:15.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:15.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[18629]: cluster 2024-09-10T19:01:14.135121+0000 mgr.y (mgr.24526) 482 : cluster [DBG] pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[18629]: audit 2024-09-10T19:01:14.566085+0000 mon.c (mon.2) 639 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[18629]: audit 2024-09-10T19:01:14.800588+0000 mon.c (mon.2) 640 : audit [INF] from='client.? v1:172.21.15.83:0/3680034989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[18629]: audit 2024-09-10T19:01:14.800925+0000 mon.a (mon.0) 2697 : audit [INF] from='client.49454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[32381]: cluster 2024-09-10T19:01:14.135121+0000 mgr.y (mgr.24526) 482 : cluster [DBG] pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[32381]: audit 2024-09-10T19:01:14.566085+0000 mon.c (mon.2) 639 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[32381]: audit 2024-09-10T19:01:14.800588+0000 mon.c (mon.2) 640 : audit [INF] from='client.? v1:172.21.15.83:0/3680034989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.449 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:15 smithi083 bash[32381]: audit 2024-09-10T19:01:14.800925+0000 mon.a (mon.0) 2697 : audit [INF] from='client.49454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.466 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:15.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:15.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:15 smithi165 bash[22591]: cluster 2024-09-10T19:01:14.135121+0000 mgr.y (mgr.24526) 482 : cluster [DBG] pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:15.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:15 smithi165 bash[22591]: audit 2024-09-10T19:01:14.566085+0000 mon.c (mon.2) 639 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:15.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:15 smithi165 bash[22591]: audit 2024-09-10T19:01:14.800588+0000 mon.c (mon.2) 640 : audit [INF] from='client.? v1:172.21.15.83:0/3680034989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.579 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:15 smithi165 bash[22591]: audit 2024-09-10T19:01:14.800925+0000 mon.a (mon.0) 2697 : audit [INF] from='client.49454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:01:15.593 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:15.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:15.728 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:15.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:15.865 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:15.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:15.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:16.004 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:16.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:16.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:16.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:16.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:16.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:16.386 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:16 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.240854+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:16.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:16 smithi165 bash[22591]: audit 2024-09-10T19:01:15.251683+0000 mon.a (mon.0) 2699 : audit [INF] from='client.49454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:01:16.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:16 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.251858+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:01:16.408 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:16.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:16.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.240854+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:16.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[18629]: audit 2024-09-10T19:01:15.251683+0000 mon.a (mon.0) 2699 : audit [INF] from='client.49454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:01:16.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.251858+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:01:16.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.240854+0000 mon.a (mon.0) 2698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:16.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[32381]: audit 2024-09-10T19:01:15.251683+0000 mon.a (mon.0) 2699 : audit [INF] from='client.49454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:01:16.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:16 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.251858+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:01:16.546 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:16.650 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:16.685 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:16.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:16.820 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:16.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:16.960 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:16.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:17.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:17.097 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:17.201 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:17.236 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:17.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:17.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:17.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.544073+0000 osd.0 (osd.0) 250 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.552102+0000 osd.0 (osd.0) 251 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.855646+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:15.858904+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.137427+0000 mgr.y (mgr.24526) 483 : cluster [DBG] pgmap v727: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.365438+0000 osd.2 (osd.2) 199 : cluster [DBG] 12.a scrub starts 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.368975+0000 osd.2 (osd.2) 200 : cluster [DBG] 12.a scrub ok 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.517464+0000 osd.5 (osd.5) 251 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.522439+0000 osd.5 (osd.5) 252 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.763884+0000 osd.6 (osd.6) 215 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:01:17.483 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[18629]: cluster 2024-09-10T19:01:16.768095+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.544073+0000 osd.0 (osd.0) 250 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.552102+0000 osd.0 (osd.0) 251 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.855646+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:15.858904+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.137427+0000 mgr.y (mgr.24526) 483 : cluster [DBG] pgmap v727: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.365438+0000 osd.2 (osd.2) 199 : cluster [DBG] 12.a scrub starts 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.368975+0000 osd.2 (osd.2) 200 : cluster [DBG] 12.a scrub ok 2024-09-10T19:01:17.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.517464+0000 osd.5 (osd.5) 251 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:01:17.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.522439+0000 osd.5 (osd.5) 252 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:01:17.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.763884+0000 osd.6 (osd.6) 215 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:01:17.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:17 smithi083 bash[32381]: cluster 2024-09-10T19:01:16.768095+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:01:17.511 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.544073+0000 osd.0 (osd.0) 250 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.552102+0000 osd.0 (osd.0) 251 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.855646+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:15.858904+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.137427+0000 mgr.y (mgr.24526) 483 : cluster [DBG] pgmap v727: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.365438+0000 osd.2 (osd.2) 199 : cluster [DBG] 12.a scrub starts 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.368975+0000 osd.2 (osd.2) 200 : cluster [DBG] 12.a scrub ok 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.517464+0000 osd.5 (osd.5) 251 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:01:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.522439+0000 osd.5 (osd.5) 252 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:01:17.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.763884+0000 osd.6 (osd.6) 215 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:01:17.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:17 smithi165 bash[22591]: cluster 2024-09-10T19:01:16.768095+0000 osd.6 (osd.6) 216 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:01:17.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:17.650 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:17.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:17.787 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:17.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:17.927 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:17.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:18.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:18.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:18.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:18.203 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:18.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:18.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[18629]: cluster 2024-09-10T19:01:17.505629+0000 osd.5 (osd.5) 253 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:01:18.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[18629]: cluster 2024-09-10T19:01:17.509170+0000 osd.5 (osd.5) 254 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:01:18.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.065565+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub starts 2024-09-10T19:01:18.344 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.068808+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.c scrub ok 2024-09-10T19:01:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:18.345 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[32381]: cluster 2024-09-10T19:01:17.505629+0000 osd.5 (osd.5) 253 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:01:18.345 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[32381]: cluster 2024-09-10T19:01:17.509170+0000 osd.5 (osd.5) 254 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:01:18.346 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.065565+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub starts 2024-09-10T19:01:18.346 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:18 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.068808+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.c scrub ok 2024-09-10T19:01:18.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:18 smithi165 bash[22591]: cluster 2024-09-10T19:01:17.505629+0000 osd.5 (osd.5) 253 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:01:18.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:18 smithi165 bash[22591]: cluster 2024-09-10T19:01:17.509170+0000 osd.5 (osd.5) 254 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:01:18.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:18 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.065565+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.c scrub starts 2024-09-10T19:01:18.459 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:18 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.068808+0000 osd.4 (osd.4) 291 : cluster [DBG] 4.c scrub ok 2024-09-10T19:01:18.482 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:18.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:18.620 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:18.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.757 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:18.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.892 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:18.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:18.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:19.029 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:19.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:19.167 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:19.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:19.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:19.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:19.432 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:19.534 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: audit 2024-09-10T19:01:18.035585+0000 mgr.y (mgr.24526) 484 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:19.534 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.138428+0000 mgr.y (mgr.24526) 485 : cluster [DBG] pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.502353+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.506093+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.065153+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.068724+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.170736+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: audit 2024-09-10T19:01:18.035585+0000 mgr.y (mgr.24526) 484 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.138428+0000 mgr.y (mgr.24526) 485 : cluster [DBG] pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.502353+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.506093+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.065153+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:01:19.535 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.068724+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:01:19.536 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:19 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.170736+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:01:19.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:19.559 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: audit 2024-09-10T19:01:18.035585+0000 mgr.y (mgr.24526) 484 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.138428+0000 mgr.y (mgr.24526) 485 : cluster [DBG] pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.502353+0000 osd.5 (osd.5) 255 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.506093+0000 osd.5 (osd.5) 256 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.065153+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.068724+0000 osd.4 (osd.4) 293 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:01:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:19 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.170736+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:01:19.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:19.685 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:19.701 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:01:19.702 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:01:19.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.766+0000 7f5e4e787640 1 -- 172.21.15.83:0/1746556031 >> v1:172.21.15.83:6790/0 conn(0x7f5e48100c70 legacy=0x7f5e48101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.766+0000 7f5e4e787640 1 -- 172.21.15.83:0/1746556031 shutdown_connections 2024-09-10T19:01:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.766+0000 7f5e4e787640 1 -- 172.21.15.83:0/1746556031 >> 172.21.15.83:0/1746556031 conn(0x7f5e48074a20 msgr2=0x7f5e480f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.766+0000 7f5e4e787640 1 -- 172.21.15.83:0/1746556031 shutdown_connections 2024-09-10T19:01:19.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.766+0000 7f5e4e787640 1 -- 172.21.15.83:0/1746556031 wait complete. 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 librados: starting msgr at 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 librados: starting objecter 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 Processor -- start 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- start start 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 librados: setting wanted keys 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 librados: calling monclient init 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e4811e1f0 con 0x7f5e48103a50 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e4811f3f0 con 0x7f5e481075b0 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5e481205f0 con 0x7f5e48100c70 2024-09-10T19:01:19.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e47fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5e48100c70 0x7f5e481000b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:52752/0 (socket says 172.21.15.83:52752) 2024-09-10T19:01:19.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e47fff640 1 -- 172.21.15.83:0/576851639 learned_addr learned my addr 172.21.15.83:0/576851639 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:19.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e477fe640 1 --1- 172.21.15.83:0/576851639 >> v1:172.21.15.83:6789/0 conn(0x7f5e48103a50 0x7f5e48118fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:57128/0 (socket says 172.21.15.83:57128) 2024-09-10T19:01:19.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter ms_handle_connect 0x7f5e48100c70 2024-09-10T19:01:19.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:19.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter ms_handle_connect 0x7f5e48103a50 2024-09-10T19:01:19.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter ms_handle_connect 0x7f5e481075b0 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758218399 0 0) 0x7f5e481205f0 con 0x7f5e48100c70 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e30003200 con 0x7f5e48100c70 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152158440 0 0) 0x7f5e4811e1f0 con 0x7f5e48103a50 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e481205f0 con 0x7f5e48103a50 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604051755 0 0) 0x7f5e4811f3f0 con 0x7f5e481075b0 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4811e1f0 con 0x7f5e481075b0 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381444633 0 0) 0x7f5e481205f0 con 0x7f5e48103a50 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e4811f3f0 con 0x7f5e48103a50 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e3c0018a0 con 0x7f5e48103a50 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 755378328 0 0) 0x7f5e30003200 con 0x7f5e48100c70 2024-09-10T19:01:19.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e481205f0 con 0x7f5e48100c70 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e38002d10 con 0x7f5e48100c70 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732107291 0 0) 0x7f5e4811e1f0 con 0x7f5e481075b0 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e30003200 con 0x7f5e481075b0 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 903693302 0 0) 0x7f5e4811f3f0 con 0x7f5e48103a50 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 >> v1:172.21.15.83:6790/0 conn(0x7f5e48100c70 legacy=0x7f5e481000b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 >> v1:172.21.15.165:6789/0 conn(0x7f5e481075b0 legacy=0x7f5e4811c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e481217f0 con 0x7f5e48103a50 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4811e3e0 con 0x7f5e48103a50 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 10 client.25389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4811e830 con 0x7f5e48103a50 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e4e787640 1 librados: init done 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5e3c001f60 con 0x7f5e48103a50 2024-09-10T19:01:19.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.770+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5e3c0059a0 con 0x7f5e48103a50 2024-09-10T19:01:19.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34005010 con 0x7f5e48103a50 2024-09-10T19:01:19.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5e3c001b00 con 0x7f5e48103a50 2024-09-10T19:01:19.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(484..484 src has 69..484) v4 ==== 8004+0+0 (unknown 3275483595 0 0) 0x7f5e3c08a350 con 0x7f5e48103a50 2024-09-10T19:01:19.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 10 client.25389.objecter ms_dispatch 0x7f5e48000b90 osd_map(484..484 src has 69..484) v4 2024-09-10T19:01:19.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 3 client.25389.objecter handle_osd_map got epochs [484,484] > 0 2024-09-10T19:01:19.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 3 client.25389.objecter handle_osd_map decoding full epoch 484 2024-09-10T19:01:19.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 20 client.25389.objecter dump_active .. 0 homeless 2024-09-10T19:01:19.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.774+0000 7f5e44ff9640 10 client.25389.objecter ms_handle_connect 0x7f5e30071970 2024-09-10T19:01:19.787 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:19.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.782+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5e3c056910 con 0x7f5e48103a50 2024-09-10T19:01:19.811 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:19.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:19.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:19.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.966+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5e340052c0 con 0x7f5e48103a50 2024-09-10T19:01:19.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e44ff9640 1 -- 172.21.15.83:0/576851639 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v484) v1 ==== 81+0+47584 (unknown 3230229014 0 2341648163) 0x7f5e3c0562b0 con 0x7f5e48103a50 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: "epoch": 484, 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: "fsid": "707a4ada-6fa3-11ef-bcea-c7b262605968", 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: "created": "2024-09-10T18:38:38.475667+0000", 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: "modified": "2024-09-10T19:01:19.158759+0000", 2024-09-10T19:01:19.976 INFO:teuthology.orchestra.run.smithi083.stdout: "last_up_change": "2024-09-10T18:48:34.513026+0000", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "last_in_change": "2024-09-10T18:48:07.544682+0000", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_num": 5804032, 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_set": [ 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "nodeep-scrub", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "pglog_hardlimit", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "purged_snapdirs", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_deletes", 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "sortbitwise" 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_version": 110, 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:01:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_snapshot": "", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_max": 219, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "max_osd": 8, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "min_compat_client": "luminous", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "require_osd_release": "quincy", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "pools": [ 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 1, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": ".mgr", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:44:06.952078+0000", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:19.978 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 1, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 1, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 1, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 1, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 1, 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:19.979 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "467", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:19.980 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:19.981 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:19.982 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "options": { 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_max": 32, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_min": 1 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "mgr": {} 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 2, 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": ".rgw.root", 2024-09-10T19:01:19.983 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:49:16.239173+0000", 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:19.984 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "71", 2024-09-10T19:01:19.985 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:19.986 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:19.987 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:19.988 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": {} 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 3, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:49:18.096790+0000", 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:19.989 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:19.990 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "73", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:19.991 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:19.992 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:19.993 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": {} 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 4, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:49:20.070555+0000", 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:19.994 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:19.995 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "75", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:19.996 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:19.997 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:19.998 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": {} 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 5, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "datapool", 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:49:20.615000+0000", 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 8193, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:19.999 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 3, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 3, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 3, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 3, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 3, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:20.000 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "80", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 3, 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 80, 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:20.001 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:20.002 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:20.003 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "rbd": {} 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 6, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:49:22.582739+0000", 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:20.004 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:20.005 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "77", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:20.006 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:20.007 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:20.008 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "options": { 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_bias": 4 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "rgw": {} 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 12, 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:54:28.259368+0000", 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:20.009 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:20.010 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "128", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:20.011 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:20.012 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:20.013 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": {} 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 64, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "test-rados-api-smithi083-77828-6", 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T18:54:55.217195+0000", 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 8193, 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:01:20.014 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:20.015 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "480", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "421", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "421", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "421", 2024-09-10T19:01:20.016 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 36, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 480, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:20.017 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:20.018 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "options": { 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "fingerprint_algorithm": "sha1" 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: "rados": {} 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.019 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 219, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_name": "test-rados-api-smithi083-77828-71", 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "create_time": "2024-09-10T19:01:06.618978+0000", 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "flags": 1, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "flags_names": "hashpspool", 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "type": 1, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 3, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "min_size": 2, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_rule": 0, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:01:20.020 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "object_hash": 2, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num": 32, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num": 32, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_placement_num_target": 32, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_target": 32, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_num_pending": 32, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "last_pg_merge_meta": { 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "source_pgid": "0.0", 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "ready_epoch": 0, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_started": 0, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "last_epoch_clean": 0, 2024-09-10T19:01:20.021 INFO:teuthology.orchestra.run.smithi083.stdout: "source_version": "0'0", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "target_version": "0'0" 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "last_change": "479", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend": "0", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "auid": 0, 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_seq": 0, 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "snap_epoch": 0, 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "pool_snaps": [], 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps": "[]", 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_bytes": 0, 2024-09-10T19:01:20.022 INFO:teuthology.orchestra.run.smithi083.stdout: "quota_max_objects": 0, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "tiers": [], 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "tier_of": -1, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "read_tier": -1, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "write_tier": -1, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_mode": "none", 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_bytes": 0, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "target_max_objects": 0, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_flush_age": 0, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "cache_min_evict_age": 0, 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profile": "", 2024-09-10T19:01:20.023 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_params": { 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "none" 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_period": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_count": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "use_gmt_hitset": true, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "grade_table": [], 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "stripe_width": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "expected_num_objects": 0, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "fast_read": false, 2024-09-10T19:01:20.024 INFO:teuthology.orchestra.run.smithi083.stdout: "options": {}, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "application_metadata": { 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "rados": {} 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "osds": [ 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 0, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "35ab8328-74f1-4af8-aa84-797914199445", 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.025 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 8, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6801", 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2535527483 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.026 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6802", 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2535527483 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6804", 2024-09-10T19:01:20.027 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2535527483 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6803", 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2535527483 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6801/2535527483", 2024-09-10T19:01:20.028 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.83:6802/2535527483", 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.83:6804/2535527483", 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.83:6803/2535527483", 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 1, 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "8e1d1072-767b-49fb-b2f2-fce60263e2ad", 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.029 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 13, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6805", 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2309624637 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.030 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6806", 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2309624637 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.031 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6808", 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2309624637 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6807", 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 2309624637 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.032 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6805/2309624637", 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.83:6806/2309624637", 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.83:6808/2309624637", 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.83:6807/2309624637", 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 2, 2024-09-10T19:01:20.033 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "6193b508-03d9-458a-b49f-6b2e24283282", 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 19, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.034 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6809", 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 386094220 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6810", 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 386094220 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.035 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6812", 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 386094220 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.036 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6811", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 386094220 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6809/386094220", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.83:6810/386094220", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.83:6812/386094220", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.83:6811/386094220", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.037 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 3, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "a28f94ff-2fff-458e-8b69-c5c65d720d6c", 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 29, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.038 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6813", 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1279688021 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6814", 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1279688021 2024-09-10T19:01:20.039 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6816", 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1279688021 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.040 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.83:6815", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1279688021 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.83:6813/1279688021", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.83:6814/1279688021", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.83:6816/1279688021", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.83:6815/1279688021", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.041 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 4, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b", 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 35, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 474, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.042 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6800", 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3225515265 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6801", 2024-09-10T19:01:20.043 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3225515265 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6803", 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3225515265 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.044 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6802", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 3225515265 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.165:6800/3225515265", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.165:6801/3225515265", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.165:6803/3225515265", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.165:6802/3225515265", 2024-09-10T19:01:20.045 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 5, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "10543223-db5a-493b-88f3-8705f2b31a8e", 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.046 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 43, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6804", 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 571173777 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.047 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6805", 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 571173777 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6807", 2024-09-10T19:01:20.048 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 571173777 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6806", 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 571173777 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.049 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.165:6804/571173777", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.165:6805/571173777", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.165:6807/571173777", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.165:6806/571173777", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 6, 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "fcea0ca4-4521-4161-b407-9b1fab1a2ae2", 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.050 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 52, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6808", 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1317922936 2024-09-10T19:01:20.051 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6809", 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1317922936 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.052 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6811", 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1317922936 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6810", 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1317922936 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.053 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.165:6808/1317922936", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.165:6809/1317922936", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.165:6811/1317922936", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.165:6810/1317922936", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 7, 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "uuid": "8531d286-5ebb-4c96-a026-63d8a17cb21f", 2024-09-10T19:01:20.054 INFO:teuthology.orchestra.run.smithi083.stdout: "up": 1, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "in": 1, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "weight": 1, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_affinity": 1, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_begin": 0, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "last_clean_end": 0, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "up_from": 62, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "up_thru": 477, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "down_at": 0, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "lost_at": 0, 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addrs": { 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6812", 2024-09-10T19:01:20.055 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1610151650 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addrs": { 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6813", 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1610151650 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.056 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addrs": { 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6815", 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1610151650 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addrs": { 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "addrvec": [ 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "type": "v1", 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "addr": "172.21.15.165:6814", 2024-09-10T19:01:20.057 INFO:teuthology.orchestra.run.smithi083.stdout: "nonce": 1610151650 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "public_addr": "172.21.15.165:6812/1610151650", 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "cluster_addr": "172.21.15.165:6813/1610151650", 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1610151650", 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1610151650", 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "state": [ 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "exists", 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "up" 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_xinfo": [ 2024-09-10T19:01:20.058 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 0, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:59.590021+0000", 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 1, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.059 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:57.294565+0000", 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 2, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:39.740354+0000", 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.060 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 3, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:16.328804+0000", 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 4, 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.061 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:46.302822+0000", 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 5, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:00:48.176374+0000", 2024-09-10T19:01:20.062 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 6, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:01:12.175938+0000", 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "osd": 7, 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "down_stamp": "0.000000", 2024-09-10T19:01:20.063 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_probability": 0, 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "laggy_interval": 0, 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "features": 4540138320759226367, 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "old_weight": 0, 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "last_purged_snaps_scrub": "2024-09-10T19:01:02.260088+0000", 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "dead_epoch": 0 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_upmap": [], 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_upmap_items": [ 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "pgid": "2.f", 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: "mappings": [ 2024-09-10T19:01:20.064 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "from": 7, 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "to": 2 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "pgid": "4.12", 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "mappings": [ 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "from": 1, 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: "to": 5 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.065 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "pgid": "4.18", 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "mappings": [ 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "from": 1, 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "to": 2 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "pgid": "4.1f", 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "mappings": [ 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.066 INFO:teuthology.orchestra.run.smithi083.stdout: "from": 1, 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "to": 2 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "pg_temp": [], 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "primary_temp": [], 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "blocklist": { 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/3645093631": "2024-09-11T18:50:14.421453+0000", 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:6800/3275901971": "2024-09-11T18:39:06.336264+0000", 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/3555551632": "2024-09-11T18:39:40.343252+0000", 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/3076437878": "2024-09-11T18:39:06.336264+0000", 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/4112054239": "2024-09-11T18:39:06.336264+0000", 2024-09-10T19:01:20.067 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:6800/2452184936": "2024-09-11T18:39:40.343252+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/2659452540": "2024-09-11T18:39:40.343252+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/867303943": "2024-09-11T18:39:06.336264+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/1156549307": "2024-09-11T18:39:40.343252+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/1687990153": "2024-09-11T18:50:14.421453+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/1246256198": "2024-09-11T18:50:14.421453+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/3225538750": "2024-09-11T18:50:14.421453+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:6800/2630800463": "2024-09-11T18:50:14.421453+0000", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "172.21.15.83:0/1423545952": "2024-09-11T18:50:14.421453+0000" 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "range_blocklist": {}, 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "erasure_code_profiles": { 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "default": { 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "crush-failure-domain": "osd", 2024-09-10T19:01:20.068 INFO:teuthology.orchestra.run.smithi083.stdout: "k": "2", 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "m": "1", 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "plugin": "jerasure", 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "technique": "reed_sol_van" 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "removed_snaps_queue": [ 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 22, 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 5, 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 7, 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 23, 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 9, 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 60, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 2, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 6, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 80, 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.071 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 5, 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 7, 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 88, 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.072 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 2, 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 95, 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 6, 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.073 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 97, 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 4, 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 4 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 108, 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.074 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 2, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 113, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 1, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 3, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.075 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 5, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 7, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 128, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 2, 2024-09-10T19:01:20.076 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 6, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 10, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 136, 2024-09-10T19:01:20.077 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 12, 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 156, 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 14, 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.078 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 166, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 17, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 171, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 19, 2024-09-10T19:01:20.079 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 2 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: "pool": 183, 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: "snaps": [ 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: "begin": 24, 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: "length": 1 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: ], 2024-09-10T19:01:20.080 INFO:teuthology.orchestra.run.smithi083.stdout: "new_removed_snaps": [], 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "new_purged_snaps": [], 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "crush_node_flags": {}, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "device_class_flags": {}, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode": { 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode_enabled": false, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_bucket_count": 0, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: "stretch_mode_bucket": 0 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:01:20.081 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:01:20.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 10 librados: watch_flush enter 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 10 librados: watch_flush exit 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 20 client.25389.objecter shutdown clearing up homeless session... 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 10 client.25389.objecter successfully canceled tick 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5e30071970 legacy=0x7f5e30073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.970+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 >> v1:172.21.15.83:6789/0 conn(0x7f5e48103a50 legacy=0x7f5e48118fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.974+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 shutdown_connections 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.974+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 >> 172.21.15.83:0/576851639 conn(0x7f5e48074a20 msgr2=0x7f5e480f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.974+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 shutdown_connections 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.974+0000 7f5e4e787640 1 -- 172.21.15.83:0/576851639 wait complete. 2024-09-10T19:01:20.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:19.974+0000 7f5e4e787640 1 librados: shutdown 2024-09-10T19:01:20.120 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:20.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:20.257 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:20.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:20 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.856784+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:01:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:20 smithi165 bash[22591]: cluster 2024-09-10T19:01:18.860086+0000 osd.1 (osd.1) 270 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:01:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:20 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.260446+0000 osd.7 (osd.7) 268 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:01:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:20 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.268311+0000 osd.7 (osd.7) 269 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:01:20.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:20 smithi165 bash[22591]: audit 2024-09-10T19:01:19.973040+0000 mon.a (mon.0) 2702 : audit [DBG] from='client.? v1:172.21.15.83:0/576851639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:01:20.393 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:20.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:20.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.856784+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[18629]: cluster 2024-09-10T19:01:18.860086+0000 osd.1 (osd.1) 270 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.260446+0000 osd.7 (osd.7) 268 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.268311+0000 osd.7 (osd.7) 269 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[18629]: audit 2024-09-10T19:01:19.973040+0000 mon.a (mon.0) 2702 : audit [DBG] from='client.? v1:172.21.15.83:0/576851639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.856784+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:01:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[32381]: cluster 2024-09-10T19:01:18.860086+0000 osd.1 (osd.1) 270 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:01:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.260446+0000 osd.7 (osd.7) 268 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:01:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.268311+0000 osd.7 (osd.7) 269 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:01:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:20 smithi083 bash[32381]: audit 2024-09-10T19:01:19.973040+0000 mon.a (mon.0) 2702 : audit [DBG] from='client.? v1:172.21.15.83:0/576851639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:01:20.560 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:01:20.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:20.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:20.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:20.803 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:20.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:20.937 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:20.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:20.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:01:21.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:21.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:21.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:21.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:21.340 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:21.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:21.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.954095+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.b scrub starts 2024-09-10T19:01:21.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:19.956884+0000 osd.3 (osd.3) 281 : cluster [DBG] 6.b scrub ok 2024-09-10T19:01:21.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:20.139301+0000 mgr.y (mgr.24526) 486 : cluster [DBG] pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:21.455 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:20.308533+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:20.312112+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.075558+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.e scrub starts 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.078604+0000 osd.4 (osd.4) 295 : cluster [DBG] 6.e scrub ok 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.954095+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.b scrub starts 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:19.956884+0000 osd.3 (osd.3) 281 : cluster [DBG] 6.b scrub ok 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:20.139301+0000 mgr.y (mgr.24526) 486 : cluster [DBG] pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:20.308533+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:20.312112+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.075558+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.e scrub starts 2024-09-10T19:01:21.456 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:21 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.078604+0000 osd.4 (osd.4) 295 : cluster [DBG] 6.e scrub ok 2024-09-10T19:01:21.475 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:21.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:21.592 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.954095+0000 osd.3 (osd.3) 280 : cluster [DBG] 6.b scrub starts 2024-09-10T19:01:21.592 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:19.956884+0000 osd.3 (osd.3) 281 : cluster [DBG] 6.b scrub ok 2024-09-10T19:01:21.592 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:20.139301+0000 mgr.y (mgr.24526) 486 : cluster [DBG] pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:01:21.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:20.308533+0000 osd.7 (osd.7) 270 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:01:21.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:20.312112+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:01:21.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.075558+0000 osd.4 (osd.4) 294 : cluster [DBG] 6.e scrub starts 2024-09-10T19:01:21.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.078604+0000 osd.4 (osd.4) 295 : cluster [DBG] 6.e scrub ok 2024-09-10T19:01:21.593 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:21.614 INFO:tasks.thrashosds.thrasher:j is {'epoch': 484, 'fsid': '707a4ada-6fa3-11ef-bcea-c7b262605968', 'created': '2024-09-10T18:38:38.475667+0000', 'modified': '2024-09-10T19:01:19.158759+0000', 'last_up_change': '2024-09-10T18:48:34.513026+0000', 'last_in_change': '2024-09-10T18:48:07.544682+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 110, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 219, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:44:06.952078+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '467', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T18:49:16.239173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T18:49:18.096790+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T18:49:20.070555+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'datapool', 'create_time': '2024-09-10T18:49:20.615000+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 80, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T18:49:22.582739+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 12, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-09-10T18:54:28.259368+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 64, 'pool_name': 'test-rados-api-smithi083-77828-6', 'create_time': '2024-09-10T18:54:55.217195+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '480', 'last_force_op_resend': '421', 'last_force_op_resend_prenautilus': '421', 'last_force_op_resend_preluminous': '421', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 36, 'snap_epoch': 480, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}, {'pool': 219, 'pool_name': 'test-rados-api-smithi083-77828-71', 'create_time': '2024-09-10T19:01:06.618978+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '479', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '35ab8328-74f1-4af8-aa84-797914199445', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6801', 'nonce': 2535527483}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6802', 'nonce': 2535527483}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6804', 'nonce': 2535527483}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6803', 'nonce': 2535527483}]}, 'public_addr': '172.21.15.83:6801/2535527483', 'cluster_addr': '172.21.15.83:6802/2535527483', 'heartbeat_back_addr': '172.21.15.83:6804/2535527483', 'heartbeat_front_addr': '172.21.15.83:6803/2535527483', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8e1d1072-767b-49fb-b2f2-fce60263e2ad', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6805', 'nonce': 2309624637}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6806', 'nonce': 2309624637}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6808', 'nonce': 2309624637}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6807', 'nonce': 2309624637}]}, 'public_addr': '172.21.15.83:6805/2309624637', 'cluster_addr': '172.21.15.83:6806/2309624637', 'heartbeat_back_addr': '172.21.15.83:6808/2309624637', 'heartbeat_front_addr': '172.21.15.83:6807/2309624637', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6193b508-03d9-458a-b49f-6b2e24283282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6809', 'nonce': 386094220}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6810', 'nonce': 386094220}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6812', 'nonce': 386094220}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6811', 'nonce': 386094220}]}, 'public_addr': '172.21.15.83:6809/386094220', 'cluster_addr': '172.21.15.83:6810/386094220', 'heartbeat_back_addr': '172.21.15.83:6812/386094220', 'heartbeat_front_addr': '172.21.15.83:6811/386094220', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a28f94ff-2fff-458e-8b69-c5c65d720d6c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 29, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6813', 'nonce': 1279688021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6814', 'nonce': 1279688021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6816', 'nonce': 1279688021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.83:6815', 'nonce': 1279688021}]}, 'public_addr': '172.21.15.83:6813/1279688021', 'cluster_addr': '172.21.15.83:6814/1279688021', 'heartbeat_back_addr': '172.21.15.83:6816/1279688021', 'heartbeat_front_addr': '172.21.15.83:6815/1279688021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 474, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 3225515265}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 3225515265}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 3225515265}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 3225515265}]}, 'public_addr': '172.21.15.165:6800/3225515265', 'cluster_addr': '172.21.15.165:6801/3225515265', 'heartbeat_back_addr': '172.21.15.165:6803/3225515265', 'heartbeat_front_addr': '172.21.15.165:6802/3225515265', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '10543223-db5a-493b-88f3-8705f2b31a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 43, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 571173777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 571173777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 571173777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 571173777}]}, 'public_addr': '172.21.15.165:6804/571173777', 'cluster_addr': '172.21.15.165:6805/571173777', 'heartbeat_back_addr': '172.21.15.165:6807/571173777', 'heartbeat_front_addr': '172.21.15.165:6806/571173777', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'fcea0ca4-4521-4161-b407-9b1fab1a2ae2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 1317922936}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 1317922936}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 1317922936}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 1317922936}]}, 'public_addr': '172.21.15.165:6808/1317922936', 'cluster_addr': '172.21.15.165:6809/1317922936', 'heartbeat_back_addr': '172.21.15.165:6811/1317922936', 'heartbeat_front_addr': '172.21.15.165:6810/1317922936', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '8531d286-5ebb-4c96-a026-63d8a17cb21f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 62, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1610151650}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1610151650}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1610151650}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1610151650}]}, 'public_addr': '172.21.15.165:6812/1610151650', 'cluster_addr': '172.21.15.165:6813/1610151650', 'heartbeat_back_addr': '172.21.15.165:6815/1610151650', 'heartbeat_front_addr': '172.21.15.165:6814/1610151650', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:59.590021+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:57.294565+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:39.740354+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:16.328804+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:46.302822+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:00:48.176374+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:01:12.175938+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:01:02.260088+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.f', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.18', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.83:0/3645093631': '2024-09-11T18:50:14.421453+0000', '172.21.15.83:6800/3275901971': '2024-09-11T18:39:06.336264+0000', '172.21.15.83:0/3555551632': '2024-09-11T18:39:40.343252+0000', '172.21.15.83:0/3076437878': '2024-09-11T18:39:06.336264+0000', '172.21.15.83:0/4112054239': '2024-09-11T18:39:06.336264+0000', '172.21.15.83:6800/2452184936': '2024-09-11T18:39:40.343252+0000', '172.21.15.83:0/2659452540': '2024-09-11T18:39:40.343252+0000', '172.21.15.83:0/867303943': '2024-09-11T18:39:06.336264+0000', '172.21.15.83:0/1156549307': '2024-09-11T18:39:40.343252+0000', '172.21.15.83:0/1687990153': '2024-09-11T18:50:14.421453+0000', '172.21.15.83:0/1246256198': '2024-09-11T18:50:14.421453+0000', '172.21.15.83:0/3225538750': '2024-09-11T18:50:14.421453+0000', '172.21.15.83:6800/2630800463': '2024-09-11T18:50:14.421453+0000', '172.21.15.83:0/1423545952': '2024-09-11T18:50:14.421453+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 22, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 23, 'snaps': [{'begin': 9, 'length': 2}]}, {'pool': 60, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 80, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 88, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 95, 'snaps': [{'begin': 6, 'length': 2}]}, {'pool': 97, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 108, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 113, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 136, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 156, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 166, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 171, 'snaps': [{'begin': 19, 'length': 2}]}, {'pool': 183, 'snaps': [{'begin': 24, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:01:21.631 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-10T19:01:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.632 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:21.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:21.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:21.766 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:21.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:21.902 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:21.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:22.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:22.016 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:22.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:22.144 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:22.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:22.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:22.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:22.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:22.447 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:22.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.742983+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.746892+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.087024+0000 osd.4 (osd.4) 296 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.090471+0000 osd.4 (osd.4) 297 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.742983+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.746892+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.087024+0000 osd.4 (osd.4) 296 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:01:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.090471+0000 osd.4 (osd.4) 297 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:01:22.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:22.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:22 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.742983+0000 osd.6 (osd.6) 217 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:01:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:22 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.746892+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:01:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:22 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.087024+0000 osd.4 (osd.4) 296 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:01:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:22 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.090471+0000 osd.4 (osd.4) 297 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:01:22.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:22.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:22.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:22.861 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:22.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:22.996 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:22.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:22] "GET /metrics HTTP/1.1" 200 228883 "" "Prometheus/2.43.0" 2024-09-10T19:01:23.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:23.134 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:23.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:23.273 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:23.376 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:23.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.949567+0000 osd.3 (osd.3) 282 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:01:23.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:21.953288+0000 osd.3 (osd.3) 283 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:01:23.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.141498+0000 mgr.y (mgr.24526) 487 : cluster [DBG] pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 385 B/s wr, 3 op/s 2024-09-10T19:01:23.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.291178+0000 osd.7 (osd.7) 272 : cluster [DBG] 3.e scrub starts 2024-09-10T19:01:23.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.297242+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.e scrub ok 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.120795+0000 osd.4 (osd.4) 298 : cluster [DBG] 4.d scrub starts 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.124605+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.d scrub ok 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.949567+0000 osd.3 (osd.3) 282 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:21.953288+0000 osd.3 (osd.3) 283 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.141498+0000 mgr.y (mgr.24526) 487 : cluster [DBG] pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 385 B/s wr, 3 op/s 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.291178+0000 osd.7 (osd.7) 272 : cluster [DBG] 3.e scrub starts 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.297242+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.e scrub ok 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.120795+0000 osd.4 (osd.4) 298 : cluster [DBG] 4.d scrub starts 2024-09-10T19:01:23.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:23 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.124605+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.d scrub ok 2024-09-10T19:01:23.409 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:23.512 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:23.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.949567+0000 osd.3 (osd.3) 282 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:21.953288+0000 osd.3 (osd.3) 283 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.141498+0000 mgr.y (mgr.24526) 487 : cluster [DBG] pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 385 B/s wr, 3 op/s 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.291178+0000 osd.7 (osd.7) 272 : cluster [DBG] 3.e scrub starts 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.297242+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.e scrub ok 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.120795+0000 osd.4 (osd.4) 298 : cluster [DBG] 4.d scrub starts 2024-09-10T19:01:23.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:23 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.124605+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.d scrub ok 2024-09-10T19:01:23.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:23.685 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:23.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:23.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:23.962 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:23.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:24.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:24.101 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:24.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:24.238 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:24.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:24.354 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.569673+0000 osd.0 (osd.0) 252 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.572604+0000 osd.0 (osd.0) 253 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.934020+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:22.937621+0000 osd.3 (osd.3) 285 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.798548+0000 osd.6 (osd.6) 219 : cluster [DBG] 12.f scrub starts 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.802097+0000 osd.6 (osd.6) 220 : cluster [DBG] 12.f scrub ok 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.569673+0000 osd.0 (osd.0) 252 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.572604+0000 osd.0 (osd.0) 253 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.934020+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:22.937621+0000 osd.3 (osd.3) 285 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:01:24.355 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.798548+0000 osd.6 (osd.6) 219 : cluster [DBG] 12.f scrub starts 2024-09-10T19:01:24.356 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:24 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.802097+0000 osd.6 (osd.6) 220 : cluster [DBG] 12.f scrub ok 2024-09-10T19:01:24.372 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:24.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.569673+0000 osd.0 (osd.0) 252 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.572604+0000 osd.0 (osd.0) 253 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.934020+0000 osd.3 (osd.3) 284 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:22.937621+0000 osd.3 (osd.3) 285 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.798548+0000 osd.6 (osd.6) 219 : cluster [DBG] 12.f scrub starts 2024-09-10T19:01:24.489 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:24 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.802097+0000 osd.6 (osd.6) 220 : cluster [DBG] 12.f scrub ok 2024-09-10T19:01:24.511 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:24.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:24.641 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:24.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:24.780 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:24.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:24.916 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:24.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:25.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:25.053 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:25.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:25.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:25.293 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:25.329 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:25.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:25.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:25 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.541209+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:01:25.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:25 smithi165 bash[22591]: cluster 2024-09-10T19:01:23.545426+0000 osd.0 (osd.0) 255 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:01:25.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:25 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.142116+0000 mgr.y (mgr.24526) 488 : cluster [DBG] pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-09-10T19:01:25.464 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:25.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.541209+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[32381]: cluster 2024-09-10T19:01:23.545426+0000 osd.0 (osd.0) 255 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.142116+0000 mgr.y (mgr.24526) 488 : cluster [DBG] pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.541209+0000 osd.0 (osd.0) 254 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[18629]: cluster 2024-09-10T19:01:23.545426+0000 osd.0 (osd.0) 255 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:01:25.581 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:25 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.142116+0000 mgr.y (mgr.24526) 488 : cluster [DBG] pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 345 B/s wr, 3 op/s 2024-09-10T19:01:25.602 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:25.706 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:25.737 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:25.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:25.873 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:25.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:25.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:26.013 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:26.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.146 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:26.146 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:26.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:26.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:26.288 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:26.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.508932+0000 osd.3 (osd.3) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:26.403 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.509326+0000 osd.3 (osd.3) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.554950+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.560013+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.880290+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:24.883723+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.218057+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.221751+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.323816+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.378467+0000 osd.5 (osd.5) 257 : cluster [DBG] 64.c scrub starts 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.384781+0000 osd.5 (osd.5) 258 : cluster [DBG] 64.c scrub ok 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:26.147699+0000 osd.4 (osd.4) 300 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:01:26.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:26 smithi165 bash[22591]: cluster 2024-09-10T19:01:26.151449+0000 osd.4 (osd.4) 301 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:01:26.425 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:26.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.508932+0000 osd.3 (osd.3) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.509326+0000 osd.3 (osd.3) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.554950+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.560013+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.880290+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:24.883723+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.218057+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.221751+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.323816+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:01:26.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.378467+0000 osd.5 (osd.5) 257 : cluster [DBG] 64.c scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.384781+0000 osd.5 (osd.5) 258 : cluster [DBG] 64.c scrub ok 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:26.147699+0000 osd.4 (osd.4) 300 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[32381]: cluster 2024-09-10T19:01:26.151449+0000 osd.4 (osd.4) 301 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.508932+0000 osd.3 (osd.3) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.509326+0000 osd.3 (osd.3) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.554950+0000 osd.0 (osd.0) 256 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.560013+0000 osd.0 (osd.0) 257 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.880290+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:24.883723+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.218057+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:01:26.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.221751+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:01:26.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.323816+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:01:26.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.378467+0000 osd.5 (osd.5) 257 : cluster [DBG] 64.c scrub starts 2024-09-10T19:01:26.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.384781+0000 osd.5 (osd.5) 258 : cluster [DBG] 64.c scrub ok 2024-09-10T19:01:26.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:26.147699+0000 osd.4 (osd.4) 300 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:01:26.545 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:26 smithi083 bash[18629]: cluster 2024-09-10T19:01:26.151449+0000 osd.4 (osd.4) 301 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:01:26.563 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:26.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:01:26.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:01:26.632 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T19:01:26.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:01:26.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:26.701 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:26.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:26.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:26.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.964 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:26.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:27.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:27.102 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:27.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- 172.21.15.83:0/885193548 >> v1:172.21.15.83:6790/0 conn(0x7f5c18107470 legacy=0x7f5c18109850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- 172.21.15.83:0/885193548 shutdown_connections 2024-09-10T19:01:27.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- 172.21.15.83:0/885193548 >> 172.21.15.83:0/885193548 conn(0x7f5c180fa290 msgr2=0x7f5c180fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:27.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- 172.21.15.83:0/885193548 shutdown_connections 2024-09-10T19:01:27.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- 172.21.15.83:0/885193548 wait complete. 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 librados: starting msgr at 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 librados: starting objecter 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 Processor -- start 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- start start 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 librados: setting wanted keys 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 librados: calling monclient init 2024-09-10T19:01:27.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c1811df50 con 0x7f5c181046e0 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c1811f110 con 0x7f5c18107470 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1de5e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c18120310 con 0x7f5c1810b000 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1d65d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5c1810b000 0x7f5c1811c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49960/0 (socket says 172.21.15.83:49960) 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c1d65d640 1 -- 172.21.15.83:0/3616996159 learned_addr learned my addr 172.21.15.83:0/3616996159 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c157fa640 10 client.?.objecter ms_handle_connect 0x7f5c1810b000 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c157fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c157fa640 10 client.?.objecter ms_handle_connect 0x7f5c181046e0 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c157fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.182+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156854882 0 0) 0x7f5c18120310 con 0x7f5c1810b000 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000034e0 con 0x7f5c1810b000 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 10 client.?.objecter ms_handle_connect 0x7f5c18107470 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983827292 0 0) 0x7f5c1811df50 con 0x7f5c181046e0 2024-09-10T19:01:27.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18120310 con 0x7f5c181046e0 2024-09-10T19:01:27.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139744742 0 0) 0x7f5c1811f110 con 0x7f5c18107470 2024-09-10T19:01:27.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1811df50 con 0x7f5c18107470 2024-09-10T19:01:27.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303289149 0 0) 0x7f5c000034e0 con 0x7f5c1810b000 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c1811f110 con 0x7f5c1810b000 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5c10001c20 con 0x7f5c1810b000 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698282049 0 0) 0x7f5c18120310 con 0x7f5c181046e0 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c000034e0 con 0x7f5c181046e0 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5c08001600 con 0x7f5c181046e0 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274898729 0 0) 0x7f5c1811df50 con 0x7f5c18107470 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c18120310 con 0x7f5c18107470 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5c0c001510 con 0x7f5c18107470 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3198728438 0 0) 0x7f5c1811f110 con 0x7f5c1810b000 2024-09-10T19:01:27.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.165:6789/0 conn(0x7f5c18107470 legacy=0x7f5c18100a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6789/0 conn(0x7f5c181046e0 legacy=0x7f5c18100310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18121510 con 0x7f5c1810b000 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1811e140 con 0x7f5c1810b000 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c1de5e640 10 client.49466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1811e590 con 0x7f5c1810b000 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c1de5e640 1 librados: init done 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5c10002610 con 0x7f5c1810b000 2024-09-10T19:01:27.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5c10005ca0 con 0x7f5c1810b000 2024-09-10T19:01:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.186+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5c04004ff0 con 0x7f5c1810b000 2024-09-10T19:01:27.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5c1001c0c0 con 0x7f5c1810b000 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(485..485 src has 69..485) v4 ==== 8004+0+0 (unknown 3346187605 0 0) 0x7f5c1008a7b0 con 0x7f5c1810b000 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 osd_map(485..485 src has 69..485) v4 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 3 client.49466.objecter handle_osd_map got epochs [485,485] > 0 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 3 client.49466.objecter handle_osd_map decoding full epoch 485 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 20 client.49466.objecter dump_active .. 0 homeless 2024-09-10T19:01:27.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c000717a0 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v485) v1 ==== 54+0+16 (unknown 499309183 0 2420000192) 0x7f5c1008cb90 con 0x7f5c1810b000 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 1 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 1 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 0 1 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 4 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 10 client.49466.objecter _send_command 1 2024-09-10T19:01:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c04005290 con 0x7f5c04004610 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.190+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c04004610 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.194+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c04005290 con 0x7f5c04004610 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.194+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.194+0000 7f5c157fa640 10 client.49466.objecter _finish_command 1 = system:0 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.194+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 4 2024-09-10T19:01:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.194+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 0 1 2024-09-10T19:01:27.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 2 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 2 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04005810 osd=0 3 2024-09-10T19:01:27.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 0 2 2024-09-10T19:01:27.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 4 2024-09-10T19:01:27.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 10 client.49466.objecter _send_command 2 2024-09-10T19:01:27.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c04004cc0 con 0x7f5c04004610 2024-09-10T19:01:27.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c04004cc0 con 0x7f5c04004610 2024-09-10T19:01:27.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c157fa640 10 client.49466.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 4 2024-09-10T19:01:27.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 0 2 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 3 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 3 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 1 3 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 4 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 10 client.49466.objecter _send_command 3 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.210+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c040022b0 con 0x7f5c04004110 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c04004110 2024-09-10T19:01:27.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c040022b0 con 0x7f5c04004110 2024-09-10T19:01:27.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:01:27.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 10 client.49466.objecter _finish_command 3 = system:0 2024-09-10T19:01:27.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 4 2024-09-10T19:01:27.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.214+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 1 3 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 4 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 4 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04003fa0 osd=1 3 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 1 4 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 4 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 10 client.49466.objecter _send_command 4 2024-09-10T19:01:27.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.226+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c04001890 con 0x7f5c04004110 2024-09-10T19:01:27.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c04001890 con 0x7f5c04004110 2024-09-10T19:01:27.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c157fa640 10 client.49466.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 4 2024-09-10T19:01:27.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 1 4 2024-09-10T19:01:27.234 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:01:27.234 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 5 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 5 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 2 5 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 4 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 10 client.49466.objecter _send_command 5 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.230+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c04003df0 con 0x7f5c0400d350 2024-09-10T19:01:27.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c0400d350 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c04003df0 con 0x7f5c0400d350 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 10 client.49466.objecter _finish_command 5 = system:0 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 4 2024-09-10T19:01:27.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.234+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 2 5 2024-09-10T19:01:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:27.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 6 2024-09-10T19:01:27.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 6 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c040035e0 osd=2 3 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 2 6 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 4 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 10 client.49466.objecter _send_command 6 2024-09-10T19:01:27.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c04003060 con 0x7f5c0400d350 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c04003060 con 0x7f5c0400d350 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 10 client.49466.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 4 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 2 6 2024-09-10T19:01:27.257 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 7 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 7 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 3 7 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 4 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 10 client.49466.objecter _send_command 7 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c0400d010 con 0x7f5c04010e50 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.254+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c04010e50 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.258+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c0400d010 con 0x7f5c04010e50 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.258+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:01:27.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.258+0000 7f5c157fa640 10 client.49466.objecter _finish_command 7 = system:0 2024-09-10T19:01:27.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.258+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 4 2024-09-10T19:01:27.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.258+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 3 7 2024-09-10T19:01:27.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 8 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 8 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0400c990 osd=3 3 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 3 8 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 4 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 10 client.49466.objecter _send_command 8 2024-09-10T19:01:27.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.270+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c040132d0 con 0x7f5c04010e50 2024-09-10T19:01:27.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c040132d0 con 0x7f5c04010e50 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 10 client.49466.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 4 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 3 8 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:01:27.275 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 9 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 9 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 4 9 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 4 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 10 client.49466.objecter _send_command 9 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c0401a260 con 0x7f5c04016760 2024-09-10T19:01:27.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c04016760 2024-09-10T19:01:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c0401a260 con 0x7f5c04016760 2024-09-10T19:01:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:01:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 10 client.49466.objecter _finish_command 9 = system:0 2024-09-10T19:01:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 4 2024-09-10T19:01:27.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.274+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 4 9 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 10 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 10 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c040132d0 osd=4 3 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 4 10 2024-09-10T19:01:27.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 4 2024-09-10T19:01:27.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 10 client.49466.objecter _send_command 10 2024-09-10T19:01:27.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.290+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c0401a260 con 0x7f5c04016760 2024-09-10T19:01:27.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c0401a260 con 0x7f5c04016760 2024-09-10T19:01:27.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 10 client.49466.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 4 2024-09-10T19:01:27.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 4 10 2024-09-10T19:01:27.297 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:01:27.297 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:01:27.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 11 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 11 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 5 11 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 4 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 10 client.49466.objecter _send_command 11 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c0401e570 con 0x7f5c0401a910 2024-09-10T19:01:27.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.294+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c0401a910 2024-09-10T19:01:27.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.298+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c0401e570 con 0x7f5c0401a910 2024-09-10T19:01:27.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.298+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:01:27.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.298+0000 7f5c157fa640 10 client.49466.objecter _finish_command 11 = system:0 2024-09-10T19:01:27.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.298+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 4 2024-09-10T19:01:27.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.298+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 5 11 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 12 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 12 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0401a260 osd=5 3 2024-09-10T19:01:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 5 12 2024-09-10T19:01:27.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 4 2024-09-10T19:01:27.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 10 client.49466.objecter _send_command 12 2024-09-10T19:01:27.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.318+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c0401f6f0 con 0x7f5c0401a910 2024-09-10T19:01:27.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c0401f6f0 con 0x7f5c0401a910 2024-09-10T19:01:27.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 10 client.49466.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 4 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 5 12 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 13 2024-09-10T19:01:27.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 13 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 6 13 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 4 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 10 client.49466.objecter _send_command 13 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c04013120 con 0x7f5c040205d0 2024-09-10T19:01:27.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.322+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c040205d0 2024-09-10T19:01:27.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.326+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c04013120 con 0x7f5c040205d0 2024-09-10T19:01:27.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.326+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:01:27.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.326+0000 7f5c157fa640 10 client.49466.objecter _finish_command 13 = system:0 2024-09-10T19:01:27.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.326+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 4 2024-09-10T19:01:27.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.326+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 6 13 2024-09-10T19:01:27.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 14 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 14 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c0401f6f0 osd=6 3 2024-09-10T19:01:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 6 14 2024-09-10T19:01:27.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 4 2024-09-10T19:01:27.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 10 client.49466.objecter _send_command 14 2024-09-10T19:01:27.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.338+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c04013120 con 0x7f5c040205d0 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.342+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c04013120 con 0x7f5c040205d0 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.342+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.342+0000 7f5c157fa640 10 client.49466.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.342+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 4 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.342+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 6 14 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:01:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 15 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 15 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 7 15 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 4 2024-09-10T19:01:27.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 10 client.49466.objecter _send_command 15 2024-09-10T19:01:27.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c04026de0 con 0x7f5c04020ba0 2024-09-10T19:01:27.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 10 client.49466.objecter ms_handle_connect 0x7f5c04020ba0 2024-09-10T19:01:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5c04026de0 con 0x7f5c04020ba0 2024-09-10T19:01:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:01:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 10 client.49466.objecter _finish_command 15 = system:0 2024-09-10T19:01:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 4 2024-09-10T19:01:27.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.346+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 7 15 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.541173+0000 osd.0 (osd.0) 258 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.547465+0000 osd.0 (osd.0) 259 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:25.998075+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:26.001274+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:26.144806+0000 mgr.y (mgr.24526) 489 : cluster [DBG] pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.188978+0000 osd.4 (osd.4) 302 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: audit 2024-09-10T19:01:27.190954+0000 mon.c (mon.2) 641 : audit [DBG] from='client.? v1:172.21.15.83:0/3616996159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.192349+0000 osd.4 (osd.4) 303 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.541173+0000 osd.0 (osd.0) 258 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:01:27.351 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.547465+0000 osd.0 (osd.0) 259 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:25.998075+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:26.001274+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:26.144806+0000 mgr.y (mgr.24526) 489 : cluster [DBG] pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.188978+0000 osd.4 (osd.4) 302 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: audit 2024-09-10T19:01:27.190954+0000 mon.c (mon.2) 641 : audit [DBG] from='client.? v1:172.21.15.83:0/3616996159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:27.352 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:27 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.192349+0000 osd.4 (osd.4) 303 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 10 client.49466.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign -1 16 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 20 client.49466.objecter _get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_remove -1 16 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 20 client.49466.objecter get_session s=0x7f5c04013120 osd=7 3 2024-09-10T19:01:27.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 15 client.49466.objecter _session_command_op_assign 7 16 2024-09-10T19:01:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 4 2024-09-10T19:01:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 10 client.49466.objecter _send_command 16 2024-09-10T19:01:27.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.362+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c04028680 con 0x7f5c04020ba0 2024-09-10T19:01:27.364 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c157fa640 1 -- 172.21.15.83:0/3616996159 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f5c04028680 con 0x7f5c04020ba0 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c157fa640 10 client.49466.objecter ms_dispatch 0x7f5c18000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c157fa640 10 client.49466.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c157fa640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 4 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c157fa640 15 client.49466.objecter _session_command_op_remove 7 16 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 librados: watch_flush enter 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 librados: watch_flush exit 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.0 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6801/2535527483 conn(0x7f5c04004610 legacy=0x7f5c04002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04005810 osd=0 1 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.1 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6805/2309624637 conn(0x7f5c04004110 legacy=0x7f5c0400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04003fa0 osd=1 1 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.2 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6809/386094220 conn(0x7f5c0400d350 legacy=0x7f5c0400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040035e0 osd=2 1 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.3 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6813/1279688021 conn(0x7f5c04010e50 legacy=0x7f5c04011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1d65d640 1 -- 172.21.15.83:0/3616996159 reap_dead start 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0400c990 osd=3 1 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.4 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.165:6800/3225515265 conn(0x7f5c04016760 legacy=0x7f5c04018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1d65d640 1 -- 172.21.15.83:0/3616996159 reap_dead start 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c040132d0 osd=4 1 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.5 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.165:6804/571173777 conn(0x7f5c0401a910 legacy=0x7f5c0401ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401a260 osd=5 1 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.6 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.165:6808/1317922936 conn(0x7f5c040205d0 legacy=0x7f5c04011990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c0401f6f0 osd=6 1 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter close_session for osd.7 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.165:6812/1610151650 conn(0x7f5c04020ba0 legacy=0x7f5c04025530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter put_session s=0x7f5c04013120 osd=7 1 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 20 client.49466.objecter shutdown clearing up homeless session... 2024-09-10T19:01:27.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 10 client.49466.objecter successfully canceled tick 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5c000717a0 legacy=0x7f5c00073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> v1:172.21.15.83:6790/0 conn(0x7f5c1810b000 legacy=0x7f5c1811c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1d65d640 1 -- 172.21.15.83:0/3616996159 reap_dead start 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1d65d640 1 -- 172.21.15.83:0/3616996159 reap_dead start 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 shutdown_connections 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 >> 172.21.15.83:0/3616996159 conn(0x7f5c180fa290 msgr2=0x7f5c1810e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 shutdown_connections 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 -- 172.21.15.83:0/3616996159 wait complete. 2024-09-10T19:01:27.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:27.366+0000 7f5c1de5e640 1 librados: shutdown 2024-09-10T19:01:27.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:27.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:27.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.541173+0000 osd.0 (osd.0) 258 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.547465+0000 osd.0 (osd.0) 259 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:25.998075+0000 osd.3 (osd.3) 288 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:26.001274+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:26.144806+0000 mgr.y (mgr.24526) 489 : cluster [DBG] pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 383 B/s wr, 3 op/s 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.188978+0000 osd.4 (osd.4) 302 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: audit 2024-09-10T19:01:27.190954+0000 mon.c (mon.2) 641 : audit [DBG] from='client.? v1:172.21.15.83:0/3616996159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:27 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.192349+0000 osd.4 (osd.4) 303 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:01:27.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:27.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:27.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:27.777 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:27.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:27.911 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:27.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:28.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:28.052 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:28.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.188 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:28.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.323 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:28.349 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:28.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:28.460 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.250131+0000 osd.7 (osd.7) 276 : cluster [DBG] 3.d scrub starts 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.255300+0000 osd.7 (osd.7) 277 : cluster [DBG] 3.d scrub ok 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.396439+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.401168+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: audit 2024-09-10T19:01:28.045481+0000 mgr.y (mgr.24526) 490 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:28.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.145676+0000 mgr.y (mgr.24526) 491 : cluster [DBG] pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 342 B/s wr, 3 op/s 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.250131+0000 osd.7 (osd.7) 276 : cluster [DBG] 3.d scrub starts 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.255300+0000 osd.7 (osd.7) 277 : cluster [DBG] 3.d scrub ok 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.396439+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.401168+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: audit 2024-09-10T19:01:28.045481+0000 mgr.y (mgr.24526) 490 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:28.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:28 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.145676+0000 mgr.y (mgr.24526) 491 : cluster [DBG] pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 342 B/s wr, 3 op/s 2024-09-10T19:01:28.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:28.596 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.250131+0000 osd.7 (osd.7) 276 : cluster [DBG] 3.d scrub starts 2024-09-10T19:01:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.255300+0000 osd.7 (osd.7) 277 : cluster [DBG] 3.d scrub ok 2024-09-10T19:01:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.396439+0000 osd.5 (osd.5) 259 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:01:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.401168+0000 osd.5 (osd.5) 260 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:01:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: audit 2024-09-10T19:01:28.045481+0000 mgr.y (mgr.24526) 490 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:28.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:28 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.145676+0000 mgr.y (mgr.24526) 491 : cluster [DBG] pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 342 B/s wr, 3 op/s 2024-09-10T19:01:28.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.733 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:28.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:28.868 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:28.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:28.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:29.001 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:29.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:29.136 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:29.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:29.274 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:29.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:29.411 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:29.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:29.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.857362+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.a scrub starts 2024-09-10T19:01:29.528 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:27.860537+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.a scrub ok 2024-09-10T19:01:29.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.211383+0000 osd.7 (osd.7) 278 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:01:29.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.217440+0000 osd.7 (osd.7) 279 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:01:29.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.740134+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:01:29.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.743825+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:01:29.529 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:29 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.168793+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:01:29.549 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:29.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.857362+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.a scrub starts 2024-09-10T19:01:29.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:27.860537+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.a scrub ok 2024-09-10T19:01:29.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.211383+0000 osd.7 (osd.7) 278 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:01:29.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.217440+0000 osd.7 (osd.7) 279 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.740134+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.743825+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.168793+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.857362+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.a scrub starts 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:27.860537+0000 osd.1 (osd.1) 274 : cluster [DBG] 2.a scrub ok 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.211383+0000 osd.7 (osd.7) 278 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.217440+0000 osd.7 (osd.7) 279 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.740134+0000 osd.6 (osd.6) 221 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:01:29.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.743825+0000 osd.6 (osd.6) 222 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:01:29.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:29 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.168793+0000 mon.a (mon.0) 2704 : cluster [DBG] osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:01:29.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:29.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:29.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:29.826 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:29.929 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:29.962 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:29.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:30.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.100 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:30.204 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:30.238 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:30.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.372 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:30.475 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:30.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.872342+0000 osd.1 (osd.1) 275 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:01:30.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.876695+0000 osd.1 (osd.1) 276 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:01:30.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.911273+0000 osd.3 (osd.3) 290 : cluster [DBG] 64.3 scrub starts 2024-09-10T19:01:30.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:28.916631+0000 osd.3 (osd.3) 291 : cluster [DBG] 64.3 scrub ok 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.172319+0000 osd.7 (osd.7) 280 : cluster [DBG] 12.d scrub starts 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.180916+0000 osd.7 (osd.7) 281 : cluster [DBG] 12.d scrub ok 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.229183+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.231284+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: audit 2024-09-10T19:01:29.566496+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.146518+0000 mgr.y (mgr.24526) 492 : cluster [DBG] pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: audit 2024-09-10T19:01:30.312439+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: audit 2024-09-10T19:01:30.314504+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: audit 2024-09-10T19:01:30.315738+0000 mon.a (mon.0) 2705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[18629]: audit 2024-09-10T19:01:30.317743+0000 mon.a (mon.0) 2706 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.872342+0000 osd.1 (osd.1) 275 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.876695+0000 osd.1 (osd.1) 276 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.911273+0000 osd.3 (osd.3) 290 : cluster [DBG] 64.3 scrub starts 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:28.916631+0000 osd.3 (osd.3) 291 : cluster [DBG] 64.3 scrub ok 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.172319+0000 osd.7 (osd.7) 280 : cluster [DBG] 12.d scrub starts 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.180916+0000 osd.7 (osd.7) 281 : cluster [DBG] 12.d scrub ok 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.229183+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.231284+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: audit 2024-09-10T19:01:29.566496+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.146518+0000 mgr.y (mgr.24526) 492 : cluster [DBG] pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: audit 2024-09-10T19:01:30.312439+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: audit 2024-09-10T19:01:30.314504+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: audit 2024-09-10T19:01:30.315738+0000 mon.a (mon.0) 2705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.491 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:30 smithi083 bash[32381]: audit 2024-09-10T19:01:30.317743+0000 mon.a (mon.0) 2706 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.510 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.872342+0000 osd.1 (osd.1) 275 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.876695+0000 osd.1 (osd.1) 276 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.911273+0000 osd.3 (osd.3) 290 : cluster [DBG] 64.3 scrub starts 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:28.916631+0000 osd.3 (osd.3) 291 : cluster [DBG] 64.3 scrub ok 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.172319+0000 osd.7 (osd.7) 280 : cluster [DBG] 12.d scrub starts 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.180916+0000 osd.7 (osd.7) 281 : cluster [DBG] 12.d scrub ok 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.229183+0000 osd.2 (osd.2) 201 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.231284+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: audit 2024-09-10T19:01:29.566496+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.146518+0000 mgr.y (mgr.24526) 492 : cluster [DBG] pgmap v737: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: audit 2024-09-10T19:01:30.312439+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: audit 2024-09-10T19:01:30.314504+0000 mon.b (mon.1) 370 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: audit 2024-09-10T19:01:30.315738+0000 mon.a (mon.0) 2705 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:30.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:30 smithi165 bash[22591]: audit 2024-09-10T19:01:30.317743+0000 mon.a (mon.0) 2706 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-71"}]: dispatch 2024-09-10T19:01:30.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:30.646 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:30.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:30.784 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:30.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.923 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:30.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:30.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:01:31.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:31.166 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:31.200 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:31.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:31.339 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:31.374 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.870162+0000 osd.1 (osd.1) 277 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.870162+0000 osd.1 (osd.1) 277 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.873713+0000 osd.1 (osd.1) 278 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.907190+0000 osd.3 (osd.3) 292 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:29.910542+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.327734+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.331208+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.702971+0000 osd.6 (osd.6) 223 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:01:31.375 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.706632+0000 osd.6 (osd.6) 224 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:01:31.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.478 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:31.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.873713+0000 osd.1 (osd.1) 278 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:01:31.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.907190+0000 osd.3 (osd.3) 292 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:01:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:29.910542+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:01:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.327734+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:01:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.331208+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:01:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.702971+0000 osd.6 (osd.6) 223 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:01:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:31 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.706632+0000 osd.6 (osd.6) 224 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:01:31.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:31.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.870162+0000 osd.1 (osd.1) 277 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.873713+0000 osd.1 (osd.1) 278 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.907190+0000 osd.3 (osd.3) 292 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:29.910542+0000 osd.3 (osd.3) 293 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.327734+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.331208+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.702971+0000 osd.6 (osd.6) 223 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:01:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:31 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.706632+0000 osd.6 (osd.6) 224 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:01:31.614 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:31.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:31.753 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:31.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.890 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:31.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:31.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:32.027 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:32.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:32.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:32.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:32.291 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:32.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:32.418 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:32.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.821969+0000 osd.1 (osd.1) 279 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:30.825521+0000 osd.1 (osd.1) 280 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.191609+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.194784+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.202655+0000 osd.4 (osd.4) 304 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.205154+0000 osd.4 (osd.4) 305 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.366380+0000 osd.5 (osd.5) 263 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.368949+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.656049+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:31.660082+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:01:32.535 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:32 smithi165 bash[22591]: cluster 2024-09-10T19:01:32.147279+0000 mgr.y (mgr.24526) 493 : cluster [DBG] pgmap v739: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:32.556 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:32.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:32.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.821969+0000 osd.1 (osd.1) 279 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:30.825521+0000 osd.1 (osd.1) 280 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.191609+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.194784+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.202655+0000 osd.4 (osd.4) 304 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.205154+0000 osd.4 (osd.4) 305 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.366380+0000 osd.5 (osd.5) 263 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.368949+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.656049+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:31.660082+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:01:32.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18629]: cluster 2024-09-10T19:01:32.147279+0000 mgr.y (mgr.24526) 493 : cluster [DBG] pgmap v739: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.821969+0000 osd.1 (osd.1) 279 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:30.825521+0000 osd.1 (osd.1) 280 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.191609+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.194784+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.202655+0000 osd.4 (osd.4) 304 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:01:32.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.205154+0000 osd.4 (osd.4) 305 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:01:32.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.366380+0000 osd.5 (osd.5) 263 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:01:32.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.368949+0000 mon.a (mon.0) 2707 : cluster [DBG] osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:01:32.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.656049+0000 osd.6 (osd.6) 225 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:01:32.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:31.660082+0000 osd.6 (osd.6) 226 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:01:32.691 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[32381]: cluster 2024-09-10T19:01:32.147279+0000 mgr.y (mgr.24526) 493 : cluster [DBG] pgmap v739: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:32.694 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:32.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:32.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:32.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:32.949 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:32] "GET /metrics HTTP/1.1" 200 228887 "" "Prometheus/2.43.0" 2024-09-10T19:01:32.968 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:32.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:33.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:33.105 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:33.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.242 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:33.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:33.380 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:33.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.517 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:33.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:33.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: cluster 2024-09-10T19:01:32.163837+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:01:33.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: cluster 2024-09-10T19:01:32.326773+0000 osd.5 (osd.5) 264 : cluster [DBG] 3.c scrub starts 2024-09-10T19:01:33.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: cluster 2024-09-10T19:01:32.331234+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.c scrub ok 2024-09-10T19:01:33.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: cluster 2024-09-10T19:01:32.389721+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: audit 2024-09-10T19:01:32.399395+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[18629]: audit 2024-09-10T19:01:32.404040+0000 mon.a (mon.0) 2709 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: cluster 2024-09-10T19:01:32.163837+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: cluster 2024-09-10T19:01:32.326773+0000 osd.5 (osd.5) 264 : cluster [DBG] 3.c scrub starts 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: cluster 2024-09-10T19:01:32.331234+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.c scrub ok 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: cluster 2024-09-10T19:01:32.389721+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:01:33.632 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: audit 2024-09-10T19:01:32.399395+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:33 smithi083 bash[32381]: audit 2024-09-10T19:01:32.404040+0000 mon.a (mon.0) 2709 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:33.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: cluster 2024-09-10T19:01:32.163837+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:01:33.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: cluster 2024-09-10T19:01:32.326773+0000 osd.5 (osd.5) 264 : cluster [DBG] 3.c scrub starts 2024-09-10T19:01:33.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: cluster 2024-09-10T19:01:32.331234+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.c scrub ok 2024-09-10T19:01:33.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: cluster 2024-09-10T19:01:32.389721+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:01:33.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: audit 2024-09-10T19:01:32.399395+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:33 smithi165 bash[22591]: audit 2024-09-10T19:01:32.404040+0000 mon.a (mon.0) 2709 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:33.707 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:33.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/1667561469 >> v1:172.21.15.83:6789/0 conn(0x7f97200fb3a0 legacy=0x7f97200fb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/1667561469 shutdown_connections 2024-09-10T19:01:33.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/1667561469 >> 172.21.15.83:0/1667561469 conn(0x7f97200f7070 msgr2=0x7f97200f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:33.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/1667561469 shutdown_connections 2024-09-10T19:01:33.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/1667561469 wait complete. 2024-09-10T19:01:33.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 librados: starting msgr at 2024-09-10T19:01:33.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 librados: starting objecter 2024-09-10T19:01:33.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 Processor -- start 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- start start 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 librados: setting wanted keys 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 librados: calling monclient init 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f972011e1d0 con 0x7f97200fb3a0 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f972011f390 con 0x7f97200fbe80 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9720120590 con 0x7f972010b1d0 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f971e59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97200fb3a0 0x7f9720107580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36166/0 (socket says 172.21.15.83:36166) 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f971e59c640 1 -- 172.21.15.83:0/3762626229 learned_addr learned my addr 172.21.15.83:0/3762626229 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter ms_handle_connect 0x7f97200fb3a0 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter ms_handle_connect 0x7f972010b1d0 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:33.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784839216 0 0) 0x7f972011e1d0 con 0x7f97200fb3a0 2024-09-10T19:01:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97080039c0 con 0x7f97200fb3a0 2024-09-10T19:01:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter ms_handle_connect 0x7f97200fbe80 2024-09-10T19:01:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537058105 0 0) 0x7f9720120590 con 0x7f972010b1d0 2024-09-10T19:01:33.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f972011e1d0 con 0x7f972010b1d0 2024-09-10T19:01:33.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785717709 0 0) 0x7f972011f390 con 0x7f97200fbe80 2024-09-10T19:01:33.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720120590 con 0x7f97200fbe80 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035403640 0 0) 0x7f97080039c0 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f972011f390 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9714001500 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336360884 0 0) 0x7f972011e1d0 con 0x7f972010b1d0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97080039c0 con 0x7f972010b1d0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9718001a20 con 0x7f972010b1d0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2222547421 0 0) 0x7f972011f390 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 >> v1:172.21.15.83:6790/0 conn(0x7f972010b1d0 legacy=0x7f972011c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 >> v1:172.21.15.165:6789/0 conn(0x7f97200fbe80 legacy=0x7f9720107c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9720121790 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9720120780 con 0x7f97200fb3a0 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 10 client.25401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:33.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9720120c50 con 0x7f97200fb3a0 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.745+0000 7f9724e55640 1 librados: init done 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f97140022c0 con 0x7f97200fb3a0 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f97140035a0 con 0x7f97200fb3a0 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9700005010 con 0x7f97200fb3a0 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T19:01:33.753 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:01:33.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9714001e60 con 0x7f97200fb3a0 2024-09-10T19:01:33.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(489..489 src has 69..489) v4 ==== 8004+0+0 (unknown 4095677405 0 0) 0x7f971408a860 con 0x7f97200fb3a0 2024-09-10T19:01:33.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 10 client.25401.objecter ms_dispatch 0x7f9720000b90 osd_map(489..489 src has 69..489) v4 2024-09-10T19:01:33.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 3 client.25401.objecter handle_osd_map got epochs [489,489] > 0 2024-09-10T19:01:33.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 3 client.25401.objecter handle_osd_map decoding full epoch 489 2024-09-10T19:01:33.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.749+0000 7f970effd640 20 client.25401.objecter dump_active .. 0 homeless 2024-09-10T19:01:33.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.753+0000 7f970effd640 10 client.25401.objecter ms_handle_connect 0x7f97080714d0 2024-09-10T19:01:33.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.757+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9714056e20 con 0x7f97200fb3a0 2024-09-10T19:01:33.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.794 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:33.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:33.928 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:33.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:33.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:33.941+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f97000052c0 con 0x7f97200fb3a0 2024-09-10T19:01:34.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:34.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:34.167 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.173+0000 7f970effd640 1 -- 172.21.15.83:0/3762626229 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v490) v1 ==== 101+0+0 (unknown 3348327898 0 0) 0x7f97140567c0 con 0x7f97200fb3a0 2024-09-10T19:01:34.179 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.177+0000 7f9724e55640 10 librados: watch_flush enter 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.177+0000 7f9724e55640 10 librados: watch_flush exit 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 20 client.25401.objecter shutdown clearing up homeless session... 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 10 client.25401.objecter successfully canceled tick 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 >> v1:172.21.15.83:6800/3071793735 conn(0x7f97080714d0 legacy=0x7f9708073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:34.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 >> v1:172.21.15.83:6789/0 conn(0x7f97200fb3a0 legacy=0x7f9720107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 shutdown_connections 2024-09-10T19:01:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 >> 172.21.15.83:0/3762626229 conn(0x7f97200f7070 msgr2=0x7f972010d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 shutdown_connections 2024-09-10T19:01:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 -- 172.21.15.83:0/3762626229 wait complete. 2024-09-10T19:01:34.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:34.181+0000 7f9724e55640 1 librados: shutdown 2024-09-10T19:01:34.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:34.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.337 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:34.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:34.455 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T19:01:34.218082928Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:01:34.455 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[57601]: logger=cleanup t=2024-09-10T19:01:34.2749018Z level=info msg="Completed cleanup jobs" duration=67.361729ms 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.185582+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.189389+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.338013+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.342417+0000 osd.5 (osd.5) 267 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: audit 2024-09-10T19:01:33.395982+0000 mon.a (mon.0) 2710 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.396354+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:01:34.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: audit 2024-09-10T19:01:33.405137+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:34.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: audit 2024-09-10T19:01:33.945297+0000 mon.a (mon.0) 2712 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:01:34.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.148046+0000 mgr.y (mgr.24526) 494 : cluster [DBG] pgmap v742: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:34.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.165452+0000 mon.a (mon.0) 2713 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:01:34.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: audit 2024-09-10T19:01:34.176630+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:01:34.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:34 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.176729+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:01:34.474 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:34.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:34.613 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:34.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.185582+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.189389+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.338013+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.342417+0000 osd.5 (osd.5) 267 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: audit 2024-09-10T19:01:33.395982+0000 mon.a (mon.0) 2710 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.396354+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:01:34.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: audit 2024-09-10T19:01:33.405137+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: audit 2024-09-10T19:01:33.945297+0000 mon.a (mon.0) 2712 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.148046+0000 mgr.y (mgr.24526) 494 : cluster [DBG] pgmap v742: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.165452+0000 mon.a (mon.0) 2713 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: audit 2024-09-10T19:01:34.176630+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.176729+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.185582+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.189389+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.338013+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.342417+0000 osd.5 (osd.5) 267 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: audit 2024-09-10T19:01:33.395982+0000 mon.a (mon.0) 2710 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.396354+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:01:34.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: audit 2024-09-10T19:01:33.405137+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:34.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: audit 2024-09-10T19:01:33.945297+0000 mon.a (mon.0) 2712 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:01:34.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.148046+0000 mgr.y (mgr.24526) 494 : cluster [DBG] pgmap v742: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:34.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.165452+0000 mon.a (mon.0) 2713 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:01:34.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: audit 2024-09-10T19:01:34.176630+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? v1:172.21.15.83:0/3762626229' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:01:34.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:34 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.176729+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:01:34.750 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:34.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:34.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:34.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:35.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:35.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:35.153 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:35.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:35.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:35.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.430 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:35.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:35.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.815038+0000 osd.3 (osd.3) 294 : cluster [DBG] 64.10 scrub starts 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:33.818539+0000 osd.3 (osd.3) 295 : cluster [DBG] 64.10 scrub ok 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.170307+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.e scrub starts 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.178336+0000 osd.2 (osd.2) 204 : cluster [DBG] 2.e scrub ok 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.220048+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.223501+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.711345+0000 osd.6 (osd.6) 227 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:01:35.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:34.714931+0000 osd.6 (osd.6) 228 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.176917+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.815038+0000 osd.3 (osd.3) 294 : cluster [DBG] 64.10 scrub starts 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:33.818539+0000 osd.3 (osd.3) 295 : cluster [DBG] 64.10 scrub ok 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.170307+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.e scrub starts 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.178336+0000 osd.2 (osd.2) 204 : cluster [DBG] 2.e scrub ok 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.220048+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.223501+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.711345+0000 osd.6 (osd.6) 227 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:34.714931+0000 osd.6 (osd.6) 228 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:01:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:35 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.176917+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:01:35.568 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:35.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.815038+0000 osd.3 (osd.3) 294 : cluster [DBG] 64.10 scrub starts 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:33.818539+0000 osd.3 (osd.3) 295 : cluster [DBG] 64.10 scrub ok 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.170307+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.e scrub starts 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.178336+0000 osd.2 (osd.2) 204 : cluster [DBG] 2.e scrub ok 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.220048+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:01:35.684 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.223501+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:01:35.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.711345+0000 osd.6 (osd.6) 227 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:01:35.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:34.714931+0000 osd.6 (osd.6) 228 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:01:35.685 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:35 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.176917+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:01:35.701 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:35.805 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:35.840 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:35.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.976 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:35.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:36.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:36.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:36.254 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:36.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:36.394 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:36.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:36.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:36.532 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:36.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.165870+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.b scrub starts 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.169206+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.b scrub ok 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.181730+0000 osd.2 (osd.2) 205 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.187676+0000 osd.2 (osd.2) 206 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.368081+0000 osd.5 (osd.5) 268 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.373237+0000 osd.5 (osd.5) 269 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.479441+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:35.482806+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:01:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:36 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.150260+0000 mgr.y (mgr.24526) 495 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:01:36.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:36.671 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:36.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:36.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.165870+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.b scrub starts 2024-09-10T19:01:36.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.169206+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.b scrub ok 2024-09-10T19:01:36.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.181730+0000 osd.2 (osd.2) 205 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:01:36.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.187676+0000 osd.2 (osd.2) 206 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.368081+0000 osd.5 (osd.5) 268 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.373237+0000 osd.5 (osd.5) 269 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.479441+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:35.482806+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.150260+0000 mgr.y (mgr.24526) 495 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.165870+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.b scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.169206+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.b scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.181730+0000 osd.2 (osd.2) 205 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.187676+0000 osd.2 (osd.2) 206 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.368081+0000 osd.5 (osd.5) 268 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.373237+0000 osd.5 (osd.5) 269 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.479441+0000 osd.0 (osd.0) 260 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:01:36.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:35.482806+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:01:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:36 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.150260+0000 mgr.y (mgr.24526) 495 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:01:36.810 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:36.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:36.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:37.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:37.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:37.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:37.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:37.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:37.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:37.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:37.495 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:37.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.182993+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.187061+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.197643+0000 osd.7 (osd.7) 289 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.202234+0000 osd.7 (osd.7) 290 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.389899+0000 osd.5 (osd.5) 270 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.392799+0000 osd.5 (osd.5) 271 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.749184+0000 osd.6 (osd.6) 229 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:01:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:37 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.753736+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:01:37.630 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:37.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:37.772 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.182993+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:01:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.187061+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.197643+0000 osd.7 (osd.7) 289 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.202234+0000 osd.7 (osd.7) 290 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.389899+0000 osd.5 (osd.5) 270 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.392799+0000 osd.5 (osd.5) 271 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.749184+0000 osd.6 (osd.6) 229 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.753736+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.182993+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.187061+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.197643+0000 osd.7 (osd.7) 289 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:01:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.202234+0000 osd.7 (osd.7) 290 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:01:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.389899+0000 osd.5 (osd.5) 270 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:01:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.392799+0000 osd.5 (osd.5) 271 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:01:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.749184+0000 osd.6 (osd.6) 229 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:01:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:37 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.753736+0000 osd.6 (osd.6) 230 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:01:37.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:37.909 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:37.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:38.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:38.049 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:38.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:38.184 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:38.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:38.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:38.300 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:38.321 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:38.424 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:38.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.829606+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.833589+0000 osd.3 (osd.3) 297 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.857172+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:36.860795+0000 osd.1 (osd.1) 282 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.414750+0000 osd.5 (osd.5) 272 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.418143+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.568144+0000 osd.0 (osd.0) 262 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.571736+0000 osd.0 (osd.0) 263 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.779781+0000 osd.6 (osd.6) 231 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.783288+0000 osd.6 (osd.6) 232 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: audit 2024-09-10T19:01:38.056685+0000 mgr.y (mgr.24526) 496 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[32381]: cluster 2024-09-10T19:01:38.150863+0000 mgr.y (mgr.24526) 497 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.829606+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.833589+0000 osd.3 (osd.3) 297 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.857172+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:36.860795+0000 osd.1 (osd.1) 282 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.414750+0000 osd.5 (osd.5) 272 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.418143+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.568144+0000 osd.0 (osd.0) 262 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.571736+0000 osd.0 (osd.0) 263 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.779781+0000 osd.6 (osd.6) 231 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.783288+0000 osd.6 (osd.6) 232 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: audit 2024-09-10T19:01:38.056685+0000 mgr.y (mgr.24526) 496 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:38 smithi083 bash[18629]: cluster 2024-09-10T19:01:38.150863+0000 mgr.y (mgr.24526) 497 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:38.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:38.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.829606+0000 osd.3 (osd.3) 296 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:01:38.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.833589+0000 osd.3 (osd.3) 297 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:01:38.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.857172+0000 osd.1 (osd.1) 281 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:36.860795+0000 osd.1 (osd.1) 282 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.414750+0000 osd.5 (osd.5) 272 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.418143+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.568144+0000 osd.0 (osd.0) 262 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.571736+0000 osd.0 (osd.0) 263 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.779781+0000 osd.6 (osd.6) 231 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.783288+0000 osd.6 (osd.6) 232 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: audit 2024-09-10T19:01:38.056685+0000 mgr.y (mgr.24526) 496 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:38.571 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:38 smithi165 bash[22591]: cluster 2024-09-10T19:01:38.150863+0000 mgr.y (mgr.24526) 497 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:38.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:38.598 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:38.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:38.728 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:38.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:01:38.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:38.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:38.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:38.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:39.011 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:39.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:39.140 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:39.243 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:39.275 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:39.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:39.412 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:39.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.870971+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.f scrub starts 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:37.874463+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.f scrub ok 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:38.100107+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:38.102770+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:38.577562+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:01:39.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:38.580981+0000 osd.0 (osd.0) 265 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[32381]: cluster 2024-09-10T19:01:39.179316+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.870971+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.f scrub starts 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:37.874463+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.f scrub ok 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:38.100107+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:38.102770+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:38.577562+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:38.580981+0000 osd.0 (osd.0) 265 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:01:39.529 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:39 smithi083 bash[18629]: cluster 2024-09-10T19:01:39.179316+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:01:39.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:39.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:39.687 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:39.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:39.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.870971+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.f scrub starts 2024-09-10T19:01:39.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:37.874463+0000 osd.1 (osd.1) 284 : cluster [DBG] 4.f scrub ok 2024-09-10T19:01:39.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:38.100107+0000 osd.2 (osd.2) 209 : cluster [DBG] 2.c scrub starts 2024-09-10T19:01:39.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:38.102770+0000 osd.2 (osd.2) 210 : cluster [DBG] 2.c scrub ok 2024-09-10T19:01:39.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:38.577562+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:01:39.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:38.580981+0000 osd.0 (osd.0) 265 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:01:39.804 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:39 smithi165 bash[22591]: cluster 2024-09-10T19:01:39.179316+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:01:39.824 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:39.928 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:39.962 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:39.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:40.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:40.098 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:40.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.233 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:40.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:40.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.506 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:40.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.581+0000 7f7ec1b67640 1 -- 172.21.15.83:0/266637928 >> v1:172.21.15.83:6790/0 conn(0x7f7ebc107e30 legacy=0x7f7ebc10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:40.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.581+0000 7f7ec1b67640 1 -- 172.21.15.83:0/266637928 shutdown_connections 2024-09-10T19:01:40.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.581+0000 7f7ec1b67640 1 -- 172.21.15.83:0/266637928 >> 172.21.15.83:0/266637928 conn(0x7f7ebc075290 msgr2=0x7f7ebc0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:40.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.581+0000 7f7ec1b67640 1 -- 172.21.15.83:0/266637928 shutdown_connections 2024-09-10T19:01:40.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.581+0000 7f7ec1b67640 1 -- 172.21.15.83:0/266637928 wait complete. 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 librados: starting msgr at 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 librados: starting objecter 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 Processor -- start 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- start start 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 librados: setting wanted keys 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 librados: calling monclient init 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ebc111240 con 0x7f7ebc1042a0 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ebc11c330 con 0x7f7ebc107e30 2024-09-10T19:01:40.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ebc11d510 con 0x7f7ebc1014e0 2024-09-10T19:01:40.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ebb7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7ebc1014e0 0x7f7ebc1106a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36880/0 (socket says 172.21.15.83:36880) 2024-09-10T19:01:40.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ebb7fe640 1 -- 172.21.15.83:0/4213776336 learned_addr learned my addr 172.21.15.83:0/4213776336 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:40.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter ms_handle_connect 0x7f7ebc1014e0 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter ms_handle_connect 0x7f7ebc107e30 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter ms_handle_connect 0x7f7ebc1042a0 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591054908 0 0) 0x7f7ebc11c330 con 0x7f7ebc107e30 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea4003720 con 0x7f7ebc107e30 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325639004 0 0) 0x7f7ebc11d510 con 0x7f7ebc1014e0 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ebc11c330 con 0x7f7ebc1014e0 2024-09-10T19:01:40.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240541630 0 0) 0x7f7ebc111240 con 0x7f7ebc1042a0 2024-09-10T19:01:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ebc11d510 con 0x7f7ebc1042a0 2024-09-10T19:01:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 269835637 0 0) 0x7f7ebc11c330 con 0x7f7ebc1014e0 2024-09-10T19:01:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc111240 con 0x7f7ebc1014e0 2024-09-10T19:01:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7eb4001c80 con 0x7f7ebc1014e0 2024-09-10T19:01:40.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2787306232 0 0) 0x7f7ea4003720 con 0x7f7ebc107e30 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc11c330 con 0x7f7ebc107e30 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7eac001430 con 0x7f7ebc107e30 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 699530000 0 0) 0x7f7ebc11d510 con 0x7f7ebc1042a0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea4003720 con 0x7f7ebc1042a0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7eb00018c0 con 0x7f7ebc1042a0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 800902452 0 0) 0x7f7ebc111240 con 0x7f7ebc1014e0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 >> v1:172.21.15.165:6789/0 conn(0x7f7ebc107e30 legacy=0x7f7ebc119c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 >> v1:172.21.15.83:6789/0 conn(0x7f7ebc1042a0 legacy=0x7f7ebc1164d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc11e6f0 con 0x7f7ebc1014e0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc11c520 con 0x7f7ebc1014e0 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 10 client.49478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:40.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc11c9f0 con 0x7f7ebc1014e0 2024-09-10T19:01:40.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.585+0000 7f7ec1b67640 1 librados: init done 2024-09-10T19:01:40.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.589+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7eb4002380 con 0x7f7ebc1014e0 2024-09-10T19:01:40.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.589+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc0fcc60 con 0x7f7ebc1014e0 2024-09-10T19:01:40.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.589+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7eb4005e00 con 0x7f7ebc1014e0 2024-09-10T19:01:40.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7eb4001f20 con 0x7f7ebc1014e0 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(492..492 src has 69..492) v4 ==== 8004+0+0 (unknown 3782380859 0 0) 0x7f7eb408aa40 con 0x7f7ebc1014e0 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 10 client.49478.objecter ms_dispatch 0x7f7ebc000b90 osd_map(492..492 src has 69..492) v4 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 3 client.49478.objecter handle_osd_map got epochs [492,492] > 0 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 3 client.49478.objecter handle_osd_map decoding full epoch 492 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 20 client.49478.objecter dump_active .. 0 homeless 2024-09-10T19:01:40.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.593+0000 7f7ea3fff640 10 client.49478.objecter ms_handle_connect 0x7f7ea4075d40 2024-09-10T19:01:40.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.597+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7eb4057040 con 0x7f7ebc1014e0 2024-09-10T19:01:40.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:40.644 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:40.651 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[18629]: cluster 2024-09-10T19:01:39.121663+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:01:40.651 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[18629]: cluster 2024-09-10T19:01:39.125523+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:01:40.651 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[18629]: cluster 2024-09-10T19:01:39.505431+0000 osd.5 (osd.5) 274 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:01:40.651 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[18629]: cluster 2024-09-10T19:01:39.509084+0000 osd.5 (osd.5) 275 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.151667+0000 mgr.y (mgr.24526) 498 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[32381]: cluster 2024-09-10T19:01:39.121663+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[32381]: cluster 2024-09-10T19:01:39.125523+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[32381]: cluster 2024-09-10T19:01:39.505431+0000 osd.5 (osd.5) 274 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[32381]: cluster 2024-09-10T19:01:39.509084+0000 osd.5 (osd.5) 275 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:01:40.652 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:40 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.151667+0000 mgr.y (mgr.24526) 498 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:40.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:40 smithi165 bash[22591]: cluster 2024-09-10T19:01:39.121663+0000 osd.2 (osd.2) 211 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:01:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:40 smithi165 bash[22591]: cluster 2024-09-10T19:01:39.125523+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:01:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:40 smithi165 bash[22591]: cluster 2024-09-10T19:01:39.505431+0000 osd.5 (osd.5) 274 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:01:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:40 smithi165 bash[22591]: cluster 2024-09-10T19:01:39.509084+0000 osd.5 (osd.5) 275 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:01:40.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:40 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.151667+0000 mgr.y (mgr.24526) 498 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-09-10T19:01:40.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.777+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ebc10d9d0 con 0x7f7ea4075d40 2024-09-10T19:01:40.784 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:40.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.797+0000 7f7ea3fff640 1 -- 172.21.15.83:0/4213776336 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+625596 (unknown 2965378022 0 141506683) 0x7f7ebc10d9d0 con 0x7f7ea4075d40 2024-09-10T19:01:40.806 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:01:40.810 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 10 librados: watch_flush enter 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 10 librados: watch_flush exit 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 20 client.49478.objecter shutdown clearing up homeless session... 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 10 client.49478.objecter successfully canceled tick 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7ea4075d40 legacy=0x7f7ea40781f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 >> v1:172.21.15.83:6790/0 conn(0x7f7ebc1014e0 legacy=0x7f7ebc1106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 shutdown_connections 2024-09-10T19:01:40.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 >> 172.21.15.83:0/4213776336 conn(0x7f7ebc075290 msgr2=0x7f7ebc10b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:40.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 shutdown_connections 2024-09-10T19:01:40.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 -- 172.21.15.83:0/4213776336 wait complete. 2024-09-10T19:01:40.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:40.801+0000 7f7ec1b67640 1 librados: shutdown 2024-09-10T19:01:40.887 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:40.919 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:40.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:41.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:41.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:41.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:41.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:41.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:41.335 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:41.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:41.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:41.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.229580+0000 osd.4 (osd.4) 308 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.232577+0000 osd.4 (osd.4) 309 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.509042+0000 osd.5 (osd.5) 276 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.519183+0000 osd.5 (osd.5) 277 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:01:41.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:41 smithi165 bash[22591]: audit 2024-09-10T19:01:40.780096+0000 mgr.y (mgr.24526) 499 : audit [DBG] from='client.49478 v1:172.21.15.83:0/4213776336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:01:41.612 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:41.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:41.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.229580+0000 osd.4 (osd.4) 308 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:01:41.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.232577+0000 osd.4 (osd.4) 309 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:01:41.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.509042+0000 osd.5 (osd.5) 276 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:01:41.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.519183+0000 osd.5 (osd.5) 277 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:01:41.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[18629]: audit 2024-09-10T19:01:40.780096+0000 mgr.y (mgr.24526) 499 : audit [DBG] from='client.49478 v1:172.21.15.83:0/4213776336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:01:41.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.229580+0000 osd.4 (osd.4) 308 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:01:41.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.232577+0000 osd.4 (osd.4) 309 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:01:41.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.509042+0000 osd.5 (osd.5) 276 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:01:41.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.519183+0000 osd.5 (osd.5) 277 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:01:41.730 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:41 smithi083 bash[32381]: audit 2024-09-10T19:01:40.780096+0000 mgr.y (mgr.24526) 499 : audit [DBG] from='client.49478 v1:172.21.15.83:0/4213776336' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:01:41.748 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:41.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:41.886 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:41.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:41.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:42.022 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:42.085 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-10T19:01:40.151161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792105,"num_objects":370,"num_object_clones":7,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":0,"num_read":2141,"num_read_kb":1892,"num_write":1466,"num_write_kb":14391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":590460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2913,"ondisk_log_size":2913,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":663,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2522256,"kb_used_data":85032,"kb_used_omap":13,"kb_used_meta":2436978,"kb_avail":747438960,"statfs":{"total":767960285184,"available":765377495040,"internally_reserved":0,"allocated":87072768,"data_stored":60932698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2495466176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":13,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003603"},"pg_stats":[{"pgid":"220.1f","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852052+0000","last_change":"2024-09-10T19:01:33.417711+0000","last_active":"2024-09-10T19:01:34.852052+0000","last_peered":"2024-09-10T19:01:34.852052+0000","last_clean":"2024-09-10T19:01:34.852052+0000","last_became_active":"2024-09-10T19:01:33.417397+0000","last_became_peered":"2024-09-10T19:01:33.417397+0000","last_unstale":"2024-09-10T19:01:34.852052+0000","last_undegraded":"2024-09-10T19:01:34.852052+0000","last_fullsized":"2024-09-10T19:01:34.852052+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:55.599283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.1e","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851318+0000","last_change":"2024-09-10T19:01:33.418508+0000","last_active":"2024-09-10T19:01:34.851318+0000","last_peered":"2024-09-10T19:01:34.851318+0000","last_clean":"2024-09-10T19:01:34.851318+0000","last_became_active":"2024-09-10T19:01:33.418160+0000","last_became_peered":"2024-09-10T19:01:33.418160+0000","last_unstale":"2024-09-10T19:01:34.851318+0000","last_undegraded":"2024-09-10T19:01:34.851318+0000","last_fullsized":"2024-09-10T19:01:34.851318+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:01.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.1d","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851479+0000","last_change":"2024-09-10T19:01:33.418175+0000","last_active":"2024-09-10T19:01:34.851479+0000","last_peered":"2024-09-10T19:01:34.851479+0000","last_clean":"2024-09-10T19:01:34.851479+0000","last_became_active":"2024-09-10T19:01:33.417675+0000","last_became_peered":"2024-09-10T19:01:33.417675+0000","last_unstale":"2024-09-10T19:01:34.851479+0000","last_undegraded":"2024-09-10T19:01:34.851479+0000","last_fullsized":"2024-09-10T19:01:34.851479+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.006564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.1c","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175129+0000","last_change":"2024-09-10T19:01:33.418973+0000","last_active":"2024-09-10T19:01:34.175129+0000","last_peered":"2024-09-10T19:01:34.175129+0000","last_clean":"2024-09-10T19:01:34.175129+0000","last_became_active":"2024-09-10T19:01:33.418701+0000","last_became_peered":"2024-09-10T19:01:33.418701+0000","last_unstale":"2024-09-10T19:01:34.175129+0000","last_undegraded":"2024-09-10T19:01:34.175129+0000","last_fullsized":"2024-09-10T19:01:34.175129+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:35.561202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"220.1b","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851179+0000","last_change":"2024-09-10T19:01:33.414548+0000","last_active":"2024-09-10T19:01:34.851179+0000","last_peered":"2024-09-10T19:01:34.851179+0000","last_clean":"2024-09-10T19:01:34.851179+0000","last_became_active":"2024-09-10T19:01:33.414231+0000","last_became_peered":"2024-09-10T19:01:33.414231+0000","last_unstale":"2024-09-10T19:01:34.851179+0000","last_undegraded":"2024-09-10T19:01:34.851179+0000","last_fullsized":"2024-09-10T19:01:34.851179+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.664618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.1a","version":"491'2","reported_seq":14,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.196908+0000","last_change":"2024-09-10T19:01:33.417521+0000","last_active":"2024-09-10T19:01:35.196908+0000","last_peered":"2024-09-10T19:01:35.196908+0000","last_clean":"2024-09-10T19:01:35.196908+0000","last_became_active":"2024-09-10T19:01:33.417227+0000","last_became_peered":"2024-09-10T19:01:33.417227+0000","last_unstale":"2024-09-10T19:01:35.196908+0000","last_undegraded":"2024-09-10T19:01:35.196908+0000","last_fullsized":"2024-09-10T19:01:35.196908+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.504410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"220.19","version":"491'4","reported_seq":15,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.197322+0000","last_change":"2024-09-10T19:01:33.415164+0000","last_active":"2024-09-10T19:01:35.197322+0000","last_peered":"2024-09-10T19:01:35.197322+0000","last_clean":"2024-09-10T19:01:35.197322+0000","last_became_active":"2024-09-10T19:01:33.414807+0000","last_became_peered":"2024-09-10T19:01:33.414807+0000","last_unstale":"2024-09-10T19:01:35.197322+0000","last_undegraded":"2024-09-10T19:01:35.197322+0000","last_fullsized":"2024-09-10T19:01:35.197322+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.658775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.18","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176107+0000","last_change":"2024-09-10T19:01:33.412946+0000","last_active":"2024-09-10T19:01:34.176107+0000","last_peered":"2024-09-10T19:01:34.176107+0000","last_clean":"2024-09-10T19:01:34.176107+0000","last_became_active":"2024-09-10T19:01:33.412395+0000","last_became_peered":"2024-09-10T19:01:33.412395+0000","last_unstale":"2024-09-10T19:01:34.176107+0000","last_undegraded":"2024-09-10T19:01:34.176107+0000","last_fullsized":"2024-09-10T19:01:34.176107+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.378345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"220.17","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181131+0000","last_change":"2024-09-10T19:01:33.411435+0000","last_active":"2024-09-10T19:01:34.181131+0000","last_peered":"2024-09-10T19:01:34.181131+0000","last_clean":"2024-09-10T19:01:34.181131+0000","last_became_active":"2024-09-10T19:01:33.411112+0000","last_became_peered":"2024-09-10T19:01:33.411112+0000","last_unstale":"2024-09-10T19:01:34.181131+0000","last_undegraded":"2024-09-10T19:01:34.181131+0000","last_fullsized":"2024-09-10T19:01:34.181131+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.542932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"220.16","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178854+0000","last_change":"2024-09-10T19:01:33.409539+0000","last_active":"2024-09-10T19:01:34.178854+0000","last_peered":"2024-09-10T19:01:34.178854+0000","last_clean":"2024-09-10T19:01:34.178854+0000","last_became_active":"2024-09-10T19:01:33.409261+0000","last_became_peered":"2024-09-10T19:01:33.409261+0000","last_unstale":"2024-09-10T19:01:34.178854+0000","last_undegraded":"2024-09-10T19:01:34.178854+0000","last_fullsized":"2024-09-10T19:01:34.178854+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.730090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"220.15","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853626+0000","last_change":"2024-09-10T19:01:33.416667+0000","last_active":"2024-09-10T19:01:34.853626+0000","last_peered":"2024-09-10T19:01:34.853626+0000","last_clean":"2024-09-10T19:01:34.853626+0000","last_became_active":"2024-09-10T19:01:33.416321+0000","last_became_peered":"2024-09-10T19:01:33.416321+0000","last_unstale":"2024-09-10T19:01:34.853626+0000","last_undegraded":"2024-09-10T19:01:34.853626+0000","last_fullsized":"2024-09-10T19:01:34.853626+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:35.549762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"220.14","version":"491'4","reported_seq":17,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.208494+0000","last_change":"2024-09-10T19:01:33.411946+0000","last_active":"2024-09-10T19:01:35.208494+0000","last_peered":"2024-09-10T19:01:35.208494+0000","last_clean":"2024-09-10T19:01:35.208494+0000","last_became_active":"2024-09-10T19:01:33.411650+0000","last_became_peered":"2024-09-10T19:01:33.411650+0000","last_unstale":"2024-09-10T19:01:35.208494+0000","last_undegraded":"2024-09-10T19:01:35.208494+0000","last_fullsized":"2024-09-10T19:01:35.208494+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.800252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"220.13","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851747+0000","last_change":"2024-09-10T19:01:33.414844+0000","last_active":"2024-09-10T19:01:34.851747+0000","last_peered":"2024-09-10T19:01:34.851747+0000","last_clean":"2024-09-10T19:01:34.851747+0000","last_became_active":"2024-09-10T19:01:33.414525+0000","last_became_peered":"2024-09-10T19:01:33.414525+0000","last_unstale":"2024-09-10T19:01:34.851747+0000","last_undegraded":"2024-09-10T19:01:34.851747+0000","last_fullsized":"2024-09-10T19:01:34.851747+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.898269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.12","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850347+0000","last_change":"2024-09-10T19:01:33.419651+0000","last_active":"2024-09-10T19:01:34.850347+0000","last_peered":"2024-09-10T19:01:34.850347+0000","last_clean":"2024-09-10T19:01:34.850347+0000","last_became_active":"2024-09-10T19:01:33.419319+0000","last_became_peered":"2024-09-10T19:01:33.419319+0000","last_unstale":"2024-09-10T19:01:34.850347+0000","last_undegraded":"2024-09-10T19:01:34.850347+0000","last_fullsized":"2024-09-10T19:01:34.850347+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:55.871016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.11","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181697+0000","last_change":"2024-09-10T19:01:33.405994+0000","last_active":"2024-09-10T19:01:34.181697+0000","last_peered":"2024-09-10T19:01:34.181697+0000","last_clean":"2024-09-10T19:01:34.181697+0000","last_became_active":"2024-09-10T19:01:33.404898+0000","last_became_peered":"2024-09-10T19:01:33.404898+0000","last_unstale":"2024-09-10T19:01:34.181697+0000","last_undegraded":"2024-09-10T19:01:34.181697+0000","last_fullsized":"2024-09-10T19:01:34.181697+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:40.800888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"220.10","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178907+0000","last_change":"2024-09-10T19:01:33.408258+0000","last_active":"2024-09-10T19:01:34.178907+0000","last_peered":"2024-09-10T19:01:34.178907+0000","last_clean":"2024-09-10T19:01:34.178907+0000","last_became_active":"2024-09-10T19:01:33.407964+0000","last_became_peered":"2024-09-10T19:01:33.407964+0000","last_unstale":"2024-09-10T19:01:34.178907+0000","last_undegraded":"2024-09-10T19:01:34.178907+0000","last_fullsized":"2024-09-10T19:01:34.178907+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:01.502050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"220.f","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852327+0000","last_change":"2024-09-10T19:01:33.418905+0000","last_active":"2024-09-10T19:01:34.852327+0000","last_peered":"2024-09-10T19:01:34.852327+0000","last_clean":"2024-09-10T19:01:34.852327+0000","last_became_active":"2024-09-10T19:01:33.418122+0000","last_became_peered":"2024-09-10T19:01:33.418122+0000","last_unstale":"2024-09-10T19:01:34.852327+0000","last_undegraded":"2024-09-10T19:01:34.852327+0000","last_fullsized":"2024-09-10T19:01:34.852327+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.713586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"220.e","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183703+0000","last_change":"2024-09-10T19:01:33.403434+0000","last_active":"2024-09-10T19:01:34.183703+0000","last_peered":"2024-09-10T19:01:34.183703+0000","last_clean":"2024-09-10T19:01:34.183703+0000","last_became_active":"2024-09-10T19:01:33.402937+0000","last_became_peered":"2024-09-10T19:01:33.402937+0000","last_unstale":"2024-09-10T19:01:34.183703+0000","last_undegraded":"2024-09-10T19:01:34.183703+0000","last_fullsized":"2024-09-10T19:01:34.183703+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:37.611927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"220.d","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180356+0000","last_change":"2024-09-10T19:01:33.412915+0000","last_active":"2024-09-10T19:01:34.180356+0000","last_peered":"2024-09-10T19:01:34.180356+0000","last_clean":"2024-09-10T19:01:34.180356+0000","last_became_active":"2024-09-10T19:01:33.412590+0000","last_became_peered":"2024-09-10T19:01:33.412590+0000","last_unstale":"2024-09-10T19:01:34.180356+0000","last_undegraded":"2024-09-10T19:01:34.180356+0000","last_fullsized":"2024-09-10T19:01:34.180356+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.913000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"220.c","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176473+0000","last_change":"2024-09-10T19:01:33.420363+0000","last_active":"2024-09-10T19:01:34.176473+0000","last_peered":"2024-09-10T19:01:34.176473+0000","last_clean":"2024-09-10T19:01:34.176473+0000","last_became_active":"2024-09-10T19:01:33.420035+0000","last_became_peered":"2024-09-10T19:01:33.420035+0000","last_unstale":"2024-09-10T19:01:34.176473+0000","last_undegraded":"2024-09-10T19:01:34.176473+0000","last_fullsized":"2024-09-10T19:01:34.176473+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.687840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"220.b","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179968+0000","last_change":"2024-09-10T19:01:33.407086+0000","last_active":"2024-09-10T19:01:34.179968+0000","last_peered":"2024-09-10T19:01:34.179968+0000","last_clean":"2024-09-10T19:01:34.179968+0000","last_became_active":"2024-09-10T19:01:33.406575+0000","last_became_peered":"2024-09-10T19:01:33.406575+0000","last_unstale":"2024-09-10T19:01:34.179968+0000","last_undegraded":"2024-09-10T19:01:34.179968+0000","last_fullsized":"2024-09-10T19:01:34.179968+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:55.466265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"220.a","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181894+0000","last_change":"2024-09-10T19:01:33.423997+0000","last_active":"2024-09-10T19:01:34.181894+0000","last_peered":"2024-09-10T19:01:34.181894+0000","last_clean":"2024-09-10T19:01:34.181894+0000","last_became_active":"2024-09-10T19:01:33.423219+0000","last_became_peered":"2024-09-10T19:01:33.423219+0000","last_unstale":"2024-09-10T19:01:34.181894+0000","last_undegraded":"2024-09-10T19:01:34.181894+0000","last_fullsized":"2024-09-10T19:01:34.181894+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.149264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"220.9","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173832+0000","last_change":"2024-09-10T19:01:33.420318+0000","last_active":"2024-09-10T19:01:34.173832+0000","last_peered":"2024-09-10T19:01:34.173832+0000","last_clean":"2024-09-10T19:01:34.173832+0000","last_became_active":"2024-09-10T19:01:33.420048+0000","last_became_peered":"2024-09-10T19:01:33.420048+0000","last_unstale":"2024-09-10T19:01:34.173832+0000","last_undegraded":"2024-09-10T19:01:34.173832+0000","last_fullsized":"2024-09-10T19:01:34.173832+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.133327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"220.8","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173108+0000","last_change":"2024-09-10T19:01:33.409851+0000","last_active":"2024-09-10T19:01:34.173108+0000","last_peered":"2024-09-10T19:01:34.173108+0000","last_clean":"2024-09-10T19:01:34.173108+0000","last_became_active":"2024-09-10T19:01:33.409578+0000","last_became_peered":"2024-09-10T19:01:33.409578+0000","last_unstale":"2024-09-10T19:01:34.173108+0000","last_undegraded":"2024-09-10T19:01:34.173108+0000","last_fullsized":"2024-09-10T19:01:34.173108+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.110558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"220.7","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183228+0000","last_change":"2024-09-10T19:01:33.421861+0000","last_active":"2024-09-10T19:01:34.183228+0000","last_peered":"2024-09-10T19:01:34.183228+0000","last_clean":"2024-09-10T19:01:34.183228+0000","last_became_active":"2024-09-10T19:01:33.421578+0000","last_became_peered":"2024-09-10T19:01:33.421578+0000","last_unstale":"2024-09-10T19:01:34.183228+0000","last_undegraded":"2024-09-10T19:01:34.183228+0000","last_fullsized":"2024-09-10T19:01:34.183228+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.986011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"220.6","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183188+0000","last_change":"2024-09-10T19:01:33.416936+0000","last_active":"2024-09-10T19:01:34.183188+0000","last_peered":"2024-09-10T19:01:34.183188+0000","last_clean":"2024-09-10T19:01:34.183188+0000","last_became_active":"2024-09-10T19:01:33.416634+0000","last_became_peered":"2024-09-10T19:01:33.416634+0000","last_unstale":"2024-09-10T19:01:34.183188+0000","last_undegraded":"2024-09-10T19:01:34.183188+0000","last_fullsized":"2024-09-10T19:01:34.183188+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:32.499416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"220.5","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182504+0000","last_change":"2024-09-10T19:01:33.421476+0000","last_active":"2024-09-10T19:01:34.182504+0000","last_peered":"2024-09-10T19:01:34.182504+0000","last_clean":"2024-09-10T19:01:34.182504+0000","last_became_active":"2024-09-10T19:01:33.421151+0000","last_became_peered":"2024-09-10T19:01:33.421151+0000","last_unstale":"2024-09-10T19:01:34.182504+0000","last_undegraded":"2024-09-10T19:01:34.182504+0000","last_fullsized":"2024-09-10T19:01:34.182504+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:54.915607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"220.4","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180958+0000","last_change":"2024-09-10T19:01:33.414303+0000","last_active":"2024-09-10T19:01:34.180958+0000","last_peered":"2024-09-10T19:01:34.180958+0000","last_clean":"2024-09-10T19:01:34.180958+0000","last_became_active":"2024-09-10T19:01:33.414009+0000","last_became_peered":"2024-09-10T19:01:33.414009+0000","last_unstale":"2024-09-10T19:01:34.180958+0000","last_undegraded":"2024-09-10T19:01:34.180958+0000","last_fullsized":"2024-09-10T19:01:34.180958+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:48.446372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"220.3","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182016+0000","last_change":"2024-09-10T19:01:33.418018+0000","last_active":"2024-09-10T19:01:34.182016+0000","last_peered":"2024-09-10T19:01:34.182016+0000","last_clean":"2024-09-10T19:01:34.182016+0000","last_became_active":"2024-09-10T19:01:33.417746+0000","last_became_peered":"2024-09-10T19:01:33.417746+0000","last_unstale":"2024-09-10T19:01:34.182016+0000","last_undegraded":"2024-09-10T19:01:34.182016+0000","last_fullsized":"2024-09-10T19:01:34.182016+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:57.713774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"220.2","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181988+0000","last_change":"2024-09-10T19:01:33.421369+0000","last_active":"2024-09-10T19:01:34.181988+0000","last_peered":"2024-09-10T19:01:34.181988+0000","last_clean":"2024-09-10T19:01:34.181988+0000","last_became_active":"2024-09-10T19:01:33.419403+0000","last_became_peered":"2024-09-10T19:01:33.419403+0000","last_unstale":"2024-09-10T19:01:34.181988+0000","last_undegraded":"2024-09-10T19:01:34.181988+0000","last_fullsized":"2024-09-10T19:01:34.181988+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.885468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"220.1","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181841+0000","last_change":"2024-09-10T19:01:33.403848+0000","last_active":"2024-09-10T19:01:34.181841+0000","last_peered":"2024-09-10T19:01:34.181841+0000","last_clean":"2024-09-10T19:01:34.181841+0000","last_became_active":"2024-09-10T19:01:33.403417+0000","last_became_peered":"2024-09-10T19:01:33.403417+0000","last_unstale":"2024-09-10T19:01:34.181841+0000","last_undegraded":"2024-09-10T19:01:34.181841+0000","last_fullsized":"2024-09-10T19:01:34.181841+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.268064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"220.0","version":"0'0","reported_seq":11,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.849633+0000","last_change":"2024-09-10T19:01:33.414585+0000","last_active":"2024-09-10T19:01:34.849633+0000","last_peered":"2024-09-10T19:01:34.849633+0000","last_clean":"2024-09-10T19:01:34.849633+0000","last_became_active":"2024-09-10T19:01:33.414116+0000","last_became_peered":"2024-09-10T19:01:33.414116+0000","last_unstale":"2024-09-10T19:01:34.849633+0000","last_undegraded":"2024-09-10T19:01:34.849633+0000","last_fullsized":"2024-09-10T19:01:34.849633+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":488,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:32.374095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.374095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:01.928595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":469,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854517+0000","last_change":"2024-09-10T19:00:48.295345+0000","last_active":"2024-09-10T19:01:34.854517+0000","last_peered":"2024-09-10T19:01:34.854517+0000","last_clean":"2024-09-10T19:01:34.854517+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:01:34.854517+0000","last_undegraded":"2024-09-10T19:01:34.854517+0000","last_fullsized":"2024-09-10T19:01:34.854517+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'39","last_scrub_stamp":"2024-09-10T18:58:52.881834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.881834+0000","objects_scrubbed":3,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029256130000000001,"objects_trimmed":0,"snaptrim_duration":0.00067620899999999999,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"475'72","reported_seq":515,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853363+0000","last_change":"2024-09-10T19:01:25.547641+0000","last_active":"2024-09-10T19:01:34.853363+0000","last_peered":"2024-09-10T19:01:34.853363+0000","last_clean":"2024-09-10T19:01:34.853363+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:01:34.853363+0000","last_undegraded":"2024-09-10T19:01:34.853363+0000","last_fullsized":"2024-09-10T19:01:34.853363+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'72","last_scrub_stamp":"2024-09-10T19:01:25.547476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.547476+0000","objects_scrubbed":10,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.233345+0000","scrub_duration":0.0061775670000000001,"objects_trimmed":0,"snaptrim_duration":0.00056750799999999999,"stat_sum":{"num_bytes":74,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"425'25","reported_seq":476,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851764+0000","last_change":"2024-09-10T19:00:48.280317+0000","last_active":"2024-09-10T19:01:34.851764+0000","last_peered":"2024-09-10T19:01:34.851764+0000","last_clean":"2024-09-10T19:01:34.851764+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:01:34.851764+0000","last_undegraded":"2024-09-10T19:01:34.851764+0000","last_fullsized":"2024-09-10T19:01:34.851764+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'25","last_scrub_stamp":"2024-09-10T19:00:03.071902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.071902+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041116920000000001,"objects_trimmed":0,"snaptrim_duration":0.002538663,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":496,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177947+0000","last_change":"2024-09-10T19:01:28.876894+0000","last_active":"2024-09-10T19:01:34.177947+0000","last_peered":"2024-09-10T19:01:34.177947+0000","last_clean":"2024-09-10T19:01:34.177947+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:01:34.177947+0000","last_undegraded":"2024-09-10T19:01:34.177947+0000","last_fullsized":"2024-09-10T19:01:34.177947+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T19:01:28.876707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.876707+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:37.578566+0000","scrub_duration":0.004224579,"objects_trimmed":0,"snaptrim_duration":0.0016033009999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":501,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180013+0000","last_change":"2024-09-10T19:00:48.297711+0000","last_active":"2024-09-10T19:01:34.180013+0000","last_peered":"2024-09-10T19:01:34.180013+0000","last_clean":"2024-09-10T19:01:34.180013+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:01:34.180013+0000","last_undegraded":"2024-09-10T19:01:34.180013+0000","last_fullsized":"2024-09-10T19:01:34.180013+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T18:59:02.768814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.768814+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045607750000000004,"objects_trimmed":0,"snaptrim_duration":0.0018816099999999999,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850833+0000","last_change":"2024-09-10T19:00:48.282302+0000","last_active":"2024-09-10T19:01:34.850833+0000","last_peered":"2024-09-10T19:01:34.850833+0000","last_clean":"2024-09-10T19:01:34.850833+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:01:34.850833+0000","last_undegraded":"2024-09-10T19:01:34.850833+0000","last_fullsized":"2024-09-10T19:01:34.850833+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T18:59:58.060206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.060206+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042048110000000001,"objects_trimmed":0,"snaptrim_duration":0.00071753500000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"454'47","reported_seq":500,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174333+0000","last_change":"2024-09-10T19:00:48.181330+0000","last_active":"2024-09-10T19:01:34.174333+0000","last_peered":"2024-09-10T19:01:34.174333+0000","last_clean":"2024-09-10T19:01:34.174333+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:01:34.174333+0000","last_undegraded":"2024-09-10T19:01:34.174333+0000","last_fullsized":"2024-09-10T19:01:34.174333+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'45","last_scrub_stamp":"2024-09-10T19:00:30.901206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.901206+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:55.729039+0000","scrub_duration":0.0053195880000000001,"objects_trimmed":0,"snaptrim_duration":0.0025425780000000002,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"464'120","reported_seq":544,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181580+0000","last_change":"2024-09-10T19:00:50.283591+0000","last_active":"2024-09-10T19:01:34.181580+0000","last_peered":"2024-09-10T19:01:34.181580+0000","last_clean":"2024-09-10T19:01:34.181580+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:01:34.181580+0000","last_undegraded":"2024-09-10T19:01:34.181580+0000","last_fullsized":"2024-09-10T19:01:34.181580+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T19:00:13.172545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.172545+0000","objects_scrubbed":7,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057922160000000002,"objects_trimmed":0,"snaptrim_duration":1.070613525,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"428'23","reported_seq":467,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180894+0000","last_change":"2024-09-10T19:00:48.179573+0000","last_active":"2024-09-10T19:01:34.180894+0000","last_peered":"2024-09-10T19:01:34.180894+0000","last_clean":"2024-09-10T19:01:34.180894+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:01:34.180894+0000","last_undegraded":"2024-09-10T19:01:34.180894+0000","last_fullsized":"2024-09-10T19:01:34.180894+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'23","last_scrub_stamp":"2024-09-10T19:00:20.541045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:20.541045+0000","objects_scrubbed":4,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.871740+0000","scrub_duration":0.0049668359999999996,"objects_trimmed":0,"snaptrim_duration":0.0012828710000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"387'24","reported_seq":470,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178927+0000","last_change":"2024-09-10T19:00:48.298750+0000","last_active":"2024-09-10T19:01:34.178927+0000","last_peered":"2024-09-10T19:01:34.178927+0000","last_clean":"2024-09-10T19:01:34.178927+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:01:34.178927+0000","last_undegraded":"2024-09-10T19:01:34.178927+0000","last_fullsized":"2024-09-10T19:01:34.178927+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.0016237720000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"428'79","reported_seq":526,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182312+0000","last_change":"2024-09-10T19:00:48.183035+0000","last_active":"2024-09-10T19:01:34.182312+0000","last_peered":"2024-09-10T19:01:34.182312+0000","last_clean":"2024-09-10T19:01:34.182312+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:01:34.182312+0000","last_undegraded":"2024-09-10T19:01:34.182312+0000","last_fullsized":"2024-09-10T19:01:34.182312+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'79","last_scrub_stamp":"2024-09-10T19:00:27.181327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.181327+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:35.433070+0000","scrub_duration":0.005077341,"objects_trimmed":0,"snaptrim_duration":0.002291235,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"434'100","reported_seq":542,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178056+0000","last_change":"2024-09-10T19:00:48.298137+0000","last_active":"2024-09-10T19:01:34.178056+0000","last_peered":"2024-09-10T19:01:34.178056+0000","last_clean":"2024-09-10T19:01:34.178056+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:01:34.178056+0000","last_undegraded":"2024-09-10T19:01:34.178056+0000","last_fullsized":"2024-09-10T19:01:34.178056+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'97","last_scrub_stamp":"2024-09-10T18:59:01.793673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.793673+0000","objects_scrubbed":8,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055476459999999998,"objects_trimmed":0,"snaptrim_duration":0.002046174,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"448'4","reported_seq":466,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182056+0000","last_change":"2024-09-10T19:00:48.182407+0000","last_active":"2024-09-10T19:01:34.182056+0000","last_peered":"2024-09-10T19:01:34.182056+0000","last_clean":"2024-09-10T19:01:34.182056+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:01:34.182056+0000","last_undegraded":"2024-09-10T19:01:34.182056+0000","last_fullsized":"2024-09-10T19:01:34.182056+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'4","last_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_deep_scrub":"448'4","last_deep_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.186496+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:43.223489+0000","scrub_duration":0.0071427519999999996,"objects_trimmed":0,"snaptrim_duration":0.00085743500000000003,"stat_sum":{"num_bytes":12,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":482,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182104+0000","last_change":"2024-09-10T19:01:16.522626+0000","last_active":"2024-09-10T19:01:34.182104+0000","last_peered":"2024-09-10T19:01:34.182104+0000","last_clean":"2024-09-10T19:01:34.182104+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:01:34.182104+0000","last_undegraded":"2024-09-10T19:01:34.182104+0000","last_fullsized":"2024-09-10T19:01:34.182104+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T19:01:16.522456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.522456+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:26.628007+0000","scrub_duration":0.0048536079999999997,"objects_trimmed":0,"snaptrim_duration":0.0024268039999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"452'56","reported_seq":497,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.172958+0000","last_change":"2024-09-10T19:00:48.285951+0000","last_active":"2024-09-10T19:01:34.172958+0000","last_peered":"2024-09-10T19:01:34.172958+0000","last_clean":"2024-09-10T19:01:34.172958+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T19:01:34.172958+0000","last_undegraded":"2024-09-10T19:01:34.172958+0000","last_fullsized":"2024-09-10T19:01:34.172958+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'56","last_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_deep_scrub":"452'56","last_deep_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.174126+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.989849+0000","scrub_duration":0.011240307,"objects_trimmed":0,"snaptrim_duration":0.00098096800000000003,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":476,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173380+0000","last_change":"2024-09-10T19:01:33.818709+0000","last_active":"2024-09-10T19:01:34.173380+0000","last_peered":"2024-09-10T19:01:34.173380+0000","last_clean":"2024-09-10T19:01:34.173380+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:01:34.173380+0000","last_undegraded":"2024-09-10T19:01:34.173380+0000","last_fullsized":"2024-09-10T19:01:34.173380+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'35","last_scrub_stamp":"2024-09-10T19:01:33.818552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.818552+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.869584+0000","scrub_duration":0.0033660140000000001,"objects_trimmed":0,"snaptrim_duration":0.0039003229999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"431'70","reported_seq":509,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183666+0000","last_change":"2024-09-10T19:00:48.276527+0000","last_active":"2024-09-10T19:01:34.183666+0000","last_peered":"2024-09-10T19:01:34.183666+0000","last_clean":"2024-09-10T19:01:34.183666+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:01:34.183666+0000","last_undegraded":"2024-09-10T19:01:34.183666+0000","last_fullsized":"2024-09-10T19:01:34.183666+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'70","last_scrub_stamp":"2024-09-10T19:00:18.252236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.252236+0000","objects_scrubbed":7,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064748799999999997,"objects_trimmed":0,"snaptrim_duration":0.00074153100000000003,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"484'136","reported_seq":561,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180610+0000","last_change":"2024-09-10T19:00:48.179467+0000","last_active":"2024-09-10T19:01:34.180610+0000","last_peered":"2024-09-10T19:01:34.180610+0000","last_clean":"2024-09-10T19:01:34.180610+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:01:34.180610+0000","last_undegraded":"2024-09-10T19:01:34.180610+0000","last_fullsized":"2024-09-10T19:01:34.180610+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'119","last_scrub_stamp":"2024-09-10T19:00:12.403246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.403246+0000","objects_scrubbed":7,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056039189999999997,"objects_trimmed":0,"snaptrim_duration":0.001433337,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":43,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"491'34","reported_seq":481,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.203701+0000","last_change":"2024-09-10T19:00:48.287101+0000","last_active":"2024-09-10T19:01:35.203701+0000","last_peered":"2024-09-10T19:01:35.203701+0000","last_clean":"2024-09-10T19:01:35.203701+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T19:01:35.203701+0000","last_undegraded":"2024-09-10T19:01:35.203701+0000","last_fullsized":"2024-09-10T19:01:35.203701+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.00069466700000000005,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"377'23","reported_seq":469,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182819+0000","last_change":"2024-09-10T19:01:25.384905+0000","last_active":"2024-09-10T19:01:34.182819+0000","last_peered":"2024-09-10T19:01:34.182819+0000","last_clean":"2024-09-10T19:01:34.182819+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:01:34.182819+0000","last_undegraded":"2024-09-10T19:01:34.182819+0000","last_fullsized":"2024-09-10T19:01:34.182819+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'23","last_scrub_stamp":"2024-09-10T19:01:25.384788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.384788+0000","objects_scrubbed":3,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:51.492529+0000","scrub_duration":0.0062438620000000002,"objects_trimmed":0,"snaptrim_duration":0.001841777,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":478,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180158+0000","last_change":"2024-09-10T19:00:48.298164+0000","last_active":"2024-09-10T19:01:34.180158+0000","last_peered":"2024-09-10T19:01:34.180158+0000","last_clean":"2024-09-10T19:01:34.180158+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:01:34.180158+0000","last_undegraded":"2024-09-10T19:01:34.180158+0000","last_fullsized":"2024-09-10T19:01:34.180158+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'36","last_scrub_stamp":"2024-09-10T19:00:28.730904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:28.730904+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.146970+0000","scrub_duration":0.0045585440000000003,"objects_trimmed":0,"snaptrim_duration":0.001811174,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"434'20","reported_seq":479,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182015+0000","last_change":"2024-09-10T19:00:48.180109+0000","last_active":"2024-09-10T19:01:34.182015+0000","last_peered":"2024-09-10T19:01:34.182015+0000","last_clean":"2024-09-10T19:01:34.182015+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:01:34.182015+0000","last_undegraded":"2024-09-10T19:01:34.182015+0000","last_fullsized":"2024-09-10T19:01:34.182015+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'20","last_scrub_stamp":"2024-09-10T19:00:18.161542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.161542+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.711115+0000","scrub_duration":0.0033164079999999999,"objects_trimmed":0,"snaptrim_duration":0.00088964800000000004,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":459,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181846+0000","last_change":"2024-09-10T19:00:48.179437+0000","last_active":"2024-09-10T19:01:34.181846+0000","last_peered":"2024-09-10T19:01:34.181846+0000","last_clean":"2024-09-10T19:01:34.181846+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:01:34.181846+0000","last_undegraded":"2024-09-10T19:01:34.181846+0000","last_fullsized":"2024-09-10T19:01:34.181846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.225132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.225132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:55.143010+0000","scrub_duration":0.0035652290000000001,"objects_trimmed":0,"snaptrim_duration":0.00064644099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"234'51","reported_seq":491,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178921+0000","last_change":"2024-09-10T19:00:48.177743+0000","last_active":"2024-09-10T19:01:34.178921+0000","last_peered":"2024-09-10T19:01:34.178921+0000","last_clean":"2024-09-10T19:01:34.178921+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:01:34.178921+0000","last_undegraded":"2024-09-10T19:01:34.178921+0000","last_fullsized":"2024-09-10T19:01:34.178921+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T19:00:03.402967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.402967+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043031060000000001,"objects_trimmed":0,"snaptrim_duration":0.001434334,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"409'160","reported_seq":608,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183229+0000","last_change":"2024-09-10T19:00:48.185601+0000","last_active":"2024-09-10T19:01:34.183229+0000","last_peered":"2024-09-10T19:01:34.183229+0000","last_clean":"2024-09-10T19:01:34.183229+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:01:34.183229+0000","last_undegraded":"2024-09-10T19:01:34.183229+0000","last_fullsized":"2024-09-10T19:01:34.183229+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'160","last_scrub_stamp":"2024-09-10T19:00:30.192760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.192760+0000","objects_scrubbed":12,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.851352+0000","scrub_duration":0.0047442129999999997,"objects_trimmed":0,"snaptrim_duration":0.0011147100000000001,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"388'99","reported_seq":595,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851612+0000","last_change":"2024-09-10T19:01:28.217626+0000","last_active":"2024-09-10T19:01:34.851612+0000","last_peered":"2024-09-10T19:01:34.851612+0000","last_clean":"2024-09-10T19:01:34.851612+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:01:34.851612+0000","last_undegraded":"2024-09-10T19:01:34.851612+0000","last_fullsized":"2024-09-10T19:01:34.851612+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T19:01:28.217457+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.217457+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.532699+0000","scrub_duration":0.0059368959999999997,"objects_trimmed":0,"snaptrim_duration":0.001143469,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":492,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183352+0000","last_change":"2024-09-10T19:00:48.275852+0000","last_active":"2024-09-10T19:01:34.183352+0000","last_peered":"2024-09-10T19:01:34.183352+0000","last_clean":"2024-09-10T19:01:34.183352+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:01:34.183352+0000","last_undegraded":"2024-09-10T19:01:34.183352+0000","last_fullsized":"2024-09-10T19:01:34.183352+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T19:00:33.093969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.093969+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045796200000000004,"objects_trimmed":0,"snaptrim_duration":0.0010303630000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"472'113","reported_seq":546,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851895+0000","last_change":"2024-09-10T19:00:48.280852+0000","last_active":"2024-09-10T19:01:34.851895+0000","last_peered":"2024-09-10T19:01:34.851895+0000","last_clean":"2024-09-10T19:01:34.851895+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:01:34.851895+0000","last_undegraded":"2024-09-10T19:01:34.851895+0000","last_fullsized":"2024-09-10T19:01:34.851895+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'107","last_scrub_stamp":"2024-09-10T19:00:06.091203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:06.091203+0000","objects_scrubbed":7,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051957840000000002,"objects_trimmed":0,"snaptrim_duration":0.0025913289999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"431'106","reported_seq":542,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176858+0000","last_change":"2024-09-10T19:01:28.916818+0000","last_active":"2024-09-10T19:01:34.176858+0000","last_peered":"2024-09-10T19:01:34.176858+0000","last_clean":"2024-09-10T19:01:34.176858+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:01:34.176858+0000","last_undegraded":"2024-09-10T19:01:34.176858+0000","last_fullsized":"2024-09-10T19:01:34.176858+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'106","last_scrub_stamp":"2024-09-10T19:01:28.916641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.916641+0000","objects_scrubbed":9,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.061376+0000","scrub_duration":0.0052486110000000002,"objects_trimmed":0,"snaptrim_duration":0.0012640469999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180063+0000","last_change":"2024-09-10T19:00:48.180783+0000","last_active":"2024-09-10T19:01:34.180063+0000","last_peered":"2024-09-10T19:01:34.180063+0000","last_clean":"2024-09-10T19:01:34.180063+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:01:34.180063+0000","last_undegraded":"2024-09-10T19:01:34.180063+0000","last_fullsized":"2024-09-10T19:01:34.180063+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T19:00:15.443270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.443270+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.357357+0000","scrub_duration":0.0044317829999999999,"objects_trimmed":0,"snaptrim_duration":0.00053455899999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"409'74","reported_seq":519,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182147+0000","last_change":"2024-09-10T19:00:48.271955+0000","last_active":"2024-09-10T19:01:34.182147+0000","last_peered":"2024-09-10T19:01:34.182147+0000","last_clean":"2024-09-10T19:01:34.182147+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:01:34.182147+0000","last_undegraded":"2024-09-10T19:01:34.182147+0000","last_fullsized":"2024-09-10T19:01:34.182147+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'74","last_scrub_stamp":"2024-09-10T19:00:31.135496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.135496+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.789352+0000","scrub_duration":0.0081214419999999996,"objects_trimmed":0,"snaptrim_duration":0.001038976,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":541,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181657+0000","last_change":"2024-09-10T19:00:48.185243+0000","last_active":"2024-09-10T19:01:34.181657+0000","last_peered":"2024-09-10T19:01:34.181657+0000","last_clean":"2024-09-10T19:01:34.181657+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:01:34.181657+0000","last_undegraded":"2024-09-10T19:01:34.181657+0000","last_fullsized":"2024-09-10T19:01:34.181657+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T19:00:14.192507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.192507+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.168955+0000","scrub_duration":0.0088236210000000002,"objects_trimmed":0,"snaptrim_duration":0.00087170800000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":464,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176216+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T19:01:34.176216+0000","last_peered":"2024-09-10T19:01:34.176216+0000","last_clean":"2024-09-10T19:01:34.176216+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:01:34.176216+0000","last_undegraded":"2024-09-10T19:01:34.176216+0000","last_fullsized":"2024-09-10T19:01:34.176216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854900+0000","last_change":"2024-09-10T18:59:59.106249+0000","last_active":"2024-09-10T19:01:34.854900+0000","last_peered":"2024-09-10T19:01:34.854900+0000","last_clean":"2024-09-10T19:01:34.854900+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:01:34.854900+0000","last_undegraded":"2024-09-10T19:01:34.854900+0000","last_fullsized":"2024-09-10T19:01:34.854900+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.106063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.106063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034646490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":498,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174768+0000","last_change":"2024-09-10T19:00:10.940056+0000","last_active":"2024-09-10T19:01:34.174768+0000","last_peered":"2024-09-10T19:01:34.174768+0000","last_clean":"2024-09-10T19:01:34.174768+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:01:34.174768+0000","last_undegraded":"2024-09-10T19:01:34.174768+0000","last_fullsized":"2024-09-10T19:01:34.174768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:00:10.939877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.939877+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051523760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":476,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176309+0000","last_change":"2024-09-10T18:58:52.233066+0000","last_active":"2024-09-10T19:01:34.176309+0000","last_peered":"2024-09-10T19:01:34.176309+0000","last_clean":"2024-09-10T19:01:34.176309+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:01:34.176309+0000","last_undegraded":"2024-09-10T19:01:34.176309+0000","last_fullsized":"2024-09-10T19:01:34.176309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:52.232903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:52.232903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030918450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":390,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181938+0000","last_change":"2024-09-10T18:59:53.417704+0000","last_active":"2024-09-10T19:01:34.181938+0000","last_peered":"2024-09-10T19:01:34.181938+0000","last_clean":"2024-09-10T19:01:34.181938+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:01:34.181938+0000","last_undegraded":"2024-09-10T19:01:34.181938+0000","last_fullsized":"2024-09-10T19:01:34.181938+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.417551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.417551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034045260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":490,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175695+0000","last_change":"2024-09-10T19:01:29.910705+0000","last_active":"2024-09-10T19:01:34.175695+0000","last_peered":"2024-09-10T19:01:34.175695+0000","last_clean":"2024-09-10T19:01:34.175695+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:01:34.175695+0000","last_undegraded":"2024-09-10T19:01:34.175695+0000","last_fullsized":"2024-09-10T19:01:34.175695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.910553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.910553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.621492+0000","scrub_duration":0.0032680439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":506,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854665+0000","last_change":"2024-09-10T19:00:03.037143+0000","last_active":"2024-09-10T19:01:34.854665+0000","last_peered":"2024-09-10T19:01:34.854665+0000","last_clean":"2024-09-10T19:01:34.854665+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:01:34.854665+0000","last_undegraded":"2024-09-10T19:01:34.854665+0000","last_fullsized":"2024-09-10T19:01:34.854665+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:03.036974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.036974+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043269010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":485,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.172690+0000","last_change":"2024-09-10T19:00:01.347416+0000","last_active":"2024-09-10T19:01:34.172690+0000","last_peered":"2024-09-10T19:01:34.172690+0000","last_clean":"2024-09-10T19:01:34.172690+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:01:34.172690+0000","last_undegraded":"2024-09-10T19:01:34.172690+0000","last_fullsized":"2024-09-10T19:01:34.172690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:01.347241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.347241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003764398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175094+0000","last_change":"2024-09-10T19:01:22.090669+0000","last_active":"2024-09-10T19:01:34.175094+0000","last_peered":"2024-09-10T19:01:34.175094+0000","last_clean":"2024-09-10T19:01:34.175094+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T19:01:34.175094+0000","last_undegraded":"2024-09-10T19:01:34.175094+0000","last_fullsized":"2024-09-10T19:01:34.175094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.090498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.090498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.520013+0000","scrub_duration":0.0033351800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853329+0000","last_change":"2024-09-10T19:01:15.552219+0000","last_active":"2024-09-10T19:01:34.853329+0000","last_peered":"2024-09-10T19:01:34.853329+0000","last_clean":"2024-09-10T19:01:34.853329+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:01:34.853329+0000","last_undegraded":"2024-09-10T19:01:34.853329+0000","last_fullsized":"2024-09-10T19:01:34.853329+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.552108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.552108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:30.190395+0000","scrub_duration":0.0079911449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852123+0000","last_change":"2024-09-10T19:01:25.221951+0000","last_active":"2024-09-10T19:01:34.852123+0000","last_peered":"2024-09-10T19:01:34.852123+0000","last_clean":"2024-09-10T19:01:34.852123+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:01:34.852123+0000","last_undegraded":"2024-09-10T19:01:34.852123+0000","last_fullsized":"2024-09-10T19:01:34.852123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:25.221762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.221762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.213404+0000","scrub_duration":0.003571613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":521,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183046+0000","last_change":"2024-09-10T19:00:09.145311+0000","last_active":"2024-09-10T19:01:34.183046+0000","last_peered":"2024-09-10T19:01:34.183046+0000","last_clean":"2024-09-10T19:01:34.183046+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:01:34.183046+0000","last_undegraded":"2024-09-10T19:01:34.183046+0000","last_fullsized":"2024-09-10T19:01:34.183046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:09.145229+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.145229+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090468159999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":485,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.223828+0000","last_change":"2024-09-10T19:01:34.223664+0000","last_active":"2024-09-10T19:01:34.223828+0000","last_peered":"2024-09-10T19:01:34.223828+0000","last_clean":"2024-09-10T19:01:34.223828+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T19:01:34.223828+0000","last_undegraded":"2024-09-10T19:01:34.223828+0000","last_fullsized":"2024-09-10T19:01:34.223828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:34.223516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.223516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.044571+0000","scrub_duration":0.0033528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":470,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182052+0000","last_change":"2024-09-10T19:00:03.208547+0000","last_active":"2024-09-10T19:01:34.182052+0000","last_peered":"2024-09-10T19:01:34.182052+0000","last_clean":"2024-09-10T19:01:34.182052+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:01:34.182052+0000","last_undegraded":"2024-09-10T19:01:34.182052+0000","last_fullsized":"2024-09-10T19:01:34.182052+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.208468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.208468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019202379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180232+0000","last_change":"2024-09-10T19:01:30.825681+0000","last_active":"2024-09-10T19:01:34.180232+0000","last_peered":"2024-09-10T19:01:34.180232+0000","last_clean":"2024-09-10T19:01:34.180232+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:01:34.180232+0000","last_undegraded":"2024-09-10T19:01:34.180232+0000","last_fullsized":"2024-09-10T19:01:34.180232+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.825536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.825536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.619051+0000","scrub_duration":0.0034703239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":489,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853292+0000","last_change":"2024-09-10T19:01:33.189562+0000","last_active":"2024-09-10T19:01:34.853292+0000","last_peered":"2024-09-10T19:01:34.853292+0000","last_clean":"2024-09-10T19:01:34.853292+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:01:34.853292+0000","last_undegraded":"2024-09-10T19:01:34.853292+0000","last_fullsized":"2024-09-10T19:01:34.853292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:33.189399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.189399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.714879+0000","scrub_duration":0.0037148820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":528,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183108+0000","last_change":"2024-09-10T19:01:33.342581+0000","last_active":"2024-09-10T19:01:34.183108+0000","last_peered":"2024-09-10T19:01:34.183108+0000","last_clean":"2024-09-10T19:01:34.183108+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:01:34.183108+0000","last_undegraded":"2024-09-10T19:01:34.183108+0000","last_fullsized":"2024-09-10T19:01:34.183108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:01:33.342427+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.342427+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.239579+0000","scrub_duration":0.004310543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":485,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174134+0000","last_change":"2024-09-10T19:00:35.255523+0000","last_active":"2024-09-10T19:01:34.174134+0000","last_peered":"2024-09-10T19:01:34.174134+0000","last_clean":"2024-09-10T19:01:34.174134+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T19:01:34.174134+0000","last_undegraded":"2024-09-10T19:01:34.174134+0000","last_fullsized":"2024-09-10T19:01:34.174134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:35.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:00:35.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:46.150006+0000","scrub_duration":0.0028367790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":471,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851917+0000","last_change":"2024-09-10T18:59:04.788976+0000","last_active":"2024-09-10T19:01:34.851917+0000","last_peered":"2024-09-10T19:01:34.851917+0000","last_clean":"2024-09-10T19:01:34.851917+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:01:34.851917+0000","last_undegraded":"2024-09-10T19:01:34.851917+0000","last_fullsized":"2024-09-10T19:01:34.851917+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:04.788729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.788729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038128509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":395,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174582+0000","last_change":"2024-09-10T19:00:14.868536+0000","last_active":"2024-09-10T19:01:34.174582+0000","last_peered":"2024-09-10T19:01:34.174582+0000","last_clean":"2024-09-10T19:01:34.174582+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:01:34.174582+0000","last_undegraded":"2024-09-10T19:01:34.174582+0000","last_fullsized":"2024-09-10T19:01:34.174582+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.868377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.868377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036007919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":539,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180254+0000","last_change":"2024-09-10T19:00:16.474150+0000","last_active":"2024-09-10T19:01:34.180254+0000","last_peered":"2024-09-10T19:01:34.180254+0000","last_clean":"2024-09-10T19:01:34.180254+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:01:34.180254+0000","last_undegraded":"2024-09-10T19:01:34.180254+0000","last_fullsized":"2024-09-10T19:01:34.180254+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:00:16.474006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.474006+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.225107+0000","scrub_duration":0.0057543610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.169559+0000","last_change":"2024-09-10T19:01:35.169404+0000","last_active":"2024-09-10T19:01:35.169559+0000","last_peered":"2024-09-10T19:01:35.169559+0000","last_clean":"2024-09-10T19:01:35.169559+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:01:35.169559+0000","last_undegraded":"2024-09-10T19:01:35.169559+0000","last_fullsized":"2024-09-10T19:01:35.169559+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.169217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.169217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:03.816865+0000","scrub_duration":0.0032414449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":643,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175750+0000","last_change":"2024-09-10T19:01:23.124779+0000","last_active":"2024-09-10T19:01:34.175750+0000","last_peered":"2024-09-10T19:01:34.175750+0000","last_clean":"2024-09-10T19:01:34.175750+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T19:01:34.175750+0000","last_undegraded":"2024-09-10T19:01:34.175750+0000","last_fullsized":"2024-09-10T19:01:34.175750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T19:01:23.124615+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.124615+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:27.487775+0000","scrub_duration":0.0037027000000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":492,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183424+0000","last_change":"2024-09-10T19:00:33.141061+0000","last_active":"2024-09-10T19:01:34.183424+0000","last_peered":"2024-09-10T19:01:34.183424+0000","last_clean":"2024-09-10T19:01:34.183424+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:01:34.183424+0000","last_undegraded":"2024-09-10T19:01:34.183424+0000","last_fullsized":"2024-09-10T19:01:34.183424+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.140938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.140938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:02.048809+0000","scrub_duration":0.0029466620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":393,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180632+0000","last_change":"2024-09-10T18:59:59.486802+0000","last_active":"2024-09-10T19:01:34.180632+0000","last_peered":"2024-09-10T19:01:34.180632+0000","last_clean":"2024-09-10T19:01:34.180632+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:01:34.180632+0000","last_undegraded":"2024-09-10T19:01:34.180632+0000","last_fullsized":"2024-09-10T19:01:34.180632+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.486653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.486653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003394902,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":507,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176726+0000","last_change":"2024-09-10T18:59:57.094883+0000","last_active":"2024-09-10T19:01:34.176726+0000","last_peered":"2024-09-10T19:01:34.176726+0000","last_clean":"2024-09-10T19:01:34.176726+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:01:34.176726+0000","last_undegraded":"2024-09-10T19:01:34.176726+0000","last_fullsized":"2024-09-10T19:01:34.176726+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T18:59:57.094721+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.094721+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045594349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":497,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177814+0000","last_change":"2024-09-10T19:01:27.860664+0000","last_active":"2024-09-10T19:01:34.177814+0000","last_peered":"2024-09-10T19:01:34.177814+0000","last_clean":"2024-09-10T19:01:34.177814+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:01:34.177814+0000","last_undegraded":"2024-09-10T19:01:34.177814+0000","last_fullsized":"2024-09-10T19:01:34.177814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:27.860545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.860545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:51.717349+0000","scrub_duration":0.0031084049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":488,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173648+0000","last_change":"2024-09-10T19:01:18.068985+0000","last_active":"2024-09-10T19:01:34.173648+0000","last_peered":"2024-09-10T19:01:34.173648+0000","last_clean":"2024-09-10T19:01:34.173648+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T19:01:34.173648+0000","last_undegraded":"2024-09-10T19:01:34.173648+0000","last_fullsized":"2024-09-10T19:01:34.173648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.068817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.068817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:24.861245+0000","scrub_duration":0.003161531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175974+0000","last_change":"2024-09-10T19:01:21.078775+0000","last_active":"2024-09-10T19:01:34.175974+0000","last_peered":"2024-09-10T19:01:34.175974+0000","last_clean":"2024-09-10T19:01:34.175974+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T19:01:34.175974+0000","last_undegraded":"2024-09-10T19:01:34.175974+0000","last_fullsized":"2024-09-10T19:01:34.175974+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.078620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.078620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:27.738168+0000","scrub_duration":0.0029631150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":393,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181087+0000","last_change":"2024-09-10T19:00:07.323465+0000","last_active":"2024-09-10T19:01:34.181087+0000","last_peered":"2024-09-10T19:01:34.181087+0000","last_clean":"2024-09-10T19:01:34.181087+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:01:34.181087+0000","last_undegraded":"2024-09-10T19:01:34.181087+0000","last_fullsized":"2024-09-10T19:01:34.181087+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:07.323364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:07.323364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028330460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":521,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179028+0000","last_change":"2024-09-10T19:00:12.906006+0000","last_active":"2024-09-10T19:01:34.179028+0000","last_peered":"2024-09-10T19:01:34.179028+0000","last_clean":"2024-09-10T19:01:34.179028+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:01:34.179028+0000","last_undegraded":"2024-09-10T19:01:34.179028+0000","last_fullsized":"2024-09-10T19:01:34.179028+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:00:12.905889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.905889+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:42.533979+0000","scrub_duration":0.003872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":487,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176929+0000","last_change":"2024-09-10T19:01:24.883894+0000","last_active":"2024-09-10T19:01:34.176929+0000","last_peered":"2024-09-10T19:01:34.176929+0000","last_clean":"2024-09-10T19:01:34.176929+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:01:34.176929+0000","last_undegraded":"2024-09-10T19:01:34.176929+0000","last_fullsized":"2024-09-10T19:01:34.176929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:24.883733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:24.883733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.277814+0000","scrub_duration":0.0033285659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175700+0000","last_change":"2024-09-10T18:58:43.840446+0000","last_active":"2024-09-10T19:01:34.175700+0000","last_peered":"2024-09-10T19:01:34.175700+0000","last_clean":"2024-09-10T19:01:34.175700+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:01:34.175700+0000","last_undegraded":"2024-09-10T19:01:34.175700+0000","last_fullsized":"2024-09-10T19:01:34.175700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:43.840238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:43.840238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032455280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182914+0000","last_change":"2024-09-10T18:59:57.328269+0000","last_active":"2024-09-10T19:01:34.182914+0000","last_peered":"2024-09-10T19:01:34.182914+0000","last_clean":"2024-09-10T19:01:34.182914+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:01:34.182914+0000","last_undegraded":"2024-09-10T19:01:34.182914+0000","last_fullsized":"2024-09-10T19:01:34.182914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:57.328111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.328111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032441140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177175+0000","last_change":"2024-09-10T19:00:14.537922+0000","last_active":"2024-09-10T19:01:34.177175+0000","last_peered":"2024-09-10T19:01:34.177175+0000","last_clean":"2024-09-10T19:01:34.177175+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:01:34.177175+0000","last_undegraded":"2024-09-10T19:01:34.177175+0000","last_fullsized":"2024-09-10T19:01:34.177175+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.537765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.537765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033849069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":524,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173923+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T19:01:34.173923+0000","last_peered":"2024-09-10T19:01:34.173923+0000","last_clean":"2024-09-10T19:01:34.173923+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:01:34.173923+0000","last_undegraded":"2024-09-10T19:01:34.173923+0000","last_fullsized":"2024-09-10T19:01:34.173923+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851029+0000","last_change":"2024-09-10T19:00:24.104254+0000","last_active":"2024-09-10T19:01:34.851029+0000","last_peered":"2024-09-10T19:01:34.851029+0000","last_clean":"2024-09-10T19:01:34.851029+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:01:34.851029+0000","last_undegraded":"2024-09-10T19:01:34.851029+0000","last_fullsized":"2024-09-10T19:01:34.851029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:24.104066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:24.104066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.917926+0000","scrub_duration":0.0038382149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":643,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176105+0000","last_change":"2024-09-10T19:00:05.221386+0000","last_active":"2024-09-10T19:01:34.176105+0000","last_peered":"2024-09-10T19:01:34.176105+0000","last_clean":"2024-09-10T19:01:34.176105+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T19:01:34.176105+0000","last_undegraded":"2024-09-10T19:01:34.176105+0000","last_fullsized":"2024-09-10T19:01:34.176105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T19:00:05.221191+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.221191+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039722840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":475,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176847+0000","last_change":"2024-09-10T19:00:22.885319+0000","last_active":"2024-09-10T19:01:34.176847+0000","last_peered":"2024-09-10T19:01:34.176847+0000","last_clean":"2024-09-10T19:01:34.176847+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:01:34.176847+0000","last_undegraded":"2024-09-10T19:01:34.176847+0000","last_fullsized":"2024-09-10T19:01:34.176847+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:00:22.885146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:22.885146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.180722+0000","scrub_duration":0.0048287809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179262+0000","last_change":"2024-09-10T19:01:29.873888+0000","last_active":"2024-09-10T19:01:34.179262+0000","last_peered":"2024-09-10T19:01:34.179262+0000","last_clean":"2024-09-10T19:01:34.179262+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:01:34.179262+0000","last_undegraded":"2024-09-10T19:01:34.179262+0000","last_fullsized":"2024-09-10T19:01:34.179262+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.873730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.873730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.635541+0000","scrub_duration":0.0034669100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":511,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854036+0000","last_change":"2024-09-10T19:01:23.545508+0000","last_active":"2024-09-10T19:01:34.854036+0000","last_peered":"2024-09-10T19:01:34.854036+0000","last_clean":"2024-09-10T19:01:34.854036+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:01:34.854036+0000","last_undegraded":"2024-09-10T19:01:34.854036+0000","last_fullsized":"2024-09-10T19:01:34.854036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:01:23.545431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.545431+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.986385+0000","scrub_duration":0.004183679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181607+0000","last_change":"2024-09-10T19:01:21.747071+0000","last_active":"2024-09-10T19:01:34.181607+0000","last_peered":"2024-09-10T19:01:34.181607+0000","last_clean":"2024-09-10T19:01:34.181607+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:01:34.181607+0000","last_undegraded":"2024-09-10T19:01:34.181607+0000","last_fullsized":"2024-09-10T19:01:34.181607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.746907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.746907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.629492+0000","scrub_duration":0.003822261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":485,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173438+0000","last_change":"2024-09-10T19:00:10.266358+0000","last_active":"2024-09-10T19:01:34.173438+0000","last_peered":"2024-09-10T19:01:34.173438+0000","last_clean":"2024-09-10T19:01:34.173438+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T19:01:34.173438+0000","last_undegraded":"2024-09-10T19:01:34.173438+0000","last_fullsized":"2024-09-10T19:01:34.173438+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.266243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.266243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003123285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1206,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.175706+0000","last_change":"2024-09-10T19:01:26.001371+0000","last_active":"2024-09-10T19:01:35.175706+0000","last_peered":"2024-09-10T19:01:35.175706+0000","last_clean":"2024-09-10T19:01:35.175706+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:01:35.175706+0000","last_undegraded":"2024-09-10T19:01:35.175706+0000","last_fullsized":"2024-09-10T19:01:35.175706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:01:26.001280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.001280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:49.352524+0000","scrub_duration":0.003156715,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":712,"num_read_kb":707,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852167+0000","last_change":"2024-09-10T19:00:04.106819+0000","last_active":"2024-09-10T19:01:34.852167+0000","last_peered":"2024-09-10T19:01:34.852167+0000","last_clean":"2024-09-10T19:01:34.852167+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:01:34.852167+0000","last_undegraded":"2024-09-10T19:01:34.852167+0000","last_fullsized":"2024-09-10T19:01:34.852167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.106660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.106660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031927650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":414,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181877+0000","last_change":"2024-09-10T19:00:31.183145+0000","last_active":"2024-09-10T19:01:34.181877+0000","last_peered":"2024-09-10T19:01:34.181877+0000","last_clean":"2024-09-10T19:01:34.181877+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:01:34.181877+0000","last_undegraded":"2024-09-10T19:01:34.181877+0000","last_fullsized":"2024-09-10T19:01:34.181877+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:31.183027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.183027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.165521+0000","scrub_duration":0.0084730650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":511,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176644+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T19:01:34.176644+0000","last_peered":"2024-09-10T19:01:34.176644+0000","last_clean":"2024-09-10T19:01:34.176644+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:01:34.176644+0000","last_undegraded":"2024-09-10T19:01:34.176644+0000","last_fullsized":"2024-09-10T19:01:34.176644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":487,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179490+0000","last_change":"2024-09-10T18:58:57.770410+0000","last_active":"2024-09-10T19:01:34.179490+0000","last_peered":"2024-09-10T19:01:34.179490+0000","last_clean":"2024-09-10T19:01:34.179490+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:01:34.179490+0000","last_undegraded":"2024-09-10T19:01:34.179490+0000","last_fullsized":"2024-09-10T19:01:34.179490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:57.770136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.770136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045536739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":623,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178828+0000","last_change":"2024-09-10T18:59:59.066338+0000","last_active":"2024-09-10T19:01:34.178828+0000","last_peered":"2024-09-10T19:01:34.178828+0000","last_clean":"2024-09-10T19:01:34.178828+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:01:34.178828+0000","last_undegraded":"2024-09-10T19:01:34.178828+0000","last_fullsized":"2024-09-10T19:01:34.178828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:59:59.066136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.066136+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038648509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":492,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173150+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T19:01:34.173150+0000","last_peered":"2024-09-10T19:01:34.173150+0000","last_clean":"2024-09-10T19:01:34.173150+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:01:34.173150+0000","last_undegraded":"2024-09-10T19:01:34.173150+0000","last_fullsized":"2024-09-10T19:01:34.173150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173925+0000","last_change":"2024-09-10T19:01:19.068892+0000","last_active":"2024-09-10T19:01:34.173925+0000","last_peered":"2024-09-10T19:01:34.173925+0000","last_clean":"2024-09-10T19:01:34.173925+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T19:01:34.173925+0000","last_undegraded":"2024-09-10T19:01:34.173925+0000","last_fullsized":"2024-09-10T19:01:34.173925+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.068733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.068733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:35.414773+0000","scrub_duration":0.0034618380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":395,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178727+0000","last_change":"2024-09-10T19:01:21.953462+0000","last_active":"2024-09-10T19:01:34.178727+0000","last_peered":"2024-09-10T19:01:34.178727+0000","last_clean":"2024-09-10T19:01:34.178727+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:01:34.178727+0000","last_undegraded":"2024-09-10T19:01:34.178727+0000","last_fullsized":"2024-09-10T19:01:34.178727+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.953303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.953303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:22.871763+0000","scrub_duration":0.0036359589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":710,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178555+0000","last_change":"2024-09-10T19:00:01.595171+0000","last_active":"2024-09-10T19:01:34.178555+0000","last_peered":"2024-09-10T19:01:34.178555+0000","last_clean":"2024-09-10T19:01:34.178555+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:01:34.178555+0000","last_undegraded":"2024-09-10T19:01:34.178555+0000","last_fullsized":"2024-09-10T19:01:34.178555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:00:01.595006+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.595006+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053873050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853325+0000","last_change":"2024-09-10T19:01:20.312372+0000","last_active":"2024-09-10T19:01:34.853325+0000","last_peered":"2024-09-10T19:01:34.853325+0000","last_clean":"2024-09-10T19:01:34.853325+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:01:34.853325+0000","last_undegraded":"2024-09-10T19:01:34.853325+0000","last_fullsized":"2024-09-10T19:01:34.853325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:20.312136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:20.312136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.789362+0000","scrub_duration":0.0033891630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":631,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852888+0000","last_change":"2024-09-10T18:59:56.107053+0000","last_active":"2024-09-10T19:01:34.852888+0000","last_peered":"2024-09-10T19:01:34.852888+0000","last_clean":"2024-09-10T19:01:34.852888+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:01:34.852888+0000","last_undegraded":"2024-09-10T19:01:34.852888+0000","last_fullsized":"2024-09-10T19:01:34.852888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:59:56.106896+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:59:56.106896+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036065950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":490,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180135+0000","last_change":"2024-09-10T19:01:16.768276+0000","last_active":"2024-09-10T19:01:34.180135+0000","last_peered":"2024-09-10T19:01:34.180135+0000","last_clean":"2024-09-10T19:01:34.180135+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:01:34.180135+0000","last_undegraded":"2024-09-10T19:01:34.180135+0000","last_fullsized":"2024-09-10T19:01:34.180135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:01:16.768112+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.768112+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.561140+0000","scrub_duration":0.0040962680000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":481,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174633+0000","last_change":"2024-09-10T19:01:18.860189+0000","last_active":"2024-09-10T19:01:34.174633+0000","last_peered":"2024-09-10T19:01:34.174633+0000","last_clean":"2024-09-10T19:01:34.174633+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:01:34.174633+0000","last_undegraded":"2024-09-10T19:01:34.174633+0000","last_fullsized":"2024-09-10T19:01:34.174633+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:26.756938+0000","scrub_duration":0.0032709060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":395,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177090+0000","last_change":"2024-09-10T19:00:31.934239+0000","last_active":"2024-09-10T19:01:34.177090+0000","last_peered":"2024-09-10T19:01:34.177090+0000","last_clean":"2024-09-10T19:01:34.177090+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:01:34.177090+0000","last_undegraded":"2024-09-10T19:01:34.177090+0000","last_fullsized":"2024-09-10T19:01:34.177090+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:31.933970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.933970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.584006+0000","scrub_duration":0.003362608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":521,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183179+0000","last_change":"2024-09-10T19:01:27.401342+0000","last_active":"2024-09-10T19:01:34.183179+0000","last_peered":"2024-09-10T19:01:34.183179+0000","last_clean":"2024-09-10T19:01:34.183179+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:01:34.183179+0000","last_undegraded":"2024-09-10T19:01:34.183179+0000","last_fullsized":"2024-09-10T19:01:34.183179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'26","last_scrub_stamp":"2024-09-10T19:01:27.401178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.401178+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.489469+0000","scrub_duration":0.0045996170000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":487,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178633+0000","last_change":"2024-09-10T19:01:15.859080+0000","last_active":"2024-09-10T19:01:34.178633+0000","last_peered":"2024-09-10T19:01:34.178633+0000","last_clean":"2024-09-10T19:01:34.178633+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:01:34.178633+0000","last_undegraded":"2024-09-10T19:01:34.178633+0000","last_fullsized":"2024-09-10T19:01:34.178633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.858920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.858920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:17.797614+0000","scrub_duration":0.003170703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177545+0000","last_change":"2024-09-10T18:58:34.812400+0000","last_active":"2024-09-10T19:01:34.177545+0000","last_peered":"2024-09-10T19:01:34.177545+0000","last_clean":"2024-09-10T19:01:34.177545+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:01:34.177545+0000","last_undegraded":"2024-09-10T19:01:34.177545+0000","last_fullsized":"2024-09-10T19:01:34.177545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:34.812228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T18:58:34.812228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034860970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":471,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852370+0000","last_change":"2024-09-10T19:00:00.056759+0000","last_active":"2024-09-10T19:01:34.852370+0000","last_peered":"2024-09-10T19:01:34.852370+0000","last_clean":"2024-09-10T19:01:34.852370+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:01:34.852370+0000","last_undegraded":"2024-09-10T19:01:34.852370+0000","last_fullsized":"2024-09-10T19:01:34.852370+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.056600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.056600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028960119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":414,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182119+0000","last_change":"2024-09-10T19:01:16.369137+0000","last_active":"2024-09-10T19:01:34.182119+0000","last_peered":"2024-09-10T19:01:34.182119+0000","last_clean":"2024-09-10T19:01:34.182119+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:01:34.182119+0000","last_undegraded":"2024-09-10T19:01:34.182119+0000","last_fullsized":"2024-09-10T19:01:34.182119+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:16.368989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.368989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:17.537485+0000","scrub_duration":0.0034355140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":538,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175440+0000","last_change":"2024-09-10T19:00:27.165321+0000","last_active":"2024-09-10T19:01:34.175440+0000","last_peered":"2024-09-10T19:01:34.175440+0000","last_clean":"2024-09-10T19:01:34.175440+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T19:01:34.175440+0000","last_undegraded":"2024-09-10T19:01:34.175440+0000","last_fullsized":"2024-09-10T19:01:34.175440+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:00:27.165232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.165232+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:49.580033+0000","scrub_duration":0.0047657200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182550+0000","last_change":"2024-09-10T19:01:30.331378+0000","last_active":"2024-09-10T19:01:34.182550+0000","last_peered":"2024-09-10T19:01:34.182550+0000","last_clean":"2024-09-10T19:01:34.182550+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:01:34.182550+0000","last_undegraded":"2024-09-10T19:01:34.182550+0000","last_fullsized":"2024-09-10T19:01:34.182550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.331218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.331218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:48.513446+0000","scrub_duration":0.0033888009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178473+0000","last_change":"2024-09-10T19:00:25.638042+0000","last_active":"2024-09-10T19:01:34.178473+0000","last_peered":"2024-09-10T19:01:34.178473+0000","last_clean":"2024-09-10T19:01:34.178473+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:01:34.178473+0000","last_undegraded":"2024-09-10T19:01:34.178473+0000","last_fullsized":"2024-09-10T19:01:34.178473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.637874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.637874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033414320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":464,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174615+0000","last_change":"2024-09-10T18:58:39.317608+0000","last_active":"2024-09-10T19:01:34.174615+0000","last_peered":"2024-09-10T19:01:34.174615+0000","last_clean":"2024-09-10T19:01:34.174615+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:01:34.174615+0000","last_undegraded":"2024-09-10T19:01:34.174615+0000","last_fullsized":"2024-09-10T19:01:34.174615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853236+0000","last_change":"2024-09-10T19:00:21.205302+0000","last_active":"2024-09-10T19:01:34.853236+0000","last_peered":"2024-09-10T19:01:34.853236+0000","last_clean":"2024-09-10T19:01:34.853236+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:01:34.853236+0000","last_undegraded":"2024-09-10T19:01:34.853236+0000","last_fullsized":"2024-09-10T19:01:34.853236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.205176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.205176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:35.277081+0000","scrub_duration":0.0033764950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":543,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177073+0000","last_change":"2024-09-10T19:00:13.487926+0000","last_active":"2024-09-10T19:01:34.177073+0000","last_peered":"2024-09-10T19:01:34.177073+0000","last_clean":"2024-09-10T19:01:34.177073+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:01:34.177073+0000","last_undegraded":"2024-09-10T19:01:34.177073+0000","last_fullsized":"2024-09-10T19:01:34.177073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.487819+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003630566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":508,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181677+0000","last_change":"2024-09-10T19:01:29.231379+0000","last_active":"2024-09-10T19:01:34.181677+0000","last_peered":"2024-09-10T19:01:34.181677+0000","last_clean":"2024-09-10T19:01:34.181677+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:01:34.181677+0000","last_undegraded":"2024-09-10T19:01:34.181677+0000","last_fullsized":"2024-09-10T19:01:34.181677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.231291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.231291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:41.516391+0000","scrub_duration":0.0020441320000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179336+0000","last_change":"2024-09-10T18:58:36.851480+0000","last_active":"2024-09-10T19:01:34.179336+0000","last_peered":"2024-09-10T19:01:34.179336+0000","last_clean":"2024-09-10T19:01:34.179336+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:01:34.179336+0000","last_undegraded":"2024-09-10T19:01:34.179336+0000","last_fullsized":"2024-09-10T19:01:34.179336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:36.851360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:36.851360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034182689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852249+0000","last_change":"2024-09-10T19:00:17.050264+0000","last_active":"2024-09-10T19:01:34.852249+0000","last_peered":"2024-09-10T19:01:34.852249+0000","last_clean":"2024-09-10T19:01:34.852249+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:01:34.852249+0000","last_undegraded":"2024-09-10T19:01:34.852249+0000","last_fullsized":"2024-09-10T19:01:34.852249+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.050058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.050058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.678802+0000","scrub_duration":0.0038046009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181254+0000","last_change":"2024-09-10T19:01:23.802298+0000","last_active":"2024-09-10T19:01:34.181254+0000","last_peered":"2024-09-10T19:01:34.181254+0000","last_clean":"2024-09-10T19:01:34.181254+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:01:34.181254+0000","last_undegraded":"2024-09-10T19:01:34.181254+0000","last_fullsized":"2024-09-10T19:01:34.181254+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:23.802111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.802111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.627831+0000","scrub_duration":0.0034655179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":517,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852269+0000","last_change":"2024-09-10T18:58:55.922693+0000","last_active":"2024-09-10T19:01:34.852269+0000","last_peered":"2024-09-10T19:01:34.852269+0000","last_clean":"2024-09-10T19:01:34.852269+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:01:34.852269+0000","last_undegraded":"2024-09-10T19:01:34.852269+0000","last_fullsized":"2024-09-10T19:01:34.852269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'14","last_scrub_stamp":"2024-09-10T18:58:55.922524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.922524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049593559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850018+0000","last_change":"2024-09-10T19:00:26.116209+0000","last_active":"2024-09-10T19:01:34.850018+0000","last_peered":"2024-09-10T19:01:34.850018+0000","last_clean":"2024-09-10T19:01:34.850018+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:01:34.850018+0000","last_undegraded":"2024-09-10T19:01:34.850018+0000","last_fullsized":"2024-09-10T19:01:34.850018+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.116042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039421509999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854140+0000","last_change":"2024-09-10T18:59:03.826329+0000","last_active":"2024-09-10T19:01:34.854140+0000","last_peered":"2024-09-10T19:01:34.854140+0000","last_clean":"2024-09-10T19:01:34.854140+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:01:34.854140+0000","last_undegraded":"2024-09-10T19:01:34.854140+0000","last_fullsized":"2024-09-10T19:01:34.854140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":481,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178213+0000","last_change":"2024-09-10T19:00:03.612383+0000","last_active":"2024-09-10T19:01:34.178213+0000","last_peered":"2024-09-10T19:01:34.178213+0000","last_clean":"2024-09-10T19:01:34.178213+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:01:34.178213+0000","last_undegraded":"2024-09-10T19:01:34.178213+0000","last_fullsized":"2024-09-10T19:01:34.178213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.612225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033384840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854291+0000","last_change":"2024-09-10T19:00:25.167720+0000","last_active":"2024-09-10T19:01:34.854291+0000","last_peered":"2024-09-10T19:01:34.854291+0000","last_clean":"2024-09-10T19:01:34.854291+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:01:34.854291+0000","last_undegraded":"2024-09-10T19:01:34.854291+0000","last_fullsized":"2024-09-10T19:01:34.854291+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.167452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.167452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:47.782239+0000","scrub_duration":0.0033805139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":503,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177401+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T19:01:34.177401+0000","last_peered":"2024-09-10T19:01:34.177401+0000","last_clean":"2024-09-10T19:01:34.177401+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:01:34.177401+0000","last_undegraded":"2024-09-10T19:01:34.177401+0000","last_fullsized":"2024-09-10T19:01:34.177401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182468+0000","last_change":"2024-09-10T19:00:15.230885+0000","last_active":"2024-09-10T19:01:34.182468+0000","last_peered":"2024-09-10T19:01:34.182468+0000","last_clean":"2024-09-10T19:01:34.182468+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:01:34.182468+0000","last_undegraded":"2024-09-10T19:01:34.182468+0000","last_fullsized":"2024-09-10T19:01:34.182468+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.230765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:38.515881+0000","scrub_duration":0.002962732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":534,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850320+0000","last_change":"2024-09-10T19:00:12.005658+0000","last_active":"2024-09-10T19:01:34.850320+0000","last_peered":"2024-09-10T19:01:34.850320+0000","last_clean":"2024-09-10T19:01:34.850320+0000","last_became_active":"2024-09-10T18:48:36.450938+0000","last_became_peered":"2024-09-10T18:48:36.450938+0000","last_unstale":"2024-09-10T19:01:34.850320+0000","last_undegraded":"2024-09-10T19:01:34.850320+0000","last_fullsized":"2024-09-10T19:01:34.850320+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":482,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181163+0000","last_change":"2024-09-10T19:01:28.744025+0000","last_active":"2024-09-10T19:01:34.181163+0000","last_peered":"2024-09-10T19:01:34.181163+0000","last_clean":"2024-09-10T19:01:34.181163+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:01:34.181163+0000","last_undegraded":"2024-09-10T19:01:34.181163+0000","last_fullsized":"2024-09-10T19:01:34.181163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:28.743841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.743841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:38.125924+0000","scrub_duration":0.0035976770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":470,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183438+0000","last_change":"2024-09-10T19:01:31.371325+0000","last_active":"2024-09-10T19:01:34.183438+0000","last_peered":"2024-09-10T19:01:34.183438+0000","last_clean":"2024-09-10T19:01:34.183438+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:01:34.183438+0000","last_undegraded":"2024-09-10T19:01:34.183438+0000","last_fullsized":"2024-09-10T19:01:34.183438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":404,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852780+0000","last_change":"2024-09-10T19:01:29.180999+0000","last_active":"2024-09-10T19:01:34.852780+0000","last_peered":"2024-09-10T19:01:34.852780+0000","last_clean":"2024-09-10T19:01:34.852780+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:01:34.852780+0000","last_undegraded":"2024-09-10T19:01:34.852780+0000","last_fullsized":"2024-09-10T19:01:34.852780+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.180921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.180921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.972784+0000","scrub_duration":0.0085685290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":541,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852749+0000","last_change":"2024-09-10T19:01:27.255478+0000","last_active":"2024-09-10T19:01:34.852749+0000","last_peered":"2024-09-10T19:01:34.852749+0000","last_clean":"2024-09-10T19:01:34.852749+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:01:34.852749+0000","last_undegraded":"2024-09-10T19:01:34.852749+0000","last_fullsized":"2024-09-10T19:01:34.852749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:01:27.255315+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.255315+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.261030+0000","scrub_duration":0.0050413439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":498,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182657+0000","last_change":"2024-09-10T19:00:10.225407+0000","last_active":"2024-09-10T19:01:34.182657+0000","last_peered":"2024-09-10T19:01:34.182657+0000","last_clean":"2024-09-10T19:01:34.182657+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:01:34.182657+0000","last_undegraded":"2024-09-10T19:01:34.182657+0000","last_fullsized":"2024-09-10T19:01:34.182657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.225321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.225321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027779139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":482,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180402+0000","last_change":"2024-09-10T19:00:29.731792+0000","last_active":"2024-09-10T19:01:34.180402+0000","last_peered":"2024-09-10T19:01:34.180402+0000","last_clean":"2024-09-10T19:01:34.180402+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:01:34.180402+0000","last_undegraded":"2024-09-10T19:01:34.180402+0000","last_fullsized":"2024-09-10T19:01:34.180402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:00:29.731727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.097458+0000","scrub_duration":0.012104484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850944+0000","last_change":"2024-09-10T19:01:31.194872+0000","last_active":"2024-09-10T19:01:34.850944+0000","last_peered":"2024-09-10T19:01:34.850944+0000","last_clean":"2024-09-10T19:01:34.850944+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:01:34.850944+0000","last_undegraded":"2024-09-10T19:01:34.850944+0000","last_fullsized":"2024-09-10T19:01:34.850944+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.194789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.194789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.340897+0000","scrub_duration":0.003152789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":394,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173315+0000","last_change":"2024-09-10T18:58:47.523353+0000","last_active":"2024-09-10T19:01:34.173315+0000","last_peered":"2024-09-10T19:01:34.173315+0000","last_clean":"2024-09-10T19:01:34.173315+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:01:34.173315+0000","last_undegraded":"2024-09-10T19:01:34.173315+0000","last_fullsized":"2024-09-10T19:01:34.173315+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:47.523106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:47.523106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052195840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":516,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183108+0000","last_change":"2024-09-10T19:01:32.331406+0000","last_active":"2024-09-10T19:01:34.183108+0000","last_peered":"2024-09-10T19:01:34.183108+0000","last_clean":"2024-09-10T19:01:34.183108+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:01:34.183108+0000","last_undegraded":"2024-09-10T19:01:34.183108+0000","last_fullsized":"2024-09-10T19:01:34.183108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:01:32.331254+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.331254+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.473619+0000","scrub_duration":0.0043664410000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":487,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179801+0000","last_change":"2024-09-10T19:00:34.708486+0000","last_active":"2024-09-10T19:01:34.179801+0000","last_peered":"2024-09-10T19:01:34.179801+0000","last_clean":"2024-09-10T19:01:34.179801+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:01:34.179801+0000","last_undegraded":"2024-09-10T19:01:34.179801+0000","last_fullsized":"2024-09-10T19:01:34.179801+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:34.708428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:34.708428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.828851+0000","scrub_duration":0.0023757650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":463,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852580+0000","last_change":"2024-09-10T18:58:54.918467+0000","last_active":"2024-09-10T19:01:34.852580+0000","last_peered":"2024-09-10T19:01:34.852580+0000","last_clean":"2024-09-10T19:01:34.852580+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:01:34.852580+0000","last_undegraded":"2024-09-10T19:01:34.852580+0000","last_fullsized":"2024-09-10T19:01:34.852580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:54.918318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:54.918318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080733249999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":471,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.483135+0000","last_change":"2024-09-10T19:01:35.482982+0000","last_active":"2024-09-10T19:01:35.483135+0000","last_peered":"2024-09-10T19:01:35.483135+0000","last_clean":"2024-09-10T19:01:35.483135+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:01:35.483135+0000","last_undegraded":"2024-09-10T19:01:35.483135+0000","last_fullsized":"2024-09-10T19:01:35.483135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.482815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.482815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.047812+0000","scrub_duration":0.0032822699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853556+0000","last_change":"2024-09-10T19:00:32.154082+0000","last_active":"2024-09-10T19:01:34.853556+0000","last_peered":"2024-09-10T19:01:34.853556+0000","last_clean":"2024-09-10T19:01:34.853556+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:01:34.853556+0000","last_undegraded":"2024-09-10T19:01:34.853556+0000","last_fullsized":"2024-09-10T19:01:34.853556+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:32.153923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.153923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.158934+0000","scrub_duration":0.003341105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":534,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852717+0000","last_change":"2024-09-10T19:00:27.155044+0000","last_active":"2024-09-10T19:01:34.852717+0000","last_peered":"2024-09-10T19:01:34.852717+0000","last_clean":"2024-09-10T19:01:34.852717+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:01:34.852717+0000","last_undegraded":"2024-09-10T19:01:34.852717+0000","last_fullsized":"2024-09-10T19:01:34.852717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:27.154715+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.154715+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005022098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":520,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183225+0000","last_change":"2024-09-10T19:01:34.178468+0000","last_active":"2024-09-10T19:01:34.183225+0000","last_peered":"2024-09-10T19:01:34.183225+0000","last_clean":"2024-09-10T19:01:34.183225+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:01:34.183225+0000","last_undegraded":"2024-09-10T19:01:34.183225+0000","last_fullsized":"2024-09-10T19:01:34.183225+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:01:34.178344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.178344+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.702660+0000","scrub_duration":0.007965204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":479,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181671+0000","last_change":"2024-09-10T19:00:00.313745+0000","last_active":"2024-09-10T19:01:34.181671+0000","last_peered":"2024-09-10T19:01:34.181671+0000","last_clean":"2024-09-10T19:01:34.181671+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:01:34.181671+0000","last_undegraded":"2024-09-10T19:01:34.181671+0000","last_fullsized":"2024-09-10T19:01:34.181671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.313582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.313582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034259580000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182461+0000","last_change":"2024-09-10T19:00:30.155345+0000","last_active":"2024-09-10T19:01:34.182461+0000","last_peered":"2024-09-10T19:01:34.182461+0000","last_clean":"2024-09-10T19:01:34.182461+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:01:34.182461+0000","last_undegraded":"2024-09-10T19:01:34.182461+0000","last_fullsized":"2024-09-10T19:01:34.182461+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:30.155089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.155089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:42.013903+0000","scrub_duration":0.0038319109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":404,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.172570+0000","last_change":"2024-09-10T19:01:31.205233+0000","last_active":"2024-09-10T19:01:34.172570+0000","last_peered":"2024-09-10T19:01:34.172570+0000","last_clean":"2024-09-10T19:01:34.172570+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T19:01:34.172570+0000","last_undegraded":"2024-09-10T19:01:34.172570+0000","last_fullsized":"2024-09-10T19:01:34.172570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.205158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.205158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.392310+0000","scrub_duration":0.0024743550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":515,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852447+0000","last_change":"2024-09-10T19:01:22.297477+0000","last_active":"2024-09-10T19:01:34.852447+0000","last_peered":"2024-09-10T19:01:34.852447+0000","last_clean":"2024-09-10T19:01:34.852447+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:01:34.852447+0000","last_undegraded":"2024-09-10T19:01:34.852447+0000","last_fullsized":"2024-09-10T19:01:34.852447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:01:22.297269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.297269+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.277078+0000","scrub_duration":0.005830773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":528,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176778+0000","last_change":"2024-09-10T18:59:59.421165+0000","last_active":"2024-09-10T19:01:34.176778+0000","last_peered":"2024-09-10T19:01:34.176778+0000","last_clean":"2024-09-10T19:01:34.176778+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:01:34.176778+0000","last_undegraded":"2024-09-10T19:01:34.176778+0000","last_fullsized":"2024-09-10T19:01:34.176778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:59:59.420998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.420998+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045326000000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":642,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.173664+0000","last_change":"2024-09-10T19:00:15.117567+0000","last_active":"2024-09-10T19:01:34.173664+0000","last_peered":"2024-09-10T19:01:34.173664+0000","last_clean":"2024-09-10T19:01:34.173664+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T19:01:34.173664+0000","last_undegraded":"2024-09-10T19:01:34.173664+0000","last_fullsized":"2024-09-10T19:01:34.173664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:15.117380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.117380+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:45.034441+0000","scrub_duration":0.004280982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":474,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176322+0000","last_change":"2024-09-10T19:01:19.956970+0000","last_active":"2024-09-10T19:01:34.176322+0000","last_peered":"2024-09-10T19:01:34.176322+0000","last_clean":"2024-09-10T19:01:34.176322+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:01:34.176322+0000","last_undegraded":"2024-09-10T19:01:34.176322+0000","last_fullsized":"2024-09-10T19:01:34.176322+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.956889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.956889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.716004+0000","scrub_duration":0.0027515830000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":406,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179155+0000","last_change":"2024-09-10T18:59:55.520903+0000","last_active":"2024-09-10T19:01:34.179155+0000","last_peered":"2024-09-10T19:01:34.179155+0000","last_clean":"2024-09-10T19:01:34.179155+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:01:34.179155+0000","last_undegraded":"2024-09-10T19:01:34.179155+0000","last_fullsized":"2024-09-10T19:01:34.179155+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T18:59:55.520655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.520655+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073248150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":515,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850866+0000","last_change":"2024-09-10T19:00:02.055337+0000","last_active":"2024-09-10T19:01:34.850866+0000","last_peered":"2024-09-10T19:01:34.850866+0000","last_clean":"2024-09-10T19:01:34.850866+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:01:34.850866+0000","last_undegraded":"2024-09-10T19:01:34.850866+0000","last_fullsized":"2024-09-10T19:01:34.850866+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:02.055169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.055169+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050398609999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":498,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181947+0000","last_change":"2024-09-10T19:00:23.196206+0000","last_active":"2024-09-10T19:01:34.181947+0000","last_peered":"2024-09-10T19:01:34.181947+0000","last_clean":"2024-09-10T19:01:34.181947+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:01:34.181947+0000","last_undegraded":"2024-09-10T19:01:34.181947+0000","last_fullsized":"2024-09-10T19:01:34.181947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034334669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853927+0000","last_change":"2024-09-10T19:00:12.129140+0000","last_active":"2024-09-10T19:01:34.853927+0000","last_peered":"2024-09-10T19:01:34.853927+0000","last_clean":"2024-09-10T19:01:34.853927+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:01:34.853927+0000","last_undegraded":"2024-09-10T19:01:34.853927+0000","last_fullsized":"2024-09-10T19:01:34.853927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:12.128979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.128979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":482,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182293+0000","last_change":"2024-09-10T18:59:53.990233+0000","last_active":"2024-09-10T19:01:34.182293+0000","last_peered":"2024-09-10T19:01:34.182293+0000","last_clean":"2024-09-10T19:01:34.182293+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:01:34.182293+0000","last_undegraded":"2024-09-10T19:01:34.182293+0000","last_fullsized":"2024-09-10T19:01:34.182293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.990034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.990034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00353419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":394,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176362+0000","last_change":"2024-09-10T18:58:59.481915+0000","last_active":"2024-09-10T19:01:34.176362+0000","last_peered":"2024-09-10T19:01:34.176362+0000","last_clean":"2024-09-10T19:01:34.176362+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:01:34.176362+0000","last_undegraded":"2024-09-10T19:01:34.176362+0000","last_fullsized":"2024-09-10T19:01:34.176362+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.481794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.481794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002861125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":498,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179172+0000","last_change":"2024-09-10T19:00:21.565543+0000","last_active":"2024-09-10T19:01:34.179172+0000","last_peered":"2024-09-10T19:01:34.179172+0000","last_clean":"2024-09-10T19:01:34.179172+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:01:34.179172+0000","last_undegraded":"2024-09-10T19:01:34.179172+0000","last_fullsized":"2024-09-10T19:01:34.179172+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:00:21.565373+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.565373+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037082009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179771+0000","last_change":"2024-09-10T19:00:10.348880+0000","last_active":"2024-09-10T19:01:34.179771+0000","last_peered":"2024-09-10T19:01:34.179771+0000","last_clean":"2024-09-10T19:01:34.179771+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:01:34.179771+0000","last_undegraded":"2024-09-10T19:01:34.179771+0000","last_fullsized":"2024-09-10T19:01:34.179771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.348709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.348709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033057529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":466,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176761+0000","last_change":"2024-09-10T18:58:32.229947+0000","last_active":"2024-09-10T19:01:34.176761+0000","last_peered":"2024-09-10T19:01:34.176761+0000","last_clean":"2024-09-10T19:01:34.176761+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:01:34.176761+0000","last_undegraded":"2024-09-10T19:01:34.176761+0000","last_fullsized":"2024-09-10T19:01:34.176761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:32.229885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T18:58:32.229885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043153200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852016+0000","last_change":"2024-09-10T19:00:23.089654+0000","last_active":"2024-09-10T19:01:34.852016+0000","last_peered":"2024-09-10T19:01:34.852016+0000","last_clean":"2024-09-10T19:01:34.852016+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:01:34.852016+0000","last_undegraded":"2024-09-10T19:01:34.852016+0000","last_fullsized":"2024-09-10T19:01:34.852016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.089483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.089483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:41.384760+0000","scrub_duration":0.003188331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.715222+0000","last_change":"2024-09-10T19:01:34.715113+0000","last_active":"2024-09-10T19:01:34.715222+0000","last_peered":"2024-09-10T19:01:34.715222+0000","last_clean":"2024-09-10T19:01:34.715222+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:01:34.715222+0000","last_undegraded":"2024-09-10T19:01:34.715222+0000","last_fullsized":"2024-09-10T19:01:34.715222+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:34.714947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.714947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:36.903905+0000","scrub_duration":0.0035091329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":520,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853503+0000","last_change":"2024-09-10T19:01:19.268405+0000","last_active":"2024-09-10T19:01:34.853503+0000","last_peered":"2024-09-10T19:01:34.853503+0000","last_clean":"2024-09-10T19:01:34.853503+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:01:34.853503+0000","last_undegraded":"2024-09-10T19:01:34.853503+0000","last_fullsized":"2024-09-10T19:01:34.853503+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:01:19.268316+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.268316+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.325891+0000","scrub_duration":0.0078237800000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":476,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182626+0000","last_change":"2024-09-10T19:00:05.196515+0000","last_active":"2024-09-10T19:01:34.182626+0000","last_peered":"2024-09-10T19:01:34.182626+0000","last_clean":"2024-09-10T19:01:34.182626+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:01:34.182626+0000","last_undegraded":"2024-09-10T19:01:34.182626+0000","last_fullsized":"2024-09-10T19:01:34.182626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:05.196346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.196346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033251359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":636,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178251+0000","last_change":"2024-09-10T19:00:26.925255+0000","last_active":"2024-09-10T19:01:34.178251+0000","last_peered":"2024-09-10T19:01:34.178251+0000","last_clean":"2024-09-10T19:01:34.178251+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:01:34.178251+0000","last_undegraded":"2024-09-10T19:01:34.178251+0000","last_fullsized":"2024-09-10T19:01:34.178251+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:26.925124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.925124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:51.740275+0000","scrub_duration":0.0037103370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":471,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853677+0000","last_change":"2024-09-10T19:00:18.141379+0000","last_active":"2024-09-10T19:01:34.853677+0000","last_peered":"2024-09-10T19:01:34.853677+0000","last_clean":"2024-09-10T19:01:34.853677+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:01:34.853677+0000","last_undegraded":"2024-09-10T19:01:34.853677+0000","last_fullsized":"2024-09-10T19:01:34.853677+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.141228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.141228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003005354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853844+0000","last_change":"2024-09-10T19:00:33.116212+0000","last_active":"2024-09-10T19:01:34.853844+0000","last_peered":"2024-09-10T19:01:34.853844+0000","last_clean":"2024-09-10T19:01:34.853844+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:01:34.853844+0000","last_undegraded":"2024-09-10T19:01:34.853844+0000","last_fullsized":"2024-09-10T19:01:34.853844+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.116061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.408011+0000","scrub_duration":0.003119681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":500,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853222+0000","last_change":"2024-09-10T19:00:16.130225+0000","last_active":"2024-09-10T19:01:34.853222+0000","last_peered":"2024-09-10T19:01:34.853222+0000","last_clean":"2024-09-10T19:01:34.853222+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:01:34.853222+0000","last_undegraded":"2024-09-10T19:01:34.853222+0000","last_fullsized":"2024-09-10T19:01:34.853222+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:16.130067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.130067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042760100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854108+0000","last_change":"2024-09-10T18:58:59.930392+0000","last_active":"2024-09-10T19:01:34.854108+0000","last_peered":"2024-09-10T19:01:34.854108+0000","last_clean":"2024-09-10T19:01:34.854108+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:01:34.854108+0000","last_undegraded":"2024-09-10T19:01:34.854108+0000","last_fullsized":"2024-09-10T19:01:34.854108+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:59.930304+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.930304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034696100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":626,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183029+0000","last_change":"2024-09-10T19:01:18.506277+0000","last_active":"2024-09-10T19:01:34.183029+0000","last_peered":"2024-09-10T19:01:34.183029+0000","last_clean":"2024-09-10T19:01:34.183029+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:01:34.183029+0000","last_undegraded":"2024-09-10T19:01:34.183029+0000","last_fullsized":"2024-09-10T19:01:34.183029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:01:18.506103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.506103+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.190637+0000","scrub_duration":0.0036564269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176496+0000","last_change":"2024-09-10T19:00:18.076688+0000","last_active":"2024-09-10T19:01:34.176496+0000","last_peered":"2024-09-10T19:01:34.176496+0000","last_clean":"2024-09-10T19:01:34.176496+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T19:01:34.176496+0000","last_undegraded":"2024-09-10T19:01:34.176496+0000","last_fullsized":"2024-09-10T19:01:34.176496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.076465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.076465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043459409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":382,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854637+0000","last_change":"2024-09-10T18:58:49.915969+0000","last_active":"2024-09-10T19:01:34.854637+0000","last_peered":"2024-09-10T19:01:34.854637+0000","last_clean":"2024-09-10T19:01:34.854637+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:01:34.854637+0000","last_undegraded":"2024-09-10T19:01:34.854637+0000","last_fullsized":"2024-09-10T19:01:34.854637+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.915801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.915801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003511148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":509,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852517+0000","last_change":"2024-09-10T19:00:32.173178+0000","last_active":"2024-09-10T19:01:34.852517+0000","last_peered":"2024-09-10T19:01:34.852517+0000","last_clean":"2024-09-10T19:01:34.852517+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:01:34.852517+0000","last_undegraded":"2024-09-10T19:01:34.852517+0000","last_fullsized":"2024-09-10T19:01:34.852517+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:32.173009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.173009+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:53.697500+0000","scrub_duration":0.0042085289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181592+0000","last_change":"2024-09-10T19:01:31.660345+0000","last_active":"2024-09-10T19:01:34.181592+0000","last_peered":"2024-09-10T19:01:34.181592+0000","last_clean":"2024-09-10T19:01:34.181592+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:01:34.181592+0000","last_undegraded":"2024-09-10T19:01:34.181592+0000","last_fullsized":"2024-09-10T19:01:34.181592+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.660098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.660098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.200581+0000","scrub_duration":0.0038432570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":470,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.182243+0000","last_change":"2024-09-10T19:00:19.289035+0000","last_active":"2024-09-10T19:01:34.182243+0000","last_peered":"2024-09-10T19:01:34.182243+0000","last_clean":"2024-09-10T19:01:34.182243+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:01:34.182243+0000","last_undegraded":"2024-09-10T19:01:34.182243+0000","last_fullsized":"2024-09-10T19:01:34.182243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:19.288883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:19.288883+0000","last_clean_scrub_stamp":"2024-09-10T19:00:19.288883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090246279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":474,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.851774+0000","last_change":"2024-09-10T19:00:04.060523+0000","last_active":"2024-09-10T19:01:34.851774+0000","last_peered":"2024-09-10T19:01:34.851774+0000","last_clean":"2024-09-10T19:01:34.851774+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:01:34.851774+0000","last_undegraded":"2024-09-10T19:01:34.851774+0000","last_fullsized":"2024-09-10T19:01:34.851774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.060371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.060371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031455099999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":404,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174350+0000","last_change":"2024-09-10T19:01:27.192516+0000","last_active":"2024-09-10T19:01:34.174350+0000","last_peered":"2024-09-10T19:01:34.174350+0000","last_clean":"2024-09-10T19:01:34.174350+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:01:34.174350+0000","last_undegraded":"2024-09-10T19:01:34.174350+0000","last_fullsized":"2024-09-10T19:01:34.174350+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:27.192364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.192364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:41.552810+0000","scrub_duration":0.0032891259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":522,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174459+0000","last_change":"2024-09-10T19:00:13.188540+0000","last_active":"2024-09-10T19:01:34.174459+0000","last_peered":"2024-09-10T19:01:34.174459+0000","last_clean":"2024-09-10T19:01:34.174459+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T19:01:34.174459+0000","last_undegraded":"2024-09-10T19:01:34.174459+0000","last_fullsized":"2024-09-10T19:01:34.174459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:00:13.188382+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.188382+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046400579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":487,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179717+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T19:01:34.179717+0000","last_peered":"2024-09-10T19:01:34.179717+0000","last_clean":"2024-09-10T19:01:34.179717+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:01:34.179717+0000","last_undegraded":"2024-09-10T19:01:34.179717+0000","last_fullsized":"2024-09-10T19:01:34.179717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":475,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174858+0000","last_change":"2024-09-10T18:59:02.444264+0000","last_active":"2024-09-10T19:01:34.174858+0000","last_peered":"2024-09-10T19:01:34.174858+0000","last_clean":"2024-09-10T19:01:34.174858+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:01:34.174858+0000","last_undegraded":"2024-09-10T19:01:34.174858+0000","last_fullsized":"2024-09-10T19:01:34.174858+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:02.444100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T18:59:02.444100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033322680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175029+0000","last_change":"2024-09-10T18:59:55.150083+0000","last_active":"2024-09-10T19:01:34.175029+0000","last_peered":"2024-09-10T19:01:34.175029+0000","last_clean":"2024-09-10T19:01:34.175029+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:01:34.175029+0000","last_undegraded":"2024-09-10T19:01:34.175029+0000","last_fullsized":"2024-09-10T19:01:34.175029+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:55.150009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.150009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024462989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":404,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175561+0000","last_change":"2024-09-10T19:01:26.151595+0000","last_active":"2024-09-10T19:01:34.175561+0000","last_peered":"2024-09-10T19:01:34.175561+0000","last_clean":"2024-09-10T19:01:34.175561+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T19:01:34.175561+0000","last_undegraded":"2024-09-10T19:01:34.175561+0000","last_fullsized":"2024-09-10T19:01:34.175561+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:26.151456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.151456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:48.172643+0000","scrub_duration":0.0036896630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":496,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.854711+0000","last_change":"2024-09-10T19:01:24.560200+0000","last_active":"2024-09-10T19:01:34.854711+0000","last_peered":"2024-09-10T19:01:34.854711+0000","last_clean":"2024-09-10T19:01:34.854711+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:01:34.854711+0000","last_undegraded":"2024-09-10T19:01:34.854711+0000","last_fullsized":"2024-09-10T19:01:34.854711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:01:24.560030+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T19:01:24.560030+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:25.538132+0000","scrub_duration":0.0049336370000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183864+0000","last_change":"2024-09-10T19:01:17.509364+0000","last_active":"2024-09-10T19:01:34.183864+0000","last_peered":"2024-09-10T19:01:34.183864+0000","last_clean":"2024-09-10T19:01:34.183864+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:01:34.183864+0000","last_undegraded":"2024-09-10T19:01:34.183864+0000","last_fullsized":"2024-09-10T19:01:34.183864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:17.509179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:01:17.509179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:33.422311+0000","scrub_duration":0.0034198829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":469,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178411+0000","last_change":"2024-09-10T18:59:53.179273+0000","last_active":"2024-09-10T19:01:34.178411+0000","last_peered":"2024-09-10T19:01:34.178411+0000","last_clean":"2024-09-10T19:01:34.178411+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:01:34.178411+0000","last_undegraded":"2024-09-10T19:01:34.178411+0000","last_fullsized":"2024-09-10T19:01:34.178411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.179202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.179202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015188129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.850811+0000","last_change":"2024-09-10T19:00:08.102047+0000","last_active":"2024-09-10T19:01:34.850811+0000","last_peered":"2024-09-10T19:01:34.850811+0000","last_clean":"2024-09-10T19:01:34.850811+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:01:34.850811+0000","last_undegraded":"2024-09-10T19:01:34.850811+0000","last_fullsized":"2024-09-10T19:01:34.850811+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:08.101901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:00:08.101901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028849869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":393,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176967+0000","last_change":"2024-09-10T18:58:55.811349+0000","last_active":"2024-09-10T19:01:34.176967+0000","last_peered":"2024-09-10T19:01:34.176967+0000","last_clean":"2024-09-10T19:01:34.176967+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:01:34.176967+0000","last_undegraded":"2024-09-10T19:01:34.176967+0000","last_fullsized":"2024-09-10T19:01:34.176967+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T18:58:55.811185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037025199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":506,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.373649+0000","last_change":"2024-09-10T19:01:35.373516+0000","last_active":"2024-09-10T19:01:35.373649+0000","last_peered":"2024-09-10T19:01:35.373649+0000","last_clean":"2024-09-10T19:01:35.373649+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:01:35.373649+0000","last_undegraded":"2024-09-10T19:01:35.373649+0000","last_fullsized":"2024-09-10T19:01:35.373649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:01:35.373263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.373263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.279384+0000","scrub_duration":0.0048975939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181473+0000","last_change":"2024-09-10T19:00:11.386471+0000","last_active":"2024-09-10T19:01:34.181473+0000","last_peered":"2024-09-10T19:01:34.181473+0000","last_clean":"2024-09-10T19:01:34.181473+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:01:34.181473+0000","last_undegraded":"2024-09-10T19:01:34.181473+0000","last_fullsized":"2024-09-10T19:01:34.181473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:11.386293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:11.386293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033555310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":466,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175410+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T19:01:34.175410+0000","last_peered":"2024-09-10T19:01:34.175410+0000","last_clean":"2024-09-10T19:01:34.175410+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:01:34.175410+0000","last_undegraded":"2024-09-10T19:01:34.175410+0000","last_fullsized":"2024-09-10T19:01:34.175410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":474,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176495+0000","last_change":"2024-09-10T19:00:23.839038+0000","last_active":"2024-09-10T19:01:34.176495+0000","last_peered":"2024-09-10T19:01:34.176495+0000","last_clean":"2024-09-10T19:01:34.176495+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:01:34.176495+0000","last_undegraded":"2024-09-10T19:01:34.176495+0000","last_fullsized":"2024-09-10T19:01:34.176495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.838880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.838880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:52.392526+0000","scrub_duration":0.0033929009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":392,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853179+0000","last_change":"2024-09-10T19:01:22.572687+0000","last_active":"2024-09-10T19:01:34.853179+0000","last_peered":"2024-09-10T19:01:34.853179+0000","last_clean":"2024-09-10T19:01:34.853179+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:01:34.853179+0000","last_undegraded":"2024-09-10T19:01:34.853179+0000","last_fullsized":"2024-09-10T19:01:34.853179+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.572609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.572609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:39.057857+0000","scrub_duration":0.0029042989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":477,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.852387+0000","last_change":"2024-09-10T19:01:32.170266+0000","last_active":"2024-09-10T19:01:34.852387+0000","last_peered":"2024-09-10T19:01:34.852387+0000","last_clean":"2024-09-10T19:01:34.852387+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:01:34.852387+0000","last_undegraded":"2024-09-10T19:01:34.852387+0000","last_fullsized":"2024-09-10T19:01:34.852387+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":535,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175693+0000","last_change":"2024-09-10T18:58:59.788654+0000","last_active":"2024-09-10T19:01:34.175693+0000","last_peered":"2024-09-10T19:01:34.175693+0000","last_clean":"2024-09-10T19:01:34.175693+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:01:34.175693+0000","last_undegraded":"2024-09-10T19:01:34.175693+0000","last_fullsized":"2024-09-10T19:01:34.175693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"383'22","last_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.788487+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181788+0000","last_change":"2024-09-10T19:00:27.159256+0000","last_active":"2024-09-10T19:01:34.181788+0000","last_peered":"2024-09-10T19:01:34.181788+0000","last_clean":"2024-09-10T19:01:34.181788+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:01:34.181788+0000","last_undegraded":"2024-09-10T19:01:34.181788+0000","last_fullsized":"2024-09-10T19:01:34.181788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:27.159007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.159007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:54.356014+0000","scrub_duration":0.0031545399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":473,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853692+0000","last_change":"2024-09-10T19:00:23.207215+0000","last_active":"2024-09-10T19:01:34.853692+0000","last_peered":"2024-09-10T19:01:34.853692+0000","last_clean":"2024-09-10T19:01:34.853692+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:01:34.853692+0000","last_undegraded":"2024-09-10T19:01:34.853692+0000","last_fullsized":"2024-09-10T19:01:34.853692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.207050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:37.229672+0000","scrub_duration":0.0061363199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":393,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180789+0000","last_change":"2024-09-10T18:59:58.485524+0000","last_active":"2024-09-10T19:01:34.180789+0000","last_peered":"2024-09-10T19:01:34.180789+0000","last_clean":"2024-09-10T19:01:34.180789+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:01:34.180789+0000","last_undegraded":"2024-09-10T19:01:34.180789+0000","last_fullsized":"2024-09-10T19:01:34.180789+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:58.485240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:58.485240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043729700000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":481,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.179305+0000","last_change":"2024-09-10T19:00:09.572036+0000","last_active":"2024-09-10T19:01:34.179305+0000","last_peered":"2024-09-10T19:01:34.179305+0000","last_clean":"2024-09-10T19:01:34.179305+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:01:34.179305+0000","last_undegraded":"2024-09-10T19:01:34.179305+0000","last_fullsized":"2024-09-10T19:01:34.179305+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:09.571847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.571847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034123030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":493,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.178927+0000","last_change":"2024-09-10T18:58:38.293832+0000","last_active":"2024-09-10T19:01:34.178927+0000","last_peered":"2024-09-10T19:01:34.178927+0000","last_clean":"2024-09-10T19:01:34.178927+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:01:34.178927+0000","last_undegraded":"2024-09-10T19:01:34.178927+0000","last_fullsized":"2024-09-10T19:01:34.178927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'9","last_scrub_stamp":"2024-09-10T18:58:38.293671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:38.293671+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050140979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177051+0000","last_change":"2024-09-10T19:01:22.937789+0000","last_active":"2024-09-10T19:01:34.177051+0000","last_peered":"2024-09-10T19:01:34.177051+0000","last_clean":"2024-09-10T19:01:34.177051+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:01:34.177051+0000","last_undegraded":"2024-09-10T19:01:34.177051+0000","last_fullsized":"2024-09-10T19:01:34.177051+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.937630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.937630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.259214+0000","scrub_duration":0.003520343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":656,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181448+0000","last_change":"2024-09-10T19:00:25.684751+0000","last_active":"2024-09-10T19:01:34.181448+0000","last_peered":"2024-09-10T19:01:34.181448+0000","last_clean":"2024-09-10T19:01:34.181448+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:01:34.181448+0000","last_undegraded":"2024-09-10T19:01:34.181448+0000","last_fullsized":"2024-09-10T19:01:34.181448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:00:25.684577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.684577+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:39.541358+0000","scrub_duration":0.0038437710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":403,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.181385+0000","last_change":"2024-09-10T19:01:30.706798+0000","last_active":"2024-09-10T19:01:34.181385+0000","last_peered":"2024-09-10T19:01:34.181385+0000","last_clean":"2024-09-10T19:01:34.181385+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:01:34.181385+0000","last_undegraded":"2024-09-10T19:01:34.181385+0000","last_fullsized":"2024-09-10T19:01:34.181385+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.706647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.706647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:54.500476+0000","scrub_duration":0.003580083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":483,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.175971+0000","last_change":"2024-09-10T19:00:21.100228+0000","last_active":"2024-09-10T19:01:34.175971+0000","last_peered":"2024-09-10T19:01:34.175971+0000","last_clean":"2024-09-10T19:01:34.175971+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:01:34.175971+0000","last_undegraded":"2024-09-10T19:01:34.175971+0000","last_fullsized":"2024-09-10T19:01:34.175971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.100058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.100058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:40.121511+0000","scrub_duration":0.0034414139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":485,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180510+0000","last_change":"2024-09-10T18:58:59.660568+0000","last_active":"2024-09-10T19:01:34.180510+0000","last_peered":"2024-09-10T19:01:34.180510+0000","last_clean":"2024-09-10T19:01:34.180510+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:01:34.180510+0000","last_undegraded":"2024-09-10T19:01:34.180510+0000","last_fullsized":"2024-09-10T19:01:34.180510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T18:58:59.660394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.660394+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003383774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":488,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.853550+0000","last_change":"2024-09-10T19:00:02.052175+0000","last_active":"2024-09-10T19:01:34.853550+0000","last_peered":"2024-09-10T19:01:34.853550+0000","last_clean":"2024-09-10T19:01:34.853550+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:01:34.853550+0000","last_undegraded":"2024-09-10T19:01:34.853550+0000","last_fullsized":"2024-09-10T19:01:34.853550+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:00:02.051995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.051995+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042242440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":504,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-10T19:01:35.187861+0000","last_change":"2024-09-10T19:01:35.187804+0000","last_active":"2024-09-10T19:01:35.187861+0000","last_peered":"2024-09-10T19:01:35.187861+0000","last_clean":"2024-09-10T19:01:35.187861+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:01:35.187861+0000","last_undegraded":"2024-09-10T19:01:35.187861+0000","last_fullsized":"2024-09-10T19:01:35.187861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.187688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.187688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:47.009815+0000","scrub_duration":0.0058675740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":390,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.183370+0000","last_change":"2024-09-10T18:59:54.415459+0000","last_active":"2024-09-10T19:01:34.183370+0000","last_peered":"2024-09-10T19:01:34.183370+0000","last_clean":"2024-09-10T19:01:34.183370+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:01:34.183370+0000","last_undegraded":"2024-09-10T19:01:34.183370+0000","last_fullsized":"2024-09-10T19:01:34.183370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:54.415297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:54.415297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031427790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":474,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176248+0000","last_change":"2024-09-10T19:00:16.862735+0000","last_active":"2024-09-10T19:01:34.176248+0000","last_peered":"2024-09-10T19:01:34.176248+0000","last_clean":"2024-09-10T19:01:34.176248+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:01:34.176248+0000","last_undegraded":"2024-09-10T19:01:34.176248+0000","last_fullsized":"2024-09-10T19:01:34.176248+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:16.862586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.862586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011943657999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":480,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.176957+0000","last_change":"2024-09-10T19:00:17.847339+0000","last_active":"2024-09-10T19:01:34.176957+0000","last_peered":"2024-09-10T19:01:34.176957+0000","last_clean":"2024-09-10T19:01:34.176957+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:01:34.176957+0000","last_undegraded":"2024-09-10T19:01:34.176957+0000","last_fullsized":"2024-09-10T19:01:34.176957+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.847185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.847185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:01:36.511351+0000","scrub_duration":0.0036286880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":512,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.174754+0000","last_change":"2024-09-10T19:00:14.152011+0000","last_active":"2024-09-10T19:01:34.174754+0000","last_peered":"2024-09-10T19:01:34.174754+0000","last_clean":"2024-09-10T19:01:34.174754+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:01:34.174754+0000","last_undegraded":"2024-09-10T19:01:34.174754+0000","last_fullsized":"2024-09-10T19:01:34.174754+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:14.151862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.151862+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041793289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":466,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.177697+0000","last_change":"2024-09-10T18:58:49.198392+0000","last_active":"2024-09-10T19:01:34.177697+0000","last_peered":"2024-09-10T19:01:34.177697+0000","last_clean":"2024-09-10T19:01:34.177697+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:01:34.177697+0000","last_undegraded":"2024-09-10T19:01:34.177697+0000","last_fullsized":"2024-09-10T19:01:34.177697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:49.198233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:58:49.198233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00346847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":393,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-10T19:01:34.180623+0000","last_change":"2024-09-10T18:59:03.762921+0000","last_active":"2024-09-10T19:01:34.180623+0000","last_peered":"2024-09-10T19:01:34.180623+0000","last_clean":"2024-09-10T19:01:34.180623+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:01:34.180623+0000","last_undegraded":"2024-09-10T19:01:34.180623+0000","last_fullsized":"2024-09-10T19:01:34.180623+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.762743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.762743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003463383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":220,"num_pg":32,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195872,"num_objects":169,"num_object_clones":7,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":111,"num_read_kb":10,"num_write":620,"num_write_kb":12709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":24587649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1941,"ondisk_log_size":1941,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":712,"num_read_kb":707,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972511,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317536,"kb_used_data":16016,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93427616,"statfs":{"total":95995035648,"available":95669878784,"internally_reserved":0,"allocated":16400384,"data_stored":13103853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.88500000000000001,"15min":0.90500000000000003},"min":{"1min":0.54900000000000004,"5min":0.311,"15min":0.311},"max":{"1min":1.1850000000000001,"5min":1.3839999999999999,"15min":1.9330000000000001},"last":1.109},{"interface":"front","average":{"1min":1.0860000000000001,"5min":0.96599999999999997,"15min":0.93400000000000005},"min":{"1min":0.72299999999999998,"5min":0.372,"15min":0.34699999999999998},"max":{"1min":2.242,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":0.98899999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89900000000000002,"15min":0.88900000000000001},"min":{"1min":0.72099999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1619999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.96299999999999997,"15min":0.99299999999999999},"min":{"1min":0.56200000000000006,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.266,"5min":1.421,"15min":1.8280000000000001},"last":1.069}]},{"osd":2,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.63300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.218,"5min":1.6739999999999999,"15min":3.4990000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.032,"15min":0.997},"min":{"1min":0.62,"5min":0.443,"15min":0.443},"max":{"1min":2.3250000000000002,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.89600000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.0740000000000001,"15min":1.0229999999999999},"min":{"1min":0.70199999999999996,"5min":0.56799999999999995,"15min":0.48299999999999998},"max":{"1min":2.3879999999999999,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.96899999999999997,"15min":0.97399999999999998},"min":{"1min":0.63100000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.099,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.94999999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.026,"15min":1.0149999999999999},"min":{"1min":0.67800000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.3120000000000001,"5min":1.4279999999999999,"15min":2.1989999999999998},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.1379999999999999,"15min":1.0660000000000001},"min":{"1min":0.67500000000000004,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4470000000000001,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":0.82599999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.008,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":3.1970000000000001,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.1599999999999999,"15min":1.165},"min":{"1min":0.64000000000000001,"5min":0.505,"15min":0.505},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":1.014}]},{"osd":6,"last update":"Tue Sep 10 19:00:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.1879999999999999,"15min":1.208},"min":{"1min":0.77100000000000002,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.516,"5min":1.591,"15min":2.2749999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.302,"5min":1.248,"15min":1.202},"min":{"1min":1.034,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":2.96,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":0.93799999999999994}]}]},{"osd":6,"up_from":52,"seq":223338299561,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313616,"kb_used_data":7912,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93431536,"statfs":{"total":95995035648,"available":95673892864,"internally_reserved":0,"allocated":8101888,"data_stored":4907595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.877,"15min":0.879},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.1910000000000001,"5min":2.032,"15min":2.0550000000000002},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.95999999999999996,"15min":0.95499999999999996},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.2370000000000001,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":0.85899999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":1,"15min":1.0269999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.296,"5min":2.2410000000000001,"15min":18.494},"last":1.363},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.91500000000000004,"15min":0.90500000000000003},"min":{"1min":0.56999999999999995,"5min":0.434,"15min":0.434},"max":{"1min":1.1830000000000001,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":0.97699999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96899999999999997},"min":{"1min":0.627,"5min":0.47499999999999998,"15min":0.35299999999999998},"max":{"1min":1.3140000000000001,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.995,"15min":0.94399999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.242,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":0.68600000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.04,"15min":1.002},"min":{"1min":0.54900000000000004,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.357,"5min":2.234,"15min":2.278},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.012,"15min":0.98599999999999999},"min":{"1min":0.67000000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.278,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.026,"15min":0.95099999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52000000000000002},"max":{"1min":1.327,"5min":6.008,"15min":6.008},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.056,"5min":1.1279999999999999,"15min":1.0600000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.5680000000000001,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.444}]},{"osd":5,"last update":"Tue Sep 10 19:00:58 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.117,"15min":1.042},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.635,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.03},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1899999999999999,"15min":1.129},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.5269999999999999,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.599}]},{"osd":7,"last update":"Tue Sep 10 19:00:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.1140000000000001,"15min":1.071},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.4339999999999999,"5min":2.069,"15min":2.069},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.2729999999999999,"15min":1.194},"min":{"1min":0.65500000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.474,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.123}]}]},{"osd":5,"up_from":43,"seq":184683593907,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313076,"kb_used_data":11556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93432076,"statfs":{"total":95995035648,"available":95674445824,"internally_reserved":0,"allocated":11833344,"data_stored":8520445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70299999999999996,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.3680000000000001,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.179},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96799999999999997,"15min":0.91900000000000004},"min":{"1min":0.50600000000000001,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":1.2470000000000001,"5min":5.0519999999999996,"15min":7.3070000000000004},"last":0.94299999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.254,"15min":1.1539999999999999},"min":{"1min":0.63,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.516,"5min":15.824,"15min":31.548999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.999,"15min":1.081},"min":{"1min":0.71099999999999997,"5min":0.55100000000000005,"15min":0.48899999999999999},"max":{"1min":1.2450000000000001,"5min":5.0839999999999996,"15min":31.847000000000001},"last":0.79200000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.224,"15min":1.2450000000000001},"min":{"1min":0.65800000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4159999999999999,"5min":20.536000000000001,"15min":61.347000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.069,"15min":1.0129999999999999},"min":{"1min":0.71599999999999997,"5min":0.60299999999999998,"15min":0.53200000000000003},"max":{"1min":1.6140000000000001,"5min":5.2489999999999997,"15min":6.1600000000000001},"last":1.123}]},{"osd":3,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.1180000000000001,"15min":1.0109999999999999},"min":{"1min":0.746,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.482,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.1819999999999999,"15min":1.2589999999999999},"min":{"1min":0.84799999999999998,"5min":0.66400000000000003,"15min":0.57399999999999995},"max":{"1min":1.304,"5min":7.7679999999999998,"15min":53.173000000000002},"last":1.236}]},{"osd":4,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1499999999999999,"15min":0.97099999999999997},"min":{"1min":0.63500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4399999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1599999999999999,"15min":1.0349999999999999},"min":{"1min":0.72999999999999998,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.4690000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.093}]},{"osd":6,"last update":"Tue Sep 10 19:00:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.268,"15min":1.157},"min":{"1min":0.76100000000000001,"5min":0.56399999999999995,"15min":0.49199999999999999},"max":{"1min":1.349,"5min":7.7199999999999998,"15min":7.7199999999999998},"last":1.034},{"interface":"front","average":{"1min":1.159,"5min":1.292,"15min":1.198},"min":{"1min":0.80700000000000005,"5min":0.53000000000000003,"15min":0.47199999999999998},"max":{"1min":1.6879999999999999,"5min":9.6760000000000002,"15min":9.6760000000000002},"last":1.2829999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:00:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.3700000000000001,"15min":1.218},"min":{"1min":0.74299999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.679,"5min":15.613,"15min":15.613},"last":1.165},{"interface":"front","average":{"1min":1.125,"5min":1.415,"15min":1.2709999999999999},"min":{"1min":0.81899999999999995,"5min":0.66100000000000003,"15min":0.54200000000000004},"max":{"1min":1.4470000000000001,"5min":16.965,"15min":16.965},"last":1.4390000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855551,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317216,"kb_used_data":7360,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93427936,"statfs":{"total":95995035648,"available":95670206464,"internally_reserved":0,"allocated":7536640,"data_stored":4327049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.86199999999999999,"15min":0.84999999999999998},"min":{"1min":0.53100000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.331,"5min":1.403,"15min":1.546},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.90800000000000003},"min":{"1min":0.57999999999999996,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.325,"5min":1.6100000000000001,"15min":1.7070000000000001},"last":1.1000000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1850000000000001,"15min":0.97499999999999998},"min":{"1min":0.61899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.405,"5min":27.716000000000001,"15min":27.716000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.216,"15min":1.0149999999999999},"min":{"1min":0.65400000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.446,"5min":27.93,"15min":27.93},"last":1.026}]},{"osd":2,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.94599999999999995,"15min":0.93899999999999995},"min":{"1min":0.67500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.321,"5min":1.5209999999999999,"15min":1.5509999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.95499999999999996,"15min":0.94199999999999995},"min":{"1min":0.60799999999999998,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.3759999999999999,"5min":1.482,"15min":1.7569999999999999},"last":0.98699999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.014,"15min":0.96599999999999997},"min":{"1min":0.70099999999999996,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.3160000000000001,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.626,"5min":0.442,"15min":0.442},"max":{"1min":1.258,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.91100000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:01:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.97499999999999998,"15min":0.93799999999999994},"min":{"1min":0.46999999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.5049999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":0.749},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.071,"15min":1.012},"min":{"1min":0.73099999999999998,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.4259999999999999,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.171}]},{"osd":6,"last update":"Tue Sep 10 19:01:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.056,"15min":1.008},"min":{"1min":0.67400000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.2310000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.1859999999999999,"15min":1.133},"min":{"1min":0.99099999999999999,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.9590000000000001,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":0.95899999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:00:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.153,"15min":1.1559999999999999},"min":{"1min":0.61799999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.591,"5min":1.629,"15min":2.008},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1819999999999999,"15min":1.167},"min":{"1min":0.65900000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.4570000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.151}]}]},{"osd":3,"up_from":29,"seq":124554051786,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309036,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436116,"statfs":{"total":95995035648,"available":95678582784,"internally_reserved":0,"allocated":7671808,"data_stored":4318499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.755,"15min":0.82999999999999996},"min":{"1min":0.30199999999999999,"5min":0.29499999999999998,"15min":0.222},"max":{"1min":1.044,"5min":8.6180000000000003,"15min":26.594000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77000000000000002,"15min":0.89200000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.252},"max":{"1min":1.444,"5min":5.3479999999999999,"15min":24.204999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.76600000000000001,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":1.6200000000000001,"5min":5.3879999999999999,"15min":53.661999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84599999999999997,"15min":0.88900000000000001},"min":{"1min":0.40200000000000002,"5min":0.39900000000000002,"15min":0.27900000000000003},"max":{"1min":2.181,"5min":4.0549999999999997,"15min":15.189},"last":0.85799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86899999999999999,"15min":1.006},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32200000000000001},"max":{"1min":1.3380000000000001,"5min":5.1920000000000002,"15min":26.558},"last":1.397},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.83999999999999997,"15min":0.94799999999999995},"min":{"1min":0.32600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.4740000000000002,"5min":4.4740000000000002,"15min":25.536999999999999},"last":1.121}]},{"osd":4,"last update":"Tue Sep 10 19:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1699999999999999,"15min":1.153},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.46999999999999997},"max":{"1min":1.663,"5min":7.3550000000000004,"15min":10.359999999999999},"last":1.244},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2709999999999999,"15min":1.2470000000000001},"min":{"1min":0.81399999999999995,"5min":0.67000000000000004,"15min":0.53500000000000003},"max":{"1min":1.73,"5min":7.4189999999999996,"15min":10.391},"last":1.302}]},{"osd":5,"last update":"Tue Sep 10 19:01:13 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.1739999999999999,"15min":1.2210000000000001},"min":{"1min":0.749,"5min":0.70399999999999996,"15min":0.501},"max":{"1min":1.6000000000000001,"5min":8.6679999999999993,"15min":17.535},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.107,"15min":1.0509999999999999},"min":{"1min":0.65900000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.55,"5min":8.7880000000000003,"15min":8.7880000000000003},"last":1.1579999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:01:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1339999999999999,"15min":1.196},"min":{"1min":0.70799999999999996,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.4830000000000001,"5min":2.5289999999999999,"15min":10.51},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.095,"15min":1.101},"min":{"1min":0.56999999999999995,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.996,"5min":2.996,"15min":10.773999999999999},"last":1.2}]},{"osd":7,"last update":"Tue Sep 10 19:00:57 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.087,"15min":1.1779999999999999},"min":{"1min":0.59899999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.494,"5min":4.0229999999999997,"15min":18.312999999999999},"last":1.002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.3069999999999999,"15min":1.3140000000000001},"min":{"1min":0.89700000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8009999999999999,"5min":7.298,"15min":10.611000000000001},"last":1.351}]}]},{"osd":2,"up_from":19,"seq":81604378835,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325300,"kb_used_data":15444,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93419852,"statfs":{"total":95995035648,"available":95661928448,"internally_reserved":0,"allocated":15814656,"data_stored":12521433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.67000000000000004,"15min":0.72999999999999998},"min":{"1min":0.36799999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.204,"5min":1.5629999999999999,"15min":9.516},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.71999999999999997,"15min":0.78100000000000003},"min":{"1min":0.39800000000000002,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.1120000000000001,"5min":1.2350000000000001,"15min":15.577999999999999},"last":0.82399999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.72899999999999998,"15min":0.72399999999999998},"min":{"1min":0.35299999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.111,"5min":4.2380000000000004,"15min":5.516},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.75900000000000001,"15min":0.81899999999999995},"min":{"1min":0.433,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.022,"5min":1.26,"15min":9.0649999999999995},"last":0.59899999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83199999999999996,"15min":0.872},"min":{"1min":0.47699999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":5.1479999999999997},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.82099999999999995,"15min":0.85399999999999998},"min":{"1min":0.45900000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.262,"5min":2.0539999999999998,"15min":15.736000000000001},"last":0.873}]},{"osd":4,"last update":"Tue Sep 10 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.052,"15min":1.081},"min":{"1min":0.80200000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.5469999999999999,"5min":3.851,"15min":21.571999999999999},"last":1.081},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.2110000000000001,"15min":1.161},"min":{"1min":0.79200000000000004,"5min":0.55100000000000005,"15min":0.46000000000000002},"max":{"1min":1.5289999999999999,"5min":10.881,"15min":10.881},"last":0.91900000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:01:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.0649999999999999,"15min":1.093},"min":{"1min":0.68400000000000005,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.661,"5min":1.8340000000000001,"15min":5.1959999999999997},"last":1.056},{"interface":"front","average":{"1min":1.099,"5min":1.0760000000000001,"15min":1.117},"min":{"1min":0.754,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5329999999999999,"5min":3.4980000000000002,"15min":21.654},"last":1.2050000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:01:09 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1060000000000001,"15min":1.165},"min":{"1min":0.71699999999999997,"5min":0.503,"15min":0.503},"max":{"1min":1.706,"5min":1.706,"15min":9.0389999999999997},"last":1},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.091,"15min":1.1359999999999999},"min":{"1min":0.86099999999999999,"5min":0.52100000000000002,"15min":0.505},"max":{"1min":1.5549999999999999,"5min":3.367,"15min":15.971},"last":0.75800000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:01:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.109,"15min":1.1819999999999999},"min":{"1min":0.74399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.601,"5min":1.905,"15min":15.025},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.212,"15min":1.204},"min":{"1min":0.875,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.474,"5min":6.851,"15min":14.816000000000001},"last":0.84699999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316972,"kb_used_data":11276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93428180,"statfs":{"total":95995035648,"available":95670456320,"internally_reserved":0,"allocated":11546624,"data_stored":8317678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:00:42 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.76300000000000001,"15min":0.82199999999999995},"min":{"1min":0.45400000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.0439999999999996,"5min":5.0439999999999996,"15min":32.677999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.83399999999999996,"15min":0.89500000000000002},"min":{"1min":0.51900000000000002,"5min":0.30399999999999999,"15min":0.253},"max":{"1min":1.9199999999999999,"5min":3.8250000000000002,"15min":32.398000000000003},"last":0.89900000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:00:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":0.84599999999999997,"15min":0.88900000000000001},"min":{"1min":0.53400000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.7999999999999998,"5min":5.7999999999999998,"15min":26.097000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.90500000000000003,"15min":1.0049999999999999},"min":{"1min":0.58999999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.7629999999999999,"5min":4.0890000000000004,"15min":32.363},"last":0.67900000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:01:30 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.999,"15min":1.018},"min":{"1min":0.59199999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.8359999999999999,"5min":3.7360000000000002,"15min":25.704000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.90700000000000003,"15min":1.0489999999999999},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.26600000000000001},"max":{"1min":3.8029999999999999,"5min":6.0099999999999998,"15min":60.999000000000002},"last":0.83299999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1930000000000001,"15min":1.123},"min":{"1min":0.67300000000000004,"5min":0.47399999999999998,"15min":0.40899999999999997},"max":{"1min":1.6719999999999999,"5min":3.5379999999999998,"15min":12.808999999999999},"last":1.054},{"interface":"front","average":{"1min":1.181,"5min":1.1599999999999999,"15min":1.2410000000000001},"min":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.378},"max":{"1min":3.6779999999999999,"5min":6.5430000000000001,"15min":60.529000000000003},"last":0.94199999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:01:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.127,"15min":1.133},"min":{"1min":0.77800000000000002,"5min":0.52200000000000002,"15min":0.38},"max":{"1min":5.2439999999999998,"5min":5.2439999999999998,"15min":26.050999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.274,"5min":1.228,"15min":1.2130000000000001},"min":{"1min":0.71599999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.7070000000000001,"5min":4.5869999999999997,"15min":12.84},"last":0.98799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:01:07 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3089999999999999,"15min":1.29},"min":{"1min":0.89200000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.7549999999999999,"5min":8.1760000000000002,"15min":25.608000000000001},"last":1.21},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.228,"15min":1.252},"min":{"1min":0.91100000000000003,"5min":0.46400000000000002,"15min":0.41499999999999998},"max":{"1min":3.7759999999999998,"5min":6.1219999999999999,"15min":33.643000000000001},"last":1.087}]},{"osd":7,"last update":"Tue Sep 10 19:01:07 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3300000000000001,"15min":1.3380000000000001},"min":{"1min":0.93700000000000006,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":2.1859999999999999,"5min":3.6499999999999999,"15min":25.565999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.353,"5min":1.262,"15min":1.3220000000000001},"min":{"1min":0.73499999999999999,"5min":0.59299999999999997,"15min":0.48399999999999999},"max":{"1min":3.7370000000000001,"5min":6.2599999999999998,"15min":33.673999999999999},"last":1.0089999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309504,"kb_used_data":7976,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93435648,"statfs":{"total":95995035648,"available":95678103552,"internally_reserved":0,"allocated":8167424,"data_stored":4916146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:00:55 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.569,"15min":0.996},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.26300000000000001},"max":{"1min":6.758,"5min":80.710999999999999,"15min":80.710999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.663,"15min":1.0960000000000001},"min":{"1min":0.44600000000000001,"5min":0.41999999999999998,"15min":0.25600000000000001},"max":{"1min":3.3109999999999999,"5min":86.009,"15min":86.009},"last":0.72499999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79900000000000004,"15min":1.0309999999999999},"min":{"1min":0.41399999999999998,"5min":0.216,"15min":0.216},"max":{"1min":1.1659999999999999,"5min":3.427,"15min":36.682000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.83499999999999996,"15min":0.92800000000000005},"min":{"1min":0.442,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.2090000000000001,"5min":3.2090000000000001,"15min":24.614000000000001},"last":0.84499999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.91300000000000003,"15min":0.92100000000000004},"min":{"1min":0.49299999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.4749999999999996,"5min":4.4749999999999996,"15min":21.620999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86399999999999999,"15min":1.0429999999999999},"min":{"1min":0.45700000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.2210000000000001,"5min":2.012,"15min":45.765000000000001},"last":0.95799999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.095,"15min":1.0609999999999999},"min":{"1min":0.871,"5min":0.64300000000000002,"15min":0.5},"max":{"1min":1.4099999999999999,"5min":3.3279999999999998,"15min":6.3630000000000004},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.099,"5min":1.1200000000000001,"15min":1.111},"min":{"1min":0.68400000000000005,"5min":0.61399999999999999,"15min":0.51100000000000001},"max":{"1min":1.5089999999999999,"5min":3.419,"15min":4.2389999999999999},"last":1.143}]},{"osd":5,"last update":"Tue Sep 10 19:01:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1040000000000001,"15min":1.129},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.442},"max":{"1min":1.6279999999999999,"5min":2.609,"15min":10.048},"last":0.875},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.087,"15min":1.1040000000000001},"min":{"1min":0.70699999999999996,"5min":0.63,"15min":0.42999999999999999},"max":{"1min":1.7010000000000001,"5min":1.7010000000000001,"15min":3.8959999999999999},"last":1.228}]},{"osd":6,"last update":"Tue Sep 10 19:01:09 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.1499999999999999,"15min":1.1439999999999999},"min":{"1min":0.80400000000000005,"5min":0.69099999999999995,"15min":0.56899999999999995},"max":{"1min":1.6180000000000001,"5min":2.4550000000000001,"15min":3.9550000000000001},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1659999999999999,"15min":1.204},"min":{"1min":0.72699999999999998,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.8149999999999999,"5min":2.4929999999999999,"15min":11.894},"last":1.0980000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:00:55 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.173,"15min":1.171},"min":{"1min":0.73299999999999998,"5min":0.66500000000000004,"15min":0.48499999999999999},"max":{"1min":2.468,"5min":2.468,"15min":6.2949999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.3540000000000001,"15min":1.2829999999999999},"min":{"1min":0.82599999999999996,"5min":0.75900000000000001,"15min":0.54900000000000004},"max":{"1min":1.6990000000000001,"5min":8.2840000000000007,"15min":8.2840000000000007},"last":1.2749999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4227072,"data_stored":4000562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8474624,"data_stored":8194985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4497408,"data_stored":4195031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8462336,"data_stored":8194936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":220,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:01:42.095 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T19:01:42.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-10T19:01:42.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:42.160 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:42.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:42.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:42.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:42.434 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:42.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:42 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.869523+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:01:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:42 smithi165 bash[22591]: cluster 2024-09-10T19:01:40.872528+0000 osd.1 (osd.1) 286 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:01:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:42 smithi165 bash[22591]: cluster 2024-09-10T19:01:41.590974+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-09-10T19:01:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:42 smithi165 bash[22591]: cluster 2024-09-10T19:01:41.594116+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-09-10T19:01:42.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:42 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.153484+0000 mgr.y (mgr.24526) 500 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 897 B/s wr, 4 op/s 2024-09-10T19:01:42.569 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:42.570 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:42.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.869523+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18629]: cluster 2024-09-10T19:01:40.872528+0000 osd.1 (osd.1) 286 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18629]: cluster 2024-09-10T19:01:41.590974+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18629]: cluster 2024-09-10T19:01:41.594116+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.153484+0000 mgr.y (mgr.24526) 500 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 897 B/s wr, 4 op/s 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.869523+0000 osd.1 (osd.1) 285 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[32381]: cluster 2024-09-10T19:01:40.872528+0000 osd.1 (osd.1) 286 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[32381]: cluster 2024-09-10T19:01:41.590974+0000 osd.0 (osd.0) 266 : cluster [DBG] 4.b scrub starts 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[32381]: cluster 2024-09-10T19:01:41.594116+0000 osd.0 (osd.0) 267 : cluster [DBG] 4.b scrub ok 2024-09-10T19:01:42.686 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.153484+0000 mgr.y (mgr.24526) 500 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 897 B/s wr, 4 op/s 2024-09-10T19:01:42.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:42.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:42.843 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:42.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:42.959 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:42] "GET /metrics HTTP/1.1" 200 228887 "" "Prometheus/2.43.0" 2024-09-10T19:01:42.980 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:42.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:43.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:43.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:43.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.251 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:43.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.391 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:43.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:43.529 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:43.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.290380+0000 osd.4 (osd.4) 310 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:01:43.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.293649+0000 osd.4 (osd.4) 311 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:01:43.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.459371+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:01:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.462675+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:01:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.290380+0000 osd.4 (osd.4) 310 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:01:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.293649+0000 osd.4 (osd.4) 311 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:01:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.459371+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:01:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:43 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.462675+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:01:43.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.666 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:43.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.803 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:43 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.290380+0000 osd.4 (osd.4) 310 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:01:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:43 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.293649+0000 osd.4 (osd.4) 311 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:01:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:43 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.459371+0000 osd.5 (osd.5) 278 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:01:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:43 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.462675+0000 osd.5 (osd.5) 279 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:01:43.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:43.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:44.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:44.078 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:44.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:44.216 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:44.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:44.356 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:44.460 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:44.487 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:44.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:44.599 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:44.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:44.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.932463+0000 osd.1 (osd.1) 287 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:42.936085+0000 osd.1 (osd.1) 288 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:43.218460+0000 osd.2 (osd.2) 213 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:43.224573+0000 osd.2 (osd.2) 214 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:43.823611+0000 osd.6 (osd.6) 233 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:43.827729+0000 osd.6 (osd.6) 234 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.154302+0000 mgr.y (mgr.24526) 501 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 798 B/s wr, 3 op/s 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.932463+0000 osd.1 (osd.1) 287 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:42.936085+0000 osd.1 (osd.1) 288 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:01:44.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:43.218460+0000 osd.2 (osd.2) 213 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:01:44.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:43.224573+0000 osd.2 (osd.2) 214 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:01:44.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:43.823611+0000 osd.6 (osd.6) 233 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:01:44.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:43.827729+0000 osd.6 (osd.6) 234 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:01:44.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:44 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.154302+0000 mgr.y (mgr.24526) 501 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 798 B/s wr, 3 op/s 2024-09-10T19:01:44.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.932463+0000 osd.1 (osd.1) 287 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:01:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:42.936085+0000 osd.1 (osd.1) 288 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:01:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:43.218460+0000 osd.2 (osd.2) 213 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:01:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:43.224573+0000 osd.2 (osd.2) 214 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:01:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:43.823611+0000 osd.6 (osd.6) 233 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:01:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:43.827729+0000 osd.6 (osd.6) 234 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:01:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:44 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.154302+0000 mgr.y (mgr.24526) 501 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 798 B/s wr, 3 op/s 2024-09-10T19:01:44.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:44.896 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:44.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:45.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:45.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:45.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:45.172 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:45.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:45.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:45.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:45.444 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:45.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:45.583 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:45.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:45.715 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:45.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:45.854 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:45.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:45.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:45.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:46.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- 172.21.15.83:0/243828454 >> v1:172.21.15.83:6790/0 conn(0x7f9468107e30 legacy=0x7f946810a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- 172.21.15.83:0/243828454 shutdown_connections 2024-09-10T19:01:46.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- 172.21.15.83:0/243828454 >> 172.21.15.83:0/243828454 conn(0x7f9468074a20 msgr2=0x7f94680f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- 172.21.15.83:0/243828454 shutdown_connections 2024-09-10T19:01:46.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- 172.21.15.83:0/243828454 wait complete. 2024-09-10T19:01:46.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 librados: starting msgr at 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 librados: starting objecter 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 Processor -- start 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- start start 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 librados: setting wanted keys 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 librados: calling monclient init 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946811c060 con 0x7f9468107e30 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946811d220 con 0x7f94681014f0 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946f7e8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f946811e420 con 0x7f94681042d0 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946cd83640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f94681042d0 0x7f94680fee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48908/0 (socket says 172.21.15.83:48908) 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946cd83640 1 -- 172.21.15.83:0/4223053077 learned_addr learned my addr 172.21.15.83:0/4223053077 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f946dd85640 1 --1- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6789/0 conn(0x7f9468107e30 0x7f946811a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58588/0 (socket says 172.21.15.83:58588) 2024-09-10T19:01:46.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f945dffb640 10 client.?.objecter ms_handle_connect 0x7f94681042d0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f945dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f945dffb640 10 client.?.objecter ms_handle_connect 0x7f9468107e30 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.029+0000 7f945dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376124862 0 0) 0x7f946811e420 con 0x7f94681042d0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540038e0 con 0x7f94681042d0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 10 client.?.objecter ms_handle_connect 0x7f94681014f0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431488115 0 0) 0x7f946811c060 con 0x7f9468107e30 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946811e420 con 0x7f9468107e30 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736658436 0 0) 0x7f946811d220 con 0x7f94681014f0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f946811c060 con 0x7f94681014f0 2024-09-10T19:01:46.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619840900 0 0) 0x7f94540038e0 con 0x7f94681042d0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946811d220 con 0x7f94681042d0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f94640019f0 con 0x7f94681042d0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153210887 0 0) 0x7f946811e420 con 0x7f9468107e30 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94540038e0 con 0x7f9468107e30 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9460001930 con 0x7f9468107e30 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2376195284 0 0) 0x7f946811c060 con 0x7f94681014f0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946811e420 con 0x7f94681014f0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9458001830 con 0x7f94681014f0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 611951310 0 0) 0x7f946811d220 con 0x7f94681042d0 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.165:6789/0 conn(0x7f94681014f0 legacy=0x7f94680fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6789/0 conn(0x7f9468107e30 legacy=0x7f946811a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946811f620 con 0x7f94681042d0 2024-09-10T19:01:46.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946811c250 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f946f7e8640 10 client.49484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f946811c720 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f946f7e8640 1 librados: init done 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9464002320 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f94640058c0 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f944c004ff0 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f94640025a0 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(493..493 src has 69..493) v4 ==== 7619+0+0 (unknown 3334736599 0 0) 0x7f946408a270 con 0x7f94681042d0 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 osd_map(493..493 src has 69..493) v4 2024-09-10T19:01:46.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 3 client.49484.objecter handle_osd_map got epochs [493,493] > 0 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 3 client.49484.objecter handle_osd_map decoding full epoch 493 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 20 client.49484.objecter dump_active .. 0 homeless 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v493) v1 ==== 54+0+16 (unknown 1253011255 0 2420000192) 0x7f946408c690 con 0x7f94681042d0 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.033+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f9454075b40 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 10 client.49484.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 1 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 1 2024-09-10T19:01:46.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 0 1 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 4 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 10 client.49484.objecter _send_command 1 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c005290 con 0x7f944c004610 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c004610 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c005290 con 0x7f944c004610 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 10 client.49484.objecter _finish_command 1 = system:0 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 4 2024-09-10T19:01:46.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.037+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 0 1 2024-09-10T19:01:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 10 client.49484.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 2 2024-09-10T19:01:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 2 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c005810 osd=0 3 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 0 2 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 4 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 10 client.49484.objecter _send_command 2 2024-09-10T19:01:46.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c004cc0 con 0x7f944c004610 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c004cc0 con 0x7f944c004610 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f945dffb640 10 client.49484.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 4 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.053+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 0 2 2024-09-10T19:01:46.058 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 10 client.49484.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 1 3 2024-09-10T19:01:46.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 4 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 10 client.49484.objecter _send_command 3 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c0022b0 con 0x7f944c004110 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c004110 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c0022b0 con 0x7f944c004110 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 10 client.49484.objecter _finish_command 3 = system:0 2024-09-10T19:01:46.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 4 2024-09-10T19:01:46.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.057+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 1 3 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 10 client.49484.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 4 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 4 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c003fa0 osd=1 3 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 1 4 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 4 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 10 client.49484.objecter _send_command 4 2024-09-10T19:01:46.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.073+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c001890 con 0x7f944c004110 2024-09-10T19:01:46.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c001890 con 0x7f944c004110 2024-09-10T19:01:46.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 10 client.49484.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 4 2024-09-10T19:01:46.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 1 4 2024-09-10T19:01:46.080 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 10 client.49484.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 5 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 5 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 2 5 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 4 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 10 client.49484.objecter _send_command 5 2024-09-10T19:01:46.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c003df0 con 0x7f944c00d350 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.077+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c00d350 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.081+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c003df0 con 0x7f944c00d350 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.081+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.081+0000 7f945dffb640 10 client.49484.objecter _finish_command 5 = system:0 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.081+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 4 2024-09-10T19:01:46.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.081+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 2 5 2024-09-10T19:01:46.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 10 client.49484.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 6 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 6 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0035e0 osd=2 3 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 2 6 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 4 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 10 client.49484.objecter _send_command 6 2024-09-10T19:01:46.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c0030d0 con 0x7f944c00d350 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c0030d0 con 0x7f944c00d350 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f945dffb640 10 client.49484.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 4 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.097+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 2 6 2024-09-10T19:01:46.102 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 10 client.49484.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 7 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 7 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 3 7 2024-09-10T19:01:46.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 4 2024-09-10T19:01:46.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 10 client.49484.objecter _send_command 7 2024-09-10T19:01:46.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c00d0d0 con 0x7f944c010e50 2024-09-10T19:01:46.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c010e50 2024-09-10T19:01:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.272778+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:01:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.276591+0000 osd.4 (osd.4) 313 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:01:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.397393+0000 osd.5 (osd.5) 280 : cluster [DBG] 6.d scrub starts 2024-09-10T19:01:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.400814+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.d scrub ok 2024-09-10T19:01:46.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:44.576850+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:44.579461+0000 mon.c (mon.2) 643 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.602374+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.606386+0000 osd.0 (osd.0) 269 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.822631+0000 osd.6 (osd.6) 235 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: cluster 2024-09-10T19:01:44.825951+0000 osd.6 (osd.6) 236 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:45.244248+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:45.246290+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:45.247437+0000 mon.a (mon.0) 2719 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.105 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:46 smithi165 bash[22591]: audit 2024-09-10T19:01:45.249345+0000 mon.a (mon.0) 2720 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c00d0d0 con 0x7f944c010e50 2024-09-10T19:01:46.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:01:46.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 10 client.49484.objecter _finish_command 7 = system:0 2024-09-10T19:01:46.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 4 2024-09-10T19:01:46.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.101+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 3 7 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 10 client.49484.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 8 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 8 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c00c990 osd=3 3 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 3 8 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 4 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 10 client.49484.objecter _send_command 8 2024-09-10T19:01:46.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c0132d0 con 0x7f944c010e50 2024-09-10T19:01:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c0132d0 con 0x7f944c010e50 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f945dffb640 10 client.49484.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 4 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 3 8 2024-09-10T19:01:46.125 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 10 client.49484.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 9 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 9 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 4 9 2024-09-10T19:01:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 4 2024-09-10T19:01:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 10 client.49484.objecter _send_command 9 2024-09-10T19:01:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.121+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c01a260 con 0x7f944c016760 2024-09-10T19:01:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c016760 2024-09-10T19:01:46.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c01a260 con 0x7f944c016760 2024-09-10T19:01:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:01:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 10 client.49484.objecter _finish_command 9 = system:0 2024-09-10T19:01:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 4 2024-09-10T19:01:46.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.125+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 4 9 2024-09-10T19:01:46.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 10 client.49484.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 10 2024-09-10T19:01:46.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 10 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0132d0 osd=4 3 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 4 10 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 4 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 10 client.49484.objecter _send_command 10 2024-09-10T19:01:46.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.141+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c01a260 con 0x7f944c016760 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c01a260 con 0x7f944c016760 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 10 client.49484.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 4 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 4 10 2024-09-10T19:01:46.147 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 10 client.49484.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 11 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 11 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 5 11 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 4 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 10 client.49484.objecter _send_command 11 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c01e610 con 0x7f944c01a920 2024-09-10T19:01:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c01a920 2024-09-10T19:01:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c01e610 con 0x7f944c01a920 2024-09-10T19:01:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:01:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 10 client.49484.objecter _finish_command 11 = system:0 2024-09-10T19:01:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 4 2024-09-10T19:01:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.145+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 5 11 2024-09-10T19:01:46.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 10 client.49484.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 12 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 12 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c01a260 osd=5 3 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 5 12 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 4 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 10 client.49484.objecter _send_command 12 2024-09-10T19:01:46.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.161+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c00c240 con 0x7f944c01a920 2024-09-10T19:01:46.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c00c240 con 0x7f944c01a920 2024-09-10T19:01:46.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 10 client.49484.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 4 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 5 12 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 10 client.49484.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 13 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 13 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 6 13 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 4 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 10 client.49484.objecter _send_command 13 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c0130b0 con 0x7f944c0208b0 2024-09-10T19:01:46.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.165+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c0208b0 2024-09-10T19:01:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.169+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c0130b0 con 0x7f944c0208b0 2024-09-10T19:01:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.169+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:01:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.169+0000 7f945dffb640 10 client.49484.objecter _finish_command 13 = system:0 2024-09-10T19:01:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.169+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 4 2024-09-10T19:01:46.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.169+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 6 13 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 10 client.49484.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 14 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 14 2024-09-10T19:01:46.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c00c240 osd=6 3 2024-09-10T19:01:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 6 14 2024-09-10T19:01:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 4 2024-09-10T19:01:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 10 client.49484.objecter _send_command 14 2024-09-10T19:01:46.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.189+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c0130b0 con 0x7f944c0208b0 2024-09-10T19:01:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.193+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c0130b0 con 0x7f944c0208b0 2024-09-10T19:01:46.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.193+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.193+0000 7f945dffb640 10 client.49484.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.193+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 4 2024-09-10T19:01:46.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.193+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 6 14 2024-09-10T19:01:46.198 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 10 client.49484.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 15 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 15 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 7 15 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 4 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 10 client.49484.objecter _send_command 15 2024-09-10T19:01:46.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c0206e0 con 0x7f944c0235c0 2024-09-10T19:01:46.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 10 client.49484.objecter ms_handle_connect 0x7f944c0235c0 2024-09-10T19:01:46.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f944c0206e0 con 0x7f944c0235c0 2024-09-10T19:01:46.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:01:46.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 10 client.49484.objecter _finish_command 15 = system:0 2024-09-10T19:01:46.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 4 2024-09-10T19:01:46.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.197+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 7 15 2024-09-10T19:01:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 10 client.49484.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:01:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign -1 16 2024-09-10T19:01:46.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 20 client.49484.objecter _get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_remove -1 16 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 20 client.49484.objecter get_session s=0x7f944c0130b0 osd=7 3 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 15 client.49484.objecter _session_command_op_assign 7 16 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 4 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 10 client.49484.objecter _send_command 16 2024-09-10T19:01:46.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.209+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f944c0206e0 con 0x7f944c0235c0 2024-09-10T19:01:46.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.213+0000 7f945dffb640 1 -- 172.21.15.83:0/4223053077 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f944c0206e0 con 0x7f944c0235c0 2024-09-10T19:01:46.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.213+0000 7f945dffb640 10 client.49484.objecter ms_dispatch 0x7f9468000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:01:46.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.213+0000 7f945dffb640 10 client.49484.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:01:46.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.213+0000 7f945dffb640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 4 2024-09-10T19:01:46.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.213+0000 7f945dffb640 15 client.49484.objecter _session_command_op_remove 7 16 2024-09-10T19:01:46.218 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:01:46.219 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:01:46.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 librados: watch_flush enter 2024-09-10T19:01:46.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 librados: watch_flush exit 2024-09-10T19:01:46.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.0 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6801/2535527483 conn(0x7f944c004610 legacy=0x7f944c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c005810 osd=0 1 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.1 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6805/2309624637 conn(0x7f944c004110 legacy=0x7f944c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c003fa0 osd=1 1 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.2 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6809/386094220 conn(0x7f944c00d350 legacy=0x7f944c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0035e0 osd=2 1 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.3 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6813/1279688021 conn(0x7f944c010e50 legacy=0x7f944c011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c990 osd=3 1 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.4 2024-09-10T19:01:46.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.165:6800/3225515265 conn(0x7f944c016760 legacy=0x7f944c018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0132d0 osd=4 1 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.5 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.165:6804/571173777 conn(0x7f944c01a920 legacy=0x7f944c01cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c01a260 osd=5 1 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.6 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.165:6808/1317922936 conn(0x7f944c0208b0 legacy=0x7f944c011990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c00c240 osd=6 1 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter close_session for osd.7 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.165:6812/1610151650 conn(0x7f944c0235c0 legacy=0x7f944c0259a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter put_session s=0x7f944c0130b0 osd=7 1 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 20 client.49484.objecter shutdown clearing up homeless session... 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 10 client.49484.objecter successfully canceled tick 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946dd85640 1 -- 172.21.15.83:0/4223053077 reap_dead start 2024-09-10T19:01:46.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946dd85640 1 -- 172.21.15.83:0/4223053077 reap_dead start 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946dd85640 1 -- 172.21.15.83:0/4223053077 reap_dead start 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946dd85640 1 -- 172.21.15.83:0/4223053077 reap_dead start 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946dd85640 1 -- 172.21.15.83:0/4223053077 reap_dead start 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9454075b40 legacy=0x7f9454077ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> v1:172.21.15.83:6790/0 conn(0x7f94681042d0 legacy=0x7f94680fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 shutdown_connections 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 >> 172.21.15.83:0/4223053077 conn(0x7f9468074a20 msgr2=0x7f94680f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 shutdown_connections 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 -- 172.21.15.83:0/4223053077 wait complete. 2024-09-10T19:01:46.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:46.217+0000 7f946f7e8640 1 librados: shutdown 2024-09-10T19:01:46.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.272778+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.276591+0000 osd.4 (osd.4) 313 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.397393+0000 osd.5 (osd.5) 280 : cluster [DBG] 6.d scrub starts 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.400814+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.d scrub ok 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:44.576850+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:44.579461+0000 mon.c (mon.2) 643 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.602374+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.606386+0000 osd.0 (osd.0) 269 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.822631+0000 osd.6 (osd.6) 235 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: cluster 2024-09-10T19:01:44.825951+0000 osd.6 (osd.6) 236 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:45.244248+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:45.246290+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:45.247437+0000 mon.a (mon.0) 2719 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[32381]: audit 2024-09-10T19:01:45.249345+0000 mon.a (mon.0) 2720 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.272778+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.276591+0000 osd.4 (osd.4) 313 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.397393+0000 osd.5 (osd.5) 280 : cluster [DBG] 6.d scrub starts 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.400814+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.d scrub ok 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:44.576850+0000 mon.a (mon.0) 2718 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:44.579461+0000 mon.c (mon.2) 643 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.602374+0000 osd.0 (osd.0) 268 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.606386+0000 osd.0 (osd.0) 269 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.822631+0000 osd.6 (osd.6) 235 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: cluster 2024-09-10T19:01:44.825951+0000 osd.6 (osd.6) 236 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:45.244248+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:45.246290+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:45.247437+0000 mon.a (mon.0) 2719 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:46.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:46 smithi083 bash[18629]: audit 2024-09-10T19:01:45.249345+0000 mon.a (mon.0) 2720 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-73"}]: dispatch 2024-09-10T19:01:46.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:46.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.383 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:46.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:46.521 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:46.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:46.654 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:46.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.793 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:46.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:46.929 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:46.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:47.033 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:47.059 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:47.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:47.197 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:47.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:47.312 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:45.583451+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:01:47.312 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:45.868750+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: audit 2024-09-10T19:01:46.037233+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? v1:172.21.15.83:0/4223053077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.155495+0000 mgr.y (mgr.24526) 502 : cluster [DBG] pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.180457+0000 osd.2 (osd.2) 215 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.187992+0000 osd.2 (osd.2) 216 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.585611+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: audit 2024-09-10T19:01:46.589839+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.313 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:47 smithi165 bash[22591]: audit 2024-09-10T19:01:46.592982+0000 mon.a (mon.0) 2723 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.332 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:45.583451+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:45.868750+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: audit 2024-09-10T19:01:46.037233+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? v1:172.21.15.83:0/4223053077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.155495+0000 mgr.y (mgr.24526) 502 : cluster [DBG] pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.180457+0000 osd.2 (osd.2) 215 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.187992+0000 osd.2 (osd.2) 216 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.585611+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: audit 2024-09-10T19:01:46.589839+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[18629]: audit 2024-09-10T19:01:46.592982+0000 mon.a (mon.0) 2723 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:45.583451+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:45.868750+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:01:47.357 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: audit 2024-09-10T19:01:46.037233+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? v1:172.21.15.83:0/4223053077' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.155495+0000 mgr.y (mgr.24526) 502 : cluster [DBG] pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.180457+0000 osd.2 (osd.2) 215 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.187992+0000 osd.2 (osd.2) 216 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.585611+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: audit 2024-09-10T19:01:46.589839+0000 mon.b (mon.1) 375 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.358 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:47 smithi083 bash[32381]: audit 2024-09-10T19:01:46.592982+0000 mon.a (mon.0) 2723 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:47.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:47.469 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:47.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:47.610 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:47.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:47.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:47.740 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:47.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:47.876 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:47.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:47.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:48.014 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:48.117 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:48.152 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:48.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:48.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:48.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.391647+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:01:48.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: cluster 2024-09-10T19:01:46.395219+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:01:48.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.143718+0000 osd.2 (osd.2) 217 : cluster [DBG] 64.a scrub starts 2024-09-10T19:01:48.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.148009+0000 osd.2 (osd.2) 218 : cluster [DBG] 64.a scrub ok 2024-09-10T19:01:48.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: audit 2024-09-10T19:01:47.587926+0000 mon.a (mon.0) 2724 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.588115+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[32381]: audit 2024-09-10T19:01:47.595375+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.391647+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: cluster 2024-09-10T19:01:46.395219+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.143718+0000 osd.2 (osd.2) 217 : cluster [DBG] 64.a scrub starts 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.148009+0000 osd.2 (osd.2) 218 : cluster [DBG] 64.a scrub ok 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: audit 2024-09-10T19:01:47.587926+0000 mon.a (mon.0) 2724 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.588115+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:01:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:48 smithi083 bash[18629]: audit 2024-09-10T19:01:47.595375+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:48.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.391647+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.12 deep-scrub starts 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: cluster 2024-09-10T19:01:46.395219+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.12 deep-scrub ok 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.143718+0000 osd.2 (osd.2) 217 : cluster [DBG] 64.a scrub starts 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.148009+0000 osd.2 (osd.2) 218 : cluster [DBG] 64.a scrub ok 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: audit 2024-09-10T19:01:47.587926+0000 mon.a (mon.0) 2724 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.588115+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:01:48.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:48 smithi165 bash[22591]: audit 2024-09-10T19:01:47.595375+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:48.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.415 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:48.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.552 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:48.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:48.690 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:48.760 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:48.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:48.826 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:48.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.963 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:48.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:49.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:49.101 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:49.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:49.217 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.268636+0000 osd.7 (osd.7) 291 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:01:49.217 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.273604+0000 osd.7 (osd.7) 292 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.871685+0000 osd.6 (osd.6) 237 : cluster [DBG] 64.e scrub starts 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.877279+0000 osd.6 (osd.6) 238 : cluster [DBG] 64.e scrub ok 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.936261+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:47.939844+0000 osd.3 (osd.3) 300 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: audit 2024-09-10T19:01:48.065302+0000 mgr.y (mgr.24526) 503 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:48.156152+0000 mgr.y (mgr.24526) 504 : cluster [DBG] pgmap v755: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:48.594691+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:48.749641+0000 osd.1 (osd.1) 289 : cluster [DBG] 64.14 scrub starts 2024-09-10T19:01:49.218 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[22591]: cluster 2024-09-10T19:01:48.758326+0000 osd.1 (osd.1) 290 : cluster [DBG] 64.14 scrub ok 2024-09-10T19:01:49.232 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:49.336 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.268636+0000 osd.7 (osd.7) 291 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.273604+0000 osd.7 (osd.7) 292 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.871685+0000 osd.6 (osd.6) 237 : cluster [DBG] 64.e scrub starts 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.877279+0000 osd.6 (osd.6) 238 : cluster [DBG] 64.e scrub ok 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.936261+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:01:49.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:47.939844+0000 osd.3 (osd.3) 300 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: audit 2024-09-10T19:01:48.065302+0000 mgr.y (mgr.24526) 503 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:48.156152+0000 mgr.y (mgr.24526) 504 : cluster [DBG] pgmap v755: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:48.594691+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:48.749641+0000 osd.1 (osd.1) 289 : cluster [DBG] 64.14 scrub starts 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[18629]: cluster 2024-09-10T19:01:48.758326+0000 osd.1 (osd.1) 290 : cluster [DBG] 64.14 scrub ok 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.268636+0000 osd.7 (osd.7) 291 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.273604+0000 osd.7 (osd.7) 292 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.871685+0000 osd.6 (osd.6) 237 : cluster [DBG] 64.e scrub starts 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.877279+0000 osd.6 (osd.6) 238 : cluster [DBG] 64.e scrub ok 2024-09-10T19:01:49.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.936261+0000 osd.3 (osd.3) 299 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:47.939844+0000 osd.3 (osd.3) 300 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: audit 2024-09-10T19:01:48.065302+0000 mgr.y (mgr.24526) 503 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:48.156152+0000 mgr.y (mgr.24526) 504 : cluster [DBG] pgmap v755: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:48.594691+0000 mon.a (mon.0) 2726 : cluster [DBG] osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:48.749641+0000 osd.1 (osd.1) 289 : cluster [DBG] 64.14 scrub starts 2024-09-10T19:01:49.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[32381]: cluster 2024-09-10T19:01:48.758326+0000 osd.1 (osd.1) 290 : cluster [DBG] 64.14 scrub ok 2024-09-10T19:01:49.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:49.466 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:49.500 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:49.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:49.630 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:49.638 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[50483]: debug 2024-09-10T19:01:49.633+0000 7f6640c27640 -1 snap_mapper.add_oid found existing snaps mapped on 64:cb46d41c:test-rados-api-smithi083-77828-76::foo:28, removing 2024-09-10T19:01:49.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:49.764 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:49.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:49.882 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:01:49 smithi165 bash[48829]: debug 2024-09-10T19:01:49.632+0000 7f107a447640 -1 snap_mapper.add_oid found existing snaps mapped on 64:cb46d41c:test-rados-api-smithi083-77828-76::foo:28, removing 2024-09-10T19:01:49.902 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:49.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:50.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:50.040 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:01:49 smithi083 bash[35790]: debug 2024-09-10T19:01:49.633+0000 7f0e95e04640 -1 snap_mapper.add_oid found existing snaps mapped on 64:cb46d41c:test-rados-api-smithi083-77828-76::foo:28, removing 2024-09-10T19:01:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:50.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:50.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:50.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:50.311 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:50.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:01:50.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:50.452 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:50.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:50.588 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:50.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:50.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: cluster 2024-09-10T19:01:49.527589+0000 osd.0 (osd.0) 270 : cluster [DBG] 64.1f scrub starts 2024-09-10T19:01:50.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: cluster 2024-09-10T19:01:49.532535+0000 osd.0 (osd.0) 271 : cluster [DBG] 64.1f scrub ok 2024-09-10T19:01:50.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: cluster 2024-09-10T19:01:49.593200+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:01:50.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: audit 2024-09-10T19:01:49.676729+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: audit 2024-09-10T19:01:49.678266+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: audit 2024-09-10T19:01:49.679813+0000 mon.a (mon.0) 2728 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: audit 2024-09-10T19:01:49.681217+0000 mon.a (mon.0) 2729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.707 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:50 smithi165 bash[22591]: cluster 2024-09-10T19:01:50.156936+0000 mgr.y (mgr.24526) 505 : cluster [DBG] pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:50.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:50.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:50.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:50.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: cluster 2024-09-10T19:01:49.527589+0000 osd.0 (osd.0) 270 : cluster [DBG] 64.1f scrub starts 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: cluster 2024-09-10T19:01:49.532535+0000 osd.0 (osd.0) 271 : cluster [DBG] 64.1f scrub ok 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: cluster 2024-09-10T19:01:49.593200+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: audit 2024-09-10T19:01:49.676729+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: audit 2024-09-10T19:01:49.678266+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: audit 2024-09-10T19:01:49.679813+0000 mon.a (mon.0) 2728 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: audit 2024-09-10T19:01:49.681217+0000 mon.a (mon.0) 2729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[18629]: cluster 2024-09-10T19:01:50.156936+0000 mgr.y (mgr.24526) 505 : cluster [DBG] pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: cluster 2024-09-10T19:01:49.527589+0000 osd.0 (osd.0) 270 : cluster [DBG] 64.1f scrub starts 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: cluster 2024-09-10T19:01:49.532535+0000 osd.0 (osd.0) 271 : cluster [DBG] 64.1f scrub ok 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: cluster 2024-09-10T19:01:49.593200+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: audit 2024-09-10T19:01:49.676729+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: audit 2024-09-10T19:01:49.678266+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: audit 2024-09-10T19:01:49.679813+0000 mon.a (mon.0) 2728 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:50.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: audit 2024-09-10T19:01:49.681217+0000 mon.a (mon.0) 2729 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-75"}]: dispatch 2024-09-10T19:01:50.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:50 smithi083 bash[32381]: cluster 2024-09-10T19:01:50.156936+0000 mgr.y (mgr.24526) 505 : cluster [DBG] pgmap v758: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:50.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:50.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:51.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:51.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:51.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:51.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:51.262 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:51.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:51.396 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:51.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:51.534 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:51.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:51.667 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:51.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:51.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:51.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[22591]: cluster 2024-09-10T19:01:50.600060+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:01:51.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[22591]: cluster 2024-09-10T19:01:50.758031+0000 osd.1 (osd.1) 291 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:01:51.783 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[22591]: cluster 2024-09-10T19:01:50.763197+0000 osd.1 (osd.1) 292 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:01:51.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.117972+0000 osd.7 (osd.7) 293 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:01:51.784 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.122238+0000 osd.7 (osd.7) 294 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:01:51.784 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:01:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:01:51.803 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:51.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:51.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[18629]: cluster 2024-09-10T19:01:50.600060+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:01:51.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[18629]: cluster 2024-09-10T19:01:50.758031+0000 osd.1 (osd.1) 291 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:01:51.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[18629]: cluster 2024-09-10T19:01:50.763197+0000 osd.1 (osd.1) 292 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:01:51.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.117972+0000 osd.7 (osd.7) 293 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:01:51.919 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.122238+0000 osd.7 (osd.7) 294 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:01:51.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[32381]: cluster 2024-09-10T19:01:50.600060+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:01:51.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[32381]: cluster 2024-09-10T19:01:50.758031+0000 osd.1 (osd.1) 291 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:01:51.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[32381]: cluster 2024-09-10T19:01:50.763197+0000 osd.1 (osd.1) 292 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:01:51.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.117972+0000 osd.7 (osd.7) 293 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:01:51.920 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:51 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.122238+0000 osd.7 (osd.7) 294 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:01:51.941 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:51.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:52.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:52.076 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:52.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:52.253 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:01:52.253 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:01:52.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:52.344 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:52.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:52.524 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:52.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:52.653 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:52.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:52.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.157768+0000 osd.2 (osd.2) 219 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.163558+0000 osd.2 (osd.2) 220 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.612369+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: audit 2024-09-10T19:01:51.617279+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: audit 2024-09-10T19:01:51.623643+0000 mon.a (mon.0) 2732 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.932706+0000 osd.3 (osd.3) 301 : cluster [DBG] 4.1d deep-scrub starts 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:51.936442+0000 osd.3 (osd.3) 302 : cluster [DBG] 4.1d deep-scrub ok 2024-09-10T19:01:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:52 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.157729+0000 mgr.y (mgr.24526) 506 : cluster [DBG] pgmap v761: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:52.880 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:01:52] "GET /metrics HTTP/1.1" 200 228890 "" "Prometheus/2.43.0" 2024-09-10T19:01:52.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.157768+0000 osd.2 (osd.2) 219 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:01:52.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.163558+0000 osd.2 (osd.2) 220 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.612369+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: audit 2024-09-10T19:01:51.617279+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: audit 2024-09-10T19:01:51.623643+0000 mon.a (mon.0) 2732 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.932706+0000 osd.3 (osd.3) 301 : cluster [DBG] 4.1d deep-scrub starts 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:51.936442+0000 osd.3 (osd.3) 302 : cluster [DBG] 4.1d deep-scrub ok 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.157729+0000 mgr.y (mgr.24526) 506 : cluster [DBG] pgmap v761: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.157768+0000 osd.2 (osd.2) 219 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.163558+0000 osd.2 (osd.2) 220 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:01:52.883 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.612369+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:01:52.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: audit 2024-09-10T19:01:51.617279+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: audit 2024-09-10T19:01:51.623643+0000 mon.a (mon.0) 2732 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:52.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.932706+0000 osd.3 (osd.3) 301 : cluster [DBG] 4.1d deep-scrub starts 2024-09-10T19:01:52.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:51.936442+0000 osd.3 (osd.3) 302 : cluster [DBG] 4.1d deep-scrub ok 2024-09-10T19:01:52.884 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:52 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.157729+0000 mgr.y (mgr.24526) 506 : cluster [DBG] pgmap v761: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:52.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:52.926 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:52.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:53.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:53.061 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:53.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:53.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:53.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:53.335 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:53.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:53.473 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:53.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:53.612 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:53.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:53.749 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:53.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:53.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:53.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:53.938 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[42805]: debug 2024-09-10T19:01:53.665+0000 7f7d99d87640 -1 snap_mapper.add_oid found existing snaps mapped on 64:1046e01d:test-rados-api-smithi083-77828-78::foo:29, removing 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.600568+0000 osd.5 (osd.5) 284 : cluster [DBG] 64.f scrub starts 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.606032+0000 osd.5 (osd.5) 285 : cluster [DBG] 64.f scrub ok 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.609605+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: audit 2024-09-10T19:01:52.621750+0000 mon.a (mon.0) 2734 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.621879+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: audit 2024-09-10T19:01:52.623720+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:53.939 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: audit 2024-09-10T19:01:52.760526+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.888424+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 deep-scrub starts 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.890187+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 deep-scrub ok 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.600568+0000 osd.5 (osd.5) 284 : cluster [DBG] 64.f scrub starts 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.606032+0000 osd.5 (osd.5) 285 : cluster [DBG] 64.f scrub ok 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.609605+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: audit 2024-09-10T19:01:52.621750+0000 mon.a (mon.0) 2734 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.621879+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: audit 2024-09-10T19:01:52.623720+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: audit 2024-09-10T19:01:52.760526+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.888424+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 deep-scrub starts 2024-09-10T19:01:53.940 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:53 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.890187+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 deep-scrub ok 2024-09-10T19:01:53.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:54.002 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[26469]: debug 2024-09-10T19:01:53.664+0000 7f37c9619640 -1 snap_mapper.add_oid found existing snaps mapped on 64:1046e01d:test-rados-api-smithi083-77828-78::foo:29, removing 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.600568+0000 osd.5 (osd.5) 284 : cluster [DBG] 64.f scrub starts 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.606032+0000 osd.5 (osd.5) 285 : cluster [DBG] 64.f scrub ok 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.609605+0000 mon.a (mon.0) 2733 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: audit 2024-09-10T19:01:52.621750+0000 mon.a (mon.0) 2734 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.621879+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: audit 2024-09-10T19:01:52.623720+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: audit 2024-09-10T19:01:52.760526+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.888424+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.19 deep-scrub starts 2024-09-10T19:01:54.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.890187+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.19 deep-scrub ok 2024-09-10T19:01:54.004 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:01:53 smithi165 bash[41096]: debug 2024-09-10T19:01:53.664+0000 7fa5aba7e640 -1 snap_mapper.add_oid found existing snaps mapped on 64:1046e01d:test-rados-api-smithi083-77828-78::foo:29, removing 2024-09-10T19:01:54.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:54.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:54.163 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:54.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:54.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:54.438 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:54.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.576 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:54.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:01:54.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:54.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:54.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:54.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.783928+0000 osd.1 (osd.1) 293 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:01:54.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:52.787375+0000 osd.1 (osd.1) 294 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.150499+0000 osd.7 (osd.7) 295 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.156951+0000 osd.7 (osd.7) 296 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.219287+0000 osd.2 (osd.2) 221 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.222985+0000 osd.2 (osd.2) 222 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.640741+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.643873+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.649694+0000 osd.0 (osd.0) 273 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: audit 2024-09-10T19:01:53.695964+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: audit 2024-09-10T19:01:53.697728+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: audit 2024-09-10T19:01:53.699178+0000 mon.a (mon.0) 2737 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: audit 2024-09-10T19:01:53.700659+0000 mon.a (mon.0) 2738 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.807260+0000 osd.6 (osd.6) 239 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.810777+0000 osd.6 (osd.6) 240 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:01:54.830 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.076788+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:54.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.077279+0000 osd.5 (osd.5) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:54.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.158544+0000 mgr.y (mgr.24526) 507 : cluster [DBG] pgmap v764: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:54.831 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:54 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.181357+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:01:54.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:54.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.987 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:54.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.783928+0000 osd.1 (osd.1) 293 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:52.787375+0000 osd.1 (osd.1) 294 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.150499+0000 osd.7 (osd.7) 295 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.156951+0000 osd.7 (osd.7) 296 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.219287+0000 osd.2 (osd.2) 221 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.222985+0000 osd.2 (osd.2) 222 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.640741+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.643873+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:01:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.649694+0000 osd.0 (osd.0) 273 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: audit 2024-09-10T19:01:53.695964+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: audit 2024-09-10T19:01:53.697728+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: audit 2024-09-10T19:01:53.699178+0000 mon.a (mon.0) 2737 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: audit 2024-09-10T19:01:53.700659+0000 mon.a (mon.0) 2738 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.807260+0000 osd.6 (osd.6) 239 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.810777+0000 osd.6 (osd.6) 240 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.076788+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.077279+0000 osd.5 (osd.5) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.158544+0000 mgr.y (mgr.24526) 507 : cluster [DBG] pgmap v764: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.181357+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.783928+0000 osd.1 (osd.1) 293 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:01:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:52.787375+0000 osd.1 (osd.1) 294 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.150499+0000 osd.7 (osd.7) 295 : cluster [DBG] 64.4 scrub starts 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.156951+0000 osd.7 (osd.7) 296 : cluster [DBG] 64.4 scrub ok 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.219287+0000 osd.2 (osd.2) 221 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.222985+0000 osd.2 (osd.2) 222 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.640741+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.643873+0000 mon.a (mon.0) 2736 : cluster [DBG] osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.649694+0000 osd.0 (osd.0) 273 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: audit 2024-09-10T19:01:53.695964+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: audit 2024-09-10T19:01:53.697728+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: audit 2024-09-10T19:01:53.699178+0000 mon.a (mon.0) 2737 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: audit 2024-09-10T19:01:53.700659+0000 mon.a (mon.0) 2738 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-77"}]: dispatch 2024-09-10T19:01:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.807260+0000 osd.6 (osd.6) 239 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:01:55.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.810777+0000 osd.6 (osd.6) 240 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:01:55.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.076788+0000 osd.5 (osd.5) 286 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:55.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.077279+0000 osd.5 (osd.5) 287 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:55.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.158544+0000 mgr.y (mgr.24526) 507 : cluster [DBG] pgmap v764: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:01:55.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:54 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.181357+0000 mon.a (mon.0) 2739 : cluster [DBG] osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:01:55.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:55.127 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:01:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:01:55.231 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:55.262 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:55.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/4090750508 >> v1:172.21.15.83:6790/0 conn(0x7f771410b130 legacy=0x7f771410d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:55.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/4090750508 shutdown_connections 2024-09-10T19:01:55.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/4090750508 >> 172.21.15.83:0/4090750508 conn(0x7f77140fa3d0 msgr2=0x7f77140fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:55.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/4090750508 shutdown_connections 2024-09-10T19:01:55.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/4090750508 wait complete. 2024-09-10T19:01:55.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 librados: starting msgr at 2024-09-10T19:01:55.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 librados: starting objecter 2024-09-10T19:01:55.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 Processor -- start 2024-09-10T19:01:55.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- start start 2024-09-10T19:01:55.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 librados: setting wanted keys 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 librados: calling monclient init 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7714122660 con 0x7f7714104810 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7714123840 con 0x7f77141075a0 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7714124a40 con 0x7f771410b130 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f77137fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7714104810 0x7f77141008a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38366/0 (socket says 172.21.15.83:38366) 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f77137fe640 1 -- 172.21.15.83:0/751661980 learned_addr learned my addr 172.21.15.83:0/751661980 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter ms_handle_connect 0x7f771410b130 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter ms_handle_connect 0x7f7714104810 2024-09-10T19:01:55.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:55.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834122293 0 0) 0x7f7714124a40 con 0x7f771410b130 2024-09-10T19:01:55.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77000034e0 con 0x7f771410b130 2024-09-10T19:01:55.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter ms_handle_connect 0x7f77141075a0 2024-09-10T19:01:55.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:55.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48339826 0 0) 0x7f7714122660 con 0x7f7714104810 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7714124a40 con 0x7f7714104810 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771840946 0 0) 0x7f7714123840 con 0x7f77141075a0 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7714122660 con 0x7f77141075a0 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 739693132 0 0) 0x7f77000034e0 con 0x7f771410b130 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7714123840 con 0x7f771410b130 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f770c0023d0 con 0x7f771410b130 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995704800 0 0) 0x7f7714124a40 con 0x7f7714104810 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77000034e0 con 0x7f7714104810 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7704001980 con 0x7f7714104810 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299690228 0 0) 0x7f7714123840 con 0x7f771410b130 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 >> v1:172.21.15.165:6789/0 conn(0x7f77141075a0 legacy=0x7f7714100fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 >> v1:172.21.15.83:6789/0 conn(0x7f7714104810 legacy=0x7f77141008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714125c40 con 0x7f771410b130 2024-09-10T19:01:55.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7714123a30 con 0x7f771410b130 2024-09-10T19:01:55.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.361+0000 7f7719aaf640 10 client.49490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:55.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7714123e80 con 0x7f771410b130 2024-09-10T19:01:55.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7719aaf640 1 librados: init done 2024-09-10T19:01:55.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f770c0026a0 con 0x7f771410b130 2024-09-10T19:01:55.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f770c0038d0 con 0x7f771410b130 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77140fff30 con 0x7f771410b130 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.365+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f770c019cf0 con 0x7f771410b130 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(503..503 src has 69..503) v4 ==== 7991+0+0 (unknown 1763675483 0 0) 0x7f770c088350 con 0x7f771410b130 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 10 client.49490.objecter ms_dispatch 0x7f7714000b90 osd_map(503..503 src has 69..503) v4 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 3 client.49490.objecter handle_osd_map got epochs [503,503] > 0 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 3 client.49490.objecter handle_osd_map decoding full epoch 503 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 20 client.49490.objecter dump_active .. 0 homeless 2024-09-10T19:01:55.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.369+0000 7f7710ff9640 10 client.49490.objecter ms_handle_connect 0x7f7700071500 2024-09-10T19:01:55.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.373+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f770c054920 con 0x7f771410b130 2024-09-10T19:01:55.403 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:55.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:55.541 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:55.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:55.577+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f7714110bc0 con 0x7f771410b130 2024-09-10T19:01:55.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.677 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:55.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.804229+0000 osd.1 (osd.1) 295 : cluster [DBG] 64.1b scrub starts 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:53.808894+0000 osd.1 (osd.1) 296 : cluster [DBG] 64.1b scrub ok 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.616590+0000 osd.0 (osd.0) 274 : cluster [DBG] 4.6 deep-scrub starts 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.785926+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:54.789172+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.187027+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: audit 2024-09-10T19:01:55.190567+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: audit 2024-09-10T19:01:55.194024+0000 mon.a (mon.0) 2741 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: audit 2024-09-10T19:01:55.580613+0000 mon.c (mon.2) 646 : audit [INF] from='client.? v1:172.21.15.83:0/751661980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[18629]: audit 2024-09-10T19:01:55.581179+0000 mon.a (mon.0) 2742 : audit [INF] from='client.49490 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.804229+0000 osd.1 (osd.1) 295 : cluster [DBG] 64.1b scrub starts 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:53.808894+0000 osd.1 (osd.1) 296 : cluster [DBG] 64.1b scrub ok 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.616590+0000 osd.0 (osd.0) 274 : cluster [DBG] 4.6 deep-scrub starts 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.785926+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:54.789172+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.187027+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: audit 2024-09-10T19:01:55.190567+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: audit 2024-09-10T19:01:55.194024+0000 mon.a (mon.0) 2741 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: audit 2024-09-10T19:01:55.580613+0000 mon.c (mon.2) 646 : audit [INF] from='client.? v1:172.21.15.83:0/751661980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:55 smithi083 bash[32381]: audit 2024-09-10T19:01:55.581179+0000 mon.a (mon.0) 2742 : audit [INF] from='client.49490 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.817 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:55.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.804229+0000 osd.1 (osd.1) 295 : cluster [DBG] 64.1b scrub starts 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:53.808894+0000 osd.1 (osd.1) 296 : cluster [DBG] 64.1b scrub ok 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.616590+0000 osd.0 (osd.0) 274 : cluster [DBG] 4.6 deep-scrub starts 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.785926+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:54.789172+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.187027+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: audit 2024-09-10T19:01:55.190567+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: audit 2024-09-10T19:01:55.194024+0000 mon.a (mon.0) 2741 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:01:55.934 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: audit 2024-09-10T19:01:55.580613+0000 mon.c (mon.2) 646 : audit [INF] from='client.? v1:172.21.15.83:0/751661980' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.935 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:55 smithi165 bash[22591]: audit 2024-09-10T19:01:55.581179+0000 mon.a (mon.0) 2742 : audit [INF] from='client.49490 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:01:55.956 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:55.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:56.059 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:56.094 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:56.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:56.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.193+0000 7f7710ff9640 1 -- 172.21.15.83:0/751661980 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v504) v1 ==== 87+0+0 (unknown 3839522426 0 0) 0x7f770c0542c0 con 0x7f771410b130 2024-09-10T19:01:56.198 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.197+0000 7f7719aaf640 10 librados: watch_flush enter 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 10 librados: watch_flush exit 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 20 client.49490.objecter shutdown clearing up homeless session... 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 10 client.49490.objecter successfully canceled tick 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7700071500 legacy=0x7f77000739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:56.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 >> v1:172.21.15.83:6790/0 conn(0x7f771410b130 legacy=0x7f7714120f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:56.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 shutdown_connections 2024-09-10T19:01:56.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 >> 172.21.15.83:0/751661980 conn(0x7f77140fa3d0 msgr2=0x7f77140fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:56.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.201+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 shutdown_connections 2024-09-10T19:01:56.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.205+0000 7f7719aaf640 1 -- 172.21.15.83:0/751661980 wait complete. 2024-09-10T19:01:56.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:56.205+0000 7f7719aaf640 1 librados: shutdown 2024-09-10T19:01:56.225 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:56.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:56.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:56.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:56.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:56.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:56.634 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:56.668 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.212318+0000 osd.2 (osd.2) 223 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:01:56.668 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.215335+0000 osd.2 (osd.2) 224 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:01:56.668 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.597686+0000 osd.0 (osd.0) 275 : cluster [DBG] 6.10 deep-scrub starts 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.601265+0000 osd.0 (osd.0) 276 : cluster [DBG] 6.10 deep-scrub ok 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.603358+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:55.606422+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:56.160874+0000 mgr.y (mgr.24526) 508 : cluster [DBG] pgmap v767: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:56.180914+0000 mon.a (mon.0) 2743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: audit 2024-09-10T19:01:56.193953+0000 mon.a (mon.0) 2744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:56.669 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: audit 2024-09-10T19:01:56.194475+0000 mon.a (mon.0) 2745 : audit [INF] from='client.49490 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:01:56.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:56.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.212318+0000 osd.2 (osd.2) 223 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:01:56.752 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.215335+0000 osd.2 (osd.2) 224 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.597686+0000 osd.0 (osd.0) 275 : cluster [DBG] 6.10 deep-scrub starts 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.601265+0000 osd.0 (osd.0) 276 : cluster [DBG] 6.10 deep-scrub ok 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.603358+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:55.606422+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:56.160874+0000 mgr.y (mgr.24526) 508 : cluster [DBG] pgmap v767: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:56.180914+0000 mon.a (mon.0) 2743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: audit 2024-09-10T19:01:56.193953+0000 mon.a (mon.0) 2744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: audit 2024-09-10T19:01:56.194475+0000 mon.a (mon.0) 2745 : audit [INF] from='client.49490 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: cluster 2024-09-10T19:01:56.194564+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:01:56.753 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:56 smithi165 bash[22591]: audit 2024-09-10T19:01:56.194914+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:56.773 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:56.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:56.909 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:56.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:57.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:57.026 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: cluster 2024-09-10T19:01:56.194564+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:01:57.026 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[18629]: audit 2024-09-10T19:01:56.194914+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:57.026 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.212318+0000 osd.2 (osd.2) 223 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.215335+0000 osd.2 (osd.2) 224 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.597686+0000 osd.0 (osd.0) 275 : cluster [DBG] 6.10 deep-scrub starts 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.601265+0000 osd.0 (osd.0) 276 : cluster [DBG] 6.10 deep-scrub ok 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.603358+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:55.606422+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:56.160874+0000 mgr.y (mgr.24526) 508 : cluster [DBG] pgmap v767: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:56.180914+0000 mon.a (mon.0) 2743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: audit 2024-09-10T19:01:56.193953+0000 mon.a (mon.0) 2744 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: audit 2024-09-10T19:01:56.194475+0000 mon.a (mon.0) 2745 : audit [INF] from='client.49490 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: cluster 2024-09-10T19:01:56.194564+0000 mon.a (mon.0) 2746 : cluster [DBG] osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:01:57.027 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:56 smithi083 bash[32381]: audit 2024-09-10T19:01:56.194914+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:01:57.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:57.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:57.188 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:57.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:57.328 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:01:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:01:57.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:57.466 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:57.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:57.605 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:57.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:57.742 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:57.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:57.852 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[18629]: cluster 2024-09-10T19:01:56.692980+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:57.852 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[18629]: cluster 2024-09-10T19:01:56.693302+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:57.852 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[18629]: cluster 2024-09-10T19:01:57.194103+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:01:57.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[32381]: cluster 2024-09-10T19:01:56.692980+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:57.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[32381]: cluster 2024-09-10T19:01:56.693302+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:57.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:57 smithi083 bash[32381]: cluster 2024-09-10T19:01:57.194103+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:01:57.854 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:57 smithi165 bash[22591]: cluster 2024-09-10T19:01:56.692980+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:01:57.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:57 smithi165 bash[22591]: cluster 2024-09-10T19:01:56.693302+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:01:57.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:57 smithi165 bash[22591]: cluster 2024-09-10T19:01:57.194103+0000 mon.a (mon.0) 2747 : cluster [DBG] osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:01:57.870 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:57.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:57.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:58.005 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:58.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- 172.21.15.83:0/1642912161 >> v1:172.21.15.83:6790/0 conn(0x7fbe580fe150 legacy=0x7fbe581075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:58.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- 172.21.15.83:0/1642912161 shutdown_connections 2024-09-10T19:01:58.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- 172.21.15.83:0/1642912161 >> 172.21.15.83:0/1642912161 conn(0x7fbe580f7060 msgr2=0x7fbe580f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:58.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- 172.21.15.83:0/1642912161 shutdown_connections 2024-09-10T19:01:58.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- 172.21.15.83:0/1642912161 wait complete. 2024-09-10T19:01:58.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 librados: starting msgr at 2024-09-10T19:01:58.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 librados: starting objecter 2024-09-10T19:01:58.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 Processor -- start 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- start start 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 librados: setting wanted keys 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 librados: calling monclient init 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe58107b20 con 0x7fbe58108ee0 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe5811f7a0 con 0x7fbe580fd5b0 2024-09-10T19:01:58.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe5d571640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbe581209a0 con 0x7fbe581075f0 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe567fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fbe58108ee0 0x7fbe58119920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:38390/0 (socket says 172.21.15.83:38390) 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe567fc640 1 -- 172.21.15.83:0/2661446621 learned_addr learned my addr 172.21.15.83:0/2661446621 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe4f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe58108ee0 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe4f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe581075f0 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.037+0000 7fbe4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 10 client.?.objecter ms_handle_connect 0x7fbe580fd5b0 2024-09-10T19:01:58.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771770056 0 0) 0x7fbe58107b20 con 0x7fbe58108ee0 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe440038e0 con 0x7fbe58108ee0 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771837848 0 0) 0x7fbe581209a0 con 0x7fbe581075f0 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe58107b20 con 0x7fbe581075f0 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420579332 0 0) 0x7fbe5811f7a0 con 0x7fbe580fd5b0 2024-09-10T19:01:58.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe581209a0 con 0x7fbe580fd5b0 2024-09-10T19:01:58.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777643584 0 0) 0x7fbe58107b20 con 0x7fbe581075f0 2024-09-10T19:01:58.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe5811f7a0 con 0x7fbe581075f0 2024-09-10T19:01:58.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbe480018d0 con 0x7fbe581075f0 2024-09-10T19:01:58.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2222587492 0 0) 0x7fbe440038e0 con 0x7fbe58108ee0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe58107b20 con 0x7fbe58108ee0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbe40001670 con 0x7fbe58108ee0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2216175522 0 0) 0x7fbe581209a0 con 0x7fbe580fd5b0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe440038e0 con 0x7fbe580fd5b0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbe50001c40 con 0x7fbe580fd5b0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3131519875 0 0) 0x7fbe5811f7a0 con 0x7fbe581075f0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 >> v1:172.21.15.165:6789/0 conn(0x7fbe580fd5b0 legacy=0x7fbe58106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 >> v1:172.21.15.83:6789/0 conn(0x7fbe58108ee0 legacy=0x7fbe58119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58121ba0 con 0x7fbe581075f0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe58120b90 con 0x7fbe581075f0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fbe48002140 con 0x7fbe581075f0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbe48002fe0 con 0x7fbe581075f0 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe5d571640 10 client.49496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:01:58.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe58120fe0 con 0x7fbe581075f0 2024-09-10T19:01:58.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe5d571640 1 librados: init done 2024-09-10T19:01:58.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.041+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe58064d10 con 0x7fbe581075f0 2024-09-10T19:01:58.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fbe48001ce0 con 0x7fbe581075f0 2024-09-10T19:01:58.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(505..505 src has 69..505) v4 ==== 8004+0+0 (unknown 2962086461 0 0) 0x7fbe48087bb0 con 0x7fbe581075f0 2024-09-10T19:01:58.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 10 client.49496.objecter ms_dispatch 0x7fbe58000b90 osd_map(505..505 src has 69..505) v4 2024-09-10T19:01:58.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 3 client.49496.objecter handle_osd_map got epochs [505,505] > 0 2024-09-10T19:01:58.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 3 client.49496.objecter handle_osd_map decoding full epoch 505 2024-09-10T19:01:58.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 20 client.49496.objecter dump_active .. 0 homeless 2024-09-10T19:01:58.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.045+0000 7fbe4f7fe640 10 client.49496.objecter ms_handle_connect 0x7fbe440718b0 2024-09-10T19:01:58.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.049+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fbe480541d0 con 0x7fbe581075f0 2024-09-10T19:01:58.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:58.129 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:01:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:01:58.162 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:01:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:01:58.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:58.229+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) v1 -- 0x7fbe580664b0 con 0x7fbe581075f0 2024-09-10T19:01:58.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:58.295 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:58.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:58.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:58.564 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:58.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:58.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:58.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.842 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:58.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.974 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:58.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:59.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:59.110 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:01:59.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.197+0000 7fbe4f7fe640 1 -- 172.21.15.83:0/2661446621 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v507) v1 ==== 129+0+0 (unknown 3694982214 0 0) 0x7fbe48053b70 con 0x7fbe581075f0 2024-09-10T19:01:59.200 INFO:teuthology.orchestra.run.smithi083.stderr:set pool 1 pg_num to 11 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 10 librados: watch_flush enter 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 10 librados: watch_flush exit 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 20 client.49496.objecter shutdown clearing up homeless session... 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 10 client.49496.objecter successfully canceled tick 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 >> v1:172.21.15.83:6800/3071793735 conn(0x7fbe440718b0 legacy=0x7fbe44073d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:59.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.201+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 >> v1:172.21.15.83:6790/0 conn(0x7fbe581075f0 legacy=0x7fbe5811d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:59.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.205+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 shutdown_connections 2024-09-10T19:01:59.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.205+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 >> 172.21.15.83:0/2661446621 conn(0x7fbe580f7060 msgr2=0x7fbe580f9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:59.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.205+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 shutdown_connections 2024-09-10T19:01:59.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.205+0000 7fbe5d571640 1 -- 172.21.15.83:0/2661446621 wait complete. 2024-09-10T19:01:59.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:01:59.205+0000 7fbe5d571640 1 librados: shutdown 2024-09-10T19:01:59.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.073183+0000 mgr.y (mgr.24526) 509 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: cluster 2024-09-10T19:01:58.161651+0000 mgr.y (mgr.24526) 510 : cluster [DBG] pgmap v770: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: cluster 2024-09-10T19:01:58.197091+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.221814+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.224548+0000 mgr.y (mgr.24526) 511 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.234034+0000 mon.c (mon.2) 647 : audit [INF] from='client.? v1:172.21.15.83:0/2661446621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.234390+0000 mon.a (mon.0) 2749 : audit [INF] from='client.49496 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.224 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: cluster 2024-09-10T19:01:58.667087+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:59.224 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.952969+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.224 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:01:59 smithi165 bash[22591]: audit 2024-09-10T19:01:58.964355+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.236 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:01:59.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:01:59.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:59.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:59.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.073183+0000 mgr.y (mgr.24526) 509 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:59.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: cluster 2024-09-10T19:01:58.161651+0000 mgr.y (mgr.24526) 510 : cluster [DBG] pgmap v770: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:59.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: cluster 2024-09-10T19:01:58.197091+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.221814+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.224548+0000 mgr.y (mgr.24526) 511 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.234034+0000 mon.c (mon.2) 647 : audit [INF] from='client.? v1:172.21.15.83:0/2661446621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.234390+0000 mon.a (mon.0) 2749 : audit [INF] from='client.49496 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: cluster 2024-09-10T19:01:58.667087+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.952969+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[32381]: audit 2024-09-10T19:01:58.964355+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.073183+0000 mgr.y (mgr.24526) 509 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: cluster 2024-09-10T19:01:58.161651+0000 mgr.y (mgr.24526) 510 : cluster [DBG] pgmap v770: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: cluster 2024-09-10T19:01:58.197091+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.221814+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.224548+0000 mgr.y (mgr.24526) 511 : audit [DBG] from='mon.? v1:172.21.15.165:0/668304138' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.15"}]: dispatch 2024-09-10T19:01:59.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.234034+0000 mon.c (mon.2) 647 : audit [INF] from='client.? v1:172.21.15.83:0/2661446621' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.234390+0000 mon.a (mon.0) 2749 : audit [INF] from='client.49496 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:01:59.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: cluster 2024-09-10T19:01:58.667087+0000 mon.a (mon.0) 2750 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:59.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.952969+0000 mon.a (mon.0) 2751 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:01:59 smithi083 bash[18629]: audit 2024-09-10T19:01:58.964355+0000 mon.a (mon.0) 2752 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:01:59.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:59.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:01:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:01:59.617 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:59.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:01:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:01:59.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:59.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:01:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:01:59.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:01:59.894 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:59.925 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:01:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:01:59.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:00.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:00.056 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:00.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.188 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:00.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:00.327 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:00.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:00.465 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: cluster 2024-09-10T19:01:58.914287+0000 osd.1 (osd.1) 297 : cluster [DBG] 223.15 deep-scrub starts 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: cluster 2024-09-10T19:01:58.919808+0000 osd.1 (osd.1) 298 : cluster [DBG] 223.15 deep-scrub ok 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: cluster 2024-09-10T19:01:59.056920+0000 osd.1 (osd.1) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: cluster 2024-09-10T19:01:59.057124+0000 osd.1 (osd.1) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: audit 2024-09-10T19:01:59.197048+0000 mon.a (mon.0) 2753 : audit [INF] from='client.49496 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: cluster 2024-09-10T19:01:59.197134+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: audit 2024-09-10T19:01:59.575182+0000 mon.a (mon.0) 2755 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: audit 2024-09-10T19:01:59.577785+0000 mon.c (mon.2) 648 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: audit 2024-09-10T19:02:00.163006+0000 mon.c (mon.2) 649 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.468 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[32381]: audit 2024-09-10T19:02:00.164953+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: cluster 2024-09-10T19:01:58.914287+0000 osd.1 (osd.1) 297 : cluster [DBG] 223.15 deep-scrub starts 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: cluster 2024-09-10T19:01:58.919808+0000 osd.1 (osd.1) 298 : cluster [DBG] 223.15 deep-scrub ok 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: cluster 2024-09-10T19:01:59.056920+0000 osd.1 (osd.1) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: cluster 2024-09-10T19:01:59.057124+0000 osd.1 (osd.1) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: audit 2024-09-10T19:01:59.197048+0000 mon.a (mon.0) 2753 : audit [INF] from='client.49496 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: cluster 2024-09-10T19:01:59.197134+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: audit 2024-09-10T19:01:59.575182+0000 mon.a (mon.0) 2755 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:00.469 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: audit 2024-09-10T19:01:59.577785+0000 mon.c (mon.2) 648 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:00.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: audit 2024-09-10T19:02:00.163006+0000 mon.c (mon.2) 649 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.470 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:00 smithi083 bash[18629]: audit 2024-09-10T19:02:00.164953+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:00.558 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:00.559 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:00.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:00.599 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: cluster 2024-09-10T19:01:58.914287+0000 osd.1 (osd.1) 297 : cluster [DBG] 223.15 deep-scrub starts 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: cluster 2024-09-10T19:01:58.919808+0000 osd.1 (osd.1) 298 : cluster [DBG] 223.15 deep-scrub ok 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: cluster 2024-09-10T19:01:59.056920+0000 osd.1 (osd.1) 299 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: cluster 2024-09-10T19:01:59.057124+0000 osd.1 (osd.1) 300 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: audit 2024-09-10T19:01:59.197048+0000 mon.a (mon.0) 2753 : audit [INF] from='client.49496 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: cluster 2024-09-10T19:01:59.197134+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: audit 2024-09-10T19:01:59.575182+0000 mon.a (mon.0) 2755 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: audit 2024-09-10T19:01:59.577785+0000 mon.c (mon.2) 648 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: audit 2024-09-10T19:02:00.163006+0000 mon.c (mon.2) 649 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:00 smithi165 bash[22591]: audit 2024-09-10T19:02:00.164953+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:00.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.738 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:00.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.875 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:00.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:00.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:01.010 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:01.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:01.144 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:01.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:01.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:01 smithi165 bash[22591]: cluster 2024-09-10T19:02:00.162459+0000 mgr.y (mgr.24526) 512 : cluster [DBG] pgmap v773: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:01.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:01 smithi165 bash[22591]: audit 2024-09-10T19:02:00.584481+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:02:01.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:01 smithi165 bash[22591]: cluster 2024-09-10T19:02:00.584598+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:02:01.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:01.418 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:01.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:01.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[18629]: cluster 2024-09-10T19:02:00.162459+0000 mgr.y (mgr.24526) 512 : cluster [DBG] pgmap v773: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[18629]: audit 2024-09-10T19:02:00.584481+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[18629]: cluster 2024-09-10T19:02:00.584598+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[32381]: cluster 2024-09-10T19:02:00.162459+0000 mgr.y (mgr.24526) 512 : cluster [DBG] pgmap v773: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[32381]: audit 2024-09-10T19:02:00.584481+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:02:01.530 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:01 smithi083 bash[32381]: cluster 2024-09-10T19:02:00.584598+0000 mon.a (mon.0) 2758 : cluster [DBG] osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:02:01.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:01.651 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:01.682 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:01.785 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:01.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:01.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:01.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:01.959 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:01.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:02.062 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:02.093 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:02.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:02.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.364 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:02.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:02.500 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:02.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:02.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.762 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:02.852 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:02] "GET /metrics HTTP/1.1" 200 228885 "" "Prometheus/2.43.0" 2024-09-10T19:02:02.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: cluster 2024-09-10T19:02:01.589081+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:02:02.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: cluster 2024-09-10T19:02:02.163423+0000 mgr.y (mgr.24526) 513 : cluster [DBG] pgmap v776: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:02.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: audit 2024-09-10T19:02:02.164108+0000 mon.c (mon.2) 650 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: audit 2024-09-10T19:02:02.166031+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: audit 2024-09-10T19:02:02.373327+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[32381]: audit 2024-09-10T19:02:02.384115+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: cluster 2024-09-10T19:02:01.589081+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: cluster 2024-09-10T19:02:02.163423+0000 mgr.y (mgr.24526) 513 : cluster [DBG] pgmap v776: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: audit 2024-09-10T19:02:02.164108+0000 mon.c (mon.2) 650 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: audit 2024-09-10T19:02:02.166031+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: audit 2024-09-10T19:02:02.373327+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.853 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:02 smithi083 bash[18629]: audit 2024-09-10T19:02:02.384115+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: cluster 2024-09-10T19:02:01.589081+0000 mon.a (mon.0) 2759 : cluster [DBG] osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:02:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: cluster 2024-09-10T19:02:02.163423+0000 mgr.y (mgr.24526) 513 : cluster [DBG] pgmap v776: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: audit 2024-09-10T19:02:02.164108+0000 mon.c (mon.2) 650 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: audit 2024-09-10T19:02:02.166031+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:02:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: audit 2024-09-10T19:02:02.373327+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:02 smithi165 bash[22591]: audit 2024-09-10T19:02:02.384115+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:02.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:02.898 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:02.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:03.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:03.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:03.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:03.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:03.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:03.309 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:03.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:03.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:03.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:03.579 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:03.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:03 smithi165 bash[22591]: audit 2024-09-10T19:02:02.602710+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:02:03.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:03 smithi165 bash[22591]: cluster 2024-09-10T19:02:02.602794+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:02:03.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:03 smithi165 bash[22591]: audit 2024-09-10T19:02:02.793780+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:03.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:03 smithi165 bash[22591]: audit 2024-09-10T19:02:02.795075+0000 mon.c (mon.2) 652 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:03.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:03 smithi165 bash[22591]: audit 2024-09-10T19:02:02.803494+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:03.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:03.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:03.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:03.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[18629]: audit 2024-09-10T19:02:02.602710+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[18629]: cluster 2024-09-10T19:02:02.602794+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[18629]: audit 2024-09-10T19:02:02.793780+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[18629]: audit 2024-09-10T19:02:02.795075+0000 mon.c (mon.2) 652 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[18629]: audit 2024-09-10T19:02:02.803494+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[32381]: audit 2024-09-10T19:02:02.602710+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24526 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[32381]: cluster 2024-09-10T19:02:02.602794+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[32381]: audit 2024-09-10T19:02:02.793780+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[32381]: audit 2024-09-10T19:02:02.795075+0000 mon.c (mon.2) 652 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:03 smithi083 bash[32381]: audit 2024-09-10T19:02:02.803494+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:04.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.120 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:04.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:04.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:04.386 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:04.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.520 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:04.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:04.646 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:04.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:04.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[18629]: cluster 2024-09-10T19:02:03.601635+0000 mon.a (mon.0) 2766 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:02:04.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[18629]: cluster 2024-09-10T19:02:04.164390+0000 mgr.y (mgr.24526) 514 : cluster [DBG] pgmap v779: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:04.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[18629]: cluster 2024-09-10T19:02:04.193521+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:02:04.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[32381]: cluster 2024-09-10T19:02:03.601635+0000 mon.a (mon.0) 2766 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:02:04.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[32381]: cluster 2024-09-10T19:02:04.164390+0000 mgr.y (mgr.24526) 514 : cluster [DBG] pgmap v779: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:04.763 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:04 smithi083 bash[32381]: cluster 2024-09-10T19:02:04.193521+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:02:04.783 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:04.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:04.899 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:04.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:04 smithi165 bash[22591]: cluster 2024-09-10T19:02:03.601635+0000 mon.a (mon.0) 2766 : cluster [DBG] osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:02:04.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:04 smithi165 bash[22591]: cluster 2024-09-10T19:02:04.164390+0000 mgr.y (mgr.24526) 514 : cluster [DBG] pgmap v779: 238 pgs: 10 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:02:04.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:04 smithi165 bash[22591]: cluster 2024-09-10T19:02:04.193521+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:02:04.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:04.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:05.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:05.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:05.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:02:05.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:02:05.079 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:02:05.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:02:05.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:05.198 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:05.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.335 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:05.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:05.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:05.609 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:05.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:05.745 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:05.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:05.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:05.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:05.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:06.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:06.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:06.156 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:06.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:06.293 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:06.396 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:06.424 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:06.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:06.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:06.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:06.830 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:06.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- 172.21.15.83:0/1168554594 >> v1:172.21.15.83:6789/0 conn(0x7fc25c107e00 legacy=0x7fc25c10a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:06.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- 172.21.15.83:0/1168554594 shutdown_connections 2024-09-10T19:02:06.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- 172.21.15.83:0/1168554594 >> 172.21.15.83:0/1168554594 conn(0x7fc25c0f70b0 msgr2=0x7fc25c0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:06.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- 172.21.15.83:0/1168554594 shutdown_connections 2024-09-10T19:02:06.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- 172.21.15.83:0/1168554594 wait complete. 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 librados: starting msgr at 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 librados: starting objecter 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 Processor -- start 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- start start 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 librados: setting wanted keys 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 librados: calling monclient init 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c107bc0 con 0x7fc25c107e00 2024-09-10T19:02:06.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c11f690 con 0x7fc25c101530 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.945+0000 7fc264638640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc25c120830 con 0x7fc25c1042a0 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc262bd5640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc25c107e00 0x7fc25c11cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34174/0 (socket says 172.21.15.83:34174) 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc261bd3640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc25c1042a0 0x7fc25c119830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33884/0 (socket says 172.21.15.83:33884) 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc262bd5640 1 -- 172.21.15.83:0/4178580181 learned_addr learned my addr 172.21.15.83:0/4178580181 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter ms_handle_connect 0x7fc25c1042a0 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter ms_handle_connect 0x7fc25c107e00 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter ms_handle_connect 0x7fc25c101530 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534403264 0 0) 0x7fc25c120830 con 0x7fc25c1042a0 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2480034e0 con 0x7fc25c1042a0 2024-09-10T19:02:06.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355476011 0 0) 0x7fc25c11f690 con 0x7fc25c101530 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c120830 con 0x7fc25c101530 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3251458361 0 0) 0x7fc2480034e0 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c11f690 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc258001a50 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3892842014 0 0) 0x7fc25c11f690 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.165:6789/0 conn(0x7fc25c101530 legacy=0x7fc25c1072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6789/0 conn(0x7fc25c107e00 legacy=0x7fc25c11cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c121a10 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c11e6e0 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc264638640 10 client.49502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c11eb70 con 0x7fc25c1042a0 2024-09-10T19:02:06.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc264638640 1 librados: init done 2024-09-10T19:02:06.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc258002490 con 0x7fc25c1042a0 2024-09-10T19:02:06.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.949+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc258005af0 con 0x7fc25c1042a0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc258002030 con 0x7fc25c1042a0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc240004ff0 con 0x7fc25c1042a0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(512..512 src has 69..512) v4 ==== 8004+0+0 (unknown 2155148716 0 0) 0x7fc25808a530 con 0x7fc25c1042a0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 osd_map(512..512 src has 69..512) v4 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 3 client.49502.objecter handle_osd_map got epochs [512,512] > 0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 3 client.49502.objecter handle_osd_map decoding full epoch 512 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 20 client.49502.objecter dump_active .. 0 homeless 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v512) v1 ==== 54+0+16 (unknown 3993684219 0 2420000192) 0x7fc258056b50 con 0x7fc25c1042a0 2024-09-10T19:02:06.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc248071660 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 10 client.49502.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 1 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 1 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 0 1 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 4 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 10 client.49502.objecter _send_command 1 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240005290 con 0x7fc240004610 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc240004610 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc240005290 con 0x7fc240004610 2024-09-10T19:02:06.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:02:06.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 10 client.49502.objecter _finish_command 1 = system:0 2024-09-10T19:02:06.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 4 2024-09-10T19:02:06.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.953+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 0 1 2024-09-10T19:02:06.960 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:06.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 10 client.49502.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 2 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 2 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240005810 osd=0 3 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 0 2 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 4 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 10 client.49502.objecter _send_command 2 2024-09-10T19:02:06.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240004cc0 con 0x7fc240004610 2024-09-10T19:02:06.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc240004cc0 con 0x7fc240004610 2024-09-10T19:02:06.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:06.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc252ffd640 10 client.49502.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:06.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 4 2024-09-10T19:02:06.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 0 2 2024-09-10T19:02:06.974 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 10 client.49502.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 1 3 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 4 2024-09-10T19:02:06.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 10 client.49502.objecter _send_command 3 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.969+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc2400022b0 con 0x7fc240004110 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc240004110 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc2400022b0 con 0x7fc240004110 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 10 client.49502.objecter _finish_command 3 = system:0 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 4 2024-09-10T19:02:06.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.973+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 1 3 2024-09-10T19:02:06.977 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:02:06.977 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 10 client.49502.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 4 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 4 2024-09-10T19:02:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240003fa0 osd=1 3 2024-09-10T19:02:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 1 4 2024-09-10T19:02:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 4 2024-09-10T19:02:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 10 client.49502.objecter _send_command 4 2024-09-10T19:02:06.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240001890 con 0x7fc240004110 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc240001890 con 0x7fc240004110 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc252ffd640 10 client.49502.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 4 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 1 4 2024-09-10T19:02:06.997 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 10 client.49502.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 5 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 5 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 2 5 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 4 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 10 client.49502.objecter _send_command 5 2024-09-10T19:02:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.993+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240003df0 con 0x7fc24000d350 2024-09-10T19:02:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc24000d350 2024-09-10T19:02:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc240003df0 con 0x7fc24000d350 2024-09-10T19:02:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:02:06.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 10 client.49502.objecter _finish_command 5 = system:0 2024-09-10T19:02:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 4 2024-09-10T19:02:07.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:06.997+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 2 5 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 10 client.49502.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 6 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 6 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc2400035e0 osd=2 3 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 2 6 2024-09-10T19:02:07.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 4 2024-09-10T19:02:07.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 10 client.49502.objecter _send_command 6 2024-09-10T19:02:07.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc2400030d0 con 0x7fc24000d350 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc2400030d0 con 0x7fc24000d350 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc252ffd640 10 client.49502.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 4 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.017+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 2 6 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:02:07.022 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 10 client.49502.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 7 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 7 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 3 7 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 4 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 10 client.49502.objecter _send_command 7 2024-09-10T19:02:07.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24000d080 con 0x7fc240010e50 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc240010e50 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc24000d080 con 0x7fc240010e50 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 10 client.49502.objecter _finish_command 7 = system:0 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 4 2024-09-10T19:02:07.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.021+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 3 7 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 10 client.49502.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 8 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 8 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24000c990 osd=3 3 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 3 8 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 4 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 10 client.49502.objecter _send_command 8 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc2400132d0 con 0x7fc240010e50 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc2400132d0 con 0x7fc240010e50 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 10 client.49502.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 4 2024-09-10T19:02:07.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 3 8 2024-09-10T19:02:07.045 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:02:07.045 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 10 client.49502.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 9 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 9 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 4 9 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 4 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 10 client.49502.objecter _send_command 9 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24001a260 con 0x7fc240016760 2024-09-10T19:02:07.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.041+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc240016760 2024-09-10T19:02:07.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.045+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc24001a260 con 0x7fc240016760 2024-09-10T19:02:07.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.045+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:02:07.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.045+0000 7fc252ffd640 10 client.49502.objecter _finish_command 9 = system:0 2024-09-10T19:02:07.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.045+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 4 2024-09-10T19:02:07.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.045+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 4 9 2024-09-10T19:02:07.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 10 client.49502.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 10 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 10 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc2400132d0 osd=4 3 2024-09-10T19:02:07.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 4 10 2024-09-10T19:02:07.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 4 2024-09-10T19:02:07.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 10 client.49502.objecter _send_command 10 2024-09-10T19:02:07.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.069+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc24001a260 con 0x7fc240016760 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc24001a260 con 0x7fc240016760 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc252ffd640 10 client.49502.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 4 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 4 10 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:02:07.077 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 10 client.49502.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 11 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 11 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 5 11 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 4 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 10 client.49502.objecter _send_command 11 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.073+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24001e590 con 0x7fc24001a8e0 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc24001a8e0 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc24001e590 con 0x7fc24001a8e0 2024-09-10T19:02:07.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:02:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 10 client.49502.objecter _finish_command 11 = system:0 2024-09-10T19:02:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 4 2024-09-10T19:02:07.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.077+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 5 11 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 10 client.49502.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 12 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 12 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24001a260 osd=5 3 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 5 12 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 4 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 10 client.49502.objecter _send_command 12 2024-09-10T19:02:07.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.089+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc24001f710 con 0x7fc24001a8e0 2024-09-10T19:02:07.098 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:07.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc24001f710 con 0x7fc24001a8e0 2024-09-10T19:02:07.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 10 client.49502.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 4 2024-09-10T19:02:07.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 5 12 2024-09-10T19:02:07.100 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:02:07.100 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:02:07.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 10 client.49502.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:07.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 13 2024-09-10T19:02:07.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 13 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 6 13 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 4 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 10 client.49502.objecter _send_command 13 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240013120 con 0x7fc2400205f0 2024-09-10T19:02:07.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.097+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc2400205f0 2024-09-10T19:02:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:07.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.101+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc240013120 con 0x7fc2400205f0 2024-09-10T19:02:07.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.101+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:02:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.101+0000 7fc252ffd640 10 client.49502.objecter _finish_command 13 = system:0 2024-09-10T19:02:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.101+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 4 2024-09-10T19:02:07.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.101+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 6 13 2024-09-10T19:02:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 10 client.49502.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 14 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 14 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc24001f710 osd=6 3 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 6 14 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 4 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 10 client.49502.objecter _send_command 14 2024-09-10T19:02:07.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.121+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240013120 con 0x7fc2400205f0 2024-09-10T19:02:07.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc240013120 con 0x7fc2400205f0 2024-09-10T19:02:07.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc252ffd640 10 client.49502.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 4 2024-09-10T19:02:07.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 6 14 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 10 client.49502.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 15 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 15 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 7 15 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 4 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 10 client.49502.objecter _send_command 15 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.125+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc240026de0 con 0x7fc240020ba0 2024-09-10T19:02:07.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 10 client.49502.objecter ms_handle_connect 0x7fc240020ba0 2024-09-10T19:02:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc240026de0 con 0x7fc240020ba0 2024-09-10T19:02:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:02:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 10 client.49502.objecter _finish_command 15 = system:0 2024-09-10T19:02:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 4 2024-09-10T19:02:07.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.129+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 7 15 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 10 client.49502.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign -1 16 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 20 client.49502.objecter _get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 15 client.49502.objecter _session_command_op_remove -1 16 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 20 client.49502.objecter get_session s=0x7fc240013120 osd=7 3 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 15 client.49502.objecter _session_command_op_assign 7 16 2024-09-10T19:02:07.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 4 2024-09-10T19:02:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 10 client.49502.objecter _send_command 16 2024-09-10T19:02:07.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.141+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc240028680 con 0x7fc240020ba0 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.145+0000 7fc252ffd640 1 -- 172.21.15.83:0/4178580181 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc240028680 con 0x7fc240020ba0 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.145+0000 7fc252ffd640 10 client.49502.objecter ms_dispatch 0x7fc25c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.145+0000 7fc252ffd640 10 client.49502.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.145+0000 7fc252ffd640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 4 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.145+0000 7fc252ffd640 15 client.49502.objecter _session_command_op_remove 7 16 2024-09-10T19:02:07.150 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 librados: watch_flush enter 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 librados: watch_flush exit 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.0 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6801/2535527483 conn(0x7fc240004610 legacy=0x7fc240002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240005810 osd=0 1 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.1 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6805/2309624637 conn(0x7fc240004110 legacy=0x7fc24000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240003fa0 osd=1 1 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.2 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6809/386094220 conn(0x7fc24000d350 legacy=0x7fc24000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400035e0 osd=2 1 2024-09-10T19:02:07.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.3 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6813/1279688021 conn(0x7fc240010e50 legacy=0x7fc240011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24000c990 osd=3 1 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.4 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.165:6800/3225515265 conn(0x7fc240016760 legacy=0x7fc240018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc2400132d0 osd=4 1 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.5 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc262bd5640 1 -- 172.21.15.83:0/4178580181 reap_dead start 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.165:6804/571173777 conn(0x7fc24001a8e0 legacy=0x7fc24001cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc262bd5640 1 -- 172.21.15.83:0/4178580181 reap_dead start 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc262bd5640 1 -- 172.21.15.83:0/4178580181 reap_dead start 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001a260 osd=5 1 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.6 2024-09-10T19:02:07.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.165:6808/1317922936 conn(0x7fc2400205f0 legacy=0x7fc240011990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc24001f710 osd=6 1 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter close_session for osd.7 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.165:6812/1610151650 conn(0x7fc240020ba0 legacy=0x7fc240025550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter put_session s=0x7fc240013120 osd=7 1 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 20 client.49502.objecter shutdown clearing up homeless session... 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 10 client.49502.objecter successfully canceled tick 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc248071660 legacy=0x7fc248073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> v1:172.21.15.83:6790/0 conn(0x7fc25c1042a0 legacy=0x7fc25c119830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc262bd5640 1 -- 172.21.15.83:0/4178580181 reap_dead start 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 shutdown_connections 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 >> 172.21.15.83:0/4178580181 conn(0x7fc25c0f70b0 msgr2=0x7fc25c0fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 shutdown_connections 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 -- 172.21.15.83:0/4178580181 wait complete. 2024-09-10T19:02:07.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:07.149+0000 7fc264638640 1 librados: shutdown 2024-09-10T19:02:07.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:07.238 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:07.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:07 smithi165 bash[22591]: cluster 2024-09-10T19:02:06.166728+0000 mgr.y (mgr.24526) 515 : cluster [DBG] pgmap v781: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-10T19:02:07.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:07 smithi165 bash[22591]: audit 2024-09-10T19:02:06.955203+0000 mon.c (mon.2) 653 : audit [DBG] from='client.? v1:172.21.15.83:0/4178580181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:07.376 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:07.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:07.518 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:07 smithi083 bash[18629]: cluster 2024-09-10T19:02:06.166728+0000 mgr.y (mgr.24526) 515 : cluster [DBG] pgmap v781: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-10T19:02:07.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:07 smithi083 bash[18629]: audit 2024-09-10T19:02:06.955203+0000 mon.c (mon.2) 653 : audit [DBG] from='client.? v1:172.21.15.83:0/4178580181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:07 smithi083 bash[32381]: cluster 2024-09-10T19:02:06.166728+0000 mgr.y (mgr.24526) 515 : cluster [DBG] pgmap v781: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-09-10T19:02:07.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:07 smithi083 bash[32381]: audit 2024-09-10T19:02:06.955203+0000 mon.c (mon.2) 653 : audit [DBG] from='client.? v1:172.21.15.83:0/4178580181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:07.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:07.654 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:07.757 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:07.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:07.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:07.929 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:07.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:08.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:08.063 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:08.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:08.204 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:08.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:08.341 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:08.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:08.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:08.481 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:08.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:08.619 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:08.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:08.758 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:08.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:02:08.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:08.893 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:08.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:08.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:09.035 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:09.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:09.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:09.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:09.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[18629]: audit 2024-09-10T19:02:08.084444+0000 mgr.y (mgr.24526) 516 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:09.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[18629]: cluster 2024-09-10T19:02:08.167506+0000 mgr.y (mgr.24526) 517 : cluster [DBG] pgmap v782: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[18629]: cluster 2024-09-10T19:02:08.705637+0000 osd.4 (osd.4) 314 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[18629]: cluster 2024-09-10T19:02:08.705956+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[32381]: audit 2024-09-10T19:02:08.084444+0000 mgr.y (mgr.24526) 516 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[32381]: cluster 2024-09-10T19:02:08.167506+0000 mgr.y (mgr.24526) 517 : cluster [DBG] pgmap v782: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[32381]: cluster 2024-09-10T19:02:08.705637+0000 osd.4 (osd.4) 314 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:09.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:09 smithi083 bash[32381]: cluster 2024-09-10T19:02:08.705956+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:09.310 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:09.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:09.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:09 smithi165 bash[22591]: audit 2024-09-10T19:02:08.084444+0000 mgr.y (mgr.24526) 516 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:09.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:09 smithi165 bash[22591]: cluster 2024-09-10T19:02:08.167506+0000 mgr.y (mgr.24526) 517 : cluster [DBG] pgmap v782: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:02:09.427 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:09 smithi165 bash[22591]: cluster 2024-09-10T19:02:08.705637+0000 osd.4 (osd.4) 314 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:09.428 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:09 smithi165 bash[22591]: cluster 2024-09-10T19:02:08.705956+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:09.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:09.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:09.581 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:09.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:09.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:09.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:09.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:09.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:09.985 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:09.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:10.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:10.117 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:10.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:10.253 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:10.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:10.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:10.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:10.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:10 smithi083 bash[18629]: cluster 2024-09-10T19:02:09.236195+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:02:10.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:10 smithi083 bash[32381]: cluster 2024-09-10T19:02:09.236195+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:02:10.530 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:10.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:10.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:10 smithi165 bash[22591]: cluster 2024-09-10T19:02:09.236195+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:02:10.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:10.665 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:10.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:10.804 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:10.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:10.943 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:10.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:11.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:11.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.113+0000 7fd892de4640 1 -- 172.21.15.83:0/3329229207 >> v1:172.21.15.83:6789/0 conn(0x7fd88c1075a0 legacy=0x7fd88c109980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.113+0000 7fd892de4640 1 -- 172.21.15.83:0/3329229207 shutdown_connections 2024-09-10T19:02:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.113+0000 7fd892de4640 1 -- 172.21.15.83:0/3329229207 >> 172.21.15.83:0/3329229207 conn(0x7fd88c0fa3d0 msgr2=0x7fd88c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.113+0000 7fd892de4640 1 -- 172.21.15.83:0/3329229207 shutdown_connections 2024-09-10T19:02:11.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.113+0000 7fd892de4640 1 -- 172.21.15.83:0/3329229207 wait complete. 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 librados: starting msgr at 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 librados: starting objecter 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 Processor -- start 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- start start 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 librados: setting wanted keys 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 librados: calling monclient init 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd88c11e2f0 con 0x7fd88c10b130 2024-09-10T19:02:11.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd88c11f4b0 con 0x7fd88c104810 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd88c1206b0 con 0x7fd88c1075a0 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd88bfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fd88c1075a0 0x7fd88c1010f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33904/0 (socket says 172.21.15.83:33904) 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd88bfff640 1 -- 172.21.15.83:0/750618570 learned_addr learned my addr 172.21.15.83:0/750618570 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter ms_handle_connect 0x7fd88c10b130 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter ms_handle_connect 0x7fd88c1075a0 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter ms_handle_connect 0x7fd88c104810 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:11.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691910367 0 0) 0x7fd88c11e2f0 con 0x7fd88c10b130 2024-09-10T19:02:11.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003120 con 0x7fd88c10b130 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753675336 0 0) 0x7fd88c11f4b0 con 0x7fd88c104810 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88c11e2f0 con 0x7fd88c104810 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579105117 0 0) 0x7fd88c1206b0 con 0x7fd88c1075a0 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88c11f4b0 con 0x7fd88c1075a0 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1753459929 0 0) 0x7fd874003120 con 0x7fd88c10b130 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd88c1206b0 con 0x7fd88c10b130 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd884001b80 con 0x7fd88c10b130 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320766955 0 0) 0x7fd88c11f4b0 con 0x7fd88c1075a0 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd874003120 con 0x7fd88c1075a0 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294237836 0 0) 0x7fd88c11e2f0 con 0x7fd88c104810 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd88c11f4b0 con 0x7fd88c104810 2024-09-10T19:02:11.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd8800018d0 con 0x7fd88c104810 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd87c0021f0 con 0x7fd88c1075a0 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1943248412 0 0) 0x7fd88c1206b0 con 0x7fd88c10b130 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 >> v1:172.21.15.83:6790/0 conn(0x7fd88c1075a0 legacy=0x7fd88c1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 >> v1:172.21.15.165:6789/0 conn(0x7fd88c104810 legacy=0x7fd88c1009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c1218b0 con 0x7fd88c10b130 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c11f6a0 con 0x7fd88c10b130 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 10 client.25434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c11faf0 con 0x7fd88c10b130 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd892de4640 1 librados: init done 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd884003100 con 0x7fd88c10b130 2024-09-10T19:02:11.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.117+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd884003450 con 0x7fd88c10b130 2024-09-10T19:02:11.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd878005010 con 0x7fd88c10b130 2024-09-10T19:02:11.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd884001ce0 con 0x7fd88c10b130 2024-09-10T19:02:11.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(513..513 src has 69..513) v4 ==== 8004+0+0 (unknown 2218645810 0 0) 0x7fd884087e10 con 0x7fd88c10b130 2024-09-10T19:02:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 10 client.25434.objecter ms_dispatch 0x7fd88c000b90 osd_map(513..513 src has 69..513) v4 2024-09-10T19:02:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 3 client.25434.objecter handle_osd_map got epochs [513,513] > 0 2024-09-10T19:02:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 3 client.25434.objecter handle_osd_map decoding full epoch 513 2024-09-10T19:02:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 20 client.25434.objecter dump_active .. 0 homeless 2024-09-10T19:02:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.121+0000 7fd8897fa640 10 client.25434.objecter ms_handle_connect 0x7fd874071740 2024-09-10T19:02:11.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.129+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd884053f50 con 0x7fd88c10b130 2024-09-10T19:02:11.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:11.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:11.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:11.321+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd8780052c0 con 0x7fd88c10b130 2024-09-10T19:02:11.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:11.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:11.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:11 smithi083 bash[18629]: cluster 2024-09-10T19:02:10.168383+0000 mgr.y (mgr.24526) 518 : cluster [DBG] pgmap v784: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-10T19:02:11.465 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:11 smithi083 bash[32381]: cluster 2024-09-10T19:02:10.168383+0000 mgr.y (mgr.24526) 518 : cluster [DBG] pgmap v784: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-10T19:02:11.487 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:11.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:11.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:02:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:11 smithi165 bash[22591]: cluster 2024-09-10T19:02:10.168383+0000 mgr.y (mgr.24526) 518 : cluster [DBG] pgmap v784: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-10T19:02:11.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:11.619 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:11.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:11.754 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:11.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:11.892 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:11.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:11.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:12.030 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:12.134 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:12.168 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:12.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:12.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.268+0000 7fd8897fa640 1 -- 172.21.15.83:0/750618570 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v514) v1 ==== 97+0+0 (unknown 3240407928 0 0) 0x7fd884053d70 con 0x7fd88c10b130 2024-09-10T19:02:12.276 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 10 librados: watch_flush enter 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 10 librados: watch_flush exit 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 20 client.25434.objecter shutdown clearing up homeless session... 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 10 client.25434.objecter successfully canceled tick 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd874071740 legacy=0x7fd874073bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 >> v1:172.21.15.83:6789/0 conn(0x7fd88c10b130 legacy=0x7fd88c11c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 shutdown_connections 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 >> 172.21.15.83:0/750618570 conn(0x7fd88c0fa3d0 msgr2=0x7fd88c10b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 shutdown_connections 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 -- 172.21.15.83:0/750618570 wait complete. 2024-09-10T19:02:12.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:12.276+0000 7fd892de4640 1 librados: shutdown 2024-09-10T19:02:12.295 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:12.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:12.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:12.412 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:12 smithi083 bash[18629]: audit 2024-09-10T19:02:11.326407+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:12.412 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:12 smithi083 bash[32381]: audit 2024-09-10T19:02:11.326407+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:12.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:12.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:12.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:12 smithi165 bash[22591]: audit 2024-09-10T19:02:11.326407+0000 mon.a (mon.0) 2769 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:12.570 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:12.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:12.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:12.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:12] "GET /metrics HTTP/1.1" 200 228885 "" "Prometheus/2.43.0" 2024-09-10T19:02:12.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:12.842 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:12.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:12.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:12.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:12.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:13.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:13.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:13.221 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:13.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:13.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:13 smithi165 bash[22591]: cluster 2024-09-10T19:02:12.171327+0000 mgr.y (mgr.24526) 519 : cluster [DBG] pgmap v785: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:02:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:13 smithi165 bash[22591]: cluster 2024-09-10T19:02:12.259929+0000 mon.a (mon.0) 2770 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:13 smithi165 bash[22591]: audit 2024-09-10T19:02:12.272498+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:13 smithi165 bash[22591]: cluster 2024-09-10T19:02:12.272583+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:02:13.395 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:13.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:13.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:13.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:13.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[32381]: cluster 2024-09-10T19:02:12.171327+0000 mgr.y (mgr.24526) 519 : cluster [DBG] pgmap v785: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:02:13.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[32381]: cluster 2024-09-10T19:02:12.259929+0000 mon.a (mon.0) 2770 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[32381]: audit 2024-09-10T19:02:12.272498+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[32381]: cluster 2024-09-10T19:02:12.272583+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[18629]: cluster 2024-09-10T19:02:12.171327+0000 mgr.y (mgr.24526) 519 : cluster [DBG] pgmap v785: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[18629]: cluster 2024-09-10T19:02:12.259929+0000 mon.a (mon.0) 2770 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[18629]: audit 2024-09-10T19:02:12.272498+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? v1:172.21.15.83:0/750618570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:13 smithi083 bash[18629]: cluster 2024-09-10T19:02:12.272583+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:02:13.570 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:13.589 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:13.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:13.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:13.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:13.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:13.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:13.939 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:13.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:14.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:14.079 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:14.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:14.215 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:14.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:14.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.356 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:14.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:14.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:14.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:14.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:14.631 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:14.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:14.769 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:14.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:14.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:14.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:15.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.038 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:15.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:15.169 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:02:15.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:15.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.289 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:15 smithi165 bash[22591]: cluster 2024-09-10T19:02:14.172063+0000 mgr.y (mgr.24526) 520 : cluster [DBG] pgmap v787: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:15.290 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:15 smithi165 bash[22591]: audit 2024-09-10T19:02:14.567396+0000 mon.c (mon.2) 654 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:15.309 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:15.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:15.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:15.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:15 smithi083 bash[18629]: cluster 2024-09-10T19:02:14.172063+0000 mgr.y (mgr.24526) 520 : cluster [DBG] pgmap v787: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:15.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:15 smithi083 bash[18629]: audit 2024-09-10T19:02:14.567396+0000 mon.c (mon.2) 654 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:15 smithi083 bash[32381]: cluster 2024-09-10T19:02:14.172063+0000 mgr.y (mgr.24526) 520 : cluster [DBG] pgmap v787: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:15 smithi083 bash[32381]: audit 2024-09-10T19:02:14.567396+0000 mon.c (mon.2) 654 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:15.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:15.584 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:15.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.720 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:15.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:15.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:15.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:15.985 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:15.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:16.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:16.127 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:16.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.264 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:16.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:16.399 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:16.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:16.532 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:16.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:16.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:16.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:16.690 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:16.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.802 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:16.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.933 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:16.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:17.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.071 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:17.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:17.186 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:02:17.187 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:02:17.187 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:02:17.187 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:02:17.200 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:17.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:17.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:17 smithi083 bash[18629]: cluster 2024-09-10T19:02:16.175013+0000 mgr.y (mgr.24526) 521 : cluster [DBG] pgmap v788: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:17.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:17 smithi083 bash[32381]: cluster 2024-09-10T19:02:16.175013+0000 mgr.y (mgr.24526) 521 : cluster [DBG] pgmap v788: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:17.334 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:17.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:17.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:17.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:17 smithi165 bash[22591]: cluster 2024-09-10T19:02:16.175013+0000 mgr.y (mgr.24526) 521 : cluster [DBG] pgmap v788: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:17.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:17.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:17.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:17.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:02:17.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:17.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:17.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:18.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:18.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:18.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:18.293 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:18.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:18.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[22591]: audit 2024-09-10T19:02:18.256568+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[22591]: audit 2024-09-10T19:02:18.258833+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[22591]: audit 2024-09-10T19:02:18.259697+0000 mon.a (mon.0) 2773 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[22591]: audit 2024-09-10T19:02:18.261746+0000 mon.a (mon.0) 2774 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.363 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[48829]: debug 2024-09-10T19:02:18.247+0000 7f107d44d640 -1 snap_mapper.add_oid found existing snaps mapped on 223:ae1c1308:test-rados-api-smithi083-77828-80::foo:2, removing 2024-09-10T19:02:18.363 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:02:18 smithi165 bash[41096]: debug 2024-09-10T19:02:18.247+0000 7fa5a6a74640 -1 snap_mapper.add_oid found existing snaps mapped on 223:ae1c1308:test-rados-api-smithi083-77828-80::foo:2, removing 2024-09-10T19:02:18.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:18.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:18.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[18629]: audit 2024-09-10T19:02:18.256568+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[18629]: audit 2024-09-10T19:02:18.258833+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[18629]: audit 2024-09-10T19:02:18.259697+0000 mon.a (mon.0) 2773 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.550 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[18629]: audit 2024-09-10T19:02:18.261746+0000 mon.a (mon.0) 2774 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[32381]: audit 2024-09-10T19:02:18.256568+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[32381]: audit 2024-09-10T19:02:18.258833+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[32381]: audit 2024-09-10T19:02:18.259697+0000 mon.a (mon.0) 2773 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:18.551 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[32381]: audit 2024-09-10T19:02:18.261746+0000 mon.a (mon.0) 2774 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-79"}]: dispatch 2024-09-10T19:02:18.551 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:02:18 smithi083 bash[42805]: debug 2024-09-10T19:02:18.252+0000 7f7da0594640 -1 snap_mapper.add_oid found existing snaps mapped on 223:ae1c1308:test-rados-api-smithi083-77828-80::foo:2, removing 2024-09-10T19:02:18.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- 172.21.15.83:0/2985270775 >> v1:172.21.15.83:6789/0 conn(0x7fe1bc107de0 legacy=0x7fe1bc10a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:18.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- 172.21.15.83:0/2985270775 shutdown_connections 2024-09-10T19:02:18.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- 172.21.15.83:0/2985270775 >> 172.21.15.83:0/2985270775 conn(0x7fe1bc0f7070 msgr2=0x7fe1bc0f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:18.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- 172.21.15.83:0/2985270775 shutdown_connections 2024-09-10T19:02:18.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- 172.21.15.83:0/2985270775 wait complete. 2024-09-10T19:02:18.567 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:18.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 librados: starting msgr at 2024-09-10T19:02:18.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 librados: starting objecter 2024-09-10T19:02:18.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 Processor -- start 2024-09-10T19:02:18.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- start start 2024-09-10T19:02:18.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 librados: setting wanted keys 2024-09-10T19:02:18.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 librados: calling monclient init 2024-09-10T19:02:18.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1bc11bd80 con 0x7fe1bc101510 2024-09-10T19:02:18.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1bc11cf80 con 0x7fe1bc104280 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c40e6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1bc11e180 con 0x7fe1bc107de0 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c1e82640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1bc101510 0x7fe1bc065c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47210/0 (socket says 172.21.15.83:47210) 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1c1e82640 1 -- 172.21.15.83:0/1585807089 learned_addr learned my addr 172.21.15.83:0/1585807089 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1bc101510 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1bc107de0 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter ms_handle_connect 0x7fe1bc104280 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915620658 0 0) 0x7fe1bc11bd80 con 0x7fe1bc101510 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a4003120 con 0x7fe1bc101510 2024-09-10T19:02:18.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710907191 0 0) 0x7fe1bc11e180 con 0x7fe1bc107de0 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc11bd80 con 0x7fe1bc107de0 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943903263 0 0) 0x7fe1bc11cf80 con 0x7fe1bc104280 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc11e180 con 0x7fe1bc104280 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052721344 0 0) 0x7fe1a4003120 con 0x7fe1bc101510 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1bc11cf80 con 0x7fe1bc101510 2024-09-10T19:02:18.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe1ac001a80 con 0x7fe1bc101510 2024-09-10T19:02:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:18.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621389418 0 0) 0x7fe1bc11bd80 con 0x7fe1bc107de0 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1a4003120 con 0x7fe1bc107de0 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1755496958 0 0) 0x7fe1bc11e180 con 0x7fe1bc104280 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1bc11bd80 con 0x7fe1bc104280 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe1b0002ce0 con 0x7fe1bc107de0 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2874475420 0 0) 0x7fe1bc11cf80 con 0x7fe1bc101510 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 >> v1:172.21.15.83:6790/0 conn(0x7fe1bc107de0 legacy=0x7fe1bc11a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 >> v1:172.21.15.165:6789/0 conn(0x7fe1bc104280 legacy=0x7fe1bc066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.564+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc11f380 con 0x7fe1bc101510 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc11e370 con 0x7fe1bc101510 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1c40e6640 10 client.25440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe1ac004cd0 con 0x7fe1bc101510 2024-09-10T19:02:18.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc11e840 con 0x7fe1bc101510 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1c40e6640 1 librados: init done 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe1ac0059d0 con 0x7fe1bc101510 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc0fcc10 con 0x7fe1bc101510 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe1ac001f50 con 0x7fe1bc101510 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(514..514 src has 69..514) v4 ==== 8004+0+0 (unknown 2296818832 0 0) 0x7fe1ac08a670 con 0x7fe1bc101510 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 10 client.25440.objecter ms_dispatch 0x7fe1bc000b90 osd_map(514..514 src has 69..514) v4 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 3 client.25440.objecter handle_osd_map got epochs [514,514] > 0 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 3 client.25440.objecter handle_osd_map decoding full epoch 514 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 20 client.25440.objecter dump_active .. 0 homeless 2024-09-10T19:02:18.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.568+0000 7fe1ba7fc640 10 client.25440.objecter ms_handle_connect 0x7fe1a4071810 2024-09-10T19:02:18.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.576+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe1ac056c90 con 0x7fe1bc101510 2024-09-10T19:02:18.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:18.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:18.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.744+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1bc10d860 con 0x7fe1a4071810 2024-09-10T19:02:18.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.764+0000 7fe1ba7fc640 1 -- 172.21.15.83:0/1585807089 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+651293 (unknown 2965378022 0 3006525839) 0x7fe1bc10d860 con 0x7fe1a4071810 2024-09-10T19:02:18.781 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:02:18.787 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:02:18.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 10 librados: watch_flush enter 2024-09-10T19:02:18.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 10 librados: watch_flush exit 2024-09-10T19:02:18.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 20 client.25440.objecter shutdown clearing up homeless session... 2024-09-10T19:02:18.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 10 client.25440.objecter successfully canceled tick 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe1a4071810 legacy=0x7fe1a4073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 >> v1:172.21.15.83:6789/0 conn(0x7fe1bc101510 legacy=0x7fe1bc065c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 shutdown_connections 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 >> 172.21.15.83:0/1585807089 conn(0x7fe1bc0f7070 msgr2=0x7fe1bc064b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 shutdown_connections 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 -- 172.21.15.83:0/1585807089 wait complete. 2024-09-10T19:02:18.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:18.772+0000 7fe1c40e6640 1 librados: shutdown 2024-09-10T19:02:18.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:18.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:18.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:19.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:19.127 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:19.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:19.257 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:19.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:19.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:19 smithi083 bash[18629]: audit 2024-09-10T19:02:18.093003+0000 mgr.y (mgr.24526) 522 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:19.372 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:19 smithi083 bash[18629]: cluster 2024-09-10T19:02:18.175845+0000 mgr.y (mgr.24526) 523 : cluster [DBG] pgmap v789: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:02:19.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:19 smithi083 bash[32381]: audit 2024-09-10T19:02:18.093003+0000 mgr.y (mgr.24526) 522 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:19.372 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:19 smithi083 bash[32381]: cluster 2024-09-10T19:02:18.175845+0000 mgr.y (mgr.24526) 523 : cluster [DBG] pgmap v789: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:02:19.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:19.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:19.527 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:19 smithi165 bash[22591]: audit 2024-09-10T19:02:18.093003+0000 mgr.y (mgr.24526) 522 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:19 smithi165 bash[22591]: cluster 2024-09-10T19:02:18.175845+0000 mgr.y (mgr.24526) 523 : cluster [DBG] pgmap v789: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:02:19.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:19.663 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:19.766 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:19.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:19.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:19.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:19.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:20.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:20.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:20.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:20.209 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:20.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:20.343 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:20.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:20.478 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:20.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:20.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:20 smithi083 bash[18629]: audit 2024-09-10T19:02:18.750445+0000 mgr.y (mgr.24526) 524 : audit [DBG] from='client.25440 v1:172.21.15.83:0/1585807089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:02:20.596 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:20 smithi083 bash[18629]: cluster 2024-09-10T19:02:19.290436+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:02:20.596 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:20 smithi083 bash[32381]: audit 2024-09-10T19:02:18.750445+0000 mgr.y (mgr.24526) 524 : audit [DBG] from='client.25440 v1:172.21.15.83:0/1585807089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:02:20.597 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:20 smithi083 bash[32381]: cluster 2024-09-10T19:02:19.290436+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:02:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:20 smithi165 bash[22591]: audit 2024-09-10T19:02:18.750445+0000 mgr.y (mgr.24526) 524 : audit [DBG] from='client.25440 v1:172.21.15.83:0/1585807089' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:02:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:20 smithi165 bash[22591]: cluster 2024-09-10T19:02:19.290436+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:02:20.615 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:20.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:20.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:20.806 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-10T19:02:18.175344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792160,"num_objects":376,"num_object_clones":12,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":376,"num_whiteouts":0,"num_read":2168,"num_read_kb":1921,"num_write":1480,"num_write_kb":14405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3750,"ondisk_log_size":3750,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2532900,"kb_used_data":87460,"kb_used_omap":13,"kb_used_meta":2445298,"kb_avail":747428316,"statfs":{"total":767960285184,"available":765366595584,"internally_reserved":0,"allocated":89559040,"data_stored":62524530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2503985856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.939788"},"pg_stats":[{"pgid":"223.1f","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547930+0000","last_change":"2024-09-10T19:01:56.216079+0000","last_active":"2024-09-10T19:02:09.547930+0000","last_peered":"2024-09-10T19:02:09.547930+0000","last_clean":"2024-09-10T19:02:09.547930+0000","last_became_active":"2024-09-10T19:01:56.215805+0000","last_became_peered":"2024-09-10T19:01:56.215805+0000","last_unstale":"2024-09-10T19:02:09.547930+0000","last_undegraded":"2024-09-10T19:02:09.547930+0000","last_fullsized":"2024-09-10T19:02:09.547930+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:02.559002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.1e","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544424+0000","last_change":"2024-09-10T19:01:56.214360+0000","last_active":"2024-09-10T19:02:09.544424+0000","last_peered":"2024-09-10T19:02:09.544424+0000","last_clean":"2024-09-10T19:02:09.544424+0000","last_became_active":"2024-09-10T19:01:56.213999+0000","last_became_peered":"2024-09-10T19:01:56.213999+0000","last_unstale":"2024-09-10T19:02:09.544424+0000","last_undegraded":"2024-09-10T19:02:09.544424+0000","last_fullsized":"2024-09-10T19:02:09.544424+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:12.083962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"223.1d","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242427+0000","last_change":"2024-09-10T19:01:56.202727+0000","last_active":"2024-09-10T19:02:09.242427+0000","last_peered":"2024-09-10T19:02:09.242427+0000","last_clean":"2024-09-10T19:02:09.242427+0000","last_became_active":"2024-09-10T19:01:56.202384+0000","last_became_peered":"2024-09-10T19:01:56.202384+0000","last_unstale":"2024-09-10T19:02:09.242427+0000","last_undegraded":"2024-09-10T19:02:09.242427+0000","last_fullsized":"2024-09-10T19:02:09.242427+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:16.834946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"223.1c","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545022+0000","last_change":"2024-09-10T19:01:56.211709+0000","last_active":"2024-09-10T19:02:09.545022+0000","last_peered":"2024-09-10T19:02:09.545022+0000","last_clean":"2024-09-10T19:02:09.545022+0000","last_became_active":"2024-09-10T19:01:56.211443+0000","last_became_peered":"2024-09-10T19:01:56.211443+0000","last_unstale":"2024-09-10T19:02:09.545022+0000","last_undegraded":"2024-09-10T19:02:09.545022+0000","last_fullsized":"2024-09-10T19:02:09.545022+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:16.824870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.1b","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545975+0000","last_change":"2024-09-10T19:01:56.216243+0000","last_active":"2024-09-10T19:02:09.545975+0000","last_peered":"2024-09-10T19:02:09.545975+0000","last_clean":"2024-09-10T19:02:09.545975+0000","last_became_active":"2024-09-10T19:01:56.215898+0000","last_became_peered":"2024-09-10T19:01:56.215898+0000","last_unstale":"2024-09-10T19:02:09.545975+0000","last_undegraded":"2024-09-10T19:02:09.545975+0000","last_fullsized":"2024-09-10T19:02:09.545975+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:04.744926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"223.1a","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544853+0000","last_change":"2024-09-10T19:01:56.208047+0000","last_active":"2024-09-10T19:02:09.544853+0000","last_peered":"2024-09-10T19:02:09.544853+0000","last_clean":"2024-09-10T19:02:09.544853+0000","last_became_active":"2024-09-10T19:01:56.207754+0000","last_became_peered":"2024-09-10T19:01:56.207754+0000","last_unstale":"2024-09-10T19:02:09.544853+0000","last_undegraded":"2024-09-10T19:02:09.544853+0000","last_fullsized":"2024-09-10T19:02:09.544853+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:20.824141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"223.19","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544771+0000","last_change":"2024-09-10T19:01:56.218231+0000","last_active":"2024-09-10T19:02:09.544771+0000","last_peered":"2024-09-10T19:02:09.544771+0000","last_clean":"2024-09-10T19:02:09.544771+0000","last_became_active":"2024-09-10T19:01:56.216139+0000","last_became_peered":"2024-09-10T19:01:56.216139+0000","last_unstale":"2024-09-10T19:02:09.544771+0000","last_undegraded":"2024-09-10T19:02:09.544771+0000","last_fullsized":"2024-09-10T19:02:09.544771+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:10.999915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.18","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.232583+0000","last_change":"2024-09-10T19:01:56.200588+0000","last_active":"2024-09-10T19:02:09.232583+0000","last_peered":"2024-09-10T19:02:09.232583+0000","last_clean":"2024-09-10T19:02:09.232583+0000","last_became_active":"2024-09-10T19:01:56.200219+0000","last_became_peered":"2024-09-10T19:01:56.200219+0000","last_unstale":"2024-09-10T19:02:09.232583+0000","last_undegraded":"2024-09-10T19:02:09.232583+0000","last_fullsized":"2024-09-10T19:02:09.232583+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:21.719049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"223.17","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243429+0000","last_change":"2024-09-10T19:01:56.214619+0000","last_active":"2024-09-10T19:02:09.243429+0000","last_peered":"2024-09-10T19:02:09.243429+0000","last_clean":"2024-09-10T19:02:09.243429+0000","last_became_active":"2024-09-10T19:01:56.214319+0000","last_became_peered":"2024-09-10T19:01:56.214319+0000","last_unstale":"2024-09-10T19:02:09.243429+0000","last_undegraded":"2024-09-10T19:02:09.243429+0000","last_fullsized":"2024-09-10T19:02:09.243429+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:10.286979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"223.16","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547650+0000","last_change":"2024-09-10T19:01:56.218361+0000","last_active":"2024-09-10T19:02:09.547650+0000","last_peered":"2024-09-10T19:02:09.547650+0000","last_clean":"2024-09-10T19:02:09.547650+0000","last_became_active":"2024-09-10T19:01:56.218102+0000","last_became_peered":"2024-09-10T19:01:56.218102+0000","last_unstale":"2024-09-10T19:02:09.547650+0000","last_undegraded":"2024-09-10T19:02:09.547650+0000","last_fullsized":"2024-09-10T19:02:09.547650+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:09.532468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.15","version":"506'8","reported_seq":36,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547920+0000","last_change":"2024-09-10T19:01:58.919978+0000","last_active":"2024-09-10T19:02:09.547920+0000","last_peered":"2024-09-10T19:02:09.547920+0000","last_clean":"2024-09-10T19:02:09.547920+0000","last_became_active":"2024-09-10T19:01:56.216564+0000","last_became_peered":"2024-09-10T19:01:56.216564+0000","last_unstale":"2024-09-10T19:02:09.547920+0000","last_undegraded":"2024-09-10T19:02:09.547920+0000","last_fullsized":"2024-09-10T19:02:09.547920+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'8","last_scrub_stamp":"2024-09-10T19:01:58.919819+0000","last_deep_scrub":"506'8","last_deep_scrub_stamp":"2024-09-10T19:01:58.919819+0000","last_clean_scrub_stamp":"2024-09-10T19:01:58.919819+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:06.512829+0000","scrub_duration":0.0054066230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.14","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234272+0000","last_change":"2024-09-10T19:01:56.211261+0000","last_active":"2024-09-10T19:02:09.234272+0000","last_peered":"2024-09-10T19:02:09.234272+0000","last_clean":"2024-09-10T19:02:09.234272+0000","last_became_active":"2024-09-10T19:01:56.210553+0000","last_became_peered":"2024-09-10T19:01:56.210553+0000","last_unstale":"2024-09-10T19:02:09.234272+0000","last_undegraded":"2024-09-10T19:02:09.234272+0000","last_fullsized":"2024-09-10T19:02:09.234272+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:16.035545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"223.13","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545903+0000","last_change":"2024-09-10T19:01:56.216145+0000","last_active":"2024-09-10T19:02:09.545903+0000","last_peered":"2024-09-10T19:02:09.545903+0000","last_clean":"2024-09-10T19:02:09.545903+0000","last_became_active":"2024-09-10T19:01:56.215887+0000","last_became_peered":"2024-09-10T19:01:56.215887+0000","last_unstale":"2024-09-10T19:02:09.545903+0000","last_undegraded":"2024-09-10T19:02:09.545903+0000","last_fullsized":"2024-09-10T19:02:09.545903+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:57.292711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.12","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544432+0000","last_change":"2024-09-10T19:01:56.211430+0000","last_active":"2024-09-10T19:02:09.544432+0000","last_peered":"2024-09-10T19:02:09.544432+0000","last_clean":"2024-09-10T19:02:09.544432+0000","last_became_active":"2024-09-10T19:01:56.211146+0000","last_became_peered":"2024-09-10T19:01:56.211146+0000","last_unstale":"2024-09-10T19:02:09.544432+0000","last_undegraded":"2024-09-10T19:02:09.544432+0000","last_fullsized":"2024-09-10T19:02:09.544432+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:57.062386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.11","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.232984+0000","last_change":"2024-09-10T19:01:56.209850+0000","last_active":"2024-09-10T19:02:09.232984+0000","last_peered":"2024-09-10T19:02:09.232984+0000","last_clean":"2024-09-10T19:02:09.232984+0000","last_became_active":"2024-09-10T19:01:56.209527+0000","last_became_peered":"2024-09-10T19:01:56.209527+0000","last_unstale":"2024-09-10T19:02:09.232984+0000","last_undegraded":"2024-09-10T19:02:09.232984+0000","last_fullsized":"2024-09-10T19:02:09.232984+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:19.198162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"223.10","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542488+0000","last_change":"2024-09-10T19:01:56.215942+0000","last_active":"2024-09-10T19:02:09.542488+0000","last_peered":"2024-09-10T19:02:09.542488+0000","last_clean":"2024-09-10T19:02:09.542488+0000","last_became_active":"2024-09-10T19:01:56.215337+0000","last_became_peered":"2024-09-10T19:01:56.215337+0000","last_unstale":"2024-09-10T19:02:09.542488+0000","last_undegraded":"2024-09-10T19:02:09.542488+0000","last_fullsized":"2024-09-10T19:02:09.542488+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:11.692995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.f","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548045+0000","last_change":"2024-09-10T19:01:56.218342+0000","last_active":"2024-09-10T19:02:09.548045+0000","last_peered":"2024-09-10T19:02:09.548045+0000","last_clean":"2024-09-10T19:02:09.548045+0000","last_became_active":"2024-09-10T19:01:56.218065+0000","last_became_peered":"2024-09-10T19:01:56.218065+0000","last_unstale":"2024-09-10T19:02:09.548045+0000","last_undegraded":"2024-09-10T19:02:09.548045+0000","last_fullsized":"2024-09-10T19:02:09.548045+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:17.642435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.e","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547525+0000","last_change":"2024-09-10T19:01:56.204932+0000","last_active":"2024-09-10T19:02:09.547525+0000","last_peered":"2024-09-10T19:02:09.547525+0000","last_clean":"2024-09-10T19:02:09.547525+0000","last_became_active":"2024-09-10T19:01:56.204648+0000","last_became_peered":"2024-09-10T19:01:56.204648+0000","last_unstale":"2024-09-10T19:02:09.547525+0000","last_undegraded":"2024-09-10T19:02:09.547525+0000","last_fullsized":"2024-09-10T19:02:09.547525+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:22.229531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"223.d","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235333+0000","last_change":"2024-09-10T19:01:56.212095+0000","last_active":"2024-09-10T19:02:09.235333+0000","last_peered":"2024-09-10T19:02:09.235333+0000","last_clean":"2024-09-10T19:02:09.235333+0000","last_became_active":"2024-09-10T19:01:56.211827+0000","last_became_peered":"2024-09-10T19:01:56.211827+0000","last_unstale":"2024-09-10T19:02:09.235333+0000","last_undegraded":"2024-09-10T19:02:09.235333+0000","last_fullsized":"2024-09-10T19:02:09.235333+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:19.762576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"223.c","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543205+0000","last_change":"2024-09-10T19:01:56.211719+0000","last_active":"2024-09-10T19:02:09.543205+0000","last_peered":"2024-09-10T19:02:09.543205+0000","last_clean":"2024-09-10T19:02:09.543205+0000","last_became_active":"2024-09-10T19:01:56.211407+0000","last_became_peered":"2024-09-10T19:01:56.211407+0000","last_unstale":"2024-09-10T19:02:09.543205+0000","last_undegraded":"2024-09-10T19:02:09.543205+0000","last_fullsized":"2024-09-10T19:02:09.543205+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:05.591337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.b","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548907+0000","last_change":"2024-09-10T19:01:56.205394+0000","last_active":"2024-09-10T19:02:09.548907+0000","last_peered":"2024-09-10T19:02:09.548907+0000","last_clean":"2024-09-10T19:02:09.548907+0000","last_became_active":"2024-09-10T19:01:56.205072+0000","last_became_peered":"2024-09-10T19:01:56.205072+0000","last_unstale":"2024-09-10T19:02:09.548907+0000","last_undegraded":"2024-09-10T19:02:09.548907+0000","last_fullsized":"2024-09-10T19:02:09.548907+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:12.375789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"223.a","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241245+0000","last_change":"2024-09-10T19:01:56.208871+0000","last_active":"2024-09-10T19:02:09.241245+0000","last_peered":"2024-09-10T19:02:09.241245+0000","last_clean":"2024-09-10T19:02:09.241245+0000","last_became_active":"2024-09-10T19:01:56.208579+0000","last_became_peered":"2024-09-10T19:01:56.208579+0000","last_unstale":"2024-09-10T19:02:09.241245+0000","last_undegraded":"2024-09-10T19:02:09.241245+0000","last_fullsized":"2024-09-10T19:02:09.241245+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:17.911731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"223.9","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542694+0000","last_change":"2024-09-10T19:01:56.215983+0000","last_active":"2024-09-10T19:02:09.542694+0000","last_peered":"2024-09-10T19:02:09.542694+0000","last_clean":"2024-09-10T19:02:09.542694+0000","last_became_active":"2024-09-10T19:01:56.215655+0000","last_became_peered":"2024-09-10T19:01:56.215655+0000","last_unstale":"2024-09-10T19:02:09.542694+0000","last_undegraded":"2024-09-10T19:02:09.542694+0000","last_fullsized":"2024-09-10T19:02:09.542694+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:21.958488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"223.8","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545851+0000","last_change":"2024-09-10T19:01:56.212663+0000","last_active":"2024-09-10T19:02:09.545851+0000","last_peered":"2024-09-10T19:02:09.545851+0000","last_clean":"2024-09-10T19:02:09.545851+0000","last_became_active":"2024-09-10T19:01:56.212239+0000","last_became_peered":"2024-09-10T19:01:56.212239+0000","last_unstale":"2024-09-10T19:02:09.545851+0000","last_undegraded":"2024-09-10T19:02:09.545851+0000","last_fullsized":"2024-09-10T19:02:09.545851+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:22.494732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.7","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235444+0000","last_change":"2024-09-10T19:01:56.206434+0000","last_active":"2024-09-10T19:02:09.235444+0000","last_peered":"2024-09-10T19:02:09.235444+0000","last_clean":"2024-09-10T19:02:09.235444+0000","last_became_active":"2024-09-10T19:01:56.206187+0000","last_became_peered":"2024-09-10T19:01:56.206187+0000","last_unstale":"2024-09-10T19:02:09.235444+0000","last_undegraded":"2024-09-10T19:02:09.235444+0000","last_fullsized":"2024-09-10T19:02:09.235444+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:02.348224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"223.6","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549621+0000","last_change":"2024-09-10T19:01:56.209347+0000","last_active":"2024-09-10T19:02:09.549621+0000","last_peered":"2024-09-10T19:02:09.549621+0000","last_clean":"2024-09-10T19:02:09.549621+0000","last_became_active":"2024-09-10T19:01:56.208836+0000","last_became_peered":"2024-09-10T19:01:56.208836+0000","last_unstale":"2024-09-10T19:02:09.549621+0000","last_undegraded":"2024-09-10T19:02:09.549621+0000","last_fullsized":"2024-09-10T19:02:09.549621+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.600405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"223.5","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241212+0000","last_change":"2024-09-10T19:01:56.213223+0000","last_active":"2024-09-10T19:02:09.241212+0000","last_peered":"2024-09-10T19:02:09.241212+0000","last_clean":"2024-09-10T19:02:09.241212+0000","last_became_active":"2024-09-10T19:01:56.212979+0000","last_became_peered":"2024-09-10T19:01:56.212979+0000","last_unstale":"2024-09-10T19:02:09.241212+0000","last_undegraded":"2024-09-10T19:02:09.241212+0000","last_fullsized":"2024-09-10T19:02:09.241212+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:18.673269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"223.4","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549168+0000","last_change":"2024-09-10T19:01:56.207329+0000","last_active":"2024-09-10T19:02:09.549168+0000","last_peered":"2024-09-10T19:02:09.549168+0000","last_clean":"2024-09-10T19:02:09.549168+0000","last_became_active":"2024-09-10T19:01:56.207032+0000","last_became_peered":"2024-09-10T19:01:56.207032+0000","last_unstale":"2024-09-10T19:02:09.549168+0000","last_undegraded":"2024-09-10T19:02:09.549168+0000","last_fullsized":"2024-09-10T19:02:09.549168+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:23.044129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"223.3","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239435+0000","last_change":"2024-09-10T19:01:56.217180+0000","last_active":"2024-09-10T19:02:09.239435+0000","last_peered":"2024-09-10T19:02:09.239435+0000","last_clean":"2024-09-10T19:02:09.239435+0000","last_became_active":"2024-09-10T19:01:56.216853+0000","last_became_peered":"2024-09-10T19:01:56.216853+0000","last_unstale":"2024-09-10T19:02:09.239435+0000","last_undegraded":"2024-09-10T19:02:09.239435+0000","last_fullsized":"2024-09-10T19:02:09.239435+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:02.323287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"223.2","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544518+0000","last_change":"2024-09-10T19:01:56.219128+0000","last_active":"2024-09-10T19:02:09.544518+0000","last_peered":"2024-09-10T19:02:09.544518+0000","last_clean":"2024-09-10T19:02:09.544518+0000","last_became_active":"2024-09-10T19:01:56.218873+0000","last_became_peered":"2024-09-10T19:01:56.218873+0000","last_unstale":"2024-09-10T19:02:09.544518+0000","last_undegraded":"2024-09-10T19:02:09.544518+0000","last_fullsized":"2024-09-10T19:02:09.544518+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.237592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"223.1","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545461+0000","last_change":"2024-09-10T19:01:56.212038+0000","last_active":"2024-09-10T19:02:09.545461+0000","last_peered":"2024-09-10T19:02:09.545461+0000","last_clean":"2024-09-10T19:02:09.545461+0000","last_became_active":"2024-09-10T19:01:56.211638+0000","last_became_peered":"2024-09-10T19:01:56.211638+0000","last_unstale":"2024-09-10T19:02:09.545461+0000","last_undegraded":"2024-09-10T19:02:09.545461+0000","last_fullsized":"2024-09-10T19:02:09.545461+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:06.018271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"223.0","version":"0'0","reported_seq":19,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.540647+0000","last_change":"2024-09-10T19:01:56.212592+0000","last_active":"2024-09-10T19:02:09.540647+0000","last_peered":"2024-09-10T19:02:09.540647+0000","last_clean":"2024-09-10T19:02:09.540647+0000","last_became_active":"2024-09-10T19:01:56.212265+0000","last_became_peered":"2024-09-10T19:01:56.212265+0000","last_unstale":"2024-09-10T19:02:09.540647+0000","last_undegraded":"2024-09-10T19:02:09.540647+0000","last_fullsized":"2024-09-10T19:02:09.540647+0000","mapping_epoch":503,"log_start":"0'0","ondisk_log_start":"0'0","created":503,"last_epoch_clean":504,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.176574+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.176574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:24.919066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.1f","version":"377'41","reported_seq":502,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545949+0000","last_change":"2024-09-10T19:01:49.532718+0000","last_active":"2024-09-10T19:02:09.545949+0000","last_peered":"2024-09-10T19:02:09.545949+0000","last_clean":"2024-09-10T19:02:09.545949+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:02:09.545949+0000","last_undegraded":"2024-09-10T19:02:09.545949+0000","last_fullsized":"2024-09-10T19:02:09.545949+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'41","last_scrub_stamp":"2024-09-10T19:01:49.532552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:49.532552+0000","objects_scrubbed":5,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:18.707758+0000","scrub_duration":0.0048290629999999998,"objects_trimmed":0,"snaptrim_duration":0.00067620899999999999,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"506'74","reported_seq":540,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545728+0000","last_change":"2024-09-10T19:01:25.547641+0000","last_active":"2024-09-10T19:02:09.545728+0000","last_peered":"2024-09-10T19:02:09.545728+0000","last_clean":"2024-09-10T19:02:09.545728+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:02:09.545728+0000","last_undegraded":"2024-09-10T19:02:09.545728+0000","last_fullsized":"2024-09-10T19:02:09.545728+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'72","last_scrub_stamp":"2024-09-10T19:01:25.547476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.547476+0000","objects_scrubbed":10,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.233345+0000","scrub_duration":0.0061775670000000001,"objects_trimmed":0,"snaptrim_duration":0.00056750799999999999,"stat_sum":{"num_bytes":84,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":28,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"425'25","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543777+0000","last_change":"2024-09-10T19:01:36.202430+0000","last_active":"2024-09-10T19:02:09.543777+0000","last_peered":"2024-09-10T19:02:09.543777+0000","last_clean":"2024-09-10T19:02:09.543777+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:02:09.543777+0000","last_undegraded":"2024-09-10T19:02:09.543777+0000","last_fullsized":"2024-09-10T19:02:09.543777+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'25","last_scrub_stamp":"2024-09-10T19:01:36.202245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.202245+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.753358+0000","scrub_duration":0.0044600810000000003,"objects_trimmed":0,"snaptrim_duration":0.002538663,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"244'51","reported_seq":519,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543656+0000","last_change":"2024-09-10T19:01:28.876894+0000","last_active":"2024-09-10T19:02:09.543656+0000","last_peered":"2024-09-10T19:02:09.543656+0000","last_clean":"2024-09-10T19:02:09.543656+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:02:09.543656+0000","last_undegraded":"2024-09-10T19:02:09.543656+0000","last_fullsized":"2024-09-10T19:02:09.543656+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T19:01:28.876707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.876707+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:37.578566+0000","scrub_duration":0.004224579,"objects_trimmed":0,"snaptrim_duration":0.0016033009999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"240'68","reported_seq":534,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543801+0000","last_change":"2024-09-10T19:01:53.809068+0000","last_active":"2024-09-10T19:02:09.543801+0000","last_peered":"2024-09-10T19:02:09.543801+0000","last_clean":"2024-09-10T19:02:09.543801+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:02:09.543801+0000","last_undegraded":"2024-09-10T19:02:09.543801+0000","last_fullsized":"2024-09-10T19:02:09.543801+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T19:01:53.808906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.808906+0000","objects_scrubbed":4,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:15.383247+0000","scrub_duration":0.0045370890000000002,"objects_trimmed":0,"snaptrim_duration":0.0018816099999999999,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"228'34","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542175+0000","last_change":"2024-09-10T19:01:51.122423+0000","last_active":"2024-09-10T19:02:09.542175+0000","last_peered":"2024-09-10T19:02:09.542175+0000","last_clean":"2024-09-10T19:02:09.542175+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:02:09.542175+0000","last_undegraded":"2024-09-10T19:02:09.542175+0000","last_fullsized":"2024-09-10T19:02:09.542175+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T19:01:51.122251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.122251+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.072386+0000","scrub_duration":0.0041375079999999998,"objects_trimmed":0,"snaptrim_duration":0.00071753500000000005,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"454'47","reported_seq":523,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.240935+0000","last_change":"2024-09-10T19:00:48.181330+0000","last_active":"2024-09-10T19:02:09.240935+0000","last_peered":"2024-09-10T19:02:09.240935+0000","last_clean":"2024-09-10T19:02:09.240935+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:02:09.240935+0000","last_undegraded":"2024-09-10T19:02:09.240935+0000","last_fullsized":"2024-09-10T19:02:09.240935+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'45","last_scrub_stamp":"2024-09-10T19:00:30.901206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.901206+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053195880000000001,"objects_trimmed":0,"snaptrim_duration":0.0025425780000000002,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"464'120","reported_seq":567,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.540820+0000","last_change":"2024-09-10T19:00:50.283591+0000","last_active":"2024-09-10T19:02:09.540820+0000","last_peered":"2024-09-10T19:02:09.540820+0000","last_clean":"2024-09-10T19:02:09.540820+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:02:09.540820+0000","last_undegraded":"2024-09-10T19:02:09.540820+0000","last_fullsized":"2024-09-10T19:02:09.540820+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'104","last_scrub_stamp":"2024-09-10T19:00:13.172545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.172545+0000","objects_scrubbed":7,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057922160000000002,"objects_trimmed":0,"snaptrim_duration":1.070613525,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"428'23","reported_seq":490,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547552+0000","last_change":"2024-09-10T19:00:48.179573+0000","last_active":"2024-09-10T19:02:09.547552+0000","last_peered":"2024-09-10T19:02:09.547552+0000","last_clean":"2024-09-10T19:02:09.547552+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:02:09.547552+0000","last_undegraded":"2024-09-10T19:02:09.547552+0000","last_fullsized":"2024-09-10T19:02:09.547552+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'23","last_scrub_stamp":"2024-09-10T19:00:20.541045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:20.541045+0000","objects_scrubbed":4,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049668359999999996,"objects_trimmed":0,"snaptrim_duration":0.0012828710000000001,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"387'24","reported_seq":493,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547678+0000","last_change":"2024-09-10T19:00:48.298750+0000","last_active":"2024-09-10T19:02:09.547678+0000","last_peered":"2024-09-10T19:02:09.547678+0000","last_clean":"2024-09-10T19:02:09.547678+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:02:09.547678+0000","last_undegraded":"2024-09-10T19:02:09.547678+0000","last_fullsized":"2024-09-10T19:02:09.547678+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.724817+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0069835599999999998,"objects_trimmed":0,"snaptrim_duration":0.0016237720000000001,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"428'79","reported_seq":559,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243241+0000","last_change":"2024-09-10T19:01:43.224775+0000","last_active":"2024-09-10T19:02:09.243241+0000","last_peered":"2024-09-10T19:02:09.243241+0000","last_clean":"2024-09-10T19:02:09.243241+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:02:09.243241+0000","last_undegraded":"2024-09-10T19:02:09.243241+0000","last_fullsized":"2024-09-10T19:02:09.243241+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'79","last_scrub_stamp":"2024-09-10T19:01:43.224593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.224593+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.541363+0000","scrub_duration":0.0059262669999999998,"objects_trimmed":0,"snaptrim_duration":0.002291235,"stat_sum":{"num_bytes":70,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"434'100","reported_seq":575,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543897+0000","last_change":"2024-09-10T19:01:48.758486+0000","last_active":"2024-09-10T19:02:09.543897+0000","last_peered":"2024-09-10T19:02:09.543897+0000","last_clean":"2024-09-10T19:02:09.543897+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:02:09.543897+0000","last_undegraded":"2024-09-10T19:02:09.543897+0000","last_fullsized":"2024-09-10T19:02:09.543897+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'100","last_scrub_stamp":"2024-09-10T19:01:48.758338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:48.758338+0000","objects_scrubbed":9,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:55.296057+0000","scrub_duration":0.0080646439999999993,"objects_trimmed":0,"snaptrim_duration":0.002046174,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":33,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"497'21","reported_seq":516,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239544+0000","last_change":"2024-09-10T19:01:51.163739+0000","last_active":"2024-09-10T19:02:09.239544+0000","last_peered":"2024-09-10T19:02:09.239544+0000","last_clean":"2024-09-10T19:02:09.239544+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:02:09.239544+0000","last_undegraded":"2024-09-10T19:02:09.239544+0000","last_fullsized":"2024-09-10T19:02:09.239544+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'21","last_scrub_stamp":"2024-09-10T19:01:51.163574+0000","last_deep_scrub":"448'4","last_deep_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.163574+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:59.682837+0000","scrub_duration":0.0056664030000000004,"objects_trimmed":0,"snaptrim_duration":0.00085743500000000003,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":505,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548791+0000","last_change":"2024-09-10T19:01:16.522626+0000","last_active":"2024-09-10T19:02:09.548791+0000","last_peered":"2024-09-10T19:02:09.548791+0000","last_clean":"2024-09-10T19:02:09.548791+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:02:09.548791+0000","last_undegraded":"2024-09-10T19:02:09.548791+0000","last_fullsized":"2024-09-10T19:02:09.548791+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T19:01:16.522456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.522456+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:26.628007+0000","scrub_duration":0.0048536079999999997,"objects_trimmed":0,"snaptrim_duration":0.0024268039999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"452'56","reported_seq":520,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.232864+0000","last_change":"2024-09-10T19:00:48.285951+0000","last_active":"2024-09-10T19:02:09.232864+0000","last_peered":"2024-09-10T19:02:09.232864+0000","last_clean":"2024-09-10T19:02:09.232864+0000","last_became_active":"2024-09-10T18:54:56.802625+0000","last_became_peered":"2024-09-10T18:54:56.802625+0000","last_unstale":"2024-09-10T19:02:09.232864+0000","last_undegraded":"2024-09-10T19:02:09.232864+0000","last_fullsized":"2024-09-10T19:02:09.232864+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'56","last_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_deep_scrub":"452'56","last_deep_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.174126+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011240307,"objects_trimmed":0,"snaptrim_duration":0.00098096800000000003,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":499,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239011+0000","last_change":"2024-09-10T19:01:33.818709+0000","last_active":"2024-09-10T19:02:09.239011+0000","last_peered":"2024-09-10T19:02:09.239011+0000","last_clean":"2024-09-10T19:02:09.239011+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:02:09.239011+0000","last_undegraded":"2024-09-10T19:02:09.239011+0000","last_fullsized":"2024-09-10T19:02:09.239011+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'35","last_scrub_stamp":"2024-09-10T19:01:33.818552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.818552+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.869584+0000","scrub_duration":0.0033660140000000001,"objects_trimmed":0,"snaptrim_duration":0.0039003229999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"431'70","reported_seq":542,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.550274+0000","last_change":"2024-09-10T19:01:52.606206+0000","last_active":"2024-09-10T19:02:09.550274+0000","last_peered":"2024-09-10T19:02:09.550274+0000","last_clean":"2024-09-10T19:02:09.550274+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:02:09.550274+0000","last_undegraded":"2024-09-10T19:02:09.550274+0000","last_fullsized":"2024-09-10T19:02:09.550274+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'70","last_scrub_stamp":"2024-09-10T19:01:52.606042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.606042+0000","objects_scrubbed":7,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.697367+0000","scrub_duration":0.0053694540000000001,"objects_trimmed":0,"snaptrim_duration":0.00074153100000000003,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"484'136","reported_seq":594,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547632+0000","last_change":"2024-09-10T19:01:47.877452+0000","last_active":"2024-09-10T19:02:09.547632+0000","last_peered":"2024-09-10T19:02:09.547632+0000","last_clean":"2024-09-10T19:02:09.547632+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:02:09.547632+0000","last_undegraded":"2024-09-10T19:02:09.547632+0000","last_fullsized":"2024-09-10T19:02:09.547632+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'136","last_scrub_stamp":"2024-09-10T19:01:47.877294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.877294+0000","objects_scrubbed":9,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:05.824125+0000","scrub_duration":0.0054641070000000002,"objects_trimmed":0,"snaptrim_duration":0.001433337,"stat_sum":{"num_bytes":97,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":43,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"491'34","reported_seq":505,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235559+0000","last_change":"2024-09-10T19:00:48.287101+0000","last_active":"2024-09-10T19:02:09.235559+0000","last_peered":"2024-09-10T19:02:09.235559+0000","last_clean":"2024-09-10T19:02:09.235559+0000","last_became_active":"2024-09-10T18:54:56.782101+0000","last_became_peered":"2024-09-10T18:54:56.782101+0000","last_unstale":"2024-09-10T19:02:09.235559+0000","last_undegraded":"2024-09-10T19:02:09.235559+0000","last_fullsized":"2024-09-10T19:02:09.235559+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'20","last_scrub_stamp":"2024-09-10T18:58:56.450180+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T18:58:56.450180+0000","objects_scrubbed":3,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.00069466700000000005,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"506'25","reported_seq":494,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549318+0000","last_change":"2024-09-10T19:01:25.384905+0000","last_active":"2024-09-10T19:02:09.549318+0000","last_peered":"2024-09-10T19:02:09.549318+0000","last_clean":"2024-09-10T19:02:09.549318+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:02:09.549318+0000","last_undegraded":"2024-09-10T19:02:09.549318+0000","last_fullsized":"2024-09-10T19:02:09.549318+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'23","last_scrub_stamp":"2024-09-10T19:01:25.384788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.384788+0000","objects_scrubbed":3,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:51.492529+0000","scrub_duration":0.0062438620000000002,"objects_trimmed":0,"snaptrim_duration":0.001841777,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"334'36","reported_seq":501,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544003+0000","last_change":"2024-09-10T19:00:48.298164+0000","last_active":"2024-09-10T19:02:09.544003+0000","last_peered":"2024-09-10T19:02:09.544003+0000","last_clean":"2024-09-10T19:02:09.544003+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:02:09.544003+0000","last_undegraded":"2024-09-10T19:02:09.544003+0000","last_fullsized":"2024-09-10T19:02:09.544003+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'36","last_scrub_stamp":"2024-09-10T19:00:28.730904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:28.730904+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045585440000000003,"objects_trimmed":0,"snaptrim_duration":0.001811174,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":3917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"434'20","reported_seq":512,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.238916+0000","last_change":"2024-09-10T19:01:47.148195+0000","last_active":"2024-09-10T19:02:09.238916+0000","last_peered":"2024-09-10T19:02:09.238916+0000","last_clean":"2024-09-10T19:02:09.238916+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:02:09.238916+0000","last_undegraded":"2024-09-10T19:02:09.238916+0000","last_fullsized":"2024-09-10T19:02:09.238916+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'20","last_scrub_stamp":"2024-09-10T19:01:47.148028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.148028+0000","objects_scrubbed":2,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.250984+0000","scrub_duration":0.0041924129999999999,"objects_trimmed":0,"snaptrim_duration":0.00088964800000000004,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":482,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239140+0000","last_change":"2024-09-10T19:00:48.179437+0000","last_active":"2024-09-10T19:02:09.239140+0000","last_peered":"2024-09-10T19:02:09.239140+0000","last_clean":"2024-09-10T19:02:09.239140+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:02:09.239140+0000","last_undegraded":"2024-09-10T19:02:09.239140+0000","last_fullsized":"2024-09-10T19:02:09.239140+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.225132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.225132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035652290000000001,"objects_trimmed":0,"snaptrim_duration":0.00064644099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"501'59","reported_seq":524,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541067+0000","last_change":"2024-09-10T19:00:48.177743+0000","last_active":"2024-09-10T19:02:09.541067+0000","last_peered":"2024-09-10T19:02:09.541067+0000","last_clean":"2024-09-10T19:02:09.541067+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:02:09.541067+0000","last_undegraded":"2024-09-10T19:02:09.541067+0000","last_fullsized":"2024-09-10T19:02:09.541067+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'51","last_scrub_stamp":"2024-09-10T19:00:03.402967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.402967+0000","objects_scrubbed":3,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043031060000000001,"objects_trimmed":0,"snaptrim_duration":0.001434334,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"409'160","reported_seq":641,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243535+0000","last_change":"2024-09-10T19:01:46.188252+0000","last_active":"2024-09-10T19:02:09.243535+0000","last_peered":"2024-09-10T19:02:09.243535+0000","last_clean":"2024-09-10T19:02:09.243535+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:02:09.243535+0000","last_undegraded":"2024-09-10T19:02:09.243535+0000","last_fullsized":"2024-09-10T19:02:09.243535+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'160","last_scrub_stamp":"2024-09-10T19:01:46.188079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:46.188079+0000","objects_scrubbed":12,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.912070+0000","scrub_duration":0.0051260070000000001,"objects_trimmed":0,"snaptrim_duration":0.0011147100000000001,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":13,"num_read_kb":2,"num_write":49,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"388'99","reported_seq":618,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544130+0000","last_change":"2024-09-10T19:01:28.217626+0000","last_active":"2024-09-10T19:02:09.544130+0000","last_peered":"2024-09-10T19:02:09.544130+0000","last_clean":"2024-09-10T19:02:09.544130+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:02:09.544130+0000","last_undegraded":"2024-09-10T19:02:09.544130+0000","last_fullsized":"2024-09-10T19:02:09.544130+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T19:01:28.217457+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.217457+0000","objects_scrubbed":6,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.532699+0000","scrub_duration":0.0059368959999999997,"objects_trimmed":0,"snaptrim_duration":0.001143469,"stat_sum":{"num_bytes":4194359,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":28,"num_write_kb":8218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":515,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549887+0000","last_change":"2024-09-10T19:00:48.275852+0000","last_active":"2024-09-10T19:02:09.549887+0000","last_peered":"2024-09-10T19:02:09.549887+0000","last_clean":"2024-09-10T19:02:09.549887+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:02:09.549887+0000","last_undegraded":"2024-09-10T19:02:09.549887+0000","last_fullsized":"2024-09-10T19:02:09.549887+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T19:00:33.093969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.093969+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045796200000000004,"objects_trimmed":0,"snaptrim_duration":0.0010303630000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"472'113","reported_seq":579,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543637+0000","last_change":"2024-09-10T19:01:53.157091+0000","last_active":"2024-09-10T19:02:09.543637+0000","last_peered":"2024-09-10T19:02:09.543637+0000","last_clean":"2024-09-10T19:02:09.543637+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:02:09.543637+0000","last_undegraded":"2024-09-10T19:02:09.543637+0000","last_fullsized":"2024-09-10T19:02:09.543637+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'113","last_scrub_stamp":"2024-09-10T19:01:53.156959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.156959+0000","objects_scrubbed":9,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.405592+0000","scrub_duration":0.0063436309999999997,"objects_trimmed":0,"snaptrim_duration":0.0025913289999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"431'106","reported_seq":565,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242097+0000","last_change":"2024-09-10T19:01:28.916818+0000","last_active":"2024-09-10T19:02:09.242097+0000","last_peered":"2024-09-10T19:02:09.242097+0000","last_clean":"2024-09-10T19:02:09.242097+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:02:09.242097+0000","last_undegraded":"2024-09-10T19:02:09.242097+0000","last_fullsized":"2024-09-10T19:02:09.242097+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'106","last_scrub_stamp":"2024-09-10T19:01:28.916641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.916641+0000","objects_scrubbed":9,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.061376+0000","scrub_duration":0.0052486110000000002,"objects_trimmed":0,"snaptrim_duration":0.0012640469999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":34,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":496,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546709+0000","last_change":"2024-09-10T19:00:48.180783+0000","last_active":"2024-09-10T19:02:09.546709+0000","last_peered":"2024-09-10T19:02:09.546709+0000","last_clean":"2024-09-10T19:02:09.546709+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:02:09.546709+0000","last_undegraded":"2024-09-10T19:02:09.546709+0000","last_fullsized":"2024-09-10T19:02:09.546709+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T19:00:15.443270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.443270+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044317829999999999,"objects_trimmed":0,"snaptrim_duration":0.00053455899999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"409'74","reported_seq":542,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548534+0000","last_change":"2024-09-10T19:00:48.271955+0000","last_active":"2024-09-10T19:02:09.548534+0000","last_peered":"2024-09-10T19:02:09.548534+0000","last_clean":"2024-09-10T19:02:09.548534+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:02:09.548534+0000","last_undegraded":"2024-09-10T19:02:09.548534+0000","last_fullsized":"2024-09-10T19:02:09.548534+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'74","last_scrub_stamp":"2024-09-10T19:00:31.135496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.135496+0000","objects_scrubbed":9,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081214419999999996,"objects_trimmed":0,"snaptrim_duration":0.001038976,"stat_sum":{"num_bytes":81,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"246'102","reported_seq":574,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.237752+0000","last_change":"2024-09-10T19:01:55.215425+0000","last_active":"2024-09-10T19:02:09.237752+0000","last_peered":"2024-09-10T19:02:09.237752+0000","last_clean":"2024-09-10T19:02:09.237752+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:02:09.237752+0000","last_undegraded":"2024-09-10T19:02:09.237752+0000","last_fullsized":"2024-09-10T19:02:09.237752+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T19:01:55.215341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.215341+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:08.731997+0000","scrub_duration":0.0029714390000000002,"objects_trimmed":0,"snaptrim_duration":0.00087170800000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":487,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241952+0000","last_change":"2024-09-10T18:58:48.167777+0000","last_active":"2024-09-10T19:02:09.241952+0000","last_peered":"2024-09-10T19:02:09.241952+0000","last_clean":"2024-09-10T19:02:09.241952+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:02:09.241952+0000","last_undegraded":"2024-09-10T19:02:09.241952+0000","last_fullsized":"2024-09-10T19:02:09.241952+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:48.167659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T18:58:48.167659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076014869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":500,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546917+0000","last_change":"2024-09-10T18:59:59.106249+0000","last_active":"2024-09-10T19:02:09.546917+0000","last_peered":"2024-09-10T19:02:09.546917+0000","last_clean":"2024-09-10T19:02:09.546917+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:02:09.546917+0000","last_undegraded":"2024-09-10T19:02:09.546917+0000","last_fullsized":"2024-09-10T19:02:09.546917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:59.106063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.106063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034646490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":521,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242776+0000","last_change":"2024-09-10T19:00:10.940056+0000","last_active":"2024-09-10T19:02:09.242776+0000","last_peered":"2024-09-10T19:02:09.242776+0000","last_clean":"2024-09-10T19:02:09.242776+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:02:09.242776+0000","last_undegraded":"2024-09-10T19:02:09.242776+0000","last_fullsized":"2024-09-10T19:02:09.242776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:00:10.939877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.939877+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051523760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241935+0000","last_change":"2024-09-10T19:01:52.890306+0000","last_active":"2024-09-10T19:02:09.241935+0000","last_peered":"2024-09-10T19:02:09.241935+0000","last_clean":"2024-09-10T19:02:09.241935+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:02:09.241935+0000","last_undegraded":"2024-09-10T19:02:09.241935+0000","last_fullsized":"2024-09-10T19:02:09.241935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.890190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:57.382848+0000","scrub_duration":0.0017323970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":423,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547110+0000","last_change":"2024-09-10T19:01:39.509250+0000","last_active":"2024-09-10T19:02:09.547110+0000","last_peered":"2024-09-10T19:02:09.547110+0000","last_clean":"2024-09-10T19:02:09.547110+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:02:09.547110+0000","last_undegraded":"2024-09-10T19:02:09.547110+0000","last_fullsized":"2024-09-10T19:02:09.547110+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.509093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.509093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:04.467858+0000","scrub_duration":0.0035638950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":513,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243227+0000","last_change":"2024-09-10T19:01:29.910705+0000","last_active":"2024-09-10T19:02:09.243227+0000","last_peered":"2024-09-10T19:02:09.243227+0000","last_clean":"2024-09-10T19:02:09.243227+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:02:09.243227+0000","last_undegraded":"2024-09-10T19:02:09.243227+0000","last_fullsized":"2024-09-10T19:02:09.243227+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.910553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.910553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.621492+0000","scrub_duration":0.0032680439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":529,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546379+0000","last_change":"2024-09-10T19:00:03.037143+0000","last_active":"2024-09-10T19:02:09.546379+0000","last_peered":"2024-09-10T19:02:09.546379+0000","last_clean":"2024-09-10T19:02:09.546379+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:02:09.546379+0000","last_undegraded":"2024-09-10T19:02:09.546379+0000","last_fullsized":"2024-09-10T19:02:09.546379+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:00:03.036974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.036974+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043269010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":508,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233323+0000","last_change":"2024-09-10T19:00:01.347416+0000","last_active":"2024-09-10T19:02:09.233323+0000","last_peered":"2024-09-10T19:02:09.233323+0000","last_clean":"2024-09-10T19:02:09.233323+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:02:09.233323+0000","last_undegraded":"2024-09-10T19:02:09.233323+0000","last_fullsized":"2024-09-10T19:02:09.233323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:01.347241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.347241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003764398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233586+0000","last_change":"2024-09-10T19:01:22.090669+0000","last_active":"2024-09-10T19:02:09.233586+0000","last_peered":"2024-09-10T19:02:09.233586+0000","last_clean":"2024-09-10T19:02:09.233586+0000","last_became_active":"2024-09-10T18:49:24.049223+0000","last_became_peered":"2024-09-10T18:49:24.049223+0000","last_unstale":"2024-09-10T19:02:09.233586+0000","last_undegraded":"2024-09-10T19:02:09.233586+0000","last_fullsized":"2024-09-10T19:02:09.233586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.090498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.090498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.520013+0000","scrub_duration":0.0033351800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547293+0000","last_change":"2024-09-10T19:01:15.552219+0000","last_active":"2024-09-10T19:02:09.547293+0000","last_peered":"2024-09-10T19:02:09.547293+0000","last_clean":"2024-09-10T19:02:09.547293+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:02:09.547293+0000","last_undegraded":"2024-09-10T19:02:09.547293+0000","last_fullsized":"2024-09-10T19:02:09.547293+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.552108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.552108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:30.190395+0000","scrub_duration":0.0079911449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543953+0000","last_change":"2024-09-10T19:01:25.221951+0000","last_active":"2024-09-10T19:02:09.543953+0000","last_peered":"2024-09-10T19:02:09.543953+0000","last_clean":"2024-09-10T19:02:09.543953+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:02:09.543953+0000","last_undegraded":"2024-09-10T19:02:09.543953+0000","last_fullsized":"2024-09-10T19:02:09.543953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:25.221762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:25.221762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.213404+0000","scrub_duration":0.003571613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":554,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549574+0000","last_change":"2024-09-10T19:01:40.519369+0000","last_active":"2024-09-10T19:02:09.549574+0000","last_peered":"2024-09-10T19:02:09.549574+0000","last_clean":"2024-09-10T19:02:09.549574+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:02:09.549574+0000","last_undegraded":"2024-09-10T19:02:09.549574+0000","last_fullsized":"2024-09-10T19:02:09.549574+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:01:40.519200+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.519200+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.894993+0000","scrub_duration":0.010009456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":508,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234524+0000","last_change":"2024-09-10T19:01:34.223664+0000","last_active":"2024-09-10T19:02:09.234524+0000","last_peered":"2024-09-10T19:02:09.234524+0000","last_clean":"2024-09-10T19:02:09.234524+0000","last_became_active":"2024-09-10T18:49:22.049104+0000","last_became_peered":"2024-09-10T18:49:22.049104+0000","last_unstale":"2024-09-10T19:02:09.234524+0000","last_undegraded":"2024-09-10T19:02:09.234524+0000","last_fullsized":"2024-09-10T19:02:09.234524+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:34.223516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.223516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.044571+0000","scrub_duration":0.0033528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548015+0000","last_change":"2024-09-10T19:01:55.606548+0000","last_active":"2024-09-10T19:02:09.548015+0000","last_peered":"2024-09-10T19:02:09.548015+0000","last_clean":"2024-09-10T19:02:09.548015+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:02:09.548015+0000","last_undegraded":"2024-09-10T19:02:09.548015+0000","last_fullsized":"2024-09-10T19:02:09.548015+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.606429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.606429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:11.896569+0000","scrub_duration":0.002983313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544271+0000","last_change":"2024-09-10T19:01:30.825681+0000","last_active":"2024-09-10T19:02:09.544271+0000","last_peered":"2024-09-10T19:02:09.544271+0000","last_clean":"2024-09-10T19:02:09.544271+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:02:09.544271+0000","last_undegraded":"2024-09-10T19:02:09.544271+0000","last_fullsized":"2024-09-10T19:02:09.544271+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.825536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.825536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.619051+0000","scrub_duration":0.0034703239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":512,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544840+0000","last_change":"2024-09-10T19:01:33.189562+0000","last_active":"2024-09-10T19:02:09.544840+0000","last_peered":"2024-09-10T19:02:09.544840+0000","last_clean":"2024-09-10T19:02:09.544840+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:02:09.544840+0000","last_undegraded":"2024-09-10T19:02:09.544840+0000","last_fullsized":"2024-09-10T19:02:09.544840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:33.189399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.189399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.714879+0000","scrub_duration":0.0037148820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":551,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549773+0000","last_change":"2024-09-10T19:01:33.342581+0000","last_active":"2024-09-10T19:02:09.549773+0000","last_peered":"2024-09-10T19:02:09.549773+0000","last_clean":"2024-09-10T19:02:09.549773+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:02:09.549773+0000","last_undegraded":"2024-09-10T19:02:09.549773+0000","last_fullsized":"2024-09-10T19:02:09.549773+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:01:33.342427+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.342427+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.239579+0000","scrub_duration":0.004310543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":508,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233989+0000","last_change":"2024-09-10T19:00:35.255523+0000","last_active":"2024-09-10T19:02:09.233989+0000","last_peered":"2024-09-10T19:02:09.233989+0000","last_clean":"2024-09-10T19:02:09.233989+0000","last_became_active":"2024-09-10T18:49:22.041544+0000","last_became_peered":"2024-09-10T18:49:22.041544+0000","last_unstale":"2024-09-10T19:02:09.233989+0000","last_undegraded":"2024-09-10T19:02:09.233989+0000","last_fullsized":"2024-09-10T19:02:09.233989+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:35.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:00:35.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028367790000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":504,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546597+0000","last_change":"2024-09-10T19:01:38.581158+0000","last_active":"2024-09-10T19:02:09.546597+0000","last_peered":"2024-09-10T19:02:09.546597+0000","last_clean":"2024-09-10T19:02:09.546597+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:02:09.546597+0000","last_undegraded":"2024-09-10T19:02:09.546597+0000","last_fullsized":"2024-09-10T19:02:09.546597+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:38.580997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:38.580997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:39.595690+0000","scrub_duration":0.0033368260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":418,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241488+0000","last_change":"2024-09-10T19:00:14.868536+0000","last_active":"2024-09-10T19:02:09.241488+0000","last_peered":"2024-09-10T19:02:09.241488+0000","last_clean":"2024-09-10T19:02:09.241488+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:02:09.241488+0000","last_undegraded":"2024-09-10T19:02:09.241488+0000","last_fullsized":"2024-09-10T19:02:09.241488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.868377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.868377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036007919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":20,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.540419+0000","last_change":"2024-09-10T19:02:03.653468+0000","last_active":"2024-09-10T19:02:09.540419+0000","last_peered":"2024-09-10T19:02:09.540419+0000","last_clean":"2024-09-10T19:02:09.540419+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:02:09.540419+0000","last_undegraded":"2024-09-10T19:02:09.540419+0000","last_fullsized":"2024-09-10T19:02:09.540419+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":316358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":562,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545922+0000","last_change":"2024-09-10T19:00:16.474150+0000","last_active":"2024-09-10T19:02:09.545922+0000","last_peered":"2024-09-10T19:02:09.545922+0000","last_clean":"2024-09-10T19:02:09.545922+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:02:09.545922+0000","last_undegraded":"2024-09-10T19:02:09.545922+0000","last_fullsized":"2024-09-10T19:02:09.545922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:00:16.474006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.474006+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057543610000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546342+0000","last_change":"2024-09-10T19:01:35.169404+0000","last_active":"2024-09-10T19:02:09.546342+0000","last_peered":"2024-09-10T19:02:09.546342+0000","last_clean":"2024-09-10T19:02:09.546342+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:02:09.546342+0000","last_undegraded":"2024-09-10T19:02:09.546342+0000","last_fullsized":"2024-09-10T19:02:09.546342+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.169217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.169217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:03.816865+0000","scrub_duration":0.0032414449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"97'12","reported_seq":674,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.397181+0000","last_change":"2024-09-10T19:01:23.124779+0000","last_active":"2024-09-10T19:02:11.397181+0000","last_peered":"2024-09-10T19:02:11.397181+0000","last_clean":"2024-09-10T19:02:11.397181+0000","last_became_active":"2024-09-10T18:49:22.044117+0000","last_became_peered":"2024-09-10T18:49:22.044117+0000","last_unstale":"2024-09-10T19:02:11.397181+0000","last_undegraded":"2024-09-10T19:02:11.397181+0000","last_fullsized":"2024-09-10T19:02:11.397181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'12","last_scrub_stamp":"2024-09-10T19:01:23.124615+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.124615+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:27.487775+0000","scrub_duration":0.0037027000000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":515,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243870+0000","last_change":"2024-09-10T19:00:33.141061+0000","last_active":"2024-09-10T19:02:09.243870+0000","last_peered":"2024-09-10T19:02:09.243870+0000","last_clean":"2024-09-10T19:02:09.243870+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:02:09.243870+0000","last_undegraded":"2024-09-10T19:02:09.243870+0000","last_fullsized":"2024-09-10T19:02:09.243870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.140938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.140938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029466620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547543+0000","last_change":"2024-09-10T19:01:53.810938+0000","last_active":"2024-09-10T19:02:09.547543+0000","last_peered":"2024-09-10T19:02:09.547543+0000","last_clean":"2024-09-10T19:02:09.547543+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:02:09.547543+0000","last_undegraded":"2024-09-10T19:02:09.547543+0000","last_fullsized":"2024-09-10T19:02:09.547543+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:53.810787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.810787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:23.197836+0000","scrub_duration":0.003435802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":577,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541391+0000","last_change":"2024-09-10T19:02:03.641148+0000","last_active":"2024-09-10T19:02:09.541391+0000","last_peered":"2024-09-10T19:02:09.541391+0000","last_clean":"2024-09-10T19:02:09.541391+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:02:09.541391+0000","last_undegraded":"2024-09-10T19:02:09.541391+0000","last_fullsized":"2024-09-10T19:02:09.541391+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":530,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242192+0000","last_change":"2024-09-10T18:59:57.094883+0000","last_active":"2024-09-10T19:02:09.242192+0000","last_peered":"2024-09-10T19:02:09.242192+0000","last_clean":"2024-09-10T19:02:09.242192+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:02:09.242192+0000","last_undegraded":"2024-09-10T19:02:09.242192+0000","last_fullsized":"2024-09-10T19:02:09.242192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T18:59:57.094721+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T18:59:57.094721+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045594349999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":520,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544042+0000","last_change":"2024-09-10T19:01:27.860664+0000","last_active":"2024-09-10T19:02:09.544042+0000","last_peered":"2024-09-10T19:02:09.544042+0000","last_clean":"2024-09-10T19:02:09.544042+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:02:09.544042+0000","last_undegraded":"2024-09-10T19:02:09.544042+0000","last_fullsized":"2024-09-10T19:02:09.544042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:27.860545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.860545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:51.717349+0000","scrub_duration":0.0031084049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":511,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234607+0000","last_change":"2024-09-10T19:01:18.068985+0000","last_active":"2024-09-10T19:02:09.234607+0000","last_peered":"2024-09-10T19:02:09.234607+0000","last_clean":"2024-09-10T19:02:09.234607+0000","last_became_active":"2024-09-10T18:49:22.041018+0000","last_became_peered":"2024-09-10T18:49:22.041018+0000","last_unstale":"2024-09-10T19:02:09.234607+0000","last_undegraded":"2024-09-10T19:02:09.234607+0000","last_fullsized":"2024-09-10T19:02:09.234607+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.068817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.068817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:24.861245+0000","scrub_duration":0.003161531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235954+0000","last_change":"2024-09-10T19:01:21.078775+0000","last_active":"2024-09-10T19:02:09.235954+0000","last_peered":"2024-09-10T19:02:09.235954+0000","last_clean":"2024-09-10T19:02:09.235954+0000","last_became_active":"2024-09-10T18:49:24.044413+0000","last_became_peered":"2024-09-10T18:49:24.044413+0000","last_unstale":"2024-09-10T19:02:09.235954+0000","last_undegraded":"2024-09-10T19:02:09.235954+0000","last_fullsized":"2024-09-10T19:02:09.235954+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.078620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.078620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:27.738168+0000","scrub_duration":0.0029631150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":416,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545877+0000","last_change":"2024-09-10T19:00:07.323465+0000","last_active":"2024-09-10T19:02:09.545877+0000","last_peered":"2024-09-10T19:02:09.545877+0000","last_clean":"2024-09-10T19:02:09.545877+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:02:09.545877+0000","last_undegraded":"2024-09-10T19:02:09.545877+0000","last_fullsized":"2024-09-10T19:02:09.545877+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:07.323364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:07.323364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028330460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":13,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548676+0000","last_change":"2024-09-10T19:02:03.643555+0000","last_active":"2024-09-10T19:02:09.548676+0000","last_peered":"2024-09-10T19:02:09.548676+0000","last_clean":"2024-09-10T19:02:09.548676+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:02:09.548676+0000","last_undegraded":"2024-09-10T19:02:09.548676+0000","last_fullsized":"2024-09-10T19:02:09.548676+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":544,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.240605+0000","last_change":"2024-09-10T19:00:12.906006+0000","last_active":"2024-09-10T19:02:09.240605+0000","last_peered":"2024-09-10T19:02:09.240605+0000","last_clean":"2024-09-10T19:02:09.240605+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:02:09.240605+0000","last_undegraded":"2024-09-10T19:02:09.240605+0000","last_fullsized":"2024-09-10T19:02:09.240605+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:00:12.905889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.905889+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003872931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543939+0000","last_change":"2024-09-10T19:01:24.883894+0000","last_active":"2024-09-10T19:02:09.543939+0000","last_peered":"2024-09-10T19:02:09.543939+0000","last_clean":"2024-09-10T19:02:09.543939+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:02:09.543939+0000","last_undegraded":"2024-09-10T19:02:09.543939+0000","last_fullsized":"2024-09-10T19:02:09.543939+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:24.883733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:24.883733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.277814+0000","scrub_duration":0.0033285659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":516,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548346+0000","last_change":"2024-09-10T19:01:37.874644+0000","last_active":"2024-09-10T19:02:09.548346+0000","last_peered":"2024-09-10T19:02:09.548346+0000","last_clean":"2024-09-10T19:02:09.548346+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:02:09.548346+0000","last_undegraded":"2024-09-10T19:02:09.548346+0000","last_fullsized":"2024-09-10T19:02:09.548346+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.874480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.874480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.675868+0000","scrub_duration":0.0034122890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549505+0000","last_change":"2024-09-10T19:01:44.400980+0000","last_active":"2024-09-10T19:02:09.549505+0000","last_peered":"2024-09-10T19:02:09.549505+0000","last_clean":"2024-09-10T19:02:09.549505+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:02:09.549505+0000","last_undegraded":"2024-09-10T19:02:09.549505+0000","last_fullsized":"2024-09-10T19:02:09.549505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.400822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.400822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:13.087644+0000","scrub_duration":0.00334009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548441+0000","last_change":"2024-09-10T19:00:14.537922+0000","last_active":"2024-09-10T19:02:09.548441+0000","last_peered":"2024-09-10T19:02:09.548441+0000","last_clean":"2024-09-10T19:02:09.548441+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:02:09.548441+0000","last_undegraded":"2024-09-10T19:02:09.548441+0000","last_fullsized":"2024-09-10T19:02:09.548441+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.537765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.537765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033849069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":547,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234084+0000","last_change":"2024-09-10T18:58:57.446647+0000","last_active":"2024-09-10T19:02:09.234084+0000","last_peered":"2024-09-10T19:02:09.234084+0000","last_clean":"2024-09-10T19:02:09.234084+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:02:09.234084+0000","last_undegraded":"2024-09-10T19:02:09.234084+0000","last_fullsized":"2024-09-10T19:02:09.234084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'17","last_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T18:58:57.446527+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011008198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542594+0000","last_change":"2024-09-10T19:00:24.104254+0000","last_active":"2024-09-10T19:02:09.542594+0000","last_peered":"2024-09-10T19:02:09.542594+0000","last_clean":"2024-09-10T19:02:09.542594+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:02:09.542594+0000","last_undegraded":"2024-09-10T19:02:09.542594+0000","last_fullsized":"2024-09-10T19:02:09.542594+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:24.104066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:24.104066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038382149999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"95'12","reported_seq":674,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.397129+0000","last_change":"2024-09-10T19:00:05.221386+0000","last_active":"2024-09-10T19:02:11.397129+0000","last_peered":"2024-09-10T19:02:11.397129+0000","last_clean":"2024-09-10T19:02:11.397129+0000","last_became_active":"2024-09-10T18:49:22.044191+0000","last_became_peered":"2024-09-10T18:49:22.044191+0000","last_unstale":"2024-09-10T19:02:11.397129+0000","last_undegraded":"2024-09-10T19:02:11.397129+0000","last_fullsized":"2024-09-10T19:02:11.397129+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'12","last_scrub_stamp":"2024-09-10T19:00:05.221191+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:00:05.221191+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039722840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":498,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242237+0000","last_change":"2024-09-10T19:00:22.885319+0000","last_active":"2024-09-10T19:02:09.242237+0000","last_peered":"2024-09-10T19:02:09.242237+0000","last_clean":"2024-09-10T19:02:09.242237+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:02:09.242237+0000","last_undegraded":"2024-09-10T19:02:09.242237+0000","last_fullsized":"2024-09-10T19:02:09.242237+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:00:22.885146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:22.885146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048287809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547852+0000","last_change":"2024-09-10T19:01:29.873888+0000","last_active":"2024-09-10T19:02:09.547852+0000","last_peered":"2024-09-10T19:02:09.547852+0000","last_clean":"2024-09-10T19:02:09.547852+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:02:09.547852+0000","last_undegraded":"2024-09-10T19:02:09.547852+0000","last_fullsized":"2024-09-10T19:02:09.547852+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.873730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.873730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:52.635541+0000","scrub_duration":0.0034669100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":18,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241832+0000","last_change":"2024-09-10T19:02:03.625372+0000","last_active":"2024-09-10T19:02:09.241832+0000","last_peered":"2024-09-10T19:02:09.241832+0000","last_clean":"2024-09-10T19:02:09.241832+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:02:09.241832+0000","last_undegraded":"2024-09-10T19:02:09.241832+0000","last_fullsized":"2024-09-10T19:02:09.241832+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":534,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546163+0000","last_change":"2024-09-10T19:01:23.545508+0000","last_active":"2024-09-10T19:02:09.546163+0000","last_peered":"2024-09-10T19:02:09.546163+0000","last_clean":"2024-09-10T19:02:09.546163+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:02:09.546163+0000","last_undegraded":"2024-09-10T19:02:09.546163+0000","last_fullsized":"2024-09-10T19:02:09.546163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:01:23.545431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.545431+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.986385+0000","scrub_duration":0.004183679,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548353+0000","last_change":"2024-09-10T19:01:21.747071+0000","last_active":"2024-09-10T19:02:09.548353+0000","last_peered":"2024-09-10T19:02:09.548353+0000","last_clean":"2024-09-10T19:02:09.548353+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:02:09.548353+0000","last_undegraded":"2024-09-10T19:02:09.548353+0000","last_fullsized":"2024-09-10T19:02:09.548353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.746907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.746907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.629492+0000","scrub_duration":0.003822261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":508,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233495+0000","last_change":"2024-09-10T19:00:10.266358+0000","last_active":"2024-09-10T19:02:09.233495+0000","last_peered":"2024-09-10T19:02:09.233495+0000","last_clean":"2024-09-10T19:02:09.233495+0000","last_became_active":"2024-09-10T18:49:22.041065+0000","last_became_peered":"2024-09-10T18:49:22.041065+0000","last_unstale":"2024-09-10T19:02:09.233495+0000","last_undegraded":"2024-09-10T19:02:09.233495+0000","last_fullsized":"2024-09-10T19:02:09.233495+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:10.266243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:00:10.266243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003123285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1269,"reported_epoch":514,"state":"active+clean","last_fresh":"2024-09-10T19:02:14.361084+0000","last_change":"2024-09-10T19:01:26.001371+0000","last_active":"2024-09-10T19:02:14.361084+0000","last_peered":"2024-09-10T19:02:14.361084+0000","last_clean":"2024-09-10T19:02:14.361084+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:02:14.361084+0000","last_undegraded":"2024-09-10T19:02:14.361084+0000","last_fullsized":"2024-09-10T19:02:14.361084+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:01:26.001280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.001280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:49.352524+0000","scrub_duration":0.003156715,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":752,"num_read_kb":747,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543878+0000","last_change":"2024-09-10T19:00:04.106819+0000","last_active":"2024-09-10T19:02:09.543878+0000","last_peered":"2024-09-10T19:02:09.543878+0000","last_clean":"2024-09-10T19:02:09.543878+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:02:09.543878+0000","last_undegraded":"2024-09-10T19:02:09.543878+0000","last_fullsized":"2024-09-10T19:02:09.543878+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:04.106660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:00:04.106660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031927650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":447,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.238945+0000","last_change":"2024-09-10T19:01:53.223149+0000","last_active":"2024-09-10T19:02:09.238945+0000","last_peered":"2024-09-10T19:02:09.238945+0000","last_clean":"2024-09-10T19:02:09.238945+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:02:09.238945+0000","last_undegraded":"2024-09-10T19:02:09.238945+0000","last_fullsized":"2024-09-10T19:02:09.238945+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:53.223000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.223000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:19.891027+0000","scrub_duration":0.0036103200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":13,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235910+0000","last_change":"2024-09-10T19:02:03.640653+0000","last_active":"2024-09-10T19:02:09.235910+0000","last_peered":"2024-09-10T19:02:09.235910+0000","last_clean":"2024-09-10T19:02:09.235910+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:02:09.235910+0000","last_undegraded":"2024-09-10T19:02:09.235910+0000","last_fullsized":"2024-09-10T19:02:09.235910+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":534,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243502+0000","last_change":"2024-09-10T18:58:59.323611+0000","last_active":"2024-09-10T19:02:09.243502+0000","last_peered":"2024-09-10T19:02:09.243502+0000","last_clean":"2024-09-10T19:02:09.243502+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:02:09.243502+0000","last_undegraded":"2024-09-10T19:02:09.243502+0000","last_fullsized":"2024-09-10T19:02:09.243502+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'13","last_scrub_stamp":"2024-09-10T18:58:59.323529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.323529+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010243271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":520,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548248+0000","last_change":"2024-09-10T19:01:36.860957+0000","last_active":"2024-09-10T19:02:09.548248+0000","last_peered":"2024-09-10T19:02:09.548248+0000","last_clean":"2024-09-10T19:02:09.548248+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:02:09.548248+0000","last_undegraded":"2024-09-10T19:02:09.548248+0000","last_fullsized":"2024-09-10T19:02:09.548248+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:36.860807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.860807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:54.793076+0000","scrub_duration":0.003533815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"77'11","reported_seq":654,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.399453+0000","last_change":"2024-09-10T18:59:59.066338+0000","last_active":"2024-09-10T19:02:11.399453+0000","last_peered":"2024-09-10T19:02:11.399453+0000","last_clean":"2024-09-10T19:02:11.399453+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:02:11.399453+0000","last_undegraded":"2024-09-10T19:02:11.399453+0000","last_fullsized":"2024-09-10T19:02:11.399453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T18:59:59.066136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.066136+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038648509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":515,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233230+0000","last_change":"2024-09-10T18:59:04.451994+0000","last_active":"2024-09-10T19:02:09.233230+0000","last_peered":"2024-09-10T19:02:09.233230+0000","last_clean":"2024-09-10T19:02:09.233230+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:02:09.233230+0000","last_undegraded":"2024-09-10T19:02:09.233230+0000","last_fullsized":"2024-09-10T19:02:09.233230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T18:59:04.451827+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T18:59:04.451827+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004116354,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233796+0000","last_change":"2024-09-10T19:01:19.068892+0000","last_active":"2024-09-10T19:02:09.233796+0000","last_peered":"2024-09-10T19:02:09.233796+0000","last_clean":"2024-09-10T19:02:09.233796+0000","last_became_active":"2024-09-10T18:49:24.044983+0000","last_became_peered":"2024-09-10T18:49:24.044983+0000","last_unstale":"2024-09-10T19:02:09.233796+0000","last_undegraded":"2024-09-10T19:02:09.233796+0000","last_fullsized":"2024-09-10T19:02:09.233796+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.068733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.068733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:35.414773+0000","scrub_duration":0.0034618380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":418,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239756+0000","last_change":"2024-09-10T19:01:21.953462+0000","last_active":"2024-09-10T19:02:09.239756+0000","last_peered":"2024-09-10T19:02:09.239756+0000","last_clean":"2024-09-10T19:02:09.239756+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:02:09.239756+0000","last_undegraded":"2024-09-10T19:02:09.239756+0000","last_fullsized":"2024-09-10T19:02:09.239756+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.953303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.953303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:22.871763+0000","scrub_duration":0.0036359589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":92,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547377+0000","last_change":"2024-09-10T19:02:03.613321+0000","last_active":"2024-09-10T19:02:09.547377+0000","last_peered":"2024-09-10T19:02:09.547377+0000","last_clean":"2024-09-10T19:02:09.547377+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:02:09.547377+0000","last_undegraded":"2024-09-10T19:02:09.547377+0000","last_fullsized":"2024-09-10T19:02:09.547377+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":741,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.400006+0000","last_change":"2024-09-10T19:00:01.595171+0000","last_active":"2024-09-10T19:02:11.400006+0000","last_peered":"2024-09-10T19:02:11.400006+0000","last_clean":"2024-09-10T19:02:11.400006+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:02:11.400006+0000","last_undegraded":"2024-09-10T19:02:11.400006+0000","last_fullsized":"2024-09-10T19:02:11.400006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:00:01.595006+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:00:01.595006+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053873050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548179+0000","last_change":"2024-09-10T19:01:20.312372+0000","last_active":"2024-09-10T19:02:09.548179+0000","last_peered":"2024-09-10T19:02:09.548179+0000","last_clean":"2024-09-10T19:02:09.548179+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:02:09.548179+0000","last_undegraded":"2024-09-10T19:02:09.548179+0000","last_fullsized":"2024-09-10T19:02:09.548179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:20.312136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:20.312136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.789362+0000","scrub_duration":0.0033891630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":662,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.399234+0000","last_change":"2024-09-10T18:59:56.107053+0000","last_active":"2024-09-10T19:02:11.399234+0000","last_peered":"2024-09-10T19:02:11.399234+0000","last_clean":"2024-09-10T19:02:11.399234+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:02:11.399234+0000","last_undegraded":"2024-09-10T19:02:11.399234+0000","last_fullsized":"2024-09-10T19:02:11.399234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T18:59:56.106896+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T18:59:56.106896+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036065950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":513,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545519+0000","last_change":"2024-09-10T19:01:16.768276+0000","last_active":"2024-09-10T19:02:09.545519+0000","last_peered":"2024-09-10T19:02:09.545519+0000","last_clean":"2024-09-10T19:02:09.545519+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:02:09.545519+0000","last_undegraded":"2024-09-10T19:02:09.545519+0000","last_fullsized":"2024-09-10T19:02:09.545519+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:01:16.768112+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.768112+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.561140+0000","scrub_duration":0.0040962680000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":504,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543203+0000","last_change":"2024-09-10T19:01:18.860189+0000","last_active":"2024-09-10T19:02:09.543203+0000","last_peered":"2024-09-10T19:02:09.543203+0000","last_clean":"2024-09-10T19:02:09.543203+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:02:09.543203+0000","last_undegraded":"2024-09-10T19:02:09.543203+0000","last_fullsized":"2024-09-10T19:02:09.543203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:26.756938+0000","scrub_duration":0.0032709060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":418,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242356+0000","last_change":"2024-09-10T19:00:31.934239+0000","last_active":"2024-09-10T19:02:09.242356+0000","last_peered":"2024-09-10T19:02:09.242356+0000","last_clean":"2024-09-10T19:02:09.242356+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:02:09.242356+0000","last_undegraded":"2024-09-10T19:02:09.242356+0000","last_fullsized":"2024-09-10T19:02:09.242356+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:31.933970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:31.933970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003362608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":92,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547399+0000","last_change":"2024-09-10T19:02:03.615773+0000","last_active":"2024-09-10T19:02:09.547399+0000","last_peered":"2024-09-10T19:02:09.547399+0000","last_clean":"2024-09-10T19:02:09.547399+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:02:09.547399+0000","last_undegraded":"2024-09-10T19:02:09.547399+0000","last_fullsized":"2024-09-10T19:02:09.547399+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":544,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549823+0000","last_change":"2024-09-10T19:01:27.401342+0000","last_active":"2024-09-10T19:02:09.549823+0000","last_peered":"2024-09-10T19:02:09.549823+0000","last_clean":"2024-09-10T19:02:09.549823+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:02:09.549823+0000","last_undegraded":"2024-09-10T19:02:09.549823+0000","last_fullsized":"2024-09-10T19:02:09.549823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'26","last_scrub_stamp":"2024-09-10T19:01:27.401178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.401178+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.489469+0000","scrub_duration":0.0045996170000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547119+0000","last_change":"2024-09-10T19:01:15.859080+0000","last_active":"2024-09-10T19:02:09.547119+0000","last_peered":"2024-09-10T19:02:09.547119+0000","last_clean":"2024-09-10T19:02:09.547119+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:02:09.547119+0000","last_undegraded":"2024-09-10T19:02:09.547119+0000","last_fullsized":"2024-09-10T19:02:09.547119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.858920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.858920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:17.797614+0000","scrub_duration":0.003170703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543341+0000","last_change":"2024-09-10T19:01:40.872639+0000","last_active":"2024-09-10T19:02:09.543341+0000","last_peered":"2024-09-10T19:02:09.543341+0000","last_clean":"2024-09-10T19:02:09.543341+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:02:09.543341+0000","last_undegraded":"2024-09-10T19:02:09.543341+0000","last_fullsized":"2024-09-10T19:02:09.543341+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:40.872535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.872535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.545270+0000","scrub_duration":0.0029510880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":494,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547059+0000","last_change":"2024-09-10T19:00:00.056759+0000","last_active":"2024-09-10T19:02:09.547059+0000","last_peered":"2024-09-10T19:02:09.547059+0000","last_clean":"2024-09-10T19:02:09.547059+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:02:09.547059+0000","last_undegraded":"2024-09-10T19:02:09.547059+0000","last_fullsized":"2024-09-10T19:02:09.547059+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:00.056600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:00:00.056600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028960119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":437,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239058+0000","last_change":"2024-09-10T19:01:16.369137+0000","last_active":"2024-09-10T19:02:09.239058+0000","last_peered":"2024-09-10T19:02:09.239058+0000","last_clean":"2024-09-10T19:02:09.239058+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:02:09.239058+0000","last_undegraded":"2024-09-10T19:02:09.239058+0000","last_fullsized":"2024-09-10T19:02:09.239058+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:16.368989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:16.368989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:17.537485+0000","scrub_duration":0.0034355140000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":13,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.540615+0000","last_change":"2024-09-10T19:02:03.641229+0000","last_active":"2024-09-10T19:02:09.540615+0000","last_peered":"2024-09-10T19:02:09.540615+0000","last_clean":"2024-09-10T19:02:09.540615+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:02:09.540615+0000","last_undegraded":"2024-09-10T19:02:09.540615+0000","last_fullsized":"2024-09-10T19:02:09.540615+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":561,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235204+0000","last_change":"2024-09-10T19:00:27.165321+0000","last_active":"2024-09-10T19:02:09.235204+0000","last_peered":"2024-09-10T19:02:09.235204+0000","last_clean":"2024-09-10T19:02:09.235204+0000","last_became_active":"2024-09-10T18:49:19.633834+0000","last_became_peered":"2024-09-10T18:49:19.633834+0000","last_unstale":"2024-09-10T19:02:09.235204+0000","last_undegraded":"2024-09-10T19:02:09.235204+0000","last_fullsized":"2024-09-10T19:02:09.235204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:00:27.165232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.165232+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047657200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549435+0000","last_change":"2024-09-10T19:01:30.331378+0000","last_active":"2024-09-10T19:02:09.549435+0000","last_peered":"2024-09-10T19:02:09.549435+0000","last_clean":"2024-09-10T19:02:09.549435+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:02:09.549435+0000","last_undegraded":"2024-09-10T19:02:09.549435+0000","last_fullsized":"2024-09-10T19:02:09.549435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.331218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.331218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:48.513446+0000","scrub_duration":0.0033888009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544904+0000","last_change":"2024-09-10T19:00:25.638042+0000","last_active":"2024-09-10T19:02:09.544904+0000","last_peered":"2024-09-10T19:02:09.544904+0000","last_clean":"2024-09-10T19:02:09.544904+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:02:09.544904+0000","last_undegraded":"2024-09-10T19:02:09.544904+0000","last_fullsized":"2024-09-10T19:02:09.544904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.637874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.637874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033414320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":490,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242679+0000","last_change":"2024-09-10T19:01:45.880130+0000","last_active":"2024-09-10T19:02:09.242679+0000","last_peered":"2024-09-10T19:02:09.242679+0000","last_clean":"2024-09-10T19:02:09.242679+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:02:09.242679+0000","last_undegraded":"2024-09-10T19:02:09.242679+0000","last_fullsized":"2024-09-10T19:02:09.242679+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:39.317483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:39.317483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003112138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545251+0000","last_change":"2024-09-10T19:00:21.205302+0000","last_active":"2024-09-10T19:02:09.545251+0000","last_peered":"2024-09-10T19:02:09.545251+0000","last_clean":"2024-09-10T19:02:09.545251+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:02:09.545251+0000","last_undegraded":"2024-09-10T19:02:09.545251+0000","last_fullsized":"2024-09-10T19:02:09.545251+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.205176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.205176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033764950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":577,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541461+0000","last_change":"2024-09-10T19:02:03.614492+0000","last_active":"2024-09-10T19:02:09.541461+0000","last_peered":"2024-09-10T19:02:09.541461+0000","last_clean":"2024-09-10T19:02:09.541461+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:02:09.541461+0000","last_undegraded":"2024-09-10T19:02:09.541461+0000","last_fullsized":"2024-09-10T19:02:09.541461+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":21,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":566,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542948+0000","last_change":"2024-09-10T19:00:13.487926+0000","last_active":"2024-09-10T19:02:09.542948+0000","last_peered":"2024-09-10T19:02:09.542948+0000","last_clean":"2024-09-10T19:02:09.542948+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:02:09.542948+0000","last_undegraded":"2024-09-10T19:02:09.542948+0000","last_fullsized":"2024-09-10T19:02:09.542948+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.487819+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003630566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":531,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.238629+0000","last_change":"2024-09-10T19:01:29.231379+0000","last_active":"2024-09-10T19:02:09.238629+0000","last_peered":"2024-09-10T19:02:09.238629+0000","last_clean":"2024-09-10T19:02:09.238629+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:02:09.238629+0000","last_undegraded":"2024-09-10T19:02:09.238629+0000","last_fullsized":"2024-09-10T19:02:09.238629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.231291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:12.280600+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.231291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:41.516391+0000","scrub_duration":0.0020441320000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548530+0000","last_change":"2024-09-10T19:01:42.936277+0000","last_active":"2024-09-10T19:02:09.548530+0000","last_peered":"2024-09-10T19:02:09.548530+0000","last_clean":"2024-09-10T19:02:09.548530+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:02:09.548530+0000","last_undegraded":"2024-09-10T19:02:09.548530+0000","last_fullsized":"2024-09-10T19:02:09.548530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.936095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:42.936095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.936095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:03.472082+0000","scrub_duration":0.0035076700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544166+0000","last_change":"2024-09-10T19:00:17.050264+0000","last_active":"2024-09-10T19:02:09.544166+0000","last_peered":"2024-09-10T19:02:09.544166+0000","last_clean":"2024-09-10T19:02:09.544166+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:02:09.544166+0000","last_undegraded":"2024-09-10T19:02:09.544166+0000","last_fullsized":"2024-09-10T19:02:09.544166+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.050058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.050058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038046009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547748+0000","last_change":"2024-09-10T19:01:23.802298+0000","last_active":"2024-09-10T19:02:09.547748+0000","last_peered":"2024-09-10T19:02:09.547748+0000","last_clean":"2024-09-10T19:02:09.547748+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:02:09.547748+0000","last_undegraded":"2024-09-10T19:02:09.547748+0000","last_fullsized":"2024-09-10T19:02:09.547748+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:23.802111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:23.802111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:29.627831+0000","scrub_duration":0.0034655179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":13,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543238+0000","last_change":"2024-09-10T19:02:03.616590+0000","last_active":"2024-09-10T19:02:09.543238+0000","last_peered":"2024-09-10T19:02:09.543238+0000","last_clean":"2024-09-10T19:02:09.543238+0000","last_became_active":"2024-09-10T19:02:03.616064+0000","last_became_peered":"2024-09-10T19:02:03.616064+0000","last_unstale":"2024-09-10T19:02:09.543238+0000","last_undegraded":"2024-09-10T19:02:09.543238+0000","last_fullsized":"2024-09-10T19:02:09.543238+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":550,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544155+0000","last_change":"2024-09-10T19:01:53.649846+0000","last_active":"2024-09-10T19:02:09.544155+0000","last_peered":"2024-09-10T19:02:09.544155+0000","last_clean":"2024-09-10T19:02:09.544155+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:02:09.544155+0000","last_undegraded":"2024-09-10T19:02:09.544155+0000","last_fullsized":"2024-09-10T19:02:09.544155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'28","last_scrub_stamp":"2024-09-10T19:01:53.649703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.649703+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:06.692256+0000","scrub_duration":0.0088661820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541611+0000","last_change":"2024-09-10T19:00:26.116209+0000","last_active":"2024-09-10T19:02:09.541611+0000","last_peered":"2024-09-10T19:02:09.541611+0000","last_clean":"2024-09-10T19:02:09.541611+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:02:09.541611+0000","last_undegraded":"2024-09-10T19:02:09.541611+0000","last_fullsized":"2024-09-10T19:02:09.541611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.116042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039421509999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":499,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546233+0000","last_change":"2024-09-10T19:01:54.655470+0000","last_active":"2024-09-10T19:02:09.546233+0000","last_peered":"2024-09-10T19:02:09.546233+0000","last_clean":"2024-09-10T19:02:09.546233+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:02:09.546233+0000","last_undegraded":"2024-09-10T19:02:09.546233+0000","last_fullsized":"2024-09-10T19:02:09.546233+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:03.826239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:03.826239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002998045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":504,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544266+0000","last_change":"2024-09-10T19:00:03.612383+0000","last_active":"2024-09-10T19:02:09.544266+0000","last_peered":"2024-09-10T19:02:09.544266+0000","last_clean":"2024-09-10T19:02:09.544266+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:02:09.544266+0000","last_undegraded":"2024-09-10T19:02:09.544266+0000","last_fullsized":"2024-09-10T19:02:09.544266+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:03.612225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:00:03.612225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033384840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546441+0000","last_change":"2024-09-10T19:00:25.167720+0000","last_active":"2024-09-10T19:02:09.546441+0000","last_peered":"2024-09-10T19:02:09.546441+0000","last_clean":"2024-09-10T19:02:09.546441+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:02:09.546441+0000","last_undegraded":"2024-09-10T19:02:09.546441+0000","last_fullsized":"2024-09-10T19:02:09.546441+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:25.167452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.167452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033805139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":526,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241542+0000","last_change":"2024-09-10T18:58:58.282166+0000","last_active":"2024-09-10T19:02:09.241542+0000","last_peered":"2024-09-10T19:02:09.241542+0000","last_clean":"2024-09-10T19:02:09.241542+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:02:09.241542+0000","last_undegraded":"2024-09-10T19:02:09.241542+0000","last_fullsized":"2024-09-10T19:02:09.241542+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'15","last_scrub_stamp":"2024-09-10T18:58:58.282011+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.282011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071078970000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549147+0000","last_change":"2024-09-10T19:00:15.230885+0000","last_active":"2024-09-10T19:02:09.549147+0000","last_peered":"2024-09-10T19:02:09.549147+0000","last_clean":"2024-09-10T19:02:09.549147+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:02:09.549147+0000","last_undegraded":"2024-09-10T19:02:09.549147+0000","last_fullsized":"2024-09-10T19:02:09.549147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.230765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002962732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":568,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541992+0000","last_change":"2024-09-10T19:02:01.597766+0000","last_active":"2024-09-10T19:02:09.541992+0000","last_peered":"2024-09-10T19:02:09.541992+0000","last_clean":"2024-09-10T19:02:09.541992+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:02:09.541992+0000","last_undegraded":"2024-09-10T19:02:09.541992+0000","last_fullsized":"2024-09-10T19:02:09.541992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043677539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":21,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":505,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548319+0000","last_change":"2024-09-10T19:01:28.744025+0000","last_active":"2024-09-10T19:02:09.548319+0000","last_peered":"2024-09-10T19:02:09.548319+0000","last_clean":"2024-09-10T19:02:09.548319+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:02:09.548319+0000","last_undegraded":"2024-09-10T19:02:09.548319+0000","last_fullsized":"2024-09-10T19:02:09.548319+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:28.743841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.743841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:38.125924+0000","scrub_duration":0.0035976770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":493,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.550106+0000","last_change":"2024-09-10T19:01:31.371325+0000","last_active":"2024-09-10T19:02:09.550106+0000","last_peered":"2024-09-10T19:02:09.550106+0000","last_clean":"2024-09-10T19:02:09.550106+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:02:09.550106+0000","last_undegraded":"2024-09-10T19:02:09.550106+0000","last_fullsized":"2024-09-10T19:02:09.550106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:59.453913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:58:59.453913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00343364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544858+0000","last_change":"2024-09-10T19:01:29.180999+0000","last_active":"2024-09-10T19:02:09.544858+0000","last_peered":"2024-09-10T19:02:09.544858+0000","last_clean":"2024-09-10T19:02:09.544858+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:02:09.544858+0000","last_undegraded":"2024-09-10T19:02:09.544858+0000","last_fullsized":"2024-09-10T19:02:09.544858+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.180921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.180921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.972784+0000","scrub_duration":0.0085685290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":564,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548107+0000","last_change":"2024-09-10T19:01:27.255478+0000","last_active":"2024-09-10T19:02:09.548107+0000","last_peered":"2024-09-10T19:02:09.548107+0000","last_clean":"2024-09-10T19:02:09.548107+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:02:09.548107+0000","last_undegraded":"2024-09-10T19:02:09.548107+0000","last_fullsized":"2024-09-10T19:02:09.548107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:01:27.255315+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.255315+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.261030+0000","scrub_duration":0.0050413439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":531,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241145+0000","last_change":"2024-09-10T19:01:38.102909+0000","last_active":"2024-09-10T19:02:09.241145+0000","last_peered":"2024-09-10T19:02:09.241145+0000","last_clean":"2024-09-10T19:02:09.241145+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:02:09.241145+0000","last_undegraded":"2024-09-10T19:02:09.241145+0000","last_fullsized":"2024-09-10T19:02:09.241145+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:38.102784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:38.102784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:50.243407+0000","scrub_duration":0.0025983339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":505,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546141+0000","last_change":"2024-09-10T19:00:29.731792+0000","last_active":"2024-09-10T19:02:09.546141+0000","last_peered":"2024-09-10T19:02:09.546141+0000","last_clean":"2024-09-10T19:02:09.546141+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:02:09.546141+0000","last_undegraded":"2024-09-10T19:02:09.546141+0000","last_fullsized":"2024-09-10T19:02:09.546141+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:00:29.731727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012104484,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542745+0000","last_change":"2024-09-10T19:01:31.194872+0000","last_active":"2024-09-10T19:02:09.542745+0000","last_peered":"2024-09-10T19:02:09.542745+0000","last_clean":"2024-09-10T19:02:09.542745+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:02:09.542745+0000","last_undegraded":"2024-09-10T19:02:09.542745+0000","last_fullsized":"2024-09-10T19:02:09.542745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.194789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.194789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.340897+0000","scrub_duration":0.003152789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.233322+0000","last_change":"2024-09-10T19:01:42.293853+0000","last_active":"2024-09-10T19:02:09.233322+0000","last_peered":"2024-09-10T19:02:09.233322+0000","last_clean":"2024-09-10T19:02:09.233322+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:02:09.233322+0000","last_undegraded":"2024-09-10T19:02:09.233322+0000","last_fullsized":"2024-09-10T19:02:09.233322+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.293664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.293664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.123000+0000","scrub_duration":0.0031423390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":539,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549773+0000","last_change":"2024-09-10T19:01:32.331406+0000","last_active":"2024-09-10T19:02:09.549773+0000","last_peered":"2024-09-10T19:02:09.549773+0000","last_clean":"2024-09-10T19:02:09.549773+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:02:09.549773+0000","last_undegraded":"2024-09-10T19:02:09.549773+0000","last_fullsized":"2024-09-10T19:02:09.549773+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:01:32.331254+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:01:32.331254+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:34.473619+0000","scrub_duration":0.0043664410000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548307+0000","last_change":"2024-09-10T19:00:34.708486+0000","last_active":"2024-09-10T19:02:09.548307+0000","last_peered":"2024-09-10T19:02:09.548307+0000","last_clean":"2024-09-10T19:02:09.548307+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:02:09.548307+0000","last_undegraded":"2024-09-10T19:02:09.548307+0000","last_fullsized":"2024-09-10T19:02:09.548307+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:34.708428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:34.708428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023757650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":496,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546052+0000","last_change":"2024-09-10T19:01:41.594329+0000","last_active":"2024-09-10T19:02:09.546052+0000","last_peered":"2024-09-10T19:02:09.546052+0000","last_clean":"2024-09-10T19:02:09.546052+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:02:09.546052+0000","last_undegraded":"2024-09-10T19:02:09.546052+0000","last_fullsized":"2024-09-10T19:02:09.546052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.594132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.594132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.455719+0000","scrub_duration":0.003049576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":494,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543937+0000","last_change":"2024-09-10T19:01:35.482982+0000","last_active":"2024-09-10T19:02:09.543937+0000","last_peered":"2024-09-10T19:02:09.543937+0000","last_clean":"2024-09-10T19:02:09.543937+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:02:09.543937+0000","last_undegraded":"2024-09-10T19:02:09.543937+0000","last_fullsized":"2024-09-10T19:02:09.543937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.482815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.482815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.047812+0000","scrub_duration":0.0032822699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546047+0000","last_change":"2024-09-10T19:00:32.154082+0000","last_active":"2024-09-10T19:02:09.546047+0000","last_peered":"2024-09-10T19:02:09.546047+0000","last_clean":"2024-09-10T19:02:09.546047+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:02:09.546047+0000","last_undegraded":"2024-09-10T19:02:09.546047+0000","last_fullsized":"2024-09-10T19:02:09.546047+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:32.153923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.153923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003341105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":557,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544511+0000","last_change":"2024-09-10T19:00:27.155044+0000","last_active":"2024-09-10T19:02:09.544511+0000","last_peered":"2024-09-10T19:02:09.544511+0000","last_clean":"2024-09-10T19:02:09.544511+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:02:09.544511+0000","last_undegraded":"2024-09-10T19:02:09.544511+0000","last_fullsized":"2024-09-10T19:02:09.544511+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:00:27.154715+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.154715+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005022098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":543,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243322+0000","last_change":"2024-09-10T19:01:34.178468+0000","last_active":"2024-09-10T19:02:09.243322+0000","last_peered":"2024-09-10T19:02:09.243322+0000","last_clean":"2024-09-10T19:02:09.243322+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:02:09.243322+0000","last_undegraded":"2024-09-10T19:02:09.243322+0000","last_fullsized":"2024-09-10T19:02:09.243322+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:01:34.178344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.178344+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.702660+0000","scrub_duration":0.007965204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":512,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541070+0000","last_change":"2024-09-10T19:01:37.418361+0000","last_active":"2024-09-10T19:02:09.541070+0000","last_peered":"2024-09-10T19:02:09.541070+0000","last_clean":"2024-09-10T19:02:09.541070+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:02:09.541070+0000","last_undegraded":"2024-09-10T19:02:09.541070+0000","last_fullsized":"2024-09-10T19:02:09.541070+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.418152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.418152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.115944+0000","scrub_duration":0.0033093839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":500,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549255+0000","last_change":"2024-09-10T19:00:30.155345+0000","last_active":"2024-09-10T19:02:09.549255+0000","last_peered":"2024-09-10T19:02:09.549255+0000","last_clean":"2024-09-10T19:02:09.549255+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:02:09.549255+0000","last_undegraded":"2024-09-10T19:02:09.549255+0000","last_fullsized":"2024-09-10T19:02:09.549255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:30.155089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.155089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038319109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.232858+0000","last_change":"2024-09-10T19:01:31.205233+0000","last_active":"2024-09-10T19:02:09.232858+0000","last_peered":"2024-09-10T19:02:09.232858+0000","last_clean":"2024-09-10T19:02:09.232858+0000","last_became_active":"2024-09-10T18:54:37.499139+0000","last_became_peered":"2024-09-10T18:54:37.499139+0000","last_unstale":"2024-09-10T19:02:09.232858+0000","last_undegraded":"2024-09-10T19:02:09.232858+0000","last_fullsized":"2024-09-10T19:02:09.232858+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.205158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.205158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.392310+0000","scrub_duration":0.0024743550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":538,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546158+0000","last_change":"2024-09-10T19:01:22.297477+0000","last_active":"2024-09-10T19:02:09.546158+0000","last_peered":"2024-09-10T19:02:09.546158+0000","last_clean":"2024-09-10T19:02:09.546158+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:02:09.546158+0000","last_undegraded":"2024-09-10T19:02:09.546158+0000","last_fullsized":"2024-09-10T19:02:09.546158+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:01:22.297269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.297269+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:28.277078+0000","scrub_duration":0.005830773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":551,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.236135+0000","last_change":"2024-09-10T18:59:59.421165+0000","last_active":"2024-09-10T19:02:09.236135+0000","last_peered":"2024-09-10T19:02:09.236135+0000","last_clean":"2024-09-10T19:02:09.236135+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:02:09.236135+0000","last_undegraded":"2024-09-10T19:02:09.236135+0000","last_fullsized":"2024-09-10T19:02:09.236135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T18:59:59.420998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:59:59.420998+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045326000000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"77'11","reported_seq":673,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.397203+0000","last_change":"2024-09-10T19:00:15.117567+0000","last_active":"2024-09-10T19:02:11.397203+0000","last_peered":"2024-09-10T19:02:11.397203+0000","last_clean":"2024-09-10T19:02:11.397203+0000","last_became_active":"2024-09-10T18:49:22.044250+0000","last_became_peered":"2024-09-10T18:49:22.044250+0000","last_unstale":"2024-09-10T19:02:11.397203+0000","last_undegraded":"2024-09-10T19:02:11.397203+0000","last_fullsized":"2024-09-10T19:02:11.397203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:15.117380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:15.117380+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004280982,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241908+0000","last_change":"2024-09-10T19:01:19.956970+0000","last_active":"2024-09-10T19:02:09.241908+0000","last_peered":"2024-09-10T19:02:09.241908+0000","last_clean":"2024-09-10T19:02:09.241908+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:02:09.241908+0000","last_undegraded":"2024-09-10T19:02:09.241908+0000","last_fullsized":"2024-09-10T19:02:09.241908+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.956889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.956889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.716004+0000","scrub_duration":0.0027515830000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":439,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545093+0000","last_change":"2024-09-10T19:01:43.827904+0000","last_active":"2024-09-10T19:02:09.545093+0000","last_peered":"2024-09-10T19:02:09.545093+0000","last_clean":"2024-09-10T19:02:09.545093+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:02:09.545093+0000","last_undegraded":"2024-09-10T19:02:09.545093+0000","last_fullsized":"2024-09-10T19:02:09.545093+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:01:43.827739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.827739+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:46.320818+0000","scrub_duration":0.0040040049999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":548,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542353+0000","last_change":"2024-09-10T19:01:47.273790+0000","last_active":"2024-09-10T19:02:09.542353+0000","last_peered":"2024-09-10T19:02:09.542353+0000","last_clean":"2024-09-10T19:02:09.542353+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:02:09.542353+0000","last_undegraded":"2024-09-10T19:02:09.542353+0000","last_fullsized":"2024-09-10T19:02:09.542353+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:01:47.273620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.273620+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:02.229650+0000","scrub_duration":0.0048496030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":531,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.238421+0000","last_change":"2024-09-10T19:01:39.125687+0000","last_active":"2024-09-10T19:02:09.238421+0000","last_peered":"2024-09-10T19:02:09.238421+0000","last_clean":"2024-09-10T19:02:09.238421+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:02:09.238421+0000","last_undegraded":"2024-09-10T19:02:09.238421+0000","last_fullsized":"2024-09-10T19:02:09.238421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.125534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.125534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.289329+0000","scrub_duration":0.0037788880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":496,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546077+0000","last_change":"2024-09-10T19:00:12.129140+0000","last_active":"2024-09-10T19:02:09.546077+0000","last_peered":"2024-09-10T19:02:09.546077+0000","last_clean":"2024-09-10T19:02:09.546077+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:02:09.546077+0000","last_undegraded":"2024-09-10T19:02:09.546077+0000","last_fullsized":"2024-09-10T19:02:09.546077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:12.128979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.128979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031738389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":515,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.240353+0000","last_change":"2024-09-10T19:01:36.187242+0000","last_active":"2024-09-10T19:02:09.240353+0000","last_peered":"2024-09-10T19:02:09.240353+0000","last_clean":"2024-09-10T19:02:09.240353+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:02:09.240353+0000","last_undegraded":"2024-09-10T19:02:09.240353+0000","last_fullsized":"2024-09-10T19:02:09.240353+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:36.187070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.187070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:43.659276+0000","scrub_duration":0.003983828,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.236462+0000","last_change":"2024-09-10T19:01:40.232693+0000","last_active":"2024-09-10T19:02:09.236462+0000","last_peered":"2024-09-10T19:02:09.236462+0000","last_clean":"2024-09-10T19:02:09.236462+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:02:09.236462+0000","last_undegraded":"2024-09-10T19:02:09.236462+0000","last_fullsized":"2024-09-10T19:02:09.236462+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:40.232583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.232583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:04.573489+0000","scrub_duration":0.0029621439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":521,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542180+0000","last_change":"2024-09-10T19:00:21.565543+0000","last_active":"2024-09-10T19:02:09.542180+0000","last_peered":"2024-09-10T19:02:09.542180+0000","last_clean":"2024-09-10T19:02:09.542180+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:02:09.542180+0000","last_undegraded":"2024-09-10T19:02:09.542180+0000","last_fullsized":"2024-09-10T19:02:09.542180+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:00:21.565373+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.565373+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037082009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":519,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545314+0000","last_change":"2024-09-10T19:01:54.789334+0000","last_active":"2024-09-10T19:02:09.545314+0000","last_peered":"2024-09-10T19:02:09.545314+0000","last_clean":"2024-09-10T19:02:09.545314+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:02:09.545314+0000","last_undegraded":"2024-09-10T19:02:09.545314+0000","last_fullsized":"2024-09-10T19:02:09.545314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:54.789186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:54.789186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:08.001040+0000","scrub_duration":0.0031670539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":499,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243418+0000","last_change":"2024-09-10T19:01:47.940008+0000","last_active":"2024-09-10T19:02:09.243418+0000","last_peered":"2024-09-10T19:02:09.243418+0000","last_clean":"2024-09-10T19:02:09.243418+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:02:09.243418+0000","last_undegraded":"2024-09-10T19:02:09.243418+0000","last_fullsized":"2024-09-10T19:02:09.243418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:47.939853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.939853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:17.397918+0000","scrub_duration":0.0034949289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544048+0000","last_change":"2024-09-10T19:00:23.089654+0000","last_active":"2024-09-10T19:02:09.544048+0000","last_peered":"2024-09-10T19:02:09.544048+0000","last_clean":"2024-09-10T19:02:09.544048+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:02:09.544048+0000","last_undegraded":"2024-09-10T19:02:09.544048+0000","last_fullsized":"2024-09-10T19:02:09.544048+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.089483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.089483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003188331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548146+0000","last_change":"2024-09-10T19:01:34.715113+0000","last_active":"2024-09-10T19:02:09.548146+0000","last_peered":"2024-09-10T19:02:09.548146+0000","last_clean":"2024-09-10T19:02:09.548146+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:02:09.548146+0000","last_undegraded":"2024-09-10T19:02:09.548146+0000","last_fullsized":"2024-09-10T19:02:09.548146+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:34.714947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.714947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:36.903905+0000","scrub_duration":0.0035091329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":543,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546079+0000","last_change":"2024-09-10T19:01:19.268405+0000","last_active":"2024-09-10T19:02:09.546079+0000","last_peered":"2024-09-10T19:02:09.546079+0000","last_clean":"2024-09-10T19:02:09.546079+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:02:09.546079+0000","last_undegraded":"2024-09-10T19:02:09.546079+0000","last_fullsized":"2024-09-10T19:02:09.546079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:01:19.268316+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.268316+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.325891+0000","scrub_duration":0.0078237800000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549769+0000","last_change":"2024-09-10T19:01:42.462844+0000","last_active":"2024-09-10T19:02:09.549769+0000","last_peered":"2024-09-10T19:02:09.549769+0000","last_clean":"2024-09-10T19:02:09.549769+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:02:09.549769+0000","last_undegraded":"2024-09-10T19:02:09.549769+0000","last_fullsized":"2024-09-10T19:02:09.549769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.462690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.462690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:54.927778+0000","scrub_duration":0.0032025000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":667,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.400228+0000","last_change":"2024-09-10T19:00:26.925255+0000","last_active":"2024-09-10T19:02:11.400228+0000","last_peered":"2024-09-10T19:02:11.400228+0000","last_clean":"2024-09-10T19:02:11.400228+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:02:11.400228+0000","last_undegraded":"2024-09-10T19:02:11.400228+0000","last_fullsized":"2024-09-10T19:02:11.400228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:26.925124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.925124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037103370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":494,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545911+0000","last_change":"2024-09-10T19:00:18.141379+0000","last_active":"2024-09-10T19:02:09.545911+0000","last_peered":"2024-09-10T19:02:09.545911+0000","last_clean":"2024-09-10T19:02:09.545911+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:02:09.545911+0000","last_undegraded":"2024-09-10T19:02:09.545911+0000","last_fullsized":"2024-09-10T19:02:09.545911+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.141228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.141228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003005354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545496+0000","last_change":"2024-09-10T19:00:33.116212+0000","last_active":"2024-09-10T19:02:09.545496+0000","last_peered":"2024-09-10T19:02:09.545496+0000","last_clean":"2024-09-10T19:02:09.545496+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:02:09.545496+0000","last_undegraded":"2024-09-10T19:02:09.545496+0000","last_fullsized":"2024-09-10T19:02:09.545496+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:00:33.116061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003119681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":523,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545097+0000","last_change":"2024-09-10T19:00:16.130225+0000","last_active":"2024-09-10T19:02:09.545097+0000","last_peered":"2024-09-10T19:02:09.545097+0000","last_clean":"2024-09-10T19:02:09.545097+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:02:09.545097+0000","last_undegraded":"2024-09-10T19:02:09.545097+0000","last_fullsized":"2024-09-10T19:02:09.545097+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:16.130067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.130067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042760100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546818+0000","last_change":"2024-09-10T19:01:44.606532+0000","last_active":"2024-09-10T19:02:09.546818+0000","last_peered":"2024-09-10T19:02:09.546818+0000","last_clean":"2024-09-10T19:02:09.546818+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:02:09.546818+0000","last_undegraded":"2024-09-10T19:02:09.546818+0000","last_fullsized":"2024-09-10T19:02:09.546818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.606395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:58.635913+0000","scrub_duration":0.0039478229999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":657,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.397081+0000","last_change":"2024-09-10T19:01:18.506277+0000","last_active":"2024-09-10T19:02:11.397081+0000","last_peered":"2024-09-10T19:02:11.397081+0000","last_clean":"2024-09-10T19:02:11.397081+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:02:11.397081+0000","last_undegraded":"2024-09-10T19:02:11.397081+0000","last_fullsized":"2024-09-10T19:02:11.397081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:01:18.506103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.506103+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.190637+0000","scrub_duration":0.0036564269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235858+0000","last_change":"2024-09-10T19:00:18.076688+0000","last_active":"2024-09-10T19:02:09.235858+0000","last_peered":"2024-09-10T19:02:09.235858+0000","last_clean":"2024-09-10T19:02:09.235858+0000","last_became_active":"2024-09-10T18:49:24.046024+0000","last_became_peered":"2024-09-10T18:49:24.046024+0000","last_unstale":"2024-09-10T19:02:09.235858+0000","last_undegraded":"2024-09-10T19:02:09.235858+0000","last_fullsized":"2024-09-10T19:02:09.235858+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:18.076465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:18.076465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043459409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545682+0000","last_change":"2024-09-10T19:01:37.571899+0000","last_active":"2024-09-10T19:02:09.545682+0000","last_peered":"2024-09-10T19:02:09.545682+0000","last_clean":"2024-09-10T19:02:09.545682+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:02:09.545682+0000","last_undegraded":"2024-09-10T19:02:09.545682+0000","last_fullsized":"2024-09-10T19:02:09.545682+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.571747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:07.105731+0000","scrub_duration":0.0035003059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":532,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544487+0000","last_change":"2024-09-10T19:00:32.173178+0000","last_active":"2024-09-10T19:02:09.544487+0000","last_peered":"2024-09-10T19:02:09.544487+0000","last_clean":"2024-09-10T19:02:09.544487+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:02:09.544487+0000","last_undegraded":"2024-09-10T19:02:09.544487+0000","last_fullsized":"2024-09-10T19:02:09.544487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:32.173009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:32.173009+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042085289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546537+0000","last_change":"2024-09-10T19:01:31.660345+0000","last_active":"2024-09-10T19:02:09.546537+0000","last_peered":"2024-09-10T19:02:09.546537+0000","last_clean":"2024-09-10T19:02:09.546537+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:02:09.546537+0000","last_undegraded":"2024-09-10T19:02:09.546537+0000","last_fullsized":"2024-09-10T19:02:09.546537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.660098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.660098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:45.200581+0000","scrub_duration":0.0038432570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.549018+0000","last_change":"2024-09-10T19:01:46.395332+0000","last_active":"2024-09-10T19:02:09.549018+0000","last_peered":"2024-09-10T19:02:09.549018+0000","last_clean":"2024-09-10T19:02:09.549018+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:02:09.549018+0000","last_undegraded":"2024-09-10T19:02:09.549018+0000","last_fullsized":"2024-09-10T19:02:09.549018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:01:46.395225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:12.736915+0000","scrub_duration":0.0035309550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":507,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546300+0000","last_change":"2024-09-10T19:01:55.601470+0000","last_active":"2024-09-10T19:02:09.546300+0000","last_peered":"2024-09-10T19:02:09.546300+0000","last_clean":"2024-09-10T19:02:09.546300+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:02:09.546300+0000","last_undegraded":"2024-09-10T19:02:09.546300+0000","last_fullsized":"2024-09-10T19:02:09.546300+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.601279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:16.533046+0000","scrub_duration":0.0034556629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234057+0000","last_change":"2024-09-10T19:01:27.192516+0000","last_active":"2024-09-10T19:02:09.234057+0000","last_peered":"2024-09-10T19:02:09.234057+0000","last_clean":"2024-09-10T19:02:09.234057+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:02:09.234057+0000","last_undegraded":"2024-09-10T19:02:09.234057+0000","last_fullsized":"2024-09-10T19:02:09.234057+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:27.192364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.192364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:41.552810+0000","scrub_duration":0.0032891259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":545,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234927+0000","last_change":"2024-09-10T19:00:13.188540+0000","last_active":"2024-09-10T19:02:09.234927+0000","last_peered":"2024-09-10T19:02:09.234927+0000","last_clean":"2024-09-10T19:02:09.234927+0000","last_became_active":"2024-09-10T18:49:19.629118+0000","last_became_peered":"2024-09-10T18:49:19.629118+0000","last_unstale":"2024-09-10T19:02:09.234927+0000","last_undegraded":"2024-09-10T19:02:09.234927+0000","last_fullsized":"2024-09-10T19:02:09.234927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:00:13.188382+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:00:13.188382+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046400579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":510,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548250+0000","last_change":"2024-09-10T18:58:58.776508+0000","last_active":"2024-09-10T19:02:09.548250+0000","last_peered":"2024-09-10T19:02:09.548250+0000","last_clean":"2024-09-10T19:02:09.548250+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:02:09.548250+0000","last_undegraded":"2024-09-10T19:02:09.548250+0000","last_fullsized":"2024-09-10T19:02:09.548250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:58:58.776325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.776325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043627020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":508,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234773+0000","last_change":"2024-09-10T19:01:44.276771+0000","last_active":"2024-09-10T19:02:09.234773+0000","last_peered":"2024-09-10T19:02:09.234773+0000","last_clean":"2024-09-10T19:02:09.234773+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:02:09.234773+0000","last_undegraded":"2024-09-10T19:02:09.234773+0000","last_fullsized":"2024-09-10T19:02:09.234773+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.276607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.276607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:09.976319+0000","scrub_duration":0.0037398430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":500,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241168+0000","last_change":"2024-09-10T18:59:55.150083+0000","last_active":"2024-09-10T19:02:09.241168+0000","last_peered":"2024-09-10T19:02:09.241168+0000","last_clean":"2024-09-10T19:02:09.241168+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:02:09.241168+0000","last_undegraded":"2024-09-10T19:02:09.241168+0000","last_fullsized":"2024-09-10T19:02:09.241168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:55.150009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T18:59:55.150009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024462989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":427,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234729+0000","last_change":"2024-09-10T19:01:26.151595+0000","last_active":"2024-09-10T19:02:09.234729+0000","last_peered":"2024-09-10T19:02:09.234729+0000","last_clean":"2024-09-10T19:02:09.234729+0000","last_became_active":"2024-09-10T18:54:37.591777+0000","last_became_peered":"2024-09-10T18:54:37.591777+0000","last_unstale":"2024-09-10T19:02:09.234729+0000","last_undegraded":"2024-09-10T19:02:09.234729+0000","last_fullsized":"2024-09-10T19:02:09.234729+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:26.151456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.151456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:48.172643+0000","scrub_duration":0.0036896630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":519,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546631+0000","last_change":"2024-09-10T19:01:24.560200+0000","last_active":"2024-09-10T19:02:09.546631+0000","last_peered":"2024-09-10T19:02:09.546631+0000","last_clean":"2024-09-10T19:02:09.546631+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:02:09.546631+0000","last_undegraded":"2024-09-10T19:02:09.546631+0000","last_fullsized":"2024-09-10T19:02:09.546631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:01:24.560030+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T19:01:24.560030+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:25.538132+0000","scrub_duration":0.0049336370000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.550156+0000","last_change":"2024-09-10T19:01:17.509364+0000","last_active":"2024-09-10T19:02:09.550156+0000","last_peered":"2024-09-10T19:02:09.550156+0000","last_clean":"2024-09-10T19:02:09.550156+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:02:09.550156+0000","last_undegraded":"2024-09-10T19:02:09.550156+0000","last_fullsized":"2024-09-10T19:02:09.550156+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:17.509179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:01:17.509179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:33.422311+0000","scrub_duration":0.0034198829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":492,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.239562+0000","last_change":"2024-09-10T18:59:53.179273+0000","last_active":"2024-09-10T19:02:09.239562+0000","last_peered":"2024-09-10T19:02:09.239562+0000","last_clean":"2024-09-10T19:02:09.239562+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:02:09.239562+0000","last_undegraded":"2024-09-10T19:02:09.239562+0000","last_fullsized":"2024-09-10T19:02:09.239562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:53.179202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T18:59:53.179202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015188129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542491+0000","last_change":"2024-09-10T19:00:08.102047+0000","last_active":"2024-09-10T19:02:09.542491+0000","last_peered":"2024-09-10T19:02:09.542491+0000","last_clean":"2024-09-10T19:02:09.542491+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:02:09.542491+0000","last_undegraded":"2024-09-10T19:02:09.542491+0000","last_fullsized":"2024-09-10T19:02:09.542491+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:08.101901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:00:08.101901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028849869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.542928+0000","last_change":"2024-09-10T19:01:52.787451+0000","last_active":"2024-09-10T19:02:09.542928+0000","last_peered":"2024-09-10T19:02:09.542928+0000","last_clean":"2024-09-10T19:02:09.542928+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:02:09.542928+0000","last_undegraded":"2024-09-10T19:02:09.542928+0000","last_fullsized":"2024-09-10T19:02:09.542928+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:52.787380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.787380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:08.272688+0000","scrub_duration":0.0034234759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":529,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.550035+0000","last_change":"2024-09-10T19:01:35.373516+0000","last_active":"2024-09-10T19:02:09.550035+0000","last_peered":"2024-09-10T19:02:09.550035+0000","last_clean":"2024-09-10T19:02:09.550035+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:02:09.550035+0000","last_undegraded":"2024-09-10T19:02:09.550035+0000","last_fullsized":"2024-09-10T19:02:09.550035+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:01:35.373263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.373263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:56.279384+0000","scrub_duration":0.0048975939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":509,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.548395+0000","last_change":"2024-09-10T19:00:11.386471+0000","last_active":"2024-09-10T19:02:09.548395+0000","last_peered":"2024-09-10T19:02:09.548395+0000","last_clean":"2024-09-10T19:02:09.548395+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:02:09.548395+0000","last_undegraded":"2024-09-10T19:02:09.548395+0000","last_fullsized":"2024-09-10T19:02:09.548395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:11.386293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:11.386293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033555310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":489,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241262+0000","last_change":"2024-09-10T18:59:01.274400+0000","last_active":"2024-09-10T19:02:09.241262+0000","last_peered":"2024-09-10T19:02:09.241262+0000","last_clean":"2024-09-10T19:02:09.241262+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:02:09.241262+0000","last_undegraded":"2024-09-10T19:02:09.241262+0000","last_fullsized":"2024-09-10T19:02:09.241262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:01.274336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T18:59:01.274336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019104549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":497,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242068+0000","last_change":"2024-09-10T19:00:23.839038+0000","last_active":"2024-09-10T19:02:09.242068+0000","last_peered":"2024-09-10T19:02:09.242068+0000","last_clean":"2024-09-10T19:02:09.242068+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:02:09.242068+0000","last_undegraded":"2024-09-10T19:02:09.242068+0000","last_fullsized":"2024-09-10T19:02:09.242068+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.838880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.838880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033929009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":415,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.544869+0000","last_change":"2024-09-10T19:01:22.572687+0000","last_active":"2024-09-10T19:02:09.544869+0000","last_peered":"2024-09-10T19:02:09.544869+0000","last_clean":"2024-09-10T19:02:09.544869+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:02:09.544869+0000","last_undegraded":"2024-09-10T19:02:09.544869+0000","last_fullsized":"2024-09-10T19:02:09.544869+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.572609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.572609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:39.057857+0000","scrub_duration":0.0029042989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":500,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543972+0000","last_change":"2024-09-10T19:01:32.170266+0000","last_active":"2024-09-10T19:02:09.543972+0000","last_peered":"2024-09-10T19:02:09.543972+0000","last_clean":"2024-09-10T19:02:09.543972+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:02:09.543972+0000","last_undegraded":"2024-09-10T19:02:09.543972+0000","last_fullsized":"2024-09-10T19:02:09.543972+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T18:58:58.248227+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T18:58:58.248227+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043299469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":568,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.543434+0000","last_change":"2024-09-10T19:01:50.763426+0000","last_active":"2024-09-10T19:02:09.543434+0000","last_peered":"2024-09-10T19:02:09.543434+0000","last_clean":"2024-09-10T19:02:09.543434+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:02:09.543434+0000","last_undegraded":"2024-09-10T19:02:09.543434+0000","last_fullsized":"2024-09-10T19:02:09.543434+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:01:50.763213+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:01:50.763213+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:53.149200+0000","scrub_duration":0.0049851260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.541224+0000","last_change":"2024-09-10T19:00:27.159256+0000","last_active":"2024-09-10T19:02:09.541224+0000","last_peered":"2024-09-10T19:02:09.541224+0000","last_clean":"2024-09-10T19:02:09.541224+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:02:09.541224+0000","last_undegraded":"2024-09-10T19:02:09.541224+0000","last_fullsized":"2024-09-10T19:02:09.541224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:27.159007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:27.159007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031545399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":496,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.545890+0000","last_change":"2024-09-10T19:00:23.207215+0000","last_active":"2024-09-10T19:02:09.545890+0000","last_peered":"2024-09-10T19:02:09.545890+0000","last_clean":"2024-09-10T19:02:09.545890+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:02:09.545890+0000","last_undegraded":"2024-09-10T19:02:09.545890+0000","last_fullsized":"2024-09-10T19:02:09.545890+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.207050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061363199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547751+0000","last_change":"2024-09-10T19:01:44.826152+0000","last_active":"2024-09-10T19:02:09.547751+0000","last_peered":"2024-09-10T19:02:09.547751+0000","last_clean":"2024-09-10T19:02:09.547751+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:02:09.547751+0000","last_undegraded":"2024-09-10T19:02:09.547751+0000","last_fullsized":"2024-09-10T19:02:09.547751+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.825961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.825961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:07.942188+0000","scrub_duration":0.0032122380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":504,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547652+0000","last_change":"2024-09-10T19:00:09.572036+0000","last_active":"2024-09-10T19:02:09.547652+0000","last_peered":"2024-09-10T19:02:09.547652+0000","last_clean":"2024-09-10T19:02:09.547652+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:02:09.547652+0000","last_undegraded":"2024-09-10T19:02:09.547652+0000","last_fullsized":"2024-09-10T19:02:09.547652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:09.571847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:00:09.571847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034123030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":526,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.240354+0000","last_change":"2024-09-10T19:01:36.833738+0000","last_active":"2024-09-10T19:02:09.240354+0000","last_peered":"2024-09-10T19:02:09.240354+0000","last_clean":"2024-09-10T19:02:09.240354+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:02:09.240354+0000","last_undegraded":"2024-09-10T19:02:09.240354+0000","last_fullsized":"2024-09-10T19:02:09.240354+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:01:36.833597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.833597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:39.866708+0000","scrub_duration":0.003904022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.241836+0000","last_change":"2024-09-10T19:01:22.937789+0000","last_active":"2024-09-10T19:02:09.241836+0000","last_peered":"2024-09-10T19:02:09.241836+0000","last_clean":"2024-09-10T19:02:09.241836+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:02:09.241836+0000","last_undegraded":"2024-09-10T19:02:09.241836+0000","last_fullsized":"2024-09-10T19:02:09.241836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.937630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.937630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:42.259214+0000","scrub_duration":0.003520343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":687,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:11.397377+0000","last_change":"2024-09-10T19:00:25.684751+0000","last_active":"2024-09-10T19:02:11.397377+0000","last_peered":"2024-09-10T19:02:11.397377+0000","last_clean":"2024-09-10T19:02:11.397377+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:02:11.397377+0000","last_undegraded":"2024-09-10T19:02:11.397377+0000","last_fullsized":"2024-09-10T19:02:11.397377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:00:25.684577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:25.684577+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038437710000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547948+0000","last_change":"2024-09-10T19:01:30.706798+0000","last_active":"2024-09-10T19:02:09.547948+0000","last_peered":"2024-09-10T19:02:09.547948+0000","last_clean":"2024-09-10T19:02:09.547948+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:02:09.547948+0000","last_undegraded":"2024-09-10T19:02:09.547948+0000","last_fullsized":"2024-09-10T19:02:09.547948+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.706647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.706647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:54.500476+0000","scrub_duration":0.003580083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":506,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.235733+0000","last_change":"2024-09-10T19:00:21.100228+0000","last_active":"2024-09-10T19:02:09.235733+0000","last_peered":"2024-09-10T19:02:09.235733+0000","last_clean":"2024-09-10T19:02:09.235733+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:02:09.235733+0000","last_undegraded":"2024-09-10T19:02:09.235733+0000","last_fullsized":"2024-09-10T19:02:09.235733+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:21.100058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:00:21.100058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034414139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":518,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546678+0000","last_change":"2024-09-10T19:01:36.753924+0000","last_active":"2024-09-10T19:02:09.546678+0000","last_peered":"2024-09-10T19:02:09.546678+0000","last_clean":"2024-09-10T19:02:09.546678+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:02:09.546678+0000","last_undegraded":"2024-09-10T19:02:09.546678+0000","last_fullsized":"2024-09-10T19:02:09.546678+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:01:36.753746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.753746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:00.576476+0000","scrub_duration":0.0044503609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":511,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.546596+0000","last_change":"2024-09-10T19:00:02.052175+0000","last_active":"2024-09-10T19:02:09.546596+0000","last_peered":"2024-09-10T19:02:09.546596+0000","last_clean":"2024-09-10T19:02:09.546596+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:02:09.546596+0000","last_undegraded":"2024-09-10T19:02:09.546596+0000","last_fullsized":"2024-09-10T19:02:09.546596+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:00:02.051995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:02.051995+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042242440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":527,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.240925+0000","last_change":"2024-09-10T19:01:35.187804+0000","last_active":"2024-09-10T19:02:09.240925+0000","last_peered":"2024-09-10T19:02:09.240925+0000","last_clean":"2024-09-10T19:02:09.240925+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:02:09.240925+0000","last_undegraded":"2024-09-10T19:02:09.240925+0000","last_fullsized":"2024-09-10T19:02:09.240925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.187688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.187688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:47.009815+0000","scrub_duration":0.0058675740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":423,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.550009+0000","last_change":"2024-09-10T19:01:36.393078+0000","last_active":"2024-09-10T19:02:09.550009+0000","last_peered":"2024-09-10T19:02:09.550009+0000","last_clean":"2024-09-10T19:02:09.550009+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:02:09.550009+0000","last_undegraded":"2024-09-10T19:02:09.550009+0000","last_fullsized":"2024-09-10T19:02:09.550009+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:36.392815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.392815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:44.308063+0000","scrub_duration":0.0027879290000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":497,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.243331+0000","last_change":"2024-09-10T19:00:16.862735+0000","last_active":"2024-09-10T19:02:09.243331+0000","last_peered":"2024-09-10T19:02:09.243331+0000","last_clean":"2024-09-10T19:02:09.243331+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:02:09.243331+0000","last_undegraded":"2024-09-10T19:02:09.243331+0000","last_fullsized":"2024-09-10T19:02:09.243331+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:16.862586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:00:16.862586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011943657999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":503,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242316+0000","last_change":"2024-09-10T19:00:17.847339+0000","last_active":"2024-09-10T19:02:09.242316+0000","last_peered":"2024-09-10T19:02:09.242316+0000","last_clean":"2024-09-10T19:02:09.242316+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:02:09.242316+0000","last_undegraded":"2024-09-10T19:02:09.242316+0000","last_fullsized":"2024-09-10T19:02:09.242316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:17.847185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:00:17.847185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036286880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":535,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.234303+0000","last_change":"2024-09-10T19:00:14.152011+0000","last_active":"2024-09-10T19:02:09.234303+0000","last_peered":"2024-09-10T19:02:09.234303+0000","last_clean":"2024-09-10T19:02:09.234303+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:02:09.234303+0000","last_undegraded":"2024-09-10T19:02:09.234303+0000","last_fullsized":"2024-09-10T19:02:09.234303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:00:14.151862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.151862+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041793289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":499,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.242500+0000","last_change":"2024-09-10T19:01:51.936609+0000","last_active":"2024-09-10T19:02:09.242500+0000","last_peered":"2024-09-10T19:02:09.242500+0000","last_clean":"2024-09-10T19:02:09.242500+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:02:09.242500+0000","last_undegraded":"2024-09-10T19:02:09.242500+0000","last_fullsized":"2024-09-10T19:02:09.242500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.936453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:06.566772+0000","scrub_duration":0.0036318919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":426,"reported_epoch":513,"state":"active+clean","last_fresh":"2024-09-10T19:02:09.547390+0000","last_change":"2024-09-10T19:01:37.783455+0000","last_active":"2024-09-10T19:02:09.547390+0000","last_peered":"2024-09-10T19:02:09.547390+0000","last_clean":"2024-09-10T19:02:09.547390+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:02:09.547390+0000","last_undegraded":"2024-09-10T19:02:09.547390+0000","last_fullsized":"2024-09-10T19:02:09.547390+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.783302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.783302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:02:51.804414+0000","scrub_duration":0.003418712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":223,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8195937,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":0,"num_read":118,"num_read_kb":12,"num_write":640,"num_write_kb":12724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26701824,"data_stored":24587859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1970,"ondisk_log_size":1970,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":752,"num_read_kb":747,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1130,"num_read_kb":887,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":115,"num_read_kb":223,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972519,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321752,"kb_used_data":16072,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93423400,"statfs":{"total":95995035648,"available":95665561600,"internally_reserved":0,"allocated":16457728,"data_stored":13044440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.89100000000000001,"15min":0.90100000000000002},"min":{"1min":0.45300000000000001,"5min":0.40300000000000002,"15min":0.311},"max":{"1min":1.2470000000000001,"5min":1.3839999999999999,"15min":1.9330000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.876,"5min":0.96799999999999997,"15min":0.93000000000000005},"min":{"1min":0.505,"5min":0.498,"15min":0.34699999999999998},"max":{"1min":1.2789999999999999,"5min":4.7000000000000002,"15min":4.7000000000000002},"last":1.0980000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89800000000000002,"15min":0.88500000000000001},"min":{"1min":0.58099999999999996,"5min":0.47299999999999998,"15min":0.41199999999999998},"max":{"1min":1.169,"5min":5.6539999999999999,"15min":5.6539999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.95199999999999996,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3620000000000001,"5min":1.421,"15min":1.8280000000000001},"last":0.95699999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.98599999999999999,"15min":1.0149999999999999},"min":{"1min":0.54400000000000004,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":1.2649999999999999,"5min":1.6739999999999999,"15min":3.4990000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.0229999999999999,"15min":0.99399999999999999},"min":{"1min":0.60999999999999999,"5min":0.443,"15min":0.443},"max":{"1min":1.351,"5min":4.1749999999999998,"15min":4.1749999999999998},"last":0.78100000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0649999999999999,"15min":1.0209999999999999},"min":{"1min":0.68000000000000005,"5min":0.65400000000000003,"15min":0.48299999999999998},"max":{"1min":1.3280000000000001,"5min":4.2080000000000002,"15min":4.2080000000000002},"last":1.026},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.98199999999999998,"15min":0.97499999999999998},"min":{"1min":0.67900000000000005,"5min":0.56200000000000006,"15min":0.46600000000000003},"max":{"1min":1.1619999999999999,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.999}]},{"osd":4,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.01,"15min":1.012},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.347,"5min":1.4279999999999999,"15min":2.1989999999999998},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.1060000000000001,"15min":1.0660000000000001},"min":{"1min":0.56499999999999995,"5min":0.49199999999999999,"15min":0.48099999999999998},"max":{"1min":1.3819999999999999,"5min":4.2400000000000002,"15min":4.2400000000000002},"last":1.149}]},{"osd":5,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.997,"15min":0.99099999999999999},"min":{"1min":0.50900000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.226,"5min":5.4710000000000001,"15min":5.4710000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.141,"15min":1.157},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.411,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":1.212}]},{"osd":6,"last update":"Tue Sep 10 19:01:54 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.1699999999999999,"15min":1.2},"min":{"1min":0.502,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.444,"5min":1.591,"15min":2.2749999999999999},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.232,"15min":1.2},"min":{"1min":0.63600000000000001,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.425,"5min":4.1310000000000002,"15min":4.1310000000000002},"last":1.1910000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299569,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313336,"kb_used_data":7656,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93431816,"statfs":{"total":95995035648,"available":95674179584,"internally_reserved":0,"allocated":7839744,"data_stored":4520512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.88500000000000001,"15min":0.88100000000000001},"min":{"1min":0.56000000000000005,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3600000000000001,"5min":2.032,"15min":2.0550000000000002},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.98099999999999998,"15min":0.95499999999999996},"min":{"1min":0.64000000000000001,"5min":0.40300000000000002,"15min":0.36399999999999999},"max":{"1min":1.266,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":0.89100000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.008,"15min":1.0349999999999999},"min":{"1min":0.56899999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.454,"5min":2.2410000000000001,"15min":18.494},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.93500000000000005,"15min":0.90700000000000003},"min":{"1min":0.65400000000000003,"5min":0.434,"15min":0.434},"max":{"1min":1.218,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":0.85599999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96499999999999997,"15min":0.97199999999999998},"min":{"1min":0.65100000000000002,"5min":0.47499999999999998,"15min":0.35299999999999998},"max":{"1min":1.2050000000000001,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.98899999999999999,"15min":0.94299999999999995},"min":{"1min":0.68600000000000005,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2150000000000001,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.01}]},{"osd":3,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.038,"15min":1.004},"min":{"1min":0.76600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.5209999999999999,"5min":2.234,"15min":2.278},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.0249999999999999,"15min":0.98599999999999999},"min":{"1min":0.64700000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.262,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":1.083}]},{"osd":4,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.012,"15min":0.95599999999999996},"min":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.52000000000000002},"max":{"1min":1.359,"5min":6.008,"15min":6.008},"last":1.123},{"interface":"front","average":{"1min":1.085,"5min":1.1259999999999999,"15min":1.0649999999999999},"min":{"1min":0.57299999999999995,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.444,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.151}]},{"osd":5,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.123,"15min":1.0489999999999999},"min":{"1min":0.77900000000000003,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":1.3169999999999999,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.135,"5min":1.1879999999999999,"15min":1.1319999999999999},"min":{"1min":0.72799999999999998,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.599,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":1.034}]},{"osd":7,"last update":"Tue Sep 10 19:01:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.101,"15min":1.0720000000000001},"min":{"1min":0.77300000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.292,"5min":2.069,"15min":2.069},"last":1.234},{"interface":"front","average":{"1min":1.173,"5min":1.286,"15min":1.196},"min":{"1min":0.98699999999999999,"5min":0.65500000000000003,"15min":0.63900000000000001},"max":{"1min":1.4910000000000001,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1839999999999999}]}]},{"osd":5,"up_from":43,"seq":184683593915,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313940,"kb_used_data":12420,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93431212,"statfs":{"total":95995035648,"available":95673561088,"internally_reserved":0,"allocated":12718080,"data_stored":9305486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.073,"15min":0.96299999999999997},"min":{"1min":0.70299999999999996,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":1.3680000000000001,"5min":8.3949999999999996,"15min":8.3949999999999996},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96799999999999997,"15min":0.91900000000000004},"min":{"1min":0.50600000000000001,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":1.2470000000000001,"5min":5.0519999999999996,"15min":7.3070000000000004},"last":0.75700000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.254,"15min":1.1539999999999999},"min":{"1min":0.63,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.516,"5min":15.824,"15min":31.548999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.999,"15min":1.081},"min":{"1min":0.71099999999999997,"5min":0.55100000000000005,"15min":0.48899999999999999},"max":{"1min":1.2450000000000001,"5min":5.0839999999999996,"15min":31.847000000000001},"last":1.0449999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.224,"15min":1.2450000000000001},"min":{"1min":0.65800000000000003,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.4159999999999999,"5min":20.536000000000001,"15min":61.347000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.069,"15min":1.0129999999999999},"min":{"1min":0.71599999999999997,"5min":0.60299999999999998,"15min":0.53200000000000003},"max":{"1min":1.6140000000000001,"5min":5.2489999999999997,"15min":6.1600000000000001},"last":0.81799999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.1180000000000001,"15min":1.0109999999999999},"min":{"1min":0.746,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.482,"5min":7.5979999999999999,"15min":7.5979999999999999},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.075,"5min":1.1819999999999999,"15min":1.2589999999999999},"min":{"1min":0.84799999999999998,"5min":0.66400000000000003,"15min":0.57399999999999995},"max":{"1min":1.304,"5min":7.7679999999999998,"15min":53.173000000000002},"last":1.163}]},{"osd":4,"last update":"Tue Sep 10 19:01:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.1499999999999999,"15min":0.97099999999999997},"min":{"1min":0.63500000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4399999999999999,"5min":13.005000000000001,"15min":13.005000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.1599999999999999,"15min":1.0349999999999999},"min":{"1min":0.72999999999999998,"5min":0.44900000000000001,"15min":0.40600000000000003},"max":{"1min":1.4690000000000001,"5min":7.6299999999999999,"15min":7.6299999999999999},"last":1.1140000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2090000000000001,"15min":1.161},"min":{"1min":0.67700000000000005,"5min":0.63400000000000001,"15min":0.49199999999999999},"max":{"1min":1.292,"5min":7.5380000000000003,"15min":7.7199999999999998},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.196,"15min":1.202},"min":{"1min":0.629,"5min":0.53000000000000003,"15min":0.47199999999999998},"max":{"1min":1.3879999999999999,"5min":5.1769999999999996,"15min":9.6760000000000002},"last":1.272}]},{"osd":7,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.325,"15min":1.2130000000000001},"min":{"1min":0.627,"5min":0.59699999999999998,"15min":0.43099999999999999},"max":{"1min":1.232,"5min":15.613,"15min":15.613},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.377,"15min":1.2649999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.54200000000000004},"max":{"1min":1.5800000000000001,"5min":16.965,"15min":16.965},"last":1.2989999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855559,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317500,"kb_used_data":7660,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93427652,"statfs":{"total":95995035648,"available":95669915648,"internally_reserved":0,"allocated":7843840,"data_stored":4521752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.86199999999999999,"15min":0.84999999999999998},"min":{"1min":0.53100000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.331,"5min":1.403,"15min":1.546},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.94299999999999995,"15min":0.90800000000000003},"min":{"1min":0.57999999999999996,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.325,"5min":1.6100000000000001,"15min":1.7070000000000001},"last":1.101}]},{"osd":1,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.1850000000000001,"15min":0.97499999999999998},"min":{"1min":0.61899999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.405,"5min":27.716000000000001,"15min":27.716000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.216,"15min":1.0149999999999999},"min":{"1min":0.65400000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.446,"5min":27.93,"15min":27.93},"last":0.79000000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.94599999999999995,"15min":0.93899999999999995},"min":{"1min":0.67500000000000004,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.321,"5min":1.5209999999999999,"15min":1.5509999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.95499999999999996,"15min":0.94199999999999995},"min":{"1min":0.60799999999999998,"5min":0.57499999999999996,"15min":0.497},"max":{"1min":1.3759999999999999,"5min":1.482,"15min":1.7569999999999999},"last":1.2050000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:01:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.014,"15min":0.96599999999999997},"min":{"1min":0.70099999999999996,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.3160000000000001,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.97999999999999998,"15min":0.96499999999999997},"min":{"1min":0.626,"5min":0.442,"15min":0.442},"max":{"1min":1.258,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.0269999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.99199999999999999,"15min":0.95299999999999996},"min":{"1min":0.60599999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.4359999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.089,"15min":1.0289999999999999},"min":{"1min":0.78000000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.355,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.355}]},{"osd":6,"last update":"Tue Sep 10 19:02:10 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.0589999999999999,"15min":1.0209999999999999},"min":{"1min":0.89000000000000001,"5min":0.53900000000000003,"15min":0.501},"max":{"1min":1.764,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.204,"15min":1.1419999999999999},"min":{"1min":0.91200000000000003,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.446,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.044}]},{"osd":7,"last update":"Tue Sep 10 19:02:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.167,"15min":1.157},"min":{"1min":0.68600000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.587,"5min":1.629,"15min":2.008},"last":1.7},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.167},"min":{"1min":0.80900000000000005,"5min":0.58499999999999996,"15min":0.58399999999999996},"max":{"1min":1.8999999999999999,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.97799999999999998}]}]},{"osd":3,"up_from":29,"seq":124554051794,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309888,"kb_used_data":8368,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93435264,"statfs":{"total":95995035648,"available":95677710336,"internally_reserved":0,"allocated":8568832,"data_stored":5112092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.755,"15min":0.82999999999999996},"min":{"1min":0.30199999999999999,"5min":0.29499999999999998,"15min":0.222},"max":{"1min":1.044,"5min":8.6180000000000003,"15min":26.594000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77000000000000002,"15min":0.89200000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.252},"max":{"1min":1.444,"5min":5.3479999999999999,"15min":24.204999999999998},"last":0.84899999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.76600000000000001,"15min":0.91000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":1.6200000000000001,"5min":5.3879999999999999,"15min":53.661999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.84599999999999997,"15min":0.88900000000000001},"min":{"1min":0.40200000000000002,"5min":0.39900000000000002,"15min":0.27900000000000003},"max":{"1min":2.181,"5min":4.0549999999999997,"15min":15.189},"last":0.88300000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86899999999999999,"15min":1.006},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.32200000000000001},"max":{"1min":1.3380000000000001,"5min":5.1920000000000002,"15min":26.558},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.83999999999999997,"15min":0.94799999999999995},"min":{"1min":0.32600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.4740000000000002,"5min":4.4740000000000002,"15min":25.536999999999999},"last":0.77100000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:01:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.1699999999999999,"15min":1.153},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.46999999999999997},"max":{"1min":1.663,"5min":7.3550000000000004,"15min":10.359999999999999},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2709999999999999,"15min":1.2470000000000001},"min":{"1min":0.81399999999999995,"5min":0.67000000000000004,"15min":0.53500000000000003},"max":{"1min":1.73,"5min":7.4189999999999996,"15min":10.391},"last":1.3280000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.1040000000000001,"15min":1.218},"min":{"1min":0.62,"5min":0.62,"15min":0.501},"max":{"1min":1.544,"5min":1.8009999999999999,"15min":17.535},"last":1.2689999999999999},{"interface":"front","average":{"1min":1,"5min":1.048,"15min":1.052},"min":{"1min":0.63500000000000001,"5min":0.60399999999999998,"15min":0.45900000000000002},"max":{"1min":1.3240000000000001,"5min":4.6619999999999999,"15min":8.7880000000000003},"last":1.0089999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:02:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1359999999999999,"15min":1.1930000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.55600000000000005},"max":{"1min":1.5600000000000001,"5min":1.9550000000000001,"15min":10.51},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.65800000000000003,"5min":0.56999999999999995,"15min":0.48499999999999999},"max":{"1min":1.405,"5min":2.996,"15min":10.773999999999999},"last":1.2310000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:01:59 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.101,"15min":1.171},"min":{"1min":0.68300000000000005,"5min":0.59899999999999998,"15min":0.45600000000000002},"max":{"1min":1.3799999999999999,"5min":4.0229999999999997,"15min":18.312999999999999},"last":1.095},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.27,"15min":1.3149999999999999},"min":{"1min":0.79300000000000004,"5min":0.70099999999999996,"15min":0.57299999999999995},"max":{"1min":2.0299999999999998,"5min":4.5350000000000001,"15min":10.611000000000001},"last":1.369}]}]},{"osd":2,"up_from":19,"seq":81604378843,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325840,"kb_used_data":16000,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93419312,"statfs":{"total":95995035648,"available":95661375488,"internally_reserved":0,"allocated":16384000,"data_stored":12978824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68300000000000005,"15min":0.73999999999999999},"min":{"1min":0.497,"5min":0.219,"15min":0.219},"max":{"1min":1.034,"5min":1.5629999999999999,"15min":9.516},"last":0.628},{"interface":"front","average":{"1min":0.754,"5min":0.746,"15min":0.78600000000000003},"min":{"1min":0.317,"5min":0.30299999999999999,"15min":0.23599999999999999},"max":{"1min":1.125,"5min":1.2350000000000001,"15min":15.577999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Tue Sep 10 19:01:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.747,"15min":0.72499999999999998},"min":{"1min":0.42799999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.014,"5min":4.2380000000000004,"15min":5.516},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.78200000000000003,"15min":0.82599999999999996},"min":{"1min":0.47099999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.121,"5min":1.26,"15min":9.0649999999999995},"last":0.86699999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.83199999999999996,"15min":0.872},"min":{"1min":0.47699999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":5.1479999999999997},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.82099999999999995,"15min":0.85399999999999998},"min":{"1min":0.45900000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.262,"5min":2.0539999999999998,"15min":15.736000000000001},"last":0.83899999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:01:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.052,"15min":1.081},"min":{"1min":0.80200000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.5469999999999999,"5min":3.851,"15min":21.571999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.2110000000000001,"15min":1.161},"min":{"1min":0.79200000000000004,"5min":0.55100000000000005,"15min":0.46000000000000002},"max":{"1min":1.5289999999999999,"5min":10.881,"15min":10.881},"last":1.0189999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.071,"15min":1.093},"min":{"1min":0.76100000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4179999999999999,"5min":1.8340000000000001,"15min":5.1959999999999997},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.085,"15min":1.1220000000000001},"min":{"1min":0.59999999999999998,"5min":0.46200000000000002,"15min":0.45400000000000001},"max":{"1min":1.2869999999999999,"5min":3.4980000000000002,"15min":21.654},"last":0.98999999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1140000000000001,"15min":1.1619999999999999},"min":{"1min":0.66600000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":9.0389999999999997},"last":1.085},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.0880000000000001,"15min":1.139},"min":{"1min":0.75800000000000001,"5min":0.57899999999999996,"15min":0.505},"max":{"1min":1.8100000000000001,"5min":3.367,"15min":15.971},"last":1.242}]},{"osd":7,"last update":"Tue Sep 10 19:02:15 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.127,"15min":1.179},"min":{"1min":0.72299999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.7150000000000001,"5min":1.905,"15min":15.025},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.212,"15min":1.206},"min":{"1min":0.80200000000000005,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.581,"5min":6.851,"15min":14.816000000000001},"last":1.1779999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317276,"kb_used_data":11596,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93427876,"statfs":{"total":95995035648,"available":95670145024,"internally_reserved":0,"allocated":11874304,"data_stored":8520953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:01:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77200000000000002,"15min":0.82699999999999996},"min":{"1min":0.32700000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.089,"5min":5.0439999999999996,"15min":32.677999999999997},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.84399999999999997,"15min":0.90000000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.253},"max":{"1min":1.2130000000000001,"5min":3.8250000000000002,"15min":32.398000000000003},"last":0.64500000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:01:42 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.85399999999999998,"15min":0.89100000000000001},"min":{"1min":0.38400000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0489999999999999,"5min":5.7999999999999998,"15min":26.097000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.90200000000000002,"15min":1.002},"min":{"1min":0.39900000000000002,"5min":0.29999999999999999,"15min":0.26000000000000001},"max":{"1min":1.264,"5min":4.0890000000000004,"15min":32.363},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:01:30 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.999,"15min":1.018},"min":{"1min":0.59199999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.8359999999999999,"5min":3.7360000000000002,"15min":25.704000000000001},"last":0.751},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.90700000000000003,"15min":1.0489999999999999},"min":{"1min":0.439,"5min":0.39500000000000002,"15min":0.26600000000000001},"max":{"1min":3.8029999999999999,"5min":6.0099999999999998,"15min":60.999000000000002},"last":0.84899999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1930000000000001,"15min":1.123},"min":{"1min":0.67300000000000004,"5min":0.47399999999999998,"15min":0.40899999999999997},"max":{"1min":1.6719999999999999,"5min":3.5379999999999998,"15min":12.808999999999999},"last":1.056},{"interface":"front","average":{"1min":1.181,"5min":1.1599999999999999,"15min":1.2410000000000001},"min":{"1min":0.64100000000000001,"5min":0.45400000000000001,"15min":0.378},"max":{"1min":3.6779999999999999,"5min":6.5430000000000001,"15min":60.529000000000003},"last":1.163}]},{"osd":5,"last update":"Tue Sep 10 19:02:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.1180000000000001,"15min":1.131},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.38},"max":{"1min":1.5,"5min":5.2439999999999998,"15min":26.050999999999998},"last":1.022},{"interface":"front","average":{"1min":1.169,"5min":1.2070000000000001,"15min":1.2150000000000001},"min":{"1min":0.71199999999999997,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.593,"5min":2.7949999999999999,"15min":12.84},"last":1.133}]},{"osd":6,"last update":"Tue Sep 10 19:02:08 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.27,"15min":1.3029999999999999},"min":{"1min":0.81200000000000006,"5min":0.59399999999999997,"15min":0.53800000000000003},"max":{"1min":1.782,"5min":3.121,"15min":25.608000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.097,"5min":1.24,"15min":1.2529999999999999},"min":{"1min":0.68999999999999995,"5min":0.63300000000000001,"15min":0.41499999999999998},"max":{"1min":1.5580000000000001,"5min":6.1219999999999999,"15min":33.643000000000001},"last":0.98499999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:02:08 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.3300000000000001,"15min":1.339},"min":{"1min":0.78100000000000003,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.881,"5min":3.0880000000000001,"15min":25.565999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.2769999999999999,"15min":1.3200000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.48399999999999999},"max":{"1min":1.8280000000000001,"5min":6.2599999999999998,"15min":33.673999999999999},"last":1.081}]}]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313368,"kb_used_data":7688,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93431784,"statfs":{"total":95995035648,"available":95674146816,"internally_reserved":0,"allocated":7872512,"data_stored":4520471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:01:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73899999999999999,"15min":1.004},"min":{"1min":0.54200000000000004,"5min":0.32400000000000001,"15min":0.26300000000000001},"max":{"1min":0.90800000000000003,"5min":6.758,"15min":80.710999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.82699999999999996,"15min":1.1040000000000001},"min":{"1min":0.50600000000000001,"5min":0.44600000000000001,"15min":0.25600000000000001},"max":{"1min":1.081,"5min":3.6850000000000001,"15min":86.009},"last":0.79500000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:01:36 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.80500000000000005,"15min":1.0389999999999999},"min":{"1min":0.38200000000000001,"5min":0.32900000000000001,"15min":0.216},"max":{"1min":1.2829999999999999,"5min":3.427,"15min":36.682000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.873,"5min":0.83699999999999997,"15min":0.93899999999999995},"min":{"1min":0.44800000000000001,"5min":0.442,"15min":0.23599999999999999},"max":{"1min":1.45,"5min":3.2090000000000001,"15min":24.614000000000001},"last":0.83199999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:01:36 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.93300000000000005,"15min":0.93000000000000005},"min":{"1min":0.57899999999999996,"5min":0.38500000000000001,"15min":0.26400000000000001},"max":{"1min":1.1519999999999999,"5min":4.4749999999999996,"15min":21.620999999999999},"last":0.629},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.88200000000000001,"15min":1.052},"min":{"1min":0.42599999999999999,"5min":0.39800000000000002,"15min":0.30299999999999999},"max":{"1min":1.117,"5min":2.012,"15min":45.765000000000001},"last":0.95599999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:01:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.095,"15min":1.0609999999999999},"min":{"1min":0.871,"5min":0.64300000000000002,"15min":0.5},"max":{"1min":1.4099999999999999,"5min":3.3279999999999998,"15min":6.3630000000000004},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.099,"5min":1.1200000000000001,"15min":1.111},"min":{"1min":0.68400000000000005,"5min":0.61399999999999999,"15min":0.51100000000000001},"max":{"1min":1.5089999999999999,"5min":3.419,"15min":4.2389999999999999},"last":1.133}]},{"osd":5,"last update":"Tue Sep 10 19:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.105,"15min":1.137},"min":{"1min":0.83699999999999997,"5min":0.67800000000000005,"15min":0.442},"max":{"1min":1.617,"5min":2.609,"15min":10.048},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.109,"15min":1.1100000000000001},"min":{"1min":0.872,"5min":0.63,"15min":0.42999999999999999},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":3.8959999999999999},"last":1.0800000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:02:12 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.181,"15min":1.1519999999999999},"min":{"1min":0.78200000000000003,"5min":0.755,"15min":0.56899999999999995},"max":{"1min":1.899,"5min":2.4550000000000001,"15min":3.9550000000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.173,"15min":1.2090000000000001},"min":{"1min":0.75,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.7909999999999999,"5min":2.4929999999999999,"15min":11.894},"last":1.0569999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:01:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1850000000000001,"15min":1.1759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.48499999999999999},"max":{"1min":1.5669999999999999,"5min":2.468,"15min":6.2949999999999999},"last":1.107},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2529999999999999,"15min":1.28},"min":{"1min":0.81599999999999995,"5min":0.78400000000000003,"15min":0.54900000000000004},"max":{"1min":1.782,"5min":3.3780000000000001,"15min":8.2840000000000007},"last":1.2270000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4000582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8486912,"data_stored":8195015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4501504,"data_stored":4195041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8478720,"data_stored":8194976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":223,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:02:20.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-10T19:02:20.855 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:20.888 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:20.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:20.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:21.020 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:21.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:21.158 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:21.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:21.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:21.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[18629]: cluster 2024-09-10T19:02:20.176730+0000 mgr.y (mgr.24526) 525 : cluster [DBG] pgmap v791: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:21.342 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[18629]: cluster 2024-09-10T19:02:20.319701+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:02:21.342 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[18629]: audit 2024-09-10T19:02:20.324032+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.342 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[18629]: audit 2024-09-10T19:02:20.328086+0000 mon.a (mon.0) 2777 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.367 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:21.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:21.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[32381]: cluster 2024-09-10T19:02:20.176730+0000 mgr.y (mgr.24526) 525 : cluster [DBG] pgmap v791: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:21.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[32381]: cluster 2024-09-10T19:02:20.319701+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:02:21.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[32381]: audit 2024-09-10T19:02:20.324032+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:21 smithi083 bash[32381]: audit 2024-09-10T19:02:20.328086+0000 mon.a (mon.0) 2777 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:21.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:21.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:21.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:21 smithi165 bash[22591]: cluster 2024-09-10T19:02:20.176730+0000 mgr.y (mgr.24526) 525 : cluster [DBG] pgmap v791: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:02:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:21 smithi165 bash[22591]: cluster 2024-09-10T19:02:20.319701+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:02:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:21 smithi165 bash[22591]: audit 2024-09-10T19:02:20.324032+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:21 smithi165 bash[22591]: audit 2024-09-10T19:02:20.328086+0000 mon.a (mon.0) 2777 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:21.614 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:21.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:21.703 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:21.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:21.837 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:21.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:21.975 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:21.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:22.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:22.109 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:22.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:22.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:22.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:22.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:22.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:22.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: audit 2024-09-10T19:02:21.328938+0000 mon.a (mon.0) 2778 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:22.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: cluster 2024-09-10T19:02:21.329795+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:02:22.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: audit 2024-09-10T19:02:21.344623+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:22.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: audit 2024-09-10T19:02:21.348080+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.494 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: audit 2024-09-10T19:02:21.351956+0000 mon.a (mon.0) 2780 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18629]: cluster 2024-09-10T19:02:22.177617+0000 mgr.y (mgr.24526) 526 : cluster [DBG] pgmap v794: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: audit 2024-09-10T19:02:21.328938+0000 mon.a (mon.0) 2778 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: cluster 2024-09-10T19:02:21.329795+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: audit 2024-09-10T19:02:21.344623+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: audit 2024-09-10T19:02:21.348080+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: audit 2024-09-10T19:02:21.351956+0000 mon.a (mon.0) 2780 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.495 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[32381]: cluster 2024-09-10T19:02:22.177617+0000 mgr.y (mgr.24526) 526 : cluster [DBG] pgmap v794: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:02:22.513 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:22.604 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: audit 2024-09-10T19:02:21.328938+0000 mon.a (mon.0) 2778 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: cluster 2024-09-10T19:02:21.329795+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:02:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: audit 2024-09-10T19:02:21.344623+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: audit 2024-09-10T19:02:21.348080+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: audit 2024-09-10T19:02:21.351956+0000 mon.a (mon.0) 2780 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:22.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:22 smithi165 bash[22591]: cluster 2024-09-10T19:02:22.177617+0000 mgr.y (mgr.24526) 526 : cluster [DBG] pgmap v794: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:02:22.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:22.645 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:22.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:22.781 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:22.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:22] "GET /metrics HTTP/1.1" 200 228872 "" "Prometheus/2.43.0" 2024-09-10T19:02:22.885 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:22.920 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:22.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:23.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:23.057 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:23.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:23.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:23.330 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:23.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:23.467 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: cluster 2024-09-10T19:02:21.503673+0000 osd.0 (osd.0) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: cluster 2024-09-10T19:02:21.504006+0000 osd.0 (osd.0) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: audit 2024-09-10T19:02:22.355887+0000 mon.a (mon.0) 2781 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: cluster 2024-09-10T19:02:22.356045+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: audit 2024-09-10T19:02:22.358946+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[32381]: audit 2024-09-10T19:02:22.361975+0000 mon.a (mon.0) 2783 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: cluster 2024-09-10T19:02:21.503673+0000 osd.0 (osd.0) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: cluster 2024-09-10T19:02:21.504006+0000 osd.0 (osd.0) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: audit 2024-09-10T19:02:22.355887+0000 mon.a (mon.0) 2781 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: cluster 2024-09-10T19:02:22.356045+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: audit 2024-09-10T19:02:22.358946+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:23 smithi083 bash[18629]: audit 2024-09-10T19:02:22.361975+0000 mon.a (mon.0) 2783 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:23.584 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: cluster 2024-09-10T19:02:21.503673+0000 osd.0 (osd.0) 277 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:23.584 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: cluster 2024-09-10T19:02:21.504006+0000 osd.0 (osd.0) 278 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:23.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: audit 2024-09-10T19:02:22.355887+0000 mon.a (mon.0) 2781 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:23.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: cluster 2024-09-10T19:02:22.356045+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:02:23.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: audit 2024-09-10T19:02:22.358946+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:23 smithi165 bash[22591]: audit 2024-09-10T19:02:22.361975+0000 mon.a (mon.0) 2783 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:23.605 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:23.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:23.741 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:23.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.808+0000 7ff4f7353640 1 -- 172.21.15.83:0/2267007788 >> v1:172.21.15.83:6790/0 conn(0x7ff4f0103a50 legacy=0x7ff4f0105e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:23.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- 172.21.15.83:0/2267007788 shutdown_connections 2024-09-10T19:02:23.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- 172.21.15.83:0/2267007788 >> 172.21.15.83:0/2267007788 conn(0x7ff4f0074a20 msgr2=0x7ff4f00f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:23.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- 172.21.15.83:0/2267007788 shutdown_connections 2024-09-10T19:02:23.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- 172.21.15.83:0/2267007788 wait complete. 2024-09-10T19:02:23.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 librados: starting msgr at 2024-09-10T19:02:23.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 librados: starting objecter 2024-09-10T19:02:23.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 Processor -- start 2024-09-10T19:02:23.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- start start 2024-09-10T19:02:23.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 librados: setting wanted keys 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 librados: calling monclient init 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4f0112e00 con 0x7ff4f0100c70 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4f0100300 con 0x7ff4f01075b0 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f7353640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4f01004f0 con 0x7ff4f0103a50 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f48ee640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff4f0103a50 0x7ff4f00fcf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44184/0 (socket says 172.21.15.83:44184) 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f48ee640 1 -- 172.21.15.83:0/1629362500 learned_addr learned my addr 172.21.15.83:0/1629362500 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4f50ef640 1 --1- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6789/0 conn(0x7ff4f0100c70 0x7ff4f00fc840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50138/0 (socket says 172.21.15.83:50138) 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff4f0100c70 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:23.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff4f0103a50 2024-09-10T19:02:23.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317142669 0 0) 0x7ff4f010b340 con 0x7ff4f0100c70 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc003120 con 0x7ff4f0100c70 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff4f01075b0 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555366634 0 0) 0x7ff4f01004f0 con 0x7ff4f0103a50 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f010b340 con 0x7ff4f0103a50 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831418148 0 0) 0x7ff4dc003120 con 0x7ff4f0100c70 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4f01004f0 con 0x7ff4f0100c70 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff4ec002750 con 0x7ff4f0100c70 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288700788 0 0) 0x7ff4f0100300 con 0x7ff4f01075b0 2024-09-10T19:02:23.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.812+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4dc003120 con 0x7ff4f01075b0 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904339404 0 0) 0x7ff4f01004f0 con 0x7ff4f0100c70 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6790/0 conn(0x7ff4f0103a50 legacy=0x7ff4f00fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.165:6789/0 conn(0x7ff4f01075b0 legacy=0x7ff4f01217f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f01006e0 con 0x7ff4f0100c70 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f0122f70 con 0x7ff4f0100c70 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4f7353640 10 client.25446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f01233c0 con 0x7ff4f0100c70 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4f7353640 1 librados: init done 2024-09-10T19:02:23.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff4ec001710 con 0x7ff4f0100c70 2024-09-10T19:02:23.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff4ec005780 con 0x7ff4f0100c70 2024-09-10T19:02:23.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff4d4004ff0 con 0x7ff4f0100c70 2024-09-10T19:02:23.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff4ec005a20 con 0x7ff4f0100c70 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(519..519 src has 69..519) v4 ==== 8036+0+0 (unknown 2616460030 0 0) 0x7ff4ec08a360 con 0x7ff4f0100c70 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 osd_map(519..519 src has 69..519) v4 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 3 client.25446.objecter handle_osd_map got epochs [519,519] > 0 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 3 client.25446.objecter handle_osd_map decoding full epoch 519 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 20 client.25446.objecter dump_active .. 0 homeless 2024-09-10T19:02:23.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.816+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v519) v1 ==== 54+0+16 (unknown 3666528676 0 2420000192) 0x7ff4ec08c790 con 0x7ff4f0100c70 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4dc0715f0 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 10 client.25446.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 1 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 1 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 0 1 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 4 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 10 client.25446.objecter _send_command 1 2024-09-10T19:02:23.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d4005290 con 0x7ff4d4004610 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d4004610 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d4005290 con 0x7ff4d4004610 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 1 = system:0 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 4 2024-09-10T19:02:23.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.820+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 0 1 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 10 client.25446.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 2 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 2 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4005810 osd=0 3 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 0 2 2024-09-10T19:02:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 4 2024-09-10T19:02:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 10 client.25446.objecter _send_command 2 2024-09-10T19:02:23.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d4004cc0 con 0x7ff4d4004610 2024-09-10T19:02:23.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d4004cc0 con 0x7ff4d4004610 2024-09-10T19:02:23.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 4 2024-09-10T19:02:23.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 0 2 2024-09-10T19:02:23.842 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 10 client.25446.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 1 3 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 4 2024-09-10T19:02:23.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 10 client.25446.objecter _send_command 3 2024-09-10T19:02:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.836+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d40022b0 con 0x7ff4d4004110 2024-09-10T19:02:23.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d4004110 2024-09-10T19:02:23.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d40022b0 con 0x7ff4d4004110 2024-09-10T19:02:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:02:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 3 = system:0 2024-09-10T19:02:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 4 2024-09-10T19:02:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.840+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 1 3 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 10 client.25446.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 4 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 4 2024-09-10T19:02:23.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4003fa0 osd=1 3 2024-09-10T19:02:23.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 1 4 2024-09-10T19:02:23.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 4 2024-09-10T19:02:23.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 10 client.25446.objecter _send_command 4 2024-09-10T19:02:23.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d4001890 con 0x7ff4d4004110 2024-09-10T19:02:23.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d4001890 con 0x7ff4d4004110 2024-09-10T19:02:23.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 4 2024-09-10T19:02:23.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 1 4 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 10 client.25446.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 5 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 5 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 2 5 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 4 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 10 client.25446.objecter _send_command 5 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.860+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d4003d90 con 0x7ff4d400d350 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d400d350 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d4003d90 con 0x7ff4d400d350 2024-09-10T19:02:23.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:02:23.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 5 = system:0 2024-09-10T19:02:23.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 4 2024-09-10T19:02:23.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.864+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 2 5 2024-09-10T19:02:23.876 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:23.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:23.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 10 client.25446.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 6 2024-09-10T19:02:23.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 6 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d40035e0 osd=2 3 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 2 6 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 4 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 10 client.25446.objecter _send_command 6 2024-09-10T19:02:23.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d40030d0 con 0x7ff4d400d350 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d40030d0 con 0x7ff4d400d350 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 4 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 2 6 2024-09-10T19:02:23.884 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 10 client.25446.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 7 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 7 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 3 7 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 4 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 10 client.25446.objecter _send_command 7 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d400d100 con 0x7ff4d4011fd0 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d4011fd0 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d400d100 con 0x7ff4d4011fd0 2024-09-10T19:02:23.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:02:23.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 7 = system:0 2024-09-10T19:02:23.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 4 2024-09-10T19:02:23.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.880+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 3 7 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 10 client.25446.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 8 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 8 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d400c990 osd=3 3 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 3 8 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 4 2024-09-10T19:02:23.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 10 client.25446.objecter _send_command 8 2024-09-10T19:02:23.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.896+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d40132d0 con 0x7ff4d4011fd0 2024-09-10T19:02:23.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d40132d0 con 0x7ff4d4011fd0 2024-09-10T19:02:23.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 4 2024-09-10T19:02:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 3 8 2024-09-10T19:02:23.904 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:02:23.904 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 10 client.25446.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 9 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 9 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 4 9 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 4 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 10 client.25446.objecter _send_command 9 2024-09-10T19:02:23.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.900+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d401a210 con 0x7ff4d4016710 2024-09-10T19:02:23.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d4016710 2024-09-10T19:02:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d401a210 con 0x7ff4d4016710 2024-09-10T19:02:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:02:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 9 = system:0 2024-09-10T19:02:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 4 2024-09-10T19:02:23.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.904+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 4 9 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 10 client.25446.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 10 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 10 2024-09-10T19:02:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d40132d0 osd=4 3 2024-09-10T19:02:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 4 10 2024-09-10T19:02:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 4 2024-09-10T19:02:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 10 client.25446.objecter _send_command 10 2024-09-10T19:02:23.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.920+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d400c240 con 0x7ff4d4016710 2024-09-10T19:02:23.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d400c240 con 0x7ff4d4016710 2024-09-10T19:02:23.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 4 2024-09-10T19:02:23.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 4 10 2024-09-10T19:02:23.930 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:02:23.930 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:02:23.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 10 client.25446.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 11 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 11 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 5 11 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 4 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 10 client.25446.objecter _send_command 11 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.924+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d4011b20 con 0x7ff4d4011610 2024-09-10T19:02:23.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d4011610 2024-09-10T19:02:23.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d4011b20 con 0x7ff4d4011610 2024-09-10T19:02:23.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:02:23.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 11 = system:0 2024-09-10T19:02:23.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 4 2024-09-10T19:02:23.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.928+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 5 11 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 10 client.25446.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 12 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 12 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d400c240 osd=5 3 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 5 12 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 4 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 10 client.25446.objecter _send_command 12 2024-09-10T19:02:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.948+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d4011b20 con 0x7ff4d4011610 2024-09-10T19:02:23.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d4011b20 con 0x7ff4d4011610 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 4 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 5 12 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:02:23.957 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 10 client.25446.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 13 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 13 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 6 13 2024-09-10T19:02:23.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 4 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 10 client.25446.objecter _send_command 13 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d4011db0 con 0x7ff4d401f140 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.952+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d401f140 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.956+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d4011db0 con 0x7ff4d401f140 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.956+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.956+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 13 = system:0 2024-09-10T19:02:23.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.956+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 4 2024-09-10T19:02:23.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.956+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 6 13 2024-09-10T19:02:23.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 10 client.25446.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:23.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 14 2024-09-10T19:02:23.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 14 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4011b20 osd=6 3 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 6 14 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 4 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 10 client.25446.objecter _send_command 14 2024-09-10T19:02:23.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.972+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d4022c40 con 0x7ff4d401f140 2024-09-10T19:02:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d4022c40 con 0x7ff4d401f140 2024-09-10T19:02:23.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:23.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 4 2024-09-10T19:02:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 6 14 2024-09-10T19:02:23.980 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:02:23.980 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 10 client.25446.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 15 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 15 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 7 15 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 4 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 10 client.25446.objecter _send_command 15 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4d4024ec0 con 0x7ff4d40232c0 2024-09-10T19:02:23.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 10 client.25446.objecter ms_handle_connect 0x7ff4d40232c0 2024-09-10T19:02:23.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff4d4024ec0 con 0x7ff4d40232c0 2024-09-10T19:02:23.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:02:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 15 = system:0 2024-09-10T19:02:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 4 2024-09-10T19:02:23.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.976+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 7 15 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 10 client.25446.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign -1 16 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 20 client.25446.objecter _get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_remove -1 16 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 20 client.25446.objecter get_session s=0x7ff4d4022c40 osd=7 3 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 15 client.25446.objecter _session_command_op_assign 7 16 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 4 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 10 client.25446.objecter _send_command 16 2024-09-10T19:02:24.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:23.996+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff4d4024ec0 con 0x7ff4d40232c0 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4e5ffb640 1 -- 172.21.15.83:0/1629362500 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff4d4024ec0 con 0x7ff4d40232c0 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4e5ffb640 10 client.25446.objecter ms_dispatch 0x7ff4f0000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4e5ffb640 10 client.25446.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4e5ffb640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 4 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4e5ffb640 15 client.25446.objecter _session_command_op_remove 7 16 2024-09-10T19:02:24.005 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:02:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:02:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 librados: watch_flush enter 2024-09-10T19:02:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 librados: watch_flush exit 2024-09-10T19:02:24.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.0 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6801/2535527483 conn(0x7ff4d4004610 legacy=0x7ff4d4002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4005810 osd=0 1 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.1 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6805/2309624637 conn(0x7ff4d4004110 legacy=0x7ff4d400aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4003fa0 osd=1 1 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.2 2024-09-10T19:02:24.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6809/386094220 conn(0x7ff4d400d350 legacy=0x7ff4d400f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40035e0 osd=2 1 2024-09-10T19:02:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.3 2024-09-10T19:02:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6813/1279688021 conn(0x7ff4d4011fd0 legacy=0x7ff4d4012400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c990 osd=3 1 2024-09-10T19:02:24.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.4 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.165:6800/3225515265 conn(0x7ff4d4016710 legacy=0x7ff4d4018af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d40132d0 osd=4 1 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.5 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.165:6804/571173777 conn(0x7ff4d4011610 legacy=0x7ff4d401b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d400c240 osd=5 1 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.6 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.165:6808/1317922936 conn(0x7ff4d401f140 legacy=0x7ff4d4021520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4011b20 osd=6 1 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter close_session for osd.7 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff4d40232c0 legacy=0x7ff4d4023730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter put_session s=0x7ff4d4022c40 osd=7 1 2024-09-10T19:02:24.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 20 client.25446.objecter shutdown clearing up homeless session... 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 10 client.25446.objecter successfully canceled tick 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f58f0640 1 -- 172.21.15.83:0/1629362500 reap_dead start 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f58f0640 1 -- 172.21.15.83:0/1629362500 reap_dead start 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f58f0640 1 -- 172.21.15.83:0/1629362500 reap_dead start 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f58f0640 1 -- 172.21.15.83:0/1629362500 reap_dead start 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f58f0640 1 -- 172.21.15.83:0/1629362500 reap_dead start 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff4dc0715f0 legacy=0x7ff4dc073aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> v1:172.21.15.83:6789/0 conn(0x7ff4f0100c70 legacy=0x7ff4f00fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 shutdown_connections 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.000+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 >> 172.21.15.83:0/1629362500 conn(0x7ff4f0074a20 msgr2=0x7ff4f00f9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.004+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 shutdown_connections 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.004+0000 7ff4f7353640 1 -- 172.21.15.83:0/1629362500 wait complete. 2024-09-10T19:02:24.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:24.004+0000 7ff4f7353640 1 librados: shutdown 2024-09-10T19:02:24.110 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:24.140 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:24.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:24.279 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:24.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:24.415 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:24.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:24.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:23.365528+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:23.366763+0000 mon.a (mon.0) 2784 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:24.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: cluster 2024-09-10T19:02:23.366883+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:02:24.530 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:23.368348+0000 mon.a (mon.0) 2786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:23.821244+0000 mon.a (mon.0) 2787 : audit [DBG] from='client.? v1:172.21.15.83:0/1629362500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: cluster 2024-09-10T19:02:24.178508+0000 mgr.y (mgr.24526) 527 : cluster [DBG] pgmap v797: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:24.197251+0000 mon.a (mon.0) 2788 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: cluster 2024-09-10T19:02:24.197556+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:24.198904+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.531 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:24 smithi165 bash[22591]: audit 2024-09-10T19:02:24.202370+0000 mon.a (mon.0) 2790 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.551 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:24.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:23.365528+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:23.366763+0000 mon.a (mon.0) 2784 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: cluster 2024-09-10T19:02:23.366883+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:23.368348+0000 mon.a (mon.0) 2786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:23.821244+0000 mon.a (mon.0) 2787 : audit [DBG] from='client.? v1:172.21.15.83:0/1629362500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:24.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: cluster 2024-09-10T19:02:24.178508+0000 mgr.y (mgr.24526) 527 : cluster [DBG] pgmap v797: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:24.197251+0000 mon.a (mon.0) 2788 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: cluster 2024-09-10T19:02:24.197556+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:24.198904+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[18629]: audit 2024-09-10T19:02:24.202370+0000 mon.a (mon.0) 2790 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:23.365528+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:23.366763+0000 mon.a (mon.0) 2784 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: cluster 2024-09-10T19:02:23.366883+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:23.368348+0000 mon.a (mon.0) 2786 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:23.821244+0000 mon.a (mon.0) 2787 : audit [DBG] from='client.? v1:172.21.15.83:0/1629362500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: cluster 2024-09-10T19:02:24.178508+0000 mgr.y (mgr.24526) 527 : cluster [DBG] pgmap v797: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:02:24.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:24.197251+0000 mon.a (mon.0) 2788 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:24.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: cluster 2024-09-10T19:02:24.197556+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:02:24.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:24.198904+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.690 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:24 smithi083 bash[32381]: audit 2024-09-10T19:02:24.202370+0000 mon.a (mon.0) 2790 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:24.690 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:24.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:24.822 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:24.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:24.956 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:24.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:25.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.095 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:25.198 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:25.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:25.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.357 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:25.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:25.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:25.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:25.630 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:25.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:25.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:25.904 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:25.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:25.976 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:26.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:26.032 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:26.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:26.167 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:26.185 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:02:26.185 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:02:26.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:26.284 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[32381]: audit 2024-09-10T19:02:25.199691+0000 mon.a (mon.0) 2791 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:26.284 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[32381]: cluster 2024-09-10T19:02:25.200073+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:02:26.284 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[32381]: audit 2024-09-10T19:02:25.226327+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.284 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[32381]: audit 2024-09-10T19:02:25.229121+0000 mon.a (mon.0) 2793 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.284 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[18629]: audit 2024-09-10T19:02:25.199691+0000 mon.a (mon.0) 2791 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:26.285 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[18629]: cluster 2024-09-10T19:02:25.200073+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:02:26.285 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[18629]: audit 2024-09-10T19:02:25.226327+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.285 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:26 smithi083 bash[18629]: audit 2024-09-10T19:02:25.229121+0000 mon.a (mon.0) 2793 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.305 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:26.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:26.436 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:26.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:26 smithi165 bash[22591]: audit 2024-09-10T19:02:25.199691+0000 mon.a (mon.0) 2791 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:26.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:26 smithi165 bash[22591]: cluster 2024-09-10T19:02:25.200073+0000 mon.a (mon.0) 2792 : cluster [DBG] osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:02:26.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:26 smithi165 bash[22591]: audit 2024-09-10T19:02:25.226327+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:26 smithi165 bash[22591]: audit 2024-09-10T19:02:25.229121+0000 mon.a (mon.0) 2793 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:02:26.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:26.574 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:26.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:26.711 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:26.815 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:26.850 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:26.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:02:26.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:26.982 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:26.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:27.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:27.117 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:27.221 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:27.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:27 smithi165 bash[22591]: cluster 2024-09-10T19:02:26.180894+0000 mgr.y (mgr.24526) 528 : cluster [DBG] pgmap v800: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:27.222 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:27 smithi165 bash[22591]: audit 2024-09-10T19:02:26.202438+0000 mon.a (mon.0) 2794 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:02:27.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:27 smithi165 bash[22591]: cluster 2024-09-10T19:02:26.202514+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:02:27.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:27 smithi165 bash[22591]: audit 2024-09-10T19:02:26.221259+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:27.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:27.387 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:27.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:02:27.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:27.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[18629]: cluster 2024-09-10T19:02:26.180894+0000 mgr.y (mgr.24526) 528 : cluster [DBG] pgmap v800: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:27.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[18629]: audit 2024-09-10T19:02:26.202438+0000 mon.a (mon.0) 2794 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:02:27.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[18629]: cluster 2024-09-10T19:02:26.202514+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:02:27.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[18629]: audit 2024-09-10T19:02:26.221259+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.505 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[18629]: audit 2024-09-10T19:02:26.224221+0000 mon.a (mon.0) 2796 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[32381]: cluster 2024-09-10T19:02:26.180894+0000 mgr.y (mgr.24526) 528 : cluster [DBG] pgmap v800: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:27.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[32381]: audit 2024-09-10T19:02:26.202438+0000 mon.a (mon.0) 2794 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:02:27.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[32381]: cluster 2024-09-10T19:02:26.202514+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:02:27.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[32381]: audit 2024-09-10T19:02:26.221259+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.506 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:27 smithi083 bash[32381]: audit 2024-09-10T19:02:26.224221+0000 mon.a (mon.0) 2796 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:27 smithi165 bash[22591]: audit 2024-09-10T19:02:26.224221+0000 mon.a (mon.0) 2796 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:02:27.626 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:27.653 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:27.756 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:27.789 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:27.893 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:27.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:27.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:28.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:28.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:28.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:28.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:28.301 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[18629]: audit 2024-09-10T19:02:27.217050+0000 mon.a (mon.0) 2797 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[18629]: cluster 2024-09-10T19:02:27.217175+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[18629]: audit 2024-09-10T19:02:27.240813+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[18629]: audit 2024-09-10T19:02:27.244068+0000 mon.a (mon.0) 2799 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[32381]: audit 2024-09-10T19:02:27.217050+0000 mon.a (mon.0) 2797 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[32381]: cluster 2024-09-10T19:02:27.217175+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:02:28.313 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[32381]: audit 2024-09-10T19:02:27.240813+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.314 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:28 smithi083 bash[32381]: audit 2024-09-10T19:02:27.244068+0000 mon.a (mon.0) 2799 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.334 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- 172.21.15.83:0/615344947 >> v1:172.21.15.83:6790/0 conn(0x7f94000fd630 legacy=0x7f94000fda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- 172.21.15.83:0/615344947 shutdown_connections 2024-09-10T19:02:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- 172.21.15.83:0/615344947 >> 172.21.15.83:0/615344947 conn(0x7f9400075290 msgr2=0x7f94000f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- 172.21.15.83:0/615344947 shutdown_connections 2024-09-10T19:02:28.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- 172.21.15.83:0/615344947 wait complete. 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 librados: starting msgr at 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 librados: starting objecter 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 Processor -- start 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- start start 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 librados: setting wanted keys 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 librados: calling monclient init 2024-09-10T19:02:28.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f940011c080 con 0x7f94000fd630 2024-09-10T19:02:28.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f940011d240 con 0x7f9400105610 2024-09-10T19:02:28.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f9404d64640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f940011e440 con 0x7f9400109110 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93fed9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9400109110 0x7f940011a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:44200/0 (socket says 172.21.15.83:44200) 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93fed9d640 1 -- 172.21.15.83:0/2445066366 learned_addr learned my addr 172.21.15.83:0/2445066366 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 10 client.?.objecter ms_handle_connect 0x7f9400109110 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 10 client.?.objecter ms_handle_connect 0x7f94000fd630 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392617888 0 0) 0x7f940011e440 con 0x7f9400109110 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.348+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e80038e0 con 0x7f9400109110 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 10 client.?.objecter ms_handle_connect 0x7f9400105610 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099273526 0 0) 0x7f940011c080 con 0x7f94000fd630 2024-09-10T19:02:28.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f940011e440 con 0x7f94000fd630 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982588089 0 0) 0x7f940011d240 con 0x7f9400105610 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f940011c080 con 0x7f9400105610 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748599927 0 0) 0x7f93e80038e0 con 0x7f9400109110 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f940011d240 con 0x7f9400109110 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f93f8001d70 con 0x7f9400109110 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997384913 0 0) 0x7f940011e440 con 0x7f94000fd630 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93e80038e0 con 0x7f94000fd630 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f93f0002c30 con 0x7f94000fd630 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1056411239 0 0) 0x7f940011c080 con 0x7f9400105610 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f940011e440 con 0x7f9400105610 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4060937663 0 0) 0x7f940011d240 con 0x7f9400109110 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 >> v1:172.21.15.165:6789/0 conn(0x7f9400105610 legacy=0x7f9400102180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:28.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 >> v1:172.21.15.83:6789/0 conn(0x7f94000fd630 legacy=0x7f9400101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940011f640 con 0x7f9400109110 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940011c270 con 0x7f9400109110 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f9404d64640 10 client.49526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f940011c6c0 con 0x7f9400109110 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f9404d64640 1 librados: init done 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f93f80026a0 con 0x7f9400109110 2024-09-10T19:02:28.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f93f8005d70 con 0x7f9400109110 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e0005010 con 0x7f9400109110 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.352+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f93f801c250 con 0x7f9400109110 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(524..524 src has 69..524) v4 ==== 8071+0+0 (unknown 867314882 0 0) 0x7f93f808a8b0 con 0x7f9400109110 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 10 client.49526.objecter ms_dispatch 0x7f9400000b90 osd_map(524..524 src has 69..524) v4 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 3 client.49526.objecter handle_osd_map got epochs [524,524] > 0 2024-09-10T19:02:28.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 3 client.49526.objecter handle_osd_map decoding full epoch 524 2024-09-10T19:02:28.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 20 client.49526.objecter dump_active .. 0 homeless 2024-09-10T19:02:28.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.356+0000 7f93eeffd640 10 client.49526.objecter ms_handle_connect 0x7f93e8075ba0 2024-09-10T19:02:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:28 smithi165 bash[22591]: audit 2024-09-10T19:02:27.217050+0000 mon.a (mon.0) 2797 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:02:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:28 smithi165 bash[22591]: cluster 2024-09-10T19:02:27.217175+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:02:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:28 smithi165 bash[22591]: audit 2024-09-10T19:02:27.240813+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:28 smithi165 bash[22591]: audit 2024-09-10T19:02:27.244068+0000 mon.a (mon.0) 2799 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:28.363 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:28.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.360+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f93f8056e70 con 0x7f9400109110 2024-09-10T19:02:28.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:28.471 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:28.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:28.560+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f93e00052c0 con 0x7f9400109110 2024-09-10T19:02:28.575 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:28.604 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:28.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.734 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:28.837 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.873 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:28.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:28.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.011 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:29.115 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.145 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:29.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.256+0000 7f93eeffd640 1 -- 172.21.15.83:0/2445066366 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v525) v1 ==== 91+0+0 (unknown 3673212267 0 0) 0x7f93f8056810 con 0x7f9400109110 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.101097+0000 mgr.y (mgr.24526) 529 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: cluster 2024-09-10T19:02:28.181859+0000 mgr.y (mgr.24526) 530 : cluster [DBG] pgmap v803: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.235906+0000 mon.a (mon.0) 2800 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: cluster 2024-09-10T19:02:28.235981+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.283281+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.284619+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.286118+0000 mon.a (mon.0) 2802 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.287307+0000 mon.a (mon.0) 2803 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.562789+0000 mon.c (mon.2) 655 : audit [INF] from='client.? v1:172.21.15.83:0/2445066366' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:29 smithi165 bash[22591]: audit 2024-09-10T19:02:28.563481+0000 mon.a (mon.0) 2804 : audit [INF] from='client.49526 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.260 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 10 librados: watch_flush enter 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 10 librados: watch_flush exit 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 20 client.49526.objecter shutdown clearing up homeless session... 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 10 client.49526.objecter successfully canceled tick 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 >> v1:172.21.15.83:6800/3071793735 conn(0x7f93e8075ba0 legacy=0x7f93e8078050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 >> v1:172.21.15.83:6790/0 conn(0x7f9400109110 legacy=0x7f940011a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 shutdown_connections 2024-09-10T19:02:29.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 >> 172.21.15.83:0/2445066366 conn(0x7f9400075290 msgr2=0x7f94001079f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:29.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 shutdown_connections 2024-09-10T19:02:29.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 -- 172.21.15.83:0/2445066366 wait complete. 2024-09-10T19:02:29.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:29.260+0000 7f9404d64640 1 librados: shutdown 2024-09-10T19:02:29.274 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:29.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:29.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.101097+0000 mgr.y (mgr.24526) 529 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:29.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: cluster 2024-09-10T19:02:28.181859+0000 mgr.y (mgr.24526) 530 : cluster [DBG] pgmap v803: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:29.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.235906+0000 mon.a (mon.0) 2800 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: cluster 2024-09-10T19:02:28.235981+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.283281+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.284619+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.286118+0000 mon.a (mon.0) 2802 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.287307+0000 mon.a (mon.0) 2803 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.562789+0000 mon.c (mon.2) 655 : audit [INF] from='client.? v1:172.21.15.83:0/2445066366' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[18629]: audit 2024-09-10T19:02:28.563481+0000 mon.a (mon.0) 2804 : audit [INF] from='client.49526 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.101097+0000 mgr.y (mgr.24526) 529 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:29.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: cluster 2024-09-10T19:02:28.181859+0000 mgr.y (mgr.24526) 530 : cluster [DBG] pgmap v803: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.235906+0000 mon.a (mon.0) 2800 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: cluster 2024-09-10T19:02:28.235981+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.283281+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.284619+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.286118+0000 mon.a (mon.0) 2802 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.287307+0000 mon.a (mon.0) 2803 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-81"}]: dispatch 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.562789+0000 mon.c (mon.2) 655 : audit [INF] from='client.? v1:172.21.15.83:0/2445066366' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:29 smithi083 bash[32381]: audit 2024-09-10T19:02:28.563481+0000 mon.a (mon.0) 2804 : audit [INF] from='client.49526 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:02:29.405 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:29.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:29.540 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:29.643 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.669 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:29.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:29.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:29.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.936 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:29.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:30.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:30.073 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:30.177 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:30.210 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:30.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.345 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:30.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.232364+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: audit 2024-09-10T19:02:29.248640+0000 mon.a (mon.0) 2806 : audit [INF] from='client.49526 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.248718+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: audit 2024-09-10T19:02:29.567579+0000 mon.c (mon.2) 656 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.980312+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.980697+0000 osd.7 (osd.7) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.075979+0000 osd.7 (osd.7) 299 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:02:30.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.080836+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.232364+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: audit 2024-09-10T19:02:29.248640+0000 mon.a (mon.0) 2806 : audit [INF] from='client.49526 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.248718+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: audit 2024-09-10T19:02:29.567579+0000 mon.c (mon.2) 656 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.980312+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.980697+0000 osd.7 (osd.7) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.075979+0000 osd.7 (osd.7) 299 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:02:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:30 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.080836+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:02:30.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.232364+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: audit 2024-09-10T19:02:29.248640+0000 mon.a (mon.0) 2806 : audit [INF] from='client.49526 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.248718+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: audit 2024-09-10T19:02:29.567579+0000 mon.c (mon.2) 656 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.980312+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.980697+0000 osd.7 (osd.7) 298 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.075979+0000 osd.7 (osd.7) 299 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:02:30.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:30 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.080836+0000 osd.7 (osd.7) 300 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:02:30.622 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:30.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.758 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:30.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:30.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:30.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:30.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:31.028 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:31.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:31.165 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:31.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:31.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.537520+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:02:31.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:29.540110+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:02:31.274 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.182687+0000 mgr.y (mgr.24526) 531 : cluster [DBG] pgmap v806: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.257904+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: audit 2024-09-10T19:02:30.261290+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: audit 2024-09-10T19:02:30.265673+0000 mon.a (mon.0) 2809 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.478310+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:30.481225+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.084089+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:02:31.275 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.088015+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:02:31.279 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.537520+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:29.540110+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.182687+0000 mgr.y (mgr.24526) 531 : cluster [DBG] pgmap v806: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.257904+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: audit 2024-09-10T19:02:30.261290+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: audit 2024-09-10T19:02:30.265673+0000 mon.a (mon.0) 2809 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.478310+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:30.481225+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.084089+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:02:31.280 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.088015+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:02:31.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:31.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:31.432 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:31.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.537520+0000 osd.0 (osd.0) 279 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:29.540110+0000 osd.0 (osd.0) 280 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.182687+0000 mgr.y (mgr.24526) 531 : cluster [DBG] pgmap v806: 206 pgs: 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.257904+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: audit 2024-09-10T19:02:30.261290+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: audit 2024-09-10T19:02:30.265673+0000 mon.a (mon.0) 2809 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.478310+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:30.481225+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.084089+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:02:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:31 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.088015+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:02:31.547 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:31.568 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:31.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.695 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:31.715 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:31.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- 172.21.15.83:0/2596416056 >> v1:172.21.15.83:6790/0 conn(0x7ff428101510 legacy=0x7ff4281018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:31.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- 172.21.15.83:0/2596416056 shutdown_connections 2024-09-10T19:02:31.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- 172.21.15.83:0/2596416056 >> 172.21.15.83:0/2596416056 conn(0x7ff4280f70d0 msgr2=0x7ff4280f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:31.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- 172.21.15.83:0/2596416056 shutdown_connections 2024-09-10T19:02:31.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- 172.21.15.83:0/2596416056 wait complete. 2024-09-10T19:02:31.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 librados: starting msgr at 2024-09-10T19:02:31.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 librados: starting objecter 2024-09-10T19:02:31.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 Processor -- start 2024-09-10T19:02:31.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- start start 2024-09-10T19:02:31.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 librados: setting wanted keys 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 librados: calling monclient init 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff42811c270 con 0x7ff428101510 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff42811d410 con 0x7ff428107de0 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff42e179640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff42811e5f0 con 0x7ff428104280 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff4277fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff428101510 0x7ff428107150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50164/0 (socket says 172.21.15.83:50164) 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.776+0000 7ff4277fe640 1 -- 172.21.15.83:0/2710522894 learned_addr learned my addr 172.21.15.83:0/2710522894 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter ms_handle_connect 0x7ff428101510 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter ms_handle_connect 0x7ff428104280 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:31.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341192983 0 0) 0x7ff42811c270 con 0x7ff428101510 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff410003720 con 0x7ff428101510 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter ms_handle_connect 0x7ff428107de0 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916615300 0 0) 0x7ff42811e5f0 con 0x7ff428104280 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42811c270 con 0x7ff428104280 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851317463 0 0) 0x7ff42811d410 con 0x7ff428107de0 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42811e5f0 con 0x7ff428107de0 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156543958 0 0) 0x7ff410003720 con 0x7ff428101510 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff42811d410 con 0x7ff428101510 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff418001950 con 0x7ff428101510 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220673464 0 0) 0x7ff42811c270 con 0x7ff428104280 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff410003720 con 0x7ff428104280 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff41c001a10 con 0x7ff428104280 2024-09-10T19:02:31.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 949400847 0 0) 0x7ff42811d410 con 0x7ff428101510 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 >> v1:172.21.15.83:6790/0 conn(0x7ff428104280 legacy=0x7ff428117420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 >> v1:172.21.15.165:6789/0 conn(0x7ff428107de0 legacy=0x7ff42811ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42811f7f0 con 0x7ff428101510 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42811e7e0 con 0x7ff428101510 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff42e179640 10 client.25458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42811ec30 con 0x7ff428101510 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff42e179640 1 librados: init done 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff418003220 con 0x7ff428101510 2024-09-10T19:02:31.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff418003910 con 0x7ff428101510 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff414005010 con 0x7ff428101510 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.780+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff418001c00 con 0x7ff428101510 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(527..527 src has 69..527) v4 ==== 8004+0+0 (unknown 3217184390 0 0) 0x7ff41808abc0 con 0x7ff428101510 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 10 client.25458.objecter ms_dispatch 0x7ff428000b90 osd_map(527..527 src has 69..527) v4 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 3 client.25458.objecter handle_osd_map got epochs [527,527] > 0 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 3 client.25458.objecter handle_osd_map decoding full epoch 527 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 20 client.25458.objecter dump_active .. 0 homeless 2024-09-10T19:02:31.788 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.784+0000 7ff40ffff640 10 client.25458.objecter ms_handle_connect 0x7ff4100716d0 2024-09-10T19:02:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.792+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff418057180 con 0x7ff428101510 2024-09-10T19:02:31.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.831 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:31.934 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:31.967 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:31.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:31.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:31.972+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) v1 -- 0x7ff414005840 con 0x7ff428101510 2024-09-10T19:02:32.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:32.107 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:32.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:32.245 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:32.293 INFO:teuthology.orchestra.run.smithi083.stderr:set pool 1 pgp_num to 11 2024-09-10T19:02:32.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.288+0000 7ff40ffff640 1 -- 172.21.15.83:0/2710522894 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v528) v1 ==== 131+0+0 (unknown 2983202303 0 0) 0x7ff418056b20 con 0x7ff428101510 2024-09-10T19:02:32.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 10 librados: watch_flush enter 2024-09-10T19:02:32.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 10 librados: watch_flush exit 2024-09-10T19:02:32.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 20 client.25458.objecter shutdown clearing up homeless session... 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 10 client.25458.objecter successfully canceled tick 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff4100716d0 legacy=0x7ff410073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 >> v1:172.21.15.83:6789/0 conn(0x7ff428101510 legacy=0x7ff428107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 shutdown_connections 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.292+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 >> 172.21.15.83:0/2710522894 conn(0x7ff4280f70d0 msgr2=0x7ff42810aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.296+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 shutdown_connections 2024-09-10T19:02:32.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.296+0000 7ff42e179640 1 -- 172.21.15.83:0/2710522894 wait complete. 2024-09-10T19:02:32.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:32.296+0000 7ff42e179640 1 librados: shutdown 2024-09-10T19:02:32.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: audit 2024-09-10T19:02:31.262966+0000 mon.a (mon.0) 2810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.263271+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:02:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: audit 2024-09-10T19:02:31.281971+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: audit 2024-09-10T19:02:31.290476+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: audit 2024-09-10T19:02:31.293140+0000 mon.a (mon.0) 2812 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.887885+0000 osd.6 (osd.6) 243 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.891913+0000 osd.6 (osd.6) 244 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: audit 2024-09-10T19:02:31.978532+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.133945+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:02:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:32 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.137440+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:02:32.382 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:32.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:32.521 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:32.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: audit 2024-09-10T19:02:31.262966+0000 mon.a (mon.0) 2810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:32.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.263271+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: audit 2024-09-10T19:02:31.281971+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: audit 2024-09-10T19:02:31.290476+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: audit 2024-09-10T19:02:31.293140+0000 mon.a (mon.0) 2812 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.887885+0000 osd.6 (osd.6) 243 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.891913+0000 osd.6 (osd.6) 244 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: audit 2024-09-10T19:02:31.978532+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.133945+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.137440+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: audit 2024-09-10T19:02:31.262966+0000 mon.a (mon.0) 2810 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.263271+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:02:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: audit 2024-09-10T19:02:31.281971+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: audit 2024-09-10T19:02:31.290476+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: audit 2024-09-10T19:02:31.293140+0000 mon.a (mon.0) 2812 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.887885+0000 osd.6 (osd.6) 243 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.891913+0000 osd.6 (osd.6) 244 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: audit 2024-09-10T19:02:31.978532+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.133945+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:02:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.137440+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:02:32.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:32.659 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:32.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:32.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:32] "GET /metrics HTTP/1.1" 200 226458 "" "Prometheus/2.43.0" 2024-09-10T19:02:32.794 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:32.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:32.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:02:32.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:32.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:33.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:33.059 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:33.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:33.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:33.296 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:33.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.493583+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.496972+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.709047+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:31.711894+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.183541+0000 mgr.y (mgr.24526) 532 : cluster [DBG] pgmap v809: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: audit 2024-09-10T19:02:32.291738+0000 mon.b (mon.1) 402 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: audit 2024-09-10T19:02:32.291829+0000 mon.a (mon.0) 2814 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: audit 2024-09-10T19:02:32.291923+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.291981+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: audit 2024-09-10T19:02:32.296977+0000 mon.a (mon.0) 2817 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.937165+0000 osd.6 (osd.6) 245 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:02:33.435 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:32.941483+0000 osd.6 (osd.6) 246 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:33.096215+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:02:33.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:33 smithi165 bash[22591]: cluster 2024-09-10T19:02:33.099509+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:02:33.455 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.493583+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.496972+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.709047+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:31.711894+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.183541+0000 mgr.y (mgr.24526) 532 : cluster [DBG] pgmap v809: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: audit 2024-09-10T19:02:32.291738+0000 mon.b (mon.1) 402 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: audit 2024-09-10T19:02:32.291829+0000 mon.a (mon.0) 2814 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: audit 2024-09-10T19:02:32.291923+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.291981+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: audit 2024-09-10T19:02:32.296977+0000 mon.a (mon.0) 2817 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.937165+0000 osd.6 (osd.6) 245 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:32.941483+0000 osd.6 (osd.6) 246 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:33.096215+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[18629]: cluster 2024-09-10T19:02:33.099509+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.493583+0000 osd.0 (osd.0) 281 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.496972+0000 osd.0 (osd.0) 282 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.709047+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:02:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:31.711894+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.183541+0000 mgr.y (mgr.24526) 532 : cluster [DBG] pgmap v809: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: audit 2024-09-10T19:02:32.291738+0000 mon.b (mon.1) 402 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: audit 2024-09-10T19:02:32.291829+0000 mon.a (mon.0) 2814 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: audit 2024-09-10T19:02:32.291923+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? v1:172.21.15.83:0/2710522894' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.291981+0000 mon.a (mon.0) 2816 : cluster [DBG] osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: audit 2024-09-10T19:02:32.296977+0000 mon.a (mon.0) 2817 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.937165+0000 osd.6 (osd.6) 245 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:32.941483+0000 osd.6 (osd.6) 246 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:33.096215+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:02:33.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:33 smithi083 bash[32381]: cluster 2024-09-10T19:02:33.099509+0000 osd.7 (osd.7) 306 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:02:33.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:33.590 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:33.678 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:33.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:33.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:33.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:33.861 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:33.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:33.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:34.001 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:34.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:34.136 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:34.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.270 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:34.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:33.303659+0000 mon.a (mon.0) 2818 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: cluster 2024-09-10T19:02:33.303943+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:33.305728+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:33.308572+0000 mon.a (mon.0) 2820 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: cluster 2024-09-10T19:02:33.455825+0000 osd.5 (osd.5) 292 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: cluster 2024-09-10T19:02:33.461812+0000 osd.5 (osd.5) 293 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:34.197755+0000 mon.a (mon.0) 2821 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.197863+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:02:34.387 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:34.198495+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.388 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:34 smithi165 bash[22591]: audit 2024-09-10T19:02:34.206074+0000 mon.a (mon.0) 2823 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.407 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:34.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.525 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:33.303659+0000 mon.a (mon.0) 2818 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: cluster 2024-09-10T19:02:33.303943+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:33.305728+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:33.308572+0000 mon.a (mon.0) 2820 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: cluster 2024-09-10T19:02:33.455825+0000 osd.5 (osd.5) 292 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: cluster 2024-09-10T19:02:33.461812+0000 osd.5 (osd.5) 293 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:34.197755+0000 mon.a (mon.0) 2821 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.197863+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:34.198495+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[32381]: audit 2024-09-10T19:02:34.206074+0000 mon.a (mon.0) 2823 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.526 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:33.303659+0000 mon.a (mon.0) 2818 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: cluster 2024-09-10T19:02:33.303943+0000 mon.a (mon.0) 2819 : cluster [DBG] osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:33.305728+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:33.308572+0000 mon.a (mon.0) 2820 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: cluster 2024-09-10T19:02:33.455825+0000 osd.5 (osd.5) 292 : cluster [DBG] 64.18 scrub starts 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: cluster 2024-09-10T19:02:33.461812+0000 osd.5 (osd.5) 293 : cluster [DBG] 64.18 scrub ok 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:34.197755+0000 mon.a (mon.0) 2821 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.197863+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:34.198495+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:34 smithi083 bash[18629]: audit 2024-09-10T19:02:34.206074+0000 mon.a (mon.0) 2823 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:34.538 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:34.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:34.677 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:34.781 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.815 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:34.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:34.957 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:34.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:35.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.096 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:35.146 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:02:35.146 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:02:35.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:35.227 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:35.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.138952+0000 osd.3 (osd.3) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.139340+0000 osd.3 (osd.3) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.184327+0000 mgr.y (mgr.24526) 533 : cluster [DBG] pgmap v812: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.219328+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.225851+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.868384+0000 osd.6 (osd.6) 247 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.875762+0000 osd.6 (osd.6) 248 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.098715+0000 osd.7 (osd.7) 307 : cluster [DBG] 3.f scrub starts 2024-09-10T19:02:35.353 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.103726+0000 osd.7 (osd.7) 308 : cluster [DBG] 3.f scrub ok 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:35.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:35.500 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:35.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.138952+0000 osd.3 (osd.3) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.139340+0000 osd.3 (osd.3) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.184327+0000 mgr.y (mgr.24526) 533 : cluster [DBG] pgmap v812: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.219328+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.225851+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.868384+0000 osd.6 (osd.6) 247 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.875762+0000 osd.6 (osd.6) 248 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.098715+0000 osd.7 (osd.7) 307 : cluster [DBG] 3.f scrub starts 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.103726+0000 osd.7 (osd.7) 308 : cluster [DBG] 3.f scrub ok 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: audit 2024-09-10T19:02:35.209271+0000 mon.a (mon.0) 2824 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:35 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.209363+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:02:35.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: audit 2024-09-10T19:02:35.209271+0000 mon.a (mon.0) 2824 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:35.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.209363+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.138952+0000 osd.3 (osd.3) 307 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.139340+0000 osd.3 (osd.3) 308 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.184327+0000 mgr.y (mgr.24526) 533 : cluster [DBG] pgmap v812: 238 pgs: 32 creating+peering, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.219328+0000 osd.4 (osd.4) 316 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.225851+0000 osd.4 (osd.4) 317 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.868384+0000 osd.6 (osd.6) 247 : cluster [DBG] 64.8 scrub starts 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.875762+0000 osd.6 (osd.6) 248 : cluster [DBG] 64.8 scrub ok 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.098715+0000 osd.7 (osd.7) 307 : cluster [DBG] 3.f scrub starts 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.103726+0000 osd.7 (osd.7) 308 : cluster [DBG] 3.f scrub ok 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: audit 2024-09-10T19:02:35.209271+0000 mon.a (mon.0) 2824 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:35.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:35 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.209363+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:02:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.641 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:35.742 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:35.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:35.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:02:35.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:35.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:36.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:36.136 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:36.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.306 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:36.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.514837+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:02:36.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:34.523126+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:02:36.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.249054+0000 osd.4 (osd.4) 318 : cluster [DBG] 64.d scrub starts 2024-09-10T19:02:36.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.257645+0000 osd.4 (osd.4) 319 : cluster [DBG] 64.d scrub ok 2024-09-10T19:02:36.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.525646+0000 osd.5 (osd.5) 294 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:02:36.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.530035+0000 osd.5 (osd.5) 295 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:02:36.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.909979+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:02:36.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.912757+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:02:36.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.216252+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:02:36.424 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.514837+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:34.523126+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.249054+0000 osd.4 (osd.4) 318 : cluster [DBG] 64.d scrub starts 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.257645+0000 osd.4 (osd.4) 319 : cluster [DBG] 64.d scrub ok 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.525646+0000 osd.5 (osd.5) 294 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.530035+0000 osd.5 (osd.5) 295 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:02:36.425 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.909979+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:02:36.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.912757+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:02:36.426 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:36 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.216252+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:02:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:36.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.514837+0000 osd.0 (osd.0) 283 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:02:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:34.523126+0000 osd.0 (osd.0) 284 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.249054+0000 osd.4 (osd.4) 318 : cluster [DBG] 64.d scrub starts 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.257645+0000 osd.4 (osd.4) 319 : cluster [DBG] 64.d scrub ok 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.525646+0000 osd.5 (osd.5) 294 : cluster [DBG] 64.5 scrub starts 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.530035+0000 osd.5 (osd.5) 295 : cluster [DBG] 64.5 scrub ok 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.909979+0000 osd.6 (osd.6) 249 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.912757+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:02:36.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:36 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.216252+0000 mon.a (mon.0) 2826 : cluster [DBG] osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:02:36.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.716 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:36.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:36.854 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:36.958 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.991 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:36.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:37.096 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:37.130 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:37.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:37.259 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:37.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:37.391 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:37.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:37.528 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.799012+0000 osd.3 (osd.3) 309 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:35.804526+0000 osd.3 (osd.3) 310 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.186708+0000 mgr.y (mgr.24526) 534 : cluster [DBG] pgmap v815: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.497192+0000 osd.5 (osd.5) 296 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.500622+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.879815+0000 osd.6 (osd.6) 251 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.884119+0000 osd.6 (osd.6) 252 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: cluster 2024-09-10T19:02:37.216727+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: audit 2024-09-10T19:02:37.271559+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: audit 2024-09-10T19:02:37.273277+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: audit 2024-09-10T19:02:37.274468+0000 mon.a (mon.0) 2828 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[18629]: audit 2024-09-10T19:02:37.275999+0000 mon.a (mon.0) 2829 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.799012+0000 osd.3 (osd.3) 309 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:35.804526+0000 osd.3 (osd.3) 310 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.186708+0000 mgr.y (mgr.24526) 534 : cluster [DBG] pgmap v815: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.497192+0000 osd.5 (osd.5) 296 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.500622+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.879815+0000 osd.6 (osd.6) 251 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.884119+0000 osd.6 (osd.6) 252 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: cluster 2024-09-10T19:02:37.216727+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:02:37.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: audit 2024-09-10T19:02:37.271559+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: audit 2024-09-10T19:02:37.273277+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: audit 2024-09-10T19:02:37.274468+0000 mon.a (mon.0) 2828 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:37 smithi083 bash[32381]: audit 2024-09-10T19:02:37.275999+0000 mon.a (mon.0) 2829 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.799012+0000 osd.3 (osd.3) 309 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:35.804526+0000 osd.3 (osd.3) 310 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.186708+0000 mgr.y (mgr.24526) 534 : cluster [DBG] pgmap v815: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.497192+0000 osd.5 (osd.5) 296 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.500622+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.879815+0000 osd.6 (osd.6) 251 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.884119+0000 osd.6 (osd.6) 252 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: cluster 2024-09-10T19:02:37.216727+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: audit 2024-09-10T19:02:37.271559+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: audit 2024-09-10T19:02:37.273277+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: audit 2024-09-10T19:02:37.274468+0000 mon.a (mon.0) 2828 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:37 smithi165 bash[22591]: audit 2024-09-10T19:02:37.275999+0000 mon.a (mon.0) 2829 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-83"}]: dispatch 2024-09-10T19:02:37.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:37.662 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:37.738 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:37.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:37.794 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- 172.21.15.83:0/2020618550 >> v1:172.21.15.83:6789/0 conn(0x7efc401042a0 legacy=0x7efc40106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- 172.21.15.83:0/2020618550 shutdown_connections 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- 172.21.15.83:0/2020618550 >> 172.21.15.83:0/2020618550 conn(0x7efc400f7070 msgr2=0x7efc400f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- 172.21.15.83:0/2020618550 shutdown_connections 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- 172.21.15.83:0/2020618550 wait complete. 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 librados: starting msgr at 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 librados: starting objecter 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 Processor -- start 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 -- start start 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 librados: setting wanted keys 2024-09-10T19:02:37.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.812+0000 7efc44863640 1 librados: calling monclient init 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc40111140 con 0x7efc40107e00 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc401112d0 con 0x7efc40101530 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efc40120890 con 0x7efc401042a0 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc3ed9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efc40107e00 0x7efc4011d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46876/0 (socket says 172.21.15.83:46876) 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc3ed9d640 1 -- 172.21.15.83:0/1904481941 learned_addr learned my addr 172.21.15.83:0/1904481941 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter ms_handle_connect 0x7efc401042a0 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:37.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter ms_handle_connect 0x7efc40107e00 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter ms_handle_connect 0x7efc40101530 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820425710 0 0) 0x7efc40120890 con 0x7efc401042a0 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28003720 con 0x7efc401042a0 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390997533 0 0) 0x7efc40111140 con 0x7efc40107e00 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40120890 con 0x7efc40107e00 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280447205 0 0) 0x7efc401112d0 con 0x7efc40101530 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40111140 con 0x7efc40101530 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 570389443 0 0) 0x7efc28003720 con 0x7efc401042a0 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc401112d0 con 0x7efc401042a0 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efc34001910 con 0x7efc401042a0 2024-09-10T19:02:37.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200005081 0 0) 0x7efc40120890 con 0x7efc40107e00 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc28003720 con 0x7efc40107e00 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efc38001c20 con 0x7efc40107e00 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 432875063 0 0) 0x7efc40111140 con 0x7efc40101530 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc40120890 con 0x7efc40101530 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efc30001800 con 0x7efc40101530 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3085828524 0 0) 0x7efc401112d0 con 0x7efc401042a0 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.165:6789/0 conn(0x7efc40101530 legacy=0x7efc401109a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6789/0 conn(0x7efc40107e00 legacy=0x7efc4011d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40121a30 con 0x7efc401042a0 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc40120a80 con 0x7efc401042a0 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7efc34002340 con 0x7efc401042a0 2024-09-10T19:02:37.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efc34003620 con 0x7efc401042a0 2024-09-10T19:02:37.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 10 client.49538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:37.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc40120ed0 con 0x7efc401042a0 2024-09-10T19:02:37.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 librados: init done 2024-09-10T19:02:37.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.816+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efc400fcc30 con 0x7efc401042a0 2024-09-10T19:02:37.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7efc34001ee0 con 0x7efc401042a0 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(533..533 src has 69..533) v4 ==== 8071+0+0 (unknown 3187821597 0 0) 0x7efc34088130 con 0x7efc401042a0 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 osd_map(533..533 src has 69..533) v4 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 3 client.49538.objecter handle_osd_map got epochs [533,533] > 0 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 3 client.49538.objecter handle_osd_map decoding full epoch 533 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 20 client.49538.objecter dump_active .. 0 homeless 2024-09-10T19:02:37.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v533) v1 ==== 54+0+16 (unknown 1636486758 0 2420000192) 0x7efc3408a520 con 0x7efc401042a0 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 10 client.49538.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 1 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 1 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 0 1 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 4 2024-09-10T19:02:37.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 10 client.49538.objecter _send_command 1 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc4010d980 con 0x7efc40108280 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc280716f0 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc40108280 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc4010d980 con 0x7efc40108280 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 10 client.49538.objecter _finish_command 1 = system:0 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 4 2024-09-10T19:02:37.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.820+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 0 1 2024-09-10T19:02:37.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 10 client.49538.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 2 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 2 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc4004bf60 osd=0 3 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 0 2 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 4 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 10 client.49538.objecter _send_command 2 2024-09-10T19:02:37.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.840+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc40108d60 con 0x7efc40108280 2024-09-10T19:02:37.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc40108d60 con 0x7efc40108280 2024-09-10T19:02:37.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 10 client.49538.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 4 2024-09-10T19:02:37.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 0 2 2024-09-10T19:02:37.848 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 10 client.49538.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 1 3 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 4 2024-09-10T19:02:37.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 10 client.49538.objecter _send_command 3 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc4002ade0 con 0x7efc40121bc0 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc40121bc0 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc4002ade0 con 0x7efc40121bc0 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 10 client.49538.objecter _finish_command 3 = system:0 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 4 2024-09-10T19:02:37.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.844+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 1 3 2024-09-10T19:02:37.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 10 client.49538.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 4 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 4 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40109170 osd=1 3 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 1 4 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 4 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 10 client.49538.objecter _send_command 4 2024-09-10T19:02:37.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.860+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc400664c0 con 0x7efc40121bc0 2024-09-10T19:02:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc400664c0 con 0x7efc40121bc0 2024-09-10T19:02:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 10 client.49538.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 4 2024-09-10T19:02:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 1 4 2024-09-10T19:02:37.869 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:02:37.869 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:02:37.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 10 client.49538.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 5 2024-09-10T19:02:37.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 5 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 2 5 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 4 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 10 client.49538.objecter _send_command 5 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc40129f70 con 0x7efc4005e820 2024-09-10T19:02:37.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.864+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc4005e820 2024-09-10T19:02:37.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.868+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc40129f70 con 0x7efc4005e820 2024-09-10T19:02:37.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.868+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:02:37.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.868+0000 7efc2effd640 10 client.49538.objecter _finish_command 5 = system:0 2024-09-10T19:02:37.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.868+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 4 2024-09-10T19:02:37.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.868+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 2 5 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.884+0000 7efc44863640 10 client.49538.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.884+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 6 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.884+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 6 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc400668d0 osd=2 3 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 2 6 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 4 2024-09-10T19:02:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 10 client.49538.objecter _send_command 6 2024-09-10T19:02:37.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc4005dd90 con 0x7efc4005e820 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc4005dd90 con 0x7efc4005e820 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 10 client.49538.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 4 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 2 6 2024-09-10T19:02:37.892 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 10 client.49538.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 7 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 7 2024-09-10T19:02:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 3 7 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 4 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 10 client.49538.objecter _send_command 7 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc4012e990 con 0x7efc40126390 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc40126390 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc4012e990 con 0x7efc40126390 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:02:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 10 client.49538.objecter _finish_command 7 = system:0 2024-09-10T19:02:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 4 2024-09-10T19:02:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.888+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 3 7 2024-09-10T19:02:37.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:37.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 10 client.49538.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 8 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 8 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc4005e1a0 osd=3 3 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 3 8 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 4 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 10 client.49538.objecter _send_command 8 2024-09-10T19:02:37.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.912+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc40125900 con 0x7efc40126390 2024-09-10T19:02:37.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc40125900 con 0x7efc40126390 2024-09-10T19:02:37.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 10 client.49538.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 4 2024-09-10T19:02:37.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 3 8 2024-09-10T19:02:37.920 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:02:37.920 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 10 client.49538.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 9 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 9 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 4 9 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 4 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 10 client.49538.objecter _send_command 9 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc401258a0 con 0x7efc4012adb0 2024-09-10T19:02:37.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.916+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc4012adb0 2024-09-10T19:02:37.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.920+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc401258a0 con 0x7efc4012adb0 2024-09-10T19:02:37.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.920+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:02:37.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.920+0000 7efc2effd640 10 client.49538.objecter _finish_command 9 = system:0 2024-09-10T19:02:37.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.920+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 4 2024-09-10T19:02:37.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.920+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 4 9 2024-09-10T19:02:37.924 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:37.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 10 client.49538.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 10 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 10 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc401256c0 osd=4 3 2024-09-10T19:02:37.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 4 10 2024-09-10T19:02:37.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 4 2024-09-10T19:02:37.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 10 client.49538.objecter _send_command 10 2024-09-10T19:02:37.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.932+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc401258a0 con 0x7efc4012adb0 2024-09-10T19:02:37.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc401258a0 con 0x7efc4012adb0 2024-09-10T19:02:37.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc2effd640 10 client.49538.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 4 2024-09-10T19:02:37.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 4 10 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 10 client.49538.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 11 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 11 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 5 11 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 4 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 10 client.49538.objecter _send_command 11 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.936+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc40137640 con 0x7efc40133a30 2024-09-10T19:02:37.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc40133a30 2024-09-10T19:02:37.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc40137640 con 0x7efc40133a30 2024-09-10T19:02:37.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:02:37.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 10 client.49538.objecter _finish_command 11 = system:0 2024-09-10T19:02:37.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 4 2024-09-10T19:02:37.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.940+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 5 11 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 10 client.49538.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 12 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 12 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc401333b0 osd=5 3 2024-09-10T19:02:37.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 5 12 2024-09-10T19:02:37.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 4 2024-09-10T19:02:37.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 10 client.49538.objecter _send_command 12 2024-09-10T19:02:37.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.956+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc40138ee0 con 0x7efc40133a30 2024-09-10T19:02:37.962 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc40138ee0 con 0x7efc40133a30 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 10 client.49538.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 4 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 5 12 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 10 client.49538.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 13 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 13 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 6 13 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 4 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 10 client.49538.objecter _send_command 13 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc40139ad0 con 0x7efc401395a0 2024-09-10T19:02:37.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc401395a0 2024-09-10T19:02:37.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc40139ad0 con 0x7efc401395a0 2024-09-10T19:02:37.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:02:37.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 10 client.49538.objecter _finish_command 13 = system:0 2024-09-10T19:02:37.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 4 2024-09-10T19:02:37.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.960+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 6 13 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 10 client.49538.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 14 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 14 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40138ee0 osd=6 3 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 6 14 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 4 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 10 client.49538.objecter _send_command 14 2024-09-10T19:02:37.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.984+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc40139ad0 con 0x7efc401395a0 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc40139ad0 con 0x7efc401395a0 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 10 client.49538.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 4 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 6 14 2024-09-10T19:02:37.991 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 10 client.49538.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 15 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 15 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 7 15 2024-09-10T19:02:37.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 4 2024-09-10T19:02:37.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 10 client.49538.objecter _send_command 15 2024-09-10T19:02:37.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc40139ed0 con 0x7efc4013c5f0 2024-09-10T19:02:37.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.988+0000 7efc2effd640 10 client.49538.objecter ms_handle_connect 0x7efc4013c5f0 2024-09-10T19:02:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.992+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7efc40139ed0 con 0x7efc4013c5f0 2024-09-10T19:02:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.992+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:02:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.992+0000 7efc2effd640 10 client.49538.objecter _finish_command 15 = system:0 2024-09-10T19:02:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.992+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 4 2024-09-10T19:02:37.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:37.992+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 7 15 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 10 client.49538.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign -1 16 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 20 client.49538.objecter _get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 15 client.49538.objecter _session_command_op_remove -1 16 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 20 client.49538.objecter get_session s=0x7efc40139ad0 osd=7 3 2024-09-10T19:02:38.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 15 client.49538.objecter _session_command_op_assign 7 16 2024-09-10T19:02:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 4 2024-09-10T19:02:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 10 client.49538.objecter _send_command 16 2024-09-10T19:02:38.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.012+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc40139ed0 con 0x7efc4013c5f0 2024-09-10T19:02:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc2effd640 1 -- 172.21.15.83:0/1904481941 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7efc40139ed0 con 0x7efc4013c5f0 2024-09-10T19:02:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc2effd640 10 client.49538.objecter ms_dispatch 0x7efc40000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc2effd640 10 client.49538.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc2effd640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 4 2024-09-10T19:02:38.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc2effd640 15 client.49538.objecter _session_command_op_remove 7 16 2024-09-10T19:02:38.020 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 librados: watch_flush enter 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 librados: watch_flush exit 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.0 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6801/2535527483 conn(0x7efc40108280 legacy=0x7efc40108650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4004bf60 osd=0 1 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.1 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6805/2309624637 conn(0x7efc40121bc0 legacy=0x7efc40123fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40109170 osd=1 1 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.2 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6809/386094220 conn(0x7efc4005e820 legacy=0x7efc40128850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc3ed9d640 1 -- 172.21.15.83:0/1904481941 reap_dead start 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc400668d0 osd=2 1 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.3 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6813/1279688021 conn(0x7efc40126390 legacy=0x7efc4012d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc4005e1a0 osd=3 1 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.4 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.165:6800/3225515265 conn(0x7efc4012adb0 legacy=0x7efc40131c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401256c0 osd=4 1 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.5 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.165:6804/571173777 conn(0x7efc40133a30 legacy=0x7efc40135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc401333b0 osd=5 1 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.6 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.165:6808/1317922936 conn(0x7efc401395a0 legacy=0x7efc40137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40138ee0 osd=6 1 2024-09-10T19:02:38.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter close_session for osd.7 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.165:6812/1610151650 conn(0x7efc4013c5f0 legacy=0x7efc4013e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter put_session s=0x7efc40139ad0 osd=7 1 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 20 client.49538.objecter shutdown clearing up homeless session... 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 10 client.49538.objecter successfully canceled tick 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc3ed9d640 1 -- 172.21.15.83:0/1904481941 reap_dead start 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6800/3071793735 conn(0x7efc280716f0 legacy=0x7efc28073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> v1:172.21.15.83:6790/0 conn(0x7efc401042a0 legacy=0x7efc40119a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 shutdown_connections 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 >> 172.21.15.83:0/1904481941 conn(0x7efc400f7070 msgr2=0x7efc40065b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 shutdown_connections 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 -- 172.21.15.83:0/1904481941 wait complete. 2024-09-10T19:02:38.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:38.016+0000 7efc44863640 1 librados: shutdown 2024-09-10T19:02:38.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:38.053 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:38.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:38.192 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:38.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:38.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:02:38.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:02:38.196 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-10T19:02:38.196 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.543847 2024-09-10T19:02:38.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 0.5438472847761285 2024-09-10T19:02:38.295 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:38.328 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:38.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.530586+0000 osd.0 (osd.0) 285 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.534110+0000 osd.0 (osd.0) 286 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.990769+0000 osd.1 (osd.1) 301 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:36.994175+0000 osd.1 (osd.1) 302 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:37.408103+0000 osd.2 (osd.2) 227 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: cluster 2024-09-10T19:02:37.412051+0000 osd.2 (osd.2) 228 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:02:38.444 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[18629]: audit 2024-09-10T19:02:37.823725+0000 mon.c (mon.2) 657 : audit [DBG] from='client.? v1:172.21.15.83:0/1904481941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.530586+0000 osd.0 (osd.0) 285 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.534110+0000 osd.0 (osd.0) 286 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.990769+0000 osd.1 (osd.1) 301 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:36.994175+0000 osd.1 (osd.1) 302 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:37.408103+0000 osd.2 (osd.2) 227 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: cluster 2024-09-10T19:02:37.412051+0000 osd.2 (osd.2) 228 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:02:38.445 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:38 smithi083 bash[32381]: audit 2024-09-10T19:02:37.823725+0000 mon.c (mon.2) 657 : audit [DBG] from='client.? v1:172.21.15.83:0/1904481941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:38.462 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:38.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:38.594 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.530586+0000 osd.0 (osd.0) 285 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.534110+0000 osd.0 (osd.0) 286 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.990769+0000 osd.1 (osd.1) 301 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:36.994175+0000 osd.1 (osd.1) 302 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:37.408103+0000 osd.2 (osd.2) 227 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:02:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: cluster 2024-09-10T19:02:37.412051+0000 osd.2 (osd.2) 228 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:02:38.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[22591]: audit 2024-09-10T19:02:37.823725+0000 mon.c (mon.2) 657 : audit [DBG] from='client.? v1:172.21.15.83:0/1904481941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:38.613 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:38.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:38.729 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:38.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:38.868 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:38.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:38.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:38.999 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:39.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:39.140 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:39.243 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:39.271 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:39.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:39.397 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:39.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:39.535 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:39.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:37.539248+0000 osd.0 (osd.0) 287 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:37.543321+0000 osd.0 (osd.0) 288 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: audit 2024-09-10T19:02:38.112423+0000 mgr.y (mgr.24526) 535 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.180440+0000 osd.6 (osd.6) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.180775+0000 osd.6 (osd.6) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.187244+0000 mgr.y (mgr.24526) 536 : cluster [DBG] pgmap v818: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.341114+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:37.539248+0000 osd.0 (osd.0) 287 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:37.543321+0000 osd.0 (osd.0) 288 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: audit 2024-09-10T19:02:38.112423+0000 mgr.y (mgr.24526) 535 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.180440+0000 osd.6 (osd.6) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:39.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.180775+0000 osd.6 (osd.6) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:39.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.187244+0000 mgr.y (mgr.24526) 536 : cluster [DBG] pgmap v818: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:39.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:39 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.341114+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:02:39.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:37.539248+0000 osd.0 (osd.0) 287 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:02:39.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:37.543321+0000 osd.0 (osd.0) 288 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:02:39.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: audit 2024-09-10T19:02:38.112423+0000 mgr.y (mgr.24526) 535 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.180440+0000 osd.6 (osd.6) 253 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:02:39.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.180775+0000 osd.6 (osd.6) 254 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:02:39.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.187244+0000 mgr.y (mgr.24526) 536 : cluster [DBG] pgmap v818: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:02:39.615 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:39 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.341114+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:02:39.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:39.676 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:39.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:39.814 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:39.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:39.953 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:39.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:40.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:40.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:40.195 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:40.223 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:40.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:40.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:40.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.986907+0000 osd.1 (osd.1) 303 : cluster [DBG] 64.16 scrub starts 2024-09-10T19:02:40.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:38.991385+0000 osd.1 (osd.1) 304 : cluster [DBG] 64.16 scrub ok 2024-09-10T19:02:40.480 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.353577+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: audit 2024-09-10T19:02:39.356775+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: audit 2024-09-10T19:02:39.360400+0000 mon.a (mon.0) 2832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.439679+0000 osd.2 (osd.2) 229 : cluster [DBG] 6.f scrub starts 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.443167+0000 osd.2 (osd.2) 230 : cluster [DBG] 6.f scrub ok 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.581118+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.584439+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.854067+0000 osd.6 (osd.6) 255 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.859091+0000 osd.6 (osd.6) 256 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.035487+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.038946+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:02:40.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.188038+0000 mgr.y (mgr.24526) 537 : cluster [DBG] pgmap v821: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.986907+0000 osd.1 (osd.1) 303 : cluster [DBG] 64.16 scrub starts 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:38.991385+0000 osd.1 (osd.1) 304 : cluster [DBG] 64.16 scrub ok 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.353577+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: audit 2024-09-10T19:02:39.356775+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: audit 2024-09-10T19:02:39.360400+0000 mon.a (mon.0) 2832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.439679+0000 osd.2 (osd.2) 229 : cluster [DBG] 6.f scrub starts 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.443167+0000 osd.2 (osd.2) 230 : cluster [DBG] 6.f scrub ok 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.581118+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.584439+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.854067+0000 osd.6 (osd.6) 255 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.859091+0000 osd.6 (osd.6) 256 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.035487+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:02:40.482 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.038946+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:02:40.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:40 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.188038+0000 mgr.y (mgr.24526) 537 : cluster [DBG] pgmap v821: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:40.499 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:40.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:40.634 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:40.683 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:40.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:40.772 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.986907+0000 osd.1 (osd.1) 303 : cluster [DBG] 64.16 scrub starts 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:38.991385+0000 osd.1 (osd.1) 304 : cluster [DBG] 64.16 scrub ok 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.353577+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: audit 2024-09-10T19:02:39.356775+0000 mon.b (mon.1) 407 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: audit 2024-09-10T19:02:39.360400+0000 mon.a (mon.0) 2832 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.439679+0000 osd.2 (osd.2) 229 : cluster [DBG] 6.f scrub starts 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.443167+0000 osd.2 (osd.2) 230 : cluster [DBG] 6.f scrub ok 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.581118+0000 osd.5 (osd.5) 298 : cluster [DBG] 6.a scrub starts 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.584439+0000 osd.5 (osd.5) 299 : cluster [DBG] 6.a scrub ok 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.854067+0000 osd.6 (osd.6) 255 : cluster [DBG] 64.17 scrub starts 2024-09-10T19:02:40.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.859091+0000 osd.6 (osd.6) 256 : cluster [DBG] 64.17 scrub ok 2024-09-10T19:02:40.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.035487+0000 osd.7 (osd.7) 309 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:02:40.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.038946+0000 osd.7 (osd.7) 310 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:02:40.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:40 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.188038+0000 mgr.y (mgr.24526) 537 : cluster [DBG] pgmap v821: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:40.874 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:40.909 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:40.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:41.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.048 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:41.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.182 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:41.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:41.320 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:41.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:41.454 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:41.558 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.572 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.603671+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:02:41.572 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.607973+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.985748+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:39.991745+0000 osd.1 (osd.1) 306 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.349531+0000 mon.a (mon.0) 2833 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: audit 2024-09-10T19:02:40.355940+0000 mon.a (mon.0) 2834 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.356257+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: audit 2024-09-10T19:02:40.372060+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: audit 2024-09-10T19:02:40.375399+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: audit 2024-09-10T19:02:40.379446+0000 mon.a (mon.0) 2836 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.465234+0000 osd.2 (osd.2) 231 : cluster [DBG] 12.a scrub starts 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.468909+0000 osd.2 (osd.2) 232 : cluster [DBG] 12.a scrub ok 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.581149+0000 osd.5 (osd.5) 300 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:02:41.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.587124+0000 osd.5 (osd.5) 301 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.822960+0000 osd.6 (osd.6) 257 : cluster [DBG] 3.a scrub starts 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:40.828302+0000 osd.6 (osd.6) 258 : cluster [DBG] 3.a scrub ok 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.052665+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.056064+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.603671+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.607973+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.985748+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:39.991745+0000 osd.1 (osd.1) 306 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.349531+0000 mon.a (mon.0) 2833 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:41.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: audit 2024-09-10T19:02:40.355940+0000 mon.a (mon.0) 2834 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.356257+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: audit 2024-09-10T19:02:40.372060+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: audit 2024-09-10T19:02:40.375399+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: audit 2024-09-10T19:02:40.379446+0000 mon.a (mon.0) 2836 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.465234+0000 osd.2 (osd.2) 231 : cluster [DBG] 12.a scrub starts 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.468909+0000 osd.2 (osd.2) 232 : cluster [DBG] 12.a scrub ok 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.581149+0000 osd.5 (osd.5) 300 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.587124+0000 osd.5 (osd.5) 301 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.822960+0000 osd.6 (osd.6) 257 : cluster [DBG] 3.a scrub starts 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:40.828302+0000 osd.6 (osd.6) 258 : cluster [DBG] 3.a scrub ok 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.052665+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:02:41.575 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:41 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.056064+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:02:41.589 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.603671+0000 osd.0 (osd.0) 289 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.607973+0000 osd.0 (osd.0) 290 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.985748+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:39.991745+0000 osd.1 (osd.1) 306 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.349531+0000 mon.a (mon.0) 2833 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:41.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: audit 2024-09-10T19:02:40.355940+0000 mon.a (mon.0) 2834 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.356257+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: audit 2024-09-10T19:02:40.372060+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: audit 2024-09-10T19:02:40.375399+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: audit 2024-09-10T19:02:40.379446+0000 mon.a (mon.0) 2836 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.465234+0000 osd.2 (osd.2) 231 : cluster [DBG] 12.a scrub starts 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.468909+0000 osd.2 (osd.2) 232 : cluster [DBG] 12.a scrub ok 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.581149+0000 osd.5 (osd.5) 300 : cluster [DBG] 64.1 scrub starts 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.587124+0000 osd.5 (osd.5) 301 : cluster [DBG] 64.1 scrub ok 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.822960+0000 osd.6 (osd.6) 257 : cluster [DBG] 3.a scrub starts 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:40.828302+0000 osd.6 (osd.6) 258 : cluster [DBG] 3.a scrub ok 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.052665+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:02:41.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:41 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.056064+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:02:41.692 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:41.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:41.861 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:41.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:41.945 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:02:41.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:42.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:42.127 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:42.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:42.266 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:42.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:42.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: audit 2024-09-10T19:02:41.376686+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: audit 2024-09-10T19:02:41.377270+0000 mon.a (mon.0) 2837 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.377462+0000 mon.a (mon.0) 2838 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: audit 2024-09-10T19:02:41.379780+0000 mon.a (mon.0) 2839 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.576170+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.580806+0000 osd.5 (osd.5) 303 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.188850+0000 mgr.y (mgr.24526) 538 : cluster [DBG] pgmap v824: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: audit 2024-09-10T19:02:42.378994+0000 mon.a (mon.0) 2840 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:42.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.379125+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:02:42.395 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:42.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:42.534 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:42.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:42.647 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: audit 2024-09-10T19:02:41.376686+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.647 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: audit 2024-09-10T19:02:41.377270+0000 mon.a (mon.0) 2837 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:42.647 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.377462+0000 mon.a (mon.0) 2838 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:02:42.647 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: audit 2024-09-10T19:02:41.379780+0000 mon.a (mon.0) 2839 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.648 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.576170+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:02:42.648 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.580806+0000 osd.5 (osd.5) 303 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:02:42.648 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.188850+0000 mgr.y (mgr.24526) 538 : cluster [DBG] pgmap v824: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-10T19:02:42.648 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: audit 2024-09-10T19:02:42.378994+0000 mon.a (mon.0) 2840 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:42.648 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.379125+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:02:42.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: audit 2024-09-10T19:02:41.376686+0000 mon.b (mon.1) 410 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: audit 2024-09-10T19:02:41.377270+0000 mon.a (mon.0) 2837 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:42.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.377462+0000 mon.a (mon.0) 2838 : cluster [DBG] osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:02:42.651 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: audit 2024-09-10T19:02:41.379780+0000 mon.a (mon.0) 2839 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:42.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.576170+0000 osd.5 (osd.5) 302 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:02:42.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.580806+0000 osd.5 (osd.5) 303 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:02:42.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.188850+0000 mgr.y (mgr.24526) 538 : cluster [DBG] pgmap v824: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-09-10T19:02:42.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: audit 2024-09-10T19:02:42.378994+0000 mon.a (mon.0) 2840 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_tier","val": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:02:42.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:42 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.379125+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:02:42.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:42.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- 172.21.15.83:0/2217298914 >> v1:172.21.15.83:6790/0 conn(0x7f05381042a0 legacy=0x7f0538106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:42.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- 172.21.15.83:0/2217298914 shutdown_connections 2024-09-10T19:02:42.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- 172.21.15.83:0/2217298914 >> 172.21.15.83:0/2217298914 conn(0x7f0538075290 msgr2=0x7f05380f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:42.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- 172.21.15.83:0/2217298914 shutdown_connections 2024-09-10T19:02:42.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- 172.21.15.83:0/2217298914 wait complete. 2024-09-10T19:02:42.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 librados: starting msgr at 2024-09-10T19:02:42.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 librados: starting objecter 2024-09-10T19:02:42.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 Processor -- start 2024-09-10T19:02:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- start start 2024-09-10T19:02:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 librados: setting wanted keys 2024-09-10T19:02:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 librados: calling monclient init 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0538065d60 con 0x7f05381042a0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f053811d410 con 0x7f0538107e30 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f053d247640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f053811e5b0 con 0x7f05381014e0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f0536d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f05381014e0 0x7f05380651b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56204/0 (socket says 172.21.15.83:56204) 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f0536d9d640 1 -- 172.21.15.83:0/3875041104 learned_addr learned my addr 172.21.15.83:0/3875041104 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter ms_handle_connect 0x7f05381042a0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter ms_handle_connect 0x7f05381014e0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942330893 0 0) 0x7f0538065d60 con 0x7f05381042a0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0524003720 con 0x7f05381042a0 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter ms_handle_connect 0x7f0538107e30 2024-09-10T19:02:42.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:42.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154443143 0 0) 0x7f053811e5b0 con 0x7f05381014e0 2024-09-10T19:02:42.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0538065d60 con 0x7f05381014e0 2024-09-10T19:02:42.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.756+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573029862 0 0) 0x7f053811d410 con 0x7f0538107e30 2024-09-10T19:02:42.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053811e5b0 con 0x7f0538107e30 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247604650 0 0) 0x7f0524003720 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f053811d410 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0520001910 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3988973702 0 0) 0x7f053811d410 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 >> v1:172.21.15.83:6790/0 conn(0x7f05381014e0 legacy=0x7f05380651b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 >> v1:172.21.15.165:6789/0 conn(0x7f0538107e30 legacy=0x7f053811ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053811f790 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f053811e7a0 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f053d247640 10 client.25470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f053811ec70 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f053d247640 1 librados: init done 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f05200010c0 con 0x7f05381042a0 2024-09-10T19:02:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f05200034c0 con 0x7f05381042a0 2024-09-10T19:02:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518005010 con 0x7f05381042a0 2024-09-10T19:02:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.760+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f0520001ee0 con 0x7f05381042a0 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(538..538 src has 69..538) v4 ==== 8036+0+0 (unknown 563012704 0 0) 0x7f0520087e90 con 0x7f05381042a0 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 10 client.25470.objecter ms_dispatch 0x7f0538000b90 osd_map(538..538 src has 69..538) v4 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 3 client.25470.objecter handle_osd_map got epochs [538,538] > 0 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 3 client.25470.objecter handle_osd_map decoding full epoch 538 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 20 client.25470.objecter dump_active .. 0 homeless 2024-09-10T19:02:42.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.764+0000 7f052f7fe640 10 client.25470.objecter ms_handle_connect 0x7f0524071380 2024-09-10T19:02:42.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.768+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f0520053fb0 con 0x7f05381042a0 2024-09-10T19:02:42.776 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:42.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:42.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:42.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:42.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:42.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:42.960+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f05180052c0 con 0x7f05381042a0 2024-09-10T19:02:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:42] "GET /metrics HTTP/1.1" 200 226458 "" "Prometheus/2.43.0" 2024-09-10T19:02:43.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:43.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:43.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:43.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:43.352 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:43.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.388+0000 7f052f7fe640 1 -- 172.21.15.83:0/3875041104 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v539) v1 ==== 101+0+0 (unknown 1432795979 0 0) 0x7f0520053dd0 con 0x7f05381042a0 2024-09-10T19:02:43.397 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:02:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 10 librados: watch_flush enter 2024-09-10T19:02:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 10 librados: watch_flush exit 2024-09-10T19:02:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 20 client.25470.objecter shutdown clearing up homeless session... 2024-09-10T19:02:43.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 10 client.25470.objecter successfully canceled tick 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 >> v1:172.21.15.83:6800/3071793735 conn(0x7f0524071380 legacy=0x7f0524073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 >> v1:172.21.15.83:6789/0 conn(0x7f05381042a0 legacy=0x7f05381175b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 shutdown_connections 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 >> 172.21.15.83:0/3875041104 conn(0x7f0538075290 msgr2=0x7f053810b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 shutdown_connections 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 -- 172.21.15.83:0/3875041104 wait complete. 2024-09-10T19:02:43.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:43.396+0000 7f053d247640 1 librados: shutdown 2024-09-10T19:02:43.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.677424+0000 osd.0 (osd.0) 291 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: cluster 2024-09-10T19:02:41.680328+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: audit 2024-09-10T19:02:42.379677+0000 mon.b (mon.1) 411 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: audit 2024-09-10T19:02:42.387284+0000 mon.a (mon.0) 2842 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.790415+0000 osd.6 (osd.6) 259 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.794715+0000 osd.6 (osd.6) 260 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:02:43.470 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: audit 2024-09-10T19:02:42.964045+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:43.471 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:43 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.382946+0000 mon.a (mon.0) 2844 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:02:43.489 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:43.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.599 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.677424+0000 osd.0 (osd.0) 291 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:02:43.599 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: cluster 2024-09-10T19:02:41.680328+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:02:43.599 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: audit 2024-09-10T19:02:42.379677+0000 mon.b (mon.1) 411 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.599 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: audit 2024-09-10T19:02:42.387284+0000 mon.a (mon.0) 2842 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.790415+0000 osd.6 (osd.6) 259 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.794715+0000 osd.6 (osd.6) 260 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: audit 2024-09-10T19:02:42.964045+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.382946+0000 mon.a (mon.0) 2844 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.677424+0000 osd.0 (osd.0) 291 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: cluster 2024-09-10T19:02:41.680328+0000 osd.0 (osd.0) 292 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: audit 2024-09-10T19:02:42.379677+0000 mon.b (mon.1) 411 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: audit 2024-09-10T19:02:42.387284+0000 mon.a (mon.0) 2842 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.790415+0000 osd.6 (osd.6) 259 : cluster [DBG] 64.2 scrub starts 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.794715+0000 osd.6 (osd.6) 260 : cluster [DBG] 64.2 scrub ok 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: audit 2024-09-10T19:02:42.964045+0000 mon.a (mon.0) 2843 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:02:43.600 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:43 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.382946+0000 mon.a (mon.0) 2844 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:02:43.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:43.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.762 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:43.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:43.900 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:43.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:44.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:44.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:44.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.174 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:44.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:44.309 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:44.328 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:02:44.328 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:02:44.328 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:02:44.328 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:02:44.406 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.654662+0000 osd.0 (osd.0) 293 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:42.658108+0000 osd.0 (osd.0) 294 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.360557+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.365358+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: audit 2024-09-10T19:02:43.391879+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: audit 2024-09-10T19:02:43.392097+0000 mon.a (mon.0) 2845 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: audit 2024-09-10T19:02:43.392291+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.392442+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: audit 2024-09-10T19:02:43.396087+0000 mon.a (mon.0) 2848 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.755498+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.a scrub starts 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.758385+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.a scrub ok 2024-09-10T19:02:44.407 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.189696+0000 mgr.y (mgr.24526) 539 : cluster [DBG] pgmap v827: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: audit 2024-09-10T19:02:44.204970+0000 mon.a (mon.0) 2849 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.205112+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.654662+0000 osd.0 (osd.0) 293 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:42.658108+0000 osd.0 (osd.0) 294 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.360557+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.365358+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: audit 2024-09-10T19:02:43.391879+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: audit 2024-09-10T19:02:43.392097+0000 mon.a (mon.0) 2845 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: audit 2024-09-10T19:02:43.392291+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:44.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.392442+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: audit 2024-09-10T19:02:43.396087+0000 mon.a (mon.0) 2848 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.755498+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.a scrub starts 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.758385+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.a scrub ok 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.189696+0000 mgr.y (mgr.24526) 539 : cluster [DBG] pgmap v827: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: audit 2024-09-10T19:02:44.204970+0000 mon.a (mon.0) 2849 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:44.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:44 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.205112+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:02:44.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:44.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.582 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.654662+0000 osd.0 (osd.0) 293 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:02:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:42.658108+0000 osd.0 (osd.0) 294 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:02:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.360557+0000 osd.4 (osd.4) 320 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.365358+0000 osd.4 (osd.4) 321 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: audit 2024-09-10T19:02:43.391879+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: audit 2024-09-10T19:02:43.392097+0000 mon.a (mon.0) 2845 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: audit 2024-09-10T19:02:43.392291+0000 mon.a (mon.0) 2846 : audit [INF] from='client.? v1:172.21.15.83:0/3875041104' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.392442+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: audit 2024-09-10T19:02:43.396087+0000 mon.a (mon.0) 2848 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.755498+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.a scrub starts 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.758385+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.a scrub ok 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.189696+0000 mgr.y (mgr.24526) 539 : cluster [DBG] pgmap v827: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: audit 2024-09-10T19:02:44.204970+0000 mon.a (mon.0) 2849 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:44 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.205112+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:02:44.686 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.720 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:44.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.833 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:44.855 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:44.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.994 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:44.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:45.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:02:45.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:45.129 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:45.233 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:45.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:45.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:45.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:45.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:45.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.667571+0000 osd.0 (osd.0) 295 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:02:45.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: cluster 2024-09-10T19:02:43.672156+0000 osd.0 (osd.0) 296 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:02:45.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: audit 2024-09-10T19:02:44.575190+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:45.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: audit 2024-09-10T19:02:44.577976+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.678966+0000 osd.5 (osd.5) 304 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.683982+0000 osd.5 (osd.5) 305 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[32381]: cluster 2024-09-10T19:02:45.219598+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.667571+0000 osd.0 (osd.0) 295 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: cluster 2024-09-10T19:02:43.672156+0000 osd.0 (osd.0) 296 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: audit 2024-09-10T19:02:44.575190+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: audit 2024-09-10T19:02:44.577976+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.678966+0000 osd.5 (osd.5) 304 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.683982+0000 osd.5 (osd.5) 305 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:02:45.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:45 smithi083 bash[18629]: cluster 2024-09-10T19:02:45.219598+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:02:45.530 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:45.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:45.670 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:45.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:45.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.667571+0000 osd.0 (osd.0) 295 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:02:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: cluster 2024-09-10T19:02:43.672156+0000 osd.0 (osd.0) 296 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:02:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: audit 2024-09-10T19:02:44.575190+0000 mon.a (mon.0) 2851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:02:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: audit 2024-09-10T19:02:44.577976+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.678966+0000 osd.5 (osd.5) 304 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:02:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.683982+0000 osd.5 (osd.5) 305 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:02:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:45 smithi165 bash[22591]: cluster 2024-09-10T19:02:45.219598+0000 mon.a (mon.0) 2852 : cluster [DBG] osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:02:45.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:45.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:45.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:46.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:46.080 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:46.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.219 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:46.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.357 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:02:46.359 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:46.462 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:46.488 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:46.592 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.630614+0000 osd.0 (osd.0) 297 : cluster [DBG] 12.c scrub starts 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.639293+0000 osd.0 (osd.0) 298 : cluster [DBG] 12.c scrub ok 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.793176+0000 osd.3 (osd.3) 311 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:44.800548+0000 osd.3 (osd.3) 312 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:45.677983+0000 osd.5 (osd.5) 306 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:45.681406+0000 osd.5 (osd.5) 307 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.191989+0000 mgr.y (mgr.24526) 540 : cluster [DBG] pgmap v830: 238 pgs: 238 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.221565+0000 mon.a (mon.0) 2853 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: audit 2024-09-10T19:02:46.270554+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: audit 2024-09-10T19:02:46.272400+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: audit 2024-09-10T19:02:46.273525+0000 mon.a (mon.0) 2854 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:46 smithi165 bash[22591]: audit 2024-09-10T19:02:46.275160+0000 mon.a (mon.0) 2855 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.620 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.630614+0000 osd.0 (osd.0) 297 : cluster [DBG] 12.c scrub starts 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.639293+0000 osd.0 (osd.0) 298 : cluster [DBG] 12.c scrub ok 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.793176+0000 osd.3 (osd.3) 311 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:44.800548+0000 osd.3 (osd.3) 312 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:45.677983+0000 osd.5 (osd.5) 306 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:45.681406+0000 osd.5 (osd.5) 307 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.191989+0000 mgr.y (mgr.24526) 540 : cluster [DBG] pgmap v830: 238 pgs: 238 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.221565+0000 mon.a (mon.0) 2853 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: audit 2024-09-10T19:02:46.270554+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: audit 2024-09-10T19:02:46.272400+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.622 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: audit 2024-09-10T19:02:46.273525+0000 mon.a (mon.0) 2854 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[32381]: audit 2024-09-10T19:02:46.275160+0000 mon.a (mon.0) 2855 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.630614+0000 osd.0 (osd.0) 297 : cluster [DBG] 12.c scrub starts 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.639293+0000 osd.0 (osd.0) 298 : cluster [DBG] 12.c scrub ok 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.793176+0000 osd.3 (osd.3) 311 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:44.800548+0000 osd.3 (osd.3) 312 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:45.677983+0000 osd.5 (osd.5) 306 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:45.681406+0000 osd.5 (osd.5) 307 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.191989+0000 mgr.y (mgr.24526) 540 : cluster [DBG] pgmap v830: 238 pgs: 238 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.221565+0000 mon.a (mon.0) 2853 : cluster [DBG] osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: audit 2024-09-10T19:02:46.270554+0000 mon.b (mon.1) 413 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.623 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: audit 2024-09-10T19:02:46.272400+0000 mon.b (mon.1) 414 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: audit 2024-09-10T19:02:46.273525+0000 mon.a (mon.0) 2854 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:46.624 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:46 smithi083 bash[18629]: audit 2024-09-10T19:02:46.275160+0000 mon.a (mon.0) 2855 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-85"}]: dispatch 2024-09-10T19:02:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:46.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.761 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:46.862 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:46.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:46.890 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:46.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:46.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:46.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:47.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:47.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:47.166 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:47.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:47.291 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:47.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:47.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:47.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:47.568 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:47.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:47.694 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:47.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:47.827 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:47.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.340032+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.342848+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.750048+0000 osd.6 (osd.6) 263 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.753092+0000 osd.6 (osd.6) 264 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:47.125996+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.8 deep-scrub starts 2024-09-10T19:02:47.942 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:47.129577+0000 osd.7 (osd.7) 314 : cluster [DBG] 2.8 deep-scrub ok 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[18629]: cluster 2024-09-10T19:02:47.223842+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.340032+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.342848+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.750048+0000 osd.6 (osd.6) 263 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.753092+0000 osd.6 (osd.6) 264 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:47.125996+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.8 deep-scrub starts 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:47.129577+0000 osd.7 (osd.7) 314 : cluster [DBG] 2.8 deep-scrub ok 2024-09-10T19:02:47.943 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:47 smithi083 bash[32381]: cluster 2024-09-10T19:02:47.223842+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:02:47.963 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:47.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:48.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:48.097 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.340032+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.342848+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.750048+0000 osd.6 (osd.6) 263 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.753092+0000 osd.6 (osd.6) 264 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:47.125996+0000 osd.7 (osd.7) 313 : cluster [DBG] 2.8 deep-scrub starts 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:47.129577+0000 osd.7 (osd.7) 314 : cluster [DBG] 2.8 deep-scrub ok 2024-09-10T19:02:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:47 smithi165 bash[22591]: cluster 2024-09-10T19:02:47.223842+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:02:48.201 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:48.245 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:48.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:48.383 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:48.385 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:48.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:48.520 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:48.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:48.657 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:48.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:48.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.599477+0000 osd.0 (osd.0) 299 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:46.602781+0000 osd.0 (osd.0) 300 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:47.639068+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:47.645021+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: audit 2024-09-10T19:02:48.120973+0000 mgr.y (mgr.24526) 541 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.141618+0000 osd.7 (osd.7) 315 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.146421+0000 osd.7 (osd.7) 316 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.192769+0000 mgr.y (mgr.24526) 542 : cluster [DBG] pgmap v833: 206 pgs: 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.228571+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: audit 2024-09-10T19:02:48.235316+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[32381]: audit 2024-09-10T19:02:48.237993+0000 mon.a (mon.0) 2858 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.599477+0000 osd.0 (osd.0) 299 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:46.602781+0000 osd.0 (osd.0) 300 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:47.639068+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:47.645021+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: audit 2024-09-10T19:02:48.120973+0000 mgr.y (mgr.24526) 541 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.141618+0000 osd.7 (osd.7) 315 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.146421+0000 osd.7 (osd.7) 316 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.192769+0000 mgr.y (mgr.24526) 542 : cluster [DBG] pgmap v833: 206 pgs: 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.228571+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: audit 2024-09-10T19:02:48.235316+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:48.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:48 smithi083 bash[18629]: audit 2024-09-10T19:02:48.237993+0000 mon.a (mon.0) 2858 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:48.795 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:48.899 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:48.925 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:48.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:49.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:49.040 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.599477+0000 osd.0 (osd.0) 299 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:46.602781+0000 osd.0 (osd.0) 300 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:47.639068+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 deep-scrub starts 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:47.645021+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.1 deep-scrub ok 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: audit 2024-09-10T19:02:48.120973+0000 mgr.y (mgr.24526) 541 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.141618+0000 osd.7 (osd.7) 315 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.146421+0000 osd.7 (osd.7) 316 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.192769+0000 mgr.y (mgr.24526) 542 : cluster [DBG] pgmap v833: 206 pgs: 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.228571+0000 mon.a (mon.0) 2857 : cluster [DBG] osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: audit 2024-09-10T19:02:48.235316+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:49.041 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:48 smithi165 bash[22591]: audit 2024-09-10T19:02:48.237993+0000 mon.a (mon.0) 2858 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:49.059 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:49.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:49.196 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:49.237 INFO:tasks.workunit.client.0.smithi083.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18086 ms) 2024-09-10T19:02:49.237 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23099 ms) 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2835 ms) 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3016 ms) 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3037 ms) 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3050 ms) 2024-09-10T19:02:49.238 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4018 ms) 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40006 ms) 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17047 ms) 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4051 ms) 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-10T19:02:49.239 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2825 ms) 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25195 ms) 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14211 ms) 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5026 ms) 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.240 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3578 ms) 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrubs... 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25105 ms) 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9960 ms) 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9093 ms) 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-10T19:02:49.241 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:02:49.242 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8880 ms) 2024-09-10T19:02:49.242 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-10T19:02:49.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:49.329 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:49.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:49.464 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:49.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:49.602 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:49.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:49.730 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:49.818 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:49.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.199209+0000 mon.a (mon.0) 2859 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[18629]: audit 2024-09-10T19:02:49.230895+0000 mon.a (mon.0) 2860 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.231142+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[18629]: audit 2024-09-10T19:02:49.232058+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.199209+0000 mon.a (mon.0) 2859 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[32381]: audit 2024-09-10T19:02:49.230895+0000 mon.a (mon.0) 2860 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.231142+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:02:49.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:49 smithi083 bash[32381]: audit 2024-09-10T19:02:49.232058+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:49.864 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:49.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:49.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:49.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:49 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.199209+0000 mon.a (mon.0) 2859 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:49.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:49 smithi165 bash[22591]: audit 2024-09-10T19:02:49.230895+0000 mon.a (mon.0) 2860 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:49.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:49 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.231142+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:02:49.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:49 smithi165 bash[22591]: audit 2024-09-10T19:02:49.232058+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:49.998 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:50.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- 172.21.15.83:0/3118719435 >> v1:172.21.15.83:6789/0 conn(0x7fc9081014e0 legacy=0x7fc9081018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:50.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- 172.21.15.83:0/3118719435 shutdown_connections 2024-09-10T19:02:50.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- 172.21.15.83:0/3118719435 >> 172.21.15.83:0/3118719435 conn(0x7fc908075290 msgr2=0x7fc9080f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- 172.21.15.83:0/3118719435 shutdown_connections 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- 172.21.15.83:0/3118719435 wait complete. 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 librados: starting msgr at 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 librados: starting objecter 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 Processor -- start 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- start start 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 librados: setting wanted keys 2024-09-10T19:02:50.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 librados: calling monclient init 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc90811e160 con 0x7fc9081042a0 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc90811f320 con 0x7fc908107e30 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc90ea36640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc908120520 con 0x7fc9081014e0 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc907fff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc9081014e0 0x7fc9080fd050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35110/0 (socket says 172.21.15.83:35110) 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc907fff640 1 -- 172.21.15.83:0/829902274 learned_addr learned my addr 172.21.15.83:0/829902274 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:50.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc9077fe640 1 --1- 172.21.15.83:0/829902274 >> v1:172.21.15.83:6789/0 conn(0x7fc9081042a0 0x7fc9080fd760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:40546/0 (socket says 172.21.15.83:40546) 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter ms_handle_connect 0x7fc9081014e0 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter ms_handle_connect 0x7fc9081042a0 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter ms_handle_connect 0x7fc908107e30 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471190514 0 0) 0x7fc90811e160 con 0x7fc9081042a0 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8f40034e0 con 0x7fc9081042a0 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737252248 0 0) 0x7fc90811f320 con 0x7fc908107e30 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90811e160 con 0x7fc908107e30 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242043398 0 0) 0x7fc908120520 con 0x7fc9081014e0 2024-09-10T19:02:50.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90811f320 con 0x7fc9081014e0 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262722456 0 0) 0x7fc8f40034e0 con 0x7fc9081042a0 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc908120520 con 0x7fc9081042a0 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.040+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc8f00018c0 con 0x7fc9081042a0 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543356209 0 0) 0x7fc908120520 con 0x7fc9081042a0 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 >> v1:172.21.15.83:6790/0 conn(0x7fc9081014e0 legacy=0x7fc9080fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:50.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 >> v1:172.21.15.165:6789/0 conn(0x7fc908107e30 legacy=0x7fc90811c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc908121720 con 0x7fc9081042a0 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90811e350 con 0x7fc9081042a0 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc90ea36640 10 client.25476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90811e820 con 0x7fc9081042a0 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc90ea36640 1 librados: init done 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc8f0001f40 con 0x7fc9081042a0 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc8f0005a10 con 0x7fc9081042a0 2024-09-10T19:02:50.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e8005010 con 0x7fc9081042a0 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc8f0001ae0 con 0x7fc9081042a0 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(545..545 src has 69..545) v4 ==== 8004+0+0 (unknown 219695858 0 0) 0x7fc8f008a4f0 con 0x7fc9081042a0 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 10 client.25476.objecter ms_dispatch 0x7fc908000b90 osd_map(545..545 src has 69..545) v4 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 3 client.25476.objecter handle_osd_map got epochs [545,545] > 0 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.044+0000 7fc904ff9640 3 client.25476.objecter handle_osd_map decoding full epoch 545 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.048+0000 7fc904ff9640 20 client.25476.objecter dump_active .. 0 homeless 2024-09-10T19:02:50.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.048+0000 7fc904ff9640 10 client.25476.objecter ms_handle_connect 0x7fc8f4071600 2024-09-10T19:02:50.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.052+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc8f0056b10 con 0x7fc9081042a0 2024-09-10T19:02:50.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:50.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:50.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:50.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:50.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:50.256+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285} v 0) v1 -- 0x7fc8e8005840 con 0x7fc9081042a0 2024-09-10T19:02:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:50.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:50.396 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:50.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:50.532 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:50.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:50.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:50.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:50.787 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.599758+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:48.603241+0000 osd.0 (osd.0) 302 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.370150+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.375053+0000 osd.4 (osd.4) 325 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.892066+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:49.895142+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.193511+0000 mgr.y (mgr.24526) 543 : cluster [DBG] pgmap v836: 238 pgs: 32 unknown, 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.235172+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: audit 2024-09-10T19:02:50.236766+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: audit 2024-09-10T19:02:50.241453+0000 mon.a (mon.0) 2863 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.788 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:50 smithi165 bash[22591]: audit 2024-09-10T19:02:50.259819+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]: dispatch 2024-09-10T19:02:50.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:50.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:50.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.599758+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:48.603241+0000 osd.0 (osd.0) 302 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.370150+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.375053+0000 osd.4 (osd.4) 325 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.892066+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:49.895142+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:02:50.947 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.193511+0000 mgr.y (mgr.24526) 543 : cluster [DBG] pgmap v836: 238 pgs: 32 unknown, 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.235172+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: audit 2024-09-10T19:02:50.236766+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: audit 2024-09-10T19:02:50.241453+0000 mon.a (mon.0) 2863 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[32381]: audit 2024-09-10T19:02:50.259819+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]: dispatch 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.599758+0000 osd.0 (osd.0) 301 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:48.603241+0000 osd.0 (osd.0) 302 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.370150+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.375053+0000 osd.4 (osd.4) 325 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.892066+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:02:50.948 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:49.895142+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:02:50.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.193511+0000 mgr.y (mgr.24526) 543 : cluster [DBG] pgmap v836: 238 pgs: 32 unknown, 206 active+clean; 8.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:50.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.235172+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:02:50.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: audit 2024-09-10T19:02:50.236766+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: audit 2024-09-10T19:02:50.241453+0000 mon.a (mon.0) 2863 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:50.949 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:50 smithi083 bash[18629]: audit 2024-09-10T19:02:50.259819+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]: dispatch 2024-09-10T19:02:51.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:51.079 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:51.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:51.217 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:51.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.240+0000 7fc904ff9640 1 -- 172.21.15.83:0/829902274 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]=0 set osd.4 primary-affinity to 0.543847 (8356412) v547) v1 ==== 155+0+0 (unknown 1730973246 0 0) 0x7fc8f00564b0 con 0x7fc9081042a0 2024-09-10T19:02:51.246 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.4 primary-affinity to 0.543847 (8356412) 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.244+0000 7fc90ea36640 10 librados: watch_flush enter 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.244+0000 7fc90ea36640 10 librados: watch_flush exit 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.244+0000 7fc90ea36640 20 client.25476.objecter shutdown clearing up homeless session... 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.244+0000 7fc90ea36640 10 client.25476.objecter successfully canceled tick 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.248+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc8f4071600 legacy=0x7fc8f4073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:51.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.248+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 >> v1:172.21.15.83:6789/0 conn(0x7fc9081042a0 legacy=0x7fc9080fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:51.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.248+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 shutdown_connections 2024-09-10T19:02:51.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.248+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 >> 172.21.15.83:0/829902274 conn(0x7fc908075290 msgr2=0x7fc908065770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:51.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.252+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 shutdown_connections 2024-09-10T19:02:51.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.252+0000 7fc90ea36640 1 -- 172.21.15.83:0/829902274 wait complete. 2024-09-10T19:02:51.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:51.252+0000 7fc90ea36640 1 librados: shutdown 2024-09-10T19:02:51.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:51.350 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:51.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:51.487 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:51.527 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:51.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:51.618 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:51.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:51.754 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:51.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.622692+0000 osd.2 (osd.2) 235 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:02:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.626289+0000 osd.2 (osd.2) 236 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:02:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.720882+0000 osd.5 (osd.5) 308 : cluster [DBG] 1.a deep-scrub starts 2024-09-10T19:02:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.724347+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.a deep-scrub ok 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.923131+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:50.928005+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: audit 2024-09-10T19:02:51.238906+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: audit 2024-09-10T19:02:51.239515+0000 mon.a (mon.0) 2865 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: audit 2024-09-10T19:02:51.240143+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]': finished 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.240346+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:02:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:51 smithi165 bash[22591]: audit 2024-09-10T19:02:51.243726+0000 mon.a (mon.0) 2868 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:51.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:51.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:51.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.622692+0000 osd.2 (osd.2) 235 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.626289+0000 osd.2 (osd.2) 236 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.720882+0000 osd.5 (osd.5) 308 : cluster [DBG] 1.a deep-scrub starts 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.724347+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.a deep-scrub ok 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.923131+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:50.928005+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: audit 2024-09-10T19:02:51.238906+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: audit 2024-09-10T19:02:51.239515+0000 mon.a (mon.0) 2865 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: audit 2024-09-10T19:02:51.240143+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]': finished 2024-09-10T19:02:52.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.240346+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[18629]: audit 2024-09-10T19:02:51.243726+0000 mon.a (mon.0) 2868 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.622692+0000 osd.2 (osd.2) 235 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.626289+0000 osd.2 (osd.2) 236 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.720882+0000 osd.5 (osd.5) 308 : cluster [DBG] 1.a deep-scrub starts 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.724347+0000 osd.5 (osd.5) 309 : cluster [DBG] 1.a deep-scrub ok 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.923131+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:50.928005+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: audit 2024-09-10T19:02:51.238906+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: audit 2024-09-10T19:02:51.239515+0000 mon.a (mon.0) 2865 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: audit 2024-09-10T19:02:51.240143+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.83:0/829902274' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5438472847761285}]': finished 2024-09-10T19:02:52.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.240346+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:02:52.011 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:51 smithi083 bash[32381]: audit 2024-09-10T19:02:51.243726+0000 mon.a (mon.0) 2868 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]: dispatch 2024-09-10T19:02:52.030 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:52.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.168 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:52.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:52.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:52.398 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.434 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:52.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:52.570 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:52.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:52.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:52.709 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:52.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:52.844 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.513234+0000 osd.0 (osd.0) 303 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.516942+0000 osd.0 (osd.0) 304 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.602317+0000 osd.2 (osd.2) 237 : cluster [DBG] 2.e scrub starts 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.606815+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.e scrub ok 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.689159+0000 osd.5 (osd.5) 310 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.693501+0000 osd.5 (osd.5) 311 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.972350+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:51.975694+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.107291+0000 osd.7 (osd.7) 317 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.111046+0000 osd.7 (osd.7) 318 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.194454+0000 mgr.y (mgr.24526) 544 : cluster [DBG] pgmap v839: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: audit 2024-09-10T19:02:52.239349+0000 mon.a (mon.0) 2869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]': finished 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.239594+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:02:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: audit 2024-09-10T19:02:52.241355+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:52 smithi165 bash[22591]: audit 2024-09-10T19:02:52.246904+0000 mon.a (mon.0) 2871 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:52.963 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:02:52] "GET /metrics HTTP/1.1" 200 228899 "" "Prometheus/2.43.0" 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.513234+0000 osd.0 (osd.0) 303 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.516942+0000 osd.0 (osd.0) 304 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.602317+0000 osd.2 (osd.2) 237 : cluster [DBG] 2.e scrub starts 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.606815+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.e scrub ok 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.689159+0000 osd.5 (osd.5) 310 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.693501+0000 osd.5 (osd.5) 311 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.972350+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:02:52.964 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:51.975694+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.107291+0000 osd.7 (osd.7) 317 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.111046+0000 osd.7 (osd.7) 318 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.194454+0000 mgr.y (mgr.24526) 544 : cluster [DBG] pgmap v839: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: audit 2024-09-10T19:02:52.239349+0000 mon.a (mon.0) 2869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]': finished 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.239594+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: audit 2024-09-10T19:02:52.241355+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[18629]: audit 2024-09-10T19:02:52.246904+0000 mon.a (mon.0) 2871 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.513234+0000 osd.0 (osd.0) 303 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.516942+0000 osd.0 (osd.0) 304 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:02:52.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.602317+0000 osd.2 (osd.2) 237 : cluster [DBG] 2.e scrub starts 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.606815+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.e scrub ok 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.689159+0000 osd.5 (osd.5) 310 : cluster [DBG] 64.12 scrub starts 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.693501+0000 osd.5 (osd.5) 311 : cluster [DBG] 64.12 scrub ok 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.972350+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:51.975694+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.107291+0000 osd.7 (osd.7) 317 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.111046+0000 osd.7 (osd.7) 318 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.194454+0000 mgr.y (mgr.24526) 544 : cluster [DBG] pgmap v839: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: audit 2024-09-10T19:02:52.239349+0000 mon.a (mon.0) 2869 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_tier","val": "test-rados-api-smithi083-77828-89-test-flush"}]': finished 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.239594+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: audit 2024-09-10T19:02:52.241355+0000 mon.b (mon.1) 419 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.966 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:52 smithi083 bash[32381]: audit 2024-09-10T19:02:52.246904+0000 mon.a (mon.0) 2871 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:02:52.982 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:52.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:53.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:53.119 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:53.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:53.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:53.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:53.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:53.376 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:53.380 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:02:53.381 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:02:53.381 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T19:02:53.381 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T19:02:53.381 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:02:53.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:53.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- 172.21.15.83:0/2764222301 >> v1:172.21.15.83:6790/0 conn(0x7f8748103a50 legacy=0x7f8748105e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- 172.21.15.83:0/2764222301 shutdown_connections 2024-09-10T19:02:53.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- 172.21.15.83:0/2764222301 >> 172.21.15.83:0/2764222301 conn(0x7f8748074a20 msgr2=0x7f87480f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:53.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- 172.21.15.83:0/2764222301 shutdown_connections 2024-09-10T19:02:53.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- 172.21.15.83:0/2764222301 wait complete. 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 librados: starting msgr at 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 librados: starting objecter 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 Processor -- start 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 -- start start 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 librados: setting wanted keys 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.403+0000 7f8750399640 1 librados: calling monclient init 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f87481005a0 con 0x7f87481075b0 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f874811d350 con 0x7f8748103a50 2024-09-10T19:02:53.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f874811e4f0 con 0x7f8748100c70 2024-09-10T19:02:53.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f874e135640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f8748100c70 0x7f87480fea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60892/0 (socket says 172.21.15.83:60892) 2024-09-10T19:02:53.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f874e936640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f87481075b0 0x7f874811ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46338/0 (socket says 172.21.15.83:46338) 2024-09-10T19:02:53.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f874e135640 1 -- 172.21.15.83:0/3934553369 learned_addr learned my addr 172.21.15.83:0/3934553369 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:53.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter ms_handle_connect 0x7f8748100c70 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter ms_handle_connect 0x7f87481075b0 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039509314 0 0) 0x7f874811e4f0 con 0x7f8748100c70 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734003120 con 0x7f8748100c70 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter ms_handle_connect 0x7f8748103a50 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028308058 0 0) 0x7f87481005a0 con 0x7f87481075b0 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874811e4f0 con 0x7f87481075b0 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196616440 0 0) 0x7f874811d350 con 0x7f8748103a50 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87481005a0 con 0x7f8748103a50 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578125134 0 0) 0x7f8734003120 con 0x7f8748100c70 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f874811d350 con 0x7f8748100c70 2024-09-10T19:02:53.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8744001990 con 0x7f8748100c70 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2017408441 0 0) 0x7f874811e4f0 con 0x7f87481075b0 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8734003120 con 0x7f87481075b0 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8740001cd0 con 0x7f87481075b0 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1082957191 0 0) 0x7f87481005a0 con 0x7f8748103a50 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f874811e4f0 con 0x7f8748103a50 2024-09-10T19:02:53.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8738001550 con 0x7f8748103a50 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4203483105 0 0) 0x7f8734003120 con 0x7f87481075b0 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6790/0 conn(0x7f8748100c70 legacy=0x7f87480fea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.165:6789/0 conn(0x7f8748103a50 legacy=0x7f87480ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874811f690 con 0x7f87481075b0 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874811c3a0 con 0x7f87481075b0 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 10 client.25482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f874811c7f0 con 0x7f87481075b0 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f8750399640 1 librados: init done 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8740002600 con 0x7f87481075b0 2024-09-10T19:02:53.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.407+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8740005d60 con 0x7f87481075b0 2024-09-10T19:02:53.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f872c004ff0 con 0x7f87481075b0 2024-09-10T19:02:53.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f87400021a0 con 0x7f87481075b0 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(549..549 src has 69..549) v4 ==== 8470+0+0 (unknown 2570604959 0 0) 0x7f874008a8b0 con 0x7f87481075b0 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 osd_map(549..549 src has 69..549) v4 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 3 client.25482.objecter handle_osd_map got epochs [549,549] > 0 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 3 client.25482.objecter handle_osd_map decoding full epoch 549 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 20 client.25482.objecter dump_active .. 0 homeless 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v549) v1 ==== 54+0+16 (unknown 2534082343 0 2420000192) 0x7f874008ac60 con 0x7f87481075b0 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f8734075bd0 2024-09-10T19:02:53.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 10 client.25482.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 1 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 1 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 0 1 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 4 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 10 client.25482.objecter _send_command 1 2024-09-10T19:02:53.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c004a60 con 0x7f872c004610 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c004610 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c004a60 con 0x7f872c004610 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 10 client.25482.objecter _finish_command 1 = system:0 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 4 2024-09-10T19:02:53.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.411+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 0 1 2024-09-10T19:02:53.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 10 client.25482.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 2 2024-09-10T19:02:53.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 2 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c005290 osd=0 3 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 0 2 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 4 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 10 client.25482.objecter _send_command 2 2024-09-10T19:02:53.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.431+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c004e40 con 0x7f872c004610 2024-09-10T19:02:53.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c004e40 con 0x7f872c004610 2024-09-10T19:02:53.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 10 client.25482.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 4 2024-09-10T19:02:53.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 0 2 2024-09-10T19:02:53.439 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:02:53.439 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 10 client.25482.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 1 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 4 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 10 client.25482.objecter _send_command 3 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c0022f0 con 0x7f872c004110 2024-09-10T19:02:53.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c004110 2024-09-10T19:02:53.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c0022f0 con 0x7f872c004110 2024-09-10T19:02:53.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:02:53.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 10 client.25482.objecter _finish_command 3 = system:0 2024-09-10T19:02:53.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 4 2024-09-10T19:02:53.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.435+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 1 3 2024-09-10T19:02:53.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 10 client.25482.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 4 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 4 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c003fa0 osd=1 3 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 1 4 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 4 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 10 client.25482.objecter _send_command 4 2024-09-10T19:02:53.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.451+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c0018f0 con 0x7f872c004110 2024-09-10T19:02:53.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c0018f0 con 0x7f872c004110 2024-09-10T19:02:53.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 10 client.25482.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 4 2024-09-10T19:02:53.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 1 4 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 10 client.25482.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 5 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 5 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 2 5 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 4 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 10 client.25482.objecter _send_command 5 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c003e30 con 0x7f872c00d350 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.455+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c00d350 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.459+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c003e30 con 0x7f872c00d350 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.459+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:02:53.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.459+0000 7f873effd640 10 client.25482.objecter _finish_command 5 = system:0 2024-09-10T19:02:53.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.459+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 4 2024-09-10T19:02:53.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.459+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 2 5 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 10 client.25482.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 6 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 6 2024-09-10T19:02:53.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c0035e0 osd=2 3 2024-09-10T19:02:53.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 2 6 2024-09-10T19:02:53.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 4 2024-09-10T19:02:53.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 10 client.25482.objecter _send_command 6 2024-09-10T19:02:53.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c003060 con 0x7f872c00d350 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c003060 con 0x7f872c00d350 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 10 client.25482.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 4 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 2 6 2024-09-10T19:02:53.481 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 10 client.25482.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 7 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 7 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 3 7 2024-09-10T19:02:53.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 4 2024-09-10T19:02:53.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 10 client.25482.objecter _send_command 7 2024-09-10T19:02:53.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c00d010 con 0x7f872c010e80 2024-09-10T19:02:53.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.475+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c010e80 2024-09-10T19:02:53.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.479+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c00d010 con 0x7f872c010e80 2024-09-10T19:02:53.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.479+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:02:53.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.479+0000 7f873effd640 10 client.25482.objecter _finish_command 7 = system:0 2024-09-10T19:02:53.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.479+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 4 2024-09-10T19:02:53.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.479+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 3 7 2024-09-10T19:02:53.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:53.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 10 client.25482.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 8 2024-09-10T19:02:53.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 8 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c00c990 osd=3 3 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 3 8 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 4 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 10 client.25482.objecter _send_command 8 2024-09-10T19:02:53.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c0132e0 con 0x7f872c010e80 2024-09-10T19:02:53.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c0132e0 con 0x7f872c010e80 2024-09-10T19:02:53.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 10 client.25482.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 4 2024-09-10T19:02:53.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 3 8 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 10 client.25482.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 9 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 9 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 4 9 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 4 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 10 client.25482.objecter _send_command 9 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c01a290 con 0x7f872c016790 2024-09-10T19:02:53.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.503+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c016790 2024-09-10T19:02:53.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.507+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c01a290 con 0x7f872c016790 2024-09-10T19:02:53.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.507+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:02:53.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.507+0000 7f873effd640 10 client.25482.objecter _finish_command 9 = system:0 2024-09-10T19:02:53.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.507+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 4 2024-09-10T19:02:53.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.507+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 4 9 2024-09-10T19:02:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:53.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 10 client.25482.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 10 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 10 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c012b20 osd=4 3 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 4 10 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 4 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 10 client.25482.objecter _send_command 10 2024-09-10T19:02:53.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.519+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c00c410 con 0x7f872c016790 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c00c410 con 0x7f872c016790 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f873effd640 10 client.25482.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 4 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 4 10 2024-09-10T19:02:53.529 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 10 client.25482.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 11 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 11 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 5 11 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 4 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 10 client.25482.objecter _send_command 11 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.523+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c012860 con 0x7f872c00c210 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c00c210 2024-09-10T19:02:53.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c012860 con 0x7f872c00c210 2024-09-10T19:02:53.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:02:53.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 10 client.25482.objecter _finish_command 11 = system:0 2024-09-10T19:02:53.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 4 2024-09-10T19:02:53.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.527+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 5 11 2024-09-10T19:02:53.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 10 client.25482.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 12 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 12 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c00c820 osd=5 3 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 5 12 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 4 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 10 client.25482.objecter _send_command 12 2024-09-10T19:02:53.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.543+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c012860 con 0x7f872c00c210 2024-09-10T19:02:53.554 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:02:53.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.547+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c012860 con 0x7f872c00c210 2024-09-10T19:02:53.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.547+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.547+0000 7f873effd640 10 client.25482.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.547+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 4 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.547+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 5 12 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 10 client.25482.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 13 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 13 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 6 13 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 4 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 10 client.25482.objecter _send_command 13 2024-09-10T19:02:53.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c022cc0 con 0x7f872c01f050 2024-09-10T19:02:53.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c01f050 2024-09-10T19:02:53.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c022cc0 con 0x7f872c01f050 2024-09-10T19:02:53.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:02:53.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 10 client.25482.objecter _finish_command 13 = system:0 2024-09-10T19:02:53.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 4 2024-09-10T19:02:53.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.551+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 6 13 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 10 client.25482.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 14 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 14 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c012860 osd=6 3 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 6 14 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 4 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 10 client.25482.objecter _send_command 14 2024-09-10T19:02:53.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.571+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c024560 con 0x7f872c01f050 2024-09-10T19:02:53.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c024560 con 0x7f872c01f050 2024-09-10T19:02:53.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 10 client.25482.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 4 2024-09-10T19:02:53.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 6 14 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 10 client.25482.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 15 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 15 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 7 15 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 4 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 10 client.25482.objecter _send_command 15 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f872c025100 con 0x7f872c024c10 2024-09-10T19:02:53.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 10 client.25482.objecter ms_handle_connect 0x7f872c024c10 2024-09-10T19:02:53.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f872c025100 con 0x7f872c024c10 2024-09-10T19:02:53.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:02:53.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 10 client.25482.objecter _finish_command 15 = system:0 2024-09-10T19:02:53.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 4 2024-09-10T19:02:53.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.575+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 7 15 2024-09-10T19:02:53.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 10 client.25482.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:02:53.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign -1 16 2024-09-10T19:02:53.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 20 client.25482.objecter _get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 15 client.25482.objecter _session_command_op_remove -1 16 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 20 client.25482.objecter get_session s=0x7f872c024560 osd=7 3 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 15 client.25482.objecter _session_command_op_assign 7 16 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 4 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 10 client.25482.objecter _send_command 16 2024-09-10T19:02:53.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.595+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f872c025100 con 0x7f872c024c10 2024-09-10T19:02:53.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f873effd640 1 -- 172.21.15.83:0/3934553369 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f872c025100 con 0x7f872c024c10 2024-09-10T19:02:53.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f873effd640 10 client.25482.objecter ms_dispatch 0x7f8748000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:02:53.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f873effd640 10 client.25482.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:02:53.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f873effd640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 4 2024-09-10T19:02:53.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f873effd640 15 client.25482.objecter _session_command_op_remove 7 16 2024-09-10T19:02:53.603 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 librados: watch_flush enter 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 librados: watch_flush exit 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.0 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6801/2535527483 conn(0x7f872c004610 legacy=0x7f872c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c005290 osd=0 1 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.1 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6805/2309624637 conn(0x7f872c004110 legacy=0x7f872c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c003fa0 osd=1 1 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.2 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6809/386094220 conn(0x7f872c00d350 legacy=0x7f872c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c0035e0 osd=2 1 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.3 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6813/1279688021 conn(0x7f872c010e80 legacy=0x7f872c011290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c990 osd=3 1 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.4 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.165:6800/3225515265 conn(0x7f872c016790 legacy=0x7f872c018b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012b20 osd=4 1 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.5 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.165:6804/571173777 conn(0x7f872c00c210 legacy=0x7f872c01d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c00c820 osd=5 1 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.6 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.165:6808/1317922936 conn(0x7f872c01f050 legacy=0x7f872c021430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c012860 osd=6 1 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter close_session for osd.7 2024-09-10T19:02:53.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.165:6812/1610151650 conn(0x7f872c024c10 legacy=0x7f872c022cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter put_session s=0x7f872c024560 osd=7 1 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 20 client.25482.objecter shutdown clearing up homeless session... 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 10 client.25482.objecter successfully canceled tick 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f874e936640 1 -- 172.21.15.83:0/3934553369 reap_dead start 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f874e936640 1 -- 172.21.15.83:0/3934553369 reap_dead start 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8734075bd0 legacy=0x7f8734078080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> v1:172.21.15.83:6789/0 conn(0x7f87481075b0 legacy=0x7f874811ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f874e936640 1 -- 172.21.15.83:0/3934553369 reap_dead start 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f874e936640 1 -- 172.21.15.83:0/3934553369 reap_dead start 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f874e936640 1 -- 172.21.15.83:0/3934553369 reap_dead start 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 shutdown_connections 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 >> 172.21.15.83:0/3934553369 conn(0x7f8748074a20 msgr2=0x7f8748070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:53.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 shutdown_connections 2024-09-10T19:02:53.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 -- 172.21.15.83:0/3934553369 wait complete. 2024-09-10T19:02:53.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:53.599+0000 7f8750399640 1 librados: shutdown 2024-09-10T19:02:53.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:53.655 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:53.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:53.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.044660+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.049265+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.533470+0000 osd.0 (osd.0) 305 : cluster [DBG] 12.e scrub starts 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.537500+0000 osd.0 (osd.0) 306 : cluster [DBG] 12.e scrub ok 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.784138+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.e scrub starts 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:52.788335+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.e scrub ok 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.068670+0000 osd.7 (osd.7) 319 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.073995+0000 osd.7 (osd.7) 320 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: audit 2024-09-10T19:02:53.245285+0000 mon.a (mon.0) 2872 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.245385+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: audit 2024-09-10T19:02:53.246313+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: audit 2024-09-10T19:02:53.248947+0000 mon.a (mon.0) 2874 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:53 smithi165 bash[22591]: audit 2024-09-10T19:02:53.415168+0000 mon.a (mon.0) 2875 : audit [DBG] from='client.? v1:172.21.15.83:0/3934553369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:53.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.044660+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:02:53.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.049265+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:02:53.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.533470+0000 osd.0 (osd.0) 305 : cluster [DBG] 12.e scrub starts 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.537500+0000 osd.0 (osd.0) 306 : cluster [DBG] 12.e scrub ok 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.784138+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.e scrub starts 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:52.788335+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.e scrub ok 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.068670+0000 osd.7 (osd.7) 319 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.073995+0000 osd.7 (osd.7) 320 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: audit 2024-09-10T19:02:53.245285+0000 mon.a (mon.0) 2872 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.245385+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: audit 2024-09-10T19:02:53.246313+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: audit 2024-09-10T19:02:53.248947+0000 mon.a (mon.0) 2874 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[32381]: audit 2024-09-10T19:02:53.415168+0000 mon.a (mon.0) 2875 : audit [DBG] from='client.? v1:172.21.15.83:0/3934553369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.044660+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.049265+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.533470+0000 osd.0 (osd.0) 305 : cluster [DBG] 12.e scrub starts 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.537500+0000 osd.0 (osd.0) 306 : cluster [DBG] 12.e scrub ok 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.784138+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.e scrub starts 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:52.788335+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.e scrub ok 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.068670+0000 osd.7 (osd.7) 319 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.073995+0000 osd.7 (osd.7) 320 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: audit 2024-09-10T19:02:53.245285+0000 mon.a (mon.0) 2872 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.245385+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: audit 2024-09-10T19:02:53.246313+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: audit 2024-09-10T19:02:53.248947+0000 mon.a (mon.0) 2874 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:02:53.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:53 smithi083 bash[18629]: audit 2024-09-10T19:02:53.415168+0000 mon.a (mon.0) 2875 : audit [DBG] from='client.? v1:172.21.15.83:0/3934553369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:02:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:53.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:53.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:54.007 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:54.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:02:54.045 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:54.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:54.182 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:54.285 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:54.314 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:54.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:54.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:54.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:54.590 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:54.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:54.728 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.444160+0000 osd.4 (osd.4) 326 : cluster [DBG] 2.f scrub starts 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.457850+0000 osd.4 (osd.4) 327 : cluster [DBG] 2.f scrub ok 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.625872+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:53.636190+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.031562+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.034437+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.195350+0000 mgr.y (mgr.24526) 545 : cluster [DBG] pgmap v842: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.201145+0000 mon.a (mon.0) 2876 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: audit 2024-09-10T19:02:54.211664+0000 mon.a (mon.0) 2877 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.211810+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.444160+0000 osd.4 (osd.4) 326 : cluster [DBG] 2.f scrub starts 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.457850+0000 osd.4 (osd.4) 327 : cluster [DBG] 2.f scrub ok 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.625872+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:53.636190+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.031562+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.034437+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.195350+0000 mgr.y (mgr.24526) 545 : cluster [DBG] pgmap v842: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.201145+0000 mon.a (mon.0) 2876 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: audit 2024-09-10T19:02:54.211664+0000 mon.a (mon.0) 2877 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:54 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.211810+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:02:54.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:54.863 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:54.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:54.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:55.002 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:55.106 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.444160+0000 osd.4 (osd.4) 326 : cluster [DBG] 2.f scrub starts 2024-09-10T19:02:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.457850+0000 osd.4 (osd.4) 327 : cluster [DBG] 2.f scrub ok 2024-09-10T19:02:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.625872+0000 osd.2 (osd.2) 239 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:02:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:53.636190+0000 osd.2 (osd.2) 240 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:02:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.031562+0000 osd.7 (osd.7) 321 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:02:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.034437+0000 osd.7 (osd.7) 322 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:02:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.195350+0000 mgr.y (mgr.24526) 545 : cluster [DBG] pgmap v842: 270 pgs: 32 creating+peering, 238 active+clean; 8.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.201145+0000 mon.a (mon.0) 2876 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:02:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: audit 2024-09-10T19:02:54.211664+0000 mon.a (mon.0) 2877 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:02:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:54 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.211810+0000 mon.a (mon.0) 2878 : cluster [DBG] osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:02:55.143 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:55.246 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:55.272 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:55.375 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:55.410 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:55.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:55.549 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:55.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:55.682 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:55.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.083354+0000 osd.1 (osd.1) 309 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.086479+0000 osd.1 (osd.1) 310 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.764855+0000 osd.5 (osd.5) 312 : cluster [DBG] 64.11 scrub starts 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:54.769799+0000 osd.5 (osd.5) 313 : cluster [DBG] 64.11 scrub ok 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.040784+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.044452+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.224661+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:02:55.796 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: audit 2024-09-10T19:02:55.278051+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: audit 2024-09-10T19:02:55.279289+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: audit 2024-09-10T19:02:55.280760+0000 mon.a (mon.0) 2880 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.797 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:55 smithi165 bash[22591]: audit 2024-09-10T19:02:55.281803+0000 mon.a (mon.0) 2881 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.819 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:55.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:55.935 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.083354+0000 osd.1 (osd.1) 309 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:02:55.935 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.086479+0000 osd.1 (osd.1) 310 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:02:55.935 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.764855+0000 osd.5 (osd.5) 312 : cluster [DBG] 64.11 scrub starts 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:54.769799+0000 osd.5 (osd.5) 313 : cluster [DBG] 64.11 scrub ok 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.040784+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.044452+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.224661+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: audit 2024-09-10T19:02:55.278051+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: audit 2024-09-10T19:02:55.279289+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: audit 2024-09-10T19:02:55.280760+0000 mon.a (mon.0) 2880 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[18629]: audit 2024-09-10T19:02:55.281803+0000 mon.a (mon.0) 2881 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.083354+0000 osd.1 (osd.1) 309 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:02:55.936 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.086479+0000 osd.1 (osd.1) 310 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.764855+0000 osd.5 (osd.5) 312 : cluster [DBG] 64.11 scrub starts 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:54.769799+0000 osd.5 (osd.5) 313 : cluster [DBG] 64.11 scrub ok 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.040784+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.044452+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.224661+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: audit 2024-09-10T19:02:55.278051+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: audit 2024-09-10T19:02:55.279289+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: audit 2024-09-10T19:02:55.280760+0000 mon.a (mon.0) 2880 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:02:55.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:55 smithi083 bash[32381]: audit 2024-09-10T19:02:55.281803+0000 mon.a (mon.0) 2881 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-87"}]: dispatch 2024-09-10T19:02:55.950 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:55.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:56.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.090 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:56.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:56.238 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:56.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:56.368 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:56.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:56.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:56.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.635 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:56.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:56.868 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:56.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:56.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:57.005 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.126666+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.130537+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.389085+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.392686+0000 osd.4 (osd.4) 329 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.944789+0000 osd.7 (osd.7) 323 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:55.948606+0000 osd.7 (osd.7) 324 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.197944+0000 mgr.y (mgr.24526) 546 : cluster [DBG] pgmap v845: 238 pgs: 238 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:57.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.227495+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.126666+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.130537+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.389085+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.392686+0000 osd.4 (osd.4) 329 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.944789+0000 osd.7 (osd.7) 323 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:55.948606+0000 osd.7 (osd.7) 324 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.197944+0000 mgr.y (mgr.24526) 546 : cluster [DBG] pgmap v845: 238 pgs: 238 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:57.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:56 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.227495+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:02:57.036 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.126666+0000 osd.1 (osd.1) 311 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:02:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.130537+0000 osd.1 (osd.1) 312 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.389085+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.392686+0000 osd.4 (osd.4) 329 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.944789+0000 osd.7 (osd.7) 323 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:55.948606+0000 osd.7 (osd.7) 324 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.197944+0000 mgr.y (mgr.24526) 546 : cluster [DBG] pgmap v845: 238 pgs: 238 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:57.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:56 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.227495+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:02:57.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:57.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:57.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:57.312 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:02:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:02:57.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:57.447 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:57.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:02:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:57.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:57.585 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:57.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:57.717 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:57.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:57.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.746340+0000 osd.6 (osd.6) 267 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:02:57.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.749614+0000 osd.6 (osd.6) 268 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:02:57.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.804055+0000 osd.5 (osd.5) 314 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:02:57.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: cluster 2024-09-10T19:02:56.808876+0000 osd.5 (osd.5) 315 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:02:57.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: cluster 2024-09-10T19:02:57.232795+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:02:57.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: audit 2024-09-10T19:02:57.234867+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:57 smithi165 bash[22591]: audit 2024-09-10T19:02:57.238363+0000 mon.a (mon.0) 2884 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.855 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:57.959 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:57.971 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.746340+0000 osd.6 (osd.6) 267 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.749614+0000 osd.6 (osd.6) 268 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.804055+0000 osd.5 (osd.5) 314 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: cluster 2024-09-10T19:02:56.808876+0000 osd.5 (osd.5) 315 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: cluster 2024-09-10T19:02:57.232795+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: audit 2024-09-10T19:02:57.234867+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[18629]: audit 2024-09-10T19:02:57.238363+0000 mon.a (mon.0) 2884 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.746340+0000 osd.6 (osd.6) 267 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.749614+0000 osd.6 (osd.6) 268 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.804055+0000 osd.5 (osd.5) 314 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:02:57.972 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: cluster 2024-09-10T19:02:56.808876+0000 osd.5 (osd.5) 315 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:02:57.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: cluster 2024-09-10T19:02:57.232795+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:02:57.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: audit 2024-09-10T19:02:57.234867+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:57 smithi083 bash[32381]: audit 2024-09-10T19:02:57.238363+0000 mon.a (mon.0) 2884 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:02:57.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:02:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:57.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:02:58.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:02:58.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:02:58.023 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-10T19:02:58.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set-full-ratio .001 2024-09-10T19:02:58.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:58.128 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:58.232 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:58.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:58.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- 172.21.15.83:0/2159375793 >> v1:172.21.15.83:6790/0 conn(0x7f281410f680 legacy=0x7f2814111a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:58.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- 172.21.15.83:0/2159375793 shutdown_connections 2024-09-10T19:02:58.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- 172.21.15.83:0/2159375793 >> 172.21.15.83:0/2159375793 conn(0x7f28140fa370 msgr2=0x7f28140fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:58.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- 172.21.15.83:0/2159375793 shutdown_connections 2024-09-10T19:02:58.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- 172.21.15.83:0/2159375793 wait complete. 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 librados: starting msgr at 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 librados: starting objecter 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 Processor -- start 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- start start 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 librados: setting wanted keys 2024-09-10T19:02:58.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 librados: calling monclient init 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f281410c7b0 con 0x7f2814071c40 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2814123a30 con 0x7f281410f680 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2818f2d640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2814124bd0 con 0x7f2814072720 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2811d9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f2814072720 0x7f281411dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60910/0 (socket says 172.21.15.83:60910) 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2811d9b640 1 -- 172.21.15.83:0/1070009144 learned_addr learned my addr 172.21.15.83:0/1070009144 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter ms_handle_connect 0x7f2814072720 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688945115 0 0) 0x7f2814124bd0 con 0x7f2814072720 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc0039c0 con 0x7f2814072720 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter ms_handle_connect 0x7f2814071c40 2024-09-10T19:02:58.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:58.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter ms_handle_connect 0x7f281410f680 2024-09-10T19:02:58.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:02:58.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509761730 0 0) 0x7f281410c7b0 con 0x7f2814071c40 2024-09-10T19:02:58.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2814124bd0 con 0x7f2814071c40 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.343+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403496454 0 0) 0x7f2814123a30 con 0x7f281410f680 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281410c7b0 con 0x7f281410f680 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2071224557 0 0) 0x7f27fc0039c0 con 0x7f2814072720 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2814123a30 con 0x7f2814072720 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2808001aa0 con 0x7f2814072720 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1924368710 0 0) 0x7f2814124bd0 con 0x7f2814071c40 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27fc0039c0 con 0x7f2814071c40 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f28040018c0 con 0x7f2814071c40 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992105467 0 0) 0x7f2814123a30 con 0x7f2814072720 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 >> v1:172.21.15.165:6789/0 conn(0x7f281410f680 legacy=0x7f2814121300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 >> v1:172.21.15.83:6789/0 conn(0x7f2814071c40 legacy=0x7f281410bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814125dd0 con 0x7f2814072720 2024-09-10T19:02:58.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2814122a80 con 0x7f2814072720 2024-09-10T19:02:58.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2818f2d640 10 client.49562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:02:58.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2814122ed0 con 0x7f2814072720 2024-09-10T19:02:58.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2818f2d640 1 librados: init done 2024-09-10T19:02:58.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2808004cf0 con 0x7f2814072720 2024-09-10T19:02:58.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2808005910 con 0x7f2814072720 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f4005010 con 0x7f2814072720 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2808001f70 con 0x7f2814072720 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(554..554 src has 69..554) v4 ==== 8036+0+0 (unknown 1160435192 0 0) 0x7f280808a520 con 0x7f2814072720 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 10 client.49562.objecter ms_dispatch 0x7f2814000b90 osd_map(554..554 src has 69..554) v4 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 3 client.49562.objecter handle_osd_map got epochs [554,554] > 0 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 3 client.49562.objecter handle_osd_map decoding full epoch 554 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 20 client.49562.objecter dump_active .. 0 homeless 2024-09-10T19:02:58.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.347+0000 7f2802ffd640 10 client.49562.objecter ms_handle_connect 0x7f27fc0716c0 2024-09-10T19:02:58.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.355+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2808056640 con 0x7f2814072720 2024-09-10T19:02:58.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:02:58.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:58.400 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:58.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:58.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:58.539 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:58.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:58.539+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f27f4005840 con 0x7f2814072720 2024-09-10T19:02:58.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:58.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:02:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:02:58.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:58.783 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:58.806 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:02:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:02:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:57.150369+0000 osd.1 (osd.1) 313 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:02:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:57.154159+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:02:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:58.001785+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.b scrub starts 2024-09-10T19:02:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:58.006265+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.b scrub ok 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.128893+0000 mgr.y (mgr.24526) 547 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:58.198642+0000 mgr.y (mgr.24526) 548 : cluster [DBG] pgmap v848: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.233871+0000 mon.a (mon.0) 2885 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: cluster 2024-09-10T19:02:58.234067+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.243263+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.247137+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.250528+0000 mon.a (mon.0) 2887 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.545940+0000 mon.c (mon.2) 659 : audit [INF] from='client.? v1:172.21.15.83:0/1070009144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:02:58 smithi165 bash[22591]: audit 2024-09-10T19:02:58.546436+0000 mon.a (mon.0) 2888 : audit [INF] from='client.49562 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:58.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:58.944 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:58.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:57.150369+0000 osd.1 (osd.1) 313 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:02:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:57.154159+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:02:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:58.001785+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.b scrub starts 2024-09-10T19:02:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:58.006265+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.b scrub ok 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.128893+0000 mgr.y (mgr.24526) 547 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:58.198642+0000 mgr.y (mgr.24526) 548 : cluster [DBG] pgmap v848: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.233871+0000 mon.a (mon.0) 2885 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: cluster 2024-09-10T19:02:58.234067+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.243263+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.247137+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.250528+0000 mon.a (mon.0) 2887 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.545940+0000 mon.c (mon.2) 659 : audit [INF] from='client.? v1:172.21.15.83:0/1070009144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[18629]: audit 2024-09-10T19:02:58.546436+0000 mon.a (mon.0) 2888 : audit [INF] from='client.49562 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:57.150369+0000 osd.1 (osd.1) 313 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:02:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:57.154159+0000 osd.1 (osd.1) 314 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:58.001785+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.b scrub starts 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:58.006265+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.b scrub ok 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.128893+0000 mgr.y (mgr.24526) 547 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:58.198642+0000 mgr.y (mgr.24526) 548 : cluster [DBG] pgmap v848: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.233871+0000 mon.a (mon.0) 2885 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: cluster 2024-09-10T19:02:58.234067+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.243263+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.247137+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.250528+0000 mon.a (mon.0) 2887 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.545940+0000 mon.c (mon.2) 659 : audit [INF] from='client.? v1:172.21.15.83:0/1070009144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:59.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:02:58 smithi083 bash[32381]: audit 2024-09-10T19:02:58.546436+0000 mon.a (mon.0) 2888 : audit [INF] from='client.49562 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:02:59.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.083 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:59.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:59.222 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:02:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:02:59.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2802ffd640 1 -- 172.21.15.83:0/1070009144 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v555) v1 ==== 87+0+0 (unknown 2246312402 0 0) 0x7f2808056460 con 0x7f2814072720 2024-09-10T19:02:59.238 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 10 librados: watch_flush enter 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 10 librados: watch_flush exit 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 20 client.49562.objecter shutdown clearing up homeless session... 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 10 client.49562.objecter successfully canceled tick 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 >> v1:172.21.15.83:6800/3071793735 conn(0x7f27fc0716c0 legacy=0x7f27fc073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.235+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 >> v1:172.21.15.83:6790/0 conn(0x7f2814072720 legacy=0x7f281411dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.239+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 shutdown_connections 2024-09-10T19:02:59.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.239+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 >> 172.21.15.83:0/1070009144 conn(0x7f28140fa370 msgr2=0x7f2814111a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:59.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.239+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 shutdown_connections 2024-09-10T19:02:59.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.239+0000 7f2818f2d640 1 -- 172.21.15.83:0/1070009144 wait complete. 2024-09-10T19:02:59.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:02:59.239+0000 7f2818f2d640 1 librados: shutdown 2024-09-10T19:02:59.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:59.359 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:02:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:02:59.463 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:59.497 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:59.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:59.626 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:02:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:02:59.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.758 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:02:59.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.896 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:02:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:02:59.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:00.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:00.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:00.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:00.173 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:00.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:00.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:00.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: cluster 2024-09-10T19:02:59.011634+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.0 deep-scrub starts 2024-09-10T19:03:00.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: cluster 2024-09-10T19:02:59.017497+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 deep-scrub ok 2024-09-10T19:03:00.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: cluster 2024-09-10T19:02:59.226622+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:00.334 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: audit 2024-09-10T19:02:59.235642+0000 mon.a (mon.0) 2890 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:03:00.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: audit 2024-09-10T19:02:59.235746+0000 mon.a (mon.0) 2891 : audit [INF] from='client.49562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:03:00.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: cluster 2024-09-10T19:02:59.235800+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:03:00.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: audit 2024-09-10T19:02:59.576164+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:00.335 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:00 smithi165 bash[22591]: audit 2024-09-10T19:02:59.581531+0000 mon.c (mon.2) 660 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:00.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:00.445 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: cluster 2024-09-10T19:02:59.011634+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.0 deep-scrub starts 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: cluster 2024-09-10T19:02:59.017497+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 deep-scrub ok 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: cluster 2024-09-10T19:02:59.226622+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: audit 2024-09-10T19:02:59.235642+0000 mon.a (mon.0) 2890 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: audit 2024-09-10T19:02:59.235746+0000 mon.a (mon.0) 2891 : audit [INF] from='client.49562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: cluster 2024-09-10T19:02:59.235800+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: audit 2024-09-10T19:02:59.576164+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[18629]: audit 2024-09-10T19:02:59.581531+0000 mon.c (mon.2) 660 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: cluster 2024-09-10T19:02:59.011634+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.0 deep-scrub starts 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: cluster 2024-09-10T19:02:59.017497+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.0 deep-scrub ok 2024-09-10T19:03:00.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: cluster 2024-09-10T19:02:59.226622+0000 mon.a (mon.0) 2889 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: audit 2024-09-10T19:02:59.235642+0000 mon.a (mon.0) 2890 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:03:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: audit 2024-09-10T19:02:59.235746+0000 mon.a (mon.0) 2891 : audit [INF] from='client.49562 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:03:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: cluster 2024-09-10T19:02:59.235800+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:03:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: audit 2024-09-10T19:02:59.576164+0000 mon.a (mon.0) 2893 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:00 smithi083 bash[32381]: audit 2024-09-10T19:02:59.581531+0000 mon.c (mon.2) 660 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:00.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:00.581 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:00.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:00.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:00.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:00.851 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:00.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:00.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:00.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:01.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.126 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:01.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:01.252 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:01 smithi165 bash[22591]: cluster 2024-09-10T19:03:00.199392+0000 mgr.y (mgr.24526) 549 : cluster [DBG] pgmap v851: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:01.252 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:01 smithi165 bash[22591]: cluster 2024-09-10T19:03:00.239387+0000 mon.a (mon.0) 2894 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:03:01.252 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:01 smithi165 bash[22591]: cluster 2024-09-10T19:03:00.571190+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:01.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.393 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:01.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.510 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:01.528 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:01.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[18629]: cluster 2024-09-10T19:03:00.199392+0000 mgr.y (mgr.24526) 549 : cluster [DBG] pgmap v851: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:01.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[18629]: cluster 2024-09-10T19:03:00.239387+0000 mon.a (mon.0) 2894 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:03:01.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[18629]: cluster 2024-09-10T19:03:00.571190+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[32381]: cluster 2024-09-10T19:03:00.199392+0000 mgr.y (mgr.24526) 549 : cluster [DBG] pgmap v851: 238 pgs: 32 unknown, 206 active+clean; 8.3 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[32381]: cluster 2024-09-10T19:03:00.239387+0000 mon.a (mon.0) 2894 : cluster [DBG] osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:03:01.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:01 smithi083 bash[32381]: cluster 2024-09-10T19:03:00.571190+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:01.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:01.666 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:01.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:01.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:01.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:01.941 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:01.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:02.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:02.079 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:02.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:02.218 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:02.272 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:03:02.273 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:03:02.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:02 smithi165 bash[22591]: cluster 2024-09-10T19:03:01.242468+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:03:02.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:02 smithi165 bash[22591]: audit 2024-09-10T19:03:01.300441+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:02 smithi165 bash[22591]: audit 2024-09-10T19:03:01.302368+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:02 smithi165 bash[22591]: audit 2024-09-10T19:03:01.303303+0000 mon.a (mon.0) 2897 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.299 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:02 smithi165 bash[22591]: audit 2024-09-10T19:03:01.305262+0000 mon.a (mon.0) 2898 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:02.352 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:02.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:02.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:03:02.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:02.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18629]: cluster 2024-09-10T19:03:01.242468+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:03:02.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18629]: audit 2024-09-10T19:03:01.300441+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18629]: audit 2024-09-10T19:03:01.302368+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18629]: audit 2024-09-10T19:03:01.303303+0000 mon.a (mon.0) 2897 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18629]: audit 2024-09-10T19:03:01.305262+0000 mon.a (mon.0) 2898 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[32381]: cluster 2024-09-10T19:03:01.242468+0000 mon.a (mon.0) 2896 : cluster [DBG] osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:03:02.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[32381]: audit 2024-09-10T19:03:01.300441+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[32381]: audit 2024-09-10T19:03:01.302368+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[32381]: audit 2024-09-10T19:03:01.303303+0000 mon.a (mon.0) 2897 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:02.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[32381]: audit 2024-09-10T19:03:01.305262+0000 mon.a (mon.0) 2898 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-90"}]: dispatch 2024-09-10T19:03:02.488 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:02.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:02.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:02.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:02.739 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:02] "GET /metrics HTTP/1.1" 200 228924 "" "Prometheus/2.43.0" 2024-09-10T19:03:02.763 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:02.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:02.881 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:02.881 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:02.898 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:02.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:02.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:03:03.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:03.036 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:03.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:03.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:03.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:03.264 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:03.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[18629]: cluster 2024-09-10T19:03:02.199974+0000 mgr.y (mgr.24526) 550 : cluster [DBG] pgmap v854: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:03.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[18629]: cluster 2024-09-10T19:03:02.267513+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:03:03.296 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[18629]: audit 2024-09-10T19:03:02.855708+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:03.296 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[32381]: cluster 2024-09-10T19:03:02.199974+0000 mgr.y (mgr.24526) 550 : cluster [DBG] pgmap v854: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:03.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[32381]: cluster 2024-09-10T19:03:02.267513+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:03:03.297 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:03 smithi083 bash[32381]: audit 2024-09-10T19:03:02.855708+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:03.392 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:03.421 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:03.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:03.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:03 smithi165 bash[22591]: cluster 2024-09-10T19:03:02.199974+0000 mgr.y (mgr.24526) 550 : cluster [DBG] pgmap v854: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:03.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:03 smithi165 bash[22591]: cluster 2024-09-10T19:03:02.267513+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:03:03.538 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:03 smithi165 bash[22591]: audit 2024-09-10T19:03:02.855708+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:03.560 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:03.663 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:03.689 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:03.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:03.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:03.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:03.967 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:03.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:04.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.107 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:04.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:04.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:04.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: cluster 2024-09-10T19:03:03.279434+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:03:04.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:03.281323+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:03.284255+0000 mon.a (mon.0) 2901 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:03.318594+0000 mon.c (mon.2) 662 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:03.319316+0000 mon.c (mon.2) 663 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:03.325285+0000 mon.a (mon.0) 2902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:04.219957+0000 mon.a (mon.0) 2903 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: cluster 2024-09-10T19:03:04.220170+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:03:04.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:04 smithi165 bash[22591]: audit 2024-09-10T19:03:04.221177+0000 mon.b (mon.1) 429 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:04.384 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:04.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.522 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: cluster 2024-09-10T19:03:03.279434+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:03.281323+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:03.284255+0000 mon.a (mon.0) 2901 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:03.318594+0000 mon.c (mon.2) 662 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:03.319316+0000 mon.c (mon.2) 663 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:03.325285+0000 mon.a (mon.0) 2902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:04.219957+0000 mon.a (mon.0) 2903 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: cluster 2024-09-10T19:03:04.220170+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:03:04.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[32381]: audit 2024-09-10T19:03:04.221177+0000 mon.b (mon.1) 429 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: cluster 2024-09-10T19:03:03.279434+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:03.281323+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:03.284255+0000 mon.a (mon.0) 2901 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:03.318594+0000 mon.c (mon.2) 662 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:03.319316+0000 mon.c (mon.2) 663 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:03.325285+0000 mon.a (mon.0) 2902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:04.219957+0000 mon.a (mon.0) 2903 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: cluster 2024-09-10T19:03:04.220170+0000 mon.a (mon.0) 2904 : cluster [DBG] osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:03:04.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:04 smithi083 bash[18629]: audit 2024-09-10T19:03:04.221177+0000 mon.b (mon.1) 429 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:04.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:04.660 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:04.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:04.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:04.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:04.934 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:04.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:05.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:05.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:05.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:05.211 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:05.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:05.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:05.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:05.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:05.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:05 smithi083 bash[18629]: cluster 2024-09-10T19:03:04.200806+0000 mgr.y (mgr.24526) 551 : cluster [DBG] pgmap v857: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:05.583 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:05 smithi083 bash[18629]: cluster 2024-09-10T19:03:05.220561+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:03:05.584 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:05 smithi083 bash[32381]: cluster 2024-09-10T19:03:04.200806+0000 mgr.y (mgr.24526) 551 : cluster [DBG] pgmap v857: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:05.584 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:05 smithi083 bash[32381]: cluster 2024-09-10T19:03:05.220561+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:03:05.597 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:05 smithi165 bash[22591]: cluster 2024-09-10T19:03:04.200806+0000 mgr.y (mgr.24526) 551 : cluster [DBG] pgmap v857: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:05.597 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:05 smithi165 bash[22591]: cluster 2024-09-10T19:03:05.220561+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:03:05.617 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:05.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:05.751 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:05.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:05.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:05.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:05.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:06.024 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:06.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.161 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:06.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:06.296 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:06.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.436 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:06.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:06.574 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:06.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:06.712 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:06.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:06.839 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:06.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.978 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:06.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:07.082 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:07.112 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:07.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:07.215 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:07.243 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: cluster 2024-09-10T19:03:06.203072+0000 mgr.y (mgr.24526) 552 : cluster [DBG] pgmap v860: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: cluster 2024-09-10T19:03:06.207467+0000 osd.2 (osd.2) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: cluster 2024-09-10T19:03:06.207855+0000 osd.2 (osd.2) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: cluster 2024-09-10T19:03:06.226328+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: audit 2024-09-10T19:03:06.273810+0000 mon.b (mon.1) 430 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: audit 2024-09-10T19:03:06.275687+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: audit 2024-09-10T19:03:06.276774+0000 mon.a (mon.0) 2907 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: audit 2024-09-10T19:03:06.278514+0000 mon.a (mon.0) 2908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[18629]: cluster 2024-09-10T19:03:06.326827+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: cluster 2024-09-10T19:03:06.203072+0000 mgr.y (mgr.24526) 552 : cluster [DBG] pgmap v860: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: cluster 2024-09-10T19:03:06.207467+0000 osd.2 (osd.2) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: cluster 2024-09-10T19:03:06.207855+0000 osd.2 (osd.2) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: cluster 2024-09-10T19:03:06.226328+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: audit 2024-09-10T19:03:06.273810+0000 mon.b (mon.1) 430 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: audit 2024-09-10T19:03:06.275687+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: audit 2024-09-10T19:03:06.276774+0000 mon.a (mon.0) 2907 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: audit 2024-09-10T19:03:06.278514+0000 mon.a (mon.0) 2908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:07 smithi083 bash[32381]: cluster 2024-09-10T19:03:06.326827+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:07.305 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:07.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:07.377 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:07.480 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:07.512 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: cluster 2024-09-10T19:03:06.203072+0000 mgr.y (mgr.24526) 552 : cluster [DBG] pgmap v860: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: cluster 2024-09-10T19:03:06.207467+0000 osd.2 (osd.2) 241 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: cluster 2024-09-10T19:03:06.207855+0000 osd.2 (osd.2) 242 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: cluster 2024-09-10T19:03:06.226328+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: audit 2024-09-10T19:03:06.273810+0000 mon.b (mon.1) 430 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: audit 2024-09-10T19:03:06.275687+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: audit 2024-09-10T19:03:06.276774+0000 mon.a (mon.0) 2907 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: audit 2024-09-10T19:03:06.278514+0000 mon.a (mon.0) 2908 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-92"}]: dispatch 2024-09-10T19:03:07.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:07 smithi165 bash[22591]: cluster 2024-09-10T19:03:06.326827+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:07.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:07.647 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:07.696 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:07.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:07.776 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:07.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:07.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:07.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:08.008 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:08.042 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:08.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:08.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:08.296 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:08.316 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:08.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:08.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:08.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:08.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:08 smithi165 bash[22591]: cluster 2024-09-10T19:03:07.337178+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:03:08.581 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:08.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:08.717 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:08 smithi083 bash[18629]: cluster 2024-09-10T19:03:07.337178+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:03:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:08 smithi083 bash[32381]: cluster 2024-09-10T19:03:07.337178+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:03:08.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:08.853 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:08.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:08.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:08.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:09.097 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:09.129 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:09.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:09.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:09.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:09.391 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:09.441 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:09 smithi165 bash[22591]: audit 2024-09-10T19:03:08.136928+0000 mgr.y (mgr.24526) 553 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:09 smithi165 bash[22591]: cluster 2024-09-10T19:03:08.203880+0000 mgr.y (mgr.24526) 554 : cluster [DBG] pgmap v863: 206 pgs: 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:09.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:09 smithi165 bash[22591]: audit 2024-09-10T19:03:08.351774+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:09 smithi165 bash[22591]: cluster 2024-09-10T19:03:08.352314+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:03:09.442 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:09 smithi165 bash[22591]: audit 2024-09-10T19:03:08.355314+0000 mon.a (mon.0) 2912 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:09.532 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:09.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:09.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:09.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:09.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[18629]: audit 2024-09-10T19:03:08.136928+0000 mgr.y (mgr.24526) 553 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[18629]: cluster 2024-09-10T19:03:08.203880+0000 mgr.y (mgr.24526) 554 : cluster [DBG] pgmap v863: 206 pgs: 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[18629]: audit 2024-09-10T19:03:08.351774+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[18629]: cluster 2024-09-10T19:03:08.352314+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[18629]: audit 2024-09-10T19:03:08.355314+0000 mon.a (mon.0) 2912 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[32381]: audit 2024-09-10T19:03:08.136928+0000 mgr.y (mgr.24526) 553 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[32381]: cluster 2024-09-10T19:03:08.203880+0000 mgr.y (mgr.24526) 554 : cluster [DBG] pgmap v863: 206 pgs: 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[32381]: audit 2024-09-10T19:03:08.351774+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[32381]: cluster 2024-09-10T19:03:08.352314+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:03:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:09 smithi083 bash[32381]: audit 2024-09-10T19:03:08.355314+0000 mon.a (mon.0) 2912 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:09.804 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:09.910 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:09.944 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:09.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:10.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:10.086 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:10.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:10.228 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:10.332 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:10.367 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:10.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:10.503 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:10.607 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: audit 2024-09-10T19:03:09.357212+0000 mon.a (mon.0) 2913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: audit 2024-09-10T19:03:09.357862+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: cluster 2024-09-10T19:03:09.358022+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: cluster 2024-09-10T19:03:09.882434+0000 osd.5 (osd.5) 316 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: cluster 2024-09-10T19:03:09.882949+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[18629]: cluster 2024-09-10T19:03:10.204742+0000 mgr.y (mgr.24526) 555 : cluster [DBG] pgmap v866: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:10.619 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: audit 2024-09-10T19:03:09.357212+0000 mon.a (mon.0) 2913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:10.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: audit 2024-09-10T19:03:09.357862+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:10.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: cluster 2024-09-10T19:03:09.358022+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:03:10.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: cluster 2024-09-10T19:03:09.882434+0000 osd.5 (osd.5) 316 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:10.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: cluster 2024-09-10T19:03:09.882949+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:10.620 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:10 smithi083 bash[32381]: cluster 2024-09-10T19:03:10.204742+0000 mgr.y (mgr.24526) 555 : cluster [DBG] pgmap v866: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:10.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:10.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: audit 2024-09-10T19:03:09.357212+0000 mon.a (mon.0) 2913 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:10.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: audit 2024-09-10T19:03:09.357862+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:10.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: cluster 2024-09-10T19:03:09.358022+0000 mon.a (mon.0) 2914 : cluster [DBG] osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:03:10.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: cluster 2024-09-10T19:03:09.882434+0000 osd.5 (osd.5) 316 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:10.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: cluster 2024-09-10T19:03:09.882949+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:10.758 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:10 smithi165 bash[22591]: cluster 2024-09-10T19:03:10.204742+0000 mgr.y (mgr.24526) 555 : cluster [DBG] pgmap v866: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:10.776 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:10.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:10.914 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:10.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:11.018 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:11.045 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/2894675868 >> v1:172.21.15.83:6789/0 conn(0x7f9f080fb3d0 legacy=0x7f9f080fb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/2894675868 shutdown_connections 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/2894675868 >> 172.21.15.83:0/2894675868 conn(0x7f9f080f70a0 msgr2=0x7f9f080f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/2894675868 shutdown_connections 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/2894675868 wait complete. 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 librados: starting msgr at 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 librados: starting objecter 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 Processor -- start 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- start start 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 librados: setting wanted keys 2024-09-10T19:03:11.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 librados: calling monclient init 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f0811e220 con 0x7f9f080fbeb0 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f0811f420 con 0x7f9f0810b200 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f08120620 con 0x7f9f080fb3d0 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f06d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9f080fb3d0 0x7f9f08107620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:41132/0 (socket says 172.21.15.83:41132) 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efe59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9f080fbeb0 0x7f9f08107d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52322/0 (socket says 172.21.15.83:52322) 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f06d9d640 1 -- 172.21.15.83:0/1194532718 learned_addr learned my addr 172.21.15.83:0/1194532718 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:11.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter ms_handle_connect 0x7f9f080fbeb0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter ms_handle_connect 0x7f9f080fb3d0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter ms_handle_connect 0x7f9f0810b200 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499689255 0 0) 0x7f9f08120620 con 0x7f9f080fb3d0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef4003120 con 0x7f9f080fb3d0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532022050 0 0) 0x7f9f0811e220 con 0x7f9f080fbeb0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f08120620 con 0x7f9f080fbeb0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734157440 0 0) 0x7f9f0811f420 con 0x7f9f0810b200 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0811e220 con 0x7f9f0810b200 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073843304 0 0) 0x7f9ef4003120 con 0x7f9f080fb3d0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0811f420 con 0x7f9f080fb3d0 2024-09-10T19:03:11.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9ef8002c60 con 0x7f9f080fb3d0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2319252815 0 0) 0x7f9f08120620 con 0x7f9f080fbeb0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ef4003120 con 0x7f9f080fbeb0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9ef0001b30 con 0x7f9f080fbeb0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524216353 0 0) 0x7f9f0811e220 con 0x7f9f0810b200 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f08120620 con 0x7f9f0810b200 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 202742602 0 0) 0x7f9f0811f420 con 0x7f9f080fb3d0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.165:6789/0 conn(0x7f9f0810b200 legacy=0x7f9f0811c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6789/0 conn(0x7f9f080fbeb0 legacy=0x7f9f08107d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08121820 con 0x7f9f080fb3d0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f08120810 con 0x7f9f080fb3d0 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 10 client.49568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:11.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08120ce0 con 0x7f9f080fb3d0 2024-09-10T19:03:11.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.051+0000 7f9f0d1a6640 1 librados: init done 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9ef8002340 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9f08066430 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9ef8003610 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9ef8003790 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(566..566 src has 69..566) v4 ==== 8036+0+0 (unknown 2141178134 0 0) 0x7f9ef808a7c0 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 osd_map(566..566 src has 69..566) v4 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 3 client.49568.objecter handle_osd_map got epochs [566,566] > 0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 3 client.49568.objecter handle_osd_map decoding full epoch 566 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 20 client.49568.objecter dump_active .. 0 homeless 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v566) v1 ==== 54+0+16 (unknown 652654156 0 2420000192) 0x7f9ef808abd0 con 0x7f9f080fb3d0 2024-09-10T19:03:11.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 1 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9ef40717b0 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 1 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 0 1 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 4 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 1 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f08110cc0 con 0x7f9f0811f5b0 2024-09-10T19:03:11.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.055+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f0811f5b0 2024-09-10T19:03:11.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.059+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f08110cc0 con 0x7f9f0811f5b0 2024-09-10T19:03:11.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.059+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:03:11.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.059+0000 7f9efffff640 10 client.49568.objecter _finish_command 1 = system:0 2024-09-10T19:03:11.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.059+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 4 2024-09-10T19:03:11.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.059+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 0 1 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 2 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 2 2024-09-10T19:03:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f08108a40 osd=0 3 2024-09-10T19:03:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 0 2 2024-09-10T19:03:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 4 2024-09-10T19:03:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 2 2024-09-10T19:03:11.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f081202d0 con 0x7f9f0811f5b0 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f081202d0 con 0x7f9f0811f5b0 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 10 client.49568.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 4 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 0 2 2024-09-10T19:03:11.081 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:03:11.082 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:03:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 3 2024-09-10T19:03:11.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 1 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 4 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 3 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0805e700 con 0x7f9f0805e310 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.075+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f0805e310 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.079+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f0805e700 con 0x7f9f0805e310 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.079+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.079+0000 7f9efffff640 10 client.49568.objecter _finish_command 3 = system:0 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.079+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 4 2024-09-10T19:03:11.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.079+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 1 3 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 4 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 4 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f08120440 osd=1 3 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 1 4 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 4 2024-09-10T19:03:11.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 4 2024-09-10T19:03:11.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.095+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f0805eab0 con 0x7f9f0805e310 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f0805eab0 con 0x7f9f0805e310 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 10 client.49568.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 4 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 1 4 2024-09-10T19:03:11.103 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 5 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 5 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 2 5 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 4 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 5 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0802aca0 con 0x7f9f0805df00 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f0805df00 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f0802aca0 con 0x7f9f0805df00 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 10 client.49568.objecter _finish_command 5 = system:0 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 4 2024-09-10T19:03:11.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.099+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 2 5 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 6 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 6 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0805dd90 osd=2 3 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 2 6 2024-09-10T19:03:11.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 4 2024-09-10T19:03:11.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 6 2024-09-10T19:03:11.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.119+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f0802a210 con 0x7f9f0805df00 2024-09-10T19:03:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f0802a210 con 0x7f9f0805df00 2024-09-10T19:03:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 10 client.49568.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 4 2024-09-10T19:03:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 2 6 2024-09-10T19:03:11.128 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 7 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 7 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 3 7 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 4 2024-09-10T19:03:11.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 7 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f08102830 con 0x7f9f0812b290 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f0812b290 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f08102830 con 0x7f9f0812b290 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 10 client.49568.objecter _finish_command 7 = system:0 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 4 2024-09-10T19:03:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.123+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 3 7 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 8 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 8 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081021b0 osd=3 3 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 3 8 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 4 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 8 2024-09-10T19:03:11.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.139+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f081029d0 con 0x7f9f0812b290 2024-09-10T19:03:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f081029d0 con 0x7f9f0812b290 2024-09-10T19:03:11.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 10 client.49568.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 4 2024-09-10T19:03:11.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 3 8 2024-09-10T19:03:11.149 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:03:11.149 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 9 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 9 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 4 9 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 4 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 9 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0812ad20 con 0x7f9f0812a8d0 2024-09-10T19:03:11.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.143+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f0812a8d0 2024-09-10T19:03:11.151 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:11.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.147+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f0812ad20 con 0x7f9f0812a8d0 2024-09-10T19:03:11.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.147+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:03:11.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.147+0000 7f9efffff640 10 client.49568.objecter _finish_command 9 = system:0 2024-09-10T19:03:11.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.147+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 4 2024-09-10T19:03:11.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.147+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 4 9 2024-09-10T19:03:11.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 10 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 10 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081029d0 osd=4 3 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 4 10 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 4 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 10 2024-09-10T19:03:11.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.171+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f0812ad20 con 0x7f9f0812a8d0 2024-09-10T19:03:11.178 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:11.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f0812ad20 con 0x7f9f0812a8d0 2024-09-10T19:03:11.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 10 client.49568.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 4 2024-09-10T19:03:11.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 4 10 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 11 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 11 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 5 11 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 4 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 11 2024-09-10T19:03:11.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0812b120 con 0x7f9f08133b50 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f08133b50 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f0812b120 con 0x7f9f08133b50 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 10 client.49568.objecter _finish_command 11 = system:0 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 4 2024-09-10T19:03:11.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.179+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 5 11 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 12 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 12 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0812ad20 osd=5 3 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 5 12 2024-09-10T19:03:11.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 4 2024-09-10T19:03:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 12 2024-09-10T19:03:11.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.203+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f0812b120 con 0x7f9f08133b50 2024-09-10T19:03:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f0812b120 con 0x7f9f08133b50 2024-09-10T19:03:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9efffff640 10 client.49568.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 4 2024-09-10T19:03:11.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 5 12 2024-09-10T19:03:11.213 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:03:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:03:11.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 13 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 13 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 6 13 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 4 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 13 2024-09-10T19:03:11.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.207+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f081386d0 con 0x7f9f08139290 2024-09-10T19:03:11.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f08139290 2024-09-10T19:03:11.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f081386d0 con 0x7f9f08139290 2024-09-10T19:03:11.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:03:11.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 10 client.49568.objecter _finish_command 13 = system:0 2024-09-10T19:03:11.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 4 2024-09-10T19:03:11.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.211+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 6 13 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 14 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 14 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f0812b120 osd=6 3 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 6 14 2024-09-10T19:03:11.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 4 2024-09-10T19:03:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 14 2024-09-10T19:03:11.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.231+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f081386d0 con 0x7f9f08139290 2024-09-10T19:03:11.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f081386d0 con 0x7f9f08139290 2024-09-10T19:03:11.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9efffff640 10 client.49568.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 4 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 6 14 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:03:11.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 15 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 15 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 7 15 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 4 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 15 2024-09-10T19:03:11.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.235+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0813fda0 con 0x7f9f08138940 2024-09-10T19:03:11.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 10 client.49568.objecter ms_handle_connect 0x7f9f08138940 2024-09-10T19:03:11.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9f0813fda0 con 0x7f9f08138940 2024-09-10T19:03:11.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:03:11.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 10 client.49568.objecter _finish_command 15 = system:0 2024-09-10T19:03:11.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 4 2024-09-10T19:03:11.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.239+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 7 15 2024-09-10T19:03:11.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 10 client.49568.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:11.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign -1 16 2024-09-10T19:03:11.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 20 client.49568.objecter _get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_remove -1 16 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 20 client.49568.objecter get_session s=0x7f9f081386d0 osd=7 3 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 15 client.49568.objecter _session_command_op_assign 7 16 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 4 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 10 client.49568.objecter _send_command 16 2024-09-10T19:03:11.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.255+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9f08141680 con 0x7f9f08138940 2024-09-10T19:03:11.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9efffff640 1 -- 172.21.15.83:0/1194532718 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f9f08141680 con 0x7f9f08138940 2024-09-10T19:03:11.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9efffff640 10 client.49568.objecter ms_dispatch 0x7f9f08000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:11.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9efffff640 10 client.49568.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:11.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9efffff640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 4 2024-09-10T19:03:11.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9efffff640 15 client.49568.objecter _session_command_op_remove 7 16 2024-09-10T19:03:11.265 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 librados: watch_flush enter 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 librados: watch_flush exit 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.0 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6801/2535527483 conn(0x7f9f0811f5b0 legacy=0x7f9f0811f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08108a40 osd=0 1 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.1 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6805/2309624637 conn(0x7f9f0805e310 legacy=0x7f9f080298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f08120440 osd=1 1 2024-09-10T19:03:11.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.2 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6809/386094220 conn(0x7f9f0805df00 legacy=0x7f9f081289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0805dd90 osd=2 1 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.3 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6813/1279688021 conn(0x7f9f0812b290 legacy=0x7f9f0812d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081021b0 osd=3 1 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.4 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.165:6800/3225515265 conn(0x7f9f0812a8d0 legacy=0x7f9f08131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081029d0 osd=4 1 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.5 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.165:6804/571173777 conn(0x7f9f08133b50 legacy=0x7f9f08135f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812ad20 osd=5 1 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.6 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.165:6808/1317922936 conn(0x7f9f08139290 legacy=0x7f9f08139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f0812b120 osd=6 1 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter close_session for osd.7 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0759e640 1 -- 172.21.15.83:0/1194532718 reap_dead start 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.165:6812/1610151650 conn(0x7f9f08138940 legacy=0x7f9f0813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0759e640 1 -- 172.21.15.83:0/1194532718 reap_dead start 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter put_session s=0x7f9f081386d0 osd=7 1 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 20 client.49568.objecter shutdown clearing up homeless session... 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 10 client.49568.objecter successfully canceled tick 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0759e640 1 -- 172.21.15.83:0/1194532718 reap_dead start 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9ef40717b0 legacy=0x7f9ef4073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0759e640 1 -- 172.21.15.83:0/1194532718 reap_dead start 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> v1:172.21.15.83:6790/0 conn(0x7f9f080fb3d0 legacy=0x7f9f08107620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 shutdown_connections 2024-09-10T19:03:11.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 >> 172.21.15.83:0/1194532718 conn(0x7f9f080f70a0 msgr2=0x7f9f0810d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:11.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 shutdown_connections 2024-09-10T19:03:11.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 -- 172.21.15.83:0/1194532718 wait complete. 2024-09-10T19:03:11.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.259+0000 7f9f0d1a6640 1 librados: shutdown 2024-09-10T19:03:11.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:11.308 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:03:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:03:11.308 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:03:11.308 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:03:11.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.323+0000 7f1df1d6c640 1 -- 172.21.15.83:0/2395083294 >> v1:172.21.15.83:6790/0 conn(0x7f1dec107e30 legacy=0x7f1dec10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.323+0000 7f1df1d6c640 1 -- 172.21.15.83:0/2395083294 shutdown_connections 2024-09-10T19:03:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.323+0000 7f1df1d6c640 1 -- 172.21.15.83:0/2395083294 >> 172.21.15.83:0/2395083294 conn(0x7f1dec075290 msgr2=0x7f1dec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.323+0000 7f1df1d6c640 1 -- 172.21.15.83:0/2395083294 shutdown_connections 2024-09-10T19:03:11.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.323+0000 7f1df1d6c640 1 -- 172.21.15.83:0/2395083294 wait complete. 2024-09-10T19:03:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 librados: starting msgr at 2024-09-10T19:03:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 librados: starting objecter 2024-09-10T19:03:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 Processor -- start 2024-09-10T19:03:11.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- start start 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 librados: setting wanted keys 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 librados: calling monclient init 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dec119f80 con 0x7f1dec1014e0 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dec11b140 con 0x7f1dec1042a0 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1dec11c340 con 0x7f1dec107e30 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1deb7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1dec1014e0 0x7f1dec0fd1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52328/0 (socket says 172.21.15.83:52328) 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1debfff640 1 --1- 172.21.15.83:0/982853815 >> v1:172.21.15.83:6790/0 conn(0x7f1dec107e30 0x7f1dec118680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:41144/0 (socket says 172.21.15.83:41144) 2024-09-10T19:03:11.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1deb7fe640 1 -- 172.21.15.83:0/982853815 learned_addr learned my addr 172.21.15.83:0/982853815 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:11.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dec107e30 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dec1042a0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter ms_handle_connect 0x7f1dec1014e0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811630442 0 0) 0x7f1dec11b140 con 0x7f1dec1042a0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd80034e0 con 0x7f1dec1042a0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755345506 0 0) 0x7f1dec11c340 con 0x7f1dec107e30 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec11b140 con 0x7f1dec107e30 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312205615 0 0) 0x7f1dec119f80 con 0x7f1dec1014e0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec11c340 con 0x7f1dec1014e0 2024-09-10T19:03:11.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316021876 0 0) 0x7f1dd80034e0 con 0x7f1dec1042a0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dec119f80 con 0x7f1dec1042a0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1dd40019d0 con 0x7f1dec1042a0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374673795 0 0) 0x7f1dec11c340 con 0x7f1dec1014e0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dd80034e0 con 0x7f1dec1014e0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1ddc001800 con 0x7f1dec1014e0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492178900 0 0) 0x7f1dec11b140 con 0x7f1dec107e30 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dec11c340 con 0x7f1dec107e30 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1de4001950 con 0x7f1dec107e30 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 200036635 0 0) 0x7f1dec119f80 con 0x7f1dec1042a0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 >> v1:172.21.15.83:6790/0 conn(0x7f1dec107e30 legacy=0x7f1dec118680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 >> v1:172.21.15.83:6789/0 conn(0x7f1dec1014e0 legacy=0x7f1dec0fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec11d540 con 0x7f1dec1042a0 2024-09-10T19:03:11.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec11c530 con 0x7f1dec1042a0 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 10 client.50083.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec11ca00 con 0x7f1dec1042a0 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1df1d6c640 1 librados: init done 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1dd40023c0 con 0x7f1dec1042a0 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.327+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1dd40031c0 con 0x7f1dec1042a0 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc005010 con 0x7f1dec1042a0 2024-09-10T19:03:11.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1dd4019660 con 0x7f1dec1042a0 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(566..566 src has 69..566) v4 ==== 8036+0+0 (unknown 2141178134 0 0) 0x7f1dd4087bb0 con 0x7f1dec1042a0 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 10 client.50083.objecter ms_dispatch 0x7f1dec000b90 osd_map(566..566 src has 69..566) v4 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 3 client.50083.objecter handle_osd_map got epochs [566,566] > 0 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 3 client.50083.objecter handle_osd_map decoding full epoch 566 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 20 client.50083.objecter dump_active .. 0 homeless 2024-09-10T19:03:11.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.331+0000 7f1de8ff9640 10 client.50083.objecter ms_handle_connect 0x7f1dd80739f0 2024-09-10T19:03:11.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.339+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1dd4053cd0 con 0x7f1dec1042a0 2024-09-10T19:03:11.389 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:11.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:11.443 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:11.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:11.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:11.543+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd set-full-ratio", "ratio": 0.001} v 0) v1 -- 0x7f1dcc0052c0 con 0x7f1dec1042a0 2024-09-10T19:03:11.582 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:11.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:11.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:11 smithi165 bash[22591]: cluster 2024-09-10T19:03:10.373840+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:03:11.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:11 smithi165 bash[22591]: audit 2024-09-10T19:03:11.060702+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? v1:172.21.15.83:0/1194532718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:11.716 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:11 smithi083 bash[18629]: cluster 2024-09-10T19:03:10.373840+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:03:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:11 smithi083 bash[18629]: audit 2024-09-10T19:03:11.060702+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? v1:172.21.15.83:0/1194532718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:11 smithi083 bash[32381]: cluster 2024-09-10T19:03:10.373840+0000 mon.a (mon.0) 2915 : cluster [DBG] osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:03:11.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:11 smithi083 bash[32381]: audit 2024-09-10T19:03:11.060702+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? v1:172.21.15.83:0/1194532718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:11.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:11.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:11.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:11.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:11.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:12.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:03:12.093 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:12.126 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:12.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:12.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.392 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:12.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.399+0000 7f1de8ff9640 1 -- 172.21.15.83:0/982853815 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd set-full-ratio", "ratio": 0.001}]=0 osd set-full-ratio 0.001 v568) v1 ==== 106+0+0 (unknown 1567548824 0 0) 0x7f1dd4053af0 con 0x7f1dec1042a0 2024-09-10T19:03:12.406 INFO:teuthology.orchestra.run.smithi083.stderr:osd set-full-ratio 0.001 2024-09-10T19:03:12.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 10 librados: watch_flush enter 2024-09-10T19:03:12.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 10 librados: watch_flush exit 2024-09-10T19:03:12.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 20 client.50083.objecter shutdown clearing up homeless session... 2024-09-10T19:03:12.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 10 client.50083.objecter successfully canceled tick 2024-09-10T19:03:12.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1dd80739f0 legacy=0x7f1dd8075ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 >> v1:172.21.15.165:6789/0 conn(0x7f1dec1042a0 legacy=0x7f1dec0fd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 shutdown_connections 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 >> 172.21.15.83:0/982853815 conn(0x7f1dec075290 msgr2=0x7f1dec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 shutdown_connections 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 -- 172.21.15.83:0/982853815 wait complete. 2024-09-10T19:03:12.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:12.407+0000 7f1df1d6c640 1 librados: shutdown 2024-09-10T19:03:12.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:12.527 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:12.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:12.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[32381]: cluster 2024-09-10T19:03:11.397229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:03:12.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[32381]: audit 2024-09-10T19:03:11.546661+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.83:0/982853815' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[32381]: audit 2024-09-10T19:03:11.549400+0000 mon.a (mon.0) 2917 : audit [INF] from='client.50083 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[32381]: cluster 2024-09-10T19:03:12.205632+0000 mgr.y (mgr.24526) 556 : cluster [DBG] pgmap v869: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:03:12.645 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18629]: debug 2024-09-10T19:03:12.387+0000 7faca7e6f640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-10T19:03:12.645 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18629]: cluster 2024-09-10T19:03:11.397229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:03:12.645 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18629]: audit 2024-09-10T19:03:11.546661+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.83:0/982853815' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.645 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18629]: audit 2024-09-10T19:03:11.549400+0000 mon.a (mon.0) 2917 : audit [INF] from='client.50083 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.645 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18629]: cluster 2024-09-10T19:03:12.205632+0000 mgr.y (mgr.24526) 556 : cluster [DBG] pgmap v869: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:03:12.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:12.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:12.791 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:12 smithi165 bash[22591]: cluster 2024-09-10T19:03:11.397229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:03:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:12 smithi165 bash[22591]: audit 2024-09-10T19:03:11.546661+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.83:0/982853815' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:12 smithi165 bash[22591]: audit 2024-09-10T19:03:11.549400+0000 mon.a (mon.0) 2917 : audit [INF] from='client.50083 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-09-10T19:03:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:12 smithi165 bash[22591]: cluster 2024-09-10T19:03:12.205632+0000 mgr.y (mgr.24526) 556 : cluster [DBG] pgmap v869: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:03:12.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.930 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:12.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:12.995 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:12] "GET /metrics HTTP/1.1" 200 228924 "" "Prometheus/2.43.0" 2024-09-10T19:03:13.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.067 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:13.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:13.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:13.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:13.342 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:13.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:13.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.619 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:13.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:13.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[18629]: cluster 2024-09-10T19:03:12.391569+0000 mon.a (mon.0) 2918 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-10T19:03:13.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[18629]: audit 2024-09-10T19:03:12.403677+0000 mon.a (mon.0) 2919 : audit [INF] from='client.50083 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-10T19:03:13.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[18629]: cluster 2024-09-10T19:03:12.403770+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:03:13.735 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[32381]: cluster 2024-09-10T19:03:12.391569+0000 mon.a (mon.0) 2918 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-10T19:03:13.736 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[32381]: audit 2024-09-10T19:03:12.403677+0000 mon.a (mon.0) 2919 : audit [INF] from='client.50083 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-10T19:03:13.736 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:13 smithi083 bash[32381]: cluster 2024-09-10T19:03:12.403770+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:03:13.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:13.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:13 smithi165 bash[22591]: cluster 2024-09-10T19:03:12.391569+0000 mon.a (mon.0) 2918 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-10T19:03:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:13 smithi165 bash[22591]: audit 2024-09-10T19:03:12.403677+0000 mon.a (mon.0) 2919 : audit [INF] from='client.50083 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-09-10T19:03:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:13 smithi165 bash[22591]: cluster 2024-09-10T19:03:12.403770+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:03:13.894 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:13.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:13.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:14.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:14.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:14.170 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:14.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:14.320 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:14.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:14.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:14.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:14.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:14.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:14.714 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:14 smithi083 bash[18629]: cluster 2024-09-10T19:03:14.206528+0000 mgr.y (mgr.24526) 557 : cluster [DBG] pgmap v871: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:03:14.715 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:14 smithi083 bash[32381]: cluster 2024-09-10T19:03:14.206528+0000 mgr.y (mgr.24526) 557 : cluster [DBG] pgmap v871: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:03:14.725 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:14.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:14 smithi165 bash[22591]: cluster 2024-09-10T19:03:14.206528+0000 mgr.y (mgr.24526) 557 : cluster [DBG] pgmap v871: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:03:14.864 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:14.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:14.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:15.004 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:15.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:15.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:03:15.142 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:15.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:15.282 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:15.340 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-10T19:03:15.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set-full-ratio .95 2024-09-10T19:03:15.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:15.418 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:15.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:15.548 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:15.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- 172.21.15.83:0/2666120087 >> v1:172.21.15.83:6790/0 conn(0x7f7b9c10da80 legacy=0x7f7b9c10fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:15.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- 172.21.15.83:0/2666120087 shutdown_connections 2024-09-10T19:03:15.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- 172.21.15.83:0/2666120087 >> 172.21.15.83:0/2666120087 conn(0x7f7b9c06b990 msgr2=0x7f7b9c06bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:15.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- 172.21.15.83:0/2666120087 shutdown_connections 2024-09-10T19:03:15.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- 172.21.15.83:0/2666120087 wait complete. 2024-09-10T19:03:15.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 librados: starting msgr at 2024-09-10T19:03:15.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 librados: starting objecter 2024-09-10T19:03:15.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 Processor -- start 2024-09-10T19:03:15.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- start start 2024-09-10T19:03:15.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 librados: setting wanted keys 2024-09-10T19:03:15.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 librados: calling monclient init 2024-09-10T19:03:15.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b9c124860 con 0x7f7b9c111710 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b9c125a20 con 0x7f7b9c10da80 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba40a0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7b9c126c20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba263d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7b9c111710 0x7f7b9c122f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59442/0 (socket says 172.21.15.83:59442) 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba263d640 1 -- 172.21.15.83:0/174953459 learned_addr learned my addr 172.21.15.83:0/174953459 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7ba1e3c640 1 --1- 172.21.15.83:0/174953459 >> v1:172.21.15.83:6790/0 conn(0x7f7b9c104bf0 0x7f7b9c109f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48908/0 (socket says 172.21.15.83:48908) 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter ms_handle_connect 0x7f7b9c104bf0 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter ms_handle_connect 0x7f7b9c111710 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter ms_handle_connect 0x7f7b9c10da80 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.579+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167788264 0 0) 0x7f7b9c126c20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88003720 con 0x7f7b9c104bf0 2024-09-10T19:03:15.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253467281 0 0) 0x7f7b9c124860 con 0x7f7b9c111710 2024-09-10T19:03:15.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c126c20 con 0x7f7b9c111710 2024-09-10T19:03:15.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258952328 0 0) 0x7f7b9c125a20 con 0x7f7b9c10da80 2024-09-10T19:03:15.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c124860 con 0x7f7b9c10da80 2024-09-10T19:03:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931106360 0 0) 0x7f7b88003720 con 0x7f7b9c104bf0 2024-09-10T19:03:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b9c125a20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7b98001800 con 0x7f7b9c104bf0 2024-09-10T19:03:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145867562 0 0) 0x7f7b9c126c20 con 0x7f7b9c111710 2024-09-10T19:03:15.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b88003720 con 0x7f7b9c111710 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7b94001c60 con 0x7f7b9c111710 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 826489933 0 0) 0x7f7b9c124860 con 0x7f7b9c10da80 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b9c126c20 con 0x7f7b9c10da80 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7b8c002d90 con 0x7f7b9c10da80 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3396956935 0 0) 0x7f7b9c125a20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 >> v1:172.21.15.165:6789/0 conn(0x7f7b9c10da80 legacy=0x7f7b9c10a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 >> v1:172.21.15.83:6789/0 conn(0x7f7b9c111710 legacy=0x7f7b9c122f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c127e20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7ba40a0640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c124a50 con 0x7f7b9c104bf0 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7ba40a0640 10 client.49580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7ba40a0640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c124ea0 con 0x7f7b9c104bf0 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7ba40a0640 1 librados: init done 2024-09-10T19:03:15.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7b98001d70 con 0x7f7b9c104bf0 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7b98005850 con 0x7f7b9c104bf0 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c005010 con 0x7f7b9c104bf0 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.583+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7b9801bc70 con 0x7f7b9c104bf0 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(568..568 src has 69..568) v4 ==== 8036+0+0 (unknown 1657096033 0 0) 0x7f7b9808a440 con 0x7f7b9c104bf0 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 10 client.49580.objecter ms_dispatch 0x7f7b9c000b90 osd_map(568..568 src has 69..568) v4 2024-09-10T19:03:15.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 3 client.49580.objecter handle_osd_map got epochs [568,568] > 0 2024-09-10T19:03:15.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 3 client.49580.objecter handle_osd_map decoding full epoch 568 2024-09-10T19:03:15.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 20 client.49580.objecter dump_active .. 0 homeless 2024-09-10T19:03:15.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.587+0000 7f7b927fc640 10 client.49580.objecter ms_handle_connect 0x7f7b88071740 2024-09-10T19:03:15.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.591+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7b98056a20 con 0x7f7b9c104bf0 2024-09-10T19:03:15.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:15.683 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:15.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:15.775+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7b7c0052c0 con 0x7f7b9c104bf0 2024-09-10T19:03:15.787 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:15.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:15 smithi083 bash[18629]: audit 2024-09-10T19:03:14.578617+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:15.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:15 smithi083 bash[18629]: audit 2024-09-10T19:03:14.581239+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:15.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:15 smithi083 bash[32381]: audit 2024-09-10T19:03:14.578617+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:15 smithi083 bash[32381]: audit 2024-09-10T19:03:14.581239+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:15.815 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:15 smithi165 bash[22591]: audit 2024-09-10T19:03:14.578617+0000 mon.a (mon.0) 2921 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:15 smithi165 bash[22591]: audit 2024-09-10T19:03:14.581239+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:15.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:15.953 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:16.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:16.092 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:16.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:16.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.229 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:16.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:16.465 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:16.500 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:16.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.591+0000 7f7b927fc640 1 -- 172.21.15.83:0/174953459 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v569) v1 ==== 97+0+0 (unknown 553416395 0 0) 0x7f7b980563c0 con 0x7f7b9c104bf0 2024-09-10T19:03:16.596 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 10 librados: watch_flush enter 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 10 librados: watch_flush exit 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 20 client.49580.objecter shutdown clearing up homeless session... 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 10 client.49580.objecter successfully canceled tick 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7b88071740 legacy=0x7f7b88073bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 >> v1:172.21.15.83:6790/0 conn(0x7f7b9c104bf0 legacy=0x7f7b9c109f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 shutdown_connections 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 >> 172.21.15.83:0/174953459 conn(0x7f7b9c06b990 msgr2=0x7f7b9c06bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 shutdown_connections 2024-09-10T19:03:16.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 -- 172.21.15.83:0/174953459 wait complete. 2024-09-10T19:03:16.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:16.595+0000 7f7b87fff640 1 librados: shutdown 2024-09-10T19:03:16.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.627 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:16.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:16.764 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[18629]: audit 2024-09-10T19:03:15.781973+0000 mon.c (mon.2) 666 : audit [INF] from='client.? v1:172.21.15.83:0/174953459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[18629]: audit 2024-09-10T19:03:15.782515+0000 mon.a (mon.0) 2922 : audit [INF] from='client.49580 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[18629]: cluster 2024-09-10T19:03:16.208816+0000 mgr.y (mgr.24526) 558 : cluster [DBG] pgmap v872: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[32381]: audit 2024-09-10T19:03:15.781973+0000 mon.c (mon.2) 666 : audit [INF] from='client.? v1:172.21.15.83:0/174953459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[32381]: audit 2024-09-10T19:03:15.782515+0000 mon.a (mon.0) 2922 : audit [INF] from='client.49580 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:16 smithi083 bash[32381]: cluster 2024-09-10T19:03:16.208816+0000 mgr.y (mgr.24526) 558 : cluster [DBG] pgmap v872: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T19:03:16.803 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:16 smithi165 bash[22591]: audit 2024-09-10T19:03:15.781973+0000 mon.c (mon.2) 666 : audit [INF] from='client.? v1:172.21.15.83:0/174953459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:16 smithi165 bash[22591]: audit 2024-09-10T19:03:15.782515+0000 mon.a (mon.0) 2922 : audit [INF] from='client.49580 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:16 smithi165 bash[22591]: cluster 2024-09-10T19:03:16.208816+0000 mgr.y (mgr.24526) 558 : cluster [DBG] pgmap v872: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-09-10T19:03:16.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:16.900 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:16.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:17.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.039 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:17.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:17.177 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:17.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.313 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:17.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:17.450 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:17.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.589 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:17.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:17.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[18629]: cluster 2024-09-10T19:03:16.576295+0000 mon.a (mon.0) 2923 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:17.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[18629]: audit 2024-09-10T19:03:16.591053+0000 mon.a (mon.0) 2924 : audit [INF] from='client.49580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:17.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[18629]: cluster 2024-09-10T19:03:16.591153+0000 mon.a (mon.0) 2925 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:03:17.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[32381]: cluster 2024-09-10T19:03:16.576295+0000 mon.a (mon.0) 2923 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:17.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[32381]: audit 2024-09-10T19:03:16.591053+0000 mon.a (mon.0) 2924 : audit [INF] from='client.49580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:17.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:17 smithi083 bash[32381]: cluster 2024-09-10T19:03:16.591153+0000 mon.a (mon.0) 2925 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:03:17.726 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:17.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:17.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:17 smithi165 bash[22591]: cluster 2024-09-10T19:03:16.576295+0000 mon.a (mon.0) 2923 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:17.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:17 smithi165 bash[22591]: audit 2024-09-10T19:03:16.591053+0000 mon.a (mon.0) 2924 : audit [INF] from='client.49580 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:17.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:17 smithi165 bash[22591]: cluster 2024-09-10T19:03:16.591153+0000 mon.a (mon.0) 2925 : cluster [DBG] osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:03:17.867 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:17.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:17.996 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:17.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:18.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:18.131 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:18.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:18.269 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:18.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:18.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:18.409 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:18.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:18.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:18.545 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:18.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:18.679 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:18.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:18 smithi165 bash[22591]: audit 2024-09-10T19:03:18.145172+0000 mgr.y (mgr.24526) 559 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:18.776 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:18 smithi165 bash[22591]: cluster 2024-09-10T19:03:18.209614+0000 mgr.y (mgr.24526) 560 : cluster [DBG] pgmap v874: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 150 B/s wr, 3 op/s 2024-09-10T19:03:18.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:18.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:18 smithi083 bash[18629]: audit 2024-09-10T19:03:18.145172+0000 mgr.y (mgr.24526) 559 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:18.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:18 smithi083 bash[18629]: cluster 2024-09-10T19:03:18.209614+0000 mgr.y (mgr.24526) 560 : cluster [DBG] pgmap v874: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 150 B/s wr, 3 op/s 2024-09-10T19:03:18.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:18 smithi083 bash[32381]: audit 2024-09-10T19:03:18.145172+0000 mgr.y (mgr.24526) 559 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:18.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:18 smithi083 bash[32381]: cluster 2024-09-10T19:03:18.209614+0000 mgr.y (mgr.24526) 560 : cluster [DBG] pgmap v874: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 150 B/s wr, 3 op/s 2024-09-10T19:03:18.816 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:18.920 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:18.956 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:18.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:19.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:19.095 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:19.198 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:19.235 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:19.338 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:19.372 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:19.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:19.494 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:03:19.506 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:19.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:19.637 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:19.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:19.772 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:19.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:19.900 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:19.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:19.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:20.002 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:20.004 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:20.005 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:20.008 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:03:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:03:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T19:03:20.008 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T19:03:20.008 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:03:20.029 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:20.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:20.163 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:20.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:20.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:20 smithi083 bash[18629]: cluster 2024-09-10T19:03:19.224432+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:03:20.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:20 smithi083 bash[32381]: cluster 2024-09-10T19:03:19.224432+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:03:20.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:20.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:20.417 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:20 smithi165 bash[22591]: cluster 2024-09-10T19:03:19.224432+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:03:20.438 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:20.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:20.576 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:20.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:03:20.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:20.716 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:20.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:20.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:20.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:20.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:20.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:21.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:21.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.267 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:21.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:21.405 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:21.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:21.522 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:21 smithi165 bash[22591]: cluster 2024-09-10T19:03:20.210499+0000 mgr.y (mgr.24526) 561 : cluster [DBG] pgmap v876: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 B/s wr, 3 op/s 2024-09-10T19:03:21.522 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:21.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:21 smithi083 bash[18629]: cluster 2024-09-10T19:03:20.210499+0000 mgr.y (mgr.24526) 561 : cluster [DBG] pgmap v876: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 B/s wr, 3 op/s 2024-09-10T19:03:21.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:21 smithi083 bash[32381]: cluster 2024-09-10T19:03:20.210499+0000 mgr.y (mgr.24526) 561 : cluster [DBG] pgmap v876: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 B/s wr, 3 op/s 2024-09-10T19:03:21.543 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:21.647 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:21.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:21.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:21.815 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:21.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.953 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:21.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:22.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:22.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:22.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:22.226 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:22.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.368 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:22.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.499 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:22.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:22.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:22.768 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:22.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.884 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:22] "GET /metrics HTTP/1.1" 200 228664 "" "Prometheus/2.43.0" 2024-09-10T19:03:22.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:22.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:23.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:23.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:23.144 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:23.174 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:23.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:23.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:23.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:23.442 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:23.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[18629]: cluster 2024-09-10T19:03:22.212808+0000 mgr.y (mgr.24526) 562 : cluster [DBG] pgmap v877: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[18629]: cluster 2024-09-10T19:03:22.246700+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[18629]: cluster 2024-09-10T19:03:22.494414+0000 osd.1 (osd.1) 315 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[18629]: cluster 2024-09-10T19:03:22.494803+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[32381]: cluster 2024-09-10T19:03:22.212808+0000 mgr.y (mgr.24526) 562 : cluster [DBG] pgmap v877: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[32381]: cluster 2024-09-10T19:03:22.246700+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[32381]: cluster 2024-09-10T19:03:22.494414+0000 osd.1 (osd.1) 315 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:23.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:23 smithi083 bash[32381]: cluster 2024-09-10T19:03:22.494803+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:23.545 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:23.574 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:23 smithi165 bash[22591]: cluster 2024-09-10T19:03:22.212808+0000 mgr.y (mgr.24526) 562 : cluster [DBG] pgmap v877: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-09-10T19:03:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:23 smithi165 bash[22591]: cluster 2024-09-10T19:03:22.246700+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:03:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:23 smithi165 bash[22591]: cluster 2024-09-10T19:03:22.494414+0000 osd.1 (osd.1) 315 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:23 smithi165 bash[22591]: cluster 2024-09-10T19:03:22.494803+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:23.678 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:23.712 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:23.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:23.845 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:23.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:23.980 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:23.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:24.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:24.119 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:24.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.256 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:24.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:24 smithi165 bash[22591]: cluster 2024-09-10T19:03:23.252289+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:03:24.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:24.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:24 smithi083 bash[18629]: cluster 2024-09-10T19:03:23.252289+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:24 smithi083 bash[32381]: cluster 2024-09-10T19:03:23.252289+0000 mon.a (mon.0) 2928 : cluster [DBG] osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:03:24.536 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:24.603 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:24.640 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:24.675 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:24.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:24.814 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:24.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:24.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:24.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:25.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:25.080 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:25.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:25.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:25.320 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:25.354 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:25.427 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:25.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:25.468 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:25 smithi083 bash[18629]: cluster 2024-09-10T19:03:24.213685+0000 mgr.y (mgr.24526) 563 : cluster [DBG] pgmap v880: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:25.469 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:25 smithi083 bash[32381]: cluster 2024-09-10T19:03:24.213685+0000 mgr.y (mgr.24526) 563 : cluster [DBG] pgmap v880: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:25.489 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:25.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:25 smithi165 bash[22591]: cluster 2024-09-10T19:03:24.213685+0000 mgr.y (mgr.24526) 563 : cluster [DBG] pgmap v880: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:25.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:25.628 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:25.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:25.763 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:25.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:25.900 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:25.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:26.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:26.038 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:26.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:26.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:26.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:26.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:26.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:26.414 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.451 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:26.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:26.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:26.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.726 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:26.830 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.862 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:26.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:26.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:26.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:27.097 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:27.123 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:27.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:27.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:27.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/2833645975 >> v1:172.21.15.83:6789/0 conn(0x7f1cb810b120 legacy=0x7f1cb810d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/2833645975 shutdown_connections 2024-09-10T19:03:27.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/2833645975 >> 172.21.15.83:0/2833645975 conn(0x7f1cb80f81a0 msgr2=0x7f1cb80fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/2833645975 shutdown_connections 2024-09-10T19:03:27.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/2833645975 wait complete. 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 librados: starting msgr at 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 librados: starting objecter 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 Processor -- start 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- start start 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 librados: setting wanted keys 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 librados: calling monclient init 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cb810ae70 con 0x7f1cb806fa30 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cb8100340 con 0x7f1cb8070510 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1cb8100530 con 0x7f1cb810b120 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb7fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1cb806fa30 0x7f1cb810a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49658/0 (socket says 172.21.15.83:49658) 2024-09-10T19:03:27.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb7fff640 1 -- 172.21.15.83:0/1416733847 learned_addr learned my addr 172.21.15.83:0/1416733847 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1cb810b120 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1cb806fa30 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1cb8070510 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358628558 0 0) 0x7f1cb8100530 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ca00034e0 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693744155 0 0) 0x7f1cb810ae70 con 0x7f1cb806fa30 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb8100530 con 0x7f1cb806fa30 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130415238 0 0) 0x7f1cb8100340 con 0x7f1cb8070510 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb810ae70 con 0x7f1cb8070510 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725699807 0 0) 0x7f1ca00034e0 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cb8100340 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1cb0002f80 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660881022 0 0) 0x7f1cb8100530 con 0x7f1cb806fa30 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ca00034e0 con 0x7f1cb806fa30 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1ca8001980 con 0x7f1cb806fa30 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 201190347 0 0) 0x7f1cb8100340 con 0x7f1cb810b120 2024-09-10T19:03:27.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 >> v1:172.21.15.165:6789/0 conn(0x7f1cb8070510 legacy=0x7f1cb80ff520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 >> v1:172.21.15.83:6789/0 conn(0x7f1cb806fa30 legacy=0x7f1cb810a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8100720 con 0x7f1cb810b120 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8122fd0 con 0x7f1cb810b120 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 10 client.49586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8123420 con 0x7f1cb810b120 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cbe481640 1 librados: init done 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1cb00014a0 con 0x7f1cb810b120 2024-09-10T19:03:27.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.279+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1cb0003830 con 0x7f1cb810b120 2024-09-10T19:03:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca4005010 con 0x7f1cb810b120 2024-09-10T19:03:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1cb0002230 con 0x7f1cb810b120 2024-09-10T19:03:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(572..572 src has 69..572) v4 ==== 8064+0+0 (unknown 2607818616 0 0) 0x7f1cb0088280 con 0x7f1cb810b120 2024-09-10T19:03:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 10 client.49586.objecter ms_dispatch 0x7f1cb8000b90 osd_map(572..572 src has 69..572) v4 2024-09-10T19:03:27.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 3 client.49586.objecter handle_osd_map got epochs [572,572] > 0 2024-09-10T19:03:27.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 3 client.49586.objecter handle_osd_map decoding full epoch 572 2024-09-10T19:03:27.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 20 client.49586.objecter dump_active .. 0 homeless 2024-09-10T19:03:27.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.283+0000 7f1cb4ff9640 10 client.49586.objecter ms_handle_connect 0x7f1ca00714e0 2024-09-10T19:03:27.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.291+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1cb0054800 con 0x7f1cb810b120 2024-09-10T19:03:27.347 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:27 smithi083 bash[32381]: cluster 2024-09-10T19:03:26.216041+0000 mgr.y (mgr.24526) 564 : cluster [DBG] pgmap v881: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:03:27.347 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:27 smithi083 bash[18629]: cluster 2024-09-10T19:03:26.216041+0000 mgr.y (mgr.24526) 564 : cluster [DBG] pgmap v881: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:03:27.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:27.403 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:27.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:27.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.511+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/189108513 >> v1:172.21.15.83:6789/0 conn(0x7fe79c101530 legacy=0x7fe79c101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.511+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/189108513 shutdown_connections 2024-09-10T19:03:27.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.511+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/189108513 >> 172.21.15.83:0/189108513 conn(0x7fe79c0f70b0 msgr2=0x7fe79c0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.511+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/189108513 shutdown_connections 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.511+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/189108513 wait complete. 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 librados: starting msgr at 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 librados: starting objecter 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 Processor -- start 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- start start 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 librados: setting wanted keys 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 librados: calling monclient init 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe79c06f650 con 0x7fe79c107e00 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe79c113b20 con 0x7fe79c101530 2024-09-10T19:03:27.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe79c114cc0 con 0x7fe79c1042a0 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a205f640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe79c107e00 0x7fe79c1113f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49676/0 (socket says 172.21.15.83:49676) 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a205f640 1 -- 172.21.15.83:0/2972070099 learned_addr learned my addr 172.21.15.83:0/2972070099 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter ms_handle_connect 0x7fe79c107e00 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter ms_handle_connect 0x7fe79c1042a0 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670414421 0 0) 0x7fe79c114cc0 con 0x7fe79c1042a0 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe788005640 con 0x7fe79c1042a0 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127693889 0 0) 0x7fe79c06f650 con 0x7fe79c107e00 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c114cc0 con 0x7fe79c107e00 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2013430600 0 0) 0x7fe788005640 con 0x7fe79c1042a0 2024-09-10T19:03:27.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe79c06f650 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268802580 0 0) 0x7fe79c114cc0 con 0x7fe79c107e00 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe788005640 con 0x7fe79c107e00 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter ms_handle_connect 0x7fe79c101530 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe794001780 con 0x7fe79c107e00 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe798001c60 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1536670084 0 0) 0x7fe79c06f650 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.165:6789/0 conn(0x7fe79c101530 legacy=0x7fe79c104810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6789/0 conn(0x7fe79c107e00 legacy=0x7fe79c1113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c115e60 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c114eb0 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 10 client.49592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c115300 con 0x7fe79c1042a0 2024-09-10T19:03:27.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7a3ac2640 1 librados: init done 2024-09-10T19:03:27.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe798002560 con 0x7fe79c1042a0 2024-09-10T19:03:27.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.515+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe798003470 con 0x7fe79c1042a0 2024-09-10T19:03:27.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe780004ff0 con 0x7fe79c1042a0 2024-09-10T19:03:27.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe798002100 con 0x7fe79c1042a0 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(572..572 src has 69..572) v4 ==== 8064+0+0 (unknown 2607818616 0 0) 0x7fe798087ea0 con 0x7fe79c1042a0 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 osd_map(572..572 src has 69..572) v4 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 3 client.49592.objecter handle_osd_map got epochs [572,572] > 0 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 3 client.49592.objecter handle_osd_map decoding full epoch 572 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 20 client.49592.objecter dump_active .. 0 homeless 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v572) v1 ==== 54+0+16 (unknown 1682992726 0 2420000192) 0x7fe7980882a0 con 0x7fe79c1042a0 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 1 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 1 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 0 1 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 4 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 1 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780005290 con 0x7fe780004610 2024-09-10T19:03:27.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe7880717b0 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.519+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe780004610 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.523+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe780005290 con 0x7fe780004610 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.523+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.523+0000 7fe7927fc640 10 client.49592.objecter _finish_command 1 = system:0 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.523+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 4 2024-09-10T19:03:27.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.523+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 0 1 2024-09-10T19:03:27.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 2 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 2 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780005810 osd=0 3 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 0 2 2024-09-10T19:03:27.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 4 2024-09-10T19:03:27.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 2 2024-09-10T19:03:27.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.535+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe780004cc0 con 0x7fe780004610 2024-09-10T19:03:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe780004cc0 con 0x7fe780004610 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 10 client.49592.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 4 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 0 2 2024-09-10T19:03:27.544 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 3 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 3 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 1 3 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 4 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 3 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7800022b0 con 0x7fe780004110 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe780004110 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe7800022b0 con 0x7fe780004110 2024-09-10T19:03:27.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 10 client.49592.objecter _finish_command 3 = system:0 2024-09-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 4 2024-09-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.539+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 1 3 2024-09-10T19:03:27.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.543+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set-full-ratio", "ratio": 0.95} v 0) v1 -- 0x7f1ca40052c0 con 0x7f1cb810b120 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 4 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 4 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780003fa0 osd=1 3 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 1 4 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 4 2024-09-10T19:03:27.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 4 2024-09-10T19:03:27.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.559+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe780001890 con 0x7fe780004110 2024-09-10T19:03:27.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe780001890 con 0x7fe780004110 2024-09-10T19:03:27.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 10 client.49592.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 4 2024-09-10T19:03:27.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 1 4 2024-09-10T19:03:27.568 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 5 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 5 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 2 5 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 4 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 5 2024-09-10T19:03:27.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780003df0 con 0x7fe78000d350 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe78000d350 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe780003df0 con 0x7fe78000d350 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 10 client.49592.objecter _finish_command 5 = system:0 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 4 2024-09-10T19:03:27.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.563+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 2 5 2024-09-10T19:03:27.578 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:27 smithi165 bash[22591]: cluster 2024-09-10T19:03:26.216041+0000 mgr.y (mgr.24526) 564 : cluster [DBG] pgmap v881: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 6 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 6 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe7800035e0 osd=2 3 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 2 6 2024-09-10T19:03:27.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 4 2024-09-10T19:03:27.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 6 2024-09-10T19:03:27.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.579+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe7800030d0 con 0x7fe78000d350 2024-09-10T19:03:27.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7800030d0 con 0x7fe78000d350 2024-09-10T19:03:27.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 10 client.49592.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 4 2024-09-10T19:03:27.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 2 6 2024-09-10T19:03:27.588 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:03:27.588 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:03:27.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 7 2024-09-10T19:03:27.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 7 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 3 7 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 4 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 7 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe78000d0d0 con 0x7fe780010e50 2024-09-10T19:03:27.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe780010e50 2024-09-10T19:03:27.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe78000d0d0 con 0x7fe780010e50 2024-09-10T19:03:27.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:03:27.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 10 client.49592.objecter _finish_command 7 = system:0 2024-09-10T19:03:27.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 4 2024-09-10T19:03:27.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.583+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 3 7 2024-09-10T19:03:27.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 8 2024-09-10T19:03:27.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 8 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe78000c990 osd=3 3 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 3 8 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 4 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 8 2024-09-10T19:03:27.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.603+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe7800132d0 con 0x7fe780010e50 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe7800132d0 con 0x7fe780010e50 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7927fc640 10 client.49592.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 4 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 3 8 2024-09-10T19:03:27.613 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 9 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 9 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 4 9 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 4 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 9 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.607+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe78001a260 con 0x7fe780016760 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe780016760 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe78001a260 con 0x7fe780016760 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:03:27.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 10 client.49592.objecter _finish_command 9 = system:0 2024-09-10T19:03:27.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 4 2024-09-10T19:03:27.616 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.611+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 4 9 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 10 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 10 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe7800132d0 osd=4 3 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 4 10 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 4 2024-09-10T19:03:27.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 10 2024-09-10T19:03:27.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.627+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe78000c240 con 0x7fe780016760 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe78000c240 con 0x7fe780016760 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 10 client.49592.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 4 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 4 10 2024-09-10T19:03:27.635 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 11 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 11 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 5 11 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 4 2024-09-10T19:03:27.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 11 2024-09-10T19:03:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780012df0 con 0x7fe7800128a0 2024-09-10T19:03:27.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.631+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe7800128a0 2024-09-10T19:03:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.635+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe780012df0 con 0x7fe7800128a0 2024-09-10T19:03:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.635+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:03:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.635+0000 7fe7927fc640 10 client.49592.objecter _finish_command 11 = system:0 2024-09-10T19:03:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.635+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 4 2024-09-10T19:03:27.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.635+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 5 11 2024-09-10T19:03:27.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 12 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 12 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe78000c240 osd=5 3 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 5 12 2024-09-10T19:03:27.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 4 2024-09-10T19:03:27.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 12 2024-09-10T19:03:27.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.651+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe780012df0 con 0x7fe7800128a0 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe780012df0 con 0x7fe7800128a0 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 10 client.49592.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 4 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 5 12 2024-09-10T19:03:27.660 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 13 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 13 2024-09-10T19:03:27.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 6 13 2024-09-10T19:03:27.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 4 2024-09-10T19:03:27.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 13 2024-09-10T19:03:27.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780022b20 con 0x7fe78001f020 2024-09-10T19:03:27.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.655+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe78001f020 2024-09-10T19:03:27.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.659+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe780022b20 con 0x7fe78001f020 2024-09-10T19:03:27.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.659+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:03:27.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.659+0000 7fe7927fc640 10 client.49592.objecter _finish_command 13 = system:0 2024-09-10T19:03:27.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.659+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 4 2024-09-10T19:03:27.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.659+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 6 13 2024-09-10T19:03:27.678 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 14 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 14 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780012df0 osd=6 3 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 6 14 2024-09-10T19:03:27.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 4 2024-09-10T19:03:27.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 14 2024-09-10T19:03:27.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.675+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe780022b20 con 0x7fe78001f020 2024-09-10T19:03:27.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe780022b20 con 0x7fe78001f020 2024-09-10T19:03:27.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 10 client.49592.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 4 2024-09-10T19:03:27.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 6 14 2024-09-10T19:03:27.685 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:03:27.685 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 15 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 15 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 7 15 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 4 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 15 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe780024e50 con 0x7fe7800231a0 2024-09-10T19:03:27.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.679+0000 7fe7927fc640 10 client.49592.objecter ms_handle_connect 0x7fe7800231a0 2024-09-10T19:03:27.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.683+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe780024e50 con 0x7fe7800231a0 2024-09-10T19:03:27.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.683+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:03:27.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.683+0000 7fe7927fc640 10 client.49592.objecter _finish_command 15 = system:0 2024-09-10T19:03:27.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.683+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 4 2024-09-10T19:03:27.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.683+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 7 15 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 10 client.49592.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign -1 16 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 20 client.49592.objecter _get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_remove -1 16 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 20 client.49592.objecter get_session s=0x7fe780022b20 osd=7 3 2024-09-10T19:03:27.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 15 client.49592.objecter _session_command_op_assign 7 16 2024-09-10T19:03:27.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 4 2024-09-10T19:03:27.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 10 client.49592.objecter _send_command 16 2024-09-10T19:03:27.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.699+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe780024e50 con 0x7fe7800231a0 2024-09-10T19:03:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7927fc640 1 -- 172.21.15.83:0/2972070099 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fe780024e50 con 0x7fe7800231a0 2024-09-10T19:03:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7927fc640 10 client.49592.objecter ms_dispatch 0x7fe79c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7927fc640 10 client.49592.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:27.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7927fc640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 4 2024-09-10T19:03:27.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7927fc640 15 client.49592.objecter _session_command_op_remove 7 16 2024-09-10T19:03:27.709 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 librados: watch_flush enter 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 librados: watch_flush exit 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.0 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6801/2535527483 conn(0x7fe780004610 legacy=0x7fe780002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780005810 osd=0 1 2024-09-10T19:03:27.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.1 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6805/2309624637 conn(0x7fe780004110 legacy=0x7fe78000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780003fa0 osd=1 1 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.2 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6809/386094220 conn(0x7fe78000d350 legacy=0x7fe78000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800035e0 osd=2 1 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.3 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6813/1279688021 conn(0x7fe780010e50 legacy=0x7fe780011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a205f640 1 -- 172.21.15.83:0/2972070099 reap_dead start 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c990 osd=3 1 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.4 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.165:6800/3225515265 conn(0x7fe780016760 legacy=0x7fe780018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a205f640 1 -- 172.21.15.83:0/2972070099 reap_dead start 2024-09-10T19:03:27.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe7800132d0 osd=4 1 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.5 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.165:6804/571173777 conn(0x7fe7800128a0 legacy=0x7fe78001d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe78000c240 osd=5 1 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.6 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.165:6808/1317922936 conn(0x7fe78001f020 legacy=0x7fe780021400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780012df0 osd=6 1 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter close_session for osd.7 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.165:6812/1610151650 conn(0x7fe7800231a0 legacy=0x7fe7800235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter put_session s=0x7fe780022b20 osd=7 1 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 20 client.49592.objecter shutdown clearing up homeless session... 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 10 client.49592.objecter successfully canceled tick 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe7880717b0 legacy=0x7fe788073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> v1:172.21.15.83:6790/0 conn(0x7fe79c1042a0 legacy=0x7fe79c104f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.703+0000 7fe7a205f640 1 -- 172.21.15.83:0/2972070099 reap_dead start 2024-09-10T19:03:27.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.707+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 shutdown_connections 2024-09-10T19:03:27.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.707+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 >> 172.21.15.83:0/2972070099 conn(0x7fe79c0f70b0 msgr2=0x7fe79c0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:27.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.707+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 shutdown_connections 2024-09-10T19:03:27.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.707+0000 7fe7a3ac2640 1 -- 172.21.15.83:0/2972070099 wait complete. 2024-09-10T19:03:27.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:27.707+0000 7fe7a3ac2640 1 librados: shutdown 2024-09-10T19:03:27.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:27.814 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:27.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:27.947 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:27.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:28.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:28.076 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:28.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:28.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.283+0000 7f1cb4ff9640 1 -- 172.21.15.83:0/1416733847 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set-full-ratio", "ratio": 0.95}]=0 osd set-full-ratio 0.95 v573) v1 ==== 104+0+0 (unknown 2720085048 0 0) 0x7f1cb00541a0 con 0x7f1cb810b120 2024-09-10T19:03:28.290 INFO:teuthology.orchestra.run.smithi083.stderr:osd set-full-ratio 0.95 2024-09-10T19:03:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.287+0000 7f1cbe481640 10 librados: watch_flush enter 2024-09-10T19:03:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.287+0000 7f1cbe481640 10 librados: watch_flush exit 2024-09-10T19:03:28.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.287+0000 7f1cbe481640 20 client.49586.objecter shutdown clearing up homeless session... 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.287+0000 7f1cbe481640 10 client.49586.objecter successfully canceled tick 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1ca00714e0 legacy=0x7f1ca0073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 >> v1:172.21.15.83:6790/0 conn(0x7f1cb810b120 legacy=0x7f1cb80ffc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 shutdown_connections 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 >> 172.21.15.83:0/1416733847 conn(0x7f1cb80f81a0 msgr2=0x7f1cb810d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 shutdown_connections 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 -- 172.21.15.83:0/1416733847 wait complete. 2024-09-10T19:03:28.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:28.291+0000 7f1cbe481640 1 librados: shutdown 2024-09-10T19:03:28.310 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:28.335 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:28.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:28 smithi165 bash[22591]: audit 2024-09-10T19:03:27.524233+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? v1:172.21.15.83:0/2972070099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:28 smithi165 bash[22591]: audit 2024-09-10T19:03:27.547018+0000 mon.c (mon.2) 668 : audit [INF] from='client.? v1:172.21.15.83:0/1416733847' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:28 smithi165 bash[22591]: audit 2024-09-10T19:03:27.547459+0000 mon.a (mon.0) 2929 : audit [INF] from='client.49586 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[18629]: audit 2024-09-10T19:03:27.524233+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? v1:172.21.15.83:0/2972070099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[18629]: audit 2024-09-10T19:03:27.547018+0000 mon.c (mon.2) 668 : audit [INF] from='client.? v1:172.21.15.83:0/1416733847' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[18629]: audit 2024-09-10T19:03:27.547459+0000 mon.a (mon.0) 2929 : audit [INF] from='client.49586 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[32381]: audit 2024-09-10T19:03:27.524233+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? v1:172.21.15.83:0/2972070099' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[32381]: audit 2024-09-10T19:03:27.547018+0000 mon.c (mon.2) 668 : audit [INF] from='client.? v1:172.21.15.83:0/1416733847' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.451 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:28 smithi083 bash[32381]: audit 2024-09-10T19:03:27.547459+0000 mon.a (mon.0) 2929 : audit [INF] from='client.49586 ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-09-10T19:03:28.470 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:28.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.605 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:28.709 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:28.742 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:28.767 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:03:28.767 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:03:28.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:28.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:28.983 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:29.017 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:29.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:29.151 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:29.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:29.291 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:29.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: audit 2024-09-10T19:03:28.156527+0000 mgr.y (mgr.24526) 565 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: cluster 2024-09-10T19:03:28.216885+0000 mgr.y (mgr.24526) 566 : cluster [DBG] pgmap v882: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: cluster 2024-09-10T19:03:28.274929+0000 mon.a (mon.0) 2930 : cluster [INF] Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: audit 2024-09-10T19:03:28.287278+0000 mon.a (mon.0) 2931 : audit [INF] from='client.49586 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: cluster 2024-09-10T19:03:28.287361+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[18629]: cluster 2024-09-10T19:03:29.229979+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: audit 2024-09-10T19:03:28.156527+0000 mgr.y (mgr.24526) 565 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: cluster 2024-09-10T19:03:28.216885+0000 mgr.y (mgr.24526) 566 : cluster [DBG] pgmap v882: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: cluster 2024-09-10T19:03:28.274929+0000 mon.a (mon.0) 2930 : cluster [INF] Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: audit 2024-09-10T19:03:28.287278+0000 mon.a (mon.0) 2931 : audit [INF] from='client.49586 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-10T19:03:29.408 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: cluster 2024-09-10T19:03:28.287361+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:03:29.409 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:29 smithi083 bash[32381]: cluster 2024-09-10T19:03:29.229979+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:03:29.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:03:29.427 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:29.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:29.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: audit 2024-09-10T19:03:28.156527+0000 mgr.y (mgr.24526) 565 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:29.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: cluster 2024-09-10T19:03:28.216885+0000 mgr.y (mgr.24526) 566 : cluster [DBG] pgmap v882: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:29.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: cluster 2024-09-10T19:03:28.274929+0000 mon.a (mon.0) 2930 : cluster [INF] Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-09-10T19:03:29.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: audit 2024-09-10T19:03:28.287278+0000 mon.a (mon.0) 2931 : audit [INF] from='client.49586 ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-09-10T19:03:29.541 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: cluster 2024-09-10T19:03:28.287361+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:03:29.542 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:29 smithi165 bash[22591]: cluster 2024-09-10T19:03:29.229979+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:03:29.559 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:29.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.695 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:29.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.830 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:29.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:29.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:29.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:29.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:30.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:30.107 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:30.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:30.242 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:30.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:30.381 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:30.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:30.519 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:30.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[32381]: audit 2024-09-10T19:03:29.577481+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[32381]: audit 2024-09-10T19:03:29.581116+0000 mon.c (mon.2) 669 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[32381]: cluster 2024-09-10T19:03:30.217696+0000 mgr.y (mgr.24526) 567 : cluster [DBG] pgmap v885: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[18629]: audit 2024-09-10T19:03:29.577481+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[18629]: audit 2024-09-10T19:03:29.581116+0000 mon.c (mon.2) 669 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:30.657 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:30 smithi083 bash[18629]: cluster 2024-09-10T19:03:30.217696+0000 mgr.y (mgr.24526) 567 : cluster [DBG] pgmap v885: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:30.661 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:30.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:30.795 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:30 smithi165 bash[22591]: audit 2024-09-10T19:03:29.577481+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:03:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:30 smithi165 bash[22591]: audit 2024-09-10T19:03:29.581116+0000 mon.c (mon.2) 669 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:30 smithi165 bash[22591]: cluster 2024-09-10T19:03:30.217696+0000 mgr.y (mgr.24526) 567 : cluster [DBG] pgmap v885: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:30.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:30.932 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:30.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:31.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:31.071 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:31.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:31.210 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:31.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:31.350 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:31.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:31.467 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:31.485 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:31.589 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:31.620 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/38293578 >> v1:172.21.15.83:6790/0 conn(0x7f5fe0104810 legacy=0x7f5fe0104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/38293578 shutdown_connections 2024-09-10T19:03:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/38293578 >> 172.21.15.83:0/38293578 conn(0x7f5fe00fa370 msgr2=0x7f5fe00fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/38293578 shutdown_connections 2024-09-10T19:03:31.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/38293578 wait complete. 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 librados: starting msgr at 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 librados: starting objecter 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 Processor -- start 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- start start 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 librados: setting wanted keys 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 librados: calling monclient init 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fe011e190 con 0x7f5fe010b130 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fe011f390 con 0x7f5fe01075a0 2024-09-10T19:03:31.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fe0120590 con 0x7f5fe0104810 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe4d4e640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5fe0104810 0x7f5fe01007d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56372/0 (socket says 172.21.15.83:56372) 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe4d4e640 1 -- 172.21.15.83:0/3263791707 learned_addr learned my addr 172.21.15.83:0/3263791707 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe0104810 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe010b130 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter ms_handle_connect 0x7f5fe01075a0 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778730461 0 0) 0x7f5fe0120590 con 0x7f5fe0104810 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc0034e0 con 0x7f5fe0104810 2024-09-10T19:03:31.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042047166 0 0) 0x7f5fe011e190 con 0x7f5fe010b130 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe0120590 con 0x7f5fe010b130 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249730811 0 0) 0x7f5fe011f390 con 0x7f5fe01075a0 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe011e190 con 0x7f5fe01075a0 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110060254 0 0) 0x7f5fcc0034e0 con 0x7f5fe0104810 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe011f390 con 0x7f5fe0104810 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5fd8003050 con 0x7f5fe0104810 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232228277 0 0) 0x7f5fe0120590 con 0x7f5fe010b130 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fcc0034e0 con 0x7f5fe010b130 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5fd0001840 con 0x7f5fe010b130 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1090717317 0 0) 0x7f5fe011e190 con 0x7f5fe01075a0 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe0120590 con 0x7f5fe01075a0 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 524480880 0 0) 0x7f5fe011f390 con 0x7f5fe0104810 2024-09-10T19:03:31.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 >> v1:172.21.15.165:6789/0 conn(0x7f5fe01075a0 legacy=0x7f5fe0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 >> v1:172.21.15.83:6789/0 conn(0x7f5fe010b130 legacy=0x7f5fe011c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe0121790 con 0x7f5fe0104810 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe011e380 con 0x7f5fe0104810 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 10 client.49598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe011e850 con 0x7f5fe0104810 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fe6fb2640 1 librados: init done 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5fd8001900 con 0x7f5fe0104810 2024-09-10T19:03:31.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.707+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5fd8003a80 con 0x7f5fe0104810 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0005010 con 0x7f5fe0104810 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5fd8002600 con 0x7f5fe0104810 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(574..574 src has 69..574) v4 ==== 8064+0+0 (unknown 324049046 0 0) 0x7f5fd808ac50 con 0x7f5fe0104810 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 10 client.49598.objecter ms_dispatch 0x7f5fe0000b90 osd_map(574..574 src has 69..574) v4 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 3 client.49598.objecter handle_osd_map got epochs [574,574] > 0 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 3 client.49598.objecter handle_osd_map decoding full epoch 574 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 20 client.49598.objecter dump_active .. 0 homeless 2024-09-10T19:03:31.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.711+0000 7f5fdd7fa640 10 client.49598.objecter ms_handle_connect 0x7f5fcc071770 2024-09-10T19:03:31.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.719+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5fd8057230 con 0x7f5fe0104810 2024-09-10T19:03:31.724 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:31.748 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:31.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:31.887 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:31.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:31.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.899+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5fc00052c0 con 0x7f5fe0104810 2024-09-10T19:03:31.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.963+0000 7f5fdd7fa640 1 -- 172.21.15.83:0/3263791707 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v575) v1 ==== 91+0+0 (unknown 914004965 0 0) 0x7f5fd8056bd0 con 0x7f5fe0104810 2024-09-10T19:03:31.970 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:03:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.967+0000 7f5fe6fb2640 10 librados: watch_flush enter 2024-09-10T19:03:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.967+0000 7f5fe6fb2640 10 librados: watch_flush exit 2024-09-10T19:03:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.967+0000 7f5fe6fb2640 20 client.49598.objecter shutdown clearing up homeless session... 2024-09-10T19:03:31.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.967+0000 7f5fe6fb2640 10 client.49598.objecter successfully canceled tick 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5fcc071770 legacy=0x7f5fcc073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 >> v1:172.21.15.83:6790/0 conn(0x7f5fe0104810 legacy=0x7f5fe01007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 shutdown_connections 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 >> 172.21.15.83:0/3263791707 conn(0x7f5fe00fa370 msgr2=0x7f5fe0064740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 shutdown_connections 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 -- 172.21.15.83:0/3263791707 wait complete. 2024-09-10T19:03:31.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:31.971+0000 7f5fe6fb2640 1 librados: shutdown 2024-09-10T19:03:31.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:32.022 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:32.126 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:32.159 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:32.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:32.272 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:31 smithi165 bash[22591]: audit 2024-09-10T19:03:31.906470+0000 mon.c (mon.2) 670 : audit [INF] from='client.? v1:172.21.15.83:0/3263791707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:31 smithi165 bash[22591]: audit 2024-09-10T19:03:31.907013+0000 mon.a (mon.0) 2935 : audit [INF] from='client.49598 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.273 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:31 smithi165 bash[22591]: cluster 2024-09-10T19:03:31.957782+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:32.288 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[18629]: audit 2024-09-10T19:03:31.906470+0000 mon.c (mon.2) 670 : audit [INF] from='client.? v1:172.21.15.83:0/3263791707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[18629]: audit 2024-09-10T19:03:31.907013+0000 mon.a (mon.0) 2935 : audit [INF] from='client.49598 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[18629]: cluster 2024-09-10T19:03:31.957782+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[32381]: audit 2024-09-10T19:03:31.906470+0000 mon.c (mon.2) 670 : audit [INF] from='client.? v1:172.21.15.83:0/3263791707' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[32381]: audit 2024-09-10T19:03:31.907013+0000 mon.a (mon.0) 2935 : audit [INF] from='client.49598 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:03:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:31 smithi083 bash[32381]: cluster 2024-09-10T19:03:31.957782+0000 mon.a (mon.0) 2936 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:03:32.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:32.426 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:32.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:32.563 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:32.659 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:03:32.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:32.699 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:32.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:32.834 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:32.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:32.951 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:32] "GET /metrics HTTP/1.1" 200 228970 "" "Prometheus/2.43.0" 2024-09-10T19:03:32.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:32.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:33.074 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:33.101 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:33.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:33.239 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: cluster 2024-09-10T19:03:31.924385+0000 osd.4 (osd.4) 330 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: cluster 2024-09-10T19:03:31.924734+0000 osd.4 (osd.4) 331 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: audit 2024-09-10T19:03:31.965860+0000 mon.a (mon.0) 2937 : audit [INF] from='client.49598 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: cluster 2024-09-10T19:03:31.966083+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: cluster 2024-09-10T19:03:32.219012+0000 mgr.y (mgr.24526) 568 : cluster [DBG] pgmap v887: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: audit 2024-09-10T19:03:32.287926+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: audit 2024-09-10T19:03:32.289618+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: audit 2024-09-10T19:03:32.290868+0000 mon.a (mon.0) 2939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:32 smithi165 bash[22591]: audit 2024-09-10T19:03:32.292081+0000 mon.a (mon.0) 2940 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: cluster 2024-09-10T19:03:31.924385+0000 osd.4 (osd.4) 330 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: cluster 2024-09-10T19:03:31.924734+0000 osd.4 (osd.4) 331 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: audit 2024-09-10T19:03:31.965860+0000 mon.a (mon.0) 2937 : audit [INF] from='client.49598 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: cluster 2024-09-10T19:03:31.966083+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: cluster 2024-09-10T19:03:32.219012+0000 mgr.y (mgr.24526) 568 : cluster [DBG] pgmap v887: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: audit 2024-09-10T19:03:32.287926+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: audit 2024-09-10T19:03:32.289618+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: audit 2024-09-10T19:03:32.290868+0000 mon.a (mon.0) 2939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[32381]: audit 2024-09-10T19:03:32.292081+0000 mon.a (mon.0) 2940 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: cluster 2024-09-10T19:03:31.924385+0000 osd.4 (osd.4) 330 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: cluster 2024-09-10T19:03:31.924734+0000 osd.4 (osd.4) 331 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: audit 2024-09-10T19:03:31.965860+0000 mon.a (mon.0) 2937 : audit [INF] from='client.49598 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: cluster 2024-09-10T19:03:31.966083+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: cluster 2024-09-10T19:03:32.219012+0000 mgr.y (mgr.24526) 568 : cluster [DBG] pgmap v887: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: audit 2024-09-10T19:03:32.287926+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: audit 2024-09-10T19:03:32.289618+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: audit 2024-09-10T19:03:32.290868+0000 mon.a (mon.0) 2939 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:33.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:32 smithi083 bash[18629]: audit 2024-09-10T19:03:32.292081+0000 mon.a (mon.0) 2940 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-94"}]: dispatch 2024-09-10T19:03:33.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:33.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:33.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:33.511 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:33.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:33.648 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:33.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:33.787 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:33.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:33.925 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:33.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:34.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:34.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:33 smithi165 bash[22591]: cluster 2024-09-10T19:03:32.983471+0000 osd.1 (osd.1) 317 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:03:34.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:33 smithi165 bash[22591]: cluster 2024-09-10T19:03:32.987086+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:03:34.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:33 smithi165 bash[22591]: cluster 2024-09-10T19:03:32.999248+0000 osd.1 (osd.1) 318 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:03:34.036 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:33 smithi165 bash[22591]: cluster 2024-09-10T19:03:33.151325+0000 osd.3 (osd.3) 325 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:03:34.051 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:34.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:34.188 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:34.216 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[32381]: cluster 2024-09-10T19:03:32.983471+0000 osd.1 (osd.1) 317 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[32381]: cluster 2024-09-10T19:03:32.987086+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[32381]: cluster 2024-09-10T19:03:32.999248+0000 osd.1 (osd.1) 318 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[32381]: cluster 2024-09-10T19:03:33.151325+0000 osd.3 (osd.3) 325 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[18629]: cluster 2024-09-10T19:03:32.983471+0000 osd.1 (osd.1) 317 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[18629]: cluster 2024-09-10T19:03:32.987086+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[18629]: cluster 2024-09-10T19:03:32.999248+0000 osd.1 (osd.1) 318 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:03:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:33 smithi083 bash[18629]: cluster 2024-09-10T19:03:33.151325+0000 osd.3 (osd.3) 325 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:03:34.293 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:34.325 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:34.428 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:34.465 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:34.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:34.600 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:34.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:34.740 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:34.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:03:34.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:34.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:34.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:34.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:35.016 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:35.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.132 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:33.929968+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:33.933318+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:33.952335+0000 osd.7 (osd.7) 325 : cluster [DBG] 3.e scrub starts 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.004235+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: audit 2024-09-10T19:03:34.005244+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: audit 2024-09-10T19:03:34.008150+0000 mon.a (mon.0) 2943 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.084170+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.087797+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.219952+0000 mgr.y (mgr.24526) 569 : cluster [DBG] pgmap v890: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: audit 2024-09-10T19:03:34.234070+0000 mon.a (mon.0) 2944 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:35.133 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.234959+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[32381]: audit 2024-09-10T19:03:34.235050+0000 mon.b (mon.1) 438 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: cluster 2024-09-10T19:03:33.929968+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: cluster 2024-09-10T19:03:33.933318+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: cluster 2024-09-10T19:03:33.952335+0000 osd.7 (osd.7) 325 : cluster [DBG] 3.e scrub starts 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.004235+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: audit 2024-09-10T19:03:34.005244+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: audit 2024-09-10T19:03:34.008150+0000 mon.a (mon.0) 2943 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:34 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.084170+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.087797+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:03:35.134 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.219952+0000 mgr.y (mgr.24526) 569 : cluster [DBG] pgmap v890: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:35.135 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: audit 2024-09-10T19:03:34.234070+0000 mon.a (mon.0) 2944 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:35.135 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.234959+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:03:35.135 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: audit 2024-09-10T19:03:34.235050+0000 mon.b (mon.1) 438 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:35.154 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:35.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:35.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:33.929968+0000 osd.5 (osd.5) 318 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:03:35.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:33.933318+0000 osd.5 (osd.5) 319 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:33.952335+0000 osd.7 (osd.7) 325 : cluster [DBG] 3.e scrub starts 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.004235+0000 mon.a (mon.0) 2942 : cluster [DBG] osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: audit 2024-09-10T19:03:34.005244+0000 mon.b (mon.1) 437 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: audit 2024-09-10T19:03:34.008150+0000 mon.a (mon.0) 2943 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.084170+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.087797+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.219952+0000 mgr.y (mgr.24526) 569 : cluster [DBG] pgmap v890: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: audit 2024-09-10T19:03:34.234070+0000 mon.a (mon.0) 2944 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.234959+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:03:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:34 smithi165 bash[22591]: audit 2024-09-10T19:03:34.235050+0000 mon.b (mon.1) 438 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:03:35.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:35.416 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:35.519 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:35.554 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:35.658 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.687 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:35.734 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:03:35.734 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:03:35.734 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:03:35.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:03:35.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:35.823 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:35.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:35.959 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:35.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:36.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:36.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:36.207 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:35 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.940450+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:03:36.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:35 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.945665+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:03:36.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:35 smithi165 bash[22591]: cluster 2024-09-10T19:03:34.995420+0000 mon.a (mon.0) 2946 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:36.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:35 smithi165 bash[22591]: cluster 2024-09-10T19:03:35.255063+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:03:36.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:36.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.940450+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:03:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:35 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.945665+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:03:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[18629]: cluster 2024-09-10T19:03:34.995420+0000 mon.a (mon.0) 2946 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[18629]: cluster 2024-09-10T19:03:35.255063+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:03:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.940450+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:03:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.945665+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:03:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[32381]: cluster 2024-09-10T19:03:34.995420+0000 mon.a (mon.0) 2946 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:36 smithi083 bash[32381]: cluster 2024-09-10T19:03:35.255063+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:03:36.332 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:36.365 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:36.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.501 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:36.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:36.641 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:36.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.779 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:36.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.917 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:36.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:37.020 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:37.048 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:37.151 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:37.185 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:37.289 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:35.732712+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:35.736164+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:35.930761+0000 osd.5 (osd.5) 322 : cluster [DBG] 3.c scrub starts 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:35.935197+0000 osd.5 (osd.5) 323 : cluster [DBG] 3.c scrub ok 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:36.111362+0000 osd.1 (osd.1) 319 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:36.114654+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:36.222316+0000 mgr.y (mgr.24526) 570 : cluster [DBG] pgmap v893: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:36.621212+0000 osd.2 (osd.2) 243 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:03:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[18629]: cluster 2024-09-10T19:03:36.628113+0000 osd.2 (osd.2) 244 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:35.732712+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:35.736164+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:35.930761+0000 osd.5 (osd.5) 322 : cluster [DBG] 3.c scrub starts 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:35.935197+0000 osd.5 (osd.5) 323 : cluster [DBG] 3.c scrub ok 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:36.111362+0000 osd.1 (osd.1) 319 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:36.114654+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:36.222316+0000 mgr.y (mgr.24526) 570 : cluster [DBG] pgmap v893: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:36.621212+0000 osd.2 (osd.2) 243 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:03:37.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:37 smithi083 bash[32381]: cluster 2024-09-10T19:03:36.628113+0000 osd.2 (osd.2) 244 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:03:37.322 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:35.732712+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:35.736164+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:35.930761+0000 osd.5 (osd.5) 322 : cluster [DBG] 3.c scrub starts 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:35.935197+0000 osd.5 (osd.5) 323 : cluster [DBG] 3.c scrub ok 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:36.111362+0000 osd.1 (osd.1) 319 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:36.114654+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:36.222316+0000 mgr.y (mgr.24526) 570 : cluster [DBG] pgmap v893: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:36.621212+0000 osd.2 (osd.2) 243 : cluster [DBG] 64.7 scrub starts 2024-09-10T19:03:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:37 smithi165 bash[22591]: cluster 2024-09-10T19:03:36.628113+0000 osd.2 (osd.2) 244 : cluster [DBG] 64.7 scrub ok 2024-09-10T19:03:37.426 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:37.434 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:37.458 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:37.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:37.567 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:03:37.567 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:03:37.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:37.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:37.729 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:37.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:37.868 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:37.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:37.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:38.006 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:38.109 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:38.137 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:38.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:38.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:03:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:38.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:38.413 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:38.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:38.529 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:38.548 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:38.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:38.687 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:38.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:38.822 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:38.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:38.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:38.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:39.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:39.083 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.288769+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.289252+0000 osd.7 (osd.7) 327 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.748743+0000 osd.6 (osd.6) 271 : cluster [DBG] 12.f scrub starts 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.751967+0000 osd.6 (osd.6) 272 : cluster [DBG] 12.f scrub ok 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.752471+0000 osd.0 (osd.0) 307 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:03:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.757885+0000 osd.0 (osd.0) 308 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:03:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.990743+0000 osd.4 (osd.4) 334 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:03:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:37.995051+0000 osd.4 (osd.4) 335 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:03:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.016868+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:03:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: audit 2024-09-10T19:03:38.167876+0000 mgr.y (mgr.24526) 571 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:39 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.223110+0000 mgr.y (mgr.24526) 572 : cluster [DBG] pgmap v895: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:03:39.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:39.220 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:39.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.288769+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.289252+0000 osd.7 (osd.7) 327 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.748743+0000 osd.6 (osd.6) 271 : cluster [DBG] 12.f scrub starts 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.751967+0000 osd.6 (osd.6) 272 : cluster [DBG] 12.f scrub ok 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.752471+0000 osd.0 (osd.0) 307 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.757885+0000 osd.0 (osd.0) 308 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.990743+0000 osd.4 (osd.4) 334 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:37.995051+0000 osd.4 (osd.4) 335 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.016868+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: audit 2024-09-10T19:03:38.167876+0000 mgr.y (mgr.24526) 571 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.223110+0000 mgr.y (mgr.24526) 572 : cluster [DBG] pgmap v895: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.288769+0000 osd.7 (osd.7) 326 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:39.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.289252+0000 osd.7 (osd.7) 327 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.748743+0000 osd.6 (osd.6) 271 : cluster [DBG] 12.f scrub starts 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.751967+0000 osd.6 (osd.6) 272 : cluster [DBG] 12.f scrub ok 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.752471+0000 osd.0 (osd.0) 307 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.757885+0000 osd.0 (osd.0) 308 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.990743+0000 osd.4 (osd.4) 334 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:37.995051+0000 osd.4 (osd.4) 335 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.016868+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: audit 2024-09-10T19:03:38.167876+0000 mgr.y (mgr.24526) 571 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:39 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.223110+0000 mgr.y (mgr.24526) 572 : cluster [DBG] pgmap v895: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:03:39.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:39.357 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:39.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:39.465 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:39.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:39.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:39.628 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:39.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:39.766 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:39.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:39.904 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:39.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:40.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:40.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:40.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.174 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:40.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.733433+0000 osd.0 (osd.0) 309 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.734087+0000 osd.6 (osd.6) 273 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.738843+0000 osd.0 (osd.0) 310 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.739205+0000 osd.6 (osd.6) 274 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.914485+0000 osd.5 (osd.5) 324 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:03:40.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:38.924276+0000 osd.5 (osd.5) 325 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.011938+0000 osd.7 (osd.7) 328 : cluster [DBG] 12.d scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.015974+0000 osd.7 (osd.7) 329 : cluster [DBG] 12.d scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.105020+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.107857+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.628050+0000 osd.2 (osd.2) 245 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.631370+0000 osd.2 (osd.2) 246 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.733433+0000 osd.0 (osd.0) 309 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.734087+0000 osd.6 (osd.6) 273 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.738843+0000 osd.0 (osd.0) 310 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.739205+0000 osd.6 (osd.6) 274 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.914485+0000 osd.5 (osd.5) 324 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:38.924276+0000 osd.5 (osd.5) 325 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:03:40.287 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.011938+0000 osd.7 (osd.7) 328 : cluster [DBG] 12.d scrub starts 2024-09-10T19:03:40.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.015974+0000 osd.7 (osd.7) 329 : cluster [DBG] 12.d scrub ok 2024-09-10T19:03:40.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.105020+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:03:40.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.107857+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:03:40.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.628050+0000 osd.2 (osd.2) 245 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:03:40.288 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:40 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.631370+0000 osd.2 (osd.2) 246 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:03:40.305 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.733433+0000 osd.0 (osd.0) 309 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:03:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.734087+0000 osd.6 (osd.6) 273 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:03:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.738843+0000 osd.0 (osd.0) 310 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:03:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.739205+0000 osd.6 (osd.6) 274 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:03:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.914485+0000 osd.5 (osd.5) 324 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:38.924276+0000 osd.5 (osd.5) 325 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.011938+0000 osd.7 (osd.7) 328 : cluster [DBG] 12.d scrub starts 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.015974+0000 osd.7 (osd.7) 329 : cluster [DBG] 12.d scrub ok 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.105020+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.107857+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.628050+0000 osd.2 (osd.2) 245 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:03:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:40 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.631370+0000 osd.2 (osd.2) 246 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:03:40.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.443 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:40.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:40.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:40.746 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:40.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:40.859 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:40.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:40.962 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:40.997 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:41.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:41.136 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:41.213 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.948785+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:03:41.213 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:39.952739+0000 osd.5 (osd.5) 327 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:03:41.213 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.022281+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.025681+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.086549+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.091937+0000 osd.3 (osd.3) 329 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.223925+0000 mgr.y (mgr.24526) 573 : cluster [DBG] pgmap v896: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.620175+0000 osd.2 (osd.2) 247 : cluster [DBG] 4.d scrub starts 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.624100+0000 osd.2 (osd.2) 248 : cluster [DBG] 4.d scrub ok 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.948785+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:39.952739+0000 osd.5 (osd.5) 327 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.022281+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:03:41.214 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.025681+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:03:41.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.086549+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:03:41.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.091937+0000 osd.3 (osd.3) 329 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:03:41.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.223925+0000 mgr.y (mgr.24526) 573 : cluster [DBG] pgmap v896: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:41.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.620175+0000 osd.2 (osd.2) 247 : cluster [DBG] 4.d scrub starts 2024-09-10T19:03:41.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:41 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.624100+0000 osd.2 (osd.2) 248 : cluster [DBG] 4.d scrub ok 2024-09-10T19:03:41.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:41.273 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.948785+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:39.952739+0000 osd.5 (osd.5) 327 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.022281+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.025681+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.086549+0000 osd.3 (osd.3) 328 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.091937+0000 osd.3 (osd.3) 329 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:03:41.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.223925+0000 mgr.y (mgr.24526) 573 : cluster [DBG] pgmap v896: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:03:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.620175+0000 osd.2 (osd.2) 247 : cluster [DBG] 4.d scrub starts 2024-09-10T19:03:41.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.624100+0000 osd.2 (osd.2) 248 : cluster [DBG] 4.d scrub ok 2024-09-10T19:03:41.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:41.407 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:41.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:41.545 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:41.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:41.661 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:41.677 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:41.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:41.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:41.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:41.953 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:41.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:42.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:42.067 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.909946+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:03:42.087 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:42.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:42.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:42.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:42.341 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:42 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.909946+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:03:42.341 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:42 smithi165 bash[22591]: cluster 2024-09-10T19:03:40.913517+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:03:42.341 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:42 smithi165 bash[22591]: cluster 2024-09-10T19:03:41.592355+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-10T19:03:42.341 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:42 smithi165 bash[22591]: cluster 2024-09-10T19:03:41.595704+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-10T19:03:42.360 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:42.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:42.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:42.471 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[18629]: cluster 2024-09-10T19:03:40.913517+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:03:42.471 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[18629]: cluster 2024-09-10T19:03:41.592355+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-10T19:03:42.472 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[18629]: cluster 2024-09-10T19:03:41.595704+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-10T19:03:42.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.909946+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:03:42.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[32381]: cluster 2024-09-10T19:03:40.913517+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:03:42.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[32381]: cluster 2024-09-10T19:03:41.592355+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.c scrub starts 2024-09-10T19:03:42.472 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[32381]: cluster 2024-09-10T19:03:41.595704+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.c scrub ok 2024-09-10T19:03:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:42.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:42.631 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:42.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:42.767 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:42.777 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:42] "GET /metrics HTTP/1.1" 200 228970 "" "Prometheus/2.43.0" 2024-09-10T19:03:42.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:42.904 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:42.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:43.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:43.024 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:43.037 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:43.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:43.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:43.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.313 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:41.819268+0000 osd.6 (osd.6) 275 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:41.822728+0000 osd.6 (osd.6) 276 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:42.076032+0000 osd.7 (osd.7) 332 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:42.079931+0000 osd.7 (osd.7) 333 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:42.121864+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:03:43.315 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:42.125250+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[32381]: cluster 2024-09-10T19:03:42.226257+0000 mgr.y (mgr.24526) 574 : cluster [DBG] pgmap v897: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:41.819268+0000 osd.6 (osd.6) 275 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:41.822728+0000 osd.6 (osd.6) 276 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:42.076032+0000 osd.7 (osd.7) 332 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:42.079931+0000 osd.7 (osd.7) 333 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:42.121864+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:42.125250+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:03:43.316 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:43 smithi083 bash[18629]: cluster 2024-09-10T19:03:42.226257+0000 mgr.y (mgr.24526) 574 : cluster [DBG] pgmap v897: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:43.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:41.819268+0000 osd.6 (osd.6) 275 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:41.822728+0000 osd.6 (osd.6) 276 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:42.076032+0000 osd.7 (osd.7) 332 : cluster [DBG] 64.1d scrub starts 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:42.079931+0000 osd.7 (osd.7) 333 : cluster [DBG] 64.1d scrub ok 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:42.121864+0000 osd.3 (osd.3) 330 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:03:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:42.125250+0000 osd.3 (osd.3) 331 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:03:43.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:43 smithi165 bash[22591]: cluster 2024-09-10T19:03:42.226257+0000 mgr.y (mgr.24526) 574 : cluster [DBG] pgmap v897: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:43.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:43.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:43.589 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:43.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:43.828 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:43.863 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:43.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:43.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:43.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:44.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:44.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:44 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.055767+0000 osd.4 (osd.4) 336 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:03:44.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:44 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.058003+0000 osd.4 (osd.4) 337 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:03:44.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:44.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:44.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:44 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.055767+0000 osd.4 (osd.4) 336 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:03:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:44 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.058003+0000 osd.4 (osd.4) 337 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:03:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:44 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.055767+0000 osd.4 (osd.4) 336 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:03:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:44 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.058003+0000 osd.4 (osd.4) 337 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:03:44.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:44.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:44.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:44.533 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:44.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:44.668 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:44.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:44.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:44.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:44.940 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:44.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:45.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.030+0000 7fc75fd46640 1 -- 172.21.15.83:0/382028983 >> v1:172.21.15.83:6790/0 conn(0x7fc7581014e0 legacy=0x7fc7581018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.030+0000 7fc75fd46640 1 -- 172.21.15.83:0/382028983 shutdown_connections 2024-09-10T19:03:45.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.030+0000 7fc75fd46640 1 -- 172.21.15.83:0/382028983 >> 172.21.15.83:0/382028983 conn(0x7fc758075290 msgr2=0x7fc7580f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:45.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.030+0000 7fc75fd46640 1 -- 172.21.15.83:0/382028983 shutdown_connections 2024-09-10T19:03:45.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.030+0000 7fc75fd46640 1 -- 172.21.15.83:0/382028983 wait complete. 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 librados: starting msgr at 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 librados: starting objecter 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 Processor -- start 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- start start 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 librados: setting wanted keys 2024-09-10T19:03:45.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 librados: calling monclient init 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7581135f0 con 0x7fc758107e30 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc758100c10 con 0x7fc7581014e0 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc758100da0 con 0x7fc7581042a0 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75e2e3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc758107e30 0x7fc758121770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45794/0 (socket says 172.21.15.83:45794) 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75e2e3640 1 -- 172.21.15.83:0/2740930518 learned_addr learned my addr 172.21.15.83:0/2740930518 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75d2e1640 1 --1- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6790/0 conn(0x7fc7581042a0 0x7fc75811e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40984/0 (socket says 172.21.15.83:40984) 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7581014e0 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc758107e30 2024-09-10T19:03:45.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter ms_handle_connect 0x7fc7581042a0 2024-09-10T19:03:45.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245317972 0 0) 0x7fc7581135f0 con 0x7fc758107e30 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744003120 con 0x7fc758107e30 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097269038 0 0) 0x7fc758100c10 con 0x7fc7581014e0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7581135f0 con 0x7fc7581014e0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187307795 0 0) 0x7fc758100da0 con 0x7fc7581042a0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc758100c10 con 0x7fc7581042a0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313485231 0 0) 0x7fc744003120 con 0x7fc758107e30 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc758100da0 con 0x7fc758107e30 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc750002c90 con 0x7fc758107e30 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964300075 0 0) 0x7fc7581135f0 con 0x7fc7581014e0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc744003120 con 0x7fc7581014e0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc754002050 con 0x7fc7581014e0 2024-09-10T19:03:45.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3434727739 0 0) 0x7fc758100c10 con 0x7fc7581042a0 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7581135f0 con 0x7fc7581042a0 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7480018c0 con 0x7fc7581042a0 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2892696325 0 0) 0x7fc758100da0 con 0x7fc758107e30 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6790/0 conn(0x7fc7581042a0 legacy=0x7fc75811e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.165:6789/0 conn(0x7fc7581014e0 legacy=0x7fc758100500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758125ec0 con 0x7fc758107e30 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc758122e90 con 0x7fc758107e30 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 10 client.25527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7581232e0 con 0x7fc758107e30 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc75fd46640 1 librados: init done 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc750001a90 con 0x7fc758107e30 2024-09-10T19:03:45.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.034+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc750005d20 con 0x7fc758107e30 2024-09-10T19:03:45.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc73c004ff0 con 0x7fc758107e30 2024-09-10T19:03:45.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc75001c1c0 con 0x7fc758107e30 2024-09-10T19:03:45.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(582..582 src has 69..582) v4 ==== 8036+0+0 (unknown 2504768174 0 0) 0x7fc75008a810 con 0x7fc758107e30 2024-09-10T19:03:45.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 osd_map(582..582 src has 69..582) v4 2024-09-10T19:03:45.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 3 client.25527.objecter handle_osd_map got epochs [582,582] > 0 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 3 client.25527.objecter handle_osd_map decoding full epoch 582 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 20 client.25527.objecter dump_active .. 0 homeless 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v582) v1 ==== 54+0+16 (unknown 1624389807 0 2420000192) 0x7fc75008cbb0 con 0x7fc758107e30 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc744071760 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 10 client.25527.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 1 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 1 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 0 1 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 4 2024-09-10T19:03:45.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 10 client.25527.objecter _send_command 1 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c005290 con 0x7fc73c004610 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c004610 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c005290 con 0x7fc73c004610 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 1 = system:0 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 4 2024-09-10T19:03:45.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.038+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 0 1 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 10 client.25527.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 2 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 2 2024-09-10T19:03:45.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c005810 osd=0 3 2024-09-10T19:03:45.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 0 2 2024-09-10T19:03:45.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 4 2024-09-10T19:03:45.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 10 client.25527.objecter _send_command 2 2024-09-10T19:03:45.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.058+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c004cc0 con 0x7fc73c004610 2024-09-10T19:03:45.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c004cc0 con 0x7fc73c004610 2024-09-10T19:03:45.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 4 2024-09-10T19:03:45.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 0 2 2024-09-10T19:03:45.068 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:03:45.068 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:03:45.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 10 client.25527.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 1 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 4 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 10 client.25527.objecter _send_command 3 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c0022b0 con 0x7fc73c004110 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c004110 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c0022b0 con 0x7fc73c004110 2024-09-10T19:03:45.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:03:45.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 3 = system:0 2024-09-10T19:03:45.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 4 2024-09-10T19:03:45.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.062+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 1 3 2024-09-10T19:03:45.074 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.701640+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.702002+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.739218+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.743695+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.841547+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.845114+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.916954+0000 osd.5 (osd.5) 330 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:43.920087+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.060795+0000 mon.a (mon.0) 2949 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.097753+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.108761+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.185411+0000 osd.3 (osd.3) 332 : cluster [DBG] 12.b scrub starts 2024-09-10T19:03:45.076 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.188397+0000 osd.3 (osd.3) 333 : cluster [DBG] 12.b scrub ok 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.225231+0000 mon.a (mon.0) 2950 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.227039+0000 mgr.y (mgr.24526) 575 : cluster [DBG] pgmap v899: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.236562+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: audit 2024-09-10T19:03:44.569683+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[32381]: audit 2024-09-10T19:03:45.043262+0000 mon.a (mon.0) 2952 : audit [DBG] from='client.? v1:172.21.15.83:0/2740930518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.701640+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.702002+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:45.077 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.739218+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.743695+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.841547+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.845114+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.916954+0000 osd.5 (osd.5) 330 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:43.920087+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.060795+0000 mon.a (mon.0) 2949 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.097753+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.108761+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.185411+0000 osd.3 (osd.3) 332 : cluster [DBG] 12.b scrub starts 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.188397+0000 osd.3 (osd.3) 333 : cluster [DBG] 12.b scrub ok 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.225231+0000 mon.a (mon.0) 2950 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.227039+0000 mgr.y (mgr.24526) 575 : cluster [DBG] pgmap v899: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.236562+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:03:45.078 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: audit 2024-09-10T19:03:44.569683+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:45.079 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[18629]: audit 2024-09-10T19:03:45.043262+0000 mon.a (mon.0) 2952 : audit [DBG] from='client.? v1:172.21.15.83:0/2740930518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 10 client.25527.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 4 2024-09-10T19:03:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 4 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c003fa0 osd=1 3 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 1 4 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 4 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 10 client.25527.objecter _send_command 4 2024-09-10T19:03:45.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.078+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c001890 con 0x7fc73c004110 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c001890 con 0x7fc73c004110 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 4 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 1 4 2024-09-10T19:03:45.088 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 10 client.25527.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 5 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 5 2024-09-10T19:03:45.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 2 5 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 4 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 10 client.25527.objecter _send_command 5 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c003df0 con 0x7fc73c00d350 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c00d350 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c003df0 con 0x7fc73c00d350 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 5 = system:0 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 4 2024-09-10T19:03:45.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.082+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 2 5 2024-09-10T19:03:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 10 client.25527.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 6 2024-09-10T19:03:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 6 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c0035e0 osd=2 3 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 2 6 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 4 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 10 client.25527.objecter _send_command 6 2024-09-10T19:03:45.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c003060 con 0x7fc73c00d350 2024-09-10T19:03:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c003060 con 0x7fc73c00d350 2024-09-10T19:03:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 4 2024-09-10T19:03:45.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 2 6 2024-09-10T19:03:45.105 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:03:45.105 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:03:45.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 10 client.25527.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 7 2024-09-10T19:03:45.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 7 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 3 7 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 4 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 10 client.25527.objecter _send_command 7 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c00d010 con 0x7fc73c010e50 2024-09-10T19:03:45.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.098+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c010e50 2024-09-10T19:03:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.102+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c00d010 con 0x7fc73c010e50 2024-09-10T19:03:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.102+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:03:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.102+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 7 = system:0 2024-09-10T19:03:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.102+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 4 2024-09-10T19:03:45.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.102+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 3 7 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 10 client.25527.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 8 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 8 2024-09-10T19:03:45.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c00c990 osd=3 3 2024-09-10T19:03:45.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 3 8 2024-09-10T19:03:45.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 4 2024-09-10T19:03:45.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 10 client.25527.objecter _send_command 8 2024-09-10T19:03:45.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c0132d0 con 0x7fc73c010e50 2024-09-10T19:03:45.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c0132d0 con 0x7fc73c010e50 2024-09-10T19:03:45.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 4 2024-09-10T19:03:45.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 3 8 2024-09-10T19:03:45.126 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 10 client.25527.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 9 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 9 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 4 9 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 4 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 10 client.25527.objecter _send_command 9 2024-09-10T19:03:45.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.118+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c01a260 con 0x7fc73c016760 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c016760 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c01a260 con 0x7fc73c016760 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 9 = system:0 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 4 2024-09-10T19:03:45.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.122+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 4 9 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 10 client.25527.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 10 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 10 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c0132d0 osd=4 3 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 4 10 2024-09-10T19:03:45.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 4 2024-09-10T19:03:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 10 client.25527.objecter _send_command 10 2024-09-10T19:03:45.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.134+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c01a260 con 0x7fc73c016760 2024-09-10T19:03:45.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.138+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c01a260 con 0x7fc73c016760 2024-09-10T19:03:45.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.138+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.138+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.138+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 4 2024-09-10T19:03:45.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.138+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 4 10 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 10 client.25527.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 11 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 11 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 5 11 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 4 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 10 client.25527.objecter _send_command 11 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c01e5f0 con 0x7fc73c01a910 2024-09-10T19:03:45.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c01a910 2024-09-10T19:03:45.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c01e5f0 con 0x7fc73c01a910 2024-09-10T19:03:45.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:03:45.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 11 = system:0 2024-09-10T19:03:45.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 4 2024-09-10T19:03:45.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.142+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 5 11 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 10 client.25527.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 12 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 12 2024-09-10T19:03:45.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c01a260 osd=5 3 2024-09-10T19:03:45.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 5 12 2024-09-10T19:03:45.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 4 2024-09-10T19:03:45.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 10 client.25527.objecter _send_command 12 2024-09-10T19:03:45.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.166+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c01e5f0 con 0x7fc73c01a910 2024-09-10T19:03:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c01e5f0 con 0x7fc73c01a910 2024-09-10T19:03:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 4 2024-09-10T19:03:45.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 5 12 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 10 client.25527.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 13 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 13 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 6 13 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 4 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 10 client.25527.objecter _send_command 13 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c013120 con 0x7fc73c020650 2024-09-10T19:03:45.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c020650 2024-09-10T19:03:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c013120 con 0x7fc73c020650 2024-09-10T19:03:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:03:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 13 = system:0 2024-09-10T19:03:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 4 2024-09-10T19:03:45.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.170+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 6 13 2024-09-10T19:03:45.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 10 client.25527.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 14 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 14 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c01e5f0 osd=6 3 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 6 14 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 4 2024-09-10T19:03:45.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 10 client.25527.objecter _send_command 14 2024-09-10T19:03:45.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.186+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c013120 con 0x7fc73c020650 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c013120 con 0x7fc73c020650 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 4 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 6 14 2024-09-10T19:03:45.197 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 10 client.25527.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 15 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 15 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 7 15 2024-09-10T19:03:45.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 4 2024-09-10T19:03:45.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 10 client.25527.objecter _send_command 15 2024-09-10T19:03:45.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.190+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc73c026e80 con 0x7fc73c020c00 2024-09-10T19:03:45.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 10 client.25527.objecter ms_handle_connect 0x7fc73c020c00 2024-09-10T19:03:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fc73c026e80 con 0x7fc73c020c00 2024-09-10T19:03:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:03:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 15 = system:0 2024-09-10T19:03:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 4 2024-09-10T19:03:45.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.194+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 7 15 2024-09-10T19:03:45.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 10 client.25527.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign -1 16 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 20 client.25527.objecter _get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_remove -1 16 2024-09-10T19:03:45.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 20 client.25527.objecter get_session s=0x7fc73c013120 osd=7 3 2024-09-10T19:03:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 15 client.25527.objecter _session_command_op_assign 7 16 2024-09-10T19:03:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 4 2024-09-10T19:03:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 10 client.25527.objecter _send_command 16 2024-09-10T19:03:45.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.214+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fc73c026e40 con 0x7fc73c020c00 2024-09-10T19:03:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc74e7fc640 1 -- 172.21.15.83:0/2740930518 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fc73c026e40 con 0x7fc73c020c00 2024-09-10T19:03:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc74e7fc640 10 client.25527.objecter ms_dispatch 0x7fc758000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:03:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc74e7fc640 10 client.25527.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:03:45.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc74e7fc640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 4 2024-09-10T19:03:45.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc74e7fc640 15 client.25527.objecter _session_command_op_remove 7 16 2024-09-10T19:03:45.224 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 librados: watch_flush enter 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 librados: watch_flush exit 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.0 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6801/2535527483 conn(0x7fc73c004610 legacy=0x7fc73c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c005810 osd=0 1 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.1 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6805/2309624637 conn(0x7fc73c004110 legacy=0x7fc73c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c003fa0 osd=1 1 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.2 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6809/386094220 conn(0x7fc73c00d350 legacy=0x7fc73c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0035e0 osd=2 1 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.3 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75e2e3640 1 -- 172.21.15.83:0/2740930518 reap_dead start 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6813/1279688021 conn(0x7fc73c010e50 legacy=0x7fc73c011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c00c990 osd=3 1 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.4 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.165:6800/3225515265 conn(0x7fc73c016760 legacy=0x7fc73c018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c0132d0 osd=4 1 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.5 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.165:6804/571173777 conn(0x7fc73c01a910 legacy=0x7fc73c01cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01a260 osd=5 1 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.6 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.165:6808/1317922936 conn(0x7fc73c020650 legacy=0x7fc73c011990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c01e5f0 osd=6 1 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter close_session for osd.7 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.165:6812/1610151650 conn(0x7fc73c020c00 legacy=0x7fc73c0255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter put_session s=0x7fc73c013120 osd=7 1 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 20 client.25527.objecter shutdown clearing up homeless session... 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 10 client.25527.objecter successfully canceled tick 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75e2e3640 1 -- 172.21.15.83:0/2740930518 reap_dead start 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc744071760 legacy=0x7fc744073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> v1:172.21.15.83:6789/0 conn(0x7fc758107e30 legacy=0x7fc758121770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 shutdown_connections 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 >> 172.21.15.83:0/2740930518 conn(0x7fc758075290 msgr2=0x7fc7580f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:45.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 shutdown_connections 2024-09-10T19:03:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 -- 172.21.15.83:0/2740930518 wait complete. 2024-09-10T19:03:45.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:45.218+0000 7fc75fd46640 1 librados: shutdown 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.701640+0000 osd.0 (osd.0) 311 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.702002+0000 osd.0 (osd.0) 312 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.739218+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.743695+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.841547+0000 osd.6 (osd.6) 277 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.845114+0000 osd.6 (osd.6) 278 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.916954+0000 osd.5 (osd.5) 330 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:43.920087+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:03:45.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.060795+0000 mon.a (mon.0) 2949 : cluster [DBG] osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.097753+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.108761+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.185411+0000 osd.3 (osd.3) 332 : cluster [DBG] 12.b scrub starts 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.188397+0000 osd.3 (osd.3) 333 : cluster [DBG] 12.b scrub ok 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.225231+0000 mon.a (mon.0) 2950 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.227039+0000 mgr.y (mgr.24526) 575 : cluster [DBG] pgmap v899: 238 pgs: 1 active+clean+scrubbing, 237 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.236562+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: audit 2024-09-10T19:03:44.569683+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:45.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[22591]: audit 2024-09-10T19:03:45.043262+0000 mon.a (mon.0) 2952 : audit [DBG] from='client.? v1:172.21.15.83:0/2740930518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:03:45.310 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:45.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:45.341 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[35790]: debug 2024-09-10T19:03:45.286+0000 7f0e91dfc640 -1 snap_mapper.add_oid found existing snaps mapped on 64:c8ca7e0a:test-rados-api-smithi083-77828-97::foo:31, removing 2024-09-10T19:03:45.341 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:03:45 smithi083 bash[50483]: debug 2024-09-10T19:03:45.286+0000 7f6640c27640 -1 snap_mapper.add_oid found existing snaps mapped on 64:c8ca7e0a:test-rados-api-smithi083-77828-97::foo:31, removing 2024-09-10T19:03:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:45.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:03:45 smithi165 bash[48829]: debug 2024-09-10T19:03:45.286+0000 7f107a447640 -1 snap_mapper.add_oid found existing snaps mapped on 64:c8ca7e0a:test-rados-api-smithi083-77828-97::foo:31, removing 2024-09-10T19:03:45.443 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.476 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:45.580 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.611 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:45.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.744 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:45.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.880 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:45.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:45.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:46.019 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:46.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.745477+0000 osd.0 (osd.0) 315 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.751815+0000 osd.0 (osd.0) 316 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.905034+0000 osd.5 (osd.5) 332 : cluster [DBG] 64.c scrub starts 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:44.909213+0000 osd.5 (osd.5) 333 : cluster [DBG] 64.c scrub ok 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:45.161230+0000 osd.3 (osd.3) 334 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: cluster 2024-09-10T19:03:45.165502+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:03:46.136 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: audit 2024-09-10T19:03:45.322493+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: audit 2024-09-10T19:03:45.324321+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: audit 2024-09-10T19:03:45.325290+0000 mon.a (mon.0) 2953 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[32381]: audit 2024-09-10T19:03:45.326890+0000 mon.a (mon.0) 2954 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.745477+0000 osd.0 (osd.0) 315 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.751815+0000 osd.0 (osd.0) 316 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.905034+0000 osd.5 (osd.5) 332 : cluster [DBG] 64.c scrub starts 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:44.909213+0000 osd.5 (osd.5) 333 : cluster [DBG] 64.c scrub ok 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:45.161230+0000 osd.3 (osd.3) 334 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: cluster 2024-09-10T19:03:45.165502+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: audit 2024-09-10T19:03:45.322493+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: audit 2024-09-10T19:03:45.324321+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.138 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: audit 2024-09-10T19:03:45.325290+0000 mon.a (mon.0) 2953 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.138 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:46 smithi083 bash[18629]: audit 2024-09-10T19:03:45.326890+0000 mon.a (mon.0) 2954 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:46.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:46.279 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:03:46.279 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:03:46.279 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T19:03:46.279 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T19:03:46.279 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:03:46.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:46.297 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.745477+0000 osd.0 (osd.0) 315 : cluster [DBG] 64.1e scrub starts 2024-09-10T19:03:46.297 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.751815+0000 osd.0 (osd.0) 316 : cluster [DBG] 64.1e scrub ok 2024-09-10T19:03:46.297 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.905034+0000 osd.5 (osd.5) 332 : cluster [DBG] 64.c scrub starts 2024-09-10T19:03:46.297 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:44.909213+0000 osd.5 (osd.5) 333 : cluster [DBG] 64.c scrub ok 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:45.161230+0000 osd.3 (osd.3) 334 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: cluster 2024-09-10T19:03:45.165502+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: audit 2024-09-10T19:03:45.322493+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: audit 2024-09-10T19:03:45.324321+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: audit 2024-09-10T19:03:45.325290+0000 mon.a (mon.0) 2953 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:46.298 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:46 smithi165 bash[22591]: audit 2024-09-10T19:03:45.326890+0000 mon.a (mon.0) 2954 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-96"}]: dispatch 2024-09-10T19:03:46.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:46.414 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:46.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:46.551 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:46.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:46.687 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:46.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:46.821 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:46.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:03:46.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:46.965 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:46.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:47.069 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:47.095 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:47.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:47.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:45.988548+0000 osd.4 (osd.4) 338 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:03:47.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:45.992373+0000 osd.4 (osd.4) 339 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:03:47.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.064199+0000 osd.1 (osd.1) 321 : cluster [DBG] 2.d scrub starts 2024-09-10T19:03:47.212 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.067808+0000 osd.1 (osd.1) 322 : cluster [DBG] 2.d scrub ok 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.084662+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.087832+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.170057+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.172656+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.229348+0000 mgr.y (mgr.24526) 576 : cluster [DBG] pgmap v901: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:47.213 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:47 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.240587+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:03:47.233 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:47.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:47.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:47.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:47.480 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:45.988548+0000 osd.4 (osd.4) 338 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:45.992373+0000 osd.4 (osd.4) 339 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.064199+0000 osd.1 (osd.1) 321 : cluster [DBG] 2.d scrub starts 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.067808+0000 osd.1 (osd.1) 322 : cluster [DBG] 2.d scrub ok 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.084662+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.087832+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.170057+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.172656+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.229348+0000 mgr.y (mgr.24526) 576 : cluster [DBG] pgmap v901: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.240587+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:03:47.481 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:45.988548+0000 osd.4 (osd.4) 338 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:45.992373+0000 osd.4 (osd.4) 339 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.064199+0000 osd.1 (osd.1) 321 : cluster [DBG] 2.d scrub starts 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.067808+0000 osd.1 (osd.1) 322 : cluster [DBG] 2.d scrub ok 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.084662+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.087832+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.170057+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.172656+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.229348+0000 mgr.y (mgr.24526) 576 : cluster [DBG] pgmap v901: 238 pgs: 238 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-09-10T19:03:47.482 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:47 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.240587+0000 mon.a (mon.0) 2955 : cluster [DBG] osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:03:47.509 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:47.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:47.645 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:47.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:47.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:47.780 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:47.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:47.920 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:47.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:48.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:48.055 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:48.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.996420+0000 osd.4 (osd.4) 340 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:03:48.143 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:46.999477+0000 osd.4 (osd.4) 341 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:03:48.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:48.192 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:48.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.996420+0000 osd.4 (osd.4) 340 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:46.999477+0000 osd.4 (osd.4) 341 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.000923+0000 osd.5 (osd.5) 334 : cluster [DBG] 64.f scrub starts 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.005245+0000 osd.5 (osd.5) 335 : cluster [DBG] 64.f scrub ok 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.030990+0000 osd.1 (osd.1) 323 : cluster [DBG] 64.b scrub starts 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.036743+0000 osd.1 (osd.1) 324 : cluster [DBG] 64.b scrub ok 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.179709+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub starts 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.184050+0000 osd.3 (osd.3) 339 : cluster [DBG] 6.c scrub ok 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.244357+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:03:48.307 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: audit 2024-09-10T19:03:47.248328+0000 mon.b (mon.1) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.308 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:48 smithi165 bash[22591]: audit 2024-09-10T19:03:47.255509+0000 mon.a (mon.0) 2957 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.326 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:48.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.459 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- 172.21.15.83:0/2835646742 >> v1:172.21.15.83:6789/0 conn(0x7f23e0108ec0 legacy=0x7f23e010b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- 172.21.15.83:0/2835646742 shutdown_connections 2024-09-10T19:03:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- 172.21.15.83:0/2835646742 >> 172.21.15.83:0/2835646742 conn(0x7f23e00f7010 msgr2=0x7f23e00f9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- 172.21.15.83:0/2835646742 shutdown_connections 2024-09-10T19:03:48.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- 172.21.15.83:0/2835646742 wait complete. 2024-09-10T19:03:48.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 librados: starting msgr at 2024-09-10T19:03:48.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 librados: starting objecter 2024-09-10T19:03:48.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 Processor -- start 2024-09-10T19:03:48.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 -- start start 2024-09-10T19:03:48.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 librados: setting wanted keys 2024-09-10T19:03:48.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.474+0000 7f23e85ce640 1 librados: calling monclient init 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23e011c070 con 0x7f23e00fe050 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23e011d270 con 0x7f23e0108ec0 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23e011e470 con 0x7f23e00fd570 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e636a640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f23e00fd570 0x7f23e01053d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40990/0 (socket says 172.21.15.83:40990) 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e636a640 1 -- 172.21.15.83:0/1736342998 learned_addr learned my addr 172.21.15.83:0/1736342998 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter ms_handle_connect 0x7f23e00fd570 2024-09-10T19:03:48.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter ms_handle_connect 0x7f23e0108ec0 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter ms_handle_connect 0x7f23e00fe050 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464126826 0 0) 0x7f23e011e470 con 0x7f23e00fd570 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c80034e0 con 0x7f23e00fd570 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448064505 0 0) 0x7f23e011d270 con 0x7f23e0108ec0 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e011e470 con 0x7f23e0108ec0 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612515452 0 0) 0x7f23e011c070 con 0x7f23e00fe050 2024-09-10T19:03:48.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23e011d270 con 0x7f23e00fe050 2024-09-10T19:03:48.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035699384 0 0) 0x7f23c80034e0 con 0x7f23e00fd570 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23e011c070 con 0x7f23e00fd570 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f23d8001d00 con 0x7f23e00fd570 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3795717285 0 0) 0x7f23e011d270 con 0x7f23e00fe050 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c80034e0 con 0x7f23e00fe050 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152824023 0 0) 0x7f23e011e470 con 0x7f23e0108ec0 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23e011d270 con 0x7f23e0108ec0 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f23d4001830 con 0x7f23e00fe050 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f23d0001e50 con 0x7f23e0108ec0 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1988359503 0 0) 0x7f23e011c070 con 0x7f23e00fd570 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 >> v1:172.21.15.165:6789/0 conn(0x7f23e0108ec0 legacy=0x7f23e011a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 >> v1:172.21.15.83:6789/0 conn(0x7f23e00fe050 legacy=0x7f23e0105ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:48.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e011f670 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e011d460 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 10 client.49610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e011d8b0 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23e85ce640 1 librados: init done 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f23d8003080 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.478+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f23d8005e70 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc005010 con 0x7f23e00fd570 2024-09-10T19:03:48.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f23d80022e0 con 0x7f23e00fd570 2024-09-10T19:03:48.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(585..585 src has 69..585) v4 ==== 8036+0+0 (unknown 148819625 0 0) 0x7f23d808a970 con 0x7f23e00fd570 2024-09-10T19:03:48.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23deffd640 10 client.49610.objecter ms_dispatch 0x7f23e0000b90 osd_map(585..585 src has 69..585) v4 2024-09-10T19:03:48.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23deffd640 3 client.49610.objecter handle_osd_map got epochs [585,585] > 0 2024-09-10T19:03:48.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.482+0000 7f23deffd640 3 client.49610.objecter handle_osd_map decoding full epoch 585 2024-09-10T19:03:48.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.486+0000 7f23deffd640 20 client.49610.objecter dump_active .. 0 homeless 2024-09-10T19:03:48.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.486+0000 7f23deffd640 10 client.49610.objecter ms_handle_connect 0x7f23c8075bd0 2024-09-10T19:03:48.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.490+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f23d8056a90 con 0x7f23e00fd570 2024-09-10T19:03:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.000923+0000 osd.5 (osd.5) 334 : cluster [DBG] 64.f scrub starts 2024-09-10T19:03:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.005245+0000 osd.5 (osd.5) 335 : cluster [DBG] 64.f scrub ok 2024-09-10T19:03:48.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.030990+0000 osd.1 (osd.1) 323 : cluster [DBG] 64.b scrub starts 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.036743+0000 osd.1 (osd.1) 324 : cluster [DBG] 64.b scrub ok 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.179709+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub starts 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.184050+0000 osd.3 (osd.3) 339 : cluster [DBG] 6.c scrub ok 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.244357+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: audit 2024-09-10T19:03:47.248328+0000 mon.b (mon.1) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[18629]: audit 2024-09-10T19:03:47.255509+0000 mon.a (mon.0) 2957 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.996420+0000 osd.4 (osd.4) 340 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:46.999477+0000 osd.4 (osd.4) 341 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.000923+0000 osd.5 (osd.5) 334 : cluster [DBG] 64.f scrub starts 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.005245+0000 osd.5 (osd.5) 335 : cluster [DBG] 64.f scrub ok 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.030990+0000 osd.1 (osd.1) 323 : cluster [DBG] 64.b scrub starts 2024-09-10T19:03:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.036743+0000 osd.1 (osd.1) 324 : cluster [DBG] 64.b scrub ok 2024-09-10T19:03:48.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.179709+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub starts 2024-09-10T19:03:48.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.184050+0000 osd.3 (osd.3) 339 : cluster [DBG] 6.c scrub ok 2024-09-10T19:03:48.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.244357+0000 mon.a (mon.0) 2956 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:03:48.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: audit 2024-09-10T19:03:47.248328+0000 mon.b (mon.1) 441 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:48 smithi083 bash[32381]: audit 2024-09-10T19:03:47.255509+0000 mon.a (mon.0) 2957 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:48.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:48.596 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:48.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:48.678+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f23cc0052c0 con 0x7f23e00fd570 2024-09-10T19:03:48.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:48.733 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:48.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.871 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:48.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:48.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:49.011 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.899131+0000 osd.6 (osd.6) 279 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:03:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:47.901788+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:03:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.036504+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.039055+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.177967+0000 mgr.y (mgr.24526) 577 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.230220+0000 mgr.y (mgr.24526) 578 : cluster [DBG] pgmap v904: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.248319+0000 mon.a (mon.0) 2958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.248453+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.273848+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.276411+0000 mon.a (mon.0) 2960 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.685988+0000 mon.c (mon.2) 672 : audit [INF] from='client.? v1:172.21.15.83:0/1736342998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:49 smithi165 bash[22591]: audit 2024-09-10T19:03:48.686502+0000 mon.a (mon.0) 2961 : audit [INF] from='client.49610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.144 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:49.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.238+0000 7f23deffd640 1 -- 172.21.15.83:0/1736342998 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v586) v1 ==== 101+0+0 (unknown 3288334485 0 0) 0x7f23d80568b0 con 0x7f23e00fd570 2024-09-10T19:03:49.243 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 10 librados: watch_flush enter 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 10 librados: watch_flush exit 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 20 client.49610.objecter shutdown clearing up homeless session... 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 10 client.49610.objecter successfully canceled tick 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 >> v1:172.21.15.83:6800/3071793735 conn(0x7f23c8075bd0 legacy=0x7f23c8078080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 >> v1:172.21.15.83:6790/0 conn(0x7f23e00fd570 legacy=0x7f23e01053d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 shutdown_connections 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 >> 172.21.15.83:0/1736342998 conn(0x7f23e00f7010 msgr2=0x7f23e010b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 shutdown_connections 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 -- 172.21.15.83:0/1736342998 wait complete. 2024-09-10T19:03:49.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:49.242+0000 7f23e85ce640 1 librados: shutdown 2024-09-10T19:03:49.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.270 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:49.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.899131+0000 osd.6 (osd.6) 279 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:47.901788+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.036504+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.039055+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.177967+0000 mgr.y (mgr.24526) 577 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.230220+0000 mgr.y (mgr.24526) 578 : cluster [DBG] pgmap v904: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.248319+0000 mon.a (mon.0) 2958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.248453+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:03:49.387 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.273848+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.276411+0000 mon.a (mon.0) 2960 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.685988+0000 mon.c (mon.2) 672 : audit [INF] from='client.? v1:172.21.15.83:0/1736342998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[32381]: audit 2024-09-10T19:03:48.686502+0000 mon.a (mon.0) 2961 : audit [INF] from='client.49610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.899131+0000 osd.6 (osd.6) 279 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:47.901788+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.036504+0000 osd.5 (osd.5) 336 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.039055+0000 osd.5 (osd.5) 337 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.177967+0000 mgr.y (mgr.24526) 577 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.230220+0000 mgr.y (mgr.24526) 578 : cluster [DBG] pgmap v904: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 op/s 2024-09-10T19:03:49.388 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.248319+0000 mon.a (mon.0) 2958 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:03:49.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.248453+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:03:49.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.273848+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.276411+0000 mon.a (mon.0) 2960 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:03:49.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.685988+0000 mon.c (mon.2) 672 : audit [INF] from='client.? v1:172.21.15.83:0/1736342998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.389 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:49 smithi083 bash[18629]: audit 2024-09-10T19:03:48.686502+0000 mon.a (mon.0) 2961 : audit [INF] from='client.49610 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:03:49.406 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:49.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:03:49.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:49.568 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:49.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:49.697 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:49.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:49.827 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:49.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:49.963 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:49.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:50.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:50.102 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:50.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:50.239 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- 172.21.15.83:0/1874952112 >> v1:172.21.15.83:6790/0 conn(0x7fdb7c107e00 legacy=0x7fdb7c10a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- 172.21.15.83:0/1874952112 shutdown_connections 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- 172.21.15.83:0/1874952112 >> 172.21.15.83:0/1874952112 conn(0x7fdb7c0f7070 msgr2=0x7fdb7c0f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- 172.21.15.83:0/1874952112 shutdown_connections 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- 172.21.15.83:0/1874952112 wait complete. 2024-09-10T19:03:50.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 librados: starting msgr at 2024-09-10T19:03:50.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 librados: starting objecter 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 Processor -- start 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 -- start start 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 librados: setting wanted keys 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.270+0000 7fdb83ad1640 1 librados: calling monclient init 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb7c11af00 con 0x7fdb7c1042a0 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb7c11c100 con 0x7fdb7c107e00 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb7c11d300 con 0x7fdb7c101530 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb8186d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fdb7c101530 0x7fdb7c104810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:41010/0 (socket says 172.21.15.83:41010) 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb8186d640 1 -- 172.21.15.83:0/138250730 learned_addr learned my addr 172.21.15.83:0/138250730 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter ms_handle_connect 0x7fdb7c1042a0 2024-09-10T19:03:50.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter ms_handle_connect 0x7fdb7c101530 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080375115 0 0) 0x7fdb7c11af00 con 0x7fdb7c1042a0 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb68003120 con 0x7fdb7c1042a0 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259729250 0 0) 0x7fdb7c11d300 con 0x7fdb7c101530 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c11af00 con 0x7fdb7c101530 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter ms_handle_connect 0x7fdb7c107e00 2024-09-10T19:03:50.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:03:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820224274 0 0) 0x7fdb7c11c100 con 0x7fdb7c107e00 2024-09-10T19:03:50.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c11d300 con 0x7fdb7c107e00 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 718316623 0 0) 0x7fdb68003120 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb7c11c100 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdb6c0019a0 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759007048 0 0) 0x7fdb7c11af00 con 0x7fdb7c101530 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb68003120 con 0x7fdb7c101530 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdb74001c40 con 0x7fdb7c101530 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1339044648 0 0) 0x7fdb7c11c100 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 >> v1:172.21.15.83:6790/0 conn(0x7fdb7c101530 legacy=0x7fdb7c104810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 >> v1:172.21.15.165:6789/0 conn(0x7fdb7c107e00 legacy=0x7fdb7c119600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c11e500 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c11d4f0 con 0x7fdb7c1042a0 2024-09-10T19:03:50.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fdb6c0022d0 con 0x7fdb7c1042a0 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fdb6c0031b0 con 0x7fdb7c1042a0 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 10 client.25539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c11d940 con 0x7fdb7c1042a0 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 librados: init done 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7c0fcc30 con 0x7fdb7c1042a0 2024-09-10T19:03:50.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.274+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fdb6c001e70 con 0x7fdb7c1042a0 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(587..587 src has 69..587) v4 ==== 8044+0+0 (unknown 3326422856 0 0) 0x7fdb6c087d20 con 0x7fdb7c1042a0 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 10 client.25539.objecter ms_dispatch 0x7fdb7c000b90 osd_map(587..587 src has 69..587) v4 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 3 client.25539.objecter handle_osd_map got epochs [587,587] > 0 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 3 client.25539.objecter handle_osd_map decoding full epoch 587 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 20 client.25539.objecter dump_active .. 0 homeless 2024-09-10T19:03:50.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.278+0000 7fdb727fc640 10 client.25539.objecter ms_handle_connect 0x7fdb68071760 2024-09-10T19:03:50.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.282+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fdb6c0542b0 con 0x7fdb7c1042a0 2024-09-10T19:03:50.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.992797+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:48.996258+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.227993+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.229090+0000 mon.a (mon.0) 2963 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: audit 2024-09-10T19:03:49.240609+0000 mon.a (mon.0) 2964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: audit 2024-09-10T19:03:49.240792+0000 mon.a (mon.0) 2965 : audit [INF] from='client.49610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.240896+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: audit 2024-09-10T19:03:49.241127+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: audit 2024-09-10T19:03:49.250943+0000 mon.a (mon.0) 2967 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.609983+0000 osd.2 (osd.2) 251 : cluster [DBG] 64.a scrub starts 2024-09-10T19:03:50.357 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:50 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.619305+0000 osd.2 (osd.2) 252 : cluster [DBG] 64.a scrub ok 2024-09-10T19:03:50.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.992797+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:03:50.419 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:48.996258+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.227993+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.229090+0000 mon.a (mon.0) 2963 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: audit 2024-09-10T19:03:49.240609+0000 mon.a (mon.0) 2964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: audit 2024-09-10T19:03:49.240792+0000 mon.a (mon.0) 2965 : audit [INF] from='client.49610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.240896+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: audit 2024-09-10T19:03:49.241127+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: audit 2024-09-10T19:03:49.250943+0000 mon.a (mon.0) 2967 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.609983+0000 osd.2 (osd.2) 251 : cluster [DBG] 64.a scrub starts 2024-09-10T19:03:50.420 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.619305+0000 osd.2 (osd.2) 252 : cluster [DBG] 64.a scrub ok 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.992797+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:48.996258+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.227993+0000 mon.a (mon.0) 2962 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.229090+0000 mon.a (mon.0) 2963 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: audit 2024-09-10T19:03:49.240609+0000 mon.a (mon.0) 2964 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: audit 2024-09-10T19:03:49.240792+0000 mon.a (mon.0) 2965 : audit [INF] from='client.49610 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.240896+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: audit 2024-09-10T19:03:49.241127+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: audit 2024-09-10T19:03:49.250943+0000 mon.a (mon.0) 2967 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]: dispatch 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.609983+0000 osd.2 (osd.2) 251 : cluster [DBG] 64.a scrub starts 2024-09-10T19:03:50.421 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:50 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.619305+0000 osd.2 (osd.2) 252 : cluster [DBG] 64.a scrub ok 2024-09-10T19:03:50.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.454+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb7c10da10 con 0x7fdb68071760 2024-09-10T19:03:50.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:50.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.478+0000 7fdb727fc640 1 -- 172.21.15.83:0/138250730 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+645679 (unknown 2965378022 0 4273381595) 0x7fdb7c10da10 con 0x7fdb68071760 2024-09-10T19:03:50.490 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 10 librados: watch_flush enter 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 10 librados: watch_flush exit 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 20 client.25539.objecter shutdown clearing up homeless session... 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 10 client.25539.objecter successfully canceled tick 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 >> v1:172.21.15.83:6800/3071793735 conn(0x7fdb68071760 legacy=0x7fdb68073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:50.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 >> v1:172.21.15.83:6789/0 conn(0x7fdb7c1042a0 legacy=0x7fdb7c104f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:50.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 shutdown_connections 2024-09-10T19:03:50.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 >> 172.21.15.83:0/138250730 conn(0x7fdb7c0f7070 msgr2=0x7fdb7c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:50.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 shutdown_connections 2024-09-10T19:03:50.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 -- 172.21.15.83:0/138250730 wait complete. 2024-09-10T19:03:50.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:03:50.486+0000 7fdb83ad1640 1 librados: shutdown 2024-09-10T19:03:50.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:50.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:50.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:50.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:50.786 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:50.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:50.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:50.920 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:50.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:51.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:51.059 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:51.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:51.195 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:51.299 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:51.330 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":908,"stamp":"2024-09-10T19:03:50.250823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8589462,"num_objects":396,"num_object_clones":16,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":2259,"num_read_kb":1876,"num_write":1535,"num_write_kb":13166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":435112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3884,"ondisk_log_size":3884,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2536060,"kb_used_data":94724,"kb_used_omap":13,"kb_used_meta":2441138,"kb_avail":747425156,"statfs":{"total":767960285184,"available":765363359744,"internally_reserved":0,"allocated":96997376,"data_stored":67252531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2499726016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014254"},"pg_stats":[{"pgid":"233.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"233.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:03:47.234541+0000","last_change":"2024-09-10T19:03:47.234541+0000","last_active":"2024-09-10T19:03:47.234541+0000","last_peered":"2024-09-10T19:03:47.234541+0000","last_clean":"2024-09-10T19:03:47.234541+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:03:47.234541+0000","last_undegraded":"2024-09-10T19:03:47.234541+0000","last_fullsized":"2024-09-10T19:03:47.234541+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:47.234541+0000","last_clean_scrub_stamp":"2024-09-10T19:03:47.234541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"542'44","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064791+0000","last_change":"2024-09-10T19:03:22.257357+0000","last_active":"2024-09-10T19:03:44.064791+0000","last_peered":"2024-09-10T19:03:44.064791+0000","last_clean":"2024-09-10T19:03:44.064791+0000","last_became_active":"2024-09-10T18:54:56.848338+0000","last_became_peered":"2024-09-10T18:54:56.848338+0000","last_unstale":"2024-09-10T19:03:44.064791+0000","last_undegraded":"2024-09-10T19:03:44.064791+0000","last_fullsized":"2024-09-10T19:03:44.064791+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'41","last_scrub_stamp":"2024-09-10T19:01:49.532552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:49.532552+0000","objects_scrubbed":5,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048290629999999998,"objects_trimmed":0,"snaptrim_duration":0.00094149900000000005,"stat_sum":{"num_bytes":1258,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":19,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1e","version":"542'75","reported_seq":625,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.752091+0000","last_change":"2024-09-10T19:03:44.752007+0000","last_active":"2024-09-10T19:03:44.752091+0000","last_peered":"2024-09-10T19:03:44.752091+0000","last_clean":"2024-09-10T19:03:44.752091+0000","last_became_active":"2024-09-10T18:54:56.843994+0000","last_became_peered":"2024-09-10T18:54:56.843994+0000","last_unstale":"2024-09-10T19:03:44.752091+0000","last_undegraded":"2024-09-10T19:03:44.752091+0000","last_fullsized":"2024-09-10T19:03:44.752091+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'75","last_scrub_stamp":"2024-09-10T19:03:44.751831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:03:44.751831+0000","objects_scrubbed":12,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:02.274000+0000","scrub_duration":0.0062126860000000002,"objects_trimmed":0,"snaptrim_duration":0.00093746300000000003,"stat_sum":{"num_bytes":2394,"num_objects":12,"num_object_clones":1,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1d","version":"523'29","reported_seq":596,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061953+0000","last_change":"2024-09-10T19:03:42.080127+0000","last_active":"2024-09-10T19:03:44.061953+0000","last_peered":"2024-09-10T19:03:44.061953+0000","last_clean":"2024-09-10T19:03:44.061953+0000","last_became_active":"2024-09-10T18:54:56.869589+0000","last_became_peered":"2024-09-10T18:54:56.869589+0000","last_unstale":"2024-09-10T19:03:44.061953+0000","last_undegraded":"2024-09-10T19:03:44.061953+0000","last_fullsized":"2024-09-10T19:03:44.061953+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'29","last_scrub_stamp":"2024-09-10T19:03:42.079946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:03:42.079946+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:08.506863+0000","scrub_duration":0.0037576100000000002,"objects_trimmed":0,"snaptrim_duration":0.0019427209999999999,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1c","version":"575'77","reported_seq":619,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150468+0000","last_change":"2024-09-10T19:03:24.253359+0000","last_active":"2024-09-10T19:03:44.150468+0000","last_peered":"2024-09-10T19:03:44.150468+0000","last_clean":"2024-09-10T19:03:44.150468+0000","last_became_active":"2024-09-10T18:54:56.873127+0000","last_became_peered":"2024-09-10T18:54:56.873127+0000","last_unstale":"2024-09-10T19:03:44.150468+0000","last_undegraded":"2024-09-10T19:03:44.150468+0000","last_fullsized":"2024-09-10T19:03:44.150468+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"244'51","last_scrub_stamp":"2024-09-10T19:01:28.876707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.876707+0000","objects_scrubbed":3,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004224579,"objects_trimmed":0,"snaptrim_duration":0.25725277899999999,"stat_sum":{"num_bytes":83,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":25,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1b","version":"542'75","reported_seq":617,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150533+0000","last_change":"2024-09-10T19:03:22.251559+0000","last_active":"2024-09-10T19:03:44.150533+0000","last_peered":"2024-09-10T19:03:44.150533+0000","last_clean":"2024-09-10T19:03:44.150533+0000","last_became_active":"2024-09-10T18:54:56.860515+0000","last_became_peered":"2024-09-10T18:54:56.860515+0000","last_unstale":"2024-09-10T19:03:44.150533+0000","last_undegraded":"2024-09-10T19:03:44.150533+0000","last_fullsized":"2024-09-10T19:03:44.150533+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'68","last_scrub_stamp":"2024-09-10T19:01:53.808906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.808906+0000","objects_scrubbed":4,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045370890000000002,"objects_trimmed":0,"snaptrim_duration":0.0046995329999999997,"stat_sum":{"num_bytes":744,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":30,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"557'48","reported_seq":599,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061276+0000","last_change":"2024-09-10T19:03:22.251024+0000","last_active":"2024-09-10T19:03:44.061276+0000","last_peered":"2024-09-10T19:03:44.061276+0000","last_clean":"2024-09-10T19:03:44.061276+0000","last_became_active":"2024-09-10T18:54:56.789695+0000","last_became_peered":"2024-09-10T18:54:56.789695+0000","last_unstale":"2024-09-10T19:03:44.061276+0000","last_undegraded":"2024-09-10T19:03:44.061276+0000","last_fullsized":"2024-09-10T19:03:44.061276+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'34","last_scrub_stamp":"2024-09-10T19:01:51.122251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.122251+0000","objects_scrubbed":2,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041375079999999998,"objects_trimmed":0,"snaptrim_duration":0.0013146880000000001,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.19","version":"454'47","reported_seq":596,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063269+0000","last_change":"2024-09-10T19:03:22.252561+0000","last_active":"2024-09-10T19:03:44.063269+0000","last_peered":"2024-09-10T19:03:44.063269+0000","last_clean":"2024-09-10T19:03:44.063269+0000","last_became_active":"2024-09-10T18:54:56.869422+0000","last_became_peered":"2024-09-10T18:54:56.869422+0000","last_unstale":"2024-09-10T19:03:44.063269+0000","last_undegraded":"2024-09-10T19:03:44.063269+0000","last_fullsized":"2024-09-10T19:03:44.063269+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'45","last_scrub_stamp":"2024-09-10T19:00:30.901206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:30.901206+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053195880000000001,"objects_trimmed":0,"snaptrim_duration":0.0015825800000000001,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.18","version":"533'128","reported_seq":657,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144324+0000","last_change":"2024-09-10T19:03:22.255455+0000","last_active":"2024-09-10T19:03:44.144324+0000","last_peered":"2024-09-10T19:03:44.144324+0000","last_clean":"2024-09-10T19:03:44.144324+0000","last_became_active":"2024-09-10T18:54:56.864984+0000","last_became_peered":"2024-09-10T18:54:56.864984+0000","last_unstale":"2024-09-10T19:03:44.144324+0000","last_undegraded":"2024-09-10T19:03:44.144324+0000","last_fullsized":"2024-09-10T19:03:44.144324+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'122","last_scrub_stamp":"2024-09-10T19:02:33.461826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:33.461826+0000","objects_scrubbed":9,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:02.352012+0000","scrub_duration":0.0058390489999999998,"objects_trimmed":0,"snaptrim_duration":0.00054257299999999997,"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":4,"num_write":47,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.17","version":"542'34","reported_seq":584,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105243+0000","last_change":"2024-09-10T19:03:22.354562+0000","last_active":"2024-09-10T19:03:44.105243+0000","last_peered":"2024-09-10T19:03:44.105243+0000","last_clean":"2024-09-10T19:03:44.105243+0000","last_became_active":"2024-09-10T18:54:56.821308+0000","last_became_peered":"2024-09-10T18:54:56.821308+0000","last_unstale":"2024-09-10T19:03:44.105243+0000","last_undegraded":"2024-09-10T19:03:44.105243+0000","last_fullsized":"2024-09-10T19:03:44.105243+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'32","last_scrub_stamp":"2024-09-10T19:02:39.859112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:39.859112+0000","objects_scrubbed":5,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:46.643334+0000","scrub_duration":0.0049073909999999997,"objects_trimmed":0,"snaptrim_duration":0.0011889559999999999,"stat_sum":{"num_bytes":3478,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.16","version":"542'25","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151903+0000","last_change":"2024-09-10T19:03:22.252315+0000","last_active":"2024-09-10T19:03:44.151903+0000","last_peered":"2024-09-10T19:03:44.151903+0000","last_clean":"2024-09-10T19:03:44.151903+0000","last_became_active":"2024-09-10T18:54:56.826749+0000","last_became_peered":"2024-09-10T18:54:56.826749+0000","last_unstale":"2024-09-10T19:03:44.151903+0000","last_undegraded":"2024-09-10T19:03:44.151903+0000","last_fullsized":"2024-09-10T19:03:44.151903+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"387'24","last_scrub_stamp":"2024-09-10T19:02:38.991395+0000","last_deep_scrub":"387'24","last_deep_scrub_stamp":"2024-09-10T18:59:03.724817+0000","last_clean_scrub_stamp":"2024-09-10T19:02:38.991395+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:54.524961+0000","scrub_duration":0.0043679799999999996,"objects_trimmed":0,"snaptrim_duration":0.0044208290000000003,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.15","version":"533'80","reported_seq":631,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148721+0000","last_change":"2024-09-10T19:03:22.355554+0000","last_active":"2024-09-10T19:03:44.148721+0000","last_peered":"2024-09-10T19:03:44.148721+0000","last_clean":"2024-09-10T19:03:44.148721+0000","last_became_active":"2024-09-10T18:54:56.892497+0000","last_became_peered":"2024-09-10T18:54:56.892497+0000","last_unstale":"2024-09-10T19:03:44.148721+0000","last_undegraded":"2024-09-10T19:03:44.148721+0000","last_fullsized":"2024-09-10T19:03:44.148721+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'79","last_scrub_stamp":"2024-09-10T19:01:43.224593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.224593+0000","objects_scrubbed":8,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059262669999999998,"objects_trimmed":0,"snaptrim_duration":0.00066040700000000001,"stat_sum":{"num_bytes":1285,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.14","version":"533'102","reported_seq":650,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150528+0000","last_change":"2024-09-10T19:03:22.252360+0000","last_active":"2024-09-10T19:03:44.150528+0000","last_peered":"2024-09-10T19:03:44.150528+0000","last_clean":"2024-09-10T19:03:44.150528+0000","last_became_active":"2024-09-10T18:54:56.884115+0000","last_became_peered":"2024-09-10T18:54:56.884115+0000","last_unstale":"2024-09-10T19:03:44.150528+0000","last_undegraded":"2024-09-10T19:03:44.150528+0000","last_fullsized":"2024-09-10T19:03:44.150528+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'100","last_scrub_stamp":"2024-09-10T19:01:48.758338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:48.758338+0000","objects_scrubbed":9,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080646439999999993,"objects_trimmed":0,"snaptrim_duration":0.0045734970000000002,"stat_sum":{"num_bytes":86,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":36,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.13","version":"582'31","reported_seq":600,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-09-10T19:03:45.300052+0000","last_change":"2024-09-10T19:03:22.353758+0000","last_active":"2024-09-10T19:03:45.300052+0000","last_peered":"2024-09-10T19:03:45.300052+0000","last_clean":"2024-09-10T19:03:45.300052+0000","last_became_active":"2024-09-10T18:54:56.862640+0000","last_became_peered":"2024-09-10T18:54:56.862640+0000","last_unstale":"2024-09-10T19:03:45.300052+0000","last_undegraded":"2024-09-10T19:03:45.300052+0000","last_fullsized":"2024-09-10T19:03:45.300052+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'21","last_scrub_stamp":"2024-09-10T19:01:51.163574+0000","last_deep_scrub":"448'4","last_deep_scrub_stamp":"2024-09-10T19:00:15.186496+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.163574+0000","objects_scrubbed":5,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056664030000000004,"objects_trimmed":0,"snaptrim_duration":0.001059656,"stat_sum":{"num_bytes":583,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":20,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.12","version":"248'51","reported_seq":587,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144859+0000","last_change":"2024-09-10T19:03:22.256694+0000","last_active":"2024-09-10T19:03:44.144859+0000","last_peered":"2024-09-10T19:03:44.144859+0000","last_clean":"2024-09-10T19:03:44.144859+0000","last_became_active":"2024-09-10T18:54:56.870161+0000","last_became_peered":"2024-09-10T18:54:56.870161+0000","last_unstale":"2024-09-10T19:03:44.144859+0000","last_undegraded":"2024-09-10T19:03:44.144859+0000","last_fullsized":"2024-09-10T19:03:44.144859+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-09-10T19:02:51.693511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:51.693511+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:07.220084+0000","scrub_duration":0.0042330099999999997,"objects_trimmed":0,"snaptrim_duration":0.001097055,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.11","version":"533'57","reported_seq":600,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144471+0000","last_change":"2024-09-10T19:03:22.255806+0000","last_active":"2024-09-10T19:03:44.144471+0000","last_peered":"2024-09-10T19:03:44.144471+0000","last_clean":"2024-09-10T19:03:44.144471+0000","last_became_active":"2024-09-10T19:02:52.255441+0000","last_became_peered":"2024-09-10T19:02:52.255441+0000","last_unstale":"2024-09-10T19:03:44.144471+0000","last_undegraded":"2024-09-10T19:03:44.144471+0000","last_fullsized":"2024-09-10T19:03:44.144471+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'57","last_scrub_stamp":"2024-09-10T19:02:54.769809+0000","last_deep_scrub":"452'56","last_deep_scrub_stamp":"2024-09-10T19:00:25.174126+0000","last_clean_scrub_stamp":"2024-09-10T19:02:54.769809+0000","objects_scrubbed":5,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:12.675191+0000","scrub_duration":0.0048406200000000003,"objects_trimmed":0,"snaptrim_duration":0.00056593300000000002,"stat_sum":{"num_bytes":608,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.10","version":"377'35","reported_seq":572,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062280+0000","last_change":"2024-09-10T19:03:22.248313+0000","last_active":"2024-09-10T19:03:44.062280+0000","last_peered":"2024-09-10T19:03:44.062280+0000","last_clean":"2024-09-10T19:03:44.062280+0000","last_became_active":"2024-09-10T18:54:56.902007+0000","last_became_peered":"2024-09-10T18:54:56.902007+0000","last_unstale":"2024-09-10T19:03:44.062280+0000","last_undegraded":"2024-09-10T19:03:44.062280+0000","last_fullsized":"2024-09-10T19:03:44.062280+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'35","last_scrub_stamp":"2024-09-10T19:01:33.818552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.818552+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033660140000000001,"objects_trimmed":0,"snaptrim_duration":0.00095859999999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.f","version":"524'74","reported_seq":618,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147605+0000","last_change":"2024-09-10T19:03:22.257848+0000","last_active":"2024-09-10T19:03:44.147605+0000","last_peered":"2024-09-10T19:03:44.147605+0000","last_clean":"2024-09-10T19:03:44.147605+0000","last_became_active":"2024-09-10T18:54:56.827690+0000","last_became_peered":"2024-09-10T18:54:56.827690+0000","last_unstale":"2024-09-10T19:03:44.147605+0000","last_undegraded":"2024-09-10T19:03:44.147605+0000","last_fullsized":"2024-09-10T19:03:44.147605+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'70","last_scrub_stamp":"2024-09-10T19:01:52.606042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.606042+0000","objects_scrubbed":7,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053694540000000001,"objects_trimmed":0,"snaptrim_duration":0.000364557,"stat_sum":{"num_bytes":64,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":33,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.e","version":"542'137","reported_seq":667,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105388+0000","last_change":"2024-09-10T19:03:22.353816+0000","last_active":"2024-09-10T19:03:44.105388+0000","last_peered":"2024-09-10T19:03:44.105388+0000","last_clean":"2024-09-10T19:03:44.105388+0000","last_became_active":"2024-09-10T18:54:56.820648+0000","last_became_peered":"2024-09-10T18:54:56.820648+0000","last_unstale":"2024-09-10T19:03:44.105388+0000","last_undegraded":"2024-09-10T19:03:44.105388+0000","last_fullsized":"2024-09-10T19:03:44.105388+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'136","last_scrub_stamp":"2024-09-10T19:01:47.877294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.877294+0000","objects_scrubbed":9,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054641070000000002,"objects_trimmed":0,"snaptrim_duration":0.00081802700000000003,"stat_sum":{"num_bytes":1434,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":45,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.d","version":"491'34","reported_seq":531,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145596+0000","last_change":"2024-09-10T19:03:22.257919+0000","last_active":"2024-09-10T19:03:44.145596+0000","last_peered":"2024-09-10T19:03:44.145596+0000","last_clean":"2024-09-10T19:03:44.145596+0000","last_became_active":"2024-09-10T19:02:52.256207+0000","last_became_peered":"2024-09-10T19:02:52.256207+0000","last_unstale":"2024-09-10T19:03:44.145596+0000","last_undegraded":"2024-09-10T19:03:44.145596+0000","last_fullsized":"2024-09-10T19:03:44.145596+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'34","last_scrub_stamp":"2024-09-10T19:02:35.257651+0000","last_deep_scrub":"243'18","last_deep_scrub_stamp":"2024-09-10T18:56:39.950183+0000","last_clean_scrub_stamp":"2024-09-10T19:02:35.257651+0000","objects_scrubbed":3,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00460977,"objects_trimmed":0,"snaptrim_duration":0.001946962,"stat_sum":{"num_bytes":65,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.c","version":"514'26","reported_seq":577,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.909459+0000","last_change":"2024-09-10T19:03:44.909376+0000","last_active":"2024-09-10T19:03:44.909459+0000","last_peered":"2024-09-10T19:03:44.909459+0000","last_clean":"2024-09-10T19:03:44.909459+0000","last_became_active":"2024-09-10T18:54:56.837685+0000","last_became_peered":"2024-09-10T18:54:56.837685+0000","last_unstale":"2024-09-10T19:03:44.909459+0000","last_undegraded":"2024-09-10T19:03:44.909459+0000","last_fullsized":"2024-09-10T19:03:44.909459+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'26","last_scrub_stamp":"2024-09-10T19:03:44.909223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:03:44.909223+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:12.115694+0000","scrub_duration":0.0040749940000000002,"objects_trimmed":0,"snaptrim_duration":0.0016513470000000001,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.b","version":"524'48","reported_seq":586,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150557+0000","last_change":"2024-09-10T19:03:22.251945+0000","last_active":"2024-09-10T19:03:44.150557+0000","last_peered":"2024-09-10T19:03:44.150557+0000","last_clean":"2024-09-10T19:03:44.150557+0000","last_became_active":"2024-09-10T18:54:56.850888+0000","last_became_peered":"2024-09-10T18:54:56.850888+0000","last_unstale":"2024-09-10T19:03:44.150557+0000","last_undegraded":"2024-09-10T19:03:44.150557+0000","last_fullsized":"2024-09-10T19:03:44.150557+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'36","last_scrub_stamp":"2024-09-10T19:00:28.730904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:00:28.730904+0000","objects_scrubbed":4,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045585440000000003,"objects_trimmed":0,"snaptrim_duration":0.0047199490000000002,"stat_sum":{"num_bytes":4000030,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":6,"num_write":30,"num_write_kb":3927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.a","version":"523'22","reported_seq":585,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147831+0000","last_change":"2024-09-10T19:03:22.354110+0000","last_active":"2024-09-10T19:03:44.147831+0000","last_peered":"2024-09-10T19:03:44.147831+0000","last_clean":"2024-09-10T19:03:44.147831+0000","last_became_active":"2024-09-10T18:54:56.844640+0000","last_became_peered":"2024-09-10T18:54:56.844640+0000","last_unstale":"2024-09-10T19:03:44.147831+0000","last_undegraded":"2024-09-10T19:03:44.147831+0000","last_fullsized":"2024-09-10T19:03:44.147831+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'20","last_scrub_stamp":"2024-09-10T19:01:47.148028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.148028+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041924129999999999,"objects_trimmed":0,"snaptrim_duration":0.0015502409999999999,"stat_sum":{"num_bytes":11,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.9","version":"551'11","reported_seq":575,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147773+0000","last_change":"2024-09-10T19:03:22.354407+0000","last_active":"2024-09-10T19:03:44.147773+0000","last_peered":"2024-09-10T19:03:44.147773+0000","last_clean":"2024-09-10T19:03:44.147773+0000","last_became_active":"2024-09-10T18:54:56.882782+0000","last_became_peered":"2024-09-10T18:54:56.882782+0000","last_unstale":"2024-09-10T19:03:44.147773+0000","last_undegraded":"2024-09-10T19:03:44.147773+0000","last_fullsized":"2024-09-10T19:03:44.147773+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"524'8","last_scrub_stamp":"2024-09-10T19:02:37.412067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:37.412067+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:03.667077+0000","scrub_duration":0.0038186159999999999,"objects_trimmed":0,"snaptrim_duration":0.001503398,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":15,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.8","version":"523'63","reported_seq":610,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.101503+0000","last_change":"2024-09-10T19:03:22.349721+0000","last_active":"2024-09-10T19:03:44.101503+0000","last_peered":"2024-09-10T19:03:44.101503+0000","last_clean":"2024-09-10T19:03:44.101503+0000","last_became_active":"2024-09-10T18:54:56.815105+0000","last_became_peered":"2024-09-10T18:54:56.815105+0000","last_unstale":"2024-09-10T19:03:44.101503+0000","last_undegraded":"2024-09-10T19:03:44.101503+0000","last_fullsized":"2024-09-10T19:03:44.101503+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"523'63","last_scrub_stamp":"2024-09-10T19:02:34.875773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:34.875773+0000","objects_scrubbed":5,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:46.358231+0000","scrub_duration":0.0069491049999999997,"objects_trimmed":0,"snaptrim_duration":0.00089944899999999997,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":25,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.7","version":"542'163","reported_seq":725,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151000+0000","last_change":"2024-09-10T19:03:36.628335+0000","last_active":"2024-09-10T19:03:44.151000+0000","last_peered":"2024-09-10T19:03:44.151000+0000","last_clean":"2024-09-10T19:03:44.151000+0000","last_became_active":"2024-09-10T18:54:56.873143+0000","last_became_peered":"2024-09-10T18:54:56.873143+0000","last_unstale":"2024-09-10T19:03:44.151000+0000","last_undegraded":"2024-09-10T19:03:44.151000+0000","last_fullsized":"2024-09-10T19:03:44.151000+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"542'163","last_scrub_stamp":"2024-09-10T19:03:36.628130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:03:36.628130+0000","objects_scrubbed":12,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:54.786938+0000","scrub_duration":0.0067295009999999997,"objects_trimmed":0,"snaptrim_duration":0.00059419400000000004,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":15,"num_read_kb":3,"num_write":52,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.6","version":"533'100","reported_seq":692,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062570+0000","last_change":"2024-09-10T19:03:22.252746+0000","last_active":"2024-09-10T19:03:44.062570+0000","last_peered":"2024-09-10T19:03:44.062570+0000","last_clean":"2024-09-10T19:03:44.062570+0000","last_became_active":"2024-09-10T18:54:56.877222+0000","last_became_peered":"2024-09-10T18:54:56.877222+0000","last_unstale":"2024-09-10T19:03:44.062570+0000","last_undegraded":"2024-09-10T19:03:44.062570+0000","last_fullsized":"2024-09-10T19:03:44.062570+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-10T19:01:28.217457+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-10T18:59:04.213299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.217457+0000","objects_scrubbed":6,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059368959999999997,"objects_trimmed":0,"snaptrim_duration":0.0025182770000000002,"stat_sum":{"num_bytes":4196669,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":30,"num_write_kb":8221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.5","version":"305'53","reported_seq":597,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146496+0000","last_change":"2024-09-10T19:03:22.258137+0000","last_active":"2024-09-10T19:03:44.146496+0000","last_peered":"2024-09-10T19:03:44.146496+0000","last_clean":"2024-09-10T19:03:44.146496+0000","last_became_active":"2024-09-10T18:54:56.796474+0000","last_became_peered":"2024-09-10T18:54:56.796474+0000","last_unstale":"2024-09-10T19:03:44.146496+0000","last_undegraded":"2024-09-10T19:03:44.146496+0000","last_fullsized":"2024-09-10T19:03:44.146496+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'53","last_scrub_stamp":"2024-09-10T19:02:35.530044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:35.530044+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:58.237409+0000","scrub_duration":0.004296762,"objects_trimmed":0,"snaptrim_duration":0.00033404600000000001,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.4","version":"523'115","reported_seq":654,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061048+0000","last_change":"2024-09-10T19:03:22.252677+0000","last_active":"2024-09-10T19:03:44.061048+0000","last_peered":"2024-09-10T19:03:44.061048+0000","last_clean":"2024-09-10T19:03:44.061048+0000","last_became_active":"2024-09-10T18:54:56.831483+0000","last_became_peered":"2024-09-10T18:54:56.831483+0000","last_unstale":"2024-09-10T19:03:44.061048+0000","last_undegraded":"2024-09-10T19:03:44.061048+0000","last_fullsized":"2024-09-10T19:03:44.061048+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'113","last_scrub_stamp":"2024-09-10T19:01:53.156959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.156959+0000","objects_scrubbed":9,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063436309999999997,"objects_trimmed":0,"snaptrim_duration":0.0016984459999999999,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":37,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.3","version":"523'108","reported_seq":640,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066846+0000","last_change":"2024-09-10T19:03:22.255229+0000","last_active":"2024-09-10T19:03:44.066846+0000","last_peered":"2024-09-10T19:03:44.066846+0000","last_clean":"2024-09-10T19:03:44.066846+0000","last_became_active":"2024-09-10T18:54:56.870166+0000","last_became_peered":"2024-09-10T18:54:56.870166+0000","last_unstale":"2024-09-10T19:03:44.066846+0000","last_undegraded":"2024-09-10T19:03:44.066846+0000","last_fullsized":"2024-09-10T19:03:44.066846+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'106","last_scrub_stamp":"2024-09-10T19:01:28.916641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:28.916641+0000","objects_scrubbed":9,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052486110000000002,"objects_trimmed":0,"snaptrim_duration":0.00056547800000000001,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":37,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"64.2","version":"230'34","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.102972+0000","last_change":"2024-09-10T19:03:22.351307+0000","last_active":"2024-09-10T19:03:44.102972+0000","last_peered":"2024-09-10T19:03:44.102972+0000","last_clean":"2024-09-10T19:03:44.102972+0000","last_became_active":"2024-09-10T18:54:56.837832+0000","last_became_peered":"2024-09-10T18:54:56.837832+0000","last_unstale":"2024-09-10T19:03:44.102972+0000","last_undegraded":"2024-09-10T19:03:44.102972+0000","last_fullsized":"2024-09-10T19:03:44.102972+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'34","last_scrub_stamp":"2024-09-10T19:02:42.794731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:42.794731+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:59.915687+0000","scrub_duration":0.0041933780000000002,"objects_trimmed":0,"snaptrim_duration":0.001159967,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1","version":"533'77","reported_seq":627,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145130+0000","last_change":"2024-09-10T19:03:22.256618+0000","last_active":"2024-09-10T19:03:44.145130+0000","last_peered":"2024-09-10T19:03:44.145130+0000","last_clean":"2024-09-10T19:03:44.145130+0000","last_became_active":"2024-09-10T18:54:56.799579+0000","last_became_peered":"2024-09-10T18:54:56.799579+0000","last_unstale":"2024-09-10T19:03:44.145130+0000","last_undegraded":"2024-09-10T19:03:44.145130+0000","last_fullsized":"2024-09-10T19:03:44.145130+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'77","last_scrub_stamp":"2024-09-10T19:02:40.587148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:02:40.587148+0000","objects_scrubbed":10,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:01.116184+0000","scrub_duration":0.0057731290000000001,"objects_trimmed":0,"snaptrim_duration":0.00099837100000000007,"stat_sum":{"num_bytes":1296,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.0","version":"533'105","reported_seq":649,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146909+0000","last_change":"2024-09-10T19:03:22.353156+0000","last_active":"2024-09-10T19:03:44.146909+0000","last_peered":"2024-09-10T19:03:44.146909+0000","last_clean":"2024-09-10T19:03:44.146909+0000","last_became_active":"2024-09-10T18:54:56.882651+0000","last_became_peered":"2024-09-10T18:54:56.882651+0000","last_unstale":"2024-09-10T19:03:44.146909+0000","last_undegraded":"2024-09-10T19:03:44.146909+0000","last_fullsized":"2024-09-10T19:03:44.146909+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":142,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":0,"last_scrub":"246'102","last_scrub_stamp":"2024-09-10T19:01:55.215341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:55.359884+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.215341+0000","objects_scrubbed":6,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029714390000000002,"objects_trimmed":0,"snaptrim_duration":0.0010685619999999999,"stat_sum":{"num_bytes":766,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":35,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":565,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066489+0000","last_change":"2024-09-10T19:02:44.800707+0000","last_active":"2024-09-10T19:03:44.066489+0000","last_peered":"2024-09-10T19:03:44.066489+0000","last_clean":"2024-09-10T19:03:44.066489+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:03:44.066489+0000","last_undegraded":"2024-09-10T19:03:44.066489+0000","last_fullsized":"2024-09-10T19:03:44.066489+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:44.800558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:56.774029+0000","last_clean_scrub_stamp":"2024-09-10T19:02:44.800558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:11.624406+0000","scrub_duration":0.0072805439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.065274+0000","last_change":"2024-09-10T19:02:36.534303+0000","last_active":"2024-09-10T19:03:44.065274+0000","last_peered":"2024-09-10T19:03:44.065274+0000","last_clean":"2024-09-10T19:03:44.065274+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:03:44.065274+0000","last_undegraded":"2024-09-10T19:03:44.065274+0000","last_fullsized":"2024-09-10T19:03:44.065274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:36.534120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:36.534120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:58.466335+0000","scrub_duration":0.0034190319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":599,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067843+0000","last_change":"2024-09-10T19:03:40.092114+0000","last_active":"2024-09-10T19:03:44.067843+0000","last_peered":"2024-09-10T19:03:44.067843+0000","last_clean":"2024-09-10T19:03:44.067843+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:03:44.067843+0000","last_undegraded":"2024-09-10T19:03:44.067843+0000","last_fullsized":"2024-09-10T19:03:44.067843+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:03:40.091947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:03:40.091947+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:07.532307+0000","scrub_duration":0.005285634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067113+0000","last_change":"2024-09-10T19:01:52.890306+0000","last_active":"2024-09-10T19:03:44.067113+0000","last_peered":"2024-09-10T19:03:44.067113+0000","last_clean":"2024-09-10T19:03:44.067113+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:03:44.067113+0000","last_undegraded":"2024-09-10T19:03:44.067113+0000","last_fullsized":"2024-09-10T19:03:44.067113+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.890190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017323970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":490,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144653+0000","last_change":"2024-09-10T19:01:39.509250+0000","last_active":"2024-09-10T19:03:44.144653+0000","last_peered":"2024-09-10T19:03:44.144653+0000","last_clean":"2024-09-10T19:03:44.144653+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:03:44.144653+0000","last_undegraded":"2024-09-10T19:03:44.144653+0000","last_fullsized":"2024-09-10T19:03:44.144653+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.509093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.509093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035638950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.068668+0000","last_change":"2024-09-10T19:01:29.910705+0000","last_active":"2024-09-10T19:03:44.068668+0000","last_peered":"2024-09-10T19:03:44.068668+0000","last_clean":"2024-09-10T19:03:44.068668+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:03:44.068668+0000","last_undegraded":"2024-09-10T19:03:44.068668+0000","last_fullsized":"2024-09-10T19:03:44.068668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.910553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.910553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032680439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":606,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.065057+0000","last_change":"2024-09-10T19:02:43.672359+0000","last_active":"2024-09-10T19:03:44.065057+0000","last_peered":"2024-09-10T19:03:44.065057+0000","last_clean":"2024-09-10T19:03:44.065057+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:03:44.065057+0000","last_undegraded":"2024-09-10T19:03:44.065057+0000","last_fullsized":"2024-09-10T19:03:44.065057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:02:43.672172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:43.672172+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:08.603284+0000","scrub_duration":0.0044597270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":586,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144883+0000","last_change":"2024-09-10T19:02:55.392846+0000","last_active":"2024-09-10T19:03:44.144883+0000","last_peered":"2024-09-10T19:03:44.144883+0000","last_clean":"2024-09-10T19:03:44.144883+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:03:44.144883+0000","last_undegraded":"2024-09-10T19:03:44.144883+0000","last_fullsized":"2024-09-10T19:03:44.144883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:55.392695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:55.392695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:08.458578+0000","scrub_duration":0.0035130090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":57,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145458+0000","last_change":"2024-09-10T19:03:43.920259+0000","last_active":"2024-09-10T19:03:44.145458+0000","last_peered":"2024-09-10T19:03:44.145458+0000","last_clean":"2024-09-10T19:03:44.145458+0000","last_became_active":"2024-09-10T19:02:52.254895+0000","last_became_peered":"2024-09-10T19:02:52.254895+0000","last_unstale":"2024-09-10T19:03:44.145458+0000","last_undegraded":"2024-09-10T19:03:44.145458+0000","last_fullsized":"2024-09-10T19:03:44.145458+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:43.920102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:03:43.920102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:02.218731+0000","scrub_duration":0.0030485299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":482,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063307+0000","last_change":"2024-09-10T19:01:15.552219+0000","last_active":"2024-09-10T19:03:44.063307+0000","last_peered":"2024-09-10T19:03:44.063307+0000","last_clean":"2024-09-10T19:03:44.063307+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:03:44.063307+0000","last_undegraded":"2024-09-10T19:03:44.063307+0000","last_fullsized":"2024-09-10T19:03:44.063307+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.552108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.552108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079911449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":587,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.108943+0000","last_change":"2024-09-10T19:03:44.108853+0000","last_active":"2024-09-10T19:03:44.108943+0000","last_peered":"2024-09-10T19:03:44.108943+0000","last_clean":"2024-09-10T19:03:44.108943+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:03:44.108943+0000","last_undegraded":"2024-09-10T19:03:44.108943+0000","last_fullsized":"2024-09-10T19:03:44.108943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:44.108766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:03:44.108766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:03.495310+0000","scrub_duration":0.010945592000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":621,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145916+0000","last_change":"2024-09-10T19:01:40.519369+0000","last_active":"2024-09-10T19:03:44.145916+0000","last_peered":"2024-09-10T19:03:44.145916+0000","last_clean":"2024-09-10T19:03:44.145916+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:03:44.145916+0000","last_undegraded":"2024-09-10T19:03:44.145916+0000","last_fullsized":"2024-09-10T19:03:44.145916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:01:40.519200+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.519200+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010009456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066697+0000","last_change":"2024-09-10T19:02:52.252521+0000","last_active":"2024-09-10T19:03:44.066697+0000","last_peered":"2024-09-10T19:03:44.066697+0000","last_clean":"2024-09-10T19:03:44.066697+0000","last_became_active":"2024-09-10T19:02:52.250472+0000","last_became_peered":"2024-09-10T19:02:52.250472+0000","last_unstale":"2024-09-10T19:03:44.066697+0000","last_undegraded":"2024-09-10T19:03:44.066697+0000","last_fullsized":"2024-09-10T19:03:44.066697+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:34.223516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:01:34.223516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":570,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144883+0000","last_change":"2024-09-10T19:01:55.606548+0000","last_active":"2024-09-10T19:03:44.144883+0000","last_peered":"2024-09-10T19:03:44.144883+0000","last_clean":"2024-09-10T19:03:44.144883+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:03:44.144883+0000","last_undegraded":"2024-09-10T19:03:44.144883+0000","last_fullsized":"2024-09-10T19:03:44.144883+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.606429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.606429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002983313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":494,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150804+0000","last_change":"2024-09-10T19:01:30.825681+0000","last_active":"2024-09-10T19:03:44.150804+0000","last_peered":"2024-09-10T19:03:44.150804+0000","last_clean":"2024-09-10T19:03:44.150804+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:03:44.150804+0000","last_undegraded":"2024-09-10T19:03:44.150804+0000","last_fullsized":"2024-09-10T19:03:44.150804+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.825536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.825536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034703239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":580,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061795+0000","last_change":"2024-09-10T19:01:33.189562+0000","last_active":"2024-09-10T19:03:44.061795+0000","last_peered":"2024-09-10T19:03:44.061795+0000","last_clean":"2024-09-10T19:03:44.061795+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:03:44.061795+0000","last_undegraded":"2024-09-10T19:03:44.061795+0000","last_fullsized":"2024-09-10T19:03:44.061795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:33.189399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:33.189399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037148820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":628,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146169+0000","last_change":"2024-09-10T19:03:34.945847+0000","last_active":"2024-09-10T19:03:44.146169+0000","last_peered":"2024-09-10T19:03:44.146169+0000","last_clean":"2024-09-10T19:03:44.146169+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:03:44.146169+0000","last_undegraded":"2024-09-10T19:03:44.146169+0000","last_fullsized":"2024-09-10T19:03:44.146169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:03:34.945682+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:03:34.945682+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:03.589904+0000","scrub_duration":0.0051108630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062760+0000","last_change":"2024-09-10T19:02:52.257264+0000","last_active":"2024-09-10T19:03:44.062760+0000","last_peered":"2024-09-10T19:03:44.062760+0000","last_clean":"2024-09-10T19:03:44.062760+0000","last_became_active":"2024-09-10T19:02:52.256764+0000","last_became_peered":"2024-09-10T19:02:52.256764+0000","last_unstale":"2024-09-10T19:03:44.062760+0000","last_undegraded":"2024-09-10T19:03:44.062760+0000","last_fullsized":"2024-09-10T19:03:44.062760+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:35.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:00:35.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":571,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061891+0000","last_change":"2024-09-10T19:01:38.581158+0000","last_active":"2024-09-10T19:03:44.061891+0000","last_peered":"2024-09-10T19:03:44.061891+0000","last_clean":"2024-09-10T19:03:44.061891+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:03:44.061891+0000","last_undegraded":"2024-09-10T19:03:44.061891+0000","last_fullsized":"2024-09-10T19:03:44.061891+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:38.580997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:38.580997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033368260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":489,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062676+0000","last_change":"2024-09-10T19:03:33.157568+0000","last_active":"2024-09-10T19:03:44.062676+0000","last_peered":"2024-09-10T19:03:44.062676+0000","last_clean":"2024-09-10T19:03:44.062676+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:03:44.062676+0000","last_undegraded":"2024-09-10T19:03:44.062676+0000","last_fullsized":"2024-09-10T19:03:44.062676+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:14.868377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:00:14.868377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036007919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":97,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144306+0000","last_change":"2024-09-10T19:02:44.684150+0000","last_active":"2024-09-10T19:03:44.144306+0000","last_peered":"2024-09-10T19:03:44.144306+0000","last_clean":"2024-09-10T19:03:44.144306+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:03:44.144306+0000","last_undegraded":"2024-09-10T19:03:44.144306+0000","last_fullsized":"2024-09-10T19:03:44.144306+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:44.683992+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:44.683992+0000","last_clean_scrub_stamp":"2024-09-10T19:02:44.683992+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:59.198641+0000","scrub_duration":0.0048470179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":639,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.103503+0000","last_change":"2024-09-10T19:02:40.828497+0000","last_active":"2024-09-10T19:03:44.103503+0000","last_peered":"2024-09-10T19:03:44.103503+0000","last_clean":"2024-09-10T19:03:44.103503+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:03:44.103503+0000","last_undegraded":"2024-09-10T19:03:44.103503+0000","last_fullsized":"2024-09-10T19:03:44.103503+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:02:40.828317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:40.828317+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:51.929050+0000","scrub_duration":0.0052059460000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062581+0000","last_change":"2024-09-10T19:01:35.169404+0000","last_active":"2024-09-10T19:03:44.062581+0000","last_peered":"2024-09-10T19:03:44.062581+0000","last_clean":"2024-09-10T19:03:44.062581+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:03:44.062581+0000","last_undegraded":"2024-09-10T19:03:44.062581+0000","last_fullsized":"2024-09-10T19:03:44.062581+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.169217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.169217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032414449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"548'13","reported_seq":133,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.149476+0000","last_change":"2024-09-10T19:03:40.624271+0000","last_active":"2024-09-10T19:03:44.149476+0000","last_peered":"2024-09-10T19:03:44.149476+0000","last_clean":"2024-09-10T19:03:44.149476+0000","last_became_active":"2024-09-10T19:02:52.256843+0000","last_became_peered":"2024-09-10T19:02:52.256843+0000","last_unstale":"2024-09-10T19:03:44.149476+0000","last_undegraded":"2024-09-10T19:03:44.149476+0000","last_fullsized":"2024-09-10T19:03:44.149476+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:03:40.624111+0000","last_deep_scrub":"97'12","last_deep_scrub_stamp":"2024-09-10T18:52:17.252498+0000","last_clean_scrub_stamp":"2024-09-10T19:03:40.624111+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:06.779728+0000","scrub_duration":0.00383418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":591,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151293+0000","last_change":"2024-09-10T19:02:39.443321+0000","last_active":"2024-09-10T19:03:44.151293+0000","last_peered":"2024-09-10T19:03:44.151293+0000","last_clean":"2024-09-10T19:03:44.151293+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:03:44.151293+0000","last_undegraded":"2024-09-10T19:03:44.151293+0000","last_fullsized":"2024-09-10T19:03:44.151293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:39.443181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:02:39.443181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033941349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":493,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.104737+0000","last_change":"2024-09-10T19:01:53.810938+0000","last_active":"2024-09-10T19:03:44.104737+0000","last_peered":"2024-09-10T19:03:44.104737+0000","last_clean":"2024-09-10T19:03:44.104737+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:03:44.104737+0000","last_undegraded":"2024-09-10T19:03:44.104737+0000","last_fullsized":"2024-09-10T19:03:44.104737+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:53.810787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.810787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003435802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":655,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.060213+0000","last_change":"2024-09-10T19:02:55.948787+0000","last_active":"2024-09-10T19:03:44.060213+0000","last_peered":"2024-09-10T19:03:44.060213+0000","last_clean":"2024-09-10T19:03:44.060213+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:03:44.060213+0000","last_undegraded":"2024-09-10T19:03:44.060213+0000","last_fullsized":"2024-09-10T19:03:44.060213+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:55.948623+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:55.948623+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:07.711954+0000","scrub_duration":0.003654577,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":608,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067255+0000","last_change":"2024-09-10T19:02:58.006429+0000","last_active":"2024-09-10T19:03:44.067255+0000","last_peered":"2024-09-10T19:03:44.067255+0000","last_clean":"2024-09-10T19:03:44.067255+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:03:44.067255+0000","last_undegraded":"2024-09-10T19:03:44.067255+0000","last_fullsized":"2024-09-10T19:03:44.067255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:02:58.006279+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:02:58.006279+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:00.387569+0000","scrub_duration":0.0043625670000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":588,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150694+0000","last_change":"2024-09-10T19:01:27.860664+0000","last_active":"2024-09-10T19:03:44.150694+0000","last_peered":"2024-09-10T19:03:44.150694+0000","last_clean":"2024-09-10T19:03:44.150694+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:03:44.150694+0000","last_undegraded":"2024-09-10T19:03:44.150694+0000","last_fullsized":"2024-09-10T19:03:44.150694+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:27.860545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.860545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031084049999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067235+0000","last_change":"2024-09-10T19:02:52.254699+0000","last_active":"2024-09-10T19:03:44.067235+0000","last_peered":"2024-09-10T19:03:44.067235+0000","last_clean":"2024-09-10T19:03:44.067235+0000","last_became_active":"2024-09-10T19:02:52.254392+0000","last_became_peered":"2024-09-10T19:02:52.254392+0000","last_unstale":"2024-09-10T19:03:44.067235+0000","last_undegraded":"2024-09-10T19:03:44.067235+0000","last_fullsized":"2024-09-10T19:03:44.067235+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.068817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.068817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152238+0000","last_change":"2024-09-10T19:02:52.258790+0000","last_active":"2024-09-10T19:03:44.152238+0000","last_peered":"2024-09-10T19:03:44.152238+0000","last_clean":"2024-09-10T19:03:44.152238+0000","last_became_active":"2024-09-10T19:02:52.258431+0000","last_became_peered":"2024-09-10T19:02:52.258431+0000","last_unstale":"2024-09-10T19:03:44.152238+0000","last_undegraded":"2024-09-10T19:03:44.152238+0000","last_fullsized":"2024-09-10T19:03:44.152238+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.078620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.078620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":493,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.103691+0000","last_change":"2024-09-10T19:02:31.892100+0000","last_active":"2024-09-10T19:03:44.103691+0000","last_peered":"2024-09-10T19:03:44.103691+0000","last_clean":"2024-09-10T19:03:44.103691+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:03:44.103691+0000","last_undegraded":"2024-09-10T19:03:44.103691+0000","last_fullsized":"2024-09-10T19:03:44.103691+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:31.891924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:02:31.891924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003931538,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":90,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144541+0000","last_change":"2024-09-10T19:02:50.724546+0000","last_active":"2024-09-10T19:03:44.144541+0000","last_peered":"2024-09-10T19:03:44.144541+0000","last_clean":"2024-09-10T19:03:44.144541+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:03:44.144541+0000","last_undegraded":"2024-09-10T19:03:44.144541+0000","last_fullsized":"2024-09-10T19:03:44.144541+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:02:50.724359+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:05.498297+0000","scrub_duration":0.0033033110000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":622,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-09-10T19:03:45.165678+0000","last_change":"2024-09-10T19:03:45.165625+0000","last_active":"2024-09-10T19:03:45.165678+0000","last_peered":"2024-09-10T19:03:45.165678+0000","last_clean":"2024-09-10T19:03:45.165678+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:03:45.165678+0000","last_undegraded":"2024-09-10T19:03:45.165678+0000","last_fullsized":"2024-09-10T19:03:45.165678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:03:45.165510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:03:45.165510+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:45.515273+0000","scrub_duration":0.0042102010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.149903+0000","last_change":"2024-09-10T19:01:24.883894+0000","last_active":"2024-09-10T19:03:44.149903+0000","last_peered":"2024-09-10T19:03:44.149903+0000","last_clean":"2024-09-10T19:03:44.149903+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:03:44.149903+0000","last_undegraded":"2024-09-10T19:03:44.149903+0000","last_fullsized":"2024-09-10T19:03:44.149903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:24.883733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:24.883733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033285659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":584,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152367+0000","last_change":"2024-09-10T19:01:37.874644+0000","last_active":"2024-09-10T19:03:44.152367+0000","last_peered":"2024-09-10T19:03:44.152367+0000","last_clean":"2024-09-10T19:03:44.152367+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:03:44.152367+0000","last_undegraded":"2024-09-10T19:03:44.152367+0000","last_fullsized":"2024-09-10T19:03:44.152367+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.874480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.874480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034122890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145737+0000","last_change":"2024-09-10T19:01:44.400980+0000","last_active":"2024-09-10T19:03:44.145737+0000","last_peered":"2024-09-10T19:03:44.145737+0000","last_clean":"2024-09-10T19:03:44.145737+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:03:44.145737+0000","last_undegraded":"2024-09-10T19:03:44.145737+0000","last_fullsized":"2024-09-10T19:03:44.145737+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.400822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.400822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00334009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":504,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152420+0000","last_change":"2024-09-10T19:03:32.999346+0000","last_active":"2024-09-10T19:03:44.152420+0000","last_peered":"2024-09-10T19:03:44.152420+0000","last_clean":"2024-09-10T19:03:44.152420+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:03:44.152420+0000","last_undegraded":"2024-09-10T19:03:44.152420+0000","last_fullsized":"2024-09-10T19:03:44.152420+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:32.999256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:32.999256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:54.862217+0000","scrub_duration":0.015712033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":625,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146834+0000","last_change":"2024-09-10T19:02:43.365539+0000","last_active":"2024-09-10T19:03:44.146834+0000","last_peered":"2024-09-10T19:03:44.146834+0000","last_clean":"2024-09-10T19:03:44.146834+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:03:44.146834+0000","last_undegraded":"2024-09-10T19:03:44.146834+0000","last_fullsized":"2024-09-10T19:03:44.146834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'23","last_scrub_stamp":"2024-09-10T19:02:43.365374+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T19:02:43.365374+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:09.735403+0000","scrub_duration":0.0046908729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":587,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.060329+0000","last_change":"2024-09-10T19:02:47.129748+0000","last_active":"2024-09-10T19:03:44.060329+0000","last_peered":"2024-09-10T19:03:44.060329+0000","last_clean":"2024-09-10T19:03:44.060329+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:03:44.060329+0000","last_undegraded":"2024-09-10T19:03:44.060329+0000","last_fullsized":"2024-09-10T19:03:44.060329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:02:47.129587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:00.754993+0000","scrub_duration":0.0034779490000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"548'13","reported_seq":125,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105973+0000","last_change":"2024-09-10T19:02:52.788518+0000","last_active":"2024-09-10T19:03:44.105973+0000","last_peered":"2024-09-10T19:03:44.105973+0000","last_clean":"2024-09-10T19:03:44.105973+0000","last_became_active":"2024-09-10T19:02:52.257016+0000","last_became_peered":"2024-09-10T19:02:52.257016+0000","last_unstale":"2024-09-10T19:03:44.105973+0000","last_undegraded":"2024-09-10T19:03:44.105973+0000","last_fullsized":"2024-09-10T19:03:44.105973+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:02:52.788351+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:02:52.788351+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:54.179723+0000","scrub_duration":0.004076199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":566,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067010+0000","last_change":"2024-09-10T19:00:22.885319+0000","last_active":"2024-09-10T19:03:44.067010+0000","last_peered":"2024-09-10T19:03:44.067010+0000","last_clean":"2024-09-10T19:03:44.067010+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:03:44.067010+0000","last_undegraded":"2024-09-10T19:03:44.067010+0000","last_fullsized":"2024-09-10T19:03:44.067010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:00:22.885146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:22.885146+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048287809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":494,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152048+0000","last_change":"2024-09-10T19:01:29.873888+0000","last_active":"2024-09-10T19:03:44.152048+0000","last_peered":"2024-09-10T19:03:44.152048+0000","last_clean":"2024-09-10T19:03:44.152048+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:03:44.152048+0000","last_undegraded":"2024-09-10T19:03:44.152048+0000","last_fullsized":"2024-09-10T19:03:44.152048+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:29.873730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:29.873730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034669100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":86,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066792+0000","last_change":"2024-09-10T19:02:03.625372+0000","last_active":"2024-09-10T19:03:44.066792+0000","last_peered":"2024-09-10T19:03:44.066792+0000","last_clean":"2024-09-10T19:03:44.066792+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:03:44.066792+0000","last_undegraded":"2024-09-10T19:03:44.066792+0000","last_fullsized":"2024-09-10T19:03:44.066792+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":611,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064519+0000","last_change":"2024-09-10T19:03:43.743847+0000","last_active":"2024-09-10T19:03:44.064519+0000","last_peered":"2024-09-10T19:03:44.064519+0000","last_clean":"2024-09-10T19:03:44.064519+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:03:44.064519+0000","last_undegraded":"2024-09-10T19:03:44.064519+0000","last_fullsized":"2024-09-10T19:03:44.064519+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:03:43.743707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:03:43.743707+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:50.643956+0000","scrub_duration":0.0043821509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":586,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.106425+0000","last_change":"2024-09-10T19:03:35.736339+0000","last_active":"2024-09-10T19:03:44.106425+0000","last_peered":"2024-09-10T19:03:44.106425+0000","last_clean":"2024-09-10T19:03:44.106425+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:03:44.106425+0000","last_undegraded":"2024-09-10T19:03:44.106425+0000","last_fullsized":"2024-09-10T19:03:44.106425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:35.736176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:03:35.736176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:56.234191+0000","scrub_duration":0.0033747270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145009+0000","last_change":"2024-09-10T19:02:52.257087+0000","last_active":"2024-09-10T19:03:44.145009+0000","last_peered":"2024-09-10T19:03:44.145009+0000","last_clean":"2024-09-10T19:03:44.145009+0000","last_became_active":"2024-09-10T19:02:52.254942+0000","last_became_peered":"2024-09-10T19:02:52.254942+0000","last_unstale":"2024-09-10T19:03:44.145009+0000","last_undegraded":"2024-09-10T19:03:44.145009+0000","last_fullsized":"2024-09-10T19:03:44.145009+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:46.342860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:02:46.342860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:05.591792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1427,"reported_epoch":582,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.562575+0000","last_change":"2024-09-10T19:01:26.001371+0000","last_active":"2024-09-10T19:03:44.562575+0000","last_peered":"2024-09-10T19:03:44.562575+0000","last_clean":"2024-09-10T19:03:44.562575+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:03:44.562575+0000","last_undegraded":"2024-09-10T19:03:44.562575+0000","last_fullsized":"2024-09-10T19:03:44.562575+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:01:26.001280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.001280+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003156715,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":842,"num_read_kb":837,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062320+0000","last_change":"2024-09-10T19:02:31.088205+0000","last_active":"2024-09-10T19:03:44.062320+0000","last_peered":"2024-09-10T19:03:44.062320+0000","last_clean":"2024-09-10T19:03:44.062320+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:03:44.062320+0000","last_undegraded":"2024-09-10T19:03:44.062320+0000","last_fullsized":"2024-09-10T19:03:44.062320+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:31.088025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:02:31.088025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:56.862609+0000","scrub_duration":0.0038374709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":513,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147622+0000","last_change":"2024-09-10T19:01:53.223149+0000","last_active":"2024-09-10T19:03:44.147622+0000","last_peered":"2024-09-10T19:03:44.147622+0000","last_clean":"2024-09-10T19:03:44.147622+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:03:44.147622+0000","last_undegraded":"2024-09-10T19:03:44.147622+0000","last_fullsized":"2024-09-10T19:03:44.147622+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:53.223000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.223000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036103200000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":91,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148274+0000","last_change":"2024-09-10T19:03:37.995351+0000","last_active":"2024-09-10T19:03:44.148274+0000","last_peered":"2024-09-10T19:03:44.148274+0000","last_clean":"2024-09-10T19:03:44.148274+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:03:44.148274+0000","last_undegraded":"2024-09-10T19:03:44.148274+0000","last_fullsized":"2024-09-10T19:03:44.148274+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:03:37.995076+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:03:37.995076+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:40.563839+0000","scrub_duration":0.0039698720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":612,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.069053+0000","last_change":"2024-09-10T19:02:50.928327+0000","last_active":"2024-09-10T19:03:44.069053+0000","last_peered":"2024-09-10T19:03:44.069053+0000","last_clean":"2024-09-10T19:03:44.069053+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:03:44.069053+0000","last_undegraded":"2024-09-10T19:03:44.069053+0000","last_fullsized":"2024-09-10T19:03:44.069053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'17","last_scrub_stamp":"2024-09-10T19:02:50.928028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:50.928028+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:10.231316+0000","scrub_duration":0.0046314269999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":588,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152422+0000","last_change":"2024-09-10T19:01:36.860957+0000","last_active":"2024-09-10T19:03:44.152422+0000","last_peered":"2024-09-10T19:03:44.152422+0000","last_clean":"2024-09-10T19:03:44.152422+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:03:44.152422+0000","last_undegraded":"2024-09-10T19:03:44.152422+0000","last_fullsized":"2024-09-10T19:03:44.152422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:36.860807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.860807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003533815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":751,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062733+0000","last_change":"2024-09-10T19:02:59.017671+0000","last_active":"2024-09-10T19:03:44.062733+0000","last_peered":"2024-09-10T19:03:44.062733+0000","last_clean":"2024-09-10T19:03:44.062733+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:03:44.062733+0000","last_undegraded":"2024-09-10T19:03:44.062733+0000","last_fullsized":"2024-09-10T19:03:44.062733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:02:59.017508+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:23.900581+0000","scrub_duration":0.0057392149999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":593,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145607+0000","last_change":"2024-09-10T19:02:34.226016+0000","last_active":"2024-09-10T19:03:44.145607+0000","last_peered":"2024-09-10T19:03:44.145607+0000","last_clean":"2024-09-10T19:03:44.145607+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:03:44.145607+0000","last_undegraded":"2024-09-10T19:03:44.145607+0000","last_fullsized":"2024-09-10T19:03:44.145607+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:02:34.225866+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:02:34.225866+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:57.674668+0000","scrub_duration":0.0064430909999999997,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062930+0000","last_change":"2024-09-10T19:02:52.257088+0000","last_active":"2024-09-10T19:03:44.062930+0000","last_peered":"2024-09-10T19:03:44.062930+0000","last_clean":"2024-09-10T19:03:44.062930+0000","last_became_active":"2024-09-10T19:02:52.256316+0000","last_became_peered":"2024-09-10T19:02:52.256316+0000","last_unstale":"2024-09-10T19:03:44.062930+0000","last_undegraded":"2024-09-10T19:03:44.062930+0000","last_fullsized":"2024-09-10T19:03:44.062930+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.068733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.068733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":486,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062655+0000","last_change":"2024-09-10T19:01:21.953462+0000","last_active":"2024-09-10T19:03:44.062655+0000","last_peered":"2024-09-10T19:03:44.062655+0000","last_clean":"2024-09-10T19:03:44.062655+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:03:44.062655+0000","last_undegraded":"2024-09-10T19:03:44.062655+0000","last_fullsized":"2024-09-10T19:03:44.062655+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:21.953303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:21.953303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036359589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":169,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105135+0000","last_change":"2024-09-10T19:02:56.749697+0000","last_active":"2024-09-10T19:03:44.105135+0000","last_peered":"2024-09-10T19:03:44.105135+0000","last_clean":"2024-09-10T19:03:44.105135+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:03:44.105135+0000","last_undegraded":"2024-09-10T19:03:44.105135+0000","last_fullsized":"2024-09-10T19:03:44.105135+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:56.749619+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:56.749619+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:58.782904+0000","scrub_duration":0.0032093069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":837,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151659+0000","last_change":"2024-09-10T19:02:39.992036+0000","last_active":"2024-09-10T19:03:44.151659+0000","last_peered":"2024-09-10T19:03:44.151659+0000","last_clean":"2024-09-10T19:03:44.151659+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:03:44.151659+0000","last_undegraded":"2024-09-10T19:03:44.151659+0000","last_fullsized":"2024-09-10T19:03:44.151659+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:02:39.991768+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:02:39.991768+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:58.087091+0000","scrub_duration":0.0057588450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062689+0000","last_change":"2024-09-10T19:01:20.312372+0000","last_active":"2024-09-10T19:03:44.062689+0000","last_peered":"2024-09-10T19:03:44.062689+0000","last_clean":"2024-09-10T19:03:44.062689+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:03:44.062689+0000","last_undegraded":"2024-09-10T19:03:44.062689+0000","last_fullsized":"2024-09-10T19:03:44.062689+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:20.312136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:20.312136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033891630000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":757,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063602+0000","last_change":"2024-09-10T19:02:34.523418+0000","last_active":"2024-09-10T19:03:44.063602+0000","last_peered":"2024-09-10T19:03:44.063602+0000","last_clean":"2024-09-10T19:03:44.063602+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:03:44.063602+0000","last_undegraded":"2024-09-10T19:03:44.063602+0000","last_fullsized":"2024-09-10T19:03:44.063602+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:02:34.523145+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:02:34.523145+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:56.526443+0000","scrub_duration":0.0074923719999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":590,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.103196+0000","last_change":"2024-09-10T19:03:43.845204+0000","last_active":"2024-09-10T19:03:44.103196+0000","last_peered":"2024-09-10T19:03:44.103196+0000","last_clean":"2024-09-10T19:03:44.103196+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:03:44.103196+0000","last_undegraded":"2024-09-10T19:03:44.103196+0000","last_fullsized":"2024-09-10T19:03:44.103196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:03:43.845119+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:03:43.845119+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:54.407811+0000","scrub_duration":0.0035452930000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":572,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148113+0000","last_change":"2024-09-10T19:01:18.860189+0000","last_active":"2024-09-10T19:03:44.148113+0000","last_peered":"2024-09-10T19:03:44.148113+0000","last_clean":"2024-09-10T19:03:44.148113+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:03:44.148113+0000","last_undegraded":"2024-09-10T19:03:44.148113+0000","last_fullsized":"2024-09-10T19:03:44.148113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:18.860092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:01:18.860092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032709060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":496,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.188622+0000","last_change":"2024-09-10T19:03:44.188526+0000","last_active":"2024-09-10T19:03:44.188622+0000","last_peered":"2024-09-10T19:03:44.188622+0000","last_clean":"2024-09-10T19:03:44.188622+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:03:44.188622+0000","last_undegraded":"2024-09-10T19:03:44.188622+0000","last_fullsized":"2024-09-10T19:03:44.188622+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:44.188404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:44.188404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:47.848433+0000","scrub_duration":0.0029092940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":169,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105263+0000","last_change":"2024-09-10T19:02:46.753210+0000","last_active":"2024-09-10T19:03:44.105263+0000","last_peered":"2024-09-10T19:03:44.105263+0000","last_clean":"2024-09-10T19:03:44.105263+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:03:44.105263+0000","last_undegraded":"2024-09-10T19:03:44.105263+0000","last_fullsized":"2024-09-10T19:03:44.105263+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:46.753104+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:46.753104+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:53.402601+0000","scrub_duration":0.0029584860000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"410'26","reported_seq":621,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146311+0000","last_change":"2024-09-10T19:02:56.809058+0000","last_active":"2024-09-10T19:03:44.146311+0000","last_peered":"2024-09-10T19:03:44.146311+0000","last_clean":"2024-09-10T19:03:44.146311+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:03:44.146311+0000","last_undegraded":"2024-09-10T19:03:44.146311+0000","last_fullsized":"2024-09-10T19:03:44.146311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'26","last_scrub_stamp":"2024-09-10T19:02:56.808899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:56.808899+0000","objects_scrubbed":7,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:18.953764+0000","scrub_duration":0.0046558989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151736+0000","last_change":"2024-09-10T19:01:15.859080+0000","last_active":"2024-09-10T19:03:44.151736+0000","last_peered":"2024-09-10T19:03:44.151736+0000","last_clean":"2024-09-10T19:03:44.151736+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:03:44.151736+0000","last_undegraded":"2024-09-10T19:03:44.151736+0000","last_fullsized":"2024-09-10T19:03:44.151736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:15.858920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:15.858920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003170703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":574,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150767+0000","last_change":"2024-09-10T19:01:40.872639+0000","last_active":"2024-09-10T19:03:44.150767+0000","last_peered":"2024-09-10T19:03:44.150767+0000","last_clean":"2024-09-10T19:03:44.150767+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:03:44.150767+0000","last_undegraded":"2024-09-10T19:03:44.150767+0000","last_fullsized":"2024-09-10T19:03:44.150767+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:40.872535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.872535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029510880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":571,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062375+0000","last_change":"2024-09-10T19:02:31.497142+0000","last_active":"2024-09-10T19:03:44.062375+0000","last_peered":"2024-09-10T19:03:44.062375+0000","last_clean":"2024-09-10T19:03:44.062375+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:03:44.062375+0000","last_undegraded":"2024-09-10T19:03:44.062375+0000","last_fullsized":"2024-09-10T19:03:44.062375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:31.496981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:02:31.496981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033055509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":513,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147968+0000","last_change":"2024-09-10T19:02:40.469113+0000","last_active":"2024-09-10T19:03:44.147968+0000","last_peered":"2024-09-10T19:03:44.147968+0000","last_clean":"2024-09-10T19:03:44.147968+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:03:44.147968+0000","last_undegraded":"2024-09-10T19:03:44.147968+0000","last_fullsized":"2024-09-10T19:03:44.147968+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:40.468926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:02:40.468926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:05.884472+0000","scrub_duration":0.0035903279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":90,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144260+0000","last_change":"2024-09-10T19:02:45.681688+0000","last_active":"2024-09-10T19:03:44.144260+0000","last_peered":"2024-09-10T19:03:44.144260+0000","last_clean":"2024-09-10T19:03:44.144260+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:03:44.144260+0000","last_undegraded":"2024-09-10T19:03:44.144260+0000","last_fullsized":"2024-09-10T19:03:44.144260+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:45.681423+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:45.681423+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:48.756053+0000","scrub_duration":0.003132035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":503,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063616+0000","last_change":"2024-09-10T19:03:37.758064+0000","last_active":"2024-09-10T19:03:44.063616+0000","last_peered":"2024-09-10T19:03:44.063616+0000","last_clean":"2024-09-10T19:03:44.063616+0000","last_became_active":"2024-09-10T19:02:52.256210+0000","last_became_peered":"2024-09-10T19:02:52.256210+0000","last_unstale":"2024-09-10T19:03:44.063616+0000","last_undegraded":"2024-09-10T19:03:44.063616+0000","last_fullsized":"2024-09-10T19:03:44.063616+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:03:37.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:03:37.757901+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:56.814084+0000","scrub_duration":0.0053013890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":583,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147042+0000","last_change":"2024-09-10T19:03:40.913701+0000","last_active":"2024-09-10T19:03:44.147042+0000","last_peered":"2024-09-10T19:03:44.147042+0000","last_clean":"2024-09-10T19:03:44.147042+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:03:44.147042+0000","last_undegraded":"2024-09-10T19:03:44.147042+0000","last_fullsized":"2024-09-10T19:03:44.147042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:40.913533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:55:08.199434+0000","last_clean_scrub_stamp":"2024-09-10T19:03:40.913533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:45.769579+0000","scrub_duration":0.0034909279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":584,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151611+0000","last_change":"2024-09-10T19:02:55.130793+0000","last_active":"2024-09-10T19:03:44.151611+0000","last_peered":"2024-09-10T19:03:44.151611+0000","last_clean":"2024-09-10T19:03:44.151611+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:03:44.151611+0000","last_undegraded":"2024-09-10T19:03:44.151611+0000","last_fullsized":"2024-09-10T19:03:44.151611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:55.130554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:55.130554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:57.511486+0000","scrub_duration":0.0036957909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":568,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067890+0000","last_change":"2024-09-10T19:02:31.712009+0000","last_active":"2024-09-10T19:03:44.067890+0000","last_peered":"2024-09-10T19:03:44.067890+0000","last_clean":"2024-09-10T19:03:44.067890+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:03:44.067890+0000","last_undegraded":"2024-09-10T19:03:44.067890+0000","last_fullsized":"2024-09-10T19:03:44.067890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:31.711901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:31.711901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:57.606454+0000","scrub_duration":0.0027961449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":492,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063773+0000","last_change":"2024-09-10T19:02:44.639367+0000","last_active":"2024-09-10T19:03:44.063773+0000","last_peered":"2024-09-10T19:03:44.063773+0000","last_clean":"2024-09-10T19:03:44.063773+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:03:44.063773+0000","last_undegraded":"2024-09-10T19:03:44.063773+0000","last_fullsized":"2024-09-10T19:03:44.063773+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:44.639297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:02:44.639297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:12.362996+0000","scrub_duration":0.0086554149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":655,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.060882+0000","last_change":"2024-09-10T19:02:54.034512+0000","last_active":"2024-09-10T19:03:44.060882+0000","last_peered":"2024-09-10T19:03:44.060882+0000","last_clean":"2024-09-10T19:03:44.060882+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:03:44.060882+0000","last_undegraded":"2024-09-10T19:03:44.060882+0000","last_fullsized":"2024-09-10T19:03:44.060882+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:54.034442+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:54.034442+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:17.226335+0000","scrub_duration":0.0028298059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":644,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.149083+0000","last_change":"2024-09-10T19:02:52.049399+0000","last_active":"2024-09-10T19:03:44.149083+0000","last_peered":"2024-09-10T19:03:44.149083+0000","last_clean":"2024-09-10T19:03:44.149083+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:03:44.149083+0000","last_undegraded":"2024-09-10T19:03:44.149083+0000","last_fullsized":"2024-09-10T19:03:44.149083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:02:52.049278+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:02:52.049278+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:04.310068+0000","scrub_duration":0.0045365220000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":607,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147366+0000","last_change":"2024-09-10T19:02:47.645163+0000","last_active":"2024-09-10T19:03:44.147366+0000","last_peered":"2024-09-10T19:03:44.147366+0000","last_clean":"2024-09-10T19:03:44.147366+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:03:44.147366+0000","last_undegraded":"2024-09-10T19:03:44.147366+0000","last_fullsized":"2024-09-10T19:03:44.147366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:02:47.645030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:16.818987+0000","scrub_duration":0.0058619759999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":574,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152493+0000","last_change":"2024-09-10T19:01:42.936277+0000","last_active":"2024-09-10T19:03:44.152493+0000","last_peered":"2024-09-10T19:03:44.152493+0000","last_clean":"2024-09-10T19:03:44.152493+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:03:44.152493+0000","last_undegraded":"2024-09-10T19:03:44.152493+0000","last_fullsized":"2024-09-10T19:03:44.152493+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.936095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:42.936095+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.936095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035076700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062116+0000","last_change":"2024-09-10T19:02:41.056245+0000","last_active":"2024-09-10T19:03:44.062116+0000","last_peered":"2024-09-10T19:03:44.062116+0000","last_clean":"2024-09-10T19:03:44.062116+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:03:44.062116+0000","last_undegraded":"2024-09-10T19:03:44.062116+0000","last_fullsized":"2024-09-10T19:03:44.062116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:41.056074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:41.056074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:57.509887+0000","scrub_duration":0.0033083850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":503,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105918+0000","last_change":"2024-09-10T19:03:37.752087+0000","last_active":"2024-09-10T19:03:44.105918+0000","last_peered":"2024-09-10T19:03:44.105918+0000","last_clean":"2024-09-10T19:03:44.105918+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:03:44.105918+0000","last_undegraded":"2024-09-10T19:03:44.105918+0000","last_fullsized":"2024-09-10T19:03:44.105918+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:37.751974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:37.751974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:47.864134+0000","scrub_duration":0.0031723350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":81,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150412+0000","last_change":"2024-09-10T19:02:03.616590+0000","last_active":"2024-09-10T19:03:44.150412+0000","last_peered":"2024-09-10T19:03:44.150412+0000","last_clean":"2024-09-10T19:03:44.150412+0000","last_became_active":"2024-09-10T19:02:03.616064+0000","last_became_peered":"2024-09-10T19:02:03.616064+0000","last_unstale":"2024-09-10T19:03:44.150412+0000","last_undegraded":"2024-09-10T19:03:44.150412+0000","last_fullsized":"2024-09-10T19:03:44.150412+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:00:12.005490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:00:12.005490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":617,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062378+0000","last_change":"2024-09-10T19:01:53.649846+0000","last_active":"2024-09-10T19:03:44.062378+0000","last_peered":"2024-09-10T19:03:44.062378+0000","last_clean":"2024-09-10T19:03:44.062378+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:03:44.062378+0000","last_undegraded":"2024-09-10T19:03:44.062378+0000","last_fullsized":"2024-09-10T19:03:44.062378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'28","last_scrub_stamp":"2024-09-10T19:01:53.649703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:53.649703+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088661820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":587,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.059897+0000","last_change":"2024-09-10T19:02:33.099677+0000","last_active":"2024-09-10T19:03:44.059897+0000","last_peered":"2024-09-10T19:03:44.059897+0000","last_clean":"2024-09-10T19:03:44.059897+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:03:44.059897+0000","last_undegraded":"2024-09-10T19:03:44.059897+0000","last_fullsized":"2024-09-10T19:03:44.059897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:33.099518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:02:33.099518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:51.563288+0000","scrub_duration":0.0031881829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":576,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064700+0000","last_change":"2024-09-10T19:02:29.540230+0000","last_active":"2024-09-10T19:03:44.064700+0000","last_peered":"2024-09-10T19:03:44.064700+0000","last_clean":"2024-09-10T19:03:44.064700+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:03:44.064700+0000","last_undegraded":"2024-09-10T19:03:44.064700+0000","last_fullsized":"2024-09-10T19:03:44.064700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:29.540120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:29.540120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025188570000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":582,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150709+0000","last_change":"2024-09-10T19:02:54.086644+0000","last_active":"2024-09-10T19:03:44.150709+0000","last_peered":"2024-09-10T19:03:44.150709+0000","last_clean":"2024-09-10T19:03:44.150709+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:03:44.150709+0000","last_undegraded":"2024-09-10T19:03:44.150709+0000","last_fullsized":"2024-09-10T19:03:44.150709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:54.086488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:16.451524+0000","last_clean_scrub_stamp":"2024-09-10T19:02:54.086488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:56.725351+0000","scrub_duration":0.0030412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":492,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064677+0000","last_change":"2024-09-10T19:02:52.537774+0000","last_active":"2024-09-10T19:03:44.064677+0000","last_peered":"2024-09-10T19:03:44.064677+0000","last_clean":"2024-09-10T19:03:44.064677+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:03:44.064677+0000","last_undegraded":"2024-09-10T19:03:44.064677+0000","last_fullsized":"2024-09-10T19:03:44.064677+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:52.537524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:02:52.537524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:03.554517+0000","scrub_duration":0.0038019759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":604,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062824+0000","last_change":"2024-09-10T19:02:35.804692+0000","last_active":"2024-09-10T19:03:44.062824+0000","last_peered":"2024-09-10T19:03:44.062824+0000","last_clean":"2024-09-10T19:03:44.062824+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:03:44.062824+0000","last_undegraded":"2024-09-10T19:03:44.062824+0000","last_fullsized":"2024-09-10T19:03:44.062824+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:02:35.804540+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:02:35.804540+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:58.917239+0000","scrub_duration":0.0054189540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":583,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145259+0000","last_change":"2024-09-10T19:02:36.500794+0000","last_active":"2024-09-10T19:03:44.145259+0000","last_peered":"2024-09-10T19:03:44.145259+0000","last_clean":"2024-09-10T19:03:44.145259+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:03:44.145259+0000","last_undegraded":"2024-09-10T19:03:44.145259+0000","last_fullsized":"2024-09-10T19:03:44.145259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:36.500638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:02:36.500638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:00.102720+0000","scrub_duration":0.0033480509999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":646,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.060020+0000","last_change":"2024-09-10T19:02:52.111149+0000","last_active":"2024-09-10T19:03:44.060020+0000","last_peered":"2024-09-10T19:03:44.060020+0000","last_clean":"2024-09-10T19:03:44.060020+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:03:44.060020+0000","last_undegraded":"2024-09-10T19:03:44.060020+0000","last_fullsized":"2024-09-10T19:03:44.060020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:02:52.111052+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:02:52.111052+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:17.915793+0000","scrub_duration":0.0036843819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":582,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105150+0000","last_change":"2024-09-10T19:03:41.822899+0000","last_active":"2024-09-10T19:03:44.105150+0000","last_peered":"2024-09-10T19:03:44.105150+0000","last_clean":"2024-09-10T19:03:44.105150+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:03:44.105150+0000","last_undegraded":"2024-09-10T19:03:44.105150+0000","last_fullsized":"2024-09-10T19:03:44.105150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:41.822743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:03:41.822743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:03.488404+0000","scrub_duration":0.0033709899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":570,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147172+0000","last_change":"2024-09-10T19:02:30.481343+0000","last_active":"2024-09-10T19:03:44.147172+0000","last_peered":"2024-09-10T19:03:44.147172+0000","last_clean":"2024-09-10T19:03:44.147172+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:03:44.147172+0000","last_undegraded":"2024-09-10T19:03:44.147172+0000","last_fullsized":"2024-09-10T19:03:44.147172+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:30.481232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:30.481232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:54.316207+0000","scrub_duration":0.0028715030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":505,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062463+0000","last_change":"2024-09-10T19:03:39.016176+0000","last_active":"2024-09-10T19:03:44.062463+0000","last_peered":"2024-09-10T19:03:44.062463+0000","last_clean":"2024-09-10T19:03:44.062463+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:03:44.062463+0000","last_undegraded":"2024-09-10T19:03:44.062463+0000","last_fullsized":"2024-09-10T19:03:44.062463+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:39.015985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:30.496126+0000","last_clean_scrub_stamp":"2024-09-10T19:03:39.015985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:07.432898+0000","scrub_duration":0.0039411059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":632,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062722+0000","last_change":"2024-09-10T19:01:27.255478+0000","last_active":"2024-09-10T19:03:44.062722+0000","last_peered":"2024-09-10T19:03:44.062722+0000","last_clean":"2024-09-10T19:03:44.062722+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:03:44.062722+0000","last_undegraded":"2024-09-10T19:03:44.062722+0000","last_fullsized":"2024-09-10T19:03:44.062722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:01:27.255315+0000","last_deep_scrub":"376'25","last_deep_scrub_stamp":"2024-09-10T18:58:52.253376+0000","last_clean_scrub_stamp":"2024-09-10T19:01:27.255315+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050413439999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":607,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150067+0000","last_change":"2024-09-10T19:03:41.595880+0000","last_active":"2024-09-10T19:03:44.150067+0000","last_peered":"2024-09-10T19:03:44.150067+0000","last_clean":"2024-09-10T19:03:44.150067+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:03:44.150067+0000","last_undegraded":"2024-09-10T19:03:44.150067+0000","last_fullsized":"2024-09-10T19:03:44.150067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:41.595722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:03:41.595722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:55.477250+0000","scrub_duration":0.003241584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":582,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.103717+0000","last_change":"2024-09-10T19:02:43.758465+0000","last_active":"2024-09-10T19:03:44.103717+0000","last_peered":"2024-09-10T19:03:44.103717+0000","last_clean":"2024-09-10T19:03:44.103717+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:03:44.103717+0000","last_undegraded":"2024-09-10T19:03:44.103717+0000","last_fullsized":"2024-09-10T19:03:44.103717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:43.758389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:02:43.758389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:45.464933+0000","scrub_duration":0.0028585310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.060422+0000","last_change":"2024-09-10T19:03:40.025804+0000","last_active":"2024-09-10T19:03:44.060422+0000","last_peered":"2024-09-10T19:03:44.060422+0000","last_clean":"2024-09-10T19:03:44.060422+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:03:44.060422+0000","last_undegraded":"2024-09-10T19:03:44.060422+0000","last_fullsized":"2024-09-10T19:03:44.060422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:40.025687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:03:40.025687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:52.971945+0000","scrub_duration":0.003329944,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":495,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146593+0000","last_change":"2024-09-10T19:01:42.293853+0000","last_active":"2024-09-10T19:03:44.146593+0000","last_peered":"2024-09-10T19:03:44.146593+0000","last_clean":"2024-09-10T19:03:44.146593+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:03:44.146593+0000","last_undegraded":"2024-09-10T19:03:44.146593+0000","last_fullsized":"2024-09-10T19:03:44.146593+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.293664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.293664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031423390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":616,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146116+0000","last_change":"2024-09-10T19:03:35.935370+0000","last_active":"2024-09-10T19:03:44.146116+0000","last_peered":"2024-09-10T19:03:44.146116+0000","last_clean":"2024-09-10T19:03:44.146116+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:03:44.146116+0000","last_undegraded":"2024-09-10T19:03:44.146116+0000","last_fullsized":"2024-09-10T19:03:44.146116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:03:35.935207+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:03:35.935207+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:55.719205+0000","scrub_duration":0.0043315480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152316+0000","last_change":"2024-09-10T19:00:34.708486+0000","last_active":"2024-09-10T19:03:44.152316+0000","last_peered":"2024-09-10T19:03:44.152316+0000","last_clean":"2024-09-10T19:03:44.152316+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:03:44.152316+0000","last_undegraded":"2024-09-10T19:03:44.152316+0000","last_fullsized":"2024-09-10T19:03:44.152316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:34.708428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:00:34.708428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023757650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":563,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063472+0000","last_change":"2024-09-10T19:01:41.594329+0000","last_active":"2024-09-10T19:03:44.063472+0000","last_peered":"2024-09-10T19:03:44.063472+0000","last_clean":"2024-09-10T19:03:44.063472+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:03:44.063472+0000","last_undegraded":"2024-09-10T19:03:44.063472+0000","last_fullsized":"2024-09-10T19:03:44.063472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.594132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.594132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003049576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":561,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062188+0000","last_change":"2024-09-10T19:01:35.482982+0000","last_active":"2024-09-10T19:03:44.062188+0000","last_peered":"2024-09-10T19:03:44.062188+0000","last_clean":"2024-09-10T19:03:44.062188+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:03:44.062188+0000","last_undegraded":"2024-09-10T19:03:44.062188+0000","last_fullsized":"2024-09-10T19:03:44.062188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:35.482815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:25.583492+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.482815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032822699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":492,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063487+0000","last_change":"2024-09-10T19:02:46.602916+0000","last_active":"2024-09-10T19:03:44.063487+0000","last_peered":"2024-09-10T19:03:44.063487+0000","last_clean":"2024-09-10T19:03:44.063487+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:03:44.063487+0000","last_undegraded":"2024-09-10T19:03:44.063487+0000","last_fullsized":"2024-09-10T19:03:44.063487+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:46.602792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:46.602792+0000","last_clean_scrub_stamp":"2024-09-10T19:02:46.602792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:53.076316+0000","scrub_duration":0.0032351569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":635,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062863+0000","last_change":"2024-09-10T19:02:35.103909+0000","last_active":"2024-09-10T19:03:44.062863+0000","last_peered":"2024-09-10T19:03:44.062863+0000","last_clean":"2024-09-10T19:03:44.062863+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:03:44.062863+0000","last_undegraded":"2024-09-10T19:03:44.062863+0000","last_fullsized":"2024-09-10T19:03:44.062863+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:02:35.103742+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:02:35.103742+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:04.327362+0000","scrub_duration":0.0048878819999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":619,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150933+0000","last_change":"2024-09-10T19:02:51.606998+0000","last_active":"2024-09-10T19:03:44.150933+0000","last_peered":"2024-09-10T19:03:44.150933+0000","last_clean":"2024-09-10T19:03:44.150933+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:03:44.150933+0000","last_undegraded":"2024-09-10T19:03:44.150933+0000","last_fullsized":"2024-09-10T19:03:44.150933+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:02:51.606831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:51.606831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:12.411061+0000","scrub_duration":0.0043976960000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":579,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144612+0000","last_change":"2024-09-10T19:01:37.418361+0000","last_active":"2024-09-10T19:03:44.144612+0000","last_peered":"2024-09-10T19:03:44.144612+0000","last_clean":"2024-09-10T19:03:44.144612+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:03:44.144612+0000","last_undegraded":"2024-09-10T19:03:44.144612+0000","last_fullsized":"2024-09-10T19:03:44.144612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.418152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.418152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033093839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145629+0000","last_change":"2024-09-10T19:02:39.584598+0000","last_active":"2024-09-10T19:03:44.145629+0000","last_peered":"2024-09-10T19:03:44.145629+0000","last_clean":"2024-09-10T19:03:44.145629+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:03:44.145629+0000","last_undegraded":"2024-09-10T19:03:44.145629+0000","last_fullsized":"2024-09-10T19:03:44.145629+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:39.584449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:39.584449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:00.194255+0000","scrub_duration":0.0032346950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":47,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.101762+0000","last_change":"2024-09-10T19:02:52.257186+0000","last_active":"2024-09-10T19:03:44.101762+0000","last_peered":"2024-09-10T19:03:44.101762+0000","last_clean":"2024-09-10T19:03:44.101762+0000","last_became_active":"2024-09-10T19:02:52.256701+0000","last_became_peered":"2024-09-10T19:02:52.256701+0000","last_unstale":"2024-09-10T19:03:44.101762+0000","last_undegraded":"2024-09-10T19:03:44.101762+0000","last_fullsized":"2024-09-10T19:03:44.101762+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.205158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.205158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":609,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063115+0000","last_change":"2024-09-10T19:03:33.963661+0000","last_active":"2024-09-10T19:03:44.063115+0000","last_peered":"2024-09-10T19:03:44.063115+0000","last_clean":"2024-09-10T19:03:44.063115+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:03:44.063115+0000","last_undegraded":"2024-09-10T19:03:44.063115+0000","last_fullsized":"2024-09-10T19:03:44.063115+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:01:22.297269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.297269+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005830773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"70'2","reported_seq":629,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148984+0000","last_change":"2024-09-10T19:02:53.458035+0000","last_active":"2024-09-10T19:03:44.148984+0000","last_peered":"2024-09-10T19:03:44.148984+0000","last_clean":"2024-09-10T19:03:44.148984+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:03:44.148984+0000","last_undegraded":"2024-09-10T19:03:44.148984+0000","last_fullsized":"2024-09-10T19:03:44.148984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-09-10T19:02:53.457861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:53.457861+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:55.088443+0000","scrub_duration":0.013569238000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"548'12","reported_seq":89,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148420+0000","last_change":"2024-09-10T19:03:36.114792+0000","last_active":"2024-09-10T19:03:44.148420+0000","last_peered":"2024-09-10T19:03:44.148420+0000","last_clean":"2024-09-10T19:03:44.148420+0000","last_became_active":"2024-09-10T19:02:52.258504+0000","last_became_peered":"2024-09-10T19:02:52.258504+0000","last_unstale":"2024-09-10T19:03:44.148420+0000","last_undegraded":"2024-09-10T19:03:44.148420+0000","last_fullsized":"2024-09-10T19:03:44.148420+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-10T19:03:36.114662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:03:36.114662+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:39.730327+0000","scrub_duration":0.003209234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":565,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066551+0000","last_change":"2024-09-10T19:01:19.956970+0000","last_active":"2024-09-10T19:03:44.066551+0000","last_peered":"2024-09-10T19:03:44.066551+0000","last_clean":"2024-09-10T19:03:44.066551+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:03:44.066551+0000","last_undegraded":"2024-09-10T19:03:44.066551+0000","last_fullsized":"2024-09-10T19:03:44.066551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:19.956889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.956889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027515830000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":506,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.101985+0000","last_change":"2024-09-10T19:01:43.827904+0000","last_active":"2024-09-10T19:03:44.101985+0000","last_peered":"2024-09-10T19:03:44.101985+0000","last_clean":"2024-09-10T19:03:44.101985+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:03:44.101985+0000","last_undegraded":"2024-09-10T19:03:44.101985+0000","last_fullsized":"2024-09-10T19:03:44.101985+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:01:43.827739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.827739+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040040049999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":616,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061038+0000","last_change":"2024-09-10T19:01:47.273790+0000","last_active":"2024-09-10T19:03:44.061038+0000","last_peered":"2024-09-10T19:03:44.061038+0000","last_clean":"2024-09-10T19:03:44.061038+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:03:44.061038+0000","last_undegraded":"2024-09-10T19:03:44.061038+0000","last_fullsized":"2024-09-10T19:03:44.061038+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:01:47.273620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.273620+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048496030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":597,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147489+0000","last_change":"2024-09-10T19:01:39.125687+0000","last_active":"2024-09-10T19:03:44.147489+0000","last_peered":"2024-09-10T19:03:44.147489+0000","last_clean":"2024-09-10T19:03:44.147489+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:03:44.147489+0000","last_undegraded":"2024-09-10T19:03:44.147489+0000","last_fullsized":"2024-09-10T19:03:44.147489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.125534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.125534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037788880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":573,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064468+0000","last_change":"2024-09-10T19:02:41.680449+0000","last_active":"2024-09-10T19:03:44.064468+0000","last_peered":"2024-09-10T19:03:44.064468+0000","last_clean":"2024-09-10T19:03:44.064468+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:03:44.064468+0000","last_undegraded":"2024-09-10T19:03:44.064468+0000","last_fullsized":"2024-09-10T19:03:44.064468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:41.680338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:41.680338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:59.362552+0000","scrub_duration":0.0028326079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":591,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148844+0000","last_change":"2024-09-10T19:02:50.626472+0000","last_active":"2024-09-10T19:03:44.148844+0000","last_peered":"2024-09-10T19:03:44.148844+0000","last_clean":"2024-09-10T19:03:44.148844+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:03:44.148844+0000","last_undegraded":"2024-09-10T19:03:44.148844+0000","last_fullsized":"2024-09-10T19:03:44.148844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:50.626308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:50.626308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:07.081748+0000","scrub_duration":0.003463954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":495,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.149244+0000","last_change":"2024-09-10T19:01:40.232693+0000","last_active":"2024-09-10T19:03:44.149244+0000","last_peered":"2024-09-10T19:03:44.149244+0000","last_clean":"2024-09-10T19:03:44.149244+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:03:44.149244+0000","last_undegraded":"2024-09-10T19:03:44.149244+0000","last_fullsized":"2024-09-10T19:03:44.149244+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:40.232583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:40.232583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029621439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":598,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.104307+0000","last_change":"2024-09-10T19:02:32.941683+0000","last_active":"2024-09-10T19:03:44.104307+0000","last_peered":"2024-09-10T19:03:44.104307+0000","last_clean":"2024-09-10T19:03:44.104307+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:03:44.104307+0000","last_undegraded":"2024-09-10T19:03:44.104307+0000","last_fullsized":"2024-09-10T19:03:44.104307+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:02:32.941501+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:02:32.941501+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:46.579201+0000","scrub_duration":0.0041743359999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":586,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.103420+0000","last_change":"2024-09-10T19:01:54.789334+0000","last_active":"2024-09-10T19:03:44.103420+0000","last_peered":"2024-09-10T19:03:44.103420+0000","last_clean":"2024-09-10T19:03:44.103420+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:03:44.103420+0000","last_undegraded":"2024-09-10T19:03:44.103420+0000","last_fullsized":"2024-09-10T19:03:44.103420+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:54.789186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:54.789186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031670539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":567,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.068977+0000","last_change":"2024-09-10T19:01:47.940008+0000","last_active":"2024-09-10T19:03:44.068977+0000","last_peered":"2024-09-10T19:03:44.068977+0000","last_clean":"2024-09-10T19:03:44.068977+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:03:44.068977+0000","last_undegraded":"2024-09-10T19:03:44.068977+0000","last_fullsized":"2024-09-10T19:03:44.068977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:47.939853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:01:47.939853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034949289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062042+0000","last_change":"2024-09-10T19:02:40.039115+0000","last_active":"2024-09-10T19:03:44.062042+0000","last_peered":"2024-09-10T19:03:44.062042+0000","last_clean":"2024-09-10T19:03:44.062042+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:03:44.062042+0000","last_undegraded":"2024-09-10T19:03:44.062042+0000","last_fullsized":"2024-09-10T19:03:44.062042+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:40.038956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:40.038956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:01.600095+0000","scrub_duration":0.0033627840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":503,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105511+0000","last_change":"2024-09-10T19:03:38.739480+0000","last_active":"2024-09-10T19:03:44.105511+0000","last_peered":"2024-09-10T19:03:44.105511+0000","last_clean":"2024-09-10T19:03:44.105511+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:03:44.105511+0000","last_undegraded":"2024-09-10T19:03:44.105511+0000","last_fullsized":"2024-09-10T19:03:44.105511+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:38.739222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:38.739222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:47.275282+0000","scrub_duration":0.0049343039999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":611,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062687+0000","last_change":"2024-09-10T19:01:19.268405+0000","last_active":"2024-09-10T19:03:44.062687+0000","last_peered":"2024-09-10T19:03:44.062687+0000","last_clean":"2024-09-10T19:03:44.062687+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:03:44.062687+0000","last_undegraded":"2024-09-10T19:03:44.062687+0000","last_fullsized":"2024-09-10T19:03:44.062687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:01:19.268316+0000","last_deep_scrub":"108'16","last_deep_scrub_stamp":"2024-09-10T18:52:48.328639+0000","last_clean_scrub_stamp":"2024-09-10T19:01:19.268316+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078237800000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":576,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147189+0000","last_change":"2024-09-10T19:01:42.462844+0000","last_active":"2024-09-10T19:03:44.147189+0000","last_peered":"2024-09-10T19:03:44.147189+0000","last_clean":"2024-09-10T19:03:44.147189+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:03:44.147189+0000","last_undegraded":"2024-09-10T19:03:44.147189+0000","last_fullsized":"2024-09-10T19:03:44.147189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:42.462690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:42.462690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032025000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"77'11","reported_seq":753,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067393+0000","last_change":"2024-09-10T19:00:26.925255+0000","last_active":"2024-09-10T19:03:44.067393+0000","last_peered":"2024-09-10T19:03:44.067393+0000","last_clean":"2024-09-10T19:03:44.067393+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:03:44.067393+0000","last_undegraded":"2024-09-10T19:03:44.067393+0000","last_fullsized":"2024-09-10T19:03:44.067393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:00:26.925124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:00:26.925124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037103370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":571,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064345+0000","last_change":"2024-09-10T19:02:37.543495+0000","last_active":"2024-09-10T19:03:44.064345+0000","last_peered":"2024-09-10T19:03:44.064345+0000","last_clean":"2024-09-10T19:03:44.064345+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:03:44.064345+0000","last_undegraded":"2024-09-10T19:03:44.064345+0000","last_fullsized":"2024-09-10T19:03:44.064345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:37.543331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:20.402922+0000","last_clean_scrub_stamp":"2024-09-10T19:02:37.543331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:55.398279+0000","scrub_duration":0.0039877619999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":492,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064041+0000","last_change":"2024-09-10T19:02:51.517128+0000","last_active":"2024-09-10T19:03:44.064041+0000","last_peered":"2024-09-10T19:03:44.064041+0000","last_clean":"2024-09-10T19:03:44.064041+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:03:44.064041+0000","last_undegraded":"2024-09-10T19:03:44.064041+0000","last_fullsized":"2024-09-10T19:03:44.064041+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:51.516968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:02:51.516968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:17.359585+0000","scrub_duration":0.0035538850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":600,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063561+0000","last_change":"2024-09-10T19:02:39.608134+0000","last_active":"2024-09-10T19:03:44.063561+0000","last_peered":"2024-09-10T19:03:44.063561+0000","last_clean":"2024-09-10T19:03:44.063561+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:03:44.063561+0000","last_undegraded":"2024-09-10T19:03:44.063561+0000","last_fullsized":"2024-09-10T19:03:44.063561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:02:39.607984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:39.607984+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:52.700221+0000","scrub_duration":0.0042023340000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":577,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063983+0000","last_change":"2024-09-10T19:01:44.606532+0000","last_active":"2024-09-10T19:03:44.063983+0000","last_peered":"2024-09-10T19:03:44.063983+0000","last_clean":"2024-09-10T19:03:44.063983+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:03:44.063983+0000","last_undegraded":"2024-09-10T19:03:44.063983+0000","last_fullsized":"2024-09-10T19:03:44.063983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.606395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039478229999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":752,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145960+0000","last_change":"2024-09-10T19:03:39.952915+0000","last_active":"2024-09-10T19:03:44.145960+0000","last_peered":"2024-09-10T19:03:44.145960+0000","last_clean":"2024-09-10T19:03:44.145960+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:03:44.145960+0000","last_undegraded":"2024-09-10T19:03:44.145960+0000","last_fullsized":"2024-09-10T19:03:44.145960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:03:39.952752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:03:39.952752+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:05.837717+0000","scrub_duration":0.0038584940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":56,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151286+0000","last_change":"2024-09-10T19:02:53.636260+0000","last_active":"2024-09-10T19:03:44.151286+0000","last_peered":"2024-09-10T19:03:44.151286+0000","last_clean":"2024-09-10T19:03:44.151286+0000","last_became_active":"2024-09-10T19:02:52.256751+0000","last_became_peered":"2024-09-10T19:02:52.256751+0000","last_unstale":"2024-09-10T19:03:44.151286+0000","last_undegraded":"2024-09-10T19:03:44.151286+0000","last_fullsized":"2024-09-10T19:03:44.151286+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:53.636196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:02:53.636196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:18.243825+0000","scrub_duration":0.010290598999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":482,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064596+0000","last_change":"2024-09-10T19:01:37.571899+0000","last_active":"2024-09-10T19:03:44.064596+0000","last_peered":"2024-09-10T19:03:44.064596+0000","last_clean":"2024-09-10T19:03:44.064596+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:03:44.064596+0000","last_undegraded":"2024-09-10T19:03:44.064596+0000","last_fullsized":"2024-09-10T19:03:44.064596+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.571747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035003059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":610,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062294+0000","last_change":"2024-09-10T19:02:48.146598+0000","last_active":"2024-09-10T19:03:44.062294+0000","last_peered":"2024-09-10T19:03:44.062294+0000","last_clean":"2024-09-10T19:03:44.062294+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:03:44.062294+0000","last_undegraded":"2024-09-10T19:03:44.062294+0000","last_fullsized":"2024-09-10T19:03:44.062294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:02:48.146432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:48.146432+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:10.028394+0000","scrub_duration":0.0046942729999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":576,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.104353+0000","last_change":"2024-09-10T19:01:31.660345+0000","last_active":"2024-09-10T19:03:44.104353+0000","last_peered":"2024-09-10T19:03:44.104353+0000","last_clean":"2024-09-10T19:03:44.104353+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:03:44.104353+0000","last_undegraded":"2024-09-10T19:03:44.104353+0000","last_fullsized":"2024-09-10T19:03:44.104353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:31.660098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:31.660098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038432570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":570,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.145289+0000","last_change":"2024-09-10T19:01:46.395332+0000","last_active":"2024-09-10T19:03:44.145289+0000","last_peered":"2024-09-10T19:03:44.145289+0000","last_clean":"2024-09-10T19:03:44.145289+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:03:44.145289+0000","last_undegraded":"2024-09-10T19:03:44.145289+0000","last_fullsized":"2024-09-10T19:03:44.145289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:01:46.395225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035309550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":574,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061702+0000","last_change":"2024-09-10T19:01:55.601470+0000","last_active":"2024-09-10T19:03:44.061702+0000","last_peered":"2024-09-10T19:03:44.061702+0000","last_clean":"2024-09-10T19:03:44.061702+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:03:44.061702+0000","last_undegraded":"2024-09-10T19:03:44.061702+0000","last_fullsized":"2024-09-10T19:03:44.061702+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:01:55.601279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034556629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":505,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.147396+0000","last_change":"2024-09-10T19:03:43.058097+0000","last_active":"2024-09-10T19:03:44.147396+0000","last_peered":"2024-09-10T19:03:44.147396+0000","last_clean":"2024-09-10T19:03:44.147396+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:03:44.147396+0000","last_undegraded":"2024-09-10T19:03:44.147396+0000","last_fullsized":"2024-09-10T19:03:44.147396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:43.058009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:43.058009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:02.544089+0000","scrub_duration":0.0021985189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":487,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061647+0000","last_change":"2024-09-10T19:02:53.074185+0000","last_active":"2024-09-10T19:03:44.061647+0000","last_peered":"2024-09-10T19:03:44.061647+0000","last_clean":"2024-09-10T19:03:44.061647+0000","last_became_active":"2024-09-10T19:02:52.257159+0000","last_became_peered":"2024-09-10T19:02:52.257159+0000","last_unstale":"2024-09-10T19:03:44.061647+0000","last_undegraded":"2024-09-10T19:03:44.061647+0000","last_fullsized":"2024-09-10T19:03:44.061647+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:02:53.074013+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:02:53.074013+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:53.396060+0000","scrub_duration":0.0052217920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":588,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.152225+0000","last_change":"2024-09-10T19:02:36.994344+0000","last_active":"2024-09-10T19:03:44.152225+0000","last_peered":"2024-09-10T19:03:44.152225+0000","last_clean":"2024-09-10T19:03:44.152225+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:03:44.152225+0000","last_undegraded":"2024-09-10T19:03:44.152225+0000","last_fullsized":"2024-09-10T19:03:44.152225+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:36.994187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:36.994187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:06.849040+0000","scrub_duration":0.0033462570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":576,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148350+0000","last_change":"2024-09-10T19:01:44.276771+0000","last_active":"2024-09-10T19:03:44.148350+0000","last_peered":"2024-09-10T19:03:44.148350+0000","last_clean":"2024-09-10T19:03:44.148350+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:03:44.148350+0000","last_undegraded":"2024-09-10T19:03:44.148350+0000","last_fullsized":"2024-09-10T19:03:44.148350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.276607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.276607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037398430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066422+0000","last_change":"2024-09-10T19:02:51.975878+0000","last_active":"2024-09-10T19:03:44.066422+0000","last_peered":"2024-09-10T19:03:44.066422+0000","last_clean":"2024-09-10T19:03:44.066422+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:03:44.066422+0000","last_undegraded":"2024-09-10T19:03:44.066422+0000","last_fullsized":"2024-09-10T19:03:44.066422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:51.975710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:51.975710+0000","last_clean_scrub_stamp":"2024-09-10T19:02:51.975710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:14.054390+0000","scrub_duration":0.0032292699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":48,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.150819+0000","last_change":"2024-09-10T19:02:52.259262+0000","last_active":"2024-09-10T19:03:44.150819+0000","last_peered":"2024-09-10T19:03:44.150819+0000","last_clean":"2024-09-10T19:03:44.150819+0000","last_became_active":"2024-09-10T19:02:52.258536+0000","last_became_peered":"2024-09-10T19:02:52.258536+0000","last_unstale":"2024-09-10T19:03:44.150819+0000","last_undegraded":"2024-09-10T19:03:44.150819+0000","last_fullsized":"2024-09-10T19:03:44.150819+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:26.151456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:26.151456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":596,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064969+0000","last_change":"2024-09-10T19:03:38.739129+0000","last_active":"2024-09-10T19:03:44.064969+0000","last_peered":"2024-09-10T19:03:44.064969+0000","last_clean":"2024-09-10T19:03:44.064969+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:03:44.064969+0000","last_undegraded":"2024-09-10T19:03:44.064969+0000","last_fullsized":"2024-09-10T19:03:44.064969+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:03:38.738861+0000","last_deep_scrub":"108'9","last_deep_scrub_stamp":"2024-09-10T18:52:39.623723+0000","last_clean_scrub_stamp":"2024-09-10T19:03:38.738861+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:01.716999+0000","scrub_duration":0.005197485,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":583,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146967+0000","last_change":"2024-09-10T19:03:33.933501+0000","last_active":"2024-09-10T19:03:44.146967+0000","last_peered":"2024-09-10T19:03:44.146967+0000","last_clean":"2024-09-10T19:03:44.146967+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:03:44.146967+0000","last_undegraded":"2024-09-10T19:03:44.146967+0000","last_fullsized":"2024-09-10T19:03:44.146967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:33.933333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:03:33.933333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:02.310978+0000","scrub_duration":0.0032241119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":570,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062573+0000","last_change":"2024-09-10T19:02:55.044624+0000","last_active":"2024-09-10T19:03:44.062573+0000","last_peered":"2024-09-10T19:03:44.062573+0000","last_clean":"2024-09-10T19:03:44.062573+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:03:44.062573+0000","last_undegraded":"2024-09-10T19:03:44.062573+0000","last_fullsized":"2024-09-10T19:03:44.062573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:55.044466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:38.650775+0000","last_clean_scrub_stamp":"2024-09-10T19:02:55.044466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:10.707600+0000","scrub_duration":0.0035938279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061293+0000","last_change":"2024-09-10T19:02:32.137610+0000","last_active":"2024-09-10T19:03:44.061293+0000","last_peered":"2024-09-10T19:03:44.061293+0000","last_clean":"2024-09-10T19:03:44.061293+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:03:44.061293+0000","last_undegraded":"2024-09-10T19:03:44.061293+0000","last_fullsized":"2024-09-10T19:03:44.061293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:32.137455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:02:32.137455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034123140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":494,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148976+0000","last_change":"2024-09-10T19:01:52.787451+0000","last_active":"2024-09-10T19:03:44.148976+0000","last_peered":"2024-09-10T19:03:44.148976+0000","last_clean":"2024-09-10T19:03:44.148976+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:03:44.148976+0000","last_undegraded":"2024-09-10T19:03:44.148976+0000","last_fullsized":"2024-09-10T19:03:44.148976+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:52.787380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:55.811185+0000","last_clean_scrub_stamp":"2024-09-10T19:01:52.787380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034234759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"409'15","reported_seq":596,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.146737+0000","last_change":"2024-09-10T19:01:35.373516+0000","last_active":"2024-09-10T19:03:44.146737+0000","last_peered":"2024-09-10T19:03:44.146737+0000","last_clean":"2024-09-10T19:03:44.146737+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:03:44.146737+0000","last_undegraded":"2024-09-10T19:03:44.146737+0000","last_fullsized":"2024-09-10T19:03:44.146737+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:01:35.373263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:35.373263+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048975939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":586,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.106238+0000","last_change":"2024-09-10T19:02:35.912840+0000","last_active":"2024-09-10T19:03:44.106238+0000","last_peered":"2024-09-10T19:03:44.106238+0000","last_clean":"2024-09-10T19:03:44.106238+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:03:44.106238+0000","last_undegraded":"2024-09-10T19:03:44.106238+0000","last_fullsized":"2024-09-10T19:03:44.106238+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:35.912761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:35.912761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:54.746305+0000","scrub_duration":0.0027534460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":567,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066607+0000","last_change":"2024-09-10T19:02:49.895259+0000","last_active":"2024-09-10T19:03:44.066607+0000","last_peered":"2024-09-10T19:03:44.066607+0000","last_clean":"2024-09-10T19:03:44.066607+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:03:44.066607+0000","last_undegraded":"2024-09-10T19:03:44.066607+0000","last_fullsized":"2024-09-10T19:03:44.066607+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:49.895149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:49.895149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:53.181301+0000","scrub_duration":0.0030212360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":565,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.066764+0000","last_change":"2024-09-10T19:00:23.839038+0000","last_active":"2024-09-10T19:03:44.066764+0000","last_peered":"2024-09-10T19:03:44.066764+0000","last_clean":"2024-09-10T19:03:44.066764+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:03:44.066764+0000","last_undegraded":"2024-09-10T19:03:44.066764+0000","last_fullsized":"2024-09-10T19:03:44.066764+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:00:23.838880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:00:23.838880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033929009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":482,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062674+0000","last_change":"2024-09-10T19:01:22.572687+0000","last_active":"2024-09-10T19:03:44.062674+0000","last_peered":"2024-09-10T19:03:44.062674+0000","last_clean":"2024-09-10T19:03:44.062674+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:03:44.062674+0000","last_undegraded":"2024-09-10T19:03:44.062674+0000","last_fullsized":"2024-09-10T19:03:44.062674+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.572609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.572609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029042989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":578,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.061214+0000","last_change":"2024-09-10T19:02:30.081130+0000","last_active":"2024-09-10T19:03:44.061214+0000","last_peered":"2024-09-10T19:03:44.061214+0000","last_clean":"2024-09-10T19:03:44.061214+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:03:44.061214+0000","last_undegraded":"2024-09-10T19:03:44.061214+0000","last_fullsized":"2024-09-10T19:03:44.061214+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:02:30.080861+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:02:30.080861+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046029529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":636,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148231+0000","last_change":"2024-09-10T19:01:50.763426+0000","last_active":"2024-09-10T19:03:44.148231+0000","last_peered":"2024-09-10T19:03:44.148231+0000","last_clean":"2024-09-10T19:03:44.148231+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:03:44.148231+0000","last_undegraded":"2024-09-10T19:03:44.148231+0000","last_fullsized":"2024-09-10T19:03:44.148231+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:01:50.763213+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:01:50.763213+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049851260000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":583,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.144915+0000","last_change":"2024-09-10T19:02:41.581075+0000","last_active":"2024-09-10T19:03:44.144915+0000","last_peered":"2024-09-10T19:03:44.144915+0000","last_clean":"2024-09-10T19:03:44.144915+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:03:44.144915+0000","last_undegraded":"2024-09-10T19:03:44.144915+0000","last_fullsized":"2024-09-10T19:03:44.144915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:41.580822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:02:41.580822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:03.147769+0000","scrub_duration":0.0044399460000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":573,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.064325+0000","last_change":"2024-09-10T19:02:48.603415+0000","last_active":"2024-09-10T19:03:44.064325+0000","last_peered":"2024-09-10T19:03:44.064325+0000","last_clean":"2024-09-10T19:03:44.064325+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:03:44.064325+0000","last_undegraded":"2024-09-10T19:03:44.064325+0000","last_fullsized":"2024-09-10T19:03:44.064325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:48.603252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:02:48.603252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:55.641516+0000","scrub_duration":0.0033719000000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":493,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.105545+0000","last_change":"2024-09-10T19:01:44.826152+0000","last_active":"2024-09-10T19:03:44.105545+0000","last_peered":"2024-09-10T19:03:44.105545+0000","last_clean":"2024-09-10T19:03:44.105545+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:03:44.105545+0000","last_undegraded":"2024-09-10T19:03:44.105545+0000","last_fullsized":"2024-09-10T19:03:44.105545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:44.825961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:44.825961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032122380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":582,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.151898+0000","last_change":"2024-09-10T19:02:57.154343+0000","last_active":"2024-09-10T19:03:44.151898+0000","last_peered":"2024-09-10T19:03:44.151898+0000","last_clean":"2024-09-10T19:03:44.151898+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:03:44.151898+0000","last_undegraded":"2024-09-10T19:03:44.151898+0000","last_fullsized":"2024-09-10T19:03:44.151898+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:02:57.154169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:02:57.154169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:26.725162+0000","scrub_duration":0.0037032549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":594,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.062874+0000","last_change":"2024-09-10T19:01:36.833738+0000","last_active":"2024-09-10T19:03:44.062874+0000","last_peered":"2024-09-10T19:03:44.062874+0000","last_clean":"2024-09-10T19:03:44.062874+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:03:44.062874+0000","last_undegraded":"2024-09-10T19:03:44.062874+0000","last_fullsized":"2024-09-10T19:03:44.062874+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:01:36.833597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.833597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003904022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":571,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067253+0000","last_change":"2024-09-10T19:01:22.937789+0000","last_active":"2024-09-10T19:03:44.067253+0000","last_peered":"2024-09-10T19:03:44.067253+0000","last_clean":"2024-09-10T19:03:44.067253+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:03:44.067253+0000","last_undegraded":"2024-09-10T19:03:44.067253+0000","last_fullsized":"2024-09-10T19:03:44.067253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:22.937630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:22.937630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003520343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":782,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.106345+0000","last_change":"2024-09-10T19:02:36.884303+0000","last_active":"2024-09-10T19:03:44.106345+0000","last_peered":"2024-09-10T19:03:44.106345+0000","last_clean":"2024-09-10T19:03:44.106345+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:03:44.106345+0000","last_undegraded":"2024-09-10T19:03:44.106345+0000","last_fullsized":"2024-09-10T19:03:44.106345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:02:36.884134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:02:36.884134+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:03:44.957096+0000","scrub_duration":0.0042044819999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":493,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.106027+0000","last_change":"2024-09-10T19:01:30.706798+0000","last_active":"2024-09-10T19:03:44.106027+0000","last_peered":"2024-09-10T19:03:44.106027+0000","last_clean":"2024-09-10T19:03:44.106027+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:03:44.106027+0000","last_undegraded":"2024-09-10T19:03:44.106027+0000","last_fullsized":"2024-09-10T19:03:44.106027+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:30.706647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:01:30.706647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003580083,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":584,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148383+0000","last_change":"2024-09-10T19:03:34.088071+0000","last_active":"2024-09-10T19:03:44.148383+0000","last_peered":"2024-09-10T19:03:44.148383+0000","last_clean":"2024-09-10T19:03:44.148383+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:03:44.148383+0000","last_undegraded":"2024-09-10T19:03:44.148383+0000","last_fullsized":"2024-09-10T19:03:44.148383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:34.087819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:03:34.087819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:48.507340+0000","scrub_duration":0.0034220600000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":585,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.104096+0000","last_change":"2024-09-10T19:01:36.753924+0000","last_active":"2024-09-10T19:03:44.104096+0000","last_peered":"2024-09-10T19:03:44.104096+0000","last_clean":"2024-09-10T19:03:44.104096+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:03:44.104096+0000","last_undegraded":"2024-09-10T19:03:44.104096+0000","last_fullsized":"2024-09-10T19:03:44.104096+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:01:36.753746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:01:36.753746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044503609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":588,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.063755+0000","last_change":"2024-09-10T19:02:42.658220+0000","last_active":"2024-09-10T19:03:44.063755+0000","last_peered":"2024-09-10T19:03:44.063755+0000","last_clean":"2024-09-10T19:03:44.063755+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:03:44.063755+0000","last_undegraded":"2024-09-10T19:03:44.063755+0000","last_fullsized":"2024-09-10T19:03:44.063755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:02:42.658114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:42.658114+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:03.187692+0000","scrub_duration":0.0033964870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":603,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.149608+0000","last_change":"2024-09-10T19:03:39.631553+0000","last_active":"2024-09-10T19:03:44.149608+0000","last_peered":"2024-09-10T19:03:44.149608+0000","last_clean":"2024-09-10T19:03:44.149608+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:03:44.149608+0000","last_undegraded":"2024-09-10T19:03:44.149608+0000","last_fullsized":"2024-09-10T19:03:44.149608+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:39.631389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:56.652676+0000","last_clean_scrub_stamp":"2024-09-10T19:03:39.631389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:54.008721+0000","scrub_duration":0.0032340799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":500,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148329+0000","last_change":"2024-09-10T19:03:38.924394+0000","last_active":"2024-09-10T19:03:44.148329+0000","last_peered":"2024-09-10T19:03:44.148329+0000","last_clean":"2024-09-10T19:03:44.148329+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:03:44.148329+0000","last_undegraded":"2024-09-10T19:03:44.148329+0000","last_fullsized":"2024-09-10T19:03:44.148329+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:38.924286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:03:38.924286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:55.230465+0000","scrub_duration":0.0097366650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":575,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.068873+0000","last_change":"2024-09-10T19:03:39.107941+0000","last_active":"2024-09-10T19:03:44.068873+0000","last_peered":"2024-09-10T19:03:44.068873+0000","last_clean":"2024-09-10T19:03:44.068873+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:03:44.068873+0000","last_undegraded":"2024-09-10T19:03:44.068873+0000","last_fullsized":"2024-09-10T19:03:44.068873+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:39.107862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:03:39.107862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:50.992773+0000","scrub_duration":0.0028008949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":581,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067264+0000","last_change":"2024-09-10T19:03:42.125422+0000","last_active":"2024-09-10T19:03:44.067264+0000","last_peered":"2024-09-10T19:03:44.067264+0000","last_clean":"2024-09-10T19:03:44.067264+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:03:44.067264+0000","last_undegraded":"2024-09-10T19:03:44.067264+0000","last_fullsized":"2024-09-10T19:03:44.067264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:03:42.125265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:03:42.125265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:05:10.580347+0000","scrub_duration":0.003293168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":613,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.148009+0000","last_change":"2024-09-10T19:02:49.375248+0000","last_active":"2024-09-10T19:03:44.148009+0000","last_peered":"2024-09-10T19:03:44.148009+0000","last_clean":"2024-09-10T19:03:44.148009+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:03:44.148009+0000","last_undegraded":"2024-09-10T19:03:44.148009+0000","last_fullsized":"2024-09-10T19:03:44.148009+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:02:49.375074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:02:49.375074+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:04:15.090445+0000","scrub_duration":0.0047769680000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":567,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.067656+0000","last_change":"2024-09-10T19:01:51.936609+0000","last_active":"2024-09-10T19:03:44.067656+0000","last_peered":"2024-09-10T19:03:44.067656+0000","last_clean":"2024-09-10T19:03:44.067656+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:03:44.067656+0000","last_undegraded":"2024-09-10T19:03:44.067656+0000","last_fullsized":"2024-09-10T19:03:44.067656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:01:51.936453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036318919999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":493,"reported_epoch":581,"state":"active+clean","last_fresh":"2024-09-10T19:03:44.104480+0000","last_change":"2024-09-10T19:01:37.783455+0000","last_active":"2024-09-10T19:03:44.104480+0000","last_peered":"2024-09-10T19:03:44.104480+0000","last_clean":"2024-09-10T19:03:44.104480+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:03:44.104480+0000","last_undegraded":"2024-09-10T19:03:44.104480+0000","last_fullsized":"2024-09-10T19:03:44.104480+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:37.783302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:01:37.783302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003418712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":233,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":8213609,"num_objects":200,"num_object_clones":16,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":216,"num_read_kb":62,"num_write":791,"num_write_kb":12820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26972160,"data_stored":24640926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2108,"ondisk_log_size":2108,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":842,"num_read_kb":837,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1128,"num_read_kb":885,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":370028,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":40,"num_write":21,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":435020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972537,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326844,"kb_used_data":16996,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93418308,"statfs":{"total":95995035648,"available":95660347392,"internally_reserved":0,"allocated":17403904,"data_stored":13635692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.90500000000000003,"15min":0.90100000000000002},"min":{"1min":0.65000000000000002,"5min":0.40300000000000002,"15min":0.311},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.9330000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.93000000000000005,"15min":0.93100000000000005},"min":{"1min":0.56599999999999995,"5min":0.498,"15min":0.34699999999999998},"max":{"1min":1.25,"5min":2.242,"15min":4.7000000000000002},"last":1.111}]},{"osd":1,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.86899999999999999,"15min":0.88600000000000001},"min":{"1min":0.60799999999999998,"5min":0.52400000000000002,"15min":0.41199999999999998},"max":{"1min":1.1140000000000001,"5min":1.208,"15min":5.6539999999999999},"last":0.873},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.90800000000000003,"15min":0.96799999999999997},"min":{"1min":0.59899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.133,"5min":1.421,"15min":1.8280000000000001},"last":0.69699999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":1.286,"15min":1.1080000000000001},"min":{"1min":0.66300000000000003,"5min":0.45000000000000001,"15min":0.41499999999999998},"max":{"1min":31.166,"5min":31.166,"15min":31.166},"last":1.069},{"interface":"front","average":{"1min":2.2949999999999999,"5min":1.26,"15min":1.081},"min":{"1min":0.72599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":27.709,"5min":27.709,"15min":27.709},"last":0.93999999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":1.0229999999999999,"15min":1.0149999999999999},"min":{"1min":0.79000000000000004,"5min":0.65400000000000003,"15min":0.48299999999999998},"max":{"1min":1.1479999999999999,"5min":2.3879999999999999,"15min":4.2080000000000002},"last":1.236},{"interface":"front","average":{"1min":1.008,"5min":0.95399999999999996,"15min":0.97899999999999998},"min":{"1min":0.71599999999999997,"5min":0.56200000000000006,"15min":0.46600000000000003},"max":{"1min":1.306,"5min":1.833,"15min":5.6050000000000004},"last":1.014}]},{"osd":4,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.03,"15min":1.016},"min":{"1min":0.64200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":2.1989999999999998},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.093,"5min":1.071,"15min":1.071},"min":{"1min":0.74099999999999999,"5min":0.49199999999999999,"15min":0.48099999999999998},"max":{"1min":1.607,"5min":2.4470000000000001,"15min":4.2400000000000002},"last":1.3029999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.95699999999999996,"15min":0.99099999999999999},"min":{"1min":0.60099999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.3109999999999999,"5min":3.1970000000000001,"15min":5.4710000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.169,"5min":1.1379999999999999,"15min":1.1539999999999999},"min":{"1min":0.74199999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.6599999999999999,"5min":3.8119999999999998,"15min":3.8119999999999998},"last":1.1499999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.161,"15min":1.196},"min":{"1min":0.65700000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.732,"5min":1.732,"15min":2.2749999999999999},"last":1.21},{"interface":"front","average":{"1min":1.175,"5min":1.204,"15min":1.1990000000000001},"min":{"1min":0.85999999999999999,"5min":0.58099999999999996,"15min":0.53100000000000003},"max":{"1min":1.4890000000000001,"5min":2.96,"15min":4.1310000000000002},"last":1.3540000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299587,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318396,"kb_used_data":8548,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93426756,"statfs":{"total":95995035648,"available":95668998144,"internally_reserved":0,"allocated":8753152,"data_stored":5110873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.88900000000000001,"15min":0.88300000000000001},"min":{"1min":0.52700000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.337,"5min":1.8839999999999999,"15min":2.0550000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.98399999999999999,"15min":0.95499999999999996},"min":{"1min":0.65600000000000003,"5min":0.40300000000000002,"15min":0.36399999999999999},"max":{"1min":1.1359999999999999,"5min":5.7309999999999999,"15min":6.8959999999999999},"last":0.97299999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.997,"15min":1.04},"min":{"1min":0.47999999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.266,"5min":2.2410000000000001,"15min":18.494},"last":1.075},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.90900000000000003,"15min":0.90100000000000002},"min":{"1min":0.55100000000000005,"5min":0.434,"15min":0.434},"max":{"1min":1.1279999999999999,"5min":1.7689999999999999,"15min":4.0229999999999997},"last":0.74099999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.94699999999999995,"15min":0.97199999999999998},"min":{"1min":0.496,"5min":0.47499999999999998,"15min":0.35299999999999998},"max":{"1min":1.2330000000000001,"5min":1.3440000000000001,"15min":9.4800000000000004},"last":1.01},{"interface":"front","average":{"1min":0.873,"5min":0.97499999999999998,"15min":0.93899999999999995},"min":{"1min":0.58799999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.3300000000000001,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":0.89500000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.0169999999999999,"15min":1.006},"min":{"1min":0.46700000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3640000000000001,"5min":2.1949999999999998,"15min":2.278},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.99199999999999999,"15min":0.97999999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.115,"5min":2.7090000000000001,"15min":3.4279999999999999},"last":0.83799999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":1.008,"15min":0.95899999999999996},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52000000000000002},"max":{"1min":1.2649999999999999,"5min":6.008,"15min":6.008},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.083,"15min":1.0609999999999999},"min":{"1min":0.52400000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.52,"5min":2.3039999999999998,"15min":2.3079999999999998},"last":1.1779999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.1180000000000001,"15min":1.0529999999999999},"min":{"1min":0.59399999999999997,"5min":0.57399999999999995,"15min":0.46300000000000002},"max":{"1min":2.02,"5min":5.9560000000000004,"15min":5.9560000000000004},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.1479999999999999,"15min":1.1299999999999999},"min":{"1min":0.60499999999999998,"5min":0.59499999999999997,"15min":0.46500000000000002},"max":{"1min":1.492,"5min":3.5169999999999999,"15min":3.5169999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.056,"15min":1.0680000000000001},"min":{"1min":0.55600000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":2.069},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.2490000000000001,"15min":1.1910000000000001},"min":{"1min":0.67000000000000004,"5min":0.65500000000000003,"15min":0.63900000000000001},"max":{"1min":1.7450000000000001,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.333}]}]},{"osd":5,"up_from":43,"seq":184683593933,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319024,"kb_used_data":13336,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93426128,"statfs":{"total":95995035648,"available":95668355072,"internally_reserved":0,"allocated":13656064,"data_stored":9900296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:03:18 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.93300000000000005,"15min":0.98199999999999998},"min":{"1min":0.65300000000000002,"5min":0.52900000000000003,"15min":0.36399999999999999},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":8.3949999999999996},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.879,"15min":0.91900000000000004},"min":{"1min":0.53100000000000003,"5min":0.50600000000000001,"15min":0.39700000000000002},"max":{"1min":1.23,"5min":1.3160000000000001,"15min":7.3070000000000004},"last":0.77900000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.94599999999999995,"15min":1.1659999999999999},"min":{"1min":0.64000000000000001,"5min":0.55600000000000005,"15min":0.41199999999999998},"max":{"1min":1.389,"5min":1.516,"15min":31.548999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.90100000000000002,"15min":1.0880000000000001},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.48899999999999999},"max":{"1min":1.325,"5min":1.325,"15min":31.847000000000001},"last":1.0369999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:03:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.94499999999999995,"15min":1.262},"min":{"1min":0.73799999999999999,"5min":0.59299999999999997,"15min":0.39900000000000002},"max":{"1min":1.298,"5min":1.4159999999999999,"15min":61.347000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.96399999999999997,"15min":1.0209999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.23,"5min":1.6140000000000001,"15min":6.1600000000000001},"last":0.95899999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.97499999999999998,"15min":1.0209999999999999},"min":{"1min":0.71399999999999997,"5min":0.58799999999999997,"15min":0.51700000000000002},"max":{"1min":1.206,"5min":1.482,"15min":7.5979999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0209999999999999,"15min":1.2729999999999999},"min":{"1min":0.61599999999999999,"5min":0.60799999999999998,"15min":0.57399999999999995},"max":{"1min":1.4019999999999999,"5min":1.5109999999999999,"15min":53.173000000000002},"last":1.2569999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.91000000000000003,"15min":0.98599999999999999},"min":{"1min":0.61899999999999999,"5min":0.52500000000000002,"15min":0.36499999999999999},"max":{"1min":1.3859999999999999,"5min":1.4399999999999999,"15min":13.005000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.03,"5min":0.99199999999999999,"15min":1.056},"min":{"1min":0.66900000000000004,"5min":0.53200000000000003,"15min":0.40600000000000003},"max":{"1min":1.546,"5min":1.546,"15min":7.6299999999999999},"last":1.1699999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:03:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.103,"15min":1.167},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.49199999999999999},"max":{"1min":1.3420000000000001,"5min":5.2009999999999996,"15min":7.7199999999999998},"last":1.377},{"interface":"front","average":{"1min":1.202,"5min":1.1659999999999999,"15min":1.214},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1619999999999999,"5min":5.1769999999999996,"15min":9.6760000000000002},"last":1.1399999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:03:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.1100000000000001,"15min":1.2110000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.43099999999999999},"max":{"1min":1.4750000000000001,"5min":5.7210000000000001,"15min":15.613},"last":1.343},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1539999999999999,"15min":1.2609999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.456,"5min":4.2480000000000002,"15min":16.965},"last":1.464}]}]},{"osd":4,"up_from":35,"seq":150323855577,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305932,"kb_used_data":8564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93439220,"statfs":{"total":95995035648,"available":95681761280,"internally_reserved":0,"allocated":8769536,"data_stored":5110253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:03:28 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.88700000000000001,"15min":0.85399999999999998},"min":{"1min":0.53300000000000003,"5min":0.371,"15min":0.371},"max":{"1min":2.0179999999999998,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":1.018},{"interface":"front","average":{"1min":1.014,"5min":0.96199999999999997,"15min":0.92800000000000005},"min":{"1min":0.68100000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.5620000000000001,"5min":1.6100000000000001,"15min":1.7070000000000001},"last":0.94899999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:03:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.1859999999999999,"15min":0.97899999999999998},"min":{"1min":0.68400000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.2749999999999999,"5min":27.716000000000001,"15min":27.716000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.242,"15min":1.032},"min":{"1min":0.622,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.282,"5min":27.93,"15min":27.93},"last":1.3280000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:03:28 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.179,"15min":1.0289999999999999},"min":{"1min":0.74099999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.4850000000000001,"5min":18.853000000000002,"15min":18.853000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.054,"5min":1.266,"15min":1.0529999999999999},"min":{"1min":0.81999999999999995,"5min":0.56599999999999995,"15min":0.497},"max":{"1min":1.4710000000000001,"5min":25.652999999999999,"15min":25.652999999999999},"last":0.84699999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:03:28 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0529999999999999,"15min":0.98399999999999999},"min":{"1min":0.67400000000000004,"5min":0.54900000000000004,"15min":0.46300000000000002},"max":{"1min":1.923,"5min":4.2670000000000003,"15min":4.2670000000000003},"last":1.292},{"interface":"front","average":{"1min":1.0149999999999999,"5min":0.998,"15min":0.98299999999999998},"min":{"1min":0.69199999999999995,"5min":0.49099999999999999,"15min":0.442},"max":{"1min":1.3779999999999999,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":0.996}]},{"osd":5,"last update":"Tue Sep 10 19:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.98799999999999999,"15min":0.96099999999999997},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.6060000000000001,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.0900000000000001,"15min":1.0449999999999999},"min":{"1min":0.67300000000000004,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.407,"5min":1.5369999999999999,"15min":1.7909999999999999},"last":1.0860000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:03:13 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0529999999999999,"15min":1.03},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.419,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.2050000000000001,"15min":1.151},"min":{"1min":0.71199999999999997,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.679,"5min":2.9590000000000001,"15min":2.9590000000000001},"last":1.379}]},{"osd":7,"last update":"Tue Sep 10 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.175,"15min":1.1599999999999999},"min":{"1min":0.66500000000000004,"5min":0.61799999999999999,"15min":0.46899999999999997},"max":{"1min":1.7,"5min":1.7,"15min":2.008},"last":1.228},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1850000000000001,"15min":1.1659999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.675,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":1.0569999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051812,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314944,"kb_used_data":9256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430208,"statfs":{"total":95995035648,"available":95672532992,"internally_reserved":0,"allocated":9478144,"data_stored":5702452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:03:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65700000000000003,"15min":0.83499999999999996},"min":{"1min":0.33400000000000002,"5min":0.29499999999999998,"15min":0.222},"max":{"1min":1.093,"5min":1.093,"15min":26.594000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.89100000000000001},"min":{"1min":0.45200000000000001,"5min":0.34899999999999998,"15min":0.252},"max":{"1min":0.96699999999999997,"5min":1.444,"15min":24.204999999999998},"last":0.77700000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:03:34 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67200000000000004,"15min":0.91300000000000003},"min":{"1min":0.36399999999999999,"5min":0.27800000000000002,"15min":0.26500000000000001},"max":{"1min":0.98199999999999998,"5min":1.6200000000000001,"15min":53.661999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.79500000000000004,"15min":0.89500000000000002},"min":{"1min":0.56499999999999995,"5min":0.40200000000000002,"15min":0.27900000000000003},"max":{"1min":1.119,"5min":2.181,"15min":15.189},"last":0.92700000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:03:34 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84599999999999997,"15min":1.0169999999999999},"min":{"1min":0.57499999999999996,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":1.1919999999999999,"5min":1.397,"15min":26.558},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81399999999999995,"15min":0.95299999999999996},"min":{"1min":0.60399999999999998,"5min":0.32600000000000001,"15min":0.26900000000000002},"max":{"1min":1.1299999999999999,"5min":4.4740000000000002,"15min":25.536999999999999},"last":0.89900000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:03:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0720000000000001,"15min":1.1599999999999999},"min":{"1min":0.75900000000000001,"5min":0.63200000000000001,"15min":0.46999999999999997},"max":{"1min":1.3580000000000001,"5min":1.663,"15min":10.359999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1870000000000001,"15min":1.26},"min":{"1min":0.875,"5min":0.64400000000000002,"15min":0.53500000000000003},"max":{"1min":1.6419999999999999,"5min":1.958,"15min":10.391},"last":1.498}]},{"osd":5,"last update":"Tue Sep 10 19:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1259999999999999,"15min":1.2250000000000001},"min":{"1min":0.68799999999999994,"5min":0.62,"15min":0.501},"max":{"1min":1.5900000000000001,"5min":1.8009999999999999,"15min":17.535},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.995,"5min":1.038,"15min":1.054},"min":{"1min":0.82099999999999995,"5min":0.63500000000000001,"15min":0.45900000000000002},"max":{"1min":1.2809999999999999,"5min":4.6619999999999999,"15min":8.7880000000000003},"last":1.4490000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:03:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.141,"15min":1.1970000000000001},"min":{"1min":0.88900000000000001,"5min":0.59499999999999997,"15min":0.55600000000000005},"max":{"1min":1.581,"5min":1.9159999999999999,"15min":10.51},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0900000000000001,"15min":1.101},"min":{"1min":0.76600000000000001,"5min":0.56999999999999995,"15min":0.48499999999999999},"max":{"1min":1.4370000000000001,"5min":2.996,"15min":10.773999999999999},"last":1.105}]},{"osd":7,"last update":"Tue Sep 10 19:03:01 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.087,"15min":1.1699999999999999},"min":{"1min":0.82399999999999995,"5min":0.59899999999999998,"15min":0.45600000000000002},"max":{"1min":1.3979999999999999,"5min":4.0229999999999997,"15min":18.312999999999999},"last":1.294},{"interface":"front","average":{"1min":1.284,"5min":1.2769999999999999,"15min":1.3169999999999999},"min":{"1min":0.79800000000000004,"5min":0.70099999999999996,"15min":0.57299999999999995},"max":{"1min":1.5980000000000001,"5min":4.5350000000000001,"15min":10.611000000000001},"last":1.754}]}]},{"osd":2,"up_from":19,"seq":81604378861,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314292,"kb_used_data":16924,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93430860,"statfs":{"total":95995035648,"available":95673200640,"internally_reserved":0,"allocated":17330176,"data_stored":13574298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.68500000000000005,"15min":0.74199999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.219},"max":{"1min":0.97599999999999998,"5min":1.2749999999999999,"15min":9.516},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.012,"5min":0.80600000000000005,"15min":0.80700000000000005},"min":{"1min":0.38400000000000001,"5min":0.30299999999999999,"15min":0.23599999999999999},"max":{"1min":6.1139999999999999,"5min":6.1139999999999999,"15min":15.577999999999999},"last":1.123}]},{"osd":1,"last update":"Tue Sep 10 19:02:53 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.26600000000000001},"max":{"1min":3.524,"5min":3.524,"15min":5.516},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.82399999999999995},"min":{"1min":0.45500000000000002,"5min":0.41599999999999998,"15min":0.28299999999999997},"max":{"1min":1.0760000000000001,"5min":1.121,"15min":9.0649999999999995},"last":0.82099999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":5.1479999999999997},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":0.91200000000000003,"15min":0.89000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":9.8219999999999992,"5min":9.8219999999999992,"15min":15.736000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:03:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.121,"15min":1.107},"min":{"1min":0.495,"5min":0.49299999999999999,"15min":0.39300000000000002},"max":{"1min":6.556,"5min":6.556,"15min":21.571999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.1950000000000001,"15min":1.153},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.367,"5min":10.881,"15min":10.881},"last":0.96099999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:03:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0800000000000001,"15min":1.0880000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.38500000000000001},"max":{"1min":1.516,"5min":1.8340000000000001,"15min":5.1959999999999997},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.153,"15min":1.1459999999999999},"min":{"1min":0.65600000000000003,"5min":0.59999999999999998,"15min":0.45400000000000001},"max":{"1min":6.0620000000000003,"5min":6.0620000000000003,"15min":21.654},"last":1.0780000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:03:15 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.121,"15min":1.1539999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.503},"max":{"1min":1.5620000000000001,"5min":1.8280000000000001,"15min":9.0389999999999997},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.0980000000000001,"15min":1.1499999999999999},"min":{"1min":0.624,"5min":0.624,"15min":0.505},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":15.971},"last":1.044}]},{"osd":7,"last update":"Tue Sep 10 19:03:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.204,"15min":1.1930000000000001},"min":{"1min":0.67800000000000005,"5min":0.59699999999999998,"15min":0.48399999999999999},"max":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":15.025},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.2529999999999999,"15min":1.2250000000000001},"min":{"1min":0.72999999999999998,"5min":0.56499999999999995,"15min":0.53100000000000003},"max":{"1min":6.048,"5min":6.851,"15min":14.816000000000001},"last":0.98999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575096,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322332,"kb_used_data":12492,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93422820,"statfs":{"total":95995035648,"available":95664967680,"internally_reserved":0,"allocated":12791808,"data_stored":9107792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:03:44 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.74399999999999999,"15min":0.82999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32700000000000001,"15min":0.23499999999999999},"max":{"1min":0.92300000000000004,"5min":5.0439999999999996,"15min":32.677999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78300000000000003,"15min":0.89900000000000002},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.253},"max":{"1min":1.575,"5min":1.9199999999999999,"15min":32.398000000000003},"last":0.60499999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:03:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.79000000000000004,"15min":0.89100000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":1.0189999999999999,"5min":5.7999999999999998,"15min":26.097000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.83199999999999996,"15min":1.008},"min":{"1min":0.437,"5min":0.39400000000000002,"15min":0.26000000000000001},"max":{"1min":1.4970000000000001,"5min":2.7629999999999999,"15min":32.363},"last":0.92400000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:03:35 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.99299999999999999,"15min":1.038},"min":{"1min":0.309,"5min":0.309,"15min":0.28899999999999998},"max":{"1min":2.27,"5min":2.8359999999999999,"15min":25.704000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.82999999999999996,"15min":1.0529999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.26600000000000001},"max":{"1min":1.002,"5min":3.8029999999999999,"15min":60.999000000000002},"last":0.874}]},{"osd":4,"last update":"Tue Sep 10 19:03:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.119,"15min":1.131},"min":{"1min":0.68899999999999995,"5min":0.67300000000000004,"15min":0.40899999999999997},"max":{"1min":1.7270000000000001,"5min":2.0760000000000001,"15min":12.808999999999999},"last":1.145},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.0209999999999999,"15min":1.2529999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.378},"max":{"1min":1.3,"5min":3.6779999999999999,"15min":60.529000000000003},"last":1.1120000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:03:09 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0960000000000001,"15min":1.129},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.38},"max":{"1min":1.651,"5min":5.2439999999999998,"15min":26.050999999999998},"last":1},{"interface":"front","average":{"1min":1.054,"5min":1.1879999999999999,"15min":1.2090000000000001},"min":{"1min":0.746,"5min":0.53200000000000003,"15min":0.49299999999999999},"max":{"1min":1.518,"5min":2.7949999999999999,"15min":12.84},"last":1.2769999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:03:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.236,"15min":1.3029999999999999},"min":{"1min":0.76500000000000001,"5min":0.67900000000000005,"15min":0.53800000000000003},"max":{"1min":1.542,"5min":3.121,"15min":25.608000000000001},"last":1.472},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.145,"15min":1.25},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.41499999999999998},"max":{"1min":1.4570000000000001,"5min":3.7759999999999998,"15min":33.643000000000001},"last":1.1970000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:03:09 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.288,"15min":1.333},"min":{"1min":0.76800000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.8380000000000001,"5min":3.0880000000000001,"15min":25.565999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1799999999999999,"15min":1.3129999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.48399999999999999},"max":{"1min":1.379,"5min":3.7370000000000001,"15min":33.673999999999999},"last":1.3300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738628,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314296,"kb_used_data":8608,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93430856,"statfs":{"total":95995035648,"available":95673196544,"internally_reserved":0,"allocated":8814592,"data_stored":5110875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.76800000000000002,"15min":1.016},"min":{"1min":0.53900000000000003,"5min":0.32400000000000001,"15min":0.26300000000000001},"max":{"1min":1.613,"5min":6.758,"15min":80.710999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.82699999999999996,"15min":1.103},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.25600000000000001},"max":{"1min":1.0660000000000001,"5min":3.6850000000000001,"15min":86.009},"last":0.97599999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.78800000000000003,"15min":1.052},"min":{"1min":0.54500000000000004,"5min":0.36799999999999999,"15min":0.216},"max":{"1min":1.1830000000000001,"5min":1.593,"15min":36.682000000000002},"last":0.87},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.85199999999999998,"15min":0.95199999999999996},"min":{"1min":0.54100000000000004,"5min":0.442,"15min":0.23599999999999999},"max":{"1min":1.028,"5min":3.2090000000000001,"15min":24.614000000000001},"last":0.80500000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:03:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.89000000000000001,"15min":0.93700000000000006},"min":{"1min":0.60699999999999998,"5min":0.48799999999999999,"15min":0.26400000000000001},"max":{"1min":1.22,"5min":4.4749999999999996,"15min":21.620999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.86699999999999999,"15min":1.0640000000000001},"min":{"1min":0.67800000000000005,"5min":0.42499999999999999,"15min":0.30299999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":45.765000000000001},"last":1.0600000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:03:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.099,"15min":1.087},"min":{"1min":0.81200000000000006,"5min":0.64300000000000002,"15min":0.5},"max":{"1min":2.4239999999999999,"5min":2.5190000000000001,"15min":6.3630000000000004},"last":1.611},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1259999999999999,"15min":1.129},"min":{"1min":0.89600000000000002,"5min":0.61399999999999999,"15min":0.51100000000000001},"max":{"1min":1.4630000000000001,"5min":1.6870000000000001,"15min":4.2389999999999999},"last":1.6910000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.093,"15min":1.145},"min":{"1min":0.82199999999999995,"5min":0.67800000000000005,"15min":0.442},"max":{"1min":1.4670000000000001,"5min":1.6279999999999999,"15min":10.048},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.145,"15min":1.121},"min":{"1min":0.97399999999999998,"5min":0.63,"15min":0.42999999999999999},"max":{"1min":1.431,"5min":2.0779999999999998,"15min":3.8959999999999999},"last":1.3120000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:03:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.1799999999999999,"15min":1.161},"min":{"1min":0.752,"5min":0.752,"15min":0.56899999999999995},"max":{"1min":1.5549999999999999,"5min":1.899,"15min":3.9550000000000001},"last":1.502},{"interface":"front","average":{"1min":1.248,"5min":1.1819999999999999,"15min":1.22},"min":{"1min":0.96099999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.804,"5min":1.8149999999999999,"15min":11.894},"last":1.546}]},{"osd":7,"last update":"Tue Sep 10 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.2170000000000001,"15min":1.1830000000000001},"min":{"1min":0.98499999999999999,"5min":0.69699999999999995,"15min":0.48499999999999999},"max":{"1min":2.1200000000000001,"5min":2.468,"15min":6.2949999999999999},"last":1.4179999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.252,"15min":1.2769999999999999},"min":{"1min":0.82299999999999995,"5min":0.78400000000000003,"15min":0.54900000000000004},"max":{"1min":2.0009999999999999,"5min":2.0129999999999999,"15min":8.2840000000000007},"last":1.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":6567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4003068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8527872,"data_stored":8206106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":6792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":4553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4542464,"data_stored":4205478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":6487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8531968,"data_stored":8201875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:03:51.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-10T19:03:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.343 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:51.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.773854+0000 osd.0 (osd.0) 317 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.781991+0000 osd.0 (osd.0) 318 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.967141+0000 osd.4 (osd.4) 342 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:49.970950+0000 osd.4 (osd.4) 343 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.010796+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:03:51.353 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.014700+0000 osd.1 (osd.1) 326 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.159075+0000 osd.3 (osd.3) 340 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.172610+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.230928+0000 mgr.y (mgr.24526) 579 : cluster [DBG] pgmap v907: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.247143+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: audit 2024-09-10T19:03:50.256537+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: audit 2024-09-10T19:03:50.257599+0000 mon.a (mon.0) 2969 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]': finished 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.257711+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[18629]: audit 2024-09-10T19:03:50.259300+0000 mon.a (mon.0) 2971 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.773854+0000 osd.0 (osd.0) 317 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.781991+0000 osd.0 (osd.0) 318 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.967141+0000 osd.4 (osd.4) 342 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:03:51.354 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:49.970950+0000 osd.4 (osd.4) 343 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.010796+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.014700+0000 osd.1 (osd.1) 326 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.159075+0000 osd.3 (osd.3) 340 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.172610+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.230928+0000 mgr.y (mgr.24526) 579 : cluster [DBG] pgmap v907: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.247143+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: audit 2024-09-10T19:03:50.256537+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: audit 2024-09-10T19:03:50.257599+0000 mon.a (mon.0) 2969 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]': finished 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.257711+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:03:51.355 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:51 smithi083 bash[32381]: audit 2024-09-10T19:03:50.259300+0000 mon.a (mon.0) 2971 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.773854+0000 osd.0 (osd.0) 317 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.781991+0000 osd.0 (osd.0) 318 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.967141+0000 osd.4 (osd.4) 342 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:49.970950+0000 osd.4 (osd.4) 343 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.010796+0000 osd.1 (osd.1) 325 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.014700+0000 osd.1 (osd.1) 326 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.159075+0000 osd.3 (osd.3) 340 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.172610+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:03:51.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.230928+0000 mgr.y (mgr.24526) 579 : cluster [DBG] pgmap v907: 238 pgs: 32 unknown, 206 active+clean; 8.2 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.247143+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:03:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: audit 2024-09-10T19:03:50.256537+0000 mon.b (mon.1) 444 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: audit 2024-09-10T19:03:50.257599+0000 mon.a (mon.0) 2969 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-98", "mode": "writeback"}]': finished 2024-09-10T19:03:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.257711+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:03:51.365 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[22591]: audit 2024-09-10T19:03:50.259300+0000 mon.a (mon.0) 2971 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:51.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:51.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:51.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:51.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:51.674 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:03:51.714 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:51.726 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:03:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:51.749 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:51.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:51.888 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:51.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:51.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:51.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:52.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:52.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:52.166 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:52.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: audit 2024-09-10T19:03:50.460538+0000 mgr.y (mgr.24526) 580 : audit [DBG] from='client.25539 v1:172.21.15.83:0/138250730' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.895580+0000 osd.6 (osd.6) 281 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: cluster 2024-09-10T19:03:50.898628+0000 osd.6 (osd.6) 282 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: audit 2024-09-10T19:03:51.259548+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: audit 2024-09-10T19:03:51.260113+0000 mon.a (mon.0) 2972 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:52.278 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: cluster 2024-09-10T19:03:51.260330+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:03:52.279 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:52 smithi165 bash[22591]: audit 2024-09-10T19:03:51.262231+0000 mon.a (mon.0) 2974 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: audit 2024-09-10T19:03:50.460538+0000 mgr.y (mgr.24526) 580 : audit [DBG] from='client.25539 v1:172.21.15.83:0/138250730' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:52.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.895580+0000 osd.6 (osd.6) 281 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: cluster 2024-09-10T19:03:50.898628+0000 osd.6 (osd.6) 282 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: audit 2024-09-10T19:03:51.259548+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: audit 2024-09-10T19:03:51.260113+0000 mon.a (mon.0) 2972 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: cluster 2024-09-10T19:03:51.260330+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18629]: audit 2024-09-10T19:03:51.262231+0000 mon.a (mon.0) 2974 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: audit 2024-09-10T19:03:50.460538+0000 mgr.y (mgr.24526) 580 : audit [DBG] from='client.25539 v1:172.21.15.83:0/138250730' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.895580+0000 osd.6 (osd.6) 281 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:03:52.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: cluster 2024-09-10T19:03:50.898628+0000 osd.6 (osd.6) 282 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:03:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: audit 2024-09-10T19:03:51.259548+0000 mon.b (mon.1) 445 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: audit 2024-09-10T19:03:51.260113+0000 mon.a (mon.0) 2972 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: cluster 2024-09-10T19:03:51.260330+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:03:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[32381]: audit 2024-09-10T19:03:51.262231+0000 mon.a (mon.0) 2974 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:03:52.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:52.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:52.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:03:52.433 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:52.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:52.566 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:52.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:52.699 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:52.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:52.836 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:52.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:52.951 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:03:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:03:52] "GET /metrics HTTP/1.1" 200 228996 "" "Prometheus/2.43.0" 2024-09-10T19:03:52.972 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:52.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:52.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:53.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:53.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:53.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:53.249 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:53.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:51.942333+0000 osd.6 (osd.6) 283 : cluster [DBG] 12.15 deep-scrub starts 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:51.945971+0000 osd.6 (osd.6) 284 : cluster [DBG] 12.15 deep-scrub ok 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.010760+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.016114+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.028906+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.032096+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.231733+0000 mgr.y (mgr.24526) 581 : cluster [DBG] pgmap v910: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.252500+0000 mon.a (mon.0) 2975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: audit 2024-09-10T19:03:52.262423+0000 mon.a (mon.0) 2976 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.262630+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:03:53.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: audit 2024-09-10T19:03:52.264581+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[18629]: audit 2024-09-10T19:03:52.267384+0000 mon.a (mon.0) 2978 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:51.942333+0000 osd.6 (osd.6) 283 : cluster [DBG] 12.15 deep-scrub starts 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:51.945971+0000 osd.6 (osd.6) 284 : cluster [DBG] 12.15 deep-scrub ok 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.010760+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.016114+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.028906+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.032096+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.231733+0000 mgr.y (mgr.24526) 581 : cluster [DBG] pgmap v910: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:53.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.252500+0000 mon.a (mon.0) 2975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:03:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: audit 2024-09-10T19:03:52.262423+0000 mon.a (mon.0) 2976 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:03:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.262630+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:03:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: audit 2024-09-10T19:03:52.264581+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:53 smithi083 bash[32381]: audit 2024-09-10T19:03:52.267384+0000 mon.a (mon.0) 2978 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:51.942333+0000 osd.6 (osd.6) 283 : cluster [DBG] 12.15 deep-scrub starts 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:51.945971+0000 osd.6 (osd.6) 284 : cluster [DBG] 12.15 deep-scrub ok 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.010760+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.016114+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.028906+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:03:53.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.032096+0000 osd.1 (osd.1) 328 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.231733+0000 mgr.y (mgr.24526) 581 : cluster [DBG] pgmap v910: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.252500+0000 mon.a (mon.0) 2975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: audit 2024-09-10T19:03:52.262423+0000 mon.a (mon.0) 2976 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.262630+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: audit 2024-09-10T19:03:52.264581+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:53 smithi165 bash[22591]: audit 2024-09-10T19:03:52.267384+0000 mon.a (mon.0) 2978 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:03:53.385 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:53.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:53.522 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:53.626 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:53.661 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:53.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:53.798 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:53.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:53.934 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:53.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:54.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:54.071 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:54.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.629418+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.629694+0000 osd.3 (osd.3) 343 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.963515+0000 osd.6 (osd.6) 285 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: cluster 2024-09-10T19:03:52.966550+0000 osd.6 (osd.6) 286 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: audit 2024-09-10T19:03:53.268125+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.188 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: audit 2024-09-10T19:03:53.268448+0000 mon.a (mon.0) 2979 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:03:54.189 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: cluster 2024-09-10T19:03:53.268565+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:03:54.189 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:54 smithi165 bash[22591]: audit 2024-09-10T19:03:53.272086+0000 mon.a (mon.0) 2981 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.209 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:54.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.629418+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.629694+0000 osd.3 (osd.3) 343 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.963515+0000 osd.6 (osd.6) 285 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: cluster 2024-09-10T19:03:52.966550+0000 osd.6 (osd.6) 286 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: audit 2024-09-10T19:03:53.268125+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: audit 2024-09-10T19:03:53.268448+0000 mon.a (mon.0) 2979 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: cluster 2024-09-10T19:03:53.268565+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[18629]: audit 2024-09-10T19:03:53.272086+0000 mon.a (mon.0) 2981 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.629418+0000 osd.3 (osd.3) 342 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.629694+0000 osd.3 (osd.3) 343 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:54.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.963515+0000 osd.6 (osd.6) 285 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:03:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: cluster 2024-09-10T19:03:52.966550+0000 osd.6 (osd.6) 286 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:03:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: audit 2024-09-10T19:03:53.268125+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: audit 2024-09-10T19:03:53.268448+0000 mon.a (mon.0) 2979 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:03:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: cluster 2024-09-10T19:03:53.268565+0000 mon.a (mon.0) 2980 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:03:54.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:54 smithi083 bash[32381]: audit 2024-09-10T19:03:53.272086+0000 mon.a (mon.0) 2981 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:03:54.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:54.346 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:54.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:54.483 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:54.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:54.618 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:54.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:54.855 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:54.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:54.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:54.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:54.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:55.026 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.28627.0:5 66.c 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-10T19:03:55.034 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:01.890647+0000", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 533.13724367199995, 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.10262373599999999, 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.28627", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4009038338", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890647+0000", 2024-09-10T19:03:55.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890647+0000", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890645+0000", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890655+0000", 2024-09-10T19:03:55.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0265000000000001e-05 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890667+0000", 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1912e-05 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.890676+0000", 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7560000000000005e-06 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898312+0000", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076363990000000003 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898418+0000", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105642 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.898609+0000", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019089899999999999 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.908199+0000", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095898189999999994 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.908363+0000", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016451399999999999 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993116+0000", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.084753172000000002 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993218+0000", 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102069 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:01.993271+0000", 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2924999999999999e-05 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.33280.0:2 72.6 72:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-09-10T19:03:55.040 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:03.454746+0000", 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 531.57314546299995, 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.079654984999999998, 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.33280", 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/2788138944", 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454746+0000", 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454746+0000", 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454743+0000", 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454753+0000", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0618999999999999e-05 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454767+0000", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3655000000000001e-05 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.454776+0000", 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7560000000000005e-06 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.470739+0000", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015963356000000001 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.470746+0000", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1609999999999997e-06 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.482837+0000", 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012090386999999999 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.482846+0000", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0299999999999999e-06 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520469+0000", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.037623707999999999 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520651+0000", 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018204199999999999 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.520847+0000", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000195231 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.521329+0000", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048234599999999998 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.521779+0000", 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00045026799999999998 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534312+0000", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012532585000000001 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534363+0000", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1341000000000002e-05 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:03.534401+0000", 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7416000000000002e-05 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.047 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.28627.0:9 66.c 66:37fc6798:test-rados-api-smithi083-77943-8::foo:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e149)", 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:04.450461+0000", 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 530.57742975300005, 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.073743234000000005, 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.28627", 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4009038338", 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 9 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.048 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450461+0000", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450461+0000", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450458+0000", 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T19:03:55.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450472+0000", 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4127e-05 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450504+0000", 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1578999999999997e-05 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450519+0000", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5812999999999999e-05 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450673+0000", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015369099999999999 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.450689+0000", 2024-09-10T19:03:55.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6147999999999999e-05 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471156+0000", 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020466640000000001 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471261+0000", 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104817 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:03:55.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.471420+0000", 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015986900000000001 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.479203+0000", 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077824490000000003 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.487328+0000", 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081247579999999993 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524074+0000", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036746094999999999 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524141+0000", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6767999999999995e-05 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.524205+0000", 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4077000000000003e-05 2024-09-10T19:03:55.054 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.35008.0:2 73.1f 73:fd8eaea7:nspace::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:04.484368+0000", 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 530.54352358300002, 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058343063000000001, 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.35008", 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/466281865", 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T19:03:55.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484368+0000", 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484368+0000", 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484365+0000", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484378+0000", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2665e-05 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484390+0000", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2615e-05 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484415+0000", 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4341999999999998e-05 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484466+0000", 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1864999999999999e-05 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.484475+0000", 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3219999999999993e-06 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510456+0000", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025981364 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510580+0000", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000123545 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.510696+0000", 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116555 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.511041+0000", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034510399999999999 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.540117+0000", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.029075515 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542572+0000", 2024-09-10T19:03:55.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024546239999999999 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542665+0000", 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3058000000000002e-05 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:04.542711+0000", 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6054999999999999e-05 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.061 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:224 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.737589+0000", 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 523.29030232900004, 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027100756, 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737589+0000", 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737589+0000", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.737584+0000", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746765+0000", 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091805819999999996 2024-09-10T19:03:55.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746788+0000", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2648000000000001e-05 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746795+0000", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8380000000000002e-06 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746877+0000", 2024-09-10T19:03:55.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1819999999999999e-05 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.746921+0000", 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3693000000000003e-05 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.764625+0000", 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017703855000000001 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.764690+0000", 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4727999999999994e-05 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:225 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.765592+0000", 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 523.26229873800003, 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.024732403, 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765592+0000", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765592+0000", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.765590+0000", 2024-09-10T19:03:55.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770186+0000", 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045959460000000001 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770237+0000", 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1035000000000002e-05 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.770249+0000", 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510999999999999e-05 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.771257+0000", 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010075220000000001 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.771286+0000", 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9459999999999999e-05 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790249+0000", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018962372000000002 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790325+0000", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5965999999999997e-05 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.4.0:226 11.1b e155/124 -13:dff2cc3b:::temp_11.1b_0_24259_1:head v 0'0, mlcod=0'0)", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.790974+0000", 2024-09-10T19:03:55.070 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 523.23691700799998, 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018381779000000001, 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790974+0000", 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790974+0000", 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.071 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.790972+0000", 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999981 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795203+0000", 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042301700000000001 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.072 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795236+0000", 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2966000000000003e-05 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795245+0000", 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8549999999999997e-06 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795347+0000", 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010155299999999999 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.795377+0000", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9929000000000001e-05 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.809306+0000", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013929182 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.809356+0000", 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9796999999999998e-05 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.074 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.6.0:339 11.3 e155/124 -13:c5d8c2df:::temp_11.3_0_24313_1:head v 0'0, mlcod=0'0)", 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.833266+0000", 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 523.19462542799999, 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021768810999999999, 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833266+0000", 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833266+0000", 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.833265+0000", 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838111+0000", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004845388 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838167+0000", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.643e-05 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838186+0000", 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9040000000000001e-05 2024-09-10T19:03:55.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838286+0000", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9642000000000006e-05 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.838324+0000", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8491999999999997e-05 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.854981+0000", 2024-09-10T19:03:55.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016656823000000001 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.855035+0000", 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3582000000000003e-05 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.6.0:340 11.3 e155/124 -13:c5d8c2df:::temp_11.3_0_24313_1:head v 0'0, mlcod=0'0)", 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:11.868202+0000", 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 523.15968926300002, 2024-09-10T19:03:55.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.02096164, 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868202+0000", 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868202+0000", 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.868198+0000", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873323+0000", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051244960000000001 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873366+0000", 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3547000000000002e-05 2024-09-10T19:03:55.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873386+0000", 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9375e-05 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873465+0000", 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9726000000000002e-05 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.873487+0000", 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2079e-05 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.889103+0000", 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015615891999999999 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:11.889164+0000", 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0149000000000003e-05 2024-09-10T19:03:55.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.24699.0:2 111.6 111:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:45.971302+0000", 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 489.05658899899998, 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.063178365, 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.24699", 2024-09-10T19:03:55.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1632292978", 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971302+0000", 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971302+0000", 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971299+0000", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971309+0000", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6299999999999993e-06 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971321+0000", 2024-09-10T19:03:55.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2330999999999999e-05 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971330+0000", 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9430000000000006e-06 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971392+0000", 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2025000000000003e-05 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.971399+0000", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7089999999999996e-06 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.992622+0000", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021222477 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:45.992630+0000", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3049999999999993e-06 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024190+0000", 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031560438000000003 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024336+0000", 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145344 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.024507+0000", 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017166499999999999 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.025077+0000", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056921300000000005 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.027024+0000", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00194736 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034341+0000", 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073167370000000002 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034401+0000", 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0884999999999997e-05 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:46.034481+0000", 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9021000000000001e-05 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.24732.0:2 118.11 118:8e20cd32:test-rados-api-smithi083-77788-26::foo:head [writefull 0~8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:03:55.091 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:55:53.983682+0000", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 481.04420907100001, 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.058042643999999997, 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.24732", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/4260234155", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983682+0000", 2024-09-10T19:03:55.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983682+0000", 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.983680+0000", 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995043+0000", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011363009 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995068+0000", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4885e-05 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995078+0000", 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0448999999999999e-05 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995241+0000", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016265999999999999 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995489+0000", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024785900000000001 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:53.995841+0000", 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035254999999999998 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.013679+0000", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017837493999999999 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.033265+0000", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019586587999999999 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041553+0000", 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082874439999999997 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041675+0000", 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012207 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:55:54.041725+0000", 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9957000000000003e-05 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.097 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:1538 141.3 141:c3978ec1:test-rados-api-smithi083-77828-18::bar:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e231)", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:27.038867+0000", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.98902405199999, 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.032955234999999999, 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1538 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038867+0000", 2024-09-10T19:03:55.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038867+0000", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038864+0000", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038873+0000", 2024-09-10T19:03:55.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0820000000000005e-06 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038888+0000", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.45e-05 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.038894+0000", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4840000000000001e-06 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.039184+0000", 2024-09-10T19:03:55.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028970999999999999 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.039192+0000", 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5950000000000001e-06 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.041143+0000", 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001951709 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.041357+0000", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021345700000000001 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.061665+0000", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020307819000000001 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.061814+0000", 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014969000000000001 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:27.071822+0000", 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010007920999999999 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.48947.0:2 151.1e 151:7cdf4dc8:::oid_XattrsRoundTripPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:40.085858+0000", 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 434.942033493, 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.028557097, 2024-09-10T19:03:55.103 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.48947", 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1872315293", 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085858+0000", 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085858+0000", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085855+0000", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085865+0000", 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0526000000000001e-05 2024-09-10T19:03:55.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085881+0000", 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5911999999999999e-05 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.085889+0000", 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7910000000000005e-06 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.091064+0000", 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051752680000000002 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.091074+0000", 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4010000000000003e-06 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112086+0000", 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021012005 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112224+0000", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013811800000000001 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112386+0000", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016208100000000001 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.112991+0000", 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00060498400000000003 2024-09-10T19:03:55.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114306+0000", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013147180000000001 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114352+0000", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6020000000000003e-05 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114388+0000", 2024-09-10T19:03:55.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6556e-05 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:40.114415+0000", 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6568e-05 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.48947.0:4 151.1e 151:7cdf4dc8:::oid_XattrsRoundTripPP_:head [setxattr attr1 (12) in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:56:42.091646+0000", 2024-09-10T19:03:55.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 432.93624471700002, 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.023530008000000002, 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.48947", 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1872315293", 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091646+0000", 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091646+0000", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091643+0000", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091656+0000", 2024-09-10T19:03:55.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2510000000000001e-05 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091680+0000", 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4394999999999999e-05 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091690+0000", 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9850000000000001e-06 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091784+0000", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4003000000000001e-05 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.091904+0000", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119968 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.092079+0000", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017448299999999999 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.092809+0000", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073068899999999997 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.108846+0000", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016036594000000001 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115065+0000", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062191290000000003 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115133+0000", 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8429000000000005e-05 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:56:42.115176+0000", 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3093999999999998e-05 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2189 64.1d 64:b8b5cecb:test-rados-api-smithi083-77828-28::foo:head [writefull 0~5] snapc 11=[11] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:08.065256+0000", 2024-09-10T19:03:55.116 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 406.96263490400003, 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020636341999999998, 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2189 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065256+0000", 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065256+0000", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065253+0000", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065267+0000", 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3886999999999999e-05 2024-09-10T19:03:55.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065284+0000", 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7597000000000001e-05 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065295+0000", 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0247e-05 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065387+0000", 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2594000000000001e-05 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.065399+0000", 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2310000000000001e-05 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074391+0000", 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089920130000000001 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074498+0000", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010673100000000001 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.074689+0000", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019112299999999999 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.077280+0000", 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025910450000000002 2024-09-10T19:03:55.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.078598+0000", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013178689999999999 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085819+0000", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072206199999999996 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085860+0000", 2024-09-10T19:03:55.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0979000000000002e-05 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:08.085893+0000", 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.273e-05 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2299 64.6 64:646b25c2:test-rados-api-smithi083-77828-30::foo:head [writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:17.494930+0000", 2024-09-10T19:03:55.123 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 397.532961382, 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021475062, 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2299 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494930+0000", 2024-09-10T19:03:55.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494930+0000", 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.494925+0000", 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501000+0000", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060746059999999998 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501046+0000", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6699000000000001e-05 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501063+0000", 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6350000000000001e-05 2024-09-10T19:03:55.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501166+0000", 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103325 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501335+0000", 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000168992 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.501499+0000", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016421 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.512384+0000", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010885018999999999 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.515420+0000", 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030353709999999998 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516115+0000", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069553900000000001 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516284+0000", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016939599999999999 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:17.516405+0000", 2024-09-10T19:03:55.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012037200000000001 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:2302 64.6 64:646b25c2:test-rados-api-smithi083-77828-30::foo:head [writefull 0~5] snapc 14=[14,13] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:57:19.038678+0000", 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 395.98921285900002, 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022007105999999999, 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:03:55.130 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2302 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038678+0000", 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038678+0000", 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038674+0000", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038695+0000", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1912999999999998e-05 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038747+0000", 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1486000000000001e-05 2024-09-10T19:03:55.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038769+0000", 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1739999999999999e-05 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038950+0000", 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018124100000000001 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.038977+0000", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6874000000000001e-05 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052474+0000", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013497055000000001 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052577+0000", 2024-09-10T19:03:55.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103589 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.052780+0000", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020221099999999999 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.054276+0000", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014968329999999999 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.055530+0000", 2024-09-10T19:03:55.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012535039999999999 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060595+0000", 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050655250000000004 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060644+0000", 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8114e-05 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:57:19.060685+0000", 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1786999999999998e-05 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.25170.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:58:24.092116+0000", 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 330.93577507399999, 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009413197, 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.137 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.25170", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/1833223569", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092116+0000", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092116+0000", 2024-09-10T19:03:55.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092114+0000", 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092122+0000", 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4670000000000004e-06 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092135+0000", 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.308e-05 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.092144+0000", 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3909999999999998e-06 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.093221+0000", 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010772430000000001 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:24.093225+0000", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1699999999999999e-06 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.097615+0000", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.004389607 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.097806+0000", 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019083199999999999 2024-09-10T19:03:55.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.098187+0000", 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0003814 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.098953+0000", 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00076624899999999999 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.101477+0000", 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025242659999999998 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:25.101529+0000", 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1836999999999999e-05 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.25176.0:1 195.6s0 195:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-10T19:03:55.143 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:58:27.093700+0000", 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 327.93419132100001, 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0292627759999999, 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.25176", 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3151444780", 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093700+0000", 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093700+0000", 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093695+0000", 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999952 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093714+0000", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9188999999999999e-05 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093743+0000", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8665000000000001e-05 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.093760+0000", 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7371e-05 2024-09-10T19:03:55.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.096045+0000", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002284861 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:27.096052+0000", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5370000000000003e-06 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121277+0000", 2024-09-10T19:03:55.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0252254999999999 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121413+0000", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013598500000000001 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.121815+0000", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040199300000000001 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122598+0000", 2024-09-10T19:03:55.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078350900000000005 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122881+0000", 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028243400000000003 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:58:28.122963+0000", 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1678e-05 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:5779 64.1d 64:bd59bd67:test-rados-api-smithi083-77828-54::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e425)", 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 246.715856403, 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019346334999999999, 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5779 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:03:55.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312032+0000", 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312044+0000", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2741999999999999e-05 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312067+0000", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2921999999999998e-05 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312077+0000", 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0118e-05 2024-09-10T19:03:55.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312150+0000", 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2853000000000002e-05 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312159+0000", 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3010000000000007e-06 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:03:55.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.319968+0000", 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078098359999999997 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.320183+0000", 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021412800000000001 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.320350+0000", 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016759300000000001 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.321096+0000", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000745544 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.324693+0000", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035975019999999998 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331284+0000", 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065907539999999999 2024-09-10T19:03:55.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331338+0000", 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3717000000000001e-05 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331381+0000", 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3377000000000002e-05 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.156 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:03:55.157 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:03:55.157 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:03:55.157 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.013975+0000 osd.6 (osd.6) 287 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.017904+0000 osd.6 (osd.6) 288 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.231143+0000 mon.a (mon.0) 2982 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.232640+0000 mgr.y (mgr.24526) 582 : cluster [DBG] pgmap v913: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: audit 2024-09-10T19:03:54.272109+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: audit 2024-09-10T19:03:54.272335+0000 mon.a (mon.0) 2983 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.272570+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:03:55.183 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:55 smithi165 bash[22591]: audit 2024-09-10T19:03:54.274869+0000 mon.a (mon.0) 2985 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.190 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:55.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.013975+0000 osd.6 (osd.6) 287 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.017904+0000 osd.6 (osd.6) 288 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.231143+0000 mon.a (mon.0) 2982 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.232640+0000 mgr.y (mgr.24526) 582 : cluster [DBG] pgmap v913: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: audit 2024-09-10T19:03:54.272109+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: audit 2024-09-10T19:03:54.272335+0000 mon.a (mon.0) 2983 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.272570+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[18629]: audit 2024-09-10T19:03:54.274869+0000 mon.a (mon.0) 2985 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.013975+0000 osd.6 (osd.6) 287 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:03:55.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.017904+0000 osd.6 (osd.6) 288 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.231143+0000 mon.a (mon.0) 2982 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.232640+0000 mgr.y (mgr.24526) 582 : cluster [DBG] pgmap v913: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: audit 2024-09-10T19:03:54.272109+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: audit 2024-09-10T19:03:54.272335+0000 mon.a (mon.0) 2983 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.272570+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:03:55.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:55 smithi083 bash[32381]: audit 2024-09-10T19:03:54.274869+0000 mon.a (mon.0) 2985 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:03:55.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:55.319 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:55.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:55.457 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:55.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:55.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:55.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:03:55.735 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:55.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:55.870 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:55.874 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:55.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:55.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:56.010 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:56.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:56.142 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:56.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:56.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.788252+0000 osd.0 (osd.0) 319 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:03:56.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.793711+0000 osd.0 (osd.0) 320 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:03:56.259 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.943730+0000 osd.6 (osd.6) 289 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.944040+0000 osd.6 (osd.6) 290 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.976070+0000 osd.6 (osd.6) 291 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:54.979766+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:55.018774+0000 osd.1 (osd.1) 329 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:55.022422+0000 osd.1 (osd.1) 330 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: audit 2024-09-10T19:03:55.272365+0000 mon.a (mon.0) 2986 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:55.272487+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: audit 2024-09-10T19:03:55.312824+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: audit 2024-09-10T19:03:55.315556+0000 mon.a (mon.0) 2988 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:55.589181+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:03:56.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:56 smithi165 bash[22591]: cluster 2024-09-10T19:03:55.592368+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:03:56.279 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:56.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:56.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.788252+0000 osd.0 (osd.0) 319 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.793711+0000 osd.0 (osd.0) 320 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.943730+0000 osd.6 (osd.6) 289 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.944040+0000 osd.6 (osd.6) 290 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.976070+0000 osd.6 (osd.6) 291 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:54.979766+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:55.018774+0000 osd.1 (osd.1) 329 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:55.022422+0000 osd.1 (osd.1) 330 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: audit 2024-09-10T19:03:55.272365+0000 mon.a (mon.0) 2986 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:55.272487+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: audit 2024-09-10T19:03:55.312824+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.392 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: audit 2024-09-10T19:03:55.315556+0000 mon.a (mon.0) 2988 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:55.589181+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[32381]: cluster 2024-09-10T19:03:55.592368+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.788252+0000 osd.0 (osd.0) 319 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.793711+0000 osd.0 (osd.0) 320 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.943730+0000 osd.6 (osd.6) 289 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.944040+0000 osd.6 (osd.6) 290 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.976070+0000 osd.6 (osd.6) 291 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:54.979766+0000 osd.6 (osd.6) 292 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:55.018774+0000 osd.1 (osd.1) 329 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:55.022422+0000 osd.1 (osd.1) 330 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:03:56.393 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: audit 2024-09-10T19:03:55.272365+0000 mon.a (mon.0) 2986 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:03:56.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:55.272487+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:03:56.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: audit 2024-09-10T19:03:55.312824+0000 mon.b (mon.1) 449 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: audit 2024-09-10T19:03:55.315556+0000 mon.a (mon.0) 2988 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:03:56.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:55.589181+0000 osd.2 (osd.2) 253 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:03:56.394 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:56 smithi083 bash[18629]: cluster 2024-09-10T19:03:55.592368+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:03:56.409 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:56.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:56.545 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:56.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:56.684 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:56.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:56.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:56.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:56.887 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:56.923 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:56.955 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:56.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:57.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:57.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:57.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:57.226 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:57.329 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.067578+0000 osd.1 (osd.1) 331 : cluster [DBG] 6.e scrub starts 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.071322+0000 osd.1 (osd.1) 332 : cluster [DBG] 6.e scrub ok 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.235041+0000 mgr.y (mgr.24526) 583 : cluster [DBG] pgmap v916: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: audit 2024-09-10T19:03:56.274654+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: audit 2024-09-10T19:03:56.274873+0000 mon.a (mon.0) 2989 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.274990+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: audit 2024-09-10T19:03:56.278175+0000 mon.a (mon.0) 2991 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.983056+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:03:57.339 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[18629]: cluster 2024-09-10T19:03:56.986641+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.067578+0000 osd.1 (osd.1) 331 : cluster [DBG] 6.e scrub starts 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.071322+0000 osd.1 (osd.1) 332 : cluster [DBG] 6.e scrub ok 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.235041+0000 mgr.y (mgr.24526) 583 : cluster [DBG] pgmap v916: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: audit 2024-09-10T19:03:56.274654+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: audit 2024-09-10T19:03:56.274873+0000 mon.a (mon.0) 2989 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.274990+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: audit 2024-09-10T19:03:56.278175+0000 mon.a (mon.0) 2991 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.983056+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:03:57.340 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:57 smithi083 bash[32381]: cluster 2024-09-10T19:03:56.986641+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:03:57.357 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:57.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.495 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:57.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.067578+0000 osd.1 (osd.1) 331 : cluster [DBG] 6.e scrub starts 2024-09-10T19:03:57.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.071322+0000 osd.1 (osd.1) 332 : cluster [DBG] 6.e scrub ok 2024-09-10T19:03:57.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.235041+0000 mgr.y (mgr.24526) 583 : cluster [DBG] pgmap v916: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:57.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: audit 2024-09-10T19:03:56.274654+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: audit 2024-09-10T19:03:56.274873+0000 mon.a (mon.0) 2989 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:03:57.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.274990+0000 mon.a (mon.0) 2990 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:03:57.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: audit 2024-09-10T19:03:56.278175+0000 mon.a (mon.0) 2991 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]: dispatch 2024-09-10T19:03:57.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.983056+0000 osd.5 (osd.5) 340 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:03:57.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:57 smithi165 bash[22591]: cluster 2024-09-10T19:03:56.986641+0000 osd.5 (osd.5) 341 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:03:57.631 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:57.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:57.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:57.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:57.905 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:57.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:58.008 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:58.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:58.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:58.175 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:03:58.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:03:58.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:58.313 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:58.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.152401+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.156371+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: audit 2024-09-10T19:03:57.279749+0000 mon.a (mon.0) 2992 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.280026+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.943736+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:03:58.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.949012+0000 osd.5 (osd.5) 343 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:57.997212+0000 osd.7 (osd.7) 340 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[18629]: cluster 2024-09-10T19:03:58.002490+0000 osd.7 (osd.7) 341 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.152401+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.156371+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: audit 2024-09-10T19:03:57.279749+0000 mon.a (mon.0) 2992 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.280026+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.943736+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.949012+0000 osd.5 (osd.5) 343 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:57.997212+0000 osd.7 (osd.7) 340 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:03:58.428 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:58 smithi083 bash[32381]: cluster 2024-09-10T19:03:58.002490+0000 osd.7 (osd.7) 341 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:03:58.441 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:58.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:58.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.152401+0000 osd.3 (osd.3) 344 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.156371+0000 osd.3 (osd.3) 345 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: audit 2024-09-10T19:03:57.279749+0000 mon.a (mon.0) 2992 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-98"}]': finished 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.280026+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.943736+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.949012+0000 osd.5 (osd.5) 343 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:57.997212+0000 osd.7 (osd.7) 340 : cluster [DBG] 64.6 scrub starts 2024-09-10T19:03:58.559 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:58 smithi165 bash[22591]: cluster 2024-09-10T19:03:58.002490+0000 osd.7 (osd.7) 341 : cluster [DBG] 64.6 scrub ok 2024-09-10T19:03:58.581 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:58.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:58.721 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:58.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:58.860 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:03:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:58.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:03:58.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:58.997 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:59.101 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:59.139 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:03:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:03:59.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:59.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:03:59.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:59.415 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:03:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:03:59.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: audit 2024-09-10T19:03:58.182155+0000 mgr.y (mgr.24526) 584 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: cluster 2024-09-10T19:03:58.235839+0000 mgr.y (mgr.24526) 585 : cluster [DBG] pgmap v919: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: cluster 2024-09-10T19:03:58.299552+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: cluster 2024-09-10T19:03:59.246947+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: audit 2024-09-10T19:03:59.250113+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[32381]: audit 2024-09-10T19:03:59.259905+0000 mon.a (mon.0) 2996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: audit 2024-09-10T19:03:58.182155+0000 mgr.y (mgr.24526) 584 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:59.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: cluster 2024-09-10T19:03:58.235839+0000 mgr.y (mgr.24526) 585 : cluster [DBG] pgmap v919: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:59.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: cluster 2024-09-10T19:03:58.299552+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:03:59.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: cluster 2024-09-10T19:03:59.246947+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:03:59.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: audit 2024-09-10T19:03:59.250113+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:03:59 smithi083 bash[18629]: audit 2024-09-10T19:03:59.259905+0000 mon.a (mon.0) 2996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.551 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:03:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:03:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: audit 2024-09-10T19:03:58.182155+0000 mgr.y (mgr.24526) 584 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: cluster 2024-09-10T19:03:58.235839+0000 mgr.y (mgr.24526) 585 : cluster [DBG] pgmap v919: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:03:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: cluster 2024-09-10T19:03:58.299552+0000 mon.a (mon.0) 2994 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:03:59.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: cluster 2024-09-10T19:03:59.246947+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:03:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: audit 2024-09-10T19:03:59.250113+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:03:59 smithi165 bash[22591]: audit 2024-09-10T19:03:59.259905+0000 mon.a (mon.0) 2996 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:03:59.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:59.685 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:03:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:03:59.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:59.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:03:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:03:59.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:59.955 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:03:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:59.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:00.058 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:00.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:00.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.228 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:00.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.364 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: cluster 2024-09-10T19:03:59.145526+0000 osd.3 (osd.3) 346 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: cluster 2024-09-10T19:03:59.153160+0000 osd.3 (osd.3) 347 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: audit 2024-09-10T19:03:59.578417+0000 mon.a (mon.0) 2997 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: audit 2024-09-10T19:03:59.581027+0000 mon.c (mon.2) 673 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: cluster 2024-09-10T19:03:59.614093+0000 osd.2 (osd.2) 255 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: cluster 2024-09-10T19:03:59.620045+0000 osd.2 (osd.2) 256 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:04:00.369 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: audit 2024-09-10T19:04:00.252245+0000 mon.a (mon.0) 2998 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:00.252482+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: audit 2024-09-10T19:04:00.252921+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[18629]: audit 2024-09-10T19:04:00.256559+0000 mon.a (mon.0) 3000 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: cluster 2024-09-10T19:03:59.145526+0000 osd.3 (osd.3) 346 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: cluster 2024-09-10T19:03:59.153160+0000 osd.3 (osd.3) 347 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: audit 2024-09-10T19:03:59.578417+0000 mon.a (mon.0) 2997 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: audit 2024-09-10T19:03:59.581027+0000 mon.c (mon.2) 673 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: cluster 2024-09-10T19:03:59.614093+0000 osd.2 (osd.2) 255 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: cluster 2024-09-10T19:03:59.620045+0000 osd.2 (osd.2) 256 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: audit 2024-09-10T19:04:00.252245+0000 mon.a (mon.0) 2998 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:00.370 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:00.252482+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:04:00.371 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: audit 2024-09-10T19:04:00.252921+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.371 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:00 smithi083 bash[32381]: audit 2024-09-10T19:04:00.256559+0000 mon.a (mon.0) 3000 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: cluster 2024-09-10T19:03:59.145526+0000 osd.3 (osd.3) 346 : cluster [DBG] 64.19 scrub starts 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: cluster 2024-09-10T19:03:59.153160+0000 osd.3 (osd.3) 347 : cluster [DBG] 64.19 scrub ok 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: audit 2024-09-10T19:03:59.578417+0000 mon.a (mon.0) 2997 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: audit 2024-09-10T19:03:59.581027+0000 mon.c (mon.2) 673 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: cluster 2024-09-10T19:03:59.614093+0000 osd.2 (osd.2) 255 : cluster [DBG] 64.15 scrub starts 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: cluster 2024-09-10T19:03:59.620045+0000 osd.2 (osd.2) 256 : cluster [DBG] 64.15 scrub ok 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: audit 2024-09-10T19:04:00.252245+0000 mon.a (mon.0) 2998 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:00.252482+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: audit 2024-09-10T19:04:00.252921+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.481 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:00 smithi165 bash[22591]: audit 2024-09-10T19:04:00.256559+0000 mon.a (mon.0) 3000 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:00.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:00.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:00.628 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:00.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:00.765 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:00.868 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:00.902 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:00.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:01.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:01.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:01.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:01.177 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:01.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:01.313 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:01.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:01.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:01.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:01.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: cluster 2024-09-10T19:04:00.236606+0000 mgr.y (mgr.24526) 586 : cluster [DBG] pgmap v922: 238 pgs: 32 unknown, 206 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:01.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: cluster 2024-09-10T19:04:00.573029+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:01.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: cluster 2024-09-10T19:04:00.954715+0000 osd.7 (osd.7) 342 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:04:01.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: cluster 2024-09-10T19:04:00.958119+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:04:01.556 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: audit 2024-09-10T19:04:01.251433+0000 mon.a (mon.0) 3002 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: cluster 2024-09-10T19:04:01.251712+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: audit 2024-09-10T19:04:01.253182+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[18629]: audit 2024-09-10T19:04:01.255987+0000 mon.a (mon.0) 3004 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: cluster 2024-09-10T19:04:00.236606+0000 mgr.y (mgr.24526) 586 : cluster [DBG] pgmap v922: 238 pgs: 32 unknown, 206 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: cluster 2024-09-10T19:04:00.573029+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: cluster 2024-09-10T19:04:00.954715+0000 osd.7 (osd.7) 342 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: cluster 2024-09-10T19:04:00.958119+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: audit 2024-09-10T19:04:01.251433+0000 mon.a (mon.0) 3002 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: cluster 2024-09-10T19:04:01.251712+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: audit 2024-09-10T19:04:01.253182+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.557 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:01 smithi083 bash[32381]: audit 2024-09-10T19:04:01.255987+0000 mon.a (mon.0) 3004 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.591 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:01.593 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:01.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: cluster 2024-09-10T19:04:00.236606+0000 mgr.y (mgr.24526) 586 : cluster [DBG] pgmap v922: 238 pgs: 32 unknown, 206 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:01.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: cluster 2024-09-10T19:04:00.573029+0000 mon.a (mon.0) 3001 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: cluster 2024-09-10T19:04:00.954715+0000 osd.7 (osd.7) 342 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: cluster 2024-09-10T19:04:00.958119+0000 osd.7 (osd.7) 343 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: audit 2024-09-10T19:04:01.251433+0000 mon.a (mon.0) 3002 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: cluster 2024-09-10T19:04:01.251712+0000 mon.a (mon.0) 3003 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: audit 2024-09-10T19:04:01.253182+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:01 smithi165 bash[22591]: audit 2024-09-10T19:04:01.255987+0000 mon.a (mon.0) 3004 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:01.696 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:01.725 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:01.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:01.863 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:01.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:01.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:01.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:02.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:02.137 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:02.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:02.269 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:02.372 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:02.409 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:02.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:02.544 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:02.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:02.683 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:02.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:02.821 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:02.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:02.938 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:02] "GET /metrics HTTP/1.1" 200 229002 "" "Prometheus/2.43.0" 2024-09-10T19:04:02.957 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:02.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:03.061 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:03.090 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:03.091 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:03.194 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:03.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:03.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:03.336 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:01.947893+0000 osd.0 (osd.0) 321 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:01.951640+0000 osd.0 (osd.0) 322 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.067070+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.067508+0000 osd.1 (osd.1) 333 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.071930+0000 osd.3 (osd.3) 349 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.073987+0000 osd.1 (osd.1) 334 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.237434+0000 mgr.y (mgr.24526) 587 : cluster [DBG] pgmap v925: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: audit 2024-09-10T19:04:02.255017+0000 mon.a (mon.0) 3005 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.255131+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: audit 2024-09-10T19:04:02.257521+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: audit 2024-09-10T19:04:02.260146+0000 mon.a (mon.0) 3007 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.651696+0000 osd.2 (osd.2) 257 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.657056+0000 osd.2 (osd.2) 258 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:04:03.337 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.953535+0000 osd.5 (osd.5) 344 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[18629]: cluster 2024-09-10T19:04:02.956911+0000 osd.5 (osd.5) 345 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:01.947893+0000 osd.0 (osd.0) 321 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:01.951640+0000 osd.0 (osd.0) 322 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.067070+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.067508+0000 osd.1 (osd.1) 333 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.071930+0000 osd.3 (osd.3) 349 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.073987+0000 osd.1 (osd.1) 334 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.237434+0000 mgr.y (mgr.24526) 587 : cluster [DBG] pgmap v925: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: audit 2024-09-10T19:04:02.255017+0000 mon.a (mon.0) 3005 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.255131+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:04:03.338 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: audit 2024-09-10T19:04:02.257521+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: audit 2024-09-10T19:04:02.260146+0000 mon.a (mon.0) 3007 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.651696+0000 osd.2 (osd.2) 257 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:04:03.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.657056+0000 osd.2 (osd.2) 258 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:04:03.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.953535+0000 osd.5 (osd.5) 344 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:04:03.339 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:03 smithi083 bash[32381]: cluster 2024-09-10T19:04:02.956911+0000 osd.5 (osd.5) 345 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:04:03.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:03.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:03.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:03.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:01.947893+0000 osd.0 (osd.0) 321 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:01.951640+0000 osd.0 (osd.0) 322 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.067070+0000 osd.3 (osd.3) 348 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.067508+0000 osd.1 (osd.1) 333 : cluster [DBG] 64.1c scrub starts 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.071930+0000 osd.3 (osd.3) 349 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.073987+0000 osd.1 (osd.1) 334 : cluster [DBG] 64.1c scrub ok 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.237434+0000 mgr.y (mgr.24526) 587 : cluster [DBG] pgmap v925: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:03.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: audit 2024-09-10T19:04:02.255017+0000 mon.a (mon.0) 3005 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-6", "overlaypool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.255131+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: audit 2024-09-10T19:04:02.257521+0000 mon.b (mon.1) 454 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: audit 2024-09-10T19:04:02.260146+0000 mon.a (mon.0) 3007 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]: dispatch 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.651696+0000 osd.2 (osd.2) 257 : cluster [DBG] 64.0 scrub starts 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.657056+0000 osd.2 (osd.2) 258 : cluster [DBG] 64.0 scrub ok 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.953535+0000 osd.5 (osd.5) 344 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:04:03.608 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:03 smithi165 bash[22591]: cluster 2024-09-10T19:04:02.956911+0000 osd.5 (osd.5) 345 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:04:03.623 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:03.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:03.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:03.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:03.881 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:03.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:03.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:04.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:04.153 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:04.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.248658+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.268131+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.268658+0000 mon.a (mon.0) 3009 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]': finished 2024-09-10T19:04:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.268791+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:04:04.260 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.270666+0000 mon.a (mon.0) 3011 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.374164+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.657308+0000 osd.2 (osd.2) 259 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.662169+0000 osd.2 (osd.2) 260 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.808942+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.810433+0000 mon.c (mon.2) 676 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: audit 2024-09-10T19:04:03.817272+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.906690+0000 osd.6 (osd.6) 293 : cluster [DBG] 64.e scrub starts 2024-09-10T19:04:04.261 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:04 smithi165 bash[22591]: cluster 2024-09-10T19:04:03.918433+0000 osd.6 (osd.6) 294 : cluster [DBG] 64.e scrub ok 2024-09-10T19:04:04.281 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:04.384 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.248658+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.268131+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.268658+0000 mon.a (mon.0) 3009 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]': finished 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.268791+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.270666+0000 mon.a (mon.0) 3011 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.374164+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.657308+0000 osd.2 (osd.2) 259 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:04:04.397 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.662169+0000 osd.2 (osd.2) 260 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.808942+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.810433+0000 mon.c (mon.2) 676 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: audit 2024-09-10T19:04:03.817272+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.906690+0000 osd.6 (osd.6) 293 : cluster [DBG] 64.e scrub starts 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[18629]: cluster 2024-09-10T19:04:03.918433+0000 osd.6 (osd.6) 294 : cluster [DBG] 64.e scrub ok 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.248658+0000 mon.a (mon.0) 3008 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.268131+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.268658+0000 mon.a (mon.0) 3009 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-100", "mode": "writeback"}]': finished 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.268791+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.270666+0000 mon.a (mon.0) 3011 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:04.398 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.374164+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.657308+0000 osd.2 (osd.2) 259 : cluster [DBG] 64.13 scrub starts 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.662169+0000 osd.2 (osd.2) 260 : cluster [DBG] 64.13 scrub ok 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.808942+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.810433+0000 mon.c (mon.2) 676 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: audit 2024-09-10T19:04:03.817272+0000 mon.a (mon.0) 3012 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.906690+0000 osd.6 (osd.6) 293 : cluster [DBG] 64.e scrub starts 2024-09-10T19:04:04.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:04 smithi083 bash[32381]: cluster 2024-09-10T19:04:03.918433+0000 osd.6 (osd.6) 294 : cluster [DBG] 64.e scrub ok 2024-09-10T19:04:04.413 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:04.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:04.551 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:04.655 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:04.686 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:04.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:04.824 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:04.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.958 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:04.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:05.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- 172.21.15.83:0/1752917118 >> v1:172.21.15.83:6789/0 conn(0x7fbf78103c60 legacy=0x7fbf78106040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- 172.21.15.83:0/1752917118 shutdown_connections 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- 172.21.15.83:0/1752917118 >> 172.21.15.83:0/1752917118 conn(0x7fbf78074c70 msgr2=0x7fbf780f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- 172.21.15.83:0/1752917118 shutdown_connections 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- 172.21.15.83:0/1752917118 wait complete. 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 librados: starting msgr at 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 librados: starting objecter 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 Processor -- start 2024-09-10T19:04:05.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- start start 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 librados: setting wanted keys 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 librados: calling monclient init 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf7811e430 con 0x7fbf78100e80 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf7811f5f0 con 0x7fbf781077c0 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf8005e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf781207f0 con 0x7fbf78103c60 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf7d5f9640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fbf78103c60 0x7fbf781191f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:46784/0 (socket says 172.21.15.83:46784) 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf7d5f9640 1 -- 172.21.15.83:0/893182552 learned_addr learned my addr 172.21.15.83:0/893182552 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf78103c60 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf78100e80 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982897497 0 0) 0x7fbf781207f0 con 0x7fbf78103c60 2024-09-10T19:04:05.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.058+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf64003120 con 0x7fbf78103c60 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 10 client.?.objecter ms_handle_connect 0x7fbf781077c0 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962165838 0 0) 0x7fbf7811e430 con 0x7fbf78100e80 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf781207f0 con 0x7fbf78100e80 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707670592 0 0) 0x7fbf7811f5f0 con 0x7fbf781077c0 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf7811e430 con 0x7fbf781077c0 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3965576379 0 0) 0x7fbf64003120 con 0x7fbf78103c60 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf7811f5f0 con 0x7fbf78103c60 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbf680015d0 con 0x7fbf78103c60 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 524523041 0 0) 0x7fbf781207f0 con 0x7fbf78100e80 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf64003120 con 0x7fbf78100e80 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbf740018c0 con 0x7fbf78100e80 2024-09-10T19:04:05.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3719714792 0 0) 0x7fbf7811f5f0 con 0x7fbf78103c60 2024-09-10T19:04:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.165:6789/0 conn(0x7fbf781077c0 legacy=0x7fbf7811cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6789/0 conn(0x7fbf78100e80 legacy=0x7fbf781002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf781219f0 con 0x7fbf78103c60 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7811e620 con 0x7fbf78103c60 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf8005e640 10 client.49622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7811ea70 con 0x7fbf78103c60 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf8005e640 1 librados: init done 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fbf68001aa0 con 0x7fbf78103c60 2024-09-10T19:04:05.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbf5c004ff0 con 0x7fbf78103c60 2024-09-10T19:04:05.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.062+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fbf68003600 con 0x7fbf78103c60 2024-09-10T19:04:05.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fbf68003840 con 0x7fbf78103c60 2024-09-10T19:04:05.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(601..601 src has 69..601) v4 ==== 8045+0+0 (unknown 3923779525 0 0) 0x7fbf6808a8b0 con 0x7fbf78103c60 2024-09-10T19:04:05.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 osd_map(601..601 src has 69..601) v4 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 3 client.49622.objecter handle_osd_map got epochs [601,601] > 0 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 3 client.49622.objecter handle_osd_map decoding full epoch 601 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 20 client.49622.objecter dump_active .. 0 homeless 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v601) v1 ==== 54+0+16 (unknown 2917459496 0 2420000192) 0x7fbf6808acb0 con 0x7fbf78103c60 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf640712e0 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 10 client.49622.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 1 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 1 2024-09-10T19:04:05.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 0 1 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 4 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 10 client.49622.objecter _send_command 1 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c005290 con 0x7fbf5c004610 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c004610 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c005290 con 0x7fbf5c004610 2024-09-10T19:04:05.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:04:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 1 = system:0 2024-09-10T19:04:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 4 2024-09-10T19:04:05.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.066+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 0 1 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 10 client.49622.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 2 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 2 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c005810 osd=0 3 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 0 2 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 4 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 10 client.49622.objecter _send_command 2 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.086+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c004cc0 con 0x7fbf5c004610 2024-09-10T19:04:05.093 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c004cc0 con 0x7fbf5c004610 2024-09-10T19:04:05.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 4 2024-09-10T19:04:05.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 0 2 2024-09-10T19:04:05.095 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:04:05.095 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 10 client.49622.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 1 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 4 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 10 client.49622.objecter _send_command 3 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c0022b0 con 0x7fbf5c004110 2024-09-10T19:04:05.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c004110 2024-09-10T19:04:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c0022b0 con 0x7fbf5c004110 2024-09-10T19:04:05.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:04:05.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 3 = system:0 2024-09-10T19:04:05.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 4 2024-09-10T19:04:05.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.090+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 1 3 2024-09-10T19:04:05.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 10 client.49622.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 4 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 4 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c003fa0 osd=1 3 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 1 4 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 4 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 10 client.49622.objecter _send_command 4 2024-09-10T19:04:05.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.110+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c001890 con 0x7fbf5c004110 2024-09-10T19:04:05.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c001890 con 0x7fbf5c004110 2024-09-10T19:04:05.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 4 2024-09-10T19:04:05.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 1 4 2024-09-10T19:04:05.119 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:04:05.119 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 10 client.49622.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 5 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 5 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 2 5 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 4 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 10 client.49622.objecter _send_command 5 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c003df0 con 0x7fbf5c00d350 2024-09-10T19:04:05.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c00d350 2024-09-10T19:04:05.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c003df0 con 0x7fbf5c00d350 2024-09-10T19:04:05.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:04:05.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 5 = system:0 2024-09-10T19:04:05.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 4 2024-09-10T19:04:05.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.114+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 2 5 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 10 client.49622.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 6 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 6 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c0035e0 osd=2 3 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 2 6 2024-09-10T19:04:05.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 4 2024-09-10T19:04:05.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 10 client.49622.objecter _send_command 6 2024-09-10T19:04:05.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.126+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c0030d0 con 0x7fbf5c00d350 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c0030d0 con 0x7fbf5c00d350 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 4 2024-09-10T19:04:05.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 2 6 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 10 client.49622.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 7 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 7 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 3 7 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 4 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 10 client.49622.objecter _send_command 7 2024-09-10T19:04:05.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c00d0d0 con 0x7fbf5c010e50 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.130+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c010e50 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.134+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c00d0d0 con 0x7fbf5c010e50 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.134+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.134+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 7 = system:0 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.134+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 4 2024-09-10T19:04:05.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.134+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 3 7 2024-09-10T19:04:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 10 client.49622.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 8 2024-09-10T19:04:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 8 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c00c990 osd=3 3 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 3 8 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 4 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 10 client.49622.objecter _send_command 8 2024-09-10T19:04:05.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.150+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c00d0d0 con 0x7fbf5c010e50 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c00d0d0 con 0x7fbf5c010e50 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 4 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 3 8 2024-09-10T19:04:05.159 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 10 client.49622.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 9 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 9 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 4 9 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 4 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 10 client.49622.objecter _send_command 9 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c013560 con 0x7fbf5c016760 2024-09-10T19:04:05.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c016760 2024-09-10T19:04:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c013560 con 0x7fbf5c016760 2024-09-10T19:04:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:04:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 9 = system:0 2024-09-10T19:04:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 4 2024-09-10T19:04:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.154+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 4 9 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 10 client.49622.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 10 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 10 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c00d0d0 osd=4 3 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 4 10 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 4 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 10 client.49622.objecter _send_command 10 2024-09-10T19:04:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.170+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c01a260 con 0x7fbf5c016760 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c01a260 con 0x7fbf5c016760 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 4 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 4 10 2024-09-10T19:04:05.180 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 10 client.49622.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 11 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 11 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 5 11 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 4 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 10 client.49622.objecter _send_command 11 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c01c510 con 0x7fbf5c01a910 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.174+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c01a910 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.178+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c01c510 con 0x7fbf5c01a910 2024-09-10T19:04:05.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.178+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:04:05.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.178+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 11 = system:0 2024-09-10T19:04:05.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.178+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 4 2024-09-10T19:04:05.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.178+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 5 11 2024-09-10T19:04:05.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 10 client.49622.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 12 2024-09-10T19:04:05.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 12 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c01a260 osd=5 3 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 5 12 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 4 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 10 client.49622.objecter _send_command 12 2024-09-10T19:04:05.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.190+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c01c510 con 0x7fbf5c01a910 2024-09-10T19:04:05.198 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c01c510 con 0x7fbf5c01a910 2024-09-10T19:04:05.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 4 2024-09-10T19:04:05.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 5 12 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 10 client.49622.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 13 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 13 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 6 13 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 4 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 10 client.49622.objecter _send_command 13 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c022b20 con 0x7fbf5c01f020 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.194+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c01f020 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.198+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c022b20 con 0x7fbf5c01f020 2024-09-10T19:04:05.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.198+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:04:05.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.198+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 13 = system:0 2024-09-10T19:04:05.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.198+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 4 2024-09-10T19:04:05.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.198+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 6 13 2024-09-10T19:04:05.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 10 client.49622.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 14 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 14 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c01c510 osd=6 3 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 6 14 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 4 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 10 client.49622.objecter _send_command 14 2024-09-10T19:04:05.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.218+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c022b20 con 0x7fbf5c01f020 2024-09-10T19:04:05.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.222+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c022b20 con 0x7fbf5c01f020 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.222+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.222+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.222+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 4 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.222+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 6 14 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:04:05.230 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 10 client.49622.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 15 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 15 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 7 15 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 4 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 10 client.49622.objecter _send_command 15 2024-09-10T19:04:05.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5c026de0 con 0x7fbf5c0231d0 2024-09-10T19:04:05.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 10 client.49622.objecter ms_handle_connect 0x7fbf5c0231d0 2024-09-10T19:04:05.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fbf5c026de0 con 0x7fbf5c0231d0 2024-09-10T19:04:05.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:04:05.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 15 = system:0 2024-09-10T19:04:05.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 4 2024-09-10T19:04:05.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.226+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 7 15 2024-09-10T19:04:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:05.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 10 client.49622.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:05.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign -1 16 2024-09-10T19:04:05.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 20 client.49622.objecter _get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_remove -1 16 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 20 client.49622.objecter get_session s=0x7fbf5c022b20 osd=7 3 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 15 client.49622.objecter _session_command_op_assign 7 16 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 4 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 10 client.49622.objecter _send_command 16 2024-09-10T19:04:05.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.250+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fbf5c026de0 con 0x7fbf5c0231d0 2024-09-10T19:04:05.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf6e7fc640 1 -- 172.21.15.83:0/893182552 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fbf5c026de0 con 0x7fbf5c0231d0 2024-09-10T19:04:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf6e7fc640 10 client.49622.objecter ms_dispatch 0x7fbf78000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf6e7fc640 10 client.49622.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf6e7fc640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 4 2024-09-10T19:04:05.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf6e7fc640 15 client.49622.objecter _session_command_op_remove 7 16 2024-09-10T19:04:05.257 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:04:05.258 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:04:05.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 librados: watch_flush enter 2024-09-10T19:04:05.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 librados: watch_flush exit 2024-09-10T19:04:05.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.0 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6801/2535527483 conn(0x7fbf5c004610 legacy=0x7fbf5c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c005810 osd=0 1 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.1 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6805/2309624637 conn(0x7fbf5c004110 legacy=0x7fbf5c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c003fa0 osd=1 1 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.2 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6809/386094220 conn(0x7fbf5c00d350 legacy=0x7fbf5c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c0035e0 osd=2 1 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.3 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6813/1279688021 conn(0x7fbf5c010e50 legacy=0x7fbf5c011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00c990 osd=3 1 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.4 2024-09-10T19:04:05.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.165:6800/3225515265 conn(0x7fbf5c016760 legacy=0x7fbf5c018b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c00d0d0 osd=4 1 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.5 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.165:6804/571173777 conn(0x7fbf5c01a910 legacy=0x7fbf5c01ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01a260 osd=5 1 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.6 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.165:6808/1317922936 conn(0x7fbf5c01f020 legacy=0x7fbf5c021400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c01c510 osd=6 1 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter close_session for osd.7 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.165:6812/1610151650 conn(0x7fbf5c0231d0 legacy=0x7fbf5c0255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter put_session s=0x7fbf5c022b20 osd=7 1 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 20 client.49622.objecter shutdown clearing up homeless session... 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 10 client.49622.objecter successfully canceled tick 2024-09-10T19:04:05.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf7e5fb640 1 -- 172.21.15.83:0/893182552 reap_dead start 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf7e5fb640 1 -- 172.21.15.83:0/893182552 reap_dead start 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf7e5fb640 1 -- 172.21.15.83:0/893182552 reap_dead start 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf7e5fb640 1 -- 172.21.15.83:0/893182552 reap_dead start 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf7e5fb640 1 -- 172.21.15.83:0/893182552 reap_dead start 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6800/3071793735 conn(0x7fbf640712e0 legacy=0x7fbf64073790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> v1:172.21.15.83:6790/0 conn(0x7fbf78103c60 legacy=0x7fbf781191f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 shutdown_connections 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 >> 172.21.15.83:0/893182552 conn(0x7fbf78074c70 msgr2=0x7fbf78070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 shutdown_connections 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 -- 172.21.15.83:0/893182552 wait complete. 2024-09-10T19:04:05.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:05.254+0000 7fbf8005e640 1 librados: shutdown 2024-09-10T19:04:05.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:05.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.084724+0000 osd.3 (osd.3) 350 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:04:05.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.088391+0000 osd.3 (osd.3) 351 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:04:05.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.238265+0000 mgr.y (mgr.24526) 588 : cluster [DBG] pgmap v928: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: audit 2024-09-10T19:04:04.276790+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: audit 2024-09-10T19:04:04.276908+0000 mon.a (mon.0) 3013 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.277100+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: audit 2024-09-10T19:04:04.279402+0000 mon.a (mon.0) 3015 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.917152+0000 osd.6 (osd.6) 295 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.917397+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.f deep-scrub starts 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.920337+0000 osd.6 (osd.6) 296 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.923901+0000 osd.4 (osd.4) 345 : cluster [DBG] 2.f deep-scrub ok 2024-09-10T19:04:05.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:05 smithi165 bash[22591]: audit 2024-09-10T19:04:05.071394+0000 mon.c (mon.2) 677 : audit [DBG] from='client.? v1:172.21.15.83:0/893182552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:05.374 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:05.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.084724+0000 osd.3 (osd.3) 350 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:04:05.489 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.088391+0000 osd.3 (osd.3) 351 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.238265+0000 mgr.y (mgr.24526) 588 : cluster [DBG] pgmap v928: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: audit 2024-09-10T19:04:04.276790+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: audit 2024-09-10T19:04:04.276908+0000 mon.a (mon.0) 3013 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.277100+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: audit 2024-09-10T19:04:04.279402+0000 mon.a (mon.0) 3015 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.917152+0000 osd.6 (osd.6) 295 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.917397+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.f deep-scrub starts 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.920337+0000 osd.6 (osd.6) 296 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.923901+0000 osd.4 (osd.4) 345 : cluster [DBG] 2.f deep-scrub ok 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[32381]: audit 2024-09-10T19:04:05.071394+0000 mon.c (mon.2) 677 : audit [DBG] from='client.? v1:172.21.15.83:0/893182552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:05.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.084724+0000 osd.3 (osd.3) 350 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.088391+0000 osd.3 (osd.3) 351 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.238265+0000 mgr.y (mgr.24526) 588 : cluster [DBG] pgmap v928: 238 pgs: 238 active+clean; 8.1 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: audit 2024-09-10T19:04:04.276790+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: audit 2024-09-10T19:04:04.276908+0000 mon.a (mon.0) 3013 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.277100+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: audit 2024-09-10T19:04:04.279402+0000 mon.a (mon.0) 3015 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.917152+0000 osd.6 (osd.6) 295 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.917397+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.f deep-scrub starts 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.920337+0000 osd.6 (osd.6) 296 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.923901+0000 osd.4 (osd.4) 345 : cluster [DBG] 2.f deep-scrub ok 2024-09-10T19:04:05.491 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:05 smithi083 bash[18629]: audit 2024-09-10T19:04:05.071394+0000 mon.c (mon.2) 677 : audit [DBG] from='client.? v1:172.21.15.83:0/893182552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:05.506 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:05.610 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:05.637 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:05.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:05.776 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:05.879 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:05.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:06.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:06.048 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:06.152 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:06.184 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:06.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:06.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:06.313 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:06.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:06.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.929479+0000 osd.0 (osd.0) 323 : cluster [DBG] 4.b scrub starts 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: cluster 2024-09-10T19:04:04.932678+0000 osd.0 (osd.0) 324 : cluster [DBG] 4.b scrub ok 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: audit 2024-09-10T19:04:05.280773+0000 mon.a (mon.0) 3016 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: cluster 2024-09-10T19:04:05.280876+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: audit 2024-09-10T19:04:05.281720+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: audit 2024-09-10T19:04:05.284861+0000 mon.a (mon.0) 3018 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: cluster 2024-09-10T19:04:05.599717+0000 osd.2 (osd.2) 261 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:04:06.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[18629]: cluster 2024-09-10T19:04:05.604735+0000 osd.2 (osd.2) 262 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.929479+0000 osd.0 (osd.0) 323 : cluster [DBG] 4.b scrub starts 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: cluster 2024-09-10T19:04:04.932678+0000 osd.0 (osd.0) 324 : cluster [DBG] 4.b scrub ok 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: audit 2024-09-10T19:04:05.280773+0000 mon.a (mon.0) 3016 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: cluster 2024-09-10T19:04:05.280876+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: audit 2024-09-10T19:04:05.281720+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: audit 2024-09-10T19:04:05.284861+0000 mon.a (mon.0) 3018 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: cluster 2024-09-10T19:04:05.599717+0000 osd.2 (osd.2) 261 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:04:06.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:06 smithi083 bash[32381]: cluster 2024-09-10T19:04:05.604735+0000 osd.2 (osd.2) 262 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:04:06.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:06.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:06.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.929479+0000 osd.0 (osd.0) 323 : cluster [DBG] 4.b scrub starts 2024-09-10T19:04:06.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: cluster 2024-09-10T19:04:04.932678+0000 osd.0 (osd.0) 324 : cluster [DBG] 4.b scrub ok 2024-09-10T19:04:06.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: audit 2024-09-10T19:04:05.280773+0000 mon.a (mon.0) 3016 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:06.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: cluster 2024-09-10T19:04:05.280876+0000 mon.a (mon.0) 3017 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:04:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: audit 2024-09-10T19:04:05.281720+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: audit 2024-09-10T19:04:05.284861+0000 mon.a (mon.0) 3018 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: cluster 2024-09-10T19:04:05.599717+0000 osd.2 (osd.2) 261 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:04:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:06 smithi165 bash[22591]: cluster 2024-09-10T19:04:05.604735+0000 osd.2 (osd.2) 262 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:04:06.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:06.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:06.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:06.861 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:06.964 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:06.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:06.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:07.096 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:07.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:07.231 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:07.266 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:07.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.030377+0000 osd.1 (osd.1) 335 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.033491+0000 osd.1 (osd.1) 336 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.120669+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.124238+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.240663+0000 mgr.y (mgr.24526) 589 : cluster [DBG] pgmap v931: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.276621+0000 mon.a (mon.0) 3019 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: audit 2024-09-10T19:04:06.286245+0000 mon.a (mon.0) 3020 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:07.382 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.286445+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: audit 2024-09-10T19:04:06.286606+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: audit 2024-09-10T19:04:06.295447+0000 mon.a (mon.0) 3022 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.872720+0000 osd.4 (osd.4) 346 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.876777+0000 osd.4 (osd.4) 347 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.916011+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.919397+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.981730+0000 osd.7 (osd.7) 344 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:06.987270+0000 osd.7 (osd.7) 345 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.065982+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:04:07.383 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.069590+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.030377+0000 osd.1 (osd.1) 335 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.033491+0000 osd.1 (osd.1) 336 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.120669+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.124238+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.240663+0000 mgr.y (mgr.24526) 589 : cluster [DBG] pgmap v931: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.276621+0000 mon.a (mon.0) 3019 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: audit 2024-09-10T19:04:06.286245+0000 mon.a (mon.0) 3020 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.286445+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: audit 2024-09-10T19:04:06.286606+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: audit 2024-09-10T19:04:06.295447+0000 mon.a (mon.0) 3022 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.384 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.872720+0000 osd.4 (osd.4) 346 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.876777+0000 osd.4 (osd.4) 347 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.916011+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.919397+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.981730+0000 osd.7 (osd.7) 344 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:06.987270+0000 osd.7 (osd.7) 345 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.065982+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:04:07.385 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:07 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.069590+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:04:07.403 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:07.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:07.543 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:07.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.030377+0000 osd.1 (osd.1) 335 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:04:07.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.033491+0000 osd.1 (osd.1) 336 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:04:07.545 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.120669+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.124238+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.240663+0000 mgr.y (mgr.24526) 589 : cluster [DBG] pgmap v931: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.276621+0000 mon.a (mon.0) 3019 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: audit 2024-09-10T19:04:06.286245+0000 mon.a (mon.0) 3020 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.286445+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: audit 2024-09-10T19:04:06.286606+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: audit 2024-09-10T19:04:06.295447+0000 mon.a (mon.0) 3022 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.872720+0000 osd.4 (osd.4) 346 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.876777+0000 osd.4 (osd.4) 347 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.916011+0000 osd.6 (osd.6) 297 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.919397+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.981730+0000 osd.7 (osd.7) 344 : cluster [DBG] 64.1a scrub starts 2024-09-10T19:04:07.546 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:06.987270+0000 osd.7 (osd.7) 345 : cluster [DBG] 64.1a scrub ok 2024-09-10T19:04:07.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.065982+0000 osd.5 (osd.5) 346 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:04:07.547 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:07 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.069590+0000 osd.5 (osd.5) 347 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:04:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:07.648 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:07.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- 172.21.15.83:0/3940322559 >> v1:172.21.15.83:6790/0 conn(0x7f9800101520 legacy=0x7f98001018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- 172.21.15.83:0/3940322559 shutdown_connections 2024-09-10T19:04:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- 172.21.15.83:0/3940322559 >> 172.21.15.83:0/3940322559 conn(0x7f98000f7080 msgr2=0x7f98000f9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- 172.21.15.83:0/3940322559 shutdown_connections 2024-09-10T19:04:07.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- 172.21.15.83:0/3940322559 wait complete. 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 librados: starting msgr at 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 librados: starting objecter 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 Processor -- start 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- start start 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 librados: setting wanted keys 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 librados: calling monclient init 2024-09-10T19:04:07.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f980011b270 con 0x7f9800104290 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f980011c410 con 0x7f9800107df0 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9807656640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f980011d5b0 con 0x7f9800101520 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9804bf1640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9800104290 0x7f9800116420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37134/0 (socket says 172.21.15.83:37134) 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f9804bf1640 1 -- 172.21.15.83:0/3127413876 learned_addr learned my addr 172.21.15.83:0/3127413876 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f98053f2640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9800101520 0x7f9800106f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:46814/0 (socket says 172.21.15.83:46814) 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter ms_handle_connect 0x7f9800101520 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter ms_handle_connect 0x7f9800104290 2024-09-10T19:04:07.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044224816 0 0) 0x7f980011d5b0 con 0x7f9800101520 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ec003120 con 0x7f9800101520 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter ms_handle_connect 0x7f9800107df0 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069546586 0 0) 0x7f980011b270 con 0x7f9800104290 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980011d5b0 con 0x7f9800104290 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428783585 0 0) 0x7f980011c410 con 0x7f9800107df0 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980011b270 con 0x7f9800107df0 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863901825 0 0) 0x7f97ec003120 con 0x7f9800101520 2024-09-10T19:04:07.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f980011c410 con 0x7f9800101520 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f97f8003010 con 0x7f9800101520 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.734+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918945945 0 0) 0x7f980011d5b0 con 0x7f9800104290 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97ec003120 con 0x7f9800104290 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f97f0001860 con 0x7f9800104290 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1361661035 0 0) 0x7f980011b270 con 0x7f9800107df0 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f980011d5b0 con 0x7f9800107df0 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f97fc0017c0 con 0x7f9800107df0 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2092044970 0 0) 0x7f980011c410 con 0x7f9800101520 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 >> v1:172.21.15.165:6789/0 conn(0x7f9800107df0 legacy=0x7f9800119b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 >> v1:172.21.15.83:6789/0 conn(0x7f9800104290 legacy=0x7f9800116420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980011e750 con 0x7f9800101520 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980011d7a0 con 0x7f9800101520 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f9807656640 10 client.49628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:07.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f980011dbf0 con 0x7f9800101520 2024-09-10T19:04:07.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f9807656640 1 librados: init done 2024-09-10T19:04:07.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f97f8001560 con 0x7f9800101520 2024-09-10T19:04:07.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f97f8003880 con 0x7f9800101520 2024-09-10T19:04:07.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e4005010 con 0x7f9800101520 2024-09-10T19:04:07.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.738+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f97f8003ac0 con 0x7f9800101520 2024-09-10T19:04:07.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(604..604 src has 69..604) v4 ==== 8045+0+0 (unknown 3025393611 0 0) 0x7f97f808ab50 con 0x7f9800101520 2024-09-10T19:04:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 10 client.49628.objecter ms_dispatch 0x7f9800000b90 osd_map(604..604 src has 69..604) v4 2024-09-10T19:04:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 3 client.49628.objecter handle_osd_map got epochs [604,604] > 0 2024-09-10T19:04:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 3 client.49628.objecter handle_osd_map decoding full epoch 604 2024-09-10T19:04:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 20 client.49628.objecter dump_active .. 0 homeless 2024-09-10T19:04:07.743 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.742+0000 7f97f5ffb640 10 client.49628.objecter ms_handle_connect 0x7f97ec0718d0 2024-09-10T19:04:07.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.746+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f97f80570e0 con 0x7f9800101520 2024-09-10T19:04:07.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:07.815 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:07.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:07.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:07.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.950+0000 7f69285b5640 1 -- 172.21.15.83:0/2253341220 >> v1:172.21.15.83:6790/0 conn(0x7f6920101710 legacy=0x7f6920101ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.950+0000 7f69285b5640 1 -- 172.21.15.83:0/2253341220 shutdown_connections 2024-09-10T19:04:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.950+0000 7f69285b5640 1 -- 172.21.15.83:0/2253341220 >> 172.21.15.83:0/2253341220 conn(0x7f6920074c70 msgr2=0x7f69200f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:07.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.950+0000 7f69285b5640 1 -- 172.21.15.83:0/2253341220 shutdown_connections 2024-09-10T19:04:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.950+0000 7f69285b5640 1 -- 172.21.15.83:0/2253341220 wait complete. 2024-09-10T19:04:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 librados: starting msgr at 2024-09-10T19:04:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 librados: starting objecter 2024-09-10T19:04:07.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 Processor -- start 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- start start 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 librados: setting wanted keys 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 librados: calling monclient init 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f692011a010 con 0x7f69201044f0 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f692011b1d0 con 0x7f6920108050 2024-09-10T19:04:07.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f692011c3d0 con 0x7f6920101710 2024-09-10T19:04:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6925b50640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f69201044f0 0x7f69200fd9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37148/0 (socket says 172.21.15.83:37148) 2024-09-10T19:04:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6925b50640 1 -- 172.21.15.83:0/615540304 learned_addr learned my addr 172.21.15.83:0/615540304 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:07.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter ms_handle_connect 0x7f6920101710 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter ms_handle_connect 0x7f69201044f0 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810754327 0 0) 0x7f692011c3d0 con 0x7f6920101710 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c0042a0 con 0x7f6920101710 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809231861 0 0) 0x7f692011a010 con 0x7f69201044f0 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692011c3d0 con 0x7f69201044f0 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter ms_handle_connect 0x7f6920108050 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887582702 0 0) 0x7f692011c3d0 con 0x7f69201044f0 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f692011a010 con 0x7f69201044f0 2024-09-10T19:04:07.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6910001900 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078420327 0 0) 0x7f692011b1d0 con 0x7f6920108050 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692011c3d0 con 0x7f6920108050 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227207769 0 0) 0x7f690c0042a0 con 0x7f6920101710 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f692011b1d0 con 0x7f6920101710 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f691c0015d0 con 0x7f6920101710 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2766011481 0 0) 0x7f692011a010 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 >> v1:172.21.15.83:6790/0 conn(0x7f6920101710 legacy=0x7f69200fd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 >> v1:172.21.15.165:6789/0 conn(0x7f6920108050 legacy=0x7f6920118710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692011d5d0 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692011a200 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 10 client.25557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f692011a6d0 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 librados: init done 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6910002360 con 0x7f69201044f0 2024-09-10T19:04:07.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6910005a60 con 0x7f69201044f0 2024-09-10T19:04:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) v1 -- 0x7f97e4005840 con 0x7f9800101520 2024-09-10T19:04:07.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.954+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6904005010 con 0x7f69201044f0 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f6910001e70 con 0x7f69201044f0 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(604..604 src has 69..604) v4 ==== 8045+0+0 (unknown 3025393611 0 0) 0x7f691008a670 con 0x7f69201044f0 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 10 client.25557.objecter ms_dispatch 0x7f6920000b90 osd_map(604..604 src has 69..604) v4 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 3 client.25557.objecter handle_osd_map got epochs [604,604] > 0 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 3 client.25557.objecter handle_osd_map decoding full epoch 604 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 20 client.25557.objecter dump_active .. 0 homeless 2024-09-10T19:04:07.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.958+0000 7f6916ffd640 10 client.25557.objecter ms_handle_connect 0x7f690c0716a0 2024-09-10T19:04:07.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:07.962+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f6910056780 con 0x7f69201044f0 2024-09-10T19:04:08.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:08.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:08.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.154+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f6904005840 con 0x7f69201044f0 2024-09-10T19:04:08.180 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.214 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:08.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f97f5ffb640 1 -- 172.21.15.83:0/3127413876 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v605) v1 ==== 131+0+0 (unknown 2470416485 0 0) 0x7f97f8056a80 con 0x7f9800101520 2024-09-10T19:04:08.323 INFO:teuthology.orchestra.run.smithi083.stderr:set pool 1 pgp_num to 11 2024-09-10T19:04:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f6916ffd640 1 -- 172.21.15.83:0/615540304 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v605) v1 ==== 87+0+0 (unknown 4170266258 0 0) 0x7f69100565a0 con 0x7f69201044f0 2024-09-10T19:04:08.325 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 10 librados: watch_flush enter 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 10 librados: watch_flush exit 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 20 client.49628.objecter shutdown clearing up homeless session... 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 10 client.49628.objecter successfully canceled tick 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 >> v1:172.21.15.83:6800/3071793735 conn(0x7f97ec0718d0 legacy=0x7f97ec073d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 >> v1:172.21.15.83:6790/0 conn(0x7f9800101520 legacy=0x7f9800106f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 shutdown_connections 2024-09-10T19:04:08.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.322+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 >> 172.21.15.83:0/3127413876 conn(0x7f98000f7080 msgr2=0x7f98000f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:08.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 shutdown_connections 2024-09-10T19:04:08.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f9807656640 1 -- 172.21.15.83:0/3127413876 wait complete. 2024-09-10T19:04:08.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f9807656640 1 librados: shutdown 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 10 librados: watch_flush enter 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 10 librados: watch_flush exit 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 20 client.25557.objecter shutdown clearing up homeless session... 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 10 client.25557.objecter successfully canceled tick 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 >> v1:172.21.15.83:6800/3071793735 conn(0x7f690c0716a0 legacy=0x7f690c073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 >> v1:172.21.15.83:6789/0 conn(0x7f69201044f0 legacy=0x7f69200fd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 shutdown_connections 2024-09-10T19:04:08.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 >> 172.21.15.83:0/615540304 conn(0x7f6920074c70 msgr2=0x7f69200715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 shutdown_connections 2024-09-10T19:04:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 -- 172.21.15.83:0/615540304 wait complete. 2024-09-10T19:04:08.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:08.326+0000 7f69285b5640 1 librados: shutdown 2024-09-10T19:04:08.344 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:08.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.112629+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.117372+0000 osd.3 (osd.3) 355 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:07.308106+0000 mon.a (mon.0) 3023 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.308191+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:07.354555+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:07.357216+0000 mon.a (mon.0) 3025 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.460 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.406168+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.406710+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.616521+0000 osd.2 (osd.2) 265 : cluster [DBG] 6.f scrub starts 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.618984+0000 osd.2 (osd.2) 266 : cluster [DBG] 6.f scrub ok 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:07.957570+0000 mon.c (mon.2) 678 : audit [INF] from='client.? v1:172.21.15.83:0/3127413876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:07.957890+0000 mon.a (mon.0) 3026 : audit [INF] from='client.49628 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.966595+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.d scrub starts 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.971653+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.d scrub ok 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.036058+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.039687+0000 osd.5 (osd.5) 349 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[18629]: audit 2024-09-10T19:04:08.156576+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:04:08.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.112629+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.117372+0000 osd.3 (osd.3) 355 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:07.308106+0000 mon.a (mon.0) 3023 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.308191+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:07.354555+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:07.357216+0000 mon.a (mon.0) 3025 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.406168+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.406710+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.616521+0000 osd.2 (osd.2) 265 : cluster [DBG] 6.f scrub starts 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.618984+0000 osd.2 (osd.2) 266 : cluster [DBG] 6.f scrub ok 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:07.957570+0000 mon.c (mon.2) 678 : audit [INF] from='client.? v1:172.21.15.83:0/3127413876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:07.957890+0000 mon.a (mon.0) 3026 : audit [INF] from='client.49628 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.966595+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.d scrub starts 2024-09-10T19:04:08.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.971653+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.d scrub ok 2024-09-10T19:04:08.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.036058+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:04:08.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.039687+0000 osd.5 (osd.5) 349 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:04:08.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:08 smithi083 bash[32381]: audit 2024-09-10T19:04:08.156576+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:04:08.483 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:08.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:08.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:08.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.112629+0000 osd.3 (osd.3) 354 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:04:08.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.117372+0000 osd.3 (osd.3) 355 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:04:08.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:07.308106+0000 mon.a (mon.0) 3023 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:04:08.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.308191+0000 mon.a (mon.0) 3024 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:07.354555+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:07.357216+0000 mon.a (mon.0) 3025 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.406168+0000 osd.2 (osd.2) 263 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.406710+0000 osd.2 (osd.2) 264 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.616521+0000 osd.2 (osd.2) 265 : cluster [DBG] 6.f scrub starts 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.618984+0000 osd.2 (osd.2) 266 : cluster [DBG] 6.f scrub ok 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:07.957570+0000 mon.c (mon.2) 678 : audit [INF] from='client.? v1:172.21.15.83:0/3127413876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:07.957890+0000 mon.a (mon.0) 3026 : audit [INF] from='client.49628 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.966595+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.d scrub starts 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.971653+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.d scrub ok 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.036058+0000 osd.5 (osd.5) 348 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.039687+0000 osd.5 (osd.5) 349 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:04:08.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:08 smithi165 bash[22591]: audit 2024-09-10T19:04:08.156576+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:04:08.618 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:08.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.754 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:08.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:08.885 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:08.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:08.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:09.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:04:09.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:09.165 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:09.268 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:09.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.950387+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T19:04:09.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:07.952987+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T19:04:09.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.090017+0000 osd.3 (osd.3) 356 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:04:09.409 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.092994+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.192800+0000 mgr.y (mgr.24526) 590 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.241539+0000 mgr.y (mgr.24526) 591 : cluster [DBG] pgmap v934: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.304426+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.321262+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.322153+0000 mon.a (mon.0) 3029 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.322272+0000 mon.a (mon.0) 3030 : audit [INF] from='client.49628 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.322340+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.322379+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:08.323869+0000 mon.a (mon.0) 3033 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.594287+0000 osd.2 (osd.2) 267 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:08.598175+0000 osd.2 (osd.2) 268 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:09.239612+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:09.410 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: audit 2024-09-10T19:04:09.249327+0000 mon.a (mon.0) 3035 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:09.411 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:09 smithi165 bash[22591]: cluster 2024-09-10T19:04:09.249425+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:04:09.431 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:09.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:09.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:09.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.950387+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T19:04:09.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:07.952987+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T19:04:09.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.090017+0000 osd.3 (osd.3) 356 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:04:09.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.092994+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:04:09.589 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.192800+0000 mgr.y (mgr.24526) 590 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.241539+0000 mgr.y (mgr.24526) 591 : cluster [DBG] pgmap v934: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.304426+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.321262+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.322153+0000 mon.a (mon.0) 3029 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.322272+0000 mon.a (mon.0) 3030 : audit [INF] from='client.49628 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.322340+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.322379+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:08.323869+0000 mon.a (mon.0) 3033 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.594287+0000 osd.2 (osd.2) 267 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:08.598175+0000 osd.2 (osd.2) 268 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:09.239612+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: audit 2024-09-10T19:04:09.249327+0000 mon.a (mon.0) 3035 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:09.590 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[18629]: cluster 2024-09-10T19:04:09.249425+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.950387+0000 osd.0 (osd.0) 325 : cluster [DBG] 6.9 deep-scrub starts 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:07.952987+0000 osd.0 (osd.0) 326 : cluster [DBG] 6.9 deep-scrub ok 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.090017+0000 osd.3 (osd.3) 356 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.092994+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.192800+0000 mgr.y (mgr.24526) 590 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.241539+0000 mgr.y (mgr.24526) 591 : cluster [DBG] pgmap v934: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.304426+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.321262+0000 mon.b (mon.1) 460 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.591 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.322153+0000 mon.a (mon.0) 3029 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]': finished 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.322272+0000 mon.a (mon.0) 3030 : audit [INF] from='client.49628 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.322340+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.83:0/615540304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.322379+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:08.323869+0000 mon.a (mon.0) 3033 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]: dispatch 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.594287+0000 osd.2 (osd.2) 267 : cluster [DBG] 64.9 scrub starts 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:08.598175+0000 osd.2 (osd.2) 268 : cluster [DBG] 64.9 scrub ok 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:09.239612+0000 mon.a (mon.0) 3034 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: audit 2024-09-10T19:04:09.249327+0000 mon.a (mon.0) 3035 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-100"}]': finished 2024-09-10T19:04:09.592 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:09 smithi083 bash[32381]: cluster 2024-09-10T19:04:09.249425+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:04:09.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:09.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:09.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:09.841 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:09.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.980 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:09.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:10.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:10.115 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:10.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:10.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:10.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:10.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:10.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:10.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:10.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:10.671 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:10.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:10.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:10.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:10.945 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:10.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:11.048 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:11.079 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:11.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:11.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:11.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.349 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:11.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.464 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:11 smithi165 bash[22591]: cluster 2024-09-10T19:04:10.242305+0000 mgr.y (mgr.24526) 592 : cluster [DBG] pgmap v937: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:11.464 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:11 smithi165 bash[22591]: cluster 2024-09-10T19:04:10.254504+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:04:11.464 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:11.487 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:11.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:11 smithi083 bash[18629]: cluster 2024-09-10T19:04:10.242305+0000 mgr.y (mgr.24526) 592 : cluster [DBG] pgmap v937: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:11.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:11 smithi083 bash[18629]: cluster 2024-09-10T19:04:10.254504+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:04:11.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:11 smithi083 bash[32381]: cluster 2024-09-10T19:04:10.242305+0000 mgr.y (mgr.24526) 592 : cluster [DBG] pgmap v937: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:11.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:11 smithi083 bash[32381]: cluster 2024-09-10T19:04:10.254504+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:04:11.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:11.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:11.762 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:11.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.897 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:11.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:12.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.027 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:12.038 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:04:12.038 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:04:12.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:12.163 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:12.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:12.304 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:12.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:12.439 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[18629]: cluster 2024-09-10T19:04:11.270444+0000 mon.a (mon.0) 3038 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[18629]: audit 2024-09-10T19:04:11.273747+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[18629]: audit 2024-09-10T19:04:11.276743+0000 mon.a (mon.0) 3039 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[32381]: cluster 2024-09-10T19:04:11.270444+0000 mon.a (mon.0) 3038 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[32381]: audit 2024-09-10T19:04:11.273747+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[32381]: audit 2024-09-10T19:04:11.276743+0000 mon.a (mon.0) 3039 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:12.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:12 smithi165 bash[22591]: cluster 2024-09-10T19:04:11.270444+0000 mon.a (mon.0) 3038 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:04:12.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:12 smithi165 bash[22591]: audit 2024-09-10T19:04:11.273747+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.554 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:12 smithi165 bash[22591]: audit 2024-09-10T19:04:11.276743+0000 mon.a (mon.0) 3039 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.576 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:12.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:12.714 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:12.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.850 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:12.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.967 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:12] "GET /metrics HTTP/1.1" 200 229002 "" "Prometheus/2.43.0" 2024-09-10T19:04:12.990 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:12.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:13.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:04:13.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:13.131 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:13.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:04:13.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:13.271 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:13.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: cluster 2024-09-10T19:04:12.243067+0000 mgr.y (mgr.24526) 593 : cluster [DBG] pgmap v940: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:13.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: audit 2024-09-10T19:04:12.278027+0000 mon.a (mon.0) 3040 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: cluster 2024-09-10T19:04:12.278407+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:04:13.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: audit 2024-09-10T19:04:12.284290+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:04:13.286 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: audit 2024-09-10T19:04:12.290231+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.287 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[18629]: audit 2024-09-10T19:04:12.292964+0000 mon.a (mon.0) 3042 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: cluster 2024-09-10T19:04:12.243067+0000 mgr.y (mgr.24526) 593 : cluster [DBG] pgmap v940: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: audit 2024-09-10T19:04:12.278027+0000 mon.a (mon.0) 3040 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: cluster 2024-09-10T19:04:12.278407+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:04:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: audit 2024-09-10T19:04:12.284290+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:04:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: audit 2024-09-10T19:04:12.290231+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:13 smithi165 bash[22591]: audit 2024-09-10T19:04:12.292964+0000 mon.a (mon.0) 3042 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.410 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:13.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: cluster 2024-09-10T19:04:12.243067+0000 mgr.y (mgr.24526) 593 : cluster [DBG] pgmap v940: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: audit 2024-09-10T19:04:12.278027+0000 mon.a (mon.0) 3040 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: cluster 2024-09-10T19:04:12.278407+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: audit 2024-09-10T19:04:12.284290+0000 mon.b (mon.1) 462 : audit [DBG] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: audit 2024-09-10T19:04:12.290231+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:13 smithi083 bash[32381]: audit 2024-09-10T19:04:12.292964+0000 mon.a (mon.0) 3042 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:04:13.545 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:13.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.676 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:13.779 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:13.813 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:13.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.951 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:13.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:14.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:14.092 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:14.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:14.229 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:14.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:14.359 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:14.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:14.476 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:14 smithi165 bash[22591]: audit 2024-09-10T19:04:13.280262+0000 mon.a (mon.0) 3043 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:04:14.476 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:14 smithi165 bash[22591]: cluster 2024-09-10T19:04:13.280374+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:04:14.476 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:14 smithi165 bash[22591]: audit 2024-09-10T19:04:13.288958+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.476 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:14 smithi165 bash[22591]: audit 2024-09-10T19:04:13.291414+0000 mon.a (mon.0) 3045 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.476 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:14 smithi165 bash[22591]: cluster 2024-09-10T19:04:14.242479+0000 mon.a (mon.0) 3046 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:14.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[18629]: audit 2024-09-10T19:04:13.280262+0000 mon.a (mon.0) 3043 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:04:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[18629]: cluster 2024-09-10T19:04:13.280374+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:04:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[18629]: audit 2024-09-10T19:04:13.288958+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[18629]: audit 2024-09-10T19:04:13.291414+0000 mon.a (mon.0) 3045 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[18629]: cluster 2024-09-10T19:04:14.242479+0000 mon.a (mon.0) 3046 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[32381]: audit 2024-09-10T19:04:13.280262+0000 mon.a (mon.0) 3043 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:04:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[32381]: cluster 2024-09-10T19:04:13.280374+0000 mon.a (mon.0) 3044 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:04:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[32381]: audit 2024-09-10T19:04:13.288958+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[32381]: audit 2024-09-10T19:04:13.291414+0000 mon.a (mon.0) 3045 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:04:14.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:14 smithi083 bash[32381]: cluster 2024-09-10T19:04:14.242479+0000 mon.a (mon.0) 3046 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:14.557 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:14.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:14.634 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:14.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:14.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:14.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:14.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:15.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:15.047 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:15.150 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:15.184 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:15.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:15.322 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:15.425 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:15.439 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: cluster 2024-09-10T19:04:14.243910+0000 mgr.y (mgr.24526) 594 : cluster [DBG] pgmap v943: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: audit 2024-09-10T19:04:14.302738+0000 mon.a (mon.0) 3047 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: cluster 2024-09-10T19:04:14.302931+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: audit 2024-09-10T19:04:14.307283+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: audit 2024-09-10T19:04:14.309937+0000 mon.a (mon.0) 3049 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: audit 2024-09-10T19:04:14.579653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[18629]: audit 2024-09-10T19:04:14.582393+0000 mon.c (mon.2) 679 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: cluster 2024-09-10T19:04:14.243910+0000 mgr.y (mgr.24526) 594 : cluster [DBG] pgmap v943: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: audit 2024-09-10T19:04:14.302738+0000 mon.a (mon.0) 3047 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:04:15.440 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: cluster 2024-09-10T19:04:14.302931+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:04:15.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: audit 2024-09-10T19:04:14.307283+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: audit 2024-09-10T19:04:14.309937+0000 mon.a (mon.0) 3049 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: audit 2024-09-10T19:04:14.579653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:15.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:15 smithi083 bash[32381]: audit 2024-09-10T19:04:14.582393+0000 mon.c (mon.2) 679 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:15.458 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:15.562 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:15.593 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: cluster 2024-09-10T19:04:14.243910+0000 mgr.y (mgr.24526) 594 : cluster [DBG] pgmap v943: 238 pgs: 32 unknown, 206 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: audit 2024-09-10T19:04:14.302738+0000 mon.a (mon.0) 3047 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:04:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: cluster 2024-09-10T19:04:14.302931+0000 mon.a (mon.0) 3048 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:04:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: audit 2024-09-10T19:04:14.307283+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: audit 2024-09-10T19:04:14.309937+0000 mon.a (mon.0) 3049 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:04:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: audit 2024-09-10T19:04:14.579653+0000 mon.a (mon.0) 3050 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:15 smithi165 bash[22591]: audit 2024-09-10T19:04:14.582393+0000 mon.c (mon.2) 679 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:15.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:15.751 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:15.835 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:04:15.835 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T19:04:15.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:15.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:15.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:15.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:16.025 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:16.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:16.164 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:16.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:16.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:16.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:16.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: audit 2024-09-10T19:04:15.307782+0000 mon.a (mon.0) 3051 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:04:16.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: cluster 2024-09-10T19:04:15.307911+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:04:16.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: audit 2024-09-10T19:04:15.346104+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: audit 2024-09-10T19:04:15.347823+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.413 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: audit 2024-09-10T19:04:15.348818+0000 mon.a (mon.0) 3053 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[18629]: audit 2024-09-10T19:04:15.350349+0000 mon.a (mon.0) 3054 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: audit 2024-09-10T19:04:15.307782+0000 mon.a (mon.0) 3051 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: cluster 2024-09-10T19:04:15.307911+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: audit 2024-09-10T19:04:15.346104+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: audit 2024-09-10T19:04:15.347823+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: audit 2024-09-10T19:04:15.348818+0000 mon.a (mon.0) 3053 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.414 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:16 smithi083 bash[32381]: audit 2024-09-10T19:04:15.350349+0000 mon.a (mon.0) 3054 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.434 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:16.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:16.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: audit 2024-09-10T19:04:15.307782+0000 mon.a (mon.0) 3051 : audit [INF] from='client.27490 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:04:16.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: cluster 2024-09-10T19:04:15.307911+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:04:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: audit 2024-09-10T19:04:15.346104+0000 mon.b (mon.1) 466 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: audit 2024-09-10T19:04:15.347823+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.83:0/3932924178' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: audit 2024-09-10T19:04:15.348818+0000 mon.a (mon.0) 3053 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-6"}]: dispatch 2024-09-10T19:04:16.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:16 smithi165 bash[22591]: audit 2024-09-10T19:04:15.350349+0000 mon.a (mon.0) 3054 : audit [INF] from='client.27490 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-6", "tierpool": "test-rados-api-smithi083-77828-102"}]: dispatch 2024-09-10T19:04:16.571 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:16.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:16.709 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:16.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:16.845 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:16.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:16.969 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:16.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:17.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:17.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:17.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:17.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:17.340 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:17 smithi165 bash[22591]: cluster 2024-09-10T19:04:16.245584+0000 mgr.y (mgr.24526) 595 : cluster [DBG] pgmap v946: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:04:17.341 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:17 smithi165 bash[22591]: cluster 2024-09-10T19:04:16.326940+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:04:17.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:17.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:17.496 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:17.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:17.636 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:17.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:17.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:17 smithi083 bash[18629]: cluster 2024-09-10T19:04:16.245584+0000 mgr.y (mgr.24526) 595 : cluster [DBG] pgmap v946: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:04:17.752 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:17 smithi083 bash[18629]: cluster 2024-09-10T19:04:16.326940+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:04:17.753 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:17 smithi083 bash[32381]: cluster 2024-09-10T19:04:16.245584+0000 mgr.y (mgr.24526) 595 : cluster [DBG] pgmap v946: 238 pgs: 238 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-09-10T19:04:17.753 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:17 smithi083 bash[32381]: cluster 2024-09-10T19:04:16.326940+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:04:17.769 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:17.832 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:17.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:17.900 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:17.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:18.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:18.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:18.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:18.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:18.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:18.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:18.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:18.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:18.445 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: cluster 2024-09-10T19:04:17.331262+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.363420+0000 mon.c (mon.2) 680 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.364036+0000 mon.a (mon.0) 3057 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.365752+0000 mon.c (mon.2) 681 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.366115+0000 mon.a (mon.0) 3058 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.366684+0000 mon.c (mon.2) 682 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[18629]: audit 2024-09-10T19:04:17.366980+0000 mon.a (mon.0) 3059 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: cluster 2024-09-10T19:04:17.331262+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.363420+0000 mon.c (mon.2) 680 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.364036+0000 mon.a (mon.0) 3057 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.365752+0000 mon.c (mon.2) 681 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.366115+0000 mon.a (mon.0) 3058 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.366684+0000 mon.c (mon.2) 682 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:18 smithi083 bash[32381]: audit 2024-09-10T19:04:17.366980+0000 mon.a (mon.0) 3059 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: cluster 2024-09-10T19:04:17.331262+0000 mon.a (mon.0) 3056 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.363420+0000 mon.c (mon.2) 680 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.364036+0000 mon.a (mon.0) 3057 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.365752+0000 mon.c (mon.2) 681 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.366115+0000 mon.a (mon.0) 3058 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.366684+0000 mon.c (mon.2) 682 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:18 smithi165 bash[22591]: audit 2024-09-10T19:04:17.366980+0000 mon.a (mon.0) 3059 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:18.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:18.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:18.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:18.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:18.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:18.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:19.096 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:19.133 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:19.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:19.271 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:19.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:19.411 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:19.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:19.543 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:19.635 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:19.648 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:19.676 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:19.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:19.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: audit 2024-09-10T19:04:18.200903+0000 mgr.y (mgr.24526) 596 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: cluster 2024-09-10T19:04:18.246432+0000 mgr.y (mgr.24526) 597 : cluster [DBG] pgmap v949: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: audit 2024-09-10T19:04:18.347427+0000 mon.a (mon.0) 3060 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: cluster 2024-09-10T19:04:18.347527+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: audit 2024-09-10T19:04:18.359867+0000 mon.c (mon.2) 683 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: audit 2024-09-10T19:04:18.360291+0000 mon.a (mon.0) 3062 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[18629]: cluster 2024-09-10T19:04:19.244185+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: audit 2024-09-10T19:04:18.200903+0000 mgr.y (mgr.24526) 596 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: cluster 2024-09-10T19:04:18.246432+0000 mgr.y (mgr.24526) 597 : cluster [DBG] pgmap v949: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: audit 2024-09-10T19:04:18.347427+0000 mon.a (mon.0) 3060 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: cluster 2024-09-10T19:04:18.347527+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:04:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: audit 2024-09-10T19:04:18.359867+0000 mon.c (mon.2) 683 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: audit 2024-09-10T19:04:18.360291+0000 mon.a (mon.0) 3062 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:19 smithi083 bash[32381]: cluster 2024-09-10T19:04:19.244185+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:19.811 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:19.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:19.950 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:19.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:20.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:20.087 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: audit 2024-09-10T19:04:18.200903+0000 mgr.y (mgr.24526) 596 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: cluster 2024-09-10T19:04:18.246432+0000 mgr.y (mgr.24526) 597 : cluster [DBG] pgmap v949: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: audit 2024-09-10T19:04:18.347427+0000 mon.a (mon.0) 3060 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: cluster 2024-09-10T19:04:18.347527+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: audit 2024-09-10T19:04:18.359867+0000 mon.c (mon.2) 683 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: audit 2024-09-10T19:04:18.360291+0000 mon.a (mon.0) 3062 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:19 smithi165 bash[22591]: cluster 2024-09-10T19:04:19.244185+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:20.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:20.223 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:20.327 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:20.359 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:20.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:20.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:20.491 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:20.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:20.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:20.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:20.738 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:20 smithi165 bash[22591]: cluster 2024-09-10T19:04:19.698438+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:04:20.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:20 smithi165 bash[22591]: cluster 2024-09-10T19:04:20.247233+0000 mgr.y (mgr.24526) 598 : cluster [DBG] pgmap v952: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:20.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:20 smithi165 bash[22591]: audit 2024-09-10T19:04:20.259369+0000 mon.a (mon.0) 3065 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:20.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:20 smithi165 bash[22591]: cluster 2024-09-10T19:04:20.259477+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:04:20.760 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:20.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:20.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- 172.21.15.83:0/1190778558 >> v1:172.21.15.83:6790/0 conn(0x7f613c0fbf30 legacy=0x7f613c10e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:20.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- 172.21.15.83:0/1190778558 shutdown_connections 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- 172.21.15.83:0/1190778558 >> 172.21.15.83:0/1190778558 conn(0x7f613c0f70b0 msgr2=0x7f613c0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- 172.21.15.83:0/1190778558 shutdown_connections 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- 172.21.15.83:0/1190778558 wait complete. 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 librados: starting msgr at 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 librados: starting objecter 2024-09-10T19:04:20.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 Processor -- start 2024-09-10T19:04:20.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- start start 2024-09-10T19:04:20.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 librados: setting wanted keys 2024-09-10T19:04:20.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 librados: calling monclient init 2024-09-10T19:04:20.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f613c122aa0 con 0x7f613c0fb390 2024-09-10T19:04:20.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f613c123ca0 con 0x7f613c10f870 2024-09-10T19:04:20.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.889+0000 7f6144038640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f613c124ea0 con 0x7f613c11ed10 2024-09-10T19:04:20.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6141dd4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f613c0fb390 0x7f613c10bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41240/0 (socket says 172.21.15.83:41240) 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6141dd4640 1 -- 172.21.15.83:0/483015771 learned_addr learned my addr 172.21.15.83:0/483015771 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter ms_handle_connect 0x7f613c0fb390 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter ms_handle_connect 0x7f613c11ed10 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter ms_handle_connect 0x7f613c10f870 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276792094 0 0) 0x7f613c122aa0 con 0x7f613c0fb390 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61280034e0 con 0x7f613c0fb390 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072350546 0 0) 0x7f613c123ca0 con 0x7f613c10f870 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c122aa0 con 0x7f613c10f870 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755149325 0 0) 0x7f613c124ea0 con 0x7f613c11ed10 2024-09-10T19:04:20.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c123ca0 con 0x7f613c11ed10 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244176139 0 0) 0x7f61280034e0 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f613c124ea0 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f612c001920 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1722342587 0 0) 0x7f613c124ea0 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6790/0 conn(0x7f613c11ed10 legacy=0x7f613c1211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.165:6789/0 conn(0x7f613c10f870 legacy=0x7f613c10c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c1260a0 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c125090 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6144038640 10 client.25569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c1254e0 con 0x7f613c0fb390 2024-09-10T19:04:20.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6144038640 1 librados: init done 2024-09-10T19:04:20.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f612c002420 con 0x7f613c0fb390 2024-09-10T19:04:20.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f612c005a50 con 0x7f613c0fb390 2024-09-10T19:04:20.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.893+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6120004ff0 con 0x7f613c0fb390 2024-09-10T19:04:20.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f612c001fc0 con 0x7f613c0fb390 2024-09-10T19:04:20.899 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:20.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(617..617 src has 69..617) v4 ==== 8046+0+0 (unknown 464188656 0 0) 0x7f612c08a640 con 0x7f613c0fb390 2024-09-10T19:04:20.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 osd_map(617..617 src has 69..617) v4 2024-09-10T19:04:20.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 3 client.25569.objecter handle_osd_map got epochs [617,617] > 0 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 3 client.25569.objecter handle_osd_map decoding full epoch 617 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 20 client.25569.objecter dump_active .. 0 homeless 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v617) v1 ==== 54+0+16 (unknown 1533357929 0 2420000192) 0x7f612c08a9b0 con 0x7f613c0fb390 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 10 client.25569.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 1 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 1 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 0 1 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 4 2024-09-10T19:04:20.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 10 client.25569.objecter _send_command 1 2024-09-10T19:04:20.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6120005290 con 0x7f6120004610 2024-09-10T19:04:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f61280713c0 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f6120004610 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6120005290 con 0x7f6120004610 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 10 client.25569.objecter _finish_command 1 = system:0 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 4 2024-09-10T19:04:20.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.897+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 0 1 2024-09-10T19:04:20.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 10 client.25569.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 2 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 2 2024-09-10T19:04:20.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120005810 osd=0 3 2024-09-10T19:04:20.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 0 2 2024-09-10T19:04:20.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 4 2024-09-10T19:04:20.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 10 client.25569.objecter _send_command 2 2024-09-10T19:04:20.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.917+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6120004c60 con 0x7f6120004610 2024-09-10T19:04:20.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6120004c60 con 0x7f6120004610 2024-09-10T19:04:20.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:20.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 10 client.25569.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:20.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 4 2024-09-10T19:04:20.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 0 2 2024-09-10T19:04:20.923 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 10 client.25569.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 1 3 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 4 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 10 client.25569.objecter _send_command 3 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6120003b10 con 0x7f61200035e0 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f61200035e0 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6120003b10 con 0x7f61200035e0 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 10 client.25569.objecter _finish_command 3 = system:0 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 4 2024-09-10T19:04:20.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.921+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 1 3 2024-09-10T19:04:20.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 10 client.25569.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 4 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 4 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120004e10 osd=1 3 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 1 4 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 4 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 10 client.25569.objecter _send_command 4 2024-09-10T19:04:20.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6120003060 con 0x7f61200035e0 2024-09-10T19:04:20.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6120003060 con 0x7f61200035e0 2024-09-10T19:04:20.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:20.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 10 client.25569.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:20.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 4 2024-09-10T19:04:20.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 1 4 2024-09-10T19:04:20.941 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 10 client.25569.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 5 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 5 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 2 5 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 4 2024-09-10T19:04:20.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 10 client.25569.objecter _send_command 5 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61200022b0 con 0x7f612000d350 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.937+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f612000d350 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.941+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f61200022b0 con 0x7f612000d350 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.941+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.941+0000 7f61327fc640 10 client.25569.objecter _finish_command 5 = system:0 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.941+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 4 2024-09-10T19:04:20.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.941+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 2 5 2024-09-10T19:04:20.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 10 client.25569.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:20.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 6 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 6 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120001b60 osd=2 3 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 2 6 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 4 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 10 client.25569.objecter _send_command 6 2024-09-10T19:04:20.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6120001650 con 0x7f612000d350 2024-09-10T19:04:20.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6120001650 con 0x7f612000d350 2024-09-10T19:04:20.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:20.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f61327fc640 10 client.25569.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:20.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 4 2024-09-10T19:04:20.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 2 6 2024-09-10T19:04:20.958 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 10 client.25569.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 7 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 7 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 3 7 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 4 2024-09-10T19:04:20.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 10 client.25569.objecter _send_command 7 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.953+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f612000d170 con 0x7f6120011fd0 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f6120011fd0 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f612000d170 con 0x7f6120011fd0 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 10 client.25569.objecter _finish_command 7 = system:0 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 4 2024-09-10T19:04:20.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.957+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 3 7 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 10 client.25569.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 8 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 8 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c990 osd=3 3 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 3 8 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 4 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 10 client.25569.objecter _send_command 8 2024-09-10T19:04:20.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.969+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f612000c410 con 0x7f6120011fd0 2024-09-10T19:04:20.976 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f612000c410 con 0x7f6120011fd0 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 10 client.25569.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 4 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 3 8 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 10 client.25569.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 9 2024-09-10T19:04:20.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 9 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 4 9 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 4 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 10 client.25569.objecter _send_command 9 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f612000c340 con 0x7f6120011b20 2024-09-10T19:04:20.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.973+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f6120011b20 2024-09-10T19:04:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.977+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f612000c340 con 0x7f6120011b20 2024-09-10T19:04:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.977+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:04:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.977+0000 7f61327fc640 10 client.25569.objecter _finish_command 9 = system:0 2024-09-10T19:04:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.977+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 4 2024-09-10T19:04:20.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.977+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 4 9 2024-09-10T19:04:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 10 client.25569.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:20.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 10 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 10 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c1d0 osd=4 3 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 4 10 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 4 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 10 client.25569.objecter _send_command 10 2024-09-10T19:04:20.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.993+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f612000c340 con 0x7f6120011b20 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f612000c340 con 0x7f6120011b20 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 10 client.25569.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 4 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 4 10 2024-09-10T19:04:21.001 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 10 client.25569.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 11 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 11 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.002 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 5 11 2024-09-10T19:04:21.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 4 2024-09-10T19:04:21.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 10 client.25569.objecter _send_command 11 2024-09-10T19:04:21.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6120017e90 con 0x7f61200178b0 2024-09-10T19:04:21.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:20.997+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f61200178b0 2024-09-10T19:04:21.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:21.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.001+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6120017e90 con 0x7f61200178b0 2024-09-10T19:04:21.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.001+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:04:21.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.001+0000 7f61327fc640 10 client.25569.objecter _finish_command 11 = system:0 2024-09-10T19:04:21.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.001+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 4 2024-09-10T19:04:21.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.001+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 5 11 2024-09-10T19:04:21.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 10 client.25569.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:21.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 12 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 12 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f612000c340 osd=5 3 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 5 12 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 4 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 10 client.25569.objecter _send_command 12 2024-09-10T19:04:21.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.021+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6120017e50 con 0x7f61200178b0 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6120017e50 con 0x7f61200178b0 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f61327fc640 10 client.25569.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 4 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 5 12 2024-09-10T19:04:21.033 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 10 client.25569.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 13 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 13 2024-09-10T19:04:21.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 6 13 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 4 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 10 client.25569.objecter _send_command 13 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.025+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6120022ca0 con 0x7f612001eff0 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f612001eff0 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6120022ca0 con 0x7f612001eff0 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 10 client.25569.objecter _finish_command 13 = system:0 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 4 2024-09-10T19:04:21.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.029+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 6 13 2024-09-10T19:04:21.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[18629]: cluster 2024-09-10T19:04:19.698438+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:04:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[18629]: cluster 2024-09-10T19:04:20.247233+0000 mgr.y (mgr.24526) 598 : cluster [DBG] pgmap v952: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[18629]: audit 2024-09-10T19:04:20.259369+0000 mon.a (mon.0) 3065 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:21.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[18629]: cluster 2024-09-10T19:04:20.259477+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:04:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[32381]: cluster 2024-09-10T19:04:19.698438+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:04:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[32381]: cluster 2024-09-10T19:04:20.247233+0000 mgr.y (mgr.24526) 598 : cluster [DBG] pgmap v952: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[32381]: audit 2024-09-10T19:04:20.259369+0000 mon.a (mon.0) 3065 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi083-77828-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:20 smithi083 bash[32381]: cluster 2024-09-10T19:04:20.259477+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 10 client.25569.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 14 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 14 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f6120017e50 osd=6 3 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 6 14 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 4 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 10 client.25569.objecter _send_command 14 2024-09-10T19:04:21.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f61200008d0 con 0x7f612001eff0 2024-09-10T19:04:21.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f61200008d0 con 0x7f612001eff0 2024-09-10T19:04:21.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:21.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f61327fc640 10 client.25569.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:21.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 4 2024-09-10T19:04:21.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 6 14 2024-09-10T19:04:21.054 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:04:21.054 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:04:21.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 10 client.25569.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 15 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 15 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 7 15 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 4 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 10 client.25569.objecter _send_command 15 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.049+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6120023cb0 con 0x7f6120024a50 2024-09-10T19:04:21.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 10 client.25569.objecter ms_handle_connect 0x7f6120024a50 2024-09-10T19:04:21.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6120023cb0 con 0x7f6120024a50 2024-09-10T19:04:21.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:04:21.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 10 client.25569.objecter _finish_command 15 = system:0 2024-09-10T19:04:21.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 4 2024-09-10T19:04:21.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.053+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 7 15 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 10 client.25569.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign -1 16 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 20 client.25569.objecter _get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 15 client.25569.objecter _session_command_op_remove -1 16 2024-09-10T19:04:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 20 client.25569.objecter get_session s=0x7f61200008d0 osd=7 3 2024-09-10T19:04:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 15 client.25569.objecter _session_command_op_assign 7 16 2024-09-10T19:04:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 4 2024-09-10T19:04:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 10 client.25569.objecter _send_command 16 2024-09-10T19:04:21.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.065+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6120023cb0 con 0x7f6120024a50 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f61327fc640 1 -- 172.21.15.83:0/483015771 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f6120023cb0 con 0x7f6120024a50 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f61327fc640 10 client.25569.objecter ms_dispatch 0x7f613c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f61327fc640 10 client.25569.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f61327fc640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 4 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f61327fc640 15 client.25569.objecter _session_command_op_remove 7 16 2024-09-10T19:04:21.073 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 librados: watch_flush enter 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 librados: watch_flush exit 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.0 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6801/2535527483 conn(0x7f6120004610 legacy=0x7f6120002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120005810 osd=0 1 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.1 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6805/2309624637 conn(0x7f61200035e0 legacy=0x7f612000aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120004e10 osd=1 1 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.2 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6809/386094220 conn(0x7f612000d350 legacy=0x7f612000f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120001b60 osd=2 1 2024-09-10T19:04:21.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.3 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6813/1279688021 conn(0x7f6120011fd0 legacy=0x7f61200143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c990 osd=3 1 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.4 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.165:6800/3225515265 conn(0x7f6120011b20 legacy=0x7f6120016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c1d0 osd=4 1 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.5 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.165:6804/571173777 conn(0x7f61200178b0 legacy=0x7f612001d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f612000c340 osd=5 1 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.6 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.165:6808/1317922936 conn(0x7f612001eff0 legacy=0x7f61200213f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f6120017e50 osd=6 1 2024-09-10T19:04:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter close_session for osd.7 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.165:6812/1610151650 conn(0x7f6120024a50 legacy=0x7f6120024ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter put_session s=0x7f61200008d0 osd=7 1 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 20 client.25569.objecter shutdown clearing up homeless session... 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 10 client.25569.objecter successfully canceled tick 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.069+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6800/3071793735 conn(0x7f61280713c0 legacy=0x7f6128073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> v1:172.21.15.83:6789/0 conn(0x7f613c0fb390 legacy=0x7f613c10bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f61425d5640 1 -- 172.21.15.83:0/483015771 reap_dead start 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f61425d5640 1 -- 172.21.15.83:0/483015771 reap_dead start 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f61425d5640 1 -- 172.21.15.83:0/483015771 reap_dead start 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f61425d5640 1 -- 172.21.15.83:0/483015771 reap_dead start 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f61425d5640 1 -- 172.21.15.83:0/483015771 reap_dead start 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 shutdown_connections 2024-09-10T19:04:21.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 >> 172.21.15.83:0/483015771 conn(0x7f613c0f70b0 msgr2=0x7f613c0f7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:21.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 shutdown_connections 2024-09-10T19:04:21.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 -- 172.21.15.83:0/483015771 wait complete. 2024-09-10T19:04:21.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:21.073+0000 7f6144038640 1 librados: shutdown 2024-09-10T19:04:21.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:21.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:21.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.296 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:21.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:21.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.551 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:21.566 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:21.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:21.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:21.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:21.835 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:21.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:21.970 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:21.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:22.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.086 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:22 smithi165 bash[22591]: audit 2024-09-10T19:04:20.897416+0000 mon.a (mon.0) 3067 : audit [DBG] from='client.? v1:172.21.15.83:0/483015771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:22.086 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:22 smithi165 bash[22591]: cluster 2024-09-10T19:04:21.263707+0000 mon.a (mon.0) 3068 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:04:22.108 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:22.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:22.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:22 smithi083 bash[18629]: audit 2024-09-10T19:04:20.897416+0000 mon.a (mon.0) 3067 : audit [DBG] from='client.? v1:172.21.15.83:0/483015771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:22.224 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:22 smithi083 bash[18629]: cluster 2024-09-10T19:04:21.263707+0000 mon.a (mon.0) 3068 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:04:22.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:22 smithi083 bash[32381]: audit 2024-09-10T19:04:20.897416+0000 mon.a (mon.0) 3067 : audit [DBG] from='client.? v1:172.21.15.83:0/483015771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:22.224 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:22 smithi083 bash[32381]: cluster 2024-09-10T19:04:21.263707+0000 mon.a (mon.0) 3068 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:04:22.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:22.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:22.373 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:22.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:22.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:22.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:22.753 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:22.786 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:22.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:22.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:22.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:23.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:22] "GET /metrics HTTP/1.1" 200 228989 "" "Prometheus/2.43.0" 2024-09-10T19:04:23.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:23.163 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:23.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:23.333 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:23.335 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[18629]: cluster 2024-09-10T19:04:22.248049+0000 mgr.y (mgr.24526) 599 : cluster [DBG] pgmap v955: 182 pgs: 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:23.336 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[18629]: cluster 2024-09-10T19:04:22.263807+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:04:23.336 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[18629]: audit 2024-09-10T19:04:22.269715+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.336 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[18629]: audit 2024-09-10T19:04:22.274116+0000 mon.a (mon.0) 3070 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.470 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:23.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:23.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:23 smithi165 bash[22591]: cluster 2024-09-10T19:04:22.248049+0000 mgr.y (mgr.24526) 599 : cluster [DBG] pgmap v955: 182 pgs: 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:23.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:23 smithi165 bash[22591]: cluster 2024-09-10T19:04:22.263807+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:04:23.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:23 smithi165 bash[22591]: audit 2024-09-10T19:04:22.269715+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.586 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:23 smithi165 bash[22591]: audit 2024-09-10T19:04:22.274116+0000 mon.a (mon.0) 3070 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[32381]: cluster 2024-09-10T19:04:22.248049+0000 mgr.y (mgr.24526) 599 : cluster [DBG] pgmap v955: 182 pgs: 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:23.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[32381]: cluster 2024-09-10T19:04:22.263807+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:04:23.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[32381]: audit 2024-09-10T19:04:22.269715+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:23 smithi083 bash[32381]: audit 2024-09-10T19:04:22.274116+0000 mon.a (mon.0) 3070 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.607 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:23.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:23.743 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:23.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.879 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:23.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:23.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:24.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:24.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:24.259 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:24.300 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:24.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[18629]: audit 2024-09-10T19:04:23.266525+0000 mon.a (mon.0) 3071 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[18629]: cluster 2024-09-10T19:04:23.266674+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:04:24.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[18629]: cluster 2024-09-10T19:04:24.248306+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:24.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[32381]: audit 2024-09-10T19:04:23.266525+0000 mon.a (mon.0) 3071 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[32381]: cluster 2024-09-10T19:04:23.266674+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:04:24.391 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:24 smithi083 bash[32381]: cluster 2024-09-10T19:04:24.248306+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:24.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:24.435 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:24.539 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.572 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:24 smithi165 bash[22591]: audit 2024-09-10T19:04:23.266525+0000 mon.a (mon.0) 3071 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:24 smithi165 bash[22591]: cluster 2024-09-10T19:04:23.266674+0000 mon.a (mon.0) 3072 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:04:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:24 smithi165 bash[22591]: cluster 2024-09-10T19:04:24.248306+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:24.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:24.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:24.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.842 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:24.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:24.978 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:24.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:25.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.118 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:25.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:25.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:25.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:25 smithi165 bash[22591]: cluster 2024-09-10T19:04:24.249087+0000 mgr.y (mgr.24526) 600 : cluster [DBG] pgmap v958: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:25.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:25 smithi165 bash[22591]: cluster 2024-09-10T19:04:24.288136+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:04:25.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:25 smithi165 bash[22591]: audit 2024-09-10T19:04:24.296261+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:25 smithi165 bash[22591]: audit 2024-09-10T19:04:24.300701+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:25.390 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:25.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[32381]: cluster 2024-09-10T19:04:24.249087+0000 mgr.y (mgr.24526) 600 : cluster [DBG] pgmap v958: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[32381]: cluster 2024-09-10T19:04:24.288136+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:04:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[32381]: audit 2024-09-10T19:04:24.296261+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[32381]: audit 2024-09-10T19:04:24.300701+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[18629]: cluster 2024-09-10T19:04:24.249087+0000 mgr.y (mgr.24526) 600 : cluster [DBG] pgmap v958: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:25.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[18629]: cluster 2024-09-10T19:04:24.288136+0000 mon.a (mon.0) 3074 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:04:25.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[18629]: audit 2024-09-10T19:04:24.296261+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:25 smithi083 bash[18629]: audit 2024-09-10T19:04:24.300701+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:25.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:04:25.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:25.662 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:25.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.799 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:25.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:25.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:25.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:26.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:26.072 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:26.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:26.205 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:26.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:26.328 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:26.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:26.467 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:26.570 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:26.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[18629]: audit 2024-09-10T19:04:25.302468+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[18629]: cluster 2024-09-10T19:04:25.302582+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:04:26.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[18629]: audit 2024-09-10T19:04:25.303407+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[18629]: audit 2024-09-10T19:04:25.307618+0000 mon.a (mon.0) 3078 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[32381]: audit 2024-09-10T19:04:25.302468+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[32381]: cluster 2024-09-10T19:04:25.302582+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:04:26.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[32381]: audit 2024-09-10T19:04:25.303407+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:26 smithi083 bash[32381]: audit 2024-09-10T19:04:25.307618+0000 mon.a (mon.0) 3078 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.603 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:26 smithi165 bash[22591]: audit 2024-09-10T19:04:25.302468+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:26 smithi165 bash[22591]: cluster 2024-09-10T19:04:25.302582+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:04:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:26 smithi165 bash[22591]: audit 2024-09-10T19:04:25.303407+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:26 smithi165 bash[22591]: audit 2024-09-10T19:04:25.307618+0000 mon.a (mon.0) 3078 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:26.706 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:26.740 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:26.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:26.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:26.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.021 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:27.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:27.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:27.297 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:27.401 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:27.430 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:27.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:27.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:27.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: cluster 2024-09-10T19:04:26.251412+0000 mgr.y (mgr.24526) 601 : cluster [DBG] pgmap v961: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: audit 2024-09-10T19:04:26.305512+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: audit 2024-09-10T19:04:26.305782+0000 mon.a (mon.0) 3079 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: cluster 2024-09-10T19:04:26.305866+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: audit 2024-09-10T19:04:26.308786+0000 mon.a (mon.0) 3081 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: audit 2024-09-10T19:04:27.307985+0000 mon.a (mon.0) 3082 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.573 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[32381]: cluster 2024-09-10T19:04:27.308194+0000 mon.a (mon.0) 3083 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: cluster 2024-09-10T19:04:26.251412+0000 mgr.y (mgr.24526) 601 : cluster [DBG] pgmap v961: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: audit 2024-09-10T19:04:26.305512+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: audit 2024-09-10T19:04:26.305782+0000 mon.a (mon.0) 3079 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: cluster 2024-09-10T19:04:26.305866+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: audit 2024-09-10T19:04:26.308786+0000 mon.a (mon.0) 3081 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: audit 2024-09-10T19:04:27.307985+0000 mon.a (mon.0) 3082 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.574 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:27 smithi083 bash[18629]: cluster 2024-09-10T19:04:27.308194+0000 mon.a (mon.0) 3083 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:04:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: cluster 2024-09-10T19:04:26.251412+0000 mgr.y (mgr.24526) 601 : cluster [DBG] pgmap v961: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: audit 2024-09-10T19:04:26.305512+0000 mon.b (mon.1) 471 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: audit 2024-09-10T19:04:26.305782+0000 mon.a (mon.0) 3079 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: cluster 2024-09-10T19:04:26.305866+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:04:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: audit 2024-09-10T19:04:26.308786+0000 mon.a (mon.0) 3081 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: audit 2024-09-10T19:04:27.307985+0000 mon.a (mon.0) 3082 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-107", "overlaypool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:27.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:27 smithi165 bash[22591]: cluster 2024-09-10T19:04:27.308194+0000 mon.a (mon.0) 3083 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:04:27.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:27.712 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:27.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:27.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:27.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.986 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:27.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:28.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:28.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:28.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:28.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:28.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:28 smithi165 bash[22591]: audit 2024-09-10T19:04:27.309127+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:28 smithi165 bash[22591]: audit 2024-09-10T19:04:27.312703+0000 mon.a (mon.0) 3084 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:28 smithi165 bash[22591]: cluster 2024-09-10T19:04:28.301914+0000 mon.a (mon.0) 3085 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:28.363 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:28.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:28.393 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:28.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:28.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[18629]: audit 2024-09-10T19:04:27.309127+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[18629]: audit 2024-09-10T19:04:27.312703+0000 mon.a (mon.0) 3084 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[18629]: cluster 2024-09-10T19:04:28.301914+0000 mon.a (mon.0) 3085 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:28.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[32381]: audit 2024-09-10T19:04:27.309127+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[32381]: audit 2024-09-10T19:04:27.312703+0000 mon.a (mon.0) 3084 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:28.512 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:28 smithi083 bash[32381]: cluster 2024-09-10T19:04:28.301914+0000 mon.a (mon.0) 3085 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:28.529 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:28.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:28.667 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:28.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:28.804 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:28.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:28.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:28.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:29.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:29.078 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:29.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:29.216 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:29.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:29.338 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:28.208899+0000 mgr.y (mgr.24526) 602 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: cluster 2024-09-10T19:04:28.252122+0000 mgr.y (mgr.24526) 603 : cluster [DBG] pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:28.310921+0000 mon.a (mon.0) 3086 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: cluster 2024-09-10T19:04:28.311092+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:28.341104+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:28.343726+0000 mon.a (mon.0) 3088 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:29.260253+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:29.260293+0000 mon.a (mon.0) 3089 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]': finished 2024-09-10T19:04:29.341 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: cluster 2024-09-10T19:04:29.260548+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[18629]: audit 2024-09-10T19:04:29.263597+0000 mon.a (mon.0) 3091 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:28.208899+0000 mgr.y (mgr.24526) 602 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: cluster 2024-09-10T19:04:28.252122+0000 mgr.y (mgr.24526) 603 : cluster [DBG] pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:28.310921+0000 mon.a (mon.0) 3086 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: cluster 2024-09-10T19:04:28.311092+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:28.341104+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:28.343726+0000 mon.a (mon.0) 3088 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:29.260253+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.342 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:29.260293+0000 mon.a (mon.0) 3089 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]': finished 2024-09-10T19:04:29.343 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: cluster 2024-09-10T19:04:29.260548+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:04:29.343 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:29 smithi083 bash[32381]: audit 2024-09-10T19:04:29.263597+0000 mon.a (mon.0) 3091 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:29.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:29.448 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:04:29.448 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:04:29.448 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T19:04:29.448 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T19:04:29.448 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:28.208899+0000 mgr.y (mgr.24526) 602 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: cluster 2024-09-10T19:04:28.252122+0000 mgr.y (mgr.24526) 603 : cluster [DBG] pgmap v964: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:28.310921+0000 mon.a (mon.0) 3086 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: cluster 2024-09-10T19:04:28.311092+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:28.341104+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:28.343726+0000 mon.a (mon.0) 3088 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]: dispatch 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:29.260253+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.83:0/781376496' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:29.260293+0000 mon.a (mon.0) 3089 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-107"}]': finished 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: cluster 2024-09-10T19:04:29.260548+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:04:29.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:29 smithi165 bash[22591]: audit 2024-09-10T19:04:29.263597+0000 mon.a (mon.0) 3091 : audit [INF] from='client.50158 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]: dispatch 2024-09-10T19:04:29.477 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:29.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:29.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:29.747 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:29.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:29.881 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:29.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:29.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.020 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:30.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:30.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:30.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:30.289 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:30.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.429 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:30.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.564 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:30.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:30.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:30.701 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: audit 2024-09-10T19:04:29.576640+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: audit 2024-09-10T19:04:29.579738+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: cluster 2024-09-10T19:04:30.252937+0000 mgr.y (mgr.24526) 604 : cluster [DBG] pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: cluster 2024-09-10T19:04:30.254424+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: audit 2024-09-10T19:04:30.264553+0000 mon.a (mon.0) 3094 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[32381]: cluster 2024-09-10T19:04:30.264665+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: audit 2024-09-10T19:04:29.576640+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: audit 2024-09-10T19:04:29.579738+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: cluster 2024-09-10T19:04:30.252937+0000 mgr.y (mgr.24526) 604 : cluster [DBG] pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: cluster 2024-09-10T19:04:30.254424+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: audit 2024-09-10T19:04:30.264553+0000 mon.a (mon.0) 3094 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:30.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:30 smithi083 bash[18629]: cluster 2024-09-10T19:04:30.264665+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:04:30.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:30.834 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: audit 2024-09-10T19:04:29.576640+0000 mon.a (mon.0) 3092 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: audit 2024-09-10T19:04:29.579738+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: cluster 2024-09-10T19:04:30.252937+0000 mgr.y (mgr.24526) 604 : cluster [DBG] pgmap v967: 246 pgs: 246 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: cluster 2024-09-10T19:04:30.254424+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: audit 2024-09-10T19:04:30.264553+0000 mon.a (mon.0) 3094 : audit [INF] from='client.50158 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-107", "tierpool": "test-rados-api-smithi083-77828-107-cache"}]': finished 2024-09-10T19:04:30.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:30 smithi165 bash[22591]: cluster 2024-09-10T19:04:30.264665+0000 mon.a (mon.0) 3095 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:04:30.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:30.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:30.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:31.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- 172.21.15.83:0/2302054289 >> v1:172.21.15.83:6790/0 conn(0x7f552410b130 legacy=0x7f552410d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- 172.21.15.83:0/2302054289 shutdown_connections 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- 172.21.15.83:0/2302054289 >> 172.21.15.83:0/2302054289 conn(0x7f55240fa370 msgr2=0x7f55240fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- 172.21.15.83:0/2302054289 shutdown_connections 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- 172.21.15.83:0/2302054289 wait complete. 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 librados: starting msgr at 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 librados: starting objecter 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 Processor -- start 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- start start 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 librados: setting wanted keys 2024-09-10T19:04:31.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 librados: calling monclient init 2024-09-10T19:04:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5524114520 con 0x7f55241075a0 2024-09-10T19:04:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f552411f610 con 0x7f5524104810 2024-09-10T19:04:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.069+0000 7f552a8ca640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55241207f0 con 0x7f552410b130 2024-09-10T19:04:31.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5528e67640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f552410b130 0x7f552411cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51822/0 (socket says 172.21.15.83:51822) 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f55237fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f55241075a0 0x7f55241197b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41456/0 (socket says 172.21.15.83:41456) 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5528e67640 1 -- 172.21.15.83:0/2612781513 learned_addr learned my addr 172.21.15.83:0/2612781513 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter ms_handle_connect 0x7f552410b130 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter ms_handle_connect 0x7f55241075a0 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter ms_handle_connect 0x7f5524104810 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195299509 0 0) 0x7f55241207f0 con 0x7f552410b130 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c003120 con 0x7f552410b130 2024-09-10T19:04:31.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199606854 0 0) 0x7f5524114520 con 0x7f55241075a0 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55241207f0 con 0x7f55241075a0 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365786321 0 0) 0x7f552411f610 con 0x7f5524104810 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5524114520 con 0x7f5524104810 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1430927484 0 0) 0x7f550c003120 con 0x7f552410b130 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f552411f610 con 0x7f552410b130 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f551c001950 con 0x7f552410b130 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908839520 0 0) 0x7f5524114520 con 0x7f5524104810 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f550c003120 con 0x7f5524104810 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784563175 0 0) 0x7f55241207f0 con 0x7f55241075a0 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5524114520 con 0x7f55241075a0 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5514001910 con 0x7f5524104810 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5518001840 con 0x7f55241075a0 2024-09-10T19:04:31.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 223398649 0 0) 0x7f552411f610 con 0x7f552410b130 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 >> v1:172.21.15.165:6789/0 conn(0x7f5524104810 legacy=0x7f5524113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 >> v1:172.21.15.83:6789/0 conn(0x7f55241075a0 legacy=0x7f55241197b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55241219d0 con 0x7f552410b130 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552411e660 con 0x7f552410b130 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f552a8ca640 10 client.49658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f552411eb30 con 0x7f552410b130 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f552a8ca640 1 librados: init done 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f551c001550 con 0x7f552410b130 2024-09-10T19:04:31.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f551c003ad0 con 0x7f552410b130 2024-09-10T19:04:31.079 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:31.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5510005010 con 0x7f552410b130 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.073+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f551c001c00 con 0x7f552410b130 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(627..627 src has 69..627) v4 ==== 8824+0+0 (unknown 3441864438 0 0) 0x7f551c08ae00 con 0x7f552410b130 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 10 client.49658.objecter ms_dispatch 0x7f5524000b90 osd_map(627..627 src has 69..627) v4 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 3 client.49658.objecter handle_osd_map got epochs [627,627] > 0 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 3 client.49658.objecter handle_osd_map decoding full epoch 627 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 20 client.49658.objecter dump_active .. 0 homeless 2024-09-10T19:04:31.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.077+0000 7f5520ff9640 10 client.49658.objecter ms_handle_connect 0x7f550c0718a0 2024-09-10T19:04:31.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.085+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f551c057090 con 0x7f552410b130 2024-09-10T19:04:31.103 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:31.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:31.242 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:31.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.273+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f55100052c0 con 0x7f552410b130 2024-09-10T19:04:31.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:04:31.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:31.380 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:31.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:31.496 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:31.520 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- 172.21.15.83:0/1853747243 >> v1:172.21.15.83:6789/0 conn(0x7f58c0107e00 legacy=0x7f58c010a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- 172.21.15.83:0/1853747243 shutdown_connections 2024-09-10T19:04:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- 172.21.15.83:0/1853747243 >> 172.21.15.83:0/1853747243 conn(0x7f58c00f7070 msgr2=0x7f58c00f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- 172.21.15.83:0/1853747243 shutdown_connections 2024-09-10T19:04:31.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- 172.21.15.83:0/1853747243 wait complete. 2024-09-10T19:04:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 librados: starting msgr at 2024-09-10T19:04:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 librados: starting objecter 2024-09-10T19:04:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 Processor -- start 2024-09-10T19:04:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- start start 2024-09-10T19:04:31.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 librados: setting wanted keys 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 librados: calling monclient init 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f58c011e130 con 0x7f58c0101530 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f58c011f2f0 con 0x7f58c0107e00 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58c62f2640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f58c01204f0 con 0x7f58c01042a0 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bf7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f58c01042a0 0x7f58c0065e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51834/0 (socket says 172.21.15.83:51834) 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bf7fe640 1 -- 172.21.15.83:0/3180615956 learned_addr learned my addr 172.21.15.83:0/3180615956 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bffff640 1 --1- 172.21.15.83:0/3180615956 >> v1:172.21.15.83:6789/0 conn(0x7f58c0101530 0x7f58c0065770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41470/0 (socket says 172.21.15.83:41470) 2024-09-10T19:04:31.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 10 client.?.objecter ms_handle_connect 0x7f58c01042a0 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 10 client.?.objecter ms_handle_connect 0x7f58c0101530 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048657333 0 0) 0x7f58c01204f0 con 0x7f58c01042a0 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.585+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a80038e0 con 0x7f58c01042a0 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 10 client.?.objecter ms_handle_connect 0x7f58c0107e00 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852346827 0 0) 0x7f58c011e130 con 0x7f58c0101530 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c01204f0 con 0x7f58c0101530 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771348050 0 0) 0x7f58c011f2f0 con 0x7f58c0107e00 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c011e130 con 0x7f58c0107e00 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3039413407 0 0) 0x7f58a80038e0 con 0x7f58c01042a0 2024-09-10T19:04:31.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c011f2f0 con 0x7f58c01042a0 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076682431 0 0) 0x7f58c01204f0 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58a8004490 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f58b0001850 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f58b4002990 con 0x7f58c01042a0 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 503627156 0 0) 0x7f58c011e130 con 0x7f58c0107e00 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c01204f0 con 0x7f58c0107e00 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f58b8002490 con 0x7f58c0107e00 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 41331743 0 0) 0x7f58a8004490 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 >> v1:172.21.15.83:6790/0 conn(0x7f58c01042a0 legacy=0x7f58c0065e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 >> v1:172.21.15.165:6789/0 conn(0x7f58c0107e00 legacy=0x7f58c011c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c01216f0 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c011e320 con 0x7f58c0101530 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58c62f2640 10 client.25587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:31.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c011e7f0 con 0x7f58c0101530 2024-09-10T19:04:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58c62f2640 1 librados: init done 2024-09-10T19:04:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f58b0002180 con 0x7f58c0101530 2024-09-10T19:04:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f58b0005770 con 0x7f58c0101530 2024-09-10T19:04:31.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac005010 con 0x7f58c0101530 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.589+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f58b0001d20 con 0x7f58c0101530 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(628..628 src has 69..628) v4 ==== 8432+0+0 (unknown 3521363818 0 0) 0x7f58b008a3f0 con 0x7f58c0101530 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 10 client.25587.objecter ms_dispatch 0x7f58c0000b90 osd_map(628..628 src has 69..628) v4 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 3 client.25587.objecter handle_osd_map got epochs [628,628] > 0 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 3 client.25587.objecter handle_osd_map decoding full epoch 628 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 20 client.25587.objecter dump_active .. 0 homeless 2024-09-10T19:04:31.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.593+0000 7f58bcff9640 10 client.25587.objecter ms_handle_connect 0x7f58a8071980 2024-09-10T19:04:31.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.597+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f58b0056800 con 0x7f58c0101530 2024-09-10T19:04:31.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:31.659 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:31.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:31.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:31.789+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f58ac0052c0 con 0x7f58c0101530 2024-09-10T19:04:31.795 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:31.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:31.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:31.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:32.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:32.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:32.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:32.206 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:32.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.269+0000 7f58bcff9640 1 -- 172.21.15.83:0/3180615956 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v629) v1 ==== 113+0+0 (unknown 1052755109 0 0) 0x7f58b00561a0 con 0x7f58c0101530 2024-09-10T19:04:32.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.269+0000 7f5520ff9640 1 -- 172.21.15.83:0/2612781513 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v629) v1 ==== 97+0+0 (unknown 2997954089 0 0) 0x7f551c056a30 con 0x7f552410b130 2024-09-10T19:04:32.273 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:04:32.274 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T19:04:32.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 10 librados: watch_flush enter 2024-09-10T19:04:32.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 10 librados: watch_flush exit 2024-09-10T19:04:32.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 20 client.49658.objecter shutdown clearing up homeless session... 2024-09-10T19:04:32.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 10 client.49658.objecter successfully canceled tick 2024-09-10T19:04:32.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 10 librados: watch_flush enter 2024-09-10T19:04:32.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 10 librados: watch_flush exit 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 20 client.25587.objecter shutdown clearing up homeless session... 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 10 client.25587.objecter successfully canceled tick 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 >> v1:172.21.15.83:6800/3071793735 conn(0x7f58a8071980 legacy=0x7f58a8073e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 >> v1:172.21.15.83:6789/0 conn(0x7f58c0101530 legacy=0x7f58c0065770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 shutdown_connections 2024-09-10T19:04:32.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 >> 172.21.15.83:0/3180615956 conn(0x7f58c00f7070 msgr2=0x7f58c010aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:32.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 >> v1:172.21.15.83:6800/3071793735 conn(0x7f550c0718a0 legacy=0x7f550c073d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:32.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 >> v1:172.21.15.83:6790/0 conn(0x7f552410b130 legacy=0x7f552411cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:32.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 shutdown_connections 2024-09-10T19:04:32.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.273+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 >> 172.21.15.83:0/2612781513 conn(0x7f55240fa370 msgr2=0x7f55240fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:32.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 shutdown_connections 2024-09-10T19:04:32.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 shutdown_connections 2024-09-10T19:04:32.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f58c62f2640 1 -- 172.21.15.83:0/3180615956 wait complete. 2024-09-10T19:04:32.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f58c62f2640 1 librados: shutdown 2024-09-10T19:04:32.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f552a8ca640 1 -- 172.21.15.83:0/2612781513 wait complete. 2024-09-10T19:04:32.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:32.281+0000 7f552a8ca640 1 librados: shutdown 2024-09-10T19:04:32.306 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[18629]: cluster 2024-09-10T19:04:31.268063+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:04:32.306 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[18629]: audit 2024-09-10T19:04:31.278181+0000 mon.c (mon.2) 685 : audit [INF] from='client.? v1:172.21.15.83:0/2612781513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[18629]: audit 2024-09-10T19:04:31.278595+0000 mon.a (mon.0) 3097 : audit [INF] from='client.49658 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[18629]: audit 2024-09-10T19:04:31.790905+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[32381]: cluster 2024-09-10T19:04:31.268063+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[32381]: audit 2024-09-10T19:04:31.278181+0000 mon.c (mon.2) 685 : audit [INF] from='client.? v1:172.21.15.83:0/2612781513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[32381]: audit 2024-09-10T19:04:31.278595+0000 mon.a (mon.0) 3097 : audit [INF] from='client.49658 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.307 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[32381]: audit 2024-09-10T19:04:31.790905+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:04:32.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:32.341 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:32.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:32.477 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:32.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:32.593 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:32 smithi165 bash[22591]: cluster 2024-09-10T19:04:31.268063+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:04:32.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:32 smithi165 bash[22591]: audit 2024-09-10T19:04:31.278181+0000 mon.c (mon.2) 685 : audit [INF] from='client.? v1:172.21.15.83:0/2612781513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:32 smithi165 bash[22591]: audit 2024-09-10T19:04:31.278595+0000 mon.a (mon.0) 3097 : audit [INF] from='client.49658 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:04:32.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:32 smithi165 bash[22591]: audit 2024-09-10T19:04:31.790905+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:04:32.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:32.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:32.750 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:32.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:32.867 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:32] "GET /metrics HTTP/1.1" 200 231407 "" "Prometheus/2.43.0" 2024-09-10T19:04:32.885 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:32.989 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.024 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:33.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:33.161 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:33.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:33.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:33.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:33.416 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: cluster 2024-09-10T19:04:32.253788+0000 mgr.y (mgr.24526) 605 : cluster [DBG] pgmap v970: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: cluster 2024-09-10T19:04:32.261107+0000 mon.a (mon.0) 3099 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: audit 2024-09-10T19:04:32.270105+0000 mon.a (mon.0) 3100 : audit [INF] from='client.49658 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: audit 2024-09-10T19:04:32.270297+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: cluster 2024-09-10T19:04:32.270422+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: audit 2024-09-10T19:04:32.306960+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.417 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: audit 2024-09-10T19:04:32.310440+0000 mon.a (mon.0) 3104 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.418 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[32381]: audit 2024-09-10T19:04:32.310925+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:33.418 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: cluster 2024-09-10T19:04:32.253788+0000 mgr.y (mgr.24526) 605 : cluster [DBG] pgmap v970: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:33.418 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: cluster 2024-09-10T19:04:32.261107+0000 mon.a (mon.0) 3099 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:33.418 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: audit 2024-09-10T19:04:32.270105+0000 mon.a (mon.0) 3100 : audit [INF] from='client.49658 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:04:33.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: audit 2024-09-10T19:04:32.270297+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:04:33.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: cluster 2024-09-10T19:04:32.270422+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:04:33.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: audit 2024-09-10T19:04:32.306960+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: audit 2024-09-10T19:04:32.310440+0000 mon.a (mon.0) 3104 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.419 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:33 smithi083 bash[18629]: audit 2024-09-10T19:04:32.310925+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:33.436 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:33.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:33.577 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: cluster 2024-09-10T19:04:32.253788+0000 mgr.y (mgr.24526) 605 : cluster [DBG] pgmap v970: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: cluster 2024-09-10T19:04:32.261107+0000 mon.a (mon.0) 3099 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: audit 2024-09-10T19:04:32.270105+0000 mon.a (mon.0) 3100 : audit [INF] from='client.49658 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:04:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: audit 2024-09-10T19:04:32.270297+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? v1:172.21.15.83:0/3180615956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:04:33.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: cluster 2024-09-10T19:04:32.270422+0000 mon.a (mon.0) 3102 : cluster [DBG] osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:04:33.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: audit 2024-09-10T19:04:32.306960+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: audit 2024-09-10T19:04:32.310440+0000 mon.a (mon.0) 3104 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:33.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:33 smithi165 bash[22591]: audit 2024-09-10T19:04:32.310925+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:33.681 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:33.707 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:33.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:33.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T19:04:33.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.835 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:33.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.968 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:33.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:34.071 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:34.118 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:34.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:34.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:34.373 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:34.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:34.512 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:34.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:34.643 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:34.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:34.776 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:34.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.914 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:34.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:35.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:35.031 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:34 smithi165 bash[22591]: audit 2024-09-10T19:04:33.297098+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:35.031 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:34 smithi165 bash[22591]: cluster 2024-09-10T19:04:33.297169+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:04:35.031 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:34 smithi165 bash[22591]: audit 2024-09-10T19:04:33.298621+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:35.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[18629]: audit 2024-09-10T19:04:33.297098+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[18629]: cluster 2024-09-10T19:04:33.297169+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:04:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[18629]: audit 2024-09-10T19:04:33.298621+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[32381]: audit 2024-09-10T19:04:33.297098+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[32381]: cluster 2024-09-10T19:04:33.297169+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:04:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:34 smithi083 bash[32381]: audit 2024-09-10T19:04:33.298621+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:35.051 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:35.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:35.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:35.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:35.330 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:35.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:35.469 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:35.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:35.603 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:35.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:35.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:04:35.730 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:35.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:35.845 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:35 smithi165 bash[22591]: cluster 2024-09-10T19:04:34.254934+0000 mgr.y (mgr.24526) 606 : cluster [DBG] pgmap v973: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:35.845 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:35 smithi165 bash[22591]: cluster 2024-09-10T19:04:34.761959+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:04:35.845 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:35 smithi165 bash[22591]: audit 2024-09-10T19:04:35.318230+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:35.845 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:35 smithi165 bash[22591]: cluster 2024-09-10T19:04:35.318418+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:04:35.869 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:35.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:35.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.005 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[18629]: cluster 2024-09-10T19:04:34.254934+0000 mgr.y (mgr.24526) 606 : cluster [DBG] pgmap v973: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[18629]: cluster 2024-09-10T19:04:34.761959+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[18629]: audit 2024-09-10T19:04:35.318230+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[18629]: cluster 2024-09-10T19:04:35.318418+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[32381]: cluster 2024-09-10T19:04:34.254934+0000 mgr.y (mgr.24526) 606 : cluster [DBG] pgmap v973: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:36.009 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[32381]: cluster 2024-09-10T19:04:34.761959+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:04:36.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[32381]: audit 2024-09-10T19:04:35.318230+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:36.010 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:35 smithi083 bash[32381]: cluster 2024-09-10T19:04:35.318418+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:04:36.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:36.145 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:36.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:36.381 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.408 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:36.511 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:36.543 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:36.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:36.679 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:36.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:36.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:36.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:37.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:37.082 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:37.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:37.219 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:37.249 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:37.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:37.358 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:37.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:37.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:37 smithi165 bash[22591]: cluster 2024-09-10T19:04:36.257286+0000 mgr.y (mgr.24526) 607 : cluster [DBG] pgmap v976: 190 pgs: 6 creating+peering, 2 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:37.475 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:37 smithi165 bash[22591]: cluster 2024-09-10T19:04:36.322345+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:04:37.475 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:37 smithi165 bash[22591]: audit 2024-09-10T19:04:36.330323+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:37.475 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:37 smithi165 bash[22591]: cluster 2024-09-10T19:04:36.766096+0000 mon.a (mon.0) 3114 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:37.496 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:37.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:37.634 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:37.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:37.769 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:37.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[18629]: cluster 2024-09-10T19:04:36.257286+0000 mgr.y (mgr.24526) 607 : cluster [DBG] pgmap v976: 190 pgs: 6 creating+peering, 2 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[18629]: cluster 2024-09-10T19:04:36.322345+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[18629]: audit 2024-09-10T19:04:36.330323+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[18629]: cluster 2024-09-10T19:04:36.766096+0000 mon.a (mon.0) 3114 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[32381]: cluster 2024-09-10T19:04:36.257286+0000 mgr.y (mgr.24526) 607 : cluster [DBG] pgmap v976: 190 pgs: 6 creating+peering, 2 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[32381]: cluster 2024-09-10T19:04:36.322345+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[32381]: audit 2024-09-10T19:04:36.330323+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:37 smithi083 bash[32381]: cluster 2024-09-10T19:04:36.766096+0000 mon.a (mon.0) 3114 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:37.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:37.905 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:37.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:38.008 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:38.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.175 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:38.253 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:38.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:38.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:38.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:38.414 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[18629]: audit 2024-09-10T19:04:37.325723+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[18629]: cluster 2024-09-10T19:04:37.325975+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[18629]: audit 2024-09-10T19:04:37.339033+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[32381]: audit 2024-09-10T19:04:37.325723+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[32381]: cluster 2024-09-10T19:04:37.325975+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:04:38.427 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:38 smithi083 bash[32381]: audit 2024-09-10T19:04:37.339033+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:38.449 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:38.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.584 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:38.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:38 smithi165 bash[22591]: audit 2024-09-10T19:04:37.325723+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:38 smithi165 bash[22591]: cluster 2024-09-10T19:04:37.325975+0000 mon.a (mon.0) 3116 : cluster [DBG] osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:04:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:38 smithi165 bash[22591]: audit 2024-09-10T19:04:37.339033+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:38.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:38.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:38.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:38.860 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:38.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:38.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:38.997 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:39.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:39.135 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:39.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:39.274 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:39.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:39.412 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: audit 2024-09-10T19:04:38.220226+0000 mgr.y (mgr.24526) 608 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: cluster 2024-09-10T19:04:38.257884+0000 mgr.y (mgr.24526) 609 : cluster [DBG] pgmap v979: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: audit 2024-09-10T19:04:38.343812+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: cluster 2024-09-10T19:04:38.343901+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: audit 2024-09-10T19:04:38.345123+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: audit 2024-09-10T19:04:39.266058+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: cluster 2024-09-10T19:04:39.266396+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[32381]: audit 2024-09-10T19:04:39.273285+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:39.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: audit 2024-09-10T19:04:38.220226+0000 mgr.y (mgr.24526) 608 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: cluster 2024-09-10T19:04:38.257884+0000 mgr.y (mgr.24526) 609 : cluster [DBG] pgmap v979: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: audit 2024-09-10T19:04:38.343812+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: cluster 2024-09-10T19:04:38.343901+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: audit 2024-09-10T19:04:38.345123+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: audit 2024-09-10T19:04:39.266058+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: cluster 2024-09-10T19:04:39.266396+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:04:39.512 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:39 smithi083 bash[18629]: audit 2024-09-10T19:04:39.273285+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:39.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:39.554 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: audit 2024-09-10T19:04:38.220226+0000 mgr.y (mgr.24526) 608 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: cluster 2024-09-10T19:04:38.257884+0000 mgr.y (mgr.24526) 609 : cluster [DBG] pgmap v979: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: audit 2024-09-10T19:04:38.343812+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: cluster 2024-09-10T19:04:38.343901+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:04:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: audit 2024-09-10T19:04:38.345123+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: audit 2024-09-10T19:04:39.266058+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-109", "overlaypool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: cluster 2024-09-10T19:04:39.266396+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:04:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:39 smithi165 bash[22591]: audit 2024-09-10T19:04:39.273285+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:39.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:39.686 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:39.789 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:39.817 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:39.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:39.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:39.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:40.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:40.094 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:40.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:40.232 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:40.285 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:40.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- 172.21.15.83:0/3535391076 >> v1:172.21.15.83:6790/0 conn(0x7f19c41075b0 legacy=0x7f19c4109a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- 172.21.15.83:0/3535391076 shutdown_connections 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- 172.21.15.83:0/3535391076 >> 172.21.15.83:0/3535391076 conn(0x7f19c4074a20 msgr2=0x7f19c40f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- 172.21.15.83:0/3535391076 shutdown_connections 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- 172.21.15.83:0/3535391076 wait complete. 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 librados: starting msgr at 2024-09-10T19:04:40.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 librados: starting objecter 2024-09-10T19:04:40.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 Processor -- start 2024-09-10T19:04:40.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- start start 2024-09-10T19:04:40.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 librados: setting wanted keys 2024-09-10T19:04:40.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 librados: calling monclient init 2024-09-10T19:04:40.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19c41226d0 con 0x7f19c41075b0 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19c41238b0 con 0x7f19c4103a50 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c8e50640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19c4124ab0 con 0x7f19c4100c70 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f19c4100c70 0x7f19c4100150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50408/0 (socket says 172.21.15.83:50408) 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c259c640 1 -- 172.21.15.83:0/1620849063 learned_addr learned my addr 172.21.15.83:0/1620849063 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.333+0000 7f19c2d9d640 1 --1- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6789/0 conn(0x7f19c41075b0 0x7f19c4120ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46762/0 (socket says 172.21.15.83:46762) 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter ms_handle_connect 0x7f19c41075b0 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter ms_handle_connect 0x7f19c4100c70 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575673607 0 0) 0x7f19c41226d0 con 0x7f19c41075b0 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac004a60 con 0x7f19c41075b0 2024-09-10T19:04:40.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339244065 0 0) 0x7f19c4124ab0 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c41226d0 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter ms_handle_connect 0x7f19c4103a50 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355779937 0 0) 0x7f19c41226d0 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c4124ab0 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19b4001900 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880976069 0 0) 0x7f19c41238b0 con 0x7f19c4103a50 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c41226d0 con 0x7f19c4103a50 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3615201981 0 0) 0x7f19c4124ab0 con 0x7f19c4100c70 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.165:6789/0 conn(0x7f19c4103a50 legacy=0x7f19c411d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6789/0 conn(0x7f19c41075b0 legacy=0x7f19c4120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4125cb0 con 0x7f19c4100c70 2024-09-10T19:04:40.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c4124ca0 con 0x7f19c4100c70 2024-09-10T19:04:40.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19c8e50640 10 client.49676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:40.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c41250f0 con 0x7f19c4100c70 2024-09-10T19:04:40.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19c8e50640 1 librados: init done 2024-09-10T19:04:40.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f19b4002290 con 0x7f19c4100c70 2024-09-10T19:04:40.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19b4005a90 con 0x7f19c4100c70 2024-09-10T19:04:40.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.337+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f19c40fc3e0 con 0x7f19c4100c70 2024-09-10T19:04:40.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f19b401bfb0 con 0x7f19c4100c70 2024-09-10T19:04:40.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19ac0716b0 2024-09-10T19:04:40.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(637..637 src has 69..637) v4 ==== 9220+0+0 (unknown 350006385 0 0) 0x7f19b4057790 con 0x7f19c4100c70 2024-09-10T19:04:40.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 osd_map(637..637 src has 69..637) v4 2024-09-10T19:04:40.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 3 client.49676.objecter handle_osd_map got epochs [637,637] > 0 2024-09-10T19:04:40.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 3 client.49676.objecter handle_osd_map decoding full epoch 637 2024-09-10T19:04:40.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.341+0000 7f19b2ffd640 20 client.49676.objecter dump_active .. 0 homeless 2024-09-10T19:04:40.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v637) v1 ==== 54+0+16 (unknown 3725107037 0 2420000192) 0x7f19b4057a50 con 0x7f19c4100c70 2024-09-10T19:04:40.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 10 client.49676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 1 2024-09-10T19:04:40.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 1 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 0 1 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 4 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 10 client.49676.objecter _send_command 1 2024-09-10T19:04:40.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c404bf60 con 0x7f19c4123a40 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c4123a40 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c404bf60 con 0x7f19c4123a40 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 1 = system:0 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 4 2024-09-10T19:04:40.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.345+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 0 1 2024-09-10T19:04:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 10 client.49676.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 2 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 2 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c40fd850 osd=0 3 2024-09-10T19:04:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 0 2 2024-09-10T19:04:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 4 2024-09-10T19:04:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 10 client.49676.objecter _send_command 2 2024-09-10T19:04:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c410c840 con 0x7f19c4123a40 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c410c840 con 0x7f19c4123a40 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 4 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.365+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 0 2 2024-09-10T19:04:40.370 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 10 client.49676.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 3 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 3 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 1 3 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 4 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 10 client.49676.objecter _send_command 3 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c402a8d0 con 0x7f19c4108050 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c4108050 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c402a8d0 con 0x7f19c4108050 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:04:40.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 3 = system:0 2024-09-10T19:04:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 4 2024-09-10T19:04:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.369+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 1 3 2024-09-10T19:04:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 10 client.49676.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 4 2024-09-10T19:04:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 4 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41079d0 osd=1 3 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 1 4 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 4 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 10 client.49676.objecter _send_command 4 2024-09-10T19:04:40.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.381+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c402ac80 con 0x7f19c4108050 2024-09-10T19:04:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c402ac80 con 0x7f19c4108050 2024-09-10T19:04:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 4 2024-09-10T19:04:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 1 4 2024-09-10T19:04:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:04:40.389 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:04:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 10 client.49676.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 5 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 5 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 2 5 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 4 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 10 client.49676.objecter _send_command 5 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c405e760 con 0x7f19c405e310 2024-09-10T19:04:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.385+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c405e310 2024-09-10T19:04:40.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.389+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c405e760 con 0x7f19c405e310 2024-09-10T19:04:40.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.389+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:04:40.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.389+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 5 = system:0 2024-09-10T19:04:40.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.389+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 4 2024-09-10T19:04:40.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.389+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 2 5 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 10 client.49676.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 6 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 6 2024-09-10T19:04:40.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c402adf0 osd=2 3 2024-09-10T19:04:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 2 6 2024-09-10T19:04:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 4 2024-09-10T19:04:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 10 client.49676.objecter _send_command 6 2024-09-10T19:04:40.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c405eb60 con 0x7f19c405e310 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c405eb60 con 0x7f19c405e310 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 4 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 2 6 2024-09-10T19:04:40.413 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 10 client.49676.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 7 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 7 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 3 7 2024-09-10T19:04:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 4 2024-09-10T19:04:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 10 client.49676.objecter _send_command 7 2024-09-10T19:04:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c412ace0 con 0x7f19c405df00 2024-09-10T19:04:40.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.409+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c405df00 2024-09-10T19:04:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.413+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c412ace0 con 0x7f19c405df00 2024-09-10T19:04:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.413+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:04:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.413+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 7 = system:0 2024-09-10T19:04:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.413+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 4 2024-09-10T19:04:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.413+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 3 7 2024-09-10T19:04:40.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 10 client.49676.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 8 2024-09-10T19:04:40.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 8 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c405dd90 osd=3 3 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 3 8 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 4 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 10 client.49676.objecter _send_command 8 2024-09-10T19:04:40.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c412a0a0 con 0x7f19c405df00 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c412a0a0 con 0x7f19c405df00 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 4 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 3 8 2024-09-10T19:04:40.437 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 10 client.49676.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 9 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 9 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 4 9 2024-09-10T19:04:40.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 4 2024-09-10T19:04:40.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 10 client.49676.objecter _send_command 9 2024-09-10T19:04:40.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.433+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c4130dd0 con 0x7f19c4129e60 2024-09-10T19:04:40.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c4129e60 2024-09-10T19:04:40.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c4130dd0 con 0x7f19c4129e60 2024-09-10T19:04:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:04:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 9 = system:0 2024-09-10T19:04:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 4 2024-09-10T19:04:40.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.437+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 4 9 2024-09-10T19:04:40.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 10 client.49676.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 10 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 10 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c412a4b0 osd=4 3 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 4 10 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 4 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 10 client.49676.objecter _send_command 10 2024-09-10T19:04:40.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.449+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c4130dd0 con 0x7f19c4129e60 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.453+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c4130dd0 con 0x7f19c4129e60 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.453+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.453+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.453+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 4 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.453+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 4 10 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:04:40.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 10 client.49676.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 11 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 11 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 5 11 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 4 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 10 client.49676.objecter _send_command 11 2024-09-10T19:04:40.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c41387c0 con 0x7f19c412ec20 2024-09-10T19:04:40.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c412ec20 2024-09-10T19:04:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c41387c0 con 0x7f19c412ec20 2024-09-10T19:04:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:04:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 11 = system:0 2024-09-10T19:04:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 4 2024-09-10T19:04:40.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.457+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 5 11 2024-09-10T19:04:40.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: cluster 2024-09-10T19:04:39.986431+0000 osd.5 (osd.5) 350 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: cluster 2024-09-10T19:04:39.986846+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: cluster 2024-09-10T19:04:40.269425+0000 mon.a (mon.0) 3124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: audit 2024-09-10T19:04:40.277308+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: cluster 2024-09-10T19:04:40.277510+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[18629]: audit 2024-09-10T19:04:40.282426+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:40.465 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: cluster 2024-09-10T19:04:39.986431+0000 osd.5 (osd.5) 350 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: cluster 2024-09-10T19:04:39.986846+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: cluster 2024-09-10T19:04:40.269425+0000 mon.a (mon.0) 3124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: audit 2024-09-10T19:04:40.277308+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: cluster 2024-09-10T19:04:40.277510+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:04:40.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:40 smithi083 bash[32381]: audit 2024-09-10T19:04:40.282426+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:40.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 10 client.49676.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 12 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 12 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c4130dd0 osd=5 3 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 5 12 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 4 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 10 client.49676.objecter _send_command 12 2024-09-10T19:04:40.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.477+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c4138780 con 0x7f19c412ec20 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c4138780 con 0x7f19c412ec20 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 4 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 5 12 2024-09-10T19:04:40.484 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 10 client.49676.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 13 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 13 2024-09-10T19:04:40.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 6 13 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 4 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 10 client.49676.objecter _send_command 13 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c413fe00 con 0x7f19c41389d0 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.481+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c41389d0 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.485+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c413fe00 con 0x7f19c41389d0 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.485+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.485+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 13 = system:0 2024-09-10T19:04:40.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.485+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 4 2024-09-10T19:04:40.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.485+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 6 13 2024-09-10T19:04:40.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 10 client.49676.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 14 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 14 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41387c0 osd=6 3 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 6 14 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 4 2024-09-10T19:04:40.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 10 client.49676.objecter _send_command 14 2024-09-10T19:04:40.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c41416a0 con 0x7f19c41389d0 2024-09-10T19:04:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c41416a0 con 0x7f19c41389d0 2024-09-10T19:04:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 4 2024-09-10T19:04:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 6 14 2024-09-10T19:04:40.506 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:04:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:04:40.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 10 client.49676.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 15 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 15 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 7 15 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 4 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 10 client.49676.objecter _send_command 15 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.501+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19c4142250 con 0x7f19c4141d20 2024-09-10T19:04:40.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 10 client.49676.objecter ms_handle_connect 0x7f19c4141d20 2024-09-10T19:04:40.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f19c4142250 con 0x7f19c4141d20 2024-09-10T19:04:40.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:04:40.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 15 = system:0 2024-09-10T19:04:40.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 4 2024-09-10T19:04:40.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.505+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 7 15 2024-09-10T19:04:40.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 10 client.49676.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:04:40.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign -1 16 2024-09-10T19:04:40.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 20 client.49676.objecter _get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_remove -1 16 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 20 client.49676.objecter get_session s=0x7f19c41416a0 osd=7 3 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 15 client.49676.objecter _session_command_op_assign 7 16 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 4 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 10 client.49676.objecter _send_command 16 2024-09-10T19:04:40.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.529+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f19c4142250 con 0x7f19c4141d20 2024-09-10T19:04:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19b2ffd640 1 -- 172.21.15.83:0/1620849063 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f19c4142250 con 0x7f19c4141d20 2024-09-10T19:04:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19b2ffd640 10 client.49676.objecter ms_dispatch 0x7f19c4000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:04:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19b2ffd640 10 client.49676.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:04:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19b2ffd640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 4 2024-09-10T19:04:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19b2ffd640 15 client.49676.objecter _session_command_op_remove 7 16 2024-09-10T19:04:40.535 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 librados: watch_flush enter 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 librados: watch_flush exit 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.0 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6801/2535527483 conn(0x7f19c4123a40 legacy=0x7f19c4123e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c40fd850 osd=0 1 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.1 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6805/2309624637 conn(0x7f19c4108050 legacy=0x7f19c4108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41079d0 osd=1 1 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.2 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6809/386094220 conn(0x7f19c405e310 legacy=0x7f19c412cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c402adf0 osd=2 1 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.3 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6813/1279688021 conn(0x7f19c405df00 legacy=0x7f19c412f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c405dd90 osd=3 1 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.4 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.165:6800/3225515265 conn(0x7f19c4129e60 legacy=0x7f19c4135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c412a4b0 osd=4 1 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.5 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.165:6804/571173777 conn(0x7f19c412ec20 legacy=0x7f19c412f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c2d9d640 1 -- 172.21.15.83:0/1620849063 reap_dead start 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c4130dd0 osd=5 1 2024-09-10T19:04:40.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.6 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.165:6808/1317922936 conn(0x7f19c41389d0 legacy=0x7f19c413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c2d9d640 1 -- 172.21.15.83:0/1620849063 reap_dead start 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41387c0 osd=6 1 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter close_session for osd.7 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.165:6812/1610151650 conn(0x7f19c4141d20 legacy=0x7f19c413fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter put_session s=0x7f19c41416a0 osd=7 1 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 20 client.49676.objecter shutdown clearing up homeless session... 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 10 client.49676.objecter successfully canceled tick 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c2d9d640 1 -- 172.21.15.83:0/1620849063 reap_dead start 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c2d9d640 1 -- 172.21.15.83:0/1620849063 reap_dead start 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6800/3071793735 conn(0x7f19ac0716b0 legacy=0x7f19ac073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> v1:172.21.15.83:6790/0 conn(0x7f19c4100c70 legacy=0x7f19c4100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:40.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 shutdown_connections 2024-09-10T19:04:40.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 >> 172.21.15.83:0/1620849063 conn(0x7f19c4074a20 msgr2=0x7f19c4070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:40.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 shutdown_connections 2024-09-10T19:04:40.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 -- 172.21.15.83:0/1620849063 wait complete. 2024-09-10T19:04:40.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:40.533+0000 7f19c8e50640 1 librados: shutdown 2024-09-10T19:04:40.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: cluster 2024-09-10T19:04:39.986431+0000 osd.5 (osd.5) 350 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: cluster 2024-09-10T19:04:39.986846+0000 osd.5 (osd.5) 351 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: cluster 2024-09-10T19:04:40.269425+0000 mon.a (mon.0) 3124 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: audit 2024-09-10T19:04:40.277308+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: cluster 2024-09-10T19:04:40.277510+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:04:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:40 smithi165 bash[22591]: audit 2024-09-10T19:04:40.282426+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:04:40.634 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:40.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:40.771 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:40.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:40.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:40.909 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:40.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:41.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.045 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:41.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.183 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:41.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:41.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:41.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:41.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:41.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[18629]: cluster 2024-09-10T19:04:40.258571+0000 mgr.y (mgr.24526) 610 : cluster [DBG] pgmap v982: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[18629]: audit 2024-09-10T19:04:40.344428+0000 mon.c (mon.2) 686 : audit [DBG] from='client.? v1:172.21.15.83:0/1620849063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[18629]: audit 2024-09-10T19:04:41.280764+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[18629]: cluster 2024-09-10T19:04:41.280919+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[18629]: audit 2024-09-10T19:04:41.283570+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[32381]: cluster 2024-09-10T19:04:40.258571+0000 mgr.y (mgr.24526) 610 : cluster [DBG] pgmap v982: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:41.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[32381]: audit 2024-09-10T19:04:40.344428+0000 mon.c (mon.2) 686 : audit [DBG] from='client.? v1:172.21.15.83:0/1620849063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:41.559 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[32381]: audit 2024-09-10T19:04:41.280764+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:41.559 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[32381]: cluster 2024-09-10T19:04:41.280919+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:04:41.559 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:41 smithi083 bash[32381]: audit 2024-09-10T19:04:41.283570+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:41.569 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:41.569 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[22591]: cluster 2024-09-10T19:04:40.258571+0000 mgr.y (mgr.24526) 610 : cluster [DBG] pgmap v982: 222 pgs: 6 creating+peering, 34 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:41.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[22591]: audit 2024-09-10T19:04:40.344428+0000 mon.c (mon.2) 686 : audit [DBG] from='client.? v1:172.21.15.83:0/1620849063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:04:41.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[22591]: audit 2024-09-10T19:04:41.280764+0000 mon.a (mon.0) 3128 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:04:41.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[22591]: cluster 2024-09-10T19:04:41.280919+0000 mon.a (mon.0) 3129 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:04:41.570 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:41 smithi165 bash[22591]: audit 2024-09-10T19:04:41.283570+0000 mon.a (mon.0) 3130 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:04:41.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:41.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.727 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:41.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:41.867 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:41.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:41.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:42.008 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:42.112 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:42.146 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:42.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:42.286 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:42.391 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:42.425 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:42.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:42.562 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:42.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:42.701 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:42.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:42.841 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:42.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:42.980 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:42.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:42] "GET /metrics HTTP/1.1" 200 231407 "" "Prometheus/2.43.0" 2024-09-10T19:04:43.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:43.119 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:43.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:43.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:43.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:43 smithi165 bash[22591]: cluster 2024-09-10T19:04:42.259664+0000 mgr.y (mgr.24526) 611 : cluster [DBG] pgmap v985: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:43 smithi165 bash[22591]: audit 2024-09-10T19:04:42.282445+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:43 smithi165 bash[22591]: cluster 2024-09-10T19:04:42.282638+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:04:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:43 smithi165 bash[22591]: audit 2024-09-10T19:04:42.283538+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:43.398 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:43.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:43.534 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:43.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[18629]: cluster 2024-09-10T19:04:42.259664+0000 mgr.y (mgr.24526) 611 : cluster [DBG] pgmap v985: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[18629]: audit 2024-09-10T19:04:42.282445+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[18629]: cluster 2024-09-10T19:04:42.282638+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[18629]: audit 2024-09-10T19:04:42.283538+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[32381]: cluster 2024-09-10T19:04:42.259664+0000 mgr.y (mgr.24526) 611 : cluster [DBG] pgmap v985: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[32381]: audit 2024-09-10T19:04:42.282445+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[32381]: cluster 2024-09-10T19:04:42.282638+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:04:43.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:43 smithi083 bash[32381]: audit 2024-09-10T19:04:42.283538+0000 mon.a (mon.0) 3133 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:04:43.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:43.668 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:43.771 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:43.799 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:43.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:43.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:43.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:44.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:44.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:44.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.209 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:44.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:44.322 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: cluster 2024-09-10T19:04:43.277472+0000 mon.a (mon.0) 3134 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:44.322 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: audit 2024-09-10T19:04:43.288687+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:44.322 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: cluster 2024-09-10T19:04:43.288755+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:04:44.322 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: audit 2024-09-10T19:04:43.290177+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:04:44.323 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: cluster 2024-09-10T19:04:43.510946+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:44.323 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: cluster 2024-09-10T19:04:43.511374+0000 osd.1 (osd.1) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:44.323 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: audit 2024-09-10T19:04:44.290924+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:04:44.323 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[32381]: cluster 2024-09-10T19:04:44.291117+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:04:44.323 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: cluster 2024-09-10T19:04:43.277472+0000 mon.a (mon.0) 3134 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: audit 2024-09-10T19:04:43.288687+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: cluster 2024-09-10T19:04:43.288755+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: audit 2024-09-10T19:04:43.290177+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: cluster 2024-09-10T19:04:43.510946+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: cluster 2024-09-10T19:04:43.511374+0000 osd.1 (osd.1) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:44.324 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: audit 2024-09-10T19:04:44.290924+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:04:44.325 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:44 smithi083 bash[18629]: cluster 2024-09-10T19:04:44.291117+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:04:44.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:44.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:44.442 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: cluster 2024-09-10T19:04:43.277472+0000 mon.a (mon.0) 3134 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: audit 2024-09-10T19:04:43.288687+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: cluster 2024-09-10T19:04:43.288755+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: audit 2024-09-10T19:04:43.290177+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: cluster 2024-09-10T19:04:43.510946+0000 osd.1 (osd.1) 337 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: cluster 2024-09-10T19:04:43.511374+0000 osd.1 (osd.1) 338 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:04:44.456 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: audit 2024-09-10T19:04:44.290924+0000 mon.a (mon.0) 3138 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:04:44.457 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:44 smithi165 bash[22591]: cluster 2024-09-10T19:04:44.291117+0000 mon.a (mon.0) 3139 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:04:44.476 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:44.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:44.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:44.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:44.746 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:44.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:44.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:44.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:44.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:45.022 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:45.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:45.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:45.290 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:45.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:45 smithi165 bash[22591]: cluster 2024-09-10T19:04:44.260479+0000 mgr.y (mgr.24526) 612 : cluster [DBG] pgmap v988: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:45 smithi165 bash[22591]: audit 2024-09-10T19:04:44.344638+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:45.404 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:45 smithi165 bash[22591]: audit 2024-09-10T19:04:44.578634+0000 mon.a (mon.0) 3141 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:45.405 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:45 smithi165 bash[22591]: audit 2024-09-10T19:04:44.581379+0000 mon.c (mon.2) 687 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:45.427 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:45.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:04:45.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[32381]: cluster 2024-09-10T19:04:44.260479+0000 mgr.y (mgr.24526) 612 : cluster [DBG] pgmap v988: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:45.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[32381]: audit 2024-09-10T19:04:44.344638+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:45.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[32381]: audit 2024-09-10T19:04:44.578634+0000 mon.a (mon.0) 3141 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:45.547 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[32381]: audit 2024-09-10T19:04:44.581379+0000 mon.c (mon.2) 687 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[18629]: cluster 2024-09-10T19:04:44.260479+0000 mgr.y (mgr.24526) 612 : cluster [DBG] pgmap v988: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[18629]: audit 2024-09-10T19:04:44.344638+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[18629]: audit 2024-09-10T19:04:44.578634+0000 mon.a (mon.0) 3141 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:04:45.548 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:45 smithi083 bash[18629]: audit 2024-09-10T19:04:44.581379+0000 mon.c (mon.2) 687 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:45.565 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:45.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:45.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:45.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:45.800 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:45.835 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:45.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:45.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:46.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:46.109 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:46.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:46.243 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:46.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:46.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:46.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:46.512 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:46.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[18629]: audit 2024-09-10T19:04:45.316989+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[18629]: cluster 2024-09-10T19:04:45.317099+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:04:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[18629]: audit 2024-09-10T19:04:45.318418+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[32381]: audit 2024-09-10T19:04:45.316989+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[32381]: cluster 2024-09-10T19:04:45.317099+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:04:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:46 smithi083 bash[32381]: audit 2024-09-10T19:04:45.318418+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:46 smithi165 bash[22591]: audit 2024-09-10T19:04:45.316989+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:46 smithi165 bash[22591]: cluster 2024-09-10T19:04:45.317099+0000 mon.a (mon.0) 3143 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:04:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:46 smithi165 bash[22591]: audit 2024-09-10T19:04:45.318418+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]: dispatch 2024-09-10T19:04:46.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:46.646 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:46.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:46.784 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:46.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:46.921 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:46.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:47.025 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:47.060 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:47.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.199 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:47.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:47.340 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:47 smithi165 bash[22591]: cluster 2024-09-10T19:04:46.262267+0000 mgr.y (mgr.24526) 613 : cluster [DBG] pgmap v991: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:47.340 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:47 smithi165 bash[22591]: audit 2024-09-10T19:04:46.321846+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:47.340 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:47 smithi165 bash[22591]: cluster 2024-09-10T19:04:46.322282+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:04:47.343 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:47.446 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:47.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:47.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[32381]: cluster 2024-09-10T19:04:46.262267+0000 mgr.y (mgr.24526) 613 : cluster [DBG] pgmap v991: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[32381]: audit 2024-09-10T19:04:46.321846+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[32381]: cluster 2024-09-10T19:04:46.322282+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[18629]: cluster 2024-09-10T19:04:46.262267+0000 mgr.y (mgr.24526) 613 : cluster [DBG] pgmap v991: 222 pgs: 222 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[18629]: audit 2024-09-10T19:04:46.321846+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-109", "tierpool": "test-rados-api-smithi083-77828-109-cache"}]': finished 2024-09-10T19:04:47.616 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:47 smithi083 bash[18629]: cluster 2024-09-10T19:04:46.322282+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:04:47.620 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:47.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.759 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:47.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:47.886 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:47.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:47.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:48.003 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:04:48.003 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:04:48.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:48.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:48.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:48.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:48.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:48.293 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:48.396 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:48.410 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:48 smithi083 bash[18629]: cluster 2024-09-10T19:04:47.335011+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:04:48.410 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:48 smithi083 bash[32381]: cluster 2024-09-10T19:04:47.335011+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:04:48.429 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:48.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:48.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:48 smithi165 bash[22591]: cluster 2024-09-10T19:04:47.335011+0000 mon.a (mon.0) 3147 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:04:48.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:48.705 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:48.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:48.845 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:48.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:48.982 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:48.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:49.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:49.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:49.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:49.253 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:49.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:49.387 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:49.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:49.525 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:49.584 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: audit 2024-09-10T19:04:48.228857+0000 mgr.y (mgr.24526) 614 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: cluster 2024-09-10T19:04:48.263082+0000 mgr.y (mgr.24526) 615 : cluster [DBG] pgmap v994: 190 pgs: 190 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: cluster 2024-09-10T19:04:48.342628+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: audit 2024-09-10T19:04:48.352467+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: audit 2024-09-10T19:04:49.269521+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: cluster 2024-09-10T19:04:49.269671+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[18629]: audit 2024-09-10T19:04:49.271385+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: audit 2024-09-10T19:04:48.228857+0000 mgr.y (mgr.24526) 614 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: cluster 2024-09-10T19:04:48.263082+0000 mgr.y (mgr.24526) 615 : cluster [DBG] pgmap v994: 190 pgs: 190 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:49.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: cluster 2024-09-10T19:04:48.342628+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:04:49.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: audit 2024-09-10T19:04:48.352467+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: audit 2024-09-10T19:04:49.269521+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:49.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: cluster 2024-09-10T19:04:49.269671+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:04:49.586 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:49 smithi083 bash[32381]: audit 2024-09-10T19:04:49.271385+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: audit 2024-09-10T19:04:48.228857+0000 mgr.y (mgr.24526) 614 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: cluster 2024-09-10T19:04:48.263082+0000 mgr.y (mgr.24526) 615 : cluster [DBG] pgmap v994: 190 pgs: 190 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: cluster 2024-09-10T19:04:48.342628+0000 mon.a (mon.0) 3148 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: audit 2024-09-10T19:04:48.352467+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: audit 2024-09-10T19:04:49.269521+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: cluster 2024-09-10T19:04:49.269671+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:04:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:49 smithi165 bash[22591]: audit 2024-09-10T19:04:49.271385+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]: dispatch 2024-09-10T19:04:49.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:49.663 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:49.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:49.799 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:49.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:49.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:49.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:50.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:50.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:50.173 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:50.209 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:50.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:50.348 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:50.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 -- 172.21.15.83:0/690281824 >> v1:172.21.15.83:6790/0 conn(0x7ffaa8107e00 legacy=0x7ffaa810a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 -- 172.21.15.83:0/690281824 shutdown_connections 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 -- 172.21.15.83:0/690281824 >> 172.21.15.83:0/690281824 conn(0x7ffaa80f70b0 msgr2=0x7ffaa80f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 -- 172.21.15.83:0/690281824 shutdown_connections 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 -- 172.21.15.83:0/690281824 wait complete. 2024-09-10T19:04:50.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 librados: starting msgr at 2024-09-10T19:04:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.461+0000 7ffaaea0f640 1 librados: starting objecter 2024-09-10T19:04:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 Processor -- start 2024-09-10T19:04:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- start start 2024-09-10T19:04:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 librados: setting wanted keys 2024-09-10T19:04:50.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 librados: calling monclient init 2024-09-10T19:04:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffaa811e210 con 0x7ffaa81042a0 2024-09-10T19:04:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffaa811f410 con 0x7ffaa8101530 2024-09-10T19:04:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffaa8120610 con 0x7ffaa8107e00 2024-09-10T19:04:50.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaacfac640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ffaa8107e00 0x7ffaa811c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51488/0 (socket says 172.21.15.83:51488) 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ffaa81042a0 0x7ffaa8066420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37412/0 (socket says 172.21.15.83:37412) 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaacfac640 1 -- 172.21.15.83:0/2589921183 learned_addr learned my addr 172.21.15.83:0/2589921183 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter ms_handle_connect 0x7ffaa8107e00 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter ms_handle_connect 0x7ffaa81042a0 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter ms_handle_connect 0x7ffaa8101530 2024-09-10T19:04:50.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321670529 0 0) 0x7ffaa811e210 con 0x7ffaa81042a0 2024-09-10T19:04:50.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa900038e0 con 0x7ffaa81042a0 2024-09-10T19:04:50.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810024397 0 0) 0x7ffaa8120610 con 0x7ffaa8107e00 2024-09-10T19:04:50.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa811e210 con 0x7ffaa8107e00 2024-09-10T19:04:50.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041720607 0 0) 0x7ffaa811f410 con 0x7ffaa8101530 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8120610 con 0x7ffaa8101530 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 39603578 0 0) 0x7ffa900038e0 con 0x7ffaa81042a0 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaa811f410 con 0x7ffaa81042a0 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ffa9c0018c0 con 0x7ffaa81042a0 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2016895652 0 0) 0x7ffaa811e210 con 0x7ffaa8107e00 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa900038e0 con 0x7ffaa8107e00 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ffaa0002dc0 con 0x7ffaa8107e00 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3582208315 0 0) 0x7ffaa811f410 con 0x7ffaa81042a0 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 >> v1:172.21.15.83:6790/0 conn(0x7ffaa8107e00 legacy=0x7ffaa811c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 >> v1:172.21.15.165:6789/0 conn(0x7ffaa8101530 legacy=0x7ffaa8065d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa8121810 con 0x7ffaa81042a0 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa811e400 con 0x7ffaa81042a0 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 10 client.25605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa811e850 con 0x7ffaa81042a0 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaaea0f640 1 librados: init done 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ffa9c001f40 con 0x7ffaa81042a0 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.465+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ffa9c0059a0 con 0x7ffaa81042a0 2024-09-10T19:04:50.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa94005010 con 0x7ffaa81042a0 2024-09-10T19:04:50.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ffa9c01bdc0 con 0x7ffaa81042a0 2024-09-10T19:04:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(647..647 src has 69..647) v4 ==== 8046+0+0 (unknown 392117237 0 0) 0x7ffa9c08a330 con 0x7ffaa81042a0 2024-09-10T19:04:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 10 client.25605.objecter ms_dispatch 0x7ffaa8000b90 osd_map(647..647 src has 69..647) v4 2024-09-10T19:04:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 3 client.25605.objecter handle_osd_map got epochs [647,647] > 0 2024-09-10T19:04:50.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 3 client.25605.objecter handle_osd_map decoding full epoch 647 2024-09-10T19:04:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.469+0000 7ffaa4ff9640 20 client.25605.objecter dump_active .. 0 homeless 2024-09-10T19:04:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.473+0000 7ffaa4ff9640 10 client.25605.objecter ms_handle_connect 0x7ffa90071500 2024-09-10T19:04:50.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:50.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.473+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ffa9c0564a0 con 0x7ffaa81042a0 2024-09-10T19:04:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:50.568 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:04:50.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:50.616 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:50.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.661+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7ffa940052c0 con 0x7ffaa81042a0 2024-09-10T19:04:50.720 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:50.750 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:50.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:50.886 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:50.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.933+0000 7fed7da9c640 1 -- 172.21.15.83:0/1649029510 >> v1:172.21.15.83:6789/0 conn(0x7fed780fe150 legacy=0x7fed781075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.933+0000 7fed7da9c640 1 -- 172.21.15.83:0/1649029510 shutdown_connections 2024-09-10T19:04:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.933+0000 7fed7da9c640 1 -- 172.21.15.83:0/1649029510 >> 172.21.15.83:0/1649029510 conn(0x7fed780f7060 msgr2=0x7fed780f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.933+0000 7fed7da9c640 1 -- 172.21.15.83:0/1649029510 shutdown_connections 2024-09-10T19:04:50.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.933+0000 7fed7da9c640 1 -- 172.21.15.83:0/1649029510 wait complete. 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 librados: starting msgr at 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 librados: starting objecter 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 Processor -- start 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- start start 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 librados: setting wanted keys 2024-09-10T19:04:50.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 librados: calling monclient init 2024-09-10T19:04:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed7811c650 con 0x7fed78107740 2024-09-10T19:04:50.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed7811d7f0 con 0x7fed78108ee0 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fed7811e9f0 con 0x7fed780fd5b0 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed777fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fed780fd5b0 0x7fed78106e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51506/0 (socket says 172.21.15.83:51506) 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed777fe640 1 -- 172.21.15.83:0/3509403301 learned_addr learned my addr 172.21.15.83:0/3509403301 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter ms_handle_connect 0x7fed78107740 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter ms_handle_connect 0x7fed780fd5b0 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118693980 0 0) 0x7fed7811c650 con 0x7fed78107740 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed600034e0 con 0x7fed78107740 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter ms_handle_connect 0x7fed78108ee0 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338008403 0 0) 0x7fed7811e9f0 con 0x7fed780fd5b0 2024-09-10T19:04:50.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7811c650 con 0x7fed780fd5b0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621170306 0 0) 0x7fed7811d7f0 con 0x7fed78108ee0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7811e9f0 con 0x7fed78108ee0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2582942503 0 0) 0x7fed600034e0 con 0x7fed78107740 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7811d7f0 con 0x7fed78107740 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fed70001c40 con 0x7fed78107740 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2730537685 0 0) 0x7fed7811c650 con 0x7fed780fd5b0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed600034e0 con 0x7fed780fd5b0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fed68001900 con 0x7fed780fd5b0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707092144 0 0) 0x7fed7811e9f0 con 0x7fed78108ee0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7811c650 con 0x7fed78108ee0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fed6c002050 con 0x7fed78108ee0 2024-09-10T19:04:50.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1397236440 0 0) 0x7fed7811d7f0 con 0x7fed78107740 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 >> v1:172.21.15.83:6790/0 conn(0x7fed780fd5b0 legacy=0x7fed78106e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 >> v1:172.21.15.165:6789/0 conn(0x7fed78108ee0 legacy=0x7fed78117670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7811fbf0 con 0x7fed78107740 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7811ebe0 con 0x7fed78107740 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 10 client.25611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7811f030 con 0x7fed78107740 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.937+0000 7fed7da9c640 1 librados: init done 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fed70002570 con 0x7fed78107740 2024-09-10T19:04:50.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fed70003450 con 0x7fed78107740 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed64005010 con 0x7fed78107740 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fed70002110 con 0x7fed78107740 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(647..647 src has 69..647) v4 ==== 8046+0+0 (unknown 392117237 0 0) 0x7fed70087e40 con 0x7fed78107740 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 10 client.25611.objecter ms_dispatch 0x7fed78000b90 osd_map(647..647 src has 69..647) v4 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 3 client.25611.objecter handle_osd_map got epochs [647,647] > 0 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 3 client.25611.objecter handle_osd_map decoding full epoch 647 2024-09-10T19:04:50.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 20 client.25611.objecter dump_active .. 0 homeless 2024-09-10T19:04:50.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.941+0000 7fed5ffff640 10 client.25611.objecter ms_handle_connect 0x7fed60075c00 2024-09-10T19:04:50.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:50.949+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fed700543d0 con 0x7fed78107740 2024-09-10T19:04:50.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:51.024 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:51.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:51.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.129+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fed640052c0 con 0x7fed78107740 2024-09-10T19:04:51.158 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:51.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:51.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.277+0000 7ffaa4ff9640 1 -- 172.21.15.83:0/2589921183 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v648) v1 ==== 113+0+0 (unknown 820768957 0 0) 0x7ffa9c0562c0 con 0x7ffaa81042a0 2024-09-10T19:04:51.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.277+0000 7fed5ffff640 1 -- 172.21.15.83:0/3509403301 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v648) v1 ==== 91+0+0 (unknown 899967187 0 0) 0x7fed70053d70 con 0x7fed78107740 2024-09-10T19:04:51.280 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T19:04:51.281 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:04:51.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 10 librados: watch_flush enter 2024-09-10T19:04:51.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 10 librados: watch_flush exit 2024-09-10T19:04:51.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 20 client.25605.objecter shutdown clearing up homeless session... 2024-09-10T19:04:51.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 10 client.25605.objecter successfully canceled tick 2024-09-10T19:04:51.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 10 librados: watch_flush enter 2024-09-10T19:04:51.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 10 librados: watch_flush exit 2024-09-10T19:04:51.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 20 client.25611.objecter shutdown clearing up homeless session... 2024-09-10T19:04:51.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 10 client.25611.objecter successfully canceled tick 2024-09-10T19:04:51.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 >> v1:172.21.15.83:6800/3071793735 conn(0x7ffa90071500 legacy=0x7ffa900739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:51.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 >> v1:172.21.15.83:6789/0 conn(0x7ffaa81042a0 legacy=0x7ffaa8066420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:51.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 shutdown_connections 2024-09-10T19:04:51.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 >> 172.21.15.83:0/2589921183 conn(0x7ffaa80f70b0 msgr2=0x7ffaa810aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:51.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 >> v1:172.21.15.83:6800/3071793735 conn(0x7fed60075c00 legacy=0x7fed600780b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:51.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 >> v1:172.21.15.83:6789/0 conn(0x7fed78107740 legacy=0x7fed7811ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:04:51.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 shutdown_connections 2024-09-10T19:04:51.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 >> 172.21.15.83:0/3509403301 conn(0x7fed780f7060 msgr2=0x7fed7810b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:04:51.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 shutdown_connections 2024-09-10T19:04:51.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 shutdown_connections 2024-09-10T19:04:51.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 -- 172.21.15.83:0/3509403301 wait complete. 2024-09-10T19:04:51.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 -- 172.21.15.83:0/2589921183 wait complete. 2024-09-10T19:04:51.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7ffaaea0f640 1 librados: shutdown 2024-09-10T19:04:51.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:04:51.281+0000 7fed7da9c640 1 librados: shutdown 2024-09-10T19:04:51.290 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:51.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:51.428 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:51.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:51.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: cluster 2024-09-10T19:04:50.263991+0000 mgr.y (mgr.24526) 616 : cluster [DBG] pgmap v997: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: audit 2024-09-10T19:04:50.276788+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: cluster 2024-09-10T19:04:50.276874+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:04:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: cluster 2024-09-10T19:04:50.349297+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:51.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: audit 2024-09-10T19:04:50.664589+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[18629]: audit 2024-09-10T19:04:51.131921+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: cluster 2024-09-10T19:04:50.263991+0000 mgr.y (mgr.24526) 616 : cluster [DBG] pgmap v997: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: audit 2024-09-10T19:04:50.276788+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: cluster 2024-09-10T19:04:50.276874+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: cluster 2024-09-10T19:04:50.349297+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: audit 2024-09-10T19:04:50.664589+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:04:51.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:51 smithi083 bash[32381]: audit 2024-09-10T19:04:51.131921+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:04:51.566 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:51.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:51.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: cluster 2024-09-10T19:04:50.263991+0000 mgr.y (mgr.24526) 616 : cluster [DBG] pgmap v997: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: audit 2024-09-10T19:04:50.276788+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.83:0/2000689439' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-109"}]': finished 2024-09-10T19:04:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: cluster 2024-09-10T19:04:50.276874+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:04:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: cluster 2024-09-10T19:04:50.349297+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: audit 2024-09-10T19:04:50.664589+0000 mon.a (mon.0) 3156 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:04:51.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:51 smithi165 bash[22591]: audit 2024-09-10T19:04:51.131921+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:04:51.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:51.700 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:51.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:51.838 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:51.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:51.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:51.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:52.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:52.112 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:52.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:52.248 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:52.351 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.268575+0000 mon.a (mon.0) 3158 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:51.278755+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:51.278870+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.278931+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:51.280479+0000 mon.c (mon.2) 688 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:51.282769+0000 mon.a (mon.0) 3162 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.482950+0000 osd.6 (osd.6) 299 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.486201+0000 osd.6 (osd.6) 300 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.927332+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.930750+0000 osd.4 (osd.4) 349 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:52.279168+0000 mon.a (mon.0) 3163 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.279294+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:52.280340+0000 mon.c (mon.2) 689 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:52 smithi165 bash[22591]: audit 2024-09-10T19:04:52.280944+0000 mon.a (mon.0) 3165 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.385 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:52.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:52.523 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:52.626 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.268575+0000 mon.a (mon.0) 3158 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:51.278755+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:51.278870+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.278931+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:51.280479+0000 mon.c (mon.2) 688 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:51.282769+0000 mon.a (mon.0) 3162 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.482950+0000 osd.6 (osd.6) 299 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.486201+0000 osd.6 (osd.6) 300 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.927332+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.930750+0000 osd.4 (osd.4) 349 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:52.279168+0000 mon.a (mon.0) 3163 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:52.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.279294+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:52.280340+0000 mon.c (mon.2) 689 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[32381]: audit 2024-09-10T19:04:52.280944+0000 mon.a (mon.0) 3165 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.268575+0000 mon.a (mon.0) 3158 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:51.278755+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? v1:172.21.15.83:0/2589921183' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:51.278870+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? v1:172.21.15.83:0/3509403301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.278931+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:51.280479+0000 mon.c (mon.2) 688 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:51.282769+0000 mon.a (mon.0) 3162 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.482950+0000 osd.6 (osd.6) 299 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:04:52.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.486201+0000 osd.6 (osd.6) 300 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.927332+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.930750+0000 osd.4 (osd.4) 349 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:52.279168+0000 mon.a (mon.0) 3163 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.279294+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:52.280340+0000 mon.c (mon.2) 689 : audit [INF] from='client.? v1:172.21.15.83:0/3480138515' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18629]: audit 2024-09-10T19:04:52.280944+0000 mon.a (mon.0) 3165 : audit [INF] from='client.49640 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]: dispatch 2024-09-10T19:04:52.663 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:52.685 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T19:04:52.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:52.803 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:52.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:52.942 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:52.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:04:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:04:52] "GET /metrics HTTP/1.1" 200 231387 "" "Prometheus/2.43.0" 2024-09-10T19:04:53.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.080 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:53.085 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:53.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.218 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:53.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:53.349 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:53.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.466 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.942393+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:51.946381+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.196000+0000 osd.5 (osd.5) 352 : cluster [DBG] 6.d scrub starts 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.199557+0000 osd.5 (osd.5) 353 : cluster [DBG] 6.d scrub ok 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.264729+0000 mgr.y (mgr.24526) 617 : cluster [DBG] pgmap v1000: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: audit 2024-09-10T19:04:53.285089+0000 mon.a (mon.0) 3166 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:53.467 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:53 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.285207+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:04:53.488 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.942393+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:51.946381+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.196000+0000 osd.5 (osd.5) 352 : cluster [DBG] 6.d scrub starts 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.199557+0000 osd.5 (osd.5) 353 : cluster [DBG] 6.d scrub ok 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.264729+0000 mgr.y (mgr.24526) 617 : cluster [DBG] pgmap v1000: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: audit 2024-09-10T19:04:53.285089+0000 mon.a (mon.0) 3166 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:53.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.285207+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.942393+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:51.946381+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.196000+0000 osd.5 (osd.5) 352 : cluster [DBG] 6.d scrub starts 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.199557+0000 osd.5 (osd.5) 353 : cluster [DBG] 6.d scrub ok 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.264729+0000 mgr.y (mgr.24526) 617 : cluster [DBG] pgmap v1000: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: audit 2024-09-10T19:04:53.285089+0000 mon.a (mon.0) 3166 : audit [INF] from='client.49640 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi083-77828-104"}]': finished 2024-09-10T19:04:53.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:53 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.285207+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:04:53.589 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:53.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:53.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:53.762 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:53.866 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:53.895 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:53.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:53.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:54.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:54.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:54.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:54.278 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:54.316 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:54.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:54.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:54.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.566704+0000 osd.0 (osd.0) 327 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:52.571571+0000 osd.0 (osd.0) 328 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.310390+0000 mon.c (mon.2) 690 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.310780+0000 mon.a (mon.0) 3168 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.312934+0000 mon.c (mon.2) 691 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.313230+0000 mon.a (mon.0) 3169 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.313786+0000 mon.c (mon.2) 692 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: audit 2024-09-10T19:04:53.314118+0000 mon.a (mon.0) 3170 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.557490+0000 osd.6 (osd.6) 301 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.561974+0000 osd.6 (osd.6) 302 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.818673+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.e scrub starts 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.827468+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.e scrub ok 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.890892+0000 osd.4 (osd.4) 350 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:54 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.895942+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:04:54.569 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.566704+0000 osd.0 (osd.0) 327 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:04:54.569 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:52.571571+0000 osd.0 (osd.0) 328 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.310390+0000 mon.c (mon.2) 690 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.310780+0000 mon.a (mon.0) 3168 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.312934+0000 mon.c (mon.2) 691 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.313230+0000 mon.a (mon.0) 3169 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.313786+0000 mon.c (mon.2) 692 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: audit 2024-09-10T19:04:53.314118+0000 mon.a (mon.0) 3170 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.557490+0000 osd.6 (osd.6) 301 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.561974+0000 osd.6 (osd.6) 302 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.818673+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.e scrub starts 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.827468+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.e scrub ok 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.890892+0000 osd.4 (osd.4) 350 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:04:54.570 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.895942+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.566704+0000 osd.0 (osd.0) 327 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:52.571571+0000 osd.0 (osd.0) 328 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.310390+0000 mon.c (mon.2) 690 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.310780+0000 mon.a (mon.0) 3168 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.312934+0000 mon.c (mon.2) 691 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.313230+0000 mon.a (mon.0) 3169 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.313786+0000 mon.c (mon.2) 692 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: audit 2024-09-10T19:04:53.314118+0000 mon.a (mon.0) 3170 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.557490+0000 osd.6 (osd.6) 301 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.561974+0000 osd.6 (osd.6) 302 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.818673+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.e scrub starts 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.827468+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.e scrub ok 2024-09-10T19:04:54.571 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.890892+0000 osd.4 (osd.4) 350 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:04:54.572 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:54 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.895942+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:04:54.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:54.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:04:54.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:54.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:54.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:54.847 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:54.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:54.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:55.087 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:55.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:55.182 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:55.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:55.254 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:55.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:55.393 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:55.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:55.527 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:55.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:55.661 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:55.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:55.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.966274+0000 osd.1 (osd.1) 341 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:04:55.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: cluster 2024-09-10T19:04:53.970546+0000 osd.1 (osd.1) 342 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:04:55.773 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: cluster 2024-09-10T19:04:54.265485+0000 mgr.y (mgr.24526) 618 : cluster [DBG] pgmap v1003: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:55.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: audit 2024-09-10T19:04:54.299361+0000 mon.a (mon.0) 3171 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:55.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: cluster 2024-09-10T19:04:54.299474+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:04:55.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: audit 2024-09-10T19:04:54.300927+0000 mon.c (mon.2) 693 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:55.774 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[18629]: audit 2024-09-10T19:04:54.302728+0000 mon.a (mon.0) 3173 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:55.774 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.966274+0000 osd.1 (osd.1) 341 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: cluster 2024-09-10T19:04:53.970546+0000 osd.1 (osd.1) 342 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: cluster 2024-09-10T19:04:54.265485+0000 mgr.y (mgr.24526) 618 : cluster [DBG] pgmap v1003: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: audit 2024-09-10T19:04:54.299361+0000 mon.a (mon.0) 3171 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: cluster 2024-09-10T19:04:54.299474+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: audit 2024-09-10T19:04:54.300927+0000 mon.c (mon.2) 693 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:55.775 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:55 smithi083 bash[32381]: audit 2024-09-10T19:04:54.302728+0000 mon.a (mon.0) 3173 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:55.791 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:55.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:55.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:55.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:56.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:56.062 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.966274+0000 osd.1 (osd.1) 341 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:04:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: cluster 2024-09-10T19:04:53.970546+0000 osd.1 (osd.1) 342 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:04:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: cluster 2024-09-10T19:04:54.265485+0000 mgr.y (mgr.24526) 618 : cluster [DBG] pgmap v1003: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:04:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: audit 2024-09-10T19:04:54.299361+0000 mon.a (mon.0) 3171 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi083-77828-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: cluster 2024-09-10T19:04:54.299474+0000 mon.a (mon.0) 3172 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:04:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: audit 2024-09-10T19:04:54.300927+0000 mon.c (mon.2) 693 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:55 smithi165 bash[22591]: audit 2024-09-10T19:04:54.302728+0000 mon.a (mon.0) 3173 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:04:56.165 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:56.201 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:56.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:56.331 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:56.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:56.461 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:56.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:56.600 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:56.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:56.738 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:56.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:54.540597+0000 osd.0 (osd.0) 329 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:54.545663+0000 osd.0 (osd.0) 330 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.120224+0000 osd.5 (osd.5) 354 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.124081+0000 osd.5 (osd.5) 355 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.562321+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.571876+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.751468+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:04:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.817046+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:04:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:55.824799+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:04:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: audit 2024-09-10T19:04:56.311693+0000 mon.a (mon.0) 3175 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:04:56.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:56 smithi165 bash[22591]: cluster 2024-09-10T19:04:56.311913+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:04:56.876 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:56.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:56.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:57.010 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:54.540597+0000 osd.0 (osd.0) 329 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:54.545663+0000 osd.0 (osd.0) 330 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.120224+0000 osd.5 (osd.5) 354 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.124081+0000 osd.5 (osd.5) 355 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.562321+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.571876+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.751468+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:04:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.817046+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:55.824799+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: audit 2024-09-10T19:04:56.311693+0000 mon.a (mon.0) 3175 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[18629]: cluster 2024-09-10T19:04:56.311913+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:54.540597+0000 osd.0 (osd.0) 329 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:54.545663+0000 osd.0 (osd.0) 330 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.120224+0000 osd.5 (osd.5) 354 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.124081+0000 osd.5 (osd.5) 355 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.562321+0000 osd.6 (osd.6) 303 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:04:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.571876+0000 osd.6 (osd.6) 304 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:04:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.751468+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:04:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.817046+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:04:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:55.824799+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:04:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: audit 2024-09-10T19:04:56.311693+0000 mon.a (mon.0) 3175 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi083-77828-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:04:57.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:56 smithi083 bash[32381]: cluster 2024-09-10T19:04:56.311913+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:04:57.114 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:57.147 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:57.208 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:57.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:57.286 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:57.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:57.424 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:04:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:04:57.528 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:57.559 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:57.663 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:57.699 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:57.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:57.835 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:57.939 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:57.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: cluster 2024-09-10T19:04:56.267360+0000 mgr.y (mgr.24526) 619 : cluster [DBG] pgmap v1006: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:57.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: cluster 2024-09-10T19:04:56.844259+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub starts 2024-09-10T19:04:57.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: cluster 2024-09-10T19:04:56.847664+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.b scrub ok 2024-09-10T19:04:57.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: cluster 2024-09-10T19:04:57.315691+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:04:57.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: audit 2024-09-10T19:04:57.324678+0000 mon.c (mon.2) 694 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[18629]: audit 2024-09-10T19:04:57.327485+0000 mon.a (mon.0) 3178 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: cluster 2024-09-10T19:04:56.267360+0000 mgr.y (mgr.24526) 619 : cluster [DBG] pgmap v1006: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: cluster 2024-09-10T19:04:56.844259+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub starts 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: cluster 2024-09-10T19:04:56.847664+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.b scrub ok 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: cluster 2024-09-10T19:04:57.315691+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: audit 2024-09-10T19:04:57.324678+0000 mon.c (mon.2) 694 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:57 smithi083 bash[32381]: audit 2024-09-10T19:04:57.327485+0000 mon.a (mon.0) 3178 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.973 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:57.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:58.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:58.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: cluster 2024-09-10T19:04:56.267360+0000 mgr.y (mgr.24526) 619 : cluster [DBG] pgmap v1006: 174 pgs: 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:58.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: cluster 2024-09-10T19:04:56.844259+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.b scrub starts 2024-09-10T19:04:58.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: cluster 2024-09-10T19:04:56.847664+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.b scrub ok 2024-09-10T19:04:58.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: cluster 2024-09-10T19:04:57.315691+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:04:58.091 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: audit 2024-09-10T19:04:57.324678+0000 mon.c (mon.2) 694 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.092 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:57 smithi165 bash[22591]: audit 2024-09-10T19:04:57.327485+0000 mon.a (mon.0) 3178 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.113 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:58.216 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:58.248 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:04:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:04:58.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:58.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:04:58.375 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:58.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:58.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.650 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:58.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:58.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:58.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:58.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:58.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:59.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: cluster 2024-09-10T19:04:57.120456+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: cluster 2024-09-10T19:04:57.124655+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: audit 2024-09-10T19:04:58.240327+0000 mgr.y (mgr.24526) 620 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.268195+0000 mgr.y (mgr.24526) 621 : cluster [DBG] pgmap v1009: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: audit 2024-09-10T19:04:58.314709+0000 mon.a (mon.0) 3179 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.315072+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: audit 2024-09-10T19:04:58.341089+0000 mon.c (mon.2) 695 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[32381]: audit 2024-09-10T19:04:58.341409+0000 mon.a (mon.0) 3181 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: cluster 2024-09-10T19:04:57.120456+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: cluster 2024-09-10T19:04:57.124655+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: audit 2024-09-10T19:04:58.240327+0000 mgr.y (mgr.24526) 620 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.268195+0000 mgr.y (mgr.24526) 621 : cluster [DBG] pgmap v1009: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: audit 2024-09-10T19:04:58.314709+0000 mon.a (mon.0) 3179 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.315072+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:04:59.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: audit 2024-09-10T19:04:58.341089+0000 mon.c (mon.2) 695 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:58 smithi083 bash[18629]: audit 2024-09-10T19:04:58.341409+0000 mon.a (mon.0) 3181 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: cluster 2024-09-10T19:04:57.120456+0000 osd.5 (osd.5) 356 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: cluster 2024-09-10T19:04:57.124655+0000 osd.5 (osd.5) 357 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: audit 2024-09-10T19:04:58.240327+0000 mgr.y (mgr.24526) 620 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.268195+0000 mgr.y (mgr.24526) 621 : cluster [DBG] pgmap v1009: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: audit 2024-09-10T19:04:58.314709+0000 mon.a (mon.0) 3179 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.315072+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: audit 2024-09-10T19:04:58.341089+0000 mon.c (mon.2) 695 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.044 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:58 smithi165 bash[22591]: audit 2024-09-10T19:04:58.341409+0000 mon.a (mon.0) 3181 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:59.064 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:04:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:04:59.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:59.203 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:04:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:04:59.238 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:04:59.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:59.336 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:59.441 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:59.468 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:04:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:04:59.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:59.606 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:04:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:04:59.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:59.744 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:04:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:04:59.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.010533+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.014036+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.491740+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub starts 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.494979+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub ok 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.765016+0000 mon.a (mon.0) 3182 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.919573+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:58.924271+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.077411+0000 osd.3 (osd.3) 358 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.080356+0000 osd.3 (osd.3) 359 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: audit 2024-09-10T19:04:59.288081+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.288243+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: audit 2024-09-10T19:04:59.289392+0000 mon.c (mon.2) 696 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: audit 2024-09-10T19:04:59.290488+0000 mon.a (mon.0) 3185 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:04:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:04:59 smithi165 bash[22591]: audit 2024-09-10T19:04:59.571842+0000 mon.c (mon.2) 697 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:59.881 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:04:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:04:59.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:04:59.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:00.020 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.010533+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.014036+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.491740+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub starts 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.494979+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub ok 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.765016+0000 mon.a (mon.0) 3182 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.919573+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:58.924271+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.077411+0000 osd.3 (osd.3) 358 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.080356+0000 osd.3 (osd.3) 359 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: audit 2024-09-10T19:04:59.288081+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:00.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.288243+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: audit 2024-09-10T19:04:59.289392+0000 mon.c (mon.2) 696 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: audit 2024-09-10T19:04:59.290488+0000 mon.a (mon.0) 3185 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[32381]: audit 2024-09-10T19:04:59.571842+0000 mon.c (mon.2) 697 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.010533+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.a scrub starts 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.014036+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.a scrub ok 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.491740+0000 osd.6 (osd.6) 305 : cluster [DBG] 4.a scrub starts 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.494979+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub ok 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.765016+0000 mon.a (mon.0) 3182 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.919573+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:58.924271+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:05:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.077411+0000 osd.3 (osd.3) 358 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.080356+0000 osd.3 (osd.3) 359 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: audit 2024-09-10T19:04:59.288081+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.288243+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: audit 2024-09-10T19:04:59.289392+0000 mon.c (mon.2) 696 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: audit 2024-09-10T19:04:59.290488+0000 mon.a (mon.0) 3185 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:00.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:04:59 smithi083 bash[18629]: audit 2024-09-10T19:04:59.571842+0000 mon.c (mon.2) 697 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:00.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:00.160 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:00.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:00.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:00.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:00.436 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:00.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:00.574 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:00.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:00.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:00.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:00.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:00.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:00.847 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:00.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:00.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.010241+0000 osd.1 (osd.1) 345 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:05:00.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.013617+0000 osd.1 (osd.1) 346 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:05:00.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.458930+0000 osd.6 (osd.6) 307 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:05:00.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.464198+0000 osd.6 (osd.6) 308 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.523838+0000 osd.0 (osd.0) 331 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.527238+0000 osd.0 (osd.0) 332 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.882552+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.882953+0000 osd.4 (osd.4) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.898436+0000 osd.2 (osd.2) 269 : cluster [DBG] 12.a scrub starts 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:04:59.901755+0000 osd.2 (osd.2) 270 : cluster [DBG] 12.a scrub ok 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.269031+0000 mgr.y (mgr.24526) 622 : cluster [DBG] pgmap v1012: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: audit 2024-09-10T19:05:00.290767+0000 mon.a (mon.0) 3186 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.290868+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: audit 2024-09-10T19:05:00.343806+0000 mon.c (mon.2) 698 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:00.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:00 smithi165 bash[22591]: audit 2024-09-10T19:05:00.344465+0000 mon.a (mon.0) 3188 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:00.984 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:00.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.010241+0000 osd.1 (osd.1) 345 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:05:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.013617+0000 osd.1 (osd.1) 346 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:05:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.458930+0000 osd.6 (osd.6) 307 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:05:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.464198+0000 osd.6 (osd.6) 308 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.523838+0000 osd.0 (osd.0) 331 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.527238+0000 osd.0 (osd.0) 332 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.882552+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.882953+0000 osd.4 (osd.4) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.898436+0000 osd.2 (osd.2) 269 : cluster [DBG] 12.a scrub starts 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:04:59.901755+0000 osd.2 (osd.2) 270 : cluster [DBG] 12.a scrub ok 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.269031+0000 mgr.y (mgr.24526) 622 : cluster [DBG] pgmap v1012: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: audit 2024-09-10T19:05:00.290767+0000 mon.a (mon.0) 3186 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.290868+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: audit 2024-09-10T19:05:00.343806+0000 mon.c (mon.2) 698 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:01.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[18629]: audit 2024-09-10T19:05:00.344465+0000 mon.a (mon.0) 3188 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.010241+0000 osd.1 (osd.1) 345 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.013617+0000 osd.1 (osd.1) 346 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.458930+0000 osd.6 (osd.6) 307 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.464198+0000 osd.6 (osd.6) 308 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.523838+0000 osd.0 (osd.0) 331 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.527238+0000 osd.0 (osd.0) 332 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.882552+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.882953+0000 osd.4 (osd.4) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.898436+0000 osd.2 (osd.2) 269 : cluster [DBG] 12.a scrub starts 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:04:59.901755+0000 osd.2 (osd.2) 270 : cluster [DBG] 12.a scrub ok 2024-09-10T19:05:01.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.269031+0000 mgr.y (mgr.24526) 622 : cluster [DBG] pgmap v1012: 214 pgs: 40 unknown, 174 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:01.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: audit 2024-09-10T19:05:00.290767+0000 mon.a (mon.0) 3186 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:01.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.290868+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:05:01.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: audit 2024-09-10T19:05:00.343806+0000 mon.c (mon.2) 698 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:01.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:00 smithi083 bash[32381]: audit 2024-09-10T19:05:00.344465+0000 mon.a (mon.0) 3188 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:01.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 -- 172.21.15.83:0/2539878914 >> v1:172.21.15.83:6790/0 conn(0x7f97081014f0 legacy=0x7f97081018c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 -- 172.21.15.83:0/2539878914 shutdown_connections 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 -- 172.21.15.83:0/2539878914 >> 172.21.15.83:0/2539878914 conn(0x7f9708074a20 msgr2=0x7f97080f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 -- 172.21.15.83:0/2539878914 shutdown_connections 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 -- 172.21.15.83:0/2539878914 wait complete. 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 librados: starting msgr at 2024-09-10T19:05:01.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 librados: starting objecter 2024-09-10T19:05:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.084+0000 7f970cee0640 1 Processor -- start 2024-09-10T19:05:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- start start 2024-09-10T19:05:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 librados: setting wanted keys 2024-09-10T19:05:01.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 librados: calling monclient init 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9708100ef0 con 0x7f97081014f0 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f970811c2e0 con 0x7f9708107e30 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f970811d4e0 con 0x7f97081042d0 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f9705d9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f97081042d0 0x7f9708116480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:36492/0 (socket says 172.21.15.83:36492) 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f9705d9b640 1 -- 172.21.15.83:0/3373410296 learned_addr learned my addr 172.21.15.83:0/3373410296 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f97081014f0 0x7f9708100340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34382/0 (socket says 172.21.15.83:34382) 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter ms_handle_connect 0x7f97081042d0 2024-09-10T19:05:01.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter ms_handle_connect 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628620788 0 0) 0x7f9708100ef0 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f0003720 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter ms_handle_connect 0x7f9708107e30 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2924539674 0 0) 0x7f96f0003720 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9708100ef0 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f96f80015d0 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388694372 0 0) 0x7f970811c2e0 con 0x7f9708107e30 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f0003720 con 0x7f9708107e30 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3477887784 0 0) 0x7f9708100ef0 con 0x7f97081014f0 2024-09-10T19:05:01.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6790/0 conn(0x7f97081042d0 legacy=0x7f9708116480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.165:6789/0 conn(0x7f9708107e30 legacy=0x7f9708119bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970811e6e0 con 0x7f97081014f0 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970811b330 con 0x7f97081014f0 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 10 client.25623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f970811b800 con 0x7f97081014f0 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 librados: init done 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f96f8002400 con 0x7f97081014f0 2024-09-10T19:05:01.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f96f8003650 con 0x7f97081014f0 2024-09-10T19:05:01.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.088+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f96e8004ff0 con 0x7f97081014f0 2024-09-10T19:05:01.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f96f8001880 con 0x7f97081014f0 2024-09-10T19:05:01.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(657..657 src has 569..657) v4 ==== 8419+0+0 (unknown 1805042588 0 0) 0x7f96f808a840 con 0x7f97081014f0 2024-09-10T19:05:01.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 osd_map(657..657 src has 569..657) v4 2024-09-10T19:05:01.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 3 client.25623.objecter handle_osd_map got epochs [657,657] > 0 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 3 client.25623.objecter handle_osd_map decoding full epoch 657 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 20 client.25623.objecter dump_active .. 0 homeless 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v657) v1 ==== 54+0+16 (unknown 784002423 0 2420000192) 0x7f96f808ac00 con 0x7f97081014f0 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96f0073780 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 10 client.25623.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 1 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 1 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 0 1 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 4 2024-09-10T19:05:01.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 10 client.25623.objecter _send_command 1 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e8005290 con 0x7f96e8004610 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e8004610 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e8005290 con 0x7f96e8004610 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 1 = system:0 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 4 2024-09-10T19:05:01.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.092+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 0 1 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 10 client.25623.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 2 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 2 2024-09-10T19:05:01.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8005810 osd=0 3 2024-09-10T19:05:01.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 0 2 2024-09-10T19:05:01.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 4 2024-09-10T19:05:01.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 10 client.25623.objecter _send_command 2 2024-09-10T19:05:01.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e8004c60 con 0x7f96e8004610 2024-09-10T19:05:01.116 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:01.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e8004c60 con 0x7f96e8004610 2024-09-10T19:05:01.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 4 2024-09-10T19:05:01.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 0 2 2024-09-10T19:05:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:01.118 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:05:01.119 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:05:01.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 10 client.25623.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 1 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 4 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 10 client.25623.objecter _send_command 3 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e8003b10 con 0x7f96e80035e0 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.112+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e80035e0 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.116+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e8003b10 con 0x7f96e80035e0 2024-09-10T19:05:01.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.116+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:05:01.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.116+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 3 = system:0 2024-09-10T19:05:01.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.116+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 4 2024-09-10T19:05:01.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.116+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 1 3 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 10 client.25623.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 4 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 4 2024-09-10T19:05:01.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8004e10 osd=1 3 2024-09-10T19:05:01.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 1 4 2024-09-10T19:05:01.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 4 2024-09-10T19:05:01.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 10 client.25623.objecter _send_command 4 2024-09-10T19:05:01.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e8003060 con 0x7f96e80035e0 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e8003060 con 0x7f96e80035e0 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 4 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.128+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 1 4 2024-09-10T19:05:01.134 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 10 client.25623.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 5 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 5 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 2 5 2024-09-10T19:05:01.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 4 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 10 client.25623.objecter _send_command 5 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e80022a0 con 0x7f96e8011760 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e8011760 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e80022a0 con 0x7f96e8011760 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 5 = system:0 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 4 2024-09-10T19:05:01.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.132+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 2 5 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 10 client.25623.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 6 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 6 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8001b60 osd=2 3 2024-09-10T19:05:01.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 2 6 2024-09-10T19:05:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 4 2024-09-10T19:05:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 10 client.25623.objecter _send_command 6 2024-09-10T19:05:01.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.152+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e8001650 con 0x7f96e8011760 2024-09-10T19:05:01.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e8001650 con 0x7f96e8011760 2024-09-10T19:05:01.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 4 2024-09-10T19:05:01.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 2 6 2024-09-10T19:05:01.159 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 10 client.25623.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 7 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 7 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 3 7 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 4 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 10 client.25623.objecter _send_command 7 2024-09-10T19:05:01.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e800d140 con 0x7f96e80163e0 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e80163e0 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e800d140 con 0x7f96e80163e0 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 7 = system:0 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 4 2024-09-10T19:05:01.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.156+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 3 7 2024-09-10T19:05:01.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 10 client.25623.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 8 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 8 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c990 osd=3 3 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 3 8 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 4 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 10 client.25623.objecter _send_command 8 2024-09-10T19:05:01.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.172+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e800c410 con 0x7f96e80163e0 2024-09-10T19:05:01.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e800c410 con 0x7f96e80163e0 2024-09-10T19:05:01.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 4 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 3 8 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 10 client.25623.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 9 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 9 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 4 9 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 4 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 10 client.25623.objecter _send_command 9 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e800c370 con 0x7f96e8015f30 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e8015f30 2024-09-10T19:05:01.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e800c370 con 0x7f96e8015f30 2024-09-10T19:05:01.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:05:01.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 9 = system:0 2024-09-10T19:05:01.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 4 2024-09-10T19:05:01.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.176+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 4 9 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 10 client.25623.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 10 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 10 2024-09-10T19:05:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c1d0 osd=4 3 2024-09-10T19:05:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 4 10 2024-09-10T19:05:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 4 2024-09-10T19:05:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 10 client.25623.objecter _send_command 10 2024-09-10T19:05:01.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.196+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e800c370 con 0x7f96e8015f30 2024-09-10T19:05:01.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e800c370 con 0x7f96e8015f30 2024-09-10T19:05:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 4 2024-09-10T19:05:01.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 4 10 2024-09-10T19:05:01.205 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 10 client.25623.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 11 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 11 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 5 11 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 4 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 10 client.25623.objecter _send_command 11 2024-09-10T19:05:01.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e801c1f0 con 0x7f96e801bcc0 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.200+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e801bcc0 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.204+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e801c1f0 con 0x7f96e801bcc0 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.204+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.204+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 11 = system:0 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.204+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 4 2024-09-10T19:05:01.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.204+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 5 11 2024-09-10T19:05:01.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 10 client.25623.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 12 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 12 2024-09-10T19:05:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e800c370 osd=5 3 2024-09-10T19:05:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 5 12 2024-09-10T19:05:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 4 2024-09-10T19:05:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 10 client.25623.objecter _send_command 12 2024-09-10T19:05:01.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.224+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e801c1f0 con 0x7f96e801bcc0 2024-09-10T19:05:01.232 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:05:01.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e801c1f0 con 0x7f96e801bcc0 2024-09-10T19:05:01.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 4 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 5 12 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 10 client.25623.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 13 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 13 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 6 13 2024-09-10T19:05:01.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 4 2024-09-10T19:05:01.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 10 client.25623.objecter _send_command 13 2024-09-10T19:05:01.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e8026ee0 con 0x7f96e80233e0 2024-09-10T19:05:01.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.228+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e80233e0 2024-09-10T19:05:01.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.232+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e8026ee0 con 0x7f96e80233e0 2024-09-10T19:05:01.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.232+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:05:01.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.232+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 13 = system:0 2024-09-10T19:05:01.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.232+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 4 2024-09-10T19:05:01.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.232+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 6 13 2024-09-10T19:05:01.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:01.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 10 client.25623.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 14 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 14 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e801c1f0 osd=6 3 2024-09-10T19:05:01.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 6 14 2024-09-10T19:05:01.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 4 2024-09-10T19:05:01.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 10 client.25623.objecter _send_command 14 2024-09-10T19:05:01.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.248+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e8026ee0 con 0x7f96e80233e0 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e8026ee0 con 0x7f96e80233e0 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 4 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 6 14 2024-09-10T19:05:01.257 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 10 client.25623.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 15 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 15 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 7 15 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 4 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 10 client.25623.objecter _send_command 15 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.252+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96e80291e0 con 0x7f96e8027560 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 10 client.25623.objecter ms_handle_connect 0x7f96e8027560 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f96e80291e0 con 0x7f96e8027560 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:05:01.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 15 = system:0 2024-09-10T19:05:01.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 4 2024-09-10T19:05:01.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.256+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 7 15 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 10 client.25623.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign -1 16 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 20 client.25623.objecter _get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 15 client.25623.objecter _session_command_op_remove -1 16 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 20 client.25623.objecter get_session s=0x7f96e8026ee0 osd=7 3 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 15 client.25623.objecter _session_command_op_assign 7 16 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 4 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 10 client.25623.objecter _send_command 16 2024-09-10T19:05:01.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.268+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f96e80291e0 con 0x7f96e8027560 2024-09-10T19:05:01.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.272+0000 7f96f6ffd640 1 -- 172.21.15.83:0/3373410296 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f96e80291e0 con 0x7f96e8027560 2024-09-10T19:05:01.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.272+0000 7f96f6ffd640 10 client.25623.objecter ms_dispatch 0x7f9708000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:01.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.272+0000 7f96f6ffd640 10 client.25623.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:01.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.272+0000 7f96f6ffd640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 4 2024-09-10T19:05:01.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.272+0000 7f96f6ffd640 15 client.25623.objecter _session_command_op_remove 7 16 2024-09-10T19:05:01.279 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 librados: watch_flush enter 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 librados: watch_flush exit 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.0 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6801/2535527483 conn(0x7f96e8004610 legacy=0x7f96e8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8005810 osd=0 1 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.1 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6805/2309624637 conn(0x7f96e80035e0 legacy=0x7f96e800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8004e10 osd=1 1 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.2 2024-09-10T19:05:01.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6809/386094220 conn(0x7f96e8011760 legacy=0x7f96e8013b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8001b60 osd=2 1 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.3 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6813/1279688021 conn(0x7f96e80163e0 legacy=0x7f96e80187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c990 osd=3 1 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.4 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.165:6800/3225515265 conn(0x7f96e8015f30 legacy=0x7f96e801b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c1d0 osd=4 1 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.5 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.165:6804/571173777 conn(0x7f96e801bcc0 legacy=0x7f96e80217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e800c370 osd=5 1 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.6 2024-09-10T19:05:01.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.165:6808/1317922936 conn(0x7f96e80233e0 legacy=0x7f96e80257c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e801c1f0 osd=6 1 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter close_session for osd.7 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.165:6812/1610151650 conn(0x7f96e8027560 legacy=0x7f96e8027990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter put_session s=0x7f96e8026ee0 osd=7 1 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 20 client.25623.objecter shutdown clearing up homeless session... 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 10 client.25623.objecter successfully canceled tick 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f9706d9d640 1 -- 172.21.15.83:0/3373410296 reap_dead start 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f9706d9d640 1 -- 172.21.15.83:0/3373410296 reap_dead start 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f9706d9d640 1 -- 172.21.15.83:0/3373410296 reap_dead start 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f9706d9d640 1 -- 172.21.15.83:0/3373410296 reap_dead start 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6800/3071793735 conn(0x7f96f0073780 legacy=0x7f96f0075c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f9706d9d640 1 -- 172.21.15.83:0/3373410296 reap_dead start 2024-09-10T19:05:01.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> v1:172.21.15.83:6789/0 conn(0x7f97081014f0 legacy=0x7f9708100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 shutdown_connections 2024-09-10T19:05:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 >> 172.21.15.83:0/3373410296 conn(0x7f9708074a20 msgr2=0x7f970810b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 shutdown_connections 2024-09-10T19:05:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 -- 172.21.15.83:0/3373410296 wait complete. 2024-09-10T19:05:01.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:01.276+0000 7f970cee0640 1 librados: shutdown 2024-09-10T19:05:01.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:01.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:01.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:01.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:01.491 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:01.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:01.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:01.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:01.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:01.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:01.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:01.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.002889+0000 osd.1 (osd.1) 347 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.009571+0000 osd.1 (osd.1) 348 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.878123+0000 osd.2 (osd.2) 271 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.881349+0000 osd.2 (osd.2) 272 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:01.082587+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.c scrub starts 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:01.085063+0000 osd.3 (osd.3) 361 : cluster [DBG] 4.c scrub ok 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: audit 2024-09-10T19:05:01.094337+0000 mon.a (mon.0) 3189 : audit [DBG] from='client.? v1:172.21.15.83:0/3373410296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: audit 2024-09-10T19:05:01.293218+0000 mon.a (mon.0) 3190 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: cluster 2024-09-10T19:05:01.293397+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: audit 2024-09-10T19:05:01.296565+0000 mon.c (mon.2) 699 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[32381]: audit 2024-09-10T19:05:01.297090+0000 mon.a (mon.0) 3192 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:01.905 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.002889+0000 osd.1 (osd.1) 347 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.009571+0000 osd.1 (osd.1) 348 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.878123+0000 osd.2 (osd.2) 271 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.881349+0000 osd.2 (osd.2) 272 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:01.082587+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.c scrub starts 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:01.085063+0000 osd.3 (osd.3) 361 : cluster [DBG] 4.c scrub ok 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: audit 2024-09-10T19:05:01.094337+0000 mon.a (mon.0) 3189 : audit [DBG] from='client.? v1:172.21.15.83:0/3373410296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: audit 2024-09-10T19:05:01.293218+0000 mon.a (mon.0) 3190 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: cluster 2024-09-10T19:05:01.293397+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: audit 2024-09-10T19:05:01.296565+0000 mon.c (mon.2) 699 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:01.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:01 smithi083 bash[18629]: audit 2024-09-10T19:05:01.297090+0000 mon.a (mon.0) 3192 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:01.926 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:01.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:01.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:02.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:02.065 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.002889+0000 osd.1 (osd.1) 347 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:05:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.009571+0000 osd.1 (osd.1) 348 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:05:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.878123+0000 osd.2 (osd.2) 271 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.881349+0000 osd.2 (osd.2) 272 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:01.082587+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.c scrub starts 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:01.085063+0000 osd.3 (osd.3) 361 : cluster [DBG] 4.c scrub ok 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: audit 2024-09-10T19:05:01.094337+0000 mon.a (mon.0) 3189 : audit [DBG] from='client.? v1:172.21.15.83:0/3373410296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: audit 2024-09-10T19:05:01.293218+0000 mon.a (mon.0) 3190 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: cluster 2024-09-10T19:05:01.293397+0000 mon.a (mon.0) 3191 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: audit 2024-09-10T19:05:01.296565+0000 mon.c (mon.2) 699 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:01 smithi165 bash[22591]: audit 2024-09-10T19:05:01.297090+0000 mon.a (mon.0) 3192 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]: dispatch 2024-09-10T19:05:02.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:02.202 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:02.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:02.332 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:02.435 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:02.469 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:02.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:02.606 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:02.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:02.744 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:02.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:02.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:02.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:02.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:02.984 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:02.997 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:02] "GET /metrics HTTP/1.1" 200 228944 "" "Prometheus/2.43.0" 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.980914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:00.983264+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:01.735681+0000 osd.0 (osd.0) 333 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:01.736057+0000 osd.0 (osd.0) 334 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.094838+0000 osd.3 (osd.3) 362 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.098367+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.269856+0000 mgr.y (mgr.24526) 623 : cluster [DBG] pgmap v1015: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:02.998 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: audit 2024-09-10T19:05:02.297752+0000 mon.a (mon.0) 3193 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.297878+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.441207+0000 osd.0 (osd.0) 335 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.980914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:00.983264+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:01.735681+0000 osd.0 (osd.0) 333 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:01.736057+0000 osd.0 (osd.0) 334 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.094838+0000 osd.3 (osd.3) 362 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:05:02.999 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.098367+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:05:03.000 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.269856+0000 mgr.y (mgr.24526) 623 : cluster [DBG] pgmap v1015: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:03.000 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: audit 2024-09-10T19:05:02.297752+0000 mon.a (mon.0) 3193 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:03.000 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.297878+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:05:03.000 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:02 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.441207+0000 osd.0 (osd.0) 335 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:05:03.015 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.980914+0000 osd.1 (osd.1) 349 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:00.983264+0000 osd.1 (osd.1) 350 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:01.735681+0000 osd.0 (osd.0) 333 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:01.736057+0000 osd.0 (osd.0) 334 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.094838+0000 osd.3 (osd.3) 362 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.098367+0000 osd.3 (osd.3) 363 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.269856+0000 mgr.y (mgr.24526) 623 : cluster [DBG] pgmap v1015: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: audit 2024-09-10T19:05:02.297752+0000 mon.a (mon.0) 3193 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-112"}]': finished 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.297878+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:05:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:02 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.441207+0000 osd.0 (osd.0) 335 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:05:03.119 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:03.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:03.290 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:03.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.425 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:03.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:03.563 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:03.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:03.694 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:03.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:03.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:03.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.344233+0000 osd.7 (osd.7) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.344540+0000 osd.7 (osd.7) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.424931+0000 osd.6 (osd.6) 309 : cluster [DBG] 3.a scrub starts 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.431166+0000 osd.6 (osd.6) 310 : cluster [DBG] 3.a scrub ok 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.444164+0000 osd.0 (osd.0) 336 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:05:03.941 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.781587+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:05:03.942 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:02.785040+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:05:03.942 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:03 smithi165 bash[22591]: cluster 2024-09-10T19:05:03.301177+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:05:03.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:03.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.344233+0000 osd.7 (osd.7) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.344540+0000 osd.7 (osd.7) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.424931+0000 osd.6 (osd.6) 309 : cluster [DBG] 3.a scrub starts 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.431166+0000 osd.6 (osd.6) 310 : cluster [DBG] 3.a scrub ok 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.444164+0000 osd.0 (osd.0) 336 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.781587+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:02.785040+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:05:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[18629]: cluster 2024-09-10T19:05:03.301177+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.344233+0000 osd.7 (osd.7) 354 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.344540+0000 osd.7 (osd.7) 355 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.424931+0000 osd.6 (osd.6) 309 : cluster [DBG] 3.a scrub starts 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.431166+0000 osd.6 (osd.6) 310 : cluster [DBG] 3.a scrub ok 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.444164+0000 osd.0 (osd.0) 336 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.781587+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:05:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:02.785040+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:05:04.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:03 smithi083 bash[32381]: cluster 2024-09-10T19:05:03.301177+0000 mon.a (mon.0) 3195 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:05:04.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:04.091 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:04.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:04.221 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:04.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:04.349 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:04.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:04.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:04.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:04.623 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:04.727 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:04.759 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:04.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:04.897 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:04.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:05.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:03.173105+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:03.177053+0000 osd.5 (osd.5) 359 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:03.429134+0000 osd.6 (osd.6) 311 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:03.771066+0000 osd.7 (osd.7) 358 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:03.873973+0000 mon.c (mon.2) 700 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.270664+0000 mgr.y (mgr.24526) 624 : cluster [DBG] pgmap v1018: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.278525+0000 mon.a (mon.0) 3196 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.303908+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:04.309487+0000 mon.c (mon.2) 701 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.015 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:04.311014+0000 mon.a (mon.0) 3198 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:04.326573+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:04.327967+0000 mon.c (mon.2) 703 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: audit 2024-09-10T19:05:04.335254+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.438469+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.441623+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:03.173105+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:03.177053+0000 osd.5 (osd.5) 359 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:03.429134+0000 osd.6 (osd.6) 311 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:03.771066+0000 osd.7 (osd.7) 358 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:05:05.016 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:03.873973+0000 mon.c (mon.2) 700 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.270664+0000 mgr.y (mgr.24526) 624 : cluster [DBG] pgmap v1018: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.278525+0000 mon.a (mon.0) 3196 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.303908+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:04.309487+0000 mon.c (mon.2) 701 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:04.311014+0000 mon.a (mon.0) 3198 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:04.326573+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:04.327967+0000 mon.c (mon.2) 703 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: audit 2024-09-10T19:05:04.335254+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.438469+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:05:05.017 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:04 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.441623+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:05:05.033 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:03.173105+0000 osd.5 (osd.5) 358 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:03.177053+0000 osd.5 (osd.5) 359 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:03.429134+0000 osd.6 (osd.6) 311 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:03.771066+0000 osd.7 (osd.7) 358 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:03.873973+0000 mon.c (mon.2) 700 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.270664+0000 mgr.y (mgr.24526) 624 : cluster [DBG] pgmap v1018: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.278525+0000 mon.a (mon.0) 3196 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.303908+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:04.309487+0000 mon.c (mon.2) 701 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:04.311014+0000 mon.a (mon.0) 3198 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:04.326573+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:04.327967+0000 mon.c (mon.2) 703 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: audit 2024-09-10T19:05:04.335254+0000 mon.a (mon.0) 3199 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.438469+0000 osd.0 (osd.0) 337 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:05:05.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:04 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.441623+0000 osd.0 (osd.0) 338 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:05:05.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.174 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:05.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:05.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:05.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:05.447 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:05.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:05.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:05.723 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:05.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:05.860 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:05.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:05.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:05:05.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.997 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:05.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:06.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.147041+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.149046+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.473863+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.e scrub starts 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.477045+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.e scrub ok 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.754576+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:05:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.757940+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.964442+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.e scrub starts 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:04.968126+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.e scrub ok 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: audit 2024-09-10T19:05:05.305473+0000 mon.a (mon.0) 3200 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.305770+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: audit 2024-09-10T19:05:05.379707+0000 mon.c (mon.2) 704 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:05 smithi165 bash[22591]: audit 2024-09-10T19:05:05.380340+0000 mon.a (mon.0) 3202 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.132 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:06.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.271 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.147041+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:05:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.149046+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:05:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.473863+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.e scrub starts 2024-09-10T19:05:06.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.477045+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.e scrub ok 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.754576+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.757940+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.964442+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.e scrub starts 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:04.968126+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.e scrub ok 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: audit 2024-09-10T19:05:05.305473+0000 mon.a (mon.0) 3200 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.305770+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: audit 2024-09-10T19:05:05.379707+0000 mon.c (mon.2) 704 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[18629]: audit 2024-09-10T19:05:05.380340+0000 mon.a (mon.0) 3202 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.147041+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.149046+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.473863+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.e scrub starts 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.477045+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.e scrub ok 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.754576+0000 osd.7 (osd.7) 359 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.757940+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.964442+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.e scrub starts 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:04.968126+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.e scrub ok 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: audit 2024-09-10T19:05:05.305473+0000 mon.a (mon.0) 3200 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:06.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.305770+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:05:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: audit 2024-09-10T19:05:05.379707+0000 mon.c (mon.2) 704 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.294 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:05 smithi083 bash[32381]: audit 2024-09-10T19:05:05.380340+0000 mon.a (mon.0) 3202 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:06.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.407 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:06.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:06.542 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:06.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:06.679 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:06.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:06.813 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:06.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:06.945 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:07.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.946716+0000 osd.4 (osd.4) 356 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.950783+0000 osd.4 (osd.4) 357 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.993155+0000 osd.3 (osd.3) 364 : cluster [DBG] 6.b scrub starts 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.996570+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub ok 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.272680+0000 mgr.y (mgr.24526) 625 : cluster [DBG] pgmap v1021: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:05:07.062 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: audit 2024-09-10T19:05:06.308800+0000 mon.a (mon.0) 3203 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:07.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.308999+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:05:07.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: audit 2024-09-10T19:05:06.311095+0000 mon.c (mon.2) 705 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:06 smithi165 bash[22591]: audit 2024-09-10T19:05:06.311880+0000 mon.a (mon.0) 3205 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.081 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:07.184 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.198 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.946716+0000 osd.4 (osd.4) 356 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.950783+0000 osd.4 (osd.4) 357 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.993155+0000 osd.3 (osd.3) 364 : cluster [DBG] 6.b scrub starts 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.996570+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub ok 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.272680+0000 mgr.y (mgr.24526) 625 : cluster [DBG] pgmap v1021: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: audit 2024-09-10T19:05:06.308800+0000 mon.a (mon.0) 3203 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.308999+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: audit 2024-09-10T19:05:06.311095+0000 mon.c (mon.2) 705 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[18629]: audit 2024-09-10T19:05:06.311880+0000 mon.a (mon.0) 3205 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.946716+0000 osd.4 (osd.4) 356 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.950783+0000 osd.4 (osd.4) 357 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:05:07.199 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.993155+0000 osd.3 (osd.3) 364 : cluster [DBG] 6.b scrub starts 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.996570+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.b scrub ok 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.272680+0000 mgr.y (mgr.24526) 625 : cluster [DBG] pgmap v1021: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: audit 2024-09-10T19:05:06.308800+0000 mon.a (mon.0) 3203 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.308999+0000 mon.a (mon.0) 3204 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: audit 2024-09-10T19:05:06.311095+0000 mon.c (mon.2) 705 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.200 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:06 smithi083 bash[32381]: audit 2024-09-10T19:05:06.311880+0000 mon.a (mon.0) 3205 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:07.217 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:07.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:07.351 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:07.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:07.590 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.625 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:07.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:07.758 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:07.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:07.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:07.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:08.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.886562+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:05.889671+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.478130+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.481808+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.706368+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.709400+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.946081+0000 osd.4 (osd.4) 358 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.950083+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.019941+0000 osd.2 (osd.2) 275 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.022251+0000 osd.2 (osd.2) 276 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: audit 2024-09-10T19:05:07.310660+0000 mon.a (mon.0) 3206 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.310733+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: audit 2024-09-10T19:05:07.312670+0000 mon.c (mon.2) 706 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:07 smithi165 bash[22591]: audit 2024-09-10T19:05:07.313254+0000 mon.a (mon.0) 3208 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:08.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.886562+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:05.889671+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.478130+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.481808+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.706368+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.709400+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.946081+0000 osd.4 (osd.4) 358 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.950083+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.019941+0000 osd.2 (osd.2) 275 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.022251+0000 osd.2 (osd.2) 276 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: audit 2024-09-10T19:05:07.310660+0000 mon.a (mon.0) 3206 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:08.151 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.310733+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: audit 2024-09-10T19:05:07.312670+0000 mon.c (mon.2) 706 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[32381]: audit 2024-09-10T19:05:07.313254+0000 mon.a (mon.0) 3208 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.886562+0000 osd.1 (osd.1) 351 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:05.889671+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.478130+0000 osd.6 (osd.6) 314 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.481808+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.706368+0000 osd.7 (osd.7) 361 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.709400+0000 osd.7 (osd.7) 362 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.946081+0000 osd.4 (osd.4) 358 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:05:08.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.950083+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.019941+0000 osd.2 (osd.2) 275 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.022251+0000 osd.2 (osd.2) 276 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: audit 2024-09-10T19:05:07.310660+0000 mon.a (mon.0) 3206 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.310733+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: audit 2024-09-10T19:05:07.312670+0000 mon.c (mon.2) 706 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.153 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:07 smithi083 bash[18629]: audit 2024-09-10T19:05:07.313254+0000 mon.a (mon.0) 3208 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]: dispatch 2024-09-10T19:05:08.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:08.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.288 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:08.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:08.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:08.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:08.551 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:08.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:08.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.725 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:08.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:08.853 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:08.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.883024+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.f scrub starts 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:06.885540+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.f scrub ok 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.217461+0000 osd.5 (osd.5) 362 : cluster [DBG] 6.a scrub starts 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.220808+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.a scrub ok 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.661415+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.665269+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.737827+0000 osd.3 (osd.3) 366 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:08.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.738332+0000 osd.3 (osd.3) 367 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.903361+0000 osd.4 (osd.4) 360 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.910207+0000 osd.4 (osd.4) 361 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.985956+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:07.989460+0000 osd.3 (osd.3) 369 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: audit 2024-09-10T19:05:08.251667+0000 mgr.y (mgr.24526) 626 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:08.273426+0000 mgr.y (mgr.24526) 627 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:08.304561+0000 mon.a (mon.0) 3209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: audit 2024-09-10T19:05:08.316094+0000 mon.a (mon.0) 3210 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]': finished 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: cluster 2024-09-10T19:05:08.316518+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: audit 2024-09-10T19:05:08.382047+0000 mon.c (mon.2) 707 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:08.970 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:08 smithi165 bash[22591]: audit 2024-09-10T19:05:08.382700+0000 mon.a (mon.0) 3212 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:08.986 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:08.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:09.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:09.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:09.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:09.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.883024+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.f scrub starts 2024-09-10T19:05:09.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:06.885540+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.f scrub ok 2024-09-10T19:05:09.239 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.217461+0000 osd.5 (osd.5) 362 : cluster [DBG] 6.a scrub starts 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.220808+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.a scrub ok 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.661415+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.665269+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.737827+0000 osd.3 (osd.3) 366 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.738332+0000 osd.3 (osd.3) 367 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.903361+0000 osd.4 (osd.4) 360 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.910207+0000 osd.4 (osd.4) 361 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.985956+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:07.989460+0000 osd.3 (osd.3) 369 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: audit 2024-09-10T19:05:08.251667+0000 mgr.y (mgr.24526) 626 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:08.273426+0000 mgr.y (mgr.24526) 627 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:09.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:08.304561+0000 mon.a (mon.0) 3209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: audit 2024-09-10T19:05:08.316094+0000 mon.a (mon.0) 3210 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]': finished 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: cluster 2024-09-10T19:05:08.316518+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: audit 2024-09-10T19:05:08.382047+0000 mon.c (mon.2) 707 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[32381]: audit 2024-09-10T19:05:08.382700+0000 mon.a (mon.0) 3212 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.883024+0000 osd.1 (osd.1) 353 : cluster [DBG] 4.f scrub starts 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:06.885540+0000 osd.1 (osd.1) 354 : cluster [DBG] 4.f scrub ok 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.217461+0000 osd.5 (osd.5) 362 : cluster [DBG] 6.a scrub starts 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.220808+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.a scrub ok 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.661415+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.665269+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.737827+0000 osd.3 (osd.3) 366 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:09.241 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.738332+0000 osd.3 (osd.3) 367 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.903361+0000 osd.4 (osd.4) 360 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.910207+0000 osd.4 (osd.4) 361 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.985956+0000 osd.3 (osd.3) 368 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:07.989460+0000 osd.3 (osd.3) 369 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: audit 2024-09-10T19:05:08.251667+0000 mgr.y (mgr.24526) 626 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:08.273426+0000 mgr.y (mgr.24526) 627 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:08.304561+0000 mon.a (mon.0) 3209 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: audit 2024-09-10T19:05:08.316094+0000 mon.a (mon.0) 3210 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-114", "mode": "writeback"}]': finished 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: cluster 2024-09-10T19:05:08.316518+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: audit 2024-09-10T19:05:08.382047+0000 mon.c (mon.2) 707 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:09.242 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:08 smithi083 bash[18629]: audit 2024-09-10T19:05:08.382700+0000 mon.a (mon.0) 3212 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:09.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:09.390 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:09.477 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:05:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:05:09.477 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T19:05:09.477 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:05:09.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:09.525 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:09.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:09.667 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:09.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:09.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:09.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:09.943 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:09.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:10.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:10.082 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: cluster 2024-09-10T19:05:08.918243+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: cluster 2024-09-10T19:05:08.921697+0000 osd.4 (osd.4) 363 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: audit 2024-09-10T19:05:09.289518+0000 mon.a (mon.0) 3213 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: cluster 2024-09-10T19:05:09.289847+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: audit 2024-09-10T19:05:09.292226+0000 mon.c (mon.2) 708 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.110 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[18629]: audit 2024-09-10T19:05:09.292834+0000 mon.a (mon.0) 3215 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: cluster 2024-09-10T19:05:08.918243+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: cluster 2024-09-10T19:05:08.921697+0000 osd.4 (osd.4) 363 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: audit 2024-09-10T19:05:09.289518+0000 mon.a (mon.0) 3213 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: cluster 2024-09-10T19:05:09.289847+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: audit 2024-09-10T19:05:09.292226+0000 mon.c (mon.2) 708 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.111 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:09 smithi083 bash[32381]: audit 2024-09-10T19:05:09.292834+0000 mon.a (mon.0) 3215 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: cluster 2024-09-10T19:05:08.918243+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: cluster 2024-09-10T19:05:08.921697+0000 osd.4 (osd.4) 363 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: audit 2024-09-10T19:05:09.289518+0000 mon.a (mon.0) 3213 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: cluster 2024-09-10T19:05:09.289847+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: audit 2024-09-10T19:05:09.292226+0000 mon.c (mon.2) 708 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:09 smithi165 bash[22591]: audit 2024-09-10T19:05:09.292834+0000 mon.a (mon.0) 3215 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]: dispatch 2024-09-10T19:05:10.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:10.223 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:10.326 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:10.356 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:10.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:10.491 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:10.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.631 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:10.734 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:10.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:09.493303+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:09.497830+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:09.937415+0000 osd.3 (osd.3) 370 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:09.941688+0000 osd.3 (osd.3) 371 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.274168+0000 mgr.y (mgr.24526) 628 : cluster [DBG] pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.283834+0000 mon.a (mon.0) 3216 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: audit 2024-09-10T19:05:10.291839+0000 mon.a (mon.0) 3217 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:10.881 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.292033+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:09.493303+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:09.497830+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:09.937415+0000 osd.3 (osd.3) 370 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:09.941688+0000 osd.3 (osd.3) 371 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.274168+0000 mgr.y (mgr.24526) 628 : cluster [DBG] pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.283834+0000 mon.a (mon.0) 3216 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: audit 2024-09-10T19:05:10.291839+0000 mon.a (mon.0) 3217 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:10.882 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:10 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.292033+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:05:10.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:10.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:11.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:09.493303+0000 osd.6 (osd.6) 316 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:09.497830+0000 osd.6 (osd.6) 317 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:09.937415+0000 osd.3 (osd.3) 370 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:09.941688+0000 osd.3 (osd.3) 371 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.274168+0000 mgr.y (mgr.24526) 628 : cluster [DBG] pgmap v1027: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.283834+0000 mon.a (mon.0) 3216 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: audit 2024-09-10T19:05:10.291839+0000 mon.a (mon.0) 3217 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-114"}]': finished 2024-09-10T19:05:11.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:10 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.292033+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:05:11.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:11.173 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:11.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:11.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:11.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:11.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:05:11.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.548+0000 7fab6d15c640 1 -- 172.21.15.83:0/1755218445 >> v1:172.21.15.83:6790/0 conn(0x7fab6810fe70 legacy=0x7fab68112250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.548+0000 7fab6d15c640 1 -- 172.21.15.83:0/1755218445 shutdown_connections 2024-09-10T19:05:11.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.548+0000 7fab6d15c640 1 -- 172.21.15.83:0/1755218445 >> 172.21.15.83:0/1755218445 conn(0x7fab6806b990 msgr2=0x7fab6806bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:11.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- 172.21.15.83:0/1755218445 shutdown_connections 2024-09-10T19:05:11.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- 172.21.15.83:0/1755218445 wait complete. 2024-09-10T19:05:11.555 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 librados: starting msgr at 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 librados: starting objecter 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 Processor -- start 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- start start 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 librados: setting wanted keys 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 librados: calling monclient init 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab6811cd20 con 0x7fab6810fe70 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab68127e40 con 0x7fab68113970 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab68129020 con 0x7fab6806de90 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fab6810fe70 0x7fab68121fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53444/0 (socket says 172.21.15.83:53444) 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6659c640 1 -- 172.21.15.83:0/4148936281 learned_addr learned my addr 172.21.15.83:0/4148936281 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:11.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab66d9d640 1 --1- 172.21.15.83:0/4148936281 >> v1:172.21.15.83:6790/0 conn(0x7fab6806de90 0x7fab6811c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47650/0 (socket says 172.21.15.83:47650) 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter ms_handle_connect 0x7fab68113970 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter ms_handle_connect 0x7fab6806de90 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter ms_handle_connect 0x7fab6810fe70 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932407947 0 0) 0x7fab68129020 con 0x7fab6806de90 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab50003720 con 0x7fab6806de90 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412768942 0 0) 0x7fab68127e40 con 0x7fab68113970 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab68129020 con 0x7fab68113970 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234142878 0 0) 0x7fab50003720 con 0x7fab6806de90 2024-09-10T19:05:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab68127e40 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fab60001cc0 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 30691517 0 0) 0x7fab68127e40 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 >> v1:172.21.15.165:6789/0 conn(0x7fab68113970 legacy=0x7fab68125710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 >> v1:172.21.15.83:6789/0 conn(0x7fab6810fe70 legacy=0x7fab68121fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6812a240 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab68129210 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 10 client.49703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab681296a0 con 0x7fab6806de90 2024-09-10T19:05:11.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab6d15c640 1 librados: init done 2024-09-10T19:05:11.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.552+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fab60002410 con 0x7fab6806de90 2024-09-10T19:05:11.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.556+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fab60002150 con 0x7fab6806de90 2024-09-10T19:05:11.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.556+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab54005010 con 0x7fab6806de90 2024-09-10T19:05:11.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.556+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fab60002700 con 0x7fab6806de90 2024-09-10T19:05:11.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(668..668 src has 569..668) v4 ==== 8038+0+0 (unknown 2101224108 0 0) 0x7fab6008a540 con 0x7fab6806de90 2024-09-10T19:05:11.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 10 client.49703.objecter ms_dispatch 0x7fab68000b90 osd_map(668..668 src has 569..668) v4 2024-09-10T19:05:11.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 3 client.49703.objecter handle_osd_map got epochs [668,668] > 0 2024-09-10T19:05:11.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 3 client.49703.objecter handle_osd_map decoding full epoch 668 2024-09-10T19:05:11.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 20 client.49703.objecter dump_active .. 0 homeless 2024-09-10T19:05:11.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 10 client.49703.objecter ms_handle_connect 0x7fab50071450 2024-09-10T19:05:11.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.560+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fab60056ae0 con 0x7fab6806de90 2024-09-10T19:05:11.573 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:11.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/3088637124 >> v1:172.21.15.83:6790/0 conn(0x7ff4a0111700 legacy=0x7ff4a0113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/3088637124 shutdown_connections 2024-09-10T19:05:11.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/3088637124 >> 172.21.15.83:0/3088637124 conn(0x7ff4a006b990 msgr2=0x7ff4a006bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:11.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/3088637124 shutdown_connections 2024-09-10T19:05:11.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/3088637124 wait complete. 2024-09-10T19:05:11.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 librados: starting msgr at 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 librados: starting objecter 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 Processor -- start 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- start start 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 librados: setting wanted keys 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 librados: calling monclient init 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4a010d7b0 con 0x7ff4a010da80 2024-09-10T19:05:11.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4a0125d10 con 0x7ff4a0111700 2024-09-10T19:05:11.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff49effd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff4a010da80 0x7ff4a011feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53466/0 (socket says 172.21.15.83:53466) 2024-09-10T19:05:11.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff49effd640 1 -- 172.21.15.83:0/287495755 learned_addr learned my addr 172.21.15.83:0/287495755 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:11.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.640+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4a0126eb0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter ms_handle_connect 0x7ff4a0104ba0 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter ms_handle_connect 0x7ff4a010da80 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter ms_handle_connect 0x7ff4a0111700 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944372635 0 0) 0x7ff4a0126eb0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff488003720 con 0x7ff4a0104ba0 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915801092 0 0) 0x7ff4a0125d10 con 0x7ff4a0111700 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a0126eb0 con 0x7ff4a0111700 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1379966768 0 0) 0x7ff488003720 con 0x7ff4a0104ba0 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4a0125d10 con 0x7ff4a0104ba0 2024-09-10T19:05:11.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff4940017f0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1554630183 0 0) 0x7ff4a0125d10 con 0x7ff4a0104ba0 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 >> v1:172.21.15.165:6789/0 conn(0x7ff4a0111700 legacy=0x7ff4a01235e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 >> v1:172.21.15.83:6789/0 conn(0x7ff4a010da80 legacy=0x7ff4a011feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0128090 con 0x7ff4a0104ba0 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a01270a0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff4a5da2640 10 client.49709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a01274f0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff4a5da2640 1 librados: init done 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff494001f40 con 0x7ff4a0104ba0 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a006f350 con 0x7ff4a0104ba0 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff4940031c0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.644+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff4940033e0 con 0x7ff4a0104ba0 2024-09-10T19:05:11.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(668..668 src has 569..668) v4 ==== 8038+0+0 (unknown 2101224108 0 0) 0x7ff494087b70 con 0x7ff4a0104ba0 2024-09-10T19:05:11.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 10 client.49709.objecter ms_dispatch 0x7ff4a0000b90 osd_map(668..668 src has 569..668) v4 2024-09-10T19:05:11.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 3 client.49709.objecter handle_osd_map got epochs [668,668] > 0 2024-09-10T19:05:11.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 3 client.49709.objecter handle_osd_map decoding full epoch 668 2024-09-10T19:05:11.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 20 client.49709.objecter dump_active .. 0 homeless 2024-09-10T19:05:11.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.648+0000 7ff487fff640 10 client.49709.objecter ms_handle_connect 0x7ff4880711b0 2024-09-10T19:05:11.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.656+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff494054c20 con 0x7ff4a0104ba0 2024-09-10T19:05:11.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:11.703 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:11.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.768+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fab540052c0 con 0x7fab6806de90 2024-09-10T19:05:11.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:11.834 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:11.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:11.860+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff4a004bf60 con 0x7ff4a0104ba0 2024-09-10T19:05:11.929 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:11.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.973 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:11.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:11.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:12.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:12.102 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.477809+0000 osd.0 (osd.0) 339 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.481672+0000 osd.0 (osd.0) 340 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.510579+0000 osd.6 (osd.6) 318 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.513958+0000 osd.6 (osd.6) 319 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.917524+0000 osd.4 (osd.4) 364 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.920877+0000 osd.4 (osd.4) 365 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.986524+0000 osd.3 (osd.3) 372 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:10.990109+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: cluster 2024-09-10T19:05:11.300118+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: audit 2024-09-10T19:05:11.771647+0000 mon.c (mon.2) 709 : audit [INF] from='client.? v1:172.21.15.83:0/4148936281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.103 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[18629]: audit 2024-09-10T19:05:11.783144+0000 mon.a (mon.0) 3220 : audit [INF] from='client.49703 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.477809+0000 osd.0 (osd.0) 339 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.481672+0000 osd.0 (osd.0) 340 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.510579+0000 osd.6 (osd.6) 318 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.513958+0000 osd.6 (osd.6) 319 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.917524+0000 osd.4 (osd.4) 364 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.920877+0000 osd.4 (osd.4) 365 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.986524+0000 osd.3 (osd.3) 372 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:10.990109+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: cluster 2024-09-10T19:05:11.300118+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: audit 2024-09-10T19:05:11.771647+0000 mon.c (mon.2) 709 : audit [INF] from='client.? v1:172.21.15.83:0/4148936281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:11 smithi083 bash[32381]: audit 2024-09-10T19:05:11.783144+0000 mon.a (mon.0) 3220 : audit [INF] from='client.49703 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.107 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.477809+0000 osd.0 (osd.0) 339 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.481672+0000 osd.0 (osd.0) 340 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.510579+0000 osd.6 (osd.6) 318 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.513958+0000 osd.6 (osd.6) 319 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.917524+0000 osd.4 (osd.4) 364 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:05:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.920877+0000 osd.4 (osd.4) 365 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:05:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.986524+0000 osd.3 (osd.3) 372 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:05:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:10.990109+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:05:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: cluster 2024-09-10T19:05:11.300118+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:05:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: audit 2024-09-10T19:05:11.771647+0000 mon.c (mon.2) 709 : audit [INF] from='client.? v1:172.21.15.83:0/4148936281' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:11 smithi165 bash[22591]: audit 2024-09-10T19:05:11.783144+0000 mon.a (mon.0) 3220 : audit [INF] from='client.49703 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:05:12.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:12.243 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:12.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.300+0000 7fab4f7fe640 1 -- 172.21.15.83:0/4148936281 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v669) v1 ==== 113+0+0 (unknown 1810890812 0 0) 0x7fab60056480 con 0x7fab6806de90 2024-09-10T19:05:12.306 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T19:05:12.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.300+0000 7ff487fff640 1 -- 172.21.15.83:0/287495755 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v669) v1 ==== 101+0+0 (unknown 1031052377 0 0) 0x7ff494001ae0 con 0x7ff4a0104ba0 2024-09-10T19:05:12.309 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:05:12.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 10 librados: watch_flush enter 2024-09-10T19:05:12.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 10 librados: watch_flush exit 2024-09-10T19:05:12.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 20 client.49703.objecter shutdown clearing up homeless session... 2024-09-10T19:05:12.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 10 client.49703.objecter successfully canceled tick 2024-09-10T19:05:12.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 >> v1:172.21.15.83:6800/3071793735 conn(0x7fab50071450 legacy=0x7fab50073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:12.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 >> v1:172.21.15.83:6790/0 conn(0x7fab6806de90 legacy=0x7fab6811c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:12.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 shutdown_connections 2024-09-10T19:05:12.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 >> 172.21.15.83:0/4148936281 conn(0x7fab6806b990 msgr2=0x7fab68064800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:12.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.308+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 shutdown_connections 2024-09-10T19:05:12.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7fab6d15c640 1 -- 172.21.15.83:0/4148936281 wait complete. 2024-09-10T19:05:12.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7fab6d15c640 1 librados: shutdown 2024-09-10T19:05:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 10 librados: watch_flush enter 2024-09-10T19:05:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 10 librados: watch_flush exit 2024-09-10T19:05:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 20 client.49709.objecter shutdown clearing up homeless session... 2024-09-10T19:05:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 10 client.49709.objecter successfully canceled tick 2024-09-10T19:05:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff4880711b0 legacy=0x7ff488073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 >> v1:172.21.15.83:6790/0 conn(0x7ff4a0104ba0 legacy=0x7ff4a010ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 shutdown_connections 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 >> 172.21.15.83:0/287495755 conn(0x7ff4a006b990 msgr2=0x7ff4a006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 shutdown_connections 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 -- 172.21.15.83:0/287495755 wait complete. 2024-09-10T19:05:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:12.312+0000 7ff4a5da2640 1 librados: shutdown 2024-09-10T19:05:12.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:12.373 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:12.477 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:12.513 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:12.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:12.648 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:12.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:12.781 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:12.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:12.916 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:12.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:13.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:11.478235+0000 osd.6 (osd.6) 320 : cluster [DBG] 12.f scrub starts 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:11.862884+0000 mon.c (mon.2) 710 : audit [INF] from='client.? v1:172.21.15.83:0/287495755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:11.863292+0000 mon.a (mon.0) 3221 : audit [INF] from='client.49709 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.069139+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.072921+0000 osd.2 (osd.2) 278 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.274988+0000 mgr.y (mgr.24526) 629 : cluster [DBG] pgmap v1030: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:13.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.290346+0000 mon.a (mon.0) 3222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:05:13.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:12.304722+0000 mon.a (mon.0) 3223 : audit [INF] from='client.49703 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:05:13.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:12.304916+0000 mon.a (mon.0) 3224 : audit [INF] from='client.49709 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:13.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.305370+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:05:13.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:12.308361+0000 mon.c (mon.2) 711 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.035 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:12 smithi165 bash[22591]: audit 2024-09-10T19:05:12.311215+0000 mon.a (mon.0) 3226 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:12] "GET /metrics HTTP/1.1" 200 228944 "" "Prometheus/2.43.0" 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:11.478235+0000 osd.6 (osd.6) 320 : cluster [DBG] 12.f scrub starts 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:11.862884+0000 mon.c (mon.2) 710 : audit [INF] from='client.? v1:172.21.15.83:0/287495755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:11.863292+0000 mon.a (mon.0) 3221 : audit [INF] from='client.49709 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.069139+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.072921+0000 osd.2 (osd.2) 278 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.274988+0000 mgr.y (mgr.24526) 629 : cluster [DBG] pgmap v1030: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.290346+0000 mon.a (mon.0) 3222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:12.304722+0000 mon.a (mon.0) 3223 : audit [INF] from='client.49703 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:12.304916+0000 mon.a (mon.0) 3224 : audit [INF] from='client.49709 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.305370+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:12.308361+0000 mon.c (mon.2) 711 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[18629]: audit 2024-09-10T19:05:12.311215+0000 mon.a (mon.0) 3226 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:11.478235+0000 osd.6 (osd.6) 320 : cluster [DBG] 12.f scrub starts 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:11.862884+0000 mon.c (mon.2) 710 : audit [INF] from='client.? v1:172.21.15.83:0/287495755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:11.863292+0000 mon.a (mon.0) 3221 : audit [INF] from='client.49709 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.069139+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.072921+0000 osd.2 (osd.2) 278 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.274988+0000 mgr.y (mgr.24526) 629 : cluster [DBG] pgmap v1030: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.290346+0000 mon.a (mon.0) 3222 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:12.304722+0000 mon.a (mon.0) 3223 : audit [INF] from='client.49703 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:12.304916+0000 mon.a (mon.0) 3224 : audit [INF] from='client.49709 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.305370+0000 mon.a (mon.0) 3225 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:05:13.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:12.308361+0000 mon.c (mon.2) 711 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:12 smithi083 bash[32381]: audit 2024-09-10T19:05:12.311215+0000 mon.a (mon.0) 3226 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:13.053 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:13.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:13.192 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:13.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:13.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.470 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:13.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.606 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:13.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:13.744 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:13.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:13.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:13.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:13.994 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.496907+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.500497+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.511750+0000 osd.0 (osd.0) 341 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.516911+0000 osd.0 (osd.0) 342 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.922419+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:12.925810+0000 osd.1 (osd.1) 356 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: audit 2024-09-10T19:05:13.308372+0000 mon.a (mon.0) 3227 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[32381]: cluster 2024-09-10T19:05:13.308582+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.496907+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:05:13.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.500497+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.511750+0000 osd.0 (osd.0) 341 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.516911+0000 osd.0 (osd.0) 342 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.922419+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:12.925810+0000 osd.1 (osd.1) 356 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: audit 2024-09-10T19:05:13.308372+0000 mon.a (mon.0) 3227 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:13.996 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:13 smithi083 bash[18629]: cluster 2024-09-10T19:05:13.308582+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:05:14.014 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:14.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:14.152 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:14.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.496907+0000 osd.6 (osd.6) 321 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.500497+0000 osd.6 (osd.6) 322 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.511750+0000 osd.0 (osd.0) 341 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.516911+0000 osd.0 (osd.0) 342 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.922419+0000 osd.1 (osd.1) 355 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:12.925810+0000 osd.1 (osd.1) 356 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: audit 2024-09-10T19:05:13.308372+0000 mon.a (mon.0) 3227 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:14.269 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:13 smithi165 bash[22591]: cluster 2024-09-10T19:05:13.308582+0000 mon.a (mon.0) 3228 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:05:14.282 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:14.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:14.422 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:14.497 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T19:05:14.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:14.563 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:14.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:14.691 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:14.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:14.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:14.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:14.966 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:14.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:15.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:15.083 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: cluster 2024-09-10T19:05:13.543080+0000 osd.6 (osd.6) 323 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:05:15.083 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: cluster 2024-09-10T19:05:13.547242+0000 osd.6 (osd.6) 324 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.275817+0000 mgr.y (mgr.24526) 630 : cluster [DBG] pgmap v1033: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.305343+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: audit 2024-09-10T19:05:14.346114+0000 mon.c (mon.2) 712 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: audit 2024-09-10T19:05:14.346752+0000 mon.a (mon.0) 3230 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: audit 2024-09-10T19:05:14.579394+0000 mon.a (mon.0) 3231 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:15.084 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:14 smithi165 bash[22591]: audit 2024-09-10T19:05:14.581491+0000 mon.c (mon.2) 713 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:15.101 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:15.205 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:15.218 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: cluster 2024-09-10T19:05:13.543080+0000 osd.6 (osd.6) 323 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: cluster 2024-09-10T19:05:13.547242+0000 osd.6 (osd.6) 324 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.275817+0000 mgr.y (mgr.24526) 630 : cluster [DBG] pgmap v1033: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.305343+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: audit 2024-09-10T19:05:14.346114+0000 mon.c (mon.2) 712 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: audit 2024-09-10T19:05:14.346752+0000 mon.a (mon.0) 3230 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: audit 2024-09-10T19:05:14.579394+0000 mon.a (mon.0) 3231 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[18629]: audit 2024-09-10T19:05:14.581491+0000 mon.c (mon.2) 713 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: cluster 2024-09-10T19:05:13.543080+0000 osd.6 (osd.6) 323 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: cluster 2024-09-10T19:05:13.547242+0000 osd.6 (osd.6) 324 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:05:15.219 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.275817+0000 mgr.y (mgr.24526) 630 : cluster [DBG] pgmap v1033: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:15.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.305343+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:05:15.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: audit 2024-09-10T19:05:14.346114+0000 mon.c (mon.2) 712 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: audit 2024-09-10T19:05:14.346752+0000 mon.a (mon.0) 3230 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:15.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: audit 2024-09-10T19:05:14.579394+0000 mon.a (mon.0) 3231 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:15.220 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:14 smithi083 bash[32381]: audit 2024-09-10T19:05:14.581491+0000 mon.c (mon.2) 713 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:15.239 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:15.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:15.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:15.479 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:15.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:15.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:15.635 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:15.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:15.768 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:15.871 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:15.899 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:15.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:16.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.039 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.577641+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.580377+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.883660+0000 osd.1 (osd.1) 357 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.886288+0000 osd.1 (osd.1) 358 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.975561+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:14.978916+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: audit 2024-09-10T19:05:15.309266+0000 mon.a (mon.0) 3232 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.309357+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: audit 2024-09-10T19:05:15.310899+0000 mon.c (mon.2) 714 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: audit 2024-09-10T19:05:15.312381+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.844038+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.577641+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.580377+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.883660+0000 osd.1 (osd.1) 357 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.886288+0000 osd.1 (osd.1) 358 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.975561+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:14.978916+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: audit 2024-09-10T19:05:15.309266+0000 mon.a (mon.0) 3232 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.309357+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:05:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: audit 2024-09-10T19:05:15.310899+0000 mon.c (mon.2) 714 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: audit 2024-09-10T19:05:15.312381+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:15 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.844038+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:05:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:16.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.577641+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.580377+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.883660+0000 osd.1 (osd.1) 357 : cluster [DBG] 12.18 deep-scrub starts 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.886288+0000 osd.1 (osd.1) 358 : cluster [DBG] 12.18 deep-scrub ok 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.975561+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:14.978916+0000 osd.3 (osd.3) 375 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: audit 2024-09-10T19:05:15.309266+0000 mon.a (mon.0) 3232 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.309357+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: audit 2024-09-10T19:05:15.310899+0000 mon.c (mon.2) 714 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: audit 2024-09-10T19:05:15.312381+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:16.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:15 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.844038+0000 osd.1 (osd.1) 359 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:05:16.176 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:16.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.315 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:16.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:16.455 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:16.498 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:05:16.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.594 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:16.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:16.733 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:16.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:16.872 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:16.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:16.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.008 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.157623+0000 osd.5 (osd.5) 364 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:05:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.161155+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:05:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.461395+0000 osd.0 (osd.0) 343 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.465071+0000 osd.0 (osd.0) 344 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.597671+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.605056+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:15.852272+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.277654+0000 mgr.y (mgr.24526) 631 : cluster [DBG] pgmap v1036: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: audit 2024-09-10T19:05:16.311424+0000 mon.a (mon.0) 3235 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.311590+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: audit 2024-09-10T19:05:16.313786+0000 mon.c (mon.2) 715 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:16 smithi165 bash[22591]: audit 2024-09-10T19:05:16.314386+0000 mon.a (mon.0) 3237 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:17.145 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:17.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.157623+0000 osd.5 (osd.5) 364 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:05:17.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.161155+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:05:17.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.461395+0000 osd.0 (osd.0) 343 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:05:17.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.465071+0000 osd.0 (osd.0) 344 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:05:17.185 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.597671+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.605056+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:15.852272+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.277654+0000 mgr.y (mgr.24526) 631 : cluster [DBG] pgmap v1036: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: audit 2024-09-10T19:05:16.311424+0000 mon.a (mon.0) 3235 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.311590+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: audit 2024-09-10T19:05:16.313786+0000 mon.c (mon.2) 715 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[32381]: audit 2024-09-10T19:05:16.314386+0000 mon.a (mon.0) 3237 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.157623+0000 osd.5 (osd.5) 364 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:05:17.186 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.161155+0000 osd.5 (osd.5) 365 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.461395+0000 osd.0 (osd.0) 343 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.465071+0000 osd.0 (osd.0) 344 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.597671+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.605056+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:15.852272+0000 osd.1 (osd.1) 360 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.277654+0000 mgr.y (mgr.24526) 631 : cluster [DBG] pgmap v1036: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: audit 2024-09-10T19:05:16.311424+0000 mon.a (mon.0) 3235 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.311590+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: audit 2024-09-10T19:05:16.313786+0000 mon.c (mon.2) 715 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.187 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:16 smithi083 bash[18629]: audit 2024-09-10T19:05:16.314386+0000 mon.a (mon.0) 3237 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]: dispatch 2024-09-10T19:05:17.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:17.284 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2475880457 >> v1:172.21.15.83:6789/0 conn(0x7f2ae4108f70 legacy=0x7f2ae410b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2475880457 shutdown_connections 2024-09-10T19:05:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2475880457 >> 172.21.15.83:0/2475880457 conn(0x7f2ae4074a80 msgr2=0x7f2ae40f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2475880457 shutdown_connections 2024-09-10T19:05:17.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2475880457 wait complete. 2024-09-10T19:05:17.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 librados: starting msgr at 2024-09-10T19:05:17.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 librados: starting objecter 2024-09-10T19:05:17.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 Processor -- start 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- start start 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 librados: setting wanted keys 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 librados: calling monclient init 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ae411e0f0 con 0x7f2ae4105320 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ae411f290 con 0x7f2ae4108f70 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2aeb8fd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ae4120490 con 0x7f2ae4102620 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ae8e98640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2ae4105320 0x7f2ae40fc350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53290/0 (socket says 172.21.15.83:53290) 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ae9699640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f2ae4102620 0x7f2ae40fbc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50466/0 (socket says 172.21.15.83:50466) 2024-09-10T19:05:17.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ae8e98640 1 -- 172.21.15.83:0/2237818844 learned_addr learned my addr 172.21.15.83:0/2237818844 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:17.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f2ae4102620 2024-09-10T19:05:17.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f2ae4105320 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f2ae4108f70 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875080029 0 0) 0x7f2ae411e0f0 con 0x7f2ae4105320 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad00039c0 con 0x7f2ae4105320 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885034398 0 0) 0x7f2ae4120490 con 0x7f2ae4102620 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae411e0f0 con 0x7f2ae4102620 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879612540 0 0) 0x7f2ae411f290 con 0x7f2ae4108f70 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae4120490 con 0x7f2ae4108f70 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034395643 0 0) 0x7f2ad00039c0 con 0x7f2ae4105320 2024-09-10T19:05:17.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae411f290 con 0x7f2ae4105320 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ae00019f0 con 0x7f2ae4105320 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230859151 0 0) 0x7f2ae411e0f0 con 0x7f2ae4102620 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad00039c0 con 0x7f2ae4102620 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ad4001840 con 0x7f2ae4102620 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826287814 0 0) 0x7f2ae4120490 con 0x7f2ae4108f70 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae411e0f0 con 0x7f2ae4108f70 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2adc002fb0 con 0x7f2ae4108f70 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712618469 0 0) 0x7f2ae411f290 con 0x7f2ae4105320 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.364+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6790/0 conn(0x7f2ae4102620 legacy=0x7f2ae40fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.165:6789/0 conn(0x7f2ae4108f70 legacy=0x7f2ae411c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4121690 con 0x7f2ae4105320 2024-09-10T19:05:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae411e2e0 con 0x7f2ae4105320 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2aeb8fd640 10 client.25629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae411e730 con 0x7f2ae4105320 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2aeb8fd640 1 librados: init done 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2ae00015a0 con 0x7f2ae4105320 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ae00059a0 con 0x7f2ae4105320 2024-09-10T19:05:17.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2ac8004ff0 con 0x7f2ae4105320 2024-09-10T19:05:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2ae0001f10 con 0x7f2ae4105320 2024-09-10T19:05:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(674..674 src has 569..674) v4 ==== 8419+0+0 (unknown 2530455051 0 0) 0x7f2ae008a580 con 0x7f2ae4105320 2024-09-10T19:05:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 osd_map(674..674 src has 569..674) v4 2024-09-10T19:05:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 3 client.25629.objecter handle_osd_map got epochs [674,674] > 0 2024-09-10T19:05:17.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 3 client.25629.objecter handle_osd_map decoding full epoch 674 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 20 client.25629.objecter dump_active .. 0 homeless 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v674) v1 ==== 54+0+16 (unknown 3339583437 0 2420000192) 0x7f2ae008c990 con 0x7f2ae4105320 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.368+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ad00717a0 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 1 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 1 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 0 1 2024-09-10T19:05:17.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 4 2024-09-10T19:05:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 1 2024-09-10T19:05:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac8004a60 con 0x7f2ac8004610 2024-09-10T19:05:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8004610 2024-09-10T19:05:17.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac8004a60 con 0x7f2ac8004610 2024-09-10T19:05:17.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:05:17.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 1 = system:0 2024-09-10T19:05:17.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 4 2024-09-10T19:05:17.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.372+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 0 1 2024-09-10T19:05:17.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 2 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 2 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8005290 osd=0 3 2024-09-10T19:05:17.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 0 2 2024-09-10T19:05:17.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 4 2024-09-10T19:05:17.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 2 2024-09-10T19:05:17.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.388+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac8004e40 con 0x7f2ac8004610 2024-09-10T19:05:17.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac8004e40 con 0x7f2ac8004610 2024-09-10T19:05:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 4 2024-09-10T19:05:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 0 2 2024-09-10T19:05:17.395 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 1 3 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 4 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 3 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac80022f0 con 0x7f2ac8004110 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8004110 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac80022f0 con 0x7f2ac8004110 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 3 = system:0 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 4 2024-09-10T19:05:17.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.392+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 1 3 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 4 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 4 2024-09-10T19:05:17.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8003fa0 osd=1 3 2024-09-10T19:05:17.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 1 4 2024-09-10T19:05:17.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 4 2024-09-10T19:05:17.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 4 2024-09-10T19:05:17.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac80018f0 con 0x7f2ac8004110 2024-09-10T19:05:17.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac80018f0 con 0x7f2ac8004110 2024-09-10T19:05:17.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 4 2024-09-10T19:05:17.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 1 4 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 5 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 5 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 2 5 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 4 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 5 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.408+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac8003e30 con 0x7f2ac800d350 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac800d350 2024-09-10T19:05:17.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac8003e30 con 0x7f2ac800d350 2024-09-10T19:05:17.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:05:17.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 5 = system:0 2024-09-10T19:05:17.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 4 2024-09-10T19:05:17.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.412+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 2 5 2024-09-10T19:05:17.421 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 6 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 6 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac80035e0 osd=2 3 2024-09-10T19:05:17.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 2 6 2024-09-10T19:05:17.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 4 2024-09-10T19:05:17.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 6 2024-09-10T19:05:17.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.424+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac80030d0 con 0x7f2ac800d350 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac80030d0 con 0x7f2ac800d350 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 4 2024-09-10T19:05:17.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 2 6 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 7 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 7 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 3 7 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 4 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 7 2024-09-10T19:05:17.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac800d100 con 0x7f2ac8010e50 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.428+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8010e50 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.432+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac800d100 con 0x7f2ac8010e50 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.432+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.432+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 7 = system:0 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.432+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 4 2024-09-10T19:05:17.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.432+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 3 7 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 8 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 8 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac800c990 osd=3 3 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 3 8 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 4 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 8 2024-09-10T19:05:17.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.452+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac8012130 con 0x7f2ac8010e50 2024-09-10T19:05:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac8012130 con 0x7f2ac8010e50 2024-09-10T19:05:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 4 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 3 8 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 9 2024-09-10T19:05:17.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 9 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 4 9 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 4 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 9 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac801a290 con 0x7f2ac8016790 2024-09-10T19:05:17.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8016790 2024-09-10T19:05:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac801a290 con 0x7f2ac8016790 2024-09-10T19:05:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:05:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 9 = system:0 2024-09-10T19:05:17.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 4 2024-09-10T19:05:17.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.456+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 4 9 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 10 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 10 2024-09-10T19:05:17.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8011bb0 osd=4 3 2024-09-10T19:05:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 4 10 2024-09-10T19:05:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 4 2024-09-10T19:05:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 10 2024-09-10T19:05:17.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.476+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac801a290 con 0x7f2ac8016790 2024-09-10T19:05:17.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac801a290 con 0x7f2ac8016790 2024-09-10T19:05:17.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 4 2024-09-10T19:05:17.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 4 10 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 11 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 11 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 5 11 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 4 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 11 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac801e610 con 0x7f2ac801a950 2024-09-10T19:05:17.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.480+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac801a950 2024-09-10T19:05:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.484+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac801e610 con 0x7f2ac801a950 2024-09-10T19:05:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.484+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:05:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.484+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 11 = system:0 2024-09-10T19:05:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.484+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 4 2024-09-10T19:05:17.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.484+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 5 11 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 12 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 12 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac801a290 osd=5 3 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 5 12 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 4 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 12 2024-09-10T19:05:17.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.496+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac801feb0 con 0x7f2ac801a950 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac801feb0 con 0x7f2ac801a950 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 4 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 5 12 2024-09-10T19:05:17.505 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 13 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 13 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 6 13 2024-09-10T19:05:17.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 4 2024-09-10T19:05:17.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 13 2024-09-10T19:05:17.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.500+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac8020a60 con 0x7f2ac8020530 2024-09-10T19:05:17.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8020530 2024-09-10T19:05:17.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac8020a60 con 0x7f2ac8020530 2024-09-10T19:05:17.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:05:17.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 13 = system:0 2024-09-10T19:05:17.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 4 2024-09-10T19:05:17.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.504+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 6 13 2024-09-10T19:05:17.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 14 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 14 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac801feb0 osd=6 3 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 6 14 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 4 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 14 2024-09-10T19:05:17.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.528+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac8020a60 con 0x7f2ac8020530 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac8020a60 con 0x7f2ac8020530 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 4 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 6 14 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:05:17.535 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 15 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 15 2024-09-10T19:05:17.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 7 15 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 4 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 15 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ac8026ee0 con 0x7f2ac8020c50 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 10 client.25629.objecter ms_handle_connect 0x7f2ac8020c50 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f2ac8026ee0 con 0x7f2ac8020c50 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 15 = system:0 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 4 2024-09-10T19:05:17.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.532+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 7 15 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 10 client.25629.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign -1 16 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 20 client.25629.objecter _get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_remove -1 16 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 20 client.25629.objecter get_session s=0x7f2ac8020a60 osd=7 3 2024-09-10T19:05:17.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 15 client.25629.objecter _session_command_op_assign 7 16 2024-09-10T19:05:17.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 4 2024-09-10T19:05:17.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 10 client.25629.objecter _send_command 16 2024-09-10T19:05:17.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.552+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2ac8028780 con 0x7f2ac8020c50 2024-09-10T19:05:17.559 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ad9ffb640 1 -- 172.21.15.83:0/2237818844 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f2ac8028780 con 0x7f2ac8020c50 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ad9ffb640 10 client.25629.objecter ms_dispatch 0x7f2ae4000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ad9ffb640 10 client.25629.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ad9ffb640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 4 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ad9ffb640 15 client.25629.objecter _session_command_op_remove 7 16 2024-09-10T19:05:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.562 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:05:17.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 librados: watch_flush enter 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 librados: watch_flush exit 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.0 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6801/2535527483 conn(0x7f2ac8004610 legacy=0x7f2ac8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8005290 osd=0 1 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.1 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6805/2309624637 conn(0x7f2ac8004110 legacy=0x7f2ac800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8003fa0 osd=1 1 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.2 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6809/386094220 conn(0x7f2ac800d350 legacy=0x7f2ac800f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac80035e0 osd=2 1 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.3 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2ae9e9a640 1 -- 172.21.15.83:0/2237818844 reap_dead start 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6813/1279688021 conn(0x7f2ac8010e50 legacy=0x7f2ac8011260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac800c990 osd=3 1 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.4 2024-09-10T19:05:17.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.165:6800/3225515265 conn(0x7f2ac8016790 legacy=0x7f2ac8018b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8011bb0 osd=4 1 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.5 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.556+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.165:6804/571173777 conn(0x7f2ac801a950 legacy=0x7f2ac801cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801a290 osd=5 1 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.6 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.165:6808/1317922936 conn(0x7f2ac8020530 legacy=0x7f2ac801e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac801feb0 osd=6 1 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 10 client.25629.objecter close_session for osd.7 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.165:6812/1610151650 conn(0x7f2ac8020c50 legacy=0x7f2ac8025630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 20 client.25629.objecter put_session s=0x7f2ac8020a60 osd=7 1 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 20 client.25629.objecter shutdown clearing up homeless session... 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 10 client.25629.objecter successfully canceled tick 2024-09-10T19:05:17.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2ae9e9a640 1 -- 172.21.15.83:0/2237818844 reap_dead start 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2ad00717a0 legacy=0x7f2ad0073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> v1:172.21.15.83:6789/0 conn(0x7f2ae4105320 legacy=0x7f2ae40fc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 shutdown_connections 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 >> 172.21.15.83:0/2237818844 conn(0x7f2ae4074a80 msgr2=0x7f2ae40fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 shutdown_connections 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 -- 172.21.15.83:0/2237818844 wait complete. 2024-09-10T19:05:17.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:17.560+0000 7f2aeb8fd640 1 librados: shutdown 2024-09-10T19:05:17.664 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:17.692 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:17.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:17.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:17.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.500045+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.504221+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.575241+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.577880+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.991791+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:16.995399+0000 osd.3 (osd.3) 377 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.073774+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.074129+0000 osd.6 (osd.6) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.305236+0000 mon.a (mon.0) 3238 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: audit 2024-09-10T19:05:17.313758+0000 mon.a (mon.0) 3239 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]': finished 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.314215+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:05:17.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: audit 2024-09-10T19:05:17.338264+0000 mon.c (mon.2) 716 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:17.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: audit 2024-09-10T19:05:17.338499+0000 mgr.y (mgr.24526) 632 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:17.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: audit 2024-09-10T19:05:17.372972+0000 mon.a (mon.0) 3241 : audit [DBG] from='client.? v1:172.21.15.83:0/2237818844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:17.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.404597+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:05:17.952 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:17 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.407500+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:05:17.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:17.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.500045+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.504221+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.575241+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.577880+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.991791+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:16.995399+0000 osd.3 (osd.3) 377 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:05:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.073774+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.074129+0000 osd.6 (osd.6) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.305236+0000 mon.a (mon.0) 3238 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: audit 2024-09-10T19:05:17.313758+0000 mon.a (mon.0) 3239 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]': finished 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.314215+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: audit 2024-09-10T19:05:17.338264+0000 mon.c (mon.2) 716 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: audit 2024-09-10T19:05:17.338499+0000 mgr.y (mgr.24526) 632 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: audit 2024-09-10T19:05:17.372972+0000 mon.a (mon.0) 3241 : audit [DBG] from='client.? v1:172.21.15.83:0/2237818844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.404597+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.407500+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:05:18.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.500045+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.504221+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.575241+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.577880+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.991791+0000 osd.3 (osd.3) 376 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:16.995399+0000 osd.3 (osd.3) 377 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.073774+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.074129+0000 osd.6 (osd.6) 328 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.305236+0000 mon.a (mon.0) 3238 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: audit 2024-09-10T19:05:17.313758+0000 mon.a (mon.0) 3239 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-116", "mode": "writeback"}]': finished 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.314215+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: audit 2024-09-10T19:05:17.338264+0000 mon.c (mon.2) 716 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:18.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: audit 2024-09-10T19:05:17.338499+0000 mgr.y (mgr.24526) 632 : audit [DBG] from='mon.? v1:172.21.15.83:0/306346173' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.5"}]: dispatch 2024-09-10T19:05:18.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: audit 2024-09-10T19:05:17.372972+0000 mon.a (mon.0) 3241 : audit [DBG] from='client.? v1:172.21.15.83:0/2237818844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:18.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.404597+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:05:18.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:17 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.407500+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:05:18.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:18.110 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:18.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:18.247 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:18.350 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:18.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:18.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:18.523 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:18.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:18.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:18.658 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:18.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:18.829 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:18.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:18.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:18.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:18.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:19.035 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.209516+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.a scrub starts 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.213603+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.a scrub ok 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.964092+0000 osd.2 (osd.2) 279 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: cluster 2024-09-10T19:05:17.967993+0000 osd.2 (osd.2) 280 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: audit 2024-09-10T19:05:18.260825+0000 mgr.y (mgr.24526) 633 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.278512+0000 mgr.y (mgr.24526) 634 : cluster [DBG] pgmap v1039: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:19.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.209516+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.a scrub starts 2024-09-10T19:05:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.213603+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.a scrub ok 2024-09-10T19:05:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.964092+0000 osd.2 (osd.2) 279 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:05:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: cluster 2024-09-10T19:05:17.967993+0000 osd.2 (osd.2) 280 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:05:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: audit 2024-09-10T19:05:18.260825+0000 mgr.y (mgr.24526) 633 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:18 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.278512+0000 mgr.y (mgr.24526) 634 : cluster [DBG] pgmap v1039: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:19.047 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:19.067 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:19.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:19.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.209516+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.a scrub starts 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.213603+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.a scrub ok 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.964092+0000 osd.2 (osd.2) 279 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: cluster 2024-09-10T19:05:17.967993+0000 osd.2 (osd.2) 280 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: audit 2024-09-10T19:05:18.260825+0000 mgr.y (mgr.24526) 633 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:19.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:18 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.278512+0000 mgr.y (mgr.24526) 634 : cluster [DBG] pgmap v1039: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:05:19.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:19.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:19.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:19.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:19.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:19.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:19.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:19.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:19.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:19.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:19.906 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.771583+0000 osd.4 (osd.4) 366 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:05:19.907 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.775127+0000 osd.4 (osd.4) 367 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:05:19.907 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.960519+0000 osd.3 (osd.3) 378 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:05:19.907 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.964827+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:05:19.907 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.983624+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-10T19:05:19.907 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[18629]: cluster 2024-09-10T19:05:18.986891+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-10T19:05:19.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:20.025 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.771583+0000 osd.4 (osd.4) 366 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:05:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.775127+0000 osd.4 (osd.4) 367 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:05:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.960519+0000 osd.3 (osd.3) 378 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:05:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.964827+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:05:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.983624+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-10T19:05:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:19 smithi165 bash[22591]: cluster 2024-09-10T19:05:18.986891+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-10T19:05:20.129 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:20.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:20.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.771583+0000 osd.4 (osd.4) 366 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:05:20.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.775127+0000 osd.4 (osd.4) 367 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:05:20.188 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.960519+0000 osd.3 (osd.3) 378 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:05:20.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.964827+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:05:20.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.983624+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.c scrub starts 2024-09-10T19:05:20.189 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:19 smithi083 bash[32381]: cluster 2024-09-10T19:05:18.986891+0000 osd.2 (osd.2) 282 : cluster [DBG] 2.c scrub ok 2024-09-10T19:05:20.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:20.304 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:20.407 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:20.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:05:20.442 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:20.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:20.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:20.578 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:20.682 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:20.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:20.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:20.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:20.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:20.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.525493+0000 osd.6 (osd.6) 329 : cluster [DBG] 244.5 scrub starts 2024-09-10T19:05:20.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.528846+0000 osd.6 (osd.6) 330 : cluster [DBG] 244.5 scrub ok 2024-09-10T19:05:20.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.614840+0000 osd.7 (osd.7) 371 : cluster [DBG] 3.f scrub starts 2024-09-10T19:05:20.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.619501+0000 osd.7 (osd.7) 372 : cluster [DBG] 3.f scrub ok 2024-09-10T19:05:20.968 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.899038+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.902888+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.916297+0000 osd.3 (osd.3) 380 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:19.919216+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[18629]: cluster 2024-09-10T19:05:20.279304+0000 mgr.y (mgr.24526) 635 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.525493+0000 osd.6 (osd.6) 329 : cluster [DBG] 244.5 scrub starts 2024-09-10T19:05:20.969 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.528846+0000 osd.6 (osd.6) 330 : cluster [DBG] 244.5 scrub ok 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.614840+0000 osd.7 (osd.7) 371 : cluster [DBG] 3.f scrub starts 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.619501+0000 osd.7 (osd.7) 372 : cluster [DBG] 3.f scrub ok 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.899038+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.902888+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.916297+0000 osd.3 (osd.3) 380 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:19.919216+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:05:20.970 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:20 smithi083 bash[32381]: cluster 2024-09-10T19:05:20.279304+0000 mgr.y (mgr.24526) 635 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:05:20.987 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:20.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:21.076 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:21.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.525493+0000 osd.6 (osd.6) 329 : cluster [DBG] 244.5 scrub starts 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.528846+0000 osd.6 (osd.6) 330 : cluster [DBG] 244.5 scrub ok 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.614840+0000 osd.7 (osd.7) 371 : cluster [DBG] 3.f scrub starts 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.619501+0000 osd.7 (osd.7) 372 : cluster [DBG] 3.f scrub ok 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.899038+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.902888+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.916297+0000 osd.3 (osd.3) 380 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:19.919216+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:05:21.104 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:20 smithi165 bash[22591]: cluster 2024-09-10T19:05:20.279304+0000 mgr.y (mgr.24526) 635 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:05:21.124 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:21.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:21.262 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:21.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:21.398 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:21.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:21.537 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:21.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:21.654 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:21.672 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:21.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:21.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:21.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:21.949 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:21.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:22.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:22.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:20.125927+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:20.129387+0000 osd.5 (osd.5) 369 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:20.881173+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:20.886256+0000 osd.3 (osd.3) 383 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:21.346259+0000 osd.0 (osd.0) 347 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[18629]: cluster 2024-09-10T19:05:21.348963+0000 osd.0 (osd.0) 348 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:05:22.173 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:20.125927+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:20.129387+0000 osd.5 (osd.5) 369 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:20.881173+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:20.886256+0000 osd.3 (osd.3) 383 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:21.346259+0000 osd.0 (osd.0) 347 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:21 smithi083 bash[32381]: cluster 2024-09-10T19:05:21.348963+0000 osd.0 (osd.0) 348 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:05:22.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:20.125927+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:20.129387+0000 osd.5 (osd.5) 369 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:20.881173+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:20.886256+0000 osd.3 (osd.3) 383 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:21.346259+0000 osd.0 (osd.0) 347 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:05:22.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:21 smithi165 bash[22591]: cluster 2024-09-10T19:05:21.348963+0000 osd.0 (osd.0) 348 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:05:22.230 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:22.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:22.367 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:22.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:22.500 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:22.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:22.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:22.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:22.774 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:22.814 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:22] "GET /metrics HTTP/1.1" 200 229010 "" "Prometheus/2.43.0" 2024-09-10T19:05:22.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:22.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:22.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:23.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:23.040 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:23.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:23.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[18629]: cluster 2024-09-10T19:05:21.848338+0000 osd.3 (osd.3) 384 : cluster [DBG] 3.b scrub starts 2024-09-10T19:05:23.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[18629]: cluster 2024-09-10T19:05:21.852548+0000 osd.3 (osd.3) 385 : cluster [DBG] 3.b scrub ok 2024-09-10T19:05:23.152 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.281437+0000 mgr.y (mgr.24526) 636 : cluster [DBG] pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:05:23.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[32381]: cluster 2024-09-10T19:05:21.848338+0000 osd.3 (osd.3) 384 : cluster [DBG] 3.b scrub starts 2024-09-10T19:05:23.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[32381]: cluster 2024-09-10T19:05:21.852548+0000 osd.3 (osd.3) 385 : cluster [DBG] 3.b scrub ok 2024-09-10T19:05:23.153 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:22 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.281437+0000 mgr.y (mgr.24526) 636 : cluster [DBG] pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:05:23.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:23.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:23.288 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:22 smithi165 bash[22591]: cluster 2024-09-10T19:05:21.848338+0000 osd.3 (osd.3) 384 : cluster [DBG] 3.b scrub starts 2024-09-10T19:05:23.288 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:22 smithi165 bash[22591]: cluster 2024-09-10T19:05:21.852548+0000 osd.3 (osd.3) 385 : cluster [DBG] 3.b scrub ok 2024-09-10T19:05:23.288 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:22 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.281437+0000 mgr.y (mgr.24526) 636 : cluster [DBG] pgmap v1041: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-09-10T19:05:23.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:23.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:23.452 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:23.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:23.585 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:23.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:23.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:23.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:23.863 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:23.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:23.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:23.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.533469+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:05:23.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.537361+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:05:23.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.648871+0000 osd.7 (osd.7) 373 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:05:23.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.653034+0000 osd.7 (osd.7) 374 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:05:23.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.962984+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:05:23.979 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:23 smithi165 bash[22591]: cluster 2024-09-10T19:05:22.966296+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:05:23.999 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:24.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.533469+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:05:24.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.537361+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:05:24.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.648871+0000 osd.7 (osd.7) 373 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:05:24.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.653034+0000 osd.7 (osd.7) 374 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:05:24.094 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.962984+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[32381]: cluster 2024-09-10T19:05:22.966296+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.533469+0000 osd.6 (osd.6) 331 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.537361+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.648871+0000 osd.7 (osd.7) 373 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.653034+0000 osd.7 (osd.7) 374 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.962984+0000 osd.2 (osd.2) 283 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:05:24.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:23 smithi083 bash[18629]: cluster 2024-09-10T19:05:22.966296+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:05:24.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:24.125 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:24.228 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.260 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:24.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:24.396 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:24.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.534 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:24.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.664 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:24.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:24.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:24.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:24.944 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:24.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:25.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.082 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:25.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:24 smithi165 bash[22591]: cluster 2024-09-10T19:05:23.156196+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:05:25.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:24 smithi165 bash[22591]: cluster 2024-09-10T19:05:23.161171+0000 osd.5 (osd.5) 371 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:05:25.199 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:24 smithi165 bash[22591]: cluster 2024-09-10T19:05:24.282172+0000 mgr.y (mgr.24526) 637 : cluster [DBG] pgmap v1042: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-09-10T19:05:25.219 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[18629]: cluster 2024-09-10T19:05:23.156196+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[18629]: cluster 2024-09-10T19:05:23.161171+0000 osd.5 (osd.5) 371 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[18629]: cluster 2024-09-10T19:05:24.282172+0000 mgr.y (mgr.24526) 637 : cluster [DBG] pgmap v1042: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[32381]: cluster 2024-09-10T19:05:23.156196+0000 osd.5 (osd.5) 370 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[32381]: cluster 2024-09-10T19:05:23.161171+0000 osd.5 (osd.5) 371 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:05:25.271 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:24 smithi083 bash[32381]: cluster 2024-09-10T19:05:24.282172+0000 mgr.y (mgr.24526) 637 : cluster [DBG] pgmap v1042: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 1 op/s 2024-09-10T19:05:25.323 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:25.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:25.463 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:25.497 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:25.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:25.636 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:25.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:25.740 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:25.770 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:25.878 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.913 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:25.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:26.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:26.050 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[18629]: cluster 2024-09-10T19:05:24.942998+0000 osd.2 (osd.2) 285 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[18629]: cluster 2024-09-10T19:05:24.946809+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[18629]: cluster 2024-09-10T19:05:25.275692+0000 osd.0 (osd.0) 349 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[18629]: cluster 2024-09-10T19:05:25.279394+0000 osd.0 (osd.0) 350 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[32381]: cluster 2024-09-10T19:05:24.942998+0000 osd.2 (osd.2) 285 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[32381]: cluster 2024-09-10T19:05:24.946809+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:05:26.052 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[32381]: cluster 2024-09-10T19:05:25.275692+0000 osd.0 (osd.0) 349 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:05:26.053 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:25 smithi083 bash[32381]: cluster 2024-09-10T19:05:25.279394+0000 osd.0 (osd.0) 350 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:05:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:26.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:26.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:25 smithi165 bash[22591]: cluster 2024-09-10T19:05:24.942998+0000 osd.2 (osd.2) 285 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:05:26.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:25 smithi165 bash[22591]: cluster 2024-09-10T19:05:24.946809+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:05:26.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:25 smithi165 bash[22591]: cluster 2024-09-10T19:05:25.275692+0000 osd.0 (osd.0) 349 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:05:26.168 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:25 smithi165 bash[22591]: cluster 2024-09-10T19:05:25.279394+0000 osd.0 (osd.0) 350 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:05:26.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:26.293 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:26.326 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:26.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:26.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:26.465 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:26.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:26.600 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:26.702 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:26.738 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:26.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:26.877 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:26.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:26.981 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:26.994 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[18629]: cluster 2024-09-10T19:05:25.900258+0000 osd.3 (osd.3) 386 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:05:26.994 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[18629]: cluster 2024-09-10T19:05:25.904943+0000 osd.3 (osd.3) 387 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:05:26.995 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[18629]: cluster 2024-09-10T19:05:26.284245+0000 mgr.y (mgr.24526) 638 : cluster [DBG] pgmap v1043: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:26.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[32381]: cluster 2024-09-10T19:05:25.900258+0000 osd.3 (osd.3) 386 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:05:26.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[32381]: cluster 2024-09-10T19:05:25.904943+0000 osd.3 (osd.3) 387 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:05:26.995 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:26 smithi083 bash[32381]: cluster 2024-09-10T19:05:26.284245+0000 mgr.y (mgr.24526) 638 : cluster [DBG] pgmap v1043: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:27.014 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:26 smithi165 bash[22591]: cluster 2024-09-10T19:05:25.900258+0000 osd.3 (osd.3) 386 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:05:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:26 smithi165 bash[22591]: cluster 2024-09-10T19:05:25.904943+0000 osd.3 (osd.3) 387 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:05:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:26 smithi165 bash[22591]: cluster 2024-09-10T19:05:26.284245+0000 mgr.y (mgr.24526) 638 : cluster [DBG] pgmap v1043: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:27.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:27.151 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:27.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:27.291 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:27.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.428 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:27.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:27.565 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:27.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:27.695 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:27.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.829 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:27.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.965 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:27.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:28.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:28.103 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:28.207 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:26.555931+0000 osd.6 (osd.6) 333 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:26.559370+0000 osd.6 (osd.6) 334 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:26.698292+0000 osd.7 (osd.7) 375 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:26.703417+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:27.216503+0000 osd.0 (osd.0) 351 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:05:28.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:27 smithi165 bash[22591]: cluster 2024-09-10T19:05:27.220342+0000 osd.0 (osd.0) 352 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:05:28.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:28.280 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:26.555931+0000 osd.6 (osd.6) 333 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:05:28.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:26.559370+0000 osd.6 (osd.6) 334 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:26.698292+0000 osd.7 (osd.7) 375 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:26.703417+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:27.216503+0000 osd.0 (osd.0) 351 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:27.220342+0000 osd.0 (osd.0) 352 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:26.555931+0000 osd.6 (osd.6) 333 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:26.559370+0000 osd.6 (osd.6) 334 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:26.698292+0000 osd.7 (osd.7) 375 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:26.703417+0000 osd.7 (osd.7) 376 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:27.216503+0000 osd.0 (osd.0) 351 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:05:28.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:27.220342+0000 osd.0 (osd.0) 352 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:05:28.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:28.381 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:28.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:28.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:28.519 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:28.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:28.658 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:28.761 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:28.794 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:28.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:28.932 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:28.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:29.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:27.869444+0000 osd.1 (osd.1) 363 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:27.874070+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.254171+0000 osd.0 (osd.0) 353 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.258157+0000 osd.0 (osd.0) 354 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: audit 2024-09-10T19:05:28.272239+0000 mgr.y (mgr.24526) 639 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.285105+0000 mgr.y (mgr.24526) 640 : cluster [DBG] pgmap v1044: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:27.869444+0000 osd.1 (osd.1) 363 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:27.874070+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.254171+0000 osd.0 (osd.0) 353 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.258157+0000 osd.0 (osd.0) 354 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:05:29.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: audit 2024-09-10T19:05:28.272239+0000 mgr.y (mgr.24526) 639 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:28 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.285105+0000 mgr.y (mgr.24526) 640 : cluster [DBG] pgmap v1044: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:05:29.047 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: cluster 2024-09-10T19:05:27.869444+0000 osd.1 (osd.1) 363 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:05:29.047 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: cluster 2024-09-10T19:05:27.874070+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:05:29.047 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.254171+0000 osd.0 (osd.0) 353 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:05:29.047 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.258157+0000 osd.0 (osd.0) 354 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:05:29.047 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: audit 2024-09-10T19:05:28.272239+0000 mgr.y (mgr.24526) 639 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:28 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.285105+0000 mgr.y (mgr.24526) 640 : cluster [DBG] pgmap v1044: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:05:29.069 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:29.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:29.208 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:29.312 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.344 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:29.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:29.482 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:29.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:29.613 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:29.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.749 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:29.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:29.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:29.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:30.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:30.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:30.166 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:30.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/1520744621 >> v1:172.21.15.83:6789/0 conn(0x7f68301040d0 legacy=0x7f68301064b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:30.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/1520744621 shutdown_connections 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/1520744621 >> 172.21.15.83:0/1520744621 conn(0x7f68300f6f20 msgr2=0x7f68300f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/1520744621 shutdown_connections 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/1520744621 wait complete. 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 librados: starting msgr at 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 librados: starting objecter 2024-09-10T19:05:30.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 Processor -- start 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- start start 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 librados: setting wanted keys 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 librados: calling monclient init 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f683011c150 con 0x7f68301012f0 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f683011d350 con 0x7f6830107c30 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f683011e550 con 0x7f68301040d0 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f682effd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f68301040d0 0x7f6830117270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39572/0 (socket says 172.21.15.83:39572) 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f682effd640 1 -- 172.21.15.83:0/3165502206 learned_addr learned my addr 172.21.15.83:0/3165502206 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter ms_handle_connect 0x7f68301040d0 2024-09-10T19:05:30.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172303827 0 0) 0x7f683011e550 con 0x7f68301040d0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681c003720 con 0x7f68301040d0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter ms_handle_connect 0x7f68301012f0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter ms_handle_connect 0x7f6830107c30 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732237770 0 0) 0x7f683011d350 con 0x7f6830107c30 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f683011e550 con 0x7f6830107c30 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2830012834 0 0) 0x7f681c003720 con 0x7f68301040d0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f683011d350 con 0x7f68301040d0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451075900 0 0) 0x7f683011c150 con 0x7f68301012f0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681c003720 con 0x7f68301012f0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6818001550 con 0x7f68301040d0 2024-09-10T19:05:30.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805552612 0 0) 0x7f683011d350 con 0x7f68301040d0 2024-09-10T19:05:30.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 >> v1:172.21.15.165:6789/0 conn(0x7f6830107c30 legacy=0x7f683011a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:30.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 >> v1:172.21.15.83:6789/0 conn(0x7f68301012f0 legacy=0x7f6830065380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:30.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683011f750 con 0x7f68301040d0 2024-09-10T19:05:30.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683011c340 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 10 client.49721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f683011c790 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6834843640 1 librados: init done 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6818001230 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.256+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6818003600 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6810005010 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f6818002300 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(674..674 src has 569..674) v4 ==== 8419+0+0 (unknown 2530455051 0 0) 0x7f681808a9b0 con 0x7f68301040d0 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 10 client.49721.objecter ms_dispatch 0x7f6830000b90 osd_map(674..674 src has 569..674) v4 2024-09-10T19:05:30.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 3 client.49721.objecter handle_osd_map got epochs [674,674] > 0 2024-09-10T19:05:30.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 3 client.49721.objecter handle_osd_map decoding full epoch 674 2024-09-10T19:05:30.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 20 client.49721.objecter dump_active .. 0 homeless 2024-09-10T19:05:30.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.260+0000 7f6827fff640 10 client.49721.objecter ms_handle_connect 0x7f681c0759f0 2024-09-10T19:05:30.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.264+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f6818056df0 con 0x7f68301040d0 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.133409+0000 osd.5 (osd.5) 372 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.136746+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.915817+0000 osd.3 (osd.3) 388 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: cluster 2024-09-10T19:05:28.918868+0000 osd.3 (osd.3) 389 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: audit 2024-09-10T19:05:29.579817+0000 mon.a (mon.0) 3242 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[18629]: audit 2024-09-10T19:05:29.584473+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:30.281 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.133409+0000 osd.5 (osd.5) 372 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:05:30.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.136746+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:05:30.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.915817+0000 osd.3 (osd.3) 388 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T19:05:30.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: cluster 2024-09-10T19:05:28.918868+0000 osd.3 (osd.3) 389 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T19:05:30.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: audit 2024-09-10T19:05:29.579817+0000 mon.a (mon.0) 3242 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:30.282 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:29 smithi083 bash[32381]: audit 2024-09-10T19:05:29.584473+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:30.296 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.133409+0000 osd.5 (osd.5) 372 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.136746+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.915817+0000 osd.3 (osd.3) 388 : cluster [DBG] 4.10 deep-scrub starts 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: cluster 2024-09-10T19:05:28.918868+0000 osd.3 (osd.3) 389 : cluster [DBG] 4.10 deep-scrub ok 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: audit 2024-09-10T19:05:29.579817+0000 mon.a (mon.0) 3242 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:30.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:29 smithi165 bash[22591]: audit 2024-09-10T19:05:29.584473+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:30.399 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.432 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:30.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.460+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f68100052c0 con 0x7f68301040d0 2024-09-10T19:05:30.537 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:30.570 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:30.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.706 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:30.810 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:30.845 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:30.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:30.978 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.972+0000 7f6827fff640 1 -- 172.21.15.83:0/3165502206 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v675) v1 ==== 113+0+0 (unknown 723799239 0 0) 0x7f6818056790 con 0x7f68301040d0 2024-09-10T19:05:30.978 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T19:05:30.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 10 librados: watch_flush enter 2024-09-10T19:05:30.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 10 librados: watch_flush exit 2024-09-10T19:05:30.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 20 client.49721.objecter shutdown clearing up homeless session... 2024-09-10T19:05:30.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 10 client.49721.objecter successfully canceled tick 2024-09-10T19:05:30.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 >> v1:172.21.15.83:6800/3071793735 conn(0x7f681c0759f0 legacy=0x7f681c077ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:30.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 >> v1:172.21.15.83:6790/0 conn(0x7f68301040d0 legacy=0x7f6830117270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:30.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 shutdown_connections 2024-09-10T19:05:30.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 >> 172.21.15.83:0/3165502206 conn(0x7f68300f6f20 msgr2=0x7f68301080b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:30.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 shutdown_connections 2024-09-10T19:05:30.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 -- 172.21.15.83:0/3165502206 wait complete. 2024-09-10T19:05:30.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:30.980+0000 7f6834843640 1 librados: shutdown 2024-09-10T19:05:30.985 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:30.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:31.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:31.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.791277+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:05:31.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.794807+0000 osd.7 (osd.7) 378 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:05:31.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.895372+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T19:05:31.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.898474+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.936087+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:29.941219+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.240605+0000 osd.0 (osd.0) 355 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.243802+0000 osd.0 (osd.0) 356 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.285640+0000 mgr.y (mgr.24526) 641 : cluster [DBG] pgmap v1045: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: audit 2024-09-10T19:05:30.465305+0000 mon.c (mon.2) 718 : audit [INF] from='client.? v1:172.21.15.83:0/3165502206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:30 smithi165 bash[22591]: audit 2024-09-10T19:05:30.465725+0000 mon.a (mon.0) 3243 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.119 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:31.222 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:31.248 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:05:31.249 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:05:31.249 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:31.266 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.791277+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.794807+0000 osd.7 (osd.7) 378 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.895372+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.898474+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.936087+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:29.941219+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.240605+0000 osd.0 (osd.0) 355 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.243802+0000 osd.0 (osd.0) 356 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.285640+0000 mgr.y (mgr.24526) 641 : cluster [DBG] pgmap v1045: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: audit 2024-09-10T19:05:30.465305+0000 mon.c (mon.2) 718 : audit [INF] from='client.? v1:172.21.15.83:0/3165502206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[18629]: audit 2024-09-10T19:05:30.465725+0000 mon.a (mon.0) 3243 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.267 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.791277+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.794807+0000 osd.7 (osd.7) 378 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.895372+0000 osd.3 (osd.3) 390 : cluster [DBG] 6.1b deep-scrub starts 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.898474+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.1b deep-scrub ok 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.936087+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:29.941219+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.240605+0000 osd.0 (osd.0) 355 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.243802+0000 osd.0 (osd.0) 356 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.285640+0000 mgr.y (mgr.24526) 641 : cluster [DBG] pgmap v1045: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: audit 2024-09-10T19:05:30.465305+0000 mon.c (mon.2) 718 : audit [INF] from='client.? v1:172.21.15.83:0/3165502206' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.268 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:30 smithi083 bash[32381]: audit 2024-09-10T19:05:30.465725+0000 mon.a (mon.0) 3243 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:05:31.353 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:31.383 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:31.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:31.517 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:31.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:31.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:31.773 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:31.793 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:31.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:31.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:31.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:32.034 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:32.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.530108+0000 osd.6 (osd.6) 335 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:05:32.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.533511+0000 osd.6 (osd.6) 336 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:05:32.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.903876+0000 osd.2 (osd.2) 287 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:32.048 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.904086+0000 osd.2 (osd.2) 288 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.918223+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.921814+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: audit 2024-09-10T19:05:30.976141+0000 mon.a (mon.0) 3244 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:30.976537+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:31.064178+0000 osd.5 (osd.5) 374 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:05:32.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:31 smithi165 bash[22591]: cluster 2024-09-10T19:05:31.067302+0000 osd.5 (osd.5) 375 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:05:32.070 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:32.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:32.207 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.530108+0000 osd.6 (osd.6) 335 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.533511+0000 osd.6 (osd.6) 336 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.903876+0000 osd.2 (osd.2) 287 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.904086+0000 osd.2 (osd.2) 288 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.918223+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.921814+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: audit 2024-09-10T19:05:30.976141+0000 mon.a (mon.0) 3244 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:30.976537+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:31.064178+0000 osd.5 (osd.5) 374 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:05:32.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[32381]: cluster 2024-09-10T19:05:31.067302+0000 osd.5 (osd.5) 375 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.530108+0000 osd.6 (osd.6) 335 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.533511+0000 osd.6 (osd.6) 336 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.903876+0000 osd.2 (osd.2) 287 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.904086+0000 osd.2 (osd.2) 288 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.918223+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.921814+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: audit 2024-09-10T19:05:30.976141+0000 mon.a (mon.0) 3244 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:30.976537+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:31.064178+0000 osd.5 (osd.5) 374 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:05:32.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:31 smithi083 bash[18629]: cluster 2024-09-10T19:05:31.067302+0000 osd.5 (osd.5) 375 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:05:32.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:32.344 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:32.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:32.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:32.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:32.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:32.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:32.752 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:32.796 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:32] "GET /metrics HTTP/1.1" 200 229005 "" "Prometheus/2.43.0" 2024-09-10T19:05:32.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:32.892 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:32.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:32.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:33.020 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:33.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:33.154 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[32381]: cluster 2024-09-10T19:05:31.502846+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[32381]: cluster 2024-09-10T19:05:31.506006+0000 osd.6 (osd.6) 338 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.087562+0000 osd.5 (osd.5) 376 : cluster [DBG] 3.c scrub starts 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.092289+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.c scrub ok 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.286487+0000 mgr.y (mgr.24526) 642 : cluster [DBG] pgmap v1047: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18629]: cluster 2024-09-10T19:05:31.502846+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:05:33.182 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18629]: cluster 2024-09-10T19:05:31.506006+0000 osd.6 (osd.6) 338 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:05:33.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.087562+0000 osd.5 (osd.5) 376 : cluster [DBG] 3.c scrub starts 2024-09-10T19:05:33.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.092289+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.c scrub ok 2024-09-10T19:05:33.183 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:32 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.286487+0000 mgr.y (mgr.24526) 642 : cluster [DBG] pgmap v1047: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:33.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T19:05:33.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:33.285 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:33.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:32 smithi165 bash[22591]: cluster 2024-09-10T19:05:31.502846+0000 osd.6 (osd.6) 337 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:05:33.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:32 smithi165 bash[22591]: cluster 2024-09-10T19:05:31.506006+0000 osd.6 (osd.6) 338 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:05:33.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:32 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.087562+0000 osd.5 (osd.5) 376 : cluster [DBG] 3.c scrub starts 2024-09-10T19:05:33.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:32 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.092289+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.c scrub ok 2024-09-10T19:05:33.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:32 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.286487+0000 mgr.y (mgr.24526) 642 : cluster [DBG] pgmap v1047: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:33.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:05:33.390 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:33.412 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:33.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:33.546 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- 172.21.15.83:0/2578131707 >> v1:172.21.15.83:6790/0 conn(0x7fde340fb3d0 legacy=0x7fde340fb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- 172.21.15.83:0/2578131707 shutdown_connections 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- 172.21.15.83:0/2578131707 >> 172.21.15.83:0/2578131707 conn(0x7fde34075290 msgr2=0x7fde340f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- 172.21.15.83:0/2578131707 shutdown_connections 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- 172.21.15.83:0/2578131707 wait complete. 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 librados: starting msgr at 2024-09-10T19:05:33.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 librados: starting objecter 2024-09-10T19:05:33.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 Processor -- start 2024-09-10T19:05:33.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- start start 2024-09-10T19:05:33.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 librados: setting wanted keys 2024-09-10T19:05:33.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 librados: calling monclient init 2024-09-10T19:05:33.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde3411e5c0 con 0x7fde340fb3d0 2024-09-10T19:05:33.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde3411f760 con 0x7fde3410b200 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde3b67b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde34120900 con 0x7fde340fbeb0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde39417640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fde340fb3d0 0x7fde3410a3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37080/0 (socket says 172.21.15.83:37080) 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde39417640 1 -- 172.21.15.83:0/1803303755 learned_addr learned my addr 172.21.15.83:0/1803303755 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter ms_handle_connect 0x7fde340fb3d0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter ms_handle_connect 0x7fde340fbeb0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854910519 0 0) 0x7fde34120900 con 0x7fde340fbeb0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde20003720 con 0x7fde340fbeb0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter ms_handle_connect 0x7fde3410b200 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736817525 0 0) 0x7fde3411e5c0 con 0x7fde340fb3d0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde34120900 con 0x7fde340fb3d0 2024-09-10T19:05:33.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220247588 0 0) 0x7fde3411f760 con 0x7fde3410b200 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde3411e5c0 con 0x7fde3410b200 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135214768 0 0) 0x7fde20003720 con 0x7fde340fbeb0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde3411f760 con 0x7fde340fbeb0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313616807 0 0) 0x7fde34120900 con 0x7fde340fb3d0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde20003720 con 0x7fde340fb3d0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fde240017f0 con 0x7fde340fbeb0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.624+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fde300015d0 con 0x7fde340fb3d0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 944054746 0 0) 0x7fde3411e5c0 con 0x7fde3410b200 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde34120900 con 0x7fde3410b200 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fde2c001c40 con 0x7fde3410b200 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 381143007 0 0) 0x7fde3411f760 con 0x7fde340fbeb0 2024-09-10T19:05:33.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 >> v1:172.21.15.165:6789/0 conn(0x7fde3410b200 legacy=0x7fde3411cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:33.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 >> v1:172.21.15.83:6789/0 conn(0x7fde340fb3d0 legacy=0x7fde3410a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:33.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34121aa0 con 0x7fde340fbeb0 2024-09-10T19:05:33.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde34120af0 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fde24001e70 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fde240030d0 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde3b67b640 10 client.49727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde34120f40 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde3b67b640 1 librados: init done 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde34065760 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fde24001a10 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(676..676 src has 569..676) v4 ==== 8531+0+0 (unknown 3621200281 0 0) 0x7fde24087e90 con 0x7fde340fbeb0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 10 client.49727.objecter ms_dispatch 0x7fde34000b90 osd_map(676..676 src has 569..676) v4 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 3 client.49727.objecter handle_osd_map got epochs [676,676] > 0 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 3 client.49727.objecter handle_osd_map decoding full epoch 676 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 20 client.49727.objecter dump_active .. 0 homeless 2024-09-10T19:05:33.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.628+0000 7fde29ffb640 10 client.49727.objecter ms_handle_connect 0x7fde20071720 2024-09-10T19:05:33.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.636+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fde24054240 con 0x7fde340fbeb0 2024-09-10T19:05:33.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:33.678 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:33.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:33.813 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:33.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:33.828+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fde34110d40 con 0x7fde340fbeb0 2024-09-10T19:05:33.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:33.951 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:33.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:34.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.000+0000 7fde29ffb640 1 -- 172.21.15.83:0/1803303755 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v677) v1 ==== 87+0+0 (unknown 4159056446 0 0) 0x7fde24053be0 con 0x7fde340fbeb0 2024-09-10T19:05:34.005 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:05:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 10 librados: watch_flush enter 2024-09-10T19:05:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 10 librados: watch_flush exit 2024-09-10T19:05:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 20 client.49727.objecter shutdown clearing up homeless session... 2024-09-10T19:05:34.011 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 10 client.49727.objecter successfully canceled tick 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 >> v1:172.21.15.83:6800/3071793735 conn(0x7fde20071720 legacy=0x7fde20073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 >> v1:172.21.15.83:6790/0 conn(0x7fde340fbeb0 legacy=0x7fde34119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 shutdown_connections 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 >> 172.21.15.83:0/1803303755 conn(0x7fde34075290 msgr2=0x7fde340f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 shutdown_connections 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 -- 172.21.15.83:0/1803303755 wait complete. 2024-09-10T19:05:34.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:34.008+0000 7fde3b67b640 1 librados: shutdown 2024-09-10T19:05:34.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.749122+0000 osd.7 (osd.7) 379 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.752007+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.916840+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.921108+0000 osd.3 (osd.3) 395 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: cluster 2024-09-10T19:05:32.998227+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: audit 2024-09-10T19:05:33.036486+0000 mon.c (mon.2) 719 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: audit 2024-09-10T19:05:33.036971+0000 mon.a (mon.0) 3247 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: audit 2024-09-10T19:05:33.833904+0000 mon.c (mon.2) 720 : audit [INF] from='client.? v1:172.21.15.83:0/1803303755' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.066 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[18629]: audit 2024-09-10T19:05:33.834405+0000 mon.a (mon.0) 3248 : audit [INF] from='client.49727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.749122+0000 osd.7 (osd.7) 379 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.752007+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.916840+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.921108+0000 osd.3 (osd.3) 395 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: cluster 2024-09-10T19:05:32.998227+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: audit 2024-09-10T19:05:33.036486+0000 mon.c (mon.2) 719 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: audit 2024-09-10T19:05:33.036971+0000 mon.a (mon.0) 3247 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: audit 2024-09-10T19:05:33.833904+0000 mon.c (mon.2) 720 : audit [INF] from='client.? v1:172.21.15.83:0/1803303755' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:34 smithi083 bash[32381]: audit 2024-09-10T19:05:33.834405+0000 mon.a (mon.0) 3248 : audit [INF] from='client.49727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.089 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:34.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.749122+0000 osd.7 (osd.7) 379 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.752007+0000 osd.7 (osd.7) 380 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.916840+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.921108+0000 osd.3 (osd.3) 395 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: cluster 2024-09-10T19:05:32.998227+0000 mon.a (mon.0) 3246 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: audit 2024-09-10T19:05:33.036486+0000 mon.c (mon.2) 719 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: audit 2024-09-10T19:05:33.036971+0000 mon.a (mon.0) 3247 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:34.205 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: audit 2024-09-10T19:05:33.833904+0000 mon.c (mon.2) 720 : audit [INF] from='client.? v1:172.21.15.83:0/1803303755' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.206 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:34 smithi165 bash[22591]: audit 2024-09-10T19:05:33.834405+0000 mon.a (mon.0) 3248 : audit [INF] from='client.49727 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:05:34.227 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:34.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:34.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:34.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:34.495 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:34.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:34.633 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:34.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:34.768 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:34.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:34.902 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:34.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:35.006 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.033 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:35.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:35.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:35.275 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:33.489344+0000 osd.6 (osd.6) 339 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:33.491955+0000 osd.6 (osd.6) 340 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:33.930808+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:33.935982+0000 osd.3 (osd.3) 397 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:33.991652+0000 mon.a (mon.0) 3249 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:35.289 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: audit 2024-09-10T19:05:34.003210+0000 mon.a (mon.0) 3250 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: audit 2024-09-10T19:05:34.003420+0000 mon.a (mon.0) 3251 : audit [INF] from='client.49727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.003573+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: audit 2024-09-10T19:05:34.004747+0000 mon.c (mon.2) 721 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: audit 2024-09-10T19:05:34.005369+0000 mon.a (mon.0) 3253 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.015974+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.022366+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.287362+0000 mgr.y (mgr.24526) 643 : cluster [DBG] pgmap v1050: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.291281+0000 mon.a (mon.0) 3254 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: audit 2024-09-10T19:05:34.300691+0000 mon.a (mon.0) 3255 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[32381]: cluster 2024-09-10T19:05:34.300861+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:05:35.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:33.489344+0000 osd.6 (osd.6) 339 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:33.491955+0000 osd.6 (osd.6) 340 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:33.930808+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:33.935982+0000 osd.3 (osd.3) 397 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:33.991652+0000 mon.a (mon.0) 3249 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: audit 2024-09-10T19:05:34.003210+0000 mon.a (mon.0) 3250 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: audit 2024-09-10T19:05:34.003420+0000 mon.a (mon.0) 3251 : audit [INF] from='client.49727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.003573+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: audit 2024-09-10T19:05:34.004747+0000 mon.c (mon.2) 721 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: audit 2024-09-10T19:05:34.005369+0000 mon.a (mon.0) 3253 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.015974+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.022366+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.287362+0000 mgr.y (mgr.24526) 643 : cluster [DBG] pgmap v1050: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.291281+0000 mon.a (mon.0) 3254 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:35.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: audit 2024-09-10T19:05:34.300691+0000 mon.a (mon.0) 3255 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:35.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:35 smithi083 bash[18629]: cluster 2024-09-10T19:05:34.300861+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:05:35.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:35.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:33.489344+0000 osd.6 (osd.6) 339 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:33.491955+0000 osd.6 (osd.6) 340 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:33.930808+0000 osd.3 (osd.3) 396 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:33.935982+0000 osd.3 (osd.3) 397 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:33.991652+0000 mon.a (mon.0) 3249 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: audit 2024-09-10T19:05:34.003210+0000 mon.a (mon.0) 3250 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: audit 2024-09-10T19:05:34.003420+0000 mon.a (mon.0) 3251 : audit [INF] from='client.49727 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.003573+0000 mon.a (mon.0) 3252 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: audit 2024-09-10T19:05:34.004747+0000 mon.c (mon.2) 721 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: audit 2024-09-10T19:05:34.005369+0000 mon.a (mon.0) 3253 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]: dispatch 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.015974+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.022366+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:05:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.287362+0000 mgr.y (mgr.24526) 643 : cluster [DBG] pgmap v1050: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:05:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.291281+0000 mon.a (mon.0) 3254 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: audit 2024-09-10T19:05:34.300691+0000 mon.a (mon.0) 3255 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-116"}]': finished 2024-09-10T19:05:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:35 smithi165 bash[22591]: cluster 2024-09-10T19:05:34.300861+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:05:35.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:35.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:35.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:35.590 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:35.693 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.717 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:35.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:35.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.991 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:35.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:36.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:36.131 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:36.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:36.270 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:36.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:36 smithi165 bash[22591]: cluster 2024-09-10T19:05:35.311809+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:05:36.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:36.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:36.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:36.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:36.656 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:36 smithi083 bash[18629]: cluster 2024-09-10T19:05:35.311809+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:05:36.656 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:36 smithi083 bash[32381]: cluster 2024-09-10T19:05:35.311809+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:05:36.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:36.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:36.818 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:36.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- 172.21.15.83:0/960158793 >> v1:172.21.15.83:6789/0 conn(0x7fae181075b0 legacy=0x7fae18109a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:36.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- 172.21.15.83:0/960158793 shutdown_connections 2024-09-10T19:05:36.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- 172.21.15.83:0/960158793 >> 172.21.15.83:0/960158793 conn(0x7fae18074a20 msgr2=0x7fae180f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:36.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- 172.21.15.83:0/960158793 shutdown_connections 2024-09-10T19:05:36.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- 172.21.15.83:0/960158793 wait complete. 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 librados: starting msgr at 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 librados: starting objecter 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 Processor -- start 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- start start 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 librados: setting wanted keys 2024-09-10T19:05:36.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 librados: calling monclient init 2024-09-10T19:05:36.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fae18119e60 con 0x7fae18103a50 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fae1811b060 con 0x7fae181075b0 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1cc7b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fae1811c260 con 0x7fae18100c70 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae15d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fae18103a50 0x7fae18114bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:37104/0 (socket says 172.21.15.83:37104) 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae15d9b640 1 -- 172.21.15.83:0/3456433517 learned_addr learned my addr 172.21.15.83:0/3456433517 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae1659c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fae18100c70 0x7fae18100150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:46904/0 (socket says 172.21.15.83:46904) 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter ms_handle_connect 0x7fae18103a50 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter ms_handle_connect 0x7fae18100c70 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:36.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter ms_handle_connect 0x7fae181075b0 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992457366 0 0) 0x7fae18119e60 con 0x7fae18103a50 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae000034e0 con 0x7fae18103a50 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251604390 0 0) 0x7fae1811c260 con 0x7fae18100c70 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae18119e60 con 0x7fae18100c70 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637269854 0 0) 0x7fae1811b060 con 0x7fae181075b0 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae1811c260 con 0x7fae181075b0 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3219104958 0 0) 0x7fae000034e0 con 0x7fae18103a50 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae1811b060 con 0x7fae18103a50 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fae0c002a10 con 0x7fae18103a50 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3437497825 0 0) 0x7fae18119e60 con 0x7fae18100c70 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae000034e0 con 0x7fae18100c70 2024-09-10T19:05:36.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.915+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fae080018c0 con 0x7fae18100c70 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335610496 0 0) 0x7fae1811c260 con 0x7fae181075b0 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae18119e60 con 0x7fae181075b0 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fae100016f0 con 0x7fae181075b0 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3070282167 0 0) 0x7fae000034e0 con 0x7fae18100c70 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.165:6789/0 conn(0x7fae181075b0 legacy=0x7fae18118560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6789/0 conn(0x7fae18103a50 legacy=0x7fae18114bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1811d460 con 0x7fae18100c70 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1811b250 con 0x7fae18100c70 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae1cc7b640 10 client.49733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1811b6a0 con 0x7fae18100c70 2024-09-10T19:05:36.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae1cc7b640 1 librados: init done 2024-09-10T19:05:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fae080022b0 con 0x7fae18100c70 2024-09-10T19:05:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fae08005990 con 0x7fae18100c70 2024-09-10T19:05:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fadf8004ff0 con 0x7fae18100c70 2024-09-10T19:05:36.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.919+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fae0801bdb0 con 0x7fae18100c70 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(680..680 src has 569..680) v4 ==== 8467+0+0 (unknown 723430534 0 0) 0x7fae0808a660 con 0x7fae18100c70 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 osd_map(680..680 src has 569..680) v4 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 3 client.49733.objecter handle_osd_map got epochs [680,680] > 0 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 3 client.49733.objecter handle_osd_map decoding full epoch 680 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 20 client.49733.objecter dump_active .. 0 homeless 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v680) v1 ==== 54+0+16 (unknown 2243635159 0 2420000192) 0x7fae0808ca00 con 0x7fae18100c70 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fae00072990 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 1 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 1 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 0 1 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 4 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 10 client.49733.objecter _send_command 1 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf8004a60 con 0x7fadf8004610 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8004610 2024-09-10T19:05:36.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf8004a60 con 0x7fadf8004610 2024-09-10T19:05:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:05:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 10 client.49733.objecter _finish_command 1 = system:0 2024-09-10T19:05:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 4 2024-09-10T19:05:36.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.923+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 0 1 2024-09-10T19:05:36.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:36.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 2 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 2 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8005290 osd=0 3 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 0 2 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 4 2024-09-10T19:05:36.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 10 client.49733.objecter _send_command 2 2024-09-10T19:05:36.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf8004e40 con 0x7fadf8004610 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf8004e40 con 0x7fadf8004610 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 10 client.49733.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 4 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 0 2 2024-09-10T19:05:36.949 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:05:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:05:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 3 2024-09-10T19:05:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 1 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 4 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 10 client.49733.objecter _send_command 3 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf80022e0 con 0x7fadf8004110 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.943+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8004110 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.947+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf80022e0 con 0x7fadf8004110 2024-09-10T19:05:36.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.947+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:05:36.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.947+0000 7fae06ffd640 10 client.49733.objecter _finish_command 3 = system:0 2024-09-10T19:05:36.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.947+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 4 2024-09-10T19:05:36.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.947+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 1 3 2024-09-10T19:05:36.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 4 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 4 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8003fa0 osd=1 3 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 1 4 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 4 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 10 client.49733.objecter _send_command 4 2024-09-10T19:05:36.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.963+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf800d350 con 0x7fadf8004110 2024-09-10T19:05:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf800d350 con 0x7fadf8004110 2024-09-10T19:05:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 10 client.49733.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 4 2024-09-10T19:05:36.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 1 4 2024-09-10T19:05:36.973 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:05:36.973 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:05:36.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:36.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 5 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 5 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 2 5 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 4 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 10 client.49733.objecter _send_command 5 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf80150e0 con 0x7fadf8003af0 2024-09-10T19:05:36.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.967+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8003af0 2024-09-10T19:05:36.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.971+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf80150e0 con 0x7fadf8003af0 2024-09-10T19:05:36.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.971+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:05:36.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.971+0000 7fae06ffd640 10 client.49733.objecter _finish_command 5 = system:0 2024-09-10T19:05:36.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.971+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 4 2024-09-10T19:05:36.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.971+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 2 5 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 6 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 6 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8001c00 osd=2 3 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 2 6 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 4 2024-09-10T19:05:36.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 10 client.49733.objecter _send_command 6 2024-09-10T19:05:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf8003060 con 0x7fadf8003af0 2024-09-10T19:05:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf8003060 con 0x7fadf8003af0 2024-09-10T19:05:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:36.993 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae06ffd640 10 client.49733.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 4 2024-09-10T19:05:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 2 6 2024-09-10T19:05:36.994 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:05:36.994 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 7 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 7 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 3 7 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 4 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 10 client.49733.objecter _send_command 7 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.987+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf800d0d0 con 0x7fadf8015250 2024-09-10T19:05:36.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8015250 2024-09-10T19:05:36.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf800d0d0 con 0x7fadf8015250 2024-09-10T19:05:36.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:05:36.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 10 client.49733.objecter _finish_command 7 = system:0 2024-09-10T19:05:36.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 4 2024-09-10T19:05:36.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:36.991+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 3 7 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 8 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:37.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 8 2024-09-10T19:05:37.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf800c990 osd=3 3 2024-09-10T19:05:37.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 3 8 2024-09-10T19:05:37.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 4 2024-09-10T19:05:37.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 10 client.49733.objecter _send_command 8 2024-09-10T19:05:37.010 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.003+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf8017700 con 0x7fadf8015250 2024-09-10T19:05:37.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf8017700 con 0x7fadf8015250 2024-09-10T19:05:37.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:37.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 10 client.49733.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:37.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 4 2024-09-10T19:05:37.012 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 3 8 2024-09-10T19:05:37.013 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:05:37.013 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:05:37.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:37.013 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 9 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 9 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 4 9 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 4 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 10 client.49733.objecter _send_command 9 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf801e640 con 0x7fadf801ab40 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.007+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf801ab40 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.011+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf801e640 con 0x7fadf801ab40 2024-09-10T19:05:37.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.011+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:05:37.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.011+0000 7fae06ffd640 10 client.49733.objecter _finish_command 9 = system:0 2024-09-10T19:05:37.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.011+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 4 2024-09-10T19:05:37.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.011+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 4 9 2024-09-10T19:05:37.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:37.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 10 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 10 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8017920 osd=4 3 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 4 10 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 4 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 10 client.49733.objecter _send_command 10 2024-09-10T19:05:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.027+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf801f7c0 con 0x7fadf801ab40 2024-09-10T19:05:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.031+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf801f7c0 con 0x7fadf801ab40 2024-09-10T19:05:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.031+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.031+0000 7fae06ffd640 10 client.49733.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.031+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 4 2024-09-10T19:05:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.031+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 4 10 2024-09-10T19:05:37.039 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:05:37.041 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:05:37.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:37.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 11 2024-09-10T19:05:37.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 11 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 5 11 2024-09-10T19:05:37.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 4 2024-09-10T19:05:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 10 client.49733.objecter _send_command 11 2024-09-10T19:05:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf8017550 con 0x7fadf8020670 2024-09-10T19:05:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8020670 2024-09-10T19:05:37.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf8017550 con 0x7fadf8020670 2024-09-10T19:05:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:05:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 10 client.49733.objecter _finish_command 11 = system:0 2024-09-10T19:05:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 4 2024-09-10T19:05:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.035+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 5 11 2024-09-10T19:05:37.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 12 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 12 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf801f7c0 osd=5 3 2024-09-10T19:05:37.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 5 12 2024-09-10T19:05:37.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 4 2024-09-10T19:05:37.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 10 client.49733.objecter _send_command 12 2024-09-10T19:05:37.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.051+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf8017550 con 0x7fadf8020670 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.055+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf8017550 con 0x7fadf8020670 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.055+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.055+0000 7fae06ffd640 10 client.49733.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.055+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 4 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.055+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 5 12 2024-09-10T19:05:37.062 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 13 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 13 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 6 13 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 4 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 10 client.49733.objecter _send_command 13 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf8026e40 con 0x7fadf8020c20 2024-09-10T19:05:37.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8020c20 2024-09-10T19:05:37.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf8026e40 con 0x7fadf8020c20 2024-09-10T19:05:37.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:05:37.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 10 client.49733.objecter _finish_command 13 = system:0 2024-09-10T19:05:37.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 4 2024-09-10T19:05:37.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.059+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 6 13 2024-09-10T19:05:37.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:37.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 14 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 14 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf8017550 osd=6 3 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 6 14 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 4 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 10 client.49733.objecter _send_command 14 2024-09-10T19:05:37.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.075+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf80286e0 con 0x7fadf8020c20 2024-09-10T19:05:37.084 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf80286e0 con 0x7fadf8020c20 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 10 client.49733.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 4 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 6 14 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:37.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 15 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 15 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 7 15 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 4 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 10 client.49733.objecter _send_command 15 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadf80292c0 con 0x7fadf8028d90 2024-09-10T19:05:37.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.079+0000 7fae06ffd640 10 client.49733.objecter ms_handle_connect 0x7fadf8028d90 2024-09-10T19:05:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.083+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fadf80292c0 con 0x7fadf8028d90 2024-09-10T19:05:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.083+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:05:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.083+0000 7fae06ffd640 10 client.49733.objecter _finish_command 15 = system:0 2024-09-10T19:05:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.083+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 4 2024-09-10T19:05:37.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.083+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 7 15 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 10 client.49733.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign -1 16 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 20 client.49733.objecter _get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_remove -1 16 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 20 client.49733.objecter get_session s=0x7fadf80286e0 osd=7 3 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 15 client.49733.objecter _session_command_op_assign 7 16 2024-09-10T19:05:37.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 4 2024-09-10T19:05:37.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 10 client.49733.objecter _send_command 16 2024-09-10T19:05:37.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.099+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fadf80292c0 con 0x7fadf8028d90 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae06ffd640 1 -- 172.21.15.83:0/3456433517 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fadf80292c0 con 0x7fadf8028d90 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae06ffd640 10 client.49733.objecter ms_dispatch 0x7fae18000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae06ffd640 10 client.49733.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae06ffd640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 4 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae06ffd640 15 client.49733.objecter _session_command_op_remove 7 16 2024-09-10T19:05:37.109 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:05:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:05:37.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 librados: watch_flush enter 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 librados: watch_flush exit 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.0 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6801/2535527483 conn(0x7fadf8004610 legacy=0x7fadf8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8005290 osd=0 1 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.1 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6805/2309624637 conn(0x7fadf8004110 legacy=0x7fadf800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8003fa0 osd=1 1 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.2 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6809/386094220 conn(0x7fadf8003af0 legacy=0x7fadf8013920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae16d9d640 1 -- 172.21.15.83:0/3456433517 reap_dead start 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8001c00 osd=2 1 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.3 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6813/1279688021 conn(0x7fadf8015250 legacy=0x7fadf80156b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf800c990 osd=3 1 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.4 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.165:6800/3225515265 conn(0x7fadf801ab40 legacy=0x7fadf801cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017920 osd=4 1 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.5 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.165:6804/571173777 conn(0x7fadf8020670 legacy=0x7fadf8015dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf801f7c0 osd=5 1 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.6 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.165:6808/1317922936 conn(0x7fadf8020c20 legacy=0x7fadf80255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf8017550 osd=6 1 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter close_session for osd.7 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.165:6812/1610151650 conn(0x7fadf8028d90 legacy=0x7fadf8026e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter put_session s=0x7fadf80286e0 osd=7 1 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 20 client.49733.objecter shutdown clearing up homeless session... 2024-09-10T19:05:37.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae1cc7b640 10 client.49733.objecter successfully canceled tick 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.103+0000 7fae16d9d640 1 -- 172.21.15.83:0/3456433517 reap_dead start 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6800/3071793735 conn(0x7fae00072990 legacy=0x7fae00074e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> v1:172.21.15.83:6790/0 conn(0x7fae18100c70 legacy=0x7fae18100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 shutdown_connections 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 >> 172.21.15.83:0/3456433517 conn(0x7fae18074a20 msgr2=0x7fae18070d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 shutdown_connections 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 -- 172.21.15.83:0/3456433517 wait complete. 2024-09-10T19:05:37.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:37.107+0000 7fae1cc7b640 1 librados: shutdown 2024-09-10T19:05:37.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:37.214 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:37.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:37.357 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:37.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:37.494 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:37.548 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[18629]: cluster 2024-09-10T19:05:36.289336+0000 mgr.y (mgr.24526) 644 : cluster [DBG] pgmap v1053: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[18629]: cluster 2024-09-10T19:05:36.324083+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[18629]: audit 2024-09-10T19:05:36.328792+0000 mon.c (mon.2) 722 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[18629]: audit 2024-09-10T19:05:36.330810+0000 mon.a (mon.0) 3259 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[18629]: audit 2024-09-10T19:05:36.925998+0000 mon.c (mon.2) 723 : audit [DBG] from='client.? v1:172.21.15.83:0/3456433517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[32381]: cluster 2024-09-10T19:05:36.289336+0000 mgr.y (mgr.24526) 644 : cluster [DBG] pgmap v1053: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:37.562 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[32381]: cluster 2024-09-10T19:05:36.324083+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:05:37.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[32381]: audit 2024-09-10T19:05:36.328792+0000 mon.c (mon.2) 722 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[32381]: audit 2024-09-10T19:05:36.330810+0000 mon.a (mon.0) 3259 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.563 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:37 smithi083 bash[32381]: audit 2024-09-10T19:05:36.925998+0000 mon.c (mon.2) 723 : audit [DBG] from='client.? v1:172.21.15.83:0/3456433517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:37.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:37 smithi165 bash[22591]: cluster 2024-09-10T19:05:36.289336+0000 mgr.y (mgr.24526) 644 : cluster [DBG] pgmap v1053: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:37 smithi165 bash[22591]: cluster 2024-09-10T19:05:36.324083+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:05:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:37 smithi165 bash[22591]: audit 2024-09-10T19:05:36.328792+0000 mon.c (mon.2) 722 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:37 smithi165 bash[22591]: audit 2024-09-10T19:05:36.330810+0000 mon.a (mon.0) 3259 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:37 smithi165 bash[22591]: audit 2024-09-10T19:05:36.925998+0000 mon.c (mon.2) 723 : audit [DBG] from='client.? v1:172.21.15.83:0/3456433517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:37.632 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:37.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:37.772 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:37.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:37.910 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:37.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:38.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:38.040 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:38.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:38.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:38.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:38.315 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:38.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:38.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:38.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:38 smithi165 bash[22591]: audit 2024-09-10T19:05:37.332008+0000 mon.a (mon.0) 3260 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:38.432 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:38 smithi165 bash[22591]: cluster 2024-09-10T19:05:37.332590+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:05:38.455 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:38.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:38.558 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:38.592 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:38.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:38 smithi083 bash[32381]: audit 2024-09-10T19:05:37.332008+0000 mon.a (mon.0) 3260 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:38.593 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:38 smithi083 bash[32381]: cluster 2024-09-10T19:05:37.332590+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:05:38.593 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:38 smithi083 bash[18629]: audit 2024-09-10T19:05:37.332008+0000 mon.a (mon.0) 3260 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:38.594 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:38 smithi083 bash[18629]: cluster 2024-09-10T19:05:37.332590+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:05:38.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:38.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:38.729 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:38.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:38.832 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:38.865 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:38.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:38.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:39.001 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:39.104 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:39.135 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:39.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.273 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:39.377 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:38.283503+0000 mgr.y (mgr.24526) 645 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:39.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: cluster 2024-09-10T19:05:38.290136+0000 mgr.y (mgr.24526) 646 : cluster [DBG] pgmap v1056: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:39.389 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: cluster 2024-09-10T19:05:38.354353+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:38.364218+0000 mon.c (mon.2) 724 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:38.364603+0000 mon.a (mon.0) 3263 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:39.303478+0000 mon.a (mon.0) 3264 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: cluster 2024-09-10T19:05:39.303570+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:39.307635+0000 mon.c (mon.2) 725 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[32381]: audit 2024-09-10T19:05:39.311795+0000 mon.a (mon.0) 3266 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:38.283503+0000 mgr.y (mgr.24526) 645 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: cluster 2024-09-10T19:05:38.290136+0000 mgr.y (mgr.24526) 646 : cluster [DBG] pgmap v1056: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: cluster 2024-09-10T19:05:38.354353+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:05:39.390 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:38.364218+0000 mon.c (mon.2) 724 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:38.364603+0000 mon.a (mon.0) 3263 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:39.303478+0000 mon.a (mon.0) 3264 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: cluster 2024-09-10T19:05:39.303570+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:05:39.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:39.307635+0000 mon.c (mon.2) 725 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.391 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:39 smithi083 bash[18629]: audit 2024-09-10T19:05:39.311795+0000 mon.a (mon.0) 3266 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.409 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:39.512 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:39.538 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:39.538 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:39.549 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:38.283503+0000 mgr.y (mgr.24526) 645 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: cluster 2024-09-10T19:05:38.290136+0000 mgr.y (mgr.24526) 646 : cluster [DBG] pgmap v1056: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-09-10T19:05:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: cluster 2024-09-10T19:05:38.354353+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:05:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:38.364218+0000 mon.c (mon.2) 724 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:38.364603+0000 mon.a (mon.0) 3263 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:39.303478+0000 mon.a (mon.0) 3264 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: cluster 2024-09-10T19:05:39.303570+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:05:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:39.307635+0000 mon.c (mon.2) 725 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:39 smithi165 bash[22591]: audit 2024-09-10T19:05:39.311795+0000 mon.a (mon.0) 3266 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:39.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:39.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.826 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:39.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:39.963 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:39.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:40.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.088 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:05:40.104 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:40.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.236 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:40.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:40.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:40.607 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.639 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:40.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.773 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:40.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:40.913 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:40.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:41.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:41.050 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:41.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.188 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:41.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:41.326 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:41.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.436 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:41.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[22591]: cluster 2024-09-10T19:05:40.290743+0000 mgr.y (mgr.24526) 647 : cluster [DBG] pgmap v1059: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:41.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[22591]: audit 2024-09-10T19:05:40.318274+0000 mon.a (mon.0) 3267 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:41.436 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[22591]: cluster 2024-09-10T19:05:40.318578+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:05:41.437 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[22591]: audit 2024-09-10T19:05:40.321250+0000 mon.c (mon.2) 726 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.437 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:41 smithi165 bash[22591]: audit 2024-09-10T19:05:40.321610+0000 mon.a (mon.0) 3269 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.452 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:41.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.567 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:41.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:41.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:41.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[32381]: cluster 2024-09-10T19:05:40.290743+0000 mgr.y (mgr.24526) 647 : cluster [DBG] pgmap v1059: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[32381]: audit 2024-09-10T19:05:40.318274+0000 mon.a (mon.0) 3267 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[32381]: cluster 2024-09-10T19:05:40.318578+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:05:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[32381]: audit 2024-09-10T19:05:40.321250+0000 mon.c (mon.2) 726 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[32381]: audit 2024-09-10T19:05:40.321610+0000 mon.a (mon.0) 3269 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[18629]: cluster 2024-09-10T19:05:40.290743+0000 mgr.y (mgr.24526) 647 : cluster [DBG] pgmap v1059: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:41.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[18629]: audit 2024-09-10T19:05:40.318274+0000 mon.a (mon.0) 3267 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:41.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[18629]: cluster 2024-09-10T19:05:40.318578+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:05:41.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[18629]: audit 2024-09-10T19:05:40.321250+0000 mon.c (mon.2) 726 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:41 smithi083 bash[18629]: audit 2024-09-10T19:05:40.321610+0000 mon.a (mon.0) 3269 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]: dispatch 2024-09-10T19:05:41.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:41.862 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:41.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:41.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:41.998 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:42.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.135 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:42.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:42.274 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:42.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.404 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:42.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[18629]: cluster 2024-09-10T19:05:41.311036+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:42.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[18629]: audit 2024-09-10T19:05:41.317659+0000 mon.a (mon.0) 3271 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]': finished 2024-09-10T19:05:42.465 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[18629]: cluster 2024-09-10T19:05:41.317739+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:05:42.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[32381]: cluster 2024-09-10T19:05:41.311036+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:42.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[32381]: audit 2024-09-10T19:05:41.317659+0000 mon.a (mon.0) 3271 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]': finished 2024-09-10T19:05:42.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[32381]: cluster 2024-09-10T19:05:41.317739+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:05:42.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.544 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:42 smithi165 bash[22591]: cluster 2024-09-10T19:05:41.311036+0000 mon.a (mon.0) 3270 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:42 smithi165 bash[22591]: audit 2024-09-10T19:05:41.317659+0000 mon.a (mon.0) 3271 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-118", "mode": "writeback"}]': finished 2024-09-10T19:05:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:42 smithi165 bash[22591]: cluster 2024-09-10T19:05:41.317739+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:05:42.648 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:42.678 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:42.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:42.785 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:42.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:42] "GET /metrics HTTP/1.1" 200 229005 "" "Prometheus/2.43.0" 2024-09-10T19:05:42.815 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:42.919 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:42.946 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:42.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:43.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:43.084 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:43.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:43.222 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:43.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:43.354 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:43.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:43.491 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:43.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:43.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:43 smithi165 bash[22591]: cluster 2024-09-10T19:05:42.291525+0000 mgr.y (mgr.24526) 648 : cluster [DBG] pgmap v1062: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:43.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:43 smithi165 bash[22591]: cluster 2024-09-10T19:05:42.332059+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:05:43.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:43 smithi165 bash[22591]: audit 2024-09-10T19:05:42.387965+0000 mon.c (mon.2) 727 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.609 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:43 smithi165 bash[22591]: audit 2024-09-10T19:05:42.388447+0000 mon.a (mon.0) 3274 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.628 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:43.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:43.740 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[18629]: cluster 2024-09-10T19:05:42.291525+0000 mgr.y (mgr.24526) 648 : cluster [DBG] pgmap v1062: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[18629]: cluster 2024-09-10T19:05:42.332059+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[18629]: audit 2024-09-10T19:05:42.387965+0000 mon.c (mon.2) 727 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[18629]: audit 2024-09-10T19:05:42.388447+0000 mon.a (mon.0) 3274 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[32381]: cluster 2024-09-10T19:05:42.291525+0000 mgr.y (mgr.24526) 648 : cluster [DBG] pgmap v1062: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[32381]: cluster 2024-09-10T19:05:42.332059+0000 mon.a (mon.0) 3273 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[32381]: audit 2024-09-10T19:05:42.387965+0000 mon.c (mon.2) 727 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.741 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:43 smithi083 bash[32381]: audit 2024-09-10T19:05:42.388447+0000 mon.a (mon.0) 3274 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:43.765 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:43.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:43.904 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:43.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:44.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:44.044 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:44.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:44.180 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:44.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:44.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:44.398 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: audit 2024-09-10T19:05:43.355392+0000 mon.a (mon.0) 3275 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: cluster 2024-09-10T19:05:43.355486+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: audit 2024-09-10T19:05:43.357255+0000 mon.c (mon.2) 728 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: audit 2024-09-10T19:05:43.357949+0000 mon.a (mon.0) 3277 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: cluster 2024-09-10T19:05:44.297884+0000 mon.a (mon.0) 3278 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: audit 2024-09-10T19:05:44.306045+0000 mon.a (mon.0) 3279 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[18629]: cluster 2024-09-10T19:05:44.306143+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: audit 2024-09-10T19:05:43.355392+0000 mon.a (mon.0) 3275 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: cluster 2024-09-10T19:05:43.355486+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: audit 2024-09-10T19:05:43.357255+0000 mon.c (mon.2) 728 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: audit 2024-09-10T19:05:43.357949+0000 mon.a (mon.0) 3277 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.399 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: cluster 2024-09-10T19:05:44.297884+0000 mon.a (mon.0) 3278 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:44.400 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: audit 2024-09-10T19:05:44.306045+0000 mon.a (mon.0) 3279 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:44.400 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:44 smithi083 bash[32381]: cluster 2024-09-10T19:05:44.306143+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:05:44.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:44.446 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:44.550 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:44.586 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: audit 2024-09-10T19:05:43.355392+0000 mon.a (mon.0) 3275 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: cluster 2024-09-10T19:05:43.355486+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: audit 2024-09-10T19:05:43.357255+0000 mon.c (mon.2) 728 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: audit 2024-09-10T19:05:43.357949+0000 mon.a (mon.0) 3277 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]: dispatch 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: cluster 2024-09-10T19:05:44.297884+0000 mon.a (mon.0) 3278 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: audit 2024-09-10T19:05:44.306045+0000 mon.a (mon.0) 3279 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-118"}]': finished 2024-09-10T19:05:44.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:44 smithi165 bash[22591]: cluster 2024-09-10T19:05:44.306143+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:05:44.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:44.721 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:44.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:44.863 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:44.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:44.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.001 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:45.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:45.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:45.269 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:45.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:45.381 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:45 smithi165 bash[22591]: cluster 2024-09-10T19:05:44.292456+0000 mgr.y (mgr.24526) 649 : cluster [DBG] pgmap v1065: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:45.381 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:45 smithi165 bash[22591]: audit 2024-09-10T19:05:44.579811+0000 mon.a (mon.0) 3281 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:45.381 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:45 smithi165 bash[22591]: audit 2024-09-10T19:05:44.584976+0000 mon.c (mon.2) 729 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:45.400 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:45.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[18629]: cluster 2024-09-10T19:05:44.292456+0000 mgr.y (mgr.24526) 649 : cluster [DBG] pgmap v1065: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[18629]: audit 2024-09-10T19:05:44.579811+0000 mon.a (mon.0) 3281 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[18629]: audit 2024-09-10T19:05:44.584976+0000 mon.c (mon.2) 729 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[32381]: cluster 2024-09-10T19:05:44.292456+0000 mgr.y (mgr.24526) 649 : cluster [DBG] pgmap v1065: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[32381]: audit 2024-09-10T19:05:44.579811+0000 mon.a (mon.0) 3281 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:05:45.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:45 smithi083 bash[32381]: audit 2024-09-10T19:05:44.584976+0000 mon.c (mon.2) 729 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:45.532 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:45.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:45.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:45.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:45.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:45.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:45.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:46.016 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:46.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:46.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:46.215 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:46.318 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.353 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:46.401 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:46 smithi083 bash[18629]: cluster 2024-09-10T19:05:45.371894+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:05:46.401 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:46 smithi083 bash[18629]: cluster 2024-09-10T19:05:46.294576+0000 mgr.y (mgr.24526) 650 : cluster [DBG] pgmap v1068: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.401 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:46 smithi083 bash[32381]: cluster 2024-09-10T19:05:45.371894+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:05:46.402 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:46 smithi083 bash[32381]: cluster 2024-09-10T19:05:46.294576+0000 mgr.y (mgr.24526) 650 : cluster [DBG] pgmap v1068: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.484 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:46.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:46 smithi165 bash[22591]: cluster 2024-09-10T19:05:45.371894+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:05:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:46 smithi165 bash[22591]: cluster 2024-09-10T19:05:46.294576+0000 mgr.y (mgr.24526) 650 : cluster [DBG] pgmap v1068: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.623 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:46.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:46.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:46.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:46.897 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:46.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:47.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:47.034 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:47.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:47.171 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:47.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:47.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:47.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[18629]: cluster 2024-09-10T19:05:46.397560+0000 mon.a (mon.0) 3283 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:05:47.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[18629]: audit 2024-09-10T19:05:46.401240+0000 mon.c (mon.2) 730 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.423 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[18629]: audit 2024-09-10T19:05:46.402504+0000 mon.a (mon.0) 3284 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.423 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[32381]: cluster 2024-09-10T19:05:46.397560+0000 mon.a (mon.0) 3283 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:05:47.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[32381]: audit 2024-09-10T19:05:46.401240+0000 mon.c (mon.2) 730 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.424 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:47 smithi083 bash[32381]: audit 2024-09-10T19:05:46.402504+0000 mon.a (mon.0) 3284 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.441 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:47.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.557 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:47 smithi165 bash[22591]: cluster 2024-09-10T19:05:46.397560+0000 mon.a (mon.0) 3283 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:05:47.558 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:47 smithi165 bash[22591]: audit 2024-09-10T19:05:46.401240+0000 mon.c (mon.2) 730 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.558 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:47 smithi165 bash[22591]: audit 2024-09-10T19:05:46.402504+0000 mon.a (mon.0) 3284 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:47.576 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:47.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:47.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:47.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.848 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:47.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:47.986 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:47.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:48.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:48.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:48.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:48.263 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:48.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:48.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:48.401 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:48.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:48.537 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:48.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: audit 2024-09-10T19:05:47.410028+0000 mon.a (mon.0) 3285 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:48.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: cluster 2024-09-10T19:05:47.410530+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:05:48.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: audit 2024-09-10T19:05:47.434588+0000 mon.c (mon.2) 731 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: audit 2024-09-10T19:05:47.435034+0000 mon.a (mon.0) 3287 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.557 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: audit 2024-09-10T19:05:48.289852+0000 mgr.y (mgr.24526) 651 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[18629]: cluster 2024-09-10T19:05:48.295502+0000 mgr.y (mgr.24526) 652 : cluster [DBG] pgmap v1071: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: audit 2024-09-10T19:05:47.410028+0000 mon.a (mon.0) 3285 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: cluster 2024-09-10T19:05:47.410530+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: audit 2024-09-10T19:05:47.434588+0000 mon.c (mon.2) 731 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: audit 2024-09-10T19:05:47.435034+0000 mon.a (mon.0) 3287 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: audit 2024-09-10T19:05:48.289852+0000 mgr.y (mgr.24526) 651 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:48.558 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:48 smithi083 bash[32381]: cluster 2024-09-10T19:05:48.295502+0000 mgr.y (mgr.24526) 652 : cluster [DBG] pgmap v1071: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:48.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:48.676 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:48.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:48.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: audit 2024-09-10T19:05:47.410028+0000 mon.a (mon.0) 3285 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:48.792 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: cluster 2024-09-10T19:05:47.410530+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:05:48.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: audit 2024-09-10T19:05:47.434588+0000 mon.c (mon.2) 731 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: audit 2024-09-10T19:05:47.435034+0000 mon.a (mon.0) 3287 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:48.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: audit 2024-09-10T19:05:48.289852+0000 mgr.y (mgr.24526) 651 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:48.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:48 smithi165 bash[22591]: cluster 2024-09-10T19:05:48.295502+0000 mgr.y (mgr.24526) 652 : cluster [DBG] pgmap v1071: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:48.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:48.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:48.949 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:48.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:49.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:49.087 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:49.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:49.225 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:49.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:49.352 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:49.455 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:49.487 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:48.419540+0000 mon.a (mon.0) 3288 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: cluster 2024-09-10T19:05:48.419689+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:48.426494+0000 mon.c (mon.2) 732 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:48.427305+0000 mon.a (mon.0) 3290 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:49.311148+0000 mon.a (mon.0) 3291 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:49.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: cluster 2024-09-10T19:05:49.311574+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:49.314833+0000 mon.c (mon.2) 733 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[18629]: audit 2024-09-10T19:05:49.321482+0000 mon.a (mon.0) 3293 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:48.419540+0000 mon.a (mon.0) 3288 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: cluster 2024-09-10T19:05:48.419689+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:48.426494+0000 mon.c (mon.2) 732 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:48.427305+0000 mon.a (mon.0) 3290 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:49.311148+0000 mon.a (mon.0) 3291 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:49.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: cluster 2024-09-10T19:05:49.311574+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:05:49.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:49.314833+0000 mon.c (mon.2) 733 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:49 smithi083 bash[32381]: audit 2024-09-10T19:05:49.321482+0000 mon.a (mon.0) 3293 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:49.622 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:49.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:49.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:49.761 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:48.419540+0000 mon.a (mon.0) 3288 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: cluster 2024-09-10T19:05:48.419689+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:05:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:48.426494+0000 mon.c (mon.2) 732 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:48.427305+0000 mon.a (mon.0) 3290 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:49.311148+0000 mon.a (mon.0) 3291 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: cluster 2024-09-10T19:05:49.311574+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:05:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:49.314833+0000 mon.c (mon.2) 733 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:49 smithi165 bash[22591]: audit 2024-09-10T19:05:49.321482+0000 mon.a (mon.0) 3293 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:49.894 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:49.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:49.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:50.031 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:50.134 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:50.168 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:50.213 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:05:50.213 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 20, 2024-09-10T19:05:50.213 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 600, 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [ 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.48707.0:9 113.6s0 113:78e31a71:test-rados-api-smithi083-77943-18::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:53.971736+0000", 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 596.23673576700003, 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.030410614999999998, 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.48707", 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2936751915", 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 9 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.214 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971736+0000", 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971736+0000", 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.215 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971733+0000", 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971746+0000", 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3066e-05 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971786+0000", 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.0725000000000001e-05 2024-09-10T19:05:50.216 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971799+0000", 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2821e-05 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971886+0000", 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.6423999999999998e-05 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.217 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.971896+0000", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0156999999999999e-05 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.988465+0000", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.016568994 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.988584+0000", 2024-09-10T19:05:50.218 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000119589 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.989573+0000", 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00098832 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:53.991324+0000", 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0017509940000000001 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.219 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.002071+0000", 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.010747359999999999 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.002146+0000", 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.5222999999999995e-05 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.220 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_repop(client.24732.0:2 118.11 e198/196 118:8e20cd32:test-rados-api-smithi083-77788-26::foo:head v 198'1, mlcod=0'0)", 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:55:54.007367+0000", 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 596.20110458900001, 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.037710684000000001, 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.221 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.007367+0000", 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.007367+0000", 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.222 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.007366+0000", 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036240+0000", 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.028873427 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.223 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036268+0000", 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.8056e-05 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036279+0000", 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0564000000000001e-05 2024-09-10T19:05:50.224 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036337+0000", 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8922999999999998e-05 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.225 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.036358+0000", 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.0449e-05 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.044988+0000", 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0086301320000000004 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.226 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:55:54.045078+0000", 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.9537999999999996e-05 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49040.0:1 168.6s0 168:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:57:08.069143+0000", 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 522.13932862299998, 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.496765363, 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.228 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49040", 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2502095084", 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069143+0000", 2024-09-10T19:05:50.229 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069143+0000", 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.230 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069140+0000", 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069165+0000", 2024-09-10T19:05:50.231 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5468000000000001e-05 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069178+0000", 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3162999999999999e-05 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.232 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.069187+0000", 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.1640000000000007e-06 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.075919+0000", 2024-09-10T19:05:50.233 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0067312300000000004 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:08.075923+0000", 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.7269999999999998e-06 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.234 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.562862+0000", 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4869386760000001 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.563146+0000", 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00028359400000000002 2024-09-10T19:05:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.564176+0000", 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0010307529999999999 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.565233+0000", 2024-09-10T19:05:50.236 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0010566830000000001 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.565789+0000", 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00055606099999999995 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.237 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:09.565908+0000", 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000118987 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49142.0:1 181.6s0 181:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:57:43.101928+0000", 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 487.10654300200002, 2024-09-10T19:05:50.238 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.98632019199999998, 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49142", 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3965626681", 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.239 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101928+0000", 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101928+0000", 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101927+0000", 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999981 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.240 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101935+0000", 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.0579999999999998e-06 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101947+0000", 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1977000000000001e-05 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.241 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.101953+0000", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.4849999999999997e-06 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.102021+0000", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.7366999999999994e-05 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:43.102024+0000", 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.1089999999999998e-06 2024-09-10T19:05:50.242 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.082959+0000", 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.98093487999999995 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.243 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.083077+0000", 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000118162 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.084419+0000", 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0013421329999999999 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.086472+0000", 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0020527990000000001 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.244 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.088186+0000", 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001714281 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:57:44.088249+0000", 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.2745999999999999e-05 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.245 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:1995 4.9 4:93e5b521:::notify.7:head [watch reconnect cookie 94430445882480 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e547)", 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:02:51.433237+0000", 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 178.77523484400001, 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.82845558900000005, 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1995 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.246 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433237+0000", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433237+0000", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433232+0000", 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T19:05:50.247 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433249+0000", 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6575999999999999e-05 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433295+0000", 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6189000000000001e-05 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.248 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433315+0000", 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.0313e-05 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433558+0000", 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024273799999999999 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:51.433597+0000", 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8509e-05 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.249 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:52.258815+0000", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.82521820099999998 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:52.258996+0000", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00018154 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:02:52.261692+0000", 2024-09-10T19:05:50.250 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0026957830000000002 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:41 242.0 242:00000000::::head [pgnls start_epoch 659] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e659)", 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:02.301319+0000", 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 47.907152562, 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0027448469999999999, 2024-09-10T19:05:50.251 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 41 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301319+0000", 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.252 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301319+0000", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301314+0000", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301335+0000", 2024-09-10T19:05:50.253 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1710000000000001e-05 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301367+0000", 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.1288000000000003e-05 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301379+0000", 2024-09-10T19:05:50.254 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2774e-05 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301544+0000", 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016445000000000001 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.301563+0000", 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8936999999999998e-05 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.255 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.303838+0000", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0022749929999999999 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.303886+0000", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8782999999999997e-05 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:02.304064+0000", 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00017727199999999999 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:109 243.10 243:091f2506:test-rados-api-smithi083-77828-115::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e665)", 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:08.322051+0000", 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 41.886420518999998, 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0076699639999999996, 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.257 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 109 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322051+0000", 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.258 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322051+0000", 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322047+0000", 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322062+0000", 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.517e-05 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.259 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322085+0000", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3108e-05 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322095+0000", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.4579999999999999e-06 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322160+0000", 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.5586999999999994e-05 2024-09-10T19:05:50.260 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.322434+0000", 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00027348999999999999 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.327780+0000", 2024-09-10T19:05:50.261 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0053461280000000003 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.327895+0000", 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011527 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329379+0000", 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001483418 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.262 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329478+0000", 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.9480000000000003e-05 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329721+0000", 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024276199999999999 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.263 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:110 243.1d 243:bff37f92:test-rados-api-smithi083-77828-115::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e665)", 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:08.329981+0000", 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 41.878490194000001, 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0033406379999999999, 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "reached pg", 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 110 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.264 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329981+0000", 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329981+0000", 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.265 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329978+0000", 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.266 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.267 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.329988+0000", 2024-09-10T19:05:50.268 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.4509999999999998e-06 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.330006+0000", 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8261e-05 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.269 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.330014+0000", 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.7190000000000001e-06 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.330075+0000", 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.1171000000000001e-05 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.330334+0000", 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00025942899999999999 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.270 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.332383+0000", 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002048575 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.332455+0000", 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.1721e-05 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.271 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.333228+0000", 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00077375499999999999 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.333322+0000", 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3539e-05 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.272 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:177 243.10 243:091f2506:test-rados-api-smithi083-77828-115::foo:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e665)", 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:08.376101+0000", 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 41.832370083000001, 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0021801379999999999, 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 177 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.273 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376101+0000", 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376101+0000", 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.274 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376099+0000", 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376110+0000", 2024-09-10T19:05:50.275 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.078e-05 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376133+0000", 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3002000000000001e-05 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376142+0000", 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.577e-06 2024-09-10T19:05:50.276 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376236+0000", 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3430000000000005e-05 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.277 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376386+0000", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000150247 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 0,2", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.376529+0000", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00014247399999999999 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.377287+0000", 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00075842800000000001 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.278 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.377932+0000", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00064499999999999996 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.378077+0000", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00014462200000000001 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.378188+0000", 2024-09-10T19:05:50.279 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000111834 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.378282+0000", 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3086999999999998e-05 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:179 243.1d 243:bff37f92:test-rados-api-smithi083-77828-115::bar:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e665)", 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:08.379274+0000", 2024-09-10T19:05:50.280 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 41.829197321000002, 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002278169, 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 179 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.281 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379274+0000", 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379274+0000", 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379272+0000", 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.282 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379282+0000", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0114e-05 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379303+0000", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.0730999999999999e-05 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379312+0000", 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3489999999999998e-06 2024-09-10T19:05:50.283 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379398+0000", 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.6490999999999996e-05 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379588+0000", 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00018913 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:05:50.284 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.379772+0000", 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00018400099999999999 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op_commit", 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.380449+0000", 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00067696200000000003 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.380831+0000", 2024-09-10T19:05:50.285 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000382013 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.381362+0000", 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00053170500000000005 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.381446+0000", 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.3560999999999995e-05 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.286 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:08.381552+0000", 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00010646599999999999 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:181 243.10 243:08000000::::head [pgnls start_epoch 667] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e667)", 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:10.303734+0000", 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 39.904737017000002, 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0033505990000000001, 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.287 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 181 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303734+0000", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303734+0000", 2024-09-10T19:05:50.288 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303729+0000", 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303750+0000", 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.143e-05 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.289 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303790+0000", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.9813e-05 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303804+0000", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3443e-05 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303900+0000", 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.6119000000000006e-05 2024-09-10T19:05:50.290 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.303906+0000", 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.0569999999999998e-06 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.291 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.306864+0000", 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002958292 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.292 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.306916+0000", 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.1705e-05 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.293 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:10.307085+0000", 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016914299999999999 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.294 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:249 241.6s0 241:70b7fb74:test-rados-api-smithi083-77828-117::bam:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:13.336183+0000", 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 36.872288308999998, 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0012082519999999999, 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.295 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 249 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.296 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.297 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.297 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336183+0000", 2024-09-10T19:05:50.297 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.297 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.297 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.298 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.298 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336183+0000", 2024-09-10T19:05:50.298 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.298 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.298 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336181+0000", 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.299 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.300 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336189+0000", 2024-09-10T19:05:50.300 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.5760000000000006e-06 2024-09-10T19:05:50.300 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.300 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.300 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336205+0000", 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5048999999999999e-05 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.301 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336210+0000", 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.3009999999999997e-06 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336265+0000", 2024-09-10T19:05:50.302 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.5313999999999999e-05 2024-09-10T19:05:50.303 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.303 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.303 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.303 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336356+0000", 2024-09-10T19:05:50.303 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.0838e-05 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.336664+0000", 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030802 2024-09-10T19:05:50.304 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.305 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.305 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.305 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.337038+0000", 2024-09-10T19:05:50.305 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00037424499999999999 2024-09-10T19:05:50.305 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.337350+0000", 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031193799999999999 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.306 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:13.337391+0000", 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.1238000000000001e-05 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.307 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDECSubOpWrite(241.7s2 671/653 ECSubWrite(tid=752, reqid=client.49694.0:252, at_version=671'3, trim_to=0'0, roll_forward_to=670'1))", 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:14.337647+0000", 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 35.870824730999999, 2024-09-10T19:05:50.308 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000959599, 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "reached pg", 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.309 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337647+0000", 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337647+0000", 2024-09-10T19:05:50.310 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337646+0000", 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.311 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337681+0000", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.4703000000000003e-05 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337697+0000", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6116e-05 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337708+0000", 2024-09-10T19:05:50.312 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0484e-05 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337796+0000", 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.8382999999999997e-05 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.337810+0000", 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4324999999999999e-05 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.338571+0000", 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00076061200000000003 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.338606+0000", 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.5371999999999999e-05 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.314 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:254 241.6s0 241:70b7fb74:test-rados-api-smithi083-77828-117::bam:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:14.343188+0000", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 35.865283300000002, 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0024620520000000002, 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 254 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.315 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343188+0000", 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343188+0000", 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343184+0000", 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.316 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343199+0000", 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4725999999999999e-05 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343228+0000", 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.8739e-05 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.317 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343238+0000", 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0869999999999999e-05 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343382+0000", 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000143683 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.343560+0000", 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00017744000000000001 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.318 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_started", 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.344349+0000", 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00078959600000000005 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "sub_op_committed", 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.345075+0000", 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00072574100000000002 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "commit_sent", 2024-09-10T19:05:50.319 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.345580+0000", 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00050467199999999995 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:14.345650+0000", 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.0586999999999993e-05 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.320 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDScrubReserve(2.8 REQUEST e672)", 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:15.600033+0000", 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 34.608438044000003, 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0038679859999999999, 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600033+0000", 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.321 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600033+0000", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600033+0000", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999995 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600039+0000", 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6960000000000002e-06 2024-09-10T19:05:50.322 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600066+0000", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.7577e-05 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600078+0000", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1525999999999999e-05 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600219+0000", 2024-09-10T19:05:50.323 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000141643 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.600237+0000", 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.7232999999999999e-05 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.603822+0000", 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0035850130000000002 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.324 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.603845+0000", 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.302e-05 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:15.603901+0000", 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.6867999999999998e-05 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(osd.2.19:114 241.6s0 241:70b7fb74:test-rados-api-smithi083-77828-117::bam:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e674)", 2024-09-10T19:05:50.325 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:17.329798+0000", 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 32.878673182999997, 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0039979289999999999, 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329798+0000", 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.326 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329798+0000", 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329794+0000", 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329810+0000", 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5526e-05 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.327 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329861+0000", 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.1613000000000002e-05 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.329875+0000", 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3334e-05 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.328 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330035+0000", 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016034799999999999 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330045+0000", 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0604e-05 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.333526+0000", 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0034807789999999998 2024-09-10T19:05:50.329 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.333660+0000", 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013378500000000001 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.333796+0000", 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000136193 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.330 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(osd.2.19:115 241.6s0 241:70b7fb74:test-rados-api-smithi083-77828-117::bam:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e674)", 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:17.330225+0000", 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 32.878246644000001, 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0048317109999999998, 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330225+0000", 2024-09-10T19:05:50.331 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330225+0000", 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330223+0000", 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999986 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.332 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330234+0000", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0443e-05 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330249+0000", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5237e-05 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330258+0000", 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2709999999999998e-06 2024-09-10T19:05:50.333 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330337+0000", 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.8677999999999994e-05 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting_for_map not empty", 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.330377+0000", 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.0608999999999997e-05 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.334 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.333826+0000", 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0034486880000000001 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.333927+0000", 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000100716 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.335057+0000", 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0011296539999999999 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:402 245.7 245:e0000000:%01:::head [pgnls start_epoch 686] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e686)", 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:42.376608+0000", 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 7.83186318, 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0056238349999999998, 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 402 2024-09-10T19:05:50.336 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376608+0000", 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376608+0000", 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.337 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376605+0000", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376618+0000", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2692e-05 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376644+0000", 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6015000000000001e-05 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.338 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376658+0000", 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4156e-05 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376772+0000", 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011403 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:05:50.339 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.376783+0000", 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.131e-05 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.382073+0000", 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0052899280000000002 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.382106+0000", 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.2752000000000001e-05 2024-09-10T19:05:50.340 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:42.382232+0000", 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00012615400000000001 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:2303 4.9 4:93e5b521:::notify.7:head [watch ping cookie 94430445882480 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:46.566671+0000", 2024-09-10T19:05:50.341 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.6418000259999999, 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0016538659999999999, 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2303 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566671+0000", 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.342 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566671+0000", 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566665+0000", 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999933 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.343 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566688+0000", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3475000000000001e-05 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566758+0000", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.9772999999999998e-05 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.566781+0000", 2024-09-10T19:05:50.344 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.2991999999999999e-05 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567744+0000", 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00096310199999999997 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.568024+0000", 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00028004700000000001 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.345 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.568325+0000", 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030114699999999998 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:2304 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94430445895968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e685)", 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:46.567087+0000", 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.641384623, 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0018994249999999999, 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:05:50.346 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2304 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567087+0000", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567087+0000", 2024-09-10T19:05:50.347 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567084+0000", 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567103+0000", 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9156000000000001e-05 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.348 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567128+0000", 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5072999999999999e-05 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.349 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.567142+0000", 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3346000000000001e-05 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.568633+0000", 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0014910909999999999 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.350 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.568830+0000", 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019718200000000001 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:46.568986+0000", 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00015629000000000001 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:05:50.351 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:05:50.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:50.378 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:50.482 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:50.515 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:50.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[18629]: cluster 2024-09-10T19:05:50.296633+0000 mgr.y (mgr.24526) 653 : cluster [DBG] pgmap v1074: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[18629]: cluster 2024-09-10T19:05:50.315887+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[18629]: audit 2024-09-10T19:05:50.324397+0000 mon.a (mon.0) 3295 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]': finished 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[18629]: cluster 2024-09-10T19:05:50.324595+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[32381]: cluster 2024-09-10T19:05:50.296633+0000 mgr.y (mgr.24526) 653 : cluster [DBG] pgmap v1074: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:50.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[32381]: cluster 2024-09-10T19:05:50.315887+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:50.634 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[32381]: audit 2024-09-10T19:05:50.324397+0000 mon.a (mon.0) 3295 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]': finished 2024-09-10T19:05:50.634 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:50 smithi083 bash[32381]: cluster 2024-09-10T19:05:50.324595+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:05:50.654 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:50.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:50.758 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:50.790 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:50 smithi165 bash[22591]: cluster 2024-09-10T19:05:50.296633+0000 mgr.y (mgr.24526) 653 : cluster [DBG] pgmap v1074: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:50 smithi165 bash[22591]: cluster 2024-09-10T19:05:50.315887+0000 mon.a (mon.0) 3294 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:50 smithi165 bash[22591]: audit 2024-09-10T19:05:50.324397+0000 mon.a (mon.0) 3295 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-120", "mode": "writeback"}]': finished 2024-09-10T19:05:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:50 smithi165 bash[22591]: cluster 2024-09-10T19:05:50.324595+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:05:50.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:50.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:50.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:51.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:51.066 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:51.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:51.203 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:51.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:51.342 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:51.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:51.458 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:05:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:51.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:51 smithi165 bash[22591]: audit 2024-09-10T19:05:50.440562+0000 mon.c (mon.2) 734 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.458 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:51 smithi165 bash[22591]: audit 2024-09-10T19:05:50.441039+0000 mon.a (mon.0) 3297 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:51.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.531+0000 7fcca6107640 1 -- 172.21.15.83:0/1911657077 >> v1:172.21.15.83:6790/0 conn(0x7fcca00fb320 legacy=0x7fcca00fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:51.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.531+0000 7fcca6107640 1 -- 172.21.15.83:0/1911657077 shutdown_connections 2024-09-10T19:05:51.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.531+0000 7fcca6107640 1 -- 172.21.15.83:0/1911657077 >> 172.21.15.83:0/1911657077 conn(0x7fcca00f7070 msgr2=0x7fcca00f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:51.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.531+0000 7fcca6107640 1 -- 172.21.15.83:0/1911657077 shutdown_connections 2024-09-10T19:05:51.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.531+0000 7fcca6107640 1 -- 172.21.15.83:0/1911657077 wait complete. 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 librados: starting msgr at 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 librados: starting objecter 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 Processor -- start 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- start start 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 librados: setting wanted keys 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 librados: calling monclient init 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca0122840 con 0x7fcca00fbe00 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca0123a40 con 0x7fcca00fb320 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca0124c40 con 0x7fcca010b150 2024-09-10T19:05:51.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc9ffff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fcca010b150 0x7fcca0120f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38176/0 (socket says 172.21.15.83:38176) 2024-09-10T19:05:51.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc9ffff640 1 -- 172.21.15.83:0/2178677863 learned_addr learned my addr 172.21.15.83:0/2178677863 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:51.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter ms_handle_connect 0x7fcca010b150 2024-09-10T19:05:51.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:51.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter ms_handle_connect 0x7fcca00fbe00 2024-09-10T19:05:51.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter ms_handle_connect 0x7fcca00fb320 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996168170 0 0) 0x7fcca0124c40 con 0x7fcca010b150 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc88003720 con 0x7fcca010b150 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46829791 0 0) 0x7fcca0122840 con 0x7fcca00fbe00 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca0124c40 con 0x7fcca00fbe00 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444316583 0 0) 0x7fcca0123a40 con 0x7fcca00fb320 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca0122840 con 0x7fcca00fb320 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3385271123 0 0) 0x7fcc88003720 con 0x7fcca010b150 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcca0123a40 con 0x7fcca010b150 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc98001bb0 con 0x7fcca010b150 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2123267152 0 0) 0x7fcca0124c40 con 0x7fcca00fbe00 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc88003720 con 0x7fcca00fbe00 2024-09-10T19:05:51.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc94001950 con 0x7fcca00fbe00 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1676751286 0 0) 0x7fcca0123a40 con 0x7fcca010b150 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 >> v1:172.21.15.165:6789/0 conn(0x7fcca00fb320 legacy=0x7fcca01031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 >> v1:172.21.15.83:6789/0 conn(0x7fcca00fbe00 legacy=0x7fcca01038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0125e40 con 0x7fcca010b150 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca0122a30 con 0x7fcca010b150 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 10 client.49739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca0122e80 con 0x7fcca010b150 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcca6107640 1 librados: init done 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fcc98002640 con 0x7fcca010b150 2024-09-10T19:05:51.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.535+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fcc98005d20 con 0x7fcca010b150 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8c005010 con 0x7fcca010b150 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fcc980021e0 con 0x7fcca010b150 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(695..695 src has 569..695) v4 ==== 8475+0+0 (unknown 1183564890 0 0) 0x7fcc9808aa80 con 0x7fcca010b150 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 10 client.49739.objecter ms_dispatch 0x7fcca0000b90 osd_map(695..695 src has 569..695) v4 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 3 client.49739.objecter handle_osd_map got epochs [695,695] > 0 2024-09-10T19:05:51.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 3 client.49739.objecter handle_osd_map decoding full epoch 695 2024-09-10T19:05:51.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 20 client.49739.objecter dump_active .. 0 homeless 2024-09-10T19:05:51.545 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.539+0000 7fcc87fff640 10 client.49739.objecter ms_handle_connect 0x7fcc88071720 2024-09-10T19:05:51.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.543+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fcc98056e80 con 0x7fcca010b150 2024-09-10T19:05:51.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:51.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:51.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:51.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:51.731+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fcc8c0052c0 con 0x7fcca010b150 2024-09-10T19:05:51.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:51.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:51 smithi083 bash[18629]: audit 2024-09-10T19:05:50.440562+0000 mon.c (mon.2) 734 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:51 smithi083 bash[18629]: audit 2024-09-10T19:05:50.441039+0000 mon.a (mon.0) 3297 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:51 smithi083 bash[32381]: audit 2024-09-10T19:05:50.440562+0000 mon.c (mon.2) 734 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:51 smithi083 bash[32381]: audit 2024-09-10T19:05:50.441039+0000 mon.a (mon.0) 3297 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:51.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:51.883 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:51.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:51.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:52.020 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:52.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:52.157 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:52.260 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:52.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:52.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:52.431 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:52.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.439+0000 7fcc87fff640 1 -- 172.21.15.83:0/2178677863 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v696) v1 ==== 113+0+0 (unknown 3424379364 0 0) 0x7fcc98056820 con 0x7fcca010b150 2024-09-10T19:05:52.443 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T19:05:52.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 10 librados: watch_flush enter 2024-09-10T19:05:52.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 10 librados: watch_flush exit 2024-09-10T19:05:52.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 20 client.49739.objecter shutdown clearing up homeless session... 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 10 client.49739.objecter successfully canceled tick 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 >> v1:172.21.15.83:6800/3071793735 conn(0x7fcc88071720 legacy=0x7fcc88073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 >> v1:172.21.15.83:6790/0 conn(0x7fcca010b150 legacy=0x7fcca0120f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 shutdown_connections 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 >> 172.21.15.83:0/2178677863 conn(0x7fcca00f7070 msgr2=0x7fcca010d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 shutdown_connections 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 -- 172.21.15.83:0/2178677863 wait complete. 2024-09-10T19:05:52.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:52.443+0000 7fcca6107640 1 librados: shutdown 2024-09-10T19:05:52.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:52.567 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:52.671 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:52.699 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: audit 2024-09-10T19:05:51.437874+0000 mon.a (mon.0) 3298 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: cluster 2024-09-10T19:05:51.437998+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: audit 2024-09-10T19:05:51.441819+0000 mon.c (mon.2) 735 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: audit 2024-09-10T19:05:51.443150+0000 mon.a (mon.0) 3300 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: audit 2024-09-10T19:05:51.737165+0000 mon.c (mon.2) 736 : audit [INF] from='client.? v1:172.21.15.83:0/2178677863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: audit 2024-09-10T19:05:51.737859+0000 mon.a (mon.0) 3301 : audit [INF] from='client.49739 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18629]: cluster 2024-09-10T19:05:52.297448+0000 mgr.y (mgr.24526) 654 : cluster [DBG] pgmap v1077: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:05:52.700 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: audit 2024-09-10T19:05:51.437874+0000 mon.a (mon.0) 3298 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: cluster 2024-09-10T19:05:51.437998+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: audit 2024-09-10T19:05:51.441819+0000 mon.c (mon.2) 735 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: audit 2024-09-10T19:05:51.443150+0000 mon.a (mon.0) 3300 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: audit 2024-09-10T19:05:51.737165+0000 mon.c (mon.2) 736 : audit [INF] from='client.? v1:172.21.15.83:0/2178677863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: audit 2024-09-10T19:05:51.737859+0000 mon.a (mon.0) 3301 : audit [INF] from='client.49739 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[32381]: cluster 2024-09-10T19:05:52.297448+0000 mgr.y (mgr.24526) 654 : cluster [DBG] pgmap v1077: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:05:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:52.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:52.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: audit 2024-09-10T19:05:51.437874+0000 mon.a (mon.0) 3298 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:05:52.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: cluster 2024-09-10T19:05:51.437998+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:05:52.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: audit 2024-09-10T19:05:51.441819+0000 mon.c (mon.2) 735 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.820 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: audit 2024-09-10T19:05:51.443150+0000 mon.a (mon.0) 3300 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]: dispatch 2024-09-10T19:05:52.820 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: audit 2024-09-10T19:05:51.737165+0000 mon.c (mon.2) 736 : audit [INF] from='client.? v1:172.21.15.83:0/2178677863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.820 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: audit 2024-09-10T19:05:51.737859+0000 mon.a (mon.0) 3301 : audit [INF] from='client.49739 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:05:52.820 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:52 smithi165 bash[22591]: cluster 2024-09-10T19:05:52.297448+0000 mgr.y (mgr.24526) 654 : cluster [DBG] pgmap v1077: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:05:52.840 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:52.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:52.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:52.844 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:05:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:05:52] "GET /metrics HTTP/1.1" 200 229032 "" "Prometheus/2.43.0" 2024-09-10T19:05:52.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:52.976 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:53.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:05:53.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:53.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:53.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:53.275 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:53.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:53.410 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:53.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:53.542 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:53.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:53.679 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:53.783 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[18629]: cluster 2024-09-10T19:05:52.433019+0000 mon.a (mon.0) 3302 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[18629]: audit 2024-09-10T19:05:52.442335+0000 mon.a (mon.0) 3303 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[18629]: audit 2024-09-10T19:05:52.442531+0000 mon.a (mon.0) 3304 : audit [INF] from='client.49739 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:05:53.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[18629]: cluster 2024-09-10T19:05:52.442695+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:05:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[32381]: cluster 2024-09-10T19:05:52.433019+0000 mon.a (mon.0) 3302 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[32381]: audit 2024-09-10T19:05:52.442335+0000 mon.a (mon.0) 3303 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[32381]: audit 2024-09-10T19:05:52.442531+0000 mon.a (mon.0) 3304 : audit [INF] from='client.49739 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:05:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:53 smithi083 bash[32381]: cluster 2024-09-10T19:05:52.442695+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:05:53.813 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:53 smithi165 bash[22591]: cluster 2024-09-10T19:05:52.433019+0000 mon.a (mon.0) 3302 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:53 smithi165 bash[22591]: audit 2024-09-10T19:05:52.442335+0000 mon.a (mon.0) 3303 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-120"}]': finished 2024-09-10T19:05:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:53 smithi165 bash[22591]: audit 2024-09-10T19:05:52.442531+0000 mon.a (mon.0) 3304 : audit [INF] from='client.49739 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:05:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:53 smithi165 bash[22591]: cluster 2024-09-10T19:05:52.442695+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:05:53.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:53.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:53.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:54.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:54.076 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:54.179 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.212 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:54.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:54.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:54.473 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:54.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:54.610 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[32381]: cluster 2024-09-10T19:05:53.460636+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[32381]: cluster 2024-09-10T19:05:54.298282+0000 mgr.y (mgr.24526) 655 : cluster [DBG] pgmap v1080: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[32381]: cluster 2024-09-10T19:05:54.313409+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[32381]: audit 2024-09-10T19:05:54.315724+0000 mon.c (mon.2) 737 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[32381]: audit 2024-09-10T19:05:54.316291+0000 mon.a (mon.0) 3308 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.685 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[18629]: cluster 2024-09-10T19:05:53.460636+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:05:54.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[18629]: cluster 2024-09-10T19:05:54.298282+0000 mgr.y (mgr.24526) 655 : cluster [DBG] pgmap v1080: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:54.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[18629]: cluster 2024-09-10T19:05:54.313409+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:05:54.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[18629]: audit 2024-09-10T19:05:54.315724+0000 mon.c (mon.2) 737 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:54 smithi083 bash[18629]: audit 2024-09-10T19:05:54.316291+0000 mon.a (mon.0) 3308 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T19:05:54.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.753 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3069508758 >> v1:172.21.15.83:6790/0 conn(0x7f59481042a0 legacy=0x7f5948106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3069508758 shutdown_connections 2024-09-10T19:05:54.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3069508758 >> 172.21.15.83:0/3069508758 conn(0x7f59480f7070 msgr2=0x7f59480f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3069508758 shutdown_connections 2024-09-10T19:05:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3069508758 wait complete. 2024-09-10T19:05:54.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 librados: starting msgr at 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 librados: starting objecter 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 Processor -- start 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- start start 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 librados: setting wanted keys 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 librados: calling monclient init 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f594811e260 con 0x7f59481042a0 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f594811f420 con 0x7f5948101530 2024-09-10T19:05:54.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5948120620 con 0x7f5948107e00 2024-09-10T19:05:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:54.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f5946d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5948107e00 0x7f594811c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38724/0 (socket says 172.21.15.83:38724) 2024-09-10T19:05:54.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f5946d9d640 1 -- 172.21.15.83:0/3822153721 learned_addr learned my addr 172.21.15.83:0/3822153721 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:54.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter ms_handle_connect 0x7f5948107e00 2024-09-10T19:05:54.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:54.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter ms_handle_connect 0x7f59481042a0 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484242336 0 0) 0x7f5948120620 con 0x7f5948107e00 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5934003720 con 0x7f5948107e00 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter ms_handle_connect 0x7f5948101530 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641859375 0 0) 0x7f594811e260 con 0x7f59481042a0 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948120620 con 0x7f59481042a0 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813321991 0 0) 0x7f594811f420 con 0x7f5948101530 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594811e260 con 0x7f5948101530 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 904097571 0 0) 0x7f5934003720 con 0x7f5948107e00 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594811f420 con 0x7f5948107e00 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5940001ca0 con 0x7f5948107e00 2024-09-10T19:05:54.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4198710052 0 0) 0x7f594811f420 con 0x7f5948107e00 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 >> v1:172.21.15.165:6789/0 conn(0x7f5948101530 legacy=0x7f5948065910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 >> v1:172.21.15.83:6789/0 conn(0x7f59481042a0 legacy=0x7f5948066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5948121820 con 0x7f5948107e00 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594811e450 con 0x7f5948107e00 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 10 client.49745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f594811e8a0 con 0x7f5948107e00 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.751+0000 7f594cf92640 1 librados: init done 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f59400027a0 con 0x7f5948107e00 2024-09-10T19:05:54.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5940005e00 con 0x7f5948107e00 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f594001c240 con 0x7f5948107e00 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928005040 con 0x7f5948107e00 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(698..698 src has 569..698) v4 ==== 8467+0+0 (unknown 3526396698 0 0) 0x7f594008b8a0 con 0x7f5948107e00 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 10 client.49745.objecter ms_dispatch 0x7f5948000b90 osd_map(698..698 src has 569..698) v4 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 3 client.49745.objecter handle_osd_map got epochs [698,698] > 0 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 3 client.49745.objecter handle_osd_map decoding full epoch 698 2024-09-10T19:05:54.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 20 client.49745.objecter dump_active .. 0 homeless 2024-09-10T19:05:54.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.755+0000 7f593effd640 10 client.49745.objecter ms_handle_connect 0x7f59340715f0 2024-09-10T19:05:54.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.759+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5940057810 con 0x7f5948107e00 2024-09-10T19:05:54.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:54 smithi165 bash[22591]: cluster 2024-09-10T19:05:53.460636+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:05:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:54 smithi165 bash[22591]: cluster 2024-09-10T19:05:54.298282+0000 mgr.y (mgr.24526) 655 : cluster [DBG] pgmap v1080: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:05:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:54 smithi165 bash[22591]: cluster 2024-09-10T19:05:54.313409+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:05:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:54 smithi165 bash[22591]: audit 2024-09-10T19:05:54.315724+0000 mon.c (mon.2) 737 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:54 smithi165 bash[22591]: audit 2024-09-10T19:05:54.316291+0000 mon.a (mon.0) 3308 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.889 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:54.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:54.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:54.951+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f59280052f0 con 0x7f5948107e00 2024-09-10T19:05:54.993 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:55.028 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:55.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.169 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:55.272 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:55.306 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:55.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.319+0000 7f593effd640 1 -- 172.21.15.83:0/3822153721 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v699) v1 ==== 97+0+0 (unknown 2922990973 0 0) 0x7f5940057630 con 0x7f5948107e00 2024-09-10T19:05:55.330 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:05:55.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 10 librados: watch_flush enter 2024-09-10T19:05:55.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 10 librados: watch_flush exit 2024-09-10T19:05:55.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 20 client.49745.objecter shutdown clearing up homeless session... 2024-09-10T19:05:55.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 10 client.49745.objecter successfully canceled tick 2024-09-10T19:05:55.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 >> v1:172.21.15.83:6800/3071793735 conn(0x7f59340715f0 legacy=0x7f5934073aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 >> v1:172.21.15.83:6790/0 conn(0x7f5948107e00 legacy=0x7f594811c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 shutdown_connections 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.335+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 >> 172.21.15.83:0/3822153721 conn(0x7f59480f7070 msgr2=0x7f59480fa550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.339+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 shutdown_connections 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.339+0000 7f594cf92640 1 -- 172.21.15.83:0/3822153721 wait complete. 2024-09-10T19:05:55.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:55.339+0000 7f594cf92640 1 librados: shutdown 2024-09-10T19:05:55.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:55.445 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:54.954515+0000 mon.c (mon.2) 738 : audit [INF] from='client.? v1:172.21.15.83:0/3822153721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:54.955029+0000 mon.a (mon.0) 3309 : audit [INF] from='client.49745 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: cluster 2024-09-10T19:05:55.309051+0000 mon.a (mon.0) 3310 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:55.318360+0000 mon.a (mon.0) 3311 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:55.318881+0000 mon.a (mon.0) 3312 : audit [INF] from='client.49745 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: cluster 2024-09-10T19:05:55.318972+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:05:55.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:55.345830+0000 mon.c (mon.2) 739 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[18629]: audit 2024-09-10T19:05:55.346488+0000 mon.a (mon.0) 3314 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:54.954515+0000 mon.c (mon.2) 738 : audit [INF] from='client.? v1:172.21.15.83:0/3822153721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:54.955029+0000 mon.a (mon.0) 3309 : audit [INF] from='client.49745 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: cluster 2024-09-10T19:05:55.309051+0000 mon.a (mon.0) 3310 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:55.318360+0000 mon.a (mon.0) 3311 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:55.318881+0000 mon.a (mon.0) 3312 : audit [INF] from='client.49745 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: cluster 2024-09-10T19:05:55.318972+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:55.345830+0000 mon.c (mon.2) 739 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:55 smithi083 bash[32381]: audit 2024-09-10T19:05:55.346488+0000 mon.a (mon.0) 3314 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:55.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:55.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:54.954515+0000 mon.c (mon.2) 738 : audit [INF] from='client.? v1:172.21.15.83:0/3822153721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:54.955029+0000 mon.a (mon.0) 3309 : audit [INF] from='client.49745 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: cluster 2024-09-10T19:05:55.309051+0000 mon.a (mon.0) 3310 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:55.318360+0000 mon.a (mon.0) 3311 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:55.318881+0000 mon.a (mon.0) 3312 : audit [INF] from='client.49745 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: cluster 2024-09-10T19:05:55.318972+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:05:55.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:55.345830+0000 mon.c (mon.2) 739 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:55 smithi165 bash[22591]: audit 2024-09-10T19:05:55.346488+0000 mon.a (mon.0) 3314 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.725 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:55.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.858 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:55.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:55.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:55.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:56.098 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:56.126 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:56.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:56.266 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:56.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:56.403 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:05:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:05:56.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:56.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:56.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:56.669 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:56.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:56.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:56.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:56.947 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:56.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:57.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:57.084 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:57.188 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:57.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:57.325 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:57.349 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/2577558953 >> v1:172.21.15.83:6789/0 conn(0x7fa28410b150 legacy=0x7fa28410d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/2577558953 shutdown_connections 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/2577558953 >> 172.21.15.83:0/2577558953 conn(0x7fa2840f70d0 msgr2=0x7fa2840f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/2577558953 shutdown_connections 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/2577558953 wait complete. 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 librados: starting msgr at 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 librados: starting objecter 2024-09-10T19:05:57.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 Processor -- start 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- start start 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 librados: setting wanted keys 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 librados: calling monclient init 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa28411e1b0 con 0x7fa2840fbe00 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa28411f370 con 0x7fa2840fb320 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa284120570 con 0x7fa28410b150 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa289271640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fa28410b150 0x7fa28411c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:38738/0 (socket says 172.21.15.83:38738) 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa289271640 1 -- 172.21.15.83:0/1914356137 learned_addr learned my addr 172.21.15.83:0/1914356137 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa283fff640 1 --1- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6789/0 conn(0x7fa2840fbe00 0x7fa284107c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50098/0 (socket says 172.21.15.83:50098) 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter ms_handle_connect 0x7fa2840fbe00 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter ms_handle_connect 0x7fa28410b150 2024-09-10T19:05:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter ms_handle_connect 0x7fa2840fb320 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913313500 0 0) 0x7fa28411e1b0 con 0x7fa2840fbe00 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2740038e0 con 0x7fa2840fbe00 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984422671 0 0) 0x7fa284120570 con 0x7fa28410b150 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa28411e1b0 con 0x7fa28410b150 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399871906 0 0) 0x7fa28411f370 con 0x7fa2840fb320 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa284120570 con 0x7fa2840fb320 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184873003 0 0) 0x7fa28411e1b0 con 0x7fa28410b150 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2740041e0 con 0x7fa28410b150 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978424072 0 0) 0x7fa2740038e0 con 0x7fa2840fbe00 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa28411e1b0 con 0x7fa2840fbe00 2024-09-10T19:05:57.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa27c003070 con 0x7fa28410b150 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa270001a80 con 0x7fa2840fbe00 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2541149431 0 0) 0x7fa2740041e0 con 0x7fa28410b150 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.165:6789/0 conn(0x7fa2840fb320 legacy=0x7fa284107560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6789/0 conn(0x7fa2840fbe00 legacy=0x7fa284107c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa284121770 con 0x7fa28410b150 2024-09-10T19:05:57.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa284120760 con 0x7fa28410b150 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 10 client.49751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa284120c30 con 0x7fa28410b150 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.355+0000 7fa28acd4640 1 librados: init done 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa27c001bf0 con 0x7fa28410b150 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa27c003990 con 0x7fa28410b150 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa268004ff0 con 0x7fa28410b150 2024-09-10T19:05:57.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa27c002620 con 0x7fa28410b150 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(701..701 src has 569..701) v4 ==== 8475+0+0 (unknown 1631042639 0 0) 0x7fa27c08abd0 con 0x7fa28410b150 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 osd_map(701..701 src has 569..701) v4 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 3 client.49751.objecter handle_osd_map got epochs [701,701] > 0 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 3 client.49751.objecter handle_osd_map decoding full epoch 701 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 20 client.49751.objecter dump_active .. 0 homeless 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v701) v1 ==== 54+0+16 (unknown 171647306 0 2420000192) 0x7fa27c08af30 con 0x7fa28410b150 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa274071650 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 10 client.49751.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 1 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 1 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 0 1 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 4 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 10 client.49751.objecter _send_command 1 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.359+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa268004a60 con 0x7fa268004610 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268004610 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa268004a60 con 0x7fa268004610 2024-09-10T19:05:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:05:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 10 client.49751.objecter _finish_command 1 = system:0 2024-09-10T19:05:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 4 2024-09-10T19:05:57.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.363+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 0 1 2024-09-10T19:05:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 10 client.49751.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 2 2024-09-10T19:05:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 2 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268005290 osd=0 3 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 0 2 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 4 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 10 client.49751.objecter _send_command 2 2024-09-10T19:05:57.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.379+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa268004e40 con 0x7fa268004610 2024-09-10T19:05:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa268004e40 con 0x7fa268004610 2024-09-10T19:05:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 10 client.49751.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 4 2024-09-10T19:05:57.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 0 2 2024-09-10T19:05:57.387 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:05:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:05:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 10 client.49751.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 1 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 4 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 10 client.49751.objecter _send_command 3 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2680022f0 con 0x7fa268004110 2024-09-10T19:05:57.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268004110 2024-09-10T19:05:57.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa2680022f0 con 0x7fa268004110 2024-09-10T19:05:57.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:05:57.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 10 client.49751.objecter _finish_command 3 = system:0 2024-09-10T19:05:57.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 4 2024-09-10T19:05:57.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.383+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 1 3 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 10 client.49751.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 4 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 4 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268003fa0 osd=1 3 2024-09-10T19:05:57.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 1 4 2024-09-10T19:05:57.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 4 2024-09-10T19:05:57.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 10 client.49751.objecter _send_command 4 2024-09-10T19:05:57.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2680018f0 con 0x7fa268004110 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa2680018f0 con 0x7fa268004110 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 10 client.49751.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 4 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 1 4 2024-09-10T19:05:57.405 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 10 client.49751.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 5 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 5 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 2 5 2024-09-10T19:05:57.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 4 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 10 client.49751.objecter _send_command 5 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa268003e30 con 0x7fa26800d350 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.399+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa26800d350 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.403+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa268003e30 con 0x7fa26800d350 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.403+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.403+0000 7fa2817fa640 10 client.49751.objecter _finish_command 5 = system:0 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.403+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 4 2024-09-10T19:05:57.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.403+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 2 5 2024-09-10T19:05:57.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 10 client.49751.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 6 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 6 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa2680035e0 osd=2 3 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 2 6 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 4 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 10 client.49751.objecter _send_command 6 2024-09-10T19:05:57.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.419+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa268003060 con 0x7fa26800d350 2024-09-10T19:05:57.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa268003060 con 0x7fa26800d350 2024-09-10T19:05:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 10 client.49751.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 4 2024-09-10T19:05:57.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 2 6 2024-09-10T19:05:57.427 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 10 client.49751.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 7 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 7 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 3 7 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 4 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 10 client.49751.objecter _send_command 7 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa26800d010 con 0x7fa268010e80 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268010e80 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa26800d010 con 0x7fa268010e80 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 10 client.49751.objecter _finish_command 7 = system:0 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 4 2024-09-10T19:05:57.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.423+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 3 7 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 10 client.49751.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 8 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 8 2024-09-10T19:05:57.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c990 osd=3 3 2024-09-10T19:05:57.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 3 8 2024-09-10T19:05:57.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 4 2024-09-10T19:05:57.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 10 client.49751.objecter _send_command 8 2024-09-10T19:05:57.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.443+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2680132e0 con 0x7fa268010e80 2024-09-10T19:05:57.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:57.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa2680132e0 con 0x7fa268010e80 2024-09-10T19:05:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 10 client.49751.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 4 2024-09-10T19:05:57.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 3 8 2024-09-10T19:05:57.453 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 10 client.49751.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 9 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 9 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 4 9 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 4 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 10 client.49751.objecter _send_command 9 2024-09-10T19:05:57.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa26801a290 con 0x7fa268016790 2024-09-10T19:05:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.447+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268016790 2024-09-10T19:05:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.451+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa278017e60 con 0x7fa268016790 2024-09-10T19:05:57.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.451+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:05:57.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.451+0000 7fa2817fa640 10 client.49751.objecter _finish_command 9 = system:0 2024-09-10T19:05:57.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.451+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 4 2024-09-10T19:05:57.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.451+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 4 9 2024-09-10T19:05:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 10 client.49751.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 10 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 10 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa268012b20 osd=4 3 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 4 10 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 4 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 10 client.49751.objecter _send_command 10 2024-09-10T19:05:57.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa26800c410 con 0x7fa268016790 2024-09-10T19:05:57.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa26800c410 con 0x7fa268016790 2024-09-10T19:05:57.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa2817fa640 10 client.49751.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 4 2024-09-10T19:05:57.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 4 10 2024-09-10T19:05:57.478 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:05:57.478 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 10 client.49751.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 11 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 11 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 5 11 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 4 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 10 client.49751.objecter _send_command 11 2024-09-10T19:05:57.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.471+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa26800c340 con 0x7fa268012670 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268012670 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa26800c340 con 0x7fa268012670 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 10 client.49751.objecter _finish_command 11 = system:0 2024-09-10T19:05:57.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 4 2024-09-10T19:05:57.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.475+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 5 11 2024-09-10T19:05:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 10 client.49751.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 12 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 12 2024-09-10T19:05:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c1d0 osd=5 3 2024-09-10T19:05:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 5 12 2024-09-10T19:05:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 4 2024-09-10T19:05:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 10 client.49751.objecter _send_command 12 2024-09-10T19:05:57.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.495+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa26800c340 con 0x7fa268012670 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa26800c340 con 0x7fa268012670 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 10 client.49751.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 4 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 5 12 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:05:57.504 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 10 client.49751.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 13 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 13 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 6 13 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 4 2024-09-10T19:05:57.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 10 client.49751.objecter _send_command 13 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa268022e30 con 0x7fa26801f1c0 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.499+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa26801f1c0 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.503+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa268022e30 con 0x7fa26801f1c0 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.503+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.503+0000 7fa2817fa640 10 client.49751.objecter _finish_command 13 = system:0 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.503+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 4 2024-09-10T19:05:57.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.503+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 6 13 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 10 client.49751.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 14 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 14 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa26800c340 osd=6 3 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 6 14 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 4 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 10 client.49751.objecter _send_command 14 2024-09-10T19:05:57.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.515+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2680246d0 con 0x7fa26801f1c0 2024-09-10T19:05:57.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa2680246d0 con 0x7fa26801f1c0 2024-09-10T19:05:57.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 10 client.49751.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 4 2024-09-10T19:05:57.525 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 6 14 2024-09-10T19:05:57.525 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 10 client.49751.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 15 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 15 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 7 15 2024-09-10T19:05:57.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 4 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 10 client.49751.objecter _send_command 15 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa268025250 con 0x7fa268024d80 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.519+0000 7fa2817fa640 10 client.49751.objecter ms_handle_connect 0x7fa268024d80 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.523+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fa278017e60 con 0x7fa268024d80 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.523+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:05:57.527 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.523+0000 7fa2817fa640 10 client.49751.objecter _finish_command 15 = system:0 2024-09-10T19:05:57.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.523+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 4 2024-09-10T19:05:57.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.523+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 7 15 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.539+0000 7fa28acd4640 10 client.49751.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign -1 16 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 20 client.49751.objecter _get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_remove -1 16 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 20 client.49751.objecter get_session s=0x7fa2680246d0 osd=7 3 2024-09-10T19:05:57.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 15 client.49751.objecter _session_command_op_assign 7 16 2024-09-10T19:05:57.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 4 2024-09-10T19:05:57.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 10 client.49751.objecter _send_command 16 2024-09-10T19:05:57.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa268025440 con 0x7fa268024d80 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa2817fa640 1 -- 172.21.15.83:0/1914356137 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7fa278012020 con 0x7fa268024d80 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa2817fa640 10 client.49751.objecter ms_dispatch 0x7fa284000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.543+0000 7fa2817fa640 10 client.49751.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa2817fa640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 4 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa2817fa640 15 client.49751.objecter _session_command_op_remove 7 16 2024-09-10T19:05:57.550 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:05:57.551 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 librados: watch_flush enter 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 librados: watch_flush exit 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.0 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6801/2535527483 conn(0x7fa268004610 legacy=0x7fa268002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268005290 osd=0 1 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.1 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6805/2309624637 conn(0x7fa268004110 legacy=0x7fa26800aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268003fa0 osd=1 1 2024-09-10T19:05:57.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.2 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6809/386094220 conn(0x7fa26800d350 legacy=0x7fa26800f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa289271640 1 -- 172.21.15.83:0/1914356137 reap_dead start 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680035e0 osd=2 1 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.3 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6813/1279688021 conn(0x7fa268010e80 legacy=0x7fa268011290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c990 osd=3 1 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.4 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.165:6800/3225515265 conn(0x7fa268016790 legacy=0x7fa268018b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa268012b20 osd=4 1 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.5 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.165:6804/571173777 conn(0x7fa268012670 legacy=0x7fa26801d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c1d0 osd=5 1 2024-09-10T19:05:57.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.6 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.165:6808/1317922936 conn(0x7fa26801f1c0 legacy=0x7fa2680215a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa26800c340 osd=6 1 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter close_session for osd.7 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.165:6812/1610151650 conn(0x7fa268024d80 legacy=0x7fa268022e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter put_session s=0x7fa2680246d0 osd=7 1 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 20 client.49751.objecter shutdown clearing up homeless session... 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 10 client.49751.objecter successfully canceled tick 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa289271640 1 -- 172.21.15.83:0/1914356137 reap_dead start 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa274071650 legacy=0x7fa274073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> v1:172.21.15.83:6790/0 conn(0x7fa28410b150 legacy=0x7fa28411c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 shutdown_connections 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 >> 172.21.15.83:0/1914356137 conn(0x7fa2840f70d0 msgr2=0x7fa2840f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:05:57.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 shutdown_connections 2024-09-10T19:05:57.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 -- 172.21.15.83:0/1914356137 wait complete. 2024-09-10T19:05:57.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:05:57.547+0000 7fa28acd4640 1 librados: shutdown 2024-09-10T19:05:57.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[18629]: cluster 2024-09-10T19:05:56.300378+0000 mgr.y (mgr.24526) 656 : cluster [DBG] pgmap v1083: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[18629]: audit 2024-09-10T19:05:56.320163+0000 mon.a (mon.0) 3315 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[18629]: cluster 2024-09-10T19:05:56.320303+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[18629]: audit 2024-09-10T19:05:56.322517+0000 mon.c (mon.2) 740 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[18629]: audit 2024-09-10T19:05:56.325073+0000 mon.a (mon.0) 3317 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[32381]: cluster 2024-09-10T19:05:56.300378+0000 mgr.y (mgr.24526) 656 : cluster [DBG] pgmap v1083: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:57.587 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[32381]: audit 2024-09-10T19:05:56.320163+0000 mon.a (mon.0) 3315 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:57.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[32381]: cluster 2024-09-10T19:05:56.320303+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:05:57.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[32381]: audit 2024-09-10T19:05:56.322517+0000 mon.c (mon.2) 740 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:57 smithi083 bash[32381]: audit 2024-09-10T19:05:56.325073+0000 mon.a (mon.0) 3317 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:57 smithi165 bash[22591]: cluster 2024-09-10T19:05:56.300378+0000 mgr.y (mgr.24526) 656 : cluster [DBG] pgmap v1083: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:05:57.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:57 smithi165 bash[22591]: audit 2024-09-10T19:05:56.320163+0000 mon.a (mon.0) 3315 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:57.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:57 smithi165 bash[22591]: cluster 2024-09-10T19:05:56.320303+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:05:57.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:57 smithi165 bash[22591]: audit 2024-09-10T19:05:56.322517+0000 mon.c (mon.2) 740 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:57 smithi165 bash[22591]: audit 2024-09-10T19:05:56.325073+0000 mon.a (mon.0) 3317 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:05:57.616 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:57.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:57.752 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:57.856 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:57.890 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:57.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:57.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:58.027 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:58.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:58.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:05:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:05:58.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:58.299 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:58.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:58.433 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: audit 2024-09-10T19:05:57.324025+0000 mon.a (mon.0) 3318 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: cluster 2024-09-10T19:05:57.324143+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: audit 2024-09-10T19:05:57.325437+0000 mon.c (mon.2) 741 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: audit 2024-09-10T19:05:57.327186+0000 mon.a (mon.0) 3320 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: audit 2024-09-10T19:05:57.364474+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? v1:172.21.15.83:0/1914356137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: cluster 2024-09-10T19:05:58.317441+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: audit 2024-09-10T19:05:58.327028+0000 mon.a (mon.0) 3322 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]': finished 2024-09-10T19:05:58.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[32381]: cluster 2024-09-10T19:05:58.327624+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: audit 2024-09-10T19:05:57.324025+0000 mon.a (mon.0) 3318 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: cluster 2024-09-10T19:05:57.324143+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: audit 2024-09-10T19:05:57.325437+0000 mon.c (mon.2) 741 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: audit 2024-09-10T19:05:57.327186+0000 mon.a (mon.0) 3320 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: audit 2024-09-10T19:05:57.364474+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? v1:172.21.15.83:0/1914356137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: cluster 2024-09-10T19:05:58.317441+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: audit 2024-09-10T19:05:58.327028+0000 mon.a (mon.0) 3322 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]': finished 2024-09-10T19:05:58.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:58 smithi083 bash[18629]: cluster 2024-09-10T19:05:58.327624+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:05:58.536 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:58.569 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:58.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:05:58.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: audit 2024-09-10T19:05:57.324025+0000 mon.a (mon.0) 3318 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:05:58.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: cluster 2024-09-10T19:05:57.324143+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: audit 2024-09-10T19:05:57.325437+0000 mon.c (mon.2) 741 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: audit 2024-09-10T19:05:57.327186+0000 mon.a (mon.0) 3320 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]: dispatch 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: audit 2024-09-10T19:05:57.364474+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? v1:172.21.15.83:0/1914356137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: cluster 2024-09-10T19:05:58.317441+0000 mon.a (mon.0) 3321 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: audit 2024-09-10T19:05:58.327028+0000 mon.a (mon.0) 3322 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-122", "mode": "writeback"}]': finished 2024-09-10T19:05:58.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:58 smithi165 bash[22591]: cluster 2024-09-10T19:05:58.327624+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:05:58.673 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:58.708 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:58.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:58.847 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:05:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:05:58.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:58.984 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:05:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:58.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:05:59.022 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:05:59.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:59.120 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:59.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:59.248 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:59.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:59.380 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:05:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:05:59.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:59.515 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:59.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:59.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[18629]: cluster 2024-09-10T19:05:58.301217+0000 mgr.y (mgr.24526) 657 : cluster [DBG] pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:59.632 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[18629]: audit 2024-09-10T19:05:58.301500+0000 mgr.y (mgr.24526) 658 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[18629]: audit 2024-09-10T19:05:58.412852+0000 mon.c (mon.2) 743 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[18629]: audit 2024-09-10T19:05:58.413413+0000 mon.a (mon.0) 3324 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[32381]: cluster 2024-09-10T19:05:58.301217+0000 mgr.y (mgr.24526) 657 : cluster [DBG] pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[32381]: audit 2024-09-10T19:05:58.301500+0000 mgr.y (mgr.24526) 658 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[32381]: audit 2024-09-10T19:05:58.412852+0000 mon.c (mon.2) 743 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.633 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:05:59 smithi083 bash[32381]: audit 2024-09-10T19:05:58.413413+0000 mon.a (mon.0) 3324 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.652 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:05:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:05:59.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:59.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:05:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:05:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:59 smithi165 bash[22591]: cluster 2024-09-10T19:05:58.301217+0000 mgr.y (mgr.24526) 657 : cluster [DBG] pgmap v1086: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:59 smithi165 bash[22591]: audit 2024-09-10T19:05:58.301500+0000 mgr.y (mgr.24526) 658 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:59 smithi165 bash[22591]: audit 2024-09-10T19:05:58.412852+0000 mon.c (mon.2) 743 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:05:59 smithi165 bash[22591]: audit 2024-09-10T19:05:58.413413+0000 mon.a (mon.0) 3324 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:05:59.896 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:59.929 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:05:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:05:59.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:05:59.952 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:59.953 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:00.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:00.065 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:00.169 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:00.203 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:00.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:00.342 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:00.446 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:00.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:06:00.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:00.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: audit 2024-09-10T19:05:59.367051+0000 mon.a (mon.0) 3325 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: cluster 2024-09-10T19:05:59.367225+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:06:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: audit 2024-09-10T19:05:59.368149+0000 mon.c (mon.2) 744 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: audit 2024-09-10T19:05:59.370112+0000 mon.a (mon.0) 3327 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: audit 2024-09-10T19:05:59.580605+0000 mon.a (mon.0) 3328 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:00.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:00.609 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: audit 2024-09-10T19:05:59.367051+0000 mon.a (mon.0) 3325 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: cluster 2024-09-10T19:05:59.367225+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:06:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: audit 2024-09-10T19:05:59.368149+0000 mon.c (mon.2) 744 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: audit 2024-09-10T19:05:59.370112+0000 mon.a (mon.0) 3327 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: audit 2024-09-10T19:05:59.580605+0000 mon.a (mon.0) 3328 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: audit 2024-09-10T19:05:59.582677+0000 mon.c (mon.2) 745 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:00.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:00 smithi165 bash[22591]: cluster 2024-09-10T19:06:00.360868+0000 mon.a (mon.0) 3329 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:00.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:00.747 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:00.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: audit 2024-09-10T19:05:59.582677+0000 mon.c (mon.2) 745 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[18629]: cluster 2024-09-10T19:06:00.360868+0000 mon.a (mon.0) 3329 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: audit 2024-09-10T19:05:59.367051+0000 mon.a (mon.0) 3325 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: cluster 2024-09-10T19:05:59.367225+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: audit 2024-09-10T19:05:59.368149+0000 mon.c (mon.2) 744 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: audit 2024-09-10T19:05:59.370112+0000 mon.a (mon.0) 3327 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]: dispatch 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: audit 2024-09-10T19:05:59.580605+0000 mon.a (mon.0) 3328 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: audit 2024-09-10T19:05:59.582677+0000 mon.c (mon.2) 745 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:00 smithi083 bash[32381]: cluster 2024-09-10T19:06:00.360868+0000 mon.a (mon.0) 3329 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:00.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:00.884 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:00.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:00.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:00.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:01.021 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:01.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:01.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:01.264 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:01.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:01.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:01.407 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:01 smithi165 bash[22591]: cluster 2024-09-10T19:06:00.302045+0000 mgr.y (mgr.24526) 659 : cluster [DBG] pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:01.407 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:01 smithi165 bash[22591]: audit 2024-09-10T19:06:00.372421+0000 mon.a (mon.0) 3330 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:06:01.407 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:01 smithi165 bash[22591]: cluster 2024-09-10T19:06:00.372580+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:06:01.407 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:01.421 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:01.524 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:01.537 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[32381]: cluster 2024-09-10T19:06:00.302045+0000 mgr.y (mgr.24526) 659 : cluster [DBG] pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:01.537 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[32381]: audit 2024-09-10T19:06:00.372421+0000 mon.a (mon.0) 3330 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:06:01.537 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[32381]: cluster 2024-09-10T19:06:00.372580+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:06:01.538 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[18629]: cluster 2024-09-10T19:06:00.302045+0000 mgr.y (mgr.24526) 659 : cluster [DBG] pgmap v1089: 214 pgs: 214 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:01.538 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[18629]: audit 2024-09-10T19:06:00.372421+0000 mon.a (mon.0) 3330 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-122"}]': finished 2024-09-10T19:06:01.538 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:01 smithi083 bash[18629]: cluster 2024-09-10T19:06:00.372580+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:06:01.559 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:01.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:01.694 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:01.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:01.818 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:01.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:01.957 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:01.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:02.060 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:02.096 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:02.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:02.233 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:02.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:02.372 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:02.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:02.489 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:02 smithi083 bash[18629]: cluster 2024-09-10T19:06:01.392362+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:06:02.490 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:02 smithi083 bash[18629]: cluster 2024-09-10T19:06:02.302999+0000 mgr.y (mgr.24526) 660 : cluster [DBG] pgmap v1092: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:02 smithi083 bash[32381]: cluster 2024-09-10T19:06:01.392362+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:06:02.490 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:02 smithi083 bash[32381]: cluster 2024-09-10T19:06:02.302999+0000 mgr.y (mgr.24526) 660 : cluster [DBG] pgmap v1092: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.504 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:02.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:02.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:02 smithi165 bash[22591]: cluster 2024-09-10T19:06:01.392362+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:06:02.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:02 smithi165 bash[22591]: cluster 2024-09-10T19:06:02.302999+0000 mgr.y (mgr.24526) 660 : cluster [DBG] pgmap v1092: 182 pgs: 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.638 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:02.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:02.778 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:02.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:02] "GET /metrics HTTP/1.1" 200 228991 "" "Prometheus/2.43.0" 2024-09-10T19:06:02.881 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:02.917 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:02.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:03.000 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:03.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:03.056 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:03.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:03.188 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:03.292 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.327 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:03.430 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:03.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[32381]: cluster 2024-09-10T19:06:02.407219+0000 mon.a (mon.0) 3333 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:06:03.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[32381]: audit 2024-09-10T19:06:02.412082+0000 mon.c (mon.2) 746 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.441 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[32381]: audit 2024-09-10T19:06:02.412678+0000 mon.a (mon.0) 3334 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:03.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:03.587 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:03.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:03.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:03 smithi165 bash[22591]: cluster 2024-09-10T19:06:02.407219+0000 mon.a (mon.0) 3333 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:06:03.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:03 smithi165 bash[22591]: audit 2024-09-10T19:06:02.412082+0000 mon.c (mon.2) 746 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.702 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:03 smithi165 bash[22591]: audit 2024-09-10T19:06:02.412678+0000 mon.a (mon.0) 3334 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:03.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[18629]: cluster 2024-09-10T19:06:02.407219+0000 mon.a (mon.0) 3333 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:06:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[18629]: audit 2024-09-10T19:06:02.412082+0000 mon.c (mon.2) 746 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:03 smithi083 bash[18629]: audit 2024-09-10T19:06:02.412678+0000 mon.a (mon.0) 3334 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:03.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.863 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:03.966 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.995 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:03.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:04.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:04.133 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:04.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:04.274 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:04.377 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:04.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:04.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:04.541 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:04.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:04.672 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:04.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: audit 2024-09-10T19:06:03.414248+0000 mon.a (mon.0) 3335 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: cluster 2024-09-10T19:06:03.414540+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: cluster 2024-09-10T19:06:04.303859+0000 mgr.y (mgr.24526) 661 : cluster [DBG] pgmap v1095: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: cluster 2024-09-10T19:06:04.320221+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: audit 2024-09-10T19:06:04.341519+0000 mon.c (mon.2) 747 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: audit 2024-09-10T19:06:04.342060+0000 mon.a (mon.0) 3338 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[32381]: audit 2024-09-10T19:06:04.382012+0000 mon.c (mon.2) 748 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: audit 2024-09-10T19:06:03.414248+0000 mon.a (mon.0) 3335 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: cluster 2024-09-10T19:06:03.414540+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: cluster 2024-09-10T19:06:04.303859+0000 mgr.y (mgr.24526) 661 : cluster [DBG] pgmap v1095: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:04.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: cluster 2024-09-10T19:06:04.320221+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:06:04.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: audit 2024-09-10T19:06:04.341519+0000 mon.c (mon.2) 747 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: audit 2024-09-10T19:06:04.342060+0000 mon.a (mon.0) 3338 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.676 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:04 smithi083 bash[18629]: audit 2024-09-10T19:06:04.382012+0000 mon.c (mon.2) 748 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:04.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:04.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: audit 2024-09-10T19:06:03.414248+0000 mon.a (mon.0) 3335 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: cluster 2024-09-10T19:06:03.414540+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: cluster 2024-09-10T19:06:04.303859+0000 mgr.y (mgr.24526) 661 : cluster [DBG] pgmap v1095: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: cluster 2024-09-10T19:06:04.320221+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: audit 2024-09-10T19:06:04.341519+0000 mon.c (mon.2) 747 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: audit 2024-09-10T19:06:04.342060+0000 mon.a (mon.0) 3338 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:04.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:04 smithi165 bash[22591]: audit 2024-09-10T19:06:04.382012+0000 mon.c (mon.2) 748 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:04.807 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:04.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:04.940 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:04.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:05.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:05.079 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:05.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:05.211 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:05.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:05.449 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:05.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:05.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:05.624 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:05.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:04.834233+0000 mon.c (mon.2) 749 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:04.835799+0000 mon.c (mon.2) 750 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:04.844110+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:05.338444+0000 mon.a (mon.0) 3340 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: cluster 2024-09-10T19:06:05.338515+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:05.339180+0000 mon.c (mon.2) 751 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.741 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[18629]: audit 2024-09-10T19:06:05.339571+0000 mon.a (mon.0) 3342 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:04.834233+0000 mon.c (mon.2) 749 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:04.835799+0000 mon.c (mon.2) 750 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:04.844110+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:05.338444+0000 mon.a (mon.0) 3340 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: cluster 2024-09-10T19:06:05.338515+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:05.339180+0000 mon.c (mon.2) 751 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.742 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:05 smithi083 bash[32381]: audit 2024-09-10T19:06:05.339571+0000 mon.a (mon.0) 3342 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.760 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:04.834233+0000 mon.c (mon.2) 749 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:04.835799+0000 mon.c (mon.2) 750 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:04.844110+0000 mon.a (mon.0) 3339 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:05.338444+0000 mon.a (mon.0) 3340 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: cluster 2024-09-10T19:06:05.338515+0000 mon.a (mon.0) 3341 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:06:05.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:05.339180+0000 mon.c (mon.2) 751 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:05 smithi165 bash[22591]: audit 2024-09-10T19:06:05.339571+0000 mon.a (mon.0) 3342 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:05.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:05.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:05.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:06.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:06.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:06.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:06.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:06.311 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:06.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:06.448 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:06.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:06.587 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:06.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.724 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:06.826 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:06.856 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:06.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.993 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:06.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:07.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:07.131 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:07.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:07.242 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:07.270 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:07.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:07.405 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:07.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:07.510 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[18629]: cluster 2024-09-10T19:06:06.306087+0000 mgr.y (mgr.24526) 662 : cluster [DBG] pgmap v1098: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[18629]: audit 2024-09-10T19:06:06.337102+0000 mon.a (mon.0) 3343 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[18629]: cluster 2024-09-10T19:06:06.337331+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[18629]: audit 2024-09-10T19:06:06.339244+0000 mon.c (mon.2) 752 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[18629]: audit 2024-09-10T19:06:06.340165+0000 mon.a (mon.0) 3345 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[32381]: cluster 2024-09-10T19:06:06.306087+0000 mgr.y (mgr.24526) 662 : cluster [DBG] pgmap v1098: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[32381]: audit 2024-09-10T19:06:06.337102+0000 mon.a (mon.0) 3343 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[32381]: cluster 2024-09-10T19:06:06.337331+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[32381]: audit 2024-09-10T19:06:06.339244+0000 mon.c (mon.2) 752 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.511 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:07 smithi083 bash[32381]: audit 2024-09-10T19:06:06.340165+0000 mon.a (mon.0) 3345 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:07 smithi165 bash[22591]: cluster 2024-09-10T19:06:06.306087+0000 mgr.y (mgr.24526) 662 : cluster [DBG] pgmap v1098: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:07 smithi165 bash[22591]: audit 2024-09-10T19:06:06.337102+0000 mon.a (mon.0) 3343 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:07 smithi165 bash[22591]: cluster 2024-09-10T19:06:06.337331+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:06:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:07 smithi165 bash[22591]: audit 2024-09-10T19:06:06.339244+0000 mon.c (mon.2) 752 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:07 smithi165 bash[22591]: audit 2024-09-10T19:06:06.340165+0000 mon.a (mon.0) 3345 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]: dispatch 2024-09-10T19:06:07.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:07.679 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:07.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:07.818 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:07.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:07.957 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:07.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:08.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:08.095 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:08.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:08.234 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:08.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:08.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:08.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:08 smithi165 bash[22591]: cluster 2024-09-10T19:06:07.330415+0000 mon.a (mon.0) 3346 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:08.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:08 smithi165 bash[22591]: audit 2024-09-10T19:06:07.342469+0000 mon.a (mon.0) 3347 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]': finished 2024-09-10T19:06:08.497 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:08 smithi165 bash[22591]: cluster 2024-09-10T19:06:07.342604+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:06:08.520 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:08.623 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[18629]: cluster 2024-09-10T19:06:07.330415+0000 mon.a (mon.0) 3346 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[18629]: audit 2024-09-10T19:06:07.342469+0000 mon.a (mon.0) 3347 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]': finished 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[18629]: cluster 2024-09-10T19:06:07.342604+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[32381]: cluster 2024-09-10T19:06:07.330415+0000 mon.a (mon.0) 3346 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[32381]: audit 2024-09-10T19:06:07.342469+0000 mon.a (mon.0) 3347 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-124", "mode": "writeback"}]': finished 2024-09-10T19:06:08.636 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:08 smithi083 bash[32381]: cluster 2024-09-10T19:06:07.342604+0000 mon.a (mon.0) 3348 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:06:08.657 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:08.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:08.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:08.796 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:08.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.934 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:08.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:09.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:09.068 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:09.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:09.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:09.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:09.336 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:09.439 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:09.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:09.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:09.590 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: cluster 2024-09-10T19:06:08.306940+0000 mgr.y (mgr.24526) 663 : cluster [DBG] pgmap v1101: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: cluster 2024-09-10T19:06:08.308451+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: cluster 2024-09-10T19:06:08.308855+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: audit 2024-09-10T19:06:08.312652+0000 mgr.y (mgr.24526) 664 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: cluster 2024-09-10T19:06:08.364106+0000 mon.a (mon.0) 3349 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: audit 2024-09-10T19:06:08.399933+0000 mon.c (mon.2) 753 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:09 smithi165 bash[22591]: audit 2024-09-10T19:06:08.400439+0000 mon.a (mon.0) 3350 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:09.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:09.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: cluster 2024-09-10T19:06:08.306940+0000 mgr.y (mgr.24526) 663 : cluster [DBG] pgmap v1101: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:09.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: cluster 2024-09-10T19:06:08.308451+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:09.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: cluster 2024-09-10T19:06:08.308855+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: audit 2024-09-10T19:06:08.312652+0000 mgr.y (mgr.24526) 664 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: cluster 2024-09-10T19:06:08.364106+0000 mon.a (mon.0) 3349 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: audit 2024-09-10T19:06:08.399933+0000 mon.c (mon.2) 753 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[32381]: audit 2024-09-10T19:06:08.400439+0000 mon.a (mon.0) 3350 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: cluster 2024-09-10T19:06:08.306940+0000 mgr.y (mgr.24526) 663 : cluster [DBG] pgmap v1101: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: cluster 2024-09-10T19:06:08.308451+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: cluster 2024-09-10T19:06:08.308855+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: audit 2024-09-10T19:06:08.312652+0000 mgr.y (mgr.24526) 664 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: cluster 2024-09-10T19:06:08.364106+0000 mon.a (mon.0) 3349 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:06:09.729 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: audit 2024-09-10T19:06:08.399933+0000 mon.c (mon.2) 753 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.730 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:09 smithi083 bash[18629]: audit 2024-09-10T19:06:08.400439+0000 mon.a (mon.0) 3350 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:09.745 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:09.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:09.882 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:09.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:09.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:10.017 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:10.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.147 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:10.250 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.287 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:10.394 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.422 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:10.525 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:10.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: audit 2024-09-10T19:06:09.377392+0000 mon.a (mon.0) 3351 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:10.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: cluster 2024-09-10T19:06:09.377480+0000 mon.a (mon.0) 3352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:06:10.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: audit 2024-09-10T19:06:09.379485+0000 mon.c (mon.2) 754 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: audit 2024-09-10T19:06:09.381447+0000 mon.a (mon.0) 3353 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: cluster 2024-09-10T19:06:09.554513+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[18629]: cluster 2024-09-10T19:06:09.554976+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: audit 2024-09-10T19:06:09.377392+0000 mon.a (mon.0) 3351 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: cluster 2024-09-10T19:06:09.377480+0000 mon.a (mon.0) 3352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: audit 2024-09-10T19:06:09.379485+0000 mon.c (mon.2) 754 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: audit 2024-09-10T19:06:09.381447+0000 mon.a (mon.0) 3353 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: cluster 2024-09-10T19:06:09.554513+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:10.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:10 smithi083 bash[32381]: cluster 2024-09-10T19:06:09.554976+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:10.560 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:10.663 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:10.702 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:10.806 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:10.838 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: audit 2024-09-10T19:06:09.377392+0000 mon.a (mon.0) 3351 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: cluster 2024-09-10T19:06:09.377480+0000 mon.a (mon.0) 3352 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:06:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: audit 2024-09-10T19:06:09.379485+0000 mon.c (mon.2) 754 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: audit 2024-09-10T19:06:09.381447+0000 mon.a (mon.0) 3353 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]: dispatch 2024-09-10T19:06:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: cluster 2024-09-10T19:06:09.554513+0000 osd.7 (osd.7) 381 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:10 smithi165 bash[22591]: cluster 2024-09-10T19:06:09.554976+0000 osd.7 (osd.7) 382 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:10.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:10.969 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:10.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:11.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.105 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:11.209 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:11.231 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:11.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.355+0000 7f86d4cfb640 1 -- 172.21.15.83:0/3980208174 >> v1:172.21.15.83:6789/0 conn(0x7f86d01042a0 legacy=0x7f86d0106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:11.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.355+0000 7f86d4cfb640 1 -- 172.21.15.83:0/3980208174 shutdown_connections 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.355+0000 7f86d4cfb640 1 -- 172.21.15.83:0/3980208174 >> 172.21.15.83:0/3980208174 conn(0x7f86d0075290 msgr2=0x7f86d00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.355+0000 7f86d4cfb640 1 -- 172.21.15.83:0/3980208174 shutdown_connections 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.355+0000 7f86d4cfb640 1 -- 172.21.15.83:0/3980208174 wait complete. 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 librados: starting msgr at 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 librados: starting objecter 2024-09-10T19:06:11.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 Processor -- start 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- start start 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 librados: setting wanted keys 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 librados: calling monclient init 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d011ae20 con 0x7f86d01042a0 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d011bfe0 con 0x7f86d0107e30 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86d011d1e0 con 0x7f86d01014e0 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86cdd9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f86d01042a0 0x7f86d00fd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55532/0 (socket says 172.21.15.83:55532) 2024-09-10T19:06:11.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86cdd9b640 1 -- 172.21.15.83:0/921995516 learned_addr learned my addr 172.21.15.83:0/921995516 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d01042a0 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d01014e0 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335909292 0 0) 0x7f86d011ae20 con 0x7f86d01042a0 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc0039c0 con 0x7f86d01042a0 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702408409 0 0) 0x7f86d011d1e0 con 0x7f86d01014e0 2024-09-10T19:06:11.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d011ae20 con 0x7f86d01014e0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter ms_handle_connect 0x7f86d0107e30 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3988568508 0 0) 0x7f86bc0039c0 con 0x7f86d01042a0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d011d1e0 con 0x7f86d01042a0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f86c8003080 con 0x7f86d01042a0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893569267 0 0) 0x7f86d011bfe0 con 0x7f86d0107e30 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc0039c0 con 0x7f86d0107e30 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904772337 0 0) 0x7f86d011ae20 con 0x7f86d01014e0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d011bfe0 con 0x7f86d01014e0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f86b80018d0 con 0x7f86d01014e0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4084449218 0 0) 0x7f86d011d1e0 con 0x7f86d01042a0 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 >> v1:172.21.15.83:6790/0 conn(0x7f86d01014e0 legacy=0x7f86d00fd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 >> v1:172.21.15.165:6789/0 conn(0x7f86d0107e30 legacy=0x7f86d0119520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:11.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d011e3e0 con 0x7f86d01042a0 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d011d3d0 con 0x7f86d01042a0 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 10 client.25671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d011d8a0 con 0x7f86d01042a0 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86d4cfb640 1 librados: init done 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f86c8002b50 con 0x7f86d01042a0 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.359+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f86c8003a40 con 0x7f86d01042a0 2024-09-10T19:06:11.367 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b0005010 con 0x7f86d01042a0 2024-09-10T19:06:11.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f86c8001c20 con 0x7f86d01042a0 2024-09-10T19:06:11.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(714..714 src has 569..714) v4 ==== 8523+0+0 (unknown 1864289857 0 0) 0x7f86c808af00 con 0x7f86d01042a0 2024-09-10T19:06:11.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 10 client.25671.objecter ms_dispatch 0x7f86d0000b90 osd_map(714..714 src has 569..714) v4 2024-09-10T19:06:11.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 3 client.25671.objecter handle_osd_map got epochs [714,714] > 0 2024-09-10T19:06:11.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 3 client.25671.objecter handle_osd_map decoding full epoch 714 2024-09-10T19:06:11.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 20 client.25671.objecter dump_active .. 0 homeless 2024-09-10T19:06:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.363+0000 7f86c6ffd640 10 client.25671.objecter ms_handle_connect 0x7f86bc0737f0 2024-09-10T19:06:11.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:11.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.371+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f86c80572b0 con 0x7f86d01042a0 2024-09-10T19:06:11.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[18629]: cluster 2024-09-10T19:06:10.307802+0000 mgr.y (mgr.24526) 665 : cluster [DBG] pgmap v1104: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[18629]: cluster 2024-09-10T19:06:10.373482+0000 mon.a (mon.0) 3354 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[18629]: audit 2024-09-10T19:06:10.382873+0000 mon.a (mon.0) 3355 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[18629]: cluster 2024-09-10T19:06:10.383051+0000 mon.a (mon.0) 3356 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[32381]: cluster 2024-09-10T19:06:10.307802+0000 mgr.y (mgr.24526) 665 : cluster [DBG] pgmap v1104: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:11.485 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[32381]: cluster 2024-09-10T19:06:10.373482+0000 mon.a (mon.0) 3354 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:11.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[32381]: audit 2024-09-10T19:06:10.382873+0000 mon.a (mon.0) 3355 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:11.486 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:11 smithi083 bash[32381]: cluster 2024-09-10T19:06:10.383051+0000 mon.a (mon.0) 3356 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:06:11.501 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:11.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:11.571+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f86b00052c0 con 0x7f86d01042a0 2024-09-10T19:06:11.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:11 smithi165 bash[22591]: cluster 2024-09-10T19:06:10.307802+0000 mgr.y (mgr.24526) 665 : cluster [DBG] pgmap v1104: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:11 smithi165 bash[22591]: cluster 2024-09-10T19:06:10.373482+0000 mon.a (mon.0) 3354 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:11 smithi165 bash[22591]: audit 2024-09-10T19:06:10.382873+0000 mon.a (mon.0) 3355 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-124"}]': finished 2024-09-10T19:06:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:11 smithi165 bash[22591]: cluster 2024-09-10T19:06:10.383051+0000 mon.a (mon.0) 3356 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:06:11.638 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:11.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:11.779 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:11.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:11.915 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:11.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:12.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:12.051 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:12.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.190 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:12.294 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.324 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:12.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.419+0000 7f86c6ffd640 1 -- 172.21.15.83:0/921995516 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v716) v1 ==== 113+0+0 (unknown 1682585808 0 0) 0x7f86c8056c50 con 0x7f86d01042a0 2024-09-10T19:06:12.425 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T19:06:12.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 10 librados: watch_flush enter 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 10 librados: watch_flush exit 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 20 client.25671.objecter shutdown clearing up homeless session... 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 10 client.25671.objecter successfully canceled tick 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 >> v1:172.21.15.83:6800/3071793735 conn(0x7f86bc0737f0 legacy=0x7f86bc075ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 >> v1:172.21.15.83:6789/0 conn(0x7f86d01042a0 legacy=0x7f86d00fd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 shutdown_connections 2024-09-10T19:06:12.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 >> 172.21.15.83:0/921995516 conn(0x7f86d0075290 msgr2=0x7f86d00f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:12.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 shutdown_connections 2024-09-10T19:06:12.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.423+0000 7f86d4cfb640 1 -- 172.21.15.83:0/921995516 wait complete. 2024-09-10T19:06:12.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:12.427+0000 7f86d4cfb640 1 librados: shutdown 2024-09-10T19:06:12.450 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:12.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:12 smithi165 bash[22591]: cluster 2024-09-10T19:06:11.403757+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:12 smithi165 bash[22591]: audit 2024-09-10T19:06:11.839039+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:12 smithi165 bash[22591]: cluster 2024-09-10T19:06:12.308605+0000 mgr.y (mgr.24526) 666 : cluster [DBG] pgmap v1107: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:12.589 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:12.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[18629]: cluster 2024-09-10T19:06:11.403757+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:06:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[18629]: audit 2024-09-10T19:06:11.839039+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:06:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[18629]: cluster 2024-09-10T19:06:12.308605+0000 mgr.y (mgr.24526) 666 : cluster [DBG] pgmap v1107: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:12.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[32381]: cluster 2024-09-10T19:06:11.403757+0000 mon.a (mon.0) 3357 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:06:12.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[32381]: audit 2024-09-10T19:06:11.839039+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:06:12.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[32381]: cluster 2024-09-10T19:06:12.308605+0000 mgr.y (mgr.24526) 666 : cluster [DBG] pgmap v1107: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:12.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:12.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:12.861 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:12.965 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:12.974 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:12] "GET /metrics HTTP/1.1" 200 228991 "" "Prometheus/2.43.0" 2024-09-10T19:06:12.990 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:12.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:13.043 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:06:13.043 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:06:13.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:13.126 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:13.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:13.266 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:13.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:13.403 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:13.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:13.540 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: audit 2024-09-10T19:06:12.423143+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: cluster 2024-09-10T19:06:12.423677+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: audit 2024-09-10T19:06:12.424965+0000 mon.c (mon.2) 755 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: audit 2024-09-10T19:06:12.426600+0000 mon.a (mon.0) 3361 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: cluster 2024-09-10T19:06:12.655223+0000 osd.1 (osd.1) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[18629]: cluster 2024-09-10T19:06:12.655627+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: audit 2024-09-10T19:06:12.423143+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: cluster 2024-09-10T19:06:12.423677+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: audit 2024-09-10T19:06:12.424965+0000 mon.c (mon.2) 755 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: audit 2024-09-10T19:06:12.426600+0000 mon.a (mon.0) 3361 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: cluster 2024-09-10T19:06:12.655223+0000 osd.1 (osd.1) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:13 smithi083 bash[32381]: cluster 2024-09-10T19:06:12.655627+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:13.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:13.658 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: audit 2024-09-10T19:06:12.423143+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.83:0/921995516' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:06:13.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: cluster 2024-09-10T19:06:12.423677+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:06:13.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: audit 2024-09-10T19:06:12.424965+0000 mon.c (mon.2) 755 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: audit 2024-09-10T19:06:12.426600+0000 mon.a (mon.0) 3361 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:13.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: cluster 2024-09-10T19:06:12.655223+0000 osd.1 (osd.1) 367 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:13.659 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:13 smithi165 bash[22591]: cluster 2024-09-10T19:06:12.655627+0000 osd.1 (osd.1) 368 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:13.674 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:13.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:13.777 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:13.810 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:13.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:13.951 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:13.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:14.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:14.092 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:14.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:14.228 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:14.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:14.361 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:14.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:14.477 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:13.420923+0000 mon.a (mon.0) 3362 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:14.477 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: cluster 2024-09-10T19:06:13.421299+0000 mon.a (mon.0) 3363 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:13.436368+0000 mon.c (mon.2) 756 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:13.438183+0000 mon.a (mon.0) 3364 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: cluster 2024-09-10T19:06:14.309418+0000 mgr.y (mgr.24526) 667 : cluster [DBG] pgmap v1110: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:14.328073+0000 mon.a (mon.0) 3365 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: cluster 2024-09-10T19:06:14.328274+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:14.328908+0000 mon.c (mon.2) 757 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:14 smithi165 bash[22591]: audit 2024-09-10T19:06:14.329365+0000 mon.a (mon.0) 3367 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:14.600 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:14.628 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:13.420923+0000 mon.a (mon.0) 3362 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: cluster 2024-09-10T19:06:13.421299+0000 mon.a (mon.0) 3363 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:13.436368+0000 mon.c (mon.2) 756 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:13.438183+0000 mon.a (mon.0) 3364 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: cluster 2024-09-10T19:06:14.309418+0000 mgr.y (mgr.24526) 667 : cluster [DBG] pgmap v1110: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:14.328073+0000 mon.a (mon.0) 3365 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: cluster 2024-09-10T19:06:14.328274+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:14.328908+0000 mon.c (mon.2) 757 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.630 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[32381]: audit 2024-09-10T19:06:14.329365+0000 mon.a (mon.0) 3367 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:13.420923+0000 mon.a (mon.0) 3362 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: cluster 2024-09-10T19:06:13.421299+0000 mon.a (mon.0) 3363 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:13.436368+0000 mon.c (mon.2) 756 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:13.438183+0000 mon.a (mon.0) 3364 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: cluster 2024-09-10T19:06:14.309418+0000 mgr.y (mgr.24526) 667 : cluster [DBG] pgmap v1110: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:14.328073+0000 mon.a (mon.0) 3365 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: cluster 2024-09-10T19:06:14.328274+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:14.328908+0000 mon.c (mon.2) 757 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.631 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:14 smithi083 bash[18629]: audit 2024-09-10T19:06:14.329365+0000 mon.a (mon.0) 3367 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:14.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:14.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:14.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:14.903 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:14.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:15.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:15.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:15.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T19:06:15.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:15.168 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:15.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:15.305 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:15.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:15.443 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:15.548 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:15.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:15.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: audit 2024-09-10T19:06:14.580671+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: audit 2024-09-10T19:06:14.588419+0000 mon.c (mon.2) 758 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: cluster 2024-09-10T19:06:15.211282+0000 osd.4 (osd.4) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: cluster 2024-09-10T19:06:15.211615+0000 osd.4 (osd.4) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: audit 2024-09-10T19:06:15.323584+0000 mon.a (mon.0) 3369 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: cluster 2024-09-10T19:06:15.323674+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: audit 2024-09-10T19:06:15.330667+0000 mon.c (mon.2) 759 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[18629]: audit 2024-09-10T19:06:15.330987+0000 mon.a (mon.0) 3371 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: audit 2024-09-10T19:06:14.580671+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: audit 2024-09-10T19:06:14.588419+0000 mon.c (mon.2) 758 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: cluster 2024-09-10T19:06:15.211282+0000 osd.4 (osd.4) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: cluster 2024-09-10T19:06:15.211615+0000 osd.4 (osd.4) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: audit 2024-09-10T19:06:15.323584+0000 mon.a (mon.0) 3369 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: cluster 2024-09-10T19:06:15.323674+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: audit 2024-09-10T19:06:15.330667+0000 mon.c (mon.2) 759 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:15 smithi083 bash[32381]: audit 2024-09-10T19:06:15.330987+0000 mon.a (mon.0) 3371 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: audit 2024-09-10T19:06:14.580671+0000 mon.a (mon.0) 3368 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: audit 2024-09-10T19:06:14.588419+0000 mon.c (mon.2) 758 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: cluster 2024-09-10T19:06:15.211282+0000 osd.4 (osd.4) 368 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: cluster 2024-09-10T19:06:15.211615+0000 osd.4 (osd.4) 369 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: audit 2024-09-10T19:06:15.323584+0000 mon.a (mon.0) 3369 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: cluster 2024-09-10T19:06:15.323674+0000 mon.a (mon.0) 3370 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: audit 2024-09-10T19:06:15.330667+0000 mon.c (mon.2) 759 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:15 smithi165 bash[22591]: audit 2024-09-10T19:06:15.330987+0000 mon.a (mon.0) 3371 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]: dispatch 2024-09-10T19:06:15.852 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:15.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:15.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:15.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:16.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.083+0000 7effec643640 1 -- 172.21.15.83:0/4242270353 >> v1:172.21.15.83:6789/0 conn(0x7effe4101510 legacy=0x7effe41018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:16.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/4242270353 shutdown_connections 2024-09-10T19:06:16.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/4242270353 >> 172.21.15.83:0/4242270353 conn(0x7effe40f7090 msgr2=0x7effe40f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:16.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/4242270353 shutdown_connections 2024-09-10T19:06:16.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/4242270353 wait complete. 2024-09-10T19:06:16.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 librados: starting msgr at 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 librados: starting objecter 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 Processor -- start 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- start start 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 librados: setting wanted keys 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 librados: calling monclient init 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7effe4106070 con 0x7effe4104280 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7effe411d3c0 con 0x7effe4101510 2024-09-10T19:06:16.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7effe411e560 con 0x7effe4107de0 2024-09-10T19:06:16.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effeabe0640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7effe4107de0 0x7effe411ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:51854/0 (socket says 172.21.15.83:51854) 2024-09-10T19:06:16.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effeabe0640 1 -- 172.21.15.83:0/2803894814 learned_addr learned my addr 172.21.15.83:0/2803894814 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:16.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe4104280 2024-09-10T19:06:16.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:16.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe4107de0 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe4101510 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879102352 0 0) 0x7effe4106070 con 0x7effe4104280 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd0003720 con 0x7effe4104280 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097716275 0 0) 0x7effe411e560 con 0x7effe4107de0 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe4106070 con 0x7effe4107de0 2024-09-10T19:06:16.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085838900 0 0) 0x7effe411d3c0 con 0x7effe4101510 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe411e560 con 0x7effe4101510 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1457885846 0 0) 0x7effd0003720 con 0x7effe4104280 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe411d3c0 con 0x7effe4104280 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 451959991 0 0) 0x7effe4106070 con 0x7effe4107de0 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd0003720 con 0x7effe4107de0 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7effd4001840 con 0x7effe4104280 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7effdc001b40 con 0x7effe4107de0 2024-09-10T19:06:16.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4210675254 0 0) 0x7effe411d3c0 con 0x7effe4104280 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 >> v1:172.21.15.83:6790/0 conn(0x7effe4107de0 legacy=0x7effe411ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 >> v1:172.21.15.165:6789/0 conn(0x7effe4101510 legacy=0x7effe4105550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe411f740 con 0x7effe4104280 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe411e750 con 0x7effe4104280 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 10 client.25677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effe411ec20 con 0x7effe4104280 2024-09-10T19:06:16.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.087+0000 7effec643640 1 librados: init done 2024-09-10T19:06:16.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:16.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7effd4001ec0 con 0x7effe4104280 2024-09-10T19:06:16.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7effd40030d0 con 0x7effe4104280 2024-09-10T19:06:16.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc8005010 con 0x7effe4104280 2024-09-10T19:06:16.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7effd4019570 con 0x7effe4104280 2024-09-10T19:06:16.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(719..719 src has 569..719) v4 ==== 8503+0+0 (unknown 4185144290 0 0) 0x7effd4087ce0 con 0x7effe4104280 2024-09-10T19:06:16.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 10 client.25677.objecter ms_dispatch 0x7effe4000b90 osd_map(719..719 src has 569..719) v4 2024-09-10T19:06:16.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 3 client.25677.objecter handle_osd_map got epochs [719,719] > 0 2024-09-10T19:06:16.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 3 client.25677.objecter handle_osd_map decoding full epoch 719 2024-09-10T19:06:16.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 20 client.25677.objecter dump_active .. 0 homeless 2024-09-10T19:06:16.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.091+0000 7effdaffd640 10 client.25677.objecter ms_handle_connect 0x7effd0071770 2024-09-10T19:06:16.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.095+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7effd4054b40 con 0x7effe4104280 2024-09-10T19:06:16.123 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:16.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:06:16.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:16.251 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:16.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.287+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7effc80052c0 con 0x7effe4104280 2024-09-10T19:06:16.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.323+0000 7effdaffd640 1 -- 172.21.15.83:0/2803894814 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v720) v1 ==== 91+0+0 (unknown 1873689781 0 0) 0x7effd4054960 con 0x7effe4104280 2024-09-10T19:06:16.329 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:06:16.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 10 librados: watch_flush enter 2024-09-10T19:06:16.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 10 librados: watch_flush exit 2024-09-10T19:06:16.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 20 client.25677.objecter shutdown clearing up homeless session... 2024-09-10T19:06:16.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 10 client.25677.objecter successfully canceled tick 2024-09-10T19:06:16.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 >> v1:172.21.15.83:6800/3071793735 conn(0x7effd0071770 legacy=0x7effd0073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:16.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 >> v1:172.21.15.83:6789/0 conn(0x7effe4104280 legacy=0x7effe4117560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:16.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 shutdown_connections 2024-09-10T19:06:16.334 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 >> 172.21.15.83:0/2803894814 conn(0x7effe40f7090 msgr2=0x7effe40fbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:16.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 shutdown_connections 2024-09-10T19:06:16.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 -- 172.21.15.83:0/2803894814 wait complete. 2024-09-10T19:06:16.335 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:16.327+0000 7effec643640 1 librados: shutdown 2024-09-10T19:06:16.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:16.378 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:16.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:16.516 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:16.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:16.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:16.757 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:16.792 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: audit 2024-09-10T19:06:16.292717+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:06:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.310706+0000 mgr.y (mgr.24526) 668 : cluster [DBG] pgmap v1113: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.320659+0000 mon.a (mon.0) 3373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.320703+0000 mon.a (mon.0) 3374 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:06:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: audit 2024-09-10T19:06:16.327419+0000 mon.a (mon.0) 3375 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]': finished 2024-09-10T19:06:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: audit 2024-09-10T19:06:16.327642+0000 mon.a (mon.0) 3376 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:06:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:16 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.327732+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:06:16.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:16.932 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:16.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:17.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:17.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: audit 2024-09-10T19:06:16.292717+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.310706+0000 mgr.y (mgr.24526) 668 : cluster [DBG] pgmap v1113: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.320659+0000 mon.a (mon.0) 3373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.320703+0000 mon.a (mon.0) 3374 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: audit 2024-09-10T19:06:16.327419+0000 mon.a (mon.0) 3375 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]': finished 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: audit 2024-09-10T19:06:16.327642+0000 mon.a (mon.0) 3376 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.327732+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: audit 2024-09-10T19:06:16.292717+0000 mon.a (mon.0) 3372 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.310706+0000 mgr.y (mgr.24526) 668 : cluster [DBG] pgmap v1113: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.320659+0000 mon.a (mon.0) 3373 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.320703+0000 mon.a (mon.0) 3374 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:06:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: audit 2024-09-10T19:06:16.327419+0000 mon.a (mon.0) 3375 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-126", "mode": "writeback"}]': finished 2024-09-10T19:06:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: audit 2024-09-10T19:06:16.327642+0000 mon.a (mon.0) 3376 : audit [INF] from='client.? v1:172.21.15.83:0/2803894814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:06:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:16 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.327732+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:06:17.068 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:17.171 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:17.205 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:17.309 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:17.345 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:17.449 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:17.478 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:17.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:17.618 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:17.721 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:17.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: audit 2024-09-10T19:06:16.647707+0000 mon.c (mon.2) 760 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: audit 2024-09-10T19:06:16.648326+0000 mon.a (mon.0) 3378 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.735 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.830606+0000 osd.3 (osd.3) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:17.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.831070+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:17.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.977251+0000 osd.5 (osd.5) 382 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:06:17.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:16.980808+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:06:17.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:17.159975+0000 osd.3 (osd.3) 400 : cluster [DBG] 12.b scrub starts 2024-09-10T19:06:17.736 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[18629]: cluster 2024-09-10T19:06:17.163644+0000 osd.3 (osd.3) 401 : cluster [DBG] 12.b scrub ok 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: audit 2024-09-10T19:06:16.647707+0000 mon.c (mon.2) 760 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: audit 2024-09-10T19:06:16.648326+0000 mon.a (mon.0) 3378 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.830606+0000 osd.3 (osd.3) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.831070+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.977251+0000 osd.5 (osd.5) 382 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:16.980808+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:17.159975+0000 osd.3 (osd.3) 400 : cluster [DBG] 12.b scrub starts 2024-09-10T19:06:17.737 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:17 smithi083 bash[32381]: cluster 2024-09-10T19:06:17.163644+0000 osd.3 (osd.3) 401 : cluster [DBG] 12.b scrub ok 2024-09-10T19:06:17.755 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:17.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: audit 2024-09-10T19:06:16.647707+0000 mon.c (mon.2) 760 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: audit 2024-09-10T19:06:16.648326+0000 mon.a (mon.0) 3378 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.830606+0000 osd.3 (osd.3) 398 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.831070+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.977251+0000 osd.5 (osd.5) 382 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:16.980808+0000 osd.5 (osd.5) 383 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:17.159975+0000 osd.3 (osd.3) 400 : cluster [DBG] 12.b scrub starts 2024-09-10T19:06:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:17 smithi165 bash[22591]: cluster 2024-09-10T19:06:17.163644+0000 osd.3 (osd.3) 401 : cluster [DBG] 12.b scrub ok 2024-09-10T19:06:17.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:17.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:17.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:18.038 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:18.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:18.172 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:18.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:18.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:18.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:18.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:18.444 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:18.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:18.581 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:18.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- 172.21.15.83:0/46835191 >> v1:172.21.15.83:6789/0 conn(0x7f75f4100c70 legacy=0x7f75f4101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- 172.21.15.83:0/46835191 shutdown_connections 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- 172.21.15.83:0/46835191 >> 172.21.15.83:0/46835191 conn(0x7f75f4074a20 msgr2=0x7f75f40f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- 172.21.15.83:0/46835191 shutdown_connections 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- 172.21.15.83:0/46835191 wait complete. 2024-09-10T19:06:18.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 librados: starting msgr at 2024-09-10T19:06:18.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 librados: starting objecter 2024-09-10T19:06:18.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: audit 2024-09-10T19:06:17.602854+0000 mon.a (mon.0) 3379 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:18.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:17.602910+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:06:18.698 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: audit 2024-09-10T19:06:17.608897+0000 mon.c (mon.2) 761 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: audit 2024-09-10T19:06:17.609375+0000 mon.a (mon.0) 3381 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.037921+0000 osd.4 (osd.4) 370 : cluster [DBG] 2.f scrub starts 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.041025+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub ok 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.050618+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.050995+0000 osd.0 (osd.0) 358 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.311358+0000 mgr.y (mgr.24526) 669 : cluster [DBG] pgmap v1116: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:18.699 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:18 smithi165 bash[22591]: audit 2024-09-10T19:06:18.320737+0000 mgr.y (mgr.24526) 670 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 Processor -- start 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- start start 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 librados: setting wanted keys 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 librados: calling monclient init 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f75f4100510 con 0x7f75f4103a50 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f75f41006c0 con 0x7f75f41075b0 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.690+0000 7f75fc028640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f75f411e360 con 0x7f75f4100c70 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75f95c3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f75f4103a50 0x7f75f40feb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35598/0 (socket says 172.21.15.83:35598) 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75f95c3640 1 -- 172.21.15.83:0/1782444934 learned_addr learned my addr 172.21.15.83:0/1782444934 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:18.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter ms_handle_connect 0x7f75f4103a50 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter ms_handle_connect 0x7f75f4100c70 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter ms_handle_connect 0x7f75f41075b0 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413581411 0 0) 0x7f75f4100510 con 0x7f75f4103a50 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e0003b70 con 0x7f75f4103a50 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573266457 0 0) 0x7f75f411e360 con 0x7f75f4100c70 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f4100510 con 0x7f75f4100c70 2024-09-10T19:06:18.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37380722 0 0) 0x7f75f41006c0 con 0x7f75f41075b0 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f411e360 con 0x7f75f41075b0 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523296440 0 0) 0x7f75f4100510 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75f41006c0 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f75f00015b0 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008909708 0 0) 0x7f75e0003b70 con 0x7f75f4103a50 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75f4100510 con 0x7f75f4103a50 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3593137730 0 0) 0x7f75f411e360 con 0x7f75f41075b0 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e0003b70 con 0x7f75f41075b0 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 748713734 0 0) 0x7f75f41006c0 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.165:6789/0 conn(0x7f75f41075b0 legacy=0x7f75f411ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6789/0 conn(0x7f75f4103a50 legacy=0x7f75f40feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f411f580 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f411e550 con 0x7f75f4100c70 2024-09-10T19:06:18.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75fc028640 10 client.49769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f411ea20 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75fc028640 1 librados: init done 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f75f0001210 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f75f00034b0 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.694+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f75d8004ff0 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f75f00199e0 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(722..722 src has 569..722) v4 ==== 8495+0+0 (unknown 2458890307 0 0) 0x7f75f00880c0 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 osd_map(722..722 src has 569..722) v4 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 3 client.49769.objecter handle_osd_map got epochs [722,722] > 0 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 3 client.49769.objecter handle_osd_map decoding full epoch 722 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 20 client.49769.objecter dump_active .. 0 homeless 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v722) v1 ==== 54+0+16 (unknown 2168025390 0 2420000192) 0x7f75f008a4b0 con 0x7f75f4100c70 2024-09-10T19:06:18.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75e0071780 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 10 client.49769.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 1 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 1 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 0 1 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 4 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 10 client.49769.objecter _send_command 1 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d8004a60 con 0x7f75d8004610 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d8004610 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d8004a60 con 0x7f75d8004610 2024-09-10T19:06:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:06:18.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 1 = system:0 2024-09-10T19:06:18.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 4 2024-09-10T19:06:18.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.698+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 0 1 2024-09-10T19:06:18.717 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 10 client.49769.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 2 2024-09-10T19:06:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.722 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 2 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8005290 osd=0 3 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 0 2 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 4 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 10 client.49769.objecter _send_command 2 2024-09-10T19:06:18.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.714+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d8004e40 con 0x7f75d8004610 2024-09-10T19:06:18.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d8004e40 con 0x7f75d8004610 2024-09-10T19:06:18.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 4 2024-09-10T19:06:18.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 0 2 2024-09-10T19:06:18.725 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 10 client.49769.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 3 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 3 2024-09-10T19:06:18.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 1 3 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 4 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 10 client.49769.objecter _send_command 3 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d80022f0 con 0x7f75d8004110 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.718+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d8004110 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.722+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d80022f0 con 0x7f75d8004110 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.722+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:06:18.727 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.722+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 3 = system:0 2024-09-10T19:06:18.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.722+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 4 2024-09-10T19:06:18.728 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.722+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 1 3 2024-09-10T19:06:18.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 10 client.49769.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 4 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 4 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8003fa0 osd=1 3 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 1 4 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 4 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 10 client.49769.objecter _send_command 4 2024-09-10T19:06:18.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.742+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d80018f0 con 0x7f75d8004110 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d80018f0 con 0x7f75d8004110 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 4 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 1 4 2024-09-10T19:06:18.752 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:06:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:06:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 10 client.49769.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 5 2024-09-10T19:06:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 5 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 2 5 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 4 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 10 client.49769.objecter _send_command 5 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d8003dd0 con 0x7f75d8011760 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.746+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d8011760 2024-09-10T19:06:18.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.750+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d8003dd0 con 0x7f75d8011760 2024-09-10T19:06:18.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.750+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:06:18.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.750+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 5 = system:0 2024-09-10T19:06:18.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.750+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 4 2024-09-10T19:06:18.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.750+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 2 5 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 10 client.49769.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 6 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 6 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d80035e0 osd=2 3 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 2 6 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 4 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 10 client.49769.objecter _send_command 6 2024-09-10T19:06:18.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d80030d0 con 0x7f75d8011760 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d80030d0 con 0x7f75d8011760 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 4 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 2 6 2024-09-10T19:06:18.777 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 10 client.49769.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 7 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 7 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 3 7 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 4 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 10 client.49769.objecter _send_command 7 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d8011550 con 0x7f75d80163e0 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.770+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d80163e0 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.774+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d8011550 con 0x7f75d80163e0 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.774+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.774+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 7 = system:0 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.774+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 4 2024-09-10T19:06:18.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.774+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 3 7 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 10 client.49769.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 8 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 8 2024-09-10T19:06:18.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8010da0 osd=3 3 2024-09-10T19:06:18.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 3 8 2024-09-10T19:06:18.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 4 2024-09-10T19:06:18.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 10 client.49769.objecter _send_command 8 2024-09-10T19:06:18.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d8017730 con 0x7f75d80163e0 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d8017730 con 0x7f75d80163e0 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 4 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 3 8 2024-09-10T19:06:18.793 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 10 client.49769.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 9 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 9 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 4 9 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 4 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 10 client.49769.objecter _send_command 9 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.786+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d801e640 con 0x7f75d801ab40 2024-09-10T19:06:18.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d801ab40 2024-09-10T19:06:18.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d801e640 con 0x7f75d801ab40 2024-09-10T19:06:18.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:06:18.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 9 = system:0 2024-09-10T19:06:18.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 4 2024-09-10T19:06:18.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.790+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 4 9 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 10 client.49769.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 10 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 10 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8017730 osd=4 3 2024-09-10T19:06:18.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 4 10 2024-09-10T19:06:18.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 4 2024-09-10T19:06:18.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 10 client.49769.objecter _send_command 10 2024-09-10T19:06:18.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.810+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d801e640 con 0x7f75d801ab40 2024-09-10T19:06:18.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:18.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d801e640 con 0x7f75d801ab40 2024-09-10T19:06:18.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 4 2024-09-10T19:06:18.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 4 10 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 10 client.49769.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 11 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 11 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 5 11 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 4 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 10 client.49769.objecter _send_command 11 2024-09-10T19:06:18.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.814+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d80209b0 con 0x7f75d801ecc0 2024-09-10T19:06:18.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d801ecc0 2024-09-10T19:06:18.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d80209b0 con 0x7f75d801ecc0 2024-09-10T19:06:18.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:06:18.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 11 = system:0 2024-09-10T19:06:18.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 4 2024-09-10T19:06:18.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.818+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 5 11 2024-09-10T19:06:18.842 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: audit 2024-09-10T19:06:17.602854+0000 mon.a (mon.0) 3379 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:17.602910+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: audit 2024-09-10T19:06:17.608897+0000 mon.c (mon.2) 761 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: audit 2024-09-10T19:06:17.609375+0000 mon.a (mon.0) 3381 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.037921+0000 osd.4 (osd.4) 370 : cluster [DBG] 2.f scrub starts 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.041025+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub ok 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.050618+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:18.844 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.050995+0000 osd.0 (osd.0) 358 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.311358+0000 mgr.y (mgr.24526) 669 : cluster [DBG] pgmap v1116: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[18629]: audit 2024-09-10T19:06:18.320737+0000 mgr.y (mgr.24526) 670 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: audit 2024-09-10T19:06:17.602854+0000 mon.a (mon.0) 3379 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:17.602910+0000 mon.a (mon.0) 3380 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: audit 2024-09-10T19:06:17.608897+0000 mon.c (mon.2) 761 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: audit 2024-09-10T19:06:17.609375+0000 mon.a (mon.0) 3381 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]: dispatch 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.037921+0000 osd.4 (osd.4) 370 : cluster [DBG] 2.f scrub starts 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.041025+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub ok 2024-09-10T19:06:18.845 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.050618+0000 osd.0 (osd.0) 357 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:18.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.050995+0000 osd.0 (osd.0) 358 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:18.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.311358+0000 mgr.y (mgr.24526) 669 : cluster [DBG] pgmap v1116: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:18.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:18 smithi083 bash[32381]: audit 2024-09-10T19:06:18.320737+0000 mgr.y (mgr.24526) 670 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 10 client.49769.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 12 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 12 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d801e640 osd=5 3 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 5 12 2024-09-10T19:06:18.847 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 4 2024-09-10T19:06:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 10 client.49769.objecter _send_command 12 2024-09-10T19:06:18.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.838+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d80209b0 con 0x7f75d801ecc0 2024-09-10T19:06:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d80209b0 con 0x7f75d801ecc0 2024-09-10T19:06:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 4 2024-09-10T19:06:18.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 5 12 2024-09-10T19:06:18.850 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:06:18.850 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:06:18.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 10 client.49769.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 13 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 13 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 6 13 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 4 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 10 client.49769.objecter _send_command 13 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.842+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d8020e80 con 0x7f75d8023570 2024-09-10T19:06:18.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d8023570 2024-09-10T19:06:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75ec00cb70 con 0x7f75d8023570 2024-09-10T19:06:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:06:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 13 = system:0 2024-09-10T19:06:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 4 2024-09-10T19:06:18.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.846+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 6 13 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 10 client.49769.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 14 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 14 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d80209b0 osd=6 3 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 6 14 2024-09-10T19:06:18.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 4 2024-09-10T19:06:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 10 client.49769.objecter _send_command 14 2024-09-10T19:06:18.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.866+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d8010650 con 0x7f75d8023570 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d8010650 con 0x7f75d8023570 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 4 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 6 14 2024-09-10T19:06:18.875 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 10 client.49769.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 15 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 15 2024-09-10T19:06:18.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 7 15 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 4 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 10 client.49769.objecter _send_command 15 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75d8015ef0 con 0x7f75d8015a20 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 10 client.49769.objecter ms_handle_connect 0x7f75d8015a20 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f75d8015ef0 con 0x7f75d8015a20 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 15 = system:0 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 4 2024-09-10T19:06:18.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.870+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 7 15 2024-09-10T19:06:18.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 10 client.49769.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:06:18.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign -1 16 2024-09-10T19:06:18.897 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 20 client.49769.objecter _get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 15 client.49769.objecter _session_command_op_remove -1 16 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 20 client.49769.objecter get_session s=0x7f75d8010650 osd=7 3 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 15 client.49769.objecter _session_command_op_assign 7 16 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 4 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 10 client.49769.objecter _send_command 16 2024-09-10T19:06:18.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.890+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f75d8015ef0 con 0x7f75d8015a20 2024-09-10T19:06:18.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75ea7fc640 1 -- 172.21.15.83:0/1782444934 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f75d8015ef0 con 0x7f75d8015a20 2024-09-10T19:06:18.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75ea7fc640 10 client.49769.objecter ms_dispatch 0x7f75f4000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:18.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75ea7fc640 10 client.49769.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:18.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75ea7fc640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 4 2024-09-10T19:06:18.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75ea7fc640 15 client.49769.objecter _session_command_op_remove 7 16 2024-09-10T19:06:18.902 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 librados: watch_flush enter 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 librados: watch_flush exit 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.0 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6801/2535527483 conn(0x7f75d8004610 legacy=0x7f75d8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8005290 osd=0 1 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.1 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6805/2309624637 conn(0x7f75d8004110 legacy=0x7f75d800eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8003fa0 osd=1 1 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.2 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6809/386094220 conn(0x7f75d8011760 legacy=0x7f75d8013b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80035e0 osd=2 1 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.3 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6813/1279688021 conn(0x7f75d80163e0 legacy=0x7f75d8016810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010da0 osd=3 1 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.4 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.165:6800/3225515265 conn(0x7f75d801ab40 legacy=0x7f75d801cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8017730 osd=4 1 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.5 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.165:6804/571173777 conn(0x7f75d801ecc0 legacy=0x7f75d801f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d801e640 osd=5 1 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.6 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.165:6808/1317922936 conn(0x7f75d8023570 legacy=0x7f75d8025950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d80209b0 osd=6 1 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter close_session for osd.7 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.165:6812/1610151650 conn(0x7f75d8015a20 legacy=0x7f75d8027070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter put_session s=0x7f75d8010650 osd=7 1 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 20 client.49769.objecter shutdown clearing up homeless session... 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.894+0000 7f75fc028640 10 client.49769.objecter successfully canceled tick 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fa5c5640 1 -- 172.21.15.83:0/1782444934 reap_dead start 2024-09-10T19:06:18.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fa5c5640 1 -- 172.21.15.83:0/1782444934 reap_dead start 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fa5c5640 1 -- 172.21.15.83:0/1782444934 reap_dead start 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fa5c5640 1 -- 172.21.15.83:0/1782444934 reap_dead start 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fa5c5640 1 -- 172.21.15.83:0/1782444934 reap_dead start 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6800/3071793735 conn(0x7f75e0071780 legacy=0x7f75e0073c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> v1:172.21.15.83:6790/0 conn(0x7f75f4100c70 legacy=0x7f75f40fe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 shutdown_connections 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 >> 172.21.15.83:0/1782444934 conn(0x7f75f4074a20 msgr2=0x7f75f40f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 shutdown_connections 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 -- 172.21.15.83:0/1782444934 wait complete. 2024-09-10T19:06:18.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:18.898+0000 7f75fc028640 1 librados: shutdown 2024-09-10T19:06:18.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:18.977 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:18.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:19.081 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:19.111 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:19.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:19.251 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:19.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:19.383 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:19.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:19.521 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:19.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:19.637 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9009 ms) 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6034 ms) 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5072 ms) 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25653 ms) 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13250 ms) 2024-09-10T19:06:19.638 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12061 ms) 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000012 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11957 ms) 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6068 ms) 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (559573 ms total) 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-10T19:06:19.639 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1292 ms) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10723 ms) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18004 ms) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30021 ms total) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6989 ms) 2024-09-10T19:06:19.640 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7999 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrub... 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24018 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10060 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8083 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7937 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10006 ms) 2024-09-10T19:06:19.641 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-10T19:06:19.653 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:19.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:19.791 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:19.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:17.960046+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:17.966254+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.071598+0000 osd.2 (osd.2) 289 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.075086+0000 osd.2 (osd.2) 290 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.598663+0000 mon.a (mon.0) 3382 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:19.905 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.599883+0000 osd.6 (osd.6) 341 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.604365+0000 osd.6 (osd.6) 342 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: audit 2024-09-10T19:06:18.609238+0000 mon.a (mon.0) 3383 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:18.609390+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: audit 2024-09-10T19:06:18.703162+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? v1:172.21.15.83:0/1782444934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:19.085586+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:06:19.906 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:19 smithi165 bash[22591]: cluster 2024-09-10T19:06:19.089869+0000 osd.3 (osd.3) 403 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:06:19.922 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:19.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:20.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:17.960046+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:17.966254+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.071598+0000 osd.2 (osd.2) 289 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.075086+0000 osd.2 (osd.2) 290 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.598663+0000 mon.a (mon.0) 3382 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.599883+0000 osd.6 (osd.6) 341 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.604365+0000 osd.6 (osd.6) 342 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: audit 2024-09-10T19:06:18.609238+0000 mon.a (mon.0) 3383 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:18.609390+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:06:20.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: audit 2024-09-10T19:06:18.703162+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? v1:172.21.15.83:0/1782444934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:19.085586+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[18629]: cluster 2024-09-10T19:06:19.089869+0000 osd.3 (osd.3) 403 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:17.960046+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:17.966254+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.071598+0000 osd.2 (osd.2) 289 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.075086+0000 osd.2 (osd.2) 290 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.598663+0000 mon.a (mon.0) 3382 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.599883+0000 osd.6 (osd.6) 341 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.604365+0000 osd.6 (osd.6) 342 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: audit 2024-09-10T19:06:18.609238+0000 mon.a (mon.0) 3383 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-126"}]': finished 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:18.609390+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: audit 2024-09-10T19:06:18.703162+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? v1:172.21.15.83:0/1782444934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:19.085586+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:06:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:19 smithi083 bash[32381]: cluster 2024-09-10T19:06:19.089869+0000 osd.3 (osd.3) 403 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:06:20.058 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:20.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:20.300 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:20.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:06:20.329 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:20.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.461 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:20.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:20.599 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:20.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:19.007182+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:06:20.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:19.010830+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:06:20.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:19.632431+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:06:20.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.001402+0000 osd.4 (osd.4) 372 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:06:20.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.005787+0000 osd.4 (osd.4) 373 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:06:20.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.037926+0000 osd.3 (osd.3) 404 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:06:20.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.041966+0000 osd.3 (osd.3) 405 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:06:20.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.311928+0000 mgr.y (mgr.24526) 671 : cluster [DBG] pgmap v1119: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:20.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:20.735 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:20.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:20.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:19.007182+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:06:20.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:19.010830+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:19.632431+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.001402+0000 osd.4 (osd.4) 372 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.005787+0000 osd.4 (osd.4) 373 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.037926+0000 osd.3 (osd.3) 404 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.041966+0000 osd.3 (osd.3) 405 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:06:20.852 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:20 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.311928+0000 mgr.y (mgr.24526) 671 : cluster [DBG] pgmap v1119: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:20.872 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:20.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:20.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:21.012 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:21.040 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:19.007182+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:19.010830+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:19.632431+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.001402+0000 osd.4 (osd.4) 372 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.005787+0000 osd.4 (osd.4) 373 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.037926+0000 osd.3 (osd.3) 404 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.041966+0000 osd.3 (osd.3) 405 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:06:21.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:20 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.311928+0000 mgr.y (mgr.24526) 671 : cluster [DBG] pgmap v1119: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:21.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.152 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:21.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:21.289 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:21.316 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:21.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.407 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:21.427 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:21.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.567 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:21.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:21.693 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:21.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.115782+0000 osd.2 (osd.2) 291 : cluster [DBG] 6.f scrub starts 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.119141+0000 osd.2 (osd.2) 292 : cluster [DBG] 6.f scrub ok 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.647255+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: audit 2024-09-10T19:06:20.650681+0000 mon.c (mon.2) 763 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: audit 2024-09-10T19:06:20.652123+0000 mon.a (mon.0) 3387 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.903051+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:20.906528+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:21.344606+0000 osd.0 (osd.0) 359 : cluster [DBG] 12.e scrub starts 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[18629]: cluster 2024-09-10T19:06:21.347899+0000 osd.0 (osd.0) 360 : cluster [DBG] 12.e scrub ok 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.115782+0000 osd.2 (osd.2) 291 : cluster [DBG] 6.f scrub starts 2024-09-10T19:06:21.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.119141+0000 osd.2 (osd.2) 292 : cluster [DBG] 6.f scrub ok 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.647255+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: audit 2024-09-10T19:06:20.650681+0000 mon.c (mon.2) 763 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: audit 2024-09-10T19:06:20.652123+0000 mon.a (mon.0) 3387 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.903051+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:20.906528+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:21.344606+0000 osd.0 (osd.0) 359 : cluster [DBG] 12.e scrub starts 2024-09-10T19:06:21.696 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:21 smithi083 bash[32381]: cluster 2024-09-10T19:06:21.347899+0000 osd.0 (osd.0) 360 : cluster [DBG] 12.e scrub ok 2024-09-10T19:06:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:21.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:06:21.832 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:21.935 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:21.968 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:21.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:22.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:22.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.115782+0000 osd.2 (osd.2) 291 : cluster [DBG] 6.f scrub starts 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.119141+0000 osd.2 (osd.2) 292 : cluster [DBG] 6.f scrub ok 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.647255+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: audit 2024-09-10T19:06:20.650681+0000 mon.c (mon.2) 763 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: audit 2024-09-10T19:06:20.652123+0000 mon.a (mon.0) 3387 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.903051+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:06:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:20.906528+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:06:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:21.344606+0000 osd.0 (osd.0) 359 : cluster [DBG] 12.e scrub starts 2024-09-10T19:06:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:21 smithi165 bash[22591]: cluster 2024-09-10T19:06:21.347899+0000 osd.0 (osd.0) 360 : cluster [DBG] 12.e scrub ok 2024-09-10T19:06:22.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:22.244 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:22.335 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:22.348 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:22.372 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:22.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:22.508 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:22.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:22.647 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:22.751 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:22.785 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: audit 2024-09-10T19:06:21.650758+0000 mon.a (mon.0) 3388 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:21.651237+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:06:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: audit 2024-09-10T19:06:21.680754+0000 mon.c (mon.2) 764 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: audit 2024-09-10T19:06:21.681442+0000 mon.a (mon.0) 3390 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.003540+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.007136+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.023585+0000 osd.3 (osd.3) 406 : cluster [DBG] 6.c scrub starts 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.027044+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.c scrub ok 2024-09-10T19:06:22.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:22 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.312816+0000 mgr.y (mgr.24526) 672 : cluster [DBG] pgmap v1122: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:22.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:22.926 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:22.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:23.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:22] "GET /metrics HTTP/1.1" 200 229002 "" "Prometheus/2.43.0" 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: audit 2024-09-10T19:06:21.650758+0000 mon.a (mon.0) 3388 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:21.651237+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: audit 2024-09-10T19:06:21.680754+0000 mon.c (mon.2) 764 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: audit 2024-09-10T19:06:21.681442+0000 mon.a (mon.0) 3390 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.003540+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.007136+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.023585+0000 osd.3 (osd.3) 406 : cluster [DBG] 6.c scrub starts 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.027044+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.c scrub ok 2024-09-10T19:06:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.312816+0000 mgr.y (mgr.24526) 672 : cluster [DBG] pgmap v1122: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: audit 2024-09-10T19:06:21.650758+0000 mon.a (mon.0) 3388 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:21.651237+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: audit 2024-09-10T19:06:21.680754+0000 mon.c (mon.2) 764 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: audit 2024-09-10T19:06:21.681442+0000 mon.a (mon.0) 3390 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.003540+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.007136+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.023585+0000 osd.3 (osd.3) 406 : cluster [DBG] 6.c scrub starts 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.027044+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.c scrub ok 2024-09-10T19:06:23.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:22 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.312816+0000 mgr.y (mgr.24526) 672 : cluster [DBG] pgmap v1122: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:23.063 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:23.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:23.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:23.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:23.342 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:23.445 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:23.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:23.584 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:23.619 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:23.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:23.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:23.756 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.006225+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:06:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.009435+0000 osd.1 (osd.1) 374 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:06:23.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.178365+0000 osd.2 (osd.2) 293 : cluster [DBG] 12.a scrub starts 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.182103+0000 osd.2 (osd.2) 294 : cluster [DBG] 12.a scrub ok 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: audit 2024-09-10T19:06:22.669912+0000 mon.a (mon.0) 3391 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.670046+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: audit 2024-09-10T19:06:22.675929+0000 mon.c (mon.2) 765 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: audit 2024-09-10T19:06:22.681824+0000 mon.a (mon.0) 3393 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.824828+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.828097+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.916482+0000 osd.7 (osd.7) 383 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:22.919519+0000 osd.7 (osd.7) 384 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.072995+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.076547+0000 osd.3 (osd.3) 409 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.361515+0000 osd.0 (osd.0) 361 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:06:23.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.367343+0000 osd.0 (osd.0) 362 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.006225+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.009435+0000 osd.1 (osd.1) 374 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.178365+0000 osd.2 (osd.2) 293 : cluster [DBG] 12.a scrub starts 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.182103+0000 osd.2 (osd.2) 294 : cluster [DBG] 12.a scrub ok 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: audit 2024-09-10T19:06:22.669912+0000 mon.a (mon.0) 3391 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.670046+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: audit 2024-09-10T19:06:22.675929+0000 mon.c (mon.2) 765 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: audit 2024-09-10T19:06:22.681824+0000 mon.a (mon.0) 3393 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.824828+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:06:23.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.828097+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.916482+0000 osd.7 (osd.7) 383 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:22.919519+0000 osd.7 (osd.7) 384 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.072995+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.076547+0000 osd.3 (osd.3) 409 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.361515+0000 osd.0 (osd.0) 361 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:06:23.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:23 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.367343+0000 osd.0 (osd.0) 362 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:06:23.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:23.894 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:23.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:23.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.006225+0000 osd.1 (osd.1) 373 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.009435+0000 osd.1 (osd.1) 374 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.178365+0000 osd.2 (osd.2) 293 : cluster [DBG] 12.a scrub starts 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.182103+0000 osd.2 (osd.2) 294 : cluster [DBG] 12.a scrub ok 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: audit 2024-09-10T19:06:22.669912+0000 mon.a (mon.0) 3391 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.670046+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:06:24.011 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: audit 2024-09-10T19:06:22.675929+0000 mon.c (mon.2) 765 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: audit 2024-09-10T19:06:22.681824+0000 mon.a (mon.0) 3393 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.824828+0000 osd.5 (osd.5) 386 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.828097+0000 osd.5 (osd.5) 387 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.916482+0000 osd.7 (osd.7) 383 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:22.919519+0000 osd.7 (osd.7) 384 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.072995+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.076547+0000 osd.3 (osd.3) 409 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.361515+0000 osd.0 (osd.0) 361 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:06:24.012 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:23 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.367343+0000 osd.0 (osd.0) 362 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:06:24.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:24.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.172 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:24.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:24.304 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:24.366 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:24.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:24.437 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:24.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:24.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: audit 2024-09-10T19:06:23.695122+0000 mon.a (mon.0) 3394 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.695209+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: audit 2024-09-10T19:06:23.697259+0000 mon.c (mon.2) 766 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: audit 2024-09-10T19:06:23.698057+0000 mon.a (mon.0) 3396 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.966881+0000 osd.4 (osd.4) 376 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:23.971470+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:24.313681+0000 mgr.y (mgr.24526) 673 : cluster [DBG] pgmap v1125: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:24.320546+0000 mon.a (mon.0) 3397 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: audit 2024-09-10T19:06:24.331947+0000 mon.a (mon.0) 3398 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]': finished 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[18629]: cluster 2024-09-10T19:06:24.332036+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: audit 2024-09-10T19:06:23.695122+0000 mon.a (mon.0) 3394 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.695209+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: audit 2024-09-10T19:06:23.697259+0000 mon.c (mon.2) 766 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: audit 2024-09-10T19:06:23.698057+0000 mon.a (mon.0) 3396 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.966881+0000 osd.4 (osd.4) 376 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:23.971470+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:24.313681+0000 mgr.y (mgr.24526) 673 : cluster [DBG] pgmap v1125: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:24.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:24.320546+0000 mon.a (mon.0) 3397 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:24.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: audit 2024-09-10T19:06:24.331947+0000 mon.a (mon.0) 3398 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]': finished 2024-09-10T19:06:24.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:24 smithi083 bash[32381]: cluster 2024-09-10T19:06:24.332036+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:06:24.819 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:24.849 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:24.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: audit 2024-09-10T19:06:23.695122+0000 mon.a (mon.0) 3394 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:24.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.695209+0000 mon.a (mon.0) 3395 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:06:24.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: audit 2024-09-10T19:06:23.697259+0000 mon.c (mon.2) 766 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: audit 2024-09-10T19:06:23.698057+0000 mon.a (mon.0) 3396 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]: dispatch 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.966881+0000 osd.4 (osd.4) 376 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:23.971470+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:24.313681+0000 mgr.y (mgr.24526) 673 : cluster [DBG] pgmap v1125: 214 pgs: 18 creating+peering, 14 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:24.320546+0000 mon.a (mon.0) 3397 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: audit 2024-09-10T19:06:24.331947+0000 mon.a (mon.0) 3398 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-128", "mode": "writeback"}]': finished 2024-09-10T19:06:24.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:24 smithi165 bash[22591]: cluster 2024-09-10T19:06:24.332036+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:06:24.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:24.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:24.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:25.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:25.125 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:25.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.261 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:25.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.399 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:25.503 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:25.537 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:25.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:25.676 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:25.780 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.816 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:25.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:25.950 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:25.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:26.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:26.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:25 smithi083 bash[32381]: cluster 2024-09-10T19:06:25.032622+0000 osd.1 (osd.1) 375 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:06:26.074 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:25 smithi083 bash[32381]: cluster 2024-09-10T19:06:25.036479+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:06:26.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:25 smithi083 bash[18629]: cluster 2024-09-10T19:06:25.032622+0000 osd.1 (osd.1) 375 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:06:26.074 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:25 smithi083 bash[18629]: cluster 2024-09-10T19:06:25.036479+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:06:26.091 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:26.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:26.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:25 smithi165 bash[22591]: cluster 2024-09-10T19:06:25.032622+0000 osd.1 (osd.1) 375 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:06:26.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:25 smithi165 bash[22591]: cluster 2024-09-10T19:06:25.036479+0000 osd.1 (osd.1) 376 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:06:26.228 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:26.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:26.366 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:26.469 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:26.505 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:26.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:26.645 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:26.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:26.786 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:26.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:26.925 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:26.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:27.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.028770+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.032261+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.268005+0000 osd.0 (osd.0) 363 : cluster [DBG] 12.c scrub starts 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.272824+0000 osd.0 (osd.0) 364 : cluster [DBG] 12.c scrub ok 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.315898+0000 mgr.y (mgr.24526) 674 : cluster [DBG] pgmap v1127: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 438 B/s wr, 1 op/s 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.028770+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.032261+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.268005+0000 osd.0 (osd.0) 363 : cluster [DBG] 12.c scrub starts 2024-09-10T19:06:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.272824+0000 osd.0 (osd.0) 364 : cluster [DBG] 12.c scrub ok 2024-09-10T19:06:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:26 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.315898+0000 mgr.y (mgr.24526) 674 : cluster [DBG] pgmap v1127: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 438 B/s wr, 1 op/s 2024-09-10T19:06:27.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:27.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:27.178 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:26 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.028770+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:06:27.178 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:26 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.032261+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:06:27.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:26 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.268005+0000 osd.0 (osd.0) 363 : cluster [DBG] 12.c scrub starts 2024-09-10T19:06:27.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:26 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.272824+0000 osd.0 (osd.0) 364 : cluster [DBG] 12.c scrub ok 2024-09-10T19:06:27.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:26 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.315898+0000 mgr.y (mgr.24526) 674 : cluster [DBG] pgmap v1127: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 438 B/s wr, 1 op/s 2024-09-10T19:06:27.201 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:27.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:27.341 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:27.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:27.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:27.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:27.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:27.634 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:27.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:27.751 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:27.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:27.891 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:27.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:27.994 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.026 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.728121+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:06:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.733231+0000 osd.5 (osd.5) 389 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:06:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.917076+0000 osd.6 (osd.6) 343 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:26.917407+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:27.075479+0000 osd.1 (osd.1) 379 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:06:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:27 smithi165 bash[22591]: cluster 2024-09-10T19:06:27.078729+0000 osd.1 (osd.1) 380 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:06:28.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.163 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:28.263 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.728121+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.733231+0000 osd.5 (osd.5) 389 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.917076+0000 osd.6 (osd.6) 343 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:26.917407+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:27.075479+0000 osd.1 (osd.1) 379 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[18629]: cluster 2024-09-10T19:06:27.078729+0000 osd.1 (osd.1) 380 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.728121+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.733231+0000 osd.5 (osd.5) 389 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.917076+0000 osd.6 (osd.6) 343 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:26.917407+0000 osd.6 (osd.6) 344 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:28.264 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:27.075479+0000 osd.1 (osd.1) 379 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:06:28.265 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:27 smithi083 bash[32381]: cluster 2024-09-10T19:06:27.078729+0000 osd.1 (osd.1) 380 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:06:28.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.299 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:28.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.438 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:28.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.580 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:28.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.716 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:28.820 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:28.850 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:28.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:28.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:29.091 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[32381]: cluster 2024-09-10T19:06:27.858493+0000 osd.4 (osd.4) 378 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:06:29.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[32381]: cluster 2024-09-10T19:06:27.862786+0000 osd.4 (osd.4) 379 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:06:29.104 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[32381]: cluster 2024-09-10T19:06:27.927967+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.316773+0000 mgr.y (mgr.24526) 675 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 362 B/s wr, 1 op/s 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[32381]: audit 2024-09-10T19:06:28.328770+0000 mgr.y (mgr.24526) 676 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[18629]: cluster 2024-09-10T19:06:27.858493+0000 osd.4 (osd.4) 378 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[18629]: cluster 2024-09-10T19:06:27.862786+0000 osd.4 (osd.4) 379 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[18629]: cluster 2024-09-10T19:06:27.927967+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.316773+0000 mgr.y (mgr.24526) 675 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 362 B/s wr, 1 op/s 2024-09-10T19:06:29.105 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:28 smithi083 bash[18629]: audit 2024-09-10T19:06:28.328770+0000 mgr.y (mgr.24526) 676 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:29.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:29.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:29.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[22591]: cluster 2024-09-10T19:06:27.858493+0000 osd.4 (osd.4) 378 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:06:29.241 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[22591]: cluster 2024-09-10T19:06:27.862786+0000 osd.4 (osd.4) 379 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:06:29.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[22591]: cluster 2024-09-10T19:06:27.927967+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:06:29.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.316773+0000 mgr.y (mgr.24526) 675 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 362 B/s wr, 1 op/s 2024-09-10T19:06:29.242 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:28 smithi165 bash[22591]: audit 2024-09-10T19:06:28.328770+0000 mgr.y (mgr.24526) 676 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:29.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:29.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.391 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:29.495 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.528 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:29.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:29.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:29.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:29.799 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:29.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:29.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:29.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:30.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:30.073 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.607163+0000 osd.6 (osd.6) 345 : cluster [DBG] 12.f scrub starts 2024-09-10T19:06:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.611502+0000 osd.6 (osd.6) 346 : cluster [DBG] 12.f scrub ok 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.753087+0000 osd.5 (osd.5) 390 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.763133+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.858325+0000 osd.4 (osd.4) 380 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:28.866456+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.092322+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.097313+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.331720+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: audit 2024-09-10T19:06:29.468431+0000 mon.c (mon.2) 767 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: audit 2024-09-10T19:06:29.469177+0000 mon.a (mon.0) 3402 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: audit 2024-09-10T19:06:29.581851+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:29 smithi165 bash[22591]: audit 2024-09-10T19:06:29.584364+0000 mon.c (mon.2) 768 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:30.177 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:30.216 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.607163+0000 osd.6 (osd.6) 345 : cluster [DBG] 12.f scrub starts 2024-09-10T19:06:30.216 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.611502+0000 osd.6 (osd.6) 346 : cluster [DBG] 12.f scrub ok 2024-09-10T19:06:30.216 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.753087+0000 osd.5 (osd.5) 390 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:06:30.216 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.763133+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:06:30.216 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.858325+0000 osd.4 (osd.4) 380 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:28.866456+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.092322+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.097313+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.331720+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: audit 2024-09-10T19:06:29.468431+0000 mon.c (mon.2) 767 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: audit 2024-09-10T19:06:29.469177+0000 mon.a (mon.0) 3402 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: audit 2024-09-10T19:06:29.581851+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[18629]: audit 2024-09-10T19:06:29.584364+0000 mon.c (mon.2) 768 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.607163+0000 osd.6 (osd.6) 345 : cluster [DBG] 12.f scrub starts 2024-09-10T19:06:30.217 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.611502+0000 osd.6 (osd.6) 346 : cluster [DBG] 12.f scrub ok 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.753087+0000 osd.5 (osd.5) 390 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.763133+0000 osd.5 (osd.5) 391 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.858325+0000 osd.4 (osd.4) 380 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:28.866456+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.092322+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.097313+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.331720+0000 mon.a (mon.0) 3401 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: audit 2024-09-10T19:06:29.468431+0000 mon.c (mon.2) 767 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: audit 2024-09-10T19:06:29.469177+0000 mon.a (mon.0) 3402 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: audit 2024-09-10T19:06:29.581851+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:30.218 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:29 smithi083 bash[32381]: audit 2024-09-10T19:06:29.584364+0000 mon.c (mon.2) 768 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:30.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:30.354 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:30.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:30.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:30.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:30.630 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:30.733 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:30.764 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:30.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:30.903 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:30.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:31.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:31.040 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:31.143 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:31.176 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:31.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.610883+0000 osd.6 (osd.6) 347 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.614059+0000 osd.6 (osd.6) 348 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.740542+0000 osd.5 (osd.5) 392 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:29.746737+0000 osd.5 (osd.5) 393 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.045253+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.049292+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.078048+0000 osd.1 (osd.1) 381 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.081594+0000 osd.1 (osd.1) 382 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:06:31.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.302836+0000 osd.0 (osd.0) 365 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.306751+0000 osd.0 (osd.0) 366 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.317711+0000 mgr.y (mgr.24526) 677 : cluster [DBG] pgmap v1131: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: audit 2024-09-10T19:06:30.346364+0000 mon.a (mon.0) 3404 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.346537+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: audit 2024-09-10T19:06:30.348588+0000 mon.c (mon.2) 769 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[18629]: audit 2024-09-10T19:06:30.349111+0000 mon.a (mon.0) 3406 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.610883+0000 osd.6 (osd.6) 347 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.614059+0000 osd.6 (osd.6) 348 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.740542+0000 osd.5 (osd.5) 392 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:06:31.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:29.746737+0000 osd.5 (osd.5) 393 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.045253+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.049292+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.078048+0000 osd.1 (osd.1) 381 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.081594+0000 osd.1 (osd.1) 382 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.302836+0000 osd.0 (osd.0) 365 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.306751+0000 osd.0 (osd.0) 366 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.317711+0000 mgr.y (mgr.24526) 677 : cluster [DBG] pgmap v1131: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: audit 2024-09-10T19:06:30.346364+0000 mon.a (mon.0) 3404 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.346537+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: audit 2024-09-10T19:06:30.348588+0000 mon.c (mon.2) 769 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:30 smithi083 bash[32381]: audit 2024-09-10T19:06:30.349111+0000 mon.a (mon.0) 3406 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.610883+0000 osd.6 (osd.6) 347 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.614059+0000 osd.6 (osd.6) 348 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.740542+0000 osd.5 (osd.5) 392 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:29.746737+0000 osd.5 (osd.5) 393 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.045253+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.049292+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.078048+0000 osd.1 (osd.1) 381 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.081594+0000 osd.1 (osd.1) 382 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:06:31.294 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.302836+0000 osd.0 (osd.0) 365 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.306751+0000 osd.0 (osd.0) 366 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.317711+0000 mgr.y (mgr.24526) 677 : cluster [DBG] pgmap v1131: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: audit 2024-09-10T19:06:30.346364+0000 mon.a (mon.0) 3404 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.346537+0000 mon.a (mon.0) 3405 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: audit 2024-09-10T19:06:30.348588+0000 mon.c (mon.2) 769 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.295 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:30 smithi165 bash[22591]: audit 2024-09-10T19:06:30.349111+0000 mon.a (mon.0) 3406 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]: dispatch 2024-09-10T19:06:31.313 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:31.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:31.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:31.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:31.590 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:31.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:31.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:31.721 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:31.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:31.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:31.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:31.988 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:31.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:32.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:32.126 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:32.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:32.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.659148+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:06:32.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:30.665216+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:06:32.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.066979+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:06:32.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.070730+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:06:32.244 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.115631+0000 osd.1 (osd.1) 383 : cluster [DBG] 6.e scrub starts 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.119795+0000 osd.1 (osd.1) 384 : cluster [DBG] 6.e scrub ok 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.185220+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.188228+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.340196+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: audit 2024-09-10T19:06:31.349665+0000 mon.a (mon.0) 3408 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.350014+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.659148+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:30.665216+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.066979+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.070730+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:06:32.245 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.115631+0000 osd.1 (osd.1) 383 : cluster [DBG] 6.e scrub starts 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.119795+0000 osd.1 (osd.1) 384 : cluster [DBG] 6.e scrub ok 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.185220+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.188228+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.340196+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: audit 2024-09-10T19:06:31.349665+0000 mon.a (mon.0) 3408 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:32.246 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:31 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.350014+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:06:32.263 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.659148+0000 osd.6 (osd.6) 349 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:30.665216+0000 osd.6 (osd.6) 350 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.066979+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.070730+0000 osd.3 (osd.3) 415 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.115631+0000 osd.1 (osd.1) 383 : cluster [DBG] 6.e scrub starts 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.119795+0000 osd.1 (osd.1) 384 : cluster [DBG] 6.e scrub ok 2024-09-10T19:06:32.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.185220+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:06:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.188228+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:06:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.340196+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: audit 2024-09-10T19:06:31.349665+0000 mon.a (mon.0) 3408 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-128"}]': finished 2024-09-10T19:06:32.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:31 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.350014+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:06:32.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:32.397 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:32.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:32.537 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:32.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:32.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:32.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:32.804 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:32.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:32.944 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:32.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:32.975 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:32] "GET /metrics HTTP/1.1" 200 229054 "" "Prometheus/2.43.0" 2024-09-10T19:06:33.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.816339+0000 osd.5 (osd.5) 394 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:31.820042+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.118233+0000 osd.1 (osd.1) 385 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.121677+0000 osd.1 (osd.1) 386 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.318614+0000 mgr.y (mgr.24526) 678 : cluster [DBG] pgmap v1134: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:33.059 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:32 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.357126+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:06:33.079 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:33.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:33.217 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:33.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- 172.21.15.83:0/916251943 >> v1:172.21.15.83:6790/0 conn(0x7f324c107e30 legacy=0x7f324c10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:33.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- 172.21.15.83:0/916251943 shutdown_connections 2024-09-10T19:06:33.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- 172.21.15.83:0/916251943 >> 172.21.15.83:0/916251943 conn(0x7f324c075290 msgr2=0x7f324c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:33.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- 172.21.15.83:0/916251943 shutdown_connections 2024-09-10T19:06:33.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- 172.21.15.83:0/916251943 wait complete. 2024-09-10T19:06:33.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 librados: starting msgr at 2024-09-10T19:06:33.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 librados: starting objecter 2024-09-10T19:06:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 Processor -- start 2024-09-10T19:06:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- start start 2024-09-10T19:06:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 librados: setting wanted keys 2024-09-10T19:06:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 librados: calling monclient init 2024-09-10T19:06:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f324c0fe0c0 con 0x7f324c107e30 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f324c11c2c0 con 0x7f324c1014e0 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f32523b7640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f324c11d460 con 0x7f324c1042a0 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f324b7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f324c1042a0 0x7f324c116460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37008/0 (socket says 172.21.15.83:37008) 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f324b7fe640 1 -- 172.21.15.83:0/3917683644 learned_addr learned my addr 172.21.15.83:0/3917683644 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter ms_handle_connect 0x7f324c1042a0 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter ms_handle_connect 0x7f324c107e30 2024-09-10T19:06:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:33.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter ms_handle_connect 0x7f324c1014e0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.250+0000 7f3248ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098744092 0 0) 0x7f324c11d460 con 0x7f324c1042a0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3234003120 con 0x7f324c1042a0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002977085 0 0) 0x7f324c11c2c0 con 0x7f324c1014e0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324c11d460 con 0x7f324c1014e0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328832712 0 0) 0x7f324c0fe0c0 con 0x7f324c107e30 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f324c11c2c0 con 0x7f324c107e30 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172787211 0 0) 0x7f3234003120 con 0x7f324c1042a0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f324c0fe0c0 con 0x7f324c1042a0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f323c0018e0 con 0x7f324c1042a0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377458656 0 0) 0x7f324c11d460 con 0x7f324c1014e0 2024-09-10T19:06:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3234003120 con 0x7f324c1014e0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f32400018c0 con 0x7f324c1014e0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872343730 0 0) 0x7f324c11c2c0 con 0x7f324c107e30 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f324c11d460 con 0x7f324c107e30 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1815820662 0 0) 0x7f324c0fe0c0 con 0x7f324c1042a0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 >> v1:172.21.15.165:6789/0 conn(0x7f324c1014e0 legacy=0x7f324c0fd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 >> v1:172.21.15.83:6789/0 conn(0x7f324c107e30 legacy=0x7f324c119b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c11e600 con 0x7f324c1042a0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c11b310 con 0x7f324c1042a0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f32523b7640 10 client.49775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c11b760 con 0x7f324c1042a0 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f32523b7640 1 librados: init done 2024-09-10T19:06:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f323c002030 con 0x7f324c1042a0 2024-09-10T19:06:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f323c005b40 con 0x7f324c1042a0 2024-09-10T19:06:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.254+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3238005010 con 0x7f324c1042a0 2024-09-10T19:06:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f323c001bd0 con 0x7f324c1042a0 2024-09-10T19:06:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(733..733 src has 569..733) v4 ==== 8122+0+0 (unknown 4038412090 0 0) 0x7f323c08a5c0 con 0x7f324c1042a0 2024-09-10T19:06:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 10 client.49775.objecter ms_dispatch 0x7f324c000b90 osd_map(733..733 src has 569..733) v4 2024-09-10T19:06:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 3 client.49775.objecter handle_osd_map got epochs [733,733] > 0 2024-09-10T19:06:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 3 client.49775.objecter handle_osd_map decoding full epoch 733 2024-09-10T19:06:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 20 client.49775.objecter dump_active .. 0 homeless 2024-09-10T19:06:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.258+0000 7f3248ff9640 10 client.49775.objecter ms_handle_connect 0x7f3234071a10 2024-09-10T19:06:33.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.266+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f323c056b60 con 0x7f324c1042a0 2024-09-10T19:06:33.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.816339+0000 osd.5 (osd.5) 394 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:06:33.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:31.820042+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:06:33.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.118233+0000 osd.1 (osd.1) 385 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:06:33.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.121677+0000 osd.1 (osd.1) 386 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.318614+0000 mgr.y (mgr.24526) 678 : cluster [DBG] pgmap v1134: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.357126+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.816339+0000 osd.5 (osd.5) 394 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:31.820042+0000 osd.5 (osd.5) 395 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.118233+0000 osd.1 (osd.1) 385 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.121677+0000 osd.1 (osd.1) 386 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.318614+0000 mgr.y (mgr.24526) 678 : cluster [DBG] pgmap v1134: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:33.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:32 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.357126+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:06:33.321 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:33.347 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:33.362 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[57601]: logger=sqlstore.transactions t=2024-09-10T19:06:33.115848934Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:06:33.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:33.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:33.466+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f32380052c0 con 0x7f324c1042a0 2024-09-10T19:06:33.474 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:33.577 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:33.605 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:33.708 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:33.741 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:33.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:33.876 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:33.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:33.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:34.009 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.775812+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:06:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.786550+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:06:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: cluster 2024-09-10T19:06:32.961997+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: cluster 2024-09-10T19:06:33.354850+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:06:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: audit 2024-09-10T19:06:33.360935+0000 mon.c (mon.2) 770 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: audit 2024-09-10T19:06:33.361410+0000 mon.a (mon.0) 3413 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: audit 2024-09-10T19:06:33.471394+0000 mon.c (mon.2) 771 : audit [INF] from='client.? v1:172.21.15.83:0/3917683644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:33 smithi165 bash[22591]: audit 2024-09-10T19:06:33.472054+0000 mon.a (mon.0) 3414 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:34.145 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:34.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.261 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.775812+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.786550+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: cluster 2024-09-10T19:06:32.961997+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: cluster 2024-09-10T19:06:33.354850+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: audit 2024-09-10T19:06:33.360935+0000 mon.c (mon.2) 770 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: audit 2024-09-10T19:06:33.361410+0000 mon.a (mon.0) 3413 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: audit 2024-09-10T19:06:33.471394+0000 mon.c (mon.2) 771 : audit [INF] from='client.? v1:172.21.15.83:0/3917683644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[18629]: audit 2024-09-10T19:06:33.472054+0000 mon.a (mon.0) 3414 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.775812+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.786550+0000 osd.5 (osd.5) 397 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:06:34.262 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: cluster 2024-09-10T19:06:32.961997+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:34.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: cluster 2024-09-10T19:06:33.354850+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:06:34.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: audit 2024-09-10T19:06:33.360935+0000 mon.c (mon.2) 770 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: audit 2024-09-10T19:06:33.361410+0000 mon.a (mon.0) 3413 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:34.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: audit 2024-09-10T19:06:33.471394+0000 mon.c (mon.2) 771 : audit [INF] from='client.? v1:172.21.15.83:0/3917683644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.263 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:33 smithi083 bash[32381]: audit 2024-09-10T19:06:33.472054+0000 mon.a (mon.0) 3414 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:06:34.277 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:34.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.354+0000 7f3248ff9640 1 -- 172.21.15.83:0/3917683644 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v735) v1 ==== 113+0+0 (unknown 2649009087 0 0) 0x7f323c056500 con 0x7f324c1042a0 2024-09-10T19:06:34.363 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T19:06:34.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 10 librados: watch_flush enter 2024-09-10T19:06:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 10 librados: watch_flush exit 2024-09-10T19:06:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 20 client.49775.objecter shutdown clearing up homeless session... 2024-09-10T19:06:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 10 client.49775.objecter successfully canceled tick 2024-09-10T19:06:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 >> v1:172.21.15.83:6800/3071793735 conn(0x7f3234071a10 legacy=0x7f3234073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:34.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 >> v1:172.21.15.83:6790/0 conn(0x7f324c1042a0 legacy=0x7f324c116460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 shutdown_connections 2024-09-10T19:06:34.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.362+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 >> 172.21.15.83:0/3917683644 conn(0x7f324c075290 msgr2=0x7f324c0f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:34.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.366+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 shutdown_connections 2024-09-10T19:06:34.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.370+0000 7f32523b7640 1 -- 172.21.15.83:0/3917683644 wait complete. 2024-09-10T19:06:34.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:34.370+0000 7f32523b7640 1 librados: shutdown 2024-09-10T19:06:34.380 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.406 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:34.509 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.545 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:34.649 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:34.684 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:34.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.821 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:34.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:34.960 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:34.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:35.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:35.097 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:35.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:35.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:33.678377+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:06:35.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:33.681835+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:06:35.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.050699+0000 osd.3 (osd.3) 416 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:06:35.214 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.054618+0000 osd.3 (osd.3) 417 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.319502+0000 mgr.y (mgr.24526) 679 : cluster [DBG] pgmap v1137: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.349599+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.355495+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: audit 2024-09-10T19:06:34.357581+0000 mon.a (mon.0) 3415 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: audit 2024-09-10T19:06:34.357874+0000 mon.a (mon.0) 3416 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.357975+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: audit 2024-09-10T19:06:34.362785+0000 mon.c (mon.2) 772 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[18629]: audit 2024-09-10T19:06:34.363837+0000 mon.a (mon.0) 3418 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:33.678377+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:33.681835+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.050699+0000 osd.3 (osd.3) 416 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.054618+0000 osd.3 (osd.3) 417 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:06:35.215 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.319502+0000 mgr.y (mgr.24526) 679 : cluster [DBG] pgmap v1137: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.349599+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.355495+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: audit 2024-09-10T19:06:34.357581+0000 mon.a (mon.0) 3415 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: audit 2024-09-10T19:06:34.357874+0000 mon.a (mon.0) 3416 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.357975+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: audit 2024-09-10T19:06:34.362785+0000 mon.c (mon.2) 772 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.216 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:34 smithi083 bash[32381]: audit 2024-09-10T19:06:34.363837+0000 mon.a (mon.0) 3418 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.230 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:35.333 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:33.678377+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:33.681835+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.050699+0000 osd.3 (osd.3) 416 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.054618+0000 osd.3 (osd.3) 417 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.319502+0000 mgr.y (mgr.24526) 679 : cluster [DBG] pgmap v1137: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.349599+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.355495+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: audit 2024-09-10T19:06:34.357581+0000 mon.a (mon.0) 3415 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: audit 2024-09-10T19:06:34.357874+0000 mon.a (mon.0) 3416 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:06:35.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.357975+0000 mon.a (mon.0) 3417 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:06:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: audit 2024-09-10T19:06:34.362785+0000 mon.c (mon.2) 772 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.364 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:34 smithi165 bash[22591]: audit 2024-09-10T19:06:34.363837+0000 mon.a (mon.0) 3418 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:35.371 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:35.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:35.509 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:35.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:35.650 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:35.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:35.789 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:35.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:35.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:35.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:36.030 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.885830+0000 osd.7 (osd.7) 385 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:06:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: cluster 2024-09-10T19:06:34.889810+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:06:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.118670+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.d scrub starts 2024-09-10T19:06:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.121907+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.d scrub ok 2024-09-10T19:06:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: audit 2024-09-10T19:06:35.367974+0000 mon.a (mon.0) 3419 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:36.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.368195+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:06:36.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: audit 2024-09-10T19:06:35.369410+0000 mon.c (mon.2) 773 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:35 smithi165 bash[22591]: audit 2024-09-10T19:06:35.369863+0000 mon.a (mon.0) 3421 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.167 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.885830+0000 osd.7 (osd.7) 385 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: cluster 2024-09-10T19:06:34.889810+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.118670+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.d scrub starts 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.121907+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.d scrub ok 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: audit 2024-09-10T19:06:35.367974+0000 mon.a (mon.0) 3419 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.368195+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: audit 2024-09-10T19:06:35.369410+0000 mon.c (mon.2) 773 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[18629]: audit 2024-09-10T19:06:35.369863+0000 mon.a (mon.0) 3421 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.885830+0000 osd.7 (osd.7) 385 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: cluster 2024-09-10T19:06:34.889810+0000 osd.7 (osd.7) 386 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.118670+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.d scrub starts 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.121907+0000 osd.1 (osd.1) 388 : cluster [DBG] 2.d scrub ok 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: audit 2024-09-10T19:06:35.367974+0000 mon.a (mon.0) 3419 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.368195+0000 mon.a (mon.0) 3420 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: audit 2024-09-10T19:06:35.369410+0000 mon.c (mon.2) 773 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:35 smithi083 bash[32381]: audit 2024-09-10T19:06:35.369863+0000 mon.a (mon.0) 3421 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:36.307 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:36.339 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:36.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:36.479 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:36.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:36.617 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:36.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:36.757 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:36.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:36.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:36.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:36.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:37.030 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:37.037 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:06:37.037 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:06:37.037 INFO:teuthology.orchestra.run.smithi083.stdout: "complaint_time": 30, 2024-09-10T19:06:37.037 INFO:teuthology.orchestra.run.smithi083.stdout: "num_blocked_ops": 0 2024-09-10T19:06:37.037 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:06:37.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:37.166 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.854586+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.858045+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.859368+0000 osd.5 (osd.5) 398 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:35.862112+0000 osd.5 (osd.5) 399 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.101334+0000 osd.1 (osd.1) 389 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.107784+0000 osd.1 (osd.1) 390 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.126778+0000 osd.2 (osd.2) 297 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.131414+0000 osd.2 (osd.2) 298 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:06:37.169 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.321676+0000 mgr.y (mgr.24526) 680 : cluster [DBG] pgmap v1140: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: audit 2024-09-10T19:06:36.367412+0000 mon.a (mon.0) 3422 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.367699+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: audit 2024-09-10T19:06:36.370592+0000 mon.c (mon.2) 774 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[32381]: audit 2024-09-10T19:06:36.371050+0000 mon.a (mon.0) 3424 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.854586+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.858045+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.859368+0000 osd.5 (osd.5) 398 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:35.862112+0000 osd.5 (osd.5) 399 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.101334+0000 osd.1 (osd.1) 389 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.107784+0000 osd.1 (osd.1) 390 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:06:37.170 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.126778+0000 osd.2 (osd.2) 297 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.131414+0000 osd.2 (osd.2) 298 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.321676+0000 mgr.y (mgr.24526) 680 : cluster [DBG] pgmap v1140: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: audit 2024-09-10T19:06:36.367412+0000 mon.a (mon.0) 3422 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.367699+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: audit 2024-09-10T19:06:36.370592+0000 mon.c (mon.2) 774 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.171 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:36 smithi083 bash[18629]: audit 2024-09-10T19:06:36.371050+0000 mon.a (mon.0) 3424 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:37.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:37.306 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:37.308 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T19:06:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.854586+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:06:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.858045+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:06:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.859368+0000 osd.5 (osd.5) 398 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:06:37.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:35.862112+0000 osd.5 (osd.5) 399 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.101334+0000 osd.1 (osd.1) 389 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.107784+0000 osd.1 (osd.1) 390 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.126778+0000 osd.2 (osd.2) 297 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.131414+0000 osd.2 (osd.2) 298 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.321676+0000 mgr.y (mgr.24526) 680 : cluster [DBG] pgmap v1140: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: audit 2024-09-10T19:06:36.367412+0000 mon.a (mon.0) 3422 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.367699+0000 mon.a (mon.0) 3423 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: audit 2024-09-10T19:06:36.370592+0000 mon.c (mon.2) 774 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:36 smithi165 bash[22591]: audit 2024-09-10T19:06:36.371050+0000 mon.a (mon.0) 3424 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]: dispatch 2024-09-10T19:06:37.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:37.440 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:37.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:37.579 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:37.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:37.712 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:37.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:37.853 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:37.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:37.991 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:37.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:38.095 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:38.130 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:38.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:38.248 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.740686+0000 osd.4 (osd.4) 382 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.744156+0000 osd.6 (osd.6) 353 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.747808+0000 osd.6 (osd.6) 354 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.751249+0000 osd.4 (osd.4) 383 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.896753+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:36.900137+0000 osd.7 (osd.7) 390 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.098244+0000 osd.2 (osd.2) 299 : cluster [DBG] 2.e scrub starts 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.103432+0000 osd.2 (osd.2) 300 : cluster [DBG] 2.e scrub ok 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.358856+0000 mon.a (mon.0) 3425 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: audit 2024-09-10T19:06:37.368255+0000 mon.a (mon.0) 3426 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]': finished 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.368437+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: audit 2024-09-10T19:06:37.508838+0000 mon.c (mon.2) 775 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.249 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[18629]: audit 2024-09-10T19:06:37.509294+0000 mon.a (mon.0) 3428 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.740686+0000 osd.4 (osd.4) 382 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.744156+0000 osd.6 (osd.6) 353 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.747808+0000 osd.6 (osd.6) 354 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.751249+0000 osd.4 (osd.4) 383 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.896753+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:36.900137+0000 osd.7 (osd.7) 390 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.098244+0000 osd.2 (osd.2) 299 : cluster [DBG] 2.e scrub starts 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.103432+0000 osd.2 (osd.2) 300 : cluster [DBG] 2.e scrub ok 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.358856+0000 mon.a (mon.0) 3425 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: audit 2024-09-10T19:06:37.368255+0000 mon.a (mon.0) 3426 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]': finished 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.368437+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: audit 2024-09-10T19:06:37.508838+0000 mon.c (mon.2) 775 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.250 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:38 smithi083 bash[32381]: audit 2024-09-10T19:06:37.509294+0000 mon.a (mon.0) 3428 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.270 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.740686+0000 osd.4 (osd.4) 382 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.744156+0000 osd.6 (osd.6) 353 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.747808+0000 osd.6 (osd.6) 354 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.751249+0000 osd.4 (osd.4) 383 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.896753+0000 osd.7 (osd.7) 389 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:36.900137+0000 osd.7 (osd.7) 390 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.098244+0000 osd.2 (osd.2) 299 : cluster [DBG] 2.e scrub starts 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.103432+0000 osd.2 (osd.2) 300 : cluster [DBG] 2.e scrub ok 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.358856+0000 mon.a (mon.0) 3425 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:38.342 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: audit 2024-09-10T19:06:37.368255+0000 mon.a (mon.0) 3426 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-130", "mode": "writeback"}]': finished 2024-09-10T19:06:38.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.368437+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:06:38.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: audit 2024-09-10T19:06:37.508838+0000 mon.c (mon.2) 775 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.343 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[22591]: audit 2024-09-10T19:06:37.509294+0000 mon.a (mon.0) 3428 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:38.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:38.399 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:38.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:38.533 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:38.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:38.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:38.673 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:38.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:38.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/835871655 >> v1:172.21.15.83:6789/0 conn(0x7f55ec104810 legacy=0x7f55ec104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:38.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/835871655 shutdown_connections 2024-09-10T19:06:38.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/835871655 >> 172.21.15.83:0/835871655 conn(0x7f55ec0fa3d0 msgr2=0x7f55ec0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:38.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/835871655 shutdown_connections 2024-09-10T19:06:38.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/835871655 wait complete. 2024-09-10T19:06:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 librados: starting msgr at 2024-09-10T19:06:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 librados: starting objecter 2024-09-10T19:06:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 Processor -- start 2024-09-10T19:06:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- start start 2024-09-10T19:06:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 librados: setting wanted keys 2024-09-10T19:06:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 librados: calling monclient init 2024-09-10T19:06:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55ec11e180 con 0x7f55ec10b130 2024-09-10T19:06:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55ec11f340 con 0x7f55ec104810 2024-09-10T19:06:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f0bc4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f55ec10b130 0x7f55ec11c880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34620/0 (socket says 172.21.15.83:34620) 2024-09-10T19:06:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f0bc4640 1 -- 172.21.15.83:0/3032126763 learned_addr learned my addr 172.21.15.83:0/3032126763 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55eb7fe640 1 --1- 172.21.15.83:0/3032126763 >> v1:172.21.15.83:6790/0 conn(0x7f55ec1075a0 0x7f55ec100f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37028/0 (socket says 172.21.15.83:37028) 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55ec120540 con 0x7f55ec1075a0 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter ms_handle_connect 0x7f55ec10b130 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter ms_handle_connect 0x7f55ec1075a0 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262385652 0 0) 0x7f55ec11e180 con 0x7f55ec10b130 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d80039c0 con 0x7f55ec10b130 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter ms_handle_connect 0x7f55ec104810 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160491593 0 0) 0x7f55ec120540 con 0x7f55ec1075a0 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec11e180 con 0x7f55ec1075a0 2024-09-10T19:06:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311446941 0 0) 0x7f55ec11f340 con 0x7f55ec104810 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ec120540 con 0x7f55ec104810 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.774+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248276492 0 0) 0x7f55d80039c0 con 0x7f55ec10b130 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55ec11f340 con 0x7f55ec10b130 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f55e4001890 con 0x7f55ec10b130 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 622887006 0 0) 0x7f55ec11e180 con 0x7f55ec1075a0 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55d80039c0 con 0x7f55ec1075a0 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2274228551 0 0) 0x7f55ec11f340 con 0x7f55ec10b130 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 >> v1:172.21.15.83:6790/0 conn(0x7f55ec1075a0 legacy=0x7f55ec100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 >> v1:172.21.15.165:6789/0 conn(0x7f55ec104810 legacy=0x7f55ec100820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec121740 con 0x7f55ec10b130 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec120730 con 0x7f55ec10b130 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f55e40022e0 con 0x7f55ec10b130 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f55e40059a0 con 0x7f55ec10b130 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55f2627640 10 client.25695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec120b80 con 0x7f55ec10b130 2024-09-10T19:06:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55f2627640 1 librados: init done 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec0fff30 con 0x7f55ec10b130 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f55e401bdc0 con 0x7f55ec10b130 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(739..739 src has 569..739) v4 ==== 8503+0+0 (unknown 2423569303 0 0) 0x7f55e408a790 con 0x7f55ec10b130 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 10 client.25695.objecter ms_dispatch 0x7f55ec000b90 osd_map(739..739 src has 569..739) v4 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 3 client.25695.objecter handle_osd_map got epochs [739,739] > 0 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 3 client.25695.objecter handle_osd_map decoding full epoch 739 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 20 client.25695.objecter dump_active .. 0 homeless 2024-09-10T19:06:38.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.778+0000 7f55e8ff9640 10 client.25695.objecter ms_handle_connect 0x7f55d8071760 2024-09-10T19:06:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.786+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f55e40566e0 con 0x7f55ec10b130 2024-09-10T19:06:38.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:38.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:38.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:38.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:38.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:38.978+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f55ec110bb0 con 0x7f55ec10b130 2024-09-10T19:06:39.055 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:39.090 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:39.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:06:39.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:39.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.897401+0000 osd.5 (osd.5) 400 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:06:39.208 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.901100+0000 osd.5 (osd.5) 401 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.931908+0000 osd.7 (osd.7) 391 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:37.935530+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.065290+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.068508+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.322573+0000 mgr.y (mgr.24526) 681 : cluster [DBG] pgmap v1143: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: audit 2024-09-10T19:06:38.340053+0000 mgr.y (mgr.24526) 682 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: audit 2024-09-10T19:06:38.369223+0000 mon.a (mon.0) 3429 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.369411+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: audit 2024-09-10T19:06:38.372988+0000 mon.c (mon.2) 776 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: audit 2024-09-10T19:06:38.373532+0000 mon.a (mon.0) 3431 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.209 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:39 smithi165 bash[22591]: audit 2024-09-10T19:06:38.983518+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:06:39.230 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.897401+0000 osd.5 (osd.5) 400 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.901100+0000 osd.5 (osd.5) 401 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.931908+0000 osd.7 (osd.7) 391 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:37.935530+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.065290+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.068508+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.322573+0000 mgr.y (mgr.24526) 681 : cluster [DBG] pgmap v1143: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: audit 2024-09-10T19:06:38.340053+0000 mgr.y (mgr.24526) 682 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: audit 2024-09-10T19:06:38.369223+0000 mon.a (mon.0) 3429 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.369411+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: audit 2024-09-10T19:06:38.372988+0000 mon.c (mon.2) 776 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: audit 2024-09-10T19:06:38.373532+0000 mon.a (mon.0) 3431 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[32381]: audit 2024-09-10T19:06:38.983518+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.897401+0000 osd.5 (osd.5) 400 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.901100+0000 osd.5 (osd.5) 401 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.931908+0000 osd.7 (osd.7) 391 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:37.935530+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.065290+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.068508+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-09-10T19:06:39.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.322573+0000 mgr.y (mgr.24526) 681 : cluster [DBG] pgmap v1143: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: audit 2024-09-10T19:06:38.340053+0000 mgr.y (mgr.24526) 682 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: audit 2024-09-10T19:06:38.369223+0000 mon.a (mon.0) 3429 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.369411+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: audit 2024-09-10T19:06:38.372988+0000 mon.c (mon.2) 776 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: audit 2024-09-10T19:06:38.373532+0000 mon.a (mon.0) 3431 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]: dispatch 2024-09-10T19:06:39.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:39 smithi083 bash[18629]: audit 2024-09-10T19:06:38.983518+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:06:39.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.334+0000 7f55e8ff9640 1 -- 172.21.15.83:0/3032126763 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v740) v1 ==== 101+0+0 (unknown 4197111409 0 0) 0x7f55e4056500 con 0x7f55ec10b130 2024-09-10T19:06:39.341 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 10 librados: watch_flush enter 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 10 librados: watch_flush exit 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 20 client.25695.objecter shutdown clearing up homeless session... 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 10 client.25695.objecter successfully canceled tick 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 >> v1:172.21.15.83:6800/3071793735 conn(0x7f55d8071760 legacy=0x7f55d8073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:39.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 >> v1:172.21.15.83:6789/0 conn(0x7f55ec10b130 legacy=0x7f55ec11c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:39.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 shutdown_connections 2024-09-10T19:06:39.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.338+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 >> 172.21.15.83:0/3032126763 conn(0x7f55ec0fa3d0 msgr2=0x7f55ec10e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:39.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.342+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 shutdown_connections 2024-09-10T19:06:39.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.342+0000 7f55f2627640 1 -- 172.21.15.83:0/3032126763 wait complete. 2024-09-10T19:06:39.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:39.342+0000 7f55f2627640 1 librados: shutdown 2024-09-10T19:06:39.360 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:39.464 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:39.499 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:39.602 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:39.636 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:39.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:39.774 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:39.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:39.908 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:39.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:40.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.687448+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:38.692245+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.137172+0000 osd.2 (osd.2) 301 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.141596+0000 osd.2 (osd.2) 302 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.326771+0000 mon.a (mon.0) 3433 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.328449+0000 mon.a (mon.0) 3434 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.328536+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: audit 2024-09-10T19:06:39.340230+0000 mon.a (mon.0) 3436 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:40.027 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: audit 2024-09-10T19:06:39.340490+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:06:40.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.340630+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:06:40.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.390433+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:06:40.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.394043+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:06:40.041 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.687448+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:38.692245+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.137172+0000 osd.2 (osd.2) 301 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.141596+0000 osd.2 (osd.2) 302 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.326771+0000 mon.a (mon.0) 3433 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.328449+0000 mon.a (mon.0) 3434 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:40.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.328536+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:06:40.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: audit 2024-09-10T19:06:39.340230+0000 mon.a (mon.0) 3436 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:40.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: audit 2024-09-10T19:06:39.340490+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:06:40.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.340630+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:06:40.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.390433+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:06:40.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:40 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.394043+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:06:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:40.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:40.181 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:40.284 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.687448+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:06:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:38.692245+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:06:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.137172+0000 osd.2 (osd.2) 301 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:06:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.141596+0000 osd.2 (osd.2) 302 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:06:40.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.326771+0000 mon.a (mon.0) 3433 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.328449+0000 mon.a (mon.0) 3434 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.328536+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: audit 2024-09-10T19:06:39.340230+0000 mon.a (mon.0) 3436 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-130"}]': finished 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: audit 2024-09-10T19:06:39.340490+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? v1:172.21.15.83:0/3032126763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.340630+0000 mon.a (mon.0) 3438 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.390433+0000 osd.0 (osd.0) 369 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:06:40.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:40 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.394043+0000 osd.0 (osd.0) 370 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:06:40.421 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.445 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:40.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:40.583 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:40.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:40.723 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:40.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:40.864 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:40.967 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.993 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:40.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:41.099 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.098+0000 7f71f1c5b640 1 -- 172.21.15.83:0/230689365 >> v1:172.21.15.83:6790/0 conn(0x7f71ec107580 legacy=0x7f71ec109a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.098+0000 7f71f1c5b640 1 -- 172.21.15.83:0/230689365 shutdown_connections 2024-09-10T19:06:41.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.098+0000 7f71f1c5b640 1 -- 172.21.15.83:0/230689365 >> 172.21.15.83:0/230689365 conn(0x7f71ec074a80 msgr2=0x7f71ec0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:41.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- 172.21.15.83:0/230689365 shutdown_connections 2024-09-10T19:06:41.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- 172.21.15.83:0/230689365 wait complete. 2024-09-10T19:06:41.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 librados: starting msgr at 2024-09-10T19:06:41.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 librados: starting objecter 2024-09-10T19:06:41.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 Processor -- start 2024-09-10T19:06:41.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- start start 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 librados: setting wanted keys 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 librados: calling monclient init 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71ec100580 con 0x7f71ec107580 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71ec11f710 con 0x7f71ec100c40 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71ec1208b0 con 0x7f71ec103a20 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71ebfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f71ec107580 0x7f71ec11cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34636/0 (socket says 172.21.15.83:34636) 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71ebfff640 1 -- 172.21.15.83:0/2647152064 learned_addr learned my addr 172.21.15.83:0/2647152064 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71eaffd640 1 --1- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6790/0 conn(0x7f71ec103a20 0x7f71ec0fcf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:37042/0 (socket says 172.21.15.83:37042) 2024-09-10T19:06:41.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter ms_handle_connect 0x7f71ec107580 2024-09-10T19:06:41.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:41.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter ms_handle_connect 0x7f71ec100c40 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter ms_handle_connect 0x7f71ec103a20 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376898793 0 0) 0x7f71ec100580 con 0x7f71ec107580 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d40034e0 con 0x7f71ec107580 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861280950 0 0) 0x7f71ec11f710 con 0x7f71ec100c40 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ec100580 con 0x7f71ec100c40 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810020368 0 0) 0x7f71ec1208b0 con 0x7f71ec103a20 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ec11f710 con 0x7f71ec103a20 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622837067 0 0) 0x7f71ec11f710 con 0x7f71ec103a20 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71ec1208b0 con 0x7f71ec103a20 2024-09-10T19:06:41.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f71e0001900 con 0x7f71ec103a20 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271033644 0 0) 0x7f71ec100580 con 0x7f71ec100c40 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71ec11f710 con 0x7f71ec100c40 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f71dc001840 con 0x7f71ec100c40 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889574866 0 0) 0x7f71d40034e0 con 0x7f71ec107580 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71ec100580 con 0x7f71ec107580 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f71e40023d0 con 0x7f71ec107580 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 163914309 0 0) 0x7f71ec1208b0 con 0x7f71ec103a20 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.165:6789/0 conn(0x7f71ec100c40 legacy=0x7f71ec0fc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6789/0 conn(0x7f71ec107580 legacy=0x7f71ec11cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec121a50 con 0x7f71ec103a20 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ec11e760 con 0x7f71ec103a20 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f71e0002ca0 con 0x7f71ec103a20 2024-09-10T19:06:41.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f71e0005b70 con 0x7f71ec103a20 2024-09-10T19:06:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.926929+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-09-10T19:06:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:39.931016+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-09-10T19:06:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:40.091817+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:06:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:40.095523+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:06:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:40.323148+0000 mgr.y (mgr.24526) 683 : cluster [DBG] pgmap v1146: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[22591]: cluster 2024-09-10T19:06:40.353666+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:06:41.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 10 client.49787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:41.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ec11ebb0 con 0x7f71ec103a20 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.102+0000 7f71f1c5b640 1 librados: init done 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f71ec0fc3b0 con 0x7f71ec103a20 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f71e0001ee0 con 0x7f71ec103a20 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(741..741 src has 569..741) v4 ==== 8122+0+0 (unknown 40871520 0 0) 0x7f71e008a590 con 0x7f71ec103a20 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 osd_map(741..741 src has 569..741) v4 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 3 client.49787.objecter handle_osd_map got epochs [741,741] > 0 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 3 client.49787.objecter handle_osd_map decoding full epoch 741 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 20 client.49787.objecter dump_active .. 0 homeless 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v741) v1 ==== 54+0+16 (unknown 1127790896 0 2420000192) 0x7f71e008c9e0 con 0x7f71ec103a20 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71d40718e0 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 1 2024-09-10T19:06:41.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 1 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 0 1 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 4 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 10 client.49787.objecter _send_command 1 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.106+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec10b690 con 0x7f71ec120a40 2024-09-10T19:06:41.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec120a40 2024-09-10T19:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec10b690 con 0x7f71ec120a40 2024-09-10T19:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 10 client.49787.objecter _finish_command 1 = system:0 2024-09-10T19:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 4 2024-09-10T19:06:41.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.110+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 0 1 2024-09-10T19:06:41.125 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:41.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 2 2024-09-10T19:06:41.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 2 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec04bf60 osd=0 3 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 0 2 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 4 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 10 client.49787.objecter _send_command 2 2024-09-10T19:06:41.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.126+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec121520 con 0x7f71ec120a40 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec121520 con 0x7f71ec120a40 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 10 client.49787.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 4 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 0 2 2024-09-10T19:06:41.135 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 3 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 3 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 1 3 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 4 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 10 client.49787.objecter _send_command 3 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec1089b0 con 0x7f71ec107ed0 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec107ed0 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec1089b0 con 0x7f71ec107ed0 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 10 client.49787.objecter _finish_command 3 = system:0 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 4 2024-09-10T19:06:41.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.130+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 1 3 2024-09-10T19:06:41.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 4 2024-09-10T19:06:41.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 4 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec1218d0 osd=1 3 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 1 4 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 4 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 10 client.49787.objecter _send_command 4 2024-09-10T19:06:41.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.146+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec02ab10 con 0x7f71ec107ed0 2024-09-10T19:06:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec02ab10 con 0x7f71ec107ed0 2024-09-10T19:06:41.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 10 client.49787.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 4 2024-09-10T19:06:41.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 1 4 2024-09-10T19:06:41.156 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 5 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 5 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 2 5 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 4 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 10 client.49787.objecter _send_command 5 2024-09-10T19:06:41.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec129d80 con 0x7f71ec126280 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec126280 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec129d80 con 0x7f71ec126280 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 10 client.49787.objecter _finish_command 5 = system:0 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 4 2024-09-10T19:06:41.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.150+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 2 5 2024-09-10T19:06:41.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 6 2024-09-10T19:06:41.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 6 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec125c00 osd=2 3 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 2 6 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 4 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 10 client.49787.objecter _send_command 6 2024-09-10T19:06:41.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.170+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec10aba0 con 0x7f71ec126280 2024-09-10T19:06:41.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec10aba0 con 0x7f71ec126280 2024-09-10T19:06:41.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 10 client.49787.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 4 2024-09-10T19:06:41.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 2 6 2024-09-10T19:06:41.179 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 7 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 7 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 3 7 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 4 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 10 client.49787.objecter _send_command 7 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec05e9f0 con 0x7f71ec12b070 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec12b070 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec05e9f0 con 0x7f71ec12b070 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 10 client.49787.objecter _finish_command 7 = system:0 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 4 2024-09-10T19:06:41.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.174+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 3 7 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 8 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 8 2024-09-10T19:06:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec05e310 osd=3 3 2024-09-10T19:06:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 3 8 2024-09-10T19:06:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 4 2024-09-10T19:06:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 10 client.49787.objecter _send_command 8 2024-09-10T19:06:41.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec12c390 con 0x7f71ec12b070 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec12c390 con 0x7f71ec12b070 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71d3fff640 10 client.49787.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 4 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.194+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 3 8 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:06:41.202 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 9 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 9 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 4 9 2024-09-10T19:06:41.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 4 2024-09-10T19:06:41.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 10 client.49787.objecter _send_command 9 2024-09-10T19:06:41.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec1332d0 con 0x7f71ec12f7d0 2024-09-10T19:06:41.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec12f7d0 2024-09-10T19:06:41.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec1332d0 con 0x7f71ec12f7d0 2024-09-10T19:06:41.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:06:41.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 10 client.49787.objecter _finish_command 9 = system:0 2024-09-10T19:06:41.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 4 2024-09-10T19:06:41.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.198+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 4 9 2024-09-10T19:06:41.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 10 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 10 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec12c390 osd=4 3 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 4 10 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 4 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 10 client.49787.objecter _send_command 10 2024-09-10T19:06:41.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.214+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec05dbc0 con 0x7f71ec12f7d0 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec05dbc0 con 0x7f71ec12f7d0 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 10 client.49787.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 4 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 4 10 2024-09-10T19:06:41.223 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 11 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 11 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 5 11 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 4 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 10 client.49787.objecter _send_command 11 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec12abd0 con 0x7f71ec12a6b0 2024-09-10T19:06:41.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec12a6b0 2024-09-10T19:06:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec12abd0 con 0x7f71ec12a6b0 2024-09-10T19:06:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:06:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 10 client.49787.objecter _finish_command 11 = system:0 2024-09-10T19:06:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 4 2024-09-10T19:06:41.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.218+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 5 11 2024-09-10T19:06:41.228 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:41.239 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.926929+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:39.931016+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:40.091817+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:40.095523+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:40.323148+0000 mgr.y (mgr.24526) 683 : cluster [DBG] pgmap v1146: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[18629]: cluster 2024-09-10T19:06:40.353666+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.926929+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:39.931016+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:40.091817+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:06:41.240 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:40.095523+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:06:41.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:40.323148+0000 mgr.y (mgr.24526) 683 : cluster [DBG] pgmap v1146: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:41.241 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:41 smithi083 bash[32381]: cluster 2024-09-10T19:06:40.353666+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:06:41.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 12 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 12 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec05dbc0 osd=5 3 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 5 12 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 4 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 10 client.49787.objecter _send_command 12 2024-09-10T19:06:41.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.238+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec12abd0 con 0x7f71ec12a6b0 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec12abd0 con 0x7f71ec12a6b0 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 10 client.49787.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 4 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 5 12 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:06:41.247 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 13 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 13 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 6 13 2024-09-10T19:06:41.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 4 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 10 client.49787.objecter _send_command 13 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec13bb90 con 0x7f71ec138090 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec138090 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec13bb90 con 0x7f71ec138090 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:06:41.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 10 client.49787.objecter _finish_command 13 = system:0 2024-09-10T19:06:41.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 4 2024-09-10T19:06:41.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.242+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 6 13 2024-09-10T19:06:41.258 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 14 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 14 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec12abd0 osd=6 3 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 6 14 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 4 2024-09-10T19:06:41.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 10 client.49787.objecter _send_command 14 2024-09-10T19:06:41.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.258+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec13d400 con 0x7f71ec138090 2024-09-10T19:06:41.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec13d400 con 0x7f71ec138090 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71d3fff640 10 client.49787.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 4 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 6 14 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:06:41.270 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 15 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 15 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 7 15 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 4 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 10 client.49787.objecter _send_command 15 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.262+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71ec13df40 con 0x7f71ec13da80 2024-09-10T19:06:41.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 10 client.49787.objecter ms_handle_connect 0x7f71ec13da80 2024-09-10T19:06:41.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f71ec13df40 con 0x7f71ec13da80 2024-09-10T19:06:41.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:06:41.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 10 client.49787.objecter _finish_command 15 = system:0 2024-09-10T19:06:41.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 4 2024-09-10T19:06:41.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.266+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 7 15 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.278+0000 7f71f1c5b640 10 client.49787.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign -1 16 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 20 client.49787.objecter _get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_remove -1 16 2024-09-10T19:06:41.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 20 client.49787.objecter get_session s=0x7f71ec13d400 osd=7 3 2024-09-10T19:06:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 15 client.49787.objecter _session_command_op_assign 7 16 2024-09-10T19:06:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 4 2024-09-10T19:06:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 10 client.49787.objecter _send_command 16 2024-09-10T19:06:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.282+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f71ec13df40 con 0x7f71ec13da80 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71d3fff640 1 -- 172.21.15.83:0/2647152064 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f71ec13df40 con 0x7f71ec13da80 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71d3fff640 10 client.49787.objecter ms_dispatch 0x7f71ec000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71d3fff640 10 client.49787.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71d3fff640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 4 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71d3fff640 15 client.49787.objecter _session_command_op_remove 7 16 2024-09-10T19:06:41.290 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:06:41.291 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:06:41.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 librados: watch_flush enter 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 librados: watch_flush exit 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.0 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6801/2535527483 conn(0x7f71ec120a40 legacy=0x7f71ec120e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec04bf60 osd=0 1 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.1 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6805/2309624637 conn(0x7f71ec107ed0 legacy=0x7f71ec1082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec1218d0 osd=1 1 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.2 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6809/386094220 conn(0x7f71ec126280 legacy=0x7f71ec128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec125c00 osd=2 1 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.3 2024-09-10T19:06:41.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6813/1279688021 conn(0x7f71ec12b070 legacy=0x7f71ec12b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71ebfff640 1 -- 172.21.15.83:0/2647152064 reap_dead start 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05e310 osd=3 1 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.4 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.165:6800/3225515265 conn(0x7f71ec12f7d0 legacy=0x7f71ec131bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12c390 osd=4 1 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.5 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.165:6804/571173777 conn(0x7f71ec12a6b0 legacy=0x7f71ec136460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec05dbc0 osd=5 1 2024-09-10T19:06:41.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.6 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.165:6808/1317922936 conn(0x7f71ec138090 legacy=0x7f71ec13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec12abd0 osd=6 1 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter close_session for osd.7 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.165:6812/1610151650 conn(0x7f71ec13da80 legacy=0x7f71ec13bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter put_session s=0x7f71ec13d400 osd=7 1 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 20 client.49787.objecter shutdown clearing up homeless session... 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 10 client.49787.objecter successfully canceled tick 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71ebfff640 1 -- 172.21.15.83:0/2647152064 reap_dead start 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6800/3071793735 conn(0x7f71d40718e0 legacy=0x7f71d4073d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> v1:172.21.15.83:6790/0 conn(0x7f71ec103a20 legacy=0x7f71ec0fcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:41.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 shutdown_connections 2024-09-10T19:06:41.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 >> 172.21.15.83:0/2647152064 conn(0x7f71ec074a80 msgr2=0x7f71ec10a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:41.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 shutdown_connections 2024-09-10T19:06:41.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 -- 172.21.15.83:0/2647152064 wait complete. 2024-09-10T19:06:41.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:41.286+0000 7f71f1c5b640 1 librados: shutdown 2024-09-10T19:06:41.361 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.383 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:41.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:41.519 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:41.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:41.636 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:41.659 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:41.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:41.799 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:41.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.939 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:41.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:41.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:42.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: audit 2024-09-10T19:06:41.112509+0000 mon.c (mon.2) 777 : audit [DBG] from='client.? v1:172.21.15.83:0/2647152064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:42.034 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: cluster 2024-09-10T19:06:41.114964+0000 osd.1 (osd.1) 395 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:06:42.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:42.073 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:42.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.212 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: audit 2024-09-10T19:06:41.112509+0000 mon.c (mon.2) 777 : audit [DBG] from='client.? v1:172.21.15.83:0/2647152064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: cluster 2024-09-10T19:06:41.114964+0000 osd.1 (osd.1) 395 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: cluster 2024-09-10T19:06:41.358479+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: audit 2024-09-10T19:06:41.364821+0000 mon.c (mon.2) 778 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: audit 2024-09-10T19:06:41.368207+0000 mon.a (mon.0) 3441 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: cluster 2024-09-10T19:06:41.384200+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T19:06:42.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18629]: cluster 2024-09-10T19:06:41.386456+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: audit 2024-09-10T19:06:41.112509+0000 mon.c (mon.2) 777 : audit [DBG] from='client.? v1:172.21.15.83:0/2647152064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: cluster 2024-09-10T19:06:41.114964+0000 osd.1 (osd.1) 395 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: cluster 2024-09-10T19:06:41.358479+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: audit 2024-09-10T19:06:41.364821+0000 mon.c (mon.2) 778 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: audit 2024-09-10T19:06:41.368207+0000 mon.a (mon.0) 3441 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: cluster 2024-09-10T19:06:41.384200+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T19:06:42.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[32381]: cluster 2024-09-10T19:06:41.386456+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T19:06:42.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: cluster 2024-09-10T19:06:41.358479+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:06:42.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: audit 2024-09-10T19:06:41.364821+0000 mon.c (mon.2) 778 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: audit 2024-09-10T19:06:41.368207+0000 mon.a (mon.0) 3441 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:42.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: cluster 2024-09-10T19:06:41.384200+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.17 deep-scrub starts 2024-09-10T19:06:42.329 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:42 smithi165 bash[22591]: cluster 2024-09-10T19:06:41.386456+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.17 deep-scrub ok 2024-09-10T19:06:42.349 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:42.453 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.484 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:42.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:42.622 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:42.725 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:42.757 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:42.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:42.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:42.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:42.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:43.011 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:06:43.025 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:42] "GET /metrics HTTP/1.1" 200 229054 "" "Prometheus/2.43.0" 2024-09-10T19:06:43.036 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:43.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:43.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:43.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.312 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:43.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:43.449 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:43.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.129897+0000 osd.3 (osd.3) 418 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:06:43.566 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.134990+0000 osd.3 (osd.3) 419 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.195664+0000 osd.2 (osd.2) 303 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.199195+0000 osd.2 (osd.2) 304 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.323839+0000 mgr.y (mgr.24526) 684 : cluster [DBG] pgmap v1149: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: audit 2024-09-10T19:06:42.361966+0000 mon.a (mon.0) 3442 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: cluster 2024-09-10T19:06:42.362236+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: audit 2024-09-10T19:06:42.368093+0000 mon.c (mon.2) 779 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:43 smithi165 bash[22591]: audit 2024-09-10T19:06:42.368795+0000 mon.a (mon.0) 3444 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:43.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.719 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.129897+0000 osd.3 (osd.3) 418 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.134990+0000 osd.3 (osd.3) 419 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.195664+0000 osd.2 (osd.2) 303 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.199195+0000 osd.2 (osd.2) 304 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.323839+0000 mgr.y (mgr.24526) 684 : cluster [DBG] pgmap v1149: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: audit 2024-09-10T19:06:42.361966+0000 mon.a (mon.0) 3442 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: cluster 2024-09-10T19:06:42.362236+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: audit 2024-09-10T19:06:42.368093+0000 mon.c (mon.2) 779 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[18629]: audit 2024-09-10T19:06:42.368795+0000 mon.a (mon.0) 3444 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.129897+0000 osd.3 (osd.3) 418 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.134990+0000 osd.3 (osd.3) 419 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.195664+0000 osd.2 (osd.2) 303 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.199195+0000 osd.2 (osd.2) 304 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.323839+0000 mgr.y (mgr.24526) 684 : cluster [DBG] pgmap v1149: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: audit 2024-09-10T19:06:42.361966+0000 mon.a (mon.0) 3442 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: cluster 2024-09-10T19:06:42.362236+0000 mon.a (mon.0) 3443 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: audit 2024-09-10T19:06:42.368093+0000 mon.c (mon.2) 779 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:43 smithi083 bash[32381]: audit 2024-09-10T19:06:42.368795+0000 mon.a (mon.0) 3444 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:43.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.857 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:43.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:43.960 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:43.997 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:44.100 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:44.130 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:44.234 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:44.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:44.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:06:44.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:44.412 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:44.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: audit 2024-09-10T19:06:43.369290+0000 mon.a (mon.0) 3445 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:43.369465+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: audit 2024-09-10T19:06:43.371870+0000 mon.c (mon.2) 780 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: audit 2024-09-10T19:06:43.379790+0000 mon.a (mon.0) 3447 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:43.396997+0000 osd.0 (osd.0) 373 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:43.403290+0000 osd.0 (osd.0) 374 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:43.977853+0000 osd.7 (osd.7) 393 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:43.983493+0000 osd.7 (osd.7) 394 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:06:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[32381]: cluster 2024-09-10T19:06:44.330366+0000 mon.a (mon.0) 3448 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:44.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: audit 2024-09-10T19:06:43.369290+0000 mon.a (mon.0) 3445 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:44.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:43.369465+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:06:44.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: audit 2024-09-10T19:06:43.371870+0000 mon.c (mon.2) 780 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: audit 2024-09-10T19:06:43.379790+0000 mon.a (mon.0) 3447 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:43.396997+0000 osd.0 (osd.0) 373 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:06:44.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:43.403290+0000 osd.0 (osd.0) 374 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:06:44.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:43.977853+0000 osd.7 (osd.7) 393 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:06:44.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:43.983493+0000 osd.7 (osd.7) 394 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:06:44.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:44 smithi083 bash[18629]: cluster 2024-09-10T19:06:44.330366+0000 mon.a (mon.0) 3448 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:44.550 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:44.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:44.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: audit 2024-09-10T19:06:43.369290+0000 mon.a (mon.0) 3445 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:44.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:43.369465+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:06:44.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: audit 2024-09-10T19:06:43.371870+0000 mon.c (mon.2) 780 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.668 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: audit 2024-09-10T19:06:43.379790+0000 mon.a (mon.0) 3447 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:44.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:43.396997+0000 osd.0 (osd.0) 373 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:06:44.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:43.403290+0000 osd.0 (osd.0) 374 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:06:44.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:43.977853+0000 osd.7 (osd.7) 393 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:06:44.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:43.983493+0000 osd.7 (osd.7) 394 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:06:44.669 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:44 smithi165 bash[22591]: cluster 2024-09-10T19:06:44.330366+0000 mon.a (mon.0) 3448 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:44.688 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:44.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:44.827 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:44.842 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:44.931 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:44.965 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:44.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:45.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:45.102 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:45.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:45.238 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:45.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:45.366 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:45.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:45.504 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: cluster 2024-09-10T19:06:44.169634+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.c scrub starts 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: cluster 2024-09-10T19:06:44.173152+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.c scrub ok 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: cluster 2024-09-10T19:06:44.324727+0000 mgr.y (mgr.24526) 685 : cluster [DBG] pgmap v1152: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: audit 2024-09-10T19:06:44.394774+0000 mon.a (mon.0) 3449 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: cluster 2024-09-10T19:06:44.395012+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: audit 2024-09-10T19:06:44.396052+0000 mon.c (mon.2) 781 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: audit 2024-09-10T19:06:44.396574+0000 mon.a (mon.0) 3451 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: audit 2024-09-10T19:06:44.581424+0000 mon.a (mon.0) 3452 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[18629]: audit 2024-09-10T19:06:44.584499+0000 mon.c (mon.2) 782 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:45.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: cluster 2024-09-10T19:06:44.169634+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.c scrub starts 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: cluster 2024-09-10T19:06:44.173152+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.c scrub ok 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: cluster 2024-09-10T19:06:44.324727+0000 mgr.y (mgr.24526) 685 : cluster [DBG] pgmap v1152: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: audit 2024-09-10T19:06:44.394774+0000 mon.a (mon.0) 3449 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: cluster 2024-09-10T19:06:44.395012+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: audit 2024-09-10T19:06:44.396052+0000 mon.c (mon.2) 781 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: audit 2024-09-10T19:06:44.396574+0000 mon.a (mon.0) 3451 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: audit 2024-09-10T19:06:44.581424+0000 mon.a (mon.0) 3452 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:45 smithi083 bash[32381]: audit 2024-09-10T19:06:44.584499+0000 mon.c (mon.2) 782 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:45.609 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:45.642 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:45.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:45.783 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: cluster 2024-09-10T19:06:44.169634+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.c scrub starts 2024-09-10T19:06:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: cluster 2024-09-10T19:06:44.173152+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.c scrub ok 2024-09-10T19:06:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: cluster 2024-09-10T19:06:44.324727+0000 mgr.y (mgr.24526) 685 : cluster [DBG] pgmap v1152: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: audit 2024-09-10T19:06:44.394774+0000 mon.a (mon.0) 3449 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: cluster 2024-09-10T19:06:44.395012+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:06:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: audit 2024-09-10T19:06:44.396052+0000 mon.c (mon.2) 781 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: audit 2024-09-10T19:06:44.396574+0000 mon.a (mon.0) 3451 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]: dispatch 2024-09-10T19:06:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: audit 2024-09-10T19:06:44.581424+0000 mon.a (mon.0) 3452 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:06:45.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:45 smithi165 bash[22591]: audit 2024-09-10T19:06:44.584499+0000 mon.c (mon.2) 782 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:45.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:45.916 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:46.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:46.048 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:46.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:46.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:46.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:46.326 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:46.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.335181+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.340039+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.388054+0000 mon.a (mon.0) 3453 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: audit 2024-09-10T19:06:45.399597+0000 mon.a (mon.0) 3454 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]': finished 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.399822+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:06:46.443 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.646428+0000 osd.4 (osd.4) 386 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:06:46.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.656791+0000 osd.4 (osd.4) 387 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:06:46.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.990192+0000 osd.7 (osd.7) 395 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:06:46.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:45.993843+0000 osd.7 (osd.7) 396 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:06:46.444 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:46 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.326431+0000 mgr.y (mgr.24526) 686 : cluster [DBG] pgmap v1155: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:46.462 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:46.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:46.604 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:46.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:46.742 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.335181+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.340039+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.388054+0000 mon.a (mon.0) 3453 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: audit 2024-09-10T19:06:45.399597+0000 mon.a (mon.0) 3454 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]': finished 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.399822+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:06:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.646428+0000 osd.4 (osd.4) 386 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.656791+0000 osd.4 (osd.4) 387 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.990192+0000 osd.7 (osd.7) 395 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:45.993843+0000 osd.7 (osd.7) 396 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.326431+0000 mgr.y (mgr.24526) 686 : cluster [DBG] pgmap v1155: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.335181+0000 osd.0 (osd.0) 375 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.340039+0000 osd.0 (osd.0) 376 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.388054+0000 mon.a (mon.0) 3453 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: audit 2024-09-10T19:06:45.399597+0000 mon.a (mon.0) 3454 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-132", "mode": "writeback"}]': finished 2024-09-10T19:06:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.399822+0000 mon.a (mon.0) 3455 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:06:46.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.646428+0000 osd.4 (osd.4) 386 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:06:46.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.656791+0000 osd.4 (osd.4) 387 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:06:46.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.990192+0000 osd.7 (osd.7) 395 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:06:46.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:45.993843+0000 osd.7 (osd.7) 396 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:06:46.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:46 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.326431+0000 mgr.y (mgr.24526) 686 : cluster [DBG] pgmap v1155: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:46.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:46.871 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:46.878 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:46.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:46.982 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:47.015 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:47.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:47.151 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:47.255 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:47.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:47.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:47.428 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.123179+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.126817+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.377523+0000 osd.0 (osd.0) 377 : cluster [DBG] 12.19 deep-scrub starts 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.380918+0000 osd.0 (osd.0) 378 : cluster [DBG] 12.19 deep-scrub ok 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.420277+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.612558+0000 osd.4 (osd.4) 388 : cluster [DBG] 1.5 deep-scrub starts 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:46.615563+0000 osd.4 (osd.4) 389 : cluster [DBG] 1.5 deep-scrub ok 2024-09-10T19:06:47.466 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.001551+0000 osd.5 (osd.5) 404 : cluster [DBG] 241.7 deep-scrub starts 2024-09-10T19:06:47.467 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.004341+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:06:47.467 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.006560+0000 osd.5 (osd.5) 405 : cluster [DBG] 241.7 deep-scrub ok 2024-09-10T19:06:47.467 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.011522+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:06:47.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:47.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.123179+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:06:47.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.126817+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.377523+0000 osd.0 (osd.0) 377 : cluster [DBG] 12.19 deep-scrub starts 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.380918+0000 osd.0 (osd.0) 378 : cluster [DBG] 12.19 deep-scrub ok 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.420277+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.612558+0000 osd.4 (osd.4) 388 : cluster [DBG] 1.5 deep-scrub starts 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:46.615563+0000 osd.4 (osd.4) 389 : cluster [DBG] 1.5 deep-scrub ok 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.001551+0000 osd.5 (osd.5) 404 : cluster [DBG] 241.7 deep-scrub starts 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.004341+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.006560+0000 osd.5 (osd.5) 405 : cluster [DBG] 241.7 deep-scrub ok 2024-09-10T19:06:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:47 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.011522+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:06:47.566 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:47.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:47.697 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:47.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:47.813 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.123179+0000 osd.3 (osd.3) 420 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.126817+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.377523+0000 osd.0 (osd.0) 377 : cluster [DBG] 12.19 deep-scrub starts 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.380918+0000 osd.0 (osd.0) 378 : cluster [DBG] 12.19 deep-scrub ok 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.420277+0000 mon.a (mon.0) 3456 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.612558+0000 osd.4 (osd.4) 388 : cluster [DBG] 1.5 deep-scrub starts 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:46.615563+0000 osd.4 (osd.4) 389 : cluster [DBG] 1.5 deep-scrub ok 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.001551+0000 osd.5 (osd.5) 404 : cluster [DBG] 241.7 deep-scrub starts 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.004341+0000 osd.7 (osd.7) 397 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.006560+0000 osd.5 (osd.5) 405 : cluster [DBG] 241.7 deep-scrub ok 2024-09-10T19:06:47.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:47 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.011522+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:06:47.836 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:47.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:47.974 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:47.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:48.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:48.112 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:48.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:48.215 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:48.251 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:48.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:48.390 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:48.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:48.523 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:48.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.423583+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: audit 2024-09-10T19:06:47.531481+0000 mon.c (mon.2) 783 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: audit 2024-09-10T19:06:47.531967+0000 mon.a (mon.0) 3458 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.844101+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub starts 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: cluster 2024-09-10T19:06:47.846679+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub ok 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.327234+0000 mgr.y (mgr.24526) 687 : cluster [DBG] pgmap v1158: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:48 smithi165 bash[22591]: audit 2024-09-10T19:06:48.351380+0000 mgr.y (mgr.24526) 688 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:48.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.423583+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: audit 2024-09-10T19:06:47.531481+0000 mon.c (mon.2) 783 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: audit 2024-09-10T19:06:47.531967+0000 mon.a (mon.0) 3458 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.844101+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub starts 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: cluster 2024-09-10T19:06:47.846679+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub ok 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.327234+0000 mgr.y (mgr.24526) 687 : cluster [DBG] pgmap v1158: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[18629]: audit 2024-09-10T19:06:48.351380+0000 mgr.y (mgr.24526) 688 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.423583+0000 mon.a (mon.0) 3457 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: audit 2024-09-10T19:06:47.531481+0000 mon.c (mon.2) 783 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: audit 2024-09-10T19:06:47.531967+0000 mon.a (mon.0) 3458 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:48.642 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.844101+0000 osd.6 (osd.6) 355 : cluster [DBG] 4.a scrub starts 2024-09-10T19:06:48.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: cluster 2024-09-10T19:06:47.846679+0000 osd.6 (osd.6) 356 : cluster [DBG] 4.a scrub ok 2024-09-10T19:06:48.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.327234+0000 mgr.y (mgr.24526) 687 : cluster [DBG] pgmap v1158: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:48.643 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:48 smithi083 bash[32381]: audit 2024-09-10T19:06:48.351380+0000 mgr.y (mgr.24526) 688 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:48.660 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:48.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:48.800 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:48.904 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:48.936 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:48.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:49.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:49.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:49.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:49.214 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:49.317 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:49.353 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:49.456 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:49.487 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:49.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:49.625 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:49.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.123737+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.127897+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.193922+0000 osd.3 (osd.3) 422 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.197378+0000 osd.3 (osd.3) 423 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.334857+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.b scrub starts 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.338284+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.b scrub ok 2024-09-10T19:06:49.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: audit 2024-09-10T19:06:48.441264+0000 mon.a (mon.0) 3459 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:48.441398+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: audit 2024-09-10T19:06:48.494248+0000 mon.c (mon.2) 784 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: audit 2024-09-10T19:06:48.494865+0000 mon.a (mon.0) 3461 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:49.332650+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: audit 2024-09-10T19:06:49.344525+0000 mon.a (mon.0) 3463 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:49.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:49 smithi165 bash[22591]: cluster 2024-09-10T19:06:49.344684+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:06:49.760 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:49.863 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.123737+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.127897+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.193922+0000 osd.3 (osd.3) 422 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.197378+0000 osd.3 (osd.3) 423 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.334857+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.b scrub starts 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.338284+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.b scrub ok 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: audit 2024-09-10T19:06:48.441264+0000 mon.a (mon.0) 3459 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:48.441398+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:06:49.876 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: audit 2024-09-10T19:06:48.494248+0000 mon.c (mon.2) 784 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: audit 2024-09-10T19:06:48.494865+0000 mon.a (mon.0) 3461 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:49.332650+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: audit 2024-09-10T19:06:49.344525+0000 mon.a (mon.0) 3463 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[18629]: cluster 2024-09-10T19:06:49.344684+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.123737+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.127897+0000 osd.1 (osd.1) 397 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.193922+0000 osd.3 (osd.3) 422 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.197378+0000 osd.3 (osd.3) 423 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.334857+0000 osd.0 (osd.0) 379 : cluster [DBG] 4.b scrub starts 2024-09-10T19:06:49.877 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.338284+0000 osd.0 (osd.0) 380 : cluster [DBG] 4.b scrub ok 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: audit 2024-09-10T19:06:48.441264+0000 mon.a (mon.0) 3459 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:48.441398+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: audit 2024-09-10T19:06:48.494248+0000 mon.c (mon.2) 784 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: audit 2024-09-10T19:06:48.494865+0000 mon.a (mon.0) 3461 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:49.332650+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: audit 2024-09-10T19:06:49.344525+0000 mon.a (mon.0) 3463 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:49.878 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:49 smithi083 bash[32381]: cluster 2024-09-10T19:06:49.344684+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:06:49.893 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:49.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:49.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:50.026 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:50.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:50.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:50.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:50.295 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:50.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:50.430 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:50.533 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:50.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: cluster 2024-09-10T19:06:49.168432+0000 osd.1 (osd.1) 398 : cluster [DBG] 241.6 scrub starts 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: cluster 2024-09-10T19:06:49.172937+0000 osd.1 (osd.1) 399 : cluster [DBG] 241.6 scrub ok 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.328065+0000 mgr.y (mgr.24526) 689 : cluster [DBG] pgmap v1161: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.345382+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: audit 2024-09-10T19:06:50.374705+0000 mon.c (mon.2) 785 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[18629]: audit 2024-09-10T19:06:50.375110+0000 mon.a (mon.0) 3466 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: cluster 2024-09-10T19:06:49.168432+0000 osd.1 (osd.1) 398 : cluster [DBG] 241.6 scrub starts 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: cluster 2024-09-10T19:06:49.172937+0000 osd.1 (osd.1) 399 : cluster [DBG] 241.6 scrub ok 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.328065+0000 mgr.y (mgr.24526) 689 : cluster [DBG] pgmap v1161: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:50.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.345382+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:06:50.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: audit 2024-09-10T19:06:50.374705+0000 mon.c (mon.2) 785 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:50 smithi083 bash[32381]: audit 2024-09-10T19:06:50.375110+0000 mon.a (mon.0) 3466 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.564 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:50.668 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:50.702 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:50.810 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:50.846 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: cluster 2024-09-10T19:06:49.168432+0000 osd.1 (osd.1) 398 : cluster [DBG] 241.6 scrub starts 2024-09-10T19:06:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: cluster 2024-09-10T19:06:49.172937+0000 osd.1 (osd.1) 399 : cluster [DBG] 241.6 scrub ok 2024-09-10T19:06:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.328065+0000 mgr.y (mgr.24526) 689 : cluster [DBG] pgmap v1161: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.345382+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:06:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: audit 2024-09-10T19:06:50.374705+0000 mon.c (mon.2) 785 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:50 smithi165 bash[22591]: audit 2024-09-10T19:06:50.375110+0000 mon.a (mon.0) 3466 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:50.950 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:50.978 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:50.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:51.082 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:51.116 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:51.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:51.256 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:51.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:51.391 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:51.409 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:51.494 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:51.528 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:51.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:51.663 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.062827+0000 osd.2 (osd.2) 307 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.067873+0000 osd.2 (osd.2) 308 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.187540+0000 osd.1 (osd.1) 400 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.191194+0000 osd.1 (osd.1) 401 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.851786+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:50.856436+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:06:51.665 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:51.094083+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:51.097397+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: audit 2024-09-10T19:06:51.348143+0000 mon.a (mon.0) 3467 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: cluster 2024-09-10T19:06:51.348417+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: audit 2024-09-10T19:06:51.350450+0000 mon.c (mon.2) 786 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[18629]: audit 2024-09-10T19:06:51.351016+0000 mon.a (mon.0) 3469 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.062827+0000 osd.2 (osd.2) 307 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.067873+0000 osd.2 (osd.2) 308 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.187540+0000 osd.1 (osd.1) 400 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:06:51.666 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.191194+0000 osd.1 (osd.1) 401 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.851786+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:50.856436+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:51.094083+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:51.097397+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: audit 2024-09-10T19:06:51.348143+0000 mon.a (mon.0) 3467 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: cluster 2024-09-10T19:06:51.348417+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: audit 2024-09-10T19:06:51.350450+0000 mon.c (mon.2) 786 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.667 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:51 smithi083 bash[32381]: audit 2024-09-10T19:06:51.351016+0000 mon.a (mon.0) 3469 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:51.769 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:51.801 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:51.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.062827+0000 osd.2 (osd.2) 307 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:06:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.067873+0000 osd.2 (osd.2) 308 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:06:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.187540+0000 osd.1 (osd.1) 400 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.191194+0000 osd.1 (osd.1) 401 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.851786+0000 osd.6 (osd.6) 357 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:50.856436+0000 osd.6 (osd.6) 358 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:51.094083+0000 osd.5 (osd.5) 406 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:51.097397+0000 osd.5 (osd.5) 407 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: audit 2024-09-10T19:06:51.348143+0000 mon.a (mon.0) 3467 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: cluster 2024-09-10T19:06:51.348417+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: audit 2024-09-10T19:06:51.350450+0000 mon.c (mon.2) 786 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:51 smithi165 bash[22591]: audit 2024-09-10T19:06:51.351016+0000 mon.a (mon.0) 3469 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]: dispatch 2024-09-10T19:06:51.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:51.934 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:51.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:52.038 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:52.070 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:52.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:52.210 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:52.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:52.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:52.348 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:52.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:52.485 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:52.588 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:52.620 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:51.682881+0000 osd.4 (osd.4) 390 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:51.691252+0000 osd.4 (osd.4) 391 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.114566+0000 osd.7 (osd.7) 399 : cluster [DBG] 241.5 scrub starts 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.119038+0000 osd.7 (osd.7) 400 : cluster [DBG] 241.5 scrub ok 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.329006+0000 mgr.y (mgr.24526) 690 : cluster [DBG] pgmap v1164: 214 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 208 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:06:52.674 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.342081+0000 mon.a (mon.0) 3470 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: audit 2024-09-10T19:06:52.352125+0000 mon.a (mon.0) 3471 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.352240+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:51.682881+0000 osd.4 (osd.4) 390 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:51.691252+0000 osd.4 (osd.4) 391 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.114566+0000 osd.7 (osd.7) 399 : cluster [DBG] 241.5 scrub starts 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.119038+0000 osd.7 (osd.7) 400 : cluster [DBG] 241.5 scrub ok 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.329006+0000 mgr.y (mgr.24526) 690 : cluster [DBG] pgmap v1164: 214 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 208 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.342081+0000 mon.a (mon.0) 3470 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: audit 2024-09-10T19:06:52.352125+0000 mon.a (mon.0) 3471 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:52.675 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.352240+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:06:52.723 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:52.760 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:52.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:51.682881+0000 osd.4 (osd.4) 390 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:51.691252+0000 osd.4 (osd.4) 391 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.114566+0000 osd.7 (osd.7) 399 : cluster [DBG] 241.5 scrub starts 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.119038+0000 osd.7 (osd.7) 400 : cluster [DBG] 241.5 scrub ok 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.329006+0000 mgr.y (mgr.24526) 690 : cluster [DBG] pgmap v1164: 214 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 208 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.342081+0000 mon.a (mon.0) 3470 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:52.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: audit 2024-09-10T19:06:52.352125+0000 mon.a (mon.0) 3471 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-132"}]': finished 2024-09-10T19:06:52.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:52 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.352240+0000 mon.a (mon.0) 3472 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:06:52.893 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:52.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:52.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.011 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:06:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:06:52] "GET /metrics HTTP/1.1" 200 229013 "" "Prometheus/2.43.0" 2024-09-10T19:06:53.030 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:53.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:53.169 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:53.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:53.308 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:53.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:53.441 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:53.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.578 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:53.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:53.711 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:53.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:53.848 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:53.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.960 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.162236+0000 osd.3 (osd.3) 424 : cluster [DBG] 2.19 deep-scrub starts 2024-09-10T19:06:53.960 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.165612+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.19 deep-scrub ok 2024-09-10T19:06:53.960 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.413418+0000 osd.2 (osd.2) 309 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:53.960 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.413876+0000 osd.2 (osd.2) 310 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:53.960 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.868081+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:52.872465+0000 osd.6 (osd.6) 360 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.012517+0000 osd.5 (osd.5) 408 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.015979+0000 osd.5 (osd.5) 409 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.354552+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.162236+0000 osd.3 (osd.3) 424 : cluster [DBG] 2.19 deep-scrub starts 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.165612+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.19 deep-scrub ok 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.413418+0000 osd.2 (osd.2) 309 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.413876+0000 osd.2 (osd.2) 310 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.868081+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:52.872465+0000 osd.6 (osd.6) 360 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:06:53.961 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.012517+0000 osd.5 (osd.5) 408 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:06:53.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.015979+0000 osd.5 (osd.5) 409 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:06:53.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:53 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.354552+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:06:53.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:53.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:54.074 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:54.110 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.162236+0000 osd.3 (osd.3) 424 : cluster [DBG] 2.19 deep-scrub starts 2024-09-10T19:06:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.165612+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.19 deep-scrub ok 2024-09-10T19:06:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.413418+0000 osd.2 (osd.2) 309 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:06:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.413876+0000 osd.2 (osd.2) 310 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:06:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.868081+0000 osd.6 (osd.6) 359 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:06:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:52.872465+0000 osd.6 (osd.6) 360 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:06:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.012517+0000 osd.5 (osd.5) 408 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:06:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.015979+0000 osd.5 (osd.5) 409 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:06:54.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:53 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.354552+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:06:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:54.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:54.248 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:54.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:54.383 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:54.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:54.519 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:54.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:54.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:54.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:54.797 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:54.902 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.343356+0000 osd.0 (osd.0) 381 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.347347+0000 osd.0 (osd.0) 382 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.827881+0000 osd.6 (osd.6) 361 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.837823+0000 osd.6 (osd.6) 362 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.987225+0000 osd.5 (osd.5) 410 : cluster [DBG] 6.a scrub starts 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:53.989386+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.a scrub ok 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.078060+0000 osd.2 (osd.2) 311 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.081801+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.329873+0000 mgr.y (mgr.24526) 691 : cluster [DBG] pgmap v1167: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.364479+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: audit 2024-09-10T19:06:54.369092+0000 mon.c (mon.2) 787 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:54.916 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[18629]: audit 2024-09-10T19:06:54.371854+0000 mon.a (mon.0) 3475 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.343356+0000 osd.0 (osd.0) 381 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.347347+0000 osd.0 (osd.0) 382 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.827881+0000 osd.6 (osd.6) 361 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.837823+0000 osd.6 (osd.6) 362 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.987225+0000 osd.5 (osd.5) 410 : cluster [DBG] 6.a scrub starts 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:53.989386+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.a scrub ok 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.078060+0000 osd.2 (osd.2) 311 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.081801+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.329873+0000 mgr.y (mgr.24526) 691 : cluster [DBG] pgmap v1167: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.364479+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:06:54.917 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: audit 2024-09-10T19:06:54.369092+0000 mon.c (mon.2) 787 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:54.918 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:54 smithi083 bash[32381]: audit 2024-09-10T19:06:54.371854+0000 mon.a (mon.0) 3475 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:54.934 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:54.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:55.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:55.070 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:55.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.343356+0000 osd.0 (osd.0) 381 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:06:55.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.347347+0000 osd.0 (osd.0) 382 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:06:55.072 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.827881+0000 osd.6 (osd.6) 361 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.837823+0000 osd.6 (osd.6) 362 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.987225+0000 osd.5 (osd.5) 410 : cluster [DBG] 6.a scrub starts 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:53.989386+0000 osd.5 (osd.5) 411 : cluster [DBG] 6.a scrub ok 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.078060+0000 osd.2 (osd.2) 311 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.081801+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.329873+0000 mgr.y (mgr.24526) 691 : cluster [DBG] pgmap v1167: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.364479+0000 mon.a (mon.0) 3474 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: audit 2024-09-10T19:06:54.369092+0000 mon.c (mon.2) 787 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:55.073 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:54 smithi165 bash[22591]: audit 2024-09-10T19:06:54.371854+0000 mon.a (mon.0) 3475 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:55.175 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:55.209 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:55.312 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:55.344 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:55.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:55.488 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:55.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:55.627 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:55.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:55.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.742+0000 7f71370d1640 1 -- 172.21.15.83:0/926857733 >> v1:172.21.15.83:6790/0 conn(0x7f7130107de0 legacy=0x7f713010a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:55.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.742+0000 7f71370d1640 1 -- 172.21.15.83:0/926857733 shutdown_connections 2024-09-10T19:06:55.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.742+0000 7f71370d1640 1 -- 172.21.15.83:0/926857733 >> 172.21.15.83:0/926857733 conn(0x7f71300f7090 msgr2=0x7f71300f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:55.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.742+0000 7f71370d1640 1 -- 172.21.15.83:0/926857733 shutdown_connections 2024-09-10T19:06:55.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.742+0000 7f71370d1640 1 -- 172.21.15.83:0/926857733 wait complete. 2024-09-10T19:06:55.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 librados: starting msgr at 2024-09-10T19:06:55.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 librados: starting objecter 2024-09-10T19:06:55.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 Processor -- start 2024-09-10T19:06:55.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- start start 2024-09-10T19:06:55.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 librados: setting wanted keys 2024-09-10T19:06:55.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 librados: calling monclient init 2024-09-10T19:06:55.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713011c270 con 0x7f7130107de0 2024-09-10T19:06:55.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713011d410 con 0x7f7130104280 2024-09-10T19:06:55.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f713011e5f0 con 0x7f7130101510 2024-09-10T19:06:55.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f7134e6d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f7130101510 0x7f7130107180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45496/0 (socket says 172.21.15.83:45496) 2024-09-10T19:06:55.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f7134e6d640 1 -- 172.21.15.83:0/845998119 learned_addr learned my addr 172.21.15.83:0/845998119 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter ms_handle_connect 0x7f7130101510 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter ms_handle_connect 0x7f7130107de0 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter ms_handle_connect 0x7f7130104280 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785806472 0 0) 0x7f713011e5f0 con 0x7f7130101510 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711c003720 con 0x7f7130101510 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091582969 0 0) 0x7f713011d410 con 0x7f7130104280 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713011e5f0 con 0x7f7130104280 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865996212 0 0) 0x7f713011c270 con 0x7f7130107de0 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713011d410 con 0x7f7130107de0 2024-09-10T19:06:55.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3794782183 0 0) 0x7f711c003720 con 0x7f7130101510 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f713011c270 con 0x7f7130101510 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7128001c30 con 0x7f7130101510 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 560248813 0 0) 0x7f713011c270 con 0x7f7130101510 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 >> v1:172.21.15.165:6789/0 conn(0x7f7130104280 legacy=0x7f7130117420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 >> v1:172.21.15.83:6789/0 conn(0x7f7130107de0 legacy=0x7f713011ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713011f7d0 con 0x7f7130101510 2024-09-10T19:06:55.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713011d600 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f71280026e0 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7128005df0 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 10 client.49793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f713011da50 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.746+0000 7f71370d1640 1 librados: init done 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7128002280 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71300fcc10 con 0x7f7130101510 2024-09-10T19:06:55.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(756..756 src has 704..756) v4 ==== 8564+0+0 (unknown 2014160745 0 0) 0x7f712808ad30 con 0x7f7130101510 2024-09-10T19:06:55.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 10 client.49793.objecter ms_dispatch 0x7f7130000b90 osd_map(756..756 src has 704..756) v4 2024-09-10T19:06:55.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 3 client.49793.objecter handle_osd_map got epochs [756,756] > 0 2024-09-10T19:06:55.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 3 client.49793.objecter handle_osd_map decoding full epoch 756 2024-09-10T19:06:55.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 20 client.49793.objecter dump_active .. 0 homeless 2024-09-10T19:06:55.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.750+0000 7f712d7fa640 10 client.49793.objecter ms_handle_connect 0x7f711c0715c0 2024-09-10T19:06:55.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.754+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7128056c60 con 0x7f7130101510 2024-09-10T19:06:55.766 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:06:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:06:55.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:55.901 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:55.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:55.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:55.946+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f713004bf60 con 0x7f7130101510 2024-09-10T19:06:56.004 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:56.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.219275+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:06:56.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.222832+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:06:56.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.649561+0000 osd.4 (osd.4) 392 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:06:56.018 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:54.652929+0000 osd.4 (osd.4) 393 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.017273+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.7 deep-scrub starts 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.021195+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.7 deep-scrub ok 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: audit 2024-09-10T19:06:55.367121+0000 mon.a (mon.0) 3476 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.367325+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: audit 2024-09-10T19:06:55.369576+0000 mon.c (mon.2) 788 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.019 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[32381]: audit 2024-09-10T19:06:55.370922+0000 mon.a (mon.0) 3478 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.020 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.219275+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:06:56.020 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.222832+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:06:56.020 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.649561+0000 osd.4 (osd.4) 392 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:06:56.020 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:54.652929+0000 osd.4 (osd.4) 393 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:06:56.021 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.017273+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.7 deep-scrub starts 2024-09-10T19:06:56.021 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.021195+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.7 deep-scrub ok 2024-09-10T19:06:56.021 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: audit 2024-09-10T19:06:55.367121+0000 mon.a (mon.0) 3476 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:56.021 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.367325+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:06:56.021 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: audit 2024-09-10T19:06:55.369576+0000 mon.c (mon.2) 788 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.022 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:55 smithi083 bash[18629]: audit 2024-09-10T19:06:55.370922+0000 mon.a (mon.0) 3478 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.039 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.219275+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:06:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.222832+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.649561+0000 osd.4 (osd.4) 392 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:54.652929+0000 osd.4 (osd.4) 393 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.017273+0000 osd.5 (osd.5) 412 : cluster [DBG] 6.7 deep-scrub starts 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.021195+0000 osd.5 (osd.5) 413 : cluster [DBG] 6.7 deep-scrub ok 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: audit 2024-09-10T19:06:55.367121+0000 mon.a (mon.0) 3476 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.367325+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: audit 2024-09-10T19:06:55.369576+0000 mon.c (mon.2) 788 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:55 smithi165 bash[22591]: audit 2024-09-10T19:06:55.370922+0000 mon.a (mon.0) 3478 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:56.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:56.178 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:56.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:56.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.370+0000 7f712d7fa640 1 -- 172.21.15.83:0/845998119 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v757) v1 ==== 113+0+0 (unknown 482030643 0 0) 0x7f712805d4a0 con 0x7f7130101510 2024-09-10T19:06:56.374 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T19:06:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 10 librados: watch_flush enter 2024-09-10T19:06:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 10 librados: watch_flush exit 2024-09-10T19:06:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 20 client.49793.objecter shutdown clearing up homeless session... 2024-09-10T19:06:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 10 client.49793.objecter successfully canceled tick 2024-09-10T19:06:56.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 >> v1:172.21.15.83:6800/3071793735 conn(0x7f711c0715c0 legacy=0x7f711c073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 >> v1:172.21.15.83:6790/0 conn(0x7f7130101510 legacy=0x7f7130107180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 shutdown_connections 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 >> 172.21.15.83:0/845998119 conn(0x7f71300f7090 msgr2=0x7f71300fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 shutdown_connections 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 -- 172.21.15.83:0/845998119 wait complete. 2024-09-10T19:06:56.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:06:56.374+0000 7f71370d1640 1 librados: shutdown 2024-09-10T19:06:56.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:56.455 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:56.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:56.592 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:56.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:56.726 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:56.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:56.859 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:56.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:56.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.201529+0000 osd.1 (osd.1) 404 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:06:56.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.204918+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:55.947984+0000 mon.c (mon.2) 789 : audit [INF] from='client.? v1:172.21.15.83:0/845998119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:55.948554+0000 mon.a (mon.0) 3479 : audit [INF] from='client.49793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.981441+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:55.984754+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.332043+0000 mgr.y (mgr.24526) 692 : cluster [DBG] pgmap v1170: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:56.369033+0000 mon.a (mon.0) 3480 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:56.369290+0000 mon.a (mon.0) 3481 : audit [INF] from='client.49793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.369538+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:56.373753+0000 mon.c (mon.2) 790 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:56.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[18629]: audit 2024-09-10T19:06:56.374476+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.201529+0000 osd.1 (osd.1) 404 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.204918+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:55.947984+0000 mon.c (mon.2) 789 : audit [INF] from='client.? v1:172.21.15.83:0/845998119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:55.948554+0000 mon.a (mon.0) 3479 : audit [INF] from='client.49793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.981441+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:55.984754+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.332043+0000 mgr.y (mgr.24526) 692 : cluster [DBG] pgmap v1170: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:56.369033+0000 mon.a (mon.0) 3480 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:56.369290+0000 mon.a (mon.0) 3481 : audit [INF] from='client.49793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.369538+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:56.373753+0000 mon.c (mon.2) 790 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:56.978 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:56 smithi083 bash[32381]: audit 2024-09-10T19:06:56.374476+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:56.989 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:56.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:57.017 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:06:57.017 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:06:57.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.201529+0000 osd.1 (osd.1) 404 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.204918+0000 osd.1 (osd.1) 405 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:55.947984+0000 mon.c (mon.2) 789 : audit [INF] from='client.? v1:172.21.15.83:0/845998119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:55.948554+0000 mon.a (mon.0) 3479 : audit [INF] from='client.49793 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.981441+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:55.984754+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.332043+0000 mgr.y (mgr.24526) 692 : cluster [DBG] pgmap v1170: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:56.369033+0000 mon.a (mon.0) 3480 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:56.369290+0000 mon.a (mon.0) 3481 : audit [INF] from='client.49793 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.369538+0000 mon.a (mon.0) 3482 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:56.373753+0000 mon.c (mon.2) 790 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:57.107 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:56 smithi165 bash[22591]: audit 2024-09-10T19:06:56.374476+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:06:57.127 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:57.231 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:57.267 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:06:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:06:57.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:57.394 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:57.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.529 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:57.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.668 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:57.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.805 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:57.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:57.943 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:06:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:57.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.242702+0000 osd.1 (osd.1) 406 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.247881+0000 osd.1 (osd.1) 407 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.673577+0000 osd.4 (osd.4) 394 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: cluster 2024-09-10T19:06:56.683582+0000 osd.4 (osd.4) 395 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: audit 2024-09-10T19:06:57.373046+0000 mon.a (mon.0) 3484 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:06:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: cluster 2024-09-10T19:06:57.373222+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: audit 2024-09-10T19:06:57.375699+0000 mon.c (mon.2) 791 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[18629]: audit 2024-09-10T19:06:57.376310+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.242702+0000 osd.1 (osd.1) 406 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.247881+0000 osd.1 (osd.1) 407 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.673577+0000 osd.4 (osd.4) 394 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: cluster 2024-09-10T19:06:56.683582+0000 osd.4 (osd.4) 395 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: audit 2024-09-10T19:06:57.373046+0000 mon.a (mon.0) 3484 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: cluster 2024-09-10T19:06:57.373222+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: audit 2024-09-10T19:06:57.375699+0000 mon.c (mon.2) 791 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:57 smithi083 bash[32381]: audit 2024-09-10T19:06:57.376310+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.084 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.242702+0000 osd.1 (osd.1) 406 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.247881+0000 osd.1 (osd.1) 407 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.673577+0000 osd.4 (osd.4) 394 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: cluster 2024-09-10T19:06:56.683582+0000 osd.4 (osd.4) 395 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: audit 2024-09-10T19:06:57.373046+0000 mon.a (mon.0) 3484 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: cluster 2024-09-10T19:06:57.373222+0000 mon.a (mon.0) 3485 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:06:58.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: audit 2024-09-10T19:06:57.375699+0000 mon.c (mon.2) 791 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:57 smithi165 bash[22591]: audit 2024-09-10T19:06:57.376310+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]: dispatch 2024-09-10T19:06:58.186 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.221 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:58.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:58.355 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:58.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:58.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:58.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:58.627 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:58.628 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:06:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:58.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:58.768 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:58.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.906 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:58.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:59.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:59.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:57.209074+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:06:59.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:57.213962+0000 osd.7 (osd.7) 402 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:06:59.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.332857+0000 mgr.y (mgr.24526) 693 : cluster [DBG] pgmap v1173: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:59.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: audit 2024-09-10T19:06:58.362722+0000 mgr.y (mgr.24526) 694 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:59.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.365553+0000 mon.a (mon.0) 3487 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: audit 2024-09-10T19:06:58.372071+0000 mon.a (mon.0) 3488 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]': finished 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.372186+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.398980+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.401356+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: audit 2024-09-10T19:06:58.450989+0000 mon.c (mon.2) 792 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:58 smithi165 bash[22591]: audit 2024-09-10T19:06:58.451624+0000 mon.a (mon.0) 3490 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.040 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:59.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:59.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:59.178 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:06:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:06:59.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:57.209074+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:57.213962+0000 osd.7 (osd.7) 402 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.332857+0000 mgr.y (mgr.24526) 693 : cluster [DBG] pgmap v1173: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: audit 2024-09-10T19:06:58.362722+0000 mgr.y (mgr.24526) 694 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.365553+0000 mon.a (mon.0) 3487 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: audit 2024-09-10T19:06:58.372071+0000 mon.a (mon.0) 3488 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]': finished 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.372186+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.398980+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.401356+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: audit 2024-09-10T19:06:58.450989+0000 mon.c (mon.2) 792 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[32381]: audit 2024-09-10T19:06:58.451624+0000 mon.a (mon.0) 3490 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:57.209074+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:57.213962+0000 osd.7 (osd.7) 402 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.332857+0000 mgr.y (mgr.24526) 693 : cluster [DBG] pgmap v1173: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: audit 2024-09-10T19:06:58.362722+0000 mgr.y (mgr.24526) 694 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.365553+0000 mon.a (mon.0) 3487 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: audit 2024-09-10T19:06:58.372071+0000 mon.a (mon.0) 3488 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-134", "mode": "writeback"}]': finished 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.372186+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:06:59.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.398980+0000 osd.0 (osd.0) 383 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:06:59.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.401356+0000 osd.0 (osd.0) 384 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:06:59.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: audit 2024-09-10T19:06:58.450989+0000 mon.c (mon.2) 792 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.293 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:58 smithi083 bash[18629]: audit 2024-09-10T19:06:58.451624+0000 mon.a (mon.0) 3490 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:06:59.316 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:06:59.367 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T19:06:59.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:06:59.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:59.442 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:06:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:06:59.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:59.571 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:59.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:59.708 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:06:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:06:59.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:06:59.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:59.852 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:06:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:06:59.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:59.981 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:06:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:06:59.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:00.085 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.098 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.883760+0000 osd.5 (osd.5) 414 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:07:00.098 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: cluster 2024-09-10T19:06:58.888180+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: audit 2024-09-10T19:06:59.352234+0000 mon.a (mon.0) 3491 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: cluster 2024-09-10T19:06:59.352393+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: audit 2024-09-10T19:06:59.353442+0000 mon.c (mon.2) 793 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: audit 2024-09-10T19:06:59.354020+0000 mon.a (mon.0) 3493 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: audit 2024-09-10T19:06:59.578518+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[18629]: audit 2024-09-10T19:06:59.582674+0000 mon.c (mon.2) 794 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.883760+0000 osd.5 (osd.5) 414 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: cluster 2024-09-10T19:06:58.888180+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: audit 2024-09-10T19:06:59.352234+0000 mon.a (mon.0) 3491 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:00.099 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: cluster 2024-09-10T19:06:59.352393+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:07:00.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: audit 2024-09-10T19:06:59.353442+0000 mon.c (mon.2) 793 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: audit 2024-09-10T19:06:59.354020+0000 mon.a (mon.0) 3493 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: audit 2024-09-10T19:06:59.578518+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:00.100 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:06:59 smithi083 bash[32381]: audit 2024-09-10T19:06:59.582674+0000 mon.c (mon.2) 794 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.883760+0000 osd.5 (osd.5) 414 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: cluster 2024-09-10T19:06:58.888180+0000 osd.5 (osd.5) 415 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: audit 2024-09-10T19:06:59.352234+0000 mon.a (mon.0) 3491 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: cluster 2024-09-10T19:06:59.352393+0000 mon.a (mon.0) 3492 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: audit 2024-09-10T19:06:59.353442+0000 mon.c (mon.2) 793 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: audit 2024-09-10T19:06:59.354020+0000 mon.a (mon.0) 3493 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]: dispatch 2024-09-10T19:07:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: audit 2024-09-10T19:06:59.578518+0000 mon.a (mon.0) 3494 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:06:59 smithi165 bash[22591]: audit 2024-09-10T19:06:59.582674+0000 mon.c (mon.2) 794 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:00.117 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:00.220 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:00.255 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:00.359 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.383 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:00.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:00.523 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:00.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- 172.21.15.83:0/4186328959 >> v1:172.21.15.83:6790/0 conn(0x7efce8102620 legacy=0x7efce81029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:00.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- 172.21.15.83:0/4186328959 shutdown_connections 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- 172.21.15.83:0/4186328959 >> 172.21.15.83:0/4186328959 conn(0x7efce8074a80 msgr2=0x7efce80f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- 172.21.15.83:0/4186328959 shutdown_connections 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- 172.21.15.83:0/4186328959 wait complete. 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 librados: starting msgr at 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 librados: starting objecter 2024-09-10T19:07:00.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 Processor -- start 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- start start 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 librados: setting wanted keys 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 librados: calling monclient init 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efce811bfd0 con 0x7efce8102620 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efce811d1d0 con 0x7efce8108f10 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcef34f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efce811e3d0 con 0x7efce81053b0 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcec8ea640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7efce81053b0 0x7efce80fecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:45522/0 (socket says 172.21.15.83:45522) 2024-09-10T19:07:00.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcec8ea640 1 -- 172.21.15.83:0/3415263065 learned_addr learned my addr 172.21.15.83:0/3415263065 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter ms_handle_connect 0x7efce8102620 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter ms_handle_connect 0x7efce81053b0 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739356055 0 0) 0x7efce811bfd0 con 0x7efce8102620 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003120 con 0x7efce8102620 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter ms_handle_connect 0x7efce8108f10 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131355312 0 0) 0x7efce811e3d0 con 0x7efce81053b0 2024-09-10T19:07:00.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.585+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce811bfd0 con 0x7efce81053b0 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902349160 0 0) 0x7efcd4003120 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efce811e3d0 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efce4001510 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883749044 0 0) 0x7efce811d1d0 con 0x7efce8108f10 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd4003120 con 0x7efce8108f10 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2216725157 0 0) 0x7efce811e3d0 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 >> v1:172.21.15.83:6790/0 conn(0x7efce81053b0 legacy=0x7efce80fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 >> v1:172.21.15.165:6789/0 conn(0x7efce8108f10 legacy=0x7efce811a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce811f5d0 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce811e5c0 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcef34f640 10 client.25713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efce811e9b0 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcef34f640 1 librados: init done 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7efce4002300 con 0x7efce8102620 2024-09-10T19:07:00.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efce4003620 con 0x7efce8102620 2024-09-10T19:07:00.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efccc005010 con 0x7efce8102620 2024-09-10T19:07:00.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7efce4002af0 con 0x7efce8102620 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(761..761 src has 704..761) v4 ==== 8523+0+0 (unknown 1826003215 0 0) 0x7efce40881b0 con 0x7efce8102620 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 10 client.25713.objecter ms_dispatch 0x7efce8000b90 osd_map(761..761 src has 704..761) v4 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 3 client.25713.objecter handle_osd_map got epochs [761,761] > 0 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 3 client.25713.objecter handle_osd_map decoding full epoch 761 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 20 client.25713.objecter dump_active .. 0 homeless 2024-09-10T19:07:00.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.589+0000 7efcddffb640 10 client.25713.objecter ms_handle_connect 0x7efcd4071620 2024-09-10T19:07:00.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.597+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7efce4054580 con 0x7efce8102620 2024-09-10T19:07:00.626 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.656 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:00.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:00.793 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:00.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:00.809+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7efccc0052c0 con 0x7efce8102620 2024-09-10T19:07:00.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:00.932 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:00.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:01.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:06:59.208754+0000 osd.1 (osd.1) 408 : cluster [DBG] 4.f scrub starts 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:06:59.211820+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.f scrub ok 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:06:59.930327+0000 osd.5 (osd.5) 416 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:06:59.942472+0000 osd.5 (osd.5) 417 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:00.147165+0000 osd.3 (osd.3) 426 : cluster [DBG] 6.b scrub starts 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:00.155678+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b scrub ok 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:00.333694+0000 mgr.y (mgr.24526) 695 : cluster [DBG] pgmap v1176: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:00.345409+0000 mon.a (mon.0) 3495 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: audit 2024-09-10T19:07:00.356743+0000 mon.a (mon.0) 3496 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:00.357123+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:07:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[18629]: audit 2024-09-10T19:07:00.811764+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:06:59.208754+0000 osd.1 (osd.1) 408 : cluster [DBG] 4.f scrub starts 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:06:59.211820+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.f scrub ok 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:06:59.930327+0000 osd.5 (osd.5) 416 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:06:59.942472+0000 osd.5 (osd.5) 417 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:00.147165+0000 osd.3 (osd.3) 426 : cluster [DBG] 6.b scrub starts 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:00.155678+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b scrub ok 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:00.333694+0000 mgr.y (mgr.24526) 695 : cluster [DBG] pgmap v1176: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:00.345409+0000 mon.a (mon.0) 3495 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: audit 2024-09-10T19:07:00.356743+0000 mon.a (mon.0) 3496 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:00.357123+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:07:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:00 smithi083 bash[32381]: audit 2024-09-10T19:07:00.811764+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:06:59.208754+0000 osd.1 (osd.1) 408 : cluster [DBG] 4.f scrub starts 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:06:59.211820+0000 osd.1 (osd.1) 409 : cluster [DBG] 4.f scrub ok 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:06:59.930327+0000 osd.5 (osd.5) 416 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:06:59.942472+0000 osd.5 (osd.5) 417 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:00.147165+0000 osd.3 (osd.3) 426 : cluster [DBG] 6.b scrub starts 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:00.155678+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.b scrub ok 2024-09-10T19:07:01.049 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:00.333694+0000 mgr.y (mgr.24526) 695 : cluster [DBG] pgmap v1176: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:01.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:00.345409+0000 mon.a (mon.0) 3495 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:01.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: audit 2024-09-10T19:07:00.356743+0000 mon.a (mon.0) 3496 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-134"}]': finished 2024-09-10T19:07:01.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:00.357123+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:07:01.050 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:00 smithi165 bash[22591]: audit 2024-09-10T19:07:00.811764+0000 mon.a (mon.0) 3498 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:07:01.068 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:01.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.205 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:01.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:01.340 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:01.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.357+0000 7efcddffb640 1 -- 172.21.15.83:0/3415263065 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v762) v1 ==== 87+0+0 (unknown 1745787904 0 0) 0x7efce4053f20 con 0x7efce8102620 2024-09-10T19:07:01.359 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:07:01.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 10 librados: watch_flush enter 2024-09-10T19:07:01.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 10 librados: watch_flush exit 2024-09-10T19:07:01.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 20 client.25713.objecter shutdown clearing up homeless session... 2024-09-10T19:07:01.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 10 client.25713.objecter successfully canceled tick 2024-09-10T19:07:01.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 >> v1:172.21.15.83:6800/3071793735 conn(0x7efcd4071620 legacy=0x7efcd4073ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:01.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 >> v1:172.21.15.83:6789/0 conn(0x7efce8102620 legacy=0x7efce80fe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:01.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 shutdown_connections 2024-09-10T19:07:01.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.361+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 >> 172.21.15.83:0/3415263065 conn(0x7efce8074a80 msgr2=0x7efce80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:01.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.365+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 shutdown_connections 2024-09-10T19:07:01.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.365+0000 7efcef34f640 1 -- 172.21.15.83:0/3415263065 wait complete. 2024-09-10T19:07:01.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:01.365+0000 7efcef34f640 1 librados: shutdown 2024-09-10T19:07:01.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:01.480 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:01.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:01.595 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:01.614 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:01.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.741 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:01.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.879 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:01.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:01.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:02.018 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:01 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.348230+0000 mon.a (mon.0) 3499 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:01 smithi165 bash[22591]: audit 2024-09-10T19:07:01.358698+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:07:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:01 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.358775+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:07:02.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:02.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.348230+0000 mon.a (mon.0) 3499 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:02.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[18629]: audit 2024-09-10T19:07:01.358698+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:07:02.138 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.358775+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:07:02.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.348230+0000 mon.a (mon.0) 3499 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:02.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[32381]: audit 2024-09-10T19:07:01.358698+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.83:0/3415263065' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:07:02.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:01 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.358775+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:07:02.158 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:02.262 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:02.297 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:02.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:02.431 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:02.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:02.535 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:02.571 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:02.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:02.706 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:02.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:02.846 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:02.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:02.984 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:02.986 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:02] "GET /metrics HTTP/1.1" 200 229035 "" "Prometheus/2.43.0" 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.187805+0000 osd.1 (osd.1) 410 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.193194+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.710062+0000 osd.6 (osd.6) 363 : cluster [DBG] 3.a scrub starts 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:01.718000+0000 osd.6 (osd.6) 364 : cluster [DBG] 3.a scrub ok 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:02.334545+0000 mgr.y (mgr.24526) 696 : cluster [DBG] pgmap v1179: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: cluster 2024-09-10T19:07:02.360768+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: audit 2024-09-10T19:07:02.363381+0000 mon.c (mon.2) 795 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[18629]: audit 2024-09-10T19:07:02.364145+0000 mon.a (mon.0) 3503 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:02.987 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.187805+0000 osd.1 (osd.1) 410 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.193194+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.710062+0000 osd.6 (osd.6) 363 : cluster [DBG] 3.a scrub starts 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:01.718000+0000 osd.6 (osd.6) 364 : cluster [DBG] 3.a scrub ok 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:02.334545+0000 mgr.y (mgr.24526) 696 : cluster [DBG] pgmap v1179: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: cluster 2024-09-10T19:07:02.360768+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: audit 2024-09-10T19:07:02.363381+0000 mon.c (mon.2) 795 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:02.988 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:02 smithi083 bash[32381]: audit 2024-09-10T19:07:02.364145+0000 mon.a (mon.0) 3503 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:02.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:03.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.187805+0000 osd.1 (osd.1) 410 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.193194+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.710062+0000 osd.6 (osd.6) 363 : cluster [DBG] 3.a scrub starts 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:01.718000+0000 osd.6 (osd.6) 364 : cluster [DBG] 3.a scrub ok 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:02.334545+0000 mgr.y (mgr.24526) 696 : cluster [DBG] pgmap v1179: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:03.102 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: cluster 2024-09-10T19:07:02.360768+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:07:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: audit 2024-09-10T19:07:02.363381+0000 mon.c (mon.2) 795 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:03.103 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:02 smithi165 bash[22591]: audit 2024-09-10T19:07:02.364145+0000 mon.a (mon.0) 3503 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:03.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:03.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- 172.21.15.83:0/2985965319 >> v1:172.21.15.83:6789/0 conn(0x7f46201053b0 legacy=0x7f4620107790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- 172.21.15.83:0/2985965319 shutdown_connections 2024-09-10T19:07:03.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- 172.21.15.83:0/2985965319 >> 172.21.15.83:0/2985965319 conn(0x7f4620074a80 msgr2=0x7f46200f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:03.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- 172.21.15.83:0/2985965319 shutdown_connections 2024-09-10T19:07:03.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- 172.21.15.83:0/2985965319 wait complete. 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 librados: starting msgr at 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 librados: starting objecter 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 Processor -- start 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- start start 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 librados: setting wanted keys 2024-09-10T19:07:03.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 librados: calling monclient init 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f462011be80 con 0x7f46201053b0 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f462011d040 con 0x7f4620102620 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f4626ce6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f462011e240 con 0x7f4620108f10 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f46201053b0 0x7f46200fe980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50488/0 (socket says 172.21.15.83:50488) 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461ffff640 1 -- 172.21.15.83:0/1844530206 learned_addr learned my addr 172.21.15.83:0/1844530206 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter ms_handle_connect 0x7f46201053b0 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:03.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter ms_handle_connect 0x7f4620108f10 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter ms_handle_connect 0x7f4620102620 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261799699 0 0) 0x7f462011be80 con 0x7f46201053b0 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4608003720 con 0x7f46201053b0 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466338599 0 0) 0x7f462011e240 con 0x7f4620108f10 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462011be80 con 0x7f4620108f10 2024-09-10T19:07:03.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181043888 0 0) 0x7f462011d040 con 0x7f4620102620 2024-09-10T19:07:03.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462011e240 con 0x7f4620102620 2024-09-10T19:07:03.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620346285 0 0) 0x7f4608003720 con 0x7f46201053b0 2024-09-10T19:07:03.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.137+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f462011d040 con 0x7f46201053b0 2024-09-10T19:07:03.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4610002c10 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 988506676 0 0) 0x7f462011d040 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6790/0 conn(0x7f4620108f10 legacy=0x7f462011a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.165:6789/0 conn(0x7f4620102620 legacy=0x7f46200fe270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462011f440 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462011e430 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f4626ce6640 10 client.25719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f462011e880 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f4626ce6640 1 librados: init done 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4610001230 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4610003540 con 0x7f46201053b0 2024-09-10T19:07:03.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f460c005020 con 0x7f46201053b0 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.141+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f46100017f0 con 0x7f46201053b0 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(763..763 src has 704..763) v4 ==== 8523+0+0 (unknown 1849626363 0 0) 0x7f461008a810 con 0x7f46201053b0 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 osd_map(763..763 src has 704..763) v4 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 3 client.25719.objecter handle_osd_map got epochs [763,763] > 0 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 3 client.25719.objecter handle_osd_map decoding full epoch 763 2024-09-10T19:07:03.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 20 client.25719.objecter dump_active .. 0 homeless 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v763) v1 ==== 54+0+16 (unknown 2229522718 0 2420000192) 0x7f461008abd0 con 0x7f46201053b0 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f4608071510 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 10 client.25719.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 1 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 1 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 0 1 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 4 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 10 client.25719.objecter _send_command 1 2024-09-10T19:07:03.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c0052c0 con 0x7f460c004640 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c004640 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c0052c0 con 0x7f460c004640 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 10 client.25719.objecter _finish_command 1 = system:0 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 4 2024-09-10T19:07:03.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.145+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 0 1 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 10 client.25719.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 2 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 2 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c005840 osd=0 3 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 0 2 2024-09-10T19:07:03.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 4 2024-09-10T19:07:03.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 10 client.25719.objecter _send_command 2 2024-09-10T19:07:03.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c004c90 con 0x7f460c004640 2024-09-10T19:07:03.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c004c90 con 0x7f460c004640 2024-09-10T19:07:03.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 10 client.25719.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 4 2024-09-10T19:07:03.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 0 2 2024-09-10T19:07:03.169 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:07:03.169 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 10 client.25719.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 1 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 4 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 10 client.25719.objecter _send_command 3 2024-09-10T19:07:03.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c003b40 con 0x7f460c003610 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.165+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c003610 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.169+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c003b40 con 0x7f460c003610 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.169+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.169+0000 7f461d7fa640 10 client.25719.objecter _finish_command 3 = system:0 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.169+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 4 2024-09-10T19:07:03.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.169+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 1 3 2024-09-10T19:07:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 10 client.25719.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 4 2024-09-10T19:07:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 4 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c004e40 osd=1 3 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 1 4 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 4 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 10 client.25719.objecter _send_command 4 2024-09-10T19:07:03.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.185+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c003090 con 0x7f460c003610 2024-09-10T19:07:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c003090 con 0x7f460c003610 2024-09-10T19:07:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 10 client.25719.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 4 2024-09-10T19:07:03.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 1 4 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 10 client.25719.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 5 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 5 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 2 5 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 4 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 10 client.25719.objecter _send_command 5 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c002380 con 0x7f460c011790 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c011790 2024-09-10T19:07:03.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c002380 con 0x7f460c011790 2024-09-10T19:07:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:07:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 10 client.25719.objecter _finish_command 5 = system:0 2024-09-10T19:07:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 4 2024-09-10T19:07:03.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.189+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 2 5 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 10 client.25719.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 6 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 6 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c001b90 osd=2 3 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 2 6 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 4 2024-09-10T19:07:03.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 10 client.25719.objecter _send_command 6 2024-09-10T19:07:03.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.209+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c001680 con 0x7f460c011790 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c001680 con 0x7f460c011790 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 10 client.25719.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 4 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 2 6 2024-09-10T19:07:03.215 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 10 client.25719.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 7 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 7 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 3 7 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 4 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 10 client.25719.objecter _send_command 7 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c011580 con 0x7f460c016410 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c016410 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c011580 con 0x7f460c016410 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:07:03.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 10 client.25719.objecter _finish_command 7 = system:0 2024-09-10T19:07:03.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 4 2024-09-10T19:07:03.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.213+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 3 7 2024-09-10T19:07:03.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:03.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 10 client.25719.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 8 2024-09-10T19:07:03.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 8 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010dd0 osd=3 3 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 3 8 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 4 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 10 client.25719.objecter _send_command 8 2024-09-10T19:07:03.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c010850 con 0x7f460c016410 2024-09-10T19:07:03.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c010850 con 0x7f460c016410 2024-09-10T19:07:03.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f461d7fa640 10 client.25719.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 4 2024-09-10T19:07:03.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.229+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 3 8 2024-09-10T19:07:03.235 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:07:03.235 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:07:03.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 10 client.25719.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 9 2024-09-10T19:07:03.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 9 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 4 9 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 4 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 10 client.25719.objecter _send_command 9 2024-09-10T19:07:03.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c010780 con 0x7f460c015f60 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c015f60 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c010780 con 0x7f460c015f60 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 10 client.25719.objecter _finish_command 9 = system:0 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 4 2024-09-10T19:07:03.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.233+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 4 9 2024-09-10T19:07:03.257 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 10 client.25719.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 10 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 10 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010610 osd=4 3 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 4 10 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 4 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 10 client.25719.objecter _send_command 10 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.257+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c010780 con 0x7f460c015f60 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c010780 con 0x7f460c015f60 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f461d7fa640 10 client.25719.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 4 2024-09-10T19:07:03.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 4 10 2024-09-10T19:07:03.267 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:07:03.267 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:07:03.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f4626ce6640 10 client.25719.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 11 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.261+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 11 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 5 11 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 4 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 10 client.25719.objecter _send_command 11 2024-09-10T19:07:03.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c01c320 con 0x7f460c01bce0 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c01bce0 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c01c320 con 0x7f460c01bce0 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 10 client.25719.objecter _finish_command 11 = system:0 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 4 2024-09-10T19:07:03.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.265+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 5 11 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 10 client.25719.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 12 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 12 2024-09-10T19:07:03.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c010780 osd=5 3 2024-09-10T19:07:03.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 5 12 2024-09-10T19:07:03.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 4 2024-09-10T19:07:03.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 10 client.25719.objecter _send_command 12 2024-09-10T19:07:03.286 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.281+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c01c2e0 con 0x7f460c01bce0 2024-09-10T19:07:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c01c2e0 con 0x7f460c01bce0 2024-09-10T19:07:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f461d7fa640 10 client.25719.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 4 2024-09-10T19:07:03.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 5 12 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 10 client.25719.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 13 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 13 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 6 13 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 4 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 10 client.25719.objecter _send_command 13 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.285+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c026f50 con 0x7f460c023410 2024-09-10T19:07:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c023410 2024-09-10T19:07:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c026f50 con 0x7f460c023410 2024-09-10T19:07:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:07:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 10 client.25719.objecter _finish_command 13 = system:0 2024-09-10T19:07:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 4 2024-09-10T19:07:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.289+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 6 13 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 10 client.25719.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 14 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 14 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c01c2e0 osd=6 3 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 6 14 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 4 2024-09-10T19:07:03.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 10 client.25719.objecter _send_command 14 2024-09-10T19:07:03.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.313+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c026f10 con 0x7f460c023410 2024-09-10T19:07:03.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c026f10 con 0x7f460c023410 2024-09-10T19:07:03.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 10 client.25719.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 4 2024-09-10T19:07:03.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 6 14 2024-09-10T19:07:03.321 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 10 client.25719.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 15 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 15 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 7 15 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 4 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 10 client.25719.objecter _send_command 15 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f460c028320 con 0x7f460c027640 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.317+0000 7f461d7fa640 10 client.25719.objecter ms_handle_connect 0x7f460c027640 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.321+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f460c028320 con 0x7f460c027640 2024-09-10T19:07:03.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.321+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:07:03.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.321+0000 7f461d7fa640 10 client.25719.objecter _finish_command 15 = system:0 2024-09-10T19:07:03.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.321+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 4 2024-09-10T19:07:03.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.321+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 7 15 2024-09-10T19:07:03.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 10 client.25719.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:03.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign -1 16 2024-09-10T19:07:03.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 20 client.25719.objecter _get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_remove -1 16 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 20 client.25719.objecter get_session s=0x7f460c026f50 osd=7 3 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 15 client.25719.objecter _session_command_op_assign 7 16 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 4 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 10 client.25719.objecter _send_command 16 2024-09-10T19:07:03.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.333+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f460c028320 con 0x7f460c027640 2024-09-10T19:07:03.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f461d7fa640 1 -- 172.21.15.83:0/1844530206 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f460c028320 con 0x7f460c027640 2024-09-10T19:07:03.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f461d7fa640 10 client.25719.objecter ms_dispatch 0x7f4620000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:03.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f461d7fa640 10 client.25719.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:03.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f461d7fa640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 4 2024-09-10T19:07:03.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f461d7fa640 15 client.25719.objecter _session_command_op_remove 7 16 2024-09-10T19:07:03.343 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:07:03.344 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:07:03.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 10 librados: watch_flush enter 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 10 librados: watch_flush exit 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.0 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6801/2535527483 conn(0x7f460c004640 legacy=0x7f460c002740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c005840 osd=0 1 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.1 2024-09-10T19:07:03.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6805/2309624637 conn(0x7f460c003610 legacy=0x7f460c00eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c004e40 osd=1 1 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.2 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.337+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6809/386094220 conn(0x7f460c011790 legacy=0x7f460c013b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c001b90 osd=2 1 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.3 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4625283640 1 -- 172.21.15.83:0/1844530206 reap_dead start 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6813/1279688021 conn(0x7f460c016410 legacy=0x7f460c0187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010dd0 osd=3 1 2024-09-10T19:07:03.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.4 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.165:6800/3225515265 conn(0x7f460c015f60 legacy=0x7f460c01b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010610 osd=4 1 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.5 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.165:6804/571173777 conn(0x7f460c01bce0 legacy=0x7f460c0217e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c010780 osd=5 1 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.6 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.165:6808/1317922936 conn(0x7f460c023410 legacy=0x7f460c0257f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c01c2e0 osd=6 1 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter close_session for osd.7 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.165:6812/1610151650 conn(0x7f460c027640 legacy=0x7f460c027ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter put_session s=0x7f460c026f50 osd=7 1 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 20 client.25719.objecter shutdown clearing up homeless session... 2024-09-10T19:07:03.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 10 client.25719.objecter successfully canceled tick 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4625283640 1 -- 172.21.15.83:0/1844530206 reap_dead start 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6800/3071793735 conn(0x7f4608071510 legacy=0x7f46080739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> v1:172.21.15.83:6789/0 conn(0x7f46201053b0 legacy=0x7f46200fe980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 shutdown_connections 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 >> 172.21.15.83:0/1844530206 conn(0x7f4620074a80 msgr2=0x7f46200fab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 shutdown_connections 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 -- 172.21.15.83:0/1844530206 wait complete. 2024-09-10T19:07:03.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:03.341+0000 7f4626ce6640 1 librados: shutdown 2024-09-10T19:07:03.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:03.394 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:03.496 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:03.533 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:03.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:03.670 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:03.775 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:03.809 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:03.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:03.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: cluster 2024-09-10T19:07:02.274324+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: audit 2024-09-10T19:07:03.144930+0000 mon.a (mon.0) 3504 : audit [DBG] from='client.? v1:172.21.15.83:0/1844530206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: audit 2024-09-10T19:07:03.364447+0000 mon.a (mon.0) 3505 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: cluster 2024-09-10T19:07:03.365946+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: audit 2024-09-10T19:07:03.368625+0000 mon.c (mon.2) 796 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:03.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:03 smithi165 bash[22591]: audit 2024-09-10T19:07:03.370943+0000 mon.a (mon.0) 3507 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:03.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:03.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: cluster 2024-09-10T19:07:02.274324+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:04.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: audit 2024-09-10T19:07:03.144930+0000 mon.a (mon.0) 3504 : audit [DBG] from='client.? v1:172.21.15.83:0/1844530206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:04.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: audit 2024-09-10T19:07:03.364447+0000 mon.a (mon.0) 3505 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:04.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: cluster 2024-09-10T19:07:03.365946+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:07:04.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: audit 2024-09-10T19:07:03.368625+0000 mon.c (mon.2) 796 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:04.044 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[18629]: audit 2024-09-10T19:07:03.370943+0000 mon.a (mon.0) 3507 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: cluster 2024-09-10T19:07:02.274324+0000 osd.7 (osd.7) 403 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: audit 2024-09-10T19:07:03.144930+0000 mon.a (mon.0) 3504 : audit [DBG] from='client.? v1:172.21.15.83:0/1844530206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: audit 2024-09-10T19:07:03.364447+0000 mon.a (mon.0) 3505 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: cluster 2024-09-10T19:07:03.365946+0000 mon.a (mon.0) 3506 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: audit 2024-09-10T19:07:03.368625+0000 mon.c (mon.2) 796 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:04.045 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:03 smithi083 bash[32381]: audit 2024-09-10T19:07:03.370943+0000 mon.a (mon.0) 3507 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:04.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:04.085 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:04.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:04.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:04.362 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:04.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.498 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:04.602 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:04.634 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:04.738 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:04.770 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:04.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.906 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:04.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:04.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:07:05.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.042 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:05.146 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:05.176 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:05.280 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:05.309 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:05.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:05.439 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: cluster 2024-09-10T19:07:04.335367+0000 mgr.y (mgr.24526) 697 : cluster [DBG] pgmap v1182: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:04.365613+0000 mon.a (mon.0) 3508 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: cluster 2024-09-10T19:07:04.365712+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:04.368051+0000 mon.c (mon.2) 797 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:04.369220+0000 mon.a (mon.0) 3510 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:04.907935+0000 mon.c (mon.2) 798 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:05.356607+0000 mon.c (mon.2) 799 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:05.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[18629]: audit 2024-09-10T19:07:05.357715+0000 mon.c (mon.2) 800 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: cluster 2024-09-10T19:07:04.335367+0000 mgr.y (mgr.24526) 697 : cluster [DBG] pgmap v1182: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:04.365613+0000 mon.a (mon.0) 3508 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: cluster 2024-09-10T19:07:04.365712+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:04.368051+0000 mon.c (mon.2) 797 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:04.369220+0000 mon.a (mon.0) 3510 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:04.907935+0000 mon.c (mon.2) 798 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:05.356607+0000 mon.c (mon.2) 799 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:05.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:05 smithi083 bash[32381]: audit 2024-09-10T19:07:05.357715+0000 mon.c (mon.2) 800 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:05.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.575 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:05.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:05.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: cluster 2024-09-10T19:07:04.335367+0000 mgr.y (mgr.24526) 697 : cluster [DBG] pgmap v1182: 214 pgs: 32 unknown, 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:05.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:04.365613+0000 mon.a (mon.0) 3508 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: cluster 2024-09-10T19:07:04.365712+0000 mon.a (mon.0) 3509 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:04.368051+0000 mon.c (mon.2) 797 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:04.369220+0000 mon.a (mon.0) 3510 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:04.907935+0000 mon.c (mon.2) 798 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:05.356607+0000 mon.c (mon.2) 799 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:05.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:05 smithi165 bash[22591]: audit 2024-09-10T19:07:05.357715+0000 mon.c (mon.2) 800 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:05.715 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:05.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.850 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:05.929 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:05.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.987 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:05.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:06.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:06.122 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:06.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:06.264 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:06.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:06.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:06.432 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:07:06.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:06.530 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:06.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:06.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: audit 2024-09-10T19:07:05.368214+0000 mon.a (mon.0) 3511 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:06.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: cluster 2024-09-10T19:07:05.368373+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:07:06.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: audit 2024-09-10T19:07:05.369243+0000 mon.c (mon.2) 801 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: audit 2024-09-10T19:07:05.370966+0000 mon.a (mon.0) 3513 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.643 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: audit 2024-09-10T19:07:05.375982+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: cluster 2024-09-10T19:07:06.361915+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[18629]: audit 2024-09-10T19:07:06.370890+0000 mon.a (mon.0) 3516 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]': finished 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: audit 2024-09-10T19:07:05.368214+0000 mon.a (mon.0) 3511 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: cluster 2024-09-10T19:07:05.368373+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: audit 2024-09-10T19:07:05.369243+0000 mon.c (mon.2) 801 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: audit 2024-09-10T19:07:05.370966+0000 mon.a (mon.0) 3513 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: audit 2024-09-10T19:07:05.375982+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:06.644 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: cluster 2024-09-10T19:07:06.361915+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:06.645 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:06 smithi083 bash[32381]: audit 2024-09-10T19:07:06.370890+0000 mon.a (mon.0) 3516 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]': finished 2024-09-10T19:07:06.655 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:06.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: audit 2024-09-10T19:07:05.368214+0000 mon.a (mon.0) 3511 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: cluster 2024-09-10T19:07:05.368373+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: audit 2024-09-10T19:07:05.369243+0000 mon.c (mon.2) 801 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: audit 2024-09-10T19:07:05.370966+0000 mon.a (mon.0) 3513 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]: dispatch 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: audit 2024-09-10T19:07:05.375982+0000 mon.a (mon.0) 3514 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: cluster 2024-09-10T19:07:06.361915+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:06.772 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:06 smithi165 bash[22591]: audit 2024-09-10T19:07:06.370890+0000 mon.a (mon.0) 3516 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-136", "mode": "writeback"}]': finished 2024-09-10T19:07:06.791 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:06.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:06.929 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:06.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:07.032 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:07.067 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:07.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:07.207 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:07.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:07.346 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:07.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:07.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:07 smithi165 bash[22591]: cluster 2024-09-10T19:07:06.337417+0000 mgr.y (mgr.24526) 698 : cluster [DBG] pgmap v1185: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:07.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:07 smithi165 bash[22591]: cluster 2024-09-10T19:07:06.371106+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:07:07.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:07 smithi165 bash[22591]: audit 2024-09-10T19:07:06.481750+0000 mon.c (mon.2) 802 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.462 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:07 smithi165 bash[22591]: audit 2024-09-10T19:07:06.482411+0000 mon.a (mon.0) 3518 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.482 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:07.586 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:07.618 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:07.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:07.750 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[18629]: cluster 2024-09-10T19:07:06.337417+0000 mgr.y (mgr.24526) 698 : cluster [DBG] pgmap v1185: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[18629]: cluster 2024-09-10T19:07:06.371106+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[18629]: audit 2024-09-10T19:07:06.481750+0000 mon.c (mon.2) 802 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[18629]: audit 2024-09-10T19:07:06.482411+0000 mon.a (mon.0) 3518 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[32381]: cluster 2024-09-10T19:07:06.337417+0000 mgr.y (mgr.24526) 698 : cluster [DBG] pgmap v1185: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[32381]: cluster 2024-09-10T19:07:06.371106+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[32381]: audit 2024-09-10T19:07:06.481750+0000 mon.c (mon.2) 802 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:07 smithi083 bash[32381]: audit 2024-09-10T19:07:06.482411+0000 mon.a (mon.0) 3518 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:07.854 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:07.887 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:07.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:07.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:08.020 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:08.124 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:08.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:08.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:08.291 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:08.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:08.422 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:08.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:08.557 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:08.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:08.673 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[18629]: audit 2024-09-10T19:07:07.390481+0000 mon.a (mon.0) 3519 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:08.673 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[18629]: cluster 2024-09-10T19:07:07.390571+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-09-10T19:07:08.673 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[18629]: audit 2024-09-10T19:07:07.393659+0000 mon.c (mon.2) 803 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.673 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[18629]: audit 2024-09-10T19:07:07.394656+0000 mon.a (mon.0) 3521 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.673 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[32381]: audit 2024-09-10T19:07:07.390481+0000 mon.a (mon.0) 3519 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:08.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[32381]: cluster 2024-09-10T19:07:07.390571+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-09-10T19:07:08.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[32381]: audit 2024-09-10T19:07:07.393659+0000 mon.c (mon.2) 803 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.674 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:08 smithi083 bash[32381]: audit 2024-09-10T19:07:07.394656+0000 mon.a (mon.0) 3521 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:08.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:08.811 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:08.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:08 smithi165 bash[22591]: audit 2024-09-10T19:07:07.390481+0000 mon.a (mon.0) 3519 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:08.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:08 smithi165 bash[22591]: cluster 2024-09-10T19:07:07.390571+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-09-10T19:07:08.811 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:08 smithi165 bash[22591]: audit 2024-09-10T19:07:07.393659+0000 mon.c (mon.2) 803 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.812 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:08 smithi165 bash[22591]: audit 2024-09-10T19:07:07.394656+0000 mon.a (mon.0) 3521 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]: dispatch 2024-09-10T19:07:08.833 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:08.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:08.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:09.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:09.105 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:09.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:09.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.376 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:09.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.515 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:09.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:09.651 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:09.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:09.786 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:09.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: cluster 2024-09-10T19:07:08.338312+0000 mgr.y (mgr.24526) 699 : cluster [DBG] pgmap v1188: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: audit 2024-09-10T19:07:08.374020+0000 mgr.y (mgr.24526) 700 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: cluster 2024-09-10T19:07:08.386292+0000 mon.a (mon.0) 3522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: audit 2024-09-10T19:07:08.403212+0000 mon.a (mon.0) 3523 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: cluster 2024-09-10T19:07:08.403314+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[18629]: cluster 2024-09-10T19:07:09.359346+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-09-10T19:07:09.902 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: cluster 2024-09-10T19:07:08.338312+0000 mgr.y (mgr.24526) 699 : cluster [DBG] pgmap v1188: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:09.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: audit 2024-09-10T19:07:08.374020+0000 mgr.y (mgr.24526) 700 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:09.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: cluster 2024-09-10T19:07:08.386292+0000 mon.a (mon.0) 3522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:09.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: audit 2024-09-10T19:07:08.403212+0000 mon.a (mon.0) 3523 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:09.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: cluster 2024-09-10T19:07:08.403314+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-09-10T19:07:09.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:09 smithi083 bash[32381]: cluster 2024-09-10T19:07:09.359346+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-09-10T19:07:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:09.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:10.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.056 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: cluster 2024-09-10T19:07:08.338312+0000 mgr.y (mgr.24526) 699 : cluster [DBG] pgmap v1188: 214 pgs: 214 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: audit 2024-09-10T19:07:08.374020+0000 mgr.y (mgr.24526) 700 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: cluster 2024-09-10T19:07:08.386292+0000 mon.a (mon.0) 3522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: audit 2024-09-10T19:07:08.403212+0000 mon.a (mon.0) 3523 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-136"}]': finished 2024-09-10T19:07:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: cluster 2024-09-10T19:07:08.403314+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-09-10T19:07:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:09 smithi165 bash[22591]: cluster 2024-09-10T19:07:09.359346+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-09-10T19:07:10.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:10.194 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:10.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.325 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:10.334 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:10.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.472 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:10.576 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:10.608 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:10.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:10.744 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:10.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:10.880 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:10.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:10.984 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.017 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:11.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:11.155 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:11.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.291 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:11.395 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[18629]: cluster 2024-09-10T19:07:10.339029+0000 mgr.y (mgr.24526) 701 : cluster [DBG] pgmap v1191: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[18629]: cluster 2024-09-10T19:07:10.357052+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[18629]: audit 2024-09-10T19:07:10.361445+0000 mon.c (mon.2) 804 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[18629]: audit 2024-09-10T19:07:10.362658+0000 mon.a (mon.0) 3527 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[32381]: cluster 2024-09-10T19:07:10.339029+0000 mgr.y (mgr.24526) 701 : cluster [DBG] pgmap v1191: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[32381]: cluster 2024-09-10T19:07:10.357052+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[32381]: audit 2024-09-10T19:07:10.361445+0000 mon.c (mon.2) 804 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.403 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:11 smithi083 bash[32381]: audit 2024-09-10T19:07:10.362658+0000 mon.a (mon.0) 3527 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.414 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:11.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:11.553 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:11.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:11.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:11 smithi165 bash[22591]: cluster 2024-09-10T19:07:10.339029+0000 mgr.y (mgr.24526) 701 : cluster [DBG] pgmap v1191: 182 pgs: 182 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:11.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:11 smithi165 bash[22591]: cluster 2024-09-10T19:07:10.357052+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-09-10T19:07:11.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:11 smithi165 bash[22591]: audit 2024-09-10T19:07:10.361445+0000 mon.c (mon.2) 804 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:11 smithi165 bash[22591]: audit 2024-09-10T19:07:10.362658+0000 mon.a (mon.0) 3527 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:11.657 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:11.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:11.829 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:11.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.965 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:11.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:12.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:12.103 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:12.206 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:12.242 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:12.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:12.373 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:12.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:12.515 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[18629]: audit 2024-09-10T19:07:11.371786+0000 mon.a (mon.0) 3528 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[18629]: cluster 2024-09-10T19:07:11.372397+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-09-10T19:07:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[18629]: audit 2024-09-10T19:07:11.382826+0000 mon.c (mon.2) 805 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[18629]: audit 2024-09-10T19:07:11.383622+0000 mon.a (mon.0) 3530 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[32381]: audit 2024-09-10T19:07:11.371786+0000 mon.a (mon.0) 3528 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[32381]: cluster 2024-09-10T19:07:11.372397+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-09-10T19:07:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[32381]: audit 2024-09-10T19:07:11.382826+0000 mon.c (mon.2) 805 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[32381]: audit 2024-09-10T19:07:11.383622+0000 mon.a (mon.0) 3530 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:12 smithi165 bash[22591]: audit 2024-09-10T19:07:11.371786+0000 mon.a (mon.0) 3528 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:12.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:12 smithi165 bash[22591]: cluster 2024-09-10T19:07:11.372397+0000 mon.a (mon.0) 3529 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-09-10T19:07:12.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:12 smithi165 bash[22591]: audit 2024-09-10T19:07:11.382826+0000 mon.c (mon.2) 805 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:12 smithi165 bash[22591]: audit 2024-09-10T19:07:11.383622+0000 mon.a (mon.0) 3530 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:12.619 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:12.653 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:12.757 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:12.790 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:12.894 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:12.895 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:12.906 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:12] "GET /metrics HTTP/1.1" 200 229035 "" "Prometheus/2.43.0" 2024-09-10T19:07:12.924 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:12.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:13.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:13.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:13.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:13.199 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:13.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:13.337 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:13.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:13.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:13 smithi165 bash[22591]: cluster 2024-09-10T19:07:12.339890+0000 mgr.y (mgr.24526) 702 : cluster [DBG] pgmap v1194: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:13.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:13 smithi165 bash[22591]: audit 2024-09-10T19:07:12.374608+0000 mon.a (mon.0) 3531 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:13.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:13 smithi165 bash[22591]: cluster 2024-09-10T19:07:12.374796+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-09-10T19:07:13.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:13 smithi165 bash[22591]: audit 2024-09-10T19:07:12.378017+0000 mon.c (mon.2) 806 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.452 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:13 smithi165 bash[22591]: audit 2024-09-10T19:07:12.383226+0000 mon.a (mon.0) 3533 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.475 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:13.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[18629]: cluster 2024-09-10T19:07:12.339890+0000 mgr.y (mgr.24526) 702 : cluster [DBG] pgmap v1194: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[18629]: audit 2024-09-10T19:07:12.374608+0000 mon.a (mon.0) 3531 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[18629]: cluster 2024-09-10T19:07:12.374796+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[18629]: audit 2024-09-10T19:07:12.378017+0000 mon.c (mon.2) 806 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[18629]: audit 2024-09-10T19:07:12.383226+0000 mon.a (mon.0) 3533 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[32381]: cluster 2024-09-10T19:07:12.339890+0000 mgr.y (mgr.24526) 702 : cluster [DBG] pgmap v1194: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[32381]: audit 2024-09-10T19:07:12.374608+0000 mon.a (mon.0) 3531 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[32381]: cluster 2024-09-10T19:07:12.374796+0000 mon.a (mon.0) 3532 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[32381]: audit 2024-09-10T19:07:12.378017+0000 mon.c (mon.2) 806 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.580 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:13 smithi083 bash[32381]: audit 2024-09-10T19:07:12.383226+0000 mon.a (mon.0) 3533 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:13.612 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:13.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:13.748 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:13.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:13.887 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:13.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:13.991 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:14.026 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:14.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:14.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/2453137649 >> v1:172.21.15.83:6790/0 conn(0x7f7de80fb340 legacy=0x7f7de80fb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/2453137649 shutdown_connections 2024-09-10T19:07:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/2453137649 >> 172.21.15.83:0/2453137649 conn(0x7f7de80f7070 msgr2=0x7f7de80f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/2453137649 shutdown_connections 2024-09-10T19:07:14.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/2453137649 wait complete. 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 librados: starting msgr at 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 librados: starting objecter 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 Processor -- start 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- start start 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 librados: setting wanted keys 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 librados: calling monclient init 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7de811dfe0 con 0x7f7de810b170 2024-09-10T19:07:14.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7de811f1a0 con 0x7f7de80fb340 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7de81203a0 con 0x7f7de80fbe20 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7de77fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f7de810b170 0x7f7de811c760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60938/0 (socket says 172.21.15.83:60938) 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7de77fe640 1 -- 172.21.15.83:0/847387100 learned_addr learned my addr 172.21.15.83:0/847387100 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7de67fc640 1 --1- 172.21.15.83:0/847387100 >> v1:172.21.15.83:6790/0 conn(0x7f7de80fbe20 0x7f7de8107890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:33554/0 (socket says 172.21.15.83:33554) 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7de810b170 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7de80fbe20 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter ms_handle_connect 0x7f7de80fb340 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:14.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626226687 0 0) 0x7f7de811dfe0 con 0x7f7de810b170 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd00039c0 con 0x7f7de810b170 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489497222 0 0) 0x7f7de811f1a0 con 0x7f7de80fb340 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de811dfe0 con 0x7f7de80fb340 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1929935470 0 0) 0x7f7dd00039c0 con 0x7f7de810b170 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7de811f1a0 con 0x7f7de810b170 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7ddc0019f0 con 0x7f7de810b170 2024-09-10T19:07:14.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622036776 0 0) 0x7f7de81203a0 con 0x7f7de80fbe20 2024-09-10T19:07:14.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd00039c0 con 0x7f7de80fbe20 2024-09-10T19:07:14.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 944578485 0 0) 0x7f7de811f1a0 con 0x7f7de810b170 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 >> v1:172.21.15.83:6790/0 conn(0x7f7de80fbe20 legacy=0x7f7de8107890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 >> v1:172.21.15.165:6789/0 conn(0x7f7de80fb340 legacy=0x7f7de8107180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de81215a0 con 0x7f7de810b170 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de8120590 con 0x7f7de810b170 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 10 client.25725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de8120a60 con 0x7f7de810b170 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.149+0000 7f7ded3ec640 1 librados: init done 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7ddc002450 con 0x7f7de810b170 2024-09-10T19:07:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7ddc005b10 con 0x7f7de810b170 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4005010 con 0x7f7de810b170 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7ddc01bf30 con 0x7f7de810b170 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(774..774 src has 704..774) v4 ==== 8531+0+0 (unknown 2135546088 0 0) 0x7f7ddc08a9e0 con 0x7f7de810b170 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 10 client.25725.objecter ms_dispatch 0x7f7de8000b90 osd_map(774..774 src has 704..774) v4 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 3 client.25725.objecter handle_osd_map got epochs [774,774] > 0 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 3 client.25725.objecter handle_osd_map decoding full epoch 774 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 20 client.25725.objecter dump_active .. 0 homeless 2024-09-10T19:07:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.153+0000 7f7dcf7fe640 10 client.25725.objecter ms_handle_connect 0x7f7dd00713c0 2024-09-10T19:07:14.159 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.157+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f7ddc057570 con 0x7f7de810b170 2024-09-10T19:07:14.172 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:07:14.172 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:07:14.172 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T19:07:14.173 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:07:14.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:14.298 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:14.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.341+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7dd4000b90 con 0x7f7dd00713c0 2024-09-10T19:07:14.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.369+0000 7f7dcf7fe640 1 -- 172.21.15.83:0/847387100 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+591420 (unknown 2965378022 0 2705556503) 0x7f7dd4000b90 con 0x7f7dd00713c0 2024-09-10T19:07:14.382 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:07:14.387 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:07:14.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 10 librados: watch_flush enter 2024-09-10T19:07:14.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 10 librados: watch_flush exit 2024-09-10T19:07:14.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 20 client.25725.objecter shutdown clearing up homeless session... 2024-09-10T19:07:14.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 10 client.25725.objecter successfully canceled tick 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 >> v1:172.21.15.83:6800/3071793735 conn(0x7f7dd00713c0 legacy=0x7f7dd0073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 >> v1:172.21.15.83:6789/0 conn(0x7f7de810b170 legacy=0x7f7de811c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 shutdown_connections 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 >> 172.21.15.83:0/847387100 conn(0x7f7de80f7070 msgr2=0x7f7de80f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 shutdown_connections 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.373+0000 7f7ded3ec640 1 -- 172.21.15.83:0/847387100 wait complete. 2024-09-10T19:07:14.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:14.377+0000 7f7ded3ec640 1 librados: shutdown 2024-09-10T19:07:14.403 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:14.457 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:14.483 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:13.395096+0000 mon.a (mon.0) 3534 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: cluster 2024-09-10T19:07:13.395217+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:13.397852+0000 mon.c (mon.2) 807 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:13.398383+0000 mon.a (mon.0) 3536 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:14.360154+0000 mon.a (mon.0) 3537 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: cluster 2024-09-10T19:07:14.360227+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:14.361177+0000 mon.c (mon.2) 808 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[32381]: audit 2024-09-10T19:07:14.361871+0000 mon.a (mon.0) 3539 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.484 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:13.395096+0000 mon.a (mon.0) 3534 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: cluster 2024-09-10T19:07:13.395217+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:13.397852+0000 mon.c (mon.2) 807 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:13.398383+0000 mon.a (mon.0) 3536 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:14.360154+0000 mon.a (mon.0) 3537 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: cluster 2024-09-10T19:07:14.360227+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:14.361177+0000 mon.c (mon.2) 808 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.485 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:14 smithi083 bash[18629]: audit 2024-09-10T19:07:14.361871+0000 mon.a (mon.0) 3539 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:14.596 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:14.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:13.395096+0000 mon.a (mon.0) 3534 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:14.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: cluster 2024-09-10T19:07:13.395217+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-09-10T19:07:14.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:13.397852+0000 mon.c (mon.2) 807 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:13.398383+0000 mon.a (mon.0) 3536 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:14.360154+0000 mon.a (mon.0) 3537 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: cluster 2024-09-10T19:07:14.360227+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-09-10T19:07:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:14.361177+0000 mon.c (mon.2) 808 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:14 smithi165 bash[22591]: audit 2024-09-10T19:07:14.361871+0000 mon.a (mon.0) 3539 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:14.699 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:14.735 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:14.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:14.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:14.873 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:14.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:14.977 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.010 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:15.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.145 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:15.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:15.285 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:15.388 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: cluster 2024-09-10T19:07:14.340480+0000 mgr.y (mgr.24526) 703 : cluster [DBG] pgmap v1197: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: audit 2024-09-10T19:07:14.345611+0000 mgr.y (mgr.24526) 704 : audit [DBG] from='client.25725 v1:172.21.15.83:0/847387100' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: audit 2024-09-10T19:07:14.580132+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: audit 2024-09-10T19:07:14.584999+0000 mon.c (mon.2) 809 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: audit 2024-09-10T19:07:15.362816+0000 mon.a (mon.0) 3541 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:15.395 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:15 smithi165 bash[22591]: cluster 2024-09-10T19:07:15.363017+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-09-10T19:07:15.417 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:15.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:15.566 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: cluster 2024-09-10T19:07:14.340480+0000 mgr.y (mgr.24526) 703 : cluster [DBG] pgmap v1197: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:15.566 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: audit 2024-09-10T19:07:14.345611+0000 mgr.y (mgr.24526) 704 : audit [DBG] from='client.25725 v1:172.21.15.83:0/847387100' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:15.566 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: audit 2024-09-10T19:07:14.580132+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:15.566 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: audit 2024-09-10T19:07:14.584999+0000 mon.c (mon.2) 809 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: audit 2024-09-10T19:07:15.362816+0000 mon.a (mon.0) 3541 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[32381]: cluster 2024-09-10T19:07:15.363017+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: cluster 2024-09-10T19:07:14.340480+0000 mgr.y (mgr.24526) 703 : cluster [DBG] pgmap v1197: 214 pgs: 32 creating+peering, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: audit 2024-09-10T19:07:14.345611+0000 mgr.y (mgr.24526) 704 : audit [DBG] from='client.25725 v1:172.21.15.83:0/847387100' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: audit 2024-09-10T19:07:14.580132+0000 mon.a (mon.0) 3540 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: audit 2024-09-10T19:07:14.584999+0000 mon.c (mon.2) 809 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: audit 2024-09-10T19:07:15.362816+0000 mon.a (mon.0) 3541 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:15 smithi083 bash[18629]: cluster 2024-09-10T19:07:15.363017+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-09-10T19:07:15.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:15.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:15.933 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.966 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:15.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:16.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:16.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:16.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:16.242 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:16.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.382 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:16.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.516 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:16 smithi083 bash[18629]: cluster 2024-09-10T19:07:16.342703+0000 mgr.y (mgr.24526) 705 : cluster [DBG] pgmap v1200: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:16 smithi083 bash[32381]: cluster 2024-09-10T19:07:16.342703+0000 mgr.y (mgr.24526) 705 : cluster [DBG] pgmap v1200: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:16 smithi165 bash[22591]: cluster 2024-09-10T19:07:16.342703+0000 mgr.y (mgr.24526) 705 : cluster [DBG] pgmap v1200: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:16.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.649 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:16.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.785 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:16.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:16.927 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:16.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:16.955 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:07:17.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:17.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:17.198 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:17.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.332 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:17.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:17.457 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[18629]: audit 2024-09-10T19:07:16.400620+0000 mon.c (mon.2) 810 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.457 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[18629]: audit 2024-09-10T19:07:16.407166+0000 mon.a (mon.0) 3543 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.457 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[18629]: audit 2024-09-10T19:07:16.408036+0000 mon.c (mon.2) 811 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.457 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[18629]: audit 2024-09-10T19:07:16.408398+0000 mon.a (mon.0) 3544 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.473 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:17.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:17 smithi165 bash[22591]: audit 2024-09-10T19:07:16.400620+0000 mon.c (mon.2) 810 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:17 smithi165 bash[22591]: audit 2024-09-10T19:07:16.407166+0000 mon.a (mon.0) 3543 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:17 smithi165 bash[22591]: audit 2024-09-10T19:07:16.408036+0000 mon.c (mon.2) 811 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.591 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:17 smithi165 bash[22591]: audit 2024-09-10T19:07:16.408398+0000 mon.a (mon.0) 3544 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.610 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:17.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:17.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[32381]: audit 2024-09-10T19:07:16.400620+0000 mon.c (mon.2) 810 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.727 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[32381]: audit 2024-09-10T19:07:16.407166+0000 mon.a (mon.0) 3543 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:17.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[32381]: audit 2024-09-10T19:07:16.408036+0000 mon.c (mon.2) 811 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:17 smithi083 bash[32381]: audit 2024-09-10T19:07:16.408398+0000 mon.a (mon.0) 3544 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]: dispatch 2024-09-10T19:07:17.747 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:17.850 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:17.882 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:17.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:17.985 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:18.015 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:18.036 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":1197,"stamp":"2024-09-10T19:07:14.340126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4596282,"num_objects":214,"num_object_clones":1,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":2243,"num_read_kb":1982,"num_write":760,"num_write_kb":3989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1860,"ondisk_log_size":1860,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":606,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2424200,"kb_used_data":24528,"kb_used_omap":13,"kb_used_meta":2399538,"kb_avail":747537016,"statfs":{"total":767960285184,"available":765477904384,"internally_reserved":0,"allocated":25116672,"data_stored":13719133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2457127616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4000008,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":3907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001553"},"pg_stats":[{"pgid":"255.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362003+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:11.638658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.357632+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:17.046631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.366887+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:26.449086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.361637+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:30.244900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.359865+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:24.208215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.369270+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:36.217157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362410+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.458421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.360480+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.386970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.371637+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:31.772406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.371649+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:34.155154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.363108+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:27.613936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.363979+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:25.974513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.359270+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:16.744271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.368430+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:20.164337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.360432+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.042245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.361222+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:27.315467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.364114+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:39.259424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.356229+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:31.719570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"255.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.356212+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:36.253902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"255.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362117+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:25.713304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.365990+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:31.498118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362499+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.656195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.367851+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:25.776271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.355970+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:35.559974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"255.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.360879+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:36.665331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"241.7","version":"767'7","reported_seq":162,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362575+0000","last_change":"2024-09-10T19:06:50.360648+0000","last_active":"2024-09-10T19:07:09.362575+0000","last_peered":"2024-09-10T19:07:09.362575+0000","last_clean":"2024-09-10T19:07:09.362575+0000","last_became_active":"2024-09-10T19:04:57.320799+0000","last_became_peered":"2024-09-10T19:04:57.320799+0000","last_unstale":"2024-09-10T19:07:09.362575+0000","last_undegraded":"2024-09-10T19:07:09.362575+0000","last_fullsized":"2024-09-10T19:07:09.362575+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'6","last_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_deep_scrub":"730'6","last_deep_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.006586+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:14.953341+0000","scrub_duration":0.004755039,"objects_trimmed":0,"snaptrim_duration":0.00095805300000000005,"stat_sum":{"num_bytes":4000013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"255.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.356100+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:18.549554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"241.6","version":"738'9","reported_seq":169,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812336+0000","last_change":"2024-09-10T19:06:50.365365+0000","last_active":"2024-09-10T19:07:09.812336+0000","last_peered":"2024-09-10T19:07:09.812336+0000","last_clean":"2024-09-10T19:07:09.812336+0000","last_became_active":"2024-09-10T19:04:57.324770+0000","last_became_peered":"2024-09-10T19:04:57.324770+0000","last_unstale":"2024-09-10T19:07:09.812336+0000","last_undegraded":"2024-09-10T19:07:09.812336+0000","last_fullsized":"2024-09-10T19:07:09.812336+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'9","last_scrub_stamp":"2024-09-10T19:06:49.172954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:49.172954+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.846639+0000","scrub_duration":0.0043854150000000001,"objects_trimmed":0,"snaptrim_duration":0.00099442499999999995,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"255.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362977+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.5","version":"720'6","reported_seq":163,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811092+0000","last_change":"2024-09-10T19:06:52.119235+0000","last_active":"2024-09-10T19:07:09.811092+0000","last_peered":"2024-09-10T19:07:09.811092+0000","last_clean":"2024-09-10T19:07:09.811092+0000","last_became_active":"2024-09-10T19:04:57.322094+0000","last_became_peered":"2024-09-10T19:04:57.322094+0000","last_unstale":"2024-09-10T19:07:09.811092+0000","last_undegraded":"2024-09-10T19:07:09.811092+0000","last_fullsized":"2024-09-10T19:07:09.811092+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"720'6","last_scrub_stamp":"2024-09-10T19:06:52.119058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.119058+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:18.898465+0000","scrub_duration":0.0043290079999999996,"objects_trimmed":0,"snaptrim_duration":0.0010204319999999999,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"255.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.362955+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:35.401465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.4","version":"759'11","reported_seq":167,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363932+0000","last_change":"2024-09-10T19:06:50.352548+0000","last_active":"2024-09-10T19:07:09.363932+0000","last_peered":"2024-09-10T19:07:09.363932+0000","last_clean":"2024-09-10T19:07:09.363932+0000","last_became_active":"2024-09-10T19:04:57.322911+0000","last_became_peered":"2024-09-10T19:04:57.322911+0000","last_unstale":"2024-09-10T19:07:09.363932+0000","last_undegraded":"2024-09-10T19:07:09.363932+0000","last_fullsized":"2024-09-10T19:07:09.363932+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'6","last_scrub_stamp":"2024-09-10T19:06:39.141613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.141613+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.735457+0000","scrub_duration":0.0042889529999999999,"objects_trimmed":0,"snaptrim_duration":0.0017454879999999999,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"255.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.360743+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:37.364774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"241.3","version":"712'10","reported_seq":165,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.611000+0000","last_change":"2024-09-10T19:06:50.345918+0000","last_active":"2024-09-10T19:07:09.611000+0000","last_peered":"2024-09-10T19:07:09.611000+0000","last_clean":"2024-09-10T19:07:09.611000+0000","last_became_active":"2024-09-10T19:04:57.324250+0000","last_became_peered":"2024-09-10T19:04:57.324250+0000","last_unstale":"2024-09-10T19:07:09.611000+0000","last_undegraded":"2024-09-10T19:07:09.611000+0000","last_fullsized":"2024-09-10T19:07:09.611000+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'10","last_scrub_stamp":"2024-09-10T19:06:27.862802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:27.862802+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.207401+0000","scrub_duration":0.0041735690000000002,"objects_trimmed":0,"snaptrim_duration":0.0036847949999999998,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.360693+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:39.222884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"241.2","version":"712'11","reported_seq":170,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.609122+0000","last_change":"2024-09-10T19:06:50.344565+0000","last_active":"2024-09-10T19:07:09.609122+0000","last_peered":"2024-09-10T19:07:09.609122+0000","last_clean":"2024-09-10T19:07:09.609122+0000","last_became_active":"2024-09-10T19:04:57.322254+0000","last_became_peered":"2024-09-10T19:04:57.322254+0000","last_unstale":"2024-09-10T19:07:09.609122+0000","last_undegraded":"2024-09-10T19:07:09.609122+0000","last_fullsized":"2024-09-10T19:07:09.609122+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'11","last_scrub_stamp":"2024-09-10T19:06:45.656802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.656802+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.262287+0000","scrub_duration":0.010299141,"objects_trimmed":0,"snaptrim_duration":0.0028767139999999998,"stat_sum":{"num_bytes":10,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"255.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.357758+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:23.646393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"241.1","version":"751'15","reported_seq":164,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815604+0000","last_change":"2024-09-10T19:06:52.359146+0000","last_active":"2024-09-10T19:07:09.815604+0000","last_peered":"2024-09-10T19:07:09.815604+0000","last_clean":"2024-09-10T19:07:09.815604+0000","last_became_active":"2024-09-10T19:04:57.329766+0000","last_became_peered":"2024-09-10T19:04:57.329766+0000","last_unstale":"2024-09-10T19:07:09.815604+0000","last_undegraded":"2024-09-10T19:07:09.815604+0000","last_fullsized":"2024-09-10T19:07:09.815604+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:04:56.297048+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52769914500000004,"stat_sum":{"num_bytes":10,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"255.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-09-10T19:07:10.350308+0000","last_change":"2024-09-10T19:07:10.358074+0000","last_active":"2024-09-10T19:07:10.350308+0000","last_peered":"2024-09-10T19:07:10.350308+0000","last_clean":"2024-09-10T19:07:10.350308+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:07:10.350308+0000","last_undegraded":"2024-09-10T19:07:10.350308+0000","last_fullsized":"2024-09-10T19:07:10.350308+0000","mapping_epoch":771,"log_start":"0'0","ondisk_log_start":"0'0","created":771,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:10.350308+0000","last_clean_scrub_stamp":"2024-09-10T19:07:10.350308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:29.446782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"241.0","version":"662'1","reported_seq":157,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.355476+0000","last_change":"2024-09-10T19:06:50.345068+0000","last_active":"2024-09-10T19:07:09.355476+0000","last_peered":"2024-09-10T19:07:09.355476+0000","last_clean":"2024-09-10T19:07:09.355476+0000","last_became_active":"2024-09-10T19:04:57.320088+0000","last_became_peered":"2024-09-10T19:04:57.320088+0000","last_unstale":"2024-09-10T19:07:09.355476+0000","last_undegraded":"2024-09-10T19:07:09.355476+0000","last_fullsized":"2024-09-10T19:07:09.355476+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'1","last_scrub_stamp":"2024-09-10T19:06:36.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.131427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.839793+0000","scrub_duration":0.0044912399999999996,"objects_trimmed":0,"snaptrim_duration":0.0018060820000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":764,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362887+0000","last_change":"2024-09-10T19:05:29.898575+0000","last_active":"2024-09-10T19:07:09.362887+0000","last_peered":"2024-09-10T19:07:09.362887+0000","last_clean":"2024-09-10T19:07:09.362887+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:07:09.362887+0000","last_undegraded":"2024-09-10T19:07:09.362887+0000","last_fullsized":"2024-09-10T19:07:09.362887+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_clean_scrub_stamp":"2024-09-10T19:05:29.898480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030543649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":774,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810165+0000","last_change":"2024-09-10T19:05:27.220506+0000","last_active":"2024-09-10T19:07:09.810165+0000","last_peered":"2024-09-10T19:07:09.810165+0000","last_clean":"2024-09-10T19:07:09.810165+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:07:09.810165+0000","last_undegraded":"2024-09-10T19:07:09.810165+0000","last_fullsized":"2024-09-10T19:07:09.810165+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:27.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:27.220358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037569050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":798,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365487+0000","last_change":"2024-09-10T19:06:29.097504+0000","last_active":"2024-09-10T19:07:09.365487+0000","last_peered":"2024-09-10T19:07:09.365487+0000","last_clean":"2024-09-10T19:07:09.365487+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:07:09.365487+0000","last_undegraded":"2024-09-10T19:07:09.365487+0000","last_fullsized":"2024-09-10T19:07:09.365487+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:06:29.097329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.097329+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.402785+0000","scrub_duration":0.0048839570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":776,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363459+0000","last_change":"2024-09-10T19:05:10.990324+0000","last_active":"2024-09-10T19:07:09.363459+0000","last_peered":"2024-09-10T19:07:09.363459+0000","last_clean":"2024-09-10T19:07:09.363459+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:07:09.363459+0000","last_undegraded":"2024-09-10T19:07:09.363459+0000","last_fullsized":"2024-09-10T19:07:09.363459+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.990119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.990119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003497137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":693,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.355739+0000","last_change":"2024-09-10T19:06:35.862278+0000","last_active":"2024-09-10T19:07:09.355739+0000","last_peered":"2024-09-10T19:07:09.355739+0000","last_clean":"2024-09-10T19:07:09.355739+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:07:09.355739+0000","last_undegraded":"2024-09-10T19:07:09.355739+0000","last_fullsized":"2024-09-10T19:07:09.355739+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.862120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.862120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.659615+0000","scrub_duration":0.0026679030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366242+0000","last_change":"2024-09-10T19:05:07.989627+0000","last_active":"2024-09-10T19:07:09.366242+0000","last_peered":"2024-09-10T19:07:09.366242+0000","last_clean":"2024-09-10T19:07:09.366242+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:07:09.366242+0000","last_undegraded":"2024-09-10T19:07:09.366242+0000","last_fullsized":"2024-09-10T19:07:09.366242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:07.989469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:07.989469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00341515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":803,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817700+0000","last_change":"2024-09-10T19:06:23.367532+0000","last_active":"2024-09-10T19:07:09.817700+0000","last_peered":"2024-09-10T19:07:09.817700+0000","last_clean":"2024-09-10T19:07:09.817700+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:07:09.817700+0000","last_undegraded":"2024-09-10T19:07:09.817700+0000","last_fullsized":"2024-09-10T19:07:09.817700+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:23.367353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.367353+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:38.115423+0000","scrub_duration":0.0057060319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":792,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.609893+0000","last_change":"2024-09-10T19:06:22.007353+0000","last_active":"2024-09-10T19:07:09.609893+0000","last_peered":"2024-09-10T19:07:09.609893+0000","last_clean":"2024-09-10T19:07:09.609893+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:07:09.609893+0000","last_undegraded":"2024-09-10T19:07:09.609893+0000","last_fullsized":"2024-09-10T19:07:09.609893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.007181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.007181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.778494+0000","scrub_duration":0.0034040429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":250,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357685+0000","last_change":"2024-09-10T19:06:16.980981+0000","last_active":"2024-09-10T19:07:09.357685+0000","last_peered":"2024-09-10T19:07:09.357685+0000","last_clean":"2024-09-10T19:07:09.357685+0000","last_became_active":"2024-09-10T19:02:52.254895+0000","last_became_peered":"2024-09-10T19:02:52.254895+0000","last_unstale":"2024-09-10T19:07:09.357685+0000","last_undegraded":"2024-09-10T19:07:09.357685+0000","last_fullsized":"2024-09-10T19:07:09.357685+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:16.980824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:16.980824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.357064+0000","scrub_duration":0.0034584770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.816660+0000","last_change":"2024-09-10T19:06:43.403406+0000","last_active":"2024-09-10T19:07:09.816660+0000","last_peered":"2024-09-10T19:07:09.816660+0000","last_clean":"2024-09-10T19:07:09.816660+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:07:09.816660+0000","last_undegraded":"2024-09-10T19:07:09.816660+0000","last_fullsized":"2024-09-10T19:07:09.816660+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:43.403297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.403297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:11.191457+0000","scrub_duration":0.0062412609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":784,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811415+0000","last_change":"2024-09-10T19:06:37.935702+0000","last_active":"2024-09-10T19:07:09.811415+0000","last_peered":"2024-09-10T19:07:09.811415+0000","last_clean":"2024-09-10T19:07:09.811415+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:07:09.811415+0000","last_undegraded":"2024-09-10T19:07:09.811415+0000","last_fullsized":"2024-09-10T19:07:09.811415+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.935545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.935545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.697682+0000","scrub_duration":0.003538518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":824,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361531+0000","last_change":"2024-09-10T19:06:29.746879+0000","last_active":"2024-09-10T19:07:09.361531+0000","last_peered":"2024-09-10T19:07:09.361531+0000","last_clean":"2024-09-10T19:07:09.361531+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:07:09.361531+0000","last_undegraded":"2024-09-10T19:07:09.361531+0000","last_fullsized":"2024-09-10T19:07:09.361531+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:29.746751+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.746751+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.290888+0000","scrub_duration":0.0059564860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":246,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363508+0000","last_change":"2024-09-10T19:05:02.098565+0000","last_active":"2024-09-10T19:07:09.363508+0000","last_peered":"2024-09-10T19:07:09.363508+0000","last_clean":"2024-09-10T19:07:09.363508+0000","last_became_active":"2024-09-10T19:02:52.250472+0000","last_became_peered":"2024-09-10T19:02:52.250472+0000","last_unstale":"2024-09-10T19:07:09.363508+0000","last_undegraded":"2024-09-10T19:07:09.363508+0000","last_fullsized":"2024-09-10T19:07:09.363508+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.098384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.098384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034406860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":773,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356130+0000","last_change":"2024-09-10T19:06:31.820211+0000","last_active":"2024-09-10T19:07:09.356130+0000","last_peered":"2024-09-10T19:07:09.356130+0000","last_clean":"2024-09-10T19:07:09.356130+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:07:09.356130+0000","last_undegraded":"2024-09-10T19:07:09.356130+0000","last_fullsized":"2024-09-10T19:07:09.356130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.820051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.820051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.110063+0000","scrub_duration":0.003619667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":701,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812281+0000","last_change":"2024-09-10T19:06:36.108291+0000","last_active":"2024-09-10T19:07:09.812281+0000","last_peered":"2024-09-10T19:07:09.812281+0000","last_clean":"2024-09-10T19:07:09.812281+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:07:09.812281+0000","last_undegraded":"2024-09-10T19:07:09.812281+0000","last_fullsized":"2024-09-10T19:07:09.812281+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.107798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.107798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:36.908011+0000","scrub_duration":0.0058806149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":787,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808010+0000","last_change":"2024-09-10T19:06:36.900403+0000","last_active":"2024-09-10T19:07:09.808010+0000","last_peered":"2024-09-10T19:07:09.808010+0000","last_clean":"2024-09-10T19:07:09.808010+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:07:09.808010+0000","last_undegraded":"2024-09-10T19:07:09.808010+0000","last_fullsized":"2024-09-10T19:07:09.808010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.900156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.900156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.830082+0000","scrub_duration":0.0031758419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":821,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363052+0000","last_change":"2024-09-10T19:06:26.733413+0000","last_active":"2024-09-10T19:07:09.363052+0000","last_peered":"2024-09-10T19:07:09.363052+0000","last_clean":"2024-09-10T19:07:09.363052+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:07:09.363052+0000","last_undegraded":"2024-09-10T19:07:09.363052+0000","last_fullsized":"2024-09-10T19:07:09.363052+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:06:26.733241+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.733241+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.224379+0000","scrub_duration":0.0050063570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":256,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361706+0000","last_change":"2024-09-10T19:06:30.049476+0000","last_active":"2024-09-10T19:07:09.361706+0000","last_peered":"2024-09-10T19:07:09.361706+0000","last_clean":"2024-09-10T19:07:09.361706+0000","last_became_active":"2024-09-10T19:02:52.256764+0000","last_became_peered":"2024-09-10T19:02:52.256764+0000","last_unstale":"2024-09-10T19:07:09.361706+0000","last_undegraded":"2024-09-10T19:07:09.361706+0000","last_fullsized":"2024-09-10T19:07:09.361706+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.049307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.049307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.430304+0000","scrub_duration":0.0039529120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815257+0000","last_change":"2024-09-10T19:06:45.340292+0000","last_active":"2024-09-10T19:07:09.815257+0000","last_peered":"2024-09-10T19:07:09.815257+0000","last_clean":"2024-09-10T19:07:09.815257+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:07:09.815257+0000","last_undegraded":"2024-09-10T19:07:09.815257+0000","last_fullsized":"2024-09-10T19:07:09.815257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.340065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.340065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.926466+0000","scrub_duration":0.004677952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":688,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361579+0000","last_change":"2024-09-10T19:04:59.080471+0000","last_active":"2024-09-10T19:07:09.361579+0000","last_peered":"2024-09-10T19:07:09.361579+0000","last_clean":"2024-09-10T19:07:09.361579+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:07:09.361579+0000","last_undegraded":"2024-09-10T19:07:09.361579+0000","last_fullsized":"2024-09-10T19:07:09.361579+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.080362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.080362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028902760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":300,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356342+0000","last_change":"2024-09-10T19:06:58.888376+0000","last_active":"2024-09-10T19:07:09.356342+0000","last_peered":"2024-09-10T19:07:09.356342+0000","last_clean":"2024-09-10T19:07:09.356342+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:07:09.356342+0000","last_undegraded":"2024-09-10T19:07:09.356342+0000","last_fullsized":"2024-09-10T19:07:09.356342+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:58.888195+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:44.683992+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.888195+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:15.759441+0000","scrub_duration":0.0043236180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":842,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.772752+0000","last_change":"2024-09-10T19:07:01.719205+0000","last_active":"2024-09-10T19:07:09.772752+0000","last_peered":"2024-09-10T19:07:09.772752+0000","last_clean":"2024-09-10T19:07:09.772752+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:07:09.772752+0000","last_undegraded":"2024-09-10T19:07:09.772752+0000","last_fullsized":"2024-09-10T19:07:09.772752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:07:01.718013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.718013+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:28.914109+0000","scrub_duration":0.0078103089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808010+0000","last_change":"2024-09-10T19:07:02.278308+0000","last_active":"2024-09-10T19:07:09.808010+0000","last_peered":"2024-09-10T19:07:09.808010+0000","last_clean":"2024-09-10T19:07:09.808010+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:07:09.808010+0000","last_undegraded":"2024-09-10T19:07:09.808010+0000","last_fullsized":"2024-09-10T19:07:09.808010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:56.847674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:04:56.847674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033132719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"548'13","reported_seq":368,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366695+0000","last_change":"2024-09-10T19:05:24.947023+0000","last_active":"2024-09-10T19:07:09.366695+0000","last_peered":"2024-09-10T19:07:09.366695+0000","last_clean":"2024-09-10T19:07:09.366695+0000","last_became_active":"2024-09-10T19:02:52.256843+0000","last_became_peered":"2024-09-10T19:02:52.256843+0000","last_unstale":"2024-09-10T19:07:09.366695+0000","last_undegraded":"2024-09-10T19:07:09.366695+0000","last_fullsized":"2024-09-10T19:07:09.366695+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_deep_scrub":"548'13","last_deep_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_clean_scrub_stamp":"2024-09-10T19:05:24.946829+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036489069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":795,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.368861+0000","last_change":"2024-09-10T19:06:20.119294+0000","last_active":"2024-09-10T19:07:09.368861+0000","last_peered":"2024-09-10T19:07:09.368861+0000","last_clean":"2024-09-10T19:07:09.368861+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:07:09.368861+0000","last_undegraded":"2024-09-10T19:07:09.368861+0000","last_fullsized":"2024-09-10T19:07:09.368861+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:20.119151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.119151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:20.911363+0000","scrub_duration":0.0032863219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768199+0000","last_change":"2024-09-10T19:06:36.747901+0000","last_active":"2024-09-10T19:07:09.768199+0000","last_peered":"2024-09-10T19:07:09.768199+0000","last_clean":"2024-09-10T19:07:09.768199+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:07:09.768199+0000","last_undegraded":"2024-09-10T19:07:09.768199+0000","last_fullsized":"2024-09-10T19:07:09.768199+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.747814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.747814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.677402+0000","scrub_duration":0.0036046889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":852,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808761+0000","last_change":"2024-09-10T19:05:22.653245+0000","last_active":"2024-09-10T19:07:09.808761+0000","last_peered":"2024-09-10T19:07:09.808761+0000","last_clean":"2024-09-10T19:07:09.808761+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:07:09.808761+0000","last_undegraded":"2024-09-10T19:07:09.808761+0000","last_fullsized":"2024-09-10T19:07:09.808761+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:22.653050+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:22.653050+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040316939999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":807,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.364020+0000","last_change":"2024-09-10T19:05:21.852708+0000","last_active":"2024-09-10T19:07:09.364020+0000","last_peered":"2024-09-10T19:07:09.364020+0000","last_clean":"2024-09-10T19:07:09.364020+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:07:09.364020+0000","last_undegraded":"2024-09-10T19:07:09.364020+0000","last_fullsized":"2024-09-10T19:07:09.364020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:05:21.852558+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:05:21.852558+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041159459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":795,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809993+0000","last_change":"2024-09-10T19:06:38.068653+0000","last_active":"2024-09-10T19:07:09.809993+0000","last_peered":"2024-09-10T19:07:09.809993+0000","last_clean":"2024-09-10T19:07:09.809993+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:07:09.809993+0000","last_undegraded":"2024-09-10T19:07:09.809993+0000","last_fullsized":"2024-09-10T19:07:09.809993+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:38.068517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:06:38.068517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.453563+0000","scrub_duration":0.0031386930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":246,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365964+0000","last_change":"2024-09-10T19:05:01.085144+0000","last_active":"2024-09-10T19:07:09.365964+0000","last_peered":"2024-09-10T19:07:09.365964+0000","last_clean":"2024-09-10T19:07:09.365964+0000","last_became_active":"2024-09-10T19:02:52.254392+0000","last_became_peered":"2024-09-10T19:02:52.254392+0000","last_unstale":"2024-09-10T19:07:09.365964+0000","last_undegraded":"2024-09-10T19:07:09.365964+0000","last_fullsized":"2024-09-10T19:07:09.365964+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:01.085070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:01.085070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024364389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":254,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.813427+0000","last_change":"2024-09-10T19:06:31.119974+0000","last_active":"2024-09-10T19:07:09.813427+0000","last_peered":"2024-09-10T19:07:09.813427+0000","last_clean":"2024-09-10T19:07:09.813427+0000","last_became_active":"2024-09-10T19:02:52.258431+0000","last_became_peered":"2024-09-10T19:02:52.258431+0000","last_unstale":"2024-09-10T19:07:09.813427+0000","last_undegraded":"2024-09-10T19:07:09.813427+0000","last_fullsized":"2024-09-10T19:07:09.813427+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.119813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.119813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:35.705097+0000","scrub_duration":0.0040758110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768344+0000","last_change":"2024-09-10T19:06:53.837927+0000","last_active":"2024-09-10T19:07:09.768344+0000","last_peered":"2024-09-10T19:07:09.768344+0000","last_clean":"2024-09-10T19:07:09.768344+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:07:09.768344+0000","last_undegraded":"2024-09-10T19:07:09.768344+0000","last_fullsized":"2024-09-10T19:07:09.768344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.837830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.837830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.063555+0000","scrub_duration":0.0098916300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":283,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356858+0000","last_change":"2024-09-10T19:05:17.213897+0000","last_active":"2024-09-10T19:07:09.356858+0000","last_peered":"2024-09-10T19:07:09.356858+0000","last_clean":"2024-09-10T19:07:09.356858+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:07:09.356858+0000","last_undegraded":"2024-09-10T19:07:09.356858+0000","last_fullsized":"2024-09-10T19:07:09.356858+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:17.213629+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:05:17.213629+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038456739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":821,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.360527+0000","last_change":"2024-09-10T19:05:33.936164+0000","last_active":"2024-09-10T19:07:09.360527+0000","last_peered":"2024-09-10T19:07:09.360527+0000","last_clean":"2024-09-10T19:07:09.360527+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:07:09.360527+0000","last_undegraded":"2024-09-10T19:07:09.360527+0000","last_fullsized":"2024-09-10T19:07:09.360527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:05:33.935997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.935997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050458739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809878+0000","last_change":"2024-09-10T19:06:40.095694+0000","last_active":"2024-09-10T19:07:09.809878+0000","last_peered":"2024-09-10T19:07:09.809878+0000","last_clean":"2024-09-10T19:07:09.809878+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:07:09.809878+0000","last_undegraded":"2024-09-10T19:07:09.809878+0000","last_fullsized":"2024-09-10T19:07:09.809878+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:40.095535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:40.095535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.980205+0000","scrub_duration":0.003623636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":791,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812513+0000","last_change":"2024-09-10T19:06:59.211985+0000","last_active":"2024-09-10T19:07:09.812513+0000","last_peered":"2024-09-10T19:07:09.812513+0000","last_clean":"2024-09-10T19:07:09.812513+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:07:09.812513+0000","last_undegraded":"2024-09-10T19:07:09.812513+0000","last_fullsized":"2024-09-10T19:07:09.812513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:59.211830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.211830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.292523+0000","scrub_duration":0.0029718470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362473+0000","last_change":"2024-09-10T19:06:39.931182+0000","last_active":"2024-09-10T19:07:09.362473+0000","last_peered":"2024-09-10T19:07:09.362473+0000","last_clean":"2024-09-10T19:07:09.362473+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:07:09.362473+0000","last_undegraded":"2024-09-10T19:07:09.362473+0000","last_fullsized":"2024-09-10T19:07:09.362473+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:39.931027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.931027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:47.756981+0000","scrub_duration":0.0039840149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":701,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810754+0000","last_change":"2024-09-10T19:06:27.078822+0000","last_active":"2024-09-10T19:07:09.810754+0000","last_peered":"2024-09-10T19:07:09.810754+0000","last_clean":"2024-09-10T19:07:09.810754+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:07:09.810754+0000","last_undegraded":"2024-09-10T19:07:09.810754+0000","last_fullsized":"2024-09-10T19:07:09.810754+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:27.078734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:27.078734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:27.146592+0000","scrub_duration":0.0032307350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":831,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.614543+0000","last_change":"2024-09-10T19:06:38.692442+0000","last_active":"2024-09-10T19:07:09.614543+0000","last_peered":"2024-09-10T19:07:09.614543+0000","last_clean":"2024-09-10T19:07:09.614543+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:07:09.614543+0000","last_undegraded":"2024-09-10T19:07:09.614543+0000","last_fullsized":"2024-09-10T19:07:09.614543+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'23","last_scrub_stamp":"2024-09-10T19:06:38.692261+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T19:06:38.692261+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:53.304601+0000","scrub_duration":0.0046551179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":784,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809589+0000","last_change":"2024-09-10T19:05:15.605213+0000","last_active":"2024-09-10T19:07:09.809589+0000","last_peered":"2024-09-10T19:07:09.809589+0000","last_clean":"2024-09-10T19:07:09.809589+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:07:09.809589+0000","last_undegraded":"2024-09-10T19:07:09.809589+0000","last_fullsized":"2024-09-10T19:07:09.809589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.605072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.605072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072968429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"548'13","reported_seq":359,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.776156+0000","last_change":"2024-09-10T19:05:04.477156+0000","last_active":"2024-09-10T19:07:09.776156+0000","last_peered":"2024-09-10T19:07:09.776156+0000","last_clean":"2024-09-10T19:07:09.776156+0000","last_became_active":"2024-09-10T19:02:52.257016+0000","last_became_peered":"2024-09-10T19:02:52.257016+0000","last_unstale":"2024-09-10T19:07:09.776156+0000","last_undegraded":"2024-09-10T19:07:09.776156+0000","last_fullsized":"2024-09-10T19:07:09.776156+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:04.477051+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.477051+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031221360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":775,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365303+0000","last_change":"2024-09-10T19:06:22.027150+0000","last_active":"2024-09-10T19:07:09.365303+0000","last_peered":"2024-09-10T19:07:09.365303+0000","last_clean":"2024-09-10T19:07:09.365303+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:07:09.365303+0000","last_undegraded":"2024-09-10T19:07:09.365303+0000","last_fullsized":"2024-09-10T19:07:09.365303+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:06:22.027051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.027051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:23.425859+0000","scrub_duration":0.003390789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":694,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812537+0000","last_change":"2024-09-10T19:06:41.123978+0000","last_active":"2024-09-10T19:07:09.812537+0000","last_peered":"2024-09-10T19:07:09.812537+0000","last_clean":"2024-09-10T19:07:09.812537+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:07:09.812537+0000","last_undegraded":"2024-09-10T19:07:09.812537+0000","last_fullsized":"2024-09-10T19:07:09.812537+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.013632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.013632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003320365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":295,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365161+0000","last_change":"2024-09-10T19:06:42.135170+0000","last_active":"2024-09-10T19:07:09.365161+0000","last_peered":"2024-09-10T19:07:09.365161+0000","last_clean":"2024-09-10T19:07:09.365161+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:07:09.365161+0000","last_undegraded":"2024-09-10T19:07:09.365161+0000","last_fullsized":"2024-09-10T19:07:09.365161+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:42.135006+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.135006+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.854577+0000","scrub_duration":0.0049906150000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":808,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817282+0000","last_change":"2024-09-10T19:06:34.355770+0000","last_active":"2024-09-10T19:07:09.817282+0000","last_peered":"2024-09-10T19:07:09.817282+0000","last_clean":"2024-09-10T19:07:09.817282+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:07:09.817282+0000","last_undegraded":"2024-09-10T19:07:09.817282+0000","last_fullsized":"2024-09-10T19:07:09.817282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:34.355514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.355514+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.626969+0000","scrub_duration":0.005673483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.769679+0000","last_change":"2024-09-10T19:05:16.504389+0000","last_active":"2024-09-10T19:07:09.769679+0000","last_peered":"2024-09-10T19:07:09.769679+0000","last_clean":"2024-09-10T19:07:09.769679+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:07:09.769679+0000","last_undegraded":"2024-09-10T19:07:09.769679+0000","last_fullsized":"2024-09-10T19:07:09.769679+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.504234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.504234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040815139999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":240,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356332+0000","last_change":"2024-09-10T19:05:20.129549+0000","last_active":"2024-09-10T19:07:09.356332+0000","last_peered":"2024-09-10T19:07:09.356332+0000","last_clean":"2024-09-10T19:07:09.356332+0000","last_became_active":"2024-09-10T19:02:52.254942+0000","last_became_peered":"2024-09-10T19:02:52.254942+0000","last_unstale":"2024-09-10T19:07:09.356332+0000","last_undegraded":"2024-09-10T19:07:09.356332+0000","last_fullsized":"2024-09-10T19:07:09.356332+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:20.129398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:05:20.129398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033769329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1830,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:10.033933+0000","last_change":"2024-09-10T19:05:09.941903+0000","last_active":"2024-09-10T19:07:10.033933+0000","last_peered":"2024-09-10T19:07:10.033933+0000","last_clean":"2024-09-10T19:07:10.033933+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:07:10.033933+0000","last_undegraded":"2024-09-10T19:07:10.033933+0000","last_fullsized":"2024-09-10T19:07:10.033933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:05:09.941698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:09.941698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041460869999999997,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1046,"num_read_kb":1041,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810088+0000","last_change":"2024-09-10T19:05:07.665521+0000","last_active":"2024-09-10T19:07:09.810088+0000","last_peered":"2024-09-10T19:07:09.810088+0000","last_clean":"2024-09-10T19:07:09.810088+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:07:09.810088+0000","last_undegraded":"2024-09-10T19:07:09.810088+0000","last_fullsized":"2024-09-10T19:07:09.810088+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:07.665287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:05:07.665287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003665391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":717,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357545+0000","last_change":"2024-09-10T19:06:18.075252+0000","last_active":"2024-09-10T19:07:09.357545+0000","last_peered":"2024-09-10T19:07:09.357545+0000","last_clean":"2024-09-10T19:07:09.357545+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:07:09.357545+0000","last_undegraded":"2024-09-10T19:07:09.357545+0000","last_fullsized":"2024-09-10T19:07:09.357545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:18.075101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.075101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:43.342791+0000","scrub_duration":0.00339478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":297,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.609728+0000","last_change":"2024-09-10T19:06:46.615668+0000","last_active":"2024-09-10T19:07:09.609728+0000","last_peered":"2024-09-10T19:07:09.609728+0000","last_clean":"2024-09-10T19:07:09.609728+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:07:09.609728+0000","last_undegraded":"2024-09-10T19:07:09.609728+0000","last_fullsized":"2024-09-10T19:07:09.609728+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.615570+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:08.032708+0000","scrub_duration":0.0029648029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":811,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366305+0000","last_change":"2024-09-10T19:05:25.905116+0000","last_active":"2024-09-10T19:07:09.366305+0000","last_peered":"2024-09-10T19:07:09.366305+0000","last_clean":"2024-09-10T19:07:09.366305+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:07:09.366305+0000","last_undegraded":"2024-09-10T19:07:09.366305+0000","last_fullsized":"2024-09-10T19:07:09.366305+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'17","last_scrub_stamp":"2024-09-10T19:05:25.904954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:25.904954+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045887740000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":795,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810646+0000","last_change":"2024-09-10T19:06:48.128049+0000","last_active":"2024-09-10T19:07:09.810646+0000","last_peered":"2024-09-10T19:07:09.810646+0000","last_clean":"2024-09-10T19:07:09.810646+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:07:09.810646+0000","last_undegraded":"2024-09-10T19:07:09.810646+0000","last_fullsized":"2024-09-10T19:07:09.810646+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.127911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.127911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.454765+0000","scrub_duration":0.004079208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":991,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.360254+0000","last_change":"2024-09-10T19:05:32.921313+0000","last_active":"2024-09-10T19:07:09.360254+0000","last_peered":"2024-09-10T19:07:09.360254+0000","last_clean":"2024-09-10T19:07:09.360254+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:07:09.360254+0000","last_undegraded":"2024-09-10T19:07:09.360254+0000","last_fullsized":"2024-09-10T19:07:09.360254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:05:32.921124+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.921124+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004136547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":799,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.610243+0000","last_change":"2024-09-10T19:06:20.005964+0000","last_active":"2024-09-10T19:07:09.610243+0000","last_peered":"2024-09-10T19:07:09.610243+0000","last_clean":"2024-09-10T19:07:09.610243+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:07:09.610243+0000","last_undegraded":"2024-09-10T19:07:09.610243+0000","last_fullsized":"2024-09-10T19:07:09.610243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:06:20.005802+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.005802+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:41.479507+0000","scrub_duration":0.0042833380000000002,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":256,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362154+0000","last_change":"2024-09-10T19:06:23.076714+0000","last_active":"2024-09-10T19:07:09.362154+0000","last_peered":"2024-09-10T19:07:09.362154+0000","last_clean":"2024-09-10T19:07:09.362154+0000","last_became_active":"2024-09-10T19:02:52.256316+0000","last_became_peered":"2024-09-10T19:02:52.256316+0000","last_unstale":"2024-09-10T19:07:09.362154+0000","last_undegraded":"2024-09-10T19:07:09.362154+0000","last_fullsized":"2024-09-10T19:07:09.362154+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:23.076562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.076562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.850014+0000","scrub_duration":0.0034726789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":695,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.360247+0000","last_change":"2024-09-10T19:06:48.197543+0000","last_active":"2024-09-10T19:07:09.360247+0000","last_peered":"2024-09-10T19:07:09.360247+0000","last_clean":"2024-09-10T19:07:09.360247+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:07:09.360247+0000","last_undegraded":"2024-09-10T19:07:09.360247+0000","last_fullsized":"2024-09-10T19:07:09.360247+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.197394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.197394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.035900+0000","scrub_duration":0.003369632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":362,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.775167+0000","last_change":"2024-09-10T19:05:09.498115+0000","last_active":"2024-09-10T19:07:09.775167+0000","last_peered":"2024-09-10T19:07:09.775167+0000","last_clean":"2024-09-10T19:07:09.775167+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:07:09.775167+0000","last_undegraded":"2024-09-10T19:07:09.775167+0000","last_fullsized":"2024-09-10T19:07:09.775167+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:09.497849+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:09.497849+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043080979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":1075,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.814989+0000","last_change":"2024-09-10T19:05:27.874167+0000","last_active":"2024-09-10T19:07:09.814989+0000","last_peered":"2024-09-10T19:07:09.814989+0000","last_clean":"2024-09-10T19:07:09.814989+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:07:09.814989+0000","last_undegraded":"2024-09-10T19:07:09.814989+0000","last_fullsized":"2024-09-10T19:07:09.814989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:05:27.874075+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:05:27.874075+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046021589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":784,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808097+0000","last_change":"2024-09-10T19:06:35.858337+0000","last_active":"2024-09-10T19:07:09.808097+0000","last_peered":"2024-09-10T19:07:09.808097+0000","last_clean":"2024-09-10T19:07:09.808097+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:07:09.808097+0000","last_undegraded":"2024-09-10T19:07:09.808097+0000","last_fullsized":"2024-09-10T19:07:09.808097+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.858069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.858069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.148804+0000","scrub_duration":0.0032278900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":995,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.816148+0000","last_change":"2024-09-10T19:05:25.279549+0000","last_active":"2024-09-10T19:07:09.816148+0000","last_peered":"2024-09-10T19:07:09.816148+0000","last_clean":"2024-09-10T19:07:09.816148+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:07:09.816148+0000","last_undegraded":"2024-09-10T19:07:09.816148+0000","last_fullsized":"2024-09-10T19:07:09.816148+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:05:25.279409+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:05:25.279409+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036183930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":783,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.772490+0000","last_change":"2024-09-10T19:05:12.500579+0000","last_active":"2024-09-10T19:07:09.772490+0000","last_peered":"2024-09-10T19:07:09.772490+0000","last_clean":"2024-09-10T19:07:09.772490+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:07:09.772490+0000","last_undegraded":"2024-09-10T19:07:09.772490+0000","last_fullsized":"2024-09-10T19:07:09.772490+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:05:12.500502+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:05:12.500502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035677880000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811287+0000","last_change":"2024-09-10T19:06:30.081763+0000","last_active":"2024-09-10T19:07:09.811287+0000","last_peered":"2024-09-10T19:07:09.811287+0000","last_clean":"2024-09-10T19:07:09.811287+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:07:09.811287+0000","last_undegraded":"2024-09-10T19:07:09.811287+0000","last_fullsized":"2024-09-10T19:07:09.811287+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.081613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.081613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:45.513032+0000","scrub_duration":0.003447449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":695,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.364390+0000","last_change":"2024-09-10T19:06:17.163823+0000","last_active":"2024-09-10T19:07:09.364390+0000","last_peered":"2024-09-10T19:07:09.364390+0000","last_clean":"2024-09-10T19:07:09.364390+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:07:09.364390+0000","last_undegraded":"2024-09-10T19:07:09.364390+0000","last_fullsized":"2024-09-10T19:07:09.364390+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.163656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.163656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:23.635993+0000","scrub_duration":0.0035671100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":365,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.776285+0000","last_change":"2024-09-10T19:06:18.604662+0000","last_active":"2024-09-10T19:07:09.776285+0000","last_peered":"2024-09-10T19:07:09.776285+0000","last_clean":"2024-09-10T19:07:09.776285+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:07:09.776285+0000","last_undegraded":"2024-09-10T19:07:09.776285+0000","last_fullsized":"2024-09-10T19:07:09.776285+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:18.604383+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.604383+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:24.775342+0000","scrub_duration":0.0042316089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"760'33","reported_seq":831,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363594+0000","last_change":"2024-09-10T19:06:59.942589+0000","last_active":"2024-09-10T19:07:09.363594+0000","last_peered":"2024-09-10T19:07:09.363594+0000","last_clean":"2024-09-10T19:07:09.363594+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:07:09.363594+0000","last_undegraded":"2024-09-10T19:07:09.363594+0000","last_fullsized":"2024-09-10T19:07:09.363594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"760'33","last_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_deep_scrub":"760'33","last_deep_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.942479+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:07.358058+0000","scrub_duration":0.012061568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815080+0000","last_change":"2024-09-10T19:06:26.032434+0000","last_active":"2024-09-10T19:07:09.815080+0000","last_peered":"2024-09-10T19:07:09.815080+0000","last_clean":"2024-09-10T19:07:09.815080+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:07:09.815080+0000","last_undegraded":"2024-09-10T19:07:09.815080+0000","last_fullsized":"2024-09-10T19:07:09.815080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.032276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.032276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.801865+0000","scrub_duration":0.0033796479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":781,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812342+0000","last_change":"2024-09-10T19:06:55.205119+0000","last_active":"2024-09-10T19:07:09.812342+0000","last_peered":"2024-09-10T19:07:09.812342+0000","last_clean":"2024-09-10T19:07:09.812342+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:07:09.812342+0000","last_undegraded":"2024-09-10T19:07:09.812342+0000","last_fullsized":"2024-09-10T19:07:09.812342+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.204933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.204933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.228743+0000","scrub_duration":0.0032554939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":768,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815827+0000","last_change":"2024-09-10T19:05:04.441891+0000","last_active":"2024-09-10T19:07:09.815827+0000","last_peered":"2024-09-10T19:07:09.815827+0000","last_clean":"2024-09-10T19:07:09.815827+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:07:09.815827+0000","last_undegraded":"2024-09-10T19:07:09.815827+0000","last_fullsized":"2024-09-10T19:07:09.815827+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:04.441647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.441647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029219010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":717,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.358463+0000","last_change":"2024-09-10T19:06:22.182306+0000","last_active":"2024-09-10T19:07:09.358463+0000","last_peered":"2024-09-10T19:07:09.358463+0000","last_clean":"2024-09-10T19:07:09.358463+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:07:09.358463+0000","last_undegraded":"2024-09-10T19:07:09.358463+0000","last_fullsized":"2024-09-10T19:07:09.358463+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.182115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.182115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:36.337406+0000","scrub_duration":0.0036403120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":293,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.355513+0000","last_change":"2024-09-10T19:06:53.016161+0000","last_active":"2024-09-10T19:07:09.355513+0000","last_peered":"2024-09-10T19:07:09.355513+0000","last_clean":"2024-09-10T19:07:09.355513+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:07:09.355513+0000","last_undegraded":"2024-09-10T19:07:09.355513+0000","last_fullsized":"2024-09-10T19:07:09.355513+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:53.015994+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.015994+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.394220+0000","scrub_duration":0.0033683329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":700,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815657+0000","last_change":"2024-09-10T19:06:39.394175+0000","last_active":"2024-09-10T19:07:09.815657+0000","last_peered":"2024-09-10T19:07:09.815657+0000","last_clean":"2024-09-10T19:07:09.815657+0000","last_became_active":"2024-09-10T19:02:52.256210+0000","last_became_peered":"2024-09-10T19:02:52.256210+0000","last_unstale":"2024-09-10T19:07:09.815657+0000","last_undegraded":"2024-09-10T19:07:09.815657+0000","last_fullsized":"2024-09-10T19:07:09.815657+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:06:39.394055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.394055+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.691949+0000","scrub_duration":0.0034977820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":776,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357986+0000","last_change":"2024-09-10T19:05:28.136902+0000","last_active":"2024-09-10T19:07:09.357986+0000","last_peered":"2024-09-10T19:07:09.357986+0000","last_clean":"2024-09-10T19:07:09.357986+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:07:09.357986+0000","last_undegraded":"2024-09-10T19:07:09.357986+0000","last_fullsized":"2024-09-10T19:07:09.357986+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.136755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032446879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":781,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.814927+0000","last_change":"2024-09-10T19:05:19.903060+0000","last_active":"2024-09-10T19:07:09.814927+0000","last_peered":"2024-09-10T19:07:09.814927+0000","last_clean":"2024-09-10T19:07:09.814927+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:07:09.814927+0000","last_undegraded":"2024-09-10T19:07:09.814927+0000","last_fullsized":"2024-09-10T19:07:09.814927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:19.902898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.902898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037634550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":767,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365376+0000","last_change":"2024-09-10T19:05:19.919334+0000","last_active":"2024-09-10T19:07:09.365376+0000","last_peered":"2024-09-10T19:07:09.365376+0000","last_clean":"2024-09-10T19:07:09.365376+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:07:09.365376+0000","last_undegraded":"2024-09-10T19:07:09.365376+0000","last_fullsized":"2024-09-10T19:07:09.365376+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:19.919226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.919226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809853+0000","last_change":"2024-09-10T19:06:26.273000+0000","last_active":"2024-09-10T19:07:09.809853+0000","last_peered":"2024-09-10T19:07:09.809853+0000","last_clean":"2024-09-10T19:07:09.809853+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:07:09.809853+0000","last_undegraded":"2024-09-10T19:07:09.809853+0000","last_fullsized":"2024-09-10T19:07:09.809853+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.272837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.272837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.951213+0000","scrub_duration":0.0046824379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":852,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809337+0000","last_change":"2024-09-10T19:05:29.794987+0000","last_active":"2024-09-10T19:07:09.809337+0000","last_peered":"2024-09-10T19:07:09.809337+0000","last_clean":"2024-09-10T19:07:09.809337+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:07:09.809337+0000","last_undegraded":"2024-09-10T19:07:09.809337+0000","last_fullsized":"2024-09-10T19:07:09.809337+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:29.794822+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:29.794822+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003393437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":851,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811149+0000","last_change":"2024-09-10T19:07:01.193377+0000","last_active":"2024-09-10T19:07:09.811149+0000","last_peered":"2024-09-10T19:07:09.811149+0000","last_clean":"2024-09-10T19:07:09.811149+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:07:09.811149+0000","last_undegraded":"2024-09-10T19:07:09.811149+0000","last_fullsized":"2024-09-10T19:07:09.811149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:07:01.193206+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.193206+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.775475+0000","scrub_duration":0.005269892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":811,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357418+0000","last_change":"2024-09-10T19:06:50.068060+0000","last_active":"2024-09-10T19:07:09.357418+0000","last_peered":"2024-09-10T19:07:09.357418+0000","last_clean":"2024-09-10T19:07:09.357418+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:07:09.357418+0000","last_undegraded":"2024-09-10T19:07:09.357418+0000","last_fullsized":"2024-09-10T19:07:09.357418+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:50.067884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.067884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:59.606576+0000","scrub_duration":0.0049528419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":781,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812653+0000","last_change":"2024-09-10T19:06:54.223011+0000","last_active":"2024-09-10T19:07:09.812653+0000","last_peered":"2024-09-10T19:07:09.812653+0000","last_clean":"2024-09-10T19:07:09.812653+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:07:09.812653+0000","last_undegraded":"2024-09-10T19:07:09.812653+0000","last_fullsized":"2024-09-10T19:07:09.812653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.222842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:12.925817+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.222842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.955782+0000","scrub_duration":0.0034650459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811789+0000","last_change":"2024-09-10T19:05:14.580485+0000","last_active":"2024-09-10T19:07:09.811789+0000","last_peered":"2024-09-10T19:07:09.811789+0000","last_clean":"2024-09-10T19:07:09.811789+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:07:09.811789+0000","last_undegraded":"2024-09-10T19:07:09.811789+0000","last_fullsized":"2024-09-10T19:07:09.811789+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:14.580383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:14.580383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026901339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":699,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.769315+0000","last_change":"2024-09-10T19:06:28.611791+0000","last_active":"2024-09-10T19:07:09.769315+0000","last_peered":"2024-09-10T19:07:09.769315+0000","last_clean":"2024-09-10T19:07:09.769315+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:07:09.769315+0000","last_undegraded":"2024-09-10T19:07:09.769315+0000","last_fullsized":"2024-09-10T19:07:09.769315+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:28.611520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.611520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:38.340443+0000","scrub_duration":0.0041400660000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":288,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811981+0000","last_change":"2024-09-10T19:06:25.036690+0000","last_active":"2024-09-10T19:07:09.811981+0000","last_peered":"2024-09-10T19:07:09.811981+0000","last_clean":"2024-09-10T19:07:09.811981+0000","last_became_active":"2024-09-10T19:02:03.616064+0000","last_became_peered":"2024-09-10T19:02:03.616064+0000","last_unstale":"2024-09-10T19:07:09.811981+0000","last_undegraded":"2024-09-10T19:07:09.811981+0000","last_fullsized":"2024-09-10T19:07:09.811981+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:25.036490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:25.036490+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.640822+0000","scrub_duration":0.0037518489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":814,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815912+0000","last_change":"2024-09-10T19:04:54.545871+0000","last_active":"2024-09-10T19:07:09.815912+0000","last_peered":"2024-09-10T19:07:09.815912+0000","last_clean":"2024-09-10T19:07:09.815912+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:07:09.815912+0000","last_undegraded":"2024-09-10T19:07:09.815912+0000","last_fullsized":"2024-09-10T19:07:09.815912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'28","last_scrub_stamp":"2024-09-10T19:04:54.545674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:04:54.545674+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049500350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":784,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808975+0000","last_change":"2024-09-10T19:05:04.758120+0000","last_active":"2024-09-10T19:07:09.808975+0000","last_peered":"2024-09-10T19:07:09.808975+0000","last_clean":"2024-09-10T19:07:09.808975+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:07:09.808975+0000","last_undegraded":"2024-09-10T19:07:09.808975+0000","last_fullsized":"2024-09-10T19:07:09.808975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:04.757950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.757950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032582150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":773,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817381+0000","last_change":"2024-09-10T19:05:10.481860+0000","last_active":"2024-09-10T19:07:09.817381+0000","last_peered":"2024-09-10T19:07:09.817381+0000","last_clean":"2024-09-10T19:07:09.817381+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:07:09.817381+0000","last_undegraded":"2024-09-10T19:07:09.817381+0000","last_fullsized":"2024-09-10T19:07:09.817381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.481693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.481693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037720449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.814752+0000","last_change":"2024-09-10T19:05:15.852430+0000","last_active":"2024-09-10T19:07:09.814752+0000","last_peered":"2024-09-10T19:07:09.814752+0000","last_clean":"2024-09-10T19:07:09.814752+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:07:09.814752+0000","last_undegraded":"2024-09-10T19:07:09.814752+0000","last_fullsized":"2024-09-10T19:07:09.814752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.852281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081435180000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810216+0000","last_change":"2024-09-10T19:06:21.347985+0000","last_active":"2024-09-10T19:07:09.810216+0000","last_peered":"2024-09-10T19:07:09.810216+0000","last_clean":"2024-09-10T19:07:09.810216+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:07:09.810216+0000","last_undegraded":"2024-09-10T19:07:09.810216+0000","last_fullsized":"2024-09-10T19:07:09.810216+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:21.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:21.347904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:28.385674+0000","scrub_duration":0.003263929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":803,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361846+0000","last_change":"2024-09-10T19:05:20.886436+0000","last_active":"2024-09-10T19:07:09.361846+0000","last_peered":"2024-09-10T19:07:09.361846+0000","last_clean":"2024-09-10T19:07:09.361846+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:07:09.361846+0000","last_undegraded":"2024-09-10T19:07:09.361846+0000","last_fullsized":"2024-09-10T19:07:09.361846+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:20.886267+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:05:20.886267+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049540019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":786,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.359351+0000","last_change":"2024-09-10T19:06:51.097568+0000","last_active":"2024-09-10T19:07:09.359351+0000","last_peered":"2024-09-10T19:07:09.359351+0000","last_clean":"2024-09-10T19:07:09.359351+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:07:09.359351+0000","last_undegraded":"2024-09-10T19:07:09.359351+0000","last_fullsized":"2024-09-10T19:07:09.359351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:51.097406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:06:51.097406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:16.803993+0000","scrub_duration":0.003232005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":843,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809185+0000","last_change":"2024-09-10T19:05:32.752121+0000","last_active":"2024-09-10T19:07:09.809185+0000","last_peered":"2024-09-10T19:07:09.809185+0000","last_clean":"2024-09-10T19:07:09.809185+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:07:09.809185+0000","last_undegraded":"2024-09-10T19:07:09.809185+0000","last_fullsized":"2024-09-10T19:07:09.809185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:32.752014+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:05:32.752014+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.752014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028322999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":775,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.775777+0000","last_change":"2024-09-10T19:05:22.537528+0000","last_active":"2024-09-10T19:07:09.775777+0000","last_peered":"2024-09-10T19:07:09.775777+0000","last_clean":"2024-09-10T19:07:09.775777+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:07:09.775777+0000","last_undegraded":"2024-09-10T19:07:09.775777+0000","last_fullsized":"2024-09-10T19:07:09.775777+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:22.537370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:05:22.537370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038042839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":773,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362890+0000","last_change":"2024-09-10T19:06:55.021461+0000","last_active":"2024-09-10T19:07:09.362890+0000","last_peered":"2024-09-10T19:07:09.362890+0000","last_clean":"2024-09-10T19:07:09.362890+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:07:09.362890+0000","last_undegraded":"2024-09-10T19:07:09.362890+0000","last_fullsized":"2024-09-10T19:07:09.362890+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.021218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.148405+0000","scrub_duration":0.0037286939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":702,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.808198+0000","last_change":"2024-09-10T19:06:45.994023+0000","last_active":"2024-09-10T19:07:09.808198+0000","last_peered":"2024-09-10T19:07:09.808198+0000","last_clean":"2024-09-10T19:07:09.808198+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:07:09.808198+0000","last_undegraded":"2024-09-10T19:07:09.808198+0000","last_fullsized":"2024-09-10T19:07:09.808198+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.993852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.778147+0000","scrub_duration":0.0035603079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":839,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812102+0000","last_change":"2024-09-10T19:06:47.011623+0000","last_active":"2024-09-10T19:07:09.812102+0000","last_peered":"2024-09-10T19:07:09.812102+0000","last_clean":"2024-09-10T19:07:09.812102+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:07:09.812102+0000","last_undegraded":"2024-09-10T19:07:09.812102+0000","last_fullsized":"2024-09-10T19:07:09.812102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_deep_scrub":"410'39","last_deep_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.011531+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.270825+0000","scrub_duration":0.0070935080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":811,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365687+0000","last_change":"2024-09-10T19:06:44.173326+0000","last_active":"2024-09-10T19:07:09.365687+0000","last_peered":"2024-09-10T19:07:09.365687+0000","last_clean":"2024-09-10T19:07:09.365687+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:07:09.365687+0000","last_undegraded":"2024-09-10T19:07:09.365687+0000","last_fullsized":"2024-09-10T19:07:09.365687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:44.173162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:44.173162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:44.806723+0000","scrub_duration":0.0034336560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.772674+0000","last_change":"2024-09-10T19:06:47.846775+0000","last_active":"2024-09-10T19:07:09.772674+0000","last_peered":"2024-09-10T19:07:09.772674+0000","last_clean":"2024-09-10T19:07:09.772674+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:07:09.772674+0000","last_undegraded":"2024-09-10T19:07:09.772674+0000","last_fullsized":"2024-09-10T19:07:09.772674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:47.846685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.846685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.395515+0000","scrub_duration":0.002537961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809740+0000","last_change":"2024-09-10T19:06:34.889988+0000","last_active":"2024-09-10T19:07:09.809740+0000","last_peered":"2024-09-10T19:07:09.809740+0000","last_clean":"2024-09-10T19:07:09.809740+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:07:09.809740+0000","last_undegraded":"2024-09-10T19:07:09.809740+0000","last_fullsized":"2024-09-10T19:07:09.809740+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:34.889826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.889826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.178210+0000","scrub_duration":0.0038908760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":711,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.607857+0000","last_change":"2024-09-10T19:06:51.691510+0000","last_active":"2024-09-10T19:07:09.607857+0000","last_peered":"2024-09-10T19:07:09.607857+0000","last_clean":"2024-09-10T19:07:09.607857+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:07:09.607857+0000","last_undegraded":"2024-09-10T19:07:09.607857+0000","last_fullsized":"2024-09-10T19:07:09.607857+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:51.691275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:51.691275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.627326+0000","scrub_duration":0.0081710110000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":809,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361701+0000","last_change":"2024-09-10T19:05:32.092434+0000","last_active":"2024-09-10T19:07:09.361701+0000","last_peered":"2024-09-10T19:07:09.361701+0000","last_clean":"2024-09-10T19:07:09.361701+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:07:09.361701+0000","last_undegraded":"2024-09-10T19:07:09.361701+0000","last_fullsized":"2024-09-10T19:07:09.361701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:32.092303+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.092303+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045903259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810663+0000","last_change":"2024-09-10T19:06:35.122084+0000","last_active":"2024-09-10T19:07:09.810663+0000","last_peered":"2024-09-10T19:07:09.810663+0000","last_clean":"2024-09-10T19:07:09.810663+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:07:09.810663+0000","last_undegraded":"2024-09-10T19:07:09.810663+0000","last_fullsized":"2024-09-10T19:07:09.810663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.121923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.121923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.025804+0000","scrub_duration":0.003146706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":770,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815474+0000","last_change":"2024-09-10T19:06:48.338457+0000","last_active":"2024-09-10T19:07:09.815474+0000","last_peered":"2024-09-10T19:07:09.815474+0000","last_clean":"2024-09-10T19:07:09.815474+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:07:09.815474+0000","last_undegraded":"2024-09-10T19:07:09.815474+0000","last_fullsized":"2024-09-10T19:07:09.815474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.338294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.338294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.950760+0000","scrub_duration":0.0033152189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":768,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815819+0000","last_change":"2024-09-10T19:06:53.347534+0000","last_active":"2024-09-10T19:07:09.815819+0000","last_peered":"2024-09-10T19:07:09.815819+0000","last_clean":"2024-09-10T19:07:09.815819+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:07:09.815819+0000","last_undegraded":"2024-09-10T19:07:09.815819+0000","last_fullsized":"2024-09-10T19:07:09.815819+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.347358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:07.952993+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.347358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.522684+0000","scrub_duration":0.0038827929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809703+0000","last_change":"2024-09-10T19:05:15.465247+0000","last_active":"2024-09-10T19:07:09.809703+0000","last_peered":"2024-09-10T19:07:09.809703+0000","last_clean":"2024-09-10T19:07:09.809703+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:07:09.809703+0000","last_undegraded":"2024-09-10T19:07:09.809703+0000","last_fullsized":"2024-09-10T19:07:09.809703+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.465081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:46.602792+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.465081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036040730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":832,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.813542+0000","last_change":"2024-09-10T19:05:19.619655+0000","last_active":"2024-09-10T19:07:09.813542+0000","last_peered":"2024-09-10T19:07:09.813542+0000","last_clean":"2024-09-10T19:07:09.813542+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:07:09.813542+0000","last_undegraded":"2024-09-10T19:07:09.813542+0000","last_fullsized":"2024-09-10T19:07:09.813542+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:05:19.619510+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.619510+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045528269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":823,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.368153+0000","last_change":"2024-09-10T19:06:37.103632+0000","last_active":"2024-09-10T19:07:09.368153+0000","last_peered":"2024-09-10T19:07:09.368153+0000","last_clean":"2024-09-10T19:07:09.368153+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:07:09.368153+0000","last_undegraded":"2024-09-10T19:07:09.368153+0000","last_fullsized":"2024-09-10T19:07:09.368153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:06:37.103448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.103448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:53.430585+0000","scrub_duration":0.0050576179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":782,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356605+0000","last_change":"2024-09-10T19:06:22.828266+0000","last_active":"2024-09-10T19:07:09.356605+0000","last_peered":"2024-09-10T19:07:09.356605+0000","last_clean":"2024-09-10T19:07:09.356605+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:07:09.356605+0000","last_undegraded":"2024-09-10T19:07:09.356605+0000","last_fullsized":"2024-09-10T19:07:09.356605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.828113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.828113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.146242+0000","scrub_duration":0.0031827000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357602+0000","last_change":"2024-09-10T19:06:53.989492+0000","last_active":"2024-09-10T19:07:09.357602+0000","last_peered":"2024-09-10T19:07:09.357602+0000","last_clean":"2024-09-10T19:07:09.357602+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:07:09.357602+0000","last_undegraded":"2024-09-10T19:07:09.357602+0000","last_fullsized":"2024-09-10T19:07:09.357602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.989397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.989397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.968837+0000","scrub_duration":0.0021082940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":250,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.771662+0000","last_change":"2024-09-10T19:06:29.614256+0000","last_active":"2024-09-10T19:07:09.771662+0000","last_peered":"2024-09-10T19:07:09.771662+0000","last_clean":"2024-09-10T19:07:09.771662+0000","last_became_active":"2024-09-10T19:02:52.256701+0000","last_became_peered":"2024-09-10T19:02:52.256701+0000","last_unstale":"2024-09-10T19:07:09.771662+0000","last_undegraded":"2024-09-10T19:07:09.771662+0000","last_fullsized":"2024-09-10T19:07:09.771662+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:29.614071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.614071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.354381+0000","scrub_duration":0.0030890639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":806,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.813111+0000","last_change":"2024-09-10T19:04:53.827646+0000","last_active":"2024-09-10T19:07:09.813111+0000","last_peered":"2024-09-10T19:07:09.813111+0000","last_clean":"2024-09-10T19:07:09.813111+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:07:09.813111+0000","last_undegraded":"2024-09-10T19:07:09.813111+0000","last_fullsized":"2024-09-10T19:07:09.813111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:04:53.827480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:04:53.827480+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086584599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"601'4","reported_seq":837,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.614724+0000","last_change":"2024-09-10T19:06:18.041112+0000","last_active":"2024-09-10T19:07:09.614724+0000","last_peered":"2024-09-10T19:07:09.614724+0000","last_clean":"2024-09-10T19:07:09.614724+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:07:09.614724+0000","last_undegraded":"2024-09-10T19:07:09.614724+0000","last_fullsized":"2024-09-10T19:07:09.614724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'4","last_scrub_stamp":"2024-09-10T19:06:18.041030+0000","last_deep_scrub":"601'4","last_deep_scrub_stamp":"2024-09-10T19:04:04.923921+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.041030+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.678787+0000","scrub_duration":0.0030506399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"548'12","reported_seq":327,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811518+0000","last_change":"2024-09-10T19:06:22.009512+0000","last_active":"2024-09-10T19:07:09.811518+0000","last_peered":"2024-09-10T19:07:09.811518+0000","last_clean":"2024-09-10T19:07:09.811518+0000","last_became_active":"2024-09-10T19:02:52.258504+0000","last_became_peered":"2024-09-10T19:02:52.258504+0000","last_unstale":"2024-09-10T19:07:09.811518+0000","last_undegraded":"2024-09-10T19:07:09.811518+0000","last_fullsized":"2024-09-10T19:07:09.811518+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-10T19:06:22.009440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.009440+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:43.875145+0000","scrub_duration":0.003198631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":774,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363361+0000","last_change":"2024-09-10T19:07:00.155780+0000","last_active":"2024-09-10T19:07:09.363361+0000","last_peered":"2024-09-10T19:07:09.363361+0000","last_clean":"2024-09-10T19:07:09.363361+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:07:09.363361+0000","last_undegraded":"2024-09-10T19:07:09.363361+0000","last_fullsized":"2024-09-10T19:07:09.363361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:00.155684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:07:00.155684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.453486+0000","scrub_duration":0.0084607129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":709,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.767866+0000","last_change":"2024-09-10T19:05:13.547418+0000","last_active":"2024-09-10T19:07:09.767866+0000","last_peered":"2024-09-10T19:07:09.767866+0000","last_clean":"2024-09-10T19:07:09.767866+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:07:09.767866+0000","last_undegraded":"2024-09-10T19:07:09.767866+0000","last_fullsized":"2024-09-10T19:07:09.767866+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:05:13.547259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:13.547259+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040562979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":823,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809156+0000","last_change":"2024-09-10T19:06:57.214131+0000","last_active":"2024-09-10T19:07:09.809156+0000","last_peered":"2024-09-10T19:07:09.809156+0000","last_clean":"2024-09-10T19:07:09.809156+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:07:09.809156+0000","last_undegraded":"2024-09-10T19:07:09.809156+0000","last_fullsized":"2024-09-10T19:07:09.809156+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:57.213972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:57.213972+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.939374+0000","scrub_duration":0.0047875110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":811,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356601+0000","last_change":"2024-09-10T19:06:55.984909+0000","last_active":"2024-09-10T19:07:09.356601+0000","last_peered":"2024-09-10T19:07:09.356601+0000","last_clean":"2024-09-10T19:07:09.356601+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:07:09.356601+0000","last_undegraded":"2024-09-10T19:07:09.356601+0000","last_fullsized":"2024-09-10T19:07:09.356601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.984769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.984769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.226594+0000","scrub_duration":0.0032115210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":770,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817172+0000","last_change":"2024-09-10T19:05:28.258466+0000","last_active":"2024-09-10T19:07:09.817172+0000","last_peered":"2024-09-10T19:07:09.817172+0000","last_clean":"2024-09-10T19:07:09.817172+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:07:09.817172+0000","last_undegraded":"2024-09-10T19:07:09.817172+0000","last_fullsized":"2024-09-10T19:07:09.817172+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.258181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.258181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037729090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":795,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365216+0000","last_change":"2024-09-10T19:06:31.188342+0000","last_active":"2024-09-10T19:07:09.365216+0000","last_peered":"2024-09-10T19:07:09.365216+0000","last_clean":"2024-09-10T19:07:09.365216+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:07:09.365216+0000","last_undegraded":"2024-09-10T19:07:09.365216+0000","last_fullsized":"2024-09-10T19:07:09.365216+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.188235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.188235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.059595+0000","scrub_duration":0.002958315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":711,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.614786+0000","last_change":"2024-09-10T19:06:56.683660+0000","last_active":"2024-09-10T19:07:09.614786+0000","last_peered":"2024-09-10T19:07:09.614786+0000","last_clean":"2024-09-10T19:07:09.614786+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:07:09.614786+0000","last_undegraded":"2024-09-10T19:07:09.614786+0000","last_fullsized":"2024-09-10T19:07:09.614786+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:56.683587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:56.683587+0000","last_clean_scrub_stamp":"2024-09-10T19:06:56.683587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.008641+0000","scrub_duration":0.0099833200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":801,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.772096+0000","last_change":"2024-09-10T19:06:52.872642+0000","last_active":"2024-09-10T19:07:09.772096+0000","last_peered":"2024-09-10T19:07:09.772096+0000","last_clean":"2024-09-10T19:07:09.772096+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:07:09.772096+0000","last_undegraded":"2024-09-10T19:07:09.772096+0000","last_fullsized":"2024-09-10T19:07:09.772096+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:06:52.872480+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.872480+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.358965+0000","scrub_duration":0.0042870599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":789,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.767488+0000","last_change":"2024-09-10T19:06:33.682027+0000","last_active":"2024-09-10T19:07:09.767488+0000","last_peered":"2024-09-10T19:07:09.767488+0000","last_clean":"2024-09-10T19:07:09.767488+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:07:09.767488+0000","last_undegraded":"2024-09-10T19:07:09.767488+0000","last_fullsized":"2024-09-10T19:07:09.767488+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:33.681850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:33.681850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.843090+0000","scrub_duration":0.0033460909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":766,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366100+0000","last_change":"2024-09-10T19:05:16.995554+0000","last_active":"2024-09-10T19:07:09.366100+0000","last_peered":"2024-09-10T19:07:09.366100+0000","last_clean":"2024-09-10T19:07:09.366100+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:07:09.366100+0000","last_undegraded":"2024-09-10T19:07:09.366100+0000","last_fullsized":"2024-09-10T19:07:09.366100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.995408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.995408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035441219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811628+0000","last_change":"2024-09-10T19:05:16.578048+0000","last_active":"2024-09-10T19:07:09.811628+0000","last_peered":"2024-09-10T19:07:09.811628+0000","last_clean":"2024-09-10T19:07:09.811628+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:07:09.811628+0000","last_undegraded":"2024-09-10T19:07:09.811628+0000","last_fullsized":"2024-09-10T19:07:09.811628+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.577898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.577898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025345699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.769472+0000","last_change":"2024-09-10T19:05:10.514126+0000","last_active":"2024-09-10T19:07:09.769472+0000","last_peered":"2024-09-10T19:07:09.769472+0000","last_clean":"2024-09-10T19:07:09.769472+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:07:09.769472+0000","last_undegraded":"2024-09-10T19:07:09.769472+0000","last_fullsized":"2024-09-10T19:07:09.769472+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.513974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.513974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003294848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":818,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810473+0000","last_change":"2024-09-10T19:06:43.983711+0000","last_active":"2024-09-10T19:07:09.810473+0000","last_peered":"2024-09-10T19:07:09.810473+0000","last_clean":"2024-09-10T19:07:09.810473+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:07:09.810473+0000","last_undegraded":"2024-09-10T19:07:09.810473+0000","last_fullsized":"2024-09-10T19:07:09.810473+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_deep_scrub":"410'24","last_deep_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.983521+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:08.138671+0000","scrub_duration":0.005511228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.358145+0000","last_change":"2024-09-10T19:06:32.786736+0000","last_active":"2024-09-10T19:07:09.358145+0000","last_peered":"2024-09-10T19:07:09.358145+0000","last_clean":"2024-09-10T19:07:09.358145+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:07:09.358145+0000","last_undegraded":"2024-09-10T19:07:09.358145+0000","last_fullsized":"2024-09-10T19:07:09.358145+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.786573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.786573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.715827+0000","scrub_duration":0.010562732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"586'12","reported_seq":1004,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366270+0000","last_change":"2024-09-10T19:06:20.042046+0000","last_active":"2024-09-10T19:07:09.366270+0000","last_peered":"2024-09-10T19:07:09.366270+0000","last_clean":"2024-09-10T19:07:09.366270+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:07:09.366270+0000","last_undegraded":"2024-09-10T19:07:09.366270+0000","last_fullsized":"2024-09-10T19:07:09.366270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"586'12","last_scrub_stamp":"2024-09-10T19:06:20.041985+0000","last_deep_scrub":"586'12","last_deep_scrub_stamp":"2024-09-10T19:03:50.172621+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.041985+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:24.587884+0000","scrub_duration":0.003891285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":768,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.816971+0000","last_change":"2024-09-10T19:05:17.407644+0000","last_active":"2024-09-10T19:07:09.816971+0000","last_peered":"2024-09-10T19:07:09.816971+0000","last_clean":"2024-09-10T19:07:09.816971+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:07:09.816971+0000","last_undegraded":"2024-09-10T19:07:09.816971+0000","last_fullsized":"2024-09-10T19:07:09.816971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:17.407512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.407512+0000","last_clean_scrub_stamp":"2024-09-10T19:05:17.407512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028230099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809659+0000","last_change":"2024-09-10T19:06:30.306917+0000","last_active":"2024-09-10T19:07:09.809659+0000","last_peered":"2024-09-10T19:07:09.809659+0000","last_clean":"2024-09-10T19:07:09.809659+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:07:09.809659+0000","last_undegraded":"2024-09-10T19:07:09.809659+0000","last_fullsized":"2024-09-10T19:07:09.809659+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.306762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.306762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.871043+0000","scrub_duration":0.0038348879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":797,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.816081+0000","last_change":"2024-09-10T19:05:12.517087+0000","last_active":"2024-09-10T19:07:09.816081+0000","last_peered":"2024-09-10T19:07:09.816081+0000","last_clean":"2024-09-10T19:07:09.816081+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:07:09.816081+0000","last_undegraded":"2024-09-10T19:07:09.816081+0000","last_fullsized":"2024-09-10T19:07:09.816081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:05:12.516921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:12.516921+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050411789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":784,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809601+0000","last_change":"2024-09-10T19:06:58.401445+0000","last_active":"2024-09-10T19:07:09.809601+0000","last_peered":"2024-09-10T19:07:09.809601+0000","last_clean":"2024-09-10T19:07:09.809601+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:07:09.809601+0000","last_undegraded":"2024-09-10T19:07:09.809601+0000","last_fullsized":"2024-09-10T19:07:09.809601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:58.401361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.401361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.596261+0000","scrub_duration":0.002332808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":986,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362791+0000","last_change":"2024-09-10T19:06:28.763314+0000","last_active":"2024-09-10T19:07:09.362791+0000","last_peered":"2024-09-10T19:07:09.362791+0000","last_clean":"2024-09-10T19:07:09.362791+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:07:09.362791+0000","last_undegraded":"2024-09-10T19:07:09.362791+0000","last_fullsized":"2024-09-10T19:07:09.362791+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:06:28.763149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.763149+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.664851+0000","scrub_duration":0.0099338919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":260,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.368854+0000","last_change":"2024-09-10T19:06:54.081950+0000","last_active":"2024-09-10T19:07:09.368854+0000","last_peered":"2024-09-10T19:07:09.368854+0000","last_clean":"2024-09-10T19:07:09.368854+0000","last_became_active":"2024-09-10T19:02:52.256751+0000","last_became_peered":"2024-09-10T19:02:52.256751+0000","last_unstale":"2024-09-10T19:07:09.368854+0000","last_undegraded":"2024-09-10T19:07:09.368854+0000","last_fullsized":"2024-09-10T19:07:09.368854+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.081811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.081811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.018335+0000","scrub_duration":0.0036574749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":679,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.814570+0000","last_change":"2024-09-10T19:04:59.527415+0000","last_active":"2024-09-10T19:07:09.814570+0000","last_peered":"2024-09-10T19:07:09.814570+0000","last_clean":"2024-09-10T19:07:09.814570+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:07:09.814570+0000","last_undegraded":"2024-09-10T19:07:09.814570+0000","last_fullsized":"2024-09-10T19:07:09.814570+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.527252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.527252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033075349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":807,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811894+0000","last_change":"2024-09-10T19:05:26.703616+0000","last_active":"2024-09-10T19:07:09.811894+0000","last_peered":"2024-09-10T19:07:09.811894+0000","last_clean":"2024-09-10T19:07:09.811894+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:07:09.811894+0000","last_undegraded":"2024-09-10T19:07:09.811894+0000","last_fullsized":"2024-09-10T19:07:09.811894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:05:26.703433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:26.703433+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050260740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768056+0000","last_change":"2024-09-10T19:05:31.506173+0000","last_active":"2024-09-10T19:07:09.768056+0000","last_peered":"2024-09-10T19:07:09.768056+0000","last_clean":"2024-09-10T19:07:09.768056+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:07:09.768056+0000","last_undegraded":"2024-09-10T19:07:09.768056+0000","last_fullsized":"2024-09-10T19:07:09.768056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:31.506021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:31.506021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030748609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":773,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.357460+0000","last_change":"2024-09-10T19:06:37.901263+0000","last_active":"2024-09-10T19:07:09.357460+0000","last_peered":"2024-09-10T19:07:09.357460+0000","last_clean":"2024-09-10T19:07:09.357460+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:07:09.357460+0000","last_undegraded":"2024-09-10T19:07:09.357460+0000","last_fullsized":"2024-09-10T19:07:09.357460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.901111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.901111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.610396+0000","scrub_duration":0.0036067199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":771,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815129+0000","last_change":"2024-09-10T19:05:02.444342+0000","last_active":"2024-09-10T19:07:09.815129+0000","last_peered":"2024-09-10T19:07:09.815129+0000","last_clean":"2024-09-10T19:07:09.815129+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:07:09.815129+0000","last_undegraded":"2024-09-10T19:07:09.815129+0000","last_fullsized":"2024-09-10T19:07:09.815129+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.444182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.444182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002871462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":711,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.607543+0000","last_change":"2024-09-10T19:06:54.653102+0000","last_active":"2024-09-10T19:07:09.607543+0000","last_peered":"2024-09-10T19:07:09.607543+0000","last_clean":"2024-09-10T19:07:09.607543+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:07:09.607543+0000","last_undegraded":"2024-09-10T19:07:09.607543+0000","last_fullsized":"2024-09-10T19:07:09.607543+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.652944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.751266+0000","scrub_duration":0.0032820850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":684,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811575+0000","last_change":"2024-09-10T19:05:06.709482+0000","last_active":"2024-09-10T19:07:09.811575+0000","last_peered":"2024-09-10T19:07:09.811575+0000","last_clean":"2024-09-10T19:07:09.811575+0000","last_became_active":"2024-09-10T19:02:52.257159+0000","last_became_peered":"2024-09-10T19:02:52.257159+0000","last_unstale":"2024-09-10T19:07:09.811575+0000","last_undegraded":"2024-09-10T19:07:09.811575+0000","last_fullsized":"2024-09-10T19:07:09.811575+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:06.709404+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:05:06.709404+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030078150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.810782+0000","last_change":"2024-09-10T19:06:17.966425+0000","last_active":"2024-09-10T19:07:09.810782+0000","last_peered":"2024-09-10T19:07:09.810782+0000","last_clean":"2024-09-10T19:07:09.810782+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:07:09.810782+0000","last_undegraded":"2024-09-10T19:07:09.810782+0000","last_fullsized":"2024-09-10T19:07:09.810782+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.966265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.966265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:31.827356+0000","scrub_duration":0.006067228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":792,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.611819+0000","last_change":"2024-09-10T19:06:28.866625+0000","last_active":"2024-09-10T19:07:09.611819+0000","last_peered":"2024-09-10T19:07:09.611819+0000","last_clean":"2024-09-10T19:07:09.611819+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:07:09.611819+0000","last_undegraded":"2024-09-10T19:07:09.611819+0000","last_fullsized":"2024-09-10T19:07:09.611819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:28.866466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.866466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.241863+0000","scrub_duration":0.0080387719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":777,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.361976+0000","last_change":"2024-09-10T19:05:30.921980+0000","last_active":"2024-09-10T19:07:09.361976+0000","last_peered":"2024-09-10T19:07:09.361976+0000","last_clean":"2024-09-10T19:07:09.361976+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:07:09.361976+0000","last_undegraded":"2024-09-10T19:07:09.361976+0000","last_fullsized":"2024-09-10T19:07:09.361976+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:30.921823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:51.975710+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.921823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003529318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":255,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.812399+0000","last_change":"2024-09-10T19:06:32.121855+0000","last_active":"2024-09-10T19:07:09.812399+0000","last_peered":"2024-09-10T19:07:09.812399+0000","last_clean":"2024-09-10T19:07:09.812399+0000","last_became_active":"2024-09-10T19:02:52.258536+0000","last_became_peered":"2024-09-10T19:02:52.258536+0000","last_unstale":"2024-09-10T19:07:09.812399+0000","last_undegraded":"2024-09-10T19:07:09.812399+0000","last_fullsized":"2024-09-10T19:07:09.812399+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.121693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.121693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.538774+0000","scrub_duration":0.0033793880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":793,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817811+0000","last_change":"2024-09-10T19:06:41.386525+0000","last_active":"2024-09-10T19:07:09.817811+0000","last_peered":"2024-09-10T19:07:09.817811+0000","last_clean":"2024-09-10T19:07:09.817811+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:07:09.817811+0000","last_undegraded":"2024-09-10T19:07:09.817811+0000","last_fullsized":"2024-09-10T19:07:09.817811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_deep_scrub":"410'15","last_deep_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_clean_scrub_stamp":"2024-09-10T19:06:41.386460+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.747797+0000","scrub_duration":0.0022228119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":776,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363728+0000","last_change":"2024-09-10T19:06:20.906700+0000","last_active":"2024-09-10T19:07:09.363728+0000","last_peered":"2024-09-10T19:07:09.363728+0000","last_clean":"2024-09-10T19:07:09.363728+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:07:09.363728+0000","last_undegraded":"2024-09-10T19:07:09.363728+0000","last_fullsized":"2024-09-10T19:07:09.363728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:20.906544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.906544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:37.590695+0000","scrub_duration":0.0034009769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":769,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.360135+0000","last_change":"2024-09-10T19:05:28.918947+0000","last_active":"2024-09-10T19:07:09.360135+0000","last_peered":"2024-09-10T19:07:09.360135+0000","last_clean":"2024-09-10T19:07:09.360135+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:07:09.360135+0000","last_undegraded":"2024-09-10T19:07:09.360135+0000","last_fullsized":"2024-09-10T19:07:09.360135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.918873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030432179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809487+0000","last_change":"2024-09-10T19:05:02.785214+0000","last_active":"2024-09-10T19:07:09.809487+0000","last_peered":"2024-09-10T19:07:09.809487+0000","last_clean":"2024-09-10T19:07:09.809487+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:07:09.809487+0000","last_undegraded":"2024-09-10T19:07:09.809487+0000","last_fullsized":"2024-09-10T19:07:09.809487+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.785055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.785055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033757409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":701,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809704+0000","last_change":"2024-09-10T19:06:50.191355+0000","last_active":"2024-09-10T19:07:09.809704+0000","last_peered":"2024-09-10T19:07:09.809704+0000","last_clean":"2024-09-10T19:07:09.809704+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:07:09.809704+0000","last_undegraded":"2024-09-10T19:07:09.809704+0000","last_fullsized":"2024-09-10T19:07:09.809704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:50.191205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:14.886297+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.191205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:17.708068+0000","scrub_duration":0.003564837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"676'19","reported_seq":803,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363605+0000","last_change":"2024-09-10T19:05:34.022493+0000","last_active":"2024-09-10T19:07:09.363605+0000","last_peered":"2024-09-10T19:07:09.363605+0000","last_clean":"2024-09-10T19:07:09.363605+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:07:09.363605+0000","last_undegraded":"2024-09-10T19:07:09.363605+0000","last_fullsized":"2024-09-10T19:07:09.363605+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'19","last_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_deep_scrub":"676'19","last_deep_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_clean_scrub_stamp":"2024-09-10T19:05:34.022377+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006303052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768761+0000","last_change":"2024-09-10T19:05:06.482047+0000","last_active":"2024-09-10T19:07:09.768761+0000","last_peered":"2024-09-10T19:07:09.768761+0000","last_clean":"2024-09-10T19:07:09.768761+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:07:09.768761+0000","last_undegraded":"2024-09-10T19:07:09.768761+0000","last_fullsized":"2024-09-10T19:07:09.768761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:06.481823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:06.481823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035879710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":766,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362406+0000","last_change":"2024-09-10T19:05:18.965121+0000","last_active":"2024-09-10T19:07:09.362406+0000","last_peered":"2024-09-10T19:07:09.362406+0000","last_clean":"2024-09-10T19:07:09.362406+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:07:09.362406+0000","last_undegraded":"2024-09-10T19:07:09.362406+0000","last_fullsized":"2024-09-10T19:07:09.362406+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:18.964844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:18.964844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040517590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":774,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363646+0000","last_change":"2024-09-10T19:06:46.126997+0000","last_active":"2024-09-10T19:07:09.363646+0000","last_peered":"2024-09-10T19:07:09.363646+0000","last_clean":"2024-09-10T19:07:09.363646+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:07:09.363646+0000","last_undegraded":"2024-09-10T19:07:09.363646+0000","last_fullsized":"2024-09-10T19:07:09.363646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.126832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.126832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.544637+0000","scrub_duration":0.003551504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":689,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.809263+0000","last_change":"2024-09-10T19:06:46.381092+0000","last_active":"2024-09-10T19:07:09.809263+0000","last_peered":"2024-09-10T19:07:09.809263+0000","last_clean":"2024-09-10T19:07:09.809263+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:07:09.809263+0000","last_undegraded":"2024-09-10T19:07:09.809263+0000","last_fullsized":"2024-09-10T19:07:09.809263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.380935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.181442+0000","scrub_duration":0.0032731539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":778,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811396+0000","last_change":"2024-09-10T19:06:22.919675+0000","last_active":"2024-09-10T19:07:09.811396+0000","last_peered":"2024-09-10T19:07:09.811396+0000","last_clean":"2024-09-10T19:07:09.811396+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:07:09.811396+0000","last_undegraded":"2024-09-10T19:07:09.811396+0000","last_fullsized":"2024-09-10T19:07:09.811396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:06:22.919527+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.919527+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.618775+0000","scrub_duration":0.002938792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":843,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.811401+0000","last_change":"2024-09-10T19:06:56.248053+0000","last_active":"2024-09-10T19:07:09.811401+0000","last_peered":"2024-09-10T19:07:09.811401+0000","last_clean":"2024-09-10T19:07:09.811401+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:07:09.811401+0000","last_undegraded":"2024-09-10T19:07:09.811401+0000","last_fullsized":"2024-09-10T19:07:09.811401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:56.247893+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:06:56.247893+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.638950+0000","scrub_duration":0.0050558469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":776,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.356909+0000","last_change":"2024-09-10T19:05:15.161330+0000","last_active":"2024-09-10T19:07:09.356909+0000","last_peered":"2024-09-10T19:07:09.356909+0000","last_clean":"2024-09-10T19:07:09.356909+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:07:09.356909+0000","last_undegraded":"2024-09-10T19:07:09.356909+0000","last_fullsized":"2024-09-10T19:07:09.356909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.161170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.161170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034575059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":770,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.817075+0000","last_change":"2024-09-10T19:05:21.349080+0000","last_active":"2024-09-10T19:07:09.817075+0000","last_peered":"2024-09-10T19:07:09.817075+0000","last_clean":"2024-09-10T19:07:09.817075+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:07:09.817075+0000","last_undegraded":"2024-09-10T19:07:09.817075+0000","last_fullsized":"2024-09-10T19:07:09.817075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:21.348970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:05:21.348970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002650716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768605+0000","last_change":"2024-09-10T19:05:30.533679+0000","last_active":"2024-09-10T19:07:09.768605+0000","last_peered":"2024-09-10T19:07:09.768605+0000","last_clean":"2024-09-10T19:07:09.768605+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:07:09.768605+0000","last_undegraded":"2024-09-10T19:07:09.768605+0000","last_fullsized":"2024-09-10T19:07:09.768605+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:30.533525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.533525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033150879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":779,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.820566+0000","last_change":"2024-09-10T19:06:19.011016+0000","last_active":"2024-09-10T19:07:09.820566+0000","last_peered":"2024-09-10T19:07:09.820566+0000","last_clean":"2024-09-10T19:07:09.820566+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:07:09.820566+0000","last_undegraded":"2024-09-10T19:07:09.820566+0000","last_fullsized":"2024-09-10T19:07:09.820566+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.010845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.010845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:35.834254+0000","scrub_duration":0.0035581089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":803,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.360403+0000","last_change":"2024-09-10T19:06:34.054784+0000","last_active":"2024-09-10T19:07:09.360403+0000","last_peered":"2024-09-10T19:07:09.360403+0000","last_clean":"2024-09-10T19:07:09.360403+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:07:09.360403+0000","last_undegraded":"2024-09-10T19:07:09.360403+0000","last_fullsized":"2024-09-10T19:07:09.360403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:06:34.054632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.054632+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.984618+0000","scrub_duration":0.0038029330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.363552+0000","last_change":"2024-09-10T19:06:52.165777+0000","last_active":"2024-09-10T19:07:09.363552+0000","last_peered":"2024-09-10T19:07:09.363552+0000","last_clean":"2024-09-10T19:07:09.363552+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:07:09.363552+0000","last_undegraded":"2024-09-10T19:07:09.363552+0000","last_fullsized":"2024-09-10T19:07:09.363552+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.165627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:18.581502+0000","scrub_duration":0.00328166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":1026,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.769004+0000","last_change":"2024-09-10T19:06:50.856618+0000","last_active":"2024-09-10T19:07:09.769004+0000","last_peered":"2024-09-10T19:07:09.769004+0000","last_clean":"2024-09-10T19:07:09.769004+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:07:09.769004+0000","last_undegraded":"2024-09-10T19:07:09.769004+0000","last_fullsized":"2024-09-10T19:07:09.769004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:06:50.856451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.856451+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.647430+0000","scrub_duration":0.0045245329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.769171+0000","last_change":"2024-09-10T19:05:26.559541+0000","last_active":"2024-09-10T19:07:09.769171+0000","last_peered":"2024-09-10T19:07:09.769171+0000","last_clean":"2024-09-10T19:07:09.769171+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:07:09.769171+0000","last_undegraded":"2024-09-10T19:07:09.769171+0000","last_fullsized":"2024-09-10T19:07:09.769171+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:26.559381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:05:26.559381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00334798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":790,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.612776+0000","last_change":"2024-09-10T19:06:36.751344+0000","last_active":"2024-09-10T19:07:09.612776+0000","last_peered":"2024-09-10T19:07:09.612776+0000","last_clean":"2024-09-10T19:07:09.612776+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:07:09.612776+0000","last_undegraded":"2024-09-10T19:07:09.612776+0000","last_fullsized":"2024-09-10T19:07:09.612776+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.751254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.751254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:41.762396+0000","scrub_duration":0.01052581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":788,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.767644+0000","last_change":"2024-09-10T19:06:30.665495+0000","last_active":"2024-09-10T19:07:09.767644+0000","last_peered":"2024-09-10T19:07:09.767644+0000","last_clean":"2024-09-10T19:07:09.767644+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:07:09.767644+0000","last_undegraded":"2024-09-10T19:07:09.767644+0000","last_fullsized":"2024-09-10T19:07:09.767644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:06:30.665240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.665240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:40.794854+0000","scrub_duration":0.0058393220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":785,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.815756+0000","last_change":"2024-09-10T19:05:30.243873+0000","last_active":"2024-09-10T19:07:09.815756+0000","last_peered":"2024-09-10T19:07:09.815756+0000","last_clean":"2024-09-10T19:07:09.815756+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:07:09.815756+0000","last_undegraded":"2024-09-10T19:07:09.815756+0000","last_fullsized":"2024-09-10T19:07:09.815756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:05:30.243807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.243807+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031694280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":807,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365473+0000","last_change":"2024-09-10T19:06:42.199376+0000","last_active":"2024-09-10T19:07:09.365473+0000","last_peered":"2024-09-10T19:07:09.365473+0000","last_clean":"2024-09-10T19:07:09.365473+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:07:09.365473+0000","last_undegraded":"2024-09-10T19:07:09.365473+0000","last_fullsized":"2024-09-10T19:07:09.365473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:42.199204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.968019+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.199204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:07.125044+0000","scrub_duration":0.0034375769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":693,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.362016+0000","last_change":"2024-09-10T19:05:31.067457+0000","last_active":"2024-09-10T19:07:09.362016+0000","last_peered":"2024-09-10T19:07:09.362016+0000","last_clean":"2024-09-10T19:07:09.362016+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:07:09.362016+0000","last_undegraded":"2024-09-10T19:07:09.362016+0000","last_fullsized":"2024-09-10T19:07:09.362016+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:31.067311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:31.067311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030309170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":774,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.365935+0000","last_change":"2024-09-10T19:06:19.090138+0000","last_active":"2024-09-10T19:07:09.365935+0000","last_peered":"2024-09-10T19:07:09.365935+0000","last_clean":"2024-09-10T19:07:09.365935+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:07:09.365935+0000","last_undegraded":"2024-09-10T19:07:09.365935+0000","last_fullsized":"2024-09-10T19:07:09.365935+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.089886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.089886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:42.552467+0000","scrub_duration":0.0040961510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":780,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.364300+0000","last_change":"2024-09-10T19:06:31.070915+0000","last_active":"2024-09-10T19:07:09.364300+0000","last_peered":"2024-09-10T19:07:09.364300+0000","last_clean":"2024-09-10T19:07:09.364300+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:07:09.364300+0000","last_undegraded":"2024-09-10T19:07:09.364300+0000","last_fullsized":"2024-09-10T19:07:09.364300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.070745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.070745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.012535+0000","scrub_duration":0.0036538719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":819,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.611297+0000","last_change":"2024-09-10T19:06:23.971596+0000","last_active":"2024-09-10T19:07:09.611297+0000","last_peered":"2024-09-10T19:07:09.611297+0000","last_clean":"2024-09-10T19:07:09.611297+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:07:09.611297+0000","last_undegraded":"2024-09-10T19:07:09.611297+0000","last_fullsized":"2024-09-10T19:07:09.611297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:23.971482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.971482+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:33.794031+0000","scrub_duration":0.0045234610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":766,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.366227+0000","last_change":"2024-09-10T19:05:14.979053+0000","last_active":"2024-09-10T19:07:09.366227+0000","last_peered":"2024-09-10T19:07:09.366227+0000","last_clean":"2024-09-10T19:07:09.366227+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:07:09.366227+0000","last_undegraded":"2024-09-10T19:07:09.366227+0000","last_fullsized":"2024-09-10T19:07:09.366227+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:14.978923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:05:14.978923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003307857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":696,"reported_epoch":770,"state":"active+clean","last_fresh":"2024-09-10T19:07:09.768266+0000","last_change":"2024-09-10T19:05:33.492062+0000","last_active":"2024-09-10T19:07:09.768266+0000","last_peered":"2024-09-10T19:07:09.768266+0000","last_clean":"2024-09-10T19:07:09.768266+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:07:09.768266+0000","last_undegraded":"2024-09-10T19:07:09.768266+0000","last_fullsized":"2024-09-10T19:07:09.768266+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:33.491961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:51.945980+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.491961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025640609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":255,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":241,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1046,"num_read_kb":1041,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1128,"num_read_kb":885,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":776,"ondisk_log_size":776,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972578,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299824,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445328,"statfs":{"total":95995035648,"available":95688015872,"internally_reserved":0,"allocated":2519040,"data_stored":1083064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.93600000000000005,"15min":0.90800000000000003},"min":{"1min":0.48399999999999999,"5min":0.432,"15min":0.311},"max":{"1min":2.0649999999999999,"5min":2.839,"15min":2.839},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.97999999999999998,"15min":0.95099999999999996},"min":{"1min":0.57099999999999995,"5min":0.497,"15min":0.34699999999999998},"max":{"1min":2.9550000000000001,"5min":2.9550000000000001,"15min":4.7000000000000002},"last":0.73099999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95599999999999996,"15min":0.90500000000000003},"min":{"1min":0.51000000000000001,"5min":0.504,"15min":0.41199999999999998},"max":{"1min":3.097,"5min":4.9299999999999997,"15min":5.6539999999999999},"last":1.244},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":0.96199999999999997},"min":{"1min":0.622,"5min":0.57899999999999996,"15min":0.31900000000000001},"max":{"1min":1.3420000000000001,"5min":2.012,"15min":2.012},"last":0.84999999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.302,"15min":1.1040000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.41499999999999998},"max":{"1min":1.385,"5min":31.166,"15min":31.166},"last":1.427},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2569999999999999,"15min":1.087},"min":{"1min":0.71499999999999997,"5min":0.55900000000000005,"15min":0.443},"max":{"1min":1.7529999999999999,"5min":27.709,"15min":27.709},"last":0.97299999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.48299999999999998},"max":{"1min":1.2729999999999999,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":1.335},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.70199999999999996,"5min":0.626,"15min":0.46600000000000003},"max":{"1min":1.3049999999999999,"5min":4.2060000000000004,"15min":5.6050000000000004},"last":1.3759999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0169999999999999,"15min":1.016},"min":{"1min":0.57599999999999996,"5min":0.51800000000000002,"15min":0.34799999999999998},"max":{"1min":1.3500000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.284},{"interface":"front","average":{"1min":1.107,"5min":1.0700000000000001,"15min":1.081},"min":{"1min":0.73299999999999998,"5min":0.59499999999999997,"15min":0.48099999999999998},"max":{"1min":1.421,"5min":2.25,"15min":4.2400000000000002},"last":1.2130000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.97299999999999998,"15min":0.98499999999999999},"min":{"1min":0.81100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.228,"5min":1.5980000000000001,"15min":5.4710000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.1459999999999999,"15min":1.1499999999999999},"min":{"1min":0.58299999999999996,"5min":0.57699999999999996,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":4.8710000000000004,"15min":4.8710000000000004},"last":1.5720000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.48699999999999999},"max":{"1min":1.444,"5min":1.766,"15min":1.766},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.153,"15min":1.1890000000000001},"min":{"1min":0.83899999999999997,"5min":0.61399999999999999,"15min":0.58099999999999996},"max":{"1min":1.353,"5min":2.4340000000000002,"15min":4.1310000000000002},"last":1.4730000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299628,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303592,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441560,"statfs":{"total":95995035648,"available":95684157440,"internally_reserved":0,"allocated":2125824,"data_stored":716228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.53000000000000003,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.575,"5min":1.696,"15min":2.0550000000000002},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.96499999999999997,"15min":0.96999999999999997},"min":{"1min":0.65300000000000002,"5min":0.438,"15min":0.36399999999999999},"max":{"1min":1.262,"5min":1.6859999999999999,"15min":6.8959999999999999},"last":1.0469999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0820000000000001,"15min":1.091},"min":{"1min":0.64800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":18.177,"5min":18.177,"15min":18.494},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.696,"5min":1.0469999999999999,"15min":0.95099999999999996},"min":{"1min":0.69899999999999995,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":17.297000000000001,"5min":17.297000000000001,"15min":17.297000000000001},"last":0.94399999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.94299999999999995,"15min":0.97399999999999998},"min":{"1min":0.59199999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3480000000000001,"5min":1.5589999999999999,"15min":9.4800000000000004},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.90900000000000003,"15min":0.93799999999999994},"min":{"1min":0.62,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":6.0970000000000004},"last":0.88300000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.99099999999999999,"15min":1.012},"min":{"1min":0.71099999999999997,"5min":0.46700000000000003,"15min":0.40400000000000003},"max":{"1min":1.538,"5min":1.8700000000000001,"15min":2.278},"last":0.626},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.95099999999999996,"15min":0.98299999999999998},"min":{"1min":0.71999999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2230000000000001,"5min":1.4730000000000001,"15min":3.4279999999999999},"last":1.0780000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.96199999999999997,"15min":0.98299999999999998},"min":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.3049999999999999,"5min":1.512,"15min":6.008},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.58699999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.492,"5min":2.0619999999999998,"15min":2.3079999999999998},"last":0.66700000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.036,"15min":1.0669999999999999},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.46300000000000002},"max":{"1min":1.415,"5min":2.02,"15min":5.9560000000000004},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.085,"15min":1.135},"min":{"1min":0.56599999999999995,"5min":0.47599999999999998,"15min":0.46500000000000002},"max":{"1min":1.611,"5min":2.145,"15min":3.5169999999999999},"last":0.92300000000000004}]},{"osd":7,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.032,"15min":1.0649999999999999},"min":{"1min":0.56100000000000005,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3520000000000001,"5min":1.8320000000000001,"15min":2.069},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.161,"15min":1.2030000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6399999999999999,"5min":1.7789999999999999,"15min":6.0380000000000003},"last":1.117}]}]},{"osd":5,"up_from":43,"seq":184683593974,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302004,"kb_used_data":4648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443148,"statfs":{"total":95995035648,"available":95685783552,"internally_reserved":0,"allocated":4759552,"data_stored":3338817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0089999999999999,"15min":1.0129999999999999},"min":{"1min":0.72799999999999998,"5min":0.46500000000000002,"15min":0.36399999999999999},"max":{"1min":1.508,"5min":5.4640000000000004,"15min":8.3949999999999996},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.62,"5min":0.53100000000000003,"15min":0.39700000000000002},"max":{"1min":1.216,"5min":5.4790000000000001,"15min":7.3070000000000004},"last":1.2170000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.012,"15min":1.208},"min":{"1min":0.68200000000000005,"5min":0.55600000000000005,"15min":0.41199999999999998},"max":{"1min":1.423,"5min":6.282,"15min":31.548999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.998,"15min":1.1140000000000001},"min":{"1min":0.56799999999999995,"5min":0.54800000000000004,"15min":0.48899999999999999},"max":{"1min":1.2869999999999999,"5min":6.0170000000000003,"15min":31.847000000000001},"last":1.1499999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.048,"15min":1.292},"min":{"1min":0.77800000000000002,"5min":0.66200000000000003,"15min":0.39900000000000002},"max":{"1min":1.3859999999999999,"5min":7.9340000000000002,"15min":61.347000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.05,"15min":1.0589999999999999},"min":{"1min":0.65900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.4419999999999999,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":1.115}]},{"osd":3,"last update":"Tue Sep 10 19:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1,"15min":1.0389999999999999},"min":{"1min":0.75,"5min":0.60199999999999998,"15min":0.51700000000000002},"max":{"1min":1.3799999999999999,"5min":1.468,"15min":7.5979999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1020000000000001,"15min":1.306},"min":{"1min":0.73299999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.482,"5min":5.3810000000000002,"15min":53.173000000000002},"last":1.2470000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.96199999999999997,"15min":1.024},"min":{"1min":0.59499999999999997,"5min":0.52500000000000002,"15min":0.36499999999999999},"max":{"1min":1.272,"5min":5.5170000000000003,"15min":13.005000000000001},"last":1.034},{"interface":"front","average":{"1min":0.999,"5min":0.97299999999999998,"15min":1.0580000000000001},"min":{"1min":0.65600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3049999999999999,"5min":1.546,"15min":7.6299999999999999},"last":0.95599999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.149,"15min":1.198},"min":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.49199999999999999},"max":{"1min":1.359,"5min":4.2880000000000003,"15min":7.7199999999999998},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.1859999999999999,"15min":1.2410000000000001},"min":{"1min":0.80400000000000005,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.518,"5min":5.4240000000000004,"15min":9.6760000000000002},"last":1.1799999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:06:11 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.135,"15min":1.2390000000000001},"min":{"1min":0.75600000000000001,"5min":0.46700000000000003,"15min":0.43099999999999999},"max":{"1min":1.591,"5min":5.5510000000000002,"15min":15.613},"last":1.27},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.1539999999999999,"15min":1.2649999999999999},"min":{"1min":0.89700000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.6339999999999999,"5min":1.718,"15min":16.965},"last":1.079}]}]},{"osd":4,"up_from":35,"seq":150323855618,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303656,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441496,"statfs":{"total":95995035648,"available":95684091904,"internally_reserved":0,"allocated":2187264,"data_stored":746224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.88900000000000001,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46899999999999997,"15min":0.371},"max":{"1min":1.357,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.995},{"interface":"front","average":{"1min":0.998,"5min":1.0029999999999999,"15min":0.94699999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.7609999999999999,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.046}]},{"osd":1,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.90800000000000003,"15min":0.98199999999999998},"min":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.42399999999999999},"max":{"1min":1.677,"5min":1.677,"15min":27.716000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98099999999999998,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.38100000000000001},"max":{"1min":1.4950000000000001,"5min":1.865,"15min":27.93},"last":1.0960000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.2230000000000001,"15min":1.0429999999999999},"min":{"1min":0.68500000000000005,"5min":0.50800000000000001,"15min":0.46600000000000003},"max":{"1min":1.3879999999999999,"5min":18.853000000000002,"15min":18.853000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.2849999999999999,"15min":1.0669999999999999},"min":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.497},"max":{"1min":1.587,"5min":25.652999999999999,"15min":25.652999999999999},"last":1.2150000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0329999999999999,"15min":1},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.46300000000000002},"max":{"1min":1.6879999999999999,"5min":1.923,"15min":4.2670000000000003},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.03,"15min":0.995},"min":{"1min":0.65700000000000003,"5min":0.622,"15min":0.442},"max":{"1min":1.5649999999999999,"5min":1.873,"15min":2.0739999999999998},"last":0.96199999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:06:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.97899999999999998,"15min":0.97199999999999998},"min":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.38700000000000001},"max":{"1min":1.4219999999999999,"5min":1.6060000000000001,"15min":1.8009999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.099,"15min":1.0700000000000001},"min":{"1min":0.72699999999999998,"5min":0.63600000000000001,"15min":0.52600000000000002},"max":{"1min":2.0259999999999998,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.302}]},{"osd":6,"last update":"Tue Sep 10 19:06:21 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.0589999999999999,"15min":1.0489999999999999},"min":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.867,"5min":1.867,"15min":2.1699999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.1830000000000001,"15min":1.171},"min":{"1min":0.873,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.802,"5min":1.9890000000000001,"15min":2.9590000000000001},"last":1.145}]},{"osd":7,"last update":"Tue Sep 10 19:06:12 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.2170000000000001,"15min":1.1719999999999999},"min":{"1min":0.77200000000000002,"5min":0.56999999999999995,"15min":0.46899999999999997},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.377},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.214,"15min":1.1819999999999999},"min":{"1min":0.82299999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":3.7850000000000001,"5min":3.7850000000000001,"15min":3.7850000000000001},"last":1.1910000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051853,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300032,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445120,"statfs":{"total":95995035648,"available":95687802880,"internally_reserved":0,"allocated":2736128,"data_stored":1316245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68899999999999995,"15min":0.85099999999999998},"min":{"1min":0.42399999999999999,"5min":0.313,"15min":0.222},"max":{"1min":0.92700000000000005,"5min":1.6539999999999999,"15min":26.594000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.73899999999999999,"15min":0.89900000000000002},"min":{"1min":0.48799999999999999,"5min":0.40400000000000003,"15min":0.252},"max":{"1min":1.139,"5min":1.4790000000000001,"15min":24.204999999999998},"last":0.64700000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70599999999999996,"15min":0.92200000000000004},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.26500000000000001},"max":{"1min":1.101,"5min":1.3240000000000001,"15min":53.661999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77900000000000003,"15min":0.88300000000000001},"min":{"1min":0.60199999999999998,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":1.071,"5min":1.7170000000000001,"15min":15.189},"last":1.028}]},{"osd":2,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.88400000000000001,"15min":1.0209999999999999},"min":{"1min":0.51400000000000001,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":2.073,"5min":2.073,"15min":26.558},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78300000000000003,"15min":0.95199999999999996},"min":{"1min":0.56599999999999995,"5min":0.497,"15min":0.26900000000000002},"max":{"1min":1.034,"5min":1.5489999999999999,"15min":25.536999999999999},"last":0.81100000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:06:34 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.095,"15min":1.1679999999999999},"min":{"1min":0.84599999999999997,"5min":0.629,"15min":0.46999999999999997},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":10.359999999999999},"last":1.119},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.202,"15min":1.27},"min":{"1min":0.81799999999999995,"5min":0.64400000000000002,"15min":0.53500000000000003},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":10.391},"last":1.2390000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.111,"15min":1.2230000000000001},"min":{"1min":0.80700000000000005,"5min":0.62,"15min":0.501},"max":{"1min":1.768,"5min":1.768,"15min":17.535},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0289999999999999,"15min":1.0649999999999999},"min":{"1min":0.77400000000000002,"5min":0.53500000000000003,"15min":0.45900000000000002},"max":{"1min":1.4059999999999999,"5min":2.0529999999999999,"15min":8.7880000000000003},"last":1.091}]},{"osd":6,"last update":"Tue Sep 10 19:06:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.1850000000000001,"15min":1.2010000000000001},"min":{"1min":0.86199999999999999,"5min":0.59499999999999997,"15min":0.55600000000000005},"max":{"1min":1.954,"5min":1.954,"15min":10.51},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0940000000000001,"15min":1.105},"min":{"1min":0.80400000000000005,"5min":0.64200000000000002,"15min":0.48499999999999999},"max":{"1min":1.8149999999999999,"5min":1.9259999999999999,"15min":10.773999999999999},"last":0.89700000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:07:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.1419999999999999,"15min":1.1799999999999999},"min":{"1min":0.84699999999999998,"5min":0.61499999999999999,"15min":0.45600000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":18.312999999999999},"last":1.149},{"interface":"front","average":{"1min":1.391,"5min":1.278,"15min":1.327},"min":{"1min":0.89000000000000001,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":10.611000000000001},"last":1.3620000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378902,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308072,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437080,"statfs":{"total":95995035648,"available":95679569920,"internally_reserved":0,"allocated":2445312,"data_stored":1017429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.153,"15min":0.89200000000000002},"min":{"1min":0.41599999999999998,"5min":0.28499999999999998,"15min":0.219},"max":{"1min":1.0880000000000001,"5min":43.927999999999997,"15min":43.927999999999997},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.875,"5min":1.3819999999999999,"15min":0.98499999999999999},"min":{"1min":0.48199999999999998,"5min":0.33000000000000002,"15min":0.23599999999999999},"max":{"1min":1.3029999999999999,"5min":49.475000000000001,"15min":49.475000000000001},"last":0.68000000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78400000000000003,"15min":0.751},"min":{"1min":0.51800000000000002,"5min":0.36499999999999999,"15min":0.26600000000000001},"max":{"1min":0.996,"5min":3.524,"15min":5.516},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.56799999999999995,"5min":0.40300000000000002,"15min":0.28299999999999997},"max":{"1min":1.135,"5min":1.77,"15min":9.0649999999999995},"last":1.101}]},{"osd":3,"last update":"Tue Sep 10 19:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":0.97199999999999998,"15min":0.90500000000000003},"min":{"1min":0.67500000000000004,"5min":0.318,"15min":0.318},"max":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.095,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":8.3710000000000004,"5min":9.8219999999999992,"15min":15.736000000000001},"last":4.8289999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.4419999999999999,"15min":1.23},"min":{"1min":0.59699999999999998,"5min":0.44900000000000001,"15min":0.39300000000000002},"max":{"1min":1.619,"5min":38.079999999999998,"15min":38.079999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.4470000000000001,"15min":1.276},"min":{"1min":0.76600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.627,"5min":37.991,"15min":37.991},"last":1.0529999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:06:18 2024","interfaces":[{"interface":"back","average":{"1min":1.133,"5min":1.119,"15min":1.1000000000000001},"min":{"1min":0.74199999999999999,"5min":0.49099999999999999,"15min":0.38500000000000001},"max":{"1min":1.444,"5min":1.7749999999999999,"15min":5.1959999999999997},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.179,"15min":1.1659999999999999},"min":{"1min":0.66600000000000004,"5min":0.59999999999999998,"15min":0.45400000000000001},"max":{"1min":1.591,"5min":6.0620000000000003,"15min":21.654},"last":0.83099999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:06:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.577,"15min":1.3009999999999999},"min":{"1min":0.85499999999999998,"5min":0.60499999999999998,"15min":0.503},"max":{"1min":1.5740000000000001,"5min":43.655999999999999,"15min":43.655999999999999},"last":1.008},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.581,"15min":1.3049999999999999},"min":{"1min":0.69599999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.5489999999999999,"5min":45.255000000000003,"15min":45.255000000000003},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:06:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.2390000000000001,"15min":1.1930000000000001},"min":{"1min":0.745,"5min":0.67800000000000005,"15min":0.48399999999999999},"max":{"1min":1.6299999999999999,"5min":6.0179999999999998,"15min":15.025},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.2250000000000001,"15min":1.2390000000000001},"min":{"1min":0.81999999999999995,"5min":0.63400000000000001,"15min":0.53100000000000003},"max":{"1min":1.8200000000000001,"5min":6.048,"15min":14.816000000000001},"last":0.78400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305576,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439576,"statfs":{"total":95995035648,"available":95682125824,"internally_reserved":0,"allocated":4157440,"data_stored":2736435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.70899999999999996,"15min":0.83699999999999997},"min":{"1min":0.46600000000000003,"5min":0.32900000000000001,"15min":0.23499999999999999},"max":{"1min":3.96,"5min":3.96,"15min":32.677999999999997},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.748,"15min":0.89600000000000002},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.253},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":32.398000000000003},"last":0.73099999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.72899999999999998,"15min":0.89600000000000002},"min":{"1min":0.432,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":26.097000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.996,"5min":0.85899999999999999,"15min":1.0149999999999999},"min":{"1min":0.45400000000000001,"5min":0.437,"15min":0.26000000000000001},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":32.363},"last":1.0860000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.5649999999999999,"15min":1.252},"min":{"1min":0.58499999999999996,"5min":0.309,"15min":0.28899999999999998},"max":{"1min":2.903,"5min":53.344999999999999,"15min":53.344999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.4319999999999999,"15min":1.2769999999999999},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.26600000000000001},"max":{"1min":4.085,"5min":53.645000000000003,"15min":60.999000000000002},"last":1.048}]},{"osd":4,"last update":"Tue Sep 10 19:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0740000000000001,"15min":1.1419999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.40899999999999997},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":12.808999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.0049999999999999,"15min":1.2649999999999999},"min":{"1min":0.70099999999999996,"5min":0.57699999999999996,"15min":0.378},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":60.529000000000003},"last":0.88700000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":0.99299999999999999,"15min":1.1359999999999999},"min":{"1min":0.70399999999999996,"5min":0.48899999999999999,"15min":0.40600000000000003},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":26.050999999999998},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.145,"15min":1.222},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.6850000000000001,"5min":2.6789999999999998,"15min":12.84},"last":1.1519999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1619999999999999,"15min":1.3129999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.53800000000000003},"max":{"1min":1.774,"5min":1.98,"15min":25.608000000000001},"last":1.242},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.0629999999999999,"15min":1.2490000000000001},"min":{"1min":0.81399999999999995,"5min":0.57699999999999996,"15min":0.46400000000000002},"max":{"1min":1.706,"5min":1.748,"15min":33.643000000000001},"last":1.127}]},{"osd":7,"last update":"Tue Sep 10 19:06:15 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.1990000000000001,"15min":1.3320000000000001},"min":{"1min":0.79700000000000004,"5min":0.76600000000000001,"15min":0.48499999999999999},"max":{"1min":1.7470000000000001,"5min":1.8859999999999999,"15min":25.565999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.0760000000000001,"15min":1.3009999999999999},"min":{"1min":0.77700000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.736,"5min":1.8280000000000001,"15min":33.673999999999999},"last":1.175}]}]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301444,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443708,"statfs":{"total":95995035648,"available":95686356992,"internally_reserved":0,"allocated":4186112,"data_stored":2764691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72599999999999998,"15min":1.032},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26300000000000001},"max":{"1min":0.89400000000000002,"5min":2.133,"15min":80.710999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80700000000000005,"15min":1.1160000000000001},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.419,"15min":86.009},"last":0.76000000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77400000000000002,"15min":1.0569999999999999},"min":{"1min":0.55000000000000004,"5min":0.36799999999999999,"15min":0.216},"max":{"1min":1.0549999999999999,"5min":2.3530000000000002,"15min":36.682000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.876,"5min":0.81399999999999995,"15min":0.95499999999999996},"min":{"1min":0.63700000000000001,"5min":0.41299999999999998,"15min":0.23599999999999999},"max":{"1min":1.294,"5min":1.651,"15min":24.614000000000001},"last":0.85299999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.81499999999999995,"15min":0.93300000000000005},"min":{"1min":0.61699999999999999,"5min":0.46300000000000002,"15min":0.26400000000000001},"max":{"1min":1.518,"5min":1.7849999999999999,"15min":21.620999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.875,"15min":1.069},"min":{"1min":0.68000000000000005,"5min":0.42499999999999999,"15min":0.30299999999999999},"max":{"1min":1.198,"5min":2.4569999999999999,"15min":45.765000000000001},"last":0.81499999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:06:34 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0840000000000001,"15min":1.103},"min":{"1min":0.81999999999999995,"5min":0.53900000000000003,"15min":0.5},"max":{"1min":1.3799999999999999,"5min":2.4239999999999999,"15min":6.3630000000000004},"last":0.997},{"interface":"front","average":{"1min":1.165,"5min":1.1639999999999999,"15min":1.1439999999999999},"min":{"1min":0.93200000000000005,"5min":0.626,"15min":0.51100000000000001},"max":{"1min":1.498,"5min":1.8979999999999999,"15min":4.2389999999999999},"last":1.1659999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:06:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1279999999999999,"15min":1.157},"min":{"1min":0.81100000000000005,"5min":0.66200000000000003,"15min":0.442},"max":{"1min":1.3939999999999999,"5min":2.5880000000000001,"15min":10.048},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1930000000000001,"15min":1.1379999999999999},"min":{"1min":0.90800000000000003,"5min":0.62,"15min":0.42999999999999999},"max":{"1min":1.365,"5min":2.0779999999999998,"15min":3.8959999999999999},"last":1.3999999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:06:17 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.23,"15min":1.1799999999999999},"min":{"1min":0.88900000000000001,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4330000000000001,"5min":1.899,"15min":3.9550000000000001},"last":1.472},{"interface":"front","average":{"1min":1.169,"5min":1.1910000000000001,"15min":1.224},"min":{"1min":0.83999999999999997,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.4770000000000001,"5min":2.5339999999999998,"15min":11.894},"last":1.4279999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2090000000000001,"15min":1.194},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.48499999999999999},"max":{"1min":1.5580000000000001,"5min":2.343,"15min":6.2949999999999999},"last":1.054},{"interface":"front","average":{"1min":1.254,"5min":1.2769999999999999,"15min":1.284},"min":{"1min":0.68100000000000005,"5min":0.56000000000000005,"15min":0.54900000000000004},"max":{"1min":1.5509999999999999,"5min":2.0009999999999999,"15min":8.2840000000000007},"last":1.3120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2043904,"data_stored":2043904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":2015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":2027520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":255,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:07:18.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:07:18.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:18.155 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:18.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:18.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:18.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:18.436 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:18.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:18.572 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:18.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:18.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[32381]: audit 2024-09-10T19:07:17.411124+0000 mon.a (mon.0) 3545 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]': finished 2024-09-10T19:07:18.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[32381]: cluster 2024-09-10T19:07:17.411227+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[32381]: cluster 2024-09-10T19:07:18.343545+0000 mgr.y (mgr.24526) 706 : cluster [DBG] pgmap v1202: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[32381]: audit 2024-09-10T19:07:18.385455+0000 mgr.y (mgr.24526) 707 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[18629]: audit 2024-09-10T19:07:17.411124+0000 mon.a (mon.0) 3545 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]': finished 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[18629]: cluster 2024-09-10T19:07:17.411227+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[18629]: cluster 2024-09-10T19:07:18.343545+0000 mgr.y (mgr.24526) 706 : cluster [DBG] pgmap v1202: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:18.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:18 smithi083 bash[18629]: audit 2024-09-10T19:07:18.385455+0000 mgr.y (mgr.24526) 707 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:18.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:18.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:18.843 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:18.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:18 smithi165 bash[22591]: audit 2024-09-10T19:07:17.411124+0000 mon.a (mon.0) 3545 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-138"}]': finished 2024-09-10T19:07:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:18 smithi165 bash[22591]: cluster 2024-09-10T19:07:17.411227+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-09-10T19:07:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:18 smithi165 bash[22591]: cluster 2024-09-10T19:07:18.343545+0000 mgr.y (mgr.24526) 706 : cluster [DBG] pgmap v1202: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:18 smithi165 bash[22591]: audit 2024-09-10T19:07:18.385455+0000 mgr.y (mgr.24526) 707 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:18.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:18.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:18.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:19.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:19.109 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:19.213 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:19.240 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:19.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:19.393 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:19.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:19.528 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:19.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:19.666 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:19.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:19.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:19 smithi165 bash[22591]: cluster 2024-09-10T19:07:18.414258+0000 mon.a (mon.0) 3547 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:19.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:19 smithi165 bash[22591]: cluster 2024-09-10T19:07:18.432866+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-09-10T19:07:19.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:19 smithi165 bash[22591]: cluster 2024-09-10T19:07:19.367763+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-09-10T19:07:19.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:19 smithi165 bash[22591]: audit 2024-09-10T19:07:19.373441+0000 mon.c (mon.2) 812 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.782 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:19 smithi165 bash[22591]: audit 2024-09-10T19:07:19.379477+0000 mon.a (mon.0) 3550 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[32381]: cluster 2024-09-10T19:07:18.414258+0000 mon.a (mon.0) 3547 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[32381]: cluster 2024-09-10T19:07:18.432866+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[32381]: cluster 2024-09-10T19:07:19.367763+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[32381]: audit 2024-09-10T19:07:19.373441+0000 mon.c (mon.2) 812 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[32381]: audit 2024-09-10T19:07:19.379477+0000 mon.a (mon.0) 3550 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[18629]: cluster 2024-09-10T19:07:18.414258+0000 mon.a (mon.0) 3547 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[18629]: cluster 2024-09-10T19:07:18.432866+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[18629]: cluster 2024-09-10T19:07:19.367763+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[18629]: audit 2024-09-10T19:07:19.373441+0000 mon.c (mon.2) 812 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:19 smithi083 bash[18629]: audit 2024-09-10T19:07:19.379477+0000 mon.a (mon.0) 3550 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.801 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:19.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:19.942 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:19.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:20.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:20.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:20.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:20.242 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:20.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:20.384 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:20.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:20.523 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:20.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:20.661 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:20.764 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:20.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:20.903 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:20.937 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:20.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:21.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:21.079 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:21.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:21.211 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- 172.21.15.83:0/2701591102 >> v1:172.21.15.83:6790/0 conn(0x7fac84105380 legacy=0x7fac84107760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- 172.21.15.83:0/2701591102 shutdown_connections 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- 172.21.15.83:0/2701591102 >> 172.21.15.83:0/2701591102 conn(0x7fac840f81a0 msgr2=0x7fac840fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- 172.21.15.83:0/2701591102 shutdown_connections 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- 172.21.15.83:0/2701591102 wait complete. 2024-09-10T19:07:21.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 librados: starting msgr at 2024-09-10T19:07:21.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 librados: starting objecter 2024-09-10T19:07:21.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 Processor -- start 2024-09-10T19:07:21.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- start start 2024-09-10T19:07:21.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 librados: setting wanted keys 2024-09-10T19:07:21.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 librados: calling monclient init 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac84101e70 con 0x7fac84108f10 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac8411d3a0 con 0x7fac841025c0 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.301+0000 7fac891bc640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fac8411e540 con 0x7fac84105380 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac8359e640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fac84108f10 0x7fac8411ac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60960/0 (socket says 172.21.15.83:60960) 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac8359e640 1 -- 172.21.15.83:0/4292134786 learned_addr learned my addr 172.21.15.83:0/4292134786 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac84108f10 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac84105380 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:21.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac841025c0 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293341079 0 0) 0x7fac84101e70 con 0x7fac84108f10 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c0034e0 con 0x7fac84108f10 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545697602 0 0) 0x7fac8411e540 con 0x7fac84105380 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac84101e70 con 0x7fac84105380 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819263250 0 0) 0x7fac8411d3a0 con 0x7fac841025c0 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac8411e540 con 0x7fac841025c0 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1414809554 0 0) 0x7fac6c0034e0 con 0x7fac84108f10 2024-09-10T19:07:21.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac8411d3a0 con 0x7fac84108f10 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac78002930 con 0x7fac84108f10 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 929288634 0 0) 0x7fac8411e540 con 0x7fac841025c0 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac6c0034e0 con 0x7fac841025c0 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac7c001c40 con 0x7fac841025c0 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529437983 0 0) 0x7fac84101e70 con 0x7fac84105380 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac8411e540 con 0x7fac84105380 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 643032329 0 0) 0x7fac8411d3a0 con 0x7fac84108f10 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 >> v1:172.21.15.83:6790/0 conn(0x7fac84105380 legacy=0x7fac84117540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 >> v1:172.21.15.165:6789/0 conn(0x7fac841025c0 legacy=0x7fac841012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8411f740 con 0x7fac84108f10 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8411e730 con 0x7fac84108f10 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac891bc640 10 client.25731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8411eb80 con 0x7fac84108f10 2024-09-10T19:07:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac891bc640 1 librados: init done 2024-09-10T19:07:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fac78002ca0 con 0x7fac84108f10 2024-09-10T19:07:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fac78005ac0 con 0x7fac84108f10 2024-09-10T19:07:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac70005040 con 0x7fac84108f10 2024-09-10T19:07:21.311 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.305+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fac78001c70 con 0x7fac84108f10 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(780..780 src has 704..780) v4 ==== 8536+0+0 (unknown 3029405982 0 0) 0x7fac7808a780 con 0x7fac84108f10 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 10 client.25731.objecter ms_dispatch 0x7fac84000b90 osd_map(780..780 src has 704..780) v4 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 3 client.25731.objecter handle_osd_map got epochs [780,780] > 0 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 3 client.25731.objecter handle_osd_map decoding full epoch 780 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 20 client.25731.objecter dump_active .. 0 homeless 2024-09-10T19:07:21.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.309+0000 7fac6b7fe640 10 client.25731.objecter ms_handle_connect 0x7fac6c071800 2024-09-10T19:07:21.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:21.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.317+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fac78056b50 con 0x7fac84108f10 2024-09-10T19:07:21.348 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:21.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: cluster 2024-09-10T19:07:20.344390+0000 mgr.y (mgr.24526) 708 : cluster [DBG] pgmap v1205: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: audit 2024-09-10T19:07:20.365445+0000 mon.a (mon.0) 3551 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: cluster 2024-09-10T19:07:20.365554+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: audit 2024-09-10T19:07:20.374313+0000 mon.c (mon.2) 813 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: audit 2024-09-10T19:07:20.376253+0000 mon.a (mon.0) 3553 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: cluster 2024-09-10T19:07:20.572158+0000 osd.7 (osd.7) 404 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[18629]: cluster 2024-09-10T19:07:20.572473+0000 osd.7 (osd.7) 405 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:21.462 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: cluster 2024-09-10T19:07:20.344390+0000 mgr.y (mgr.24526) 708 : cluster [DBG] pgmap v1205: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: audit 2024-09-10T19:07:20.365445+0000 mon.a (mon.0) 3551 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: cluster 2024-09-10T19:07:20.365554+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: audit 2024-09-10T19:07:20.374313+0000 mon.c (mon.2) 813 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: audit 2024-09-10T19:07:20.376253+0000 mon.a (mon.0) 3553 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: cluster 2024-09-10T19:07:20.572158+0000 osd.7 (osd.7) 404 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:21.463 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:21 smithi083 bash[32381]: cluster 2024-09-10T19:07:20.572473+0000 osd.7 (osd.7) 405 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:21.479 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:21.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:21.513+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fac700052f0 con 0x7fac84108f10 2024-09-10T19:07:21.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: cluster 2024-09-10T19:07:20.344390+0000 mgr.y (mgr.24526) 708 : cluster [DBG] pgmap v1205: 214 pgs: 32 unknown, 182 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: audit 2024-09-10T19:07:20.365445+0000 mon.a (mon.0) 3551 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: cluster 2024-09-10T19:07:20.365554+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: audit 2024-09-10T19:07:20.374313+0000 mon.c (mon.2) 813 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: audit 2024-09-10T19:07:20.376253+0000 mon.a (mon.0) 3553 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: cluster 2024-09-10T19:07:20.572158+0000 osd.7 (osd.7) 404 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:21.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:21 smithi165 bash[22591]: cluster 2024-09-10T19:07:20.572473+0000 osd.7 (osd.7) 405 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:21.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:21.716 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:21.748 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:21.851 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:21.884 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:21.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:21.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:22.024 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:22.128 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.164 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:22.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.301 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:22.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.381+0000 7fac6b7fe640 1 -- 172.21.15.83:0/4292134786 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v782) v1 ==== 97+0+0 (unknown 1700551136 0 0) 0x7fac780564f0 con 0x7fac84108f10 2024-09-10T19:07:22.385 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:07:22.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 10 librados: watch_flush enter 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 10 librados: watch_flush exit 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 20 client.25731.objecter shutdown clearing up homeless session... 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 10 client.25731.objecter successfully canceled tick 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 >> v1:172.21.15.83:6800/3071793735 conn(0x7fac6c071800 legacy=0x7fac6c073cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 >> v1:172.21.15.83:6789/0 conn(0x7fac84108f10 legacy=0x7fac8411ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 shutdown_connections 2024-09-10T19:07:22.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 >> 172.21.15.83:0/4292134786 conn(0x7fac840f81a0 msgr2=0x7fac84109390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:22.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 shutdown_connections 2024-09-10T19:07:22.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 -- 172.21.15.83:0/4292134786 wait complete. 2024-09-10T19:07:22.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.385+0000 7fac891bc640 1 librados: shutdown 2024-09-10T19:07:22.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:22.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.449+0000 7f1d809d4640 1 -- 172.21.15.83:0/1273234106 >> v1:172.21.15.83:6789/0 conn(0x7f1d7c0fbc50 legacy=0x7f1d7c1097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.449+0000 7f1d809d4640 1 -- 172.21.15.83:0/1273234106 shutdown_connections 2024-09-10T19:07:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.449+0000 7f1d809d4640 1 -- 172.21.15.83:0/1273234106 >> 172.21.15.83:0/1273234106 conn(0x7f1d7c0f6f20 msgr2=0x7f1d7c0f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- 172.21.15.83:0/1273234106 shutdown_connections 2024-09-10T19:07:22.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- 172.21.15.83:0/1273234106 wait complete. 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 librados: starting msgr at 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 librados: starting objecter 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 Processor -- start 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- start start 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 librados: setting wanted keys 2024-09-10T19:07:22.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 librados: calling monclient init 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d7c10ac40 con 0x7f1d7c0fb170 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d7c11f460 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d7c120600 con 0x7f1d7c10afa0 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d7ad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1d7c0fb170 0x7f1d7c10a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60984/0 (socket says 172.21.15.83:60984) 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d7ad9d640 1 -- 172.21.15.83:0/3493996965 learned_addr learned my addr 172.21.15.83:0/3493996965 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:22.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter ms_handle_connect 0x7f1d7c0fb170 2024-09-10T19:07:22.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter ms_handle_connect 0x7f1d7c0fbc50 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter ms_handle_connect 0x7f1d7c10afa0 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442742420 0 0) 0x7f1d7c10ac40 con 0x7f1d7c0fb170 2024-09-10T19:07:22.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d680034e0 con 0x7f1d7c0fb170 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794991210 0 0) 0x7f1d7c11f460 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d7c10ac40 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807717080 0 0) 0x7f1d7c120600 con 0x7f1d7c10afa0 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d7c11f460 con 0x7f1d7c10afa0 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428976889 0 0) 0x7f1d7c10ac40 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d7c120600 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d64001430 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3401070292 0 0) 0x7f1d7c11f460 con 0x7f1d7c10afa0 2024-09-10T19:07:22.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d7c10ac40 con 0x7f1d7c10afa0 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670761320 0 0) 0x7f1d680034e0 con 0x7f1d7c0fb170 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d7c11f460 con 0x7f1d7c0fb170 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d74001ae0 con 0x7f1d7c10afa0 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d6c001900 con 0x7f1d7c0fb170 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3326423548 0 0) 0x7f1d7c120600 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6790/0 conn(0x7f1d7c10afa0 legacy=0x7f1d7c11cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6789/0 conn(0x7f1d7c0fb170 legacy=0x7f1d7c10a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c1217e0 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c11f650 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 10 client.50332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:22.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c11fb20 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d809d4640 1 librados: init done 2024-09-10T19:07:22.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.453+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1d64001b30 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1d640035b0 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1d5c004ff0 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1d64002bb0 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(782..782 src has 704..782) v4 ==== 8531+0+0 (unknown 3591740027 0 0) 0x7f1d64088030 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 osd_map(782..782 src has 704..782) v4 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 3 client.50332.objecter handle_osd_map got epochs [782,782] > 0 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 3 client.50332.objecter handle_osd_map decoding full epoch 782 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 20 client.50332.objecter dump_active .. 0 homeless 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v782) v1 ==== 54+0+16 (unknown 1309096239 0 2420000192) 0x7f1d64088420 con 0x7f1d7c0fbc50 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d68071870 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 10 client.50332.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 1 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 1 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 0 1 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 4 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 10 client.50332.objecter _send_command 1 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.457+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c005290 con 0x7f1d5c004610 2024-09-10T19:07:22.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c004610 2024-09-10T19:07:22.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c005290 con 0x7f1d5c004610 2024-09-10T19:07:22.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:07:22.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 10 client.50332.objecter _finish_command 1 = system:0 2024-09-10T19:07:22.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 4 2024-09-10T19:07:22.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.461+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 0 1 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 10 client.50332.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 2 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 2 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c005810 osd=0 3 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 0 2 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 4 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 10 client.50332.objecter _send_command 2 2024-09-10T19:07:22.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.477+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c004c60 con 0x7f1d5c004610 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c004c60 con 0x7f1d5c004610 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 10 client.50332.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 4 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 0 2 2024-09-10T19:07:22.483 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 10 client.50332.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 1 3 2024-09-10T19:07:22.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 4 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 10 client.50332.objecter _send_command 3 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6805/2309624637 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c003b10 con 0x7f1d5c0035e0 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c0035e0 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.1 v1:172.21.15.83:6805/2309624637 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c003b10 con 0x7f1d5c0035e0 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 3: 0 ) v1 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 10 client.50332.objecter _finish_command 3 = system:0 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 4 2024-09-10T19:07:22.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.481+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 1 3 2024-09-10T19:07:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 10 client.50332.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 4 2024-09-10T19:07:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 4 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c004e10 osd=1 3 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 1 4 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 4 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 10 client.50332.objecter _send_command 4 2024-09-10T19:07:22.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.501+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6805/2309624637 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c003060 con 0x7f1d5c0035e0 2024-09-10T19:07:22.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.1 v1:172.21.15.83:6805/2309624637 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c003060 con 0x7f1d5c0035e0 2024-09-10T19:07:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 10 client.50332.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 4 2024-09-10T19:07:22.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 1 4 2024-09-10T19:07:22.508 INFO:teuthology.orchestra.run.smithi083.stdout:osd.1: {} 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 10 client.50332.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 5 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 5 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 2 5 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 4 2024-09-10T19:07:22.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 10 client.50332.objecter _send_command 5 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c002350 con 0x7f1d5c00d350 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c00d350 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.505+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c002350 con 0x7f1d5c00d350 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.509+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 5: 0 ) v1 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.509+0000 7f1d737fe640 10 client.50332.objecter _finish_command 5 = system:0 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.509+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 4 2024-09-10T19:07:22.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.509+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 2 5 2024-09-10T19:07:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 10 client.50332.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 6 2024-09-10T19:07:22.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 6 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c001b60 osd=2 3 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 2 6 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 4 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 10 client.50332.objecter _send_command 6 2024-09-10T19:07:22.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.529+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6809/386094220 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c001650 con 0x7f1d5c00d350 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c001650 con 0x7f1d5c00d350 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 10 client.50332.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 4 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 2 6 2024-09-10T19:07:22.535 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 10 client.50332.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 7 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 7 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 3 7 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 4 2024-09-10T19:07:22.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 10 client.50332.objecter _send_command 7 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c00d170 con 0x7f1d5c011fd0 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c011fd0 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c00d170 con 0x7f1d5c011fd0 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 7: 0 ) v1 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 10 client.50332.objecter _finish_command 7 = system:0 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 4 2024-09-10T19:07:22.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.533+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 3 7 2024-09-10T19:07:22.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[32381]: audit 2024-09-10T19:07:21.370712+0000 mon.a (mon.0) 3554 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[32381]: cluster 2024-09-10T19:07:21.370813+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-09-10T19:07:22.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[32381]: audit 2024-09-10T19:07:21.372764+0000 mon.c (mon.2) 814 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[32381]: audit 2024-09-10T19:07:21.379501+0000 mon.a (mon.0) 3556 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[32381]: audit 2024-09-10T19:07:21.518224+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18629]: audit 2024-09-10T19:07:21.370712+0000 mon.a (mon.0) 3554 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18629]: cluster 2024-09-10T19:07:21.370813+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18629]: audit 2024-09-10T19:07:21.372764+0000 mon.c (mon.2) 814 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18629]: audit 2024-09-10T19:07:21.379501+0000 mon.a (mon.0) 3556 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18629]: audit 2024-09-10T19:07:21.518224+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:07:22.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 10 client.50332.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 8 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 8 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c990 osd=3 3 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 3 8 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 4 2024-09-10T19:07:22.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 10 client.50332.objecter _send_command 8 2024-09-10T19:07:22.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.83:6813/1279688021 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c00c410 con 0x7f1d5c011fd0 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c00c410 con 0x7f1d5c011fd0 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 10 client.50332.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 4 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 3 8 2024-09-10T19:07:22.556 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 10 client.50332.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 9 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 9 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 4 9 2024-09-10T19:07:22.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 4 2024-09-10T19:07:22.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 10 client.50332.objecter _send_command 9 2024-09-10T19:07:22.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c00c380 con 0x7f1d5c011b20 2024-09-10T19:07:22.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.553+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c011b20 2024-09-10T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:22 smithi165 bash[22591]: audit 2024-09-10T19:07:21.370712+0000 mon.a (mon.0) 3554 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:22 smithi165 bash[22591]: cluster 2024-09-10T19:07:21.370813+0000 mon.a (mon.0) 3555 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-09-10T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:22 smithi165 bash[22591]: audit 2024-09-10T19:07:21.372764+0000 mon.c (mon.2) 814 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:22 smithi165 bash[22591]: audit 2024-09-10T19:07:21.379501+0000 mon.a (mon.0) 3556 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:22.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:22 smithi165 bash[22591]: audit 2024-09-10T19:07:21.518224+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:07:22.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.557+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c00c380 con 0x7f1d5c011b20 2024-09-10T19:07:22.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.557+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 9: 0 ) v1 2024-09-10T19:07:22.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.557+0000 7f1d737fe640 10 client.50332.objecter _finish_command 9 = system:0 2024-09-10T19:07:22.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.557+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 4 2024-09-10T19:07:22.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.557+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 4 9 2024-09-10T19:07:22.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 10 client.50332.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 10 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 10 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c1d0 osd=4 3 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 4 10 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 4 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 10 client.50332.objecter _send_command 10 2024-09-10T19:07:22.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.573+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6800/3225515265 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c00c380 con 0x7f1d5c011b20 2024-09-10T19:07:22.579 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:22.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.577+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c00c380 con 0x7f1d5c011b20 2024-09-10T19:07:22.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.577+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.577+0000 7f1d737fe640 10 client.50332.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.577+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 4 2024-09-10T19:07:22.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.577+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 4 10 2024-09-10T19:07:22.583 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:07:22.583 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 10 client.50332.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 11 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 11 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 5 11 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 4 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 10 client.50332.objecter _send_command 11 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c017df0 con 0x7f1d5c017870 2024-09-10T19:07:22.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c017870 2024-09-10T19:07:22.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c017df0 con 0x7f1d5c017870 2024-09-10T19:07:22.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 11: 0 ) v1 2024-09-10T19:07:22.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 10 client.50332.objecter _finish_command 11 = system:0 2024-09-10T19:07:22.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 4 2024-09-10T19:07:22.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.581+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 5 11 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 10 client.50332.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 12 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 12 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c00c380 osd=5 3 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 5 12 2024-09-10T19:07:22.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 4 2024-09-10T19:07:22.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 10 client.50332.objecter _send_command 12 2024-09-10T19:07:22.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.601+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6804/571173777 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c017df0 con 0x7f1d5c017870 2024-09-10T19:07:22.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c017df0 con 0x7f1d5c017870 2024-09-10T19:07:22.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 10 client.50332.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 4 2024-09-10T19:07:22.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 5 12 2024-09-10T19:07:22.609 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 10 client.50332.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 13 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 13 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 6 13 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 4 2024-09-10T19:07:22.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 10 client.50332.objecter _send_command 13 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c022ad0 con 0x7f1d5c01efd0 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.605+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c01efd0 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.609+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c022ad0 con 0x7f1d5c01efd0 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.609+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 13: 0 ) v1 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.609+0000 7f1d737fe640 10 client.50332.objecter _finish_command 13 = system:0 2024-09-10T19:07:22.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.609+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 4 2024-09-10T19:07:22.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.609+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 6 13 2024-09-10T19:07:22.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 10 client.50332.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 14 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 14 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c017df0 osd=6 3 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 6 14 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 4 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 10 client.50332.objecter _send_command 14 2024-09-10T19:07:22.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.625+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6808/1317922936 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c022ad0 con 0x7f1d5c01efd0 2024-09-10T19:07:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c022ad0 con 0x7f1d5c01efd0 2024-09-10T19:07:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 10 client.50332.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 4 2024-09-10T19:07:22.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 6 14 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 10 client.50332.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 15 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 15 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 7 15 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 4 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 10 client.50332.objecter _send_command 15 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d5c026dd0 con 0x7f1d5c023150 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 10 client.50332.objecter ms_handle_connect 0x7f1d5c023150 2024-09-10T19:07:22.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1d5c026dd0 con 0x7f1d5c023150 2024-09-10T19:07:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 15: 0 ) v1 2024-09-10T19:07:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 10 client.50332.objecter _finish_command 15 = system:0 2024-09-10T19:07:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 4 2024-09-10T19:07:22.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.629+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 7 15 2024-09-10T19:07:22.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 10 client.50332.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:07:22.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign -1 16 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 20 client.50332.objecter _get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_remove -1 16 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 20 client.50332.objecter get_session s=0x7f1d5c022ad0 osd=7 3 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 15 client.50332.objecter _session_command_op_assign 7 16 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 4 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 10 client.50332.objecter _send_command 16 2024-09-10T19:07:22.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.645+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 --> v1:172.21.15.165:6812/1610151650 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1d5c00c6a0 con 0x7f1d5c023150 2024-09-10T19:07:22.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d737fe640 1 -- 172.21.15.83:0/3493996965 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f1d5c00c6a0 con 0x7f1d5c023150 2024-09-10T19:07:22.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d737fe640 10 client.50332.objecter ms_dispatch 0x7f1d7c000b90 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:22.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d737fe640 10 client.50332.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:22.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d737fe640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 4 2024-09-10T19:07:22.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d737fe640 15 client.50332.objecter _session_command_op_remove 7 16 2024-09-10T19:07:22.652 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:07:22.653 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:07:22.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 librados: watch_flush enter 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 librados: watch_flush exit 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.0 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6801/2535527483 conn(0x7f1d5c004610 legacy=0x7f1d5c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c005810 osd=0 1 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.1 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6805/2309624637 conn(0x7f1d5c0035e0 legacy=0x7f1d5c00aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c004e10 osd=1 1 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.2 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6809/386094220 conn(0x7f1d5c00d350 legacy=0x7f1d5c00f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c001b60 osd=2 1 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.3 2024-09-10T19:07:22.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6813/1279688021 conn(0x7f1d5c011fd0 legacy=0x7f1d5c0143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c990 osd=3 1 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.4 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.165:6800/3225515265 conn(0x7f1d5c011b20 legacy=0x7f1d5c016c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d7b59e640 1 -- 172.21.15.83:0/3493996965 reap_dead start 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d7b59e640 1 -- 172.21.15.83:0/3493996965 reap_dead start 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c1d0 osd=4 1 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.5 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.165:6804/571173777 conn(0x7f1d5c017870 legacy=0x7f1d5c01d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c00c380 osd=5 1 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.6 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.165:6808/1317922936 conn(0x7f1d5c01efd0 legacy=0x7f1d5c0213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c017df0 osd=6 1 2024-09-10T19:07:22.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter close_session for osd.7 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.165:6812/1610151650 conn(0x7f1d5c023150 legacy=0x7f1d5c025550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter put_session s=0x7f1d5c022ad0 osd=7 1 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 20 client.50332.objecter shutdown clearing up homeless session... 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 10 client.50332.objecter successfully canceled tick 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1d68071870 legacy=0x7f1d68073d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d7b59e640 1 -- 172.21.15.83:0/3493996965 reap_dead start 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> v1:172.21.15.165:6789/0 conn(0x7f1d7c0fbc50 legacy=0x7f1d7c119600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 shutdown_connections 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 >> 172.21.15.83:0/3493996965 conn(0x7f1d7c0f6f20 msgr2=0x7f1d7c0f9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 shutdown_connections 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 -- 172.21.15.83:0/3493996965 wait complete. 2024-09-10T19:07:22.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:22.649+0000 7f1d809d4640 1 librados: shutdown 2024-09-10T19:07:22.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:22.712 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:22.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:22.848 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:22.952 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.985 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:22.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:22] "GET /metrics HTTP/1.1" 200 228991 "" "Prometheus/2.43.0" 2024-09-10T19:07:23.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:23.129 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:23.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.267 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:23.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:23.401 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:23.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:23.538 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:23.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: cluster 2024-09-10T19:07:22.345092+0000 mgr.y (mgr.24526) 709 : cluster [DBG] pgmap v1208: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: cluster 2024-09-10T19:07:22.374636+0000 mon.a (mon.0) 3558 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:22.383189+0000 mon.a (mon.0) 3559 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:22.383410+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: cluster 2024-09-10T19:07:22.383504+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:22.386048+0000 mon.c (mon.2) 815 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:22.388962+0000 mon.a (mon.0) 3562 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:22.457692+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.83:0/3493996965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:23.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: audit 2024-09-10T19:07:23.386294+0000 mon.a (mon.0) 3563 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:23.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:23 smithi165 bash[22591]: cluster 2024-09-10T19:07:23.386435+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-09-10T19:07:23.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:23.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: cluster 2024-09-10T19:07:22.345092+0000 mgr.y (mgr.24526) 709 : cluster [DBG] pgmap v1208: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:23.737 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: cluster 2024-09-10T19:07:22.374636+0000 mon.a (mon.0) 3558 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:22.383189+0000 mon.a (mon.0) 3559 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:22.383410+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: cluster 2024-09-10T19:07:22.383504+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:22.386048+0000 mon.c (mon.2) 815 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:22.388962+0000 mon.a (mon.0) 3562 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:22.457692+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.83:0/3493996965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: audit 2024-09-10T19:07:23.386294+0000 mon.a (mon.0) 3563 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[18629]: cluster 2024-09-10T19:07:23.386435+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: cluster 2024-09-10T19:07:22.345092+0000 mgr.y (mgr.24526) 709 : cluster [DBG] pgmap v1208: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:23.738 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: cluster 2024-09-10T19:07:22.374636+0000 mon.a (mon.0) 3558 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:22.383189+0000 mon.a (mon.0) 3559 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:22.383410+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.83:0/4292134786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: cluster 2024-09-10T19:07:22.383504+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:22.386048+0000 mon.c (mon.2) 815 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:22.388962+0000 mon.a (mon.0) 3562 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:22.457692+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.83:0/3493996965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: audit 2024-09-10T19:07:23.386294+0000 mon.a (mon.0) 3563 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:23.739 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:23 smithi083 bash[32381]: cluster 2024-09-10T19:07:23.386435+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-09-10T19:07:23.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.814 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:23.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:23.953 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:23.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:24.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:24.092 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:24.196 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:24.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:24.366 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:24.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:24.502 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:24.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:24 smithi165 bash[22591]: cluster 2024-09-10T19:07:23.351131+0000 osd.0 (osd.0) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:24 smithi165 bash[22591]: cluster 2024-09-10T19:07:23.351494+0000 osd.0 (osd.0) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:24 smithi165 bash[22591]: audit 2024-09-10T19:07:23.388873+0000 mon.c (mon.2) 816 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:24 smithi165 bash[22591]: audit 2024-09-10T19:07:23.390026+0000 mon.a (mon.0) 3565 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:24 smithi165 bash[22591]: cluster 2024-09-10T19:07:24.357098+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:24.637 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:24.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:24.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[32381]: cluster 2024-09-10T19:07:23.351131+0000 osd.0 (osd.0) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:24.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[32381]: cluster 2024-09-10T19:07:23.351494+0000 osd.0 (osd.0) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:24.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[32381]: audit 2024-09-10T19:07:23.388873+0000 mon.c (mon.2) 816 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.754 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[32381]: audit 2024-09-10T19:07:23.390026+0000 mon.a (mon.0) 3565 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[32381]: cluster 2024-09-10T19:07:24.357098+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[18629]: cluster 2024-09-10T19:07:23.351131+0000 osd.0 (osd.0) 385 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[18629]: cluster 2024-09-10T19:07:23.351494+0000 osd.0 (osd.0) 386 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[18629]: audit 2024-09-10T19:07:23.388873+0000 mon.c (mon.2) 816 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[18629]: audit 2024-09-10T19:07:23.390026+0000 mon.a (mon.0) 3565 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:24.755 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:24 smithi083 bash[18629]: cluster 2024-09-10T19:07:24.357098+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:24.775 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:24.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:24.910 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:24.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:24.954 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:24.954 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:25.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:25.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:25.185 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:25.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:25.322 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:25.414 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:25 smithi165 bash[22591]: cluster 2024-09-10T19:07:24.345916+0000 mgr.y (mgr.24526) 710 : cluster [DBG] pgmap v1211: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:25.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:25 smithi165 bash[22591]: audit 2024-09-10T19:07:24.391746+0000 mon.a (mon.0) 3567 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:25.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:25 smithi165 bash[22591]: cluster 2024-09-10T19:07:24.391962+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-09-10T19:07:25.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:25 smithi165 bash[22591]: audit 2024-09-10T19:07:24.394839+0000 mon.c (mon.2) 817 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.415 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:25 smithi165 bash[22591]: audit 2024-09-10T19:07:24.401380+0000 mon.a (mon.0) 3569 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:25.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:25.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.585 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:25.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[18629]: cluster 2024-09-10T19:07:24.345916+0000 mgr.y (mgr.24526) 710 : cluster [DBG] pgmap v1211: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:25.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[18629]: audit 2024-09-10T19:07:24.391746+0000 mon.a (mon.0) 3567 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:25.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[18629]: cluster 2024-09-10T19:07:24.391962+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-09-10T19:07:25.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[18629]: audit 2024-09-10T19:07:24.394839+0000 mon.c (mon.2) 817 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.704 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[18629]: audit 2024-09-10T19:07:24.401380+0000 mon.a (mon.0) 3569 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[32381]: cluster 2024-09-10T19:07:24.345916+0000 mgr.y (mgr.24526) 710 : cluster [DBG] pgmap v1211: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:25.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[32381]: audit 2024-09-10T19:07:24.391746+0000 mon.a (mon.0) 3567 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:25.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[32381]: cluster 2024-09-10T19:07:24.391962+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-09-10T19:07:25.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[32381]: audit 2024-09-10T19:07:24.394839+0000 mon.c (mon.2) 817 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.705 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:25 smithi083 bash[32381]: audit 2024-09-10T19:07:24.401380+0000 mon.a (mon.0) 3569 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:25.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:25.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:07:25.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:25.856 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:25.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:25.992 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:25.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:26.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:26.129 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:26.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:26.266 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:26.370 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.405 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:26.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:26.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:07:26.542 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:26.646 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:26.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[18629]: audit 2024-09-10T19:07:25.405395+0000 mon.a (mon.0) 3570 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:07:26.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[18629]: cluster 2024-09-10T19:07:25.405576+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-09-10T19:07:26.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[18629]: cluster 2024-09-10T19:07:26.348109+0000 mgr.y (mgr.24526) 711 : cluster [DBG] pgmap v1214: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-10T19:07:26.656 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[32381]: audit 2024-09-10T19:07:25.405395+0000 mon.a (mon.0) 3570 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:07:26.656 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[32381]: cluster 2024-09-10T19:07:25.405576+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-09-10T19:07:26.656 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:26 smithi083 bash[32381]: cluster 2024-09-10T19:07:26.348109+0000 mgr.y (mgr.24526) 711 : cluster [DBG] pgmap v1214: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-10T19:07:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:26.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.804 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:26 smithi165 bash[22591]: audit 2024-09-10T19:07:25.405395+0000 mon.a (mon.0) 3570 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:07:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:26 smithi165 bash[22591]: cluster 2024-09-10T19:07:25.405576+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-09-10T19:07:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:26 smithi165 bash[22591]: cluster 2024-09-10T19:07:26.348109+0000 mgr.y (mgr.24526) 711 : cluster [DBG] pgmap v1214: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-09-10T19:07:26.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.941 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:26.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:27.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:27.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:27.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:27.215 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:27.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:27.345 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[18629]: cluster 2024-09-10T19:07:26.194877+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[18629]: cluster 2024-09-10T19:07:26.195194+0000 osd.5 (osd.5) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[18629]: cluster 2024-09-10T19:07:26.426072+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[32381]: cluster 2024-09-10T19:07:26.194877+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[32381]: cluster 2024-09-10T19:07:26.195194+0000 osd.5 (osd.5) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:27.443 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:27 smithi083 bash[32381]: cluster 2024-09-10T19:07:26.426072+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-09-10T19:07:27.448 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.478 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:27.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:27.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:27 smithi165 bash[22591]: cluster 2024-09-10T19:07:26.194877+0000 osd.5 (osd.5) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:27.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:27 smithi165 bash[22591]: cluster 2024-09-10T19:07:26.195194+0000 osd.5 (osd.5) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:27.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:27 smithi165 bash[22591]: cluster 2024-09-10T19:07:26.426072+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-09-10T19:07:27.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:27.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.753 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:27.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.894 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:27.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:27.997 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:28.021 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:28.125 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:28.156 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:28.259 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.293 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:28.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:28.426 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:28.432 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:28 smithi083 bash[32381]: cluster 2024-09-10T19:07:28.348963+0000 mgr.y (mgr.24526) 712 : cluster [DBG] pgmap v1216: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:07:28.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:28 smithi083 bash[18629]: cluster 2024-09-10T19:07:28.348963+0000 mgr.y (mgr.24526) 712 : cluster [DBG] pgmap v1216: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:07:28.562 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:28.665 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.701 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:28.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:28.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:28 smithi165 bash[22591]: cluster 2024-09-10T19:07:28.348963+0000 mgr.y (mgr.24526) 712 : cluster [DBG] pgmap v1216: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:07:28.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:28.839 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:28.942 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.977 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:28.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:29.081 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.116 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:29.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.255 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:29.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:29.386 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:29.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:29.526 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:29.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:29.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:29 smithi165 bash[22591]: audit 2024-09-10T19:07:28.392679+0000 mgr.y (mgr.24526) 713 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:29 smithi165 bash[22591]: cluster 2024-09-10T19:07:29.172196+0000 osd.4 (osd.4) 396 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:29.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:29 smithi165 bash[22591]: cluster 2024-09-10T19:07:29.172533+0000 osd.4 (osd.4) 397 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:29.642 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:29 smithi165 bash[22591]: cluster 2024-09-10T19:07:29.367805+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-09-10T19:07:29.661 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:29.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[32381]: audit 2024-09-10T19:07:28.392679+0000 mgr.y (mgr.24526) 713 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[32381]: cluster 2024-09-10T19:07:29.172196+0000 osd.4 (osd.4) 396 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[32381]: cluster 2024-09-10T19:07:29.172533+0000 osd.4 (osd.4) 397 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[32381]: cluster 2024-09-10T19:07:29.367805+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[18629]: audit 2024-09-10T19:07:28.392679+0000 mgr.y (mgr.24526) 713 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[18629]: cluster 2024-09-10T19:07:29.172196+0000 osd.4 (osd.4) 396 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[18629]: cluster 2024-09-10T19:07:29.172533+0000 osd.4 (osd.4) 397 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:29.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:29 smithi083 bash[18629]: cluster 2024-09-10T19:07:29.367805+0000 mon.a (mon.0) 3573 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-09-10T19:07:29.798 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:29.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:29.935 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:29.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:30.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:30.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:30.181 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:30.320 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:30.358 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:30.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:30.496 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:30.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:30.636 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:30.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:30.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:30 smithi165 bash[22591]: audit 2024-09-10T19:07:29.575430+0000 mon.c (mon.2) 818 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:30.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:30 smithi165 bash[22591]: cluster 2024-09-10T19:07:30.349801+0000 mgr.y (mgr.24526) 714 : cluster [DBG] pgmap v1218: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 1.3 KiB/s wr, 1 op/s 2024-09-10T19:07:30.771 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:30.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:30 smithi083 bash[18629]: audit 2024-09-10T19:07:29.575430+0000 mon.c (mon.2) 818 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:30.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:30 smithi083 bash[18629]: cluster 2024-09-10T19:07:30.349801+0000 mgr.y (mgr.24526) 714 : cluster [DBG] pgmap v1218: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 1.3 KiB/s wr, 1 op/s 2024-09-10T19:07:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:30 smithi083 bash[32381]: audit 2024-09-10T19:07:29.575430+0000 mon.c (mon.2) 818 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:30.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:30 smithi083 bash[32381]: cluster 2024-09-10T19:07:30.349801+0000 mgr.y (mgr.24526) 714 : cluster [DBG] pgmap v1218: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 858 B/s rd, 1.3 KiB/s wr, 1 op/s 2024-09-10T19:07:30.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:30.907 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:30.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:31.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:31.045 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:31.148 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:31.177 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:31.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:31.315 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:31.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:31.453 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:31.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:31.591 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:31 smithi165 bash[22591]: cluster 2024-09-10T19:07:30.820006+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:31 smithi165 bash[22591]: cluster 2024-09-10T19:07:30.820464+0000 osd.1 (osd.1) 413 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:31.613 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:31.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:31.708 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:31 smithi083 bash[18629]: cluster 2024-09-10T19:07:30.820006+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:31.708 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:31 smithi083 bash[18629]: cluster 2024-09-10T19:07:30.820464+0000 osd.1 (osd.1) 413 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:31.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:31 smithi083 bash[32381]: cluster 2024-09-10T19:07:30.820006+0000 osd.1 (osd.1) 412 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:31.709 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:31 smithi083 bash[32381]: cluster 2024-09-10T19:07:30.820464+0000 osd.1 (osd.1) 413 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:31.727 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:31.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:31.858 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 -- 172.21.15.83:0/1575660954 >> v1:172.21.15.83:6789/0 conn(0x7fad441025a0 legacy=0x7fad44102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 -- 172.21.15.83:0/1575660954 shutdown_connections 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 -- 172.21.15.83:0/1575660954 >> 172.21.15.83:0/1575660954 conn(0x7fad440f81a0 msgr2=0x7fad440fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 -- 172.21.15.83:0/1575660954 shutdown_connections 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 -- 172.21.15.83:0/1575660954 wait complete. 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 librados: starting msgr at 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 librados: starting objecter 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.893+0000 7fad494ab640 1 Processor -- start 2024-09-10T19:07:31.898 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- start start 2024-09-10T19:07:31.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 librados: setting wanted keys 2024-09-10T19:07:31.899 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 librados: calling monclient init 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad44101e40 con 0x7fad44108ef0 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad4411f770 con 0x7fad44105360 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fad44120910 con 0x7fad441025a0 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad437fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fad44108ef0 0x7fad4411d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53248/0 (socket says 172.21.15.83:53248) 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad437fe640 1 -- 172.21.15.83:0/1182162833 learned_addr learned my addr 172.21.15.83:0/1182162833 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter ms_handle_connect 0x7fad44108ef0 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:31.900 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter ms_handle_connect 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146136132 0 0) 0x7fad44120910 con 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad2c003120 con 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503051345 0 0) 0x7fad44101e40 con 0x7fad44108ef0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44120910 con 0x7fad44108ef0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter ms_handle_connect 0x7fad44105360 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563788050 0 0) 0x7fad2c003120 con 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad44101e40 con 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fad34002130 con 0x7fad441025a0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256947851 0 0) 0x7fad44120910 con 0x7fad44108ef0 2024-09-10T19:07:31.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad2c003120 con 0x7fad44108ef0 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fad3c001c80 con 0x7fad44108ef0 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744380588 0 0) 0x7fad44101e40 con 0x7fad441025a0 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 >> v1:172.21.15.165:6789/0 conn(0x7fad44105360 legacy=0x7fad44119910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 >> v1:172.21.15.83:6789/0 conn(0x7fad44108ef0 legacy=0x7fad4411d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44121ab0 con 0x7fad441025a0 2024-09-10T19:07:31.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4411e7c0 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 10 client.49829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4411ec10 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 librados: init done 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fad34002d00 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fad34003480 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad30005010 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.897+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fad340198a0 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(788..788 src has 704..788) v4 ==== 8557+0+0 (unknown 3623183036 0 0) 0x7fad34088080 con 0x7fad441025a0 2024-09-10T19:07:31.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 10 client.49829.objecter ms_dispatch 0x7fad44000b90 osd_map(788..788 src has 704..788) v4 2024-09-10T19:07:31.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 3 client.49829.objecter handle_osd_map got epochs [788,788] > 0 2024-09-10T19:07:31.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 3 client.49829.objecter handle_osd_map decoding full epoch 788 2024-09-10T19:07:31.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 20 client.49829.objecter dump_active .. 0 homeless 2024-09-10T19:07:31.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.901+0000 7fad2b7fe640 10 client.49829.objecter ms_handle_connect 0x7fad2c071650 2024-09-10T19:07:31.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:31.905+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fad34055220 con 0x7fad441025a0 2024-09-10T19:07:31.915 INFO:teuthology.orchestra.run.smithi083.stdout:ERROR: (22) Invalid argument 2024-09-10T19:07:31.916 INFO:teuthology.orchestra.run.smithi083.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:07:31.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:31.995 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:31.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:32.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.081+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad30000b90 con 0x7fad2c071650 2024-09-10T19:07:32.099 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:32.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.101+0000 7fad2b7fe640 1 -- 172.21.15.83:0/1182162833 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+592562 (unknown 2965378022 0 1859445736) 0x7fad30000b90 con 0x7fad2c071650 2024-09-10T19:07:32.114 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:07:32.119 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:07:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 10 librados: watch_flush enter 2024-09-10T19:07:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 10 librados: watch_flush exit 2024-09-10T19:07:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 20 client.49829.objecter shutdown clearing up homeless session... 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 10 client.49829.objecter successfully canceled tick 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 >> v1:172.21.15.83:6800/3071793735 conn(0x7fad2c071650 legacy=0x7fad2c073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 >> v1:172.21.15.83:6790/0 conn(0x7fad441025a0 legacy=0x7fad441014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 shutdown_connections 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.105+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 >> 172.21.15.83:0/1182162833 conn(0x7fad440f81a0 msgr2=0x7fad44109370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.109+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 shutdown_connections 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.109+0000 7fad494ab640 1 -- 172.21.15.83:0/1182162833 wait complete. 2024-09-10T19:07:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:32.109+0000 7fad494ab640 1 librados: shutdown 2024-09-10T19:07:32.135 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:32.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:32.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:32.325 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:32.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:32.410 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:32.514 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:32.550 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:32 smithi165 bash[22591]: cluster 2024-09-10T19:07:30.867827+0000 osd.3 (osd.3) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:32 smithi165 bash[22591]: cluster 2024-09-10T19:07:30.868310+0000 osd.3 (osd.3) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:32 smithi165 bash[22591]: cluster 2024-09-10T19:07:31.460029+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-09-10T19:07:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:32 smithi165 bash[22591]: audit 2024-09-10T19:07:32.085244+0000 mgr.y (mgr.24526) 715 : audit [DBG] from='client.49829 v1:172.21.15.83:0/1182162833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:32.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:32 smithi165 bash[22591]: cluster 2024-09-10T19:07:32.350361+0000 mgr.y (mgr.24526) 716 : cluster [DBG] pgmap v1220: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-09-10T19:07:32.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:32.687 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:32.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:32] "GET /metrics HTTP/1.1" 200 229033 "" "Prometheus/2.43.0" 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[32381]: cluster 2024-09-10T19:07:30.867827+0000 osd.3 (osd.3) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[32381]: cluster 2024-09-10T19:07:30.868310+0000 osd.3 (osd.3) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[32381]: cluster 2024-09-10T19:07:31.460029+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[32381]: audit 2024-09-10T19:07:32.085244+0000 mgr.y (mgr.24526) 715 : audit [DBG] from='client.49829 v1:172.21.15.83:0/1182162833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[32381]: cluster 2024-09-10T19:07:32.350361+0000 mgr.y (mgr.24526) 716 : cluster [DBG] pgmap v1220: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-09-10T19:07:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18629]: cluster 2024-09-10T19:07:30.867827+0000 osd.3 (osd.3) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18629]: cluster 2024-09-10T19:07:30.868310+0000 osd.3 (osd.3) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18629]: cluster 2024-09-10T19:07:31.460029+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-09-10T19:07:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18629]: audit 2024-09-10T19:07:32.085244+0000 mgr.y (mgr.24526) 715 : audit [DBG] from='client.49829 v1:172.21.15.83:0/1182162833' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:07:32.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:32 smithi083 bash[18629]: cluster 2024-09-10T19:07:32.350361+0000 mgr.y (mgr.24526) 716 : cluster [DBG] pgmap v1220: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-09-10T19:07:32.793 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:32.828 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:32.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:32.967 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:32.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:33.071 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:33.108 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:33.211 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:33.243 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:33.347 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:33.380 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:33.484 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:33.518 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:33.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:33.656 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:33.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:33.794 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:33.898 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:33.933 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:33.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:34.037 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:34.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:07:34.072 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:34.176 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:34.205 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:07:34.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:07:34.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:34.344 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:34.425 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":1219,"stamp":"2024-09-10T19:07:31.454702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4645518,"num_objects":216,"num_object_clones":1,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":2263,"num_read_kb":2002,"num_write":767,"num_write_kb":4038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1867,"ondisk_log_size":1867,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":642,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2434848,"kb_used_data":26868,"kb_used_omap":13,"kb_used_meta":2407858,"kb_avail":747526368,"statfs":{"total":767960285184,"available":765467000832,"internally_reserved":0,"allocated":27512832,"data_stored":15096703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2465647296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":49236,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.054767"},"pg_stats":[{"pgid":"256.1f","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753175+0000","last_change":"2024-09-10T19:07:20.384909+0000","last_active":"2024-09-10T19:07:27.753175+0000","last_peered":"2024-09-10T19:07:27.753175+0000","last_clean":"2024-09-10T19:07:27.753175+0000","last_became_active":"2024-09-10T19:07:20.384608+0000","last_became_peered":"2024-09-10T19:07:20.384608+0000","last_unstale":"2024-09-10T19:07:27.753175+0000","last_undegraded":"2024-09-10T19:07:27.753175+0000","last_fullsized":"2024-09-10T19:07:27.753175+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:28.976319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"256.1e","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419412+0000","last_change":"2024-09-10T19:07:20.386683+0000","last_active":"2024-09-10T19:07:27.419412+0000","last_peered":"2024-09-10T19:07:27.419412+0000","last_clean":"2024-09-10T19:07:27.419412+0000","last_became_active":"2024-09-10T19:07:20.386356+0000","last_became_peered":"2024-09-10T19:07:20.386356+0000","last_unstale":"2024-09-10T19:07:27.419412+0000","last_undegraded":"2024-09-10T19:07:27.419412+0000","last_fullsized":"2024-09-10T19:07:27.419412+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:41.446585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"256.1d","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755063+0000","last_change":"2024-09-10T19:07:20.385446+0000","last_active":"2024-09-10T19:07:27.755063+0000","last_peered":"2024-09-10T19:07:27.755063+0000","last_clean":"2024-09-10T19:07:27.755063+0000","last_became_active":"2024-09-10T19:07:20.384810+0000","last_became_peered":"2024-09-10T19:07:20.384810+0000","last_unstale":"2024-09-10T19:07:27.755063+0000","last_undegraded":"2024-09-10T19:07:27.755063+0000","last_fullsized":"2024-09-10T19:07:27.755063+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:47.021565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"256.1c","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690331+0000","last_change":"2024-09-10T19:07:20.383347+0000","last_active":"2024-09-10T19:07:27.690331+0000","last_peered":"2024-09-10T19:07:27.690331+0000","last_clean":"2024-09-10T19:07:27.690331+0000","last_became_active":"2024-09-10T19:07:20.381987+0000","last_became_peered":"2024-09-10T19:07:20.381987+0000","last_unstale":"2024-09-10T19:07:27.690331+0000","last_undegraded":"2024-09-10T19:07:27.690331+0000","last_fullsized":"2024-09-10T19:07:27.690331+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:28.722067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.1b","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690123+0000","last_change":"2024-09-10T19:07:20.383965+0000","last_active":"2024-09-10T19:07:27.690123+0000","last_peered":"2024-09-10T19:07:27.690123+0000","last_clean":"2024-09-10T19:07:27.690123+0000","last_became_active":"2024-09-10T19:07:20.383613+0000","last_became_peered":"2024-09-10T19:07:20.383613+0000","last_unstale":"2024-09-10T19:07:27.690123+0000","last_undegraded":"2024-09-10T19:07:27.690123+0000","last_fullsized":"2024-09-10T19:07:27.690123+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:27.670955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.1a","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689885+0000","last_change":"2024-09-10T19:07:20.387954+0000","last_active":"2024-09-10T19:07:27.689885+0000","last_peered":"2024-09-10T19:07:27.689885+0000","last_clean":"2024-09-10T19:07:27.689885+0000","last_became_active":"2024-09-10T19:07:20.387654+0000","last_became_peered":"2024-09-10T19:07:20.387654+0000","last_unstale":"2024-09-10T19:07:27.689885+0000","last_undegraded":"2024-09-10T19:07:27.689885+0000","last_fullsized":"2024-09-10T19:07:27.689885+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:45.395412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.19","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689805+0000","last_change":"2024-09-10T19:07:20.378514+0000","last_active":"2024-09-10T19:07:27.689805+0000","last_peered":"2024-09-10T19:07:27.689805+0000","last_clean":"2024-09-10T19:07:27.689805+0000","last_became_active":"2024-09-10T19:07:20.378043+0000","last_became_peered":"2024-09-10T19:07:20.378043+0000","last_unstale":"2024-09-10T19:07:27.689805+0000","last_undegraded":"2024-09-10T19:07:27.689805+0000","last_fullsized":"2024-09-10T19:07:27.689805+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:42.255102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.18","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.417572+0000","last_change":"2024-09-10T19:07:20.380287+0000","last_active":"2024-09-10T19:07:27.417572+0000","last_peered":"2024-09-10T19:07:27.417572+0000","last_clean":"2024-09-10T19:07:27.417572+0000","last_became_active":"2024-09-10T19:07:20.380006+0000","last_became_peered":"2024-09-10T19:07:20.380006+0000","last_unstale":"2024-09-10T19:07:27.417572+0000","last_undegraded":"2024-09-10T19:07:27.417572+0000","last_fullsized":"2024-09-10T19:07:27.417572+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:29.358344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"256.17","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753190+0000","last_change":"2024-09-10T19:07:20.385412+0000","last_active":"2024-09-10T19:07:27.753190+0000","last_peered":"2024-09-10T19:07:27.753190+0000","last_clean":"2024-09-10T19:07:27.753190+0000","last_became_active":"2024-09-10T19:07:20.384467+0000","last_became_peered":"2024-09-10T19:07:20.384467+0000","last_unstale":"2024-09-10T19:07:27.753190+0000","last_undegraded":"2024-09-10T19:07:27.753190+0000","last_fullsized":"2024-09-10T19:07:27.753190+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:31.333502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"256.16","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314358+0000","last_change":"2024-09-10T19:07:20.392825+0000","last_active":"2024-09-10T19:07:27.314358+0000","last_peered":"2024-09-10T19:07:27.314358+0000","last_clean":"2024-09-10T19:07:27.314358+0000","last_became_active":"2024-09-10T19:07:20.392540+0000","last_became_peered":"2024-09-10T19:07:20.392540+0000","last_unstale":"2024-09-10T19:07:27.314358+0000","last_undegraded":"2024-09-10T19:07:27.314358+0000","last_fullsized":"2024-09-10T19:07:27.314358+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:27.121199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"256.15","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755237+0000","last_change":"2024-09-10T19:07:20.385624+0000","last_active":"2024-09-10T19:07:27.755237+0000","last_peered":"2024-09-10T19:07:27.755237+0000","last_clean":"2024-09-10T19:07:27.755237+0000","last_became_active":"2024-09-10T19:07:20.385182+0000","last_became_peered":"2024-09-10T19:07:20.385182+0000","last_unstale":"2024-09-10T19:07:27.755237+0000","last_undegraded":"2024-09-10T19:07:27.755237+0000","last_fullsized":"2024-09-10T19:07:27.755237+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:34.230174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"256.14","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690543+0000","last_change":"2024-09-10T19:07:20.384294+0000","last_active":"2024-09-10T19:07:27.690543+0000","last_peered":"2024-09-10T19:07:27.690543+0000","last_clean":"2024-09-10T19:07:27.690543+0000","last_became_active":"2024-09-10T19:07:20.382687+0000","last_became_peered":"2024-09-10T19:07:20.382687+0000","last_unstale":"2024-09-10T19:07:27.690543+0000","last_undegraded":"2024-09-10T19:07:27.690543+0000","last_fullsized":"2024-09-10T19:07:27.690543+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:26.514838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.13","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690166+0000","last_change":"2024-09-10T19:07:20.383313+0000","last_active":"2024-09-10T19:07:27.690166+0000","last_peered":"2024-09-10T19:07:27.690166+0000","last_clean":"2024-09-10T19:07:27.690166+0000","last_became_active":"2024-09-10T19:07:20.382917+0000","last_became_peered":"2024-09-10T19:07:20.382917+0000","last_unstale":"2024-09-10T19:07:27.690166+0000","last_undegraded":"2024-09-10T19:07:27.690166+0000","last_fullsized":"2024-09-10T19:07:27.690166+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:48.769307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.12","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428487+0000","last_change":"2024-09-10T19:07:20.377839+0000","last_active":"2024-09-10T19:07:26.428487+0000","last_peered":"2024-09-10T19:07:26.428487+0000","last_clean":"2024-09-10T19:07:26.428487+0000","last_became_active":"2024-09-10T19:07:20.377469+0000","last_became_peered":"2024-09-10T19:07:20.377469+0000","last_unstale":"2024-09-10T19:07:26.428487+0000","last_undegraded":"2024-09-10T19:07:26.428487+0000","last_fullsized":"2024-09-10T19:07:26.428487+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:20.474447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"256.11","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425026+0000","last_change":"2024-09-10T19:07:20.380296+0000","last_active":"2024-09-10T19:07:26.425026+0000","last_peered":"2024-09-10T19:07:26.425026+0000","last_clean":"2024-09-10T19:07:26.425026+0000","last_became_active":"2024-09-10T19:07:20.380003+0000","last_became_peered":"2024-09-10T19:07:20.380003+0000","last_unstale":"2024-09-10T19:07:26.425026+0000","last_undegraded":"2024-09-10T19:07:26.425026+0000","last_fullsized":"2024-09-10T19:07:26.425026+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:47.422644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"256.10","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.312837+0000","last_change":"2024-09-10T19:07:20.392570+0000","last_active":"2024-09-10T19:07:27.312837+0000","last_peered":"2024-09-10T19:07:27.312837+0000","last_clean":"2024-09-10T19:07:27.312837+0000","last_became_active":"2024-09-10T19:07:20.389001+0000","last_became_peered":"2024-09-10T19:07:20.389001+0000","last_unstale":"2024-09-10T19:07:27.312837+0000","last_undegraded":"2024-09-10T19:07:27.312837+0000","last_fullsized":"2024-09-10T19:07:27.312837+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:28.488444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"256.f","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691283+0000","last_change":"2024-09-10T19:07:20.383654+0000","last_active":"2024-09-10T19:07:27.691283+0000","last_peered":"2024-09-10T19:07:27.691283+0000","last_clean":"2024-09-10T19:07:27.691283+0000","last_became_active":"2024-09-10T19:07:20.383360+0000","last_became_peered":"2024-09-10T19:07:20.383360+0000","last_unstale":"2024-09-10T19:07:27.691283+0000","last_undegraded":"2024-09-10T19:07:27.691283+0000","last_fullsized":"2024-09-10T19:07:27.691283+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:32.721458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.e","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419379+0000","last_change":"2024-09-10T19:07:20.387223+0000","last_active":"2024-09-10T19:07:27.419379+0000","last_peered":"2024-09-10T19:07:27.419379+0000","last_clean":"2024-09-10T19:07:27.419379+0000","last_became_active":"2024-09-10T19:07:20.386978+0000","last_became_peered":"2024-09-10T19:07:20.386978+0000","last_unstale":"2024-09-10T19:07:27.419379+0000","last_undegraded":"2024-09-10T19:07:27.419379+0000","last_fullsized":"2024-09-10T19:07:27.419379+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:39.665369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"256.d","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752462+0000","last_change":"2024-09-10T19:07:20.388803+0000","last_active":"2024-09-10T19:07:27.752462+0000","last_peered":"2024-09-10T19:07:27.752462+0000","last_clean":"2024-09-10T19:07:27.752462+0000","last_became_active":"2024-09-10T19:07:20.388491+0000","last_became_peered":"2024-09-10T19:07:20.388491+0000","last_unstale":"2024-09-10T19:07:27.752462+0000","last_undegraded":"2024-09-10T19:07:27.752462+0000","last_fullsized":"2024-09-10T19:07:27.752462+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:35.026187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"256.c","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427575+0000","last_change":"2024-09-10T19:07:20.389212+0000","last_active":"2024-09-10T19:07:26.427575+0000","last_peered":"2024-09-10T19:07:26.427575+0000","last_clean":"2024-09-10T19:07:26.427575+0000","last_became_active":"2024-09-10T19:07:20.388921+0000","last_became_peered":"2024-09-10T19:07:20.388921+0000","last_unstale":"2024-09-10T19:07:26.427575+0000","last_undegraded":"2024-09-10T19:07:26.427575+0000","last_fullsized":"2024-09-10T19:07:26.427575+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:38.602676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"256.b","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313755+0000","last_change":"2024-09-10T19:07:20.386491+0000","last_active":"2024-09-10T19:07:27.313755+0000","last_peered":"2024-09-10T19:07:27.313755+0000","last_clean":"2024-09-10T19:07:27.313755+0000","last_became_active":"2024-09-10T19:07:20.386132+0000","last_became_peered":"2024-09-10T19:07:20.386132+0000","last_unstale":"2024-09-10T19:07:27.313755+0000","last_undegraded":"2024-09-10T19:07:27.313755+0000","last_fullsized":"2024-09-10T19:07:27.313755+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:42.948633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"256.a","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751172+0000","last_change":"2024-09-10T19:07:20.384195+0000","last_active":"2024-09-10T19:07:27.751172+0000","last_peered":"2024-09-10T19:07:27.751172+0000","last_clean":"2024-09-10T19:07:27.751172+0000","last_became_active":"2024-09-10T19:07:20.383892+0000","last_became_peered":"2024-09-10T19:07:20.383892+0000","last_unstale":"2024-09-10T19:07:27.751172+0000","last_undegraded":"2024-09-10T19:07:27.751172+0000","last_fullsized":"2024-09-10T19:07:27.751172+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:47.850050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"256.9","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.417716+0000","last_change":"2024-09-10T19:07:20.382141+0000","last_active":"2024-09-10T19:07:27.417716+0000","last_peered":"2024-09-10T19:07:27.417716+0000","last_clean":"2024-09-10T19:07:27.417716+0000","last_became_active":"2024-09-10T19:07:20.381429+0000","last_became_peered":"2024-09-10T19:07:20.381429+0000","last_unstale":"2024-09-10T19:07:27.417716+0000","last_undegraded":"2024-09-10T19:07:27.417716+0000","last_fullsized":"2024-09-10T19:07:27.417716+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:26.564864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"256.8","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689409+0000","last_change":"2024-09-10T19:07:20.392072+0000","last_active":"2024-09-10T19:07:27.689409+0000","last_peered":"2024-09-10T19:07:27.689409+0000","last_clean":"2024-09-10T19:07:27.689409+0000","last_became_active":"2024-09-10T19:07:20.391769+0000","last_became_peered":"2024-09-10T19:07:20.391769+0000","last_unstale":"2024-09-10T19:07:27.689409+0000","last_undegraded":"2024-09-10T19:07:27.689409+0000","last_fullsized":"2024-09-10T19:07:27.689409+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:37.571315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"256.7","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428732+0000","last_change":"2024-09-10T19:07:20.387956+0000","last_active":"2024-09-10T19:07:26.428732+0000","last_peered":"2024-09-10T19:07:26.428732+0000","last_clean":"2024-09-10T19:07:26.428732+0000","last_became_active":"2024-09-10T19:07:20.387619+0000","last_became_peered":"2024-09-10T19:07:20.387619+0000","last_unstale":"2024-09-10T19:07:26.428732+0000","last_undegraded":"2024-09-10T19:07:26.428732+0000","last_fullsized":"2024-09-10T19:07:26.428732+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:40.371560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"256.6","version":"787'7","reported_seq":23,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-09-10T19:07:29.426501+0000","last_change":"2024-09-10T19:07:20.390941+0000","last_active":"2024-09-10T19:07:29.426501+0000","last_peered":"2024-09-10T19:07:29.426501+0000","last_clean":"2024-09-10T19:07:29.426501+0000","last_became_active":"2024-09-10T19:07:20.390578+0000","last_became_peered":"2024-09-10T19:07:20.390578+0000","last_unstale":"2024-09-10T19:07:29.426501+0000","last_undegraded":"2024-09-10T19:07:29.426501+0000","last_fullsized":"2024-09-10T19:07:29.426501+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:34.996684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49236,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"256.5","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428019+0000","last_change":"2024-09-10T19:07:20.384056+0000","last_active":"2024-09-10T19:07:26.428019+0000","last_peered":"2024-09-10T19:07:26.428019+0000","last_clean":"2024-09-10T19:07:26.428019+0000","last_became_active":"2024-09-10T19:07:20.383730+0000","last_became_peered":"2024-09-10T19:07:20.383730+0000","last_unstale":"2024-09-10T19:07:26.428019+0000","last_undegraded":"2024-09-10T19:07:26.428019+0000","last_fullsized":"2024-09-10T19:07:26.428019+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:25.208512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"256.4","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314350+0000","last_change":"2024-09-10T19:07:20.393101+0000","last_active":"2024-09-10T19:07:27.314350+0000","last_peered":"2024-09-10T19:07:27.314350+0000","last_clean":"2024-09-10T19:07:27.314350+0000","last_became_active":"2024-09-10T19:07:20.392721+0000","last_became_peered":"2024-09-10T19:07:20.392721+0000","last_unstale":"2024-09-10T19:07:27.314350+0000","last_undegraded":"2024-09-10T19:07:27.314350+0000","last_fullsized":"2024-09-10T19:07:27.314350+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:37.999791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"256.3","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.418265+0000","last_change":"2024-09-10T19:07:20.382523+0000","last_active":"2024-09-10T19:07:27.418265+0000","last_peered":"2024-09-10T19:07:27.418265+0000","last_clean":"2024-09-10T19:07:27.418265+0000","last_became_active":"2024-09-10T19:07:20.381221+0000","last_became_peered":"2024-09-10T19:07:20.381221+0000","last_unstale":"2024-09-10T19:07:27.418265+0000","last_undegraded":"2024-09-10T19:07:27.418265+0000","last_fullsized":"2024-09-10T19:07:27.418265+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:40.834756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"256.2","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754415+0000","last_change":"2024-09-10T19:07:20.384465+0000","last_active":"2024-09-10T19:07:27.754415+0000","last_peered":"2024-09-10T19:07:27.754415+0000","last_clean":"2024-09-10T19:07:27.754415+0000","last_became_active":"2024-09-10T19:07:20.384144+0000","last_became_peered":"2024-09-10T19:07:20.384144+0000","last_unstale":"2024-09-10T19:07:27.754415+0000","last_undegraded":"2024-09-10T19:07:27.754415+0000","last_fullsized":"2024-09-10T19:07:27.754415+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.910666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"256.1","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751701+0000","last_change":"2024-09-10T19:07:20.377377+0000","last_active":"2024-09-10T19:07:27.751701+0000","last_peered":"2024-09-10T19:07:27.751701+0000","last_clean":"2024-09-10T19:07:27.751701+0000","last_became_active":"2024-09-10T19:07:20.377058+0000","last_became_peered":"2024-09-10T19:07:20.377058+0000","last_unstale":"2024-09-10T19:07:27.751701+0000","last_undegraded":"2024-09-10T19:07:27.751701+0000","last_fullsized":"2024-09-10T19:07:27.751701+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:43.115629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"256.0","version":"0'0","reported_seq":16,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.312905+0000","last_change":"2024-09-10T19:07:20.392537+0000","last_active":"2024-09-10T19:07:27.312905+0000","last_peered":"2024-09-10T19:07:27.312905+0000","last_clean":"2024-09-10T19:07:27.312905+0000","last_became_active":"2024-09-10T19:07:20.387956+0000","last_became_peered":"2024-09-10T19:07:20.387956+0000","last_unstale":"2024-09-10T19:07:27.312905+0000","last_undegraded":"2024-09-10T19:07:27.312905+0000","last_fullsized":"2024-09-10T19:07:27.312905+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":779,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:07:19.352703+0000","last_clean_scrub_stamp":"2024-09-10T19:07:19.352703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:32.293080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.7","version":"767'7","reported_seq":178,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753340+0000","last_change":"2024-09-10T19:06:50.360648+0000","last_active":"2024-09-10T19:07:27.753340+0000","last_peered":"2024-09-10T19:07:27.753340+0000","last_clean":"2024-09-10T19:07:27.753340+0000","last_became_active":"2024-09-10T19:04:57.320799+0000","last_became_peered":"2024-09-10T19:04:57.320799+0000","last_unstale":"2024-09-10T19:07:27.753340+0000","last_undegraded":"2024-09-10T19:07:27.753340+0000","last_fullsized":"2024-09-10T19:07:27.753340+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'6","last_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_deep_scrub":"730'6","last_deep_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.006586+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:14.953341+0000","scrub_duration":0.004755039,"objects_trimmed":0,"snaptrim_duration":0.00095805300000000005,"stat_sum":{"num_bytes":4000013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"241.6","version":"738'9","reported_seq":185,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428594+0000","last_change":"2024-09-10T19:06:50.365365+0000","last_active":"2024-09-10T19:07:26.428594+0000","last_peered":"2024-09-10T19:07:26.428594+0000","last_clean":"2024-09-10T19:07:26.428594+0000","last_became_active":"2024-09-10T19:04:57.324770+0000","last_became_peered":"2024-09-10T19:04:57.324770+0000","last_unstale":"2024-09-10T19:07:26.428594+0000","last_undegraded":"2024-09-10T19:07:26.428594+0000","last_fullsized":"2024-09-10T19:07:26.428594+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'9","last_scrub_stamp":"2024-09-10T19:06:49.172954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:49.172954+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.846639+0000","scrub_duration":0.0043854150000000001,"objects_trimmed":0,"snaptrim_duration":0.00099442499999999995,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"241.5","version":"720'6","reported_seq":178,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419213+0000","last_change":"2024-09-10T19:06:52.119235+0000","last_active":"2024-09-10T19:07:27.419213+0000","last_peered":"2024-09-10T19:07:27.419213+0000","last_clean":"2024-09-10T19:07:27.419213+0000","last_became_active":"2024-09-10T19:04:57.322094+0000","last_became_peered":"2024-09-10T19:04:57.322094+0000","last_unstale":"2024-09-10T19:07:27.419213+0000","last_undegraded":"2024-09-10T19:07:27.419213+0000","last_fullsized":"2024-09-10T19:07:27.419213+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"720'6","last_scrub_stamp":"2024-09-10T19:06:52.119058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.119058+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:18.898465+0000","scrub_duration":0.0043290079999999996,"objects_trimmed":0,"snaptrim_duration":0.0010204319999999999,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"241.4","version":"759'11","reported_seq":182,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313945+0000","last_change":"2024-09-10T19:06:50.352548+0000","last_active":"2024-09-10T19:07:27.313945+0000","last_peered":"2024-09-10T19:07:27.313945+0000","last_clean":"2024-09-10T19:07:27.313945+0000","last_became_active":"2024-09-10T19:04:57.322911+0000","last_became_peered":"2024-09-10T19:04:57.322911+0000","last_unstale":"2024-09-10T19:07:27.313945+0000","last_undegraded":"2024-09-10T19:07:27.313945+0000","last_fullsized":"2024-09-10T19:07:27.313945+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'6","last_scrub_stamp":"2024-09-10T19:06:39.141613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.141613+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.735457+0000","scrub_duration":0.0042889529999999999,"objects_trimmed":0,"snaptrim_duration":0.0017454879999999999,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.3","version":"712'10","reported_seq":180,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424484+0000","last_change":"2024-09-10T19:06:50.345918+0000","last_active":"2024-09-10T19:07:26.424484+0000","last_peered":"2024-09-10T19:07:26.424484+0000","last_clean":"2024-09-10T19:07:26.424484+0000","last_became_active":"2024-09-10T19:04:57.324250+0000","last_became_peered":"2024-09-10T19:04:57.324250+0000","last_unstale":"2024-09-10T19:07:26.424484+0000","last_undegraded":"2024-09-10T19:07:26.424484+0000","last_fullsized":"2024-09-10T19:07:26.424484+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'10","last_scrub_stamp":"2024-09-10T19:06:27.862802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:27.862802+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.207401+0000","scrub_duration":0.0041735690000000002,"objects_trimmed":0,"snaptrim_duration":0.0036847949999999998,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"241.2","version":"712'11","reported_seq":185,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424177+0000","last_change":"2024-09-10T19:06:50.344565+0000","last_active":"2024-09-10T19:07:26.424177+0000","last_peered":"2024-09-10T19:07:26.424177+0000","last_clean":"2024-09-10T19:07:26.424177+0000","last_became_active":"2024-09-10T19:04:57.322254+0000","last_became_peered":"2024-09-10T19:04:57.322254+0000","last_unstale":"2024-09-10T19:07:26.424177+0000","last_undegraded":"2024-09-10T19:07:26.424177+0000","last_fullsized":"2024-09-10T19:07:26.424177+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"712'11","last_scrub_stamp":"2024-09-10T19:06:45.656802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.656802+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.262287+0000","scrub_duration":0.010299141,"objects_trimmed":0,"snaptrim_duration":0.0028767139999999998,"stat_sum":{"num_bytes":10,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"241.1","version":"751'15","reported_seq":180,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754079+0000","last_change":"2024-09-10T19:06:52.359146+0000","last_active":"2024-09-10T19:07:27.754079+0000","last_peered":"2024-09-10T19:07:27.754079+0000","last_clean":"2024-09-10T19:07:27.754079+0000","last_became_active":"2024-09-10T19:04:57.329766+0000","last_became_peered":"2024-09-10T19:04:57.329766+0000","last_unstale":"2024-09-10T19:07:27.754079+0000","last_undegraded":"2024-09-10T19:07:27.754079+0000","last_fullsized":"2024-09-10T19:07:27.754079+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:04:56.297048+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52769914500000004,"stat_sum":{"num_bytes":10,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"241.0","version":"662'1","reported_seq":172,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313052+0000","last_change":"2024-09-10T19:06:50.345068+0000","last_active":"2024-09-10T19:07:27.313052+0000","last_peered":"2024-09-10T19:07:27.313052+0000","last_clean":"2024-09-10T19:07:27.313052+0000","last_became_active":"2024-09-10T19:04:57.320088+0000","last_became_peered":"2024-09-10T19:04:57.320088+0000","last_unstale":"2024-09-10T19:07:27.313052+0000","last_undegraded":"2024-09-10T19:07:27.313052+0000","last_fullsized":"2024-09-10T19:07:27.313052+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'1","last_scrub_stamp":"2024-09-10T19:06:36.131427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.131427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.839793+0000","scrub_duration":0.0044912399999999996,"objects_trimmed":0,"snaptrim_duration":0.0018060820000000001,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":780,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426257+0000","last_change":"2024-09-10T19:05:29.898575+0000","last_active":"2024-09-10T19:07:26.426257+0000","last_peered":"2024-09-10T19:07:26.426257+0000","last_clean":"2024-09-10T19:07:26.426257+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:07:26.426257+0000","last_undegraded":"2024-09-10T19:07:26.426257+0000","last_fullsized":"2024-09-10T19:07:26.426257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_clean_scrub_stamp":"2024-09-10T19:05:29.898480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030543649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":790,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.756147+0000","last_change":"2024-09-10T19:05:27.220506+0000","last_active":"2024-09-10T19:07:27.756147+0000","last_peered":"2024-09-10T19:07:27.756147+0000","last_clean":"2024-09-10T19:07:27.756147+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:07:27.756147+0000","last_undegraded":"2024-09-10T19:07:27.756147+0000","last_fullsized":"2024-09-10T19:07:27.756147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:27.220358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:27.220358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037569050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":814,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426985+0000","last_change":"2024-09-10T19:06:29.097504+0000","last_active":"2024-09-10T19:07:26.426985+0000","last_peered":"2024-09-10T19:07:26.426985+0000","last_clean":"2024-09-10T19:07:26.426985+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:07:26.426985+0000","last_undegraded":"2024-09-10T19:07:26.426985+0000","last_fullsized":"2024-09-10T19:07:26.426985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:06:29.097329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.097329+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.402785+0000","scrub_duration":0.0048839570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":792,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427056+0000","last_change":"2024-09-10T19:05:10.990324+0000","last_active":"2024-09-10T19:07:26.427056+0000","last_peered":"2024-09-10T19:07:26.427056+0000","last_clean":"2024-09-10T19:07:26.427056+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:07:26.427056+0000","last_undegraded":"2024-09-10T19:07:26.427056+0000","last_fullsized":"2024-09-10T19:07:26.427056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.990119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.990119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003497137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":709,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751168+0000","last_change":"2024-09-10T19:06:35.862278+0000","last_active":"2024-09-10T19:07:27.751168+0000","last_peered":"2024-09-10T19:07:27.751168+0000","last_clean":"2024-09-10T19:07:27.751168+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:07:27.751168+0000","last_undegraded":"2024-09-10T19:07:27.751168+0000","last_fullsized":"2024-09-10T19:07:27.751168+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.862120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.862120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.659615+0000","scrub_duration":0.0026679030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":796,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427476+0000","last_change":"2024-09-10T19:05:07.989627+0000","last_active":"2024-09-10T19:07:26.427476+0000","last_peered":"2024-09-10T19:07:26.427476+0000","last_clean":"2024-09-10T19:07:26.427476+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:07:26.427476+0000","last_undegraded":"2024-09-10T19:07:26.427476+0000","last_fullsized":"2024-09-10T19:07:26.427476+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:07.989469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:07.989469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00341515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":819,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753008+0000","last_change":"2024-09-10T19:06:23.367532+0000","last_active":"2024-09-10T19:07:27.753008+0000","last_peered":"2024-09-10T19:07:27.753008+0000","last_clean":"2024-09-10T19:07:27.753008+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:07:27.753008+0000","last_undegraded":"2024-09-10T19:07:27.753008+0000","last_fullsized":"2024-09-10T19:07:27.753008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:23.367353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.367353+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:38.115423+0000","scrub_duration":0.0057060319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":807,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.423879+0000","last_change":"2024-09-10T19:06:22.007353+0000","last_active":"2024-09-10T19:07:26.423879+0000","last_peered":"2024-09-10T19:07:26.423879+0000","last_clean":"2024-09-10T19:07:26.423879+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:07:26.423879+0000","last_undegraded":"2024-09-10T19:07:26.423879+0000","last_fullsized":"2024-09-10T19:07:26.423879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.007181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.007181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.778494+0000","scrub_duration":0.0034040429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":266,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751575+0000","last_change":"2024-09-10T19:06:16.980981+0000","last_active":"2024-09-10T19:07:27.751575+0000","last_peered":"2024-09-10T19:07:27.751575+0000","last_clean":"2024-09-10T19:07:27.751575+0000","last_became_active":"2024-09-10T19:02:52.254895+0000","last_became_peered":"2024-09-10T19:02:52.254895+0000","last_unstale":"2024-09-10T19:07:27.751575+0000","last_undegraded":"2024-09-10T19:07:27.751575+0000","last_fullsized":"2024-09-10T19:07:27.751575+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:16.980824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:16.980824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.357064+0000","scrub_duration":0.0034584770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.756136+0000","last_change":"2024-09-10T19:06:43.403406+0000","last_active":"2024-09-10T19:07:27.756136+0000","last_peered":"2024-09-10T19:07:27.756136+0000","last_clean":"2024-09-10T19:07:27.756136+0000","last_became_active":"2024-09-10T18:54:37.631759+0000","last_became_peered":"2024-09-10T18:54:37.631759+0000","last_unstale":"2024-09-10T19:07:27.756136+0000","last_undegraded":"2024-09-10T19:07:27.756136+0000","last_fullsized":"2024-09-10T19:07:27.756136+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:43.403297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.403297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:11.191457+0000","scrub_duration":0.0062412609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":799,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419228+0000","last_change":"2024-09-10T19:06:37.935702+0000","last_active":"2024-09-10T19:07:27.419228+0000","last_peered":"2024-09-10T19:07:27.419228+0000","last_clean":"2024-09-10T19:07:27.419228+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:07:27.419228+0000","last_undegraded":"2024-09-10T19:07:27.419228+0000","last_fullsized":"2024-09-10T19:07:27.419228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.935545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.935545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.697682+0000","scrub_duration":0.003538518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":840,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750442+0000","last_change":"2024-09-10T19:06:29.746879+0000","last_active":"2024-09-10T19:07:27.750442+0000","last_peered":"2024-09-10T19:07:27.750442+0000","last_clean":"2024-09-10T19:07:27.750442+0000","last_became_active":"2024-09-10T18:49:19.627208+0000","last_became_peered":"2024-09-10T18:49:19.627208+0000","last_unstale":"2024-09-10T19:07:27.750442+0000","last_undegraded":"2024-09-10T19:07:27.750442+0000","last_fullsized":"2024-09-10T19:07:27.750442+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:29.746751+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.746751+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.290888+0000","scrub_duration":0.0059564860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":262,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426487+0000","last_change":"2024-09-10T19:05:02.098565+0000","last_active":"2024-09-10T19:07:26.426487+0000","last_peered":"2024-09-10T19:07:26.426487+0000","last_clean":"2024-09-10T19:07:26.426487+0000","last_became_active":"2024-09-10T19:02:52.250472+0000","last_became_peered":"2024-09-10T19:02:52.250472+0000","last_unstale":"2024-09-10T19:07:26.426487+0000","last_undegraded":"2024-09-10T19:07:26.426487+0000","last_fullsized":"2024-09-10T19:07:26.426487+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.098384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.098384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034406860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":789,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751456+0000","last_change":"2024-09-10T19:06:31.820211+0000","last_active":"2024-09-10T19:07:27.751456+0000","last_peered":"2024-09-10T19:07:27.751456+0000","last_clean":"2024-09-10T19:07:27.751456+0000","last_became_active":"2024-09-10T18:49:24.055126+0000","last_became_peered":"2024-09-10T18:49:24.055126+0000","last_unstale":"2024-09-10T19:07:27.751456+0000","last_undegraded":"2024-09-10T19:07:27.751456+0000","last_fullsized":"2024-09-10T19:07:27.751456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.820051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.820051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.110063+0000","scrub_duration":0.003619667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":717,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427285+0000","last_change":"2024-09-10T19:06:36.108291+0000","last_active":"2024-09-10T19:07:26.427285+0000","last_peered":"2024-09-10T19:07:26.427285+0000","last_clean":"2024-09-10T19:07:26.427285+0000","last_became_active":"2024-09-10T18:54:37.676054+0000","last_became_peered":"2024-09-10T18:54:37.676054+0000","last_unstale":"2024-09-10T19:07:26.427285+0000","last_undegraded":"2024-09-10T19:07:26.427285+0000","last_fullsized":"2024-09-10T19:07:26.427285+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.107798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.107798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:36.908011+0000","scrub_duration":0.0058806149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":802,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419916+0000","last_change":"2024-09-10T19:06:36.900403+0000","last_active":"2024-09-10T19:07:27.419916+0000","last_peered":"2024-09-10T19:07:27.419916+0000","last_clean":"2024-09-10T19:07:27.419916+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:07:27.419916+0000","last_undegraded":"2024-09-10T19:07:27.419916+0000","last_fullsized":"2024-09-10T19:07:27.419916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.900156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.900156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.830082+0000","scrub_duration":0.0031758419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":837,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750779+0000","last_change":"2024-09-10T19:06:26.733413+0000","last_active":"2024-09-10T19:07:27.750779+0000","last_peered":"2024-09-10T19:07:27.750779+0000","last_clean":"2024-09-10T19:07:27.750779+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:07:27.750779+0000","last_undegraded":"2024-09-10T19:07:27.750779+0000","last_fullsized":"2024-09-10T19:07:27.750779+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:06:26.733241+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.733241+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.224379+0000","scrub_duration":0.0050063570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":272,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428759+0000","last_change":"2024-09-10T19:06:30.049476+0000","last_active":"2024-09-10T19:07:26.428759+0000","last_peered":"2024-09-10T19:07:26.428759+0000","last_clean":"2024-09-10T19:07:26.428759+0000","last_became_active":"2024-09-10T19:02:52.256764+0000","last_became_peered":"2024-09-10T19:02:52.256764+0000","last_unstale":"2024-09-10T19:07:26.428759+0000","last_undegraded":"2024-09-10T19:07:26.428759+0000","last_fullsized":"2024-09-10T19:07:26.428759+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.049307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.049307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.430304+0000","scrub_duration":0.0039529120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":794,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753874+0000","last_change":"2024-09-10T19:06:45.340292+0000","last_active":"2024-09-10T19:07:27.753874+0000","last_peered":"2024-09-10T19:07:27.753874+0000","last_clean":"2024-09-10T19:07:27.753874+0000","last_became_active":"2024-09-10T18:49:24.046306+0000","last_became_peered":"2024-09-10T18:49:24.046306+0000","last_unstale":"2024-09-10T19:07:27.753874+0000","last_undegraded":"2024-09-10T19:07:27.753874+0000","last_fullsized":"2024-09-10T19:07:27.753874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.340065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.340065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.926466+0000","scrub_duration":0.004677952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":704,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428675+0000","last_change":"2024-09-10T19:04:59.080471+0000","last_active":"2024-09-10T19:07:26.428675+0000","last_peered":"2024-09-10T19:07:26.428675+0000","last_clean":"2024-09-10T19:07:26.428675+0000","last_became_active":"2024-09-10T18:54:37.625870+0000","last_became_peered":"2024-09-10T18:54:37.625870+0000","last_unstale":"2024-09-10T19:07:26.428675+0000","last_undegraded":"2024-09-10T19:07:26.428675+0000","last_fullsized":"2024-09-10T19:07:26.428675+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.080362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.080362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028902760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":316,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750610+0000","last_change":"2024-09-10T19:06:58.888376+0000","last_active":"2024-09-10T19:07:27.750610+0000","last_peered":"2024-09-10T19:07:27.750610+0000","last_clean":"2024-09-10T19:07:27.750610+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:07:27.750610+0000","last_undegraded":"2024-09-10T19:07:27.750610+0000","last_fullsized":"2024-09-10T19:07:27.750610+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:58.888195+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:44.683992+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.888195+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:15.759441+0000","scrub_duration":0.0043236180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":857,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689030+0000","last_change":"2024-09-10T19:07:01.719205+0000","last_active":"2024-09-10T19:07:27.689030+0000","last_peered":"2024-09-10T19:07:27.689030+0000","last_clean":"2024-09-10T19:07:27.689030+0000","last_became_active":"2024-09-10T18:49:19.628987+0000","last_became_peered":"2024-09-10T18:49:19.628987+0000","last_unstale":"2024-09-10T19:07:27.689030+0000","last_undegraded":"2024-09-10T19:07:27.689030+0000","last_fullsized":"2024-09-10T19:07:27.689030+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:07:01.718013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.718013+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:28.914109+0000","scrub_duration":0.0078103089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420931+0000","last_change":"2024-09-10T19:07:02.278308+0000","last_active":"2024-09-10T19:07:27.420931+0000","last_peered":"2024-09-10T19:07:27.420931+0000","last_clean":"2024-09-10T19:07:27.420931+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:07:27.420931+0000","last_undegraded":"2024-09-10T19:07:27.420931+0000","last_fullsized":"2024-09-10T19:07:27.420931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:56.847674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:04:56.847674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033132719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"548'13","reported_seq":387,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314534+0000","last_change":"2024-09-10T19:05:24.947023+0000","last_active":"2024-09-10T19:07:27.314534+0000","last_peered":"2024-09-10T19:07:27.314534+0000","last_clean":"2024-09-10T19:07:27.314534+0000","last_became_active":"2024-09-10T19:02:52.256843+0000","last_became_peered":"2024-09-10T19:02:52.256843+0000","last_unstale":"2024-09-10T19:07:27.314534+0000","last_undegraded":"2024-09-10T19:07:27.314534+0000","last_fullsized":"2024-09-10T19:07:27.314534+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_deep_scrub":"548'13","last_deep_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_clean_scrub_stamp":"2024-09-10T19:05:24.946829+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036489069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":810,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314772+0000","last_change":"2024-09-10T19:06:20.119294+0000","last_active":"2024-09-10T19:07:27.314772+0000","last_peered":"2024-09-10T19:07:27.314772+0000","last_clean":"2024-09-10T19:07:27.314772+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:07:27.314772+0000","last_undegraded":"2024-09-10T19:07:27.314772+0000","last_fullsized":"2024-09-10T19:07:27.314772+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:20.119151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.119151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032863219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690816+0000","last_change":"2024-09-10T19:06:36.747901+0000","last_active":"2024-09-10T19:07:27.690816+0000","last_peered":"2024-09-10T19:07:27.690816+0000","last_clean":"2024-09-10T19:07:27.690816+0000","last_became_active":"2024-09-10T18:54:37.708175+0000","last_became_peered":"2024-09-10T18:54:37.708175+0000","last_unstale":"2024-09-10T19:07:27.690816+0000","last_undegraded":"2024-09-10T19:07:27.690816+0000","last_fullsized":"2024-09-10T19:07:27.690816+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.747814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.747814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.677402+0000","scrub_duration":0.0036046889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":867,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.418661+0000","last_change":"2024-09-10T19:05:22.653245+0000","last_active":"2024-09-10T19:07:27.418661+0000","last_peered":"2024-09-10T19:07:27.418661+0000","last_clean":"2024-09-10T19:07:27.418661+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:07:27.418661+0000","last_undegraded":"2024-09-10T19:07:27.418661+0000","last_fullsized":"2024-09-10T19:07:27.418661+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:22.653050+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:22.653050+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040316939999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":823,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426650+0000","last_change":"2024-09-10T19:05:21.852708+0000","last_active":"2024-09-10T19:07:26.426650+0000","last_peered":"2024-09-10T19:07:26.426650+0000","last_clean":"2024-09-10T19:07:26.426650+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:07:26.426650+0000","last_undegraded":"2024-09-10T19:07:26.426650+0000","last_fullsized":"2024-09-10T19:07:26.426650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:05:21.852558+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:05:21.852558+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041159459999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":811,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426846+0000","last_change":"2024-09-10T19:06:38.068653+0000","last_active":"2024-09-10T19:07:26.426846+0000","last_peered":"2024-09-10T19:07:26.426846+0000","last_clean":"2024-09-10T19:07:26.426846+0000","last_became_active":"2024-09-10T18:49:17.614379+0000","last_became_peered":"2024-09-10T18:49:17.614379+0000","last_unstale":"2024-09-10T19:07:26.426846+0000","last_undegraded":"2024-09-10T19:07:26.426846+0000","last_fullsized":"2024-09-10T19:07:26.426846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:38.068517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:06:38.068517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.453563+0000","scrub_duration":0.0031386930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":262,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427992+0000","last_change":"2024-09-10T19:05:01.085144+0000","last_active":"2024-09-10T19:07:26.427992+0000","last_peered":"2024-09-10T19:07:26.427992+0000","last_clean":"2024-09-10T19:07:26.427992+0000","last_became_active":"2024-09-10T19:02:52.254392+0000","last_became_peered":"2024-09-10T19:02:52.254392+0000","last_unstale":"2024-09-10T19:07:26.427992+0000","last_undegraded":"2024-09-10T19:07:26.427992+0000","last_fullsized":"2024-09-10T19:07:26.427992+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:01.085070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:01.085070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024364389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":270,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428868+0000","last_change":"2024-09-10T19:06:31.119974+0000","last_active":"2024-09-10T19:07:26.428868+0000","last_peered":"2024-09-10T19:07:26.428868+0000","last_clean":"2024-09-10T19:07:26.428868+0000","last_became_active":"2024-09-10T19:02:52.258431+0000","last_became_peered":"2024-09-10T19:02:52.258431+0000","last_unstale":"2024-09-10T19:07:26.428868+0000","last_undegraded":"2024-09-10T19:07:26.428868+0000","last_fullsized":"2024-09-10T19:07:26.428868+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.119813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.119813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:35.705097+0000","scrub_duration":0.0040758110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690587+0000","last_change":"2024-09-10T19:06:53.837927+0000","last_active":"2024-09-10T19:07:27.690587+0000","last_peered":"2024-09-10T19:07:27.690587+0000","last_clean":"2024-09-10T19:07:27.690587+0000","last_became_active":"2024-09-10T18:54:37.600308+0000","last_became_peered":"2024-09-10T18:54:37.600308+0000","last_unstale":"2024-09-10T19:07:27.690587+0000","last_undegraded":"2024-09-10T19:07:27.690587+0000","last_fullsized":"2024-09-10T19:07:27.690587+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.837830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.837830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.063555+0000","scrub_duration":0.0098916300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":299,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751056+0000","last_change":"2024-09-10T19:05:17.213897+0000","last_active":"2024-09-10T19:07:27.751056+0000","last_peered":"2024-09-10T19:07:27.751056+0000","last_clean":"2024-09-10T19:07:27.751056+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:07:27.751056+0000","last_undegraded":"2024-09-10T19:07:27.751056+0000","last_fullsized":"2024-09-10T19:07:27.751056+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:17.213629+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:05:17.213629+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038456739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":837,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429906+0000","last_change":"2024-09-10T19:05:33.936164+0000","last_active":"2024-09-10T19:07:26.429906+0000","last_peered":"2024-09-10T19:07:26.429906+0000","last_clean":"2024-09-10T19:07:26.429906+0000","last_became_active":"2024-09-10T18:49:19.629260+0000","last_became_peered":"2024-09-10T18:49:19.629260+0000","last_unstale":"2024-09-10T19:07:26.429906+0000","last_undegraded":"2024-09-10T19:07:26.429906+0000","last_fullsized":"2024-09-10T19:07:26.429906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:05:33.935997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.935997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050458739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":801,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429625+0000","last_change":"2024-09-10T19:06:40.095694+0000","last_active":"2024-09-10T19:07:26.429625+0000","last_peered":"2024-09-10T19:07:26.429625+0000","last_clean":"2024-09-10T19:07:26.429625+0000","last_became_active":"2024-09-10T18:49:17.613748+0000","last_became_peered":"2024-09-10T18:49:17.613748+0000","last_unstale":"2024-09-10T19:07:26.429625+0000","last_undegraded":"2024-09-10T19:07:26.429625+0000","last_fullsized":"2024-09-10T19:07:26.429625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:40.095535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:40.095535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.980205+0000","scrub_duration":0.003623636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":807,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428862+0000","last_change":"2024-09-10T19:06:59.211985+0000","last_active":"2024-09-10T19:07:26.428862+0000","last_peered":"2024-09-10T19:07:26.428862+0000","last_clean":"2024-09-10T19:07:26.428862+0000","last_became_active":"2024-09-10T18:49:22.040698+0000","last_became_peered":"2024-09-10T18:49:22.040698+0000","last_unstale":"2024-09-10T19:07:26.428862+0000","last_undegraded":"2024-09-10T19:07:26.428862+0000","last_fullsized":"2024-09-10T19:07:26.428862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:59.211830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.211830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.292523+0000","scrub_duration":0.0029718470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":796,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752475+0000","last_change":"2024-09-10T19:06:39.931182+0000","last_active":"2024-09-10T19:07:27.752475+0000","last_peered":"2024-09-10T19:07:27.752475+0000","last_clean":"2024-09-10T19:07:27.752475+0000","last_became_active":"2024-09-10T18:49:24.052198+0000","last_became_peered":"2024-09-10T18:49:24.052198+0000","last_unstale":"2024-09-10T19:07:27.752475+0000","last_undegraded":"2024-09-10T19:07:27.752475+0000","last_fullsized":"2024-09-10T19:07:27.752475+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:39.931027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.931027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:47.756981+0000","scrub_duration":0.0039840149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":717,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428897+0000","last_change":"2024-09-10T19:06:27.078822+0000","last_active":"2024-09-10T19:07:26.428897+0000","last_peered":"2024-09-10T19:07:26.428897+0000","last_clean":"2024-09-10T19:07:26.428897+0000","last_became_active":"2024-09-10T18:54:37.595043+0000","last_became_peered":"2024-09-10T18:54:37.595043+0000","last_unstale":"2024-09-10T19:07:26.428897+0000","last_undegraded":"2024-09-10T19:07:26.428897+0000","last_fullsized":"2024-09-10T19:07:26.428897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:27.078734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:27.078734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032307350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":846,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424334+0000","last_change":"2024-09-10T19:06:38.692442+0000","last_active":"2024-09-10T19:07:26.424334+0000","last_peered":"2024-09-10T19:07:26.424334+0000","last_clean":"2024-09-10T19:07:26.424334+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:07:26.424334+0000","last_undegraded":"2024-09-10T19:07:26.424334+0000","last_fullsized":"2024-09-10T19:07:26.424334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'23","last_scrub_stamp":"2024-09-10T19:06:38.692261+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T19:06:38.692261+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:53.304601+0000","scrub_duration":0.0046551179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":799,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420947+0000","last_change":"2024-09-10T19:05:15.605213+0000","last_active":"2024-09-10T19:07:27.420947+0000","last_peered":"2024-09-10T19:07:27.420947+0000","last_clean":"2024-09-10T19:07:27.420947+0000","last_became_active":"2024-09-10T18:49:17.608936+0000","last_became_peered":"2024-09-10T18:49:17.608936+0000","last_unstale":"2024-09-10T19:07:27.420947+0000","last_undegraded":"2024-09-10T19:07:27.420947+0000","last_fullsized":"2024-09-10T19:07:27.420947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.605072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.605072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072968429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"548'13","reported_seq":378,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691400+0000","last_change":"2024-09-10T19:05:04.477156+0000","last_active":"2024-09-10T19:07:27.691400+0000","last_peered":"2024-09-10T19:07:27.691400+0000","last_clean":"2024-09-10T19:07:27.691400+0000","last_became_active":"2024-09-10T19:02:52.257016+0000","last_became_peered":"2024-09-10T19:02:52.257016+0000","last_unstale":"2024-09-10T19:07:27.691400+0000","last_undegraded":"2024-09-10T19:07:27.691400+0000","last_fullsized":"2024-09-10T19:07:27.691400+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:04.477051+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.477051+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031221360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":791,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427753+0000","last_change":"2024-09-10T19:06:22.027150+0000","last_active":"2024-09-10T19:07:26.427753+0000","last_peered":"2024-09-10T19:07:26.427753+0000","last_clean":"2024-09-10T19:07:26.427753+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:07:26.427753+0000","last_undegraded":"2024-09-10T19:07:26.427753+0000","last_fullsized":"2024-09-10T19:07:26.427753+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:06:22.027051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.027051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003390789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":710,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428694+0000","last_change":"2024-09-10T19:06:41.123978+0000","last_active":"2024-09-10T19:07:26.428694+0000","last_peered":"2024-09-10T19:07:26.428694+0000","last_clean":"2024-09-10T19:07:26.428694+0000","last_became_active":"2024-09-10T18:54:37.680497+0000","last_became_peered":"2024-09-10T18:54:37.680497+0000","last_unstale":"2024-09-10T19:07:26.428694+0000","last_undegraded":"2024-09-10T19:07:26.428694+0000","last_fullsized":"2024-09-10T19:07:26.428694+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.013632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.013632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003320365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":311,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427662+0000","last_change":"2024-09-10T19:06:42.135170+0000","last_active":"2024-09-10T19:07:26.427662+0000","last_peered":"2024-09-10T19:07:26.427662+0000","last_clean":"2024-09-10T19:07:26.427662+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:07:26.427662+0000","last_undegraded":"2024-09-10T19:07:26.427662+0000","last_fullsized":"2024-09-10T19:07:26.427662+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:42.135006+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.135006+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.854577+0000","scrub_duration":0.0049906150000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":824,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752723+0000","last_change":"2024-09-10T19:06:34.355770+0000","last_active":"2024-09-10T19:07:27.752723+0000","last_peered":"2024-09-10T19:07:27.752723+0000","last_clean":"2024-09-10T19:07:27.752723+0000","last_became_active":"2024-09-10T18:49:19.633634+0000","last_became_peered":"2024-09-10T18:49:19.633634+0000","last_unstale":"2024-09-10T19:07:27.752723+0000","last_undegraded":"2024-09-10T19:07:27.752723+0000","last_fullsized":"2024-09-10T19:07:27.752723+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:34.355514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.355514+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.626969+0000","scrub_duration":0.005673483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":794,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691774+0000","last_change":"2024-09-10T19:05:16.504389+0000","last_active":"2024-09-10T19:07:27.691774+0000","last_peered":"2024-09-10T19:07:27.691774+0000","last_clean":"2024-09-10T19:07:27.691774+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:07:27.691774+0000","last_undegraded":"2024-09-10T19:07:27.691774+0000","last_fullsized":"2024-09-10T19:07:27.691774+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.504234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.504234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040815139999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":256,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751579+0000","last_change":"2024-09-10T19:05:20.129549+0000","last_active":"2024-09-10T19:07:27.751579+0000","last_peered":"2024-09-10T19:07:27.751579+0000","last_clean":"2024-09-10T19:07:27.751579+0000","last_became_active":"2024-09-10T19:02:52.254942+0000","last_became_peered":"2024-09-10T19:02:52.254942+0000","last_unstale":"2024-09-10T19:07:27.751579+0000","last_undegraded":"2024-09-10T19:07:27.751579+0000","last_fullsized":"2024-09-10T19:07:27.751579+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:20.129398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:05:20.129398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033769329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1866,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-09-10T19:07:29.372027+0000","last_change":"2024-09-10T19:05:09.941903+0000","last_active":"2024-09-10T19:07:29.372027+0000","last_peered":"2024-09-10T19:07:29.372027+0000","last_clean":"2024-09-10T19:07:29.372027+0000","last_became_active":"2024-09-10T18:49:22.039938+0000","last_became_peered":"2024-09-10T18:49:22.039938+0000","last_unstale":"2024-09-10T19:07:29.372027+0000","last_undegraded":"2024-09-10T19:07:29.372027+0000","last_fullsized":"2024-09-10T19:07:29.372027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:05:09.941698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:09.941698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041460869999999997,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1066,"num_read_kb":1061,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420013+0000","last_change":"2024-09-10T19:05:07.665521+0000","last_active":"2024-09-10T19:07:27.420013+0000","last_peered":"2024-09-10T19:07:27.420013+0000","last_clean":"2024-09-10T19:07:27.420013+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:07:27.420013+0000","last_undegraded":"2024-09-10T19:07:27.420013+0000","last_fullsized":"2024-09-10T19:07:27.420013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:07.665287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:05:07.665287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003665391,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":732,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313492+0000","last_change":"2024-09-10T19:06:18.075252+0000","last_active":"2024-09-10T19:07:27.313492+0000","last_peered":"2024-09-10T19:07:27.313492+0000","last_clean":"2024-09-10T19:07:27.313492+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:07:27.313492+0000","last_undegraded":"2024-09-10T19:07:27.313492+0000","last_fullsized":"2024-09-10T19:07:27.313492+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:18.075101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.075101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:43.342791+0000","scrub_duration":0.00339478,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":312,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424838+0000","last_change":"2024-09-10T19:06:46.615668+0000","last_active":"2024-09-10T19:07:26.424838+0000","last_peered":"2024-09-10T19:07:26.424838+0000","last_clean":"2024-09-10T19:07:26.424838+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:07:26.424838+0000","last_undegraded":"2024-09-10T19:07:26.424838+0000","last_fullsized":"2024-09-10T19:07:26.424838+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.615570+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:08.032708+0000","scrub_duration":0.0029648029999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":827,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426883+0000","last_change":"2024-09-10T19:05:25.905116+0000","last_active":"2024-09-10T19:07:26.426883+0000","last_peered":"2024-09-10T19:07:26.426883+0000","last_clean":"2024-09-10T19:07:26.426883+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:07:26.426883+0000","last_undegraded":"2024-09-10T19:07:26.426883+0000","last_fullsized":"2024-09-10T19:07:26.426883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'17","last_scrub_stamp":"2024-09-10T19:05:25.904954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:25.904954+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045887740000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":811,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428945+0000","last_change":"2024-09-10T19:06:48.128049+0000","last_active":"2024-09-10T19:07:26.428945+0000","last_peered":"2024-09-10T19:07:26.428945+0000","last_clean":"2024-09-10T19:07:26.428945+0000","last_became_active":"2024-09-10T18:49:17.612734+0000","last_became_peered":"2024-09-10T18:49:17.612734+0000","last_unstale":"2024-09-10T19:07:26.428945+0000","last_undegraded":"2024-09-10T19:07:26.428945+0000","last_fullsized":"2024-09-10T19:07:26.428945+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.127911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.127911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.454765+0000","scrub_duration":0.004079208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":1011,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.644875+0000","last_change":"2024-09-10T19:05:32.921313+0000","last_active":"2024-09-10T19:07:26.644875+0000","last_peered":"2024-09-10T19:07:26.644875+0000","last_clean":"2024-09-10T19:07:26.644875+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:07:26.644875+0000","last_undegraded":"2024-09-10T19:07:26.644875+0000","last_fullsized":"2024-09-10T19:07:26.644875+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:05:32.921124+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.921124+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004136547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":814,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.423912+0000","last_change":"2024-09-10T19:06:20.005964+0000","last_active":"2024-09-10T19:07:26.423912+0000","last_peered":"2024-09-10T19:07:26.423912+0000","last_clean":"2024-09-10T19:07:26.423912+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:07:26.423912+0000","last_undegraded":"2024-09-10T19:07:26.423912+0000","last_fullsized":"2024-09-10T19:07:26.423912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:06:20.005802+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.005802+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:41.479507+0000","scrub_duration":0.0042833380000000002,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":272,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428993+0000","last_change":"2024-09-10T19:06:23.076714+0000","last_active":"2024-09-10T19:07:26.428993+0000","last_peered":"2024-09-10T19:07:26.428993+0000","last_clean":"2024-09-10T19:07:26.428993+0000","last_became_active":"2024-09-10T19:02:52.256316+0000","last_became_peered":"2024-09-10T19:02:52.256316+0000","last_unstale":"2024-09-10T19:07:26.428993+0000","last_undegraded":"2024-09-10T19:07:26.428993+0000","last_fullsized":"2024-09-10T19:07:26.428993+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:23.076562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.076562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.850014+0000","scrub_duration":0.0034726789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429718+0000","last_change":"2024-09-10T19:06:48.197543+0000","last_active":"2024-09-10T19:07:26.429718+0000","last_peered":"2024-09-10T19:07:26.429718+0000","last_clean":"2024-09-10T19:07:26.429718+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:07:26.429718+0000","last_undegraded":"2024-09-10T19:07:26.429718+0000","last_fullsized":"2024-09-10T19:07:26.429718+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.197394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.197394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.035900+0000","scrub_duration":0.003369632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":377,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691173+0000","last_change":"2024-09-10T19:05:09.498115+0000","last_active":"2024-09-10T19:07:27.691173+0000","last_peered":"2024-09-10T19:07:27.691173+0000","last_clean":"2024-09-10T19:07:27.691173+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:07:27.691173+0000","last_undegraded":"2024-09-10T19:07:27.691173+0000","last_fullsized":"2024-09-10T19:07:27.691173+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:09.497849+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:09.497849+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043080979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"410'52","reported_seq":1095,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.645761+0000","last_change":"2024-09-10T19:05:27.874167+0000","last_active":"2024-09-10T19:07:26.645761+0000","last_peered":"2024-09-10T19:07:26.645761+0000","last_clean":"2024-09-10T19:07:26.645761+0000","last_became_active":"2024-09-10T18:49:19.626457+0000","last_became_peered":"2024-09-10T18:49:19.626457+0000","last_unstale":"2024-09-10T19:07:26.645761+0000","last_undegraded":"2024-09-10T19:07:26.645761+0000","last_fullsized":"2024-09-10T19:07:26.645761+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:05:27.874075+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:05:27.874075+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046021589999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":799,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420848+0000","last_change":"2024-09-10T19:06:35.858337+0000","last_active":"2024-09-10T19:07:27.420848+0000","last_peered":"2024-09-10T19:07:27.420848+0000","last_clean":"2024-09-10T19:07:27.420848+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:07:27.420848+0000","last_undegraded":"2024-09-10T19:07:27.420848+0000","last_fullsized":"2024-09-10T19:07:27.420848+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.858069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.858069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.148804+0000","scrub_duration":0.0032278900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":1015,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754918+0000","last_change":"2024-09-10T19:05:25.279549+0000","last_active":"2024-09-10T19:07:27.754918+0000","last_peered":"2024-09-10T19:07:27.754918+0000","last_clean":"2024-09-10T19:07:27.754918+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:07:27.754918+0000","last_undegraded":"2024-09-10T19:07:27.754918+0000","last_fullsized":"2024-09-10T19:07:27.754918+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:05:25.279409+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:05:25.279409+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036183930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":798,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690124+0000","last_change":"2024-09-10T19:05:12.500579+0000","last_active":"2024-09-10T19:07:27.690124+0000","last_peered":"2024-09-10T19:07:27.690124+0000","last_clean":"2024-09-10T19:07:27.690124+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:07:27.690124+0000","last_undegraded":"2024-09-10T19:07:27.690124+0000","last_fullsized":"2024-09-10T19:07:27.690124+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:05:12.500502+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:05:12.500502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035677880000000001,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429368+0000","last_change":"2024-09-10T19:06:30.081763+0000","last_active":"2024-09-10T19:07:26.429368+0000","last_peered":"2024-09-10T19:07:26.429368+0000","last_clean":"2024-09-10T19:07:26.429368+0000","last_became_active":"2024-09-10T18:49:24.365217+0000","last_became_peered":"2024-09-10T18:49:24.365217+0000","last_unstale":"2024-09-10T19:07:26.429368+0000","last_undegraded":"2024-09-10T19:07:26.429368+0000","last_fullsized":"2024-09-10T19:07:26.429368+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.081613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.081613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:45.513032+0000","scrub_duration":0.003447449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427021+0000","last_change":"2024-09-10T19:06:17.163823+0000","last_active":"2024-09-10T19:07:26.427021+0000","last_peered":"2024-09-10T19:07:26.427021+0000","last_clean":"2024-09-10T19:07:26.427021+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:07:26.427021+0000","last_undegraded":"2024-09-10T19:07:26.427021+0000","last_fullsized":"2024-09-10T19:07:26.427021+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.163656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.163656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035671100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":380,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691334+0000","last_change":"2024-09-10T19:06:18.604662+0000","last_active":"2024-09-10T19:07:27.691334+0000","last_peered":"2024-09-10T19:07:27.691334+0000","last_clean":"2024-09-10T19:07:27.691334+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:07:27.691334+0000","last_undegraded":"2024-09-10T19:07:27.691334+0000","last_fullsized":"2024-09-10T19:07:27.691334+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:18.604383+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.604383+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042316089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"760'33","reported_seq":847,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750438+0000","last_change":"2024-09-10T19:06:59.942589+0000","last_active":"2024-09-10T19:07:27.750438+0000","last_peered":"2024-09-10T19:07:27.750438+0000","last_clean":"2024-09-10T19:07:27.750438+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:07:27.750438+0000","last_undegraded":"2024-09-10T19:07:27.750438+0000","last_fullsized":"2024-09-10T19:07:27.750438+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"760'33","last_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_deep_scrub":"760'33","last_deep_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.942479+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:07.358058+0000","scrub_duration":0.012061568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":801,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427785+0000","last_change":"2024-09-10T19:06:26.032434+0000","last_active":"2024-09-10T19:07:26.427785+0000","last_peered":"2024-09-10T19:07:26.427785+0000","last_clean":"2024-09-10T19:07:26.427785+0000","last_became_active":"2024-09-10T18:49:17.615381+0000","last_became_peered":"2024-09-10T18:49:17.615381+0000","last_unstale":"2024-09-10T19:07:26.427785+0000","last_undegraded":"2024-09-10T19:07:26.427785+0000","last_fullsized":"2024-09-10T19:07:26.427785+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.032276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.032276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.801865+0000","scrub_duration":0.0033796479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":797,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426255+0000","last_change":"2024-09-10T19:06:55.205119+0000","last_active":"2024-09-10T19:07:26.426255+0000","last_peered":"2024-09-10T19:07:26.426255+0000","last_clean":"2024-09-10T19:07:26.426255+0000","last_became_active":"2024-09-10T18:49:22.040325+0000","last_became_peered":"2024-09-10T18:49:22.040325+0000","last_unstale":"2024-09-10T19:07:26.426255+0000","last_undegraded":"2024-09-10T19:07:26.426255+0000","last_fullsized":"2024-09-10T19:07:26.426255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.204933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.204933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.228743+0000","scrub_duration":0.0032554939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":784,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754458+0000","last_change":"2024-09-10T19:05:04.441891+0000","last_active":"2024-09-10T19:07:27.754458+0000","last_peered":"2024-09-10T19:07:27.754458+0000","last_clean":"2024-09-10T19:07:27.754458+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:07:27.754458+0000","last_undegraded":"2024-09-10T19:07:27.754458+0000","last_fullsized":"2024-09-10T19:07:27.754458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:04.441647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.441647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029219010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":732,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313519+0000","last_change":"2024-09-10T19:06:22.182306+0000","last_active":"2024-09-10T19:07:27.313519+0000","last_peered":"2024-09-10T19:07:27.313519+0000","last_clean":"2024-09-10T19:07:27.313519+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:07:27.313519+0000","last_undegraded":"2024-09-10T19:07:27.313519+0000","last_fullsized":"2024-09-10T19:07:27.313519+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.182115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.182115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:36.337406+0000","scrub_duration":0.0036403120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":309,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751135+0000","last_change":"2024-09-10T19:06:53.016161+0000","last_active":"2024-09-10T19:07:27.751135+0000","last_peered":"2024-09-10T19:07:27.751135+0000","last_clean":"2024-09-10T19:07:27.751135+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:07:27.751135+0000","last_undegraded":"2024-09-10T19:07:27.751135+0000","last_fullsized":"2024-09-10T19:07:27.751135+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:53.015994+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.015994+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.394220+0000","scrub_duration":0.0033683329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":716,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754817+0000","last_change":"2024-09-10T19:06:39.394175+0000","last_active":"2024-09-10T19:07:27.754817+0000","last_peered":"2024-09-10T19:07:27.754817+0000","last_clean":"2024-09-10T19:07:27.754817+0000","last_became_active":"2024-09-10T19:02:52.256210+0000","last_became_peered":"2024-09-10T19:02:52.256210+0000","last_unstale":"2024-09-10T19:07:27.754817+0000","last_undegraded":"2024-09-10T19:07:27.754817+0000","last_fullsized":"2024-09-10T19:07:27.754817+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:06:39.394055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.394055+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.691949+0000","scrub_duration":0.0034977820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":792,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752058+0000","last_change":"2024-09-10T19:05:28.136902+0000","last_active":"2024-09-10T19:07:27.752058+0000","last_peered":"2024-09-10T19:07:27.752058+0000","last_clean":"2024-09-10T19:07:27.752058+0000","last_became_active":"2024-09-10T18:49:17.612726+0000","last_became_peered":"2024-09-10T18:49:17.612726+0000","last_unstale":"2024-09-10T19:07:27.752058+0000","last_undegraded":"2024-09-10T19:07:27.752058+0000","last_fullsized":"2024-09-10T19:07:27.752058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.136755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032446879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":797,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427696+0000","last_change":"2024-09-10T19:05:19.903060+0000","last_active":"2024-09-10T19:07:26.427696+0000","last_peered":"2024-09-10T19:07:26.427696+0000","last_clean":"2024-09-10T19:07:26.427696+0000","last_became_active":"2024-09-10T18:49:22.046879+0000","last_became_peered":"2024-09-10T18:49:22.046879+0000","last_unstale":"2024-09-10T19:07:26.427696+0000","last_undegraded":"2024-09-10T19:07:26.427696+0000","last_fullsized":"2024-09-10T19:07:26.427696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:19.902898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.902898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037634550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":783,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426887+0000","last_change":"2024-09-10T19:05:19.919334+0000","last_active":"2024-09-10T19:07:26.426887+0000","last_peered":"2024-09-10T19:07:26.426887+0000","last_clean":"2024-09-10T19:07:26.426887+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:07:26.426887+0000","last_undegraded":"2024-09-10T19:07:26.426887+0000","last_fullsized":"2024-09-10T19:07:26.426887+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:19.919226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.919226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028520339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755052+0000","last_change":"2024-09-10T19:06:26.273000+0000","last_active":"2024-09-10T19:07:27.755052+0000","last_peered":"2024-09-10T19:07:27.755052+0000","last_clean":"2024-09-10T19:07:27.755052+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:07:27.755052+0000","last_undegraded":"2024-09-10T19:07:27.755052+0000","last_fullsized":"2024-09-10T19:07:27.755052+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.272837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.272837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.951213+0000","scrub_duration":0.0046824379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":867,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.418279+0000","last_change":"2024-09-10T19:05:29.794987+0000","last_active":"2024-09-10T19:07:27.418279+0000","last_peered":"2024-09-10T19:07:27.418279+0000","last_clean":"2024-09-10T19:07:27.418279+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:07:27.418279+0000","last_undegraded":"2024-09-10T19:07:27.418279+0000","last_fullsized":"2024-09-10T19:07:27.418279+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:29.794822+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:05:29.794822+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003393437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":867,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425171+0000","last_change":"2024-09-10T19:07:01.193377+0000","last_active":"2024-09-10T19:07:26.425171+0000","last_peered":"2024-09-10T19:07:26.425171+0000","last_clean":"2024-09-10T19:07:26.425171+0000","last_became_active":"2024-09-10T18:49:19.625084+0000","last_became_peered":"2024-09-10T18:49:19.625084+0000","last_unstale":"2024-09-10T19:07:26.425171+0000","last_undegraded":"2024-09-10T19:07:26.425171+0000","last_fullsized":"2024-09-10T19:07:26.425171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:07:01.193206+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.193206+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.775475+0000","scrub_duration":0.005269892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":826,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313313+0000","last_change":"2024-09-10T19:06:50.068060+0000","last_active":"2024-09-10T19:07:27.313313+0000","last_peered":"2024-09-10T19:07:27.313313+0000","last_clean":"2024-09-10T19:07:27.313313+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:07:27.313313+0000","last_undegraded":"2024-09-10T19:07:27.313313+0000","last_fullsized":"2024-09-10T19:07:27.313313+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:50.067884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.067884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:59.606576+0000","scrub_duration":0.0049528419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":797,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428936+0000","last_change":"2024-09-10T19:06:54.223011+0000","last_active":"2024-09-10T19:07:26.428936+0000","last_peered":"2024-09-10T19:07:26.428936+0000","last_clean":"2024-09-10T19:07:26.428936+0000","last_became_active":"2024-09-10T18:49:22.036683+0000","last_became_peered":"2024-09-10T18:49:22.036683+0000","last_unstale":"2024-09-10T19:07:26.428936+0000","last_undegraded":"2024-09-10T19:07:26.428936+0000","last_fullsized":"2024-09-10T19:07:26.428936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.222842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:12.925817+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.222842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.955782+0000","scrub_duration":0.0034650459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419586+0000","last_change":"2024-09-10T19:05:14.580485+0000","last_active":"2024-09-10T19:07:27.419586+0000","last_peered":"2024-09-10T19:07:27.419586+0000","last_clean":"2024-09-10T19:07:27.419586+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:07:27.419586+0000","last_undegraded":"2024-09-10T19:07:27.419586+0000","last_fullsized":"2024-09-10T19:07:27.419586+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:14.580383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:14.580383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026901339999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":714,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691472+0000","last_change":"2024-09-10T19:06:28.611791+0000","last_active":"2024-09-10T19:07:27.691472+0000","last_peered":"2024-09-10T19:07:27.691472+0000","last_clean":"2024-09-10T19:07:27.691472+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:07:27.691472+0000","last_undegraded":"2024-09-10T19:07:27.691472+0000","last_fullsized":"2024-09-10T19:07:27.691472+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:28.611520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.611520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:38.340443+0000","scrub_duration":0.0041400660000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":304,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426925+0000","last_change":"2024-09-10T19:06:25.036690+0000","last_active":"2024-09-10T19:07:26.426925+0000","last_peered":"2024-09-10T19:07:26.426925+0000","last_clean":"2024-09-10T19:07:26.426925+0000","last_became_active":"2024-09-10T19:02:03.616064+0000","last_became_peered":"2024-09-10T19:02:03.616064+0000","last_unstale":"2024-09-10T19:07:26.426925+0000","last_undegraded":"2024-09-10T19:07:26.426925+0000","last_fullsized":"2024-09-10T19:07:26.426925+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:25.036490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:25.036490+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.640822+0000","scrub_duration":0.0037518489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":830,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752432+0000","last_change":"2024-09-10T19:04:54.545871+0000","last_active":"2024-09-10T19:07:27.752432+0000","last_peered":"2024-09-10T19:07:27.752432+0000","last_clean":"2024-09-10T19:07:27.752432+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:07:27.752432+0000","last_undegraded":"2024-09-10T19:07:27.752432+0000","last_fullsized":"2024-09-10T19:07:27.752432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'28","last_scrub_stamp":"2024-09-10T19:04:54.545674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:04:54.545674+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049500350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":799,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.417915+0000","last_change":"2024-09-10T19:05:04.758120+0000","last_active":"2024-09-10T19:07:27.417915+0000","last_peered":"2024-09-10T19:07:27.417915+0000","last_clean":"2024-09-10T19:07:27.417915+0000","last_became_active":"2024-09-10T18:49:17.615320+0000","last_became_peered":"2024-09-10T18:49:17.615320+0000","last_unstale":"2024-09-10T19:07:27.417915+0000","last_undegraded":"2024-09-10T19:07:27.417915+0000","last_fullsized":"2024-09-10T19:07:27.417915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:04.757950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.757950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032582150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":789,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755597+0000","last_change":"2024-09-10T19:05:10.481860+0000","last_active":"2024-09-10T19:07:27.755597+0000","last_peered":"2024-09-10T19:07:27.755597+0000","last_clean":"2024-09-10T19:07:27.755597+0000","last_became_active":"2024-09-10T18:49:22.047075+0000","last_became_peered":"2024-09-10T18:49:22.047075+0000","last_unstale":"2024-09-10T19:07:27.755597+0000","last_undegraded":"2024-09-10T19:07:27.755597+0000","last_fullsized":"2024-09-10T19:07:27.755597+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.481693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.481693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037720449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427438+0000","last_change":"2024-09-10T19:05:15.852430+0000","last_active":"2024-09-10T19:07:26.427438+0000","last_peered":"2024-09-10T19:07:26.427438+0000","last_clean":"2024-09-10T19:07:26.427438+0000","last_became_active":"2024-09-10T18:49:24.053770+0000","last_became_peered":"2024-09-10T18:49:24.053770+0000","last_unstale":"2024-09-10T19:07:26.427438+0000","last_undegraded":"2024-09-10T19:07:26.427438+0000","last_fullsized":"2024-09-10T19:07:26.427438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.852281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081435180000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755690+0000","last_change":"2024-09-10T19:06:21.347985+0000","last_active":"2024-09-10T19:07:27.755690+0000","last_peered":"2024-09-10T19:07:27.755690+0000","last_clean":"2024-09-10T19:07:27.755690+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:07:27.755690+0000","last_undegraded":"2024-09-10T19:07:27.755690+0000","last_fullsized":"2024-09-10T19:07:27.755690+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:21.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:21.347904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003263929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":819,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428840+0000","last_change":"2024-09-10T19:05:20.886436+0000","last_active":"2024-09-10T19:07:26.428840+0000","last_peered":"2024-09-10T19:07:26.428840+0000","last_clean":"2024-09-10T19:07:26.428840+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:07:26.428840+0000","last_undegraded":"2024-09-10T19:07:26.428840+0000","last_fullsized":"2024-09-10T19:07:26.428840+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:20.886267+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:05:20.886267+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049540019999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":802,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752062+0000","last_change":"2024-09-10T19:06:51.097568+0000","last_active":"2024-09-10T19:07:27.752062+0000","last_peered":"2024-09-10T19:07:27.752062+0000","last_clean":"2024-09-10T19:07:27.752062+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:07:27.752062+0000","last_undegraded":"2024-09-10T19:07:27.752062+0000","last_fullsized":"2024-09-10T19:07:27.752062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:51.097406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:06:51.097406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:16.803993+0000","scrub_duration":0.003232005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":858,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.418200+0000","last_change":"2024-09-10T19:05:32.752121+0000","last_active":"2024-09-10T19:07:27.418200+0000","last_peered":"2024-09-10T19:07:27.418200+0000","last_clean":"2024-09-10T19:07:27.418200+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:07:27.418200+0000","last_undegraded":"2024-09-10T19:07:27.418200+0000","last_fullsized":"2024-09-10T19:07:27.418200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:05:32.752014+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:05:32.752014+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.752014+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028322999999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":790,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691077+0000","last_change":"2024-09-10T19:05:22.537528+0000","last_active":"2024-09-10T19:07:27.691077+0000","last_peered":"2024-09-10T19:07:27.691077+0000","last_clean":"2024-09-10T19:07:27.691077+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:07:27.691077+0000","last_undegraded":"2024-09-10T19:07:27.691077+0000","last_fullsized":"2024-09-10T19:07:27.691077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:22.537370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:05:22.537370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038042839999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":789,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753491+0000","last_change":"2024-09-10T19:06:55.021461+0000","last_active":"2024-09-10T19:07:27.753491+0000","last_peered":"2024-09-10T19:07:27.753491+0000","last_clean":"2024-09-10T19:07:27.753491+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:07:27.753491+0000","last_undegraded":"2024-09-10T19:07:27.753491+0000","last_fullsized":"2024-09-10T19:07:27.753491+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.021218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.148405+0000","scrub_duration":0.0037286939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":717,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420388+0000","last_change":"2024-09-10T19:06:45.994023+0000","last_active":"2024-09-10T19:07:27.420388+0000","last_peered":"2024-09-10T19:07:27.420388+0000","last_clean":"2024-09-10T19:07:27.420388+0000","last_became_active":"2024-09-10T18:54:37.596409+0000","last_became_peered":"2024-09-10T18:54:37.596409+0000","last_unstale":"2024-09-10T19:07:27.420388+0000","last_undegraded":"2024-09-10T19:07:27.420388+0000","last_fullsized":"2024-09-10T19:07:27.420388+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.993852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.778147+0000","scrub_duration":0.0035603079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":854,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420632+0000","last_change":"2024-09-10T19:06:47.011623+0000","last_active":"2024-09-10T19:07:27.420632+0000","last_peered":"2024-09-10T19:07:27.420632+0000","last_clean":"2024-09-10T19:07:27.420632+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:07:27.420632+0000","last_undegraded":"2024-09-10T19:07:27.420632+0000","last_fullsized":"2024-09-10T19:07:27.420632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_deep_scrub":"410'39","last_deep_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.011531+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:00.270825+0000","scrub_duration":0.0070935080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":826,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314921+0000","last_change":"2024-09-10T19:06:44.173326+0000","last_active":"2024-09-10T19:07:27.314921+0000","last_peered":"2024-09-10T19:07:27.314921+0000","last_clean":"2024-09-10T19:07:27.314921+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:07:27.314921+0000","last_undegraded":"2024-09-10T19:07:27.314921+0000","last_fullsized":"2024-09-10T19:07:27.314921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:44.173162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:44.173162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:44.806723+0000","scrub_duration":0.0034336560000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":800,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690363+0000","last_change":"2024-09-10T19:06:47.846775+0000","last_active":"2024-09-10T19:07:27.690363+0000","last_peered":"2024-09-10T19:07:27.690363+0000","last_clean":"2024-09-10T19:07:27.690363+0000","last_became_active":"2024-09-10T18:49:22.058576+0000","last_became_peered":"2024-09-10T18:49:22.058576+0000","last_unstale":"2024-09-10T19:07:27.690363+0000","last_undegraded":"2024-09-10T19:07:27.690363+0000","last_fullsized":"2024-09-10T19:07:27.690363+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:47.846685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.846685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:48.395515+0000","scrub_duration":0.002537961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420718+0000","last_change":"2024-09-10T19:06:34.889988+0000","last_active":"2024-09-10T19:07:27.420718+0000","last_peered":"2024-09-10T19:07:27.420718+0000","last_clean":"2024-09-10T19:07:27.420718+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:07:27.420718+0000","last_undegraded":"2024-09-10T19:07:27.420718+0000","last_fullsized":"2024-09-10T19:07:27.420718+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:34.889826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.889826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.178210+0000","scrub_duration":0.0038908760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":726,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424819+0000","last_change":"2024-09-10T19:06:51.691510+0000","last_active":"2024-09-10T19:07:26.424819+0000","last_peered":"2024-09-10T19:07:26.424819+0000","last_clean":"2024-09-10T19:07:26.424819+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:07:26.424819+0000","last_undegraded":"2024-09-10T19:07:26.424819+0000","last_fullsized":"2024-09-10T19:07:26.424819+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:51.691275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:51.691275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.627326+0000","scrub_duration":0.0081710110000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":825,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750363+0000","last_change":"2024-09-10T19:05:32.092434+0000","last_active":"2024-09-10T19:07:27.750363+0000","last_peered":"2024-09-10T19:07:27.750363+0000","last_clean":"2024-09-10T19:07:27.750363+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:07:27.750363+0000","last_undegraded":"2024-09-10T19:07:27.750363+0000","last_fullsized":"2024-09-10T19:07:27.750363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:32.092303+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.092303+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045903259999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":801,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428371+0000","last_change":"2024-09-10T19:06:35.122084+0000","last_active":"2024-09-10T19:07:26.428371+0000","last_peered":"2024-09-10T19:07:26.428371+0000","last_clean":"2024-09-10T19:07:26.428371+0000","last_became_active":"2024-09-10T18:49:17.611429+0000","last_became_peered":"2024-09-10T18:49:17.611429+0000","last_unstale":"2024-09-10T19:07:26.428371+0000","last_undegraded":"2024-09-10T19:07:26.428371+0000","last_fullsized":"2024-09-10T19:07:26.428371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.121923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.121923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:49.025804+0000","scrub_duration":0.003146706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":786,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754624+0000","last_change":"2024-09-10T19:06:48.338457+0000","last_active":"2024-09-10T19:07:27.754624+0000","last_peered":"2024-09-10T19:07:27.754624+0000","last_clean":"2024-09-10T19:07:27.754624+0000","last_became_active":"2024-09-10T18:49:22.039453+0000","last_became_peered":"2024-09-10T18:49:22.039453+0000","last_unstale":"2024-09-10T19:07:27.754624+0000","last_undegraded":"2024-09-10T19:07:27.754624+0000","last_fullsized":"2024-09-10T19:07:27.754624+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.338294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.338294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.950760+0000","scrub_duration":0.0033152189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":784,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755198+0000","last_change":"2024-09-10T19:06:53.347534+0000","last_active":"2024-09-10T19:07:27.755198+0000","last_peered":"2024-09-10T19:07:27.755198+0000","last_clean":"2024-09-10T19:07:27.755198+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:07:27.755198+0000","last_undegraded":"2024-09-10T19:07:27.755198+0000","last_fullsized":"2024-09-10T19:07:27.755198+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.347358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:07.952993+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.347358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.522684+0000","scrub_duration":0.0038827929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754701+0000","last_change":"2024-09-10T19:05:15.465247+0000","last_active":"2024-09-10T19:07:27.754701+0000","last_peered":"2024-09-10T19:07:27.754701+0000","last_clean":"2024-09-10T19:07:27.754701+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:07:27.754701+0000","last_undegraded":"2024-09-10T19:07:27.754701+0000","last_fullsized":"2024-09-10T19:07:27.754701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.465081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:46.602792+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.465081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036040730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":847,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420221+0000","last_change":"2024-09-10T19:05:19.619655+0000","last_active":"2024-09-10T19:07:27.420221+0000","last_peered":"2024-09-10T19:07:27.420221+0000","last_clean":"2024-09-10T19:07:27.420221+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:07:27.420221+0000","last_undegraded":"2024-09-10T19:07:27.420221+0000","last_fullsized":"2024-09-10T19:07:27.420221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:05:19.619510+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.619510+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045528269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":838,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314867+0000","last_change":"2024-09-10T19:06:37.103632+0000","last_active":"2024-09-10T19:07:27.314867+0000","last_peered":"2024-09-10T19:07:27.314867+0000","last_clean":"2024-09-10T19:07:27.314867+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:07:27.314867+0000","last_undegraded":"2024-09-10T19:07:27.314867+0000","last_fullsized":"2024-09-10T19:07:27.314867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:06:37.103448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.103448+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:53.430585+0000","scrub_duration":0.0050576179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":798,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750820+0000","last_change":"2024-09-10T19:06:22.828266+0000","last_active":"2024-09-10T19:07:27.750820+0000","last_peered":"2024-09-10T19:07:27.750820+0000","last_clean":"2024-09-10T19:07:27.750820+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:07:27.750820+0000","last_undegraded":"2024-09-10T19:07:27.750820+0000","last_fullsized":"2024-09-10T19:07:27.750820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.828113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.828113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.146242+0000","scrub_duration":0.0031827000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":796,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751762+0000","last_change":"2024-09-10T19:06:53.989492+0000","last_active":"2024-09-10T19:07:27.751762+0000","last_peered":"2024-09-10T19:07:27.751762+0000","last_clean":"2024-09-10T19:07:27.751762+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:07:27.751762+0000","last_undegraded":"2024-09-10T19:07:27.751762+0000","last_fullsized":"2024-09-10T19:07:27.751762+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.989397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.989397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.968837+0000","scrub_duration":0.0021082940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":265,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689689+0000","last_change":"2024-09-10T19:06:29.614256+0000","last_active":"2024-09-10T19:07:27.689689+0000","last_peered":"2024-09-10T19:07:27.689689+0000","last_clean":"2024-09-10T19:07:27.689689+0000","last_became_active":"2024-09-10T19:02:52.256701+0000","last_became_peered":"2024-09-10T19:02:52.256701+0000","last_unstale":"2024-09-10T19:07:27.689689+0000","last_undegraded":"2024-09-10T19:07:27.689689+0000","last_fullsized":"2024-09-10T19:07:27.689689+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:29.614071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.614071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.354381+0000","scrub_duration":0.0030890639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":821,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420741+0000","last_change":"2024-09-10T19:04:53.827646+0000","last_active":"2024-09-10T19:07:27.420741+0000","last_peered":"2024-09-10T19:07:27.420741+0000","last_clean":"2024-09-10T19:07:27.420741+0000","last_became_active":"2024-09-10T18:49:19.632380+0000","last_became_peered":"2024-09-10T18:49:19.632380+0000","last_unstale":"2024-09-10T19:07:27.420741+0000","last_undegraded":"2024-09-10T19:07:27.420741+0000","last_fullsized":"2024-09-10T19:07:27.420741+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:04:53.827480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:04:53.827480+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086584599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"601'4","reported_seq":852,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426455+0000","last_change":"2024-09-10T19:06:18.041112+0000","last_active":"2024-09-10T19:07:26.426455+0000","last_peered":"2024-09-10T19:07:26.426455+0000","last_clean":"2024-09-10T19:07:26.426455+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:07:26.426455+0000","last_undegraded":"2024-09-10T19:07:26.426455+0000","last_fullsized":"2024-09-10T19:07:26.426455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'4","last_scrub_stamp":"2024-09-10T19:06:18.041030+0000","last_deep_scrub":"601'4","last_deep_scrub_stamp":"2024-09-10T19:04:04.923921+0000","last_clean_scrub_stamp":"2024-09-10T19:06:18.041030+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.678787+0000","scrub_duration":0.0030506399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"548'12","reported_seq":347,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.645533+0000","last_change":"2024-09-10T19:06:22.009512+0000","last_active":"2024-09-10T19:07:26.645533+0000","last_peered":"2024-09-10T19:07:26.645533+0000","last_clean":"2024-09-10T19:07:26.645533+0000","last_became_active":"2024-09-10T19:02:52.258504+0000","last_became_peered":"2024-09-10T19:02:52.258504+0000","last_unstale":"2024-09-10T19:07:26.645533+0000","last_undegraded":"2024-09-10T19:07:26.645533+0000","last_fullsized":"2024-09-10T19:07:26.645533+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-10T19:06:22.009440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.009440+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:43.875145+0000","scrub_duration":0.003198631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":790,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426315+0000","last_change":"2024-09-10T19:07:00.155780+0000","last_active":"2024-09-10T19:07:26.426315+0000","last_peered":"2024-09-10T19:07:26.426315+0000","last_clean":"2024-09-10T19:07:26.426315+0000","last_became_active":"2024-09-10T18:49:24.043472+0000","last_became_peered":"2024-09-10T18:49:24.043472+0000","last_unstale":"2024-09-10T19:07:26.426315+0000","last_undegraded":"2024-09-10T19:07:26.426315+0000","last_fullsized":"2024-09-10T19:07:26.426315+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:00.155684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:07:00.155684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:04.453486+0000","scrub_duration":0.0084607129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":724,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689851+0000","last_change":"2024-09-10T19:05:13.547418+0000","last_active":"2024-09-10T19:07:27.689851+0000","last_peered":"2024-09-10T19:07:27.689851+0000","last_clean":"2024-09-10T19:07:27.689851+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:07:27.689851+0000","last_undegraded":"2024-09-10T19:07:27.689851+0000","last_fullsized":"2024-09-10T19:07:27.689851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:05:13.547259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:13.547259+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040562979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":838,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419933+0000","last_change":"2024-09-10T19:06:57.214131+0000","last_active":"2024-09-10T19:07:27.419933+0000","last_peered":"2024-09-10T19:07:27.419933+0000","last_clean":"2024-09-10T19:07:27.419933+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:07:27.419933+0000","last_undegraded":"2024-09-10T19:07:27.419933+0000","last_fullsized":"2024-09-10T19:07:27.419933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:57.213972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:57.213972+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:10.939374+0000","scrub_duration":0.0047875110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":826,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.313393+0000","last_change":"2024-09-10T19:06:55.984909+0000","last_active":"2024-09-10T19:07:27.313393+0000","last_peered":"2024-09-10T19:07:27.313393+0000","last_clean":"2024-09-10T19:07:27.313393+0000","last_became_active":"2024-09-10T18:49:17.608041+0000","last_became_peered":"2024-09-10T18:49:17.608041+0000","last_unstale":"2024-09-10T19:07:27.313393+0000","last_undegraded":"2024-09-10T19:07:27.313393+0000","last_fullsized":"2024-09-10T19:07:27.313393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.984769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.984769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.226594+0000","scrub_duration":0.0032115210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":786,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755471+0000","last_change":"2024-09-10T19:05:28.258466+0000","last_active":"2024-09-10T19:07:27.755471+0000","last_peered":"2024-09-10T19:07:27.755471+0000","last_clean":"2024-09-10T19:07:27.755471+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:07:27.755471+0000","last_undegraded":"2024-09-10T19:07:27.755471+0000","last_fullsized":"2024-09-10T19:07:27.755471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.258181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.258181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037729090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":810,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314036+0000","last_change":"2024-09-10T19:06:31.188342+0000","last_active":"2024-09-10T19:07:27.314036+0000","last_peered":"2024-09-10T19:07:27.314036+0000","last_clean":"2024-09-10T19:07:27.314036+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:07:27.314036+0000","last_undegraded":"2024-09-10T19:07:27.314036+0000","last_fullsized":"2024-09-10T19:07:27.314036+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.188235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.188235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:52.059595+0000","scrub_duration":0.002958315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":726,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426494+0000","last_change":"2024-09-10T19:06:56.683660+0000","last_active":"2024-09-10T19:07:26.426494+0000","last_peered":"2024-09-10T19:07:26.426494+0000","last_clean":"2024-09-10T19:07:26.426494+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:07:26.426494+0000","last_undegraded":"2024-09-10T19:07:26.426494+0000","last_fullsized":"2024-09-10T19:07:26.426494+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:56.683587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:56.683587+0000","last_clean_scrub_stamp":"2024-09-10T19:06:56.683587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:05.008641+0000","scrub_duration":0.0099833200000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":816,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.689853+0000","last_change":"2024-09-10T19:06:52.872642+0000","last_active":"2024-09-10T19:07:27.689853+0000","last_peered":"2024-09-10T19:07:27.689853+0000","last_clean":"2024-09-10T19:07:27.689853+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:07:27.689853+0000","last_undegraded":"2024-09-10T19:07:27.689853+0000","last_fullsized":"2024-09-10T19:07:27.689853+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:06:52.872480+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.872480+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:56.358965+0000","scrub_duration":0.0042870599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":804,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690360+0000","last_change":"2024-09-10T19:06:33.682027+0000","last_active":"2024-09-10T19:07:27.690360+0000","last_peered":"2024-09-10T19:07:27.690360+0000","last_clean":"2024-09-10T19:07:27.690360+0000","last_became_active":"2024-09-10T18:49:17.609520+0000","last_became_peered":"2024-09-10T18:49:17.609520+0000","last_unstale":"2024-09-10T19:07:27.690360+0000","last_undegraded":"2024-09-10T19:07:27.690360+0000","last_fullsized":"2024-09-10T19:07:27.690360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:33.681850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:33.681850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.843090+0000","scrub_duration":0.0033460909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":782,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426668+0000","last_change":"2024-09-10T19:05:16.995554+0000","last_active":"2024-09-10T19:07:26.426668+0000","last_peered":"2024-09-10T19:07:26.426668+0000","last_clean":"2024-09-10T19:07:26.426668+0000","last_became_active":"2024-09-10T18:49:22.064026+0000","last_became_peered":"2024-09-10T18:49:22.064026+0000","last_unstale":"2024-09-10T19:07:26.426668+0000","last_undegraded":"2024-09-10T19:07:26.426668+0000","last_fullsized":"2024-09-10T19:07:26.426668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.995408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.995408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035441219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419604+0000","last_change":"2024-09-10T19:05:16.578048+0000","last_active":"2024-09-10T19:07:27.419604+0000","last_peered":"2024-09-10T19:07:27.419604+0000","last_clean":"2024-09-10T19:07:27.419604+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:07:27.419604+0000","last_undegraded":"2024-09-10T19:07:27.419604+0000","last_fullsized":"2024-09-10T19:07:27.419604+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.577898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.577898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025345699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691407+0000","last_change":"2024-09-10T19:05:10.514126+0000","last_active":"2024-09-10T19:07:27.691407+0000","last_peered":"2024-09-10T19:07:27.691407+0000","last_clean":"2024-09-10T19:07:27.691407+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:07:27.691407+0000","last_undegraded":"2024-09-10T19:07:27.691407+0000","last_fullsized":"2024-09-10T19:07:27.691407+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.513974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.513974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003294848,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":833,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420802+0000","last_change":"2024-09-10T19:06:43.983711+0000","last_active":"2024-09-10T19:07:27.420802+0000","last_peered":"2024-09-10T19:07:27.420802+0000","last_clean":"2024-09-10T19:07:27.420802+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:07:27.420802+0000","last_undegraded":"2024-09-10T19:07:27.420802+0000","last_fullsized":"2024-09-10T19:07:27.420802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_deep_scrub":"410'24","last_deep_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.983521+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:08.138671+0000","scrub_duration":0.005511228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752099+0000","last_change":"2024-09-10T19:06:32.786736+0000","last_active":"2024-09-10T19:07:27.752099+0000","last_peered":"2024-09-10T19:07:27.752099+0000","last_clean":"2024-09-10T19:07:27.752099+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:07:27.752099+0000","last_undegraded":"2024-09-10T19:07:27.752099+0000","last_fullsized":"2024-09-10T19:07:27.752099+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.786573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.786573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:50.715827+0000","scrub_duration":0.010562732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"586'12","reported_seq":1024,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.644692+0000","last_change":"2024-09-10T19:06:20.042046+0000","last_active":"2024-09-10T19:07:26.644692+0000","last_peered":"2024-09-10T19:07:26.644692+0000","last_clean":"2024-09-10T19:07:26.644692+0000","last_became_active":"2024-09-10T18:49:22.063335+0000","last_became_peered":"2024-09-10T18:49:22.063335+0000","last_unstale":"2024-09-10T19:07:26.644692+0000","last_undegraded":"2024-09-10T19:07:26.644692+0000","last_fullsized":"2024-09-10T19:07:26.644692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"586'12","last_scrub_stamp":"2024-09-10T19:06:20.041985+0000","last_deep_scrub":"586'12","last_deep_scrub_stamp":"2024-09-10T19:03:50.172621+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.041985+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003891285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":784,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755450+0000","last_change":"2024-09-10T19:05:17.407644+0000","last_active":"2024-09-10T19:07:27.755450+0000","last_peered":"2024-09-10T19:07:27.755450+0000","last_clean":"2024-09-10T19:07:27.755450+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:07:27.755450+0000","last_undegraded":"2024-09-10T19:07:27.755450+0000","last_fullsized":"2024-09-10T19:07:27.755450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:17.407512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.407512+0000","last_clean_scrub_stamp":"2024-09-10T19:05:17.407512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028230099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755456+0000","last_change":"2024-09-10T19:06:30.306917+0000","last_active":"2024-09-10T19:07:27.755456+0000","last_peered":"2024-09-10T19:07:27.755456+0000","last_clean":"2024-09-10T19:07:27.755456+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:07:27.755456+0000","last_undegraded":"2024-09-10T19:07:27.755456+0000","last_fullsized":"2024-09-10T19:07:27.755456+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.306762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.306762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.871043+0000","scrub_duration":0.0038348879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":813,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752618+0000","last_change":"2024-09-10T19:05:12.517087+0000","last_active":"2024-09-10T19:07:27.752618+0000","last_peered":"2024-09-10T19:07:27.752618+0000","last_clean":"2024-09-10T19:07:27.752618+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:07:27.752618+0000","last_undegraded":"2024-09-10T19:07:27.752618+0000","last_fullsized":"2024-09-10T19:07:27.752618+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:05:12.516921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:12.516921+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050411789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":800,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.754958+0000","last_change":"2024-09-10T19:06:58.401445+0000","last_active":"2024-09-10T19:07:27.754958+0000","last_peered":"2024-09-10T19:07:27.754958+0000","last_clean":"2024-09-10T19:07:27.754958+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:07:27.754958+0000","last_undegraded":"2024-09-10T19:07:27.754958+0000","last_fullsized":"2024-09-10T19:07:27.754958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:58.401361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.401361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:09.596261+0000","scrub_duration":0.002332808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":1006,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752854+0000","last_change":"2024-09-10T19:06:28.763314+0000","last_active":"2024-09-10T19:07:27.752854+0000","last_peered":"2024-09-10T19:07:27.752854+0000","last_clean":"2024-09-10T19:07:27.752854+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:07:27.752854+0000","last_undegraded":"2024-09-10T19:07:27.752854+0000","last_fullsized":"2024-09-10T19:07:27.752854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:06:28.763149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.763149+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:39.664851+0000","scrub_duration":0.0099338919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":275,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314883+0000","last_change":"2024-09-10T19:06:54.081950+0000","last_active":"2024-09-10T19:07:27.314883+0000","last_peered":"2024-09-10T19:07:27.314883+0000","last_clean":"2024-09-10T19:07:27.314883+0000","last_became_active":"2024-09-10T19:02:52.256751+0000","last_became_peered":"2024-09-10T19:02:52.256751+0000","last_unstale":"2024-09-10T19:07:27.314883+0000","last_undegraded":"2024-09-10T19:07:27.314883+0000","last_fullsized":"2024-09-10T19:07:27.314883+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.081811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.081811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.018335+0000","scrub_duration":0.0036574749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":695,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755606+0000","last_change":"2024-09-10T19:04:59.527415+0000","last_active":"2024-09-10T19:07:27.755606+0000","last_peered":"2024-09-10T19:07:27.755606+0000","last_clean":"2024-09-10T19:07:27.755606+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:07:27.755606+0000","last_undegraded":"2024-09-10T19:07:27.755606+0000","last_fullsized":"2024-09-10T19:07:27.755606+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.527252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.527252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033075349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":822,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.420026+0000","last_change":"2024-09-10T19:05:26.703616+0000","last_active":"2024-09-10T19:07:27.420026+0000","last_peered":"2024-09-10T19:07:27.420026+0000","last_clean":"2024-09-10T19:07:27.420026+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:07:27.420026+0000","last_undegraded":"2024-09-10T19:07:27.420026+0000","last_fullsized":"2024-09-10T19:07:27.420026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:05:26.703433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:26.703433+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050260740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":794,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690928+0000","last_change":"2024-09-10T19:05:31.506173+0000","last_active":"2024-09-10T19:07:27.690928+0000","last_peered":"2024-09-10T19:07:27.690928+0000","last_clean":"2024-09-10T19:07:27.690928+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:07:27.690928+0000","last_undegraded":"2024-09-10T19:07:27.690928+0000","last_fullsized":"2024-09-10T19:07:27.690928+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:31.506021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:31.506021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030748609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":789,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751529+0000","last_change":"2024-09-10T19:06:37.901263+0000","last_active":"2024-09-10T19:07:27.751529+0000","last_peered":"2024-09-10T19:07:27.751529+0000","last_clean":"2024-09-10T19:07:27.751529+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:07:27.751529+0000","last_undegraded":"2024-09-10T19:07:27.751529+0000","last_fullsized":"2024-09-10T19:07:27.751529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.901111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.901111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.610396+0000","scrub_duration":0.0036067199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":787,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753324+0000","last_change":"2024-09-10T19:05:02.444342+0000","last_active":"2024-09-10T19:07:27.753324+0000","last_peered":"2024-09-10T19:07:27.753324+0000","last_clean":"2024-09-10T19:07:27.753324+0000","last_became_active":"2024-09-10T18:49:24.048821+0000","last_became_peered":"2024-09-10T18:49:24.048821+0000","last_unstale":"2024-09-10T19:07:27.753324+0000","last_undegraded":"2024-09-10T19:07:27.753324+0000","last_fullsized":"2024-09-10T19:07:27.753324+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.444182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.444182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002871462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":726,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425109+0000","last_change":"2024-09-10T19:06:54.653102+0000","last_active":"2024-09-10T19:07:26.425109+0000","last_peered":"2024-09-10T19:07:26.425109+0000","last_clean":"2024-09-10T19:07:26.425109+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:07:26.425109+0000","last_undegraded":"2024-09-10T19:07:26.425109+0000","last_fullsized":"2024-09-10T19:07:26.425109+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.652944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:21.751266+0000","scrub_duration":0.0032820850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":699,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419589+0000","last_change":"2024-09-10T19:05:06.709482+0000","last_active":"2024-09-10T19:07:27.419589+0000","last_peered":"2024-09-10T19:07:27.419589+0000","last_clean":"2024-09-10T19:07:27.419589+0000","last_became_active":"2024-09-10T19:02:52.257159+0000","last_became_peered":"2024-09-10T19:02:52.257159+0000","last_unstale":"2024-09-10T19:07:27.419589+0000","last_undegraded":"2024-09-10T19:07:27.419589+0000","last_fullsized":"2024-09-10T19:07:27.419589+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:05:06.709404+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:05:06.709404+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030078150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":801,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.428258+0000","last_change":"2024-09-10T19:06:17.966425+0000","last_active":"2024-09-10T19:07:26.428258+0000","last_peered":"2024-09-10T19:07:26.428258+0000","last_clean":"2024-09-10T19:07:26.428258+0000","last_became_active":"2024-09-10T18:49:17.613775+0000","last_became_peered":"2024-09-10T18:49:17.613775+0000","last_unstale":"2024-09-10T19:07:26.428258+0000","last_undegraded":"2024-09-10T19:07:26.428258+0000","last_fullsized":"2024-09-10T19:07:26.428258+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.966265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.966265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:31.827356+0000","scrub_duration":0.006067228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":807,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.424908+0000","last_change":"2024-09-10T19:06:28.866625+0000","last_active":"2024-09-10T19:07:26.424908+0000","last_peered":"2024-09-10T19:07:26.424908+0000","last_clean":"2024-09-10T19:07:26.424908+0000","last_became_active":"2024-09-10T18:49:22.044440+0000","last_became_peered":"2024-09-10T18:49:22.044440+0000","last_unstale":"2024-09-10T19:07:26.424908+0000","last_undegraded":"2024-09-10T19:07:26.424908+0000","last_fullsized":"2024-09-10T19:07:26.424908+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:28.866466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.866466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.241863+0000","scrub_duration":0.0080387719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425616+0000","last_change":"2024-09-10T19:05:30.921980+0000","last_active":"2024-09-10T19:07:26.425616+0000","last_peered":"2024-09-10T19:07:26.425616+0000","last_clean":"2024-09-10T19:07:26.425616+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:07:26.425616+0000","last_undegraded":"2024-09-10T19:07:26.425616+0000","last_fullsized":"2024-09-10T19:07:26.425616+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:30.921823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:51.975710+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.921823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003529318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":271,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427466+0000","last_change":"2024-09-10T19:06:32.121855+0000","last_active":"2024-09-10T19:07:26.427466+0000","last_peered":"2024-09-10T19:07:26.427466+0000","last_clean":"2024-09-10T19:07:26.427466+0000","last_became_active":"2024-09-10T19:02:52.258536+0000","last_became_peered":"2024-09-10T19:02:52.258536+0000","last_unstale":"2024-09-10T19:07:26.427466+0000","last_undegraded":"2024-09-10T19:07:26.427466+0000","last_fullsized":"2024-09-10T19:07:26.427466+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.121693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.121693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:46.538774+0000","scrub_duration":0.0033793880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":809,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753024+0000","last_change":"2024-09-10T19:06:41.386525+0000","last_active":"2024-09-10T19:07:27.753024+0000","last_peered":"2024-09-10T19:07:27.753024+0000","last_clean":"2024-09-10T19:07:27.753024+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:07:27.753024+0000","last_undegraded":"2024-09-10T19:07:27.753024+0000","last_fullsized":"2024-09-10T19:07:27.753024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_deep_scrub":"410'15","last_deep_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_clean_scrub_stamp":"2024-09-10T19:06:41.386460+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.747797+0000","scrub_duration":0.0022228119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":792,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753134+0000","last_change":"2024-09-10T19:06:20.906700+0000","last_active":"2024-09-10T19:07:27.753134+0000","last_peered":"2024-09-10T19:07:27.753134+0000","last_clean":"2024-09-10T19:07:27.753134+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:07:27.753134+0000","last_undegraded":"2024-09-10T19:07:27.753134+0000","last_fullsized":"2024-09-10T19:07:27.753134+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:20.906544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.906544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:37.590695+0000","scrub_duration":0.0034009769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":785,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429651+0000","last_change":"2024-09-10T19:05:28.918947+0000","last_active":"2024-09-10T19:07:26.429651+0000","last_peered":"2024-09-10T19:07:26.429651+0000","last_clean":"2024-09-10T19:07:26.429651+0000","last_became_active":"2024-09-10T18:49:22.037954+0000","last_became_peered":"2024-09-10T18:49:22.037954+0000","last_unstale":"2024-09-10T19:07:26.429651+0000","last_undegraded":"2024-09-10T19:07:26.429651+0000","last_fullsized":"2024-09-10T19:07:26.429651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.918873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030432179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.418450+0000","last_change":"2024-09-10T19:05:02.785214+0000","last_active":"2024-09-10T19:07:27.418450+0000","last_peered":"2024-09-10T19:07:27.418450+0000","last_clean":"2024-09-10T19:07:27.418450+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:07:27.418450+0000","last_undegraded":"2024-09-10T19:07:27.418450+0000","last_fullsized":"2024-09-10T19:07:27.418450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.785055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.785055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033757409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":717,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425017+0000","last_change":"2024-09-10T19:06:50.191355+0000","last_active":"2024-09-10T19:07:26.425017+0000","last_peered":"2024-09-10T19:07:26.425017+0000","last_clean":"2024-09-10T19:07:26.425017+0000","last_became_active":"2024-09-10T18:54:37.614456+0000","last_became_peered":"2024-09-10T18:54:37.614456+0000","last_unstale":"2024-09-10T19:07:26.425017+0000","last_undegraded":"2024-09-10T19:07:26.425017+0000","last_fullsized":"2024-09-10T19:07:26.425017+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:50.191205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:14.886297+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.191205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:17.708068+0000","scrub_duration":0.003564837,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"676'19","reported_seq":819,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.750466+0000","last_change":"2024-09-10T19:05:34.022493+0000","last_active":"2024-09-10T19:07:27.750466+0000","last_peered":"2024-09-10T19:07:27.750466+0000","last_clean":"2024-09-10T19:07:27.750466+0000","last_became_active":"2024-09-10T18:49:19.626266+0000","last_became_peered":"2024-09-10T18:49:19.626266+0000","last_unstale":"2024-09-10T19:07:27.750466+0000","last_undegraded":"2024-09-10T19:07:27.750466+0000","last_fullsized":"2024-09-10T19:07:27.750466+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'19","last_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_deep_scrub":"676'19","last_deep_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_clean_scrub_stamp":"2024-09-10T19:05:34.022377+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006303052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":794,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691722+0000","last_change":"2024-09-10T19:05:06.482047+0000","last_active":"2024-09-10T19:07:27.691722+0000","last_peered":"2024-09-10T19:07:27.691722+0000","last_clean":"2024-09-10T19:07:27.691722+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:07:27.691722+0000","last_undegraded":"2024-09-10T19:07:27.691722+0000","last_fullsized":"2024-09-10T19:07:27.691722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:06.481823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:06.481823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035879710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":782,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425919+0000","last_change":"2024-09-10T19:05:18.965121+0000","last_active":"2024-09-10T19:07:26.425919+0000","last_peered":"2024-09-10T19:07:26.425919+0000","last_clean":"2024-09-10T19:07:26.425919+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:07:26.425919+0000","last_undegraded":"2024-09-10T19:07:26.425919+0000","last_fullsized":"2024-09-10T19:07:26.425919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:18.964844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:18.964844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040517590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":790,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426554+0000","last_change":"2024-09-10T19:06:46.126997+0000","last_active":"2024-09-10T19:07:26.426554+0000","last_peered":"2024-09-10T19:07:26.426554+0000","last_clean":"2024-09-10T19:07:26.426554+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:07:26.426554+0000","last_undegraded":"2024-09-10T19:07:26.426554+0000","last_fullsized":"2024-09-10T19:07:26.426554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.126832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.126832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:01.544637+0000","scrub_duration":0.003551504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":705,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755568+0000","last_change":"2024-09-10T19:06:46.381092+0000","last_active":"2024-09-10T19:07:27.755568+0000","last_peered":"2024-09-10T19:07:27.755568+0000","last_clean":"2024-09-10T19:07:27.755568+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:07:27.755568+0000","last_undegraded":"2024-09-10T19:07:27.755568+0000","last_fullsized":"2024-09-10T19:07:27.755568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.380935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.181442+0000","scrub_duration":0.0032731539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":793,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.419041+0000","last_change":"2024-09-10T19:06:22.919675+0000","last_active":"2024-09-10T19:07:27.419041+0000","last_peered":"2024-09-10T19:07:27.419041+0000","last_clean":"2024-09-10T19:07:27.419041+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:07:27.419041+0000","last_undegraded":"2024-09-10T19:07:27.419041+0000","last_fullsized":"2024-09-10T19:07:27.419041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:06:22.919527+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.919527+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:32.618775+0000","scrub_duration":0.002938792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":859,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429467+0000","last_change":"2024-09-10T19:06:56.248053+0000","last_active":"2024-09-10T19:07:26.429467+0000","last_peered":"2024-09-10T19:07:26.429467+0000","last_clean":"2024-09-10T19:07:26.429467+0000","last_became_active":"2024-09-10T18:49:19.633639+0000","last_became_peered":"2024-09-10T18:49:19.633639+0000","last_unstale":"2024-09-10T19:07:26.429467+0000","last_undegraded":"2024-09-10T19:07:26.429467+0000","last_fullsized":"2024-09-10T19:07:26.429467+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:56.247893+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:06:56.247893+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:57.638950+0000","scrub_duration":0.0050558469999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":792,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.751005+0000","last_change":"2024-09-10T19:05:15.161330+0000","last_active":"2024-09-10T19:07:27.751005+0000","last_peered":"2024-09-10T19:07:27.751005+0000","last_clean":"2024-09-10T19:07:27.751005+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:07:27.751005+0000","last_undegraded":"2024-09-10T19:07:27.751005+0000","last_fullsized":"2024-09-10T19:07:27.751005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.161170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.161170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034575059999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":786,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.755375+0000","last_change":"2024-09-10T19:05:21.349080+0000","last_active":"2024-09-10T19:07:27.755375+0000","last_peered":"2024-09-10T19:07:27.755375+0000","last_clean":"2024-09-10T19:07:27.755375+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:07:27.755375+0000","last_undegraded":"2024-09-10T19:07:27.755375+0000","last_fullsized":"2024-09-10T19:07:27.755375+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:21.348970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:05:21.348970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002650716,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691240+0000","last_change":"2024-09-10T19:05:30.533679+0000","last_active":"2024-09-10T19:07:27.691240+0000","last_peered":"2024-09-10T19:07:27.691240+0000","last_clean":"2024-09-10T19:07:27.691240+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:07:27.691240+0000","last_undegraded":"2024-09-10T19:07:27.691240+0000","last_fullsized":"2024-09-10T19:07:27.691240+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:30.533525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.533525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033150879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427788+0000","last_change":"2024-09-10T19:06:19.011016+0000","last_active":"2024-09-10T19:07:26.427788+0000","last_peered":"2024-09-10T19:07:26.427788+0000","last_clean":"2024-09-10T19:07:26.427788+0000","last_became_active":"2024-09-10T18:49:24.054338+0000","last_became_peered":"2024-09-10T18:49:24.054338+0000","last_unstale":"2024-09-10T19:07:26.427788+0000","last_undegraded":"2024-09-10T19:07:26.427788+0000","last_fullsized":"2024-09-10T19:07:26.427788+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.010845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.010845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:35.834254+0000","scrub_duration":0.0035581089999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":819,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.429858+0000","last_change":"2024-09-10T19:06:34.054784+0000","last_active":"2024-09-10T19:07:26.429858+0000","last_peered":"2024-09-10T19:07:26.429858+0000","last_clean":"2024-09-10T19:07:26.429858+0000","last_became_active":"2024-09-10T18:49:19.626933+0000","last_became_peered":"2024-09-10T18:49:19.626933+0000","last_unstale":"2024-09-10T19:07:26.429858+0000","last_undegraded":"2024-09-10T19:07:26.429858+0000","last_fullsized":"2024-09-10T19:07:26.429858+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:06:34.054632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.054632+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:34.984618+0000","scrub_duration":0.0038029330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":796,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.427255+0000","last_change":"2024-09-10T19:06:52.165777+0000","last_active":"2024-09-10T19:07:26.427255+0000","last_peered":"2024-09-10T19:07:26.427255+0000","last_clean":"2024-09-10T19:07:26.427255+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:07:26.427255+0000","last_undegraded":"2024-09-10T19:07:26.427255+0000","last_fullsized":"2024-09-10T19:07:26.427255+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.165627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:18.581502+0000","scrub_duration":0.00328166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":1045,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691683+0000","last_change":"2024-09-10T19:06:50.856618+0000","last_active":"2024-09-10T19:07:27.691683+0000","last_peered":"2024-09-10T19:07:27.691683+0000","last_clean":"2024-09-10T19:07:27.691683+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:07:27.691683+0000","last_undegraded":"2024-09-10T19:07:27.691683+0000","last_fullsized":"2024-09-10T19:07:27.691683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:06:50.856451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.856451+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:12.647430+0000","scrub_duration":0.0045245329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.691614+0000","last_change":"2024-09-10T19:05:26.559541+0000","last_active":"2024-09-10T19:07:27.691614+0000","last_peered":"2024-09-10T19:07:27.691614+0000","last_clean":"2024-09-10T19:07:27.691614+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:07:27.691614+0000","last_undegraded":"2024-09-10T19:07:27.691614+0000","last_fullsized":"2024-09-10T19:07:27.691614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:26.559381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:05:26.559381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00334798,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":805,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426021+0000","last_change":"2024-09-10T19:06:36.751344+0000","last_active":"2024-09-10T19:07:26.426021+0000","last_peered":"2024-09-10T19:07:26.426021+0000","last_clean":"2024-09-10T19:07:26.426021+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:07:26.426021+0000","last_undegraded":"2024-09-10T19:07:26.426021+0000","last_fullsized":"2024-09-10T19:07:26.426021+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.751254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.751254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:41.762396+0000","scrub_duration":0.01052581,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":803,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690409+0000","last_change":"2024-09-10T19:06:30.665495+0000","last_active":"2024-09-10T19:07:27.690409+0000","last_peered":"2024-09-10T19:07:27.690409+0000","last_clean":"2024-09-10T19:07:27.690409+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:07:27.690409+0000","last_undegraded":"2024-09-10T19:07:27.690409+0000","last_fullsized":"2024-09-10T19:07:27.690409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:06:30.665240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.665240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:40.794854+0000","scrub_duration":0.0058393220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":801,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.752739+0000","last_change":"2024-09-10T19:05:30.243873+0000","last_active":"2024-09-10T19:07:27.752739+0000","last_peered":"2024-09-10T19:07:27.752739+0000","last_clean":"2024-09-10T19:07:27.752739+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:07:27.752739+0000","last_undegraded":"2024-09-10T19:07:27.752739+0000","last_fullsized":"2024-09-10T19:07:27.752739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:05:30.243807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.243807+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031694280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":822,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.314535+0000","last_change":"2024-09-10T19:06:42.199376+0000","last_active":"2024-09-10T19:07:27.314535+0000","last_peered":"2024-09-10T19:07:27.314535+0000","last_clean":"2024-09-10T19:07:27.314535+0000","last_became_active":"2024-09-10T18:49:22.048289+0000","last_became_peered":"2024-09-10T18:49:22.048289+0000","last_unstale":"2024-09-10T19:07:27.314535+0000","last_undegraded":"2024-09-10T19:07:27.314535+0000","last_fullsized":"2024-09-10T19:07:27.314535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:42.199204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.968019+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.199204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:07.125044+0000","scrub_duration":0.0034375769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":709,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.753533+0000","last_change":"2024-09-10T19:05:31.067457+0000","last_active":"2024-09-10T19:07:27.753533+0000","last_peered":"2024-09-10T19:07:27.753533+0000","last_clean":"2024-09-10T19:07:27.753533+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:07:27.753533+0000","last_undegraded":"2024-09-10T19:07:27.753533+0000","last_fullsized":"2024-09-10T19:07:27.753533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:31.067311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:05:31.067311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030309170000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":790,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426433+0000","last_change":"2024-09-10T19:06:19.090138+0000","last_active":"2024-09-10T19:07:26.426433+0000","last_peered":"2024-09-10T19:07:26.426433+0000","last_clean":"2024-09-10T19:07:26.426433+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:07:26.426433+0000","last_undegraded":"2024-09-10T19:07:26.426433+0000","last_fullsized":"2024-09-10T19:07:26.426433+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.089886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.089886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:42.552467+0000","scrub_duration":0.0040961510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":796,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.426916+0000","last_change":"2024-09-10T19:06:31.070915+0000","last_active":"2024-09-10T19:07:26.426916+0000","last_peered":"2024-09-10T19:07:26.426916+0000","last_clean":"2024-09-10T19:07:26.426916+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:07:26.426916+0000","last_undegraded":"2024-09-10T19:07:26.426916+0000","last_fullsized":"2024-09-10T19:07:26.426916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.070745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.070745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:54.012535+0000","scrub_duration":0.0036538719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":834,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425347+0000","last_change":"2024-09-10T19:06:23.971596+0000","last_active":"2024-09-10T19:07:26.425347+0000","last_peered":"2024-09-10T19:07:26.425347+0000","last_clean":"2024-09-10T19:07:26.425347+0000","last_became_active":"2024-09-10T18:49:19.619272+0000","last_became_peered":"2024-09-10T18:49:19.619272+0000","last_unstale":"2024-09-10T19:07:26.425347+0000","last_undegraded":"2024-09-10T19:07:26.425347+0000","last_fullsized":"2024-09-10T19:07:26.425347+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:23.971482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.971482+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:07:33.794031+0000","scrub_duration":0.0045234610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":782,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:26.425738+0000","last_change":"2024-09-10T19:05:14.979053+0000","last_active":"2024-09-10T19:07:26.425738+0000","last_peered":"2024-09-10T19:07:26.425738+0000","last_clean":"2024-09-10T19:07:26.425738+0000","last_became_active":"2024-09-10T18:49:22.042211+0000","last_became_peered":"2024-09-10T18:49:22.042211+0000","last_unstale":"2024-09-10T19:07:26.425738+0000","last_undegraded":"2024-09-10T19:07:26.425738+0000","last_fullsized":"2024-09-10T19:07:26.425738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:14.978923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:05:14.978923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003307857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":711,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-09-10T19:07:27.690772+0000","last_change":"2024-09-10T19:05:33.492062+0000","last_active":"2024-09-10T19:07:27.690772+0000","last_peered":"2024-09-10T19:07:27.690772+0000","last_clean":"2024-09-10T19:07:27.690772+0000","last_became_active":"2024-09-10T18:54:37.706540+0000","last_became_peered":"2024-09-10T18:54:37.706540+0000","last_unstale":"2024-09-10T19:07:27.690772+0000","last_undegraded":"2024-09-10T19:07:27.690772+0000","last_fullsized":"2024-09-10T19:07:27.690772+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:33.491961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:51.945980+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.491961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025640609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":256,"num_pg":32,"stat_sum":{"num_bytes":49236,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":147708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":241,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1066,"num_read_kb":1061,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1128,"num_read_kb":885,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":776,"ondisk_log_size":776,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972582,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304292,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440860,"statfs":{"total":95995035648,"available":95683440640,"internally_reserved":0,"allocated":2842624,"data_stored":1278732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.93600000000000005,"15min":0.90800000000000003},"min":{"1min":0.48399999999999999,"5min":0.432,"15min":0.311},"max":{"1min":2.0649999999999999,"5min":2.839,"15min":2.839},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.97999999999999998,"15min":0.95099999999999996},"min":{"1min":0.57099999999999995,"5min":0.497,"15min":0.34699999999999998},"max":{"1min":2.9550000000000001,"5min":2.9550000000000001,"15min":4.7000000000000002},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95599999999999996,"15min":0.90500000000000003},"min":{"1min":0.51000000000000001,"5min":0.504,"15min":0.41199999999999998},"max":{"1min":3.097,"5min":4.9299999999999997,"15min":5.6539999999999999},"last":1.038},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":0.96199999999999997},"min":{"1min":0.622,"5min":0.57899999999999996,"15min":0.31900000000000001},"max":{"1min":1.3420000000000001,"5min":2.012,"15min":2.012},"last":0.77600000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.302,"15min":1.1040000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.41499999999999998},"max":{"1min":1.385,"5min":31.166,"15min":31.166},"last":1.2589999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.2569999999999999,"15min":1.087},"min":{"1min":0.71499999999999997,"5min":0.55900000000000005,"15min":0.443},"max":{"1min":1.7529999999999999,"5min":27.709,"15min":27.709},"last":0.88400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.0289999999999999,"15min":1.0309999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.48299999999999998},"max":{"1min":1.2729999999999999,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":1.131},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.98699999999999999,"15min":0.98299999999999998},"min":{"1min":0.70199999999999996,"5min":0.626,"15min":0.46600000000000003},"max":{"1min":1.3049999999999999,"5min":4.2060000000000004,"15min":5.6050000000000004},"last":0.84699999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.0169999999999999,"15min":1.016},"min":{"1min":0.57599999999999996,"5min":0.51800000000000002,"15min":0.34799999999999998},"max":{"1min":1.3500000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.105},{"interface":"front","average":{"1min":1.107,"5min":1.0700000000000001,"15min":1.081},"min":{"1min":0.73299999999999998,"5min":0.59499999999999997,"15min":0.48099999999999998},"max":{"1min":1.421,"5min":2.25,"15min":4.2400000000000002},"last":0.82299999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.97299999999999998,"15min":0.98499999999999999},"min":{"1min":0.81100000000000005,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.228,"5min":1.5980000000000001,"15min":5.4710000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.042,"5min":1.1459999999999999,"15min":1.1499999999999999},"min":{"1min":0.58299999999999996,"5min":0.57699999999999996,"15min":0.47499999999999998},"max":{"1min":1.4039999999999999,"5min":4.8710000000000004,"15min":4.8710000000000004},"last":0.90900000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1679999999999999,"15min":1.177},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.48699999999999999},"max":{"1min":1.444,"5min":1.766,"15min":1.766},"last":1.21},{"interface":"front","average":{"1min":1.127,"5min":1.153,"15min":1.1890000000000001},"min":{"1min":0.83899999999999997,"5min":0.61399999999999999,"15min":0.58099999999999996},"max":{"1min":1.353,"5min":2.4340000000000002,"15min":4.1310000000000002},"last":1.075}]}]},{"osd":6,"up_from":52,"seq":223338299632,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303884,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441268,"statfs":{"total":95995035648,"available":95683858432,"internally_reserved":0,"allocated":2420736,"data_stored":879575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.90700000000000003,"15min":0.88600000000000001},"min":{"1min":0.53000000000000003,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.575,"5min":1.696,"15min":2.0550000000000002},"last":1.01},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.96499999999999997,"15min":0.96999999999999997},"min":{"1min":0.65300000000000002,"5min":0.438,"15min":0.36399999999999999},"max":{"1min":1.262,"5min":1.6859999999999999,"15min":6.8959999999999999},"last":0.74099999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0820000000000001,"15min":1.091},"min":{"1min":0.64800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":18.177,"5min":18.177,"15min":18.494},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.696,"5min":1.0469999999999999,"15min":0.95099999999999996},"min":{"1min":0.69899999999999995,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":17.297000000000001,"5min":17.297000000000001,"15min":17.297000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.94299999999999995,"15min":0.97399999999999998},"min":{"1min":0.59199999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3480000000000001,"5min":1.5589999999999999,"15min":9.4800000000000004},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.90900000000000003,"15min":0.93799999999999994},"min":{"1min":0.62,"5min":0.52300000000000002,"15min":0.46500000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":6.0970000000000004},"last":1.294}]},{"osd":3,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":0.99099999999999999,"15min":1.012},"min":{"1min":0.71099999999999997,"5min":0.46700000000000003,"15min":0.40400000000000003},"max":{"1min":1.538,"5min":1.8700000000000001,"15min":2.278},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.95099999999999996,"15min":0.98299999999999998},"min":{"1min":0.71999999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.2230000000000001,"5min":1.4730000000000001,"15min":3.4279999999999999},"last":0.70399999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":0.96199999999999997,"15min":0.98299999999999998},"min":{"1min":0.44900000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.3049999999999999,"5min":1.512,"15min":6.008},"last":1.141},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0349999999999999,"15min":1.075},"min":{"1min":0.58699999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.492,"5min":2.0619999999999998,"15min":2.3079999999999998},"last":0.88200000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.036,"15min":1.0669999999999999},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.46300000000000002},"max":{"1min":1.415,"5min":2.02,"15min":5.9560000000000004},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.085,"15min":1.135},"min":{"1min":0.56599999999999995,"5min":0.47599999999999998,"15min":0.46500000000000002},"max":{"1min":1.611,"5min":2.145,"15min":3.5169999999999999},"last":1.1779999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.032,"15min":1.0649999999999999},"min":{"1min":0.56100000000000005,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3520000000000001,"5min":1.8320000000000001,"15min":2.069},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.161,"15min":1.2030000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6399999999999999,"5min":1.7789999999999999,"15min":6.0380000000000003},"last":1.0409999999999999}]}]},{"osd":5,"up_from":43,"seq":184683593978,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302280,"kb_used_data":4920,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442872,"statfs":{"total":95995035648,"available":95685500928,"internally_reserved":0,"allocated":5038080,"data_stored":3493441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.0169999999999999,"15min":1.0129999999999999},"min":{"1min":0.55400000000000005,"5min":0.46500000000000002,"15min":0.36399999999999999},"max":{"1min":1.52,"5min":5.4640000000000004,"15min":8.3949999999999996},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.98599999999999999,"15min":0.96199999999999997},"min":{"1min":0.69099999999999995,"5min":0.53100000000000003,"15min":0.39700000000000002},"max":{"1min":1.306,"5min":5.4790000000000001,"15min":7.3070000000000004},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.014,"15min":1.2070000000000001},"min":{"1min":0.60399999999999998,"5min":0.56100000000000005,"15min":0.41199999999999998},"max":{"1min":1.5609999999999999,"5min":6.282,"15min":31.548999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.026,"15min":1.1180000000000001},"min":{"1min":0.72299999999999998,"5min":0.54800000000000004,"15min":0.48899999999999999},"max":{"1min":1.5900000000000001,"5min":6.0170000000000003,"15min":31.847000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0600000000000001,"15min":1.3009999999999999},"min":{"1min":0.78600000000000003,"5min":0.66200000000000003,"15min":0.39900000000000002},"max":{"1min":1.341,"5min":7.9340000000000002,"15min":61.347000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0680000000000001,"15min":1.0609999999999999},"min":{"1min":0.77300000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.46,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":1.0149999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0089999999999999,"15min":1.044},"min":{"1min":0.76100000000000001,"5min":0.60199999999999998,"15min":0.51700000000000002},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":7.5979999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.135,"15min":1.3140000000000001},"min":{"1min":0.872,"5min":0.61599999999999999,"15min":0.60799999999999998},"max":{"1min":1.4019999999999999,"5min":5.3810000000000002,"15min":53.173000000000002},"last":1.101}]},{"osd":4,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.99199999999999999,"15min":1.0289999999999999},"min":{"1min":0.64000000000000001,"5min":0.59499999999999997,"15min":0.36499999999999999},"max":{"1min":1.5129999999999999,"5min":5.5170000000000003,"15min":13.005000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.98099999999999998,"15min":1.0609999999999999},"min":{"1min":0.73799999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.482,"5min":1.546,"15min":7.6299999999999999},"last":1.073}]},{"osd":6,"last update":"Tue Sep 10 19:07:12 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.1499999999999999,"15min":1.204},"min":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.56399999999999995},"max":{"1min":1.3859999999999999,"5min":4.2880000000000003,"15min":7.7199999999999998},"last":1.238},{"interface":"front","average":{"1min":1.107,"5min":1.1899999999999999,"15min":1.246},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.476,"5min":5.4240000000000004,"15min":9.6760000000000002},"last":1.2150000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:07:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1619999999999999,"15min":1.252},"min":{"1min":0.66700000000000004,"5min":0.46700000000000003,"15min":0.43099999999999999},"max":{"1min":1.6899999999999999,"5min":5.5510000000000002,"15min":15.613},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2709999999999999},"min":{"1min":0.72199999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.361,"5min":1.718,"15min":16.965},"last":1.125}]}]},{"osd":4,"up_from":35,"seq":150323855622,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308100,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437052,"statfs":{"total":95995035648,"available":95679541248,"internally_reserved":0,"allocated":2482176,"data_stored":909571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.88900000000000001,"15min":0.85999999999999999},"min":{"1min":0.52900000000000003,"5min":0.46899999999999997,"15min":0.371},"max":{"1min":1.357,"5min":2.0179999999999998,"15min":2.0179999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.998,"5min":1.0029999999999999,"15min":0.94699999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.44600000000000001},"max":{"1min":1.7609999999999999,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.84499999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.90800000000000003,"15min":0.98199999999999998},"min":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.42399999999999999},"max":{"1min":1.677,"5min":1.677,"15min":27.716000000000001},"last":0.997},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98099999999999998,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.38100000000000001},"max":{"1min":1.4950000000000001,"5min":1.865,"15min":27.93},"last":1.0349999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":1.2230000000000001,"15min":1.0429999999999999},"min":{"1min":0.68500000000000005,"5min":0.50800000000000001,"15min":0.46600000000000003},"max":{"1min":1.3879999999999999,"5min":18.853000000000002,"15min":18.853000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.2849999999999999,"15min":1.0669999999999999},"min":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.497},"max":{"1min":1.587,"5min":25.652999999999999,"15min":25.652999999999999},"last":1.26}]},{"osd":3,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0329999999999999,"15min":1},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.46300000000000002},"max":{"1min":1.6879999999999999,"5min":1.923,"15min":4.2670000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.03,"15min":0.995},"min":{"1min":0.65700000000000003,"5min":0.622,"15min":0.442},"max":{"1min":1.5649999999999999,"5min":1.873,"15min":2.0739999999999998},"last":0.92100000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:07:22 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95999999999999996,"15min":0.97199999999999998},"min":{"1min":0.60399999999999998,"5min":0.45000000000000001,"15min":0.38700000000000001},"max":{"1min":1.339,"5min":1.6060000000000001,"15min":1.8009999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.1000000000000001,"15min":1.0720000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.52600000000000002},"max":{"1min":1.401,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.1890000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:07:22 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0549999999999999,"15min":1.0489999999999999},"min":{"1min":0.69699999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.593,"5min":1.867,"15min":2.1699999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.1839999999999999,"15min":1.1739999999999999},"min":{"1min":0.83999999999999997,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.55,"5min":1.9890000000000001,"15min":2.9590000000000001},"last":1.103}]},{"osd":7,"last update":"Tue Sep 10 19:07:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.218,"15min":1.1739999999999999},"min":{"1min":0.65000000000000002,"5min":0.56999999999999995,"15min":0.46899999999999997},"max":{"1min":1.819,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.069},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.224,"15min":1.1890000000000001},"min":{"1min":0.81200000000000006,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.913,"5min":3.7850000000000001,"15min":3.7850000000000001},"last":1.228}]}]},{"osd":3,"up_from":29,"seq":124554051857,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300344,"kb_used_data":2988,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93444808,"statfs":{"total":95995035648,"available":95687483392,"internally_reserved":0,"allocated":3059712,"data_stored":1511913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.68899999999999995,"15min":0.85099999999999998},"min":{"1min":0.42399999999999999,"5min":0.313,"15min":0.222},"max":{"1min":0.92700000000000005,"5min":1.6539999999999999,"15min":26.594000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.73899999999999999,"15min":0.89900000000000002},"min":{"1min":0.48799999999999999,"5min":0.40400000000000003,"15min":0.252},"max":{"1min":1.139,"5min":1.4790000000000001,"15min":24.204999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70599999999999996,"15min":0.92200000000000004},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.26500000000000001},"max":{"1min":1.101,"5min":1.3240000000000001,"15min":53.661999999999999},"last":0.624},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77900000000000003,"15min":0.88300000000000001},"min":{"1min":0.60199999999999998,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":1.071,"5min":1.7170000000000001,"15min":15.189},"last":0.96899999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.88400000000000001,"15min":1.0209999999999999},"min":{"1min":0.51400000000000001,"5min":0.37,"15min":0.32200000000000001},"max":{"1min":2.073,"5min":2.073,"15min":26.558},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78300000000000003,"15min":0.95199999999999996},"min":{"1min":0.56599999999999995,"5min":0.497,"15min":0.26900000000000002},"max":{"1min":1.034,"5min":1.5489999999999999,"15min":25.536999999999999},"last":0.81100000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:06:34 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.095,"15min":1.1679999999999999},"min":{"1min":0.84599999999999997,"5min":0.629,"15min":0.46999999999999997},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":10.359999999999999},"last":1.151},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.202,"15min":1.27},"min":{"1min":0.81799999999999995,"5min":0.64400000000000002,"15min":0.53500000000000003},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":10.391},"last":1.4530000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.1559999999999999,"15min":1.2330000000000001},"min":{"1min":0.755,"5min":0.67000000000000004,"15min":0.501},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":17.535},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.052,"15min":1.0740000000000001},"min":{"1min":0.72299999999999998,"5min":0.53500000000000003,"15min":0.45900000000000002},"max":{"1min":1.55,"5min":2.0529999999999999,"15min":8.7880000000000003},"last":0.88100000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.2270000000000001,"15min":1.2110000000000001},"min":{"1min":0.78700000000000003,"5min":0.72199999999999998,"15min":0.55600000000000005},"max":{"1min":2.206,"5min":2.206,"15min":10.51},"last":1.399},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1299999999999999,"15min":1.115},"min":{"1min":0.73899999999999999,"5min":0.64200000000000002,"15min":0.48499999999999999},"max":{"1min":1.9219999999999999,"5min":1.9259999999999999,"15min":10.773999999999999},"last":1.349}]},{"osd":7,"last update":"Tue Sep 10 19:07:10 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.1419999999999999,"15min":1.1799999999999999},"min":{"1min":0.84699999999999998,"5min":0.61499999999999999,"15min":0.45600000000000002},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":18.312999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.278,"15min":1.327},"min":{"1min":0.89000000000000001,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":10.611000000000001},"last":1.496}]}]},{"osd":2,"up_from":19,"seq":81604378906,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308380,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93436772,"statfs":{"total":95995035648,"available":95679254528,"internally_reserved":0,"allocated":2768896,"data_stored":1213097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":1.153,"15min":0.89200000000000002},"min":{"1min":0.41599999999999998,"5min":0.28499999999999998,"15min":0.219},"max":{"1min":1.0880000000000001,"5min":43.927999999999997,"15min":43.927999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.875,"5min":1.3819999999999999,"15min":0.98499999999999999},"min":{"1min":0.48199999999999998,"5min":0.33000000000000002,"15min":0.23599999999999999},"max":{"1min":1.3029999999999999,"5min":49.475000000000001,"15min":49.475000000000001},"last":1.046}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78400000000000003,"15min":0.751},"min":{"1min":0.51800000000000002,"5min":0.36499999999999999,"15min":0.26600000000000001},"max":{"1min":0.996,"5min":3.524,"15min":5.516},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.56799999999999995,"5min":0.40300000000000002,"15min":0.28299999999999997},"max":{"1min":1.135,"5min":1.77,"15min":9.0649999999999995},"last":0.624}]},{"osd":3,"last update":"Tue Sep 10 19:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":0.97199999999999998,"15min":0.90500000000000003},"min":{"1min":0.67500000000000004,"5min":0.318,"15min":0.318},"max":{"1min":6.4020000000000001,"5min":6.4020000000000001,"15min":6.4020000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.095,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":8.3710000000000004,"5min":9.8219999999999992,"15min":15.736000000000001},"last":0.752}]},{"osd":4,"last update":"Tue Sep 10 19:06:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.4419999999999999,"15min":1.23},"min":{"1min":0.59699999999999998,"5min":0.44900000000000001,"15min":0.39300000000000002},"max":{"1min":1.619,"5min":38.079999999999998,"15min":38.079999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.4470000000000001,"15min":1.276},"min":{"1min":0.76600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.627,"5min":37.991,"15min":37.991},"last":1.0660000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.129,"15min":1.1040000000000001},"min":{"1min":0.79400000000000004,"5min":0.49099999999999999,"15min":0.38500000000000001},"max":{"1min":1.514,"5min":1.7749999999999999,"15min":5.1959999999999997},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.212,"15min":1.1699999999999999},"min":{"1min":0.76000000000000001,"5min":0.60899999999999999,"15min":0.45400000000000001},"max":{"1min":1.8979999999999999,"5min":6.0620000000000003,"15min":21.654},"last":0.89700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.581,"15min":1.3},"min":{"1min":0.81000000000000005,"5min":0.60499999999999998,"15min":0.503},"max":{"1min":1.5449999999999999,"5min":43.655999999999999,"15min":43.655999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.5920000000000001,"15min":1.3109999999999999},"min":{"1min":0.64700000000000002,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.54,"5min":45.255000000000003,"15min":45.255000000000003},"last":0.96299999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:07:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.2669999999999999,"15min":1.2010000000000001},"min":{"1min":0.80100000000000005,"5min":0.67800000000000005,"15min":0.48399999999999999},"max":{"1min":1.9990000000000001,"5min":6.0179999999999998,"15min":15.025},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.2310000000000001,"15min":1.2430000000000001},"min":{"1min":0.78400000000000003,"5min":0.63400000000000001,"15min":0.53100000000000003},"max":{"1min":1.5549999999999999,"5min":6.048,"15min":14.816000000000001},"last":1.0820000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305848,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439304,"statfs":{"total":95995035648,"available":95681847296,"internally_reserved":0,"allocated":4435968,"data_stored":2891059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.70899999999999996,"15min":0.83699999999999997},"min":{"1min":0.46600000000000003,"5min":0.32900000000000001,"15min":0.23499999999999999},"max":{"1min":3.96,"5min":3.96,"15min":32.677999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.748,"15min":0.89600000000000002},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.253},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":32.398000000000003},"last":1.1020000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.72899999999999998,"15min":0.89600000000000002},"min":{"1min":0.432,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":26.097000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.996,"5min":0.85899999999999999,"15min":1.0149999999999999},"min":{"1min":0.45400000000000001,"5min":0.437,"15min":0.26000000000000001},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":32.363},"last":0.79900000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.5649999999999999,"15min":1.252},"min":{"1min":0.58499999999999996,"5min":0.309,"15min":0.28899999999999998},"max":{"1min":2.903,"5min":53.344999999999999,"15min":53.344999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.4319999999999999,"15min":1.2769999999999999},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.26600000000000001},"max":{"1min":4.085,"5min":53.645000000000003,"15min":60.999000000000002},"last":0.752}]},{"osd":4,"last update":"Tue Sep 10 19:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0740000000000001,"15min":1.1419999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.40899999999999997},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":12.808999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.234,"5min":1.0049999999999999,"15min":1.2649999999999999},"min":{"1min":0.70099999999999996,"5min":0.57699999999999996,"15min":0.378},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":60.529000000000003},"last":0.92900000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.0309999999999999,"15min":1.143},"min":{"1min":0.71099999999999997,"5min":0.48899999999999999,"15min":0.40600000000000003},"max":{"1min":3.8410000000000002,"5min":3.8410000000000002,"15min":26.050999999999998},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.1439999999999999,"15min":1.224},"min":{"1min":0.68700000000000006,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":3.0310000000000001,"5min":3.0310000000000001,"15min":12.84},"last":0.95099999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1619999999999999,"15min":1.3200000000000001},"min":{"1min":0.82799999999999996,"5min":0.63200000000000001,"15min":0.53800000000000003},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":25.608000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.085,"15min":1.2569999999999999},"min":{"1min":0.65200000000000002,"5min":0.57699999999999996,"15min":0.46400000000000002},"max":{"1min":4.1710000000000003,"5min":4.1710000000000003,"15min":33.643000000000001},"last":1.165}]},{"osd":7,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2070000000000001,"15min":1.341},"min":{"1min":0.746,"5min":0.746,"15min":0.48499999999999999},"max":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":25.565999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.1120000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":4.218,"5min":4.218,"15min":33.673999999999999},"last":1.216}]}]},{"osd":0,"up_from":8,"seq":34359738673,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301720,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93443432,"statfs":{"total":95995035648,"available":95686074368,"internally_reserved":0,"allocated":4464640,"data_stored":2919315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72599999999999998,"15min":1.032},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26300000000000001},"max":{"1min":0.89400000000000002,"5min":2.133,"15min":80.710999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80700000000000005,"15min":1.1160000000000001},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.419,"15min":86.009},"last":0.98199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77400000000000002,"15min":1.0569999999999999},"min":{"1min":0.55000000000000004,"5min":0.36799999999999999,"15min":0.216},"max":{"1min":1.0549999999999999,"5min":2.3530000000000002,"15min":36.682000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.876,"5min":0.81399999999999995,"15min":0.95499999999999996},"min":{"1min":0.63700000000000001,"5min":0.41299999999999998,"15min":0.23599999999999999},"max":{"1min":1.294,"5min":1.651,"15min":24.614000000000001},"last":1.0669999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.81499999999999995,"15min":0.93300000000000005},"min":{"1min":0.61699999999999999,"5min":0.46300000000000002,"15min":0.26400000000000001},"max":{"1min":1.518,"5min":1.7849999999999999,"15min":21.620999999999999},"last":1.79},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.875,"15min":1.069},"min":{"1min":0.68000000000000005,"5min":0.42499999999999999,"15min":0.30299999999999999},"max":{"1min":1.198,"5min":2.4569999999999999,"15min":45.765000000000001},"last":0.94599999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:06:34 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0840000000000001,"15min":1.103},"min":{"1min":0.81999999999999995,"5min":0.53900000000000003,"15min":0.5},"max":{"1min":1.3799999999999999,"5min":2.4239999999999999,"15min":6.3630000000000004},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.1639999999999999,"15min":1.1439999999999999},"min":{"1min":0.93200000000000005,"5min":0.626,"15min":0.51100000000000001},"max":{"1min":1.498,"5min":1.8979999999999999,"15min":4.2389999999999999},"last":1.4239999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:07:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.1140000000000001,"15min":1.155},"min":{"1min":0.746,"5min":0.66200000000000003,"15min":0.442},"max":{"1min":1.3109999999999999,"5min":2.5880000000000001,"15min":10.048},"last":1.611},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.1950000000000001,"15min":1.145},"min":{"1min":0.73399999999999999,"5min":0.62,"15min":0.42999999999999999},"max":{"1min":1.504,"5min":1.5840000000000001,"15min":3.8959999999999999},"last":1.643}]},{"osd":6,"last update":"Tue Sep 10 19:07:20 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.238,"15min":1.1919999999999999},"min":{"1min":0.80900000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.615,"5min":1.71,"15min":3.9550000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1839999999999999,"15min":1.2250000000000001},"min":{"1min":0.70899999999999996,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.45,"5min":2.5339999999999998,"15min":11.894},"last":1.5229999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.2090000000000001,"15min":1.194},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.48499999999999999},"max":{"1min":1.5580000000000001,"5min":2.343,"15min":6.2949999999999999},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.254,"5min":1.2769999999999999,"15min":1.284},"min":{"1min":0.68100000000000005,"5min":0.56000000000000005,"15min":0.54900000000000004},"max":{"1min":1.5509999999999999,"5min":2.0009999999999999,"15min":8.2840000000000007},"last":1.7330000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2043904,"data_stored":2043904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":2015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":2027520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":49236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":49236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":256,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":49236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:07:34.436 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:07:34.436 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:07:34.436 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:07:34.436 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:07:34.437 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:07:34.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:34.478 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:34.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:34.615 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:34.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:34.731 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:34 smithi083 systemd[1]: Stopping Ceph osd.1 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:07:34.751 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:34.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:34.890 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:34.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:34.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:35.025 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:35.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:34 smithi083 bash[42805]: debug 2024-09-10T19:07:34.781+0000 7f7db7fb1640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:07:35.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:34 smithi083 bash[42805]: debug 2024-09-10T19:07:34.781+0000 7f7db7fb1640 -1 osd.1 789 *** Got signal Terminated *** 2024-09-10T19:07:35.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:34 smithi083 bash[42805]: debug 2024-09-10T19:07:34.781+0000 7f7db7fb1640 -1 osd.1 789 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:07:35.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:35.165 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:35.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:35.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:35.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[18629]: cluster 2024-09-10T19:07:34.351230+0000 mgr.y (mgr.24526) 717 : cluster [DBG] pgmap v1221: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-09-10T19:07:35.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[18629]: cluster 2024-09-10T19:07:34.363102+0000 osd.6 (osd.6) 365 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:35.403 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[18629]: cluster 2024-09-10T19:07:34.363493+0000 osd.6 (osd.6) 366 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[18629]: cluster 2024-09-10T19:07:34.370774+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[18629]: cluster 2024-09-10T19:07:34.783918+0000 mon.a (mon.0) 3576 : cluster [INF] osd.1 marked itself down and dead 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[32381]: cluster 2024-09-10T19:07:34.351230+0000 mgr.y (mgr.24526) 717 : cluster [DBG] pgmap v1221: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[32381]: cluster 2024-09-10T19:07:34.363102+0000 osd.6 (osd.6) 365 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[32381]: cluster 2024-09-10T19:07:34.363493+0000 osd.6 (osd.6) 366 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[32381]: cluster 2024-09-10T19:07:34.370774+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-09-10T19:07:35.404 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:35 smithi083 bash[32381]: cluster 2024-09-10T19:07:34.783918+0000 mon.a (mon.0) 3576 : cluster [INF] osd.1 marked itself down and dead 2024-09-10T19:07:35.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:35.426 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:35.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:35.529 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:35.563 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:35.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:35.693 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:35.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:35.797 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:35.833 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:35 smithi165 bash[22591]: cluster 2024-09-10T19:07:34.351230+0000 mgr.y (mgr.24526) 717 : cluster [DBG] pgmap v1221: 214 pgs: 214 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 645 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-09-10T19:07:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:35 smithi165 bash[22591]: cluster 2024-09-10T19:07:34.363102+0000 osd.6 (osd.6) 365 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:07:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:35 smithi165 bash[22591]: cluster 2024-09-10T19:07:34.363493+0000 osd.6 (osd.6) 366 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:07:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:35 smithi165 bash[22591]: cluster 2024-09-10T19:07:34.370774+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-09-10T19:07:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:35 smithi165 bash[22591]: cluster 2024-09-10T19:07:34.783918+0000 mon.a (mon.0) 3576 : cluster [INF] osd.1 marked itself down and dead 2024-09-10T19:07:35.937 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:35.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:35.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:35.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:36.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.108 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:36.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:36.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.400 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:36.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:36.539 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:36.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:36 smithi083 bash[18629]: cluster 2024-09-10T19:07:35.365290+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:07:36.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:36 smithi083 bash[18629]: cluster 2024-09-10T19:07:35.374864+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e790: 8 total, 7 up, 8 in 2024-09-10T19:07:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:36 smithi083 bash[32381]: cluster 2024-09-10T19:07:35.365290+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:07:36.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:36 smithi083 bash[32381]: cluster 2024-09-10T19:07:35.374864+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e790: 8 total, 7 up, 8 in 2024-09-10T19:07:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:36.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:36.656 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:36 smithi165 bash[22591]: cluster 2024-09-10T19:07:35.365290+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:07:36.657 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:36 smithi165 bash[22591]: cluster 2024-09-10T19:07:35.374864+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e790: 8 total, 7 up, 8 in 2024-09-10T19:07:36.677 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:36.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.816 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:36.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:36.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:36.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:37.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:37.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:37.192 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:37.225 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:37.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:37.362 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:37.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:37.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:37.497 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:37.601 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:37 smithi165 bash[22591]: cluster 2024-09-10T19:07:36.353819+0000 mgr.y (mgr.24526) 718 : cluster [DBG] pgmap v1224: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 38 KiB/s wr, 6 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:37 smithi165 bash[22591]: cluster 2024-09-10T19:07:36.378831+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e791: 8 total, 7 up, 8 in 2024-09-10T19:07:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:37 smithi165 bash[22591]: cluster 2024-09-10T19:07:36.384770+0000 mon.a (mon.0) 3580 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-09-10T19:07:37.614 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[18629]: cluster 2024-09-10T19:07:36.353819+0000 mgr.y (mgr.24526) 718 : cluster [DBG] pgmap v1224: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 38 KiB/s wr, 6 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:37.614 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[18629]: cluster 2024-09-10T19:07:36.378831+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e791: 8 total, 7 up, 8 in 2024-09-10T19:07:37.615 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[18629]: cluster 2024-09-10T19:07:36.384770+0000 mon.a (mon.0) 3580 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-09-10T19:07:37.615 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[32381]: cluster 2024-09-10T19:07:36.353819+0000 mgr.y (mgr.24526) 718 : cluster [DBG] pgmap v1224: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.3 KiB/s rd, 38 KiB/s wr, 6 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:37.615 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[32381]: cluster 2024-09-10T19:07:36.378831+0000 mon.a (mon.0) 3579 : cluster [DBG] osdmap e791: 8 total, 7 up, 8 in 2024-09-10T19:07:37.615 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[32381]: cluster 2024-09-10T19:07:36.384770+0000 mon.a (mon.0) 3580 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 24 pgs peering (PG_AVAILABILITY) 2024-09-10T19:07:37.633 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:37.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:37.772 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:37.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:37.911 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:37.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:38.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:38.044 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:38.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:38.181 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:38.227 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:37 smithi083 bash[204655]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:07:38.228 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[204993]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:07:38.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.319 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:38.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.447 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[32381]: audit 2024-09-10T19:07:37.486119+0000 mon.c (mon.2) 819 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[32381]: audit 2024-09-10T19:07:37.486795+0000 mon.a (mon.0) 3581 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[32381]: audit 2024-09-10T19:07:37.487786+0000 mon.c (mon.2) 820 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[32381]: audit 2024-09-10T19:07:37.488336+0000 mon.a (mon.0) 3582 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[18629]: audit 2024-09-10T19:07:37.486119+0000 mon.c (mon.2) 819 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[18629]: audit 2024-09-10T19:07:37.486795+0000 mon.a (mon.0) 3581 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[18629]: audit 2024-09-10T19:07:37.487786+0000 mon.c (mon.2) 820 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:38 smithi083 bash[18629]: audit 2024-09-10T19:07:37.488336+0000 mon.a (mon.0) 3582 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:38.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:38.564 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:38 smithi165 bash[22591]: audit 2024-09-10T19:07:37.486119+0000 mon.c (mon.2) 819 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:38 smithi165 bash[22591]: audit 2024-09-10T19:07:37.486795+0000 mon.a (mon.0) 3581 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:38.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:38 smithi165 bash[22591]: audit 2024-09-10T19:07:37.487786+0000 mon.c (mon.2) 820 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.565 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:38 smithi165 bash[22591]: audit 2024-09-10T19:07:37.488336+0000 mon.a (mon.0) 3582 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]: dispatch 2024-09-10T19:07:38.584 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:38.688 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:38.716 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:38.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.854 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:38.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:38.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:38.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:39.094 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:39.128 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:39.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:39.268 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:39.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:39.404 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:39.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:39.545 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:39.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:39.682 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:39.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:39.819 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:39.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:39.956 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:39.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:40.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.093 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[32381]: cluster 2024-09-10T19:07:38.355135+0000 mgr.y (mgr.24526) 719 : cluster [DBG] pgmap v1226: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 32 KiB/s wr, 4 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[32381]: audit 2024-09-10T19:07:38.417649+0000 mon.a (mon.0) 3583 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]': finished 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[32381]: cluster 2024-09-10T19:07:38.417738+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e792: 8 total, 7 up, 8 in 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[32381]: cluster 2024-09-10T19:07:39.375721+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e793: 8 total, 7 up, 8 in 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[18629]: cluster 2024-09-10T19:07:38.355135+0000 mgr.y (mgr.24526) 719 : cluster [DBG] pgmap v1226: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 32 KiB/s wr, 4 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:40.095 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[18629]: audit 2024-09-10T19:07:38.417649+0000 mon.a (mon.0) 3583 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]': finished 2024-09-10T19:07:40.096 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[18629]: cluster 2024-09-10T19:07:38.417738+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e792: 8 total, 7 up, 8 in 2024-09-10T19:07:40.096 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:39 smithi083 bash[18629]: cluster 2024-09-10T19:07:39.375721+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e793: 8 total, 7 up, 8 in 2024-09-10T19:07:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:39 smithi165 bash[22591]: cluster 2024-09-10T19:07:38.355135+0000 mgr.y (mgr.24526) 719 : cluster [DBG] pgmap v1226: 214 pgs: 72 peering, 142 active+clean; 4.5 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 32 KiB/s wr, 4 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-10T19:07:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:39 smithi165 bash[22591]: audit 2024-09-10T19:07:38.417649+0000 mon.a (mon.0) 3583 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-140"}]': finished 2024-09-10T19:07:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:39 smithi165 bash[22591]: cluster 2024-09-10T19:07:38.417738+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e792: 8 total, 7 up, 8 in 2024-09-10T19:07:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:39 smithi165 bash[22591]: cluster 2024-09-10T19:07:39.375721+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e793: 8 total, 7 up, 8 in 2024-09-10T19:07:40.198 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:40.226 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:40.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:40.365 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:40.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:40.503 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:40.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:40.638 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:40.728 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:40.742 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.769 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:40.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.905 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:40.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:41.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:41.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:40 smithi165 bash[22591]: audit 2024-09-10T19:07:38.403746+0000 mgr.y (mgr.24526) 720 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:41.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:40 smithi165 bash[22591]: cluster 2024-09-10T19:07:40.356368+0000 mgr.y (mgr.24526) 721 : cluster [DBG] pgmap v1229: 182 pgs: 64 peering, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 20 KiB/s wr, 3 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-10T19:07:41.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:40 smithi165 bash[22591]: cluster 2024-09-10T19:07:40.379899+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e794: 8 total, 7 up, 8 in 2024-09-10T19:07:41.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:40 smithi165 bash[22591]: audit 2024-09-10T19:07:40.381917+0000 mon.c (mon.2) 821 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.023 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:40 smithi165 bash[22591]: audit 2024-09-10T19:07:40.383478+0000 mon.a (mon.0) 3587 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[18629]: audit 2024-09-10T19:07:38.403746+0000 mgr.y (mgr.24526) 720 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[18629]: cluster 2024-09-10T19:07:40.356368+0000 mgr.y (mgr.24526) 721 : cluster [DBG] pgmap v1229: 182 pgs: 64 peering, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 20 KiB/s wr, 3 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[18629]: cluster 2024-09-10T19:07:40.379899+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e794: 8 total, 7 up, 8 in 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[18629]: audit 2024-09-10T19:07:40.381917+0000 mon.c (mon.2) 821 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[18629]: audit 2024-09-10T19:07:40.383478+0000 mon.a (mon.0) 3587 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[32381]: audit 2024-09-10T19:07:38.403746+0000 mgr.y (mgr.24526) 720 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[32381]: cluster 2024-09-10T19:07:40.356368+0000 mgr.y (mgr.24526) 721 : cluster [DBG] pgmap v1229: 182 pgs: 64 peering, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 20 KiB/s wr, 3 op/s; 11 KiB/s, 0 objects/s recovering 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[32381]: cluster 2024-09-10T19:07:40.379899+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e794: 8 total, 7 up, 8 in 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[32381]: audit 2024-09-10T19:07:40.381917+0000 mon.c (mon.2) 821 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:40 smithi083 bash[32381]: audit 2024-09-10T19:07:40.383478+0000 mon.a (mon.0) 3587 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:41.044 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:41.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.181 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:41.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.321 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:41.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:41.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.568 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:41.591 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:41.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:41.722 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:41.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.860 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:41.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:41.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:42.000 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:42.103 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.136 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:42.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:42.264 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:42.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- 172.21.15.83:0/445924139 >> v1:172.21.15.83:6790/0 conn(0x7ff818108d30 legacy=0x7ff81810b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:42.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- 172.21.15.83:0/445924139 shutdown_connections 2024-09-10T19:07:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- 172.21.15.83:0/445924139 >> 172.21.15.83:0/445924139 conn(0x7ff8180f7fe0 msgr2=0x7ff8180fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- 172.21.15.83:0/445924139 shutdown_connections 2024-09-10T19:07:42.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- 172.21.15.83:0/445924139 wait complete. 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 librados: starting msgr at 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 librados: starting objecter 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 Processor -- start 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- start start 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 librados: setting wanted keys 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 librados: calling monclient init 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8180ff030 con 0x7ff818108d30 2024-09-10T19:07:42.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff81811d1c0 con 0x7ff8181051d0 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff817fff640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff81811e360 con 0x7ff8181023f0 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff8177fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff818108d30 0x7ff81811aa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46540/0 (socket says 172.21.15.83:46540) 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff8177fe640 1 -- 172.21.15.83:0/359885117 learned_addr learned my addr 172.21.15.83:0/359885117 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter ms_handle_connect 0x7ff818108d30 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter ms_handle_connect 0x7ff8181023f0 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter ms_handle_connect 0x7ff8181051d0 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:42.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752800773 0 0) 0x7ff8180ff030 con 0x7ff818108d30 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff800003120 con 0x7ff818108d30 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835862319 0 0) 0x7ff81811e360 con 0x7ff8181023f0 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8180ff030 con 0x7ff8181023f0 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365577982 0 0) 0x7ff81811d1c0 con 0x7ff8181051d0 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff81811e360 con 0x7ff8181051d0 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064559916 0 0) 0x7ff800003120 con 0x7ff818108d30 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff81811d1c0 con 0x7ff818108d30 2024-09-10T19:07:42.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236330264 0 0) 0x7ff8180ff030 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff800003120 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.348+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff810003030 con 0x7ff818108d30 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff80c0019b0 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600590556 0 0) 0x7ff81811e360 con 0x7ff8181051d0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8180ff030 con 0x7ff8181051d0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff808001800 con 0x7ff8181051d0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 518795823 0 0) 0x7ff800003120 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 >> v1:172.21.15.165:6789/0 conn(0x7ff8181051d0 legacy=0x7ff818117360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 >> v1:172.21.15.83:6789/0 conn(0x7ff818108d30 legacy=0x7ff81811aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81811f500 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81811e550 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff80c002360 con 0x7ff8181023f0 2024-09-10T19:07:42.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff817fff640 10 client.49835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81811e9a0 con 0x7ff8181023f0 2024-09-10T19:07:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff80c0030f0 con 0x7ff8181023f0 2024-09-10T19:07:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff817fff640 1 librados: init done 2024-09-10T19:07:42.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff8180fdb60 con 0x7ff8181023f0 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.352+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff80c001f00 con 0x7ff8181023f0 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(795..795 src has 704..795) v4 ==== 8536+0+0 (unknown 473842944 0 0) 0x7ff80c087f70 con 0x7ff8181023f0 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 osd_map(795..795 src has 704..795) v4 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 3 client.49835.objecter handle_osd_map got epochs [795,795] > 0 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 3 client.49835.objecter handle_osd_map decoding full epoch 795 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 20 client.49835.objecter dump_active .. 0 homeless 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v795) v1 ==== 54+0+16 (unknown 3246866354 0 2420000192) 0x7ff80c088370 con 0x7ff8181023f0 2024-09-10T19:07:42.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff8000719b0 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 10 client.49835.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 1 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 1 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 0 1 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 4 2024-09-10T19:07:42.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 10 client.49835.objecter _send_command 1 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff81804bf60 con 0x7ff81811d350 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff81811d350 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff81804bf60 con 0x7ff81811d350 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 1 = system:0 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 4 2024-09-10T19:07:42.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.356+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 0 1 2024-09-10T19:07:42.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 10 client.49835.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 2 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 2 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8180ff740 osd=0 3 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 0 2 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 4 2024-09-10T19:07:42.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 10 client.49835.objecter _send_command 2 2024-09-10T19:07:42.387 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.380+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff81810e830 con 0x7ff81811d350 2024-09-10T19:07:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff81810e830 con 0x7ff81811d350 2024-09-10T19:07:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8180ff740 osd=0 4 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 0 2 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 10 client.49835.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 3 2024-09-10T19:07:42.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 20 client.49835.objecter _get_session osd=-1 returning homeless 2024-09-10T19:07:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 10 client.49835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=796}) v3 -- 0x7ff8181216a0 con 0x7ff8181023f0 2024-09-10T19:07:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.384+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff818121830 con 0x7ff8181023f0 2024-09-10T19:07:42.391 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 10 ==== osd_map(796..796 src has 704..796) v4 ==== 988+0+0 (unknown 828887310 0 0) 0x7ff80c054060 con 0x7ff8181023f0 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 osd_map(796..796 src has 704..796) v4 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 3 client.49835.objecter handle_osd_map got epochs [796,796] > 795 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 3 client.49835.objecter handle_osd_map decoding incremental epoch 796 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter checking command 3 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter _check_command_map_dne tid 3 current 796 map_dne_bound 0 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 20 client.49835.objecter dump_active .. 1 homeless 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== mon.2 v1:172.21.15.83:6790/0 11 ==== mon_get_version_reply(handle=1 version=796) v2 ==== 24+0+0 (unknown 3038538296 0 0) 0x7ff80c054320 con 0x7ff8181023f0 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff8157fa640 10 client.49835.objecter _check_command_map_dne tid 3 current 796 map_dne_bound 796 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff8157fa640 10 client.49835.objecter _finish_command 3 = osd:6 osd down 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff8157fa640 15 client.49835.objecter _session_command_op_remove -1 3 2024-09-10T19:07:42.392 INFO:teuthology.orchestra.run.smithi083.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 10 client.49835.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 4 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 4 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 2 4 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 4 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 10 client.49835.objecter _send_command 4 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6809/386094220 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff81811f000 con 0x7ff81805e820 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff81805e820 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff81811f000 con 0x7ff81805e820 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 4: 0 ) v1 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 4 = system:0 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 4 2024-09-10T19:07:42.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.388+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 2 4 2024-09-10T19:07:42.395 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 10 client.49835.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 5 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 5 2024-09-10T19:07:42.408 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81811ee90 osd=2 3 2024-09-10T19:07:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 2 5 2024-09-10T19:07:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 4 2024-09-10T19:07:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 10 client.49835.objecter _send_command 5 2024-09-10T19:07:42.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.404+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff818029fd0 con 0x7ff81805e820 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff818029fd0 con 0x7ff81805e820 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81811ee90 osd=2 4 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 2 5 2024-09-10T19:07:42.410 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:07:42.411 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:07:42.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 10 client.49835.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 6 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 6 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 3 6 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 4 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 10 client.49835.objecter _send_command 6 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6813/1279688021 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff81802ae10 con 0x7ff81805df00 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff81805df00 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff80800a730 con 0x7ff81805df00 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 6: 0 ) v1 2024-09-10T19:07:42.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 6 = system:0 2024-09-10T19:07:42.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 4 2024-09-10T19:07:42.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.408+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 3 6 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 10 client.49835.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 7 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 7 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81805dd90 osd=3 3 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 3 7 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 4 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 10 client.49835.objecter _send_command 7 2024-09-10T19:07:42.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff818071950 con 0x7ff81805df00 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff818071950 con 0x7ff81805df00 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81805dd90 osd=3 4 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 3 7 2024-09-10T19:07:42.429 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 10 client.49835.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 8 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 8 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 4 8 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 4 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 10 client.49835.objecter _send_command 8 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.424+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6800/3225515265 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff81802a730 con 0x7ff818109e20 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff818109e20 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff81802a730 con 0x7ff818109e20 2024-09-10T19:07:42.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 8: 0 ) v1 2024-09-10T19:07:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 8 = system:0 2024-09-10T19:07:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 4 2024-09-10T19:07:42.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.428+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 4 8 2024-09-10T19:07:42.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 10 client.49835.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 9 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 9 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff81802a5c0 osd=4 3 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 4 9 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 4 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 10 client.49835.objecter _send_command 9 2024-09-10T19:07:42.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.440+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff818109400 con 0x7ff818109e20 2024-09-10T19:07:42.448 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:07:42.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff818109400 con 0x7ff818109e20 2024-09-10T19:07:42.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff81802a5c0 osd=4 4 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 4 9 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 10 client.49835.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 10 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 10 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 5 10 2024-09-10T19:07:42.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 4 2024-09-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 10 client.49835.objecter _send_command 10 2024-09-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6804/571173777 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8181093f0 con 0x7ff81812b1c0 2024-09-10T19:07:42.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.444+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff81812b1c0 2024-09-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.448+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff8181093f0 con 0x7ff81812b1c0 2024-09-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.448+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 10: 0 ) v1 2024-09-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.448+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 10 = system:0 2024-09-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.448+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 4 2024-09-10T19:07:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.448+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 5 10 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 10 client.49835.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 11 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 11 2024-09-10T19:07:42.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8181091c0 osd=5 3 2024-09-10T19:07:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 5 11 2024-09-10T19:07:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 4 2024-09-10T19:07:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 10 client.49835.objecter _send_command 11 2024-09-10T19:07:42.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.460+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff8181093f0 con 0x7ff81812b1c0 2024-09-10T19:07:42.470 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff8181093f0 con 0x7ff81812b1c0 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8181091c0 osd=5 4 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 5 11 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:07:42.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 10 client.49835.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 12 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 12 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 6 12 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 4 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 10 client.49835.objecter _send_command 12 2024-09-10T19:07:42.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6808/1317922936 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8181311c0 con 0x7ff818130bd0 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff818130bd0 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff8181311c0 con 0x7ff818130bd0 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 12: 0 ) v1 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 12 = system:0 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 4 2024-09-10T19:07:42.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.468+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 6 12 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 10 client.49835.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 13 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 13 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff8181093f0 osd=6 3 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 6 13 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 4 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 10 client.49835.objecter _send_command 13 2024-09-10T19:07:42.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.484+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff818131180 con 0x7ff818130bd0 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff818131180 con 0x7ff818130bd0 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff8181093f0 osd=6 4 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 6 13 2024-09-10T19:07:42.493 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 10 client.49835.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 14 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 14 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 7 14 2024-09-10T19:07:42.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 4 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 10 client.49835.objecter _send_command 14 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6812/1610151650 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff81813c020 con 0x7ff8181382a0 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.488+0000 7ff7ff7fe640 10 client.49835.objecter ms_handle_connect 0x7ff8181382a0 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.492+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff81813c020 con 0x7ff8181382a0 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.492+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 14: 0 ) v1 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.492+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 14 = system:0 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.492+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 4 2024-09-10T19:07:42.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.492+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 7 14 2024-09-10T19:07:42.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 10 client.49835.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign -1 15 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 20 client.49835.objecter _get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 15 client.49835.objecter _session_command_op_remove -1 15 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 20 client.49835.objecter get_session s=0x7ff818131180 osd=7 3 2024-09-10T19:07:42.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 15 client.49835.objecter _session_command_op_assign 7 15 2024-09-10T19:07:42.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 4 2024-09-10T19:07:42.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 10 client.49835.objecter _send_command 15 2024-09-10T19:07:42.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.504+0000 7ff817fff640 1 -- 172.21.15.83:0/359885117 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff81813bfe0 con 0x7ff8181382a0 2024-09-10T19:07:42.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.508+0000 7ff7ff7fe640 1 -- 172.21.15.83:0/359885117 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7ff81813bfe0 con 0x7ff8181382a0 2024-09-10T19:07:42.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.508+0000 7ff7ff7fe640 10 client.49835.objecter ms_dispatch 0x7ff818000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:07:42.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.508+0000 7ff7ff7fe640 10 client.49835.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:07:42.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.508+0000 7ff7ff7fe640 20 client.49835.objecter put_session s=0x7ff818131180 osd=7 4 2024-09-10T19:07:42.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:42.508+0000 7ff7ff7fe640 15 client.49835.objecter _session_command_op_remove 7 15 2024-09-10T19:07:42.514 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:07:42.514 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:07:42.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18629]: audit 2024-09-10T19:07:41.381692+0000 mon.a (mon.0) 3588 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:42.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18629]: cluster 2024-09-10T19:07:41.381866+0000 mon.a (mon.0) 3589 : cluster [DBG] osdmap e795: 8 total, 7 up, 8 in 2024-09-10T19:07:42.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18629]: audit 2024-09-10T19:07:41.434638+0000 mon.c (mon.2) 822 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18629]: audit 2024-09-10T19:07:41.435251+0000 mon.a (mon.0) 3590 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18629]: audit 2024-09-10T19:07:42.357914+0000 mon.c (mon.2) 823 : audit [DBG] from='client.? v1:172.21.15.83:0/359885117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:42.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[32381]: audit 2024-09-10T19:07:41.381692+0000 mon.a (mon.0) 3588 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:42.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[32381]: cluster 2024-09-10T19:07:41.381866+0000 mon.a (mon.0) 3589 : cluster [DBG] osdmap e795: 8 total, 7 up, 8 in 2024-09-10T19:07:42.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[32381]: audit 2024-09-10T19:07:41.434638+0000 mon.c (mon.2) 822 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[32381]: audit 2024-09-10T19:07:41.435251+0000 mon.a (mon.0) 3590 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[32381]: audit 2024-09-10T19:07:42.357914+0000 mon.c (mon.2) 823 : audit [DBG] from='client.? v1:172.21.15.83:0/359885117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:42.534 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:42.637 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.671 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:42.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:42.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:42 smithi165 bash[22591]: audit 2024-09-10T19:07:41.381692+0000 mon.a (mon.0) 3588 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:42.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:42 smithi165 bash[22591]: cluster 2024-09-10T19:07:41.381866+0000 mon.a (mon.0) 3589 : cluster [DBG] osdmap e795: 8 total, 7 up, 8 in 2024-09-10T19:07:42.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:42 smithi165 bash[22591]: audit 2024-09-10T19:07:41.434638+0000 mon.c (mon.2) 822 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:42 smithi165 bash[22591]: audit 2024-09-10T19:07:41.435251+0000 mon.a (mon.0) 3590 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:42.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:42 smithi165 bash[22591]: audit 2024-09-10T19:07:42.357914+0000 mon.c (mon.2) 823 : audit [DBG] from='client.? v1:172.21.15.83:0/359885117' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:07:42.810 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:42.914 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.927 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:42] "GET /metrics HTTP/1.1" 200 229033 "" "Prometheus/2.43.0" 2024-09-10T19:07:42.946 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:42.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:43.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:43.074 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:43.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:43.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:43.351 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:43.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:43.486 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:43.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:43.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: cluster 2024-09-10T19:07:42.357384+0000 mgr.y (mgr.24526) 722 : cluster [DBG] pgmap v1232: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:43.602 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: cluster 2024-09-10T19:07:42.374735+0000 mon.a (mon.0) 3591 : cluster [WRN] Health check failed: Degraded data redundancy: 80/642 objects degraded (12.461%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: cluster 2024-09-10T19:07:42.374768+0000 mon.a (mon.0) 3592 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-09-10T19:07:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: audit 2024-09-10T19:07:42.380876+0000 mon.a (mon.0) 3593 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: cluster 2024-09-10T19:07:42.380976+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e796: 8 total, 7 up, 8 in 2024-09-10T19:07:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: audit 2024-09-10T19:07:42.386507+0000 mon.c (mon.2) 824 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.603 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:43 smithi165 bash[22591]: audit 2024-09-10T19:07:42.391776+0000 mon.a (mon.0) 3595 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.624 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:43.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: cluster 2024-09-10T19:07:42.357384+0000 mgr.y (mgr.24526) 722 : cluster [DBG] pgmap v1232: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:43.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: cluster 2024-09-10T19:07:42.374735+0000 mon.a (mon.0) 3591 : cluster [WRN] Health check failed: Degraded data redundancy: 80/642 objects degraded (12.461%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:43.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: cluster 2024-09-10T19:07:42.374768+0000 mon.a (mon.0) 3592 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-09-10T19:07:43.705 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: audit 2024-09-10T19:07:42.380876+0000 mon.a (mon.0) 3593 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: cluster 2024-09-10T19:07:42.380976+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e796: 8 total, 7 up, 8 in 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: audit 2024-09-10T19:07:42.386507+0000 mon.c (mon.2) 824 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[18629]: audit 2024-09-10T19:07:42.391776+0000 mon.a (mon.0) 3595 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: cluster 2024-09-10T19:07:42.357384+0000 mgr.y (mgr.24526) 722 : cluster [DBG] pgmap v1232: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: cluster 2024-09-10T19:07:42.374735+0000 mon.a (mon.0) 3591 : cluster [WRN] Health check failed: Degraded data redundancy: 80/642 objects degraded (12.461%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: cluster 2024-09-10T19:07:42.374768+0000 mon.a (mon.0) 3592 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: audit 2024-09-10T19:07:42.380876+0000 mon.a (mon.0) 3593 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: cluster 2024-09-10T19:07:42.380976+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e796: 8 total, 7 up, 8 in 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: audit 2024-09-10T19:07:42.386507+0000 mon.c (mon.2) 824 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.706 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:43 smithi083 bash[32381]: audit 2024-09-10T19:07:42.391776+0000 mon.a (mon.0) 3595 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:43.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.759 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:43.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.812+0000 7f672f767640 1 -- 172.21.15.83:0/3523824201 >> v1:172.21.15.83:6789/0 conn(0x7f67281014f0 legacy=0x7f67281018c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:43.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.812+0000 7f672f767640 1 -- 172.21.15.83:0/3523824201 shutdown_connections 2024-09-10T19:07:43.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.812+0000 7f672f767640 1 -- 172.21.15.83:0/3523824201 >> 172.21.15.83:0/3523824201 conn(0x7f67280f7070 msgr2=0x7f67280f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:43.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.812+0000 7f672f767640 1 -- 172.21.15.83:0/3523824201 shutdown_connections 2024-09-10T19:07:43.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- 172.21.15.83:0/3523824201 wait complete. 2024-09-10T19:07:43.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 librados: starting msgr at 2024-09-10T19:07:43.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 librados: starting objecter 2024-09-10T19:07:43.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 Processor -- start 2024-09-10T19:07:43.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- start start 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 librados: setting wanted keys 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 librados: calling monclient init 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6728107bd0 con 0x7f6728104260 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f672811d260 con 0x7f6728107dc0 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f672811e460 con 0x7f67281014f0 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672cd02640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6728104260 0x7f6728117400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52834/0 (socket says 172.21.15.83:52834) 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672cd02640 1 -- 172.21.15.83:0/203179453 learned_addr learned my addr 172.21.15.83:0/203179453 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter ms_handle_connect 0x7f67281014f0 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter ms_handle_connect 0x7f6728104260 2024-09-10T19:07:43.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347733839 0 0) 0x7f672811e460 con 0x7f67281014f0 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714003120 con 0x7f67281014f0 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter ms_handle_connect 0x7f6728107dc0 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725743645 0 0) 0x7f6728107bd0 con 0x7f6728104260 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f672811e460 con 0x7f6728104260 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616632412 0 0) 0x7f672811d260 con 0x7f6728107dc0 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6728107bd0 con 0x7f6728107dc0 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1632629022 0 0) 0x7f672811e460 con 0x7f6728104260 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f672811d260 con 0x7f6728104260 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6724001d70 con 0x7f6728104260 2024-09-10T19:07:43.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038415884 0 0) 0x7f6714003120 con 0x7f67281014f0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f672811e460 con 0x7f67281014f0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6720002050 con 0x7f67281014f0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 918226490 0 0) 0x7f6728107bd0 con 0x7f6728107dc0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6714003120 con 0x7f6728107dc0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f671c001840 con 0x7f6728107dc0 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3142535107 0 0) 0x7f672811d260 con 0x7f6728104260 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 >> v1:172.21.15.83:6790/0 conn(0x7f67281014f0 legacy=0x7f6728107000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 >> v1:172.21.15.165:6789/0 conn(0x7f6728107dc0 legacy=0x7f672811ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672811f660 con 0x7f6728104260 2024-09-10T19:07:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672811c2b0 con 0x7f6728104260 2024-09-10T19:07:43.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 10 client.25755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:07:43.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f672811c700 con 0x7f6728104260 2024-09-10T19:07:43.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f672f767640 1 librados: init done 2024-09-10T19:07:43.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f67240023f0 con 0x7f6728104260 2024-09-10T19:07:43.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.816+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6724005d70 con 0x7f6728104260 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67280fcbf0 con 0x7f6728104260 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f6724001f90 con 0x7f6728104260 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(797..797 src has 704..797) v4 ==== 8531+0+0 (unknown 477512956 0 0) 0x7f672408ab50 con 0x7f6728104260 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 10 client.25755.objecter ms_dispatch 0x7f6728000b90 osd_map(797..797 src has 704..797) v4 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 3 client.25755.objecter handle_osd_map got epochs [797,797] > 0 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 3 client.25755.objecter handle_osd_map decoding full epoch 797 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 20 client.25755.objecter dump_active .. 0 homeless 2024-09-10T19:07:43.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.820+0000 7f6719ffb640 10 client.25755.objecter ms_handle_connect 0x7f6714071810 2024-09-10T19:07:43.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:43.828+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f6724056f00 con 0x7f6728104260 2024-09-10T19:07:43.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.893 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:43.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:43.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.012+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f672810d960 con 0x7f6728104260 2024-09-10T19:07:44.028 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:44.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:44.166 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:44.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:44.305 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:44.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:44.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.416+0000 7f6719ffb640 1 -- 172.21.15.83:0/203179453 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v798) v1 ==== 91+0+0 (unknown 2012268461 0 0) 0x7f67240568a0 con 0x7f6728104260 2024-09-10T19:07:44.421 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:07:44.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 10 librados: watch_flush enter 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 10 librados: watch_flush exit 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 20 client.25755.objecter shutdown clearing up homeless session... 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 10 client.25755.objecter successfully canceled tick 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 >> v1:172.21.15.83:6800/3071793735 conn(0x7f6714071810 legacy=0x7f6714073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 >> v1:172.21.15.83:6789/0 conn(0x7f6728104260 legacy=0x7f6728117400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 shutdown_connections 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 >> 172.21.15.83:0/203179453 conn(0x7f67280f7070 msgr2=0x7f67280fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 shutdown_connections 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 -- 172.21.15.83:0/203179453 wait complete. 2024-09-10T19:07:44.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:07:44.420+0000 7f672f767640 1 librados: shutdown 2024-09-10T19:07:44.437 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:44.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:44.575 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:44.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: audit 2024-09-10T19:07:43.397048+0000 mon.a (mon.0) 3596 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: cluster 2024-09-10T19:07:43.397204+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e797: 8 total, 7 up, 8 in 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: audit 2024-09-10T19:07:43.403142+0000 mon.c (mon.2) 825 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: audit 2024-09-10T19:07:43.409814+0000 mon.a (mon.0) 3598 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: audit 2024-09-10T19:07:44.017495+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:07:44.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:44 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.358448+0000 mgr.y (mgr.24526) 723 : cluster [DBG] pgmap v1235: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:44.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: audit 2024-09-10T19:07:43.397048+0000 mon.a (mon.0) 3596 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: cluster 2024-09-10T19:07:43.397204+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e797: 8 total, 7 up, 8 in 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: audit 2024-09-10T19:07:43.403142+0000 mon.c (mon.2) 825 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: audit 2024-09-10T19:07:43.409814+0000 mon.a (mon.0) 3598 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: audit 2024-09-10T19:07:44.017495+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.358448+0000 mgr.y (mgr.24526) 723 : cluster [DBG] pgmap v1235: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: audit 2024-09-10T19:07:43.397048+0000 mon.a (mon.0) 3596 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: cluster 2024-09-10T19:07:43.397204+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e797: 8 total, 7 up, 8 in 2024-09-10T19:07:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: audit 2024-09-10T19:07:43.403142+0000 mon.c (mon.2) 825 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: audit 2024-09-10T19:07:43.409814+0000 mon.a (mon.0) 3598 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]: dispatch 2024-09-10T19:07:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: audit 2024-09-10T19:07:44.017495+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:07:44.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:44 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.358448+0000 mgr.y (mgr.24526) 723 : cluster [DBG] pgmap v1235: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80/642 objects degraded (12.461%) 2024-09-10T19:07:44.812 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:44.840 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:07:44.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:07:44.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:44.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:45.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:45.121 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:45.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:45.259 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:45.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:45.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:45.399 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:45.502 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.405191+0000 mon.a (mon.0) 3600 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.405287+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.419917+0000 mon.a (mon.0) 3602 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]': finished 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.420035+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.420087+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e798: 8 total, 7 up, 8 in 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.421703+0000 mon.c (mon.2) 826 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.422632+0000 mon.a (mon.0) 3605 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.581743+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: audit 2024-09-10T19:07:44.583605+0000 mon.c (mon.2) 827 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.701599+0000 osd.6 (osd.6) 367 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[18629]: cluster 2024-09-10T19:07:44.704494+0000 osd.6 (osd.6) 368 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.405191+0000 mon.a (mon.0) 3600 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.405287+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.419917+0000 mon.a (mon.0) 3602 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]': finished 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.420035+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.420087+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e798: 8 total, 7 up, 8 in 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.421703+0000 mon.c (mon.2) 826 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.422632+0000 mon.a (mon.0) 3605 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.518 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.581743+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: audit 2024-09-10T19:07:44.583605+0000 mon.c (mon.2) 827 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.701599+0000 osd.6 (osd.6) 367 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:07:45.519 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:45 smithi083 bash[32381]: cluster 2024-09-10T19:07:44.704494+0000 osd.6 (osd.6) 368 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:07:45.538 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:45.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.405191+0000 mon.a (mon.0) 3600 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:45.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.405287+0000 mon.a (mon.0) 3601 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.419917+0000 mon.a (mon.0) 3602 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-142", "mode": "writeback"}]': finished 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.420035+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? v1:172.21.15.83:0/203179453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.420087+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e798: 8 total, 7 up, 8 in 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.421703+0000 mon.c (mon.2) 826 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.422632+0000 mon.a (mon.0) 3605 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.581743+0000 mon.a (mon.0) 3606 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: audit 2024-09-10T19:07:44.583605+0000 mon.c (mon.2) 827 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.701599+0000 osd.6 (osd.6) 367 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:07:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:45 smithi165 bash[22591]: cluster 2024-09-10T19:07:44.704494+0000 osd.6 (osd.6) 368 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:07:45.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:45.674 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:45.778 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:45.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:45.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:45.938 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:45.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:45.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:46.041 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:46.077 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:46.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:46.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:46.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:46.317 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:46.351 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:46.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:46.481 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:46.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: audit 2024-09-10T19:07:45.424541+0000 mon.a (mon.0) 3607 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:45.424657+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e799: 8 total, 7 up, 8 in 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: audit 2024-09-10T19:07:45.431163+0000 mon.c (mon.2) 828 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: audit 2024-09-10T19:07:45.431715+0000 mon.a (mon.0) 3609 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:45.732292+0000 osd.6 (osd.6) 369 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:45.739522+0000 osd.6 (osd.6) 370 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:45.944129+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:45.951230+0000 osd.7 (osd.7) 407 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:07:46.598 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:46 smithi165 bash[22591]: cluster 2024-09-10T19:07:46.360853+0000 mgr.y (mgr.24526) 724 : cluster [DBG] pgmap v1238: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: audit 2024-09-10T19:07:45.424541+0000 mon.a (mon.0) 3607 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:45.424657+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e799: 8 total, 7 up, 8 in 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: audit 2024-09-10T19:07:45.431163+0000 mon.c (mon.2) 828 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: audit 2024-09-10T19:07:45.431715+0000 mon.a (mon.0) 3609 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:45.732292+0000 osd.6 (osd.6) 369 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:07:46.605 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:45.739522+0000 osd.6 (osd.6) 370 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:45.944129+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:45.951230+0000 osd.7 (osd.7) 407 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[18629]: cluster 2024-09-10T19:07:46.360853+0000 mgr.y (mgr.24526) 724 : cluster [DBG] pgmap v1238: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: audit 2024-09-10T19:07:45.424541+0000 mon.a (mon.0) 3607 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:45.424657+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e799: 8 total, 7 up, 8 in 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: audit 2024-09-10T19:07:45.431163+0000 mon.c (mon.2) 828 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: audit 2024-09-10T19:07:45.431715+0000 mon.a (mon.0) 3609 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:45.732292+0000 osd.6 (osd.6) 369 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:07:46.606 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:45.739522+0000 osd.6 (osd.6) 370 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:07:46.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:45.944129+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:07:46.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:45.951230+0000 osd.7 (osd.7) 407 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:07:46.607 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:46 smithi083 bash[32381]: cluster 2024-09-10T19:07:46.360853+0000 mgr.y (mgr.24526) 724 : cluster [DBG] pgmap v1238: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:46.619 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:46.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:07:46.722 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:46.753 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:46.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:07:46.758 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:07:46.758 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-10T19:07:46.759 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi083.front.sepia.ceph.com 2024-09-10T19:07:46.759 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-10T19:07:46.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:46.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:07:46.890 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:46.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:46.927 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:07:46.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:47.031 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:47.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:47.167 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:47.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:47.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:47.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.441 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: cluster 2024-09-10T19:07:46.073930+0000 osd.3 (osd.3) 430 : cluster [DBG] 4.c scrub starts 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: cluster 2024-09-10T19:07:46.076999+0000 osd.3 (osd.3) 431 : cluster [DBG] 4.c scrub ok 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: audit 2024-09-10T19:07:46.440476+0000 mon.a (mon.0) 3610 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: cluster 2024-09-10T19:07:46.440575+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e800: 8 total, 7 up, 8 in 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: audit 2024-09-10T19:07:46.444590+0000 mon.c (mon.2) 829 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[18629]: audit 2024-09-10T19:07:46.445179+0000 mon.a (mon.0) 3612 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: cluster 2024-09-10T19:07:46.073930+0000 osd.3 (osd.3) 430 : cluster [DBG] 4.c scrub starts 2024-09-10T19:07:47.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: cluster 2024-09-10T19:07:46.076999+0000 osd.3 (osd.3) 431 : cluster [DBG] 4.c scrub ok 2024-09-10T19:07:47.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: audit 2024-09-10T19:07:46.440476+0000 mon.a (mon.0) 3610 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:47.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: cluster 2024-09-10T19:07:46.440575+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e800: 8 total, 7 up, 8 in 2024-09-10T19:07:47.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: audit 2024-09-10T19:07:46.444590+0000 mon.c (mon.2) 829 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:47 smithi083 bash[32381]: audit 2024-09-10T19:07:46.445179+0000 mon.a (mon.0) 3612 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:47.579 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:47.682 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:47.717 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:47.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: cluster 2024-09-10T19:07:46.073930+0000 osd.3 (osd.3) 430 : cluster [DBG] 4.c scrub starts 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: cluster 2024-09-10T19:07:46.076999+0000 osd.3 (osd.3) 431 : cluster [DBG] 4.c scrub ok 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: audit 2024-09-10T19:07:46.440476+0000 mon.a (mon.0) 3610 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: cluster 2024-09-10T19:07:46.440575+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e800: 8 total, 7 up, 8 in 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: audit 2024-09-10T19:07:46.444590+0000 mon.c (mon.2) 829 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.832 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:47 smithi165 bash[22591]: audit 2024-09-10T19:07:46.445179+0000 mon.a (mon.0) 3612 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:47.852 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:47.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:47.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:48.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:48.124 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:48.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.264 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:48.368 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:48.395 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:48.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:48.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: cluster 2024-09-10T19:07:47.430868+0000 mon.a (mon.0) 3613 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:48.511 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: audit 2024-09-10T19:07:47.442631+0000 mon.a (mon.0) 3614 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: cluster 2024-09-10T19:07:47.442724+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e801: 8 total, 7 up, 8 in 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: audit 2024-09-10T19:07:47.450697+0000 mon.c (mon.2) 830 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: audit 2024-09-10T19:07:47.451079+0000 mon.a (mon.0) 3616 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: cluster 2024-09-10T19:07:47.920930+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: cluster 2024-09-10T19:07:47.924730+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:07:48.512 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:48 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.361982+0000 mgr.y (mgr.24526) 725 : cluster [DBG] pgmap v1241: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:48.525 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:48.572 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:07:48.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:48.639 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: cluster 2024-09-10T19:07:47.430868+0000 mon.a (mon.0) 3613 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: audit 2024-09-10T19:07:47.442631+0000 mon.a (mon.0) 3614 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: cluster 2024-09-10T19:07:47.442724+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e801: 8 total, 7 up, 8 in 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: audit 2024-09-10T19:07:47.450697+0000 mon.c (mon.2) 830 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: audit 2024-09-10T19:07:47.451079+0000 mon.a (mon.0) 3616 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: cluster 2024-09-10T19:07:47.920930+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: cluster 2024-09-10T19:07:47.924730+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.361982+0000 mgr.y (mgr.24526) 725 : cluster [DBG] pgmap v1241: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: cluster 2024-09-10T19:07:47.430868+0000 mon.a (mon.0) 3613 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:48.640 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: audit 2024-09-10T19:07:47.442631+0000 mon.a (mon.0) 3614 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: cluster 2024-09-10T19:07:47.442724+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e801: 8 total, 7 up, 8 in 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: audit 2024-09-10T19:07:47.450697+0000 mon.c (mon.2) 830 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: audit 2024-09-10T19:07:47.451079+0000 mon.a (mon.0) 3616 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: cluster 2024-09-10T19:07:47.920930+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: cluster 2024-09-10T19:07:47.924730+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:07:48.641 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:48 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.361982+0000 mgr.y (mgr.24526) 725 : cluster [DBG] pgmap v1241: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:48.653 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:48.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.789 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:48.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.925 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:48.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:49.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:49.063 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:49.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:49.204 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:49.308 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:49.342 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:49.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:49.477 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:49.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.364056+0000 osd.0 (osd.0) 387 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.367786+0000 osd.0 (osd.0) 388 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:48.414968+0000 mgr.y (mgr.24526) 726 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:48.460571+0000 mon.a (mon.0) 3617 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.460681+0000 mon.a (mon.0) 3618 : cluster [DBG] osdmap e802: 8 total, 7 up, 8 in 2024-09-10T19:07:49.594 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:48.464939+0000 mon.c (mon.2) 831 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:48.465387+0000 mon.a (mon.0) 3619 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.522529+0000 osd.4 (osd.4) 398 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.525996+0000 osd.4 (osd.4) 399 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.620367+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:48.623854+0000 osd.5 (osd.5) 421 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.368308+0000 mon.a (mon.0) 3620 : cluster [WRN] Health check update: Degraded data redundancy: 84/702 objects degraded (11.966%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:49.379515+0000 mon.a (mon.0) 3621 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.379705+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e803: 8 total, 7 up, 8 in 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:49.381799+0000 mon.c (mon.2) 832 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.595 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:49 smithi165 bash[22591]: audit 2024-09-10T19:07:49.387741+0000 mon.a (mon.0) 3623 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.613 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:49.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:49.717 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.364056+0000 osd.0 (osd.0) 387 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.367786+0000 osd.0 (osd.0) 388 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:48.414968+0000 mgr.y (mgr.24526) 726 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:48.460571+0000 mon.a (mon.0) 3617 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.460681+0000 mon.a (mon.0) 3618 : cluster [DBG] osdmap e802: 8 total, 7 up, 8 in 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:48.464939+0000 mon.c (mon.2) 831 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:48.465387+0000 mon.a (mon.0) 3619 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.731 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.522529+0000 osd.4 (osd.4) 398 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.525996+0000 osd.4 (osd.4) 399 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.620367+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:48.623854+0000 osd.5 (osd.5) 421 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.368308+0000 mon.a (mon.0) 3620 : cluster [WRN] Health check update: Degraded data redundancy: 84/702 objects degraded (11.966%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:49.379515+0000 mon.a (mon.0) 3621 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.379705+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e803: 8 total, 7 up, 8 in 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:49.381799+0000 mon.c (mon.2) 832 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[32381]: audit 2024-09-10T19:07:49.387741+0000 mon.a (mon.0) 3623 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.364056+0000 osd.0 (osd.0) 387 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.367786+0000 osd.0 (osd.0) 388 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:07:49.732 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:48.414968+0000 mgr.y (mgr.24526) 726 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:48.460571+0000 mon.a (mon.0) 3617 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.460681+0000 mon.a (mon.0) 3618 : cluster [DBG] osdmap e802: 8 total, 7 up, 8 in 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:48.464939+0000 mon.c (mon.2) 831 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:48.465387+0000 mon.a (mon.0) 3619 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.522529+0000 osd.4 (osd.4) 398 : cluster [DBG] 241.3 scrub starts 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.525996+0000 osd.4 (osd.4) 399 : cluster [DBG] 241.3 scrub ok 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.620367+0000 osd.5 (osd.5) 420 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:48.623854+0000 osd.5 (osd.5) 421 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.368308+0000 mon.a (mon.0) 3620 : cluster [WRN] Health check update: Degraded data redundancy: 84/702 objects degraded (11.966%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:49.379515+0000 mon.a (mon.0) 3621 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.379705+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e803: 8 total, 7 up, 8 in 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:49.381799+0000 mon.c (mon.2) 832 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.733 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:49 smithi083 bash[18629]: audit 2024-09-10T19:07:49.387741+0000 mon.a (mon.0) 3623 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:07:49.750 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:49.853 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:49.887 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:49.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:49.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:50.024 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:50.128 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:50.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:50.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:50.292 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:50.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:50.427 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:50.530 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:50.555 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "size": 20, 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 600, 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [ 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24818.0:1138 16.7 16:e0000000::::head [pgnls start_epoch 345] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_overlay+known_if_redirected+supports_pool_eio e345)", 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:58:22.152310+0000", 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 568.42380027599995, 2024-09-10T19:07:50.579 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0098653500000001, 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24818", 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1232835119", 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1138 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152310+0000", 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152310+0000", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152305+0000", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152324+0000", 2024-09-10T19:07:50.581 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9012e-05 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152351+0000", 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.6846e-05 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.152381+0000", 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.9889000000000001e-05 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.582 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153140+0000", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00075948400000000005 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153151+0000", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0917000000000001e-05 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.161992+0000", 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0088409679999999 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.583 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.162038+0000", 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5762e-05 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.162175+0000", 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013753999999999999 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24818.0:1138 16.7 16:e0000000::::head [pgnls start_epoch 345] snapc 0=[] RETRY=2 ondisk+retry+read+ignore_overlay+known_if_redirected+supports_pool_eio e346)", 2024-09-10T19:07:50.584 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T18:58:22.153665+0000", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 568.42244496000001, 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0093921429999999, 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24818", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/1232835119", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1138 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153665+0000", 2024-09-10T19:07:50.585 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153665+0000", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153664+0000", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999986 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153671+0000", 2024-09-10T19:07:50.586 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.1069999999999999e-06 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153679+0000", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.4090000000000003e-06 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153684+0000", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.9540000000000003e-06 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153725+0000", 2024-09-10T19:07:50.587 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.0534999999999997e-05 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for peered", 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:22.153734+0000", 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.3829999999999998e-06 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.162799+0000", 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0090648419999999 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.588 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.162856+0000", 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.7056999999999999e-05 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T18:58:23.163058+0000", 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00020143200000000001 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.589 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:255 244.15 244:a9dad0c8:test-rados-api-smithi083-77828-117::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e674)", 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:17.317314+0000", 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 153.25879686799999, 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0089220069999999992, 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 255 2024-09-10T19:07:50.590 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317314+0000", 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317314+0000", 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.591 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317310+0000", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317325+0000", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.4816e-05 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317340+0000", 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5238999999999999e-05 2024-09-10T19:07:50.592 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317348+0000", 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.5259999999999993e-06 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317417+0000", 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.8799000000000003e-05 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.593 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.317422+0000", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.3850000000000004e-06 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.318045+0000", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00062297000000000003 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.318343+0000", 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00029799299999999998 2024-09-10T19:07:50.594 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.325061+0000", 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0067179739999999998 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.325186+0000", 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00012541200000000001 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.595 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.325969+0000", 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00078343200000000003 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.326025+0000", 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.5294000000000003e-05 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:17.326236+0000", 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021088899999999999 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.596 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:412 245.4 245:20000000::::head [pgnls start_epoch 688] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e688)", 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:05:44.395462+0000", 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 126.180648537, 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.003398711, 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 412 2024-09-10T19:07:50.597 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395462+0000", 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395462+0000", 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395458+0000", 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395476+0000", 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8037e-05 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395523+0000", 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6789999999999998e-05 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.599 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395539+0000", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6846000000000001e-05 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395717+0000", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00017705600000000001 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "op must wait for map", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.395732+0000", 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5758999999999999e-05 2024-09-10T19:07:50.600 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.398642+0000", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0029101629999999999 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.398695+0000", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.2454999999999998e-05 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:05:44.398861+0000", 2024-09-10T19:07:50.601 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00016567099999999999 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:1339 251.e 251:74591401:test-rados-api-smithi083-77828-131::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e738)", 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:06:37.414137+0000", 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 73.161973144000001, 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0049547339999999997, 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:07:50.602 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1339 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414137+0000", 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.603 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414137+0000", 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414135+0000", 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414143+0000", 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.1380000000000007e-06 2024-09-10T19:07:50.604 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414159+0000", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6543000000000001e-05 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414166+0000", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 6.3080000000000004e-06 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414290+0000", 2024-09-10T19:07:50.605 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000124358 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.414481+0000", 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019103600000000001 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.418894+0000", 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0044131750000000001 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.606 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.419007+0000", 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00011243699999999999 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.419092+0000", 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.5350000000000001e-05 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:1382 251.e 251:74591401:test-rados-api-smithi083-77828-131::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e738)", 2024-09-10T19:07:50.607 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:06:37.448190+0000", 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 73.127920400999997, 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0054562889999999996, 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1382 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.608 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448190+0000", 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448190+0000", 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448187+0000", 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.609 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448198+0000", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.0678e-05 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448213+0000", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5613000000000001e-05 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448221+0000", 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.5780000000000001e-06 2024-09-10T19:07:50.610 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448301+0000", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.9983999999999999e-05 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for blocked object", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.448511+0000", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000209965 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.450728+0000", 2024-09-10T19:07:50.611 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002217138 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "waiting for rw locks", 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.450872+0000", 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000144196 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.451823+0000", 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00095052000000000005 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.612 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.451915+0000", 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.2619000000000002e-05 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:06:37.453646+0000", 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001730828 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.613 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.49694.0:1932 255.0 255:00000000::::head [pgnls start_epoch 777] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e777)", 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:17.419113+0000", 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 33.156997253999997, 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.038319910999999998, 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.49694", 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1932 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.614 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.419113+0000", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.419113+0000", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.419109+0000", 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999957 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.615 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.419127+0000", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.8386e-05 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.454720+0000", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.035592398999999997 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.454779+0000", 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.9546000000000002e-05 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.616 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.457108+0000", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.002329341 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.457184+0000", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.5975000000000004e-05 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:17.457433+0000", 2024-09-10T19:07:50.617 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024868299999999998 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1073 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e796)", 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:43.406363+0000", 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 7.1697475629999996, 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00060460799999999997, 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.618 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1073 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406363+0000", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406363+0000", 2024-09-10T19:07:50.619 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406357+0000", 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999938 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406411+0000", 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.4032e-05 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.620 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406456+0000", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5033000000000001e-05 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406475+0000", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9508999999999999e-05 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406662+0000", 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00018688099999999999 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.621 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406793+0000", 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013035499999999999 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:43.406967+0000", 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000174885 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.622 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1074 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e797)", 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:44.408571+0000", 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 6.1675398619999999, 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00068227999999999995, 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1074 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.623 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408571+0000", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408571+0000", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408567+0000", 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.624 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408584+0000", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6918999999999998e-05 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408615+0000", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.0942000000000003e-05 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408631+0000", 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6209e-05 2024-09-10T19:07:50.625 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408750+0000", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000119259 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.408941+0000", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019078000000000001 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:44.409253+0000", 2024-09-10T19:07:50.626 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000311935 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1075 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e797)", 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:45.410633+0000", 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 5.165477696, 2024-09-10T19:07:50.627 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00056931600000000005, 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1075 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410633+0000", 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410633+0000", 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410626+0000", 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999933 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.629 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410652+0000", 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5621000000000001e-05 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410692+0000", 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.0516000000000003e-05 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410708+0000", 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6090999999999999e-05 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.630 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.410845+0000", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000136329 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.411003+0000", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00015860900000000001 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.411202+0000", 2024-09-10T19:07:50.631 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00019865099999999999 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "MOSDScrubReserve(1.6 REQUEST e798)", 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:45.734671+0000", 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 4.841439856, 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00039279799999999998, 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.632 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734671+0000", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734671+0000", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734667+0000", 2024-09-10T19:07:50.633 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999967 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734681+0000", 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.3423e-05 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734714+0000", 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3062999999999998e-05 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.634 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734726+0000", 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2276000000000001e-05 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.734982+0000", 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00025604799999999999 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.735015+0000", 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3229000000000003e-05 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.635 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.735063+0000", 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.8189000000000002e-05 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "replica_scrub(pg: 1.6,from:0'0,to:0'0,epoch:798/510,start:1:60000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:45.736480+0000", 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 4.8396308970000002, 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00039910100000000001, 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.636 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736480+0000", 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736480+0000", 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.637 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736471+0000", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999919 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736491+0000", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9604000000000001e-05 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736525+0000", 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.3528999999999997e-05 2024-09-10T19:07:50.638 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736544+0000", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.9423e-05 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736741+0000", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000196992 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736787+0000", 2024-09-10T19:07:50.639 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.5924999999999998e-05 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:45.736879+0000", 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.1638000000000006e-05 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1076 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e798)", 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:46.412980+0000", 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 4.1631304350000002, 2024-09-10T19:07:50.640 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0010158299999999999, 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1076 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.412980+0000", 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.641 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.412980+0000", 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.412975+0000", 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.642 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413000+0000", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5012e-05 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413046+0000", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.6142e-05 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413062+0000", 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5863999999999999e-05 2024-09-10T19:07:50.643 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413307+0000", 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024542100000000001 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413584+0000", 2024-09-10T19:07:50.644 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00027747299999999998 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.413996+0000", 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00041130099999999998 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:2528 4.0 4:05bf5b68:::notify.1:head [watch ping cookie 94430446486304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:46.658737+0000", 2024-09-10T19:07:50.645 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.917373022, 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001101122, 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2528 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658737+0000", 2024-09-10T19:07:50.646 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658737+0000", 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658732+0000", 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.647 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658782+0000", 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.9602999999999999e-05 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658837+0000", 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.5025000000000003e-05 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658862+0000", 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.5216e-05 2024-09-10T19:07:50.648 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.659214+0000", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00035192200000000001 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.659531+0000", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00031744599999999998 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.659839+0000", 2024-09-10T19:07:50.649 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00030720699999999999 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:2529 4.14 4:2b04a3e9:::notify.6:head [watch ping cookie 94430446499360 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:46.658902+0000", 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.9172079690000001, 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001774124, 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:07:50.650 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2529 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658902+0000", 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658902+0000", 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.651 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658900+0000", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658911+0000", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.173e-05 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658934+0000", 2024-09-10T19:07:50.652 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3221999999999999e-05 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.658950+0000", 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5874000000000001e-05 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.659999+0000", 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.001048853 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.653 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660232+0000", 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000232492 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660677+0000", 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00044487999999999998 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.654 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24374.0:2530 4.9 4:93e5b521:::notify.7:head [watch ping cookie 94430445882480 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e795)", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:46.660715+0000", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.9153951290000002, 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00054966200000000002, 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24374", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.83:0/2025706139", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 2530 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.655 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660715+0000", 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660715+0000", 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660712+0000", 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999971 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.656 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660728+0000", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5707000000000001e-05 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660766+0000", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.8126000000000001e-05 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.660782+0000", 2024-09-10T19:07:50.657 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6052000000000001e-05 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.661034+0000", 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00025142799999999999 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.661115+0000", 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 8.1426000000000003e-05 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.658 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:46.661265+0000", 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000149987 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1077 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e799)", 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:47.415896+0000", 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 3.160214157, 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00071940100000000005, 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.659 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1077 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415896+0000", 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.660 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415896+0000", 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415891+0000", 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999948 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415915+0000", 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.3969999999999999e-05 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.661 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415959+0000", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4.4181999999999998e-05 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.415974+0000", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5073e-05 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.416195+0000", 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00022030800000000001 2024-09-10T19:07:50.662 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.416397+0000", 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00020294000000000001 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:47.416616+0000", 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00021815499999999999 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.663 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1078 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e800)", 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:48.417572+0000", 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 2.1585388679999999, 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00098643699999999995, 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1078 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.664 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417572+0000", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417572+0000", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417565+0000", 2024-09-10T19:07:50.665 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999933 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417592+0000", 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.764e-05 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417651+0000", 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 5.8857000000000002e-05 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.666 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417667+0000", 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.5423000000000001e-05 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.417896+0000", 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00022928900000000001 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.418144+0000", 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00024838799999999999 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.667 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:48.418558+0000", 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00041367999999999998 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1079 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e801)", 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:49.420267+0000", 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 1.1558437530000001, 2024-09-10T19:07:50.668 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00044947100000000001, 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1079 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420267+0000", 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.669 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420267+0000", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420263+0000", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999962 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420279+0000", 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.6197000000000001e-05 2024-09-10T19:07:50.670 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420311+0000", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 3.1993999999999999e-05 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420324+0000", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.2449999999999999e-05 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420472+0000", 2024-09-10T19:07:50.671 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00014861200000000001 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420581+0000", 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000108859 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.672 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:49.420716+0000", 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00013506600000000001 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "description": "osd_op(client.24493.0:1080 5.0 5:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e803)", 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "initiated_at": "2024-09-10T19:07:50.421560+0000", 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "age": 0.15455076800000001, 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.00039710400000000001, 2024-09-10T19:07:50.673 INFO:teuthology.orchestra.run.smithi083.stdout: "type_data": { 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "flag_point": "started", 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "client_info": { 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "client": "client.24493", 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "client_addr": "v1:172.21.15.165:0/2391841549", 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "tid": 1080 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "events": [ 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "initiated", 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421560+0000", 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.674 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "throttled", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421560+0000", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "header_read", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421557+0000", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 4294967295.9999976 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "all_read", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421569+0000", 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 1.1253999999999999e-05 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.675 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "dispatched", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421590+0000", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 2.1075999999999999e-05 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "queued_for_pg", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421598+0000", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 7.9889999999999992e-06 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "reached_pg", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421744+0000", 2024-09-10T19:07:50.676 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.0001462 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "started", 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421839+0000", 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 9.4913000000000004e-05 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: { 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "event": "done", 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "time": "2024-09-10T19:07:50.421957+0000", 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: "duration": 0.000117898 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.677 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-09-10T19:07:50.678 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-09-10T19:07:50.678 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:07:50.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.123279+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.127541+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.541417+0000 osd.4 (osd.4) 400 : cluster [DBG] 2.f scrub starts 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:49.547179+0000 osd.4 (osd.4) 401 : cluster [DBG] 2.f scrub ok 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.363075+0000 mgr.y (mgr.24526) 727 : cluster [DBG] pgmap v1244: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 84/702 objects degraded (11.966%) 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: audit 2024-09-10T19:07:50.392805+0000 mon.a (mon.0) 3624 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.393721+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e804: 8 total, 7 up, 8 in 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: audit 2024-09-10T19:07:50.451621+0000 mon.c (mon.2) 833 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.679 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[18629]: audit 2024-09-10T19:07:50.452253+0000 mon.a (mon.0) 3626 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.123279+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.127541+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.541417+0000 osd.4 (osd.4) 400 : cluster [DBG] 2.f scrub starts 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:49.547179+0000 osd.4 (osd.4) 401 : cluster [DBG] 2.f scrub ok 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.363075+0000 mgr.y (mgr.24526) 727 : cluster [DBG] pgmap v1244: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 84/702 objects degraded (11.966%) 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: audit 2024-09-10T19:07:50.392805+0000 mon.a (mon.0) 3624 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.393721+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e804: 8 total, 7 up, 8 in 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: audit 2024-09-10T19:07:50.451621+0000 mon.c (mon.2) 833 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.680 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:50 smithi083 bash[32381]: audit 2024-09-10T19:07:50.452253+0000 mon.a (mon.0) 3626 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.123279+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:07:50.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.127541+0000 osd.3 (osd.3) 433 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.541417+0000 osd.4 (osd.4) 400 : cluster [DBG] 2.f scrub starts 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:49.547179+0000 osd.4 (osd.4) 401 : cluster [DBG] 2.f scrub ok 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.363075+0000 mgr.y (mgr.24526) 727 : cluster [DBG] pgmap v1244: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 84/702 objects degraded (11.966%) 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: audit 2024-09-10T19:07:50.392805+0000 mon.a (mon.0) 3624 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.393721+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e804: 8 total, 7 up, 8 in 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: audit 2024-09-10T19:07:50.451621+0000 mon.c (mon.2) 833 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.694 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:50 smithi165 bash[22591]: audit 2024-09-10T19:07:50.452253+0000 mon.a (mon.0) 3626 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:50.713 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:50.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:50.851 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:50.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:50.988 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:50.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:51.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:51.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:51.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.261 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:51.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.402 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:51.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:51.518 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:51.518 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.081627+0000 osd.3 (osd.3) 434 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.086276+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.172060+0000 osd.2 (osd.2) 315 : cluster [DBG] 6.f scrub starts 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.175120+0000 osd.2 (osd.2) 316 : cluster [DBG] 6.f scrub ok 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.501456+0000 osd.4 (osd.4) 402 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.505645+0000 osd.4 (osd.4) 403 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.952306+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:50.956238+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: audit 2024-09-10T19:07:51.396691+0000 mon.a (mon.0) 3627 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: cluster 2024-09-10T19:07:51.396765+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e805: 8 total, 7 up, 8 in 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: audit 2024-09-10T19:07:51.398422+0000 mon.c (mon.2) 834 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.519 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:51 smithi165 bash[22591]: audit 2024-09-10T19:07:51.400021+0000 mon.a (mon.0) 3629 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.540 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:51.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.678 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:51.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.081627+0000 osd.3 (osd.3) 434 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.086276+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.172060+0000 osd.2 (osd.2) 315 : cluster [DBG] 6.f scrub starts 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.175120+0000 osd.2 (osd.2) 316 : cluster [DBG] 6.f scrub ok 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.501456+0000 osd.4 (osd.4) 402 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.505645+0000 osd.4 (osd.4) 403 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.952306+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:50.956238+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: audit 2024-09-10T19:07:51.396691+0000 mon.a (mon.0) 3627 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: cluster 2024-09-10T19:07:51.396765+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e805: 8 total, 7 up, 8 in 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: audit 2024-09-10T19:07:51.398422+0000 mon.c (mon.2) 834 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[18629]: audit 2024-09-10T19:07:51.400021+0000 mon.a (mon.0) 3629 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.081627+0000 osd.3 (osd.3) 434 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.086276+0000 osd.3 (osd.3) 435 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.172060+0000 osd.2 (osd.2) 315 : cluster [DBG] 6.f scrub starts 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.175120+0000 osd.2 (osd.2) 316 : cluster [DBG] 6.f scrub ok 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.501456+0000 osd.4 (osd.4) 402 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.505645+0000 osd.4 (osd.4) 403 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.952306+0000 osd.7 (osd.7) 410 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:50.956238+0000 osd.7 (osd.7) 411 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: audit 2024-09-10T19:07:51.396691+0000 mon.a (mon.0) 3627 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: cluster 2024-09-10T19:07:51.396765+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e805: 8 total, 7 up, 8 in 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: audit 2024-09-10T19:07:51.398422+0000 mon.c (mon.2) 834 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:51 smithi083 bash[32381]: audit 2024-09-10T19:07:51.400021+0000 mon.a (mon.0) 3629 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:51.812 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:51.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.952 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:51.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:52.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:52.092 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:52.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:52.228 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:52.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:52.364 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:52.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:52.478 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: cluster 2024-09-10T19:07:51.171477+0000 osd.2 (osd.2) 317 : cluster [DBG] 12.a scrub starts 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: cluster 2024-09-10T19:07:51.175636+0000 osd.2 (osd.2) 318 : cluster [DBG] 12.a scrub ok 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.364286+0000 mgr.y (mgr.24526) 728 : cluster [DBG] pgmap v1247: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: audit 2024-09-10T19:07:52.398329+0000 mon.a (mon.0) 3630 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.398464+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e806: 8 total, 7 up, 8 in 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: audit 2024-09-10T19:07:52.428974+0000 mon.c (mon.2) 835 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: audit 2024-09-10T19:07:52.429502+0000 mon.a (mon.0) 3632 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: audit 2024-09-10T19:07:52.430363+0000 mon.c (mon.2) 836 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:52 smithi165 bash[22591]: audit 2024-09-10T19:07:52.430796+0000 mon.a (mon.0) 3633 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:52.540 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: cluster 2024-09-10T19:07:51.171477+0000 osd.2 (osd.2) 317 : cluster [DBG] 12.a scrub starts 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: cluster 2024-09-10T19:07:51.175636+0000 osd.2 (osd.2) 318 : cluster [DBG] 12.a scrub ok 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.364286+0000 mgr.y (mgr.24526) 728 : cluster [DBG] pgmap v1247: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: audit 2024-09-10T19:07:52.398329+0000 mon.a (mon.0) 3630 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.398464+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e806: 8 total, 7 up, 8 in 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: audit 2024-09-10T19:07:52.428974+0000 mon.c (mon.2) 835 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: audit 2024-09-10T19:07:52.429502+0000 mon.a (mon.0) 3632 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: audit 2024-09-10T19:07:52.430363+0000 mon.c (mon.2) 836 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[32381]: audit 2024-09-10T19:07:52.430796+0000 mon.a (mon.0) 3633 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: cluster 2024-09-10T19:07:51.171477+0000 osd.2 (osd.2) 317 : cluster [DBG] 12.a scrub starts 2024-09-10T19:07:52.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: cluster 2024-09-10T19:07:51.175636+0000 osd.2 (osd.2) 318 : cluster [DBG] 12.a scrub ok 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.364286+0000 mgr.y (mgr.24526) 728 : cluster [DBG] pgmap v1247: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: audit 2024-09-10T19:07:52.398329+0000 mon.a (mon.0) 3630 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]': finished 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.398464+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e806: 8 total, 7 up, 8 in 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: audit 2024-09-10T19:07:52.428974+0000 mon.c (mon.2) 835 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: audit 2024-09-10T19:07:52.429502+0000 mon.a (mon.0) 3632 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: audit 2024-09-10T19:07:52.430363+0000 mon.c (mon.2) 836 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18629]: audit 2024-09-10T19:07:52.430796+0000 mon.a (mon.0) 3633 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-142"}]: dispatch 2024-09-10T19:07:52.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:07:52.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:52.631 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:52.735 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:52.770 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:52.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:52.899 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:52.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:53.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.032 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:53.033 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:07:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:07:52] "GET /metrics HTTP/1.1" 200 229135 "" "Prometheus/2.43.0" 2024-09-10T19:07:53.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:53.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:53.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.451 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:53.555 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:53.584 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:53.587 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.138996+0000 osd.2 (osd.2) 319 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.142729+0000 osd.2 (osd.2) 320 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.424979+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[18629]: cluster 2024-09-10T19:07:52.428918+0000 osd.4 (osd.4) 405 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.138996+0000 osd.2 (osd.2) 319 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.142729+0000 osd.2 (osd.2) 320 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.424979+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:07:53.588 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:53 smithi083 bash[32381]: cluster 2024-09-10T19:07:52.428918+0000 osd.4 (osd.4) 405 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:07:53.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:53.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:53 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.138996+0000 osd.2 (osd.2) 319 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:07:53.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:53 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.142729+0000 osd.2 (osd.2) 320 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:07:53.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:53 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.424979+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:07:53.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:53 smithi165 bash[22591]: cluster 2024-09-10T19:07:52.428918+0000 osd.4 (osd.4) 405 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:07:53.714 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:53.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:53.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:53.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.990 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:53.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:54.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.127 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:54.232 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:54.268 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:54.371 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:54.402 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:54.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:54.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[32381]: cluster 2024-09-10T19:07:53.415400+0000 osd.4 (osd.4) 406 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:07:54.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[32381]: cluster 2024-09-10T19:07:53.418790+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:07:54.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[32381]: cluster 2024-09-10T19:07:53.485845+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e807: 8 total, 7 up, 8 in 2024-09-10T19:07:54.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[32381]: cluster 2024-09-10T19:07:54.365353+0000 mgr.y (mgr.24526) 729 : cluster [DBG] pgmap v1250: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:54.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[32381]: cluster 2024-09-10T19:07:54.371182+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check update: Degraded data redundancy: 84/705 objects degraded (11.915%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:54.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[18629]: cluster 2024-09-10T19:07:53.415400+0000 osd.4 (osd.4) 406 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:07:54.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[18629]: cluster 2024-09-10T19:07:53.418790+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:07:54.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[18629]: cluster 2024-09-10T19:07:53.485845+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e807: 8 total, 7 up, 8 in 2024-09-10T19:07:54.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[18629]: cluster 2024-09-10T19:07:54.365353+0000 mgr.y (mgr.24526) 729 : cluster [DBG] pgmap v1250: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:54.517 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:54 smithi083 bash[18629]: cluster 2024-09-10T19:07:54.371182+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check update: Degraded data redundancy: 84/705 objects degraded (11.915%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:54.530 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:54.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:54 smithi165 bash[22591]: cluster 2024-09-10T19:07:53.415400+0000 osd.4 (osd.4) 406 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:07:54.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:54 smithi165 bash[22591]: cluster 2024-09-10T19:07:53.418790+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:07:54.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:54 smithi165 bash[22591]: cluster 2024-09-10T19:07:53.485845+0000 mon.a (mon.0) 3634 : cluster [DBG] osdmap e807: 8 total, 7 up, 8 in 2024-09-10T19:07:54.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:54 smithi165 bash[22591]: cluster 2024-09-10T19:07:54.365353+0000 mgr.y (mgr.24526) 729 : cluster [DBG] pgmap v1250: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 84/702 objects degraded (11.966%) 2024-09-10T19:07:54.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:54 smithi165 bash[22591]: cluster 2024-09-10T19:07:54.371182+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check update: Degraded data redundancy: 84/705 objects degraded (11.915%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:07:54.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:54.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:54.808 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:54.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.945 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:54.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:55.049 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:55.085 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:55.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.224 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:55.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.356 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:55.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.489 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:55.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[18629]: cluster 2024-09-10T19:07:54.507178+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e808: 8 total, 7 up, 8 in 2024-09-10T19:07:55.522 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[18629]: audit 2024-09-10T19:07:54.514355+0000 mon.c (mon.2) 837 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:55.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:55.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:55.755 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:55.814 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[18629]: audit 2024-09-10T19:07:54.514961+0000 mon.a (mon.0) 3637 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.814 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[18629]: cluster 2024-09-10T19:07:55.035671+0000 osd.7 (osd.7) 412 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:07:55.814 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[18629]: cluster 2024-09-10T19:07:55.039498+0000 osd.7 (osd.7) 413 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:07:55.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[32381]: cluster 2024-09-10T19:07:54.507178+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e808: 8 total, 7 up, 8 in 2024-09-10T19:07:55.814 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[32381]: audit 2024-09-10T19:07:54.514355+0000 mon.c (mon.2) 837 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[32381]: audit 2024-09-10T19:07:54.514961+0000 mon.a (mon.0) 3637 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[32381]: cluster 2024-09-10T19:07:55.035671+0000 osd.7 (osd.7) 412 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:07:55.815 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:55 smithi083 bash[32381]: cluster 2024-09-10T19:07:55.039498+0000 osd.7 (osd.7) 413 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:07:55.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:55 smithi165 bash[22591]: cluster 2024-09-10T19:07:54.507178+0000 mon.a (mon.0) 3636 : cluster [DBG] osdmap e808: 8 total, 7 up, 8 in 2024-09-10T19:07:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:55 smithi165 bash[22591]: audit 2024-09-10T19:07:54.514355+0000 mon.c (mon.2) 837 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:55 smithi165 bash[22591]: audit 2024-09-10T19:07:54.514961+0000 mon.a (mon.0) 3637 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:55 smithi165 bash[22591]: cluster 2024-09-10T19:07:55.035671+0000 osd.7 (osd.7) 412 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:07:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:55 smithi165 bash[22591]: cluster 2024-09-10T19:07:55.039498+0000 osd.7 (osd.7) 413 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:07:55.895 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:55.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:55.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:56.029 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:56.133 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:56.167 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.d 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.b 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.4 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.1d 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.1a 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:6.19 2024-09-10T19:07:56.235 INFO:teuthology.orchestra.run.smithi083.stdout:5.0 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:4.f 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:4.e 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:256.14 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:3.0 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:256.12 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:4.9 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:2.8 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:6.18 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:2.2 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:12.6 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:12.1b 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:2.a 2024-09-10T19:07:56.236 INFO:teuthology.orchestra.run.smithi083.stdout:12.4 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:241.6s0 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:2.15 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:256.1c 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:3.6 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:2.10 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:1.3 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:12.12 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:256.a 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:2.11 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:256.e 2024-09-10T19:07:56.237 INFO:teuthology.orchestra.run.smithi083.stdout:12.7 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:2.4 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:12.10 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:256.1e 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:4.a 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:6.1 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:2.d 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:12.15 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:4.6 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:4.17 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:241.7s2 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:12.13 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:12.18 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:4.14 2024-09-10T19:07:56.238 INFO:teuthology.orchestra.run.smithi083.stdout:12.5 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:12.d 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:4.7 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:2.0 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.8 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:2.9 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.18 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.19 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:2.6 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.1a 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.1c 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:256.5 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.4 2024-09-10T19:07:56.239 INFO:teuthology.orchestra.run.smithi083.stdout:3.e 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:6.e 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:256.7 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.10 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.13 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:3.a 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.1b 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.1c 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:6.10 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.2 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:3.16 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.4 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.b 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.1d 2024-09-10T19:07:56.240 INFO:teuthology.orchestra.run.smithi083.stdout:4.d 2024-09-10T19:07:56.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:56.305 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:56.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:56.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:07:56.544 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:56.569 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:55.512898+0000 mon.a (mon.0) 3638 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: cluster 2024-09-10T19:07:55.513081+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e809: 8 total, 7 up, 8 in 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:55.538014+0000 mon.c (mon.2) 838 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:55.538393+0000 mon.a (mon.0) 3640 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: cluster 2024-09-10T19:07:55.757788+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: cluster 2024-09-10T19:07:55.761219+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.367593+0000 mgr.y (mgr.24526) 730 : cluster [DBG] pgmap v1253: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:56.514946+0000 mon.a (mon.0) 3641 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.515067+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e810: 8 total, 7 up, 8 in 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:56.516272+0000 mon.c (mon.2) 839 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:56 smithi165 bash[22591]: audit 2024-09-10T19:07:56.517088+0000 mon.a (mon.0) 3643 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:56.702 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:56.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:55.512898+0000 mon.a (mon.0) 3638 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: cluster 2024-09-10T19:07:55.513081+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e809: 8 total, 7 up, 8 in 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:55.538014+0000 mon.c (mon.2) 838 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:55.538393+0000 mon.a (mon.0) 3640 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: cluster 2024-09-10T19:07:55.757788+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:07:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: cluster 2024-09-10T19:07:55.761219+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.367593+0000 mgr.y (mgr.24526) 730 : cluster [DBG] pgmap v1253: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:56.514946+0000 mon.a (mon.0) 3641 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.515067+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e810: 8 total, 7 up, 8 in 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:56.516272+0000 mon.c (mon.2) 839 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[32381]: audit 2024-09-10T19:07:56.517088+0000 mon.a (mon.0) 3643 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:55.512898+0000 mon.a (mon.0) 3638 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: cluster 2024-09-10T19:07:55.513081+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e809: 8 total, 7 up, 8 in 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:55.538014+0000 mon.c (mon.2) 838 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:55.538393+0000 mon.a (mon.0) 3640 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: cluster 2024-09-10T19:07:55.757788+0000 osd.6 (osd.6) 371 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:07:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: cluster 2024-09-10T19:07:55.761219+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:07:56.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.367593+0000 mgr.y (mgr.24526) 730 : cluster [DBG] pgmap v1253: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:56.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:56.514946+0000 mon.a (mon.0) 3641 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:56.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.515067+0000 mon.a (mon.0) 3642 : cluster [DBG] osdmap e810: 8 total, 7 up, 8 in 2024-09-10T19:07:56.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:56.516272+0000 mon.c (mon.2) 839 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:56 smithi083 bash[18629]: audit 2024-09-10T19:07:56.517088+0000 mon.a (mon.0) 3643 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:07:56.805 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:56.831 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:56.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:56.962 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:56.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:57.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:57.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:57.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:57.240 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:57.343 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:57.344 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:07:57.378 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:57.481 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:57.514 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:57.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.131230+0000 osd.3 (osd.3) 436 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:07:57.617 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.134228+0000 osd.3 (osd.3) 437 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.349210+0000 osd.0 (osd.0) 389 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.352621+0000 osd.0 (osd.0) 390 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.387312+0000 osd.4 (osd.4) 408 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.393551+0000 osd.4 (osd.4) 409 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.993253+0000 osd.7 (osd.7) 414 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:56.997055+0000 osd.7 (osd.7) 415 : cluster [DBG] 2.b scrub ok 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: audit 2024-09-10T19:07:57.517380+0000 mon.a (mon.0) 3644 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: cluster 2024-09-10T19:07:57.517547+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e811: 8 total, 7 up, 8 in 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: audit 2024-09-10T19:07:57.522940+0000 mon.c (mon.2) 840 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[32381]: audit 2024-09-10T19:07:57.523331+0000 mon.a (mon.0) 3646 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.618 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.131230+0000 osd.3 (osd.3) 436 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.134228+0000 osd.3 (osd.3) 437 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.349210+0000 osd.0 (osd.0) 389 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.352621+0000 osd.0 (osd.0) 390 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.387312+0000 osd.4 (osd.4) 408 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.393551+0000 osd.4 (osd.4) 409 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.993253+0000 osd.7 (osd.7) 414 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:56.997055+0000 osd.7 (osd.7) 415 : cluster [DBG] 2.b scrub ok 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: audit 2024-09-10T19:07:57.517380+0000 mon.a (mon.0) 3644 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: cluster 2024-09-10T19:07:57.517547+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e811: 8 total, 7 up, 8 in 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: audit 2024-09-10T19:07:57.522940+0000 mon.c (mon.2) 840 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.619 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:57 smithi083 bash[18629]: audit 2024-09-10T19:07:57.523331+0000 mon.a (mon.0) 3646 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.627 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:57.658 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:57.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:57.793 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.131230+0000 osd.3 (osd.3) 436 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:07:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.134228+0000 osd.3 (osd.3) 437 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.349210+0000 osd.0 (osd.0) 389 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.352621+0000 osd.0 (osd.0) 390 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.387312+0000 osd.4 (osd.4) 408 : cluster [DBG] 241.2 scrub starts 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.393551+0000 osd.4 (osd.4) 409 : cluster [DBG] 241.2 scrub ok 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.993253+0000 osd.7 (osd.7) 414 : cluster [DBG] 2.b scrub starts 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:56.997055+0000 osd.7 (osd.7) 415 : cluster [DBG] 2.b scrub ok 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: audit 2024-09-10T19:07:57.517380+0000 mon.a (mon.0) 3644 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: cluster 2024-09-10T19:07:57.517547+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e811: 8 total, 7 up, 8 in 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: audit 2024-09-10T19:07:57.522940+0000 mon.c (mon.2) 840 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:57 smithi165 bash[22591]: audit 2024-09-10T19:07:57.523331+0000 mon.a (mon.0) 3646 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:07:57.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:57.928 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:57.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:58.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:58.066 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:58.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:58.201 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:58.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:58.336 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:58.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:58.472 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:07:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:07:58.539 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:07:58.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: cluster 2024-09-10T19:07:57.770433+0000 osd.6 (osd.6) 373 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:07:58.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: cluster 2024-09-10T19:07:57.773836+0000 osd.6 (osd.6) 374 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:07:58.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: cluster 2024-09-10T19:07:58.368689+0000 mgr.y (mgr.24526) 731 : cluster [DBG] pgmap v1256: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:58.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: cluster 2024-09-10T19:07:58.511170+0000 mon.a (mon.0) 3647 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:58.539 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: audit 2024-09-10T19:07:58.520206+0000 mon.a (mon.0) 3648 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]': finished 2024-09-10T19:07:58.540 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:58 smithi165 bash[22591]: cluster 2024-09-10T19:07:58.520364+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e812: 8 total, 7 up, 8 in 2024-09-10T19:07:58.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:58.611 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:58.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 6.19 --file /var/log/ceph/exp.6.19.1 2024-09-10T19:07:58.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:58.745 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:58.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: cluster 2024-09-10T19:07:57.770433+0000 osd.6 (osd.6) 373 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: cluster 2024-09-10T19:07:57.773836+0000 osd.6 (osd.6) 374 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: cluster 2024-09-10T19:07:58.368689+0000 mgr.y (mgr.24526) 731 : cluster [DBG] pgmap v1256: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: cluster 2024-09-10T19:07:58.511170+0000 mon.a (mon.0) 3647 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: audit 2024-09-10T19:07:58.520206+0000 mon.a (mon.0) 3648 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]': finished 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[32381]: cluster 2024-09-10T19:07:58.520364+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e812: 8 total, 7 up, 8 in 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: cluster 2024-09-10T19:07:57.770433+0000 osd.6 (osd.6) 373 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: cluster 2024-09-10T19:07:57.773836+0000 osd.6 (osd.6) 374 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: cluster 2024-09-10T19:07:58.368689+0000 mgr.y (mgr.24526) 731 : cluster [DBG] pgmap v1256: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: cluster 2024-09-10T19:07:58.511170+0000 mon.a (mon.0) 3647 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:58.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: audit 2024-09-10T19:07:58.520206+0000 mon.a (mon.0) 3648 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-144", "mode": "readproxy"}]': finished 2024-09-10T19:07:58.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:58 smithi083 bash[18629]: cluster 2024-09-10T19:07:58.520364+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e812: 8 total, 7 up, 8 in 2024-09-10T19:07:58.821 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:07:58.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:58.882 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:58.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:58.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:59.015 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:07:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:07:59.118 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.152 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:59.256 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:59.287 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:07:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:07:59.390 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:59.415 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:07:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:07:59.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.550 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:59.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:59 smithi165 bash[22591]: cluster 2024-09-10T19:07:58.216909+0000 osd.2 (osd.2) 321 : cluster [DBG] 2.c scrub starts 2024-09-10T19:07:59.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:59 smithi165 bash[22591]: cluster 2024-09-10T19:07:58.220549+0000 osd.2 (osd.2) 322 : cluster [DBG] 2.c scrub ok 2024-09-10T19:07:59.671 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:07:59 smithi165 bash[22591]: audit 2024-09-10T19:07:58.424943+0000 mgr.y (mgr.24526) 732 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:59.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:07:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:07:59.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:59.807 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[18629]: cluster 2024-09-10T19:07:58.216909+0000 osd.2 (osd.2) 321 : cluster [DBG] 2.c scrub starts 2024-09-10T19:07:59.808 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[18629]: cluster 2024-09-10T19:07:58.220549+0000 osd.2 (osd.2) 322 : cluster [DBG] 2.c scrub ok 2024-09-10T19:07:59.808 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[18629]: audit 2024-09-10T19:07:58.424943+0000 mgr.y (mgr.24526) 732 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:59.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[32381]: cluster 2024-09-10T19:07:58.216909+0000 osd.2 (osd.2) 321 : cluster [DBG] 2.c scrub starts 2024-09-10T19:07:59.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[32381]: cluster 2024-09-10T19:07:58.220549+0000 osd.2 (osd.2) 322 : cluster [DBG] 2.c scrub ok 2024-09-10T19:07:59.808 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:07:59 smithi083 bash[32381]: audit 2024-09-10T19:07:58.424943+0000 mgr.y (mgr.24526) 732 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:59.830 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:07:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:07:59.934 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:59.971 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:07:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:07:59.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:00.075 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:00.104 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:00.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:00.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:00.241 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:00.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:00.377 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:00.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:00.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:00.515 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:00 smithi165 bash[22591]: audit 2024-09-10T19:07:59.576210+0000 mon.c (mon.2) 841 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:59.740629+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:08:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:00 smithi165 bash[22591]: cluster 2024-09-10T19:07:59.744960+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:08:00.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.369725+0000 mgr.y (mgr.24526) 733 : cluster [DBG] pgmap v1258: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:00.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:00.651 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:00.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:00.767 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[18629]: audit 2024-09-10T19:07:59.576210+0000 mon.c (mon.2) 841 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:59.740629+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[18629]: cluster 2024-09-10T19:07:59.744960+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.369725+0000 mgr.y (mgr.24526) 733 : cluster [DBG] pgmap v1258: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[32381]: audit 2024-09-10T19:07:59.576210+0000 mon.c (mon.2) 841 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:59.740629+0000 osd.6 (osd.6) 375 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[32381]: cluster 2024-09-10T19:07:59.744960+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:08:00.768 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.369725+0000 mgr.y (mgr.24526) 733 : cluster [DBG] pgmap v1258: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 210 B/s wr, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.793 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:00.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:00.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:00.931 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:00.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:01.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:01.073 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:01.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:01.209 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:01.313 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:01.348 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:01.451 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:01.486 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:01.510 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:08:01.510 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:08:01.510 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:08:01.510 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:08:01.536 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:01.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:01.622 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:01.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:01.762 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.432906+0000 osd.4 (osd.4) 410 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.438150+0000 osd.4 (osd.4) 411 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.713723+0000 osd.6 (osd.6) 377 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.714331+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub starts 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.717679+0000 osd.6 (osd.6) 378 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:08:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[32381]: cluster 2024-09-10T19:08:00.719196+0000 osd.5 (osd.5) 423 : cluster [DBG] 3.c scrub ok 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.432906+0000 osd.4 (osd.4) 410 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.438150+0000 osd.4 (osd.4) 411 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.713723+0000 osd.6 (osd.6) 377 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.714331+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub starts 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.717679+0000 osd.6 (osd.6) 378 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:08:01.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:01 smithi083 bash[18629]: cluster 2024-09-10T19:08:00.719196+0000 osd.5 (osd.5) 423 : cluster [DBG] 3.c scrub ok 2024-09-10T19:08:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.432906+0000 osd.4 (osd.4) 410 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:08:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.438150+0000 osd.4 (osd.4) 411 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:08:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.713723+0000 osd.6 (osd.6) 377 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:08:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.714331+0000 osd.5 (osd.5) 422 : cluster [DBG] 3.c scrub starts 2024-09-10T19:08:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.717679+0000 osd.6 (osd.6) 378 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:08:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:01 smithi165 bash[22591]: cluster 2024-09-10T19:08:00.719196+0000 osd.5 (osd.5) 423 : cluster [DBG] 3.c scrub ok 2024-09-10T19:08:01.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:01.896 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:01.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:02.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:02.037 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:02.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:02.171 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:02.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:08:02.274 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:02.302 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:02.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:02.436 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:02.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- 172.21.15.83:0/1383007826 >> v1:172.21.15.83:6789/0 conn(0x7f8770107e00 legacy=0x7f877010a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- 172.21.15.83:0/1383007826 shutdown_connections 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- 172.21.15.83:0/1383007826 >> 172.21.15.83:0/1383007826 conn(0x7f87700f70b0 msgr2=0x7f87700f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- 172.21.15.83:0/1383007826 shutdown_connections 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- 172.21.15.83:0/1383007826 wait complete. 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 librados: starting msgr at 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 librados: starting objecter 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 Processor -- start 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- start start 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 librados: setting wanted keys 2024-09-10T19:08:02.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 librados: calling monclient init 2024-09-10T19:08:02.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8770107b60 con 0x7f87701042a0 2024-09-10T19:08:02.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f877011d420 con 0x7f8770107e00 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f87753ac640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f877011e5c0 con 0x7f8770101530 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.464+0000 7f876e7fc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f87701042a0 0x7f87701175c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33648/0 (socket says 172.21.15.83:33648) 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f876e7fc640 1 -- 172.21.15.83:0/3975971342 learned_addr learned my addr 172.21.15.83:0/3975971342 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f876effd640 1 --1- 172.21.15.83:0/3975971342 >> v1:172.21.15.83:6790/0 conn(0x7f8770101530 0x7f8770107040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:42232/0 (socket says 172.21.15.83:42232) 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter ms_handle_connect 0x7f87701042a0 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter ms_handle_connect 0x7f8770101530 2024-09-10T19:08:02.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter ms_handle_connect 0x7f8770107e00 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742411664 0 0) 0x7f8770107b60 con 0x7f87701042a0 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87580034e0 con 0x7f87701042a0 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920763282 0 0) 0x7f877011e5c0 con 0x7f8770101530 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8770107b60 con 0x7f8770101530 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687267774 0 0) 0x7f877011d420 con 0x7f8770107e00 2024-09-10T19:08:02.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877011e5c0 con 0x7f8770107e00 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3618998178 0 0) 0x7f877011e5c0 con 0x7f8770107e00 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f877011d420 con 0x7f8770107e00 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857822476 0 0) 0x7f87580034e0 con 0x7f87701042a0 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f877011e5c0 con 0x7f87701042a0 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8768001430 con 0x7f8770107e00 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8764002910 con 0x7f87701042a0 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1182199206 0 0) 0x7f877011e5c0 con 0x7f87701042a0 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 >> v1:172.21.15.83:6790/0 conn(0x7f8770101530 legacy=0x7f8770107040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 >> v1:172.21.15.165:6789/0 conn(0x7f8770107e00 legacy=0x7f877011acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877011f7a0 con 0x7f87701042a0 2024-09-10T19:08:02.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877011e7b0 con 0x7f87701042a0 2024-09-10T19:08:02.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87753ac640 10 client.25761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:02.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f877011ec80 con 0x7f87701042a0 2024-09-10T19:08:02.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87753ac640 1 librados: init done 2024-09-10T19:08:02.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f8764001790 con 0x7f87701042a0 2024-09-10T19:08:02.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.468+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f8764005a60 con 0x7f87701042a0 2024-09-10T19:08:02.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f875c004ff0 con 0x7f87701042a0 2024-09-10T19:08:02.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f876401bf50 con 0x7f87701042a0 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(812..812 src has 750..812) v4 ==== 8531+0+0 (unknown 2322846786 0 0) 0x7f876408a680 con 0x7f87701042a0 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 osd_map(812..812 src has 750..812) v4 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 3 client.25761.objecter handle_osd_map got epochs [812,812] > 0 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 3 client.25761.objecter handle_osd_map decoding full epoch 812 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 20 client.25761.objecter dump_active .. 0 homeless 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v812) v1 ==== 54+0+16 (unknown 59176876 0 2420000192) 0x7f876408ca70 con 0x7f87701042a0 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f8758071560 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 10 client.25761.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 1 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 1 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 0 1 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 4 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 10 client.25761.objecter _send_command 1 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c004a60 con 0x7f875c004610 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c004610 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c004a60 con 0x7f875c004610 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 10 client.25761.objecter _finish_command 1 = system:0 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 4 2024-09-10T19:08:02.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.472+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 0 1 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 10 client.25761.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 2 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 2 2024-09-10T19:08:02.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c005290 osd=0 3 2024-09-10T19:08:02.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 0 2 2024-09-10T19:08:02.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 4 2024-09-10T19:08:02.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 10 client.25761.objecter _send_command 2 2024-09-10T19:08:02.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.488+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c004e40 con 0x7f875c004610 2024-09-10T19:08:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c004e40 con 0x7f875c004610 2024-09-10T19:08:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 10 client.25761.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c005290 osd=0 4 2024-09-10T19:08:02.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 0 2 2024-09-10T19:08:02.496 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 10 client.25761.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 3 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter _get_session osd=-1 returning homeless 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 10 client.25761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=813}) v3 -- 0x7f875c003fa0 con 0x7f87701042a0 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f875c004380 con 0x7f87701042a0 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_get_version_reply(handle=1 version=812) v2 ==== 24+0+0 (unknown 1932003993 0 0) 0x7f8764056a30 con 0x7f87701042a0 2024-09-10T19:08:02.497 INFO:teuthology.orchestra.run.smithi083.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f876d7fa640 10 client.25761.objecter _check_command_map_dne tid 3 current 812 map_dne_bound 812 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f876d7fa640 10 client.25761.objecter _finish_command 3 = osd:6 osd down 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f876d7fa640 15 client.25761.objecter _session_command_op_remove -1 3 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 10 client.25761.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 4 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 4 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 2 4 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 4 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 10 client.25761.objecter _send_command 4 2024-09-10T19:08:02.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6809/386094220 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c00c810 con 0x7f875c002150 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c002150 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c00c810 con 0x7f875c002150 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 4: 0 ) v1 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 10 client.25761.objecter _finish_command 4 = system:0 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 4 2024-09-10T19:08:02.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.492+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 2 4 2024-09-10T19:08:02.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- 172.21.15.83:0/464721663 >> v1:172.21.15.83:6789/0 conn(0x7f950c1014e0 legacy=0x7f950c1018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- 172.21.15.83:0/464721663 shutdown_connections 2024-09-10T19:08:02.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- 172.21.15.83:0/464721663 >> 172.21.15.83:0/464721663 conn(0x7f950c075290 msgr2=0x7f950c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:02.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- 172.21.15.83:0/464721663 shutdown_connections 2024-09-10T19:08:02.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- 172.21.15.83:0/464721663 wait complete. 2024-09-10T19:08:02.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 librados: starting msgr at 2024-09-10T19:08:02.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 librados: starting objecter 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 Processor -- start 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- start start 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 librados: setting wanted keys 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 librados: calling monclient init 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f950c100cb0 con 0x7f950c1042a0 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f950c100e60 con 0x7f950c107e30 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f9511ca4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f950c1206d0 con 0x7f950c1014e0 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f950b7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f950c1014e0 0x7f950c1003f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:42244/0 (socket says 172.21.15.83:42244) 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f950b7fe640 1 -- 172.21.15.83:0/3992153229 learned_addr learned my addr 172.21.15.83:0/3992153229 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter ms_handle_connect 0x7f950c1014e0 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter ms_handle_connect 0x7f950c107e30 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532804391 0 0) 0x7f950c1206d0 con 0x7f950c1014e0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f40034e0 con 0x7f950c1014e0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter ms_handle_connect 0x7f950c1042a0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606000597 0 0) 0x7f950c100e60 con 0x7f950c107e30 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f950c1206d0 con 0x7f950c107e30 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232494658 0 0) 0x7f950c100cb0 con 0x7f950c1042a0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f950c100e60 con 0x7f950c1042a0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152894413 0 0) 0x7f94f40034e0 con 0x7f950c1014e0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f950c100cb0 con 0x7f950c1014e0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f94fc0012a0 con 0x7f950c1014e0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571458153 0 0) 0x7f950c100e60 con 0x7f950c1042a0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94f40034e0 con 0x7f950c1042a0 2024-09-10T19:08:02.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9500001840 con 0x7f950c1042a0 2024-09-10T19:08:02.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847803630 0 0) 0x7f950c1206d0 con 0x7f950c107e30 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f950c100e60 con 0x7f950c107e30 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.504+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9504001bb0 con 0x7f950c107e30 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4173966066 0 0) 0x7f950c100cb0 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 >> v1:172.21.15.165:6789/0 conn(0x7f950c107e30 legacy=0x7f950c11cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 >> v1:172.21.15.83:6789/0 conn(0x7f950c1042a0 legacy=0x7f950c119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c1218b0 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c1208c0 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f9511ca4640 10 client.49853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c120d10 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f9511ca4640 1 librados: init done 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f94fc0014c0 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f94fc003560 con 0x7f950c1014e0 2024-09-10T19:08:02.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f8005010 con 0x7f950c1014e0 2024-09-10T19:08:02.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f94fc0036c0 con 0x7f950c1014e0 2024-09-10T19:08:02.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(812..812 src has 750..812) v4 ==== 8531+0+0 (unknown 2322846786 0 0) 0x7f94fc08a890 con 0x7f950c1014e0 2024-09-10T19:08:02.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 10 client.49853.objecter ms_dispatch 0x7f950c000b90 osd_map(812..812 src has 750..812) v4 2024-09-10T19:08:02.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 3 client.49853.objecter handle_osd_map got epochs [812,812] > 0 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.508+0000 7f94f3fff640 3 client.49853.objecter handle_osd_map decoding full epoch 812 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f94f3fff640 20 client.49853.objecter dump_active .. 0 homeless 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f94f3fff640 10 client.49853.objecter ms_handle_connect 0x7f94f40718d0 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 10 client.25761.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 5 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 5 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c001fe0 osd=2 3 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 2 5 2024-09-10T19:08:02.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 4 2024-09-10T19:08:02.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 10 client.25761.objecter _send_command 5 2024-09-10T19:08:02.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c001b30 con 0x7f875c002150 2024-09-10T19:08:02.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c001b30 con 0x7f875c002150 2024-09-10T19:08:02.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 10 client.25761.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c001fe0 osd=2 4 2024-09-10T19:08:02.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 2 5 2024-09-10T19:08:02.517 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:08:02.517 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 10 client.25761.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 6 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 6 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 3 6 2024-09-10T19:08:02.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 4 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 10 client.25761.objecter _send_command 6 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6813/1279688021 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c003df0 con 0x7f875c00db30 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c00db30 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c003df0 con 0x7f875c00db30 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 6: 0 ) v1 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 10 client.25761.objecter _finish_command 6 = system:0 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 4 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.512+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 3 6 2024-09-10T19:08:02.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.516+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f94fc056c60 con 0x7f950c1014e0 2024-09-10T19:08:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 10 client.25761.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 7 2024-09-10T19:08:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 7 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c0035e0 osd=3 3 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 3 7 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 4 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 10 client.25761.objecter _send_command 7 2024-09-10T19:08:02.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c00ef00 con 0x7f875c00db30 2024-09-10T19:08:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c00ef00 con 0x7f875c00db30 2024-09-10T19:08:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87577fe640 10 client.25761.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c0035e0 osd=3 4 2024-09-10T19:08:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 3 7 2024-09-10T19:08:02.538 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:08:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:08:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 10 client.25761.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 8 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 8 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 4 8 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 4 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 10 client.25761.objecter _send_command 8 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.532+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6800/3225515265 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c0162d0 con 0x7f875c0127d0 2024-09-10T19:08:02.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c0127d0 2024-09-10T19:08:02.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c0162d0 con 0x7f875c0127d0 2024-09-10T19:08:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 8: 0 ) v1 2024-09-10T19:08:02.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 10 client.25761.objecter _finish_command 8 = system:0 2024-09-10T19:08:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 4 2024-09-10T19:08:02.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.536+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 4 8 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 10 client.25761.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 9 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 9 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c0032a0 osd=4 3 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 4 9 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 4 2024-09-10T19:08:02.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 10 client.25761.objecter _send_command 9 2024-09-10T19:08:02.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.548+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c00e980 con 0x7f875c0127d0 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c00e980 con 0x7f875c0127d0 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 10 client.25761.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c0032a0 osd=4 4 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 4 9 2024-09-10T19:08:02.556 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 10 client.25761.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 10 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 10 2024-09-10T19:08:02.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 5 10 2024-09-10T19:08:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 4 2024-09-10T19:08:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 10 client.25761.objecter _send_command 10 2024-09-10T19:08:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6804/571173777 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c00e8b0 con 0x7f875c00d680 2024-09-10T19:08:02.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.552+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c00d680 2024-09-10T19:08:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.556+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c00e8b0 con 0x7f875c00d680 2024-09-10T19:08:02.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.556+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 10: 0 ) v1 2024-09-10T19:08:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.556+0000 7f87577fe640 10 client.25761.objecter _finish_command 10 = system:0 2024-09-10T19:08:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.556+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 4 2024-09-10T19:08:02.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.556+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 5 10 2024-09-10T19:08:02.571 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:02.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 10 client.25761.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 11 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 11 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c00e740 osd=5 3 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 5 11 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 4 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 10 client.25761.objecter _send_command 11 2024-09-10T19:08:02.578 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.572+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c00e8b0 con 0x7f875c00d680 2024-09-10T19:08:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c00e8b0 con 0x7f875c00d680 2024-09-10T19:08:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 10 client.25761.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c00e740 osd=5 4 2024-09-10T19:08:02.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 5 11 2024-09-10T19:08:02.581 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:08:02.581 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:08:02.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 10 client.25761.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 12 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 12 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 6 12 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 4 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 10 client.25761.objecter _send_command 12 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6808/1317922936 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c01ee90 con 0x7f875c01b200 2024-09-10T19:08:02.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.576+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c01b200 2024-09-10T19:08:02.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.580+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c01ee90 con 0x7f875c01b200 2024-09-10T19:08:02.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.580+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 12: 0 ) v1 2024-09-10T19:08:02.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.580+0000 7f87577fe640 10 client.25761.objecter _finish_command 12 = system:0 2024-09-10T19:08:02.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.580+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 4 2024-09-10T19:08:02.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.580+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 6 12 2024-09-10T19:08:02.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 10 client.25761.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 13 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 13 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c00e8b0 osd=6 3 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 6 13 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 4 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 10 client.25761.objecter _send_command 13 2024-09-10T19:08:02.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c020730 con 0x7f875c01b200 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c020730 con 0x7f875c01b200 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87577fe640 10 client.25761.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c00e8b0 osd=6 4 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.596+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 6 13 2024-09-10T19:08:02.602 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:08:02.603 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:08:02.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 10 client.25761.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:02.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 14 2024-09-10T19:08:02.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 14 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 7 14 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 4 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 10 client.25761.objecter _send_command 14 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6812/1610151650 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f875c0212e0 con 0x7f875c020db0 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 10 client.25761.objecter ms_handle_connect 0x7f875c020db0 2024-09-10T19:08:02.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f875c0212e0 con 0x7f875c020db0 2024-09-10T19:08:02.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 14: 0 ) v1 2024-09-10T19:08:02.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 10 client.25761.objecter _finish_command 14 = system:0 2024-09-10T19:08:02.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 4 2024-09-10T19:08:02.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.600+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 7 14 2024-09-10T19:08:02.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 10 client.25761.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign -1 15 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 20 client.25761.objecter _get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 15 client.25761.objecter _session_command_op_remove -1 15 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 20 client.25761.objecter get_session s=0x7f875c020730 osd=7 3 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 15 client.25761.objecter _session_command_op_assign 7 15 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 4 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 10 client.25761.objecter _send_command 15 2024-09-10T19:08:02.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.628+0000 7f87753ac640 1 -- 172.21.15.83:0/3975971342 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f875c0212e0 con 0x7f875c020db0 2024-09-10T19:08:02.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.632+0000 7f87577fe640 1 -- 172.21.15.83:0/3975971342 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f875c0212e0 con 0x7f875c020db0 2024-09-10T19:08:02.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.632+0000 7f87577fe640 10 client.25761.objecter ms_dispatch 0x7f8770000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:02.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.632+0000 7f87577fe640 10 client.25761.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:02.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.632+0000 7f87577fe640 20 client.25761.objecter put_session s=0x7f875c020730 osd=7 4 2024-09-10T19:08:02.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.632+0000 7f87577fe640 15 client.25761.objecter _session_command_op_remove 7 15 2024-09-10T19:08:02.637 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:08:02.637 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:08:02.674 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:02.701 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:02.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:02.732+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f94f80052c0 con 0x7f950c1014e0 2024-09-10T19:08:02.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:02] "GET /metrics HTTP/1.1" 200 229161 "" "Prometheus/2.43.0" 2024-09-10T19:08:02.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[32381]: cluster 2024-09-10T19:08:01.404573+0000 osd.4 (osd.4) 412 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:08:02.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[32381]: cluster 2024-09-10T19:08:01.407538+0000 osd.4 (osd.4) 413 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:08:02.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[32381]: cluster 2024-09-10T19:08:02.371097+0000 mgr.y (mgr.24526) 734 : cluster [DBG] pgmap v1259: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:02.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[32381]: audit 2024-09-10T19:08:02.475188+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? v1:172.21.15.83:0/3975971342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[18629]: cluster 2024-09-10T19:08:01.404573+0000 osd.4 (osd.4) 412 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:08:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[18629]: cluster 2024-09-10T19:08:01.407538+0000 osd.4 (osd.4) 413 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:08:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[18629]: cluster 2024-09-10T19:08:02.371097+0000 mgr.y (mgr.24526) 734 : cluster [DBG] pgmap v1259: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:02 smithi083 bash[18629]: audit 2024-09-10T19:08:02.475188+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? v1:172.21.15.83:0/3975971342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:02.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:02.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:02 smithi165 bash[22591]: cluster 2024-09-10T19:08:01.404573+0000 osd.4 (osd.4) 412 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:08:02.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:02 smithi165 bash[22591]: cluster 2024-09-10T19:08:01.407538+0000 osd.4 (osd.4) 413 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:08:02.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:02 smithi165 bash[22591]: cluster 2024-09-10T19:08:02.371097+0000 mgr.y (mgr.24526) 734 : cluster [DBG] pgmap v1259: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:02.817 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:02 smithi165 bash[22591]: audit 2024-09-10T19:08:02.475188+0000 mon.a (mon.0) 3650 : audit [DBG] from='client.? v1:172.21.15.83:0/3975971342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:02.840 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:02.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:02.979 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:02.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:03.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:03.119 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:03.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:03.260 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:03.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:03.396 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:03.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:03.539 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:03.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.576+0000 7f94f3fff640 1 -- 172.21.15.83:0/3992153229 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v813) v1 ==== 101+0+0 (unknown 1981131982 0 0) 0x7f94fc056600 con 0x7f950c1014e0 2024-09-10T19:08:03.580 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:08:03.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 10 librados: watch_flush enter 2024-09-10T19:08:03.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 10 librados: watch_flush exit 2024-09-10T19:08:03.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 20 client.49853.objecter shutdown clearing up homeless session... 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 10 client.49853.objecter successfully canceled tick 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 >> v1:172.21.15.83:6800/3071793735 conn(0x7f94f40718d0 legacy=0x7f94f4073d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 >> v1:172.21.15.83:6790/0 conn(0x7f950c1014e0 legacy=0x7f950c1003f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 shutdown_connections 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 >> 172.21.15.83:0/3992153229 conn(0x7f950c075290 msgr2=0x7f950c064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:03.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 shutdown_connections 2024-09-10T19:08:03.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 -- 172.21.15.83:0/3992153229 wait complete. 2024-09-10T19:08:03.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:03.580+0000 7f9511ca4640 1 librados: shutdown 2024-09-10T19:08:03.642 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:03.674 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:03.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:03.811 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: cluster 2024-09-10T19:08:02.147495+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:08:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: cluster 2024-09-10T19:08:02.150763+0000 osd.3 (osd.3) 439 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:08:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: audit 2024-09-10T19:08:02.736647+0000 mon.c (mon.2) 842 : audit [INF] from='client.? v1:172.21.15.83:0/3992153229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: audit 2024-09-10T19:08:02.737140+0000 mon.a (mon.0) 3651 : audit [INF] from='client.49853 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: cluster 2024-09-10T19:08:02.887198+0000 osd.7 (osd.7) 416 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:08:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:03 smithi165 bash[22591]: cluster 2024-09-10T19:08:02.890583+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:08:03.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:03.927 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: cluster 2024-09-10T19:08:02.147495+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: cluster 2024-09-10T19:08:02.150763+0000 osd.3 (osd.3) 439 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: audit 2024-09-10T19:08:02.736647+0000 mon.c (mon.2) 842 : audit [INF] from='client.? v1:172.21.15.83:0/3992153229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: audit 2024-09-10T19:08:02.737140+0000 mon.a (mon.0) 3651 : audit [INF] from='client.49853 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: cluster 2024-09-10T19:08:02.887198+0000 osd.7 (osd.7) 416 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[18629]: cluster 2024-09-10T19:08:02.890583+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: cluster 2024-09-10T19:08:02.147495+0000 osd.3 (osd.3) 438 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: cluster 2024-09-10T19:08:02.150763+0000 osd.3 (osd.3) 439 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: audit 2024-09-10T19:08:02.736647+0000 mon.c (mon.2) 842 : audit [INF] from='client.? v1:172.21.15.83:0/3992153229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: audit 2024-09-10T19:08:02.737140+0000 mon.a (mon.0) 3651 : audit [INF] from='client.49853 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:03.928 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: cluster 2024-09-10T19:08:02.887198+0000 osd.7 (osd.7) 416 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:08:03.929 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:03 smithi083 bash[32381]: cluster 2024-09-10T19:08:02.890583+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:08:03.948 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:03.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:04.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:04.086 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:08:04.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:08:04.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:04.223 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:04.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:04.360 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:04.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:04.491 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:04.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.563090+0000 mon.a (mon.0) 3652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:08:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: audit 2024-09-10T19:08:03.577555+0000 mon.a (mon.0) 3653 : audit [INF] from='client.49853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.577700+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e813: 8 total, 7 up, 8 in 2024-09-10T19:08:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.688037+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:08:04.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.690204+0000 osd.6 (osd.6) 380 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:08:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.856108+0000 osd.7 (osd.7) 418 : cluster [DBG] 3.f scrub starts 2024-09-10T19:08:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:03.861431+0000 osd.7 (osd.7) 419 : cluster [DBG] 3.f scrub ok 2024-09-10T19:08:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.161741+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:08:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.166151+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:08:04.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:04 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.372029+0000 mgr.y (mgr.24526) 735 : cluster [DBG] pgmap v1261: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:04.626 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:04.730 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:04.756 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:04.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.563090+0000 mon.a (mon.0) 3652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: audit 2024-09-10T19:08:03.577555+0000 mon.a (mon.0) 3653 : audit [INF] from='client.49853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.577700+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e813: 8 total, 7 up, 8 in 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.688037+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.690204+0000 osd.6 (osd.6) 380 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.856108+0000 osd.7 (osd.7) 418 : cluster [DBG] 3.f scrub starts 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:03.861431+0000 osd.7 (osd.7) 419 : cluster [DBG] 3.f scrub ok 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.161741+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.166151+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:08:04.872 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.372029+0000 mgr.y (mgr.24526) 735 : cluster [DBG] pgmap v1261: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.563090+0000 mon.a (mon.0) 3652 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: audit 2024-09-10T19:08:03.577555+0000 mon.a (mon.0) 3653 : audit [INF] from='client.49853 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.577700+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e813: 8 total, 7 up, 8 in 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.688037+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.690204+0000 osd.6 (osd.6) 380 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.856108+0000 osd.7 (osd.7) 418 : cluster [DBG] 3.f scrub starts 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:03.861431+0000 osd.7 (osd.7) 419 : cluster [DBG] 3.f scrub ok 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.161741+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.166151+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:08:04.873 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:04 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.372029+0000 mgr.y (mgr.24526) 735 : cluster [DBG] pgmap v1261: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:04.893 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:04.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:04.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:05.027 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:05.130 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:05.162 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:05.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:05.301 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:05.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:05.441 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:05.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:05.578 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:05.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:05.710 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:05.813 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:05.843 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:05.845 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.674071+0000 osd.6 (osd.6) 381 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.677454+0000 osd.6 (osd.6) 382 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.852772+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[18629]: cluster 2024-09-10T19:08:04.857659+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[18629]: audit 2024-09-10T19:08:05.425192+0000 mon.c (mon.2) 843 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.674071+0000 osd.6 (osd.6) 381 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.677454+0000 osd.6 (osd.6) 382 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.852772+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[32381]: cluster 2024-09-10T19:08:04.857659+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:08:05.846 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:05 smithi083 bash[32381]: audit 2024-09-10T19:08:05.425192+0000 mon.c (mon.2) 843 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:05 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.674071+0000 osd.6 (osd.6) 381 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:08:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:05 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.677454+0000 osd.6 (osd.6) 382 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:08:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:05 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.852772+0000 osd.7 (osd.7) 420 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:08:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:05 smithi165 bash[22591]: cluster 2024-09-10T19:08:04.857659+0000 osd.7 (osd.7) 421 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:08:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:05 smithi165 bash[22591]: audit 2024-09-10T19:08:05.425192+0000 mon.c (mon.2) 843 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:05.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:05.981 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:05.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:06.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:06.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:08:06.166 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:06.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:06.302 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:06.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:06.440 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:06.543 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:06.576 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:06.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:06.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: cluster 2024-09-10T19:08:05.491425+0000 osd.4 (osd.4) 414 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:08:06.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: cluster 2024-09-10T19:08:05.495302+0000 osd.4 (osd.4) 415 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: cluster 2024-09-10T19:08:05.882811+0000 osd.7 (osd.7) 422 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: audit 2024-09-10T19:08:05.885165+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: audit 2024-09-10T19:08:05.886626+0000 mon.c (mon.2) 845 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: cluster 2024-09-10T19:08:05.886727+0000 osd.7 (osd.7) 423 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: audit 2024-09-10T19:08:05.898300+0000 mon.a (mon.0) 3655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:06.693 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:06 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.374029+0000 mgr.y (mgr.24526) 736 : cluster [DBG] pgmap v1262: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:06.710 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:06.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: cluster 2024-09-10T19:08:05.491425+0000 osd.4 (osd.4) 414 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: cluster 2024-09-10T19:08:05.495302+0000 osd.4 (osd.4) 415 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: cluster 2024-09-10T19:08:05.882811+0000 osd.7 (osd.7) 422 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: audit 2024-09-10T19:08:05.885165+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: audit 2024-09-10T19:08:05.886626+0000 mon.c (mon.2) 845 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: cluster 2024-09-10T19:08:05.886727+0000 osd.7 (osd.7) 423 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: audit 2024-09-10T19:08:05.898300+0000 mon.a (mon.0) 3655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.374029+0000 mgr.y (mgr.24526) 736 : cluster [DBG] pgmap v1262: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:06.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: cluster 2024-09-10T19:08:05.491425+0000 osd.4 (osd.4) 414 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: cluster 2024-09-10T19:08:05.495302+0000 osd.4 (osd.4) 415 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: cluster 2024-09-10T19:08:05.882811+0000 osd.7 (osd.7) 422 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: audit 2024-09-10T19:08:05.885165+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: audit 2024-09-10T19:08:05.886626+0000 mon.c (mon.2) 845 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: cluster 2024-09-10T19:08:05.886727+0000 osd.7 (osd.7) 423 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: audit 2024-09-10T19:08:05.898300+0000 mon.a (mon.0) 3655 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:06 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.374029+0000 mgr.y (mgr.24526) 736 : cluster [DBG] pgmap v1262: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:06.814 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:06.817 INFO:teuthology.orchestra.run.smithi083.stderr:Exporting 6.19 info 6.19( empty local-lis/les=75/76 n=0 ec=75/75 lis/c=75/75 les/c/f=76/76/0 sis=75) 2024-09-10T19:08:06.817 INFO:teuthology.orchestra.run.smithi083.stderr:Export successful 2024-09-10T19:08:06.846 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:06.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:06.977 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:06.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:06.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:07.026 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:07.080 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:07.114 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:07.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:08:07.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:07.254 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:07.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:07.393 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:07.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:07.531 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:07.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.485031+0000 osd.2 (osd.2) 323 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.485398+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.684874+0000 osd.6 (osd.6) 383 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.688638+0000 osd.6 (osd.6) 384 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.770332+0000 osd.5 (osd.5) 424 : cluster [DBG] 12.14 deep-scrub starts 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.774187+0000 osd.5 (osd.5) 425 : cluster [DBG] 12.14 deep-scrub ok 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.909217+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e814: 8 total, 7 up, 8 in 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.918006+0000 osd.7 (osd.7) 424 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:08:07.654 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[32381]: cluster 2024-09-10T19:08:06.920803+0000 osd.7 (osd.7) 425 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.485031+0000 osd.2 (osd.2) 323 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.485398+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.684874+0000 osd.6 (osd.6) 383 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.688638+0000 osd.6 (osd.6) 384 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.770332+0000 osd.5 (osd.5) 424 : cluster [DBG] 12.14 deep-scrub starts 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.774187+0000 osd.5 (osd.5) 425 : cluster [DBG] 12.14 deep-scrub ok 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.909217+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e814: 8 total, 7 up, 8 in 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.918006+0000 osd.7 (osd.7) 424 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:08:07.655 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:07 smithi083 bash[18629]: cluster 2024-09-10T19:08:06.920803+0000 osd.7 (osd.7) 425 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:08:07.673 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:07.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.485031+0000 osd.2 (osd.2) 323 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.485398+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.684874+0000 osd.6 (osd.6) 383 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.688638+0000 osd.6 (osd.6) 384 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.770332+0000 osd.5 (osd.5) 424 : cluster [DBG] 12.14 deep-scrub starts 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.774187+0000 osd.5 (osd.5) 425 : cluster [DBG] 12.14 deep-scrub ok 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.909217+0000 mon.a (mon.0) 3656 : cluster [DBG] osdmap e814: 8 total, 7 up, 8 in 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.918006+0000 osd.7 (osd.7) 424 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:08:07.791 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:07 smithi165 bash[22591]: cluster 2024-09-10T19:08:06.920803+0000 osd.7 (osd.7) 425 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:08:07.810 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:07.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:07.948 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:07.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:08.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:08.087 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:08.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:08.222 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:08.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:08.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:08.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:08.496 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:08.500 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 6.19 2024-09-10T19:08:08.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:08.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:08.632 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:08.660 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:08:08.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:08.768 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:07.116623+0000 osd.2 (osd.2) 325 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:07.119949+0000 osd.2 (osd.2) 326 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:07.691826+0000 osd.6 (osd.6) 385 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:07.694464+0000 osd.6 (osd.6) 386 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.119042+0000 osd.3 (osd.3) 440 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.122929+0000 osd.3 (osd.3) 441 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.375073+0000 mgr.y (mgr.24526) 737 : cluster [DBG] pgmap v1264: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:07.116623+0000 osd.2 (osd.2) 325 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:07.119949+0000 osd.2 (osd.2) 326 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:07.691826+0000 osd.6 (osd.6) 385 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:08:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:07.694464+0000 osd.6 (osd.6) 386 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:08:08.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.119042+0000 osd.3 (osd.3) 440 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:08:08.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.122929+0000 osd.3 (osd.3) 441 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:08:08.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:08 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.375073+0000 mgr.y (mgr.24526) 737 : cluster [DBG] pgmap v1264: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:07.116623+0000 osd.2 (osd.2) 325 : cluster [DBG] 241.0 scrub starts 2024-09-10T19:08:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:07.119949+0000 osd.2 (osd.2) 326 : cluster [DBG] 241.0 scrub ok 2024-09-10T19:08:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:07.691826+0000 osd.6 (osd.6) 385 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:08:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:07.694464+0000 osd.6 (osd.6) 386 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:08:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.119042+0000 osd.3 (osd.3) 440 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:08:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.122929+0000 osd.3 (osd.3) 441 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:08:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:08 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.375073+0000 mgr.y (mgr.24526) 737 : cluster [DBG] pgmap v1264: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:08.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:08.905 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:08.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:09.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:09.041 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:09.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.178 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:09.282 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.317 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:09.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.444 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:09.548 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:09.581 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:09.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.717 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:09.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.111614+0000 osd.2 (osd.2) 327 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:08:09.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.115102+0000 osd.2 (osd.2) 328 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:08:09.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: audit 2024-09-10T19:08:08.432945+0000 mgr.y (mgr.24526) 738 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.526168+0000 osd.4 (osd.4) 416 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:08:09.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.111614+0000 osd.2 (osd.2) 327 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.115102+0000 osd.2 (osd.2) 328 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: audit 2024-09-10T19:08:08.432945+0000 mgr.y (mgr.24526) 738 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.526168+0000 osd.4 (osd.4) 416 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:08.529145+0000 osd.4 (osd.4) 417 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: audit 2024-09-10T19:08:08.812492+0000 mon.c (mon.2) 846 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: audit 2024-09-10T19:08:08.813167+0000 mon.a (mon.0) 3657 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:09.126056+0000 osd.3 (osd.3) 442 : cluster [DBG] 3.b scrub starts 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[18629]: cluster 2024-09-10T19:08:09.131156+0000 osd.3 (osd.3) 443 : cluster [DBG] 3.b scrub ok 2024-09-10T19:08:09.835 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.111614+0000 osd.2 (osd.2) 327 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.115102+0000 osd.2 (osd.2) 328 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: audit 2024-09-10T19:08:08.432945+0000 mgr.y (mgr.24526) 738 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.526168+0000 osd.4 (osd.4) 416 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:08.529145+0000 osd.4 (osd.4) 417 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: audit 2024-09-10T19:08:08.812492+0000 mon.c (mon.2) 846 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: audit 2024-09-10T19:08:08.813167+0000 mon.a (mon.0) 3657 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:09.126056+0000 osd.3 (osd.3) 442 : cluster [DBG] 3.b scrub starts 2024-09-10T19:08:09.836 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:09 smithi083 bash[32381]: cluster 2024-09-10T19:08:09.131156+0000 osd.3 (osd.3) 443 : cluster [DBG] 3.b scrub ok 2024-09-10T19:08:09.857 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:08.529145+0000 osd.4 (osd.4) 417 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:08:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: audit 2024-09-10T19:08:08.812492+0000 mon.c (mon.2) 846 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: audit 2024-09-10T19:08:08.813167+0000 mon.a (mon.0) 3657 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:09.126056+0000 osd.3 (osd.3) 442 : cluster [DBG] 3.b scrub starts 2024-09-10T19:08:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:09 smithi165 bash[22591]: cluster 2024-09-10T19:08:09.131156+0000 osd.3 (osd.3) 443 : cluster [DBG] 3.b scrub ok 2024-09-10T19:08:09.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:09.994 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:09.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:10.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:10.123 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:10.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:10.260 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:10.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:10.394 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:10.497 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:10.533 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:10.635 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:10.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:10.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:10.813 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: audit 2024-09-10T19:08:09.596382+0000 mon.a (mon.0) 3658 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:10.813 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: cluster 2024-09-10T19:08:09.596508+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e815: 8 total, 7 up, 8 in 2024-09-10T19:08:10.813 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: audit 2024-09-10T19:08:09.600242+0000 mon.c (mon.2) 847 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:10.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: audit 2024-09-10T19:08:09.606855+0000 mon.a (mon.0) 3660 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:10.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: cluster 2024-09-10T19:08:09.922498+0000 osd.7 (osd.7) 426 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:08:10.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: cluster 2024-09-10T19:08:09.925574+0000 osd.7 (osd.7) 427 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:08:10.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:10 smithi165 bash[22591]: cluster 2024-09-10T19:08:10.376098+0000 mgr.y (mgr.24526) 739 : cluster [DBG] pgmap v1266: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:10.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:10.939 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:10.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:10.944 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:08:10.945 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: audit 2024-09-10T19:08:09.596382+0000 mon.a (mon.0) 3658 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: cluster 2024-09-10T19:08:09.596508+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e815: 8 total, 7 up, 8 in 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: audit 2024-09-10T19:08:09.600242+0000 mon.c (mon.2) 847 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: audit 2024-09-10T19:08:09.606855+0000 mon.a (mon.0) 3660 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: cluster 2024-09-10T19:08:09.922498+0000 osd.7 (osd.7) 426 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: cluster 2024-09-10T19:08:09.925574+0000 osd.7 (osd.7) 427 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:08:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[18629]: cluster 2024-09-10T19:08:10.376098+0000 mgr.y (mgr.24526) 739 : cluster [DBG] pgmap v1266: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: audit 2024-09-10T19:08:09.596382+0000 mon.a (mon.0) 3658 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: cluster 2024-09-10T19:08:09.596508+0000 mon.a (mon.0) 3659 : cluster [DBG] osdmap e815: 8 total, 7 up, 8 in 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: audit 2024-09-10T19:08:09.600242+0000 mon.c (mon.2) 847 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: audit 2024-09-10T19:08:09.606855+0000 mon.a (mon.0) 3660 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: cluster 2024-09-10T19:08:09.922498+0000 osd.7 (osd.7) 426 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: cluster 2024-09-10T19:08:09.925574+0000 osd.7 (osd.7) 427 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:08:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:10 smithi083 bash[32381]: cluster 2024-09-10T19:08:10.376098+0000 mgr.y (mgr.24526) 739 : cluster [DBG] pgmap v1266: 214 pgs: 58 active+undersized, 18 active+undersized+degraded, 138 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 752 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:11.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:11.077 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:11.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:11.216 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:11.319 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.355 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:11.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.491 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:11.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:11.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:11.629 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:11.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:08:11.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:11.766 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: cluster 2024-09-10T19:08:10.601741+0000 mon.a (mon.0) 3661 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: audit 2024-09-10T19:08:10.610077+0000 mon.a (mon.0) 3662 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: cluster 2024-09-10T19:08:10.610306+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e816: 8 total, 7 up, 8 in 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: audit 2024-09-10T19:08:10.661590+0000 mon.c (mon.2) 848 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: audit 2024-09-10T19:08:10.662124+0000 mon.a (mon.0) 3664 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: audit 2024-09-10T19:08:10.662957+0000 mon.c (mon.2) 849 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: audit 2024-09-10T19:08:10.663352+0000 mon.a (mon.0) 3665 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: cluster 2024-09-10T19:08:11.163307+0000 osd.3 (osd.3) 444 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[18629]: cluster 2024-09-10T19:08:11.167019+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: cluster 2024-09-10T19:08:10.601741+0000 mon.a (mon.0) 3661 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: audit 2024-09-10T19:08:10.610077+0000 mon.a (mon.0) 3662 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: cluster 2024-09-10T19:08:10.610306+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e816: 8 total, 7 up, 8 in 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: audit 2024-09-10T19:08:10.661590+0000 mon.c (mon.2) 848 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: audit 2024-09-10T19:08:10.662124+0000 mon.a (mon.0) 3664 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: audit 2024-09-10T19:08:10.662957+0000 mon.c (mon.2) 849 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: audit 2024-09-10T19:08:10.663352+0000 mon.a (mon.0) 3665 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: cluster 2024-09-10T19:08:11.163307+0000 osd.3 (osd.3) 444 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:08:11.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:11 smithi083 bash[32381]: cluster 2024-09-10T19:08:11.167019+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:08:11.841 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:11.869 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.899 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:11.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:12.003 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:12.024 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:12.037 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: cluster 2024-09-10T19:08:10.601741+0000 mon.a (mon.0) 3661 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: audit 2024-09-10T19:08:10.610077+0000 mon.a (mon.0) 3662 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]': finished 2024-09-10T19:08:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: cluster 2024-09-10T19:08:10.610306+0000 mon.a (mon.0) 3663 : cluster [DBG] osdmap e816: 8 total, 7 up, 8 in 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: audit 2024-09-10T19:08:10.661590+0000 mon.c (mon.2) 848 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: audit 2024-09-10T19:08:10.662124+0000 mon.a (mon.0) 3664 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: audit 2024-09-10T19:08:10.662957+0000 mon.c (mon.2) 849 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: audit 2024-09-10T19:08:10.663352+0000 mon.a (mon.0) 3665 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-144"}]: dispatch 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: cluster 2024-09-10T19:08:11.163307+0000 osd.3 (osd.3) 444 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:08:12.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:11 smithi165 bash[22591]: cluster 2024-09-10T19:08:11.167019+0000 osd.3 (osd.3) 445 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:08:12.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:12.175 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:12.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:12.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:12.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:12.418 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:12.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:12.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:12.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:12.593 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[32381]: cluster 2024-09-10T19:08:11.631665+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e817: 8 total, 7 up, 8 in 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[32381]: cluster 2024-09-10T19:08:12.150398+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[32381]: cluster 2024-09-10T19:08:12.153334+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[32381]: cluster 2024-09-10T19:08:12.377153+0000 mgr.y (mgr.24526) 740 : cluster [DBG] pgmap v1269: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[18629]: cluster 2024-09-10T19:08:11.631665+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e817: 8 total, 7 up, 8 in 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[18629]: cluster 2024-09-10T19:08:12.150398+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:08:12.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[18629]: cluster 2024-09-10T19:08:12.153334+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:08:12.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[18629]: cluster 2024-09-10T19:08:12.377153+0000 mgr.y (mgr.24526) 740 : cluster [DBG] pgmap v1269: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:12.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:12.732 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:12.836 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:12.869 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:12.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:12.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:12.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:12 smithi165 bash[22591]: cluster 2024-09-10T19:08:11.631665+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e817: 8 total, 7 up, 8 in 2024-09-10T19:08:12.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:12 smithi165 bash[22591]: cluster 2024-09-10T19:08:12.150398+0000 osd.0 (osd.0) 393 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:08:12.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:12 smithi165 bash[22591]: cluster 2024-09-10T19:08:12.153334+0000 osd.0 (osd.0) 394 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:08:12.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:12 smithi165 bash[22591]: cluster 2024-09-10T19:08:12.377153+0000 mgr.y (mgr.24526) 740 : cluster [DBG] pgmap v1269: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:13.005 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:13.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:12] "GET /metrics HTTP/1.1" 200 229161 "" "Prometheus/2.43.0" 2024-09-10T19:08:13.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:13.141 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:13.244 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:13.280 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:13.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:13.418 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:13.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:13.555 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:13.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.685 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:13.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:13.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:13.819 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:13.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:13.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: cluster 2024-09-10T19:08:12.641955+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e818: 8 total, 7 up, 8 in 2024-09-10T19:08:13.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: audit 2024-09-10T19:08:12.643741+0000 mon.c (mon.2) 850 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: audit 2024-09-10T19:08:12.644892+0000 mon.a (mon.0) 3668 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.936 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: cluster 2024-09-10T19:08:13.104554+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: cluster 2024-09-10T19:08:13.109131+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: audit 2024-09-10T19:08:13.644684+0000 mon.a (mon.0) 3669 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[18629]: cluster 2024-09-10T19:08:13.644961+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e819: 8 total, 7 up, 8 in 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: cluster 2024-09-10T19:08:12.641955+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e818: 8 total, 7 up, 8 in 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: audit 2024-09-10T19:08:12.643741+0000 mon.c (mon.2) 850 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: audit 2024-09-10T19:08:12.644892+0000 mon.a (mon.0) 3668 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: cluster 2024-09-10T19:08:13.104554+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: cluster 2024-09-10T19:08:13.109131+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: audit 2024-09-10T19:08:13.644684+0000 mon.a (mon.0) 3669 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.937 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:13 smithi083 bash[32381]: cluster 2024-09-10T19:08:13.644961+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e819: 8 total, 7 up, 8 in 2024-09-10T19:08:13.957 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:13.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:14.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: cluster 2024-09-10T19:08:12.641955+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e818: 8 total, 7 up, 8 in 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: audit 2024-09-10T19:08:12.643741+0000 mon.c (mon.2) 850 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: audit 2024-09-10T19:08:12.644892+0000 mon.a (mon.0) 3668 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: cluster 2024-09-10T19:08:13.104554+0000 osd.0 (osd.0) 395 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: cluster 2024-09-10T19:08:13.109131+0000 osd.0 (osd.0) 396 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: audit 2024-09-10T19:08:13.644684+0000 mon.a (mon.0) 3669 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:14.074 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:13 smithi165 bash[22591]: cluster 2024-09-10T19:08:13.644961+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e819: 8 total, 7 up, 8 in 2024-09-10T19:08:14.092 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:14.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:14.232 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:14.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:14.370 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:14.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:14.509 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:14.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:14.647 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:14.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:13.676294+0000 mon.c (mon.2) 851 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:13.676639+0000 mon.a (mon.0) 3671 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.133322+0000 osd.0 (osd.0) 397 : cluster [DBG] 241.1 scrub starts 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.139398+0000 osd.0 (osd.0) 398 : cluster [DBG] 241.1 scrub ok 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.211071+0000 osd.3 (osd.3) 446 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.216281+0000 osd.3 (osd.3) 447 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.378177+0000 mgr.y (mgr.24526) 741 : cluster [DBG] pgmap v1272: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:14.583984+0000 mon.a (mon.0) 3672 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:14.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:14.586545+0000 mon.c (mon.2) 852 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:14.649041+0000 mon.a (mon.0) 3673 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.649281+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e820: 8 total, 7 up, 8 in 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:14.650548+0000 mon.c (mon.2) 853 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[18629]: audit 2024-09-10T19:08:14.651050+0000 mon.a (mon.0) 3675 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:13.676294+0000 mon.c (mon.2) 851 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:13.676639+0000 mon.a (mon.0) 3671 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.133322+0000 osd.0 (osd.0) 397 : cluster [DBG] 241.1 scrub starts 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.139398+0000 osd.0 (osd.0) 398 : cluster [DBG] 241.1 scrub ok 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.211071+0000 osd.3 (osd.3) 446 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:08:14.765 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.216281+0000 osd.3 (osd.3) 447 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.378177+0000 mgr.y (mgr.24526) 741 : cluster [DBG] pgmap v1272: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:14.583984+0000 mon.a (mon.0) 3672 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:14.586545+0000 mon.c (mon.2) 852 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:14.649041+0000 mon.a (mon.0) 3673 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.649281+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e820: 8 total, 7 up, 8 in 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:14.650548+0000 mon.c (mon.2) 853 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:14.766 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:14 smithi083 bash[32381]: audit 2024-09-10T19:08:14.651050+0000 mon.a (mon.0) 3675 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:14.782 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:14.886 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:14.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:14.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:15.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.046 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:13.676294+0000 mon.c (mon.2) 851 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:13.676639+0000 mon.a (mon.0) 3671 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.133322+0000 osd.0 (osd.0) 397 : cluster [DBG] 241.1 scrub starts 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.139398+0000 osd.0 (osd.0) 398 : cluster [DBG] 241.1 scrub ok 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.211071+0000 osd.3 (osd.3) 446 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.216281+0000 osd.3 (osd.3) 447 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.378177+0000 mgr.y (mgr.24526) 741 : cluster [DBG] pgmap v1272: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 84/705 objects degraded (11.915%) 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:14.583984+0000 mon.a (mon.0) 3672 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:14.586545+0000 mon.c (mon.2) 852 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:14.649041+0000 mon.a (mon.0) 3673 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.649281+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e820: 8 total, 7 up, 8 in 2024-09-10T19:08:15.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:14.650548+0000 mon.c (mon.2) 853 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:15.114 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:14 smithi165 bash[22591]: audit 2024-09-10T19:08:14.651050+0000 mon.a (mon.0) 3675 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:15.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.177 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:15.227 INFO:teuthology.orchestra.run.smithi083.stdout: marking collection for removal 2024-09-10T19:08:15.227 INFO:teuthology.orchestra.run.smithi083.stdout:setting '_remove' omap key 2024-09-10T19:08:15.227 INFO:teuthology.orchestra.run.smithi083.stdout:finish_remove_pgs 6.19_head removing 6.19 2024-09-10T19:08:15.227 INFO:teuthology.orchestra.run.smithi083.stdout:Remove successful 2024-09-10T19:08:15.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:15.314 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:15.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:15.454 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:15.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:15.591 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:15.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:15.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:15.860 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.751843+0000 osd.5 (osd.5) 426 : cluster [DBG] 1.a scrub starts 2024-09-10T19:08:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: cluster 2024-09-10T19:08:14.758251+0000 osd.5 (osd.5) 427 : cluster [DBG] 1.a scrub ok 2024-09-10T19:08:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: cluster 2024-09-10T19:08:15.140260+0000 osd.0 (osd.0) 399 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:08:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: cluster 2024-09-10T19:08:15.144602+0000 osd.0 (osd.0) 400 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:08:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: audit 2024-09-10T19:08:15.651568+0000 mon.a (mon.0) 3676 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: cluster 2024-09-10T19:08:15.651684+0000 mon.a (mon.0) 3677 : cluster [DBG] osdmap e821: 8 total, 7 up, 8 in 2024-09-10T19:08:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: audit 2024-09-10T19:08:15.653128+0000 mon.c (mon.2) 854 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:15 smithi165 bash[22591]: audit 2024-09-10T19:08:15.653704+0000 mon.a (mon.0) 3678 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:15.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.751843+0000 osd.5 (osd.5) 426 : cluster [DBG] 1.a scrub starts 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: cluster 2024-09-10T19:08:14.758251+0000 osd.5 (osd.5) 427 : cluster [DBG] 1.a scrub ok 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: cluster 2024-09-10T19:08:15.140260+0000 osd.0 (osd.0) 399 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: cluster 2024-09-10T19:08:15.144602+0000 osd.0 (osd.0) 400 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: audit 2024-09-10T19:08:15.651568+0000 mon.a (mon.0) 3676 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: cluster 2024-09-10T19:08:15.651684+0000 mon.a (mon.0) 3677 : cluster [DBG] osdmap e821: 8 total, 7 up, 8 in 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: audit 2024-09-10T19:08:15.653128+0000 mon.c (mon.2) 854 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.964 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[32381]: audit 2024-09-10T19:08:15.653704+0000 mon.a (mon.0) 3678 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.751843+0000 osd.5 (osd.5) 426 : cluster [DBG] 1.a scrub starts 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: cluster 2024-09-10T19:08:14.758251+0000 osd.5 (osd.5) 427 : cluster [DBG] 1.a scrub ok 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: cluster 2024-09-10T19:08:15.140260+0000 osd.0 (osd.0) 399 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: cluster 2024-09-10T19:08:15.144602+0000 osd.0 (osd.0) 400 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: audit 2024-09-10T19:08:15.651568+0000 mon.a (mon.0) 3676 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-77828-111", "overlaypool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: cluster 2024-09-10T19:08:15.651684+0000 mon.a (mon.0) 3677 : cluster [DBG] osdmap e821: 8 total, 7 up, 8 in 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: audit 2024-09-10T19:08:15.653128+0000 mon.c (mon.2) 854 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:15 smithi083 bash[18629]: audit 2024-09-10T19:08:15.653704+0000 mon.a (mon.0) 3678 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]: dispatch 2024-09-10T19:08:15.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:15.997 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:16.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:16.133 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:16.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:16.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:16.375 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:16.411 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:16.454 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:16.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:16.548 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:16.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:16.683 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:16.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:16.800 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:16 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.379856+0000 mgr.y (mgr.24526) 742 : cluster [DBG] pgmap v1275: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:16.800 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:16 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.639948+0000 mon.a (mon.0) 3679 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:16.800 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:16 smithi165 bash[22591]: audit 2024-09-10T19:08:16.650253+0000 mon.a (mon.0) 3680 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]': finished 2024-09-10T19:08:16.800 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:16 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.650437+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e822: 8 total, 7 up, 8 in 2024-09-10T19:08:16.821 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:16.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:16.955 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:16.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.379856+0000 mgr.y (mgr.24526) 742 : cluster [DBG] pgmap v1275: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.639948+0000 mon.a (mon.0) 3679 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[18629]: audit 2024-09-10T19:08:16.650253+0000 mon.a (mon.0) 3680 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]': finished 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.650437+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e822: 8 total, 7 up, 8 in 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.379856+0000 mgr.y (mgr.24526) 742 : cluster [DBG] pgmap v1275: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.639948+0000 mon.a (mon.0) 3679 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[32381]: audit 2024-09-10T19:08:16.650253+0000 mon.a (mon.0) 3680 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi083-77828-146", "mode": "writeback"}]': finished 2024-09-10T19:08:17.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:16 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.650437+0000 mon.a (mon.0) 3681 : cluster [DBG] osdmap e822: 8 total, 7 up, 8 in 2024-09-10T19:08:17.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:17.094 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:17.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:17.232 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:17.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:17.367 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:17.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:17.501 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.19.1 2024-09-10T19:08:17.511 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:17.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:17.644 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:17.662 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:08:17.747 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.667220+0000 osd.6 (osd.6) 387 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.667315+0000 mon.a (mon.0) 3682 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: cluster 2024-09-10T19:08:16.673791+0000 osd.6 (osd.6) 388 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: audit 2024-09-10T19:08:16.709164+0000 mon.c (mon.2) 855 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: audit 2024-09-10T19:08:16.709677+0000 mon.a (mon.0) 3683 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.294505+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.297916+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.667220+0000 osd.6 (osd.6) 387 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:08:17.761 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.667315+0000 mon.a (mon.0) 3682 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:17.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: cluster 2024-09-10T19:08:16.673791+0000 osd.6 (osd.6) 388 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:08:17.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: audit 2024-09-10T19:08:16.709164+0000 mon.c (mon.2) 855 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: audit 2024-09-10T19:08:16.709677+0000 mon.a (mon.0) 3683 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.294505+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:08:17.762 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:17 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.297916+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:08:17.783 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:17.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:17.914 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:17.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.667220+0000 osd.6 (osd.6) 387 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:08:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.667315+0000 mon.a (mon.0) 3682 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:18.002 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: cluster 2024-09-10T19:08:16.673791+0000 osd.6 (osd.6) 388 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:08:18.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: audit 2024-09-10T19:08:16.709164+0000 mon.c (mon.2) 855 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:18.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: audit 2024-09-10T19:08:16.709677+0000 mon.a (mon.0) 3683 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:18.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.294505+0000 osd.3 (osd.3) 448 : cluster [DBG] 6.11 deep-scrub starts 2024-09-10T19:08:18.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:17 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.297916+0000 osd.3 (osd.3) 449 : cluster [DBG] 6.11 deep-scrub ok 2024-09-10T19:08:18.017 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:18.047 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:18.150 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:18.182 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:18.285 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:18.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:18.419 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:18.452 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:18.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:18.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:18.594 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:18.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.093241+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.e scrub starts 2024-09-10T19:08:18.697 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:18.720 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:18.824 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:18.860 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:18.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.093241+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.e scrub starts 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.097237+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.e scrub ok 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.641699+0000 osd.6 (osd.6) 389 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.644912+0000 osd.6 (osd.6) 390 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: audit 2024-09-10T19:08:17.689272+0000 mon.a (mon.0) 3684 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:17.689365+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e823: 8 total, 7 up, 8 in 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: audit 2024-09-10T19:08:17.693850+0000 mon.c (mon.2) 856 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: audit 2024-09-10T19:08:17.694211+0000 mon.a (mon.0) 3686 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:18.023258+0000 osd.7 (osd.7) 428 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:08:18.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:18.027210+0000 osd.7 (osd.7) 429 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:08:18.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:18 smithi165 bash[22591]: cluster 2024-09-10T19:08:18.380915+0000 mgr.y (mgr.24526) 743 : cluster [DBG] pgmap v1278: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:18.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:18.997 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.097237+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.e scrub ok 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.641699+0000 osd.6 (osd.6) 389 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.644912+0000 osd.6 (osd.6) 390 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: audit 2024-09-10T19:08:17.689272+0000 mon.a (mon.0) 3684 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:17.689365+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e823: 8 total, 7 up, 8 in 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: audit 2024-09-10T19:08:17.693850+0000 mon.c (mon.2) 856 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: audit 2024-09-10T19:08:17.694211+0000 mon.a (mon.0) 3686 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:18.023258+0000 osd.7 (osd.7) 428 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:18.027210+0000 osd.7 (osd.7) 429 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[18629]: cluster 2024-09-10T19:08:18.380915+0000 mgr.y (mgr.24526) 743 : cluster [DBG] pgmap v1278: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.093241+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.e scrub starts 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.097237+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.e scrub ok 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.641699+0000 osd.6 (osd.6) 389 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.644912+0000 osd.6 (osd.6) 390 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: audit 2024-09-10T19:08:17.689272+0000 mon.a (mon.0) 3684 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:17.689365+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e823: 8 total, 7 up, 8 in 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: audit 2024-09-10T19:08:17.693850+0000 mon.c (mon.2) 856 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: audit 2024-09-10T19:08:17.694211+0000 mon.a (mon.0) 3686 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:18.023258+0000 osd.7 (osd.7) 428 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:08:19.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:18.027210+0000 osd.7 (osd.7) 429 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:08:19.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:18 smithi083 bash[32381]: cluster 2024-09-10T19:08:18.380915+0000 mgr.y (mgr.24526) 743 : cluster [DBG] pgmap v1278: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:19.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:19.135 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:19.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:19.272 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:19.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:19.406 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:19.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:19.544 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:19.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:19.680 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:19.784 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:18.444284+0000 mgr.y (mgr.24526) 744 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:18.699942+0000 mon.a (mon.0) 3687 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: cluster 2024-09-10T19:08:18.700028+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e824: 8 total, 7 up, 8 in 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:18.706310+0000 mon.c (mon.2) 857 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:18.706706+0000 mon.a (mon.0) 3689 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: cluster 2024-09-10T19:08:19.227400+0000 osd.0 (osd.0) 401 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: cluster 2024-09-10T19:08:19.231836+0000 osd.0 (osd.0) 402 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:08:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: cluster 2024-09-10T19:08:19.384386+0000 mon.a (mon.0) 3690 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:19.394308+0000 mon.a (mon.0) 3691 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: cluster 2024-09-10T19:08:19.395277+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e825: 8 total, 7 up, 8 in 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:19.396143+0000 mon.c (mon.2) 858 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[18629]: audit 2024-09-10T19:08:19.397046+0000 mon.a (mon.0) 3693 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:18.444284+0000 mgr.y (mgr.24526) 744 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:18.699942+0000 mon.a (mon.0) 3687 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: cluster 2024-09-10T19:08:18.700028+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e824: 8 total, 7 up, 8 in 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:18.706310+0000 mon.c (mon.2) 857 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:18.706706+0000 mon.a (mon.0) 3689 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: cluster 2024-09-10T19:08:19.227400+0000 osd.0 (osd.0) 401 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:08:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: cluster 2024-09-10T19:08:19.231836+0000 osd.0 (osd.0) 402 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:08:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: cluster 2024-09-10T19:08:19.384386+0000 mon.a (mon.0) 3690 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:19.394308+0000 mon.a (mon.0) 3691 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: cluster 2024-09-10T19:08:19.395277+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e825: 8 total, 7 up, 8 in 2024-09-10T19:08:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:19.396143+0000 mon.c (mon.2) 858 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:19 smithi083 bash[32381]: audit 2024-09-10T19:08:19.397046+0000 mon.a (mon.0) 3693 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.821 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:19.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:18.444284+0000 mgr.y (mgr.24526) 744 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:18.699942+0000 mon.a (mon.0) 3687 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: cluster 2024-09-10T19:08:18.700028+0000 mon.a (mon.0) 3688 : cluster [DBG] osdmap e824: 8 total, 7 up, 8 in 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:18.706310+0000 mon.c (mon.2) 857 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:18.706706+0000 mon.a (mon.0) 3689 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: cluster 2024-09-10T19:08:19.227400+0000 osd.0 (osd.0) 401 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: cluster 2024-09-10T19:08:19.231836+0000 osd.0 (osd.0) 402 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:08:19.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: cluster 2024-09-10T19:08:19.384386+0000 mon.a (mon.0) 3690 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:19.394308+0000 mon.a (mon.0) 3691 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: cluster 2024-09-10T19:08:19.395277+0000 mon.a (mon.0) 3692 : cluster [DBG] osdmap e825: 8 total, 7 up, 8 in 2024-09-10T19:08:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:19.396143+0000 mon.c (mon.2) 858 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.939 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:19 smithi165 bash[22591]: audit 2024-09-10T19:08:19.397046+0000 mon.a (mon.0) 3693 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:19.958 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:19.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:20.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:20.089 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:20.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:20.196 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:08:20.196 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:08:20.221 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:20.324 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:20.350 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:20.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- 172.21.15.83:0/3036708500 >> v1:172.21.15.83:6789/0 conn(0x7f0d5c108ef0 legacy=0x7f0d5c10b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- 172.21.15.83:0/3036708500 shutdown_connections 2024-09-10T19:08:20.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- 172.21.15.83:0/3036708500 >> 172.21.15.83:0/3036708500 conn(0x7f0d5c0f81a0 msgr2=0x7f0d5c0fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:20.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- 172.21.15.83:0/3036708500 shutdown_connections 2024-09-10T19:08:20.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- 172.21.15.83:0/3036708500 wait complete. 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 librados: starting msgr at 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 librados: starting objecter 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 Processor -- start 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- start start 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 librados: setting wanted keys 2024-09-10T19:08:20.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 librados: calling monclient init 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d5c1122d0 con 0x7f0d5c108ef0 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d5c11d3f0 con 0x7f0d5c105360 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d60e71640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0d5c11e5d0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d5a59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0d5c1025a0 0x7f0d5c111760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50348/0 (socket says 172.21.15.83:50348) 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d5a59c640 1 -- 172.21.15.83:0/618647290 learned_addr learned my addr 172.21.15.83:0/618647290 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter ms_handle_connect 0x7f0d5c1025a0 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter ms_handle_connect 0x7f0d5c108ef0 2024-09-10T19:08:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184320363 0 0) 0x7f0d5c11e5d0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d44003720 con 0x7f0d5c1025a0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter ms_handle_connect 0x7f0d5c105360 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438450271 0 0) 0x7f0d5c1122d0 con 0x7f0d5c108ef0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d5c11e5d0 con 0x7f0d5c108ef0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396729116 0 0) 0x7f0d5c11d3f0 con 0x7f0d5c105360 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d5c1122d0 con 0x7f0d5c105360 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 282451388 0 0) 0x7f0d44003720 con 0x7f0d5c1025a0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d5c11d3f0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.436+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0d54001c00 con 0x7f0d5c1025a0 2024-09-10T19:08:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869792689 0 0) 0x7f0d5c1122d0 con 0x7f0d5c105360 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d44003720 con 0x7f0d5c105360 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0d50001a10 con 0x7f0d5c105360 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1236738273 0 0) 0x7f0d5c11d3f0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 >> v1:172.21.15.165:6789/0 conn(0x7f0d5c105360 legacy=0x7f0d5c117590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 >> v1:172.21.15.83:6789/0 conn(0x7f0d5c108ef0 legacy=0x7f0d5c11acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c11f7f0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c11c440 con 0x7f0d5c1025a0 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d60e71640 10 client.49859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c11c890 con 0x7f0d5c1025a0 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d60e71640 1 librados: init done 2024-09-10T19:08:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f0d540026e0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f0d54005de0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0d3c004ff0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.440+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f0d54002280 con 0x7f0d5c1025a0 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(826..826 src has 750..826) v4 ==== 8557+0+0 (unknown 3293915329 0 0) 0x7f0d5408ab60 con 0x7f0d5c1025a0 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 osd_map(826..826 src has 750..826) v4 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 3 client.49859.objecter handle_osd_map got epochs [826,826] > 0 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 3 client.49859.objecter handle_osd_map decoding full epoch 826 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 20 client.49859.objecter dump_active .. 0 homeless 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v826) v1 ==== 54+0+16 (unknown 2469407434 0 2420000192) 0x7f0d5408af10 con 0x7f0d5c1025a0 2024-09-10T19:08:20.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 10 client.49859.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4050171745 >> v1:172.21.15.83:6790/0 conn(0x7f2e2c104260 legacy=0x7f2e2c106640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 1 2024-09-10T19:08:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4050171745 shutdown_connections 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d440714e0 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 1 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 0 1 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 4 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 10 client.49859.objecter _send_command 1 2024-09-10T19:08:20.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c004a60 con 0x7f0d3c004610 2024-09-10T19:08:20.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4050171745 >> 172.21.15.83:0/4050171745 conn(0x7f2e2c0f70b0 msgr2=0x7f2e2c0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:20.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4050171745 shutdown_connections 2024-09-10T19:08:20.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4050171745 wait complete. 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c004610 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c004a60 con 0x7f0d3c004610 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 10 client.49859.objecter _finish_command 1 = system:0 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 4 2024-09-10T19:08:20.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 0 1 2024-09-10T19:08:20.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 librados: starting msgr at 2024-09-10T19:08:20.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 librados: starting objecter 2024-09-10T19:08:20.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 Processor -- start 2024-09-10T19:08:20.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- start start 2024-09-10T19:08:20.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 librados: setting wanted keys 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 librados: calling monclient init 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e2c119e20 con 0x7f2e2c104260 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e2c11afe0 con 0x7f2e2c107dc0 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e31f9c640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2e2c11b1d0 con 0x7f2e2c1014f0 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e2affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2e2c104260 0x7f2e2c0ff250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56054/0 (socket says 172.21.15.83:56054) 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e2affd640 1 -- 172.21.15.83:0/4277497782 learned_addr learned my addr 172.21.15.83:0/4277497782 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e13fff640 10 client.?.objecter ms_handle_connect 0x7f2e2c104260 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.444+0000 7f2e13fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 10 client.?.objecter ms_handle_connect 0x7f2e2c1014f0 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203021728 0 0) 0x7f2e2c119e20 con 0x7f2e2c104260 2024-09-10T19:08:20.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e14003720 con 0x7f2e2c104260 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 10 client.?.objecter ms_handle_connect 0x7f2e2c107dc0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629389072 0 0) 0x7f2e2c11b1d0 con 0x7f2e2c1014f0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2c119e20 con 0x7f2e2c1014f0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047342666 0 0) 0x7f2e2c11afe0 con 0x7f2e2c107dc0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2c11b1d0 con 0x7f2e2c107dc0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580415773 0 0) 0x7f2e14003720 con 0x7f2e2c104260 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e2c11afe0 con 0x7f2e2c104260 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280105116 0 0) 0x7f2e2c119e20 con 0x7f2e2c1014f0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e14003720 con 0x7f2e2c1014f0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2e20002c50 con 0x7f2e2c104260 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2e1c001820 con 0x7f2e2c1014f0 2024-09-10T19:08:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1602981710 0 0) 0x7f2e2c11afe0 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 >> v1:172.21.15.83:6790/0 conn(0x7f2e2c1014f0 legacy=0x7f2e2c0feb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 >> v1:172.21.15.165:6789/0 conn(0x7f2e2c107dc0 legacy=0x7f2e2c118520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c11b3c0 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c11b670 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e31f9c640 10 client.25779.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c11bac0 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e31f9c640 1 librados: init done 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2e200021b0 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2e20003660 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18005010 con 0x7f2e2c104260 2024-09-10T19:08:20.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.448+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2e200017f0 con 0x7f2e2c104260 2024-09-10T19:08:20.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:20.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(826..826 src has 750..826) v4 ==== 8557+0+0 (unknown 3293915329 0 0) 0x7f2e2008a910 con 0x7f2e2c104260 2024-09-10T19:08:20.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 10 client.25779.objecter ms_dispatch 0x7f2e2c000b90 osd_map(826..826 src has 750..826) v4 2024-09-10T19:08:20.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 3 client.25779.objecter handle_osd_map got epochs [826,826] > 0 2024-09-10T19:08:20.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 3 client.25779.objecter handle_osd_map decoding full epoch 826 2024-09-10T19:08:20.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 20 client.25779.objecter dump_active .. 0 homeless 2024-09-10T19:08:20.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.452+0000 7f2e13fff640 10 client.25779.objecter ms_handle_connect 0x7f2e14071310 2024-09-10T19:08:20.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.456+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2e20056ca0 con 0x7f2e2c104260 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 10 client.49859.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 2 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 2 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c005290 osd=0 3 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 0 2 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 4 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 10 client.49859.objecter _send_command 2 2024-09-10T19:08:20.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c004e40 con 0x7f0d3c004610 2024-09-10T19:08:20.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c004e40 con 0x7f0d3c004610 2024-09-10T19:08:20.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d4affd640 10 client.49859.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c005290 osd=0 4 2024-09-10T19:08:20.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 0 2 2024-09-10T19:08:20.470 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 10 client.49859.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 3 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 20 client.49859.objecter _get_session osd=-1 returning homeless 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 10 client.49859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=827}) v3 -- 0x7f0d3c003fa0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.464+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d3c004380 con 0x7f0d5c1025a0 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_get_version_reply(handle=1 version=826) v2 ==== 24+0+0 (unknown 1124000704 0 0) 0x7f0d54056ef0 con 0x7f0d5c1025a0 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d58d99640 10 client.49859.objecter _check_command_map_dne tid 3 current 826 map_dne_bound 826 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d58d99640 10 client.49859.objecter _finish_command 3 = osd:6 osd down 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d58d99640 15 client.49859.objecter _session_command_op_remove -1 3 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 10 client.49859.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 4 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 4 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 2 4 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 4 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 10 client.49859.objecter _send_command 4 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6809/386094220 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c010be0 con 0x7f0d3c002150 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c002150 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c010be0 con 0x7f0d3c002150 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 4: 0 ) v1 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 10 client.49859.objecter _finish_command 4 = system:0 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 4 2024-09-10T19:08:20.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.468+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 2 4 2024-09-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 10 client.49859.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 5 2024-09-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 5 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c001fe0 osd=2 3 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 2 5 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 4 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 10 client.49859.objecter _send_command 5 2024-09-10T19:08:20.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c001b30 con 0x7f0d3c002150 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c001b30 con 0x7f0d3c002150 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 10 client.49859.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c001fe0 osd=2 4 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 2 5 2024-09-10T19:08:20.489 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 10 client.49859.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 6 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 6 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 3 6 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 4 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 10 client.49859.objecter _send_command 6 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6813/1279688021 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c003d20 con 0x7f0d3c010d50 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.484+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c010d50 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.488+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c003d20 con 0x7f0d3c010d50 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.488+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 6: 0 ) v1 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.488+0000 7f0d4affd640 10 client.49859.objecter _finish_command 6 = system:0 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.488+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 4 2024-09-10T19:08:20.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.488+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 3 6 2024-09-10T19:08:20.492 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 10 client.49859.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 7 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 7 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c0035e0 osd=3 3 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 3 7 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 4 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 10 client.49859.objecter _send_command 7 2024-09-10T19:08:20.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.512+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c012070 con 0x7f0d3c010d50 2024-09-10T19:08:20.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c012070 con 0x7f0d3c010d50 2024-09-10T19:08:20.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 10 client.49859.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c0035e0 osd=3 4 2024-09-10T19:08:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 3 7 2024-09-10T19:08:20.520 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:08:20.520 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 10 client.49859.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 8 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 8 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 4 8 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 4 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 10 client.49859.objecter _send_command 8 2024-09-10T19:08:20.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6800/3225515265 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c01a6a0 con 0x7f0d3c016ba0 2024-09-10T19:08:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.516+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c016ba0 2024-09-10T19:08:20.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.520+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c01a6a0 con 0x7f0d3c016ba0 2024-09-10T19:08:20.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.520+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 8: 0 ) v1 2024-09-10T19:08:20.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.520+0000 7f0d4affd640 10 client.49859.objecter _finish_command 8 = system:0 2024-09-10T19:08:20.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.520+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 4 2024-09-10T19:08:20.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.520+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 4 8 2024-09-10T19:08:20.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 10 client.49859.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 9 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 9 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c003310 osd=4 3 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 4 9 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 4 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 10 client.49859.objecter _send_command 9 2024-09-10T19:08:20.544 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.540+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c011af0 con 0x7f0d3c016ba0 2024-09-10T19:08:20.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c011af0 con 0x7f0d3c016ba0 2024-09-10T19:08:20.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 10 client.49859.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c003310 osd=4 4 2024-09-10T19:08:20.548 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:08:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 4 9 2024-09-10T19:08:20.548 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 10 client.49859.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 10 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 10 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 5 10 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 4 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 10 client.49859.objecter _send_command 10 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6804/571173777 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c011a20 con 0x7f0d3c013120 2024-09-10T19:08:20.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c013120 2024-09-10T19:08:20.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c011a20 con 0x7f0d3c013120 2024-09-10T19:08:20.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 10: 0 ) v1 2024-09-10T19:08:20.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 10 client.49859.objecter _finish_command 10 = system:0 2024-09-10T19:08:20.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 4 2024-09-10T19:08:20.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.544+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 5 10 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 10 client.49859.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 11 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 11 2024-09-10T19:08:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c0118b0 osd=5 3 2024-09-10T19:08:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 5 11 2024-09-10T19:08:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 4 2024-09-10T19:08:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 10 client.49859.objecter _send_command 11 2024-09-10T19:08:20.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.564+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c01f0c0 con 0x7f0d3c013120 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d5405a020 con 0x7f0d3c013120 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 10 client.49859.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c0118b0 osd=5 4 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 5 11 2024-09-10T19:08:20.572 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 10 client.49859.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 12 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 12 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 6 12 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 4 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 10 client.49859.objecter _send_command 12 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6808/1317922936 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c023570 con 0x7f0d3c01f8b0 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.568+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c01f8b0 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.572+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c023570 con 0x7f0d3c01f8b0 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.572+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 12: 0 ) v1 2024-09-10T19:08:20.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.572+0000 7f0d4affd640 10 client.49859.objecter _finish_command 12 = system:0 2024-09-10T19:08:20.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.572+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 4 2024-09-10T19:08:20.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.572+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 6 12 2024-09-10T19:08:20.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 10 client.49859.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 13 2024-09-10T19:08:20.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 13 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c01f230 osd=6 3 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 6 13 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 4 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 10 client.49859.objecter _send_command 13 2024-09-10T19:08:20.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.592+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c023570 con 0x7f0d3c01f8b0 2024-09-10T19:08:20.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c023570 con 0x7f0d3c01f8b0 2024-09-10T19:08:20.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 10 client.49859.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c01f230 osd=6 4 2024-09-10T19:08:20.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 6 13 2024-09-10T19:08:20.599 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:08:20.599 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 10 client.49859.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 14 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 14 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 7 14 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 4 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 10 client.49859.objecter _send_command 14 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6812/1610151650 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d3c024720 con 0x7f0d3c025320 2024-09-10T19:08:20.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 10 client.49859.objecter ms_handle_connect 0x7f0d3c025320 2024-09-10T19:08:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0d3c024720 con 0x7f0d3c025320 2024-09-10T19:08:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 14: 0 ) v1 2024-09-10T19:08:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 10 client.49859.objecter _finish_command 14 = system:0 2024-09-10T19:08:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 4 2024-09-10T19:08:20.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.596+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 7 14 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 10 client.49859.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign -1 15 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 20 client.49859.objecter _get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_remove -1 15 2024-09-10T19:08:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 20 client.49859.objecter get_session s=0x7f0d3c023570 osd=7 3 2024-09-10T19:08:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 15 client.49859.objecter _session_command_op_assign 7 15 2024-09-10T19:08:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 4 2024-09-10T19:08:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 10 client.49859.objecter _send_command 15 2024-09-10T19:08:20.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.616+0000 7f0d60e71640 1 -- 172.21.15.83:0/618647290 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0d3c024720 con 0x7f0d3c025320 2024-09-10T19:08:20.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.620+0000 7f0d4affd640 1 -- 172.21.15.83:0/618647290 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f0d3c024720 con 0x7f0d3c025320 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.620+0000 7f0d4affd640 10 client.49859.objecter ms_dispatch 0x7f0d5c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.620+0000 7f0d4affd640 10 client.49859.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.620+0000 7f0d4affd640 20 client.49859.objecter put_session s=0x7f0d3c023570 osd=7 4 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.620+0000 7f0d4affd640 15 client.49859.objecter _session_command_op_remove 7 15 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:08:20.625 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:08:20.628 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:20.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:20.660+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2e180052c0 con 0x7f2e2c104260 2024-09-10T19:08:20.731 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:20.764 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:20.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.046107+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:08:20.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.049809+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:08:20.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.381791+0000 mgr.y (mgr.24526) 745 : cluster [DBG] pgmap v1281: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/717 objects degraded (11.855%) 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: audit 2024-09-10T19:08:20.395091+0000 mon.a (mon.0) 3694 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.395202+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e826: 8 total, 7 up, 8 in 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: audit 2024-09-10T19:08:20.396125+0000 mon.c (mon.2) 859 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: audit 2024-09-10T19:08:20.397200+0000 mon.a (mon.0) 3696 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: audit 2024-09-10T19:08:20.446061+0000 mon.c (mon.2) 860 : audit [DBG] from='client.? v1:172.21.15.83:0/618647290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:20.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:20 smithi165 bash[22591]: audit 2024-09-10T19:08:20.666268+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:08:20.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:20.903 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:20.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:20.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:08:21.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.046107+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.049809+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.381791+0000 mgr.y (mgr.24526) 745 : cluster [DBG] pgmap v1281: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/717 objects degraded (11.855%) 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: audit 2024-09-10T19:08:20.395091+0000 mon.a (mon.0) 3694 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.395202+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e826: 8 total, 7 up, 8 in 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: audit 2024-09-10T19:08:20.396125+0000 mon.c (mon.2) 859 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: audit 2024-09-10T19:08:20.397200+0000 mon.a (mon.0) 3696 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: audit 2024-09-10T19:08:20.446061+0000 mon.c (mon.2) 860 : audit [DBG] from='client.? v1:172.21.15.83:0/618647290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[18629]: audit 2024-09-10T19:08:20.666268+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.046107+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.049809+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.381791+0000 mgr.y (mgr.24526) 745 : cluster [DBG] pgmap v1281: 214 pgs: 55 active+undersized, 18 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/717 objects degraded (11.855%) 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: audit 2024-09-10T19:08:20.395091+0000 mon.a (mon.0) 3694 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.395202+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e826: 8 total, 7 up, 8 in 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: audit 2024-09-10T19:08:20.396125+0000 mon.c (mon.2) 859 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: audit 2024-09-10T19:08:20.397200+0000 mon.a (mon.0) 3696 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: audit 2024-09-10T19:08:20.446061+0000 mon.c (mon.2) 860 : audit [DBG] from='client.? v1:172.21.15.83:0/618647290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:20 smithi083 bash[32381]: audit 2024-09-10T19:08:20.666268+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:08:21.047 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:21.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:21.174 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:21.277 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:21.310 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:21.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.396+0000 7f2e13fff640 1 -- 172.21.15.83:0/4277497782 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v827) v1 ==== 87+0+0 (unknown 848594926 0 0) 0x7f2e20056640 con 0x7f2e2c104260 2024-09-10T19:08:21.401 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is set 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 10 librados: watch_flush enter 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 10 librados: watch_flush exit 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 20 client.25779.objecter shutdown clearing up homeless session... 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 10 client.25779.objecter successfully canceled tick 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2e14071310 legacy=0x7f2e140737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:21.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 >> v1:172.21.15.83:6789/0 conn(0x7f2e2c104260 legacy=0x7f2e2c0ff250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:21.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 shutdown_connections 2024-09-10T19:08:21.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.400+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 >> 172.21.15.83:0/4277497782 conn(0x7f2e2c0f70b0 msgr2=0x7f2e2c10aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:21.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.404+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 shutdown_connections 2024-09-10T19:08:21.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.404+0000 7f2e31f9c640 1 -- 172.21.15.83:0/4277497782 wait complete. 2024-09-10T19:08:21.407 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:21.404+0000 7f2e31f9c640 1 librados: shutdown 2024-09-10T19:08:21.415 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:21.437 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:21.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:21.553 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:21.572 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:21.676 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:21.708 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:21.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.074063+0000 osd.2 (osd.2) 331 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.083400+0000 osd.2 (osd.2) 332 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.609475+0000 osd.6 (osd.6) 391 : cluster [DBG] 12.f scrub starts 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:20.617019+0000 osd.6 (osd.6) 392 : cluster [DBG] 12.f scrub ok 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.177546+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.180808+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.389679+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: audit 2024-09-10T19:08:21.399060+0000 mon.a (mon.0) 3699 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: audit 2024-09-10T19:08:21.399202+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:08:21.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:21 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.399244+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e827: 8 total, 7 up, 8 in 2024-09-10T19:08:21.844 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:21.948 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.074063+0000 osd.2 (osd.2) 331 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.083400+0000 osd.2 (osd.2) 332 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.609475+0000 osd.6 (osd.6) 391 : cluster [DBG] 12.f scrub starts 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:20.617019+0000 osd.6 (osd.6) 392 : cluster [DBG] 12.f scrub ok 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.177546+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.180808+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.389679+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: audit 2024-09-10T19:08:21.399060+0000 mon.a (mon.0) 3699 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: audit 2024-09-10T19:08:21.399202+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:08:21.961 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.399244+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e827: 8 total, 7 up, 8 in 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.074063+0000 osd.2 (osd.2) 331 : cluster [DBG] 241.4 scrub starts 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.083400+0000 osd.2 (osd.2) 332 : cluster [DBG] 241.4 scrub ok 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.609475+0000 osd.6 (osd.6) 391 : cluster [DBG] 12.f scrub starts 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:20.617019+0000 osd.6 (osd.6) 392 : cluster [DBG] 12.f scrub ok 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.177546+0000 osd.0 (osd.0) 403 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.180808+0000 osd.0 (osd.0) 404 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.389679+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: audit 2024-09-10T19:08:21.399060+0000 mon.a (mon.0) 3699 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: audit 2024-09-10T19:08:21.399202+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? v1:172.21.15.83:0/4277497782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:08:21.962 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:21 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.399244+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e827: 8 total, 7 up, 8 in 2024-09-10T19:08:21.983 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:21.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:22.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:22.126 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:22.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:22.263 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:22.367 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.401 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:22.505 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.538 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:22.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:22.675 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:22.778 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:22.805 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:22.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.946 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:22.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:23.028 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:22] "GET /metrics HTTP/1.1" 200 229155 "" "Prometheus/2.43.0" 2024-09-10T19:08:23.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.065557+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:08:23.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[18629]: cluster 2024-09-10T19:08:21.069207+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:08:23.028 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[18629]: cluster 2024-09-10T19:08:22.382923+0000 mgr.y (mgr.24526) 746 : cluster [DBG] pgmap v1284: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:23.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.065557+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:08:23.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[32381]: cluster 2024-09-10T19:08:21.069207+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:08:23.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:22 smithi083 bash[32381]: cluster 2024-09-10T19:08:22.382923+0000 mgr.y (mgr.24526) 746 : cluster [DBG] pgmap v1284: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:23.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:23.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:22 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.065557+0000 osd.2 (osd.2) 333 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:08:23.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:22 smithi165 bash[22591]: cluster 2024-09-10T19:08:21.069207+0000 osd.2 (osd.2) 334 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:08:23.063 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:22 smithi165 bash[22591]: cluster 2024-09-10T19:08:22.382923+0000 mgr.y (mgr.24526) 746 : cluster [DBG] pgmap v1284: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:23.083 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:23.187 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:23.219 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:08:23.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:08:23.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:23.358 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:23.461 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:23.495 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:23.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:23.633 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:23.737 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:23.772 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:23.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:23.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:24.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:24.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:23 smithi165 bash[22591]: cluster 2024-09-10T19:08:22.708489+0000 mon.a (mon.0) 3702 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:24.028 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:23 smithi165 bash[22591]: cluster 2024-09-10T19:08:22.708563+0000 mon.a (mon.0) 3703 : cluster [WRN] Health check update: Degraded data redundancy: 86/729 objects degraded (11.797%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:24.040 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:23 smithi083 bash[18629]: cluster 2024-09-10T19:08:22.708489+0000 mon.a (mon.0) 3702 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:23 smithi083 bash[18629]: cluster 2024-09-10T19:08:22.708563+0000 mon.a (mon.0) 3703 : cluster [WRN] Health check update: Degraded data redundancy: 86/729 objects degraded (11.797%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:23 smithi083 bash[32381]: cluster 2024-09-10T19:08:22.708489+0000 mon.a (mon.0) 3702 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:24.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:23 smithi083 bash[32381]: cluster 2024-09-10T19:08:22.708563+0000 mon.a (mon.0) 3703 : cluster [WRN] Health check update: Degraded data redundancy: 86/729 objects degraded (11.797%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:24.047 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:24.151 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:24.183 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:24.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:24.320 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:24.422 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:24.453 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:24.556 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:24.590 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:24.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:24.726 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:24.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:24.864 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:24.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:24.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:24.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:24 smithi165 bash[22591]: cluster 2024-09-10T19:08:24.384043+0000 mgr.y (mgr.24526) 747 : cluster [DBG] pgmap v1285: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:24.984 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:24 smithi083 bash[18629]: cluster 2024-09-10T19:08:24.384043+0000 mgr.y (mgr.24526) 747 : cluster [DBG] pgmap v1285: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:24.984 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:24 smithi083 bash[32381]: cluster 2024-09-10T19:08:24.384043+0000 mgr.y (mgr.24526) 747 : cluster [DBG] pgmap v1285: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 86/729 objects degraded (11.797%) 2024-09-10T19:08:25.000 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:25.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.137 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:25.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.271 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:25.302 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:08:25.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:25.406 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:25.415 INFO:teuthology.orchestra.run.smithi083.stdout:get_pg_num_history pg_num_history pg_num_history(e789 pg_nums {1={508=11},2={69=32},3={71=32},4={73=32},5={73=3},6={75=32},12={124=32},241={653=8},247={698=32},248={706=32},249={716=32},250={724=32},251={734=32},252={742=32},253={755=32},254={763=32},255={771=32},256={779=32}} deleted_pools 705,247,715,248,723,249,733,250,741,251,754,252,762,253,770,254,778,255) 2024-09-10T19:08:25.415 INFO:teuthology.orchestra.run.smithi083.stdout:Importing pgid 6.19 2024-09-10T19:08:25.415 INFO:teuthology.orchestra.run.smithi083.stdout:write_pg epoch 773 info 6.19( empty local-lis/les=75/76 n=0 ec=75/75 lis/c=75/75 les/c/f=76/76/0 sis=75) 2024-09-10T19:08:25.415 INFO:teuthology.orchestra.run.smithi083.stdout:Import successful 2024-09-10T19:08:25.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:25.544 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:25.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:25.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:08:25.683 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:25.766 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:25.786 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.817 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:25.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.957 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:25.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:25.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:26.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:26.088 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:26.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:26.225 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:26.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:26.364 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:26.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:26.499 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:26.603 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:26.639 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:26.743 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:26.757 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:26 smithi083 bash[18629]: cluster 2024-09-10T19:08:26.386459+0000 mgr.y (mgr.24526) 748 : cluster [DBG] pgmap v1286: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:26.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:26 smithi083 bash[32381]: cluster 2024-09-10T19:08:26.386459+0000 mgr.y (mgr.24526) 748 : cluster [DBG] pgmap v1286: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:26.778 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:26.862 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/exp.6.19.1 2024-09-10T19:08:26.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:26 smithi165 bash[22591]: cluster 2024-09-10T19:08:26.386459+0000 mgr.y (mgr.24526) 748 : cluster [DBG] pgmap v1286: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:26.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:26.915 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:26.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:27.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:27.046 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:27.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:27.184 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:27.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:27.322 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:27.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:27.460 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:27.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:27.598 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:27.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:27.741 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:27.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:27.870 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:27.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:27.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:28.004 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:28.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:28.134 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:28.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:28.270 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:28.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:28.407 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:28.510 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:28.543 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:28 smithi165 bash[22591]: cluster 2024-09-10T19:08:28.387133+0000 mgr.y (mgr.24526) 749 : cluster [DBG] pgmap v1287: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:28.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:28.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:28.682 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:28.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:28.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:28 smithi083 bash[18629]: cluster 2024-09-10T19:08:28.387133+0000 mgr.y (mgr.24526) 749 : cluster [DBG] pgmap v1287: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:28.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:28 smithi083 bash[32381]: cluster 2024-09-10T19:08:28.387133+0000 mgr.y (mgr.24526) 749 : cluster [DBG] pgmap v1287: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:28.820 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:28.924 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:28.959 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:28.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:29.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.096 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:29.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.234 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:29.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:29.369 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:29.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:29.501 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:08:29.501 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:08:29.501 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:08:29.501 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:08:29.503 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:29.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:29 smithi083 bash[18629]: audit 2024-09-10T19:08:28.453038+0000 mgr.y (mgr.24526) 750 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:29 smithi083 bash[18629]: cluster 2024-09-10T19:08:29.388339+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:29 smithi083 bash[32381]: audit 2024-09-10T19:08:28.453038+0000 mgr.y (mgr.24526) 750 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:29 smithi083 bash[32381]: cluster 2024-09-10T19:08:29.388339+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:29.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:29 smithi165 bash[22591]: audit 2024-09-10T19:08:28.453038+0000 mgr.y (mgr.24526) 750 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:29 smithi165 bash[22591]: cluster 2024-09-10T19:08:29.388339+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: Degraded data redundancy: 86/723 objects degraded (11.895%), 19 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:29.636 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:29.739 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.773 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:29.876 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:29.901 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:29.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:30.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:30.037 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:30.141 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:30.176 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:30.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:08:30.280 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:30.315 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:30.392 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:30.392 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:30.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:30.446 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:30.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:30.582 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:30.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:30.724 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:30.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:30.840 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[18629]: audit 2024-09-10T19:08:29.583944+0000 mon.a (mon.0) 3705 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:30.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[18629]: audit 2024-09-10T19:08:29.588203+0000 mon.c (mon.2) 861 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:30.841 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[18629]: cluster 2024-09-10T19:08:30.388173+0000 mgr.y (mgr.24526) 751 : cluster [DBG] pgmap v1288: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:30.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[32381]: audit 2024-09-10T19:08:29.583944+0000 mon.a (mon.0) 3705 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:30.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[32381]: audit 2024-09-10T19:08:29.588203+0000 mon.c (mon.2) 861 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:30.841 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:30 smithi083 bash[32381]: cluster 2024-09-10T19:08:30.388173+0000 mgr.y (mgr.24526) 751 : cluster [DBG] pgmap v1288: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:30.861 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:30 smithi165 bash[22591]: audit 2024-09-10T19:08:29.583944+0000 mon.a (mon.0) 3705 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:30 smithi165 bash[22591]: audit 2024-09-10T19:08:29.588203+0000 mon.c (mon.2) 861 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:30 smithi165 bash[22591]: cluster 2024-09-10T19:08:30.388173+0000 mgr.y (mgr.24526) 751 : cluster [DBG] pgmap v1288: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:30.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:30.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:30.999 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:31.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:31.138 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:31.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:31.276 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:31.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:31.412 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:31.516 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:31.550 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:31.591 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:31.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:31.687 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:31.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:31 smithi083 bash[32381]: audit 2024-09-10T19:08:31.429400+0000 mon.c (mon.2) 862 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:31 smithi083 bash[32381]: audit 2024-09-10T19:08:31.430079+0000 mon.a (mon.0) 3706 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:31 smithi083 bash[18629]: audit 2024-09-10T19:08:31.429400+0000 mon.c (mon.2) 862 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:31 smithi083 bash[18629]: audit 2024-09-10T19:08:31.430079+0000 mon.a (mon.0) 3706 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.826 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:31 smithi165 bash[22591]: audit 2024-09-10T19:08:31.429400+0000 mon.c (mon.2) 862 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:31 smithi165 bash[22591]: audit 2024-09-10T19:08:31.430079+0000 mon.a (mon.0) 3706 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:31.877 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:08:31.877 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:08:31.878 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:08:31.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:08:31.930 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:31.963 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:31.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:32.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:32.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:32.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:32.239 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:32.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:32.375 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:32.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:32.505 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:32.608 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:32.633 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:32.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:32.771 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:32.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:32] "GET /metrics HTTP/1.1" 200 229158 "" "Prometheus/2.43.0" 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18629]: audit 2024-09-10T19:08:31.604437+0000 mon.a (mon.0) 3707 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18629]: cluster 2024-09-10T19:08:31.604532+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e828: 8 total, 7 up, 8 in 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18629]: audit 2024-09-10T19:08:31.605959+0000 mon.c (mon.2) 863 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18629]: audit 2024-09-10T19:08:31.606737+0000 mon.a (mon.0) 3709 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[18629]: cluster 2024-09-10T19:08:32.389316+0000 mgr.y (mgr.24526) 752 : cluster [DBG] pgmap v1290: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[32381]: audit 2024-09-10T19:08:31.604437+0000 mon.a (mon.0) 3707 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[32381]: cluster 2024-09-10T19:08:31.604532+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e828: 8 total, 7 up, 8 in 2024-09-10T19:08:32.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[32381]: audit 2024-09-10T19:08:31.605959+0000 mon.c (mon.2) 863 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[32381]: audit 2024-09-10T19:08:31.606737+0000 mon.a (mon.0) 3709 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:32 smithi083 bash[32381]: cluster 2024-09-10T19:08:32.389316+0000 mgr.y (mgr.24526) 752 : cluster [DBG] pgmap v1290: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:32.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:32.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:32 smithi165 bash[22591]: audit 2024-09-10T19:08:31.604437+0000 mon.a (mon.0) 3707 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:32.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:32 smithi165 bash[22591]: cluster 2024-09-10T19:08:31.604532+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e828: 8 total, 7 up, 8 in 2024-09-10T19:08:32.888 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:32 smithi165 bash[22591]: audit 2024-09-10T19:08:31.605959+0000 mon.c (mon.2) 863 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.889 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:32 smithi165 bash[22591]: audit 2024-09-10T19:08:31.606737+0000 mon.a (mon.0) 3709 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:32.889 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:32 smithi165 bash[22591]: cluster 2024-09-10T19:08:32.389316+0000 mgr.y (mgr.24526) 752 : cluster [DBG] pgmap v1290: 214 pgs: 54 active+undersized, 19 active+undersized+degraded, 141 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s; 86/723 objects degraded (11.895%) 2024-09-10T19:08:32.907 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:32.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:33.010 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:33.045 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:33.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:33.182 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:33.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:33.320 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:33.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:33.459 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:33.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:33.599 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:33.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:33.735 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: audit 2024-09-10T19:08:32.608016+0000 mon.a (mon.0) 3710 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: cluster 2024-09-10T19:08:32.608192+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e829: 8 total, 7 up, 8 in 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: audit 2024-09-10T19:08:32.646950+0000 mon.c (mon.2) 864 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: audit 2024-09-10T19:08:32.647434+0000 mon.a (mon.0) 3712 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: audit 2024-09-10T19:08:32.648167+0000 mon.c (mon.2) 865 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[18629]: audit 2024-09-10T19:08:32.648543+0000 mon.a (mon.0) 3713 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: audit 2024-09-10T19:08:32.608016+0000 mon.a (mon.0) 3710 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: cluster 2024-09-10T19:08:32.608192+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e829: 8 total, 7 up, 8 in 2024-09-10T19:08:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: audit 2024-09-10T19:08:32.646950+0000 mon.c (mon.2) 864 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: audit 2024-09-10T19:08:32.647434+0000 mon.a (mon.0) 3712 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: audit 2024-09-10T19:08:32.648167+0000 mon.c (mon.2) 865 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:33 smithi083 bash[32381]: audit 2024-09-10T19:08:32.648543+0000 mon.a (mon.0) 3713 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:33.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: audit 2024-09-10T19:08:32.608016+0000 mon.a (mon.0) 3710 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]': finished 2024-09-10T19:08:33.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: cluster 2024-09-10T19:08:32.608192+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e829: 8 total, 7 up, 8 in 2024-09-10T19:08:33.848 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: audit 2024-09-10T19:08:32.646950+0000 mon.c (mon.2) 864 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: audit 2024-09-10T19:08:32.647434+0000 mon.a (mon.0) 3712 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:33.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: audit 2024-09-10T19:08:32.648167+0000 mon.c (mon.2) 865 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:33 smithi165 bash[22591]: audit 2024-09-10T19:08:32.648543+0000 mon.a (mon.0) 3713 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-146"}]: dispatch 2024-09-10T19:08:33.866 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:33.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:33.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:34.002 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:34.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:34.137 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:34.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:34.275 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:34.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:34.421 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:34.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:34.555 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:34.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:34.691 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[32381]: cluster 2024-09-10T19:08:33.631361+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e830: 8 total, 7 up, 8 in 2024-09-10T19:08:34.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[32381]: cluster 2024-09-10T19:08:34.390548+0000 mgr.y (mgr.24526) 753 : cluster [DBG] pgmap v1293: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[32381]: cluster 2024-09-10T19:08:34.407018+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e831: 8 total, 7 up, 8 in 2024-09-10T19:08:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[32381]: audit 2024-09-10T19:08:34.409892+0000 mon.c (mon.2) 866 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.779 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[32381]: audit 2024-09-10T19:08:34.416208+0000 mon.a (mon.0) 3716 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.779 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[18629]: cluster 2024-09-10T19:08:33.631361+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e830: 8 total, 7 up, 8 in 2024-09-10T19:08:34.779 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[18629]: cluster 2024-09-10T19:08:34.390548+0000 mgr.y (mgr.24526) 753 : cluster [DBG] pgmap v1293: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:34.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[18629]: cluster 2024-09-10T19:08:34.407018+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e831: 8 total, 7 up, 8 in 2024-09-10T19:08:34.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[18629]: audit 2024-09-10T19:08:34.409892+0000 mon.c (mon.2) 866 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.780 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:34 smithi083 bash[18629]: audit 2024-09-10T19:08:34.416208+0000 mon.a (mon.0) 3716 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:34.831 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:34.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:34 smithi165 bash[22591]: cluster 2024-09-10T19:08:33.631361+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e830: 8 total, 7 up, 8 in 2024-09-10T19:08:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:34 smithi165 bash[22591]: cluster 2024-09-10T19:08:34.390548+0000 mgr.y (mgr.24526) 753 : cluster [DBG] pgmap v1293: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 85/717 objects degraded (11.855%) 2024-09-10T19:08:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:34 smithi165 bash[22591]: cluster 2024-09-10T19:08:34.407018+0000 mon.a (mon.0) 3715 : cluster [DBG] osdmap e831: 8 total, 7 up, 8 in 2024-09-10T19:08:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:34 smithi165 bash[22591]: audit 2024-09-10T19:08:34.409892+0000 mon.c (mon.2) 866 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:34 smithi165 bash[22591]: audit 2024-09-10T19:08:34.416208+0000 mon.a (mon.0) 3716 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:34.970 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:34.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:35.034 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:35.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:35.104 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:35.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:35.241 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:35.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:35.378 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:35.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:35.511 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:35.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:35.650 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:35.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:35.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:35 smithi165 bash[22591]: cluster 2024-09-10T19:08:34.624506+0000 mon.a (mon.0) 3717 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:35.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:35 smithi165 bash[22591]: cluster 2024-09-10T19:08:34.624590+0000 mon.a (mon.0) 3718 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:08:35.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:35 smithi165 bash[22591]: audit 2024-09-10T19:08:35.411262+0000 mon.a (mon.0) 3719 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:35.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:35 smithi165 bash[22591]: cluster 2024-09-10T19:08:35.411913+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e832: 8 total, 7 up, 8 in 2024-09-10T19:08:35.786 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:35.890 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:35.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[18629]: cluster 2024-09-10T19:08:34.624506+0000 mon.a (mon.0) 3717 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:35.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[18629]: cluster 2024-09-10T19:08:34.624590+0000 mon.a (mon.0) 3718 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:08:35.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[18629]: audit 2024-09-10T19:08:35.411262+0000 mon.a (mon.0) 3719 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:35.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[18629]: cluster 2024-09-10T19:08:35.411913+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e832: 8 total, 7 up, 8 in 2024-09-10T19:08:35.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[32381]: cluster 2024-09-10T19:08:34.624506+0000 mon.a (mon.0) 3717 : cluster [WRN] Health check update: Degraded data redundancy: 85/717 objects degraded (11.855%), 18 pgs degraded (PG_DEGRADED) 2024-09-10T19:08:35.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[32381]: cluster 2024-09-10T19:08:34.624590+0000 mon.a (mon.0) 3718 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:08:35.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[32381]: audit 2024-09-10T19:08:35.411262+0000 mon.a (mon.0) 3719 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:35.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:35 smithi083 bash[32381]: cluster 2024-09-10T19:08:35.411913+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e832: 8 total, 7 up, 8 in 2024-09-10T19:08:35.922 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:35.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:36.025 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.061 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:36.164 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:36.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:36.334 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:36.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- 172.21.15.83:0/2607326062 >> v1:172.21.15.83:6789/0 conn(0x7fc7fc1053a0 legacy=0x7fc7fc107780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- 172.21.15.83:0/2607326062 shutdown_connections 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- 172.21.15.83:0/2607326062 >> 172.21.15.83:0/2607326062 conn(0x7fc7fc074a20 msgr2=0x7fc7fc0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- 172.21.15.83:0/2607326062 shutdown_connections 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- 172.21.15.83:0/2607326062 wait complete. 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 librados: starting msgr at 2024-09-10T19:08:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 librados: starting objecter 2024-09-10T19:08:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 Processor -- start 2024-09-10T19:08:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- start start 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 librados: setting wanted keys 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 librados: calling monclient init 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7fc1143f0 con 0x7fc7fc101d30 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7fc1145a0 con 0x7fc7fc108f60 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc8047c3640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7fc114790 con 0x7fc7fc102670 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.439+0000 7fc802d60640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc7fc101d30 0x7fc7fc113c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47608/0 (socket says 172.21.15.83:47608) 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc802d60640 1 -- 172.21.15.83:0/1216641132 learned_addr learned my addr 172.21.15.83:0/1216641132 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7fc101d30 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7fc102670 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327696333 0 0) 0x7fc7fc1143f0 con 0x7fc7fc101d30 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8003120 con 0x7fc7fc101d30 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066953289 0 0) 0x7fc7fc10ccd0 con 0x7fc7fc102670 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7fc1143f0 con 0x7fc7fc102670 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7fc108f60 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743412712 0 0) 0x7fc7fc1143f0 con 0x7fc7fc102670 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7fc10ccd0 con 0x7fc7fc102670 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7f8001740 con 0x7fc7fc102670 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942430083 0 0) 0x7fc7fc1145a0 con 0x7fc7fc108f60 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e8004660 con 0x7fc7fc108f60 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 104624566 0 0) 0x7fc7e8003120 con 0x7fc7fc101d30 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7fc1145a0 con 0x7fc7fc101d30 2024-09-10T19:08:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7f4001c80 con 0x7fc7fc101d30 2024-09-10T19:08:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1009932428 0 0) 0x7fc7fc10ccd0 con 0x7fc7fc102670 2024-09-10T19:08:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 >> v1:172.21.15.165:6789/0 conn(0x7fc7fc108f60 legacy=0x7fc7fc113540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 >> v1:172.21.15.83:6789/0 conn(0x7fc7fc101d30 legacy=0x7fc7fc113c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc126210 con 0x7fc7fc102670 2024-09-10T19:08:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc8047c3640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7fc1231e0 con 0x7fc7fc102670 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc8047c3640 10 client.49874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc8047c3640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc1236b0 con 0x7fc7fc102670 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc8047c3640 1 librados: init done 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7f74a27b0640 1 -- 172.21.15.83:0/711025012 >> v1:172.21.15.165:6789/0 conn(0x7f749c0fbf70 legacy=0x7f749c10e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7f74a27b0640 1 -- 172.21.15.83:0/711025012 shutdown_connections 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7f74a27b0640 1 -- 172.21.15.83:0/711025012 >> 172.21.15.83:0/711025012 conn(0x7f749c0f70d0 msgr2=0x7f749c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7f74a27b0640 1 -- 172.21.15.83:0/711025012 shutdown_connections 2024-09-10T19:08:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7f74a27b0640 1 -- 172.21.15.83:0/711025012 wait complete. 2024-09-10T19:08:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc7f8001d80 con 0x7fc7fc102670 2024-09-10T19:08:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.443+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc7f8001740 con 0x7fc7fc102670 2024-09-10T19:08:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 librados: starting msgr at 2024-09-10T19:08:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 librados: starting objecter 2024-09-10T19:08:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e4005010 con 0x7fc7fc102670 2024-09-10T19:08:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 Processor -- start 2024-09-10T19:08:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc7f80190c0 con 0x7fc7fc102670 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- start start 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 librados: setting wanted keys 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 librados: calling monclient init 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f749c12b260 con 0x7f749c06f260 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f749c12c400 con 0x7f749c10f8b0 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f749c12d5a0 con 0x7f749c0fb3d0 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f749bfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f749c0fb3d0 0x7f749c10bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:57046/0 (socket says 172.21.15.83:57046) 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f749bfff640 1 -- 172.21.15.83:0/2528857522 learned_addr learned my addr 172.21.15.83:0/2528857522 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 10 client.?.objecter ms_handle_connect 0x7f749c0fb3d0 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 10 client.?.objecter ms_handle_connect 0x7f749c10f8b0 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(833..833 src has 750..833) v4 ==== 8531+0+0 (unknown 2005715411 0 0) 0x7fc7f8087a20 con 0x7fc7fc102670 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 10 client.49874.objecter ms_dispatch 0x7fc7fc000b90 osd_map(833..833 src has 750..833) v4 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 3 client.49874.objecter handle_osd_map got epochs [833,833] > 0 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 3 client.49874.objecter handle_osd_map decoding full epoch 833 2024-09-10T19:08:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 20 client.49874.objecter dump_active .. 0 homeless 2024-09-10T19:08:36.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133758020 0 0) 0x7f749c12c400 con 0x7f749c10f8b0 2024-09-10T19:08:36.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7484003120 con 0x7f749c10f8b0 2024-09-10T19:08:36.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7fc7f2ffd640 10 client.49874.objecter ms_handle_connect 0x7fc7e8071a10 2024-09-10T19:08:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 279626088 0 0) 0x7f7484003120 con 0x7f749c10f8b0 2024-09-10T19:08:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f749c12c400 con 0x7f749c10f8b0 2024-09-10T19:08:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f7490002080 con 0x7f749c10f8b0 2024-09-10T19:08:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336335092 0 0) 0x7f749c12c400 con 0x7f749c10f8b0 2024-09-10T19:08:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 >> v1:172.21.15.83:6790/0 conn(0x7f749c0fb3d0 legacy=0x7f749c10bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 >> v1:172.21.15.83:6789/0 conn(0x7f749c06f260 legacy=0x7f749c129b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c12e7a0 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c12c5f0 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 10 client.50389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c12ca40 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 librados: init done 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f7490001b00 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f74900035d0 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.447+0000 7f74a27b0640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7488004ff0 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f7490002a60 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(833..833 src has 750..833) v4 ==== 8531+0+0 (unknown 2005715411 0 0) 0x7f7490088140 con 0x7f749c10f8b0 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 osd_map(833..833 src has 750..833) v4 2024-09-10T19:08:36.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 3 client.50389.objecter handle_osd_map got epochs [833,833] > 0 2024-09-10T19:08:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 3 client.50389.objecter handle_osd_map decoding full epoch 833 2024-09-10T19:08:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 20 client.50389.objecter dump_active .. 0 homeless 2024-09-10T19:08:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v833) v1 ==== 54+0+16 (unknown 2640300698 0 2420000192) 0x7f7490088530 con 0x7f749c10f8b0 2024-09-10T19:08:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f7484071690 2024-09-10T19:08:36.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc7f8054b20 con 0x7fc7fc102670 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 10 client.50389.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 1 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 1 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 0 1 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 4 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 10 client.50389.objecter _send_command 1 2024-09-10T19:08:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74880022e0 con 0x7f7488005a80 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.451+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f7488005a80 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.455+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f74880022e0 con 0x7f7488005a80 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.455+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.455+0000 7f7498ff9640 10 client.50389.objecter _finish_command 1 = system:0 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.455+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 4 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.455+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 0 1 2024-09-10T19:08:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 10 client.50389.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 2 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 2 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488005810 osd=0 3 2024-09-10T19:08:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 0 2 2024-09-10T19:08:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 4 2024-09-10T19:08:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 10 client.50389.objecter _send_command 2 2024-09-10T19:08:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.467+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7488009580 con 0x7f7488005a80 2024-09-10T19:08:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7488009580 con 0x7f7488005a80 2024-09-10T19:08:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 10 client.50389.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488005810 osd=0 4 2024-09-10T19:08:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 0 2 2024-09-10T19:08:36.476 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 10 client.50389.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 3 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter _get_session osd=-1 returning homeless 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 10 client.50389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=834}) v3 -- 0x7f7488001850 con 0x7f749c10f8b0 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7488001c30 con 0x7f749c10f8b0 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_get_version_reply(handle=1 version=833) v2 ==== 24+0+0 (unknown 3121706793 0 0) 0x7f7490054510 con 0x7f749c10f8b0 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f749a7fc640 10 client.50389.objecter _check_command_map_dne tid 3 current 833 map_dne_bound 833 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f749a7fc640 10 client.50389.objecter _finish_command 3 = osd:6 osd down 2024-09-10T19:08:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f749a7fc640 15 client.50389.objecter _session_command_op_remove -1 3 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 10 client.50389.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 4 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 4 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 2 4 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 4 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 10 client.50389.objecter _send_command 4 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6809/386094220 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748800b490 con 0x7f748800a870 2024-09-10T19:08:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f748800a870 2024-09-10T19:08:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f748800b490 con 0x7f748800a870 2024-09-10T19:08:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 4: 0 ) v1 2024-09-10T19:08:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 10 client.50389.objecter _finish_command 4 = system:0 2024-09-10T19:08:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 4 2024-09-10T19:08:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.471+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 2 4 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 10 client.50389.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 5 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 5 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488004e20 osd=2 3 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 2 5 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 4 2024-09-10T19:08:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 10 client.50389.objecter _send_command 5 2024-09-10T19:08:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7488004910 con 0x7f748800a870 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7488004910 con 0x7f748800a870 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f7498ff9640 10 client.50389.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488004e20 osd=2 4 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.491+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 2 5 2024-09-10T19:08:36.498 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 10 client.50389.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 6 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 6 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 3 6 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 4 2024-09-10T19:08:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 10 client.50389.objecter _send_command 6 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6813/1279688021 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7488013230 con 0x7f748800f660 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f748800f660 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f7488013230 con 0x7f748800f660 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 6: 0 ) v1 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 10 client.50389.objecter _finish_command 6 = system:0 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 4 2024-09-10T19:08:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.495+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 3 6 2024-09-10T19:08:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 10 client.50389.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 7 2024-09-10T19:08:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 7 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f748800efe0 osd=3 3 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 3 7 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 4 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 10 client.50389.objecter _send_command 7 2024-09-10T19:08:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.507+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f748800b910 con 0x7f748800f660 2024-09-10T19:08:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f748800b910 con 0x7f748800f660 2024-09-10T19:08:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 10 client.50389.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f748800efe0 osd=3 4 2024-09-10T19:08:36.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 3 7 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 10 client.50389.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 8 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 8 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 4 8 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 4 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 10 client.50389.objecter _send_command 8 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6800/3225515265 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748800a530 con 0x7f74880133a0 2024-09-10T19:08:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.511+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f74880133a0 2024-09-10T19:08:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.515+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f748800a530 con 0x7f74880133a0 2024-09-10T19:08:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.515+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 8: 0 ) v1 2024-09-10T19:08:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.515+0000 7f7498ff9640 10 client.50389.objecter _finish_command 8 = system:0 2024-09-10T19:08:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.515+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 4 2024-09-10T19:08:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.515+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 4 8 2024-09-10T19:08:36.537 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 10 client.50389.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 9 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 9 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488009eb0 osd=4 3 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 4 9 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 4 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 10 client.50389.objecter _send_command 9 2024-09-10T19:08:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.531+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f74880146c0 con 0x7f74880133a0 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f74880146c0 con 0x7f74880133a0 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 10 client.50389.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488009eb0 osd=4 4 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 4 9 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:08:36.540 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:08:36.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 10 client.50389.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 10 2024-09-10T19:08:36.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 10 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 5 10 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 4 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 10 client.50389.objecter _send_command 10 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6804/571173777 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7488014900 con 0x7f7488018ce0 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f7488018ce0 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f7488014900 con 0x7f7488018ce0 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 10: 0 ) v1 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 10 client.50389.objecter _finish_command 10 = system:0 2024-09-10T19:08:36.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 4 2024-09-10T19:08:36.543 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.535+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 5 10 2024-09-10T19:08:36.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 10 client.50389.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 11 2024-09-10T19:08:36.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 11 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f74880146c0 osd=5 3 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 5 11 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 4 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 10 client.50389.objecter _send_command 11 2024-09-10T19:08:36.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7488014900 con 0x7f7488018ce0 2024-09-10T19:08:36.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7488014900 con 0x7f7488018ce0 2024-09-10T19:08:36.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f7498ff9640 10 client.50389.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f74880146c0 osd=5 4 2024-09-10T19:08:36.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.551+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 5 11 2024-09-10T19:08:36.558 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 10 client.50389.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 12 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 12 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 6 12 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 4 2024-09-10T19:08:36.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 10 client.50389.objecter _send_command 12 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6808/1317922936 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f748801d8c0 con 0x7f748801e460 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f748801e460 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f748801d8c0 con 0x7f748801e460 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 12: 0 ) v1 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 10 client.50389.objecter _finish_command 12 = system:0 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 4 2024-09-10T19:08:36.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.555+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 6 12 2024-09-10T19:08:36.565 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:36.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 10 client.50389.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 13 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 13 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f7488014900 osd=6 3 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 6 13 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 4 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 10 client.50389.objecter _send_command 13 2024-09-10T19:08:36.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.567+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f748801d8c0 con 0x7f748801e460 2024-09-10T19:08:36.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f748801d8c0 con 0x7f748801e460 2024-09-10T19:08:36.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f7498ff9640 10 client.50389.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f7488014900 osd=6 4 2024-09-10T19:08:36.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 6 13 2024-09-10T19:08:36.578 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:08:36.579 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:08:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 10 client.50389.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 14 2024-09-10T19:08:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 14 2024-09-10T19:08:36.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 7 14 2024-09-10T19:08:36.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 4 2024-09-10T19:08:36.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 10 client.50389.objecter _send_command 14 2024-09-10T19:08:36.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.571+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6812/1610151650 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7488024f20 con 0x7f748801db00 2024-09-10T19:08:36.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 10 client.50389.objecter ms_handle_connect 0x7f748801db00 2024-09-10T19:08:36.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f7488024f20 con 0x7f748801db00 2024-09-10T19:08:36.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 14: 0 ) v1 2024-09-10T19:08:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 10 client.50389.objecter _finish_command 14 = system:0 2024-09-10T19:08:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 4 2024-09-10T19:08:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.575+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 7 14 2024-09-10T19:08:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 10 client.50389.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign -1 15 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 20 client.50389.objecter _get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 15 client.50389.objecter _session_command_op_remove -1 15 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 20 client.50389.objecter get_session s=0x7f748801d8c0 osd=7 3 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 15 client.50389.objecter _session_command_op_assign 7 15 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 4 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 10 client.50389.objecter _send_command 15 2024-09-10T19:08:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.591+0000 7f74827fc640 1 -- 172.21.15.83:0/2528857522 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7488024f20 con 0x7f748801db00 2024-09-10T19:08:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.595+0000 7f7498ff9640 1 -- 172.21.15.83:0/2528857522 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f7488024f20 con 0x7f748801db00 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.595+0000 7f7498ff9640 10 client.50389.objecter ms_dispatch 0x7f749c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.595+0000 7f7498ff9640 10 client.50389.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.595+0000 7f7498ff9640 20 client.50389.objecter put_session s=0x7f748801d8c0 osd=7 4 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.595+0000 7f7498ff9640 15 client.50389.objecter _session_command_op_remove 7 15 2024-09-10T19:08:36.599 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:08:36.600 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:08:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:36.647+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc7e40052c0 con 0x7fc7fc102670 2024-09-10T19:08:36.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:36.731 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:35.810930+0000 mon.c (mon.2) 867 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:35.811815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: cluster 2024-09-10T19:08:36.392607+0000 mgr.y (mgr.24526) 754 : cluster [DBG] pgmap v1296: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:36.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:36.408335+0000 mon.a (mon.0) 3722 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: cluster 2024-09-10T19:08:36.408481+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e833: 8 total, 7 up, 8 in 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:36.419356+0000 mon.c (mon.2) 868 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:36.419828+0000 mon.a (mon.0) 3724 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[18629]: audit 2024-09-10T19:08:36.452891+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? v1:172.21.15.83:0/2528857522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:35.810930+0000 mon.c (mon.2) 867 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:35.811815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: cluster 2024-09-10T19:08:36.392607+0000 mgr.y (mgr.24526) 754 : cluster [DBG] pgmap v1296: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:36.408335+0000 mon.a (mon.0) 3722 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:36.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: cluster 2024-09-10T19:08:36.408481+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e833: 8 total, 7 up, 8 in 2024-09-10T19:08:36.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:36.419356+0000 mon.c (mon.2) 868 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:36.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:36.419828+0000 mon.a (mon.0) 3724 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:36.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:36 smithi083 bash[32381]: audit 2024-09-10T19:08:36.452891+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? v1:172.21.15.83:0/2528857522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:36.834 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:36.867 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:36.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:36.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:37.004 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:37.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:35.810930+0000 mon.c (mon.2) 867 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:35.811815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: cluster 2024-09-10T19:08:36.392607+0000 mgr.y (mgr.24526) 754 : cluster [DBG] pgmap v1296: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:36.408335+0000 mon.a (mon.0) 3722 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: cluster 2024-09-10T19:08:36.408481+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e833: 8 total, 7 up, 8 in 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:36.419356+0000 mon.c (mon.2) 868 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:36.419828+0000 mon.a (mon.0) 3724 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:36 smithi165 bash[22591]: audit 2024-09-10T19:08:36.452891+0000 mon.b (mon.1) 476 : audit [DBG] from='client.? v1:172.21.15.83:0/2528857522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:37.144 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:37.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:37.281 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:37.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:37.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f2ffd640 1 -- 172.21.15.83:0/1216641132 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v834) v1 ==== 97+0+0 (unknown 4145088770 0 0) 0x7fc7f80544c0 con 0x7fc7fc102670 2024-09-10T19:08:37.415 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is set 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 10 librados: watch_flush enter 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 10 librados: watch_flush exit 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 20 client.49874.objecter shutdown clearing up homeless session... 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 10 client.49874.objecter successfully canceled tick 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc7e8071a10 legacy=0x7fc7e8073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 >> v1:172.21.15.83:6790/0 conn(0x7fc7fc102670 legacy=0x7fc7fc101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 shutdown_connections 2024-09-10T19:08:37.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.411+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 >> 172.21.15.83:0/1216641132 conn(0x7fc7fc074a20 msgr2=0x7fc7fc0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:37.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.415+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 shutdown_connections 2024-09-10T19:08:37.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.415+0000 7fc7f0ff9640 1 -- 172.21.15.83:0/1216641132 wait complete. 2024-09-10T19:08:37.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:37.415+0000 7fc7f0ff9640 1 librados: shutdown 2024-09-10T19:08:37.423 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:37.526 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:37.555 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:37.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:37.692 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:36.650848+0000 mon.c (mon.2) 869 : audit [INF] from='client.? v1:172.21.15.83:0/1216641132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:36.651156+0000 mon.a (mon.0) 3725 : audit [INF] from='client.49874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: cluster 2024-09-10T19:08:37.402946+0000 mon.a (mon.0) 3726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.412664+0000 mon.a (mon.0) 3727 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]': finished 2024-09-10T19:08:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.412895+0000 mon.a (mon.0) 3728 : audit [INF] from='client.49874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: cluster 2024-09-10T19:08:37.413280+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e834: 8 total, 7 up, 8 in 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.447014+0000 mon.c (mon.2) 870 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.447511+0000 mon.a (mon.0) 3730 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.448307+0000 mon.c (mon.2) 871 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[18629]: audit 2024-09-10T19:08:37.448682+0000 mon.a (mon.0) 3731 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:36.650848+0000 mon.c (mon.2) 869 : audit [INF] from='client.? v1:172.21.15.83:0/1216641132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:36.651156+0000 mon.a (mon.0) 3725 : audit [INF] from='client.49874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: cluster 2024-09-10T19:08:37.402946+0000 mon.a (mon.0) 3726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.412664+0000 mon.a (mon.0) 3727 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]': finished 2024-09-10T19:08:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.412895+0000 mon.a (mon.0) 3728 : audit [INF] from='client.49874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:37.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: cluster 2024-09-10T19:08:37.413280+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e834: 8 total, 7 up, 8 in 2024-09-10T19:08:37.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.447014+0000 mon.c (mon.2) 870 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.447511+0000 mon.a (mon.0) 3730 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.448307+0000 mon.c (mon.2) 871 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:37 smithi083 bash[32381]: audit 2024-09-10T19:08:37.448682+0000 mon.a (mon.0) 3731 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:37.829 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:37.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:36.650848+0000 mon.c (mon.2) 869 : audit [INF] from='client.? v1:172.21.15.83:0/1216641132' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:36.651156+0000 mon.a (mon.0) 3725 : audit [INF] from='client.49874 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: cluster 2024-09-10T19:08:37.402946+0000 mon.a (mon.0) 3726 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.412664+0000 mon.a (mon.0) 3727 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]': finished 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.412895+0000 mon.a (mon.0) 3728 : audit [INF] from='client.49874 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: cluster 2024-09-10T19:08:37.413280+0000 mon.a (mon.0) 3729 : cluster [DBG] osdmap e834: 8 total, 7 up, 8 in 2024-09-10T19:08:37.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.447014+0000 mon.c (mon.2) 870 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.447511+0000 mon.a (mon.0) 3730 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:37.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.448307+0000 mon.c (mon.2) 871 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.947 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:37 smithi165 bash[22591]: audit 2024-09-10T19:08:37.448682+0000 mon.a (mon.0) 3731 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-148"}]: dispatch 2024-09-10T19:08:37.967 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:37.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:38.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:38.106 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:38.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:38.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:38.244 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:38.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:38.374 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:38.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:38.510 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:38.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:38.614 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:38.642 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:38.746 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:38.747 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:08:38.747 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:08:38.778 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:38.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:38.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:38 smithi083 bash[18629]: cluster 2024-09-10T19:08:38.394181+0000 mgr.y (mgr.24526) 755 : cluster [DBG] pgmap v1299: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:38.892 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:38 smithi083 bash[18629]: cluster 2024-09-10T19:08:38.416476+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e835: 8 total, 7 up, 8 in 2024-09-10T19:08:38.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:38 smithi083 bash[32381]: cluster 2024-09-10T19:08:38.394181+0000 mgr.y (mgr.24526) 755 : cluster [DBG] pgmap v1299: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:38.892 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:38 smithi083 bash[32381]: cluster 2024-09-10T19:08:38.416476+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e835: 8 total, 7 up, 8 in 2024-09-10T19:08:38.910 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:38.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:08:39.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:08:39.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:39.041 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:38 smithi165 bash[22591]: cluster 2024-09-10T19:08:38.394181+0000 mgr.y (mgr.24526) 755 : cluster [DBG] pgmap v1299: 214 pgs: 4 creating+activating, 1 creating+activating+undersized, 58 active+undersized, 18 active+undersized+degraded, 133 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:38 smithi165 bash[22591]: cluster 2024-09-10T19:08:38.416476+0000 mon.a (mon.0) 3732 : cluster [DBG] osdmap e835: 8 total, 7 up, 8 in 2024-09-10T19:08:39.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:39.179 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:39.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:39.318 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:39.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:08:39.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:39.458 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:39.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:39.591 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:39.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:39.727 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:39.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:39.867 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:39.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:40.008 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:39 smithi165 bash[22591]: audit 2024-09-10T19:08:38.460916+0000 mgr.y (mgr.24526) 756 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:39 smithi165 bash[22591]: cluster 2024-09-10T19:08:38.856288+0000 mon.a (mon.0) 3733 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:39 smithi165 bash[22591]: cluster 2024-09-10T19:08:39.419085+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e836: 8 total, 7 up, 8 in 2024-09-10T19:08:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:39 smithi165 bash[22591]: audit 2024-09-10T19:08:39.420539+0000 mon.c (mon.2) 872 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:39 smithi165 bash[22591]: audit 2024-09-10T19:08:39.429191+0000 mon.a (mon.0) 3735 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:40.140 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:40.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[18629]: audit 2024-09-10T19:08:38.460916+0000 mgr.y (mgr.24526) 756 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:40.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[18629]: cluster 2024-09-10T19:08:38.856288+0000 mon.a (mon.0) 3733 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:40.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[18629]: cluster 2024-09-10T19:08:39.419085+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e836: 8 total, 7 up, 8 in 2024-09-10T19:08:40.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[18629]: audit 2024-09-10T19:08:39.420539+0000 mon.c (mon.2) 872 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.166 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[18629]: audit 2024-09-10T19:08:39.429191+0000 mon.a (mon.0) 3735 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[32381]: audit 2024-09-10T19:08:38.460916+0000 mgr.y (mgr.24526) 756 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:40.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[32381]: cluster 2024-09-10T19:08:38.856288+0000 mon.a (mon.0) 3733 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:40.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[32381]: cluster 2024-09-10T19:08:39.419085+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e836: 8 total, 7 up, 8 in 2024-09-10T19:08:40.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[32381]: audit 2024-09-10T19:08:39.420539+0000 mon.c (mon.2) 872 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.167 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:39 smithi083 bash[32381]: audit 2024-09-10T19:08:39.429191+0000 mon.a (mon.0) 3735 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:40.243 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.279 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:40.382 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:40.418 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:40.527 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.562 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:40.666 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.700 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:40.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:40.835 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:40.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: cluster 2024-09-10T19:08:40.395941+0000 mgr.y (mgr.24526) 757 : cluster [DBG] pgmap v1302: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/720 objects degraded (11.806%) 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: audit 2024-09-10T19:08:40.420421+0000 mon.a (mon.0) 3736 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: cluster 2024-09-10T19:08:40.420823+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e837: 8 total, 7 up, 8 in 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: audit 2024-09-10T19:08:40.462725+0000 mon.c (mon.2) 873 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: audit 2024-09-10T19:08:40.463177+0000 mon.a (mon.0) 3738 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: audit 2024-09-10T19:08:40.463934+0000 mon.c (mon.2) 874 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:40.950 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[32381]: audit 2024-09-10T19:08:40.464311+0000 mon.a (mon.0) 3739 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: cluster 2024-09-10T19:08:40.395941+0000 mgr.y (mgr.24526) 757 : cluster [DBG] pgmap v1302: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/720 objects degraded (11.806%) 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: audit 2024-09-10T19:08:40.420421+0000 mon.a (mon.0) 3736 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: cluster 2024-09-10T19:08:40.420823+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e837: 8 total, 7 up, 8 in 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: audit 2024-09-10T19:08:40.462725+0000 mon.c (mon.2) 873 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: audit 2024-09-10T19:08:40.463177+0000 mon.a (mon.0) 3738 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: audit 2024-09-10T19:08:40.463934+0000 mon.c (mon.2) 874 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:40.951 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:40 smithi083 bash[18629]: audit 2024-09-10T19:08:40.464311+0000 mon.a (mon.0) 3739 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:40.972 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:40.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:41.013 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:08:41.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.108 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: cluster 2024-09-10T19:08:40.395941+0000 mgr.y (mgr.24526) 757 : cluster [DBG] pgmap v1302: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 85/720 objects degraded (11.806%) 2024-09-10T19:08:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: audit 2024-09-10T19:08:40.420421+0000 mon.a (mon.0) 3736 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: cluster 2024-09-10T19:08:40.420823+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e837: 8 total, 7 up, 8 in 2024-09-10T19:08:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: audit 2024-09-10T19:08:40.462725+0000 mon.c (mon.2) 873 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: audit 2024-09-10T19:08:40.463177+0000 mon.a (mon.0) 3738 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: audit 2024-09-10T19:08:40.463934+0000 mon.c (mon.2) 874 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:40 smithi165 bash[22591]: audit 2024-09-10T19:08:40.464311+0000 mon.a (mon.0) 3739 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-150"}]: dispatch 2024-09-10T19:08:41.212 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:41.237 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:41.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.371 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:41.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:41.507 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:41.610 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.640 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:41.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:08:41.744 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:41.772 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:41.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:41.875 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:41.903 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:41.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:42.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:42.044 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:42.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:42.181 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:42.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:42.316 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:42.420 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:42.451 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:42.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:42.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:42 smithi165 bash[22591]: cluster 2024-09-10T19:08:41.421821+0000 mon.a (mon.0) 3740 : cluster [DBG] osdmap e838: 8 total, 7 up, 8 in 2024-09-10T19:08:42.563 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:42 smithi165 bash[22591]: cluster 2024-09-10T19:08:42.397491+0000 mgr.y (mgr.24526) 758 : cluster [DBG] pgmap v1305: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:42.585 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:42.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:42.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:42 smithi083 bash[18629]: cluster 2024-09-10T19:08:41.421821+0000 mon.a (mon.0) 3740 : cluster [DBG] osdmap e838: 8 total, 7 up, 8 in 2024-09-10T19:08:42.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:42 smithi083 bash[18629]: cluster 2024-09-10T19:08:42.397491+0000 mgr.y (mgr.24526) 758 : cluster [DBG] pgmap v1305: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:42.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:42 smithi083 bash[32381]: cluster 2024-09-10T19:08:41.421821+0000 mon.a (mon.0) 3740 : cluster [DBG] osdmap e838: 8 total, 7 up, 8 in 2024-09-10T19:08:42.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:42 smithi083 bash[32381]: cluster 2024-09-10T19:08:42.397491+0000 mgr.y (mgr.24526) 758 : cluster [DBG] pgmap v1305: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:42.718 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:42.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:42.852 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:42.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- 172.21.15.83:0/734631487 >> v1:172.21.15.83:6790/0 conn(0x7ff65c10b130 legacy=0x7ff65c10d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- 172.21.15.83:0/734631487 shutdown_connections 2024-09-10T19:08:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- 172.21.15.83:0/734631487 >> 172.21.15.83:0/734631487 conn(0x7ff65c0fa370 msgr2=0x7ff65c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- 172.21.15.83:0/734631487 shutdown_connections 2024-09-10T19:08:42.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- 172.21.15.83:0/734631487 wait complete. 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 librados: starting msgr at 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 librados: starting objecter 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 Processor -- start 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- start start 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 librados: setting wanted keys 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 librados: calling monclient init 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff65c1040b0 con 0x7ff65c104810 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff65c11f630 con 0x7ff65c1075a0 2024-09-10T19:08:42.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.963+0000 7ff6635a4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff65c1207d0 con 0x7ff65c10b130 2024-09-10T19:08:42.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff661b41640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff65c10b130 0x7ff65c11cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:57072/0 (socket says 172.21.15.83:57072) 2024-09-10T19:08:42.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff661b41640 1 -- 172.21.15.83:0/1827029381 learned_addr learned my addr 172.21.15.83:0/1827029381 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff661340640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff65c104810 0x7ff65c103500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:47650/0 (socket says 172.21.15.83:47650) 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter ms_handle_connect 0x7ff65c104810 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter ms_handle_connect 0x7ff65c10b130 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter ms_handle_connect 0x7ff65c1075a0 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220999208 0 0) 0x7ff65c1040b0 con 0x7ff65c104810 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff648003720 con 0x7ff65c104810 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836261723 0 0) 0x7ff65c1207d0 con 0x7ff65c10b130 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c1040b0 con 0x7ff65c10b130 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639101036 0 0) 0x7ff65c11f630 con 0x7ff65c1075a0 2024-09-10T19:08:42.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff65c1207d0 con 0x7ff65c1075a0 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918871949 0 0) 0x7ff648003720 con 0x7ff65c104810 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff65c11f630 con 0x7ff65c104810 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff64c0018c0 con 0x7ff65c104810 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2305463826 0 0) 0x7ff65c1040b0 con 0x7ff65c10b130 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff648003720 con 0x7ff65c10b130 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff6540017b0 con 0x7ff65c10b130 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2492204928 0 0) 0x7ff65c11f630 con 0x7ff65c104810 2024-09-10T19:08:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 >> v1:172.21.15.83:6790/0 conn(0x7ff65c10b130 legacy=0x7ff65c11cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 >> v1:172.21.15.165:6789/0 conn(0x7ff65c1075a0 legacy=0x7ff65c1197d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c1219b0 con 0x7ff65c104810 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c11f820 con 0x7ff65c104810 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff6635a4640 10 client.25797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c11fcf0 con 0x7ff65c104810 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff6635a4640 1 librados: init done 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff64c002310 con 0x7ff65c104810 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.967+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff64c005a10 con 0x7ff65c104810 2024-09-10T19:08:42.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff640005010 con 0x7ff65c104810 2024-09-10T19:08:42.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff64c001eb0 con 0x7ff65c104810 2024-09-10T19:08:42.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(839..839 src has 750..839) v4 ==== 8523+0+0 (unknown 2311606309 0 0) 0x7ff64c08a580 con 0x7ff65c104810 2024-09-10T19:08:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 10 client.25797.objecter ms_dispatch 0x7ff65c000b90 osd_map(839..839 src has 750..839) v4 2024-09-10T19:08:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 3 client.25797.objecter handle_osd_map got epochs [839,839] > 0 2024-09-10T19:08:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 3 client.25797.objecter handle_osd_map decoding full epoch 839 2024-09-10T19:08:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 20 client.25797.objecter dump_active .. 0 homeless 2024-09-10T19:08:42.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.971+0000 7ff651ffb640 10 client.25797.objecter ms_handle_connect 0x7ff6480714d0 2024-09-10T19:08:42.983 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:42.979+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff64c056950 con 0x7ff65c104810 2024-09-10T19:08:42.991 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:42.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:42] "GET /metrics HTTP/1.1" 200 229158 "" "Prometheus/2.43.0" 2024-09-10T19:08:43.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:43.126 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:43.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.139+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff640000b90 con 0x7ff6480714d0 2024-09-10T19:08:43.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.167+0000 7ff651ffb640 1 -- 172.21.15.83:0/1827029381 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+586556 (unknown 2965378022 0 4122641678) 0x7ff640000b90 con 0x7ff6480714d0 2024-09-10T19:08:43.178 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:08:43.182 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 10 librados: watch_flush enter 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 10 librados: watch_flush exit 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 20 client.25797.objecter shutdown clearing up homeless session... 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 10 client.25797.objecter successfully canceled tick 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff6480714d0 legacy=0x7ff648073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 >> v1:172.21.15.83:6789/0 conn(0x7ff65c104810 legacy=0x7ff65c103500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:43.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 shutdown_connections 2024-09-10T19:08:43.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 >> 172.21.15.83:0/1827029381 conn(0x7ff65c0fa370 msgr2=0x7ff65c0fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:43.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 shutdown_connections 2024-09-10T19:08:43.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 -- 172.21.15.83:0/1827029381 wait complete. 2024-09-10T19:08:43.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:43.175+0000 7ff6635a4640 1 librados: shutdown 2024-09-10T19:08:43.230 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:43.257 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:43.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.395 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:43.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:43.530 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:43.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: cluster 2024-09-10T19:08:42.185815+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:43.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: cluster 2024-09-10T19:08:42.186091+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:43.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: cluster 2024-09-10T19:08:42.433352+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e839: 8 total, 7 up, 8 in 2024-09-10T19:08:43.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: audit 2024-09-10T19:08:42.441016+0000 mon.c (mon.2) 875 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.647 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: audit 2024-09-10T19:08:42.441650+0000 mon.a (mon.0) 3742 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.648 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:43 smithi165 bash[22591]: audit 2024-09-10T19:08:43.145634+0000 mgr.y (mgr.24526) 759 : audit [DBG] from='client.25797 v1:172.21.15.83:0/1827029381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:08:43.669 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:43.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:43.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: cluster 2024-09-10T19:08:42.185815+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:43.786 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: cluster 2024-09-10T19:08:42.186091+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: cluster 2024-09-10T19:08:42.433352+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e839: 8 total, 7 up, 8 in 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: audit 2024-09-10T19:08:42.441016+0000 mon.c (mon.2) 875 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: audit 2024-09-10T19:08:42.441650+0000 mon.a (mon.0) 3742 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[18629]: audit 2024-09-10T19:08:43.145634+0000 mgr.y (mgr.24526) 759 : audit [DBG] from='client.25797 v1:172.21.15.83:0/1827029381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: cluster 2024-09-10T19:08:42.185815+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: cluster 2024-09-10T19:08:42.186091+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: cluster 2024-09-10T19:08:42.433352+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e839: 8 total, 7 up, 8 in 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: audit 2024-09-10T19:08:42.441016+0000 mon.c (mon.2) 875 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: audit 2024-09-10T19:08:42.441650+0000 mon.a (mon.0) 3742 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:43.787 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:43 smithi083 bash[32381]: audit 2024-09-10T19:08:43.145634+0000 mgr.y (mgr.24526) 759 : audit [DBG] from='client.25797 v1:172.21.15.83:0/1827029381' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:08:43.800 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:43.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.936 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:43.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:44.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:44.073 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:44.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:44.199 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:44.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:44.305 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":1306,"stamp":"2024-09-10T19:08:42.430632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4650152,"num_objects":240,"num_object_clones":1,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":2350,"num_read_kb":2093,"num_write":801,"num_write_kb":4165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":53762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1895,"ondisk_log_size":1895,"up":484,"acting":484,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":617,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2443836,"kb_used_data":27524,"kb_used_omap":13,"kb_used_meta":2416178,"kb_avail":747517380,"statfs":{"total":767960285184,"available":765457797120,"internally_reserved":0,"allocated":28184576,"data_stored":15469541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2474166976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020634"},"pg_stats":[{"pgid":"262.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"262.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-10T19:08:42.416918+0000","last_change":"2024-09-10T19:08:42.416918+0000","last_active":"2024-09-10T19:08:42.416918+0000","last_peered":"2024-09-10T19:08:42.416918+0000","last_clean":"2024-09-10T19:08:42.416918+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:08:42.416918+0000","last_undegraded":"2024-09-10T19:08:42.416918+0000","last_fullsized":"2024-09-10T19:08:42.416918+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:42.416918+0000","last_clean_scrub_stamp":"2024-09-10T19:08:42.416918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"241.7","version":"795'10","reported_seq":238,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.429886+0000","last_change":"2024-09-10T19:07:36.406466+0000","last_active":"2024-09-10T19:08:39.429886+0000","last_peered":"2024-09-10T19:08:39.429886+0000","last_clean":"2024-09-10T19:07:34.941540+0000","last_became_active":"2024-09-10T19:07:36.406466+0000","last_became_peered":"2024-09-10T19:07:36.406466+0000","last_unstale":"2024-09-10T19:08:39.429886+0000","last_undegraded":"2024-09-10T19:07:36.385676+0000","last_fullsized":"2024-09-10T19:07:36.385536+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'6","last_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_deep_scrub":"730'6","last_deep_scrub_stamp":"2024-09-10T19:06:47.006586+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.006586+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004755039,"objects_trimmed":0,"snaptrim_duration":0.00095805300000000005,"stat_sum":{"num_bytes":4000037,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":3912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2147483647],"acting":[5,0,2147483647],"avail_no_missing":["5(0)","0(1)"],"object_location_counts":[{"shards":"0(1),5(0)","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"241.6","version":"837'14","reported_seq":182,"reported_epoch":837,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:40.445453+0000","last_change":"2024-09-10T19:07:36.392061+0000","last_active":"2024-09-10T19:08:40.445453+0000","last_peered":"2024-09-10T19:08:40.445453+0000","last_clean":"2024-09-10T19:06:37.458414+0000","last_became_active":"2024-09-10T19:07:36.392061+0000","last_became_peered":"2024-09-10T19:07:36.392061+0000","last_unstale":"2024-09-10T19:08:40.445453+0000","last_undegraded":"2024-09-10T19:07:36.386912+0000","last_fullsized":"2024-09-10T19:07:36.386067+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'9","last_scrub_stamp":"2024-09-10T19:06:49.172954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:49.172954+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010222,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":11,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,5],"acting":[2147483647,2,5],"avail_no_missing":["2(1)","5(2)"],"object_location_counts":[{"shards":"2(1),5(2)","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.5","version":"819'8","reported_seq":232,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.431926+0000","last_change":"2024-09-10T19:06:52.119235+0000","last_active":"2024-09-10T19:08:39.431926+0000","last_peered":"2024-09-10T19:08:39.431926+0000","last_clean":"2024-09-10T19:08:39.431926+0000","last_became_active":"2024-09-10T19:04:57.322094+0000","last_became_peered":"2024-09-10T19:04:57.322094+0000","last_unstale":"2024-09-10T19:08:39.431926+0000","last_undegraded":"2024-09-10T19:08:39.431926+0000","last_fullsized":"2024-09-10T19:08:39.431926+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"720'6","last_scrub_stamp":"2024-09-10T19:06:52.119058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.119058+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043290079999999996,"objects_trimmed":0,"snaptrim_duration":0.0010204319999999999,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"241.4","version":"795'14","reported_seq":247,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424743+0000","last_change":"2024-09-10T19:08:20.083585+0000","last_active":"2024-09-10T19:08:39.424743+0000","last_peered":"2024-09-10T19:08:39.424743+0000","last_clean":"2024-09-10T19:08:39.424743+0000","last_became_active":"2024-09-10T19:04:57.322911+0000","last_became_peered":"2024-09-10T19:04:57.322911+0000","last_unstale":"2024-09-10T19:08:39.424743+0000","last_undegraded":"2024-09-10T19:08:39.424743+0000","last_fullsized":"2024-09-10T19:08:39.424743+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'14","last_scrub_stamp":"2024-09-10T19:08:20.083419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:08:20.083419+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:25.258312+0000","scrub_duration":0.0092037939999999995,"objects_trimmed":0,"snaptrim_duration":0.0017454879999999999,"stat_sum":{"num_bytes":41,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"241.3","version":"819'18","reported_seq":252,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422944+0000","last_change":"2024-09-10T19:07:48.526109+0000","last_active":"2024-09-10T19:08:39.422944+0000","last_peered":"2024-09-10T19:08:39.422944+0000","last_clean":"2024-09-10T19:08:39.422944+0000","last_became_active":"2024-09-10T19:04:57.324250+0000","last_became_peered":"2024-09-10T19:04:57.324250+0000","last_unstale":"2024-09-10T19:08:39.422944+0000","last_undegraded":"2024-09-10T19:08:39.422944+0000","last_fullsized":"2024-09-10T19:08:39.422944+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'15","last_scrub_stamp":"2024-09-10T19:07:48.526005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:07:48.526005+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:04.735431+0000","scrub_duration":0.0034068290000000001,"objects_trimmed":0,"snaptrim_duration":0.0036847949999999998,"stat_sum":{"num_bytes":69,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"241.2","version":"795'14","reported_seq":247,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430342+0000","last_change":"2024-09-10T19:07:56.393847+0000","last_active":"2024-09-10T19:08:39.430342+0000","last_peered":"2024-09-10T19:08:39.430342+0000","last_clean":"2024-09-10T19:08:39.430342+0000","last_became_active":"2024-09-10T19:04:57.322254+0000","last_became_peered":"2024-09-10T19:04:57.322254+0000","last_unstale":"2024-09-10T19:08:39.430342+0000","last_undegraded":"2024-09-10T19:08:39.430342+0000","last_fullsized":"2024-09-10T19:08:39.430342+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'14","last_scrub_stamp":"2024-09-10T19:07:56.393576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:07:56.393576+0000","objects_scrubbed":6,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:58.981621+0000","scrub_duration":0.0059973200000000004,"objects_trimmed":0,"snaptrim_duration":0.0028767139999999998,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"241.1","version":"795'19","reported_seq":244,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.415149+0000","last_change":"2024-09-10T19:08:14.139592+0000","last_active":"2024-09-10T19:08:39.415149+0000","last_peered":"2024-09-10T19:08:39.415149+0000","last_clean":"2024-09-10T19:08:39.415149+0000","last_became_active":"2024-09-10T19:04:57.329766+0000","last_became_peered":"2024-09-10T19:04:57.329766+0000","last_unstale":"2024-09-10T19:08:39.415149+0000","last_undegraded":"2024-09-10T19:08:39.415149+0000","last_fullsized":"2024-09-10T19:08:39.415149+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'19","last_scrub_stamp":"2024-09-10T19:08:14.139415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:08:14.139415+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:22.514159+0000","scrub_duration":0.005948984,"objects_trimmed":0,"snaptrim_duration":0.52769914500000004,"stat_sum":{"num_bytes":42,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":11,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"241.0","version":"833'3","reported_seq":232,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420672+0000","last_change":"2024-09-10T19:08:07.120066+0000","last_active":"2024-09-10T19:08:39.420672+0000","last_peered":"2024-09-10T19:08:39.420672+0000","last_clean":"2024-09-10T19:08:39.420672+0000","last_became_active":"2024-09-10T19:04:57.320088+0000","last_became_peered":"2024-09-10T19:04:57.320088+0000","last_unstale":"2024-09-10T19:08:39.420672+0000","last_undegraded":"2024-09-10T19:08:39.420672+0000","last_fullsized":"2024-09-10T19:08:39.420672+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":653,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'1","last_scrub_stamp":"2024-09-10T19:08:07.119957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:56.297048+0000","last_clean_scrub_stamp":"2024-09-10T19:08:07.119957+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:13.588993+0000","scrub_duration":0.0032611749999999998,"objects_trimmed":0,"snaptrim_duration":0.0018060820000000001,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":840,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417291+0000","last_change":"2024-09-10T19:08:11.167191+0000","last_active":"2024-09-10T19:08:39.417291+0000","last_peered":"2024-09-10T19:08:39.417291+0000","last_clean":"2024-09-10T19:08:39.417291+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:08:39.417291+0000","last_undegraded":"2024-09-10T19:08:39.417291+0000","last_fullsized":"2024-09-10T19:08:39.417291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:11.167033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_clean_scrub_stamp":"2024-09-10T19:08:11.167033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:30.727993+0000","scrub_duration":0.0036252020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":850,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421828+0000","last_change":"2024-09-10T19:08:15.146126+0000","last_active":"2024-09-10T19:08:39.421828+0000","last_peered":"2024-09-10T19:08:39.421828+0000","last_clean":"2024-09-10T19:08:39.421828+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:08:39.421828+0000","last_undegraded":"2024-09-10T19:08:39.421828+0000","last_fullsized":"2024-09-10T19:08:39.421828+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:15.144619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:08:15.144619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:32.972170+0000","scrub_duration":0.0041512729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"410'20","reported_seq":864,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418036+0000","last_change":"2024-09-10T19:06:29.097504+0000","last_active":"2024-09-10T19:08:39.418036+0000","last_peered":"2024-09-10T19:08:39.418036+0000","last_clean":"2024-09-10T19:08:39.418036+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:08:39.418036+0000","last_undegraded":"2024-09-10T19:08:39.418036+0000","last_fullsized":"2024-09-10T19:08:39.418036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'20","last_scrub_stamp":"2024-09-10T19:06:29.097329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.097329+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048839570000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417351+0000","last_change":"2024-09-10T19:08:02.150931+0000","last_active":"2024-09-10T19:08:39.417351+0000","last_peered":"2024-09-10T19:08:39.417351+0000","last_clean":"2024-09-10T19:08:39.417351+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:08:39.417351+0000","last_undegraded":"2024-09-10T19:08:39.417351+0000","last_fullsized":"2024-09-10T19:08:39.417351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:02.150772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:08:02.150772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:09.196188+0000","scrub_duration":0.0031741600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":759,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422959+0000","last_change":"2024-09-10T19:06:35.862278+0000","last_active":"2024-09-10T19:08:39.422959+0000","last_peered":"2024-09-10T19:08:39.422959+0000","last_clean":"2024-09-10T19:08:39.422959+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:08:39.422959+0000","last_undegraded":"2024-09-10T19:08:39.422959+0000","last_fullsized":"2024-09-10T19:08:39.422959+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.862120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.862120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026679030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":856,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420585+0000","last_change":"2024-09-10T19:07:56.134341+0000","last_active":"2024-09-10T19:08:39.420585+0000","last_peered":"2024-09-10T19:08:39.420585+0000","last_clean":"2024-09-10T19:08:39.420585+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:08:39.420585+0000","last_undegraded":"2024-09-10T19:08:39.420585+0000","last_fullsized":"2024-09-10T19:08:39.420585+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:56.134234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:07:56.134234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.207532+0000","scrub_duration":0.0029342219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"410'19","reported_seq":869,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421685+0000","last_change":"2024-09-10T19:06:23.367532+0000","last_active":"2024-09-10T19:08:39.421685+0000","last_peered":"2024-09-10T19:08:39.421685+0000","last_clean":"2024-09-10T19:08:39.421685+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:08:39.421685+0000","last_undegraded":"2024-09-10T19:08:39.421685+0000","last_fullsized":"2024-09-10T19:08:39.421685+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:23.367353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.367353+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057060319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":866,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419615+0000","last_change":"2024-09-10T19:07:53.418975+0000","last_active":"2024-09-10T19:08:39.419615+0000","last_peered":"2024-09-10T19:08:39.419615+0000","last_clean":"2024-09-10T19:08:39.419615+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:08:39.419615+0000","last_undegraded":"2024-09-10T19:08:39.419615+0000","last_fullsized":"2024-09-10T19:08:39.419615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:53.418806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:07:53.418806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:55.081457+0000","scrub_duration":0.0033107850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":323,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.423243+0000","last_change":"2024-09-10T19:07:36.401986+0000","last_active":"2024-09-10T19:08:39.423243+0000","last_peered":"2024-09-10T19:08:39.423243+0000","last_clean":"2024-09-10T19:07:34.940462+0000","last_became_active":"2024-09-10T19:07:36.401986+0000","last_became_peered":"2024-09-10T19:07:36.401986+0000","last_unstale":"2024-09-10T19:08:39.423243+0000","last_undegraded":"2024-09-10T19:08:39.423243+0000","last_fullsized":"2024-09-10T19:07:36.382177+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:16.980824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:16.980824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034584770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":762,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.416240+0000","last_change":"2024-09-10T19:07:36.404145+0000","last_active":"2024-09-10T19:08:39.416240+0000","last_peered":"2024-09-10T19:08:39.416240+0000","last_clean":"2024-09-10T19:07:34.939911+0000","last_became_active":"2024-09-10T19:07:36.404145+0000","last_became_peered":"2024-09-10T19:07:36.404145+0000","last_unstale":"2024-09-10T19:08:39.416240+0000","last_undegraded":"2024-09-10T19:08:39.416240+0000","last_fullsized":"2024-09-10T19:07:36.383752+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:43.403297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.403297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062412609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":849,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432013+0000","last_change":"2024-09-10T19:06:37.935702+0000","last_active":"2024-09-10T19:08:39.432013+0000","last_peered":"2024-09-10T19:08:39.432013+0000","last_clean":"2024-09-10T19:08:39.432013+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:08:39.432013+0000","last_undegraded":"2024-09-10T19:08:39.432013+0000","last_fullsized":"2024-09-10T19:08:39.432013+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.935545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.935545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003538518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"410'34","reported_seq":897,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.426043+0000","last_change":"2024-09-10T19:07:36.402970+0000","last_active":"2024-09-10T19:08:39.426043+0000","last_peered":"2024-09-10T19:08:39.426043+0000","last_clean":"2024-09-10T19:07:34.941528+0000","last_became_active":"2024-09-10T19:07:36.402970+0000","last_became_peered":"2024-09-10T19:07:36.402970+0000","last_unstale":"2024-09-10T19:08:39.426043+0000","last_undegraded":"2024-09-10T19:07:36.387782+0000","last_fullsized":"2024-09-10T19:07:36.387668+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:29.746751+0000","last_deep_scrub":"108'22","last_deep_scrub_stamp":"2024-09-10T18:52:28.673170+0000","last_clean_scrub_stamp":"2024-09-10T19:06:29.746751+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059564860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":319,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.417817+0000","last_change":"2024-09-10T19:07:36.391380+0000","last_active":"2024-09-10T19:08:39.417817+0000","last_peered":"2024-09-10T19:08:39.417817+0000","last_clean":"2024-09-10T19:07:34.371975+0000","last_became_active":"2024-09-10T19:07:36.391380+0000","last_became_peered":"2024-09-10T19:07:36.391380+0000","last_unstale":"2024-09-10T19:08:39.417817+0000","last_undegraded":"2024-09-10T19:08:39.417817+0000","last_fullsized":"2024-09-10T19:07:36.378055+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.098384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.098384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034406860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.423138+0000","last_change":"2024-09-10T19:07:36.392501+0000","last_active":"2024-09-10T19:08:39.423138+0000","last_peered":"2024-09-10T19:08:39.423138+0000","last_clean":"2024-09-10T19:07:34.937906+0000","last_became_active":"2024-09-10T19:07:36.392501+0000","last_became_peered":"2024-09-10T19:07:36.392501+0000","last_unstale":"2024-09-10T19:08:39.423138+0000","last_undegraded":"2024-09-10T19:08:39.423138+0000","last_fullsized":"2024-09-10T19:07:36.382172+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.820051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.820051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003619667,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.421178+0000","last_change":"2024-09-10T19:07:36.392013+0000","last_active":"2024-09-10T19:08:39.421178+0000","last_peered":"2024-09-10T19:08:39.421178+0000","last_clean":"2024-09-10T18:54:31.683713+0000","last_became_active":"2024-09-10T19:07:36.392013+0000","last_became_peered":"2024-09-10T19:07:36.392013+0000","last_unstale":"2024-09-10T19:08:39.421178+0000","last_undegraded":"2024-09-10T19:08:39.421178+0000","last_fullsized":"2024-09-10T19:07:36.378539+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.107798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.107798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":862,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418670+0000","last_change":"2024-09-10T19:08:20.050091+0000","last_active":"2024-09-10T19:08:39.418670+0000","last_peered":"2024-09-10T19:08:39.418670+0000","last_clean":"2024-09-10T19:08:39.418670+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:08:39.418670+0000","last_undegraded":"2024-09-10T19:08:39.418670+0000","last_fullsized":"2024-09-10T19:08:39.418670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:20.049826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:08:20.049826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:34.281087+0000","scrub_duration":0.0035106790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"410'23","reported_seq":887,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.426456+0000","last_change":"2024-09-10T19:06:26.733413+0000","last_active":"2024-09-10T19:08:39.426456+0000","last_peered":"2024-09-10T19:08:39.426456+0000","last_clean":"2024-09-10T19:08:39.426456+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:08:39.426456+0000","last_undegraded":"2024-09-10T19:08:39.426456+0000","last_fullsized":"2024-09-10T19:08:39.426456+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'23","last_scrub_stamp":"2024-09-10T19:06:26.733241+0000","last_deep_scrub":"95'17","last_deep_scrub_stamp":"2024-09-10T18:50:36.476884+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.733241+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050063570000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":322,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417154+0000","last_change":"2024-09-10T19:06:30.049476+0000","last_active":"2024-09-10T19:08:39.417154+0000","last_peered":"2024-09-10T19:08:39.417154+0000","last_clean":"2024-09-10T19:08:39.417154+0000","last_became_active":"2024-09-10T19:02:52.256764+0000","last_became_peered":"2024-09-10T19:02:52.256764+0000","last_unstale":"2024-09-10T19:08:39.417154+0000","last_undegraded":"2024-09-10T19:08:39.417154+0000","last_fullsized":"2024-09-10T19:08:39.417154+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.049307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.049307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039529120000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":851,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.415318+0000","last_change":"2024-09-10T19:07:36.403126+0000","last_active":"2024-09-10T19:08:39.415318+0000","last_peered":"2024-09-10T19:08:39.415318+0000","last_clean":"2024-09-10T19:07:34.938683+0000","last_became_active":"2024-09-10T19:07:36.403126+0000","last_became_peered":"2024-09-10T19:07:36.403126+0000","last_unstale":"2024-09-10T19:08:39.415318+0000","last_undegraded":"2024-09-10T19:08:39.415318+0000","last_fullsized":"2024-09-10T19:07:36.374701+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.340065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.340065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004677952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":761,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.417024+0000","last_change":"2024-09-10T19:07:36.389933+0000","last_active":"2024-09-10T19:08:39.417024+0000","last_peered":"2024-09-10T19:08:39.417024+0000","last_clean":"2024-09-10T19:07:34.369603+0000","last_became_active":"2024-09-10T19:07:36.389933+0000","last_became_peered":"2024-09-10T19:07:36.389933+0000","last_unstale":"2024-09-10T19:08:39.417024+0000","last_undegraded":"2024-09-10T19:08:39.417024+0000","last_fullsized":"2024-09-10T19:07:36.380790+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.080362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.080362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028902760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":366,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422262+0000","last_change":"2024-09-10T19:06:58.888376+0000","last_active":"2024-09-10T19:08:39.422262+0000","last_peered":"2024-09-10T19:08:39.422262+0000","last_clean":"2024-09-10T19:08:39.422262+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:08:39.422262+0000","last_undegraded":"2024-09-10T19:08:39.422262+0000","last_fullsized":"2024-09-10T19:08:39.422262+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:58.888195+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:44.683992+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.888195+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043236180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"410'35","reported_seq":912,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.441383+0000","last_change":"2024-09-10T19:07:36.398898+0000","last_active":"2024-09-10T19:08:39.441383+0000","last_peered":"2024-09-10T19:08:39.441383+0000","last_clean":"2024-09-10T19:07:34.367578+0000","last_became_active":"2024-09-10T19:07:36.398898+0000","last_became_peered":"2024-09-10T19:07:36.398898+0000","last_unstale":"2024-09-10T19:08:39.441383+0000","last_undegraded":"2024-09-10T19:07:36.383449+0000","last_fullsized":"2024-09-10T19:07:36.383273+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'35","last_scrub_stamp":"2024-09-10T19:07:01.718013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.718013+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078103089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":9}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":855,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.433347+0000","last_change":"2024-09-10T19:07:56.997220+0000","last_active":"2024-09-10T19:08:39.433347+0000","last_peered":"2024-09-10T19:08:39.433347+0000","last_clean":"2024-09-10T19:08:39.433347+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:08:39.433347+0000","last_undegraded":"2024-09-10T19:08:39.433347+0000","last_fullsized":"2024-09-10T19:08:39.433347+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:56.997064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:07:56.997064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:16.566191+0000","scrub_duration":0.0037155790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"791'14","reported_seq":457,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.430028+0000","last_change":"2024-09-10T19:07:36.394982+0000","last_active":"2024-09-10T19:08:39.430028+0000","last_peered":"2024-09-10T19:08:39.430028+0000","last_clean":"2024-09-10T19:07:34.374069+0000","last_became_active":"2024-09-10T19:07:36.394982+0000","last_became_peered":"2024-09-10T19:07:36.394982+0000","last_unstale":"2024-09-10T19:08:39.430028+0000","last_undegraded":"2024-09-10T19:07:36.386044+0000","last_fullsized":"2024-09-10T19:07:36.385901+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_deep_scrub":"548'13","last_deep_scrub_stamp":"2024-09-10T19:05:24.946829+0000","last_clean_scrub_stamp":"2024-09-10T19:05:24.946829+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036489069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":868,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430882+0000","last_change":"2024-09-10T19:07:50.175313+0000","last_active":"2024-09-10T19:08:39.430882+0000","last_peered":"2024-09-10T19:08:39.430882+0000","last_clean":"2024-09-10T19:08:39.430882+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:08:39.430882+0000","last_undegraded":"2024-09-10T19:08:39.430882+0000","last_fullsized":"2024-09-10T19:08:39.430882+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:50.175145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.907056+0000","last_clean_scrub_stamp":"2024-09-10T19:07:50.175145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:19.842911+0000","scrub_duration":0.0029091920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":766,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.439440+0000","last_change":"2024-09-10T19:07:36.400873+0000","last_active":"2024-09-10T19:08:39.439440+0000","last_peered":"2024-09-10T19:08:39.439440+0000","last_clean":"2024-09-10T19:07:34.368051+0000","last_became_active":"2024-09-10T19:07:36.400873+0000","last_became_peered":"2024-09-10T19:07:36.400873+0000","last_unstale":"2024-09-10T19:08:39.439440+0000","last_undegraded":"2024-09-10T19:08:39.439440+0000","last_fullsized":"2024-09-10T19:07:36.386176+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:36.747814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:06:36.747814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036046889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":927,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418395+0000","last_change":"2024-09-10T19:08:05.886908+0000","last_active":"2024-09-10T19:08:39.418395+0000","last_peered":"2024-09-10T19:08:39.418395+0000","last_clean":"2024-09-10T19:08:39.418395+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:08:39.418395+0000","last_undegraded":"2024-09-10T19:08:39.418395+0000","last_fullsized":"2024-09-10T19:08:39.418395+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:05.886739+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:08:05.886739+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:16.294941+0000","scrub_duration":0.0038160569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"410'19","reported_seq":883,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419524+0000","last_change":"2024-09-10T19:08:09.131340+0000","last_active":"2024-09-10T19:08:39.419524+0000","last_peered":"2024-09-10T19:08:39.419524+0000","last_clean":"2024-09-10T19:08:39.419524+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:08:39.419524+0000","last_undegraded":"2024-09-10T19:08:39.419524+0000","last_fullsized":"2024-09-10T19:08:39.419524+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:08:09.131167+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:08:09.131167+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:26.157658+0000","scrub_duration":0.0049887919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.416811+0000","last_change":"2024-09-10T19:07:36.388618+0000","last_active":"2024-09-10T19:08:39.416811+0000","last_peered":"2024-09-10T19:08:39.416811+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.388618+0000","last_became_peered":"2024-09-10T19:07:36.388618+0000","last_unstale":"2024-09-10T19:08:39.416811+0000","last_undegraded":"2024-09-10T19:08:39.416811+0000","last_fullsized":"2024-09-10T19:07:36.379722+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:38.068517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:06:38.068517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":322,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418942+0000","last_change":"2024-09-10T19:07:46.077081+0000","last_active":"2024-09-10T19:08:39.418942+0000","last_peered":"2024-09-10T19:08:39.418942+0000","last_clean":"2024-09-10T19:08:39.418942+0000","last_became_active":"2024-09-10T19:02:52.254392+0000","last_became_peered":"2024-09-10T19:02:52.254392+0000","last_unstale":"2024-09-10T19:08:39.418942+0000","last_undegraded":"2024-09-10T19:08:39.418942+0000","last_fullsized":"2024-09-10T19:08:39.418942+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:46.077003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:07:46.077003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.678339+0000","scrub_duration":0.003049618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":58,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.430290+0000","last_change":"2024-09-10T19:07:36.394915+0000","last_active":"2024-09-10T19:08:39.430290+0000","last_peered":"2024-09-10T19:08:39.430290+0000","last_clean":"2024-09-10T18:49:23.022168+0000","last_became_active":"2024-09-10T19:07:36.394915+0000","last_became_peered":"2024-09-10T19:07:36.394915+0000","last_unstale":"2024-09-10T19:08:39.430290+0000","last_undegraded":"2024-09-10T19:08:39.430290+0000","last_fullsized":"2024-09-10T19:07:36.386506+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.119813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.119813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":766,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.438614+0000","last_change":"2024-09-10T19:07:36.400035+0000","last_active":"2024-09-10T19:08:39.438614+0000","last_peered":"2024-09-10T19:08:39.438614+0000","last_clean":"2024-09-10T19:07:34.367656+0000","last_became_active":"2024-09-10T19:07:36.400035+0000","last_became_peered":"2024-09-10T19:07:36.400035+0000","last_unstale":"2024-09-10T19:08:39.438614+0000","last_undegraded":"2024-09-10T19:08:39.438614+0000","last_fullsized":"2024-09-10T19:07:36.380175+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.837830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.837830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098916300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":359,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422953+0000","last_change":"2024-09-10T19:08:14.758532+0000","last_active":"2024-09-10T19:08:39.422953+0000","last_peered":"2024-09-10T19:08:39.422953+0000","last_clean":"2024-09-10T19:08:39.422953+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:08:39.422953+0000","last_undegraded":"2024-09-10T19:08:39.422953+0000","last_fullsized":"2024-09-10T19:08:39.422953+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:14.758272+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:08:14.758272+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:42.321726+0000","scrub_duration":0.0060293760000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"410'27","reported_seq":894,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.421279+0000","last_change":"2024-09-10T19:07:36.389915+0000","last_active":"2024-09-10T19:08:39.421279+0000","last_peered":"2024-09-10T19:08:39.421279+0000","last_clean":"2024-09-10T19:07:34.374468+0000","last_became_active":"2024-09-10T19:07:36.389915+0000","last_became_peered":"2024-09-10T19:07:36.389915+0000","last_unstale":"2024-09-10T19:08:39.421279+0000","last_undegraded":"2024-09-10T19:07:36.379679+0000","last_fullsized":"2024-09-10T19:07:36.379545+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'27","last_scrub_stamp":"2024-09-10T19:05:33.935997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.935997+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050458739999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.419565+0000","last_change":"2024-09-10T19:07:36.394307+0000","last_active":"2024-09-10T19:08:39.419565+0000","last_peered":"2024-09-10T19:08:39.419565+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.394307+0000","last_became_peered":"2024-09-10T19:07:36.394307+0000","last_unstale":"2024-09-10T19:08:39.419565+0000","last_undegraded":"2024-09-10T19:08:39.419565+0000","last_fullsized":"2024-09-10T19:07:36.385657+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:40.095535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:40.095535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.421287+0000","last_change":"2024-09-10T19:07:36.407638+0000","last_active":"2024-09-10T19:08:39.421287+0000","last_peered":"2024-09-10T19:08:39.421287+0000","last_clean":"2024-09-10T18:49:21.015454+0000","last_became_active":"2024-09-10T19:07:36.407638+0000","last_became_peered":"2024-09-10T19:07:36.407638+0000","last_unstale":"2024-09-10T19:08:39.421287+0000","last_undegraded":"2024-09-10T19:08:39.421287+0000","last_fullsized":"2024-09-10T19:07:36.386574+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:59.211830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.211830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.425800+0000","last_change":"2024-09-10T19:07:36.406907+0000","last_active":"2024-09-10T19:08:39.425800+0000","last_peered":"2024-09-10T19:08:39.425800+0000","last_clean":"2024-09-10T19:07:34.941352+0000","last_became_active":"2024-09-10T19:07:36.406907+0000","last_became_peered":"2024-09-10T19:07:36.406907+0000","last_unstale":"2024-09-10T19:08:39.425800+0000","last_undegraded":"2024-09-10T19:08:39.425800+0000","last_fullsized":"2024-09-10T19:07:36.390919+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:39.931027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:50.674821+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.931027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039840149999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.435111+0000","last_change":"2024-09-10T19:07:36.396552+0000","last_active":"2024-09-10T19:08:39.435111+0000","last_peered":"2024-09-10T19:08:39.435111+0000","last_clean":"2024-09-10T18:54:31.683713+0000","last_became_active":"2024-09-10T19:07:36.396552+0000","last_became_peered":"2024-09-10T19:07:36.396552+0000","last_unstale":"2024-09-10T19:08:39.435111+0000","last_undegraded":"2024-09-10T19:08:39.435111+0000","last_fullsized":"2024-09-10T19:07:36.390174+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:27.078734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:27.078734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"409'23","reported_seq":905,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430328+0000","last_change":"2024-09-10T19:08:00.438372+0000","last_active":"2024-09-10T19:08:39.430328+0000","last_peered":"2024-09-10T19:08:39.430328+0000","last_clean":"2024-09-10T19:08:39.430328+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:08:39.430328+0000","last_undegraded":"2024-09-10T19:08:39.430328+0000","last_fullsized":"2024-09-10T19:08:39.430328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'23","last_scrub_stamp":"2024-09-10T19:08:00.438165+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T19:08:00.438165+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:02.152228+0000","scrub_duration":0.0051134609999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":856,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.436725+0000","last_change":"2024-09-10T19:07:36.396907+0000","last_active":"2024-09-10T19:08:39.436725+0000","last_peered":"2024-09-10T19:08:39.436725+0000","last_clean":"2024-09-10T19:07:34.446324+0000","last_became_active":"2024-09-10T19:07:36.396907+0000","last_became_peered":"2024-09-10T19:07:36.396907+0000","last_unstale":"2024-09-10T19:08:39.436725+0000","last_undegraded":"2024-09-10T19:08:39.436725+0000","last_fullsized":"2024-09-10T19:07:36.384433+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.605072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.605072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072968429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"791'14","reported_seq":448,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.440554+0000","last_change":"2024-09-10T19:07:36.405119+0000","last_active":"2024-09-10T19:08:39.440554+0000","last_peered":"2024-09-10T19:08:39.440554+0000","last_clean":"2024-09-10T19:07:34.368300+0000","last_became_active":"2024-09-10T19:07:36.405119+0000","last_became_peered":"2024-09-10T19:07:36.405119+0000","last_unstale":"2024-09-10T19:08:39.440554+0000","last_undegraded":"2024-09-10T19:07:36.387834+0000","last_fullsized":"2024-09-10T19:07:36.387708+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'13","last_scrub_stamp":"2024-09-10T19:05:04.477051+0000","last_deep_scrub":"95'12","last_deep_scrub_stamp":"2024-09-10T18:50:27.317817+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.477051+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031221360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":841,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418723+0000","last_change":"2024-09-10T19:06:22.027150+0000","last_active":"2024-09-10T19:08:39.418723+0000","last_peered":"2024-09-10T19:08:39.418723+0000","last_clean":"2024-09-10T19:08:39.418723+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:08:39.418723+0000","last_undegraded":"2024-09-10T19:08:39.418723+0000","last_fullsized":"2024-09-10T19:08:39.418723+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:06:22.027051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.027051+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003390789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.422813+0000","last_change":"2024-09-10T19:07:36.396451+0000","last_active":"2024-09-10T19:08:39.422813+0000","last_peered":"2024-09-10T19:08:39.422813+0000","last_clean":"2024-09-10T18:54:31.683713+0000","last_became_active":"2024-09-10T19:07:36.396451+0000","last_became_peered":"2024-09-10T19:07:36.396451+0000","last_unstale":"2024-09-10T19:08:39.422813+0000","last_undegraded":"2024-09-10T19:08:39.422813+0000","last_fullsized":"2024-09-10T19:07:36.382938+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:04:59.013632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:04:59.013632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":361,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418684+0000","last_change":"2024-09-10T19:06:42.135170+0000","last_active":"2024-09-10T19:08:39.418684+0000","last_peered":"2024-09-10T19:08:39.418684+0000","last_clean":"2024-09-10T19:08:39.418684+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:08:39.418684+0000","last_undegraded":"2024-09-10T19:08:39.418684+0000","last_fullsized":"2024-09-10T19:08:39.418684+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:42.135006+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.135006+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049906150000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"410'24","reported_seq":881,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.421467+0000","last_change":"2024-09-10T19:07:36.389098+0000","last_active":"2024-09-10T19:08:39.421467+0000","last_peered":"2024-09-10T19:08:39.421467+0000","last_clean":"2024-09-10T19:07:34.940859+0000","last_became_active":"2024-09-10T19:07:36.389098+0000","last_became_peered":"2024-09-10T19:07:36.389098+0000","last_unstale":"2024-09-10T19:08:39.421467+0000","last_undegraded":"2024-09-10T19:07:36.376693+0000","last_fullsized":"2024-09-10T19:07:36.376406+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:34.355514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.355514+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005673483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.443368+0000","last_change":"2024-09-10T19:08:03.690337+0000","last_active":"2024-09-10T19:08:39.443368+0000","last_peered":"2024-09-10T19:08:39.443368+0000","last_clean":"2024-09-10T19:08:39.443368+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:08:39.443368+0000","last_undegraded":"2024-09-10T19:08:39.443368+0000","last_fullsized":"2024-09-10T19:08:39.443368+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:03.690228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:08:03.690228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:16.282002+0000","scrub_duration":0.0020939040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":306,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.423271+0000","last_change":"2024-09-10T19:05:20.129549+0000","last_active":"2024-09-10T19:08:39.423271+0000","last_peered":"2024-09-10T19:08:39.423271+0000","last_clean":"2024-09-10T19:08:39.423271+0000","last_became_active":"2024-09-10T19:02:52.254942+0000","last_became_peered":"2024-09-10T19:02:52.254942+0000","last_unstale":"2024-09-10T19:08:39.423271+0000","last_undegraded":"2024-09-10T19:08:39.423271+0000","last_fullsized":"2024-09-10T19:08:39.423271+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:20.129398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:05:20.129398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033769329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"87'5","reported_seq":1993,"reported_epoch":837,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:40.426275+0000","last_change":"2024-09-10T19:07:36.386681+0000","last_active":"2024-09-10T19:08:40.426275+0000","last_peered":"2024-09-10T19:08:40.426275+0000","last_clean":"2024-09-10T19:07:35.091956+0000","last_became_active":"2024-09-10T19:07:36.386681+0000","last_became_peered":"2024-09-10T19:07:36.386681+0000","last_unstale":"2024-09-10T19:08:40.426275+0000","last_undegraded":"2024-09-10T19:07:36.375734+0000","last_fullsized":"2024-09-10T19:07:36.375365+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'5","last_scrub_stamp":"2024-09-10T19:05:09.941698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:09.941698+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041460869999999997,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1135,"num_read_kb":1130,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432941+0000","last_change":"2024-09-10T19:07:50.956434+0000","last_active":"2024-09-10T19:08:39.432941+0000","last_peered":"2024-09-10T19:08:39.432941+0000","last_clean":"2024-09-10T19:08:39.432941+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:08:39.432941+0000","last_undegraded":"2024-09-10T19:08:39.432941+0000","last_fullsized":"2024-09-10T19:08:39.432941+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:50.956249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:07:50.956249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:09.189993+0000","scrub_duration":0.0038448480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":790,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424197+0000","last_change":"2024-09-10T19:07:52.142872+0000","last_active":"2024-09-10T19:08:39.424197+0000","last_peered":"2024-09-10T19:08:39.424197+0000","last_clean":"2024-09-10T19:08:39.424197+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:08:39.424197+0000","last_undegraded":"2024-09-10T19:08:39.424197+0000","last_fullsized":"2024-09-10T19:08:39.424197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:52.142740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:07:52.142740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.131887+0000","scrub_duration":0.0036781520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":371,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424833+0000","last_change":"2024-09-10T19:08:08.529260+0000","last_active":"2024-09-10T19:08:39.424833+0000","last_peered":"2024-09-10T19:08:39.424833+0000","last_clean":"2024-09-10T19:08:39.424833+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:08:39.424833+0000","last_undegraded":"2024-09-10T19:08:39.424833+0000","last_fullsized":"2024-09-10T19:08:39.424833+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:08.529152+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_clean_scrub_stamp":"2024-09-10T19:08:08.529152+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:20.611442+0000","scrub_duration":0.0029299539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"410'17","reported_seq":887,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421768+0000","last_change":"2024-09-10T19:07:50.086461+0000","last_active":"2024-09-10T19:08:39.421768+0000","last_peered":"2024-09-10T19:08:39.421768+0000","last_clean":"2024-09-10T19:08:39.421768+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:08:39.421768+0000","last_undegraded":"2024-09-10T19:08:39.421768+0000","last_fullsized":"2024-09-10T19:08:39.421768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'17","last_scrub_stamp":"2024-09-10T19:07:50.086292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:07:50.086292+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:50.840336+0000","scrub_duration":0.0045139730000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":54,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.440726+0000","last_change":"2024-09-10T19:07:36.406301+0000","last_active":"2024-09-10T19:08:39.440726+0000","last_peered":"2024-09-10T19:08:39.440726+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.406301+0000","last_became_peered":"2024-09-10T19:07:36.406301+0000","last_unstale":"2024-09-10T19:08:39.440726+0000","last_undegraded":"2024-09-10T19:08:39.440726+0000","last_fullsized":"2024-09-10T19:07:36.389312+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.127911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.127911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":1075,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420979+0000","last_change":"2024-09-10T19:05:32.921313+0000","last_active":"2024-09-10T19:08:39.420979+0000","last_peered":"2024-09-10T19:08:39.420979+0000","last_clean":"2024-09-10T19:08:39.420979+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:08:39.420979+0000","last_undegraded":"2024-09-10T19:08:39.420979+0000","last_fullsized":"2024-09-10T19:08:39.420979+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:05:32.921124+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:05:32.921124+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004136547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":873,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422207+0000","last_change":"2024-09-10T19:07:50.505821+0000","last_active":"2024-09-10T19:08:39.422207+0000","last_peered":"2024-09-10T19:08:39.422207+0000","last_clean":"2024-09-10T19:08:39.422207+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:08:39.422207+0000","last_undegraded":"2024-09-10T19:08:39.422207+0000","last_fullsized":"2024-09-10T19:08:39.422207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:07:50.505662+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:07:50.505662+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:01.195289+0000","scrub_duration":0.0040820309999999999,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":322,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417605+0000","last_change":"2024-09-10T19:06:23.076714+0000","last_active":"2024-09-10T19:08:39.417605+0000","last_peered":"2024-09-10T19:08:39.417605+0000","last_clean":"2024-09-10T19:08:39.417605+0000","last_became_active":"2024-09-10T19:02:52.256316+0000","last_became_peered":"2024-09-10T19:02:52.256316+0000","last_unstale":"2024-09-10T19:08:39.417605+0000","last_undegraded":"2024-09-10T19:08:39.417605+0000","last_fullsized":"2024-09-10T19:08:39.417605+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:23.076562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.076562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034726789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":761,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420897+0000","last_change":"2024-09-10T19:06:48.197543+0000","last_active":"2024-09-10T19:08:39.420897+0000","last_peered":"2024-09-10T19:08:39.420897+0000","last_clean":"2024-09-10T19:08:39.420897+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:08:39.420897+0000","last_undegraded":"2024-09-10T19:08:39.420897+0000","last_fullsized":"2024-09-10T19:08:39.420897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.197394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.197394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003369632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":435,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.436357+0000","last_change":"2024-09-10T19:07:45.739684+0000","last_active":"2024-09-10T19:08:39.436357+0000","last_peered":"2024-09-10T19:08:39.436357+0000","last_clean":"2024-09-10T19:08:39.436357+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:08:39.436357+0000","last_undegraded":"2024-09-10T19:08:39.436357+0000","last_fullsized":"2024-09-10T19:08:39.436357+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:07:45.739532+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:07:45.739532+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:52.613888+0000","scrub_duration":0.0071414440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"791'53","reported_seq":660,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.425846+0000","last_change":"2024-09-10T19:07:36.399232+0000","last_active":"2024-09-10T19:08:39.425846+0000","last_peered":"2024-09-10T19:08:39.425846+0000","last_clean":"2024-09-10T18:59:26.377355+0000","last_became_active":"2024-09-10T19:07:36.399232+0000","last_became_peered":"2024-09-10T19:07:36.399232+0000","last_unstale":"2024-09-10T19:08:39.425846+0000","last_undegraded":"2024-09-10T19:07:36.384853+0000","last_fullsized":"2024-09-10T19:07:36.384742+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'52","last_scrub_stamp":"2024-09-10T19:05:27.874075+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:05:27.874075+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057072520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":68,"num_read_kb":53,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":849,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432661+0000","last_change":"2024-09-10T19:06:35.858337+0000","last_active":"2024-09-10T19:08:39.432661+0000","last_peered":"2024-09-10T19:08:39.432661+0000","last_clean":"2024-09-10T19:08:39.432661+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:08:39.432661+0000","last_undegraded":"2024-09-10T19:08:39.432661+0000","last_fullsized":"2024-09-10T19:08:39.432661+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.858069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.858069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032278900000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":1079,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420941+0000","last_change":"2024-09-10T19:05:25.279549+0000","last_active":"2024-09-10T19:08:39.420941+0000","last_peered":"2024-09-10T19:08:39.420941+0000","last_clean":"2024-09-10T19:08:39.420941+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:08:39.420941+0000","last_undegraded":"2024-09-10T19:08:39.420941+0000","last_fullsized":"2024-09-10T19:08:39.420941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:05:25.279409+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:05:25.279409+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036183930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":856,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.436362+0000","last_change":"2024-09-10T19:07:59.745141+0000","last_active":"2024-09-10T19:08:39.436362+0000","last_peered":"2024-09-10T19:08:39.436362+0000","last_clean":"2024-09-10T19:08:39.436362+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:08:39.436362+0000","last_undegraded":"2024-09-10T19:08:39.436362+0000","last_fullsized":"2024-09-10T19:08:39.436362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:07:59.744970+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:07:59.744970+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:08.265804+0000","scrub_duration":0.0042378249999999998,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":54,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.424225+0000","last_change":"2024-09-10T19:07:36.394151+0000","last_active":"2024-09-10T19:08:39.424225+0000","last_peered":"2024-09-10T19:08:39.424225+0000","last_clean":"2024-09-10T18:49:23.022168+0000","last_became_active":"2024-09-10T19:07:36.394151+0000","last_became_peered":"2024-09-10T19:07:36.394151+0000","last_unstale":"2024-09-10T19:08:39.424225+0000","last_undegraded":"2024-09-10T19:08:39.424225+0000","last_fullsized":"2024-09-10T19:07:36.380490+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.081613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.081613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":761,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419879+0000","last_change":"2024-09-10T19:06:17.163823+0000","last_active":"2024-09-10T19:08:39.419879+0000","last_peered":"2024-09-10T19:08:39.419879+0000","last_clean":"2024-09-10T19:08:39.419879+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:08:39.419879+0000","last_undegraded":"2024-09-10T19:08:39.419879+0000","last_fullsized":"2024-09-10T19:08:39.419879+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.163656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.163656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035671100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":438,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.440649+0000","last_change":"2024-09-10T19:08:16.673860+0000","last_active":"2024-09-10T19:08:39.440649+0000","last_peered":"2024-09-10T19:08:39.440649+0000","last_clean":"2024-09-10T19:08:39.440649+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:08:39.440649+0000","last_undegraded":"2024-09-10T19:08:39.440649+0000","last_fullsized":"2024-09-10T19:08:39.440649+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:16.673796+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:08:16.673796+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:35.842683+0000","scrub_duration":0.0065334770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"760'33","reported_seq":897,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.426556+0000","last_change":"2024-09-10T19:06:59.942589+0000","last_active":"2024-09-10T19:08:39.426556+0000","last_peered":"2024-09-10T19:08:39.426556+0000","last_clean":"2024-09-10T19:08:39.426556+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:08:39.426556+0000","last_undegraded":"2024-09-10T19:08:39.426556+0000","last_fullsized":"2024-09-10T19:08:39.426556+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"760'33","last_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_deep_scrub":"760'33","last_deep_scrub_stamp":"2024-09-10T19:06:59.942479+0000","last_clean_scrub_stamp":"2024-09-10T19:06:59.942479+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012061568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.420638+0000","last_change":"2024-09-10T19:07:36.399402+0000","last_active":"2024-09-10T19:08:39.420638+0000","last_peered":"2024-09-10T19:08:39.420638+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.399402+0000","last_became_peered":"2024-09-10T19:07:36.399402+0000","last_unstale":"2024-09-10T19:08:39.420638+0000","last_undegraded":"2024-09-10T19:08:39.420638+0000","last_fullsized":"2024-09-10T19:07:36.385278+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.032276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.032276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.423744+0000","last_change":"2024-09-10T19:07:36.402579+0000","last_active":"2024-09-10T19:08:39.423744+0000","last_peered":"2024-09-10T19:08:39.423744+0000","last_clean":"2024-09-10T18:49:21.015454+0000","last_became_active":"2024-09-10T19:07:36.402579+0000","last_became_peered":"2024-09-10T19:07:36.402579+0000","last_unstale":"2024-09-10T19:08:39.423744+0000","last_undegraded":"2024-09-10T19:08:39.423744+0000","last_fullsized":"2024-09-10T19:07:36.384161+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.204933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.204933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":844,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.415745+0000","last_change":"2024-09-10T19:07:56.352783+0000","last_active":"2024-09-10T19:08:39.415745+0000","last_peered":"2024-09-10T19:08:39.415745+0000","last_clean":"2024-09-10T19:08:39.415745+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:08:39.415745+0000","last_undegraded":"2024-09-10T19:08:39.415745+0000","last_fullsized":"2024-09-10T19:08:39.415745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:56.352635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:07:56.352635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:58.202927+0000","scrub_duration":0.0033279910000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":790,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.425865+0000","last_change":"2024-09-10T19:07:51.175909+0000","last_active":"2024-09-10T19:08:39.425865+0000","last_peered":"2024-09-10T19:08:39.425865+0000","last_clean":"2024-09-10T19:08:39.425865+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:08:39.425865+0000","last_undegraded":"2024-09-10T19:08:39.425865+0000","last_fullsized":"2024-09-10T19:08:39.425865+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:51.175658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:07:51.175658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:12.143388+0000","scrub_duration":0.0039666110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":359,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422745+0000","last_change":"2024-09-10T19:06:53.016161+0000","last_active":"2024-09-10T19:08:39.422745+0000","last_peered":"2024-09-10T19:08:39.422745+0000","last_clean":"2024-09-10T19:08:39.422745+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:08:39.422745+0000","last_undegraded":"2024-09-10T19:08:39.422745+0000","last_fullsized":"2024-09-10T19:08:39.422745+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:53.015994+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.015994+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033683329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"410'29","reported_seq":766,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420985+0000","last_change":"2024-09-10T19:06:39.394175+0000","last_active":"2024-09-10T19:08:39.420985+0000","last_peered":"2024-09-10T19:08:39.420985+0000","last_clean":"2024-09-10T19:08:39.420985+0000","last_became_active":"2024-09-10T19:02:52.256210+0000","last_became_peered":"2024-09-10T19:02:52.256210+0000","last_unstale":"2024-09-10T19:08:39.420985+0000","last_undegraded":"2024-09-10T19:08:39.420985+0000","last_fullsized":"2024-09-10T19:08:39.420985+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'29","last_scrub_stamp":"2024-09-10T19:06:39.394055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:39.394055+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034977820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":849,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.423976+0000","last_change":"2024-09-10T19:07:36.405213+0000","last_active":"2024-09-10T19:08:39.423976+0000","last_peered":"2024-09-10T19:08:39.423976+0000","last_clean":"2024-09-10T19:07:34.941183+0000","last_became_active":"2024-09-10T19:07:36.405213+0000","last_became_peered":"2024-09-10T19:07:36.405213+0000","last_unstale":"2024-09-10T19:08:39.423976+0000","last_undegraded":"2024-09-10T19:08:39.423976+0000","last_fullsized":"2024-09-10T19:07:36.396405+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.136755+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.136755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032446879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":54,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.425505+0000","last_change":"2024-09-10T19:07:36.390361+0000","last_active":"2024-09-10T19:08:39.425505+0000","last_peered":"2024-09-10T19:08:39.425505+0000","last_clean":"2024-09-10T18:49:21.015454+0000","last_became_active":"2024-09-10T19:07:36.390361+0000","last_became_peered":"2024-09-10T19:07:36.390361+0000","last_unstale":"2024-09-10T19:08:39.425505+0000","last_undegraded":"2024-09-10T19:08:39.425505+0000","last_fullsized":"2024-09-10T19:07:36.383217+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:19.902898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:19.902898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":843,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417918+0000","last_change":"2024-09-10T19:07:49.127707+0000","last_active":"2024-09-10T19:08:39.417918+0000","last_peered":"2024-09-10T19:08:39.417918+0000","last_clean":"2024-09-10T19:08:39.417918+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:08:39.417918+0000","last_undegraded":"2024-09-10T19:08:39.417918+0000","last_fullsized":"2024-09-10T19:08:39.417918+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:49.127554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:07:49.127554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:18.632591+0000","scrub_duration":0.0041833599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":755,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419960+0000","last_change":"2024-09-10T19:06:26.273000+0000","last_active":"2024-09-10T19:08:39.419960+0000","last_peered":"2024-09-10T19:08:39.419960+0000","last_clean":"2024-09-10T19:08:39.419960+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:08:39.419960+0000","last_undegraded":"2024-09-10T19:08:39.419960+0000","last_fullsized":"2024-09-10T19:08:39.419960+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:26.272837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:26.272837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046824379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":927,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419330+0000","last_change":"2024-09-10T19:08:09.925670+0000","last_active":"2024-09-10T19:08:39.419330+0000","last_peered":"2024-09-10T19:08:39.419330+0000","last_clean":"2024-09-10T19:08:39.419330+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:08:39.419330+0000","last_undegraded":"2024-09-10T19:08:39.419330+0000","last_fullsized":"2024-09-10T19:08:39.419330+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:09.925580+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:08:09.925580+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:39.523576+0000","scrub_duration":0.0030219410000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"410'38","reported_seq":504,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.421250+0000","last_change":"2024-09-10T19:07:36.390251+0000","last_active":"2024-09-10T19:08:39.421250+0000","last_peered":"2024-09-10T19:08:39.421250+0000","last_clean":"2024-09-10T18:59:26.329100+0000","last_became_active":"2024-09-10T19:07:36.390251+0000","last_became_peered":"2024-09-10T19:07:36.390251+0000","last_unstale":"2024-09-10T19:08:39.421250+0000","last_undegraded":"2024-09-10T19:07:36.382887+0000","last_fullsized":"2024-09-10T19:07:36.382717+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'38","last_scrub_stamp":"2024-09-10T19:07:01.193206+0000","last_deep_scrub":"410'38","last_deep_scrub_stamp":"2024-09-10T19:00:13.487819+0000","last_clean_scrub_stamp":"2024-09-10T19:07:01.193206+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036290649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":884,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424017+0000","last_change":"2024-09-10T19:08:21.069373+0000","last_active":"2024-09-10T19:08:39.424017+0000","last_peered":"2024-09-10T19:08:39.424017+0000","last_clean":"2024-09-10T19:08:39.424017+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:08:39.424017+0000","last_undegraded":"2024-09-10T19:08:39.424017+0000","last_fullsized":"2024-09-10T19:08:39.424017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:21.069217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:08:21.069217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:27.666197+0000","scrub_duration":0.0035598930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.430835+0000","last_change":"2024-09-10T19:07:36.406632+0000","last_active":"2024-09-10T19:08:39.430835+0000","last_peered":"2024-09-10T19:08:39.430835+0000","last_clean":"2024-09-10T18:49:21.015454+0000","last_became_active":"2024-09-10T19:07:36.406632+0000","last_became_peered":"2024-09-10T19:07:36.406632+0000","last_unstale":"2024-09-10T19:08:39.430835+0000","last_undegraded":"2024-09-10T19:08:39.430835+0000","last_fullsized":"2024-09-10T19:07:36.391018+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.222842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:12.925817+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.222842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432173+0000","last_change":"2024-09-10T19:07:55.039670+0000","last_active":"2024-09-10T19:08:39.432173+0000","last_peered":"2024-09-10T19:08:39.432173+0000","last_clean":"2024-09-10T19:08:39.432173+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:08:39.432173+0000","last_undegraded":"2024-09-10T19:08:39.432173+0000","last_fullsized":"2024-09-10T19:08:39.432173+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:55.039508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:07:55.039508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:03.636831+0000","scrub_duration":0.0037455549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":772,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.442252+0000","last_change":"2024-09-10T19:08:20.617119+0000","last_active":"2024-09-10T19:08:39.442252+0000","last_peered":"2024-09-10T19:08:39.442252+0000","last_clean":"2024-09-10T19:08:39.442252+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:08:39.442252+0000","last_undegraded":"2024-09-10T19:08:39.442252+0000","last_fullsized":"2024-09-10T19:08:39.442252+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:20.617025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:08:20.617025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:33.416232+0000","scrub_duration":0.007498691,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":130,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.429825+0000","last_change":"2024-09-10T19:07:36.398844+0000","last_active":"2024-09-10T19:08:39.429825+0000","last_peered":"2024-09-10T19:08:39.429825+0000","last_clean":"2024-09-10T18:50:16.551448+0000","last_became_active":"2024-09-10T19:07:36.398844+0000","last_became_peered":"2024-09-10T19:07:36.398844+0000","last_unstale":"2024-09-10T19:08:39.429825+0000","last_undegraded":"2024-09-10T19:08:39.429825+0000","last_fullsized":"2024-09-10T19:07:36.382054+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:06:25.036490+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:06:25.036490+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066991719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"410'28","reported_seq":890,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418346+0000","last_change":"2024-09-10T19:08:04.166541+0000","last_active":"2024-09-10T19:08:39.418346+0000","last_peered":"2024-09-10T19:08:39.418346+0000","last_clean":"2024-09-10T19:08:39.418346+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:08:39.418346+0000","last_undegraded":"2024-09-10T19:08:39.418346+0000","last_fullsized":"2024-09-10T19:08:39.418346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'28","last_scrub_stamp":"2024-09-10T19:08:04.166176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:08:04.166176+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.230091+0000","scrub_duration":0.0041306909999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":856,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.436195+0000","last_change":"2024-09-10T19:07:36.396606+0000","last_active":"2024-09-10T19:08:39.436195+0000","last_peered":"2024-09-10T19:08:39.436195+0000","last_clean":"2024-09-10T19:07:34.445564+0000","last_became_active":"2024-09-10T19:07:36.396606+0000","last_became_peered":"2024-09-10T19:07:36.396606+0000","last_unstale":"2024-09-10T19:08:39.436195+0000","last_undegraded":"2024-09-10T19:08:39.436195+0000","last_fullsized":"2024-09-10T19:07:36.383652+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:04.757950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:05:04.757950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032582150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.421501+0000","last_change":"2024-09-10T19:07:36.383447+0000","last_active":"2024-09-10T19:08:39.421501+0000","last_peered":"2024-09-10T19:08:39.421501+0000","last_clean":"2024-09-10T19:07:34.940970+0000","last_became_active":"2024-09-10T19:07:36.383447+0000","last_became_peered":"2024-09-10T19:07:36.383447+0000","last_unstale":"2024-09-10T19:08:39.421501+0000","last_undegraded":"2024-09-10T19:08:39.421501+0000","last_fullsized":"2024-09-10T19:07:36.377294+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:10.481693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:10.481693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037720449999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.426204+0000","last_change":"2024-09-10T19:07:36.405790+0000","last_active":"2024-09-10T19:08:39.426204+0000","last_peered":"2024-09-10T19:08:39.426204+0000","last_clean":"2024-09-10T18:49:23.022168+0000","last_became_active":"2024-09-10T19:07:36.405790+0000","last_became_peered":"2024-09-10T19:07:36.405790+0000","last_unstale":"2024-09-10T19:08:39.426204+0000","last_undegraded":"2024-09-10T19:08:39.426204+0000","last_fullsized":"2024-09-10T19:07:36.389368+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:15.852281+0000","last_clean_scrub_stamp":"2024-09-10T19:05:15.852281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":755,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421591+0000","last_change":"2024-09-10T19:06:21.347985+0000","last_active":"2024-09-10T19:08:39.421591+0000","last_peered":"2024-09-10T19:08:39.421591+0000","last_clean":"2024-09-10T19:08:39.421591+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:08:39.421591+0000","last_undegraded":"2024-09-10T19:08:39.421591+0000","last_fullsized":"2024-09-10T19:08:39.421591+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:21.347904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:21.347904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003263929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"410'25","reported_seq":879,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417261+0000","last_change":"2024-09-10T19:08:14.216487+0000","last_active":"2024-09-10T19:08:39.417261+0000","last_peered":"2024-09-10T19:08:39.417261+0000","last_clean":"2024-09-10T19:08:39.417261+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:08:39.417261+0000","last_undegraded":"2024-09-10T19:08:39.417261+0000","last_fullsized":"2024-09-10T19:08:39.417261+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:08:14.216297+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:08:14.216297+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:20.696166+0000","scrub_duration":0.0050864100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.425749+0000","last_change":"2024-09-10T19:06:51.097568+0000","last_active":"2024-09-10T19:08:39.425749+0000","last_peered":"2024-09-10T19:08:39.425749+0000","last_clean":"2024-09-10T19:08:39.425749+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:08:39.425749+0000","last_undegraded":"2024-09-10T19:08:39.425749+0000","last_fullsized":"2024-09-10T19:08:39.425749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:51.097406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:06:51.097406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003232005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":918,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.436377+0000","last_change":"2024-09-10T19:08:06.920954+0000","last_active":"2024-09-10T19:08:39.436377+0000","last_peered":"2024-09-10T19:08:39.436377+0000","last_clean":"2024-09-10T19:08:39.436377+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:08:39.436377+0000","last_undegraded":"2024-09-10T19:08:39.436377+0000","last_fullsized":"2024-09-10T19:08:39.436377+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:08:06.920810+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:05:32.752014+0000","last_clean_scrub_stamp":"2024-09-10T19:08:06.920810+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:33.531908+0000","scrub_duration":0.0027060840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":848,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.439860+0000","last_change":"2024-09-10T19:08:07.694616+0000","last_active":"2024-09-10T19:08:39.439860+0000","last_peered":"2024-09-10T19:08:39.439860+0000","last_clean":"2024-09-10T19:08:39.439860+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:08:39.439860+0000","last_undegraded":"2024-09-10T19:08:39.439860+0000","last_fullsized":"2024-09-10T19:08:39.439860+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:07.694472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:08:07.694472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:36.389156+0000","scrub_duration":0.0025424860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":839,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430492+0000","last_change":"2024-09-10T19:06:55.021461+0000","last_active":"2024-09-10T19:08:39.430492+0000","last_peered":"2024-09-10T19:08:39.430492+0000","last_clean":"2024-09-10T19:08:39.430492+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:08:39.430492+0000","last_undegraded":"2024-09-10T19:08:39.430492+0000","last_fullsized":"2024-09-10T19:08:39.430492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.021218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037286939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":774,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.432455+0000","last_change":"2024-09-10T19:07:36.397377+0000","last_active":"2024-09-10T19:08:39.432455+0000","last_peered":"2024-09-10T19:08:39.432455+0000","last_clean":"2024-09-10T19:07:34.449276+0000","last_became_active":"2024-09-10T19:07:36.397377+0000","last_became_peered":"2024-09-10T19:07:36.397377+0000","last_unstale":"2024-09-10T19:08:39.432455+0000","last_undegraded":"2024-09-10T19:08:39.432455+0000","last_fullsized":"2024-09-10T19:07:36.389643+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:45.993852+0000","last_clean_scrub_stamp":"2024-09-10T19:06:45.993852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035603079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"410'39","reported_seq":904,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432582+0000","last_change":"2024-09-10T19:06:47.011623+0000","last_active":"2024-09-10T19:08:39.432582+0000","last_peered":"2024-09-10T19:08:39.432582+0000","last_clean":"2024-09-10T19:08:39.432582+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:08:39.432582+0000","last_undegraded":"2024-09-10T19:08:39.432582+0000","last_fullsized":"2024-09-10T19:08:39.432582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'39","last_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_deep_scrub":"410'39","last_deep_scrub_stamp":"2024-09-10T19:06:47.011531+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.011531+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070935080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":884,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.426064+0000","last_change":"2024-09-10T19:07:58.220725+0000","last_active":"2024-09-10T19:08:39.426064+0000","last_peered":"2024-09-10T19:08:39.426064+0000","last_clean":"2024-09-10T19:08:39.426064+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:08:39.426064+0000","last_undegraded":"2024-09-10T19:08:39.426064+0000","last_fullsized":"2024-09-10T19:08:39.426064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:58.220565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:07:58.220565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:25.855331+0000","scrub_duration":0.0035541309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":855,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.438745+0000","last_change":"2024-09-10T19:07:36.396618+0000","last_active":"2024-09-10T19:08:39.438745+0000","last_peered":"2024-09-10T19:08:39.438745+0000","last_clean":"2024-09-10T19:07:34.367805+0000","last_became_active":"2024-09-10T19:07:36.396618+0000","last_became_peered":"2024-09-10T19:07:36.396618+0000","last_unstale":"2024-09-10T19:08:39.438745+0000","last_undegraded":"2024-09-10T19:08:39.438745+0000","last_fullsized":"2024-09-10T19:07:36.383336+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:47.846685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:29.731727+0000","last_clean_scrub_stamp":"2024-09-10T19:06:47.846685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002537961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":843,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.436864+0000","last_change":"2024-09-10T19:06:34.889988+0000","last_active":"2024-09-10T19:08:39.436864+0000","last_peered":"2024-09-10T19:08:39.436864+0000","last_clean":"2024-09-10T19:08:39.436864+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:08:39.436864+0000","last_undegraded":"2024-09-10T19:08:39.436864+0000","last_fullsized":"2024-09-10T19:08:39.436864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:34.889826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.889826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038908760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":785,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430586+0000","last_change":"2024-09-10T19:08:01.407700+0000","last_active":"2024-09-10T19:08:39.430586+0000","last_peered":"2024-09-10T19:08:39.430586+0000","last_clean":"2024-09-10T19:08:39.430586+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:08:39.430586+0000","last_undegraded":"2024-09-10T19:08:39.430586+0000","last_fullsized":"2024-09-10T19:08:39.430586+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:01.407551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:08:01.407551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:28.126783+0000","scrub_duration":0.0028775459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"410'25","reported_seq":885,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.427705+0000","last_change":"2024-09-10T19:08:00.719368+0000","last_active":"2024-09-10T19:08:39.427705+0000","last_peered":"2024-09-10T19:08:39.427705+0000","last_clean":"2024-09-10T19:08:39.427705+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:08:39.427705+0000","last_undegraded":"2024-09-10T19:08:39.427705+0000","last_fullsized":"2024-09-10T19:08:39.427705+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:08:00.719206+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:43.611756+0000","last_clean_scrub_stamp":"2024-09-10T19:08:00.719206+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:10.998643+0000","scrub_duration":0.0047437160000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.422152+0000","last_change":"2024-09-10T19:07:36.407722+0000","last_active":"2024-09-10T19:08:39.422152+0000","last_peered":"2024-09-10T19:08:39.422152+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.407722+0000","last_became_peered":"2024-09-10T19:07:36.407722+0000","last_unstale":"2024-09-10T19:08:39.422152+0000","last_undegraded":"2024-09-10T19:08:39.422152+0000","last_fullsized":"2024-09-10T19:07:36.386441+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:35.121923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:35.121923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":843,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.420833+0000","last_change":"2024-09-10T19:07:36.385560+0000","last_active":"2024-09-10T19:08:39.420833+0000","last_peered":"2024-09-10T19:08:39.420833+0000","last_clean":"2024-09-10T19:07:34.940499+0000","last_became_active":"2024-09-10T19:07:36.385560+0000","last_became_peered":"2024-09-10T19:07:36.385560+0000","last_unstale":"2024-09-10T19:08:39.420833+0000","last_undegraded":"2024-09-10T19:08:39.420833+0000","last_fullsized":"2024-09-10T19:07:36.375997+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:48.338294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:48.338294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033152189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":834,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418312+0000","last_change":"2024-09-10T19:06:53.347534+0000","last_active":"2024-09-10T19:08:39.418312+0000","last_peered":"2024-09-10T19:08:39.418312+0000","last_clean":"2024-09-10T19:08:39.418312+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:08:39.418312+0000","last_undegraded":"2024-09-10T19:08:39.418312+0000","last_fullsized":"2024-09-10T19:08:39.418312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.347358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:07.952993+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.347358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038827929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":765,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420899+0000","last_change":"2024-09-10T19:08:19.232096+0000","last_active":"2024-09-10T19:08:39.420899+0000","last_peered":"2024-09-10T19:08:39.420899+0000","last_clean":"2024-09-10T19:08:39.420899+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:08:39.420899+0000","last_undegraded":"2024-09-10T19:08:39.420899+0000","last_fullsized":"2024-09-10T19:08:39.420899+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:19.231852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:46.602792+0000","last_clean_scrub_stamp":"2024-09-10T19:08:19.231852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:41.137198+0000","scrub_duration":0.0040133729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"410'34","reported_seq":907,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.434826+0000","last_change":"2024-09-10T19:08:03.861633+0000","last_active":"2024-09-10T19:08:39.434826+0000","last_peered":"2024-09-10T19:08:39.434826+0000","last_clean":"2024-09-10T19:08:39.434826+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:08:39.434826+0000","last_undegraded":"2024-09-10T19:08:39.434826+0000","last_fullsized":"2024-09-10T19:08:39.434826+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:08:03.861448+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:08:03.861448+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:26.330553+0000","scrub_duration":0.0051960790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":896,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430426+0000","last_change":"2024-09-10T19:08:17.097412+0000","last_active":"2024-09-10T19:08:39.430426+0000","last_peered":"2024-09-10T19:08:39.430426+0000","last_clean":"2024-09-10T19:08:39.430426+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:08:39.430426+0000","last_undegraded":"2024-09-10T19:08:39.430426+0000","last_fullsized":"2024-09-10T19:08:39.430426+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:08:17.097247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:08:17.097247+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:18.076205+0000","scrub_duration":0.0038926,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":848,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422549+0000","last_change":"2024-09-10T19:06:22.828266+0000","last_active":"2024-09-10T19:08:39.422549+0000","last_peered":"2024-09-10T19:08:39.422549+0000","last_clean":"2024-09-10T19:08:39.422549+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:08:39.422549+0000","last_undegraded":"2024-09-10T19:08:39.422549+0000","last_fullsized":"2024-09-10T19:08:39.422549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:22.828113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.828113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031827000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.423373+0000","last_change":"2024-09-10T19:06:53.989492+0000","last_active":"2024-09-10T19:08:39.423373+0000","last_peered":"2024-09-10T19:08:39.423373+0000","last_clean":"2024-09-10T19:08:39.423373+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:08:39.423373+0000","last_undegraded":"2024-09-10T19:08:39.423373+0000","last_fullsized":"2024-09-10T19:08:39.423373+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:53.989397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:53.989397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021082940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":323,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.426636+0000","last_change":"2024-09-10T19:08:17.645083+0000","last_active":"2024-09-10T19:08:39.426636+0000","last_peered":"2024-09-10T19:08:39.426636+0000","last_clean":"2024-09-10T19:08:39.426636+0000","last_became_active":"2024-09-10T19:02:52.256701+0000","last_became_peered":"2024-09-10T19:02:52.256701+0000","last_unstale":"2024-09-10T19:08:39.426636+0000","last_undegraded":"2024-09-10T19:08:39.426636+0000","last_fullsized":"2024-09-10T19:08:39.426636+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:17.644928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:08:17.644928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:24.504368+0000","scrub_duration":0.0031311310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"410'19","reported_seq":878,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.434169+0000","last_change":"2024-09-10T19:07:36.391180+0000","last_active":"2024-09-10T19:08:39.434169+0000","last_peered":"2024-09-10T19:08:39.434169+0000","last_clean":"2024-09-10T19:07:34.449072+0000","last_became_active":"2024-09-10T19:07:36.391180+0000","last_became_peered":"2024-09-10T19:07:36.391180+0000","last_unstale":"2024-09-10T19:08:39.434169+0000","last_undegraded":"2024-09-10T19:07:36.382913+0000","last_fullsized":"2024-09-10T19:07:36.382793+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:04:53.827480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:04:53.827480+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086584599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"601'4","reported_seq":911,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.435595+0000","last_change":"2024-09-10T19:07:49.547394+0000","last_active":"2024-09-10T19:08:39.435595+0000","last_peered":"2024-09-10T19:08:39.435595+0000","last_clean":"2024-09-10T19:08:39.435595+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:08:39.435595+0000","last_undegraded":"2024-09-10T19:08:39.435595+0000","last_fullsized":"2024-09-10T19:08:39.435595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'4","last_scrub_stamp":"2024-09-10T19:07:49.547191+0000","last_deep_scrub":"601'4","last_deep_scrub_stamp":"2024-09-10T19:04:04.923921+0000","last_clean_scrub_stamp":"2024-09-10T19:07:49.547191+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:54.588591+0000","scrub_duration":0.0055618350000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"791'13","reported_seq":91,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.416914+0000","last_change":"2024-09-10T19:07:36.407153+0000","last_active":"2024-09-10T19:08:39.416914+0000","last_peered":"2024-09-10T19:08:39.416914+0000","last_clean":"2024-09-10T18:49:25.610593+0000","last_became_active":"2024-09-10T19:07:36.407153+0000","last_became_peered":"2024-09-10T19:07:36.407153+0000","last_unstale":"2024-09-10T19:08:39.416914+0000","last_undegraded":"2024-09-10T19:07:36.381960+0000","last_fullsized":"2024-09-10T19:07:36.381829+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'12","last_scrub_stamp":"2024-09-10T19:06:22.009440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:22.009440+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":847,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.419973+0000","last_change":"2024-09-10T19:07:36.388839+0000","last_active":"2024-09-10T19:08:39.419973+0000","last_peered":"2024-09-10T19:08:39.419973+0000","last_clean":"2024-09-10T19:07:34.370919+0000","last_became_active":"2024-09-10T19:07:36.388839+0000","last_became_peered":"2024-09-10T19:07:36.388839+0000","last_unstale":"2024-09-10T19:08:39.419973+0000","last_undegraded":"2024-09-10T19:08:39.419973+0000","last_fullsized":"2024-09-10T19:07:36.377177+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:00.155684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:07:00.155684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084607129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":782,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424030+0000","last_change":"2024-09-10T19:07:44.704612+0000","last_active":"2024-09-10T19:08:39.424030+0000","last_peered":"2024-09-10T19:08:39.424030+0000","last_clean":"2024-09-10T19:08:39.424030+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:08:39.424030+0000","last_undegraded":"2024-09-10T19:08:39.424030+0000","last_fullsized":"2024-09-10T19:08:39.424030+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:07:44.704506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:07:44.704506+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:08:56.640193+0000","scrub_duration":0.002809874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"410'19","reported_seq":888,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419329+0000","last_change":"2024-09-10T19:06:57.214131+0000","last_active":"2024-09-10T19:08:39.419329+0000","last_peered":"2024-09-10T19:08:39.419329+0000","last_clean":"2024-09-10T19:08:39.419329+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:08:39.419329+0000","last_undegraded":"2024-09-10T19:08:39.419329+0000","last_fullsized":"2024-09-10T19:08:39.419329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'19","last_scrub_stamp":"2024-09-10T19:06:57.213972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:57.213972+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047875110000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":881,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.421409+0000","last_change":"2024-09-10T19:07:36.395876+0000","last_active":"2024-09-10T19:08:39.421409+0000","last_peered":"2024-09-10T19:08:39.421409+0000","last_clean":"2024-09-10T19:07:34.368992+0000","last_became_active":"2024-09-10T19:07:36.395876+0000","last_became_peered":"2024-09-10T19:07:36.395876+0000","last_unstale":"2024-09-10T19:08:39.421409+0000","last_undegraded":"2024-09-10T19:08:39.421409+0000","last_fullsized":"2024-09-10T19:07:36.384123+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:55.984769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:06:55.984769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032115210000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":836,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421400+0000","last_change":"2024-09-10T19:05:28.258466+0000","last_active":"2024-09-10T19:08:39.421400+0000","last_peered":"2024-09-10T19:08:39.421400+0000","last_clean":"2024-09-10T19:08:39.421400+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:08:39.421400+0000","last_undegraded":"2024-09-10T19:08:39.421400+0000","last_fullsized":"2024-09-10T19:08:39.421400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.258181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.258181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037729090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":868,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.425212+0000","last_change":"2024-09-10T19:08:08.115280+0000","last_active":"2024-09-10T19:08:39.425212+0000","last_peered":"2024-09-10T19:08:39.425212+0000","last_clean":"2024-09-10T19:08:39.425212+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:08:39.425212+0000","last_undegraded":"2024-09-10T19:08:39.425212+0000","last_fullsized":"2024-09-10T19:08:39.425212+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:08.115119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:08.115119+0000","last_clean_scrub_stamp":"2024-09-10T19:08:08.115119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:23.320612+0000","scrub_duration":0.0033723189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":785,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.433186+0000","last_change":"2024-09-10T19:08:05.495618+0000","last_active":"2024-09-10T19:08:39.433186+0000","last_peered":"2024-09-10T19:08:39.433186+0000","last_clean":"2024-09-10T19:08:39.433186+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:08:39.433186+0000","last_undegraded":"2024-09-10T19:08:39.433186+0000","last_fullsized":"2024-09-10T19:08:39.433186+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:05.495325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:56.683587+0000","last_clean_scrub_stamp":"2024-09-10T19:08:05.495325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:09.871751+0000","scrub_duration":0.0036436580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"409'10","reported_seq":864,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.427281+0000","last_change":"2024-09-10T19:06:52.872642+0000","last_active":"2024-09-10T19:08:39.427281+0000","last_peered":"2024-09-10T19:08:39.427281+0000","last_clean":"2024-09-10T19:08:39.427281+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:08:39.427281+0000","last_undegraded":"2024-09-10T19:08:39.427281+0000","last_fullsized":"2024-09-10T19:08:39.427281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'10","last_scrub_stamp":"2024-09-10T19:06:52.872480+0000","last_deep_scrub":"113'6","last_deep_scrub_stamp":"2024-09-10T18:53:32.756731+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.872480+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042870599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":859,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.427620+0000","last_change":"2024-09-10T19:07:36.398730+0000","last_active":"2024-09-10T19:08:39.427620+0000","last_peered":"2024-09-10T19:08:39.427620+0000","last_clean":"2024-09-10T19:07:34.368000+0000","last_became_active":"2024-09-10T19:07:36.398730+0000","last_became_peered":"2024-09-10T19:07:36.398730+0000","last_unstale":"2024-09-10T19:08:39.427620+0000","last_undegraded":"2024-09-10T19:08:39.427620+0000","last_fullsized":"2024-09-10T19:07:36.378186+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:33.681850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:33.681850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033460909999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":839,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.421471+0000","last_change":"2024-09-10T19:07:36.389529+0000","last_active":"2024-09-10T19:08:39.421471+0000","last_peered":"2024-09-10T19:08:39.421471+0000","last_clean":"2024-09-10T19:07:34.374873+0000","last_became_active":"2024-09-10T19:07:36.389529+0000","last_became_peered":"2024-09-10T19:07:36.389529+0000","last_unstale":"2024-09-10T19:08:39.421471+0000","last_undegraded":"2024-09-10T19:08:39.421471+0000","last_fullsized":"2024-09-10T19:07:36.386952+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:16.995408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:05:16.995408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035441219999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432098+0000","last_change":"2024-09-10T19:08:02.890753+0000","last_active":"2024-09-10T19:08:39.432098+0000","last_peered":"2024-09-10T19:08:39.432098+0000","last_clean":"2024-09-10T19:08:39.432098+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:08:39.432098+0000","last_undegraded":"2024-09-10T19:08:39.432098+0000","last_fullsized":"2024-09-10T19:08:39.432098+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:02.890597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:08:02.890597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:27.550952+0000","scrub_duration":0.0032852060000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":769,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.440962+0000","last_change":"2024-09-10T19:07:57.774014+0000","last_active":"2024-09-10T19:08:39.440962+0000","last_peered":"2024-09-10T19:08:39.440962+0000","last_clean":"2024-09-10T19:08:39.440962+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:08:39.440962+0000","last_undegraded":"2024-09-10T19:08:39.440962+0000","last_fullsized":"2024-09-10T19:08:39.440962+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:57.773845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:07:57.773845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:12.862597+0000","scrub_duration":0.003316299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"410'24","reported_seq":883,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.433214+0000","last_change":"2024-09-10T19:06:43.983711+0000","last_active":"2024-09-10T19:08:39.433214+0000","last_peered":"2024-09-10T19:08:39.433214+0000","last_clean":"2024-09-10T19:08:39.433214+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:08:39.433214+0000","last_undegraded":"2024-09-10T19:08:39.433214+0000","last_fullsized":"2024-09-10T19:08:39.433214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'24","last_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_deep_scrub":"410'24","last_deep_scrub_stamp":"2024-09-10T19:06:43.983521+0000","last_clean_scrub_stamp":"2024-09-10T19:06:43.983521+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005511228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":845,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.424092+0000","last_change":"2024-09-10T19:06:32.786736+0000","last_active":"2024-09-10T19:08:39.424092+0000","last_peered":"2024-09-10T19:08:39.424092+0000","last_clean":"2024-09-10T19:08:39.424092+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:08:39.424092+0000","last_undegraded":"2024-09-10T19:08:39.424092+0000","last_fullsized":"2024-09-10T19:08:39.424092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.786573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.786573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010562732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"791'13","reported_seq":1096,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.422644+0000","last_change":"2024-09-10T19:07:36.388813+0000","last_active":"2024-09-10T19:08:39.422644+0000","last_peered":"2024-09-10T19:08:39.422644+0000","last_clean":"2024-09-10T19:07:34.372256+0000","last_became_active":"2024-09-10T19:07:36.388813+0000","last_became_peered":"2024-09-10T19:07:36.388813+0000","last_unstale":"2024-09-10T19:08:39.422644+0000","last_undegraded":"2024-09-10T19:07:36.380172+0000","last_fullsized":"2024-09-10T19:07:36.380081+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"586'12","last_scrub_stamp":"2024-09-10T19:06:20.041985+0000","last_deep_scrub":"586'12","last_deep_scrub_stamp":"2024-09-10T19:03:50.172621+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.041985+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003891285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":844,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420355+0000","last_change":"2024-09-10T19:08:12.153438+0000","last_active":"2024-09-10T19:08:39.420355+0000","last_peered":"2024-09-10T19:08:39.420355+0000","last_clean":"2024-09-10T19:08:39.420355+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:08:39.420355+0000","last_undegraded":"2024-09-10T19:08:39.420355+0000","last_fullsized":"2024-09-10T19:08:39.420355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:12.153343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.407512+0000","last_clean_scrub_stamp":"2024-09-10T19:08:12.153343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:37.767515+0000","scrub_duration":0.0028982579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":755,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420751+0000","last_change":"2024-09-10T19:06:30.306917+0000","last_active":"2024-09-10T19:08:39.420751+0000","last_peered":"2024-09-10T19:08:39.420751+0000","last_clean":"2024-09-10T19:08:39.420751+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:08:39.420751+0000","last_undegraded":"2024-09-10T19:08:39.420751+0000","last_fullsized":"2024-09-10T19:08:39.420751+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:30.306762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:33.116061+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.306762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038348879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"410'15","reported_seq":873,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419047+0000","last_change":"2024-09-10T19:08:13.109314+0000","last_active":"2024-09-10T19:08:39.419047+0000","last_peered":"2024-09-10T19:08:39.419047+0000","last_clean":"2024-09-10T19:08:39.419047+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:08:39.419047+0000","last_undegraded":"2024-09-10T19:08:39.419047+0000","last_fullsized":"2024-09-10T19:08:39.419047+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:08:13.109147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:08:13.109147+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:42.392925+0000","scrub_duration":0.0044673530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":850,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421130+0000","last_change":"2024-09-10T19:06:58.401445+0000","last_active":"2024-09-10T19:08:39.421130+0000","last_peered":"2024-09-10T19:08:39.421130+0000","last_clean":"2024-09-10T19:08:39.421130+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:08:39.421130+0000","last_undegraded":"2024-09-10T19:08:39.421130+0000","last_fullsized":"2024-09-10T19:08:39.421130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:58.401361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:44.606395+0000","last_clean_scrub_stamp":"2024-09-10T19:06:58.401361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002332808,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":1070,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.426171+0000","last_change":"2024-09-10T19:06:28.763314+0000","last_active":"2024-09-10T19:08:39.426171+0000","last_peered":"2024-09-10T19:08:39.426171+0000","last_clean":"2024-09-10T19:08:39.426171+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:08:39.426171+0000","last_undegraded":"2024-09-10T19:08:39.426171+0000","last_fullsized":"2024-09-10T19:08:39.426171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:06:28.763149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.763149+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099338919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":323,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430882+0000","last_change":"2024-09-10T19:06:54.081950+0000","last_active":"2024-09-10T19:08:39.430882+0000","last_peered":"2024-09-10T19:08:39.430882+0000","last_clean":"2024-09-10T19:08:39.430882+0000","last_became_active":"2024-09-10T19:02:52.256751+0000","last_became_peered":"2024-09-10T19:02:52.256751+0000","last_unstale":"2024-09-10T19:08:39.430882+0000","last_undegraded":"2024-09-10T19:08:39.430882+0000","last_fullsized":"2024-09-10T19:08:39.430882+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.081811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.081811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036574749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":755,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420537+0000","last_change":"2024-09-10T19:07:48.367935+0000","last_active":"2024-09-10T19:08:39.420537+0000","last_peered":"2024-09-10T19:08:39.420537+0000","last_clean":"2024-09-10T19:08:39.420537+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:08:39.420537+0000","last_undegraded":"2024-09-10T19:08:39.420537+0000","last_fullsized":"2024-09-10T19:08:39.420537+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:48.367798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:37.571747+0000","last_clean_scrub_stamp":"2024-09-10T19:07:48.367798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.435649+0000","scrub_duration":0.0036291650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"410'15","reported_seq":882,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432315+0000","last_change":"2024-09-10T19:08:04.857843+0000","last_active":"2024-09-10T19:08:39.432315+0000","last_peered":"2024-09-10T19:08:39.432315+0000","last_clean":"2024-09-10T19:08:39.432315+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:08:39.432315+0000","last_undegraded":"2024-09-10T19:08:39.432315+0000","last_fullsized":"2024-09-10T19:08:39.432315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:08:04.857675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:08:04.857675+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:16.730732+0000","scrub_duration":0.0047708560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.438864+0000","last_change":"2024-09-10T19:08:04.677624+0000","last_active":"2024-09-10T19:08:39.438864+0000","last_peered":"2024-09-10T19:08:39.438864+0000","last_clean":"2024-09-10T19:08:39.438864+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:08:39.438864+0000","last_undegraded":"2024-09-10T19:08:39.438864+0000","last_fullsized":"2024-09-10T19:08:39.438864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:04.677463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:08:04.677463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:33.471977+0000","scrub_duration":0.0032961219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":839,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.423143+0000","last_change":"2024-09-10T19:06:37.901263+0000","last_active":"2024-09-10T19:08:39.423143+0000","last_peered":"2024-09-10T19:08:39.423143+0000","last_clean":"2024-09-10T19:08:39.423143+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:08:39.423143+0000","last_undegraded":"2024-09-10T19:08:39.423143+0000","last_fullsized":"2024-09-10T19:08:39.423143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:37.901111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:06:37.901111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036067199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":844,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.415112+0000","last_change":"2024-09-10T19:07:36.404843+0000","last_active":"2024-09-10T19:08:39.415112+0000","last_peered":"2024-09-10T19:08:39.415112+0000","last_clean":"2024-09-10T19:07:34.938683+0000","last_became_active":"2024-09-10T19:07:36.404843+0000","last_became_peered":"2024-09-10T19:07:36.404843+0000","last_unstale":"2024-09-10T19:08:39.415112+0000","last_undegraded":"2024-09-10T19:08:39.415112+0000","last_fullsized":"2024-09-10T19:07:36.374678+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:02.444182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:05:02.444182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002871462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":775,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.431133+0000","last_change":"2024-09-10T19:06:54.653102+0000","last_active":"2024-09-10T19:08:39.431133+0000","last_peered":"2024-09-10T19:08:39.431133+0000","last_clean":"2024-09-10T19:08:39.431133+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:08:39.431133+0000","last_undegraded":"2024-09-10T19:08:39.431133+0000","last_fullsized":"2024-09-10T19:08:39.431133+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:54.652944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:54.652944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032820850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"410'25","reported_seq":759,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418516+0000","last_change":"2024-09-10T19:07:45.951762+0000","last_active":"2024-09-10T19:08:39.418516+0000","last_peered":"2024-09-10T19:08:39.418516+0000","last_clean":"2024-09-10T19:08:39.418516+0000","last_became_active":"2024-09-10T19:02:52.257159+0000","last_became_peered":"2024-09-10T19:02:52.257159+0000","last_unstale":"2024-09-10T19:08:39.418516+0000","last_undegraded":"2024-09-10T19:08:39.418516+0000","last_fullsized":"2024-09-10T19:08:39.418516+0000","mapping_epoch":547,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'25","last_scrub_stamp":"2024-09-10T19:07:45.951248+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:07:45.951248+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:07.055026+0000","scrub_duration":0.0068702210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.426654+0000","last_change":"2024-09-10T19:07:36.407170+0000","last_active":"2024-09-10T19:08:39.426654+0000","last_peered":"2024-09-10T19:08:39.426654+0000","last_clean":"2024-09-10T18:49:16.574846+0000","last_became_active":"2024-09-10T19:07:36.407170+0000","last_became_peered":"2024-09-10T19:07:36.407170+0000","last_unstale":"2024-09-10T19:08:39.426654+0000","last_undegraded":"2024-09-10T19:08:39.426654+0000","last_fullsized":"2024-09-10T19:07:36.391432+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:17.966265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:17.966265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":863,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.423377+0000","last_change":"2024-09-10T19:07:36.404939+0000","last_active":"2024-09-10T19:08:39.423377+0000","last_peered":"2024-09-10T19:08:39.423377+0000","last_clean":"2024-09-10T19:07:34.939194+0000","last_became_active":"2024-09-10T19:07:36.404939+0000","last_became_peered":"2024-09-10T19:07:36.404939+0000","last_unstale":"2024-09-10T19:08:39.423377+0000","last_undegraded":"2024-09-10T19:08:39.423377+0000","last_fullsized":"2024-09-10T19:07:36.377468+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:28.866466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:20.152369+0000","last_clean_scrub_stamp":"2024-09-10T19:06:28.866466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080387719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.416529+0000","last_change":"2024-09-10T19:08:17.298101+0000","last_active":"2024-09-10T19:08:39.416529+0000","last_peered":"2024-09-10T19:08:39.416529+0000","last_clean":"2024-09-10T19:08:39.416529+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:08:39.416529+0000","last_undegraded":"2024-09-10T19:08:39.416529+0000","last_fullsized":"2024-09-10T19:08:39.416529+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:17.297932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:17.297932+0000","last_clean_scrub_stamp":"2024-09-10T19:08:17.297932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:42.626200+0000","scrub_duration":0.003318634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":61,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.417617+0000","last_change":"2024-09-10T19:07:36.390482+0000","last_active":"2024-09-10T19:08:39.417617+0000","last_peered":"2024-09-10T19:08:39.417617+0000","last_clean":"2024-09-10T18:54:31.683713+0000","last_became_active":"2024-09-10T19:07:36.390482+0000","last_became_peered":"2024-09-10T19:07:36.390482+0000","last_unstale":"2024-09-10T19:08:39.417617+0000","last_undegraded":"2024-09-10T19:08:39.417617+0000","last_fullsized":"2024-09-10T19:07:36.376161+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":548,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:32.121693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:06:32.121693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"410'15","reported_seq":859,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421752+0000","last_change":"2024-09-10T19:06:41.386525+0000","last_active":"2024-09-10T19:08:39.421752+0000","last_peered":"2024-09-10T19:08:39.421752+0000","last_clean":"2024-09-10T19:08:39.421752+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:08:39.421752+0000","last_undegraded":"2024-09-10T19:08:39.421752+0000","last_fullsized":"2024-09-10T19:08:39.421752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_deep_scrub":"410'15","last_deep_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_clean_scrub_stamp":"2024-09-10T19:06:41.386460+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022228119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":842,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.430524+0000","last_change":"2024-09-10T19:06:20.906700+0000","last_active":"2024-09-10T19:08:39.430524+0000","last_peered":"2024-09-10T19:08:39.430524+0000","last_clean":"2024-09-10T19:08:39.430524+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:08:39.430524+0000","last_undegraded":"2024-09-10T19:08:39.430524+0000","last_fullsized":"2024-09-10T19:08:39.430524+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:20.906544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.600690+0000","last_clean_scrub_stamp":"2024-09-10T19:06:20.906544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034009769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":842,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.422277+0000","last_change":"2024-09-10T19:07:36.392454+0000","last_active":"2024-09-10T19:08:39.422277+0000","last_peered":"2024-09-10T19:08:39.422277+0000","last_clean":"2024-09-10T19:07:34.373733+0000","last_became_active":"2024-09-10T19:07:36.392454+0000","last_became_peered":"2024-09-10T19:07:36.392454+0000","last_unstale":"2024-09-10T19:08:39.422277+0000","last_undegraded":"2024-09-10T19:08:39.422277+0000","last_fullsized":"2024-09-10T19:07:36.376817+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_clean_scrub_stamp":"2024-09-10T19:05:28.918873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030432179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419442+0000","last_change":"2024-09-10T19:07:47.924901+0000","last_active":"2024-09-10T19:08:39.419442+0000","last_peered":"2024-09-10T19:08:39.419442+0000","last_clean":"2024-09-10T19:08:39.419442+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:08:39.419442+0000","last_undegraded":"2024-09-10T19:08:39.419442+0000","last_fullsized":"2024-09-10T19:08:39.419442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:47.924745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:07:47.924745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:14.376613+0000","scrub_duration":0.0037064260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.435961+0000","last_change":"2024-09-10T19:07:36.394266+0000","last_active":"2024-09-10T19:08:39.435961+0000","last_peered":"2024-09-10T19:08:39.435961+0000","last_clean":"2024-09-10T18:54:31.683713+0000","last_became_active":"2024-09-10T19:07:36.394266+0000","last_became_peered":"2024-09-10T19:07:36.394266+0000","last_unstale":"2024-09-10T19:08:39.435961+0000","last_undegraded":"2024-09-10T19:08:39.435961+0000","last_fullsized":"2024-09-10T19:07:36.381738+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:50.191205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:14.886297+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.191205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"676'19","reported_seq":876,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.430345+0000","last_change":"2024-09-10T19:07:36.406346+0000","last_active":"2024-09-10T19:08:39.430345+0000","last_peered":"2024-09-10T19:08:39.430345+0000","last_clean":"2024-09-10T19:07:34.941372+0000","last_became_active":"2024-09-10T19:07:36.406346+0000","last_became_peered":"2024-09-10T19:07:36.406346+0000","last_unstale":"2024-09-10T19:08:39.430345+0000","last_undegraded":"2024-09-10T19:07:36.399394+0000","last_fullsized":"2024-09-10T19:07:36.399279+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"676'19","last_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_deep_scrub":"676'19","last_deep_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_clean_scrub_stamp":"2024-09-10T19:05:34.022377+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006303052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.443228+0000","last_change":"2024-09-10T19:07:55.761391+0000","last_active":"2024-09-10T19:08:39.443228+0000","last_peered":"2024-09-10T19:08:39.443228+0000","last_clean":"2024-09-10T19:08:39.443228+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:08:39.443228+0000","last_undegraded":"2024-09-10T19:08:39.443228+0000","last_fullsized":"2024-09-10T19:08:39.443228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:55.761234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:07:55.761234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:21.668963+0000","scrub_duration":0.0033344680000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":842,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.416740+0000","last_change":"2024-09-10T19:08:08.123057+0000","last_active":"2024-09-10T19:08:39.416740+0000","last_peered":"2024-09-10T19:08:39.416740+0000","last_clean":"2024-09-10T19:08:39.416740+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:08:39.416740+0000","last_undegraded":"2024-09-10T19:08:39.416740+0000","last_fullsized":"2024-09-10T19:08:39.416740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:08.122936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:08:08.122936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:19.058886+0000","scrub_duration":0.003831246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":840,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417999+0000","last_change":"2024-09-10T19:06:46.126997+0000","last_active":"2024-09-10T19:08:39.417999+0000","last_peered":"2024-09-10T19:08:39.417999+0000","last_clean":"2024-09-10T19:08:39.417999+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:08:39.417999+0000","last_undegraded":"2024-09-10T19:08:39.417999+0000","last_fullsized":"2024-09-10T19:08:39.417999+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.126832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.126832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003551504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":755,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418524+0000","last_change":"2024-09-10T19:06:46.381092+0000","last_active":"2024-09-10T19:08:39.418524+0000","last_peered":"2024-09-10T19:08:39.418524+0000","last_clean":"2024-09-10T19:08:39.418524+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:08:39.418524+0000","last_undegraded":"2024-09-10T19:08:39.418524+0000","last_fullsized":"2024-09-10T19:08:39.418524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_clean_scrub_stamp":"2024-09-10T19:06:46.380935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032731539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":853,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.418319+0000","last_change":"2024-09-10T19:08:18.027391+0000","last_active":"2024-09-10T19:08:39.418319+0000","last_peered":"2024-09-10T19:08:39.418319+0000","last_clean":"2024-09-10T19:08:39.418319+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:08:39.418319+0000","last_undegraded":"2024-09-10T19:08:39.418319+0000","last_fullsized":"2024-09-10T19:08:39.418319+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:08:18.027220+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:08:18.027220+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:28.081515+0000","scrub_duration":0.003854564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"410'34","reported_seq":508,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.420611+0000","last_change":"2024-09-10T19:07:36.407974+0000","last_active":"2024-09-10T19:08:39.420611+0000","last_peered":"2024-09-10T19:08:39.420611+0000","last_clean":"2024-09-10T18:59:26.429814+0000","last_became_active":"2024-09-10T19:07:36.407974+0000","last_became_peered":"2024-09-10T19:07:36.407974+0000","last_unstale":"2024-09-10T19:08:39.420611+0000","last_undegraded":"2024-09-10T19:07:36.386486+0000","last_fullsized":"2024-09-10T19:07:36.385690+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'34","last_scrub_stamp":"2024-09-10T19:06:56.247893+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:06:56.247893+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011481923999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":852,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.422629+0000","last_change":"2024-09-10T19:07:48.624021+0000","last_active":"2024-09-10T19:08:39.422629+0000","last_peered":"2024-09-10T19:08:39.422629+0000","last_clean":"2024-09-10T19:08:39.422629+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:08:39.422629+0000","last_undegraded":"2024-09-10T19:08:39.422629+0000","last_fullsized":"2024-09-10T19:08:39.422629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:48.623862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:07:48.623862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:14.881230+0000","scrub_duration":0.003397382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.420461+0000","last_change":"2024-09-10T19:08:21.180994+0000","last_active":"2024-09-10T19:08:39.420461+0000","last_peered":"2024-09-10T19:08:39.420461+0000","last_clean":"2024-09-10T19:08:39.420461+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:08:39.420461+0000","last_undegraded":"2024-09-10T19:08:39.420461+0000","last_fullsized":"2024-09-10T19:08:39.420461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:21.180819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:23.207050+0000","last_clean_scrub_stamp":"2024-09-10T19:08:21.180819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:44.540219+0000","scrub_duration":0.0031746470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":769,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.436477+0000","last_change":"2024-09-10T19:08:06.688811+0000","last_active":"2024-09-10T19:08:39.436477+0000","last_peered":"2024-09-10T19:08:39.436477+0000","last_clean":"2024-09-10T19:08:39.436477+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:08:39.436477+0000","last_undegraded":"2024-09-10T19:08:39.436477+0000","last_fullsized":"2024-09-10T19:08:39.436477+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:06.688653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:08:06.688653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:35.985893+0000","scrub_duration":0.0036860700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":56,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.425951+0000","last_change":"2024-09-10T19:07:36.398437+0000","last_active":"2024-09-10T19:08:39.425951+0000","last_peered":"2024-09-10T19:08:39.425951+0000","last_clean":"2024-09-10T18:49:23.022168+0000","last_became_active":"2024-09-10T19:07:36.398437+0000","last_became_peered":"2024-09-10T19:07:36.398437+0000","last_unstale":"2024-09-10T19:08:39.425951+0000","last_undegraded":"2024-09-10T19:08:39.425951+0000","last_fullsized":"2024-09-10T19:07:36.386157+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.010845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:38.304790+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.010845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"409'15","reported_seq":876,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.421134+0000","last_change":"2024-09-10T19:07:36.393413+0000","last_active":"2024-09-10T19:08:39.421134+0000","last_peered":"2024-09-10T19:08:39.421134+0000","last_clean":"2024-09-10T19:07:34.374210+0000","last_became_active":"2024-09-10T19:07:36.393413+0000","last_became_peered":"2024-09-10T19:07:36.393413+0000","last_unstale":"2024-09-10T19:08:39.421134+0000","last_undegraded":"2024-09-10T19:07:36.378649+0000","last_fullsized":"2024-09-10T19:07:36.378123+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"409'15","last_scrub_stamp":"2024-09-10T19:06:34.054632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:34.054632+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038029330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.417533+0000","last_change":"2024-09-10T19:06:52.165777+0000","last_active":"2024-09-10T19:08:39.417533+0000","last_peered":"2024-09-10T19:08:39.417533+0000","last_clean":"2024-09-10T19:08:39.417533+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:08:39.417533+0000","last_undegraded":"2024-09-10T19:08:39.417533+0000","last_fullsized":"2024-09-10T19:08:39.417533+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_clean_scrub_stamp":"2024-09-10T19:06:52.165627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00328166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"85'12","reported_seq":1107,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.442602+0000","last_change":"2024-09-10T19:06:50.856618+0000","last_active":"2024-09-10T19:08:39.442602+0000","last_peered":"2024-09-10T19:08:39.442602+0000","last_clean":"2024-09-10T19:08:39.442602+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:08:39.442602+0000","last_undegraded":"2024-09-10T19:08:39.442602+0000","last_fullsized":"2024-09-10T19:08:39.442602+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-09-10T19:06:50.856451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:06:50.856451+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045245329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":769,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.442338+0000","last_change":"2024-09-10T19:08:00.717852+0000","last_active":"2024-09-10T19:08:39.442338+0000","last_peered":"2024-09-10T19:08:39.442338+0000","last_clean":"2024-09-10T19:08:39.442338+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:08:39.442338+0000","last_undegraded":"2024-09-10T19:08:39.442338+0000","last_fullsized":"2024-09-10T19:08:39.442338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:00.717688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:08:00.717688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:18.955753+0000","scrub_duration":0.0038609899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":864,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.432800+0000","last_change":"2024-09-10T19:07:52.429092+0000","last_active":"2024-09-10T19:08:39.432800+0000","last_peered":"2024-09-10T19:08:39.432800+0000","last_clean":"2024-09-10T19:08:39.432800+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:08:39.432800+0000","last_undegraded":"2024-09-10T19:08:39.432800+0000","last_fullsized":"2024-09-10T19:08:39.432800+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:07:52.428930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:07:52.428930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:05.256885+0000","scrub_duration":0.0038142369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":851,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.438977+0000","last_change":"2024-09-10T19:06:30.665495+0000","last_active":"2024-09-10T19:08:39.438977+0000","last_peered":"2024-09-10T19:08:39.438977+0000","last_clean":"2024-09-10T19:08:39.438977+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:08:39.438977+0000","last_undegraded":"2024-09-10T19:08:39.438977+0000","last_fullsized":"2024-09-10T19:08:39.438977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:06:30.665240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:06:30.665240+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058393220000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"410'7","reported_seq":851,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421027+0000","last_change":"2024-09-10T19:05:30.243873+0000","last_active":"2024-09-10T19:08:39.421027+0000","last_peered":"2024-09-10T19:08:39.421027+0000","last_clean":"2024-09-10T19:08:39.421027+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:08:39.421027+0000","last_undegraded":"2024-09-10T19:08:39.421027+0000","last_fullsized":"2024-09-10T19:08:39.421027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'7","last_scrub_stamp":"2024-09-10T19:05:30.243807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:05:30.243807+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031694280000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":877,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.425664+0000","last_change":"2024-09-10T19:07:36.390999+0000","last_active":"2024-09-10T19:08:39.425664+0000","last_peered":"2024-09-10T19:08:39.425664+0000","last_clean":"2024-09-10T19:07:34.373341+0000","last_became_active":"2024-09-10T19:07:36.390999+0000","last_became_peered":"2024-09-10T19:07:36.390999+0000","last_unstale":"2024-09-10T19:08:39.425664+0000","last_undegraded":"2024-09-10T19:08:39.425664+0000","last_fullsized":"2024-09-10T19:07:36.383681+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:42.199204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.968019+0000","last_clean_scrub_stamp":"2024-09-10T19:06:42.199204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034375769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":769,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.427981+0000","last_change":"2024-09-10T19:08:06.774495+0000","last_active":"2024-09-10T19:08:39.427981+0000","last_peered":"2024-09-10T19:08:39.427981+0000","last_clean":"2024-09-10T19:08:39.427981+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:08:39.427981+0000","last_undegraded":"2024-09-10T19:08:39.427981+0000","last_fullsized":"2024-09-10T19:08:39.427981+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:08:06.774232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:06.774232+0000","last_clean_scrub_stamp":"2024-09-10T19:08:06.774232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:09:31.737502+0000","scrub_duration":0.003646449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":840,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.421156+0000","last_change":"2024-09-10T19:06:19.090138+0000","last_active":"2024-09-10T19:08:39.421156+0000","last_peered":"2024-09-10T19:08:39.421156+0000","last_clean":"2024-09-10T19:08:39.421156+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:08:39.421156+0000","last_undegraded":"2024-09-10T19:08:39.421156+0000","last_fullsized":"2024-09-10T19:08:39.421156+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:19.089886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:06:19.089886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040961510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":846,"reported_epoch":836,"state":"active+clean","last_fresh":"2024-09-10T19:08:39.419723+0000","last_change":"2024-09-10T19:06:31.070915+0000","last_active":"2024-09-10T19:08:39.419723+0000","last_peered":"2024-09-10T19:08:39.419723+0000","last_clean":"2024-09-10T19:08:39.419723+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:08:39.419723+0000","last_undegraded":"2024-09-10T19:08:39.419723+0000","last_fullsized":"2024-09-10T19:08:39.419723+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:06:31.070745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:09.745020+0000","last_clean_scrub_stamp":"2024-09-10T19:06:31.070745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036538719999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"410'15","reported_seq":890,"reported_epoch":836,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:08:39.431394+0000","last_change":"2024-09-10T19:07:36.391503+0000","last_active":"2024-09-10T19:08:39.431394+0000","last_peered":"2024-09-10T19:08:39.431394+0000","last_clean":"2024-09-10T19:07:34.939737+0000","last_became_active":"2024-09-10T19:07:36.391503+0000","last_became_peered":"2024-09-10T19:07:36.391503+0000","last_unstale":"2024-09-10T19:08:39.431394+0000","last_undegraded":"2024-09-10T19:07:36.376840+0000","last_fullsized":"2024-09-10T19:07:36.376678+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'15","last_scrub_stamp":"2024-09-10T19:06:23.971482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:06:23.971482+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045234610000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":839,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.422032+0000","last_change":"2024-09-10T19:07:36.407182+0000","last_active":"2024-09-10T19:08:39.422032+0000","last_peered":"2024-09-10T19:08:39.422032+0000","last_clean":"2024-09-10T19:07:34.373409+0000","last_became_active":"2024-09-10T19:07:36.407182+0000","last_became_peered":"2024-09-10T19:07:36.407182+0000","last_unstale":"2024-09-10T19:08:39.422032+0000","last_undegraded":"2024-09-10T19:08:39.422032+0000","last_fullsized":"2024-09-10T19:07:36.381985+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:14.978923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:05:14.978923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003307857,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":766,"reported_epoch":836,"state":"active+undersized","last_fresh":"2024-09-10T19:08:39.441085+0000","last_change":"2024-09-10T19:07:36.400149+0000","last_active":"2024-09-10T19:08:39.441085+0000","last_peered":"2024-09-10T19:08:39.441085+0000","last_clean":"2024-09-10T19:07:34.367856+0000","last_became_active":"2024-09-10T19:07:36.400149+0000","last_became_peered":"2024-09-10T19:07:36.400149+0000","last_unstale":"2024-09-10T19:08:39.441085+0000","last_undegraded":"2024-09-10T19:08:39.441085+0000","last_fullsized":"2024-09-10T19:07:36.385769+0000","mapping_epoch":790,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:05:33.491961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:51.945980+0000","last_clean_scrub_stamp":"2024-09-10T19:05:33.491961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025640609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":262,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":86,"acting":86,"num_store_stats":8},{"poolid":241,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":43,"num_object_clones":1,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":27,"num_read_kb":9,"num_write":67,"num_write_kb":3967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6479872,"data_stored":6479872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":24,"acting":24,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1135,"num_read_kb":1130,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":8,"acting":8,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":79,"acting":79,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":71,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":85,"acting":85,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":32,"acting":32,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972596,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304352,"kb_used_data":2836,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440800,"statfs":{"total":95995035648,"available":95683379200,"internally_reserved":0,"allocated":2904064,"data_stored":1301838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.92500000000000004,"15min":0.90400000000000003},"min":{"1min":0.622,"5min":0.432,"15min":0.311},"max":{"1min":1.2410000000000001,"5min":2.839,"15min":2.839},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.97299999999999998,"15min":0.94399999999999995},"min":{"1min":0.69599999999999995,"5min":0.497,"15min":0.34699999999999998},"max":{"1min":1.3520000000000001,"5min":2.9550000000000001,"15min":4.7000000000000002},"last":0.81799999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95599999999999996,"15min":0.90500000000000003},"min":{"1min":0.51000000000000001,"5min":0.504,"15min":0.41199999999999998},"max":{"1min":3.097,"5min":4.9299999999999997,"15min":5.6539999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.97199999999999998,"15min":0.96199999999999997},"min":{"1min":0.622,"5min":0.57899999999999996,"15min":0.31900000000000001},"max":{"1min":1.3420000000000001,"5min":2.012,"15min":2.012},"last":1.1379999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0109999999999999,"15min":1.109},"min":{"1min":0.54100000000000004,"5min":0.53500000000000003,"15min":0.41499999999999998},"max":{"1min":1.427,"5min":1.601,"15min":31.166},"last":1.018},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.98399999999999999,"15min":1.085},"min":{"1min":0.66000000000000003,"5min":0.55900000000000005,"15min":0.443},"max":{"1min":1.3819999999999999,"5min":1.8560000000000001,"15min":27.709},"last":0.91600000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.046,"15min":1.0349999999999999},"min":{"1min":0.59099999999999997,"5min":0.51700000000000002,"15min":0.48299999999999998},"max":{"1min":1.335,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.98899999999999999,"15min":0.98399999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.46600000000000003},"max":{"1min":1.53,"5min":4.2060000000000004,"15min":5.6050000000000004},"last":0.78400000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.998,"15min":1.0229999999999999},"min":{"1min":0.66300000000000003,"5min":0.51800000000000002,"15min":0.34799999999999998},"max":{"1min":1.284,"5min":1.522,"15min":1.5760000000000001},"last":0.877},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0609999999999999,"15min":1.083},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.48099999999999998},"max":{"1min":1.4390000000000001,"5min":2.25,"15min":4.2400000000000002},"last":0.68999999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96099999999999997,"15min":0.98199999999999998},"min":{"1min":0.65600000000000003,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.476,"5min":1.5980000000000001,"15min":5.4710000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.123,"15min":1.145},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.47499999999999998},"max":{"1min":1.5720000000000001,"5min":4.8710000000000004,"15min":4.8710000000000004},"last":1.0660000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1479999999999999,"15min":1.1739999999999999},"min":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.48699999999999999},"max":{"1min":1.5189999999999999,"5min":1.766,"15min":1.766},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.137,"5min":1.1459999999999999,"15min":1.1859999999999999},"min":{"1min":0.71299999999999997,"5min":0.61399999999999999,"15min":0.58099999999999996},"max":{"1min":1.5049999999999999,"5min":2.4340000000000002,"15min":4.1310000000000002},"last":0.75900000000000001}]}]},{"osd":6,"up_from":52,"seq":223338299646,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304004,"kb_used_data":2480,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441148,"statfs":{"total":95995035648,"available":95683735552,"internally_reserved":0,"allocated":2539520,"data_stored":948753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.89300000000000002,"15min":0.88500000000000001},"min":{"1min":0.42899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.294,"5min":1.696,"15min":2.0550000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.97099999999999997,"15min":0.96499999999999997},"min":{"1min":0.67600000000000005,"5min":0.438,"15min":0.36399999999999999},"max":{"1min":1.3440000000000001,"5min":1.6859999999999999,"15min":6.8959999999999999},"last":1.1359999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.0820000000000001,"15min":1.091},"min":{"1min":0.64800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":18.177,"5min":18.177,"15min":18.494},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.696,"5min":1.0469999999999999,"15min":0.95099999999999996},"min":{"1min":0.69899999999999995,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":17.297000000000001,"5min":17.297000000000001,"15min":17.297000000000001},"last":1.0629999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.93700000000000006,"15min":0.96999999999999997},"min":{"1min":0.57699999999999996,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.2110000000000001,"5min":1.5589999999999999,"15min":9.4800000000000004},"last":0.874},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.92600000000000005,"15min":0.94099999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.46500000000000002},"max":{"1min":1.294,"5min":1.3779999999999999,"15min":6.0970000000000004},"last":0.70999999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.96699999999999997,"15min":1.006},"min":{"1min":0.54400000000000004,"5min":0.48099999999999998,"15min":0.40400000000000003},"max":{"1min":1.2330000000000001,"5min":1.8700000000000001,"15min":2.278},"last":0.747},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.57999999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.246,"5min":1.4730000000000001,"15min":3.4279999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.96699999999999997,"15min":0.98399999999999999},"min":{"1min":0.622,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.3620000000000001,"5min":1.512,"15min":6.008},"last":1.077},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.0289999999999999,"15min":1.0700000000000001},"min":{"1min":0.60299999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.4610000000000001,"5min":2.0619999999999998,"15min":2.3079999999999998},"last":0.82399999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.036,"15min":1.0660000000000001},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.46300000000000002},"max":{"1min":1.4199999999999999,"5min":1.5649999999999999,"15min":5.9560000000000004},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.022,"5min":1.0800000000000001,"15min":1.1299999999999999},"min":{"1min":0.48199999999999998,"5min":0.47599999999999998,"15min":0.46500000000000002},"max":{"1min":1.4019999999999999,"5min":2.145,"15min":3.5169999999999999},"last":1.002}]},{"osd":7,"last update":"Tue Sep 10 19:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.0369999999999999,"15min":1.0629999999999999},"min":{"1min":0.57399999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.3340000000000001,"5min":1.6040000000000001,"15min":2.069},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1799999999999999,"15min":1.2010000000000001},"min":{"1min":0.78500000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4950000000000001,"5min":1.7789999999999999,"15min":6.0380000000000003},"last":1.226}]}]},{"osd":5,"up_from":43,"seq":184683593992,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302384,"kb_used_data":5024,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93442768,"statfs":{"total":95995035648,"available":95685394432,"internally_reserved":0,"allocated":5144576,"data_stored":3557591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:08:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.0009999999999999,"15min":1.0089999999999999},"min":{"1min":0.66600000000000004,"5min":0.46500000000000002,"15min":0.36399999999999999},"max":{"1min":1.355,"5min":5.4640000000000004,"15min":8.3949999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.873,"5min":0.98099999999999998,"15min":0.95299999999999996},"min":{"1min":0.65500000000000003,"5min":0.53800000000000003,"15min":0.39700000000000002},"max":{"1min":1.2430000000000001,"5min":5.4790000000000001,"15min":7.3070000000000004},"last":0.76700000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:07:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.014,"15min":1.2070000000000001},"min":{"1min":0.60399999999999998,"5min":0.56100000000000005,"15min":0.41199999999999998},"max":{"1min":1.5609999999999999,"5min":6.282,"15min":31.548999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.026,"15min":1.1180000000000001},"min":{"1min":0.72299999999999998,"5min":0.54800000000000004,"15min":0.48899999999999999},"max":{"1min":1.5900000000000001,"5min":6.0170000000000003,"15min":31.847000000000001},"last":0.876}]},{"osd":2,"last update":"Tue Sep 10 19:08:26 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.0309999999999999,"15min":1.292},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.39900000000000002},"max":{"1min":1.1819999999999999,"5min":7.9340000000000002,"15min":61.347000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0720000000000001,"15min":1.0569999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.52000000000000002},"max":{"1min":1.4419999999999999,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":0.80000000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:08:26 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.002,"15min":1.036},"min":{"1min":0.69499999999999995,"5min":0.60199999999999998,"15min":0.51700000000000002},"max":{"1min":1.1799999999999999,"5min":1.6220000000000001,"15min":7.5979999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.123,"15min":1.3140000000000001},"min":{"1min":0.69899999999999995,"5min":0.63400000000000001,"15min":0.60799999999999998},"max":{"1min":1.472,"5min":5.3810000000000002,"15min":53.173000000000002},"last":0.94899999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:08:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.97699999999999998,"15min":1.0229999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.36499999999999999},"max":{"1min":1.302,"5min":5.5170000000000003,"15min":13.005000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.96099999999999997,"15min":1.0589999999999999},"min":{"1min":0.56799999999999995,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3320000000000001,"5min":1.5,"15min":7.6299999999999999},"last":0.86099999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.137,"15min":1.202},"min":{"1min":0.61899999999999999,"5min":0.58999999999999997,"15min":0.56399999999999995},"max":{"1min":1.29,"5min":4.2880000000000003,"15min":7.7199999999999998},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1459999999999999,"15min":1.2390000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.42799999999999999},"max":{"1min":1.603,"5min":5.4240000000000004,"15min":9.6760000000000002},"last":1.0680000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.1359999999999999,"15min":1.2450000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.49,"5min":5.5510000000000002,"15min":15.613},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.268},"min":{"1min":0.67500000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4350000000000001,"5min":1.718,"15min":16.965},"last":1.117}]}]},{"osd":4,"up_from":35,"seq":150323855636,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308236,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93436916,"statfs":{"total":95995035648,"available":95679401984,"internally_reserved":0,"allocated":2617344,"data_stored":1002393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.371},"max":{"1min":0.94499999999999995,"5min":1.367,"15min":2.0179999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.995,"15min":0.94799999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.44600000000000001},"max":{"1min":1.202,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.90800000000000003,"15min":0.98199999999999998},"min":{"1min":0.57799999999999996,"5min":0.55900000000000005,"15min":0.42399999999999999},"max":{"1min":1.677,"5min":1.677,"15min":27.716000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.98099999999999998,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.38100000000000001},"max":{"1min":1.4950000000000001,"5min":1.865,"15min":27.93},"last":0.92100000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.98299999999999998,"15min":1.0309999999999999},"min":{"1min":0.57299999999999995,"5min":0.50800000000000001,"15min":0.46600000000000003},"max":{"1min":1.085,"5min":2.0640000000000001,"15min":18.853000000000002},"last":0.753},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.96899999999999997,"15min":1.056},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.497},"max":{"1min":1.512,"5min":1.587,"15min":25.652999999999999},"last":0.92600000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:08:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.0349999999999999,"15min":1.0029999999999999},"min":{"1min":0.58699999999999997,"5min":0.49099999999999999,"15min":0.46300000000000002},"max":{"1min":1.421,"5min":1.6879999999999999,"15min":4.2670000000000003},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.88,"5min":1.0029999999999999,"15min":0.98499999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.442},"max":{"1min":1.292,"5min":1.873,"15min":2.0739999999999998},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.92800000000000005,"15min":0.96399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.079,"5min":1.4319999999999999,"15min":1.8009999999999999},"last":1.073},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.079,"15min":1.069},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2709999999999999,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.1759999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:08:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.0189999999999999,"15min":1.04},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.3500000000000001,"5min":1.867,"15min":2.1699999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.1439999999999999,"15min":1.159},"min":{"1min":0.54300000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1879999999999999,"5min":1.9890000000000001,"15min":2.9590000000000001},"last":0.96699999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:08:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.1870000000000001,"15min":1.167},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46899999999999997},"max":{"1min":1.589,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.2010000000000001,"15min":1.181},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.40899999999999997},"max":{"1min":1.401,"5min":3.7850000000000001,"15min":3.7850000000000001},"last":1.1259999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051871,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304548,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93440604,"statfs":{"total":95995035648,"available":95683178496,"internally_reserved":0,"allocated":3100672,"data_stored":1507279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.69399999999999995,"15min":0.85299999999999998},"min":{"1min":0.432,"5min":0.313,"15min":0.222},"max":{"1min":1.3480000000000001,"5min":1.6539999999999999,"15min":26.594000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.75800000000000001,"15min":0.90300000000000002},"min":{"1min":0.54500000000000004,"5min":0.40400000000000003,"15min":0.252},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":24.204999999999998},"last":0.60699999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:06:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.70599999999999996,"15min":0.92200000000000004},"min":{"1min":0.46500000000000002,"5min":0.34599999999999997,"15min":0.26500000000000001},"max":{"1min":1.101,"5min":1.3240000000000001,"15min":53.661999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77900000000000003,"15min":0.88300000000000001},"min":{"1min":0.60199999999999998,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":1.071,"5min":1.7170000000000001,"15min":15.189},"last":0.875}]},{"osd":2,"last update":"Tue Sep 10 19:07:41 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.90000000000000002,"15min":1.032},"min":{"1min":0.72799999999999998,"5min":0.496,"15min":0.32200000000000001},"max":{"1min":1.409,"5min":2.073,"15min":26.558},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.879,"5min":0.80900000000000005,"15min":0.95999999999999996},"min":{"1min":0.59599999999999997,"5min":0.497,"15min":0.26900000000000002},"max":{"1min":1.766,"5min":1.766,"15min":25.536999999999999},"last":0.66900000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:08:35 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":1.111,"15min":1.173},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.46999999999999997},"max":{"1min":1.226,"5min":2.5449999999999999,"15min":10.359999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.077,"5min":1.228,"15min":1.2829999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.53500000000000003},"max":{"1min":1.956,"5min":2.6019999999999999,"15min":10.391},"last":0.89000000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1319999999999999,"15min":1.2290000000000001},"min":{"1min":0.82999999999999996,"5min":0.67000000000000004,"15min":0.501},"max":{"1min":1.7709999999999999,"5min":2.2770000000000001,"15min":17.535},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.034,"15min":1.0589999999999999},"min":{"1min":0.73799999999999999,"5min":0.53500000000000003,"15min":0.45900000000000002},"max":{"1min":1.8169999999999999,"5min":2.0529999999999999,"15min":8.7880000000000003},"last":1.0029999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:08:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.2030000000000001,"15min":1.21},"min":{"1min":0.76300000000000001,"5min":0.72199999999999998,"15min":0.55600000000000005},"max":{"1min":1.8859999999999999,"5min":2.206,"15min":10.51},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.097,"5min":1.1259999999999999,"15min":1.1200000000000001},"min":{"1min":0.68300000000000005,"5min":0.64200000000000002,"15min":0.48499999999999999},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":10.773999999999999},"last":0.748}]},{"osd":7,"last update":"Tue Sep 10 19:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.145,"15min":1.1839999999999999},"min":{"1min":0.84999999999999998,"5min":0.61499999999999999,"15min":0.45600000000000002},"max":{"1min":1.986,"5min":1.986,"15min":18.312999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.226,"5min":1.2669999999999999,"15min":1.3300000000000001},"min":{"1min":0.77100000000000002,"5min":0.66000000000000003,"15min":0.57299999999999995},"max":{"1min":2.0830000000000002,"5min":2.2469999999999999,"15min":10.611000000000001},"last":0.93899999999999995}]}]},{"osd":2,"up_from":19,"seq":81604378920,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308436,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93436716,"statfs":{"total":95995035648,"available":95679197184,"internally_reserved":0,"allocated":2822144,"data_stored":1228011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:08:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.1499999999999999,"15min":0.89200000000000002},"min":{"1min":0.437,"5min":0.311,"15min":0.219},"max":{"1min":0.97399999999999998,"5min":43.927999999999997,"15min":43.927999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.3180000000000001,"15min":0.97799999999999998},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.23599999999999999},"max":{"1min":1.046,"5min":49.475000000000001,"15min":49.475000000000001},"last":0.66300000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78400000000000003,"15min":0.751},"min":{"1min":0.51800000000000002,"5min":0.36499999999999999,"15min":0.26600000000000001},"max":{"1min":0.996,"5min":3.524,"15min":5.516},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.56799999999999995,"5min":0.40300000000000002,"15min":0.28299999999999997},"max":{"1min":1.135,"5min":1.77,"15min":9.0649999999999995},"last":0.624}]},{"osd":3,"last update":"Tue Sep 10 19:08:39 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.92300000000000004,"15min":0.88800000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.318},"max":{"1min":0.746,"5min":6.4020000000000001,"15min":6.4020000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.034,"15min":0.95199999999999996},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.377},"max":{"1min":1.1970000000000001,"5min":8.3710000000000004,"15min":15.736000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:08:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.395,"15min":1.222},"min":{"1min":0.55200000000000005,"5min":0.44900000000000001,"15min":0.39300000000000002},"max":{"1min":1.401,"5min":38.079999999999998,"15min":38.079999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.879,"5min":1.4390000000000001,"15min":1.26},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.40400000000000003},"max":{"1min":1.2090000000000001,"5min":37.991,"15min":37.991},"last":0.84699999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:08:21 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.1120000000000001,"15min":1.0920000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.38500000000000001},"max":{"1min":1.22,"5min":1.7749999999999999,"15min":5.1959999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.133,"15min":1.161},"min":{"1min":0.62,"5min":0.60899999999999999,"15min":0.45400000000000001},"max":{"1min":1.554,"5min":2.0390000000000001,"15min":21.654},"last":0.91000000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:08:21 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.5620000000000001,"15min":1.2869999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.503},"max":{"1min":1.2789999999999999,"5min":43.655999999999999,"15min":43.655999999999999},"last":1.04},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.538,"15min":1.2989999999999999},"min":{"1min":0.71099999999999997,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1759999999999999,"5min":45.255000000000003,"15min":45.255000000000003},"last":0.997}]},{"osd":7,"last update":"Tue Sep 10 19:08:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.1890000000000001,"15min":1.1910000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.48399999999999999},"max":{"1min":1.6040000000000001,"5min":2.2210000000000001,"15min":15.025},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.1519999999999999,"15min":1.23},"min":{"1min":0.75900000000000001,"5min":0.63400000000000001,"15min":0.53100000000000003},"max":{"1min":1.256,"5min":2.504,"15min":14.816000000000001},"last":0.88}]}]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305848,"kb_used_data":4332,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439304,"statfs":{"total":95995035648,"available":95681847296,"internally_reserved":0,"allocated":4435968,"data_stored":2891059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.70899999999999996,"15min":0.83699999999999997},"min":{"1min":0.46600000000000003,"5min":0.32900000000000001,"15min":0.23499999999999999},"max":{"1min":3.96,"5min":3.96,"15min":32.677999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.748,"15min":0.89600000000000002},"min":{"1min":0.38100000000000001,"5min":0.34399999999999997,"15min":0.253},"max":{"1min":2.9700000000000002,"5min":2.9700000000000002,"15min":32.398000000000003},"last":1.1020000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.72899999999999998,"15min":0.89600000000000002},"min":{"1min":0.432,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":26.097000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.996,"5min":0.85899999999999999,"15min":1.0149999999999999},"min":{"1min":0.45400000000000001,"5min":0.437,"15min":0.26000000000000001},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":32.363},"last":0.79900000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:06:43 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.5649999999999999,"15min":1.252},"min":{"1min":0.58499999999999996,"5min":0.309,"15min":0.28899999999999998},"max":{"1min":2.903,"5min":53.344999999999999,"15min":53.344999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.4319999999999999,"15min":1.2769999999999999},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.26600000000000001},"max":{"1min":4.085,"5min":53.645000000000003,"15min":60.999000000000002},"last":0.752}]},{"osd":4,"last update":"Tue Sep 10 19:06:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.0740000000000001,"15min":1.1419999999999999},"min":{"1min":0.73999999999999999,"5min":0.64900000000000002,"15min":0.40899999999999997},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":12.808999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.234,"5min":1.0049999999999999,"15min":1.2649999999999999},"min":{"1min":0.70099999999999996,"5min":0.57699999999999996,"15min":0.378},"max":{"1min":4.1379999999999999,"5min":4.1379999999999999,"15min":60.529000000000003},"last":0.92900000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.0309999999999999,"15min":1.143},"min":{"1min":0.71099999999999997,"5min":0.48899999999999999,"15min":0.40600000000000003},"max":{"1min":3.8410000000000002,"5min":3.8410000000000002,"15min":26.050999999999998},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.165,"5min":1.1439999999999999,"15min":1.224},"min":{"1min":0.68700000000000006,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":3.0310000000000001,"5min":3.0310000000000001,"15min":12.84},"last":0.95099999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.1619999999999999,"15min":1.3200000000000001},"min":{"1min":0.82799999999999996,"5min":0.63200000000000001,"15min":0.53800000000000003},"max":{"1min":3.1400000000000001,"5min":3.1400000000000001,"15min":25.608000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.085,"15min":1.2569999999999999},"min":{"1min":0.65200000000000002,"5min":0.57699999999999996,"15min":0.46400000000000002},"max":{"1min":4.1710000000000003,"5min":4.1710000000000003,"15min":33.643000000000001},"last":1.165}]},{"osd":7,"last update":"Tue Sep 10 19:07:18 2024","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.2070000000000001,"15min":1.341},"min":{"1min":0.746,"5min":0.746,"15min":0.48499999999999999},"max":{"1min":3.8700000000000001,"5min":3.8700000000000001,"15min":25.565999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.1120000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":4.218,"5min":4.218,"15min":33.673999999999999},"last":1.216}]}]},{"osd":0,"up_from":8,"seq":34359738687,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306028,"kb_used_data":4512,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93439124,"statfs":{"total":95995035648,"available":95681662976,"internally_reserved":0,"allocated":4620288,"data_stored":3032617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.72599999999999998,"15min":1.032},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.26300000000000001},"max":{"1min":0.89400000000000002,"5min":2.133,"15min":80.710999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.80700000000000005,"15min":1.1160000000000001},"min":{"1min":0.36099999999999999,"5min":0.33200000000000002,"15min":0.25600000000000001},"max":{"1min":1.2430000000000001,"5min":1.419,"15min":86.009},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.77000000000000002,"15min":1.0580000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.216},"max":{"1min":1.01,"5min":2.3530000000000002,"15min":36.682000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.79300000000000004,"15min":0.94999999999999996},"min":{"1min":0.442,"5min":0.41299999999999998,"15min":0.23599999999999999},"max":{"1min":1.0669999999999999,"5min":1.294,"15min":24.614000000000001},"last":0.52600000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.81899999999999995,"15min":0.93600000000000005},"min":{"1min":0.52200000000000002,"5min":0.46300000000000002,"15min":0.26400000000000001},"max":{"1min":1.79,"5min":1.79,"15min":21.620999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.88200000000000001,"15min":1.0700000000000001},"min":{"1min":0.51000000000000001,"5min":0.42699999999999999,"15min":0.30299999999999999},"max":{"1min":1.4990000000000001,"5min":2.4569999999999999,"15min":45.765000000000001},"last":0.625}]},{"osd":4,"last update":"Tue Sep 10 19:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":1.042,"15min":1.095},"min":{"1min":0.65000000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2150000000000001,"5min":1.6599999999999999,"15min":6.3630000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.1459999999999999,"15min":1.143},"min":{"1min":0.79000000000000004,"5min":0.626,"15min":0.51100000000000001},"max":{"1min":1.2869999999999999,"5min":1.8979999999999999,"15min":4.2389999999999999},"last":0.88600000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.097,"15min":1.153},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.442},"max":{"1min":1.629,"5min":2.5880000000000001,"15min":10.048},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.077,"5min":1.1639999999999999,"15min":1.1359999999999999},"min":{"1min":0.66800000000000004,"5min":0.62,"15min":0.42999999999999999},"max":{"1min":1.643,"5min":1.643,"15min":3.8959999999999999},"last":1.1379999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:08:22 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.2110000000000001,"15min":1.1839999999999999},"min":{"1min":0.78000000000000003,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4399999999999999,"5min":1.71,"15min":3.9550000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.153,"5min":1.165,"15min":1.218},"min":{"1min":0.92700000000000005,"5min":0.57399999999999995,"15min":0.48999999999999999},"max":{"1min":1.5229999999999999,"5min":2.5339999999999998,"15min":11.894},"last":0.86499999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:08:07 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.2,"15min":1.194},"min":{"1min":0.95999999999999996,"5min":0.61599999999999999,"15min":0.48499999999999999},"max":{"1min":1.9570000000000001,"5min":2.343,"15min":6.2949999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2649999999999999,"15min":1.2809999999999999},"min":{"1min":0.85799999999999998,"5min":0.56000000000000005,"15min":0.54900000000000004},"max":{"1min":1.7330000000000001,"5min":1.8120000000000001,"15min":8.2840000000000007},"last":1.0640000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":2129920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2015232,"data_stored":2015232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2064384,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":241,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:08:44.316 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-10T19:08:44.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:44.350 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:44.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:44.480 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:44.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:44.615 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:44.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:44.752 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:44.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:44.888 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:44.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.023 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: audit 2024-09-10T19:08:43.439309+0000 mon.a (mon.0) 3743 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: cluster 2024-09-10T19:08:43.439760+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e840: 8 total, 7 up, 8 in 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: audit 2024-09-10T19:08:43.506059+0000 mon.c (mon.2) 876 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: audit 2024-09-10T19:08:43.506694+0000 mon.a (mon.0) 3745 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: audit 2024-09-10T19:08:43.507731+0000 mon.c (mon.2) 877 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: audit 2024-09-10T19:08:43.508341+0000 mon.a (mon.0) 3746 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.397678+0000 mon.a (mon.0) 3747 : cluster [WRN] Health check update: Degraded data redundancy: 85/720 objects degraded (11.806%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.397727+0000 mon.a (mon.0) 3748 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.399207+0000 mgr.y (mgr.24526) 760 : cluster [DBG] pgmap v1308: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:45.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:44 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.409384+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e841: 8 total, 7 up, 8 in 2024-09-10T19:08:45.127 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.159 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:45.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.290 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: audit 2024-09-10T19:08:43.439309+0000 mon.a (mon.0) 3743 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: cluster 2024-09-10T19:08:43.439760+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e840: 8 total, 7 up, 8 in 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: audit 2024-09-10T19:08:43.506059+0000 mon.c (mon.2) 876 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: audit 2024-09-10T19:08:43.506694+0000 mon.a (mon.0) 3745 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: audit 2024-09-10T19:08:43.507731+0000 mon.c (mon.2) 877 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: audit 2024-09-10T19:08:43.508341+0000 mon.a (mon.0) 3746 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.397678+0000 mon.a (mon.0) 3747 : cluster [WRN] Health check update: Degraded data redundancy: 85/720 objects degraded (11.806%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.397727+0000 mon.a (mon.0) 3748 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.399207+0000 mgr.y (mgr.24526) 760 : cluster [DBG] pgmap v1308: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.409384+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e841: 8 total, 7 up, 8 in 2024-09-10T19:08:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: audit 2024-09-10T19:08:43.439309+0000 mon.a (mon.0) 3743 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: cluster 2024-09-10T19:08:43.439760+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e840: 8 total, 7 up, 8 in 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: audit 2024-09-10T19:08:43.506059+0000 mon.c (mon.2) 876 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: audit 2024-09-10T19:08:43.506694+0000 mon.a (mon.0) 3745 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: audit 2024-09-10T19:08:43.507731+0000 mon.c (mon.2) 877 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: audit 2024-09-10T19:08:43.508341+0000 mon.a (mon.0) 3746 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-152"}]: dispatch 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.397678+0000 mon.a (mon.0) 3747 : cluster [WRN] Health check update: Degraded data redundancy: 85/720 objects degraded (11.806%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.397727+0000 mon.a (mon.0) 3748 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.399207+0000 mgr.y (mgr.24526) 760 : cluster [DBG] pgmap v1308: 214 pgs: 32 unknown, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 85/720 objects degraded (11.806%) 2024-09-10T19:08:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:44 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.409384+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e841: 8 total, 7 up, 8 in 2024-09-10T19:08:45.295 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:45.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.431 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:45.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.565 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:45.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:45.701 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:45.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.838 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:45.941 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:45.945 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.270135+0000 osd.4 (osd.4) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:45.945 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: cluster 2024-09-10T19:08:44.270780+0000 osd.4 (osd.4) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:45.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: audit 2024-09-10T19:08:44.863584+0000 mon.a (mon.0) 3750 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:45.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: audit 2024-09-10T19:08:44.865464+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:45.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: cluster 2024-09-10T19:08:45.412878+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e842: 8 total, 7 up, 8 in 2024-09-10T19:08:45.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: audit 2024-09-10T19:08:45.418910+0000 mon.c (mon.2) 879 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:45 smithi165 bash[22591]: audit 2024-09-10T19:08:45.420047+0000 mon.a (mon.0) 3752 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.953 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.270135+0000 osd.4 (osd.4) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: cluster 2024-09-10T19:08:44.270780+0000 osd.4 (osd.4) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: audit 2024-09-10T19:08:44.863584+0000 mon.a (mon.0) 3750 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: audit 2024-09-10T19:08:44.865464+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: cluster 2024-09-10T19:08:45.412878+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e842: 8 total, 7 up, 8 in 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: audit 2024-09-10T19:08:45.418910+0000 mon.c (mon.2) 879 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[18629]: audit 2024-09-10T19:08:45.420047+0000 mon.a (mon.0) 3752 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.270135+0000 osd.4 (osd.4) 418 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: cluster 2024-09-10T19:08:44.270780+0000 osd.4 (osd.4) 419 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:45.954 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: audit 2024-09-10T19:08:44.863584+0000 mon.a (mon.0) 3750 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:08:45.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: audit 2024-09-10T19:08:44.865464+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:45.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: cluster 2024-09-10T19:08:45.412878+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e842: 8 total, 7 up, 8 in 2024-09-10T19:08:45.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: audit 2024-09-10T19:08:45.418910+0000 mon.c (mon.2) 879 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.955 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:45 smithi083 bash[32381]: audit 2024-09-10T19:08:45.420047+0000 mon.a (mon.0) 3752 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.971 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:45.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:46.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.108 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:46.211 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:46.245 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:46.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:46.380 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:46.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.514 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:46.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:46.651 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:46.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.788 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:46.805 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:46.806 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:46.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.923 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:46.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:47.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:47.062 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:47.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: cluster 2024-09-10T19:08:45.904828+0000 osd.3 (osd.3) 450 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: cluster 2024-09-10T19:08:45.905336+0000 osd.3 (osd.3) 451 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: cluster 2024-09-10T19:08:46.402106+0000 mgr.y (mgr.24526) 761 : cluster [DBG] pgmap v1311: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.416035+0000 mon.a (mon.0) 3753 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: cluster 2024-09-10T19:08:46.416222+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e843: 8 total, 7 up, 8 in 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.423281+0000 mon.c (mon.2) 880 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.424025+0000 mon.a (mon.0) 3755 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.452785+0000 mon.c (mon.2) 881 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.453260+0000 mon.a (mon.0) 3756 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.453990+0000 mon.c (mon.2) 882 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.176 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:46 smithi165 bash[22591]: audit 2024-09-10T19:08:46.454450+0000 mon.a (mon.0) 3757 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.195 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: cluster 2024-09-10T19:08:45.904828+0000 osd.3 (osd.3) 450 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: cluster 2024-09-10T19:08:45.905336+0000 osd.3 (osd.3) 451 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: cluster 2024-09-10T19:08:46.402106+0000 mgr.y (mgr.24526) 761 : cluster [DBG] pgmap v1311: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.416035+0000 mon.a (mon.0) 3753 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: cluster 2024-09-10T19:08:46.416222+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e843: 8 total, 7 up, 8 in 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.423281+0000 mon.c (mon.2) 880 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.424025+0000 mon.a (mon.0) 3755 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.452785+0000 mon.c (mon.2) 881 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.453260+0000 mon.a (mon.0) 3756 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.453990+0000 mon.c (mon.2) 882 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[18629]: audit 2024-09-10T19:08:46.454450+0000 mon.a (mon.0) 3757 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: cluster 2024-09-10T19:08:45.904828+0000 osd.3 (osd.3) 450 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: cluster 2024-09-10T19:08:45.905336+0000 osd.3 (osd.3) 451 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: cluster 2024-09-10T19:08:46.402106+0000 mgr.y (mgr.24526) 761 : cluster [DBG] pgmap v1311: 214 pgs: 32 creating+peering, 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.416035+0000 mon.a (mon.0) 3753 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-77828-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: cluster 2024-09-10T19:08:46.416222+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e843: 8 total, 7 up, 8 in 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.423281+0000 mon.c (mon.2) 880 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.424025+0000 mon.a (mon.0) 3755 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi083-77828-111","var": "dedup_tier","val": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.452785+0000 mon.c (mon.2) 881 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.453260+0000 mon.a (mon.0) 3756 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:47.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.453990+0000 mon.c (mon.2) 882 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.293 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:46 smithi083 bash[32381]: audit 2024-09-10T19:08:46.454450+0000 mon.a (mon.0) 3757 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi083-77828-111", "tierpool": "test-rados-api-smithi083-77828-154"}]: dispatch 2024-09-10T19:08:47.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:47.327 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:47.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:47.456 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:47.559 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:47.591 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:47.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:47.728 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:47.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:47.866 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:47.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:47.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:47.999 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:48.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:48.040 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-10T19:08:48.040 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:08:48.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:48.125 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:48.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:48.260 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:48.364 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:48.402 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:48.473 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:48 smithi165 bash[22591]: cluster 2024-09-10T19:08:47.373177+0000 osd.5 (osd.5) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:48.473 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:48 smithi165 bash[22591]: cluster 2024-09-10T19:08:47.373480+0000 osd.5 (osd.5) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:48.473 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:48 smithi165 bash[22591]: cluster 2024-09-10T19:08:47.419279+0000 mon.a (mon.0) 3758 : cluster [DBG] osdmap e844: 8 total, 7 up, 8 in 2024-09-10T19:08:48.474 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:48 smithi165 bash[22591]: cluster 2024-09-10T19:08:48.402989+0000 mgr.y (mgr.24526) 762 : cluster [DBG] pgmap v1314: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:48.506 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:48.541 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:48.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:48.644 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:48.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[18629]: cluster 2024-09-10T19:08:47.373177+0000 osd.5 (osd.5) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:48.658 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[18629]: cluster 2024-09-10T19:08:47.373480+0000 osd.5 (osd.5) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[18629]: cluster 2024-09-10T19:08:47.419279+0000 mon.a (mon.0) 3758 : cluster [DBG] osdmap e844: 8 total, 7 up, 8 in 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[18629]: cluster 2024-09-10T19:08:48.402989+0000 mgr.y (mgr.24526) 762 : cluster [DBG] pgmap v1314: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[32381]: cluster 2024-09-10T19:08:47.373177+0000 osd.5 (osd.5) 428 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[32381]: cluster 2024-09-10T19:08:47.373480+0000 osd.5 (osd.5) 429 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[32381]: cluster 2024-09-10T19:08:47.419279+0000 mon.a (mon.0) 3758 : cluster [DBG] osdmap e844: 8 total, 7 up, 8 in 2024-09-10T19:08:48.659 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:48 smithi083 bash[32381]: cluster 2024-09-10T19:08:48.402989+0000 mgr.y (mgr.24526) 762 : cluster [DBG] pgmap v1314: 182 pgs: 46 active+undersized, 18 active+undersized+degraded, 118 active+clean; 4.4 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s; 85/723 objects degraded (11.757%) 2024-09-10T19:08:48.677 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:48.728 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:08:48.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:48.808 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:48.835 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:48.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:48.938 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:48.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:49.042 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:49.075 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:49.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:49.213 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:49.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:49.352 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:08:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:08:49.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.468 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: cluster 2024-09-10T19:08:48.425220+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e845: 8 total, 7 up, 8 in 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: audit 2024-09-10T19:08:48.430939+0000 mon.c (mon.2) 883 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: audit 2024-09-10T19:08:48.431312+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: cluster 2024-09-10T19:08:48.435908+0000 osd.7 (osd.7) 432 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: cluster 2024-09-10T19:08:48.436275+0000 osd.7 (osd.7) 433 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: audit 2024-09-10T19:08:48.472204+0000 mgr.y (mgr.24526) 763 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:49.469 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:49 smithi165 bash[22591]: cluster 2024-09-10T19:08:49.400562+0000 mon.a (mon.0) 3761 : cluster [WRN] Health check update: Degraded data redundancy: 85/723 objects degraded (11.757%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:49.488 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:49.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.626 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:49.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: cluster 2024-09-10T19:08:48.425220+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e845: 8 total, 7 up, 8 in 2024-09-10T19:08:49.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: audit 2024-09-10T19:08:48.430939+0000 mon.c (mon.2) 883 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.700 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: audit 2024-09-10T19:08:48.431312+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: cluster 2024-09-10T19:08:48.435908+0000 osd.7 (osd.7) 432 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: cluster 2024-09-10T19:08:48.436275+0000 osd.7 (osd.7) 433 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: audit 2024-09-10T19:08:48.472204+0000 mgr.y (mgr.24526) 763 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[18629]: cluster 2024-09-10T19:08:49.400562+0000 mon.a (mon.0) 3761 : cluster [WRN] Health check update: Degraded data redundancy: 85/723 objects degraded (11.757%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: cluster 2024-09-10T19:08:48.425220+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e845: 8 total, 7 up, 8 in 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: audit 2024-09-10T19:08:48.430939+0000 mon.c (mon.2) 883 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.701 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: audit 2024-09-10T19:08:48.431312+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:49.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: cluster 2024-09-10T19:08:48.435908+0000 osd.7 (osd.7) 432 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:08:49.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: cluster 2024-09-10T19:08:48.436275+0000 osd.7 (osd.7) 433 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:08:49.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: audit 2024-09-10T19:08:48.472204+0000 mgr.y (mgr.24526) 763 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:49.702 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:49 smithi083 bash[32381]: cluster 2024-09-10T19:08:49.400562+0000 mon.a (mon.0) 3761 : cluster [WRN] Health check update: Degraded data redundancy: 85/723 objects degraded (11.757%), 18 pgs degraded, 64 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:49.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.765 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:49.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:49.903 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:49.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:50.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:50.041 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:08:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:08:50.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:50.168 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:08:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:08:50.270 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:50.305 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:50.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:50.409 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:50.444 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:08:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:08:50.460 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8231 ms) 2024-09-10T19:08:50.460 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-10T19:08:50.460 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12724 ms) 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7997 ms) 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13001 ms) 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8001 ms) 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8002 ms) 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: ok, hit_set contains 255:602f83fe:::foo:head 2024-09-10T19:08:50.461 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9075 ms) 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,0 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first is 1725995246 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,0 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,0 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,0 2024-09-10T19:08:50.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,1725995254,1725995255,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,1725995254,1725995255,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995246,1725995248,1725995249,1725995251,1725995252,1725995254,1725995255,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1725995249,1725995251,1725995252,1725995254,1725995255,1725995257,1725995258,0 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first now 1725995249, trimmed 2024-09-10T19:08:50.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20940 ms) 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: foo0 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14112 ms) 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18146 ms) 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21998 ms) 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4788 ms) 2024-09-10T19:08:50.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3008 ms) 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2983 ms) 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3009 ms) 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231107 ms total) 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (864646 ms total) 2024-09-10T19:08:50.465 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-10T19:08:50.466 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77971 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77971 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77683 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77683 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77667 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77667 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77736 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77736 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.467 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77999 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77999 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78186 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78186 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77893 2024-09-10T19:08:50.468 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77893 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77643 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77643 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78280 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78280 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78088 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78088 2024-09-10T19:08:50.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77949 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77949 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78272 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78272 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77675 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77675 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78109 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78109 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77659 2024-09-10T19:08:50.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77659 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78018 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78018 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77781 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77781 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=78079 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 78079 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=77933 2024-09-10T19:08:50.473 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 77933 2024-09-10T19:08:50.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ exit 0 2024-09-10T19:08:50.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ cleanup 2024-09-10T19:08:50.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ pkill -P 77630 2024-09-10T19:08:50.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ true 2024-09-10T19:08:50.474 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T19:08:50.474 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T19:08:50.487 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-10T19:08:50.487 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T19:08:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[18629]: audit 2024-09-10T19:08:49.436779+0000 mon.a (mon.0) 3762 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[18629]: cluster 2024-09-10T19:08:49.436859+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e846: 8 total, 7 up, 8 in 2024-09-10T19:08:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[18629]: audit 2024-09-10T19:08:49.437943+0000 mon.c (mon.2) 884 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[18629]: audit 2024-09-10T19:08:49.438287+0000 mon.a (mon.0) 3764 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[18629]: cluster 2024-09-10T19:08:50.404511+0000 mgr.y (mgr.24526) 764 : cluster [DBG] pgmap v1317: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:50.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[32381]: audit 2024-09-10T19:08:49.436779+0000 mon.a (mon.0) 3762 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:50.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[32381]: cluster 2024-09-10T19:08:49.436859+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e846: 8 total, 7 up, 8 in 2024-09-10T19:08:50.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[32381]: audit 2024-09-10T19:08:49.437943+0000 mon.c (mon.2) 884 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[32381]: audit 2024-09-10T19:08:49.438287+0000 mon.a (mon.0) 3764 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:50 smithi083 bash[32381]: cluster 2024-09-10T19:08:50.404511+0000 mgr.y (mgr.24526) 764 : cluster [DBG] pgmap v1317: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:50.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:50.576 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:08:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:08:50.680 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:50.709 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:50.762 DEBUG:teuthology.parallel:result is None 2024-09-10T19:08:50.763 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:08:50.774 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:08:50.774 DEBUG:teuthology.orchestra.run.smithi083:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:08:50.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:50.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:50 smithi165 bash[22591]: audit 2024-09-10T19:08:49.436779+0000 mon.a (mon.0) 3762 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:50.827 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:50 smithi165 bash[22591]: cluster 2024-09-10T19:08:49.436859+0000 mon.a (mon.0) 3763 : cluster [DBG] osdmap e846: 8 total, 7 up, 8 in 2024-09-10T19:08:50.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:50 smithi165 bash[22591]: audit 2024-09-10T19:08:49.437943+0000 mon.c (mon.2) 884 : audit [INF] from='client.? v1:172.21.15.83:0/3778548406' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:50 smithi165 bash[22591]: audit 2024-09-10T19:08:49.438287+0000 mon.a (mon.0) 3764 : audit [INF] from='client.49694 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]: dispatch 2024-09-10T19:08:50.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:50 smithi165 bash[22591]: cluster 2024-09-10T19:08:50.404511+0000 mgr.y (mgr.24526) 764 : cluster [DBG] pgmap v1317: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:50.848 INFO:teuthology.orchestra.run.smithi165.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:08:50.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:08:50.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:50.981 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:08:50.982 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-10T19:08:50.995 INFO:tasks.thrashosds:joining thrashosds 2024-09-10T19:08:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:08:50.997 INFO:teuthology.orchestra.run.smithi083.stderr:Error response from daemon: Cannot kill container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:08:50.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi083 with status 1: 'sudo docker kill -s 1 ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[18629]: cluster 2024-09-10T19:08:50.431385+0000 mon.a (mon.0) 3765 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[18629]: audit 2024-09-10T19:08:50.449300+0000 mon.a (mon.0) 3766 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[18629]: cluster 2024-09-10T19:08:50.449413+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e847: 8 total, 7 up, 8 in 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[32381]: cluster 2024-09-10T19:08:50.431385+0000 mon.a (mon.0) 3765 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[32381]: audit 2024-09-10T19:08:50.449300+0000 mon.a (mon.0) 3766 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:51 smithi083 bash[32381]: cluster 2024-09-10T19:08:50.449413+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e847: 8 total, 7 up, 8 in 2024-09-10T19:08:51.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:08:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:51 smithi165 bash[22591]: cluster 2024-09-10T19:08:50.431385+0000 mon.a (mon.0) 3765 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:51 smithi165 bash[22591]: audit 2024-09-10T19:08:50.449300+0000 mon.a (mon.0) 3766 : audit [INF] from='client.49694 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi083-77828-111"}]': finished 2024-09-10T19:08:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:51 smithi165 bash[22591]: cluster 2024-09-10T19:08:50.449413+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e847: 8 total, 7 up, 8 in 2024-09-10T19:08:53.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:08:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:08:52] "GET /metrics HTTP/1.1" 200 226721 "" "Prometheus/2.43.0" 2024-09-10T19:08:53.510 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:53.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:53 smithi083 bash[18629]: cluster 2024-09-10T19:08:52.407006+0000 mgr.y (mgr.24526) 765 : cluster [DBG] pgmap v1319: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:53.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:53 smithi083 bash[32381]: cluster 2024-09-10T19:08:52.407006+0000 mgr.y (mgr.24526) 765 : cluster [DBG] pgmap v1319: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:53 smithi165 bash[22591]: cluster 2024-09-10T19:08:52.407006+0000 mgr.y (mgr.24526) 765 : cluster [DBG] pgmap v1319: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:54.728 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:54 smithi083 bash[18629]: cluster 2024-09-10T19:08:54.403246+0000 mon.a (mon.0) 3768 : cluster [WRN] Health check update: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:54.728 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:54 smithi083 bash[32381]: cluster 2024-09-10T19:08:54.403246+0000 mon.a (mon.0) 3768 : cluster [WRN] Health check update: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:54.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:54 smithi165 bash[22591]: cluster 2024-09-10T19:08:54.403246+0000 mon.a (mon.0) 3768 : cluster [WRN] Health check update: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized (PG_DEGRADED) 2024-09-10T19:08:55.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.539+0000 7f4b62e07640 1 -- 172.21.15.83:0/3291254339 >> v1:172.21.15.83:6789/0 conn(0x7f4b5c108f40 legacy=0x7f4b5c10b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:55.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.539+0000 7f4b62e07640 1 -- 172.21.15.83:0/3291254339 shutdown_connections 2024-09-10T19:08:55.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.539+0000 7f4b62e07640 1 -- 172.21.15.83:0/3291254339 >> 172.21.15.83:0/3291254339 conn(0x7f4b5c074a20 msgr2=0x7f4b5c0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:55.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.539+0000 7f4b62e07640 1 -- 172.21.15.83:0/3291254339 shutdown_connections 2024-09-10T19:08:55.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.539+0000 7f4b62e07640 1 -- 172.21.15.83:0/3291254339 wait complete. 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 librados: starting msgr at 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 librados: starting objecter 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 Processor -- start 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- start start 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 librados: setting wanted keys 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 librados: calling monclient init 2024-09-10T19:08:55.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b5c11c080 con 0x7f4b5c1053e0 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b5c11d280 con 0x7f4b5c102600 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b5c11e480 con 0x7f4b5c108f40 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b5bfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f4b5c1053e0 0x7f4b5c0fed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49334/0 (socket says 172.21.15.83:49334) 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b5bfff640 1 -- 172.21.15.83:0/479908533 learned_addr learned my addr 172.21.15.83:0/479908533 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter ms_handle_connect 0x7f4b5c1053e0 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter ms_handle_connect 0x7f4b5c108f40 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter ms_handle_connect 0x7f4b5c102600 2024-09-10T19:08:55.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:55.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289876046 0 0) 0x7f4b5c11c080 con 0x7f4b5c1053e0 2024-09-10T19:08:55.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b44003720 con 0x7f4b5c1053e0 2024-09-10T19:08:55.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908610429 0 0) 0x7f4b5c11e480 con 0x7f4b5c108f40 2024-09-10T19:08:55.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c11c080 con 0x7f4b5c108f40 2024-09-10T19:08:55.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255603164 0 0) 0x7f4b5c11d280 con 0x7f4b5c102600 2024-09-10T19:08:55.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c11e480 con 0x7f4b5c102600 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357697620 0 0) 0x7f4b44003720 con 0x7f4b5c1053e0 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b5c11d280 con 0x7f4b5c1053e0 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157175136 0 0) 0x7f4b5c11c080 con 0x7f4b5c108f40 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b44003720 con 0x7f4b5c108f40 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4b4c001cf0 con 0x7f4b5c1053e0 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4b54002d00 con 0x7f4b5c108f40 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2846395127 0 0) 0x7f4b5c11e480 con 0x7f4b5c102600 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b5c11c080 con 0x7f4b5c102600 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2425224349 0 0) 0x7f4b5c11d280 con 0x7f4b5c1053e0 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 >> v1:172.21.15.83:6790/0 conn(0x7f4b5c108f40 legacy=0x7f4b5c11a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:55.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 >> v1:172.21.15.165:6789/0 conn(0x7f4b5c102600 legacy=0x7f4b5c0fe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c11f680 con 0x7f4b5c1053e0 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c11e670 con 0x7f4b5c1053e0 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 10 client.25803.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c11eac0 con 0x7f4b5c1053e0 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.543+0000 7f4b62e07640 1 librados: init done 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4b4c003050 con 0x7f4b5c1053e0 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4b4c005d80 con 0x7f4b5c1053e0 2024-09-10T19:08:55.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4b48004ff0 con 0x7f4b5c1053e0 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4b4c002190 con 0x7f4b5c1053e0 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(847..847 src has 750..847) v4 ==== 7363+0+0 (unknown 4127979390 0 0) 0x7f4b4c08a560 con 0x7f4b5c1053e0 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 osd_map(847..847 src has 750..847) v4 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 3 client.25803.objecter handle_osd_map got epochs [847,847] > 0 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 3 client.25803.objecter handle_osd_map decoding full epoch 847 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 20 client.25803.objecter dump_active .. 0 homeless 2024-09-10T19:08:55.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v847) v1 ==== 54+0+16 (unknown 4099917860 0 2420000192) 0x7f4b4c08c950 con 0x7f4b5c1053e0 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.547+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b44075c00 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 10 client.25803.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 1 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 1 2024-09-10T19:08:55.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 0 1 2024-09-10T19:08:55.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 4 2024-09-10T19:08:55.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 10 client.25803.objecter _send_command 1 2024-09-10T19:08:55.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b48005290 con 0x7f4b48004610 2024-09-10T19:08:55.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b48004610 2024-09-10T19:08:55.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b48005290 con 0x7f4b48004610 2024-09-10T19:08:55.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:08:55.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 10 client.25803.objecter _finish_command 1 = system:0 2024-09-10T19:08:55.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 4 2024-09-10T19:08:55.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.551+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 0 1 2024-09-10T19:08:55.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 10 client.25803.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 2 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 2 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48005810 osd=0 3 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 0 2 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 4 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 10 client.25803.objecter _send_command 2 2024-09-10T19:08:55.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b48004cc0 con 0x7f4b48004610 2024-09-10T19:08:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b48004cc0 con 0x7f4b48004610 2024-09-10T19:08:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 10 client.25803.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48005810 osd=0 4 2024-09-10T19:08:55.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 0 2 2024-09-10T19:08:55.574 INFO:teuthology.orchestra.run.smithi083.stdout:osd.0: {} 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 10 client.25803.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 3 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 20 client.25803.objecter _get_session osd=-1 returning homeless 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 10 client.25803.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=848}) v3 -- 0x7f4b48003fa0 con 0x7f4b5c1053e0 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b48004380 con 0x7f4b5c1053e0 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_get_version_reply(handle=1 version=847) v2 ==== 24+0+0 (unknown 335845864 0 0) 0x7f4b4c002cc0 con 0x7f4b5c1053e0 2024-09-10T19:08:55.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b5affd640 10 client.25803.objecter _check_command_map_dne tid 3 current 847 map_dne_bound 847 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b5affd640 10 client.25803.objecter _finish_command 3 = osd:6 osd down 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.567+0000 7f4b5affd640 15 client.25803.objecter _session_command_op_remove -1 3 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 10 client.25803.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 4 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 4 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 2 4 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 4 2024-09-10T19:08:55.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 10 client.25803.objecter _send_command 4 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6809/386094220 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b4800c780 con 0x7f4b48002150 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b48002150 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b4800c780 con 0x7f4b48002150 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 4: 0 ) v1 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 10 client.25803.objecter _finish_command 4 = system:0 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 4 2024-09-10T19:08:55.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.571+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 2 4 2024-09-10T19:08:55.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 10 client.25803.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 5 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 5 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48001fe0 osd=2 3 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 2 5 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 4 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 10 client.25803.objecter _send_command 5 2024-09-10T19:08:55.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.587+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6809/386094220 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b48001b30 con 0x7f4b48002150 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b48001b30 con 0x7f4b48002150 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 5: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 10 client.25803.objecter _finish_command 5 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48001fe0 osd=2 4 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 2 5 2024-09-10T19:08:55.597 INFO:teuthology.orchestra.run.smithi083.stdout:osd.2: {} 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 10 client.25803.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 6 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 6 2024-09-10T19:08:55.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 3 6 2024-09-10T19:08:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 4 2024-09-10T19:08:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 10 client.25803.objecter _send_command 6 2024-09-10T19:08:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6813/1279688021 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b48003dd0 con 0x7f4b4800daa0 2024-09-10T19:08:55.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.591+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b4800daa0 2024-09-10T19:08:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.595+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b48003dd0 con 0x7f4b4800daa0 2024-09-10T19:08:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.595+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 6: 0 ) v1 2024-09-10T19:08:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.595+0000 7f4b597fa640 10 client.25803.objecter _finish_command 6 = system:0 2024-09-10T19:08:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.595+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 4 2024-09-10T19:08:55.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.595+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 3 6 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 10 client.25803.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 7 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 7 2024-09-10T19:08:55.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48001db0 osd=3 3 2024-09-10T19:08:55.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 3 7 2024-09-10T19:08:55.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 4 2024-09-10T19:08:55.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 10 client.25803.objecter _send_command 7 2024-09-10T19:08:55.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.615+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.83:6813/1279688021 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4800ee60 con 0x7f4b4800daa0 2024-09-10T19:08:55.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b4800ee60 con 0x7f4b4800daa0 2024-09-10T19:08:55.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 7: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 10 client.25803.objecter _finish_command 7 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48001db0 osd=3 4 2024-09-10T19:08:55.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 3 7 2024-09-10T19:08:55.624 INFO:teuthology.orchestra.run.smithi083.stdout:osd.3: {} 2024-09-10T19:08:55.624 INFO:teuthology.orchestra.run.smithi083.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:08:55.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 10 client.25803.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 8 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 8 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 4 8 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 4 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 10 client.25803.objecter _send_command 8 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6800/3225515265 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b48016240 con 0x7f4b48012740 2024-09-10T19:08:55.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.619+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b48012740 2024-09-10T19:08:55.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.623+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b48016240 con 0x7f4b48012740 2024-09-10T19:08:55.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.623+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 8: 0 ) v1 2024-09-10T19:08:55.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.623+0000 7f4b597fa640 10 client.25803.objecter _finish_command 8 = system:0 2024-09-10T19:08:55.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.623+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 4 2024-09-10T19:08:55.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.623+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 4 8 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 10 client.25803.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 9 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 9 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b480034e0 osd=4 3 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 4 9 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 4 2024-09-10T19:08:55.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 10 client.25803.objecter _send_command 9 2024-09-10T19:08:55.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.639+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6800/3225515265 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4800e8e0 con 0x7f4b48012740 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b4800e8e0 con 0x7f4b48012740 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 9: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 10 client.25803.objecter _finish_command 9 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b480034e0 osd=4 4 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 4 9 2024-09-10T19:08:55.648 INFO:teuthology.orchestra.run.smithi083.stdout:osd.4: {} 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 10 client.25803.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 10 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 10 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 5 10 2024-09-10T19:08:55.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 4 2024-09-10T19:08:55.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 10 client.25803.objecter _send_command 10 2024-09-10T19:08:55.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6804/571173777 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b4800e810 con 0x7f4b4800d5f0 2024-09-10T19:08:55.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.643+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b4800d5f0 2024-09-10T19:08:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.647+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b4800e810 con 0x7f4b4800d5f0 2024-09-10T19:08:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.647+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 10: 0 ) v1 2024-09-10T19:08:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.647+0000 7f4b597fa640 10 client.25803.objecter _finish_command 10 = system:0 2024-09-10T19:08:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.647+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 4 2024-09-10T19:08:55.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.647+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 5 10 2024-09-10T19:08:55.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 10 client.25803.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 11 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 11 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b4800e6a0 osd=5 3 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 5 11 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 4 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 10 client.25803.objecter _send_command 11 2024-09-10T19:08:55.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.663+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6804/571173777 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4801ac60 con 0x7f4b4800d5f0 2024-09-10T19:08:55.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b4801ac60 con 0x7f4b4800d5f0 2024-09-10T19:08:55.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 11: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b597fa640 10 client.25803.objecter _finish_command 11 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b4800e6a0 osd=5 4 2024-09-10T19:08:55.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 5 11 2024-09-10T19:08:55.674 INFO:teuthology.orchestra.run.smithi083.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:08:55.674 INFO:teuthology.orchestra.run.smithi083.stdout:osd.5: {} 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 10 client.25803.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 12 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 12 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 6 12 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 4 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 10 client.25803.objecter _send_command 12 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.667+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6808/1317922936 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b4801ede0 con 0x7f4b4801b2e0 2024-09-10T19:08:55.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b4801b2e0 2024-09-10T19:08:55.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b4801ede0 con 0x7f4b4801b2e0 2024-09-10T19:08:55.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 12: 0 ) v1 2024-09-10T19:08:55.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 10 client.25803.objecter _finish_command 12 = system:0 2024-09-10T19:08:55.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 4 2024-09-10T19:08:55.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.671+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 6 12 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 10 client.25803.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 13 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 13 2024-09-10T19:08:55.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b4801ac60 osd=6 3 2024-09-10T19:08:55.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 6 13 2024-09-10T19:08:55.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 4 2024-09-10T19:08:55.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 10 client.25803.objecter _send_command 13 2024-09-10T19:08:55.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.687+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6808/1317922936 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b48020650 con 0x7f4b4801b2e0 2024-09-10T19:08:55.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b48020650 con 0x7f4b4801b2e0 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 13: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 10 client.25803.objecter _finish_command 13 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b4801ac60 osd=6 4 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 6 13 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:08:55.697 INFO:teuthology.orchestra.run.smithi083.stdout:osd.6: {} 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 10 client.25803.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 14 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 14 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 7 14 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 4 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 10 client.25803.objecter _send_command 14 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6812/1610151650 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b48021200 con 0x7f4b48020cd0 2024-09-10T19:08:55.698 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.691+0000 7f4b597fa640 10 client.25803.objecter ms_handle_connect 0x7f4b48020cd0 2024-09-10T19:08:55.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.695+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f4b48021200 con 0x7f4b48020cd0 2024-09-10T19:08:55.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.695+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 14: 0 ) v1 2024-09-10T19:08:55.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.695+0000 7f4b597fa640 10 client.25803.objecter _finish_command 14 = system:0 2024-09-10T19:08:55.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.695+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 4 2024-09-10T19:08:55.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.695+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 7 14 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 10 client.25803.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign -1 15 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 20 client.25803.objecter _get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_remove -1 15 2024-09-10T19:08:55.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 20 client.25803.objecter get_session s=0x7f4b48020650 osd=7 3 2024-09-10T19:08:55.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 15 client.25803.objecter _session_command_op_assign 7 15 2024-09-10T19:08:55.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 4 2024-09-10T19:08:55.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 10 client.25803.objecter _send_command 15 2024-09-10T19:08:55.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.707+0000 7f4b62e07640 1 -- 172.21.15.83:0/479908533 --> v1:172.21.15.165:6812/1610151650 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4800eb50 con 0x7f4b48020cd0 2024-09-10T19:08:55.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.711+0000 7f4b597fa640 1 -- 172.21.15.83:0/479908533 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 ==== 35+0+3 (unknown 3797281404 0 2423995997) 0x7f4b4800eb50 con 0x7f4b48020cd0 2024-09-10T19:08:55.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.711+0000 7f4b597fa640 10 client.25803.objecter ms_dispatch 0x7f4b5c000b90 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) v1 2024-09-10T19:08:55.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.711+0000 7f4b597fa640 10 client.25803.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-09-10T19:08:55.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.711+0000 7f4b597fa640 20 client.25803.objecter put_session s=0x7f4b48020650 osd=7 4 2024-09-10T19:08:55.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:55.711+0000 7f4b597fa640 15 client.25803.objecter _session_command_op_remove 7 15 2024-09-10T19:08:55.718 INFO:teuthology.orchestra.run.smithi083.stdout:osd.7: {} 2024-09-10T19:08:55.718 INFO:teuthology.orchestra.run.smithi083.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:08:55.734 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:55 smithi083 bash[18629]: cluster 2024-09-10T19:08:54.408469+0000 mgr.y (mgr.24526) 766 : cluster [DBG] pgmap v1320: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:55.734 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:55 smithi083 bash[32381]: cluster 2024-09-10T19:08:54.408469+0000 mgr.y (mgr.24526) 766 : cluster [DBG] pgmap v1320: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:55 smithi165 bash[22591]: cluster 2024-09-10T19:08:54.408469+0000 mgr.y (mgr.24526) 766 : cluster [DBG] pgmap v1320: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76/591 objects degraded (12.860%) 2024-09-10T19:08:56.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 -- 172.21.15.83:0/3081607577 >> v1:172.21.15.83:6790/0 conn(0x7f6430101520 legacy=0x7f64301018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:56.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 -- 172.21.15.83:0/3081607577 shutdown_connections 2024-09-10T19:08:56.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 -- 172.21.15.83:0/3081607577 >> 172.21.15.83:0/3081607577 conn(0x7f64300f70a0 msgr2=0x7f64300f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:56.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 -- 172.21.15.83:0/3081607577 shutdown_connections 2024-09-10T19:08:56.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 -- 172.21.15.83:0/3081607577 wait complete. 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 librados: starting msgr at 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 librados: starting objecter 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.063+0000 7f64370b9640 1 Processor -- start 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- start start 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 librados: setting wanted keys 2024-09-10T19:08:56.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 librados: calling monclient init 2024-09-10T19:08:56.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f643011bf20 con 0x7f6430107df0 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f643011d120 con 0x7f6430104290 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f643011e320 con 0x7f6430101520 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f6434e55640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f6430101520 0x7f6430065db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55328/0 (socket says 172.21.15.83:55328) 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f6434e55640 1 -- 172.21.15.83:0/2309230782 learned_addr learned my addr 172.21.15.83:0/2309230782 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter ms_handle_connect 0x7f6430101520 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter ms_handle_connect 0x7f6430104290 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter ms_handle_connect 0x7f6430107df0 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:56.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569517924 0 0) 0x7f643011e320 con 0x7f6430101520 2024-09-10T19:08:56.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6418003720 con 0x7f6430101520 2024-09-10T19:08:56.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549770992 0 0) 0x7f643011d120 con 0x7f6430104290 2024-09-10T19:08:56.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643011e320 con 0x7f6430104290 2024-09-10T19:08:56.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158988471 0 0) 0x7f643011bf20 con 0x7f6430107df0 2024-09-10T19:08:56.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643011d120 con 0x7f6430107df0 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849688492 0 0) 0x7f6418003720 con 0x7f6430101520 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f643011bf20 con 0x7f6430101520 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6428001950 con 0x7f6430101520 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322715477 0 0) 0x7f643011e320 con 0x7f6430104290 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6418003720 con 0x7f6430104290 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f64240018c0 con 0x7f6430104290 2024-09-10T19:08:56.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1173107425 0 0) 0x7f643011bf20 con 0x7f6430101520 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 >> v1:172.21.15.165:6789/0 conn(0x7f6430104290 legacy=0x7f64300664c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 >> v1:172.21.15.83:6789/0 conn(0x7f6430107df0 legacy=0x7f643011a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643011f520 con 0x7f6430101520 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643011d310 con 0x7f6430101520 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 10 client.49895.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f643011d7e0 con 0x7f6430101520 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f64370b9640 1 librados: init done 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6428001570 con 0x7f6430101520 2024-09-10T19:08:56.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.067+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6428003980 con 0x7f6430101520 2024-09-10T19:08:56.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f6428001c00 con 0x7f6430101520 2024-09-10T19:08:56.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c005010 con 0x7f6430101520 2024-09-10T19:08:56.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(847..847 src has 750..847) v4 ==== 7363+0+0 (unknown 4127979390 0 0) 0x7f642808a9a0 con 0x7f6430101520 2024-09-10T19:08:56.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 10 client.49895.objecter ms_dispatch 0x7f6430000b90 osd_map(847..847 src has 750..847) v4 2024-09-10T19:08:56.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 3 client.49895.objecter handle_osd_map got epochs [847,847] > 0 2024-09-10T19:08:56.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 3 client.49895.objecter handle_osd_map decoding full epoch 847 2024-09-10T19:08:56.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 20 client.49895.objecter dump_active .. 0 homeless 2024-09-10T19:08:56.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.071+0000 7f642d7fa640 10 client.49895.objecter ms_handle_connect 0x7f6418071440 2024-09-10T19:08:56.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.079+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f64280571e0 con 0x7f6430101520 2024-09-10T19:08:56.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.263+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f641c0052c0 con 0x7f6430101520 2024-09-10T19:08:56.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.467+0000 7f642d7fa640 1 -- 172.21.15.83:0/2309230782 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v848) v1 ==== 91+0+0 (unknown 3113106858 0 0) 0x7f6428056b80 con 0x7f6430101520 2024-09-10T19:08:56.472 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 10 librados: watch_flush enter 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 10 librados: watch_flush exit 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 20 client.49895.objecter shutdown clearing up homeless session... 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 10 client.49895.objecter successfully canceled tick 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 >> v1:172.21.15.83:6800/3071793735 conn(0x7f6418071440 legacy=0x7f64180738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 >> v1:172.21.15.83:6790/0 conn(0x7f6430101520 legacy=0x7f6430065db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 shutdown_connections 2024-09-10T19:08:56.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 >> 172.21.15.83:0/2309230782 conn(0x7f64300f70a0 msgr2=0x7f6430064cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:56.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 shutdown_connections 2024-09-10T19:08:56.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 -- 172.21.15.83:0/2309230782 wait complete. 2024-09-10T19:08:56.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:56.471+0000 7f64370b9640 1 librados: shutdown 2024-09-10T19:08:56.715 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:56 smithi165 bash[22591]: audit 2024-09-10T19:08:55.553111+0000 mon.a (mon.0) 3769 : audit [DBG] from='client.? v1:172.21.15.83:0/479908533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:56.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:56 smithi165 bash[22591]: audit 2024-09-10T19:08:56.267799+0000 mon.c (mon.2) 885 : audit [INF] from='client.? v1:172.21.15.83:0/2309230782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.716 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:56 smithi165 bash[22591]: audit 2024-09-10T19:08:56.268446+0000 mon.a (mon.0) 3770 : audit [INF] from='client.49895 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[18629]: audit 2024-09-10T19:08:55.553111+0000 mon.a (mon.0) 3769 : audit [DBG] from='client.? v1:172.21.15.83:0/479908533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[18629]: audit 2024-09-10T19:08:56.267799+0000 mon.c (mon.2) 885 : audit [INF] from='client.? v1:172.21.15.83:0/2309230782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[18629]: audit 2024-09-10T19:08:56.268446+0000 mon.a (mon.0) 3770 : audit [INF] from='client.49895 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[32381]: audit 2024-09-10T19:08:55.553111+0000 mon.a (mon.0) 3769 : audit [DBG] from='client.? v1:172.21.15.83:0/479908533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[32381]: audit 2024-09-10T19:08:56.267799+0000 mon.c (mon.2) 885 : audit [INF] from='client.? v1:172.21.15.83:0/2309230782' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:56 smithi083 bash[32381]: audit 2024-09-10T19:08:56.268446+0000 mon.a (mon.0) 3770 : audit [INF] from='client.49895 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:08:56.841 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:08:56.841 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:08:56.841 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-10T19:08:56.842 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:08:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:08:57.306 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:08:57.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.411196+0000 mgr.y (mgr.24526) 767 : cluster [DBG] pgmap v1321: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.450738+0000 mon.a (mon.0) 3771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: audit 2024-09-10T19:08:56.470790+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49895 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.471067+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e848: 8 total, 7 up, 8 in 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.499604+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.507595+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.864171+0000 osd.7 (osd.7) 434 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:08:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[32381]: cluster 2024-09-10T19:08:56.866993+0000 osd.7 (osd.7) 435 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.411196+0000 mgr.y (mgr.24526) 767 : cluster [DBG] pgmap v1321: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.450738+0000 mon.a (mon.0) 3771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: audit 2024-09-10T19:08:56.470790+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49895 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.471067+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e848: 8 total, 7 up, 8 in 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.499604+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.507595+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.864171+0000 osd.7 (osd.7) 434 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:08:57.834 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:57 smithi083 bash[18629]: cluster 2024-09-10T19:08:56.866993+0000 osd.7 (osd.7) 435 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:08:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.411196+0000 mgr.y (mgr.24526) 767 : cluster [DBG] pgmap v1321: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.450738+0000 mon.a (mon.0) 3771 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:08:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: audit 2024-09-10T19:08:56.470790+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49895 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:08:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.471067+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e848: 8 total, 7 up, 8 in 2024-09-10T19:08:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.499604+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:08:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.507595+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:08:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.864171+0000 osd.7 (osd.7) 434 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:08:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:57 smithi165 bash[22591]: cluster 2024-09-10T19:08:56.866993+0000 osd.7 (osd.7) 435 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:08:58.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 -- 172.21.15.83:0/3432031559 >> v1:172.21.15.83:6789/0 conn(0x7ff9f4107e30 legacy=0x7ff9f410a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:58.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 -- 172.21.15.83:0/3432031559 shutdown_connections 2024-09-10T19:08:58.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 -- 172.21.15.83:0/3432031559 >> 172.21.15.83:0/3432031559 conn(0x7ff9f4075290 msgr2=0x7ff9f40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:58.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 -- 172.21.15.83:0/3432031559 shutdown_connections 2024-09-10T19:08:58.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 -- 172.21.15.83:0/3432031559 wait complete. 2024-09-10T19:08:58.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 librados: starting msgr at 2024-09-10T19:08:58.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 librados: starting objecter 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.099+0000 7ff9fb526640 1 Processor -- start 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- start start 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 librados: setting wanted keys 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 librados: calling monclient init 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f411c2c0 con 0x7ff9f4107e30 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f411d460 con 0x7ff9f41042a0 2024-09-10T19:08:58.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9f411e600 con 0x7ff9f41014e0 2024-09-10T19:08:58.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9f9ac3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff9f4107e30 0x7ff9f411aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49358/0 (socket says 172.21.15.83:49358) 2024-09-10T19:08:58.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9f92c2640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff9f41014e0 0x7ff9f40653b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55358/0 (socket says 172.21.15.83:55358) 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9f9ac3640 1 -- 172.21.15.83:0/2980890398 learned_addr learned my addr 172.21.15.83:0/2980890398 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter ms_handle_connect 0x7ff9f4107e30 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter ms_handle_connect 0x7ff9f41042a0 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter ms_handle_connect 0x7ff9f41014e0 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936788604 0 0) 0x7ff9f411e600 con 0x7ff9f41014e0 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e0003120 con 0x7ff9f41014e0 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910615503 0 0) 0x7ff9f411d460 con 0x7ff9f41042a0 2024-09-10T19:08:58.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f411e600 con 0x7ff9f41042a0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252033890 0 0) 0x7ff9f411c2c0 con 0x7ff9f4107e30 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9f411d460 con 0x7ff9f4107e30 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 478624653 0 0) 0x7ff9e0003120 con 0x7ff9f41014e0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f411c2c0 con 0x7ff9f41014e0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff9e4001840 con 0x7ff9f41014e0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 452294831 0 0) 0x7ff9f411d460 con 0x7ff9f4107e30 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9e0003120 con 0x7ff9f4107e30 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff9ec0017b0 con 0x7ff9f4107e30 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665830676 0 0) 0x7ff9f411e600 con 0x7ff9f41042a0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9f411d460 con 0x7ff9f41042a0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff9f0001990 con 0x7ff9f41042a0 2024-09-10T19:08:58.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2802558719 0 0) 0x7ff9f411c2c0 con 0x7ff9f41014e0 2024-09-10T19:08:58.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 >> v1:172.21.15.165:6789/0 conn(0x7ff9f41042a0 legacy=0x7ff9f4117470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:58.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 >> v1:172.21.15.83:6789/0 conn(0x7ff9f4107e30 legacy=0x7ff9f411aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:58.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f411f7e0 con 0x7ff9f41014e0 2024-09-10T19:08:58.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f411e7f0 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 10 client.49901.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f411ecc0 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9fb526640 1 librados: init done 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff9e4002360 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.103+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff9e4005a80 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8005010 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff9e401bea0 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(848..848 src has 750..848) v4 ==== 7363+0+0 (unknown 2337220105 0 0) 0x7ff9e408a180 con 0x7ff9f41014e0 2024-09-10T19:08:58.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 10 client.49901.objecter ms_dispatch 0x7ff9f4000b90 osd_map(848..848 src has 750..848) v4 2024-09-10T19:08:58.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 3 client.49901.objecter handle_osd_map got epochs [848,848] > 0 2024-09-10T19:08:58.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 3 client.49901.objecter handle_osd_map decoding full epoch 848 2024-09-10T19:08:58.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 20 client.49901.objecter dump_active .. 0 homeless 2024-09-10T19:08:58.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.107+0000 7ff9e9ffb640 10 client.49901.objecter ms_handle_connect 0x7ff9e00717c0 2024-09-10T19:08:58.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.111+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff9e40565a0 con 0x7ff9f41014e0 2024-09-10T19:08:58.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.287+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) v1 -- 0x7ff9d8005840 con 0x7ff9f41014e0 2024-09-10T19:08:58.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.479+0000 7ff9e9ffb640 1 -- 172.21.15.83:0/2980890398 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v849) v1 ==== 131+0+0 (unknown 3901103718 0 0) 0x7ff9e40563c0 con 0x7ff9f41014e0 2024-09-10T19:08:58.483 INFO:teuthology.orchestra.run.smithi083.stderr:set pool 1 pgp_num to 11 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 10 librados: watch_flush enter 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 10 librados: watch_flush exit 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 20 client.49901.objecter shutdown clearing up homeless session... 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 10 client.49901.objecter successfully canceled tick 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff9e00717c0 legacy=0x7ff9e0073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 >> v1:172.21.15.83:6790/0 conn(0x7ff9f41014e0 legacy=0x7ff9f40653b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 shutdown_connections 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 >> 172.21.15.83:0/2980890398 conn(0x7ff9f4075290 msgr2=0x7ff9f40f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:08:58.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 shutdown_connections 2024-09-10T19:08:58.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 -- 172.21.15.83:0/2980890398 wait complete. 2024-09-10T19:08:58.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:08:58.483+0000 7ff9fb526640 1 librados: shutdown 2024-09-10T19:08:58.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:58 smithi083 bash[18629]: audit 2024-09-10T19:08:58.292185+0000 mon.c (mon.2) 886 : audit [INF] from='client.? v1:172.21.15.83:0/2980890398' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:58.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:58 smithi083 bash[18629]: audit 2024-09-10T19:08:58.292893+0000 mon.a (mon.0) 3774 : audit [INF] from='client.49901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:58.509 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:58 smithi083 bash[32381]: audit 2024-09-10T19:08:58.292185+0000 mon.c (mon.2) 886 : audit [INF] from='client.? v1:172.21.15.83:0/2980890398' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:58.509 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:58 smithi083 bash[32381]: audit 2024-09-10T19:08:58.292893+0000 mon.a (mon.0) 3774 : audit [INF] from='client.49901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:58.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:08:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:08:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:58 smithi165 bash[22591]: audit 2024-09-10T19:08:58.292185+0000 mon.c (mon.2) 886 : audit [INF] from='client.? v1:172.21.15.83:0/2980890398' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:58 smithi165 bash[22591]: audit 2024-09-10T19:08:58.292893+0000 mon.a (mon.0) 3774 : audit [INF] from='client.49901 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.412672+0000 mgr.y (mgr.24526) 768 : cluster [DBG] pgmap v1323: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: audit 2024-09-10T19:08:58.477151+0000 mon.a (mon.0) 3775 : audit [INF] from='client.49901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.477243+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e849: 8 total, 7 up, 8 in 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: audit 2024-09-10T19:08:58.480931+0000 mgr.y (mgr.24526) 769 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.542464+0000 osd.3 (osd.3) 452 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:08:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.545437+0000 osd.3 (osd.3) 453 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.795904+0000 osd.5 (osd.5) 430 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[18629]: cluster 2024-09-10T19:08:58.799442+0000 osd.5 (osd.5) 431 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.412672+0000 mgr.y (mgr.24526) 768 : cluster [DBG] pgmap v1323: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: audit 2024-09-10T19:08:58.477151+0000 mon.a (mon.0) 3775 : audit [INF] from='client.49901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.477243+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e849: 8 total, 7 up, 8 in 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: audit 2024-09-10T19:08:58.480931+0000 mgr.y (mgr.24526) 769 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.542464+0000 osd.3 (osd.3) 452 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.545437+0000 osd.3 (osd.3) 453 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.795904+0000 osd.5 (osd.5) 430 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:08:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:08:59 smithi083 bash[32381]: cluster 2024-09-10T19:08:58.799442+0000 osd.5 (osd.5) 431 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:08:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.412672+0000 mgr.y (mgr.24526) 768 : cluster [DBG] pgmap v1323: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:08:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: audit 2024-09-10T19:08:58.477151+0000 mon.a (mon.0) 3775 : audit [INF] from='client.49901 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.477243+0000 mon.a (mon.0) 3776 : cluster [DBG] osdmap e849: 8 total, 7 up, 8 in 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: audit 2024-09-10T19:08:58.480931+0000 mgr.y (mgr.24526) 769 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.542464+0000 osd.3 (osd.3) 452 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.545437+0000 osd.3 (osd.3) 453 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.795904+0000 osd.5 (osd.5) 430 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:08:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:08:59 smithi165 bash[22591]: cluster 2024-09-10T19:08:58.799442+0000 osd.5 (osd.5) 431 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:09:00.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:09:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.355038+0000 osd.4 (osd.4) 420 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:09:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.358688+0000 osd.4 (osd.4) 421 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:09:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.516104+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.c scrub starts 2024-09-10T19:09:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.520685+0000 osd.3 (osd.3) 455 : cluster [DBG] 6.c scrub ok 2024-09-10T19:09:00.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: audit 2024-09-10T19:08:59.577505+0000 mon.c (mon.2) 887 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.584511+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.588681+0000 osd.6 (osd.6) 394 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.863029+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[18629]: cluster 2024-09-10T19:08:59.866673+0000 osd.7 (osd.7) 437 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.355038+0000 osd.4 (osd.4) 420 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.358688+0000 osd.4 (osd.4) 421 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.516104+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.c scrub starts 2024-09-10T19:09:00.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.520685+0000 osd.3 (osd.3) 455 : cluster [DBG] 6.c scrub ok 2024-09-10T19:09:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: audit 2024-09-10T19:08:59.577505+0000 mon.c (mon.2) 887 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.584511+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:09:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.588681+0000 osd.6 (osd.6) 394 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:09:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.863029+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:09:00.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:00 smithi083 bash[32381]: cluster 2024-09-10T19:08:59.866673+0000 osd.7 (osd.7) 437 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:09:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.355038+0000 osd.4 (osd.4) 420 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:09:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.358688+0000 osd.4 (osd.4) 421 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:09:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.516104+0000 osd.3 (osd.3) 454 : cluster [DBG] 6.c scrub starts 2024-09-10T19:09:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.520685+0000 osd.3 (osd.3) 455 : cluster [DBG] 6.c scrub ok 2024-09-10T19:09:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: audit 2024-09-10T19:08:59.577505+0000 mon.c (mon.2) 887 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.584511+0000 osd.6 (osd.6) 393 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:09:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.588681+0000 osd.6 (osd.6) 394 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:09:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.863029+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:09:00.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:00 smithi165 bash[22591]: cluster 2024-09-10T19:08:59.866673+0000 osd.7 (osd.7) 437 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.414129+0000 mgr.y (mgr.24526) 770 : cluster [DBG] pgmap v1325: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.519056+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.523197+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.591673+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.596019+0000 osd.6 (osd.6) 396 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.814250+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.d scrub starts 2024-09-10T19:09:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[18629]: cluster 2024-09-10T19:09:00.818966+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.d scrub ok 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.414129+0000 mgr.y (mgr.24526) 770 : cluster [DBG] pgmap v1325: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.519056+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.523197+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.591673+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.596019+0000 osd.6 (osd.6) 396 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.814250+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.d scrub starts 2024-09-10T19:09:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:01 smithi083 bash[32381]: cluster 2024-09-10T19:09:00.818966+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.d scrub ok 2024-09-10T19:09:01.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.414129+0000 mgr.y (mgr.24526) 770 : cluster [DBG] pgmap v1325: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.519056+0000 osd.0 (osd.0) 409 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.523197+0000 osd.0 (osd.0) 410 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.591673+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.596019+0000 osd.6 (osd.6) 396 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.814250+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.d scrub starts 2024-09-10T19:09:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:01 smithi165 bash[22591]: cluster 2024-09-10T19:09:00.818966+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.d scrub ok 2024-09-10T19:09:02.321 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:02.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:02] "GET /metrics HTTP/1.1" 200 224232 "" "Prometheus/2.43.0" 2024-09-10T19:09:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.314200+0000 osd.6 (osd.6) 397 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.314557+0000 osd.6 (osd.6) 398 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:02.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.411265+0000 osd.4 (osd.4) 422 : cluster [DBG] 2.f scrub starts 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.415617+0000 osd.4 (osd.4) 423 : cluster [DBG] 2.f scrub ok 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.504323+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e850: 8 total, 7 up, 8 in 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.314200+0000 osd.6 (osd.6) 397 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.314557+0000 osd.6 (osd.6) 398 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.411265+0000 osd.4 (osd.4) 422 : cluster [DBG] 2.f scrub starts 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.415617+0000 osd.4 (osd.4) 423 : cluster [DBG] 2.f scrub ok 2024-09-10T19:09:02.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:02 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.504323+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e850: 8 total, 7 up, 8 in 2024-09-10T19:09:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:02 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.314200+0000 osd.6 (osd.6) 397 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:02.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:02 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.314557+0000 osd.6 (osd.6) 398 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:02 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.411265+0000 osd.4 (osd.4) 422 : cluster [DBG] 2.f scrub starts 2024-09-10T19:09:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:02 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.415617+0000 osd.4 (osd.4) 423 : cluster [DBG] 2.f scrub ok 2024-09-10T19:09:02.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:02 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.504323+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e850: 8 total, 7 up, 8 in 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.567772+0000 osd.0 (osd.0) 411 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.571182+0000 osd.0 (osd.0) 412 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.916772+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[18629]: cluster 2024-09-10T19:09:01.920989+0000 osd.2 (osd.2) 336 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[18629]: cluster 2024-09-10T19:09:02.415647+0000 mgr.y (mgr.24526) 771 : cluster [DBG] pgmap v1327: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.567772+0000 osd.0 (osd.0) 411 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:09:03.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.571182+0000 osd.0 (osd.0) 412 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:09:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.916772+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:09:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[32381]: cluster 2024-09-10T19:09:01.920989+0000 osd.2 (osd.2) 336 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:09:03.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:03 smithi083 bash[32381]: cluster 2024-09-10T19:09:02.415647+0000 mgr.y (mgr.24526) 771 : cluster [DBG] pgmap v1327: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:03 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.567772+0000 osd.0 (osd.0) 411 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:09:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:03 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.571182+0000 osd.0 (osd.0) 412 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:09:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:03 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.916772+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:09:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:03 smithi165 bash[22591]: cluster 2024-09-10T19:09:01.920989+0000 osd.2 (osd.2) 336 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:09:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:03 smithi165 bash[22591]: cluster 2024-09-10T19:09:02.415647+0000 mgr.y (mgr.24526) 771 : cluster [DBG] pgmap v1327: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:04.362 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-10T19:09:04.362 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-10T19:09:04.362 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-10T19:09:04.362 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T19:09:04.365 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl start ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:09:04.386 INFO:journalctl@ceph.osd.1.smithi083.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2024-09-10T19:09:04.393 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:09:04.791 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:04 smithi083 systemd[1]: Started Ceph osd.1 for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:09:04.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.476679+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:09:04.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.479462+0000 osd.4 (osd.4) 425 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.506700+0000 osd.0 (osd.0) 413 : cluster [DBG] 12.e scrub starts 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.511925+0000 osd.0 (osd.0) 414 : cluster [DBG] 12.e scrub ok 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.911381+0000 osd.5 (osd.5) 432 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:03.914917+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[18629]: cluster 2024-09-10T19:09:04.416159+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e851: 8 total, 7 up, 8 in 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.476679+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.479462+0000 osd.4 (osd.4) 425 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:09:04.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.506700+0000 osd.0 (osd.0) 413 : cluster [DBG] 12.e scrub starts 2024-09-10T19:09:04.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.511925+0000 osd.0 (osd.0) 414 : cluster [DBG] 12.e scrub ok 2024-09-10T19:09:04.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.911381+0000 osd.5 (osd.5) 432 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:09:04.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:03.914917+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:09:04.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:04 smithi083 bash[32381]: cluster 2024-09-10T19:09:04.416159+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e851: 8 total, 7 up, 8 in 2024-09-10T19:09:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.476679+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:09:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.479462+0000 osd.4 (osd.4) 425 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:09:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.506700+0000 osd.0 (osd.0) 413 : cluster [DBG] 12.e scrub starts 2024-09-10T19:09:04.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.511925+0000 osd.0 (osd.0) 414 : cluster [DBG] 12.e scrub ok 2024-09-10T19:09:04.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.911381+0000 osd.5 (osd.5) 432 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:09:04.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:03.914917+0000 osd.5 (osd.5) 433 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:09:04.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:04 smithi165 bash[22591]: cluster 2024-09-10T19:09:04.416159+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e851: 8 total, 7 up, 8 in 2024-09-10T19:09:05.003 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:05.648 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:09:05.648 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-10T19:09:05.648 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-10T19:09:05.648 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-10T19:09:05.649 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[18629]: cluster 2024-09-10T19:09:04.416617+0000 mgr.y (mgr.24526) 772 : cluster [DBG] pgmap v1329: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[18629]: cluster 2024-09-10T19:09:04.485002+0000 osd.3 (osd.3) 456 : cluster [DBG] 12.b scrub starts 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[18629]: cluster 2024-09-10T19:09:04.494555+0000 osd.3 (osd.3) 457 : cluster [DBG] 12.b scrub ok 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[32381]: cluster 2024-09-10T19:09:04.416617+0000 mgr.y (mgr.24526) 772 : cluster [DBG] pgmap v1329: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[32381]: cluster 2024-09-10T19:09:04.485002+0000 osd.3 (osd.3) 456 : cluster [DBG] 12.b scrub starts 2024-09-10T19:09:05.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:05 smithi083 bash[32381]: cluster 2024-09-10T19:09:04.494555+0000 osd.3 (osd.3) 457 : cluster [DBG] 12.b scrub ok 2024-09-10T19:09:05.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:05 smithi165 bash[22591]: cluster 2024-09-10T19:09:04.416617+0000 mgr.y (mgr.24526) 772 : cluster [DBG] pgmap v1329: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:05.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:05 smithi165 bash[22591]: cluster 2024-09-10T19:09:04.485002+0000 osd.3 (osd.3) 456 : cluster [DBG] 12.b scrub starts 2024-09-10T19:09:05.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:05 smithi165 bash[22591]: cluster 2024-09-10T19:09:04.494555+0000 osd.3 (osd.3) 457 : cluster [DBG] 12.b scrub ok 2024-09-10T19:09:06.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:09:06.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: cluster 2024-09-10T19:09:05.514838+0000 osd.3 (osd.3) 458 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:09:06.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: cluster 2024-09-10T19:09:05.517846+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:09:06.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: audit 2024-09-10T19:09:05.949462+0000 mon.c (mon.2) 888 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:06.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: audit 2024-09-10T19:09:06.396515+0000 mon.c (mon.2) 889 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:06.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: audit 2024-09-10T19:09:06.397764+0000 mon.c (mon.2) 890 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:06.614 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:06 smithi165 bash[22591]: audit 2024-09-10T19:09:06.404799+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: cluster 2024-09-10T19:09:05.514838+0000 osd.3 (osd.3) 458 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: cluster 2024-09-10T19:09:05.517846+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: audit 2024-09-10T19:09:05.949462+0000 mon.c (mon.2) 888 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: audit 2024-09-10T19:09:06.396515+0000 mon.c (mon.2) 889 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: audit 2024-09-10T19:09:06.397764+0000 mon.c (mon.2) 890 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[18629]: audit 2024-09-10T19:09:06.404799+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: cluster 2024-09-10T19:09:05.514838+0000 osd.3 (osd.3) 458 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:09:06.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: cluster 2024-09-10T19:09:05.517846+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:09:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: audit 2024-09-10T19:09:05.949462+0000 mon.c (mon.2) 888 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: audit 2024-09-10T19:09:06.396515+0000 mon.c (mon.2) 889 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: audit 2024-09-10T19:09:06.397764+0000 mon.c (mon.2) 890 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:06.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:06 smithi083 bash[32381]: audit 2024-09-10T19:09:06.404799+0000 mon.a (mon.0) 3779 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-10T19:09:07.522 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:09:07.523 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[211960]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-10T19:09:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[18629]: cluster 2024-09-10T19:09:05.915216+0000 osd.2 (osd.2) 337 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:09:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[18629]: cluster 2024-09-10T19:09:05.918520+0000 osd.2 (osd.2) 338 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:09:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[18629]: cluster 2024-09-10T19:09:06.419390+0000 mgr.y (mgr.24526) 773 : cluster [DBG] pgmap v1330: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[18629]: cluster 2024-09-10T19:09:06.531294+0000 osd.3 (osd.3) 460 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:09:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[18629]: cluster 2024-09-10T19:09:06.534606+0000 osd.3 (osd.3) 461 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:09:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[32381]: cluster 2024-09-10T19:09:05.915216+0000 osd.2 (osd.2) 337 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:09:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[32381]: cluster 2024-09-10T19:09:05.918520+0000 osd.2 (osd.2) 338 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:09:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[32381]: cluster 2024-09-10T19:09:06.419390+0000 mgr.y (mgr.24526) 773 : cluster [DBG] pgmap v1330: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[32381]: cluster 2024-09-10T19:09:06.531294+0000 osd.3 (osd.3) 460 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:09:07.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:07 smithi083 bash[32381]: cluster 2024-09-10T19:09:06.534606+0000 osd.3 (osd.3) 461 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:09:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:07 smithi165 bash[22591]: cluster 2024-09-10T19:09:05.915216+0000 osd.2 (osd.2) 337 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:09:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:07 smithi165 bash[22591]: cluster 2024-09-10T19:09:05.918520+0000 osd.2 (osd.2) 338 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:09:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:07 smithi165 bash[22591]: cluster 2024-09-10T19:09:06.419390+0000 mgr.y (mgr.24526) 773 : cluster [DBG] pgmap v1330: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:07.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:07 smithi165 bash[22591]: cluster 2024-09-10T19:09:06.531294+0000 osd.3 (osd.3) 460 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:09:07.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:07 smithi165 bash[22591]: cluster 2024-09-10T19:09:06.534606+0000 osd.3 (osd.3) 461 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:09:08.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:08 smithi083 bash[18629]: cluster 2024-09-10T19:09:08.025746+0000 osd.5 (osd.5) 434 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:09:08.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:08 smithi083 bash[18629]: cluster 2024-09-10T19:09:08.030638+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:09:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:08 smithi083 bash[32381]: cluster 2024-09-10T19:09:08.025746+0000 osd.5 (osd.5) 434 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:09:08.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:08 smithi083 bash[32381]: cluster 2024-09-10T19:09:08.030638+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:09:08.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:08 smithi165 bash[22591]: cluster 2024-09-10T19:09:08.025746+0000 osd.5 (osd.5) 434 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:09:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:08 smithi165 bash[22591]: cluster 2024-09-10T19:09:08.030638+0000 osd.5 (osd.5) 435 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:09:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:09 smithi083 bash[18629]: cluster 2024-09-10T19:09:08.420950+0000 mgr.y (mgr.24526) 774 : cluster [DBG] pgmap v1331: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:09 smithi083 bash[18629]: audit 2024-09-10T19:09:08.492229+0000 mgr.y (mgr.24526) 775 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:09 smithi083 bash[32381]: cluster 2024-09-10T19:09:08.420950+0000 mgr.y (mgr.24526) 774 : cluster [DBG] pgmap v1331: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:09.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:09 smithi083 bash[32381]: audit 2024-09-10T19:09:08.492229+0000 mgr.y (mgr.24526) 775 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:09 smithi165 bash[22591]: cluster 2024-09-10T19:09:08.420950+0000 mgr.y (mgr.24526) 774 : cluster [DBG] pgmap v1331: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:09 smithi165 bash[22591]: audit 2024-09-10T19:09:08.492229+0000 mgr.y (mgr.24526) 775 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:10.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- 172.21.15.83:0/922641298 >> v1:172.21.15.83:6789/0 conn(0x7f82e4108ef0 legacy=0x7f82e410b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:10.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- 172.21.15.83:0/922641298 shutdown_connections 2024-09-10T19:09:10.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- 172.21.15.83:0/922641298 >> 172.21.15.83:0/922641298 conn(0x7f82e40f8180 msgr2=0x7f82e40fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:10.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- 172.21.15.83:0/922641298 shutdown_connections 2024-09-10T19:09:10.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- 172.21.15.83:0/922641298 wait complete. 2024-09-10T19:09:10.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 librados: starting msgr at 2024-09-10T19:09:10.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 librados: starting objecter 2024-09-10T19:09:10.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 Processor -- start 2024-09-10T19:09:10.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 -- start start 2024-09-10T19:09:10.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 librados: setting wanted keys 2024-09-10T19:09:10.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.754+0000 7f82ec468640 1 librados: calling monclient init 2024-09-10T19:09:10.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e411be40 con 0x7f82e4105360 2024-09-10T19:09:10.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e411d000 con 0x7f82e41025a0 2024-09-10T19:09:10.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82e9a03640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f82e4105360 0x7f82e40fe950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54124/0 (socket says 172.21.15.83:54124) 2024-09-10T19:09:10.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82eaa05640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f82e4108ef0 0x7f82e411a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:60604/0 (socket says 172.21.15.83:60604) 2024-09-10T19:09:10.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e411e200 con 0x7f82e4108ef0 2024-09-10T19:09:10.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82e9a03640 1 -- 172.21.15.83:0/4153394699 learned_addr learned my addr 172.21.15.83:0/4153394699 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:10.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter ms_handle_connect 0x7f82e4105360 2024-09-10T19:09:10.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:10.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter ms_handle_connect 0x7f82e4108ef0 2024-09-10T19:09:10.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter ms_handle_connect 0x7f82e41025a0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500269653 0 0) 0x7f82e411be40 con 0x7f82e4105360 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0003720 con 0x7f82e4105360 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750254615 0 0) 0x7f82e411e200 con 0x7f82e4108ef0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e411be40 con 0x7f82e4108ef0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385518552 0 0) 0x7f82e411d000 con 0x7f82e41025a0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e411e200 con 0x7f82e41025a0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2148976558 0 0) 0x7f82e411be40 con 0x7f82e4108ef0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e411d000 con 0x7f82e4108ef0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f82e0002050 con 0x7f82e4108ef0 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035386799 0 0) 0x7f82d0003720 con 0x7f82e4105360 2024-09-10T19:09:10.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e411be40 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f82d40019f0 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034613575 0 0) 0x7f82e411e200 con 0x7f82e41025a0 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d0003720 con 0x7f82e41025a0 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f82dc001bc0 con 0x7f82e41025a0 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461295667 0 0) 0x7f82e411be40 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 >> v1:172.21.15.83:6790/0 conn(0x7f82e4108ef0 legacy=0x7f82e411a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 >> v1:172.21.15.165:6789/0 conn(0x7f82e41025a0 legacy=0x7f82e40fe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e411f400 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e411e3f0 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 10 client.25821.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e411e840 con 0x7f82e4105360 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 librados: init done 2024-09-10T19:09:10.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f82d40023e0 con 0x7f82e4105360 2024-09-10T19:09:10.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f82d40059f0 con 0x7f82e4105360 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.758+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c8005040 con 0x7f82e4105360 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f82d4001f80 con 0x7f82e4105360 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(851..851 src has 750..851) v4 ==== 7363+0+0 (unknown 1120706650 0 0) 0x7f82d408a250 con 0x7f82e4105360 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 10 client.25821.objecter ms_dispatch 0x7f82e4000b90 osd_map(851..851 src has 750..851) v4 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 3 client.25821.objecter handle_osd_map got epochs [851,851] > 0 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 3 client.25821.objecter handle_osd_map decoding full epoch 851 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 20 client.25821.objecter dump_active .. 0 homeless 2024-09-10T19:09:10.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.762+0000 7f82daffd640 10 client.25821.objecter ms_handle_connect 0x7f82d0075b00 2024-09-10T19:09:10.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.766+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f82d4056a90 con 0x7f82e4105360 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:09.643105+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:09.647584+0000 osd.6 (osd.6) 400 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:09.874525+0000 osd.7 (osd.7) 440 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:09.877882+0000 osd.7 (osd.7) 441 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.090451+0000 osd.5 (osd.5) 436 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.093664+0000 osd.5 (osd.5) 437 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:09.643105+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:09:10.850 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:09.647584+0000 osd.6 (osd.6) 400 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:09:10.851 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:09.874525+0000 osd.7 (osd.7) 440 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:09:10.851 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:09.877882+0000 osd.7 (osd.7) 441 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:09:10.851 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.090451+0000 osd.5 (osd.5) 436 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:09:10.851 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:10 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.093664+0000 osd.5 (osd.5) 437 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:09.643105+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:09.647584+0000 osd.6 (osd.6) 400 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:09.874525+0000 osd.7 (osd.7) 440 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:09.877882+0000 osd.7 (osd.7) 441 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.090451+0000 osd.5 (osd.5) 436 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:09:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:10 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.093664+0000 osd.5 (osd.5) 437 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:09:10.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:10.958+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f82c80052f0 con 0x7f82e4105360 2024-09-10T19:09:11.078 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:11.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.550+0000 7f82daffd640 1 -- 172.21.15.83:0/4153394699 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v852) v1 ==== 91+0+0 (unknown 2473840836 0 0) 0x7f82d4056430 con 0x7f82e4105360 2024-09-10T19:09:11.558 INFO:teuthology.orchestra.run.smithi083.stderr:noscrub is unset 2024-09-10T19:09:11.559 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[212374]: debug 2024-09-10T19:09:11.502+0000 7f1b46267640 1 -- 172.21.15.83:0/1237526461 <== mon.0 v1:172.21.15.83:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 1957561174 0 0) 0x55e357f1ba00 con 0x55e357eecc00 2024-09-10T19:09:11.560 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:11.560 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.422503+0000 mgr.y (mgr.24526) 776 : cluster [DBG] pgmap v1332: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:11.561 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.553185+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:09:11.561 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.556527+0000 osd.3 (osd.3) 463 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:09:11.561 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.662391+0000 osd.6 (osd.6) 401 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:09:11.561 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: cluster 2024-09-10T19:09:10.666076+0000 osd.6 (osd.6) 402 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:09:11.561 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[32381]: audit 2024-09-10T19:09:10.963132+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:09:11.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 10 librados: watch_flush enter 2024-09-10T19:09:11.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 10 librados: watch_flush exit 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 20 client.25821.objecter shutdown clearing up homeless session... 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 10 client.25821.objecter successfully canceled tick 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 >> v1:172.21.15.83:6800/3071793735 conn(0x7f82d0075b00 legacy=0x7f82d0077fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.554+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 >> v1:172.21.15.83:6789/0 conn(0x7f82e4105360 legacy=0x7f82e40fe950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.558+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 shutdown_connections 2024-09-10T19:09:11.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.558+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 >> 172.21.15.83:0/4153394699 conn(0x7f82e40f8180 msgr2=0x7f82e4064f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:11.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.558+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 shutdown_connections 2024-09-10T19:09:11.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.558+0000 7f82ec468640 1 -- 172.21.15.83:0/4153394699 wait complete. 2024-09-10T19:09:11.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:11.558+0000 7f82ec468640 1 librados: shutdown 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.422503+0000 mgr.y (mgr.24526) 776 : cluster [DBG] pgmap v1332: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.553185+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.556527+0000 osd.3 (osd.3) 463 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.662391+0000 osd.6 (osd.6) 401 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: cluster 2024-09-10T19:09:10.666076+0000 osd.6 (osd.6) 402 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:09:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:11 smithi083 bash[18629]: audit 2024-09-10T19:09:10.963132+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:09:11.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.422503+0000 mgr.y (mgr.24526) 776 : cluster [DBG] pgmap v1332: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 573 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.553185+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:09:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.556527+0000 osd.3 (osd.3) 463 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:09:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.662391+0000 osd.6 (osd.6) 401 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:09:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: cluster 2024-09-10T19:09:10.666076+0000 osd.6 (osd.6) 402 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:09:11.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:11 smithi165 bash[22591]: audit 2024-09-10T19:09:10.963132+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:09:12.540 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[212374]: debug 2024-09-10T19:09:12.058+0000 7f1b48acf440 -1 Falling back to public interface 2024-09-10T19:09:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:12 smithi165 bash[22591]: cluster 2024-09-10T19:09:11.467124+0000 osd.4 (osd.4) 426 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:09:12.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:12 smithi165 bash[22591]: cluster 2024-09-10T19:09:11.471307+0000 osd.4 (osd.4) 427 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:09:12.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:12 smithi165 bash[22591]: audit 2024-09-10T19:09:11.552530+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:09:12.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:12 smithi165 bash[22591]: cluster 2024-09-10T19:09:11.552653+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e852: 8 total, 7 up, 8 in 2024-09-10T19:09:13.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[32381]: cluster 2024-09-10T19:09:11.467124+0000 osd.4 (osd.4) 426 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:09:13.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[32381]: cluster 2024-09-10T19:09:11.471307+0000 osd.4 (osd.4) 427 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:09:13.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[32381]: audit 2024-09-10T19:09:11.552530+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:09:13.029 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[32381]: cluster 2024-09-10T19:09:11.552653+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e852: 8 total, 7 up, 8 in 2024-09-10T19:09:13.030 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[18629]: cluster 2024-09-10T19:09:11.467124+0000 osd.4 (osd.4) 426 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:09:13.030 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[18629]: cluster 2024-09-10T19:09:11.471307+0000 osd.4 (osd.4) 427 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:09:13.030 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[18629]: audit 2024-09-10T19:09:11.552530+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.83:0/4153394699' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:09:13.030 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[18629]: cluster 2024-09-10T19:09:11.552653+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e852: 8 total, 7 up, 8 in 2024-09-10T19:09:13.030 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:12] "GET /metrics HTTP/1.1" 200 224232 "" "Prometheus/2.43.0" 2024-09-10T19:09:13.031 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[18629]: cluster 2024-09-10T19:09:12.423560+0000 mgr.y (mgr.24526) 777 : cluster [DBG] pgmap v1334: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[18629]: cluster 2024-09-10T19:09:12.535814+0000 osd.3 (osd.3) 464 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[18629]: cluster 2024-09-10T19:09:12.539311+0000 osd.3 (osd.3) 465 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.385355+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.389698+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[32381]: cluster 2024-09-10T19:09:12.423560+0000 mgr.y (mgr.24526) 777 : cluster [DBG] pgmap v1334: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[32381]: cluster 2024-09-10T19:09:12.535814+0000 osd.3 (osd.3) 464 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:09:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[32381]: cluster 2024-09-10T19:09:12.539311+0000 osd.3 (osd.3) 465 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:09:13.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.385355+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:09:13.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:13 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.389698+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:09:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:13 smithi165 bash[22591]: cluster 2024-09-10T19:09:12.423560+0000 mgr.y (mgr.24526) 777 : cluster [DBG] pgmap v1334: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:13 smithi165 bash[22591]: cluster 2024-09-10T19:09:12.535814+0000 osd.3 (osd.3) 464 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:09:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:13 smithi165 bash[22591]: cluster 2024-09-10T19:09:12.539311+0000 osd.3 (osd.3) 465 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:09:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:13 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.385355+0000 osd.0 (osd.0) 415 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:09:13.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:13 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.389698+0000 osd.0 (osd.0) 416 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:09:14.332 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.27490.0:5779 64.1d 64:bd59bd67:test-rados-api-smithi083-77828-54::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e425)", 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 566.01331796199997, 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019346334999999999, 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.333 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.27490", 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3932924178", 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5779 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312035+0000", 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312032+0000", 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999971 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312044+0000", 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2741999999999999e-05 2024-09-10T19:09:14.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312067+0000", 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2921999999999998e-05 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312077+0000", 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0118e-05 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312150+0000", 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2853000000000002e-05 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.312159+0000", 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3010000000000007e-06 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.319968+0000", 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078098359999999997 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.320183+0000", 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021412800000000001 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.320350+0000", 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016759300000000001 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.321096+0000", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000745544 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.324693+0000", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035975019999999998 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331284+0000", 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065907539999999999 2024-09-10T19:09:14.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331338+0000", 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3717000000000001e-05 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T18:59:48.331381+0000", 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3377000000000002e-05 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(1.7 REQUEST e660)", 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:03.429419+0000", 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 250.895934127, 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063424299999999996, 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429419+0000", 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429419+0000", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429418+0000", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429425+0000", 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6460000000000004e-06 2024-09-10T19:09:14.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429456+0000", 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1322999999999998e-05 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429470+0000", 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4133e-05 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429672+0000", 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020179700000000001 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.429699+0000", 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.743e-05 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.435616+0000", 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059164500000000002 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.435688+0000", 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2175999999999995e-05 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:03.435761+0000", 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3034999999999994e-05 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.345 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:246 241.5s0 241:a9dad0c8:test-rados-api-smithi083-77828-117::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e670)", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:13.315832+0000", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 241.00952062900001, 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014675466999999999, 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 246 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315832+0000", 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315832+0000", 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315830+0000", 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315840+0000", 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7110000000000007e-06 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315871+0000", 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0997000000000002e-05 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315881+0000", 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0094e-05 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.315945+0000", 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3887999999999995e-05 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.349 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.316088+0000", 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014390299999999999 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.316936+0000", 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084779099999999997 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.319732+0000", 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027956890000000001 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.330431+0000", 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010698993 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:13.330508+0000", 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6639999999999998e-05 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.351 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:251 241.5s0 241:a9dad0c8:test-rados-api-smithi083-77828-117::foo:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e671)", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:14.307761+0000", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 240.01759134100001, 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021644595999999999, 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 251 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307761+0000", 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307761+0000", 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307756+0000", 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999943 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307783+0000", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7025999999999998e-05 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307829+0000", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6385999999999999e-05 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307846+0000", 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6813e-05 2024-09-10T19:09:14.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307958+0000", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011189 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.307967+0000", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2820000000000002e-06 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.318935+0000", 2024-09-10T19:09:14.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010967475000000001 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.319010+0000", 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4987999999999994e-05 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_started", 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.320018+0000", 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010081739999999999 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_committed", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.321420+0000", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001402431 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.329291+0000", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0078708500000000004 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:14.329406+0000", 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000114811 2024-09-10T19:09:14.357 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(osd.3.29:122 241.5s0 241:a9dad0c8:test-rados-api-smithi083-77828-117::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e674)", 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:17.316793+0000", 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 237.008560098, 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005531635, 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316793+0000", 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316793+0000", 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316791+0000", 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.359 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316800+0000", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8310000000000007e-06 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316811+0000", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1148999999999999e-05 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316817+0000", 2024-09-10T19:09:14.360 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8629999999999996e-06 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316949+0000", 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131715 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.316959+0000", 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0274e-05 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.361 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.321324+0000", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043652259999999998 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.321391+0000", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6159000000000001e-05 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:17.322324+0000", 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093359499999999995 2024-09-10T19:09:14.362 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:475 246.19 246:9a946601:test-rados-api-smithi083-77828-121::foo:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e694)", 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:50.329618+0000", 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 203.99573474499999, 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0092052450000000008, 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.363 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 475 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329618+0000", 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329618+0000", 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.364 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329613+0000", 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999948 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329633+0000", 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9936e-05 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.365 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329678+0000", 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4913000000000002e-05 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329699+0000", 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1123000000000001e-05 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329855+0000", 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015677099999999999 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.329866+0000", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0727000000000001e-05 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.332590+0000", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027237519999999999 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.332947+0000", 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00035687700000000002 2024-09-10T19:09:14.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.336856+0000", 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039089090000000003 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.337011+0000", 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155484 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.337278+0000", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026719299999999999 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.337876+0000", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00059789200000000004 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.338500+0000", 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062412199999999996 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.338741+0000", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240158 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.338787+0000", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5964000000000002e-05 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.338823+0000", 2024-09-10T19:09:14.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6637000000000002e-05 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:578 246.19 246:9a946601:test-rados-api-smithi083-77828-121::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e694)", 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:50.430737+0000", 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 203.89461555299999, 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053183520000000001, 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.371 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 578 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430737+0000", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430737+0000", 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430734+0000", 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999967 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430748+0000", 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4566000000000001e-05 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430781+0000", 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2982999999999999e-05 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430799+0000", 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.749e-05 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.430935+0000", 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135878 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.431124+0000", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018949400000000001 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.435728+0000", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046037059999999999 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.435946+0000", 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021833700000000001 2024-09-10T19:09:14.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:50.436056+0000", 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109378 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:580 246.0 246:00000000::::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:05:52.443634+0000", 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 201.88171909900001, 2024-09-10T19:09:14.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00693824, 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 580 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443634+0000", 2024-09-10T19:09:14.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443634+0000", 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443628+0000", 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999943 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443653+0000", 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5207000000000001e-05 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443698+0000", 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5158999999999999e-05 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443719+0000", 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1386000000000002e-05 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443874+0000", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015425200000000001 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.443902+0000", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7982000000000001e-05 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.450402+0000", 2024-09-10T19:09:14.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065003769999999999 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.450439+0000", 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6538000000000002e-05 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:05:52.450572+0000", 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013325199999999999 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.381 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1101 250.7 250:e14dd380:test-rados-api-smithi083-77828-129::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:06:24.334259+0000", 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 169.991093791, 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011413339, 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1101 2024-09-10T19:09:14.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334259+0000", 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334259+0000", 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334254+0000", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999948 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334277+0000", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3649999999999999e-05 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334318+0000", 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0296000000000002e-05 2024-09-10T19:09:14.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334333+0000", 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5423000000000001e-05 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334422+0000", 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8653999999999995e-05 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.334428+0000", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2840000000000003e-06 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.339042+0000", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046142689999999998 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.339234+0000", 2024-09-10T19:09:14.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000191428 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.344330+0000", 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050963859999999996 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.344498+0000", 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016820800000000001 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.344647+0000", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014905699999999999 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.345231+0000", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058382300000000002 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.345450+0000", 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000218879 2024-09-10T19:09:14.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.345574+0000", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000124041 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.345621+0000", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6774999999999999e-05 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:24.345672+0000", 2024-09-10T19:09:14.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1456000000000003e-05 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(4.15 REQUEST e729)", 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:06:28.753431+0000", 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 165.57192150500001, 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059290560000000003, 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753431+0000", 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753431+0000", 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753431+0000", 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753437+0000", 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3199999999999996e-06 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753467+0000", 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9762999999999999e-05 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753480+0000", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3236e-05 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753602+0000", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121975 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.753627+0000", 2024-09-10T19:09:14.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4790000000000002e-05 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.759284+0000", 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056572200000000001 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.759308+0000", 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3733000000000001e-05 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:28.759360+0000", 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2605999999999997e-05 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1150 241.5s0 241:a0000000:test-rados-api-smithi083-77828-129:::head [pgnls start_epoch 730] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e730)", 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:06:29.389653+0000", 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 164.93569967600001, 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059410579999999999, 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.395 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1150 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389653+0000", 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389653+0000", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389649+0000", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389665+0000", 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6214000000000001e-05 2024-09-10T19:09:14.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389695+0000", 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9187e-05 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389704+0000", 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4809999999999995e-06 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389883+0000", 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017905500000000001 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.389894+0000", 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1402e-05 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.395316+0000", 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054215460000000002 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.395408+0000", 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2169000000000005e-05 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:29.395594+0000", 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018589100000000001 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1464 251.c 251:30000000::::head [pgnls start_epoch 740] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e740)", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:06:39.361587+0000", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 154.963766177, 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010574373999999999, 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1464 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361587+0000", 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361587+0000", 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361580+0000", 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999933 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361610+0000", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0431000000000001e-05 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361670+0000", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9324999999999999e-05 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361695+0000", 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5624999999999999e-05 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361809+0000", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011353600000000001 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.361816+0000", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7200000000000006e-06 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.371971+0000", 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010154421 2024-09-10T19:09:14.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.372015+0000", 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4162000000000002e-05 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:39.372161+0000", 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014589099999999999 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(12.6 REQUEST e741)", 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:06:41.113327+0000", 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 153.21202554800001, 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083207019999999993, 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113327+0000", 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113327+0000", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113327+0000", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999995 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113333+0000", 2024-09-10T19:09:14.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1929999999999998e-06 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113362+0000", 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8867e-05 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113375+0000", 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3638999999999999e-05 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113505+0000", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012945900000000001 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.113520+0000", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5201999999999999e-05 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.121584+0000", 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080640780000000006 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.121608+0000", 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4039000000000001e-05 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:06:41.121648+0000", 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9830999999999999e-05 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1794 254.7 254:e3109cb7:test-rados-api-smithi083-77828-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e767)", 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:07:06.381674+0000", 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 127.943679108, 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036376954000000003, 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1794 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.381674+0000", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.381674+0000", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.381669+0000", 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999957 2024-09-10T19:09:14.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.388336+0000", 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066664960000000001 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.388372+0000", 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5840999999999998e-05 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.388384+0000", 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1943999999999999e-05 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.388488+0000", 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000104042 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.388496+0000", 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3049999999999993e-06 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.392793+0000", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042970700000000001 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.393192+0000", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039887400000000002 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.399653+0000", 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064614190000000004 2024-09-10T19:09:14.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.399776+0000", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012261600000000001 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.399920+0000", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014448400000000001 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.406116+0000", 2024-09-10T19:09:14.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061951990000000002 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.417875+0000", 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011759357 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.417943+0000", 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7829000000000004e-05 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418012+0000", 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9212000000000001e-05 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418051+0000", 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8504999999999998e-05 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.418 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1795 254.7 254:e3109cb7:test-rados-api-smithi083-77828-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e767)", 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:07:06.418394+0000", 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 127.906958598, 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031943669000000001, 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1795 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418394+0000", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418394+0000", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418392+0000", 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:09:14.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418401+0000", 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4520000000000003e-06 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418419+0000", 2024-09-10T19:09:14.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8232000000000001e-05 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418426+0000", 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9639999999999997e-06 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418496+0000", 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9861000000000001e-05 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.418691+0000", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019503700000000001 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.449959+0000", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.031268140999999999 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.450196+0000", 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023658199999999999 2024-09-10T19:09:14.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:06.450338+0000", 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014173000000000001 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:1933 255.10 255:08000000::::head [pgnls start_epoch 777] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e777)", 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:07:17.456387+0000", 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 116.868965516, 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003043993, 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1933 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456387+0000", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456387+0000", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456382+0000", 2024-09-10T19:09:14.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999948 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456405+0000", 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.262e-05 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456438+0000", 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3413e-05 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456449+0000", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1141999999999999e-05 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456548+0000", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9160000000000006e-05 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.456558+0000", 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6520000000000001e-06 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.459136+0000", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002577551 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.459218+0000", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2280999999999994e-05 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:17.459431+0000", 2024-09-10T19:09:14.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000213351 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:2145 257.1c 257:3b4d2b8d:test-rados-api-smithi083-77828-143::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e804)", 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:07:50.395574+0000", 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 83.929779073000006, 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073302640000000004, 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.430 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2145 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395574+0000", 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395574+0000", 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395570+0000", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999962 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395585+0000", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4785e-05 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395608+0000", 2024-09-10T19:09:14.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3258999999999999e-05 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395619+0000", 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1104000000000001e-05 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.395743+0000", 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012407499999999999 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:07:50.402904+0000", 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071606279999999996 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:2807 259.0 259:00000000:%01:::head [pgnls start_epoch 829] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e829)", 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:08:32.608259+0000", 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.717093509000001, 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033541500000000002, 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.434 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2807 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608259+0000", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608259+0000", 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608254+0000", 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999943 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608277+0000", 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3136e-05 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608320+0000", 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3526000000000003e-05 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608340+0000", 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9296999999999998e-05 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608481+0000", 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014112700000000001 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.608492+0000", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0866999999999999e-05 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.611331+0000", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028394330000000001 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.611379+0000", 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7589999999999997e-05 2024-09-10T19:09:14.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:32.611613+0000", 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000234811 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.49694.0:2912 260.1 260:84e153a7:test-rados-api-smithi083-77828-149::bar:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:08:35.424130+0000", 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.901222681, 2024-09-10T19:09:14.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.383929097, 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.49694", 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.83:0/3778548406", 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2912 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424130+0000", 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424130+0000", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424128+0000", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.9999976 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424138+0000", 2024-09-10T19:09:14.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8640000000000008e-06 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424153+0000", 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5763000000000001e-05 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424161+0000", 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3749999999999997e-06 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424264+0000", 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102878 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.424272+0000", 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7069999999999998e-06 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.803742+0000", 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.37946943799999999 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.804016+0000", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027428199999999999 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.804331+0000", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031516900000000002 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.807373+0000", 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030421649999999999 2024-09-10T19:09:14.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.807641+0000", 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00026777600000000001 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.807806+0000", 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016500400000000001 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:35.808059+0000", 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025302899999999999 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(3.1b REQUEST e848)", 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-10T19:08:56.497825+0000", 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 17.827528061999999, 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050917740000000003, 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-10T19:09:14.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497825+0000", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497825+0000", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497824+0000", 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4294967295.999999 2024-09-10T19:09:14.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497832+0000", 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8660000000000006e-06 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497875+0000", 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3250000000000001e-05 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-10T19:09:14.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.497891+0000", 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6575999999999999e-05 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.498045+0000", 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015312799999999999 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-10T19:09:14.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.498065+0000", 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0888999999999999e-05 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.502805+0000", 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047396230000000001 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.502845+0000", 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9782e-05 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-10T19:09:14.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-10T19:08:56.502916+0000", 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1589999999999997e-05 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-10T19:09:14.451 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-10T19:09:14.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:12.889165+0000 osd.2 (osd.2) 339 : cluster [DBG] 12.a scrub starts 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:12.892815+0000 osd.2 (osd.2) 340 : cluster [DBG] 12.a scrub ok 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.765430+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.770480+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.990938+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:13.995258+0000 osd.5 (osd.5) 439 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:14.355804+0000 osd.0 (osd.0) 417 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:14.359321+0000 osd.0 (osd.0) 418 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[18629]: cluster 2024-09-10T19:09:14.422574+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e853: 8 total, 7 up, 8 in 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:12.889165+0000 osd.2 (osd.2) 339 : cluster [DBG] 12.a scrub starts 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:12.892815+0000 osd.2 (osd.2) 340 : cluster [DBG] 12.a scrub ok 2024-09-10T19:09:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.765430+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.770480+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.990938+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:13.995258+0000 osd.5 (osd.5) 439 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:14.355804+0000 osd.0 (osd.0) 417 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:14.359321+0000 osd.0 (osd.0) 418 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:09:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[32381]: cluster 2024-09-10T19:09:14.422574+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e853: 8 total, 7 up, 8 in 2024-09-10T19:09:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:12.889165+0000 osd.2 (osd.2) 339 : cluster [DBG] 12.a scrub starts 2024-09-10T19:09:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:12.892815+0000 osd.2 (osd.2) 340 : cluster [DBG] 12.a scrub ok 2024-09-10T19:09:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.765430+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:09:14.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.770480+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:09:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.990938+0000 osd.5 (osd.5) 438 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:09:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:13.995258+0000 osd.5 (osd.5) 439 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:09:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:14.355804+0000 osd.0 (osd.0) 417 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:09:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:14.359321+0000 osd.0 (osd.0) 418 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:09:14.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:14 smithi165 bash[22591]: cluster 2024-09-10T19:09:14.422574+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e853: 8 total, 7 up, 8 in 2024-09-10T19:09:15.290 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:14 smithi083 bash[212374]: debug 2024-09-10T19:09:14.850+0000 7f1b48acf440 -1 osd.1 789 log_to_monitors true 2024-09-10T19:09:15.755 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-09-10T19:09:15.756 INFO:teuthology.orchestra.run.smithi083.stdout: "ops": [], 2024-09-10T19:09:15.756 INFO:teuthology.orchestra.run.smithi083.stdout: "num_ops": 0 2024-09-10T19:09:15.756 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-09-10T19:09:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: cluster 2024-09-10T19:09:14.424898+0000 mgr.y (mgr.24526) 778 : cluster [DBG] pgmap v1336: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: audit 2024-09-10T19:09:14.582459+0000 mon.c (mon.2) 891 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: audit 2024-09-10T19:09:14.862997+0000 mon.c (mon.2) 892 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: audit 2024-09-10T19:09:14.863568+0000 mon.a (mon.0) 3784 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.036125+0000 osd.5 (osd.5) 440 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:09:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:15 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.039084+0000 osd.5 (osd.5) 441 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:09:16.003 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[212374]: debug 2024-09-10T19:09:15.678+0000 7f1b3e076640 -1 osd.1 854 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: cluster 2024-09-10T19:09:14.424898+0000 mgr.y (mgr.24526) 778 : cluster [DBG] pgmap v1336: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: audit 2024-09-10T19:09:14.582459+0000 mon.c (mon.2) 891 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: audit 2024-09-10T19:09:14.862997+0000 mon.c (mon.2) 892 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: audit 2024-09-10T19:09:14.863568+0000 mon.a (mon.0) 3784 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.036125+0000 osd.5 (osd.5) 440 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:09:16.004 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.039084+0000 osd.5 (osd.5) 441 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: cluster 2024-09-10T19:09:14.424898+0000 mgr.y (mgr.24526) 778 : cluster [DBG] pgmap v1336: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: audit 2024-09-10T19:09:14.582459+0000 mon.c (mon.2) 891 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: audit 2024-09-10T19:09:14.862997+0000 mon.c (mon.2) 892 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: audit 2024-09-10T19:09:14.863568+0000 mon.a (mon.0) 3784 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.036125+0000 osd.5 (osd.5) 440 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:09:16.005 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:15 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.039084+0000 osd.5 (osd.5) 441 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:09:16.728 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-10T19:09:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.458455+0000 osd.4 (osd.4) 428 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.463898+0000 osd.4 (osd.4) 429 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: audit 2024-09-10T19:09:15.588461+0000 mon.a (mon.0) 3785 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.588552+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e854: 8 total, 7 up, 8 in 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: audit 2024-09-10T19:09:15.591925+0000 mon.c (mon.2) 893 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: audit 2024-09-10T19:09:15.592948+0000 mon.a (mon.0) 3787 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.078437+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.081956+0000 osd.5 (osd.5) 443 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.371184+0000 osd.0 (osd.0) 419 : cluster [DBG] 12.c scrub starts 2024-09-10T19:09:16.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:16 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.374699+0000 osd.0 (osd.0) 420 : cluster [DBG] 12.c scrub ok 2024-09-10T19:09:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.458455+0000 osd.4 (osd.4) 428 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:09:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.463898+0000 osd.4 (osd.4) 429 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:09:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: audit 2024-09-10T19:09:15.588461+0000 mon.a (mon.0) 3785 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:09:17.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.588552+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e854: 8 total, 7 up, 8 in 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: audit 2024-09-10T19:09:15.591925+0000 mon.c (mon.2) 893 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: audit 2024-09-10T19:09:15.592948+0000 mon.a (mon.0) 3787 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.078437+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.081956+0000 osd.5 (osd.5) 443 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.371184+0000 osd.0 (osd.0) 419 : cluster [DBG] 12.c scrub starts 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.374699+0000 osd.0 (osd.0) 420 : cluster [DBG] 12.c scrub ok 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.458455+0000 osd.4 (osd.4) 428 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.463898+0000 osd.4 (osd.4) 429 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:09:17.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: audit 2024-09-10T19:09:15.588461+0000 mon.a (mon.0) 3785 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.588552+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e854: 8 total, 7 up, 8 in 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: audit 2024-09-10T19:09:15.591925+0000 mon.c (mon.2) 893 : audit [INF] from='osd.1 v1:172.21.15.83:6805/1048694267' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: audit 2024-09-10T19:09:15.592948+0000 mon.a (mon.0) 3787 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.078437+0000 osd.5 (osd.5) 442 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.081956+0000 osd.5 (osd.5) 443 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.371184+0000 osd.0 (osd.0) 419 : cluster [DBG] 12.c scrub starts 2024-09-10T19:09:17.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:16 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.374699+0000 osd.0 (osd.0) 420 : cluster [DBG] 12.c scrub ok 2024-09-10T19:09:17.801 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.427443+0000 mgr.y (mgr.24526) 779 : cluster [DBG] pgmap v1338: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.579022+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:09:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.582858+0000 osd.3 (osd.3) 467 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.585130+0000 mon.a (mon.0) 3788 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.593282+0000 mon.a (mon.0) 3789 : cluster [INF] osd.1 v1:172.21.15.83:6805/1048694267 boot 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: audit 2024-09-10T19:09:16.593320+0000 mon.c (mon.2) 894 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.594353+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.720084+0000 osd.7 (osd.7) 444 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:16.725224+0000 osd.7 (osd.7) 445 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:09:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:17 smithi165 bash[22591]: cluster 2024-09-10T19:09:17.597012+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-09-10T19:09:17.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.427443+0000 mgr.y (mgr.24526) 779 : cluster [DBG] pgmap v1338: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:17.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.579022+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:09:17.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.582858+0000 osd.3 (osd.3) 467 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:09:17.902 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.585130+0000 mon.a (mon.0) 3788 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.593282+0000 mon.a (mon.0) 3789 : cluster [INF] osd.1 v1:172.21.15.83:6805/1048694267 boot 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: audit 2024-09-10T19:09:16.593320+0000 mon.c (mon.2) 894 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.594353+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.720084+0000 osd.7 (osd.7) 444 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:16.725224+0000 osd.7 (osd.7) 445 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[18629]: cluster 2024-09-10T19:09:17.597012+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.427443+0000 mgr.y (mgr.24526) 779 : cluster [DBG] pgmap v1338: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:17.903 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.579022+0000 osd.3 (osd.3) 466 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.582858+0000 osd.3 (osd.3) 467 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.585130+0000 mon.a (mon.0) 3788 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.593282+0000 mon.a (mon.0) 3789 : cluster [INF] osd.1 v1:172.21.15.83:6805/1048694267 boot 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: audit 2024-09-10T19:09:16.593320+0000 mon.c (mon.2) 894 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.594353+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.720084+0000 osd.7 (osd.7) 444 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:16.725224+0000 osd.7 (osd.7) 445 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:09:17.904 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:17 smithi083 bash[32381]: cluster 2024-09-10T19:09:17.597012+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-09-10T19:09:18.865 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:18.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.842751+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:18.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:15.843275+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:18.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:17.437171+0000 osd.2 (osd.2) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:18.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:17.437692+0000 osd.2 (osd.2) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:18.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:17.523960+0000 osd.4 (osd.4) 430 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:09:18.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:17.527507+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:09:18.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.367743+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:09:18.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:18 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.371102+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.842751+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:15.843275+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:17.437171+0000 osd.2 (osd.2) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:17.437692+0000 osd.2 (osd.2) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:17.523960+0000 osd.4 (osd.4) 430 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:17.527507+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.367743+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:09:19.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.371102+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.842751+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:15.843275+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:17.437171+0000 osd.2 (osd.2) 341 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:17.437692+0000 osd.2 (osd.2) 342 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:17.523960+0000 osd.4 (osd.4) 430 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:17.527507+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.367743+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:09:19.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:18 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.371102+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.428901+0000 mgr.y (mgr.24526) 780 : cluster [DBG] pgmap v1341: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: audit 2024-09-10T19:09:18.503452+0000 mgr.y (mgr.24526) 781 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.690674+0000 osd.6 (osd.6) 403 : cluster [DBG] 4.e scrub starts 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.694544+0000 osd.6 (osd.6) 404 : cluster [DBG] 4.e scrub ok 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.781913+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.e scrub starts 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.786903+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.e scrub ok 2024-09-10T19:09:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.426617+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.428901+0000 mgr.y (mgr.24526) 780 : cluster [DBG] pgmap v1341: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: audit 2024-09-10T19:09:18.503452+0000 mgr.y (mgr.24526) 781 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.690674+0000 osd.6 (osd.6) 403 : cluster [DBG] 4.e scrub starts 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.694544+0000 osd.6 (osd.6) 404 : cluster [DBG] 4.e scrub ok 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.781913+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.e scrub starts 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.786903+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.e scrub ok 2024-09-10T19:09:19.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:19 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.426617+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-09-10T19:09:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.428901+0000 mgr.y (mgr.24526) 780 : cluster [DBG] pgmap v1341: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: audit 2024-09-10T19:09:18.503452+0000 mgr.y (mgr.24526) 781 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.690674+0000 osd.6 (osd.6) 403 : cluster [DBG] 4.e scrub starts 2024-09-10T19:09:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.694544+0000 osd.6 (osd.6) 404 : cluster [DBG] 4.e scrub ok 2024-09-10T19:09:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.781913+0000 osd.7 (osd.7) 446 : cluster [DBG] 3.e scrub starts 2024-09-10T19:09:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.786903+0000 osd.7 (osd.7) 447 : cluster [DBG] 3.e scrub ok 2024-09-10T19:09:20.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:19 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.426617+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-09-10T19:09:20.848 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.936937+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.d scrub starts 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:18.940897+0000 osd.2 (osd.2) 344 : cluster [DBG] 4.d scrub ok 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.558747+0000 osd.4 (osd.4) 432 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.567714+0000 osd.4 (osd.4) 433 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.702544+0000 osd.3 (osd.3) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.706048+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.314884+0000 osd.0 (osd.0) 423 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:09:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.318331+0000 osd.0 (osd.0) 424 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.936937+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.d scrub starts 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:18.940897+0000 osd.2 (osd.2) 344 : cluster [DBG] 4.d scrub ok 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.558747+0000 osd.4 (osd.4) 432 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.567714+0000 osd.4 (osd.4) 433 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.702544+0000 osd.3 (osd.3) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.706048+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.314884+0000 osd.0 (osd.0) 423 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:09:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:20 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.318331+0000 osd.0 (osd.0) 424 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:09:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.936937+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.d scrub starts 2024-09-10T19:09:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:18.940897+0000 osd.2 (osd.2) 344 : cluster [DBG] 4.d scrub ok 2024-09-10T19:09:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.558747+0000 osd.4 (osd.4) 432 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:09:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.567714+0000 osd.4 (osd.4) 433 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:09:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.702544+0000 osd.3 (osd.3) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:09:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.706048+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:09:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.314884+0000 osd.0 (osd.0) 423 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:09:21.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:20 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.318331+0000 osd.0 (osd.0) 424 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:09:21.629 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.931272+0000 osd.2 (osd.2) 345 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:19.934667+0000 osd.2 (osd.2) 346 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.430419+0000 mgr.y (mgr.24526) 782 : cluster [DBG] pgmap v1343: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.545400+0000 osd.4 (osd.4) 434 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.548988+0000 osd.4 (osd.4) 435 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.670013+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.673452+0000 osd.3 (osd.3) 471 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.763180+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:09:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:20.767665+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.300604+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.305965+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.931272+0000 osd.2 (osd.2) 345 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:19.934667+0000 osd.2 (osd.2) 346 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.430419+0000 mgr.y (mgr.24526) 782 : cluster [DBG] pgmap v1343: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.545400+0000 osd.4 (osd.4) 434 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.548988+0000 osd.4 (osd.4) 435 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.670013+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.673452+0000 osd.3 (osd.3) 471 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.763180+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:20.767665+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:09:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.300604+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:09:22.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:21 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.305965+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:09:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.931272+0000 osd.2 (osd.2) 345 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:09:22.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:19.934667+0000 osd.2 (osd.2) 346 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.430419+0000 mgr.y (mgr.24526) 782 : cluster [DBG] pgmap v1343: 174 pgs: 46 active+undersized, 16 active+undersized+degraded, 112 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 76/591 objects degraded (12.860%) 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.545400+0000 osd.4 (osd.4) 434 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.548988+0000 osd.4 (osd.4) 435 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.670013+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.673452+0000 osd.3 (osd.3) 471 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.763180+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:20.767665+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.300604+0000 osd.0 (osd.0) 425 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:09:22.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:21 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.305965+0000 osd.0 (osd.0) 426 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:09:22.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/1410235904 >> v1:172.21.15.83:6790/0 conn(0x7f5cec107e30 legacy=0x7f5cec10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:22.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/1410235904 shutdown_connections 2024-09-10T19:09:22.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/1410235904 >> 172.21.15.83:0/1410235904 conn(0x7f5cec075290 msgr2=0x7f5cec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:22.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/1410235904 shutdown_connections 2024-09-10T19:09:22.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/1410235904 wait complete. 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 librados: starting msgr at 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 librados: starting objecter 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 Processor -- start 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- start start 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 librados: setting wanted keys 2024-09-10T19:09:22.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 librados: calling monclient init 2024-09-10T19:09:22.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cec119f80 con 0x7f5cec107e30 2024-09-10T19:09:22.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cec11b140 con 0x7f5cec1014e0 2024-09-10T19:09:22.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cf0ee4640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cec11c340 con 0x7f5cec1042a0 2024-09-10T19:09:22.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cead9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f5cec107e30 0x7f5cec118680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:55094/0 (socket says 172.21.15.83:55094) 2024-09-10T19:09:22.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5cead9d640 1 -- 172.21.15.83:0/2794446184 learned_addr learned my addr 172.21.15.83:0/2794446184 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce3fff640 1 --1- 172.21.15.83:0/2794446184 >> v1:172.21.15.83:6790/0 conn(0x7f5cec1042a0 0x7f5cec0fd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:40246/0 (socket says 172.21.15.83:40246) 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec1042a0 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec107e30 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f5cec1014e0 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255072730 0 0) 0x7f5cec119f80 con 0x7f5cec107e30 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd80034e0 con 0x7f5cec107e30 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187098168 0 0) 0x7f5cec11c340 con 0x7f5cec1042a0 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec119f80 con 0x7f5cec1042a0 2024-09-10T19:09:22.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020942432 0 0) 0x7f5cec11b140 con 0x7f5cec1014e0 2024-09-10T19:09:22.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cec11c340 con 0x7f5cec1014e0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818385164 0 0) 0x7f5cd80034e0 con 0x7f5cec107e30 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cec11b140 con 0x7f5cec107e30 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ce40023d0 con 0x7f5cec107e30 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.858+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250535109 0 0) 0x7f5cec119f80 con 0x7f5cec1042a0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cd80034e0 con 0x7f5cec1042a0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5cd4001a50 con 0x7f5cec1042a0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2263320908 0 0) 0x7f5cec11c340 con 0x7f5cec1014e0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cec119f80 con 0x7f5cec1014e0 2024-09-10T19:09:22.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5cdc001840 con 0x7f5cec1014e0 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2864312597 0 0) 0x7f5cec11b140 con 0x7f5cec107e30 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 >> v1:172.21.15.83:6790/0 conn(0x7f5cec1042a0 legacy=0x7f5cec0fd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 >> v1:172.21.15.165:6789/0 conn(0x7f5cec1014e0 legacy=0x7f5cec0fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cec11d540 con 0x7f5cec107e30 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cec11a170 con 0x7f5cec107e30 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5cf0ee4640 10 client.25833.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cec11a5c0 con 0x7f5cec107e30 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5cf0ee4640 1 librados: init done 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5ce40026a0 con 0x7f5cec107e30 2024-09-10T19:09:22.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ce4003960 con 0x7f5cec107e30 2024-09-10T19:09:22.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc005010 con 0x7f5cec107e30 2024-09-10T19:09:22.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.862+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5ce4001b70 con 0x7f5cec107e30 2024-09-10T19:09:22.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(857..857 src has 750..857) v4 ==== 7363+0+0 (unknown 4229781586 0 0) 0x7f5ce408a860 con 0x7f5cec107e30 2024-09-10T19:09:22.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 10 client.25833.objecter ms_dispatch 0x7f5cec000b90 osd_map(857..857 src has 750..857) v4 2024-09-10T19:09:22.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 3 client.25833.objecter handle_osd_map got epochs [857,857] > 0 2024-09-10T19:09:22.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 3 client.25833.objecter handle_osd_map decoding full epoch 857 2024-09-10T19:09:22.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 20 client.25833.objecter dump_active .. 0 homeless 2024-09-10T19:09:22.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.866+0000 7f5ce2ffd640 10 client.25833.objecter ms_handle_connect 0x7f5cd8073a70 2024-09-10T19:09:22.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:22.874+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5ce40570a0 con 0x7f5cec107e30 2024-09-10T19:09:22.921 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:22] "GET /metrics HTTP/1.1" 200 224234 "" "Prometheus/2.43.0" 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.667119+0000 osd.3 (osd.3) 472 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.669912+0000 osd.3 (osd.3) 473 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.809454+0000 osd.1 (osd.1) 3 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[32381]: cluster 2024-09-10T19:09:21.812981+0000 osd.1 (osd.1) 4 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.667119+0000 osd.3 (osd.3) 472 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.669912+0000 osd.3 (osd.3) 473 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.809454+0000 osd.1 (osd.1) 3 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:09:22.922 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:22 smithi083 bash[18629]: cluster 2024-09-10T19:09:21.812981+0000 osd.1 (osd.1) 4 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:09:23.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.066+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5ccc0052c0 con 0x7f5cec107e30 2024-09-10T19:09:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:22 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.667119+0000 osd.3 (osd.3) 472 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:09:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:22 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.669912+0000 osd.3 (osd.3) 473 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:09:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:22 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.809454+0000 osd.1 (osd.1) 3 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:09:23.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:22 smithi165 bash[22591]: cluster 2024-09-10T19:09:21.812981+0000 osd.1 (osd.1) 4 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:09:23.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.654+0000 7f5ce2ffd640 1 -- 172.21.15.83:0/2794446184 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v858) v1 ==== 101+0+0 (unknown 3530457387 0 0) 0x7f5ce4056a40 con 0x7f5cec107e30 2024-09-10T19:09:23.662 INFO:teuthology.orchestra.run.smithi083.stderr:nodeep-scrub is unset 2024-09-10T19:09:23.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 10 librados: watch_flush enter 2024-09-10T19:09:23.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 10 librados: watch_flush exit 2024-09-10T19:09:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 20 client.25833.objecter shutdown clearing up homeless session... 2024-09-10T19:09:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 10 client.25833.objecter successfully canceled tick 2024-09-10T19:09:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5cd8073a70 legacy=0x7f5cd8075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:23.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 >> v1:172.21.15.83:6789/0 conn(0x7f5cec107e30 legacy=0x7f5cec118680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 shutdown_connections 2024-09-10T19:09:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 >> 172.21.15.83:0/2794446184 conn(0x7f5cec075290 msgr2=0x7f5cec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 shutdown_connections 2024-09-10T19:09:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 -- 172.21.15.83:0/2794446184 wait complete. 2024-09-10T19:09:23.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:23.658+0000 7f5cf0ee4640 1 librados: shutdown 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.184779+0000 osd.5 (osd.5) 444 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.187998+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.432283+0000 mgr.y (mgr.24526) 783 : cluster [DBG] pgmap v1344: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.628095+0000 mon.a (mon.0) 3793 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized) 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.645443+0000 osd.6 (osd.6) 405 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.649009+0000 osd.6 (osd.6) 406 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:09:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.762785+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.765840+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.821959+0000 osd.1 (osd.1) 5 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: cluster 2024-09-10T19:09:22.824979+0000 osd.1 (osd.1) 6 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[18629]: audit 2024-09-10T19:09:23.073891+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.184779+0000 osd.5 (osd.5) 444 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.187998+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.432283+0000 mgr.y (mgr.24526) 783 : cluster [DBG] pgmap v1344: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.628095+0000 mon.a (mon.0) 3793 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized) 2024-09-10T19:09:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.645443+0000 osd.6 (osd.6) 405 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.649009+0000 osd.6 (osd.6) 406 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.762785+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.765840+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.821959+0000 osd.1 (osd.1) 5 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: cluster 2024-09-10T19:09:22.824979+0000 osd.1 (osd.1) 6 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:09:24.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:23 smithi083 bash[32381]: audit 2024-09-10T19:09:23.073891+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:09:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.184779+0000 osd.5 (osd.5) 444 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:09:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.187998+0000 osd.5 (osd.5) 445 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.432283+0000 mgr.y (mgr.24526) 783 : cluster [DBG] pgmap v1344: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.628095+0000 mon.a (mon.0) 3793 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 76/591 objects degraded (12.860%), 16 pgs degraded, 62 pgs undersized) 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.645443+0000 osd.6 (osd.6) 405 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.649009+0000 osd.6 (osd.6) 406 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.762785+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.765840+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.821959+0000 osd.1 (osd.1) 5 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: cluster 2024-09-10T19:09:22.824979+0000 osd.1 (osd.1) 6 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:09:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:23 smithi165 bash[22591]: audit 2024-09-10T19:09:23.073891+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.173466+0000 osd.5 (osd.5) 446 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.177947+0000 osd.5 (osd.5) 447 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.640706+0000 mon.a (mon.0) 3795 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: audit 2024-09-10T19:09:23.660870+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.660955+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.734775+0000 osd.7 (osd.7) 452 : cluster [DBG] 12.d scrub starts 2024-09-10T19:09:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.738507+0000 osd.7 (osd.7) 453 : cluster [DBG] 12.d scrub ok 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.790107+0000 osd.1 (osd.1) 7 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:23.803495+0000 osd.1 (osd.1) 8 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.427495+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.173466+0000 osd.5 (osd.5) 446 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.177947+0000 osd.5 (osd.5) 447 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.640706+0000 mon.a (mon.0) 3795 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: audit 2024-09-10T19:09:23.660870+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.660955+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.734775+0000 osd.7 (osd.7) 452 : cluster [DBG] 12.d scrub starts 2024-09-10T19:09:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.738507+0000 osd.7 (osd.7) 453 : cluster [DBG] 12.d scrub ok 2024-09-10T19:09:25.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.790107+0000 osd.1 (osd.1) 7 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:09:25.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:23.803495+0000 osd.1 (osd.1) 8 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:09:25.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:24 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.427495+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-09-10T19:09:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.173466+0000 osd.5 (osd.5) 446 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.177947+0000 osd.5 (osd.5) 447 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.640706+0000 mon.a (mon.0) 3795 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: audit 2024-09-10T19:09:23.660870+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? v1:172.21.15.83:0/2794446184' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.660955+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.734775+0000 osd.7 (osd.7) 452 : cluster [DBG] 12.d scrub starts 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.738507+0000 osd.7 (osd.7) 453 : cluster [DBG] 12.d scrub ok 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.790107+0000 osd.1 (osd.1) 7 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:23.803495+0000 osd.1 (osd.1) 8 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:09:25.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:24 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.427495+0000 mon.a (mon.0) 3798 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.432970+0000 mgr.y (mgr.24526) 784 : cluster [DBG] pgmap v1347: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.614032+0000 osd.3 (osd.3) 474 : cluster [DBG] 12.12 deep-scrub starts 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.618953+0000 osd.4 (osd.4) 436 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.622304+0000 osd.3 (osd.3) 475 : cluster [DBG] 12.12 deep-scrub ok 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:24.622526+0000 osd.4 (osd.4) 437 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.355805+0000 osd.0 (osd.0) 427 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:09:25.975 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.362333+0000 osd.0 (osd.0) 428 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.432970+0000 mgr.y (mgr.24526) 784 : cluster [DBG] pgmap v1347: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.614032+0000 osd.3 (osd.3) 474 : cluster [DBG] 12.12 deep-scrub starts 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.618953+0000 osd.4 (osd.4) 436 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.622304+0000 osd.3 (osd.3) 475 : cluster [DBG] 12.12 deep-scrub ok 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:24.622526+0000 osd.4 (osd.4) 437 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.355805+0000 osd.0 (osd.0) 427 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:09:25.976 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:25 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.362333+0000 osd.0 (osd.0) 428 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:09:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.432970+0000 mgr.y (mgr.24526) 784 : cluster [DBG] pgmap v1347: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:09:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.614032+0000 osd.3 (osd.3) 474 : cluster [DBG] 12.12 deep-scrub starts 2024-09-10T19:09:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.618953+0000 osd.4 (osd.4) 436 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:09:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.622304+0000 osd.3 (osd.3) 475 : cluster [DBG] 12.12 deep-scrub ok 2024-09-10T19:09:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:24.622526+0000 osd.4 (osd.4) 437 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:09:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.355805+0000 osd.0 (osd.0) 427 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:09:26.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:25 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.362333+0000 osd.0 (osd.0) 428 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/1749832959 >> v1:172.21.15.83:6790/0 conn(0x7fa9cc107e00 legacy=0x7fa9cc10a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/1749832959 shutdown_connections 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/1749832959 >> 172.21.15.83:0/1749832959 conn(0x7fa9cc0f70b0 msgr2=0x7fa9cc0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/1749832959 shutdown_connections 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/1749832959 wait complete. 2024-09-10T19:09:26.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 librados: starting msgr at 2024-09-10T19:09:26.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 librados: starting objecter 2024-09-10T19:09:26.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 Processor -- start 2024-09-10T19:09:26.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- start start 2024-09-10T19:09:26.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 librados: setting wanted keys 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 librados: calling monclient init 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9cc113590 con 0x7fa9cc101530 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9cbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa9cc101530 0x7fa9cc0fe810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52280/0 (socket says 172.21.15.83:52280) 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9cbfff640 1 -- 172.21.15.83:0/2612693355 learned_addr learned my addr 172.21.15.83:0/2612693355 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9cc123760 con 0x7fa9cc107e00 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa9cc124900 con 0x7fa9cc1042a0 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9cc1042a0 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9cc101530 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9cc107e00 2024-09-10T19:09:26.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:26.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117174466 0 0) 0x7fa9cc124900 con 0x7fa9cc1042a0 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b4003720 con 0x7fa9cc1042a0 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632665415 0 0) 0x7fa9cc113590 con 0x7fa9cc101530 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.134+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc124900 con 0x7fa9cc101530 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669728537 0 0) 0x7fa9cc123760 con 0x7fa9cc107e00 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc113590 con 0x7fa9cc107e00 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3211999753 0 0) 0x7fa9cc124900 con 0x7fa9cc101530 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9cc123760 con 0x7fa9cc101530 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9bc0019d0 con 0x7fa9cc101530 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2064569077 0 0) 0x7fa9b4003720 con 0x7fa9cc1042a0 2024-09-10T19:09:26.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9cc124900 con 0x7fa9cc1042a0 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c00018c0 con 0x7fa9cc1042a0 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3209645867 0 0) 0x7fa9cc123760 con 0x7fa9cc101530 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6790/0 conn(0x7fa9cc1042a0 legacy=0x7fa9cc0fef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.165:6789/0 conn(0x7fa9cc107e00 legacy=0x7fa9cc121030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc125aa0 con 0x7fa9cc101530 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc1227b0 con 0x7fa9cc101530 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9d253c640 10 client.25839.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc122c80 con 0x7fa9cc101530 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9d253c640 1 librados: init done 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa9bc0023a0 con 0x7fa9cc101530 2024-09-10T19:09:26.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9bc005a90 con 0x7fa9cc101530 2024-09-10T19:09:26.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.138+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9b8005030 con 0x7fa9cc101530 2024-09-10T19:09:26.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa9bc002510 con 0x7fa9cc101530 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(859..859 src has 750..859) v4 ==== 7363+0+0 (unknown 2386850207 0 0) 0x7fa9bc08a270 con 0x7fa9cc101530 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 10 client.25839.objecter ms_dispatch 0x7fa9cc000b90 osd_map(859..859 src has 750..859) v4 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 3 client.25839.objecter handle_osd_map got epochs [859,859] > 0 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 3 client.25839.objecter handle_osd_map decoding full epoch 859 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 20 client.25839.objecter dump_active .. 0 homeless 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1113 (unknown 409899127 0 4169750816) 0x7fa9bc08c660 con 0x7fa9cc101530 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b40714a0 2024-09-10T19:09:26.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b8004d40 con 0x7fa9b8002730 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b8002730 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221113641 0 0) 0x7fa9b8004d40 con 0x7fa9b8002730 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9cc123760 con 0x7fa9b8002730 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1783747053 0 0) 0x7fa9cc123760 con 0x7fa9b8002730 2024-09-10T19:09:26.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b8004d40 con 0x7fa9b8002730 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c4003050 con 0x7fa9b8002730 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1702409790 0 0) 0x7fa9b8004d40 con 0x7fa9b8002730 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b800a520 con 0x7fa9b8002730 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa9c4001df0 con 0x7fa9b8002730 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6789/0 conn(0x7fa9b8002730 legacy=0x7fa9b80055a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.142+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6789/0 conn(0x7fa9b8002730 legacy=0x7fa9b80055a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b800ad80 con 0x7fa9b800a8b0 2024-09-10T19:09:26.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b800a8b0 2024-09-10T19:09:26.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874149800 0 0) 0x7fa9b800ad80 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b8004d40 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2658890890 0 0) 0x7fa9b8004d40 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b800ad80 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c4002a70 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3723939959 0 0) 0x7fa9b800ad80 con 0x7fa9b800a8b0 2024-09-10T19:09:26.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fa9b800c000 con 0x7fa9b800a8b0 2024-09-10T19:09:26.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.0 v1:172.21.15.83:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7fa9c4003450 con 0x7fa9b800a8b0 2024-09-10T19:09:26.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6789/0 conn(0x7fa9b800a8b0 legacy=0x7fa9b8008cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6789/0 conn(0x7fa9b800a8b0 legacy=0x7fa9b8008cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.162 INFO:teuthology.orchestra.run.smithi083.stdout:mon.a: {} 2024-09-10T19:09:26.162 INFO:teuthology.orchestra.run.smithi083.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:09:26.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.154+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b80021d0 con 0x7fa9b8001d70 2024-09-10T19:09:26.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b8001d70 2024-09-10T19:09:26.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021971024 0 0) 0x7fa9b80021d0 con 0x7fa9b8001d70 2024-09-10T19:09:26.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b800ad80 con 0x7fa9b8001d70 2024-09-10T19:09:26.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3666934336 0 0) 0x7fa9b800ad80 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b80021d0 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c40039a0 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2214831644 0 0) 0x7fa9b80021d0 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b8012a50 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa9c4006380 con 0x7fa9b8001d70 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.165:6789/0 conn(0x7fa9b8001d70 legacy=0x7fa9b800b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.165:6789/0 conn(0x7fa9b8001d70 legacy=0x7fa9b800b6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.158+0000 7fa9d0ad9640 1 -- 172.21.15.83:0/2612693355 reap_dead start 2024-09-10T19:09:26.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.166+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b80130f0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b8012cb0 2024-09-10T19:09:26.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756558728 0 0) 0x7fa9b80130f0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b80021d0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2275546812 0 0) 0x7fa9b80021d0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b80130f0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c40055a0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4238953504 0 0) 0x7fa9b80130f0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.170+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.165:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fa9b8014310 con 0x7fa9b8012cb0 2024-09-10T19:09:26.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.1 v1:172.21.15.165:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7fa9c40057c0 con 0x7fa9b8012cb0 2024-09-10T19:09:26.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.165:6789/0 conn(0x7fa9b8012cb0 legacy=0x7fa9b8011200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.165:6789/0 conn(0x7fa9b8012cb0 legacy=0x7fa9b8011200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.179 INFO:teuthology.orchestra.run.smithi083.stdout:mon.b: {} 2024-09-10T19:09:26.180 INFO:teuthology.orchestra.run.smithi083.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:09:26.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b8013f50 con 0x7fa9b80133a0 2024-09-10T19:09:26.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183615475 0 0) 0x7fa9b8013f50 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b80130f0 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1321646560 0 0) 0x7fa9b80130f0 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b8013f50 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c4005d20 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2165670781 0 0) 0x7fa9b8013f50 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b801ad60 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa9c4005d40 con 0x7fa9b80133a0 2024-09-10T19:09:26.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6790/0 conn(0x7fa9b80133a0 legacy=0x7fa9b80137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.174+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6790/0 conn(0x7fa9b80133a0 legacy=0x7fa9b80137c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa9b801b2a0 con 0x7fa9b801aed0 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 10 client.25839.objecter ms_handle_connect 0x7fa9b801aed0 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 10 client.25839.objecter resend_mon_ops 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754301050 0 0) 0x7fa9b801b2a0 con 0x7fa9b801aed0 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b80130f0 con 0x7fa9b801aed0 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3588895962 0 0) 0x7fa9b80130f0 con 0x7fa9b801aed0 2024-09-10T19:09:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b801b2a0 con 0x7fa9b801aed0 2024-09-10T19:09:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9c40069c0 con 0x7fa9b801aed0 2024-09-10T19:09:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3615480353 0 0) 0x7fa9b801b2a0 con 0x7fa9b801aed0 2024-09-10T19:09:26.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.190+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 --> v1:172.21.15.83:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fa9b801c440 con 0x7fa9b801aed0 2024-09-10T19:09:26.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 <== mon.2 v1:172.21.15.83:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) v1 ==== 82+0+3 (unknown 285646418 0 2423995997) 0x7fa9c4006aa0 con 0x7fa9b801aed0 2024-09-10T19:09:26.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6790/0 conn(0x7fa9b801aed0 legacy=0x7fa9b8019510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9c8ff9640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6790/0 conn(0x7fa9b801aed0 legacy=0x7fa9b8019510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T19:09:26.200 INFO:teuthology.orchestra.run.smithi083.stdout:mon.c: {} 2024-09-10T19:09:26.200 INFO:teuthology.orchestra.run.smithi083.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 10 librados: watch_flush enter 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 10 librados: watch_flush exit 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 20 client.25839.objecter shutdown clearing up homeless session... 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 10 client.25839.objecter successfully canceled tick 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa9b40714a0 legacy=0x7fa9b4073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 >> v1:172.21.15.83:6789/0 conn(0x7fa9cc101530 legacy=0x7fa9cc0fe810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d0ad9640 1 -- 172.21.15.83:0/2612693355 reap_dead start 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 shutdown_connections 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 >> 172.21.15.83:0/2612693355 conn(0x7fa9cc0f70b0 msgr2=0x7fa9cc0fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 shutdown_connections 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 -- 172.21.15.83:0/2612693355 wait complete. 2024-09-10T19:09:26.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:26.194+0000 7fa9d253c640 1 librados: shutdown 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.117754+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.125113+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.199859+0000 osd.5 (osd.5) 448 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.203419+0000 osd.5 (osd.5) 449 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.585296+0000 osd.4 (osd.4) 438 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.589025+0000 osd.4 (osd.4) 439 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.667204+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: cluster 2024-09-10T19:09:25.670471+0000 osd.6 (osd.6) 408 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:09:26.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[18629]: audit 2024-09-10T19:09:26.146762+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? v1:172.21.15.83:0/2612693355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.117754+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.125113+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.199859+0000 osd.5 (osd.5) 448 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.203419+0000 osd.5 (osd.5) 449 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.585296+0000 osd.4 (osd.4) 438 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.589025+0000 osd.4 (osd.4) 439 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.667204+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: cluster 2024-09-10T19:09:25.670471+0000 osd.6 (osd.6) 408 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:09:26.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:26 smithi083 bash[32381]: audit 2024-09-10T19:09:26.146762+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? v1:172.21.15.83:0/2612693355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:09:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.117754+0000 osd.2 (osd.2) 347 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:09:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.125113+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:09:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.199859+0000 osd.5 (osd.5) 448 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.203419+0000 osd.5 (osd.5) 449 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.585296+0000 osd.4 (osd.4) 438 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.589025+0000 osd.4 (osd.4) 439 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.667204+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: cluster 2024-09-10T19:09:25.670471+0000 osd.6 (osd.6) 408 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:09:27.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:26 smithi165 bash[22591]: audit 2024-09-10T19:09:26.146762+0000 mon.a (mon.0) 3799 : audit [DBG] from='client.? v1:172.21.15.83:0/2612693355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:09:27.219 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[18629]: cluster 2024-09-10T19:09:26.103204+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[18629]: cluster 2024-09-10T19:09:26.107382+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[18629]: cluster 2024-09-10T19:09:26.434621+0000 mgr.y (mgr.24526) 785 : cluster [DBG] pgmap v1348: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[18629]: cluster 2024-09-10T19:09:26.695539+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[18629]: cluster 2024-09-10T19:09:26.698641+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[32381]: cluster 2024-09-10T19:09:26.103204+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[32381]: cluster 2024-09-10T19:09:26.107382+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-09-10T19:09:28.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[32381]: cluster 2024-09-10T19:09:26.434621+0000 mgr.y (mgr.24526) 785 : cluster [DBG] pgmap v1348: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T19:09:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[32381]: cluster 2024-09-10T19:09:26.695539+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:09:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:27 smithi083 bash[32381]: cluster 2024-09-10T19:09:26.698641+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:09:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:27 smithi165 bash[22591]: cluster 2024-09-10T19:09:26.103204+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-09-10T19:09:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:27 smithi165 bash[22591]: cluster 2024-09-10T19:09:26.107382+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-09-10T19:09:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:27 smithi165 bash[22591]: cluster 2024-09-10T19:09:26.434621+0000 mgr.y (mgr.24526) 785 : cluster [DBG] pgmap v1348: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-10T19:09:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:27 smithi165 bash[22591]: cluster 2024-09-10T19:09:26.695539+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:09:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:27 smithi165 bash[22591]: cluster 2024-09-10T19:09:26.698641+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:09:28.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:28.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.100394+0000 osd.2 (osd.2) 351 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.103988+0000 osd.2 (osd.2) 352 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.645639+0000 osd.3 (osd.3) 476 : cluster [DBG] 5.0 deep-scrub starts 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.654253+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.0 deep-scrub ok 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.701980+0000 osd.7 (osd.7) 456 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.706737+0000 osd.7 (osd.7) 457 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.747730+0000 osd.6 (osd.6) 409 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:09:28.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:28 smithi165 bash[22591]: cluster 2024-09-10T19:09:27.751455+0000 osd.6 (osd.6) 410 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.100394+0000 osd.2 (osd.2) 351 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.103988+0000 osd.2 (osd.2) 352 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.645639+0000 osd.3 (osd.3) 476 : cluster [DBG] 5.0 deep-scrub starts 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.654253+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.0 deep-scrub ok 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.701980+0000 osd.7 (osd.7) 456 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.706737+0000 osd.7 (osd.7) 457 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.747730+0000 osd.6 (osd.6) 409 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:09:29.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[18629]: cluster 2024-09-10T19:09:27.751455+0000 osd.6 (osd.6) 410 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.100394+0000 osd.2 (osd.2) 351 : cluster [DBG] 6.f deep-scrub starts 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.103988+0000 osd.2 (osd.2) 352 : cluster [DBG] 6.f deep-scrub ok 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.645639+0000 osd.3 (osd.3) 476 : cluster [DBG] 5.0 deep-scrub starts 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.654253+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.0 deep-scrub ok 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.701980+0000 osd.7 (osd.7) 456 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.706737+0000 osd.7 (osd.7) 457 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.747730+0000 osd.6 (osd.6) 409 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:09:29.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:28 smithi083 bash[32381]: cluster 2024-09-10T19:09:27.751455+0000 osd.6 (osd.6) 410 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.435424+0000 mgr.y (mgr.24526) 786 : cluster [DBG] pgmap v1349: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: audit 2024-09-10T19:09:28.514718+0000 mgr.y (mgr.24526) 787 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.648012+0000 osd.4 (osd.4) 440 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.651464+0000 osd.4 (osd.4) 441 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.733098+0000 osd.7 (osd.7) 458 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.736355+0000 osd.7 (osd.7) 459 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:09:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.784577+0000 osd.6 (osd.6) 411 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: cluster 2024-09-10T19:09:28.788538+0000 osd.6 (osd.6) 412 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: audit 2024-09-10T19:09:29.586103+0000 mon.a (mon.0) 3800 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[18629]: audit 2024-09-10T19:09:29.588339+0000 mon.c (mon.2) 895 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.435424+0000 mgr.y (mgr.24526) 786 : cluster [DBG] pgmap v1349: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: audit 2024-09-10T19:09:28.514718+0000 mgr.y (mgr.24526) 787 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.648012+0000 osd.4 (osd.4) 440 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.651464+0000 osd.4 (osd.4) 441 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.733098+0000 osd.7 (osd.7) 458 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.736355+0000 osd.7 (osd.7) 459 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.784577+0000 osd.6 (osd.6) 411 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: cluster 2024-09-10T19:09:28.788538+0000 osd.6 (osd.6) 412 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: audit 2024-09-10T19:09:29.586103+0000 mon.a (mon.0) 3800 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:29 smithi083 bash[32381]: audit 2024-09-10T19:09:29.588339+0000 mon.c (mon.2) 895 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.435424+0000 mgr.y (mgr.24526) 786 : cluster [DBG] pgmap v1349: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: audit 2024-09-10T19:09:28.514718+0000 mgr.y (mgr.24526) 787 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.648012+0000 osd.4 (osd.4) 440 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.651464+0000 osd.4 (osd.4) 441 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.733098+0000 osd.7 (osd.7) 458 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.736355+0000 osd.7 (osd.7) 459 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.784577+0000 osd.6 (osd.6) 411 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: cluster 2024-09-10T19:09:28.788538+0000 osd.6 (osd.6) 412 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: audit 2024-09-10T19:09:29.586103+0000 mon.a (mon.0) 3800 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:09:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:29 smithi165 bash[22591]: audit 2024-09-10T19:09:29.588339+0000 mon.c (mon.2) 895 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.089354+0000 osd.2 (osd.2) 353 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.093383+0000 osd.2 (osd.2) 354 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.713506+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.717345+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.785795+0000 osd.6 (osd.6) 413 : cluster [DBG] 4.a scrub starts 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[18629]: cluster 2024-09-10T19:09:29.789187+0000 osd.6 (osd.6) 414 : cluster [DBG] 4.a scrub ok 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.089354+0000 osd.2 (osd.2) 353 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.093383+0000 osd.2 (osd.2) 354 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:09:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.713506+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:09:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.717345+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:09:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.785795+0000 osd.6 (osd.6) 413 : cluster [DBG] 4.a scrub starts 2024-09-10T19:09:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:30 smithi083 bash[32381]: cluster 2024-09-10T19:09:29.789187+0000 osd.6 (osd.6) 414 : cluster [DBG] 4.a scrub ok 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.089354+0000 osd.2 (osd.2) 353 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.093383+0000 osd.2 (osd.2) 354 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.713506+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.717345+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.785795+0000 osd.6 (osd.6) 413 : cluster [DBG] 4.a scrub starts 2024-09-10T19:09:31.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:30 smithi165 bash[22591]: cluster 2024-09-10T19:09:29.789187+0000 osd.6 (osd.6) 414 : cluster [DBG] 4.a scrub ok 2024-09-10T19:09:31.681 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:31.988 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:30.124837+0000 osd.2 (osd.2) 355 : cluster [DBG] 2.c scrub starts 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:30.128208+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.c scrub ok 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:30.436204+0000 mgr.y (mgr.24526) 788 : cluster [DBG] pgmap v1350: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.266563+0000 osd.0 (osd.0) 429 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.270436+0000 osd.0 (osd.0) 430 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.553363+0000 osd.3 (osd.3) 478 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:09:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.559067+0000 osd.3 (osd.3) 479 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:30.124837+0000 osd.2 (osd.2) 355 : cluster [DBG] 2.c scrub starts 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:30.128208+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.c scrub ok 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:30.436204+0000 mgr.y (mgr.24526) 788 : cluster [DBG] pgmap v1350: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.266563+0000 osd.0 (osd.0) 429 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.270436+0000 osd.0 (osd.0) 430 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.553363+0000 osd.3 (osd.3) 478 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:09:32.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:31 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.559067+0000 osd.3 (osd.3) 479 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:09:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:30.124837+0000 osd.2 (osd.2) 355 : cluster [DBG] 2.c scrub starts 2024-09-10T19:09:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:30.128208+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.c scrub ok 2024-09-10T19:09:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:30.436204+0000 mgr.y (mgr.24526) 788 : cluster [DBG] pgmap v1350: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.6 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-10T19:09:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.266563+0000 osd.0 (osd.0) 429 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:09:32.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.270436+0000 osd.0 (osd.0) 430 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:09:32.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.553363+0000 osd.3 (osd.3) 478 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:09:32.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:31 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.559067+0000 osd.3 (osd.3) 479 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:32] "GET /metrics HTTP/1.1" 200 224079 "" "Prometheus/2.43.0" 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.085371+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.088795+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.763941+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.767602+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.811873+0000 osd.6 (osd.6) 415 : cluster [DBG] 3.a scrub starts 2024-09-10T19:09:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[18629]: cluster 2024-09-10T19:09:31.817516+0000 osd.6 (osd.6) 416 : cluster [DBG] 3.a scrub ok 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.085371+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.088795+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.763941+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.767602+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.811873+0000 osd.6 (osd.6) 415 : cluster [DBG] 3.a scrub starts 2024-09-10T19:09:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:32 smithi083 bash[32381]: cluster 2024-09-10T19:09:31.817516+0000 osd.6 (osd.6) 416 : cluster [DBG] 3.a scrub ok 2024-09-10T19:09:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.085371+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:09:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.088795+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:09:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.763941+0000 osd.1 (osd.1) 11 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:09:33.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.767602+0000 osd.1 (osd.1) 12 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:09:33.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.811873+0000 osd.6 (osd.6) 415 : cluster [DBG] 3.a scrub starts 2024-09-10T19:09:33.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:32 smithi165 bash[22591]: cluster 2024-09-10T19:09:31.817516+0000 osd.6 (osd.6) 416 : cluster [DBG] 3.a scrub ok 2024-09-10T19:09:33.790 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[32381]: cluster 2024-09-10T19:09:32.438103+0000 mgr.y (mgr.24526) 789 : cluster [DBG] pgmap v1351: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-10T19:09:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[32381]: cluster 2024-09-10T19:09:33.267223+0000 osd.0 (osd.0) 431 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:09:33.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[32381]: cluster 2024-09-10T19:09:33.271113+0000 osd.0 (osd.0) 432 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:09:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[18629]: cluster 2024-09-10T19:09:32.438103+0000 mgr.y (mgr.24526) 789 : cluster [DBG] pgmap v1351: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-10T19:09:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[18629]: cluster 2024-09-10T19:09:33.267223+0000 osd.0 (osd.0) 431 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:09:33.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:33 smithi083 bash[18629]: cluster 2024-09-10T19:09:33.271113+0000 osd.0 (osd.0) 432 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:09:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:33 smithi165 bash[22591]: cluster 2024-09-10T19:09:32.438103+0000 mgr.y (mgr.24526) 789 : cluster [DBG] pgmap v1351: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s rd, 0 B/s wr, 82 op/s 2024-09-10T19:09:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:33 smithi165 bash[22591]: cluster 2024-09-10T19:09:33.267223+0000 osd.0 (osd.0) 431 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:09:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:33 smithi165 bash[22591]: cluster 2024-09-10T19:09:33.271113+0000 osd.0 (osd.0) 432 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:09:35.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.298585+0000 osd.0 (osd.0) 433 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:09:35.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.303148+0000 osd.0 (osd.0) 434 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:09:35.686 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.318502+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.321860+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.438921+0000 mgr.y (mgr.24526) 790 : cluster [DBG] pgmap v1352: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.751241+0000 osd.6 (osd.6) 417 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.755564+0000 osd.6 (osd.6) 418 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.298585+0000 osd.0 (osd.0) 433 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.303148+0000 osd.0 (osd.0) 434 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.318502+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.321860+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.438921+0000 mgr.y (mgr.24526) 790 : cluster [DBG] pgmap v1352: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.751241+0000 osd.6 (osd.6) 417 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:09:35.687 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:35 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.755564+0000 osd.6 (osd.6) 418 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:09:35.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.846+0000 7f1b381b6640 1 -- 172.21.15.83:0/3010806447 >> v1:172.21.15.83:6790/0 conn(0x7f1b3010f880 legacy=0x7f1b30111c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:35.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.846+0000 7f1b381b6640 1 -- 172.21.15.83:0/3010806447 shutdown_connections 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.846+0000 7f1b381b6640 1 -- 172.21.15.83:0/3010806447 >> 172.21.15.83:0/3010806447 conn(0x7f1b300f7060 msgr2=0x7f1b300f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.846+0000 7f1b381b6640 1 -- 172.21.15.83:0/3010806447 shutdown_connections 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.846+0000 7f1b381b6640 1 -- 172.21.15.83:0/3010806447 wait complete. 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 librados: starting msgr at 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 librados: starting objecter 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 Processor -- start 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- start start 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 librados: setting wanted keys 2024-09-10T19:09:35.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 librados: calling monclient init 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b301229e0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b30123be0 con 0x7f1b3010f880 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b35f52640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f1b300fb3a0 0x7f1b3010bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46118/0 (socket says 172.21.15.83:46118) 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b30124de0 con 0x7f1b3011ec50 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b36753640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f1b3011ec50 0x7f1b301210e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49476/0 (socket says 172.21.15.83:49476) 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b35f52640 1 -- 172.21.15.83:0/4024048763 learned_addr learned my addr 172.21.15.83:0/4024048763 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter ms_handle_connect 0x7f1b300fb3a0 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter ms_handle_connect 0x7f1b3011ec50 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:35.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter ms_handle_connect 0x7f1b3010f880 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019885596 0 0) 0x7f1b301229e0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c0038e0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410381806 0 0) 0x7f1b30124de0 con 0x7f1b3011ec50 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b301229e0 con 0x7f1b3011ec50 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335775635 0 0) 0x7f1b30123be0 con 0x7f1b3010f880 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30124de0 con 0x7f1b3010f880 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354130328 0 0) 0x7f1b1c0038e0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b30123be0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1b2c001a70 con 0x7f1b300fb3a0 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 949201659 0 0) 0x7f1b301229e0 con 0x7f1b3011ec50 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b1c0038e0 con 0x7f1b3011ec50 2024-09-10T19:09:35.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905237082 0 0) 0x7f1b30124de0 con 0x7f1b3010f880 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b301229e0 con 0x7f1b3010f880 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1b28002410 con 0x7f1b3011ec50 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1b20001840 con 0x7f1b3010f880 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4045985608 0 0) 0x7f1b30123be0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 >> v1:172.21.15.83:6790/0 conn(0x7f1b3011ec50 legacy=0x7f1b301210e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 >> v1:172.21.15.165:6789/0 conn(0x7f1b3010f880 legacy=0x7f1b3010c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:35.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30125fe0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b30124fd0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 10 client.25851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b30125420 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b381b6640 1 librados: init done 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1b2c002470 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.850+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1b2c005a70 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1b2c01be90 con 0x7f1b300fb3a0 2024-09-10T19:09:35.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b14005010 con 0x7f1b300fb3a0 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(859..859 src has 750..859) v4 ==== 7363+0+0 (unknown 2386850207 0 0) 0x7f1b2c08a3b0 con 0x7f1b300fb3a0 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 10 client.25851.objecter ms_dispatch 0x7f1b30000b90 osd_map(859..859 src has 750..859) v4 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 3 client.25851.objecter handle_osd_map got epochs [859,859] > 0 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 3 client.25851.objecter handle_osd_map decoding full epoch 859 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 20 client.25851.objecter dump_active .. 0 homeless 2024-09-10T19:09:35.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.854+0000 7f1b267fc640 10 client.25851.objecter ms_handle_connect 0x7f1b1c0718b0 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.298585+0000 osd.0 (osd.0) 433 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.303148+0000 osd.0 (osd.0) 434 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.318502+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.321860+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.438921+0000 mgr.y (mgr.24526) 790 : cluster [DBG] pgmap v1352: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.751241+0000 osd.6 (osd.6) 417 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:09:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:35 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.755564+0000 osd.6 (osd.6) 418 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:09:35.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:35.862+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1b2c056bf0 con 0x7f1b300fb3a0 2024-09-10T19:09:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.046+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f1b140052c0 con 0x7f1b300fb3a0 2024-09-10T19:09:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.442+0000 7f1b267fc640 1 -- 172.21.15.83:0/4024048763 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v860) v1 ==== 113+0+0 (unknown 2723452967 0 0) 0x7f1b2c056590 con 0x7f1b300fb3a0 2024-09-10T19:09:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 10 librados: watch_flush enter 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 10 librados: watch_flush exit 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 20 client.25851.objecter shutdown clearing up homeless session... 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 10 client.25851.objecter successfully canceled tick 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1b1c0718b0 legacy=0x7f1b1c073d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 >> v1:172.21.15.83:6789/0 conn(0x7f1b300fb3a0 legacy=0x7f1b3010bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 shutdown_connections 2024-09-10T19:09:36.449 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.446+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 >> 172.21.15.83:0/4024048763 conn(0x7f1b300f7060 msgr2=0x7f1b300f85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.450+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 shutdown_connections 2024-09-10T19:09:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.450+0000 7f1b381b6640 1 -- 172.21.15.83:0/4024048763 wait complete. 2024-09-10T19:09:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:36.450+0000 7f1b381b6640 1 librados: shutdown 2024-09-10T19:09:36.717 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.601317+0000 osd.3 (osd.3) 480 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[18629]: cluster 2024-09-10T19:09:34.605454+0000 osd.3 (osd.3) 481 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[18629]: cluster 2024-09-10T19:09:35.312041+0000 osd.5 (osd.5) 452 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[18629]: cluster 2024-09-10T19:09:35.317570+0000 osd.5 (osd.5) 453 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[18629]: audit 2024-09-10T19:09:36.049067+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.601317+0000 osd.3 (osd.3) 480 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[32381]: cluster 2024-09-10T19:09:34.605454+0000 osd.3 (osd.3) 481 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[32381]: cluster 2024-09-10T19:09:35.312041+0000 osd.5 (osd.5) 452 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[32381]: cluster 2024-09-10T19:09:35.317570+0000 osd.5 (osd.5) 453 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:09:36.718 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:36 smithi083 bash[32381]: audit 2024-09-10T19:09:36.049067+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:36 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.601317+0000 osd.3 (osd.3) 480 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:09:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:36 smithi165 bash[22591]: cluster 2024-09-10T19:09:34.605454+0000 osd.3 (osd.3) 481 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:09:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:36 smithi165 bash[22591]: cluster 2024-09-10T19:09:35.312041+0000 osd.5 (osd.5) 452 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:09:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:36 smithi165 bash[22591]: cluster 2024-09-10T19:09:35.317570+0000 osd.5 (osd.5) 453 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:09:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:36 smithi165 bash[22591]: audit 2024-09-10T19:09:36.049067+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:37.322 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[32381]: cluster 2024-09-10T19:09:36.274338+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.d scrub starts 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[32381]: cluster 2024-09-10T19:09:36.278062+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.d scrub ok 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[32381]: cluster 2024-09-10T19:09:36.439683+0000 mgr.y (mgr.24526) 791 : cluster [DBG] pgmap v1354: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[32381]: audit 2024-09-10T19:09:36.440857+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[32381]: cluster 2024-09-10T19:09:36.440955+0000 mon.a (mon.0) 3803 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[18629]: cluster 2024-09-10T19:09:36.274338+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.d scrub starts 2024-09-10T19:09:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[18629]: cluster 2024-09-10T19:09:36.278062+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.d scrub ok 2024-09-10T19:09:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[18629]: cluster 2024-09-10T19:09:36.439683+0000 mgr.y (mgr.24526) 791 : cluster [DBG] pgmap v1354: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[18629]: audit 2024-09-10T19:09:36.440857+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:37 smithi083 bash[18629]: cluster 2024-09-10T19:09:36.440955+0000 mon.a (mon.0) 3803 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-09-10T19:09:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:37 smithi165 bash[22591]: cluster 2024-09-10T19:09:36.274338+0000 osd.5 (osd.5) 454 : cluster [DBG] 6.d scrub starts 2024-09-10T19:09:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:37 smithi165 bash[22591]: cluster 2024-09-10T19:09:36.278062+0000 osd.5 (osd.5) 455 : cluster [DBG] 6.d scrub ok 2024-09-10T19:09:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:37 smithi165 bash[22591]: cluster 2024-09-10T19:09:36.439683+0000 mgr.y (mgr.24526) 791 : cluster [DBG] pgmap v1354: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:37 smithi165 bash[22591]: audit 2024-09-10T19:09:36.440857+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? v1:172.21.15.83:0/4024048763' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:37 smithi165 bash[22591]: cluster 2024-09-10T19:09:36.440955+0000 mon.a (mon.0) 3803 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.311550+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.315543+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.645280+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.648888+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.711935+0000 osd.6 (osd.6) 419 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[32381]: cluster 2024-09-10T19:09:37.716110+0000 osd.6 (osd.6) 420 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.311550+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.315543+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:09:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.645280+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:09:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.648888+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:09:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.711935+0000 osd.6 (osd.6) 419 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:09:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:38 smithi083 bash[18629]: cluster 2024-09-10T19:09:37.716110+0000 osd.6 (osd.6) 420 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:09:38.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.311550+0000 osd.0 (osd.0) 435 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:09:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.315543+0000 osd.0 (osd.0) 436 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:09:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.645280+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:09:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.648888+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:09:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.711935+0000 osd.6 (osd.6) 419 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:09:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:38 smithi165 bash[22591]: cluster 2024-09-10T19:09:37.716110+0000 osd.6 (osd.6) 420 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.440465+0000 mgr.y (mgr.24526) 792 : cluster [DBG] pgmap v1355: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: audit 2024-09-10T19:09:38.526079+0000 mgr.y (mgr.24526) 793 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.676060+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.680545+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.695078+0000 osd.7 (osd.7) 462 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.699039+0000 osd.7 (osd.7) 463 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.755032+0000 osd.6 (osd.6) 421 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:38.758548+0000 osd.6 (osd.6) 422 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:09:39.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[32381]: cluster 2024-09-10T19:09:39.432505+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.440465+0000 mgr.y (mgr.24526) 792 : cluster [DBG] pgmap v1355: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: audit 2024-09-10T19:09:38.526079+0000 mgr.y (mgr.24526) 793 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.676060+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.680545+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.695078+0000 osd.7 (osd.7) 462 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.699039+0000 osd.7 (osd.7) 463 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.755032+0000 osd.6 (osd.6) 421 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:38.758548+0000 osd.6 (osd.6) 422 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:09:39.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:39 smithi083 bash[18629]: cluster 2024-09-10T19:09:39.432505+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-09-10T19:09:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.440465+0000 mgr.y (mgr.24526) 792 : cluster [DBG] pgmap v1355: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 64 op/s 2024-09-10T19:09:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: audit 2024-09-10T19:09:38.526079+0000 mgr.y (mgr.24526) 793 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.676060+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.680545+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.695078+0000 osd.7 (osd.7) 462 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.699039+0000 osd.7 (osd.7) 463 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.755032+0000 osd.6 (osd.6) 421 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:38.758548+0000 osd.6 (osd.6) 422 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:09:39.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:39 smithi165 bash[22591]: cluster 2024-09-10T19:09:39.432505+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-09-10T19:09:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:40 smithi083 bash[18629]: cluster 2024-09-10T19:09:39.273766+0000 osd.5 (osd.5) 456 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:09:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:40 smithi083 bash[18629]: cluster 2024-09-10T19:09:39.277832+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:09:40.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:40 smithi083 bash[32381]: cluster 2024-09-10T19:09:39.273766+0000 osd.5 (osd.5) 456 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:09:40.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:40 smithi083 bash[32381]: cluster 2024-09-10T19:09:39.277832+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:09:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:40 smithi165 bash[22591]: cluster 2024-09-10T19:09:39.273766+0000 osd.5 (osd.5) 456 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:09:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:40 smithi165 bash[22591]: cluster 2024-09-10T19:09:39.277832+0000 osd.5 (osd.5) 457 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[18629]: cluster 2024-09-10T19:09:40.270182+0000 osd.5 (osd.5) 458 : cluster [DBG] 6.a scrub starts 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[18629]: cluster 2024-09-10T19:09:40.275717+0000 osd.5 (osd.5) 459 : cluster [DBG] 6.a scrub ok 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[18629]: cluster 2024-09-10T19:09:40.441185+0000 mgr.y (mgr.24526) 794 : cluster [DBG] pgmap v1357: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[18629]: cluster 2024-09-10T19:09:40.665460+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[18629]: cluster 2024-09-10T19:09:40.669840+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[32381]: cluster 2024-09-10T19:09:40.270182+0000 osd.5 (osd.5) 458 : cluster [DBG] 6.a scrub starts 2024-09-10T19:09:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[32381]: cluster 2024-09-10T19:09:40.275717+0000 osd.5 (osd.5) 459 : cluster [DBG] 6.a scrub ok 2024-09-10T19:09:41.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[32381]: cluster 2024-09-10T19:09:40.441185+0000 mgr.y (mgr.24526) 794 : cluster [DBG] pgmap v1357: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:41.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[32381]: cluster 2024-09-10T19:09:40.665460+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:09:41.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:41 smithi083 bash[32381]: cluster 2024-09-10T19:09:40.669840+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:09:41.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[22591]: cluster 2024-09-10T19:09:40.270182+0000 osd.5 (osd.5) 458 : cluster [DBG] 6.a scrub starts 2024-09-10T19:09:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[22591]: cluster 2024-09-10T19:09:40.275717+0000 osd.5 (osd.5) 459 : cluster [DBG] 6.a scrub ok 2024-09-10T19:09:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[22591]: cluster 2024-09-10T19:09:40.441185+0000 mgr.y (mgr.24526) 794 : cluster [DBG] pgmap v1357: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[22591]: cluster 2024-09-10T19:09:40.665460+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:09:41.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:41 smithi165 bash[22591]: cluster 2024-09-10T19:09:40.669840+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:09:42.094 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:43.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:42] "GET /metrics HTTP/1.1" 200 224079 "" "Prometheus/2.43.0" 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.307018+0000 osd.5 (osd.5) 460 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.310763+0000 osd.5 (osd.5) 461 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.443094+0000 mgr.y (mgr.24526) 795 : cluster [DBG] pgmap v1358: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.650922+0000 osd.1 (osd.1) 17 : cluster [DBG] 12.1b deep-scrub starts 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.654413+0000 osd.1 (osd.1) 18 : cluster [DBG] 12.1b deep-scrub ok 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.727178+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.b scrub starts 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.730500+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.b scrub ok 2024-09-10T19:09:43.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.760106+0000 osd.6 (osd.6) 423 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[18629]: cluster 2024-09-10T19:09:42.763572+0000 osd.6 (osd.6) 424 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.307018+0000 osd.5 (osd.5) 460 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.310763+0000 osd.5 (osd.5) 461 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.443094+0000 mgr.y (mgr.24526) 795 : cluster [DBG] pgmap v1358: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.650922+0000 osd.1 (osd.1) 17 : cluster [DBG] 12.1b deep-scrub starts 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.654413+0000 osd.1 (osd.1) 18 : cluster [DBG] 12.1b deep-scrub ok 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.727178+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.b scrub starts 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.730500+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.b scrub ok 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.760106+0000 osd.6 (osd.6) 423 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:09:43.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:43 smithi083 bash[32381]: cluster 2024-09-10T19:09:42.763572+0000 osd.6 (osd.6) 424 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:09:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.307018+0000 osd.5 (osd.5) 460 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:09:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.310763+0000 osd.5 (osd.5) 461 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:09:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.443094+0000 mgr.y (mgr.24526) 795 : cluster [DBG] pgmap v1358: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.650922+0000 osd.1 (osd.1) 17 : cluster [DBG] 12.1b deep-scrub starts 2024-09-10T19:09:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.654413+0000 osd.1 (osd.1) 18 : cluster [DBG] 12.1b deep-scrub ok 2024-09-10T19:09:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.727178+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.b scrub starts 2024-09-10T19:09:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.730500+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.b scrub ok 2024-09-10T19:09:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.760106+0000 osd.6 (osd.6) 423 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:09:43.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:43 smithi165 bash[22591]: cluster 2024-09-10T19:09:42.763572+0000 osd.6 (osd.6) 424 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:09:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:44 smithi083 bash[18629]: cluster 2024-09-10T19:09:43.226443+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:09:44.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:44 smithi083 bash[18629]: cluster 2024-09-10T19:09:43.229952+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:09:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:44 smithi083 bash[32381]: cluster 2024-09-10T19:09:43.226443+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:09:44.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:44 smithi083 bash[32381]: cluster 2024-09-10T19:09:43.229952+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:09:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:44 smithi165 bash[22591]: cluster 2024-09-10T19:09:43.226443+0000 osd.0 (osd.0) 437 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:09:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:44 smithi165 bash[22591]: cluster 2024-09-10T19:09:43.229952+0000 osd.0 (osd.0) 438 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[18629]: cluster 2024-09-10T19:09:44.443944+0000 mgr.y (mgr.24526) 796 : cluster [DBG] pgmap v1359: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[18629]: audit 2024-09-10T19:09:44.578669+0000 mon.c (mon.2) 896 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[18629]: cluster 2024-09-10T19:09:44.825805+0000 osd.6 (osd.6) 425 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[18629]: cluster 2024-09-10T19:09:44.829567+0000 osd.6 (osd.6) 426 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[32381]: cluster 2024-09-10T19:09:44.443944+0000 mgr.y (mgr.24526) 796 : cluster [DBG] pgmap v1359: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[32381]: audit 2024-09-10T19:09:44.578669+0000 mon.c (mon.2) 896 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:45.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[32381]: cluster 2024-09-10T19:09:44.825805+0000 osd.6 (osd.6) 425 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:09:45.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:45 smithi083 bash[32381]: cluster 2024-09-10T19:09:44.829567+0000 osd.6 (osd.6) 426 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- 172.21.15.83:0/2774562794 >> v1:172.21.15.83:6789/0 conn(0x7feca4104810 legacy=0x7feca4104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- 172.21.15.83:0/2774562794 shutdown_connections 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- 172.21.15.83:0/2774562794 >> 172.21.15.83:0/2774562794 conn(0x7feca40fa370 msgr2=0x7feca40fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- 172.21.15.83:0/2774562794 shutdown_connections 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- 172.21.15.83:0/2774562794 wait complete. 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 librados: starting msgr at 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 librados: starting objecter 2024-09-10T19:09:45.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 Processor -- start 2024-09-10T19:09:45.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- start start 2024-09-10T19:09:45.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 librados: setting wanted keys 2024-09-10T19:09:45.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 librados: calling monclient init 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feca411e230 con 0x7feca41075a0 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feca411f430 con 0x7feca4104810 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca88b8640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feca4120630 con 0x7feca410b130 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca1d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7feca41075a0 0x7feca4100ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46046/0 (socket says 172.21.15.83:46046) 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca1d9b640 1 -- 172.21.15.83:0/1238328775 learned_addr learned my addr 172.21.15.83:0/1238328775 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7feca2d9d640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7feca410b130 0x7feca411c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:48364/0 (socket says 172.21.15.83:48364) 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter ms_handle_connect 0x7feca410b130 2024-09-10T19:09:45.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter ms_handle_connect 0x7feca41075a0 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter ms_handle_connect 0x7feca4104810 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.845+0000 7fec92ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279392322 0 0) 0x7feca4120630 con 0x7feca410b130 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec8c0038e0 con 0x7feca410b130 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142469545 0 0) 0x7feca411e230 con 0x7feca41075a0 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4120630 con 0x7feca41075a0 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546084722 0 0) 0x7feca411f430 con 0x7feca4104810 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca411e230 con 0x7feca4104810 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283482622 0 0) 0x7fec8c0038e0 con 0x7feca410b130 2024-09-10T19:09:45.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca411f430 con 0x7feca410b130 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec9c001a80 con 0x7feca410b130 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470001231 0 0) 0x7feca4120630 con 0x7feca41075a0 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec8c0038e0 con 0x7feca41075a0 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec94001c80 con 0x7feca41075a0 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717236093 0 0) 0x7feca411e230 con 0x7feca4104810 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca4120630 con 0x7feca4104810 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec98001f90 con 0x7feca4104810 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2720812667 0 0) 0x7feca411f430 con 0x7feca410b130 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 >> v1:172.21.15.165:6789/0 conn(0x7feca4104810 legacy=0x7feca41007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:45.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 >> v1:172.21.15.83:6789/0 conn(0x7feca41075a0 legacy=0x7feca4100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4121830 con 0x7feca410b130 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca411e420 con 0x7feca410b130 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7feca88b8640 10 client.49943.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca411e870 con 0x7feca410b130 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7feca88b8640 1 librados: init done 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fec9c000fe0 con 0x7feca410b130 2024-09-10T19:09:45.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec9c0058b0 con 0x7feca410b130 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec84005010 con 0x7feca410b130 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.849+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fec9c0050b0 con 0x7feca410b130 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(861..861 src has 750..861) v4 ==== 7363+0+0 (unknown 414125000 0 0) 0x7fec9c08a1f0 con 0x7feca410b130 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 10 client.49943.objecter ms_dispatch 0x7feca4000b90 osd_map(861..861 src has 750..861) v4 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 3 client.49943.objecter handle_osd_map got epochs [861,861] > 0 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 3 client.49943.objecter handle_osd_map decoding full epoch 861 2024-09-10T19:09:45.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 20 client.49943.objecter dump_active .. 0 homeless 2024-09-10T19:09:45.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.853+0000 7fec92ffd640 10 client.49943.objecter ms_handle_connect 0x7fec8c071840 2024-09-10T19:09:45.860 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:45.857+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fec9c056a90 con 0x7feca410b130 2024-09-10T19:09:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:45 smithi165 bash[22591]: cluster 2024-09-10T19:09:44.443944+0000 mgr.y (mgr.24526) 796 : cluster [DBG] pgmap v1359: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:45 smithi165 bash[22591]: audit 2024-09-10T19:09:44.578669+0000 mon.c (mon.2) 896 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:45 smithi165 bash[22591]: cluster 2024-09-10T19:09:44.825805+0000 osd.6 (osd.6) 425 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:09:45.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:45 smithi165 bash[22591]: cluster 2024-09-10T19:09:44.829567+0000 osd.6 (osd.6) 426 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:09:46.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.049+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fec84005840 con 0x7feca410b130 2024-09-10T19:09:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.517+0000 7fec92ffd640 1 -- 172.21.15.83:0/1238328775 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v862) v1 ==== 133+0+0 (unknown 116915636 0 0) 0x7fec9c056430 con 0x7feca410b130 2024-09-10T19:09:46.519 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 10 librados: watch_flush enter 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 10 librados: watch_flush exit 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 20 client.49943.objecter shutdown clearing up homeless session... 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 10 client.49943.objecter successfully canceled tick 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 >> v1:172.21.15.83:6800/3071793735 conn(0x7fec8c071840 legacy=0x7fec8c073cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 >> v1:172.21.15.83:6790/0 conn(0x7feca410b130 legacy=0x7feca411c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 shutdown_connections 2024-09-10T19:09:46.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 >> 172.21.15.83:0/1238328775 conn(0x7feca40fa370 msgr2=0x7feca40fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:46.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 shutdown_connections 2024-09-10T19:09:46.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 -- 172.21.15.83:0/1238328775 wait complete. 2024-09-10T19:09:46.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:46.521+0000 7feca88b8640 1 librados: shutdown 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: cluster 2024-09-10T19:09:45.287899+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: cluster 2024-09-10T19:09:45.293842+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: cluster 2024-09-10T19:09:45.807297+0000 osd.6 (osd.6) 427 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: cluster 2024-09-10T19:09:45.810861+0000 osd.6 (osd.6) 428 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: audit 2024-09-10T19:09:46.054288+0000 mon.c (mon.2) 897 : audit [INF] from='client.? v1:172.21.15.83:0/1238328775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:46.759 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[18629]: audit 2024-09-10T19:09:46.054818+0000 mon.a (mon.0) 3805 : audit [INF] from='client.49943 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: cluster 2024-09-10T19:09:45.287899+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: cluster 2024-09-10T19:09:45.293842+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: cluster 2024-09-10T19:09:45.807297+0000 osd.6 (osd.6) 427 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: cluster 2024-09-10T19:09:45.810861+0000 osd.6 (osd.6) 428 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: audit 2024-09-10T19:09:46.054288+0000 mon.c (mon.2) 897 : audit [INF] from='client.? v1:172.21.15.83:0/1238328775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:46.760 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:46 smithi083 bash[32381]: audit 2024-09-10T19:09:46.054818+0000 mon.a (mon.0) 3805 : audit [INF] from='client.49943 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: cluster 2024-09-10T19:09:45.287899+0000 osd.5 (osd.5) 462 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:09:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: cluster 2024-09-10T19:09:45.293842+0000 osd.5 (osd.5) 463 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:09:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: cluster 2024-09-10T19:09:45.807297+0000 osd.6 (osd.6) 427 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:09:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: cluster 2024-09-10T19:09:45.810861+0000 osd.6 (osd.6) 428 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:09:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: audit 2024-09-10T19:09:46.054288+0000 mon.c (mon.2) 897 : audit [INF] from='client.? v1:172.21.15.83:0/1238328775' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:46.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:46 smithi165 bash[22591]: audit 2024-09-10T19:09:46.054818+0000 mon.a (mon.0) 3805 : audit [INF] from='client.49943 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:09:47.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.305230+0000 osd.0 (osd.0) 439 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.308743+0000 osd.0 (osd.0) 440 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.324573+0000 osd.5 (osd.5) 464 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.330397+0000 osd.5 (osd.5) 465 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.445875+0000 mgr.y (mgr.24526) 797 : cluster [DBG] pgmap v1360: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: audit 2024-09-10T19:09:46.518096+0000 mon.a (mon.0) 3806 : audit [INF] from='client.49943 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.518252+0000 mon.a (mon.0) 3807 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.657170+0000 osd.1 (osd.1) 19 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:09:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[18629]: cluster 2024-09-10T19:09:46.666024+0000 osd.1 (osd.1) 20 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.305230+0000 osd.0 (osd.0) 439 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.308743+0000 osd.0 (osd.0) 440 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.324573+0000 osd.5 (osd.5) 464 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.330397+0000 osd.5 (osd.5) 465 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.445875+0000 mgr.y (mgr.24526) 797 : cluster [DBG] pgmap v1360: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: audit 2024-09-10T19:09:46.518096+0000 mon.a (mon.0) 3806 : audit [INF] from='client.49943 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.518252+0000 mon.a (mon.0) 3807 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.657170+0000 osd.1 (osd.1) 19 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:09:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:47 smithi083 bash[32381]: cluster 2024-09-10T19:09:46.666024+0000 osd.1 (osd.1) 20 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:09:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.305230+0000 osd.0 (osd.0) 439 : cluster [DBG] 2.13 deep-scrub starts 2024-09-10T19:09:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.308743+0000 osd.0 (osd.0) 440 : cluster [DBG] 2.13 deep-scrub ok 2024-09-10T19:09:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.324573+0000 osd.5 (osd.5) 464 : cluster [DBG] 1.8 deep-scrub starts 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.330397+0000 osd.5 (osd.5) 465 : cluster [DBG] 1.8 deep-scrub ok 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.445875+0000 mgr.y (mgr.24526) 797 : cluster [DBG] pgmap v1360: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: audit 2024-09-10T19:09:46.518096+0000 mon.a (mon.0) 3806 : audit [INF] from='client.49943 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.518252+0000 mon.a (mon.0) 3807 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.657170+0000 osd.1 (osd.1) 19 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:09:47.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:47 smithi165 bash[22591]: cluster 2024-09-10T19:09:46.666024+0000 osd.1 (osd.1) 20 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:09:48.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:48 smithi083 bash[18629]: cluster 2024-09-10T19:09:47.275049+0000 osd.0 (osd.0) 441 : cluster [DBG] 4.b scrub starts 2024-09-10T19:09:48.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:48 smithi083 bash[18629]: cluster 2024-09-10T19:09:47.278880+0000 osd.0 (osd.0) 442 : cluster [DBG] 4.b scrub ok 2024-09-10T19:09:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:48 smithi083 bash[32381]: cluster 2024-09-10T19:09:47.275049+0000 osd.0 (osd.0) 441 : cluster [DBG] 4.b scrub starts 2024-09-10T19:09:48.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:48 smithi083 bash[32381]: cluster 2024-09-10T19:09:47.278880+0000 osd.0 (osd.0) 442 : cluster [DBG] 4.b scrub ok 2024-09-10T19:09:48.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:48 smithi165 bash[22591]: cluster 2024-09-10T19:09:47.275049+0000 osd.0 (osd.0) 441 : cluster [DBG] 4.b scrub starts 2024-09-10T19:09:48.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:48 smithi165 bash[22591]: cluster 2024-09-10T19:09:47.278880+0000 osd.0 (osd.0) 442 : cluster [DBG] 4.b scrub ok 2024-09-10T19:09:48.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:47.577405+0000 osd.3 (osd.3) 482 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:09:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:47.581422+0000 osd.3 (osd.3) 483 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:09:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.372246+0000 osd.5 (osd.5) 466 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:09:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.375748+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:09:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.446696+0000 mgr.y (mgr.24526) 798 : cluster [DBG] pgmap v1362: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: audit 2024-09-10T19:09:48.536834+0000 mgr.y (mgr.24526) 799 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.666254+0000 osd.1 (osd.1) 21 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.669787+0000 osd.1 (osd.1) 22 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.700704+0000 osd.7 (osd.7) 466 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.705186+0000 osd.7 (osd.7) 467 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[18629]: cluster 2024-09-10T19:09:49.435613+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:47.577405+0000 osd.3 (osd.3) 482 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:47.581422+0000 osd.3 (osd.3) 483 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.372246+0000 osd.5 (osd.5) 466 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:09:49.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.375748+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.446696+0000 mgr.y (mgr.24526) 798 : cluster [DBG] pgmap v1362: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: audit 2024-09-10T19:09:48.536834+0000 mgr.y (mgr.24526) 799 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.666254+0000 osd.1 (osd.1) 21 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.669787+0000 osd.1 (osd.1) 22 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.700704+0000 osd.7 (osd.7) 466 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.705186+0000 osd.7 (osd.7) 467 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:09:49.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:49 smithi083 bash[32381]: cluster 2024-09-10T19:09:49.435613+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-09-10T19:09:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:47.577405+0000 osd.3 (osd.3) 482 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:09:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:47.581422+0000 osd.3 (osd.3) 483 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:09:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.372246+0000 osd.5 (osd.5) 466 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:09:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.375748+0000 osd.5 (osd.5) 467 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.446696+0000 mgr.y (mgr.24526) 798 : cluster [DBG] pgmap v1362: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: audit 2024-09-10T19:09:48.536834+0000 mgr.y (mgr.24526) 799 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.666254+0000 osd.1 (osd.1) 21 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.669787+0000 osd.1 (osd.1) 22 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.700704+0000 osd.7 (osd.7) 466 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.705186+0000 osd.7 (osd.7) 467 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:09:49.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:49 smithi165 bash[22591]: cluster 2024-09-10T19:09:49.435613+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-09-10T19:09:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.612796+0000 osd.3 (osd.3) 484 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:09:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:48.617057+0000 osd.3 (osd.3) 485 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:09:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:49.369492+0000 osd.5 (osd.5) 468 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:09:50.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:49.372687+0000 osd.5 (osd.5) 469 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:09:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:49.700132+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-09-10T19:09:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[18629]: cluster 2024-09-10T19:09:49.703663+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-09-10T19:09:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.612796+0000 osd.3 (osd.3) 484 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:09:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:48.617057+0000 osd.3 (osd.3) 485 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:09:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:49.369492+0000 osd.5 (osd.5) 468 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:09:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:49.372687+0000 osd.5 (osd.5) 469 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:09:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:49.700132+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-09-10T19:09:50.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:50 smithi083 bash[32381]: cluster 2024-09-10T19:09:49.703663+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-09-10T19:09:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.612796+0000 osd.3 (osd.3) 484 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:09:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:48.617057+0000 osd.3 (osd.3) 485 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:09:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:49.369492+0000 osd.5 (osd.5) 468 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:09:50.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:49.372687+0000 osd.5 (osd.5) 469 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:09:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:49.700132+0000 osd.1 (osd.1) 23 : cluster [DBG] 2.d scrub starts 2024-09-10T19:09:50.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:50 smithi165 bash[22591]: cluster 2024-09-10T19:09:49.703663+0000 osd.1 (osd.1) 24 : cluster [DBG] 2.d scrub ok 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.447450+0000 mgr.y (mgr.24526) 800 : cluster [DBG] pgmap v1364: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.711491+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.e scrub starts 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.712984+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.f scrub starts 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.715089+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.e scrub ok 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.718167+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.f scrub ok 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.907446+0000 osd.6 (osd.6) 429 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:09:51.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[32381]: cluster 2024-09-10T19:09:50.910968+0000 osd.6 (osd.6) 430 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.447450+0000 mgr.y (mgr.24526) 800 : cluster [DBG] pgmap v1364: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.711491+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.e scrub starts 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.712984+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.f scrub starts 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.715089+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.e scrub ok 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.718167+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.f scrub ok 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.907446+0000 osd.6 (osd.6) 429 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:09:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:51 smithi083 bash[18629]: cluster 2024-09-10T19:09:50.910968+0000 osd.6 (osd.6) 430 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:09:51.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.447450+0000 mgr.y (mgr.24526) 800 : cluster [DBG] pgmap v1364: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.711491+0000 osd.1 (osd.1) 25 : cluster [DBG] 6.e scrub starts 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.712984+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.f scrub starts 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.715089+0000 osd.1 (osd.1) 26 : cluster [DBG] 6.e scrub ok 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.718167+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.f scrub ok 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.907446+0000 osd.6 (osd.6) 429 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:09:51.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:51 smithi165 bash[22591]: cluster 2024-09-10T19:09:50.910968+0000 osd.6 (osd.6) 430 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:09:52.118 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.343109+0000 osd.0 (osd.0) 443 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.346527+0000 osd.0 (osd.0) 444 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.682295+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.686842+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.910432+0000 osd.6 (osd.6) 431 : cluster [DBG] 12.f scrub starts 2024-09-10T19:09:52.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:52 smithi165 bash[22591]: cluster 2024-09-10T19:09:51.913765+0000 osd.6 (osd.6) 432 : cluster [DBG] 12.f scrub ok 2024-09-10T19:09:53.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:09:52] "GET /metrics HTTP/1.1" 200 224116 "" "Prometheus/2.43.0" 2024-09-10T19:09:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.343109+0000 osd.0 (osd.0) 443 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:09:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.346527+0000 osd.0 (osd.0) 444 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:09:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.682295+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:09:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.686842+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.910432+0000 osd.6 (osd.6) 431 : cluster [DBG] 12.f scrub starts 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[18629]: cluster 2024-09-10T19:09:51.913765+0000 osd.6 (osd.6) 432 : cluster [DBG] 12.f scrub ok 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.343109+0000 osd.0 (osd.0) 443 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.346527+0000 osd.0 (osd.0) 444 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.682295+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.686842+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.910432+0000 osd.6 (osd.6) 431 : cluster [DBG] 12.f scrub starts 2024-09-10T19:09:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:52 smithi083 bash[32381]: cluster 2024-09-10T19:09:51.913765+0000 osd.6 (osd.6) 432 : cluster [DBG] 12.f scrub ok 2024-09-10T19:09:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:53 smithi165 bash[22591]: cluster 2024-09-10T19:09:52.277601+0000 osd.5 (osd.5) 470 : cluster [DBG] 3.c scrub starts 2024-09-10T19:09:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:53 smithi165 bash[22591]: cluster 2024-09-10T19:09:52.282820+0000 osd.5 (osd.5) 471 : cluster [DBG] 3.c scrub ok 2024-09-10T19:09:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:53 smithi165 bash[22591]: cluster 2024-09-10T19:09:52.449308+0000 mgr.y (mgr.24526) 801 : cluster [DBG] pgmap v1365: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:53 smithi165 bash[22591]: cluster 2024-09-10T19:09:52.727490+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:09:53.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:53 smithi165 bash[22591]: cluster 2024-09-10T19:09:52.731258+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[18629]: cluster 2024-09-10T19:09:52.277601+0000 osd.5 (osd.5) 470 : cluster [DBG] 3.c scrub starts 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[18629]: cluster 2024-09-10T19:09:52.282820+0000 osd.5 (osd.5) 471 : cluster [DBG] 3.c scrub ok 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[18629]: cluster 2024-09-10T19:09:52.449308+0000 mgr.y (mgr.24526) 801 : cluster [DBG] pgmap v1365: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[18629]: cluster 2024-09-10T19:09:52.727490+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[18629]: cluster 2024-09-10T19:09:52.731258+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[32381]: cluster 2024-09-10T19:09:52.277601+0000 osd.5 (osd.5) 470 : cluster [DBG] 3.c scrub starts 2024-09-10T19:09:53.956 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[32381]: cluster 2024-09-10T19:09:52.282820+0000 osd.5 (osd.5) 471 : cluster [DBG] 3.c scrub ok 2024-09-10T19:09:53.957 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[32381]: cluster 2024-09-10T19:09:52.449308+0000 mgr.y (mgr.24526) 801 : cluster [DBG] pgmap v1365: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:09:53.957 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[32381]: cluster 2024-09-10T19:09:52.727490+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:09:53.957 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:53 smithi083 bash[32381]: cluster 2024-09-10T19:09:52.731258+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:09:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:54 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.687029+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:09:54.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:54 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.691242+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:09:54.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:54 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.924198+0000 osd.6 (osd.6) 433 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:09:54.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:54 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.927563+0000 osd.6 (osd.6) 434 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:09:55.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.687029+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.691242+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.924198+0000 osd.6 (osd.6) 433 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.927563+0000 osd.6 (osd.6) 434 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.687029+0000 osd.1 (osd.1) 29 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.691242+0000 osd.1 (osd.1) 30 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.924198+0000 osd.6 (osd.6) 433 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:09:55.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:54 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.927563+0000 osd.6 (osd.6) 434 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:09:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.687262+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:09:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:53.692552+0000 osd.3 (osd.3) 487 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:09:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.376751+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:09:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.382394+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:09:55.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.450113+0000 mgr.y (mgr.24526) 802 : cluster [DBG] pgmap v1366: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.792741+0000 osd.7 (osd.7) 472 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:09:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.796932+0000 osd.7 (osd.7) 473 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:09:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.913660+0000 osd.6 (osd.6) 435 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:09:55.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:55 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.917100+0000 osd.6 (osd.6) 436 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- 172.21.15.83:0/2151915282 >> v1:172.21.15.83:6789/0 conn(0x7f52d8104270 legacy=0x7f52d8106650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- 172.21.15.83:0/2151915282 shutdown_connections 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- 172.21.15.83:0/2151915282 >> 172.21.15.83:0/2151915282 conn(0x7f52d80f7060 msgr2=0x7f52d80f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- 172.21.15.83:0/2151915282 shutdown_connections 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- 172.21.15.83:0/2151915282 wait complete. 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 librados: starting msgr at 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 librados: starting objecter 2024-09-10T19:09:55.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 Processor -- start 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- start start 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 librados: setting wanted keys 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 librados: calling monclient init 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d8107a80 con 0x7f52d8101500 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d8107c30 con 0x7f52d8107dd0 2024-09-10T19:09:55.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52dc927640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d81206d0 con 0x7f52d8104270 2024-09-10T19:09:55.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52d659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f52d8101500 0x7f52d81071c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52926/0 (socket says 172.21.15.83:52926) 2024-09-10T19:09:55.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52d659c640 1 -- 172.21.15.83:0/3656710538 learned_addr learned my addr 172.21.15.83:0/3656710538 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:09:55.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52c6ffd640 10 client.?.objecter ms_handle_connect 0x7f52d8101500 2024-09-10T19:09:55.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987537653 0 0) 0x7f52d8107a80 con 0x7f52d8101500 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.877+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c00034e0 con 0x7f52d8101500 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 10 client.?.objecter ms_handle_connect 0x7f52d8107dd0 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 10 client.?.objecter ms_handle_connect 0x7f52d8104270 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1068124558 0 0) 0x7f52c00034e0 con 0x7f52d8101500 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52d8107a80 con 0x7f52d8101500 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824104167 0 0) 0x7f52d810df60 con 0x7f52d8107dd0 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c00034e0 con 0x7f52d8107dd0 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f52c8001980 con 0x7f52d8101500 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016415925 0 0) 0x7f52d81206d0 con 0x7f52d8104270 2024-09-10T19:09:55.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d810df60 con 0x7f52d8104270 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053267475 0 0) 0x7f52c00034e0 con 0x7f52d8107dd0 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52d81206d0 con 0x7f52d8107dd0 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f52d00019e0 con 0x7f52d8107dd0 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2173385847 0 0) 0x7f52d810df60 con 0x7f52d8104270 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c00034e0 con 0x7f52d8104270 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f52cc001f90 con 0x7f52d8104270 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3154598101 0 0) 0x7f52d8107a80 con 0x7f52d8101500 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 >> v1:172.21.15.83:6790/0 conn(0x7f52d8104270 legacy=0x7f52d8119800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 >> v1:172.21.15.165:6789/0 conn(0x7f52d8107dd0 legacy=0x7f52d811cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d81218d0 con 0x7f52d8101500 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d81208c0 con 0x7f52d8101500 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52dc927640 10 client.25863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d8120d10 con 0x7f52d8101500 2024-09-10T19:09:55.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52dc927640 1 librados: init done 2024-09-10T19:09:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f52c8002d40 con 0x7f52d8101500 2024-09-10T19:09:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f52c8005a60 con 0x7f52d8101500 2024-09-10T19:09:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.881+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b8005010 con 0x7f52d8101500 2024-09-10T19:09:55.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f52c8001e60 con 0x7f52d8101500 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(863..863 src has 750..863) v4 ==== 7363+0+0 (unknown 1794515461 0 0) 0x7f52c808a1c0 con 0x7f52d8101500 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 10 client.25863.objecter ms_dispatch 0x7f52d8000b90 osd_map(863..863 src has 750..863) v4 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 3 client.25863.objecter handle_osd_map got epochs [863,863] > 0 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 3 client.25863.objecter handle_osd_map decoding full epoch 863 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 20 client.25863.objecter dump_active .. 0 homeless 2024-09-10T19:09:55.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.885+0000 7f52c6ffd640 10 client.25863.objecter ms_handle_connect 0x7f52c0071810 2024-09-10T19:09:55.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:55.893+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f52c8056a60 con 0x7f52d8101500 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.687262+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:53.692552+0000 osd.3 (osd.3) 487 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.376751+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.382394+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.450113+0000 mgr.y (mgr.24526) 802 : cluster [DBG] pgmap v1366: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.792741+0000 osd.7 (osd.7) 472 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.796932+0000 osd.7 (osd.7) 473 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.913660+0000 osd.6 (osd.6) 435 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:09:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.917100+0000 osd.6 (osd.6) 436 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.687262+0000 osd.3 (osd.3) 486 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:53.692552+0000 osd.3 (osd.3) 487 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.376751+0000 osd.0 (osd.0) 445 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.382394+0000 osd.0 (osd.0) 446 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.450113+0000 mgr.y (mgr.24526) 802 : cluster [DBG] pgmap v1366: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.792741+0000 osd.7 (osd.7) 472 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.796932+0000 osd.7 (osd.7) 473 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.913660+0000 osd.6 (osd.6) 435 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:09:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:55 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.917100+0000 osd.6 (osd.6) 436 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:09:56.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.073+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f52b80052c0 con 0x7f52d8101500 2024-09-10T19:09:56.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.573+0000 7f52c6ffd640 1 -- 172.21.15.83:0/3656710538 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v864) v1 ==== 113+0+0 (unknown 692733661 0 0) 0x7f52c8056400 con 0x7f52d8101500 2024-09-10T19:09:56.578 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T19:09:56.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 10 librados: watch_flush enter 2024-09-10T19:09:56.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 10 librados: watch_flush exit 2024-09-10T19:09:56.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 20 client.25863.objecter shutdown clearing up homeless session... 2024-09-10T19:09:56.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 10 client.25863.objecter successfully canceled tick 2024-09-10T19:09:56.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 >> v1:172.21.15.83:6800/3071793735 conn(0x7f52c0071810 legacy=0x7f52c0073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:56.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 >> v1:172.21.15.83:6789/0 conn(0x7f52d8101500 legacy=0x7f52d81071c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:09:56.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 shutdown_connections 2024-09-10T19:09:56.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.577+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 >> 172.21.15.83:0/3656710538 conn(0x7f52d80f7060 msgr2=0x7f52d80fa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:09:56.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.581+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 shutdown_connections 2024-09-10T19:09:56.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.581+0000 7f52dc927640 1 -- 172.21.15.83:0/3656710538 wait complete. 2024-09-10T19:09:56.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:09:56.581+0000 7f52dc927640 1 librados: shutdown 2024-09-10T19:09:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.672707+0000 osd.3 (osd.3) 488 : cluster [DBG] 6.13 deep-scrub starts 2024-09-10T19:09:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:54.676350+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.13 deep-scrub ok 2024-09-10T19:09:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.293353+0000 osd.5 (osd.5) 472 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:09:56.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.297003+0000 osd.5 (osd.5) 473 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:09:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.375795+0000 osd.0 (osd.0) 447 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:09:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.379138+0000 osd.0 (osd.0) 448 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:09:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.946422+0000 osd.6 (osd.6) 437 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:09:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: cluster 2024-09-10T19:09:55.949702+0000 osd.6 (osd.6) 438 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:09:56.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:56 smithi165 bash[22591]: audit 2024-09-10T19:09:56.077974+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.672707+0000 osd.3 (osd.3) 488 : cluster [DBG] 6.13 deep-scrub starts 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:54.676350+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.13 deep-scrub ok 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.293353+0000 osd.5 (osd.5) 472 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.297003+0000 osd.5 (osd.5) 473 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.375795+0000 osd.0 (osd.0) 447 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.379138+0000 osd.0 (osd.0) 448 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.946422+0000 osd.6 (osd.6) 437 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: cluster 2024-09-10T19:09:55.949702+0000 osd.6 (osd.6) 438 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:09:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[18629]: audit 2024-09-10T19:09:56.077974+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.672707+0000 osd.3 (osd.3) 488 : cluster [DBG] 6.13 deep-scrub starts 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:54.676350+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.13 deep-scrub ok 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.293353+0000 osd.5 (osd.5) 472 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.297003+0000 osd.5 (osd.5) 473 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.375795+0000 osd.0 (osd.0) 447 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.379138+0000 osd.0 (osd.0) 448 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.946422+0000 osd.6 (osd.6) 437 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: cluster 2024-09-10T19:09:55.949702+0000 osd.6 (osd.6) 438 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:09:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:56 smithi083 bash[32381]: audit 2024-09-10T19:09:56.077974+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:09:57.522 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.398384+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.401964+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.451953+0000 mgr.y (mgr.24526) 803 : cluster [DBG] pgmap v1367: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: audit 2024-09-10T19:09:56.574575+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.575102+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-09-10T19:09:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.599513+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.b scrub starts 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.605011+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.b scrub ok 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.681764+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.685517+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.930575+0000 osd.6 (osd.6) 439 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:56.934157+0000 osd.6 (osd.6) 440 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:57.557540+0000 osd.3 (osd.3) 492 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[18629]: cluster 2024-09-10T19:09:57.560895+0000 osd.3 (osd.3) 493 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.398384+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.401964+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:09:57.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.451953+0000 mgr.y (mgr.24526) 803 : cluster [DBG] pgmap v1367: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: audit 2024-09-10T19:09:56.574575+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.575102+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.599513+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.b scrub starts 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.605011+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.b scrub ok 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.681764+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.685517+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.930575+0000 osd.6 (osd.6) 439 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:56.934157+0000 osd.6 (osd.6) 440 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:57.557540+0000 osd.3 (osd.3) 492 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:09:57.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:57 smithi083 bash[32381]: cluster 2024-09-10T19:09:57.560895+0000 osd.3 (osd.3) 493 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:09:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.398384+0000 osd.0 (osd.0) 449 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:09:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.401964+0000 osd.0 (osd.0) 450 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:09:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.451953+0000 mgr.y (mgr.24526) 803 : cluster [DBG] pgmap v1367: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:09:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: audit 2024-09-10T19:09:56.574575+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.83:0/3656710538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.575102+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.599513+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.b scrub starts 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.605011+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.b scrub ok 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.681764+0000 osd.1 (osd.1) 31 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.685517+0000 osd.1 (osd.1) 32 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.930575+0000 osd.6 (osd.6) 439 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:56.934157+0000 osd.6 (osd.6) 440 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:57.557540+0000 osd.3 (osd.3) 492 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:09:57.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:57 smithi165 bash[22591]: cluster 2024-09-10T19:09:57.560895+0000 osd.3 (osd.3) 493 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:09:58.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:09:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:09:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:58 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.536912+0000 osd.3 (osd.3) 494 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:09:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:58 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.540359+0000 osd.3 (osd.3) 495 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:09:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:58 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.536912+0000 osd.3 (osd.3) 494 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:09:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:58 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.540359+0000 osd.3 (osd.3) 495 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:09:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:58 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.536912+0000 osd.3 (osd.3) 494 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:09:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:58 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.540359+0000 osd.3 (osd.3) 495 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:09:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.398795+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:09:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.399237+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:09:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.452828+0000 mgr.y (mgr.24526) 804 : cluster [DBG] pgmap v1369: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: audit 2024-09-10T19:09:58.544671+0000 mgr.y (mgr.24526) 805 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.604646+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.714403+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:58.722108+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:59.446461+0000 osd.0 (osd.0) 451 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: cluster 2024-09-10T19:09:59.449060+0000 osd.0 (osd.0) 452 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:09:59.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:09:59 smithi165 bash[22591]: audit 2024-09-10T19:09:59.579164+0000 mon.c (mon.2) 898 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.398795+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.399237+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.452828+0000 mgr.y (mgr.24526) 804 : cluster [DBG] pgmap v1369: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:10:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: audit 2024-09-10T19:09:58.544671+0000 mgr.y (mgr.24526) 805 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.604646+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.714403+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:58.722108+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:59.446461+0000 osd.0 (osd.0) 451 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: cluster 2024-09-10T19:09:59.449060+0000 osd.0 (osd.0) 452 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[18629]: audit 2024-09-10T19:09:59.579164+0000 mon.c (mon.2) 898 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.398795+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.399237+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.452828+0000 mgr.y (mgr.24526) 804 : cluster [DBG] pgmap v1369: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: audit 2024-09-10T19:09:58.544671+0000 mgr.y (mgr.24526) 805 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.604646+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.714403+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.0 deep-scrub starts 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:58.722108+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.0 deep-scrub ok 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:59.446461+0000 osd.0 (osd.0) 451 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: cluster 2024-09-10T19:09:59.449060+0000 osd.0 (osd.0) 452 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:10:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:09:59 smithi083 bash[32381]: audit 2024-09-10T19:09:59.579164+0000 mon.c (mon.2) 898 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.000187+0000 mon.a (mon.0) 3813 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.000234+0000 mon.a (mon.0) 3814 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.000249+0000 mon.a (mon.0) 3815 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.000262+0000 mon.a (mon.0) 3816 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.000187+0000 mon.a (mon.0) 3813 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.000234+0000 mon.a (mon.0) 3814 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.000249+0000 mon.a (mon.0) 3815 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.000262+0000 mon.a (mon.0) 3816 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.000187+0000 mon.a (mon.0) 3813 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.000234+0000 mon.a (mon.0) 3814 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.000249+0000 mon.a (mon.0) 3815 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.000262+0000 mon.a (mon.0) 3816 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:01.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:01 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.453709+0000 mgr.y (mgr.24526) 806 : cluster [DBG] pgmap v1371: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:01 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.688908+0000 osd.1 (osd.1) 33 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:10:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:01 smithi165 bash[22591]: cluster 2024-09-10T19:10:00.692499+0000 osd.1 (osd.1) 34 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.453709+0000 mgr.y (mgr.24526) 806 : cluster [DBG] pgmap v1371: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.688908+0000 osd.1 (osd.1) 33 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[18629]: cluster 2024-09-10T19:10:00.692499+0000 osd.1 (osd.1) 34 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.453709+0000 mgr.y (mgr.24526) 806 : cluster [DBG] pgmap v1371: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.688908+0000 osd.1 (osd.1) 33 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:10:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:01 smithi083 bash[32381]: cluster 2024-09-10T19:10:00.692499+0000 osd.1 (osd.1) 34 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:10:02.317 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:03.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:02] "GET /metrics HTTP/1.1" 200 224111 "" "Prometheus/2.43.0" 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:02.455686+0000 mgr.y (mgr.24526) 807 : cluster [DBG] pgmap v1372: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:02.682513+0000 osd.7 (osd.7) 478 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:02.686170+0000 osd.7 (osd.7) 479 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:02.737421+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:02.740961+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:10:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:03.117784+0000 osd.4 (osd.4) 442 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:03.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:03 smithi165 bash[22591]: cluster 2024-09-10T19:10:03.118130+0000 osd.4 (osd.4) 443 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:03.924 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:02.455686+0000 mgr.y (mgr.24526) 807 : cluster [DBG] pgmap v1372: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:03.924 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:02.682513+0000 osd.7 (osd.7) 478 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:02.686170+0000 osd.7 (osd.7) 479 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:02.737421+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:02.740961+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:03.117784+0000 osd.4 (osd.4) 442 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[18629]: cluster 2024-09-10T19:10:03.118130+0000 osd.4 (osd.4) 443 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:02.455686+0000 mgr.y (mgr.24526) 807 : cluster [DBG] pgmap v1372: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:02.682513+0000 osd.7 (osd.7) 478 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:02.686170+0000 osd.7 (osd.7) 479 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:02.737421+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:10:03.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:02.740961+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:10:03.926 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:03.117784+0000 osd.4 (osd.4) 442 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:03.926 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:03 smithi083 bash[32381]: cluster 2024-09-10T19:10:03.118130+0000 osd.4 (osd.4) 443 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:04.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[18629]: cluster 2024-09-10T19:10:03.370618+0000 osd.5 (osd.5) 474 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:10:04.937 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[18629]: cluster 2024-09-10T19:10:03.373973+0000 osd.5 (osd.5) 475 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[18629]: cluster 2024-09-10T19:10:03.623823+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.441855+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[32381]: cluster 2024-09-10T19:10:03.370618+0000 osd.5 (osd.5) 474 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[32381]: cluster 2024-09-10T19:10:03.373973+0000 osd.5 (osd.5) 475 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[32381]: cluster 2024-09-10T19:10:03.623823+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-09-10T19:10:04.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:04 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.441855+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-09-10T19:10:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:04 smithi165 bash[22591]: cluster 2024-09-10T19:10:03.370618+0000 osd.5 (osd.5) 474 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:10:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:04 smithi165 bash[22591]: cluster 2024-09-10T19:10:03.373973+0000 osd.5 (osd.5) 475 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:10:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:04 smithi165 bash[22591]: cluster 2024-09-10T19:10:03.623823+0000 mon.a (mon.0) 3817 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-09-10T19:10:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:04 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.441855+0000 mon.a (mon.0) 3818 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.456401+0000 mgr.y (mgr.24526) 808 : cluster [DBG] pgmap v1375: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.593106+0000 osd.3 (osd.3) 496 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.603260+0000 osd.3 (osd.3) 497 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.679524+0000 osd.7 (osd.7) 480 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.683791+0000 osd.7 (osd.7) 481 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.712337+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.717287+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:10:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.968774+0000 osd.3 (osd.3) 498 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[18629]: cluster 2024-09-10T19:10:04.969302+0000 osd.3 (osd.3) 499 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.456401+0000 mgr.y (mgr.24526) 808 : cluster [DBG] pgmap v1375: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.593106+0000 osd.3 (osd.3) 496 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.603260+0000 osd.3 (osd.3) 497 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.679524+0000 osd.7 (osd.7) 480 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.683791+0000 osd.7 (osd.7) 481 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.712337+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.717287+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.968774+0000 osd.3 (osd.3) 498 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:06.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:05 smithi083 bash[32381]: cluster 2024-09-10T19:10:04.969302+0000 osd.3 (osd.3) 499 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.456401+0000 mgr.y (mgr.24526) 808 : cluster [DBG] pgmap v1375: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:10:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.593106+0000 osd.3 (osd.3) 496 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:10:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.603260+0000 osd.3 (osd.3) 497 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:10:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.679524+0000 osd.7 (osd.7) 480 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:10:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.683791+0000 osd.7 (osd.7) 481 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:10:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.712337+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:10:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.717287+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:10:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.968774+0000 osd.3 (osd.3) 498 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:05 smithi165 bash[22591]: cluster 2024-09-10T19:10:04.969302+0000 osd.3 (osd.3) 499 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:06.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/263645693 >> v1:172.21.15.83:6789/0 conn(0x7fb844108ed0 legacy=0x7fb84410b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:06.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/263645693 shutdown_connections 2024-09-10T19:10:06.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/263645693 >> 172.21.15.83:0/263645693 conn(0x7fb8440f70b0 msgr2=0x7fb8440f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:06.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/263645693 shutdown_connections 2024-09-10T19:10:06.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/263645693 wait complete. 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 librados: starting msgr at 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 librados: starting objecter 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 Processor -- start 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- start start 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 librados: setting wanted keys 2024-09-10T19:10:06.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 librados: calling monclient init 2024-09-10T19:10:06.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb84411e710 con 0x7fb8441151d0 2024-09-10T19:10:06.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb84411f8d0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb844120ad0 con 0x7fb844108ed0 2024-09-10T19:10:06.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb848fd4640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb8441151d0 0x7fb84411ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:50666/0 (socket says 172.21.15.83:50666) 2024-09-10T19:10:06.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb848fd4640 1 -- 172.21.15.83:0/2729378453 learned_addr learned my addr 172.21.15.83:0/2729378453 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:06.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter ms_handle_connect 0x7fb8441151d0 2024-09-10T19:10:06.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter ms_handle_connect 0x7fb844108ed0 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter ms_handle_connect 0x7fb8440fd5a0 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348159198 0 0) 0x7fb84411e710 con 0x7fb8441151d0 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8300038e0 con 0x7fb8441151d0 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344849554 0 0) 0x7fb844120ad0 con 0x7fb844108ed0 2024-09-10T19:10:06.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84411e710 con 0x7fb844108ed0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042457857 0 0) 0x7fb84411f8d0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844120ad0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2178760524 0 0) 0x7fb84411e710 con 0x7fb844108ed0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb84411f8d0 con 0x7fb844108ed0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb82c001880 con 0x7fb844108ed0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846149943 0 0) 0x7fb8300038e0 con 0x7fb8441151d0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb84411e710 con 0x7fb8441151d0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb83c002450 con 0x7fb8441151d0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289953785 0 0) 0x7fb844120ad0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8300038e0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb8340018c0 con 0x7fb8440fd5a0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1283882808 0 0) 0x7fb84411f8d0 con 0x7fb844108ed0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 >> v1:172.21.15.165:6789/0 conn(0x7fb8440fd5a0 legacy=0x7fb8441148a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 >> v1:172.21.15.83:6789/0 conn(0x7fb8441151d0 legacy=0x7fb84411ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb844121cd0 con 0x7fb844108ed0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84411fac0 con 0x7fb844108ed0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 10 client.49955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84411ff10 con 0x7fb844108ed0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.333+0000 7fb84aa37640 1 librados: init done 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb82c002280 con 0x7fb844108ed0 2024-09-10T19:10:06.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb82c0031a0 con 0x7fb844108ed0 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824005010 con 0x7fb844108ed0 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fb82c001e20 con 0x7fb844108ed0 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(868..868 src has 750..868) v4 ==== 7363+0+0 (unknown 3474423023 0 0) 0x7fb82c0878a0 con 0x7fb844108ed0 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 10 client.49955.objecter ms_dispatch 0x7fb844000b90 osd_map(868..868 src has 750..868) v4 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 3 client.49955.objecter handle_osd_map got epochs [868,868] > 0 2024-09-10T19:10:06.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.337+0000 7fb840ff9640 3 client.49955.objecter handle_osd_map decoding full epoch 868 2024-09-10T19:10:06.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.341+0000 7fb840ff9640 20 client.49955.objecter dump_active .. 0 homeless 2024-09-10T19:10:06.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.341+0000 7fb840ff9640 10 client.49955.objecter ms_handle_connect 0x7fb8300717b0 2024-09-10T19:10:06.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.345+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fb82c054140 con 0x7fb844108ed0 2024-09-10T19:10:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.553+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fb824005840 con 0x7fb844108ed0 2024-09-10T19:10:06.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.637+0000 7fb840ff9640 1 -- 172.21.15.83:0/2729378453 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v869) v1 ==== 133+0+0 (unknown 2100502275 0 0) 0x7fb82c053ae0 con 0x7fb844108ed0 2024-09-10T19:10:06.643 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-10T19:10:06.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 10 librados: watch_flush enter 2024-09-10T19:10:06.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 10 librados: watch_flush exit 2024-09-10T19:10:06.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 20 client.49955.objecter shutdown clearing up homeless session... 2024-09-10T19:10:06.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 10 client.49955.objecter successfully canceled tick 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 >> v1:172.21.15.83:6800/3071793735 conn(0x7fb8300717b0 legacy=0x7fb830073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 >> v1:172.21.15.83:6790/0 conn(0x7fb844108ed0 legacy=0x7fb8441196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 shutdown_connections 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 >> 172.21.15.83:0/2729378453 conn(0x7fb8440f70b0 msgr2=0x7fb844064ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 shutdown_connections 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 -- 172.21.15.83:0/2729378453 wait complete. 2024-09-10T19:10:06.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:06.645+0000 7fb84aa37640 1 librados: shutdown 2024-09-10T19:10:06.896 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[18629]: cluster 2024-09-10T19:10:05.636089+0000 mon.a (mon.0) 3819 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[18629]: audit 2024-09-10T19:10:06.460335+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[18629]: audit 2024-09-10T19:10:06.556692+0000 mon.c (mon.2) 900 : audit [INF] from='client.? v1:172.21.15.83:0/2729378453' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[18629]: audit 2024-09-10T19:10:06.557314+0000 mon.a (mon.0) 3820 : audit [INF] from='client.49955 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[32381]: cluster 2024-09-10T19:10:05.636089+0000 mon.a (mon.0) 3819 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[32381]: audit 2024-09-10T19:10:06.460335+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[32381]: audit 2024-09-10T19:10:06.556692+0000 mon.c (mon.2) 900 : audit [INF] from='client.? v1:172.21.15.83:0/2729378453' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:06.897 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:06 smithi083 bash[32381]: audit 2024-09-10T19:10:06.557314+0000 mon.a (mon.0) 3820 : audit [INF] from='client.49955 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:06 smithi165 bash[22591]: cluster 2024-09-10T19:10:05.636089+0000 mon.a (mon.0) 3819 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-09-10T19:10:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:06 smithi165 bash[22591]: audit 2024-09-10T19:10:06.460335+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:07.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:06 smithi165 bash[22591]: audit 2024-09-10T19:10:06.556692+0000 mon.c (mon.2) 900 : audit [INF] from='client.? v1:172.21.15.83:0/2729378453' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:07.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:06 smithi165 bash[22591]: audit 2024-09-10T19:10:06.557314+0000 mon.a (mon.0) 3820 : audit [INF] from='client.49955 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:10:07.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T19:10:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.351186+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.a scrub starts 2024-09-10T19:10:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.354688+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.a scrub ok 2024-09-10T19:10:07.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.458061+0000 mgr.y (mgr.24526) 809 : cluster [DBG] pgmap v1377: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:06.640662+0000 mon.a (mon.0) 3821 : audit [INF] from='client.49955 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.640848+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.744051+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.a scrub starts 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:06.751010+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.a scrub ok 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:06.762756+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:06.772936+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:07.205610+0000 mon.c (mon.2) 901 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:07.206946+0000 mon.c (mon.2) 902 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: audit 2024-09-10T19:10:07.217951+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:07.445197+0000 osd.0 (osd.0) 453 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:10:07.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[18629]: cluster 2024-09-10T19:10:07.448643+0000 osd.0 (osd.0) 454 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:10:07.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.351186+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.a scrub starts 2024-09-10T19:10:07.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.354688+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.a scrub ok 2024-09-10T19:10:07.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.458061+0000 mgr.y (mgr.24526) 809 : cluster [DBG] pgmap v1377: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:07.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:06.640662+0000 mon.a (mon.0) 3821 : audit [INF] from='client.49955 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:10:07.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.640848+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.744051+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.a scrub starts 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:06.751010+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.a scrub ok 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:06.762756+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:06.772936+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:07.205610+0000 mon.c (mon.2) 901 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:07.795 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:07.206946+0000 mon.c (mon.2) 902 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:07.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: audit 2024-09-10T19:10:07.217951+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:07.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:07.445197+0000 osd.0 (osd.0) 453 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:10:07.796 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:07 smithi083 bash[32381]: cluster 2024-09-10T19:10:07.448643+0000 osd.0 (osd.0) 454 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:10:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.351186+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.a scrub starts 2024-09-10T19:10:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.354688+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.a scrub ok 2024-09-10T19:10:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.458061+0000 mgr.y (mgr.24526) 809 : cluster [DBG] pgmap v1377: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:06.640662+0000 mon.a (mon.0) 3821 : audit [INF] from='client.49955 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.640848+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.744051+0000 osd.1 (osd.1) 39 : cluster [DBG] 2.a scrub starts 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:06.751010+0000 osd.1 (osd.1) 40 : cluster [DBG] 2.a scrub ok 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:06.762756+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:06.772936+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:07.205610+0000 mon.c (mon.2) 901 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:07.206946+0000 mon.c (mon.2) 902 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: audit 2024-09-10T19:10:07.217951+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:07.445197+0000 osd.0 (osd.0) 453 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:10:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:07 smithi165 bash[22591]: cluster 2024-09-10T19:10:07.448643+0000 osd.0 (osd.0) 454 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:10:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:08 smithi165 bash[22591]: cluster 2024-09-10T19:10:07.346430+0000 osd.5 (osd.5) 478 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:10:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:08 smithi165 bash[22591]: cluster 2024-09-10T19:10:07.349956+0000 osd.5 (osd.5) 479 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:10:08.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:08 smithi083 bash[18629]: cluster 2024-09-10T19:10:07.346430+0000 osd.5 (osd.5) 478 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:10:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:08 smithi083 bash[18629]: cluster 2024-09-10T19:10:07.349956+0000 osd.5 (osd.5) 479 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:10:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:08 smithi083 bash[32381]: cluster 2024-09-10T19:10:07.346430+0000 osd.5 (osd.5) 478 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:10:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:08 smithi083 bash[32381]: cluster 2024-09-10T19:10:07.349956+0000 osd.5 (osd.5) 479 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:08.458807+0000 mgr.y (mgr.24526) 810 : cluster [DBG] pgmap v1379: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: audit 2024-09-10T19:10:08.555989+0000 mgr.y (mgr.24526) 811 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:08.607344+0000 osd.3 (osd.3) 500 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:08.611041+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:08.663259+0000 osd.7 (osd.7) 482 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:08.666704+0000 osd.7 (osd.7) 483 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:10:10.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:09.465702+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[18629]: cluster 2024-09-10T19:10:09.470685+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:08.458807+0000 mgr.y (mgr.24526) 810 : cluster [DBG] pgmap v1379: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: audit 2024-09-10T19:10:08.555989+0000 mgr.y (mgr.24526) 811 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:08.607344+0000 osd.3 (osd.3) 500 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:08.611041+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:08.663259+0000 osd.7 (osd.7) 482 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:08.666704+0000 osd.7 (osd.7) 483 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:09.465702+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:10:10.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:09 smithi083 bash[32381]: cluster 2024-09-10T19:10:09.470685+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:08.458807+0000 mgr.y (mgr.24526) 810 : cluster [DBG] pgmap v1379: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: audit 2024-09-10T19:10:08.555989+0000 mgr.y (mgr.24526) 811 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:08.607344+0000 osd.3 (osd.3) 500 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:08.611041+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:08.663259+0000 osd.7 (osd.7) 482 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:10:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:08.666704+0000 osd.7 (osd.7) 483 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:10:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:09.465702+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:10:10.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:09 smithi165 bash[22591]: cluster 2024-09-10T19:10:09.470685+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.275335+0000 osd.0 (osd.0) 457 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.275774+0000 osd.0 (osd.0) 458 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.508273+0000 osd.0 (osd.0) 459 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.511634+0000 osd.0 (osd.0) 460 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.275335+0000 osd.0 (osd.0) 457 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.275774+0000 osd.0 (osd.0) 458 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:11.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.508273+0000 osd.0 (osd.0) 459 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:10:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:10 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.511634+0000 osd.0 (osd.0) 460 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:10:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:10 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.275335+0000 osd.0 (osd.0) 457 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:10 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.275774+0000 osd.0 (osd.0) 458 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:10 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.508273+0000 osd.0 (osd.0) 459 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:10:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:10 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.511634+0000 osd.0 (osd.0) 460 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:10:11.674 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:11 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.459616+0000 mgr.y (mgr.24526) 812 : cluster [DBG] pgmap v1380: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-09-10T19:10:12.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:11 smithi083 bash[18629]: cluster 2024-09-10T19:10:10.676300+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-09-10T19:10:12.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:11 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.459616+0000 mgr.y (mgr.24526) 812 : cluster [DBG] pgmap v1380: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-09-10T19:10:12.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:11 smithi083 bash[32381]: cluster 2024-09-10T19:10:10.676300+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-09-10T19:10:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:11 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.459616+0000 mgr.y (mgr.24526) 812 : cluster [DBG] pgmap v1380: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 680 B/s rd, 0 op/s 2024-09-10T19:10:12.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:11 smithi165 bash[22591]: cluster 2024-09-10T19:10:10.676300+0000 mon.a (mon.0) 3826 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-09-10T19:10:12.319 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:12] "GET /metrics HTTP/1.1" 200 224111 "" "Prometheus/2.43.0" 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[18629]: cluster 2024-09-10T19:10:11.678584+0000 osd.3 (osd.3) 502 : cluster [DBG] 4.c scrub starts 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[18629]: cluster 2024-09-10T19:10:11.681719+0000 osd.3 (osd.3) 503 : cluster [DBG] 4.c scrub ok 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[18629]: cluster 2024-09-10T19:10:11.808145+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[18629]: cluster 2024-09-10T19:10:11.813318+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:10:13.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[32381]: cluster 2024-09-10T19:10:11.678584+0000 osd.3 (osd.3) 502 : cluster [DBG] 4.c scrub starts 2024-09-10T19:10:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[32381]: cluster 2024-09-10T19:10:11.681719+0000 osd.3 (osd.3) 503 : cluster [DBG] 4.c scrub ok 2024-09-10T19:10:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[32381]: cluster 2024-09-10T19:10:11.808145+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:10:13.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:12 smithi083 bash[32381]: cluster 2024-09-10T19:10:11.813318+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:10:13.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:12 smithi165 bash[22591]: cluster 2024-09-10T19:10:11.678584+0000 osd.3 (osd.3) 502 : cluster [DBG] 4.c scrub starts 2024-09-10T19:10:13.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:12 smithi165 bash[22591]: cluster 2024-09-10T19:10:11.681719+0000 osd.3 (osd.3) 503 : cluster [DBG] 4.c scrub ok 2024-09-10T19:10:13.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:12 smithi165 bash[22591]: cluster 2024-09-10T19:10:11.808145+0000 osd.1 (osd.1) 41 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:10:13.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:12 smithi165 bash[22591]: cluster 2024-09-10T19:10:11.813318+0000 osd.1 (osd.1) 42 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:10:13.944 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.336831+0000 osd.5 (osd.5) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:13.944 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.337232+0000 osd.5 (osd.5) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:13.944 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.460923+0000 mgr.y (mgr.24526) 813 : cluster [DBG] pgmap v1382: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.512406+0000 osd.4 (osd.4) 444 : cluster [DBG] 2.f scrub starts 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.516523+0000 osd.4 (osd.4) 445 : cluster [DBG] 2.f scrub ok 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:12.694511+0000 mon.a (mon.0) 3827 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:13.511974+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[18629]: cluster 2024-09-10T19:10:13.516452+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.336831+0000 osd.5 (osd.5) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.337232+0000 osd.5 (osd.5) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.460923+0000 mgr.y (mgr.24526) 813 : cluster [DBG] pgmap v1382: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.512406+0000 osd.4 (osd.4) 444 : cluster [DBG] 2.f scrub starts 2024-09-10T19:10:13.945 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.516523+0000 osd.4 (osd.4) 445 : cluster [DBG] 2.f scrub ok 2024-09-10T19:10:13.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:12.694511+0000 mon.a (mon.0) 3827 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-09-10T19:10:13.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:13.511974+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:10:13.946 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:13 smithi083 bash[32381]: cluster 2024-09-10T19:10:13.516452+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:10:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.336831+0000 osd.5 (osd.5) 480 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.337232+0000 osd.5 (osd.5) 481 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.460923+0000 mgr.y (mgr.24526) 813 : cluster [DBG] pgmap v1382: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.512406+0000 osd.4 (osd.4) 444 : cluster [DBG] 2.f scrub starts 2024-09-10T19:10:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.516523+0000 osd.4 (osd.4) 445 : cluster [DBG] 2.f scrub ok 2024-09-10T19:10:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:12.694511+0000 mon.a (mon.0) 3827 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-09-10T19:10:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:13.511974+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:10:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:13 smithi165 bash[22591]: cluster 2024-09-10T19:10:13.516452+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: cluster 2024-09-10T19:10:13.677266+0000 osd.3 (osd.3) 504 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: cluster 2024-09-10T19:10:13.680258+0000 osd.3 (osd.3) 505 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: cluster 2024-09-10T19:10:14.461638+0000 mgr.y (mgr.24526) 814 : cluster [DBG] pgmap v1384: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: audit 2024-09-10T19:10:14.579693+0000 mon.c (mon.2) 903 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: cluster 2024-09-10T19:10:14.695770+0000 osd.7 (osd.7) 484 : cluster [DBG] 3.d scrub starts 2024-09-10T19:10:15.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[32381]: cluster 2024-09-10T19:10:14.701542+0000 osd.7 (osd.7) 485 : cluster [DBG] 3.d scrub ok 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: cluster 2024-09-10T19:10:13.677266+0000 osd.3 (osd.3) 504 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: cluster 2024-09-10T19:10:13.680258+0000 osd.3 (osd.3) 505 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: cluster 2024-09-10T19:10:14.461638+0000 mgr.y (mgr.24526) 814 : cluster [DBG] pgmap v1384: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: audit 2024-09-10T19:10:14.579693+0000 mon.c (mon.2) 903 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: cluster 2024-09-10T19:10:14.695770+0000 osd.7 (osd.7) 484 : cluster [DBG] 3.d scrub starts 2024-09-10T19:10:15.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:15 smithi083 bash[18629]: cluster 2024-09-10T19:10:14.701542+0000 osd.7 (osd.7) 485 : cluster [DBG] 3.d scrub ok 2024-09-10T19:10:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: cluster 2024-09-10T19:10:13.677266+0000 osd.3 (osd.3) 504 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:10:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: cluster 2024-09-10T19:10:13.680258+0000 osd.3 (osd.3) 505 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:10:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: cluster 2024-09-10T19:10:14.461638+0000 mgr.y (mgr.24526) 814 : cluster [DBG] pgmap v1384: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-09-10T19:10:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: audit 2024-09-10T19:10:14.579693+0000 mon.c (mon.2) 903 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:15.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: cluster 2024-09-10T19:10:14.695770+0000 osd.7 (osd.7) 484 : cluster [DBG] 3.d scrub starts 2024-09-10T19:10:15.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:15 smithi165 bash[22591]: cluster 2024-09-10T19:10:14.701542+0000 osd.7 (osd.7) 485 : cluster [DBG] 3.d scrub ok 2024-09-10T19:10:16.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/360084658 >> v1:172.21.15.83:6790/0 conn(0x7fe1e8108050 legacy=0x7fe1e810a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:16.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/360084658 shutdown_connections 2024-09-10T19:10:16.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/360084658 >> 172.21.15.83:0/360084658 conn(0x7fe1e8074c70 msgr2=0x7fe1e80f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:16.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/360084658 shutdown_connections 2024-09-10T19:10:16.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/360084658 wait complete. 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 librados: starting msgr at 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 librados: starting objecter 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 Processor -- start 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 -- start start 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 librados: setting wanted keys 2024-09-10T19:10:16.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.109+0000 7fe1ef6e7640 1 librados: calling monclient init 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1e811b270 con 0x7fe1e81044f0 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1e811c430 con 0x7fe1e8101710 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe1e811d630 con 0x7fe1e8108050 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ecc82640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe1e81044f0 0x7fe1e80fe010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:49140/0 (socket says 172.21.15.83:49140) 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ecc82640 1 -- 172.21.15.83:0/1104596248 learned_addr learned my addr 172.21.15.83:0/1104596248 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter ms_handle_connect 0x7fe1e81044f0 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297038763 0 0) 0x7fe1e811b270 con 0x7fe1e81044f0 2024-09-10T19:10:16.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d40034e0 con 0x7fe1e81044f0 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter ms_handle_connect 0x7fe1e8108050 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter ms_handle_connect 0x7fe1e8101710 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3584844979 0 0) 0x7fe1d40034e0 con 0x7fe1e81044f0 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1e811b270 con 0x7fe1e81044f0 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe1d8002970 con 0x7fe1e81044f0 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248543941 0 0) 0x7fe1e811d630 con 0x7fe1e8108050 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d40034e0 con 0x7fe1e8108050 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824082219 0 0) 0x7fe1e811c430 con 0x7fe1e8101710 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e811d630 con 0x7fe1e8101710 2024-09-10T19:10:16.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 78459409 0 0) 0x7fe1e811b270 con 0x7fe1e81044f0 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 >> v1:172.21.15.83:6790/0 conn(0x7fe1e8108050 legacy=0x7fe1e8119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 >> v1:172.21.15.165:6789/0 conn(0x7fe1e8101710 legacy=0x7fe1e80fd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e811e830 con 0x7fe1e81044f0 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e811c620 con 0x7fe1e81044f0 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 10 client.25875.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e811ca70 con 0x7fe1e81044f0 2024-09-10T19:10:16.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 librados: init done 2024-09-10T19:10:16.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe1d8002c70 con 0x7fe1e81044f0 2024-09-10T19:10:16.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe1d8005af0 con 0x7fe1e81044f0 2024-09-10T19:10:16.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.113+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc005010 con 0x7fe1e81044f0 2024-09-10T19:10:16.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe1d801bf10 con 0x7fe1e81044f0 2024-09-10T19:10:16.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(871..871 src has 750..871) v4 ==== 7363+0+0 (unknown 110426812 0 0) 0x7fe1d808a3d0 con 0x7fe1e81044f0 2024-09-10T19:10:16.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 10 client.25875.objecter ms_dispatch 0x7fe1e8000b90 osd_map(871..871 src has 750..871) v4 2024-09-10T19:10:16.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 3 client.25875.objecter handle_osd_map got epochs [871,871] > 0 2024-09-10T19:10:16.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 3 client.25875.objecter handle_osd_map decoding full epoch 871 2024-09-10T19:10:16.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 20 client.25875.objecter dump_active .. 0 homeless 2024-09-10T19:10:16.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.117+0000 7fe1ddffb640 10 client.25875.objecter ms_handle_connect 0x7fe1d4071830 2024-09-10T19:10:16.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.121+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe1d8056c10 con 0x7fe1e81044f0 2024-09-10T19:10:16.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.309+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fe1cc0052c0 con 0x7fe1e81044f0 2024-09-10T19:10:16.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.465+0000 7fe1ddffb640 1 -- 172.21.15.83:0/1104596248 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v872) v1 ==== 113+0+0 (unknown 3437091725 0 0) 0x7fe1d80565b0 con 0x7fe1e81044f0 2024-09-10T19:10:16.470 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 10 librados: watch_flush enter 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 10 librados: watch_flush exit 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 20 client.25875.objecter shutdown clearing up homeless session... 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 10 client.25875.objecter successfully canceled tick 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe1d4071830 legacy=0x7fe1d4073ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 >> v1:172.21.15.83:6789/0 conn(0x7fe1e81044f0 legacy=0x7fe1e80fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 shutdown_connections 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 >> 172.21.15.83:0/1104596248 conn(0x7fe1e8074c70 msgr2=0x7fe1e80fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 shutdown_connections 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 -- 172.21.15.83:0/1104596248 wait complete. 2024-09-10T19:10:16.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:16.469+0000 7fe1ef6e7640 1 librados: shutdown 2024-09-10T19:10:16.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:16 smithi083 bash[18629]: audit 2024-09-10T19:10:16.312366+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:16.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:16 smithi083 bash[32381]: audit 2024-09-10T19:10:16.312366+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:16.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:16 smithi165 bash[22591]: audit 2024-09-10T19:10:16.312366+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:17.342 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:15.675095+0000 osd.3 (osd.3) 506 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:15.679039+0000 osd.3 (osd.3) 507 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:16.462709+0000 mgr.y (mgr.24526) 815 : cluster [DBG] pgmap v1385: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: audit 2024-09-10T19:10:16.468873+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:16.469024+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:16.768955+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.f scrub starts 2024-09-10T19:10:17.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[18629]: cluster 2024-09-10T19:10:16.772602+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.f scrub ok 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:15.675095+0000 osd.3 (osd.3) 506 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:15.679039+0000 osd.3 (osd.3) 507 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:16.462709+0000 mgr.y (mgr.24526) 815 : cluster [DBG] pgmap v1385: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: audit 2024-09-10T19:10:16.468873+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:16.469024+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:16.768955+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.f scrub starts 2024-09-10T19:10:17.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:17 smithi083 bash[32381]: cluster 2024-09-10T19:10:16.772602+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.f scrub ok 2024-09-10T19:10:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:15.675095+0000 osd.3 (osd.3) 506 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:10:17.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:15.679039+0000 osd.3 (osd.3) 507 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:10:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:16.462709+0000 mgr.y (mgr.24526) 815 : cluster [DBG] pgmap v1385: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: audit 2024-09-10T19:10:16.468873+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? v1:172.21.15.83:0/1104596248' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:16.469024+0000 mon.a (mon.0) 3830 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-09-10T19:10:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:16.768955+0000 osd.1 (osd.1) 43 : cluster [DBG] 4.f scrub starts 2024-09-10T19:10:17.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:17 smithi165 bash[22591]: cluster 2024-09-10T19:10:16.772602+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.f scrub ok 2024-09-10T19:10:18.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.271356+0000 osd.5 (osd.5) 482 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.274833+0000 osd.5 (osd.5) 483 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.412916+0000 osd.0 (osd.0) 463 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.416888+0000 osd.0 (osd.0) 464 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.463384+0000 mgr.y (mgr.24526) 816 : cluster [DBG] pgmap v1387: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: audit 2024-09-10T19:10:18.567235+0000 mgr.y (mgr.24526) 817 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[32381]: cluster 2024-09-10T19:10:19.446210+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-09-10T19:10:19.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.271356+0000 osd.5 (osd.5) 482 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.274833+0000 osd.5 (osd.5) 483 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.412916+0000 osd.0 (osd.0) 463 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.416888+0000 osd.0 (osd.0) 464 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.463384+0000 mgr.y (mgr.24526) 816 : cluster [DBG] pgmap v1387: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: audit 2024-09-10T19:10:18.567235+0000 mgr.y (mgr.24526) 817 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:19 smithi083 bash[18629]: cluster 2024-09-10T19:10:19.446210+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.271356+0000 osd.5 (osd.5) 482 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.274833+0000 osd.5 (osd.5) 483 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.412916+0000 osd.0 (osd.0) 463 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.416888+0000 osd.0 (osd.0) 464 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.463384+0000 mgr.y (mgr.24526) 816 : cluster [DBG] pgmap v1387: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:19.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: audit 2024-09-10T19:10:18.567235+0000 mgr.y (mgr.24526) 817 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:19 smithi165 bash[22591]: cluster 2024-09-10T19:10:19.446210+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-09-10T19:10:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.875676+0000 osd.2 (osd.2) 359 : cluster [DBG] 12.a scrub starts 2024-09-10T19:10:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[18629]: cluster 2024-09-10T19:10:18.879208+0000 osd.2 (osd.2) 360 : cluster [DBG] 12.a scrub ok 2024-09-10T19:10:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[18629]: cluster 2024-09-10T19:10:19.259073+0000 osd.6 (osd.6) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:20.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[18629]: cluster 2024-09-10T19:10:19.259399+0000 osd.6 (osd.6) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:20.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.875676+0000 osd.2 (osd.2) 359 : cluster [DBG] 12.a scrub starts 2024-09-10T19:10:20.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[32381]: cluster 2024-09-10T19:10:18.879208+0000 osd.2 (osd.2) 360 : cluster [DBG] 12.a scrub ok 2024-09-10T19:10:20.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[32381]: cluster 2024-09-10T19:10:19.259073+0000 osd.6 (osd.6) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:20.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:20 smithi083 bash[32381]: cluster 2024-09-10T19:10:19.259399+0000 osd.6 (osd.6) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:20.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:20 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.875676+0000 osd.2 (osd.2) 359 : cluster [DBG] 12.a scrub starts 2024-09-10T19:10:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:20 smithi165 bash[22591]: cluster 2024-09-10T19:10:18.879208+0000 osd.2 (osd.2) 360 : cluster [DBG] 12.a scrub ok 2024-09-10T19:10:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:20 smithi165 bash[22591]: cluster 2024-09-10T19:10:19.259073+0000 osd.6 (osd.6) 441 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:20.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:20 smithi165 bash[22591]: cluster 2024-09-10T19:10:19.259399+0000 osd.6 (osd.6) 442 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[18629]: cluster 2024-09-10T19:10:20.271675+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[18629]: cluster 2024-09-10T19:10:20.275351+0000 osd.5 (osd.5) 485 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[18629]: cluster 2024-09-10T19:10:20.464160+0000 mgr.y (mgr.24526) 818 : cluster [DBG] pgmap v1389: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[32381]: cluster 2024-09-10T19:10:20.271675+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[32381]: cluster 2024-09-10T19:10:20.275351+0000 osd.5 (osd.5) 485 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:10:21.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:21 smithi083 bash[32381]: cluster 2024-09-10T19:10:20.464160+0000 mgr.y (mgr.24526) 818 : cluster [DBG] pgmap v1389: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-10T19:10:21.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:21 smithi165 bash[22591]: cluster 2024-09-10T19:10:20.271675+0000 osd.5 (osd.5) 484 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:10:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:21 smithi165 bash[22591]: cluster 2024-09-10T19:10:20.275351+0000 osd.5 (osd.5) 485 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:10:21.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:21 smithi165 bash[22591]: cluster 2024-09-10T19:10:20.464160+0000 mgr.y (mgr.24526) 818 : cluster [DBG] pgmap v1389: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-09-10T19:10:22.118 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:22.790 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:22] "GET /metrics HTTP/1.1" 200 224098 "" "Prometheus/2.43.0" 2024-09-10T19:10:22.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[32381]: cluster 2024-09-10T19:10:21.832647+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:10:22.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[32381]: cluster 2024-09-10T19:10:21.836099+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:10:22.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[32381]: cluster 2024-09-10T19:10:21.960105+0000 osd.6 (osd.6) 443 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:10:22.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[32381]: cluster 2024-09-10T19:10:21.964181+0000 osd.6 (osd.6) 444 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:10:22.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[18629]: cluster 2024-09-10T19:10:21.832647+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:10:22.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[18629]: cluster 2024-09-10T19:10:21.836099+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:10:22.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[18629]: cluster 2024-09-10T19:10:21.960105+0000 osd.6 (osd.6) 443 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:10:22.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:22 smithi083 bash[18629]: cluster 2024-09-10T19:10:21.964181+0000 osd.6 (osd.6) 444 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:10:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:22 smithi165 bash[22591]: cluster 2024-09-10T19:10:21.832647+0000 osd.1 (osd.1) 45 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:10:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:22 smithi165 bash[22591]: cluster 2024-09-10T19:10:21.836099+0000 osd.1 (osd.1) 46 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:10:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:22 smithi165 bash[22591]: cluster 2024-09-10T19:10:21.960105+0000 osd.6 (osd.6) 443 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:10:22.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:22 smithi165 bash[22591]: cluster 2024-09-10T19:10:21.964181+0000 osd.6 (osd.6) 444 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[32381]: cluster 2024-09-10T19:10:22.466062+0000 mgr.y (mgr.24526) 819 : cluster [DBG] pgmap v1390: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[32381]: cluster 2024-09-10T19:10:22.616440+0000 osd.4 (osd.4) 446 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[32381]: cluster 2024-09-10T19:10:22.619828+0000 osd.4 (osd.4) 447 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[32381]: cluster 2024-09-10T19:10:22.829593+0000 osd.1 (osd.1) 47 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[32381]: cluster 2024-09-10T19:10:22.835140+0000 osd.1 (osd.1) 48 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[18629]: cluster 2024-09-10T19:10:22.466062+0000 mgr.y (mgr.24526) 819 : cluster [DBG] pgmap v1390: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[18629]: cluster 2024-09-10T19:10:22.616440+0000 osd.4 (osd.4) 446 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:10:23.764 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[18629]: cluster 2024-09-10T19:10:22.619828+0000 osd.4 (osd.4) 447 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:10:23.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[18629]: cluster 2024-09-10T19:10:22.829593+0000 osd.1 (osd.1) 47 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:10:23.765 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:23 smithi083 bash[18629]: cluster 2024-09-10T19:10:22.835140+0000 osd.1 (osd.1) 48 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:10:23.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:23 smithi165 bash[22591]: cluster 2024-09-10T19:10:22.466062+0000 mgr.y (mgr.24526) 819 : cluster [DBG] pgmap v1390: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:23.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:23 smithi165 bash[22591]: cluster 2024-09-10T19:10:22.616440+0000 osd.4 (osd.4) 446 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:10:23.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:23 smithi165 bash[22591]: cluster 2024-09-10T19:10:22.619828+0000 osd.4 (osd.4) 447 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:10:23.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:23 smithi165 bash[22591]: cluster 2024-09-10T19:10:22.829593+0000 osd.1 (osd.1) 47 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:10:23.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:23 smithi165 bash[22591]: cluster 2024-09-10T19:10:22.835140+0000 osd.1 (osd.1) 48 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:10:24.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[18629]: cluster 2024-09-10T19:10:23.767866+0000 osd.7 (osd.7) 486 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[18629]: cluster 2024-09-10T19:10:23.771513+0000 osd.7 (osd.7) 487 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[18629]: cluster 2024-09-10T19:10:23.994506+0000 osd.6 (osd.6) 445 : cluster [DBG] 4.1f deep-scrub starts 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.001004+0000 osd.6 (osd.6) 446 : cluster [DBG] 4.1f deep-scrub ok 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[32381]: cluster 2024-09-10T19:10:23.767866+0000 osd.7 (osd.7) 486 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[32381]: cluster 2024-09-10T19:10:23.771513+0000 osd.7 (osd.7) 487 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[32381]: cluster 2024-09-10T19:10:23.994506+0000 osd.6 (osd.6) 445 : cluster [DBG] 4.1f deep-scrub starts 2024-09-10T19:10:24.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:24 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.001004+0000 osd.6 (osd.6) 446 : cluster [DBG] 4.1f deep-scrub ok 2024-09-10T19:10:24.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:24 smithi165 bash[22591]: cluster 2024-09-10T19:10:23.767866+0000 osd.7 (osd.7) 486 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:10:24.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:24 smithi165 bash[22591]: cluster 2024-09-10T19:10:23.771513+0000 osd.7 (osd.7) 487 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:10:24.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:24 smithi165 bash[22591]: cluster 2024-09-10T19:10:23.994506+0000 osd.6 (osd.6) 445 : cluster [DBG] 4.1f deep-scrub starts 2024-09-10T19:10:24.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:24 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.001004+0000 osd.6 (osd.6) 446 : cluster [DBG] 4.1f deep-scrub ok 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.466821+0000 mgr.y (mgr.24526) 820 : cluster [DBG] pgmap v1391: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.492630+0000 osd.0 (osd.0) 465 : cluster [DBG] 12.e scrub starts 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.496185+0000 osd.0 (osd.0) 466 : cluster [DBG] 12.e scrub ok 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.812792+0000 osd.1 (osd.1) 49 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:24.817532+0000 osd.1 (osd.1) 50 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:25.012390+0000 osd.6 (osd.6) 447 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[18629]: cluster 2024-09-10T19:10:25.016571+0000 osd.6 (osd.6) 448 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T19:10:25.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.466821+0000 mgr.y (mgr.24526) 820 : cluster [DBG] pgmap v1391: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.492630+0000 osd.0 (osd.0) 465 : cluster [DBG] 12.e scrub starts 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.496185+0000 osd.0 (osd.0) 466 : cluster [DBG] 12.e scrub ok 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.812792+0000 osd.1 (osd.1) 49 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:24.817532+0000 osd.1 (osd.1) 50 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:25.012390+0000 osd.6 (osd.6) 447 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T19:10:25.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:25 smithi083 bash[32381]: cluster 2024-09-10T19:10:25.016571+0000 osd.6 (osd.6) 448 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.466821+0000 mgr.y (mgr.24526) 820 : cluster [DBG] pgmap v1391: 174 pgs: 174 active+clean; 635 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.492630+0000 osd.0 (osd.0) 465 : cluster [DBG] 12.e scrub starts 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.496185+0000 osd.0 (osd.0) 466 : cluster [DBG] 12.e scrub ok 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.812792+0000 osd.1 (osd.1) 49 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:24.817532+0000 osd.1 (osd.1) 50 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:10:25.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:25.012390+0000 osd.6 (osd.6) 447 : cluster [DBG] 3.10 deep-scrub starts 2024-09-10T19:10:25.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:25 smithi165 bash[22591]: cluster 2024-09-10T19:10:25.016571+0000 osd.6 (osd.6) 448 : cluster [DBG] 3.10 deep-scrub ok 2024-09-10T19:10:26.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 -- 172.21.15.83:0/1320486131 >> v1:172.21.15.83:6790/0 conn(0x7efee810f680 legacy=0x7efee8111a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:26.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 -- 172.21.15.83:0/1320486131 shutdown_connections 2024-09-10T19:10:26.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 -- 172.21.15.83:0/1320486131 >> 172.21.15.83:0/1320486131 conn(0x7efee80fa370 msgr2=0x7efee80fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:26.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 -- 172.21.15.83:0/1320486131 shutdown_connections 2024-09-10T19:10:26.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 -- 172.21.15.83:0/1320486131 wait complete. 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 librados: starting msgr at 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.076+0000 7efeef741640 1 librados: starting objecter 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 Processor -- start 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- start start 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 librados: setting wanted keys 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 librados: calling monclient init 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efee8122630 con 0x7efee8071c40 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efee8123830 con 0x7efee810f680 2024-09-10T19:10:26.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efee8124a30 con 0x7efee8072720 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeed4dd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7efee8071c40 0x7efee810bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35350/0 (socket says 172.21.15.83:35350) 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeed4dd640 1 -- 172.21.15.83:0/1386111935 learned_addr learned my addr 172.21.15.83:0/1386111935 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter ms_handle_connect 0x7efee8071c40 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter ms_handle_connect 0x7efee8072720 2024-09-10T19:10:26.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:26.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006632443 0 0) 0x7efee8124a30 con 0x7efee8072720 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed40034e0 con 0x7efee8072720 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700605815 0 0) 0x7efee8122630 con 0x7efee8071c40 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee8124a30 con 0x7efee8071c40 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter ms_handle_connect 0x7efee810f680 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815287024 0 0) 0x7efee8123830 con 0x7efee810f680 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee8122630 con 0x7efee810f680 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3162079039 0 0) 0x7efee8124a30 con 0x7efee8071c40 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee8123830 con 0x7efee8071c40 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 440591789 0 0) 0x7efed40034e0 con 0x7efee8072720 2024-09-10T19:10:26.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee8124a30 con 0x7efee8072720 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efee40019d0 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efed8001840 con 0x7efee8072720 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4012479499 0 0) 0x7efee8123830 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 >> v1:172.21.15.83:6790/0 conn(0x7efee8072720 legacy=0x7efee810c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 >> v1:172.21.15.165:6789/0 conn(0x7efee810f680 legacy=0x7efee8120d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8125c30 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee8124c20 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 10 client.25881.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efee8125070 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeef741640 1 librados: init done 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7efee40024a0 con 0x7efee8071c40 2024-09-10T19:10:26.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.080+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efee4005b30 con 0x7efee8071c40 2024-09-10T19:10:26.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc005010 con 0x7efee8071c40 2024-09-10T19:10:26.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7efee4002040 con 0x7efee8071c40 2024-09-10T19:10:26.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(873..873 src has 750..873) v4 ==== 7363+0+0 (unknown 1559712046 0 0) 0x7efee408a3b0 con 0x7efee8071c40 2024-09-10T19:10:26.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 10 client.25881.objecter ms_dispatch 0x7efee8000b90 osd_map(873..873 src has 750..873) v4 2024-09-10T19:10:26.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 3 client.25881.objecter handle_osd_map got epochs [873,873] > 0 2024-09-10T19:10:26.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 3 client.25881.objecter handle_osd_map decoding full epoch 873 2024-09-10T19:10:26.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 20 client.25881.objecter dump_active .. 0 homeless 2024-09-10T19:10:26.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.084+0000 7efeddffb640 10 client.25881.objecter ms_handle_connect 0x7efed4071540 2024-09-10T19:10:26.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.092+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7efee4056bf0 con 0x7efee8071c40 2024-09-10T19:10:26.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.280+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7efecc005840 con 0x7efee8071c40 2024-09-10T19:10:26.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.528+0000 7efeddffb640 1 -- 172.21.15.83:0/1386111935 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v874) v1 ==== 133+0+0 (unknown 1190967290 0 0) 0x7efee4056590 con 0x7efee8071c40 2024-09-10T19:10:26.534 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 10 librados: watch_flush enter 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 10 librados: watch_flush exit 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 20 client.25881.objecter shutdown clearing up homeless session... 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 10 client.25881.objecter successfully canceled tick 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 >> v1:172.21.15.83:6800/3071793735 conn(0x7efed4071540 legacy=0x7efed40739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 >> v1:172.21.15.83:6789/0 conn(0x7efee8071c40 legacy=0x7efee810bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 shutdown_connections 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 >> 172.21.15.83:0/1386111935 conn(0x7efee80fa370 msgr2=0x7efee8111a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 shutdown_connections 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 -- 172.21.15.83:0/1386111935 wait complete. 2024-09-10T19:10:26.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:26.536+0000 7efeef741640 1 librados: shutdown 2024-09-10T19:10:26.770 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[18629]: cluster 2024-09-10T19:10:25.583556+0000 osd.4 (osd.4) 448 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:10:26.770 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[18629]: cluster 2024-09-10T19:10:25.586859+0000 osd.4 (osd.4) 449 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:10:26.770 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[18629]: audit 2024-09-10T19:10:26.283247+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:26.770 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[32381]: cluster 2024-09-10T19:10:25.583556+0000 osd.4 (osd.4) 448 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:10:26.771 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[32381]: cluster 2024-09-10T19:10:25.586859+0000 osd.4 (osd.4) 449 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:10:26.771 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:26 smithi083 bash[32381]: audit 2024-09-10T19:10:26.283247+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:26 smithi165 bash[22591]: cluster 2024-09-10T19:10:25.583556+0000 osd.4 (osd.4) 448 : cluster [DBG] 4.13 scrub starts 2024-09-10T19:10:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:26 smithi165 bash[22591]: cluster 2024-09-10T19:10:25.586859+0000 osd.4 (osd.4) 449 : cluster [DBG] 4.13 scrub ok 2024-09-10T19:10:26.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:26 smithi165 bash[22591]: audit 2024-09-10T19:10:26.283247+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:10:27.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.468691+0000 mgr.y (mgr.24526) 821 : cluster [DBG] pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: audit 2024-09-10T19:10:26.532697+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.532829+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.609087+0000 osd.4 (osd.4) 450 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.614341+0000 osd.4 (osd.4) 451 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.782312+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.787719+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:10:27.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.788024+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[18629]: cluster 2024-09-10T19:10:26.790997+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.468691+0000 mgr.y (mgr.24526) 821 : cluster [DBG] pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: audit 2024-09-10T19:10:26.532697+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.532829+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.609087+0000 osd.4 (osd.4) 450 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.614341+0000 osd.4 (osd.4) 451 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.782312+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.787719+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.788024+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:10:27.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:27 smithi083 bash[32381]: cluster 2024-09-10T19:10:26.790997+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:10:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.468691+0000 mgr.y (mgr.24526) 821 : cluster [DBG] pgmap v1392: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: audit 2024-09-10T19:10:26.532697+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? v1:172.21.15.83:0/1386111935' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:10:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.532829+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-09-10T19:10:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.609087+0000 osd.4 (osd.4) 450 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:10:27.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.614341+0000 osd.4 (osd.4) 451 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:10:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.782312+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:10:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.787719+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:10:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.788024+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:10:27.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:27 smithi165 bash[22591]: cluster 2024-09-10T19:10:26.790997+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:10:28.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:27.769958+0000 osd.3 (osd.3) 508 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:10:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:27.774018+0000 osd.3 (osd.3) 509 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:10:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:28.423988+0000 osd.2 (osd.2) 361 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:28.424476+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:29.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:28.469470+0000 mgr.y (mgr.24526) 822 : cluster [DBG] pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:29.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: cluster 2024-09-10T19:10:28.547862+0000 mon.a (mon.0) 3835 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-09-10T19:10:29.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:29 smithi165 bash[22591]: audit 2024-09-10T19:10:28.578569+0000 mgr.y (mgr.24526) 823 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:27.769958+0000 osd.3 (osd.3) 508 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:27.774018+0000 osd.3 (osd.3) 509 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:28.423988+0000 osd.2 (osd.2) 361 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:28.424476+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:28.469470+0000 mgr.y (mgr.24526) 822 : cluster [DBG] pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: cluster 2024-09-10T19:10:28.547862+0000 mon.a (mon.0) 3835 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-09-10T19:10:30.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[18629]: audit 2024-09-10T19:10:28.578569+0000 mgr.y (mgr.24526) 823 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:27.769958+0000 osd.3 (osd.3) 508 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:27.774018+0000 osd.3 (osd.3) 509 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:28.423988+0000 osd.2 (osd.2) 361 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:28.424476+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:28.469470+0000 mgr.y (mgr.24526) 822 : cluster [DBG] pgmap v1394: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: cluster 2024-09-10T19:10:28.547862+0000 mon.a (mon.0) 3835 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-09-10T19:10:30.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:29 smithi083 bash[32381]: audit 2024-09-10T19:10:28.578569+0000 mgr.y (mgr.24526) 823 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:30 smithi165 bash[22591]: audit 2024-09-10T19:10:29.579714+0000 mon.c (mon.2) 904 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:30 smithi165 bash[22591]: cluster 2024-09-10T19:10:29.654543+0000 osd.4 (osd.4) 452 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:10:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:30 smithi165 bash[22591]: cluster 2024-09-10T19:10:29.658082+0000 osd.4 (osd.4) 453 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:10:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:30 smithi165 bash[22591]: cluster 2024-09-10T19:10:29.940923+0000 osd.6 (osd.6) 449 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:10:30.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:30 smithi165 bash[22591]: cluster 2024-09-10T19:10:29.950661+0000 osd.6 (osd.6) 450 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[18629]: audit 2024-09-10T19:10:29.579714+0000 mon.c (mon.2) 904 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[18629]: cluster 2024-09-10T19:10:29.654543+0000 osd.4 (osd.4) 452 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[18629]: cluster 2024-09-10T19:10:29.658082+0000 osd.4 (osd.4) 453 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[18629]: cluster 2024-09-10T19:10:29.940923+0000 osd.6 (osd.6) 449 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[18629]: cluster 2024-09-10T19:10:29.950661+0000 osd.6 (osd.6) 450 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[32381]: audit 2024-09-10T19:10:29.579714+0000 mon.c (mon.2) 904 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:31.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[32381]: cluster 2024-09-10T19:10:29.654543+0000 osd.4 (osd.4) 452 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:10:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[32381]: cluster 2024-09-10T19:10:29.658082+0000 osd.4 (osd.4) 453 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:10:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[32381]: cluster 2024-09-10T19:10:29.940923+0000 osd.6 (osd.6) 449 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:10:31.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:30 smithi083 bash[32381]: cluster 2024-09-10T19:10:29.950661+0000 osd.6 (osd.6) 450 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:10:31.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.470239+0000 mgr.y (mgr.24526) 824 : cluster [DBG] pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.620812+0000 osd.4 (osd.4) 454 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:10:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.624553+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:10:31.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.802399+0000 osd.7 (osd.7) 490 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:10:31.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:31 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.807264+0000 osd.7 (osd.7) 491 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.470239+0000 mgr.y (mgr.24526) 824 : cluster [DBG] pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.620812+0000 osd.4 (osd.4) 454 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.624553+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.802399+0000 osd.7 (osd.7) 490 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.807264+0000 osd.7 (osd.7) 491 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:10:32.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.470239+0000 mgr.y (mgr.24526) 824 : cluster [DBG] pgmap v1396: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:32.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.620812+0000 osd.4 (osd.4) 454 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:10:32.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.624553+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:10:32.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.802399+0000 osd.7 (osd.7) 490 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:10:32.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:31 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.807264+0000 osd.7 (osd.7) 491 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:10:32.180 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.809577+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[32381]: cluster 2024-09-10T19:10:30.814376+0000 osd.3 (osd.3) 511 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[32381]: cluster 2024-09-10T19:10:31.882788+0000 osd.6 (osd.6) 451 : cluster [DBG] 4.e scrub starts 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[32381]: cluster 2024-09-10T19:10:31.886742+0000 osd.6 (osd.6) 452 : cluster [DBG] 4.e scrub ok 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.809577+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:10:32.567 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[18629]: cluster 2024-09-10T19:10:30.814376+0000 osd.3 (osd.3) 511 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:10:32.568 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[18629]: cluster 2024-09-10T19:10:31.882788+0000 osd.6 (osd.6) 451 : cluster [DBG] 4.e scrub starts 2024-09-10T19:10:32.568 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[18629]: cluster 2024-09-10T19:10:31.886742+0000 osd.6 (osd.6) 452 : cluster [DBG] 4.e scrub ok 2024-09-10T19:10:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:32 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.809577+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:10:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:32 smithi165 bash[22591]: cluster 2024-09-10T19:10:30.814376+0000 osd.3 (osd.3) 511 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:10:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:32 smithi165 bash[22591]: cluster 2024-09-10T19:10:31.882788+0000 osd.6 (osd.6) 451 : cluster [DBG] 4.e scrub starts 2024-09-10T19:10:32.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:32 smithi165 bash[22591]: cluster 2024-09-10T19:10:31.886742+0000 osd.6 (osd.6) 452 : cluster [DBG] 4.e scrub ok 2024-09-10T19:10:33.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:32] "GET /metrics HTTP/1.1" 200 224085 "" "Prometheus/2.43.0" 2024-09-10T19:10:33.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[18629]: cluster 2024-09-10T19:10:31.593725+0000 osd.0 (osd.0) 467 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:10:33.848 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[18629]: cluster 2024-09-10T19:10:31.597435+0000 osd.0 (osd.0) 468 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.472089+0000 mgr.y (mgr.24526) 825 : cluster [DBG] pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.894772+0000 osd.6 (osd.6) 453 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.898779+0000 osd.6 (osd.6) 454 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[32381]: cluster 2024-09-10T19:10:31.593725+0000 osd.0 (osd.0) 467 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[32381]: cluster 2024-09-10T19:10:31.597435+0000 osd.0 (osd.0) 468 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.472089+0000 mgr.y (mgr.24526) 825 : cluster [DBG] pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.894772+0000 osd.6 (osd.6) 453 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:10:33.849 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:33 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.898779+0000 osd.6 (osd.6) 454 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:10:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:33 smithi165 bash[22591]: cluster 2024-09-10T19:10:31.593725+0000 osd.0 (osd.0) 467 : cluster [DBG] 12.1c deep-scrub starts 2024-09-10T19:10:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:33 smithi165 bash[22591]: cluster 2024-09-10T19:10:31.597435+0000 osd.0 (osd.0) 468 : cluster [DBG] 12.1c deep-scrub ok 2024-09-10T19:10:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:33 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.472089+0000 mgr.y (mgr.24526) 825 : cluster [DBG] pgmap v1397: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:33 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.894772+0000 osd.6 (osd.6) 453 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:10:33.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:33 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.898779+0000 osd.6 (osd.6) 454 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.612627+0000 osd.0 (osd.0) 469 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.616331+0000 osd.0 (osd.0) 470 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.823332+0000 osd.3 (osd.3) 512 : cluster [DBG] 3.b scrub starts 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:32.827944+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.b scrub ok 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:33.680312+0000 osd.4 (osd.4) 456 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:10:34.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:34 smithi165 bash[22591]: cluster 2024-09-10T19:10:33.683924+0000 osd.4 (osd.4) 457 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.612627+0000 osd.0 (osd.0) 469 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.616331+0000 osd.0 (osd.0) 470 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.823332+0000 osd.3 (osd.3) 512 : cluster [DBG] 3.b scrub starts 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:32.827944+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.b scrub ok 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:33.680312+0000 osd.4 (osd.4) 456 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[18629]: cluster 2024-09-10T19:10:33.683924+0000 osd.4 (osd.4) 457 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:10:35.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.612627+0000 osd.0 (osd.0) 469 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:10:35.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.616331+0000 osd.0 (osd.0) 470 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:10:35.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.823332+0000 osd.3 (osd.3) 512 : cluster [DBG] 3.b scrub starts 2024-09-10T19:10:35.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:32.827944+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.b scrub ok 2024-09-10T19:10:35.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:33.680312+0000 osd.4 (osd.4) 456 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:10:35.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:34 smithi083 bash[32381]: cluster 2024-09-10T19:10:33.683924+0000 osd.4 (osd.4) 457 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:10:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.472856+0000 mgr.y (mgr.24526) 826 : cluster [DBG] pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.550349+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.550755+0000 osd.1 (osd.1) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.563156+0000 osd.0 (osd.0) 471 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:10:35.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.566844+0000 osd.0 (osd.0) 472 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:10:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.584014+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-09-10T19:10:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.727673+0000 osd.1 (osd.1) 55 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:10:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.731049+0000 osd.1 (osd.1) 56 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:10:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.846430+0000 osd.6 (osd.6) 455 : cluster [DBG] 12.4 deep-scrub starts 2024-09-10T19:10:35.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:35 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.850806+0000 osd.6 (osd.6) 456 : cluster [DBG] 12.4 deep-scrub ok 2024-09-10T19:10:35.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.472856+0000 mgr.y (mgr.24526) 826 : cluster [DBG] pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:35.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.550349+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:35.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.550755+0000 osd.1 (osd.1) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:35.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.563156+0000 osd.0 (osd.0) 471 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:10:35.866 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.566844+0000 osd.0 (osd.0) 472 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:10:35.867 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.584014+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-09-10T19:10:35.867 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.727673+0000 osd.1 (osd.1) 55 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:10:35.867 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.731049+0000 osd.1 (osd.1) 56 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:10:35.867 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.846430+0000 osd.6 (osd.6) 455 : cluster [DBG] 12.4 deep-scrub starts 2024-09-10T19:10:35.867 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.850806+0000 osd.6 (osd.6) 456 : cluster [DBG] 12.4 deep-scrub ok 2024-09-10T19:10:35.868 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.472856+0000 mgr.y (mgr.24526) 826 : cluster [DBG] pgmap v1398: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:35.868 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.550349+0000 osd.1 (osd.1) 53 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:10:35.868 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.550755+0000 osd.1 (osd.1) 54 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:10:35.868 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.563156+0000 osd.0 (osd.0) 471 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:10:35.869 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.566844+0000 osd.0 (osd.0) 472 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:10:35.869 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.584014+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-09-10T19:10:35.869 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.727673+0000 osd.1 (osd.1) 55 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:10:35.869 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.731049+0000 osd.1 (osd.1) 56 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:10:35.869 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.846430+0000 osd.6 (osd.6) 455 : cluster [DBG] 12.4 deep-scrub starts 2024-09-10T19:10:35.870 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:35 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.850806+0000 osd.6 (osd.6) 456 : cluster [DBG] 12.4 deep-scrub ok 2024-09-10T19:10:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- 172.21.15.83:0/279048495 >> v1:172.21.15.83:6790/0 conn(0x7f19b8108ee0 legacy=0x7f19b810b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- 172.21.15.83:0/279048495 shutdown_connections 2024-09-10T19:10:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- 172.21.15.83:0/279048495 >> 172.21.15.83:0/279048495 conn(0x7f19b80f7060 msgr2=0x7f19b80f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- 172.21.15.83:0/279048495 shutdown_connections 2024-09-10T19:10:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- 172.21.15.83:0/279048495 wait complete. 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 librados: starting msgr at 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 librados: starting objecter 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 Processor -- start 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- start start 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 librados: setting wanted keys 2024-09-10T19:10:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 librados: calling monclient init 2024-09-10T19:10:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b8114310 con 0x7f19b80fe150 2024-09-10T19:10:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b811f4b0 con 0x7f19b80fd580 2024-09-10T19:10:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19c070e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b8120650 con 0x7f19b8108ee0 2024-09-10T19:10:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19becab640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f19b8108ee0 0x7f19b811cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:53042/0 (socket says 172.21.15.83:53042) 2024-09-10T19:10:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.044+0000 7f19becab640 1 -- 172.21.15.83:0/3138125318 learned_addr learned my addr 172.21.15.83:0/3138125318 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter ms_handle_connect 0x7f19b8108ee0 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter ms_handle_connect 0x7f19b80fd580 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter ms_handle_connect 0x7f19b80fe150 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074971646 0 0) 0x7f19b811f4b0 con 0x7f19b80fd580 2024-09-10T19:10:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19a40034e0 con 0x7f19b80fd580 2024-09-10T19:10:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918853749 0 0) 0x7f19b8114310 con 0x7f19b80fe150 2024-09-10T19:10:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b811f4b0 con 0x7f19b80fe150 2024-09-10T19:10:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628679527 0 0) 0x7f19b8120650 con 0x7f19b8108ee0 2024-09-10T19:10:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b8114310 con 0x7f19b8108ee0 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 872852499 0 0) 0x7f19a40034e0 con 0x7f19b80fd580 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b8120650 con 0x7f19b80fd580 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758703853 0 0) 0x7f19b8114310 con 0x7f19b8108ee0 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19a40034e0 con 0x7f19b8108ee0 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19b40018c0 con 0x7f19b80fd580 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862079934 0 0) 0x7f19b811f4b0 con 0x7f19b80fe150 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b8114310 con 0x7f19b80fe150 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19b0002fb0 con 0x7f19b8108ee0 2024-09-10T19:10:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19a8001910 con 0x7f19b80fe150 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1141584847 0 0) 0x7f19b8120650 con 0x7f19b80fd580 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 >> v1:172.21.15.83:6790/0 conn(0x7f19b8108ee0 legacy=0x7f19b811cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 >> v1:172.21.15.83:6789/0 conn(0x7f19b80fe150 legacy=0x7f19b8119650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b8121830 con 0x7f19b80fd580 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b811f6a0 con 0x7f19b80fd580 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19c070e640 10 client.50485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b811faf0 con 0x7f19b80fd580 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19c070e640 1 librados: init done 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f19b4001f40 con 0x7f19b80fd580 2024-09-10T19:10:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f19b40030f0 con 0x7f19b80fd580 2024-09-10T19:10:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c005010 con 0x7f19b80fd580 2024-09-10T19:10:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.048+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f19b4001ae0 con 0x7f19b80fd580 2024-09-10T19:10:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 8 ==== osd_map(876..876 src has 786..876) v4 ==== 7363+0+0 (unknown 1507333563 0 0) 0x7f19b4087890 con 0x7f19b80fd580 2024-09-10T19:10:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 10 client.50485.objecter ms_dispatch 0x7f19b8000b90 osd_map(876..876 src has 786..876) v4 2024-09-10T19:10:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 3 client.50485.objecter handle_osd_map got epochs [876,876] > 0 2024-09-10T19:10:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 3 client.50485.objecter handle_osd_map decoding full epoch 876 2024-09-10T19:10:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 20 client.50485.objecter dump_active .. 0 homeless 2024-09-10T19:10:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.052+0000 7f19aeffd640 10 client.50485.objecter ms_handle_connect 0x7f19a4071ca0 2024-09-10T19:10:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.056+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f19b40540d0 con 0x7f19b80fd580 2024-09-10T19:10:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.248+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 --> v1:172.21.15.165:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f199c0052c0 con 0x7f19b80fd580 2024-09-10T19:10:36.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.588+0000 7f19aeffd640 1 -- 172.21.15.83:0/3138125318 <== mon.1 v1:172.21.15.165:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v877) v1 ==== 113+0+0 (unknown 140146458 0 0) 0x7f19b4053a70 con 0x7f19b80fd580 2024-09-10T19:10:36.591 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 10 librados: watch_flush enter 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 10 librados: watch_flush exit 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 20 client.50485.objecter shutdown clearing up homeless session... 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 10 client.50485.objecter successfully canceled tick 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 >> v1:172.21.15.83:6800/3071793735 conn(0x7f19a4071ca0 legacy=0x7f19a4074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 >> v1:172.21.15.165:6789/0 conn(0x7f19b80fd580 legacy=0x7f19b8113730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 shutdown_connections 2024-09-10T19:10:36.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 >> 172.21.15.83:0/3138125318 conn(0x7f19b80f7060 msgr2=0x7f19b810c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 shutdown_connections 2024-09-10T19:10:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 -- 172.21.15.83:0/3138125318 wait complete. 2024-09-10T19:10:36.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:36.592+0000 7f19c070e640 1 librados: shutdown 2024-09-10T19:10:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.869735+0000 osd.2 (osd.2) 363 : cluster [DBG] 12.9 deep-scrub starts 2024-09-10T19:10:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:34.872823+0000 osd.2 (osd.2) 364 : cluster [DBG] 12.9 deep-scrub ok 2024-09-10T19:10:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.202680+0000 osd.5 (osd.5) 486 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:10:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.207204+0000 osd.5 (osd.5) 487 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:10:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.775696+0000 osd.7 (osd.7) 492 : cluster [DBG] 3.e scrub starts 2024-09-10T19:10:36.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.780767+0000 osd.7 (osd.7) 493 : cluster [DBG] 3.e scrub ok 2024-09-10T19:10:36.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.842570+0000 osd.6 (osd.6) 457 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:10:36.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.845825+0000 osd.6 (osd.6) 458 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:10:36.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: audit 2024-09-10T19:10:36.249569+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.83:0/3138125318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:36.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:36 smithi165 bash[22591]: audit 2024-09-10T19:10:36.252153+0000 mon.a (mon.0) 3837 : audit [INF] from='client.50485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.869735+0000 osd.2 (osd.2) 363 : cluster [DBG] 12.9 deep-scrub starts 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:34.872823+0000 osd.2 (osd.2) 364 : cluster [DBG] 12.9 deep-scrub ok 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.202680+0000 osd.5 (osd.5) 486 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.207204+0000 osd.5 (osd.5) 487 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.775696+0000 osd.7 (osd.7) 492 : cluster [DBG] 3.e scrub starts 2024-09-10T19:10:37.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.780767+0000 osd.7 (osd.7) 493 : cluster [DBG] 3.e scrub ok 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.842570+0000 osd.6 (osd.6) 457 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.845825+0000 osd.6 (osd.6) 458 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: audit 2024-09-10T19:10:36.249569+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.83:0/3138125318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[18629]: audit 2024-09-10T19:10:36.252153+0000 mon.a (mon.0) 3837 : audit [INF] from='client.50485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.869735+0000 osd.2 (osd.2) 363 : cluster [DBG] 12.9 deep-scrub starts 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:34.872823+0000 osd.2 (osd.2) 364 : cluster [DBG] 12.9 deep-scrub ok 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.202680+0000 osd.5 (osd.5) 486 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.207204+0000 osd.5 (osd.5) 487 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.775696+0000 osd.7 (osd.7) 492 : cluster [DBG] 3.e scrub starts 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.780767+0000 osd.7 (osd.7) 493 : cluster [DBG] 3.e scrub ok 2024-09-10T19:10:37.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.842570+0000 osd.6 (osd.6) 457 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:10:37.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.845825+0000 osd.6 (osd.6) 458 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:10:37.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: audit 2024-09-10T19:10:36.249569+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.83:0/3138125318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:37.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:36 smithi083 bash[32381]: audit 2024-09-10T19:10:36.252153+0000 mon.a (mon.0) 3837 : audit [INF] from='client.50485 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:37.483 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.831851+0000 osd.2 (osd.2) 365 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:35.835587+0000 osd.2 (osd.2) 366 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.474740+0000 mgr.y (mgr.24526) 827 : cluster [DBG] pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: audit 2024-09-10T19:10:36.590068+0000 mon.a (mon.0) 3838 : audit [INF] from='client.50485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.590344+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.759986+0000 osd.7 (osd.7) 494 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:10:37.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.763913+0000 osd.7 (osd.7) 495 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.831851+0000 osd.2 (osd.2) 365 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:35.835587+0000 osd.2 (osd.2) 366 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.474740+0000 mgr.y (mgr.24526) 827 : cluster [DBG] pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: audit 2024-09-10T19:10:36.590068+0000 mon.a (mon.0) 3838 : audit [INF] from='client.50485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.590344+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.759986+0000 osd.7 (osd.7) 494 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:10:37.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:37 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.763913+0000 osd.7 (osd.7) 495 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:10:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.831851+0000 osd.2 (osd.2) 365 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:10:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:35.835587+0000 osd.2 (osd.2) 366 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:10:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.474740+0000 mgr.y (mgr.24526) 827 : cluster [DBG] pgmap v1400: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: audit 2024-09-10T19:10:36.590068+0000 mon.a (mon.0) 3838 : audit [INF] from='client.50485 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.590344+0000 mon.a (mon.0) 3839 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-09-10T19:10:37.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.759986+0000 osd.7 (osd.7) 494 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:10:37.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:37 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.763913+0000 osd.7 (osd.7) 495 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:10:38.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.872166+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.17 deep-scrub starts 2024-09-10T19:10:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:36.878356+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.17 deep-scrub ok 2024-09-10T19:10:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:37.608926+0000 osd.0 (osd.0) 473 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:10:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:37.611164+0000 osd.0 (osd.0) 474 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:10:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:37.698342+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:10:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:38 smithi165 bash[22591]: cluster 2024-09-10T19:10:37.701350+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.872166+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.17 deep-scrub starts 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:36.878356+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.17 deep-scrub ok 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:37.608926+0000 osd.0 (osd.0) 473 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:37.611164+0000 osd.0 (osd.0) 474 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:37.698342+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[18629]: cluster 2024-09-10T19:10:37.701350+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:10:39.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.872166+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.17 deep-scrub starts 2024-09-10T19:10:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:36.878356+0000 osd.2 (osd.2) 368 : cluster [DBG] 6.17 deep-scrub ok 2024-09-10T19:10:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:37.608926+0000 osd.0 (osd.0) 473 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:10:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:37.611164+0000 osd.0 (osd.0) 474 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:10:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:37.698342+0000 osd.1 (osd.1) 57 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:10:39.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:38 smithi083 bash[32381]: cluster 2024-09-10T19:10:37.701350+0000 osd.1 (osd.1) 58 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:10:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:39 smithi165 bash[22591]: cluster 2024-09-10T19:10:38.475510+0000 mgr.y (mgr.24526) 828 : cluster [DBG] pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:39 smithi165 bash[22591]: audit 2024-09-10T19:10:38.583551+0000 mgr.y (mgr.24526) 829 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:39 smithi165 bash[22591]: cluster 2024-09-10T19:10:38.640243+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:10:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:39 smithi165 bash[22591]: cluster 2024-09-10T19:10:38.650536+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[32381]: cluster 2024-09-10T19:10:38.475510+0000 mgr.y (mgr.24526) 828 : cluster [DBG] pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[32381]: audit 2024-09-10T19:10:38.583551+0000 mgr.y (mgr.24526) 829 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[32381]: cluster 2024-09-10T19:10:38.640243+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[32381]: cluster 2024-09-10T19:10:38.650536+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[18629]: cluster 2024-09-10T19:10:38.475510+0000 mgr.y (mgr.24526) 828 : cluster [DBG] pgmap v1402: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[18629]: audit 2024-09-10T19:10:38.583551+0000 mgr.y (mgr.24526) 829 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[18629]: cluster 2024-09-10T19:10:38.640243+0000 osd.0 (osd.0) 475 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:10:40.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:39 smithi083 bash[18629]: cluster 2024-09-10T19:10:38.650536+0000 osd.0 (osd.0) 476 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.665439+0000 osd.0 (osd.0) 477 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.668954+0000 osd.0 (osd.0) 478 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.726195+0000 osd.4 (osd.4) 458 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.729291+0000 osd.4 (osd.4) 459 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.792894+0000 osd.6 (osd.6) 459 : cluster [DBG] 3.a scrub starts 2024-09-10T19:10:41.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[18629]: cluster 2024-09-10T19:10:39.798099+0000 osd.6 (osd.6) 460 : cluster [DBG] 3.a scrub ok 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.665439+0000 osd.0 (osd.0) 477 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.668954+0000 osd.0 (osd.0) 478 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.726195+0000 osd.4 (osd.4) 458 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.729291+0000 osd.4 (osd.4) 459 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.792894+0000 osd.6 (osd.6) 459 : cluster [DBG] 3.a scrub starts 2024-09-10T19:10:41.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:40 smithi083 bash[32381]: cluster 2024-09-10T19:10:39.798099+0000 osd.6 (osd.6) 460 : cluster [DBG] 3.a scrub ok 2024-09-10T19:10:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.665439+0000 osd.0 (osd.0) 477 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:10:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.668954+0000 osd.0 (osd.0) 478 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:10:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.726195+0000 osd.4 (osd.4) 458 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:10:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.729291+0000 osd.4 (osd.4) 459 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:10:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.792894+0000 osd.6 (osd.6) 459 : cluster [DBG] 3.a scrub starts 2024-09-10T19:10:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:40 smithi165 bash[22591]: cluster 2024-09-10T19:10:39.798099+0000 osd.6 (osd.6) 460 : cluster [DBG] 3.a scrub ok 2024-09-10T19:10:41.635 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[18629]: cluster 2024-09-10T19:10:40.255660+0000 osd.5 (osd.5) 488 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[18629]: cluster 2024-09-10T19:10:40.259082+0000 osd.5 (osd.5) 489 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[18629]: cluster 2024-09-10T19:10:40.476344+0000 mgr.y (mgr.24526) 830 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[18629]: cluster 2024-09-10T19:10:40.723133+0000 osd.7 (osd.7) 496 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[18629]: cluster 2024-09-10T19:10:40.727064+0000 osd.7 (osd.7) 497 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[32381]: cluster 2024-09-10T19:10:40.255660+0000 osd.5 (osd.5) 488 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:10:42.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[32381]: cluster 2024-09-10T19:10:40.259082+0000 osd.5 (osd.5) 489 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:10:42.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[32381]: cluster 2024-09-10T19:10:40.476344+0000 mgr.y (mgr.24526) 830 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:42.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[32381]: cluster 2024-09-10T19:10:40.723133+0000 osd.7 (osd.7) 496 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:10:42.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:41 smithi083 bash[32381]: cluster 2024-09-10T19:10:40.727064+0000 osd.7 (osd.7) 497 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:10:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[22591]: cluster 2024-09-10T19:10:40.255660+0000 osd.5 (osd.5) 488 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:10:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[22591]: cluster 2024-09-10T19:10:40.259082+0000 osd.5 (osd.5) 489 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:10:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[22591]: cluster 2024-09-10T19:10:40.476344+0000 mgr.y (mgr.24526) 830 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[22591]: cluster 2024-09-10T19:10:40.723133+0000 osd.7 (osd.7) 496 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:10:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:41 smithi165 bash[22591]: cluster 2024-09-10T19:10:40.727064+0000 osd.7 (osd.7) 497 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:10:42.268 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:42] "GET /metrics HTTP/1.1" 200 224085 "" "Prometheus/2.43.0" 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[18629]: cluster 2024-09-10T19:10:41.276054+0000 osd.5 (osd.5) 490 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[18629]: cluster 2024-09-10T19:10:41.279287+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[18629]: cluster 2024-09-10T19:10:41.636569+0000 osd.0 (osd.0) 479 : cluster [DBG] 12.c scrub starts 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[18629]: cluster 2024-09-10T19:10:41.640058+0000 osd.0 (osd.0) 480 : cluster [DBG] 12.c scrub ok 2024-09-10T19:10:43.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[32381]: cluster 2024-09-10T19:10:41.276054+0000 osd.5 (osd.5) 490 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:10:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[32381]: cluster 2024-09-10T19:10:41.279287+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:10:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[32381]: cluster 2024-09-10T19:10:41.636569+0000 osd.0 (osd.0) 479 : cluster [DBG] 12.c scrub starts 2024-09-10T19:10:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:42 smithi083 bash[32381]: cluster 2024-09-10T19:10:41.640058+0000 osd.0 (osd.0) 480 : cluster [DBG] 12.c scrub ok 2024-09-10T19:10:43.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:42 smithi165 bash[22591]: cluster 2024-09-10T19:10:41.276054+0000 osd.5 (osd.5) 490 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:10:43.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:42 smithi165 bash[22591]: cluster 2024-09-10T19:10:41.279287+0000 osd.5 (osd.5) 491 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:10:43.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:42 smithi165 bash[22591]: cluster 2024-09-10T19:10:41.636569+0000 osd.0 (osd.0) 479 : cluster [DBG] 12.c scrub starts 2024-09-10T19:10:43.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:42 smithi165 bash[22591]: cluster 2024-09-10T19:10:41.640058+0000 osd.0 (osd.0) 480 : cluster [DBG] 12.c scrub ok 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[18629]: cluster 2024-09-10T19:10:42.478179+0000 mgr.y (mgr.24526) 831 : cluster [DBG] pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[18629]: cluster 2024-09-10T19:10:42.728399+0000 osd.7 (osd.7) 498 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[18629]: cluster 2024-09-10T19:10:42.731996+0000 osd.7 (osd.7) 499 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[32381]: cluster 2024-09-10T19:10:42.478179+0000 mgr.y (mgr.24526) 831 : cluster [DBG] pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[32381]: cluster 2024-09-10T19:10:42.728399+0000 osd.7 (osd.7) 498 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:10:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:43 smithi083 bash[32381]: cluster 2024-09-10T19:10:42.731996+0000 osd.7 (osd.7) 499 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:10:44.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:43 smithi165 bash[22591]: cluster 2024-09-10T19:10:42.478179+0000 mgr.y (mgr.24526) 831 : cluster [DBG] pgmap v1404: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:43 smithi165 bash[22591]: cluster 2024-09-10T19:10:42.728399+0000 osd.7 (osd.7) 498 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:10:44.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:43 smithi165 bash[22591]: cluster 2024-09-10T19:10:42.731996+0000 osd.7 (osd.7) 499 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.284990+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.290129+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.837407+0000 osd.4 (osd.4) 460 : cluster [DBG] 3.1a deep-scrub starts 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.846263+0000 osd.4 (osd.4) 461 : cluster [DBG] 3.1a deep-scrub ok 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[18629]: audit 2024-09-10T19:10:44.580288+0000 mon.c (mon.2) 905 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.284990+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:10:45.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.290129+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:10:45.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.837407+0000 osd.4 (osd.4) 460 : cluster [DBG] 3.1a deep-scrub starts 2024-09-10T19:10:45.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.846263+0000 osd.4 (osd.4) 461 : cluster [DBG] 3.1a deep-scrub ok 2024-09-10T19:10:45.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:44 smithi083 bash[32381]: audit 2024-09-10T19:10:44.580288+0000 mon.c (mon.2) 905 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:44 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.284990+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:10:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:44 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.290129+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:10:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:44 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.837407+0000 osd.4 (osd.4) 460 : cluster [DBG] 3.1a deep-scrub starts 2024-09-10T19:10:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:44 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.846263+0000 osd.4 (osd.4) 461 : cluster [DBG] 3.1a deep-scrub ok 2024-09-10T19:10:45.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:44 smithi165 bash[22591]: audit 2024-09-10T19:10:44.580288+0000 mon.c (mon.2) 905 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.951405+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.d scrub starts 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[32381]: cluster 2024-09-10T19:10:43.957206+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.d scrub ok 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[32381]: cluster 2024-09-10T19:10:44.478954+0000 mgr.y (mgr.24526) 832 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[32381]: cluster 2024-09-10T19:10:44.616206+0000 osd.0 (osd.0) 481 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[32381]: cluster 2024-09-10T19:10:44.620176+0000 osd.0 (osd.0) 482 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:10:45.973 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.951405+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.d scrub starts 2024-09-10T19:10:45.974 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[18629]: cluster 2024-09-10T19:10:43.957206+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.d scrub ok 2024-09-10T19:10:45.974 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[18629]: cluster 2024-09-10T19:10:44.478954+0000 mgr.y (mgr.24526) 832 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:10:45.974 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[18629]: cluster 2024-09-10T19:10:44.616206+0000 osd.0 (osd.0) 481 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:10:45.974 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:45 smithi083 bash[18629]: cluster 2024-09-10T19:10:44.620176+0000 osd.0 (osd.0) 482 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:10:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:45 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.951405+0000 osd.2 (osd.2) 369 : cluster [DBG] 4.d scrub starts 2024-09-10T19:10:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:45 smithi165 bash[22591]: cluster 2024-09-10T19:10:43.957206+0000 osd.2 (osd.2) 370 : cluster [DBG] 4.d scrub ok 2024-09-10T19:10:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:45 smithi165 bash[22591]: cluster 2024-09-10T19:10:44.478954+0000 mgr.y (mgr.24526) 832 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:10:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:45 smithi165 bash[22591]: cluster 2024-09-10T19:10:44.616206+0000 osd.0 (osd.0) 481 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:10:46.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:45 smithi165 bash[22591]: cluster 2024-09-10T19:10:44.620176+0000 osd.0 (osd.0) 482 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:10:46.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- 172.21.15.83:0/908843015 >> v1:172.21.15.83:6789/0 conn(0x7fe5840701c0 legacy=0x7fe58410df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- 172.21.15.83:0/908843015 shutdown_connections 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- 172.21.15.83:0/908843015 >> 172.21.15.83:0/908843015 conn(0x7fe5840f7fe0 msgr2=0x7fe5840fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- 172.21.15.83:0/908843015 shutdown_connections 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- 172.21.15.83:0/908843015 wait complete. 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 librados: starting msgr at 2024-09-10T19:10:46.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 librados: starting objecter 2024-09-10T19:10:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 Processor -- start 2024-09-10T19:10:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- start start 2024-09-10T19:10:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 librados: setting wanted keys 2024-09-10T19:10:46.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 librados: calling monclient init 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5841225f0 con 0x7fe58410f620 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5841237b0 con 0x7fe5840716e0 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589f43640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5841249b0 con 0x7fe5840701c0 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589742640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe58410f620 0x7fe584120cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:52594/0 (socket says 172.21.15.83:52594) 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe589742640 1 -- 172.21.15.83:0/3998876273 learned_addr learned my addr 172.21.15.83:0/3998876273 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe588f41640 1 --1- 172.21.15.83:0/3998876273 >> v1:172.21.15.83:6790/0 conn(0x7fe5840701c0 0x7fe58410bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:50098/0 (socket says 172.21.15.83:50098) 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe5817fa640 10 client.?.objecter ms_handle_connect 0x7fe5840701c0 2024-09-10T19:10:46.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe5817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe5817fa640 10 client.?.objecter ms_handle_connect 0x7fe58410f620 2024-09-10T19:10:46.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.144+0000 7fe5817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 10 client.?.objecter ms_handle_connect 0x7fe5840716e0 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609070776 0 0) 0x7fe5841225f0 con 0x7fe58410f620 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe570003120 con 0x7fe58410f620 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538936712 0 0) 0x7fe5841249b0 con 0x7fe5840701c0 2024-09-10T19:10:46.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5841225f0 con 0x7fe5840701c0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591272214 0 0) 0x7fe5841237b0 con 0x7fe5840716e0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5841249b0 con 0x7fe5840716e0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 420123089 0 0) 0x7fe570003120 con 0x7fe58410f620 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5841237b0 con 0x7fe58410f620 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe57c002cd0 con 0x7fe58410f620 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752475525 0 0) 0x7fe5841225f0 con 0x7fe5840701c0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe570003120 con 0x7fe5840701c0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe5740018c0 con 0x7fe5840701c0 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2207384808 0 0) 0x7fe5841237b0 con 0x7fe58410f620 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 >> v1:172.21.15.83:6790/0 conn(0x7fe5840701c0 legacy=0x7fe58410bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:46.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 >> v1:172.21.15.165:6789/0 conn(0x7fe5840716e0 legacy=0x7fe58410c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584125bb0 con 0x7fe58410f620 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5841227e0 con 0x7fe58410f620 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe589f43640 10 client.25893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe584122c30 con 0x7fe58410f620 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe589f43640 1 librados: init done 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe57c001af0 con 0x7fe58410f620 2024-09-10T19:10:46.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe57c005de0 con 0x7fe58410f620 2024-09-10T19:10:46.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.148+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe564005010 con 0x7fe58410f620 2024-09-10T19:10:46.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe57c01c300 con 0x7fe58410f620 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(877..877 src has 827..877) v4 ==== 7363+0+0 (unknown 1866598722 0 0) 0x7fe57c08a760 con 0x7fe58410f620 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 10 client.25893.objecter ms_dispatch 0x7fe584000b90 osd_map(877..877 src has 827..877) v4 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 3 client.25893.objecter handle_osd_map got epochs [877,877] > 0 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 3 client.25893.objecter handle_osd_map decoding full epoch 877 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 20 client.25893.objecter dump_active .. 0 homeless 2024-09-10T19:10:46.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.152+0000 7fe5817fa640 10 client.25893.objecter ms_handle_connect 0x7fe5700716b0 2024-09-10T19:10:46.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.160+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe57c056b20 con 0x7fe58410f620 2024-09-10T19:10:46.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.344+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fe564005840 con 0x7fe58410f620 2024-09-10T19:10:46.671 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-10T19:10:46.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.668+0000 7fe5817fa640 1 -- 172.21.15.83:0/3998876273 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v878) v1 ==== 133+0+0 (unknown 2530720618 0 0) 0x7fe57c056940 con 0x7fe58410f620 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 10 librados: watch_flush enter 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 10 librados: watch_flush exit 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 20 client.25893.objecter shutdown clearing up homeless session... 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 10 client.25893.objecter successfully canceled tick 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe5700716b0 legacy=0x7fe570073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 >> v1:172.21.15.83:6789/0 conn(0x7fe58410f620 legacy=0x7fe584120cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:46.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 shutdown_connections 2024-09-10T19:10:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 >> 172.21.15.83:0/3998876273 conn(0x7fe5840f7fe0 msgr2=0x7fe5840f92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 shutdown_connections 2024-09-10T19:10:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 -- 172.21.15.83:0/3998876273 wait complete. 2024-09-10T19:10:46.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:46.672+0000 7fe589f43640 1 librados: shutdown 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:44.959258+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:44.962963+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.596241+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.603327+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.759309+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:10:47.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.763078+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.770986+0000 osd.6 (osd.6) 461 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.772240+0000 osd.4 (osd.4) 462 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.774957+0000 osd.6 (osd.6) 462 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: cluster 2024-09-10T19:10:45.777649+0000 osd.4 (osd.4) 463 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[32381]: audit 2024-09-10T19:10:46.350639+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:44.959258+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:44.962963+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.596241+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.603327+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.759309+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:10:47.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.763078+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:10:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.770986+0000 osd.6 (osd.6) 461 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:10:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.772240+0000 osd.4 (osd.4) 462 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:10:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.774957+0000 osd.6 (osd.6) 462 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:10:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: cluster 2024-09-10T19:10:45.777649+0000 osd.4 (osd.4) 463 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:10:47.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:46 smithi083 bash[18629]: audit 2024-09-10T19:10:46.350639+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:47.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:44.959258+0000 osd.2 (osd.2) 371 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:10:47.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:44.962963+0000 osd.2 (osd.2) 372 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:10:47.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.596241+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:10:47.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.603327+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.759309+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.763078+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.770986+0000 osd.6 (osd.6) 461 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.772240+0000 osd.4 (osd.4) 462 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.774957+0000 osd.6 (osd.6) 462 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: cluster 2024-09-10T19:10:45.777649+0000 osd.4 (osd.4) 463 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:10:47.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:46 smithi165 bash[22591]: audit 2024-09-10T19:10:46.350639+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:10:47.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T19:10:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.480781+0000 mgr.y (mgr.24526) 833 : cluster [DBG] pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.627713+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:10:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.633825+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:10:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: audit 2024-09-10T19:10:46.668600+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.668678+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.686057+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.687756+0000 osd.3 (osd.3) 515 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.795860+0000 osd.1 (osd.1) 61 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.805076+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.809584+0000 osd.7 (osd.7) 500 : cluster [DBG] 12.d scrub starts 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.813043+0000 osd.7 (osd.7) 501 : cluster [DBG] 12.d scrub ok 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.480781+0000 mgr.y (mgr.24526) 833 : cluster [DBG] pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.627713+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:10:47.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.633825+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: audit 2024-09-10T19:10:46.668600+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.668678+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.686057+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.687756+0000 osd.3 (osd.3) 515 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.795860+0000 osd.1 (osd.1) 61 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.805076+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.809584+0000 osd.7 (osd.7) 500 : cluster [DBG] 12.d scrub starts 2024-09-10T19:10:47.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:47 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.813043+0000 osd.7 (osd.7) 501 : cluster [DBG] 12.d scrub ok 2024-09-10T19:10:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.480781+0000 mgr.y (mgr.24526) 833 : cluster [DBG] pgmap v1406: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.627713+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:10:48.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.633825+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: audit 2024-09-10T19:10:46.668600+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? v1:172.21.15.83:0/3998876273' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.668678+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.686057+0000 osd.3 (osd.3) 514 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.687756+0000 osd.3 (osd.3) 515 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.795860+0000 osd.1 (osd.1) 61 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.805076+0000 osd.1 (osd.1) 62 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.809584+0000 osd.7 (osd.7) 500 : cluster [DBG] 12.d scrub starts 2024-09-10T19:10:48.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:47 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.813043+0000 osd.7 (osd.7) 501 : cluster [DBG] 12.d scrub ok 2024-09-10T19:10:48.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.945053+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:46.948440+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.269354+0000 osd.5 (osd.5) 494 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.273137+0000 osd.5 (osd.5) 495 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.675549+0000 osd.3 (osd.3) 516 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.678392+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.681347+0000 osd.3 (osd.3) 517 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.785754+0000 osd.7 (osd.7) 502 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:10:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:48 smithi165 bash[22591]: cluster 2024-09-10T19:10:47.790614+0000 osd.7 (osd.7) 503 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:10:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.945053+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:10:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:46.948440+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:10:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.269354+0000 osd.5 (osd.5) 494 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:10:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.273137+0000 osd.5 (osd.5) 495 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:10:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.675549+0000 osd.3 (osd.3) 516 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.678392+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.681347+0000 osd.3 (osd.3) 517 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.785754+0000 osd.7 (osd.7) 502 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[18629]: cluster 2024-09-10T19:10:47.790614+0000 osd.7 (osd.7) 503 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.945053+0000 osd.2 (osd.2) 373 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:46.948440+0000 osd.2 (osd.2) 374 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.269354+0000 osd.5 (osd.5) 494 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.273137+0000 osd.5 (osd.5) 495 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.675549+0000 osd.3 (osd.3) 516 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.678392+0000 mon.a (mon.0) 3843 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-09-10T19:10:49.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.681347+0000 osd.3 (osd.3) 517 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:10:49.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.785754+0000 osd.7 (osd.7) 502 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:10:49.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:48 smithi083 bash[32381]: cluster 2024-09-10T19:10:47.790614+0000 osd.7 (osd.7) 503 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:10:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:49 smithi083 bash[18629]: cluster 2024-09-10T19:10:48.481549+0000 mgr.y (mgr.24526) 834 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:50.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:49 smithi083 bash[18629]: audit 2024-09-10T19:10:48.592578+0000 mgr.y (mgr.24526) 835 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:49 smithi083 bash[32381]: cluster 2024-09-10T19:10:48.481549+0000 mgr.y (mgr.24526) 834 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:50.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:49 smithi083 bash[32381]: audit 2024-09-10T19:10:48.592578+0000 mgr.y (mgr.24526) 835 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:49 smithi165 bash[22591]: cluster 2024-09-10T19:10:48.481549+0000 mgr.y (mgr.24526) 834 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:50.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:49 smithi165 bash[22591]: audit 2024-09-10T19:10:48.592578+0000 mgr.y (mgr.24526) 835 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:48.964173+0000 osd.2 (osd.2) 375 : cluster [DBG] 2.e scrub starts 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:48.968158+0000 osd.2 (osd.2) 376 : cluster [DBG] 2.e scrub ok 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.251790+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.255131+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.652439+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.656265+0000 osd.4 (osd.4) 465 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:10:51.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.801720+0000 osd.6 (osd.6) 463 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.805514+0000 osd.6 (osd.6) 464 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.830854+0000 osd.7 (osd.7) 504 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[18629]: cluster 2024-09-10T19:10:49.835774+0000 osd.7 (osd.7) 505 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:48.964173+0000 osd.2 (osd.2) 375 : cluster [DBG] 2.e scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:48.968158+0000 osd.2 (osd.2) 376 : cluster [DBG] 2.e scrub ok 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.251790+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.255131+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.652439+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.656265+0000 osd.4 (osd.4) 465 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.801720+0000 osd.6 (osd.6) 463 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:10:51.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.805514+0000 osd.6 (osd.6) 464 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:10:51.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.830854+0000 osd.7 (osd.7) 504 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:10:51.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:50 smithi083 bash[32381]: cluster 2024-09-10T19:10:49.835774+0000 osd.7 (osd.7) 505 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:10:51.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:48.964173+0000 osd.2 (osd.2) 375 : cluster [DBG] 2.e scrub starts 2024-09-10T19:10:51.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:48.968158+0000 osd.2 (osd.2) 376 : cluster [DBG] 2.e scrub ok 2024-09-10T19:10:51.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.251790+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.255131+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.652439+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.656265+0000 osd.4 (osd.4) 465 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.801720+0000 osd.6 (osd.6) 463 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.805514+0000 osd.6 (osd.6) 464 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.830854+0000 osd.7 (osd.7) 504 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:10:51.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:50 smithi165 bash[22591]: cluster 2024-09-10T19:10:49.835774+0000 osd.7 (osd.7) 505 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:10:51.700 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:51.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[22591]: cluster 2024-09-10T19:10:50.482347+0000 mgr.y (mgr.24526) 836 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:51.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[22591]: cluster 2024-09-10T19:10:50.680960+0000 osd.3 (osd.3) 518 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:10:51.701 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[22591]: cluster 2024-09-10T19:10:50.684928+0000 osd.3 (osd.3) 519 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[18629]: cluster 2024-09-10T19:10:50.482347+0000 mgr.y (mgr.24526) 836 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[18629]: cluster 2024-09-10T19:10:50.680960+0000 osd.3 (osd.3) 518 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[18629]: cluster 2024-09-10T19:10:50.684928+0000 osd.3 (osd.3) 519 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[18629]: cluster 2024-09-10T19:10:50.753723+0000 osd.6 (osd.6) 465 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[18629]: cluster 2024-09-10T19:10:50.757714+0000 osd.6 (osd.6) 466 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:10:52.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[32381]: cluster 2024-09-10T19:10:50.482347+0000 mgr.y (mgr.24526) 836 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:52.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[32381]: cluster 2024-09-10T19:10:50.680960+0000 osd.3 (osd.3) 518 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:10:52.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[32381]: cluster 2024-09-10T19:10:50.684928+0000 osd.3 (osd.3) 519 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:10:52.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[32381]: cluster 2024-09-10T19:10:50.753723+0000 osd.6 (osd.6) 465 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:10:52.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:51 smithi083 bash[32381]: cluster 2024-09-10T19:10:50.757714+0000 osd.6 (osd.6) 466 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:10:52.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[22591]: cluster 2024-09-10T19:10:50.753723+0000 osd.6 (osd.6) 465 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:10:52.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:51 smithi165 bash[22591]: cluster 2024-09-10T19:10:50.757714+0000 osd.6 (osd.6) 466 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:10:52.317 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:53.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:10:52] "GET /metrics HTTP/1.1" 200 224121 "" "Prometheus/2.43.0" 2024-09-10T19:10:53.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.319469+0000 osd.5 (osd.5) 498 : cluster [DBG] 6.d scrub starts 2024-09-10T19:10:53.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.323253+0000 osd.5 (osd.5) 499 : cluster [DBG] 6.d scrub ok 2024-09-10T19:10:53.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.590187+0000 osd.4 (osd.4) 466 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:10:53.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.593754+0000 osd.4 (osd.4) 467 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.735560+0000 osd.6 (osd.6) 467 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[32381]: cluster 2024-09-10T19:10:51.739262+0000 osd.6 (osd.6) 468 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.319469+0000 osd.5 (osd.5) 498 : cluster [DBG] 6.d scrub starts 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.323253+0000 osd.5 (osd.5) 499 : cluster [DBG] 6.d scrub ok 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.590187+0000 osd.4 (osd.4) 466 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.593754+0000 osd.4 (osd.4) 467 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.735560+0000 osd.6 (osd.6) 467 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:10:53.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:52 smithi083 bash[18629]: cluster 2024-09-10T19:10:51.739262+0000 osd.6 (osd.6) 468 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.319469+0000 osd.5 (osd.5) 498 : cluster [DBG] 6.d scrub starts 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.323253+0000 osd.5 (osd.5) 499 : cluster [DBG] 6.d scrub ok 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.590187+0000 osd.4 (osd.4) 466 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.593754+0000 osd.4 (osd.4) 467 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.735560+0000 osd.6 (osd.6) 467 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:10:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:52 smithi165 bash[22591]: cluster 2024-09-10T19:10:51.739262+0000 osd.6 (osd.6) 468 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[32381]: cluster 2024-09-10T19:10:52.301049+0000 osd.5 (osd.5) 500 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[32381]: cluster 2024-09-10T19:10:52.304274+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[32381]: cluster 2024-09-10T19:10:52.484146+0000 mgr.y (mgr.24526) 837 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[32381]: cluster 2024-09-10T19:10:52.825291+0000 osd.1 (osd.1) 63 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[32381]: cluster 2024-09-10T19:10:52.828875+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[18629]: cluster 2024-09-10T19:10:52.301049+0000 osd.5 (osd.5) 500 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[18629]: cluster 2024-09-10T19:10:52.304274+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[18629]: cluster 2024-09-10T19:10:52.484146+0000 mgr.y (mgr.24526) 837 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:53.965 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[18629]: cluster 2024-09-10T19:10:52.825291+0000 osd.1 (osd.1) 63 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:10:53.966 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:53 smithi083 bash[18629]: cluster 2024-09-10T19:10:52.828875+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:10:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:53 smithi165 bash[22591]: cluster 2024-09-10T19:10:52.301049+0000 osd.5 (osd.5) 500 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:10:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:53 smithi165 bash[22591]: cluster 2024-09-10T19:10:52.304274+0000 osd.5 (osd.5) 501 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:10:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:53 smithi165 bash[22591]: cluster 2024-09-10T19:10:52.484146+0000 mgr.y (mgr.24526) 837 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:10:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:53 smithi165 bash[22591]: cluster 2024-09-10T19:10:52.825291+0000 osd.1 (osd.1) 63 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:10:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:53 smithi165 bash[22591]: cluster 2024-09-10T19:10:52.828875+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.600859+0000 osd.4 (osd.4) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.603811+0000 osd.4 (osd.4) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.656591+0000 osd.0 (osd.0) 487 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.659451+0000 osd.0 (osd.0) 488 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.772848+0000 osd.7 (osd.7) 506 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:10:55.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[18629]: cluster 2024-09-10T19:10:53.776471+0000 osd.7 (osd.7) 507 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.600859+0000 osd.4 (osd.4) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.603811+0000 osd.4 (osd.4) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.656591+0000 osd.0 (osd.0) 487 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.659451+0000 osd.0 (osd.0) 488 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.772848+0000 osd.7 (osd.7) 506 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:10:55.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:54 smithi083 bash[32381]: cluster 2024-09-10T19:10:53.776471+0000 osd.7 (osd.7) 507 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.600859+0000 osd.4 (osd.4) 468 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.603811+0000 osd.4 (osd.4) 469 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.656591+0000 osd.0 (osd.0) 487 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.659451+0000 osd.0 (osd.0) 488 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.772848+0000 osd.7 (osd.7) 506 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:10:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:54 smithi165 bash[22591]: cluster 2024-09-10T19:10:53.776471+0000 osd.7 (osd.7) 507 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:10:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[32381]: cluster 2024-09-10T19:10:54.327213+0000 osd.5 (osd.5) 502 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:10:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[32381]: cluster 2024-09-10T19:10:54.332010+0000 osd.5 (osd.5) 503 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:10:56.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[32381]: cluster 2024-09-10T19:10:54.484872+0000 mgr.y (mgr.24526) 838 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-10T19:10:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[18629]: cluster 2024-09-10T19:10:54.327213+0000 osd.5 (osd.5) 502 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:10:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[18629]: cluster 2024-09-10T19:10:54.332010+0000 osd.5 (osd.5) 503 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:10:56.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:55 smithi083 bash[18629]: cluster 2024-09-10T19:10:54.484872+0000 mgr.y (mgr.24526) 838 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-10T19:10:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:55 smithi165 bash[22591]: cluster 2024-09-10T19:10:54.327213+0000 osd.5 (osd.5) 502 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:10:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:55 smithi165 bash[22591]: cluster 2024-09-10T19:10:54.332010+0000 osd.5 (osd.5) 503 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:10:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:55 smithi165 bash[22591]: cluster 2024-09-10T19:10:54.484872+0000 mgr.y (mgr.24526) 838 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-09-10T19:10:56.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- 172.21.15.83:0/1018923159 >> v1:172.21.15.83:6790/0 conn(0x7f72ec0fd5e0 legacy=0x7f72ec0fd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:56.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- 172.21.15.83:0/1018923159 shutdown_connections 2024-09-10T19:10:56.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- 172.21.15.83:0/1018923159 >> 172.21.15.83:0/1018923159 conn(0x7f72ec0f70d0 msgr2=0x7f72ec0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:56.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- 172.21.15.83:0/1018923159 shutdown_connections 2024-09-10T19:10:56.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- 172.21.15.83:0/1018923159 wait complete. 2024-09-10T19:10:56.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 librados: starting msgr at 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 librados: starting objecter 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 Processor -- start 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- start start 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 librados: setting wanted keys 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 librados: calling monclient init 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72ec11e670 con 0x7f72ec11a8c0 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72ec11f870 con 0x7f72ec108f10 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72f1951640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72ec120a70 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72eaffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f72ec0fd5e0 0x7f72ec1063c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56526/0 (socket says 172.21.15.83:56526) 2024-09-10T19:10:56.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72eaffd640 1 -- 172.21.15.83:0/2100277508 learned_addr learned my addr 172.21.15.83:0/2100277508 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:10:56.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter ms_handle_connect 0x7f72ec0fd5e0 2024-09-10T19:10:56.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878489176 0 0) 0x7f72ec120a70 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d4003120 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter ms_handle_connect 0x7f72ec11a8c0 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter ms_handle_connect 0x7f72ec108f10 2024-09-10T19:10:56.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.240+0000 7f72d37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809156092 0 0) 0x7f72ec11e670 con 0x7f72ec11a8c0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec120a70 con 0x7f72ec11a8c0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246222563 0 0) 0x7f72ec11f870 con 0x7f72ec108f10 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec11e670 con 0x7f72ec108f10 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 143400730 0 0) 0x7f72d4003120 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ec11f870 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f72e0002d50 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893274713 0 0) 0x7f72ec120a70 con 0x7f72ec11a8c0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d4003120 con 0x7f72ec11a8c0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f72e4001d70 con 0x7f72ec11a8c0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3374895714 0 0) 0x7f72ec11f870 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 >> v1:172.21.15.165:6789/0 conn(0x7f72ec108f10 legacy=0x7f72ec106b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:56.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 >> v1:172.21.15.83:6789/0 conn(0x7f72ec11a8c0 legacy=0x7f72ec11cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec121c70 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec11e860 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72f1951640 10 client.49985.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec11ecb0 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72f1951640 1 librados: init done 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f72e0002300 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f72e0003560 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d8005010 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.244+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f72e00017c0 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(879..879 src has 827..879) v4 ==== 7363+0+0 (unknown 488317071 0 0) 0x7f72e008a450 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 10 client.49985.objecter ms_dispatch 0x7f72ec000b90 osd_map(879..879 src has 827..879) v4 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 3 client.49985.objecter handle_osd_map got epochs [879,879] > 0 2024-09-10T19:10:56.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 3 client.49985.objecter handle_osd_map decoding full epoch 879 2024-09-10T19:10:56.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 20 client.49985.objecter dump_active .. 0 homeless 2024-09-10T19:10:56.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.248+0000 7f72d37fe640 10 client.49985.objecter ms_handle_connect 0x7f72d40716b0 2024-09-10T19:10:56.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.252+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f72e0056c90 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.440+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f72d80052c0 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.728+0000 7f72d37fe640 1 -- 172.21.15.83:0/2100277508 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v880) v1 ==== 113+0+0 (unknown 479177104 0 0) 0x7f72e0056630 con 0x7f72ec0fd5e0 2024-09-10T19:10:56.732 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T19:10:56.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 10 librados: watch_flush enter 2024-09-10T19:10:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 10 librados: watch_flush exit 2024-09-10T19:10:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 20 client.49985.objecter shutdown clearing up homeless session... 2024-09-10T19:10:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 10 client.49985.objecter successfully canceled tick 2024-09-10T19:10:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 >> v1:172.21.15.83:6800/3071793735 conn(0x7f72d40716b0 legacy=0x7f72d4073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:56.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 >> v1:172.21.15.83:6790/0 conn(0x7f72ec0fd5e0 legacy=0x7f72ec1063c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:10:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 shutdown_connections 2024-09-10T19:10:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 >> 172.21.15.83:0/2100277508 conn(0x7f72ec0f70d0 msgr2=0x7f72ec10c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:10:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 shutdown_connections 2024-09-10T19:10:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 -- 172.21.15.83:0/2100277508 wait complete. 2024-09-10T19:10:56.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:10:56.732+0000 7f72f1951640 1 librados: shutdown 2024-09-10T19:10:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: cluster 2024-09-10T19:10:55.796009+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:10:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: cluster 2024-09-10T19:10:55.799619+0000 osd.3 (osd.3) 521 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:10:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: cluster 2024-09-10T19:10:55.807273+0000 osd.1 (osd.1) 65 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: cluster 2024-09-10T19:10:55.810827+0000 osd.1 (osd.1) 66 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: audit 2024-09-10T19:10:56.442696+0000 mon.c (mon.2) 906 : audit [INF] from='client.? v1:172.21.15.83:0/2100277508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[32381]: audit 2024-09-10T19:10:56.443311+0000 mon.a (mon.0) 3844 : audit [INF] from='client.49985 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: cluster 2024-09-10T19:10:55.796009+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: cluster 2024-09-10T19:10:55.799619+0000 osd.3 (osd.3) 521 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:10:57.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: cluster 2024-09-10T19:10:55.807273+0000 osd.1 (osd.1) 65 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:10:57.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: cluster 2024-09-10T19:10:55.810827+0000 osd.1 (osd.1) 66 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:10:57.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: audit 2024-09-10T19:10:56.442696+0000 mon.c (mon.2) 906 : audit [INF] from='client.? v1:172.21.15.83:0/2100277508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:56 smithi083 bash[18629]: audit 2024-09-10T19:10:56.443311+0000 mon.a (mon.0) 3844 : audit [INF] from='client.49985 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: cluster 2024-09-10T19:10:55.796009+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: cluster 2024-09-10T19:10:55.799619+0000 osd.3 (osd.3) 521 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: cluster 2024-09-10T19:10:55.807273+0000 osd.1 (osd.1) 65 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: cluster 2024-09-10T19:10:55.810827+0000 osd.1 (osd.1) 66 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: audit 2024-09-10T19:10:56.442696+0000 mon.c (mon.2) 906 : audit [INF] from='client.? v1:172.21.15.83:0/2100277508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:56 smithi165 bash[22591]: audit 2024-09-10T19:10:56.443311+0000 mon.a (mon.0) 3844 : audit [INF] from='client.49985 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:10:57.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[18629]: cluster 2024-09-10T19:10:56.486246+0000 mgr.y (mgr.24526) 839 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[18629]: cluster 2024-09-10T19:10:56.708082+0000 osd.6 (osd.6) 469 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[18629]: cluster 2024-09-10T19:10:56.712085+0000 osd.6 (osd.6) 470 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[18629]: audit 2024-09-10T19:10:56.731061+0000 mon.a (mon.0) 3845 : audit [INF] from='client.49985 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[18629]: cluster 2024-09-10T19:10:56.731225+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-09-10T19:10:58.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[32381]: cluster 2024-09-10T19:10:56.486246+0000 mgr.y (mgr.24526) 839 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T19:10:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[32381]: cluster 2024-09-10T19:10:56.708082+0000 osd.6 (osd.6) 469 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:10:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[32381]: cluster 2024-09-10T19:10:56.712085+0000 osd.6 (osd.6) 470 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:10:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[32381]: audit 2024-09-10T19:10:56.731061+0000 mon.a (mon.0) 3845 : audit [INF] from='client.49985 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:10:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:57 smithi083 bash[32381]: cluster 2024-09-10T19:10:56.731225+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-09-10T19:10:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:57 smithi165 bash[22591]: cluster 2024-09-10T19:10:56.486246+0000 mgr.y (mgr.24526) 839 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 834 B/s rd, 0 op/s 2024-09-10T19:10:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:57 smithi165 bash[22591]: cluster 2024-09-10T19:10:56.708082+0000 osd.6 (osd.6) 469 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:10:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:57 smithi165 bash[22591]: cluster 2024-09-10T19:10:56.712085+0000 osd.6 (osd.6) 470 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:10:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:57 smithi165 bash[22591]: audit 2024-09-10T19:10:56.731061+0000 mon.a (mon.0) 3845 : audit [INF] from='client.49985 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:10:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:57 smithi165 bash[22591]: cluster 2024-09-10T19:10:56.731225+0000 mon.a (mon.0) 3846 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-09-10T19:10:58.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:10:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.044342+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.c scrub starts 2024-09-10T19:10:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.047598+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.c scrub ok 2024-09-10T19:10:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.503051+0000 osd.4 (osd.4) 470 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:10:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.506940+0000 osd.4 (osd.4) 471 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:10:58.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.727364+0000 osd.6 (osd.6) 471 : cluster [DBG] 4.a scrub starts 2024-09-10T19:10:58.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:58 smithi165 bash[22591]: cluster 2024-09-10T19:10:57.730668+0000 osd.6 (osd.6) 472 : cluster [DBG] 4.a scrub ok 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.044342+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.c scrub starts 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.047598+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.c scrub ok 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.503051+0000 osd.4 (osd.4) 470 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.506940+0000 osd.4 (osd.4) 471 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.727364+0000 osd.6 (osd.6) 471 : cluster [DBG] 4.a scrub starts 2024-09-10T19:10:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[18629]: cluster 2024-09-10T19:10:57.730668+0000 osd.6 (osd.6) 472 : cluster [DBG] 4.a scrub ok 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.044342+0000 osd.2 (osd.2) 377 : cluster [DBG] 2.c scrub starts 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.047598+0000 osd.2 (osd.2) 378 : cluster [DBG] 2.c scrub ok 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.503051+0000 osd.4 (osd.4) 470 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.506940+0000 osd.4 (osd.4) 471 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.727364+0000 osd.6 (osd.6) 471 : cluster [DBG] 4.a scrub starts 2024-09-10T19:10:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:58 smithi083 bash[32381]: cluster 2024-09-10T19:10:57.730668+0000 osd.6 (osd.6) 472 : cluster [DBG] 4.a scrub ok 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.454472+0000 osd.4 (osd.4) 472 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.458344+0000 osd.4 (osd.4) 473 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.486917+0000 mgr.y (mgr.24526) 840 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: audit 2024-09-10T19:10:58.596649+0000 mgr.y (mgr.24526) 841 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.832787+0000 osd.1 (osd.1) 67 : cluster [DBG] 2.d scrub starts 2024-09-10T19:11:00.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.836102+0000 osd.1 (osd.1) 68 : cluster [DBG] 2.d scrub ok 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.863466+0000 osd.7 (osd.7) 508 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.868468+0000 osd.7 (osd.7) 509 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.891830+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: cluster 2024-09-10T19:10:58.895223+0000 osd.3 (osd.3) 523 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[18629]: audit 2024-09-10T19:10:59.580911+0000 mon.c (mon.2) 907 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.454472+0000 osd.4 (osd.4) 472 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.458344+0000 osd.4 (osd.4) 473 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.486917+0000 mgr.y (mgr.24526) 840 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: audit 2024-09-10T19:10:58.596649+0000 mgr.y (mgr.24526) 841 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.832787+0000 osd.1 (osd.1) 67 : cluster [DBG] 2.d scrub starts 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.836102+0000 osd.1 (osd.1) 68 : cluster [DBG] 2.d scrub ok 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.863466+0000 osd.7 (osd.7) 508 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.868468+0000 osd.7 (osd.7) 509 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.891830+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: cluster 2024-09-10T19:10:58.895223+0000 osd.3 (osd.3) 523 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:11:00.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:10:59 smithi083 bash[32381]: audit 2024-09-10T19:10:59.580911+0000 mon.c (mon.2) 907 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.454472+0000 osd.4 (osd.4) 472 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.458344+0000 osd.4 (osd.4) 473 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.486917+0000 mgr.y (mgr.24526) 840 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: audit 2024-09-10T19:10:58.596649+0000 mgr.y (mgr.24526) 841 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.832787+0000 osd.1 (osd.1) 67 : cluster [DBG] 2.d scrub starts 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.836102+0000 osd.1 (osd.1) 68 : cluster [DBG] 2.d scrub ok 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.863466+0000 osd.7 (osd.7) 508 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.868468+0000 osd.7 (osd.7) 509 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.891830+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: cluster 2024-09-10T19:10:58.895223+0000 osd.3 (osd.3) 523 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:11:00.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:10:59 smithi165 bash[22591]: audit 2024-09-10T19:10:59.580911+0000 mon.c (mon.2) 907 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:59.838327+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[18629]: cluster 2024-09-10T19:10:59.841957+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.135076+0000 osd.5 (osd.5) 504 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.138370+0000 osd.5 (osd.5) 505 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:59.838327+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[32381]: cluster 2024-09-10T19:10:59.841957+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:11:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.135076+0000 osd.5 (osd.5) 504 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:11:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:00 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.138370+0000 osd.5 (osd.5) 505 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:11:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:59.838327+0000 osd.7 (osd.7) 510 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:11:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:00 smithi165 bash[22591]: cluster 2024-09-10T19:10:59.841957+0000 osd.7 (osd.7) 511 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:11:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:00 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.135076+0000 osd.5 (osd.5) 504 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:11:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:00 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.138370+0000 osd.5 (osd.5) 505 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:11:01.765 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.442440+0000 osd.4 (osd.4) 474 : cluster [DBG] 6.e scrub starts 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.446746+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.e scrub ok 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.487684+0000 mgr.y (mgr.24526) 842 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.887590+0000 osd.3 (osd.3) 524 : cluster [DBG] 12.b scrub starts 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:00.891252+0000 osd.3 (osd.3) 525 : cluster [DBG] 12.b scrub ok 2024-09-10T19:11:02.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.442440+0000 osd.4 (osd.4) 474 : cluster [DBG] 6.e scrub starts 2024-09-10T19:11:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.446746+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.e scrub ok 2024-09-10T19:11:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.487684+0000 mgr.y (mgr.24526) 842 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.887590+0000 osd.3 (osd.3) 524 : cluster [DBG] 12.b scrub starts 2024-09-10T19:11:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:00.891252+0000 osd.3 (osd.3) 525 : cluster [DBG] 12.b scrub ok 2024-09-10T19:11:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.442440+0000 osd.4 (osd.4) 474 : cluster [DBG] 6.e scrub starts 2024-09-10T19:11:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.446746+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.e scrub ok 2024-09-10T19:11:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.487684+0000 mgr.y (mgr.24526) 842 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:11:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.887590+0000 osd.3 (osd.3) 524 : cluster [DBG] 12.b scrub starts 2024-09-10T19:11:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:00.891252+0000 osd.3 (osd.3) 525 : cluster [DBG] 12.b scrub ok 2024-09-10T19:11:02.377 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:02.774 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:02] "GET /metrics HTTP/1.1" 200 224102 "" "Prometheus/2.43.0" 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.640188+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.642464+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.755820+0000 osd.6 (osd.6) 473 : cluster [DBG] 12.f scrub starts 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.758830+0000 osd.6 (osd.6) 474 : cluster [DBG] 12.f scrub ok 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.863753+0000 osd.7 (osd.7) 512 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.866849+0000 osd.7 (osd.7) 513 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.930978+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:01.935707+0000 osd.3 (osd.3) 527 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:02.148268+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:11:03.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[18629]: cluster 2024-09-10T19:11:02.154062+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.640188+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.642464+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.755820+0000 osd.6 (osd.6) 473 : cluster [DBG] 12.f scrub starts 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.758830+0000 osd.6 (osd.6) 474 : cluster [DBG] 12.f scrub ok 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.863753+0000 osd.7 (osd.7) 512 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.866849+0000 osd.7 (osd.7) 513 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.930978+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:01.935707+0000 osd.3 (osd.3) 527 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:02.148268+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:11:03.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:02 smithi083 bash[32381]: cluster 2024-09-10T19:11:02.154062+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.640188+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.642464+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.755820+0000 osd.6 (osd.6) 473 : cluster [DBG] 12.f scrub starts 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.758830+0000 osd.6 (osd.6) 474 : cluster [DBG] 12.f scrub ok 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.863753+0000 osd.7 (osd.7) 512 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:11:03.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.866849+0000 osd.7 (osd.7) 513 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:11:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.930978+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:11:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:01.935707+0000 osd.3 (osd.3) 527 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:11:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:02.148268+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.5 deep-scrub starts 2024-09-10T19:11:03.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:02 smithi165 bash[22591]: cluster 2024-09-10T19:11:02.154062+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.5 deep-scrub ok 2024-09-10T19:11:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:03 smithi083 bash[18629]: cluster 2024-09-10T19:11:02.489545+0000 mgr.y (mgr.24526) 843 : cluster [DBG] pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:03 smithi083 bash[32381]: cluster 2024-09-10T19:11:02.489545+0000 mgr.y (mgr.24526) 843 : cluster [DBG] pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:04.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:03 smithi165 bash[22591]: cluster 2024-09-10T19:11:02.489545+0000 mgr.y (mgr.24526) 843 : cluster [DBG] pgmap v1417: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[18629]: cluster 2024-09-10T19:11:03.718037+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:11:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[18629]: cluster 2024-09-10T19:11:03.721166+0000 osd.1 (osd.1) 70 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:11:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[18629]: cluster 2024-09-10T19:11:03.895332+0000 osd.3 (osd.3) 528 : cluster [DBG] 6.c scrub starts 2024-09-10T19:11:05.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[18629]: cluster 2024-09-10T19:11:03.898496+0000 osd.3 (osd.3) 529 : cluster [DBG] 6.c scrub ok 2024-09-10T19:11:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[32381]: cluster 2024-09-10T19:11:03.718037+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:11:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[32381]: cluster 2024-09-10T19:11:03.721166+0000 osd.1 (osd.1) 70 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:11:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[32381]: cluster 2024-09-10T19:11:03.895332+0000 osd.3 (osd.3) 528 : cluster [DBG] 6.c scrub starts 2024-09-10T19:11:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:04 smithi083 bash[32381]: cluster 2024-09-10T19:11:03.898496+0000 osd.3 (osd.3) 529 : cluster [DBG] 6.c scrub ok 2024-09-10T19:11:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:04 smithi165 bash[22591]: cluster 2024-09-10T19:11:03.718037+0000 osd.1 (osd.1) 69 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:11:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:04 smithi165 bash[22591]: cluster 2024-09-10T19:11:03.721166+0000 osd.1 (osd.1) 70 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:11:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:04 smithi165 bash[22591]: cluster 2024-09-10T19:11:03.895332+0000 osd.3 (osd.3) 528 : cluster [DBG] 6.c scrub starts 2024-09-10T19:11:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:04 smithi165 bash[22591]: cluster 2024-09-10T19:11:03.898496+0000 osd.3 (osd.3) 529 : cluster [DBG] 6.c scrub ok 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[18629]: cluster 2024-09-10T19:11:04.490355+0000 mgr.y (mgr.24526) 844 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[18629]: cluster 2024-09-10T19:11:04.706469+0000 osd.6 (osd.6) 475 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[18629]: cluster 2024-09-10T19:11:04.709793+0000 osd.6 (osd.6) 476 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[32381]: cluster 2024-09-10T19:11:04.490355+0000 mgr.y (mgr.24526) 844 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[32381]: cluster 2024-09-10T19:11:04.706469+0000 osd.6 (osd.6) 475 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:11:06.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:05 smithi083 bash[32381]: cluster 2024-09-10T19:11:04.709793+0000 osd.6 (osd.6) 476 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:11:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:05 smithi165 bash[22591]: cluster 2024-09-10T19:11:04.490355+0000 mgr.y (mgr.24526) 844 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:05 smithi165 bash[22591]: cluster 2024-09-10T19:11:04.706469+0000 osd.6 (osd.6) 475 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:11:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:05 smithi165 bash[22591]: cluster 2024-09-10T19:11:04.709793+0000 osd.6 (osd.6) 476 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:11:06.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/2799486853 >> v1:172.21.15.83:6789/0 conn(0x7fe0a8105210 legacy=0x7fe0a81075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/2799486853 shutdown_connections 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/2799486853 >> 172.21.15.83:0/2799486853 conn(0x7fe0a80f8000 msgr2=0x7fe0a80fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/2799486853 shutdown_connections 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/2799486853 wait complete. 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 librados: starting msgr at 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 librados: starting objecter 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 Processor -- start 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- start start 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 librados: setting wanted keys 2024-09-10T19:11:06.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 librados: calling monclient init 2024-09-10T19:11:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0a8108970 con 0x7fe0a8105210 2024-09-10T19:11:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0a8108b60 con 0x7fe0a8108d70 2024-09-10T19:11:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe0a8120600 con 0x7fe0a81024a0 2024-09-10T19:11:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fe0a8105210 0x7fe0a8119790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:33598/0 (socket says 172.21.15.83:33598) 2024-09-10T19:11:06.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09ffff640 1 -- 172.21.15.83:0/1019111463 learned_addr learned my addr 172.21.15.83:0/1019111463 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a6ffd640 1 --1- 172.21.15.83:0/1019111463 >> v1:172.21.15.83:6790/0 conn(0x7fe0a81024a0 0x7fe0a8071110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:56554/0 (socket says 172.21.15.83:56554) 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0a81024a0 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0a8105210 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter ms_handle_connect 0x7fe0a8108d70 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151215848 0 0) 0x7fe0a8108970 con 0x7fe0a8105210 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe094003120 con 0x7fe0a8105210 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013014428 0 0) 0x7fe0a8120600 con 0x7fe0a81024a0 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a8108970 con 0x7fe0a81024a0 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907490769 0 0) 0x7fe0a8108b60 con 0x7fe0a8108d70 2024-09-10T19:11:06.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a8120600 con 0x7fe0a8108d70 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400427209 0 0) 0x7fe094003120 con 0x7fe0a8105210 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a8108b60 con 0x7fe0a8105210 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe090001550 con 0x7fe0a8105210 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2943367959 0 0) 0x7fe0a8120600 con 0x7fe0a8108d70 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe094003120 con 0x7fe0a8108d70 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33413137 0 0) 0x7fe0a8108970 con 0x7fe0a81024a0 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a8120600 con 0x7fe0a81024a0 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe0a0001bc0 con 0x7fe0a8108d70 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe098001830 con 0x7fe0a81024a0 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2808099788 0 0) 0x7fe0a8108b60 con 0x7fe0a8105210 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 >> v1:172.21.15.83:6790/0 conn(0x7fe0a81024a0 legacy=0x7fe0a8071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 >> v1:172.21.15.165:6789/0 conn(0x7fe0a8108d70 legacy=0x7fe0a811cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:06.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a81217e0 con 0x7fe0a8105210 2024-09-10T19:11:06.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a81207f0 con 0x7fe0a8105210 2024-09-10T19:11:06.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 10 client.25905.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:06.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a8120c40 con 0x7fe0a8105210 2024-09-10T19:11:06.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.159+0000 7fe0a7fff640 1 librados: init done 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fe090001230 con 0x7fe0a8105210 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fe090003580 con 0x7fe0a8105210 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fe0900037c0 con 0x7fe0a8105210 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a80fdba0 con 0x7fe0a8105210 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(880..880 src has 827..880) v4 ==== 7363+0+0 (unknown 1616900600 0 0) 0x7fe09008b320 con 0x7fe0a8105210 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 10 client.25905.objecter ms_dispatch 0x7fe0a8000b90 osd_map(880..880 src has 827..880) v4 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 3 client.25905.objecter handle_osd_map got epochs [880,880] > 0 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 3 client.25905.objecter handle_osd_map decoding full epoch 880 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 20 client.25905.objecter dump_active .. 0 homeless 2024-09-10T19:11:06.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.163+0000 7fe09f7fe640 10 client.25905.objecter ms_handle_connect 0x7fe0940718c0 2024-09-10T19:11:06.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.171+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fe090057b60 con 0x7fe0a8105210 2024-09-10T19:11:06.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.355+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fe0a805ed70 con 0x7fe0a8105210 2024-09-10T19:11:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.791+0000 7fe09f7fe640 1 -- 172.21.15.83:0/1019111463 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v881) v1 ==== 133+0+0 (unknown 4201296484 0 0) 0x7fe090057500 con 0x7fe0a8105210 2024-09-10T19:11:06.798 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 10 librados: watch_flush enter 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 10 librados: watch_flush exit 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 20 client.25905.objecter shutdown clearing up homeless session... 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 10 client.25905.objecter successfully canceled tick 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 >> v1:172.21.15.83:6800/3071793735 conn(0x7fe0940718c0 legacy=0x7fe094073d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:06.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.795+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 >> v1:172.21.15.83:6789/0 conn(0x7fe0a8105210 legacy=0x7fe0a8119790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.799+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 shutdown_connections 2024-09-10T19:11:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.799+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 >> 172.21.15.83:0/1019111463 conn(0x7fe0a80f8000 msgr2=0x7fe0a80fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.799+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 shutdown_connections 2024-09-10T19:11:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.799+0000 7fe0a7fff640 1 -- 172.21.15.83:0/1019111463 wait complete. 2024-09-10T19:11:06.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:06.799+0000 7fe0a7fff640 1 librados: shutdown 2024-09-10T19:11:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:06 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.102024+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:11:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:06 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.105829+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:11:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:06 smithi165 bash[22591]: audit 2024-09-10T19:11:06.359887+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.102024+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.105829+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[32381]: audit 2024-09-10T19:11:06.359887+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.102024+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.105829+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:11:07.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:06 smithi083 bash[18629]: audit 2024-09-10T19:11:06.359887+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:11:07.664 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.086637+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.f scrub starts 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.089698+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.f scrub ok 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.306534+0000 osd.4 (osd.4) 476 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.310694+0000 osd.4 (osd.4) 477 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.492292+0000 mgr.y (mgr.24526) 845 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: audit 2024-09-10T19:11:06.796963+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:11:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: cluster 2024-09-10T19:11:06.797075+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: audit 2024-09-10T19:11:07.262843+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: audit 2024-09-10T19:11:07.571649+0000 mon.a (mon.0) 3850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[18629]: audit 2024-09-10T19:11:07.581467+0000 mon.a (mon.0) 3851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.086637+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.f scrub starts 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.089698+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.f scrub ok 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.306534+0000 osd.4 (osd.4) 476 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.310694+0000 osd.4 (osd.4) 477 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.492292+0000 mgr.y (mgr.24526) 845 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: audit 2024-09-10T19:11:06.796963+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: cluster 2024-09-10T19:11:06.797075+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: audit 2024-09-10T19:11:07.262843+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: audit 2024-09-10T19:11:07.571649+0000 mon.a (mon.0) 3850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:07 smithi083 bash[32381]: audit 2024-09-10T19:11:07.581467+0000 mon.a (mon.0) 3851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.086637+0000 osd.2 (osd.2) 379 : cluster [DBG] 6.f scrub starts 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.089698+0000 osd.2 (osd.2) 380 : cluster [DBG] 6.f scrub ok 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.306534+0000 osd.4 (osd.4) 476 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.310694+0000 osd.4 (osd.4) 477 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.492292+0000 mgr.y (mgr.24526) 845 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: audit 2024-09-10T19:11:06.796963+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? v1:172.21.15.83:0/1019111463' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: cluster 2024-09-10T19:11:06.797075+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-09-10T19:11:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: audit 2024-09-10T19:11:07.262843+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: audit 2024-09-10T19:11:07.571649+0000 mon.a (mon.0) 3850 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:07 smithi165 bash[22591]: audit 2024-09-10T19:11:07.581467+0000 mon.a (mon.0) 3851 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: audit 2024-09-10T19:11:08.049853+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: audit 2024-09-10T19:11:08.051230+0000 mon.c (mon.2) 910 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.051742+0000 osd.5 (osd.5) 510 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T19:11:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.057004+0000 osd.5 (osd.5) 511 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T19:11:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: audit 2024-09-10T19:11:08.061772+0000 mon.a (mon.0) 3852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.074511+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:11:08.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:08 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.077202+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: audit 2024-09-10T19:11:08.049853+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: audit 2024-09-10T19:11:08.051230+0000 mon.c (mon.2) 910 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.051742+0000 osd.5 (osd.5) 510 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.057004+0000 osd.5 (osd.5) 511 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: audit 2024-09-10T19:11:08.061772+0000 mon.a (mon.0) 3852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.074511+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:11:09.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.077202+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: audit 2024-09-10T19:11:08.049853+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: audit 2024-09-10T19:11:08.051230+0000 mon.c (mon.2) 910 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.051742+0000 osd.5 (osd.5) 510 : cluster [DBG] 3.c deep-scrub starts 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.057004+0000 osd.5 (osd.5) 511 : cluster [DBG] 3.c deep-scrub ok 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: audit 2024-09-10T19:11:08.061772+0000 mon.a (mon.0) 3852 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.074511+0000 osd.2 (osd.2) 381 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:11:09.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:08 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.077202+0000 osd.2 (osd.2) 382 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:11:10.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:10 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.493083+0000 mgr.y (mgr.24526) 846 : cluster [DBG] pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:10.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:10 smithi165 bash[22591]: audit 2024-09-10T19:11:08.608044+0000 mgr.y (mgr.24526) 847 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:10.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:10 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.628632+0000 osd.6 (osd.6) 477 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:11:10.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:10 smithi165 bash[22591]: cluster 2024-09-10T19:11:08.632457+0000 osd.6 (osd.6) 478 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:11:10.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.493083+0000 mgr.y (mgr.24526) 846 : cluster [DBG] pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:10.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[18629]: audit 2024-09-10T19:11:08.608044+0000 mgr.y (mgr.24526) 847 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:10.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.628632+0000 osd.6 (osd.6) 477 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:11:10.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[18629]: cluster 2024-09-10T19:11:08.632457+0000 osd.6 (osd.6) 478 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:11:10.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.493083+0000 mgr.y (mgr.24526) 846 : cluster [DBG] pgmap v1421: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:10.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[32381]: audit 2024-09-10T19:11:08.608044+0000 mgr.y (mgr.24526) 847 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:10.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.628632+0000 osd.6 (osd.6) 477 : cluster [DBG] 1.7 scrub starts 2024-09-10T19:11:10.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:10 smithi083 bash[32381]: cluster 2024-09-10T19:11:08.632457+0000 osd.6 (osd.6) 478 : cluster [DBG] 1.7 scrub ok 2024-09-10T19:11:11.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:11 smithi165 bash[22591]: cluster 2024-09-10T19:11:10.006870+0000 osd.5 (osd.5) 512 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:11:11.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:11 smithi165 bash[22591]: cluster 2024-09-10T19:11:10.012315+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:11:11.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:11 smithi165 bash[22591]: cluster 2024-09-10T19:11:10.765276+0000 osd.1 (osd.1) 71 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:11:11.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:11 smithi165 bash[22591]: cluster 2024-09-10T19:11:10.768666+0000 osd.1 (osd.1) 72 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[18629]: cluster 2024-09-10T19:11:10.006870+0000 osd.5 (osd.5) 512 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[18629]: cluster 2024-09-10T19:11:10.012315+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[18629]: cluster 2024-09-10T19:11:10.765276+0000 osd.1 (osd.1) 71 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[18629]: cluster 2024-09-10T19:11:10.768666+0000 osd.1 (osd.1) 72 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[32381]: cluster 2024-09-10T19:11:10.006870+0000 osd.5 (osd.5) 512 : cluster [DBG] 3.1c scrub starts 2024-09-10T19:11:11.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[32381]: cluster 2024-09-10T19:11:10.012315+0000 osd.5 (osd.5) 513 : cluster [DBG] 3.1c scrub ok 2024-09-10T19:11:11.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[32381]: cluster 2024-09-10T19:11:10.765276+0000 osd.1 (osd.1) 71 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:11:11.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:11 smithi083 bash[32381]: cluster 2024-09-10T19:11:10.768666+0000 osd.1 (osd.1) 72 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:11:11.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:12.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:12 smithi165 bash[22591]: cluster 2024-09-10T19:11:10.493870+0000 mgr.y (mgr.24526) 848 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:12.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:12 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.018363+0000 osd.5 (osd.5) 514 : cluster [DBG] 6.a scrub starts 2024-09-10T19:11:12.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:12 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.021561+0000 osd.5 (osd.5) 515 : cluster [DBG] 6.a scrub ok 2024-09-10T19:11:12.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:12 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.812137+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 deep-scrub starts 2024-09-10T19:11:12.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:12 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.815575+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 deep-scrub ok 2024-09-10T19:11:12.434 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18629]: cluster 2024-09-10T19:11:10.493870+0000 mgr.y (mgr.24526) 848 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.018363+0000 osd.5 (osd.5) 514 : cluster [DBG] 6.a scrub starts 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.021561+0000 osd.5 (osd.5) 515 : cluster [DBG] 6.a scrub ok 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.812137+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 deep-scrub starts 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.815575+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 deep-scrub ok 2024-09-10T19:11:12.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[32381]: cluster 2024-09-10T19:11:10.493870+0000 mgr.y (mgr.24526) 848 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.018363+0000 osd.5 (osd.5) 514 : cluster [DBG] 6.a scrub starts 2024-09-10T19:11:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.021561+0000 osd.5 (osd.5) 515 : cluster [DBG] 6.a scrub ok 2024-09-10T19:11:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.812137+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 deep-scrub starts 2024-09-10T19:11:12.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.815575+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 deep-scrub ok 2024-09-10T19:11:13.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:12] "GET /metrics HTTP/1.1" 200 224102 "" "Prometheus/2.43.0" 2024-09-10T19:11:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.941146+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.1e deep-scrub starts 2024-09-10T19:11:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:11.948992+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.1e deep-scrub ok 2024-09-10T19:11:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.214534+0000 osd.4 (osd.4) 478 : cluster [DBG] 12.0 deep-scrub starts 2024-09-10T19:11:13.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.218379+0000 osd.4 (osd.4) 479 : cluster [DBG] 12.0 deep-scrub ok 2024-09-10T19:11:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.530642+0000 osd.0 (osd.0) 491 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:11:13.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:13 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.534121+0000 osd.0 (osd.0) 492 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.941146+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.1e deep-scrub starts 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:11.948992+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.1e deep-scrub ok 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.214534+0000 osd.4 (osd.4) 478 : cluster [DBG] 12.0 deep-scrub starts 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.218379+0000 osd.4 (osd.4) 479 : cluster [DBG] 12.0 deep-scrub ok 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.530642+0000 osd.0 (osd.0) 491 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:11:13.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.534121+0000 osd.0 (osd.0) 492 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.941146+0000 osd.3 (osd.3) 530 : cluster [DBG] 3.1e deep-scrub starts 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:11.948992+0000 osd.3 (osd.3) 531 : cluster [DBG] 3.1e deep-scrub ok 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.214534+0000 osd.4 (osd.4) 478 : cluster [DBG] 12.0 deep-scrub starts 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.218379+0000 osd.4 (osd.4) 479 : cluster [DBG] 12.0 deep-scrub ok 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.530642+0000 osd.0 (osd.0) 491 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:11:13.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:13 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.534121+0000 osd.0 (osd.0) 492 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.495893+0000 mgr.y (mgr.24526) 849 : cluster [DBG] pgmap v1423: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.973329+0000 osd.5 (osd.5) 516 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[18629]: cluster 2024-09-10T19:11:12.976691+0000 osd.5 (osd.5) 517 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.495893+0000 mgr.y (mgr.24526) 849 : cluster [DBG] pgmap v1423: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.973329+0000 osd.5 (osd.5) 516 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:11:14.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:14 smithi083 bash[32381]: cluster 2024-09-10T19:11:12.976691+0000 osd.5 (osd.5) 517 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:11:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:14 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.495893+0000 mgr.y (mgr.24526) 849 : cluster [DBG] pgmap v1423: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:14 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.973329+0000 osd.5 (osd.5) 516 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:11:14.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:14 smithi165 bash[22591]: cluster 2024-09-10T19:11:12.976691+0000 osd.5 (osd.5) 517 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:11:15.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:15 smithi165 bash[22591]: cluster 2024-09-10T19:11:13.945100+0000 osd.3 (osd.3) 532 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:11:15.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:15 smithi165 bash[22591]: cluster 2024-09-10T19:11:13.949112+0000 osd.3 (osd.3) 533 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:11:15.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:15 smithi165 bash[22591]: cluster 2024-09-10T19:11:13.999353+0000 osd.5 (osd.5) 518 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:11:15.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:15 smithi165 bash[22591]: cluster 2024-09-10T19:11:14.002663+0000 osd.5 (osd.5) 519 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:11:15.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:15 smithi165 bash[22591]: audit 2024-09-10T19:11:14.581085+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:15.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[18629]: cluster 2024-09-10T19:11:13.945100+0000 osd.3 (osd.3) 532 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:11:15.584 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[18629]: cluster 2024-09-10T19:11:13.949112+0000 osd.3 (osd.3) 533 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[18629]: cluster 2024-09-10T19:11:13.999353+0000 osd.5 (osd.5) 518 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[18629]: cluster 2024-09-10T19:11:14.002663+0000 osd.5 (osd.5) 519 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[18629]: audit 2024-09-10T19:11:14.581085+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[32381]: cluster 2024-09-10T19:11:13.945100+0000 osd.3 (osd.3) 532 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[32381]: cluster 2024-09-10T19:11:13.949112+0000 osd.3 (osd.3) 533 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[32381]: cluster 2024-09-10T19:11:13.999353+0000 osd.5 (osd.5) 518 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[32381]: cluster 2024-09-10T19:11:14.002663+0000 osd.5 (osd.5) 519 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:11:15.585 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:15 smithi083 bash[32381]: audit 2024-09-10T19:11:14.581085+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:16 smithi165 bash[22591]: cluster 2024-09-10T19:11:14.496714+0000 mgr.y (mgr.24526) 850 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:16 smithi165 bash[22591]: cluster 2024-09-10T19:11:14.606087+0000 osd.6 (osd.6) 479 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:11:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:16 smithi165 bash[22591]: cluster 2024-09-10T19:11:14.609152+0000 osd.6 (osd.6) 480 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:11:16.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:16 smithi165 bash[22591]: cluster 2024-09-10T19:11:14.996976+0000 osd.5 (osd.5) 520 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:11:16.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:16 smithi165 bash[22591]: cluster 2024-09-10T19:11:15.000916+0000 osd.5 (osd.5) 521 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:11:16.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- 172.21.15.83:0/3145801067 >> v1:172.21.15.83:6789/0 conn(0x7f44e4108ef0 legacy=0x7f44e410b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- 172.21.15.83:0/3145801067 shutdown_connections 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- 172.21.15.83:0/3145801067 >> 172.21.15.83:0/3145801067 conn(0x7f44e40f81a0 msgr2=0x7f44e40fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- 172.21.15.83:0/3145801067 shutdown_connections 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- 172.21.15.83:0/3145801067 wait complete. 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 librados: starting msgr at 2024-09-10T19:11:16.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 librados: starting objecter 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 Processor -- start 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 -- start start 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 librados: setting wanted keys 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.411+0000 7f44ea7bd640 1 librados: calling monclient init 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f44e4113240 con 0x7f44e4108ef0 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f44e411d3a0 con 0x7f44e41025a0 2024-09-10T19:11:16.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f44e411e540 con 0x7f44e4105360 2024-09-10T19:11:16.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e37fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f44e4105360 0x7f44e4117540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55152/0 (socket says 172.21.15.83:55152) 2024-09-10T19:11:16.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e37fe640 1 -- 172.21.15.83:0/2606056886 learned_addr learned my addr 172.21.15.83:0/2606056886 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter ms_handle_connect 0x7f44e4105360 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter ms_handle_connect 0x7f44e41025a0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter ms_handle_connect 0x7f44e4108ef0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21451498 0 0) 0x7f44e411e540 con 0x7f44e4105360 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d00034e0 con 0x7f44e4105360 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147838677 0 0) 0x7f44e411d3a0 con 0x7f44e41025a0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e411e540 con 0x7f44e41025a0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518518298 0 0) 0x7f44e4113240 con 0x7f44e4108ef0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e411d3a0 con 0x7f44e4108ef0 2024-09-10T19:11:16.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2664099490 0 0) 0x7f44d00034e0 con 0x7f44e4105360 2024-09-10T19:11:16.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e4113240 con 0x7f44e4105360 2024-09-10T19:11:16.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f44cc002990 con 0x7f44e4105360 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641301966 0 0) 0x7f44e411d3a0 con 0x7f44e4108ef0 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44d00034e0 con 0x7f44e4108ef0 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860634720 0 0) 0x7f44e411e540 con 0x7f44e41025a0 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e411d3a0 con 0x7f44e41025a0 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1188841089 0 0) 0x7f44e4113240 con 0x7f44e4105360 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 >> v1:172.21.15.165:6789/0 conn(0x7f44e41025a0 legacy=0x7f44e4112720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 >> v1:172.21.15.83:6789/0 conn(0x7f44e4108ef0 legacy=0x7f44e411ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e411f6e0 con 0x7f44e4105360 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e411d590 con 0x7f44e4105360 2024-09-10T19:11:16.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 10 client.49997.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e411d9e0 con 0x7f44e4105360 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 librados: init done 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f44cc001810 con 0x7f44e4105360 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e40fdd20 con 0x7f44e4105360 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.415+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f44cc005a80 con 0x7f44e4105360 2024-09-10T19:11:16.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f44cc01bf80 con 0x7f44e4105360 2024-09-10T19:11:16.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(882..882 src has 827..882) v4 ==== 7363+0+0 (unknown 302438453 0 0) 0x7f44cc08a4c0 con 0x7f44e4105360 2024-09-10T19:11:16.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 10 client.49997.objecter ms_dispatch 0x7f44e4000b90 osd_map(882..882 src has 827..882) v4 2024-09-10T19:11:16.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 3 client.49997.objecter handle_osd_map got epochs [882,882] > 0 2024-09-10T19:11:16.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 3 client.49997.objecter handle_osd_map decoding full epoch 882 2024-09-10T19:11:16.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 20 client.49997.objecter dump_active .. 0 homeless 2024-09-10T19:11:16.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.419+0000 7f44e0ff9640 10 client.49997.objecter ms_handle_connect 0x7f44d0071760 2024-09-10T19:11:16.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.423+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f44cc001d60 con 0x7f44e4105360 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[18629]: cluster 2024-09-10T19:11:14.496714+0000 mgr.y (mgr.24526) 850 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[18629]: cluster 2024-09-10T19:11:14.606087+0000 osd.6 (osd.6) 479 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[18629]: cluster 2024-09-10T19:11:14.609152+0000 osd.6 (osd.6) 480 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[18629]: cluster 2024-09-10T19:11:14.996976+0000 osd.5 (osd.5) 520 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[18629]: cluster 2024-09-10T19:11:15.000916+0000 osd.5 (osd.5) 521 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[32381]: cluster 2024-09-10T19:11:14.496714+0000 mgr.y (mgr.24526) 850 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[32381]: cluster 2024-09-10T19:11:14.606087+0000 osd.6 (osd.6) 479 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:11:16.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[32381]: cluster 2024-09-10T19:11:14.609152+0000 osd.6 (osd.6) 480 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:11:16.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[32381]: cluster 2024-09-10T19:11:14.996976+0000 osd.5 (osd.5) 520 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:11:16.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:16 smithi083 bash[32381]: cluster 2024-09-10T19:11:15.000916+0000 osd.5 (osd.5) 521 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:11:16.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:16.607+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f44e410e9a0 con 0x7f44e4105360 2024-09-10T19:11:17.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.119+0000 7f44e0ff9640 1 -- 172.21.15.83:0/2606056886 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v883) v1 ==== 113+0+0 (unknown 3884755772 0 0) 0x7f44cc056940 con 0x7f44e4105360 2024-09-10T19:11:17.125 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T19:11:17.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 10 librados: watch_flush enter 2024-09-10T19:11:17.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 10 librados: watch_flush exit 2024-09-10T19:11:17.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 20 client.49997.objecter shutdown clearing up homeless session... 2024-09-10T19:11:17.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 10 client.49997.objecter successfully canceled tick 2024-09-10T19:11:17.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 >> v1:172.21.15.83:6800/3071793735 conn(0x7f44d0071760 legacy=0x7f44d0073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:17.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 >> v1:172.21.15.83:6790/0 conn(0x7f44e4105360 legacy=0x7f44e4117540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:17.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 shutdown_connections 2024-09-10T19:11:17.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 >> 172.21.15.83:0/2606056886 conn(0x7f44e40f81a0 msgr2=0x7f44e40f8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:17.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 shutdown_connections 2024-09-10T19:11:17.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 -- 172.21.15.83:0/2606056886 wait complete. 2024-09-10T19:11:17.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:17.123+0000 7f44ea7bd640 1 librados: shutdown 2024-09-10T19:11:17.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:15.548063+0000 osd.7 (osd.7) 514 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:17.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:15.548518+0000 osd.7 (osd.7) 515 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:17.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:15.796916+0000 osd.7 (osd.7) 516 : cluster [DBG] 2.b scrub starts 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:15.800307+0000 osd.7 (osd.7) 517 : cluster [DBG] 2.b scrub ok 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:16.119255+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: audit 2024-09-10T19:11:16.614077+0000 mon.c (mon.2) 912 : audit [INF] from='client.? v1:172.21.15.83:0/2606056886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: audit 2024-09-10T19:11:16.614710+0000 mon.a (mon.0) 3854 : audit [INF] from='client.49997 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:16.837290+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:11:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[18629]: cluster 2024-09-10T19:11:16.841182+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:15.548063+0000 osd.7 (osd.7) 514 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:15.548518+0000 osd.7 (osd.7) 515 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:15.796916+0000 osd.7 (osd.7) 516 : cluster [DBG] 2.b scrub starts 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:15.800307+0000 osd.7 (osd.7) 517 : cluster [DBG] 2.b scrub ok 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:16.119255+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: audit 2024-09-10T19:11:16.614077+0000 mon.c (mon.2) 912 : audit [INF] from='client.? v1:172.21.15.83:0/2606056886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: audit 2024-09-10T19:11:16.614710+0000 mon.a (mon.0) 3854 : audit [INF] from='client.49997 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:16.837290+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:11:17.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:17 smithi083 bash[32381]: cluster 2024-09-10T19:11:16.841182+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:15.548063+0000 osd.7 (osd.7) 514 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:15.548518+0000 osd.7 (osd.7) 515 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:15.796916+0000 osd.7 (osd.7) 516 : cluster [DBG] 2.b scrub starts 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:15.800307+0000 osd.7 (osd.7) 517 : cluster [DBG] 2.b scrub ok 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:16.119255+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-09-10T19:11:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: audit 2024-09-10T19:11:16.614077+0000 mon.c (mon.2) 912 : audit [INF] from='client.? v1:172.21.15.83:0/2606056886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: audit 2024-09-10T19:11:16.614710+0000 mon.a (mon.0) 3854 : audit [INF] from='client.49997 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:16.837290+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:11:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:17 smithi165 bash[22591]: cluster 2024-09-10T19:11:16.841182+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:11:17.995 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[18629]: cluster 2024-09-10T19:11:16.497474+0000 mgr.y (mgr.24526) 851 : cluster [DBG] pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[18629]: audit 2024-09-10T19:11:17.118565+0000 mon.a (mon.0) 3855 : audit [INF] from='client.49997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[18629]: cluster 2024-09-10T19:11:17.118854+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[18629]: cluster 2024-09-10T19:11:17.864440+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[18629]: cluster 2024-09-10T19:11:17.868344+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[32381]: cluster 2024-09-10T19:11:16.497474+0000 mgr.y (mgr.24526) 851 : cluster [DBG] pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[32381]: audit 2024-09-10T19:11:17.118565+0000 mon.a (mon.0) 3855 : audit [INF] from='client.49997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:18.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[32381]: cluster 2024-09-10T19:11:17.118854+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-09-10T19:11:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[32381]: cluster 2024-09-10T19:11:17.864440+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:11:18.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:18 smithi083 bash[32381]: cluster 2024-09-10T19:11:17.868344+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:11:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[22591]: cluster 2024-09-10T19:11:16.497474+0000 mgr.y (mgr.24526) 851 : cluster [DBG] pgmap v1426: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:11:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[22591]: audit 2024-09-10T19:11:17.118565+0000 mon.a (mon.0) 3855 : audit [INF] from='client.49997 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[22591]: cluster 2024-09-10T19:11:17.118854+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-09-10T19:11:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[22591]: cluster 2024-09-10T19:11:17.864440+0000 osd.1 (osd.1) 77 : cluster [DBG] 2.9 deep-scrub starts 2024-09-10T19:11:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[22591]: cluster 2024-09-10T19:11:17.868344+0000 osd.1 (osd.1) 78 : cluster [DBG] 2.9 deep-scrub ok 2024-09-10T19:11:19.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:19 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.427052+0000 osd.0 (osd.0) 493 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:11:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:19 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.430630+0000 osd.0 (osd.0) 494 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:11:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:19 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.427052+0000 osd.0 (osd.0) 493 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:11:19.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:19 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.430630+0000 osd.0 (osd.0) 494 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:11:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:19 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.427052+0000 osd.0 (osd.0) 493 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:11:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:19 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.430630+0000 osd.0 (osd.0) 494 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.309144+0000 osd.4 (osd.4) 480 : cluster [DBG] 2.f scrub starts 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.313852+0000 osd.4 (osd.4) 481 : cluster [DBG] 2.f scrub ok 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.498299+0000 mgr.y (mgr.24526) 852 : cluster [DBG] pgmap v1428: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: audit 2024-09-10T19:11:18.616498+0000 mgr.y (mgr.24526) 853 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.941672+0000 osd.3 (osd.3) 534 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[18629]: cluster 2024-09-10T19:11:18.946724+0000 osd.3 (osd.3) 535 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.309144+0000 osd.4 (osd.4) 480 : cluster [DBG] 2.f scrub starts 2024-09-10T19:11:20.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.313852+0000 osd.4 (osd.4) 481 : cluster [DBG] 2.f scrub ok 2024-09-10T19:11:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.498299+0000 mgr.y (mgr.24526) 852 : cluster [DBG] pgmap v1428: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: audit 2024-09-10T19:11:18.616498+0000 mgr.y (mgr.24526) 853 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.941672+0000 osd.3 (osd.3) 534 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:11:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:20 smithi083 bash[32381]: cluster 2024-09-10T19:11:18.946724+0000 osd.3 (osd.3) 535 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.309144+0000 osd.4 (osd.4) 480 : cluster [DBG] 2.f scrub starts 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.313852+0000 osd.4 (osd.4) 481 : cluster [DBG] 2.f scrub ok 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.498299+0000 mgr.y (mgr.24526) 852 : cluster [DBG] pgmap v1428: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: audit 2024-09-10T19:11:18.616498+0000 mgr.y (mgr.24526) 853 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.941672+0000 osd.3 (osd.3) 534 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:11:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:20 smithi165 bash[22591]: cluster 2024-09-10T19:11:18.946724+0000 osd.3 (osd.3) 535 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[18629]: cluster 2024-09-10T19:11:19.983854+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[18629]: cluster 2024-09-10T19:11:19.987956+0000 osd.3 (osd.3) 537 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[18629]: cluster 2024-09-10T19:11:20.407386+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[18629]: cluster 2024-09-10T19:11:20.411047+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[32381]: cluster 2024-09-10T19:11:19.983854+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[32381]: cluster 2024-09-10T19:11:19.987956+0000 osd.3 (osd.3) 537 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:11:21.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[32381]: cluster 2024-09-10T19:11:20.407386+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:11:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:21 smithi083 bash[32381]: cluster 2024-09-10T19:11:20.411047+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:11:21.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:21 smithi165 bash[22591]: cluster 2024-09-10T19:11:19.983854+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:11:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:21 smithi165 bash[22591]: cluster 2024-09-10T19:11:19.987956+0000 osd.3 (osd.3) 537 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:11:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:21 smithi165 bash[22591]: cluster 2024-09-10T19:11:20.407386+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:11:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:21 smithi165 bash[22591]: cluster 2024-09-10T19:11:20.411047+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18629]: cluster 2024-09-10T19:11:20.499074+0000 mgr.y (mgr.24526) 854 : cluster [DBG] pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.421231+0000 osd.0 (osd.0) 497 : cluster [DBG] 4.b scrub starts 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.424916+0000 osd.0 (osd.0) 498 : cluster [DBG] 4.b scrub ok 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.797133+0000 osd.1 (osd.1) 79 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.801318+0000 osd.1 (osd.1) 80 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[32381]: cluster 2024-09-10T19:11:20.499074+0000 mgr.y (mgr.24526) 854 : cluster [DBG] pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:22.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.421231+0000 osd.0 (osd.0) 497 : cluster [DBG] 4.b scrub starts 2024-09-10T19:11:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.424916+0000 osd.0 (osd.0) 498 : cluster [DBG] 4.b scrub ok 2024-09-10T19:11:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.797133+0000 osd.1 (osd.1) 79 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:11:22.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.801318+0000 osd.1 (osd.1) 80 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:11:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:22 smithi165 bash[22591]: cluster 2024-09-10T19:11:20.499074+0000 mgr.y (mgr.24526) 854 : cluster [DBG] pgmap v1429: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:22 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.421231+0000 osd.0 (osd.0) 497 : cluster [DBG] 4.b scrub starts 2024-09-10T19:11:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:22 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.424916+0000 osd.0 (osd.0) 498 : cluster [DBG] 4.b scrub ok 2024-09-10T19:11:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:22 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.797133+0000 osd.1 (osd.1) 79 : cluster [DBG] 6.1d scrub starts 2024-09-10T19:11:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:22 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.801318+0000 osd.1 (osd.1) 80 : cluster [DBG] 6.1d scrub ok 2024-09-10T19:11:22.774 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:23.040 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:22] "GET /metrics HTTP/1.1" 200 224110 "" "Prometheus/2.43.0" 2024-09-10T19:11:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:23 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.985797+0000 osd.3 (osd.3) 538 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:11:23.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:23 smithi083 bash[18629]: cluster 2024-09-10T19:11:21.989253+0000 osd.3 (osd.3) 539 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:11:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:23 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.985797+0000 osd.3 (osd.3) 538 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:11:23.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:23 smithi083 bash[32381]: cluster 2024-09-10T19:11:21.989253+0000 osd.3 (osd.3) 539 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:11:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:23 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.985797+0000 osd.3 (osd.3) 538 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:11:23.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:23 smithi165 bash[22591]: cluster 2024-09-10T19:11:21.989253+0000 osd.3 (osd.3) 539 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:11:24.415 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[32381]: cluster 2024-09-10T19:11:22.500959+0000 mgr.y (mgr.24526) 855 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:24.416 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.033847+0000 osd.3 (osd.3) 540 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:11:24.416 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.037387+0000 osd.3 (osd.3) 541 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:11:24.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[18629]: cluster 2024-09-10T19:11:22.500959+0000 mgr.y (mgr.24526) 855 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:24.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.033847+0000 osd.3 (osd.3) 540 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:11:24.416 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:24 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.037387+0000 osd.3 (osd.3) 541 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:11:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:24 smithi165 bash[22591]: cluster 2024-09-10T19:11:22.500959+0000 mgr.y (mgr.24526) 855 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:24 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.033847+0000 osd.3 (osd.3) 540 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:11:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:24 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.037387+0000 osd.3 (osd.3) 541 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.748287+0000 osd.7 (osd.7) 518 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.752025+0000 osd.7 (osd.7) 519 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.877867+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.a scrub starts 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.881376+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.a scrub ok 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.951579+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:23.954655+0000 osd.5 (osd.5) 523 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.107733+0000 osd.6 (osd.6) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:25.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.108073+0000 osd.6 (osd.6) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.173373+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.748287+0000 osd.7 (osd.7) 518 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.752025+0000 osd.7 (osd.7) 519 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.877867+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.a scrub starts 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.881376+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.a scrub ok 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.951579+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:23.954655+0000 osd.5 (osd.5) 523 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.107733+0000 osd.6 (osd.6) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.108073+0000 osd.6 (osd.6) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:25.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:25 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.173373+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.748287+0000 osd.7 (osd.7) 518 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.752025+0000 osd.7 (osd.7) 519 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.877867+0000 osd.1 (osd.1) 81 : cluster [DBG] 2.a scrub starts 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.881376+0000 osd.1 (osd.1) 82 : cluster [DBG] 2.a scrub ok 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.951579+0000 osd.5 (osd.5) 522 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:23.954655+0000 osd.5 (osd.5) 523 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:11:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.107733+0000 osd.6 (osd.6) 481 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:25.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.108073+0000 osd.6 (osd.6) 482 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:25.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:25 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.173373+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.501715+0000 mgr.y (mgr.24526) 856 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.989971+0000 osd.3 (osd.3) 542 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[18629]: cluster 2024-09-10T19:11:24.994900+0000 osd.3 (osd.3) 543 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[18629]: cluster 2024-09-10T19:11:25.843698+0000 osd.1 (osd.1) 83 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[18629]: cluster 2024-09-10T19:11:25.847475+0000 osd.1 (osd.1) 84 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.501715+0000 mgr.y (mgr.24526) 856 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.989971+0000 osd.3 (osd.3) 542 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:11:26.509 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[32381]: cluster 2024-09-10T19:11:24.994900+0000 osd.3 (osd.3) 543 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:11:26.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[32381]: cluster 2024-09-10T19:11:25.843698+0000 osd.1 (osd.1) 83 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:11:26.510 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:26 smithi083 bash[32381]: cluster 2024-09-10T19:11:25.847475+0000 osd.1 (osd.1) 84 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:11:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:26 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.501715+0000 mgr.y (mgr.24526) 856 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:26 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.989971+0000 osd.3 (osd.3) 542 : cluster [DBG] 3.18 scrub starts 2024-09-10T19:11:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:26 smithi165 bash[22591]: cluster 2024-09-10T19:11:24.994900+0000 osd.3 (osd.3) 543 : cluster [DBG] 3.18 scrub ok 2024-09-10T19:11:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:26 smithi165 bash[22591]: cluster 2024-09-10T19:11:25.843698+0000 osd.1 (osd.1) 83 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:11:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:26 smithi165 bash[22591]: cluster 2024-09-10T19:11:25.847475+0000 osd.1 (osd.1) 84 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:11:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.659+0000 7fc6f067e640 1 -- 172.21.15.83:0/1142239009 >> v1:172.21.15.83:6790/0 conn(0x7fc6e80fd3b0 legacy=0x7fc6e80faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:26.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.659+0000 7fc6f067e640 1 -- 172.21.15.83:0/1142239009 shutdown_connections 2024-09-10T19:11:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- 172.21.15.83:0/1142239009 >> 172.21.15.83:0/1142239009 conn(0x7fc6e8074a80 msgr2=0x7fc6e80f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- 172.21.15.83:0/1142239009 shutdown_connections 2024-09-10T19:11:26.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- 172.21.15.83:0/1142239009 wait complete. 2024-09-10T19:11:26.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 librados: starting msgr at 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 librados: starting objecter 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 Processor -- start 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- start start 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 librados: setting wanted keys 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 librados: calling monclient init 2024-09-10T19:11:26.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6e811e1d0 con 0x7fc6e80fd870 2024-09-10T19:11:26.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6e811f390 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6e8120590 con 0x7fc6e810b230 2024-09-10T19:11:26.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6eec1b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc6e810b230 0x7fc6e811c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:43594/0 (socket says 172.21.15.83:43594) 2024-09-10T19:11:26.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6edc19640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc6e80fd870 0x7fc6e8107ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:53920/0 (socket says 172.21.15.83:53920) 2024-09-10T19:11:26.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6eec1b640 1 -- 172.21.15.83:0/1328718064 learned_addr learned my addr 172.21.15.83:0/1328718064 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter ms_handle_connect 0x7fc6e810b230 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter ms_handle_connect 0x7fc6e80fd870 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter ms_handle_connect 0x7fc6e80fd3b0 2024-09-10T19:11:26.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405442145 0 0) 0x7fc6e8120590 con 0x7fc6e810b230 2024-09-10T19:11:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d40038b0 con 0x7fc6e810b230 2024-09-10T19:11:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753628442 0 0) 0x7fc6e811e1d0 con 0x7fc6e80fd870 2024-09-10T19:11:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e8120590 con 0x7fc6e80fd870 2024-09-10T19:11:26.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662551796 0 0) 0x7fc6e811f390 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e811e1d0 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782601103 0 0) 0x7fc6d40038b0 con 0x7fc6e810b230 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e811f390 con 0x7fc6e810b230 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6e0002cd0 con 0x7fc6e810b230 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441656724 0 0) 0x7fc6e811e1d0 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6d40038b0 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6e4002b30 con 0x7fc6e80fd3b0 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77105977 0 0) 0x7fc6e8120590 con 0x7fc6e80fd870 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6e811e1d0 con 0x7fc6e80fd870 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1344821325 0 0) 0x7fc6e811f390 con 0x7fc6e810b230 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 >> v1:172.21.15.165:6789/0 conn(0x7fc6e80fd3b0 legacy=0x7fc6e81075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:26.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 >> v1:172.21.15.83:6789/0 conn(0x7fc6e80fd870 legacy=0x7fc6e8107ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e8121790 con 0x7fc6e810b230 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e811f580 con 0x7fc6e810b230 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 10 client.50003.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e811f9d0 con 0x7fc6e810b230 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6f067e640 1 librados: init done 2024-09-10T19:11:26.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc6e0003050 con 0x7fc6e810b230 2024-09-10T19:11:26.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.663+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6e0005e00 con 0x7fc6e810b230 2024-09-10T19:11:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc005010 con 0x7fc6e810b230 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc6e0001960 con 0x7fc6e810b230 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(884..884 src has 827..884) v4 ==== 7363+0+0 (unknown 2230166114 0 0) 0x7fc6e008a550 con 0x7fc6e810b230 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 10 client.50003.objecter ms_dispatch 0x7fc6e8000b90 osd_map(884..884 src has 827..884) v4 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 3 client.50003.objecter handle_osd_map got epochs [884,884] > 0 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 3 client.50003.objecter handle_osd_map decoding full epoch 884 2024-09-10T19:11:26.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 20 client.50003.objecter dump_active .. 0 homeless 2024-09-10T19:11:26.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.667+0000 7fc6deffd640 10 client.50003.objecter ms_handle_connect 0x7fc6d4071900 2024-09-10T19:11:26.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.675+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc6e0056d90 con 0x7fc6e810b230 2024-09-10T19:11:26.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:26.859+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7fc6cc005840 con 0x7fc6e810b230 2024-09-10T19:11:27.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.187+0000 7fc6deffd640 1 -- 172.21.15.83:0/1328718064 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v885) v1 ==== 133+0+0 (unknown 1140202035 0 0) 0x7fc6e0056730 con 0x7fc6e810b230 2024-09-10T19:11:27.194 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-10T19:11:27.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 10 librados: watch_flush enter 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 10 librados: watch_flush exit 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 20 client.50003.objecter shutdown clearing up homeless session... 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 10 client.50003.objecter successfully canceled tick 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc6d4071900 legacy=0x7fc6d4073db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.191+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 >> v1:172.21.15.83:6790/0 conn(0x7fc6e810b230 legacy=0x7fc6e811c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.195+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 shutdown_connections 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.195+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 >> 172.21.15.83:0/1328718064 conn(0x7fc6e8074a80 msgr2=0x7fc6e80fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.195+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 shutdown_connections 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.195+0000 7fc6f067e640 1 -- 172.21.15.83:0/1328718064 wait complete. 2024-09-10T19:11:27.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:27.195+0000 7fc6f067e640 1 librados: shutdown 2024-09-10T19:11:27.460 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: cluster 2024-09-10T19:11:25.527597+0000 osd.6 (osd.6) 483 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: cluster 2024-09-10T19:11:25.531143+0000 osd.6 (osd.6) 484 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: cluster 2024-09-10T19:11:26.316316+0000 osd.0 (osd.0) 499 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: cluster 2024-09-10T19:11:26.319967+0000 osd.0 (osd.0) 500 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: audit 2024-09-10T19:11:26.865487+0000 mon.c (mon.2) 913 : audit [INF] from='client.? v1:172.21.15.83:0/1328718064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[32381]: audit 2024-09-10T19:11:26.866146+0000 mon.a (mon.0) 3858 : audit [INF] from='client.50003 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: cluster 2024-09-10T19:11:25.527597+0000 osd.6 (osd.6) 483 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: cluster 2024-09-10T19:11:25.531143+0000 osd.6 (osd.6) 484 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: cluster 2024-09-10T19:11:26.316316+0000 osd.0 (osd.0) 499 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:11:27.461 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: cluster 2024-09-10T19:11:26.319967+0000 osd.0 (osd.0) 500 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:11:27.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: audit 2024-09-10T19:11:26.865487+0000 mon.c (mon.2) 913 : audit [INF] from='client.? v1:172.21.15.83:0/1328718064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:27.462 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:27 smithi083 bash[18629]: audit 2024-09-10T19:11:26.866146+0000 mon.a (mon.0) 3858 : audit [INF] from='client.50003 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: cluster 2024-09-10T19:11:25.527597+0000 osd.6 (osd.6) 483 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: cluster 2024-09-10T19:11:25.531143+0000 osd.6 (osd.6) 484 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: cluster 2024-09-10T19:11:26.316316+0000 osd.0 (osd.0) 499 : cluster [DBG] 12.1d scrub starts 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: cluster 2024-09-10T19:11:26.319967+0000 osd.0 (osd.0) 500 : cluster [DBG] 12.1d scrub ok 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: audit 2024-09-10T19:11:26.865487+0000 mon.c (mon.2) 913 : audit [INF] from='client.? v1:172.21.15.83:0/1328718064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:27 smithi165 bash[22591]: audit 2024-09-10T19:11:26.866146+0000 mon.a (mon.0) 3858 : audit [INF] from='client.50003 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:11:28.073 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T19:11:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[32381]: cluster 2024-09-10T19:11:26.503730+0000 mgr.y (mgr.24526) 857 : cluster [DBG] pgmap v1433: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:11:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.013900+0000 osd.3 (osd.3) 544 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:11:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.017089+0000 osd.3 (osd.3) 545 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:11:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[32381]: audit 2024-09-10T19:11:27.189281+0000 mon.a (mon.0) 3859 : audit [INF] from='client.50003 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:28.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.189472+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-09-10T19:11:28.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[18629]: cluster 2024-09-10T19:11:26.503730+0000 mgr.y (mgr.24526) 857 : cluster [DBG] pgmap v1433: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:11:28.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.013900+0000 osd.3 (osd.3) 544 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:11:28.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.017089+0000 osd.3 (osd.3) 545 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:11:28.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[18629]: audit 2024-09-10T19:11:27.189281+0000 mon.a (mon.0) 3859 : audit [INF] from='client.50003 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:28.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:28 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.189472+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-09-10T19:11:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[22591]: cluster 2024-09-10T19:11:26.503730+0000 mgr.y (mgr.24526) 857 : cluster [DBG] pgmap v1433: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:11:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.013900+0000 osd.3 (osd.3) 544 : cluster [DBG] 2.1b scrub starts 2024-09-10T19:11:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.017089+0000 osd.3 (osd.3) 545 : cluster [DBG] 2.1b scrub ok 2024-09-10T19:11:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[22591]: audit 2024-09-10T19:11:27.189281+0000 mon.a (mon.0) 3859 : audit [INF] from='client.50003 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:11:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.189472+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-09-10T19:11:29.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.565943+0000 osd.6 (osd.6) 485 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.573695+0000 osd.6 (osd.6) 486 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.824783+0000 osd.7 (osd.7) 520 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:27.833348+0000 osd.7 (osd.7) 521 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.317902+0000 osd.0 (osd.0) 501 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.323098+0000 osd.0 (osd.0) 502 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.565943+0000 osd.6 (osd.6) 485 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:11:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.573695+0000 osd.6 (osd.6) 486 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:11:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.824783+0000 osd.7 (osd.7) 520 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:11:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:27.833348+0000 osd.7 (osd.7) 521 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:11:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.317902+0000 osd.0 (osd.0) 501 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:11:29.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:29 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.323098+0000 osd.0 (osd.0) 502 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.565943+0000 osd.6 (osd.6) 485 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.573695+0000 osd.6 (osd.6) 486 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.824783+0000 osd.7 (osd.7) 520 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:27.833348+0000 osd.7 (osd.7) 521 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.317902+0000 osd.0 (osd.0) 501 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:11:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:29 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.323098+0000 osd.0 (osd.0) 502 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:11:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.504540+0000 mgr.y (mgr.24526) 858 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.595565+0000 osd.6 (osd.6) 487 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:11:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.599079+0000 osd.6 (osd.6) 488 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: audit 2024-09-10T19:11:28.624607+0000 mgr.y (mgr.24526) 859 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.931445+0000 osd.1 (osd.1) 85 : cluster [DBG] 4.f scrub starts 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.935066+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.f scrub ok 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.949989+0000 osd.4 (osd.4) 482 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:28.950336+0000 osd.4 (osd.4) 483 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:29.218620+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:29.270737+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: cluster 2024-09-10T19:11:29.277905+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[32381]: audit 2024-09-10T19:11:29.581562+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:30.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.504540+0000 mgr.y (mgr.24526) 858 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.595565+0000 osd.6 (osd.6) 487 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.599079+0000 osd.6 (osd.6) 488 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: audit 2024-09-10T19:11:28.624607+0000 mgr.y (mgr.24526) 859 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.931445+0000 osd.1 (osd.1) 85 : cluster [DBG] 4.f scrub starts 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.935066+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.f scrub ok 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.949989+0000 osd.4 (osd.4) 482 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:28.950336+0000 osd.4 (osd.4) 483 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:29.218620+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:29.270737+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: cluster 2024-09-10T19:11:29.277905+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:11:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:30 smithi083 bash[18629]: audit 2024-09-10T19:11:29.581562+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.504540+0000 mgr.y (mgr.24526) 858 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.595565+0000 osd.6 (osd.6) 487 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:11:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.599079+0000 osd.6 (osd.6) 488 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:11:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: audit 2024-09-10T19:11:28.624607+0000 mgr.y (mgr.24526) 859 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.931445+0000 osd.1 (osd.1) 85 : cluster [DBG] 4.f scrub starts 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.935066+0000 osd.1 (osd.1) 86 : cluster [DBG] 4.f scrub ok 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.949989+0000 osd.4 (osd.4) 482 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:28.950336+0000 osd.4 (osd.4) 483 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:29.218620+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:29.270737+0000 osd.0 (osd.0) 503 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: cluster 2024-09-10T19:11:29.277905+0000 osd.0 (osd.0) 504 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:11:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:30 smithi165 bash[22591]: audit 2024-09-10T19:11:29.581562+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:31 smithi083 bash[18629]: cluster 2024-09-10T19:11:30.260916+0000 osd.0 (osd.0) 505 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:11:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:31 smithi083 bash[18629]: cluster 2024-09-10T19:11:30.264530+0000 osd.0 (osd.0) 506 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:11:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:31 smithi083 bash[32381]: cluster 2024-09-10T19:11:30.260916+0000 osd.0 (osd.0) 505 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:11:31.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:31 smithi083 bash[32381]: cluster 2024-09-10T19:11:30.264530+0000 osd.0 (osd.0) 506 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:11:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:31 smithi165 bash[22591]: cluster 2024-09-10T19:11:30.260916+0000 osd.0 (osd.0) 505 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:11:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:31 smithi165 bash[22591]: cluster 2024-09-10T19:11:30.264530+0000 osd.0 (osd.0) 506 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:11:31.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[18629]: cluster 2024-09-10T19:11:30.505358+0000 mgr.y (mgr.24526) 860 : cluster [DBG] pgmap v1437: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-09-10T19:11:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[18629]: cluster 2024-09-10T19:11:30.794148+0000 osd.3 (osd.3) 546 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[18629]: cluster 2024-09-10T19:11:30.794653+0000 osd.3 (osd.3) 547 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[18629]: cluster 2024-09-10T19:11:31.228489+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-09-10T19:11:32.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[32381]: cluster 2024-09-10T19:11:30.505358+0000 mgr.y (mgr.24526) 860 : cluster [DBG] pgmap v1437: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-09-10T19:11:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[32381]: cluster 2024-09-10T19:11:30.794148+0000 osd.3 (osd.3) 546 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[32381]: cluster 2024-09-10T19:11:30.794653+0000 osd.3 (osd.3) 547 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[32381]: cluster 2024-09-10T19:11:31.228489+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-09-10T19:11:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:32 smithi165 bash[22591]: cluster 2024-09-10T19:11:30.505358+0000 mgr.y (mgr.24526) 860 : cluster [DBG] pgmap v1437: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 808 B/s rd, 0 op/s 2024-09-10T19:11:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:32 smithi165 bash[22591]: cluster 2024-09-10T19:11:30.794148+0000 osd.3 (osd.3) 546 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:32 smithi165 bash[22591]: cluster 2024-09-10T19:11:30.794653+0000 osd.3 (osd.3) 547 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:32 smithi165 bash[22591]: cluster 2024-09-10T19:11:31.228489+0000 mon.a (mon.0) 3862 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-09-10T19:11:32.848 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:32.951 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:32] "GET /metrics HTTP/1.1" 200 224099 "" "Prometheus/2.43.0" 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:31.870629+0000 osd.7 (osd.7) 522 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:31.874351+0000 osd.7 (osd.7) 523 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:32.235171+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:32.237689+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.023764+0000 osd.5 (osd.5) 524 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.024181+0000 osd.5 (osd.5) 525 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:31.870629+0000 osd.7 (osd.7) 522 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:31.874351+0000 osd.7 (osd.7) 523 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:32.235171+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:32.237689+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.023764+0000 osd.5 (osd.5) 524 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:33.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:33 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.024181+0000 osd.5 (osd.5) 525 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:31.870629+0000 osd.7 (osd.7) 522 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:31.874351+0000 osd.7 (osd.7) 523 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:32.235171+0000 osd.0 (osd.0) 507 : cluster [DBG] 6.16 scrub starts 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:32.237689+0000 osd.0 (osd.0) 508 : cluster [DBG] 6.16 scrub ok 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.023764+0000 osd.5 (osd.5) 524 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:33 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.024181+0000 osd.5 (osd.5) 525 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[18629]: cluster 2024-09-10T19:11:32.506128+0000 mgr.y (mgr.24526) 861 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:11:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[18629]: cluster 2024-09-10T19:11:32.857268+0000 osd.7 (osd.7) 524 : cluster [DBG] 3.f scrub starts 2024-09-10T19:11:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[18629]: cluster 2024-09-10T19:11:32.862459+0000 osd.7 (osd.7) 525 : cluster [DBG] 3.f scrub ok 2024-09-10T19:11:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.242321+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-09-10T19:11:34.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[32381]: cluster 2024-09-10T19:11:32.506128+0000 mgr.y (mgr.24526) 861 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:11:34.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[32381]: cluster 2024-09-10T19:11:32.857268+0000 osd.7 (osd.7) 524 : cluster [DBG] 3.f scrub starts 2024-09-10T19:11:34.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[32381]: cluster 2024-09-10T19:11:32.862459+0000 osd.7 (osd.7) 525 : cluster [DBG] 3.f scrub ok 2024-09-10T19:11:34.544 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:34 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.242321+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-09-10T19:11:34.612 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:11:34 smithi165 bash[57601]: logger=cleanup t=2024-09-10T19:11:34.273677917Z level=info msg="Completed cleanup jobs" duration=66.790104ms 2024-09-10T19:11:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:34 smithi165 bash[22591]: cluster 2024-09-10T19:11:32.506128+0000 mgr.y (mgr.24526) 861 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:11:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:34 smithi165 bash[22591]: cluster 2024-09-10T19:11:32.857268+0000 osd.7 (osd.7) 524 : cluster [DBG] 3.f scrub starts 2024-09-10T19:11:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:34 smithi165 bash[22591]: cluster 2024-09-10T19:11:32.862459+0000 osd.7 (osd.7) 525 : cluster [DBG] 3.f scrub ok 2024-09-10T19:11:34.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:34 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.242321+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.825477+0000 osd.3 (osd.3) 548 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.829121+0000 osd.3 (osd.3) 549 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.902396+0000 osd.7 (osd.7) 526 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:33.906094+0000 osd.7 (osd.7) 527 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.189059+0000 osd.0 (osd.0) 509 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.193696+0000 osd.0 (osd.0) 510 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:11:35.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.630971+0000 osd.6 (osd.6) 489 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.634882+0000 osd.6 (osd.6) 490 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.825477+0000 osd.3 (osd.3) 548 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.829121+0000 osd.3 (osd.3) 549 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.902396+0000 osd.7 (osd.7) 526 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:33.906094+0000 osd.7 (osd.7) 527 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.189059+0000 osd.0 (osd.0) 509 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.193696+0000 osd.0 (osd.0) 510 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.630971+0000 osd.6 (osd.6) 489 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:11:35.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:35 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.634882+0000 osd.6 (osd.6) 490 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:11:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.825477+0000 osd.3 (osd.3) 548 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:11:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.829121+0000 osd.3 (osd.3) 549 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:11:35.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.902396+0000 osd.7 (osd.7) 526 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:11:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:33.906094+0000 osd.7 (osd.7) 527 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:11:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.189059+0000 osd.0 (osd.0) 509 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:11:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.193696+0000 osd.0 (osd.0) 510 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:11:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.630971+0000 osd.6 (osd.6) 489 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:11:35.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:35 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.634882+0000 osd.6 (osd.6) 490 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.506730+0000 mgr.y (mgr.24526) 862 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.875783+0000 osd.3 (osd.3) 550 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[18629]: cluster 2024-09-10T19:11:34.879325+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[18629]: cluster 2024-09-10T19:11:35.596744+0000 osd.6 (osd.6) 491 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[18629]: cluster 2024-09-10T19:11:35.600509+0000 osd.6 (osd.6) 492 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.506730+0000 mgr.y (mgr.24526) 862 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.875783+0000 osd.3 (osd.3) 550 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:11:36.527 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[32381]: cluster 2024-09-10T19:11:34.879325+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:11:36.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[32381]: cluster 2024-09-10T19:11:35.596744+0000 osd.6 (osd.6) 491 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:11:36.528 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:36 smithi083 bash[32381]: cluster 2024-09-10T19:11:35.600509+0000 osd.6 (osd.6) 492 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:11:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:36 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.506730+0000 mgr.y (mgr.24526) 862 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:36 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.875783+0000 osd.3 (osd.3) 550 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:11:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:36 smithi165 bash[22591]: cluster 2024-09-10T19:11:34.879325+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:11:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:36 smithi165 bash[22591]: cluster 2024-09-10T19:11:35.596744+0000 osd.6 (osd.6) 491 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:11:36.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:36 smithi165 bash[22591]: cluster 2024-09-10T19:11:35.600509+0000 osd.6 (osd.6) 492 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:11:36.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/3255730413 >> v1:172.21.15.83:6789/0 conn(0x7f2ed0105310 legacy=0x7f2ed01076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:36.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/3255730413 shutdown_connections 2024-09-10T19:11:36.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/3255730413 >> 172.21.15.83:0/3255730413 conn(0x7f2ed0074af0 msgr2=0x7f2ed00f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:36.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/3255730413 shutdown_connections 2024-09-10T19:11:36.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/3255730413 wait complete. 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 librados: starting msgr at 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 librados: starting objecter 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 Processor -- start 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- start start 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 librados: setting wanted keys 2024-09-10T19:11:36.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 librados: calling monclient init 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ed011c010 con 0x7f2ed0102610 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ed011d1d0 con 0x7f2ed0105310 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed6ac7640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ed011e3d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed4863640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2ed0102610 0x7f2ed00fe590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:58374/0 (socket says 172.21.15.83:58374) 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.635+0000 7f2ed4863640 1 -- 172.21.15.83:0/2498425611 learned_addr learned my addr 172.21.15.83:0/2498425611 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ed0102610 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ed0108f60 2024-09-10T19:11:36.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17054220 0 0) 0x7f2ed011e3d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb8003720 con 0x7f2ed0108f60 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ed0105310 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684472511 0 0) 0x7f2ed011c010 con 0x7f2ed0102610 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed011e3d0 con 0x7f2ed0102610 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232791067 0 0) 0x7f2ed011d1d0 con 0x7f2ed0105310 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed011c010 con 0x7f2ed0105310 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187636149 0 0) 0x7f2eb8003720 con 0x7f2ed0108f60 2024-09-10T19:11:36.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ed011d1d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ec8001bc0 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931179074 0 0) 0x7f2ed011e3d0 con 0x7f2ed0102610 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb8003720 con 0x7f2ed0102610 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ec4001900 con 0x7f2ed0102610 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574689138 0 0) 0x7f2ed011d1d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 >> v1:172.21.15.165:6789/0 conn(0x7f2ed0105310 legacy=0x7f2ed00feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 >> v1:172.21.15.83:6789/0 conn(0x7f2ed0102610 legacy=0x7f2ed00fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed011f5d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed011e5c0 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ed6ac7640 10 client.50009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed011ea10 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ed6ac7640 1 librados: init done 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2ec8002240 con 0x7f2ed0108f60 2024-09-10T19:11:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2ec80034d0 con 0x7f2ed0108f60 2024-09-10T19:11:36.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.639+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc005010 con 0x7f2ed0108f60 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f2ec80198f0 con 0x7f2ed0108f60 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(889..889 src has 838..889) v4 ==== 7363+0+0 (unknown 570741851 0 0) 0x7f2ec8087c10 con 0x7f2ed0108f60 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 10 client.50009.objecter ms_dispatch 0x7f2ed0000b90 osd_map(889..889 src has 838..889) v4 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 3 client.50009.objecter handle_osd_map got epochs [889,889] > 0 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 3 client.50009.objecter handle_osd_map decoding full epoch 889 2024-09-10T19:11:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 20 client.50009.objecter dump_active .. 0 homeless 2024-09-10T19:11:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.643+0000 7f2ecd7fa640 10 client.50009.objecter ms_handle_connect 0x7f2eb80714f0 2024-09-10T19:11:36.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.647+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2ec8053fd0 con 0x7f2ed0108f60 2024-09-10T19:11:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:36.835+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f2ebc0052c0 con 0x7f2ed0108f60 2024-09-10T19:11:37.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.487+0000 7f2ecd7fa640 1 -- 172.21.15.83:0/2498425611 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v890) v1 ==== 113+0+0 (unknown 852666049 0 0) 0x7f2ec8053df0 con 0x7f2ed0108f60 2024-09-10T19:11:37.494 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 10 librados: watch_flush enter 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 10 librados: watch_flush exit 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 20 client.50009.objecter shutdown clearing up homeless session... 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 10 client.50009.objecter successfully canceled tick 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 >> v1:172.21.15.83:6800/3071793735 conn(0x7f2eb80714f0 legacy=0x7f2eb80739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 >> v1:172.21.15.83:6790/0 conn(0x7f2ed0108f60 legacy=0x7f2ed011a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:37.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 shutdown_connections 2024-09-10T19:11:37.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 >> 172.21.15.83:0/2498425611 conn(0x7f2ed0074af0 msgr2=0x7f2ed00fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:37.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 shutdown_connections 2024-09-10T19:11:37.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 -- 172.21.15.83:0/2498425611 wait complete. 2024-09-10T19:11:37.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:37.491+0000 7f2ed6ac7640 1 librados: shutdown 2024-09-10T19:11:37.520 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:35.876182+0000 osd.7 (osd.7) 528 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:35.879600+0000 osd.7 (osd.7) 529 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.152057+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.152425+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.185323+0000 osd.0 (osd.0) 513 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.189078+0000 osd.0 (osd.0) 514 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.482521+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.584000+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.584389+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: audit 2024-09-10T19:11:36.841663+0000 mon.c (mon.2) 915 : audit [INF] from='client.? v1:172.21.15.83:0/2498425611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:37.521 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[18629]: audit 2024-09-10T19:11:36.842338+0000 mon.a (mon.0) 3865 : audit [INF] from='client.50009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:35.876182+0000 osd.7 (osd.7) 528 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:35.879600+0000 osd.7 (osd.7) 529 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.152057+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.152425+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.185323+0000 osd.0 (osd.0) 513 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.189078+0000 osd.0 (osd.0) 514 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.482521+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.584000+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.584389+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.522 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: audit 2024-09-10T19:11:36.841663+0000 mon.c (mon.2) 915 : audit [INF] from='client.? v1:172.21.15.83:0/2498425611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:37.523 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:37 smithi083 bash[32381]: audit 2024-09-10T19:11:36.842338+0000 mon.a (mon.0) 3865 : audit [INF] from='client.50009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:35.876182+0000 osd.7 (osd.7) 528 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:11:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:35.879600+0000 osd.7 (osd.7) 529 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:11:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.152057+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.152425+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.185323+0000 osd.0 (osd.0) 513 : cluster [DBG] 4.1e deep-scrub starts 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.189078+0000 osd.0 (osd.0) 514 : cluster [DBG] 4.1e deep-scrub ok 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.482521+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.584000+0000 osd.1 (osd.1) 87 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.584389+0000 osd.1 (osd.1) 88 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: audit 2024-09-10T19:11:36.841663+0000 mon.c (mon.2) 915 : audit [INF] from='client.? v1:172.21.15.83:0/2498425611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:37.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:37 smithi165 bash[22591]: audit 2024-09-10T19:11:36.842338+0000 mon.a (mon.0) 3865 : audit [INF] from='client.50009 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:38.383 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-10T19:11:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[32381]: cluster 2024-09-10T19:11:36.507378+0000 mgr.y (mgr.24526) 863 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:11:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.150637+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:11:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.154829+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:11:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[32381]: audit 2024-09-10T19:11:37.492816+0000 mon.a (mon.0) 3866 : audit [INF] from='client.50009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:38.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.493022+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-09-10T19:11:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[18629]: cluster 2024-09-10T19:11:36.507378+0000 mgr.y (mgr.24526) 863 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:11:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.150637+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:11:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.154829+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:11:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[18629]: audit 2024-09-10T19:11:37.492816+0000 mon.a (mon.0) 3866 : audit [INF] from='client.50009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:38.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:38 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.493022+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-09-10T19:11:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[22591]: cluster 2024-09-10T19:11:36.507378+0000 mgr.y (mgr.24526) 863 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:11:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.150637+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.1b scrub starts 2024-09-10T19:11:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.154829+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.1b scrub ok 2024-09-10T19:11:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[22591]: audit 2024-09-10T19:11:37.492816+0000 mon.a (mon.0) 3866 : audit [INF] from='client.50009 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:38.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.493022+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-09-10T19:11:39.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:39.540 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.872375+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.877655+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.990012+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:37.999235+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:38.591825+0000 osd.6 (osd.6) 493 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[18629]: cluster 2024-09-10T19:11:38.594833+0000 osd.6 (osd.6) 494 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.872375+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.877655+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.990012+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:37.999235+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:11:39.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:38.591825+0000 osd.6 (osd.6) 493 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:11:39.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:39 smithi083 bash[32381]: cluster 2024-09-10T19:11:38.594833+0000 osd.6 (osd.6) 494 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.872375+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.877655+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.990012+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 deep-scrub starts 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:37.999235+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 deep-scrub ok 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:38.591825+0000 osd.6 (osd.6) 493 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:11:39.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:39 smithi165 bash[22591]: cluster 2024-09-10T19:11:38.594833+0000 osd.6 (osd.6) 494 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:38.329771+0000 osd.4 (osd.4) 484 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:38.333745+0000 osd.4 (osd.4) 485 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:38.507914+0000 mgr.y (mgr.24526) 864 : cluster [DBG] pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: audit 2024-09-10T19:11:38.633090+0000 mgr.y (mgr.24526) 865 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.132000+0000 osd.0 (osd.0) 517 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.135562+0000 osd.0 (osd.0) 518 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:11:40.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.811436+0000 osd.5 (osd.5) 526 : cluster [DBG] 1.a scrub starts 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.815576+0000 osd.5 (osd.5) 527 : cluster [DBG] 1.a scrub ok 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:38.329771+0000 osd.4 (osd.4) 484 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:38.333745+0000 osd.4 (osd.4) 485 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:38.507914+0000 mgr.y (mgr.24526) 864 : cluster [DBG] pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: audit 2024-09-10T19:11:38.633090+0000 mgr.y (mgr.24526) 865 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.132000+0000 osd.0 (osd.0) 517 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.135562+0000 osd.0 (osd.0) 518 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.811436+0000 osd.5 (osd.5) 526 : cluster [DBG] 1.a scrub starts 2024-09-10T19:11:40.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:40 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.815576+0000 osd.5 (osd.5) 527 : cluster [DBG] 1.a scrub ok 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:38.329771+0000 osd.4 (osd.4) 484 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:38.333745+0000 osd.4 (osd.4) 485 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:38.507914+0000 mgr.y (mgr.24526) 864 : cluster [DBG] pgmap v1445: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: audit 2024-09-10T19:11:38.633090+0000 mgr.y (mgr.24526) 865 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.132000+0000 osd.0 (osd.0) 517 : cluster [DBG] 12.3 scrub starts 2024-09-10T19:11:40.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.135562+0000 osd.0 (osd.0) 518 : cluster [DBG] 12.3 scrub ok 2024-09-10T19:11:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.811436+0000 osd.5 (osd.5) 526 : cluster [DBG] 1.a scrub starts 2024-09-10T19:11:40.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:40 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.815576+0000 osd.5 (osd.5) 527 : cluster [DBG] 1.a scrub ok 2024-09-10T19:11:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:41 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.864199+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.b scrub starts 2024-09-10T19:11:41.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:41 smithi083 bash[18629]: cluster 2024-09-10T19:11:39.868021+0000 osd.3 (osd.3) 553 : cluster [DBG] 6.b scrub ok 2024-09-10T19:11:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:41 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.864199+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.b scrub starts 2024-09-10T19:11:41.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:41 smithi083 bash[32381]: cluster 2024-09-10T19:11:39.868021+0000 osd.3 (osd.3) 553 : cluster [DBG] 6.b scrub ok 2024-09-10T19:11:41.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:41 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.864199+0000 osd.3 (osd.3) 552 : cluster [DBG] 6.b scrub starts 2024-09-10T19:11:41.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:41 smithi165 bash[22591]: cluster 2024-09-10T19:11:39.868021+0000 osd.3 (osd.3) 553 : cluster [DBG] 6.b scrub ok 2024-09-10T19:11:41.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:40.330566+0000 osd.4 (osd.4) 486 : cluster [DBG] 4.c scrub starts 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:40.334084+0000 osd.4 (osd.4) 487 : cluster [DBG] 4.c scrub ok 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:40.508625+0000 mgr.y (mgr.24526) 866 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.798957+0000 osd.5 (osd.5) 528 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.802451+0000 osd.5 (osd.5) 529 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.856296+0000 osd.7 (osd.7) 532 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:11:42.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.860119+0000 osd.7 (osd.7) 533 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:40.330566+0000 osd.4 (osd.4) 486 : cluster [DBG] 4.c scrub starts 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:40.334084+0000 osd.4 (osd.4) 487 : cluster [DBG] 4.c scrub ok 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:40.508625+0000 mgr.y (mgr.24526) 866 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.798957+0000 osd.5 (osd.5) 528 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.802451+0000 osd.5 (osd.5) 529 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.856296+0000 osd.7 (osd.7) 532 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:11:42.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.860119+0000 osd.7 (osd.7) 533 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:40.330566+0000 osd.4 (osd.4) 486 : cluster [DBG] 4.c scrub starts 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:40.334084+0000 osd.4 (osd.4) 487 : cluster [DBG] 4.c scrub ok 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:40.508625+0000 mgr.y (mgr.24526) 866 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 703 B/s rd, 0 op/s 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.798957+0000 osd.5 (osd.5) 528 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.802451+0000 osd.5 (osd.5) 529 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:11:42.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.856296+0000 osd.7 (osd.7) 532 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:11:42.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:42 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.860119+0000 osd.7 (osd.7) 533 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:11:43.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:42] "GET /metrics HTTP/1.1" 200 224099 "" "Prometheus/2.43.0" 2024-09-10T19:11:43.161 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.372589+0000 osd.4 (osd.4) 488 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:11:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.376577+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:11:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.907120+0000 osd.3 (osd.3) 554 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:11:43.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:41.911149+0000 osd.3 (osd.3) 555 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:42.219667+0000 osd.2 (osd.2) 383 : cluster [DBG] 12.a scrub starts 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[18629]: cluster 2024-09-10T19:11:42.223066+0000 osd.2 (osd.2) 384 : cluster [DBG] 12.a scrub ok 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.372589+0000 osd.4 (osd.4) 488 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.376577+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.907120+0000 osd.3 (osd.3) 554 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:41.911149+0000 osd.3 (osd.3) 555 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:42.219667+0000 osd.2 (osd.2) 383 : cluster [DBG] 12.a scrub starts 2024-09-10T19:11:43.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:43 smithi083 bash[32381]: cluster 2024-09-10T19:11:42.223066+0000 osd.2 (osd.2) 384 : cluster [DBG] 12.a scrub ok 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.372589+0000 osd.4 (osd.4) 488 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.376577+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.907120+0000 osd.3 (osd.3) 554 : cluster [DBG] 4.14 scrub starts 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:41.911149+0000 osd.3 (osd.3) 555 : cluster [DBG] 4.14 scrub ok 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:42.219667+0000 osd.2 (osd.2) 383 : cluster [DBG] 12.a scrub starts 2024-09-10T19:11:43.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:43 smithi165 bash[22591]: cluster 2024-09-10T19:11:42.223066+0000 osd.2 (osd.2) 384 : cluster [DBG] 12.a scrub ok 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[18629]: cluster 2024-09-10T19:11:42.510506+0000 mgr.y (mgr.24526) 867 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[18629]: cluster 2024-09-10T19:11:43.226382+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[18629]: cluster 2024-09-10T19:11:43.230139+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[32381]: cluster 2024-09-10T19:11:42.510506+0000 mgr.y (mgr.24526) 867 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[32381]: cluster 2024-09-10T19:11:43.226382+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:11:44.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:44 smithi083 bash[32381]: cluster 2024-09-10T19:11:43.230139+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:11:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:44 smithi165 bash[22591]: cluster 2024-09-10T19:11:42.510506+0000 mgr.y (mgr.24526) 867 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:44 smithi165 bash[22591]: cluster 2024-09-10T19:11:43.226382+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:11:44.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:44 smithi165 bash[22591]: cluster 2024-09-10T19:11:43.230139+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:43.975731+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:43.978621+0000 osd.3 (osd.3) 557 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.275653+0000 osd.0 (osd.0) 521 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.281154+0000 osd.0 (osd.0) 522 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: audit 2024-09-10T19:11:44.581629+0000 mon.c (mon.2) 916 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.796018+0000 osd.5 (osd.5) 530 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:11:45.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.799153+0000 osd.5 (osd.5) 531 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:43.975731+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:43.978621+0000 osd.3 (osd.3) 557 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.275653+0000 osd.0 (osd.0) 521 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.281154+0000 osd.0 (osd.0) 522 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: audit 2024-09-10T19:11:44.581629+0000 mon.c (mon.2) 916 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.796018+0000 osd.5 (osd.5) 530 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:11:45.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:45 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.799153+0000 osd.5 (osd.5) 531 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:11:45.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:43.975731+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:43.978621+0000 osd.3 (osd.3) 557 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.275653+0000 osd.0 (osd.0) 521 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.281154+0000 osd.0 (osd.0) 522 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: audit 2024-09-10T19:11:44.581629+0000 mon.c (mon.2) 916 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.796018+0000 osd.5 (osd.5) 530 : cluster [DBG] 2.16 deep-scrub starts 2024-09-10T19:11:45.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:45 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.799153+0000 osd.5 (osd.5) 531 : cluster [DBG] 2.16 deep-scrub ok 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.417124+0000 osd.4 (osd.4) 490 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.420556+0000 osd.4 (osd.4) 491 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.511354+0000 mgr.y (mgr.24526) 868 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.920108+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[18629]: cluster 2024-09-10T19:11:44.923856+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.417124+0000 osd.4 (osd.4) 490 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:11:46.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.420556+0000 osd.4 (osd.4) 491 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:11:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.511354+0000 mgr.y (mgr.24526) 868 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-10T19:11:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.920108+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:11:46.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:46 smithi083 bash[32381]: cluster 2024-09-10T19:11:44.923856+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:11:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:46 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.417124+0000 osd.4 (osd.4) 490 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:11:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:46 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.420556+0000 osd.4 (osd.4) 491 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:11:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:46 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.511354+0000 mgr.y (mgr.24526) 868 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-10T19:11:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:46 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.920108+0000 osd.1 (osd.1) 91 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:11:46.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:46 smithi165 bash[22591]: cluster 2024-09-10T19:11:44.923856+0000 osd.1 (osd.1) 92 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:11:46.912 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- 172.21.15.83:0/31863396 >> v1:172.21.15.83:6789/0 conn(0x7fd63010b130 legacy=0x7fd63010d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- 172.21.15.83:0/31863396 shutdown_connections 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- 172.21.15.83:0/31863396 >> 172.21.15.83:0/31863396 conn(0x7fd6300fa3d0 msgr2=0x7fd6300fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- 172.21.15.83:0/31863396 shutdown_connections 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- 172.21.15.83:0/31863396 wait complete. 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 librados: starting msgr at 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 librados: starting objecter 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 Processor -- start 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- start start 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 librados: setting wanted keys 2024-09-10T19:11:46.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 librados: calling monclient init 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6301227a0 con 0x7fd630104810 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6301239a0 con 0x7fd6301075a0 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd636a62640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd630124ba0 con 0x7fd63010b130 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd630104810 0x7fd630100810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:46040/0 (socket says 172.21.15.83:46040) 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62ffff640 1 -- 172.21.15.83:0/2291743010 learned_addr learned my addr 172.21.15.83:0/2291743010 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter ms_handle_connect 0x7fd630104810 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter ms_handle_connect 0x7fd6301075a0 2024-09-10T19:11:46.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter ms_handle_connect 0x7fd63010b130 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056556240 0 0) 0x7fd6301227a0 con 0x7fd630104810 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c003720 con 0x7fd630104810 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759983765 0 0) 0x7fd630124ba0 con 0x7fd63010b130 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6301227a0 con 0x7fd63010b130 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840130850 0 0) 0x7fd6301239a0 con 0x7fd6301075a0 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd630124ba0 con 0x7fd6301075a0 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3547592144 0 0) 0x7fd61c003720 con 0x7fd630104810 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6301239a0 con 0x7fd630104810 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.906+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd620001cc0 con 0x7fd630104810 2024-09-10T19:11:46.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3095676466 0 0) 0x7fd6301227a0 con 0x7fd63010b130 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd61c003720 con 0x7fd63010b130 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd628001430 con 0x7fd63010b130 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 28378855 0 0) 0x7fd6301239a0 con 0x7fd630104810 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 >> v1:172.21.15.83:6790/0 conn(0x7fd63010b130 legacy=0x7fd630120f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 >> v1:172.21.15.165:6789/0 conn(0x7fd6301075a0 legacy=0x7fd630100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630125da0 con 0x7fd630104810 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd630123b90 con 0x7fd630104810 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd636a62640 10 client.25929.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd630124060 con 0x7fd630104810 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd636a62640 1 librados: init done 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd6200027a0 con 0x7fd630104810 2024-09-10T19:11:46.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd620005e00 con 0x7fd630104810 2024-09-10T19:11:46.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd62001c330 con 0x7fd630104810 2024-09-10T19:11:46.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.910+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd610005010 con 0x7fd630104810 2024-09-10T19:11:46.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(890..890 src has 838..890) v4 ==== 7363+0+0 (unknown 3951514120 0 0) 0x7fd62008a8d0 con 0x7fd630104810 2024-09-10T19:11:46.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 10 client.25929.objecter ms_dispatch 0x7fd630000b90 osd_map(890..890 src has 838..890) v4 2024-09-10T19:11:46.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 3 client.25929.objecter handle_osd_map got epochs [890,890] > 0 2024-09-10T19:11:46.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 3 client.25929.objecter handle_osd_map decoding full epoch 890 2024-09-10T19:11:46.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 20 client.25929.objecter dump_active .. 0 homeless 2024-09-10T19:11:46.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.914+0000 7fd62cff9640 10 client.25929.objecter ms_handle_connect 0x7fd61c0716d0 2024-09-10T19:11:46.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:46.918+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fd620056c90 con 0x7fd630104810 2024-09-10T19:11:47.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.106+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fd610005840 con 0x7fd630104810 2024-09-10T19:11:47.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.294+0000 7fd62cff9640 1 -- 172.21.15.83:0/2291743010 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v891) v1 ==== 133+0+0 (unknown 215403736 0 0) 0x7fd620056ab0 con 0x7fd630104810 2024-09-10T19:11:47.301 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-10T19:11:47.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 10 librados: watch_flush enter 2024-09-10T19:11:47.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 10 librados: watch_flush exit 2024-09-10T19:11:47.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 20 client.25929.objecter shutdown clearing up homeless session... 2024-09-10T19:11:47.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 10 client.25929.objecter successfully canceled tick 2024-09-10T19:11:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd61c0716d0 legacy=0x7fd61c073b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 >> v1:172.21.15.83:6789/0 conn(0x7fd630104810 legacy=0x7fd630100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 shutdown_connections 2024-09-10T19:11:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 >> 172.21.15.83:0/2291743010 conn(0x7fd6300fa3d0 msgr2=0x7fd6300fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:47.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 shutdown_connections 2024-09-10T19:11:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 -- 172.21.15.83:0/2291743010 wait complete. 2024-09-10T19:11:47.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:47.298+0000 7fd636a62640 1 librados: shutdown 2024-09-10T19:11:47.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:47 smithi165 bash[22591]: cluster 2024-09-10T19:11:45.372573+0000 osd.4 (osd.4) 492 : cluster [DBG] 4.d scrub starts 2024-09-10T19:11:47.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:47 smithi165 bash[22591]: cluster 2024-09-10T19:11:45.377170+0000 osd.4 (osd.4) 493 : cluster [DBG] 4.d scrub ok 2024-09-10T19:11:47.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:47 smithi165 bash[22591]: audit 2024-09-10T19:11:47.112004+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[18629]: cluster 2024-09-10T19:11:45.372573+0000 osd.4 (osd.4) 492 : cluster [DBG] 4.d scrub starts 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[18629]: cluster 2024-09-10T19:11:45.377170+0000 osd.4 (osd.4) 493 : cluster [DBG] 4.d scrub ok 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[18629]: audit 2024-09-10T19:11:47.112004+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[32381]: cluster 2024-09-10T19:11:45.372573+0000 osd.4 (osd.4) 492 : cluster [DBG] 4.d scrub starts 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[32381]: cluster 2024-09-10T19:11:45.377170+0000 osd.4 (osd.4) 493 : cluster [DBG] 4.d scrub ok 2024-09-10T19:11:47.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:47 smithi083 bash[32381]: audit 2024-09-10T19:11:47.112004+0000 mon.a (mon.0) 3868 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:11:48.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:46.359227+0000 osd.4 (osd.4) 494 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:46.362685+0000 osd.4 (osd.4) 495 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:46.513246+0000 mgr.y (mgr.24526) 869 : cluster [DBG] pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:47.202814+0000 osd.0 (osd.0) 523 : cluster [DBG] 12.e scrub starts 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:47.206659+0000 osd.0 (osd.0) 524 : cluster [DBG] 12.e scrub ok 2024-09-10T19:11:48.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: audit 2024-09-10T19:11:47.298222+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:47.298573+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:47.840078+0000 osd.1 (osd.1) 93 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[32381]: cluster 2024-09-10T19:11:47.849730+0000 osd.1 (osd.1) 94 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:46.359227+0000 osd.4 (osd.4) 494 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:46.362685+0000 osd.4 (osd.4) 495 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:46.513246+0000 mgr.y (mgr.24526) 869 : cluster [DBG] pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:47.202814+0000 osd.0 (osd.0) 523 : cluster [DBG] 12.e scrub starts 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:47.206659+0000 osd.0 (osd.0) 524 : cluster [DBG] 12.e scrub ok 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: audit 2024-09-10T19:11:47.298222+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:47.298573+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-09-10T19:11:48.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:47.840078+0000 osd.1 (osd.1) 93 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:11:48.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:48 smithi083 bash[18629]: cluster 2024-09-10T19:11:47.849730+0000 osd.1 (osd.1) 94 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:11:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:46.359227+0000 osd.4 (osd.4) 494 : cluster [DBG] 12.1a scrub starts 2024-09-10T19:11:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:46.362685+0000 osd.4 (osd.4) 495 : cluster [DBG] 12.1a scrub ok 2024-09-10T19:11:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:46.513246+0000 mgr.y (mgr.24526) 869 : cluster [DBG] pgmap v1449: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:48.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:47.202814+0000 osd.0 (osd.0) 523 : cluster [DBG] 12.e scrub starts 2024-09-10T19:11:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:47.206659+0000 osd.0 (osd.0) 524 : cluster [DBG] 12.e scrub ok 2024-09-10T19:11:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: audit 2024-09-10T19:11:47.298222+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? v1:172.21.15.83:0/2291743010' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:11:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:47.298573+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-09-10T19:11:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:47.840078+0000 osd.1 (osd.1) 93 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:11:48.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[22591]: cluster 2024-09-10T19:11:47.849730+0000 osd.1 (osd.1) 94 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:11:49.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:49 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.434555+0000 osd.4 (osd.4) 496 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:11:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:49 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.438375+0000 osd.4 (osd.4) 497 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:11:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:49 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.818549+0000 osd.5 (osd.5) 532 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:11:49.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:49 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.822477+0000 osd.5 (osd.5) 533 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.434555+0000 osd.4 (osd.4) 496 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.438375+0000 osd.4 (osd.4) 497 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.818549+0000 osd.5 (osd.5) 532 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.822477+0000 osd.5 (osd.5) 533 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.434555+0000 osd.4 (osd.4) 496 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:11:49.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.438375+0000 osd.4 (osd.4) 497 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:11:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.818549+0000 osd.5 (osd.5) 532 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:11:49.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:49 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.822477+0000 osd.5 (osd.5) 533 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.514041+0000 mgr.y (mgr.24526) 870 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: audit 2024-09-10T19:11:48.639496+0000 mgr.y (mgr.24526) 871 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.897321+0000 osd.3 (osd.3) 558 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: cluster 2024-09-10T19:11:48.902303+0000 osd.3 (osd.3) 559 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: cluster 2024-09-10T19:11:49.767961+0000 osd.7 (osd.7) 534 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:11:50.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:50 smithi165 bash[22591]: cluster 2024-09-10T19:11:49.771358+0000 osd.7 (osd.7) 535 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:11:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.514041+0000 mgr.y (mgr.24526) 870 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: audit 2024-09-10T19:11:48.639496+0000 mgr.y (mgr.24526) 871 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.897321+0000 osd.3 (osd.3) 558 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:11:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: cluster 2024-09-10T19:11:48.902303+0000 osd.3 (osd.3) 559 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:11:50.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: cluster 2024-09-10T19:11:49.767961+0000 osd.7 (osd.7) 534 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[32381]: cluster 2024-09-10T19:11:49.771358+0000 osd.7 (osd.7) 535 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.514041+0000 mgr.y (mgr.24526) 870 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: audit 2024-09-10T19:11:48.639496+0000 mgr.y (mgr.24526) 871 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.897321+0000 osd.3 (osd.3) 558 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: cluster 2024-09-10T19:11:48.902303+0000 osd.3 (osd.3) 559 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: cluster 2024-09-10T19:11:49.767961+0000 osd.7 (osd.7) 534 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:11:50.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:50 smithi083 bash[18629]: cluster 2024-09-10T19:11:49.771358+0000 osd.7 (osd.7) 535 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:11:51.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:11:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:51 smithi165 bash[22591]: cluster 2024-09-10T19:11:49.915173+0000 osd.3 (osd.3) 560 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:11:51.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:51 smithi165 bash[22591]: cluster 2024-09-10T19:11:49.918646+0000 osd.3 (osd.3) 561 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:11:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:51 smithi083 bash[18629]: cluster 2024-09-10T19:11:49.915173+0000 osd.3 (osd.3) 560 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:11:51.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:51 smithi083 bash[18629]: cluster 2024-09-10T19:11:49.918646+0000 osd.3 (osd.3) 561 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:11:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:51 smithi083 bash[32381]: cluster 2024-09-10T19:11:49.915173+0000 osd.3 (osd.3) 560 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:11:51.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:51 smithi083 bash[32381]: cluster 2024-09-10T19:11:49.918646+0000 osd.3 (osd.3) 561 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:11:52.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:52 smithi165 bash[22591]: cluster 2024-09-10T19:11:50.514839+0000 mgr.y (mgr.24526) 872 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:52.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:52 smithi165 bash[22591]: cluster 2024-09-10T19:11:51.179161+0000 osd.0 (osd.0) 525 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:11:52.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:52 smithi165 bash[22591]: cluster 2024-09-10T19:11:51.184570+0000 osd.0 (osd.0) 526 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:11:52.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:52 smithi165 bash[22591]: cluster 2024-09-10T19:11:51.438671+0000 osd.4 (osd.4) 498 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:11:52.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:52 smithi165 bash[22591]: cluster 2024-09-10T19:11:51.442092+0000 osd.4 (osd.4) 499 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:11:52.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[32381]: cluster 2024-09-10T19:11:50.514839+0000 mgr.y (mgr.24526) 872 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:52.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[32381]: cluster 2024-09-10T19:11:51.179161+0000 osd.0 (osd.0) 525 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:11:52.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[32381]: cluster 2024-09-10T19:11:51.184570+0000 osd.0 (osd.0) 526 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:11:52.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[32381]: cluster 2024-09-10T19:11:51.438671+0000 osd.4 (osd.4) 498 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:11:52.688 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[32381]: cluster 2024-09-10T19:11:51.442092+0000 osd.4 (osd.4) 499 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:11:52.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18629]: cluster 2024-09-10T19:11:50.514839+0000 mgr.y (mgr.24526) 872 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:52.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18629]: cluster 2024-09-10T19:11:51.179161+0000 osd.0 (osd.0) 525 : cluster [DBG] 3.1f deep-scrub starts 2024-09-10T19:11:52.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18629]: cluster 2024-09-10T19:11:51.184570+0000 osd.0 (osd.0) 526 : cluster [DBG] 3.1f deep-scrub ok 2024-09-10T19:11:52.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18629]: cluster 2024-09-10T19:11:51.438671+0000 osd.4 (osd.4) 498 : cluster [DBG] 12.2 scrub starts 2024-09-10T19:11:52.689 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18629]: cluster 2024-09-10T19:11:51.442092+0000 osd.4 (osd.4) 499 : cluster [DBG] 12.2 scrub ok 2024-09-10T19:11:52.950 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:53.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:11:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:11:52] "GET /metrics HTTP/1.1" 200 224110 "" "Prometheus/2.43.0" 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.142454+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.146110+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.407753+0000 osd.4 (osd.4) 500 : cluster [DBG] 4.e scrub starts 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.411765+0000 osd.4 (osd.4) 501 : cluster [DBG] 4.e scrub ok 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.827045+0000 osd.5 (osd.5) 534 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:11:53.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:53 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.832765+0000 osd.5 (osd.5) 535 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.142454+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.146110+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.407753+0000 osd.4 (osd.4) 500 : cluster [DBG] 4.e scrub starts 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.411765+0000 osd.4 (osd.4) 501 : cluster [DBG] 4.e scrub ok 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.827045+0000 osd.5 (osd.5) 534 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.832765+0000 osd.5 (osd.5) 535 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.142454+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:11:53.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.146110+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:11:53.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.407753+0000 osd.4 (osd.4) 500 : cluster [DBG] 4.e scrub starts 2024-09-10T19:11:53.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.411765+0000 osd.4 (osd.4) 501 : cluster [DBG] 4.e scrub ok 2024-09-10T19:11:53.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.827045+0000 osd.5 (osd.5) 534 : cluster [DBG] 3.1d scrub starts 2024-09-10T19:11:53.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:53 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.832765+0000 osd.5 (osd.5) 535 : cluster [DBG] 3.1d scrub ok 2024-09-10T19:11:54.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:54 smithi165 bash[22591]: cluster 2024-09-10T19:11:52.516648+0000 mgr.y (mgr.24526) 873 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:54.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:54 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.373462+0000 osd.4 (osd.4) 502 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:11:54.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:54 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.376697+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[18629]: cluster 2024-09-10T19:11:52.516648+0000 mgr.y (mgr.24526) 873 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.373462+0000 osd.4 (osd.4) 502 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.376697+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[32381]: cluster 2024-09-10T19:11:52.516648+0000 mgr.y (mgr.24526) 873 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.373462+0000 osd.4 (osd.4) 502 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:11:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:54 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.376697+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.883140+0000 osd.3 (osd.3) 562 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.887150+0000 osd.3 (osd.3) 563 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.924347+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:53.927915+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.167628+0000 osd.0 (osd.0) 527 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.171598+0000 osd.0 (osd.0) 528 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:11:55.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.345360+0000 osd.4 (osd.4) 504 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:11:55.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:55 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.348662+0000 osd.4 (osd.4) 505 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.883140+0000 osd.3 (osd.3) 562 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.887150+0000 osd.3 (osd.3) 563 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.924347+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:53.927915+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.167628+0000 osd.0 (osd.0) 527 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.171598+0000 osd.0 (osd.0) 528 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.345360+0000 osd.4 (osd.4) 504 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:11:55.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.348662+0000 osd.4 (osd.4) 505 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.883140+0000 osd.3 (osd.3) 562 : cluster [DBG] 4.11 scrub starts 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.887150+0000 osd.3 (osd.3) 563 : cluster [DBG] 4.11 scrub ok 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.924347+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.7 deep-scrub starts 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:53.927915+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.7 deep-scrub ok 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.167628+0000 osd.0 (osd.0) 527 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.171598+0000 osd.0 (osd.0) 528 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.345360+0000 osd.4 (osd.4) 504 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:11:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:55 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.348662+0000 osd.4 (osd.4) 505 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:11:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.517459+0000 mgr.y (mgr.24526) 874 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.943059+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:11:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:54.946805+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:11:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.132425+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:11:56.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.137052+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.145238+0000 osd.2 (osd.2) 387 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.148242+0000 osd.2 (osd.2) 388 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.346246+0000 osd.4 (osd.4) 506 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.353069+0000 osd.4 (osd.4) 507 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.832106+0000 osd.7 (osd.7) 536 : cluster [DBG] 3.d scrub starts 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.837919+0000 osd.7 (osd.7) 537 : cluster [DBG] 3.d scrub ok 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.861275+0000 osd.5 (osd.5) 536 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:11:56.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:56 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.864588+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:11:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- 172.21.15.83:0/3409967894 >> v1:172.21.15.83:6790/0 conn(0x7f78fc108f20 legacy=0x7f78fc10b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:56.707 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- 172.21.15.83:0/3409967894 shutdown_connections 2024-09-10T19:11:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- 172.21.15.83:0/3409967894 >> 172.21.15.83:0/3409967894 conn(0x7f78fc0f81f0 msgr2=0x7f78fc0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- 172.21.15.83:0/3409967894 shutdown_connections 2024-09-10T19:11:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- 172.21.15.83:0/3409967894 wait complete. 2024-09-10T19:11:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 librados: starting msgr at 2024-09-10T19:11:56.708 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 librados: starting objecter 2024-09-10T19:11:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 Processor -- start 2024-09-10T19:11:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- start start 2024-09-10T19:11:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 librados: setting wanted keys 2024-09-10T19:11:56.709 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 librados: calling monclient init 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78fc11c0f0 con 0x7f78fc108f20 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78fc11d2f0 con 0x7f78fc105390 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f7900a4b640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78fc11e4f0 con 0x7f78fc1025d0 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78fa59c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f78fc1025d0 0x7f78fc0fe780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:39212/0 (socket says 172.21.15.83:39212) 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78fad9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f78fc108f20 0x7f78fc11a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36102/0 (socket says 172.21.15.83:36102) 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78fa59c640 1 -- 172.21.15.83:0/478674786 learned_addr learned my addr 172.21.15.83:0/478674786 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter ms_handle_connect 0x7f78fc1025d0 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:56.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter ms_handle_connect 0x7f78fc108f20 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter ms_handle_connect 0x7f78fc105390 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747138239 0 0) 0x7f78fc11e4f0 con 0x7f78fc1025d0 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e80039c0 con 0x7f78fc1025d0 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834037593 0 0) 0x7f78fc11c0f0 con 0x7f78fc108f20 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc11e4f0 con 0x7f78fc108f20 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574992824 0 0) 0x7f78fc11d2f0 con 0x7f78fc105390 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.702+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc11c0f0 con 0x7f78fc105390 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992316604 0 0) 0x7f78fc11e4f0 con 0x7f78fc108f20 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78fc11d2f0 con 0x7f78fc108f20 2024-09-10T19:11:56.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f78f4002f10 con 0x7f78fc108f20 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364308061 0 0) 0x7f78e80039c0 con 0x7f78fc1025d0 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78fc11e4f0 con 0x7f78fc1025d0 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f78ec0019f0 con 0x7f78fc1025d0 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809576691 0 0) 0x7f78fc11c0f0 con 0x7f78fc105390 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78e80039c0 con 0x7f78fc105390 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f78e40018c0 con 0x7f78fc105390 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1106183453 0 0) 0x7f78fc11d2f0 con 0x7f78fc108f20 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 >> v1:172.21.15.83:6790/0 conn(0x7f78fc1025d0 legacy=0x7f78fc0fe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 >> v1:172.21.15.165:6789/0 conn(0x7f78fc105390 legacy=0x7f78fc0fee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc11f6f0 con 0x7f78fc108f20 2024-09-10T19:11:56.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc11c2e0 con 0x7f78fc108f20 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f7900a4b640 10 client.25935.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc11c730 con 0x7f78fc108f20 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f7900a4b640 1 librados: init done 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f78f4002630 con 0x7f78fc108f20 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f78f40036f0 con 0x7f78fc108f20 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc005010 con 0x7f78fc108f20 2024-09-10T19:11:56.714 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.706+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f78f4001b70 con 0x7f78fc108f20 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(891..891 src has 838..891) v4 ==== 7363+0+0 (unknown 1348337046 0 0) 0x7f78f408a700 con 0x7f78fc108f20 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 10 client.25935.objecter ms_dispatch 0x7f78fc000b90 osd_map(891..891 src has 838..891) v4 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 3 client.25935.objecter handle_osd_map got epochs [891,891] > 0 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 3 client.25935.objecter handle_osd_map decoding full epoch 891 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 20 client.25935.objecter dump_active .. 0 homeless 2024-09-10T19:11:56.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.710+0000 7f78f2ffd640 10 client.25935.objecter ms_handle_connect 0x7f78e8075cb0 2024-09-10T19:11:56.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.714+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f78f4056f80 con 0x7f78fc108f20 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.517459+0000 mgr.y (mgr.24526) 874 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.943059+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:54.946805+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.132425+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.137052+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.145238+0000 osd.2 (osd.2) 387 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.148242+0000 osd.2 (osd.2) 388 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:11:56.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.346246+0000 osd.4 (osd.4) 506 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.353069+0000 osd.4 (osd.4) 507 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.832106+0000 osd.7 (osd.7) 536 : cluster [DBG] 3.d scrub starts 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.837919+0000 osd.7 (osd.7) 537 : cluster [DBG] 3.d scrub ok 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.861275+0000 osd.5 (osd.5) 536 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.864588+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.517459+0000 mgr.y (mgr.24526) 874 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.943059+0000 osd.1 (osd.1) 97 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:54.946805+0000 osd.1 (osd.1) 98 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.132425+0000 osd.0 (osd.0) 529 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.137052+0000 osd.0 (osd.0) 530 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:11:56.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.145238+0000 osd.2 (osd.2) 387 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.148242+0000 osd.2 (osd.2) 388 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.346246+0000 osd.4 (osd.4) 506 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.353069+0000 osd.4 (osd.4) 507 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.832106+0000 osd.7 (osd.7) 536 : cluster [DBG] 3.d scrub starts 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.837919+0000 osd.7 (osd.7) 537 : cluster [DBG] 3.d scrub ok 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.861275+0000 osd.5 (osd.5) 536 : cluster [DBG] 2.2 scrub starts 2024-09-10T19:11:56.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:56 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.864588+0000 osd.5 (osd.5) 537 : cluster [DBG] 2.2 scrub ok 2024-09-10T19:11:56.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:56.898+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f78dc0052c0 con 0x7f78fc108f20 2024-09-10T19:11:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.354+0000 7f78f2ffd640 1 -- 172.21.15.83:0/478674786 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v892) v1 ==== 113+0+0 (unknown 3241727336 0 0) 0x7f78f4056920 con 0x7f78fc108f20 2024-09-10T19:11:57.363 INFO:teuthology.orchestra.run.smithi083.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 10 librados: watch_flush enter 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 10 librados: watch_flush exit 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 20 client.25935.objecter shutdown clearing up homeless session... 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 10 client.25935.objecter successfully canceled tick 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 >> v1:172.21.15.83:6800/3071793735 conn(0x7f78e8075cb0 legacy=0x7f78e8078160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:57.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 >> v1:172.21.15.83:6789/0 conn(0x7f78fc108f20 legacy=0x7f78fc11a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:11:57.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 shutdown_connections 2024-09-10T19:11:57.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 >> 172.21.15.83:0/478674786 conn(0x7f78fc0f81f0 msgr2=0x7f78fc0f85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:11:57.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 shutdown_connections 2024-09-10T19:11:57.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.362+0000 7f7900a4b640 1 -- 172.21.15.83:0/478674786 wait complete. 2024-09-10T19:11:57.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:11:57.366+0000 7f7900a4b640 1 librados: shutdown 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.920786+0000 osd.3 (osd.3) 564 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[32381]: cluster 2024-09-10T19:11:55.924330+0000 osd.3 (osd.3) 565 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.790774+0000 osd.7 (osd.7) 538 : cluster [DBG] 3.e scrub starts 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.795673+0000 osd.7 (osd.7) 539 : cluster [DBG] 3.e scrub ok 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[32381]: audit 2024-09-10T19:11:56.903279+0000 mon.a (mon.0) 3871 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:11:57.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.920786+0000 osd.3 (osd.3) 564 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:11:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[18629]: cluster 2024-09-10T19:11:55.924330+0000 osd.3 (osd.3) 565 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:11:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.790774+0000 osd.7 (osd.7) 538 : cluster [DBG] 3.e scrub starts 2024-09-10T19:11:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.795673+0000 osd.7 (osd.7) 539 : cluster [DBG] 3.e scrub ok 2024-09-10T19:11:57.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:57 smithi083 bash[18629]: audit 2024-09-10T19:11:56.903279+0000 mon.a (mon.0) 3871 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:11:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:57 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.920786+0000 osd.3 (osd.3) 564 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:11:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:57 smithi165 bash[22591]: cluster 2024-09-10T19:11:55.924330+0000 osd.3 (osd.3) 565 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:11:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:57 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.790774+0000 osd.7 (osd.7) 538 : cluster [DBG] 3.e scrub starts 2024-09-10T19:11:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:57 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.795673+0000 osd.7 (osd.7) 539 : cluster [DBG] 3.e scrub ok 2024-09-10T19:11:57.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:57 smithi165 bash[22591]: audit 2024-09-10T19:11:56.903279+0000 mon.a (mon.0) 3871 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:11:58.331 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-10T19:11:58.652 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.519327+0000 mgr.y (mgr.24526) 875 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.942154+0000 osd.3 (osd.3) 566 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.945082+0000 osd.3 (osd.3) 567 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.974836+0000 osd.2 (osd.2) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:56.975329+0000 osd.2 (osd.2) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.100925+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.106034+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.183605+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.187582+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: audit 2024-09-10T19:11:57.361474+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.361621+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-09-10T19:11:58.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.627400+0000 osd.6 (osd.6) 495 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:11:58.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.631553+0000 osd.6 (osd.6) 496 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:11:58.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.760788+0000 osd.7 (osd.7) 540 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:11:58.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.764757+0000 osd.7 (osd.7) 541 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:11:58.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.857249+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.12 deep-scrub starts 2024-09-10T19:11:58.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[22591]: cluster 2024-09-10T19:11:57.860684+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.12 deep-scrub ok 2024-09-10T19:11:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.519327+0000 mgr.y (mgr.24526) 875 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:58.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.942154+0000 osd.3 (osd.3) 566 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.945082+0000 osd.3 (osd.3) 567 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.974836+0000 osd.2 (osd.2) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:56.975329+0000 osd.2 (osd.2) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.100925+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.106034+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.183605+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.187582+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: audit 2024-09-10T19:11:57.361474+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.361621+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.627400+0000 osd.6 (osd.6) 495 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.631553+0000 osd.6 (osd.6) 496 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:11:58.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.760788+0000 osd.7 (osd.7) 540 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.764757+0000 osd.7 (osd.7) 541 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.857249+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.12 deep-scrub starts 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[32381]: cluster 2024-09-10T19:11:57.860684+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.12 deep-scrub ok 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.519327+0000 mgr.y (mgr.24526) 875 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.942154+0000 osd.3 (osd.3) 566 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.945082+0000 osd.3 (osd.3) 567 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.974836+0000 osd.2 (osd.2) 389 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:56.975329+0000 osd.2 (osd.2) 390 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.100925+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:11:58.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.106034+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.183605+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.187582+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: audit 2024-09-10T19:11:57.361474+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? v1:172.21.15.83:0/478674786' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.361621+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.627400+0000 osd.6 (osd.6) 495 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.631553+0000 osd.6 (osd.6) 496 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.760788+0000 osd.7 (osd.7) 540 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.764757+0000 osd.7 (osd.7) 541 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.857249+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.12 deep-scrub starts 2024-09-10T19:11:58.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:58 smithi083 bash[18629]: cluster 2024-09-10T19:11:57.860684+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.12 deep-scrub ok 2024-09-10T19:11:59.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:11:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[18629]: cluster 2024-09-10T19:11:58.067520+0000 osd.0 (osd.0) 533 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[18629]: cluster 2024-09-10T19:11:58.071716+0000 osd.0 (osd.0) 534 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[18629]: cluster 2024-09-10T19:11:58.348195+0000 osd.4 (osd.4) 508 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[18629]: cluster 2024-09-10T19:11:58.353303+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[32381]: cluster 2024-09-10T19:11:58.067520+0000 osd.0 (osd.0) 533 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[32381]: cluster 2024-09-10T19:11:58.071716+0000 osd.0 (osd.0) 534 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:11:59.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[32381]: cluster 2024-09-10T19:11:58.348195+0000 osd.4 (osd.4) 508 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:11:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:11:59 smithi083 bash[32381]: cluster 2024-09-10T19:11:58.353303+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:11:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:59 smithi165 bash[22591]: cluster 2024-09-10T19:11:58.067520+0000 osd.0 (osd.0) 533 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:11:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:59 smithi165 bash[22591]: cluster 2024-09-10T19:11:58.071716+0000 osd.0 (osd.0) 534 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:11:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:59 smithi165 bash[22591]: cluster 2024-09-10T19:11:58.348195+0000 osd.4 (osd.4) 508 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:11:59.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:11:59 smithi165 bash[22591]: cluster 2024-09-10T19:11:58.353303+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:12:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[18629]: cluster 2024-09-10T19:11:58.519896+0000 mgr.y (mgr.24526) 876 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[18629]: audit 2024-09-10T19:11:58.650864+0000 mgr.y (mgr.24526) 877 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[18629]: audit 2024-09-10T19:11:59.582266+0000 mon.c (mon.2) 917 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:00.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[32381]: cluster 2024-09-10T19:11:58.519896+0000 mgr.y (mgr.24526) 876 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[32381]: audit 2024-09-10T19:11:58.650864+0000 mgr.y (mgr.24526) 877 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:00 smithi083 bash[32381]: audit 2024-09-10T19:11:59.582266+0000 mon.c (mon.2) 917 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:00 smithi165 bash[22591]: cluster 2024-09-10T19:11:58.519896+0000 mgr.y (mgr.24526) 876 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:00 smithi165 bash[22591]: audit 2024-09-10T19:11:58.650864+0000 mgr.y (mgr.24526) 877 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:00 smithi165 bash[22591]: audit 2024-09-10T19:11:59.582266+0000 mon.c (mon.2) 917 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:59.977009+0000 osd.3 (osd.3) 568 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:12:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:11:59.982179+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:12:01.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:12:00.823957+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:12:00.827314+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:12:00.933126+0000 osd.5 (osd.5) 540 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[18629]: cluster 2024-09-10T19:12:00.936913+0000 osd.5 (osd.5) 541 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:59.977009+0000 osd.3 (osd.3) 568 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:11:59.982179+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:12:00.823957+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:12:00.827314+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:12:00.933126+0000 osd.5 (osd.5) 540 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:12:01.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:01 smithi083 bash[32381]: cluster 2024-09-10T19:12:00.936913+0000 osd.5 (osd.5) 541 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:59.977009+0000 osd.3 (osd.3) 568 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:11:59.982179+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:12:00.823957+0000 osd.7 (osd.7) 542 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:12:00.827314+0000 osd.7 (osd.7) 543 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:12:01.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:12:00.933126+0000 osd.5 (osd.5) 540 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:12:01.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:01 smithi165 bash[22591]: cluster 2024-09-10T19:12:00.936913+0000 osd.5 (osd.5) 541 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:12:02.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[18629]: cluster 2024-09-10T19:12:00.520623+0000 mgr.y (mgr.24526) 878 : cluster [DBG] pgmap v1458: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:02.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[18629]: cluster 2024-09-10T19:12:01.870667+0000 osd.7 (osd.7) 544 : cluster [DBG] 6.5 deep-scrub starts 2024-09-10T19:12:02.688 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[18629]: cluster 2024-09-10T19:12:01.874572+0000 osd.7 (osd.7) 545 : cluster [DBG] 6.5 deep-scrub ok 2024-09-10T19:12:02.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[32381]: cluster 2024-09-10T19:12:00.520623+0000 mgr.y (mgr.24526) 878 : cluster [DBG] pgmap v1458: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:02.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[32381]: cluster 2024-09-10T19:12:01.870667+0000 osd.7 (osd.7) 544 : cluster [DBG] 6.5 deep-scrub starts 2024-09-10T19:12:02.689 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[32381]: cluster 2024-09-10T19:12:01.874572+0000 osd.7 (osd.7) 545 : cluster [DBG] 6.5 deep-scrub ok 2024-09-10T19:12:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:02 smithi165 bash[22591]: cluster 2024-09-10T19:12:00.520623+0000 mgr.y (mgr.24526) 878 : cluster [DBG] pgmap v1458: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:02 smithi165 bash[22591]: cluster 2024-09-10T19:12:01.870667+0000 osd.7 (osd.7) 544 : cluster [DBG] 6.5 deep-scrub starts 2024-09-10T19:12:02.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:02 smithi165 bash[22591]: cluster 2024-09-10T19:12:01.874572+0000 osd.7 (osd.7) 545 : cluster [DBG] 6.5 deep-scrub ok 2024-09-10T19:12:03.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:02] "GET /metrics HTTP/1.1" 200 224102 "" "Prometheus/2.43.0" 2024-09-10T19:12:03.099 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[18629]: cluster 2024-09-10T19:12:01.941990+0000 osd.3 (osd.3) 570 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[18629]: cluster 2024-09-10T19:12:01.945441+0000 osd.3 (osd.3) 571 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[18629]: cluster 2024-09-10T19:12:02.891002+0000 osd.7 (osd.7) 546 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[18629]: cluster 2024-09-10T19:12:02.895366+0000 osd.7 (osd.7) 547 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[32381]: cluster 2024-09-10T19:12:01.941990+0000 osd.3 (osd.3) 570 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[32381]: cluster 2024-09-10T19:12:01.945441+0000 osd.3 (osd.3) 571 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:12:03.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[32381]: cluster 2024-09-10T19:12:02.891002+0000 osd.7 (osd.7) 546 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:12:03.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:03 smithi083 bash[32381]: cluster 2024-09-10T19:12:02.895366+0000 osd.7 (osd.7) 547 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:12:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:03 smithi165 bash[22591]: cluster 2024-09-10T19:12:01.941990+0000 osd.3 (osd.3) 570 : cluster [DBG] 6.11 scrub starts 2024-09-10T19:12:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:03 smithi165 bash[22591]: cluster 2024-09-10T19:12:01.945441+0000 osd.3 (osd.3) 571 : cluster [DBG] 6.11 scrub ok 2024-09-10T19:12:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:03 smithi165 bash[22591]: cluster 2024-09-10T19:12:02.891002+0000 osd.7 (osd.7) 546 : cluster [DBG] 12.d deep-scrub starts 2024-09-10T19:12:03.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:03 smithi165 bash[22591]: cluster 2024-09-10T19:12:02.895366+0000 osd.7 (osd.7) 547 : cluster [DBG] 12.d deep-scrub ok 2024-09-10T19:12:04.731 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[18629]: cluster 2024-09-10T19:12:02.522575+0000 mgr.y (mgr.24526) 879 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:04.732 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.868528+0000 osd.7 (osd.7) 548 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:12:04.732 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.873346+0000 osd.7 (osd.7) 549 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:12:04.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[32381]: cluster 2024-09-10T19:12:02.522575+0000 mgr.y (mgr.24526) 879 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:04.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.868528+0000 osd.7 (osd.7) 548 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:12:04.732 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:04 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.873346+0000 osd.7 (osd.7) 549 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:12:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:04 smithi165 bash[22591]: cluster 2024-09-10T19:12:02.522575+0000 mgr.y (mgr.24526) 879 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:04 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.868528+0000 osd.7 (osd.7) 548 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:12:04.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:04 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.873346+0000 osd.7 (osd.7) 549 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:12:05.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.928186+0000 osd.3 (osd.3) 572 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:12:05.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.932088+0000 osd.3 (osd.3) 573 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:12:05.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.980367+0000 osd.1 (osd.1) 99 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:12:05.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:03.983859+0000 osd.1 (osd.1) 100 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:12:05.694 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:04.458636+0000 osd.4 (osd.4) 510 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:04.462320+0000 osd.4 (osd.4) 511 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:04.853309+0000 osd.5 (osd.5) 542 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[18629]: cluster 2024-09-10T19:12:04.856893+0000 osd.5 (osd.5) 543 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.928186+0000 osd.3 (osd.3) 572 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.932088+0000 osd.3 (osd.3) 573 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.980367+0000 osd.1 (osd.1) 99 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:03.983859+0000 osd.1 (osd.1) 100 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:04.458636+0000 osd.4 (osd.4) 510 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:04.462320+0000 osd.4 (osd.4) 511 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:04.853309+0000 osd.5 (osd.5) 542 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:12:05.695 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:05 smithi083 bash[32381]: cluster 2024-09-10T19:12:04.856893+0000 osd.5 (osd.5) 543 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.928186+0000 osd.3 (osd.3) 572 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.932088+0000 osd.3 (osd.3) 573 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.980367+0000 osd.1 (osd.1) 99 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:03.983859+0000 osd.1 (osd.1) 100 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:04.458636+0000 osd.4 (osd.4) 510 : cluster [DBG] 12.1e deep-scrub starts 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:04.462320+0000 osd.4 (osd.4) 511 : cluster [DBG] 12.1e deep-scrub ok 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:04.853309+0000 osd.5 (osd.5) 542 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:12:05.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:05 smithi165 bash[22591]: cluster 2024-09-10T19:12:04.856893+0000 osd.5 (osd.5) 543 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:12:06.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[32381]: cluster 2024-09-10T19:12:04.523396+0000 mgr.y (mgr.24526) 880 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:06.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[32381]: cluster 2024-09-10T19:12:05.013224+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:12:06.778 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[32381]: cluster 2024-09-10T19:12:05.017587+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:12:06.779 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[18629]: cluster 2024-09-10T19:12:04.523396+0000 mgr.y (mgr.24526) 880 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:06.779 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[18629]: cluster 2024-09-10T19:12:05.013224+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:12:06.779 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:06 smithi083 bash[18629]: cluster 2024-09-10T19:12:05.017587+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:12:06.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:06 smithi165 bash[22591]: cluster 2024-09-10T19:12:04.523396+0000 mgr.y (mgr.24526) 880 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:06.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:06 smithi165 bash[22591]: cluster 2024-09-10T19:12:05.013224+0000 osd.1 (osd.1) 101 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:12:06.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:06 smithi165 bash[22591]: cluster 2024-09-10T19:12:05.017587+0000 osd.1 (osd.1) 102 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:12:06.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- 172.21.15.83:0/1560044242 >> v1:172.21.15.83:6789/0 conn(0x7efe2c1042a0 legacy=0x7efe2c106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- 172.21.15.83:0/1560044242 shutdown_connections 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- 172.21.15.83:0/1560044242 >> 172.21.15.83:0/1560044242 conn(0x7efe2c0f70b0 msgr2=0x7efe2c0f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- 172.21.15.83:0/1560044242 shutdown_connections 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- 172.21.15.83:0/1560044242 wait complete. 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 librados: starting msgr at 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 librados: starting objecter 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 Processor -- start 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- start start 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 librados: setting wanted keys 2024-09-10T19:12:06.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 librados: calling monclient init 2024-09-10T19:12:06.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe2c11aee0 con 0x7efe2c107e00 2024-09-10T19:12:06.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe2c11c0e0 con 0x7efe2c1042a0 2024-09-10T19:12:06.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe34761640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe2c11d2e0 con 0x7efe2c101530 2024-09-10T19:12:06.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe324fd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7efe2c101530 0x7efe2c104830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:55552/0 (socket says 172.21.15.83:55552) 2024-09-10T19:12:06.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe324fd640 1 -- 172.21.15.83:0/2774650232 learned_addr learned my addr 172.21.15.83:0/2774650232 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter ms_handle_connect 0x7efe2c101530 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter ms_handle_connect 0x7efe2c107e00 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter ms_handle_connect 0x7efe2c1042a0 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082599741 0 0) 0x7efe2c11d2e0 con 0x7efe2c101530 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe14003720 con 0x7efe2c101530 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985152513 0 0) 0x7efe2c11aee0 con 0x7efe2c107e00 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2c11d2e0 con 0x7efe2c107e00 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028296532 0 0) 0x7efe2c11c0e0 con 0x7efe2c1042a0 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2c11aee0 con 0x7efe2c1042a0 2024-09-10T19:12:06.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426655218 0 0) 0x7efe14003720 con 0x7efe2c101530 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe2c11c0e0 con 0x7efe2c101530 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2748713483 0 0) 0x7efe2c11d2e0 con 0x7efe2c107e00 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe14003720 con 0x7efe2c107e00 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe1c002950 con 0x7efe2c101530 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe24001840 con 0x7efe2c107e00 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203996156 0 0) 0x7efe2c11aee0 con 0x7efe2c1042a0 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe2c11d2e0 con 0x7efe2c1042a0 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe280023f0 con 0x7efe2c1042a0 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3061612909 0 0) 0x7efe2c11c0e0 con 0x7efe2c101530 2024-09-10T19:12:06.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 >> v1:172.21.15.165:6789/0 conn(0x7efe2c1042a0 legacy=0x7efe2c104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 >> v1:172.21.15.83:6789/0 conn(0x7efe2c107e00 legacy=0x7efe2c1195e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.958+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c11e4e0 con 0x7efe2c101530 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c11b0d0 con 0x7efe2c101530 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7efe1c001750 con 0x7efe2c101530 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7efe1c005ad0 con 0x7efe2c101530 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe34761640 10 client.50027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c11b5a0 con 0x7efe2c101530 2024-09-10T19:12:06.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe34761640 1 librados: init done 2024-09-10T19:12:06.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c0fcc30 con 0x7efe2c101530 2024-09-10T19:12:06.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7efe1c005d60 con 0x7efe2c101530 2024-09-10T19:12:06.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(892..892 src has 838..892) v4 ==== 7363+0+0 (unknown 2104240223 0 0) 0x7efe1c08a340 con 0x7efe2c101530 2024-09-10T19:12:06.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 10 client.50027.objecter ms_dispatch 0x7efe2c000b90 osd_map(892..892 src has 838..892) v4 2024-09-10T19:12:06.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 3 client.50027.objecter handle_osd_map got epochs [892,892] > 0 2024-09-10T19:12:06.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 3 client.50027.objecter handle_osd_map decoding full epoch 892 2024-09-10T19:12:06.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.962+0000 7efe22ffd640 20 client.50027.objecter dump_active .. 0 homeless 2024-09-10T19:12:06.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.966+0000 7efe22ffd640 10 client.50027.objecter ms_handle_connect 0x7efe14073a40 2024-09-10T19:12:06.979 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:06.974+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7efe1c056b80 con 0x7efe2c101530 2024-09-10T19:12:07.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.154+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7efe2c105680 con 0x7efe2c101530 2024-09-10T19:12:07.425 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.418+0000 7efe22ffd640 1 -- 172.21.15.83:0/2774650232 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v893) v1 ==== 133+0+0 (unknown 3972264857 0 0) 0x7efe1c056520 con 0x7efe2c101530 2024-09-10T19:12:07.425 INFO:teuthology.orchestra.run.smithi083.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-10T19:12:07.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 10 librados: watch_flush enter 2024-09-10T19:12:07.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 10 librados: watch_flush exit 2024-09-10T19:12:07.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 20 client.50027.objecter shutdown clearing up homeless session... 2024-09-10T19:12:07.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 10 client.50027.objecter successfully canceled tick 2024-09-10T19:12:07.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 >> v1:172.21.15.83:6800/3071793735 conn(0x7efe14073a40 legacy=0x7efe14075ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:07.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 >> v1:172.21.15.83:6790/0 conn(0x7efe2c101530 legacy=0x7efe2c104830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:07.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 shutdown_connections 2024-09-10T19:12:07.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 >> 172.21.15.83:0/2774650232 conn(0x7efe2c0f70b0 msgr2=0x7efe2c0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:07.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 shutdown_connections 2024-09-10T19:12:07.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 -- 172.21.15.83:0/2774650232 wait complete. 2024-09-10T19:12:07.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:07.422+0000 7efe34761640 1 librados: shutdown 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: cluster 2024-09-10T19:12:05.959680+0000 osd.3 (osd.3) 574 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: cluster 2024-09-10T19:12:05.963360+0000 osd.3 (osd.3) 575 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: cluster 2024-09-10T19:12:06.473082+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.1a deep-scrub starts 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: cluster 2024-09-10T19:12:06.476570+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a deep-scrub ok 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: audit 2024-09-10T19:12:07.162283+0000 mon.c (mon.2) 918 : audit [INF] from='client.? v1:172.21.15.83:0/2774650232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[32381]: audit 2024-09-10T19:12:07.163017+0000 mon.a (mon.0) 3874 : audit [INF] from='client.50027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:07.697 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: cluster 2024-09-10T19:12:05.959680+0000 osd.3 (osd.3) 574 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:12:07.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: cluster 2024-09-10T19:12:05.963360+0000 osd.3 (osd.3) 575 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:12:07.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: cluster 2024-09-10T19:12:06.473082+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.1a deep-scrub starts 2024-09-10T19:12:07.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: cluster 2024-09-10T19:12:06.476570+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a deep-scrub ok 2024-09-10T19:12:07.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: audit 2024-09-10T19:12:07.162283+0000 mon.c (mon.2) 918 : audit [INF] from='client.? v1:172.21.15.83:0/2774650232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:07.698 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:07 smithi083 bash[18629]: audit 2024-09-10T19:12:07.163017+0000 mon.a (mon.0) 3874 : audit [INF] from='client.50027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: cluster 2024-09-10T19:12:05.959680+0000 osd.3 (osd.3) 574 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: cluster 2024-09-10T19:12:05.963360+0000 osd.3 (osd.3) 575 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: cluster 2024-09-10T19:12:06.473082+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.1a deep-scrub starts 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: cluster 2024-09-10T19:12:06.476570+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a deep-scrub ok 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: audit 2024-09-10T19:12:07.162283+0000 mon.c (mon.2) 918 : audit [INF] from='client.? v1:172.21.15.83:0/2774650232' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:07.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:07 smithi165 bash[22591]: audit 2024-09-10T19:12:07.163017+0000 mon.a (mon.0) 3874 : audit [INF] from='client.50027 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:12:08.329 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-10T19:12:08.329 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-10T19:12:08.329 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-10T19:12:08.329 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-10T19:12:08.330 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T19:12:08.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:06.525240+0000 mgr.y (mgr.24526) 881 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:06.942683+0000 osd.3 (osd.3) 576 : cluster [DBG] 12.b scrub starts 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:06.944857+0000 osd.3 (osd.3) 577 : cluster [DBG] 12.b scrub ok 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: audit 2024-09-10T19:12:07.421217+0000 mon.a (mon.0) 3875 : audit [INF] from='client.50027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:07.421357+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:07.839557+0000 osd.7 (osd.7) 550 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: cluster 2024-09-10T19:12:07.845951+0000 osd.7 (osd.7) 551 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[18629]: audit 2024-09-10T19:12:08.114622+0000 mon.c (mon.2) 919 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:08.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:06.525240+0000 mgr.y (mgr.24526) 881 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:06.942683+0000 osd.3 (osd.3) 576 : cluster [DBG] 12.b scrub starts 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:06.944857+0000 osd.3 (osd.3) 577 : cluster [DBG] 12.b scrub ok 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: audit 2024-09-10T19:12:07.421217+0000 mon.a (mon.0) 3875 : audit [INF] from='client.50027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:07.421357+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:07.839557+0000 osd.7 (osd.7) 550 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: cluster 2024-09-10T19:12:07.845951+0000 osd.7 (osd.7) 551 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:12:08.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:08 smithi083 bash[32381]: audit 2024-09-10T19:12:08.114622+0000 mon.c (mon.2) 919 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:08.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:06.525240+0000 mgr.y (mgr.24526) 881 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:06.942683+0000 osd.3 (osd.3) 576 : cluster [DBG] 12.b scrub starts 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:06.944857+0000 osd.3 (osd.3) 577 : cluster [DBG] 12.b scrub ok 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: audit 2024-09-10T19:12:07.421217+0000 mon.a (mon.0) 3875 : audit [INF] from='client.50027 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:07.421357+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-09-10T19:12:08.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:07.839557+0000 osd.7 (osd.7) 550 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:12:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: cluster 2024-09-10T19:12:07.845951+0000 osd.7 (osd.7) 551 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:12:08.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:08 smithi165 bash[22591]: audit 2024-09-10T19:12:08.114622+0000 mon.c (mon.2) 919 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:07.979055+0000 osd.3 (osd.3) 578 : cluster [DBG] 3.b scrub starts 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:07.985705+0000 osd.3 (osd.3) 579 : cluster [DBG] 3.b scrub ok 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.803020+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.806355+0000 osd.5 (osd.5) 545 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.826266+0000 osd.7 (osd.7) 552 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:12:09.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.831625+0000 osd.7 (osd.7) 553 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:07.979055+0000 osd.3 (osd.3) 578 : cluster [DBG] 3.b scrub starts 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:07.985705+0000 osd.3 (osd.3) 579 : cluster [DBG] 3.b scrub ok 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.803020+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.806355+0000 osd.5 (osd.5) 545 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.826266+0000 osd.7 (osd.7) 552 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:12:09.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:09 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.831625+0000 osd.7 (osd.7) 553 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:07.979055+0000 osd.3 (osd.3) 578 : cluster [DBG] 3.b scrub starts 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:07.985705+0000 osd.3 (osd.3) 579 : cluster [DBG] 3.b scrub ok 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.803020+0000 osd.5 (osd.5) 544 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.806355+0000 osd.5 (osd.5) 545 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.826266+0000 osd.7 (osd.7) 552 : cluster [DBG] 3.13 deep-scrub starts 2024-09-10T19:12:09.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:09 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.831625+0000 osd.7 (osd.7) 553 : cluster [DBG] 3.13 deep-scrub ok 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.525972+0000 mgr.y (mgr.24526) 882 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: audit 2024-09-10T19:12:08.662175+0000 mgr.y (mgr.24526) 883 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.935929+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:08.942291+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.143463+0000 osd.2 (osd.2) 393 : cluster [DBG] 2.c scrub starts 2024-09-10T19:12:10.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.147260+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.c scrub ok 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.475309+0000 osd.4 (osd.4) 514 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.479937+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.530872+0000 osd.6 (osd.6) 497 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.534588+0000 osd.6 (osd.6) 498 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.855026+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.858526+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.525972+0000 mgr.y (mgr.24526) 882 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: audit 2024-09-10T19:12:08.662175+0000 mgr.y (mgr.24526) 883 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.935929+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:08.942291+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.143463+0000 osd.2 (osd.2) 393 : cluster [DBG] 2.c scrub starts 2024-09-10T19:12:10.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.147260+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.c scrub ok 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.475309+0000 osd.4 (osd.4) 514 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.479937+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.530872+0000 osd.6 (osd.6) 497 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.534588+0000 osd.6 (osd.6) 498 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.855026+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:12:10.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:10 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.858526+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.525972+0000 mgr.y (mgr.24526) 882 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: audit 2024-09-10T19:12:08.662175+0000 mgr.y (mgr.24526) 883 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.935929+0000 osd.1 (osd.1) 103 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:08.942291+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.143463+0000 osd.2 (osd.2) 393 : cluster [DBG] 2.c scrub starts 2024-09-10T19:12:10.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.147260+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.c scrub ok 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.475309+0000 osd.4 (osd.4) 514 : cluster [DBG] 3.9 scrub starts 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.479937+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.9 scrub ok 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.530872+0000 osd.6 (osd.6) 497 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.534588+0000 osd.6 (osd.6) 498 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.855026+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:12:10.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:10 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.858526+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.974999+0000 osd.1 (osd.1) 105 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.978554+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.985572+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.c scrub starts 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:09.989760+0000 osd.3 (osd.3) 581 : cluster [DBG] 6.c scrub ok 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:10.511059+0000 osd.6 (osd.6) 499 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:12:11.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[18629]: cluster 2024-09-10T19:12:10.515240+0000 osd.6 (osd.6) 500 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.974999+0000 osd.1 (osd.1) 105 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.978554+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.985572+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.c scrub starts 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:09.989760+0000 osd.3 (osd.3) 581 : cluster [DBG] 6.c scrub ok 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:10.511059+0000 osd.6 (osd.6) 499 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:12:11.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:11 smithi083 bash[32381]: cluster 2024-09-10T19:12:10.515240+0000 osd.6 (osd.6) 500 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:12:11.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:11.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.974999+0000 osd.1 (osd.1) 105 : cluster [DBG] 6.4 deep-scrub starts 2024-09-10T19:12:11.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.978554+0000 osd.1 (osd.1) 106 : cluster [DBG] 6.4 deep-scrub ok 2024-09-10T19:12:11.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.985572+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.c scrub starts 2024-09-10T19:12:11.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:09.989760+0000 osd.3 (osd.3) 581 : cluster [DBG] 6.c scrub ok 2024-09-10T19:12:11.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:10.511059+0000 osd.6 (osd.6) 499 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:12:11.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:11 smithi165 bash[22591]: cluster 2024-09-10T19:12:10.515240+0000 osd.6 (osd.6) 500 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:12] "GET /metrics HTTP/1.1" 200 224102 "" "Prometheus/2.43.0" 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:10.526764+0000 mgr.y (mgr.24526) 884 : cluster [DBG] pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.021922+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.d scrub starts 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.025510+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.d scrub ok 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.084009+0000 osd.0 (osd.0) 535 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:12:12.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.087605+0000 osd.0 (osd.0) 536 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.528985+0000 osd.6 (osd.6) 501 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.533074+0000 osd.6 (osd.6) 502 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.767798+0000 osd.5 (osd.5) 546 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.771284+0000 osd.5 (osd.5) 547 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.895505+0000 osd.7 (osd.7) 556 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[32381]: cluster 2024-09-10T19:12:11.900701+0000 osd.7 (osd.7) 557 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:10.526764+0000 mgr.y (mgr.24526) 884 : cluster [DBG] pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.021922+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.d scrub starts 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.025510+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.d scrub ok 2024-09-10T19:12:12.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.084009+0000 osd.0 (osd.0) 535 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.087605+0000 osd.0 (osd.0) 536 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.528985+0000 osd.6 (osd.6) 501 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.533074+0000 osd.6 (osd.6) 502 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.767798+0000 osd.5 (osd.5) 546 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.771284+0000 osd.5 (osd.5) 547 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.895505+0000 osd.7 (osd.7) 556 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:12:12.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:12 smithi083 bash[18629]: cluster 2024-09-10T19:12:11.900701+0000 osd.7 (osd.7) 557 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:10.526764+0000 mgr.y (mgr.24526) 884 : cluster [DBG] pgmap v1464: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.021922+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.d scrub starts 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.025510+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.d scrub ok 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.084009+0000 osd.0 (osd.0) 535 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.087605+0000 osd.0 (osd.0) 536 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.528985+0000 osd.6 (osd.6) 501 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:12:12.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.533074+0000 osd.6 (osd.6) 502 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:12:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.767798+0000 osd.5 (osd.5) 546 : cluster [DBG] 6.d deep-scrub starts 2024-09-10T19:12:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.771284+0000 osd.5 (osd.5) 547 : cluster [DBG] 6.d deep-scrub ok 2024-09-10T19:12:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.895505+0000 osd.7 (osd.7) 556 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:12:12.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:12 smithi165 bash[22591]: cluster 2024-09-10T19:12:11.900701+0000 osd.7 (osd.7) 557 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:12:12.920 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:13.790 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:13 smithi083 bash[18629]: cluster 2024-09-10T19:12:12.528595+0000 mgr.y (mgr.24526) 885 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:13 smithi083 bash[32381]: cluster 2024-09-10T19:12:12.528595+0000 mgr.y (mgr.24526) 885 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:13.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:13 smithi165 bash[22591]: cluster 2024-09-10T19:12:12.528595+0000 mgr.y (mgr.24526) 885 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: cluster 2024-09-10T19:12:14.070297+0000 osd.0 (osd.0) 537 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: cluster 2024-09-10T19:12:14.074823+0000 osd.0 (osd.0) 538 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.168570+0000 mon.a (mon.0) 3877 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.177080+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.303188+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.312210+0000 mon.a (mon.0) 3880 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.582042+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.697813+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:15.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.698871+0000 mon.c (mon.2) 922 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[32381]: audit 2024-09-10T19:12:14.708552+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: cluster 2024-09-10T19:12:14.070297+0000 osd.0 (osd.0) 537 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: cluster 2024-09-10T19:12:14.074823+0000 osd.0 (osd.0) 538 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.168570+0000 mon.a (mon.0) 3877 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.177080+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.303188+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.312210+0000 mon.a (mon.0) 3880 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.582042+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.697813+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:15.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.698871+0000 mon.c (mon.2) 922 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:15.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:15 smithi083 bash[18629]: audit 2024-09-10T19:12:14.708552+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: cluster 2024-09-10T19:12:14.070297+0000 osd.0 (osd.0) 537 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: cluster 2024-09-10T19:12:14.074823+0000 osd.0 (osd.0) 538 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.168570+0000 mon.a (mon.0) 3877 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.177080+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.303188+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.312210+0000 mon.a (mon.0) 3880 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.582042+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.697813+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.698871+0000 mon.c (mon.2) 922 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:15.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:15 smithi165 bash[22591]: audit 2024-09-10T19:12:14.708552+0000 mon.a (mon.0) 3881 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[32381]: cluster 2024-09-10T19:12:14.505306+0000 osd.6 (osd.6) 503 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[32381]: cluster 2024-09-10T19:12:14.507939+0000 osd.6 (osd.6) 504 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[32381]: cluster 2024-09-10T19:12:14.529155+0000 mgr.y (mgr.24526) 886 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.176497+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.178652+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:12:16.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[18629]: cluster 2024-09-10T19:12:14.505306+0000 osd.6 (osd.6) 503 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:12:16.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[18629]: cluster 2024-09-10T19:12:14.507939+0000 osd.6 (osd.6) 504 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:12:16.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[18629]: cluster 2024-09-10T19:12:14.529155+0000 mgr.y (mgr.24526) 886 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:16.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.176497+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:12:16.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:16 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.178652+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:12:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:16 smithi165 bash[22591]: cluster 2024-09-10T19:12:14.505306+0000 osd.6 (osd.6) 503 : cluster [DBG] 12.17 scrub starts 2024-09-10T19:12:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:16 smithi165 bash[22591]: cluster 2024-09-10T19:12:14.507939+0000 osd.6 (osd.6) 504 : cluster [DBG] 12.17 scrub ok 2024-09-10T19:12:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:16 smithi165 bash[22591]: cluster 2024-09-10T19:12:14.529155+0000 mgr.y (mgr.24526) 886 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:16 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.176497+0000 osd.2 (osd.2) 395 : cluster [DBG] 4.1c scrub starts 2024-09-10T19:12:16.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:16 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.178652+0000 osd.2 (osd.2) 396 : cluster [DBG] 4.1c scrub ok 2024-09-10T19:12:17.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 -- 172.21.15.83:0/2323116311 >> v1:172.21.15.83:6789/0 conn(0x7fa96c10f680 legacy=0x7fa96c111a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:17.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 -- 172.21.15.83:0/2323116311 shutdown_connections 2024-09-10T19:12:17.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 -- 172.21.15.83:0/2323116311 >> 172.21.15.83:0/2323116311 conn(0x7fa96c0fa370 msgr2=0x7fa96c0fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:17.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 -- 172.21.15.83:0/2323116311 shutdown_connections 2024-09-10T19:12:17.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 -- 172.21.15.83:0/2323116311 wait complete. 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 librados: starting msgr at 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 librados: starting objecter 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.066+0000 7fa9742af640 1 Processor -- start 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- start start 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 librados: setting wanted keys 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 librados: calling monclient init 2024-09-10T19:12:17.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa96c122660 con 0x7fa96c10f680 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa96c123860 con 0x7fa96c072720 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa96c124a60 con 0x7fa96c071c40 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa97284c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa96c10f680 0x7fa96c120d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41610/0 (socket says 172.21.15.83:41610) 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa97284c640 1 -- 172.21.15.83:0/2439670450 learned_addr learned my addr 172.21.15.83:0/2439670450 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c10f680 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c071c40 2024-09-10T19:12:17.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697934258 0 0) 0x7fa96c122660 con 0x7fa96c10f680 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa958003200 con 0x7fa96c10f680 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c072720 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403209073 0 0) 0x7fa96c124a60 con 0x7fa96c071c40 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c122660 con 0x7fa96c071c40 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230947262 0 0) 0x7fa96c123860 con 0x7fa96c072720 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c124a60 con 0x7fa96c072720 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274608757 0 0) 0x7fa96c122660 con 0x7fa96c071c40 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa96c123860 con 0x7fa96c071c40 2024-09-10T19:12:17.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa95c0018c0 con 0x7fa96c071c40 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760156690 0 0) 0x7fa958003200 con 0x7fa96c10f680 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa96c122660 con 0x7fa96c10f680 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa9640014f0 con 0x7fa96c10f680 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046840555 0 0) 0x7fa96c123860 con 0x7fa96c071c40 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 >> v1:172.21.15.165:6789/0 conn(0x7fa96c072720 legacy=0x7fa96c10c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 >> v1:172.21.15.83:6789/0 conn(0x7fa96c10f680 legacy=0x7fa96c120d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c125c60 con 0x7fa96c071c40 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96c124c50 con 0x7fa96c071c40 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 10 client.50033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96c1250a0 con 0x7fa96c071c40 2024-09-10T19:12:17.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 librados: init done 2024-09-10T19:12:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fa95c001f40 con 0x7fa96c071c40 2024-09-10T19:12:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fa95c0031f0 con 0x7fa96c071c40 2024-09-10T19:12:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.070+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa950005010 con 0x7fa96c071c40 2024-09-10T19:12:17.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fa95c001ae0 con 0x7fa96c071c40 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(893..893 src has 838..893) v4 ==== 7363+0+0 (unknown 3333851073 0 0) 0x7fa95c087ad0 con 0x7fa96c071c40 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 10 client.50033.objecter ms_dispatch 0x7fa96c000b90 osd_map(893..893 src has 838..893) v4 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 3 client.50033.objecter handle_osd_map got epochs [893,893] > 0 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 3 client.50033.objecter handle_osd_map decoding full epoch 893 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 20 client.50033.objecter dump_active .. 0 homeless 2024-09-10T19:12:17.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.074+0000 7fa962ffd640 10 client.50033.objecter ms_handle_connect 0x7fa9580718d0 2024-09-10T19:12:17.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.082+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fa95c054310 con 0x7fa96c071c40 2024-09-10T19:12:17.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.262+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9500052c0 con 0x7fa96c071c40 2024-09-10T19:12:17.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.262+0000 7fa962ffd640 1 -- 172.21.15.83:0/2439670450 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v893) v1 ==== 74+0+22041 (unknown 3660431869 0 3612933177) 0x7fa95c053cb0 con 0x7fa96c071c40 2024-09-10T19:12:17.266 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:12:17.266 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":893,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","created":"2024-09-10T18:38:38.475667+0000","modified":"2024-09-10T19:12:07.411756+0000","last_up_change":"2024-09-10T19:09:16.583972+0000","last_in_change":"2024-09-10T18:48:07.544682+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":117,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":263,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:44:06.952078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"849","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"508","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T18:49:16.239173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:49:18.096790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:49:20.070555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"datapool","create_time":"2024-09-10T18:49:20.615000+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":80,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:49:22.582739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-10T18:54:28.259368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"35ab8328-74f1-4af8-aa84-797914199445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6801","nonce":2535527483}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6802","nonce":2535527483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6804","nonce":2535527483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6803","nonce":2535527483}]},"public_addr":"172.21.15.83:6801/2535527483","cluster_addr":"172.21.15.83:6802/2535527483","heartbeat_back_addr":"172.21.15.83:6804/2535527483","heartbeat_front_addr":"172.21.15.83:6803/2535527483","state":["exists","up"]},{"osd":1,"uuid":"8e1d1072-767b-49fb-b2f2-fce60263e2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":789,"up_from":855,"up_thru":855,"down_at":790,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6805","nonce":1048694267}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6806","nonce":1048694267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6808","nonce":1048694267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6807","nonce":1048694267}]},"public_addr":"172.21.15.83:6805/1048694267","cluster_addr":"172.21.15.83:6806/1048694267","heartbeat_back_addr":"172.21.15.83:6808/1048694267","heartbeat_front_addr":"172.21.15.83:6807/1048694267","state":["exists","up"]},{"osd":2,"uuid":"6193b508-03d9-458a-b49f-6b2e24283282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6809","nonce":386094220}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6810","nonce":386094220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6812","nonce":386094220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6811","nonce":386094220}]},"public_addr":"172.21.15.83:6809/386094220","cluster_addr":"172.21.15.83:6810/386094220","heartbeat_back_addr":"172.21.15.83:6812/386094220","heartbeat_front_addr":"172.21.15.83:6811/386094220","state":["exists","up"]},{"osd":3,"uuid":"a28f94ff-2fff-458e-8b69-c5c65d720d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6813","nonce":1279688021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6814","nonce":1279688021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6816","nonce":1279688021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6815","nonce":1279688021}]},"public_addr":"172.21.15.83:6813/1279688021","cluster_addr":"172.21.15.83:6814/1279688021","heartbeat_back_addr":"172.21.15.83:6816/1279688021","heartbeat_front_addr":"172.21.15.83:6815/1279688021","state":["exists","up"]},{"osd":4,"uuid":"a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":878,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":3225515265}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":3225515265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":3225515265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":3225515265}]},"public_addr":"172.21.15.165:6800/3225515265","cluster_addr":"172.21.15.165:6801/3225515265","heartbeat_back_addr":"172.21.15.165:6803/3225515265","heartbeat_front_addr":"172.21.15.165:6802/3225515265","state":["exists","up"]},{"osd":5,"uuid":"10543223-db5a-493b-88f3-8705f2b31a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":571173777}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":571173777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":571173777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":571173777}]},"public_addr":"172.21.15.165:6804/571173777","cluster_addr":"172.21.15.165:6805/571173777","heartbeat_back_addr":"172.21.15.165:6807/571173777","heartbeat_front_addr":"172.21.15.165:6806/571173777","state":["exists","up"]},{"osd":6,"uuid":"fcea0ca4-4521-4161-b407-9b1fab1a2ae2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":1317922936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":1317922936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":1317922936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":1317922936}]},"public_addr":"172.21.15.165:6808/1317922936","cluster_addr":"172.21.15.165:6809/1317922936","heartbeat_back_addr":"172.21.15.165:6811/1317922936","heartbeat_front_addr":"172.21.15.165:6810/1317922936","state":["exists","up"]},{"osd":7,"uuid":"8531d286-5ebb-4c96-a026-63d8a17cb21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1610151650}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1610151650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1610151650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1610151650}]},"public_addr":"172.21.15.165:6812/1610151650","cluster_addr":"172.21.15.165:6813/1610151650","heartbeat_back_addr":"172.21.15.165:6815/1610151650","heartbeat_front_addr":"172.21.15.165:6814/1610151650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:36.152430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:36.584416+0000","dead_epoch":789},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:56.975333+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:30.794658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:28.950343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:33.024186+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:24.108078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:15.548526+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/3645093631":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/3275901971":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3555551632":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/3076437878":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/4112054239":"2024-09-11T18:39:06.336264+0000","172.21.15.83:6800/2452184936":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/2659452540":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/867303943":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/1156549307":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/1687990153":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1246256198":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/3225538750":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/2630800463":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1423545952":"2024-09-11T18:50:14.421453+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":22,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":23,"snaps":[{"begin":9,"length":2}]},{"pool":60,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":80,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":88,"snaps":[{"begin":2,"length":1}]},{"pool":95,"snaps":[{"begin":6,"length":2}]},{"pool":97,"snaps":[{"begin":4,"length":4}]},{"pool":108,"snaps":[{"begin":2,"length":1}]},{"pool":113,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":128,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":136,"snaps":[{"begin":12,"length":1}]},{"pool":156,"snaps":[{"begin":14,"length":1}]},{"pool":166,"snaps":[{"begin":17,"length":1}]},{"pool":171,"snaps":[{"begin":19,"length":2}]},{"pool":183,"snaps":[{"begin":24,"length":1}]},{"pool":244,"snaps":[{"begin":2,"length":1}]},{"pool":245,"snaps":[{"begin":4,"length":1}]},{"pool":248,"snaps":[{"begin":6,"length":1}]},{"pool":252,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 10 librados: watch_flush enter 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 10 librados: watch_flush exit 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 20 client.50033.objecter shutdown clearing up homeless session... 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 10 client.50033.objecter successfully canceled tick 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 >> v1:172.21.15.83:6800/3071793735 conn(0x7fa9580718d0 legacy=0x7fa958073d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 >> v1:172.21.15.83:6790/0 conn(0x7fa96c071c40 legacy=0x7fa96c10bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 shutdown_connections 2024-09-10T19:12:17.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 >> 172.21.15.83:0/2439670450 conn(0x7fa96c0fa370 msgr2=0x7fa96c111a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:17.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 shutdown_connections 2024-09-10T19:12:17.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 -- 172.21.15.83:0/2439670450 wait complete. 2024-09-10T19:12:17.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:17.266+0000 7fa9742af640 1 librados: shutdown 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.595323+0000 osd.4 (osd.4) 516 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.599140+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.730604+0000 osd.5 (osd.5) 548 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.733851+0000 osd.5 (osd.5) 549 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.911841+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:15.916439+0000 osd.3 (osd.3) 583 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.001453+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.4 deep-scrub starts 2024-09-10T19:12:17.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.005005+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.4 deep-scrub ok 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.077537+0000 osd.0 (osd.0) 539 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.081269+0000 osd.0 (osd.0) 540 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.595323+0000 osd.4 (osd.4) 516 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.599140+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.730604+0000 osd.5 (osd.5) 548 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.733851+0000 osd.5 (osd.5) 549 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.911841+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:15.916439+0000 osd.3 (osd.3) 583 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.001453+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.4 deep-scrub starts 2024-09-10T19:12:17.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.005005+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.4 deep-scrub ok 2024-09-10T19:12:17.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.077537+0000 osd.0 (osd.0) 539 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:12:17.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:17 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.081269+0000 osd.0 (osd.0) 540 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.595323+0000 osd.4 (osd.4) 516 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.599140+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.730604+0000 osd.5 (osd.5) 548 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.733851+0000 osd.5 (osd.5) 549 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.911841+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub starts 2024-09-10T19:12:17.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:15.916439+0000 osd.3 (osd.3) 583 : cluster [DBG] 1.4 scrub ok 2024-09-10T19:12:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.001453+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.4 deep-scrub starts 2024-09-10T19:12:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.005005+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.4 deep-scrub ok 2024-09-10T19:12:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.077537+0000 osd.0 (osd.0) 539 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:12:17.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:17 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.081269+0000 osd.0 (osd.0) 540 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:12:18.118 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T19:12:18.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.531067+0000 mgr.y (mgr.24526) 887 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.776640+0000 osd.5 (osd.5) 550 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:16.781147+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.024120+0000 osd.1 (osd.1) 111 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.027913+0000 osd.1 (osd.1) 112 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.228882+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-09-10T19:12:18.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.231996+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[18629]: audit 2024-09-10T19:12:17.262814+0000 mon.c (mon.2) 923 : audit [DBG] from='client.? v1:172.21.15.83:0/2439670450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.531067+0000 mgr.y (mgr.24526) 887 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.776640+0000 osd.5 (osd.5) 550 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:16.781147+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.024120+0000 osd.1 (osd.1) 111 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.027913+0000 osd.1 (osd.1) 112 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.228882+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.231996+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-09-10T19:12:18.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:18 smithi083 bash[32381]: audit 2024-09-10T19:12:17.262814+0000 mon.c (mon.2) 923 : audit [DBG] from='client.? v1:172.21.15.83:0/2439670450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.531067+0000 mgr.y (mgr.24526) 887 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.776640+0000 osd.5 (osd.5) 550 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:16.781147+0000 osd.5 (osd.5) 551 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.024120+0000 osd.1 (osd.1) 111 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.027913+0000 osd.1 (osd.1) 112 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:12:18.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.228882+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-09-10T19:12:18.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.231996+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-09-10T19:12:18.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[22591]: audit 2024-09-10T19:12:17.262814+0000 mon.c (mon.2) 923 : audit [DBG] from='client.? v1:172.21.15.83:0/2439670450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:19.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.447552+0000 osd.6 (osd.6) 505 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.451150+0000 osd.6 (osd.6) 506 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.617111+0000 osd.4 (osd.4) 518 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:17.620825+0000 osd.4 (osd.4) 519 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.024877+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.028357+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.064996+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.6 deep-scrub starts 2024-09-10T19:12:19.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.069090+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.6 deep-scrub ok 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.447552+0000 osd.6 (osd.6) 505 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.451150+0000 osd.6 (osd.6) 506 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.617111+0000 osd.4 (osd.4) 518 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:17.620825+0000 osd.4 (osd.4) 519 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.024877+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.028357+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.064996+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.6 deep-scrub starts 2024-09-10T19:12:19.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:19 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.069090+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.6 deep-scrub ok 2024-09-10T19:12:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.447552+0000 osd.6 (osd.6) 505 : cluster [DBG] 12.5 scrub starts 2024-09-10T19:12:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.451150+0000 osd.6 (osd.6) 506 : cluster [DBG] 12.5 scrub ok 2024-09-10T19:12:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.617111+0000 osd.4 (osd.4) 518 : cluster [DBG] 4.1a scrub starts 2024-09-10T19:12:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:17.620825+0000 osd.4 (osd.4) 519 : cluster [DBG] 4.1a scrub ok 2024-09-10T19:12:19.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.024877+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:12:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.028357+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:12:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.064996+0000 osd.1 (osd.1) 113 : cluster [DBG] 2.6 deep-scrub starts 2024-09-10T19:12:19.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:19 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.069090+0000 osd.1 (osd.1) 114 : cluster [DBG] 2.6 deep-scrub ok 2024-09-10T19:12:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.421584+0000 osd.6 (osd.6) 507 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:12:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.425029+0000 osd.6 (osd.6) 508 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:12:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.531849+0000 mgr.y (mgr.24526) 888 : cluster [DBG] pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:12:20.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: audit 2024-09-10T19:12:18.670747+0000 mgr.y (mgr.24526) 889 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.977853+0000 osd.3 (osd.3) 584 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:18.981623+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.015169+0000 osd.0 (osd.0) 543 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.018614+0000 osd.0 (osd.0) 544 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.220127+0000 osd.2 (osd.2) 399 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.223568+0000 osd.2 (osd.2) 400 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.752840+0000 osd.5 (osd.5) 552 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.758496+0000 osd.5 (osd.5) 553 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.421584+0000 osd.6 (osd.6) 507 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:12:20.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.425029+0000 osd.6 (osd.6) 508 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.531849+0000 mgr.y (mgr.24526) 888 : cluster [DBG] pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: audit 2024-09-10T19:12:18.670747+0000 mgr.y (mgr.24526) 889 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.977853+0000 osd.3 (osd.3) 584 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:18.981623+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.015169+0000 osd.0 (osd.0) 543 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.018614+0000 osd.0 (osd.0) 544 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.220127+0000 osd.2 (osd.2) 399 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.223568+0000 osd.2 (osd.2) 400 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.752840+0000 osd.5 (osd.5) 552 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T19:12:20.543 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:20 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.758496+0000 osd.5 (osd.5) 553 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T19:12:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.421584+0000 osd.6 (osd.6) 507 : cluster [DBG] 12.1f scrub starts 2024-09-10T19:12:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.425029+0000 osd.6 (osd.6) 508 : cluster [DBG] 12.1f scrub ok 2024-09-10T19:12:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.531849+0000 mgr.y (mgr.24526) 888 : cluster [DBG] pgmap v1468: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:12:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: audit 2024-09-10T19:12:18.670747+0000 mgr.y (mgr.24526) 889 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.977853+0000 osd.3 (osd.3) 584 : cluster [DBG] 6.1f scrub starts 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:18.981623+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.1f scrub ok 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.015169+0000 osd.0 (osd.0) 543 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.018614+0000 osd.0 (osd.0) 544 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.220127+0000 osd.2 (osd.2) 399 : cluster [DBG] 6.14 deep-scrub starts 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.223568+0000 osd.2 (osd.2) 400 : cluster [DBG] 6.14 deep-scrub ok 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.752840+0000 osd.5 (osd.5) 552 : cluster [DBG] 3.1c deep-scrub starts 2024-09-10T19:12:20.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:20 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.758496+0000 osd.5 (osd.5) 553 : cluster [DBG] 3.1c deep-scrub ok 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.546674+0000 osd.4 (osd.4) 520 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:19.551663+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.009892+0000 osd.3 (osd.3) 586 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.014757+0000 osd.3 (osd.3) 587 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.252367+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.e scrub starts 2024-09-10T19:12:21.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.256573+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.e scrub ok 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.546674+0000 osd.4 (osd.4) 520 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:19.551663+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.009892+0000 osd.3 (osd.3) 586 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.014757+0000 osd.3 (osd.3) 587 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.252367+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.e scrub starts 2024-09-10T19:12:21.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:21 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.256573+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.e scrub ok 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.546674+0000 osd.4 (osd.4) 520 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:19.551663+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.009892+0000 osd.3 (osd.3) 586 : cluster [DBG] 3.8 scrub starts 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.014757+0000 osd.3 (osd.3) 587 : cluster [DBG] 3.8 scrub ok 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.252367+0000 osd.2 (osd.2) 401 : cluster [DBG] 2.e scrub starts 2024-09-10T19:12:21.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:21 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.256573+0000 osd.2 (osd.2) 402 : cluster [DBG] 2.e scrub ok 2024-09-10T19:12:22.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.432215+0000 osd.6 (osd.6) 509 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:12:22.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.436300+0000 osd.6 (osd.6) 510 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:12:22.544 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[18629]: cluster 2024-09-10T19:12:20.532579+0000 mgr.y (mgr.24526) 890 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:22.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.432215+0000 osd.6 (osd.6) 509 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:12:22.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.436300+0000 osd.6 (osd.6) 510 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:12:22.545 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[32381]: cluster 2024-09-10T19:12:20.532579+0000 mgr.y (mgr.24526) 890 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:22 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.432215+0000 osd.6 (osd.6) 509 : cluster [DBG] 2.11 scrub starts 2024-09-10T19:12:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:22 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.436300+0000 osd.6 (osd.6) 510 : cluster [DBG] 2.11 scrub ok 2024-09-10T19:12:22.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:22 smithi165 bash[22591]: cluster 2024-09-10T19:12:20.532579+0000 mgr.y (mgr.24526) 890 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:22.893 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:22.999 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:22] "GET /metrics HTTP/1.1" 200 224120 "" "Prometheus/2.43.0" 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[18629]: cluster 2024-09-10T19:12:22.431964+0000 osd.6 (osd.6) 511 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[18629]: cluster 2024-09-10T19:12:22.435756+0000 osd.6 (osd.6) 512 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[18629]: cluster 2024-09-10T19:12:22.534484+0000 mgr.y (mgr.24526) 891 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[32381]: cluster 2024-09-10T19:12:22.431964+0000 osd.6 (osd.6) 511 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[32381]: cluster 2024-09-10T19:12:22.435756+0000 osd.6 (osd.6) 512 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:12:24.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:24 smithi083 bash[32381]: cluster 2024-09-10T19:12:22.534484+0000 mgr.y (mgr.24526) 891 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:24 smithi165 bash[22591]: cluster 2024-09-10T19:12:22.431964+0000 osd.6 (osd.6) 511 : cluster [DBG] 4.a deep-scrub starts 2024-09-10T19:12:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:24 smithi165 bash[22591]: cluster 2024-09-10T19:12:22.435756+0000 osd.6 (osd.6) 512 : cluster [DBG] 4.a deep-scrub ok 2024-09-10T19:12:24.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:24 smithi165 bash[22591]: cluster 2024-09-10T19:12:22.534484+0000 mgr.y (mgr.24526) 891 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:25.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:25 smithi083 bash[18629]: cluster 2024-09-10T19:12:23.409540+0000 osd.6 (osd.6) 513 : cluster [DBG] 3.a scrub starts 2024-09-10T19:12:25.515 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:25 smithi083 bash[18629]: cluster 2024-09-10T19:12:23.415293+0000 osd.6 (osd.6) 514 : cluster [DBG] 3.a scrub ok 2024-09-10T19:12:25.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:25 smithi083 bash[32381]: cluster 2024-09-10T19:12:23.409540+0000 osd.6 (osd.6) 513 : cluster [DBG] 3.a scrub starts 2024-09-10T19:12:25.516 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:25 smithi083 bash[32381]: cluster 2024-09-10T19:12:23.415293+0000 osd.6 (osd.6) 514 : cluster [DBG] 3.a scrub ok 2024-09-10T19:12:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:25 smithi165 bash[22591]: cluster 2024-09-10T19:12:23.409540+0000 osd.6 (osd.6) 513 : cluster [DBG] 3.a scrub starts 2024-09-10T19:12:25.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:25 smithi165 bash[22591]: cluster 2024-09-10T19:12:23.415293+0000 osd.6 (osd.6) 514 : cluster [DBG] 3.a scrub ok 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[32381]: cluster 2024-09-10T19:12:24.455335+0000 osd.6 (osd.6) 515 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[32381]: cluster 2024-09-10T19:12:24.460172+0000 osd.6 (osd.6) 516 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[32381]: cluster 2024-09-10T19:12:24.535118+0000 mgr.y (mgr.24526) 892 : cluster [DBG] pgmap v1471: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[18629]: cluster 2024-09-10T19:12:24.455335+0000 osd.6 (osd.6) 515 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[18629]: cluster 2024-09-10T19:12:24.460172+0000 osd.6 (osd.6) 516 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:12:26.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:26 smithi083 bash[18629]: cluster 2024-09-10T19:12:24.535118+0000 mgr.y (mgr.24526) 892 : cluster [DBG] pgmap v1471: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:26 smithi165 bash[22591]: cluster 2024-09-10T19:12:24.455335+0000 osd.6 (osd.6) 515 : cluster [DBG] 5.2 scrub starts 2024-09-10T19:12:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:26 smithi165 bash[22591]: cluster 2024-09-10T19:12:24.460172+0000 osd.6 (osd.6) 516 : cluster [DBG] 5.2 scrub ok 2024-09-10T19:12:26.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:26 smithi165 bash[22591]: cluster 2024-09-10T19:12:24.535118+0000 mgr.y (mgr.24526) 892 : cluster [DBG] pgmap v1471: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:26.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.869+0000 7f02388db640 1 -- 172.21.15.83:0/10136762 >> v1:172.21.15.83:6789/0 conn(0x7f0234071c40 legacy=0x7f0234072010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:26.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.869+0000 7f02388db640 1 -- 172.21.15.83:0/10136762 shutdown_connections 2024-09-10T19:12:26.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.869+0000 7f02388db640 1 -- 172.21.15.83:0/10136762 >> 172.21.15.83:0/10136762 conn(0x7f02340fa3a0 msgr2=0x7f02340fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:26.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.869+0000 7f02388db640 1 -- 172.21.15.83:0/10136762 shutdown_connections 2024-09-10T19:12:26.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.869+0000 7f02388db640 1 -- 172.21.15.83:0/10136762 wait complete. 2024-09-10T19:12:26.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 librados: starting msgr at 2024-09-10T19:12:26.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 librados: starting objecter 2024-09-10T19:12:26.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 Processor -- start 2024-09-10T19:12:26.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- start start 2024-09-10T19:12:26.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 librados: setting wanted keys 2024-09-10T19:12:26.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 librados: calling monclient init 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0234118930 con 0x7f0234072720 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0234123a20 con 0x7f023410f670 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f023259c640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f0234071c40 0x7f0234117d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:53370/0 (socket says 172.21.15.83:53370) 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0234124c00 con 0x7f0234071c40 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f023259c640 1 -- 172.21.15.83:0/3781656744 learned_addr learned my addr 172.21.15.83:0/3781656744 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter ms_handle_connect 0x7f0234072720 2024-09-10T19:12:26.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:26.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter ms_handle_connect 0x7f0234071c40 2024-09-10T19:12:26.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:26.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757284271 0 0) 0x7f0234118930 con 0x7f0234072720 2024-09-10T19:12:26.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220003120 con 0x7f0234072720 2024-09-10T19:12:26.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671748241 0 0) 0x7f0234124c00 con 0x7f0234071c40 2024-09-10T19:12:26.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234118930 con 0x7f0234071c40 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter ms_handle_connect 0x7f023410f670 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3381764007 0 0) 0x7f0234118930 con 0x7f0234071c40 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0234124c00 con 0x7f0234071c40 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f021c002050 con 0x7f0234071c40 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 818533190 0 0) 0x7f0220003120 con 0x7f0234072720 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0234118930 con 0x7f0234072720 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f022c001ca0 con 0x7f0234072720 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676824395 0 0) 0x7f0234123a20 con 0x7f023410f670 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0220003120 con 0x7f023410f670 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1881220217 0 0) 0x7f0234124c00 con 0x7f0234071c40 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 >> v1:172.21.15.165:6789/0 conn(0x7f023410f670 legacy=0x7f02341212f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:26.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 >> v1:172.21.15.83:6789/0 conn(0x7f0234072720 legacy=0x7f023411dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234125de0 con 0x7f0234071c40 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0234124df0 con 0x7f0234071c40 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 10 client.50039.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0234125240 con 0x7f0234071c40 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f02388db640 1 librados: init done 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f021c002390 con 0x7f0234071c40 2024-09-10T19:12:26.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.873+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f021c0035d0 con 0x7f0234071c40 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214005010 con 0x7f0234071c40 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f021c0037b0 con 0x7f0234071c40 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(893..893 src has 838..893) v4 ==== 7363+0+0 (unknown 3333851073 0 0) 0x7f021c08a520 con 0x7f0234071c40 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 10 client.50039.objecter ms_dispatch 0x7f0234000b90 osd_map(893..893 src has 838..893) v4 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 3 client.50039.objecter handle_osd_map got epochs [893,893] > 0 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 3 client.50039.objecter handle_osd_map decoding full epoch 893 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 20 client.50039.objecter dump_active .. 0 homeless 2024-09-10T19:12:26.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.877+0000 7f022affd640 10 client.50039.objecter ms_handle_connect 0x7f0220071760 2024-09-10T19:12:26.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:26.881+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f021c056dc0 con 0x7f0234071c40 2024-09-10T19:12:27.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.053+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02140052c0 con 0x7f0234071c40 2024-09-10T19:12:27.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.057+0000 7f022affd640 1 -- 172.21.15.83:0/3781656744 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v893) v1 ==== 74+0+22041 (unknown 3660431869 0 3612933177) 0x7f021c056760 con 0x7f0234071c40 2024-09-10T19:12:27.059 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:12:27.060 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":893,"fsid":"707a4ada-6fa3-11ef-bcea-c7b262605968","created":"2024-09-10T18:38:38.475667+0000","modified":"2024-09-10T19:12:07.411756+0000","last_up_change":"2024-09-10T19:09:16.583972+0000","last_in_change":"2024-09-10T18:48:07.544682+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":117,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":263,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:44:06.952078+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"849","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"508","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T18:49:16.239173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-09-10T18:49:18.096790+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-09-10T18:49:20.070555+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"datapool","create_time":"2024-09-10T18:49:20.615000+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":80,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T18:49:22.582739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-10T18:54:28.259368+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"35ab8328-74f1-4af8-aa84-797914199445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6801","nonce":2535527483}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6802","nonce":2535527483}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6804","nonce":2535527483}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6803","nonce":2535527483}]},"public_addr":"172.21.15.83:6801/2535527483","cluster_addr":"172.21.15.83:6802/2535527483","heartbeat_back_addr":"172.21.15.83:6804/2535527483","heartbeat_front_addr":"172.21.15.83:6803/2535527483","state":["exists","up"]},{"osd":1,"uuid":"8e1d1072-767b-49fb-b2f2-fce60263e2ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":789,"up_from":855,"up_thru":855,"down_at":790,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6805","nonce":1048694267}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6806","nonce":1048694267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6808","nonce":1048694267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6807","nonce":1048694267}]},"public_addr":"172.21.15.83:6805/1048694267","cluster_addr":"172.21.15.83:6806/1048694267","heartbeat_back_addr":"172.21.15.83:6808/1048694267","heartbeat_front_addr":"172.21.15.83:6807/1048694267","state":["exists","up"]},{"osd":2,"uuid":"6193b508-03d9-458a-b49f-6b2e24283282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6809","nonce":386094220}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6810","nonce":386094220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6812","nonce":386094220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6811","nonce":386094220}]},"public_addr":"172.21.15.83:6809/386094220","cluster_addr":"172.21.15.83:6810/386094220","heartbeat_back_addr":"172.21.15.83:6812/386094220","heartbeat_front_addr":"172.21.15.83:6811/386094220","state":["exists","up"]},{"osd":3,"uuid":"a28f94ff-2fff-458e-8b69-c5c65d720d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":29,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6813","nonce":1279688021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6814","nonce":1279688021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6816","nonce":1279688021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.83:6815","nonce":1279688021}]},"public_addr":"172.21.15.83:6813/1279688021","cluster_addr":"172.21.15.83:6814/1279688021","heartbeat_back_addr":"172.21.15.83:6816/1279688021","heartbeat_front_addr":"172.21.15.83:6815/1279688021","state":["exists","up"]},{"osd":4,"uuid":"a7db50aa-4bf2-4694-88f7-9e5b6e8fa56b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":878,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":3225515265}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":3225515265}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":3225515265}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":3225515265}]},"public_addr":"172.21.15.165:6800/3225515265","cluster_addr":"172.21.15.165:6801/3225515265","heartbeat_back_addr":"172.21.15.165:6803/3225515265","heartbeat_front_addr":"172.21.15.165:6802/3225515265","state":["exists","up"]},{"osd":5,"uuid":"10543223-db5a-493b-88f3-8705f2b31a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":571173777}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":571173777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":571173777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":571173777}]},"public_addr":"172.21.15.165:6804/571173777","cluster_addr":"172.21.15.165:6805/571173777","heartbeat_back_addr":"172.21.15.165:6807/571173777","heartbeat_front_addr":"172.21.15.165:6806/571173777","state":["exists","up"]},{"osd":6,"uuid":"fcea0ca4-4521-4161-b407-9b1fab1a2ae2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":1317922936}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":1317922936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":1317922936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":1317922936}]},"public_addr":"172.21.15.165:6808/1317922936","cluster_addr":"172.21.15.165:6809/1317922936","heartbeat_back_addr":"172.21.15.165:6811/1317922936","heartbeat_front_addr":"172.21.15.165:6810/1317922936","state":["exists","up"]},{"osd":7,"uuid":"8531d286-5ebb-4c96-a026-63d8a17cb21f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1610151650}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1610151650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1610151650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1610151650}]},"public_addr":"172.21.15.165:6812/1610151650","cluster_addr":"172.21.15.165:6813/1610151650","heartbeat_back_addr":"172.21.15.165:6815/1610151650","heartbeat_front_addr":"172.21.15.165:6814/1610151650","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:36.152430+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:36.584416+0000","dead_epoch":789},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:56.975333+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:30.794658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:28.950343+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:33.024186+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:24.108078+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:11:15.548526+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.83:0/3645093631":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/3275901971":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/3555551632":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/3076437878":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/4112054239":"2024-09-11T18:39:06.336264+0000","172.21.15.83:6800/2452184936":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/2659452540":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/867303943":"2024-09-11T18:39:06.336264+0000","172.21.15.83:0/1156549307":"2024-09-11T18:39:40.343252+0000","172.21.15.83:0/1687990153":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1246256198":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/3225538750":"2024-09-11T18:50:14.421453+0000","172.21.15.83:6800/2630800463":"2024-09-11T18:50:14.421453+0000","172.21.15.83:0/1423545952":"2024-09-11T18:50:14.421453+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":22,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":23,"snaps":[{"begin":9,"length":2}]},{"pool":60,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":80,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":88,"snaps":[{"begin":2,"length":1}]},{"pool":95,"snaps":[{"begin":6,"length":2}]},{"pool":97,"snaps":[{"begin":4,"length":4}]},{"pool":108,"snaps":[{"begin":2,"length":1}]},{"pool":113,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":128,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":136,"snaps":[{"begin":12,"length":1}]},{"pool":156,"snaps":[{"begin":14,"length":1}]},{"pool":166,"snaps":[{"begin":17,"length":1}]},{"pool":171,"snaps":[{"begin":19,"length":2}]},{"pool":183,"snaps":[{"begin":24,"length":1}]},{"pool":244,"snaps":[{"begin":2,"length":1}]},{"pool":245,"snaps":[{"begin":4,"length":1}]},{"pool":248,"snaps":[{"begin":6,"length":1}]},{"pool":252,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:12:27.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 10 librados: watch_flush enter 2024-09-10T19:12:27.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 10 librados: watch_flush exit 2024-09-10T19:12:27.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 20 client.50039.objecter shutdown clearing up homeless session... 2024-09-10T19:12:27.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 10 client.50039.objecter successfully canceled tick 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 >> v1:172.21.15.83:6800/3071793735 conn(0x7f0220071760 legacy=0x7f0220073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 >> v1:172.21.15.83:6790/0 conn(0x7f0234071c40 legacy=0x7f0234117d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 shutdown_connections 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 >> 172.21.15.83:0/3781656744 conn(0x7f02340fa3a0 msgr2=0x7f02340fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 shutdown_connections 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 -- 172.21.15.83:0/3781656744 wait complete. 2024-09-10T19:12:27.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:27.061+0000 7f02388db640 1 librados: shutdown 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.847062+0000 osd.5 (osd.5) 554 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.850438+0000 osd.5 (osd.5) 555 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[18629]: audit 2024-09-10T19:12:27.058036+0000 mon.c (mon.2) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/3781656744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.847062+0000 osd.5 (osd.5) 554 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.850438+0000 osd.5 (osd.5) 555 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:12:27.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:27 smithi083 bash[32381]: audit 2024-09-10T19:12:27.058036+0000 mon.c (mon.2) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/3781656744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:27 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.847062+0000 osd.5 (osd.5) 554 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:12:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:27 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.850438+0000 osd.5 (osd.5) 555 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:12:27.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:27 smithi165 bash[22591]: audit 2024-09-10T19:12:27.058036+0000 mon.c (mon.2) 924 : audit [DBG] from='client.? v1:172.21.15.83:0/3781656744' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:12:27.905 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T19:12:27.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T19:12:27.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T19:12:27.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T19:12:27.907 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T19:12:27.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T19:12:27.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T19:12:27.909 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T19:12:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.453039+0000 osd.6 (osd.6) 517 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:12:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.456430+0000 osd.6 (osd.6) 518 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:12:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.530439+0000 osd.4 (osd.4) 522 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:12:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.534607+0000 osd.4 (osd.4) 523 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:12:28.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[18629]: cluster 2024-09-10T19:12:26.536996+0000 mgr.y (mgr.24526) 893 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.453039+0000 osd.6 (osd.6) 517 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:12:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.456430+0000 osd.6 (osd.6) 518 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:12:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.530439+0000 osd.4 (osd.4) 522 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:12:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.534607+0000 osd.4 (osd.4) 523 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:12:28.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:28 smithi083 bash[32381]: cluster 2024-09-10T19:12:26.536996+0000 mgr.y (mgr.24526) 893 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.453039+0000 osd.6 (osd.6) 517 : cluster [DBG] 12.16 scrub starts 2024-09-10T19:12:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.456430+0000 osd.6 (osd.6) 518 : cluster [DBG] 12.16 scrub ok 2024-09-10T19:12:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.530439+0000 osd.4 (osd.4) 522 : cluster [DBG] 12.1b scrub starts 2024-09-10T19:12:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.534607+0000 osd.4 (osd.4) 523 : cluster [DBG] 12.1b scrub ok 2024-09-10T19:12:28.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[22591]: cluster 2024-09-10T19:12:26.536996+0000 mgr.y (mgr.24526) 893 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:29.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:29 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.795943+0000 osd.5 (osd.5) 556 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:12:29.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:29 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.801217+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:12:29.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:29 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.795943+0000 osd.5 (osd.5) 556 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:12:29.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:29 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.801217+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:12:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:29 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.795943+0000 osd.5 (osd.5) 556 : cluster [DBG] 3.5 scrub starts 2024-09-10T19:12:29.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:29 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.801217+0000 osd.5 (osd.5) 557 : cluster [DBG] 3.5 scrub ok 2024-09-10T19:12:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.457392+0000 osd.6 (osd.6) 519 : cluster [DBG] 12.f scrub starts 2024-09-10T19:12:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.460653+0000 osd.6 (osd.6) 520 : cluster [DBG] 12.f scrub ok 2024-09-10T19:12:30.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.514297+0000 osd.4 (osd.4) 524 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.517979+0000 osd.4 (osd.4) 525 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.537806+0000 mgr.y (mgr.24526) 894 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: audit 2024-09-10T19:12:28.682148+0000 mgr.y (mgr.24526) 895 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.790861+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b deep-scrub starts 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:28.794109+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b deep-scrub ok 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.491204+0000 osd.6 (osd.6) 521 : cluster [DBG] 1.7 deep-scrub starts 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.494902+0000 osd.6 (osd.6) 522 : cluster [DBG] 1.7 deep-scrub ok 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[32381]: audit 2024-09-10T19:12:29.583054+0000 mon.c (mon.2) 925 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.457392+0000 osd.6 (osd.6) 519 : cluster [DBG] 12.f scrub starts 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.460653+0000 osd.6 (osd.6) 520 : cluster [DBG] 12.f scrub ok 2024-09-10T19:12:30.542 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.514297+0000 osd.4 (osd.4) 524 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.517979+0000 osd.4 (osd.4) 525 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.537806+0000 mgr.y (mgr.24526) 894 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: audit 2024-09-10T19:12:28.682148+0000 mgr.y (mgr.24526) 895 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.790861+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b deep-scrub starts 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:28.794109+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b deep-scrub ok 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.491204+0000 osd.6 (osd.6) 521 : cluster [DBG] 1.7 deep-scrub starts 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.494902+0000 osd.6 (osd.6) 522 : cluster [DBG] 1.7 deep-scrub ok 2024-09-10T19:12:30.543 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:30 smithi083 bash[18629]: audit 2024-09-10T19:12:29.583054+0000 mon.c (mon.2) 925 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.457392+0000 osd.6 (osd.6) 519 : cluster [DBG] 12.f scrub starts 2024-09-10T19:12:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.460653+0000 osd.6 (osd.6) 520 : cluster [DBG] 12.f scrub ok 2024-09-10T19:12:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.514297+0000 osd.4 (osd.4) 524 : cluster [DBG] 4.1b scrub starts 2024-09-10T19:12:30.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.517979+0000 osd.4 (osd.4) 525 : cluster [DBG] 4.1b scrub ok 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.537806+0000 mgr.y (mgr.24526) 894 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: audit 2024-09-10T19:12:28.682148+0000 mgr.y (mgr.24526) 895 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.790861+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b deep-scrub starts 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:28.794109+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b deep-scrub ok 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.491204+0000 osd.6 (osd.6) 521 : cluster [DBG] 1.7 deep-scrub starts 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.494902+0000 osd.6 (osd.6) 522 : cluster [DBG] 1.7 deep-scrub ok 2024-09-10T19:12:30.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:30 smithi165 bash[22591]: audit 2024-09-10T19:12:29.583054+0000 mon.c (mon.2) 925 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.499649+0000 osd.4 (osd.4) 526 : cluster [DBG] 6.e scrub starts 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.503334+0000 osd.4 (osd.4) 527 : cluster [DBG] 6.e scrub ok 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.748066+0000 osd.7 (osd.7) 560 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:29.751526+0000 osd.7 (osd.7) 561 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:30.476257+0000 osd.6 (osd.6) 523 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:12:31.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[18629]: cluster 2024-09-10T19:12:30.479632+0000 osd.6 (osd.6) 524 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.499649+0000 osd.4 (osd.4) 526 : cluster [DBG] 6.e scrub starts 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.503334+0000 osd.4 (osd.4) 527 : cluster [DBG] 6.e scrub ok 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.748066+0000 osd.7 (osd.7) 560 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:29.751526+0000 osd.7 (osd.7) 561 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:30.476257+0000 osd.6 (osd.6) 523 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:12:31.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:31 smithi083 bash[32381]: cluster 2024-09-10T19:12:30.479632+0000 osd.6 (osd.6) 524 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:12:31.612 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.499649+0000 osd.4 (osd.4) 526 : cluster [DBG] 6.e scrub starts 2024-09-10T19:12:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.503334+0000 osd.4 (osd.4) 527 : cluster [DBG] 6.e scrub ok 2024-09-10T19:12:31.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.748066+0000 osd.7 (osd.7) 560 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:12:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:29.751526+0000 osd.7 (osd.7) 561 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:12:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:30.476257+0000 osd.6 (osd.6) 523 : cluster [DBG] 2.17 scrub starts 2024-09-10T19:12:31.613 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:31 smithi165 bash[22591]: cluster 2024-09-10T19:12:30.479632+0000 osd.6 (osd.6) 524 : cluster [DBG] 2.17 scrub ok 2024-09-10T19:12:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18629]: cluster 2024-09-10T19:12:30.538624+0000 mgr.y (mgr.24526) 896 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.070649+0000 osd.0 (osd.0) 545 : cluster [DBG] 4.b scrub starts 2024-09-10T19:12:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.074087+0000 osd.0 (osd.0) 546 : cluster [DBG] 4.b scrub ok 2024-09-10T19:12:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.729049+0000 osd.5 (osd.5) 558 : cluster [DBG] 6.a scrub starts 2024-09-10T19:12:32.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.731881+0000 osd.5 (osd.5) 559 : cluster [DBG] 6.a scrub ok 2024-09-10T19:12:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[32381]: cluster 2024-09-10T19:12:30.538624+0000 mgr.y (mgr.24526) 896 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.070649+0000 osd.0 (osd.0) 545 : cluster [DBG] 4.b scrub starts 2024-09-10T19:12:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.074087+0000 osd.0 (osd.0) 546 : cluster [DBG] 4.b scrub ok 2024-09-10T19:12:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.729049+0000 osd.5 (osd.5) 558 : cluster [DBG] 6.a scrub starts 2024-09-10T19:12:32.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.731881+0000 osd.5 (osd.5) 559 : cluster [DBG] 6.a scrub ok 2024-09-10T19:12:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:32 smithi165 bash[22591]: cluster 2024-09-10T19:12:30.538624+0000 mgr.y (mgr.24526) 896 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:32 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.070649+0000 osd.0 (osd.0) 545 : cluster [DBG] 4.b scrub starts 2024-09-10T19:12:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:32 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.074087+0000 osd.0 (osd.0) 546 : cluster [DBG] 4.b scrub ok 2024-09-10T19:12:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:32 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.729049+0000 osd.5 (osd.5) 558 : cluster [DBG] 6.a scrub starts 2024-09-10T19:12:32.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:32 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.731881+0000 osd.5 (osd.5) 559 : cluster [DBG] 6.a scrub ok 2024-09-10T19:12:32.789 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.790 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.790 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.791 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.792 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.795 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.801 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:32.972 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:32] "GET /metrics HTTP/1.1" 200 224098 "" "Prometheus/2.43.0" 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.530972+0000 osd.4 (osd.4) 528 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[18629]: cluster 2024-09-10T19:12:31.534336+0000 osd.4 (osd.4) 529 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[18629]: cluster 2024-09-10T19:12:32.017133+0000 osd.3 (osd.3) 588 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[18629]: cluster 2024-09-10T19:12:32.020577+0000 osd.3 (osd.3) 589 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.530972+0000 osd.4 (osd.4) 528 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[32381]: cluster 2024-09-10T19:12:31.534336+0000 osd.4 (osd.4) 529 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[32381]: cluster 2024-09-10T19:12:32.017133+0000 osd.3 (osd.3) 588 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:12:33.541 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:33 smithi083 bash[32381]: cluster 2024-09-10T19:12:32.020577+0000 osd.3 (osd.3) 589 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:12:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:33 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.530972+0000 osd.4 (osd.4) 528 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:12:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:33 smithi165 bash[22591]: cluster 2024-09-10T19:12:31.534336+0000 osd.4 (osd.4) 529 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:12:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:33 smithi165 bash[22591]: cluster 2024-09-10T19:12:32.017133+0000 osd.3 (osd.3) 588 : cluster [DBG] 4.1d scrub starts 2024-09-10T19:12:33.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:33 smithi165 bash[22591]: cluster 2024-09-10T19:12:32.020577+0000 osd.3 (osd.3) 589 : cluster [DBG] 4.1d scrub ok 2024-09-10T19:12:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[18629]: cluster 2024-09-10T19:12:32.540454+0000 mgr.y (mgr.24526) 897 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[18629]: cluster 2024-09-10T19:12:33.103900+0000 osd.0 (osd.0) 547 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:12:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[18629]: cluster 2024-09-10T19:12:33.107844+0000 osd.0 (osd.0) 548 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:12:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[18629]: cluster 2024-09-10T19:12:33.673837+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:12:34.541 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[18629]: cluster 2024-09-10T19:12:33.678952+0000 osd.5 (osd.5) 561 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:12:34.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[32381]: cluster 2024-09-10T19:12:32.540454+0000 mgr.y (mgr.24526) 897 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:34.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[32381]: cluster 2024-09-10T19:12:33.103900+0000 osd.0 (osd.0) 547 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:12:34.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[32381]: cluster 2024-09-10T19:12:33.107844+0000 osd.0 (osd.0) 548 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:12:34.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[32381]: cluster 2024-09-10T19:12:33.673837+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:12:34.542 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:34 smithi083 bash[32381]: cluster 2024-09-10T19:12:33.678952+0000 osd.5 (osd.5) 561 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:12:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:34 smithi165 bash[22591]: cluster 2024-09-10T19:12:32.540454+0000 mgr.y (mgr.24526) 897 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:34 smithi165 bash[22591]: cluster 2024-09-10T19:12:33.103900+0000 osd.0 (osd.0) 547 : cluster [DBG] 12.1d deep-scrub starts 2024-09-10T19:12:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:34 smithi165 bash[22591]: cluster 2024-09-10T19:12:33.107844+0000 osd.0 (osd.0) 548 : cluster [DBG] 12.1d deep-scrub ok 2024-09-10T19:12:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:34 smithi165 bash[22591]: cluster 2024-09-10T19:12:33.673837+0000 osd.5 (osd.5) 560 : cluster [DBG] 3.16 scrub starts 2024-09-10T19:12:34.612 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:34 smithi165 bash[22591]: cluster 2024-09-10T19:12:33.678952+0000 osd.5 (osd.5) 561 : cluster [DBG] 3.16 scrub ok 2024-09-10T19:12:35.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:35 smithi083 bash[18629]: cluster 2024-09-10T19:12:34.541060+0000 mgr.y (mgr.24526) 898 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:35.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:35 smithi083 bash[32381]: cluster 2024-09-10T19:12:34.541060+0000 mgr.y (mgr.24526) 898 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:35.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:35 smithi165 bash[22591]: cluster 2024-09-10T19:12:34.541060+0000 mgr.y (mgr.24526) 898 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:36.768 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[18629]: cluster 2024-09-10T19:12:35.247713+0000 osd.1 (osd.1) 115 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[18629]: cluster 2024-09-10T19:12:35.251448+0000 osd.1 (osd.1) 116 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[18629]: cluster 2024-09-10T19:12:35.365593+0000 osd.4 (osd.4) 530 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[18629]: cluster 2024-09-10T19:12:35.370038+0000 osd.4 (osd.4) 531 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[32381]: cluster 2024-09-10T19:12:35.247713+0000 osd.1 (osd.1) 115 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[32381]: cluster 2024-09-10T19:12:35.251448+0000 osd.1 (osd.1) 116 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[32381]: cluster 2024-09-10T19:12:35.365593+0000 osd.4 (osd.4) 530 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:12:36.769 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:36 smithi083 bash[32381]: cluster 2024-09-10T19:12:35.370038+0000 osd.4 (osd.4) 531 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:12:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:36 smithi165 bash[22591]: cluster 2024-09-10T19:12:35.247713+0000 osd.1 (osd.1) 115 : cluster [DBG] 12.7 deep-scrub starts 2024-09-10T19:12:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:36 smithi165 bash[22591]: cluster 2024-09-10T19:12:35.251448+0000 osd.1 (osd.1) 116 : cluster [DBG] 12.7 deep-scrub ok 2024-09-10T19:12:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:36 smithi165 bash[22591]: cluster 2024-09-10T19:12:35.365593+0000 osd.4 (osd.4) 530 : cluster [DBG] 1.5 scrub starts 2024-09-10T19:12:36.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:36 smithi165 bash[22591]: cluster 2024-09-10T19:12:35.370038+0000 osd.4 (osd.4) 531 : cluster [DBG] 1.5 scrub ok 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.412863+0000 osd.4 (osd.4) 532 : cluster [DBG] 2.f scrub starts 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.417384+0000 osd.4 (osd.4) 533 : cluster [DBG] 2.f scrub ok 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.542904+0000 mgr.y (mgr.24526) 899 : cluster [DBG] pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.709227+0000 osd.5 (osd.5) 562 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.709635+0000 osd.5 (osd.5) 563 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.713362+0000 osd.5 (osd.5) 564 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:12:37.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[32381]: cluster 2024-09-10T19:12:36.716718+0000 osd.5 (osd.5) 565 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.412863+0000 osd.4 (osd.4) 532 : cluster [DBG] 2.f scrub starts 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.417384+0000 osd.4 (osd.4) 533 : cluster [DBG] 2.f scrub ok 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.542904+0000 mgr.y (mgr.24526) 899 : cluster [DBG] pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.709227+0000 osd.5 (osd.5) 562 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.709635+0000 osd.5 (osd.5) 563 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.713362+0000 osd.5 (osd.5) 564 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:12:37.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:37 smithi083 bash[18629]: cluster 2024-09-10T19:12:36.716718+0000 osd.5 (osd.5) 565 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.412863+0000 osd.4 (osd.4) 532 : cluster [DBG] 2.f scrub starts 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.417384+0000 osd.4 (osd.4) 533 : cluster [DBG] 2.f scrub ok 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.542904+0000 mgr.y (mgr.24526) 899 : cluster [DBG] pgmap v1477: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.709227+0000 osd.5 (osd.5) 562 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.709635+0000 osd.5 (osd.5) 563 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.713362+0000 osd.5 (osd.5) 564 : cluster [DBG] 12.14 scrub starts 2024-09-10T19:12:37.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:37 smithi165 bash[22591]: cluster 2024-09-10T19:12:36.716718+0000 osd.5 (osd.5) 565 : cluster [DBG] 12.14 scrub ok 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.176620+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.179883+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.376370+0000 osd.6 (osd.6) 525 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.380149+0000 osd.6 (osd.6) 526 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.459287+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:12:38.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.463291+0000 osd.4 (osd.4) 535 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.520725+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.750196+0000 osd.7 (osd.7) 562 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[18629]: cluster 2024-09-10T19:12:37.756929+0000 osd.7 (osd.7) 563 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.176620+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.179883+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.376370+0000 osd.6 (osd.6) 525 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.380149+0000 osd.6 (osd.6) 526 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.459287+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.463291+0000 osd.4 (osd.4) 535 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:12:38.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.520725+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-09-10T19:12:38.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.750196+0000 osd.7 (osd.7) 562 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:12:38.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:38 smithi083 bash[32381]: cluster 2024-09-10T19:12:37.756929+0000 osd.7 (osd.7) 563 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:12:38.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.176620+0000 osd.1 (osd.1) 117 : cluster [DBG] 4.2 scrub starts 2024-09-10T19:12:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.179883+0000 osd.1 (osd.1) 118 : cluster [DBG] 4.2 scrub ok 2024-09-10T19:12:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.376370+0000 osd.6 (osd.6) 525 : cluster [DBG] 12.15 scrub starts 2024-09-10T19:12:38.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.380149+0000 osd.6 (osd.6) 526 : cluster [DBG] 12.15 scrub ok 2024-09-10T19:12:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.459287+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:12:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.463291+0000 osd.4 (osd.4) 535 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:12:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.520725+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-09-10T19:12:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.750196+0000 osd.7 (osd.7) 562 : cluster [DBG] 1.9 scrub starts 2024-09-10T19:12:38.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:38 smithi165 bash[22591]: cluster 2024-09-10T19:12:37.756929+0000 osd.7 (osd.7) 563 : cluster [DBG] 1.9 scrub ok 2024-09-10T19:12:39.851 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[18629]: cluster 2024-09-10T19:12:38.543666+0000 mgr.y (mgr.24526) 900 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:39.851 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[18629]: cluster 2024-09-10T19:12:38.683800+0000 osd.5 (osd.5) 566 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:12:39.851 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[18629]: cluster 2024-09-10T19:12:38.687206+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:12:39.851 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[18629]: audit 2024-09-10T19:12:38.692544+0000 mgr.y (mgr.24526) 901 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.851 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[32381]: cluster 2024-09-10T19:12:38.543666+0000 mgr.y (mgr.24526) 900 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:39.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[32381]: cluster 2024-09-10T19:12:38.683800+0000 osd.5 (osd.5) 566 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:12:39.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[32381]: cluster 2024-09-10T19:12:38.687206+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:12:39.852 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:39 smithi083 bash[32381]: audit 2024-09-10T19:12:38.692544+0000 mgr.y (mgr.24526) 901 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:39 smithi165 bash[22591]: cluster 2024-09-10T19:12:38.543666+0000 mgr.y (mgr.24526) 900 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:39 smithi165 bash[22591]: cluster 2024-09-10T19:12:38.683800+0000 osd.5 (osd.5) 566 : cluster [DBG] 6.7 scrub starts 2024-09-10T19:12:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:39 smithi165 bash[22591]: cluster 2024-09-10T19:12:38.687206+0000 osd.5 (osd.5) 567 : cluster [DBG] 6.7 scrub ok 2024-09-10T19:12:39.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:39 smithi165 bash[22591]: audit 2024-09-10T19:12:38.692544+0000 mgr.y (mgr.24526) 901 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:38.980377+0000 osd.3 (osd.3) 590 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:38.983635+0000 osd.3 (osd.3) 591 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:39.098010+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:39.101264+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:39.689326+0000 osd.7 (osd.7) 564 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:12:40.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[18629]: cluster 2024-09-10T19:12:39.694332+0000 osd.7 (osd.7) 565 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:38.980377+0000 osd.3 (osd.3) 590 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:38.983635+0000 osd.3 (osd.3) 591 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:39.098010+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:39.101264+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:39.689326+0000 osd.7 (osd.7) 564 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:12:40.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:40 smithi083 bash[32381]: cluster 2024-09-10T19:12:39.694332+0000 osd.7 (osd.7) 565 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:38.980377+0000 osd.3 (osd.3) 590 : cluster [DBG] 12.12 scrub starts 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:38.983635+0000 osd.3 (osd.3) 591 : cluster [DBG] 12.12 scrub ok 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:39.098010+0000 osd.1 (osd.1) 119 : cluster [DBG] 6.1 scrub starts 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:39.101264+0000 osd.1 (osd.1) 120 : cluster [DBG] 6.1 scrub ok 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:39.689326+0000 osd.7 (osd.7) 564 : cluster [DBG] 2.1c scrub starts 2024-09-10T19:12:40.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:40 smithi165 bash[22591]: cluster 2024-09-10T19:12:39.694332+0000 osd.7 (osd.7) 565 : cluster [DBG] 2.1c scrub ok 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[18629]: cluster 2024-09-10T19:12:40.544484+0000 mgr.y (mgr.24526) 902 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[18629]: cluster 2024-09-10T19:12:40.690902+0000 osd.5 (osd.5) 568 : cluster [DBG] 3.c scrub starts 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[18629]: cluster 2024-09-10T19:12:40.695545+0000 osd.5 (osd.5) 569 : cluster [DBG] 3.c scrub ok 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[32381]: cluster 2024-09-10T19:12:40.544484+0000 mgr.y (mgr.24526) 902 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[32381]: cluster 2024-09-10T19:12:40.690902+0000 osd.5 (osd.5) 568 : cluster [DBG] 3.c scrub starts 2024-09-10T19:12:41.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:41 smithi083 bash[32381]: cluster 2024-09-10T19:12:40.695545+0000 osd.5 (osd.5) 569 : cluster [DBG] 3.c scrub ok 2024-09-10T19:12:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:41 smithi165 bash[22591]: cluster 2024-09-10T19:12:40.544484+0000 mgr.y (mgr.24526) 902 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:41 smithi165 bash[22591]: cluster 2024-09-10T19:12:40.690902+0000 osd.5 (osd.5) 568 : cluster [DBG] 3.c scrub starts 2024-09-10T19:12:41.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:41 smithi165 bash[22591]: cluster 2024-09-10T19:12:40.695545+0000 osd.5 (osd.5) 569 : cluster [DBG] 3.c scrub ok 2024-09-10T19:12:41.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:40.986031+0000 osd.3 (osd.3) 592 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:40.989554+0000 osd.3 (osd.3) 593 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:41.092656+0000 osd.0 (osd.0) 549 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:41.096636+0000 osd.0 (osd.0) 550 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:41.494419+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:41.494847+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:42.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:42 smithi165 bash[22591]: cluster 2024-09-10T19:12:41.547175+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-09-10T19:12:43.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:42] "GET /metrics HTTP/1.1" 200 224098 "" "Prometheus/2.43.0" 2024-09-10T19:12:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:40.986031+0000 osd.3 (osd.3) 592 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:12:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:40.989554+0000 osd.3 (osd.3) 593 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:12:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:41.092656+0000 osd.0 (osd.0) 549 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:12:43.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:41.096636+0000 osd.0 (osd.0) 550 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:41.494419+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:41.494847+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[18629]: cluster 2024-09-10T19:12:41.547175+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:40.986031+0000 osd.3 (osd.3) 592 : cluster [DBG] 12.8 scrub starts 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:40.989554+0000 osd.3 (osd.3) 593 : cluster [DBG] 12.8 scrub ok 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:41.092656+0000 osd.0 (osd.0) 549 : cluster [DBG] 4.3 scrub starts 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:41.096636+0000 osd.0 (osd.0) 550 : cluster [DBG] 4.3 scrub ok 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:41.494419+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:41.494847+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:43.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:42 smithi083 bash[32381]: cluster 2024-09-10T19:12:41.547175+0000 mon.a (mon.0) 3883 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-09-10T19:12:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:43 smithi165 bash[22591]: cluster 2024-09-10T19:12:42.090843+0000 osd.0 (osd.0) 551 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:12:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:43 smithi165 bash[22591]: cluster 2024-09-10T19:12:42.093864+0000 osd.0 (osd.0) 552 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:12:43.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:43 smithi165 bash[22591]: cluster 2024-09-10T19:12:42.545165+0000 mgr.y (mgr.24526) 903 : cluster [DBG] pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[18629]: cluster 2024-09-10T19:12:42.090843+0000 osd.0 (osd.0) 551 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[18629]: cluster 2024-09-10T19:12:42.093864+0000 osd.0 (osd.0) 552 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[18629]: cluster 2024-09-10T19:12:42.545165+0000 mgr.y (mgr.24526) 903 : cluster [DBG] pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[32381]: cluster 2024-09-10T19:12:42.090843+0000 osd.0 (osd.0) 551 : cluster [DBG] 2.13 scrub starts 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[32381]: cluster 2024-09-10T19:12:42.093864+0000 osd.0 (osd.0) 552 : cluster [DBG] 2.13 scrub ok 2024-09-10T19:12:44.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:43 smithi083 bash[32381]: cluster 2024-09-10T19:12:42.545165+0000 mgr.y (mgr.24526) 903 : cluster [DBG] pgmap v1482: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:44.809 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.259593+0000 osd.7 (osd.7) 566 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:44.809 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.260041+0000 osd.7 (osd.7) 567 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:44.809 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.437374+0000 osd.6 (osd.6) 527 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:12:44.809 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.440807+0000 osd.6 (osd.6) 528 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:12:44.809 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.563554+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-09-10T19:12:44.810 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.755812+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:12:44.810 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[18629]: cluster 2024-09-10T19:12:43.759741+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:12:44.810 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.259593+0000 osd.7 (osd.7) 566 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:44.810 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.260041+0000 osd.7 (osd.7) 567 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:44.810 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.437374+0000 osd.6 (osd.6) 527 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:12:44.811 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.440807+0000 osd.6 (osd.6) 528 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:12:44.811 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.563554+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-09-10T19:12:44.811 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.755812+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:12:44.811 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:44 smithi083 bash[32381]: cluster 2024-09-10T19:12:43.759741+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:12:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.259593+0000 osd.7 (osd.7) 566 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.260041+0000 osd.7 (osd.7) 567 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:44.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.437374+0000 osd.6 (osd.6) 527 : cluster [DBG] 4.5 scrub starts 2024-09-10T19:12:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.440807+0000 osd.6 (osd.6) 528 : cluster [DBG] 4.5 scrub ok 2024-09-10T19:12:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.563554+0000 mon.a (mon.0) 3884 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-09-10T19:12:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.755812+0000 osd.7 (osd.7) 568 : cluster [DBG] 1.0 scrub starts 2024-09-10T19:12:44.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:44 smithi165 bash[22591]: cluster 2024-09-10T19:12:43.759741+0000 osd.7 (osd.7) 569 : cluster [DBG] 1.0 scrub ok 2024-09-10T19:12:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.921+0000 7fd89cc6f640 1 -- 172.21.15.83:0/2613646489 >> v1:172.21.15.83:6790/0 conn(0x7fd8980fe180 legacy=0x7fd8981075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.921+0000 7fd89cc6f640 1 -- 172.21.15.83:0/2613646489 shutdown_connections 2024-09-10T19:12:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.921+0000 7fd89cc6f640 1 -- 172.21.15.83:0/2613646489 >> 172.21.15.83:0/2613646489 conn(0x7fd8980f70b0 msgr2=0x7fd8980f94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.921+0000 7fd89cc6f640 1 -- 172.21.15.83:0/2613646489 shutdown_connections 2024-09-10T19:12:44.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.921+0000 7fd89cc6f640 1 -- 172.21.15.83:0/2613646489 wait complete. 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 librados: starting msgr at 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 librados: starting objecter 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 Processor -- start 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- start start 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 librados: setting wanted keys 2024-09-10T19:12:44.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 librados: calling monclient init 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd89811e830 con 0x7fd898108f10 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd89811f9d0 con 0x7fd8980fd5e0 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd898120bf0 con 0x7fd898107800 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd895d9b640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd898108f10 0x7fd898119950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36522/0 (socket says 172.21.15.83:36522) 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd895d9b640 1 -- 172.21.15.83:0/1817389336 learned_addr learned my addr 172.21.15.83:0/1817389336 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter ms_handle_connect 0x7fd898107800 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter ms_handle_connect 0x7fd898108f10 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter ms_handle_connect 0x7fd8980fd5e0 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419172606 0 0) 0x7fd89811e830 con 0x7fd898108f10 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8800034e0 con 0x7fd898108f10 2024-09-10T19:12:44.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336401683 0 0) 0x7fd898120bf0 con 0x7fd898107800 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89811e830 con 0x7fd898107800 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162250548 0 0) 0x7fd89811f9d0 con 0x7fd8980fd5e0 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898120bf0 con 0x7fd8980fd5e0 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717808508 0 0) 0x7fd8800034e0 con 0x7fd898108f10 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd89811f9d0 con 0x7fd898108f10 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd888001430 con 0x7fd898108f10 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517814080 0 0) 0x7fd89811e830 con 0x7fd898107800 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8800034e0 con 0x7fd898107800 2024-09-10T19:12:44.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd890001bc0 con 0x7fd898107800 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1678528336 0 0) 0x7fd898120bf0 con 0x7fd8980fd5e0 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd89811e830 con 0x7fd8980fd5e0 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd88c0018c0 con 0x7fd8980fd5e0 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 801519045 0 0) 0x7fd89811f9d0 con 0x7fd898108f10 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 >> v1:172.21.15.83:6790/0 conn(0x7fd898107800 legacy=0x7fd89811d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 >> v1:172.21.15.165:6789/0 conn(0x7fd8980fd5e0 legacy=0x7fd8981070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898121df0 con 0x7fd898108f10 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89811ea20 con 0x7fd898108f10 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 10 client.25959.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:44.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89811ee70 con 0x7fd898108f10 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 1 librados: init done 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fd888002400 con 0x7fd898108f10 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fd888003570 con 0x7fd898108f10 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 10 client.25959.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.925+0000 7fd89cc6f640 15 client.25959.objecter _session_command_op_assign -1 1 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd89cc6f640 20 client.25959.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd89cc6f640 10 client.25959.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:44.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8780028b0 con 0x7fd898108f10 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fd8880037b0 con 0x7fd898108f10 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(896..896 src has 838..896) v4 ==== 7363+0+0 (unknown 2110548846 0 0) 0x7fd88808a5f0 con 0x7fd898108f10 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 10 client.25959.objecter ms_dispatch 0x7fd898000b90 osd_map(896..896 src has 838..896) v4 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 3 client.25959.objecter handle_osd_map got epochs [896,896] > 0 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 3 client.25959.objecter handle_osd_map decoding full epoch 896 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 10 client.25959.objecter checking command 1 2024-09-10T19:12:44.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter _get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 15 client.25959.objecter _session_command_op_remove -1 1 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter _get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 15 client.25959.objecter _session_command_op_assign 0 1 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 4 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 10 client.25959.objecter _send_command 1 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6801/2535527483 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd880081a10 con 0x7fd88007dd10 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 20 client.25959.objecter dump_active .. 0 homeless 2024-09-10T19:12:44.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=896) v2 ==== 24+0+0 (unknown 2995603004 0 0) 0x7fd88808a9c0 con 0x7fd898108f10 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 10 client.25959.objecter ms_handle_connect 0x7fd880071860 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.929+0000 7fd886ffd640 10 client.25959.objecter ms_handle_connect 0x7fd88007dd10 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.933+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== osd.0 v1:172.21.15.83:6801/2535527483 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fd880081a10 con 0x7fd88007dd10 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.933+0000 7fd886ffd640 10 client.25959.objecter ms_dispatch 0x7fd898000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.933+0000 7fd886ffd640 10 client.25959.objecter _finish_command 1 = system:0 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.933+0000 7fd886ffd640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 4 2024-09-10T19:12:44.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.933+0000 7fd886ffd640 15 client.25959.objecter _session_command_op_remove 0 1 2024-09-10T19:12:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 client.25959.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 15 client.25959.objecter _session_command_op_assign -1 2 2024-09-10T19:12:44.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter _get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter _get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 15 client.25959.objecter _session_command_op_remove -1 2 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter get_session s=0x7fd88007d660 osd=0 3 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 15 client.25959.objecter _session_command_op_assign 0 2 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 4 2024-09-10T19:12:44.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 client.25959.objecter _send_command 2 2024-09-10T19:12:44.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 --> v1:172.21.15.83:6801/2535527483 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd878001190 con 0x7fd88007dd10 2024-09-10T19:12:44.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd886ffd640 1 -- 172.21.15.83:0/1817389336 <== osd.0 v1:172.21.15.83:6801/2535527483 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1521147262) 0x7fd878001190 con 0x7fd88007dd10 2024-09-10T19:12:44.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd886ffd640 10 client.25959.objecter ms_dispatch 0x7fd898000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:44.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd886ffd640 10 client.25959.objecter _finish_command 2 = system:0 2024-09-10T19:12:44.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd886ffd640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 4 2024-09-10T19:12:44.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd886ffd640 15 client.25959.objecter _session_command_op_remove 0 2 2024-09-10T19:12:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 librados: watch_flush enter 2024-09-10T19:12:44.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 librados: watch_flush exit 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 client.25959.objecter close_session for osd.0 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 >> v1:172.21.15.83:6801/2535527483 conn(0x7fd88007dd10 legacy=0x7fd8800801a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter put_session s=0x7fd88007d660 osd=0 1 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 20 client.25959.objecter shutdown clearing up homeless session... 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 10 client.25959.objecter successfully canceled tick 2024-09-10T19:12:44.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 >> v1:172.21.15.83:6800/3071793735 conn(0x7fd880071860 legacy=0x7fd880073d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 >> v1:172.21.15.83:6789/0 conn(0x7fd898108f10 legacy=0x7fd898119950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.949+0000 7fd896d9d640 1 -- 172.21.15.83:0/1817389336 reap_dead start 2024-09-10T19:12:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.953+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 shutdown_connections 2024-09-10T19:12:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.953+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 >> 172.21.15.83:0/1817389336 conn(0x7fd8980f70b0 msgr2=0x7fd89810b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:44.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.953+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 shutdown_connections 2024-09-10T19:12:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.953+0000 7fd89cc6f640 1 -- 172.21.15.83:0/1817389336 wait complete. 2024-09-10T19:12:44.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:44.953+0000 7fd89cc6f640 1 librados: shutdown 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: cluster 2024-09-10T19:12:44.545789+0000 mgr.y (mgr.24526) 904 : cluster [DBG] pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: audit 2024-09-10T19:12:44.583471+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: cluster 2024-09-10T19:12:44.720313+0000 osd.5 (osd.5) 570 : cluster [DBG] 1.a scrub starts 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: cluster 2024-09-10T19:12:44.724114+0000 osd.5 (osd.5) 571 : cluster [DBG] 1.a scrub ok 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: cluster 2024-09-10T19:12:44.829763+0000 osd.6 (osd.6) 529 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:45.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:45 smithi165 bash[22591]: cluster 2024-09-10T19:12:44.830070+0000 osd.6 (osd.6) 530 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: cluster 2024-09-10T19:12:44.545789+0000 mgr.y (mgr.24526) 904 : cluster [DBG] pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: audit 2024-09-10T19:12:44.583471+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: cluster 2024-09-10T19:12:44.720313+0000 osd.5 (osd.5) 570 : cluster [DBG] 1.a scrub starts 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: cluster 2024-09-10T19:12:44.724114+0000 osd.5 (osd.5) 571 : cluster [DBG] 1.a scrub ok 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: cluster 2024-09-10T19:12:44.829763+0000 osd.6 (osd.6) 529 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:46.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[18629]: cluster 2024-09-10T19:12:44.830070+0000 osd.6 (osd.6) 530 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: cluster 2024-09-10T19:12:44.545789+0000 mgr.y (mgr.24526) 904 : cluster [DBG] pgmap v1484: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 op/s 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: audit 2024-09-10T19:12:44.583471+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: cluster 2024-09-10T19:12:44.720313+0000 osd.5 (osd.5) 570 : cluster [DBG] 1.a scrub starts 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: cluster 2024-09-10T19:12:44.724114+0000 osd.5 (osd.5) 571 : cluster [DBG] 1.a scrub ok 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: cluster 2024-09-10T19:12:44.829763+0000 osd.6 (osd.6) 529 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:46.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:45 smithi083 bash[32381]: cluster 2024-09-10T19:12:44.830070+0000 osd.6 (osd.6) 530 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.102267+0000 osd.0 (osd.0) 553 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:12:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.106055+0000 osd.0 (osd.0) 554 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:12:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.432047+0000 osd.6 (osd.6) 531 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:12:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.435881+0000 osd.6 (osd.6) 532 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:12:46.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.568549+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-09-10T19:12:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.102267+0000 osd.0 (osd.0) 553 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:12:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.106055+0000 osd.0 (osd.0) 554 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:12:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.432047+0000 osd.6 (osd.6) 531 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:12:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.435881+0000 osd.6 (osd.6) 532 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:12:46.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:46 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.568549+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-09-10T19:12:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:46 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.102267+0000 osd.0 (osd.0) 553 : cluster [DBG] 4.1e scrub starts 2024-09-10T19:12:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:46 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.106055+0000 osd.0 (osd.0) 554 : cluster [DBG] 4.1e scrub ok 2024-09-10T19:12:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:46 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.432047+0000 osd.6 (osd.6) 531 : cluster [DBG] 2.14 scrub starts 2024-09-10T19:12:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:46 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.435881+0000 osd.6 (osd.6) 532 : cluster [DBG] 2.14 scrub ok 2024-09-10T19:12:46.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:46 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.568549+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-09-10T19:12:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:47 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.971906+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:12:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:47 smithi165 bash[22591]: cluster 2024-09-10T19:12:45.975981+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:12:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:47 smithi165 bash[22591]: cluster 2024-09-10T19:12:46.547781+0000 mgr.y (mgr.24526) 905 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:47 smithi165 bash[22591]: cluster 2024-09-10T19:12:46.744908+0000 osd.5 (osd.5) 572 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:12:47.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:47 smithi165 bash[22591]: cluster 2024-09-10T19:12:46.748625+0000 osd.5 (osd.5) 573 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:12:48.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.971906+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[18629]: cluster 2024-09-10T19:12:45.975981+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[18629]: cluster 2024-09-10T19:12:46.547781+0000 mgr.y (mgr.24526) 905 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[18629]: cluster 2024-09-10T19:12:46.744908+0000 osd.5 (osd.5) 572 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[18629]: cluster 2024-09-10T19:12:46.748625+0000 osd.5 (osd.5) 573 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.971906+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.1d deep-scrub starts 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[32381]: cluster 2024-09-10T19:12:45.975981+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.1d deep-scrub ok 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[32381]: cluster 2024-09-10T19:12:46.547781+0000 mgr.y (mgr.24526) 905 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[32381]: cluster 2024-09-10T19:12:46.744908+0000 osd.5 (osd.5) 572 : cluster [DBG] 1.1 scrub starts 2024-09-10T19:12:48.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:47 smithi083 bash[32381]: cluster 2024-09-10T19:12:46.748625+0000 osd.5 (osd.5) 573 : cluster [DBG] 1.1 scrub ok 2024-09-10T19:12:48.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.198377+0000 osd.0 (osd.0) 555 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:12:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.202037+0000 osd.0 (osd.0) 556 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:12:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.566309+0000 osd.4 (osd.4) 536 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.566665+0000 osd.4 (osd.4) 537 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:48.863 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.581834+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-09-10T19:12:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.822127+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.f scrub starts 2024-09-10T19:12:48.864 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:48 smithi165 bash[22591]: cluster 2024-09-10T19:12:47.831193+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.f scrub ok 2024-09-10T19:12:49.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.198377+0000 osd.0 (osd.0) 555 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:12:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.202037+0000 osd.0 (osd.0) 556 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:12:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.566309+0000 osd.4 (osd.4) 536 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.566665+0000 osd.4 (osd.4) 537 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.581834+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-09-10T19:12:49.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.822127+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.f scrub starts 2024-09-10T19:12:49.043 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[18629]: cluster 2024-09-10T19:12:47.831193+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.f scrub ok 2024-09-10T19:12:49.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.198377+0000 osd.0 (osd.0) 555 : cluster [DBG] 6.0 scrub starts 2024-09-10T19:12:49.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.202037+0000 osd.0 (osd.0) 556 : cluster [DBG] 6.0 scrub ok 2024-09-10T19:12:49.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.566309+0000 osd.4 (osd.4) 536 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:49.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.566665+0000 osd.4 (osd.4) 537 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:49.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.581834+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-09-10T19:12:49.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.822127+0000 osd.7 (osd.7) 570 : cluster [DBG] 3.f scrub starts 2024-09-10T19:12:49.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:48 smithi083 bash[32381]: cluster 2024-09-10T19:12:47.831193+0000 osd.7 (osd.7) 571 : cluster [DBG] 3.f scrub ok 2024-09-10T19:12:49.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.577+0000 7ff712569640 1 -- 172.21.15.83:0/3437259181 >> v1:172.21.15.83:6790/0 conn(0x7ff70c107e30 legacy=0x7ff70c10a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/3437259181 shutdown_connections 2024-09-10T19:12:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/3437259181 >> 172.21.15.83:0/3437259181 conn(0x7ff70c075290 msgr2=0x7ff70c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/3437259181 shutdown_connections 2024-09-10T19:12:49.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/3437259181 wait complete. 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 librados: starting msgr at 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 librados: starting objecter 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 Processor -- start 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- start start 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 librados: setting wanted keys 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 librados: calling monclient init 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff70c11e1c0 con 0x7ff70c1014e0 2024-09-10T19:12:49.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff70c11f3c0 con 0x7ff70c1042a0 2024-09-10T19:12:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff70c1205c0 con 0x7ff70c107e30 2024-09-10T19:12:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff710b06640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7ff70c107e30 0x7ff70c11c8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59512/0 (socket says 172.21.15.83:59512) 2024-09-10T19:12:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff710b06640 1 -- 172.21.15.83:0/1418466785 learned_addr learned my addr 172.21.15.83:0/1418466785 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:49.585 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff70bfff640 1 --1- 172.21.15.83:0/1418466785 >> v1:172.21.15.83:6789/0 conn(0x7ff70c1014e0 0x7ff70c064ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36550/0 (socket says 172.21.15.83:36550) 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter ms_handle_connect 0x7ff70c107e30 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter ms_handle_connect 0x7ff70c1014e0 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter ms_handle_connect 0x7ff70c1042a0 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561672639 0 0) 0x7ff70c1205c0 con 0x7ff70c107e30 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f40034e0 con 0x7ff70c107e30 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732530444 0 0) 0x7ff70c11f3c0 con 0x7ff70c1042a0 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff70c1205c0 con 0x7ff70c1042a0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246957260 0 0) 0x7ff70c11e1c0 con 0x7ff70c1014e0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff70c11f3c0 con 0x7ff70c1014e0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034032491 0 0) 0x7ff6f40034e0 con 0x7ff70c107e30 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff70c11e1c0 con 0x7ff70c107e30 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff704002dd0 con 0x7ff70c107e30 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4203158128 0 0) 0x7ff70c11f3c0 con 0x7ff70c1014e0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f40034e0 con 0x7ff70c1014e0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446477171 0 0) 0x7ff70c1205c0 con 0x7ff70c1042a0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff70c11f3c0 con 0x7ff70c1042a0 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3967738417 0 0) 0x7ff70c11e1c0 con 0x7ff70c107e30 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 >> v1:172.21.15.165:6789/0 conn(0x7ff70c1042a0 legacy=0x7ff70c0655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 >> v1:172.21.15.83:6789/0 conn(0x7ff70c1014e0 legacy=0x7ff70c064ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c1217c0 con 0x7ff70c107e30 2024-09-10T19:12:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c11f5b0 con 0x7ff70c107e30 2024-09-10T19:12:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 10 client.50051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c11fa00 con 0x7ff70c107e30 2024-09-10T19:12:49.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.581+0000 7ff712569640 1 librados: init done 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff704001ea0 con 0x7ff70c107e30 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff704003a00 con 0x7ff70c107e30 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff712569640 10 client.50051.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff712569640 15 client.50051.objecter _session_command_op_assign -1 1 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff712569640 20 client.50051.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff712569640 10 client.50051.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6f80028b0 con 0x7ff70c107e30 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff704003b60 con 0x7ff70c107e30 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7ff70408a7e0 con 0x7ff70c107e30 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 10 client.50051.objecter ms_dispatch 0x7ff70c000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:49.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 3 client.50051.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 3 client.50051.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 10 client.50051.objecter checking command 1 2024-09-10T19:12:49.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter _get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 15 client.50051.objecter _session_command_op_remove -1 1 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter _get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 15 client.50051.objecter _session_command_op_assign 7 1 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 4 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 10 client.50051.objecter _send_command 1 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.165:6812/1610151650 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6f40819c0 con 0x7ff6f407dcc0 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 20 client.50051.objecter dump_active .. 0 homeless 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7ff70408abf0 con 0x7ff70c107e30 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.585+0000 7ff708ff9640 10 client.50051.objecter ms_handle_connect 0x7ff6f4071810 2024-09-10T19:12:49.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 10 client.50051.objecter ms_handle_connect 0x7ff6f407dcc0 2024-09-10T19:12:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== osd.7 v1:172.21.15.165:6812/1610151650 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7ff6f40819c0 con 0x7ff6f407dcc0 2024-09-10T19:12:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 10 client.50051.objecter ms_dispatch 0x7ff70c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 10 client.50051.objecter _finish_command 1 = system:0 2024-09-10T19:12:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 4 2024-09-10T19:12:49.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.589+0000 7ff708ff9640 15 client.50051.objecter _session_command_op_remove 7 1 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 client.50051.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 15 client.50051.objecter _session_command_op_assign -1 2 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter _get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter _get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 15 client.50051.objecter _session_command_op_remove -1 2 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter get_session s=0x7ff6f407d610 osd=7 3 2024-09-10T19:12:49.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 15 client.50051.objecter _session_command_op_assign 7 2 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 4 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 client.50051.objecter _send_command 2 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 --> v1:172.21.15.165:6812/1610151650 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff6f8001190 con 0x7ff6f407dcc0 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff708ff9640 1 -- 172.21.15.83:0/1418466785 <== osd.7 v1:172.21.15.165:6812/1610151650 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3687268154) 0x7ff6f8001190 con 0x7ff6f407dcc0 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff708ff9640 10 client.50051.objecter ms_dispatch 0x7ff70c000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff708ff9640 10 client.50051.objecter _finish_command 2 = system:0 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff708ff9640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 4 2024-09-10T19:12:49.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff708ff9640 15 client.50051.objecter _session_command_op_remove 7 2 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 librados: watch_flush enter 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 librados: watch_flush exit 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 client.50051.objecter close_session for osd.7 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 >> v1:172.21.15.165:6812/1610151650 conn(0x7ff6f407dcc0 legacy=0x7ff6f4080150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter put_session s=0x7ff6f407d610 osd=7 1 2024-09-10T19:12:49.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 20 client.50051.objecter shutdown clearing up homeless session... 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 10 client.50051.objecter successfully canceled tick 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff6f4071810 legacy=0x7ff6f4073cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 >> v1:172.21.15.83:6790/0 conn(0x7ff70c107e30 legacy=0x7ff70c11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff710b06640 1 -- 172.21.15.83:0/1418466785 reap_dead start 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 shutdown_connections 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 >> 172.21.15.83:0/1418466785 conn(0x7ff70c075290 msgr2=0x7ff70c0f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 shutdown_connections 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 -- 172.21.15.83:0/1418466785 wait complete. 2024-09-10T19:12:49.615 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:49.609+0000 7ff712569640 1 librados: shutdown 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.442053+0000 osd.4 (osd.4) 538 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.446916+0000 osd.4 (osd.4) 539 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.548581+0000 mgr.y (mgr.24526) 906 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.581138+0000 osd.3 (osd.3) 594 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.581591+0000 osd.3 (osd.3) 595 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: cluster 2024-09-10T19:12:48.598782+0000 mon.a (mon.0) 3887 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-09-10T19:12:49.756 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[18629]: audit 2024-09-10T19:12:48.700517+0000 mgr.y (mgr.24526) 907 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.442053+0000 osd.4 (osd.4) 538 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:12:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.446916+0000 osd.4 (osd.4) 539 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:12:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.548581+0000 mgr.y (mgr.24526) 906 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.581138+0000 osd.3 (osd.3) 594 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:49.757 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.581591+0000 osd.3 (osd.3) 595 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:49.758 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: cluster 2024-09-10T19:12:48.598782+0000 mon.a (mon.0) 3887 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-09-10T19:12:49.758 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:49 smithi083 bash[32381]: audit 2024-09-10T19:12:48.700517+0000 mgr.y (mgr.24526) 907 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.851 INFO:teuthology.orchestra.run.smithi083.stdout:34359738736 2024-09-10T19:12:49.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.442053+0000 osd.4 (osd.4) 538 : cluster [DBG] 5.1 scrub starts 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.446916+0000 osd.4 (osd.4) 539 : cluster [DBG] 5.1 scrub ok 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.548581+0000 mgr.y (mgr.24526) 906 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.581138+0000 osd.3 (osd.3) 594 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.581591+0000 osd.3 (osd.3) 595 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: cluster 2024-09-10T19:12:48.598782+0000 mon.a (mon.0) 3887 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-09-10T19:12:49.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:49 smithi165 bash[22591]: audit 2024-09-10T19:12:48.700517+0000 mgr.y (mgr.24526) 907 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:51.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:12:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:51 smithi165 bash[22591]: cluster 2024-09-10T19:12:50.549605+0000 mgr.y (mgr.24526) 908 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:51 smithi165 bash[22591]: cluster 2024-09-10T19:12:50.750403+0000 osd.5 (osd.5) 574 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:12:51.862 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:51 smithi165 bash[22591]: cluster 2024-09-10T19:12:50.753829+0000 osd.5 (osd.5) 575 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:12:51.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[18629]: cluster 2024-09-10T19:12:50.549605+0000 mgr.y (mgr.24526) 908 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:51.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[18629]: cluster 2024-09-10T19:12:50.750403+0000 osd.5 (osd.5) 574 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:12:51.897 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[18629]: cluster 2024-09-10T19:12:50.753829+0000 osd.5 (osd.5) 575 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:12:51.897 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[32381]: cluster 2024-09-10T19:12:50.549605+0000 mgr.y (mgr.24526) 908 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:12:51.898 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[32381]: cluster 2024-09-10T19:12:50.750403+0000 osd.5 (osd.5) 574 : cluster [DBG] 2.18 scrub starts 2024-09-10T19:12:51.898 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:51 smithi083 bash[32381]: cluster 2024-09-10T19:12:50.753829+0000 osd.5 (osd.5) 575 : cluster [DBG] 2.18 scrub ok 2024-09-10T19:12:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- 172.21.15.83:0/986167681 >> v1:172.21.15.83:6790/0 conn(0x7f91c41014e0 legacy=0x7f91c41018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- 172.21.15.83:0/986167681 shutdown_connections 2024-09-10T19:12:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- 172.21.15.83:0/986167681 >> 172.21.15.83:0/986167681 conn(0x7f91c4075290 msgr2=0x7f91c40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- 172.21.15.83:0/986167681 shutdown_connections 2024-09-10T19:12:51.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- 172.21.15.83:0/986167681 wait complete. 2024-09-10T19:12:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 librados: starting msgr at 2024-09-10T19:12:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 librados: starting objecter 2024-09-10T19:12:52.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 Processor -- start 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- start start 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 librados: setting wanted keys 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 librados: calling monclient init 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c4100e40 con 0x7f91c41042a0 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c411f6c0 con 0x7f91c41014e0 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c96b6640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c4120860 con 0x7f91c4107e30 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c2ffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f91c41014e0 0x7f91c41005b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.83:50102/0 (socket says 172.21.15.83:50102) 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c2ffd640 1 -- 172.21.15.83:0/1207545381 learned_addr learned my addr 172.21.15.83:0/1207545381 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:52.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91c27fc640 1 --1- 172.21.15.83:0/1207545381 >> v1:172.21.15.83:6789/0 conn(0x7f91c41042a0 0x7f91c4119860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36574/0 (socket says 172.21.15.83:36574) 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter ms_handle_connect 0x7f91c4107e30 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter ms_handle_connect 0x7f91c41042a0 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter ms_handle_connect 0x7f91c41014e0 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847938234 0 0) 0x7f91c4120860 con 0x7f91c4107e30 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b0003120 con 0x7f91c4107e30 2024-09-10T19:12:52.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683408475 0 0) 0x7f91c4100e40 con 0x7f91c41042a0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4120860 con 0x7f91c41042a0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288907766 0 0) 0x7f91c411f6c0 con 0x7f91c41014e0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4100e40 con 0x7f91c41014e0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2565805710 0 0) 0x7f91b0003120 con 0x7f91c4107e30 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c411f6c0 con 0x7f91c4107e30 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f91bc001b80 con 0x7f91c4107e30 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886274342 0 0) 0x7f91c4120860 con 0x7f91c41042a0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91b0003120 con 0x7f91c41042a0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f91b40018e0 con 0x7f91c41042a0 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 205375844 0 0) 0x7f91c411f6c0 con 0x7f91c4107e30 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 >> v1:172.21.15.165:6789/0 conn(0x7f91c41014e0 legacy=0x7f91c41005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:51.997+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 >> v1:172.21.15.83:6789/0 conn(0x7f91c41042a0 legacy=0x7f91c4119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c4121a00 con 0x7f91c4107e30 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c411e710 con 0x7f91c4107e30 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 10 client.50057.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c411eb60 con 0x7f91c4107e30 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 1 librados: init done 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f91bc002250 con 0x7f91c4107e30 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f91bc0034d0 con 0x7f91c4107e30 2024-09-10T19:12:52.005 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 10 client.50057.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 15 client.50057.objecter _session_command_op_assign -1 1 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 20 client.50057.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 10 client.50057.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f91a40028b0 con 0x7f91c4107e30 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.001+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f91bc001e40 con 0x7f91c4107e30 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7f91bc087d50 con 0x7f91c4107e30 2024-09-10T19:12:52.006 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter ms_dispatch 0x7f91c4000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 3 client.50057.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 3 client.50057.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter checking command 1 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter _get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 15 client.50057.objecter _session_command_op_remove -1 1 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter _get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 15 client.50057.objecter _session_command_op_assign 5 1 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 4 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter _send_command 1 2024-09-10T19:12:52.007 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.165:6804/571173777 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91b0081520 con 0x7f91b007d820 2024-09-10T19:12:52.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter dump_active .. 0 homeless 2024-09-10T19:12:52.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter ms_handle_connect 0x7f91b0071470 2024-09-10T19:12:52.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7f91bc0880a0 con 0x7f91c4107e30 2024-09-10T19:12:52.008 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter ms_handle_connect 0x7f91b007d820 2024-09-10T19:12:52.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== osd.5 v1:172.21.15.165:6804/571173777 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f91b0081520 con 0x7f91b007d820 2024-09-10T19:12:52.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter ms_dispatch 0x7f91c4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:52.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 10 client.50057.objecter _finish_command 1 = system:0 2024-09-10T19:12:52.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 4 2024-09-10T19:12:52.009 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.005+0000 7f91bb7fe640 15 client.50057.objecter _session_command_op_remove 5 1 2024-09-10T19:12:52.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- 172.21.15.83:0/2584034093 >> v1:172.21.15.83:6790/0 conn(0x7f41541014f0 legacy=0x7f41541018c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- 172.21.15.83:0/2584034093 shutdown_connections 2024-09-10T19:12:52.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- 172.21.15.83:0/2584034093 >> 172.21.15.83:0/2584034093 conn(0x7f4154074a20 msgr2=0x7f41540f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- 172.21.15.83:0/2584034093 shutdown_connections 2024-09-10T19:12:52.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- 172.21.15.83:0/2584034093 wait complete. 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 librados: starting msgr at 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 librados: starting objecter 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 Processor -- start 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- start start 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 librados: setting wanted keys 2024-09-10T19:12:52.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 librados: calling monclient init 2024-09-10T19:12:52.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f415411c0a0 con 0x7f41541014f0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f415411d2a0 con 0x7f4154107e30 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415aac0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f415411e4a0 con 0x7f41541042d0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415885c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f41541014f0 0x7f41540fac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36588/0 (socket says 172.21.15.83:36588) 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f415885c640 1 -- 172.21.15.83:0/2675830669 learned_addr learned my addr 172.21.15.83:0/2675830669 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f4153fff640 1 --1- 172.21.15.83:0/2675830669 >> v1:172.21.15.83:6790/0 conn(0x7f41541042d0 0x7f41540fb3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59562/0 (socket says 172.21.15.83:59562) 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f41517fa640 10 client.?.objecter ms_handle_connect 0x7f41541014f0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f41517fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f41517fa640 10 client.?.objecter ms_handle_connect 0x7f41541042d0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f41517fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.021+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463131748 0 0) 0x7f415411e4a0 con 0x7f41541042d0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41440039c0 con 0x7f41541042d0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180104211 0 0) 0x7f415411c0a0 con 0x7f41541014f0 2024-09-10T19:12:52.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415411e4a0 con 0x7f41541014f0 2024-09-10T19:12:52.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 10 client.?.objecter ms_handle_connect 0x7f4154107e30 2024-09-10T19:12:52.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431306615 0 0) 0x7f415411e4a0 con 0x7f41541014f0 2024-09-10T19:12:52.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415411c0a0 con 0x7f41541014f0 2024-09-10T19:12:52.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4148001500 con 0x7f41541014f0 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871892468 0 0) 0x7f41440039c0 con 0x7f41541042d0 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415411e4a0 con 0x7f41541042d0 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498600483 0 0) 0x7f415411d2a0 con 0x7f4154107e30 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41440039c0 con 0x7f4154107e30 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4140001840 con 0x7f41541042d0 2024-09-10T19:12:52.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 428799286 0 0) 0x7f415411c0a0 con 0x7f41541014f0 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 >> v1:172.21.15.83:6790/0 conn(0x7f41541042d0 legacy=0x7f41540fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 >> v1:172.21.15.165:6789/0 conn(0x7f4154107e30 legacy=0x7f415411a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415411f6a0 con 0x7f41541014f0 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415411e690 con 0x7f41541014f0 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 10 client.25977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415411eb60 con 0x7f41541014f0 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 1 librados: init done 2024-09-10T19:12:52.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4148002340 con 0x7f41541014f0 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4148003580 con 0x7f41541014f0 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 10 client.25977.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 15 client.25977.objecter _session_command_op_assign -1 1 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 20 client.25977.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 10 client.25977.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f41380028b0 con 0x7f41541014f0 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 10 client.50057.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 15 client.50057.objecter _session_command_op_assign -1 2 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 20 client.50057.objecter _get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 20 client.50057.objecter _get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 15 client.50057.objecter _session_command_op_remove -1 2 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 20 client.50057.objecter get_session s=0x7f91b007d170 osd=5 3 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 15 client.50057.objecter _session_command_op_assign 5 2 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 4 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 10 client.50057.objecter _send_command 2 2024-09-10T19:12:52.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.025+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 --> v1:172.21.15.165:6804/571173777 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f91a4001190 con 0x7f91b007d820 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4148001ee0 con 0x7f41541014f0 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91bb7fe640 1 -- 172.21.15.83:0/1207545381 <== osd.5 v1:172.21.15.165:6804/571173777 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 383958530) 0x7f91a4001190 con 0x7f91b007d820 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91bb7fe640 10 client.50057.objecter ms_dispatch 0x7f91c4000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91bb7fe640 10 client.50057.objecter _finish_command 2 = system:0 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91bb7fe640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 4 2024-09-10T19:12:52.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91bb7fe640 15 client.50057.objecter _session_command_op_remove 5 2 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 10 librados: watch_flush enter 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 10 librados: watch_flush exit 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 10 client.50057.objecter close_session for osd.5 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 >> v1:172.21.15.165:6804/571173777 conn(0x7f91b007d820 legacy=0x7f91b007fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 20 client.50057.objecter put_session s=0x7f91b007d170 osd=5 1 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 20 client.50057.objecter shutdown clearing up homeless session... 2024-09-10T19:12:52.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 10 client.50057.objecter successfully canceled tick 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 >> v1:172.21.15.83:6800/3071793735 conn(0x7f91b0071470 legacy=0x7f91b0073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7f414808a460 con 0x7f41541014f0 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 10 client.25977.objecter ms_dispatch 0x7f4154000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 3 client.25977.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 3 client.25977.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:52.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 >> v1:172.21.15.83:6790/0 conn(0x7f91c4107e30 legacy=0x7f91c411cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c37fe640 1 -- 172.21.15.83:0/1207545381 reap_dead start 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 shutdown_connections 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 >> 172.21.15.83:0/1207545381 conn(0x7f91c4075290 msgr2=0x7f91c4064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 shutdown_connections 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 10 client.25977.objecter checking command 1 2024-09-10T19:12:52.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 -- 172.21.15.83:0/1207545381 wait complete. 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter _get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 15 client.25977.objecter _session_command_op_remove -1 1 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter _get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 15 client.25977.objecter _session_command_op_assign 6 1 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 4 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 10 client.25977.objecter _send_command 1 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.165:6808/1317922936 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41440818a0 con 0x7f414407dba0 2024-09-10T19:12:52.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 20 client.25977.objecter dump_active .. 0 homeless 2024-09-10T19:12:52.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7f414808a860 con 0x7f41541014f0 2024-09-10T19:12:52.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f91c96b6640 1 librados: shutdown 2024-09-10T19:12:52.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 10 client.25977.objecter ms_handle_connect 0x7f41440716f0 2024-09-10T19:12:52.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.029+0000 7f41517fa640 10 client.25977.objecter ms_handle_connect 0x7f414407dba0 2024-09-10T19:12:52.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.033+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== osd.6 v1:172.21.15.165:6808/1317922936 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f41440818a0 con 0x7f414407dba0 2024-09-10T19:12:52.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.033+0000 7f41517fa640 10 client.25977.objecter ms_dispatch 0x7f4154000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:52.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.033+0000 7f41517fa640 10 client.25977.objecter _finish_command 1 = system:0 2024-09-10T19:12:52.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.033+0000 7f41517fa640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 4 2024-09-10T19:12:52.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.033+0000 7f41517fa640 15 client.25977.objecter _session_command_op_remove 6 1 2024-09-10T19:12:52.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 client.25977.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:52.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 15 client.25977.objecter _session_command_op_assign -1 2 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter _get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter _get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 15 client.25977.objecter _session_command_op_remove -1 2 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter get_session s=0x7f414407d4f0 osd=6 3 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 15 client.25977.objecter _session_command_op_assign 6 2 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 4 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 client.25977.objecter _send_command 2 2024-09-10T19:12:52.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 --> v1:172.21.15.165:6808/1317922936 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4138001190 con 0x7f414407dba0 2024-09-10T19:12:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f41517fa640 1 -- 172.21.15.83:0/2675830669 <== osd.6 v1:172.21.15.165:6808/1317922936 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 4242665900) 0x7f4138001190 con 0x7f414407dba0 2024-09-10T19:12:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f41517fa640 10 client.25977.objecter ms_dispatch 0x7f4154000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f41517fa640 10 client.25977.objecter _finish_command 2 = system:0 2024-09-10T19:12:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f41517fa640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 4 2024-09-10T19:12:52.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f41517fa640 15 client.25977.objecter _session_command_op_remove 6 2 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 librados: watch_flush enter 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 librados: watch_flush exit 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 client.25977.objecter close_session for osd.6 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 >> v1:172.21.15.165:6808/1317922936 conn(0x7f414407dba0 legacy=0x7f4144080030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter put_session s=0x7f414407d4f0 osd=6 1 2024-09-10T19:12:52.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 20 client.25977.objecter shutdown clearing up homeless session... 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 10 client.25977.objecter successfully canceled tick 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 >> v1:172.21.15.83:6800/3071793735 conn(0x7f41440716f0 legacy=0x7f4144073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 >> v1:172.21.15.83:6789/0 conn(0x7f41541014f0 legacy=0x7f41540fac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415905d640 1 -- 172.21.15.83:0/2675830669 reap_dead start 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 shutdown_connections 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 >> 172.21.15.83:0/2675830669 conn(0x7f4154074a20 msgr2=0x7f41540f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 shutdown_connections 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 -- 172.21.15.83:0/2675830669 wait complete. 2024-09-10T19:12:52.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.053+0000 7f415aac0640 1 librados: shutdown 2024-09-10T19:12:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- 172.21.15.83:0/2364000735 >> v1:172.21.15.83:6790/0 conn(0x7f600c103a50 legacy=0x7f600c105e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- 172.21.15.83:0/2364000735 shutdown_connections 2024-09-10T19:12:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- 172.21.15.83:0/2364000735 >> 172.21.15.83:0/2364000735 conn(0x7f600c074a20 msgr2=0x7f600c0f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- 172.21.15.83:0/2364000735 shutdown_connections 2024-09-10T19:12:52.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- 172.21.15.83:0/2364000735 wait complete. 2024-09-10T19:12:52.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 librados: starting msgr at 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 librados: starting objecter 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 Processor -- start 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- start start 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 librados: setting wanted keys 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 librados: calling monclient init 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600c1004b0 con 0x7f600c100c70 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600c1006a0 con 0x7f600c103a50 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f60121ca640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600c11c380 con 0x7f600c1075b0 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f600bfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f600c1075b0 0x7f600c118c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:59574/0 (socket says 172.21.15.83:59574) 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f600bfff640 1 -- 172.21.15.83:0/1085206019 learned_addr learned my addr 172.21.15.83:0/1085206019 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:52.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f600b7fe640 1 --1- 172.21.15.83:0/1085206019 >> v1:172.21.15.83:6789/0 conn(0x7f600c100c70 0x7f600c0fc840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36606/0 (socket says 172.21.15.83:36606) 2024-09-10T19:12:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f600c1075b0 2024-09-10T19:12:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f600c100c70 2024-09-10T19:12:52.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f600c103a50 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335591518 0 0) 0x7f600c11c380 con 0x7f600c1075b0 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff40034e0 con 0x7f600c1075b0 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378481820 0 0) 0x7f600c1004b0 con 0x7f600c100c70 2024-09-10T19:12:52.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.201+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c11c380 con 0x7f600c100c70 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768690152 0 0) 0x7f600c1006a0 con 0x7f600c103a50 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c1004b0 con 0x7f600c103a50 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3640154441 0 0) 0x7f5ff40034e0 con 0x7f600c1075b0 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c1006a0 con 0x7f600c1075b0 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6004002c70 con 0x7f600c1075b0 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501200223 0 0) 0x7f600c11c380 con 0x7f600c100c70 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ff40034e0 con 0x7f600c100c70 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ffc0018c0 con 0x7f600c100c70 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3540813916 0 0) 0x7f600c1006a0 con 0x7f600c1075b0 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 >> v1:172.21.15.165:6789/0 conn(0x7f600c103a50 legacy=0x7f600c0fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 >> v1:172.21.15.83:6789/0 conn(0x7f600c100c70 legacy=0x7f600c0fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c11d560 con 0x7f600c1075b0 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c11c570 con 0x7f600c1075b0 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 10 client.50069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c11ca40 con 0x7f600c1075b0 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 1 librados: init done 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6004001b10 con 0x7f600c1075b0 2024-09-10T19:12:52.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6004005d60 con 0x7f600c1075b0 2024-09-10T19:12:52.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 10 client.50069.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:52.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 15 client.50069.objecter _session_command_op_assign -1 1 2024-09-10T19:12:52.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 20 client.50069.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:52.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 10 client.50069.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ff80028b0 con 0x7f600c1075b0 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f600401c260 con 0x7f600c1075b0 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7f600408a710 con 0x7f600c1075b0 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 10 client.50069.objecter ms_dispatch 0x7f600c000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 3 client.50069.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 3 client.50069.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 10 client.50069.objecter checking command 1 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter _get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 15 client.50069.objecter _session_command_op_remove -1 1 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter _get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 15 client.50069.objecter _session_command_op_assign 3 1 2024-09-10T19:12:52.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 4 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 10 client.50069.objecter _send_command 1 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6813/1279688021 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ff4085b70 con 0x7f5ff4081e70 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 20 client.50069.objecter dump_active .. 0 homeless 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.205+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7f600408ab40 con 0x7f600c1075b0 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 10 client.50069.objecter ms_handle_connect 0x7f5ff4075ac0 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 10 client.50069.objecter ms_handle_connect 0x7f5ff4081e70 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== osd.3 v1:172.21.15.83:6813/1279688021 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f5ff4085b70 con 0x7f5ff4081e70 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 10 client.50069.objecter ms_dispatch 0x7f600c000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 10 client.50069.objecter _finish_command 1 = system:0 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 4 2024-09-10T19:12:52.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.209+0000 7f5ff3fff640 15 client.50069.objecter _session_command_op_remove 3 1 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 client.50069.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 15 client.50069.objecter _session_command_op_assign -1 2 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter _get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter _get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 15 client.50069.objecter _session_command_op_remove -1 2 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter get_session s=0x7f5ff40817c0 osd=3 3 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 15 client.50069.objecter _session_command_op_assign 3 2 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 4 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 client.50069.objecter _send_command 2 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 --> v1:172.21.15.83:6813/1279688021 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5ff8001190 con 0x7f5ff4081e70 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f5ff3fff640 1 -- 172.21.15.83:0/1085206019 <== osd.3 v1:172.21.15.83:6813/1279688021 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1740048643) 0x7f5ff8001190 con 0x7f5ff4081e70 2024-09-10T19:12:52.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f5ff3fff640 10 client.50069.objecter ms_dispatch 0x7f600c000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:52.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f5ff3fff640 10 client.50069.objecter _finish_command 2 = system:0 2024-09-10T19:12:52.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f5ff3fff640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 4 2024-09-10T19:12:52.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f5ff3fff640 15 client.50069.objecter _session_command_op_remove 3 2 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 librados: watch_flush enter 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 librados: watch_flush exit 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 client.50069.objecter close_session for osd.3 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 >> v1:172.21.15.83:6813/1279688021 conn(0x7f5ff4081e70 legacy=0x7f5ff4084300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter put_session s=0x7f5ff40817c0 osd=3 1 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 20 client.50069.objecter shutdown clearing up homeless session... 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 10 client.50069.objecter successfully canceled tick 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5ff4075ac0 legacy=0x7f5ff4077f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 >> v1:172.21.15.83:6790/0 conn(0x7f600c1075b0 legacy=0x7f600c118c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f600bfff640 1 -- 172.21.15.83:0/1085206019 reap_dead start 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 shutdown_connections 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 >> 172.21.15.83:0/1085206019 conn(0x7f600c074a20 msgr2=0x7f600c10a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 shutdown_connections 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 -- 172.21.15.83:0/1085206019 wait complete. 2024-09-10T19:12:52.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.229+0000 7f60121ca640 1 librados: shutdown 2024-09-10T19:12:52.278 INFO:teuthology.orchestra.run.smithi083.stdout:266287972646 2024-09-10T19:12:52.278 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T19:12:52.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- 172.21.15.83:0/2269979693 >> v1:172.21.15.83:6789/0 conn(0x7f61a01042a0 legacy=0x7f61a0106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- 172.21.15.83:0/2269979693 shutdown_connections 2024-09-10T19:12:52.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- 172.21.15.83:0/2269979693 >> 172.21.15.83:0/2269979693 conn(0x7f61a0075290 msgr2=0x7f61a00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- 172.21.15.83:0/2269979693 shutdown_connections 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- 172.21.15.83:0/2269979693 wait complete. 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 librados: starting msgr at 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 librados: starting objecter 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 Processor -- start 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- start start 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 librados: setting wanted keys 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 librados: calling monclient init 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a0100c00 con 0x7f61a01042a0 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a0100db0 con 0x7f61a01014e0 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a7a15640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a01206d0 con 0x7f61a0107e30 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.921+0000 7f61a4fb0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f61a01042a0 0x7f61a0119860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:36612/0 (socket says 172.21.15.83:36612) 2024-09-10T19:12:52.928 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a4fb0640 1 -- 172.21.15.83:0/2182295128 learned_addr learned my addr 172.21.15.83:0/2182295128 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter ms_handle_connect 0x7f61a0107e30 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter ms_handle_connect 0x7f61a01042a0 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter ms_handle_connect 0x7f61a01014e0 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388665603 0 0) 0x7f61a01206d0 con 0x7f61a0107e30 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61880034e0 con 0x7f61a0107e30 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516404403 0 0) 0x7f61a0100c00 con 0x7f61a01042a0 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a01206d0 con 0x7f61a01042a0 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291317396 0 0) 0x7f61a0100db0 con 0x7f61a01014e0 2024-09-10T19:12:52.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0100c00 con 0x7f61a01014e0 2024-09-10T19:12:52.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3169021453 0 0) 0x7f61880034e0 con 0x7f61a0107e30 2024-09-10T19:12:52.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a0100db0 con 0x7f61a0107e30 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f61980018c0 con 0x7f61a0107e30 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948446656 0 0) 0x7f61a01206d0 con 0x7f61a01042a0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61880034e0 con 0x7f61a01042a0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6194001510 con 0x7f61a01042a0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3173931742 0 0) 0x7f61a0100c00 con 0x7f61a01014e0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a01206d0 con 0x7f61a01014e0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6190001980 con 0x7f61a01014e0 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1035503614 0 0) 0x7f61a0100db0 con 0x7f61a0107e30 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 >> v1:172.21.15.165:6789/0 conn(0x7f61a01014e0 legacy=0x7f61a01003d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 >> v1:172.21.15.83:6789/0 conn(0x7f61a01042a0 legacy=0x7f61a0119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a0121870 con 0x7f61a0107e30 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a01208c0 con 0x7f61a0107e30 2024-09-10T19:12:52.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 10 client.50075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a0120d10 con 0x7f61a0107e30 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 1 librados: init done 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f6198001f90 con 0x7f61a0107e30 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f6198003230 con 0x7f61a0107e30 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 10 client.50075.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 15 client.50075.objecter _session_command_op_assign -1 1 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 20 client.50075.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 10 client.50075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.925+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f618c0028b0 con 0x7f61a0107e30 2024-09-10T19:12:52.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f61980196d0 con 0x7f61a0107e30 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7f6198087aa0 con 0x7f61a0107e30 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter ms_dispatch 0x7f61a0000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 3 client.50075.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 3 client.50075.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter checking command 1 2024-09-10T19:12:52.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter _get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 15 client.50075.objecter _session_command_op_remove -1 1 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter _get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 15 client.50075.objecter _session_command_op_assign 2 1 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 4 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter _send_command 1 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6809/386094220 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6188081820 con 0x7f618807db20 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter dump_active .. 0 homeless 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter ms_handle_connect 0x7f61880717f0 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7f6198087e70 con 0x7f61a0107e30 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter ms_handle_connect 0x7f618807db20 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== osd.2 v1:172.21.15.83:6809/386094220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6188081820 con 0x7f618807db20 2024-09-10T19:12:52.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter ms_dispatch 0x7f61a0000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:52.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 10 client.50075.objecter _finish_command 1 = system:0 2024-09-10T19:12:52.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 4 2024-09-10T19:12:52.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.929+0000 7f619dffb640 15 client.50075.objecter _session_command_op_remove 2 1 2024-09-10T19:12:52.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 client.50075.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:52.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 15 client.50075.objecter _session_command_op_assign -1 2 2024-09-10T19:12:52.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter _get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter _get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 15 client.50075.objecter _session_command_op_remove -1 2 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter get_session s=0x7f618807d470 osd=2 3 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 15 client.50075.objecter _session_command_op_assign 2 2 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 4 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 client.50075.objecter _send_command 2 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 --> v1:172.21.15.83:6809/386094220 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f618c001190 con 0x7f618807db20 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f619dffb640 1 -- 172.21.15.83:0/2182295128 <== osd.2 v1:172.21.15.83:6809/386094220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1121936251) 0x7f618c001190 con 0x7f618807db20 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f619dffb640 10 client.50075.objecter ms_dispatch 0x7f61a0000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f619dffb640 10 client.50075.objecter _finish_command 2 = system:0 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f619dffb640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 4 2024-09-10T19:12:52.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f619dffb640 15 client.50075.objecter _session_command_op_remove 2 2 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 librados: watch_flush enter 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 librados: watch_flush exit 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 client.50075.objecter close_session for osd.2 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 >> v1:172.21.15.83:6809/386094220 conn(0x7f618807db20 legacy=0x7f618807ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter put_session s=0x7f618807d470 osd=2 1 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 20 client.50075.objecter shutdown clearing up homeless session... 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 10 client.50075.objecter successfully canceled tick 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 >> v1:172.21.15.83:6800/3071793735 conn(0x7f61880717f0 legacy=0x7f6188073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 >> v1:172.21.15.83:6790/0 conn(0x7f61a0107e30 legacy=0x7f61a011cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:52.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a5fb2640 1 -- 172.21.15.83:0/2182295128 reap_dead start 2024-09-10T19:12:52.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 shutdown_connections 2024-09-10T19:12:52.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 >> 172.21.15.83:0/2182295128 conn(0x7f61a0075290 msgr2=0x7f61a0064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:52.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 shutdown_connections 2024-09-10T19:12:52.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 -- 172.21.15.83:0/2182295128 wait complete. 2024-09-10T19:12:52.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:52.949+0000 7f61a7a15640 1 librados: shutdown 2024-09-10T19:12:52.966 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:12:52] "GET /metrics HTTP/1.1" 200 224097 "" "Prometheus/2.43.0" 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:50.951749+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:50.955122+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.203091+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.206555+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.515077+0000 osd.2 (osd.2) 403 : cluster [DBG] 12.a scrub starts 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.518588+0000 osd.2 (osd.2) 404 : cluster [DBG] 12.a scrub ok 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.753786+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:12:52.967 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[18629]: cluster 2024-09-10T19:12:51.757128+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:50.951749+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:50.955122+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.203091+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.206555+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.515077+0000 osd.2 (osd.2) 403 : cluster [DBG] 12.a scrub starts 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.518588+0000 osd.2 (osd.2) 404 : cluster [DBG] 12.a scrub ok 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.753786+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:12:52.968 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:52 smithi083 bash[32381]: cluster 2024-09-10T19:12:51.757128+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:12:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:50.951749+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.9 scrub starts 2024-09-10T19:12:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:50.955122+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.9 scrub ok 2024-09-10T19:12:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.203091+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.9 scrub starts 2024-09-10T19:12:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.206555+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.9 scrub ok 2024-09-10T19:12:53.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.515077+0000 osd.2 (osd.2) 403 : cluster [DBG] 12.a scrub starts 2024-09-10T19:12:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.518588+0000 osd.2 (osd.2) 404 : cluster [DBG] 12.a scrub ok 2024-09-10T19:12:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.753786+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.5 scrub starts 2024-09-10T19:12:53.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:52 smithi165 bash[22591]: cluster 2024-09-10T19:12:51.757128+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.5 scrub ok 2024-09-10T19:12:53.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.145+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/700387662 >> v1:172.21.15.83:6790/0 conn(0x7fb6b41042a0 legacy=0x7fb6b4106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/700387662 shutdown_connections 2024-09-10T19:12:53.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/700387662 >> 172.21.15.83:0/700387662 conn(0x7fb6b4075290 msgr2=0x7fb6b40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:53.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/700387662 shutdown_connections 2024-09-10T19:12:53.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/700387662 wait complete. 2024-09-10T19:12:53.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 librados: starting msgr at 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 librados: starting objecter 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 Processor -- start 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- start start 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 librados: setting wanted keys 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 librados: calling monclient init 2024-09-10T19:12:53.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6b411c1e0 con 0x7fb6b4107e30 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6b411d3a0 con 0x7fb6b41014e0 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6b411e5a0 con 0x7fb6b41042a0 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b2d9d640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fb6b4107e30 0x7fb6b411a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56262/0 (socket says 172.21.15.83:56262) 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b2d9d640 1 -- 172.21.15.83:0/3212045809 learned_addr learned my addr 172.21.15.83:0/3212045809 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b1d9b640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fb6b41042a0 0x7fb6b40fd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:57686/0 (socket says 172.21.15.83:57686) 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb6b4107e30 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb6b41042a0 2024-09-10T19:12:53.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:53.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627985960 0 0) 0x7fb6b411e5a0 con 0x7fb6b41042a0 2024-09-10T19:12:53.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c0034e0 con 0x7fb6b41042a0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb6b41014e0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454405950 0 0) 0x7fb6b411c1e0 con 0x7fb6b4107e30 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b411e5a0 con 0x7fb6b4107e30 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269774692 0 0) 0x7fb6b411d3a0 con 0x7fb6b41014e0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b411c1e0 con 0x7fb6b41014e0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365293237 0 0) 0x7fb6b411e5a0 con 0x7fb6b4107e30 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b411d3a0 con 0x7fb6b4107e30 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb6ac0019d0 con 0x7fb6b4107e30 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 174703277 0 0) 0x7fb69c0034e0 con 0x7fb6b41042a0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b411e5a0 con 0x7fb6b41042a0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb6a80013a0 con 0x7fb6b41042a0 2024-09-10T19:12:53.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2725278030 0 0) 0x7fb6b411d3a0 con 0x7fb6b4107e30 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 >> v1:172.21.15.83:6790/0 conn(0x7fb6b41042a0 legacy=0x7fb6b40fd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 >> v1:172.21.15.165:6789/0 conn(0x7fb6b41014e0 legacy=0x7fb6b40fd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b411f7a0 con 0x7fb6b4107e30 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b411e790 con 0x7fb6b4107e30 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 10 client.25995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b411ebe0 con 0x7fb6b4107e30 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.149+0000 7fb6b8ffe640 1 librados: init done 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fb6ac002d60 con 0x7fb6b4107e30 2024-09-10T19:12:53.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fb6ac005c40 con 0x7fb6b4107e30 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6b8ffe640 10 client.25995.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6b8ffe640 15 client.25995.objecter _session_command_op_assign -1 1 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6b8ffe640 20 client.25995.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6b8ffe640 10 client.25995.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.83:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6940028b0 con 0x7fb6b4107e30 2024-09-10T19:12:53.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.153+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fb6ac01c100 con 0x7fb6b4107e30 2024-09-10T19:12:53.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(899..899 src has 849..899) v4 ==== 7363+0+0 (unknown 993204670 0 0) 0x7fb6ac08a480 con 0x7fb6b4107e30 2024-09-10T19:12:53.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter ms_dispatch 0x7fb6b4000b90 osd_map(899..899 src has 849..899) v4 2024-09-10T19:12:53.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 3 client.25995.objecter handle_osd_map got epochs [899,899] > 0 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 3 client.25995.objecter handle_osd_map decoding full epoch 899 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter checking command 1 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter _get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 15 client.25995.objecter _session_command_op_remove -1 1 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter _get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 15 client.25995.objecter _session_command_op_assign 4 1 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 4 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter _send_command 1 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.165:6800/3225515265 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb69c0817b0 con 0x7fb69c07dab0 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter dump_active .. 0 homeless 2024-09-10T19:12:53.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_get_version_reply(handle=1 version=899) v2 ==== 24+0+0 (unknown 3747228287 0 0) 0x7fb6ac08a850 con 0x7fb6b4107e30 2024-09-10T19:12:53.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter ms_handle_connect 0x7fb69c071700 2024-09-10T19:12:53.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter ms_handle_connect 0x7fb69c07dab0 2024-09-10T19:12:53.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== osd.4 v1:172.21.15.165:6800/3225515265 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fb69c0817b0 con 0x7fb69c07dab0 2024-09-10T19:12:53.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter ms_dispatch 0x7fb6b4000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:53.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 10 client.25995.objecter _finish_command 1 = system:0 2024-09-10T19:12:53.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 4 2024-09-10T19:12:53.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.157+0000 7fb6a2ffd640 15 client.25995.objecter _session_command_op_remove 4 1 2024-09-10T19:12:53.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 client.25995.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 15 client.25995.objecter _session_command_op_assign -1 2 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter _get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter _get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 15 client.25995.objecter _session_command_op_remove -1 2 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter get_session s=0x7fb69c07d400 osd=4 3 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 15 client.25995.objecter _session_command_op_assign 4 2 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 4 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 client.25995.objecter _send_command 2 2024-09-10T19:12:53.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 --> v1:172.21.15.165:6800/3225515265 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb694001190 con 0x7fb69c07dab0 2024-09-10T19:12:53.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6a2ffd640 1 -- 172.21.15.83:0/3212045809 <== osd.4 v1:172.21.15.165:6800/3225515265 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2726041861) 0x7fb694001190 con 0x7fb69c07dab0 2024-09-10T19:12:53.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6a2ffd640 10 client.25995.objecter ms_dispatch 0x7fb6b4000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:53.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6a2ffd640 10 client.25995.objecter _finish_command 2 = system:0 2024-09-10T19:12:53.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6a2ffd640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 4 2024-09-10T19:12:53.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6a2ffd640 15 client.25995.objecter _session_command_op_remove 4 2 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 librados: watch_flush enter 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 librados: watch_flush exit 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 client.25995.objecter close_session for osd.4 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 >> v1:172.21.15.165:6800/3225515265 conn(0x7fb69c07dab0 legacy=0x7fb69c07ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter put_session s=0x7fb69c07d400 osd=4 1 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 20 client.25995.objecter shutdown clearing up homeless session... 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 10 client.25995.objecter successfully canceled tick 2024-09-10T19:12:53.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 >> v1:172.21.15.83:6800/3071793735 conn(0x7fb69c071700 legacy=0x7fb69c073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 >> v1:172.21.15.83:6789/0 conn(0x7fb6b4107e30 legacy=0x7fb6b411a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b2d9d640 1 -- 172.21.15.83:0/3212045809 reap_dead start 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 shutdown_connections 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 >> 172.21.15.83:0/3212045809 conn(0x7fb6b4075290 msgr2=0x7fb6b40f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 shutdown_connections 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 -- 172.21.15.83:0/3212045809 wait complete. 2024-09-10T19:12:53.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:53.173+0000 7fb6b8ffe640 1 librados: shutdown 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.095952+0000 osd.3 (osd.3) 596 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.099383+0000 osd.3 (osd.3) 597 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.219362+0000 osd.0 (osd.0) 559 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.223714+0000 osd.0 (osd.0) 560 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.442579+0000 osd.6 (osd.6) 533 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.445989+0000 osd.6 (osd.6) 534 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:12:54.037 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[18629]: cluster 2024-09-10T19:12:52.551505+0000 mgr.y (mgr.24526) 909 : cluster [DBG] pgmap v1491: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.095952+0000 osd.3 (osd.3) 596 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.099383+0000 osd.3 (osd.3) 597 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.219362+0000 osd.0 (osd.0) 559 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.223714+0000 osd.0 (osd.0) 560 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.442579+0000 osd.6 (osd.6) 533 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.445989+0000 osd.6 (osd.6) 534 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:12:54.038 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:53 smithi083 bash[32381]: cluster 2024-09-10T19:12:52.551505+0000 mgr.y (mgr.24526) 909 : cluster [DBG] pgmap v1491: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.095952+0000 osd.3 (osd.3) 596 : cluster [DBG] 3.1e scrub starts 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.099383+0000 osd.3 (osd.3) 597 : cluster [DBG] 3.1e scrub ok 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.219362+0000 osd.0 (osd.0) 559 : cluster [DBG] 3.1 scrub starts 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.223714+0000 osd.0 (osd.0) 560 : cluster [DBG] 3.1 scrub ok 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.442579+0000 osd.6 (osd.6) 533 : cluster [DBG] 12.4 scrub starts 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.445989+0000 osd.6 (osd.6) 534 : cluster [DBG] 12.4 scrub ok 2024-09-10T19:12:54.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:53 smithi165 bash[22591]: cluster 2024-09-10T19:12:52.551505+0000 mgr.y (mgr.24526) 909 : cluster [DBG] pgmap v1491: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.041076+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.045399+0000 osd.3 (osd.3) 599 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.115487+0000 osd.0 (osd.0) 561 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.115984+0000 osd.0 (osd.0) 562 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.181705+0000 osd.0 (osd.0) 563 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.186545+0000 osd.0 (osd.0) 564 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.041076+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:12:54.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.045399+0000 osd.3 (osd.3) 599 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:12:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.115487+0000 osd.0 (osd.0) 561 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.115984+0000 osd.0 (osd.0) 562 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.181705+0000 osd.0 (osd.0) 563 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:12:54.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:54 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.186545+0000 osd.0 (osd.0) 564 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:12:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.041076+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.0 scrub starts 2024-09-10T19:12:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.045399+0000 osd.3 (osd.3) 599 : cluster [DBG] 5.0 scrub ok 2024-09-10T19:12:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.115487+0000 osd.0 (osd.0) 561 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:12:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.115984+0000 osd.0 (osd.0) 562 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:12:55.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.181705+0000 osd.0 (osd.0) 563 : cluster [DBG] 3.12 scrub starts 2024-09-10T19:12:55.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:54 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.186545+0000 osd.0 (osd.0) 564 : cluster [DBG] 3.12 scrub ok 2024-09-10T19:12:55.470 INFO:teuthology.orchestra.run.smithi083.stdout:124554051922 2024-09-10T19:12:55.471 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T19:12:55.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.625+0000 7f933f216640 1 -- 172.21.15.83:0/3649945929 >> v1:172.21.15.83:6789/0 conn(0x7f933810b2f0 legacy=0x7f933810d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.625+0000 7f933f216640 1 -- 172.21.15.83:0/3649945929 shutdown_connections 2024-09-10T19:12:55.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.625+0000 7f933f216640 1 -- 172.21.15.83:0/3649945929 >> 172.21.15.83:0/3649945929 conn(0x7f9338074a80 msgr2=0x7f93380f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:55.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.625+0000 7f933f216640 1 -- 172.21.15.83:0/3649945929 shutdown_connections 2024-09-10T19:12:55.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.625+0000 7f933f216640 1 -- 172.21.15.83:0/3649945929 wait complete. 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 librados: starting msgr at 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 librados: starting objecter 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 Processor -- start 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- start start 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 librados: setting wanted keys 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 librados: calling monclient init 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f933811e200 con 0x7f933810b2f0 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f933811f400 con 0x7f93380fd8b0 2024-09-10T19:12:55.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9338120600 con 0x7f93380fd3f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933d7b3640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f933810b2f0 0x7f933811c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56280/0 (socket says 172.21.15.83:56280) 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933d7b3640 1 -- 172.21.15.83:0/2930963708 learned_addr learned my addr 172.21.15.83:0/2930963708 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter ms_handle_connect 0x7f93380fd3f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter ms_handle_connect 0x7f933810b2f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736863639 0 0) 0x7f9338120600 con 0x7f93380fd3f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93200039c0 con 0x7f93380fd3f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter ms_handle_connect 0x7f93380fd8b0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 10 client.?.objecter resend_mon_ops 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216561463 0 0) 0x7f933811e200 con 0x7f933810b2f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9338120600 con 0x7f933810b2f0 2024-09-10T19:12:55.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614624095 0 0) 0x7f933811f400 con 0x7f93380fd8b0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f933811e200 con 0x7f93380fd8b0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433179405 0 0) 0x7f93200039c0 con 0x7f93380fd3f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f933811f400 con 0x7f93380fd3f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9330001b70 con 0x7f93380fd3f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609448779 0 0) 0x7f9338120600 con 0x7f933810b2f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93200039c0 con 0x7f933810b2f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f932c002cb0 con 0x7f933810b2f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1707050724 0 0) 0x7f933811f400 con 0x7f93380fd3f0 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 >> v1:172.21.15.165:6789/0 conn(0x7f93380fd8b0 legacy=0x7f9338107e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 >> v1:172.21.15.83:6789/0 conn(0x7f933810b2f0 legacy=0x7f933811c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338121800 con 0x7f93380fd3f0 2024-09-10T19:12:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933811e3f0 con 0x7f93380fd3f0 2024-09-10T19:12:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 10 client.50087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:55.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f933811e840 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 librados: init done 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f93300026c0 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f93300034d0 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 10 client.50087.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 15 client.50087.objecter _session_command_op_assign -1 1 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 20 client.50087.objecter _get_session osd=-1 returning homeless 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 10 client.50087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.629+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93240028e0 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f9330002260 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(900..900 src has 849..900) v4 ==== 7363+0+0 (unknown 369282167 0 0) 0x7f9330087cb0 con 0x7f93380fd3f0 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter ms_dispatch 0x7f9338000b90 osd_map(900..900 src has 849..900) v4 2024-09-10T19:12:55.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 3 client.50087.objecter handle_osd_map got epochs [900,900] > 0 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 3 client.50087.objecter handle_osd_map decoding full epoch 900 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter checking command 1 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter _get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 15 client.50087.objecter _session_command_op_remove -1 1 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter _get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 15 client.50087.objecter _session_command_op_assign 1 1 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 4 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter _send_command 1 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6805/1048694267 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9320081880 con 0x7f932007db80 2024-09-10T19:12:55.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter dump_active .. 0 homeless 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_get_version_reply(handle=1 version=900) v2 ==== 24+0+0 (unknown 178710119 0 0) 0x7f9330088080 con 0x7f93380fd3f0 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter ms_handle_connect 0x7f9320071750 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter ms_handle_connect 0x7f932007db80 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== osd.1 v1:172.21.15.83:6805/1048694267 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f9320081880 con 0x7f932007db80 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter ms_dispatch 0x7f9338000b90 command_reply(tid 1: 0 ) v1 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 10 client.50087.objecter _finish_command 1 = system:0 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 4 2024-09-10T19:12:55.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.633+0000 7f93357fa640 15 client.50087.objecter _session_command_op_remove 1 1 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 10 client.50087.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 15 client.50087.objecter _session_command_op_assign -1 2 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 20 client.50087.objecter _get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 20 client.50087.objecter _get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 15 client.50087.objecter _session_command_op_remove -1 2 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 20 client.50087.objecter get_session s=0x7f932007d4d0 osd=1 3 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 15 client.50087.objecter _session_command_op_assign 1 2 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 4 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 10 client.50087.objecter _send_command 2 2024-09-10T19:12:55.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.649+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 --> v1:172.21.15.83:6805/1048694267 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f93240011c0 con 0x7f932007db80 2024-09-10T19:12:55.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f93357fa640 1 -- 172.21.15.83:0/2930963708 <== osd.1 v1:172.21.15.83:6805/1048694267 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+13 (unknown 0 0 2511293266) 0x7f93240011c0 con 0x7f932007db80 2024-09-10T19:12:55.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f93357fa640 10 client.50087.objecter ms_dispatch 0x7f9338000b90 command_reply(tid 2: 0 ) v1 2024-09-10T19:12:55.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f93357fa640 10 client.50087.objecter _finish_command 2 = system:0 2024-09-10T19:12:55.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f93357fa640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 4 2024-09-10T19:12:55.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f93357fa640 15 client.50087.objecter _session_command_op_remove 1 2 2024-09-10T19:12:55.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 10 librados: watch_flush enter 2024-09-10T19:12:55.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 10 librados: watch_flush exit 2024-09-10T19:12:55.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 10 client.50087.objecter close_session for osd.1 2024-09-10T19:12:55.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 >> v1:172.21.15.83:6805/1048694267 conn(0x7f932007db80 legacy=0x7f9320080010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 20 client.50087.objecter put_session s=0x7f932007d4d0 osd=1 1 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 20 client.50087.objecter shutdown clearing up homeless session... 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 10 client.50087.objecter successfully canceled tick 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 >> v1:172.21.15.83:6800/3071793735 conn(0x7f9320071750 legacy=0x7f9320073c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 >> v1:172.21.15.83:6790/0 conn(0x7f93380fd3f0 legacy=0x7f9338107760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933d7b3640 1 -- 172.21.15.83:0/2930963708 reap_dead start 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 shutdown_connections 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 >> 172.21.15.83:0/2930963708 conn(0x7f9338074a80 msgr2=0x7f93380fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 shutdown_connections 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 -- 172.21.15.83:0/2930963708 wait complete. 2024-09-10T19:12:55.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:12:55.653+0000 7f933f216640 1 librados: shutdown 2024-09-10T19:12:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.552565+0000 mgr.y (mgr.24526) 910 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-10T19:12:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.626805+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-09-10T19:12:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.029921+0000 osd.3 (osd.3) 600 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:12:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.033536+0000 osd.3 (osd.3) 601 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:12:55.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.162717+0000 osd.0 (osd.0) 565 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.165741+0000 osd.0 (osd.0) 566 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.552565+0000 mgr.y (mgr.24526) 910 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.626805+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.029921+0000 osd.3 (osd.3) 600 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.033536+0000 osd.3 (osd.3) 601 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.162717+0000 osd.0 (osd.0) 565 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:12:55.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:55 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.165741+0000 osd.0 (osd.0) 566 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:12:55.813 INFO:teuthology.orchestra.run.smithi083.stdout:184683594043 2024-09-10T19:12:55.813 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T19:12:55.816 INFO:teuthology.orchestra.run.smithi083.stdout:223338299697 2024-09-10T19:12:55.816 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T19:12:55.827 INFO:teuthology.orchestra.run.smithi083.stdout:150323855687 2024-09-10T19:12:55.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T19:12:55.828 INFO:teuthology.orchestra.run.smithi083.stdout:81604378971 2024-09-10T19:12:55.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.552565+0000 mgr.y (mgr.24526) 910 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.626805+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.029921+0000 osd.3 (osd.3) 600 : cluster [DBG] 4.10 scrub starts 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.033536+0000 osd.3 (osd.3) 601 : cluster [DBG] 4.10 scrub ok 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.162717+0000 osd.0 (osd.0) 565 : cluster [DBG] 4.16 scrub starts 2024-09-10T19:12:56.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:55 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.165741+0000 osd.0 (osd.0) 566 : cluster [DBG] 4.16 scrub ok 2024-09-10T19:12:56.562 INFO:teuthology.orchestra.run.smithi083.stdout:3672197038126 2024-09-10T19:12:56.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.969897+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.a scrub starts 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[18629]: cluster 2024-09-10T19:12:54.973559+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.a scrub ok 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.837134+0000 osd.5 (osd.5) 576 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[18629]: cluster 2024-09-10T19:12:55.843214+0000 osd.5 (osd.5) 577 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.969897+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.a scrub starts 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[32381]: cluster 2024-09-10T19:12:54.973559+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.a scrub ok 2024-09-10T19:12:57.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.837134+0000 osd.5 (osd.5) 576 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T19:12:57.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:56 smithi083 bash[32381]: cluster 2024-09-10T19:12:55.843214+0000 osd.5 (osd.5) 577 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T19:12:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:56 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.969897+0000 osd.1 (osd.1) 127 : cluster [DBG] 2.a scrub starts 2024-09-10T19:12:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:56 smithi165 bash[22591]: cluster 2024-09-10T19:12:54.973559+0000 osd.1 (osd.1) 128 : cluster [DBG] 2.a scrub ok 2024-09-10T19:12:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:56 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.837134+0000 osd.5 (osd.5) 576 : cluster [DBG] 3.1d deep-scrub starts 2024-09-10T19:12:57.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:56 smithi165 bash[22591]: cluster 2024-09-10T19:12:55.843214+0000 osd.5 (osd.5) 577 : cluster [DBG] 3.1d deep-scrub ok 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.394437+0000 osd.4 (osd.4) 540 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.398821+0000 osd.4 (osd.4) 541 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.418075+0000 osd.6 (osd.6) 535 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.421501+0000 osd.6 (osd.6) 536 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.554474+0000 mgr.y (mgr.24526) 911 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.956647+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.b scrub starts 2024-09-10T19:12:58.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[18629]: cluster 2024-09-10T19:12:56.960350+0000 osd.3 (osd.3) 603 : cluster [DBG] 6.b scrub ok 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.394437+0000 osd.4 (osd.4) 540 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.398821+0000 osd.4 (osd.4) 541 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.418075+0000 osd.6 (osd.6) 535 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.421501+0000 osd.6 (osd.6) 536 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.554474+0000 mgr.y (mgr.24526) 911 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.956647+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.b scrub starts 2024-09-10T19:12:58.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:57 smithi083 bash[32381]: cluster 2024-09-10T19:12:56.960350+0000 osd.3 (osd.3) 603 : cluster [DBG] 6.b scrub ok 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.394437+0000 osd.4 (osd.4) 540 : cluster [DBG] 4.d deep-scrub starts 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.398821+0000 osd.4 (osd.4) 541 : cluster [DBG] 4.d deep-scrub ok 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.418075+0000 osd.6 (osd.6) 535 : cluster [DBG] 2.7 scrub starts 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.421501+0000 osd.6 (osd.6) 536 : cluster [DBG] 2.7 scrub ok 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.554474+0000 mgr.y (mgr.24526) 911 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.956647+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.b scrub starts 2024-09-10T19:12:58.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:57 smithi165 bash[22591]: cluster 2024-09-10T19:12:56.960350+0000 osd.3 (osd.3) 603 : cluster [DBG] 6.b scrub ok 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[18629]: cluster 2024-09-10T19:12:57.053400+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.f scrub starts 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[18629]: cluster 2024-09-10T19:12:57.057412+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.f scrub ok 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[18629]: cluster 2024-09-10T19:12:57.378526+0000 osd.4 (osd.4) 542 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[18629]: cluster 2024-09-10T19:12:57.381949+0000 osd.4 (osd.4) 543 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[32381]: cluster 2024-09-10T19:12:57.053400+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.f scrub starts 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[32381]: cluster 2024-09-10T19:12:57.057412+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.f scrub ok 2024-09-10T19:12:59.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[32381]: cluster 2024-09-10T19:12:57.378526+0000 osd.4 (osd.4) 542 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:12:59.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:58 smithi083 bash[32381]: cluster 2024-09-10T19:12:57.381949+0000 osd.4 (osd.4) 543 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:12:59.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:12:58 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:12:59.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:58 smithi165 bash[22591]: cluster 2024-09-10T19:12:57.053400+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.f scrub starts 2024-09-10T19:12:59.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:58 smithi165 bash[22591]: cluster 2024-09-10T19:12:57.057412+0000 osd.1 (osd.1) 130 : cluster [DBG] 4.f scrub ok 2024-09-10T19:12:59.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:58 smithi165 bash[22591]: cluster 2024-09-10T19:12:57.378526+0000 osd.4 (osd.4) 542 : cluster [DBG] 4.13 deep-scrub starts 2024-09-10T19:12:59.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:58 smithi165 bash[22591]: cluster 2024-09-10T19:12:57.381949+0000 osd.4 (osd.4) 543 : cluster [DBG] 4.13 deep-scrub ok 2024-09-10T19:12:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[18629]: cluster 2024-09-10T19:12:58.389858+0000 osd.4 (osd.4) 544 : cluster [DBG] 12.1a deep-scrub starts 2024-09-10T19:12:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[18629]: cluster 2024-09-10T19:12:58.393001+0000 osd.4 (osd.4) 545 : cluster [DBG] 12.1a deep-scrub ok 2024-09-10T19:12:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[18629]: cluster 2024-09-10T19:12:58.555254+0000 mgr.y (mgr.24526) 912 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:12:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[18629]: audit 2024-09-10T19:12:58.708628+0000 mgr.y (mgr.24526) 913 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:59.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[18629]: audit 2024-09-10T19:12:59.583891+0000 mon.c (mon.2) 927 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:59.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[32381]: cluster 2024-09-10T19:12:58.389858+0000 osd.4 (osd.4) 544 : cluster [DBG] 12.1a deep-scrub starts 2024-09-10T19:12:59.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[32381]: cluster 2024-09-10T19:12:58.393001+0000 osd.4 (osd.4) 545 : cluster [DBG] 12.1a deep-scrub ok 2024-09-10T19:12:59.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[32381]: cluster 2024-09-10T19:12:58.555254+0000 mgr.y (mgr.24526) 912 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:12:59.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[32381]: audit 2024-09-10T19:12:58.708628+0000 mgr.y (mgr.24526) 913 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:59.793 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:12:59 smithi083 bash[32381]: audit 2024-09-10T19:12:59.583891+0000 mon.c (mon.2) 927 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:59 smithi165 bash[22591]: cluster 2024-09-10T19:12:58.389858+0000 osd.4 (osd.4) 544 : cluster [DBG] 12.1a deep-scrub starts 2024-09-10T19:13:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:59 smithi165 bash[22591]: cluster 2024-09-10T19:12:58.393001+0000 osd.4 (osd.4) 545 : cluster [DBG] 12.1a deep-scrub ok 2024-09-10T19:13:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:59 smithi165 bash[22591]: cluster 2024-09-10T19:12:58.555254+0000 mgr.y (mgr.24526) 912 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:13:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:59 smithi165 bash[22591]: audit 2024-09-10T19:12:58.708628+0000 mgr.y (mgr.24526) 913 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:00.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:12:59 smithi165 bash[22591]: audit 2024-09-10T19:12:59.583891+0000 mon.c (mon.2) 927 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:00.374 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:00.375 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:00.375 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:00.376 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[18629]: cluster 2024-09-10T19:12:59.389241+0000 osd.4 (osd.4) 546 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[18629]: cluster 2024-09-10T19:12:59.394105+0000 osd.4 (osd.4) 547 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.160708+0000 osd.0 (osd.0) 567 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.163898+0000 osd.0 (osd.0) 568 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[32381]: cluster 2024-09-10T19:12:59.389241+0000 osd.4 (osd.4) 546 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[32381]: cluster 2024-09-10T19:12:59.394105+0000 osd.4 (osd.4) 547 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:13:01.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.160708+0000 osd.0 (osd.0) 567 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:13:01.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:00 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.163898+0000 osd.0 (osd.0) 568 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:13:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:00 smithi165 bash[22591]: cluster 2024-09-10T19:12:59.389241+0000 osd.4 (osd.4) 546 : cluster [DBG] 3.14 scrub starts 2024-09-10T19:13:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:00 smithi165 bash[22591]: cluster 2024-09-10T19:12:59.394105+0000 osd.4 (osd.4) 547 : cluster [DBG] 3.14 scrub ok 2024-09-10T19:13:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:00 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.160708+0000 osd.0 (osd.0) 567 : cluster [DBG] 6.16 deep-scrub starts 2024-09-10T19:13:01.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:00 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.163898+0000 osd.0 (osd.0) 568 : cluster [DBG] 6.16 deep-scrub ok 2024-09-10T19:13:01.659 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.055655+0000 osd.1 (osd.1) 131 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.059239+0000 osd.1 (osd.1) 132 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.555287+0000 osd.7 (osd.7) 574 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.556004+0000 mgr.y (mgr.24526) 914 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:00.558790+0000 osd.7 (osd.7) 575 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:01.162728+0000 osd.0 (osd.0) 569 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:13:02.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[18629]: cluster 2024-09-10T19:13:01.166534+0000 osd.0 (osd.0) 570 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.055655+0000 osd.1 (osd.1) 131 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.059239+0000 osd.1 (osd.1) 132 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.555287+0000 osd.7 (osd.7) 574 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.556004+0000 mgr.y (mgr.24526) 914 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:00.558790+0000 osd.7 (osd.7) 575 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:01.162728+0000 osd.0 (osd.0) 569 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:13:02.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:01 smithi083 bash[32381]: cluster 2024-09-10T19:13:01.166534+0000 osd.0 (osd.0) 570 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:13:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.055655+0000 osd.1 (osd.1) 131 : cluster [DBG] 2.15 scrub starts 2024-09-10T19:13:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.059239+0000 osd.1 (osd.1) 132 : cluster [DBG] 2.15 scrub ok 2024-09-10T19:13:02.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.555287+0000 osd.7 (osd.7) 574 : cluster [DBG] 1.2 scrub starts 2024-09-10T19:13:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.556004+0000 mgr.y (mgr.24526) 914 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:00.558790+0000 osd.7 (osd.7) 575 : cluster [DBG] 1.2 scrub ok 2024-09-10T19:13:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:01.162728+0000 osd.0 (osd.0) 569 : cluster [DBG] 12.3 deep-scrub starts 2024-09-10T19:13:02.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:01 smithi165 bash[22591]: cluster 2024-09-10T19:13:01.166534+0000 osd.0 (osd.0) 570 : cluster [DBG] 12.3 deep-scrub ok 2024-09-10T19:13:02.978 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:02 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:02] "GET /metrics HTTP/1.1" 200 224151 "" "Prometheus/2.43.0" 2024-09-10T19:13:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[18629]: cluster 2024-09-10T19:13:02.557892+0000 mgr.y (mgr.24526) 915 : cluster [DBG] pgmap v1497: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[18629]: cluster 2024-09-10T19:13:03.157279+0000 osd.0 (osd.0) 571 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:13:04.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[18629]: cluster 2024-09-10T19:13:03.161029+0000 osd.0 (osd.0) 572 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:13:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[32381]: cluster 2024-09-10T19:13:02.557892+0000 mgr.y (mgr.24526) 915 : cluster [DBG] pgmap v1497: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:04.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[32381]: cluster 2024-09-10T19:13:03.157279+0000 osd.0 (osd.0) 571 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:13:04.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:03 smithi083 bash[32381]: cluster 2024-09-10T19:13:03.161029+0000 osd.0 (osd.0) 572 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:13:04.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:03 smithi165 bash[22591]: cluster 2024-09-10T19:13:02.557892+0000 mgr.y (mgr.24526) 915 : cluster [DBG] pgmap v1497: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:04.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:03 smithi165 bash[22591]: cluster 2024-09-10T19:13:03.157279+0000 osd.0 (osd.0) 571 : cluster [DBG] 6.10 scrub starts 2024-09-10T19:13:04.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:03 smithi165 bash[22591]: cluster 2024-09-10T19:13:03.161029+0000 osd.0 (osd.0) 572 : cluster [DBG] 6.10 scrub ok 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[18629]: cluster 2024-09-10T19:13:03.519521+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[18629]: cluster 2024-09-10T19:13:03.523905+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.155844+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.161163+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[32381]: cluster 2024-09-10T19:13:03.519521+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[32381]: cluster 2024-09-10T19:13:03.523905+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:13:05.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.155844+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:13:05.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:04 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.161163+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:13:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:04 smithi165 bash[22591]: cluster 2024-09-10T19:13:03.519521+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-09-10T19:13:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:04 smithi165 bash[22591]: cluster 2024-09-10T19:13:03.523905+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-09-10T19:13:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:04 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.155844+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.6 scrub starts 2024-09-10T19:13:05.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:04 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.161163+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.6 scrub ok 2024-09-10T19:13:05.924 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.067092+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:13:05.924 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.072796+0000 osd.1 (osd.1) 134 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:13:05.924 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.356352+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:13:05.924 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.360206+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:13:05.924 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.490618+0000 osd.2 (osd.2) 405 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.492967+0000 osd.2 (osd.2) 406 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:04.558548+0000 mgr.y (mgr.24526) 916 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.122055+0000 osd.0 (osd.0) 575 : cluster [DBG] 3.1b deep-scrub starts 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.129433+0000 osd.0 (osd.0) 576 : cluster [DBG] 3.1b deep-scrub ok 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.067092+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.072796+0000 osd.1 (osd.1) 134 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.356352+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.360206+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.490618+0000 osd.2 (osd.2) 405 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:13:05.925 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.492967+0000 osd.2 (osd.2) 406 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:13:05.926 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:04.558548+0000 mgr.y (mgr.24526) 916 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:05.926 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.122055+0000 osd.0 (osd.0) 575 : cluster [DBG] 3.1b deep-scrub starts 2024-09-10T19:13:05.926 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:05 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.129433+0000 osd.0 (osd.0) 576 : cluster [DBG] 3.1b deep-scrub ok 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.067092+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.19 scrub starts 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.072796+0000 osd.1 (osd.1) 134 : cluster [DBG] 3.19 scrub ok 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.356352+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1e scrub starts 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.360206+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.1e scrub ok 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.490618+0000 osd.2 (osd.2) 405 : cluster [DBG] 12.9 scrub starts 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.492967+0000 osd.2 (osd.2) 406 : cluster [DBG] 12.9 scrub ok 2024-09-10T19:13:06.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:04.558548+0000 mgr.y (mgr.24526) 916 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-10T19:13:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.122055+0000 osd.0 (osd.0) 575 : cluster [DBG] 3.1b deep-scrub starts 2024-09-10T19:13:06.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:05 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.129433+0000 osd.0 (osd.0) 576 : cluster [DBG] 3.1b deep-scrub ok 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.043062+0000 osd.1 (osd.1) 135 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.046607+0000 osd.1 (osd.1) 136 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.346307+0000 osd.4 (osd.4) 550 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.350798+0000 osd.4 (osd.4) 551 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.800051+0000 osd.5 (osd.5) 578 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:13:06.793 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.803909+0000 osd.5 (osd.5) 579 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.944174+0000 osd.3 (osd.3) 604 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[18629]: cluster 2024-09-10T19:13:05.947595+0000 osd.3 (osd.3) 605 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.043062+0000 osd.1 (osd.1) 135 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.046607+0000 osd.1 (osd.1) 136 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.346307+0000 osd.4 (osd.4) 550 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.350798+0000 osd.4 (osd.4) 551 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.800051+0000 osd.5 (osd.5) 578 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.803909+0000 osd.5 (osd.5) 579 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.944174+0000 osd.3 (osd.3) 604 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:13:06.794 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:06 smithi083 bash[32381]: cluster 2024-09-10T19:13:05.947595+0000 osd.3 (osd.3) 605 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.043062+0000 osd.1 (osd.1) 135 : cluster [DBG] 12.18 scrub starts 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.046607+0000 osd.1 (osd.1) 136 : cluster [DBG] 12.18 scrub ok 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.346307+0000 osd.4 (osd.4) 550 : cluster [DBG] 4.e deep-scrub starts 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.350798+0000 osd.4 (osd.4) 551 : cluster [DBG] 4.e deep-scrub ok 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.800051+0000 osd.5 (osd.5) 578 : cluster [DBG] 12.11 scrub starts 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.803909+0000 osd.5 (osd.5) 579 : cluster [DBG] 12.11 scrub ok 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.944174+0000 osd.3 (osd.3) 604 : cluster [DBG] 4.17 scrub starts 2024-09-10T19:13:07.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:06 smithi165 bash[22591]: cluster 2024-09-10T19:13:05.947595+0000 osd.3 (osd.3) 605 : cluster [DBG] 4.17 scrub ok 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.082784+0000 osd.1 (osd.1) 137 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.089630+0000 osd.1 (osd.1) 138 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.422802+0000 osd.6 (osd.6) 537 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.426836+0000 osd.6 (osd.6) 538 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.560437+0000 mgr.y (mgr.24526) 917 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:13:08.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.562925+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:06.567361+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:07.103886+0000 osd.0 (osd.0) 577 : cluster [DBG] 12.e scrub starts 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[18629]: cluster 2024-09-10T19:13:07.107985+0000 osd.0 (osd.0) 578 : cluster [DBG] 12.e scrub ok 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.082784+0000 osd.1 (osd.1) 137 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.089630+0000 osd.1 (osd.1) 138 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.422802+0000 osd.6 (osd.6) 537 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.426836+0000 osd.6 (osd.6) 538 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.560437+0000 mgr.y (mgr.24526) 917 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.562925+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:06.567361+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:13:08.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:07.103886+0000 osd.0 (osd.0) 577 : cluster [DBG] 12.e scrub starts 2024-09-10T19:13:08.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:07 smithi083 bash[32381]: cluster 2024-09-10T19:13:07.107985+0000 osd.0 (osd.0) 578 : cluster [DBG] 12.e scrub ok 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.082784+0000 osd.1 (osd.1) 137 : cluster [DBG] 3.0 scrub starts 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.089630+0000 osd.1 (osd.1) 138 : cluster [DBG] 3.0 scrub ok 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.422802+0000 osd.6 (osd.6) 537 : cluster [DBG] 1.6 scrub starts 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.426836+0000 osd.6 (osd.6) 538 : cluster [DBG] 1.6 scrub ok 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.560437+0000 mgr.y (mgr.24526) 917 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:13:08.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.562925+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.1c scrub starts 2024-09-10T19:13:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:06.567361+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.1c scrub ok 2024-09-10T19:13:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:07.103886+0000 osd.0 (osd.0) 577 : cluster [DBG] 12.e scrub starts 2024-09-10T19:13:08.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:07 smithi165 bash[22591]: cluster 2024-09-10T19:13:07.107985+0000 osd.0 (osd.0) 578 : cluster [DBG] 12.e scrub ok 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[32381]: cluster 2024-09-10T19:13:07.513009+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[32381]: cluster 2024-09-10T19:13:07.513490+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[32381]: cluster 2024-09-10T19:13:07.682220+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[18629]: cluster 2024-09-10T19:13:07.513009+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[18629]: cluster 2024-09-10T19:13:07.513490+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:09.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:08 smithi083 bash[18629]: cluster 2024-09-10T19:13:07.682220+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-09-10T19:13:09.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:13:08 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:13:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:08 smithi165 bash[22591]: cluster 2024-09-10T19:13:07.513009+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:08 smithi165 bash[22591]: cluster 2024-09-10T19:13:07.513490+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:09.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:08 smithi165 bash[22591]: cluster 2024-09-10T19:13:07.682220+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: cluster 2024-09-10T19:13:08.332205+0000 osd.4 (osd.4) 552 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: cluster 2024-09-10T19:13:08.335304+0000 osd.4 (osd.4) 553 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: cluster 2024-09-10T19:13:08.561419+0000 mgr.y (mgr.24526) 918 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: audit 2024-09-10T19:13:08.716459+0000 mgr.y (mgr.24526) 919 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: cluster 2024-09-10T19:13:09.046123+0000 osd.0 (osd.0) 579 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[18629]: cluster 2024-09-10T19:13:09.048250+0000 osd.0 (osd.0) 580 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:13:09.938 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: cluster 2024-09-10T19:13:08.332205+0000 osd.4 (osd.4) 552 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:13:09.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: cluster 2024-09-10T19:13:08.335304+0000 osd.4 (osd.4) 553 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:13:09.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: cluster 2024-09-10T19:13:08.561419+0000 mgr.y (mgr.24526) 918 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:09.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: audit 2024-09-10T19:13:08.716459+0000 mgr.y (mgr.24526) 919 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:09.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: cluster 2024-09-10T19:13:09.046123+0000 osd.0 (osd.0) 579 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:13:09.939 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:09 smithi083 bash[32381]: cluster 2024-09-10T19:13:09.048250+0000 osd.0 (osd.0) 580 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: cluster 2024-09-10T19:13:08.332205+0000 osd.4 (osd.4) 552 : cluster [DBG] 4.1 scrub starts 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: cluster 2024-09-10T19:13:08.335304+0000 osd.4 (osd.4) 553 : cluster [DBG] 4.1 scrub ok 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: cluster 2024-09-10T19:13:08.561419+0000 mgr.y (mgr.24526) 918 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: audit 2024-09-10T19:13:08.716459+0000 mgr.y (mgr.24526) 919 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: cluster 2024-09-10T19:13:09.046123+0000 osd.0 (osd.0) 579 : cluster [DBG] 12.1c scrub starts 2024-09-10T19:13:10.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:09 smithi165 bash[22591]: cluster 2024-09-10T19:13:09.048250+0000 osd.0 (osd.0) 580 : cluster [DBG] 12.1c scrub ok 2024-09-10T19:13:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:10 smithi083 bash[18629]: cluster 2024-09-10T19:13:09.075031+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:13:11.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:10 smithi083 bash[18629]: cluster 2024-09-10T19:13:09.078036+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:13:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:10 smithi083 bash[32381]: cluster 2024-09-10T19:13:09.075031+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:13:11.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:10 smithi083 bash[32381]: cluster 2024-09-10T19:13:09.078036+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:13:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:10 smithi165 bash[22591]: cluster 2024-09-10T19:13:09.075031+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.4 scrub starts 2024-09-10T19:13:11.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:10 smithi165 bash[22591]: cluster 2024-09-10T19:13:09.078036+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.4 scrub ok 2024-09-10T19:13:11.710 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:11.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[22591]: cluster 2024-09-10T19:13:10.327764+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:13:11.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[22591]: cluster 2024-09-10T19:13:10.333463+0000 osd.4 (osd.4) 555 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:13:11.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[22591]: cluster 2024-09-10T19:13:10.562228+0000 mgr.y (mgr.24526) 920 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:11.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[22591]: cluster 2024-09-10T19:13:10.914625+0000 osd.3 (osd.3) 606 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T19:13:11.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:11 smithi165 bash[22591]: cluster 2024-09-10T19:13:10.917712+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T19:13:11.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[18629]: cluster 2024-09-10T19:13:10.327764+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:13:11.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[18629]: cluster 2024-09-10T19:13:10.333463+0000 osd.4 (osd.4) 555 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:13:11.976 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[18629]: cluster 2024-09-10T19:13:10.562228+0000 mgr.y (mgr.24526) 920 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[18629]: cluster 2024-09-10T19:13:10.914625+0000 osd.3 (osd.3) 606 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[18629]: cluster 2024-09-10T19:13:10.917712+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[32381]: cluster 2024-09-10T19:13:10.327764+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub starts 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[32381]: cluster 2024-09-10T19:13:10.333463+0000 osd.4 (osd.4) 555 : cluster [DBG] 3.3 scrub ok 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[32381]: cluster 2024-09-10T19:13:10.562228+0000 mgr.y (mgr.24526) 920 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[32381]: cluster 2024-09-10T19:13:10.914625+0000 osd.3 (osd.3) 606 : cluster [DBG] 2.1b deep-scrub starts 2024-09-10T19:13:11.977 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:11 smithi083 bash[32381]: cluster 2024-09-10T19:13:10.917712+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b deep-scrub ok 2024-09-10T19:13:13.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:12 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:12] "GET /metrics HTTP/1.1" 200 224151 "" "Prometheus/2.43.0" 2024-09-10T19:13:13.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 -- 172.21.15.83:0/1687593881 >> v1:172.21.15.83:6789/0 conn(0x7f99440fd5c0 legacy=0x7f99440fd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:13.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 -- 172.21.15.83:0/1687593881 shutdown_connections 2024-09-10T19:13:13.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 -- 172.21.15.83:0/1687593881 >> 172.21.15.83:0/1687593881 conn(0x7f99440f70d0 msgr2=0x7f99440f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:13.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 -- 172.21.15.83:0/1687593881 shutdown_connections 2024-09-10T19:13:13.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 -- 172.21.15.83:0/1687593881 wait complete. 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 librados: starting msgr at 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.276+0000 7f994b444640 1 librados: starting objecter 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 Processor -- start 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- start start 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 librados: setting wanted keys 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 librados: calling monclient init 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f994411a3c0 con 0x7f9944116630 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f994411b5c0 con 0x7f99440fd5c0 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f994411c7c0 con 0x7f9944108ef0 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f99489df640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f9944108ef0 0x7f9944102830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49484/0 (socket says 172.21.15.83:49484) 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f99489df640 1 -- 172.21.15.83:0/1880579388 learned_addr learned my addr 172.21.15.83:0/1880579388 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter ms_handle_connect 0x7f9944108ef0 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:13.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter ms_handle_connect 0x7f99440fd5c0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter ms_handle_connect 0x7f9944116630 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727755977 0 0) 0x7f994411c7c0 con 0x7f9944108ef0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99300034e0 con 0x7f9944108ef0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574587073 0 0) 0x7f994411a3c0 con 0x7f9944116630 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994411c7c0 con 0x7f9944116630 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792269314 0 0) 0x7f994411b5c0 con 0x7f99440fd5c0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994411a3c0 con 0x7f99440fd5c0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2497249926 0 0) 0x7f99300034e0 con 0x7f9944108ef0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f994411b5c0 con 0x7f9944108ef0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f99340029d0 con 0x7f9944108ef0 2024-09-10T19:13:13.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1148855268 0 0) 0x7f994411b5c0 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 >> v1:172.21.15.165:6789/0 conn(0x7f99440fd5c0 legacy=0x7f9944102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 >> v1:172.21.15.83:6789/0 conn(0x7f9944116630 legacy=0x7f9944118ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994411d9c0 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994411b7b0 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 10 client.50093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f994411bc00 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 librados: init done 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f9934002d60 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f9934005bd0 con 0x7f9944108ef0 2024-09-10T19:13:13.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.280+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9928005010 con 0x7f9944108ef0 2024-09-10T19:13:13.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f99340016e0 con 0x7f9944108ef0 2024-09-10T19:13:13.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f993408a3c0 con 0x7f9944108ef0 2024-09-10T19:13:13.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 10 client.50093.objecter ms_dispatch 0x7f9944000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:13.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 3 client.50093.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:13.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 3 client.50093.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:13.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 20 client.50093.objecter dump_active .. 0 homeless 2024-09-10T19:13:13.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.284+0000 7f9939ffb640 10 client.50093.objecter ms_handle_connect 0x7f99300755e0 2024-09-10T19:13:13.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.288+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f9934056c00 con 0x7f9944108ef0 2024-09-10T19:13:13.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.464+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f99280052c0 con 0x7f9944108ef0 2024-09-10T19:13:13.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.464+0000 7f9939ffb640 1 -- 172.21.15.83:0/1880579388 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 4011120053) 0x7f99340565a0 con 0x7f9944108ef0 2024-09-10T19:13:13.471 INFO:teuthology.orchestra.run.smithi083.stdout:266287972651 2024-09-10T19:13:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 10 librados: watch_flush enter 2024-09-10T19:13:13.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 10 librados: watch_flush exit 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 20 client.50093.objecter shutdown clearing up homeless session... 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 10 client.50093.objecter successfully canceled tick 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 >> v1:172.21.15.83:6800/3071793735 conn(0x7f99300755e0 legacy=0x7f9930077a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 >> v1:172.21.15.83:6790/0 conn(0x7f9944108ef0 legacy=0x7f9944102830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 shutdown_connections 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 >> 172.21.15.83:0/1880579388 conn(0x7f99440f70d0 msgr2=0x7f9944109390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:13.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 shutdown_connections 2024-09-10T19:13:13.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 -- 172.21.15.83:0/1880579388 wait complete. 2024-09-10T19:13:13.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:13.468+0000 7f994b444640 1 librados: shutdown 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:12.351807+0000 osd.4 (osd.4) 556 : cluster [DBG] 4.c scrub starts 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:12.355520+0000 osd.4 (osd.4) 557 : cluster [DBG] 4.c scrub ok 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:12.482982+0000 osd.2 (osd.2) 409 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:12.487056+0000 osd.2 (osd.2) 410 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:12.564098+0000 mgr.y (mgr.24526) 921 : cluster [DBG] pgmap v1503: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.021938+0000 osd.0 (osd.0) 581 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.025517+0000 osd.0 (osd.0) 582 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:13:13.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[32381]: audit 2024-09-10T19:13:13.469835+0000 mon.c (mon.2) 928 : audit [DBG] from='client.? v1:172.21.15.83:0/1880579388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:12.351807+0000 osd.4 (osd.4) 556 : cluster [DBG] 4.c scrub starts 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:12.355520+0000 osd.4 (osd.4) 557 : cluster [DBG] 4.c scrub ok 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:12.482982+0000 osd.2 (osd.2) 409 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:12.487056+0000 osd.2 (osd.2) 410 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:12.564098+0000 mgr.y (mgr.24526) 921 : cluster [DBG] pgmap v1503: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.021938+0000 osd.0 (osd.0) 581 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.025517+0000 osd.0 (osd.0) 582 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:13:13.792 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:13 smithi083 bash[18629]: audit 2024-09-10T19:13:13.469835+0000 mon.c (mon.2) 928 : audit [DBG] from='client.? v1:172.21.15.83:0/1880579388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:13:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:12.351807+0000 osd.4 (osd.4) 556 : cluster [DBG] 4.c scrub starts 2024-09-10T19:13:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:12.355520+0000 osd.4 (osd.4) 557 : cluster [DBG] 4.c scrub ok 2024-09-10T19:13:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:12.482982+0000 osd.2 (osd.2) 409 : cluster [DBG] 2.1 scrub starts 2024-09-10T19:13:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:12.487056+0000 osd.2 (osd.2) 410 : cluster [DBG] 2.1 scrub ok 2024-09-10T19:13:14.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:12.564098+0000 mgr.y (mgr.24526) 921 : cluster [DBG] pgmap v1503: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.021938+0000 osd.0 (osd.0) 581 : cluster [DBG] 12.c deep-scrub starts 2024-09-10T19:13:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.025517+0000 osd.0 (osd.0) 582 : cluster [DBG] 12.c deep-scrub ok 2024-09-10T19:13:14.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:13 smithi165 bash[22591]: audit 2024-09-10T19:13:13.469835+0000 mon.c (mon.2) 928 : audit [DBG] from='client.? v1:172.21.15.83:0/1880579388' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:13:14.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/773630781 >> v1:172.21.15.83:6790/0 conn(0x7f5ef01042a0 legacy=0x7f5ef0106680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:14.632 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/773630781 shutdown_connections 2024-09-10T19:13:14.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/773630781 >> 172.21.15.83:0/773630781 conn(0x7f5ef0075290 msgr2=0x7f5ef00f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:14.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/773630781 shutdown_connections 2024-09-10T19:13:14.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/773630781 wait complete. 2024-09-10T19:13:14.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 librados: starting msgr at 2024-09-10T19:13:14.633 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 librados: starting objecter 2024-09-10T19:13:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 Processor -- start 2024-09-10T19:13:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- start start 2024-09-10T19:13:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 librados: setting wanted keys 2024-09-10T19:13:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 librados: calling monclient init 2024-09-10T19:13:14.634 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ef011bff0 con 0x7f5ef01014e0 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ef011d1b0 con 0x7f5ef0107e30 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5ef5b6f640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ef011e3b0 con 0x7f5ef01042a0 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5eeeffd640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f5ef01042a0 0x7f5ef00fecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49496/0 (socket says 172.21.15.83:49496) 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.628+0000 7f5eeeffd640 1 -- 172.21.15.83:0/4206975613 learned_addr learned my addr 172.21.15.83:0/4206975613 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter ms_handle_connect 0x7f5ef01042a0 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter ms_handle_connect 0x7f5ef01014e0 2024-09-10T19:13:14.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter ms_handle_connect 0x7f5ef0107e30 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831755054 0 0) 0x7f5ef011e3b0 con 0x7f5ef01042a0 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed80034e0 con 0x7f5ef01042a0 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799535102 0 0) 0x7f5ef011bff0 con 0x7f5ef01014e0 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef011e3b0 con 0x7f5ef01014e0 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813648247 0 0) 0x7f5ef011d1b0 con 0x7f5ef0107e30 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef011bff0 con 0x7f5ef0107e30 2024-09-10T19:13:14.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774585708 0 0) 0x7f5ef011e3b0 con 0x7f5ef01014e0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ef011d1b0 con 0x7f5ef01014e0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1254555904 0 0) 0x7f5ed80034e0 con 0x7f5ef01042a0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ef011e3b0 con 0x7f5ef01042a0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ee0001980 con 0x7f5ef01014e0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ee4002030 con 0x7f5ef01042a0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2510925631 0 0) 0x7f5ef011d1b0 con 0x7f5ef01014e0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 >> v1:172.21.15.83:6790/0 conn(0x7f5ef01042a0 legacy=0x7f5ef00fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 >> v1:172.21.15.165:6789/0 conn(0x7f5ef0107e30 legacy=0x7f5ef011a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef011f5b0 con 0x7f5ef01014e0 2024-09-10T19:13:14.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef011c1e0 con 0x7f5ef01014e0 2024-09-10T19:13:14.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ef5b6f640 10 client.26013.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:14.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef011c630 con 0x7f5ef01014e0 2024-09-10T19:13:14.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ef5b6f640 1 librados: init done 2024-09-10T19:13:14.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f5ee00021f0 con 0x7f5ef01014e0 2024-09-10T19:13:14.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.632+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f5ee0003100 con 0x7f5ef01014e0 2024-09-10T19:13:14.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f5ee0003340 con 0x7f5ef01014e0 2024-09-10T19:13:14.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc005010 con 0x7f5ef01014e0 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f5ee0087920 con 0x7f5ef01014e0 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 10 client.26013.objecter ms_dispatch 0x7f5ef0000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 3 client.26013.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 3 client.26013.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 20 client.26013.objecter dump_active .. 0 homeless 2024-09-10T19:13:14.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.636+0000 7f5ed7fff640 10 client.26013.objecter ms_handle_connect 0x7f5ed8075aa0 2024-09-10T19:13:14.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.640+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f5ee0053ce0 con 0x7f5ef01014e0 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.775807+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.779273+0000 osd.5 (osd.5) 581 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.907122+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[18629]: cluster 2024-09-10T19:13:13.910658+0000 osd.3 (osd.3) 609 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[18629]: audit 2024-09-10T19:13:14.583987+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.775807+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.779273+0000 osd.5 (osd.5) 581 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:13:14.791 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.907122+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:13:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[32381]: cluster 2024-09-10T19:13:13.910658+0000 osd.3 (osd.3) 609 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:13:14.792 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:14 smithi083 bash[32381]: audit 2024-09-10T19:13:14.583987+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.820+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5edc0052c0 con 0x7f5ef01014e0 2024-09-10T19:13:14.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.820+0000 7f5ed7fff640 1 -- 172.21.15.83:0/4206975613 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 2453635953) 0x7f5ee0053b00 con 0x7f5ef01014e0 2024-09-10T19:13:14.824 INFO:teuthology.orchestra.run.smithi083.stdout:150323855691 2024-09-10T19:13:14.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 10 librados: watch_flush enter 2024-09-10T19:13:14.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 10 librados: watch_flush exit 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 20 client.26013.objecter shutdown clearing up homeless session... 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 10 client.26013.objecter successfully canceled tick 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 >> v1:172.21.15.83:6800/3071793735 conn(0x7f5ed8075aa0 legacy=0x7f5ed8077f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 >> v1:172.21.15.83:6789/0 conn(0x7f5ef01014e0 legacy=0x7f5ef00fe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 shutdown_connections 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 >> 172.21.15.83:0/4206975613 conn(0x7f5ef0075290 msgr2=0x7f5ef0064ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:14.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 shutdown_connections 2024-09-10T19:13:14.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 -- 172.21.15.83:0/4206975613 wait complete. 2024-09-10T19:13:14.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:14.824+0000 7f5ef5b6f640 1 librados: shutdown 2024-09-10T19:13:15.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/1963704470 >> v1:172.21.15.83:6789/0 conn(0x7f2910108fb0 legacy=0x7f291010b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:15.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/1963704470 shutdown_connections 2024-09-10T19:13:15.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/1963704470 >> 172.21.15.83:0/1963704470 conn(0x7f2910074a20 msgr2=0x7f29100f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:15.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/1963704470 shutdown_connections 2024-09-10T19:13:15.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/1963704470 wait complete. 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 librados: starting msgr at 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 librados: starting objecter 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 Processor -- start 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- start start 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 librados: setting wanted keys 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 librados: calling monclient init 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2910106610 con 0x7f2910101590 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f291011d4b0 con 0x7f2910108fb0 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f291011e650 con 0x7f2910105450 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2910101590 0x7f29101059c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41720/0 (socket says 172.21.15.83:41720) 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290ffff640 1 -- 172.21.15.83:0/251441584 learned_addr learned my addr 172.21.15.83:0/251441584 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290f7fe640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f2910105450 0x7f2910117650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49512/0 (socket says 172.21.15.83:49512) 2024-09-10T19:13:15.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter ms_handle_connect 0x7f2910105450 2024-09-10T19:13:15.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:15.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter ms_handle_connect 0x7f2910101590 2024-09-10T19:13:15.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041282530 0 0) 0x7f291011e650 con 0x7f2910105450 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8003720 con 0x7f2910105450 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter ms_handle_connect 0x7f2910108fb0 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230301837 0 0) 0x7f2910106610 con 0x7f2910101590 2024-09-10T19:13:15.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f291011e650 con 0x7f2910101590 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967479381 0 0) 0x7f291011d4b0 con 0x7f2910108fb0 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910106610 con 0x7f2910108fb0 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320911368 0 0) 0x7f291011e650 con 0x7f2910101590 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f291011d4b0 con 0x7f2910101590 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2021997228 0 0) 0x7f28f8003720 con 0x7f2910105450 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f291011e650 con 0x7f2910105450 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f29000018c0 con 0x7f2910101590 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f2904001860 con 0x7f2910105450 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 918439637 0 0) 0x7f291011d4b0 con 0x7f2910101590 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 >> v1:172.21.15.83:6790/0 conn(0x7f2910105450 legacy=0x7f2910117650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 >> v1:172.21.15.165:6789/0 conn(0x7f2910108fb0 legacy=0x7f291011ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:15.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291011f850 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291011c500 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 10 client.26019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f291011c990 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f2916902640 1 librados: init done 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f2900002360 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.024+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f29000059a0 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f290001bdc0 con 0x7f2910101590 2024-09-10T19:13:15.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc005040 con 0x7f2910101590 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f290008a250 con 0x7f2910101590 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 10 client.26019.objecter ms_dispatch 0x7f2910000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 3 client.26019.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 3 client.26019.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 20 client.26019.objecter dump_active .. 0 homeless 2024-09-10T19:13:15.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.028+0000 7f290cff9640 10 client.26019.objecter ms_handle_connect 0x7f28f8071770 2024-09-10T19:13:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.032+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f2900056670 con 0x7f2910101590 2024-09-10T19:13:15.111 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:14 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.775807+0000 osd.5 (osd.5) 580 : cluster [DBG] 2.2 deep-scrub starts 2024-09-10T19:13:15.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:14 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.779273+0000 osd.5 (osd.5) 581 : cluster [DBG] 2.2 deep-scrub ok 2024-09-10T19:13:15.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:14 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.907122+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.19 scrub starts 2024-09-10T19:13:15.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:14 smithi165 bash[22591]: cluster 2024-09-10T19:13:13.910658+0000 osd.3 (osd.3) 609 : cluster [DBG] 2.19 scrub ok 2024-09-10T19:13:15.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:14 smithi165 bash[22591]: audit 2024-09-10T19:13:14.583987+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:15.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.212+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f28fc0052f0 con 0x7f2910101590 2024-09-10T19:13:15.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.212+0000 7f290cff9640 1 -- 172.21.15.83:0/251441584 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 2351066234) 0x7f2900056490 con 0x7f2910101590 2024-09-10T19:13:15.216 INFO:teuthology.orchestra.run.smithi083.stdout:34359738742 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 10 librados: watch_flush enter 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 10 librados: watch_flush exit 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 20 client.26019.objecter shutdown clearing up homeless session... 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 10 client.26019.objecter successfully canceled tick 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 >> v1:172.21.15.83:6800/3071793735 conn(0x7f28f8071770 legacy=0x7f28f8073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:15.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 >> v1:172.21.15.83:6789/0 conn(0x7f2910101590 legacy=0x7f29101059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:15.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 shutdown_connections 2024-09-10T19:13:15.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 >> 172.21.15.83:0/251441584 conn(0x7f2910074a20 msgr2=0x7f29100f8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:15.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 shutdown_connections 2024-09-10T19:13:15.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 -- 172.21.15.83:0/251441584 wait complete. 2024-09-10T19:13:15.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:15.216+0000 7f2916902640 1 librados: shutdown 2024-09-10T19:13:16.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.028+0000 7fc49dd51640 1 -- 172.21.15.83:0/3896992945 >> v1:172.21.15.83:6789/0 conn(0x7fc4981014e0 legacy=0x7fc4981018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:16.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.028+0000 7fc49dd51640 1 -- 172.21.15.83:0/3896992945 shutdown_connections 2024-09-10T19:13:16.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.028+0000 7fc49dd51640 1 -- 172.21.15.83:0/3896992945 >> 172.21.15.83:0/3896992945 conn(0x7fc498075290 msgr2=0x7fc4980f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:16.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.028+0000 7fc49dd51640 1 -- 172.21.15.83:0/3896992945 shutdown_connections 2024-09-10T19:13:16.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.028+0000 7fc49dd51640 1 -- 172.21.15.83:0/3896992945 wait complete. 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 librados: starting msgr at 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 librados: starting objecter 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 Processor -- start 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- start start 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 librados: setting wanted keys 2024-09-10T19:13:16.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 librados: calling monclient init 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc498065dc0 con 0x7fc4981042a0 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc49811d380 con 0x7fc4981014e0 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc496ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc4981042a0 0x7fc498117520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41736/0 (socket says 172.21.15.83:41736) 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc496ffd640 1 -- 172.21.15.83:0/3669644208 learned_addr learned my addr 172.21.15.83:0/3669644208 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter ms_handle_connect 0x7fc4981042a0 2024-09-10T19:13:16.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter ms_handle_connect 0x7fc498107e30 2024-09-10T19:13:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc49811c370 con 0x7fc498107e30 2024-09-10T19:13:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter ms_handle_connect 0x7fc4981014e0 2024-09-10T19:13:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403735989 0 0) 0x7fc498065dc0 con 0x7fc4981042a0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4800034e0 con 0x7fc4981042a0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263028201 0 0) 0x7fc49811c370 con 0x7fc498107e30 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc498065dc0 con 0x7fc498107e30 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868805134 0 0) 0x7fc49811d380 con 0x7fc4981014e0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc49811c370 con 0x7fc4981014e0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6592204 0 0) 0x7fc4800034e0 con 0x7fc4981042a0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc49811d380 con 0x7fc4981042a0 2024-09-10T19:13:16.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc488002950 con 0x7fc4981042a0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641626001 0 0) 0x7fc498065dc0 con 0x7fc498107e30 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4800034e0 con 0x7fc498107e30 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc490001c00 con 0x7fc498107e30 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461631221 0 0) 0x7fc49811c370 con 0x7fc4981014e0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc498065dc0 con 0x7fc4981014e0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc48c001f90 con 0x7fc4981014e0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2102202983 0 0) 0x7fc49811d380 con 0x7fc4981042a0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 >> v1:172.21.15.83:6790/0 conn(0x7fc498107e30 legacy=0x7fc49811ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 >> v1:172.21.15.165:6789/0 conn(0x7fc4981014e0 legacy=0x7fc498065210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49811c540 con 0x7fc4981042a0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49811c7f0 con 0x7fc4981042a0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 10 client.26025.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49811cc40 con 0x7fc4981042a0 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc49dd51640 1 librados: init done 2024-09-10T19:13:16.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc4880017d0 con 0x7fc4981042a0 2024-09-10T19:13:16.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.032+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc488005af0 con 0x7fc4981042a0 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: cluster 2024-09-10T19:13:14.465732+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: cluster 2024-09-10T19:13:14.471956+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: cluster 2024-09-10T19:13:14.564852+0000 mgr.y (mgr.24526) 922 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:14.763003+0000 mon.c (mon.2) 930 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:14.823960+0000 mon.a (mon.0) 3890 : audit [DBG] from='client.? v1:172.21.15.83:0/4206975613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:15.215821+0000 mon.a (mon.0) 3891 : audit [DBG] from='client.? v1:172.21.15.83:0/251441584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:15.216598+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:15.217524+0000 mon.c (mon.2) 932 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[18629]: audit 2024-09-10T19:13:15.225525+0000 mon.a (mon.0) 3892 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:16.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: cluster 2024-09-10T19:13:14.465732+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: cluster 2024-09-10T19:13:14.471956+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: cluster 2024-09-10T19:13:14.564852+0000 mgr.y (mgr.24526) 922 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:14.763003+0000 mon.c (mon.2) 930 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:14.823960+0000 mon.a (mon.0) 3890 : audit [DBG] from='client.? v1:172.21.15.83:0/4206975613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:15.215821+0000 mon.a (mon.0) 3891 : audit [DBG] from='client.? v1:172.21.15.83:0/251441584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:15.216598+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:15.217524+0000 mon.c (mon.2) 932 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:16.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:15 smithi083 bash[32381]: audit 2024-09-10T19:13:15.225525+0000 mon.a (mon.0) 3892 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc488006d80 con 0x7fc4981042a0 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc484005010 con 0x7fc4981042a0 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7fc48808a350 con 0x7fc4981042a0 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 10 client.26025.objecter ms_dispatch 0x7fc498000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 3 client.26025.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 3 client.26025.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.036+0000 7fc47ffff640 20 client.26025.objecter dump_active .. 0 homeless 2024-09-10T19:13:16.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.040+0000 7fc47ffff640 10 client.26025.objecter ms_handle_connect 0x7fc480071760 2024-09-10T19:13:16.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.044+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc488056710 con 0x7fc4981042a0 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: cluster 2024-09-10T19:13:14.465732+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d deep-scrub starts 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: cluster 2024-09-10T19:13:14.471956+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d deep-scrub ok 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: cluster 2024-09-10T19:13:14.564852+0000 mgr.y (mgr.24526) 922 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:14.763003+0000 mon.c (mon.2) 930 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:14.823960+0000 mon.a (mon.0) 3890 : audit [DBG] from='client.? v1:172.21.15.83:0/4206975613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:15.215821+0000 mon.a (mon.0) 3891 : audit [DBG] from='client.? v1:172.21.15.83:0/251441584' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:13:16.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:15.216598+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:15.217524+0000 mon.c (mon.2) 932 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:16.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:15 smithi165 bash[22591]: audit 2024-09-10T19:13:15.225525+0000 mon.a (mon.0) 3892 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:16.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc4840052c0 con 0x7fc4981042a0 2024-09-10T19:13:16.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc47ffff640 1 -- 172.21.15.83:0/3669644208 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+14 (unknown 832126871 0 1007691690) 0x7fc488056530 con 0x7fc4981042a0 2024-09-10T19:13:16.227 INFO:teuthology.orchestra.run.smithi083.stdout:3672197038129 2024-09-10T19:13:16.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 10 librados: watch_flush enter 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 10 librados: watch_flush exit 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 20 client.26025.objecter shutdown clearing up homeless session... 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 10 client.26025.objecter successfully canceled tick 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc480071760 legacy=0x7fc480073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 >> v1:172.21.15.83:6789/0 conn(0x7fc4981042a0 legacy=0x7fc498117520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 shutdown_connections 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 >> 172.21.15.83:0/3669644208 conn(0x7fc498075290 msgr2=0x7fc4980f9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 shutdown_connections 2024-09-10T19:13:16.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 -- 172.21.15.83:0/3669644208 wait complete. 2024-09-10T19:13:16.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:16.224+0000 7fc49dd51640 1 librados: shutdown 2024-09-10T19:13:16.417 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972646 got 266287972651 for osd.7 2024-09-10T19:13:16.417 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.044131+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.046919+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.485352+0000 osd.7 (osd.7) 582 : cluster [DBG] 12.d scrub starts 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.489292+0000 osd.7 (osd.7) 583 : cluster [DBG] 12.d scrub ok 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.725312+0000 osd.5 (osd.5) 582 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: cluster 2024-09-10T19:13:15.729914+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:13:16.742 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[18629]: audit 2024-09-10T19:13:16.226983+0000 mon.a (mon.0) 3893 : audit [DBG] from='client.? v1:172.21.15.83:0/3669644208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.044131+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.046919+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.485352+0000 osd.7 (osd.7) 582 : cluster [DBG] 12.d scrub starts 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.489292+0000 osd.7 (osd.7) 583 : cluster [DBG] 12.d scrub ok 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.725312+0000 osd.5 (osd.5) 582 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: cluster 2024-09-10T19:13:15.729914+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:13:16.743 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:16 smithi083 bash[32381]: audit 2024-09-10T19:13:16.226983+0000 mon.a (mon.0) 3893 : audit [DBG] from='client.? v1:172.21.15.83:0/3669644208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.044131+0000 osd.1 (osd.1) 141 : cluster [DBG] 1.3 scrub starts 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.046919+0000 osd.1 (osd.1) 142 : cluster [DBG] 1.3 scrub ok 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.485352+0000 osd.7 (osd.7) 582 : cluster [DBG] 12.d scrub starts 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.489292+0000 osd.7 (osd.7) 583 : cluster [DBG] 12.d scrub ok 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.725312+0000 osd.5 (osd.5) 582 : cluster [DBG] 2.16 scrub starts 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: cluster 2024-09-10T19:13:15.729914+0000 osd.5 (osd.5) 583 : cluster [DBG] 2.16 scrub ok 2024-09-10T19:13:17.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:16 smithi165 bash[22591]: audit 2024-09-10T19:13:16.226983+0000 mon.a (mon.0) 3893 : audit [DBG] from='client.? v1:172.21.15.83:0/3669644208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[18629]: cluster 2024-09-10T19:13:16.566773+0000 mgr.y (mgr.24526) 923 : cluster [DBG] pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.030122+0000 osd.0 (osd.0) 583 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.035081+0000 osd.0 (osd.0) 584 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.308056+0000 osd.4 (osd.4) 558 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.311413+0000 osd.4 (osd.4) 559 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[32381]: cluster 2024-09-10T19:13:16.566773+0000 mgr.y (mgr.24526) 923 : cluster [DBG] pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:18.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.030122+0000 osd.0 (osd.0) 583 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:13:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.035081+0000 osd.0 (osd.0) 584 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:13:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.308056+0000 osd.4 (osd.4) 558 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:13:18.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:17 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.311413+0000 osd.4 (osd.4) 559 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:13:18.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:17 smithi165 bash[22591]: cluster 2024-09-10T19:13:16.566773+0000 mgr.y (mgr.24526) 923 : cluster [DBG] pgmap v1505: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:18.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:17 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.030122+0000 osd.0 (osd.0) 583 : cluster [DBG] 3.17 scrub starts 2024-09-10T19:13:18.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:17 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.035081+0000 osd.0 (osd.0) 584 : cluster [DBG] 3.17 scrub ok 2024-09-10T19:13:18.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:17 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.308056+0000 osd.4 (osd.4) 558 : cluster [DBG] 6.17 scrub starts 2024-09-10T19:13:18.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:17 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.311413+0000 osd.4 (osd.4) 559 : cluster [DBG] 6.17 scrub ok 2024-09-10T19:13:18.202 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855687 got 150323855691 for osd.4 2024-09-10T19:13:18.202 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:18.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- 172.21.15.83:0/3481871007 >> v1:172.21.15.83:6789/0 conn(0x7f134c0fe1d0 legacy=0x7f134c107680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- 172.21.15.83:0/3481871007 shutdown_connections 2024-09-10T19:13:18.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- 172.21.15.83:0/3481871007 >> 172.21.15.83:0/3481871007 conn(0x7f134c075290 msgr2=0x7f134c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- 172.21.15.83:0/3481871007 shutdown_connections 2024-09-10T19:13:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- 172.21.15.83:0/3481871007 wait complete. 2024-09-10T19:13:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 librados: starting msgr at 2024-09-10T19:13:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 librados: starting objecter 2024-09-10T19:13:18.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 Processor -- start 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- start start 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 librados: setting wanted keys 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 librados: calling monclient init 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.336+0000 7f13517d7640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134c1079d0 con 0x7f134c0fd600 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134c107b80 con 0x7f134c0fe1d0 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134c1207e0 con 0x7f134c108f30 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f134affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f134c0fd600 0x7f134c107110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41764/0 (socket says 172.21.15.83:41764) 2024-09-10T19:13:18.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f134affd640 1 -- 172.21.15.83:0/39762659 learned_addr learned my addr 172.21.15.83:0/39762659 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f134c0fd600 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f134c108f30 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter ms_handle_connect 0x7f134c0fe1d0 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345577620 0 0) 0x7f134c1079d0 con 0x7f134c0fd600 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334003120 con 0x7f134c0fd600 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546587589 0 0) 0x7f134c1207e0 con 0x7f134c108f30 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c1079d0 con 0x7f134c108f30 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261180725 0 0) 0x7f134c107b80 con 0x7f134c0fe1d0 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c1207e0 con 0x7f134c0fe1d0 2024-09-10T19:13:18.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057751485 0 0) 0x7f134c1079d0 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134c107b80 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1344001b80 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474664092 0 0) 0x7f1334003120 con 0x7f134c0fd600 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134c1079d0 con 0x7f134c0fd600 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f133c002990 con 0x7f134c0fd600 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 430541020 0 0) 0x7f134c107b80 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 >> v1:172.21.15.165:6789/0 conn(0x7f134c0fe1d0 legacy=0x7f134c119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 >> v1:172.21.15.83:6789/0 conn(0x7f134c0fd600 legacy=0x7f134c107110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c1219c0 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c1209d0 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 10 client.50117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c120e20 con 0x7f134c108f30 2024-09-10T19:13:18.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 librados: init done 2024-09-10T19:13:18.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f1344003100 con 0x7f134c108f30 2024-09-10T19:13:18.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f1344003450 con 0x7f134c108f30 2024-09-10T19:13:18.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338005010 con 0x7f134c108f30 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.340+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f1344002570 con 0x7f134c108f30 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f1344087c80 con 0x7f134c108f30 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 10 client.50117.objecter ms_dispatch 0x7f134c000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 3 client.50117.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 3 client.50117.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 20 client.50117.objecter dump_active .. 0 homeless 2024-09-10T19:13:18.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f13337fe640 10 client.50117.objecter ms_handle_connect 0x7f1334071700 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- 172.21.15.83:0/688317825 >> v1:172.21.15.83:6789/0 conn(0x7f837c10b220 legacy=0x7f837c10d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- 172.21.15.83:0/688317825 shutdown_connections 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- 172.21.15.83:0/688317825 >> 172.21.15.83:0/688317825 conn(0x7f837c074af0 msgr2=0x7f837c0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- 172.21.15.83:0/688317825 shutdown_connections 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- 172.21.15.83:0/688317825 wait complete. 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 librados: starting msgr at 2024-09-10T19:13:18.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 librados: starting objecter 2024-09-10T19:13:18.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 Processor -- start 2024-09-10T19:13:18.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- start start 2024-09-10T19:13:18.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 librados: setting wanted keys 2024-09-10T19:13:18.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 librados: calling monclient init 2024-09-10T19:13:18.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f837c10af60 con 0x7f837c0fd3a0 2024-09-10T19:13:18.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f837c11f6f0 con 0x7f837c0fd860 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f83823cd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f837c120890 con 0x7f837c10b220 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f838096a640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7f837c10b220 0x7f837c11cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49550/0 (socket says 172.21.15.83:49550) 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f838096a640 1 -- 172.21.15.83:0/2525664497 learned_addr learned my addr 172.21.15.83:0/2525664497 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter ms_handle_connect 0x7f837c10b220 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter ms_handle_connect 0x7f837c0fd3a0 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter ms_handle_connect 0x7f837c0fd860 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752086955 0 0) 0x7f837c10af60 con 0x7f837c0fd3a0 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.344+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8364003720 con 0x7f837c0fd3a0 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126269117 0 0) 0x7f837c11f6f0 con 0x7f837c0fd860 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c10af60 con 0x7f837c0fd860 2024-09-10T19:13:18.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709073179 0 0) 0x7f8364003720 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f837c11f6f0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f836c002820 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1031598899 0 0) 0x7f837c11f6f0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 >> v1:172.21.15.83:6790/0 conn(0x7f837c10b220 legacy=0x7f837c11cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 >> v1:172.21.15.165:6789/0 conn(0x7f837c0fd860 legacy=0x7f837c119890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c121a70 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c11e740 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f836c0016c0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f836c003210 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f83823cd640 10 client.26037.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c11eb90 con 0x7f837c0fd3a0 2024-09-10T19:13:18.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f83823cd640 1 librados: init done 2024-09-10T19:13:18.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f836c001260 con 0x7f837c0fd3a0 2024-09-10T19:13:18.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f1344054040 con 0x7f134c108f30 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c0fbca0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f836c087b80 con 0x7f837c0fd3a0 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 10 client.26037.objecter ms_dispatch 0x7f837c000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 3 client.26037.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 3 client.26037.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 20 client.26037.objecter dump_active .. 0 homeless 2024-09-10T19:13:18.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.348+0000 7f8378ff9640 10 client.26037.objecter ms_handle_connect 0x7f8364071280 2024-09-10T19:13:18.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.352+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f836c0543c0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.528 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.524+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f13380052c0 con 0x7f134c108f30 2024-09-10T19:13:18.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.524+0000 7f13337fe640 1 -- 172.21.15.83:0/39762659 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 2573945879) 0x7f1344053e60 con 0x7f134c108f30 2024-09-10T19:13:18.529 INFO:teuthology.orchestra.run.smithi083.stdout:223338299702 2024-09-10T19:13:18.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 10 librados: watch_flush enter 2024-09-10T19:13:18.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 10 librados: watch_flush exit 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 20 client.50117.objecter shutdown clearing up homeless session... 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 10 client.50117.objecter successfully canceled tick 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 >> v1:172.21.15.83:6800/3071793735 conn(0x7f1334071700 legacy=0x7f1334073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 >> v1:172.21.15.83:6790/0 conn(0x7f134c108f30 legacy=0x7f134c11d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 shutdown_connections 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 >> 172.21.15.83:0/39762659 conn(0x7f134c075290 msgr2=0x7f134c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 shutdown_connections 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 -- 172.21.15.83:0/39762659 wait complete. 2024-09-10T19:13:18.532 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.528+0000 7f13517d7640 1 librados: shutdown 2024-09-10T19:13:18.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.532+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f837c110cc0 con 0x7f837c0fd3a0 2024-09-10T19:13:18.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f8378ff9640 1 -- 172.21.15.83:0/2525664497 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 4191994152) 0x7f836c053d60 con 0x7f837c0fd3a0 2024-09-10T19:13:18.538 INFO:teuthology.orchestra.run.smithi083.stdout:81604378976 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 10 librados: watch_flush enter 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 10 librados: watch_flush exit 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 20 client.26037.objecter shutdown clearing up homeless session... 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 10 client.26037.objecter successfully canceled tick 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 >> v1:172.21.15.83:6800/3071793735 conn(0x7f8364071280 legacy=0x7f8364073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 >> v1:172.21.15.83:6789/0 conn(0x7f837c0fd3a0 legacy=0x7f837c10a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:18.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 shutdown_connections 2024-09-10T19:13:18.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 >> 172.21.15.83:0/2525664497 conn(0x7f837c074af0 msgr2=0x7f837c10e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:18.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 shutdown_connections 2024-09-10T19:13:18.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 -- 172.21.15.83:0/2525664497 wait complete. 2024-09-10T19:13:18.542 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:18.536+0000 7f83823cd640 1 librados: shutdown 2024-09-10T19:13:18.821 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.053863+0000 osd.1 (osd.1) 143 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.057368+0000 osd.1 (osd.1) 144 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.521675+0000 osd.7 (osd.7) 584 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.525056+0000 osd.7 (osd.7) 585 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.883853+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.18 deep-scrub starts 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:17.892078+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.18 deep-scrub ok 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.286920+0000 osd.4 (osd.4) 560 : cluster [DBG] 12.2 deep-scrub starts 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.290176+0000 osd.4 (osd.4) 561 : cluster [DBG] 12.2 deep-scrub ok 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: audit 2024-09-10T19:13:18.528607+0000 mon.c (mon.2) 933 : audit [DBG] from='client.? v1:172.21.15.83:0/39762659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:13:18.822 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[32381]: audit 2024-09-10T19:13:18.538339+0000 mon.a (mon.0) 3894 : audit [DBG] from='client.? v1:172.21.15.83:0/2525664497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.053863+0000 osd.1 (osd.1) 143 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.057368+0000 osd.1 (osd.1) 144 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.521675+0000 osd.7 (osd.7) 584 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.525056+0000 osd.7 (osd.7) 585 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.883853+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.18 deep-scrub starts 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:17.892078+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.18 deep-scrub ok 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.286920+0000 osd.4 (osd.4) 560 : cluster [DBG] 12.2 deep-scrub starts 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.290176+0000 osd.4 (osd.4) 561 : cluster [DBG] 12.2 deep-scrub ok 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: audit 2024-09-10T19:13:18.528607+0000 mon.c (mon.2) 933 : audit [DBG] from='client.? v1:172.21.15.83:0/39762659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:13:18.823 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:18 smithi083 bash[18629]: audit 2024-09-10T19:13:18.538339+0000 mon.a (mon.0) 3894 : audit [DBG] from='client.? v1:172.21.15.83:0/2525664497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:13:18.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738736 got 34359738742 for osd.0 2024-09-10T19:13:18.824 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:19.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:13:19.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.053863+0000 osd.1 (osd.1) 143 : cluster [DBG] 12.6 scrub starts 2024-09-10T19:13:19.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.057368+0000 osd.1 (osd.1) 144 : cluster [DBG] 12.6 scrub ok 2024-09-10T19:13:19.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.521675+0000 osd.7 (osd.7) 584 : cluster [DBG] 6.5 scrub starts 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.525056+0000 osd.7 (osd.7) 585 : cluster [DBG] 6.5 scrub ok 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.883853+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.18 deep-scrub starts 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:17.892078+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.18 deep-scrub ok 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.286920+0000 osd.4 (osd.4) 560 : cluster [DBG] 12.2 deep-scrub starts 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.290176+0000 osd.4 (osd.4) 561 : cluster [DBG] 12.2 deep-scrub ok 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: audit 2024-09-10T19:13:18.528607+0000 mon.c (mon.2) 933 : audit [DBG] from='client.? v1:172.21.15.83:0/39762659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:13:19.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:18 smithi165 bash[22591]: audit 2024-09-10T19:13:18.538339+0000 mon.a (mon.0) 3894 : audit [DBG] from='client.? v1:172.21.15.83:0/2525664497' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:13:19.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.936+0000 7ff73d072640 1 -- 172.21.15.83:0/761385631 >> v1:172.21.15.83:6789/0 conn(0x7ff7380fb370 legacy=0x7ff7380fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:19.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.936+0000 7ff73d072640 1 -- 172.21.15.83:0/761385631 shutdown_connections 2024-09-10T19:13:19.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.936+0000 7ff73d072640 1 -- 172.21.15.83:0/761385631 >> 172.21.15.83:0/761385631 conn(0x7ff738075290 msgr2=0x7ff7380f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.936+0000 7ff73d072640 1 -- 172.21.15.83:0/761385631 shutdown_connections 2024-09-10T19:13:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.936+0000 7ff73d072640 1 -- 172.21.15.83:0/761385631 wait complete. 2024-09-10T19:13:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 librados: starting msgr at 2024-09-10T19:13:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 librados: starting objecter 2024-09-10T19:13:19.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 Processor -- start 2024-09-10T19:13:19.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- start start 2024-09-10T19:13:19.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 librados: setting wanted keys 2024-09-10T19:13:19.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 librados: calling monclient init 2024-09-10T19:13:19.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7381155e0 con 0x7ff7380fbe50 2024-09-10T19:13:19.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff73811f6f0 con 0x7ff7380fb370 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff738120890 con 0x7ff73810b1a0 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73659c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7ff7380fbe50 0x7ff738119890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41800/0 (socket says 172.21.15.83:41800) 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73659c640 1 -- 172.21.15.83:0/1663118599 learned_addr learned my addr 172.21.15.83:0/1663118599 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7380fbe50 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff73810b1a0 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7380fb370 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280276941 0 0) 0x7ff7381155e0 con 0x7ff7380fbe50 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7180034e0 con 0x7ff7380fbe50 2024-09-10T19:13:19.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235728940 0 0) 0x7ff738120890 con 0x7ff73810b1a0 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7381155e0 con 0x7ff73810b1a0 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015432627 0 0) 0x7ff73811f6f0 con 0x7ff7380fb370 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff738120890 con 0x7ff7380fb370 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 986631930 0 0) 0x7ff7180034e0 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff73811f6f0 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff7280028f0 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1368179091 0 0) 0x7ff73811f6f0 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 >> v1:172.21.15.83:6790/0 conn(0x7ff73810b1a0 legacy=0x7ff73811cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 >> v1:172.21.15.165:6789/0 conn(0x7ff7380fb370 legacy=0x7ff738114a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff738121a70 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738120a80 con 0x7ff7380fbe50 2024-09-10T19:13:19.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 10 client.26043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:19.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff738120ed0 con 0x7ff7380fbe50 2024-09-10T19:13:19.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff73d072640 1 librados: init done 2024-09-10T19:13:19.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7ff728001810 con 0x7ff7380fbe50 2024-09-10T19:13:19.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.940+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7ff728005a20 con 0x7ff7380fbe50 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c005010 con 0x7ff7380fbe50 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7ff72801bf20 con 0x7ff7380fbe50 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7ff72808a360 con 0x7ff7380fbe50 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 10 client.26043.objecter ms_dispatch 0x7ff738000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 3 client.26043.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 3 client.26043.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:19.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 20 client.26043.objecter dump_active .. 0 homeless 2024-09-10T19:13:19.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.944+0000 7ff72f7fe640 10 client.26043.objecter ms_handle_connect 0x7ff718071400 2024-09-10T19:13:19.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:19.952+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7ff728057500 con 0x7ff7380fbe50 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.567244+0000 mgr.y (mgr.24526) 924 : cluster [DBG] pgmap v1506: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: audit 2024-09-10T19:13:18.727799+0000 mgr.y (mgr.24526) 925 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.835335+0000 osd.3 (osd.3) 612 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.839249+0000 osd.3 (osd.3) 613 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.965908+0000 osd.0 (osd.0) 585 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[18629]: cluster 2024-09-10T19:13:18.969393+0000 osd.0 (osd.0) 586 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.567244+0000 mgr.y (mgr.24526) 924 : cluster [DBG] pgmap v1506: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: audit 2024-09-10T19:13:18.727799+0000 mgr.y (mgr.24526) 925 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.835335+0000 osd.3 (osd.3) 612 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:13:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.839249+0000 osd.3 (osd.3) 613 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:13:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.965908+0000 osd.0 (osd.0) 585 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:13:20.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:19 smithi083 bash[32381]: cluster 2024-09-10T19:13:18.969393+0000 osd.0 (osd.0) 586 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.567244+0000 mgr.y (mgr.24526) 924 : cluster [DBG] pgmap v1506: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: audit 2024-09-10T19:13:18.727799+0000 mgr.y (mgr.24526) 925 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.835335+0000 osd.3 (osd.3) 612 : cluster [DBG] 4.14 deep-scrub starts 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.839249+0000 osd.3 (osd.3) 613 : cluster [DBG] 4.14 deep-scrub ok 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.965908+0000 osd.0 (osd.0) 585 : cluster [DBG] 12.19 scrub starts 2024-09-10T19:13:20.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:19 smithi165 bash[22591]: cluster 2024-09-10T19:13:18.969393+0000 osd.0 (osd.0) 586 : cluster [DBG] 12.19 scrub ok 2024-09-10T19:13:20.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3672197038126 got 3672197038129 for osd.1 2024-09-10T19:13:20.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.128+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff71c0052c0 con 0x7ff7380fbe50 2024-09-10T19:13:20.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.128+0000 7ff72f7fe640 1 -- 172.21.15.83:0/1663118599 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 591188304) 0x7ff728057320 con 0x7ff7380fbe50 2024-09-10T19:13:20.134 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:20.134 INFO:teuthology.orchestra.run.smithi083.stdout:184683594048 2024-09-10T19:13:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 10 librados: watch_flush enter 2024-09-10T19:13:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 10 librados: watch_flush exit 2024-09-10T19:13:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 20 client.26043.objecter shutdown clearing up homeless session... 2024-09-10T19:13:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 10 client.26043.objecter successfully canceled tick 2024-09-10T19:13:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 >> v1:172.21.15.83:6800/3071793735 conn(0x7ff718071400 legacy=0x7ff7180738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 >> v1:172.21.15.83:6789/0 conn(0x7ff7380fbe50 legacy=0x7ff738119890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 shutdown_connections 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 >> 172.21.15.83:0/1663118599 conn(0x7ff738075290 msgr2=0x7ff7380f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 shutdown_connections 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 -- 172.21.15.83:0/1663118599 wait complete. 2024-09-10T19:13:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.132+0000 7ff73d072640 1 librados: shutdown 2024-09-10T19:13:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- 172.21.15.83:0/1166104515 >> v1:172.21.15.83:6790/0 conn(0x7fec901014e0 legacy=0x7fec901018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- 172.21.15.83:0/1166104515 shutdown_connections 2024-09-10T19:13:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- 172.21.15.83:0/1166104515 >> 172.21.15.83:0/1166104515 conn(0x7fec90075290 msgr2=0x7fec900f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:20.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- 172.21.15.83:0/1166104515 shutdown_connections 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- 172.21.15.83:0/1166104515 wait complete. 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 librados: starting msgr at 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 librados: starting objecter 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 Processor -- start 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- start start 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 librados: setting wanted keys 2024-09-10T19:13:20.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 librados: calling monclient init 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec900ff3f0 con 0x7fec90107e30 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec9011d450 con 0x7fec901014e0 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec97dfd640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec9011e630 con 0x7fec901042a0 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec95398640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fec901042a0 0x7fec901175f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:49604/0 (socket says 172.21.15.83:49604) 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec9639a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fec90107e30 0x7fec9011ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:41826/0 (socket says 172.21.15.83:41826) 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec95398640 1 -- 172.21.15.83:0/30177808 learned_addr learned my addr 172.21.15.83:0/30177808 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter ms_handle_connect 0x7fec901042a0 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter ms_handle_connect 0x7fec90107e30 2024-09-10T19:13:20.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter ms_handle_connect 0x7fec901014e0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196327713 0 0) 0x7fec900ff3f0 con 0x7fec90107e30 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec7c0034e0 con 0x7fec90107e30 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922289892 0 0) 0x7fec9011e630 con 0x7fec901042a0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec900ff3f0 con 0x7fec901042a0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624873093 0 0) 0x7fec9011d450 con 0x7fec901014e0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9011e630 con 0x7fec901014e0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 264444187 0 0) 0x7fec7c0034e0 con 0x7fec90107e30 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec9011d450 con 0x7fec90107e30 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec8c0019d0 con 0x7fec90107e30 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451951708 0 0) 0x7fec900ff3f0 con 0x7fec901042a0 2024-09-10T19:13:20.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec7c0034e0 con 0x7fec901042a0 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec80002930 con 0x7fec901042a0 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.680+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180933957 0 0) 0x7fec9011e630 con 0x7fec901014e0 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec900ff3f0 con 0x7fec901014e0 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec880017b0 con 0x7fec901014e0 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2589464129 0 0) 0x7fec9011d450 con 0x7fec90107e30 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 >> v1:172.21.15.83:6790/0 conn(0x7fec901042a0 legacy=0x7fec901175f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 >> v1:172.21.15.165:6789/0 conn(0x7fec901014e0 legacy=0x7fec900fe840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9011f830 con 0x7fec90107e30 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9011c4a0 con 0x7fec90107e30 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec97dfd640 10 client.26049.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9011c8f0 con 0x7fec90107e30 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec97dfd640 1 librados: init done 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fec8c002350 con 0x7fec90107e30 2024-09-10T19:13:20.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fec8c005a10 con 0x7fec90107e30 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec74005010 con 0x7fec90107e30 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fec8c01be30 con 0x7fec90107e30 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7fec8c08a080 con 0x7fec90107e30 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 10 client.26049.objecter ms_dispatch 0x7fec90000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 3 client.26049.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 3 client.26049.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:20.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.684+0000 7fec867fc640 20 client.26049.objecter dump_active .. 0 homeless 2024-09-10T19:13:20.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.688+0000 7fec867fc640 10 client.26049.objecter ms_handle_connect 0x7fec7c071720 2024-09-10T19:13:20.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.692+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fec8c0566b0 con 0x7fec90107e30 2024-09-10T19:13:20.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.872+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 --> v1:172.21.15.83:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fec740052c0 con 0x7fec90107e30 2024-09-10T19:13:20.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.872+0000 7fec867fc640 1 -- 172.21.15.83:0/30177808 <== mon.0 v1:172.21.15.83:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 637375854) 0x7fec8c0564d0 con 0x7fec90107e30 2024-09-10T19:13:20.878 INFO:teuthology.orchestra.run.smithi083.stdout:124554051927 2024-09-10T19:13:20.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 10 librados: watch_flush enter 2024-09-10T19:13:20.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 10 librados: watch_flush exit 2024-09-10T19:13:20.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 20 client.26049.objecter shutdown clearing up homeless session... 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 10 client.26049.objecter successfully canceled tick 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 >> v1:172.21.15.83:6800/3071793735 conn(0x7fec7c071720 legacy=0x7fec7c073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 >> v1:172.21.15.83:6789/0 conn(0x7fec90107e30 legacy=0x7fec9011ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 shutdown_connections 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 >> 172.21.15.83:0/30177808 conn(0x7fec90075290 msgr2=0x7fec900f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:20.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 shutdown_connections 2024-09-10T19:13:20.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 -- 172.21.15.83:0/30177808 wait complete. 2024-09-10T19:13:20.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:20.876+0000 7fec97dfd640 1 librados: shutdown 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:19.454957+0000 osd.2 (osd.2) 411 : cluster [DBG] 2.c scrub starts 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:19.458613+0000 osd.2 (osd.2) 412 : cluster [DBG] 2.c scrub ok 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:19.692195+0000 osd.5 (osd.5) 584 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:19.697475+0000 osd.5 (osd.5) 585 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: audit 2024-09-10T19:13:20.133461+0000 mon.a (mon.0) 3895 : audit [DBG] from='client.? v1:172.21.15.83:0/1663118599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.240600+0000 osd.4 (osd.4) 562 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:13:21.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.244145+0000 osd.4 (osd.4) 563 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:19.454957+0000 osd.2 (osd.2) 411 : cluster [DBG] 2.c scrub starts 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:19.458613+0000 osd.2 (osd.2) 412 : cluster [DBG] 2.c scrub ok 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:19.692195+0000 osd.5 (osd.5) 584 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:19.697475+0000 osd.5 (osd.5) 585 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: audit 2024-09-10T19:13:20.133461+0000 mon.a (mon.0) 3895 : audit [DBG] from='client.? v1:172.21.15.83:0/1663118599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.240600+0000 osd.4 (osd.4) 562 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:13:21.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:20 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.244145+0000 osd.4 (osd.4) 563 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:19.454957+0000 osd.2 (osd.2) 411 : cluster [DBG] 2.c scrub starts 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:19.458613+0000 osd.2 (osd.2) 412 : cluster [DBG] 2.c scrub ok 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:19.692195+0000 osd.5 (osd.5) 584 : cluster [DBG] 4.15 scrub starts 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:19.697475+0000 osd.5 (osd.5) 585 : cluster [DBG] 4.15 scrub ok 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: audit 2024-09-10T19:13:20.133461+0000 mon.a (mon.0) 3895 : audit [DBG] from='client.? v1:172.21.15.83:0/1663118599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.240600+0000 osd.4 (osd.4) 562 : cluster [DBG] 4.18 scrub starts 2024-09-10T19:13:21.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:20 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.244145+0000 osd.4 (osd.4) 563 : cluster [DBG] 4.18 scrub ok 2024-09-10T19:13:21.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378971 got 81604378976 for osd.2 2024-09-10T19:13:21.265 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.438707+0000 osd.7 (osd.7) 586 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.443106+0000 osd.7 (osd.7) 587 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.567991+0000 mgr.y (mgr.24526) 926 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: audit 2024-09-10T19:13:20.877803+0000 mon.a (mon.0) 3896 : audit [DBG] from='client.? v1:172.21.15.83:0/30177808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.982429+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:13:21.769 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:21 smithi165 bash[22591]: cluster 2024-09-10T19:13:20.988285+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:13:21.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299697 got 223338299702 for osd.6 2024-09-10T19:13:21.912 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.438707+0000 osd.7 (osd.7) 586 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.443106+0000 osd.7 (osd.7) 587 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.567991+0000 mgr.y (mgr.24526) 926 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: audit 2024-09-10T19:13:20.877803+0000 mon.a (mon.0) 3896 : audit [DBG] from='client.? v1:172.21.15.83:0/30177808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.982429+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:13:22.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[18629]: cluster 2024-09-10T19:13:20.988285+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.438707+0000 osd.7 (osd.7) 586 : cluster [DBG] 3.15 scrub starts 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.443106+0000 osd.7 (osd.7) 587 : cluster [DBG] 3.15 scrub ok 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.567991+0000 mgr.y (mgr.24526) 926 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: audit 2024-09-10T19:13:20.877803+0000 mon.a (mon.0) 3896 : audit [DBG] from='client.? v1:172.21.15.83:0/30177808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.982429+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1f scrub starts 2024-09-10T19:13:22.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:21 smithi083 bash[32381]: cluster 2024-09-10T19:13:20.988285+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1f scrub ok 2024-09-10T19:13:22.509 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683594043 got 184683594048 for osd.5 2024-09-10T19:13:22.510 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:22.774 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:22] "GET /metrics HTTP/1.1" 200 224156 "" "Prometheus/2.43.0" 2024-09-10T19:13:22.866 INFO:tasks.cephadm.ceph_manager.ceph:need seq 124554051922 got 124554051927 for osd.3 2024-09-10T19:13:22.866 DEBUG:teuthology.parallel:result is None 2024-09-10T19:13:22.866 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T19:13:22.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[32381]: cluster 2024-09-10T19:13:21.087199+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.d scrub starts 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[32381]: cluster 2024-09-10T19:13:21.090656+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.d scrub ok 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[32381]: cluster 2024-09-10T19:13:21.391283+0000 osd.7 (osd.7) 588 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[32381]: cluster 2024-09-10T19:13:21.395109+0000 osd.7 (osd.7) 589 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[18629]: cluster 2024-09-10T19:13:21.087199+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.d scrub starts 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[18629]: cluster 2024-09-10T19:13:21.090656+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.d scrub ok 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[18629]: cluster 2024-09-10T19:13:21.391283+0000 osd.7 (osd.7) 588 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:13:23.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:22 smithi083 bash[18629]: cluster 2024-09-10T19:13:21.395109+0000 osd.7 (osd.7) 589 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:13:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:22 smithi165 bash[22591]: cluster 2024-09-10T19:13:21.087199+0000 osd.1 (osd.1) 145 : cluster [DBG] 2.d scrub starts 2024-09-10T19:13:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:22 smithi165 bash[22591]: cluster 2024-09-10T19:13:21.090656+0000 osd.1 (osd.1) 146 : cluster [DBG] 2.d scrub ok 2024-09-10T19:13:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:22 smithi165 bash[22591]: cluster 2024-09-10T19:13:21.391283+0000 osd.7 (osd.7) 588 : cluster [DBG] 2.8 scrub starts 2024-09-10T19:13:23.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:22 smithi165 bash[22591]: cluster 2024-09-10T19:13:21.395109+0000 osd.7 (osd.7) 589 : cluster [DBG] 2.8 scrub ok 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:22.545358+0000 osd.2 (osd.2) 413 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:22.549315+0000 osd.2 (osd.2) 414 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:22.569930+0000 mgr.y (mgr.24526) 927 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:22.797251+0000 osd.3 (osd.3) 614 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:22.801996+0000 osd.3 (osd.3) 615 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:13:24.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.297037+0000 osd.4 (osd.4) 564 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.300765+0000 osd.4 (osd.4) 565 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:22.545358+0000 osd.2 (osd.2) 413 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:22.549315+0000 osd.2 (osd.2) 414 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:22.569930+0000 mgr.y (mgr.24526) 927 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:22.797251+0000 osd.3 (osd.3) 614 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:22.801996+0000 osd.3 (osd.3) 615 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.297037+0000 osd.4 (osd.4) 564 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:13:24.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:23 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.300765+0000 osd.4 (osd.4) 565 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:22.545358+0000 osd.2 (osd.2) 413 : cluster [DBG] 6.14 scrub starts 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:22.549315+0000 osd.2 (osd.2) 414 : cluster [DBG] 6.14 scrub ok 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:22.569930+0000 mgr.y (mgr.24526) 927 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:22.797251+0000 osd.3 (osd.3) 614 : cluster [DBG] 3.7 scrub starts 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:22.801996+0000 osd.3 (osd.3) 615 : cluster [DBG] 3.7 scrub ok 2024-09-10T19:13:24.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.297037+0000 osd.4 (osd.4) 564 : cluster [DBG] 6.2 scrub starts 2024-09-10T19:13:24.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:23 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.300765+0000 osd.4 (osd.4) 565 : cluster [DBG] 6.2 scrub ok 2024-09-10T19:13:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.065998+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:13:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.068816+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:13:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.414245+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.e deep-scrub starts 2024-09-10T19:13:25.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.425451+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.e deep-scrub ok 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.524380+0000 osd.2 (osd.2) 415 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[18629]: cluster 2024-09-10T19:13:23.527699+0000 osd.2 (osd.2) 416 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.065998+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.068816+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.414245+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.e deep-scrub starts 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.425451+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.e deep-scrub ok 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.524380+0000 osd.2 (osd.2) 415 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:13:25.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:24 smithi083 bash[32381]: cluster 2024-09-10T19:13:23.527699+0000 osd.2 (osd.2) 416 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.065998+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.7 scrub starts 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.068816+0000 osd.1 (osd.1) 148 : cluster [DBG] 4.7 scrub ok 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.414245+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.e deep-scrub starts 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.425451+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.e deep-scrub ok 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.524380+0000 osd.2 (osd.2) 415 : cluster [DBG] 2.10 scrub starts 2024-09-10T19:13:25.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:24 smithi165 bash[22591]: cluster 2024-09-10T19:13:23.527699+0000 osd.2 (osd.2) 416 : cluster [DBG] 2.10 scrub ok 2024-09-10T19:13:26.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:25 smithi083 bash[18629]: cluster 2024-09-10T19:13:24.570742+0000 mgr.y (mgr.24526) 928 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:26.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:25 smithi083 bash[32381]: cluster 2024-09-10T19:13:24.570742+0000 mgr.y (mgr.24526) 928 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:26.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:25 smithi165 bash[22591]: cluster 2024-09-10T19:13:24.570742+0000 mgr.y (mgr.24526) 928 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[18629]: cluster 2024-09-10T19:13:25.663221+0000 osd.5 (osd.5) 586 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[18629]: cluster 2024-09-10T19:13:25.666724+0000 osd.5 (osd.5) 587 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[18629]: cluster 2024-09-10T19:13:25.819993+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[18629]: cluster 2024-09-10T19:13:25.823269+0000 osd.3 (osd.3) 617 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[32381]: cluster 2024-09-10T19:13:25.663221+0000 osd.5 (osd.5) 586 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:13:27.041 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[32381]: cluster 2024-09-10T19:13:25.666724+0000 osd.5 (osd.5) 587 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:13:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[32381]: cluster 2024-09-10T19:13:25.819993+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:13:27.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:26 smithi083 bash[32381]: cluster 2024-09-10T19:13:25.823269+0000 osd.3 (osd.3) 617 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:13:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:26 smithi165 bash[22591]: cluster 2024-09-10T19:13:25.663221+0000 osd.5 (osd.5) 586 : cluster [DBG] 6.19 scrub starts 2024-09-10T19:13:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:26 smithi165 bash[22591]: cluster 2024-09-10T19:13:25.666724+0000 osd.5 (osd.5) 587 : cluster [DBG] 6.19 scrub ok 2024-09-10T19:13:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:26 smithi165 bash[22591]: cluster 2024-09-10T19:13:25.819993+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.13 scrub starts 2024-09-10T19:13:27.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:26 smithi165 bash[22591]: cluster 2024-09-10T19:13:25.823269+0000 osd.3 (osd.3) 617 : cluster [DBG] 6.13 scrub ok 2024-09-10T19:13:27.642 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.093514+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:13:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.099699+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:13:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.431208+0000 osd.6 (osd.6) 539 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:13:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.435585+0000 osd.6 (osd.6) 540 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:13:28.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.572631+0000 mgr.y (mgr.24526) 929 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.648661+0000 osd.5 (osd.5) 588 : cluster [DBG] 6.d scrub starts 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:26.652372+0000 osd.5 (osd.5) 589 : cluster [DBG] 6.d scrub ok 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.292409+0000 osd.4 (osd.4) 566 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.298008+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.093514+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.099699+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.431208+0000 osd.6 (osd.6) 539 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.435585+0000 osd.6 (osd.6) 540 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.572631+0000 mgr.y (mgr.24526) 929 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.648661+0000 osd.5 (osd.5) 588 : cluster [DBG] 6.d scrub starts 2024-09-10T19:13:28.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:26.652372+0000 osd.5 (osd.5) 589 : cluster [DBG] 6.d scrub ok 2024-09-10T19:13:28.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.292409+0000 osd.4 (osd.4) 566 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T19:13:28.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:27 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.298008+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.093514+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.4 scrub starts 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.099699+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.4 scrub ok 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.431208+0000 osd.6 (osd.6) 539 : cluster [DBG] 4.1f scrub starts 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.435585+0000 osd.6 (osd.6) 540 : cluster [DBG] 4.1f scrub ok 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.572631+0000 mgr.y (mgr.24526) 929 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:28.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.648661+0000 osd.5 (osd.5) 588 : cluster [DBG] 6.d scrub starts 2024-09-10T19:13:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:26.652372+0000 osd.5 (osd.5) 589 : cluster [DBG] 6.d scrub ok 2024-09-10T19:13:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.292409+0000 osd.4 (osd.4) 566 : cluster [DBG] 3.9 deep-scrub starts 2024-09-10T19:13:28.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:27 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.298008+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.9 deep-scrub ok 2024-09-10T19:13:29.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:13:28 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:13:29.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:28 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.450957+0000 osd.7 (osd.7) 592 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:13:29.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:28 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.454299+0000 osd.7 (osd.7) 593 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:13:29.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:28 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.670912+0000 osd.5 (osd.5) 590 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:13:29.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:28 smithi165 bash[22591]: cluster 2024-09-10T19:13:27.674181+0000 osd.5 (osd.5) 591 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:13:29.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.450957+0000 osd.7 (osd.7) 592 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:13:29.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.454299+0000 osd.7 (osd.7) 593 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:13:29.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.670912+0000 osd.5 (osd.5) 590 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:13:29.164 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[18629]: cluster 2024-09-10T19:13:27.674181+0000 osd.5 (osd.5) 591 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:13:29.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.450957+0000 osd.7 (osd.7) 592 : cluster [DBG] 6.15 scrub starts 2024-09-10T19:13:29.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.454299+0000 osd.7 (osd.7) 593 : cluster [DBG] 6.15 scrub ok 2024-09-10T19:13:29.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.670912+0000 osd.5 (osd.5) 590 : cluster [DBG] 4.8 scrub starts 2024-09-10T19:13:29.165 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:28 smithi083 bash[32381]: cluster 2024-09-10T19:13:27.674181+0000 osd.5 (osd.5) 591 : cluster [DBG] 4.8 scrub ok 2024-09-10T19:13:29.804 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:28.044208+0000 osd.1 (osd.1) 151 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:13:29.804 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:28.047647+0000 osd.1 (osd.1) 152 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:13:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:28.044208+0000 osd.1 (osd.1) 151 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:13:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:28.047647+0000 osd.1 (osd.1) 152 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:13:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:28.405701+0000 osd.7 (osd.7) 594 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:13:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:28.409472+0000 osd.7 (osd.7) 595 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:13:30.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:28.573390+0000 mgr.y (mgr.24526) 930 : cluster [DBG] pgmap v1511: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: audit 2024-09-10T19:13:28.736395+0000 mgr.y (mgr.24526) 931 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:29.314090+0000 osd.4 (osd.4) 568 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:13:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: cluster 2024-09-10T19:13:29.317581+0000 osd.4 (osd.4) 569 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:13:30.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:29 smithi165 bash[22591]: audit 2024-09-10T19:13:29.584508+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:28.044208+0000 osd.1 (osd.1) 151 : cluster [DBG] 12.13 scrub starts 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:28.047647+0000 osd.1 (osd.1) 152 : cluster [DBG] 12.13 scrub ok 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:28.405701+0000 osd.7 (osd.7) 594 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:28.409472+0000 osd.7 (osd.7) 595 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:28.573390+0000 mgr.y (mgr.24526) 930 : cluster [DBG] pgmap v1511: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: audit 2024-09-10T19:13:28.736395+0000 mgr.y (mgr.24526) 931 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:30.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:29.314090+0000 osd.4 (osd.4) 568 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: cluster 2024-09-10T19:13:29.317581+0000 osd.4 (osd.4) 569 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[32381]: audit 2024-09-10T19:13:29.584508+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:28.405701+0000 osd.7 (osd.7) 594 : cluster [DBG] 2.0 scrub starts 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:28.409472+0000 osd.7 (osd.7) 595 : cluster [DBG] 2.0 scrub ok 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:28.573390+0000 mgr.y (mgr.24526) 930 : cluster [DBG] pgmap v1511: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: audit 2024-09-10T19:13:28.736395+0000 mgr.y (mgr.24526) 931 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:29.314090+0000 osd.4 (osd.4) 568 : cluster [DBG] 6.1a scrub starts 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: cluster 2024-09-10T19:13:29.317581+0000 osd.4 (osd.4) 569 : cluster [DBG] 6.1a scrub ok 2024-09-10T19:13:30.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:29 smithi083 bash[18629]: audit 2024-09-10T19:13:29.584508+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:31.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- 172.21.15.83:0/2411183372 >> v1:172.21.15.83:6789/0 conn(0x7f499c0fd5c0 legacy=0x7f499c0fd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- 172.21.15.83:0/2411183372 shutdown_connections 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- 172.21.15.83:0/2411183372 >> 172.21.15.83:0/2411183372 conn(0x7f499c0f7050 msgr2=0x7f499c0f9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- 172.21.15.83:0/2411183372 shutdown_connections 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- 172.21.15.83:0/2411183372 wait complete. 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 librados: starting msgr at 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 librados: starting objecter 2024-09-10T19:13:31.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 Processor -- start 2024-09-10T19:13:31.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- start start 2024-09-10T19:13:31.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 librados: setting wanted keys 2024-09-10T19:13:31.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 librados: calling monclient init 2024-09-10T19:13:31.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f499c11c360 con 0x7f499c108ef0 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f499c11d560 con 0x7f499c1185f0 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f49a28d0640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f499c11e760 con 0x7f499c0fd5c0 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f499b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f499c108ef0 0x7f499c1047f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35434/0 (socket says 172.21.15.83:35434) 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f499b7fe640 1 -- 172.21.15.83:0/724317947 learned_addr learned my addr 172.21.15.83:0/724317947 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f499bfff640 1 --1- 172.21.15.83:0/724317947 >> v1:172.21.15.83:6790/0 conn(0x7f499c0fd5c0 0x7f499c1040e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:47848/0 (socket says 172.21.15.83:47848) 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f4998ff9640 10 client.?.objecter ms_handle_connect 0x7f499c0fd5c0 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f4998ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f4998ff9640 10 client.?.objecter ms_handle_connect 0x7f499c108ef0 2024-09-10T19:13:31.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f4998ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.644+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680083797 0 0) 0x7f499c11c360 con 0x7f499c108ef0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49840038b0 con 0x7f499c108ef0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 10 client.?.objecter ms_handle_connect 0x7f499c1185f0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278266914 0 0) 0x7f499c11e760 con 0x7f499c0fd5c0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c11c360 con 0x7f499c0fd5c0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117095819 0 0) 0x7f499c11d560 con 0x7f499c1185f0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c11e760 con 0x7f499c1185f0 2024-09-10T19:13:31.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292034489 0 0) 0x7f499c11c360 con 0x7f499c0fd5c0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f499c11d560 con 0x7f499c0fd5c0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4990002110 con 0x7f499c0fd5c0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031987131 0 0) 0x7f49840038b0 con 0x7f499c108ef0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f499c11c360 con 0x7f499c108ef0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f498c001aa0 con 0x7f499c108ef0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2979139923 0 0) 0x7f499c11e760 con 0x7f499c1185f0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49840038b0 con 0x7f499c1185f0 2024-09-10T19:13:31.653 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f4994001c40 con 0x7f499c1185f0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1270124267 0 0) 0x7f499c11d560 con 0x7f499c0fd5c0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 >> v1:172.21.15.165:6789/0 conn(0x7f499c1185f0 legacy=0x7f499c11aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 >> v1:172.21.15.83:6789/0 conn(0x7f499c108ef0 legacy=0x7f499c1047f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499c11f960 con 0x7f499c0fd5c0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499c11c550 con 0x7f499c0fd5c0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f49a28d0640 10 client.50135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f499c11c9a0 con 0x7f499c0fd5c0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f49a28d0640 1 librados: init done 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f4990002ca0 con 0x7f499c0fd5c0 2024-09-10T19:13:31.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f49900035b0 con 0x7f499c0fd5c0 2024-09-10T19:13:31.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.648+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988005010 con 0x7f499c0fd5c0 2024-09-10T19:13:31.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f4990001410 con 0x7f499c0fd5c0 2024-09-10T19:13:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f499008a590 con 0x7f499c0fd5c0 2024-09-10T19:13:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 10 client.50135.objecter ms_dispatch 0x7f499c000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 3 client.50135.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 3 client.50135.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:31.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 20 client.50135.objecter dump_active .. 0 homeless 2024-09-10T19:13:31.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.652+0000 7f4998ff9640 10 client.50135.objecter ms_handle_connect 0x7f49840716f0 2024-09-10T19:13:31.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.656+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f4990056dd0 con 0x7f499c0fd5c0 2024-09-10T19:13:31.807 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:31.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.832+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4988000b90 con 0x7f49840716f0 2024-09-10T19:13:31.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.844+0000 7f4998ff9640 1 -- 172.21.15.83:0/724317947 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491974 (unknown 2965378022 0 1094713349) 0x7f4988000b90 con 0x7f49840716f0 2024-09-10T19:13:31.857 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:13:31.861 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:13:31.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 10 librados: watch_flush enter 2024-09-10T19:13:31.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 10 librados: watch_flush exit 2024-09-10T19:13:31.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 20 client.50135.objecter shutdown clearing up homeless session... 2024-09-10T19:13:31.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 10 client.50135.objecter successfully canceled tick 2024-09-10T19:13:31.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 >> v1:172.21.15.83:6800/3071793735 conn(0x7f49840716f0 legacy=0x7f4984073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:31.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 >> v1:172.21.15.83:6790/0 conn(0x7f499c0fd5c0 legacy=0x7f499c1040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:31.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 shutdown_connections 2024-09-10T19:13:31.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.852+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 >> 172.21.15.83:0/724317947 conn(0x7f499c0f7050 msgr2=0x7f499c10b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:31.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.856+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 shutdown_connections 2024-09-10T19:13:31.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.856+0000 7f49a28d0640 1 -- 172.21.15.83:0/724317947 wait complete. 2024-09-10T19:13:31.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:31.856+0000 7f49a28d0640 1 librados: shutdown 2024-09-10T19:13:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[22591]: cluster 2024-09-10T19:13:30.574118+0000 mgr.y (mgr.24526) 932 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[22591]: cluster 2024-09-10T19:13:30.645948+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.e scrub starts 2024-09-10T19:13:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[22591]: cluster 2024-09-10T19:13:30.649972+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.e scrub ok 2024-09-10T19:13:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[22591]: cluster 2024-09-10T19:13:31.276272+0000 osd.4 (osd.4) 570 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:13:32.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:31 smithi165 bash[22591]: cluster 2024-09-10T19:13:31.279698+0000 osd.4 (osd.4) 571 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:13:32.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[18629]: cluster 2024-09-10T19:13:30.574118+0000 mgr.y (mgr.24526) 932 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:32.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[18629]: cluster 2024-09-10T19:13:30.645948+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.e scrub starts 2024-09-10T19:13:32.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[18629]: cluster 2024-09-10T19:13:30.649972+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.e scrub ok 2024-09-10T19:13:32.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[18629]: cluster 2024-09-10T19:13:31.276272+0000 osd.4 (osd.4) 570 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:13:32.127 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[18629]: cluster 2024-09-10T19:13:31.279698+0000 osd.4 (osd.4) 571 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:13:32.128 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[32381]: cluster 2024-09-10T19:13:30.574118+0000 mgr.y (mgr.24526) 932 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:32.129 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[32381]: cluster 2024-09-10T19:13:30.645948+0000 osd.2 (osd.2) 417 : cluster [DBG] 2.e scrub starts 2024-09-10T19:13:32.129 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[32381]: cluster 2024-09-10T19:13:30.649972+0000 osd.2 (osd.2) 418 : cluster [DBG] 2.e scrub ok 2024-09-10T19:13:32.129 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[32381]: cluster 2024-09-10T19:13:31.276272+0000 osd.4 (osd.4) 570 : cluster [DBG] 12.1e scrub starts 2024-09-10T19:13:32.129 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:31 smithi083 bash[32381]: cluster 2024-09-10T19:13:31.279698+0000 osd.4 (osd.4) 571 : cluster [DBG] 12.1e scrub ok 2024-09-10T19:13:32.760 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2024-09-10T19:13:30.573676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":3022,"num_read_kb":2773,"num_write":1009,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2116,"ondisk_log_size":2116,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2454384,"kb_used_data":15600,"kb_used_omap":101,"kb_used_meta":2438618,"kb_avail":747506832,"statfs":{"total":767960285184,"available":765446995968,"internally_reserved":0,"allocated":15974400,"data_stored":5897779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103861,"internal_metadata":2497145419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006669"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":924,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628580+0000","last_change":"2024-09-10T19:12:05.963530+0000","last_active":"2024-09-10T19:12:54.628580+0000","last_peered":"2024-09-10T19:12:54.628580+0000","last_clean":"2024-09-10T19:12:54.628580+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:12:54.628580+0000","last_undegraded":"2024-09-10T19:12:54.628580+0000","last_fullsized":"2024-09-10T19:12:54.628580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.963371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.963371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035715529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.632665+0000","last_change":"2024-09-10T19:12:16.081436+0000","last_active":"2024-09-10T19:12:54.632665+0000","last_peered":"2024-09-10T19:12:54.632665+0000","last_clean":"2024-09-10T19:12:54.632665+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:12:54.632665+0000","last_undegraded":"2024-09-10T19:12:54.632665+0000","last_fullsized":"2024-09-10T19:12:54.632665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:16.081285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.081285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.314031+0000","scrub_duration":0.0036441989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"881'35","reported_seq":978,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630369+0000","last_change":"2024-09-10T19:12:52.099506+0000","last_active":"2024-09-10T19:12:54.630369+0000","last_peered":"2024-09-10T19:12:54.630369+0000","last_clean":"2024-09-10T19:12:54.630369+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:12:54.630369+0000","last_undegraded":"2024-09-10T19:12:54.630369+0000","last_fullsized":"2024-09-10T19:12:54.630369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"881'35","last_scrub_stamp":"2024-09-10T19:12:52.099394+0000","last_deep_scrub":"881'35","last_deep_scrub_stamp":"2024-09-10T19:11:11.949004+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.099394+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.338297+0000","scrub_duration":0.0033541360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":936,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628948+0000","last_change":"2024-09-10T19:11:55.924492+0000","last_active":"2024-09-10T19:12:54.628948+0000","last_peered":"2024-09-10T19:12:54.628948+0000","last_clean":"2024-09-10T19:12:54.628948+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:12:54.628948+0000","last_undegraded":"2024-09-10T19:12:54.628948+0000","last_fullsized":"2024-09-10T19:12:54.628948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.924340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.924340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.735622+0000","scrub_duration":0.0034557429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.804312+0000","last_change":"2024-09-10T19:13:05.804094+0000","last_active":"2024-09-10T19:13:05.804312+0000","last_peered":"2024-09-10T19:13:05.804312+0000","last_clean":"2024-09-10T19:13:05.804312+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:13:05.804312+0000","last_undegraded":"2024-09-10T19:13:05.804312+0000","last_fullsized":"2024-09-10T19:13:05.804312+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.803920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.803920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.317826+0000","scrub_duration":0.0037495219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630651+0000","last_change":"2024-09-10T19:11:49.918802+0000","last_active":"2024-09-10T19:12:54.630651+0000","last_peered":"2024-09-10T19:12:54.630651+0000","last_clean":"2024-09-10T19:12:54.630651+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:12:54.630651+0000","last_undegraded":"2024-09-10T19:12:54.630651+0000","last_fullsized":"2024-09-10T19:12:54.630651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:49.918655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.918655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.174706+0000","scrub_duration":0.003368209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"877'32","reported_seq":990,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.988719+0000","last_change":"2024-09-10T19:13:20.988583+0000","last_active":"2024-09-10T19:13:20.988719+0000","last_peered":"2024-09-10T19:13:20.988719+0000","last_clean":"2024-09-10T19:13:20.988719+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:13:20.988719+0000","last_undegraded":"2024-09-10T19:13:20.988719+0000","last_fullsized":"2024-09-10T19:13:20.988719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'32","last_scrub_stamp":"2024-09-10T19:13:20.988309+0000","last_deep_scrub":"877'32","last_deep_scrub_stamp":"2024-09-10T19:11:51.184586+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.988309+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.979932+0000","scrub_duration":0.0056414710000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":970,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.244392+0000","last_change":"2024-09-10T19:13:20.244315+0000","last_active":"2024-09-10T19:13:20.244392+0000","last_peered":"2024-09-10T19:13:20.244392+0000","last_clean":"2024-09-10T19:13:20.244392+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:13:20.244392+0000","last_undegraded":"2024-09-10T19:13:20.244392+0000","last_fullsized":"2024-09-10T19:13:20.244392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:20.244160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.244160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.925366+0000","scrub_duration":0.0034380309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":582,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627518+0000","last_change":"2024-09-10T19:12:06.476732+0000","last_active":"2024-09-10T19:12:54.627518+0000","last_peered":"2024-09-10T19:12:54.627518+0000","last_clean":"2024-09-10T19:12:54.627518+0000","last_became_active":"2024-09-10T19:10:47.694960+0000","last_became_peered":"2024-09-10T19:10:47.694960+0000","last_unstale":"2024-09-10T19:12:54.627518+0000","last_undegraded":"2024-09-10T19:12:54.627518+0000","last_fullsized":"2024-09-10T19:12:54.627518+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.476579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:06.476579+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.476579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:24.860201+0000","scrub_duration":0.00339406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":866,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630656+0000","last_change":"2024-09-10T19:12:19.018785+0000","last_active":"2024-09-10T19:12:54.630656+0000","last_peered":"2024-09-10T19:12:54.630656+0000","last_clean":"2024-09-10T19:12:54.630656+0000","last_became_active":"2024-09-10T19:09:17.624105+0000","last_became_peered":"2024-09-10T19:09:17.624105+0000","last_unstale":"2024-09-10T19:12:54.630656+0000","last_undegraded":"2024-09-10T19:12:54.630656+0000","last_fullsized":"2024-09-10T19:12:54.630656+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:19.018625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.018625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:42.479971+0000","scrub_duration":0.0033536490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.782411+0000","last_change":"2024-09-10T19:12:09.858704+0000","last_active":"2024-09-10T19:12:54.782411+0000","last_peered":"2024-09-10T19:12:54.782411+0000","last_clean":"2024-09-10T19:12:54.782411+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:12:54.782411+0000","last_undegraded":"2024-09-10T19:12:54.782411+0000","last_fullsized":"2024-09-10T19:12:54.782411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:09.858541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.858541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.070102+0000","scrub_duration":0.0033805240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"859'46","reported_seq":1019,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781638+0000","last_change":"2024-09-10T19:12:19.758669+0000","last_active":"2024-09-10T19:12:54.781638+0000","last_peered":"2024-09-10T19:12:54.781638+0000","last_clean":"2024-09-10T19:12:54.781638+0000","last_became_active":"2024-09-10T19:09:17.624161+0000","last_became_peered":"2024-09-10T19:09:17.624161+0000","last_unstale":"2024-09-10T19:12:54.781638+0000","last_undegraded":"2024-09-10T19:12:54.781638+0000","last_fullsized":"2024-09-10T19:12:54.781638+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:12:19.758507+0000","last_deep_scrub":"859'46","last_deep_scrub_stamp":"2024-09-10T19:12:19.758507+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.758507+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.472714+0000","scrub_duration":0.0055441459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":594,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627622+0000","last_change":"2024-09-10T19:12:28.518152+0000","last_active":"2024-09-10T19:12:54.627622+0000","last_peered":"2024-09-10T19:12:54.627622+0000","last_clean":"2024-09-10T19:12:54.627622+0000","last_became_active":"2024-09-10T19:10:47.695582+0000","last_became_peered":"2024-09-10T19:10:47.695582+0000","last_unstale":"2024-09-10T19:12:54.627622+0000","last_undegraded":"2024-09-10T19:12:54.627622+0000","last_fullsized":"2024-09-10T19:12:54.627622+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.517993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.517993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.359419+0000","scrub_duration":0.00357822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":960,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:25.666981+0000","last_change":"2024-09-10T19:13:25.666878+0000","last_active":"2024-09-10T19:13:25.666981+0000","last_peered":"2024-09-10T19:13:25.666981+0000","last_clean":"2024-09-10T19:13:25.666981+0000","last_became_active":"2024-09-10T19:09:17.623986+0000","last_became_peered":"2024-09-10T19:09:17.623986+0000","last_unstale":"2024-09-10T19:13:25.666981+0000","last_undegraded":"2024-09-10T19:13:25.666981+0000","last_fullsized":"2024-09-10T19:13:25.666981+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:25.666738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:25.666738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.854461+0000","scrub_duration":0.0034230649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":797,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778636+0000","last_change":"2024-09-10T19:12:17.028101+0000","last_active":"2024-09-10T19:12:54.778636+0000","last_peered":"2024-09-10T19:12:54.778636+0000","last_clean":"2024-09-10T19:12:54.778636+0000","last_became_active":"2024-09-10T19:09:17.622887+0000","last_became_peered":"2024-09-10T19:09:17.622887+0000","last_unstale":"2024-09-10T19:12:54.778636+0000","last_undegraded":"2024-09-10T19:12:54.778636+0000","last_fullsized":"2024-09-10T19:12:54.778636+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:17.027929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:17.027929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:23.525589+0000","scrub_duration":0.0036852719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":956,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780662+0000","last_change":"2024-09-10T19:12:39.694607+0000","last_active":"2024-09-10T19:12:54.780662+0000","last_peered":"2024-09-10T19:12:54.780662+0000","last_clean":"2024-09-10T19:12:54.780662+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:12:54.780662+0000","last_undegraded":"2024-09-10T19:12:54.780662+0000","last_fullsized":"2024-09-10T19:12:54.780662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:39.694347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:39.694347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:53.849112+0000","scrub_duration":0.004782723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"859'29","reported_seq":1000,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.843533+0000","last_change":"2024-09-10T19:12:55.843394+0000","last_active":"2024-09-10T19:12:55.843533+0000","last_peered":"2024-09-10T19:12:55.843533+0000","last_clean":"2024-09-10T19:12:55.843533+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:12:55.843533+0000","last_undegraded":"2024-09-10T19:12:55.843533+0000","last_fullsized":"2024-09-10T19:12:55.843533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'29","last_scrub_stamp":"2024-09-10T19:12:55.843230+0000","last_deep_scrub":"859'29","last_deep_scrub_stamp":"2024-09-10T19:12:55.843230+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.843230+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.269290+0000","scrub_duration":0.0059692130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":594,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627809+0000","last_change":"2024-09-10T19:12:17.621011+0000","last_active":"2024-09-10T19:12:54.627809+0000","last_peered":"2024-09-10T19:12:54.627809+0000","last_clean":"2024-09-10T19:12:54.627809+0000","last_became_active":"2024-09-10T19:10:47.692260+0000","last_became_peered":"2024-09-10T19:10:47.692260+0000","last_unstale":"2024-09-10T19:12:54.627809+0000","last_undegraded":"2024-09-10T19:12:54.627809+0000","last_fullsized":"2024-09-10T19:12:54.627809+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:17.620834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:47.473795+0000","last_clean_scrub_stamp":"2024-09-10T19:12:17.620834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:33.376760+0000","scrub_duration":0.0036247440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":955,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629377+0000","last_change":"2024-09-10T19:12:18.028526+0000","last_active":"2024-09-10T19:12:54.629377+0000","last_peered":"2024-09-10T19:12:54.629377+0000","last_clean":"2024-09-10T19:12:54.629377+0000","last_became_active":"2024-09-10T19:09:17.624498+0000","last_became_peered":"2024-09-10T19:09:17.624498+0000","last_unstale":"2024-09-10T19:12:54.629377+0000","last_undegraded":"2024-09-10T19:12:54.629377+0000","last_fullsized":"2024-09-10T19:12:54.629377+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.028367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.028367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.903157+0000","scrub_duration":0.003381682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629826+0000","last_change":"2024-09-10T19:12:38.983791+0000","last_active":"2024-09-10T19:12:54.629826+0000","last_peered":"2024-09-10T19:12:54.629826+0000","last_clean":"2024-09-10T19:12:54.629826+0000","last_became_active":"2024-09-10T19:09:17.624188+0000","last_became_peered":"2024-09-10T19:09:17.624188+0000","last_unstale":"2024-09-10T19:12:54.629826+0000","last_undegraded":"2024-09-10T19:12:54.629826+0000","last_fullsized":"2024-09-10T19:12:54.629826+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:38.983644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:24.622321+0000","last_clean_scrub_stamp":"2024-09-10T19:12:38.983644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.093397+0000","scrub_duration":0.0031681270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":460,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776821+0000","last_change":"2024-09-10T19:12:16.781316+0000","last_active":"2024-09-10T19:12:54.776821+0000","last_peered":"2024-09-10T19:12:54.776821+0000","last_clean":"2024-09-10T19:12:54.776821+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:12:54.776821+0000","last_undegraded":"2024-09-10T19:12:54.776821+0000","last_fullsized":"2024-09-10T19:12:54.776821+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:16.781157+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:09:46.330407+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.781157+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:28.455349+0000","scrub_duration":0.0044085089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"859'51","reported_seq":1039,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778449+0000","last_change":"2024-09-10T19:12:23.415504+0000","last_active":"2024-09-10T19:12:54.778449+0000","last_peered":"2024-09-10T19:12:54.778449+0000","last_clean":"2024-09-10T19:12:54.778449+0000","last_became_active":"2024-09-10T19:09:17.613942+0000","last_became_peered":"2024-09-10T19:09:17.613942+0000","last_unstale":"2024-09-10T19:12:54.778449+0000","last_undegraded":"2024-09-10T19:12:54.778449+0000","last_fullsized":"2024-09-10T19:12:54.778449+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'51","last_scrub_stamp":"2024-09-10T19:12:23.415308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:23.415308+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:25.951689+0000","scrub_duration":0.0056407159999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":949,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780132+0000","last_change":"2024-09-10T19:12:28.794331+0000","last_active":"2024-09-10T19:12:54.780132+0000","last_peered":"2024-09-10T19:12:54.780132+0000","last_clean":"2024-09-10T19:12:54.780132+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:12:54.780132+0000","last_undegraded":"2024-09-10T19:12:54.780132+0000","last_fullsized":"2024-09-10T19:12:54.780132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.794119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:28.794119+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.794119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.934709+0000","scrub_duration":0.0031531430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"879'16","reported_seq":800,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915342+0000","last_change":"2024-09-10T19:12:56.399010+0000","last_active":"2024-09-10T19:13:16.915342+0000","last_peered":"2024-09-10T19:13:16.915342+0000","last_clean":"2024-09-10T19:13:16.915342+0000","last_became_active":"2024-09-10T19:10:47.694901+0000","last_became_peered":"2024-09-10T19:10:47.694901+0000","last_unstale":"2024-09-10T19:13:16.915342+0000","last_undegraded":"2024-09-10T19:13:16.915342+0000","last_fullsized":"2024-09-10T19:13:16.915342+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'16","last_scrub_stamp":"2024-09-10T19:12:56.398838+0000","last_deep_scrub":"879'16","last_deep_scrub_stamp":"2024-09-10T19:12:56.398838+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.398838+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.826926+0000","scrub_duration":0.0042448329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":962,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781374+0000","last_change":"2024-09-10T19:12:17.232150+0000","last_active":"2024-09-10T19:12:54.781374+0000","last_peered":"2024-09-10T19:12:54.781374+0000","last_clean":"2024-09-10T19:12:54.781374+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:12:54.781374+0000","last_undegraded":"2024-09-10T19:12:54.781374+0000","last_fullsized":"2024-09-10T19:12:54.781374+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:17.232008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:27.103997+0000","last_clean_scrub_stamp":"2024-09-10T19:12:17.232008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:34.473259+0000","scrub_duration":0.0030401159999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778034+0000","last_change":"2024-09-10T19:12:17.451314+0000","last_active":"2024-09-10T19:12:54.778034+0000","last_peered":"2024-09-10T19:12:54.778034+0000","last_clean":"2024-09-10T19:12:54.778034+0000","last_became_active":"2024-09-10T19:09:17.614891+0000","last_became_peered":"2024-09-10T19:09:17.614891+0000","last_unstale":"2024-09-10T19:12:54.778034+0000","last_undegraded":"2024-09-10T19:12:54.778034+0000","last_fullsized":"2024-09-10T19:12:54.778034+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:17.451166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:12:17.451166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:43.562729+0000","scrub_duration":0.0035160650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":1021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777160+0000","last_change":"2024-09-10T19:12:37.757201+0000","last_active":"2024-09-10T19:12:54.777160+0000","last_peered":"2024-09-10T19:12:54.777160+0000","last_clean":"2024-09-10T19:12:54.777160+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:12:54.777160+0000","last_undegraded":"2024-09-10T19:12:54.777160+0000","last_fullsized":"2024-09-10T19:12:54.777160+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:37.756949+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.756949+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.451929+0000","scrub_duration":0.0065141599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"859'25","reported_seq":976,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628795+0000","last_change":"2024-09-10T19:12:07.985950+0000","last_active":"2024-09-10T19:12:54.628795+0000","last_peered":"2024-09-10T19:12:54.628795+0000","last_clean":"2024-09-10T19:12:54.628795+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:12:54.628795+0000","last_undegraded":"2024-09-10T19:12:54.628795+0000","last_fullsized":"2024-09-10T19:12:54.628795+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'25","last_scrub_stamp":"2024-09-10T19:12:07.985723+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:12:07.985723+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:33.206339+0000","scrub_duration":0.0064713729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":893,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.973839+0000","last_change":"2024-09-10T19:12:54.973728+0000","last_active":"2024-09-10T19:12:54.973839+0000","last_peered":"2024-09-10T19:12:54.973839+0000","last_clean":"2024-09-10T19:12:54.973839+0000","last_became_active":"2024-09-10T19:09:17.627988+0000","last_became_peered":"2024-09-10T19:09:17.627988+0000","last_unstale":"2024-09-10T19:12:54.973839+0000","last_undegraded":"2024-09-10T19:12:54.973839+0000","last_fullsized":"2024-09-10T19:12:54.973839+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.973568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.973568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.631809+0000","scrub_duration":0.0035731980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":597,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:12.355934+0000","last_change":"2024-09-10T19:13:12.355799+0000","last_active":"2024-09-10T19:13:12.355934+0000","last_peered":"2024-09-10T19:13:12.355934+0000","last_clean":"2024-09-10T19:13:12.355934+0000","last_became_active":"2024-09-10T19:10:47.692181+0000","last_became_peered":"2024-09-10T19:10:47.692181+0000","last_unstale":"2024-09-10T19:13:12.355934+0000","last_undegraded":"2024-09-10T19:13:12.355934+0000","last_fullsized":"2024-09-10T19:13:12.355934+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:12.355544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:12.355544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.491359+0000","scrub_duration":0.0034917949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":592,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629079+0000","last_change":"2024-09-10T19:12:29.503496+0000","last_active":"2024-09-10T19:12:54.629079+0000","last_peered":"2024-09-10T19:12:54.629079+0000","last_clean":"2024-09-10T19:12:54.629079+0000","last_became_active":"2024-09-10T19:10:47.697379+0000","last_became_peered":"2024-09-10T19:10:47.697379+0000","last_unstale":"2024-09-10T19:12:54.629079+0000","last_undegraded":"2024-09-10T19:12:54.629079+0000","last_fullsized":"2024-09-10T19:12:54.629079+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:29.503343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.503343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.655335+0000","scrub_duration":0.0035948239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777367+0000","last_change":"2024-09-10T19:12:52.446169+0000","last_active":"2024-09-10T19:12:54.777367+0000","last_peered":"2024-09-10T19:12:54.777367+0000","last_clean":"2024-09-10T19:12:54.777367+0000","last_became_active":"2024-09-10T19:09:17.615697+0000","last_became_peered":"2024-09-10T19:09:17.615697+0000","last_unstale":"2024-09-10T19:12:54.777367+0000","last_undegraded":"2024-09-10T19:12:54.777367+0000","last_fullsized":"2024-09-10T19:12:54.777367+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.446009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:34.850823+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.446009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.085539+0000","scrub_duration":0.003327637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":453,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778506+0000","last_change":"2024-09-10T19:12:44.724285+0000","last_active":"2024-09-10T19:12:54.778506+0000","last_peered":"2024-09-10T19:12:54.778506+0000","last_clean":"2024-09-10T19:12:54.778506+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:12:54.778506+0000","last_undegraded":"2024-09-10T19:12:54.778506+0000","last_fullsized":"2024-09-10T19:12:54.778506+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:44.724127+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.724127+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.988635+0000","scrub_duration":0.003706642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"859'39","reported_seq":1016,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628380+0000","last_change":"2024-09-10T19:12:20.014879+0000","last_active":"2024-09-10T19:12:54.628380+0000","last_peered":"2024-09-10T19:12:54.628380+0000","last_clean":"2024-09-10T19:12:54.628380+0000","last_became_active":"2024-09-10T19:09:17.625369+0000","last_became_peered":"2024-09-10T19:09:17.625369+0000","last_unstale":"2024-09-10T19:12:54.628380+0000","last_undegraded":"2024-09-10T19:12:54.628380+0000","last_fullsized":"2024-09-10T19:12:54.628380+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'39","last_scrub_stamp":"2024-09-10T19:12:20.014768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:20.014768+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:48.537536+0000","scrub_duration":0.0047882580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777456+0000","last_change":"2024-09-10T19:12:50.955288+0000","last_active":"2024-09-10T19:12:54.777456+0000","last_peered":"2024-09-10T19:12:54.777456+0000","last_clean":"2024-09-10T19:12:54.777456+0000","last_became_active":"2024-09-10T19:09:17.624964+0000","last_became_peered":"2024-09-10T19:09:17.624964+0000","last_unstale":"2024-09-10T19:12:54.777456+0000","last_undegraded":"2024-09-10T19:12:54.777456+0000","last_fullsized":"2024-09-10T19:12:54.777456+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.955131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:17.868358+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.955131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.325447+0000","scrub_duration":0.0032922939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":866,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:57.057858+0000","last_change":"2024-09-10T19:12:57.057682+0000","last_active":"2024-09-10T19:12:57.057858+0000","last_peered":"2024-09-10T19:12:57.057858+0000","last_clean":"2024-09-10T19:12:57.057858+0000","last_became_active":"2024-09-10T19:09:17.628731+0000","last_became_peered":"2024-09-10T19:09:17.628731+0000","last_unstale":"2024-09-10T19:12:57.057858+0000","last_undegraded":"2024-09-10T19:12:57.057858+0000","last_fullsized":"2024-09-10T19:12:57.057858+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.057429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.057429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.533873+0000","scrub_duration":0.003824095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":957,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781258+0000","last_change":"2024-09-10T19:12:11.771451+0000","last_active":"2024-09-10T19:12:54.781258+0000","last_peered":"2024-09-10T19:12:54.781258+0000","last_clean":"2024-09-10T19:12:54.781258+0000","last_became_active":"2024-09-10T19:09:17.624545+0000","last_became_peered":"2024-09-10T19:09:17.624545+0000","last_unstale":"2024-09-10T19:12:54.781258+0000","last_undegraded":"2024-09-10T19:12:54.781258+0000","last_fullsized":"2024-09-10T19:12:54.781258+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.771295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:11.771295+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.771295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.567598+0000","scrub_duration":0.0034001090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":789,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779295+0000","last_change":"2024-09-10T19:12:35.251646+0000","last_active":"2024-09-10T19:12:54.779295+0000","last_peered":"2024-09-10T19:12:54.779295+0000","last_clean":"2024-09-10T19:12:54.779295+0000","last_became_active":"2024-09-10T19:09:17.626924+0000","last_became_peered":"2024-09-10T19:09:17.626924+0000","last_unstale":"2024-09-10T19:12:54.779295+0000","last_undegraded":"2024-09-10T19:12:54.779295+0000","last_fullsized":"2024-09-10T19:12:54.779295+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:35.251459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:35.251459+0000","last_clean_scrub_stamp":"2024-09-10T19:12:35.251459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.999862+0000","scrub_duration":0.0036221130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"859'29","reported_seq":1008,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627795+0000","last_change":"2024-09-10T19:12:09.480112+0000","last_active":"2024-09-10T19:12:54.627795+0000","last_peered":"2024-09-10T19:12:54.627795+0000","last_clean":"2024-09-10T19:12:54.627795+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:12:54.627795+0000","last_undegraded":"2024-09-10T19:12:54.627795+0000","last_fullsized":"2024-09-10T19:12:54.627795+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'29","last_scrub_stamp":"2024-09-10T19:12:09.479947+0000","last_deep_scrub":"93'17","last_deep_scrub_stamp":"2024-09-10T18:58:57.446527+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.479947+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:23.656619+0000","scrub_duration":0.0045088869999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":970,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:21.395350+0000","last_change":"2024-09-10T19:13:21.395273+0000","last_active":"2024-09-10T19:13:21.395350+0000","last_peered":"2024-09-10T19:13:21.395350+0000","last_clean":"2024-09-10T19:13:21.395350+0000","last_became_active":"2024-09-10T19:09:17.633970+0000","last_became_peered":"2024-09-10T19:09:17.633970+0000","last_unstale":"2024-09-10T19:13:21.395350+0000","last_undegraded":"2024-09-10T19:13:21.395350+0000","last_fullsized":"2024-09-10T19:13:21.395350+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:21.395118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:13:21.395118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.732701+0000","scrub_duration":0.0037239640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"879'16","reported_seq":800,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915356+0000","last_change":"2024-09-10T19:13:05.350996+0000","last_active":"2024-09-10T19:13:16.915356+0000","last_peered":"2024-09-10T19:13:16.915356+0000","last_clean":"2024-09-10T19:13:16.915356+0000","last_became_active":"2024-09-10T19:10:47.695744+0000","last_became_peered":"2024-09-10T19:10:47.695744+0000","last_unstale":"2024-09-10T19:13:16.915356+0000","last_undegraded":"2024-09-10T19:13:16.915356+0000","last_fullsized":"2024-09-10T19:13:16.915356+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'16","last_scrub_stamp":"2024-09-10T19:13:05.350810+0000","last_deep_scrub":"879'16","last_deep_scrub_stamp":"2024-09-10T19:13:05.350810+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.350810+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:11.865095+0000","scrub_duration":0.0043700359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":935,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629123+0000","last_change":"2024-09-10T19:12:09.989957+0000","last_active":"2024-09-10T19:12:54.629123+0000","last_peered":"2024-09-10T19:12:54.629123+0000","last_clean":"2024-09-10T19:12:54.629123+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:12:54.629123+0000","last_undegraded":"2024-09-10T19:12:54.629123+0000","last_fullsized":"2024-09-10T19:12:54.629123+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:12:09.989770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.989770+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:26.724461+0000","scrub_duration":0.0040537400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":801,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.057617+0000","last_change":"2024-09-10T19:13:17.057544+0000","last_active":"2024-09-10T19:13:17.057617+0000","last_peered":"2024-09-10T19:13:17.057617+0000","last_clean":"2024-09-10T19:13:17.057617+0000","last_became_active":"2024-09-10T19:09:17.630225+0000","last_became_peered":"2024-09-10T19:09:17.630225+0000","last_unstale":"2024-09-10T19:13:17.057617+0000","last_undegraded":"2024-09-10T19:13:17.057617+0000","last_fullsized":"2024-09-10T19:13:17.057617+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.057383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.057383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:39.844914+0000","scrub_duration":0.003413592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":455,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629055+0000","last_change":"2024-09-10T19:12:15.916614+0000","last_active":"2024-09-10T19:12:54.629055+0000","last_peered":"2024-09-10T19:12:54.629055+0000","last_clean":"2024-09-10T19:12:54.629055+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:12:54.629055+0000","last_undegraded":"2024-09-10T19:12:54.629055+0000","last_fullsized":"2024-09-10T19:12:54.629055+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:15.916455+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:15.916455+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.319759+0000","scrub_duration":0.0044933289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"859'36","reported_seq":1013,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.161475+0000","last_change":"2024-09-10T19:13:04.161342+0000","last_active":"2024-09-10T19:13:04.161475+0000","last_peered":"2024-09-10T19:13:04.161475+0000","last_clean":"2024-09-10T19:13:04.161475+0000","last_became_active":"2024-09-10T19:09:17.623922+0000","last_became_peered":"2024-09-10T19:09:17.623922+0000","last_unstale":"2024-09-10T19:13:04.161475+0000","last_undegraded":"2024-09-10T19:13:04.161475+0000","last_fullsized":"2024-09-10T19:13:04.161475+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'36","last_scrub_stamp":"2024-09-10T19:13:04.161180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.161180+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.680463+0000","scrub_duration":0.0052040009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:56.421748+0000","last_change":"2024-09-10T19:12:56.421657+0000","last_active":"2024-09-10T19:12:56.421748+0000","last_peered":"2024-09-10T19:12:56.421748+0000","last_clean":"2024-09-10T19:12:56.421748+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:12:56.421748+0000","last_undegraded":"2024-09-10T19:12:56.421748+0000","last_fullsized":"2024-09-10T19:12:56.421748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.421509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.421509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.031135+0000","scrub_duration":0.0033371899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":604,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:08.335492+0000","last_change":"2024-09-10T19:13:08.335428+0000","last_active":"2024-09-10T19:13:08.335492+0000","last_peered":"2024-09-10T19:13:08.335492+0000","last_clean":"2024-09-10T19:13:08.335492+0000","last_became_active":"2024-09-10T19:10:47.689065+0000","last_became_peered":"2024-09-10T19:10:47.689065+0000","last_unstale":"2024-09-10T19:13:08.335492+0000","last_undegraded":"2024-09-10T19:13:08.335492+0000","last_fullsized":"2024-09-10T19:13:08.335492+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.335311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.335311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:33.200985+0000","scrub_duration":0.0030415720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"859'6","reported_seq":2381,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:24.216910+0000","last_change":"2024-09-10T19:12:54.045560+0000","last_active":"2024-09-10T19:13:24.216910+0000","last_peered":"2024-09-10T19:13:24.216910+0000","last_clean":"2024-09-10T19:13:24.216910+0000","last_became_active":"2024-09-10T19:09:17.624538+0000","last_became_peered":"2024-09-10T19:09:17.624538+0000","last_unstale":"2024-09-10T19:13:24.216910+0000","last_undegraded":"2024-09-10T19:13:24.216910+0000","last_fullsized":"2024-09-10T19:13:24.216910+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'6","last_scrub_stamp":"2024-09-10T19:12:54.045409+0000","last_deep_scrub":"859'6","last_deep_scrub_stamp":"2024-09-10T19:09:27.654271+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.045409+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:04.756787+0000","scrub_duration":0.0042215819999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1417,"num_read_kb":1412,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779643+0000","last_change":"2024-09-10T19:12:00.827494+0000","last_active":"2024-09-10T19:12:54.779643+0000","last_peered":"2024-09-10T19:12:54.779643+0000","last_clean":"2024-09-10T19:12:54.779643+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:12:54.779643+0000","last_undegraded":"2024-09-10T19:12:54.779643+0000","last_fullsized":"2024-09-10T19:12:54.779643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.827329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.827329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.678266+0000","scrub_duration":0.0032610619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.493101+0000","last_change":"2024-09-10T19:13:04.493044+0000","last_active":"2024-09-10T19:13:04.493101+0000","last_peered":"2024-09-10T19:13:04.493101+0000","last_clean":"2024-09-10T19:13:04.493101+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:13:04.493101+0000","last_undegraded":"2024-09-10T19:13:04.493101+0000","last_fullsized":"2024-09-10T19:13:04.493101+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.492972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:34.872832+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.492972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:05.971471+0000","scrub_duration":0.0023311849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":465,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628010+0000","last_change":"2024-09-10T19:12:35.370263+0000","last_active":"2024-09-10T19:12:54.628010+0000","last_peered":"2024-09-10T19:12:54.628010+0000","last_clean":"2024-09-10T19:12:54.628010+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:12:54.628010+0000","last_undegraded":"2024-09-10T19:12:54.628010+0000","last_fullsized":"2024-09-10T19:12:54.628010+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:35.370049+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_clean_scrub_stamp":"2024-09-10T19:12:35.370049+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.411760+0000","scrub_duration":0.0043445589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"859'21","reported_seq":987,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:22.802330+0000","last_change":"2024-09-10T19:13:22.802218+0000","last_active":"2024-09-10T19:13:22.802330+0000","last_peered":"2024-09-10T19:13:22.802330+0000","last_clean":"2024-09-10T19:13:22.802330+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:13:22.802330+0000","last_undegraded":"2024-09-10T19:13:22.802330+0000","last_fullsized":"2024-09-10T19:13:22.802330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:22.802012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:22.802012+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:51.225843+0000","scrub_duration":0.0046020039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":902,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778831+0000","last_change":"2024-09-10T19:12:18.069246+0000","last_active":"2024-09-10T19:12:54.778831+0000","last_peered":"2024-09-10T19:12:54.778831+0000","last_clean":"2024-09-10T19:12:54.778831+0000","last_became_active":"2024-09-10T19:09:17.619020+0000","last_became_peered":"2024-09-10T19:09:17.619020+0000","last_unstale":"2024-09-10T19:12:54.778831+0000","last_undegraded":"2024-09-10T19:12:54.778831+0000","last_fullsized":"2024-09-10T19:12:54.778831+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.069100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:18.069100+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.069100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:32.274496+0000","scrub_duration":0.0040091909999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":1226,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915529+0000","last_change":"2024-09-10T19:12:03.932393+0000","last_active":"2024-09-10T19:13:16.915529+0000","last_peered":"2024-09-10T19:13:16.915529+0000","last_clean":"2024-09-10T19:13:16.915529+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:13:16.915529+0000","last_undegraded":"2024-09-10T19:13:16.915529+0000","last_fullsized":"2024-09-10T19:13:16.915529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:12:03.932114+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:12:03.932114+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003643079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":977,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627086+0000","last_change":"2024-09-10T19:12:48.447199+0000","last_active":"2024-09-10T19:12:54.627086+0000","last_peered":"2024-09-10T19:12:54.627086+0000","last_clean":"2024-09-10T19:12:54.627086+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:12:54.627086+0000","last_undegraded":"2024-09-10T19:12:54.627086+0000","last_fullsized":"2024-09-10T19:12:54.627086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:12:48.446936+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.446936+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:50.864977+0000","scrub_duration":0.004606176,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":602,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.301017+0000","last_change":"2024-09-10T19:13:23.300939+0000","last_active":"2024-09-10T19:13:23.301017+0000","last_peered":"2024-09-10T19:13:23.301017+0000","last_clean":"2024-09-10T19:13:23.301017+0000","last_became_active":"2024-09-10T19:10:47.691892+0000","last_became_peered":"2024-09-10T19:10:47.691892+0000","last_unstale":"2024-09-10T19:13:23.301017+0000","last_undegraded":"2024-09-10T19:13:23.301017+0000","last_fullsized":"2024-09-10T19:13:23.301017+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.300782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.300782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.347037+0000","scrub_duration":0.003615732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628117+0000","last_change":"2024-09-10T19:12:40.989724+0000","last_active":"2024-09-10T19:12:54.628117+0000","last_peered":"2024-09-10T19:12:54.628117+0000","last_clean":"2024-09-10T19:12:54.628117+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:12:54.628117+0000","last_undegraded":"2024-09-10T19:12:54.628117+0000","last_fullsized":"2024-09-10T19:12:54.628117+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:40.989570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:40.989570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:08.420730+0000","scrub_duration":0.003425071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":538,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.427126+0000","last_change":"2024-09-10T19:13:06.427010+0000","last_active":"2024-09-10T19:13:06.427126+0000","last_peered":"2024-09-10T19:13:06.427126+0000","last_clean":"2024-09-10T19:13:06.427126+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:13:06.427126+0000","last_undegraded":"2024-09-10T19:13:06.427126+0000","last_fullsized":"2024-09-10T19:13:06.427126+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:06.426853+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.426853+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.227721+0000","scrub_duration":0.0039253079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"859'66","reported_seq":1215,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915283+0000","last_change":"2024-09-10T19:12:08.942471+0000","last_active":"2024-09-10T19:13:16.915283+0000","last_peered":"2024-09-10T19:13:16.915283+0000","last_clean":"2024-09-10T19:13:16.915283+0000","last_became_active":"2024-09-10T19:09:17.624136+0000","last_became_peered":"2024-09-10T19:09:17.624136+0000","last_unstale":"2024-09-10T19:13:16.915283+0000","last_undegraded":"2024-09-10T19:13:16.915283+0000","last_fullsized":"2024-09-10T19:13:16.915283+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'66","last_scrub_stamp":"2024-09-10T19:12:08.942301+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:12:08.942301+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062192020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":953,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.783976+0000","last_change":"2024-09-10T19:12:51.757296+0000","last_active":"2024-09-10T19:12:54.783976+0000","last_peered":"2024-09-10T19:12:54.783976+0000","last_clean":"2024-09-10T19:12:54.783976+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:12:54.783976+0000","last_undegraded":"2024-09-10T19:12:54.783976+0000","last_fullsized":"2024-09-10T19:12:54.783976+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.757142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.757142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.706460+0000","scrub_duration":0.0032464379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":1240,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915409+0000","last_change":"2024-09-10T19:12:41.096730+0000","last_active":"2024-09-10T19:13:16.915409+0000","last_peered":"2024-09-10T19:13:16.915409+0000","last_clean":"2024-09-10T19:13:16.915409+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:13:16.915409+0000","last_undegraded":"2024-09-10T19:13:16.915409+0000","last_fullsized":"2024-09-10T19:13:16.915409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:12:41.096641+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:12:41.096641+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.538296+0000","scrub_duration":0.003951225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":949,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778065+0000","last_change":"2024-09-10T19:12:24.460352+0000","last_active":"2024-09-10T19:12:54.778065+0000","last_peered":"2024-09-10T19:12:54.778065+0000","last_clean":"2024-09-10T19:12:54.778065+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:12:54.778065+0000","last_undegraded":"2024-09-10T19:12:54.778065+0000","last_fullsized":"2024-09-10T19:12:54.778065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:12:24.460189+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:12:24.460189+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.529216+0000","scrub_duration":0.0047295499999999999,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777560+0000","last_change":"2024-09-10T19:12:39.101426+0000","last_active":"2024-09-10T19:12:54.777560+0000","last_peered":"2024-09-10T19:12:54.777560+0000","last_clean":"2024-09-10T19:12:54.777560+0000","last_became_active":"2024-09-10T19:09:17.611408+0000","last_became_peered":"2024-09-10T19:09:17.611408+0000","last_unstale":"2024-09-10T19:12:54.777560+0000","last_undegraded":"2024-09-10T19:12:54.777560+0000","last_fullsized":"2024-09-10T19:12:54.777560+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:39.101274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:11.815591+0000","last_clean_scrub_stamp":"2024-09-10T19:12:39.101274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.570649+0000","scrub_duration":0.0031667710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628941+0000","last_change":"2024-09-10T19:12:06.944923+0000","last_active":"2024-09-10T19:12:54.628941+0000","last_peered":"2024-09-10T19:12:54.628941+0000","last_clean":"2024-09-10T19:12:54.628941+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:12:54.628941+0000","last_undegraded":"2024-09-10T19:12:54.628941+0000","last_fullsized":"2024-09-10T19:12:54.628941+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.944861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.944861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.267746+0000","scrub_duration":0.0021456119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":531,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778794+0000","last_change":"2024-09-10T19:12:29.495085+0000","last_active":"2024-09-10T19:12:54.778794+0000","last_peered":"2024-09-10T19:12:54.778794+0000","last_clean":"2024-09-10T19:12:54.778794+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:12:54.778794+0000","last_undegraded":"2024-09-10T19:12:54.778794+0000","last_fullsized":"2024-09-10T19:12:54.778794+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:29.494918+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:12:29.494918+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.494918+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.808922+0000","scrub_duration":0.0035680629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"859'45","reported_seq":1009,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781707+0000","last_change":"2024-09-10T19:12:28.801393+0000","last_active":"2024-09-10T19:12:54.781707+0000","last_peered":"2024-09-10T19:12:54.781707+0000","last_clean":"2024-09-10T19:12:54.781707+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:12:54.781707+0000","last_undegraded":"2024-09-10T19:12:54.781707+0000","last_fullsized":"2024-09-10T19:12:54.781707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'45","last_scrub_stamp":"2024-09-10T19:12:28.801232+0000","last_deep_scrub":"859'45","last_deep_scrub_stamp":"2024-09-10T19:11:02.154077+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.801232+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.125987+0000","scrub_duration":0.0051599180000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":873,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778792+0000","last_change":"2024-09-10T19:12:16.005163+0000","last_active":"2024-09-10T19:12:54.778792+0000","last_peered":"2024-09-10T19:12:54.778792+0000","last_clean":"2024-09-10T19:12:54.778792+0000","last_became_active":"2024-09-10T19:09:17.630951+0000","last_became_peered":"2024-09-10T19:09:17.630951+0000","last_unstale":"2024-09-10T19:12:54.778792+0000","last_undegraded":"2024-09-10T19:12:54.778792+0000","last_fullsized":"2024-09-10T19:12:54.778792+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:16.005014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:16.005014+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.005014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:41.641050+0000","scrub_duration":0.0034657170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777952+0000","last_change":"2024-09-10T19:12:37.180001+0000","last_active":"2024-09-10T19:12:54.777952+0000","last_peered":"2024-09-10T19:12:54.777952+0000","last_clean":"2024-09-10T19:12:54.777952+0000","last_became_active":"2024-09-10T19:09:17.622457+0000","last_became_peered":"2024-09-10T19:09:17.622457+0000","last_unstale":"2024-09-10T19:12:54.777952+0000","last_undegraded":"2024-09-10T19:12:54.777952+0000","last_fullsized":"2024-09-10T19:12:54.777952+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:37.179890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.179890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.583108+0000","scrub_duration":0.0032060080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":938,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630061+0000","last_change":"2024-09-10T19:12:47.202242+0000","last_active":"2024-09-10T19:12:54.630061+0000","last_peered":"2024-09-10T19:12:54.630061+0000","last_clean":"2024-09-10T19:12:54.630061+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:12:54.630061+0000","last_undegraded":"2024-09-10T19:12:54.630061+0000","last_fullsized":"2024-09-10T19:12:54.630061+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:47.202053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.202053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.505049+0000","scrub_duration":0.0035756780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779843+0000","last_change":"2024-09-10T19:12:51.518758+0000","last_active":"2024-09-10T19:12:54.779843+0000","last_peered":"2024-09-10T19:12:54.779843+0000","last_clean":"2024-09-10T19:12:54.779843+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:12:54.779843+0000","last_undegraded":"2024-09-10T19:12:54.779843+0000","last_fullsized":"2024-09-10T19:12:54.779843+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.518604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.518604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.107929+0000","scrub_duration":0.0034235910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":453,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776215+0000","last_change":"2024-09-10T19:12:46.748804+0000","last_active":"2024-09-10T19:12:54.776215+0000","last_peered":"2024-09-10T19:12:54.776215+0000","last_clean":"2024-09-10T19:12:54.776215+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:12:54.776215+0000","last_undegraded":"2024-09-10T19:12:54.776215+0000","last_fullsized":"2024-09-10T19:12:54.776215+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:46.748636+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.748636+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.753366+0000","scrub_duration":0.0036174229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"859'39","reported_seq":866,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:10.333726+0000","last_change":"2024-09-10T19:13:10.333638+0000","last_active":"2024-09-10T19:13:10.333726+0000","last_peered":"2024-09-10T19:13:10.333726+0000","last_clean":"2024-09-10T19:13:10.333726+0000","last_became_active":"2024-09-10T19:10:47.693090+0000","last_became_peered":"2024-09-10T19:10:47.693090+0000","last_unstale":"2024-09-10T19:13:10.333726+0000","last_undegraded":"2024-09-10T19:13:10.333726+0000","last_fullsized":"2024-09-10T19:13:10.333726+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'39","last_scrub_stamp":"2024-09-10T19:13:10.333475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:10.333475+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.237967+0000","scrub_duration":0.0055781709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.779540+0000","last_change":"2024-09-10T19:13:13.779437+0000","last_active":"2024-09-10T19:13:13.779540+0000","last_peered":"2024-09-10T19:13:13.779540+0000","last_clean":"2024-09-10T19:13:13.779540+0000","last_became_active":"2024-09-10T19:09:17.624324+0000","last_became_peered":"2024-09-10T19:09:17.624324+0000","last_unstale":"2024-09-10T19:13:13.779540+0000","last_undegraded":"2024-09-10T19:13:13.779540+0000","last_fullsized":"2024-09-10T19:13:13.779540+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.779283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:13.779283+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.779283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.997020+0000","scrub_duration":0.0033720930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":866,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:09.078299+0000","last_change":"2024-09-10T19:13:09.078185+0000","last_active":"2024-09-10T19:13:09.078299+0000","last_peered":"2024-09-10T19:13:09.078299+0000","last_clean":"2024-09-10T19:13:09.078299+0000","last_became_active":"2024-09-10T19:09:17.629427+0000","last_became_peered":"2024-09-10T19:09:17.629427+0000","last_unstale":"2024-09-10T19:13:09.078299+0000","last_undegraded":"2024-09-10T19:13:09.078299+0000","last_fullsized":"2024-09-10T19:13:09.078299+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.078046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.078046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.316720+0000","scrub_duration":0.0029229659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":927,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630257+0000","last_change":"2024-09-10T19:11:56.945174+0000","last_active":"2024-09-10T19:12:54.630257+0000","last_peered":"2024-09-10T19:12:54.630257+0000","last_clean":"2024-09-10T19:12:54.630257+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:12:54.630257+0000","last_undegraded":"2024-09-10T19:12:54.630257+0000","last_fullsized":"2024-09-10T19:12:54.630257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.945087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.945087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028862369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.025803+0000","last_change":"2024-09-10T19:13:13.025692+0000","last_active":"2024-09-10T19:13:13.025803+0000","last_peered":"2024-09-10T19:13:13.025803+0000","last_clean":"2024-09-10T19:13:13.025803+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:13:13.025803+0000","last_undegraded":"2024-09-10T19:13:13.025803+0000","last_fullsized":"2024-09-10T19:13:13.025803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.025532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:13.025532+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.025532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.233120+0000","scrub_duration":0.003487637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":1021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.559137+0000","last_change":"2024-09-10T19:13:00.558993+0000","last_active":"2024-09-10T19:13:00.559137+0000","last_peered":"2024-09-10T19:13:00.559137+0000","last_clean":"2024-09-10T19:13:00.559137+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:13:00.559137+0000","last_undegraded":"2024-09-10T19:13:00.559137+0000","last_fullsized":"2024-09-10T19:13:00.559137+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:00.558805+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.558805+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:05.158493+0000","scrub_duration":0.0033727290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"859'50","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.090155+0000","last_change":"2024-09-10T19:13:06.089934+0000","last_active":"2024-09-10T19:13:06.090155+0000","last_peered":"2024-09-10T19:13:06.090155+0000","last_clean":"2024-09-10T19:13:06.090155+0000","last_became_active":"2024-09-10T19:09:17.619352+0000","last_became_peered":"2024-09-10T19:09:17.619352+0000","last_unstale":"2024-09-10T19:13:06.090155+0000","last_undegraded":"2024-09-10T19:13:06.090155+0000","last_fullsized":"2024-09-10T19:13:06.090155+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'50","last_scrub_stamp":"2024-09-10T19:13:06.089649+0000","last_deep_scrub":"859'50","last_deep_scrub_stamp":"2024-09-10T19:11:37.999246+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.089649+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.912537+0000","scrub_duration":0.0065939689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":988,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:12.487351+0000","last_change":"2024-09-10T19:13:12.487233+0000","last_active":"2024-09-10T19:13:12.487351+0000","last_peered":"2024-09-10T19:13:12.487351+0000","last_clean":"2024-09-10T19:13:12.487351+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:13:12.487351+0000","last_undegraded":"2024-09-10T19:13:12.487351+0000","last_fullsized":"2024-09-10T19:13:12.487351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:12.487067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:13:12.487067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.969526+0000","scrub_duration":0.0039078300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":851,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.068942+0000","last_change":"2024-09-10T19:13:23.068902+0000","last_active":"2024-09-10T19:13:23.068942+0000","last_peered":"2024-09-10T19:13:23.068942+0000","last_clean":"2024-09-10T19:13:23.068942+0000","last_became_active":"2024-09-10T19:09:17.622531+0000","last_became_peered":"2024-09-10T19:09:17.622531+0000","last_unstale":"2024-09-10T19:13:23.068942+0000","last_undegraded":"2024-09-10T19:13:23.068942+0000","last_fullsized":"2024-09-10T19:13:23.068942+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.068822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.927927+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.068822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.020193+0000","scrub_duration":0.0027733609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.525310+0000","last_change":"2024-09-10T19:13:17.525229+0000","last_active":"2024-09-10T19:13:17.525310+0000","last_peered":"2024-09-10T19:13:17.525310+0000","last_clean":"2024-09-10T19:13:17.525310+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:13:17.525310+0000","last_undegraded":"2024-09-10T19:13:17.525310+0000","last_fullsized":"2024-09-10T19:13:17.525310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.525065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:01.874587+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.525065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:35.063352+0000","scrub_duration":0.0032966100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779249+0000","last_change":"2024-09-10T19:12:28.460824+0000","last_active":"2024-09-10T19:12:54.779249+0000","last_peered":"2024-09-10T19:12:54.779249+0000","last_clean":"2024-09-10T19:12:54.779249+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:12:54.779249+0000","last_undegraded":"2024-09-10T19:12:54.779249+0000","last_fullsized":"2024-09-10T19:12:54.779249+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.460668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.460668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:40.663215+0000","scrub_duration":0.003172182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":376,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.047088+0000","last_change":"2024-09-10T19:13:15.047037+0000","last_active":"2024-09-10T19:13:15.047088+0000","last_peered":"2024-09-10T19:13:15.047088+0000","last_clean":"2024-09-10T19:13:15.047088+0000","last_became_active":"2024-09-10T19:09:17.615000+0000","last_became_peered":"2024-09-10T19:09:17.615000+0000","last_unstale":"2024-09-10T19:13:15.047088+0000","last_undegraded":"2024-09-10T19:13:15.047088+0000","last_fullsized":"2024-09-10T19:13:15.047088+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:15.046929+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.046929+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:28.227610+0000","scrub_duration":0.0027016319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"859'42","reported_seq":1005,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630903+0000","last_change":"2024-09-10T19:12:52.223833+0000","last_active":"2024-09-10T19:12:54.630903+0000","last_peered":"2024-09-10T19:12:54.630903+0000","last_clean":"2024-09-10T19:12:54.630903+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:12:54.630903+0000","last_undegraded":"2024-09-10T19:12:54.630903+0000","last_fullsized":"2024-09-10T19:12:54.630903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'42","last_scrub_stamp":"2024-09-10T19:12:52.223726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.223726+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.659214+0000","scrub_duration":0.0042711950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":960,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780150+0000","last_change":"2024-09-10T19:11:57.764932+0000","last_active":"2024-09-10T19:12:54.780150+0000","last_peered":"2024-09-10T19:12:54.780150+0000","last_clean":"2024-09-10T19:12:54.780150+0000","last_became_active":"2024-09-10T19:09:17.624516+0000","last_became_peered":"2024-09-10T19:09:17.624516+0000","last_unstale":"2024-09-10T19:12:54.780150+0000","last_undegraded":"2024-09-10T19:12:54.780150+0000","last_fullsized":"2024-09-10T19:12:54.780150+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:57.764772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.764772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:24.744343+0000","scrub_duration":0.0038741769999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":950,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.632171+0000","last_change":"2024-09-10T19:12:14.075183+0000","last_active":"2024-09-10T19:12:54.632171+0000","last_peered":"2024-09-10T19:12:54.632171+0000","last_clean":"2024-09-10T19:12:54.632171+0000","last_became_active":"2024-09-10T19:09:17.624029+0000","last_became_peered":"2024-09-10T19:09:17.624029+0000","last_unstale":"2024-09-10T19:12:54.632171+0000","last_undegraded":"2024-09-10T19:12:54.632171+0000","last_fullsized":"2024-09-10T19:12:54.632171+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:14.074847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:14.074847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:32.092954+0000","scrub_duration":0.004273415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779425+0000","last_change":"2024-09-10T19:12:09.978721+0000","last_active":"2024-09-10T19:12:54.779425+0000","last_peered":"2024-09-10T19:12:54.779425+0000","last_clean":"2024-09-10T19:12:54.779425+0000","last_became_active":"2024-09-10T19:09:17.630321+0000","last_became_peered":"2024-09-10T19:09:17.630321+0000","last_unstale":"2024-09-10T19:12:54.779425+0000","last_undegraded":"2024-09-10T19:12:54.779425+0000","last_fullsized":"2024-09-10T19:12:54.779425+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:09.978565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:09.978565+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.978565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.552818+0000","scrub_duration":0.003434084,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:07.108379+0000","last_change":"2024-09-10T19:13:07.108208+0000","last_active":"2024-09-10T19:13:07.108379+0000","last_peered":"2024-09-10T19:13:07.108379+0000","last_clean":"2024-09-10T19:13:07.108379+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:13:07.108379+0000","last_undegraded":"2024-09-10T19:13:07.108379+0000","last_fullsized":"2024-09-10T19:13:07.108379+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:07.108000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:07.108000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.241782+0000","scrub_duration":0.0039466789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"859'35","reported_seq":978,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629907+0000","last_change":"2024-09-10T19:11:59.982386+0000","last_active":"2024-09-10T19:12:54.629907+0000","last_peered":"2024-09-10T19:12:54.629907+0000","last_clean":"2024-09-10T19:12:54.629907+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:12:54.629907+0000","last_undegraded":"2024-09-10T19:12:54.629907+0000","last_fullsized":"2024-09-10T19:12:54.629907+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:11:59.982219+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.982219+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050655860000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781104+0000","last_change":"2024-09-10T19:12:08.806518+0000","last_active":"2024-09-10T19:12:54.781104+0000","last_peered":"2024-09-10T19:12:54.781104+0000","last_clean":"2024-09-10T19:12:54.781104+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:12:54.781104+0000","last_undegraded":"2024-09-10T19:12:54.781104+0000","last_fullsized":"2024-09-10T19:12:54.781104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:08.806364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:12:08.806364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.487417+0000","scrub_duration":0.0032522279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":1012,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780443+0000","last_change":"2024-09-10T19:12:43.759930+0000","last_active":"2024-09-10T19:12:54.780443+0000","last_peered":"2024-09-10T19:12:54.780443+0000","last_clean":"2024-09-10T19:12:54.780443+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:12:54.780443+0000","last_undegraded":"2024-09-10T19:12:54.780443+0000","last_fullsized":"2024-09-10T19:12:54.780443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:43.759757+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:09:58.722119+0000","last_clean_scrub_stamp":"2024-09-10T19:12:43.759757+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:59.816579+0000","scrub_duration":0.0038125199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":941,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778523+0000","last_change":"2024-09-10T19:12:43.440980+0000","last_active":"2024-09-10T19:12:54.778523+0000","last_peered":"2024-09-10T19:12:54.778523+0000","last_clean":"2024-09-10T19:12:54.778523+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:12:54.778523+0000","last_undegraded":"2024-09-10T19:12:54.778523+0000","last_fullsized":"2024-09-10T19:12:54.778523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:43.440817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:12:43.440817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:09.757706+0000","scrub_duration":0.003326421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.782124+0000","last_change":"2024-09-10T19:12:38.687371+0000","last_active":"2024-09-10T19:12:54.782124+0000","last_peered":"2024-09-10T19:12:54.782124+0000","last_clean":"2024-09-10T19:12:54.782124+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:12:54.782124+0000","last_undegraded":"2024-09-10T19:12:54.782124+0000","last_fullsized":"2024-09-10T19:12:54.782124+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:38.687215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_clean_scrub_stamp":"2024-09-10T19:12:38.687215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:07.271979+0000","scrub_duration":0.0033182039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":888,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.489558+0000","last_change":"2024-09-10T19:13:15.489477+0000","last_active":"2024-09-10T19:13:15.489558+0000","last_peered":"2024-09-10T19:13:15.489558+0000","last_clean":"2024-09-10T19:13:15.489558+0000","last_became_active":"2024-09-10T19:09:17.624085+0000","last_became_peered":"2024-09-10T19:09:17.624085+0000","last_unstale":"2024-09-10T19:13:15.489558+0000","last_undegraded":"2024-09-10T19:13:15.489558+0000","last_fullsized":"2024-09-10T19:13:15.489558+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:02.895391+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.489308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.774790+0000","scrub_duration":0.0038349460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"859'53","reported_seq":1029,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:14.472225+0000","last_change":"2024-09-10T19:13:14.472143+0000","last_active":"2024-09-10T19:13:14.472225+0000","last_peered":"2024-09-10T19:13:14.472225+0000","last_clean":"2024-09-10T19:13:14.472225+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:13:14.472225+0000","last_undegraded":"2024-09-10T19:13:14.472225+0000","last_fullsized":"2024-09-10T19:13:14.472225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'53","last_scrub_stamp":"2024-09-10T19:13:14.471973+0000","last_deep_scrub":"859'53","last_deep_scrub_stamp":"2024-09-10T19:13:14.471973+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.471973+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.038006+0000","scrub_duration":0.0061128839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":988,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:19.459010+0000","last_change":"2024-09-10T19:13:19.458902+0000","last_active":"2024-09-10T19:13:19.459010+0000","last_peered":"2024-09-10T19:13:19.459010+0000","last_clean":"2024-09-10T19:13:19.459010+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:13:19.459010+0000","last_undegraded":"2024-09-10T19:13:19.459010+0000","last_fullsized":"2024-09-10T19:13:19.459010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:19.458638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:19.458638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.204429+0000","scrub_duration":0.0035443229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":958,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778689+0000","last_change":"2024-09-10T19:12:22.435931+0000","last_active":"2024-09-10T19:12:54.778689+0000","last_peered":"2024-09-10T19:12:54.778689+0000","last_clean":"2024-09-10T19:12:54.778689+0000","last_became_active":"2024-09-10T19:09:17.613101+0000","last_became_peered":"2024-09-10T19:09:17.613101+0000","last_unstale":"2024-09-10T19:12:54.778689+0000","last_undegraded":"2024-09-10T19:12:54.778689+0000","last_fullsized":"2024-09-10T19:12:54.778689+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:22.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:22.435765+0000","last_clean_scrub_stamp":"2024-09-10T19:12:22.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:31.022433+0000","scrub_duration":0.0037076259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:03.524244+0000","last_change":"2024-09-10T19:13:03.524074+0000","last_active":"2024-09-10T19:13:03.524244+0000","last_peered":"2024-09-10T19:13:03.524244+0000","last_clean":"2024-09-10T19:13:03.524244+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:13:03.524244+0000","last_undegraded":"2024-09-10T19:13:03.524244+0000","last_fullsized":"2024-09-10T19:13:03.524244+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:03.523914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:03.523914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.969934+0000","scrub_duration":0.0043000859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":889,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:18.290450+0000","last_change":"2024-09-10T19:13:18.290372+0000","last_active":"2024-09-10T19:13:18.290450+0000","last_peered":"2024-09-10T19:13:18.290450+0000","last_clean":"2024-09-10T19:13:18.290450+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:13:18.290450+0000","last_undegraded":"2024-09-10T19:13:18.290450+0000","last_fullsized":"2024-09-10T19:13:18.290450+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:18.290186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:18.290186+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.290186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:44.810909+0000","scrub_duration":0.0031641389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"859'35","reported_seq":994,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781853+0000","last_change":"2024-09-10T19:12:40.695724+0000","last_active":"2024-09-10T19:12:54.781853+0000","last_peered":"2024-09-10T19:12:54.781853+0000","last_clean":"2024-09-10T19:12:54.781853+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:12:54.781853+0000","last_undegraded":"2024-09-10T19:12:54.781853+0000","last_fullsized":"2024-09-10T19:12:54.781853+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:12:40.695561+0000","last_deep_scrub":"859'35","last_deep_scrub_stamp":"2024-09-10T19:11:08.057014+0000","last_clean_scrub_stamp":"2024-09-10T19:12:40.695561+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.662645+0000","scrub_duration":0.0045341169999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":890,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:21.090862+0000","last_change":"2024-09-10T19:13:21.090793+0000","last_active":"2024-09-10T19:13:21.090862+0000","last_peered":"2024-09-10T19:13:21.090862+0000","last_clean":"2024-09-10T19:13:21.090862+0000","last_became_active":"2024-09-10T19:09:17.627724+0000","last_became_peered":"2024-09-10T19:09:17.627724+0000","last_unstale":"2024-09-10T19:13:21.090862+0000","last_undegraded":"2024-09-10T19:13:21.090862+0000","last_fullsized":"2024-09-10T19:13:21.090862+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:21.090668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:21.090668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:38.110398+0000","scrub_duration":0.0033861249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631241+0000","last_change":"2024-09-10T19:12:31.074289+0000","last_active":"2024-09-10T19:12:54.631241+0000","last_peered":"2024-09-10T19:12:54.631241+0000","last_clean":"2024-09-10T19:12:54.631241+0000","last_became_active":"2024-09-10T19:09:17.623063+0000","last_became_peered":"2024-09-10T19:09:17.623063+0000","last_unstale":"2024-09-10T19:12:54.631241+0000","last_undegraded":"2024-09-10T19:12:54.631241+0000","last_fullsized":"2024-09-10T19:12:54.631241+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:31.074104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:31.074104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:34.686834+0000","scrub_duration":0.003352233,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":928,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630831+0000","last_change":"2024-09-10T19:12:51.206710+0000","last_active":"2024-09-10T19:12:54.630831+0000","last_peered":"2024-09-10T19:12:54.630831+0000","last_clean":"2024-09-10T19:12:54.630831+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:12:54.630831+0000","last_undegraded":"2024-09-10T19:12:54.630831+0000","last_fullsized":"2024-09-10T19:12:54.630831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.206565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:07.952993+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.206565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.323393+0000","scrub_duration":0.0033822589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:01.166984+0000","last_change":"2024-09-10T19:13:01.166809+0000","last_active":"2024-09-10T19:13:01.166984+0000","last_peered":"2024-09-10T19:13:01.166984+0000","last_clean":"2024-09-10T19:13:01.166984+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:13:01.166984+0000","last_undegraded":"2024-09-10T19:13:01.166984+0000","last_fullsized":"2024-09-10T19:13:01.166984+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.166559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:01.166559+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.166559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.234005+0000","scrub_duration":0.0035870870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"859'46","reported_seq":1019,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.783310+0000","last_change":"2024-09-10T19:12:47.831330+0000","last_active":"2024-09-10T19:12:54.783310+0000","last_peered":"2024-09-10T19:12:54.783310+0000","last_clean":"2024-09-10T19:12:54.783310+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:12:54.783310+0000","last_undegraded":"2024-09-10T19:12:54.783310+0000","last_fullsized":"2024-09-10T19:12:54.783310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:12:47.831200+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.831200+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.699467+0000","scrub_duration":0.0089948319999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":990,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781070+0000","last_change":"2024-09-10T19:12:20.256789+0000","last_active":"2024-09-10T19:12:54.781070+0000","last_peered":"2024-09-10T19:12:54.781070+0000","last_clean":"2024-09-10T19:12:54.781070+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:12:54.781070+0000","last_undegraded":"2024-09-10T19:12:54.781070+0000","last_fullsized":"2024-09-10T19:12:54.781070+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:12:20.256589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:20.256589+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.628673+0000","scrub_duration":0.0040982650000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":942,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776243+0000","last_change":"2024-09-10T19:12:04.857069+0000","last_active":"2024-09-10T19:12:54.776243+0000","last_peered":"2024-09-10T19:12:54.776243+0000","last_clean":"2024-09-10T19:12:54.776243+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:12:54.776243+0000","last_undegraded":"2024-09-10T19:12:54.776243+0000","last_fullsized":"2024-09-10T19:12:54.776243+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:04.856902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.856902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.137273+0000","scrub_duration":0.0034979120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780469+0000","last_change":"2024-09-10T19:12:31.732013+0000","last_active":"2024-09-10T19:12:54.780469+0000","last_peered":"2024-09-10T19:12:54.780469+0000","last_clean":"2024-09-10T19:12:54.780469+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:12:54.780469+0000","last_undegraded":"2024-09-10T19:12:54.780469+0000","last_fullsized":"2024-09-10T19:12:54.780469+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:31.731888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:31.731888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:39.391093+0000","scrub_duration":0.0027531270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":513,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.626750+0000","last_change":"2024-09-10T19:12:31.534516+0000","last_active":"2024-09-10T19:12:54.626750+0000","last_peered":"2024-09-10T19:12:54.626750+0000","last_clean":"2024-09-10T19:12:54.626750+0000","last_became_active":"2024-09-10T19:10:47.689638+0000","last_became_peered":"2024-09-10T19:10:47.689638+0000","last_unstale":"2024-09-10T19:12:54.626750+0000","last_undegraded":"2024-09-10T19:12:54.626750+0000","last_fullsized":"2024-09-10T19:12:54.626750+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:31.534346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:12.218396+0000","last_clean_scrub_stamp":"2024-09-10T19:12:31.534346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.026784+0000","scrub_duration":0.003248552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"901'32","reported_seq":1009,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.425730+0000","last_change":"2024-09-10T19:13:23.425654+0000","last_active":"2024-09-10T19:13:23.425730+0000","last_peered":"2024-09-10T19:13:23.425730+0000","last_clean":"2024-09-10T19:13:23.425730+0000","last_became_active":"2024-09-10T19:09:17.633035+0000","last_became_peered":"2024-09-10T19:09:17.633035+0000","last_unstale":"2024-09-10T19:13:23.425730+0000","last_undegraded":"2024-09-10T19:13:23.425730+0000","last_fullsized":"2024-09-10T19:13:23.425730+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'32","last_scrub_stamp":"2024-09-10T19:13:23.425467+0000","last_deep_scrub":"901'32","last_deep_scrub_stamp":"2024-09-10T19:13:23.425467+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.425467+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.478631+0000","scrub_duration":0.011078439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"601'4","reported_seq":1015,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629697+0000","last_change":"2024-09-10T19:12:36.417556+0000","last_active":"2024-09-10T19:12:54.629697+0000","last_peered":"2024-09-10T19:12:54.629697+0000","last_clean":"2024-09-10T19:12:54.629697+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:12:54.629697+0000","last_undegraded":"2024-09-10T19:12:54.629697+0000","last_fullsized":"2024-09-10T19:12:54.629697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'4","last_scrub_stamp":"2024-09-10T19:12:36.417396+0000","last_deep_scrub":"601'4","last_deep_scrub_stamp":"2024-09-10T19:04:04.923921+0000","last_clean_scrub_stamp":"2024-09-10T19:12:36.417396+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.972382+0000","scrub_duration":0.004415385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"879'15","reported_seq":799,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.915247+0000","last_change":"2024-09-10T19:12:37.463471+0000","last_active":"2024-09-10T19:13:16.915247+0000","last_peered":"2024-09-10T19:13:16.915247+0000","last_clean":"2024-09-10T19:13:16.915247+0000","last_became_active":"2024-09-10T19:10:47.694903+0000","last_became_peered":"2024-09-10T19:10:47.694903+0000","last_unstale":"2024-09-10T19:13:16.915247+0000","last_undegraded":"2024-09-10T19:13:16.915247+0000","last_fullsized":"2024-09-10T19:13:16.915247+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'15","last_scrub_stamp":"2024-09-10T19:12:37.463308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.463308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:41.922827+0000","scrub_duration":0.0038917520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":951,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:56.960638+0000","last_change":"2024-09-10T19:12:56.960511+0000","last_active":"2024-09-10T19:12:56.960638+0000","last_peered":"2024-09-10T19:12:56.960638+0000","last_clean":"2024-09-10T19:12:56.960638+0000","last_became_active":"2024-09-10T19:09:17.620179+0000","last_became_peered":"2024-09-10T19:09:17.620179+0000","last_unstale":"2024-09-10T19:12:56.960638+0000","last_undegraded":"2024-09-10T19:12:56.960638+0000","last_fullsized":"2024-09-10T19:12:56.960638+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.960359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.960359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.915896+0000","scrub_duration":0.0036065630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":875,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778854+0000","last_change":"2024-09-10T19:12:11.533250+0000","last_active":"2024-09-10T19:12:54.778854+0000","last_peered":"2024-09-10T19:12:54.778854+0000","last_clean":"2024-09-10T19:12:54.778854+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:12:54.778854+0000","last_undegraded":"2024-09-10T19:12:54.778854+0000","last_fullsized":"2024-09-10T19:12:54.778854+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:12:11.533087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.533087+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.792717+0000","scrub_duration":0.0039834850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"859'27","reported_seq":994,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778190+0000","last_change":"2024-09-10T19:12:11.900879+0000","last_active":"2024-09-10T19:12:54.778190+0000","last_peered":"2024-09-10T19:12:54.778190+0000","last_clean":"2024-09-10T19:12:54.778190+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:12:54.778190+0000","last_undegraded":"2024-09-10T19:12:54.778190+0000","last_fullsized":"2024-09-10T19:12:54.778190+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'27","last_scrub_stamp":"2024-09-10T19:12:11.900712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.900712+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:31.032241+0000","scrub_duration":0.0050889890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":995,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.527959+0000","last_change":"2024-09-10T19:13:23.527868+0000","last_active":"2024-09-10T19:13:23.527959+0000","last_peered":"2024-09-10T19:13:23.527959+0000","last_clean":"2024-09-10T19:13:23.527959+0000","last_became_active":"2024-09-10T19:09:17.615024+0000","last_became_peered":"2024-09-10T19:09:17.615024+0000","last_unstale":"2024-09-10T19:13:23.527959+0000","last_undegraded":"2024-09-10T19:13:23.527959+0000","last_fullsized":"2024-09-10T19:13:23.527959+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.527711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.527711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.875509+0000","scrub_duration":0.0032316559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.165992+0000","last_change":"2024-09-10T19:12:55.165889+0000","last_active":"2024-09-10T19:12:55.165992+0000","last_peered":"2024-09-10T19:12:55.165992+0000","last_clean":"2024-09-10T19:12:55.165992+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:12:55.165992+0000","last_undegraded":"2024-09-10T19:12:55.165992+0000","last_fullsized":"2024-09-10T19:12:55.165992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.165750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.165750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.773616+0000","scrub_duration":0.0029434719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":972,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:22.549558+0000","last_change":"2024-09-10T19:13:22.549477+0000","last_active":"2024-09-10T19:13:22.549558+0000","last_peered":"2024-09-10T19:13:22.549558+0000","last_clean":"2024-09-10T19:13:22.549558+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:13:22.549558+0000","last_undegraded":"2024-09-10T19:13:22.549558+0000","last_fullsized":"2024-09-10T19:13:22.549558+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:22.549324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:19.223584+0000","last_clean_scrub_stamp":"2024-09-10T19:13:22.549324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.638964+0000","scrub_duration":0.0038758019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629437+0000","last_change":"2024-09-10T19:12:04.462494+0000","last_active":"2024-09-10T19:12:54.629437+0000","last_peered":"2024-09-10T19:12:54.629437+0000","last_clean":"2024-09-10T19:12:54.629437+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:12:54.629437+0000","last_undegraded":"2024-09-10T19:12:54.629437+0000","last_fullsized":"2024-09-10T19:12:54.629437+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:04.462331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:04.462331+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.462331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.141660+0000","scrub_duration":0.003556937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"859'14","reported_seq":963,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777271+0000","last_change":"2024-09-10T19:11:57.631685+0000","last_active":"2024-09-10T19:12:54.777271+0000","last_peered":"2024-09-10T19:12:54.777271+0000","last_clean":"2024-09-10T19:12:54.777271+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:12:54.777271+0000","last_undegraded":"2024-09-10T19:12:54.777271+0000","last_fullsized":"2024-09-10T19:12:54.777271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'14","last_scrub_stamp":"2024-09-10T19:11:57.631561+0000","last_deep_scrub":"859'14","last_deep_scrub_stamp":"2024-09-10T19:10:25.016589+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.631561+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:26.509894+0000","scrub_duration":0.0040774849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":962,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779937+0000","last_change":"2024-09-10T19:12:20.436572+0000","last_active":"2024-09-10T19:12:54.779937+0000","last_peered":"2024-09-10T19:12:54.779937+0000","last_clean":"2024-09-10T19:12:54.779937+0000","last_became_active":"2024-09-10T19:09:17.614809+0000","last_became_peered":"2024-09-10T19:09:17.614809+0000","last_unstale":"2024-09-10T19:12:54.779937+0000","last_undegraded":"2024-09-10T19:12:54.779937+0000","last_fullsized":"2024-09-10T19:12:54.779937+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:20.436322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:20.436322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:47.028739+0000","scrub_duration":0.0038737590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.947899+0000","last_change":"2024-09-10T19:13:05.947765+0000","last_active":"2024-09-10T19:13:05.947899+0000","last_peered":"2024-09-10T19:13:05.947899+0000","last_clean":"2024-09-10T19:13:05.947899+0000","last_became_active":"2024-09-10T19:09:17.622794+0000","last_became_peered":"2024-09-10T19:09:17.622794+0000","last_unstale":"2024-09-10T19:13:05.947899+0000","last_undegraded":"2024-09-10T19:13:05.947899+0000","last_fullsized":"2024-09-10T19:13:05.947899+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.947610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.947610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.828022+0000","scrub_duration":0.0033327069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.782642+0000","last_change":"2024-09-10T19:12:07.846572+0000","last_active":"2024-09-10T19:12:54.782642+0000","last_peered":"2024-09-10T19:12:54.782642+0000","last_clean":"2024-09-10T19:12:54.782642+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:12:54.782642+0000","last_undegraded":"2024-09-10T19:12:54.782642+0000","last_fullsized":"2024-09-10T19:12:54.782642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:07.845968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:07.845968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:22.138786+0000","scrub_duration":0.0063041770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779023+0000","last_change":"2024-09-10T19:12:18.425215+0000","last_active":"2024-09-10T19:12:54.779023+0000","last_peered":"2024-09-10T19:12:54.779023+0000","last_clean":"2024-09-10T19:12:54.779023+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:12:54.779023+0000","last_undegraded":"2024-09-10T19:12:54.779023+0000","last_fullsized":"2024-09-10T19:12:54.779023+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.425043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.425043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.366833+0000","scrub_duration":0.0033576000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"859'32","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780079+0000","last_change":"2024-09-10T19:12:08.831808+0000","last_active":"2024-09-10T19:12:54.780079+0000","last_peered":"2024-09-10T19:12:54.780079+0000","last_clean":"2024-09-10T19:12:54.780079+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:12:54.780079+0000","last_undegraded":"2024-09-10T19:12:54.780079+0000","last_fullsized":"2024-09-10T19:12:54.780079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'32","last_scrub_stamp":"2024-09-10T19:12:08.831642+0000","last_deep_scrub":"859'32","last_deep_scrub_stamp":"2024-09-10T19:12:08.831642+0000","last_clean_scrub_stamp":"2024-09-10T19:12:08.831642+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.338553+0000","scrub_duration":0.0052310309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":939,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781058+0000","last_change":"2024-09-10T19:11:57.860845+0000","last_active":"2024-09-10T19:12:54.781058+0000","last_peered":"2024-09-10T19:12:54.781058+0000","last_clean":"2024-09-10T19:12:54.781058+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:12:54.781058+0000","last_undegraded":"2024-09-10T19:12:54.781058+0000","last_fullsized":"2024-09-10T19:12:54.781058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:57.860693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.860693+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.860693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:23.608479+0000","scrub_duration":0.0033432570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"856'14","reported_seq":1258,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:18.839703+0000","last_change":"2024-09-10T19:13:18.839483+0000","last_active":"2024-09-10T19:13:18.839703+0000","last_peered":"2024-09-10T19:13:18.839703+0000","last_clean":"2024-09-10T19:13:18.839703+0000","last_became_active":"2024-09-10T19:09:17.630818+0000","last_became_peered":"2024-09-10T19:09:17.630818+0000","last_unstale":"2024-09-10T19:13:18.839703+0000","last_undegraded":"2024-09-10T19:13:18.839703+0000","last_fullsized":"2024-09-10T19:13:18.839703+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'14","last_scrub_stamp":"2024-09-10T19:13:18.839267+0000","last_deep_scrub":"856'14","last_deep_scrub_stamp":"2024-09-10T19:13:18.839267+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.839267+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.300397+0000","scrub_duration":0.0037778780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":938,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.164056+0000","last_change":"2024-09-10T19:13:00.163996+0000","last_active":"2024-09-10T19:13:00.164056+0000","last_peered":"2024-09-10T19:13:00.164056+0000","last_clean":"2024-09-10T19:13:00.164056+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:13:00.164056+0000","last_undegraded":"2024-09-10T19:13:00.164056+0000","last_fullsized":"2024-09-10T19:13:00.164056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.163904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:00.163904+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.163904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.326070+0000","scrub_duration":0.003138348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:09.048361+0000","last_change":"2024-09-10T19:13:09.048324+0000","last_active":"2024-09-10T19:13:09.048361+0000","last_peered":"2024-09-10T19:13:09.048361+0000","last_clean":"2024-09-10T19:13:09.048361+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:13:09.048361+0000","last_undegraded":"2024-09-10T19:13:09.048361+0000","last_fullsized":"2024-09-10T19:13:09.048361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.048254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:31.597446+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.048254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.209263+0000","scrub_duration":0.0021055919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"859'21","reported_seq":976,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631249+0000","last_change":"2024-09-10T19:12:54.186725+0000","last_active":"2024-09-10T19:12:54.631249+0000","last_peered":"2024-09-10T19:12:54.631249+0000","last_clean":"2024-09-10T19:12:54.631249+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:12:54.631249+0000","last_undegraded":"2024-09-10T19:12:54.631249+0000","last_fullsized":"2024-09-10T19:12:54.631249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:12:54.186556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.186556+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.956911+0000","scrub_duration":0.0047097140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631603+0000","last_change":"2024-09-10T19:12:42.094004+0000","last_active":"2024-09-10T19:12:54.631603+0000","last_peered":"2024-09-10T19:12:54.631603+0000","last_clean":"2024-09-10T19:12:54.631603+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:12:54.631603+0000","last_undegraded":"2024-09-10T19:12:54.631603+0000","last_fullsized":"2024-09-10T19:12:54.631603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:42.093876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:46.308759+0000","last_clean_scrub_stamp":"2024-09-10T19:12:42.093876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.092846+0000","scrub_duration":0.002962592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":1231,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:19.697910+0000","last_change":"2024-09-10T19:13:19.697733+0000","last_active":"2024-09-10T19:13:19.697910+0000","last_peered":"2024-09-10T19:13:19.697910+0000","last_clean":"2024-09-10T19:13:19.697910+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:13:19.697910+0000","last_undegraded":"2024-09-10T19:13:19.697910+0000","last_fullsized":"2024-09-10T19:13:19.697910+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:13:19.697499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:19.697499+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:33.844040+0000","scrub_duration":0.0051654370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":592,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.311660+0000","last_change":"2024-09-10T19:13:17.311578+0000","last_active":"2024-09-10T19:13:17.311660+0000","last_peered":"2024-09-10T19:13:17.311660+0000","last_clean":"2024-09-10T19:13:17.311660+0000","last_became_active":"2024-09-10T19:10:47.684527+0000","last_became_peered":"2024-09-10T19:10:47.684527+0000","last_unstale":"2024-09-10T19:13:17.311660+0000","last_undegraded":"2024-09-10T19:13:17.311660+0000","last_fullsized":"2024-09-10T19:13:17.311660+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.311423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:36.878367+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.311423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:39.495046+0000","scrub_duration":0.0032662540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631818+0000","last_change":"2024-09-10T19:12:33.108017+0000","last_active":"2024-09-10T19:12:54.631818+0000","last_peered":"2024-09-10T19:12:54.631818+0000","last_clean":"2024-09-10T19:12:54.631818+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:12:54.631818+0000","last_undegraded":"2024-09-10T19:12:54.631818+0000","last_fullsized":"2024-09-10T19:12:54.631818+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:33.107854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:33.107854+0000","last_clean_scrub_stamp":"2024-09-10T19:12:33.107854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:52.672454+0000","scrub_duration":0.0038379249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"859'21","reported_seq":995,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.443450+0000","last_change":"2024-09-10T19:13:20.443309+0000","last_active":"2024-09-10T19:13:20.443450+0000","last_peered":"2024-09-10T19:13:20.443450+0000","last_clean":"2024-09-10T19:13:20.443450+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:13:20.443450+0000","last_undegraded":"2024-09-10T19:13:20.443450+0000","last_fullsized":"2024-09-10T19:13:20.443450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:20.443121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.443121+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.371252+0000","scrub_duration":0.0042781690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778298+0000","last_change":"2024-09-10T19:12:45.436062+0000","last_active":"2024-09-10T19:12:54.778298+0000","last_peered":"2024-09-10T19:12:54.778298+0000","last_clean":"2024-09-10T19:12:54.778298+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:12:54.778298+0000","last_undegraded":"2024-09-10T19:12:54.778298+0000","last_fullsized":"2024-09-10T19:12:54.778298+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.435891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.435891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.633103+0000","scrub_duration":0.0037274909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778988+0000","last_change":"2024-09-10T19:12:26.850598+0000","last_active":"2024-09-10T19:12:54.778988+0000","last_peered":"2024-09-10T19:12:54.778988+0000","last_clean":"2024-09-10T19:12:54.778988+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:12:54.778988+0000","last_undegraded":"2024-09-10T19:12:54.778988+0000","last_fullsized":"2024-09-10T19:12:54.778988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.850447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.850447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:28.973815+0000","scrub_duration":0.0032896179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":958,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:03.161227+0000","last_change":"2024-09-10T19:13:03.161153+0000","last_active":"2024-09-10T19:13:03.161227+0000","last_peered":"2024-09-10T19:13:03.161227+0000","last_clean":"2024-09-10T19:13:03.161227+0000","last_became_active":"2024-09-10T19:09:17.624937+0000","last_became_peered":"2024-09-10T19:09:17.624937+0000","last_unstale":"2024-09-10T19:13:03.161227+0000","last_undegraded":"2024-09-10T19:13:03.161227+0000","last_fullsized":"2024-09-10T19:13:03.161227+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:03.161037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:13:03.161037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.924458+0000","scrub_duration":0.0037071690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:58.393290+0000","last_change":"2024-09-10T19:12:58.393174+0000","last_active":"2024-09-10T19:12:58.393290+0000","last_peered":"2024-09-10T19:12:58.393290+0000","last_clean":"2024-09-10T19:12:58.393290+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:12:58.393290+0000","last_undegraded":"2024-09-10T19:12:58.393290+0000","last_fullsized":"2024-09-10T19:12:58.393290+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.393011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:58.393011+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.393011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.466325+0000","scrub_duration":0.003042051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"859'35","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:59.394453+0000","last_change":"2024-09-10T19:12:59.394320+0000","last_active":"2024-09-10T19:12:59.394453+0000","last_peered":"2024-09-10T19:12:59.394453+0000","last_clean":"2024-09-10T19:12:59.394453+0000","last_became_active":"2024-09-10T19:10:47.697814+0000","last_became_peered":"2024-09-10T19:10:47.697814+0000","last_unstale":"2024-09-10T19:12:59.394453+0000","last_undegraded":"2024-09-10T19:12:59.394453+0000","last_fullsized":"2024-09-10T19:12:59.394453+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:12:59.394122+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.394122+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.412944+0000","scrub_duration":0.004757173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.059524+0000","last_change":"2024-09-10T19:13:00.059400+0000","last_active":"2024-09-10T19:13:00.059524+0000","last_peered":"2024-09-10T19:13:00.059524+0000","last_clean":"2024-09-10T19:13:00.059524+0000","last_became_active":"2024-09-10T19:09:17.622828+0000","last_became_peered":"2024-09-10T19:09:17.622828+0000","last_unstale":"2024-09-10T19:13:00.059524+0000","last_undegraded":"2024-09-10T19:13:00.059524+0000","last_fullsized":"2024-09-10T19:13:00.059524+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.059254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.059254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.787520+0000","scrub_duration":0.003501016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":977,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:57.382246+0000","last_change":"2024-09-10T19:12:57.382119+0000","last_active":"2024-09-10T19:12:57.382246+0000","last_peered":"2024-09-10T19:12:57.382246+0000","last_clean":"2024-09-10T19:12:57.382246+0000","last_became_active":"2024-09-10T19:09:17.613499+0000","last_became_peered":"2024-09-10T19:09:17.613499+0000","last_unstale":"2024-09-10T19:12:57.382246+0000","last_undegraded":"2024-09-10T19:12:57.382246+0000","last_fullsized":"2024-09-10T19:12:57.382246+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.381964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:57.381964+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.381964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.527513+0000","scrub_duration":0.00331124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":937,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628566+0000","last_change":"2024-09-10T19:12:01.945608+0000","last_active":"2024-09-10T19:12:54.628566+0000","last_peered":"2024-09-10T19:12:54.628566+0000","last_clean":"2024-09-10T19:12:54.628566+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:12:54.628566+0000","last_undegraded":"2024-09-10T19:12:54.628566+0000","last_fullsized":"2024-09-10T19:12:54.628566+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:01.945455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:17.297932+0000","last_clean_scrub_stamp":"2024-09-10T19:12:01.945455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.898125+0000","scrub_duration":0.003361845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":513,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627659+0000","last_change":"2024-09-10T19:12:26.534868+0000","last_active":"2024-09-10T19:12:54.627659+0000","last_peered":"2024-09-10T19:12:54.627659+0000","last_clean":"2024-09-10T19:12:54.627659+0000","last_became_active":"2024-09-10T19:10:47.695555+0000","last_became_peered":"2024-09-10T19:10:47.695555+0000","last_unstale":"2024-09-10T19:12:54.627659+0000","last_undegraded":"2024-09-10T19:12:54.627659+0000","last_fullsized":"2024-09-10T19:12:54.627659+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.534630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:42.654425+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.534630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.975141+0000","scrub_duration":0.003960842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"859'21","reported_seq":972,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.035350+0000","last_change":"2024-09-10T19:13:17.035260+0000","last_active":"2024-09-10T19:13:17.035350+0000","last_peered":"2024-09-10T19:13:17.035350+0000","last_clean":"2024-09-10T19:13:17.035350+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:13:17.035350+0000","last_undegraded":"2024-09-10T19:13:17.035350+0000","last_fullsized":"2024-09-10T19:13:17.035350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:17.035097+0000","last_deep_scrub":"410'15","last_deep_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.035097+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.333276+0000","scrub_duration":0.0048530689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":946,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.730333+0000","last_change":"2024-09-10T19:13:15.730201+0000","last_active":"2024-09-10T19:13:15.730333+0000","last_peered":"2024-09-10T19:13:15.730333+0000","last_clean":"2024-09-10T19:13:15.730333+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:13:15.730333+0000","last_undegraded":"2024-09-10T19:13:15.730333+0000","last_fullsized":"2024-09-10T19:13:15.730333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.729942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.799168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.729942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.722915+0000","scrub_duration":0.0044025180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.033799+0000","last_change":"2024-09-10T19:12:55.033696+0000","last_active":"2024-09-10T19:12:55.033799+0000","last_peered":"2024-09-10T19:12:55.033799+0000","last_clean":"2024-09-10T19:12:55.033799+0000","last_became_active":"2024-09-10T19:09:17.624214+0000","last_became_peered":"2024-09-10T19:09:17.624214+0000","last_unstale":"2024-09-10T19:12:55.033799+0000","last_undegraded":"2024-09-10T19:12:55.033799+0000","last_fullsized":"2024-09-10T19:12:55.033799+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.033547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.033547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:59.195451+0000","scrub_duration":0.0035322420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778816+0000","last_change":"2024-09-10T19:12:29.751735+0000","last_active":"2024-09-10T19:12:54.778816+0000","last_peered":"2024-09-10T19:12:54.778816+0000","last_clean":"2024-09-10T19:12:54.778816+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:12:54.778816+0000","last_undegraded":"2024-09-10T19:12:54.778816+0000","last_fullsized":"2024-09-10T19:12:54.778816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:29.751545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.751545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.935489+0000","scrub_duration":0.0033452339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":767,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.046899+0000","last_change":"2024-09-10T19:13:05.046783+0000","last_active":"2024-09-10T19:13:05.046899+0000","last_peered":"2024-09-10T19:13:05.046899+0000","last_clean":"2024-09-10T19:13:05.046899+0000","last_became_active":"2024-09-10T19:09:17.627510+0000","last_became_peered":"2024-09-10T19:09:17.627510+0000","last_unstale":"2024-09-10T19:13:05.046899+0000","last_undegraded":"2024-09-10T19:13:05.046899+0000","last_fullsized":"2024-09-10T19:13:05.046899+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.046623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:14.886297+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.046623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.429597+0000","scrub_duration":0.0034561169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"859'25","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781288+0000","last_change":"2024-09-10T19:12:33.679136+0000","last_active":"2024-09-10T19:12:54.781288+0000","last_peered":"2024-09-10T19:12:54.781288+0000","last_clean":"2024-09-10T19:12:54.781288+0000","last_became_active":"2024-09-10T19:09:17.623786+0000","last_became_peered":"2024-09-10T19:09:17.623786+0000","last_unstale":"2024-09-10T19:12:54.781288+0000","last_undegraded":"2024-09-10T19:12:54.781288+0000","last_fullsized":"2024-09-10T19:12:54.781288+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'25","last_scrub_stamp":"2024-09-10T19:12:33.678963+0000","last_deep_scrub":"676'19","last_deep_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_clean_scrub_stamp":"2024-09-10T19:12:33.678963+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.728656+0000","scrub_duration":0.0049950280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780089+0000","last_change":"2024-09-10T19:12:30.479805+0000","last_active":"2024-09-10T19:12:54.780089+0000","last_peered":"2024-09-10T19:12:54.780089+0000","last_clean":"2024-09-10T19:12:54.780089+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:12:54.780089+0000","last_undegraded":"2024-09-10T19:12:54.780089+0000","last_fullsized":"2024-09-10T19:12:54.780089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:30.479648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:30.479648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.977977+0000","scrub_duration":0.0032843109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":926,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628680+0000","last_change":"2024-09-10T19:11:53.887313+0000","last_active":"2024-09-10T19:12:54.628680+0000","last_peered":"2024-09-10T19:12:54.628680+0000","last_clean":"2024-09-10T19:12:54.628680+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:12:54.628680+0000","last_undegraded":"2024-09-10T19:12:54.628680+0000","last_fullsized":"2024-09-10T19:12:54.628680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.887159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.887159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:14.288677+0000","scrub_duration":0.0039200759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":924,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628737+0000","last_change":"2024-09-10T19:11:43.978746+0000","last_active":"2024-09-10T19:12:54.628737+0000","last_peered":"2024-09-10T19:12:54.628737+0000","last_clean":"2024-09-10T19:12:54.628737+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:12:54.628737+0000","last_undegraded":"2024-09-10T19:12:54.628737+0000","last_fullsized":"2024-09-10T19:12:54.628737+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.978628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:54.676361+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.978628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028283230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:18.969644+0000","last_change":"2024-09-10T19:13:18.969563+0000","last_active":"2024-09-10T19:13:18.969644+0000","last_peered":"2024-09-10T19:13:18.969644+0000","last_clean":"2024-09-10T19:13:18.969644+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:13:18.969644+0000","last_undegraded":"2024-09-10T19:13:18.969644+0000","last_fullsized":"2024-09-10T19:13:18.969644+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:18.969409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.969409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.167954+0000","scrub_duration":0.0034001330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.567668+0000","last_change":"2024-09-10T19:13:06.567544+0000","last_active":"2024-09-10T19:13:06.567668+0000","last_peered":"2024-09-10T19:13:06.567668+0000","last_clean":"2024-09-10T19:13:06.567668+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:13:06.567668+0000","last_undegraded":"2024-09-10T19:13:06.567668+0000","last_fullsized":"2024-09-10T19:13:06.567668+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:13:06.567370+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.567370+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.454237+0000","scrub_duration":0.0043217389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"859'46","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.073180+0000","last_change":"2024-09-10T19:13:04.072985+0000","last_active":"2024-09-10T19:13:04.073180+0000","last_peered":"2024-09-10T19:13:04.073180+0000","last_clean":"2024-09-10T19:13:04.073180+0000","last_became_active":"2024-09-10T19:09:17.626306+0000","last_became_peered":"2024-09-10T19:09:17.626306+0000","last_unstale":"2024-09-10T19:13:04.073180+0000","last_undegraded":"2024-09-10T19:13:04.073180+0000","last_fullsized":"2024-09-10T19:13:04.073180+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:13:04.072812+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.072812+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.672103+0000","scrub_duration":0.0055804890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776582+0000","last_change":"2024-09-10T19:12:50.753994+0000","last_active":"2024-09-10T19:12:54.776582+0000","last_peered":"2024-09-10T19:12:54.776582+0000","last_clean":"2024-09-10T19:12:54.776582+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:12:54.776582+0000","last_undegraded":"2024-09-10T19:12:54.776582+0000","last_fullsized":"2024-09-10T19:12:54.776582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.753839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.753839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.661916+0000","scrub_duration":0.0033401160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631990+0000","last_change":"2024-09-10T19:12:45.106271+0000","last_active":"2024-09-10T19:12:54.631990+0000","last_peered":"2024-09-10T19:12:54.631990+0000","last_clean":"2024-09-10T19:12:54.631990+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:12:54.631990+0000","last_undegraded":"2024-09-10T19:12:54.631990+0000","last_fullsized":"2024-09-10T19:12:54.631990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.106072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:36.189093+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.106072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.364792+0000","scrub_duration":0.0037103420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778259+0000","last_change":"2024-09-10T19:12:26.456594+0000","last_active":"2024-09-10T19:12:54.778259+0000","last_peered":"2024-09-10T19:12:54.778259+0000","last_clean":"2024-09-10T19:12:54.778259+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:12:54.778259+0000","last_undegraded":"2024-09-10T19:12:54.778259+0000","last_fullsized":"2024-09-10T19:12:54.778259+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.456445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.456445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.415998+0000","scrub_duration":0.0033016310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779502+0000","last_change":"2024-09-10T19:12:45.976155+0000","last_active":"2024-09-10T19:12:54.779502+0000","last_peered":"2024-09-10T19:12:54.779502+0000","last_clean":"2024-09-10T19:12:54.779502+0000","last_became_active":"2024-09-10T19:09:17.625423+0000","last_became_peered":"2024-09-10T19:09:17.625423+0000","last_unstale":"2024-09-10T19:12:54.779502+0000","last_undegraded":"2024-09-10T19:12:54.779502+0000","last_fullsized":"2024-09-10T19:12:54.779502+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.975996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:45.975996+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.975996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.533211+0000","scrub_duration":0.0039800649999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"901'25","reported_seq":993,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.892307+0000","last_change":"2024-09-10T19:13:17.892228+0000","last_active":"2024-09-10T19:13:17.892307+0000","last_peered":"2024-09-10T19:13:17.892307+0000","last_clean":"2024-09-10T19:13:17.892307+0000","last_became_active":"2024-09-10T19:09:17.624891+0000","last_became_peered":"2024-09-10T19:09:17.624891+0000","last_unstale":"2024-09-10T19:13:17.892307+0000","last_undegraded":"2024-09-10T19:13:17.892307+0000","last_fullsized":"2024-09-10T19:13:17.892307+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'25","last_scrub_stamp":"2024-09-10T19:13:17.892088+0000","last_deep_scrub":"901'25","last_deep_scrub_stamp":"2024-09-10T19:13:17.892088+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.892088+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.410727+0000","scrub_duration":0.0081228310000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":940,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.910898+0000","last_change":"2024-09-10T19:13:13.910819+0000","last_active":"2024-09-10T19:13:13.910898+0000","last_peered":"2024-09-10T19:13:13.910898+0000","last_clean":"2024-09-10T19:13:13.910898+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:13:13.910898+0000","last_undegraded":"2024-09-10T19:13:13.910898+0000","last_fullsized":"2024-09-10T19:13:13.910898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.910669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.910669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.344950+0000","scrub_duration":0.0034429040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"873'13","reported_seq":1258,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:16.914151+0000","last_change":"2024-09-10T19:12:10.515429+0000","last_active":"2024-09-10T19:13:16.914151+0000","last_peered":"2024-09-10T19:13:16.914151+0000","last_clean":"2024-09-10T19:13:16.914151+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:13:16.914151+0000","last_undegraded":"2024-09-10T19:13:16.914151+0000","last_fullsized":"2024-09-10T19:13:16.914151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"873'13","last_scrub_stamp":"2024-09-10T19:12:10.515255+0000","last_deep_scrub":"873'13","last_deep_scrub_stamp":"2024-09-10T19:10:24.001015+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.515255+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:22.952544+0000","scrub_duration":0.0040584669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779604+0000","last_change":"2024-09-10T19:12:14.508081+0000","last_active":"2024-09-10T19:12:54.779604+0000","last_peered":"2024-09-10T19:12:54.779604+0000","last_clean":"2024-09-10T19:12:54.779604+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:12:54.779604+0000","last_undegraded":"2024-09-10T19:12:54.779604+0000","last_fullsized":"2024-09-10T19:12:54.779604+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:14.507954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:12:14.507954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.494942+0000","scrub_duration":0.0025421939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":968,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.360640+0000","last_change":"2024-09-10T19:13:04.360465+0000","last_active":"2024-09-10T19:13:04.360640+0000","last_peered":"2024-09-10T19:13:04.360640+0000","last_clean":"2024-09-10T19:13:04.360640+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:13:04.360640+0000","last_undegraded":"2024-09-10T19:13:04.360640+0000","last_fullsized":"2024-09-10T19:13:04.360640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.360223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.360223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.574679+0000","scrub_duration":0.0036477329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779083+0000","last_change":"2024-09-10T19:12:09.534767+0000","last_active":"2024-09-10T19:12:54.779083+0000","last_peered":"2024-09-10T19:12:54.779083+0000","last_clean":"2024-09-10T19:12:54.779083+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:12:54.779083+0000","last_undegraded":"2024-09-10T19:12:54.779083+0000","last_fullsized":"2024-09-10T19:12:54.779083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:12:09.534602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.534602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.543114+0000","scrub_duration":0.0035922110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"900'11","reported_seq":960,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.129734+0000","last_change":"2024-09-10T19:13:05.129607+0000","last_active":"2024-09-10T19:13:05.129734+0000","last_peered":"2024-09-10T19:13:05.129734+0000","last_clean":"2024-09-10T19:13:05.129734+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:13:05.129734+0000","last_undegraded":"2024-09-10T19:13:05.129734+0000","last_fullsized":"2024-09-10T19:13:05.129734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'11","last_scrub_stamp":"2024-09-10T19:13:05.129451+0000","last_deep_scrub":"900'11","last_deep_scrub_stamp":"2024-09-10T19:13:05.129451+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.129451+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.801663+0000","scrub_duration":0.0072537269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":981,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780917+0000","last_change":"2024-09-10T19:12:15.178732+0000","last_active":"2024-09-10T19:12:54.780917+0000","last_peered":"2024-09-10T19:12:54.780917+0000","last_clean":"2024-09-10T19:12:54.780917+0000","last_became_active":"2024-09-10T19:09:17.615219+0000","last_became_peered":"2024-09-10T19:09:17.615219+0000","last_unstale":"2024-09-10T19:12:54.780917+0000","last_undegraded":"2024-09-10T19:12:54.780917+0000","last_fullsized":"2024-09-10T19:12:54.780917+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:15.178658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.968019+0000","last_clean_scrub_stamp":"2024-09-10T19:12:15.178658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.379149+0000","scrub_duration":0.0021276839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781980+0000","last_change":"2024-09-10T19:12:36.716875+0000","last_active":"2024-09-10T19:12:54.781980+0000","last_peered":"2024-09-10T19:12:54.781980+0000","last_clean":"2024-09-10T19:12:54.781980+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:12:54.781980+0000","last_undegraded":"2024-09-10T19:12:54.781980+0000","last_fullsized":"2024-09-10T19:12:54.781980+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:36.716727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:06.774232+0000","last_clean_scrub_stamp":"2024-09-10T19:12:36.716727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.432775+0000","scrub_duration":0.0032699080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":934,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630756+0000","last_change":"2024-09-10T19:12:18.981781+0000","last_active":"2024-09-10T19:12:54.630756+0000","last_peered":"2024-09-10T19:12:54.630756+0000","last_clean":"2024-09-10T19:12:54.630756+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:12:54.630756+0000","last_undegraded":"2024-09-10T19:12:54.630756+0000","last_fullsized":"2024-09-10T19:12:54.630756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.981632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.981632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:48.322860+0000","scrub_duration":0.0036854710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":940,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:10.917971+0000","last_change":"2024-09-10T19:13:10.917868+0000","last_active":"2024-09-10T19:13:10.917971+0000","last_peered":"2024-09-10T19:13:10.917971+0000","last_clean":"2024-09-10T19:13:10.917971+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:13:10.917971+0000","last_undegraded":"2024-09-10T19:13:10.917971+0000","last_fullsized":"2024-09-10T19:13:10.917971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:10.917727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:10.917727+0000","last_clean_scrub_stamp":"2024-09-10T19:13:10.917727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.898023+0000","scrub_duration":0.003015958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"877'25","reported_seq":1007,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628245+0000","last_change":"2024-09-10T19:12:19.551853+0000","last_active":"2024-09-10T19:12:54.628245+0000","last_peered":"2024-09-10T19:12:54.628245+0000","last_clean":"2024-09-10T19:12:54.628245+0000","last_became_active":"2024-09-10T19:09:17.613813+0000","last_became_peered":"2024-09-10T19:09:17.613813+0000","last_unstale":"2024-09-10T19:12:54.628245+0000","last_undegraded":"2024-09-10T19:12:54.628245+0000","last_fullsized":"2024-09-10T19:12:54.628245+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-10T19:12:19.551679+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-10T19:10:43.846279+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.551679+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.808352+0000","scrub_duration":0.0048679129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629589+0000","last_change":"2024-09-10T19:12:32.020746+0000","last_active":"2024-09-10T19:12:54.629589+0000","last_peered":"2024-09-10T19:12:54.629589+0000","last_clean":"2024-09-10T19:12:54.629589+0000","last_became_active":"2024-09-10T19:09:17.621963+0000","last_became_peered":"2024-09-10T19:09:17.621963+0000","last_unstale":"2024-09-10T19:12:54.629589+0000","last_undegraded":"2024-09-10T19:12:54.629589+0000","last_fullsized":"2024-09-10T19:12:54.629589+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:32.020586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:12:32.020586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.091894+0000","scrub_duration":0.0033598090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777803+0000","last_change":"2024-09-10T19:12:37.380309+0000","last_active":"2024-09-10T19:12:54.777803+0000","last_peered":"2024-09-10T19:12:54.777803+0000","last_clean":"2024-09-10T19:12:54.777803+0000","last_became_active":"2024-09-10T19:09:17.615496+0000","last_became_peered":"2024-09-10T19:09:17.615496+0000","last_unstale":"2024-09-10T19:12:54.777803+0000","last_undegraded":"2024-09-10T19:12:54.777803+0000","last_fullsized":"2024-09-10T19:12:54.777803+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:37.380157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:51.945980+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.380157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.489922+0000","scrub_duration":0.0036922259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1417,"num_read_kb":1412,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1552,"num_read_kb":1309,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1089,"ondisk_log_size":1089,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972654,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307736,"kb_used_data":2064,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437416,"statfs":{"total":95995035648,"available":95679913984,"internally_reserved":0,"allocated":2113536,"data_stored":844590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.93600000000000005,"15min":0.92200000000000004},"min":{"1min":0.66500000000000004,"5min":0.42999999999999999,"15min":0.40300000000000002},"max":{"1min":1.3149999999999999,"5min":2.1589999999999998,"15min":2.839},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95599999999999996,"15min":0.95299999999999996},"min":{"1min":0.53400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":2.8079999999999998,"15min":2.9550000000000001},"last":0.92500000000000004}]},{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.93300000000000005},"min":{"1min":0.72299999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.151,"5min":1.921,"15min":1.921},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.3220000000000001,"5min":1.881,"15min":1.881},"last":0.79100000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.98699999999999999,"15min":1.095},"min":{"1min":0.504,"5min":0.504,"15min":0.45000000000000001},"max":{"1min":1.3260000000000001,"5min":1.9750000000000001,"15min":31.166},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.69799999999999995,"5min":0.54200000000000004,"15min":0.443},"max":{"1min":5.218,"5min":5.218,"15min":27.709},"last":1.0740000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0069999999999999,"15min":1.0249999999999999},"min":{"1min":0.76000000000000001,"5min":0.59199999999999997,"15min":0.51700000000000002},"max":{"1min":1.4850000000000001,"5min":2.8799999999999999,"15min":4.4800000000000004},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96699999999999997},"min":{"1min":0.57599999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3,"5min":2.0459999999999998,"15min":4.2060000000000004},"last":0.97099999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96299999999999997,"15min":0.997},"min":{"1min":0.63600000000000001,"5min":0.54800000000000004,"15min":0.34799999999999998},"max":{"1min":1.339,"5min":1.3660000000000001,"15min":1.5760000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.76500000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2649999999999999,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95099999999999996,"15min":0.95699999999999996},"min":{"1min":0.73399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.234,"5min":1.3200000000000001,"15min":3.1970000000000001},"last":1.054},{"interface":"front","average":{"1min":1.181,"5min":1.1220000000000001,"15min":1.1279999999999999},"min":{"1min":0.85599999999999998,"5min":0.52900000000000003,"15min":0.47499999999999998},"max":{"1min":1.421,"5min":1.7829999999999999,"15min":4.8710000000000004},"last":1.3799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.1699999999999999,"15min":1.1599999999999999},"min":{"1min":1.0229999999999999,"5min":0.56299999999999994,"15min":0.48699999999999999},"max":{"1min":1.4019999999999999,"5min":1.706,"15min":1.766},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.2090000000000001,"15min":1.1859999999999999},"min":{"1min":0.80200000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5960000000000001,"5min":2.9430000000000001,"15min":2.96},"last":1.137}]}]},{"osd":6,"up_from":52,"seq":223338299704,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307396,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437756,"statfs":{"total":95995035648,"available":95680262144,"internally_reserved":0,"allocated":1765376,"data_stored":515149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89700000000000002,"15min":0.89300000000000002},"min":{"1min":0.58999999999999997,"5min":0.47999999999999998,"15min":0.26900000000000002},"max":{"1min":1.3360000000000001,"5min":1.385,"15min":1.8839999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.95399999999999996,"15min":0.96999999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.40300000000000002},"max":{"1min":1.254,"5min":4.1420000000000003,"15min":5.7309999999999999},"last":0.71499999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.008,"15min":0.94899999999999995},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.345,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.097,"5min":1.129,"15min":1.107},"min":{"1min":0.76700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.387,"5min":4.1870000000000003,"15min":4.1870000000000003},"last":0.73499999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.94599999999999995},"min":{"1min":0.624,"5min":0.42399999999999999,"15min":0.33500000000000002},"max":{"1min":1.1850000000000001,"5min":4.069,"15min":4.069},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.94199999999999995,"15min":0.94699999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":4.0350000000000001,"15min":6.0970000000000004},"last":0.76000000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0169999999999999,"15min":1},"min":{"1min":0.77400000000000002,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.2250000000000001,"5min":3.972,"15min":3.972},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96199999999999997,"15min":0.97099999999999997},"min":{"1min":0.74199999999999999,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.345,"5min":4.2270000000000003,"15min":4.2270000000000003},"last":0.55000000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.95799999999999996,"15min":0.97799999999999998},"min":{"1min":0.65500000000000003,"5min":0.55800000000000005,"15min":0.44400000000000001},"max":{"1min":1.3120000000000001,"5min":1.79,"15min":6.008},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.626,"5min":0.59499999999999997,"15min":0.42299999999999999},"max":{"1min":1.3799999999999999,"5min":1.48,"15min":2.3039999999999998},"last":0.41099999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0389999999999999,"15min":1.0640000000000001},"min":{"1min":0.72499999999999998,"5min":0.52300000000000002,"15min":0.5},"max":{"1min":1.2090000000000001,"5min":1.9179999999999999,"15min":5.9560000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1000000000000001,"15min":1.109},"min":{"1min":0.76300000000000001,"5min":0.60699999999999998,"15min":0.47599999999999998},"max":{"1min":1.3879999999999999,"5min":1.5820000000000001,"15min":3.5169999999999999},"last":0.52400000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.755,"5min":0.51000000000000001,"15min":0.47899999999999998},"max":{"1min":1.363,"5min":1.393,"15min":1.8320000000000001},"last":0.438},{"interface":"front","average":{"1min":1.131,"5min":1.151,"15min":1.1930000000000001},"min":{"1min":0.88200000000000001,"5min":0.77000000000000002,"15min":0.51200000000000001},"max":{"1min":1.369,"5min":2.02,"15min":6.0380000000000003},"last":0.80100000000000005}]}]},{"osd":5,"up_from":43,"seq":184683594050,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441336,"statfs":{"total":95995035648,"available":95683928064,"internally_reserved":0,"allocated":2359296,"data_stored":1105591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.92000000000000004,"15min":0.95899999999999996},"min":{"1min":0.60999999999999999,"5min":0.55200000000000005,"15min":0.46500000000000002},"max":{"1min":1.2390000000000001,"5min":2.1840000000000002,"15min":5.4640000000000004},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.92500000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2190000000000001,"5min":1.2430000000000001,"15min":5.4790000000000001},"last":1.3080000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.93400000000000005,"15min":0.93100000000000005},"min":{"1min":0.64100000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2130000000000001,"5min":1.387,"15min":1.387},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.97199999999999998,"15min":0.98099999999999998},"min":{"1min":0.72599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.2170000000000001,"5min":1.377,"15min":1.377},"last":1.127}]},{"osd":2,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.95299999999999996,"15min":0.98899999999999999},"min":{"1min":0.629,"5min":0.59299999999999997,"15min":0.51000000000000001},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":7.9340000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":0.997},"min":{"1min":0.60999999999999999,"5min":0.55500000000000005,"15min":0.52000000000000002},"max":{"1min":1.2569999999999999,"5min":1.532,"15min":6.3280000000000003},"last":0.97399999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.0129999999999999,"15min":1.006},"min":{"1min":0.77100000000000002,"5min":0.69499999999999995,"15min":0.56299999999999994},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.60799999999999998},"max":{"1min":1.2629999999999999,"5min":2.0990000000000002,"15min":5.3810000000000002},"last":1.1759999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.879,"15min":0.93899999999999995},"min":{"1min":0.56899999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.2949999999999999,"5min":1.3200000000000001,"15min":5.5170000000000003},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.59599999999999997,"5min":0.56799999999999995,"15min":0.39400000000000002},"max":{"1min":2.895,"5min":2.895,"15min":4.1970000000000001},"last":1.0880000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":1.0329999999999999,"5min":0.63400000000000001,"15min":0.57499999999999996},"max":{"1min":1.423,"5min":2.8399999999999999,"15min":5.2009999999999996},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1040000000000001,"15min":1.1379999999999999},"min":{"1min":0.76900000000000002,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":1.4570000000000001,"5min":2.343,"15min":5.4240000000000004},"last":1.286}]},{"osd":7,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0389999999999999,"15min":1.095},"min":{"1min":0.71399999999999997,"5min":0.57099999999999995,"15min":0.41999999999999998},"max":{"1min":1.429,"5min":1.4690000000000001,"15min":5.7210000000000001},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.288,"5min":1.1599999999999999,"15min":1.151},"min":{"1min":0.98499999999999999,"5min":0.69099999999999995,"15min":0.42599999999999999},"max":{"1min":1.7649999999999999,"5min":3.1059999999999999,"15min":4.2480000000000002},"last":1.4079999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855694,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311588,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433564,"statfs":{"total":95995035648,"available":95675969536,"internally_reserved":0,"allocated":1798144,"data_stored":516473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.875,"15min":0.879},"min":{"1min":0.60099999999999998,"5min":0.48099999999999998,"15min":0.371},"max":{"1min":1.3100000000000001,"5min":1.3819999999999999,"15min":2.0179999999999998},"last":0.749},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.629,"5min":0.52400000000000002,"15min":0.44600000000000001},"max":{"1min":1.55,"5min":1.55,"15min":2.1930000000000001},"last":0.88600000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.92000000000000004,"15min":0.90000000000000002},"min":{"1min":0.63500000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3480000000000001,"5min":1.387,"15min":1.387},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.069,"15min":1.085},"min":{"1min":0.78100000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.0069999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.93600000000000005,"15min":1.04},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.46600000000000003},"max":{"1min":1.272,"5min":1.4550000000000001,"15min":18.853000000000002},"last":1.032},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1,"15min":1.0820000000000001},"min":{"1min":0.72299999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.409,"5min":1.579,"15min":25.652999999999999},"last":0.79400000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.96999999999999997,"15min":1.0149999999999999},"min":{"1min":0.76200000000000001,"5min":0.51900000000000002,"15min":0.49099999999999999},"max":{"1min":1.345,"5min":1.421,"15min":4.2670000000000003},"last":1.141},{"interface":"front","average":{"1min":1.012,"5min":0.94099999999999995,"15min":0.98299999999999998},"min":{"1min":0.78500000000000003,"5min":0.51000000000000001,"15min":0.49099999999999999},"max":{"1min":1.214,"5min":1.405,"15min":2.0739999999999998},"last":1.1060000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88600000000000001,"15min":0.94599999999999995},"min":{"1min":0.58499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.8009999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.042,"5min":1.022,"15min":1.0700000000000001},"min":{"1min":0.63900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2989999999999999,"5min":1.5089999999999999,"15min":2.0259999999999998},"last":1.23}]},{"osd":6,"last update":"Tue Sep 10 19:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.94899999999999995,"15min":1.0209999999999999},"min":{"1min":0.751,"5min":0.307,"15min":0.307},"max":{"1min":1.2949999999999999,"5min":1.5589999999999999,"15min":2.1699999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.087,"15min":1.1579999999999999},"min":{"1min":0.72999999999999998,"5min":0.54300000000000004,"15min":0.48799999999999999},"max":{"1min":1.46,"5min":1.46,"15min":2.9590000000000001},"last":1.0820000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1399999999999999,"15min":1.167},"min":{"1min":0.86499999999999999,"5min":0.71199999999999997,"15min":0.56499999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":2.6739999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.0760000000000001,"15min":1.1539999999999999},"min":{"1min":0.79800000000000004,"5min":0.73799999999999999,"15min":0.40899999999999997},"max":{"1min":1.2989999999999999,"5min":1.536,"15min":3.7850000000000001},"last":1.1850000000000001}]}]},{"osd":3,"up_from":29,"seq":124554051929,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441336,"statfs":{"total":95995035648,"available":95683928064,"internally_reserved":0,"allocated":2359296,"data_stored":1106443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.29499999999999998},"max":{"1min":0.96499999999999997,"5min":1.4830000000000001,"15min":1.6539999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.749,"15min":0.73999999999999999},"min":{"1min":0.45900000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.96599999999999997,"5min":1.79,"15min":1.8660000000000001},"last":0.65700000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:13:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.59199999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1060000000000001,"5min":1.704,"15min":1.704},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.99099999999999999,"5min":1.365,"15min":1.365},"last":0.61799999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.83799999999999997},"min":{"1min":0.442,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1850000000000001,"5min":1.661,"15min":2.073},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80200000000000005,"15min":0.81399999999999995},"min":{"1min":0.45000000000000001,"5min":0.379,"15min":0.32600000000000001},"max":{"1min":1.016,"5min":2.032,"15min":4.4740000000000002},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0489999999999999,"15min":1.097},"min":{"1min":0.84399999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.55,"5min":1.748,"15min":3.2469999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1479999999999999,"15min":1.2010000000000001},"min":{"1min":0.92200000000000004,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.6339999999999999,"5min":1.956,"15min":3.2850000000000001},"last":0.89500000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0580000000000001,"15min":1.1060000000000001},"min":{"1min":0.73799999999999999,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.593,"5min":1.8,"15min":2.2770000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97299999999999998,"15min":1.024},"min":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.53500000000000003},"max":{"1min":1.6080000000000001,"5min":1.8169999999999999,"15min":4.6619999999999999},"last":1.05}]},{"osd":6,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.135,"15min":1.1659999999999999},"min":{"1min":0.95199999999999996,"5min":0.69399999999999995,"15min":0.59499999999999997},"max":{"1min":1.679,"5min":1.8859999999999999,"15min":2.206},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.125,"15min":1.111},"min":{"1min":0.82699999999999996,"5min":0.68300000000000005,"15min":0.56999999999999995},"max":{"1min":1.46,"5min":2.2200000000000002,"15min":2.996},"last":0.88400000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.097},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.59899999999999998},"max":{"1min":1.359,"5min":1.778,"15min":4.0229999999999997},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.177,"15min":1.24},"min":{"1min":0.871,"5min":0.70799999999999996,"15min":0.66000000000000003},"max":{"1min":1.3140000000000001,"5min":1.921,"15min":4.5350000000000001},"last":0.94499999999999995}]}]},{"osd":2,"up_from":19,"seq":81604378978,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311824,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433328,"statfs":{"total":95995035648,"available":95675727872,"internally_reserved":0,"allocated":2039808,"data_stored":778955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66600000000000004,"15min":0.83399999999999996},"min":{"1min":0.47699999999999998,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.1080000000000001,"5min":1.2170000000000001,"15min":43.927999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.74199999999999999,"15min":0.95499999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.30299999999999999},"max":{"1min":1.1240000000000001,"5min":1.26,"15min":49.475000000000001},"last":0.78200000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:12:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.73099999999999998,"15min":0.72699999999999998},"min":{"1min":0.54100000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0089999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.88,"5min":0.84199999999999997,"15min":0.82399999999999995},"min":{"1min":0.625,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0549999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.76300000000000001,"15min":0.85899999999999999},"min":{"1min":0.56200000000000006,"5min":0.38900000000000001,"15min":0.318},"max":{"1min":1.869,"5min":1.869,"15min":6.4020000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.80600000000000005,"15min":0.92200000000000004},"min":{"1min":0.61099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.2869999999999999,"5min":1.361,"15min":9.8219999999999992},"last":0.56200000000000006}]},{"osd":4,"last update":"Tue Sep 10 19:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.002,"15min":1.177},"min":{"1min":0.79200000000000004,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":38.079999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.143,"5min":1.042,"15min":1.2390000000000001},"min":{"1min":0.876,"5min":0.42199999999999999,"15min":0.40400000000000003},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":37.991},"last":0.95999999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0349999999999999,"15min":1.0780000000000001},"min":{"1min":0.86799999999999999,"5min":0.59999999999999998,"15min":0.38500000000000001},"max":{"1min":1.706,"5min":1.706,"15min":1.8340000000000001},"last":1.139},{"interface":"front","average":{"1min":1.089,"5min":1.0669999999999999,"15min":1.121},"min":{"1min":0.71599999999999997,"5min":0.62,"15min":0.46200000000000002},"max":{"1min":1.583,"5min":3.0819999999999999,"15min":6.0620000000000003},"last":0.90500000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.073,"15min":1.256},"min":{"1min":0.79000000000000004,"5min":0.53500000000000003,"15min":0.503},"max":{"1min":1.661,"5min":1.6919999999999999,"15min":43.655999999999999},"last":1.026},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.03,"15min":1.2370000000000001},"min":{"1min":0.77900000000000003,"5min":0.71099999999999997,"15min":0.48999999999999999},"max":{"1min":1.472,"5min":1.6339999999999999,"15min":45.255000000000003},"last":1.087}]},{"osd":7,"last update":"Tue Sep 10 19:12:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.121,"15min":1.171},"min":{"1min":0.876,"5min":0.68000000000000005,"15min":0.48399999999999999},"max":{"1min":1.679,"5min":1.679,"15min":6.0179999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0740000000000001,"15min":1.1719999999999999},"min":{"1min":0.84299999999999997,"5min":0.66600000000000004,"15min":0.53100000000000003},"max":{"1min":1.645,"5min":1.645,"15min":6.851},"last":1.175}]}]},{"osd":1,"up_from":855,"seq":3672197038132,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304968,"kb_used_data":1728,"kb_used_omap":89,"kb_used_meta":303142,"kb_avail":93440184,"statfs":{"total":95995035648,"available":95682748416,"internally_reserved":0,"allocated":1769472,"data_stored":515497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91933,"internal_metadata":310417635},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70199999999999996,"15min":0.68600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94399999999999995,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.80100000000000005,"15min":0.77700000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.202,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.69099999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.1259999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.753,"15min":0.751},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.121,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.878,"15min":0.878},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2050000000000001,"5min":1.405,"15min":1.405},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86399999999999999},"min":{"1min":0.55500000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.0289999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.67100000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.294,"5min":1.48,"15min":1.48},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.093,"15min":1.097},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3819999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":1.0189999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.337,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.002,"5min":1.083,"15min":1.1220000000000001},"min":{"1min":0.79900000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.2609999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.85199999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.169,"15min":1.1830000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.349,"5min":1.833,"15min":1.833},"last":1.111},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.175,"15min":1.1830000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.4970000000000001,"5min":1.845,"15min":1.845},"last":1.1799999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.196,"15min":1.242},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.3129999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.234,"15min":1.262},"min":{"1min":0.79600000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.407,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.98099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738745,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303240,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441912,"statfs":{"total":95995035648,"available":95684517888,"internally_reserved":0,"allocated":1769472,"data_stored":515081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1579999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.80900000000000005,"15min":0.81000000000000005},"min":{"1min":0.51500000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.226,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.66700000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":0.77700000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.36799999999999999},"max":{"1min":1.25,"5min":1.802,"15min":3.427},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.68700000000000006,"15min":0.77900000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.04,"5min":1.125,"15min":3.2090000000000001},"last":0.627}]},{"osd":3,"last update":"Tue Sep 10 19:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.78300000000000003,"15min":0.83699999999999997},"min":{"1min":0.438,"5min":0.317,"15min":0.317},"max":{"1min":1.3260000000000001,"5min":1.4470000000000001,"15min":4.4749999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.86099999999999999,"15min":0.874},"min":{"1min":0.54900000000000004,"5min":0.45400000000000001,"15min":0.42499999999999999},"max":{"1min":1.1439999999999999,"5min":1.8819999999999999,"15min":2.4569999999999999},"last":0.77600000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.91600000000000004,"15min":1.036},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.48199999999999998},"max":{"1min":1.198,"5min":1.4590000000000001,"15min":3.3279999999999998},"last":0.879},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0720000000000001,"15min":1.1299999999999999},"min":{"1min":0.82699999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"max":{"1min":1.304,"5min":1.7070000000000001,"15min":3.419},"last":0.93700000000000006}]},{"osd":5,"last update":"Tue Sep 10 19:12:30 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0649999999999999,"15min":1.0940000000000001},"min":{"1min":0.70999999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.335,"5min":1.8380000000000001,"15min":2.609},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.085,"15min":1.129},"min":{"1min":0.78200000000000003,"5min":0.629,"15min":0.62},"max":{"1min":1.357,"5min":1.643,"15min":2.0779999999999998},"last":1.109}]},{"osd":6,"last update":"Tue Sep 10 19:12:30 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1399999999999999,"15min":1.1859999999999999},"min":{"1min":0.90500000000000003,"5min":0.71099999999999997,"15min":0.52400000000000002},"max":{"1min":1.3,"5min":2.1070000000000002,"15min":2.4550000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.169,"15min":1.175},"min":{"1min":0.872,"5min":0.72699999999999998,"15min":0.48999999999999999},"max":{"1min":1.3979999999999999,"5min":1.9850000000000001,"15min":2.5339999999999998},"last":0.86499999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.079,"15min":1.165},"min":{"1min":0.83599999999999997,"5min":0.63500000000000001,"15min":0.61599999999999999},"max":{"1min":1.466,"5min":1.5129999999999999,"15min":2.468},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.22,"5min":1.214,"15min":1.244},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.163}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:13:32.770 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:13:33.041 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:32] "GET /metrics HTTP/1.1" 200 224114 "" "Prometheus/2.43.0" 2024-09-10T19:13:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[18629]: audit 2024-09-10T19:13:31.835310+0000 mgr.y (mgr.24526) 933 : audit [DBG] from='client.50135 v1:172.21.15.83:0/724317947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.139300+0000 osd.0 (osd.0) 589 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:13:33.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.142394+0000 osd.0 (osd.0) 590 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:13:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[32381]: audit 2024-09-10T19:13:31.835310+0000 mgr.y (mgr.24526) 933 : audit [DBG] from='client.50135 v1:172.21.15.83:0/724317947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.139300+0000 osd.0 (osd.0) 589 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:13:33.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:32 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.142394+0000 osd.0 (osd.0) 590 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:13:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:32 smithi165 bash[22591]: audit 2024-09-10T19:13:31.835310+0000 mgr.y (mgr.24526) 933 : audit [DBG] from='client.50135 v1:172.21.15.83:0/724317947' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:32 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.139300+0000 osd.0 (osd.0) 589 : cluster [DBG] 4.6 scrub starts 2024-09-10T19:13:33.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:32 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.142394+0000 osd.0 (osd.0) 590 : cluster [DBG] 4.6 scrub ok 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.099712+0000 osd.1 (osd.1) 153 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.103300+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.479554+0000 osd.6 (osd.6) 541 : cluster [DBG] 3.a scrub starts 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.485608+0000 osd.6 (osd.6) 542 : cluster [DBG] 3.a scrub ok 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.575962+0000 mgr.y (mgr.24526) 934 : cluster [DBG] pgmap v1513: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.855245+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:13:34.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:33 smithi165 bash[22591]: cluster 2024-09-10T19:13:32.859455+0000 osd.3 (osd.3) 619 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.099712+0000 osd.1 (osd.1) 153 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.103300+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.479554+0000 osd.6 (osd.6) 541 : cluster [DBG] 3.a scrub starts 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.485608+0000 osd.6 (osd.6) 542 : cluster [DBG] 3.a scrub ok 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.575962+0000 mgr.y (mgr.24526) 934 : cluster [DBG] pgmap v1513: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.855245+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:13:34.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[18629]: cluster 2024-09-10T19:13:32.859455+0000 osd.3 (osd.3) 619 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.099712+0000 osd.1 (osd.1) 153 : cluster [DBG] 6.4 scrub starts 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.103300+0000 osd.1 (osd.1) 154 : cluster [DBG] 6.4 scrub ok 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.479554+0000 osd.6 (osd.6) 541 : cluster [DBG] 3.a scrub starts 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.485608+0000 osd.6 (osd.6) 542 : cluster [DBG] 3.a scrub ok 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.575962+0000 mgr.y (mgr.24526) 934 : cluster [DBG] pgmap v1513: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.855245+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.2 scrub starts 2024-09-10T19:13:34.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:33 smithi083 bash[32381]: cluster 2024-09-10T19:13:32.859455+0000 osd.3 (osd.3) 619 : cluster [DBG] 3.2 scrub ok 2024-09-10T19:13:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:35 smithi165 bash[22591]: cluster 2024-09-10T19:13:34.576733+0000 mgr.y (mgr.24526) 935 : cluster [DBG] pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:35 smithi165 bash[22591]: cluster 2024-09-10T19:13:34.684376+0000 osd.2 (osd.2) 419 : cluster [DBG] 6.f scrub starts 2024-09-10T19:13:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:35 smithi165 bash[22591]: cluster 2024-09-10T19:13:34.687951+0000 osd.2 (osd.2) 420 : cluster [DBG] 6.f scrub ok 2024-09-10T19:13:36.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:35 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.109179+0000 osd.0 (osd.0) 591 : cluster [DBG] 4.b deep-scrub starts 2024-09-10T19:13:36.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:35 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.113265+0000 osd.0 (osd.0) 592 : cluster [DBG] 4.b deep-scrub ok 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[18629]: cluster 2024-09-10T19:13:34.576733+0000 mgr.y (mgr.24526) 935 : cluster [DBG] pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[18629]: cluster 2024-09-10T19:13:34.684376+0000 osd.2 (osd.2) 419 : cluster [DBG] 6.f scrub starts 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[18629]: cluster 2024-09-10T19:13:34.687951+0000 osd.2 (osd.2) 420 : cluster [DBG] 6.f scrub ok 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.109179+0000 osd.0 (osd.0) 591 : cluster [DBG] 4.b deep-scrub starts 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.113265+0000 osd.0 (osd.0) 592 : cluster [DBG] 4.b deep-scrub ok 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[32381]: cluster 2024-09-10T19:13:34.576733+0000 mgr.y (mgr.24526) 935 : cluster [DBG] pgmap v1514: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[32381]: cluster 2024-09-10T19:13:34.684376+0000 osd.2 (osd.2) 419 : cluster [DBG] 6.f scrub starts 2024-09-10T19:13:36.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[32381]: cluster 2024-09-10T19:13:34.687951+0000 osd.2 (osd.2) 420 : cluster [DBG] 6.f scrub ok 2024-09-10T19:13:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.109179+0000 osd.0 (osd.0) 591 : cluster [DBG] 4.b deep-scrub starts 2024-09-10T19:13:36.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:35 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.113265+0000 osd.0 (osd.0) 592 : cluster [DBG] 4.b deep-scrub ok 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.123558+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.127141+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.274813+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.277984+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.338924+0000 osd.4 (osd.4) 572 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T19:13:37.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:36 smithi165 bash[22591]: cluster 2024-09-10T19:13:35.342438+0000 osd.4 (osd.4) 573 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.123558+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.127141+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.274813+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.277984+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.338924+0000 osd.4 (osd.4) 572 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[32381]: cluster 2024-09-10T19:13:35.342438+0000 osd.4 (osd.4) 573 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.123558+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.6 scrub starts 2024-09-10T19:13:37.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.127141+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.6 scrub ok 2024-09-10T19:13:37.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.274813+0000 osd.7 (osd.7) 596 : cluster [DBG] 6.3 scrub starts 2024-09-10T19:13:37.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.277984+0000 osd.7 (osd.7) 597 : cluster [DBG] 6.3 scrub ok 2024-09-10T19:13:37.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.338924+0000 osd.4 (osd.4) 572 : cluster [DBG] 4.1a deep-scrub starts 2024-09-10T19:13:37.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:36 smithi083 bash[18629]: cluster 2024-09-10T19:13:35.342438+0000 osd.4 (osd.4) 573 : cluster [DBG] 4.1a deep-scrub ok 2024-09-10T19:13:37.553 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[18629]: cluster 2024-09-10T19:13:36.279798+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:13:38.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[18629]: cluster 2024-09-10T19:13:36.283442+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:13:38.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[18629]: cluster 2024-09-10T19:13:36.578663+0000 mgr.y (mgr.24526) 936 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:38.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[18629]: cluster 2024-09-10T19:13:36.731399+0000 osd.5 (osd.5) 592 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:13:38.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[18629]: cluster 2024-09-10T19:13:36.734663+0000 osd.5 (osd.5) 593 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:13:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[32381]: cluster 2024-09-10T19:13:36.279798+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:13:38.042 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[32381]: cluster 2024-09-10T19:13:36.283442+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:13:38.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[32381]: cluster 2024-09-10T19:13:36.578663+0000 mgr.y (mgr.24526) 936 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:38.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[32381]: cluster 2024-09-10T19:13:36.731399+0000 osd.5 (osd.5) 592 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:13:38.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:37 smithi083 bash[32381]: cluster 2024-09-10T19:13:36.734663+0000 osd.5 (osd.5) 593 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:13:38.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:37 smithi165 bash[22591]: cluster 2024-09-10T19:13:36.279798+0000 osd.7 (osd.7) 598 : cluster [DBG] 2.1d scrub starts 2024-09-10T19:13:38.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:37 smithi165 bash[22591]: cluster 2024-09-10T19:13:36.283442+0000 osd.7 (osd.7) 599 : cluster [DBG] 2.1d scrub ok 2024-09-10T19:13:38.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:37 smithi165 bash[22591]: cluster 2024-09-10T19:13:36.578663+0000 mgr.y (mgr.24526) 936 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:38.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:37 smithi165 bash[22591]: cluster 2024-09-10T19:13:36.731399+0000 osd.5 (osd.5) 592 : cluster [DBG] 2.12 scrub starts 2024-09-10T19:13:38.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:37 smithi165 bash[22591]: cluster 2024-09-10T19:13:36.734663+0000 osd.5 (osd.5) 593 : cluster [DBG] 2.12 scrub ok 2024-09-10T19:13:39.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:13:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:37.374472+0000 osd.6 (osd.6) 543 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:13:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:37.379115+0000 osd.6 (osd.6) 544 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:13:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:37.980068+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:13:39.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:37.983899+0000 osd.3 (osd.3) 621 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:13:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.084244+0000 osd.0 (osd.0) 593 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:13:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.088432+0000 osd.0 (osd.0) 594 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:13:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.761964+0000 osd.2 (osd.2) 421 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:13:39.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:38 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.766404+0000 osd.2 (osd.2) 422 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:37.374472+0000 osd.6 (osd.6) 543 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:37.379115+0000 osd.6 (osd.6) 544 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:37.980068+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:37.983899+0000 osd.3 (osd.3) 621 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.084244+0000 osd.0 (osd.0) 593 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.088432+0000 osd.0 (osd.0) 594 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.761964+0000 osd.2 (osd.2) 421 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:13:39.137 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.766404+0000 osd.2 (osd.2) 422 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:37.374472+0000 osd.6 (osd.6) 543 : cluster [DBG] 3.10 scrub starts 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:37.379115+0000 osd.6 (osd.6) 544 : cluster [DBG] 3.10 scrub ok 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:37.980068+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.6 scrub starts 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:37.983899+0000 osd.3 (osd.3) 621 : cluster [DBG] 6.6 scrub ok 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.084244+0000 osd.0 (osd.0) 593 : cluster [DBG] 2.1f scrub starts 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.088432+0000 osd.0 (osd.0) 594 : cluster [DBG] 2.1f scrub ok 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.761964+0000 osd.2 (osd.2) 421 : cluster [DBG] 4.1c deep-scrub starts 2024-09-10T19:13:39.138 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:38 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.766404+0000 osd.2 (osd.2) 422 : cluster [DBG] 4.1c deep-scrub ok 2024-09-10T19:13:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.345256+0000 osd.6 (osd.6) 545 : cluster [DBG] 4.a scrub starts 2024-09-10T19:13:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.348825+0000 osd.6 (osd.6) 546 : cluster [DBG] 4.a scrub ok 2024-09-10T19:13:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.579431+0000 mgr.y (mgr.24526) 937 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: audit 2024-09-10T19:13:38.747633+0000 mgr.y (mgr.24526) 938 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:38.997246+0000 osd.3 (osd.3) 622 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:13:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.000587+0000 osd.3 (osd.3) 623 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:13:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.106259+0000 osd.0 (osd.0) 595 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:13:40.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:39 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.110188+0000 osd.0 (osd.0) 596 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.345256+0000 osd.6 (osd.6) 545 : cluster [DBG] 4.a scrub starts 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.348825+0000 osd.6 (osd.6) 546 : cluster [DBG] 4.a scrub ok 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.579431+0000 mgr.y (mgr.24526) 937 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: audit 2024-09-10T19:13:38.747633+0000 mgr.y (mgr.24526) 938 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:38.997246+0000 osd.3 (osd.3) 622 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.000587+0000 osd.3 (osd.3) 623 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.106259+0000 osd.0 (osd.0) 595 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:13:40.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.110188+0000 osd.0 (osd.0) 596 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.345256+0000 osd.6 (osd.6) 545 : cluster [DBG] 4.a scrub starts 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.348825+0000 osd.6 (osd.6) 546 : cluster [DBG] 4.a scrub ok 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.579431+0000 mgr.y (mgr.24526) 937 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: audit 2024-09-10T19:13:38.747633+0000 mgr.y (mgr.24526) 938 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:38.997246+0000 osd.3 (osd.3) 622 : cluster [DBG] 6.1b scrub starts 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.000587+0000 osd.3 (osd.3) 623 : cluster [DBG] 6.1b scrub ok 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.106259+0000 osd.0 (osd.0) 595 : cluster [DBG] 6.18 scrub starts 2024-09-10T19:13:40.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:39 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.110188+0000 osd.0 (osd.0) 596 : cluster [DBG] 6.18 scrub ok 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.157798+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.160656+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.275414+0000 osd.7 (osd.7) 600 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.280337+0000 osd.7 (osd.7) 601 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.792086+0000 osd.5 (osd.5) 594 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:13:41.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:39.796710+0000 osd.5 (osd.5) 595 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:13:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.007593+0000 osd.3 (osd.3) 624 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:13:41.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:40 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.010882+0000 osd.3 (osd.3) 625 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:13:41.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.157798+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:13:41.290 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.160656+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.275414+0000 osd.7 (osd.7) 600 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.280337+0000 osd.7 (osd.7) 601 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.792086+0000 osd.5 (osd.5) 594 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:39.796710+0000 osd.5 (osd.5) 595 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.007593+0000 osd.3 (osd.3) 624 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.010882+0000 osd.3 (osd.3) 625 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.157798+0000 osd.1 (osd.1) 157 : cluster [DBG] 12.7 scrub starts 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.160656+0000 osd.1 (osd.1) 158 : cluster [DBG] 12.7 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.275414+0000 osd.7 (osd.7) 600 : cluster [DBG] 3.11 scrub starts 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.280337+0000 osd.7 (osd.7) 601 : cluster [DBG] 3.11 scrub ok 2024-09-10T19:13:41.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.792086+0000 osd.5 (osd.5) 594 : cluster [DBG] 1.8 scrub starts 2024-09-10T19:13:41.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:39.796710+0000 osd.5 (osd.5) 595 : cluster [DBG] 1.8 scrub ok 2024-09-10T19:13:41.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.007593+0000 osd.3 (osd.3) 624 : cluster [DBG] 4.0 scrub starts 2024-09-10T19:13:41.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:40 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.010882+0000 osd.3 (osd.3) 625 : cluster [DBG] 4.0 scrub ok 2024-09-10T19:13:41.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- 172.21.15.83:0/4128376010 >> v1:172.21.15.83:6790/0 conn(0x7f64000fbeb0 legacy=0x7f6400109a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:41.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- 172.21.15.83:0/4128376010 shutdown_connections 2024-09-10T19:13:41.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- 172.21.15.83:0/4128376010 >> 172.21.15.83:0/4128376010 conn(0x7f6400075290 msgr2=0x7f64000f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:41.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- 172.21.15.83:0/4128376010 shutdown_connections 2024-09-10T19:13:41.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- 172.21.15.83:0/4128376010 wait complete. 2024-09-10T19:13:41.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 librados: starting msgr at 2024-09-10T19:13:41.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 librados: starting objecter 2024-09-10T19:13:41.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 Processor -- start 2024-09-10T19:13:41.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- start start 2024-09-10T19:13:41.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 librados: setting wanted keys 2024-09-10T19:13:41.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 librados: calling monclient init 2024-09-10T19:13:41.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f640011e840 con 0x7f64000fb3d0 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f640011fa00 con 0x7f640010ad00 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f64068ab640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6400120bc0 con 0x7f640010b200 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f64000fb3d0 0x7f640010a3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45020/0 (socket says 172.21.15.83:45020) 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fffff640 1 -- 172.21.15.83:0/286588446 learned_addr learned my addr 172.21.15.83:0/286588446 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fcff9640 10 client.?.objecter ms_handle_connect 0x7f64000fb3d0 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fcff9640 10 client.?.objecter ms_handle_connect 0x7f640010b200 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.455+0000 7f63fcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186269460 0 0) 0x7f640011e840 con 0x7f64000fb3d0 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ec0034e0 con 0x7f64000fb3d0 2024-09-10T19:13:41.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312991929 0 0) 0x7f6400120bc0 con 0x7f640010b200 2024-09-10T19:13:41.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640011e840 con 0x7f640010b200 2024-09-10T19:13:41.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 10 client.?.objecter ms_handle_connect 0x7f640010ad00 2024-09-10T19:13:41.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474378717 0 0) 0x7f640011e840 con 0x7f640010b200 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6400120bc0 con 0x7f640010b200 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57838181 0 0) 0x7f640011fa00 con 0x7f640010ad00 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640011e840 con 0x7f640010ad00 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f63e80017b0 con 0x7f640010b200 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517264070 0 0) 0x7f63ec0034e0 con 0x7f64000fb3d0 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f640011fa00 con 0x7f64000fb3d0 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f63f8001980 con 0x7f64000fb3d0 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4157638730 0 0) 0x7f6400120bc0 con 0x7f640010b200 2024-09-10T19:13:41.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 >> v1:172.21.15.165:6789/0 conn(0x7f640010ad00 legacy=0x7f640011cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 >> v1:172.21.15.83:6789/0 conn(0x7f64000fb3d0 legacy=0x7f640010a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400121dc0 con 0x7f640010b200 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6400120db0 con 0x7f640010b200 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f64068ab640 10 client.50141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6400121200 con 0x7f640010b200 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f64068ab640 1 librados: init done 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7f63e8001eb0 con 0x7f640010b200 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7f63e80037d0 con 0x7f640010b200 2024-09-10T19:13:41.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e0005010 con 0x7f640010b200 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7f63e8019bf0 con 0x7f640010b200 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(901..901 src has 849..901) v4 ==== 7363+0+0 (unknown 2916639721 0 0) 0x7f63e8087fd0 con 0x7f640010b200 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 10 client.50141.objecter ms_dispatch 0x7f6400000b90 osd_map(901..901 src has 849..901) v4 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 3 client.50141.objecter handle_osd_map got epochs [901,901] > 0 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.459+0000 7f63fcff9640 3 client.50141.objecter handle_osd_map decoding full epoch 901 2024-09-10T19:13:41.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.463+0000 7f63fcff9640 20 client.50141.objecter dump_active .. 0 homeless 2024-09-10T19:13:41.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.463+0000 7f63fcff9640 10 client.50141.objecter ms_handle_connect 0x7f63ec071790 2024-09-10T19:13:41.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.467+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7f63e8054870 con 0x7f640010b200 2024-09-10T19:13:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.643+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 --> v1:172.21.15.83:6800/3071793735 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63e0000b90 con 0x7f63ec071790 2024-09-10T19:13:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.663+0000 7f63fcff9640 1 -- 172.21.15.83:0/286588446 <== mgr.24526 v1:172.21.15.83:6800/3071793735 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+492111 (unknown 2965378022 0 2598047972) 0x7f63e0000b90 con 0x7f63ec071790 2024-09-10T19:13:41.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-10T19:13:41.680 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-10T19:13:41.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 10 librados: watch_flush enter 2024-09-10T19:13:41.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 10 librados: watch_flush exit 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 20 client.50141.objecter shutdown clearing up homeless session... 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 10 client.50141.objecter successfully canceled tick 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 >> v1:172.21.15.83:6800/3071793735 conn(0x7f63ec071790 legacy=0x7f63ec073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 >> v1:172.21.15.83:6790/0 conn(0x7f640010b200 legacy=0x7f6400119770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 shutdown_connections 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 >> 172.21.15.83:0/286588446 conn(0x7f6400075290 msgr2=0x7f64000f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 shutdown_connections 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 -- 172.21.15.83:0/286588446 wait complete. 2024-09-10T19:13:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:41.671+0000 7f64068ab640 1 librados: shutdown 2024-09-10T19:13:41.859 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.317842+0000 osd.6 (osd.6) 547 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:13:42.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.322084+0000 osd.6 (osd.6) 548 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.580198+0000 mgr.y (mgr.24526) 939 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.968299+0000 osd.3 (osd.3) 626 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:40.971720+0000 osd.3 (osd.3) 627 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.221463+0000 osd.4 (osd.4) 574 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.225018+0000 osd.4 (osd.4) 575 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.244992+0000 osd.7 (osd.7) 602 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:13:42.042 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.249782+0000 osd.7 (osd.7) 603 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:13:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.317842+0000 osd.6 (osd.6) 547 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:13:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.322084+0000 osd.6 (osd.6) 548 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:13:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.580198+0000 mgr.y (mgr.24526) 939 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.968299+0000 osd.3 (osd.3) 626 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:13:42.043 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:40.971720+0000 osd.3 (osd.3) 627 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:13:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.221463+0000 osd.4 (osd.4) 574 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:13:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.225018+0000 osd.4 (osd.4) 575 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:13:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.244992+0000 osd.7 (osd.7) 602 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:13:42.044 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:41 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.249782+0000 osd.7 (osd.7) 603 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:13:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.317842+0000 osd.6 (osd.6) 547 : cluster [DBG] 12.1 scrub starts 2024-09-10T19:13:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.322084+0000 osd.6 (osd.6) 548 : cluster [DBG] 12.1 scrub ok 2024-09-10T19:13:42.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.580198+0000 mgr.y (mgr.24526) 939 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.968299+0000 osd.3 (osd.3) 626 : cluster [DBG] 2.1e scrub starts 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:40.971720+0000 osd.3 (osd.3) 627 : cluster [DBG] 2.1e scrub ok 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.221463+0000 osd.4 (osd.4) 574 : cluster [DBG] 12.0 scrub starts 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.225018+0000 osd.4 (osd.4) 575 : cluster [DBG] 12.0 scrub ok 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.244992+0000 osd.7 (osd.7) 602 : cluster [DBG] 3.13 scrub starts 2024-09-10T19:13:42.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:41 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.249782+0000 osd.7 (osd.7) 603 : cluster [DBG] 3.13 scrub ok 2024-09-10T19:13:42.602 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":1517,"stamp":"2024-09-10T19:13:40.579752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596193,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":3032,"num_read_kb":2783,"num_write":1009,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2116,"ondisk_log_size":2116,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2454384,"kb_used_data":15600,"kb_used_omap":101,"kb_used_meta":2438618,"kb_avail":747506832,"statfs":{"total":767960285184,"available":765446995968,"internally_reserved":0,"allocated":15974400,"data_stored":5897779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103861,"internal_metadata":2497145419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006816"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":924,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628580+0000","last_change":"2024-09-10T19:12:05.963530+0000","last_active":"2024-09-10T19:12:54.628580+0000","last_peered":"2024-09-10T19:12:54.628580+0000","last_clean":"2024-09-10T19:12:54.628580+0000","last_became_active":"2024-09-10T18:49:24.370456+0000","last_became_peered":"2024-09-10T18:49:24.370456+0000","last_unstale":"2024-09-10T19:12:54.628580+0000","last_undegraded":"2024-09-10T19:12:54.628580+0000","last_fullsized":"2024-09-10T19:12:54.628580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.963371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:29.898480+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.963371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035715529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.632665+0000","last_change":"2024-09-10T19:12:16.081436+0000","last_active":"2024-09-10T19:12:54.632665+0000","last_peered":"2024-09-10T19:12:54.632665+0000","last_clean":"2024-09-10T19:12:54.632665+0000","last_became_active":"2024-09-10T18:49:17.614716+0000","last_became_peered":"2024-09-10T18:49:17.614716+0000","last_unstale":"2024-09-10T19:12:54.632665+0000","last_undegraded":"2024-09-10T19:12:54.632665+0000","last_fullsized":"2024-09-10T19:12:54.632665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:16.081285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.081285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.314031+0000","scrub_duration":0.0036441989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"881'35","reported_seq":978,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630369+0000","last_change":"2024-09-10T19:12:52.099506+0000","last_active":"2024-09-10T19:12:54.630369+0000","last_peered":"2024-09-10T19:12:54.630369+0000","last_clean":"2024-09-10T19:12:54.630369+0000","last_became_active":"2024-09-10T18:49:19.625971+0000","last_became_peered":"2024-09-10T18:49:19.625971+0000","last_unstale":"2024-09-10T19:12:54.630369+0000","last_undegraded":"2024-09-10T19:12:54.630369+0000","last_fullsized":"2024-09-10T19:12:54.630369+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"881'35","last_scrub_stamp":"2024-09-10T19:12:52.099394+0000","last_deep_scrub":"881'35","last_deep_scrub_stamp":"2024-09-10T19:11:11.949004+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.099394+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.338297+0000","scrub_duration":0.0033541360000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":936,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628948+0000","last_change":"2024-09-10T19:11:55.924492+0000","last_active":"2024-09-10T19:12:54.628948+0000","last_peered":"2024-09-10T19:12:54.628948+0000","last_clean":"2024-09-10T19:12:54.628948+0000","last_became_active":"2024-09-10T18:49:22.049899+0000","last_became_peered":"2024-09-10T18:49:22.049899+0000","last_unstale":"2024-09-10T19:12:54.628948+0000","last_undegraded":"2024-09-10T19:12:54.628948+0000","last_fullsized":"2024-09-10T19:12:54.628948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.924340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:52.890190+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.924340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.735622+0000","scrub_duration":0.0034557429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.804312+0000","last_change":"2024-09-10T19:13:05.804094+0000","last_active":"2024-09-10T19:13:05.804312+0000","last_peered":"2024-09-10T19:13:05.804312+0000","last_clean":"2024-09-10T19:13:05.804312+0000","last_became_active":"2024-09-10T18:54:37.588711+0000","last_became_peered":"2024-09-10T18:54:37.588711+0000","last_unstale":"2024-09-10T19:13:05.804312+0000","last_undegraded":"2024-09-10T19:13:05.804312+0000","last_fullsized":"2024-09-10T19:13:05.804312+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.803920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.803920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.317826+0000","scrub_duration":0.0037495219999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630651+0000","last_change":"2024-09-10T19:11:49.918802+0000","last_active":"2024-09-10T19:12:54.630651+0000","last_peered":"2024-09-10T19:12:54.630651+0000","last_clean":"2024-09-10T19:12:54.630651+0000","last_became_active":"2024-09-10T18:49:17.612045+0000","last_became_peered":"2024-09-10T18:49:17.612045+0000","last_unstale":"2024-09-10T19:12:54.630651+0000","last_undegraded":"2024-09-10T19:12:54.630651+0000","last_fullsized":"2024-09-10T19:12:54.630651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:49.918655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.918655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.174706+0000","scrub_duration":0.003368209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"877'32","reported_seq":990,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.988719+0000","last_change":"2024-09-10T19:13:20.988583+0000","last_active":"2024-09-10T19:13:20.988719+0000","last_peered":"2024-09-10T19:13:20.988719+0000","last_clean":"2024-09-10T19:13:20.988719+0000","last_became_active":"2024-09-10T18:49:19.629889+0000","last_became_peered":"2024-09-10T18:49:19.629889+0000","last_unstale":"2024-09-10T19:13:20.988719+0000","last_undegraded":"2024-09-10T19:13:20.988719+0000","last_fullsized":"2024-09-10T19:13:20.988719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'32","last_scrub_stamp":"2024-09-10T19:13:20.988309+0000","last_deep_scrub":"877'32","last_deep_scrub_stamp":"2024-09-10T19:11:51.184586+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.988309+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.979932+0000","scrub_duration":0.0056414710000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":970,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.244392+0000","last_change":"2024-09-10T19:13:20.244315+0000","last_active":"2024-09-10T19:13:20.244392+0000","last_peered":"2024-09-10T19:13:20.244392+0000","last_clean":"2024-09-10T19:13:20.244392+0000","last_became_active":"2024-09-10T18:49:41.639430+0000","last_became_peered":"2024-09-10T18:49:41.639430+0000","last_unstale":"2024-09-10T19:13:20.244392+0000","last_undegraded":"2024-09-10T19:13:20.244392+0000","last_fullsized":"2024-09-10T19:13:20.244392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:20.244160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.244160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.925366+0000","scrub_duration":0.0034380309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":592,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:29.317845+0000","last_change":"2024-09-10T19:13:29.317767+0000","last_active":"2024-09-10T19:13:29.317845+0000","last_peered":"2024-09-10T19:13:29.317845+0000","last_clean":"2024-09-10T19:13:29.317845+0000","last_became_active":"2024-09-10T19:10:47.694960+0000","last_became_peered":"2024-09-10T19:10:47.694960+0000","last_unstale":"2024-09-10T19:13:29.317845+0000","last_undegraded":"2024-09-10T19:13:29.317845+0000","last_fullsized":"2024-09-10T19:13:29.317845+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:29.317591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:06.476579+0000","last_clean_scrub_stamp":"2024-09-10T19:13:29.317591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:58.614222+0000","scrub_duration":0.0034066550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":866,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630656+0000","last_change":"2024-09-10T19:12:19.018785+0000","last_active":"2024-09-10T19:12:54.630656+0000","last_peered":"2024-09-10T19:12:54.630656+0000","last_clean":"2024-09-10T19:12:54.630656+0000","last_became_active":"2024-09-10T19:09:17.624105+0000","last_became_peered":"2024-09-10T19:09:17.624105+0000","last_unstale":"2024-09-10T19:12:54.630656+0000","last_undegraded":"2024-09-10T19:12:54.630656+0000","last_fullsized":"2024-09-10T19:12:54.630656+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:19.018625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.018625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:42.479971+0000","scrub_duration":0.0033536490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.782411+0000","last_change":"2024-09-10T19:12:09.858704+0000","last_active":"2024-09-10T19:12:54.782411+0000","last_peered":"2024-09-10T19:12:54.782411+0000","last_clean":"2024-09-10T19:12:54.782411+0000","last_became_active":"2024-09-10T18:49:17.610269+0000","last_became_peered":"2024-09-10T18:49:17.610269+0000","last_unstale":"2024-09-10T19:12:54.782411+0000","last_undegraded":"2024-09-10T19:12:54.782411+0000","last_fullsized":"2024-09-10T19:12:54.782411+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:09.858541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.858541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.070102+0000","scrub_duration":0.0033805240000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"859'46","reported_seq":1019,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781638+0000","last_change":"2024-09-10T19:12:19.758669+0000","last_active":"2024-09-10T19:12:54.781638+0000","last_peered":"2024-09-10T19:12:54.781638+0000","last_clean":"2024-09-10T19:12:54.781638+0000","last_became_active":"2024-09-10T19:09:17.624161+0000","last_became_peered":"2024-09-10T19:09:17.624161+0000","last_unstale":"2024-09-10T19:12:54.781638+0000","last_undegraded":"2024-09-10T19:12:54.781638+0000","last_fullsized":"2024-09-10T19:12:54.781638+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:12:19.758507+0000","last_deep_scrub":"859'46","last_deep_scrub_stamp":"2024-09-10T19:12:19.758507+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.758507+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.472714+0000","scrub_duration":0.0055441459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":594,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627622+0000","last_change":"2024-09-10T19:12:28.518152+0000","last_active":"2024-09-10T19:12:54.627622+0000","last_peered":"2024-09-10T19:12:54.627622+0000","last_clean":"2024-09-10T19:12:54.627622+0000","last_became_active":"2024-09-10T19:10:47.695582+0000","last_became_peered":"2024-09-10T19:10:47.695582+0000","last_unstale":"2024-09-10T19:12:54.627622+0000","last_undegraded":"2024-09-10T19:12:54.627622+0000","last_fullsized":"2024-09-10T19:12:54.627622+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.517993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:28.116084+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.517993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.359419+0000","scrub_duration":0.00357822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":960,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:25.666981+0000","last_change":"2024-09-10T19:13:25.666878+0000","last_active":"2024-09-10T19:13:25.666981+0000","last_peered":"2024-09-10T19:13:25.666981+0000","last_clean":"2024-09-10T19:13:25.666981+0000","last_became_active":"2024-09-10T19:09:17.623986+0000","last_became_peered":"2024-09-10T19:09:17.623986+0000","last_unstale":"2024-09-10T19:13:25.666981+0000","last_undegraded":"2024-09-10T19:13:25.666981+0000","last_fullsized":"2024-09-10T19:13:25.666981+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:25.666738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:25.666738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.854461+0000","scrub_duration":0.0034230649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":807,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:28.047888+0000","last_change":"2024-09-10T19:13:28.047815+0000","last_active":"2024-09-10T19:13:28.047888+0000","last_peered":"2024-09-10T19:13:28.047888+0000","last_clean":"2024-09-10T19:13:28.047888+0000","last_became_active":"2024-09-10T19:09:17.622887+0000","last_became_peered":"2024-09-10T19:09:17.622887+0000","last_unstale":"2024-09-10T19:13:28.047888+0000","last_undegraded":"2024-09-10T19:13:28.047888+0000","last_fullsized":"2024-09-10T19:13:28.047888+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:28.047656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:28.047656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.945407+0000","scrub_duration":0.0033500510000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":956,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780662+0000","last_change":"2024-09-10T19:12:39.694607+0000","last_active":"2024-09-10T19:12:54.780662+0000","last_peered":"2024-09-10T19:12:54.780662+0000","last_clean":"2024-09-10T19:12:54.780662+0000","last_became_active":"2024-09-10T18:49:17.611581+0000","last_became_peered":"2024-09-10T18:49:17.611581+0000","last_unstale":"2024-09-10T19:12:54.780662+0000","last_undegraded":"2024-09-10T19:12:54.780662+0000","last_fullsized":"2024-09-10T19:12:54.780662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:39.694347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:39.694347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:53.849112+0000","scrub_duration":0.004782723,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"859'29","reported_seq":1000,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.843533+0000","last_change":"2024-09-10T19:12:55.843394+0000","last_active":"2024-09-10T19:12:55.843533+0000","last_peered":"2024-09-10T19:12:55.843533+0000","last_clean":"2024-09-10T19:12:55.843533+0000","last_became_active":"2024-09-10T18:49:19.627185+0000","last_became_peered":"2024-09-10T18:49:19.627185+0000","last_unstale":"2024-09-10T19:12:55.843533+0000","last_undegraded":"2024-09-10T19:12:55.843533+0000","last_fullsized":"2024-09-10T19:12:55.843533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'29","last_scrub_stamp":"2024-09-10T19:12:55.843230+0000","last_deep_scrub":"859'29","last_deep_scrub_stamp":"2024-09-10T19:12:55.843230+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.843230+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.269290+0000","scrub_duration":0.0059692130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":604,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:35.342674+0000","last_change":"2024-09-10T19:13:35.342593+0000","last_active":"2024-09-10T19:13:35.342674+0000","last_peered":"2024-09-10T19:13:35.342674+0000","last_clean":"2024-09-10T19:13:35.342674+0000","last_became_active":"2024-09-10T19:10:47.692260+0000","last_became_peered":"2024-09-10T19:10:47.692260+0000","last_unstale":"2024-09-10T19:13:35.342674+0000","last_undegraded":"2024-09-10T19:13:35.342674+0000","last_fullsized":"2024-09-10T19:13:35.342674+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:35.342448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:35.342448+0000","last_clean_scrub_stamp":"2024-09-10T19:13:35.342448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.179956+0000","scrub_duration":0.003417865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":955,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629377+0000","last_change":"2024-09-10T19:12:18.028526+0000","last_active":"2024-09-10T19:12:54.629377+0000","last_peered":"2024-09-10T19:12:54.629377+0000","last_clean":"2024-09-10T19:12:54.629377+0000","last_became_active":"2024-09-10T19:09:17.624498+0000","last_became_peered":"2024-09-10T19:09:17.624498+0000","last_unstale":"2024-09-10T19:12:54.629377+0000","last_undegraded":"2024-09-10T19:12:54.629377+0000","last_fullsized":"2024-09-10T19:12:54.629377+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.028367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.028367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.903157+0000","scrub_duration":0.003381682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629826+0000","last_change":"2024-09-10T19:12:38.983791+0000","last_active":"2024-09-10T19:12:54.629826+0000","last_peered":"2024-09-10T19:12:54.629826+0000","last_clean":"2024-09-10T19:12:54.629826+0000","last_became_active":"2024-09-10T19:09:17.624188+0000","last_became_peered":"2024-09-10T19:09:17.624188+0000","last_unstale":"2024-09-10T19:12:54.629826+0000","last_undegraded":"2024-09-10T19:12:54.629826+0000","last_fullsized":"2024-09-10T19:12:54.629826+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:38.983644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:24.622321+0000","last_clean_scrub_stamp":"2024-09-10T19:12:38.983644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.093397+0000","scrub_duration":0.0031681270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"93'81","reported_seq":460,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776821+0000","last_change":"2024-09-10T19:12:16.781316+0000","last_active":"2024-09-10T19:12:54.776821+0000","last_peered":"2024-09-10T19:12:54.776821+0000","last_clean":"2024-09-10T19:12:54.776821+0000","last_became_active":"2024-09-10T19:02:03.643069+0000","last_became_peered":"2024-09-10T19:02:03.643069+0000","last_unstale":"2024-09-10T19:12:54.776821+0000","last_undegraded":"2024-09-10T19:12:54.776821+0000","last_fullsized":"2024-09-10T19:12:54.776821+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:16.781157+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:09:46.330407+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.781157+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:28.455349+0000","scrub_duration":0.0044085089999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"859'51","reported_seq":1049,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:32.485890+0000","last_change":"2024-09-10T19:13:32.485797+0000","last_active":"2024-09-10T19:13:32.485890+0000","last_peered":"2024-09-10T19:13:32.485890+0000","last_clean":"2024-09-10T19:13:32.485890+0000","last_became_active":"2024-09-10T19:09:17.613942+0000","last_became_peered":"2024-09-10T19:09:17.613942+0000","last_unstale":"2024-09-10T19:13:32.485890+0000","last_undegraded":"2024-09-10T19:13:32.485890+0000","last_fullsized":"2024-09-10T19:13:32.485890+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'51","last_scrub_stamp":"2024-09-10T19:13:32.485624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:32.485624+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.547333+0000","scrub_duration":0.005941979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":949,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780132+0000","last_change":"2024-09-10T19:12:28.794331+0000","last_active":"2024-09-10T19:12:54.780132+0000","last_peered":"2024-09-10T19:12:54.780132+0000","last_clean":"2024-09-10T19:12:54.780132+0000","last_became_active":"2024-09-10T18:49:17.610776+0000","last_became_peered":"2024-09-10T18:49:17.610776+0000","last_unstale":"2024-09-10T19:12:54.780132+0000","last_undegraded":"2024-09-10T19:12:54.780132+0000","last_fullsized":"2024-09-10T19:12:54.780132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.794119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:28.794119+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.794119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.934709+0000","scrub_duration":0.0031531430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"879'16","reported_seq":802,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.922012+0000","last_change":"2024-09-10T19:12:56.399010+0000","last_active":"2024-09-10T19:13:26.922012+0000","last_peered":"2024-09-10T19:13:26.922012+0000","last_clean":"2024-09-10T19:13:26.922012+0000","last_became_active":"2024-09-10T19:10:47.694901+0000","last_became_peered":"2024-09-10T19:10:47.694901+0000","last_unstale":"2024-09-10T19:13:26.922012+0000","last_undegraded":"2024-09-10T19:13:26.922012+0000","last_fullsized":"2024-09-10T19:13:26.922012+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'16","last_scrub_stamp":"2024-09-10T19:12:56.398838+0000","last_deep_scrub":"879'16","last_deep_scrub_stamp":"2024-09-10T19:12:56.398838+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.398838+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.826926+0000","scrub_duration":0.0042448329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":972,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:34.688211+0000","last_change":"2024-09-10T19:13:34.688134+0000","last_active":"2024-09-10T19:13:34.688211+0000","last_peered":"2024-09-10T19:13:34.688211+0000","last_clean":"2024-09-10T19:13:34.688211+0000","last_became_active":"2024-09-10T18:49:24.051437+0000","last_became_peered":"2024-09-10T18:49:24.051437+0000","last_unstale":"2024-09-10T19:13:34.688211+0000","last_undegraded":"2024-09-10T19:13:34.688211+0000","last_fullsized":"2024-09-10T19:13:34.688211+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:34.687968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:27.103997+0000","last_clean_scrub_stamp":"2024-09-10T19:13:34.687968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:38.739467+0000","scrub_duration":0.0034660530000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778034+0000","last_change":"2024-09-10T19:12:17.451314+0000","last_active":"2024-09-10T19:12:54.778034+0000","last_peered":"2024-09-10T19:12:54.778034+0000","last_clean":"2024-09-10T19:12:54.778034+0000","last_became_active":"2024-09-10T19:09:17.614891+0000","last_became_peered":"2024-09-10T19:09:17.614891+0000","last_unstale":"2024-09-10T19:12:54.778034+0000","last_undegraded":"2024-09-10T19:12:54.778034+0000","last_fullsized":"2024-09-10T19:12:54.778034+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:17.451166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:38.781169+0000","last_clean_scrub_stamp":"2024-09-10T19:12:17.451166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:43.562729+0000","scrub_duration":0.0035160650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"93'81","reported_seq":1021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777160+0000","last_change":"2024-09-10T19:12:37.757201+0000","last_active":"2024-09-10T19:12:54.777160+0000","last_peered":"2024-09-10T19:12:54.777160+0000","last_clean":"2024-09-10T19:12:54.777160+0000","last_became_active":"2024-09-10T19:02:03.640426+0000","last_became_peered":"2024-09-10T19:02:03.640426+0000","last_unstale":"2024-09-10T19:12:54.777160+0000","last_undegraded":"2024-09-10T19:12:54.777160+0000","last_fullsized":"2024-09-10T19:12:54.777160+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:37.756949+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.756949+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.451929+0000","scrub_duration":0.0065141599999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"859'25","reported_seq":976,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628795+0000","last_change":"2024-09-10T19:12:07.985950+0000","last_active":"2024-09-10T19:12:54.628795+0000","last_peered":"2024-09-10T19:12:54.628795+0000","last_clean":"2024-09-10T19:12:54.628795+0000","last_became_active":"2024-09-10T18:49:19.633531+0000","last_became_peered":"2024-09-10T18:49:19.633531+0000","last_unstale":"2024-09-10T19:12:54.628795+0000","last_undegraded":"2024-09-10T19:12:54.628795+0000","last_fullsized":"2024-09-10T19:12:54.628795+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'25","last_scrub_stamp":"2024-09-10T19:12:07.985723+0000","last_deep_scrub":"95'13","last_deep_scrub_stamp":"2024-09-10T18:50:39.845392+0000","last_clean_scrub_stamp":"2024-09-10T19:12:07.985723+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:33.206339+0000","scrub_duration":0.0064713729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":893,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.973839+0000","last_change":"2024-09-10T19:12:54.973728+0000","last_active":"2024-09-10T19:12:54.973839+0000","last_peered":"2024-09-10T19:12:54.973839+0000","last_clean":"2024-09-10T19:12:54.973839+0000","last_became_active":"2024-09-10T19:09:17.627988+0000","last_became_peered":"2024-09-10T19:09:17.627988+0000","last_unstale":"2024-09-10T19:12:54.973839+0000","last_undegraded":"2024-09-10T19:12:54.973839+0000","last_fullsized":"2024-09-10T19:12:54.973839+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.973568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:58:52.884077+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.973568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.631809+0000","scrub_duration":0.0035731980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":597,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:12.355934+0000","last_change":"2024-09-10T19:13:12.355799+0000","last_active":"2024-09-10T19:13:12.355934+0000","last_peered":"2024-09-10T19:13:12.355934+0000","last_clean":"2024-09-10T19:13:12.355934+0000","last_became_active":"2024-09-10T19:10:47.692181+0000","last_became_peered":"2024-09-10T19:10:47.692181+0000","last_unstale":"2024-09-10T19:13:12.355934+0000","last_undegraded":"2024-09-10T19:13:12.355934+0000","last_fullsized":"2024-09-10T19:13:12.355934+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:12.355544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:12.355544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.491359+0000","scrub_duration":0.0034917949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":592,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629079+0000","last_change":"2024-09-10T19:12:29.503496+0000","last_active":"2024-09-10T19:12:54.629079+0000","last_peered":"2024-09-10T19:12:54.629079+0000","last_clean":"2024-09-10T19:12:54.629079+0000","last_became_active":"2024-09-10T19:10:47.697379+0000","last_became_peered":"2024-09-10T19:10:47.697379+0000","last_unstale":"2024-09-10T19:12:54.629079+0000","last_undegraded":"2024-09-10T19:12:54.629079+0000","last_fullsized":"2024-09-10T19:12:54.629079+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:29.503343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.503343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.655335+0000","scrub_duration":0.0035948239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777367+0000","last_change":"2024-09-10T19:12:52.446169+0000","last_active":"2024-09-10T19:12:54.777367+0000","last_peered":"2024-09-10T19:12:54.777367+0000","last_clean":"2024-09-10T19:12:54.777367+0000","last_became_active":"2024-09-10T19:09:17.615697+0000","last_became_peered":"2024-09-10T19:09:17.615697+0000","last_unstale":"2024-09-10T19:12:54.777367+0000","last_undegraded":"2024-09-10T19:12:54.777367+0000","last_fullsized":"2024-09-10T19:12:54.777367+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.446009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:34.850823+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.446009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.085539+0000","scrub_duration":0.003327637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"93'81","reported_seq":453,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778506+0000","last_change":"2024-09-10T19:12:44.724285+0000","last_active":"2024-09-10T19:12:54.778506+0000","last_peered":"2024-09-10T19:12:54.778506+0000","last_clean":"2024-09-10T19:12:54.778506+0000","last_became_active":"2024-09-10T19:02:03.642988+0000","last_became_peered":"2024-09-10T19:02:03.642988+0000","last_unstale":"2024-09-10T19:12:54.778506+0000","last_undegraded":"2024-09-10T19:12:54.778506+0000","last_fullsized":"2024-09-10T19:12:54.778506+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:44.724127+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:02:50.724359+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.724127+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.988635+0000","scrub_duration":0.003706642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"859'39","reported_seq":1016,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628380+0000","last_change":"2024-09-10T19:12:20.014879+0000","last_active":"2024-09-10T19:12:54.628380+0000","last_peered":"2024-09-10T19:12:54.628380+0000","last_clean":"2024-09-10T19:12:54.628380+0000","last_became_active":"2024-09-10T19:09:17.625369+0000","last_became_peered":"2024-09-10T19:09:17.625369+0000","last_unstale":"2024-09-10T19:12:54.628380+0000","last_undegraded":"2024-09-10T19:12:54.628380+0000","last_fullsized":"2024-09-10T19:12:54.628380+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'39","last_scrub_stamp":"2024-09-10T19:12:20.014768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:20.014768+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:48.537536+0000","scrub_duration":0.0047882580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":885,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777456+0000","last_change":"2024-09-10T19:12:50.955288+0000","last_active":"2024-09-10T19:12:54.777456+0000","last_peered":"2024-09-10T19:12:54.777456+0000","last_clean":"2024-09-10T19:12:54.777456+0000","last_became_active":"2024-09-10T19:09:17.624964+0000","last_became_peered":"2024-09-10T19:09:17.624964+0000","last_unstale":"2024-09-10T19:12:54.777456+0000","last_undegraded":"2024-09-10T19:12:54.777456+0000","last_fullsized":"2024-09-10T19:12:54.777456+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.955131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:17.868358+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.955131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.325447+0000","scrub_duration":0.0032922939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":866,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:57.057858+0000","last_change":"2024-09-10T19:12:57.057682+0000","last_active":"2024-09-10T19:12:57.057858+0000","last_peered":"2024-09-10T19:12:57.057858+0000","last_clean":"2024-09-10T19:12:57.057858+0000","last_became_active":"2024-09-10T19:09:17.628731+0000","last_became_peered":"2024-09-10T19:09:17.628731+0000","last_unstale":"2024-09-10T19:12:57.057858+0000","last_undegraded":"2024-09-10T19:12:57.057858+0000","last_fullsized":"2024-09-10T19:12:57.057858+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.057429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.057429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.533873+0000","scrub_duration":0.003824095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":967,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.652617+0000","last_change":"2024-09-10T19:13:26.652537+0000","last_active":"2024-09-10T19:13:26.652617+0000","last_peered":"2024-09-10T19:13:26.652617+0000","last_clean":"2024-09-10T19:13:26.652617+0000","last_became_active":"2024-09-10T19:09:17.624545+0000","last_became_peered":"2024-09-10T19:09:17.624545+0000","last_unstale":"2024-09-10T19:13:26.652617+0000","last_undegraded":"2024-09-10T19:13:26.652617+0000","last_fullsized":"2024-09-10T19:13:26.652617+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:26.652388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:11.771295+0000","last_clean_scrub_stamp":"2024-09-10T19:13:26.652388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.963192+0000","scrub_duration":0.003625578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":789,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779295+0000","last_change":"2024-09-10T19:12:35.251646+0000","last_active":"2024-09-10T19:12:54.779295+0000","last_peered":"2024-09-10T19:12:54.779295+0000","last_clean":"2024-09-10T19:12:54.779295+0000","last_became_active":"2024-09-10T19:09:17.626924+0000","last_became_peered":"2024-09-10T19:09:17.626924+0000","last_unstale":"2024-09-10T19:12:54.779295+0000","last_undegraded":"2024-09-10T19:12:54.779295+0000","last_fullsized":"2024-09-10T19:12:54.779295+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:35.251459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:35.251459+0000","last_clean_scrub_stamp":"2024-09-10T19:12:35.251459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.999862+0000","scrub_duration":0.0036221130000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"859'29","reported_seq":1018,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:27.298327+0000","last_change":"2024-09-10T19:13:27.298201+0000","last_active":"2024-09-10T19:13:27.298327+0000","last_peered":"2024-09-10T19:13:27.298327+0000","last_clean":"2024-09-10T19:13:27.298327+0000","last_became_active":"2024-09-10T18:49:19.628221+0000","last_became_peered":"2024-09-10T18:49:19.628221+0000","last_unstale":"2024-09-10T19:13:27.298327+0000","last_undegraded":"2024-09-10T19:13:27.298327+0000","last_fullsized":"2024-09-10T19:13:27.298327+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'29","last_scrub_stamp":"2024-09-10T19:13:27.298024+0000","last_deep_scrub":"859'29","last_deep_scrub_stamp":"2024-09-10T19:13:27.298024+0000","last_clean_scrub_stamp":"2024-09-10T19:13:27.298024+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:56.120005+0000","scrub_duration":0.0054813450000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":970,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:21.395350+0000","last_change":"2024-09-10T19:13:21.395273+0000","last_active":"2024-09-10T19:13:21.395350+0000","last_peered":"2024-09-10T19:13:21.395350+0000","last_clean":"2024-09-10T19:13:21.395350+0000","last_became_active":"2024-09-10T19:09:17.633970+0000","last_became_peered":"2024-09-10T19:09:17.633970+0000","last_unstale":"2024-09-10T19:13:21.395350+0000","last_undegraded":"2024-09-10T19:13:21.395350+0000","last_fullsized":"2024-09-10T19:13:21.395350+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:21.395118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.129587+0000","last_clean_scrub_stamp":"2024-09-10T19:13:21.395118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.732701+0000","scrub_duration":0.0037239640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"879'16","reported_seq":802,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.921869+0000","last_change":"2024-09-10T19:13:05.350996+0000","last_active":"2024-09-10T19:13:26.921869+0000","last_peered":"2024-09-10T19:13:26.921869+0000","last_clean":"2024-09-10T19:13:26.921869+0000","last_became_active":"2024-09-10T19:10:47.695744+0000","last_became_peered":"2024-09-10T19:10:47.695744+0000","last_unstale":"2024-09-10T19:13:26.921869+0000","last_undegraded":"2024-09-10T19:13:26.921869+0000","last_fullsized":"2024-09-10T19:13:26.921869+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'16","last_scrub_stamp":"2024-09-10T19:13:05.350810+0000","last_deep_scrub":"879'16","last_deep_scrub_stamp":"2024-09-10T19:13:05.350810+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.350810+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:11.865095+0000","scrub_duration":0.0043700359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"77'1","reported_seq":935,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629123+0000","last_change":"2024-09-10T19:12:09.989957+0000","last_active":"2024-09-10T19:12:54.629123+0000","last_peered":"2024-09-10T19:12:54.629123+0000","last_clean":"2024-09-10T19:12:54.629123+0000","last_became_active":"2024-09-10T18:49:24.366060+0000","last_became_peered":"2024-09-10T18:49:24.366060+0000","last_unstale":"2024-09-10T19:12:54.629123+0000","last_undegraded":"2024-09-10T19:12:54.629123+0000","last_fullsized":"2024-09-10T19:12:54.629123+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'1","last_scrub_stamp":"2024-09-10T19:12:09.989770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.989770+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:26.724461+0000","scrub_duration":0.0040537400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":801,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.057617+0000","last_change":"2024-09-10T19:13:17.057544+0000","last_active":"2024-09-10T19:13:17.057617+0000","last_peered":"2024-09-10T19:13:17.057617+0000","last_clean":"2024-09-10T19:13:17.057617+0000","last_became_active":"2024-09-10T19:09:17.630225+0000","last_became_peered":"2024-09-10T19:09:17.630225+0000","last_unstale":"2024-09-10T19:13:17.057617+0000","last_undegraded":"2024-09-10T19:13:17.057617+0000","last_fullsized":"2024-09-10T19:13:17.057617+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.057383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.057383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:39.844914+0000","scrub_duration":0.003413592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"93'81","reported_seq":455,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629055+0000","last_change":"2024-09-10T19:12:15.916614+0000","last_active":"2024-09-10T19:12:54.629055+0000","last_peered":"2024-09-10T19:12:54.629055+0000","last_clean":"2024-09-10T19:12:54.629055+0000","last_became_active":"2024-09-10T19:02:03.618024+0000","last_became_peered":"2024-09-10T19:02:03.618024+0000","last_unstale":"2024-09-10T19:12:54.629055+0000","last_undegraded":"2024-09-10T19:12:54.629055+0000","last_fullsized":"2024-09-10T19:12:54.629055+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:15.916455+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:15.916455+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.319759+0000","scrub_duration":0.0044933289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"859'36","reported_seq":1013,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.161475+0000","last_change":"2024-09-10T19:13:04.161342+0000","last_active":"2024-09-10T19:13:04.161475+0000","last_peered":"2024-09-10T19:13:04.161475+0000","last_clean":"2024-09-10T19:13:04.161475+0000","last_became_active":"2024-09-10T19:09:17.623922+0000","last_became_peered":"2024-09-10T19:09:17.623922+0000","last_unstale":"2024-09-10T19:13:04.161475+0000","last_undegraded":"2024-09-10T19:13:04.161475+0000","last_fullsized":"2024-09-10T19:13:04.161475+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'36","last_scrub_stamp":"2024-09-10T19:13:04.161180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.161180+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.680463+0000","scrub_duration":0.0052040009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:56.421748+0000","last_change":"2024-09-10T19:12:56.421657+0000","last_active":"2024-09-10T19:12:56.421748+0000","last_peered":"2024-09-10T19:12:56.421748+0000","last_clean":"2024-09-10T19:12:56.421748+0000","last_became_active":"2024-09-10T18:49:17.612244+0000","last_became_peered":"2024-09-10T18:49:17.612244+0000","last_unstale":"2024-09-10T19:12:56.421748+0000","last_undegraded":"2024-09-10T19:12:56.421748+0000","last_fullsized":"2024-09-10T19:12:56.421748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.421509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:19.756020+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.421509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.031135+0000","scrub_duration":0.0033371899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":604,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:08.335492+0000","last_change":"2024-09-10T19:13:08.335428+0000","last_active":"2024-09-10T19:13:08.335492+0000","last_peered":"2024-09-10T19:13:08.335492+0000","last_clean":"2024-09-10T19:13:08.335492+0000","last_became_active":"2024-09-10T19:10:47.689065+0000","last_became_peered":"2024-09-10T19:10:47.689065+0000","last_unstale":"2024-09-10T19:13:08.335492+0000","last_undegraded":"2024-09-10T19:13:08.335492+0000","last_fullsized":"2024-09-10T19:13:08.335492+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.335311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:41.770525+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.335311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:33.200985+0000","scrub_duration":0.0030415720000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"859'6","reported_seq":2391,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:34.240351+0000","last_change":"2024-09-10T19:12:54.045560+0000","last_active":"2024-09-10T19:13:34.240351+0000","last_peered":"2024-09-10T19:13:34.240351+0000","last_clean":"2024-09-10T19:13:34.240351+0000","last_became_active":"2024-09-10T19:09:17.624538+0000","last_became_peered":"2024-09-10T19:09:17.624538+0000","last_unstale":"2024-09-10T19:13:34.240351+0000","last_undegraded":"2024-09-10T19:13:34.240351+0000","last_fullsized":"2024-09-10T19:13:34.240351+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'6","last_scrub_stamp":"2024-09-10T19:12:54.045409+0000","last_deep_scrub":"859'6","last_deep_scrub_stamp":"2024-09-10T19:09:27.654271+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.045409+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:04.756787+0000","scrub_duration":0.0042215819999999998,"objects_trimmed":0,"snaptrim_duration":0.0042911650000000004,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1427,"num_read_kb":1422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:35.278260+0000","last_change":"2024-09-10T19:13:35.278156+0000","last_active":"2024-09-10T19:13:35.278260+0000","last_peered":"2024-09-10T19:13:35.278260+0000","last_clean":"2024-09-10T19:13:35.278260+0000","last_became_active":"2024-09-10T18:49:24.364844+0000","last_became_peered":"2024-09-10T18:49:24.364844+0000","last_unstale":"2024-09-10T19:13:35.278260+0000","last_undegraded":"2024-09-10T19:13:35.278260+0000","last_fullsized":"2024-09-10T19:13:35.278260+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:35.278002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:49.386404+0000","last_clean_scrub_stamp":"2024-09-10T19:13:35.278002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:59.995484+0000","scrub_duration":0.0030851030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.493101+0000","last_change":"2024-09-10T19:13:04.493044+0000","last_active":"2024-09-10T19:13:04.493101+0000","last_peered":"2024-09-10T19:13:04.493101+0000","last_clean":"2024-09-10T19:13:04.493101+0000","last_became_active":"2024-09-10T18:54:37.674460+0000","last_became_peered":"2024-09-10T18:54:37.674460+0000","last_unstale":"2024-09-10T19:13:04.493101+0000","last_undegraded":"2024-09-10T19:13:04.493101+0000","last_fullsized":"2024-09-10T19:13:04.493101+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.492972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:34.872832+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.492972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:05.971471+0000","scrub_duration":0.0023311849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"93'81","reported_seq":465,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628010+0000","last_change":"2024-09-10T19:12:35.370263+0000","last_active":"2024-09-10T19:12:54.628010+0000","last_peered":"2024-09-10T19:12:54.628010+0000","last_clean":"2024-09-10T19:12:54.628010+0000","last_became_active":"2024-09-10T19:02:03.639765+0000","last_became_peered":"2024-09-10T19:02:03.639765+0000","last_unstale":"2024-09-10T19:12:54.628010+0000","last_undegraded":"2024-09-10T19:12:54.628010+0000","last_fullsized":"2024-09-10T19:12:54.628010+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:35.370049+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:06:46.615570+0000","last_clean_scrub_stamp":"2024-09-10T19:12:35.370049+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:03.411760+0000","scrub_duration":0.0043445589999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"859'21","reported_seq":987,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:22.802330+0000","last_change":"2024-09-10T19:13:22.802218+0000","last_active":"2024-09-10T19:13:22.802330+0000","last_peered":"2024-09-10T19:13:22.802330+0000","last_clean":"2024-09-10T19:13:22.802330+0000","last_became_active":"2024-09-10T18:49:19.630368+0000","last_became_peered":"2024-09-10T18:49:19.630368+0000","last_unstale":"2024-09-10T19:13:22.802330+0000","last_undegraded":"2024-09-10T19:13:22.802330+0000","last_fullsized":"2024-09-10T19:13:22.802330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:22.802012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:22.802012+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:51.225843+0000","scrub_duration":0.0046020039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":912,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:35.127388+0000","last_change":"2024-09-10T19:13:35.127303+0000","last_active":"2024-09-10T19:13:35.127388+0000","last_peered":"2024-09-10T19:13:35.127388+0000","last_clean":"2024-09-10T19:13:35.127388+0000","last_became_active":"2024-09-10T19:09:17.619020+0000","last_became_peered":"2024-09-10T19:09:17.619020+0000","last_unstale":"2024-09-10T19:13:35.127388+0000","last_undegraded":"2024-09-10T19:13:35.127388+0000","last_fullsized":"2024-09-10T19:13:35.127388+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:35.127150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:18.069100+0000","last_clean_scrub_stamp":"2024-09-10T19:13:35.127150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:57.681072+0000","scrub_duration":0.0035019460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"554'12","reported_seq":1228,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.923311+0000","last_change":"2024-09-10T19:12:03.932393+0000","last_active":"2024-09-10T19:13:26.923311+0000","last_peered":"2024-09-10T19:13:26.923311+0000","last_clean":"2024-09-10T19:13:26.923311+0000","last_became_active":"2024-09-10T18:49:22.062355+0000","last_became_peered":"2024-09-10T18:49:22.062355+0000","last_unstale":"2024-09-10T19:13:26.923311+0000","last_undegraded":"2024-09-10T19:13:26.923311+0000","last_fullsized":"2024-09-10T19:13:26.923311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'12","last_scrub_stamp":"2024-09-10T19:12:03.932114+0000","last_deep_scrub":"554'12","last_deep_scrub_stamp":"2024-09-10T19:02:59.017508+0000","last_clean_scrub_stamp":"2024-09-10T19:12:03.932114+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003643079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"112'2","reported_seq":977,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627086+0000","last_change":"2024-09-10T19:12:48.447199+0000","last_active":"2024-09-10T19:12:54.627086+0000","last_peered":"2024-09-10T19:12:54.627086+0000","last_clean":"2024-09-10T19:12:54.627086+0000","last_became_active":"2024-09-10T18:49:22.059926+0000","last_became_peered":"2024-09-10T18:49:22.059926+0000","last_unstale":"2024-09-10T19:12:54.627086+0000","last_undegraded":"2024-09-10T19:12:54.627086+0000","last_fullsized":"2024-09-10T19:12:54.627086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-09-10T19:12:48.446936+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-09-10T18:53:12.153557+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.446936+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:50.864977+0000","scrub_duration":0.004606176,"objects_trimmed":0,"snaptrim_duration":0.0012876400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":602,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.301017+0000","last_change":"2024-09-10T19:13:23.300939+0000","last_active":"2024-09-10T19:13:23.301017+0000","last_peered":"2024-09-10T19:13:23.301017+0000","last_clean":"2024-09-10T19:13:23.301017+0000","last_became_active":"2024-09-10T19:10:47.691892+0000","last_became_peered":"2024-09-10T19:10:47.691892+0000","last_unstale":"2024-09-10T19:13:23.301017+0000","last_undegraded":"2024-09-10T19:13:23.301017+0000","last_fullsized":"2024-09-10T19:13:23.301017+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.300782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:57:50.792298+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.300782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.347037+0000","scrub_duration":0.003615732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628117+0000","last_change":"2024-09-10T19:12:40.989724+0000","last_active":"2024-09-10T19:12:54.628117+0000","last_peered":"2024-09-10T19:12:54.628117+0000","last_clean":"2024-09-10T19:12:54.628117+0000","last_became_active":"2024-09-10T18:54:37.726291+0000","last_became_peered":"2024-09-10T18:54:37.726291+0000","last_unstale":"2024-09-10T19:12:54.628117+0000","last_undegraded":"2024-09-10T19:12:54.628117+0000","last_fullsized":"2024-09-10T19:12:54.628117+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:40.989570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:40.989570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:08.420730+0000","scrub_duration":0.003425071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"93'81","reported_seq":538,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.427126+0000","last_change":"2024-09-10T19:13:06.427010+0000","last_active":"2024-09-10T19:13:06.427126+0000","last_peered":"2024-09-10T19:13:06.427126+0000","last_clean":"2024-09-10T19:13:06.427126+0000","last_became_active":"2024-09-10T19:02:03.612956+0000","last_became_peered":"2024-09-10T19:02:03.612956+0000","last_unstale":"2024-09-10T19:13:06.427126+0000","last_undegraded":"2024-09-10T19:13:06.427126+0000","last_fullsized":"2024-09-10T19:13:06.427126+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:06.426853+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.426853+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.227721+0000","scrub_duration":0.0039253079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"859'66","reported_seq":1227,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.923503+0000","last_change":"2024-09-10T19:13:26.099884+0000","last_active":"2024-09-10T19:13:26.923503+0000","last_peered":"2024-09-10T19:13:26.923503+0000","last_clean":"2024-09-10T19:13:26.923503+0000","last_became_active":"2024-09-10T19:09:17.624136+0000","last_became_peered":"2024-09-10T19:09:17.624136+0000","last_unstale":"2024-09-10T19:13:26.923503+0000","last_undegraded":"2024-09-10T19:13:26.923503+0000","last_fullsized":"2024-09-10T19:13:26.923503+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'66","last_scrub_stamp":"2024-09-10T19:13:26.099715+0000","last_deep_scrub":"113'40","last_deep_scrub_stamp":"2024-09-10T18:53:33.355780+0000","last_clean_scrub_stamp":"2024-09-10T19:13:26.099715+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.191070+0000","scrub_duration":0.0060579420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":953,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.783976+0000","last_change":"2024-09-10T19:12:51.757296+0000","last_active":"2024-09-10T19:12:54.783976+0000","last_peered":"2024-09-10T19:12:54.783976+0000","last_clean":"2024-09-10T19:12:54.783976+0000","last_became_active":"2024-09-10T18:49:17.616029+0000","last_became_peered":"2024-09-10T18:49:17.616029+0000","last_unstale":"2024-09-10T19:12:54.783976+0000","last_undegraded":"2024-09-10T19:12:54.783976+0000","last_fullsized":"2024-09-10T19:12:54.783976+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.757142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.757142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.706460+0000","scrub_duration":0.0032464379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'12","reported_seq":1242,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.923318+0000","last_change":"2024-09-10T19:12:41.096730+0000","last_active":"2024-09-10T19:13:26.923318+0000","last_peered":"2024-09-10T19:13:26.923318+0000","last_clean":"2024-09-10T19:13:26.923318+0000","last_became_active":"2024-09-10T18:49:22.061644+0000","last_became_peered":"2024-09-10T18:49:22.061644+0000","last_unstale":"2024-09-10T19:13:26.923318+0000","last_undegraded":"2024-09-10T19:13:26.923318+0000","last_fullsized":"2024-09-10T19:13:26.923318+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'12","last_scrub_stamp":"2024-09-10T19:12:41.096641+0000","last_deep_scrub":"104'12","last_deep_scrub_stamp":"2024-09-10T18:52:07.501543+0000","last_clean_scrub_stamp":"2024-09-10T19:12:41.096641+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.538296+0000","scrub_duration":0.003951225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"105'3","reported_seq":949,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778065+0000","last_change":"2024-09-10T19:12:24.460352+0000","last_active":"2024-09-10T19:12:54.778065+0000","last_peered":"2024-09-10T19:12:54.778065+0000","last_clean":"2024-09-10T19:12:54.778065+0000","last_became_active":"2024-09-10T18:49:22.042316+0000","last_became_peered":"2024-09-10T18:49:22.042316+0000","last_unstale":"2024-09-10T19:12:54.778065+0000","last_undegraded":"2024-09-10T19:12:54.778065+0000","last_fullsized":"2024-09-10T19:12:54.778065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-09-10T19:12:24.460189+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-09-10T18:52:15.552791+0000","last_clean_scrub_stamp":"2024-09-10T19:12:24.460189+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.529216+0000","scrub_duration":0.0047295499999999999,"objects_trimmed":0,"snaptrim_duration":0.0014539869999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":869,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777560+0000","last_change":"2024-09-10T19:12:39.101426+0000","last_active":"2024-09-10T19:12:54.777560+0000","last_peered":"2024-09-10T19:12:54.777560+0000","last_clean":"2024-09-10T19:12:54.777560+0000","last_became_active":"2024-09-10T19:09:17.611408+0000","last_became_peered":"2024-09-10T19:09:17.611408+0000","last_unstale":"2024-09-10T19:12:54.777560+0000","last_undegraded":"2024-09-10T19:12:54.777560+0000","last_fullsized":"2024-09-10T19:12:54.777560+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:39.101274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:11.815591+0000","last_clean_scrub_stamp":"2024-09-10T19:12:39.101274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.570649+0000","scrub_duration":0.0031667710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":855,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628941+0000","last_change":"2024-09-10T19:12:06.944923+0000","last_active":"2024-09-10T19:12:54.628941+0000","last_peered":"2024-09-10T19:12:54.628941+0000","last_clean":"2024-09-10T19:12:54.628941+0000","last_became_active":"2024-09-10T18:54:37.710229+0000","last_became_peered":"2024-09-10T18:54:37.710229+0000","last_unstale":"2024-09-10T19:12:54.628941+0000","last_undegraded":"2024-09-10T19:12:54.628941+0000","last_fullsized":"2024-09-10T19:12:54.628941+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.944861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.944861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.267746+0000","scrub_duration":0.0021456119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"93'81","reported_seq":531,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778794+0000","last_change":"2024-09-10T19:12:29.495085+0000","last_active":"2024-09-10T19:12:54.778794+0000","last_peered":"2024-09-10T19:12:54.778794+0000","last_clean":"2024-09-10T19:12:54.778794+0000","last_became_active":"2024-09-10T19:02:03.615380+0000","last_became_peered":"2024-09-10T19:02:03.615380+0000","last_unstale":"2024-09-10T19:12:54.778794+0000","last_undegraded":"2024-09-10T19:12:54.778794+0000","last_fullsized":"2024-09-10T19:12:54.778794+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:29.494918+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:12:29.494918+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.494918+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:56.808922+0000","scrub_duration":0.0035680629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"859'45","reported_seq":1009,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781707+0000","last_change":"2024-09-10T19:12:28.801393+0000","last_active":"2024-09-10T19:12:54.781707+0000","last_peered":"2024-09-10T19:12:54.781707+0000","last_clean":"2024-09-10T19:12:54.781707+0000","last_became_active":"2024-09-10T18:49:19.615090+0000","last_became_peered":"2024-09-10T18:49:19.615090+0000","last_unstale":"2024-09-10T19:12:54.781707+0000","last_undegraded":"2024-09-10T19:12:54.781707+0000","last_fullsized":"2024-09-10T19:12:54.781707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'45","last_scrub_stamp":"2024-09-10T19:12:28.801232+0000","last_deep_scrub":"859'45","last_deep_scrub_stamp":"2024-09-10T19:11:02.154077+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.801232+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.125987+0000","scrub_duration":0.0051599180000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":873,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778792+0000","last_change":"2024-09-10T19:12:16.005163+0000","last_active":"2024-09-10T19:12:54.778792+0000","last_peered":"2024-09-10T19:12:54.778792+0000","last_clean":"2024-09-10T19:12:54.778792+0000","last_became_active":"2024-09-10T19:09:17.630951+0000","last_became_peered":"2024-09-10T19:09:17.630951+0000","last_unstale":"2024-09-10T19:12:54.778792+0000","last_undegraded":"2024-09-10T19:12:54.778792+0000","last_fullsized":"2024-09-10T19:12:54.778792+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:16.005014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:16.005014+0000","last_clean_scrub_stamp":"2024-09-10T19:12:16.005014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:41.641050+0000","scrub_duration":0.0034657170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":852,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777952+0000","last_change":"2024-09-10T19:12:37.180001+0000","last_active":"2024-09-10T19:12:54.777952+0000","last_peered":"2024-09-10T19:12:54.777952+0000","last_clean":"2024-09-10T19:12:54.777952+0000","last_became_active":"2024-09-10T19:09:17.622457+0000","last_became_peered":"2024-09-10T19:09:17.622457+0000","last_unstale":"2024-09-10T19:12:54.777952+0000","last_undegraded":"2024-09-10T19:12:54.777952+0000","last_fullsized":"2024-09-10T19:12:54.777952+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:37.179890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:46.247300+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.179890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.583108+0000","scrub_duration":0.0032060080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":938,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630061+0000","last_change":"2024-09-10T19:12:47.202242+0000","last_active":"2024-09-10T19:12:54.630061+0000","last_peered":"2024-09-10T19:12:54.630061+0000","last_clean":"2024-09-10T19:12:54.630061+0000","last_became_active":"2024-09-10T18:49:24.047181+0000","last_became_peered":"2024-09-10T18:49:24.047181+0000","last_unstale":"2024-09-10T19:12:54.630061+0000","last_undegraded":"2024-09-10T19:12:54.630061+0000","last_fullsized":"2024-09-10T19:12:54.630061+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:47.202053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:51.813903+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.202053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.505049+0000","scrub_duration":0.0035756780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":894,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779843+0000","last_change":"2024-09-10T19:12:51.518758+0000","last_active":"2024-09-10T19:12:54.779843+0000","last_peered":"2024-09-10T19:12:54.779843+0000","last_clean":"2024-09-10T19:12:54.779843+0000","last_became_active":"2024-09-10T18:54:37.666724+0000","last_became_peered":"2024-09-10T18:54:37.666724+0000","last_unstale":"2024-09-10T19:12:54.779843+0000","last_undegraded":"2024-09-10T19:12:54.779843+0000","last_fullsized":"2024-09-10T19:12:54.779843+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.518604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.518604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.107929+0000","scrub_duration":0.0034235910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"93'81","reported_seq":453,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776215+0000","last_change":"2024-09-10T19:12:46.748804+0000","last_active":"2024-09-10T19:12:54.776215+0000","last_peered":"2024-09-10T19:12:54.776215+0000","last_clean":"2024-09-10T19:12:54.776215+0000","last_became_active":"2024-09-10T19:02:03.640262+0000","last_became_peered":"2024-09-10T19:02:03.640262+0000","last_unstale":"2024-09-10T19:12:54.776215+0000","last_undegraded":"2024-09-10T19:12:54.776215+0000","last_fullsized":"2024-09-10T19:12:54.776215+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:46.748636+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.748636+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.753366+0000","scrub_duration":0.0036174229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"859'39","reported_seq":866,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:10.333726+0000","last_change":"2024-09-10T19:13:10.333638+0000","last_active":"2024-09-10T19:13:10.333726+0000","last_peered":"2024-09-10T19:13:10.333726+0000","last_clean":"2024-09-10T19:13:10.333726+0000","last_became_active":"2024-09-10T19:10:47.693090+0000","last_became_peered":"2024-09-10T19:10:47.693090+0000","last_unstale":"2024-09-10T19:13:10.333726+0000","last_undegraded":"2024-09-10T19:13:10.333726+0000","last_fullsized":"2024-09-10T19:13:10.333726+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'39","last_scrub_stamp":"2024-09-10T19:13:10.333475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:10.333475+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.237967+0000","scrub_duration":0.0055781709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.779540+0000","last_change":"2024-09-10T19:13:13.779437+0000","last_active":"2024-09-10T19:13:13.779540+0000","last_peered":"2024-09-10T19:13:13.779540+0000","last_clean":"2024-09-10T19:13:13.779540+0000","last_became_active":"2024-09-10T19:09:17.624324+0000","last_became_peered":"2024-09-10T19:09:17.624324+0000","last_unstale":"2024-09-10T19:13:13.779540+0000","last_undegraded":"2024-09-10T19:13:13.779540+0000","last_fullsized":"2024-09-10T19:13:13.779540+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.779283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:13.779283+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.779283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.997020+0000","scrub_duration":0.0033720930000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":866,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:09.078299+0000","last_change":"2024-09-10T19:13:09.078185+0000","last_active":"2024-09-10T19:13:09.078299+0000","last_peered":"2024-09-10T19:13:09.078299+0000","last_clean":"2024-09-10T19:13:09.078299+0000","last_became_active":"2024-09-10T19:09:17.629427+0000","last_became_peered":"2024-09-10T19:09:17.629427+0000","last_unstale":"2024-09-10T19:13:09.078299+0000","last_undegraded":"2024-09-10T19:13:09.078299+0000","last_fullsized":"2024-09-10T19:13:09.078299+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.078046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.078046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.316720+0000","scrub_duration":0.0029229659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":927,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630257+0000","last_change":"2024-09-10T19:11:56.945174+0000","last_active":"2024-09-10T19:12:54.630257+0000","last_peered":"2024-09-10T19:12:54.630257+0000","last_clean":"2024-09-10T19:12:54.630257+0000","last_became_active":"2024-09-10T18:49:24.043444+0000","last_became_peered":"2024-09-10T18:49:24.043444+0000","last_unstale":"2024-09-10T19:12:54.630257+0000","last_undegraded":"2024-09-10T19:12:54.630257+0000","last_fullsized":"2024-09-10T19:12:54.630257+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.945087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.945087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028862369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.025803+0000","last_change":"2024-09-10T19:13:13.025692+0000","last_active":"2024-09-10T19:13:13.025803+0000","last_peered":"2024-09-10T19:13:13.025803+0000","last_clean":"2024-09-10T19:13:13.025803+0000","last_became_active":"2024-09-10T18:54:37.639056+0000","last_became_peered":"2024-09-10T18:54:37.639056+0000","last_unstale":"2024-09-10T19:13:13.025803+0000","last_undegraded":"2024-09-10T19:13:13.025803+0000","last_fullsized":"2024-09-10T19:13:13.025803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.025532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:13.025532+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.025532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.233120+0000","scrub_duration":0.003487637,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'81","reported_seq":1021,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.559137+0000","last_change":"2024-09-10T19:13:00.558993+0000","last_active":"2024-09-10T19:13:00.559137+0000","last_peered":"2024-09-10T19:13:00.559137+0000","last_clean":"2024-09-10T19:13:00.559137+0000","last_became_active":"2024-09-10T19:02:03.614074+0000","last_became_peered":"2024-09-10T19:02:03.614074+0000","last_unstale":"2024-09-10T19:13:00.559137+0000","last_undegraded":"2024-09-10T19:13:00.559137+0000","last_fullsized":"2024-09-10T19:13:00.559137+0000","mapping_epoch":510,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":511,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:00.558805+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.558805+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:05.158493+0000","scrub_duration":0.0033727290000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"859'50","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.090155+0000","last_change":"2024-09-10T19:13:06.089934+0000","last_active":"2024-09-10T19:13:06.090155+0000","last_peered":"2024-09-10T19:13:06.090155+0000","last_clean":"2024-09-10T19:13:06.090155+0000","last_became_active":"2024-09-10T19:09:17.619352+0000","last_became_peered":"2024-09-10T19:09:17.619352+0000","last_unstale":"2024-09-10T19:13:06.090155+0000","last_undegraded":"2024-09-10T19:13:06.090155+0000","last_fullsized":"2024-09-10T19:13:06.090155+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'50","last_scrub_stamp":"2024-09-10T19:13:06.089649+0000","last_deep_scrub":"859'50","last_deep_scrub_stamp":"2024-09-10T19:11:37.999246+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.089649+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.912537+0000","scrub_duration":0.0065939689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":988,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:12.487351+0000","last_change":"2024-09-10T19:13:12.487233+0000","last_active":"2024-09-10T19:13:12.487351+0000","last_peered":"2024-09-10T19:13:12.487351+0000","last_clean":"2024-09-10T19:13:12.487351+0000","last_became_active":"2024-09-10T18:49:17.603670+0000","last_became_peered":"2024-09-10T18:49:17.603670+0000","last_unstale":"2024-09-10T19:13:12.487351+0000","last_undegraded":"2024-09-10T19:13:12.487351+0000","last_fullsized":"2024-09-10T19:13:12.487351+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:12.487067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:02:47.645030+0000","last_clean_scrub_stamp":"2024-09-10T19:13:12.487067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.969526+0000","scrub_duration":0.0039078300000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":851,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.068942+0000","last_change":"2024-09-10T19:13:23.068902+0000","last_active":"2024-09-10T19:13:23.068942+0000","last_peered":"2024-09-10T19:13:23.068942+0000","last_clean":"2024-09-10T19:13:23.068942+0000","last_became_active":"2024-09-10T19:09:17.622531+0000","last_became_peered":"2024-09-10T19:09:17.622531+0000","last_unstale":"2024-09-10T19:13:23.068942+0000","last_undegraded":"2024-09-10T19:13:23.068942+0000","last_fullsized":"2024-09-10T19:13:23.068942+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.068822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.927927+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.068822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.020193+0000","scrub_duration":0.0027733609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.525310+0000","last_change":"2024-09-10T19:13:17.525229+0000","last_active":"2024-09-10T19:13:17.525310+0000","last_peered":"2024-09-10T19:13:17.525310+0000","last_clean":"2024-09-10T19:13:17.525310+0000","last_became_active":"2024-09-10T18:49:24.364995+0000","last_became_peered":"2024-09-10T18:49:24.364995+0000","last_unstale":"2024-09-10T19:13:17.525310+0000","last_undegraded":"2024-09-10T19:13:17.525310+0000","last_fullsized":"2024-09-10T19:13:17.525310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.525065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:01.874587+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.525065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:35.063352+0000","scrub_duration":0.0032966100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":865,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779249+0000","last_change":"2024-09-10T19:12:28.460824+0000","last_active":"2024-09-10T19:12:54.779249+0000","last_peered":"2024-09-10T19:12:54.779249+0000","last_clean":"2024-09-10T19:12:54.779249+0000","last_became_active":"2024-09-10T18:54:37.708976+0000","last_became_peered":"2024-09-10T18:54:37.708976+0000","last_unstale":"2024-09-10T19:12:54.779249+0000","last_undegraded":"2024-09-10T19:12:54.779249+0000","last_fullsized":"2024-09-10T19:12:54.779249+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:28.460668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:28.460668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:40.663215+0000","scrub_duration":0.003172182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"93'81","reported_seq":376,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.047088+0000","last_change":"2024-09-10T19:13:15.047037+0000","last_active":"2024-09-10T19:13:15.047088+0000","last_peered":"2024-09-10T19:13:15.047088+0000","last_clean":"2024-09-10T19:13:15.047088+0000","last_became_active":"2024-09-10T19:09:17.615000+0000","last_became_peered":"2024-09-10T19:09:17.615000+0000","last_unstale":"2024-09-10T19:13:15.047088+0000","last_undegraded":"2024-09-10T19:13:15.047088+0000","last_fullsized":"2024-09-10T19:13:15.047088+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":508,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:13:15.046929+0000","last_deep_scrub":"30'74","last_deep_scrub_stamp":"2024-09-10T18:49:07.003725+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.046929+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:28.227610+0000","scrub_duration":0.0027016319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"859'42","reported_seq":1005,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630903+0000","last_change":"2024-09-10T19:12:52.223833+0000","last_active":"2024-09-10T19:12:54.630903+0000","last_peered":"2024-09-10T19:12:54.630903+0000","last_clean":"2024-09-10T19:12:54.630903+0000","last_became_active":"2024-09-10T18:49:19.635802+0000","last_became_peered":"2024-09-10T18:49:19.635802+0000","last_unstale":"2024-09-10T19:12:54.630903+0000","last_undegraded":"2024-09-10T19:12:54.630903+0000","last_fullsized":"2024-09-10T19:12:54.630903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'42","last_scrub_stamp":"2024-09-10T19:12:52.223726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.223726+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.659214+0000","scrub_duration":0.0042711950000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":970,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:28.409723+0000","last_change":"2024-09-10T19:13:28.409645+0000","last_active":"2024-09-10T19:13:28.409723+0000","last_peered":"2024-09-10T19:13:28.409723+0000","last_clean":"2024-09-10T19:13:28.409723+0000","last_became_active":"2024-09-10T19:09:17.624516+0000","last_became_peered":"2024-09-10T19:09:17.624516+0000","last_unstale":"2024-09-10T19:13:28.409723+0000","last_undegraded":"2024-09-10T19:13:28.409723+0000","last_fullsized":"2024-09-10T19:13:28.409723+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:28.409488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:26.116042+0000","last_clean_scrub_stamp":"2024-09-10T19:13:28.409488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.946435+0000","scrub_duration":0.0037011169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":960,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:32.142641+0000","last_change":"2024-09-10T19:13:32.142551+0000","last_active":"2024-09-10T19:13:32.142641+0000","last_peered":"2024-09-10T19:13:32.142641+0000","last_clean":"2024-09-10T19:13:32.142641+0000","last_became_active":"2024-09-10T19:09:17.624029+0000","last_became_peered":"2024-09-10T19:09:17.624029+0000","last_unstale":"2024-09-10T19:13:32.142641+0000","last_undegraded":"2024-09-10T19:13:32.142641+0000","last_fullsized":"2024-09-10T19:13:32.142641+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:32.142409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:32.142409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:59.684136+0000","scrub_duration":0.0029776659999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":862,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:32.103540+0000","last_change":"2024-09-10T19:13:32.103466+0000","last_active":"2024-09-10T19:13:32.103540+0000","last_peered":"2024-09-10T19:13:32.103540+0000","last_clean":"2024-09-10T19:13:32.103540+0000","last_became_active":"2024-09-10T19:09:17.630321+0000","last_became_peered":"2024-09-10T19:09:17.630321+0000","last_unstale":"2024-09-10T19:13:32.103540+0000","last_undegraded":"2024-09-10T19:13:32.103540+0000","last_fullsized":"2024-09-10T19:13:32.103540+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:32.103309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:09.978565+0000","last_clean_scrub_stamp":"2024-09-10T19:13:32.103309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:01.005342+0000","scrub_duration":0.0035133500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:07.108379+0000","last_change":"2024-09-10T19:13:07.108208+0000","last_active":"2024-09-10T19:13:07.108379+0000","last_peered":"2024-09-10T19:13:07.108379+0000","last_clean":"2024-09-10T19:13:07.108379+0000","last_became_active":"2024-09-10T18:54:37.579356+0000","last_became_peered":"2024-09-10T18:54:37.579356+0000","last_unstale":"2024-09-10T19:13:07.108379+0000","last_undegraded":"2024-09-10T19:13:07.108379+0000","last_fullsized":"2024-09-10T19:13:07.108379+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:07.108000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:13:07.108000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.241782+0000","scrub_duration":0.0039466789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"859'35","reported_seq":988,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:32.859677+0000","last_change":"2024-09-10T19:13:32.859598+0000","last_active":"2024-09-10T19:13:32.859677+0000","last_peered":"2024-09-10T19:13:32.859677+0000","last_clean":"2024-09-10T19:13:32.859677+0000","last_became_active":"2024-09-10T18:49:19.621597+0000","last_became_peered":"2024-09-10T18:49:19.621597+0000","last_unstale":"2024-09-10T19:13:32.859677+0000","last_undegraded":"2024-09-10T19:13:32.859677+0000","last_fullsized":"2024-09-10T19:13:32.859677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:13:32.859466+0000","last_deep_scrub":"109'15","last_deep_scrub_stamp":"2024-09-10T18:52:54.743264+0000","last_clean_scrub_stamp":"2024-09-10T19:13:32.859466+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.222566+0000","scrub_duration":0.004120355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781104+0000","last_change":"2024-09-10T19:12:08.806518+0000","last_active":"2024-09-10T19:12:54.781104+0000","last_peered":"2024-09-10T19:12:54.781104+0000","last_clean":"2024-09-10T19:12:54.781104+0000","last_became_active":"2024-09-10T18:49:17.612439+0000","last_became_peered":"2024-09-10T18:49:17.612439+0000","last_unstale":"2024-09-10T19:12:54.781104+0000","last_undegraded":"2024-09-10T19:12:54.781104+0000","last_fullsized":"2024-09-10T19:12:54.781104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:08.806364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:00:15.230765+0000","last_clean_scrub_stamp":"2024-09-10T19:12:08.806364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.487417+0000","scrub_duration":0.0032522279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'81","reported_seq":1012,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780443+0000","last_change":"2024-09-10T19:12:43.759930+0000","last_active":"2024-09-10T19:12:54.780443+0000","last_peered":"2024-09-10T19:12:54.780443+0000","last_clean":"2024-09-10T19:12:54.780443+0000","last_became_active":"2024-09-10T19:02:01.597407+0000","last_became_peered":"2024-09-10T19:02:01.597407+0000","last_unstale":"2024-09-10T19:12:54.780443+0000","last_undegraded":"2024-09-10T19:12:54.780443+0000","last_fullsized":"2024-09-10T19:12:54.780443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":509,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'81","last_scrub_stamp":"2024-09-10T19:12:43.759757+0000","last_deep_scrub":"93'81","last_deep_scrub_stamp":"2024-09-10T19:09:58.722119+0000","last_clean_scrub_stamp":"2024-09-10T19:12:43.759757+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:59.816579+0000","scrub_duration":0.0038125199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":941,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778523+0000","last_change":"2024-09-10T19:12:43.440980+0000","last_active":"2024-09-10T19:12:54.778523+0000","last_peered":"2024-09-10T19:12:54.778523+0000","last_clean":"2024-09-10T19:12:54.778523+0000","last_became_active":"2024-09-10T18:49:22.058613+0000","last_became_peered":"2024-09-10T18:49:22.058613+0000","last_unstale":"2024-09-10T19:12:54.778523+0000","last_undegraded":"2024-09-10T19:12:54.778523+0000","last_fullsized":"2024-09-10T19:12:54.778523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:43.440817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:03.329632+0000","last_clean_scrub_stamp":"2024-09-10T19:12:43.440817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:09.757706+0000","scrub_duration":0.003326421,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.782124+0000","last_change":"2024-09-10T19:12:38.687371+0000","last_active":"2024-09-10T19:12:54.782124+0000","last_peered":"2024-09-10T19:12:54.782124+0000","last_clean":"2024-09-10T19:12:54.782124+0000","last_became_active":"2024-09-10T18:49:24.048939+0000","last_became_peered":"2024-09-10T18:49:24.048939+0000","last_unstale":"2024-09-10T19:12:54.782124+0000","last_undegraded":"2024-09-10T19:12:54.782124+0000","last_fullsized":"2024-09-10T19:12:54.782124+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:38.687215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:55.021218+0000","last_clean_scrub_stamp":"2024-09-10T19:12:38.687215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:07.271979+0000","scrub_duration":0.0033182039999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":888,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.489558+0000","last_change":"2024-09-10T19:13:15.489477+0000","last_active":"2024-09-10T19:13:15.489558+0000","last_peered":"2024-09-10T19:13:15.489558+0000","last_clean":"2024-09-10T19:13:15.489558+0000","last_became_active":"2024-09-10T19:09:17.624085+0000","last_became_peered":"2024-09-10T19:09:17.624085+0000","last_unstale":"2024-09-10T19:13:15.489558+0000","last_undegraded":"2024-09-10T19:13:15.489558+0000","last_fullsized":"2024-09-10T19:13:15.489558+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.489308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:02.895391+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.489308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.774790+0000","scrub_duration":0.0038349460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"859'53","reported_seq":1029,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:14.472225+0000","last_change":"2024-09-10T19:13:14.472143+0000","last_active":"2024-09-10T19:13:14.472225+0000","last_peered":"2024-09-10T19:13:14.472225+0000","last_clean":"2024-09-10T19:13:14.472225+0000","last_became_active":"2024-09-10T18:49:19.631584+0000","last_became_peered":"2024-09-10T18:49:19.631584+0000","last_unstale":"2024-09-10T19:13:14.472225+0000","last_undegraded":"2024-09-10T19:13:14.472225+0000","last_fullsized":"2024-09-10T19:13:14.472225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'53","last_scrub_stamp":"2024-09-10T19:13:14.471973+0000","last_deep_scrub":"859'53","last_deep_scrub_stamp":"2024-09-10T19:13:14.471973+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.471973+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.038006+0000","scrub_duration":0.0061128839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":988,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:19.459010+0000","last_change":"2024-09-10T19:13:19.458902+0000","last_active":"2024-09-10T19:13:19.459010+0000","last_peered":"2024-09-10T19:13:19.459010+0000","last_clean":"2024-09-10T19:13:19.459010+0000","last_became_active":"2024-09-10T18:49:17.613208+0000","last_became_peered":"2024-09-10T18:49:17.613208+0000","last_unstale":"2024-09-10T19:13:19.459010+0000","last_undegraded":"2024-09-10T19:13:19.459010+0000","last_fullsized":"2024-09-10T19:13:19.459010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:19.458638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:19.458638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.204429+0000","scrub_duration":0.0035443229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":958,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778689+0000","last_change":"2024-09-10T19:12:22.435931+0000","last_active":"2024-09-10T19:12:54.778689+0000","last_peered":"2024-09-10T19:12:54.778689+0000","last_clean":"2024-09-10T19:12:54.778689+0000","last_became_active":"2024-09-10T19:09:17.613101+0000","last_became_peered":"2024-09-10T19:09:17.613101+0000","last_unstale":"2024-09-10T19:12:54.778689+0000","last_undegraded":"2024-09-10T19:12:54.778689+0000","last_fullsized":"2024-09-10T19:12:54.778689+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:22.435765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:22.435765+0000","last_clean_scrub_stamp":"2024-09-10T19:12:22.435765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:31.022433+0000","scrub_duration":0.0037076259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:03.524244+0000","last_change":"2024-09-10T19:13:03.524074+0000","last_active":"2024-09-10T19:13:03.524244+0000","last_peered":"2024-09-10T19:13:03.524244+0000","last_clean":"2024-09-10T19:13:03.524244+0000","last_became_active":"2024-09-10T18:49:24.054983+0000","last_became_peered":"2024-09-10T18:49:24.054983+0000","last_unstale":"2024-09-10T19:13:03.524244+0000","last_undegraded":"2024-09-10T19:13:03.524244+0000","last_fullsized":"2024-09-10T19:13:03.524244+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:03.523914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:03.523914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.969934+0000","scrub_duration":0.0043000859999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":889,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:18.290450+0000","last_change":"2024-09-10T19:13:18.290372+0000","last_active":"2024-09-10T19:13:18.290450+0000","last_peered":"2024-09-10T19:13:18.290450+0000","last_clean":"2024-09-10T19:13:18.290450+0000","last_became_active":"2024-09-10T18:54:37.534551+0000","last_became_peered":"2024-09-10T18:54:37.534551+0000","last_unstale":"2024-09-10T19:13:18.290450+0000","last_undegraded":"2024-09-10T19:13:18.290450+0000","last_fullsized":"2024-09-10T19:13:18.290450+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:18.290186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:18.290186+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.290186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:44.810909+0000","scrub_duration":0.0031641389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"859'35","reported_seq":994,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781853+0000","last_change":"2024-09-10T19:12:40.695724+0000","last_active":"2024-09-10T19:12:54.781853+0000","last_peered":"2024-09-10T19:12:54.781853+0000","last_clean":"2024-09-10T19:12:54.781853+0000","last_became_active":"2024-09-10T18:49:19.618534+0000","last_became_peered":"2024-09-10T18:49:19.618534+0000","last_unstale":"2024-09-10T19:12:54.781853+0000","last_undegraded":"2024-09-10T19:12:54.781853+0000","last_fullsized":"2024-09-10T19:12:54.781853+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:12:40.695561+0000","last_deep_scrub":"859'35","last_deep_scrub_stamp":"2024-09-10T19:11:08.057014+0000","last_clean_scrub_stamp":"2024-09-10T19:12:40.695561+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:51.662645+0000","scrub_duration":0.0045341169999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":890,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:21.090862+0000","last_change":"2024-09-10T19:13:21.090793+0000","last_active":"2024-09-10T19:13:21.090862+0000","last_peered":"2024-09-10T19:13:21.090862+0000","last_clean":"2024-09-10T19:13:21.090862+0000","last_became_active":"2024-09-10T19:09:17.627724+0000","last_became_peered":"2024-09-10T19:09:17.627724+0000","last_unstale":"2024-09-10T19:13:21.090862+0000","last_undegraded":"2024-09-10T19:13:21.090862+0000","last_fullsized":"2024-09-10T19:13:21.090862+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:21.090668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:21.090668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:38.110398+0000","scrub_duration":0.0033861249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:35.113599+0000","last_change":"2024-09-10T19:13:35.113472+0000","last_active":"2024-09-10T19:13:35.113599+0000","last_peered":"2024-09-10T19:13:35.113599+0000","last_clean":"2024-09-10T19:13:35.113599+0000","last_became_active":"2024-09-10T19:09:17.623063+0000","last_became_peered":"2024-09-10T19:09:17.623063+0000","last_unstale":"2024-09-10T19:13:35.113599+0000","last_undegraded":"2024-09-10T19:13:35.113599+0000","last_fullsized":"2024-09-10T19:13:35.113599+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:35.113285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:35.113285+0000","last_clean_scrub_stamp":"2024-09-10T19:13:35.113285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:42.112271+0000","scrub_duration":0.0040027550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":928,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630831+0000","last_change":"2024-09-10T19:12:51.206710+0000","last_active":"2024-09-10T19:12:54.630831+0000","last_peered":"2024-09-10T19:12:54.630831+0000","last_clean":"2024-09-10T19:12:54.630831+0000","last_became_active":"2024-09-10T18:49:24.047408+0000","last_became_peered":"2024-09-10T18:49:24.047408+0000","last_unstale":"2024-09-10T19:12:54.630831+0000","last_undegraded":"2024-09-10T19:12:54.630831+0000","last_fullsized":"2024-09-10T19:12:54.630831+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.206565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:04:07.952993+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.206565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.323393+0000","scrub_duration":0.0033822589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":859,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:01.166984+0000","last_change":"2024-09-10T19:13:01.166809+0000","last_active":"2024-09-10T19:13:01.166984+0000","last_peered":"2024-09-10T19:13:01.166984+0000","last_clean":"2024-09-10T19:13:01.166984+0000","last_became_active":"2024-09-10T18:54:37.670253+0000","last_became_peered":"2024-09-10T18:54:37.670253+0000","last_unstale":"2024-09-10T19:13:01.166984+0000","last_undegraded":"2024-09-10T19:13:01.166984+0000","last_fullsized":"2024-09-10T19:13:01.166984+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.166559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:01.166559+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.166559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.234005+0000","scrub_duration":0.0035870870000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"859'46","reported_seq":1019,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.783310+0000","last_change":"2024-09-10T19:12:47.831330+0000","last_active":"2024-09-10T19:12:54.783310+0000","last_peered":"2024-09-10T19:12:54.783310+0000","last_clean":"2024-09-10T19:12:54.783310+0000","last_became_active":"2024-09-10T18:49:19.633504+0000","last_became_peered":"2024-09-10T18:49:19.633504+0000","last_unstale":"2024-09-10T19:12:54.783310+0000","last_undegraded":"2024-09-10T19:12:54.783310+0000","last_fullsized":"2024-09-10T19:12:54.783310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:12:47.831200+0000","last_deep_scrub":"104'22","last_deep_scrub_stamp":"2024-09-10T18:51:47.304153+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.831200+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.699467+0000","scrub_duration":0.0089948319999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"70'1","reported_seq":1000,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:30.650296+0000","last_change":"2024-09-10T19:13:30.650158+0000","last_active":"2024-09-10T19:13:30.650296+0000","last_peered":"2024-09-10T19:13:30.650296+0000","last_clean":"2024-09-10T19:13:30.650296+0000","last_became_active":"2024-09-10T18:49:17.616032+0000","last_became_peered":"2024-09-10T18:49:17.616032+0000","last_unstale":"2024-09-10T19:13:30.650296+0000","last_undegraded":"2024-09-10T19:13:30.650296+0000","last_fullsized":"2024-09-10T19:13:30.650296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:13:30.649988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:30.649988+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:31.060014+0000","scrub_duration":0.0039130170000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":952,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:27.674449+0000","last_change":"2024-09-10T19:13:27.674373+0000","last_active":"2024-09-10T19:13:27.674449+0000","last_peered":"2024-09-10T19:13:27.674449+0000","last_clean":"2024-09-10T19:13:27.674449+0000","last_became_active":"2024-09-10T18:49:22.060313+0000","last_became_peered":"2024-09-10T18:49:22.060313+0000","last_unstale":"2024-09-10T19:13:27.674449+0000","last_undegraded":"2024-09-10T19:13:27.674449+0000","last_fullsized":"2024-09-10T19:13:27.674449+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:27.674190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:46.633831+0000","last_clean_scrub_stamp":"2024-09-10T19:13:27.674190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:46.760502+0000","scrub_duration":0.003182921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780469+0000","last_change":"2024-09-10T19:12:31.732013+0000","last_active":"2024-09-10T19:12:54.780469+0000","last_peered":"2024-09-10T19:12:54.780469+0000","last_clean":"2024-09-10T19:12:54.780469+0000","last_became_active":"2024-09-10T18:49:24.368502+0000","last_became_peered":"2024-09-10T18:49:24.368502+0000","last_unstale":"2024-09-10T19:12:54.780469+0000","last_undegraded":"2024-09-10T19:12:54.780469+0000","last_fullsized":"2024-09-10T19:12:54.780469+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:31.731888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:12:31.731888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:39.391093+0000","scrub_duration":0.0027531270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":513,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.626750+0000","last_change":"2024-09-10T19:12:31.534516+0000","last_active":"2024-09-10T19:12:54.626750+0000","last_peered":"2024-09-10T19:12:54.626750+0000","last_clean":"2024-09-10T19:12:54.626750+0000","last_became_active":"2024-09-10T19:10:47.689638+0000","last_became_peered":"2024-09-10T19:10:47.689638+0000","last_unstale":"2024-09-10T19:12:54.626750+0000","last_undegraded":"2024-09-10T19:12:54.626750+0000","last_fullsized":"2024-09-10T19:12:54.626750+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:31.534346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:12.218396+0000","last_clean_scrub_stamp":"2024-09-10T19:12:31.534346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.026784+0000","scrub_duration":0.003248552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"901'32","reported_seq":1009,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.425730+0000","last_change":"2024-09-10T19:13:23.425654+0000","last_active":"2024-09-10T19:13:23.425730+0000","last_peered":"2024-09-10T19:13:23.425730+0000","last_clean":"2024-09-10T19:13:23.425730+0000","last_became_active":"2024-09-10T19:09:17.633035+0000","last_became_peered":"2024-09-10T19:09:17.633035+0000","last_unstale":"2024-09-10T19:13:23.425730+0000","last_undegraded":"2024-09-10T19:13:23.425730+0000","last_fullsized":"2024-09-10T19:13:23.425730+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'32","last_scrub_stamp":"2024-09-10T19:13:23.425467+0000","last_deep_scrub":"901'32","last_deep_scrub_stamp":"2024-09-10T19:13:23.425467+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.425467+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.478631+0000","scrub_duration":0.011078439000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"601'4","reported_seq":1015,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629697+0000","last_change":"2024-09-10T19:12:36.417556+0000","last_active":"2024-09-10T19:12:54.629697+0000","last_peered":"2024-09-10T19:12:54.629697+0000","last_clean":"2024-09-10T19:12:54.629697+0000","last_became_active":"2024-09-10T18:49:41.640975+0000","last_became_peered":"2024-09-10T18:49:41.640975+0000","last_unstale":"2024-09-10T19:12:54.629697+0000","last_undegraded":"2024-09-10T19:12:54.629697+0000","last_fullsized":"2024-09-10T19:12:54.629697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'4","last_scrub_stamp":"2024-09-10T19:12:36.417396+0000","last_deep_scrub":"601'4","last_deep_scrub_stamp":"2024-09-10T19:04:04.923921+0000","last_clean_scrub_stamp":"2024-09-10T19:12:36.417396+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.972382+0000","scrub_duration":0.004415385,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"879'15","reported_seq":801,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.922084+0000","last_change":"2024-09-10T19:12:37.463471+0000","last_active":"2024-09-10T19:13:26.922084+0000","last_peered":"2024-09-10T19:13:26.922084+0000","last_clean":"2024-09-10T19:13:26.922084+0000","last_became_active":"2024-09-10T19:10:47.694903+0000","last_became_peered":"2024-09-10T19:10:47.694903+0000","last_unstale":"2024-09-10T19:13:26.922084+0000","last_undegraded":"2024-09-10T19:13:26.922084+0000","last_fullsized":"2024-09-10T19:13:26.922084+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'15","last_scrub_stamp":"2024-09-10T19:12:37.463308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.463308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:41.922827+0000","scrub_duration":0.0038917520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":951,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:56.960638+0000","last_change":"2024-09-10T19:12:56.960511+0000","last_active":"2024-09-10T19:12:56.960638+0000","last_peered":"2024-09-10T19:12:56.960638+0000","last_clean":"2024-09-10T19:12:56.960638+0000","last_became_active":"2024-09-10T19:09:17.620179+0000","last_became_peered":"2024-09-10T19:09:17.620179+0000","last_unstale":"2024-09-10T19:12:56.960638+0000","last_undegraded":"2024-09-10T19:12:56.960638+0000","last_fullsized":"2024-09-10T19:12:56.960638+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.960359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:40.755118+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.960359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.915896+0000","scrub_duration":0.0036065630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1","version":"126'7","reported_seq":875,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778854+0000","last_change":"2024-09-10T19:12:11.533250+0000","last_active":"2024-09-10T19:12:54.778854+0000","last_peered":"2024-09-10T19:12:54.778854+0000","last_clean":"2024-09-10T19:12:54.778854+0000","last_became_active":"2024-09-10T18:54:37.601651+0000","last_became_peered":"2024-09-10T18:54:37.601651+0000","last_unstale":"2024-09-10T19:12:54.778854+0000","last_undegraded":"2024-09-10T19:12:54.778854+0000","last_fullsized":"2024-09-10T19:12:54.778854+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'7","last_scrub_stamp":"2024-09-10T19:12:11.533087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.533087+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.792717+0000","scrub_duration":0.0039834850000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"859'27","reported_seq":994,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778190+0000","last_change":"2024-09-10T19:12:11.900879+0000","last_active":"2024-09-10T19:12:54.778190+0000","last_peered":"2024-09-10T19:12:54.778190+0000","last_clean":"2024-09-10T19:12:54.778190+0000","last_became_active":"2024-09-10T18:49:19.632593+0000","last_became_peered":"2024-09-10T18:49:19.632593+0000","last_unstale":"2024-09-10T19:12:54.778190+0000","last_undegraded":"2024-09-10T19:12:54.778190+0000","last_fullsized":"2024-09-10T19:12:54.778190+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'27","last_scrub_stamp":"2024-09-10T19:12:11.900712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.900712+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:31.032241+0000","scrub_duration":0.0050889890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":995,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:23.527959+0000","last_change":"2024-09-10T19:13:23.527868+0000","last_active":"2024-09-10T19:13:23.527959+0000","last_peered":"2024-09-10T19:13:23.527959+0000","last_clean":"2024-09-10T19:13:23.527959+0000","last_became_active":"2024-09-10T19:09:17.615024+0000","last_became_peered":"2024-09-10T19:09:17.615024+0000","last_unstale":"2024-09-10T19:13:23.527959+0000","last_undegraded":"2024-09-10T19:13:23.527959+0000","last_fullsized":"2024-09-10T19:13:23.527959+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:23.527711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:53:13.292663+0000","last_clean_scrub_stamp":"2024-09-10T19:13:23.527711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.875509+0000","scrub_duration":0.0032316559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.165992+0000","last_change":"2024-09-10T19:12:55.165889+0000","last_active":"2024-09-10T19:12:55.165992+0000","last_peered":"2024-09-10T19:12:55.165992+0000","last_clean":"2024-09-10T19:12:55.165992+0000","last_became_active":"2024-09-10T18:49:22.063917+0000","last_became_peered":"2024-09-10T18:49:22.063917+0000","last_unstale":"2024-09-10T19:12:55.165992+0000","last_undegraded":"2024-09-10T19:12:55.165992+0000","last_fullsized":"2024-09-10T19:12:55.165992+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.165750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.165750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.773616+0000","scrub_duration":0.0029434719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":972,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:22.549558+0000","last_change":"2024-09-10T19:13:22.549477+0000","last_active":"2024-09-10T19:13:22.549558+0000","last_peered":"2024-09-10T19:13:22.549558+0000","last_clean":"2024-09-10T19:13:22.549558+0000","last_became_active":"2024-09-10T18:49:24.048871+0000","last_became_peered":"2024-09-10T18:49:24.048871+0000","last_unstale":"2024-09-10T19:13:22.549558+0000","last_undegraded":"2024-09-10T19:13:22.549558+0000","last_fullsized":"2024-09-10T19:13:22.549558+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:22.549324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:19.223584+0000","last_clean_scrub_stamp":"2024-09-10T19:13:22.549324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.638964+0000","scrub_duration":0.0038758019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":889,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:31.279954+0000","last_change":"2024-09-10T19:13:31.279868+0000","last_active":"2024-09-10T19:13:31.279954+0000","last_peered":"2024-09-10T19:13:31.279954+0000","last_clean":"2024-09-10T19:13:31.279954+0000","last_became_active":"2024-09-10T18:54:37.613912+0000","last_became_peered":"2024-09-10T18:54:37.613912+0000","last_unstale":"2024-09-10T19:13:31.279954+0000","last_undegraded":"2024-09-10T19:13:31.279954+0000","last_fullsized":"2024-09-10T19:13:31.279954+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:31.279707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:04.462331+0000","last_clean_scrub_stamp":"2024-09-10T19:13:31.279707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.159860+0000","scrub_duration":0.0033326860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"859'14","reported_seq":963,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777271+0000","last_change":"2024-09-10T19:11:57.631685+0000","last_active":"2024-09-10T19:12:54.777271+0000","last_peered":"2024-09-10T19:12:54.777271+0000","last_clean":"2024-09-10T19:12:54.777271+0000","last_became_active":"2024-09-10T18:49:19.622846+0000","last_became_peered":"2024-09-10T18:49:19.622846+0000","last_unstale":"2024-09-10T19:12:54.777271+0000","last_undegraded":"2024-09-10T19:12:54.777271+0000","last_fullsized":"2024-09-10T19:12:54.777271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'14","last_scrub_stamp":"2024-09-10T19:11:57.631561+0000","last_deep_scrub":"859'14","last_deep_scrub_stamp":"2024-09-10T19:10:25.016589+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.631561+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:26.509894+0000","scrub_duration":0.0040774849999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":962,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779937+0000","last_change":"2024-09-10T19:12:20.436572+0000","last_active":"2024-09-10T19:12:54.779937+0000","last_peered":"2024-09-10T19:12:54.779937+0000","last_clean":"2024-09-10T19:12:54.779937+0000","last_became_active":"2024-09-10T19:09:17.614809+0000","last_became_peered":"2024-09-10T19:09:17.614809+0000","last_unstale":"2024-09-10T19:12:54.779937+0000","last_undegraded":"2024-09-10T19:12:54.779937+0000","last_fullsized":"2024-09-10T19:12:54.779937+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:20.436322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:20.436322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:47.028739+0000","scrub_duration":0.0038737590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.947899+0000","last_change":"2024-09-10T19:13:05.947765+0000","last_active":"2024-09-10T19:13:05.947899+0000","last_peered":"2024-09-10T19:13:05.947899+0000","last_clean":"2024-09-10T19:13:05.947899+0000","last_became_active":"2024-09-10T19:09:17.622794+0000","last_became_peered":"2024-09-10T19:09:17.622794+0000","last_unstale":"2024-09-10T19:13:05.947899+0000","last_undegraded":"2024-09-10T19:13:05.947899+0000","last_fullsized":"2024-09-10T19:13:05.947899+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.947610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:52:46.829507+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.947610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.828022+0000","scrub_duration":0.0033327069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":957,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:27.454547+0000","last_change":"2024-09-10T19:13:27.454469+0000","last_active":"2024-09-10T19:13:27.454547+0000","last_peered":"2024-09-10T19:13:27.454547+0000","last_clean":"2024-09-10T19:13:27.454547+0000","last_became_active":"2024-09-10T18:49:24.366119+0000","last_became_peered":"2024-09-10T18:49:24.366119+0000","last_unstale":"2024-09-10T19:13:27.454547+0000","last_undegraded":"2024-09-10T19:13:27.454547+0000","last_fullsized":"2024-09-10T19:13:27.454547+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:27.454309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:23.022168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:27.454309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.760259+0000","scrub_duration":0.0032222119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779023+0000","last_change":"2024-09-10T19:12:18.425215+0000","last_active":"2024-09-10T19:12:54.779023+0000","last_peered":"2024-09-10T19:12:54.779023+0000","last_clean":"2024-09-10T19:12:54.779023+0000","last_became_active":"2024-09-10T18:54:37.709707+0000","last_became_peered":"2024-09-10T18:54:37.709707+0000","last_unstale":"2024-09-10T19:12:54.779023+0000","last_undegraded":"2024-09-10T19:12:54.779023+0000","last_fullsized":"2024-09-10T19:12:54.779023+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.425043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.425043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:44.366833+0000","scrub_duration":0.0033576000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"859'32","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780079+0000","last_change":"2024-09-10T19:12:08.831808+0000","last_active":"2024-09-10T19:12:54.780079+0000","last_peered":"2024-09-10T19:12:54.780079+0000","last_clean":"2024-09-10T19:12:54.780079+0000","last_became_active":"2024-09-10T18:49:19.632374+0000","last_became_peered":"2024-09-10T18:49:19.632374+0000","last_unstale":"2024-09-10T19:12:54.780079+0000","last_undegraded":"2024-09-10T19:12:54.780079+0000","last_fullsized":"2024-09-10T19:12:54.780079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'32","last_scrub_stamp":"2024-09-10T19:12:08.831642+0000","last_deep_scrub":"859'32","last_deep_scrub_stamp":"2024-09-10T19:12:08.831642+0000","last_clean_scrub_stamp":"2024-09-10T19:12:08.831642+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.338553+0000","scrub_duration":0.0052310309999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":939,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781058+0000","last_change":"2024-09-10T19:11:57.860845+0000","last_active":"2024-09-10T19:12:54.781058+0000","last_peered":"2024-09-10T19:12:54.781058+0000","last_clean":"2024-09-10T19:12:54.781058+0000","last_became_active":"2024-09-10T18:49:17.613388+0000","last_became_peered":"2024-09-10T18:49:17.613388+0000","last_unstale":"2024-09-10T19:12:54.781058+0000","last_undegraded":"2024-09-10T19:12:54.781058+0000","last_fullsized":"2024-09-10T19:12:54.781058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:57.860693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.860693+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.860693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:23.608479+0000","scrub_duration":0.0033432570000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"856'14","reported_seq":1260,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.924058+0000","last_change":"2024-09-10T19:13:18.839483+0000","last_active":"2024-09-10T19:13:26.924058+0000","last_peered":"2024-09-10T19:13:26.924058+0000","last_clean":"2024-09-10T19:13:26.924058+0000","last_became_active":"2024-09-10T19:09:17.630818+0000","last_became_peered":"2024-09-10T19:09:17.630818+0000","last_unstale":"2024-09-10T19:13:26.924058+0000","last_undegraded":"2024-09-10T19:13:26.924058+0000","last_fullsized":"2024-09-10T19:13:26.924058+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'14","last_scrub_stamp":"2024-09-10T19:13:18.839267+0000","last_deep_scrub":"856'14","last_deep_scrub_stamp":"2024-09-10T19:13:18.839267+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.839267+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.300397+0000","scrub_duration":0.0037778780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":938,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.164056+0000","last_change":"2024-09-10T19:13:00.163996+0000","last_active":"2024-09-10T19:13:00.164056+0000","last_peered":"2024-09-10T19:13:00.164056+0000","last_clean":"2024-09-10T19:13:00.164056+0000","last_became_active":"2024-09-10T18:49:24.043593+0000","last_became_peered":"2024-09-10T18:49:24.043593+0000","last_unstale":"2024-09-10T19:13:00.164056+0000","last_undegraded":"2024-09-10T19:13:00.164056+0000","last_fullsized":"2024-09-10T19:13:00.164056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.163904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:00.163904+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.163904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.326070+0000","scrub_duration":0.003138348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:09.048361+0000","last_change":"2024-09-10T19:13:09.048324+0000","last_active":"2024-09-10T19:13:09.048361+0000","last_peered":"2024-09-10T19:13:09.048361+0000","last_clean":"2024-09-10T19:13:09.048361+0000","last_became_active":"2024-09-10T18:54:37.638708+0000","last_became_peered":"2024-09-10T18:54:37.638708+0000","last_unstale":"2024-09-10T19:13:09.048361+0000","last_undegraded":"2024-09-10T19:13:09.048361+0000","last_fullsized":"2024-09-10T19:13:09.048361+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.048254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:31.597446+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.048254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.209263+0000","scrub_duration":0.0021055919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"859'21","reported_seq":976,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631249+0000","last_change":"2024-09-10T19:12:54.186725+0000","last_active":"2024-09-10T19:12:54.631249+0000","last_peered":"2024-09-10T19:12:54.631249+0000","last_clean":"2024-09-10T19:12:54.631249+0000","last_became_active":"2024-09-10T18:49:19.630488+0000","last_became_peered":"2024-09-10T18:49:19.630488+0000","last_unstale":"2024-09-10T19:12:54.631249+0000","last_undegraded":"2024-09-10T19:12:54.631249+0000","last_fullsized":"2024-09-10T19:12:54.631249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:12:54.186556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.186556+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.956911+0000","scrub_duration":0.0047097140000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631603+0000","last_change":"2024-09-10T19:12:42.094004+0000","last_active":"2024-09-10T19:12:54.631603+0000","last_peered":"2024-09-10T19:12:54.631603+0000","last_clean":"2024-09-10T19:12:54.631603+0000","last_became_active":"2024-09-10T18:49:17.607069+0000","last_became_peered":"2024-09-10T18:49:17.607069+0000","last_unstale":"2024-09-10T19:12:54.631603+0000","last_undegraded":"2024-09-10T19:12:54.631603+0000","last_fullsized":"2024-09-10T19:12:54.631603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:42.093876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:46.308759+0000","last_clean_scrub_stamp":"2024-09-10T19:12:42.093876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.092846+0000","scrub_duration":0.002962592,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"77'11","reported_seq":1233,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.921765+0000","last_change":"2024-09-10T19:13:19.697733+0000","last_active":"2024-09-10T19:13:26.921765+0000","last_peered":"2024-09-10T19:13:26.921765+0000","last_clean":"2024-09-10T19:13:26.921765+0000","last_became_active":"2024-09-10T18:49:22.059932+0000","last_became_peered":"2024-09-10T18:49:22.059932+0000","last_unstale":"2024-09-10T19:13:26.921765+0000","last_undegraded":"2024-09-10T19:13:26.921765+0000","last_fullsized":"2024-09-10T19:13:26.921765+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'11","last_scrub_stamp":"2024-09-10T19:13:19.697499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:13:19.697499+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:33.844040+0000","scrub_duration":0.0051654370000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":592,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.311660+0000","last_change":"2024-09-10T19:13:17.311578+0000","last_active":"2024-09-10T19:13:17.311660+0000","last_peered":"2024-09-10T19:13:17.311660+0000","last_clean":"2024-09-10T19:13:17.311660+0000","last_became_active":"2024-09-10T19:10:47.684527+0000","last_became_peered":"2024-09-10T19:10:47.684527+0000","last_unstale":"2024-09-10T19:13:17.311660+0000","last_undegraded":"2024-09-10T19:13:17.311660+0000","last_fullsized":"2024-09-10T19:13:17.311660+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:17.311423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:36.878367+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.311423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:39.495046+0000","scrub_duration":0.0032662540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631818+0000","last_change":"2024-09-10T19:12:33.108017+0000","last_active":"2024-09-10T19:12:54.631818+0000","last_peered":"2024-09-10T19:12:54.631818+0000","last_clean":"2024-09-10T19:12:54.631818+0000","last_became_active":"2024-09-10T18:54:37.621269+0000","last_became_peered":"2024-09-10T18:54:37.621269+0000","last_unstale":"2024-09-10T19:12:54.631818+0000","last_undegraded":"2024-09-10T19:12:54.631818+0000","last_fullsized":"2024-09-10T19:12:54.631818+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:33.107854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:33.107854+0000","last_clean_scrub_stamp":"2024-09-10T19:12:33.107854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:52.672454+0000","scrub_duration":0.0038379249999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"859'21","reported_seq":995,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:20.443450+0000","last_change":"2024-09-10T19:13:20.443309+0000","last_active":"2024-09-10T19:13:20.443450+0000","last_peered":"2024-09-10T19:13:20.443450+0000","last_clean":"2024-09-10T19:13:20.443450+0000","last_became_active":"2024-09-10T18:49:19.632314+0000","last_became_peered":"2024-09-10T18:49:19.632314+0000","last_unstale":"2024-09-10T19:13:20.443450+0000","last_undegraded":"2024-09-10T19:13:20.443450+0000","last_fullsized":"2024-09-10T19:13:20.443450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:20.443121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:18.595041+0000","last_clean_scrub_stamp":"2024-09-10T19:13:20.443121+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.371252+0000","scrub_duration":0.0042781690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778298+0000","last_change":"2024-09-10T19:12:45.436062+0000","last_active":"2024-09-10T19:12:54.778298+0000","last_peered":"2024-09-10T19:12:54.778298+0000","last_clean":"2024-09-10T19:12:54.778298+0000","last_became_active":"2024-09-10T18:49:17.615587+0000","last_became_peered":"2024-09-10T18:49:17.615587+0000","last_unstale":"2024-09-10T19:12:54.778298+0000","last_undegraded":"2024-09-10T19:12:54.778298+0000","last_fullsized":"2024-09-10T19:12:54.778298+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.435891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.435891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.633103+0000","scrub_duration":0.0037274909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778988+0000","last_change":"2024-09-10T19:12:26.850598+0000","last_active":"2024-09-10T19:12:54.778988+0000","last_peered":"2024-09-10T19:12:54.778988+0000","last_clean":"2024-09-10T19:12:54.778988+0000","last_became_active":"2024-09-10T18:49:41.635531+0000","last_became_peered":"2024-09-10T18:49:41.635531+0000","last_unstale":"2024-09-10T19:12:54.778988+0000","last_undegraded":"2024-09-10T19:12:54.778988+0000","last_fullsized":"2024-09-10T19:12:54.778988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.850447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:46.395225+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.850447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:28.973815+0000","scrub_duration":0.0032896179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":958,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:03.161227+0000","last_change":"2024-09-10T19:13:03.161153+0000","last_active":"2024-09-10T19:13:03.161227+0000","last_peered":"2024-09-10T19:13:03.161227+0000","last_clean":"2024-09-10T19:13:03.161227+0000","last_became_active":"2024-09-10T19:09:17.624937+0000","last_became_peered":"2024-09-10T19:09:17.624937+0000","last_unstale":"2024-09-10T19:13:03.161227+0000","last_undegraded":"2024-09-10T19:13:03.161227+0000","last_fullsized":"2024-09-10T19:13:03.161227+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:03.161037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:55.601279+0000","last_clean_scrub_stamp":"2024-09-10T19:13:03.161037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.924458+0000","scrub_duration":0.0037071690000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:58.393290+0000","last_change":"2024-09-10T19:12:58.393174+0000","last_active":"2024-09-10T19:12:58.393290+0000","last_peered":"2024-09-10T19:12:58.393290+0000","last_clean":"2024-09-10T19:12:58.393290+0000","last_became_active":"2024-09-10T18:54:37.539714+0000","last_became_peered":"2024-09-10T18:54:37.539714+0000","last_unstale":"2024-09-10T19:12:58.393290+0000","last_undegraded":"2024-09-10T19:12:58.393290+0000","last_fullsized":"2024-09-10T19:12:58.393290+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.393011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:58.393011+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.393011+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.466325+0000","scrub_duration":0.003042051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"859'35","reported_seq":849,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:59.394453+0000","last_change":"2024-09-10T19:12:59.394320+0000","last_active":"2024-09-10T19:12:59.394453+0000","last_peered":"2024-09-10T19:12:59.394453+0000","last_clean":"2024-09-10T19:12:59.394453+0000","last_became_active":"2024-09-10T19:10:47.697814+0000","last_became_peered":"2024-09-10T19:10:47.697814+0000","last_unstale":"2024-09-10T19:12:59.394453+0000","last_undegraded":"2024-09-10T19:12:59.394453+0000","last_fullsized":"2024-09-10T19:12:59.394453+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'35","last_scrub_stamp":"2024-09-10T19:12:59.394122+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-09-10T18:53:38.127115+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.394122+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:25.412944+0000","scrub_duration":0.004757173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":876,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:00.059524+0000","last_change":"2024-09-10T19:13:00.059400+0000","last_active":"2024-09-10T19:13:00.059524+0000","last_peered":"2024-09-10T19:13:00.059524+0000","last_clean":"2024-09-10T19:13:00.059524+0000","last_became_active":"2024-09-10T19:09:17.622828+0000","last_became_peered":"2024-09-10T19:09:17.622828+0000","last_unstale":"2024-09-10T19:13:00.059524+0000","last_undegraded":"2024-09-10T19:13:00.059524+0000","last_fullsized":"2024-09-10T19:13:00.059524+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.059254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.059254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.787520+0000","scrub_duration":0.003501016,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":977,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:57.382246+0000","last_change":"2024-09-10T19:12:57.382119+0000","last_active":"2024-09-10T19:12:57.382246+0000","last_peered":"2024-09-10T19:12:57.382246+0000","last_clean":"2024-09-10T19:12:57.382246+0000","last_became_active":"2024-09-10T19:09:17.613499+0000","last_became_peered":"2024-09-10T19:09:17.613499+0000","last_unstale":"2024-09-10T19:12:57.382246+0000","last_undegraded":"2024-09-10T19:12:57.382246+0000","last_fullsized":"2024-09-10T19:12:57.382246+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.381964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:57.381964+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.381964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.527513+0000","scrub_duration":0.00331124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":937,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628566+0000","last_change":"2024-09-10T19:12:01.945608+0000","last_active":"2024-09-10T19:12:54.628566+0000","last_peered":"2024-09-10T19:12:54.628566+0000","last_clean":"2024-09-10T19:12:54.628566+0000","last_became_active":"2024-09-10T18:49:24.052763+0000","last_became_peered":"2024-09-10T18:49:24.052763+0000","last_unstale":"2024-09-10T19:12:54.628566+0000","last_undegraded":"2024-09-10T19:12:54.628566+0000","last_fullsized":"2024-09-10T19:12:54.628566+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:01.945455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:17.297932+0000","last_clean_scrub_stamp":"2024-09-10T19:12:01.945455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.898125+0000","scrub_duration":0.003361845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":513,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.627659+0000","last_change":"2024-09-10T19:12:26.534868+0000","last_active":"2024-09-10T19:12:54.627659+0000","last_peered":"2024-09-10T19:12:54.627659+0000","last_clean":"2024-09-10T19:12:54.627659+0000","last_became_active":"2024-09-10T19:10:47.695555+0000","last_became_peered":"2024-09-10T19:10:47.695555+0000","last_unstale":"2024-09-10T19:12:54.627659+0000","last_undegraded":"2024-09-10T19:12:54.627659+0000","last_fullsized":"2024-09-10T19:12:54.627659+0000","mapping_epoch":878,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":879,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.534630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:42.654425+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.534630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.975141+0000","scrub_duration":0.003960842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"859'21","reported_seq":972,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.035350+0000","last_change":"2024-09-10T19:13:17.035260+0000","last_active":"2024-09-10T19:13:17.035350+0000","last_peered":"2024-09-10T19:13:17.035350+0000","last_clean":"2024-09-10T19:13:17.035350+0000","last_became_active":"2024-09-10T18:49:19.630611+0000","last_became_peered":"2024-09-10T18:49:19.630611+0000","last_unstale":"2024-09-10T19:13:17.035350+0000","last_undegraded":"2024-09-10T19:13:17.035350+0000","last_fullsized":"2024-09-10T19:13:17.035350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'21","last_scrub_stamp":"2024-09-10T19:13:17.035097+0000","last_deep_scrub":"410'15","last_deep_scrub_stamp":"2024-09-10T19:06:41.386460+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.035097+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.333276+0000","scrub_duration":0.0048530689999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":946,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:15.730333+0000","last_change":"2024-09-10T19:13:15.730201+0000","last_active":"2024-09-10T19:13:15.730333+0000","last_peered":"2024-09-10T19:13:15.730333+0000","last_clean":"2024-09-10T19:13:15.730333+0000","last_became_active":"2024-09-10T18:49:17.612674+0000","last_became_peered":"2024-09-10T18:49:17.612674+0000","last_unstale":"2024-09-10T19:13:15.730333+0000","last_undegraded":"2024-09-10T19:13:15.730333+0000","last_fullsized":"2024-09-10T19:13:15.730333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.729942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.799168+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.729942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.722915+0000","scrub_duration":0.0044025180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:55.033799+0000","last_change":"2024-09-10T19:12:55.033696+0000","last_active":"2024-09-10T19:12:55.033799+0000","last_peered":"2024-09-10T19:12:55.033799+0000","last_clean":"2024-09-10T19:12:55.033799+0000","last_became_active":"2024-09-10T19:09:17.624214+0000","last_became_peered":"2024-09-10T19:09:17.624214+0000","last_unstale":"2024-09-10T19:12:55.033799+0000","last_undegraded":"2024-09-10T19:12:55.033799+0000","last_fullsized":"2024-09-10T19:12:55.033799+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.033547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:28.918873+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.033547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:59.195451+0000","scrub_duration":0.0035322420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778816+0000","last_change":"2024-09-10T19:12:29.751735+0000","last_active":"2024-09-10T19:12:54.778816+0000","last_peered":"2024-09-10T19:12:54.778816+0000","last_clean":"2024-09-10T19:12:54.778816+0000","last_became_active":"2024-09-10T18:49:24.047015+0000","last_became_peered":"2024-09-10T18:49:24.047015+0000","last_unstale":"2024-09-10T19:12:54.778816+0000","last_undegraded":"2024-09-10T19:12:54.778816+0000","last_fullsized":"2024-09-10T19:12:54.778816+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:29.751545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:50:30.205599+0000","last_clean_scrub_stamp":"2024-09-10T19:12:29.751545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:45.935489+0000","scrub_duration":0.0033452339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":767,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.046899+0000","last_change":"2024-09-10T19:13:05.046783+0000","last_active":"2024-09-10T19:13:05.046899+0000","last_peered":"2024-09-10T19:13:05.046899+0000","last_clean":"2024-09-10T19:13:05.046899+0000","last_became_active":"2024-09-10T19:09:17.627510+0000","last_became_peered":"2024-09-10T19:09:17.627510+0000","last_unstale":"2024-09-10T19:13:05.046899+0000","last_undegraded":"2024-09-10T19:13:05.046899+0000","last_fullsized":"2024-09-10T19:13:05.046899+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.046623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:14.886297+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.046623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.429597+0000","scrub_duration":0.0034561169999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"859'25","reported_seq":989,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781288+0000","last_change":"2024-09-10T19:12:33.679136+0000","last_active":"2024-09-10T19:12:54.781288+0000","last_peered":"2024-09-10T19:12:54.781288+0000","last_clean":"2024-09-10T19:12:54.781288+0000","last_became_active":"2024-09-10T19:09:17.623786+0000","last_became_peered":"2024-09-10T19:09:17.623786+0000","last_unstale":"2024-09-10T19:12:54.781288+0000","last_undegraded":"2024-09-10T19:12:54.781288+0000","last_fullsized":"2024-09-10T19:12:54.781288+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'25","last_scrub_stamp":"2024-09-10T19:12:33.678963+0000","last_deep_scrub":"676'19","last_deep_scrub_stamp":"2024-09-10T19:05:34.022377+0000","last_clean_scrub_stamp":"2024-09-10T19:12:33.678963+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.728656+0000","scrub_duration":0.0049950280000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":945,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780089+0000","last_change":"2024-09-10T19:12:30.479805+0000","last_active":"2024-09-10T19:12:54.780089+0000","last_peered":"2024-09-10T19:12:54.780089+0000","last_clean":"2024-09-10T19:12:54.780089+0000","last_became_active":"2024-09-10T18:49:17.616394+0000","last_became_peered":"2024-09-10T18:49:17.616394+0000","last_unstale":"2024-09-10T19:12:54.780089+0000","last_undegraded":"2024-09-10T19:12:54.780089+0000","last_fullsized":"2024-09-10T19:12:54.780089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:30.479648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:30.479648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.977977+0000","scrub_duration":0.0032843109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":926,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628680+0000","last_change":"2024-09-10T19:11:53.887313+0000","last_active":"2024-09-10T19:12:54.628680+0000","last_peered":"2024-09-10T19:12:54.628680+0000","last_clean":"2024-09-10T19:12:54.628680+0000","last_became_active":"2024-09-10T18:49:22.061579+0000","last_became_peered":"2024-09-10T18:49:22.061579+0000","last_unstale":"2024-09-10T19:12:54.628680+0000","last_undegraded":"2024-09-10T19:12:54.628680+0000","last_fullsized":"2024-09-10T19:12:54.628680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.887159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:21.015454+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.887159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:14.288677+0000","scrub_duration":0.0039200759999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":934,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:25.823509+0000","last_change":"2024-09-10T19:13:25.823429+0000","last_active":"2024-09-10T19:13:25.823509+0000","last_peered":"2024-09-10T19:13:25.823509+0000","last_clean":"2024-09-10T19:13:25.823509+0000","last_became_active":"2024-09-10T18:49:24.370166+0000","last_became_peered":"2024-09-10T18:49:24.370166+0000","last_unstale":"2024-09-10T19:13:25.823509+0000","last_undegraded":"2024-09-10T19:13:25.823509+0000","last_fullsized":"2024-09-10T19:13:25.823509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:25.823279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:09:54.676361+0000","last_clean_scrub_stamp":"2024-09-10T19:13:25.823279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.975751+0000","scrub_duration":0.003183281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":859,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:18.969644+0000","last_change":"2024-09-10T19:13:18.969563+0000","last_active":"2024-09-10T19:13:18.969644+0000","last_peered":"2024-09-10T19:13:18.969644+0000","last_clean":"2024-09-10T19:13:18.969644+0000","last_became_active":"2024-09-10T18:54:37.556736+0000","last_became_peered":"2024-09-10T18:54:37.556736+0000","last_unstale":"2024-09-10T19:13:18.969644+0000","last_undegraded":"2024-09-10T19:13:18.969644+0000","last_fullsized":"2024-09-10T19:13:18.969644+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:18.969409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:46.380935+0000","last_clean_scrub_stamp":"2024-09-10T19:13:18.969409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.167954+0000","scrub_duration":0.0034001330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"108'2","reported_seq":947,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:06.567668+0000","last_change":"2024-09-10T19:13:06.567544+0000","last_active":"2024-09-10T19:13:06.567668+0000","last_peered":"2024-09-10T19:13:06.567668+0000","last_clean":"2024-09-10T19:13:06.567668+0000","last_became_active":"2024-09-10T18:49:24.050051+0000","last_became_peered":"2024-09-10T18:49:24.050051+0000","last_unstale":"2024-09-10T19:13:06.567668+0000","last_undegraded":"2024-09-10T19:13:06.567668+0000","last_fullsized":"2024-09-10T19:13:06.567668+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'2","last_scrub_stamp":"2024-09-10T19:13:06.567370+0000","last_deep_scrub":"108'2","last_deep_scrub_stamp":"2024-09-10T18:52:44.380117+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.567370+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.454237+0000","scrub_duration":0.0043217389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"859'46","reported_seq":933,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.073180+0000","last_change":"2024-09-10T19:13:04.072985+0000","last_active":"2024-09-10T19:13:04.073180+0000","last_peered":"2024-09-10T19:13:04.073180+0000","last_clean":"2024-09-10T19:13:04.073180+0000","last_became_active":"2024-09-10T19:09:17.626306+0000","last_became_peered":"2024-09-10T19:09:17.626306+0000","last_unstale":"2024-09-10T19:13:04.073180+0000","last_undegraded":"2024-09-10T19:13:04.073180+0000","last_fullsized":"2024-09-10T19:13:04.073180+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"859'46","last_scrub_stamp":"2024-09-10T19:13:04.072812+0000","last_deep_scrub":"383'22","last_deep_scrub_stamp":"2024-09-10T18:58:59.788487+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.072812+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.672103+0000","scrub_duration":0.0055804890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":946,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.776582+0000","last_change":"2024-09-10T19:12:50.753994+0000","last_active":"2024-09-10T19:12:54.776582+0000","last_peered":"2024-09-10T19:12:54.776582+0000","last_clean":"2024-09-10T19:12:54.776582+0000","last_became_active":"2024-09-10T18:49:17.613593+0000","last_became_peered":"2024-09-10T18:49:17.613593+0000","last_unstale":"2024-09-10T19:12:54.776582+0000","last_undegraded":"2024-09-10T19:12:54.776582+0000","last_fullsized":"2024-09-10T19:12:54.776582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.753839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.753839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.661916+0000","scrub_duration":0.0033401160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":940,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.631990+0000","last_change":"2024-09-10T19:12:45.106271+0000","last_active":"2024-09-10T19:12:54.631990+0000","last_peered":"2024-09-10T19:12:54.631990+0000","last_clean":"2024-09-10T19:12:54.631990+0000","last_became_active":"2024-09-10T18:49:22.061591+0000","last_became_peered":"2024-09-10T18:49:22.061591+0000","last_unstale":"2024-09-10T19:12:54.631990+0000","last_undegraded":"2024-09-10T19:12:54.631990+0000","last_fullsized":"2024-09-10T19:12:54.631990+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.106072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:36.189093+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.106072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.364792+0000","scrub_duration":0.0037103420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.778259+0000","last_change":"2024-09-10T19:12:26.456594+0000","last_active":"2024-09-10T19:12:54.778259+0000","last_peered":"2024-09-10T19:12:54.778259+0000","last_clean":"2024-09-10T19:12:54.778259+0000","last_became_active":"2024-09-10T18:54:37.701587+0000","last_became_peered":"2024-09-10T18:54:37.701587+0000","last_unstale":"2024-09-10T19:12:54.778259+0000","last_undegraded":"2024-09-10T19:12:54.778259+0000","last_fullsized":"2024-09-10T19:12:54.778259+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:26.456445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:54:31.683713+0000","last_clean_scrub_stamp":"2024-09-10T19:12:26.456445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:55.415998+0000","scrub_duration":0.0033016310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":861,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779502+0000","last_change":"2024-09-10T19:12:45.976155+0000","last_active":"2024-09-10T19:12:54.779502+0000","last_peered":"2024-09-10T19:12:54.779502+0000","last_clean":"2024-09-10T19:12:54.779502+0000","last_became_active":"2024-09-10T19:09:17.625423+0000","last_became_peered":"2024-09-10T19:09:17.625423+0000","last_unstale":"2024-09-10T19:12:54.779502+0000","last_undegraded":"2024-09-10T19:12:54.779502+0000","last_fullsized":"2024-09-10T19:12:54.779502+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.975996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:45.975996+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.975996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.533211+0000","scrub_duration":0.0039800649999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"901'25","reported_seq":993,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:17.892307+0000","last_change":"2024-09-10T19:13:17.892228+0000","last_active":"2024-09-10T19:13:17.892307+0000","last_peered":"2024-09-10T19:13:17.892307+0000","last_clean":"2024-09-10T19:13:17.892307+0000","last_became_active":"2024-09-10T19:09:17.624891+0000","last_became_peered":"2024-09-10T19:09:17.624891+0000","last_unstale":"2024-09-10T19:13:17.892307+0000","last_undegraded":"2024-09-10T19:13:17.892307+0000","last_fullsized":"2024-09-10T19:13:17.892307+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'25","last_scrub_stamp":"2024-09-10T19:13:17.892088+0000","last_deep_scrub":"901'25","last_deep_scrub_stamp":"2024-09-10T19:13:17.892088+0000","last_clean_scrub_stamp":"2024-09-10T19:13:17.892088+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:26.410727+0000","scrub_duration":0.0081228310000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":940,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:13.910898+0000","last_change":"2024-09-10T19:13:13.910819+0000","last_active":"2024-09-10T19:13:13.910898+0000","last_peered":"2024-09-10T19:13:13.910898+0000","last_clean":"2024-09-10T19:13:13.910898+0000","last_became_active":"2024-09-10T18:49:17.613077+0000","last_became_peered":"2024-09-10T18:49:17.613077+0000","last_unstale":"2024-09-10T19:13:13.910898+0000","last_undegraded":"2024-09-10T19:13:13.910898+0000","last_fullsized":"2024-09-10T19:13:13.910898+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:13.910669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:06:52.165627+0000","last_clean_scrub_stamp":"2024-09-10T19:13:13.910669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.344950+0000","scrub_duration":0.0034429040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"873'13","reported_seq":1270,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:26.921866+0000","last_change":"2024-09-10T19:13:26.435758+0000","last_active":"2024-09-10T19:13:26.921866+0000","last_peered":"2024-09-10T19:13:26.921866+0000","last_clean":"2024-09-10T19:13:26.921866+0000","last_became_active":"2024-09-10T18:49:41.637119+0000","last_became_peered":"2024-09-10T18:49:41.637119+0000","last_unstale":"2024-09-10T19:13:26.921866+0000","last_undegraded":"2024-09-10T19:13:26.921866+0000","last_fullsized":"2024-09-10T19:13:26.921866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"873'13","last_scrub_stamp":"2024-09-10T19:13:26.435595+0000","last_deep_scrub":"873'13","last_deep_scrub_stamp":"2024-09-10T19:10:24.001015+0000","last_clean_scrub_stamp":"2024-09-10T19:13:26.435595+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.590649+0000","scrub_duration":0.004271525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":862,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779604+0000","last_change":"2024-09-10T19:12:14.508081+0000","last_active":"2024-09-10T19:12:54.779604+0000","last_peered":"2024-09-10T19:12:54.779604+0000","last_clean":"2024-09-10T19:12:54.779604+0000","last_became_active":"2024-09-10T18:54:37.709666+0000","last_became_peered":"2024-09-10T18:54:37.709666+0000","last_unstale":"2024-09-10T19:12:54.779604+0000","last_undegraded":"2024-09-10T19:12:54.779604+0000","last_fullsized":"2024-09-10T19:12:54.779604+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:14.507954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:00.644299+0000","last_clean_scrub_stamp":"2024-09-10T19:12:14.507954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.494942+0000","scrub_duration":0.0025421939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":968,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:04.360640+0000","last_change":"2024-09-10T19:13:04.360465+0000","last_active":"2024-09-10T19:13:04.360640+0000","last_peered":"2024-09-10T19:13:04.360640+0000","last_clean":"2024-09-10T19:13:04.360640+0000","last_became_active":"2024-09-10T18:49:24.367159+0000","last_became_peered":"2024-09-10T18:49:24.367159+0000","last_unstale":"2024-09-10T19:13:04.360640+0000","last_undegraded":"2024-09-10T19:13:04.360640+0000","last_fullsized":"2024-09-10T19:13:04.360640+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.360223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:51:41.705954+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.360223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.574679+0000","scrub_duration":0.0036477329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"70'1","reported_seq":944,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.779083+0000","last_change":"2024-09-10T19:12:09.534767+0000","last_active":"2024-09-10T19:12:54.779083+0000","last_peered":"2024-09-10T19:12:54.779083+0000","last_clean":"2024-09-10T19:12:54.779083+0000","last_became_active":"2024-09-10T18:49:17.613931+0000","last_became_peered":"2024-09-10T18:49:17.613931+0000","last_unstale":"2024-09-10T19:12:54.779083+0000","last_undegraded":"2024-09-10T19:12:54.779083+0000","last_fullsized":"2024-09-10T19:12:54.779083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'1","last_scrub_stamp":"2024-09-10T19:12:09.534602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:49:16.574846+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.534602+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:36.543114+0000","scrub_duration":0.0035922110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"900'11","reported_seq":960,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:13:05.129734+0000","last_change":"2024-09-10T19:13:05.129607+0000","last_active":"2024-09-10T19:13:05.129734+0000","last_peered":"2024-09-10T19:13:05.129734+0000","last_clean":"2024-09-10T19:13:05.129734+0000","last_became_active":"2024-09-10T18:49:19.634749+0000","last_became_peered":"2024-09-10T18:49:19.634749+0000","last_unstale":"2024-09-10T19:13:05.129734+0000","last_undegraded":"2024-09-10T19:13:05.129734+0000","last_fullsized":"2024-09-10T19:13:05.129734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'11","last_scrub_stamp":"2024-09-10T19:13:05.129451+0000","last_deep_scrub":"900'11","last_deep_scrub_stamp":"2024-09-10T19:13:05.129451+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.129451+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:29.801663+0000","scrub_duration":0.0072537269999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":981,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.780917+0000","last_change":"2024-09-10T19:12:15.178732+0000","last_active":"2024-09-10T19:12:54.780917+0000","last_peered":"2024-09-10T19:12:54.780917+0000","last_clean":"2024-09-10T19:12:54.780917+0000","last_became_active":"2024-09-10T19:09:17.615219+0000","last_became_peered":"2024-09-10T19:09:17.615219+0000","last_unstale":"2024-09-10T19:12:54.780917+0000","last_undegraded":"2024-09-10T19:12:54.780917+0000","last_fullsized":"2024-09-10T19:12:54.780917+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:15.178658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:05:17.968019+0000","last_clean_scrub_stamp":"2024-09-10T19:12:15.178658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.379149+0000","scrub_duration":0.0021276839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":863,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.781980+0000","last_change":"2024-09-10T19:12:36.716875+0000","last_active":"2024-09-10T19:12:54.781980+0000","last_peered":"2024-09-10T19:12:54.781980+0000","last_clean":"2024-09-10T19:12:54.781980+0000","last_became_active":"2024-09-10T18:54:37.588908+0000","last_became_peered":"2024-09-10T18:54:37.588908+0000","last_unstale":"2024-09-10T19:12:54.781980+0000","last_undegraded":"2024-09-10T19:12:54.781980+0000","last_fullsized":"2024-09-10T19:12:54.781980+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:36.716727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:06.774232+0000","last_clean_scrub_stamp":"2024-09-10T19:12:36.716727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.432775+0000","scrub_duration":0.0032699080000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":934,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.630756+0000","last_change":"2024-09-10T19:12:18.981781+0000","last_active":"2024-09-10T19:12:54.630756+0000","last_peered":"2024-09-10T19:12:54.630756+0000","last_clean":"2024-09-10T19:12:54.630756+0000","last_became_active":"2024-09-10T18:49:24.365551+0000","last_became_peered":"2024-09-10T18:49:24.365551+0000","last_unstale":"2024-09-10T19:12:54.630756+0000","last_undegraded":"2024-09-10T19:12:54.630756+0000","last_fullsized":"2024-09-10T19:12:54.630756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:18.981632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:56:33.964103+0000","last_clean_scrub_stamp":"2024-09-10T19:12:18.981632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:48.322860+0000","scrub_duration":0.0036854710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":940,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-10T19:13:10.917971+0000","last_change":"2024-09-10T19:13:10.917868+0000","last_active":"2024-09-10T19:13:10.917971+0000","last_peered":"2024-09-10T19:13:10.917971+0000","last_clean":"2024-09-10T19:13:10.917971+0000","last_became_active":"2024-09-10T18:49:17.612503+0000","last_became_peered":"2024-09-10T18:49:17.612503+0000","last_unstale":"2024-09-10T19:13:10.917971+0000","last_undegraded":"2024-09-10T19:13:10.917971+0000","last_fullsized":"2024-09-10T19:13:10.917971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:10.917727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:10.917727+0000","last_clean_scrub_stamp":"2024-09-10T19:13:10.917727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.898023+0000","scrub_duration":0.003015958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"877'25","reported_seq":1007,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.628245+0000","last_change":"2024-09-10T19:12:19.551853+0000","last_active":"2024-09-10T19:12:54.628245+0000","last_peered":"2024-09-10T19:12:54.628245+0000","last_clean":"2024-09-10T19:12:54.628245+0000","last_became_active":"2024-09-10T19:09:17.613813+0000","last_became_peered":"2024-09-10T19:09:17.613813+0000","last_unstale":"2024-09-10T19:12:54.628245+0000","last_undegraded":"2024-09-10T19:12:54.628245+0000","last_fullsized":"2024-09-10T19:12:54.628245+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-10T19:12:19.551679+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-10T19:10:43.846279+0000","last_clean_scrub_stamp":"2024-09-10T19:12:19.551679+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.808352+0000","scrub_duration":0.0048679129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":943,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.629589+0000","last_change":"2024-09-10T19:12:32.020746+0000","last_active":"2024-09-10T19:12:54.629589+0000","last_peered":"2024-09-10T19:12:54.629589+0000","last_clean":"2024-09-10T19:12:54.629589+0000","last_became_active":"2024-09-10T19:09:17.621963+0000","last_became_peered":"2024-09-10T19:09:17.621963+0000","last_unstale":"2024-09-10T19:12:54.629589+0000","last_undegraded":"2024-09-10T19:12:54.629589+0000","last_fullsized":"2024-09-10T19:12:54.629589+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:32.020586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:51.936453+0000","last_clean_scrub_stamp":"2024-09-10T19:12:32.020586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:35.091894+0000","scrub_duration":0.0033598090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":879,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-10T19:12:54.777803+0000","last_change":"2024-09-10T19:12:37.380309+0000","last_active":"2024-09-10T19:12:54.777803+0000","last_peered":"2024-09-10T19:12:54.777803+0000","last_clean":"2024-09-10T19:12:54.777803+0000","last_became_active":"2024-09-10T19:09:17.615496+0000","last_became_peered":"2024-09-10T19:09:17.615496+0000","last_unstale":"2024-09-10T19:12:54.777803+0000","last_undegraded":"2024-09-10T19:12:54.777803+0000","last_fullsized":"2024-09-10T19:12:54.777803+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:37.380157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:03:51.945980+0000","last_clean_scrub_stamp":"2024-09-10T19:12:37.380157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:54.489922+0000","scrub_duration":0.0036922259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1427,"num_read_kb":1422,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1552,"num_read_kb":1309,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1089,"ondisk_log_size":1089,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":891,"ondisk_log_size":891,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":62,"seq":266287972656,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307736,"kb_used_data":2064,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437416,"statfs":{"total":95995035648,"available":95679913984,"internally_reserved":0,"allocated":2113536,"data_stored":844590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.93600000000000005,"15min":0.92200000000000004},"min":{"1min":0.66500000000000004,"5min":0.42999999999999999,"15min":0.40300000000000002},"max":{"1min":1.3149999999999999,"5min":2.1589999999999998,"15min":2.839},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95599999999999996,"15min":0.95299999999999996},"min":{"1min":0.53400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.2889999999999999,"5min":2.8079999999999998,"15min":2.9550000000000001},"last":1.3080000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.94799999999999995,"15min":0.93300000000000005},"min":{"1min":0.72299999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.151,"5min":1.921,"15min":1.921},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.0569999999999999,"15min":1.0649999999999999},"min":{"1min":0.61899999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.3220000000000001,"5min":1.881,"15min":1.881},"last":0.92900000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.98699999999999999,"15min":1.095},"min":{"1min":0.504,"5min":0.504,"15min":0.45000000000000001},"max":{"1min":1.3260000000000001,"5min":1.9750000000000001,"15min":31.166},"last":1},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.103,"15min":1.1160000000000001},"min":{"1min":0.69799999999999995,"5min":0.54200000000000004,"15min":0.443},"max":{"1min":5.218,"5min":5.218,"15min":27.709},"last":1.431}]},{"osd":3,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0069999999999999,"15min":1.0249999999999999},"min":{"1min":0.76000000000000001,"5min":0.59199999999999997,"15min":0.51700000000000002},"max":{"1min":1.4850000000000001,"5min":2.8799999999999999,"15min":4.4800000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96699999999999997},"min":{"1min":0.57599999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3,"5min":2.0459999999999998,"15min":4.2060000000000004},"last":1.2350000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.96299999999999997,"15min":0.997},"min":{"1min":0.63600000000000001,"5min":0.54800000000000004,"15min":0.34799999999999998},"max":{"1min":1.339,"5min":1.3660000000000001,"15min":1.5760000000000001},"last":1.393},{"interface":"front","average":{"1min":1.054,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.76500000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2649999999999999,"5min":2.6070000000000002,"15min":2.6070000000000002},"last":1.54}]},{"osd":5,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.95099999999999996,"15min":0.95699999999999996},"min":{"1min":0.73399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.234,"5min":1.3200000000000001,"15min":3.1970000000000001},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.181,"5min":1.1220000000000001,"15min":1.1279999999999999},"min":{"1min":0.85599999999999998,"5min":0.52900000000000003,"15min":0.47499999999999998},"max":{"1min":1.421,"5min":1.7829999999999999,"15min":4.8710000000000004},"last":1.1539999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.1699999999999999,"15min":1.1599999999999999},"min":{"1min":1.0229999999999999,"5min":0.56299999999999994,"15min":0.48699999999999999},"max":{"1min":1.4019999999999999,"5min":1.706,"15min":1.766},"last":1.262},{"interface":"front","average":{"1min":1.252,"5min":1.2090000000000001,"15min":1.1859999999999999},"min":{"1min":0.80200000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.5960000000000001,"5min":2.9430000000000001,"15min":2.96},"last":1.1839999999999999}]}]},{"osd":6,"up_from":52,"seq":223338299706,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307396,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437756,"statfs":{"total":95995035648,"available":95680262144,"internally_reserved":0,"allocated":1765376,"data_stored":515149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.89700000000000002,"15min":0.89300000000000002},"min":{"1min":0.58999999999999997,"5min":0.47999999999999998,"15min":0.26900000000000002},"max":{"1min":1.3360000000000001,"5min":1.385,"15min":1.8839999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.95399999999999996,"15min":0.96999999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.40300000000000002},"max":{"1min":1.254,"5min":4.1420000000000003,"15min":5.7309999999999999},"last":0.80500000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.008,"15min":0.94899999999999995},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.345,"5min":4.1130000000000004,"15min":4.1130000000000004},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.097,"5min":1.129,"15min":1.107},"min":{"1min":0.76700000000000002,"5min":0.504,"15min":0.504},"max":{"1min":1.387,"5min":4.1870000000000003,"15min":4.1870000000000003},"last":1.163}]},{"osd":2,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.94599999999999995},"min":{"1min":0.624,"5min":0.42399999999999999,"15min":0.33500000000000002},"max":{"1min":1.1850000000000001,"5min":4.069,"15min":4.069},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.94199999999999995,"15min":0.94699999999999995},"min":{"1min":0.54600000000000004,"5min":0.51400000000000001,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":4.0350000000000001,"15min":6.0970000000000004},"last":0.64200000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0169999999999999,"15min":1},"min":{"1min":0.77400000000000002,"5min":0.58699999999999997,"15min":0.40400000000000003},"max":{"1min":1.2250000000000001,"5min":3.972,"15min":3.972},"last":1.298},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.96199999999999997,"15min":0.97099999999999997},"min":{"1min":0.74199999999999999,"5min":0.56699999999999995,"15min":0.40400000000000003},"max":{"1min":1.345,"5min":4.2270000000000003,"15min":4.2270000000000003},"last":0.97399999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.95799999999999996,"15min":0.97799999999999998},"min":{"1min":0.65500000000000003,"5min":0.55800000000000005,"15min":0.44400000000000001},"max":{"1min":1.3120000000000001,"5min":1.79,"15min":6.008},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0609999999999999,"15min":1.0580000000000001},"min":{"1min":0.626,"5min":0.59499999999999997,"15min":0.42299999999999999},"max":{"1min":1.3799999999999999,"5min":1.48,"15min":2.3039999999999998},"last":1.093}]},{"osd":5,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0389999999999999,"15min":1.0640000000000001},"min":{"1min":0.72499999999999998,"5min":0.52300000000000002,"15min":0.5},"max":{"1min":1.2090000000000001,"5min":1.9179999999999999,"15min":5.9560000000000004},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1000000000000001,"15min":1.109},"min":{"1min":0.76300000000000001,"5min":0.60699999999999998,"15min":0.47599999999999998},"max":{"1min":1.3879999999999999,"5min":1.5820000000000001,"15min":3.5169999999999999},"last":1.2090000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.0269999999999999,"15min":1.04},"min":{"1min":0.755,"5min":0.51000000000000001,"15min":0.47899999999999998},"max":{"1min":1.363,"5min":1.393,"15min":1.8320000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.131,"5min":1.151,"15min":1.1930000000000001},"min":{"1min":0.88200000000000001,"5min":0.77000000000000002,"15min":0.51200000000000001},"max":{"1min":1.369,"5min":2.02,"15min":6.0380000000000003},"last":1.0249999999999999}]}]},{"osd":5,"up_from":43,"seq":184683594052,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441336,"statfs":{"total":95995035648,"available":95683928064,"internally_reserved":0,"allocated":2359296,"data_stored":1105591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.92600000000000005,"15min":0.95299999999999996},"min":{"1min":0.68200000000000005,"5min":0.55200000000000005,"15min":0.46500000000000002},"max":{"1min":1.3899999999999999,"5min":2.1840000000000002,"15min":5.4640000000000004},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.91800000000000004,"15min":0.92600000000000005},"min":{"1min":0.69099999999999995,"5min":0.433,"15min":0.433},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":5.4790000000000001},"last":0.70799999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.93400000000000005,"15min":0.93100000000000005},"min":{"1min":0.64100000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.2130000000000001,"5min":1.387,"15min":1.387},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.97199999999999998,"15min":0.98099999999999998},"min":{"1min":0.72599999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.2170000000000001,"5min":1.377,"15min":1.377},"last":0.82399999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.70899999999999996,"5min":0.60299999999999998,"15min":0.59299999999999997},"max":{"1min":1.2509999999999999,"5min":2.9300000000000002,"15min":7.9340000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.96499999999999997,"15min":1},"min":{"1min":0.69599999999999995,"5min":0.60999999999999999,"15min":0.52000000000000002},"max":{"1min":1.294,"5min":1.532,"15min":6.3280000000000003},"last":0.86299999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0309999999999999,"15min":1.0029999999999999},"min":{"1min":0.83599999999999997,"5min":0.73099999999999998,"15min":0.58799999999999997},"max":{"1min":1.218,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":1.012},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.048,"15min":1.0640000000000001},"min":{"1min":0.80200000000000005,"5min":0.65600000000000003,"15min":0.60799999999999998},"max":{"1min":1.3180000000000001,"5min":2.0990000000000002,"15min":5.3810000000000002},"last":0.80200000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.89800000000000002,"15min":0.92800000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.2969999999999999,"5min":1.3200000000000001,"15min":5.5170000000000003},"last":0.497},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.996,"15min":0.98299999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.39400000000000002},"max":{"1min":1.4350000000000001,"5min":2.895,"15min":2.895},"last":0.54500000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.1240000000000001,"15min":1.1220000000000001},"min":{"1min":1.0329999999999999,"5min":0.63400000000000001,"15min":0.57499999999999996},"max":{"1min":1.423,"5min":2.8399999999999999,"15min":5.2009999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1040000000000001,"15min":1.1379999999999999},"min":{"1min":0.76900000000000002,"5min":0.46800000000000003,"15min":0.42799999999999999},"max":{"1min":1.4570000000000001,"5min":2.343,"15min":5.4240000000000004},"last":0.67400000000000004}]},{"osd":7,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.0389999999999999,"15min":1.095},"min":{"1min":0.71399999999999997,"5min":0.57099999999999995,"15min":0.41999999999999998},"max":{"1min":1.429,"5min":1.4690000000000001,"15min":5.7210000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.288,"5min":1.1599999999999999,"15min":1.151},"min":{"1min":0.98499999999999999,"5min":0.69099999999999995,"15min":0.42599999999999999},"max":{"1min":1.7649999999999999,"5min":3.1059999999999999,"15min":4.2480000000000002},"last":0.64800000000000002}]}]},{"osd":4,"up_from":35,"seq":150323855696,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311588,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433564,"statfs":{"total":95995035648,"available":95675969536,"internally_reserved":0,"allocated":1798144,"data_stored":516473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.875,"15min":0.879},"min":{"1min":0.60099999999999998,"5min":0.48099999999999998,"15min":0.371},"max":{"1min":1.3100000000000001,"5min":1.3819999999999999,"15min":2.0179999999999998},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.98699999999999999,"15min":0.98099999999999998},"min":{"1min":0.629,"5min":0.52400000000000002,"15min":0.44600000000000001},"max":{"1min":1.55,"5min":1.55,"15min":2.1930000000000001},"last":1.1870000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:13:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.92000000000000004,"15min":0.90000000000000002},"min":{"1min":0.63500000000000001,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.3480000000000001,"5min":1.387,"15min":1.387},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.069,"15min":1.085},"min":{"1min":0.78100000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.93600000000000005,"15min":1.04},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.46600000000000003},"max":{"1min":1.272,"5min":1.4550000000000001,"15min":18.853000000000002},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1,"15min":1.0820000000000001},"min":{"1min":0.72299999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.409,"5min":1.579,"15min":25.652999999999999},"last":1.5209999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.96999999999999997,"15min":1.0149999999999999},"min":{"1min":0.76200000000000001,"5min":0.51900000000000002,"15min":0.49099999999999999},"max":{"1min":1.345,"5min":1.421,"15min":4.2670000000000003},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.012,"5min":0.94099999999999995,"15min":0.98299999999999998},"min":{"1min":0.78500000000000003,"5min":0.51000000000000001,"15min":0.49099999999999999},"max":{"1min":1.214,"5min":1.405,"15min":2.0739999999999998},"last":1.1539999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88600000000000001,"15min":0.94599999999999995},"min":{"1min":0.58499999999999996,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.365,"5min":1.365,"15min":1.8009999999999999},"last":1.363},{"interface":"front","average":{"1min":1.042,"5min":1.022,"15min":1.0700000000000001},"min":{"1min":0.63900000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2989999999999999,"5min":1.5089999999999999,"15min":2.0259999999999998},"last":1.4279999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.94899999999999995,"15min":1.0209999999999999},"min":{"1min":0.751,"5min":0.307,"15min":0.307},"max":{"1min":1.2949999999999999,"5min":1.5589999999999999,"15min":2.1699999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.087,"15min":1.1579999999999999},"min":{"1min":0.72999999999999998,"5min":0.54300000000000004,"15min":0.48799999999999999},"max":{"1min":1.46,"5min":1.46,"15min":2.9590000000000001},"last":1.4630000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1399999999999999,"15min":1.167},"min":{"1min":0.86499999999999999,"5min":0.71199999999999997,"15min":0.56499999999999995},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":2.6739999999999999},"last":1.595},{"interface":"front","average":{"1min":1.071,"5min":1.0760000000000001,"15min":1.1539999999999999},"min":{"1min":0.79800000000000004,"5min":0.73799999999999999,"15min":0.40899999999999997},"max":{"1min":1.2989999999999999,"5min":1.536,"15min":3.7850000000000001},"last":1.2709999999999999}]}]},{"osd":3,"up_from":29,"seq":124554051931,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441336,"statfs":{"total":95995035648,"available":95683928064,"internally_reserved":0,"allocated":2359296,"data_stored":1106443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65900000000000003,"15min":0.66800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.29499999999999998},"max":{"1min":0.96499999999999997,"5min":1.4830000000000001,"15min":1.6539999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.749,"15min":0.73999999999999999},"min":{"1min":0.45900000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.96599999999999997,"5min":1.79,"15min":1.8660000000000001},"last":0.83299999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:13:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.86199999999999999,"15min":0.84599999999999997},"min":{"1min":0.59199999999999997,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1060000000000001,"5min":1.704,"15min":1.704},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.746,"15min":0.74399999999999999},"min":{"1min":0.317,"5min":0.316,"15min":0.316},"max":{"1min":0.99099999999999999,"5min":1.365,"15min":1.365},"last":0.58199999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:12:54 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.78300000000000003,"15min":0.83799999999999997},"min":{"1min":0.442,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.1850000000000001,"5min":1.661,"15min":2.073},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.80200000000000005,"15min":0.81399999999999995},"min":{"1min":0.45000000000000001,"5min":0.379,"15min":0.32600000000000001},"max":{"1min":1.016,"5min":2.032,"15min":4.4740000000000002},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0489999999999999,"15min":1.097},"min":{"1min":0.84399999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.55,"5min":1.748,"15min":3.2469999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.1479999999999999,"15min":1.2010000000000001},"min":{"1min":0.92200000000000004,"5min":0.77600000000000002,"15min":0.64400000000000002},"max":{"1min":1.6339999999999999,"5min":1.956,"15min":3.2850000000000001},"last":1.089}]},{"osd":5,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.0580000000000001,"15min":1.1060000000000001},"min":{"1min":0.73799999999999999,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.593,"5min":1.8,"15min":2.2770000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.97299999999999998,"15min":1.024},"min":{"1min":0.70799999999999996,"5min":0.68500000000000005,"15min":0.53500000000000003},"max":{"1min":1.6080000000000001,"5min":1.8169999999999999,"15min":4.6619999999999999},"last":1.032}]},{"osd":6,"last update":"Tue Sep 10 19:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.135,"15min":1.1659999999999999},"min":{"1min":0.95199999999999996,"5min":0.69399999999999995,"15min":0.59499999999999997},"max":{"1min":1.679,"5min":1.8859999999999999,"15min":2.206},"last":1.258},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.125,"15min":1.111},"min":{"1min":0.82699999999999996,"5min":0.68300000000000005,"15min":0.56999999999999995},"max":{"1min":1.46,"5min":2.2200000000000002,"15min":2.996},"last":1.1659999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.0600000000000001,"15min":1.097},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.59899999999999998},"max":{"1min":1.359,"5min":1.778,"15min":4.0229999999999997},"last":1.143},{"interface":"front","average":{"1min":1.089,"5min":1.177,"15min":1.24},"min":{"1min":0.871,"5min":0.70799999999999996,"15min":0.66000000000000003},"max":{"1min":1.3140000000000001,"5min":1.921,"15min":4.5350000000000001},"last":1.1910000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378980,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311824,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433328,"statfs":{"total":95995035648,"available":95675727872,"internally_reserved":0,"allocated":2039808,"data_stored":778955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66600000000000004,"15min":0.83399999999999996},"min":{"1min":0.47699999999999998,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.1080000000000001,"5min":1.2170000000000001,"15min":43.927999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.74199999999999999,"15min":0.95499999999999996},"min":{"1min":0.54000000000000004,"5min":0.36199999999999999,"15min":0.30299999999999999},"max":{"1min":1.1240000000000001,"5min":1.26,"15min":49.475000000000001},"last":0.81299999999999994}]},{"osd":1,"last update":"Tue Sep 10 19:13:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73299999999999998,"15min":0.72699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.23,"5min":1.23,"15min":1.23},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.85599999999999998,"15min":0.82799999999999996},"min":{"1min":0.51000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.504,"5min":1.504,"15min":1.504},"last":0.93700000000000006}]},{"osd":3,"last update":"Tue Sep 10 19:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.76300000000000001,"15min":0.85899999999999999},"min":{"1min":0.56200000000000006,"5min":0.38900000000000001,"15min":0.318},"max":{"1min":1.869,"5min":1.869,"15min":6.4020000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.80600000000000005,"15min":0.92200000000000004},"min":{"1min":0.61099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.2869999999999999,"5min":1.361,"15min":9.8219999999999992},"last":0.997}]},{"osd":4,"last update":"Tue Sep 10 19:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.002,"15min":1.177},"min":{"1min":0.79200000000000004,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":38.079999999999998},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.042,"15min":1.2390000000000001},"min":{"1min":0.876,"5min":0.42199999999999999,"15min":0.40400000000000003},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":37.991},"last":1.1020000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.087,"15min":1.093},"min":{"1min":0.875,"5min":0.60699999999999998,"15min":0.49099999999999999},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.125,"15min":1.137},"min":{"1min":0.80500000000000005,"5min":0.71599999999999997,"15min":0.59999999999999998},"max":{"1min":2.21,"5min":3.0819999999999999,"15min":6.0620000000000003},"last":1.3169999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.1160000000000001,"15min":1.266},"min":{"1min":0.78000000000000003,"5min":0.59799999999999998,"15min":0.53500000000000003},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":43.655999999999999},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.0629999999999999,"15min":1.2330000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.48999999999999999},"max":{"1min":1.4350000000000001,"5min":1.6339999999999999,"15min":45.255000000000003},"last":1.4350000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1619999999999999,"15min":1.1850000000000001},"min":{"1min":0.90500000000000003,"5min":0.78700000000000003,"15min":0.59699999999999998},"max":{"1min":2.141,"5min":2.141,"15min":6.0179999999999998},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.107,"15min":1.171},"min":{"1min":0.93600000000000005,"5min":0.66600000000000004,"15min":0.56499999999999995},"max":{"1min":1.5720000000000001,"5min":1.645,"15min":6.851},"last":1.3540000000000001}]}]},{"osd":1,"up_from":855,"seq":3672197038134,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304968,"kb_used_data":1728,"kb_used_omap":89,"kb_used_meta":303142,"kb_avail":93440184,"statfs":{"total":95995035648,"available":95682748416,"internally_reserved":0,"allocated":1769472,"data_stored":515497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":91933,"internal_metadata":310417635},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.70199999999999996,"15min":0.68600000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94399999999999995,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.80100000000000005,"15min":0.77700000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.202,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.88700000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77600000000000002,"15min":0.77400000000000002},"min":{"1min":0.53200000000000003,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.1259999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.753,"15min":0.751},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.121,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.80500000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.878,"15min":0.878},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.2050000000000001,"5min":1.405,"15min":1.405},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.85599999999999998,"15min":0.86399999999999999},"min":{"1min":0.55500000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.0289999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.59399999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0720000000000001,"15min":1.099},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.294,"5min":1.48,"15min":1.48},"last":1.01},{"interface":"front","average":{"1min":1.03,"5min":1.093,"15min":1.097},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3819999999999999,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.337,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.083,"15min":1.1220000000000001},"min":{"1min":0.79900000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.2609999999999999,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.169,"15min":1.1830000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.349,"5min":1.833,"15min":1.833},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.175,"15min":1.1830000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.4970000000000001,"5min":1.845,"15min":1.845},"last":1.147}]},{"osd":7,"last update":"Tue Sep 10 19:13:23 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.196,"15min":1.242},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.3129999999999999,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.048},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.234,"15min":1.262},"min":{"1min":0.79600000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.407,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.1679999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738747,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303240,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93441912,"statfs":{"total":95995035648,"available":95684517888,"internally_reserved":0,"allocated":1769472,"data_stored":515081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.69199999999999995,"15min":0.72799999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1579999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.80900000000000005,"15min":0.81000000000000005},"min":{"1min":0.51500000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.226,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.82799999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":0.77700000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.36799999999999999},"max":{"1min":1.25,"5min":1.802,"15min":3.427},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.68700000000000006,"15min":0.77900000000000003},"min":{"1min":0.37,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.04,"5min":1.125,"15min":3.2090000000000001},"last":0.88200000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:13:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.78300000000000003,"15min":0.83699999999999997},"min":{"1min":0.438,"5min":0.317,"15min":0.317},"max":{"1min":1.3260000000000001,"5min":1.4470000000000001,"15min":4.4749999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.86099999999999999,"15min":0.874},"min":{"1min":0.54900000000000004,"5min":0.45400000000000001,"15min":0.42499999999999999},"max":{"1min":1.1439999999999999,"5min":1.8819999999999999,"15min":2.4569999999999999},"last":0.98899999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:12:50 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.91600000000000004,"15min":1.036},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.48199999999999998},"max":{"1min":1.198,"5min":1.4590000000000001,"15min":3.3279999999999998},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0720000000000001,"15min":1.1299999999999999},"min":{"1min":0.82699999999999996,"5min":0.66200000000000003,"15min":0.61399999999999999},"max":{"1min":1.304,"5min":1.7070000000000001,"15min":3.419},"last":1.143}]},{"osd":5,"last update":"Tue Sep 10 19:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.0800000000000001,"15min":1.0900000000000001},"min":{"1min":0.83099999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.6299999999999999,"5min":1.8380000000000001,"15min":2.5880000000000001},"last":1.173},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.103,"15min":1.137},"min":{"1min":0.69799999999999995,"5min":0.629,"15min":0.62},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.94899999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:13:31 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.151,"15min":1.181},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.52400000000000002},"max":{"1min":1.8660000000000001,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.173,"15min":1.173},"min":{"1min":0.80500000000000005,"5min":0.72699999999999998,"15min":0.48999999999999999},"max":{"1min":1.577,"5min":1.9850000000000001,"15min":2.5339999999999998},"last":1.0960000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:13:19 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.079,"15min":1.165},"min":{"1min":0.83599999999999997,"5min":0.63500000000000001,"15min":0.61599999999999999},"max":{"1min":1.466,"5min":1.5129999999999999,"15min":2.468},"last":1.03},{"interface":"front","average":{"1min":1.22,"5min":1.214,"15min":1.244},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.56000000000000005},"max":{"1min":2.0550000000000002,"5min":2.0550000000000002,"15min":2.0550000000000002},"last":1.2250000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:13:42.611 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:13:42.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-10T19:13:42.868 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:42] "GET /metrics HTTP/1.1" 200 224114 "" "Prometheus/2.43.0" 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.275693+0000 osd.6 (osd.6) 549 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[18629]: cluster 2024-09-10T19:13:41.279726+0000 osd.6 (osd.6) 550 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[18629]: audit 2024-09-10T19:13:41.649187+0000 mgr.y (mgr.24526) 940 : audit [DBG] from='client.50141 v1:172.21.15.83:0/286588446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.275693+0000 osd.6 (osd.6) 549 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[32381]: cluster 2024-09-10T19:13:41.279726+0000 osd.6 (osd.6) 550 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:13:43.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:42 smithi083 bash[32381]: audit 2024-09-10T19:13:41.649187+0000 mgr.y (mgr.24526) 940 : audit [DBG] from='client.50141 v1:172.21.15.83:0/286588446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:42 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.275693+0000 osd.6 (osd.6) 549 : cluster [DBG] 2.1a scrub starts 2024-09-10T19:13:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:42 smithi165 bash[22591]: cluster 2024-09-10T19:13:41.279726+0000 osd.6 (osd.6) 550 : cluster [DBG] 2.1a scrub ok 2024-09-10T19:13:43.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:42 smithi165 bash[22591]: audit 2024-09-10T19:13:41.649187+0000 mgr.y (mgr.24526) 940 : audit [DBG] from='client.50141 v1:172.21.15.83:0/286588446' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[32381]: cluster 2024-09-10T19:13:42.582118+0000 mgr.y (mgr.24526) 941 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[32381]: cluster 2024-09-10T19:13:42.742533+0000 osd.5 (osd.5) 596 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[32381]: cluster 2024-09-10T19:13:42.745023+0000 osd.5 (osd.5) 597 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[18629]: cluster 2024-09-10T19:13:42.582118+0000 mgr.y (mgr.24526) 941 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[18629]: cluster 2024-09-10T19:13:42.742533+0000 osd.5 (osd.5) 596 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:13:44.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:43 smithi083 bash[18629]: cluster 2024-09-10T19:13:42.745023+0000 osd.5 (osd.5) 597 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:13:44.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:43 smithi165 bash[22591]: cluster 2024-09-10T19:13:42.582118+0000 mgr.y (mgr.24526) 941 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:44.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:43 smithi165 bash[22591]: cluster 2024-09-10T19:13:42.742533+0000 osd.5 (osd.5) 596 : cluster [DBG] 4.12 scrub starts 2024-09-10T19:13:44.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:43 smithi165 bash[22591]: cluster 2024-09-10T19:13:42.745023+0000 osd.5 (osd.5) 597 : cluster [DBG] 4.12 scrub ok 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:43.129283+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:43.133090+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:43.151983+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:43.152366+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:43.875996+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-09-10T19:13:45.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:44.115557+0000 osd.0 (osd.0) 597 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: cluster 2024-09-10T19:13:44.119114+0000 osd.0 (osd.0) 598 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[18629]: audit 2024-09-10T19:13:44.585449+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:43.129283+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:43.133090+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:43.151983+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:43.152366+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:43.875996+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:44.115557+0000 osd.0 (osd.0) 597 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: cluster 2024-09-10T19:13:44.119114+0000 osd.0 (osd.0) 598 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:13:45.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:44 smithi083 bash[32381]: audit 2024-09-10T19:13:44.585449+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:43.129283+0000 osd.1 (osd.1) 159 : cluster [DBG] 2.4 scrub starts 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:43.133090+0000 osd.1 (osd.1) 160 : cluster [DBG] 2.4 scrub ok 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:43.151983+0000 osd.1 (osd.1) 161 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:43.152366+0000 osd.1 (osd.1) 162 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:43.875996+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:44.115557+0000 osd.0 (osd.0) 597 : cluster [DBG] 12.10 scrub starts 2024-09-10T19:13:45.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: cluster 2024-09-10T19:13:44.119114+0000 osd.0 (osd.0) 598 : cluster [DBG] 12.10 scrub ok 2024-09-10T19:13:45.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:44 smithi165 bash[22591]: audit 2024-09-10T19:13:44.585449+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[32381]: cluster 2024-09-10T19:13:44.582878+0000 mgr.y (mgr.24526) 942 : cluster [DBG] pgmap v1520: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[32381]: cluster 2024-09-10T19:13:44.739608+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[32381]: cluster 2024-09-10T19:13:44.743251+0000 osd.5 (osd.5) 599 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[32381]: cluster 2024-09-10T19:13:45.258898+0000 osd.4 (osd.4) 576 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[32381]: cluster 2024-09-10T19:13:45.263721+0000 osd.4 (osd.4) 577 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:13:46.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[18629]: cluster 2024-09-10T19:13:44.582878+0000 mgr.y (mgr.24526) 942 : cluster [DBG] pgmap v1520: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:46.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[18629]: cluster 2024-09-10T19:13:44.739608+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:13:46.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[18629]: cluster 2024-09-10T19:13:44.743251+0000 osd.5 (osd.5) 599 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:13:46.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[18629]: cluster 2024-09-10T19:13:45.258898+0000 osd.4 (osd.4) 576 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:13:46.292 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:45 smithi083 bash[18629]: cluster 2024-09-10T19:13:45.263721+0000 osd.4 (osd.4) 577 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:13:46.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:45 smithi165 bash[22591]: cluster 2024-09-10T19:13:44.582878+0000 mgr.y (mgr.24526) 942 : cluster [DBG] pgmap v1520: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:46.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:45 smithi165 bash[22591]: cluster 2024-09-10T19:13:44.739608+0000 osd.5 (osd.5) 598 : cluster [DBG] 2.3 scrub starts 2024-09-10T19:13:46.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:45 smithi165 bash[22591]: cluster 2024-09-10T19:13:44.743251+0000 osd.5 (osd.5) 599 : cluster [DBG] 2.3 scrub ok 2024-09-10T19:13:46.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:45 smithi165 bash[22591]: cluster 2024-09-10T19:13:45.258898+0000 osd.4 (osd.4) 576 : cluster [DBG] 3.1a scrub starts 2024-09-10T19:13:46.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:45 smithi165 bash[22591]: cluster 2024-09-10T19:13:45.263721+0000 osd.4 (osd.4) 577 : cluster [DBG] 3.1a scrub ok 2024-09-10T19:13:47.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:46 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.131315+0000 osd.7 (osd.7) 604 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:13:47.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:46 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.134602+0000 osd.7 (osd.7) 605 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:13:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:46 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.131315+0000 osd.7 (osd.7) 604 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:13:47.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:46 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.134602+0000 osd.7 (osd.7) 605 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:13:47.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:46 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.131315+0000 osd.7 (osd.7) 604 : cluster [DBG] 6.12 scrub starts 2024-09-10T19:13:47.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:46 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.134602+0000 osd.7 (osd.7) 605 : cluster [DBG] 6.12 scrub ok 2024-09-10T19:13:47.391 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.584742+0000 mgr.y (mgr.24526) 943 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.696963+0000 osd.5 (osd.5) 600 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.702206+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.973617+0000 osd.3 (osd.3) 628 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:46.976913+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:47.220177+0000 osd.4 (osd.4) 578 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:13:48.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[18629]: cluster 2024-09-10T19:13:47.224811+0000 osd.4 (osd.4) 579 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.584742+0000 mgr.y (mgr.24526) 943 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.696963+0000 osd.5 (osd.5) 600 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.702206+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.973617+0000 osd.3 (osd.3) 628 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:46.976913+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:47.220177+0000 osd.4 (osd.4) 578 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:13:48.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:47 smithi083 bash[32381]: cluster 2024-09-10T19:13:47.224811+0000 osd.4 (osd.4) 579 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.584742+0000 mgr.y (mgr.24526) 943 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.696963+0000 osd.5 (osd.5) 600 : cluster [DBG] 3.16 deep-scrub starts 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.702206+0000 osd.5 (osd.5) 601 : cluster [DBG] 3.16 deep-scrub ok 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.973617+0000 osd.3 (osd.3) 628 : cluster [DBG] 4.19 scrub starts 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:46.976913+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.19 scrub ok 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:47.220177+0000 osd.4 (osd.4) 578 : cluster [DBG] 4.9 scrub starts 2024-09-10T19:13:48.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:47 smithi165 bash[22591]: cluster 2024-09-10T19:13:47.224811+0000 osd.4 (osd.4) 579 : cluster [DBG] 4.9 scrub ok 2024-09-10T19:13:49.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[54535]: debug there is no tcmu-runner data available 2024-09-10T19:13:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[22591]: cluster 2024-09-10T19:13:47.457446+0000 osd.7 (osd.7) 606 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[22591]: cluster 2024-09-10T19:13:47.457883+0000 osd.7 (osd.7) 607 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[22591]: cluster 2024-09-10T19:13:47.912316+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-09-10T19:13:49.112 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[22591]: cluster 2024-09-10T19:13:48.267875+0000 osd.4 (osd.4) 580 : cluster [DBG] 2.f scrub starts 2024-09-10T19:13:49.113 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:48 smithi165 bash[22591]: cluster 2024-09-10T19:13:48.272401+0000 osd.4 (osd.4) 581 : cluster [DBG] 2.f scrub ok 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[18629]: cluster 2024-09-10T19:13:47.457446+0000 osd.7 (osd.7) 606 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[18629]: cluster 2024-09-10T19:13:47.457883+0000 osd.7 (osd.7) 607 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[18629]: cluster 2024-09-10T19:13:47.912316+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[18629]: cluster 2024-09-10T19:13:48.267875+0000 osd.4 (osd.4) 580 : cluster [DBG] 2.f scrub starts 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[18629]: cluster 2024-09-10T19:13:48.272401+0000 osd.4 (osd.4) 581 : cluster [DBG] 2.f scrub ok 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[32381]: cluster 2024-09-10T19:13:47.457446+0000 osd.7 (osd.7) 606 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:49.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[32381]: cluster 2024-09-10T19:13:47.457883+0000 osd.7 (osd.7) 607 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:49.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[32381]: cluster 2024-09-10T19:13:47.912316+0000 mon.a (mon.0) 3898 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-09-10T19:13:49.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[32381]: cluster 2024-09-10T19:13:48.267875+0000 osd.4 (osd.4) 580 : cluster [DBG] 2.f scrub starts 2024-09-10T19:13:49.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:48 smithi083 bash[32381]: cluster 2024-09-10T19:13:48.272401+0000 osd.4 (osd.4) 581 : cluster [DBG] 2.f scrub ok 2024-09-10T19:13:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:49 smithi083 bash[18629]: cluster 2024-09-10T19:13:48.585514+0000 mgr.y (mgr.24526) 944 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:50.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:49 smithi083 bash[18629]: audit 2024-09-10T19:13:48.758848+0000 mgr.y (mgr.24526) 945 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:49 smithi083 bash[32381]: cluster 2024-09-10T19:13:48.585514+0000 mgr.y (mgr.24526) 944 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:50.291 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:49 smithi083 bash[32381]: audit 2024-09-10T19:13:48.758848+0000 mgr.y (mgr.24526) 945 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:49 smithi165 bash[22591]: cluster 2024-09-10T19:13:48.585514+0000 mgr.y (mgr.24526) 944 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:50.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:49 smithi165 bash[22591]: audit 2024-09-10T19:13:48.758848+0000 mgr.y (mgr.24526) 945 : audit [DBG] from='client.14574 v1:172.21.15.165:0/2322478610' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:51.247 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[18629]: cluster 2024-09-10T19:13:49.794330+0000 osd.5 (osd.5) 602 : cluster [DBG] 6.a scrub starts 2024-09-10T19:13:51.247 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[18629]: cluster 2024-09-10T19:13:49.797630+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.a scrub ok 2024-09-10T19:13:51.247 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[18629]: cluster 2024-09-10T19:13:49.824612+0000 osd.6 (osd.6) 551 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:51.247 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[18629]: cluster 2024-09-10T19:13:49.824915+0000 osd.6 (osd.6) 552 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[18629]: cluster 2024-09-10T19:13:49.930235+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[32381]: cluster 2024-09-10T19:13:49.794330+0000 osd.5 (osd.5) 602 : cluster [DBG] 6.a scrub starts 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[32381]: cluster 2024-09-10T19:13:49.797630+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.a scrub ok 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[32381]: cluster 2024-09-10T19:13:49.824612+0000 osd.6 (osd.6) 551 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[32381]: cluster 2024-09-10T19:13:49.824915+0000 osd.6 (osd.6) 552 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:51.248 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:50 smithi083 bash[32381]: cluster 2024-09-10T19:13:49.930235+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-09-10T19:13:51.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:50 smithi165 bash[22591]: cluster 2024-09-10T19:13:49.794330+0000 osd.5 (osd.5) 602 : cluster [DBG] 6.a scrub starts 2024-09-10T19:13:51.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:50 smithi165 bash[22591]: cluster 2024-09-10T19:13:49.797630+0000 osd.5 (osd.5) 603 : cluster [DBG] 6.a scrub ok 2024-09-10T19:13:51.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:50 smithi165 bash[22591]: cluster 2024-09-10T19:13:49.824612+0000 osd.6 (osd.6) 551 : cluster [DBG] purged_snaps scrub starts 2024-09-10T19:13:51.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:50 smithi165 bash[22591]: cluster 2024-09-10T19:13:49.824915+0000 osd.6 (osd.6) 552 : cluster [DBG] purged_snaps scrub ok 2024-09-10T19:13:51.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:50 smithi165 bash[22591]: cluster 2024-09-10T19:13:49.930235+0000 mon.a (mon.0) 3899 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-09-10T19:13:51.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.407+0000 7fc6b1b63640 1 -- 172.21.15.83:0/1536105155 >> v1:172.21.15.83:6789/0 conn(0x7fc6ac1014c0 legacy=0x7fc6ac101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:51.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.407+0000 7fc6b1b63640 1 -- 172.21.15.83:0/1536105155 shutdown_connections 2024-09-10T19:13:51.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.407+0000 7fc6b1b63640 1 -- 172.21.15.83:0/1536105155 >> 172.21.15.83:0/1536105155 conn(0x7fc6ac0f70a0 msgr2=0x7fc6ac0f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:51.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.407+0000 7fc6b1b63640 1 -- 172.21.15.83:0/1536105155 shutdown_connections 2024-09-10T19:13:51.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.407+0000 7fc6b1b63640 1 -- 172.21.15.83:0/1536105155 wait complete. 2024-09-10T19:13:51.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 librados: starting msgr at 2024-09-10T19:13:51.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 librados: starting objecter 2024-09-10T19:13:51.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 Processor -- start 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- start start 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 librados: setting wanted keys 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 librados: calling monclient init 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6ac11e370 con 0x7fc6ac104280 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6ac11f570 con 0x7fc6ac1014c0 2024-09-10T19:13:51.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- --> v1:172.21.15.83:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6ac120770 con 0x7fc6ac107e10 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6abfff640 1 --1- >> v1:172.21.15.83:6790/0 conn(0x7fc6ac107e10 0x7fc6ac11ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6790/0 says I am v1:172.21.15.83:35286/0 (socket says 172.21.15.83:35286) 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a3fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fc6ac104280 0x7fc6ac0664e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:35692/0 (socket says 172.21.15.83:35692) 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6abfff640 1 -- 172.21.15.83:0/2652457182 learned_addr learned my addr 172.21.15.83:0/2652457182 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6ac107e10 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6ac104280 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6ac1014c0 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T19:13:51.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556897172 0 0) 0x7fc6ac120770 con 0x7fc6ac107e10 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc698003720 con 0x7fc6ac107e10 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303337580 0 0) 0x7fc6ac11e370 con 0x7fc6ac104280 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ac120770 con 0x7fc6ac104280 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544694172 0 0) 0x7fc6ac11f570 con 0x7fc6ac1014c0 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ac11e370 con 0x7fc6ac1014c0 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359748599 0 0) 0x7fc698003720 con 0x7fc6ac107e10 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6ac11f570 con 0x7fc6ac107e10 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6a4001c80 con 0x7fc6ac107e10 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.0 v1:172.21.15.83:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3511713799 0 0) 0x7fc6ac120770 con 0x7fc6ac104280 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc698003720 con 0x7fc6ac104280 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.0 v1:172.21.15.83:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6940018c0 con 0x7fc6ac104280 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2316540104 0 0) 0x7fc6ac11e370 con 0x7fc6ac1014c0 2024-09-10T19:13:51.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.165:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6ac120770 con 0x7fc6ac1014c0 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1588194873 0 0) 0x7fc6ac11f570 con 0x7fc6ac107e10 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 >> v1:172.21.15.165:6789/0 conn(0x7fc6ac1014c0 legacy=0x7fc6ac065dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 >> v1:172.21.15.83:6789/0 conn(0x7fc6ac104280 legacy=0x7fc6ac0664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac121970 con 0x7fc6ac107e10 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac11f760 con 0x7fc6ac107e10 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 10 client.50147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac11fbb0 con 0x7fc6ac107e10 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6b1b63640 1 librados: init done 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 5 ==== config(24 keys) v1 ==== 1007+0+0 (unknown 2292141906 0 0) 0x7fc6a4002010 con 0x7fc6ac107e10 2024-09-10T19:13:51.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.411+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2131304546 0 0) 0x7fc6a4005ee0 con 0x7fc6ac107e10 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c005010 con 0x7fc6ac107e10 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 7 ==== mgrmap(e 21) v1 ==== 90802+0+0 (unknown 4023959538 0 0) 0x7fc6a401c300 con 0x7fc6ac107e10 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 8 ==== osd_map(904..904 src has 849..904) v4 ==== 7363+0+0 (unknown 1043877416 0 0) 0x7fc6a408a5e0 con 0x7fc6ac107e10 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 10 client.50147.objecter ms_dispatch 0x7fc6ac000b90 osd_map(904..904 src has 849..904) v4 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 3 client.50147.objecter handle_osd_map got epochs [904,904] > 0 2024-09-10T19:13:51.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 3 client.50147.objecter handle_osd_map decoding full epoch 904 2024-09-10T19:13:51.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 20 client.50147.objecter dump_active .. 0 homeless 2024-09-10T19:13:51.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.415+0000 7fc6a8ff9640 10 client.50147.objecter ms_handle_connect 0x7fc698071700 2024-09-10T19:13:51.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.423+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (unknown 1092875540 0 301564637) 0x7fc6a4056e20 con 0x7fc6ac107e10 2024-09-10T19:13:51.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.599+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 --> v1:172.21.15.83:6790/0 -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7fc68c005840 con 0x7fc6ac107e10 2024-09-10T19:13:51.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.607+0000 7fc6a8ff9640 1 -- 172.21.15.83:0/2652457182 <== mon.2 v1:172.21.15.83:6790/0 10 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v30) v1 ==== 118+0+0 (unknown 1095616568 0 0) 0x7fc6a40567c0 con 0x7fc6ac107e10 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 10 librados: watch_flush enter 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 10 librados: watch_flush exit 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 20 client.50147.objecter shutdown clearing up homeless session... 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 10 client.50147.objecter successfully canceled tick 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 >> v1:172.21.15.83:6800/3071793735 conn(0x7fc698071700 legacy=0x7fc698073bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 >> v1:172.21.15.83:6790/0 conn(0x7fc6ac107e10 legacy=0x7fc6ac11ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 shutdown_connections 2024-09-10T19:13:51.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 >> 172.21.15.83:0/2652457182 conn(0x7fc6ac0f70a0 msgr2=0x7fc6ac108270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:13:51.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 shutdown_connections 2024-09-10T19:13:51.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 -- 172.21.15.83:0/2652457182 wait complete. 2024-09-10T19:13:51.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-10T19:13:51.611+0000 7fc6b1b63640 1 librados: shutdown 2024-09-10T19:13:51.862 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[23586]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:51.939 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE Bus STOPPING 2024-09-10T19:13:51.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:13:51.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE Bus STOPPED 2024-09-10T19:13:51.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE Bus STARTING 2024-09-10T19:13:51.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE Serving on http://:::9283 2024-09-10T19:13:51.940 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18896]: [10/Sep/2024:19:13:51] ENGINE Bus STARTED 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: cluster 2024-09-10T19:13:50.586295+0000 mgr.y (mgr.24526) 946 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.603739+0000 mon.c (mon.2) 936 : audit [INF] from='client.? v1:172.21.15.83:0/2652457182' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.604391+0000 mon.a (mon.0) 3900 : audit [INF] from='client.50147 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.612735+0000 mon.a (mon.0) 3901 : audit [INF] from='client.50147 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.628650+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.630735+0000 mon.c (mon.2) 938 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.631840+0000 mon.c (mon.2) 939 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:52.291 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[18629]: audit 2024-09-10T19:13:51.640504+0000 mon.a (mon.0) 3902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: cluster 2024-09-10T19:13:50.586295+0000 mgr.y (mgr.24526) 946 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.603739+0000 mon.c (mon.2) 936 : audit [INF] from='client.? v1:172.21.15.83:0/2652457182' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.604391+0000 mon.a (mon.0) 3900 : audit [INF] from='client.50147 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.612735+0000 mon.a (mon.0) 3901 : audit [INF] from='client.50147 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.628650+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.630735+0000 mon.c (mon.2) 938 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.631840+0000 mon.c (mon.2) 939 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:52.292 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:51 smithi083 bash[32381]: audit 2024-09-10T19:13:51.640504+0000 mon.a (mon.0) 3902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: cluster 2024-09-10T19:13:50.586295+0000 mgr.y (mgr.24526) 946 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 762 B/s rd, 0 op/s 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.603739+0000 mon.c (mon.2) 936 : audit [INF] from='client.? v1:172.21.15.83:0/2652457182' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.604391+0000 mon.a (mon.0) 3900 : audit [INF] from='client.50147 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.612735+0000 mon.a (mon.0) 3901 : audit [INF] from='client.50147 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.628650+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.630735+0000 mon.c (mon.2) 938 : audit [DBG] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:52.362 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.631840+0000 mon.c (mon.2) 939 : audit [INF] from='mgr.24526 v1:172.21.15.83:0/654739710' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:52.363 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:51 smithi165 bash[22591]: audit 2024-09-10T19:13:51.640504+0000 mon.a (mon.0) 3902 : audit [INF] from='mgr.24526 ' entity='mgr.y' 2024-09-10T19:13:52.512 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-10T19:13:52.521 INFO:tasks.cephadm:Teardown begin 2024-09-10T19:13:52.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:13:52.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:13:52.555 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-10T19:13:52.555 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:13:52.589 DEBUG:teuthology.orchestra.run.smithi165:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:13:52.595 INFO:tasks.cephadm:Stopping all daemons... 2024-09-10T19:13:52.596 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-10T19:13:52.596 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a 2024-09-10T19:13:52.776 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:52 smithi083 systemd[1]: Stopping Ceph mon.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:52.776 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:52 smithi083 bash[18896]: ::ffff:172.21.15.165 - - [10/Sep/2024:19:13:52] "GET /metrics HTTP/1.1" 200 224091 "" "Prometheus/2.43.0" 2024-09-10T19:13:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:52 smithi083 bash[18629]: debug 2024-09-10T19:13:52.771+0000 7facaae75640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-10T19:13:53.041 INFO:journalctl@ceph.mon.a.smithi083.stdout:Sep 10 19:13:52 smithi083 bash[18629]: debug 2024-09-10T19:13:52.771+0000 7facaae75640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-10T19:13:53.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.a.service' 2024-09-10T19:13:53.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:53.782 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-10T19:13:53.782 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-10T19:13:53.782 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.c 2024-09-10T19:13:54.194 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE Bus STOPPING 2024-09-10T19:13:54.194 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:53 smithi083 systemd[1]: Stopping Ceph mon.c for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:54.195 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[32381]: debug 2024-09-10T19:13:54.007+0000 7f143c646640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-10T19:13:54.195 INFO:journalctl@ceph.mon.c.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[32381]: debug 2024-09-10T19:13:54.007+0000 7f143c646640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:13:54.572 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:13:54.572 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE Bus STOPPED 2024-09-10T19:13:54.572 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE Bus STARTING 2024-09-10T19:13:54.572 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE Serving on http://:::9283 2024-09-10T19:13:54.573 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:54 smithi083 bash[18896]: [10/Sep/2024:19:13:54] ENGINE Bus STARTED 2024-09-10T19:13:55.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.c.service' 2024-09-10T19:13:55.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:55.152 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-10T19:13:55.152 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-10T19:13:55.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.b 2024-09-10T19:13:55.420 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:55 smithi165 systemd[1]: Stopping Ceph mon.b for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:55.688 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:55 smithi165 bash[22591]: debug 2024-09-10T19:13:55.412+0000 7f6f0b09e640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-09-10T19:13:55.688 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 10 19:13:55 smithi165 bash[22591]: debug 2024-09-10T19:13:55.412+0000 7f6f0b09e640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:13:56.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mon.b.service' 2024-09-10T19:13:56.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:56.489 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-10T19:13:56.489 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-10T19:13:56.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y 2024-09-10T19:13:56.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:56 smithi083 systemd[1]: Stopping Ceph mgr.y for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:56.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:56 smithi083 bash[18896]: debug 2024-09-10T19:13:56.659+0000 7f5222968640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:13:56.791 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Sep 10 19:13:56 smithi083 bash[18896]: debug 2024-09-10T19:13:56.659+0000 7f5222968640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-10T19:13:57.615 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-10T19:13:57.650 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:57.650 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-10T19:13:57.650 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-10T19:13:57.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.x 2024-09-10T19:13:58.004 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 10 19:13:57 smithi165 systemd[1]: Stopping Ceph mgr.x for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:58.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-10T19:13:58.554 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:58.554 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-10T19:13:58.554 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-10T19:13:58.554 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.0 2024-09-10T19:13:59.041 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:13:58 smithi083 systemd[1]: Stopping Ceph osd.0 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:13:59.041 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:13:58 smithi083 bash[35790]: debug 2024-09-10T19:13:58.731+0000 7f0eb0827640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:13:59.041 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:13:58 smithi083 bash[35790]: debug 2024-09-10T19:13:58.731+0000 7f0eb0827640 -1 osd.0 904 *** Got signal Terminated *** 2024-09-10T19:13:59.042 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:13:58 smithi083 bash[35790]: debug 2024-09-10T19:13:58.731+0000 7f0eb0827640 -1 osd.0 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:05.279 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:14:04 smithi083 bash[222561]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-0 2024-09-10T19:14:05.279 INFO:journalctl@ceph.osd.0.smithi083.stdout:Sep 10 19:14:05 smithi083 bash[222633]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-0 2024-09-10T19:14:07.452 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.0.service' 2024-09-10T19:14:07.491 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:07.491 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-10T19:14:07.491 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:14:07.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:14:07.711 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:07 smithi083 systemd[1]: Stopping Ceph osd.1 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:08.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:07 smithi083 bash[212374]: debug 2024-09-10T19:14:07.703+0000 7f1b45265640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:08.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:07 smithi083 bash[212374]: debug 2024-09-10T19:14:07.703+0000 7f1b45265640 -1 osd.1 904 *** Got signal Terminated *** 2024-09-10T19:14:08.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:07 smithi083 bash[212374]: debug 2024-09-10T19:14:07.703+0000 7f1b45265640 -1 osd.1 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:14.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:13 smithi083 bash[222774]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:14:14.041 INFO:journalctl@ceph.osd.1.smithi083.stdout:Sep 10 19:14:13 smithi083 bash[222850]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:14:16.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:14:16.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:16.265 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:14:16.265 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-10T19:14:16.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.2 2024-09-10T19:14:16.541 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:16 smithi083 systemd[1]: Stopping Ceph osd.2 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:16.541 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:16 smithi083 bash[50483]: debug 2024-09-10T19:14:16.415+0000 7f6658644640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:16.541 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:16 smithi083 bash[50483]: debug 2024-09-10T19:14:16.415+0000 7f6658644640 -1 osd.2 904 *** Got signal Terminated *** 2024-09-10T19:14:16.541 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:16 smithi083 bash[50483]: debug 2024-09-10T19:14:16.415+0000 7f6658644640 -1 osd.2 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:22.947 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:22 smithi083 bash[223000]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-2 2024-09-10T19:14:22.947 INFO:journalctl@ceph.osd.2.smithi083.stdout:Sep 10 19:14:22 smithi083 bash[223075]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-2 2024-09-10T19:14:24.922 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.2.service' 2024-09-10T19:14:24.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:24.956 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-10T19:14:24.956 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-10T19:14:24.957 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.3 2024-09-10T19:14:25.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:25 smithi083 systemd[1]: Stopping Ceph osd.3 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:25.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:25 smithi083 bash[58218]: debug 2024-09-10T19:14:25.150+0000 7f0c5f2cd640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:25.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:25 smithi083 bash[58218]: debug 2024-09-10T19:14:25.150+0000 7f0c5f2cd640 -1 osd.3 904 *** Got signal Terminated *** 2024-09-10T19:14:25.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:25 smithi083 bash[58218]: debug 2024-09-10T19:14:25.150+0000 7f0c5f2cd640 -1 osd.3 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:26.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:26 smithi165 bash[34072]: debug 2024-09-10T19:14:26.460+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:27.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:27 smithi165 bash[41096]: debug 2024-09-10T19:14:27.224+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:27.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:27 smithi165 bash[48829]: debug 2024-09-10T19:14:27.068+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:27.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:27 smithi165 bash[26469]: debug 2024-09-10T19:14:27.528+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:27.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:27 smithi165 bash[34072]: debug 2024-09-10T19:14:27.440+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:28.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:27 smithi083 bash[58218]: debug 2024-09-10T19:14:27.978+0000 7f0c5b0e4640 -1 osd.3 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:03.530261+0000 front 2024-09-10T19:14:03.530535+0000 (oldest deadline 2024-09-10T19:14:27.030412+0000) 2024-09-10T19:14:28.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:28 smithi165 bash[41096]: debug 2024-09-10T19:14:28.236+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:28.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:28 smithi165 bash[48829]: debug 2024-09-10T19:14:28.104+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:28.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:28 smithi165 bash[26469]: debug 2024-09-10T19:14:28.568+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:28.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:28 smithi165 bash[34072]: debug 2024-09-10T19:14:28.476+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:29.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:28 smithi083 bash[58218]: debug 2024-09-10T19:14:28.974+0000 7f0c5b0e4640 -1 osd.3 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:03.530261+0000 front 2024-09-10T19:14:03.530535+0000 (oldest deadline 2024-09-10T19:14:27.030412+0000) 2024-09-10T19:14:29.472 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:29 smithi165 bash[48829]: debug 2024-09-10T19:14:29.136+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:29.472 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:29 smithi165 bash[41096]: debug 2024-09-10T19:14:29.216+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:29.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:29 smithi165 bash[26469]: debug 2024-09-10T19:14:29.616+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:29.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:29 smithi165 bash[34072]: debug 2024-09-10T19:14:29.468+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:30.291 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:29 smithi083 bash[58218]: debug 2024-09-10T19:14:29.966+0000 7f0c5b0e4640 -1 osd.3 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:03.530261+0000 front 2024-09-10T19:14:03.530535+0000 (oldest deadline 2024-09-10T19:14:27.030412+0000) 2024-09-10T19:14:30.454 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:30 smithi165 bash[48829]: debug 2024-09-10T19:14:30.128+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:30.455 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:30 smithi165 bash[41096]: debug 2024-09-10T19:14:30.236+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:30.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:30 smithi165 bash[26469]: debug 2024-09-10T19:14:30.612+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:30.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:30 smithi165 bash[34072]: debug 2024-09-10T19:14:30.448+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:31.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:31 smithi165 bash[41096]: debug 2024-09-10T19:14:31.192+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:31.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:31 smithi165 bash[48829]: debug 2024-09-10T19:14:31.088+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:31.541 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:31 smithi083 bash[223213]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-3 2024-09-10T19:14:31.541 INFO:journalctl@ceph.osd.3.smithi083.stdout:Sep 10 19:14:31 smithi083 bash[223286]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-3 2024-09-10T19:14:31.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:31 smithi165 bash[26469]: debug 2024-09-10T19:14:31.580+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:31.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:31 smithi165 bash[34072]: debug 2024-09-10T19:14:31.472+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:32.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:32 smithi165 bash[41096]: debug 2024-09-10T19:14:32.204+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:32.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:32 smithi165 bash[48829]: debug 2024-09-10T19:14:32.104+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:32.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:32 smithi165 bash[26469]: debug 2024-09-10T19:14:32.608+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:32.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:32 smithi165 bash[34072]: debug 2024-09-10T19:14:32.484+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:33.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[41096]: debug 2024-09-10T19:14:33.232+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:33.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[48829]: debug 2024-09-10T19:14:33.076+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:33.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.3.service' 2024-09-10T19:14:33.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:33.553 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-10T19:14:33.553 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-10T19:14:33.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.4 2024-09-10T19:14:33.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[34072]: debug 2024-09-10T19:14:33.524+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:33.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[34072]: debug 2024-09-10T19:14:33.524+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:33.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[26469]: debug 2024-09-10T19:14:33.564+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:33.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[26469]: debug 2024-09-10T19:14:33.564+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:33.863 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 systemd[1]: Stopping Ceph osd.4 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:33.863 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[26469]: debug 2024-09-10T19:14:33.708+0000 7f37e0835640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:33.863 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[26469]: debug 2024-09-10T19:14:33.708+0000 7f37e0835640 -1 osd.4 904 *** Got signal Terminated *** 2024-09-10T19:14:33.863 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:33 smithi165 bash[26469]: debug 2024-09-10T19:14:33.708+0000 7f37e0835640 -1 osd.4 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:34.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[48829]: debug 2024-09-10T19:14:34.076+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:34.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[41096]: debug 2024-09-10T19:14:34.252+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:34.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[26469]: debug 2024-09-10T19:14:34.520+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:34.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[26469]: debug 2024-09-10T19:14:34.520+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:34.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[34072]: debug 2024-09-10T19:14:34.556+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:34.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:34 smithi165 bash[34072]: debug 2024-09-10T19:14:34.556+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:35.478 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[48829]: debug 2024-09-10T19:14:35.120+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:35.478 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[41096]: debug 2024-09-10T19:14:35.284+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:35.479 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[41096]: debug 2024-09-10T19:14:35.284+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:35.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[26469]: debug 2024-09-10T19:14:35.472+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:35.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[26469]: debug 2024-09-10T19:14:35.472+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:35.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[34072]: debug 2024-09-10T19:14:35.552+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:35.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:35 smithi165 bash[34072]: debug 2024-09-10T19:14:35.552+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:36.432 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[48829]: debug 2024-09-10T19:14:36.128+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:36.432 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[48829]: debug 2024-09-10T19:14:36.128+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:36.432 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[41096]: debug 2024-09-10T19:14:36.264+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:36.432 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[41096]: debug 2024-09-10T19:14:36.264+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:36.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[26469]: debug 2024-09-10T19:14:36.424+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:36.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[26469]: debug 2024-09-10T19:14:36.424+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:36.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[34072]: debug 2024-09-10T19:14:36.596+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:36.863 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:36 smithi165 bash[34072]: debug 2024-09-10T19:14:36.596+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:37.463 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[48829]: debug 2024-09-10T19:14:37.124+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:37.463 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[48829]: debug 2024-09-10T19:14:37.124+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:37.463 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[41096]: debug 2024-09-10T19:14:37.252+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:37.463 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[41096]: debug 2024-09-10T19:14:37.252+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:37.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[26469]: debug 2024-09-10T19:14:37.460+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:37.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[26469]: debug 2024-09-10T19:14:37.460+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:37.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[34072]: debug 2024-09-10T19:14:37.548+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:37.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:37 smithi165 bash[34072]: debug 2024-09-10T19:14:37.548+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:38.446 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[48829]: debug 2024-09-10T19:14:38.140+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:38.447 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[48829]: debug 2024-09-10T19:14:38.140+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:38.447 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[41096]: debug 2024-09-10T19:14:38.244+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:38.447 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[41096]: debug 2024-09-10T19:14:38.244+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:38.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[26469]: debug 2024-09-10T19:14:38.444+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.771618+0000 front 2024-09-10T19:14:02.771807+0000 (oldest deadline 2024-09-10T19:14:27.372325+0000) 2024-09-10T19:14:38.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[26469]: debug 2024-09-10T19:14:38.444+0000 7f37dce4d640 -1 osd.4 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:07.373266+0000 front 2024-09-10T19:14:07.373531+0000 (oldest deadline 2024-09-10T19:14:33.273145+0000) 2024-09-10T19:14:38.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[34072]: debug 2024-09-10T19:14:38.528+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:38.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:38 smithi165 bash[34072]: debug 2024-09-10T19:14:38.528+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:39.554 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[41096]: debug 2024-09-10T19:14:39.220+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:39.555 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[41096]: debug 2024-09-10T19:14:39.220+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:39.555 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[48829]: debug 2024-09-10T19:14:39.116+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:39.555 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[48829]: debug 2024-09-10T19:14:39.116+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:39.612 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[34072]: debug 2024-09-10T19:14:39.548+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:39.612 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[34072]: debug 2024-09-10T19:14:39.548+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:39.862 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[100269]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-4 2024-09-10T19:14:40.265 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 10 19:14:39 smithi165 bash[100343]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-4 2024-09-10T19:14:40.265 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[48829]: debug 2024-09-10T19:14:40.132+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:40.265 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[48829]: debug 2024-09-10T19:14:40.132+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:40.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[41096]: debug 2024-09-10T19:14:40.259+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:40.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[41096]: debug 2024-09-10T19:14:40.259+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:40.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[34072]: debug 2024-09-10T19:14:40.519+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:40.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:40 smithi165 bash[34072]: debug 2024-09-10T19:14:40.519+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:41.338 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[48829]: debug 2024-09-10T19:14:41.115+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:41.338 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[48829]: debug 2024-09-10T19:14:41.115+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:41.338 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[41096]: debug 2024-09-10T19:14:41.247+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:41.338 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[41096]: debug 2024-09-10T19:14:41.247+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:41.612 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[34072]: debug 2024-09-10T19:14:41.515+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:41.612 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:41 smithi165 bash[34072]: debug 2024-09-10T19:14:41.515+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:41.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.4.service' 2024-09-10T19:14:42.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:42.006 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-10T19:14:42.006 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-10T19:14:42.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.5 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 systemd[1]: Stopping Ceph osd.5 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.159+0000 7fd5ede81640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.159+0000 7fd5ede81640 -1 osd.5 904 *** Got signal Terminated *** 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.159+0000 7fd5ede81640 -1 osd.5 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[48829]: debug 2024-09-10T19:14:42.151+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:42.243 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[48829]: debug 2024-09-10T19:14:42.151+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:42.564 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[41096]: debug 2024-09-10T19:14:42.235+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:42.564 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[41096]: debug 2024-09-10T19:14:42.235+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:42.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.555+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:42.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.555+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:42.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:42 smithi165 bash[34072]: debug 2024-09-10T19:14:42.555+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.864719+0000 front 2024-09-10T19:14:20.865052+0000 (oldest deadline 2024-09-10T19:14:41.964756+0000) 2024-09-10T19:14:43.540 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[41096]: debug 2024-09-10T19:14:43.199+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:43.540 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[41096]: debug 2024-09-10T19:14:43.199+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:43.541 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[48829]: debug 2024-09-10T19:14:43.115+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:43.541 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[48829]: debug 2024-09-10T19:14:43.115+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:43.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[34072]: debug 2024-09-10T19:14:43.531+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:43.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[34072]: debug 2024-09-10T19:14:43.531+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:43.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:43 smithi165 bash[34072]: debug 2024-09-10T19:14:43.531+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.864719+0000 front 2024-09-10T19:14:20.865052+0000 (oldest deadline 2024-09-10T19:14:41.964756+0000) 2024-09-10T19:14:44.573 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[41096]: debug 2024-09-10T19:14:44.183+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:44.573 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[41096]: debug 2024-09-10T19:14:44.183+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:44.573 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[48829]: debug 2024-09-10T19:14:44.123+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:44.573 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[48829]: debug 2024-09-10T19:14:44.123+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:44.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[34072]: debug 2024-09-10T19:14:44.567+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:44.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[34072]: debug 2024-09-10T19:14:44.567+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:44.862 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:44 smithi165 bash[34072]: debug 2024-09-10T19:14:44.567+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.864719+0000 front 2024-09-10T19:14:20.865052+0000 (oldest deadline 2024-09-10T19:14:41.964756+0000) 2024-09-10T19:14:45.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[41096]: debug 2024-09-10T19:14:45.207+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:45.614 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[41096]: debug 2024-09-10T19:14:45.207+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:45.614 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[41096]: debug 2024-09-10T19:14:45.207+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:45.614 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[48829]: debug 2024-09-10T19:14:45.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:45.614 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[48829]: debug 2024-09-10T19:14:45.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:46.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[34072]: debug 2024-09-10T19:14:45.615+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:46.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[34072]: debug 2024-09-10T19:14:45.615+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:46.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:45 smithi165 bash[34072]: debug 2024-09-10T19:14:45.615+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.864719+0000 front 2024-09-10T19:14:20.865052+0000 (oldest deadline 2024-09-10T19:14:41.964756+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[41096]: debug 2024-09-10T19:14:46.251+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[41096]: debug 2024-09-10T19:14:46.251+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[41096]: debug 2024-09-10T19:14:46.251+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[48829]: debug 2024-09-10T19:14:46.175+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[48829]: debug 2024-09-10T19:14:46.175+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:46.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[48829]: debug 2024-09-10T19:14:46.175+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:47.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[34072]: debug 2024-09-10T19:14:46.655+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:02.761021+0000 front 2024-09-10T19:14:02.760898+0000 (oldest deadline 2024-09-10T19:14:26.261026+0000) 2024-09-10T19:14:47.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[34072]: debug 2024-09-10T19:14:46.655+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:12.162775+0000 front 2024-09-10T19:14:12.162796+0000 (oldest deadline 2024-09-10T19:14:33.262556+0000) 2024-09-10T19:14:47.112 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:46 smithi165 bash[34072]: debug 2024-09-10T19:14:46.655+0000 7fd5ea499640 -1 osd.5 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.864719+0000 front 2024-09-10T19:14:20.865052+0000 (oldest deadline 2024-09-10T19:14:41.964756+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[41096]: debug 2024-09-10T19:14:47.243+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[41096]: debug 2024-09-10T19:14:47.243+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[41096]: debug 2024-09-10T19:14:47.243+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[48829]: debug 2024-09-10T19:14:47.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[48829]: debug 2024-09-10T19:14:47.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:47.583 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:47 smithi165 bash[48829]: debug 2024-09-10T19:14:47.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:48.362 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[100494]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-5 2024-09-10T19:14:48.362 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[100568]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-5 2024-09-10T19:14:48.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[48829]: debug 2024-09-10T19:14:48.135+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:48.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[48829]: debug 2024-09-10T19:14:48.135+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:48.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[48829]: debug 2024-09-10T19:14:48.135+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:48.363 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[41096]: debug 2024-09-10T19:14:48.263+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:48.363 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[41096]: debug 2024-09-10T19:14:48.263+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:48.363 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:48 smithi165 bash[41096]: debug 2024-09-10T19:14:48.263+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:49.599 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[41096]: debug 2024-09-10T19:14:49.231+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:49.599 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[41096]: debug 2024-09-10T19:14:49.231+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:49.599 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[41096]: debug 2024-09-10T19:14:49.231+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:49.600 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[48829]: debug 2024-09-10T19:14:49.123+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:49.600 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[48829]: debug 2024-09-10T19:14:49.123+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:49.600 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:49 smithi165 bash[48829]: debug 2024-09-10T19:14:49.123+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:50.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.5.service' 2024-09-10T19:14:50.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:50.217 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-10T19:14:50.217 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-10T19:14:50.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.6 2024-09-10T19:14:50.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[48829]: debug 2024-09-10T19:14:50.171+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:50.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[48829]: debug 2024-09-10T19:14:50.171+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[48829]: debug 2024-09-10T19:14:50.171+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.179+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.179+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.179+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 systemd[1]: Stopping Ceph osd.6 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.419+0000 7fa5bf493640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.419+0000 7fa5bf493640 -1 osd.6 904 *** Got signal Terminated *** 2024-09-10T19:14:50.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:50 smithi165 bash[41096]: debug 2024-09-10T19:14:50.419+0000 7fa5bf493640 -1 osd.6 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:51.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[41096]: debug 2024-09-10T19:14:51.163+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:51.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[41096]: debug 2024-09-10T19:14:51.163+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:51.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[41096]: debug 2024-09-10T19:14:51.163+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:51.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[48829]: debug 2024-09-10T19:14:51.163+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:51.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[48829]: debug 2024-09-10T19:14:51.163+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:51.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:51 smithi165 bash[48829]: debug 2024-09-10T19:14:51.163+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:52.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[41096]: debug 2024-09-10T19:14:52.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:52.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[41096]: debug 2024-09-10T19:14:52.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:52.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[41096]: debug 2024-09-10T19:14:52.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:52.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[48829]: debug 2024-09-10T19:14:52.191+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:52.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[48829]: debug 2024-09-10T19:14:52.191+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:52.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:52 smithi165 bash[48829]: debug 2024-09-10T19:14:52.191+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:53.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[41096]: debug 2024-09-10T19:14:53.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:53.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[41096]: debug 2024-09-10T19:14:53.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:53.613 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[41096]: debug 2024-09-10T19:14:53.191+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:53.614 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[48829]: debug 2024-09-10T19:14:53.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:53.614 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[48829]: debug 2024-09-10T19:14:53.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:53.614 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:53 smithi165 bash[48829]: debug 2024-09-10T19:14:53.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:54.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[41096]: debug 2024-09-10T19:14:54.171+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:54.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[41096]: debug 2024-09-10T19:14:54.171+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:54.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[41096]: debug 2024-09-10T19:14:54.171+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:54.612 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[41096]: debug 2024-09-10T19:14:54.171+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:28.743325+0000 front 2024-09-10T19:14:28.743200+0000 (oldest deadline 2024-09-10T19:14:53.943948+0000) 2024-09-10T19:14:54.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[48829]: debug 2024-09-10T19:14:54.107+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:54.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[48829]: debug 2024-09-10T19:14:54.107+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:54.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[48829]: debug 2024-09-10T19:14:54.107+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:54.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:54 smithi165 bash[48829]: debug 2024-09-10T19:14:54.107+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[41096]: debug 2024-09-10T19:14:55.203+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:13:58.937226+0000 front 2024-09-10T19:13:58.936954+0000 (oldest deadline 2024-09-10T19:14:27.137886+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[41096]: debug 2024-09-10T19:14:55.203+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:11.140552+0000 front 2024-09-10T19:14:11.140617+0000 (oldest deadline 2024-09-10T19:14:34.640176+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[41096]: debug 2024-09-10T19:14:55.203+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:20.541758+0000 front 2024-09-10T19:14:20.542063+0000 (oldest deadline 2024-09-10T19:14:44.641644+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[41096]: debug 2024-09-10T19:14:55.203+0000 7fa5bcaad640 -1 osd.6 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:28.743325+0000 front 2024-09-10T19:14:28.743200+0000 (oldest deadline 2024-09-10T19:14:53.943948+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[48829]: debug 2024-09-10T19:14:55.099+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:55.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[48829]: debug 2024-09-10T19:14:55.099+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:55.363 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[48829]: debug 2024-09-10T19:14:55.099+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:55.363 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:55 smithi165 bash[48829]: debug 2024-09-10T19:14:55.099+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:14:56.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[48829]: debug 2024-09-10T19:14:56.103+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:56.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[48829]: debug 2024-09-10T19:14:56.103+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:56.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[48829]: debug 2024-09-10T19:14:56.103+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:56.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[48829]: debug 2024-09-10T19:14:56.103+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:14:56.793 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[100708]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-6 2024-09-10T19:14:56.793 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 10 19:14:56 smithi165 bash[100780]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-6 2024-09-10T19:14:57.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:57 smithi165 bash[48829]: debug 2024-09-10T19:14:57.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:57.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:57 smithi165 bash[48829]: debug 2024-09-10T19:14:57.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:57.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:57 smithi165 bash[48829]: debug 2024-09-10T19:14:57.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:57.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:57 smithi165 bash[48829]: debug 2024-09-10T19:14:57.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:14:58.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:58.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:58.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:58.362 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.095+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:14:58.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.6.service' 2024-09-10T19:14:58.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:14:58.639 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-10T19:14:58.639 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-10T19:14:58.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.7 2024-09-10T19:14:58.862 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 systemd[1]: Stopping Ceph osd.7 for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:14:58.862 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.787+0000 7f1096e6e640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:14:58.862 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.787+0000 7f1096e6e640 -1 osd.7 904 *** Got signal Terminated *** 2024-09-10T19:14:58.862 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:58 smithi165 bash[48829]: debug 2024-09-10T19:14:58.787+0000 7f1096e6e640 -1 osd.7 904 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:14:59.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:59 smithi165 bash[48829]: debug 2024-09-10T19:14:59.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:14:59.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:59 smithi165 bash[48829]: debug 2024-09-10T19:14:59.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:14:59.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:59 smithi165 bash[48829]: debug 2024-09-10T19:14:59.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:14:59.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:14:59 smithi165 bash[48829]: debug 2024-09-10T19:14:59.143+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:15:00.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:00 smithi165 bash[48829]: debug 2024-09-10T19:15:00.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:15:00.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:00 smithi165 bash[48829]: debug 2024-09-10T19:15:00.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:15:00.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:00 smithi165 bash[48829]: debug 2024-09-10T19:15:00.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:15:00.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:00 smithi165 bash[48829]: debug 2024-09-10T19:15:00.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:15:00.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:00 smithi165 bash[48829]: debug 2024-09-10T19:15:00.131+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-10T19:14:35.901523+0000 front 2024-09-10T19:14:35.901570+0000 (oldest deadline 2024-09-10T19:15:00.001480+0000) 2024-09-10T19:15:01.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:01 smithi165 bash[48829]: debug 2024-09-10T19:15:01.147+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:15:01.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:01 smithi165 bash[48829]: debug 2024-09-10T19:15:01.147+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:15:01.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:01 smithi165 bash[48829]: debug 2024-09-10T19:15:01.147+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:15:01.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:01 smithi165 bash[48829]: debug 2024-09-10T19:15:01.147+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:15:01.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:01 smithi165 bash[48829]: debug 2024-09-10T19:15:01.147+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-10T19:14:35.901523+0000 front 2024-09-10T19:14:35.901570+0000 (oldest deadline 2024-09-10T19:15:00.001480+0000) 2024-09-10T19:15:02.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:02 smithi165 bash[48829]: debug 2024-09-10T19:15:02.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:15:02.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:02 smithi165 bash[48829]: debug 2024-09-10T19:15:02.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:15:02.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:02 smithi165 bash[48829]: debug 2024-09-10T19:15:02.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:15:02.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:02 smithi165 bash[48829]: debug 2024-09-10T19:15:02.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:15:02.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:02 smithi165 bash[48829]: debug 2024-09-10T19:15:02.139+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-10T19:14:35.901523+0000 front 2024-09-10T19:14:35.901570+0000 (oldest deadline 2024-09-10T19:15:00.001480+0000) 2024-09-10T19:15:03.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:03 smithi165 bash[48829]: debug 2024-09-10T19:15:03.183+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6803 osd.0 since back 2024-09-10T19:14:00.893646+0000 front 2024-09-10T19:14:00.894045+0000 (oldest deadline 2024-09-10T19:14:26.793700+0000) 2024-09-10T19:15:03.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:03 smithi165 bash[48829]: debug 2024-09-10T19:15:03.183+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6807 osd.1 since back 2024-09-10T19:14:09.695586+0000 front 2024-09-10T19:14:09.695749+0000 (oldest deadline 2024-09-10T19:14:35.595476+0000) 2024-09-10T19:15:03.612 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:03 smithi165 bash[48829]: debug 2024-09-10T19:15:03.183+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6811 osd.2 since back 2024-09-10T19:14:21.398016+0000 front 2024-09-10T19:14:21.397927+0000 (oldest deadline 2024-09-10T19:14:46.097981+0000) 2024-09-10T19:15:03.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:03 smithi165 bash[48829]: debug 2024-09-10T19:15:03.183+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.83:6815 osd.3 since back 2024-09-10T19:14:30.100208+0000 front 2024-09-10T19:14:30.100485+0000 (oldest deadline 2024-09-10T19:14:53.600261+0000) 2024-09-10T19:15:03.613 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:03 smithi165 bash[48829]: debug 2024-09-10T19:15:03.183+0000 7f1093486640 -1 osd.7 904 heartbeat_check: no reply from 172.21.15.165:6802 osd.4 since back 2024-09-10T19:14:35.901523+0000 front 2024-09-10T19:14:35.901570+0000 (oldest deadline 2024-09-10T19:15:00.001480+0000) 2024-09-10T19:15:05.101 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:04 smithi165 bash[100919]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-7 2024-09-10T19:15:05.102 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 10 19:15:04 smithi165 bash[100996]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-osd-7 2024-09-10T19:15:06.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@osd.7.service' 2024-09-10T19:15:06.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:15:06.870 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-10T19:15:06.870 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-10T19:15:06.870 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-10T19:15:07.291 INFO:journalctl@ceph.rgw.foo.a.smithi083.stdout:Sep 10 19:15:06 smithi083 systemd[1]: Stopping Ceph rgw.foo.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:07.291 INFO:journalctl@ceph.rgw.foo.a.smithi083.stdout:Sep 10 19:15:07 smithi083 bash[63728]: debug 2024-09-10T19:15:07.053+0000 7fb3f77fe640 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-09-10T19:15:07.291 INFO:journalctl@ceph.rgw.foo.a.smithi083.stdout:Sep 10 19:15:07 smithi083 bash[63728]: debug 2024-09-10T19:15:07.053+0000 7fb4002727c0 -1 shutting down 2024-09-10T19:15:12.042 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:11 smithi083 bash[67687]: ts=2024-09-10T19:15:11.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-09-10T19:15:12.042 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:11 smithi083 bash[67687]: ts=2024-09-10T19:15:11.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi165.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.165:8443: connect: connection refused" 2024-09-10T19:15:17.871 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-10T19:15:17.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:15:17.905 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-10T19:15:17.905 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-10T19:15:17.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@prometheus.a 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:17 smithi165 systemd[1]: Stopping Ceph prometheus.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T19:15:18.273 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.087Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T19:15:18.274 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.089Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T19:15:18.274 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.089Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T19:15:18.274 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 10 19:15:18 smithi165 bash[58330]: ts=2024-09-10T19:15:18.089Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T19:15:18.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-10T19:15:18.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:15:18.774 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-10T19:15:18.774 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:15:22.041 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:21 smithi083 bash[67687]: ts=2024-09-10T19:15:21.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi165.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.165:8443: connect: connection refused" 2024-09-10T19:15:22.041 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:21 smithi083 bash[67687]: ts=2024-09-10T19:15:21.629Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi083.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.83:8443: connect: connection refused" 2024-09-10T19:15:22.041 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:21 smithi083 bash[67687]: ts=2024-09-10T19:15:21.630Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi165.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.165:8443: connect: connection refused" 2024-09-10T19:15:25.652 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:25 smithi083 systemd[1]: Stopping Ceph node-exporter.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:26.291 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:25 smithi083 bash[223789]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-node-exporter-a 2024-09-10T19:15:26.291 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:26 smithi083 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:15:26.291 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:26 smithi083 bash[223864]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-node-exporter-a 2024-09-10T19:15:26.291 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:26 smithi083 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-10T19:15:26.291 INFO:journalctl@ceph.node-exporter.a.smithi083.stdout:Sep 10 19:15:26 smithi083 systemd[1]: Stopped Ceph node-exporter.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:15:26.791 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:26 smithi083 systemd[1]: Stopping Ceph alertmanager.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:26.791 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:26 smithi083 bash[67687]: ts=2024-09-10T19:15:26.550Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:15:27.541 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:27 smithi083 bash[223925]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:15:27.541 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:27 smithi083 bash[223998]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:15:27.541 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:27 smithi083 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@alertmanager.a.service: Succeeded. 2024-09-10T19:15:27.541 INFO:journalctl@ceph.alertmanager.a.smithi083.stdout:Sep 10 19:15:27 smithi083 systemd[1]: Stopped Ceph alertmanager.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:15:28.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:15:34.362 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:34 smithi165 systemd[1]: Stopping Ceph iscsi.iscsi.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:34.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:34 smithi165 bash[54535]: debug Shutdown received 2024-09-10T19:15:45.025 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:44 smithi165 bash[101527]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-10T19:15:45.317 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-10T19:15:45.317 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 bash[101610]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-10T19:15:45.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 bash[101618]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-10T19:15:45.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 bash[101665]: rm: cannot remove '/run/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@iscsi.iscsi.a.tcmu.service-pid': No such file or directory 2024-09-10T19:15:45.612 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 bash[101666]: rm: cannot remove '/run/ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@iscsi.iscsi.a.tcmu.service-cid': No such file or directory 2024-09-10T19:15:45.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-10T19:15:45.903 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 10 19:15:45 smithi165 systemd[1]: Stopped Ceph iscsi.iscsi.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:15:46.273 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 systemd[1]: Stopping Ceph node-exporter.b for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:46.694 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 bash[101745]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-node-exporter-b 2024-09-10T19:15:47.110 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:15:47.110 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 bash[101820]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-node-exporter-b 2024-09-10T19:15:47.110 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-10T19:15:47.110 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 10 19:15:46 smithi165 systemd[1]: Stopped Ceph node-exporter.b for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:15:47.362 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 systemd[1]: Stopping Ceph grafana.a for 707a4ada-6fa3-11ef-bcea-c7b262605968... 2024-09-10T19:15:47.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 bash[101882]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-grafana-a 2024-09-10T19:15:48.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:15:48.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 bash[101957]: Error response from daemon: No such container: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968-grafana-a 2024-09-10T19:15:48.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 systemd[1]: ceph-707a4ada-6fa3-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-10T19:15:48.079 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 10 19:15:47 smithi165 systemd[1]: Stopped Ceph grafana.a for 707a4ada-6fa3-11ef-bcea-c7b262605968. 2024-09-10T19:15:48.547 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:15:48.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:15:48.580 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-10T19:15:48.581 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi083/crash 2024-09-10T19:15:48.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:15:48.597 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:15:48.597 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:15:48.599 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi165/crash 2024-09-10T19:15:48.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:15:48.614 INFO:teuthology.orchestra.run.smithi165.stderr:tar: /var/lib/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:15:48.614 INFO:teuthology.orchestra.run.smithi165.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:15:48.615 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-10T19:15:48.616 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-09-10T19:15:48.660 INFO:teuthology.orchestra.run.smithi083.stderr:grep: /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-10T19:15:48.662 INFO:tasks.cephadm:Compressing logs... 2024-09-10T19:15:48.662 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:15:48.664 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:15:48.673 INFO:teuthology.orchestra.run.smithi083.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-10T19:15:48.681 INFO:teuthology.orchestra.run.smithi165.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-09-10T19:15:48.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-10T19:15:48.685 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-10T19:15:48.686 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-10T19:15:48.687 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/objectstore_tool.7.log 2024-09-10T19:15:48.687 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: gzip /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.y.log: -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-10T19:15:48.688 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/objectstore_tool.7.log: --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-10T19:15:48.688 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-10T19:15:48.690 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.1.log: -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-10T19:15:48.690 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:15:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-10T19:15:48.694 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-10T19:15:48.696 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-10T19:15:48.696 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-10T19:15:48.696 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.x.log: /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/tcmu-runner.log 2024-09-10T19:15:48.697 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-10T19:15:48.697 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:15:48.699 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/tcmu-runner.log: /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 63.1% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-10T19:15:48.699 INFO:teuthology.orchestra.run.smithi165.stderr:--verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-10T19:15:48.699 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:15:48.710 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.7.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:15:48.711 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.3.log: 95.2% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/objectstore_tool.7.log.gz 2024-09-10T19:15:48.712 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-10T19:15:48.713 INFO:teuthology.orchestra.run.smithi165.stderr: 92.2% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-10T19:15:48.717 INFO:teuthology.orchestra.run.smithi165.stderr: 90.7% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:15:48.731 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:15:48.731 INFO:teuthology.orchestra.run.smithi083.stderr: 90.7% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:15:48.749 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.2.log: /var/log/ceph/cephadm.log: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:15:48.934 INFO:teuthology.orchestra.run.smithi083.stderr: 93.5% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-10T19:15:49.165 INFO:teuthology.orchestra.run.smithi083.stderr: 90.2% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-10T19:15:50.498 INFO:teuthology.orchestra.run.smithi083.stderr: 91.8% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-10T19:15:50.702 INFO:teuthology.orchestra.run.smithi165.stderr: 92.2% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-10T19:15:51.399 INFO:teuthology.orchestra.run.smithi083.stderr: 91.2% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-10T19:15:54.702 INFO:teuthology.orchestra.run.smithi165.stderr: 93.9% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-10T19:15:54.828 INFO:teuthology.orchestra.run.smithi083.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-10T19:15:54.843 INFO:teuthology.orchestra.run.smithi165.stderr: 93.9% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-10T19:15:55.184 INFO:teuthology.orchestra.run.smithi165.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-10T19:15:55.541 INFO:teuthology.orchestra.run.smithi083.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-10T19:15:55.680 INFO:teuthology.orchestra.run.smithi083.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-10T19:15:55.739 INFO:teuthology.orchestra.run.smithi165.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-10T19:15:55.741 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T19:15:55.741 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m7.072s 2024-09-10T19:15:55.741 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m26.818s 2024-09-10T19:15:55.741 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.941s 2024-09-10T19:15:56.013 INFO:teuthology.orchestra.run.smithi083.stderr: 94.0% -- replaced with /var/log/ceph/707a4ada-6fa3-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-10T19:15:56.013 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-10T19:15:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m7.347s 2024-09-10T19:15:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m31.506s 2024-09-10T19:15:56.014 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m1.050s 2024-09-10T19:15:56.015 INFO:tasks.cephadm:Archiving logs... 2024-09-10T19:15:56.015 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi083/log 2024-09-10T19:15:56.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:16:00.336 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi165/log 2024-09-10T19:16:00.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:16:04.217 INFO:tasks.cephadm:Removing cluster... 2024-09-10T19:16:04.218 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 --force 2024-09-10T19:16:05.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 707a4ada-6fa3-11ef-bcea-c7b262605968 --force 2024-09-10T19:16:07.370 INFO:tasks.cephadm:Removing cephadm ... 2024-09-10T19:16:07.370 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T19:16:07.378 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T19:16:07.382 INFO:tasks.cephadm:Teardown complete 2024-09-10T19:16:07.383 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T19:16:07.402 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T19:16:07.402 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:16:07.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:16:07.493 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T19:16:07.505 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T19:16:07.505 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:16:07.507 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:16:07.529 INFO:teuthology.orchestra.run.smithi083.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T19:16:07.529 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================== 2024-09-10T19:16:07.529 INFO:teuthology.orchestra.run.smithi083.stdout:*hv01.front.sepi 67.205.162.81 3 u 80 128 377 0.109 -0.350 0.100 2024-09-10T19:16:07.530 INFO:teuthology.orchestra.run.smithi083.stdout:+hv02.front.sepi 63.231.80.2 3 u 72 128 377 0.093 -1.664 0.073 2024-09-10T19:16:07.530 INFO:teuthology.orchestra.run.smithi083.stdout:+hv03.front.sepi 74.6.168.72 3 u 75 128 377 0.085 0.909 0.170 2024-09-10T19:16:07.530 INFO:teuthology.orchestra.run.smithi083.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout:*hv01.front.sepi 67.205.162.81 3 u 82 128 377 0.102 1.616 0.153 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 128 377 0.111 0.291 0.240 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout:+hv03.front.sepi 74.6.168.72 3 u 58 128 377 0.081 2.676 0.154 2024-09-10T19:16:07.531 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-10T19:16:07.532 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T19:16:07.541 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T19:16:07.541 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T19:16:07.555 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T19:16:07.574 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T19:16:07.588 INFO:teuthology.task.internal:Duration was 2906.704332 seconds 2024-09-10T19:16:07.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T19:16:07.603 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T19:16:07.604 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:16:07.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:16:07.656 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T19:16:07.656 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-09-10T19:16:07.657 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:16:07.710 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T19:16:07.710 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:16:07.725 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T19:16:07.726 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:16:07.728 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:16:07.858 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T19:16:07.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:16:07.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:16:08.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T19:16:08.239 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T19:16:08.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:16:08.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:16:08.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T19:16:08.281 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:16:08.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:16:08.295 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-09-10T19:16:08.298 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-09-10T19:16:08.324 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:16:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.329 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:16:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T19:16:08.384 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T19:16:08.384 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi083 2024-09-10T19:16:08.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:16:08.442 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900236/remote/smithi165 2024-09-10T19:16:08.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:16:08.481 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T19:16:08.481 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:16:08.483 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:16:08.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T19:16:08.503 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T19:16:08.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T19:16:08.513 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T19:16:08.513 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:16:08.532 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:16:08.537 INFO:teuthology.orchestra.run.smithi083.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:16 /home/ubuntu/cephtest 2024-09-10T19:16:08.540 INFO:teuthology.orchestra.run.smithi165.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:16 /home/ubuntu/cephtest 2024-09-10T19:16:08.542 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T19:16:08.554 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T19:16:08.584 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 2906.704332113266 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-10T19:16:08.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:16:08.654 INFO:teuthology.run:pass